From patchwork Tue Apr 23 05:54:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911935 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C0EC922 for ; Tue, 23 Apr 2019 05:54:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E67028733 for ; Tue, 23 Apr 2019 05:54:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01D8028738; Tue, 23 Apr 2019 05:54:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FA8028733 for ; Tue, 23 Apr 2019 05:54:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC0406B0007; Tue, 23 Apr 2019 01:54:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A6FB06B0008; Tue, 23 Apr 2019 01:54:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 936816B000A; Tue, 23 Apr 2019 01:54:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 6F4706B0007 for ; Tue, 23 Apr 2019 01:54:39 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id e13so9177113qkl.8 for ; Mon, 22 Apr 2019 22:54:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Fc5tzu6C1dq4dNrmHXxBLR13rAWvqP8agDt0wYDJxgY=; b=JcsAAubcygl0Wdvoj8lLKiEFHL7vhRrW2+7aCZNQ6FKiBGsebyH9qsm1j3yE52x9uD H43UwvWH3A0dFQv7AkJs2L1HPTVrxwWVS7ag7uddfLW3M6RbQsJW7buJO7cupCkSzaN5 J3tQ8BbSbsb4a7XKidoCc82SYTO5nRuAF9L6RAXRP+R4P8PexMAXf1QmnHHiMpHS1xr5 /q5yhQ37bmylwBFPe4envP962YYEuFfIQqcV0Vw2hLaYnE5bv2EXIa6GeHPdWL8s3dTT /N5eIBcZhIgdj4OTHzFjBQs6KQyfIP/0s/d9Z1EfqyA/uD/KvcJTKfpalJ6eB1iD1mpJ YjpA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAXkLailYmwP7idzioMyYmxwAH8ura6244P9XhO62rrynQdTrw7j X5MUX6xYNc/Cml0PV90xC+Zwh1EDuEs+fFUOiuq/Fs2z+/kPeJz6MSXREWkXWpUD3VfxkNAyG1K iPS/RY814Lv9GN95OZJaX3uWA12Fm1C1AznHwisc0eiwIzOm5Hsz70H2fV/k1aRYI6A== X-Received: by 2002:a37:2e43:: with SMTP id u64mr17143508qkh.340.1555998879219; Mon, 22 Apr 2019 22:54:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqwBfl2ycdXeWxC37tN9xu2nicVssqquFWikJM53KZ+zCGF1gQCMet+Dryia4J7CfV9Rzek8 X-Received: by 2002:a37:2e43:: with SMTP id u64mr17143479qkh.340.1555998878198; Mon, 22 Apr 2019 22:54:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998878; cv=none; d=google.com; s=arc-20160816; b=dBLUXwBYdklgjnF+CKf3PyKCRTsrnOt+fcQslrzoRVbIKe6Q3gxmqXwu0kxITQkBB2 0IB7wC0Ju/Os5T49+6a4xpMzB4hqMYNkHiDKOdEiNKWxT6oomrjL1xrnbU/8k8IZIudI tawHAlDd9/51bBxzDLBgVxtpvrVV0ATflo27aOKCdUZRYN2vUg6cY5lgrfF4Z/Gcpq5X wlNJrUvn3oNJnjhRh4v2oqvtNWAHxd3AWtgx03aWtnATajjlu5ax2y/Gb8JOVkwvlGTb OmMQoK45foJPa3w0mkuhP2x8lH583MSzAWszji0eGprPo6XU5mon+EIY/uXfUUspnyS/ MxgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Fc5tzu6C1dq4dNrmHXxBLR13rAWvqP8agDt0wYDJxgY=; b=ncWkyfGd9hPRw1pwDmuK3eOGpgcTwYuYh5IOnqtFlljWbPt5NLh+ou+1ipzPdSCi1P /QyaHTP6QqEup+r/Re7tWBvwKRXy7V6aJ3hBsOazZNe3hcof05qvVmY22r2DmWiWcIz5 0uCPrANcGisxCJmSEtO46JMmVTqmbKSVQkF5hoE9aZP1KlWTwoiwbSFpHjOl3G2HpJOb CaS4uGvfgqjZOSvpb1nmMfY3LvFnIpeaIngTSC3e9oKbEryOZO6hAHUpgQy0FHRehaJN +0M4Abwk+vm0YptRGTOd2wJ4RgEcdSEw4WclHIkSwFSjt+G/9cGTL1iSoDv1zp9XPEOn HkfA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id 7si3742853qtz.236.2019.04.22.22.54.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:54:38 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 62C47307D8BE; Tue, 23 Apr 2019 05:54:37 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5F53119C7E; Tue, 23 Apr 2019 05:54:32 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 1/6] vhost: generalize adding used elem Date: Tue, 23 Apr 2019 01:54:15 -0400 Message-Id: <20190423055420.26408-2-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.48]); Tue, 23 Apr 2019 05:54:37 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use one generic vhost_copy_to_user() instead of two dedicated accessor. This will simplify the conversion to fine grain accessors. About 2% improvement of PPS were seen during vitio-user txonly test. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 351af88231ad..6df76ac8200a 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2255,16 +2255,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (count == 1) { - if (vhost_put_user(vq, heads[0].id, &used->id)) { - vq_err(vq, "Failed to write used id"); - return -EFAULT; - } - if (vhost_put_user(vq, heads[0].len, &used->len)) { - vq_err(vq, "Failed to write used len"); - return -EFAULT; - } - } else if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { vq_err(vq, "Failed to write used"); return -EFAULT; } From patchwork Tue Apr 23 05:54:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911945 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4DB9112C for ; Tue, 23 Apr 2019 05:54:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6F7F28733 for ; Tue, 23 Apr 2019 05:54:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BAA9328738; Tue, 23 Apr 2019 05:54:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C785B28733 for ; Tue, 23 Apr 2019 05:54:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A49556B000A; Tue, 23 Apr 2019 01:54:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9F9286B000C; Tue, 23 Apr 2019 01:54:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C1F96B000D; Tue, 23 Apr 2019 01:54:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 6825C6B000A for ; Tue, 23 Apr 2019 01:54:45 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 144so7211785qkf.12 for ; Mon, 22 Apr 2019 22:54:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=5+4MUJqHHmEntLU+rntoIA4WgcfBuduaxD6VHtuUKq4=; b=HkFfAe5blAjbYAuEjKlhhbonr2p8W89FT7thgNUeJ4PuU1PkOPqHWYdoo36TZYfgNj 90MoJ+J05Cc4DR5O3EJmMQP8fXb2Ju0fNQ4sqyqHXhee1AUgccFJ9GiDGSwTBAPartkp nwpZGSBpgNHQmRQC305zlF2752qayZscKcaolLqzih3+7m755WOPWijYM9FxNkEAWOnu NU07ZNkcmn/HSJII3GQJmHuvjTDxVACGNg73DSV8X/5r171CbHxo9C+8q3A5mg5cF49D ARlc/PBDld1MGCBKgpRVsXQr4RuFby7EOkIbJT3WPtByRw0seQ8Tl8Mzw6oXHXVEO18u X9TQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAV3tysR8D861zoAVomRhKQxoyPrOUSh8O7CWiYpA+dkUNGnmCvk g7ZKQfHI86VARO+dfCAHop0R/AduCyS8YtleIGPqpEmxWr+tL3dai9MOhT/tNcLrsSBFCyS51Ro i26poQNgimT/rcIYK9QzqftIJ6pVB4rP5OD0TsX3vLTkvESaBVOi5yuD5s01P4vO2DQ== X-Received: by 2002:a05:620a:1438:: with SMTP id k24mr17230386qkj.165.1555998885183; Mon, 22 Apr 2019 22:54:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqwf/WTTwG0pa6siIGCYVeDyenqPHxV8+NPsvpvNsK0JfcJPJwBRFxoGGQcxtTecmfBaKl3M X-Received: by 2002:a05:620a:1438:: with SMTP id k24mr17230336qkj.165.1555998883900; Mon, 22 Apr 2019 22:54:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998883; cv=none; d=google.com; s=arc-20160816; b=aXORDxTBuUa3wNoiK7JxCuodK1gLzQLxFYCOrOose4k6dAJ2sHTV1QCMBsxlYbutQa YUCCCPNOJSonM78Zx3Yu9+vGDO7ysr6ZQ3p1vJamWcp1svQ/HmhRg2sZ6q2dfd54WDIH zfe2WTRZq3v3aZicJR/9XA5fBo79DYjr4C7hbePeN26opU37iXHK1S1pTGgdLRFISMYl yYdhM64nFo0tMgNRc6S7L7JgduuD73ug/Nf8ds4Oo0BKTmv3M8L/5Ww4wBXd2YIwo5qA ahf8DpLVXdWpWjHXtGb35kMtykNEgRTSRjISOO/D/RHfL7PBUGVnjY43bP4BT28Tn+i0 f53A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=5+4MUJqHHmEntLU+rntoIA4WgcfBuduaxD6VHtuUKq4=; b=SAr0ZW2N+iToI4LfLg/IVd1lf/ASE2lnS6VBTi6R47Lm01PXPpadI0ICPET3HDZzBE 6igyc5EVEu9yWuAmUIEbdfm0RFrsLcAkkMf0jRhx68NtB86fVs7IHbvJTNAEYz/lp8d2 fcHTSfXw4yUvALrKZwBjhkmHy9VgI98cZi2NEcaScIelierxAp62z7xaEV8ufvEK3AiE lnn5NVDDvENL8rGq2YVPN/VqVWy35jnP4HYVbb4hdkLQPEgKnZJy1WcbWHB0mabBgsl3 93IW52NWXrvVJWbL9U7vinW7U6AoV8ZGae7W36XhbXLvidnXLuwBLNxzJUTL9ze2pIOK EifQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id m9si8372416qtp.48.2019.04.22.22.54.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:54:43 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 01BB185A04; Tue, 23 Apr 2019 05:54:43 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD4BF19C7E; Tue, 23 Apr 2019 05:54:37 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 2/6] vhost: fine grain userspace memory accessors Date: Tue, 23 Apr 2019 01:54:16 -0400 Message-Id: <20190423055420.26408-3-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Tue, 23 Apr 2019 05:54:43 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is used to hide the metadata address from virtqueue helpers. This will allow to implement a vmap based fast accessing to metadata. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 94 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 17 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 6df76ac8200a..7335f2275ed3 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -869,6 +869,34 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, ret; \ }) +static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), + vhost_avail_event(vq)); +} + +static inline int vhost_put_used(struct vhost_virtqueue *vq, + struct vring_used_elem *head, int idx, + int count) +{ + return vhost_copy_to_user(vq, vq->used->ring + idx, head, + count * sizeof(*head)); +} + +static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), + &vq->used->flags); +} + +static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), + &vq->used->idx); +} + #define vhost_get_user(vq, x, ptr, type) \ ({ \ int ret; \ @@ -907,6 +935,43 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) mutex_unlock(&d->vqs[i]->mutex); } +static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_avail(vq, *idx, &vq->avail->idx); +} + +static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, + __virtio16 *head, int idx) +{ + return vhost_get_avail(vq, *head, + &vq->avail->ring[idx & (vq->num - 1)]); +} + +static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, + __virtio16 *flags) +{ + return vhost_get_avail(vq, *flags, &vq->avail->flags); +} + +static inline int vhost_get_used_event(struct vhost_virtqueue *vq, + __virtio16 *event) +{ + return vhost_get_avail(vq, *event, vhost_used_event(vq)); +} + +static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_used(vq, *idx, &vq->used->idx); +} + +static inline int vhost_get_desc(struct vhost_virtqueue *vq, + struct vring_desc *desc, int idx) +{ + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); +} + static int vhost_new_umem_range(struct vhost_umem *umem, u64 start, u64 size, u64 end, u64 userspace_addr, int perm) @@ -1844,8 +1909,7 @@ EXPORT_SYMBOL_GPL(vhost_log_write); static int vhost_update_used_flags(struct vhost_virtqueue *vq) { void __user *used; - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), - &vq->used->flags) < 0) + if (vhost_put_used_flags(vq)) return -EFAULT; if (unlikely(vq->log_used)) { /* Make sure the flag is seen before log. */ @@ -1862,8 +1926,7 @@ static int vhost_update_used_flags(struct vhost_virtqueue *vq) static int vhost_update_avail_event(struct vhost_virtqueue *vq, u16 avail_event) { - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), - vhost_avail_event(vq))) + if (vhost_put_avail_event(vq)) return -EFAULT; if (unlikely(vq->log_used)) { void __user *used; @@ -1899,7 +1962,7 @@ int vhost_vq_init_access(struct vhost_virtqueue *vq) r = -EFAULT; goto err; } - r = vhost_get_used(vq, last_used_idx, &vq->used->idx); + r = vhost_get_used_idx(vq, &last_used_idx); if (r) { vq_err(vq, "Can't access used idx at %p\n", &vq->used->idx); @@ -2098,7 +2161,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, last_avail_idx = vq->last_avail_idx; if (vq->avail_idx == vq->last_avail_idx) { - if (unlikely(vhost_get_avail(vq, avail_idx, &vq->avail->idx))) { + if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { vq_err(vq, "Failed to access avail idx at %p\n", &vq->avail->idx); return -EFAULT; @@ -2125,8 +2188,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, /* Grab the next descriptor number they're advertising, and increment * the index we've seen. */ - if (unlikely(vhost_get_avail(vq, ring_head, - &vq->avail->ring[last_avail_idx & (vq->num - 1)]))) { + if (unlikely(vhost_get_avail_head(vq, &ring_head, last_avail_idx))) { vq_err(vq, "Failed to read head: idx %d address %p\n", last_avail_idx, &vq->avail->ring[last_avail_idx % vq->num]); @@ -2161,8 +2223,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, i, vq->num, head); return -EINVAL; } - ret = vhost_copy_from_user(vq, &desc, vq->desc + i, - sizeof desc); + ret = vhost_get_desc(vq, &desc, i); if (unlikely(ret)) { vq_err(vq, "Failed to get descriptor: idx %d addr %p\n", i, vq->desc + i); @@ -2255,7 +2316,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_put_used(vq, heads, start, count)) { vq_err(vq, "Failed to write used"); return -EFAULT; } @@ -2297,8 +2358,7 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, /* Make sure buffer is written before we update index. */ smp_wmb(); - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), - &vq->used->idx)) { + if (vhost_put_used_idx(vq)) { vq_err(vq, "Failed to increment used idx"); return -EFAULT; } @@ -2331,7 +2391,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { __virtio16 flags; - if (vhost_get_avail(vq, flags, &vq->avail->flags)) { + if (vhost_get_avail_flags(vq, &flags)) { vq_err(vq, "Failed to get flags"); return true; } @@ -2345,7 +2405,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (unlikely(!v)) return true; - if (vhost_get_avail(vq, event, vhost_used_event(vq))) { + if (vhost_get_used_event(vq, &event)) { vq_err(vq, "Failed to get used event idx"); return true; } @@ -2390,7 +2450,7 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (vq->avail_idx != vq->last_avail_idx) return false; - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (unlikely(r)) return false; vq->avail_idx = vhost16_to_cpu(vq, avail_idx); @@ -2426,7 +2486,7 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) /* They could have slipped one in as we were doing that: make * sure it's written, then check again. */ smp_mb(); - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (r) { vq_err(vq, "Failed to check avail idx at %p: %d\n", &vq->avail->idx, r); From patchwork Tue Apr 23 05:54:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA301922 for ; Tue, 23 Apr 2019 05:54:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCD6327D0C for ; Tue, 23 Apr 2019 05:54:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C11FD28113; Tue, 23 Apr 2019 05:54:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5534727D0C for ; Tue, 23 Apr 2019 05:54:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FF8D6B000D; Tue, 23 Apr 2019 01:54:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7AEE86B000E; Tue, 23 Apr 2019 01:54:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C5246B0010; Tue, 23 Apr 2019 01:54:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 4E0786B000D for ; Tue, 23 Apr 2019 01:54:55 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id o34so13803324qte.5 for ; Mon, 22 Apr 2019 22:54:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=p0LVi5P4xummSTVtr9ikiQ4kmmxsXBzgyT6m3PDySak=; b=Ww663fQJkCHEJUoZLeXRMVQkjy4FSGSTZwdnxVS2qYbjaPx+/4RaozC87tXaYOpsm5 bSRYyhb26XGYQxy+W5u5WsM8OqwsnUMrK7nOJwc53rOAXG02PIPwNIgbnCHHFgAsS3AN 4mhQ6ot0OVamyH7efxiM41pXmyJX6IVS2QKdDxXJeq/cWczvpkkvTdRV99V3OfPeTc2w Ewru2Lo2jmxH5aA8KDjwzXJiiJrDsAokX3um2Nn4ts2PzLwZmruy8jNWuMCwvGFcT+QT 0XZynPN24vyTJm3u3NdGmS6tbRCaSgkAigWfPZuC3FnjVP1JH7wc3wJ82M8z17pGNhXT YoEw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAXLcxSVN89RD7fh9mbR2GjNz3fDQisGL41oRXC1x6d5pwZ+0h7k 5Ej4U+ycGV6a/qB7GdLVfB8r3VPyZqj0q6jMXyd4eK8dVvUQLpvKflxXavXOS+9N8sIpG1vt+xY /9YWHxUzUTg5EK11ewN3C7s6UFy/tlNo7qLZq0PmAm7Gm+icAUfTRJhN+/Z5InL2gnQ== X-Received: by 2002:ac8:71c4:: with SMTP id i4mr18824549qtp.358.1555998895086; Mon, 22 Apr 2019 22:54:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqy6tpWV7P349pHN/Uw/IFwadoYHrr1Q2tfG5rO+vakuIzTjFckfd065Cly2GlfJJ8taE04T X-Received: by 2002:ac8:71c4:: with SMTP id i4mr18824502qtp.358.1555998894014; Mon, 22 Apr 2019 22:54:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998894; cv=none; d=google.com; s=arc-20160816; b=kNrnTbXz/BRzFn2bGWCgSpjfazBur47yrnIsiT7wjurZz9Vxl43Jdkey7Mzb3v9Lq+ jIFlXNnoajTt3x1u0gfxKi1KaTQXfPp4MvDU4JLHOBDnhzDnp2AT6uuicZJmnct3Un39 yOKkBevDmXNbJgiM0NhqnR5XYIXheLOpi14J16nrFs2C3xcDX6U0QZmDCmQw81r/JxdW iW2PhM50LVeKgyUF1ho3CvnT+yI3HwMeR7rAD7a8L6HApsUz5n9t43dA+Ps34yzOy3ng zejaKm+cu8YE21zDKGUqW2+xXTb17vVMJlQXNAZWUG0m386RkjIF8DakB9VdXYqqBI5G 08dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=p0LVi5P4xummSTVtr9ikiQ4kmmxsXBzgyT6m3PDySak=; b=hLISUodD7UDst1ac2V1lMINqYEtbIJiTeUAVywwxqQ21rq756+R9IOC2KYyqcmuqXa CgAxNFL/Pni77o+1AynTDHL2y+sfycn+e7VKh+foEFkNlozL8BFHNyoLlKVIaOo9Us7y fkY4t2nsC529ExXu3goqAm+E0v8wHs1fU4ndqnH+VpmmbwNEtggZZ7qp11ohrg5/GCNn ul/WGDAiajd56afj3Kybor7oUWs0WKuruHsTwDx3ztGzWB0KElg/grsrD9KOOlKCcEvR lkVuTah6TCwHsIFwzfoB/lkDtVhHLL1pQ7L0C+rOcjNO3KLVCwM1OqEXsjrNDw/E7NSm zfVA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id c10si1171825qvt.162.2019.04.22.22.54.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:54:54 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2483BF74AC; Tue, 23 Apr 2019 05:54:53 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD99D1A8F6; Tue, 23 Apr 2019 05:54:43 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 3/6] vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() Date: Tue, 23 Apr 2019 01:54:17 -0400 Message-Id: <20190423055420.26408-4-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 23 Apr 2019 05:54:53 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Rename the function to be more accurate since it actually tries to prefetch vq metadata address in IOTLB. And this will be used by following patch to prefetch metadata virtual addresses. Signed-off-by: Jason Wang --- drivers/vhost/net.c | 4 ++-- drivers/vhost/vhost.c | 4 ++-- drivers/vhost/vhost.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index df51a35cf537..bf55f995ebae 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -971,7 +971,7 @@ static void handle_tx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); @@ -1140,7 +1140,7 @@ static void handle_rx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 7335f2275ed3..bff4d586871d 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -1313,7 +1313,7 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, return true; } -int vq_iotlb_prefetch(struct vhost_virtqueue *vq) +int vq_meta_prefetch(struct vhost_virtqueue *vq) { size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; @@ -1332,7 +1332,7 @@ int vq_iotlb_prefetch(struct vhost_virtqueue *vq) num * sizeof(*vq->used->ring) + s, VHOST_ADDR_USED); } -EXPORT_SYMBOL_GPL(vq_iotlb_prefetch); +EXPORT_SYMBOL_GPL(vq_meta_prefetch); /* Can we log writes? */ /* Caller should have device mutex but not vq mutex */ diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 9490e7ddb340..7a7fc001265f 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -209,7 +209,7 @@ bool vhost_enable_notify(struct vhost_dev *, struct vhost_virtqueue *); int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, unsigned int log_num, u64 len, struct iovec *iov, int count); -int vq_iotlb_prefetch(struct vhost_virtqueue *vq); +int vq_meta_prefetch(struct vhost_virtqueue *vq); struct vhost_msg_node *vhost_new_msg(struct vhost_virtqueue *vq, int type); void vhost_enqueue_msg(struct vhost_dev *dev, From patchwork Tue Apr 23 05:54:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911961 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2952922 for ; Tue, 23 Apr 2019 05:55:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A582727D0C for ; Tue, 23 Apr 2019 05:55:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9997C28113; Tue, 23 Apr 2019 05:55:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A057A27D0C for ; Tue, 23 Apr 2019 05:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 989126B0010; Tue, 23 Apr 2019 01:55:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 90F016B0266; Tue, 23 Apr 2019 01:55:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FC906B0269; Tue, 23 Apr 2019 01:55:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 6019C6B0010 for ; Tue, 23 Apr 2019 01:55:01 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id o16so720670qtp.18 for ; Mon, 22 Apr 2019 22:55:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9ROmEdhXXSco/ULkOaE6eoNIg0SUqKfnbfR953ok6yc=; b=JAYwPqrYcB/fHlq0tGoS7ZmuMv+wGJWjydokqyiwkPlFXg+IUV52rC1lekKktaav9u SkSHNxJ7SprKNiWgTMhGJwC4b+WodzwTi9EpYokAMYZYpBjEpdPfIu/SD0p1b9paEdiY t3D3VqQyqyzfI9M5YIF4n8+zWEV4+8ov/IuSfnHfL2CQuvXnbvTHwTJtAeSzhQolj1ny uc7dUU3jYC5NCwh5UG+3opxucnj/F0Vxmh0Bo9nSZ57Mq7UYEtXSxD/1SOqKVlItRRg+ tApvQRjFZpx8TAhXoxCKLaZZ98gJw+MlAbFGscE2qos1Ym/dFUUi3UQf5rD0rkWYS/Yq zxZA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAWWTuKYMp40OAJ+5XL9mbpTtR+tlAcOxeNvxZIGSHt+hfYFOSYE YDQDJeTMJ4KUAFZMocqMlq8wzp6FLQKjlj/qma2iyViaTTyNQcM7t5TjQt0vv8OOzRSSEMZUuGA tVVVsidjT37B6EVXi2D7irRskKJ1P3Pxoc4VndqiFU1RH8+GYVapV1VRvlFKYPUkACg== X-Received: by 2002:a37:b7c1:: with SMTP id h184mr348670qkf.153.1555998901168; Mon, 22 Apr 2019 22:55:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqxJ2zPp6g6op6qAhmyi8VHjUXEv4X4uZFMcfXVEcZUj5LoTRGStFdcOoaQj93rpIuq3p9Y0 X-Received: by 2002:a37:b7c1:: with SMTP id h184mr348615qkf.153.1555998899515; Mon, 22 Apr 2019 22:54:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998899; cv=none; d=google.com; s=arc-20160816; b=LVa5nvAoHMSHZ7+44ECxcuYxjKqtrwI2FdxviAgT6eXmkJUKrnYv++w23AkSmociOt lGZuITSdL4MpktSrh7mF0Jh5jSAcljKG0EoZ2I/yK1zvScWwEpEPxjX3jZQlIHx/ugri cAdkcrmc1x/PIWk1K2MJ04lkn9pQ942CZofix5Uemx//oaipHaAVSxz583WnVdID3I+Y gnXNFpjcRvXf1Q9BY8tjKjgldWo1mdwBMT5XG0DZjTc2NrOUTZpcgSscvhDJFpaT90K8 T/AFrpx0FXRMtIfXmRe4PARzWeeghnMdsyCO3qFC4q1tY0KwywPQ4aRO/0RwMtDjBaT4 UxIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=9ROmEdhXXSco/ULkOaE6eoNIg0SUqKfnbfR953ok6yc=; b=qNxGgT8QMQpFarFwaeogp+D8qezxKVP8KQuARhrF8GAFNTZWVjAGLmmdW8o6Vq/BDe yW1BTeFYjBLc7fECfzdWvGKoi996CDL2Ig9jkhrHoVCVELXJ2UDVbUhxVp+MtxH83+MU xbDeRbfDW+1PZ3bfgoFCYk/hTIP1cpKJAfXwyPfXHJzBDhTKDhwqp8GPgFpGwZmicukg 6MAuGze7gH7GBdVh1V0o4kJY6k/Ahn+05uE8FI41nphc58vAVMnb4slMqBcHU8k20QVG i1zOJxiIZkhn5Qenm+HeJld3DbpVVmWQquElmNxXB0c23ZD9RRVrM6NJnz5Wz8gW699l EZzw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id g51si1987970qta.362.2019.04.22.22.54.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:54:59 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9D8D63087925; Tue, 23 Apr 2019 05:54:58 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9ECD3261D1; Tue, 23 Apr 2019 05:54:53 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 4/6] vhost: introduce helpers to get the size of metadata area Date: Tue, 23 Apr 2019 01:54:18 -0400 Message-Id: <20190423055420.26408-5-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Tue, 23 Apr 2019 05:54:58 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To avoid code duplication since it will be used by kernel VA prefetching. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 51 ++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index bff4d586871d..f3f86c3ed659 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -413,6 +413,32 @@ static void vhost_dev_free_iovecs(struct vhost_dev *dev) vhost_vq_free_iovecs(dev->vqs[i]); } +static size_t vhost_get_avail_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + size_t event __maybe_unused = + vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->avail) + + sizeof(*vq->avail->ring) * num + event; +} + +static size_t vhost_get_used_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + size_t event __maybe_unused = + vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->used) + + sizeof(*vq->used->ring) * num + event; +} + +static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, + unsigned int num) +{ + return sizeof(*vq->desc) * num; +} + void vhost_dev_init(struct vhost_dev *dev, struct vhost_virtqueue **vqs, int nvqs, int iov_limit) { @@ -1257,13 +1283,9 @@ static bool vq_access_ok(struct vhost_virtqueue *vq, unsigned int num, struct vring_used __user *used) { - size_t s __maybe_unused = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - - return access_ok(desc, num * sizeof *desc) && - access_ok(avail, - sizeof *avail + num * sizeof *avail->ring + s) && - access_ok(used, - sizeof *used + num * sizeof *used->ring + s); + return access_ok(desc, vhost_get_desc_size(vq, num)) && + access_ok(avail, vhost_get_avail_size(vq, num)) && + access_ok(used, vhost_get_used_size(vq, num)); } static void vhost_vq_meta_update(struct vhost_virtqueue *vq, @@ -1315,22 +1337,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, int vq_meta_prefetch(struct vhost_virtqueue *vq) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; if (!vq->iotlb) return 1; return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, - num * sizeof(*vq->desc), VHOST_ADDR_DESC) && + vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->avail, - sizeof *vq->avail + - num * sizeof(*vq->avail->ring) + s, + vhost_get_avail_size(vq, num), VHOST_ADDR_AVAIL) && iotlb_access_ok(vq, VHOST_ACCESS_WO, (u64)(uintptr_t)vq->used, - sizeof *vq->used + - num * sizeof(*vq->used->ring) + s, - VHOST_ADDR_USED); + vhost_get_used_size(vq, num), VHOST_ADDR_USED); } EXPORT_SYMBOL_GPL(vq_meta_prefetch); @@ -1347,13 +1365,10 @@ EXPORT_SYMBOL_GPL(vhost_log_access_ok); static bool vq_log_access_ok(struct vhost_virtqueue *vq, void __user *log_base) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - return vq_memory_access_ok(log_base, vq->umem, vhost_has_feature(vq, VHOST_F_LOG_ALL)) && (!vq->log_used || log_access_ok(log_base, vq->log_addr, - sizeof *vq->used + - vq->num * sizeof *vq->used->ring + s)); + vhost_get_used_size(vq, vq->num))); } /* Can we start vq? */ From patchwork Tue Apr 23 05:54:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911959 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5EFD9112C for ; Tue, 23 Apr 2019 05:55:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FD0D27D29 for ; Tue, 23 Apr 2019 05:55:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4304928249; Tue, 23 Apr 2019 05:55:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8041227D29 for ; Tue, 23 Apr 2019 05:55:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 875166B0269; Tue, 23 Apr 2019 01:55:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 823DD6B026A; Tue, 23 Apr 2019 01:55:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 715466B026B; Tue, 23 Apr 2019 01:55:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 4CC3D6B0269 for ; Tue, 23 Apr 2019 01:55:08 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id j20so5393366qta.23 for ; Mon, 22 Apr 2019 22:55:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=xGOYeCUAlm5LyudhK8G/pGQDHHGofUwYKXBsaq5c9JI=; b=X+4TtOyK2AT1jveZK4qHylWJeDA/WwoGAG3GtBQqQ6u+LVeBqvp/usBgfNfznyRY2I Imu2eSpZczgb4fwFS77LgaQWsKbTnkbi5T2zS6pgUPsOEk/2KfD1cctqp+ObuqtDWKBc u78qQlJuOV/a5+KU3dgjOeZSjlCHIQz0Zz8YxyoZjWEoXLuVcIdNDOw+Yy+K/WsU58fJ rTo2V70Ij3KJj7bwRAfL4jgUxTlhazLYEtFL3YqmFYnL6VUzFa2p5pFldeaQ+ky3RNIc rMf+dQfMY+fR3BgPqp+iTAM+hIY9tiYwm0C77vT52YZluLawCk2psAsyjow5MgVOXFsR aTEg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAUL1ZyD1dc0auj3gAponWA3ocOqJ4SodeeTMAkxnd1YcqMrJY0T +9BmIn9nociETFmRuqwTliCshGkuaRyVe7IvCTiApJrNIwRXTFhiJqhU7wjFFHFJksbya+6NkYy BAvx53snnw7kvFgnzCrE0Q+uMwWtxSXCmiJvP2c1tLl9SGFkdDv7PyU0aox1FISCslA== X-Received: by 2002:ac8:37b0:: with SMTP id d45mr8209451qtc.368.1555998908040; Mon, 22 Apr 2019 22:55:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqxzYOjkQhWPDMq4pm3Ks34lviE5JV6O488CPkueyhbftf9GIg0orsdAlXSshesGyMtkdFjR X-Received: by 2002:ac8:37b0:: with SMTP id d45mr8209403qtc.368.1555998906756; Mon, 22 Apr 2019 22:55:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998906; cv=none; d=google.com; s=arc-20160816; b=iBTjYz86fUvabXd9y/4bFuq98b18Akh4fya+ppq4MJnhXhuis7ke4rmCdFwUMMAVBR 0oEIwuPNu7bxvEztqygGkVDz1ISAv8fz65MsDywspZ+4JXKE3stmeI14MCcDxxFMD3qI tx7H7RqJif5w7H/52Qs5bC0mw25Eie7dhoByLqHAhvlZ7HQjYj8O84C7OvvP/AwdZcBU 8e61JoD7Cqy5y8uRa4Kdyj1yBUWA4QaHfVBK4Uaki/CemIIAaBSO7WBk6pOjGKz3s1vL vMkM1aCbrX/AHS4hHFmP2REpeAPU7HVpqxGmUSCAedpHvpEYLZnAP9Bj9YUcmrsxMPH2 UsHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=xGOYeCUAlm5LyudhK8G/pGQDHHGofUwYKXBsaq5c9JI=; b=erf2+RRRMiYFztVgZpW6WXO6GYRubBbzlDzrzJS4iIIkJAFXqEJTl6XONXxeaUz8Ya kMjYyid2XCPgOY028lZ+oB4c5HLl3RRuKs2xUa1SbaBcHwvOHjGjKQAbQzvmRTBZlq16 tBkir/OYWgj59tgH5/2JdV5iToSoqvDOislhxQ+oGXDBuBSsmrj890/SKiaoHyTclueb 6RdpbNX6gJ80eil4m/4oPrI+WaDrEs0tjM039umUeqkONq4BA8+DtoewNT/QLDKMHBAd /DqW3qbNo2ArBamF3FkawPxAfw58XH9th/Jz/CoVobgbF/PWV82MWuIgbXChTgBou2s1 2FLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id f66si10244179qtb.0.2019.04.22.22.55.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:55:06 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BFAE030917AD; Tue, 23 Apr 2019 05:55:05 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 25C64261D3; Tue, 23 Apr 2019 05:54:58 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 5/6] vhost: factor out setting vring addr and num Date: Tue, 23 Apr 2019 01:54:19 -0400 Message-Id: <20190423055420.26408-6-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Tue, 23 Apr 2019 05:55:05 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Factoring vring address and num setting which needs special care for accelerating vq metadata accessing. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 177 ++++++++++++++++++++++++------------------ 1 file changed, 103 insertions(+), 74 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index f3f86c3ed659..c2362ed5839e 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -1468,6 +1468,104 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m) return -EFAULT; } +static long vhost_vring_set_num(struct vhost_dev *d, + struct vhost_virtqueue *vq, + void __user *argp) +{ + struct vhost_vring_state s; + + /* Resizing ring with an active backend? + * You don't want to do that. */ + if (vq->private_data) + return -EBUSY; + + if (copy_from_user(&s, argp, sizeof s)) + return -EFAULT; + + if (!s.num || s.num > 0xffff || (s.num & (s.num - 1))) + return -EINVAL; + vq->num = s.num; + + return 0; +} + +static long vhost_vring_set_addr(struct vhost_dev *d, + struct vhost_virtqueue *vq, + void __user *argp) +{ + struct vhost_vring_addr a; + + if (copy_from_user(&a, argp, sizeof a)) + return -EFAULT; + if (a.flags & ~(0x1 << VHOST_VRING_F_LOG)) + return -EOPNOTSUPP; + + /* For 32bit, verify that the top 32bits of the user + data are set to zero. */ + if ((u64)(unsigned long)a.desc_user_addr != a.desc_user_addr || + (u64)(unsigned long)a.used_user_addr != a.used_user_addr || + (u64)(unsigned long)a.avail_user_addr != a.avail_user_addr) + return -EFAULT; + + /* Make sure it's safe to cast pointers to vring types. */ + BUILD_BUG_ON(__alignof__ *vq->avail > VRING_AVAIL_ALIGN_SIZE); + BUILD_BUG_ON(__alignof__ *vq->used > VRING_USED_ALIGN_SIZE); + if ((a.avail_user_addr & (VRING_AVAIL_ALIGN_SIZE - 1)) || + (a.used_user_addr & (VRING_USED_ALIGN_SIZE - 1)) || + (a.log_guest_addr & (VRING_USED_ALIGN_SIZE - 1))) + return -EINVAL; + + /* We only verify access here if backend is configured. + * If it is not, we don't as size might not have been setup. + * We will verify when backend is configured. */ + if (vq->private_data) { + if (!vq_access_ok(vq, vq->num, + (void __user *)(unsigned long)a.desc_user_addr, + (void __user *)(unsigned long)a.avail_user_addr, + (void __user *)(unsigned long)a.used_user_addr)) + return -EINVAL; + + /* Also validate log access for used ring if enabled. */ + if ((a.flags & (0x1 << VHOST_VRING_F_LOG)) && + !log_access_ok(vq->log_base, a.log_guest_addr, + sizeof *vq->used + + vq->num * sizeof *vq->used->ring)) + return -EINVAL; + } + + vq->log_used = !!(a.flags & (0x1 << VHOST_VRING_F_LOG)); + vq->desc = (void __user *)(unsigned long)a.desc_user_addr; + vq->avail = (void __user *)(unsigned long)a.avail_user_addr; + vq->log_addr = a.log_guest_addr; + vq->used = (void __user *)(unsigned long)a.used_user_addr; + + return 0; +} + +static long vhost_vring_set_num_addr(struct vhost_dev *d, + struct vhost_virtqueue *vq, + unsigned int ioctl, + void __user *argp) +{ + long r; + + mutex_lock(&vq->mutex); + + switch (ioctl) { + case VHOST_SET_VRING_NUM: + r = vhost_vring_set_num(d, vq, argp); + break; + case VHOST_SET_VRING_ADDR: + r = vhost_vring_set_addr(d, vq, argp); + break; + default: + BUG(); + } + + mutex_unlock(&vq->mutex); + + return r; +} long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *argp) { struct file *eventfp, *filep = NULL; @@ -1477,7 +1575,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg struct vhost_virtqueue *vq; struct vhost_vring_state s; struct vhost_vring_file f; - struct vhost_vring_addr a; u32 idx; long r; @@ -1490,26 +1587,14 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg idx = array_index_nospec(idx, d->nvqs); vq = d->vqs[idx]; + if (ioctl == VHOST_SET_VRING_NUM || + ioctl == VHOST_SET_VRING_ADDR) { + return vhost_vring_set_num_addr(d, vq, ioctl, argp); + } + mutex_lock(&vq->mutex); switch (ioctl) { - case VHOST_SET_VRING_NUM: - /* Resizing ring with an active backend? - * You don't want to do that. */ - if (vq->private_data) { - r = -EBUSY; - break; - } - if (copy_from_user(&s, argp, sizeof s)) { - r = -EFAULT; - break; - } - if (!s.num || s.num > 0xffff || (s.num & (s.num - 1))) { - r = -EINVAL; - break; - } - vq->num = s.num; - break; case VHOST_SET_VRING_BASE: /* Moving base with an active backend? * You don't want to do that. */ @@ -1535,62 +1620,6 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg if (copy_to_user(argp, &s, sizeof s)) r = -EFAULT; break; - case VHOST_SET_VRING_ADDR: - if (copy_from_user(&a, argp, sizeof a)) { - r = -EFAULT; - break; - } - if (a.flags & ~(0x1 << VHOST_VRING_F_LOG)) { - r = -EOPNOTSUPP; - break; - } - /* For 32bit, verify that the top 32bits of the user - data are set to zero. */ - if ((u64)(unsigned long)a.desc_user_addr != a.desc_user_addr || - (u64)(unsigned long)a.used_user_addr != a.used_user_addr || - (u64)(unsigned long)a.avail_user_addr != a.avail_user_addr) { - r = -EFAULT; - break; - } - - /* Make sure it's safe to cast pointers to vring types. */ - BUILD_BUG_ON(__alignof__ *vq->avail > VRING_AVAIL_ALIGN_SIZE); - BUILD_BUG_ON(__alignof__ *vq->used > VRING_USED_ALIGN_SIZE); - if ((a.avail_user_addr & (VRING_AVAIL_ALIGN_SIZE - 1)) || - (a.used_user_addr & (VRING_USED_ALIGN_SIZE - 1)) || - (a.log_guest_addr & (VRING_USED_ALIGN_SIZE - 1))) { - r = -EINVAL; - break; - } - - /* We only verify access here if backend is configured. - * If it is not, we don't as size might not have been setup. - * We will verify when backend is configured. */ - if (vq->private_data) { - if (!vq_access_ok(vq, vq->num, - (void __user *)(unsigned long)a.desc_user_addr, - (void __user *)(unsigned long)a.avail_user_addr, - (void __user *)(unsigned long)a.used_user_addr)) { - r = -EINVAL; - break; - } - - /* Also validate log access for used ring if enabled. */ - if ((a.flags & (0x1 << VHOST_VRING_F_LOG)) && - !log_access_ok(vq->log_base, a.log_guest_addr, - sizeof *vq->used + - vq->num * sizeof *vq->used->ring)) { - r = -EINVAL; - break; - } - } - - vq->log_used = !!(a.flags & (0x1 << VHOST_VRING_F_LOG)); - vq->desc = (void __user *)(unsigned long)a.desc_user_addr; - vq->avail = (void __user *)(unsigned long)a.avail_user_addr; - vq->log_addr = a.log_guest_addr; - vq->used = (void __user *)(unsigned long)a.used_user_addr; - break; case VHOST_SET_VRING_KICK: if (copy_from_user(&f, argp, sizeof f)) { r = -EFAULT; From patchwork Tue Apr 23 05:54:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10911967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87AD8922 for ; Tue, 23 Apr 2019 05:55:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 772AD28733 for ; Tue, 23 Apr 2019 05:55:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A9E428738; Tue, 23 Apr 2019 05:55:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 233AD28733 for ; Tue, 23 Apr 2019 05:55:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 344C36B026B; Tue, 23 Apr 2019 01:55:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2F2BD6B026C; Tue, 23 Apr 2019 01:55:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E3F26B026D; Tue, 23 Apr 2019 01:55:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id EA2636B026B for ; Tue, 23 Apr 2019 01:55:18 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id 18so13762188qtw.20 for ; Mon, 22 Apr 2019 22:55:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=LuzluN4HroU1Cwy2lijowB0NCVsvY4/LGQ9O1vgjbCI=; b=VGG5GD/6jY5mon7dbpvlg6iDRYXLDYoEJbPnBnTuMklmCtfPLyPdypWcVWNJs6NA/q 7P7YVsDX+hIr/PsJCOHGg0sAPt91P0yZDQFmzt05LSQp6c9PXlH/u49/6pUCry7aYe8F ndNP2MN/SO8YlebCIvG0lnjtyhrpnVyIiWASeEdpwZw9CHXgv2RKWm4YREZNFLQ5/syR PxO7sU9a6woHSwI5DX9u1sNBzl1n14klVbbwPToRJbdiI51lfueUoqH7G8RhMpWB8pho EXfKwTZiPB/uHbKGORLPHAP31LRfZTE8TbDmFEURhFwQO5oE8roQeuvz6Ig/eX/H9y6D kVfQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAX+yCAIsRIHwgelmr7bSkh3vsQHEB2u6+ZvE0uCJV4gggXwqSoA wbUJ4PUdsmlsbX69ct4fJusDMNvJLd5QEAbf+2QJx7ClHek2SN45wqLYr7LfEayrF47/mjgQtmy mwBBXd/s2fa4/CYDl5lKSmrkqJ7C7wKeNhf9iXwwLDzuZmgOC5hb1foZ4l/fJk4GYtw== X-Received: by 2002:aed:22c5:: with SMTP id q5mr19437732qtc.386.1555998918628; Mon, 22 Apr 2019 22:55:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqyqbCorp0FInlslp1ktoqgxHHDDjuKcgai4UUuR+9z1313U0L5S0uWuAd55numBsWQDmeq9 X-Received: by 2002:aed:22c5:: with SMTP id q5mr19437669qtc.386.1555998917041; Mon, 22 Apr 2019 22:55:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555998917; cv=none; d=google.com; s=arc-20160816; b=VuH/bvyiYJANu0kC2RlxKtjmSvOBBKg23nAgpKChrOIbWxc1h6cc2dymT2IiNbbQLg zyo+0oNNY9iln4oxA8TYc3vvSeykMMM4K08jmFXjCqZj4zYnUpb7MKQi9jmZApLqIwsx UowadLKggQq+GIIrlzgty06t9HcRcjRWyXKqGyo/EkH3pi75QNcHPmUOmd8vDIA91HTt Abxk1Qlimw4UaBZMF543flLNa2Fz0G2t8e3ikM01jxS3ePvZZr3iV91LzbZKoLRfMSUd /Beh0ofDpoROD6qJJppu3tRWmWv2nlAg0BeOH3su5e2wcFazywEyb93896LdEOW884A4 9Gtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=LuzluN4HroU1Cwy2lijowB0NCVsvY4/LGQ9O1vgjbCI=; b=rxi31m0FGCnOPNe12ovEOFCwA79xxfGGpcTkOenX3KO6fXvKrFGjtatVcjKNlrgQKn yAA1UTM9/N6VNGpzZbcjqOxxiUASEWVnK7Jy9V1B4vIN9YoftXfoJD5Y+hw6ShJpQwFU aP89Cr3JACMDwL0L0aC+DttfpHGaJwN/16MOIRHU400/ir1aCTvb/4oqjHOUXm3IaC5z Ngea6LuWunu+3epV0rX4+ki/sBSz6P3ZEIjd77kzyjcvSXbLGD8tJtBWIwyaNolgtz/G aHHbzAj0cLHTG3A0vkptA3tOvUcHVduGMOC0/rAuuMx+JTgvIkMnXR8bJhRVfmWZMvhZ QY4Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id m1si498932qvb.184.2019.04.22.22.55.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 22:55:17 -0700 (PDT) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1E2E830BC654; Tue, 23 Apr 2019 05:55:16 +0000 (UTC) Received: from hp-dl380pg8-01.lab.eng.pek2.redhat.com (hp-dl380pg8-01.lab.eng.pek2.redhat.com [10.73.8.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79EB9261D9; Tue, 23 Apr 2019 05:55:06 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, aarcange@redhat.com, James.Bottomley@hansenpartnership.com, hch@infradead.org, davem@davemloft.net, jglisse@redhat.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, christophe.de.dinechin@gmail.com, jrdr.linux@gmail.com Subject: [RFC PATCH V3 6/6] vhost: access vq metadata through kernel virtual address Date: Tue, 23 Apr 2019 01:54:20 -0400 Message-Id: <20190423055420.26408-7-jasowang@redhat.com> In-Reply-To: <20190423055420.26408-1-jasowang@redhat.com> References: <20190423055420.26408-1-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Tue, 23 Apr 2019 05:55:16 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It was noticed that the copy_to/from_user() friends that was used to access virtqueue metdata tends to be very expensive for dataplane implementation like vhost since it involves lots of software checks, speculation barriers, hardware feature toggling (e.g SMAP). The extra cost will be more obvious when transferring small packets since the time spent on metadata accessing become more significant. This patch tries to eliminate those overheads by accessing them through direct mapping of those pages. Invalidation callbacks is implemented for co-operation with general VM management (swap, KSM, THP or NUMA balancing). We will try to get the direct mapping of vq metadata before each round of packet processing if it doesn't exist. If we fail, we will simplely fallback to copy_to/from_user() friends. This invalidation and direct mapping access are synchronized through spinlock and RCU. All matedata accessing through direct map is protected by RCU, and the setup or invalidation are done under spinlock. This method might does not work for high mem page which requires temporary mapping so we just fallback to normal copy_to/from_user() and may not for arch that has virtual tagged cache since extra cache flushing is needed to eliminate the alias. This will result complex logic and bad performance. For those archs, this patch simply go for copy_to/from_user() friends. This is done by ruling out kernel mapping codes through ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE. Note that this is only done when device IOTLB is not enabled. We could use similar method to optimize IOTLB in the future. Tests shows at most about 23% improvement on TX PPS when using virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell: SMAP on | SMAP off Before: 5.2Mpps | 7.1Mpps After: 6.4Mpps | 8.2Mpps Cc: Andrea Arcangeli Cc: James Bottomley Cc: Christoph Hellwig Cc: David Miller Cc: Jerome Glisse Cc: linux-mm@kvack.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-parisc@vger.kernel.org Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 517 +++++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 32 +++ 2 files changed, 546 insertions(+), 3 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index c2362ed5839e..a9fe04f0016a 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -299,6 +299,49 @@ static void vhost_vq_meta_reset(struct vhost_dev *d) __vhost_vq_meta_reset(d->vqs[i]); } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 +static void vhost_map_unprefetch(struct vhost_map *map) +{ + kfree(map->pages); + map->pages = NULL; + map->npages = 0; + map->addr = NULL; +} + +static void vhost_uninit_vq_maps(struct vhost_virtqueue *vq) +{ + struct vhost_map *map[VHOST_NUM_ADDRS]; + int i; + + spin_lock(&vq->mmu_lock); + for (i = 0; i < VHOST_NUM_ADDRS; i++) { + map[i] = rcu_dereference_protected(vq->maps[i], + lockdep_is_held(&vq->mmu_lock)); + if (map[i]) + rcu_assign_pointer(vq->maps[i], NULL); + } + spin_unlock(&vq->mmu_lock); + + synchronize_rcu(); + + for (i = 0; i < VHOST_NUM_ADDRS; i++) + if (map[i]) + vhost_map_unprefetch(map[i]); + +} +#endif + +static void vhost_reset_vq_maps(struct vhost_virtqueue *vq) +{ +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + int i; + + vhost_uninit_vq_maps(vq); + for (i = 0; i < VHOST_NUM_ADDRS; i++) + vq->uaddrs[i].size = 0; +#endif +} + static void vhost_vq_reset(struct vhost_dev *dev, struct vhost_virtqueue *vq) { @@ -327,7 +370,9 @@ static void vhost_vq_reset(struct vhost_dev *dev, vq->busyloop_timeout = 0; vq->umem = NULL; vq->iotlb = NULL; + vq->invalidate_count = 0; __vhost_vq_meta_reset(vq); + vhost_reset_vq_maps(vq); } static int vhost_worker(void *data) @@ -439,6 +484,123 @@ static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, return sizeof(*vq->desc) * num; } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 +static bool vhost_map_range_overlap(struct vhost_uaddr *uaddr, + unsigned long start, + unsigned long end) +{ + if (unlikely(!uaddr->size)) + return false; + + return !(end < uaddr->uaddr || start > uaddr->uaddr - 1 + uaddr->size); +} + +static void vhost_invalidate_vq_start(struct vhost_virtqueue *vq, + int index, + unsigned long start, + unsigned long end) +{ + struct vhost_uaddr *uaddr = &vq->uaddrs[index]; + struct vhost_map *map; + int i; + + if (!vhost_map_range_overlap(uaddr, start, end)) + return; + + spin_lock(&vq->mmu_lock); + ++vq->invalidate_count; + + map = rcu_dereference_protected(vq->maps[index], + lockdep_is_held(&vq->mmu_lock)); + if (map) { + if (uaddr->write) { + for (i = 0; i < map->npages; i++) + set_page_dirty(map->pages[i]); + } + rcu_assign_pointer(vq->maps[index], NULL); + } + spin_unlock(&vq->mmu_lock); + + if (map) { + synchronize_rcu(); + vhost_map_unprefetch(map); + } +} + +static void vhost_invalidate_vq_end(struct vhost_virtqueue *vq, + int index, + unsigned long start, + unsigned long end) +{ + if (!vhost_map_range_overlap(&vq->uaddrs[index], start, end)) + return; + + spin_lock(&vq->mmu_lock); + --vq->invalidate_count; + spin_unlock(&vq->mmu_lock); +} + +static int vhost_invalidate_range_start(struct mmu_notifier *mn, + const struct mmu_notifier_range *range) +{ + struct vhost_dev *dev = container_of(mn, struct vhost_dev, + mmu_notifier); + int i, j; + + if (!range->blockable) + return -EAGAIN; + + for (i = 0; i < dev->nvqs; i++) { + struct vhost_virtqueue *vq = dev->vqs[i]; + + for (j = 0; j < VHOST_NUM_ADDRS; j++) + vhost_invalidate_vq_start(vq, j, + range->start, + range->end); + } + + return 0; +} + +static void vhost_invalidate_range_end(struct mmu_notifier *mn, + const struct mmu_notifier_range *range) +{ + struct vhost_dev *dev = container_of(mn, struct vhost_dev, + mmu_notifier); + int i, j; + + for (i = 0; i < dev->nvqs; i++) { + struct vhost_virtqueue *vq = dev->vqs[i]; + + for (j = 0; j < VHOST_NUM_ADDRS; j++) + vhost_invalidate_vq_end(vq, j, + range->start, + range->end); + } +} + +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { + .invalidate_range_start = vhost_invalidate_range_start, + .invalidate_range_end = vhost_invalidate_range_end, +}; +#endif + +static void vhost_init_maps(struct vhost_dev *dev) +{ +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_virtqueue *vq; + int i, j; + + dev->mmu_notifier.ops = &vhost_mmu_notifier_ops; + + for (i = 0; i < dev->nvqs; ++i) { + vq = dev->vqs[i]; + for (j = 0; j < VHOST_NUM_ADDRS; j++) + RCU_INIT_POINTER(vq->maps[j], NULL); + } +#endif +} + void vhost_dev_init(struct vhost_dev *dev, struct vhost_virtqueue **vqs, int nvqs, int iov_limit) { @@ -459,7 +621,7 @@ void vhost_dev_init(struct vhost_dev *dev, INIT_LIST_HEAD(&dev->read_list); INIT_LIST_HEAD(&dev->pending_list); spin_lock_init(&dev->iotlb_lock); - + vhost_init_maps(dev); for (i = 0; i < dev->nvqs; ++i) { vq = dev->vqs[i]; @@ -468,6 +630,7 @@ void vhost_dev_init(struct vhost_dev *dev, vq->heads = NULL; vq->dev = dev; mutex_init(&vq->mutex); + spin_lock_init(&vq->mmu_lock); vhost_vq_reset(dev, vq); if (vq->handle_kick) vhost_poll_init(&vq->poll, vq->handle_kick, @@ -547,7 +710,18 @@ long vhost_dev_set_owner(struct vhost_dev *dev) if (err) goto err_cgroup; +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + err = mmu_notifier_register(&dev->mmu_notifier, dev->mm); + if (err) + goto err_mmu_notifier; +#endif + return 0; + +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 +err_mmu_notifier: + vhost_dev_free_iovecs(dev); +#endif err_cgroup: kthread_stop(worker); dev->worker = NULL; @@ -638,6 +812,107 @@ static void vhost_clear_msg(struct vhost_dev *dev) spin_unlock(&dev->iotlb_lock); } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 +static void vhost_setup_uaddr(struct vhost_virtqueue *vq, + int index, unsigned long uaddr, + size_t size, bool write) +{ + struct vhost_uaddr *addr = &vq->uaddrs[index]; + + addr->uaddr = uaddr; + addr->size = size; + addr->write = write; +} + +static void vhost_setup_vq_uaddr(struct vhost_virtqueue *vq) +{ + vhost_setup_uaddr(vq, VHOST_ADDR_DESC, + (unsigned long)vq->desc, + vhost_get_desc_size(vq, vq->num), + false); + vhost_setup_uaddr(vq, VHOST_ADDR_AVAIL, + (unsigned long)vq->avail, + vhost_get_avail_size(vq, vq->num), + false); + vhost_setup_uaddr(vq, VHOST_ADDR_USED, + (unsigned long)vq->used, + vhost_get_used_size(vq, vq->num), + true); +} + +static int vhost_map_prefetch(struct vhost_virtqueue *vq, + int index) +{ + struct vhost_map *map; + struct vhost_uaddr *uaddr = &vq->uaddrs[index]; + struct page **pages; + int npages = DIV_ROUND_UP(uaddr->size, PAGE_SIZE); + int npinned; + void *vaddr, *v; + int err; + int i; + + spin_lock(&vq->mmu_lock); + + err = -EFAULT; + if (vq->invalidate_count) + goto err; + + err = -ENOMEM; + map = kmalloc(sizeof(*map), GFP_ATOMIC); + if (!map) + goto err; + + pages = kmalloc_array(npages, sizeof(struct page *), GFP_ATOMIC); + if (!pages) + goto err_pages; + + err = EFAULT; + npinned = __get_user_pages_fast(uaddr->uaddr, npages, + uaddr->write, pages); + if (npinned > 0) + release_pages(pages, npinned); + if (npinned != npages) + goto err_gup; + + for (i = 0; i < npinned; i++) + if (PageHighMem(pages[i])) + goto err_gup; + + vaddr = v = page_address(pages[0]); + + /* For simplicity, fallback to userspace address if VA is not + * contigious. + */ + for (i = 1; i < npinned; i++) { + v += PAGE_SIZE; + if (v != page_address(pages[i])) + goto err_gup; + } + + map->addr = vaddr + (uaddr->uaddr & (PAGE_SIZE - 1)); + map->npages = npages; + map->pages = pages; + + rcu_assign_pointer(vq->maps[index], map); + /* No need for a synchronize_rcu(). This function should be + * called by dev->worker so we are serialized with all + * readers. + */ + spin_unlock(&vq->mmu_lock); + + return 0; + +err_gup: + kfree(pages); +err_pages: + kfree(map); +err: + spin_unlock(&vq->mmu_lock); + return err; +} +#endif + void vhost_dev_cleanup(struct vhost_dev *dev) { int i; @@ -667,8 +942,16 @@ void vhost_dev_cleanup(struct vhost_dev *dev) kthread_stop(dev->worker); dev->worker = NULL; } - if (dev->mm) + if (dev->mm) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + mmu_notifier_unregister(&dev->mmu_notifier, dev->mm); +#endif mmput(dev->mm); + } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + for (i = 0; i < dev->nvqs; i++) + vhost_uninit_vq_maps(dev->vqs[i]); +#endif dev->mm = NULL; } EXPORT_SYMBOL_GPL(vhost_dev_cleanup); @@ -897,6 +1180,26 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + *((__virtio16 *)&used->ring[vq->num]) = + cpu_to_vhost16(vq, vq->avail_idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), vhost_avail_event(vq)); } @@ -905,6 +1208,27 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, struct vring_used_elem *head, int idx, int count) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_used *used; + size_t size; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + size = count * sizeof(*head); + memcpy(used->ring + idx, head, size); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_copy_to_user(vq, vq->used->ring + idx, head, count * sizeof(*head)); } @@ -912,6 +1236,25 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + used->flags = cpu_to_vhost16(vq, vq->used_flags); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), &vq->used->flags); } @@ -919,6 +1262,25 @@ static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + used->idx = cpu_to_vhost16(vq, vq->last_used_idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), &vq->used->idx); } @@ -964,12 +1326,50 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *idx = avail->idx; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *idx, &vq->avail->idx); } static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, __virtio16 *head, int idx) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *head = avail->ring[idx & (vq->num - 1)]; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *head, &vq->avail->ring[idx & (vq->num - 1)]); } @@ -977,24 +1377,98 @@ static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, __virtio16 *flags) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *flags = avail->flags; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *flags, &vq->avail->flags); } static inline int vhost_get_used_event(struct vhost_virtqueue *vq, __virtio16 *event) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_avail *avail; + + if (!vq->iotlb) { + rcu_read_lock(); + map = rcu_dereference(vq->maps[VHOST_ADDR_AVAIL]); + if (likely(map)) { + avail = map->addr; + *event = (__virtio16)avail->ring[vq->num]; + rcu_read_unlock(); + return 0; + } + rcu_read_unlock(); + } +#endif + return vhost_get_avail(vq, *event, vhost_used_event(vq)); } static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_used *used; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_USED]); + if (likely(map)) { + used = map->addr; + *idx = used->idx; + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_get_used(vq, *idx, &vq->used->idx); } static inline int vhost_get_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, int idx) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + struct vhost_map *map; + struct vring_desc *d; + + if (!vq->iotlb) { + rcu_read_lock(); + + map = rcu_dereference(vq->maps[VHOST_ADDR_DESC]); + if (likely(map)) { + d = map->addr; + *desc = *(d + idx); + rcu_read_unlock(); + return 0; + } + + rcu_read_unlock(); + } +#endif + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); } @@ -1335,12 +1809,32 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, return true; } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 +static void vhost_vq_map_prefetch(struct vhost_virtqueue *vq) +{ + struct vhost_map __rcu *map; + int i; + + for (i = 0; i < VHOST_NUM_ADDRS; i++) { + rcu_read_lock(); + map = rcu_dereference(vq->maps[i]); + rcu_read_unlock(); + if (unlikely(!map)) + vhost_map_prefetch(vq, i); + } +} +#endif + int vq_meta_prefetch(struct vhost_virtqueue *vq) { unsigned int num = vq->num; - if (!vq->iotlb) + if (!vq->iotlb) { +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + vhost_vq_map_prefetch(vq); +#endif return 1; + } return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && @@ -1551,6 +2045,16 @@ static long vhost_vring_set_num_addr(struct vhost_dev *d, mutex_lock(&vq->mutex); +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + /* Unregister MMU notifer to allow invalidation callback + * can access vq->uaddrs[] without holding a lock. + */ + if (d->mm) + mmu_notifier_unregister(&d->mmu_notifier, d->mm); + + vhost_uninit_vq_maps(vq); +#endif + switch (ioctl) { case VHOST_SET_VRING_NUM: r = vhost_vring_set_num(d, vq, argp); @@ -1562,6 +2066,13 @@ static long vhost_vring_set_num_addr(struct vhost_dev *d, BUG(); } +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + vhost_setup_vq_uaddr(vq); + + if (d->mm) + mmu_notifier_register(&d->mmu_notifier, d->mm); +#endif + mutex_unlock(&vq->mutex); return r; diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 7a7fc001265f..f33208b55875 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -12,6 +12,9 @@ #include #include #include +#include +#include +#include struct vhost_work; typedef void (*vhost_work_fn_t)(struct vhost_work *work); @@ -80,6 +83,18 @@ enum vhost_uaddr_type { VHOST_NUM_ADDRS = 3, }; +struct vhost_map { + int npages; + void *addr; + struct page **pages; +}; + +struct vhost_uaddr { + unsigned long uaddr; + size_t size; + bool write; +}; + /* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue { struct vhost_dev *dev; @@ -90,7 +105,21 @@ struct vhost_virtqueue { struct vring_desc __user *desc; struct vring_avail __user *avail; struct vring_used __user *used; + +#if ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE == 0 + /* read by memory accessors, modified by meta data + * prefetching, MMU notifier and vring ioctl(). + * Synchonrized through mmu_lock (writers) and RCU (writers + * and readers). + */ + struct vhost_map __rcu *maps[VHOST_NUM_ADDRS]; + /* read by MMU notifier, write by vring ioctl(), synchronized + * through register/unregister MMU notifier. + */ + struct vhost_uaddr uaddrs[VHOST_NUM_ADDRS]; +#endif const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS]; + struct file *kick; struct eventfd_ctx *call_ctx; struct eventfd_ctx *error_ctx; @@ -145,6 +174,8 @@ struct vhost_virtqueue { bool user_be; #endif u32 busyloop_timeout; + spinlock_t mmu_lock; + int invalidate_count; }; struct vhost_msg_node { @@ -158,6 +189,7 @@ struct vhost_msg_node { struct vhost_dev { struct mm_struct *mm; + struct mmu_notifier mmu_notifier; struct mutex mutex; struct vhost_virtqueue **vqs; int nvqs;