From patchwork Wed Mar 6 07:18:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10840479 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF95814DE for ; Wed, 6 Mar 2019 07:18:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 937FD2C9B1 for ; Wed, 6 Mar 2019 07:18:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 87CAB2CAD2; Wed, 6 Mar 2019 07:18:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DCB72C9B1 for ; Wed, 6 Mar 2019 07:18:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E81758E0004; Wed, 6 Mar 2019 02:18:26 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E09318E0001; Wed, 6 Mar 2019 02:18:26 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAB938E0004; Wed, 6 Mar 2019 02:18:26 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 9F4488E0001 for ; Wed, 6 Mar 2019 02:18:26 -0500 (EST) Received: by mail-qk1-f198.google.com with SMTP id v67so9072754qkl.22 for ; Tue, 05 Mar 2019 23:18:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=I7i8KSWluUccJGvvitm/nvSYW6LlPlWoYjbi3a/+f3A=; b=WVFM0bQMWDwM+U7ULMcNeAfOyPsOWyv4sMjy+ZGqPL/BmY874ymipaVvibIBGIEjRW nWmVtZOzLtU74tN9k8JTAS1sTcarOqZRYkwG88L3Zd9+tbNa8SfTu5Su03uq8iVNgo95 2n0+7lZVct9NhMQbC63jS7Sy73Lrdqj/RplOjvruN1DgoU1ZJoK9LsHDctgwJy5kAPDC msxCU1xNrmN8skiIUTRn2aBlwNa9iQtfXOVzW89tg8iK6rWftKa8qJ2iLWlxvlgEkqEX zOD3qfqzl2Ymi6NESAPBh7m0De7647vj3ZpfnK7jjKExu6yhbvic1Jywv951k7JXlINg DspQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAXX/eHhBzKMShWb0EjKmSWe66fDA/Cf0cpVj4MLRum0kWe2w+7Y nL3Vlz0mhJ/j0sLkW//+vPrzx7qGYwzpqWzgqMtvoFw2xzwEjI3rX9kBL01TRWuKuGsiIuMIuvJ BS7oipxfxtYyPpma1xNUQTJXml+iF/wNyBGB6BpwB+fHyzYpEr0J47/kiPBb355tTeg== X-Received: by 2002:aed:2a2f:: with SMTP id c44mr4642570qtd.144.1551856706427; Tue, 05 Mar 2019 23:18:26 -0800 (PST) X-Google-Smtp-Source: APXvYqyFefWBdACfEIePNLStFkIAw1BPNLBfcUw0n+jrvnifsNw1KzfnKZg1HIcQ2+yPSS4prgSA X-Received: by 2002:aed:2a2f:: with SMTP id c44mr4642539qtd.144.1551856705615; Tue, 05 Mar 2019 23:18:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551856705; cv=none; d=google.com; s=arc-20160816; b=Yl/q2ePhYU6KvPzJlctAzfT6pLQ5sViGuBWXPzA5D5zERY4jFeNuW44X4LzB2cFWcf KnYGehfZZomQmhwV1pxRxrJC1PanPi1LxctpcsflL1UFwf4tXRTeXeuKWlNd4FcFe57y VLoDqmdjuEcUSSJq+FV/WZYUyopqVU8WNac2SZDfwxNhu/nSFnf0r4YLVRQOTgrqQO3L x0ImlnaGzXjEuYaGhcPuXlOkfyogC/Eyr30eIm0e4pGVtfIJZCnO9FMBoD8l1zV2nDRY fKcoQqm6QyYkkLjvqNsluT1re5qimcB0DGmmna1nOh+qpFF7Bv5ry4j8/WhMXc6Ng8Nt OBag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=I7i8KSWluUccJGvvitm/nvSYW6LlPlWoYjbi3a/+f3A=; b=v5ZLP6fvZDxNwlHdItRoOidpXwDVG3tDtrIBWZL7SPNRPI75mENrupBdWOjFzZHOcC 5/BKu63KDg8FWr3zcgnm+xIKZFecpK8exKbM9qWk8kjNcU/C/SGZmNCuIUpqFF0yh5VS fPkQZDFjvi+khEvDwpr2s3ffnXQQ08P33mce89VQ4KIVB3a7SuK8AzU+u5PPrKU5qT9J 06fqpCE6iHsuPCCNgsbqZp0Qr2Q9R0IXZp85blp2LzUwYzQZeFHOd1RMgMyTIBbtrIh8 X1HH2vkQ8Punz2ir60h61dQvFSSJAnaTjQlqg/kzH28N4W0me/JjJGO93oDCakMEz3aW CWWw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id t63si499720qkh.271.2019.03.05.23.18.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Mar 2019 23:18:25 -0800 (PST) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E10C1C057F31; Wed, 6 Mar 2019 07:18:24 +0000 (UTC) Received: from hp-dl380pg8-02.lab.eng.pek2.redhat.com (hp-dl380pg8-02.lab.eng.pek2.redhat.com [10.73.8.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id D7524600C5; Wed, 6 Mar 2019 07:18:21 +0000 (UTC) From: Jason Wang To: jasowang@redhat.com, mst@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com Subject: [RFC PATCH V2 1/5] vhost: generalize adding used elem Date: Wed, 6 Mar 2019 02:18:08 -0500 Message-Id: <1551856692-3384-2-git-send-email-jasowang@redhat.com> In-Reply-To: <1551856692-3384-1-git-send-email-jasowang@redhat.com> References: <1551856692-3384-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 06 Mar 2019 07:18:24 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use one generic vhost_copy_to_user() instead of two dedicated accessor. This will simplify the conversion to fine grain accessors. About 2% improvement of PPS were seen during vitio-user txonly test. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index a2e5dc7..400aa78 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -2251,16 +2251,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (count == 1) { - if (vhost_put_user(vq, heads[0].id, &used->id)) { - vq_err(vq, "Failed to write used id"); - return -EFAULT; - } - if (vhost_put_user(vq, heads[0].len, &used->len)) { - vq_err(vq, "Failed to write used len"); - return -EFAULT; - } - } else if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { vq_err(vq, "Failed to write used"); return -EFAULT; } From patchwork Wed Mar 6 07:18:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10840483 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30F241515 for ; Wed, 6 Mar 2019 07:18:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14C572C9B1 for ; Wed, 6 Mar 2019 07:18:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08C312CAD2; Wed, 6 Mar 2019 07:18:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F5BA2CACF for ; Wed, 6 Mar 2019 07:18:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D24F68E0005; Wed, 6 Mar 2019 02:18:30 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CAAC98E0001; Wed, 6 Mar 2019 02:18:30 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B498D8E0005; Wed, 6 Mar 2019 02:18:30 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 8966B8E0001 for ; Wed, 6 Mar 2019 02:18:30 -0500 (EST) Received: by mail-qt1-f197.google.com with SMTP id f24so10676173qte.4 for ; Tue, 05 Mar 2019 23:18:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Amux5aEMfzp4ho9rtWPxMQUKoupwqgeHiImCUv6JiKs=; b=M8LTmoJpVWk5VwZGU3pAVWDTGJXFtFmh7iB/TKQ6RQ6VaV1n7OM7JWx/Uu9t4AMbV3 NHXR5UBi/GlEABPyQo8TbbeRU2G4yM7fNnsFzVsk68+q6J/fgoDvQ7wKMeTr1sK09Coj Tdr2zp8WwFG8VQlXlhhiOEkQakiyDhUdqKDFLVtG5fHuOXLJ0+15HO2LjxBs2/vE6ECl +UsXKFaAfG0kDLLhctbj7sLbj10b9IM59tD6sa0dnGKflfaYStubRGLIy1xnlb4VPc2U tUBRLmAIDqbg0Wu8DKRRpfzZxcpEDPJAMWrKruteCXzBghtvFp0yMRibzH6wIx3F6ewe MvsA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAVfLTqd2sJmdNhWh030p+rcA3WN9VCx0dKNBFFiKpOxl+2ijVji D19Z9L3augNax3AqGkAeTzTSDipj6hRwBpJxH2R0O4jI3vvJySanV59Ex4Ll+MDSiYYVo4ILlOY a7oZb9n/CqPqyEVJ+pFlgXkrqs4CTZMzYLBbxHYRYAjGKcoLe7MMFcCQ5I2cNNFtg1Q== X-Received: by 2002:a0c:ae27:: with SMTP id y36mr5243809qvc.185.1551856710319; Tue, 05 Mar 2019 23:18:30 -0800 (PST) X-Google-Smtp-Source: APXvYqwqnsHe1zOCtEK4D/XlyfmQYWFDt4Ljt6Qs7eNEqMtTd4iWMlUYerwdmKx0ZD7ToHMe09PE X-Received: by 2002:a0c:ae27:: with SMTP id y36mr5243757qvc.185.1551856709048; Tue, 05 Mar 2019 23:18:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551856709; cv=none; d=google.com; s=arc-20160816; b=cVndBTQdh3INrbdyHPogB/adF8OYRAOZsMB2WoC6IPBp/2AY84W6VDeTulqy0DWLpg Lkq1rMeAlH1nXDNFlQ2cpRFTNi9R8hdR0mMFG0ge1nRHMzQVBJOoEzCTtjGdg1+bV5Yy YbADFxQhsKdMhHjUqwrJXnCN5lMXFPS7aGZhNqti7S0eRjtGFg/NV8v6EGHuVgfjT2Rz 09Ib1Nk0zIl+FFG4WV0V9KQY+yNLb9pDA9xgmIhPRQS611ZIjjMselO59UxF+o2wQVRd YFSg2dXrXzSNCmlLxkJ5pCSiAxgme3HxDNJbwnP0SJ9b+trtx/OMGsffbLAsz1GR3zVu /7BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Amux5aEMfzp4ho9rtWPxMQUKoupwqgeHiImCUv6JiKs=; b=Oyb0pk+Ou7MtETJClGKFbsQV4NFWsfyw5iRAqtsNjqHjHmPjmSNLpDFt0425k+/RYp D8Sn3Qmth0pwWLYAxT3z1bNTwMTMmyRgsyYEDDJzB2Bu83jWJsm9LgXkFtkQkZosHzMB Gwr+iCqe0wFOiJY6DEUvq0lP3tD+Tj93SHi1nl3J71OCNnQXVoLpVrZkdPfNFsde3f69 +mtX+x9s3UpLBIoH8XFBB5+qDz97Cz2lJyMfiI7SSmEDgeuatb5aH8E5JBQe0rpPdi+d QVexv6WkI5uABqhKch8EhxiSnHxD5p1QrwAzEMz0GjS9wgIBU7C+UlrN0gPYXu6ol+Yp 4Jmw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id 17si452817qvo.189.2019.03.05.23.18.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Mar 2019 23:18:29 -0800 (PST) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3615D13AAE; Wed, 6 Mar 2019 07:18:28 +0000 (UTC) Received: from hp-dl380pg8-02.lab.eng.pek2.redhat.com (hp-dl380pg8-02.lab.eng.pek2.redhat.com [10.73.8.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AACC600C5; Wed, 6 Mar 2019 07:18:25 +0000 (UTC) From: Jason Wang To: jasowang@redhat.com, mst@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com Subject: [RFC PATCH V2 2/5] vhost: fine grain userspace memory accessors Date: Wed, 6 Mar 2019 02:18:09 -0500 Message-Id: <1551856692-3384-3-git-send-email-jasowang@redhat.com> In-Reply-To: <1551856692-3384-1-git-send-email-jasowang@redhat.com> References: <1551856692-3384-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 06 Mar 2019 07:18:28 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is used to hide the metadata address from virtqueue helpers. This will allow to implement a vmap based fast accessing to metadata. Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 94 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 77 insertions(+), 17 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 400aa78..29709e7 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -869,6 +869,34 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, ret; \ }) +static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), + vhost_avail_event(vq)); +} + +static inline int vhost_put_used(struct vhost_virtqueue *vq, + struct vring_used_elem *head, int idx, + int count) +{ + return vhost_copy_to_user(vq, vq->used->ring + idx, head, + count * sizeof(*head)); +} + +static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), + &vq->used->flags); +} + +static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) + +{ + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), + &vq->used->idx); +} + #define vhost_get_user(vq, x, ptr, type) \ ({ \ int ret; \ @@ -907,6 +935,43 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) mutex_unlock(&d->vqs[i]->mutex); } +static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_avail(vq, *idx, &vq->avail->idx); +} + +static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, + __virtio16 *head, int idx) +{ + return vhost_get_avail(vq, *head, + &vq->avail->ring[idx & (vq->num - 1)]); +} + +static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, + __virtio16 *flags) +{ + return vhost_get_avail(vq, *flags, &vq->avail->flags); +} + +static inline int vhost_get_used_event(struct vhost_virtqueue *vq, + __virtio16 *event) +{ + return vhost_get_avail(vq, *event, vhost_used_event(vq)); +} + +static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, + __virtio16 *idx) +{ + return vhost_get_used(vq, *idx, &vq->used->idx); +} + +static inline int vhost_get_desc(struct vhost_virtqueue *vq, + struct vring_desc *desc, int idx) +{ + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); +} + static int vhost_new_umem_range(struct vhost_umem *umem, u64 start, u64 size, u64 end, u64 userspace_addr, int perm) @@ -1840,8 +1905,7 @@ int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, static int vhost_update_used_flags(struct vhost_virtqueue *vq) { void __user *used; - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), - &vq->used->flags) < 0) + if (vhost_put_used_flags(vq)) return -EFAULT; if (unlikely(vq->log_used)) { /* Make sure the flag is seen before log. */ @@ -1858,8 +1922,7 @@ static int vhost_update_used_flags(struct vhost_virtqueue *vq) static int vhost_update_avail_event(struct vhost_virtqueue *vq, u16 avail_event) { - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), - vhost_avail_event(vq))) + if (vhost_put_avail_event(vq)) return -EFAULT; if (unlikely(vq->log_used)) { void __user *used; @@ -1895,7 +1958,7 @@ int vhost_vq_init_access(struct vhost_virtqueue *vq) r = -EFAULT; goto err; } - r = vhost_get_used(vq, last_used_idx, &vq->used->idx); + r = vhost_get_used_idx(vq, &last_used_idx); if (r) { vq_err(vq, "Can't access used idx at %p\n", &vq->used->idx); @@ -2094,7 +2157,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, last_avail_idx = vq->last_avail_idx; if (vq->avail_idx == vq->last_avail_idx) { - if (unlikely(vhost_get_avail(vq, avail_idx, &vq->avail->idx))) { + if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { vq_err(vq, "Failed to access avail idx at %p\n", &vq->avail->idx); return -EFAULT; @@ -2121,8 +2184,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, /* Grab the next descriptor number they're advertising, and increment * the index we've seen. */ - if (unlikely(vhost_get_avail(vq, ring_head, - &vq->avail->ring[last_avail_idx & (vq->num - 1)]))) { + if (unlikely(vhost_get_avail_head(vq, &ring_head, last_avail_idx))) { vq_err(vq, "Failed to read head: idx %d address %p\n", last_avail_idx, &vq->avail->ring[last_avail_idx % vq->num]); @@ -2157,8 +2219,7 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, i, vq->num, head); return -EINVAL; } - ret = vhost_copy_from_user(vq, &desc, vq->desc + i, - sizeof desc); + ret = vhost_get_desc(vq, &desc, i); if (unlikely(ret)) { vq_err(vq, "Failed to get descriptor: idx %d addr %p\n", i, vq->desc + i); @@ -2251,7 +2312,7 @@ static int __vhost_add_used_n(struct vhost_virtqueue *vq, start = vq->last_used_idx & (vq->num - 1); used = vq->used->ring + start; - if (vhost_copy_to_user(vq, used, heads, count * sizeof *used)) { + if (vhost_put_used(vq, heads, start, count)) { vq_err(vq, "Failed to write used"); return -EFAULT; } @@ -2293,8 +2354,7 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, /* Make sure buffer is written before we update index. */ smp_wmb(); - if (vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), - &vq->used->idx)) { + if (vhost_put_used_idx(vq)) { vq_err(vq, "Failed to increment used idx"); return -EFAULT; } @@ -2327,7 +2387,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { __virtio16 flags; - if (vhost_get_avail(vq, flags, &vq->avail->flags)) { + if (vhost_get_avail_flags(vq, &flags)) { vq_err(vq, "Failed to get flags"); return true; } @@ -2341,7 +2401,7 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (unlikely(!v)) return true; - if (vhost_get_avail(vq, event, vhost_used_event(vq))) { + if (vhost_get_used_event(vq, &event)) { vq_err(vq, "Failed to get used event idx"); return true; } @@ -2386,7 +2446,7 @@ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq) if (vq->avail_idx != vq->last_avail_idx) return false; - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (unlikely(r)) return false; vq->avail_idx = vhost16_to_cpu(vq, avail_idx); @@ -2422,7 +2482,7 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) /* They could have slipped one in as we were doing that: make * sure it's written, then check again. */ smp_mb(); - r = vhost_get_avail(vq, avail_idx, &vq->avail->idx); + r = vhost_get_avail_idx(vq, &avail_idx); if (r) { vq_err(vq, "Failed to check avail idx at %p: %d\n", &vq->avail->idx, r); From patchwork Wed Mar 6 07:18:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10840485 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCC2214DE for ; Wed, 6 Mar 2019 07:18:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A22032C9B1 for ; Wed, 6 Mar 2019 07:18:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 962E02CAD2; Wed, 6 Mar 2019 07:18:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16F402C9B1 for ; Wed, 6 Mar 2019 07:18:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AF4F8E0006; Wed, 6 Mar 2019 02:18:37 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 138EC8E0001; Wed, 6 Mar 2019 02:18:37 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0291F8E0006; Wed, 6 Mar 2019 02:18:36 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id C9EBF8E0001 for ; Wed, 6 Mar 2019 02:18:36 -0500 (EST) Received: by mail-qt1-f200.google.com with SMTP id k37so10672035qtb.20 for ; Tue, 05 Mar 2019 23:18:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OYPBbA6eAG3catNtSF1xNYKBcVS8MMZS2S/FHZ1OqKw=; b=PifkMOUSL5Xn/eWgzLtRyDkEtrIJeH8NJ0LaEGJ45FTzzN/L3s9BEozhcN5/zxhVtW Fz/LmBEQ1teXVpD0hZXlRHGDl9OAY/l2VdxqS3U8aTAn5noszzzyaCgLJKI7NmJ4QIxu i9rWsLJQopee/h6CXWNRFe0eeJPOVf8cYR1OfDmUJIVd864knCvyMcD3XkCkxS3D19aJ tmHFo6FlVWT7TydqGMcBD/rB40nyHa3/cUDYOkl5x+WeT6T2Fj7638H6xGLzFqvJwqUZ GCKr0ogS6KAzGSBfkeesNBI9j9PmSUqC2x3fhcF2fpM4B7o3aTiNr+XI8rYQEvEUA3e9 iMJQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAUYASwuN6+eDyvuKjXAOAlpdEw734FY+sD6PJ07VVj+QdnuUbC+ 7JxgGG5d3QXSXD9lfyYJJ921DfeVUlYf1Dk1Kk1c6T34xCSCDFkHdSJeMuFFUF4HKwF6g8/GY8Z X/tx1kszVHgyiZZ/aReojBOVvMG1pR7FmbBcrpcRs5+4k+G7tTEURoXpYlL81Y1UGnQ== X-Received: by 2002:ac8:3928:: with SMTP id s37mr4707909qtb.246.1551856716587; Tue, 05 Mar 2019 23:18:36 -0800 (PST) X-Google-Smtp-Source: APXvYqzFdzGDvw+RY/leoSXXAux8i6x5uKM1lieu0dXXU8AHTXLFt/CrHSPm8lB/J5jUSd9jDy0x X-Received: by 2002:ac8:3928:: with SMTP id s37mr4707849qtb.246.1551856715026; Tue, 05 Mar 2019 23:18:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551856715; cv=none; d=google.com; s=arc-20160816; b=TZq5js6AewKhVxVgYjxmUyxDk960jdRkjfXKCN2qDXmv7ZJIkckYXvGVI4AkN64VE1 Sf94yvNXXuYt4tLSz+QpGNTvaL3UNP4IZNad/3m/1UIsj5pXdfR07pq5iewjWGfl7tDT N2H/P2tJi/tSAbmtYNZWeGkD4CdChdXUu5uE5+uS7zdeB50m2CovePDvgAQJ4DIke3n2 3ilET0P28RulH4minQeJOZF/F82xn+7RSs6a/E2z1bEfqJWpdT3xAF//V/eNqjhLZHy5 pi5/6+W8gypwYkesAeJLkB9VxJ9cLpcCT0Xx+YkOsXKKhBEVEHgTDIpjQHN9e9kzxGkG kTUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=OYPBbA6eAG3catNtSF1xNYKBcVS8MMZS2S/FHZ1OqKw=; b=Pq1d+Pb76p+goGFJqVWPBW0nYq0gieDZxpnynB5mMlDPMYqZYMwWfwH1GWWXLLUPCf J0zqAbP6Ze6o8r1i4d1veI8IF8tcjko/r0Pvt3yybsn8j/Ht2H2qfCmpSpSjOh/0977E 8WCKpWgtGMWnhitLvDqdMX0mOTKmehupBHYFsrDOqU/gAvmaSLAMLYB+B85jLU1bYM5c n9RAKlpBaGq3XZ1G+fEYQIG5VYs7HdASeHlEnKfW7gAWYTXDOxgC1LFmIySzuhv8+Qyp XZM36sSW+RHH0DK+6iFqfsdr0KrnGvmLXpY1FhLavC5oO8HqWL/1ywdXckKFi4xYeO4i jVWQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id q49si488511qtj.90.2019.03.05.23.18.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Mar 2019 23:18:35 -0800 (PST) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3FE76300B915; Wed, 6 Mar 2019 07:18:34 +0000 (UTC) Received: from hp-dl380pg8-02.lab.eng.pek2.redhat.com (hp-dl380pg8-02.lab.eng.pek2.redhat.com [10.73.8.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id AFFAE600C5; Wed, 6 Mar 2019 07:18:28 +0000 (UTC) From: Jason Wang To: jasowang@redhat.com, mst@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com Subject: [RFC PATCH V2 3/5] vhost: rename vq_iotlb_prefetch() to vq_meta_prefetch() Date: Wed, 6 Mar 2019 02:18:10 -0500 Message-Id: <1551856692-3384-4-git-send-email-jasowang@redhat.com> In-Reply-To: <1551856692-3384-1-git-send-email-jasowang@redhat.com> References: <1551856692-3384-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Wed, 06 Mar 2019 07:18:34 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Rename the function to be more accurate since it actually tries to prefetch vq metadata address in IOTLB. And this will be used by following patch to prefetch metadata virtual addresses. Signed-off-by: Jason Wang --- drivers/vhost/net.c | 4 ++-- drivers/vhost/vhost.c | 4 ++-- drivers/vhost/vhost.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index df51a35..bf55f99 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -971,7 +971,7 @@ static void handle_tx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); @@ -1140,7 +1140,7 @@ static void handle_rx(struct vhost_net *net) if (!sock) goto out; - if (!vq_iotlb_prefetch(vq)) + if (!vq_meta_prefetch(vq)) goto out; vhost_disable_notify(&net->dev, vq); diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 29709e7..2025543 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -1309,7 +1309,7 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, return true; } -int vq_iotlb_prefetch(struct vhost_virtqueue *vq) +int vq_meta_prefetch(struct vhost_virtqueue *vq) { size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; @@ -1328,7 +1328,7 @@ int vq_iotlb_prefetch(struct vhost_virtqueue *vq) num * sizeof(*vq->used->ring) + s, VHOST_ADDR_USED); } -EXPORT_SYMBOL_GPL(vq_iotlb_prefetch); +EXPORT_SYMBOL_GPL(vq_meta_prefetch); /* Can we log writes? */ /* Caller should have device mutex but not vq mutex */ diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 9490e7d..7a7fc00 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -209,7 +209,7 @@ void vhost_add_used_and_signal_n(struct vhost_dev *, struct vhost_virtqueue *, int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, unsigned int log_num, u64 len, struct iovec *iov, int count); -int vq_iotlb_prefetch(struct vhost_virtqueue *vq); +int vq_meta_prefetch(struct vhost_virtqueue *vq); struct vhost_msg_node *vhost_new_msg(struct vhost_virtqueue *vq, int type); void vhost_enqueue_msg(struct vhost_dev *dev, From patchwork Wed Mar 6 07:18:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10840489 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 03D3B1515 for ; Wed, 6 Mar 2019 07:18:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDECC2C9B1 for ; Wed, 6 Mar 2019 07:18:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D25A62CAD2; Wed, 6 Mar 2019 07:18:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E5842C9B1 for ; Wed, 6 Mar 2019 07:18:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B73D28E0007; Wed, 6 Mar 2019 02:18:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AFB858E0001; Wed, 6 Mar 2019 02:18:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C47D8E0007; Wed, 6 Mar 2019 02:18:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 7176F8E0001 for ; Wed, 6 Mar 2019 02:18:45 -0500 (EST) Received: by mail-qk1-f200.google.com with SMTP id i66so9069177qke.21 for ; Tue, 05 Mar 2019 23:18:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=hgLHbNBUqAGtlADp7mooWyIXpRAaHdyS6OF7kehE+CQ=; b=FOl/ENVGAXNUPEEGO0mk4nfe7v7dPmd1knvSmg63X9KweqtEgI0TMeg4oxa4qPgArc fzoQTpCPo97TjE0fzw9Phj0tjpOf7SdGnFrsJFTQz4zeqm2B8OnF7XAH4zBKTmWzkKuX bn51uoC0LJMm15aF1TeS2j9NXE2Gd19rvhxtz292S0tzF1diTY0W8yasz3GbfYl2RT+i 2z+6PJH1Nsf2ZbSK8AsK8N1psoXnTErjj0k1J5NyozHgGbycL8518OXbneJa4xb/ImHQ o+SA3VYrplCDee6y4h1Q3IUIJiI9Uha3ShfKjPL6JEmJYqbauByhLoIyoYrhSBqOf5Sy 4G4g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAX0A5tsZKFmb/6rcZ7xpAl7+U84uiQB31dg6SoE6vhsARiXe8kZ MNtdI7XE4dFmoU3J5DPBD4/pNNzojCTKVk2HZYk8k5kBJB1Z/LpxdLDToR7xxboXq8IAEr1h2OV S9OOFcjmaYexj6nn8SAHN/yd3SdSM8MrtZIez7MF7Dz4xmEZwcdl1qZxwqe+e3Zj2MA== X-Received: by 2002:ac8:23f7:: with SMTP id r52mr4452882qtr.378.1551856725188; Tue, 05 Mar 2019 23:18:45 -0800 (PST) X-Google-Smtp-Source: APXvYqxTX+ylVUKqkeHEu3Qjok9LM5gm8Xr79/T0ZvS85HSrFZAVnlkj3OmAyzcG/pk9m5jUqN5f X-Received: by 2002:ac8:23f7:: with SMTP id r52mr4452861qtr.378.1551856724414; Tue, 05 Mar 2019 23:18:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551856724; cv=none; d=google.com; s=arc-20160816; b=hDgtZJpXBPHdiAOyr+QF6szA97qLVJwlOBy9L+Y0wo5BukWqNS5NR+fIYLRu4vmw+J +zXfb1po4yBui6WS5WAno16xNmQLys7hPpgKpeAWeh+YNt1fyFB+O5MLKk/M5Ex9O2EG LVdQTYy+Mab8YNvHoPyKan9l+PI0fNFSzVeaqR3bexwpPSFNyn0w2dOvS/7Q8+tM9ArX f+xQTLowFn2ozgqgN4M/AYEt9uaDQfzl1m9Bws0x7meF162j4dvntvJ45NN68iyQJ4X6 R98Dy0DtLy5QQtcZBaP5lFHVE9B5tU5WlvXVH4DcdzSejEyWTAbFgCR0AoQtSPRFjZQ4 4//A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=hgLHbNBUqAGtlADp7mooWyIXpRAaHdyS6OF7kehE+CQ=; b=ZxVIxtVatIh2nQhhGbPLec1YBwvKBB55bfkpZQRNx+SvxF4qIJQPN3WV+TcuHi4nos P+cWGnkO9ulnBqeDqI4uzxpaRCfNIgBSvGo+O5diVa5BfguRDIK/eeejzAid25RooWEF EM2b+RXJX0s6pUco9gWsT3GZoGjxvp1IHPnGUH4v0IGH644Cql0Sf3Tt/mtRR/Uv+Tl4 GPXNNeLgOxVxgdh9p349iLfgFLcUBIVRJQUKouDLk6Y7I+O/ZhNdBauHafvetkhXscX9 +Qm8gptnrjbdhnq4BulFSykv3BfmjtMQNOx0gvrr3TP8SAJVSka6KxwSNrvzHJjalStW SmYQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id d2si261619qvn.110.2019.03.05.23.18.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Mar 2019 23:18:44 -0800 (PST) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9E63C30224BE; Wed, 6 Mar 2019 07:18:43 +0000 (UTC) Received: from hp-dl380pg8-02.lab.eng.pek2.redhat.com (hp-dl380pg8-02.lab.eng.pek2.redhat.com [10.73.8.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6C434600C5; Wed, 6 Mar 2019 07:18:34 +0000 (UTC) From: Jason Wang To: jasowang@redhat.com, mst@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com Subject: [RFC PATCH V2 4/5] vhost: introduce helpers to get the size of metadata area Date: Wed, 6 Mar 2019 02:18:11 -0500 Message-Id: <1551856692-3384-5-git-send-email-jasowang@redhat.com> In-Reply-To: <1551856692-3384-1-git-send-email-jasowang@redhat.com> References: <1551856692-3384-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Wed, 06 Mar 2019 07:18:43 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Jason Wang --- drivers/vhost/vhost.c | 46 ++++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 2025543..1015464 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -413,6 +413,27 @@ static void vhost_dev_free_iovecs(struct vhost_dev *dev) vhost_vq_free_iovecs(dev->vqs[i]); } +static size_t vhost_get_avail_size(struct vhost_virtqueue *vq, int num) +{ + size_t event = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->avail) + + sizeof(*vq->avail->ring) * num + event; +} + +static size_t vhost_get_used_size(struct vhost_virtqueue *vq, int num) +{ + size_t event = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; + + return sizeof(*vq->used) + + sizeof(*vq->used->ring) * num + event; +} + +static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, int num) +{ + return sizeof(*vq->desc) * num; +} + void vhost_dev_init(struct vhost_dev *dev, struct vhost_virtqueue **vqs, int nvqs, int iov_limit) { @@ -1253,13 +1274,9 @@ static bool vq_access_ok(struct vhost_virtqueue *vq, unsigned int num, struct vring_used __user *used) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - - return access_ok(desc, num * sizeof *desc) && - access_ok(avail, - sizeof *avail + num * sizeof *avail->ring + s) && - access_ok(used, - sizeof *used + num * sizeof *used->ring + s); + return access_ok(desc, vhost_get_desc_size(vq, num)) && + access_ok(avail, vhost_get_avail_size(vq, num)) && + access_ok(used, vhost_get_used_size(vq, num)); } static void vhost_vq_meta_update(struct vhost_virtqueue *vq, @@ -1311,22 +1328,18 @@ static bool iotlb_access_ok(struct vhost_virtqueue *vq, int vq_meta_prefetch(struct vhost_virtqueue *vq) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; unsigned int num = vq->num; if (!vq->iotlb) return 1; return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, - num * sizeof(*vq->desc), VHOST_ADDR_DESC) && + vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->avail, - sizeof *vq->avail + - num * sizeof(*vq->avail->ring) + s, + vhost_get_avail_size(vq, num), VHOST_ADDR_AVAIL) && iotlb_access_ok(vq, VHOST_ACCESS_WO, (u64)(uintptr_t)vq->used, - sizeof *vq->used + - num * sizeof(*vq->used->ring) + s, - VHOST_ADDR_USED); + vhost_get_used_size(vq, num), VHOST_ADDR_USED); } EXPORT_SYMBOL_GPL(vq_meta_prefetch); @@ -1343,13 +1356,10 @@ bool vhost_log_access_ok(struct vhost_dev *dev) static bool vq_log_access_ok(struct vhost_virtqueue *vq, void __user *log_base) { - size_t s = vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX) ? 2 : 0; - return vq_memory_access_ok(log_base, vq->umem, vhost_has_feature(vq, VHOST_F_LOG_ALL)) && (!vq->log_used || log_access_ok(log_base, vq->log_addr, - sizeof *vq->used + - vq->num * sizeof *vq->used->ring + s)); + vhost_get_used_size(vq, vq->num))); } /* Can we start vq? */ From patchwork Wed Mar 6 07:18:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 10840491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C49B214DE for ; Wed, 6 Mar 2019 07:18:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7BC02C9B1 for ; Wed, 6 Mar 2019 07:18:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9A36F2CAD2; Wed, 6 Mar 2019 07:18:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E3DE2C9B1 for ; Wed, 6 Mar 2019 07:18:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3844A8E0008; Wed, 6 Mar 2019 02:18:52 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 30CF18E0001; Wed, 6 Mar 2019 02:18:52 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AC678E0008; Wed, 6 Mar 2019 02:18:52 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id E1F788E0001 for ; Wed, 6 Mar 2019 02:18:51 -0500 (EST) Received: by mail-qt1-f197.google.com with SMTP id m34so10664207qtb.14 for ; Tue, 05 Mar 2019 23:18:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=NmGqV+WxlicRT3LjKje1/obK8f2shvdEIYtualvbYZo=; b=ls4u5BkTvRXPvAgoSCD6gJfqDQ3C9tppbcUOxrOkgwwZLzi8O137hgKPr/nRBJGTK9 KF/OW2sTb3wWFP4Q+rTzZvh+yJAjDiNbtc/fLk80tWB6WEp90r5mDi4NW+5e1ow84qMx scf509QP+hCjc8Y08PKk5mki2Tx5O6SbsmeaUcEqAcyvXjoomgPY6Ag71mBCDPGkbbyo V1A9NYbqVqAT3S3h9k6A8l7lyPEmfbg4Hrf5ONbLnjY6fLOIDPV2rGvl7XK0OYNsDsbz KWZfRLA6ds0wISgUOhdjd6tGUJ5GEkdDEwNsq0hCbnmZXsWY+eKCeo5NbUDQlhqI1Tym Wj+w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APjAAAWwjJVZuYzJmkU7PET+AAgyZCVCgt1WuQ4pIX/SFVUkqJ7pkM0N vXORE6ofktWjQwzxj9S7WG5OKpdDvJSTiE8pQujSReeiPnnmnWxipbO3QTJ5qgUubkEx5r5scRW VG9mWjQNKp/GpIK0o6po5w6mRYvIYYJCPNXJMy387v+xw6S2NSVyZko0CFBUy4dtvNQ== X-Received: by 2002:a37:d612:: with SMTP id t18mr4601707qki.215.1551856731619; Tue, 05 Mar 2019 23:18:51 -0800 (PST) X-Google-Smtp-Source: APXvYqxY3ve+qMxH3G9r99xRauFt6lI4m8zNDmrMvyhp9C+w+QCwX9pIfkkgAnoc6GZnPLzOPukT X-Received: by 2002:a37:d612:: with SMTP id t18mr4601632qki.215.1551856729724; Tue, 05 Mar 2019 23:18:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551856729; cv=none; d=google.com; s=arc-20160816; b=c/socrVot8+i7z7KSuIGeCy3E95MeP9OHRTMfSbZM+EL4Wd5rhqLV/o26t4+HaErSv CitpOqZZ5TghN6Ge7CD3MVlpMaJ/tcT84zfiMHsF20FXvzudvLhTDpE7QxHOoAEE9EQQ Jhxc6B0axsrMOnYEjFob/jV5nrRojigaM6LqR7dw/MZwd4wY/UiqD9Zg0SSEsT7pAT73 X6QNVZd3Mh4PKMo8Pam3LG/Wsen0IC969GUsMAhBl/JffrwCLQCd9P3xNjGb98maSdW0 ywW8A4dWJIBmqwvaqT+9Xah0Sjs8thGhGYFrvrg3EJ6DD51WAqVMiAecGbsNlfMEkLaX jCFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=NmGqV+WxlicRT3LjKje1/obK8f2shvdEIYtualvbYZo=; b=1Ali+O+nxPrtdwti9VLMP+lCZm0nH8V5I4ebU3F6ZEkvb23QyWqZMkYgTNetGJcqu0 joxN1U0/AaRQlYyR/Z46jLlTnYSSK4KmP3/pipy0aouNNXv6GhfNw1BlXJPHFBclW8Wp SYYfkXVGTBXxcDIt4Fkj31g+LVZGBQ7itSoAM63Wf9IXvhYSsxBt7OLq1qalqYBWbf0I 1Luo4eXSXMFWgsWupJmQlnU8vE22w0Xk2rmr6y8uDZIp3LbUljAStKQreiRl7PYKg5Dt IbmLjcpUxGOOEYfbRO0FCju4h9F7PknAAEH9pE6vDIeRrwSXzx53VoAfEGsRmlbTMhL7 JSHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id n9si427721qvk.93.2019.03.05.23.18.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Mar 2019 23:18:49 -0800 (PST) Received-SPF: pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jasowang@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=jasowang@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D82F0309175F; Wed, 6 Mar 2019 07:18:48 +0000 (UTC) Received: from hp-dl380pg8-02.lab.eng.pek2.redhat.com (hp-dl380pg8-02.lab.eng.pek2.redhat.com [10.73.8.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id 280D7600C5; Wed, 6 Mar 2019 07:18:43 +0000 (UTC) From: Jason Wang To: jasowang@redhat.com, mst@redhat.com, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, linux-mm@kvack.org, aarcange@redhat.com Subject: [RFC PATCH V2 5/5] vhost: access vq metadata through kernel virtual address Date: Wed, 6 Mar 2019 02:18:12 -0500 Message-Id: <1551856692-3384-6-git-send-email-jasowang@redhat.com> In-Reply-To: <1551856692-3384-1-git-send-email-jasowang@redhat.com> References: <1551856692-3384-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Wed, 06 Mar 2019 07:18:48 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It was noticed that the copy_user() friends that was used to access virtqueue metdata tends to be very expensive for dataplane implementation like vhost since it involves lots of software checks, speculation barrier, hardware feature toggling (e.g SMAP). The extra cost will be more obvious when transferring small packets since the time spent on metadata accessing become more significant. This patch tries to eliminate those overheads by accessing them through kernel virtual address by vmap(). To make the pages can be migrated, instead of pinning them through GUP, we use MMU notifiers to invalidate vmaps and re-establish vmaps during each round of metadata prefetching if necessary. It looks to me .invalidate_range() is sufficient for catching this since we don't need extra TLB flush. For devices that doesn't use metadata prefetching, the memory accessors fallback to normal copy_user() implementation gracefully. The invalidation was synchronized with datapath through vq mutex, and in order to avoid hold vq mutex during range checking, MMU notifier was teared down when trying to modify vq metadata. Dirty page checking is done by calling set_page_dirty_locked() explicitly for the page that used ring stay after each round of processing. Note that this was only done when device IOTLB is not enabled. We could use similar method to optimize it in the future. Tests shows at most about 22% improvement on TX PPS when using virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell: SMAP on | SMAP off Before: 5.0Mpps | 6.6Mpps After: 6.1Mpps | 7.4Mpps Cc: Signed-off-by: Jason Wang --- drivers/vhost/net.c | 2 + drivers/vhost/vhost.c | 281 +++++++++++++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 16 +++ 3 files changed, 297 insertions(+), 2 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index bf55f99..c276371 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -982,6 +982,7 @@ static void handle_tx(struct vhost_net *net) else handle_tx_copy(net, sock); + vq_meta_prefetch_done(vq); out: mutex_unlock(&vq->mutex); } @@ -1250,6 +1251,7 @@ static void handle_rx(struct vhost_net *net) vhost_net_enable_vq(net, vq); out: vhost_net_signal_used(nvq); + vq_meta_prefetch_done(vq); mutex_unlock(&vq->mutex); } diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 1015464..36ccf7c 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -434,6 +434,74 @@ static size_t vhost_get_desc_size(struct vhost_virtqueue *vq, int num) return sizeof(*vq->desc) * num; } +static void vhost_uninit_vmap(struct vhost_vmap *map) +{ + if (map->addr) { + vunmap(map->unmap_addr); + kfree(map->pages); + map->pages = NULL; + map->npages = 0; + } + + map->addr = NULL; + map->unmap_addr = NULL; +} + +static void vhost_invalidate_vmap(struct vhost_virtqueue *vq, + struct vhost_vmap *map, + unsigned long ustart, + size_t size, + unsigned long start, + unsigned long end) +{ + if (end < ustart || start > ustart - 1 + size) + return; + + dump_stack(); + mutex_lock(&vq->mutex); + vhost_uninit_vmap(map); + mutex_unlock(&vq->mutex); +} + + +static void vhost_invalidate(struct vhost_dev *dev, + unsigned long start, unsigned long end) +{ + int i; + + for (i = 0; i < dev->nvqs; i++) { + struct vhost_virtqueue *vq = dev->vqs[i]; + + vhost_invalidate_vmap(vq, &vq->avail_ring, + (unsigned long)vq->avail, + vhost_get_avail_size(vq, vq->num), + start, end); + vhost_invalidate_vmap(vq, &vq->desc_ring, + (unsigned long)vq->desc, + vhost_get_desc_size(vq, vq->num), + start, end); + vhost_invalidate_vmap(vq, &vq->used_ring, + (unsigned long)vq->used, + vhost_get_used_size(vq, vq->num), + start, end); + } +} + + +static void vhost_invalidate_range(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct vhost_dev *dev = container_of(mn, struct vhost_dev, + mmu_notifier); + + vhost_invalidate(dev, start, end); +} + +static const struct mmu_notifier_ops vhost_mmu_notifier_ops = { + .invalidate_range = vhost_invalidate_range, +}; + void vhost_dev_init(struct vhost_dev *dev, struct vhost_virtqueue **vqs, int nvqs, int iov_limit) { @@ -449,6 +517,7 @@ void vhost_dev_init(struct vhost_dev *dev, dev->mm = NULL; dev->worker = NULL; dev->iov_limit = iov_limit; + dev->mmu_notifier.ops = &vhost_mmu_notifier_ops; init_llist_head(&dev->work_list); init_waitqueue_head(&dev->wait); INIT_LIST_HEAD(&dev->read_list); @@ -462,6 +531,9 @@ void vhost_dev_init(struct vhost_dev *dev, vq->indirect = NULL; vq->heads = NULL; vq->dev = dev; + vq->avail_ring.addr = NULL; + vq->used_ring.addr = NULL; + vq->desc_ring.addr = NULL; mutex_init(&vq->mutex); vhost_vq_reset(dev, vq); if (vq->handle_kick) @@ -542,7 +614,13 @@ long vhost_dev_set_owner(struct vhost_dev *dev) if (err) goto err_cgroup; + err = mmu_notifier_register(&dev->mmu_notifier, dev->mm); + if (err) + goto err_mmu_notifier; + return 0; +err_mmu_notifier: + vhost_dev_free_iovecs(dev); err_cgroup: kthread_stop(worker); dev->worker = NULL; @@ -633,6 +711,81 @@ static void vhost_clear_msg(struct vhost_dev *dev) spin_unlock(&dev->iotlb_lock); } +static int vhost_init_vmap(struct vhost_dev *dev, + struct vhost_vmap *map, unsigned long uaddr, + size_t size, int write) +{ + struct page **pages; + int npages = DIV_ROUND_UP(size, PAGE_SIZE); + int npinned; + void *vaddr; + int err = -EFAULT; + + err = -ENOMEM; + pages = kmalloc_array(npages, sizeof(struct page *), GFP_KERNEL); + if (!pages) + goto err_uaddr; + + err = EFAULT; + npinned = get_user_pages_fast(uaddr, npages, write, pages); + if (npinned != npages) + goto err_gup; + + vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL); + if (!vaddr) + goto err_gup; + + map->addr = vaddr + (uaddr & (PAGE_SIZE - 1)); + map->unmap_addr = vaddr; + map->npages = npages; + map->pages = pages; + +err_gup: + /* Don't pin pages, mmu notifier will notify us about page + * migration. + */ + if (npinned > 0) + release_pages(pages, npinned); +err_uaddr: + return err; +} + +static void vhost_uninit_vq_vmaps(struct vhost_virtqueue *vq) +{ + vhost_uninit_vmap(&vq->avail_ring); + vhost_uninit_vmap(&vq->desc_ring); + vhost_uninit_vmap(&vq->used_ring); +} + +static int vhost_setup_avail_vmap(struct vhost_virtqueue *vq, + unsigned long avail) +{ + return vhost_init_vmap(vq->dev, &vq->avail_ring, avail, + vhost_get_avail_size(vq, vq->num), false); +} + +static int vhost_setup_desc_vmap(struct vhost_virtqueue *vq, + unsigned long desc) +{ + return vhost_init_vmap(vq->dev, &vq->desc_ring, desc, + vhost_get_desc_size(vq, vq->num), false); +} + +static int vhost_setup_used_vmap(struct vhost_virtqueue *vq, + unsigned long used) +{ + return vhost_init_vmap(vq->dev, &vq->used_ring, used, + vhost_get_used_size(vq, vq->num), true); +} + +static void vhost_set_vmap_dirty(struct vhost_vmap *used) +{ + int i; + + for (i = 0; i < used->npages; i++) + set_page_dirty_lock(used->pages[i]); +} + void vhost_dev_cleanup(struct vhost_dev *dev) { int i; @@ -662,8 +815,12 @@ void vhost_dev_cleanup(struct vhost_dev *dev) kthread_stop(dev->worker); dev->worker = NULL; } - if (dev->mm) + for (i = 0; i < dev->nvqs; i++) + vhost_uninit_vq_vmaps(dev->vqs[i]); + if (dev->mm) { + mmu_notifier_unregister(&dev->mmu_notifier, dev->mm); mmput(dev->mm); + } dev->mm = NULL; } EXPORT_SYMBOL_GPL(vhost_dev_cleanup); @@ -892,6 +1049,16 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq, static inline int vhost_put_avail_event(struct vhost_virtqueue *vq) { + if (!vq->iotlb) { + struct vring_used *used = vq->used_ring.addr; + + if (likely(used)) { + *((__virtio16 *)&used->ring[vq->num]) = + cpu_to_vhost16(vq, vq->avail_idx); + return 0; + } + } + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->avail_idx), vhost_avail_event(vq)); } @@ -900,6 +1067,16 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, struct vring_used_elem *head, int idx, int count) { + if (!vq->iotlb) { + struct vring_used *used = vq->used_ring.addr; + + if (likely(used)) { + memcpy(used->ring + idx, head, + count * sizeof(*head)); + return 0; + } + } + return vhost_copy_to_user(vq, vq->used->ring + idx, head, count * sizeof(*head)); } @@ -907,6 +1084,15 @@ static inline int vhost_put_used(struct vhost_virtqueue *vq, static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) { + if (!vq->iotlb) { + struct vring_used *used = vq->used_ring.addr; + + if (likely(used)) { + used->flags = cpu_to_vhost16(vq, vq->used_flags); + return 0; + } + } + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->used_flags), &vq->used->flags); } @@ -914,6 +1100,15 @@ static inline int vhost_put_used_flags(struct vhost_virtqueue *vq) static inline int vhost_put_used_idx(struct vhost_virtqueue *vq) { + if (!vq->iotlb) { + struct vring_used *used = vq->used_ring.addr; + + if (likely(used)) { + used->idx = cpu_to_vhost16(vq, vq->last_used_idx); + return 0; + } + } + return vhost_put_user(vq, cpu_to_vhost16(vq, vq->last_used_idx), &vq->used->idx); } @@ -959,12 +1154,30 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { + if (!vq->iotlb) { + struct vring_avail *avail = vq->avail_ring.addr; + + if (likely(avail)) { + *idx = avail->idx; + return 0; + } + } + return vhost_get_avail(vq, *idx, &vq->avail->idx); } static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, __virtio16 *head, int idx) { + if (!vq->iotlb) { + struct vring_avail *avail = vq->avail_ring.addr; + + if (likely(avail)) { + *head = avail->ring[idx & (vq->num - 1)]; + return 0; + } + } + return vhost_get_avail(vq, *head, &vq->avail->ring[idx & (vq->num - 1)]); } @@ -972,24 +1185,60 @@ static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, static inline int vhost_get_avail_flags(struct vhost_virtqueue *vq, __virtio16 *flags) { + if (!vq->iotlb) { + struct vring_avail *avail = vq->avail_ring.addr; + + if (likely(avail)) { + *flags = avail->flags; + return 0; + } + } + return vhost_get_avail(vq, *flags, &vq->avail->flags); } static inline int vhost_get_used_event(struct vhost_virtqueue *vq, __virtio16 *event) { + if (!vq->iotlb) { + struct vring_avail *avail = vq->avail_ring.addr; + + if (likely(avail)) { + *event = (__virtio16)avail->ring[vq->num]; + return 0; + } + } + return vhost_get_avail(vq, *event, vhost_used_event(vq)); } static inline int vhost_get_used_idx(struct vhost_virtqueue *vq, __virtio16 *idx) { + if (!vq->iotlb) { + struct vring_used *used = vq->used_ring.addr; + + if (likely(used)) { + *idx = used->idx; + return 0; + } + } + return vhost_get_used(vq, *idx, &vq->used->idx); } static inline int vhost_get_desc(struct vhost_virtqueue *vq, struct vring_desc *desc, int idx) { + if (!vq->iotlb) { + struct vring_desc *d = vq->desc_ring.addr; + + if (likely(d)) { + *desc = *(d + idx); + return 0; + } + } + return vhost_copy_from_user(vq, desc, vq->desc + idx, sizeof(*desc)); } @@ -1330,8 +1579,16 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq) { unsigned int num = vq->num; - if (!vq->iotlb) + if (!vq->iotlb) { + if (unlikely(!vq->avail_ring.addr)) + vhost_setup_avail_vmap(vq, (unsigned long)vq->avail); + if (unlikely(!vq->desc_ring.addr)) + vhost_setup_desc_vmap(vq, (unsigned long)vq->desc); + if (unlikely(!vq->used_ring.addr)) + vhost_setup_used_vmap(vq, (unsigned long)vq->used); + return 1; + } return iotlb_access_ok(vq, VHOST_ACCESS_RO, (u64)(uintptr_t)vq->desc, vhost_get_desc_size(vq, num), VHOST_ADDR_DESC) && @@ -1343,6 +1600,15 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq) } EXPORT_SYMBOL_GPL(vq_meta_prefetch); +void vq_meta_prefetch_done(struct vhost_virtqueue *vq) +{ + if (vq->iotlb) + return; + if (likely(vq->used_ring.addr)) + vhost_set_vmap_dirty(&vq->used_ring); +} +EXPORT_SYMBOL_GPL(vq_meta_prefetch_done); + /* Can we log writes? */ /* Caller should have device mutex but not vq mutex */ bool vhost_log_access_ok(struct vhost_dev *dev) @@ -1483,6 +1749,13 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg mutex_lock(&vq->mutex); + /* Unregister MMU notifer to allow invalidation callback + * can access vq->avail, vq->desc , vq->used and vq->num + * without holding vq->mutex. + */ + if (d->mm) + mmu_notifier_unregister(&d->mmu_notifier, d->mm); + switch (ioctl) { case VHOST_SET_VRING_NUM: /* Resizing ring with an active backend? @@ -1499,6 +1772,7 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg r = -EINVAL; break; } + vhost_uninit_vq_vmaps(vq); vq->num = s.num; break; case VHOST_SET_VRING_BASE: @@ -1581,6 +1855,7 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg vq->avail = (void __user *)(unsigned long)a.avail_user_addr; vq->log_addr = a.log_guest_addr; vq->used = (void __user *)(unsigned long)a.used_user_addr; + vhost_uninit_vq_vmaps(vq); break; case VHOST_SET_VRING_KICK: if (copy_from_user(&f, argp, sizeof f)) { @@ -1656,6 +1931,8 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg if (pollstart && vq->handle_kick) r = vhost_poll_start(&vq->poll, vq->kick); + if (d->mm) + mmu_notifier_register(&d->mmu_notifier, d->mm); mutex_unlock(&vq->mutex); if (pollstop && vq->handle_kick) diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 7a7fc00..146076e 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -12,6 +12,8 @@ #include #include #include +#include +#include struct vhost_work; typedef void (*vhost_work_fn_t)(struct vhost_work *work); @@ -80,6 +82,13 @@ enum vhost_uaddr_type { VHOST_NUM_ADDRS = 3, }; +struct vhost_vmap { + void *addr; + void *unmap_addr; + int npages; + struct page **pages; +}; + /* The virtqueue structure describes a queue attached to a device. */ struct vhost_virtqueue { struct vhost_dev *dev; @@ -90,6 +99,11 @@ struct vhost_virtqueue { struct vring_desc __user *desc; struct vring_avail __user *avail; struct vring_used __user *used; + + struct vhost_vmap avail_ring; + struct vhost_vmap desc_ring; + struct vhost_vmap used_ring; + const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS]; struct file *kick; struct eventfd_ctx *call_ctx; @@ -158,6 +172,7 @@ struct vhost_msg_node { struct vhost_dev { struct mm_struct *mm; + struct mmu_notifier mmu_notifier; struct mutex mutex; struct vhost_virtqueue **vqs; int nvqs; @@ -210,6 +225,7 @@ int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, unsigned int log_num, u64 len, struct iovec *iov, int count); int vq_meta_prefetch(struct vhost_virtqueue *vq); +void vq_meta_prefetch_done(struct vhost_virtqueue *vq); struct vhost_msg_node *vhost_new_msg(struct vhost_virtqueue *vq, int type); void vhost_enqueue_msg(struct vhost_dev *dev,