From patchwork Wed Feb 28 14:41:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575622 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6812615A4B0; Wed, 28 Feb 2024 14:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131248; cv=none; b=YRO1PDpTOEdYVH3vDS2VVoWk2grxU3XNmCUFPfjw16d2Uu5QKIAfJz/4wOu+z0t2BvCpGZaCM5xpYfiODaGoW5GAUJuZg2NvUAC+d6Frd6+YJiUpEpQ3ol71mouBXdWDJ02sHScbUyilzOo1XsrVRRxvZSWIQbUxBRi7xzrtATM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131248; c=relaxed/simple; bh=GOzQd1+4k7aY5WsMAE+wWu8EQfHg2z+crAl/K709Yoo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HFIj0C36qOA8GHd/vvDYATftft931gi6d+lr2P0Ppp2itCkYOoOTrbaiaHRNNKpg3H6+OjisdST/0MuAiG5+Xz5A16jdLwYOBTBh2dhmUd0bq/AqEo/vZAEDaFOm4yG90Dz7Nkt61Rv7OPkLB3THtufYZ4nCwxenXpUTwfdLGH4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8M1vbnz4f3lfH; Wed, 28 Feb 2024 22:40:35 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 961511A0DAF; Wed, 28 Feb 2024 22:40:42 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S5; Wed, 28 Feb 2024 22:40:42 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 1/6] fuse: limit the length of ITER_KVEC dio by max_pages Date: Wed, 28 Feb 2024 22:41:21 +0800 Message-Id: <20240228144126.2864064-2-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S5 X-Coremail-Antispam: 1UD129KBjvJXoW3AF1DtrW3ur4UCFWrXry5Arb_yoW7Gw43pr W3KF17uFs3XF47uws3JF1UuFyrCwnrJF43Xr95Z3s3ur1UZryIkF98K3Wa9FW7CrZ7Jw1x XrsYy3sFvwn0vaDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBjb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW5JVW7JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx 0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWU JVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCF04k20xvY0x0EwI xGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k2 6cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxV AFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x07jesjbUUUUU= X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao When trying to insert a 10MB kernel module kept in a virtio-fs with cache disabled, the following warning was reported: ------------[ cut here ]------------ WARNING: CPU: 2 PID: 439 at mm/page_alloc.c:4544 ...... Modules linked in: CPU: 2 PID: 439 Comm: insmod Not tainted 6.7.0-rc7+ #33 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), ...... RIP: 0010:__alloc_pages+0x2c4/0x360 ...... Call Trace: ? __warn+0x8f/0x150 ? __alloc_pages+0x2c4/0x360 __kmalloc_large_node+0x86/0x160 __kmalloc+0xcd/0x140 virtio_fs_enqueue_req+0x240/0x6d0 virtio_fs_wake_pending_and_unlock+0x7f/0x190 queue_request_and_unlock+0x58/0x70 fuse_simple_request+0x18b/0x2e0 fuse_direct_io+0x58a/0x850 fuse_file_read_iter+0xdb/0x130 __kernel_read+0xf3/0x260 kernel_read+0x45/0x60 kernel_read_file+0x1ad/0x2b0 init_module_from_file+0x6a/0xe0 idempotent_init_module+0x179/0x230 __x64_sys_finit_module+0x5d/0xb0 do_syscall_64+0x36/0xb0 entry_SYSCALL_64_after_hwframe+0x6e/0x76 ...... ---[ end trace 0000000000000000 ]--- The warning is triggered when: 1) inserting a 10MB sized kernel module kept in a virtiofs. syscall finit_module() will handle the module insertion and it will invoke kernel_read_file() to read the content of the module first. 2) kernel_read_file() allocates a 10MB buffer by using vmalloc() and passes it to kernel_read(). kernel_read() constructs a kvec iter by using iov_iter_kvec() and passes it to fuse_file_read_iter(). 3) virtio-fs disables the cache, so fuse_file_read_iter() invokes fuse_direct_io(). As for now, the maximal read size for kvec iter is only limited by fc->max_read. For virtio-fs, max_read is UINT_MAX, so fuse_direct_io() doesn't split the 10MB buffer. It saves the address and the size of the 10MB-sized buffer in out_args[0] of a fuse request and passes the fuse request to virtio_fs_wake_pending_and_unlock(). 4) virtio_fs_wake_pending_and_unlock() uses virtio_fs_enqueue_req() to queue the request. Because the arguments in fuse request may be kept in stack, so virtio_fs_enqueue_req() uses kmalloc() to allocate a bounce buffer for all fuse args, copies these args into the bounce buffer and passed the physical address of the bounce buffer to virtiofsd. The total length of these fuse args for the passed fuse request is about 10MB, so copy_args_to_argbuf() invokes kmalloc() with a 10MB size parameter and it triggers the warning in __alloc_pages(): if (WARN_ON_ONCE_GFP(order > MAX_PAGE_ORDER, gfp)) return NULL; 5) virtio_fs_enqueue_req() will retry the memory allocation in a kworker, but it won't help, because kmalloc() will always return NULL due to the abnormal size and finit_module() will hang forever. A feasible solution is to limit the value of max_read for virtio-fs, so the length passed to kmalloc() will be limited. However it will affect the maximal read size for normal fuse read. And for virtio-fs write initiated from kernel, it has the similar problem and now there is no way to limit fc->max_write in kernel. So instead of limiting both the values of max_read and max_write in kernel, capping the maximal length of kvec iter IO by using max_pages in fuse_direct_io() just like it does for ubuf/iovec iter IO. Now the max value for max_pages is 256, so on host with 4KB page size, the maximal size passed to kmalloc() in copy_args_to_argbuf() is about 1MB+40B. The allocation of 2MB of physically contiguous memory will still incur significant stress on the memory subsystem, but the warning is fixed. Additionally, the requirement for huge physically contiguous memory will be removed in the following patch. Fixes: a62a8ef9d97d ("virtio-fs: add virtiofs filesystem") Signed-off-by: Hou Tao --- fs/fuse/file.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 148a71b8b4d0e..f90ea25e366f0 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1423,6 +1423,16 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii, return ret < 0 ? ret : 0; } +static size_t fuse_max_dio_rw_size(const struct fuse_conn *fc, + const struct iov_iter *iter, int write) +{ + unsigned int nmax = write ? fc->max_write : fc->max_read; + + if (iov_iter_is_kvec(iter)) + nmax = min(nmax, fc->max_pages << PAGE_SHIFT); + return nmax; +} + ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, loff_t *ppos, int flags) { @@ -1433,7 +1443,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, struct inode *inode = mapping->host; struct fuse_file *ff = file->private_data; struct fuse_conn *fc = ff->fm->fc; - size_t nmax = write ? fc->max_write : fc->max_read; + size_t nmax = fuse_max_dio_rw_size(fc, iter, write); loff_t pos = *ppos; size_t count = iov_iter_count(iter); pgoff_t idx_from = pos >> PAGE_SHIFT; From patchwork Wed Feb 28 14:41:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575623 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6816015A4B9; Wed, 28 Feb 2024 14:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131248; cv=none; b=RSQqi/xG6bGWDeS2UshEOdnGJ6jvfh14YHdMCtfq2QNuUHiwv70dPzk7VT8Vj6GX7DP1kI4kcolbso27/LuzDXos+WvMR8irLTvrDsGo+ql6LcQu679AaJregYb9BPlTuuo9Rxo6umxp/FDTi5vNo0ZTA4jYL8hKaTC5Mv7hHUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131248; c=relaxed/simple; bh=2KwL01sAiXpjiF+csHP1C/1iZ97LREkkPfTG1NxGkm4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pi7FFIktczMPf9zMs/6h9tGzJ8ostc9NxbYEjeh90t+4Kwkw8B8Z8USjNOSLvAqn12LfheKhMdIAyaIP9ZXPy2lPm6yrG75u06ZiPakxhmCSbfkzNKy6Z/qF2g3sXpW+6vF2v4HkcrItsEVVprfgWT9oLHquuxK11GzhoPttHEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8R52DLz4f3kJr; Wed, 28 Feb 2024 22:40:39 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 14E0D1A0DAF; Wed, 28 Feb 2024 22:40:43 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S6; Wed, 28 Feb 2024 22:40:42 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 2/6] virtiofs: move alloc/free of argbuf into separated helpers Date: Wed, 28 Feb 2024 22:41:22 +0800 Message-Id: <20240228144126.2864064-3-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S6 X-Coremail-Antispam: 1UD129KBjvJXoWxJF4kAFyDuF1rZF1fKrWUtwb_yoWrGry5pr 43tw15ZFZ3XFZrWF95KF4UAw1Fy3yS93WUGrZ3W3sxKF1UXw47XFy0yFy0qr90vrWkCF4x Ar4FqF4UWF4rXaDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBjb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW5JVW7JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx 0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWU JVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCF04k20xvY0x0EwI xGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k2 6cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxV AFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x07UC9aPUUUUU= X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao The bounce buffer for fuse args in virtiofs will be extended to support scatterd pages later. Therefore, move the allocation and the free of argbuf out of the copy procedures and factor them into virtio_fs_argbuf_{new|free}() helpers. Signed-off-by: Hou Tao --- fs/fuse/virtio_fs.c | 52 +++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 21 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index 5f1be1da92ce9..cd1330506daba 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -404,6 +404,24 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work) } } +static void virtio_fs_argbuf_free(void *argbuf) +{ + kfree(argbuf); +} + +static void *virtio_fs_argbuf_new(struct fuse_args *args, gfp_t gfp) +{ + unsigned int numargs; + unsigned int len; + + numargs = args->in_numargs - args->in_pages; + len = fuse_len_args(numargs, (struct fuse_arg *) args->in_args); + numargs = args->out_numargs - args->out_pages; + len += fuse_len_args(numargs, args->out_args); + + return kmalloc(len, gfp); +} + /* * Returns 1 if queue is full and sender should wait a bit before sending * next request, 0 otherwise. @@ -487,36 +505,24 @@ static void virtio_fs_hiprio_dispatch_work(struct work_struct *work) } } -/* Allocate and copy args into req->argbuf */ -static int copy_args_to_argbuf(struct fuse_req *req) +/* Copy args into req->argbuf */ +static void copy_args_to_argbuf(struct fuse_req *req) { struct fuse_args *args = req->args; unsigned int offset = 0; unsigned int num_in; - unsigned int num_out; - unsigned int len; unsigned int i; num_in = args->in_numargs - args->in_pages; - num_out = args->out_numargs - args->out_pages; - len = fuse_len_args(num_in, (struct fuse_arg *) args->in_args) + - fuse_len_args(num_out, args->out_args); - - req->argbuf = kmalloc(len, GFP_ATOMIC); - if (!req->argbuf) - return -ENOMEM; - for (i = 0; i < num_in; i++) { memcpy(req->argbuf + offset, args->in_args[i].value, args->in_args[i].size); offset += args->in_args[i].size; } - - return 0; } -/* Copy args out of and free req->argbuf */ +/* Copy args out of req->argbuf */ static void copy_args_from_argbuf(struct fuse_args *args, struct fuse_req *req) { unsigned int remaining; @@ -549,9 +555,6 @@ static void copy_args_from_argbuf(struct fuse_args *args, struct fuse_req *req) /* Store the actual size of the variable-length arg */ if (args->out_argvar) args->out_args[args->out_numargs - 1].size = remaining; - - kfree(req->argbuf); - req->argbuf = NULL; } /* Work function for request completion */ @@ -571,6 +574,9 @@ static void virtio_fs_request_complete(struct fuse_req *req, args = req->args; copy_args_from_argbuf(args, req); + virtio_fs_argbuf_free(req->argbuf); + req->argbuf = NULL; + if (args->out_pages && args->page_zeroing) { len = args->out_args[args->out_numargs - 1].size; ap = container_of(args, typeof(*ap), args); @@ -1149,9 +1155,13 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, } /* Use a bounce buffer since stack args cannot be mapped */ - ret = copy_args_to_argbuf(req); - if (ret < 0) + req->argbuf = virtio_fs_argbuf_new(args, GFP_ATOMIC); + if (!req->argbuf) { + ret = -ENOMEM; goto out; + } + + copy_args_to_argbuf(req); /* Request elements */ sg_init_one(&sg[out_sgs++], &req->in.h, sizeof(req->in.h)); @@ -1210,7 +1220,7 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, out: if (ret < 0 && req->argbuf) { - kfree(req->argbuf); + virtio_fs_argbuf_free(req->argbuf); req->argbuf = NULL; } if (sgs != stack_sgs) { From patchwork Wed Feb 28 14:41:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575624 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 680DC15A4AF; Wed, 28 Feb 2024 14:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; cv=none; b=gZ3e3cZX3zcX/OrEOesDduPn6j6+1h8pIIZanX529C85tBBCa2KGKGTG4Te/Iq1G23IgreKkWCPfZfVhSmSpuyp+gWp6xptijLuz54sUkQqNWfm0UIHSbNGqGXG+gPFq/qok1DqP2KrAN8UyRhk1LCUFsMm9aj26FtvNwU20I6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; c=relaxed/simple; bh=huCSXdGFSbPcRPMpXRU5L/75Pdi/x+LgE/kzjwzotB0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JCfbRNJ0ZUYJ8zTBzynjn2P++DALyyW+REZ4sVxLSVxvQQElHurXcg7K/joRXDYuAaulPISwYKBnLxxJa1VENwoE5m0d/vv88tpY4tM/LV6AwD3rvy/sOJPKn/RKyChPm0PV6sAp1SflSK9N98TcwEhNGmwzyXfjhalZhqI1Pgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8N1SVSz4f3lVP; Wed, 28 Feb 2024 22:40:36 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 86C1B1A016E; Wed, 28 Feb 2024 22:40:43 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S7; Wed, 28 Feb 2024 22:40:43 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 3/6] virtiofs: factor out more common methods for argbuf Date: Wed, 28 Feb 2024 22:41:23 +0800 Message-Id: <20240228144126.2864064-4-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S7 X-Coremail-Antispam: 1UD129KBjvJXoWxKFWkCryDJFy3Ar47Gr1xZrb_yoWxJrykpF 45tw15XFWfJFZFgFyrGF4rA3WSk393uw1xGrZ3G3sxKF1UXw47XFy8AryjkrnIvrykAF4x AFsaqr4UWF48uaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBjb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUWw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW5JVW7JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx 0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWU JVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCF04k20xvY0x0EwI xGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k2 6cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxV AFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x07UAkuxUUUUU= X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao Factor out more common methods for bounce buffer of fuse args: 1) virtio_fs_argbuf_setup_sg: set-up sgs for bounce buffer 2) virtio_fs_argbuf_copy_from_in_arg: copy each in-arg to bounce buffer 3) virtio_fs_argbuf_out_args_offset: calc the start offset of out-arg 4) virtio_fs_argbuf_copy_to_out_arg: copy bounce buffer to each out-arg These methods will be used to implement bounce buffer backed by scattered pages which are allocated separatedly. Signed-off-by: Hou Tao --- fs/fuse/virtio_fs.c | 77 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 60 insertions(+), 17 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index cd1330506daba..f10fff7f23a0f 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -86,6 +86,10 @@ struct virtio_fs_req_work { struct work_struct done_work; }; +struct virtio_fs_argbuf { + DECLARE_FLEX_ARRAY(u8, buf); +}; + static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, struct fuse_req *req, bool in_flight); @@ -404,13 +408,15 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work) } } -static void virtio_fs_argbuf_free(void *argbuf) +static void virtio_fs_argbuf_free(struct virtio_fs_argbuf *argbuf) { kfree(argbuf); } -static void *virtio_fs_argbuf_new(struct fuse_args *args, gfp_t gfp) +static struct virtio_fs_argbuf *virtio_fs_argbuf_new(struct fuse_args *args, + gfp_t gfp) { + struct virtio_fs_argbuf *argbuf; unsigned int numargs; unsigned int len; @@ -419,7 +425,41 @@ static void *virtio_fs_argbuf_new(struct fuse_args *args, gfp_t gfp) numargs = args->out_numargs - args->out_pages; len += fuse_len_args(numargs, args->out_args); - return kmalloc(len, gfp); + argbuf = kmalloc(struct_size(argbuf, buf, len), gfp); + + return argbuf; +} + +static unsigned int virtio_fs_argbuf_setup_sg(struct virtio_fs_argbuf *argbuf, + unsigned int offset, + unsigned int len, + struct scatterlist *sg) +{ + sg_init_one(sg, argbuf->buf + offset, len); + return 1; +} + +static void virtio_fs_argbuf_copy_from_in_arg(struct virtio_fs_argbuf *argbuf, + unsigned int offset, + const void *src, unsigned int len) +{ + memcpy(argbuf->buf + offset, src, len); +} + +static unsigned int +virtio_fs_argbuf_out_args_offset(struct virtio_fs_argbuf *argbuf, + const struct fuse_args *args) +{ + unsigned int num_in = args->in_numargs - args->in_pages; + + return fuse_len_args(num_in, (struct fuse_arg *)args->in_args); +} + +static void virtio_fs_argbuf_copy_to_out_arg(struct virtio_fs_argbuf *argbuf, + unsigned int offset, void *dst, + unsigned int len) +{ + memcpy(dst, argbuf->buf + offset, len); } /* @@ -515,9 +555,9 @@ static void copy_args_to_argbuf(struct fuse_req *req) num_in = args->in_numargs - args->in_pages; for (i = 0; i < num_in; i++) { - memcpy(req->argbuf + offset, - args->in_args[i].value, - args->in_args[i].size); + virtio_fs_argbuf_copy_from_in_arg(req->argbuf, offset, + args->in_args[i].value, + args->in_args[i].size); offset += args->in_args[i].size; } } @@ -525,17 +565,19 @@ static void copy_args_to_argbuf(struct fuse_req *req) /* Copy args out of req->argbuf */ static void copy_args_from_argbuf(struct fuse_args *args, struct fuse_req *req) { + struct virtio_fs_argbuf *argbuf; unsigned int remaining; unsigned int offset; - unsigned int num_in; unsigned int num_out; unsigned int i; remaining = req->out.h.len - sizeof(req->out.h); - num_in = args->in_numargs - args->in_pages; num_out = args->out_numargs - args->out_pages; - offset = fuse_len_args(num_in, (struct fuse_arg *)args->in_args); + if (!num_out) + goto out; + argbuf = req->argbuf; + offset = virtio_fs_argbuf_out_args_offset(argbuf, args); for (i = 0; i < num_out; i++) { unsigned int argsize = args->out_args[i].size; @@ -545,13 +587,16 @@ static void copy_args_from_argbuf(struct fuse_args *args, struct fuse_req *req) argsize = remaining; } - memcpy(args->out_args[i].value, req->argbuf + offset, argsize); + virtio_fs_argbuf_copy_to_out_arg(argbuf, offset, + args->out_args[i].value, + argsize); offset += argsize; if (i != args->out_numargs - 1) remaining -= argsize; } +out: /* Store the actual size of the variable-length arg */ if (args->out_argvar) args->out_args[args->out_numargs - 1].size = remaining; @@ -1100,7 +1145,6 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg, struct fuse_arg *args, unsigned int numargs, bool argpages, - void *argbuf, unsigned int *len_used) { struct fuse_args_pages *ap = container_of(req->args, typeof(*ap), args); @@ -1109,7 +1153,8 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg, len = fuse_len_args(numargs - argpages, args); if (len) - sg_init_one(&sg[total_sgs++], argbuf, len); + total_sgs += virtio_fs_argbuf_setup_sg(req->argbuf, *len_used, + len, &sg[total_sgs]); if (argpages) total_sgs += sg_init_fuse_pages(&sg[total_sgs], @@ -1117,8 +1162,7 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg, ap->num_pages, args[numargs - 1].size); - if (len_used) - *len_used = len; + *len_used = len; return total_sgs; } @@ -1168,7 +1212,7 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, out_sgs += sg_init_fuse_args(&sg[out_sgs], req, (struct fuse_arg *)args->in_args, args->in_numargs, args->in_pages, - req->argbuf, &argbuf_used); + &argbuf_used); /* Reply elements */ if (test_bit(FR_ISREPLY, &req->flags)) { @@ -1176,8 +1220,7 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, &req->out.h, sizeof(req->out.h)); in_sgs += sg_init_fuse_args(&sg[out_sgs + in_sgs], req, args->out_args, args->out_numargs, - args->out_pages, - req->argbuf + argbuf_used, NULL); + args->out_pages, &argbuf_used); } WARN_ON(out_sgs + in_sgs != total_sgs); From patchwork Wed Feb 28 14:41:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575626 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D12B15A4BA; Wed, 28 Feb 2024 14:40:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; cv=none; b=jZKQY3iMpuAKbYr1iyC/hrV+gwvtU02b/DRk2oVN3CDy0cPZD6FBNeIZL4mvz93DkRWqyB7Z0yeXyxVZL1hmqTvLVef85IbAXHLp+e2ryN7Mqm5gICnSy6cuQX5+D2101gVozO082qGxsDiEdEvmFw4ULZNo/XbRMEQHkSvLGHo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; c=relaxed/simple; bh=gGrqqWhOprABAylbdHf6d3Bgw6OyAPvbcs/3s/7xVEs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M53DM6yh8G3ytdjXlJb9tmli41yWyG1ON0crpdB8xy8nZwUceX6+VeAGDdQ2dRKzHYmB71cRh7YowtO9493J5Jspze0Ahxf8Ap5tDjRsIGbXkIiqg0uj1dwhJF7mEiEl8D/axYkjdKMWA4f0mrBt3WzmVypISKaZXIerz9Pjdeo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8S4ZRtz4f3kK1; Wed, 28 Feb 2024 22:40:40 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 0377C1A0172; Wed, 28 Feb 2024 22:40:44 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S8; Wed, 28 Feb 2024 22:40:43 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 4/6] virtiofs: support bounce buffer backed by scattered pages Date: Wed, 28 Feb 2024 22:41:24 +0800 Message-Id: <20240228144126.2864064-5-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S8 X-Coremail-Antispam: 1UD129KBjvJXoW3GrW5XFyxKFWrJr1rJw1fWFg_yoW3Gr1xpF 4Fyw15JrWfJrW7Kry8GF48AF1Skws3uw1xGrZ3X3sIkw1UXw4xXFyUAry0vrnxJrykCF1x JF1FqF18Wr4q9aUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUB2b4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVWxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAF wI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2 WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkE bVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCF04k20xvY0x 0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E 7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcV C0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UMIIF0xvE 42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6x kF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxUFgAwUUUUU X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao When reading a file kept in virtiofs from kernel (e.g., insmod a kernel module), if the cache of virtiofs is disabled, the read buffer will be passed to virtiofs through out_args[0].value instead of pages. Because virtiofs can't get the pages for the read buffer, virtio_fs_argbuf_new() will create a bounce buffer for the read buffer by using kmalloc() and copy the read buffer into bounce buffer. If the read buffer is large (e.g., 1MB), the allocation will incur significant stress on the memory subsystem. So instead of allocating bounce buffer by using kmalloc(), allocate a bounce buffer which is backed by scattered pages. The original idea is to use vmap(), but the use of GFP_ATOMIC is no possible for vmap(). To simplify the copy operations in the bounce buffer, use a bio_vec flex array to represent the argbuf. Also add an is_flat field in struct virtio_fs_argbuf to distinguish between kmalloc-ed and scattered bounce buffer. Signed-off-by: Hou Tao --- fs/fuse/virtio_fs.c | 163 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 149 insertions(+), 14 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index f10fff7f23a0f..ffea684bd100d 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -86,10 +86,27 @@ struct virtio_fs_req_work { struct work_struct done_work; }; -struct virtio_fs_argbuf { +struct virtio_fs_flat_argbuf { DECLARE_FLEX_ARRAY(u8, buf); }; +struct virtio_fs_scattered_argbuf { + unsigned int size; + unsigned int nr; + DECLARE_FLEX_ARRAY(struct bio_vec, bvec); +}; + +struct virtio_fs_argbuf { + bool is_flat; + /* There is flexible array in the end of these two struct + * definitions, so they must be the last field. + */ + union { + struct virtio_fs_flat_argbuf f; + struct virtio_fs_scattered_argbuf s; + }; +}; + static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, struct fuse_req *req, bool in_flight); @@ -408,42 +425,143 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work) } } +static unsigned int virtio_fs_argbuf_len(unsigned int in_args_len, + unsigned int out_args_len, + bool is_flat) +{ + if (is_flat) + return in_args_len + out_args_len; + + /* + * Align in_args_len with PAGE_SIZE to reduce the total number of + * sg entries when the value of out_args_len (e.g., the length of + * read buffer) is page-aligned. + */ + return round_up(in_args_len, PAGE_SIZE) + + round_up(out_args_len, PAGE_SIZE); +} + static void virtio_fs_argbuf_free(struct virtio_fs_argbuf *argbuf) { + unsigned int i; + + if (!argbuf) + return; + + if (argbuf->is_flat) + goto free_argbuf; + + for (i = 0; i < argbuf->s.nr; i++) + __free_page(argbuf->s.bvec[i].bv_page); + +free_argbuf: kfree(argbuf); } static struct virtio_fs_argbuf *virtio_fs_argbuf_new(struct fuse_args *args, - gfp_t gfp) + gfp_t gfp, bool is_flat) { struct virtio_fs_argbuf *argbuf; unsigned int numargs; - unsigned int len; + unsigned int in_len, out_len, len; + unsigned int i, nr; numargs = args->in_numargs - args->in_pages; - len = fuse_len_args(numargs, (struct fuse_arg *) args->in_args); + in_len = fuse_len_args(numargs, (struct fuse_arg *) args->in_args); numargs = args->out_numargs - args->out_pages; - len += fuse_len_args(numargs, args->out_args); + out_len = fuse_len_args(numargs, args->out_args); + len = virtio_fs_argbuf_len(in_len, out_len, is_flat); + + if (is_flat) { + argbuf = kmalloc(struct_size(argbuf, f.buf, len), gfp); + if (argbuf) + argbuf->is_flat = true; + + return argbuf; + } + + nr = len >> PAGE_SHIFT; + argbuf = kmalloc(struct_size(argbuf, s.bvec, nr), gfp); + if (!argbuf) + return NULL; + + argbuf->is_flat = false; + argbuf->s.size = len; + argbuf->s.nr = 0; + for (i = 0; i < nr; i++) { + struct page *page; + + page = alloc_page(gfp); + if (!page) { + virtio_fs_argbuf_free(argbuf); + return NULL; + } + bvec_set_page(&argbuf->s.bvec[i], page, PAGE_SIZE, 0); + argbuf->s.nr++; + } + + /* Zero the unused space for in_args */ + if (in_len & ~PAGE_MASK) { + struct iov_iter iter; + unsigned int to_zero; + + iov_iter_bvec(&iter, ITER_DEST, argbuf->s.bvec, argbuf->s.nr, + argbuf->s.size); + iov_iter_advance(&iter, in_len); - argbuf = kmalloc(struct_size(argbuf, buf, len), gfp); + to_zero = PAGE_SIZE - (in_len & ~PAGE_MASK); + iov_iter_zero(to_zero, &iter); + } return argbuf; } static unsigned int virtio_fs_argbuf_setup_sg(struct virtio_fs_argbuf *argbuf, unsigned int offset, - unsigned int len, + unsigned int *len, struct scatterlist *sg) { - sg_init_one(sg, argbuf->buf + offset, len); - return 1; + struct bvec_iter bi = { + .bi_size = offset + *len, + }; + struct scatterlist *cur; + struct bio_vec bv; + + if (argbuf->is_flat) { + sg_init_one(sg, argbuf->f.buf + offset, *len); + return 1; + } + + cur = sg; + bvec_iter_advance(argbuf->s.bvec, &bi, offset); + for_each_bvec(bv, argbuf->s.bvec, bi, bi) { + sg_init_table(cur, 1); + sg_set_page(cur, bv.bv_page, bv.bv_len, bv.bv_offset); + cur++; + } + *len = round_up(*len, PAGE_SIZE); + + return cur - sg; } static void virtio_fs_argbuf_copy_from_in_arg(struct virtio_fs_argbuf *argbuf, unsigned int offset, const void *src, unsigned int len) { - memcpy(argbuf->buf + offset, src, len); + struct iov_iter iter; + unsigned int copied; + + if (argbuf->is_flat) { + memcpy(argbuf->f.buf + offset, src, len); + return; + } + + iov_iter_bvec(&iter, ITER_DEST, argbuf->s.bvec, + argbuf->s.nr, argbuf->s.size); + iov_iter_advance(&iter, offset); + + copied = _copy_to_iter(src, len, &iter); + WARN_ON_ONCE(copied != len); } static unsigned int @@ -451,15 +569,32 @@ virtio_fs_argbuf_out_args_offset(struct virtio_fs_argbuf *argbuf, const struct fuse_args *args) { unsigned int num_in = args->in_numargs - args->in_pages; + unsigned int offset = fuse_len_args(num_in, + (struct fuse_arg *)args->in_args); - return fuse_len_args(num_in, (struct fuse_arg *)args->in_args); + if (argbuf->is_flat) + return offset; + return round_up(offset, PAGE_SIZE); } static void virtio_fs_argbuf_copy_to_out_arg(struct virtio_fs_argbuf *argbuf, unsigned int offset, void *dst, unsigned int len) { - memcpy(dst, argbuf->buf + offset, len); + struct iov_iter iter; + unsigned int copied; + + if (argbuf->is_flat) { + memcpy(dst, argbuf->f.buf + offset, len); + return; + } + + iov_iter_bvec(&iter, ITER_SOURCE, argbuf->s.bvec, + argbuf->s.nr, argbuf->s.size); + iov_iter_advance(&iter, offset); + + copied = _copy_from_iter(dst, len, &iter); + WARN_ON_ONCE(copied != len); } /* @@ -1154,7 +1289,7 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg, len = fuse_len_args(numargs - argpages, args); if (len) total_sgs += virtio_fs_argbuf_setup_sg(req->argbuf, *len_used, - len, &sg[total_sgs]); + &len, &sg[total_sgs]); if (argpages) total_sgs += sg_init_fuse_pages(&sg[total_sgs], @@ -1199,7 +1334,7 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, } /* Use a bounce buffer since stack args cannot be mapped */ - req->argbuf = virtio_fs_argbuf_new(args, GFP_ATOMIC); + req->argbuf = virtio_fs_argbuf_new(args, GFP_ATOMIC, true); if (!req->argbuf) { ret = -ENOMEM; goto out; From patchwork Wed Feb 28 14:41:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575627 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F2B415AADD; Wed, 28 Feb 2024 14:40:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; cv=none; b=k6tUObvYpDv2VPHSnwhndFJ4BawoCg92Z0lVxVl0zhiGkVUe+BsldCKTqxudZ2hlIG19A1zaYaLZJD5r4DpNxuvOGVODnQFezHGMYAnrxML8eBwPKC2vvmZmWL/bwp9BYN+ntijjDpgwUUj+u1wP1FnM0wPpFjdCe7GCbgEQo88= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131249; c=relaxed/simple; bh=5tKZoLoQHsYKSVgtcARv6ya8kA/+8jNNy0nuW/K4E4Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jaT2Hm8dY5KDfkyEPM4x2EhSna98zYmKGMuofjiNns3t8pkF53lboEjhrZpCk8OayDfFHOXyxBUoLwoNdU9Jbi+jV3fcotHm0nYg3aFnJyyK/OHWdfBWatdnkinFEyRs+wAX7SEDC9VqED5GMzdxiX1zQ94QL8w74E9ZenDDr+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8P0yqxz4f3m75; Wed, 28 Feb 2024 22:40:37 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id 75E491A0232; Wed, 28 Feb 2024 22:40:44 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S9; Wed, 28 Feb 2024 22:40:44 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 5/6] virtiofs: use scattered bounce buffer for ITER_KVEC dio Date: Wed, 28 Feb 2024 22:41:25 +0800 Message-Id: <20240228144126.2864064-6-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S9 X-Coremail-Antispam: 1UD129KBjvJXoWxKF4UZry3tr17ur43Gr4kJFb_yoW7ArW5pr 47t3W5XFWftFZrWryfKw45GFySyrsaka18KrWfJ3sxKF17Z39rXFy0ya45ZFsIqrykAF4x uF1FqF1DWF48ZaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBIb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY 6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aV CY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU13l1DUUUUU== X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao To prevent unnecessary request for large contiguous physical memory chunk, use bounce buffer backed by scattered pages for ITER_KVEC direct-io read/write when the total size of its args is greater than PAGE_SIZE. Signed-off-by: Hou Tao --- fs/fuse/virtio_fs.c | 78 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 19 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index ffea684bd100d..34b9370beba6d 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -458,20 +458,15 @@ static void virtio_fs_argbuf_free(struct virtio_fs_argbuf *argbuf) kfree(argbuf); } -static struct virtio_fs_argbuf *virtio_fs_argbuf_new(struct fuse_args *args, +static struct virtio_fs_argbuf *virtio_fs_argbuf_new(unsigned int in_len, + unsigned int out_len, gfp_t gfp, bool is_flat) { struct virtio_fs_argbuf *argbuf; - unsigned int numargs; - unsigned int in_len, out_len, len; + unsigned int len; unsigned int i, nr; - numargs = args->in_numargs - args->in_pages; - in_len = fuse_len_args(numargs, (struct fuse_arg *) args->in_args); - numargs = args->out_numargs - args->out_pages; - out_len = fuse_len_args(numargs, args->out_args); len = virtio_fs_argbuf_len(in_len, out_len, is_flat); - if (is_flat) { argbuf = kmalloc(struct_size(argbuf, f.buf, len), gfp); if (argbuf) @@ -1222,14 +1217,17 @@ static unsigned int sg_count_fuse_pages(struct fuse_page_desc *page_descs, } /* Return the number of scatter-gather list elements required */ -static unsigned int sg_count_fuse_req(struct fuse_req *req) +static unsigned int sg_count_fuse_req(struct fuse_req *req, + unsigned int in_args_len, + unsigned int out_args_len, + bool flat_argbuf) { struct fuse_args *args = req->args; struct fuse_args_pages *ap = container_of(args, typeof(*ap), args); unsigned int size, total_sgs = 1 /* fuse_in_header */; + unsigned int num_in, num_out; - if (args->in_numargs - args->in_pages) - total_sgs += 1; + num_in = args->in_numargs - args->in_pages; if (args->in_pages) { size = args->in_args[args->in_numargs - 1].size; @@ -1237,20 +1235,25 @@ static unsigned int sg_count_fuse_req(struct fuse_req *req) size); } - if (!test_bit(FR_ISREPLY, &req->flags)) - return total_sgs; + if (!test_bit(FR_ISREPLY, &req->flags)) { + num_out = 0; + goto done; + } total_sgs += 1 /* fuse_out_header */; - - if (args->out_numargs - args->out_pages) - total_sgs += 1; + num_out = args->out_numargs - args->out_pages; if (args->out_pages) { size = args->out_args[args->out_numargs - 1].size; total_sgs += sg_count_fuse_pages(ap->descs, ap->num_pages, size); } - +done: + if (flat_argbuf) + total_sgs += !!num_in + !!num_out; + else + total_sgs += virtio_fs_argbuf_len(in_args_len, out_args_len, + false) >> PAGE_SHIFT; return total_sgs; } @@ -1302,6 +1305,31 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg, return total_sgs; } +static bool use_scattered_argbuf(struct fuse_req *req) +{ + struct fuse_args *args = req->args; + + /* + * To prevent unnecessary request for contiguous physical memory chunk, + * use argbuf backed by scattered pages for ITER_KVEC direct-io + * read/write when the total size of its args is greater than PAGE_SIZE. + */ + if ((req->in.h.opcode == FUSE_WRITE && !args->in_pages) || + (req->in.h.opcode == FUSE_READ && !args->out_pages)) { + unsigned int numargs; + unsigned int len; + + numargs = args->in_numargs - args->in_pages; + len = fuse_len_args(numargs, (struct fuse_arg *)args->in_args); + numargs = args->out_numargs - args->out_pages; + len += fuse_len_args(numargs, args->out_args); + if (len > PAGE_SIZE) + return true; + } + + return false; +} + /* Add a request to a virtqueue and kick the device */ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, struct fuse_req *req, bool in_flight) @@ -1317,13 +1345,24 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, unsigned int out_sgs = 0; unsigned int in_sgs = 0; unsigned int total_sgs; + unsigned int numargs; + unsigned int in_args_len; + unsigned int out_args_len; unsigned int i; int ret; bool notify; + bool flat_argbuf; struct fuse_pqueue *fpq; + flat_argbuf = !use_scattered_argbuf(req); + numargs = args->in_numargs - args->in_pages; + in_args_len = fuse_len_args(numargs, (struct fuse_arg *) args->in_args); + numargs = args->out_numargs - args->out_pages; + out_args_len = fuse_len_args(numargs, args->out_args); + /* Does the sglist fit on the stack? */ - total_sgs = sg_count_fuse_req(req); + total_sgs = sg_count_fuse_req(req, in_args_len, out_args_len, + flat_argbuf); if (total_sgs > ARRAY_SIZE(stack_sgs)) { sgs = kmalloc_array(total_sgs, sizeof(sgs[0]), GFP_ATOMIC); sg = kmalloc_array(total_sgs, sizeof(sg[0]), GFP_ATOMIC); @@ -1334,7 +1373,8 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, } /* Use a bounce buffer since stack args cannot be mapped */ - req->argbuf = virtio_fs_argbuf_new(args, GFP_ATOMIC, true); + req->argbuf = virtio_fs_argbuf_new(in_args_len, out_args_len, + GFP_ATOMIC, flat_argbuf); if (!req->argbuf) { ret = -ENOMEM; goto out; From patchwork Wed Feb 28 14:41:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hou Tao X-Patchwork-Id: 13575628 Received: from dggsgout11.his.huawei.com (unknown [45.249.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29A4015B0E4; Wed, 28 Feb 2024 14:40:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131250; cv=none; b=aLDzB3sU0YeryUXQJCZhMg5/W3n/yPaYoXW6SnkKlHGA/vw5Lkx78DrUrx3p7kbCmL/42PxIfsCkO+WzQr2i2zgm9Re9qf72fSfnDOOHVTRXsJJc4kdKgq5zVuw2/QaF0WiLmqwmrbphUSx6ZiW4p05DaaYOTfEa0Q8ABLhASis= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709131250; c=relaxed/simple; bh=MfpUPQ3tl/f5UmupFztvkyZZv7LY1g0CBIcSjcfAuK0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pw3F1Wq8krBNJiwkr3jQpH/96JP6G46jINSsV1lqL8bdqzx1yOF/RZyRp05pNG4X3IK9mA8FOIlpM/noTnt16lU8LeJiiG+F765QH6inE2zE6lgHjrmpl5klg7mZJn4RSSBiF+K45lzlKphhyzVSrgdIA63MV2jBmHv70t2ia0Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=45.249.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4TlH8P4BdYz4f3m7B; Wed, 28 Feb 2024 22:40:37 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.112]) by mail.maildlp.com (Postfix) with ESMTP id E54671A0DCA; Wed, 28 Feb 2024 22:40:44 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.124.27]) by APP1 (Coremail) with SMTP id cCh0CgBnOBHkRd9lwGKzFQ--.18779S10; Wed, 28 Feb 2024 22:40:44 +0800 (CST) From: Hou Tao To: linux-fsdevel@vger.kernel.org Cc: Miklos Szeredi , Vivek Goyal , Stefan Hajnoczi , Bernd Schubert , "Michael S . Tsirkin" , Matthew Wilcox , Benjamin Coddington , linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, houtao1@huawei.com Subject: [PATCH v2 6/6] virtiofs: use GFP_NOFS when enqueuing request through kworker Date: Wed, 28 Feb 2024 22:41:26 +0800 Message-Id: <20240228144126.2864064-7-houtao@huaweicloud.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20240228144126.2864064-1-houtao@huaweicloud.com> References: <20240228144126.2864064-1-houtao@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: cCh0CgBnOBHkRd9lwGKzFQ--.18779S10 X-Coremail-Antispam: 1UD129KBjvJXoWxGF4kXrWUAr1rGryrAF1rXrb_yoW5tF4xpr WkAa15GFZ5JrW2gFWkKF4UCw4Ykw1kCrW7G34fX3sIkr4jqw47uFyUZFy0qFsavrykAF1x WF4FqF4DuFsrZw7anT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBIb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E 14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7 xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Y z7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY 6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aV CY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU13l1DUUUUU== X-CM-SenderInfo: xkrx3t3r6k3tpzhluzxrxghudrp/ From: Hou Tao When invoking virtio_fs_enqueue_req() through kworker, both the allocation of the sg array and the bounce buffer still use GFP_ATOMIC. Considering the size of the sg array may be greater than PAGE_SIZE, use GFP_NOFS instead of GFP_ATOMIC to lower the possibility of memory allocation failure and to avoid unnecessarily depleting the atomic reserves. GFP_NOFS is not passed to virtio_fs_enqueue_req() directly, use GFP_KERNEL and memalloc_nofs_{save|restore} helpers instead. Signed-off-by: Hou Tao --- fs/fuse/virtio_fs.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index 34b9370beba6d..9ee71051c89f2 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -108,7 +108,8 @@ struct virtio_fs_argbuf { }; static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, - struct fuse_req *req, bool in_flight); + struct fuse_req *req, bool in_flight, + gfp_t gfp); static const struct constant_table dax_param_enums[] = { {"always", FUSE_DAX_ALWAYS }, @@ -394,6 +395,8 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work) /* Dispatch pending requests */ while (1) { + unsigned int flags; + spin_lock(&fsvq->lock); req = list_first_entry_or_null(&fsvq->queued_reqs, struct fuse_req, list); @@ -404,7 +407,9 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work) list_del_init(&req->list); spin_unlock(&fsvq->lock); - ret = virtio_fs_enqueue_req(fsvq, req, true); + flags = memalloc_nofs_save(); + ret = virtio_fs_enqueue_req(fsvq, req, true, GFP_KERNEL); + memalloc_nofs_restore(flags); if (ret < 0) { if (ret == -ENOMEM || ret == -ENOSPC) { spin_lock(&fsvq->lock); @@ -1332,7 +1337,8 @@ static bool use_scattered_argbuf(struct fuse_req *req) /* Add a request to a virtqueue and kick the device */ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, - struct fuse_req *req, bool in_flight) + struct fuse_req *req, bool in_flight, + gfp_t gfp) { /* requests need at least 4 elements */ struct scatterlist *stack_sgs[6]; @@ -1364,8 +1370,8 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, total_sgs = sg_count_fuse_req(req, in_args_len, out_args_len, flat_argbuf); if (total_sgs > ARRAY_SIZE(stack_sgs)) { - sgs = kmalloc_array(total_sgs, sizeof(sgs[0]), GFP_ATOMIC); - sg = kmalloc_array(total_sgs, sizeof(sg[0]), GFP_ATOMIC); + sgs = kmalloc_array(total_sgs, sizeof(sgs[0]), gfp); + sg = kmalloc_array(total_sgs, sizeof(sg[0]), gfp); if (!sgs || !sg) { ret = -ENOMEM; goto out; @@ -1373,8 +1379,8 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq, } /* Use a bounce buffer since stack args cannot be mapped */ - req->argbuf = virtio_fs_argbuf_new(in_args_len, out_args_len, - GFP_ATOMIC, flat_argbuf); + req->argbuf = virtio_fs_argbuf_new(in_args_len, out_args_len, gfp, + flat_argbuf); if (!req->argbuf) { ret = -ENOMEM; goto out; @@ -1473,7 +1479,7 @@ __releases(fiq->lock) fuse_len_args(req->args->out_numargs, req->args->out_args)); fsvq = &fs->vqs[queue_id]; - ret = virtio_fs_enqueue_req(fsvq, req, false); + ret = virtio_fs_enqueue_req(fsvq, req, false, GFP_ATOMIC); if (ret < 0) { if (ret == -ENOMEM || ret == -ENOSPC) { /*