From patchwork Tue Mar 8 15:20:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD8B5C4332F for ; Tue, 8 Mar 2022 15:42:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347901AbiCHPm4 (ORCPT ); Tue, 8 Mar 2022 10:42:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347967AbiCHPmy (ORCPT ); Tue, 8 Mar 2022 10:42:54 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF1254ECDD for ; Tue, 8 Mar 2022 07:41:56 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154153epoutp03185f7dd25c99c8fade87f493e532278c~aciQl9yv92356523565epoutp03k for ; Tue, 8 Mar 2022 15:41:53 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154153epoutp03185f7dd25c99c8fade87f493e532278c~aciQl9yv92356523565epoutp03k DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754113; bh=ZieorIZ+j5h8EI+90pdzpAhBaEPS1ZjtIJkbUiQYYd8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Kz/OH60pw9rSaW8HcqQOiT1KrsUaXL6zSecZc/aDABFXO51GJF5TyVlC2TQcUo8WO WhVKpMmEmIJ8exh8Bopxt7qRvJdUO6Eeb1I6NH8vaJRLjmD/PP3y9kxmpWipIhQicQ ADzqDPUxgtOX2ovR6ho83SZnsJbBEnmX2/fm+cho= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20220308154152epcas5p21f1563d14c5155be4fdc7dd755c235ed~aciP6dkU42884128841epcas5p2M; Tue, 8 Mar 2022 15:41:52 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfhF0vthz4x9Pv; Tue, 8 Mar 2022 15:41:49 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 13.AB.05590.C3977226; Wed, 9 Mar 2022 00:41:48 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152653epcas5p10c31f58cf6bff125cc0baa176b4d4fac~acVKklf6q1816518165epcas5p1Y; Tue, 8 Mar 2022 15:26:53 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152653epsmtrp1fe07683fdc8270ace03a527428d4b61e~acVKjfmC00125001250epsmtrp1L; Tue, 8 Mar 2022 15:26:53 +0000 (GMT) X-AuditID: b6c32a4b-723ff700000015d6-0b-6227793cd86c Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 13.51.29871.DB577226; Wed, 9 Mar 2022 00:26:53 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152651epsmtip16d484f8aaaa4d3e5114ee5cf029c8893~acVIgR9hJ3168431684epsmtip19; Tue, 8 Mar 2022 15:26:51 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 01/17] io_uring: add support for 128-byte SQEs Date: Tue, 8 Mar 2022 20:50:49 +0530 Message-Id: <20220308152105.309618-2-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNJsWRmVeSWpSXmKPExsWy7bCmlq5tpXqSwVQui+mHFS2aJvxltpiz ahujxeq7/WwWK1cfZbJ413qOxaLz9AUmi/NvDzNZTDp0jdFi7y1ti/nLnrJbLGk9zmZxY8JT Ros1N5+yWHw+M4/Vgd/j2dVnjB47Z91l92hecIfF4/LZUo9NqzrZPDYvqffYfbOBzWPb4pes Hn1bVjF6fN4kF8AVlW2TkZqYklqkkJqXnJ+SmZduq+QdHO8cb2pmYKhraGlhrqSQl5ibaqvk 4hOg65aZA/SCkkJZYk4pUCggsbhYSd/Opii/tCRVISO/uMRWKbUgJafApECvODG3uDQvXS8v tcTK0MDAyBSoMCE74/2dfywFMyUqulZ2sTQwXhTuYuTgkBAwkbj7I7eLkYtDSGA3o8TZLZdZ IZxPjBIfbu9n7mLkBHK+MUpM210EYoM0XGxtZYco2sso8b5hARuE85lRYsqvv+wgY9kENCUu TC4FaRAR8JK4f/s92FRmgS4mibf77rOBJIQF7CRmfT7BBGKzCKhKtC/6ywzSyytgKdH4vxBi mbzEzEvf2UFsTgEriZ+3trKC2LwCghInZz5hAbGZgWqat85mBpkvIXCEQ2LToX42iGYXiYc7 V7JA2MISr45vYYewpSRe9rdB2cUSv+4chWruYJS43jATqsFe4uKev0wgBzEDPbN+lz5EWFZi 6ql1TBCL+SR6fz9hgojzSuyYB2MrStyb9JQVwhaXeDhjCZTtIbGx7yM0eHsZJR79u8M+gVFh FpKHZiF5aBbC6gWMzKsYJVMLinPTU4tNC4zzUsvhkZycn7uJEZyqtbx3MD568EHvECMTB+Mh RgkOZiUR3vvnVZKEeFMSK6tSi/Lji0pzUosPMZoCA3wis5Rocj4wW+SVxBuaWBqYmJmZmVga mxkqifOeSt+QKCSQnliSmp2aWpBaBNPHxMEp1cD0MHDLs9Nioit6Pt/6vrQx7920vS8K6laG vH42+ZeXf+z8g7WLVspvnc7/J/C576VpD10/qhW+sC/YtvHJg4pdFcenLJE+f2SqrNdHhq67 N91ev3r+R3V6des/b5UsjdNpQkpu7ltL+47VSogpNmU2TI/+H/6V3zDh8jueTRyhj/ddX373 S3THxuz49l+nZwZfCHjY6r/7q97ZXLaDeUxHVgi2mjNveuLjM3cz2xuzNR0NU8qFj3XzGDVa +9QwP7nJInPWOiz71cpp4iXZW93e/bg+x4d3RcSdF/dfzJ2woIrn7Tkhhaa7d66JHU7JSg2/ /Kv/QJjtou011rfnd3Xt4FhakvlnepTNhhvq/jWxc5RYijMSDbWYi4oTAXX1s9JeBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTndvqXqSwa/74hbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr4/2dfywFMyUqulZ2sTQwXhTuYuTkkBAwkbjY 2soOYgsJ7GaU6FiRBxEXl2i+9oMdwhaWWPnvOZDNBVTzkVHi3aJLbF2MHBxsApoSFyaXgtSI CARIHGy8DFbDLDCDSaKn+TMLSEJYwE5i1ucTTCA2i4CqRPuiv8wgvbwClhKN/wsh5stLzLz0 HWwXp4CVxM9bW1kh7rGUWLHuNxuIzSsgKHFy5hOwkcxA9c1bZzNPYBSYhSQ1C0lqASPTKkbJ 1ILi3PTcYsMCw7zUcr3ixNzi0rx0veT83E2M4GjS0tzBuH3VB71DjEwcjIcYJTiYlUR4759X SRLiTUmsrEotyo8vKs1JLT7EKM3BoiTOe6HrZLyQQHpiSWp2ampBahFMlomDU6qBSeKUWarr ErZUxrCzEc/t864c+vnE9JjV8xDhHPMddhcmb0zujdEV74tr2+F7oL42suDhtik3D87eIdfa nr1eYL2BJfdZhgSrXwZFuQp7GLVFzhb8SzkWtmxT190ZR7K0rNRPzvOcb7hB44QN9+tVCgc+ HHzLc0/9SL38chHOQ2ZK2358WS6259LGYs5X5xw377e4pj3jgeGOPVyra5bk1DhuUYve+Naw m7VX5fuWHUf3JzDuFPAIfv359buZgicXuxQHNsYvyOdXTfjM4eKqF3u45nn5kcmx7Rqq7II/ 1hZ4ljKvWGN0/t2LjwefSkx7URS3d5Gk2xPzOeVh4fqijV6WJ46Fe3a+e/XjHu8iByWW4oxE Qy3mouJEAD3/bYoVAwAA X-CMS-MailID: 20220308152653epcas5p10c31f58cf6bff125cc0baa176b4d4fac X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152653epcas5p10c31f58cf6bff125cc0baa176b4d4fac References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Jens Axboe Normal SQEs are 64-bytes in length, which is fine for all the commands we support. However, in preparation for supporting passthrough IO, provide an option for setting up a ring with 128-byte SQEs. We continue to use the same type for io_uring_sqe, it's marked and commented with a zero sized array pad at the end. This provides up to 80 bytes of data for a passthrough command - 64 bytes for the extra added data, and 16 bytes available at the end of the existing SQE. Signed-off-by: Jens Axboe --- fs/io_uring.c | 13 ++++++++++--- include/uapi/linux/io_uring.h | 7 +++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index a7412f6862fc..241ba1cd6dcf 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -7431,8 +7431,12 @@ static const struct io_uring_sqe *io_get_sqe(struct io_ring_ctx *ctx) * though the application is the one updating it. */ head = READ_ONCE(ctx->sq_array[sq_idx]); - if (likely(head < ctx->sq_entries)) + if (likely(head < ctx->sq_entries)) { + /* double index for 128-byte SQEs, twice as long */ + if (ctx->flags & IORING_SETUP_SQE128) + head <<= 1; return &ctx->sq_sqes[head]; + } /* drop invalid entries */ ctx->cq_extra--; @@ -10431,7 +10435,10 @@ static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, rings->sq_ring_entries = p->sq_entries; rings->cq_ring_entries = p->cq_entries; - size = array_size(sizeof(struct io_uring_sqe), p->sq_entries); + if (p->flags & IORING_SETUP_SQE128) + size = array_size(2 * sizeof(struct io_uring_sqe), p->sq_entries); + else + size = array_size(sizeof(struct io_uring_sqe), p->sq_entries); if (size == SIZE_MAX) { io_mem_free(ctx->rings); ctx->rings = NULL; @@ -10643,7 +10650,7 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) if (p.flags & ~(IORING_SETUP_IOPOLL | IORING_SETUP_SQPOLL | IORING_SETUP_SQ_AFF | IORING_SETUP_CQSIZE | IORING_SETUP_CLAMP | IORING_SETUP_ATTACH_WQ | - IORING_SETUP_R_DISABLED)) + IORING_SETUP_R_DISABLED | IORING_SETUP_SQE128)) return -EINVAL; return io_uring_create(entries, &p, params); diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 787f491f0d2a..c5db68433ca5 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -61,6 +61,12 @@ struct io_uring_sqe { __u32 file_index; }; __u64 __pad2[2]; + + /* + * If the ring is initializefd with IORING_SETUP_SQE128, then this field + * contains 64-bytes of padding, doubling the size of the SQE. + */ + __u64 __big_sqe_pad[0]; }; enum { @@ -101,6 +107,7 @@ enum { #define IORING_SETUP_CLAMP (1U << 4) /* clamp SQ/CQ ring sizes */ #define IORING_SETUP_ATTACH_WQ (1U << 5) /* attach to existing wq */ #define IORING_SETUP_R_DISABLED (1U << 6) /* start with ring disabled */ +#define IORING_SETUP_SQE128 (1U << 7) /* SQEs are 128b */ enum { IORING_OP_NOP, From patchwork Tue Mar 8 15:20:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16DC8C433F5 for ; Tue, 8 Mar 2022 15:42:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347912AbiCHPnB (ORCPT ); Tue, 8 Mar 2022 10:43:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347917AbiCHPm4 (ORCPT ); Tue, 8 Mar 2022 10:42:56 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F0DC4ECFE for ; Tue, 8 Mar 2022 07:42:00 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154158epoutp03ab6d676ae4858d2fab229b8f4bcb75c8~aciVbhCF02772027720epoutp03N for ; Tue, 8 Mar 2022 15:41:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154158epoutp03ab6d676ae4858d2fab229b8f4bcb75c8~aciVbhCF02772027720epoutp03N DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754118; bh=eQY981qQJsial6Teqout4wQZLQ5cka3Wr/hRVUCufOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VoBjpVsjBBRNJs3Qn2CLiaUfinCSDtsJE05MyGLUpc3OVjxZv/m1nbv5cSFjYrpE4 wMehDn1GR6MczyXCcoeDxxhcGvQlnPyEZ5UbhtJxt8hKg0bx8rJGeQlLw7bdN5uO3U /I64vA0B6xaJfzhLwnfNf5UgX4cqpjzChaSg0rSU= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20220308154157epcas5p29c6de2e29e8cdd6bbe0d9ce3231d80bb~aciUhE2X42884128841epcas5p2R; Tue, 8 Mar 2022 15:41:57 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.177]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4KCfhL0vJwz4x9Pp; Tue, 8 Mar 2022 15:41:54 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 01.E6.46822.A6677226; Wed, 9 Mar 2022 00:29:46 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20220308152655epcas5p4ae47d715e1c15069e97152dcd283fd40~acVMtmpUO1273712737epcas5p4d; Tue, 8 Mar 2022 15:26:55 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152655epsmtrp13252b6f06a1190aaa59bc4780bbeae64~acVMsu3R20125001250epsmtrp1N; Tue, 8 Mar 2022 15:26:55 +0000 (GMT) X-AuditID: b6c32a4a-de5ff7000000b6e6-f6-6227766ab17d Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 14.51.29871.FB577226; Wed, 9 Mar 2022 00:26:55 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152653epsmtip1170e3b67827d9ab191f908af9edbf8ad~acVKoUDEa1072310723epsmtip1o; Tue, 8 Mar 2022 15:26:53 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 02/17] fs: add file_operations->async_cmd() Date: Tue, 8 Mar 2022 20:50:50 +0530 Message-Id: <20220308152105.309618-3-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDJsWRmVeSWpSXmKPExsWy7bCmpm5WmXqSwexXrBbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsbv5kOMBZPYK+Y2/2JpYHzC2sXIySEhYCLxoO8vUxcjF4eQwG5GiZs9 +9ggnE+MEvcuzGeFcL4xSpz/NYUZpuXZocdgtpDAXkaJPUeyIIo+M0r8WfMXKMHBwSagKXFh cilIjYiAl8T92+/BBjELdDFJvN13nw0kISxgLbF0+iwWEJtFQFVi+53pYEN5BSwlujZNZYNY Ji8x89J3dhCbU8BK4uetrawQNYISJ2c+AetlBqpp3jqbGWSBhMABDonuOd+YQI6QEHCROL1T EGKOsMSr41vYIWwpiZf9bVB2scSvO0ehejsYJa43zGSBSNhLXNzzF2wOM9Az63fpQ4RlJaae WscEsZdPovf3EyaIOK/EjnkwtqLEvUlPoeErLvFwxhIo20Oi88c9NkjA9TJK9O+TnsCoMAvJ O7OQvDMLYfMCRuZVjJKpBcW56anFpgVGeanl8EhOzs/dxAhO1lpeOxgfPvigd4iRiYPxEKME B7OSCO/98ypJQrwpiZVVqUX58UWlOanFhxhNgeE9kVlKNDkfmC/ySuINTSwNTMzMzEwsjc0M lcR5T6dvSBQSSE8sSc1OTS1ILYLpY+LglGpgst574HNvd/lXtx0Fd1c8qGYJYki1vlq1/y/3 9PtPn/7bdGhGlObll9sKZQ+3dZ4WrhOw+Rbqvznx88L2zc974h5kBBhKCUhZhHxzZIz5x1a8 YILM36k+0pJ+nAvkOjrOHhN7m/JxRlTIFabjJyZ+Pa10v1OzKv7X07W9n6weJs96VPo/0/j6 y2KzWWrtGTLvsn60+/3bHxJ5y8yJ10TQeWdaYoyOC7/5TcHkn7d4L09jtnT7v3fnv7ZCe5sr tep/fFszny3r7BRRdnO2WLTAPkvywg02//VaIV0cb9uqs7xmrtkVFxnmFcu6KPHz3COiHzY4 tcpnB8huZXjXVPfS7uKalBX//1ZND2dbyPNeiaU4I9FQi7moOBEAOTRKRl8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmkeLIzCtJLcpLzFFi42LZdlhJTnd/qXqSwbkrGhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr43fzIcaCSewVc5t/sTQwPmHtYuTkkBAwkXh2 6DEziC0ksJtRYu2/AIi4uETztR/sELawxMp/z4FsLqCaj4wSl3umADVwcLAJaEpcmFwKUiMi ECBxsPEyWA2zwAwmiZ7mzywgCWEBa4ml02eB2SwCqhLb70wHW8YrYCnRtWkqG8QCeYmZl76D LeMUsJL4eWsrK8RBlhIr1v1mg6gXlDg58wnYHGag+uats5knMArMQpKahSS1gJFpFaNkakFx bnpusWGBYV5quV5xYm5xaV66XnJ+7iZGcDxpae5g3L7qg94hRiYOxkOMEhzMSiK898+rJAnx piRWVqUW5ccXleakFh9ilOZgURLnvdB1Ml5IID2xJDU7NbUgtQgmy8TBKdXA1PT58kQpURXJ PYbX7ZxKJ8//URD1VLV69iTLMJm0ojvOfbuv3t/722H91H+zFe9fPcz2U6jVzMuW68Hv+HUn Yo41TIv8v6Va26JK36Y4v5mxWWxln9gdMcMr+8+uf5lS8GSiw663BysityyXV9bS4T4+ybcz M+uesp2XY/HCg94axruX6Yqzme9Kkd3TcviG2N18rYhTuSsZurj5zW/ObDyzanFeKIszx2TR SWqmlk932x3gdtq0tpHpzbuPk54IF1xoKln+Rem4YsqZH7cE8iM+ND9gVd585HWrFK9giN/M ewtNqqOSjI816epN7/gmsLLwo37plpDdG+8z3G7m/v3yuFVa1bbdqUHH9m24rMRSnJFoqMVc VJwIAHF55LcWAwAA X-CMS-MailID: 20220308152655epcas5p4ae47d715e1c15069e97152dcd283fd40 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152655epcas5p4ae47d715e1c15069e97152dcd283fd40 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Jens Axboe This is a file private handler, similar to ioctls but hopefully a lot more sane and useful. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index e2d892b201b0..a32f83b70435 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1977,6 +1977,7 @@ struct dir_context { #define REMAP_FILE_ADVISORY (REMAP_FILE_CAN_SHORTEN) struct iov_iter; +struct io_uring_cmd; struct file_operations { struct module *owner; @@ -2019,6 +2020,7 @@ struct file_operations { struct file *file_out, loff_t pos_out, loff_t len, unsigned int remap_flags); int (*fadvise)(struct file *, loff_t, loff_t, int); + int (*async_cmd)(struct io_uring_cmd *ioucmd); } __randomize_layout; struct inode_operations { From patchwork Tue Mar 8 15:20:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE660C433FE for ; Tue, 8 Mar 2022 15:42:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347914AbiCHPnJ (ORCPT ); Tue, 8 Mar 2022 10:43:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347917AbiCHPnI (ORCPT ); Tue, 8 Mar 2022 10:43:08 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 968B24F441 for ; Tue, 8 Mar 2022 07:42:09 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154205epoutp0372315b20dc10b760d875b19cbd77c09c~acibqELmZ2772027720epoutp03R for ; Tue, 8 Mar 2022 15:42:05 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154205epoutp0372315b20dc10b760d875b19cbd77c09c~acibqELmZ2772027720epoutp03R DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754125; bh=ExpJtTiwCEF1LtD2DrUQNrZudMvTlavOkrHKvYLVMg4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Svn6CU/IwSTCQ9wo+39vkqcgWhTeYgCNxl7oGYgW+fYN/hgd3fGzW8lIwBku/om7G 9wYZRvn/J4PnrkKu8SKyrgnPnYNcAAx9+AFwNrEuzwyPxcYZ2C7taFknnpQpylWRVV Bgeh73tLRLlVJkpHAvVPsiWoanbv+DQWOUj8z13w= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220308154204epcas5p4a21a4771dc19f3d964bcd2881b176b33~acia1ua-V2070420704epcas5p4Y; Tue, 8 Mar 2022 15:42:04 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfhS5Zw7z4x9Pq; Tue, 8 Mar 2022 15:42:00 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 41.57.06423.84977226; Wed, 9 Mar 2022 00:42:00 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152658epcas5p3929bd1fcf75edc505fec71901158d1b5~acVOyUPPw0623006230epcas5p3S; Tue, 8 Mar 2022 15:26:58 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152658epsmtrp261e74e158fcc86a46729821df02b8a4a~acVOxXOXA2706527065epsmtrp25; Tue, 8 Mar 2022 15:26:58 +0000 (GMT) X-AuditID: b6c32a49-b01ff70000001917-cb-62277948a196 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 24.96.03370.1C577226; Wed, 9 Mar 2022 00:26:57 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152655epsmtip1327c3fde86c87962eb4b3f60922231fe~acVMwW9Uv3168431684epsmtip1_; Tue, 8 Mar 2022 15:26:55 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 03/17] io_uring: add infra and support for IORING_OP_URING_CMD Date: Tue, 8 Mar 2022 20:50:51 +0530 Message-Id: <20220308152105.309618-4-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHJsWRmVeSWpSXmKPExsWy7bCmhq5HpXqSwYGdWhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsav1kXsBacsK3Yd3MfewHhLr4uRk0NCwETix+fzLF2MXBxCArsZJZ7t mA/lfGKU+Lt9ESOE85lR4s22zYwwLX9f3GIHsYUEdjFKNG+SgCt6tuQRWxcjBwebgKbEhcml IDUiAl4S92+/ZwWpYRboYpJ4u+8+G0hCWCBAorHjHTOIzSKgKtGx7Q/YUF4BS4nnU7ewQiyT l5h56TtYnFPASuLnra2sEDWCEidnPmEBsZmBapq3zmYGWSAhcIRD4sW+/WwQzS4Sa/oWMEHY whKvjm9hh7ClJD6/2wtVUyzx685RqOYORonrDTNZIBL2Ehf3/GUC+YYZ6Jv1u/QhwrISU0+t Y4JYzCfR+/sJ1HxeiR3zYGxFiXuTnkI9IC7xcMYSKNtD4u1dkBpQaPUySnz4tY99AqPCLCQP zULy0CyE1QsYmVcxSqYWFOempxabFhjmpZbDozk5P3cTIzhha3nuYLz74IPeIUYmDsZDjBIc zEoivPfPqyQJ8aYkVlalFuXHF5XmpBYfYjQFhvhEZinR5HxgzsgriTc0sTQwMTMzM7E0NjNU Euc9nb4hUUggPbEkNTs1tSC1CKaPiYNTqoFJoe3NlBLd670880xLj1Tpuq7yeP41ukf73ceL DKLzH5q3cB+ts2X//tOUpTW/X+TKPpmQLd96vS6z9cezLq+Yn/by5sF9IqrrmASDnThPzy1V i3BY8HeBZlHZvv6zGTvcirr/vniRvzzL8+KWdcvP5vbLljsWqv1L6FBymTlhkS+r2I2a5z9U H9q5bRfJW5D7ck1ao9vd+IN8345ba8XKHsydK3ZJeI94l/k8E91ah8TDr1fv/HvAweHEQxln iZolq1JnyM6P2qO1VDtNSWw65735C5g90jyFFnKz+tzWDxTffLm84pRXYbOyYozvD66jkY6v 1nG7L8oKnNpbbLFq0osN81cF8fc5Vl/+yqDEUpyRaKjFXFScCAA3WQZaYQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphkeLIzCtJLcpLzFFi42LZdlhJTvdgqXqSQX+PlcX0w4oWTRP+MlvM WbWN0WL13X42i5WrjzJZvGs9x2LRefoCk8X5t4eZLCYdusZosfeWtsX8ZU/ZLZa0HmezuDHh KaPFmptPWSw+n5nH6sDv8ezqM0aPnbPusns0L7jD4nH5bKnHplWdbB6bl9R77L7ZwOaxbfFL Vo++LasYPT5vkgvgiuKySUnNySxLLdK3S+DK+NW6iL3glGXFroP72BsYb+l1MXJySAiYSPx9 cYu9i5GLQ0hgB6PEtZN7WCAS4hLN136wQ9jCEiv/PYcq+sgoMWfZXSCHg4NNQFPiwuRSkBoR gQCJg42XwWqYBWYwSfQ0fwYbJCzgJzG56yYziM0ioCrRse0P2FBeAUuJ51O3sEIskJeYeek7 WJxTwEri562tYHEhoJoV636zQdQLSpyc+QRsJjNQffPW2cwTGAVmIUnNQpJawMi0ilEytaA4 Nz232LDAKC+1XK84Mbe4NC9dLzk/dxMjOKK0tHYw7ln1Qe8QIxMH4yFGCQ5mJRHe++dVkoR4 UxIrq1KL8uOLSnNSiw8xSnOwKInzXug6GS8kkJ5YkpqdmlqQWgSTZeLglGpg2vWCP8hoVnXg RXsxw7SNK09KeIrxFmX+/PXlqtePrVXB+w+e998uNCNG+6/G/3N3+SZ9eCvJ9nrqJfn/G3bw J7Kq6Iv/SzCLWFr05caa/NtSM1/obXisKZRr++pX04VzWZXmLBOdQ2OOWcdUl+3byqdyJ2N/ l8Pl56pbU+5OZrqnJM7qn7hUKGbp70VJx0rWMkRn3HGV1Tv+tdzwksib0oKIY9M3Ht4yZdOi kNCVGlGJv+p2KF2qiNjhdk9n9pFrdasyrr39vsZIjmOhyRVN/0mmq2XXsT14+npRp43nzUWe krK6nRWhzk92zvylUnHxhfyllNQ9u64szpQS0dyy+Oy7jy++71bh/LdkdtWiLi8lluKMREMt 5qLiRAB6qByWFwMAAA== X-CMS-MailID: 20220308152658epcas5p3929bd1fcf75edc505fec71901158d1b5 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152658epcas5p3929bd1fcf75edc505fec71901158d1b5 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Jens Axboe This is a file private kind of request. io_uring doesn't know what's in this command type, it's for the file_operations->async_cmd() handler to deal with. Signed-off-by: Jens Axboe Signed-off-by: Kanchan Joshi Signed-off-by: Luis Chamberlain --- fs/io_uring.c | 79 +++++++++++++++++++++++++++++++---- include/linux/io_uring.h | 29 +++++++++++++ include/uapi/linux/io_uring.h | 9 +++- 3 files changed, 108 insertions(+), 9 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 241ba1cd6dcf..1f228a79e68f 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -200,13 +200,6 @@ struct io_rings { struct io_uring_cqe cqes[] ____cacheline_aligned_in_smp; }; -enum io_uring_cmd_flags { - IO_URING_F_COMPLETE_DEFER = 1, - IO_URING_F_UNLOCKED = 2, - /* int's last bit, sign checks are usually faster than a bit test */ - IO_URING_F_NONBLOCK = INT_MIN, -}; - struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; @@ -860,6 +853,7 @@ struct io_kiocb { struct io_mkdir mkdir; struct io_symlink symlink; struct io_hardlink hardlink; + struct io_uring_cmd uring_cmd; }; u8 opcode; @@ -1110,6 +1104,9 @@ static const struct io_op_def io_op_defs[] = { [IORING_OP_MKDIRAT] = {}, [IORING_OP_SYMLINKAT] = {}, [IORING_OP_LINKAT] = {}, + [IORING_OP_URING_CMD] = { + .needs_file = 1, + }, }; /* requests with any of those set should undergo io_disarm_next() */ @@ -2464,6 +2461,22 @@ static void io_req_task_submit(struct io_kiocb *req, bool *locked) io_req_complete_failed(req, -EFAULT); } +static void io_uring_cmd_work(struct io_kiocb *req, bool *locked) +{ + req->uring_cmd.driver_cb(&req->uring_cmd); +} + +void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, + void (*driver_cb)(struct io_uring_cmd *)) +{ + struct io_kiocb *req = container_of(ioucmd, struct io_kiocb, uring_cmd); + + req->uring_cmd.driver_cb = driver_cb; + req->io_task_work.func = io_uring_cmd_work; + io_req_task_work_add(req, !!(req->ctx->flags & IORING_SETUP_SQPOLL)); +} +EXPORT_SYMBOL_GPL(io_uring_cmd_complete_in_task); + static void io_req_task_queue_fail(struct io_kiocb *req, int ret) { req->result = ret; @@ -4109,6 +4122,51 @@ static int io_linkat(struct io_kiocb *req, unsigned int issue_flags) return 0; } +/* + * Called by consumers of io_uring_cmd, if they originally returned + * -EIOCBQUEUED upon receiving the command. + */ +void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret) +{ + struct io_kiocb *req = container_of(ioucmd, struct io_kiocb, uring_cmd); + + if (ret < 0) + req_set_fail(req); + io_req_complete(req, ret); +} +EXPORT_SYMBOL_GPL(io_uring_cmd_done); + +static int io_uring_cmd_prep(struct io_kiocb *req, + const struct io_uring_sqe *sqe) +{ + struct io_uring_cmd *ioucmd = &req->uring_cmd; + + if (!req->file->f_op->async_cmd || !(req->ctx->flags & IORING_SETUP_SQE128)) + return -EOPNOTSUPP; + if (req->ctx->flags & IORING_SETUP_IOPOLL) + return -EOPNOTSUPP; + ioucmd->cmd = (void *) &sqe->cmd; + ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); + ioucmd->cmd_len = READ_ONCE(sqe->cmd_len); + ioucmd->flags = 0; + return 0; +} + +static int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) +{ + struct file *file = req->file; + int ret; + struct io_uring_cmd *ioucmd = &req->uring_cmd; + + ioucmd->flags |= issue_flags; + ret = file->f_op->async_cmd(ioucmd); + /* queued async, consumer will call io_uring_cmd_done() when complete */ + if (ret == -EIOCBQUEUED) + return 0; + io_uring_cmd_done(ioucmd, ret); + return 0; +} + static int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { @@ -6588,6 +6646,8 @@ static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return io_symlinkat_prep(req, sqe); case IORING_OP_LINKAT: return io_linkat_prep(req, sqe); + case IORING_OP_URING_CMD: + return io_uring_cmd_prep(req, sqe); } printk_once(KERN_WARNING "io_uring: unhandled opcode %d\n", @@ -6871,6 +6931,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) case IORING_OP_LINKAT: ret = io_linkat(req, issue_flags); break; + case IORING_OP_URING_CMD: + ret = io_uring_cmd(req, issue_flags); + break; default: ret = -EINVAL; break; @@ -11215,6 +11278,8 @@ static int __init io_uring_init(void) BUILD_BUG_ON(ARRAY_SIZE(io_op_defs) != IORING_OP_LAST); BUILD_BUG_ON(__REQ_F_LAST_BIT > 8 * sizeof(int)); + BUILD_BUG_ON(sizeof(struct io_uring_cmd) > 64); + req_cachep = KMEM_CACHE(io_kiocb, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); return 0; diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 649a4d7c241b..cedc68201469 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -5,7 +5,29 @@ #include #include +enum io_uring_cmd_flags { + IO_URING_F_COMPLETE_DEFER = 1, + IO_URING_F_UNLOCKED = 2, + /* int's last bit, sign checks are usually faster than a bit test */ + IO_URING_F_NONBLOCK = INT_MIN, +}; + +struct io_uring_cmd { + struct file *file; + void *cmd; + /* for irq-completion - if driver requires doing stuff in task-context*/ + void (*driver_cb)(struct io_uring_cmd *cmd); + u32 flags; + u32 cmd_op; + u16 cmd_len; + u16 unused; + u8 pdu[28]; /* available inline for free use */ +}; + #if defined(CONFIG_IO_URING) +void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret); +void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, + void (*driver_cb)(struct io_uring_cmd *)); struct sock *io_uring_get_socket(struct file *file); void __io_uring_cancel(bool cancel_all); void __io_uring_free(struct task_struct *tsk); @@ -26,6 +48,13 @@ static inline void io_uring_free(struct task_struct *tsk) __io_uring_free(tsk); } #else +static inline void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret) +{ +} +static inline void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, + void (*driver_cb)(struct io_uring_cmd *)) +{ +} static inline struct sock *io_uring_get_socket(struct file *file) { return NULL; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index c5db68433ca5..9bf1d6c0ed7f 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -22,10 +22,12 @@ struct io_uring_sqe { union { __u64 off; /* offset into file */ __u64 addr2; + __u32 cmd_op; }; union { __u64 addr; /* pointer to buffer or iovecs */ __u64 splice_off_in; + __u16 cmd_len; }; __u32 len; /* buffer size or number of iovecs */ union { @@ -60,8 +62,10 @@ struct io_uring_sqe { __s32 splice_fd_in; __u32 file_index; }; - __u64 __pad2[2]; - + union { + __u64 __pad2[2]; + __u64 cmd; + }; /* * If the ring is initializefd with IORING_SETUP_SQE128, then this field * contains 64-bytes of padding, doubling the size of the SQE. @@ -150,6 +154,7 @@ enum { IORING_OP_MKDIRAT, IORING_OP_SYMLINKAT, IORING_OP_LINKAT, + IORING_OP_URING_CMD, /* this goes last, obviously */ IORING_OP_LAST, From patchwork Tue Mar 8 15:20:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E271C433FE for ; Tue, 8 Mar 2022 15:42:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347927AbiCHPnL (ORCPT ); Tue, 8 Mar 2022 10:43:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346457AbiCHPnK (ORCPT ); Tue, 8 Mar 2022 10:43:10 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5030A4EF4B for ; Tue, 8 Mar 2022 07:42:13 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220308154211epoutp014f05a4261f63067dfe4162623bcbad82~acihxac7I0712807128epoutp01X for ; Tue, 8 Mar 2022 15:42:11 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220308154211epoutp014f05a4261f63067dfe4162623bcbad82~acihxac7I0712807128epoutp01X DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754131; bh=+Gw3S7EpbsROEHcqCLKFsIh2z7rntifub50Jlm8CLsc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fMsa5Y0HhBZci8W2AiAmKkJZ2fM9vv4JWn1P7By9t9X/DYNX6Izvn3TGvaZ5fxctl 9eK98gYSqdP+tz0dFhPRFFHzNMTeLbLFRWKdQYNLsaocXZlQhY5y5x3rSaa8uwzGsZ f4KDrdRV4LwDewmvIlFsBfIotW9sWSF306oY+zPA= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20220308154211epcas5p2895be8485a317815417f0aca54971c63~acihDmOBN3013030130epcas5p2E; Tue, 8 Mar 2022 15:42:11 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.181]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4KCfhZ4Bk1z4x9Pt; Tue, 8 Mar 2022 15:42:06 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 42.57.06423.E4977226; Wed, 9 Mar 2022 00:42:06 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20220308152700epcas5p4130d20119a3a250a2515217d6552f668~acVQ_6hVA0648306483epcas5p4C; Tue, 8 Mar 2022 15:27:00 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152700epsmtrp1f92bca6e505e95dcbfe04f8e92e04356~acVQ_BlwD0125001250epsmtrp1O; Tue, 8 Mar 2022 15:27:00 +0000 (GMT) X-AuditID: b6c32a49-b13ff70000001917-d2-6227794e1152 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 15.51.29871.4C577226; Wed, 9 Mar 2022 00:27:00 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152658epsmtip1ff873fbf2c626628b6a5c1f7dc1c8700~acVO4fKK11072310723epsmtip1p; Tue, 8 Mar 2022 15:26:58 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 04/17] nvme: modify nvme_alloc_request to take an additional parameter Date: Tue, 8 Mar 2022 20:50:52 +0530 Message-Id: <20220308152105.309618-5-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmuq5fpXqSwbJFChbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsapae9ZCqYpVew/OZetgfGSTBcjJ4eEgInEuRcLmbsYuTiEBHYzSuxb PYMVJCEk8IlRYuaeUojEN0aJ3gnfWWE61rYfYYJI7GWUeHv1JQuE85lR4vrTDUAOBwebgKbE hcmlIA0iAl4S92+/ZwWpYRboYpJ4u+8+G0hCWCBCYs6JTnYQm0VAVeL7i5VgvbwClhK71qtC LJOXmHnpO1gJp4CVxM9bW8GO4BUQlDg58wkLiM0MVNO8dTbYCxICJzgkLi28ygLR7CLxt/MV E4QtLPHq+BZ2CFtK4mV/G5RdLPHrzlGo5g6gBxpmQjXbS1zc85cJ5CBmoGfW79KHCMtKTD21 jgliMZ9E7+8nUPN5JXbMg7EVJe5NegoNLXGJhzOWQNkeEjv+32aHBFYvo8SfKQeYJzAqzELy 0CwkD81CWL2AkXkVo2RqQXFuemqxaYFhXmo5PJaT83M3MYLTtZbnDsa7Dz7oHWJk4mA8xCjB wawkwnv/vEqSEG9KYmVValF+fFFpTmrxIUZTYIBPZJYSTc4HZoy8knhDE0sDEzMzMxNLYzND JXHe0+kbEoUE0hNLUrNTUwtSi2D6mDg4pRqYGrT/vVFc5PtgpfS5+4zG/qpzelOK/5yUe1iZ 5V2jseuikfG6i+8X5qzgPnjsKfPj36Vrzj8//04yuI9LZ6Nu2OKsS13CqfpPZqcIKoRvS32T +WVTa7iTgpLt1iuimg9XzJ595aC3zlK2I7GChpO8sudP1wz7dCzZc94OO4dl9bc/zTS+4mH3 KPrCuYX36sPvbTTsCk4s8g8OnmB+8dfpKSptJ6pCvF99tY787vZ2/blVN3LaJXPn8tk4JL42 usc9nzvHx1uK897Cnh02L/pOzczVDGo7yDizcHtS3KSk3TGfd3xtZi9hds45oWb76FhYfVfn GhFJ8VcCU7Y9mZJYwdm4yiP+qeCXvLfXDqxSUGIpzkg01GIuKk4EAK+jZgJgBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmkeLIzCtJLcpLzFFi42LZdlhJTvdIqXqSwf8uPovphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFcVlk5Kak1mWWqRvl8CVcWrae5aCaUoV+0/OZWtgvCTTxcjJISFgIrG2 /QhTFyMXh5DAbkaJzQdfs0MkxCWar/2AsoUlVv57zg5R9JFRontyC2MXIwcHm4CmxIXJpSA1 IgIBEgcbL4PVMAvMYJLoaf7MApIQFgiT2PRhKjOIzSKgKvH9xUoWkF5eAUuJXetVIebLS8y8 9B1sF6eAlcTPW1tZQWwhoJIV636zgdi8AoISJ2c+ARvJDFTfvHU28wRGgVlIUrOQpBYwMq1i lEwtKM5Nzy02LDDMSy3XK07MLS7NS9dLzs/dxAiOJy3NHYzbV33QO8TIxMF4iFGCg1lJhPf+ eZUkId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rwXuk7GCwmkJ5akZqemFqQWwWSZODilGphstOL7 xRU0fjYWG8nY7PSuFo+WEqiLP6W8g+nmlEWnpzF3Nc5T/CVuX2N2Q/biHLZF7DVtzpvl52zd fWvtslYdlZ23tJsZbf24rFwPMAnKWvz/zfx5W8se1VUaycZsDY/+PC7PCJD4+/Hc87MrZ016 NOmCbMQTa77F9ToxdZ6XIxOZnk+2UXKTzf3avFhZcH9RRrPf2aqc3PBJCu3JOn7T+773yEUt u7GE/eVx640e0tlemU4Sf74e+ON1t87+3GGvrK1PPX0fPxW9MTdjztoUMfFDi+dNmRIruslj 77R5s/0YL/zQ2Gce+02C7cbuhG8mW1JMXupeKQnqfVZy6rXGyUOvK+JKPth5PZkurqzEUpyR aKjFXFScCAC/ZVvRFgMAAA== X-CMS-MailID: 20220308152700epcas5p4130d20119a3a250a2515217d6552f668 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152700epcas5p4130d20119a3a250a2515217d6552f668 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch This is a prep patch. It modifies nvme_alloc_request to take an additional parameter, allowing request flags to be passed. Signed-off-by: Keith Busch --- drivers/nvme/host/core.c | 10 ++++++---- drivers/nvme/host/ioctl.c | 2 +- drivers/nvme/host/nvme.h | 3 ++- drivers/nvme/host/pci.c | 4 ++-- drivers/nvme/target/passthru.c | 2 +- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 961a5f8a44d2..159944499c4f 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -630,11 +630,13 @@ static inline void nvme_init_request(struct request *req, } struct request *nvme_alloc_request(struct request_queue *q, - struct nvme_command *cmd, blk_mq_req_flags_t flags) + struct nvme_command *cmd, blk_mq_req_flags_t flags, + unsigned int rq_flags) { + unsigned int cmd_flags = nvme_req_op(cmd) | rq_flags; struct request *req; - req = blk_mq_alloc_request(q, nvme_req_op(cmd), flags); + req = blk_mq_alloc_request(q, cmd_flags, flags); if (!IS_ERR(req)) nvme_init_request(req, cmd); return req; @@ -1075,7 +1077,7 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, int ret; if (qid == NVME_QID_ANY) - req = nvme_alloc_request(q, cmd, flags); + req = nvme_alloc_request(q, cmd, flags, 0); else req = nvme_alloc_request_qid(q, cmd, flags, qid); if (IS_ERR(req)) @@ -1271,7 +1273,7 @@ static void nvme_keep_alive_work(struct work_struct *work) } rq = nvme_alloc_request(ctrl->admin_q, &ctrl->ka_cmd, - BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT); + BLK_MQ_REQ_RESERVED | BLK_MQ_REQ_NOWAIT, 0); if (IS_ERR(rq)) { /* allocation failure, reset the controller */ dev_err(ctrl->device, "keep-alive failed: %ld\n", PTR_ERR(rq)); diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 22314962842d..5c9cd9695519 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -66,7 +66,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, void *meta = NULL; int ret; - req = nvme_alloc_request(q, cmd, 0); + req = nvme_alloc_request(q, cmd, 0, 0); if (IS_ERR(req)) return PTR_ERR(req); diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index a162f6c6da6e..b32f4e2c68fd 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -698,7 +698,8 @@ void nvme_start_freeze(struct nvme_ctrl *ctrl); #define NVME_QID_ANY -1 struct request *nvme_alloc_request(struct request_queue *q, - struct nvme_command *cmd, blk_mq_req_flags_t flags); + struct nvme_command *cmd, blk_mq_req_flags_t flags, + unsigned int rq_flags); void nvme_cleanup_cmd(struct request *req); blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req); blk_status_t nvme_fail_nonready_command(struct nvme_ctrl *ctrl, diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 6a99ed680915..655c26589ac3 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1429,7 +1429,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) req->tag, nvmeq->qid); abort_req = nvme_alloc_request(dev->ctrl.admin_q, &cmd, - BLK_MQ_REQ_NOWAIT); + BLK_MQ_REQ_NOWAIT, 0); if (IS_ERR(abort_req)) { atomic_inc(&dev->ctrl.abort_limit); return BLK_EH_RESET_TIMER; @@ -2475,7 +2475,7 @@ static int nvme_delete_queue(struct nvme_queue *nvmeq, u8 opcode) cmd.delete_queue.opcode = opcode; cmd.delete_queue.qid = cpu_to_le16(nvmeq->qid); - req = nvme_alloc_request(q, &cmd, BLK_MQ_REQ_NOWAIT); + req = nvme_alloc_request(q, &cmd, BLK_MQ_REQ_NOWAIT, 0); if (IS_ERR(req)) return PTR_ERR(req); diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c index 9e5b89ae29df..2a9e2fd3b137 100644 --- a/drivers/nvme/target/passthru.c +++ b/drivers/nvme/target/passthru.c @@ -253,7 +253,7 @@ static void nvmet_passthru_execute_cmd(struct nvmet_req *req) timeout = nvmet_req_subsys(req)->admin_timeout; } - rq = nvme_alloc_request(q, req->cmd, 0); + rq = nvme_alloc_request(q, req->cmd, 0, 0); if (IS_ERR(rq)) { status = NVME_SC_INTERNAL; goto out_put_ns; From patchwork Tue Mar 8 15:20:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3061EC433FE for ; Tue, 8 Mar 2022 15:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347957AbiCHPng (ORCPT ); Tue, 8 Mar 2022 10:43:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346457AbiCHPnZ (ORCPT ); Tue, 8 Mar 2022 10:43:25 -0500 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 915444EF65 for ; Tue, 8 Mar 2022 07:42:18 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20220308154217epoutp049e7217ca57461f83edcccbc9a52875bf~acimrqCNk1148011480epoutp04i for ; Tue, 8 Mar 2022 15:42:17 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20220308154217epoutp049e7217ca57461f83edcccbc9a52875bf~acimrqCNk1148011480epoutp04i DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754137; bh=vYkOrensJar+IrmlGpSyBioyW/8GQq4/dZ5JxuOyK8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TAPWfs8qBZkIcOplrw9wm95xtTKQJnEtyDt2yYLMfMlb7Wb1sNsuCiDGGus/EHjGX UUoQwZBSvGaz29y/rPKX/A5r3bvepycCzT38vbeLamdVyCrNQ7RqpiQM8pbh6/a0y2 qsD3UlnQCpVzb/9Mwn6Fs3wgNS97iLuOJ7VHGeLs= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20220308154216epcas5p1142420f56a22642c33b809f4cf1e1d82~acil6BfNG1651416514epcas5p1x; Tue, 8 Mar 2022 15:42:16 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.174]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4KCfhh64DWz4x9Pr; Tue, 8 Mar 2022 15:42:12 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 95.E6.46822.C7677226; Wed, 9 Mar 2022 00:30:04 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152702epcas5p1eb1880e024ac8b9531c85a82f31a4e78~acVTGhoqz1816518165epcas5p1g; Tue, 8 Mar 2022 15:27:02 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152702epsmtrp2daca8881aad176cb697ecbf213155727~acVTD5dLI2706527065epsmtrp27; Tue, 8 Mar 2022 15:27:02 +0000 (GMT) X-AuditID: b6c32a4a-dfbff7000000b6e6-0f-6227767c1c5f Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 35.96.03370.6C577226; Wed, 9 Mar 2022 00:27:02 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152700epsmtip1b2f3c1f5f38aa1933d1439fa69114b59~acVRAS1aT3168431684epsmtip1-; Tue, 8 Mar 2022 15:27:00 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 05/17] nvme: wire-up support for async-passthru on char-device. Date: Tue, 8 Mar 2022 20:50:53 +0530 Message-Id: <20220308152105.309618-6-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDJsWRmVeSWpSXmKPExsWy7bCmum5NmXqSwcFlOhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsa6t1uZClakVFybeY61gXFacBcjJ4eEgInEjNZudhBbSGA3o8SnS+ld jFxA9idGiXMTf7NCJD4zSkyfywTT8GzfE3aIol2MEi9/n2eCcICKHsxay9jFyMHBJqApcWFy KUiDiICXxP3b71lBapgFupgk3u67zwaSEBYIlDix6iHYBhYBVYmmG+8YQWxeAUuJrqe7mSG2 yUvMvPQd7DxOASuJn7e2skLUCEqcnPmEBcRmBqpp3jqbGWSBhMARDomjDxeyQzS7SFzZ0MgC YQtLvDq+BSouJfH53V42CLtY4tedo1DNHYwS1xtmQjXYS1zc85cJ5BtmoG/W79KHCMtKTD21 jgliMZ9E7+8n0GDhldgxD8ZWlLg36SkrhC0u8XDGElaQMRICHhJLZktDAquXUWLKyTeMExgV ZiH5ZxaSf2YhbF7AyLyKUTK1oDg3PbXYtMAoL7UcHsnJ+bmbGMHJWstrB+PDBx/0DjEycTAe YpTgYFYS4b1/XiVJiDclsbIqtSg/vqg0J7X4EKMpMMAnMkuJJucD80VeSbyhiaWBiZmZmYml sZmhkjjv6fQNiUIC6YklqdmpqQWpRTB9TBycUg1MC/enCh2b8yj30OkZssVlwYJ6/v2lXX0i S18dsbG8tLDXY8GT2UJuvhNV1PTP72EST4jPOl526sT+TVv7ssKuTZ9alrBJVW0no4SPkmd/ +NFzhd2ren5cdN92ecUe64kqze/f5+zc/+bn4tWX5MqPnljs6GO7Y5Nf3eu0mTcy4tMff7vK mbNfdRPfiqkrTbepfTeYKbfx6cGqYpvI+GnOB5+WuLjevjPL3ii30L7s6Or+vdlSN/+85JRa tZ2TZWe+RqJ6MruBEBff+oZvFj/i17+K2Ptrp/namWuzNq1vWXd7+zUe3xfn5vJqyr3ab9IT Htbj+lWz2X2aiOuS1StjOf7/mc6UvulAaZlnZMH/90osxRmJhlrMRcWJAEcxHHJfBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphkeLIzCtJLcpLzFFi42LZdlhJTvdYqXqSweYfChbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoErY93brUwFK1Iqrs08x9rAOC24i5GTQ0LAROLZ vifsXYxcHEICOxglDj9ZwgqREJdovvaDHcIWllj57zlU0UdGiTPdf9m6GDk42AQ0JS5MLgWp EREIkDjYeBmshllgBpNET/NnFpCEsIC/xKXpU8GGsgioSjTdeMcIYvMKWEp0Pd3NDLFAXmLm pe9gyzgFrCR+3toKVi8EVLNi3W82iHpBiZMzn4DNZAaqb946m3kCo8AsJKlZSFILGJlWMUqm FhTnpucWGxYY5aWW6xUn5haX5qXrJefnbmIER5SW1g7GPas+6B1iZOJgPMQowcGsJMJ7/7xK khBvSmJlVWpRfnxRaU5q8SFGaQ4WJXHeC10n44UE0hNLUrNTUwtSi2CyTBycUg1MBWqNZa8+ ue6bs4F1/2bbLeGPzmeeLoyMsWuIXHa0kHXxA681/2/wn1Vx4jtWXyGslaY8aU7gr4AK6VKu 5x/ynvJzJD9/vmvKFRX2bANVS8tpukm/+zROM4qI9W071/q90/vVSuUP3SsuttZuPrpsbsxa HqfZd3q6rX8fvDtbNN7C+kYm52YT3mSzD66T5ts5BkQeS5jBu2lK0mtp4RvxUbE2LZJ5v2bf 81WZGvQn93dkwrUW9rAitkN7H1Q//zOBbWFO4t1fyvpm9R4Tb27LCmYr3b6Fg3n52VcZW7Ve vz96YavCnqR/W7Yocb0MX9tVz3l2f2tv7FJnY7sij3oDl7ky5j/Mc3Orpm/6c19GiaU4I9FQ i7moOBEAVijQjxcDAAA= X-CMS-MailID: 20220308152702epcas5p1eb1880e024ac8b9531c85a82f31a4e78 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152702epcas5p1eb1880e024ac8b9531c85a82f31a4e78 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Introduce handler for fops->async_cmd(), implementing async passthru on char device (/dev/ngX). The handler supports NVME_IOCTL_IO64_CMD for read and write commands. Returns failure for other commands. This is low overhead path for processing the inline commands housed inside io_uring's sqe. Neither the commmand is fetched via copy_from_user, nor the result (inside passthru command) is updated via put_user. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- drivers/nvme/host/core.c | 1 + drivers/nvme/host/ioctl.c | 205 ++++++++++++++++++++++++++++------ drivers/nvme/host/multipath.c | 1 + drivers/nvme/host/nvme.h | 3 + 4 files changed, 178 insertions(+), 32 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 159944499c4f..3fe8f5901cd9 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3667,6 +3667,7 @@ static const struct file_operations nvme_ns_chr_fops = { .release = nvme_ns_chr_release, .unlocked_ioctl = nvme_ns_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, + .async_cmd = nvme_ns_chr_async_cmd, }; static int nvme_add_ns_cdev(struct nvme_ns *ns) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 5c9cd9695519..1df270b47af5 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -18,6 +18,76 @@ static void __user *nvme_to_user_ptr(uintptr_t ptrval) ptrval = (compat_uptr_t)ptrval; return (void __user *)ptrval; } +/* + * This overlays struct io_uring_cmd pdu. + * Expect build errors if this grows larger than that. + */ +struct nvme_uring_cmd_pdu { + u32 meta_len; + union { + struct bio *bio; + struct request *req; + }; + void *meta; /* kernel-resident buffer */ + void __user *meta_buffer; +} __packed; + +static struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu(struct io_uring_cmd *ioucmd) +{ + return (struct nvme_uring_cmd_pdu *)&ioucmd->pdu; +} + +static void nvme_pt_task_cb(struct io_uring_cmd *ioucmd) +{ + struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); + struct request *req = pdu->req; + int status; + struct bio *bio = req->bio; + + if (nvme_req(req)->flags & NVME_REQ_CANCELLED) + status = -EINTR; + else + status = nvme_req(req)->status; + + /* we can free request */ + blk_mq_free_request(req); + blk_rq_unmap_user(bio); + + if (!status && pdu->meta_buffer) { + if (copy_to_user(pdu->meta_buffer, pdu->meta, pdu->meta_len)) + status = -EFAULT; + } + kfree(pdu->meta); + + io_uring_cmd_done(ioucmd, status); +} + +static void nvme_end_async_pt(struct request *req, blk_status_t err) +{ + struct io_uring_cmd *ioucmd = req->end_io_data; + struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); + /* extract bio before reusing the same field for request */ + struct bio *bio = pdu->bio; + + pdu->req = req; + req->bio = bio; + /* this takes care of setting up task-work */ + io_uring_cmd_complete_in_task(ioucmd, nvme_pt_task_cb); +} + +static void nvme_setup_uring_cmd_data(struct request *rq, + struct io_uring_cmd *ioucmd, void *meta, + void __user *meta_buffer, u32 meta_len) +{ + struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); + + /* to free bio on completion, as req->bio will be null at that time */ + pdu->bio = rq->bio; + pdu->meta = meta; + pdu->meta_buffer = meta_buffer; + pdu->meta_len = meta_len; + rq->end_io_data = ioucmd; +} static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf, unsigned len, u32 seed, bool write) @@ -56,7 +126,8 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf, static int nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, void __user *ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, - u32 meta_seed, u64 *result, unsigned timeout) + u32 meta_seed, u64 *result, unsigned timeout, + struct io_uring_cmd *ioucmd) { bool write = nvme_is_write(cmd); struct nvme_ns *ns = q->queuedata; @@ -64,9 +135,15 @@ static int nvme_submit_user_cmd(struct request_queue *q, struct request *req; struct bio *bio = NULL; void *meta = NULL; + unsigned int rq_flags = 0; + blk_mq_req_flags_t blk_flags = 0; int ret; - req = nvme_alloc_request(q, cmd, 0, 0); + if (ioucmd && (ioucmd->flags & IO_URING_F_NONBLOCK)) { + rq_flags |= REQ_NOWAIT; + blk_flags |= BLK_MQ_REQ_NOWAIT; + } + req = nvme_alloc_request(q, cmd, blk_flags, rq_flags); if (IS_ERR(req)) return PTR_ERR(req); @@ -92,6 +169,19 @@ static int nvme_submit_user_cmd(struct request_queue *q, req->cmd_flags |= REQ_INTEGRITY; } } + if (ioucmd) { /* async dispatch */ + if (cmd->common.opcode == nvme_cmd_write || + cmd->common.opcode == nvme_cmd_read) { + nvme_setup_uring_cmd_data(req, ioucmd, meta, meta_buffer, + meta_len); + blk_execute_rq_nowait(req, 0, nvme_end_async_pt); + return 0; + } else { + /* support only read and write for now. */ + ret = -EINVAL; + goto out_meta; + } + } ret = nvme_execute_passthru_rq(req); if (result) @@ -100,6 +190,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, if (copy_to_user(meta_buffer, meta, meta_len)) ret = -EFAULT; } + out_meta: kfree(meta); out_unmap: if (bio) @@ -170,7 +261,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) return nvme_submit_user_cmd(ns->queue, &c, nvme_to_user_ptr(io.addr), length, - metadata, meta_len, lower_32_bits(io.slba), NULL, 0); + metadata, meta_len, lower_32_bits(io.slba), NULL, 0, + NULL); } static bool nvme_validate_passthru_nsid(struct nvme_ctrl *ctrl, @@ -224,7 +316,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, nvme_to_user_ptr(cmd.addr), cmd.data_len, nvme_to_user_ptr(cmd.metadata), cmd.metadata_len, - 0, &result, timeout); + 0, &result, timeout, NULL); if (status >= 0) { if (put_user(result, &ucmd->result)) @@ -235,45 +327,53 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, } static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, - struct nvme_passthru_cmd64 __user *ucmd) + struct nvme_passthru_cmd64 __user *ucmd, + struct io_uring_cmd *ioucmd) { - struct nvme_passthru_cmd64 cmd; + struct nvme_passthru_cmd64 cmd, *cptr; struct nvme_command c; unsigned timeout = 0; int status; if (!capable(CAP_SYS_ADMIN)) return -EACCES; - if (copy_from_user(&cmd, ucmd, sizeof(cmd))) - return -EFAULT; - if (cmd.flags) + if (!ioucmd) { + if (copy_from_user(&cmd, ucmd, sizeof(cmd))) + return -EFAULT; + cptr = &cmd; + } else { + if (ioucmd->cmd_len != sizeof(struct nvme_passthru_cmd64)) + return -EINVAL; + cptr = (struct nvme_passthru_cmd64 *)ioucmd->cmd; + } + if (cptr->flags) return -EINVAL; - if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid)) + if (!nvme_validate_passthru_nsid(ctrl, ns, cptr->nsid)) return -EINVAL; memset(&c, 0, sizeof(c)); - c.common.opcode = cmd.opcode; - c.common.flags = cmd.flags; - c.common.nsid = cpu_to_le32(cmd.nsid); - c.common.cdw2[0] = cpu_to_le32(cmd.cdw2); - c.common.cdw2[1] = cpu_to_le32(cmd.cdw3); - c.common.cdw10 = cpu_to_le32(cmd.cdw10); - c.common.cdw11 = cpu_to_le32(cmd.cdw11); - c.common.cdw12 = cpu_to_le32(cmd.cdw12); - c.common.cdw13 = cpu_to_le32(cmd.cdw13); - c.common.cdw14 = cpu_to_le32(cmd.cdw14); - c.common.cdw15 = cpu_to_le32(cmd.cdw15); - - if (cmd.timeout_ms) - timeout = msecs_to_jiffies(cmd.timeout_ms); + c.common.opcode = cptr->opcode; + c.common.flags = cptr->flags; + c.common.nsid = cpu_to_le32(cptr->nsid); + c.common.cdw2[0] = cpu_to_le32(cptr->cdw2); + c.common.cdw2[1] = cpu_to_le32(cptr->cdw3); + c.common.cdw10 = cpu_to_le32(cptr->cdw10); + c.common.cdw11 = cpu_to_le32(cptr->cdw11); + c.common.cdw12 = cpu_to_le32(cptr->cdw12); + c.common.cdw13 = cpu_to_le32(cptr->cdw13); + c.common.cdw14 = cpu_to_le32(cptr->cdw14); + c.common.cdw15 = cpu_to_le32(cptr->cdw15); + + if (cptr->timeout_ms) + timeout = msecs_to_jiffies(cptr->timeout_ms); status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, - nvme_to_user_ptr(cmd.addr), cmd.data_len, - nvme_to_user_ptr(cmd.metadata), cmd.metadata_len, - 0, &cmd.result, timeout); + nvme_to_user_ptr(cptr->addr), cptr->data_len, + nvme_to_user_ptr(cptr->metadata), cptr->metadata_len, + 0, &cptr->result, timeout, ioucmd); - if (status >= 0) { - if (put_user(cmd.result, &ucmd->result)) + if (!ioucmd && status >= 0) { + if (put_user(cptr->result, &ucmd->result)) return -EFAULT; } @@ -296,7 +396,7 @@ static int nvme_ctrl_ioctl(struct nvme_ctrl *ctrl, unsigned int cmd, case NVME_IOCTL_ADMIN_CMD: return nvme_user_cmd(ctrl, NULL, argp); case NVME_IOCTL_ADMIN64_CMD: - return nvme_user_cmd64(ctrl, NULL, argp); + return nvme_user_cmd64(ctrl, NULL, argp, NULL); default: return sed_ioctl(ctrl->opal_dev, cmd, argp); } @@ -340,7 +440,7 @@ static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd, case NVME_IOCTL_SUBMIT_IO: return nvme_submit_io(ns, argp); case NVME_IOCTL_IO64_CMD: - return nvme_user_cmd64(ns->ctrl, ns, argp); + return nvme_user_cmd64(ns->ctrl, ns, argp, NULL); default: return -ENOTTY; } @@ -369,6 +469,33 @@ long nvme_ns_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg) return __nvme_ioctl(ns, cmd, (void __user *)arg); } +static int nvme_ns_async_ioctl(struct nvme_ns *ns, struct io_uring_cmd *ioucmd) +{ + int ret; + + BUILD_BUG_ON(sizeof(struct nvme_uring_cmd_pdu) > sizeof(ioucmd->pdu)); + + switch (ioucmd->cmd_op) { + case NVME_IOCTL_IO64_CMD: + ret = nvme_user_cmd64(ns->ctrl, ns, NULL, ioucmd); + break; + default: + ret = -ENOTTY; + } + + if (ret >= 0) + ret = -EIOCBQUEUED; + return ret; +} + +int nvme_ns_chr_async_cmd(struct io_uring_cmd *ioucmd) +{ + struct nvme_ns *ns = container_of(file_inode(ioucmd->file)->i_cdev, + struct nvme_ns, cdev); + + return nvme_ns_async_ioctl(ns, ioucmd); +} + #ifdef CONFIG_NVME_MULTIPATH static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd, void __user *argp, struct nvme_ns_head *head, int srcu_idx) @@ -412,6 +539,20 @@ int nvme_ns_head_ioctl(struct block_device *bdev, fmode_t mode, return ret; } +int nvme_ns_head_chr_async_cmd(struct io_uring_cmd *ioucmd) +{ + struct cdev *cdev = file_inode(ioucmd->file)->i_cdev; + struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev); + int srcu_idx = srcu_read_lock(&head->srcu); + struct nvme_ns *ns = nvme_find_path(head); + int ret = -EWOULDBLOCK; + + if (ns) + ret = nvme_ns_async_ioctl(ns, ioucmd); + srcu_read_unlock(&head->srcu, srcu_idx); + return ret; +} + long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { @@ -480,7 +621,7 @@ long nvme_dev_ioctl(struct file *file, unsigned int cmd, case NVME_IOCTL_ADMIN_CMD: return nvme_user_cmd(ctrl, NULL, argp); case NVME_IOCTL_ADMIN64_CMD: - return nvme_user_cmd64(ctrl, NULL, argp); + return nvme_user_cmd64(ctrl, NULL, argp, NULL); case NVME_IOCTL_IO_CMD: return nvme_dev_user_cmd(ctrl, argp); case NVME_IOCTL_RESET: diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index f8bf6606eb2f..1d798d09456f 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -459,6 +459,7 @@ static const struct file_operations nvme_ns_head_chr_fops = { .release = nvme_ns_head_chr_release, .unlocked_ioctl = nvme_ns_head_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, + .async_cmd = nvme_ns_head_chr_async_cmd, }; static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index b32f4e2c68fd..e6a30543d7c8 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -16,6 +16,7 @@ #include #include #include +#include #include @@ -752,6 +753,8 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg); long nvme_dev_ioctl(struct file *file, unsigned int cmd, unsigned long arg); +int nvme_ns_chr_async_cmd(struct io_uring_cmd *ioucmd); +int nvme_ns_head_chr_async_cmd(struct io_uring_cmd *ioucmd); int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo); extern const struct attribute_group *nvme_ns_id_attr_groups[]; From patchwork Tue Mar 8 15:20:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23799C433F5 for ; Tue, 8 Mar 2022 15:42:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237753AbiCHPnj (ORCPT ); Tue, 8 Mar 2022 10:43:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347937AbiCHPng (ORCPT ); Tue, 8 Mar 2022 10:43:36 -0500 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B29274F45C for ; Tue, 8 Mar 2022 07:42:24 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20220308154223epoutp0438321579215122daca9e2a7520c1652c~acisYM1TB1148011480epoutp04k for ; Tue, 8 Mar 2022 15:42:23 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20220308154223epoutp0438321579215122daca9e2a7520c1652c~acisYM1TB1148011480epoutp04k DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754143; bh=KxAi2S/QOS8P47wclOO/QQaBl7MyXZwh6nrm+2MFnUA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qvYoo4GPnJphpnxrQk2PM/DsAUwQfVmeVZkiCqYugojt9c5y/KWBe8qMOE6H9TLwI hxyMxkBIyDOaPZy4UjH/RJb6fsQ6SkzRByKV30ixuz5sZg2PJ/bGd+aQgYXDd/nk9A bp7T2QqKoigRL1Id0gQ/jQRw1c5c/VrNjibyUNjs= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20220308154221epcas5p28126241c4808ab138321f959e55fea2e~acirQQVlu3013430134epcas5p2M; Tue, 8 Mar 2022 15:42:21 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.181]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4KCfhp671Pz4x9Pp; Tue, 8 Mar 2022 15:42:18 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id B6.E6.46822.28677226; Wed, 9 Mar 2022 00:30:10 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152704epcas5p16610e1f50672b25fa1df5f7c5c261bb5~acVVLiJPN1997519975epcas5p1R; Tue, 8 Mar 2022 15:27:04 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152704epsmtrp14721c045eeb6dbefeedfcf7d5cba2aeb~acVVKukz50125001250epsmtrp1R; Tue, 8 Mar 2022 15:27:04 +0000 (GMT) X-AuditID: b6c32a4a-dfbff7000000b6e6-17-6227768274cd Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 86.51.29871.8C577226; Wed, 9 Mar 2022 00:27:04 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152702epsmtip1f90559af419bb23c2a20175b5e7f4802~acVTKAjdX1072310723epsmtip1q; Tue, 8 Mar 2022 15:27:02 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 06/17] io_uring: prep for fixed-buffer enabled uring-cmd Date: Tue, 8 Mar 2022 20:50:54 +0530 Message-Id: <20220308152105.309618-7-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmpm5TmXqSQds2A4vphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO+PbF+uCPtGK/1+XMTcwHhbsYuTkkBAwkdj2bAJjFyMXh5DAbkaJn0d+ s4AkhAQ+MUqs2JcPkfjGKNE/5yE7TMfCeZPYIBJ7GSXmLz8E5XxmlFjfdoS1i5GDg01AU+LC 5FKQBhEBL4n7t9+zgtQwC3QxSbzdd58NJCEs4CHRM3cimM0ioCrx+vQrRhCbV8BS4mjTW6ht 8hIzL30HszkFrCR+3trKClEjKHFy5hOwU5mBapq3zmYGWSAhcIBDYueCx6wQzS4SE3ffhBok LPHq+BYoW0riZX8blF0s8evOUajmDkaJ6w0zWSAS9hIX9/xlAvmGGeib9bv0IcKyElNPrWOC WMwn0fv7CRNEnFdixzwYW1Hi3qSnUDeISzycsQTK9pC4dfQiNLR6GSW+ztvOMoFRYRaSh2Yh eWgWwuoFjMyrGCVTC4pz01OLTQuM8lLL4bGcnJ+7iRGcrrW8djA+fPBB7xAjEwfjIUYJDmYl Ed7751WShHhTEiurUovy44tKc1KLDzGaAkN8IrOUaHI+MGPklcQbmlgamJiZmZlYGpsZKonz nk7fkCgkkJ5YkpqdmlqQWgTTx8TBKdXANM84veFQusrcmauOnvh10VbN2Ggf32OWHb7/L729 cun935ul5wz8HEJSUpZVzLjufNSi6i3fx8DEG441cw7q7uYOTm7wrpPVFciSORq6oohH44Tl MdviVcevqkyYv+u5Y+/H6kdzBO4Evrev3iTy2oM/LOKbjVfofoO1FeuXnSv84y6mPc9sxoq3 G5rml/07k7aUQaj2/aK5cZOmXi/98TgumHf3aovvLxP5K+s/Wb853SsfcmvK5LfKKpJpfRvr gx9M2nn69c9HDcaFluGGnTO+LzLYusT5q1fXpIMHrpvZvrFeH1hjf4Q1STey877QnT93Dgew mP88PON+7MZ8nxr95Q7zGM08vH+V7DusJqrEUpyRaKjFXFScCABiHbqNYAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmkeLIzCtJLcpLzFFi42LZdlhJTvdEqXqSwYMHhhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr49sX64I+0Yr/X5cxNzAeFuxi5OSQEDCRWDhv ElsXIxeHkMBuRok/c06yQiTEJZqv/WCHsIUlVv57zg5R9JFR4sCFK0AdHBxsApoSFyaXgtSI CARIHGy8DFbDLDCDSaKn+TMLSEJYwEOiZ+5ENhCbRUBV4vXpV4wgNq+ApcTRprdQC+QlZl76 DmZzClhJ/Ly1FewIIaCaFet+s0HUC0qcnPkEbCYzUH3z1tnMExgFZiFJzUKSWsDItIpRMrWg ODc9t9iwwDAvtVyvODG3uDQvXS85P3cTIzietDR3MG5f9UHvECMTB+MhRgkOZiUR3vvnVZKE eFMSK6tSi/Lji0pzUosPMUpzsCiJ817oOhkvJJCeWJKanZpakFoEk2Xi4JRqYMrz2fkpoiRi xtWWeX5tm9vuJ+uIczku1n6W9CXku1jejVqV8rrWR+qat1+9vj772ttoqfexJSeM83Yc/tBw x6O9sOKJWenc5E3+bAvzjjPd+Wl/qlfNq0Tl92F1a+PLuhX7nnesFE5RFjW+6K8SY/lZ21CJ t0GwedGkZ0Ksq1jyr6qu3vt+p1ROXOfdd9Fxobvuv/p03nuZwGpJkf6VVlMCF/y+dIcrdkKR MIv5utd6bLbxmesjbrsnbu6dp8O94tvUg6YL0h45W26SKdt2QY5t4rm9UncbujSjQgO9D29r v39B+WK1XGCcy+cumxdzlaWnJcUmiqldcFh+ziKAIet9xPIU/k0Lb2fMfKa4S4mlOCPRUIu5 qDgRANUzlhEWAwAA X-CMS-MailID: 20220308152704epcas5p16610e1f50672b25fa1df5f7c5c261bb5 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152704epcas5p16610e1f50672b25fa1df5f7c5c261bb5 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Anuj Gupta Refactor the existing code and factor out helper that can be used for passthrough with fixed-buffer. This is a prep patch. Signed-off-by: Anuj Gupta Signed-off-by: Kanchan Joshi --- fs/io_uring.c | 19 ++++++++++++++----- include/linux/io_uring.h | 7 +++++++ 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 1f228a79e68f..ead0cbae8416 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3140,11 +3140,10 @@ static void kiocb_done(struct io_kiocb *req, ssize_t ret, } } -static int __io_import_fixed(struct io_kiocb *req, int rw, struct iov_iter *iter, - struct io_mapped_ubuf *imu) +static int __io_import_fixed(u64 buf_addr, size_t len, int rw, + struct iov_iter *iter, struct io_mapped_ubuf *imu) { - size_t len = req->rw.len; - u64 buf_end, buf_addr = req->rw.addr; + u64 buf_end; size_t offset; if (unlikely(check_add_overflow(buf_addr, (u64)len, &buf_end))) @@ -3213,9 +3212,19 @@ static int io_import_fixed(struct io_kiocb *req, int rw, struct iov_iter *iter) imu = READ_ONCE(ctx->user_bufs[index]); req->imu = imu; } - return __io_import_fixed(req, rw, iter, imu); + return __io_import_fixed(req->rw.addr, req->rw.len, rw, iter, imu); } +int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, + int rw, struct iov_iter *iter, void *ioucmd) +{ + struct io_kiocb *req = container_of(ioucmd, struct io_kiocb, uring_cmd); + struct io_mapped_ubuf *imu = req->imu; + + return __io_import_fixed(ubuf, len, rw, iter, imu); +} +EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed); + static void io_ring_submit_unlock(struct io_ring_ctx *ctx, bool needs_lock) { if (needs_lock) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index cedc68201469..1888a5ea7dbe 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -25,6 +25,8 @@ struct io_uring_cmd { }; #if defined(CONFIG_IO_URING) +int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, + int rw, struct iov_iter *iter, void *ioucmd); void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret); void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, void (*driver_cb)(struct io_uring_cmd *)); @@ -48,6 +50,11 @@ static inline void io_uring_free(struct task_struct *tsk) __io_uring_free(tsk); } #else +int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, + int rw, struct iov_iter *iter, void *ioucmd) +{ + return -1; +} static inline void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret) { } From patchwork Tue Mar 8 15:20:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53328C433F5 for ; Tue, 8 Mar 2022 15:42:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236624AbiCHPnm (ORCPT ); Tue, 8 Mar 2022 10:43:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347979AbiCHPni (ORCPT ); Tue, 8 Mar 2022 10:43:38 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2835AE4D for ; Tue, 8 Mar 2022 07:42:33 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154232epoutp037c768d9a855d10f6ad3424fdfd571c7f~aci0vmLPI2451824518epoutp03- for ; Tue, 8 Mar 2022 15:42:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154232epoutp037c768d9a855d10f6ad3424fdfd571c7f~aci0vmLPI2451824518epoutp03- DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754152; bh=U1RoaKLfaYKKu6QFXNo2zVIHlpoVqjMAIXls6xb1dJE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F0+fQGVzaa8WKvFE2lSkYZ+2nki6kTlkbRCr/6tQzEFLWWkub9G4Vi8ZxYdcbnFgO adcnKiJ8rLMZlV1dR6sdQXU+VM5jAEbT7cCqX4VufBnpI9ITHkuab0CwIDJAiFVJqV 7NoefeDySOEoVCo5sHUVrqkoTDYGRoFiPSIISidc= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220308154231epcas5p431d6bcee1f60b67d6b7c521b685471ff~aciz5kKHo2304623046epcas5p40; Tue, 8 Mar 2022 15:42:31 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.179]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfhx4kp6z4x9Pw; Tue, 8 Mar 2022 15:42:25 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F7.E6.46822.98677226; Wed, 9 Mar 2022 00:30:17 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20220308152707epcas5p430127761a7fd4bf90c2501eabe9ee96e~acVXUfkd11840518405epcas5p4L; Tue, 8 Mar 2022 15:27:07 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152707epsmtrp27029ff4754249d2e5c83ff78fba113ce~acVXTxYdo2706527065epsmtrp29; Tue, 8 Mar 2022 15:27:07 +0000 (GMT) X-AuditID: b6c32a4a-de5ff7000000b6e6-20-62277689114a Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 87.51.29871.BC577226; Wed, 9 Mar 2022 00:27:07 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152705epsmtip1ff777caf077ea0206ffaf6a7f753e9ea~acVVRlEJC0990109901epsmtip1a; Tue, 8 Mar 2022 15:27:04 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 07/17] io_uring: add support for uring_cmd with fixed-buffer Date: Tue, 8 Mar 2022 20:50:55 +0530 Message-Id: <20220308152105.309618-8-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPJsWRmVeSWpSXmKPExsWy7bCmlm5nmXqSwYlGFYvphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO2Pjvjb2gmWyFT0vjjA2MP4U72Lk5JAQMJH4tfosexcjF4eQwG5GiYsn FjNCOJ8YJab23ILKfGOUuN79iQmm5f9LmMReRokHbx9CtXxmlFi4+AZLFyMHB5uApsSFyaUg DSICXhL3b79nBalhFuhikni77z4bSEJYwEdixq/9jCA2i4CqxNLDPcwgNq+ApcS/RRPYILbJ S8y89J0dxOYUsJL4eWsrK0SNoMTJmU9YQGxmoJrmrbOZQRZICJzgkNhx9jPUqS4SB6/OZYaw hSVeHd/CDmFLSXx+txdqQbHErztHoZo7gP5smMkCkbCXuLjnLxPIN8xA36zfpQ8RlpWYemod E8RiPone30+gdvFK7JgHYytK3Jv0lBXCFpd4OGMJlO0h8ePfDmjQ9TJKtKxaxTyBUWEWkodm IXloFsLqBYzMqxglUwuKc9NTi00LjPJSy+HxnJyfu4kRnLK1vHYwPnzwQe8QIxMH4yFGCQ5m JRHe++dVkoR4UxIrq1KL8uOLSnNSiw8xmgJDfCKzlGhyPjBr5JXEG5pYGpiYmZmZWBqbGSqJ 855O35AoJJCeWJKanZpakFoE08fEwSnVwJRycUHI484tNueXxzxaI6sxJT/6QXrB7JYEv9Ce YPHDHNcjnvmfUvi+Nzj/zKG1Meb7zPZW5VdssG+Xvt3TKhAmOU9MPTp/4/2ErytV+uWqbl25 9uP2VR+Zw2/uLTqm3Kd9xsZRPO2EzC4n9kNXs78wT7tn3nvQ0frBv5cVjm+tDbY7yLHHnH+8 nudHCFuGGste2U8bvvnmtz7RPCvjLWTwduIU0ee/6o9XuDpzX6hbY7Zo2szH4toS0200JHdr /f3yJLngmOH944Zt/6fnu/1Lr1X9///7jh3L+vdbeYQUCm+1DdBneXatfnvPnsWtu3Mvd3y+ mbHtP1P+hqP2P26sMLbgUquzz9KIMRRbMlGJpTgj0VCLuag4EQA3j5XuYgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphkeLIzCtJLcpLzFFi42LZdlhJTvd0qXqSwZceNovphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFcVlk5Kak1mWWqRvl8CVsXFfG3vBMtmKnhdHGBsYf4p3MXJySAiYSPx/ eYu9i5GLQ0hgN6PEj90z2CAS4hLN136wQ9jCEiv/PYcq+sgosb/vJXMXIwcHm4CmxIXJpSA1 IgIBEgcbL4PVMAvMYJLoaf7MApIQFvCRmPFrPyOIzSKgKrH0cA8ziM0rYCnxb9EEqGXyEjMv fQdbxilgJfHz1lZWEFsIqGbFut9sEPWCEidnPgGbyQxU37x1NvMERoFZSFKzkKQWMDKtYpRM LSjOTc8tNiwwzEst1ytOzC0uzUvXS87P3cQIjigtzR2M21d90DvEyMTBeIhRgoNZSYT3/nmV JCHelMTKqtSi/Pii0pzU4kOM0hwsSuK8F7pOxgsJpCeWpGanphakFsFkmTg4pRqYzjH93nAm 9GG7Nme+PcMSmTZeqYPfthmqta2auV6I1e7Lnf1HJpc9feGwUHT6jYdlWhun97VfTtfoPP5m eTdr+IT/C2p3HCx0zzSS8T5bpKxxLyIueOXu84/rpKftOnOgq6tZhuOC1d2Z7sfn8rNsX9be +9nE4Fv3ipXp2y3fzp+/6OmGLWc2vtD4fkEz4rDyr20ct7dz3K0wfWR1cJK9qkdq22lH1wmZ HEGqHEe/15y4G2Sqsu2qbvOW3jWr+6v81RuW9a1RjWHbeuCQ4LFjNVPirhjM+RfYonf/R1uR gafZ5XvHbs08mme/1er5+VnbYk1s5mlVtXaL/NCqtQ9OTD56xnLSy8QUWft9DpuuyiqxFGck GmoxFxUnAgBLwz6ZFwMAAA== X-CMS-MailID: 20220308152707epcas5p430127761a7fd4bf90c2501eabe9ee96e X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152707epcas5p430127761a7fd4bf90c2501eabe9ee96e References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add IORING_OP_URING_CMD_FIXED opcode that enables performing the operation with previously registered buffers. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- fs/io_uring.c | 29 ++++++++++++++++++++++++++++- include/linux/io_uring.h | 1 + include/uapi/linux/io_uring.h | 1 + 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index ead0cbae8416..6a1dcea0f538 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1107,6 +1107,9 @@ static const struct io_op_def io_op_defs[] = { [IORING_OP_URING_CMD] = { .needs_file = 1, }, + [IORING_OP_URING_CMD_FIXED] = { + .needs_file = 1, + }, }; /* requests with any of those set should undergo io_disarm_next() */ @@ -4148,16 +4151,25 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_done); static int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { + struct io_ring_ctx *ctx = req->ctx; struct io_uring_cmd *ioucmd = &req->uring_cmd; if (!req->file->f_op->async_cmd || !(req->ctx->flags & IORING_SETUP_SQE128)) return -EOPNOTSUPP; if (req->ctx->flags & IORING_SETUP_IOPOLL) return -EOPNOTSUPP; + if (req->opcode == IORING_OP_URING_CMD_FIXED) { + req->imu = NULL; + io_req_set_rsrc_node(req, ctx); + req->buf_index = READ_ONCE(sqe->buf_index); + ioucmd->flags = IO_URING_F_UCMD_FIXEDBUFS; + } else { + ioucmd->flags = 0; + } + ioucmd->cmd = (void *) &sqe->cmd; ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); ioucmd->cmd_len = READ_ONCE(sqe->cmd_len); - ioucmd->flags = 0; return 0; } @@ -4167,6 +4179,19 @@ static int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) int ret; struct io_uring_cmd *ioucmd = &req->uring_cmd; + if (req->opcode == IORING_OP_URING_CMD_FIXED) { + u32 index, buf_index = req->buf_index; + struct io_ring_ctx *ctx = req->ctx; + struct io_mapped_ubuf *imu = req->imu; + + if (likely(!imu)) { + if (unlikely(buf_index >= ctx->nr_user_bufs)) + return -EFAULT; + index = array_index_nospec(buf_index, ctx->nr_user_bufs); + imu = READ_ONCE(ctx->user_bufs[index]); + req->imu = imu; + } + } ioucmd->flags |= issue_flags; ret = file->f_op->async_cmd(ioucmd); /* queued async, consumer will call io_uring_cmd_done() when complete */ @@ -6656,6 +6681,7 @@ static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) case IORING_OP_LINKAT: return io_linkat_prep(req, sqe); case IORING_OP_URING_CMD: + case IORING_OP_URING_CMD_FIXED: return io_uring_cmd_prep(req, sqe); } @@ -6941,6 +6967,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) ret = io_linkat(req, issue_flags); break; case IORING_OP_URING_CMD: + case IORING_OP_URING_CMD_FIXED: ret = io_uring_cmd(req, issue_flags); break; default: diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 1888a5ea7dbe..abad6175739e 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -8,6 +8,7 @@ enum io_uring_cmd_flags { IO_URING_F_COMPLETE_DEFER = 1, IO_URING_F_UNLOCKED = 2, + IO_URING_F_UCMD_FIXEDBUFS = 4, /* int's last bit, sign checks are usually faster than a bit test */ IO_URING_F_NONBLOCK = INT_MIN, }; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 9bf1d6c0ed7f..ee84be4b6be8 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -155,6 +155,7 @@ enum { IORING_OP_SYMLINKAT, IORING_OP_LINKAT, IORING_OP_URING_CMD, + IORING_OP_URING_CMD_FIXED, /* this goes last, obviously */ IORING_OP_LAST, From patchwork Tue Mar 8 15:20:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95BD6C433EF for ; Tue, 8 Mar 2022 15:42:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245284AbiCHPnm (ORCPT ); Tue, 8 Mar 2022 10:43:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347981AbiCHPnj (ORCPT ); Tue, 8 Mar 2022 10:43:39 -0500 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CA211A3B6 for ; Tue, 8 Mar 2022 07:42:37 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20220308154235epoutp04517be2eb8cc5a4d0e37a1a13a4055ac2~aci3-QTUp1148011480epoutp04o for ; Tue, 8 Mar 2022 15:42:35 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20220308154235epoutp04517be2eb8cc5a4d0e37a1a13a4055ac2~aci3-QTUp1148011480epoutp04o DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754155; bh=xggqX2s11RagKw4Q7ebC+vgPojKy+SUye6vkSA58oIg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XUBprxDBEA1meGt24m5+fi32r1T+0Dx/5sWsfVK1EzaOKrEddx5QdA8Ws2qcjycCa 75700AJdOOkHEyCFcXPXnFiJbUJlUGEEt4Avf4Et6ELD6IsYJnQ7DLchDqIz0ozChU xTRjvj9/Qtd5Uf34suG0s0gw9Yr3XJF8cxzSbnSU= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20220308154234epcas5p213c09e68f69d6e005bc050a6bb38bb18~aci3YCqRz2884128841epcas5p2k; Tue, 8 Mar 2022 15:42:34 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4KCfj3309Mz4x9Pv; Tue, 8 Mar 2022 15:42:31 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 29.57.06423.76977226; Wed, 9 Mar 2022 00:42:31 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152709epcas5p1f9d274a0214dc462c22c278a72d8697c~acVZbAWje1997519975epcas5p1T; Tue, 8 Mar 2022 15:27:09 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152709epsmtrp198e5583e2cca027f550c02dfb6f25123~acVZaLrYR0125001250epsmtrp1W; Tue, 8 Mar 2022 15:27:09 +0000 (GMT) X-AuditID: b6c32a49-b13ff70000001917-f6-622779675e51 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id F6.96.03370.DC577226; Wed, 9 Mar 2022 00:27:09 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152707epsmtip153ad2d104b96e7f2f57cb6e851c5fda1~acVXZd2Mt3168431684epsmtip1D; Tue, 8 Mar 2022 15:27:07 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 08/17] nvme: enable passthrough with fixed-buffer Date: Tue, 8 Mar 2022 20:50:56 +0530 Message-Id: <20220308152105.309618-9-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmhm56pXqSwc9ZQhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsafbxfZC15oVWzd9JW1gXG2chcjJ4eEgInEibN7GbsYuTiEBHYzSrz5 2s8OkhAS+MQocWGzHETiM6PEsSf9rDAd21f0MkEkdjFKTN/2nBWu6ui6KyxdjBwcbAKaEhcm l4I0iAh4Sdy//R6shlmgi0ni7b77bCAJYQFHiembH4KtYxFQldiw6yNYL6+ApcT7yZ4Qy+Ql Zl76DlbCKWAl8fPWVrAjeAUEJU7OfMICYjMD1TRvnc0MMl9C4AiHxOtZP5khml0kHt28zQJh C0u8Or6FHcKWknjZ3wZlF0v8unMUqrmDUeJ6w0yoBnuJi3v+MoEcxAz0zPpd+hBhWYmpp9Yx QSzmk+j9/YQJIs4rsWMejK0ocW/SU2hoiUs8nLEEyvaQ2L1lFzSwehklrq47zziBUWEWkodm IXloFsLqBYzMqxglUwuKc9NTi00LDPNSy+GxnJyfu4kRnK61PHcw3n3wQe8QIxMH4yFGCQ5m JRHe++dVkoR4UxIrq1KL8uOLSnNSiw8xmgIDfCKzlGhyPjBj5JXEG5pYGpiYmZmZWBqbGSqJ 855O35AoJJCeWJKanZpakFoE08fEwSnVwKTFUVC5qOnBcQuvjdwJrV5cFt9O/P3oWSN+eANP tea56BbfIxJxt7SvmM5LFYl8ePthyIQqr77pHjFGd19UzDxTZn154/Ow1Spd88K8LCwN3WTY vVPyfpusS6x+a9dkaeGsx39q4tS7ZxSVn4sbCXyYXPTI2fPn2aW2e1bJifqqax7bvNpp0RfO jTOnivFIHPvsKHdAeY2wT3X1W9aADXtc+iKNjditp0RuXb97WdtXXcPvbhOZ2fZfsV+S8cvy Z1XMvgn8F1pq7u/Qz5rWwyLusmHi+Q8TuS68Douo5OJ+kpDa2+HRNOP4gush+Sz62u2ZHWzf jdcrMTTOFj3vEKFwRuTPbsFdaYfn7tGKU2Ipzkg01GIuKk4EAEmmsJ9gBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvdsqXqSwZzHYhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr48+3i+wFL7Qqtm76ytrAOFu5i5GTQ0LARGL7 il4mEFtIYAejxJ2NfBBxcYnmaz/YIWxhiZX/ngPZXEA1Hxklpj3eCtTAwcEmoClxYXIpSI2I QIDEwcbLYDXMAjOYJHqaP7OAJIQFHCWmb34INohFQFViw66PLCC9vAKWEu8ne0LMl5eYeek7 WAmngJXEz1tbWSHusZRYse43G4jNKyAocXLmE7CRzED1zVtnM09gFJiFJDULSWoBI9MqRsnU guLc9NxiwwKjvNRyveLE3OLSvHS95PzcTYzgaNLS2sG4Z9UHvUOMTByMhxglOJiVRHjvn1dJ EuJNSaysSi3Kjy8qzUktPsQozcGiJM57oetkvJBAemJJanZqakFqEUyWiYNTqoHpSAur+syc h1sXL3p/ZL2uLOexL1f8j6rk3isyX3c9wvzkJqZ32lc6l0v/ajGaYhr2etv8dccXSh+flsGU v8pt34FtBtMLnXPtSnU9GLIkDQyXGGx88fJDvjN7/H3z/EsVpSzn3TZoWDNf/n3vwZQPTAst j+4UmMwxTdTvaa3kj2Prl/8IKrbq4O/g4f/54Fz83v5pz/7PnbCNJ7zshpGYwKqW21yLD/n7 7Zr1PeLfrJ0W0yfp7qgRdpA32Wfz6ur6vN27p+Q9l1OWt/df5pFV2rl+jjNrzMZ4/4bP13na nmcLml3prpMt+/Qs/MYUpqtCFpoJj/4L7ZrF8XvrKTGZsn8Ha/adnyHvp3FuwewyXSWW4oxE Qy3mouJEAAC9jsYVAwAA X-CMS-MailID: 20220308152709epcas5p1f9d274a0214dc462c22c278a72d8697c X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152709epcas5p1f9d274a0214dc462c22c278a72d8697c References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Anuj Gupta Add support to carry out passthrough command with pre-mapped buffers. Signed-off-by: Anuj Gupta Signed-off-by: Kanchan Joshi --- block/blk-map.c | 45 +++++++++++++++++++++++++++++++++++++++ drivers/nvme/host/ioctl.c | 27 ++++++++++++++--------- include/linux/blk-mq.h | 2 ++ 3 files changed, 64 insertions(+), 10 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index 4526adde0156..027e8216e313 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "blk.h" @@ -577,6 +578,50 @@ int blk_rq_map_user(struct request_queue *q, struct request *rq, } EXPORT_SYMBOL(blk_rq_map_user); +/* Unlike blk_rq_map_user () this is only for fixed-buffer async passthrough. */ +int blk_rq_map_user_fixedb(struct request_queue *q, struct request *rq, + u64 ubuf, unsigned long len, gfp_t gfp_mask, + struct io_uring_cmd *ioucmd) +{ + struct iov_iter iter; + size_t iter_count, nr_segs; + struct bio *bio; + int ret; + + /* + * Talk to io_uring to obtain BVEC iterator for the buffer. + * And use that iterator to form bio/request. + */ + ret = io_uring_cmd_import_fixed(ubuf, len, rq_data_dir(rq), &iter, + ioucmd); + if (unlikely(ret < 0)) + return ret; + iter_count = iov_iter_count(&iter); + nr_segs = iter.nr_segs; + + if (!iter_count || (iter_count >> 9) > queue_max_hw_sectors(q)) + return -EINVAL; + if (nr_segs > queue_max_segments(q)) + return -EINVAL; + /* no iovecs to alloc, as we already have a BVEC iterator */ + bio = bio_alloc(gfp_mask, 0); + if (!bio) + return -ENOMEM; + + ret = bio_iov_iter_get_pages(bio, &iter); + if (ret) + goto out_free; + + blk_rq_bio_prep(rq, bio, nr_segs); + return 0; + +out_free: + bio_release_pages(bio, false); + bio_put(bio); + return ret; +} +EXPORT_SYMBOL(blk_rq_map_user_fixedb); + /** * blk_rq_unmap_user - unmap a request with user data * @bio: start of bio list diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 1df270b47af5..91d893eedc82 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -123,8 +123,13 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf, return ERR_PTR(ret); } +static inline bool nvme_is_fixedb_passthru(struct io_uring_cmd *ioucmd) +{ + return ((ioucmd) && (ioucmd->flags & IO_URING_F_UCMD_FIXEDBUFS)); +} + static int nvme_submit_user_cmd(struct request_queue *q, - struct nvme_command *cmd, void __user *ubuffer, + struct nvme_command *cmd, u64 ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, u32 meta_seed, u64 *result, unsigned timeout, struct io_uring_cmd *ioucmd) @@ -152,8 +157,12 @@ static int nvme_submit_user_cmd(struct request_queue *q, nvme_req(req)->flags |= NVME_REQ_USERCMD; if (ubuffer && bufflen) { - ret = blk_rq_map_user(q, req, NULL, ubuffer, bufflen, - GFP_KERNEL); + if (likely(nvme_is_fixedb_passthru(ioucmd))) + ret = blk_rq_map_user_fixedb(q, req, ubuffer, bufflen, + GFP_KERNEL, ioucmd); + else + ret = blk_rq_map_user(q, req, NULL, nvme_to_user_ptr(ubuffer), + bufflen, GFP_KERNEL); if (ret) goto out; bio = req->bio; @@ -260,9 +269,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) c.rw.appmask = cpu_to_le16(io.appmask); return nvme_submit_user_cmd(ns->queue, &c, - nvme_to_user_ptr(io.addr), length, - metadata, meta_len, lower_32_bits(io.slba), NULL, 0, - NULL); + io.addr, length, metadata, meta_len, + lower_32_bits(io.slba), NULL, 0, NULL); } static bool nvme_validate_passthru_nsid(struct nvme_ctrl *ctrl, @@ -314,9 +322,8 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, timeout = msecs_to_jiffies(cmd.timeout_ms); status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, - nvme_to_user_ptr(cmd.addr), cmd.data_len, - nvme_to_user_ptr(cmd.metadata), cmd.metadata_len, - 0, &result, timeout, NULL); + cmd.addr, cmd.data_len, nvme_to_user_ptr(cmd.metadata), + cmd.metadata_len, 0, &result, timeout, NULL); if (status >= 0) { if (put_user(result, &ucmd->result)) @@ -368,7 +375,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, timeout = msecs_to_jiffies(cptr->timeout_ms); status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, - nvme_to_user_ptr(cptr->addr), cptr->data_len, + cptr->addr, cptr->data_len, nvme_to_user_ptr(cptr->metadata), cptr->metadata_len, 0, &cptr->result, timeout, ioucmd); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index d319ffa59354..48bcfd194bdc 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -966,6 +966,8 @@ struct rq_map_data { int blk_rq_map_user(struct request_queue *, struct request *, struct rq_map_data *, void __user *, unsigned long, gfp_t); +int blk_rq_map_user_fixedb(struct request_queue *, struct request *, + u64 ubuf, unsigned long, gfp_t, struct io_uring_cmd *); int blk_rq_map_user_iov(struct request_queue *, struct request *, struct rq_map_data *, const struct iov_iter *, gfp_t); int blk_rq_unmap_user(struct bio *); From patchwork Tue Mar 8 15:20:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E06D0C43217 for ; Tue, 8 Mar 2022 15:42:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347812AbiCHPnn (ORCPT ); Tue, 8 Mar 2022 10:43:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347931AbiCHPnk (ORCPT ); Tue, 8 Mar 2022 10:43:40 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC8FFE79 for ; Tue, 8 Mar 2022 07:42:43 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220308154242epoutp01f1da302fa6b90c58d110509ca3e5f4f6~aci_BYnrA1289012890epoutp01H for ; Tue, 8 Mar 2022 15:42:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220308154242epoutp01f1da302fa6b90c58d110509ca3e5f4f6~aci_BYnrA1289012890epoutp01H DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754162; bh=B9TCCvMpq0hZ8/Gfc3DGPxKEOS7j8oXEpRjRFqsjyFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a1qbUeCVC85O/d9o9pjyBjUdN0oxRnKCImsmwGVM0lP6UoXwT5ojM6tDfL2/fpjAW zqMmY8GwNNc1VyCl2f6gWZXBlIEN6kWSMeIjYCPpcXEKQowoNP2G3AfFJkpdmYNVgj X/yqmWCPiZxkaPf7Zg8T2GeVISUdlINdJqb8MC9w= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220308154241epcas5p45d1f0d628e028943aa92360091e60bb3~aci9ZQ1-H0458304583epcas5p4J; Tue, 8 Mar 2022 15:42:41 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4KCfjB3kgFz4x9Pp; Tue, 8 Mar 2022 15:42:38 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 7A.57.06423.E6977226; Wed, 9 Mar 2022 00:42:38 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152711epcas5p31de5d63f5de91fae94e61e5c857c0f13~acVbjD1AI0623006230epcas5p3d; Tue, 8 Mar 2022 15:27:11 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152711epsmtrp2182fc0fe1c80190c69467d6c864a0e91~acVbiFBOx2761027610epsmtrp2D; Tue, 8 Mar 2022 15:27:11 +0000 (GMT) X-AuditID: b6c32a49-b13ff70000001917-00-6227796ea243 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 38.96.03370.FC577226; Wed, 9 Mar 2022 00:27:11 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152709epsmtip1d1199c377ac2493630a28bda0b92732b~acVZhcOPT0990109901epsmtip1c; Tue, 8 Mar 2022 15:27:09 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 09/17] io_uring: plug for async bypass Date: Tue, 8 Mar 2022 20:50:57 +0530 Message-Id: <20220308152105.309618-10-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHJsWRmVeSWpSXmKPExsWy7bCmum5epXqSQdc+CYvphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO+PJjqtsBUdZKq5/3cPYwLifuYuRk0NCwETi476V7F2MXBxCArsZJXo2 LmCBcD4xSvTPWsMK4XxmlPj78hlcy9LN85ggErsYJT5fWc0GV3XhzEmgDAcHm4CmxIXJpSAN IgJeEvdvvwebxCzQxSTxdt99NpCEsICZxNtff1lBbBYBVYmDZ+aA9fIKWEl8XWsPsUxeYual 7+wgNidQ+OetrWDlvAKCEidnPmEBsZmBapq3zmYGmS8hcIRDomXHDUaIZheJQ78vs0PYwhKv jm+BsqUkPr/bywZhF0v8unMUqrmDUeJ6w0wWiIS9xMU9f8EOYgZ6Zv0ufYiwrMTUU+uYIBbz SfT+fsIEEeeV2DEPxlaUuDfpKSuELS7xcMYSKNtD4vT3pdCQ62WU6Hp4l30Co8IsJA/NQvLQ LITVCxiZVzFKphYU56anFpsWGOallsOjOTk/dxMjOGFree5gvPvgg94hRiYOxkOMEhzMSiK8 98+rJAnxpiRWVqUW5ccXleakFh9iNAUG+ERmKdHkfGDOyCuJNzSxNDAxMzMzsTQ2M1QS5z2d viFRSCA9sSQ1OzW1ILUIpo+Jg1OqgUmRc+OcgtIV99bO/mMY2WFR9vXCI8dZ/Ifcq9z9Hndt qflxfWLV/KeHZlhPW7XmUZ5lfMJ07RiBORnH5BfvWLfqxZOtisK1fqyyCocKDGub0nvlZL4+ VFwVeDSBM70naWZEpw2Po6Pzzx29anvlwiYV2NivWG7Geq0v9o13jRI33/QDjw/s7t0Y8Gt5 SFPOpHk512tk+Q+GHjIwu+SV4WWnxCH8WSVKnaHDavM8+f5vhyo4Tr//92DhnvTfXy9tPXau esqWfev6TkrnyXo1FLo8Fl3Devh9izBPpOacSQ9nda8StNsrcPrYkmofrY0OIV5yLXdXpfgy Cgr/cTh+/vS7WUmiP/mz05u+dK9b+1eJpTgj0VCLuag4EQA2GwXSYQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmkeLIzCtJLcpLzFFi42LZdlhJTvd8qXqSwf7fahbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr48mOq2wFR1kqrn/dw9jAuJ+5i5GTQ0LARGLp 5nlMXYxcHEICOxglFkx5zQKREJdovvaDHcIWllj57zk7RNFHRomru/8CORwcbAKaEhcml4LU iAgESBxsvAxWwywwg0mip/kz2CBhATOJt7/+soLYLAKqEgfPzGEC6eUVsJL4utYeYr68xMxL 38F2cQKFf97aClYuJGApsWLdbzYQm1dAUOLkzCdgI5mB6pu3zmaewCgwC0lqFpLUAkamVYyS qQXFuem5xYYFRnmp5XrFibnFpXnpesn5uZsYwfGkpbWDcc+qD3qHGJk4GA8xSnAwK4nw3j+v kiTEm5JYWZValB9fVJqTWnyIUZqDRUmc90LXyXghgfTEktTs1NSC1CKYLBMHp1QDk7qPS4C/ 0SOHp4nyAplJGmGzJ8y5dViTReLq2tZYOc6bW/gm+dRy9+aeX/BzjtnEt+0aFSXCS9kPcYWL XdHKmfF4UgUr199Hk2QO1XPtWLqId/qBmwqbdSbIlrKEdebX3lTzDzz7J+u6wQKptF9n4zPT NP93qT7ulIxmTXvEwfTjzvOoPRNUdhSeu7ypYrLdjKNi2W+2ZH3Ywp4qIi714sNRJp2WJq1D 7fwndtx4lvM+2nqCg3N0eO8l5rjlUs5psi5F0TcS5ll/SfKZM/lywDK3vS5TK4V3Pdy7KO2l 5/EvVlor5uT96jltWjy7VXumma5ESOjtC4Im30QvLDkYemD6CvFT6i06jx7Xfd8spcRSnJFo qMVcVJwIAHVdqpsWAwAA X-CMS-MailID: 20220308152711epcas5p31de5d63f5de91fae94e61e5c857c0f13 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152711epcas5p31de5d63f5de91fae94e61e5c857c0f13 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Jens Axboe Enable .plug for uring-cmd. Signed-off-by: Jens Axboe --- fs/io_uring.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/io_uring.c b/fs/io_uring.c index 6a1dcea0f538..f04bb497bd88 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1106,9 +1106,11 @@ static const struct io_op_def io_op_defs[] = { [IORING_OP_LINKAT] = {}, [IORING_OP_URING_CMD] = { .needs_file = 1, + .plug = 1, }, [IORING_OP_URING_CMD_FIXED] = { .needs_file = 1, + .plug = 1, }, }; From patchwork Tue Mar 8 15:20:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A95E4C433FE for ; Tue, 8 Mar 2022 15:42:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347919AbiCHPns (ORCPT ); Tue, 8 Mar 2022 10:43:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347918AbiCHPns (ORCPT ); Tue, 8 Mar 2022 10:43:48 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21B1A275EE for ; Tue, 8 Mar 2022 07:42:51 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154249epoutp034a4ca08a16cefe45699eeed0684cd40c~acjE9V5_02772027720epoutp03g for ; Tue, 8 Mar 2022 15:42:49 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154249epoutp034a4ca08a16cefe45699eeed0684cd40c~acjE9V5_02772027720epoutp03g DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754169; bh=2rlThWeyJ9VUC3jt2jsrqQfe5sVmTxRqTj81niQsLZE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=omUTmwGREf0oE6FWsAaQV8uHDVsrCHQi+N6CYMxuWlqDYP2sxynxNOw24ZqWaKLoO ATlt3MDFcPDldnth+dwgPFcA0i9QyjT5NFCl06BSUCL/RkOV4lr2JP81w3l0gazP79 emp8/cbtpa1kLXfrRadn4lteT9qRcrFVGlOb0bKk= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20220308154248epcas5p1ea3697922cad7004de83762173b2b0b0~acjEUFawJ2331223312epcas5p1_; Tue, 8 Mar 2022 15:42:48 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.178]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfjH5RXGz4x9Pq; Tue, 8 Mar 2022 15:42:43 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 0D.AB.05590.37977226; Wed, 9 Mar 2022 00:42:43 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20220308152714epcas5p4c5a0d16512fd7054c9a713ee28ede492~acVdtyQMf1840518405epcas5p4T; Tue, 8 Mar 2022 15:27:14 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152714epsmtrp138c5015c6abd95afe5165a8e81305e9b~acVdtABXk0125001250epsmtrp1Z; Tue, 8 Mar 2022 15:27:14 +0000 (GMT) X-AuditID: b6c32a4b-739ff700000015d6-52-6227797340bb Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id FA.96.03370.1D577226; Wed, 9 Mar 2022 00:27:13 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152711epsmtip16a2a58aa2b7d39548f005e0fddf6732a~acVbohUIj3168431684epsmtip1E; Tue, 8 Mar 2022 15:27:11 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 10/17] block: wire-up support for plugging Date: Tue, 8 Mar 2022 20:50:58 +0530 Message-Id: <20220308152105.309618-11-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHJsWRmVeSWpSXmKPExsWy7bCmpm5xpXqSQe9TdYvphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO+PL4Q+MBQfkKu79mMDYwDhHsouRk0NCwETi+u5XbF2MXBxCArsZJW5s P8MC4XxilLjXv4YdwvnMKLG1fQIrTMuZHTuZIBK7GCW+d3awgyTAqh7uiexi5OBgE9CUuDC5 FCQsIuAlcf/2e1aQemaBLiaJt/vus4EkhAWsJLasf88MYrMIqEq8m3EBzOYFil+Y85sNYpm8 xMxL38HmcwLFf97aygpRIyhxcuYTFhCbGaimeetsZpAFEgIHOCRaV76HutRF4mjrciYIW1ji 1fEt7BC2lMTL/jYou1ji152jUM0djBLXG2ayQCTsJS7u+csE8g0z0Dfrd+lDhGUlpp5axwSx mE+i9/cTqPm8EjvmwdiKEvcmPYW6QVzi4YwlULaHxJM3B5khIdfLKDF5bSPbBEaFWUgemoXk oVkIqxcwMq9ilEwtKM5NTy02LTDOSy2HR3Nyfu4mRnDC1vLewfjowQe9Q4xMHIyHGCU4mJVE eO+fV0kS4k1JrKxKLcqPLyrNSS0+xGgKDPGJzFKiyfnAnJFXEm9oYmlgYmZmZmJpbGaoJM57 Kn1DopBAemJJanZqakFqEUwfEwenVAOTpG2Tolfbox0TnM+9fq0bHHIq98l2ftb/L+ZG1M0X cpMKvJvxcueh3WFpV40fnN98vFr+70kGtr1PF3hqFzNJegqzctZUnSo9OqPmyxOvcyvrakuD nDi1efVrJjK8dhHa8zWxaVaKw4/nZ/9PWChgz3CqwyU5f1tJ0xzHf5bn+8SSQ3bPcKz7Fnmc QUK59PkVC91ZOg5Rxzo/mV0M+Ge9aMGzw6+/nvD54Jh8XdcpacPZ9KrW1XKnp2h7314kuu/3 rmfmE9dWK/vvXHxP3+yPspBp1+2AiyujLRNrJm/a/OrdJPmN34vndRp3PU8tc5z3akmWudJM g2dLwk86HV0j7z9j3zPWWOFZzXO+KJ3fr8RSnJFoqMVcVJwIAN0QxIxhBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmkeLIzCtJLcpLzFFi42LZdlhJTvdiqXqSwcc7lhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr48vhD4wFB+Qq7v2YwNjAOEeyi5GTQ0LAROLM jp1MILaQwA5GiXWtvhBxcYnmaz/YIWxhiZX/ngPZXEA1Hxkl1i+dx9zFyMHBJqApcWFyKUiN iECAxMHGy2A1zAIzmCR6mj+zgCSEBawktqx/zwxiswioSrybcQHM5gWKX5jzmw1igbzEzEvf wZZxAsV/3trKCnGQpcSKdRA1vAKCEidnPgGbyQxU37x1NvMERoFZSFKzkKQWMDKtYpRMLSjO Tc8tNiwwykst1ytOzC0uzUvXS87P3cQIjictrR2Me1Z90DvEyMTBeIhRgoNZSYT3/nmVJCHe lMTKqtSi/Pii0pzU4kOM0hwsSuK8F7pOxgsJpCeWpGanphakFsFkmTg4pRqYauInfdeNnmtt VrlP61G8hM2l5IjvlXeLF+45yNOskMy3MTnywm2G5mCjt+u2TfnH9tTTbmvMjimhWisj79Yc b2w4ZhI0Q25v7FKtPx0tRy/q1+VEtEq7/uNcsPJS2tFplbt8Ny3apRN5IkHi18Hm6w/KQ/qn xgWWHYpcFeVQWp6y/NOzTz88fUzvVGyLPCuwR7bHkHnXnCwj2b1zGpc9t54yySZveadmZffJ nf/sr/GKfVebZeavySkePslH/vfvDXwCPzOE2T5N/OKbcCHLffvOu0+Dc8rPRk7RebtImqfb 8tfmlulB2n2PJgZbFa979GkND4vAmw0NNXxFn9lbwt+x/Zp0X0dCytbbKEhdiaU4I9FQi7mo OBEAk9OYNxYDAAA= X-CMS-MailID: 20220308152714epcas5p4c5a0d16512fd7054c9a713ee28ede492 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152714epcas5p4c5a0d16512fd7054c9a713ee28ede492 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Jens Axboe Add support to use plugging if it is enabled, else use default path. Signed-off-by: Jens Axboe --- block/blk-mq.c | 90 ++++++++++++++++++++++++++------------------------ 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 1adfe4824ef5..29f65eaf3e6b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2326,6 +2326,40 @@ void __blk_mq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_mq_hctx_mark_pending(hctx, ctx); } +/* + * Allow 2x BLK_MAX_REQUEST_COUNT requests on plug queue for multiple + * queues. This is important for md arrays to benefit from merging + * requests. + */ +static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug) +{ + if (plug->multiple_queues) + return BLK_MAX_REQUEST_COUNT * 2; + return BLK_MAX_REQUEST_COUNT; +} + +static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq) +{ + struct request *last = rq_list_peek(&plug->mq_list); + + if (!plug->rq_count) { + trace_block_plug(rq->q); + } else if (plug->rq_count >= blk_plug_max_rq_count(plug) || + (!blk_queue_nomerges(rq->q) && + blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) { + blk_mq_flush_plug_list(plug, false); + trace_block_plug(rq->q); + } + + if (!plug->multiple_queues && last && last->q != rq->q) + plug->multiple_queues = true; + if (!plug->has_elevator && (rq->rq_flags & RQF_ELV)) + plug->has_elevator = true; + rq->rq_next = NULL; + rq_list_add(&plug->mq_list, rq); + plug->rq_count++; +} + /** * blk_mq_request_bypass_insert - Insert a request at dispatch list. * @rq: Pointer to request to be inserted. @@ -2339,16 +2373,20 @@ void blk_mq_request_bypass_insert(struct request *rq, bool at_head, bool run_queue) { struct blk_mq_hw_ctx *hctx = rq->mq_hctx; + struct blk_plug *plug = current->plug; - spin_lock(&hctx->lock); - if (at_head) - list_add(&rq->queuelist, &hctx->dispatch); - else - list_add_tail(&rq->queuelist, &hctx->dispatch); - spin_unlock(&hctx->lock); - - if (run_queue) - blk_mq_run_hw_queue(hctx, false); + if (plug) { + blk_add_rq_to_plug(plug, rq); + } else { + spin_lock(&hctx->lock); + if (at_head) + list_add(&rq->queuelist, &hctx->dispatch); + else + list_add_tail(&rq->queuelist, &hctx->dispatch); + spin_unlock(&hctx->lock); + if (run_queue) + blk_mq_run_hw_queue(hctx, false); + } } void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx, @@ -2666,40 +2704,6 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, hctx->queue->mq_ops->commit_rqs(hctx); } -/* - * Allow 2x BLK_MAX_REQUEST_COUNT requests on plug queue for multiple - * queues. This is important for md arrays to benefit from merging - * requests. - */ -static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug) -{ - if (plug->multiple_queues) - return BLK_MAX_REQUEST_COUNT * 2; - return BLK_MAX_REQUEST_COUNT; -} - -static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq) -{ - struct request *last = rq_list_peek(&plug->mq_list); - - if (!plug->rq_count) { - trace_block_plug(rq->q); - } else if (plug->rq_count >= blk_plug_max_rq_count(plug) || - (!blk_queue_nomerges(rq->q) && - blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) { - blk_mq_flush_plug_list(plug, false); - trace_block_plug(rq->q); - } - - if (!plug->multiple_queues && last && last->q != rq->q) - plug->multiple_queues = true; - if (!plug->has_elevator && (rq->rq_flags & RQF_ELV)) - plug->has_elevator = true; - rq->rq_next = NULL; - rq_list_add(&plug->mq_list, rq); - plug->rq_count++; -} - static bool blk_mq_attempt_bio_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs) { From patchwork Tue Mar 8 15:20:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0B48C433F5 for ; Tue, 8 Mar 2022 15:43:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347459AbiCHPnz (ORCPT ); Tue, 8 Mar 2022 10:43:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347923AbiCHPny (ORCPT ); Tue, 8 Mar 2022 10:43:54 -0500 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 705721EC67 for ; Tue, 8 Mar 2022 07:42:57 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20220308154255epoutp028b2298c2849b5e04fcfe4799840ca2e7~acjK0tEX13174931749epoutp02Q for ; Tue, 8 Mar 2022 15:42:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20220308154255epoutp028b2298c2849b5e04fcfe4799840ca2e7~acjK0tEX13174931749epoutp02Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754175; bh=t4ZrZpuvC13gfl85z8lIeeIdlnonTz/uCGTqcpBiwxQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vla7QIYupvhYCJ5O2M1R7aYv+2mpLetxZyTgphP+6MD5And8gWCiRL/GphvBEVR6J FKOjG3aqluLZymH1dsL+qAimcx6anIphmZbsMxMziJvLYvBqP6L6hfpJ13nibbeVNK w+TbEocv6DN/d63s3z1/L1YgKkaEKmwXsCIkTl2Q= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20220308154254epcas5p3d9c44455f00cfa3e2418dfaeb97a2037~acjJ7WQs-1110111101epcas5p3b; Tue, 8 Mar 2022 15:42:54 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.183]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfjQ1xGDz4x9Pq; Tue, 8 Mar 2022 15:42:50 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 0F.AB.05590.A7977226; Wed, 9 Mar 2022 00:42:50 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152716epcas5p3d38d2372c184259f1a10c969f7e4396f~acVfzwP-30623006230epcas5p3k; Tue, 8 Mar 2022 15:27:16 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152716epsmtrp20d7f75940f1233e666a3e0d035a212a3~acVfyxrEJ2706527065epsmtrp2C; Tue, 8 Mar 2022 15:27:16 +0000 (GMT) X-AuditID: b6c32a4b-723ff700000015d6-5b-6227797ade75 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 5A.51.29871.4D577226; Wed, 9 Mar 2022 00:27:16 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152714epsmtip165f9a4da945e9fd75cdc64cd6d1d83a5~acVdwBZ8h3168431684epsmtip1F; Tue, 8 Mar 2022 15:27:13 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 11/17] block: factor out helper for bio allocation from cache Date: Tue, 8 Mar 2022 20:50:59 +0530 Message-Id: <20220308152105.309618-12-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFJsWRmVeSWpSXmKPExsWy7bCmpm5VpXqSQdM9bovphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO+PIhNyCk5IV95Z9ZWtgvC3SxcjBISFgIvGwn7GLkYtDSGA3o8SFO4fY uxg5gZxPjBKtX5UgEt8YJdp2HWEGSYA0dJ57zARRtJdR4uwOLwj7M6PE/CdRIEPZBDQlLkwu BQmLCHhJ3L/9nhVkDrNAF5PE23332UASwgL+Ekv/zAGbwyKgKnGp5SCYzStgJbH98TZGiF3y EjMvfQc7iBMo/vPWVlaIGkGJkzOfsIDYzEA1zVtnM4MskBA4wCHxr2sJVLOLxK2ns1ghbGGJ V8e3sEPYUhKf3+1lg7CLJX7dOQrV3MEocb1hJgtEwl7i4p6/TCDfMAN9s36XPkRYVmLqqXVM EIv5JHp/P2GCiPNK7JgHYytK3Jv0FGqvuMTDGUugbA+JiVsuskBCtJdR4kT3F6YJjAqzkDw0 C8lDsxBWL2BkXsUomVpQnJueWmxaYJyXWg6P4uT83E2M4ESt5b2D8dGDD3qHGJk4GA8xSnAw K4nw3j+vkiTEm5JYWZValB9fVJqTWnyI0RQY4hOZpUST84G5Iq8k3tDE0sDEzMzMxNLYzFBJ nPdU+oZEIYH0xJLU7NTUgtQimD4mDk6pBqY5pXOFJA2P+vDHrFGYurLjnprXOwXnrx/02Vre ukpUHV8naCa2YfKrH4kKrxikW3VYdc/737zlUyTv2vhO46btl+P/T3ksqRJaP//pX4ETyXnP ua/NtlQz13mfFJzcL/hbOMnC48zxG8mn6tev1JrGeLmm/Q2TuiTbqh8mFuIhn38ULHno3WOm JvX53TZzObsDFg63K0WPrz99OlzgnVSC5psN00wUT/oue3XslN6Khcvfsq/xuOe+4AXjnfXK twvCm5m8J7qGuLKx+r88X7B+1a7Fa7RS/1s+cV5blc9Xpdz759wtrnl5ppNvXRbocTi08h77 lW45TpOEXacEWM/9tneSO2ofNNOo6KuL3CMlluKMREMt5qLiRAAqDQ66XQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvdKqXqSwas/3BbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr48iE3IKTkhX3ln1la2C8LdLFyMkhIWAi0Xnu MROILSSwm1Gi9aEDRFxcovnaD3YIW1hi5b/nQDYXUM1HRol3ky6wdDFycLAJaEpcmFwKUiMi ECBxsPEyWA2zwAwmiZ7mzywgCWEBX4mFm3vZQGwWAVWJSy0HwZbxClhJbH+8jRFigbzEzEvf wZZxAsV/3trKCnGQpcSKdb/ZIOoFJU7OfAI2kxmovnnrbOYJjAKzkKRmIUktYGRaxSiZWlCc m55bbFhgmJdarlecmFtcmpeul5yfu4kRHE1amjsYt6/6oHeIkYmD8RCjBAezkgjv/fMqSUK8 KYmVValF+fFFpTmpxYcYpTlYlMR5L3SdjBcSSE8sSc1OTS1ILYLJMnFwSjUwuTxhXH0s+kf8 /Z3d2655SX3RqnbMikmTTz55f13iy0BdF3ftb8yZnOeXGr2Vn9Thsv3663D+xEU3s1MKfopP Pch05+uTpPNcG63zM9OOepWfKZO028BWOE3P49vSNSdFAlUiFtuuM7a1nuCasIhNL5bnb/LB u2esFfsue72wjA56krJ7tnz8BJVZUw+/fz3V98VGySCZpBlbj95In3KLM2K61ZRZvYy2D6ft adn4PGC9+KY35jV3ztrvSJPcer30aNaUJStFqpLldtas21f/71xuzG699XfehQr8aE5eFqZm yBd/i2+pjt2qr5kpul8WML05uPznkhtqSu8qTvevYd3Aycpz1ezUFdX98VZmc5RYijMSDbWY i4oTAYJXiPcVAwAA X-CMS-MailID: 20220308152716epcas5p3d38d2372c184259f1a10c969f7e4396f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152716epcas5p3d38d2372c184259f1a10c969f7e4396f References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Factor the code to pull out bio_from_cache helper which is not tied to kiocb. This is prep patch. Signed-off-by: Kanchan Joshi --- block/bio.c | 43 ++++++++++++++++++++++++++----------------- include/linux/bio.h | 1 + 2 files changed, 27 insertions(+), 17 deletions(-) diff --git a/block/bio.c b/block/bio.c index 4312a8085396..5e12c6bd43d3 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1705,27 +1705,12 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) } EXPORT_SYMBOL(bioset_init_from_src); -/** - * bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb - * @kiocb: kiocb describing the IO - * @nr_vecs: number of iovecs to pre-allocate - * @bs: bio_set to allocate from - * - * Description: - * Like @bio_alloc_bioset, but pass in the kiocb. The kiocb is only - * used to check if we should dip into the per-cpu bio_set allocation - * cache. The allocation uses GFP_KERNEL internally. On return, the - * bio is marked BIO_PERCPU_CACHEABLE, and the final put of the bio - * MUST be done from process context, not hard/soft IRQ. - * - */ -struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, - struct bio_set *bs) +struct bio *bio_from_cache(unsigned short nr_vecs, struct bio_set *bs) { struct bio_alloc_cache *cache; struct bio *bio; - if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) + if (nr_vecs > BIO_INLINE_VECS) return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); cache = per_cpu_ptr(bs->cache, get_cpu()); @@ -1744,6 +1729,30 @@ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, bio_set_flag(bio, BIO_PERCPU_CACHE); return bio; } +EXPORT_SYMBOL_GPL(bio_from_cache); + +/** + * bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb + * @kiocb: kiocb describing the IO + * @nr_vecs: number of iovecs to pre-allocate + * @bs: bio_set to allocate from + * + * Description: + * Like @bio_alloc_bioset, but pass in the kiocb. The kiocb is only + * used to check if we should dip into the per-cpu bio_set allocation + * cache. The allocation uses GFP_KERNEL internally. On return, the + * bio is marked BIO_PERCPU_CACHEABLE, and the final put of the bio + * MUST be done from process context, not hard/soft IRQ. + * + */ +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, + struct bio_set *bs) +{ + if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE)) + return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); + + return bio_from_cache(nr_vecs, bs); +} EXPORT_SYMBOL_GPL(bio_alloc_kiocb); static int __init init_bio(void) diff --git a/include/linux/bio.h b/include/linux/bio.h index 117d7f248ac9..3216401f75b0 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -409,6 +409,7 @@ struct bio *bio_alloc_bioset(gfp_t gfp, unsigned short nr_iovecs, struct bio_set *bs); struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, struct bio_set *bs); +struct bio *bio_from_cache(unsigned short nr_vecs, struct bio_set *bs); struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs); extern void bio_put(struct bio *); From patchwork Tue Mar 8 15:21:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36DF0C433EF for ; Tue, 8 Mar 2022 15:43:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347924AbiCHPoC (ORCPT ); Tue, 8 Mar 2022 10:44:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347931AbiCHPn6 (ORCPT ); Tue, 8 Mar 2022 10:43:58 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98A7234652 for ; Tue, 8 Mar 2022 07:43:01 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220308154259epoutp01a3b8186b5abd00a6dc395ae3d9accd74~acjOp3w7h1289012890epoutp01O for ; Tue, 8 Mar 2022 15:42:59 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220308154259epoutp01a3b8186b5abd00a6dc395ae3d9accd74~acjOp3w7h1289012890epoutp01O DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754180; bh=yZrnUJ+OizD3P426LO8RDZ9aFK64rTJqSnlCEwq+Q0Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tgyRWrzGR6g0UegyOi5IZVYEWWxQhmfnzuZnZgfReyAauvnWOLFmdXzugkxsGoDoz RYT4rSboXJUTdgPuoABqyrrI7DFThh2gAV9L3ApIxDqqXGDDWr8cTCiSlkgoZlLCMl ULqPnfmnY/C/UIDMif75Wy8YsVgUtYQkHJYHW1JY= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220308154259epcas5p4378118cc100fa519e953b6d8c5b8ed84~acjONar9f2304623046epcas5p4I; Tue, 8 Mar 2022 15:42:59 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4KCfjX26Wmz4x9Pt; Tue, 8 Mar 2022 15:42:56 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 70.BB.05590.08977226; Wed, 9 Mar 2022 00:42:56 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152718epcas5p3afd2c8a628f4e9733572cbb39270989d~acVh836sV0623006230epcas5p3t; Tue, 8 Mar 2022 15:27:18 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152718epsmtrp20821a446ed5389f4dcc0574257c867fe~acVh7_rJl2706527065epsmtrp2D; Tue, 8 Mar 2022 15:27:18 +0000 (GMT) X-AuditID: b6c32a4b-723ff700000015d6-65-6227798078e9 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 7B.51.29871.6D577226; Wed, 9 Mar 2022 00:27:18 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152716epsmtip17df4dff75fe115ee2b149232b262bb53~acVf4GXN23168431684epsmtip1G; Tue, 8 Mar 2022 15:27:16 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 12/17] nvme: enable bio-cache for fixed-buffer passthru Date: Tue, 8 Mar 2022 20:51:00 +0530 Message-Id: <20220308152105.309618-13-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmlm5DpXqSwZl9nBbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsbXBZfZC27IVyxf1c/ewDhNqouRg0NCwESi56ldFyMXh5DAbkaJOXPW s0M4nxgllr26xQrhfGOUeH5qEVMXIydYR/OZScwQib2MElebtkE5nxklnpxczwgyl01AU+LC 5FKQBhEBL4n7t9+DTWIW6GKSeLvvPhtIQljAXeLs20ssIDaLgKrEtfMguzk5eAWsJNb/WskM sU1eYual72BxTqD4z1tbWSFqBCVOznwC1ssMVNO8dTZU/REOiZm/2CBsF4nby3dCXS0s8er4 FnYIW0ri87u9UDXFEr/uHAV7QEKgg1HiesNMFoiEvcTFPX+ZQJ5hBnpm/S59iLCsxNRT65gg 9vJJ9P5+AjWfV2LHPBhbUeLepKesELa4xMMZS6BsD4mz/+9BA6uXUaJ/20G2CYwKs5D8MwvJ P7MQVi9gZF7FKJlaUJybnlpsWmCcl1oOj+Xk/NxNjOB0reW9g/HRgw96hxiZOBgPMUpwMCuJ 8N4/r5IkxJuSWFmVWpQfX1Sak1p8iNEUGOATmaVEk/OBGSOvJN7QxNLAxMzMzMTS2MxQSZz3 VPqGRCGB9MSS1OzU1ILUIpg+Jg5OqQamtRkvjn+R/bxuMtshvrXF114cnOn3qNdf2GnDCs84 b9nyb2/+buYuNd+zsv/WtlOmZabTXgVYT3Tsd4k7YTU93ubXbKMlSZ2t+YlhZ+9rP+s3ltgy 8V9P8sdb1q8vfuw8uVdnLvOvyXcZM8+2PBJf9q1db8vVAzoyOjvs9tl+Obota/N/2f9+wuap PesKb51eajLnuM76oCsXin/qhl5bKX/98NdT+xhKXQ4fWFYl4ZvosDK46ahBMbNNkm8FA7PS 5OdPNHc2+z03EU18d+3585Wbjm9UcjlkrTtjqpPio7d/Z19or07pYSreGmt8PlagsyXvZqlN 9ks/pSV3lZ+uXu3sv5OJWfOjAXfc1KxGDiWW4oxEQy3mouJEAC6rD9lgBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphkeLIzCtJLcpLzFFi42LZdlhJTvdaqXqSwcnpshbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr4+uCy+wFN+Qrlq/qZ29gnCbVxcjJISFgItF8 ZhJzFyMXh5DAbkaJh+v3M0IkxCWar/1gh7CFJVb+e84OUfQRqGjTJpYuRg4ONgFNiQuTS0Fq RAQCJA42XgarYRaYwSTR0/yZBSQhLOAucfbtJTCbRUBV4tr59WBDeQWsJNb/WskMsUBeYual 72BxTqD4z1tbWUFsIQFLiRXrfrNB1AtKnJz5BGwOM1B989bZzBMYBWYhSc1CklrAyLSKUTK1 oDg3PbfYsMAwL7Vcrzgxt7g0L10vOT93EyM4orQ0dzBuX/VB7xAjEwfjIUYJDmYlEd7751WS hHhTEiurUovy44tKc1KLDzFKc7AoifNe6DoZLySQnliSmp2aWpBaBJNl4uCUamBirwt+oOTo 9640Qavs3bHvgVIWqatdIh6JR2zPF+3c+fre8xQpo4UTi/b6PBN+fMnf5YGA4dqzbRnJmgz2 KkLik43OPV26IVNc70FT+tOYGa6brBLv//Fxu+GqdL3bgmvmB/ZNx6vvejzx0Alwq7uTfebQ VCeBnqr8JN/93speR+9LzczaGVbI8kM5J2Gm/n5VF1azRacWLmWQVV3g+WBa/ScRoaWHLJ7x PDT9+f31BGvHEhv9yLypud+vx8Qnn/fedJL/dOpzdgeT2pJH85+rv/0b72zOXF/uekt0xkyZ T5L/VwnPY4/lcK5YVfX+VuC3dv7WgyIJQVe+zK3M7NKbvzFJdoq6w5mjN07eM1RiKc5INNRi LipOBACYbmLUFwMAAA== X-CMS-MailID: 20220308152718epcas5p3afd2c8a628f4e9733572cbb39270989d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152718epcas5p3afd2c8a628f4e9733572cbb39270989d References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Since we do submission/completion in task, we can have this up. Add a bio-set for nvme as we need that for bio-cache. Signed-off-by: Kanchan Joshi --- block/blk-map.c | 4 ++-- drivers/nvme/host/core.c | 9 +++++++++ drivers/nvme/host/ioctl.c | 2 +- drivers/nvme/host/nvme.h | 1 + include/linux/blk-mq.h | 3 ++- 5 files changed, 15 insertions(+), 4 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index 027e8216e313..c39917f0eb78 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -580,7 +580,7 @@ EXPORT_SYMBOL(blk_rq_map_user); /* Unlike blk_rq_map_user () this is only for fixed-buffer async passthrough. */ int blk_rq_map_user_fixedb(struct request_queue *q, struct request *rq, - u64 ubuf, unsigned long len, gfp_t gfp_mask, + u64 ubuf, unsigned long len, struct bio_set *bs, struct io_uring_cmd *ioucmd) { struct iov_iter iter; @@ -604,7 +604,7 @@ int blk_rq_map_user_fixedb(struct request_queue *q, struct request *rq, if (nr_segs > queue_max_segments(q)) return -EINVAL; /* no iovecs to alloc, as we already have a BVEC iterator */ - bio = bio_alloc(gfp_mask, 0); + bio = bio_from_cache(0, bs); if (!bio) return -ENOMEM; diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 3fe8f5901cd9..4a385001f124 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -30,6 +30,9 @@ #define NVME_MINORS (1U << MINORBITS) +#define NVME_BIO_POOL_SZ (4) +struct bio_set nvme_bio_pool; + unsigned int admin_timeout = 60; module_param(admin_timeout, uint, 0644); MODULE_PARM_DESC(admin_timeout, "timeout in seconds for admin commands"); @@ -4797,6 +4800,11 @@ static int __init nvme_core_init(void) goto unregister_generic_ns; } + result = bioset_init(&nvme_bio_pool, NVME_BIO_POOL_SZ, 0, + BIOSET_NEED_BVECS | BIOSET_PERCPU_CACHE); + if (result < 0) + goto unregister_generic_ns; + return 0; unregister_generic_ns: @@ -4819,6 +4827,7 @@ static int __init nvme_core_init(void) static void __exit nvme_core_exit(void) { + bioset_exit(&nvme_bio_pool); class_destroy(nvme_ns_chr_class); class_destroy(nvme_subsys_class); class_destroy(nvme_class); diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 91d893eedc82..a4cde210aab9 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -159,7 +159,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, if (ubuffer && bufflen) { if (likely(nvme_is_fixedb_passthru(ioucmd))) ret = blk_rq_map_user_fixedb(q, req, ubuffer, bufflen, - GFP_KERNEL, ioucmd); + &nvme_bio_pool, ioucmd); else ret = blk_rq_map_user(q, req, NULL, nvme_to_user_ptr(ubuffer), bufflen, GFP_KERNEL); diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index e6a30543d7c8..9a3e5093dedc 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -47,6 +47,7 @@ extern unsigned int admin_timeout; extern struct workqueue_struct *nvme_wq; extern struct workqueue_struct *nvme_reset_wq; extern struct workqueue_struct *nvme_delete_wq; +extern struct bio_set nvme_bio_pool; /* * List of workarounds for devices that required behavior not specified in diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 48bcfd194bdc..5f21f71b2529 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -967,7 +967,8 @@ struct rq_map_data { int blk_rq_map_user(struct request_queue *, struct request *, struct rq_map_data *, void __user *, unsigned long, gfp_t); int blk_rq_map_user_fixedb(struct request_queue *, struct request *, - u64 ubuf, unsigned long, gfp_t, struct io_uring_cmd *); + u64 ubuf, unsigned long, struct bio_set *, + struct io_uring_cmd *); int blk_rq_map_user_iov(struct request_queue *, struct request *, struct rq_map_data *, const struct iov_iter *, gfp_t); int blk_rq_unmap_user(struct bio *); From patchwork Tue Mar 8 15:21:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 832A3C433EF for ; Tue, 8 Mar 2022 15:43:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347929AbiCHPoG (ORCPT ); Tue, 8 Mar 2022 10:44:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347928AbiCHPoF (ORCPT ); Tue, 8 Mar 2022 10:44:05 -0500 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9D5D34B87 for ; Tue, 8 Mar 2022 07:43:07 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20220308154306epoutp0262dc35f9ab986d1f269ac8a866712df6~acjUeRwCe2790227902epoutp02S for ; Tue, 8 Mar 2022 15:43:06 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20220308154306epoutp0262dc35f9ab986d1f269ac8a866712df6~acjUeRwCe2790227902epoutp02S DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754186; bh=70vfAyIFdHg4l3b5yOr4vSbOCrH9gUU5cNk0pS+RxM0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CYK1GEZyE4Jl78jfYgerBsAlOZF6gmli3yCgpxx/zGyFwYFSNAhSUGOfx+0MjJ/l3 jhO++SuXxV4IqbnYkiFcSRmYZRzlQ9kEsGC61dXA0IyuSnArtSMexei+y/gveBF5DV L+ux8k5tTesHjljPl9wFVOc6IXbBEorv//L9NsPU= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20220308154305epcas5p4a76e9412f39f8c4e74fb40a43f3aaa2f~acjTfREx-0458304583epcas5p4b; Tue, 8 Mar 2022 15:43:05 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.180]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4KCfjd4dxNz4x9Pt; Tue, 8 Mar 2022 15:43:01 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 2E.57.06423.58977226; Wed, 9 Mar 2022 00:43:01 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152720epcas5p19653942458e160714444942ddb8b8579~acVkD7i7p1632216322epcas5p1Z; Tue, 8 Mar 2022 15:27:20 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152720epsmtrp18448af239b56600a66975708faa5278d~acVkDHkoW0125001250epsmtrp1e; Tue, 8 Mar 2022 15:27:20 +0000 (GMT) X-AuditID: b6c32a49-b01ff70000001917-1f-622779850f9b Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 0D.96.03370.8D577226; Wed, 9 Mar 2022 00:27:20 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152718epsmtip1a755c271693873fbf05c0285e1b44641~acViAHzMs0990109901epsmtip1g; Tue, 8 Mar 2022 15:27:18 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 13/17] nvme: allow user passthrough commands to poll Date: Tue, 8 Mar 2022 20:51:01 +0530 Message-Id: <20220308152105.309618-14-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDJsWRmVeSWpSXmKPExsWy7bCmlm5rpXqSwb2HChbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsatuf4Fa7UrzuxLb2A8qNzFyMkhIWAi8arlIUsXIxeHkMBuRollk18w QzifGCWenL3CBuF8ZpRY0TWNDaZl/ZUH7BCJXYwSzzasYoGrOrj1FpDDwcEmoClxYXIpSIOI gJfE/dvvWUFqmAW6mCTe7rsPNklYwEXi5sz97CA2i4CqxO3bK8HivAJWEs8b1jFCbJOXmHnp O1gNJ1D8562trBA1ghInZz5hAbGZgWqat84Gu1tC4ACHxK1Pr6GaXSQe7PzKDmELS7w6vgXK lpJ42d8GZRdL/LpzFKq5g1HiesNMFoiEvcTFPX+ZQL5hBvpm/S59iLCsxNRT65ggFvNJ9P5+ wgQR55XYMQ/GVpS4N+kpK4QtLvFwxhJWkDESAh4S629nQAKrl1Hi7KfbjBMYFWYh+WcWkn9m IWxewMi8ilEytaA4Nz212LTAMC+1HB7Jyfm5mxjByVrLcwfj3Qcf9A4xMnEwHmKU4GBWEuG9 f14lSYg3JbGyKrUoP76oNCe1+BCjKTDAJzJLiSbnA/NFXkm8oYmlgYmZmZmJpbGZoZI47+n0 DYlCAumJJanZqakFqUUwfUwcnFINTMae5UosMZx2zfYLr1+emjTdqXvG1iyzmx0is9wProhl N/F1b7PZN9EtLeh0MCMn49wtjmU9q1g1Vh6syJRa7ZCSo1NntOfzfafkqz+rjvPszQl1slTl uNDadrVS7sSvZ3Xrny19uWTNzEB/66jQXPv2Z+vlXdp3Sq4vf5NZcSzn8+aXszad/J7D8zhs adJ5A4l2/WW/g5czsDG3vrp+b5cb5+Vpv8t5hJn99+ffffE0ZH0EQ0j2PI4fpWdEklX/8rbf nBGpf/ljonROHjOX/p1vmu8sHScUTdXf+T5kw7U0Q4vsxO8WszmTilYeO3jYfZsNYztPjLaQ zayKPxqzfRZUij06ULEy3mo2kxqfEktxRqKhFnNRcSIAyZXjXV8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvdGqXqSwavnuhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr49Zc/4K12hVn9qU3MB5U7mLk5JAQMJFYf+UB excjF4eQwA5GiWV3nzJBJMQlmq/9YIewhSVW/nsOVfSRUaJ77VvGLkYODjYBTYkLk0tBakQE AiQONl4Gq2EWmMEk0dP8mQUkISzgInFz5n6wQSwCqhK3b69kA7F5BawknjesY4RYIC8x89J3 sBpOoPjPW1tZQWwhAUuJFet+Q9ULSpyc+QRsJjNQffPW2cwTGAVmIUnNQpJawMi0ilEytaA4 Nz232LDAKC+1XK84Mbe4NC9dLzk/dxMjOJq0tHYw7ln1Qe8QIxMH4yFGCQ5mJRHe++dVkoR4 UxIrq1KL8uOLSnNSiw8xSnOwKInzXug6GS8kkJ5YkpqdmlqQWgSTZeLglGpganfQ3fO+/+bJ mYksK3McPndGhvP5bWZaZRgdpM0ks3nVgiKGmzOfqLBXPP776Qwfh3Xz5hyHg+rGqic0mJb4 aKb1dNn7Xm+c43dtDZ+nofWKhDNL+0RnODfPOH5S2yCocMLl0BP/vm3zSK7Z4PWgcu7ElICj 9bPb3q5xytstaWwS7fKIUy1O8RPHGZGY8AubP0l6PfNK58+YrWB6Jl9M9Q5vlMmM9IOqi5br LJvQKWRu0fbrfMb3F985JzhzC4gu4yx+9bXvddR8sbSGduNff9/ms27LnbJs0q2mJT+8r/yS SKt4vvHYpezAwPluXhKvlD8dO7Vt1cKVx8/xsj56ZLVpzWrGn8Jvs8zuHC07rsRSnJFoqMVc VJwIAFME66IVAwAA X-CMS-MailID: 20220308152720epcas5p19653942458e160714444942ddb8b8579 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152720epcas5p19653942458e160714444942ddb8b8579 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch The block layer knows how to deal with polled requests. Let the NVMe driver use the previously reserved user "flags" fields to define an option to allocate the request from the polled hardware contexts. If polling is not enabled, then the block layer will automatically fallback to a non-polled request.[1] [1] https://lore.kernel.org/linux-block/20210517171443.GB2709391@dhcp-10-100-145-180.wdc.com/ Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 30 ++++++++++++++++-------------- include/uapi/linux/nvme_ioctl.h | 4 ++++ 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index a4cde210aab9..a6712fb3eb98 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -132,7 +132,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, u64 ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, u32 meta_seed, u64 *result, unsigned timeout, - struct io_uring_cmd *ioucmd) + struct io_uring_cmd *ioucmd, unsigned int rq_flags) { bool write = nvme_is_write(cmd); struct nvme_ns *ns = q->queuedata; @@ -140,7 +140,6 @@ static int nvme_submit_user_cmd(struct request_queue *q, struct request *req; struct bio *bio = NULL; void *meta = NULL; - unsigned int rq_flags = 0; blk_mq_req_flags_t blk_flags = 0; int ret; @@ -216,11 +215,12 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) struct nvme_command c; unsigned length, meta_len; void __user *metadata; + unsigned int rq_flags = 0; if (copy_from_user(&io, uio, sizeof(io))) return -EFAULT; - if (io.flags) - return -EINVAL; + if (io.flags & NVME_HIPRI) + rq_flags |= REQ_POLLED; switch (io.opcode) { case nvme_cmd_write: @@ -258,7 +258,7 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) memset(&c, 0, sizeof(c)); c.rw.opcode = io.opcode; - c.rw.flags = io.flags; + c.rw.flags = 0; c.rw.nsid = cpu_to_le32(ns->head->ns_id); c.rw.slba = cpu_to_le64(io.slba); c.rw.length = cpu_to_le16(io.nblocks); @@ -270,7 +270,7 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) return nvme_submit_user_cmd(ns->queue, &c, io.addr, length, metadata, meta_len, - lower_32_bits(io.slba), NULL, 0, NULL); + lower_32_bits(io.slba), NULL, 0, NULL, rq_flags); } static bool nvme_validate_passthru_nsid(struct nvme_ctrl *ctrl, @@ -292,6 +292,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, { struct nvme_passthru_cmd cmd; struct nvme_command c; + unsigned int rq_flags = 0; unsigned timeout = 0; u64 result; int status; @@ -300,14 +301,14 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, return -EACCES; if (copy_from_user(&cmd, ucmd, sizeof(cmd))) return -EFAULT; - if (cmd.flags) - return -EINVAL; + if (cmd.flags & NVME_HIPRI) + rq_flags |= REQ_POLLED; if (!nvme_validate_passthru_nsid(ctrl, ns, cmd.nsid)) return -EINVAL; memset(&c, 0, sizeof(c)); c.common.opcode = cmd.opcode; - c.common.flags = cmd.flags; + c.common.flags = 0; c.common.nsid = cpu_to_le32(cmd.nsid); c.common.cdw2[0] = cpu_to_le32(cmd.cdw2); c.common.cdw2[1] = cpu_to_le32(cmd.cdw3); @@ -323,7 +324,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, cmd.addr, cmd.data_len, nvme_to_user_ptr(cmd.metadata), - cmd.metadata_len, 0, &result, timeout, NULL); + cmd.metadata_len, 0, &result, timeout, NULL, rq_flags); if (status >= 0) { if (put_user(result, &ucmd->result)) @@ -339,6 +340,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, { struct nvme_passthru_cmd64 cmd, *cptr; struct nvme_command c; + unsigned int rq_flags = 0; unsigned timeout = 0; int status; @@ -353,14 +355,14 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, return -EINVAL; cptr = (struct nvme_passthru_cmd64 *)ioucmd->cmd; } - if (cptr->flags) - return -EINVAL; + if (cptr->flags & NVME_HIPRI) + rq_flags |= REQ_POLLED; if (!nvme_validate_passthru_nsid(ctrl, ns, cptr->nsid)) return -EINVAL; memset(&c, 0, sizeof(c)); c.common.opcode = cptr->opcode; - c.common.flags = cptr->flags; + c.common.flags = 0; c.common.nsid = cpu_to_le32(cptr->nsid); c.common.cdw2[0] = cpu_to_le32(cptr->cdw2); c.common.cdw2[1] = cpu_to_le32(cptr->cdw3); @@ -377,7 +379,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c, cptr->addr, cptr->data_len, nvme_to_user_ptr(cptr->metadata), cptr->metadata_len, - 0, &cptr->result, timeout, ioucmd); + 0, &cptr->result, timeout, ioucmd, rq_flags); if (!ioucmd && status >= 0) { if (put_user(cptr->result, &ucmd->result)) diff --git a/include/uapi/linux/nvme_ioctl.h b/include/uapi/linux/nvme_ioctl.h index d99b5a772698..df2c138c38d9 100644 --- a/include/uapi/linux/nvme_ioctl.h +++ b/include/uapi/linux/nvme_ioctl.h @@ -9,6 +9,10 @@ #include +enum nvme_io_flags { + NVME_HIPRI = 1 << 0, /* use polling queue if available */ +}; + struct nvme_user_io { __u8 opcode; __u8 flags; From patchwork Tue Mar 8 15:21:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9D3AC433FE for ; Tue, 8 Mar 2022 15:43:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238398AbiCHPoK (ORCPT ); Tue, 8 Mar 2022 10:44:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347942AbiCHPoJ (ORCPT ); Tue, 8 Mar 2022 10:44:09 -0500 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C0FA3586C for ; Tue, 8 Mar 2022 07:43:12 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20220308154310epoutp042130f1825dc3c5e3df6730b309145d2c~acjYse9b51434414344epoutp04B for ; Tue, 8 Mar 2022 15:43:10 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20220308154310epoutp042130f1825dc3c5e3df6730b309145d2c~acjYse9b51434414344epoutp04B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754190; bh=sJAKQq5cu+BeAFJ67QPYN1xE08QBey0SZKYq+gcQQjY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QCLr6pgvjZtLJE9I8XcJP7YQm69nAvaKqy3uk+8g/p6uT4+2vBquxURhsyoBMXZtx QcRRZAXNUEtfdO+AOKyJsr/D/IyWnKR7i02Xq8ExuHhamNDDJ5j3PV0KwyeDkrJlBU Juv8svSF94ZI7+B1KvihMrsW5wDWJa+Iyt2PeyoQ= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20220308154310epcas5p39e8f0de73e701e8816476ccce48f652d~acjYKbEDG1091910919epcas5p3z; Tue, 8 Mar 2022 15:43:10 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfjk4Zs8z4x9Pv; Tue, 8 Mar 2022 15:43:06 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id A4.F6.46822.2B677226; Wed, 9 Mar 2022 00:30:58 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152723epcas5p34460b4af720e515317f88dbb78295f06~acVmPNsAI0623006230epcas5p30; Tue, 8 Mar 2022 15:27:23 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152723epsmtrp2ee59c041947572c36d5357c97abd96cd~acVmOWS_E2706527065epsmtrp2F; Tue, 8 Mar 2022 15:27:23 +0000 (GMT) X-AuditID: b6c32a4a-de5ff7000000b6e6-58-622776b2f68a Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id BC.51.29871.AD577226; Wed, 9 Mar 2022 00:27:23 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152720epsmtip1007a153d494d25e1583d17ab1e2ec74a~acVkHsp6q3168431684epsmtip1H; Tue, 8 Mar 2022 15:27:20 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 14/17] io_uring: add polling support for uring-cmd Date: Tue, 8 Mar 2022 20:51:02 +0530 Message-Id: <20220308152105.309618-15-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNJsWRmVeSWpSXmKPExsWy7bCmhu6mMvUkgz3P5CymH1a0aJrwl9li zqptjBar7/azWaxcfZTJ4l3rORaLztMXmCzOvz3MZDHp0DVGi723tC3mL3vKbrGk9TibxY0J Txkt1tx8ymLx+cw8Vgd+j2dXnzF67Jx1l92jecEdFo/LZ0s9Nq3qZPPYvKTeY/fNBjaPbYtf snr0bVnF6PF5k1wAV1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqt kotPgK5bZg7QD0oKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9 vNQSK0MDAyNToMKE7Iz3HQkFXUoViy88Y2pgnCbdxcjBISFgIjH5f0oXIyeHkMBuRokPf827 GLmA7E+MEjOfLGeHcL4xSuxcvoQdpAqkYd25+WwQib2MEp8frIOq+swo8eP/UiaQsWwCmhIX JpeCNIgIeEncv/2eFaSGWaCLSeLtvvtsIAlhASeJjz39TCA2i4CqxIeZF5lBenkFrCQmbJKA WCYvMfPSd7DFnEDhn7e2soLYvAKCEidnPmEBsZmBapq3zmYGmS8hcIRD4szNK4wQzS4S12bu h7paWOLV8S1QtpTE53d72SDsYolfd45CNXcwSlxvmMkCkbCXuLjnL9gzzEDPrN+lDxGWlZh6 ah0TxGI+id7fT5gg4rwSO+bB2IoS9yY9ZYWwxSUezlgCZXtIzN32BxpYvYwSE/edZJ7AqDAL yUOzkDw0C2H1AkbmVYySqQXFuempxaYFRnmp5fA4Ts7P3cQITtVaXjsYHz74oHeIkYmD8RCj BAezkgjv/fMqSUK8KYmVValF+fFFpTmpxYcYTYEBPpFZSjQ5H5gt8kriDU0sDUzMzMxMLI3N DJXEeU+nb0gUEkhPLEnNTk0tSC2C6WPi4JRqYOq6NTnB2SVHW0ns8UV9w4drTf5vbznfXKTf eFqwY7bY5tdaAYVrc+YK/K64GjQnmK8smunoauMPz9Pe/OyoLloldyEtyDrRleFZ6KWP19ZL y1qrZB7gzjgoIrdk1xzhs4/nGi+IF18zs3wuy41fa6f4SZxkmvml3Opi1ndX+R1FKneuWd5a sr34gvyXFfoZ503j33/xsHp+brpFYNcD6etbvH51XawKPZ2wjenedbfWH2/UI1Qm7U427fzm liv4hDFPoPqi5gmbP6eZb07ZeTGLdUJBq8X2X7W+u344aSbwrLD+8MJik8WW8ye+bn99VZrt ZfirZ2Hbb6dPmvZ01cRFCXb/5Nh87BqeGURu9nilxFKckWioxVxUnAgAaxm27l4EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvd2qXqSwdTzTBbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr431HQkGXUsXiC8+YGhinSXcxcnJICJhIrDs3 n62LkYtDSGA3o8SRrYfYIBLiEs3XfrBD2MISK/89Z4co+sgosWD1dpYuRg4ONgFNiQuTS0Fq RAQCJA42XgarYRaYwSTR0/yZBSQhLOAk8bGnnwnEZhFQlfgw8yIzSC+vgJXEhE0SEPPlJWZe +g62ixMo/PPWVlYQW0jAUmLFut9g9/AKCEqcnPkEbCQzUH3z1tnMExgFZiFJzUKSWsDItIpR MrWgODc9t9iwwDAvtVyvODG3uDQvXS85P3cTIziatDR3MG5f9UHvECMTB+MhRgkOZiUR3vvn VZKEeFMSK6tSi/Lji0pzUosPMUpzsCiJ817oOhkvJJCeWJKanZpakFoEk2Xi4JRqYOJQ1r28 tYQ912z5GadfFRqH2VIf9fn4+CfeacorK3SZVRH4PSw/MHL2XlZDrwuaKnFHRVzVguUm75d2 Sj/eNevyn3M7n1/yLWm8ksTxpn/xMWbLYOufjUksx3edfhry4p1QkMLmyM0C77NnNPbJyvh/ zp7Vkur7O4WhpEGfTTWovDT/0dT2yS9OFByaz3DhfivnyYU/+fla7/1ZnsgXacL6T8b1tKdU y2JGg9+vQszTfp+7/XYSd4PYf77lM87nT3nOeLyxfaX+jh9SK8tn5uyU4P26I8qJfdnZVxLL NyV71k8OXN4jtSR8nq6Zz/Iv0wwMBIIiFmc/UDcWPvzFvrW5Tn7NoURbK1Vu579pSizFGYmG WsxFxYkAD8GTJBUDAAA= X-CMS-MailID: 20220308152723epcas5p34460b4af720e515317f88dbb78295f06 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152723epcas5p34460b4af720e515317f88dbb78295f06 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Pankaj Raghav Enable polling support infra for uring-cmd. Collect the bio during submission, and use that to implement polling on completion. Signed-off-by: Pankaj Raghav --- fs/io_uring.c | 51 ++++++++++++++++++++++++++++++++++------ include/linux/io_uring.h | 9 +++++-- 2 files changed, 51 insertions(+), 9 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index f04bb497bd88..8bd9401f9964 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2664,7 +2664,20 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) if (READ_ONCE(req->iopoll_completed)) break; - ret = kiocb->ki_filp->f_op->iopoll(kiocb, &iob, poll_flags); + if (req->opcode == IORING_OP_URING_CMD || + req->opcode == IORING_OP_URING_CMD_FIXED) { + /* uring_cmd structure does not contain kiocb struct */ + struct kiocb kiocb_uring_cmd; + + kiocb_uring_cmd.private = req->uring_cmd.bio; + kiocb_uring_cmd.ki_filp = req->uring_cmd.file; + ret = req->uring_cmd.file->f_op->iopoll(&kiocb_uring_cmd, + &iob, poll_flags); + } else { + ret = kiocb->ki_filp->f_op->iopoll(kiocb, &iob, + poll_flags); + } + if (unlikely(ret < 0)) return ret; else if (ret) @@ -2777,6 +2790,15 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) wq_list_empty(&ctx->iopoll_list)) break; } + + /* + * In some scenarios, completion callback has been queued up to be + * completed in-task context but polling happens in the same task + * not giving a chance for the completion callback to complete. + */ + if (current->task_works) + io_run_task_work(); + ret = io_do_iopoll(ctx, !min); if (ret < 0) break; @@ -4136,6 +4158,14 @@ static int io_linkat(struct io_kiocb *req, unsigned int issue_flags) return 0; } +static void io_complete_uring_cmd_iopoll(struct io_kiocb *req, long res) +{ + WRITE_ONCE(req->result, res); + /* order with io_iopoll_complete() checking ->result */ + smp_wmb(); + WRITE_ONCE(req->iopoll_completed, 1); +} + /* * Called by consumers of io_uring_cmd, if they originally returned * -EIOCBQUEUED upon receiving the command. @@ -4146,7 +4176,11 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret) if (ret < 0) req_set_fail(req); - io_req_complete(req, ret); + + if (req->uring_cmd.flags & IO_URING_F_UCMD_POLLED) + io_complete_uring_cmd_iopoll(req, ret); + else + io_req_complete(req, ret); } EXPORT_SYMBOL_GPL(io_uring_cmd_done); @@ -4158,15 +4192,18 @@ static int io_uring_cmd_prep(struct io_kiocb *req, if (!req->file->f_op->async_cmd || !(req->ctx->flags & IORING_SETUP_SQE128)) return -EOPNOTSUPP; - if (req->ctx->flags & IORING_SETUP_IOPOLL) - return -EOPNOTSUPP; + if (req->ctx->flags & IORING_SETUP_IOPOLL) { + ioucmd->flags = IO_URING_F_UCMD_POLLED; + ioucmd->bio = NULL; + req->iopoll_completed = 0; + } else { + ioucmd->flags = 0; + } if (req->opcode == IORING_OP_URING_CMD_FIXED) { req->imu = NULL; io_req_set_rsrc_node(req, ctx); req->buf_index = READ_ONCE(sqe->buf_index); - ioucmd->flags = IO_URING_F_UCMD_FIXEDBUFS; - } else { - ioucmd->flags = 0; + ioucmd->flags |= IO_URING_F_UCMD_FIXEDBUFS; } ioucmd->cmd = (void *) &sqe->cmd; diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index abad6175739e..65db83d703b7 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -9,6 +9,7 @@ enum io_uring_cmd_flags { IO_URING_F_COMPLETE_DEFER = 1, IO_URING_F_UNLOCKED = 2, IO_URING_F_UCMD_FIXEDBUFS = 4, + IO_URING_F_UCMD_POLLED = 8, /* int's last bit, sign checks are usually faster than a bit test */ IO_URING_F_NONBLOCK = INT_MIN, }; @@ -16,8 +17,12 @@ enum io_uring_cmd_flags { struct io_uring_cmd { struct file *file; void *cmd; - /* for irq-completion - if driver requires doing stuff in task-context*/ - void (*driver_cb)(struct io_uring_cmd *cmd); + union { + void *bio; /* used for polled completion */ + + /* for irq-completion - if driver requires doing stuff in task-context*/ + void (*driver_cb)(struct io_uring_cmd *cmd); + }; u32 flags; u32 cmd_op; u16 cmd_len; From patchwork Tue Mar 8 15:21:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC879C433EF for ; Tue, 8 Mar 2022 15:43:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347934AbiCHPoP (ORCPT ); Tue, 8 Mar 2022 10:44:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243080AbiCHPoO (ORCPT ); Tue, 8 Mar 2022 10:44:14 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C92135DF5 for ; Tue, 8 Mar 2022 07:43:17 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20220308154315epoutp0379473cf5b800af69a4af3be84a786a85~acjdfOqrE2772027720epoutp03o for ; Tue, 8 Mar 2022 15:43:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20220308154315epoutp0379473cf5b800af69a4af3be84a786a85~acjdfOqrE2772027720epoutp03o DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754195; bh=bAliuuQyhVaAA810mSYAZQpiymJdGdlUpCRULteop00=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZBvCGODlFm2X1HTgcUvSA5rQCjOSehbcgfhMICfRSnUm4V+iNMLRGe686gc0cdtbh v+cT3gZQZT0XnY7/n9uYnVBstC2ao+qsSbdB/Awj0C6qi8EeDbgerjGgOd70T/SAts Znvfea8KCr0P9VfOR1f4uI5ErUqExr5FS4f4BJQw= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20220308154314epcas5p154fd80284aae81f189328339708d8b3a~acjcftWYL2331223312epcas5p1j; Tue, 8 Mar 2022 15:43:14 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.175]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4KCfjq6RmMz4x9Ps; Tue, 8 Mar 2022 15:43:11 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 63.BB.05590.F8977226; Wed, 9 Mar 2022 00:43:11 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20220308152725epcas5p36d1ce3269a47c1c22cc0d66bdc2b9eb3~acVoStPq60962409624epcas5p3H; Tue, 8 Mar 2022 15:27:25 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152725epsmtrp19bf628287ac3275a95536afe6a11f50e~acVoRyAOv0125001250epsmtrp1h; Tue, 8 Mar 2022 15:27:25 +0000 (GMT) X-AuditID: b6c32a4b-739ff700000015d6-7a-6227798f409f Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id DD.51.29871.DD577226; Wed, 9 Mar 2022 00:27:25 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152723epsmtip1c7c06b9e44ad86e6b6710274bccc3639~acVmQMLQy0990109901epsmtip1h; Tue, 8 Mar 2022 15:27:23 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 15/17] nvme: wire-up polling for uring-passthru Date: Tue, 8 Mar 2022 20:51:03 +0530 Message-Id: <20220308152105.309618-16-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmlm5/pXqSQdcRfYvphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFZVtk5GamJJapJCal5yfkpmXbqvkHRzvHG9qZmCoa2hpYa6kkJeYm2qr 5OIToOuWmQP0g5JCWWJOKVAoILG4WEnfzqYov7QkVSEjv7jEVim1ICWnwKRArzgxt7g0L10v L7XEytDAwMgUqDAhO+PJr37Ggkk2FaeaFrE1MF437GLk5JAQMJHYfHobUxcjF4eQwG5GiYVz 1rCAJIQEPjFKTN/tD5H4xiix6ekzFpiOOzf/MEMU7WWU+PcjH6LoM6NE75JrjF2MHBxsApoS FyaXgtSICHhJ3L/9nhWkhlmgi0ni7b77bCAJYQF7iV9NN5hAbBYBVYnGJbvA4rwCVhJbutay QSyTl5h56Ts7iM0JFP95aysrRI2gxMmZT8AOYgaqad46mxlkgYTAEQ6JbX/mQF3qInH/xWJW CFtY4tXxLewQtpTEy/42KLtY4tedo1DNHYwS1xtmQjXbS1zc85cJ5BtmoG/W79KHCMtKTD21 jgliMZ9E7+8nTBBxXokd82BsRYl7k55C7RWXeDhjCZTtIfHjwns2SGj1Mkp82bGSfQKjwiwk D81C8tAshNULGJlXMUqmFhTnpqcWmxYY56WWw2M5OT93EyM4XWt572B89OCD3iFGJg7GQ4wS HMxKIrz3z6skCfGmJFZWpRblxxeV5qQWH2I0BYb4RGYp0eR8YMbIK4k3NLE0MDEzMzOxNDYz VBLnPZW+IVFIID2xJDU7NbUgtQimj4mDU6qBaX/Mtf/vYgsCjOamydjPUt7IET45Pu7Sjvn5 l7smXLBes2PK/BP8l3Y2yMW8/V9g5Tfpcd2/XV/8V6lUTrB2PLqq7uDGtjzGSQGRk8MWzLnU whqkN4tl3+IHi/gE7Xee+ev+6oGF2MUbM+54fAm5FrlE0ujtb44VvDZ13Dr7RU+9fNtf9lwn 94qc+NL1E05/0WETT9ItZpnnMnPC6vkZOxnuf3Hp3n1X4OQcKzNlzj02K9lbWt8c1Tvn0Ca0 sHRv498rU77vm6KZUuI066uzb968T1ZzzhzYwf58j+nLluUX9xfZb4udvtN/b7fYP7lHlwxm Lwqddu3B4dI/or12u0pCM9a/e6xfXz5nhhhPzoNoJZbijERDLeai4kQAT0wRymAEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphkeLIzCtJLcpLzFFi42LZdlhJTvduqXqSwd3tIhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr48mvfsaCSTYVp5oWsTUwXjfsYuTkkBAwkbhz 8w9zFyMXh5DAbkaJBftesUAkxCWar/1gh7CFJVb+e84OUfSRUWL52bOMXYwcHGwCmhIXJpeC 1IgIBEgcbLwMVsMsMINJoqf5M9ggYQF7iV9NN5hAbBYBVYnGJbvYQGxeASuJLV1r2SAWyEvM vPQdbBknUPznra2sILaQgKXEinW/oeoFJU7OfAI2kxmovnnrbOYJjAKzkKRmIUktYGRaxSiZ WlCcm55bbFhgmJdarlecmFtcmpeul5yfu4kRHFFamjsYt6/6oHeIkYmD8RCjBAezkgjv/fMq SUK8KYmVValF+fFFpTmpxYcYpTlYlMR5L3SdjBcSSE8sSc1OTS1ILYLJMnFwSjUwpfEU+jLs E637dmnOIfWPPjsnrJz1IWjeq/X/3vw+IMZ48HP2v/LeDSIX6l42zdkksuIgq7mW7VODuQY7 /EWWJ7QIf+D497D8fk77TCkz2ck/JhcF3al8X1/NtHjfdTs1h51Kyb4HZ/EkFN8+Jb5xTTTP 0h36DWsupMzOvefud7FmH8cPEXkHedbkf5OXTqtYcfpTlloEc5aH5PWOJ6mdGQ3zNItEQnLX 3c61PbVhVXwF65tICWe/n1bspUse2+ZMvXxex0SmYR7rdeF3Kv+XJvLHXnti+uzFr/VG0aGL OKwmuM33KDxZ4cjx9aN5pkBYfWzYFKuHwhXWx9ceTtm3bcW8gLNsVx4JfLu4JEFJQYmlOCPR UIu5qDgRAAG6OioXAwAA X-CMS-MailID: 20220308152725epcas5p36d1ce3269a47c1c22cc0d66bdc2b9eb3 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152725epcas5p36d1ce3269a47c1c22cc0d66bdc2b9eb3 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Pankaj Raghav add .iopoll handler for char device (/dev/ngX), which in turn redirects to bio_poll to implement polling. Signed-off-by: Pankaj Raghav --- block/blk-mq.c | 3 +- drivers/nvme/host/core.c | 1 + drivers/nvme/host/ioctl.c | 79 ++++++++++++++++++++++++++++++++++- drivers/nvme/host/multipath.c | 1 + drivers/nvme/host/nvme.h | 4 ++ include/linux/blk-mq.h | 1 + 6 files changed, 86 insertions(+), 3 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 29f65eaf3e6b..6b37774b0d59 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1193,7 +1193,7 @@ void blk_execute_rq_nowait(struct request *rq, bool at_head, rq_end_io_fn *done) } EXPORT_SYMBOL_GPL(blk_execute_rq_nowait); -static bool blk_rq_is_poll(struct request *rq) +bool blk_rq_is_poll(struct request *rq) { if (!rq->mq_hctx) return false; @@ -1203,6 +1203,7 @@ static bool blk_rq_is_poll(struct request *rq) return false; return true; } +EXPORT_SYMBOL_GPL(blk_rq_is_poll); static void blk_rq_poll_completion(struct request *rq, struct completion *wait) { diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 4a385001f124..64254771a28e 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3671,6 +3671,7 @@ static const struct file_operations nvme_ns_chr_fops = { .unlocked_ioctl = nvme_ns_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, .async_cmd = nvme_ns_chr_async_cmd, + .iopoll = nvme_iopoll, }; static int nvme_add_ns_cdev(struct nvme_ns *ns) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index a6712fb3eb98..701feaecabbe 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -37,6 +37,12 @@ static struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu(struct io_uring_cmd *ioucmd return (struct nvme_uring_cmd_pdu *)&ioucmd->pdu; } +static inline bool is_polling_enabled(struct io_uring_cmd *ioucmd, + struct request *req) +{ + return (ioucmd->flags & IO_URING_F_UCMD_POLLED) && blk_rq_is_poll(req); +} + static void nvme_pt_task_cb(struct io_uring_cmd *ioucmd) { struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); @@ -71,8 +77,17 @@ static void nvme_end_async_pt(struct request *req, blk_status_t err) pdu->req = req; req->bio = bio; - /* this takes care of setting up task-work */ - io_uring_cmd_complete_in_task(ioucmd, nvme_pt_task_cb); + + /* + * IO can be completed directly (i.e. without task work) if we are + * polling and in the task context already + */ + if (is_polling_enabled(ioucmd, req)) { + nvme_pt_task_cb(ioucmd); + } else { + /* this takes care of setting up task-work */ + io_uring_cmd_complete_in_task(ioucmd, nvme_pt_task_cb); + } } static void nvme_setup_uring_cmd_data(struct request *rq, @@ -180,6 +195,10 @@ static int nvme_submit_user_cmd(struct request_queue *q, if (ioucmd) { /* async dispatch */ if (cmd->common.opcode == nvme_cmd_write || cmd->common.opcode == nvme_cmd_read) { + if (bio && is_polling_enabled(ioucmd, req)) { + ioucmd->bio = bio; + bio->bi_opf |= REQ_POLLED; + } nvme_setup_uring_cmd_data(req, ioucmd, meta, meta_buffer, meta_len); blk_execute_rq_nowait(req, 0, nvme_end_async_pt); @@ -505,6 +524,32 @@ int nvme_ns_chr_async_cmd(struct io_uring_cmd *ioucmd) return nvme_ns_async_ioctl(ns, ioucmd); } +int nvme_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob, + unsigned int flags) +{ + struct bio *bio = NULL; + struct nvme_ns *ns = NULL; + struct request_queue *q = NULL; + int ret = 0; + + rcu_read_lock(); + bio = READ_ONCE(kiocb->private); + ns = container_of(file_inode(kiocb->ki_filp)->i_cdev, struct nvme_ns, + cdev); + q = ns->queue; + /* + * bio and driver_cb are a part of the same union inside io_uring_cmd + * struct. If driver is loaded without poll queues, completion will be + * IRQ based and driver_cb is populated. We do not want to treat that + * as bio and get into troubles. Avoid this by checking if queue is + * polled and bail out if not. + */ + if ((test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) && bio && bio->bi_bdev) + ret = bio_poll(bio, iob, flags); + rcu_read_unlock(); + return ret; +} + #ifdef CONFIG_NVME_MULTIPATH static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd, void __user *argp, struct nvme_ns_head *head, int srcu_idx) @@ -585,6 +630,36 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, srcu_read_unlock(&head->srcu, srcu_idx); return ret; } + +int nvme_ns_head_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob, + unsigned int flags) +{ + struct bio *bio = NULL; + struct request_queue *q = NULL; + struct cdev *cdev = file_inode(kiocb->ki_filp)->i_cdev; + struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev); + int srcu_idx = srcu_read_lock(&head->srcu); + struct nvme_ns *ns = nvme_find_path(head); + int ret = -EWOULDBLOCK; + + if (ns) { + bio = READ_ONCE(kiocb->private); + q = ns->queue; + /* + * bio and driver_cb are part of same union inside io_uring_cmd + * struct. If driver is loaded without poll queues, completion + * will be IRQ based and driver_cb is populated. We do not want + * to treat that as bio and get into troubles. Avoid this by + * checking if queue is polled, and bail out if not. + */ + if ((test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) && bio && + bio->bi_bdev) + ret = bio_poll(bio, iob, flags); + } + + srcu_read_unlock(&head->srcu, srcu_idx); + return ret; +} #endif /* CONFIG_NVME_MULTIPATH */ static int nvme_dev_user_cmd(struct nvme_ctrl *ctrl, void __user *argp) diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 1d798d09456f..27995358c847 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -460,6 +460,7 @@ static const struct file_operations nvme_ns_head_chr_fops = { .unlocked_ioctl = nvme_ns_head_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, .async_cmd = nvme_ns_head_chr_async_cmd, + .iopoll = nvme_ns_head_iopoll, }; static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 9a3e5093dedc..0be437c25077 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -755,7 +755,11 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, long nvme_dev_ioctl(struct file *file, unsigned int cmd, unsigned long arg); int nvme_ns_chr_async_cmd(struct io_uring_cmd *ioucmd); +int nvme_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob, + unsigned int flags); int nvme_ns_head_chr_async_cmd(struct io_uring_cmd *ioucmd); +int nvme_ns_head_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob, + unsigned int flags); int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo); extern const struct attribute_group *nvme_ns_id_attr_groups[]; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 5f21f71b2529..9f00b2a5a991 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -977,6 +977,7 @@ int blk_rq_map_kern(struct request_queue *, struct request *, void *, int blk_rq_append_bio(struct request *rq, struct bio *bio); void blk_execute_rq_nowait(struct request *rq, bool at_head, rq_end_io_fn *end_io); +bool blk_rq_is_poll(struct request *rq); blk_status_t blk_execute_rq(struct request *rq, bool at_head); struct req_iterator { From patchwork Tue Mar 8 15:21:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45CAFC433FE for ; Tue, 8 Mar 2022 15:43:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347928AbiCHPoU (ORCPT ); Tue, 8 Mar 2022 10:44:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243080AbiCHPoU (ORCPT ); Tue, 8 Mar 2022 10:44:20 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED11E35DEE for ; Tue, 8 Mar 2022 07:43:22 -0800 (PST) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220308154321epoutp01ae7facb25426c7d508559d0e416d8778~acjil30k51309213092epoutp01K for ; Tue, 8 Mar 2022 15:43:21 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220308154321epoutp01ae7facb25426c7d508559d0e416d8778~acjil30k51309213092epoutp01K DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754201; bh=uqZSicb94MLilTPRMRReRmaCfggMTLDRLlbMQ8tYAnE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hREbLhkLEIWVaDrqnSRGtk7arQmu1KMYqW0HN4CRJiKnvJ9yrjMwyvPbtf/e55zx+ LDwt7CUm2dXotjOCUP5Z85zcartcSfbmTXBZY3GUqHYOVtEcREfjIiH1gYWbZDUzp7 aEIHeFX41XVp0i+23XM19wdqU2s7GsRuu5wgMES0= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20220308154320epcas5p108c23cb00779b6ea36ac8164da3038cf~acjiBWvVG2250922509epcas5p1G; Tue, 8 Mar 2022 15:43:20 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.180]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4KCfjx1v35z4x9Pq; Tue, 8 Mar 2022 15:43:17 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 62.67.06423.59977226; Wed, 9 Mar 2022 00:43:17 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20220308152727epcas5p20e605718dd99e97c94f9232d40d04d95~acVqepfNG2419124191epcas5p2d; Tue, 8 Mar 2022 15:27:27 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20220308152727epsmtrp286d7fd65322e1a0949884579c3063f79~acVqd31na2706527065epsmtrp2J; Tue, 8 Mar 2022 15:27:27 +0000 (GMT) X-AuditID: b6c32a49-b01ff70000001917-35-622779950d8e Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 9F.51.29871.FD577226; Wed, 9 Mar 2022 00:27:27 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152725epsmtip1b0c875bce0b2916868189b89eff1b74d~acVoYvW0E1072310723epsmtip1u; Tue, 8 Mar 2022 15:27:25 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 16/17] io_uring: add support for non-inline uring-cmd Date: Tue, 8 Mar 2022 20:51:04 +0530 Message-Id: <20220308152105.309618-17-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLJsWRmVeSWpSXmKPExsWy7bCmhu7USvUkg1kzuCymH1a0aJrwl9li zqptjBar7/azWaxcfZTJ4l3rORaLztMXmCzOvz3MZDHp0DVGi723tC3mL3vKbrGk9TibxY0J Txkt1tx8ymLx+cw8Vgd+j2dXnzF67Jx1l92jecEdFo/LZ0s9Nq3qZPPYvKTeY/fNBjaPbYtf snr0bVnF6PF5k1wAV1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqt kotPgK5bZg7QD0oKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9 vNQSK0MDAyNToMKE7Ixzi34xFvSKV2zYuoatgbFJuIuRk0NCwERi1qkHzF2MXBxCArsZJSZ0 N7JBOJ8YJf42H4fKfGOUWNoxB8jhAGs5sosFIr6XUeL42ZmMEM5nRolZ108zgRSxCWhKXJhc CrJCRMBL4v7t96wgNcwCXUwSb/fdZwOpERZwlfh03xSkhkVAVeLg/3uMIDavgJXEuRnNTBDn yUvMvPSdHcTmBIr/vLWVFaJGUOLkzCcsIDYzUE3z1tlgh0oInOCQ2H72HwtEs4vE0e3noWxh iVfHt7BD2FISn9/tZYOwiyV+3TkK1dzBKHG9YSZUg73ExT1/wZ5hBnpm/S59iLCsxNRT65gg FvNJ9P5+AnUor8SOeTC2osS9SU9ZIWxxiYczlkDZHhLrNkxgggRWL6PEob6FTBMYFWYheWgW kodmIaxewMi8ilEytaA4Nz212LTAMC+1HB7Lyfm5mxjB6VrLcwfj3Qcf9A4xMnEwHmKU4GBW EuG9f14lSYg3JbGyKrUoP76oNCe1+BCjKTDEJzJLiSbnAzNGXkm8oYmlgYmZmZmJpbGZoZI4 7+n0DYlCAumJJanZqakFqUUwfUwcnFINTAV88VrpHZwn3jVLfz/V6N4XfOHflevcAh2KazKa zx22jdg3O+CZqM6dwLJPFhsmzmqJ+pi37plE9tu40qn5Mne22MyOa1vvv84zXsKt0ej9rNSH Na4Mb/l6TTUCovdmWX74+G5Xi295gQXv5eWPFU1U/iWWLtFabPJ6T1SBtKRG09IstUdpKge7 1V1bl8g+flstZcQdab7v+xStfxsuK0t/2vzt+4aUNaHvBDNPVUnLmXGcdy09ZWoUbWPpe3yS odCtN9sc/ZMsX1bnZ5688YDpC+tuduG2Jg6DK0se1J3piS1Z3bFSdPPTvTPXnp5g9ZVnYemk Hra2DVbqtQwfA07tmbvskFh6oe/OmvPZSizFGYmGWsxFxYkA/Ebj0GAEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvd+qXqSweSlqhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AK4rLJiU1J7MstUjfLoEr49yiX4wFveIVG7auYWtgbBLuYuTgkBAwkTiy i6WLkYtDSGA3o8Te+c2sXYycQHFxieZrP9ghbGGJlf+es0MUfWSUOHv2KyNIM5uApsSFyaUg NSICARIHGy+D1TALzGCS6Gn+zAJSIyzgKvHpvilIDYuAqsTB//cYQWxeASuJczOamSDmy0vM vPQdbBcnUPznra1gNwgJWEqsWPebDaJeUOLkzCcsIDYzUH3z1tnMExgFZiFJzUKSWsDItIpR MrWgODc9t9iwwDAvtVyvODG3uDQvXS85P3cTIziatDR3MG5f9UHvECMTB+MhRgkOZiUR3vvn VZKEeFMSK6tSi/Lji0pzUosPMUpzsCiJ817oOhkvJJCeWJKanZpakFoEk2Xi4JRqYJq0TaXz 8CqlC01XljccF9j7VYqpSvFTm0vQpALOgMWqrpPVNqh80JBaqhZntPNedLGEkudUp9CFRwtj 9ea7Xnf34ZaRdhY+FnSKl+N8SQLH4rTPe3fdENy7+hjL43IFr2r/XPY7+3/327D/ywxvvfXQ WMMg71HsMstPhYrcD5XXLZIP2rzm7sltvetsnx898vDGszKN2hzpHUpajRtm9faL3yt7Nkvs e+2Fr2X9FxUPsiV+9vsYLfL15mOWfxXTG6NeftF63TLr469/171iI7z9b9Y3ha+ac9fr+17f OS8/z2yIU2N5c5B9k+jUbRNadB3iz1i+3jGhLHK3WlTc9m8nXKaeFe5YyfH/x9rjt5YqsRRn JBpqMRcVJwIAHysdvBUDAAA= X-CMS-MailID: 20220308152727epcas5p20e605718dd99e97c94f9232d40d04d95 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152727epcas5p20e605718dd99e97c94f9232d40d04d95 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Lifetime of inline commands (within sqe) is upto submission, and maximum length is 80 bytes. This can be limiting for certain commands. Add option to accept command pointer via same sqe->cmd field. User need to set IORING_URING_CMD_INDIRECT flag in sqe->uring_cmd_flags. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- fs/io_uring.c | 13 +++++++++++-- include/linux/io_uring.h | 1 + include/uapi/linux/io_uring.h | 6 ++++++ 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 8bd9401f9964..d88c6601a556 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -4189,8 +4189,12 @@ static int io_uring_cmd_prep(struct io_kiocb *req, { struct io_ring_ctx *ctx = req->ctx; struct io_uring_cmd *ioucmd = &req->uring_cmd; + u32 ucmd_flags = READ_ONCE(sqe->uring_cmd_flags); - if (!req->file->f_op->async_cmd || !(req->ctx->flags & IORING_SETUP_SQE128)) + if (!req->file->f_op->async_cmd) + return -EOPNOTSUPP; + if (!(req->ctx->flags & IORING_SETUP_SQE128) && + !(ucmd_flags & IORING_URING_CMD_INDIRECT)) return -EOPNOTSUPP; if (req->ctx->flags & IORING_SETUP_IOPOLL) { ioucmd->flags = IO_URING_F_UCMD_POLLED; @@ -4206,7 +4210,12 @@ static int io_uring_cmd_prep(struct io_kiocb *req, ioucmd->flags |= IO_URING_F_UCMD_FIXEDBUFS; } - ioucmd->cmd = (void *) &sqe->cmd; + if (ucmd_flags & IORING_URING_CMD_INDIRECT) { + ioucmd->flags |= IO_URING_F_UCMD_INDIRECT; + ioucmd->cmd = (void *) sqe->cmd; + } else { + ioucmd->cmd = (void *) &sqe->cmd; + } ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); ioucmd->cmd_len = READ_ONCE(sqe->cmd_len); return 0; diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 65db83d703b7..c534a6fcef4f 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -10,6 +10,7 @@ enum io_uring_cmd_flags { IO_URING_F_UNLOCKED = 2, IO_URING_F_UCMD_FIXEDBUFS = 4, IO_URING_F_UCMD_POLLED = 8, + IO_URING_F_UCMD_INDIRECT = 16, /* int's last bit, sign checks are usually faster than a bit test */ IO_URING_F_NONBLOCK = INT_MIN, }; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index ee84be4b6be8..a4b9db37ecf1 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -47,6 +47,7 @@ struct io_uring_sqe { __u32 rename_flags; __u32 unlink_flags; __u32 hardlink_flags; + __u32 uring_cmd_flags; }; __u64 user_data; /* data to be passed back at completion time */ /* pack this to avoid bogus arm OABI complaints */ @@ -198,6 +199,11 @@ enum { #define IORING_POLL_UPDATE_EVENTS (1U << 1) #define IORING_POLL_UPDATE_USER_DATA (1U << 2) +/* + * sqe->uring_cmd_flags + */ +#define IORING_URING_CMD_INDIRECT (1U << 0) + /* * IO completion data structure (Completion Queue Entry) */ From patchwork Tue Mar 8 15:21:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 12773991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09BA2C43217 for ; Tue, 8 Mar 2022 15:43:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347940AbiCHPoZ (ORCPT ); Tue, 8 Mar 2022 10:44:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347881AbiCHPoZ (ORCPT ); Tue, 8 Mar 2022 10:44:25 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 442EA35DF0 for ; Tue, 8 Mar 2022 07:43:27 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20220308154325epoutp014e5d6f6b89d23e99cde8772ac42ea2e8~acjmmHJXk1289012890epoutp01W for ; Tue, 8 Mar 2022 15:43:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20220308154325epoutp014e5d6f6b89d23e99cde8772ac42ea2e8~acjmmHJXk1289012890epoutp01W DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1646754205; bh=xblvuUbMlm8XGc4r3UcbItBHUpyrY9AabHvMcjXEnVY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f5D7F7AYK1LMe+K8B1OekPb3XzCx51MJRBsiJtKbVT+4iw48hAkPcHfo1HKiTok1t LiZldJknkhcAwX0IIy2xAczKyuOKcoF+Yf55E7BqCZ4QE/QeHily9EKFbUqRwMoFGJ YG9vcVinnmdnpVjUvgJvyOvFkiRF2ZV1ymxEqc8g= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20220308154324epcas5p3f3f29ebd8b3578242dbe1433d85edc81~acjl2LE4K2611026110epcas5p3G; Tue, 8 Mar 2022 15:43:24 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.175]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4KCfk16sr8z4x9Pp; Tue, 8 Mar 2022 15:43:21 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id D8.F6.46822.1C677226; Wed, 9 Mar 2022 00:31:13 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20220308152729epcas5p17e82d59c68076eb46b5ef658619d65e3~acVsiydo81632216322epcas5p1e; Tue, 8 Mar 2022 15:27:29 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20220308152729epsmtrp17f1e09d2f31b8098aea660127729e8d6~acVsh7ndU0125001250epsmtrp1t; Tue, 8 Mar 2022 15:27:29 +0000 (GMT) X-AuditID: b6c32a4a-de5ff7000000b6e6-6e-622776c1034f Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id BF.96.03370.1E577226; Wed, 9 Mar 2022 00:27:29 +0900 (KST) Received: from localhost.localdomain (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20220308152727epsmtip157d88b77887039cf75d68faff64079c3~acVqghG8i1236312363epsmtip1N; Tue, 8 Mar 2022 15:27:27 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, kbusch@kernel.org, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, sbates@raithlin.com, logang@deltatee.com, pankydev8@gmail.com, javier@javigon.com, mcgrof@kernel.org, a.manzanares@samsung.com, joshiiitr@gmail.com, anuj20.g@samsung.com Subject: [PATCH 17/17] nvme: enable non-inline passthru commands Date: Tue, 8 Mar 2022 20:51:05 +0530 Message-Id: <20220308152105.309618-18-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220308152105.309618-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNJsWRmVeSWpSXmKPExsWy7bCmpu7BMvUkg48NZhbTDytaNE34y2wx Z9U2RovVd/vZLFauPspk8a71HItF5+kLTBbn3x5msph06Bqjxd5b2hbzlz1lt1jSepzN4saE p4wWa24+ZbH4fGYeqwO/x7Orzxg9ds66y+7RvOAOi8fls6Uem1Z1snlsXlLvsftmA5vHtsUv WT36tqxi9Pi8SS6AKyrbJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VW ycUnQNctMwfoByWFssScUqBQQGJxsZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpe XmqJlaGBgZEpUGFCdsbnZdNZC9aKVdybdYWpgXGmUBcjJ4eEgInExPur2LsYuTiEBHYzSnyb /oUVwvnEKPHu83Qo5zOjxOSLF9hhWho+PmADsYUEdjFKPJ5iBWEDFa287djFyMHBJqApcWFy KUhYRMBL4v7t92BzmAW6mCTe7rvPBlIjLOAgsW2/MEgNi4CqxPa/q5lAbF4BK4m3P94zQayS l5h56TvYWk6g+M9bW1khagQlTs58wgJiMwPVNG+dzQwyX0LgCIfE2Uv9LBDNLhInt7xhg7CF JV4d3wJ1v5TEy/42KLtY4tedo1DNHYwS1xtmQjXbS1zc85cJ5FBmoGfW79KHCMtKTD21jgli MZ9E7+8nUIfySuyYB2MrStyb9JQVwhaXeDhjCZTtIXG88RUjJKx6GSXe/naawKgwC8k/s5D8 Mwth8wJG5lWMkqkFxbnpqcWmBUZ5qeXwOE7Oz93ECE7VWl47GB8++KB3iJGJg/EQowQHs5II 7/3zKklCvCmJlVWpRfnxRaU5qcWHGE2BAT6RWUo0OR+YLfJK4g1NLA1MzMzMTCyNzQyVxHlP p29IFBJITyxJzU5NLUgtgulj4uCUamDSnbKBoembkoi+oez8C6u4JeKliu1t596/VLc88IB2 kVL6mt3+725u1tNZzidQtEG3gcFA+fOWG34s62d2O+7va7t02ujPn9IJ5++yOi49n33jhffb z43SwZrS5hzePv+eH5nn3HQysKghsrZ5x9V3dgdqdng9Y3D6L6L9LJGd7dzsHclz2X6mlpXM tTj5WmiS8E3H7AlLbka1yRlN/Cm3MLmh16Jbi838oRuzzfw+j7Lfy2b6dzJX1M0QjXosc7z6 2Mc7jtZvrE7ts92tw9ZfW3vTOEcsfMfe+dMFTN6vEk1+c2z3oe+HWDgEOYwzErZ8Yd2jq3r6 T3/EqaKZZY2frDKlDotITfRZaPAkNUiJpTgj0VCLuag4EQBq/uBgXgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42LZdlhJTvdhqXqSwc1+M4vphxUtmib8ZbaY s2obo8Xqu/1sFitXH2WyeNd6jsWi8/QFJovzbw8zWUw6dI3RYu8tbYv5y56yWyxpPc5mcWPC U0aLNTefslh8PjOP1YHf49nVZ4weO2fdZfdoXnCHxePy2VKPTas62Tw2L6n32H2zgc1j2+KX rB59W1YxenzeJBfAFcVlk5Kak1mWWqRvl8CV8XnZdNaCtWIV92ZdYWpgnCnUxcjJISFgItHw 8QEbiC0ksINRYnmnPERcXKL52g92CFtYYuW/50A2F1DNR0aJxytWADVwcLAJaEpcmFwKUiMi ECBxsPEyWA2zwAwmiZ7mzywgNcICDhLb9guD1LAIqEps/7uaCcTmFbCSePvjPRPEfHmJmZe+ g+3iBIr/vLWVFeIeS4kV636zQdQLSpyc+YQFxGYGqm/eOpt5AqPALCSpWUhSCxiZVjFKphYU 56bnFhsWGOWllusVJ+YWl+al6yXn525iBEeTltYOxj2rPugdYmTiYDzEKMHBrCTCe/+8SpIQ b0piZVVqUX58UWlOavEhRmkOFiVx3gtdJ+OFBNITS1KzU1MLUotgskwcnFINTDnzHrHNPPRl 371nF+coFyStvvSkiPfDChcTMect2xdv+OO49DXrqsXGW6yNVnVaXWVxXcz42v78HJmlWRv3 xh8+3rm1v/pGvFfqYrfTtS/uMGwRX3gznmH6GZVfna0ZYulHCidNj4tdorBHtSCe9dGMo5NE /7lqeE9OWtZex/Mw+Q3Hwb3VjL8NPjCHsGfvX5w+/dTRv+oR/w/Xi02f43eyMcFxS8ScSefP PijPrDKLL+Bck3GnanPIH/bJJZEdioGX5P1/9RgZKJywYqvaeVTu9OY/nyXW9rWtDtkX1PSV 8YSszZIl4jvXvH6bPUEuJrRt3dZlk+68rPklHT7ddNc18dX+XSfPn3F+sWKSyGNPJZbijERD Leai4kQAJ5cZgRUDAAA= X-CMS-MailID: 20220308152729epcas5p17e82d59c68076eb46b5ef658619d65e3 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20220308152729epcas5p17e82d59c68076eb46b5ef658619d65e3 References: <20220308152105.309618-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Anuj Gupta On submission,just fetch the commmand from userspace pointer and reuse everything else. On completion, update the result field inside the passthru command. Signed-off-by: Anuj Gupta Signed-off-by: Kanchan Joshi Signed-off-by: Jens Axboe --- drivers/nvme/host/ioctl.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 701feaecabbe..ddb7e5864be6 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -65,6 +65,14 @@ static void nvme_pt_task_cb(struct io_uring_cmd *ioucmd) } kfree(pdu->meta); + if (ioucmd->flags & IO_URING_F_UCMD_INDIRECT) { + struct nvme_passthru_cmd64 __user *ptcmd64 = ioucmd->cmd; + u64 result = le64_to_cpu(nvme_req(req)->result.u64); + + if (put_user(result, &ptcmd64->result)) + status = -EFAULT; + } + io_uring_cmd_done(ioucmd, status); } @@ -143,6 +151,13 @@ static inline bool nvme_is_fixedb_passthru(struct io_uring_cmd *ioucmd) return ((ioucmd) && (ioucmd->flags & IO_URING_F_UCMD_FIXEDBUFS)); } +static inline bool is_inline_rw(struct io_uring_cmd *ioucmd, struct nvme_command *cmd) +{ + return ((ioucmd->flags & IO_URING_F_UCMD_INDIRECT) || + (cmd->common.opcode == nvme_cmd_write || + cmd->common.opcode == nvme_cmd_read)); +} + static int nvme_submit_user_cmd(struct request_queue *q, struct nvme_command *cmd, u64 ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, @@ -193,8 +208,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, } } if (ioucmd) { /* async dispatch */ - if (cmd->common.opcode == nvme_cmd_write || - cmd->common.opcode == nvme_cmd_read) { + if (is_inline_rw(ioucmd, cmd)) { if (bio && is_polling_enabled(ioucmd, req)) { ioucmd->bio = bio; bio->bi_opf |= REQ_POLLED; @@ -204,7 +218,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, blk_execute_rq_nowait(req, 0, nvme_end_async_pt); return 0; } else { - /* support only read and write for now. */ + /* support only read and write for inline */ ret = -EINVAL; goto out_meta; } @@ -372,7 +386,14 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, } else { if (ioucmd->cmd_len != sizeof(struct nvme_passthru_cmd64)) return -EINVAL; - cptr = (struct nvme_passthru_cmd64 *)ioucmd->cmd; + if (ioucmd->flags & IO_URING_F_UCMD_INDIRECT) { + ucmd = (struct nvme_passthru_cmd64 __user *)ioucmd->cmd; + if (copy_from_user(&cmd, ucmd, sizeof(cmd))) + return -EFAULT; + cptr = &cmd; + } else { + cptr = (struct nvme_passthru_cmd64 *)ioucmd->cmd; + } } if (cptr->flags & NVME_HIPRI) rq_flags |= REQ_POLLED;