From patchwork Sat Apr 29 09:39:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2D05C77B7E for ; Sat, 29 Apr 2023 09:42:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230415AbjD2Jmr (ORCPT ); Sat, 29 Apr 2023 05:42:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230466AbjD2Jmq (ORCPT ); Sat, 29 Apr 2023 05:42:46 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BA90E54 for ; Sat, 29 Apr 2023 02:42:44 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230429094242epoutp0317b5bc5f57d2b03e9eb5ddca77aff11a~aXosU05Ou2943429434epoutp03p for ; Sat, 29 Apr 2023 09:42:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230429094242epoutp0317b5bc5f57d2b03e9eb5ddca77aff11a~aXosU05Ou2943429434epoutp03p DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761362; bh=SAKoOZSg9Ryf3M1Ys01fDubKQ5exxqRHHmS46vJCi0k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FjF9tSTnW4E6oKa2nPcNA0zAkEl/I5VTWviKH+mYQ2pBjJ3+gIGYJWD8ZmBqYeb9H Ih12C44A47xC7+wmscGOav/imZMEZWJ0W8OFxupo27IuxNbyTIf+jVAB9GCP+6sTCO taxXXrQ5Vp9vSakVdmmesxvKbxmSqHRDkAufOdx8= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230429094240epcas5p37d11b6192fc960914c687d756af34a61~aXorLFqJG0334203342epcas5p3B; Sat, 29 Apr 2023 09:42:40 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4Q7kzM5wN4z4x9Pt; Sat, 29 Apr 2023 09:42:39 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 46.50.55646.F86EC446; Sat, 29 Apr 2023 18:42:39 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230429094238epcas5p4efa3dc785fa54ab974852c7f90113025~aXopHYaOv1633616336epcas5p4q; Sat, 29 Apr 2023 09:42:38 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094238epsmtrp10dfd833b65a0679e8f1cbabc11094929~aXopGo8QR0376803768epsmtrp1p; Sat, 29 Apr 2023 09:42:38 +0000 (GMT) X-AuditID: b6c32a4b-b71fa7000001d95e-d5-644ce68fc5b0 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id C4.38.27706.E86EC446; Sat, 29 Apr 2023 18:42:38 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094236epsmtip2c319e9777540e1cfa3f2fc1b390ad953~aXonEwqLw0191301913epsmtip2_; Sat, 29 Apr 2023 09:42:36 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi Subject: [RFC PATCH 01/12] nvme: refactor nvme_alloc_io_tag_set Date: Sat, 29 Apr 2023 15:09:14 +0530 Message-Id: <20230429093925.133327-2-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrOJsWRmVeSWpSXmKPExsWy7bCmpm7/M58Ug9YzOhYfv/5msVh9t5/N 4uaBnUwWK1cfZbJ413qOxeLo/7dsFpMOXWO02HtL22L+sqfsFutev2ex2PT3JJMDt8fOWXfZ Pc7f28jicflsqcemVZ1sHjsfWnpsXlLvsftmA5tH35ZVjB6fN8kFcEZl22SkJqakFimk5iXn p2TmpdsqeQfHO8ebmhkY6hpaWpgrKeQl5qbaKrn4BOi6ZeYAnaqkUJaYUwoUCkgsLlbSt7Mp yi8tSVXIyC8usVVKLUjJKTAp0CtOzC0uzUvXy0stsTI0MDAyBSpMyM7oeTKTsaBFrWLWtqls DYzdCl2MnBwSAiYSN7dtYuxi5OIQEtjNKPF913M2COcTo8SK5mcsEM43Rom/C1qYYVqWbvrF BJHYyyhx49tvVgjnM6NE/9Ut7F2MHBxsApoSFyaXgjSICLhINK2dCjaWWeAio0TPp0MsIAlh AXuJBVfPs4HYLAKqEg0nX4Jt4BWwlJj4+hwTxDZ5iZmXvrOD2JwCVhLfZ+yGqhGUODnzCdgc ZqCa5q2zoa6byyHRekYMwnaR+H7mGTuELSzx6vgWKFtK4mV/G5SdLHFpJsyuEonHew5C2fYS raf6mUF+YQb6Zf0ufYhVfBK9v58wgYQlBHglOtqEIKoVJe5NesoKYYtLPJyxBMr2kHh0tAEa PL3A4D2ymW0Co/wsJB/MQvLBLIRtCxiZVzFKphYU56anFpsWGOellsMjNjk/dxMjOMVqee9g fPTgg94hRiYOxkOMEhzMSiK8vJXuKUK8KYmVValF+fFFpTmpxYcYTYFBPJFZSjQ5H5jk80ri DU0sDUzMzMxMLI3NDJXEedVtTyYLCaQnlqRmp6YWpBbB9DFxcEo1MKV2ipx4fnMmV8jlY/xb PjVZBW78ZXjh5tSUzmwTATF53tkp225MYZf67iyhv8TvX256xIG5y93kr4u+2ns7wzBxiXp0 q2fD+frr3NuOm9n+dpxpYf+ZP+xnskXythkPPd5KJU/4/3BV953ZuexbLRvkXDmUPCUv5RQ1 sYf8Pdd66PECh6P/V8aV/9ozxaFqZbLnijWZ11d+lKk++e95tLy7zubs2D/p83Key21aN2NS wdpf6bPE/jotTtAym872Rf3C8oTG6ccWKmg6fVPbpPypP0Zuje+z2bofWu6Y8Aif5Gndtu6C UvPkdF0dgRmnDLuW7qlRnj73gv2fVc7vVxfFGtjnmz6STotvsHI1nq/EUpyRaKjFXFScCADd 5orHOgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrJLMWRmVeSWpSXmKPExsWy7bCSvG7fM58Ug56dKhYfv/5msVh9t5/N 4uaBnUwWK1cfZbJ413qOxeLo/7dsFpMOXWO02HtL22L+sqfsFutev2ex2PT3JJMDt8fOWXfZ Pc7f28jicflsqcemVZ1sHjsfWnpsXlLvsftmA5tH35ZVjB6fN8kFcEZx2aSk5mSWpRbp2yVw ZfQ8mclY0KJWMWvbVLYGxm6FLkZODgkBE4mlm34xgdhCArsZJb4t8YeIi0s0X/vBDmELS6z8 9xzI5gKq+cgoMWXvC+YuRg4ONgFNiQuTS0FqRAS8JNrfzmIDqWEWuMkosW/3XrBmYQF7iQVX z7OB2CwCqhINJ18yg9i8ApYSE1+fY4JYIC8x89J3sHpOASuJ7zN2g80XAqppXBAPUS4ocXLm ExYQmxmovHnrbOYJjAKzkKRmIUktYGRaxSiZWlCcm55bbFhgmJdarlecmFtcmpeul5yfu4kR HBtamjsYt6/6oHeIkYmD8RCjBAezkggvb6V7ihBvSmJlVWpRfnxRaU5q8SFGaQ4WJXHeC10n 44UE0hNLUrNTUwtSi2CyTBycUg1Mp7NU2qP4VbI/rD7waLLWnKeyGrZx594qx06fHvHT3Wn9 i6lv5wXMDTgafCdohrwl060HCYd2pN3VORD78Y9mmKP0hOfaTz6rh+z2EHr558eZZay5qpz9 s5JY3wZ75Oz3mZje+WJmyZPvByQ73HXeHz/mOtOF/22C9b4kZ//rr19fnTl3QYLlSY4rR7/p mcddDT7C/fBG15kGX45tDyc/cfhrHuc4zbiNS5CnvS7QaL5Bonp38EbRM44utnHO/7UPPzDk VlIo1N7jm/u9tSTx44sIxns9oudzdgW67VDrkb+sX7J+Xr1Qr8Signd3wk/1OS29ptZ5Jqa1 bwObwd6LtcvPLbx9y5jX6WS4greyEktxRqKhFnNRcSIAmcAyTvwCAAA= X-CMS-MailID: 20230429094238epcas5p4efa3dc785fa54ab974852c7f90113025 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094238epcas5p4efa3dc785fa54ab974852c7f90113025 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Allow this function to take nr_hw_queues as a parameter. And change all the callers. This is in preparation to introduce queues which do not have to be registered with block-layer. Signed-off-by: Kanchan Joshi --- drivers/nvme/host/core.c | 5 +++-- drivers/nvme/host/fc.c | 3 ++- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 3 ++- drivers/nvme/host/rdma.c | 2 +- drivers/nvme/host/tcp.c | 3 ++- drivers/nvme/target/loop.c | 3 ++- 7 files changed, 13 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 1bfd52eae2ee..ba476c48d566 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4962,7 +4962,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_admin_tag_set); int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, const struct blk_mq_ops *ops, unsigned int nr_maps, - unsigned int cmd_size) + unsigned int cmd_size, unsigned int nr_hw_queues) { int ret; @@ -4983,9 +4983,10 @@ int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, set->flags |= BLK_MQ_F_BLOCKING; set->cmd_size = cmd_size, set->driver_data = ctrl; - set->nr_hw_queues = ctrl->queue_count - 1; + set->nr_hw_queues = nr_hw_queues - 1; set->timeout = NVME_IO_TIMEOUT; set->nr_maps = nr_maps; + ret = blk_mq_alloc_tag_set(set); if (ret) return ret; diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 456ee42a6133..c7dd084c1999 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2918,7 +2918,8 @@ nvme_fc_create_io_queues(struct nvme_fc_ctrl *ctrl) ret = nvme_alloc_io_tag_set(&ctrl->ctrl, &ctrl->tag_set, &nvme_fc_mq_ops, 1, struct_size((struct nvme_fcp_op_w_sgl *)NULL, priv, - ctrl->lport->ops->fcprqst_priv_sz)); + ctrl->lport->ops->fcprqst_priv_sz), + ctrl->ctrl.queue_count); if (ret) return ret; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..73992dc9dec7 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -748,7 +748,7 @@ int nvme_alloc_admin_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, void nvme_remove_admin_tag_set(struct nvme_ctrl *ctrl); int nvme_alloc_io_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, const struct blk_mq_ops *ops, unsigned int nr_maps, - unsigned int cmd_size); + unsigned int cmd_size, unsigned int nr_hw_queues); void nvme_remove_io_tag_set(struct nvme_ctrl *ctrl); void nvme_remove_namespaces(struct nvme_ctrl *ctrl); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7f25c0fe3a0b..3a38ee6ee129 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -3033,7 +3033,8 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (dev->online_queues > 1) { nvme_alloc_io_tag_set(&dev->ctrl, &dev->tagset, &nvme_mq_ops, - nvme_pci_nr_maps(dev), sizeof(struct nvme_iod)); + nvme_pci_nr_maps(dev), sizeof(struct nvme_iod), + dev->ctrl.queue_count); nvme_dbbuf_set(dev); } diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0eb79696fb73..2e2e131edc62 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -795,7 +795,7 @@ static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl) return nvme_alloc_io_tag_set(ctrl, &to_rdma_ctrl(ctrl)->tag_set, &nvme_rdma_mq_ops, ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, - cmd_size); + cmd_size, ctrl->queue_count); } static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..50efbd724284 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1893,7 +1893,8 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, &nvme_tcp_mq_ops, ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, - sizeof(struct nvme_tcp_request)); + sizeof(struct nvme_tcp_request), + ctrl->queue_count); if (ret) goto out_free_io_queues; } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..f02419b537f7 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -496,7 +496,8 @@ static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl) ret = nvme_alloc_io_tag_set(&ctrl->ctrl, &ctrl->tag_set, &nvme_loop_mq_ops, 1, sizeof(struct nvme_loop_iod) + - NVME_INLINE_SG_CNT * sizeof(struct scatterlist)); + NVME_INLINE_SG_CNT * sizeof(struct scatterlist), + ctrl->ctrl.queue_count); if (ret) goto out_destroy_queues; From patchwork Sat Apr 29 09:39:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8C33C7EE24 for ; Sat, 29 Apr 2023 09:42:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230490AbjD2Jmt (ORCPT ); Sat, 29 Apr 2023 05:42:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230493AbjD2Jmr (ORCPT ); Sat, 29 Apr 2023 05:42:47 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEB3A1FF6 for ; Sat, 29 Apr 2023 02:42:44 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230429094243epoutp02fca536321b0044a6315610f8f370171b~aXotWLr6U1899418994epoutp023 for ; Sat, 29 Apr 2023 09:42:43 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230429094243epoutp02fca536321b0044a6315610f8f370171b~aXotWLr6U1899418994epoutp023 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761363; bh=dRfZRENbFt4jUmmLDCY0GDG76mzOTIYXuzFg+vYW1II=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZXEV3j+xMVSJ9X29CF99Z9/ioyF2o09AA1XPXxI+msiA/OYyZ10M0nCU7gCTTwos1 LllOILUmL72oq9wlJ2ujiheP89Z5kURED05igbLv5cGYl729RnyAc0Ud76ux/o4noV WLtiDX6xMzNGFhTVhy/8V16lRUuMaLDCct10pvnk= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230429094242epcas5p2ffba7cfbd216c2de1cbe71f31e5e0cb2~aXos5fo592162921629epcas5p2E; Sat, 29 Apr 2023 09:42:42 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzP1Z6fz4x9Pv; Sat, 29 Apr 2023 09:42:41 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id FD.6C.54880.196EC446; Sat, 29 Apr 2023 18:42:41 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094240epcas5p1a7411f266412244115411b05da509e4a~aXorEo7hX2856928569epcas5p1j; Sat, 29 Apr 2023 09:42:40 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230429094240epsmtrp27997cc90dadd279ba2f8ea78308208f4~aXorD9o1-3077530775epsmtrp2e; Sat, 29 Apr 2023 09:42:40 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-7e-644ce6916e7d Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 85.38.27706.096EC446; Sat, 29 Apr 2023 18:42:40 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094238epsmtip2189ff0663bb9fdab14e535acca1e77bc~aXopOsFdO0191201912epsmtip2I; Sat, 29 Apr 2023 09:42:38 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi Subject: [RFC PATCH 02/12] pci: enable "raw_queues = N" module parameter Date: Sat, 29 Apr 2023 15:09:15 +0530 Message-Id: <20230429093925.133327-3-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrOJsWRmVeSWpSXmKPExsWy7bCmhu7EZz4pBu8ucVh8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBmVbZORmpiSWqSQmpec n5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+TiE6DrlpkDdKqSQlliTilQKCCxuFhJ386m KL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1xMrQwMDIFKgwITvj3bEfjAUXLCtW7LzH 2MC4V7+LkZNDQsBEomvBcdYuRi4OIYHdjBL/2nayQDifGCV275oF5XxjlGhafQHI4QBrub4i EyK+l1Gi695lqPbPjBIfDm9lBSliE9CUuDC5FGSFiICLRNPaqWwgNcwCFxklej4dYgFJCAt4 SGxp2MQMYrMIqEosPT2PCcTmFbCUeNYynxniPnmJmZe+s4PYnAJWEt9n7GaGqBGUODnzCdgc ZqCa5q2zmUEWSAgs5JDYseoaVLOLxIUjG9ggbGGJV8e3sEPYUhKf3+2FiidLXJp5jgnCLpF4 vOcglG0v0XqqnxnkGWagZ9bv0ofYxSfR+/sJEyQgeCU62oQgqhUl7k16ygphi0s8nLEEyvaQ OH7uFSMkfHoZJf52f2CawCg/C8kLs5C8MAth2wJG5lWMkqkFxbnpqcWmBYZ5qeXwiE3Oz93E CE6xWp47GO8++KB3iJGJg/EQowQHs5IIL2+le4oQb0piZVVqUX58UWlOavEhRlNgGE9klhJN zgcm+bySeEMTSwMTMzMzE0tjM0MlcV5125PJQgLpiSWp2ampBalFMH1MHJxSDUx5Mn8YONqM veO+71yi3td+bsKEP8++JvH236r6cnyq3v1itsp/zZyNr/lETyy2y17E9DIg6se5fndX4S7d sykRvNruDytuXCj79O11YMbUM9eN76xgCLzz/dSNK9qKlaEWS45biR+Lmr3z+G/5PMsF6ZM3 /GFjnPs0OJI9JWr2h029qQe0jhxmtjXYlHKuWuCYiMeFmKIVa0OT169YM9dqZpbktIxrndGm HQnsq/3O5NreOlZ/8JwSj+ztowu+qP6N/i035cHfmR1fXA/m/3EM+cy0+/wUA/uPr2PfXErc ulL9vaDF6Y8FBac3vM1Mn+/Q9uaRoe0J+Wf9yypUHO3DJjtExhWzlrEq+ags5C5XYinOSDTU Yi4qTgQA2KiuFzoEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrBLMWRmVeSWpSXmKPExsWy7bCSvO6EZz4pBiteGlh8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBnFZZOSmpNZllqkb5fA lfHu2A/GgguWFSt23mNsYNyr38XIwSEhYCJxfUVmFyMnh5DAbkaJl7fNQWwJAXGJ5ms/2CFs YYmV/54D2VxANR8ZJR50PmcF6WUT0JS4MLkUpEZEwEui/e0sNpAaZoGbjBL7du8FaxYW8JDY 0rCJGcRmEVCVWHp6HhOIzStgKfGsZT4zxAJ5iZmXvoPVcwpYSXyfsZsZZL4QUE3jgniIckGJ kzOfsIDYzEDlzVtnM09gFJiFJDULSWoBI9MqRsnUguLc9NxiwwLDvNRyveLE3OLSvHS95Pzc TYzgyNDS3MG4fdUHvUOMTByMhxglOJiVRHh5K91ThHhTEiurUovy44tKc1KLDzFKc7AoifNe 6DoZLySQnliSmp2aWpBaBJNl4uCUamDqV540k0+4XPVszeqWxrK1cuk2Pd7z0jedc/bo2Xp3 7Y/gDR3Wifrx63aV9IbW/WsuNGkX4NOb3Bpsq6i8dcG71q3/11QfXlpu8TJt7oeMgr01Niw7 TeP843uk1eyuyZxbWjTt5LJl2bNf+uU+t/fWDj2wTHV6pvJ0qRfh4tHOwu+f86vI5gSX2xj5 b/i9bskqBoZSLye5uxkK7w3FfwVPEz3zelvXCpfSa6mmdXFHbNOvLSmUnn+/6ZENT9ux//fq VpV/zj0X3nlHU8k3pDBxqU/h1H0n4hSX9f1PV16pV7RRUNlYZPrKlht/nvNP8juYLXDu2kfH RGPDjs83PK4t/PJw508plbV/jPrWFyqxFGckGmoxFxUnAgADuJEk+wIAAA== X-CMS-MailID: 20230429094240epcas5p1a7411f266412244115411b05da509e4a X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094240epcas5p1a7411f266412244115411b05da509e4a References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add the infrastructure that carves out N nvme queue-pairs (struct nvme_queue) which are not registered with the block layer. The last N entries in dev->nvmeq[] are available to be attached on demand. Similar to poll_queues, these are interrupt-disabled. This patch does not introduce the interface to attach/detach these queues with any user. That is to be followed in subsequent patches. Signed-off-by: Kanchan Joshi --- drivers/nvme/host/pci.c | 49 +++++++++++++++++++++++++++++------------ 1 file changed, 35 insertions(+), 14 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 3a38ee6ee129..d366a76cc304 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -101,6 +101,10 @@ static unsigned int poll_queues; module_param_cb(poll_queues, &io_queue_count_ops, &poll_queues, 0644); MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO."); +static unsigned int raw_queues; +module_param_cb(raw_queues, &io_queue_count_ops, &raw_queues, 0644); +MODULE_PARM_DESC(raw_queues, "Number of polled, unmanaged queues."); + static bool noacpi; module_param(noacpi, bool, 0444); MODULE_PARM_DESC(noacpi, "disable acpi bios quirks"); @@ -159,6 +163,7 @@ struct nvme_dev { unsigned int nr_allocated_queues; unsigned int nr_write_queues; unsigned int nr_poll_queues; + unsigned int nr_raw_queues; }; static int io_queue_depth_set(const char *val, const struct kernel_param *kp) @@ -209,6 +214,7 @@ struct nvme_queue { #define NVMEQ_SQ_CMB 1 #define NVMEQ_DELETE_ERROR 2 #define NVMEQ_POLLED 3 +#define NVMEQ_RAW 4 __le32 *dbbuf_sq_db; __le32 *dbbuf_cq_db; __le32 *dbbuf_sq_ei; @@ -1599,7 +1605,8 @@ static int nvme_setup_io_queues_trylock(struct nvme_dev *dev) return 0; } -static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled) +static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled, + bool rawq) { struct nvme_dev *dev = nvmeq->dev; int result; @@ -1613,8 +1620,11 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled) */ if (!polled) vector = dev->num_vecs == 1 ? 0 : qid; - else + else { set_bit(NVMEQ_POLLED, &nvmeq->flags); + if (rawq) + set_bit(NVMEQ_RAW, &nvmeq->flags); + } result = adapter_alloc_cq(dev, qid, nvmeq, vector); if (result) @@ -1770,7 +1780,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev) static int nvme_create_io_queues(struct nvme_dev *dev) { - unsigned i, max, rw_queues; + unsigned i, max, rw_queues, rw_poll_queues; int ret = 0; for (i = dev->ctrl.queue_count; i <= dev->max_qid; i++) { @@ -1781,17 +1791,20 @@ static int nvme_create_io_queues(struct nvme_dev *dev) } max = min(dev->max_qid, dev->ctrl.queue_count - 1); - if (max != 1 && dev->io_queues[HCTX_TYPE_POLL]) { + if (max != 1 && + (dev->io_queues[HCTX_TYPE_POLL] || dev->nr_raw_queues)) { rw_queues = dev->io_queues[HCTX_TYPE_DEFAULT] + dev->io_queues[HCTX_TYPE_READ]; + rw_poll_queues = rw_queues + dev->io_queues[HCTX_TYPE_POLL]; } else { rw_queues = max; + rw_poll_queues = max; } - for (i = dev->online_queues; i <= max; i++) { bool polled = i > rw_queues; + bool rawq = i > rw_poll_queues; - ret = nvme_create_queue(&dev->queues[i], i, polled); + ret = nvme_create_queue(&dev->queues[i], i, polled, rawq); if (ret) break; } @@ -2212,7 +2225,7 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) .calc_sets = nvme_calc_irq_sets, .priv = dev, }; - unsigned int irq_queues, poll_queues; + unsigned int irq_queues, poll_queues, raw_queues; /* * Poll queues don't need interrupts, but we need at least one I/O queue @@ -2220,6 +2233,7 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) */ poll_queues = min(dev->nr_poll_queues, nr_io_queues - 1); dev->io_queues[HCTX_TYPE_POLL] = poll_queues; + raw_queues = dev->nr_raw_queues; /* * Initialize for the single interrupt case, will be updated in @@ -2235,7 +2249,7 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) */ irq_queues = 1; if (!(dev->ctrl.quirks & NVME_QUIRK_SINGLE_VECTOR)) - irq_queues += (nr_io_queues - poll_queues); + irq_queues += (nr_io_queues - poll_queues - raw_queues); return pci_alloc_irq_vectors_affinity(pdev, 1, irq_queues, PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY, &affd); } @@ -2248,7 +2262,9 @@ static unsigned int nvme_max_io_queues(struct nvme_dev *dev) */ if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) return 1; - return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues; + + return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues + + dev->nr_raw_queues; } static int nvme_setup_io_queues(struct nvme_dev *dev) @@ -2265,6 +2281,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) */ dev->nr_write_queues = write_queues; dev->nr_poll_queues = poll_queues; + dev->nr_raw_queues = raw_queues; nr_io_queues = dev->nr_allocated_queues - 1; result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues); @@ -2329,7 +2346,8 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) dev->num_vecs = result; result = max(result - 1, 1); - dev->max_qid = result + dev->io_queues[HCTX_TYPE_POLL]; + dev->max_qid = result + dev->io_queues[HCTX_TYPE_POLL] + + dev->nr_raw_queues; /* * Should investigate if there's a performance win from allocating @@ -2356,10 +2374,11 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) nvme_suspend_io_queues(dev); goto retry; } - dev_info(dev->ctrl.device, "%d/%d/%d default/read/poll queues\n", + dev_info(dev->ctrl.device, "%d/%d/%d/%d default/read/poll queues/raw queues\n", dev->io_queues[HCTX_TYPE_DEFAULT], dev->io_queues[HCTX_TYPE_READ], - dev->io_queues[HCTX_TYPE_POLL]); + dev->io_queues[HCTX_TYPE_POLL], + dev->nr_raw_queues); return 0; out_unlock: mutex_unlock(&dev->shutdown_lock); @@ -2457,7 +2476,8 @@ static unsigned int nvme_pci_nr_maps(struct nvme_dev *dev) static void nvme_pci_update_nr_queues(struct nvme_dev *dev) { - blk_mq_update_nr_hw_queues(&dev->tagset, dev->online_queues - 1); + blk_mq_update_nr_hw_queues(&dev->tagset, + dev->online_queues - dev->nr_raw_queues - 1); /* free previously allocated queues that are no longer usable */ nvme_free_queues(dev, dev->online_queues); } @@ -2921,6 +2941,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev, dev->nr_write_queues = write_queues; dev->nr_poll_queues = poll_queues; + dev->nr_raw_queues = raw_queues; dev->nr_allocated_queues = nvme_max_io_queues(dev) + 1; dev->queues = kcalloc_node(dev->nr_allocated_queues, sizeof(struct nvme_queue), GFP_KERNEL, node); @@ -3034,7 +3055,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (dev->online_queues > 1) { nvme_alloc_io_tag_set(&dev->ctrl, &dev->tagset, &nvme_mq_ops, nvme_pci_nr_maps(dev), sizeof(struct nvme_iod), - dev->ctrl.queue_count); + dev->ctrl.queue_count - dev->nr_raw_queues); nvme_dbbuf_set(dev); } From patchwork Sat Apr 29 09:39:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04CAEC77B7E for ; Sat, 29 Apr 2023 09:42:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230497AbjD2Jmu (ORCPT ); Sat, 29 Apr 2023 05:42:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbjD2Jmt (ORCPT ); Sat, 29 Apr 2023 05:42:49 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF1FA1BDA for ; Sat, 29 Apr 2023 02:42:47 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230429094246epoutp02c70e00cd42263cf0fdac4ae04fc5bc4d~aXov3aMJo2139121391epoutp02d for ; Sat, 29 Apr 2023 09:42:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230429094246epoutp02c70e00cd42263cf0fdac4ae04fc5bc4d~aXov3aMJo2139121391epoutp02d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761366; bh=cGKDfl/XthfWKk15QDiG1O0uY0pyS8zVFh/2tUDjwjY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CooSrNQQLUGXRa84Lz00nAf3iTLu431vKL52KxL+WBYW9DkpVXtXO57FVry/CTfRH QDC5hTurcj0ci71s03AKidKqvAwoHZ8PnUWUWrZl75vK9gNFEWV7Td5un00PJNULx5 j3RsMH3zSmMKXzMc5UXR+QAMDzn4H9gUS0f/r+8Q= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230429094245epcas5p18b332f08089ce47aef7cf4c022276bb1~aXovUNIjP2857728577epcas5p1f; Sat, 29 Apr 2023 09:42:45 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q7kzR4r5Cz4x9Pp; Sat, 29 Apr 2023 09:42:43 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 58.50.55646.396EC446; Sat, 29 Apr 2023 18:42:43 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094243epcas5p13be3ca62dc2b03299d09cafaf11923c1~aXotIHsik1041010410epcas5p1k; Sat, 29 Apr 2023 09:42:43 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094243epsmtrp1f27744d43c11881159e6c70a6bf698ea~aXotHXscF0376803768epsmtrp1q; Sat, 29 Apr 2023 09:42:43 +0000 (GMT) X-AuditID: b6c32a4b-913ff7000001d95e-dc-644ce6930268 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 46.38.27706.296EC446; Sat, 29 Apr 2023 18:42:42 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094240epsmtip29cd999aa5565b7db1f8bef0f06413366~aXorL5mwJ0703907039epsmtip2R; Sat, 29 Apr 2023 09:42:40 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Anuj Gupta , Kanchan Joshi Subject: [RFC PATCH 03/12] fs, block: interface to register/unregister the raw-queue Date: Sat, 29 Apr 2023 15:09:16 +0530 Message-Id: <20230429093925.133327-4-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrNJsWRmVeSWpSXmKPExsWy7bCmuu7kZz4pBi8vKll8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZ1S2TUZqYkpq kUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqtkotPgK5bZg7QvUoKZYk5pUChgMTi YiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9vNQSK0MDAyNToMKE7Iwf+1uZC5Zw VSxdq9bAeJaji5GTQ0LARGLynalMXYxcHEICuxklPvzZwQLhfGKUePFqDVTmG6PE8nmbGWFa ei9+ZYNI7GWU+H1xGjuE85lR4vXJ3cxdjBwcbAKaEhcml4I0iAi4SDStnQrWwAwyqWn3XBaQ hLBAmMSMx6vBprIIqErsW/+UHcTmFbCU+LW6kwlim7zEzEvfweKcAlYS32eAzAepEZQ4OfMJ 2BxmoJrmrbOZQRZICCzkkFh6eQtUs4vE7zOz2SBsYYlXx7ewQ9hSEi/726DsZIlLM89B1ZdI PN5zEMq2l2g91Q/2DDPQM+t36UPs4pPo/f2ECSQsIcAr0dEmBFGtKHFv0lNWCFtc4uGMJawQ JR4SR18mQ4Knl1Hi3cUW9gmM8rOQfDALyQezEJYtYGRexSiZWlCcm55abFpgnJdaDo/X5Pzc TYzgVKvlvYPx0YMPeocYmTgYDzFKcDArifDyVrqnCPGmJFZWpRblxxeV5qQWH2I0BQbxRGYp 0eR8YLLPK4k3NLE0MDEzMzOxNDYzVBLnVbc9mSwkkJ5YkpqdmlqQWgTTx8TBKdXAtPzmbwfu pWWHn86Z8LesS/mbpyVXwdHHKZ41AZxhxU6u0devXT95Iqjp759yK9lw5Y3LZD7mCQWtSrO8 U7FHmNt8/2eDzC9fn/EuXswxL5J7odfsl+an74n1PBSsnDv5TO2re8Ezz5SkTGLy7Jg9xbdk j+u69bNvbIr5/toxLXzed+arU+dNXs1X6CBVqTltms4emzyRHi/JiiJzVcVzmRpvmk5cc/5y 2Ffsoph/SbS3tv/+y+Uzxa+XiC/M9sj45mDEkME9/W9Cx8tfzsdnbfacNOsGZ6IRm1DbhrCj u69yvd2tOpf/6IF5Mh3pRyrs+rMUIk4W8P3/sfFZePi5vnINi4XbJfjNRB49f+RyXImlOCPR UIu5qDgRAHcRTy4+BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRmVeSWpSXmKPExsWy7bCSvO6kZz4pBt1rrS0+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFkf/v2WzmHToGqPF3lvaFvOXPWW3WPf6PYvFpr8nmRx4 PHbOusvucf7eRhaPy2dLPTat6mTz2PnQ0mPzknqP3Tcb2Dz6tqxi9Pi8SS6AM4rLJiU1J7Ms tUjfLoEr48f+VuaCJVwVS9eqNTCe5ehi5OSQEDCR6L34la2LkYtDSGA3o0Tviv1MEAlxieZr P9ghbGGJlf+es0MUfWSU2PFtMpDDwcEmoClxYXIpSI2IgJdE+9tZYIOYBf4xSrz/ehlskLBA iMS5eY/ABrEIqErsW/8UzOYVsJT4tboTapm8xMxL38HinAJWEt9n7GYGmS8EVNO4IB6iXFDi 5MwnLCA2M1B589bZzBMYBWYhSc1CklrAyLSKUTK1oDg3PbfYsMAwL7Vcrzgxt7g0L10vOT93 EyM4SrQ0dzBuX/VB7xAjEwfjIUYJDmYlEV7eSvcUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rwX uk7GCwmkJ5akZqemFqQWwWSZODilGpgsk26+f5zl2zxDcVqxv4VBiNmPwnWyjg79L2d7BDlX Lryfwnf/EL/ZvmynaTd6Ot8JVn/7e1lSWqj072H2Vc66z5N35kzmnPUp9Gxap+VdFo5Dmzfs 7X3iJKzFITGhY4VH7/x/9heko/afmV7ZqLbtT+jO73H2h9SmbRGcrDZrcZhu9yXxaGfLEqVt 3sXlXN97hQxzBUOXFuttaJi0WPnFkxipCdvDuNxPak1MOiPhvv/O3JwJOo9ulZ3NYp92uq7v 7NkV/GVGCf3xFXYX1xstXNa9LkH/s6KIm7F/6JkwVZE3Gsn+E4+q/dm75lZl4Oa52naXyt9c nXp97sqk9KSA7/d5w+JvZVef2m0i26bEUpyRaKjFXFScCACTtju/AQMAAA== X-CMS-MailID: 20230429094243epcas5p13be3ca62dc2b03299d09cafaf11923c1 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094243epcas5p13be3ca62dc2b03299d09cafaf11923c1 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Anuj Gupta Introduce register/unregister callbacks to fops and mq_ops, so that callers can attach/detach to raw-queues. Signed-off-by: Anuj Gupta Signed-off-by: Kanchan Joshi --- include/linux/blk-mq.h | 2 ++ include/linux/fs.h | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 06caacd77ed6..7d6790be4847 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -657,6 +657,8 @@ struct blk_mq_ops { * information about a request. */ void (*show_rq)(struct seq_file *m, struct request *rq); + int (*register_queue)(void *data); + int (*unregister_queue)(void *data, int qid); #endif }; diff --git a/include/linux/fs.h b/include/linux/fs.h index 475d88640d3d..79acccc5e7d4 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1798,6 +1798,8 @@ struct file_operations { int (*uring_cmd)(struct io_uring_cmd *ioucmd, unsigned int issue_flags); int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *, unsigned int poll_flags); + int (*register_queue)(struct file *f); + int (*unregister_queue)(struct file *f, int qid); } __randomize_layout; struct inode_operations { From patchwork Sat Apr 29 09:39:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA42C77B7E for ; Sat, 29 Apr 2023 09:42:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231152AbjD2Jm4 (ORCPT ); Sat, 29 Apr 2023 05:42:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230507AbjD2Jmw (ORCPT ); Sat, 29 Apr 2023 05:42:52 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 66C281BEC for ; Sat, 29 Apr 2023 02:42:50 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230429094248epoutp01d6342258b0b755237185e0d9185af60d~aXoyWAPLs2540625406epoutp015 for ; Sat, 29 Apr 2023 09:42:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230429094248epoutp01d6342258b0b755237185e0d9185af60d~aXoyWAPLs2540625406epoutp015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761368; bh=4c/Krci4Z4CXyKaTGDw8RdWSrdEiCYIykDkltCWySjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=duwFNA/qyM9HNzlo4LfgsgAoIsRuRtjRvVcbsZq4HNpqKehl/66uyWHbSvMiflbq5 cCVp6hhO/Eec4vMdjrSPwUvkbw8MLt3wRScl/VXTiBFArFmcZh+ALaBejbi22L8joe yXZS+F4uCAaj8SmlnMYHtaDXe/jdc0zTgRvVOs9o= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230429094247epcas5p1dd95493cde0ef020ba1761d668b90591~aXoxUTG3c2857728577epcas5p1i; Sat, 29 Apr 2023 09:42:47 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Q7kzT75Tlz4x9Pp; Sat, 29 Apr 2023 09:42:45 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 3B.50.55646.596EC446; Sat, 29 Apr 2023 18:42:45 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230429094245epcas5p2843abc5cd54ffe301d36459543bcd228~aXovdMmAn2071420714epcas5p26; Sat, 29 Apr 2023 09:42:45 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094245epsmtrp1bd351d6ec504f4a91026fa26bbb47fd8~aXovcaQtH0376803768epsmtrp1r; Sat, 29 Apr 2023 09:42:45 +0000 (GMT) X-AuditID: b6c32a4b-913ff7000001d95e-e4-644ce69529c2 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 17.38.27706.596EC446; Sat, 29 Apr 2023 18:42:45 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094243epsmtip25ece63685e3de64d93e5fa79dc9c9281~aXotNyDAy0191201912epsmtip2J; Sat, 29 Apr 2023 09:42:43 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Anuj Gupta , Kanchan Joshi Subject: [RFC PATCH 04/12] io_uring, fs: plumb support to register/unregister raw-queue Date: Sat, 29 Apr 2023 15:09:17 +0530 Message-Id: <20230429093925.133327-5-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrDJsWRmVeSWpSXmKPExsWy7bCmpu7UZz4pBqt7zS0+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFkf/v2WzmHToGqPF3lvaFvOXPWW3WPf6PYvFpr8nmRx4 PHbOusvucf7eRhaPy2dLPTat6mTz2PnQ0mPzknqP3Tcb2Dz6tqxi9Pi8SS6AMyrbJiM1MSW1 SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VWycUnQNctMwfoXiWFssScUqBQQGJx sZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpeXmqJlaGBgZEpUGFCdsbHBftYC/5q V7x58I6xgbFLpYuRk0NCwETiRdNN5i5GLg4hgd2MEvu3nWOFcD4xSjzYtZYRwvnGKLHuzyUm mJaOHatZIBJ7GSXeP2tiA0kICXxmlHh+I6qLkYODTUBT4sLkUpCwiICLRNPaqWwg9cwgg5p2 z2UBSQgLREp0/GxnBrFZBFQlZh9pAovzClhKbFj7gB1imbzEzEvfwWxOASuJ7zN2M0PUCEqc nPkErJ4ZqKZ562ywHyQElnJI3D5wmxWi2UViUedMFghbWOLV8S1QQ6UkXva3QdnJEpdmnoP6 rETi8Z6DULa9ROupfmaQZ5iBnlm/Sx9iF59E7+8nTCBhCQFeiY42IYhqRYl7k55CbRWXeDhj CZTtIfH1zV92SFj1Mkr8+vOdcQKj/CwkL8xC8sIshG0LGJlXMUqmFhTnpqcWmxYY56WWwyM2 OT93EyM42Wp572B89OCD3iFGJg7GQ4wSHMxKIry8le4pQrwpiZVVqUX58UWlOanFhxhNgWE8 kVlKNDkfmO7zSuINTSwNTMzMzEwsjc0MlcR51W1PJgsJpCeWpGanphakFsH0MXFwSjUwbRMz 2xshc1rmWeKvTbc7FOY2GXdJ1GwwfTBfTLPXdn4iy0S7HRuuL/13aJfW9L6b7j7VO57PZ9w2 fbuu95GXustelcyraREXipPqmufQvT7SfhmPaNXV4imRuw74pl6v8Sn4VcZ16U/ctXwtkUqV x6G7Fmn/dT7zYYLhawmbv5oJi8ytNHUPbDu7dlKB8LV7i1euf35uJ0u6/8NMw/cL3R6rcwVW rCxQaWNX4NaV2ORzhJ1b1iHlkkpn2sbHMtKXVOSzTJiutLQVLpmtbf2wKqj8pvX7hK7kOsWI 3vmbTp6cs+rExGO6cyqD+C5/co44VnDmteq3T9umN4dKVn0N2nCPfcW7qFXv3h6VYVq0Woml OCPRUIu5qDgRACfhNU8/BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsWy7bCSvO7UZz4pBg/3SVp8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZxSXTUpqTmZZ apG+XQJXxscF+1gL/mpXvHnwjrGBsUuli5GTQ0LARKJjx2qWLkYuDiGB3YwSZ7dsZINIiEs0 X/vBDmELS6z895wdougjo8TWU88Zuxg5ONgENCUuTC4FqRER8JJofzuLDaSGWeAfo8T7r5eZ QBLCAuESfzdfBhvKIqAqMftIEwuIzStgKbFh7QOoBfISMy99B7M5Bawkvs/YzQwyXwiopnFB PES5oMTJmU/AWpmBypu3zmaewCgwC0lqFpLUAkamVYySqQXFuem5xYYFhnmp5XrFibnFpXnp esn5uZsYwXGipbmDcfuqD3qHGJk4GA8xSnAwK4nw8la6pwjxpiRWVqUW5ccXleakFh9ilOZg URLnvdB1Ml5IID2xJDU7NbUgtQgmy8TBKdXAZBa2cG7+oQRW7qDoaC3ekg8nTrNn8P7laYx9 uIV94VvrB4KylzPruFl0961b/G7J4q6r9+6ds2OaG+V1cbpD55G1c289+CJjUm0Wlb9cK3Ld +7n/Ga402zvemcW7bkWeypMC0TU9TPZL5x05yMTMLGt2KNeqynrbxM8R4VJHdZZ/usi2ze/9 drX2zCONd7kyS3pNSp/qV7d9YBfj7wnUynshzD0hrt72lNSbr68WmO2V2yF43IL1Y0SBk7Nv kNLU1f6Bt0qmKZfoMnHKeO/d+uJ1Ul3UHj+zY6oMCpvu/3GvOSDBpnpybth1ibMvywM/hpha BszedOmAZYa3s6nB/cOl0fxs3o8vxyxWzBBXYinOSDTUYi4qTgQADox1lAIDAAA= X-CMS-MailID: 20230429094245epcas5p2843abc5cd54ffe301d36459543bcd228 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094245epcas5p2843abc5cd54ffe301d36459543bcd228 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Anuj Gupta Extend io_uring's registration interface with - IORING_REGISTER_QUEUE: to ask for a queue. It goes down via fops->register_queue and returns identifier of the queue. This qid is stored in ring's ctx. - IORING_UNREGISTER_QUEUE: to return the previously registered queue. At max one queue is allowed to be attached with the io_uring's ring. The file for which queue is requested is expected to be in registered file-set. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- fs/file.c | 14 ++++++++ include/linux/fs.h | 2 ++ include/linux/io_uring_types.h | 3 ++ include/uapi/linux/io_uring.h | 4 +++ io_uring/io_uring.c | 60 ++++++++++++++++++++++++++++++++++ 5 files changed, 83 insertions(+) diff --git a/fs/file.c b/fs/file.c index 7893ea161d77..7dada9cd0911 100644 --- a/fs/file.c +++ b/fs/file.c @@ -1318,3 +1318,17 @@ int iterate_fd(struct files_struct *files, unsigned n, return res; } EXPORT_SYMBOL(iterate_fd); + +int file_register_queue(struct file *file) +{ + if (file->f_op->register_queue) + return file->f_op->register_queue(file); + return -EINVAL; +} + +int file_unregister_queue(struct file *file, int qid) +{ + if (file->f_op->unregister_queue) + return file->f_op->unregister_queue(file, qid); + return -EINVAL; +} diff --git a/include/linux/fs.h b/include/linux/fs.h index 79acccc5e7d4..0a82aac6868b 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3190,5 +3190,7 @@ extern int vfs_fadvise(struct file *file, loff_t offset, loff_t len, int advice); extern int generic_fadvise(struct file *file, loff_t offset, loff_t len, int advice); +int file_register_queue(struct file *file); +int file_unregister_queue(struct file *file, int qid); #endif /* _LINUX_FS_H */ diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 1b2a20a42413..8d4e721493d6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -364,6 +364,9 @@ struct io_ring_ctx { unsigned sq_thread_idle; /* protected by ->completion_lock */ unsigned evfd_last_cq_tail; + /* for io_uring attached device queue */ + int dev_qid; + int dev_fd; }; struct io_tw_state { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 0716cb17e436..a9d59bfd26f7 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -523,6 +523,10 @@ enum { /* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE = 25, + /* register a device-queue with the ring */ + IORING_REGISTER_QUEUE = 26, + IORING_UNREGISTER_QUEUE = 27, + /* this goes last */ IORING_REGISTER_LAST, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3bca7a79efda..5a9b7adf438e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -337,6 +337,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_WQ_LIST(&ctx->locked_free_list); INIT_DELAYED_WORK(&ctx->fallback_work, io_fallback_req_func); INIT_WQ_LIST(&ctx->submit_state.compl_reqs); + /* -EINVAL implies nothing is registered with this ring */ + ctx->dev_qid = -EINVAL; return ctx; err: kfree(ctx->dummy_ubuf); @@ -2822,6 +2824,51 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) mutex_unlock(&ctx->uring_lock); } +static int io_register_queue(struct io_ring_ctx *ctx, void __user *arg) +{ + struct io_fixed_file *file_slot; + struct file *file; + __s32 __user *fds = arg; + int fd, qid; + + if (ctx->dev_qid != -EINVAL) + return -EINVAL; + if (copy_from_user(&fd, fds, sizeof(*fds))) + return -EFAULT; + file_slot = io_fixed_file_slot(&ctx->file_table, + array_index_nospec(fd, ctx->nr_user_files)); + if (!file_slot->file_ptr) + return -EBADF; + file = (struct file *)(file_slot->file_ptr & FFS_MASK); + qid = file_register_queue(file); + if (qid < 0) + return qid; + ctx->dev_fd = fd; + ctx->dev_qid = qid; + return 0; +} + +static int io_unregister_queue(struct io_ring_ctx *ctx) +{ + struct io_fixed_file *file_slot; + struct file *file; + int ret; + + if (ctx->dev_qid == -EINVAL) + return 0; + file_slot = io_fixed_file_slot(&ctx->file_table, + array_index_nospec(ctx->dev_fd, ctx->nr_user_files)); + if (!file_slot) + return -EBADF; + if (!file_slot->file_ptr) + return -EBADF; + file = (struct file *)(file_slot->file_ptr & FFS_MASK); + ret = file_unregister_queue(file, ctx->dev_qid); + if (!ret) + ctx->dev_qid = -EINVAL; + return ret; +} + static void io_rsrc_node_cache_free(struct io_cache_entry *entry) { kfree(container_of(entry, struct io_rsrc_node, cache)); @@ -2835,6 +2882,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) return; mutex_lock(&ctx->uring_lock); + io_unregister_queue(ctx); if (ctx->buf_data) __io_sqe_buffers_unregister(ctx); if (ctx->file_data) @@ -4418,6 +4466,18 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_file_alloc_range(ctx, arg); break; + case IORING_REGISTER_QUEUE: + ret = -EINVAL; + if (!arg || nr_args != 1) + break; + ret = io_register_queue(ctx, arg); + break; + case IORING_UNREGISTER_QUEUE: + ret = -EINVAL; + if (arg || nr_args) + break; + ret = io_unregister_queue(ctx); + break; default: ret = -EINVAL; break; From patchwork Sat Apr 29 09:39:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AB5FC7EE23 for ; Sat, 29 Apr 2023 09:43:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230507AbjD2JnA (ORCPT ); Sat, 29 Apr 2023 05:43:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230466AbjD2Jmz (ORCPT ); Sat, 29 Apr 2023 05:42:55 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C18B010D7 for ; Sat, 29 Apr 2023 02:42:51 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230429094250epoutp039c17f37de41277dc7d3f58c8e4bb3e81~aXozuIA7m2943429434epoutp03u for ; Sat, 29 Apr 2023 09:42:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230429094250epoutp039c17f37de41277dc7d3f58c8e4bb3e81~aXozuIA7m2943429434epoutp03u DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761370; bh=j3RUN5D5cpuMgVbInnVIvXQH9V/4ewHPlZPhTRvKD5s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vAXsULS9IJXjdp6dVdsU7pY9QucKNYevMPj3Duxc/TQcBbnqnT4536B2JafLdl9Ri vEbw4LXJLs+XqztTzUCqneBmGyRw8lNinBdyjszX8ZTdr4iNQPMA6MSSBtENIG3U3F hQ68siPLOQ2VoIVBbNRvpIL8B6YXlLQi4SC+fpiM= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230429094249epcas5p1857244999a0b36c8f611849df577d598~aXoy8tr851040910409epcas5p1q; Sat, 29 Apr 2023 09:42:49 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.183]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzX1cX6z4x9Pq; Sat, 29 Apr 2023 09:42:48 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D0.7C.54880.896EC446; Sat, 29 Apr 2023 18:42:48 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230429094247epcas5p333e0f515000de60fb64dc2590cf9fcd8~aXoxThIqq2931829318epcas5p3l; Sat, 29 Apr 2023 09:42:47 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094247epsmtrp104154e422aebf5b2f558085e93778bda~aXoxSxzyo0376803768epsmtrp1t; Sat, 29 Apr 2023 09:42:47 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-8c-644ce6986599 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 73.29.28392.796EC446; Sat, 29 Apr 2023 18:42:47 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094245epsmtip2cde3811e2454169712d6ee8949816689~aXovlzz5Z0703907039epsmtip2S; Sat, 29 Apr 2023 09:42:45 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Anuj Gupta , Kanchan Joshi Subject: [RFC PATCH 05/12] nvme: wire-up register/unregister queue f_op callback Date: Sat, 29 Apr 2023 15:09:18 +0530 Message-Id: <20230429093925.133327-6-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrJJsWRmVeSWpSXmKPExsWy7bCmuu6MZz4pBn1TOCw+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFkf/v2WzmHToGqPF3lvaFvOXPWW3WPf6PYvFpr8nmRx4 PHbOusvucf7eRhaPy2dLPTat6mTz2PnQ0mPzknqP3Tcb2Dz6tqxi9Pi8SS6AMyrbJiM1MSW1 SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VWycUnQNctMwfoXiWFssScUqBQQGJx sZK+nU1RfmlJqkJGfnGJrVJqQUpOgUmBXnFibnFpXrpeXmqJlaGBgZEpUGFCdsbkaUEFk9Uq rnzdyd7AOE+hi5GTQ0LARGLrh1ZmEFtIYDejxN0NRl2MXED2J0aJd/fOM0E4nxklpnx6wQ7T sfJkDxtEYhejxNybT5jhqk78v87axcjBwSagKXFhcilIg4iAi0TT2qlgDcwC3xglmnbPZQGp ERYIkuiZIQtSwyKgKnF65xEmEJtXwFJi1d+DzBDL5CVmXvoOtphTwEri+4zdzBA1ghInZz5h AbGZgWqat86Gql/KIfH7kRCE7SKxouMVG4QtLPHq+BaoB6QkXva3QdnJEpdmnmOCsEskHu85 CGXbS7Se6mcGOZMZ6JX1u/QhVvFJ9P5+wgQSlhDglehog9qkKHFv0lNWCFtc4uGMJVC2h8Sj N6ugQdXLKLFg6yH2CYzys5B8MAvJB7MQti1gZF7FKJlaUJybnlpsWmCYl1oOj9Xk/NxNjOA0 q+W5g/Hugw96hxiZOBgPMUpwMCuJ8PJWuqcI8aYkVlalFuXHF5XmpBYfYjQFBvFEZinR5Hxg os8riTc0sTQwMTMzM7E0NjNUEudVtz2ZLCSQnliSmp2aWpBaBNPHxMEp1cCUt+f0pmWbH+/y TQkL/T/zR7HJO919O6xiD+wqFnkX9HmtgnbAnNj1XQu/ltSWHYh5t8Tc37hQ/r2/35qy4ryz 4o9kpnedXffN+pCNdEF4USj7fmP/xfEXNtpPMDP8eaRr3cMTD3lbIl+4a+++XqnPO7n+yZmZ XreqtFcra9znNrn1sFWuPfDNno3VraofhJz+XKs6dKXnUf1+rsYbzLydbm9PvT2Z7Olk6Py9 XUzy7tNzMZeaXxxhWfD49tUt4ff8Z0VqLe5bqH5+x/x1xippFTeB/lk2l8Uv/bbJb8t/nDYP VQP+LLd3O2er997/ocWkm3sbNR/OW/BBbU/0wrNpjTzTzKZlO2RuPZn2QZNZiaU4I9FQi7mo OBEA9oMVuzwEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDLMWRmVeSWpSXmKPExsWy7bCSvO70Zz4pBts/S1l8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZxSXTUpqTmZZ apG+XQJXxuRpQQWT1SqufN3J3sA4T6GLkZNDQsBEYuXJHjYQW0hgB6PEsytKEHFxieZrP9gh bGGJlf+eA9lcQDUfGSU2PjzP2MXIwcEmoClxYXIpSI2IgJdE+9tZbCA1zAL/GCXef73MBFIj LBAgsbrfAaSGRUBV4vTOI0wgNq+ApcSqvweZIebLS8y89B1sF6eAlcT3GbuZQVqFgGoaF8RD lAtKnJz5hAXEZgYqb946m3kCo8AsJKlZSFILGJlWMUqmFhTnpucWGxYY5aWW6xUn5haX5qXr JefnbmIER4iW1g7GPas+6B1iZOJgPMQowcGsJMLLW+meIsSbklhZlVqUH19UmpNafIhRmoNF SZz3QtfJeCGB9MSS1OzU1ILUIpgsEwenVAMT53f57vnT3B+ah0SdXyV4dOKf4O3HK8qmC7vd duMUnyF2KSLpeNrE4olsglxX9r+Ot53KzP9Zg+fLI4MZ+9urRayOxrS9ZzLm+Xzuxj6rM/0L HPKeP37uVmQ3963kytfTOu2M7D6fPWd0TW/Nj0+zyyvu8C/8utyx7hPr6wNRdy1YDXevNFE/ WeXvlHjPteOj5nMBh1/+98pXuV6adrD4A+O9h0uDuZi2ye/kP5xnsGH+Rplq6YBPCXlLVsYL 3zBNXZDWVhLb7Wz393/CvJuTUjYIPt3gPOuvrtqJpJ2PvsbvVXH+rDm7TSwm0r/aZIVA+kmp xsfbbbjkgi+82LLV8sbfw5OFs7MEJcryG1YbKLEUZyQaajEXFScCAGoRqoL/AgAA X-CMS-MailID: 20230429094247epcas5p333e0f515000de60fb64dc2590cf9fcd8 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094247epcas5p333e0f515000de60fb64dc2590cf9fcd8 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Anuj Gupta Implement register/unregister handlers for char-device file. Signed-off-by: Anuj Gupta Signed-off-by: Kanchan Joshi --- drivers/nvme/host/core.c | 26 +++++++++++++++++++ drivers/nvme/host/ioctl.c | 48 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/multipath.c | 2 ++ drivers/nvme/host/nvme.h | 2 ++ 4 files changed, 78 insertions(+) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index ba476c48d566..4462ce50d076 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4059,6 +4059,30 @@ static int nvme_ns_chr_release(struct inode *inode, struct file *file) return 0; } +int nvme_register_queue(struct file *file) +{ + struct nvme_ns *ns = container_of(file_inode(file)->i_cdev, + struct nvme_ns, cdev); + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = ns ? ns->queue : ctrl->admin_q; + + if (q->mq_ops && q->mq_ops->register_queue) + return q->mq_ops->register_queue(ns); + return -EINVAL; +} + +int nvme_unregister_queue(struct file *file, int qid) +{ + struct nvme_ns *ns = container_of(file_inode(file)->i_cdev, + struct nvme_ns, cdev); + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = ns ? ns->queue : ctrl->admin_q; + + if (q->mq_ops && q->mq_ops->unregister_queue) + return q->mq_ops->unregister_queue(ns, qid); + return -EINVAL; +} + static const struct file_operations nvme_ns_chr_fops = { .owner = THIS_MODULE, .open = nvme_ns_chr_open, @@ -4067,6 +4091,8 @@ static const struct file_operations nvme_ns_chr_fops = { .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_ns_chr_uring_cmd, .uring_cmd_iopoll = nvme_ns_chr_uring_cmd_iopoll, + .register_queue = nvme_register_queue, + .unregister_queue = nvme_unregister_queue, }; static int nvme_add_ns_cdev(struct nvme_ns *ns) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index d24ea2e05156..292a578686b6 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -871,6 +871,54 @@ long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, return ret; } +int nvme_ns_head_register_queue(struct file *file) +{ + struct cdev *cdev = file_inode(file)->i_cdev; + struct nvme_ns_head *head = + container_of(cdev, struct nvme_ns_head, cdev); + struct nvme_ns *ns; + struct nvme_ctrl *ctrl; + struct request_queue *q; + int srcu_idx, ret = -EINVAL; + + srcu_idx = srcu_read_lock(&head->srcu); + ns = nvme_find_path(head); + if (!ns) + goto out_unlock; + + ctrl = ns->ctrl; + q = ns ? ns->queue : ctrl->admin_q; + if (q->mq_ops && q->mq_ops->register_queue) + ret = q->mq_ops->register_queue(ns); +out_unlock: + srcu_read_unlock(&head->srcu, srcu_idx); + return ret; +} + +int nvme_ns_head_unregister_queue(struct file *file, int qid) +{ + struct cdev *cdev = file_inode(file)->i_cdev; + struct nvme_ns_head *head = + container_of(cdev, struct nvme_ns_head, cdev); + struct nvme_ns *ns; + struct nvme_ctrl *ctrl; + struct request_queue *q; + int srcu_idx, ret = -EINVAL; + + srcu_idx = srcu_read_lock(&head->srcu); + ns = nvme_find_path(head); + if (!ns) + goto out_unlock; + + ctrl = ns->ctrl; + q = ns ? ns->queue : ctrl->admin_q; + if (q->mq_ops && q->mq_ops->unregister_queue) + ret = q->mq_ops->unregister_queue(ns, qid); +out_unlock: + srcu_read_unlock(&head->srcu, srcu_idx); + return ret; +} + int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) { diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 9171452e2f6d..eed30daf1a37 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -471,6 +471,8 @@ static const struct file_operations nvme_ns_head_chr_fops = { .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_ns_head_chr_uring_cmd, .uring_cmd_iopoll = nvme_ns_head_chr_uring_cmd_iopoll, + .register_queue = nvme_ns_head_register_queue, + .unregister_queue = nvme_ns_head_unregister_queue, }; static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 73992dc9dec7..4619a7498f8e 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -843,6 +843,8 @@ int nvme_ns_head_ioctl(struct block_device *bdev, fmode_t mode, unsigned int cmd, unsigned long arg); long nvme_ns_head_chr_ioctl(struct file *file, unsigned int cmd, unsigned long arg); +int nvme_ns_head_register_queue(struct file *file); +int nvme_ns_head_unregister_queue(struct file *file, int qid); long nvme_dev_ioctl(struct file *file, unsigned int cmd, unsigned long arg); int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, From patchwork Sat Apr 29 09:39:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B8D0C77B60 for ; Sat, 29 Apr 2023 09:43:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230495AbjD2JnA (ORCPT ); Sat, 29 Apr 2023 05:43:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231157AbjD2Jm4 (ORCPT ); Sat, 29 Apr 2023 05:42:56 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC2521FFF for ; Sat, 29 Apr 2023 02:42:53 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230429094251epoutp030efd0131708c7187dd7a1fde5e4c3462~aXo1V0SqN2944229442epoutp03n for ; Sat, 29 Apr 2023 09:42:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230429094251epoutp030efd0131708c7187dd7a1fde5e4c3462~aXo1V0SqN2944229442epoutp03n DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761371; bh=DmNoMnErFd30BsWQmdnQWonqLi5WPKhdAEqCyccyvuU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ix8x4lb+rtZ0ZDeaQ8HhIP/Ky0T5I+S6Uy5+BDaYY9j6UbBB+VYMDYOgyQ1a/jk/+ ID79GU7hrYDJ6WUvs13u2Y0z1k7O63nVzgvxwUtqh6cIRnp4jWlf5gBeP5odWicLu8 AE4HO7/PJsNhU8gQEBM79s3x5zrrCk32AUUOMWqI= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230429094251epcas5p127889f83c801d6f9aee5fbe5a4da0ba6~aXo04yuRV2857728577epcas5p1m; Sat, 29 Apr 2023 09:42:51 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzZ1T9rz4x9Pq; Sat, 29 Apr 2023 09:42:50 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 53.7C.54880.A96EC446; Sat, 29 Apr 2023 18:42:50 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094249epcas5p18bd717f4e34077c0fcf28458f11de8d1~aXozX1JN72857728577epcas5p1l; Sat, 29 Apr 2023 09:42:49 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230429094249epsmtrp21015735b632716c0a8a450abd4bc8a8a~aXozXJwWV0329503295epsmtrp2Q; Sat, 29 Apr 2023 09:42:49 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-90-644ce69a7f9c Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E7.38.27706.996EC446; Sat, 29 Apr 2023 18:42:49 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094247epsmtip294491de44997f253a7fda773fafa9aa2~aXoxX4sIF0191201912epsmtip2K; Sat, 29 Apr 2023 09:42:47 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi , Anuj Gupta Subject: [RFC PATCH 06/12] pci: implement register/unregister functionality Date: Sat, 29 Apr 2023 15:09:19 +0530 Message-Id: <20230429093925.133327-7-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrNJsWRmVeSWpSXmKPExsWy7bCmlu6sZz4pBrumsVp8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZ1S2TUZqYkpq kUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqtkotPgK5bZg7QvUoKZYk5pUChgMTi YiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9vNQSK0MDAyNToMKE7Iwzbe9ZCo7r VUzZKdzA+Ey1i5GTQ0LAROLKu7ssXYxcHEICuxklTq1ZxAbhfGKUuLDyEzOE841RYsbiYyww LT9fv4Vq2cso0Xr3BCOE8xmopb+TtYuRg4NNQFPiwuRSkAYRAReJprVTwcYyg0xa9PodK0hC WMBbYtGhc2A2i4CqxJ8Fs8FsXgFLif5JmxkhtslLzLz0nR3E5hSwkvg+YzczRI2gxMmZT8Au Ygaqad46G+xUCYGlHBIH302BOtVFomvFTChbWOLV8S3sELaUxOd3e9kg7GSJSzPPMUHYJRKP 9xyEsu0lWk/1M4M8wwz0zPpd+hC7+CR6fz9hAglLCPBKdLQJQVQrStyb9JQVwhaXeDhjCZTt IfH61yMwW0igl1FiyQWZCYzys5B8MAvJB7MQli1gZF7FKJlaUJybnlpsWmCYl1oOj9fk/NxN jOBUq+W5g/Hugw96hxiZOBgPMUpwMCuJ8PJWuqcI8aYkVlalFuXHF5XmpBYfYjQFBvFEZinR 5Hxgss8riTc0sTQwMTMzM7E0NjNUEudVtz2ZLCSQnliSmp2aWpBaBNPHxMEp1cA0wdNkooL6 pdXLG33rnD1vTv+wqjf+emOKx3vmSzMyt3xTyWN138Wixfj0k8+RV1NWsubecfq8N2R568Pi r4vPdIo/vH2v/Lm4jRaPpMWKmJpjC5QjdgWsEDe0XZHP2Krg6PtZ9bPVnhmKpXeXZ4SejRDi 4/10xUNINeV36aqTr6Qt3TbV+F3tfGdhMrl87sSvjVwp+hYawncaZxxd+Pbqqv16QlesLiUx psewuIu+/e70vKr+3RGP++m6k0paTHhYmV0Cn13OzLqxQOPFNk2Ja3OP7g7/V5mxuHLjAZ2H ObtWs/yTvjrvjNn5P+mHsnWDlFaX1/TlH7xi5zjj56fO5dt/ZHbeXbfg7A7miSKMSizFGYmG WsxFxYkA9VRJCT4EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRmVeSWpSXmKPExsWy7bCSvO7MZz4pBk1z1Sw+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFkf/v2WzmHToGqPF3lvaFvOXPWW3WPf6PYvFpr8nmRx4 PHbOusvucf7eRhaPy2dLPTat6mTz2PnQ0mPzknqP3Tcb2Dz6tqxi9Pi8SS6AM4rLJiU1J7Ms tUjfLoEr40zbe5aC43oVU3YKNzA+U+1i5OSQEDCR+Pn6LUsXIxeHkMBuRonpB7exQCTEJZqv /WCHsIUlVv57zg5R9JFRYubT68xdjBwcbAKaEhcml4LUiAh4SbS/ncUGUsMs8I9R4sH7NjaQ hLCAt8SiQ+dYQWwWAVWJPwtmg9m8ApYS/ZM2M0IskJeYeek72DJOASuJ7zN2g80XAqppXBAP US4ocXLmE7DbmIHKm7fOZp7AKDALSWoWktQCRqZVjJKpBcW56bnFhgWGeanlesWJucWleel6 yfm5mxjBUaKluYNx+6oPeocYmTgYDzFKcDArifDyVrqnCPGmJFZWpRblxxeV5qQWH2KU5mBR Eue90HUyXkggPbEkNTs1tSC1CCbLxMEp1cB0bIk9J49dgbDp05N5q/oOac6zmln1rEjhTl9N emrQoRMHuFsd9bY9kFmsZdvbqrOqb8PB6hVaFSYXpzAvk0ualDjXylj1VvLbG/tMKxdMept2 M1qSX3CPcVve8raI8P/zW1RmrTfTY72ncG7+glDn8+G+909orDzJb/j9u9OtdywfZ1y5/5/j 1jIDg3zWfWsN50z4tOjjjO57X/d8aMj9WsZhfkDM5eoDnkfPlljGCklfWXCCYb+Ki+iiaqG/ e8yFpj9yrZWtmie7IKU83MRx9v/utcpFxROMTPIZdgYWXVO980TmwtYEwf0cMgE8N/zeT+G9 33+sRkru35385Sxy2t9d17x/LBMltar/9Mk6JZbijERDLeai4kQAGzGqLgEDAAA= X-CMS-MailID: 20230429094249epcas5p18bd717f4e34077c0fcf28458f11de8d1 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094249epcas5p18bd717f4e34077c0fcf28458f11de8d1 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Implement the register callback. It checks if any raw-queue is available to be attached. If found, it returns the qid. During queue registration, iod and command-id bitmap are also preallocated. Unregister callback does the opposite and returns the corresponding queue to the pool of available raw-queues. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- drivers/nvme/host/pci.c | 154 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index d366a76cc304..b4498e198e8a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -116,6 +116,15 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown); static void nvme_delete_io_queues(struct nvme_dev *dev); static void nvme_update_attrs(struct nvme_dev *dev); +enum { + Q_FREE, + Q_ALLOC +}; +struct rawq_info { + int nr_free; + int q_state[]; +}; + /* * Represents an NVM Express device. Each nvme_dev is a PCI function. */ @@ -164,6 +173,8 @@ struct nvme_dev { unsigned int nr_write_queues; unsigned int nr_poll_queues; unsigned int nr_raw_queues; + struct mutex rawq_lock; + struct rawq_info *rawqi; }; static int io_queue_depth_set(const char *val, const struct kernel_param *kp) @@ -195,6 +206,10 @@ struct nvme_queue { struct nvme_dev *dev; spinlock_t sq_lock; void *sq_cmds; + /* only used for raw queues: */ + unsigned long *cmdid_bmp; + spinlock_t cmdid_lock; + struct nvme_iod *iod; /* only used for poll queues: */ spinlock_t cq_poll_lock ____cacheline_aligned_in_smp; struct nvme_completion *cqes; @@ -1661,6 +1676,141 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled, return result; } +static int setup_rawq_info(struct nvme_dev *dev, int nr_rawq) +{ + struct rawq_info *rawqi; + int size = sizeof(struct rawq_info) + nr_rawq * sizeof(int); + + rawqi = kzalloc(size, GFP_KERNEL); + if (rawqi == NULL) + return -ENOMEM; + rawqi->nr_free = nr_rawq; + dev->rawqi = rawqi; + return 0; +} + +static int nvme_pci_get_rawq(struct nvme_dev *dev) +{ + int i, qid, nr_rawq; + struct rawq_info *rawqi = NULL; + int ret = -EINVAL; + + nr_rawq = dev->nr_raw_queues; + if (!nr_rawq) + return ret; + + mutex_lock(&dev->rawq_lock); + if (dev->rawqi == NULL) { + ret = setup_rawq_info(dev, nr_rawq); + if (ret) + goto unlock; + } + rawqi = dev->rawqi; + if (rawqi->nr_free == 0) { + ret = -EINVAL; + goto unlock; + } + for (i = 0; i < nr_rawq; i++) { + if (rawqi->q_state[i] == Q_FREE) { + rawqi->q_state[i] = Q_ALLOC; + qid = dev->nr_allocated_queues - nr_rawq - i; + rawqi->nr_free--; + ret = qid; + goto unlock; + } + } +unlock: + mutex_unlock(&dev->rawq_lock); + return ret; +} + +static int nvme_pci_put_rawq(struct nvme_dev *dev, int qid) +{ + int i, nr_rawq; + struct rawq_info *rawqi = NULL; + struct nvme_queue *nvmeq; + + nr_rawq = dev->nr_raw_queues; + if (!nr_rawq || dev->rawqi == NULL) + return -EINVAL; + + i = dev->nr_allocated_queues - nr_rawq - qid; + mutex_lock(&dev->rawq_lock); + rawqi = dev->rawqi; + if (rawqi->q_state[i] == Q_ALLOC) { + rawqi->q_state[i] = Q_FREE; + rawqi->nr_free++; + } + mutex_unlock(&dev->rawq_lock); + nvmeq = &dev->queues[qid]; + kfree(nvmeq->cmdid_bmp); + kfree(nvmeq->iod); + return 0; +} + +static int nvme_pci_alloc_cmdid_bmp(struct nvme_queue *nvmeq) +{ + int size = BITS_TO_LONGS(nvmeq->q_depth) * sizeof(unsigned long); + + if (!test_bit(NVMEQ_RAW, &nvmeq->flags)) + return -EINVAL; + nvmeq->cmdid_bmp = kzalloc(size, GFP_KERNEL); + if (!nvmeq->cmdid_bmp) + return -ENOMEM; + spin_lock_init(&nvmeq->cmdid_lock); + return 0; +} + +static int nvme_pci_alloc_iod_array(struct nvme_queue *nvmeq) +{ + if (!test_bit(NVMEQ_RAW, &nvmeq->flags)) + return -EINVAL; + nvmeq->iod = kcalloc(nvmeq->q_depth - 1, sizeof(struct nvme_iod), + GFP_KERNEL); + if (!nvmeq->iod) + return -ENOMEM; + return 0; +} + +static int nvme_pci_setup_rawq(struct nvme_queue *nvmeq) +{ + int ret; + + ret = nvme_pci_alloc_cmdid_bmp(nvmeq); + if (ret) + return ret; + ret = nvme_pci_alloc_iod_array(nvmeq); + if (ret) { + kfree(nvmeq->cmdid_bmp); + return ret; + } + return ret; +} + +static int nvme_pci_register_queue(void *data) +{ + struct nvme_ns *ns = (struct nvme_ns *) data; + struct nvme_dev *dev = to_nvme_dev(ns->ctrl); + int qid, ret; + + qid = nvme_pci_get_rawq(dev); + if (qid > 0) { + /* setup command-id bitmap and iod array */ + ret = nvme_pci_setup_rawq(&dev->queues[qid]); + if (ret < 0) + qid = ret; + } + return qid; +} + +static int nvme_pci_unregister_queue(void *data, int qid) +{ + struct nvme_ns *ns = (struct nvme_ns *) data; + struct nvme_dev *dev = to_nvme_dev(ns->ctrl); + + return nvme_pci_put_rawq(dev, qid); +} + static const struct blk_mq_ops nvme_mq_admin_ops = { .queue_rq = nvme_queue_rq, .complete = nvme_pci_complete_rq, @@ -1679,6 +1829,8 @@ static const struct blk_mq_ops nvme_mq_ops = { .map_queues = nvme_pci_map_queues, .timeout = nvme_timeout, .poll = nvme_poll, + .register_queue = nvme_pci_register_queue, + .unregister_queue = nvme_pci_unregister_queue, }; static void nvme_dev_remove_admin(struct nvme_dev *dev) @@ -2698,6 +2850,7 @@ static void nvme_pci_free_ctrl(struct nvme_ctrl *ctrl) nvme_free_tagset(dev); put_device(dev->dev); kfree(dev->queues); + kfree(dev->rawqi); kfree(dev); } @@ -2938,6 +3091,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev, return ERR_PTR(-ENOMEM); INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work); mutex_init(&dev->shutdown_lock); + mutex_init(&dev->rawq_lock); dev->nr_write_queues = write_queues; dev->nr_poll_queues = poll_queues; From patchwork Sat Apr 29 09:39:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3412C7EE22 for ; Sat, 29 Apr 2023 09:43:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231124AbjD2JnG (ORCPT ); Sat, 29 Apr 2023 05:43:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230523AbjD2JnE (ORCPT ); Sat, 29 Apr 2023 05:43:04 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F7351FD4 for ; Sat, 29 Apr 2023 02:42:56 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230429094254epoutp013f6e6713bb366e71b1ef506321809f26~aXo34QHN22551625516epoutp013 for ; Sat, 29 Apr 2023 09:42:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230429094254epoutp013f6e6713bb366e71b1ef506321809f26~aXo34QHN22551625516epoutp013 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761374; bh=a1Ko1tJ8viYuiqNUaXjSspplUkqNdUYdAc0cZ2nFHa8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sBoJIh+BEUdg+ATzYJ4kr7UxqdffA/2CBFbvs1DAudtHxbqxq40yp87OxERx7sD8U WpfGCIbgFba2MWuaKAdXpLq1j+bDwlXIeAyt9ybqga1ZPOV9Z70RKF1/RNAkSiEy+n 5gOFPWnKJPYTTYsglv3JXcfWmOWg/KnNV3i+45Ds= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230429094254epcas5p283c806bab6a38a6d31d6a904f4a00fd8~aXo3VFAsx2155821558epcas5p2Z; Sat, 29 Apr 2023 09:42:54 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Q7kzc5lKkz4x9Pr; Sat, 29 Apr 2023 09:42:52 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 91.60.55646.C96EC446; Sat, 29 Apr 2023 18:42:52 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094251epcas5p144d042853e10f090e3119338c2306546~aXo08lRSK1041010410epcas5p1r; Sat, 29 Apr 2023 09:42:51 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230429094251epsmtrp2157b2d867c16b6daf4e4dfcab05fa906~aXo078rgo3077530775epsmtrp2f; Sat, 29 Apr 2023 09:42:51 +0000 (GMT) X-AuditID: b6c32a4b-b71fa7000001d95e-ef-644ce69c2d19 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 74.29.28392.B96EC446; Sat, 29 Apr 2023 18:42:51 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094249epsmtip2e3eae92ba5f7cf001b2e1a9aff515f82~aXozaVkkq0575905759epsmtip2U; Sat, 29 Apr 2023 09:42:49 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi Subject: [RFC PATCH 07/12] io_uring: support for using registered queue in uring-cmd Date: Sat, 29 Apr 2023 15:09:20 +0530 Message-Id: <20230429093925.133327-8-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrBJsWRmVeSWpSXmKPExsWy7bCmuu6cZz4pBku36lh8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBmVbZORmpiSWqSQmpec n5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+TiE6DrlpkDdKqSQlliTilQKCCxuFhJ386m KL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1xMrQwMDIFKgwITtj24SDTAVvxSu2vj/I 1MD4WriLkZNDQsBE4uWlVsYuRi4OIYHdjBI9J/5DOZ8YJf6/XccO4XxmlLiy8gIzTMuzDbuY IRK7GCW2Tv7ECFd1bcUpti5GDg42AU2JC5NLQRpEBFwkmtZOZQOpYRa4CLTj0yEWkISwQJjE zZPnGUFsFgFViWNrellBbF4BS4m9HXNZIbbJS8y89J0dxOYUsJL4PmM3M0SNoMTJmU/A5jAD 1TRvnQ12kYTAXA6J4/MnM0E0u0hcP/+QDcIWlnh1fAs7hC0l8fndXqh4ssSlmeeg6kskHu85 CGXbS7Se6mcGeYYZ6Jn1u/QhdvFJ9P5+wgQSlhDglehoE4KoVpS4N+kp1MniEg9nLIGyPSR2 HOtigoRPL6PE+QPbWSYwys9C8sIsJC/MQti2gJF5FaNkakFxbnpqsWmBcV5qOTxmk/NzNzGC k6yW9w7GRw8+6B1iZOJgPMQowcGsJMLLW+meIsSbklhZlVqUH19UmpNafIjRFBjGE5mlRJPz gWk+ryTe0MTSwMTMzMzE0tjMUEmcV932ZLKQQHpiSWp2ampBahFMHxMHp1QD04PGg1dO5TAx XFDd+uLdmeWyE9boBfbISR55O3XCZdUFB2W37NrQyHRb7mbNSoMXf49y6tkuUruowJYpnRax pyJjrtPmzXKdev1nuSpEckuqVn3l/Xl43s5frBuirqx4uqNcLtrg5eyd845k2EjsW6DEvOTN u5Ntgo8ffIjhM81Z9/iwZuDUa5fFfkXNa6je9rb32Qf2Pzdvxa3+FP5I0dXIcQfjgo8F96WZ 606LRhcXWZj2fXa9pMJ3uDkx/LVB04KgpSZtTL7txlfX5DPNuRWebfFIoWCvW6qZ8sPVhXX3 LVktmufoHEyd9JF7Q2zzL+UVC1be+ix26s6lUq+Z/jq5bzTPH73ulbIk4orTb1ElluKMREMt 5qLiRAD2X2TAOwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrFLMWRmVeSWpSXmKPExsWy7bCSvO7sZz4pBgs3SVh8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBnFZZOSmpNZllqkb5fA lbFtwkGmgrfiFVvfH2RqYHwt3MXIySEhYCLxbMMu5i5GLg4hgR2MEs3HtjFBJMQlmq/9YIew hSVW/nvODlH0kVHi6rLfQB0cHGwCmhIXJpeC1IgIeEm0v53FBlLDLHCTUWLf7r1gzcICIRJt ax+CDWURUJU4tqaXFcTmFbCU2NsxlxVigbzEzEvfweo5Bawkvs/YDTZfCKimcUE8RLmgxMmZ T1hAbGag8uats5knMArMQpKahSS1gJFpFaNkakFxbnpusWGBUV5quV5xYm5xaV66XnJ+7iZG cHRoae1g3LPqg94hRiYOxkOMEhzMSiK8vJXuKUK8KYmVValF+fFFpTmpxYcYpTlYlMR5L3Sd jBcSSE8sSc1OTS1ILYLJMnFwSjUw9Xb/CrprpvtdguP59h+8TiozOXRSg1z73dOOL7iU9WFm jBVTybrcuLyHr7/eCHiaePt20TM7tUf7H/ElhCo5hN7QSP/ln/faInIXr1hd9VOxs3vn5XX+ u8LdtayXNSpkRrbLOYXknrVPfxs+mD7z9TObQN8Jt/dNzypu6uo7Mjm5667wjWSXpu97o/K/ n59ptP31Wdlz8dOzn2xTkwv9vu/SsR/b9ObmlumZa30VSp2we88m6R6rzf6HlL8+3bugfwUX k1TgtBXCH9UXRXw02eybPc3q34+Ypd5qswql4+qqHvu/1GN5IsUttaFMdXnDo7JFjOXMNbFh Cy/5q1m/UNUO8w1deWtLlI1Fl/xuJZbijERDLeai4kQAbCskBv0CAAA= X-CMS-MailID: 20230429094251epcas5p144d042853e10f090e3119338c2306546 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094251epcas5p144d042853e10f090e3119338c2306546 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a new flag IORING_URING_CMD_DIRECT that user-space can spcifiy in SQE. If queue is registered with the ring, this flag goes down to the provider of ->uring_cmd. Provider may choose to do things differently or ignore this flag. Also export a helper that allows retrieving the identifier of the registered queue. Signed-off-by: Kanchan Joshi --- include/linux/io_uring.h | 6 ++++++ include/uapi/linux/io_uring.h | 2 ++ io_uring/uring_cmd.c | 14 +++++++++++++- 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 35b9328ca335..bb6f900411e1 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -37,6 +37,7 @@ struct io_uring_cmd { }; #if defined(CONFIG_IO_URING) +int io_uring_cmd_import_qid(void *ioucmd); int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret, ssize_t res2, @@ -67,6 +68,11 @@ static inline void io_uring_free(struct task_struct *tsk) __io_uring_free(tsk); } #else +static inline int io_uring_cmd_import_qid(void *ioucmd) +{ + return -EOPNOTSUPP; +} + static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index a9d59bfd26f7..67fbcfd3f676 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -232,8 +232,10 @@ enum io_uring_op { * sqe->uring_cmd_flags * IORING_URING_CMD_FIXED use registered buffer; pass this flag * along with setting sqe->buf_index. + * IORING_URING_CMD_DIRECT use registered queue for this cmd. */ #define IORING_URING_CMD_FIXED (1U << 0) +#define IORING_URING_CMD_DIRECT (1U << 1) /* diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 5113c9a48583..2a543b490045 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -89,9 +89,12 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -EINVAL; ioucmd->flags = READ_ONCE(sqe->uring_cmd_flags); - if (ioucmd->flags & ~IORING_URING_CMD_FIXED) + if (ioucmd->flags & ~(IORING_URING_CMD_FIXED | IORING_URING_CMD_DIRECT)) return -EINVAL; + if (ioucmd->flags & IORING_URING_CMD_DIRECT && + req->ctx->dev_qid == -EINVAL) + return -EINVAL; if (ioucmd->flags & IORING_URING_CMD_FIXED) { struct io_ring_ctx *ctx = req->ctx; u16 index; @@ -162,3 +165,12 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, return io_import_fixed(rw, iter, req->imu, ubuf, len); } EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed); + +int io_uring_cmd_import_qid(void *ioucmd) +{ + struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); + struct io_ring_ctx *ctx = req->ctx; + + return ctx->dev_qid; +} +EXPORT_SYMBOL_GPL(io_uring_cmd_import_qid); From patchwork Sat Apr 29 09:39:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AB06C77B60 for ; Sat, 29 Apr 2023 09:43:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230523AbjD2JnH (ORCPT ); Sat, 29 Apr 2023 05:43:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbjD2JnG (ORCPT ); Sat, 29 Apr 2023 05:43:06 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5626210E for ; Sat, 29 Apr 2023 02:42:58 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230429094257epoutp0149050a1a76b115643720c791ade4ab37~aXo6SNnqP2540625406epoutp019 for ; Sat, 29 Apr 2023 09:42:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230429094257epoutp0149050a1a76b115643720c791ade4ab37~aXo6SNnqP2540625406epoutp019 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761377; bh=ZdLCtWWe0/Llin+LzaCzsXrHz2aghk9/ahbIIMl7+Xo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YiseguAM2nMEq1UrQmZ5YJxrPzVuqFa6o+zE0l960iE1oy2dHd7WjlfHFw09x/6OW mp8SBHmDsWpFBii62yChTelWOCfZy01dqqFlWPRU3NEcGkZLc37R3Oy1xnbwUv8gka VZ6EVRl5X86ia+4Nwn8L8kOwDVen/k23+XSa7rrg= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230429094255epcas5p447f0176a9d404b46dd0f40a3d864e95c~aXo5GlbYu1967019670epcas5p4R; Sat, 29 Apr 2023 09:42:55 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzf1K7zz4x9Pq; Sat, 29 Apr 2023 09:42:54 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 52.60.55646.E96EC446; Sat, 29 Apr 2023 18:42:54 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230429094253epcas5p3cfff90e1c003b6fc9c7c4a61287beecb~aXo2uScLC0334203342epcas5p3N; Sat, 29 Apr 2023 09:42:53 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094253epsmtrp1b582f9b72b57a2a0e53a6d45cb7a2113~aXo2tiA7x0376803768epsmtrp1w; Sat, 29 Apr 2023 09:42:53 +0000 (GMT) X-AuditID: b6c32a4b-913ff7000001d95e-f2-644ce69eed41 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 35.29.28392.D96EC446; Sat, 29 Apr 2023 18:42:53 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094251epsmtip227856de4b7a95bc7378b7b46d1facec0~aXo1Ed1Pr0191301913epsmtip2-; Sat, 29 Apr 2023 09:42:51 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Anuj Gupta , Kanchan Joshi Subject: [RFC PATCH 08/12] block: add mq_ops to submit and complete commands from raw-queue Date: Sat, 29 Apr 2023 15:09:21 +0530 Message-Id: <20230429093925.133327-9-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrFJsWRmVeSWpSXmKPExsWy7bCmhu68Zz4pBrsmsll8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZ1S2TUZqYkpq kUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqtkotPgK5bZg7QvUoKZYk5pUChgMTi YiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9vNQSK0MDAyNToMKE7Iy3vbvZCjay V0x5MYmxgXEqWxcjJ4eEgInE/971rF2MXBxCArsZJbbtvsEEkhAS+MQo8fEqM0TiM6PExjeL 4TomdP1nh0jsYpR4eP8iK1xV1+P/QC0cHGwCmhIXJpeCNIgIuEg0rQVZx8XBLPCNUaJp91wW kISwQKzEkp9PWUFsFgFViUPtj8E28ApYSsydvoYFYpu8xMxL39lBbE4BK4nvM3YzQ9QISpyc +QSshhmopnnrbGaI+qUcEu9nhEHYLhKzZm9mhbCFJV4d38IOYUtJvOxvg7KTJS7NPMcEYZdI PN5zEMq2l2g91Q/2CzPQL+t36UOs4pPo/f2ECSQsIcAr0dEmBFGtKHFv0lOoTeISD2csgbI9 JNqP/GaFBGgvo8TymzUTGOVnIXlgFpIHZiEsW8DIvIpRMrWgODc9tdi0wDgvtRwercn5uZsY wYlWy3sH46MHH/QOMTJxMB5ilOBgVhLh5a10TxHiTUmsrEotyo8vKs1JLT7EaAoM4YnMUqLJ +cBUn1cSb2hiaWBiZmZmYmlsZqgkzqtuezJZSCA9sSQ1OzW1ILUIpo+Jg1OqgSkzIiFHpE0x zWl5uFjL1h/76gOr+cvVY/gZcjldivvMqhu2ei6IXzM56jz3rKb5Sisv+jzQULb65hkreFMh uP7GtOJzd0o5eSd8meDcefX1q44zxQ8OHOyNucL9SezGRX7H61cPsz+ekNSnc1m1PGV7+3GR tJLzJlNFdJ3qIi+n6sgK79iqrivRz3PNfXvDyfKnT2SubT6x/a/CjF/33rTxPOF8zfLri6SW z7K8a8U3tmwufp3v77Q4/P450Xvi7A7/ba8maEdsMvp+ILT68r6SWstMVa18822eJt3bk9bP 1L1x3X5BfXogr8zvFz95jXQDJdIL9IOtvUPZbgXb2lvd93Hc+Lta7Fn6wuVTlViKMxINtZiL ihMBF2wPdz0EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRmVeSWpSXmKPExsWy7bCSvO7cZz4pBmf6hSw+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFkf/v2WzmHToGqPF3lvaFvOXPWW3WPf6PYvFpr8nmRx4 PHbOusvucf7eRhaPy2dLPTat6mTz2PnQ0mPzknqP3Tcb2Dz6tqxi9Pi8SS6AM4rLJiU1J7Ms tUjfLoEr423vbraCjewVU15MYmxgnMrWxcjJISFgIjGh6z87iC0ksINR4vcLOYi4uETztR/s ELawxMp/z4FsLqCaj4wSJ07vYepi5OBgE9CUuDC5FKRGRMBLov3tLDaQGmaBf4wS779eZgJJ CAtESzS9vgo2iEVAVeJQ+2OwxbwClhJzp69hgVggLzHz0newGk4BK4nvM3Yzg8wXAqppXBAP US4ocXLmE7ByZqDy5q2zmScwCsxCkpqFJLWAkWkVo2RqQXFuem6xYYFRXmq5XnFibnFpXrpe cn7uJkZwlGhp7WDcs+qD3iFGJg7GQ4wSHMxKIry8le4pQrwpiZVVqUX58UWlOanFhxilOViU xHkvdJ2MFxJITyxJzU5NLUgtgskycXBKNTDNS0ksTljcq95wPSH1R0nR7QLz85erpGby/y9b faJ82Q2n57wvr0fUT74TJnDz4znO+MBfLQvDGZJ2807KbM94Weim9fD9wf0ZE1udKp5/Yikx XBW2/tG859rL9nKrW1py3fa89XrivKMa+ziCD0TOZZKUF3hivOvDuwVxi5c/936XsD18nULD uqTO1PeLD2fMiHuyuGBOpX/zMQ3BOadsn7zbubXR+uyXooX6oXM/35cy5fhyS2UNV8aKjKuv mS/oLpL5Je/GLhksIxO66tXK3pa49bIbN4XEqR1kyowyetSdf71i8vEn6gcuz7Q+3TexePfz L9pCx5229F+6rNtmpXRyu9+BkPIv9y/166XmKrEUZyQaajEXFScCAKgzlO8BAwAA X-CMS-MailID: 20230429094253epcas5p3cfff90e1c003b6fc9c7c4a61287beecb X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094253epcas5p3cfff90e1c003b6fc9c7c4a61287beecb References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Anuj Gupta This patch introduces ->queue_uring_cmd and -> poll_uring_cmd in mq_ops, which will allow to submit and complete (via polling) from the specified raw-queue (denoted by qid). Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- include/linux/blk-mq.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 7d6790be4847..dcce2939ff1e 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -659,6 +659,9 @@ struct blk_mq_ops { void (*show_rq)(struct seq_file *m, struct request *rq); int (*register_queue)(void *data); int (*unregister_queue)(void *data, int qid); + int (*queue_uring_cmd)(struct io_uring_cmd *ioucmd, int qid); + int (*poll_uring_cmd)(struct io_uring_cmd *ioucmd, int qid, + struct io_comp_batch *); #endif }; From patchwork Sat Apr 29 09:39:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B112CC77B7E for ; Sat, 29 Apr 2023 09:43:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231135AbjD2JnO (ORCPT ); Sat, 29 Apr 2023 05:43:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjD2JnM (ORCPT ); Sat, 29 Apr 2023 05:43:12 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B8232139 for ; Sat, 29 Apr 2023 02:43:01 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230429094259epoutp01f19d6dfeb14260718a9aa5baa88a7471~aXo8XEwrC2551625516epoutp015 for ; Sat, 29 Apr 2023 09:42:59 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230429094259epoutp01f19d6dfeb14260718a9aa5baa88a7471~aXo8XEwrC2551625516epoutp015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761379; bh=CUKxeRq1qvH1W4pJIKUNwrDgFpmbgSlJ0NlQ+has8Hg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MrBsZ7a9J29Gqwk3I+F4NeEiqbqXFO63aiWLh5wU4cBfR8TIWBjIN4/NEiqgjhDgX HGsMFDkyAaAn7fCWB4JDvwv7wVfKpsuJjM4KKL1inrmtC+thEb8YDDKAZ6ajKy742F +TzFE8/eo+IPm2Iwu0zEtxG8BBris6tOFEVEBPbA= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230429094258epcas5p3749184eb8850fc553547f2d2b2666528~aXo7f0xtb2426524265epcas5p3v; Sat, 29 Apr 2023 09:42:58 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.183]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzh6gJHz4x9Pq; Sat, 29 Apr 2023 09:42:56 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B4.7C.54880.0A6EC446; Sat, 29 Apr 2023 18:42:56 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094255epcas5p11bcbe76772289f27c41a50ce502c998d~aXo4y0H0-1041010410epcas5p1v; Sat, 29 Apr 2023 09:42:55 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094255epsmtrp13906fe5c736973e4538870cd28553ca8~aXo4yJBAV0376803768epsmtrp1x; Sat, 29 Apr 2023 09:42:55 +0000 (GMT) X-AuditID: b6c32a49-b21fa7000001d660-9b-644ce6a06d12 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 06.29.28392.F96EC446; Sat, 29 Apr 2023 18:42:55 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094253epsmtip28a893917aee13f32a8fc91fa9611a6ad~aXo23wiAu0703907039epsmtip2U; Sat, 29 Apr 2023 09:42:53 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi , Anuj Gupta Subject: [RFC PATCH 09/12] nvme: carve out a helper to prepare nvme_command from ioucmd->cmd Date: Sat, 29 Apr 2023 15:09:22 +0530 Message-Id: <20230429093925.133327-10-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrJJsWRmVeSWpSXmKPExsWy7bCmuu6CZz4pBn9261t8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZ1S2TUZqYkpq kUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqtkotPgK5bZg7QvUoKZYk5pUChgMTi YiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9vNQSK0MDAyNToMKE7Izesw1sBUc0 Kj4/fsjewLhHqYuRk0NCwERi647PTF2MXBxCArsZJZqbF7JAOJ8YJU59mcUO4XxmlDix9ysz TEvzt052EFtIYBejxJYP6RA2UNGNg9JdjBwcbAKaEhcml4KERQRcJJrWTmUDmcMs8I1RYtHr d6wgNcICcRJPd4G1sgioSqze+5oNxOYVsJJ4e30DG8QqeYmZl76DreIEin+fsZsZokZQ4uTM JywgNjNQTfPW2cwg8yUElnJIvNw8jQmi2UXi5LT57BC2sMSr41ugbCmJl/1tUHayxKWZ56Dq SyQe7zkIZdtLtJ7qZwa5kxnol/W79CF28Un0/n7CBBKWEOCV6GgTgqhWlLg36SkrhC0u8XDG ElaIEg+JM0sVICHYyyhx/Wwb+wRG+VlIPpiF5INZCMsWMDKvYpRMLSjOTU8tNi0wzEsth8dq cn7uJkZwmtXy3MF498EHvUOMTByMhxglOJiVRHh5K91ThHhTEiurUovy44tKc1KLDzGaAoN4 IrOUaHI+MNHnlcQbmlgamJiZmZlYGpsZKonzqtueTBYSSE8sSc1OTS1ILYLpY+LglGpgcvp9 SK96t3Ov1oli1S3u06dsDGJ1m6mfwVj3xzk34PCNzuwvS3/IHZA/mNhv/DW/KLf37Kbf++V/ tLb2zn2ffp1HfuXJw8E2Ny/bm11cxvJrtui6LR86pRRzIi8zmBxeNeH3L5lpsmetF8eduMn2 s3WneZru/U0CXw5OC8th/Wec4di2n1+l5V7rF5N1skYbJlyZaOvGOz/loanJUpdPm3mvZryR Feiyn38+vsDWI/al6m3tN4eWMnHfDQjgWum456u4kg17BdfhSdnbXQ28zhyzUw27rb28+lTa 5LyUd7pHn33tkGF8Oktvy9favik1+9R2v5U/tzjoU9/qL45x+yvrhdqk2/Unaf83/3OrRoml OCPRUIu5qDgRABV1R5Y8BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsWy7bCSvO78Zz4pBlM2SVl8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZxSXTUpqTmZZ apG+XQJXRu/ZBraCIxoVnx8/ZG9g3KPUxcjJISFgItH8rZO9i5GLQ0hgB6PExxv7WSES4hLN 136wQ9jCEiv/PYcq+sgosbfvD1MXIwcHm4CmxIXJpSA1IgJeEu1vZ7GB1DAL/GOUePC+jQ0k ISwQIzH7+XOwQSwCqhKr974Gi/MKWEm8vb6BDWKBvMTMS9/BajiB4t9n7GYGmS8kYCnRuCAe olxQ4uTMJywgNjNQefPW2cwTGAVmIUnNQpJawMi0ilEytaA4Nz232LDAKC+1XK84Mbe4NC9d Lzk/dxMjOE60tHYw7ln1Qe8QIxMH4yFGCQ5mJRFe3kr3FCHelMTKqtSi/Pii0pzU4kOM0hws SuK8F7pOxgsJpCeWpGanphakFsFkmTg4pRqYlq9ba+xo5mBtE8uV9qZ32ZOZVvbv005Pce25 Zf2nmNPymMmb4CVxbw8y2dXrWnybvjzlnN9xTmbjHwddDXdGCMRnHvR7/Pbi7JBfC/59CWp5 eJBlE0e7TpHotDd6yzfseFYyb96qL3UTSr1j5itd8WPj3Vjds3vZdMErBef7vE4Ff21nqt/U 6rioOr7a02PTDrVDpbeZfhlFxW8zPBKxMDdz8fuvnMqPb4r+ni6mbP1Rg+djr/lKTu9ps5Ze DMtdxnDo8y6PVZ2e4oEvVi5/tqY5/04o/zOLmzWRV+c9ndd9aWn+V4PO+heVqoJ3MhUY1WeL XeqOePda/NtKIZ7IsKf5Qou2+a1r+fS4buPp30osxRmJhlrMRcWJACAt69ECAwAA X-CMS-MailID: 20230429094255epcas5p11bcbe76772289f27c41a50ce502c998d X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094255epcas5p11bcbe76772289f27c41a50ce502c998d References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This helper will be used in the subsequent patch. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- drivers/nvme/host/ioctl.c | 81 +++++++++++++++++++++------------------ drivers/nvme/host/nvme.h | 11 ++++++ 2 files changed, 54 insertions(+), 38 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 292a578686b6..18f4f20f5e76 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -430,14 +430,6 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, return status; } -struct nvme_uring_data { - __u64 metadata; - __u64 addr; - __u32 data_len; - __u32 metadata_len; - __u32 timeout_ms; -}; - /* * This overlays struct io_uring_cmd pdu. * Expect build errors if this grows larger than that. @@ -548,11 +540,50 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, return RQ_END_IO_NONE; } +int nvme_prep_cmd_from_ioucmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, + struct io_uring_cmd *ioucmd, struct nvme_command *c, + struct nvme_uring_data *d) +{ + const struct nvme_uring_cmd *cmd = ioucmd->cmd; + + c->common.opcode = READ_ONCE(cmd->opcode); + c->common.flags = READ_ONCE(cmd->flags); + if (c->common.flags) + return -EINVAL; + + c->common.command_id = 0; + c->common.nsid = cpu_to_le32(cmd->nsid); + if (!nvme_validate_passthru_nsid(ctrl, ns, le32_to_cpu(c->common.nsid))) + return -EINVAL; + + c->common.cdw2[0] = cpu_to_le32(READ_ONCE(cmd->cdw2)); + c->common.cdw2[1] = cpu_to_le32(READ_ONCE(cmd->cdw3)); + c->common.metadata = 0; + c->common.dptr.prp1 = c->common.dptr.prp2 = 0; + c->common.cdw10 = cpu_to_le32(READ_ONCE(cmd->cdw10)); + c->common.cdw11 = cpu_to_le32(READ_ONCE(cmd->cdw11)); + c->common.cdw12 = cpu_to_le32(READ_ONCE(cmd->cdw12)); + c->common.cdw13 = cpu_to_le32(READ_ONCE(cmd->cdw13)); + c->common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14)); + c->common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15)); + + if (!nvme_cmd_allowed(ns, c, 0, ioucmd->file->f_mode)) + return -EACCES; + + d->metadata = READ_ONCE(cmd->metadata); + d->addr = READ_ONCE(cmd->addr); + d->data_len = READ_ONCE(cmd->data_len); + d->metadata_len = READ_ONCE(cmd->metadata_len); + d->timeout_ms = READ_ONCE(cmd->timeout_ms); + return 0; +} +EXPORT_SYMBOL_GPL(nvme_prep_cmd_from_ioucmd); + + static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct io_uring_cmd *ioucmd, unsigned int issue_flags, bool vec) { struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - const struct nvme_uring_cmd *cmd = ioucmd->cmd; struct request_queue *q = ns ? ns->queue : ctrl->admin_q; struct nvme_uring_data d; struct nvme_command c; @@ -562,35 +593,9 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, void *meta = NULL; int ret; - c.common.opcode = READ_ONCE(cmd->opcode); - c.common.flags = READ_ONCE(cmd->flags); - if (c.common.flags) - return -EINVAL; - - c.common.command_id = 0; - c.common.nsid = cpu_to_le32(cmd->nsid); - if (!nvme_validate_passthru_nsid(ctrl, ns, le32_to_cpu(c.common.nsid))) - return -EINVAL; - - c.common.cdw2[0] = cpu_to_le32(READ_ONCE(cmd->cdw2)); - c.common.cdw2[1] = cpu_to_le32(READ_ONCE(cmd->cdw3)); - c.common.metadata = 0; - c.common.dptr.prp1 = c.common.dptr.prp2 = 0; - c.common.cdw10 = cpu_to_le32(READ_ONCE(cmd->cdw10)); - c.common.cdw11 = cpu_to_le32(READ_ONCE(cmd->cdw11)); - c.common.cdw12 = cpu_to_le32(READ_ONCE(cmd->cdw12)); - c.common.cdw13 = cpu_to_le32(READ_ONCE(cmd->cdw13)); - c.common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14)); - c.common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15)); - - if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode)) - return -EACCES; - - d.metadata = READ_ONCE(cmd->metadata); - d.addr = READ_ONCE(cmd->addr); - d.data_len = READ_ONCE(cmd->data_len); - d.metadata_len = READ_ONCE(cmd->metadata_len); - d.timeout_ms = READ_ONCE(cmd->timeout_ms); + ret = nvme_prep_cmd_from_ioucmd(ctrl, ns, ioucmd, &c, &d); + if (ret) + return ret; if (issue_flags & IO_URING_F_NONBLOCK) { rq_flags |= REQ_NOWAIT; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 4619a7498f8e..4eb45afc9484 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -168,6 +168,14 @@ struct nvme_request { struct nvme_ctrl *ctrl; }; +struct nvme_uring_data { + __u64 metadata; + __u64 addr; + __u32 data_len; + __u32 metadata_len; + __u32 timeout_ms; +}; + /* * Mark a bio as coming in through the mpath node. */ @@ -811,6 +819,9 @@ static inline bool nvme_is_unique_nsid(struct nvme_ctrl *ctrl, (ctrl->ctratt & NVME_CTRL_CTRATT_NVM_SETS); } +int nvme_prep_cmd_from_ioucmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, + struct io_uring_cmd *ioucmd, struct nvme_command *c, + struct nvme_uring_data *d); int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, void *buf, unsigned bufflen); int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, From patchwork Sat Apr 29 09:39:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C2CBC7EE24 for ; Sat, 29 Apr 2023 09:43:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231136AbjD2JnO (ORCPT ); Sat, 29 Apr 2023 05:43:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231148AbjD2JnN (ORCPT ); Sat, 29 Apr 2023 05:43:13 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECBCE10D7 for ; Sat, 29 Apr 2023 02:43:01 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230429094300epoutp03fdf320575ce9a596760c59dc14e61a44~aXo9NPnes3037430374epoutp03j for ; Sat, 29 Apr 2023 09:43:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230429094300epoutp03fdf320575ce9a596760c59dc14e61a44~aXo9NPnes3037430374epoutp03j DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761380; bh=DtyDvlRYNmEJxi0GYtG3RjT+t7RC5XKjuqrISgRW5bE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OoNvAOoHGOXA1xnEnAwu39oLMf0BMf8ih/m6BSUV5gr6Fb3wQG52PPrAwmeuteFhY 6hO6GwIZRk2DNAUiizftqL4hpWgkSAnH1dKKA8l062qbkYYOGusv/VApKsNdAW/iMU DNEL/LuHweHeml377Ok6gMCH6RyFWY0LVzoF4p6M= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230429094259epcas5p10dd02a8941a48e25f25dc3df91a1ae48~aXo8beI6i1041010410epcas5p1y; Sat, 29 Apr 2023 09:42:59 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Q7kzk2QLJz4x9Pv; Sat, 29 Apr 2023 09:42:58 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 85.7C.54880.2A6EC446; Sat, 29 Apr 2023 18:42:58 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230429094257epcas5p463574920bba26cd219275e57c2063d85~aXo6dL7V01668816688epcas5p46; Sat, 29 Apr 2023 09:42:57 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230429094257epsmtrp1808ec5ab05a9a48765311bf37e4cb428~aXo6cifes0376803768epsmtrp1y; Sat, 29 Apr 2023 09:42:57 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-9e-644ce6a24f54 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D8.38.27706.1A6EC446; Sat, 29 Apr 2023 18:42:57 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094255epsmtip2c15ae7be5d7b2531d5b34103fe80be40~aXo40BWrm0920909209epsmtip2J; Sat, 29 Apr 2023 09:42:55 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi Subject: [RFC PATCH 10/12] nvme: submisssion/completion of uring_cmd to/from the registered queue Date: Sat, 29 Apr 2023 15:09:23 +0530 Message-Id: <20230429093925.133327-11-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrCJsWRmVeSWpSXmKPExsWy7bCmlu6iZz4pBlcnCFh8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBmVbZORmpiSWqSQmpec n5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+TiE6DrlpkDdKqSQlliTilQKCCxuFhJ386m KL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1xMrQwMDIFKgwITtj+tMTTAW7tCumbGtk aWBcqdzFyMkhIWAicezyWrYuRi4OIYHdjBIHJi5lB0kICXxilHh6tQrC/sYocey/N0zD47t7 oBr2MkpsP3CSDaLoM6PE4p/RXYwcHGwCmhIXJpeChEUEXCSa1k4Fq2cWuMgo0fPpEAtIQlgg WeLr/dlgNouAqkTv319gNq+AlURz1w8WiGXyEjMvfQc7iBMo/n3GbmaIGkGJkzOfgNUwA9U0 b53NDLJAQmAuh0Tbi2XMIEdIAG3+tj0DYo6wxKvjW9ghbCmJl/1tUHayxKWZ55gg7BKJx3sO Qtn2Eq2n+sHGMAP9sn6XPsQqPone30+YIKbzSnS0CUFUK0rcm/SUFcIWl3g4YwmU7SFx4fB7 dkhQ9TJKrLnzlWkCo/wsJB/MQvLBLIRtCxiZVzFKphYU56anFpsWGOallsNjNTk/dxMjOLlq ee5gvPvgg94hRiYOxkOMEhzMSiK8vJXuKUK8KYmVValF+fFFpTmpxYcYTYFBPJFZSjQ5H5je 80riDU0sDUzMzMxMLI3NDJXEedVtTyYLCaQnlqRmp6YWpBbB9DFxcEo1MFl8afR2myFW9V4j ieteSqxAh+Di74WHN5nL9U7+e1b9V7fppQ/RVf1F6+XX7DPytDXd0MxdFtTsqsR4bbN492NR tox/ixbvWXll/83lwtWpt7o2Xln8d9flsqdrBY8HBR/fHtfSnXt1vveRPUVt7sL2S7ZP68p5 YTrJJ1FQJin8xsq3Gy2073/xbw+YeHvZpDSOnwwPmjYfuCgbNkf8w5zAxnPF8W3sysqGK2qX nX5qvnQ+I8v3HalHtrpwiOWxrVHfOvln8QXFpuTEDYpqWh+tdKanp/AdfC81e9GlTVN2fSi9 U5kpaDHvibHKrZiFgck/3rcGvOn8oRF49Mi+gJ6yiVfKG7s3VR7nVdw6J1WJpTgj0VCLuag4 EQCVQJDRNwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrFLMWRmVeSWpSXmKPExsWy7bCSvO7CZz4pBgu/8lp8/PqbxWL13X42 i5sHdjJZrFx9lMniXes5Fouj/9+yWUw6dI3RYu8tbYv5y56yW6x7/Z7FYtPfk0wO3B47Z91l 9zh/byOLx+WzpR6bVnWyeex8aOmxeUm9x+6bDWwefVtWMXp83iQXwBnFZZOSmpNZllqkb5fA lTH96Qmmgl3aFVO2NbI0MK5U7mLk5JAQMJF4fHcPWxcjF4eQwG5GiTWPv7JDJMQlmq/9gLKF JVb+e84OUfSRUWLF0flADgcHm4CmxIXJpSA1IgJeEu1vZ4ENYha4ySixb/desGZhgUSJp7vf MoLYLAKqEr1/f7GA2LwCVhLNXT9YIBbIS8y89B2snhMo/n3GbmaQ+UIClhKNC+IhygUlTs58 AlbODFTevHU28wRGgVlIUrOQpBYwMq1ilEwtKM5Nzy02LDDMSy3XK07MLS7NS9dLzs/dxAiO Di3NHYzbV33QO8TIxMF4iFGCg1lJhJe30j1FiDclsbIqtSg/vqg0J7X4EKM0B4uSOO+FrpPx QgLpiSWp2ampBalFMFkmDk6pBqautC0mvZeWvNEVf1V/7Pu/jJMLlt6+ey00b7EEz/2y3qZV B/MrT5vPLDzxsVR28gKT5RuunXpVd23G7pqllQV30nYbeCR9spzl7TTX/8XyCFfJHSbOtVyX DXu0fza9/F+1I2/TFV7pG6W519iK3IqeFa0Tvrpj3vvERtlVM+5ocqZEy/ZuiN1zTzJvvlJL eMJW+STu5U+fuIlcds87NOXKq4k5NZwPZid5uN0SCnXOYGbbaLiJY+0D+927O35Jub1lPd45 Uf3cgpwVEyc/3TI3duFLsffxTbUfXyRdiauw/sfcmL2AffMcXy876xNb9/wP5QvtmTPt8d4j O2JnX1S7JX6E/8jiTO7NJXV2UyemKbEUZyQaajEXFScCAE8v+f79AgAA X-CMS-MailID: 20230429094257epcas5p463574920bba26cd219275e57c2063d85 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094257epcas5p463574920bba26cd219275e57c2063d85 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If IORING_URING_CMD_DIRECT flag is set, get the registered qid and - submit the io_uring command via mq_ops->queue_uring_cmd. - complete it via mq_ops->poll_uring_cmd. If the command could not be submitted this way due to any reason, abstract it and fallback to old way of submission. This keeps IORING_URING_CMD_DIRECT flag advisory. Signed-off-by: Kanchan Joshi --- drivers/nvme/host/ioctl.c | 105 +++++++++++++++++++++++++++++++++----- drivers/nvme/host/nvme.h | 4 ++ 2 files changed, 97 insertions(+), 12 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 18f4f20f5e76..df86fb4f132b 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -653,6 +653,23 @@ static bool is_ctrl_ioctl(unsigned int cmd) return false; } +static int nvme_uring_cmd_io_direct(struct nvme_ctrl *ctrl, struct nvme_ns *ns, + struct io_uring_cmd *ioucmd, unsigned int issue_flags) +{ + struct request_queue *q = ns ? ns->queue : ctrl->admin_q; + int qid = io_uring_cmd_import_qid(ioucmd); + struct nvme_uring_direct_pdu *pdu = + (struct nvme_uring_direct_pdu *)&ioucmd->pdu; + + if ((issue_flags & IO_URING_F_IOPOLL) != IO_URING_F_IOPOLL) + return -EOPNOTSUPP; + + pdu->ns = ns; + if (q->mq_ops && q->mq_ops->queue_uring_cmd) + return q->mq_ops->queue_uring_cmd(ioucmd, qid); + return -EOPNOTSUPP; +} + static int nvme_ctrl_ioctl(struct nvme_ctrl *ctrl, unsigned int cmd, void __user *argp, fmode_t mode) { @@ -763,6 +780,14 @@ static int nvme_ns_uring_cmd(struct nvme_ns *ns, struct io_uring_cmd *ioucmd, switch (ioucmd->cmd_op) { case NVME_URING_CMD_IO: + if (ioucmd->flags & IORING_URING_CMD_DIRECT) { + ret = nvme_uring_cmd_io_direct(ctrl, ns, ioucmd, + issue_flags); + if (ret == -EIOCBQUEUED) + return ret; + /* in case of any error, just fallback */ + ioucmd->flags &= ~(IORING_URING_CMD_DIRECT); + } ret = nvme_uring_cmd_io(ctrl, ns, ioucmd, issue_flags, false); break; case NVME_URING_CMD_IO_VEC: @@ -783,6 +808,38 @@ int nvme_ns_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) return nvme_ns_uring_cmd(ns, ioucmd, issue_flags); } +/* similar to blk_mq_poll; may be possible to unify */ +int nvme_uring_cmd_iopoll_qid(struct request_queue *q, + struct io_uring_cmd *ioucmd, int qid, + struct io_comp_batch *iob, + unsigned int flags) +{ + long state = get_current_state(); + int ret; + + if (!(q->mq_ops && q->mq_ops->poll_uring_cmd)) + return 0; + do { + ret = q->mq_ops->poll_uring_cmd(ioucmd, qid, iob); + if (ret > 0) { + __set_current_state(TASK_RUNNING); + return ret; + } + if (signal_pending_state(state, current)) + __set_current_state(TASK_RUNNING); + if (task_is_running(current)) + return 1; + + if (ret < 0 || (flags & BLK_POLL_ONESHOT)) + break; + cpu_relax(); + + } while (!need_resched()); + + __set_current_state(TASK_RUNNING); + return 0; +} + int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags) @@ -792,14 +849,26 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct nvme_ns *ns; struct request_queue *q; - rcu_read_lock(); - bio = READ_ONCE(ioucmd->cookie); ns = container_of(file_inode(ioucmd->file)->i_cdev, struct nvme_ns, cdev); q = ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio && bio->bi_bdev) - ret = bio_poll(bio, iob, poll_flags); - rcu_read_unlock(); + if (!(ioucmd->flags & IORING_URING_CMD_DIRECT)) { + rcu_read_lock(); + bio = READ_ONCE(ioucmd->cookie); + if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio && bio->bi_bdev) + ret = bio_poll(bio, iob, poll_flags); + + rcu_read_unlock(); + } else { + int qid = io_uring_cmd_import_qid(ioucmd); + + if (qid <= 0) + return 0; + if (!percpu_ref_tryget(&q->q_usage_counter)) + return 0; + ret = nvme_uring_cmd_iopoll_qid(q, ioucmd, qid, iob, poll_flags); + percpu_ref_put(&q->q_usage_counter); + } return ret; } #ifdef CONFIG_NVME_MULTIPATH @@ -952,13 +1021,25 @@ int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct request_queue *q; if (ns) { - rcu_read_lock(); - bio = READ_ONCE(ioucmd->cookie); - q = ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio - && bio->bi_bdev) - ret = bio_poll(bio, iob, poll_flags); - rcu_read_unlock(); + if (!(ioucmd->flags & IORING_URING_CMD_DIRECT)) { + rcu_read_lock(); + bio = READ_ONCE(ioucmd->cookie); + q = ns->queue; + if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio + && bio->bi_bdev) + ret = bio_poll(bio, iob, poll_flags); + rcu_read_unlock(); + } else { + int qid = io_uring_cmd_import_qid(ioucmd); + + if (qid <= 0) + return 0; + if (!percpu_ref_tryget(&q->q_usage_counter)) + return 0; + ret = nvme_uring_cmd_iopoll_qid(q, ioucmd, qid, iob, + poll_flags); + percpu_ref_put(&q->q_usage_counter); + } } srcu_read_unlock(&head->srcu, srcu_idx); return ret; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 4eb45afc9484..2fd4432fbe12 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -176,6 +176,10 @@ struct nvme_uring_data { __u32 timeout_ms; }; +struct nvme_uring_direct_pdu { + struct nvme_ns *ns; +}; + /* * Mark a bio as coming in through the mpath node. */ From patchwork Sat Apr 29 09:39:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 029CEC7EE22 for ; Sat, 29 Apr 2023 09:43:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231144AbjD2JnU (ORCPT ); Sat, 29 Apr 2023 05:43:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231164AbjD2JnR (ORCPT ); Sat, 29 Apr 2023 05:43:17 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18F341FF2 for ; Sat, 29 Apr 2023 02:43:05 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230429094302epoutp01502e1da19530010ee492a430242c217d~aXo_yNEC92598925989epoutp016 for ; Sat, 29 Apr 2023 09:43:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230429094302epoutp01502e1da19530010ee492a430242c217d~aXo_yNEC92598925989epoutp016 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761382; bh=aMQGYrKQR0F3Yjruge3qlF3MmwpDpKmN737Ouil9Fvg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DjLZ1OMkEkxFruTg0IJrFdCGK/TLOTC66cvssvLViJphwGoTYlp4R7Us0TIfmnKm5 RNe/4hUiH7t5UbYMdrA5yu0PWr/OKkwu/tDWb6fZEV4hhBRlPh+95SKLgUThFBcPl0 SQmzEUtQYhsQnt6jT5e7zUM/01Q0vskAaDobVD5A= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230429094301epcas5p36aac06352f25c1a7a6cbfd232443861c~aXo_NTa2k2426524265epcas5p3z; Sat, 29 Apr 2023 09:43:01 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.178]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4Q7kzm1Wpxz4x9Pv; Sat, 29 Apr 2023 09:43:00 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 86.60.55646.4A6EC446; Sat, 29 Apr 2023 18:43:00 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230429094259epcas5p11f0f3422eb4aa4e3ebf00e0666790efa~aXo8bbf_M2856928569epcas5p10; Sat, 29 Apr 2023 09:42:59 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230429094259epsmtrp2d3c103cd3d0c04bd93d29d979136e0a6~aXo8atG-O0329503295epsmtrp2R; Sat, 29 Apr 2023 09:42:59 +0000 (GMT) X-AuditID: b6c32a4b-b71fa7000001d95e-ff-644ce6a45ea4 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id D6.29.28392.3A6EC446; Sat, 29 Apr 2023 18:42:59 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094257epsmtip248e853837ceca4e51363fca0e2ab46f4~aXo6e_z7J0191301913epsmtip2A; Sat, 29 Apr 2023 09:42:57 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Anuj Gupta Subject: [RFC PATCH 11/12] pci: modify nvme_setup_prp_simple parameters Date: Sat, 29 Apr 2023 15:09:24 +0530 Message-Id: <20230429093925.133327-12-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrOJsWRmVeSWpSXmKPExsWy7bCmhu6SZz4pBre62Sw+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFpMOXWO02HtL22L+sqfsFutev2ex2PT3JJMDt8fOWXfZ Pc7f28jicflsqcemVZ1sHjsfWnpsXlLvsftmA5tH35ZVjB6fN8kFcEZl22SkJqakFimk5iXn p2TmpdsqeQfHO8ebmhkY6hpaWpgrKeQl5qbaKrn4BOi6ZeYAnaqkUJaYUwoUCkgsLlbSt7Mp yi8tSVXIyC8usVVKLUjJKTAp0CtOzC0uzUvXy0stsTI0MDAyBSpMyM44segAe8FO3opNC74z NTB+5epi5OSQEDCRmDnlD3sXIxeHkMBuRonV25awQTifGCUWPpoGlfnMKNHceI8ZpmXZiitQ VbsYJbpOTWKFq/p1eipjFyMHB5uApsSFyaUgDSICLhJNa6eCNTALnGeUuNffBDZJWMBd4s/y lywgNouAqsS5t3fZQHp5BawkDk+qh1gmLzHz0nd2EJsTKPx9xm6wVl4BQYmTM5+AtTID1TRv nc0MMl9CYC6HRMeH/VCXukhM2L+fBcIWlnh1fAs7hC0l8fndXjYIO1ni0sxzTBB2icTjPQeh bHuJ1lP9zCD3MAP9sn6XPsQuPone30+YQMISArwSHW1CENWKEvcmPWWFsMUlHs5YAmV7SBzd MAsaiL2MEnM23GKewCg/C8kLs5C8MAth2wJG5lWMkqkFxbnpqcWmBcZ5qeXwiE3Oz93ECE6x Wt47GB89+KB3iJGJg/EQowQHs5IIL2+le4oQb0piZVVqUX58UWlOavEhRlNgEE9klhJNzgcm +bySeEMTSwMTMzMzE0tjM0MlcV5125PJQgLpiSWp2ampBalFMH1MHJxSDUzhqYLSqxnbDmdt +shmOEn5Z2OlG2dW4ys+y6QbBs4hCsqZNXcmH929Mk/bY+fJyaf9WcoDJ8xwtWXb1vTryTo3 tVXssr9XyC7InV0XFc9tmM13OvaP0o3Prv5ruqJXLOq81PN2Stz/v8aPxKuTj3I3urj8fVIx y/TQw6h/j+VO2+bd2CJl82j1/wr55fLJ7w1FduXkSkeKt/stkN2ueWXTT0NG+5eGRtmvr30P vrmO84+CxJl/cTP/nTDWag/NrFrHwbvO30pYt6Qq7MW8+RvTjd55eCdNlVI6MzfO8kSA+H3/ 1u7C2R/fJ3s0tcTNmlfo/OS41O/J6lHB639m1Xu+L3p5u+VCQkVcOe/BC0osxRmJhlrMRcWJ AFbzc6o6BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrJLMWRmVeSWpSXmKPExsWy7bCSvO7iZz4pBlcaxCw+fv3NYtE04S+z xeq7/WwWNw/sZLJYufook8W71nMsFpMOXWO02HtL22L+sqfsFutev2ex2PT3JJMDt8fOWXfZ Pc7f28jicflsqcemVZ1sHjsfWnpsXlLvsftmA5tH35ZVjB6fN8kFcEZx2aSk5mSWpRbp2yVw ZZxYdIC9YCdvxaYF35kaGL9ydTFyckgImEgsW3GFrYuRi0NIYAejxN3ZjcwQCXGJ5ms/2CFs YYmV/56zQxR9ZJQ4OH0TUAcHB5uApsSFyaUgNSICXhLtb2eBDWIWuM4ocXPldrBBwgLuEn+W v2QBsVkEVCXOvb0L1ssrYCVxeFI9xHx5iZmXvoPt4gQKf5+xmxmkREjAUqJxQTxImFdAUOLk zCdgU5iBypu3zmaewCgwC0lqFpLUAkamVYySqQXFuem5xYYFRnmp5XrFibnFpXnpesn5uZsY wbGhpbWDcc+qD3qHGJk4GA8xSnAwK4nw8la6pwjxpiRWVqUW5ccXleakFh9ilOZgURLnvdB1 Ml5IID2xJDU7NbUgtQgmy8TBKdXA5Cy53Xnx3wzTWwkRXhN3/zu/Vc5znWKy5wGPOGu/Flfp FR9a3vzcdrjFIPXxt0PGtyN7J9gsCJd6yvbz7Iq2V8/mvJHPO7tH195uv0/um70zVR7GP7oS YrB9/i2H1v/qvVUJduKrT072fCFvyrlrgl7d4ik56ydaqy8+fG31x8d9XCdtMg97zw0qbjrJ ZXX4zOKsj3Xn1NunyjVsVDUOD8kJyl2vtYm1dK0h/y3vtLPSK/65eLDtmePPk3CtNPSZC9fX DW4Te4+3pMu4P3K3mZQ6IWZ+S8P67S3vynpO+M/lfh/w9mvsoyfCyysOmG4X2ZA1e7UMd/4R S64vKycZ7L3jnRVhGvaCa9HCWeaVjUosxRmJhlrMRcWJADYdJqz8AgAA X-CMS-MailID: 20230429094259epcas5p11f0f3422eb4aa4e3ebf00e0666790efa X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094259epcas5p11f0f3422eb4aa4e3ebf00e0666790efa References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Anuj Gupta Refactor parameters of nvme_setup_prp_simple a bit. This is a prep patch. Signed-off-by: Anuj Gupta --- drivers/nvme/host/pci.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index b4498e198e8a..30d7a1a6eaab 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -750,14 +750,13 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, } static blk_status_t nvme_setup_prp_simple(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmnd, - struct bio_vec *bv) + struct nvme_iod *iod, struct nvme_rw_command *cmnd, + struct bio_vec *bv, int dma_dir) { - struct nvme_iod *iod = blk_mq_rq_to_pdu(req); unsigned int offset = bv->bv_offset & (NVME_CTRL_PAGE_SIZE - 1); unsigned int first_prp_len = NVME_CTRL_PAGE_SIZE - offset; - iod->first_dma = dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); + iod->first_dma = dma_map_bvec(dev->dev, bv, dma_dir, 0); if (dma_mapping_error(dev->dev, iod->first_dma)) return BLK_STS_RESOURCE; iod->dma_len = bv->bv_len; @@ -801,8 +800,10 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, if (!is_pci_p2pdma_page(bv.bv_page)) { if (bv.bv_offset + bv.bv_len <= NVME_CTRL_PAGE_SIZE * 2) - return nvme_setup_prp_simple(dev, req, - &cmnd->rw, &bv); + return nvme_setup_prp_simple(dev, + blk_mq_rq_to_pdu(req), + &cmnd->rw, &bv, + rq_dma_dir(req)); if (nvmeq->qid && sgl_threshold && nvme_ctrl_sgl_supported(&dev->ctrl)) From patchwork Sat Apr 29 09:39:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kanchan Joshi X-Patchwork-Id: 13226951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CD81C7EE23 for ; Sat, 29 Apr 2023 09:43:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231130AbjD2JnV (ORCPT ); Sat, 29 Apr 2023 05:43:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231126AbjD2JnT (ORCPT ); Sat, 29 Apr 2023 05:43:19 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A06213C for ; Sat, 29 Apr 2023 02:43:06 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230429094304epoutp04c18878018024ee925cfc4ca07d65a2c6~aXpBLpvQE0828308283epoutp04u for ; Sat, 29 Apr 2023 09:43:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230429094304epoutp04c18878018024ee925cfc4ca07d65a2c6~aXpBLpvQE0828308283epoutp04u DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1682761384; bh=d3noo7PR5uc2q/xqQsl9I4ZjiH2gFhIOu9fepDFlxqI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dkH0+yGiDtaZmPDeXGOk9vErTxtNuuTRLA/3adNvrX1pwAAr+fhme4BQEMpEvwFu0 xToDq/bKFePIkshLbPL5TzlPuD9MSGlN4M4aKyYky8nRPVQyTzKl8ujujWlYYI42EE AMGgrA/Ax1Px75K0tadVfNIdhrE7o/Snhx9EO/lM= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230429094303epcas5p2381f3f7bb217f5befbe78921fac9b3df~aXpAMKyhw2905329053epcas5p2N; Sat, 29 Apr 2023 09:43:03 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q7kzp23nFz4x9Pt; Sat, 29 Apr 2023 09:43:02 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 77.60.55646.6A6EC446; Sat, 29 Apr 2023 18:43:02 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230429094301epcas5p48cf45da2f83d9ca8140ee777c7446d11~aXo_k5l0I1668816688epcas5p4_; Sat, 29 Apr 2023 09:43:01 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230429094301epsmtrp2cf3e99dba583ba2eff31e2746ba61602~aXo_j9YsY3077530775epsmtrp2g; Sat, 29 Apr 2023 09:43:01 +0000 (GMT) X-AuditID: b6c32a4b-b71fa7000001d95e-03-644ce6a66daa Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id B7.29.28392.5A6EC446; Sat, 29 Apr 2023 18:43:01 +0900 (KST) Received: from localhost.localdomain (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230429094259epsmtip2b406d2d3e0d312d53ec8b113ae791d71~aXo8eZn4W0575905759epsmtip2V; Sat, 29 Apr 2023 09:42:59 +0000 (GMT) From: Kanchan Joshi To: axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kbusch@kernel.org Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, gost.dev@samsung.com, anuj1072538@gmail.com, xiaoguang.wang@linux.alibaba.com, Kanchan Joshi , Anuj Gupta Subject: [RFC PATCH 12/12] pci: implement submission/completion for rawq commands Date: Sat, 29 Apr 2023 15:09:25 +0530 Message-Id: <20230429093925.133327-13-joshi.k@samsung.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230429093925.133327-1-joshi.k@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrNJsWRmVeSWpSXmKPExsWy7bCmlu6yZz4pBo+X8Fp8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZ1S2TUZqYkpq kUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5ibqqtkotPgK5bZg7QvUoKZYk5pUChgMTi YiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLSvHS9vNQSK0MDAyNToMKE7IzWj8+YCy64 V5zbcZKpgfGCVRcjJ4eEgInEn5/rGLsYuTiEBHYzSlzqW88E4XxilDjRfokZwvnMKHG86zET TMuRLZ9ZIRK7GCUO7DrMAlc1ddMGoGEcHGwCmhIXJpeCNIgIuEg0rZ3KBlLDLPCNUWLR63es IDXCAsES7XeiQGpYBFQldky/zA5i8wpYSXxbNJkVYpm8xMxL38HinEDx7zN2M0PUCEqcnPmE BcRmBqpp3job7FIJgaUcEq+7pkA1u0i8+3yfGcIWlnh1fAs7hC0l8fndXjYIO1ni0sxzUJ+V SDzecxDKtpdoPdXPDHInM9Av63fpQ+zik+j9/YQJJCwhwCvR0SYEUa0ocW/SU6it4hIPZyyB sj0k+vcuBjtTSKCXUeLRl9wJjPKzkHwwC8kHsxCWLWBkXsUomVpQnJueWmxaYJyXWg6P1+T8 3E2M4FSr5b2D8dGDD3qHGJk4GA8xSnAwK4nw8la6pwjxpiRWVqUW5ccXleakFh9iNAUG8URm KdHkfGCyzyuJNzSxNDAxMzMzsTQ2M1QS51W3PZksJJCeWJKanZpakFoE08fEwSnVwOSZm/3g xdFbJZrMokKd0hO8Ngj+FPyw2WbDjsv1/1L7k+xdhKYdvX7mzFrlwkl/Liuf3ahSutH9cO89 vjdit7+kXVOr4ZxX1roo+3nX58YN61wcXS9P36J91niLrlPyRZ33z328txxaoPkgLcNzi2BX ELP68UbWW3I3ZC9rzFs00bJsj6LijlyNFTZ/iwVYdl/1bslrOPH9qAPThXcP1jm2cK5eMPng B2Gh14py+8IaTtcYvU3LMfl67HiP77u1No3zF3y0DPN4l/H/b2LLOabvpcaKB5YGmc9Yksp8 cN2ymFVRATlGhfMWXknXZzrjOSHvhxXj8d7Vh2Sle2TvPxblmLglUK2dXTRl2jZuzUIlluKM REMt5qLiRAB2hJK0PgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrHLMWRmVeSWpSXmKPExsWy7bCSvO7SZz4pBttnaVh8/PqbxaJpwl9m i9V3+9ksbh7YyWSxcvVRJot3redYLI7+f8tmMenQNUaLvbe0LeYve8puse71exaLTX9PMjnw eOycdZfd4/y9jSwel8+Wemxa1cnmsfOhpcfmJfUeu282sHn0bVnF6PF5k1wAZxSXTUpqTmZZ apG+XQJXRuvHZ8wFF9wrzu04ydTAeMGqi5GTQ0LAROLIls+sILaQwA5GicuzSiDi4hLN136w Q9jCEiv/PQeyuYBqPjJKnO/sYe5i5OBgE9CUuDC5FKRGRMBLov3tLDaQGmaBf4wSD963sYEk hAUCJTq3rWEBsVkEVCV2TL8MNpRXwEri26LJrBAL5CVmXvoOFucEin+fsRtsvpCApUTjgniI ckGJkzOfgI1hBipv3jqbeQKjwCwkqVlIUgsYmVYxSqYWFOem5xYbFhjlpZbrFSfmFpfmpesl 5+duYgRHiZbWDsY9qz7oHWJk4mA8xCjBwawkwstb6Z4ixJuSWFmVWpQfX1Sak1p8iFGag0VJ nPdC18l4IYH0xJLU7NTUgtQimCwTB6dUA5P14kcvMw23CDEmxEfYfjZa9m3fvM4Fx/4LsT/t KrJJkPuzIOji6/0folzrbteYltiU/btml+B2r5/L8YS57P+E681zg7rEGrenXjkqeFlnocer iibV80VZBzVOPirRklxcqVT5UXJmOPdLieV8BdY3udrz/C85FiqVPzXo+PhgzeSyzMDDMxM6 wlpyJ+cqqHuVe3VLr+MskjV4+/akZoBdNVfj6Ucb9SM+3bicwNqbMZv5lsKb21symoMvL/JN 8t39+73fk8NFeREH3Ofervn38r+DT9GrniViLyce1Jz8bvb7sJ+iOxdNlToQb2zrPaXrzuvp 4TN0912TlHgf8WB27vUfqz60ultt/mv/ebUSS3FGoqEWc1FxIgC7JmW2AQMAAA== X-CMS-MailID: 20230429094301epcas5p48cf45da2f83d9ca8140ee777c7446d11 X-Msg-Generator: CA CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230429094301epcas5p48cf45da2f83d9ca8140ee777c7446d11 References: <20230429093925.133327-1-joshi.k@samsung.com> Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Implement ->queue_uring_cmd and ->poll_uring_cmd for leaner submission and completion. Both operate on "struct io_uring_cmd" directly, without having to take hoops involving request and bio. This is currently enabled for small (single segement length), premapped (fixedbufs) IOs. Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- drivers/nvme/host/pci.c | 194 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 192 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 30d7a1a6eaab..ca0580c4e977 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include "trace.h" #include "nvme.h" @@ -210,6 +212,7 @@ struct nvme_queue { unsigned long *cmdid_bmp; spinlock_t cmdid_lock; struct nvme_iod *iod; + u8 reg_id; /* only used for poll queues: */ spinlock_t cq_poll_lock ____cacheline_aligned_in_smp; struct nvme_completion *cqes; @@ -249,7 +252,11 @@ union nvme_descriptor { * to the actual struct scatterlist. */ struct nvme_iod { - struct nvme_request req; + union { + struct nvme_request req; + /* for raw-queue */ + struct io_uring_cmd *ioucmd; + }; struct nvme_command cmd; bool aborted; s8 nr_allocations; /* PRP list pool allocations. 0 means small @@ -1025,6 +1032,13 @@ static inline void nvme_ring_cq_doorbell(struct nvme_queue *nvmeq) writel(head, nvmeq->q_db + nvmeq->dev->db_stride); } +static void nvme_pci_put_cmdid(struct nvme_queue *nvmeq, int id) +{ + spin_lock(&nvmeq->cmdid_lock); + clear_bit(id, nvmeq->cmdid_bmp); + spin_unlock(&nvmeq->cmdid_lock); +} + static inline struct blk_mq_tags *nvme_queue_tagset(struct nvme_queue *nvmeq) { if (!nvmeq->qid) @@ -1032,6 +1046,37 @@ static inline struct blk_mq_tags *nvme_queue_tagset(struct nvme_queue *nvmeq) return nvmeq->dev->tagset.tags[nvmeq->qid - 1]; } +static inline struct nvme_uring_direct_pdu *nvme_uring_cmd_direct_pdu( + struct io_uring_cmd *ioucmd) +{ + return (struct nvme_uring_direct_pdu *)&ioucmd->pdu; +} + +static inline void nvme_handle_cqe_rawq(struct nvme_queue *nvmeq, + struct nvme_completion *cqe, + __u16 command_id) +{ + struct nvme_dev *dev = nvmeq->dev; + u32 status = le16_to_cpu(cqe->status) >> 1; + u64 result = cqe->result.u64; + struct nvme_iod *iod; + int id, reg_id; + + reg_id = nvme_genctr_from_cid(command_id); + /* we should not encounter completions from past registration*/ + WARN_ON_ONCE(nvmeq->reg_id != reg_id); + + id = nvme_tag_from_cid(command_id); + iod = &nvmeq->iod[id]; + + if (iod->dma_len) + dma_unmap_page(dev->dev, iod->first_dma, iod->dma_len, + nvme_is_write(&iod->cmd) ? DMA_TO_DEVICE : DMA_FROM_DEVICE); + + nvme_pci_put_cmdid(nvmeq, id); + io_uring_cmd_done(iod->ioucmd, status, result, IO_URING_F_UNLOCKED); +} + static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, struct io_comp_batch *iob, u16 idx) { @@ -1039,6 +1084,9 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, __u16 command_id = READ_ONCE(cqe->command_id); struct request *req; + if (test_bit(NVMEQ_RAW, &nvmeq->flags)) + return nvme_handle_cqe_rawq(nvmeq, cqe, command_id); + /* * AEN requests are special as they don't time out and can * survive any kind of queue freeze and often don't respond to @@ -1151,6 +1199,25 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob) return found; } +static int nvme_poll_uring_cmd(struct io_uring_cmd *ioucmd, int qid, + struct io_comp_batch *iob) + +{ + struct nvme_uring_direct_pdu *pdu = nvme_uring_cmd_direct_pdu(ioucmd); + struct nvme_dev *dev = to_nvme_dev(pdu->ns->ctrl); + struct nvme_queue *nvmeq = &dev->queues[qid]; + bool found; + + if (!nvme_cqe_pending(nvmeq)) + return 0; + + spin_lock(&nvmeq->cq_poll_lock); + found = nvme_poll_cq(nvmeq, iob); + spin_unlock(&nvmeq->cq_poll_lock); + + return found; +} + static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) { struct nvme_dev *dev = to_nvme_dev(ctrl); @@ -1762,6 +1829,22 @@ static int nvme_pci_alloc_cmdid_bmp(struct nvme_queue *nvmeq) return 0; } +static int nvme_pci_get_cmdid(struct nvme_queue *nvmeq) +{ + int id = 0, size = nvmeq->q_depth - 1; + + spin_lock(&nvmeq->cmdid_lock); + id = find_first_zero_bit(nvmeq->cmdid_bmp, size); + if (id >= size) { + id = -EBUSY; + goto unlock; + } + set_bit(id, nvmeq->cmdid_bmp); +unlock: + spin_unlock(&nvmeq->cmdid_lock); + return id; +} + static int nvme_pci_alloc_iod_array(struct nvme_queue *nvmeq) { if (!test_bit(NVMEQ_RAW, &nvmeq->flags)) @@ -1793,13 +1876,17 @@ static int nvme_pci_register_queue(void *data) struct nvme_ns *ns = (struct nvme_ns *) data; struct nvme_dev *dev = to_nvme_dev(ns->ctrl); int qid, ret; + struct nvme_queue *nvmeq; qid = nvme_pci_get_rawq(dev); if (qid > 0) { /* setup command-id bitmap and iod array */ - ret = nvme_pci_setup_rawq(&dev->queues[qid]); + nvmeq = &dev->queues[qid]; + ret = nvme_pci_setup_rawq(nvmeq); if (ret < 0) qid = ret; + else + nvmeq->reg_id++; } return qid; } @@ -1812,6 +1899,107 @@ static int nvme_pci_unregister_queue(void *data, int qid) return nvme_pci_put_rawq(dev, qid); } +static int nvme_map_io_fb(struct request_queue *q, struct io_uring_cmd *ioucmd, + struct nvme_dev *dev, struct nvme_iod *iod, + unsigned long addr, unsigned int buf_len) +{ + struct nvme_command *cmnd = &iod->cmd; + int ret, rw = nvme_is_write(cmnd); + struct iov_iter iter; + size_t nr_iter, nr_segs; + struct bio_vec *bv; + + if (!(ioucmd->flags & IORING_URING_CMD_FIXED)) + return -EINVAL; + + ret = io_uring_cmd_import_fixed(addr, buf_len, rw, &iter, ioucmd); + if (ret < 0) + return ret; + nr_iter = iov_iter_count(&iter); + nr_segs = iter.nr_segs; + + /* device will do these checks anyway, so why do duplicate work? */ + if (!nr_iter || (nr_iter >> SECTOR_SHIFT) > queue_max_hw_sectors(q)) + goto err; + if (nr_segs > queue_max_segments(q)) + goto err; + /* this will be attempted from regular path instead */ + if (nr_segs > 1) + goto err; + + bv = (struct bio_vec *)&iter.bvec[0]; + if (bv->bv_offset + bv->bv_len <= NVME_CTRL_PAGE_SIZE * 2) + return nvme_setup_prp_simple(dev, iod, + (struct nvme_rw_command *)cmnd, + bv, rw ? DMA_TO_DEVICE : DMA_FROM_DEVICE); +err: + return -EINVAL; +} + +static int nvme_alloc_cmd_from_q(struct nvme_queue *nvmeq, int *cmdid, + struct nvme_iod **iod) +{ + int id; + + id = nvme_pci_get_cmdid(nvmeq); + if (id < 0) + return id; + *cmdid = id | nvme_cid_install_genctr(nvmeq->reg_id); + *iod = &nvmeq->iod[id]; + return 0; +} + +static int nvme_pci_queue_ucmd(struct io_uring_cmd *ioucmd, int qid) +{ + struct nvme_uring_direct_pdu *pdu = nvme_uring_cmd_direct_pdu(ioucmd); + struct nvme_ns *ns = pdu->ns; + struct nvme_ctrl *ctrl = ns->ctrl; + struct nvme_dev *dev = to_nvme_dev(ctrl); + struct nvme_command *nvme_cmd; + struct nvme_uring_data d; + int ret, cmdid; + struct nvme_iod *iod; + struct nvme_queue *nvmeq = &dev->queues[qid]; + + ret = nvme_alloc_cmd_from_q(nvmeq, &cmdid, &iod); + if (ret) + return ret; + + iod->ioucmd = ioucmd; + nvme_cmd = &iod->cmd; + ret = nvme_prep_cmd_from_ioucmd(ctrl, ns, ioucmd, nvme_cmd, &d); + if (ret) + goto out; + + nvme_cmd->common.command_id = cmdid; + iod->aborted = false; + iod->nr_allocations = -1; + iod->sgt.nents = 0; + ret = nvme_map_io_fb(ns->queue, ioucmd, dev, iod, d.addr, d.data_len); + if (ret) + goto out; + + /* + * since this nvmeq is exclusive to single submitter (io_uring + * follows one-thread-per-ring model), we do not need the lock + * ideally. But we have lifetime difference between io_uring + * and nvme sqe. io_uring SQE can be reused just after submission + * but for nvme we have to wait until completion. + * So if we run out of space, submission will be deferred to + * io_uring worker. So we can't skip the lock. + * But not all is lost! Due to one-thread-per-ring model and + * polled-completion, contention is not common in most cases. + */ + spin_lock(&nvmeq->sq_lock); + nvme_sq_copy_cmd(nvmeq, &iod->cmd); + nvme_write_sq_db(nvmeq, true); + spin_unlock(&nvmeq->sq_lock); + return -EIOCBQUEUED; +out: + nvme_pci_put_cmdid(nvmeq, cmdid); + return ret; +} + static const struct blk_mq_ops nvme_mq_admin_ops = { .queue_rq = nvme_queue_rq, .complete = nvme_pci_complete_rq, @@ -1832,6 +2020,8 @@ static const struct blk_mq_ops nvme_mq_ops = { .poll = nvme_poll, .register_queue = nvme_pci_register_queue, .unregister_queue = nvme_pci_unregister_queue, + .queue_uring_cmd = nvme_pci_queue_ucmd, + .poll_uring_cmd = nvme_poll_uring_cmd, }; static void nvme_dev_remove_admin(struct nvme_dev *dev)