From patchwork Thu Dec 16 16:38:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12682097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E903DC4332F for ; Thu, 16 Dec 2021 16:39:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233284AbhLPQjF (ORCPT ); Thu, 16 Dec 2021 11:39:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229480AbhLPQjF (ORCPT ); Thu, 16 Dec 2021 11:39:05 -0500 Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C51B2C061574 for ; Thu, 16 Dec 2021 08:39:04 -0800 (PST) Received: by mail-il1-x130.google.com with SMTP id d14so20172226ila.1 for ; Thu, 16 Dec 2021 08:39:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/pPKkXtT6Xrl9aqdgr4UGNl+ZJEtLSBkw/CIGk0Ps8Y=; b=OrdIjvCOXY0D0TMhorH3nRbwYFZcB54ze7TN98Pgntko5rub26f5OWcC/o8nw+ATvJ ROzMEapHo1abx6igYSx+SJv5yWUw2fodYo3ajfEfxj9MofnbX3x2lrp3et8IeSYo48CS ti1UsKR7wA0orUTz52eSlAWp7e0C9m0rtZBJHbUNIq6TzSNBvArOFDLElnPV0DsZ6yDj I44eRTDecpJ1D/pLGhoMs6EEa3oNO59aebTWElSx6pYfkf+Oq99xFlTxEy7Q1d/7INDQ kC3Fdr2lz8524jQZtl0KlShO4HoqGMtJMi59DNh8isyZ0fBJuGPEjmoyjPZHnCu7/lvT m5ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/pPKkXtT6Xrl9aqdgr4UGNl+ZJEtLSBkw/CIGk0Ps8Y=; b=cSq7Zu53RWQzSEZYn7qTWyfrYSRIdazn7zWoGxQDOxpQLaXs2fYVa/bRORxoVIyQo0 0Io7LoYeinJFVTMlBVH/zu2ZJ6UKwkPkj/NYhZjcZmhlztPhk7wGgytmsD5do/AE8aJG x+W63bFiuI9V/dWH3tsLJLX3ALdCyzIZ9n8TjS7JN6mS56gKhMHrlXOu1/eaa8GuY8ha 9D+x4JbO/+xWU3hIGzTKRukXOg3etbcOBe730LwrsHBrtNsFlGZeZ4tFt9CcY8fwZPs4 wbux7dgZL47nqikky40YjjiFYylSNQrnscHzhbvfEUEoo2LDDJTJ5yLHJBzSBeOOVvXj twGw== X-Gm-Message-State: AOAM532mS1hx0nverMC6fpbCTSWhwMTidLPuWD3C9Z7YvEGvsxwMofLj kxVg7cCN+y5JuqKCg1s/85MxJA== X-Google-Smtp-Source: ABdhPJwQb6QdZqPg4FFtSRHmv+JA7S9z28uquNJFXf+nY1Sz3Gy2ArQ3OY+7RONlUp8vi3Km9w7CEQ== X-Received: by 2002:a05:6e02:15c8:: with SMTP id q8mr10367323ilu.21.1639672744121; Thu, 16 Dec 2021 08:39:04 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id t17sm71816ilm.46.2021.12.16.08.39.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 08:39:03 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe , Christoph Hellwig Subject: [PATCH 1/4] block: add mq_ops->queue_rqs hook Date: Thu, 16 Dec 2021 09:38:58 -0700 Message-Id: <20211216163901.81845-2-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211216163901.81845-1-axboe@kernel.dk> References: <20211216163901.81845-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If we have a list of requests in our plug list, send it to the driver in one go, if possible. The driver must set mq_ops->queue_rqs() to support this, if not the usual one-by-one path is used. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- block/blk-mq.c | 26 +++++++++++++++++++++++--- include/linux/blk-mq.h | 8 ++++++++ 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 75154cc788db..51991232824a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2553,6 +2553,7 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { struct blk_mq_hw_ctx *this_hctx; struct blk_mq_ctx *this_ctx; + struct request *rq; unsigned int depth; LIST_HEAD(list); @@ -2561,7 +2562,28 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) plug->rq_count = 0; if (!plug->multiple_queues && !plug->has_elevator && !from_schedule) { - struct request_queue *q = rq_list_peek(&plug->mq_list)->q; + struct request_queue *q; + + rq = rq_list_peek(&plug->mq_list); + q = rq->q; + + /* + * Peek first request and see if we have a ->queue_rqs() hook. + * If we do, we can dispatch the whole plug list in one go. We + * already know at this point that all requests belong to the + * same queue, caller must ensure that's the case. + * + * Since we pass off the full list to the driver at this point, + * we do not increment the active request count for the queue. + * Bypass shared tags for now because of that. + */ + if (q->mq_ops->queue_rqs && + !(rq->mq_hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)) { + blk_mq_run_dispatch_ops(q, + q->mq_ops->queue_rqs(&plug->mq_list)); + if (rq_list_empty(plug->mq_list)) + return; + } blk_mq_run_dispatch_ops(q, blk_mq_plug_issue_direct(plug, false)); @@ -2573,8 +2595,6 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) this_ctx = NULL; depth = 0; do { - struct request *rq; - rq = rq_list_pop(&plug->mq_list); if (!this_hctx) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 772f8f921526..550996cf419c 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -492,6 +492,14 @@ struct blk_mq_ops { */ void (*commit_rqs)(struct blk_mq_hw_ctx *); + /** + * @queue_rqs: Queue a list of new requests. Driver is guaranteed + * that each request belongs to the same queue. If the driver doesn't + * empty the @rqlist completely, then the rest will be queued + * individually by the block layer upon return. + */ + void (*queue_rqs)(struct request **rqlist); + /** * @get_budget: Reserve budget before queue request, once .queue_rq is * run, it is driver's responsibility to release the From patchwork Thu Dec 16 16:38:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12682099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E44FC433F5 for ; Thu, 16 Dec 2021 16:39:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233316AbhLPQjG (ORCPT ); Thu, 16 Dec 2021 11:39:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229480AbhLPQjF (ORCPT ); Thu, 16 Dec 2021 11:39:05 -0500 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3506C061574 for ; Thu, 16 Dec 2021 08:39:05 -0800 (PST) Received: by mail-io1-xd36.google.com with SMTP id z26so35864846iod.10 for ; Thu, 16 Dec 2021 08:39:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z+2oYvndBswSZaKsj1QzPxEnhIwscC5+Oo0mdTn6+Dc=; b=yVowYWvxaczkDintg9/OYs4mta6ur4XI8QM9V7IS/4j7hJ7IVfG/p1UtKN9Oj9HKyf dvpycSXBY66fnxl4f4ODqsuvnE0bt4Iu5ypURdLq3JdGrKjsYd9KjOII+m4kzTXzqI6c zdf5Zx4EuwHP8ls6C6eCZXSXRa+kwCD/6c1e0GKV94IgCtbghoLn7mGbCRPv/sRDOM5Q 4igld9S5Imxsd/4QUU8av6SYgW2lezEhL4kyebUmOM/+Cf//4dJ+bRJDEbiniKLSqegM gW4Mu1E2LOMKCRru5ZBvUOPnoeYj7Y/jY4qnbFbZaM7rSjmcATav3eMTpQwWCK4pLneC E44g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z+2oYvndBswSZaKsj1QzPxEnhIwscC5+Oo0mdTn6+Dc=; b=6EqWa0cDWeHJvji8u35aCcvPGHq8zeJTdqtrGsgqWCxn+i3+UA8AEpNKGPLcD0o5B7 GsQE7vYIrDTCzMpv3icbxCFHLEiavsSv0ELOv91GZr87IkoAk5fCTL2M3S2gMnQVLsqj JtDHGKW0xKZlMiXoRcY0In8G7nBoLNq4QT9d/JUpwO3wBaJ77vs15e9Qy2PAB8lDEp96 UTyPyje81FH9eMBeRsRavJDOs1zpccEnuRN2a1lK0sAq/NMulEz7UnG69GzpBamFBKob z8WS4ozK2jVC3S2KimhyF10ybL/Hu255YI6L+NucOtgxZBqs3GRpzT9NiUhWpPLFiAR6 Jcqg== X-Gm-Message-State: AOAM530QgAr5m+V+opmn8Hs2ErCTvtYtCIfjev87CzQfMvGPeqHmYG8c 4ydZcuAwvLTlwb9sbJtuKIRATA== X-Google-Smtp-Source: ABdhPJyVdd7iRMbi7TlLSgBNUbLTjF4iOc3lkxSgG43dQaU+1dqqGcxHWIIAam4lk1kutc09Uj2kjA== X-Received: by 2002:a05:6638:4087:: with SMTP id m7mr10188047jam.112.1639672745102; Thu, 16 Dec 2021 08:39:05 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id t17sm71816ilm.46.2021.12.16.08.39.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 08:39:04 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe , Chaitanya Kulkarni , Hannes Reinecke , Max Gurtovoy Subject: [PATCH 2/4] nvme: split command copy into a helper Date: Thu, 16 Dec 2021 09:38:59 -0700 Message-Id: <20211216163901.81845-3-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211216163901.81845-1-axboe@kernel.dk> References: <20211216163901.81845-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We'll need it for batched submit as well. Since we now have a copy helper, get rid of the nvme_submit_cmd() wrapper. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Hannes Reinecke Reviewed-by: Max Gurtovoy Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 8637538f3fd5..9d2a36de228a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -500,22 +500,13 @@ static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq) nvmeq->last_sq_tail = nvmeq->sq_tail; } -/** - * nvme_submit_cmd() - Copy a command into a queue and ring the doorbell - * @nvmeq: The queue to use - * @cmd: The command to send - * @write_sq: whether to write to the SQ doorbell - */ -static void nvme_submit_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd, - bool write_sq) +static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, + struct nvme_command *cmd) { - spin_lock(&nvmeq->sq_lock); - memcpy(nvmeq->sq_cmds + (nvmeq->sq_tail << nvmeq->sqes), - cmd, sizeof(*cmd)); + memcpy(nvmeq->sq_cmds + (nvmeq->sq_tail << nvmeq->sqes), cmd, + sizeof(*cmd)); if (++nvmeq->sq_tail == nvmeq->q_depth) nvmeq->sq_tail = 0; - nvme_write_sq_db(nvmeq, write_sq); - spin_unlock(&nvmeq->sq_lock); } static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) @@ -957,7 +948,10 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, } blk_mq_start_request(req); - nvme_submit_cmd(nvmeq, cmnd, bd->last); + spin_lock(&nvmeq->sq_lock); + nvme_sq_copy_cmd(nvmeq, &iod->cmd); + nvme_write_sq_db(nvmeq, bd->last); + spin_unlock(&nvmeq->sq_lock); return BLK_STS_OK; out_unmap_data: nvme_unmap_data(dev, req); @@ -1140,7 +1134,11 @@ static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl) c.common.opcode = nvme_admin_async_event; c.common.command_id = NVME_AQ_BLK_MQ_DEPTH; - nvme_submit_cmd(nvmeq, &c, true); + + spin_lock(&nvmeq->sq_lock); + nvme_sq_copy_cmd(nvmeq, &c); + nvme_write_sq_db(nvmeq, true); + spin_unlock(&nvmeq->sq_lock); } static int adapter_delete_queue(struct nvme_dev *dev, u8 opcode, u16 id) From patchwork Thu Dec 16 16:39:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12682101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D2FDC43219 for ; Thu, 16 Dec 2021 16:39:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233339AbhLPQjH (ORCPT ); Thu, 16 Dec 2021 11:39:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbhLPQjG (ORCPT ); Thu, 16 Dec 2021 11:39:06 -0500 Received: from mail-il1-x12c.google.com (mail-il1-x12c.google.com [IPv6:2607:f8b0:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD602C061574 for ; Thu, 16 Dec 2021 08:39:06 -0800 (PST) Received: by mail-il1-x12c.google.com with SMTP id w1so22531018ilh.9 for ; Thu, 16 Dec 2021 08:39:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5IX74MWW+MmXrMtixah+OlhoboMuiSi2pn/mLZENDuU=; b=h+xv2L16XaQ7XBDfUh+cBCV9gZOt0jGlTL7hJlxewlVZdF/KeFEOAg7wqeIW1HM6ie jgXXOhESr35rI4HJ4X+Lu956zTOfva+wHc9X+OS+cHwQDYQQsiOlCjGGQNXxxxXL3I6Z KR8I0USQ0qocIEMrJdMij14zZSRyEl99f4ZwWWMeKgVe2FRjrLq7H62gvxYedEf7BfX8 UmznztDNj7Rca2f7nqJ3qajucZsF2SZ9IXxANYRiCevLitEFfsZyIeWMZW6HlGpKh5mR rgjQnaQhIu5ogeO3wLn88mcMp1igflNUVn8bUtGxPxVpVFJhfl5cYUEJ6Hmk4t1EPSK5 jZOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5IX74MWW+MmXrMtixah+OlhoboMuiSi2pn/mLZENDuU=; b=eqsomQg/9w5FufKbHyIUtp8y4bgGzZGP/sh9afHf2u+LSBGRcXLVcliVGXErMztqsY g3GqOWafjQ56NaxKEzebPIuWTSVwxPqy0SACdUmdJlFI711/yLh72JibvPQk11Kc+vF7 FJyeNIdJpiglGJCbf5f63b+b728LDUcXyJvw1rJtTM7vzpRh5I2FBn2xJ2biD5M090U6 eoHGc17xtAxXiSruV6igtUPN8VpvRLz6W0+A2BX2G6QcbDU2fj1HGn4QkFp3ew3Vdz+I kie3FkVDW1kNi/3dGRHkF9QaoHezchiOlS91n3/UH4IsVvnsKVt8PctB8Lr2duMQGF2D 2k7Q== X-Gm-Message-State: AOAM531JkykxB2f0fm+5NpEHXpX4IjfG2Z70qZVw9XXVV7aiQrs3pjY3 hDDee42/6+78NDPZ/7jNdSErSA== X-Google-Smtp-Source: ABdhPJypT/FJAV4gbG5r+26xb+eg8kD4Wm9SRoUbE/cBV9XcOAGBXdJ3GFjTL1jOtWhomlwQk+8iqg== X-Received: by 2002:a05:6e02:1528:: with SMTP id i8mr10122850ilu.312.1639672746003; Thu, 16 Dec 2021 08:39:06 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id t17sm71816ilm.46.2021.12.16.08.39.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 08:39:05 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe , Hannes Reinecke , Christoph Hellwig Subject: [PATCH 3/4] nvme: separate command prep and issue Date: Thu, 16 Dec 2021 09:39:00 -0700 Message-Id: <20211216163901.81845-4-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211216163901.81845-1-axboe@kernel.dk> References: <20211216163901.81845-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a nvme_prep_rq() helper to setup a command, and nvme_queue_rq() is adapted to use this helper. Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 63 +++++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 27 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 9d2a36de228a..7062128c8204 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -903,55 +903,32 @@ static blk_status_t nvme_map_metadata(struct nvme_dev *dev, struct request *req, return BLK_STS_OK; } -/* - * NOTE: ns is NULL when called on the admin queue. - */ -static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, - const struct blk_mq_queue_data *bd) +static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) { - struct nvme_ns *ns = hctx->queue->queuedata; - struct nvme_queue *nvmeq = hctx->driver_data; - struct nvme_dev *dev = nvmeq->dev; - struct request *req = bd->rq; struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - struct nvme_command *cmnd = &iod->cmd; blk_status_t ret; iod->aborted = 0; iod->npages = -1; iod->nents = 0; - /* - * We should not need to do this, but we're still using this to - * ensure we can drain requests on a dying queue. - */ - if (unlikely(!test_bit(NVMEQ_ENABLED, &nvmeq->flags))) - return BLK_STS_IOERR; - - if (!nvme_check_ready(&dev->ctrl, req, true)) - return nvme_fail_nonready_command(&dev->ctrl, req); - - ret = nvme_setup_cmd(ns, req); + ret = nvme_setup_cmd(req->q->queuedata, req); if (ret) return ret; if (blk_rq_nr_phys_segments(req)) { - ret = nvme_map_data(dev, req, cmnd); + ret = nvme_map_data(dev, req, &iod->cmd); if (ret) goto out_free_cmd; } if (blk_integrity_rq(req)) { - ret = nvme_map_metadata(dev, req, cmnd); + ret = nvme_map_metadata(dev, req, &iod->cmd); if (ret) goto out_unmap_data; } blk_mq_start_request(req); - spin_lock(&nvmeq->sq_lock); - nvme_sq_copy_cmd(nvmeq, &iod->cmd); - nvme_write_sq_db(nvmeq, bd->last); - spin_unlock(&nvmeq->sq_lock); return BLK_STS_OK; out_unmap_data: nvme_unmap_data(dev, req); @@ -960,6 +937,38 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } +/* + * NOTE: ns is NULL when called on the admin queue. + */ +static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, + const struct blk_mq_queue_data *bd) +{ + struct nvme_queue *nvmeq = hctx->driver_data; + struct nvme_dev *dev = nvmeq->dev; + struct request *req = bd->rq; + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + blk_status_t ret; + + /* + * We should not need to do this, but we're still using this to + * ensure we can drain requests on a dying queue. + */ + if (unlikely(!test_bit(NVMEQ_ENABLED, &nvmeq->flags))) + return BLK_STS_IOERR; + + if (unlikely(!nvme_check_ready(&dev->ctrl, req, true))) + return nvme_fail_nonready_command(&dev->ctrl, req); + + ret = nvme_prep_rq(dev, req); + if (unlikely(ret)) + return ret; + spin_lock(&nvmeq->sq_lock); + nvme_sq_copy_cmd(nvmeq, &iod->cmd); + nvme_write_sq_db(nvmeq, bd->last); + spin_unlock(&nvmeq->sq_lock); + return BLK_STS_OK; +} + static __always_inline void nvme_pci_unmap_rq(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); From patchwork Thu Dec 16 16:39:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12682103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EF1DC4167B for ; Thu, 16 Dec 2021 16:39:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233333AbhLPQjH (ORCPT ); Thu, 16 Dec 2021 11:39:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233361AbhLPQjH (ORCPT ); Thu, 16 Dec 2021 11:39:07 -0500 Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44005C061574 for ; Thu, 16 Dec 2021 08:39:07 -0800 (PST) Received: by mail-io1-xd2f.google.com with SMTP id z26so35864942iod.10 for ; Thu, 16 Dec 2021 08:39:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xEIyIeQItOxJNAK+GZxR0AU9fLDjbQGibSW3G09wzOM=; b=zRA+kbRsHGd+7od2RWLlx3FvBE1xCRl1+3C5MhHDQ+HqHJNy0xJHOFBUOHy1+O/22d 8LJAxe/jM0DL4lXc3HETGaIjv17lonI5G2vIzdR5O6DOKsu0v9qPob2Nds760/l/SCVl 3YtccMlZD0Etp5r1+mQtN6VkR+2CZPl7qIw5jLJUIDxb3fSO0QHV1u9Dklaa7vcf8Qmf msFaBGJtzHIa0MmvRLZvUJx2ZRI9Qf1xFyoXZU/5DHjXHb39VdfOZ+uQfpXHv/jQs2QD YomC17TZGd3Cg2BBGj/dvIoBgUCqOcGHq80iBjaz2DlIwzdnRFL7KJO6UHrw8EJoOgW6 wbmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xEIyIeQItOxJNAK+GZxR0AU9fLDjbQGibSW3G09wzOM=; b=PZ2xEhE5XoU69L3lundGEhqVyaA2tL1E8gNPd5LnMk6QlJbKG0M4tiX0aG2vnYpOmw Pxg7T/TqSlr+sCHgR2ZWVkGev1V7pl5xCxDpT5fnkpb4VG769bMh49WbyfsVo8mcQwNz 7ITXMhh+emsCZECESdTtYsKYqyuLTI1c6nM/nHmyno0oKqqh595LOrYUJW4oy7fSUPVV w0vWu169An9HflwO4rBeRJvLVv/aavZ9DD4O4L2746bsJ2iab4nM7kIEj39VraZSLHAu gHxLapWv0HJt/ixAh/Ak2E/QCV1BMRBetU07XYJ0cMmFlm79MnQeMMkQiF0wqskklRIv mI6g== X-Gm-Message-State: AOAM531l66DtOHryPZ9uWK6e5KekNfhDGbl3TQB5s7xGfBH1QQ8pQ8o4 8aEa4dlxtHA9kqzDOayb6SnMzw== X-Google-Smtp-Source: ABdhPJwab3tMpsiybMjm9sus1v4EzqXqt6Hcyi92T+FXmNaQbb7mkeQlqpuplIBRFp6xzC+i0g+knw== X-Received: by 2002:a05:6638:1923:: with SMTP id p35mr10202987jal.16.1639672746662; Thu, 16 Dec 2021 08:39:06 -0800 (PST) Received: from x1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id t17sm71816ilm.46.2021.12.16.08.39.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Dec 2021 08:39:06 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe , Hannes Reinecke , Keith Busch Subject: [PATCH 4/4] nvme: add support for mq_ops->queue_rqs() Date: Thu, 16 Dec 2021 09:39:01 -0700 Message-Id: <20211216163901.81845-5-axboe@kernel.dk> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20211216163901.81845-1-axboe@kernel.dk> References: <20211216163901.81845-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This enables the block layer to send us a full plug list of requests that need submitting. The block layer guarantees that they all belong to the same queue, but we do have to check the hardware queue mapping for each request. If errors are encountered, leave them in the passed in list. Then the block layer will handle them individually. This is good for about a 4% improvement in peak performance, taking us from 9.6M to 10M IOPS/core. Reviewed-by: Hannes Reinecke Reviewed-by: Keith Busch Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 59 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7062128c8204..51a903d91d92 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -969,6 +969,64 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_OK; } +static void nvme_submit_cmds(struct nvme_queue *nvmeq, struct request **rqlist) +{ + spin_lock(&nvmeq->sq_lock); + while (!rq_list_empty(*rqlist)) { + struct request *req = rq_list_pop(rqlist); + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + + nvme_sq_copy_cmd(nvmeq, &iod->cmd); + } + nvme_write_sq_db(nvmeq, true); + spin_unlock(&nvmeq->sq_lock); +} + +static bool nvme_prep_rq_batch(struct nvme_queue *nvmeq, struct request *req) +{ + /* + * We should not need to do this, but we're still using this to + * ensure we can drain requests on a dying queue. + */ + if (unlikely(!test_bit(NVMEQ_ENABLED, &nvmeq->flags))) + return false; + if (unlikely(!nvme_check_ready(&nvmeq->dev->ctrl, req, true))) + return false; + + req->mq_hctx->tags->rqs[req->tag] = req; + return nvme_prep_rq(nvmeq->dev, req) == BLK_STS_OK; +} + +static void nvme_queue_rqs(struct request **rqlist) +{ + struct request *req = rq_list_peek(rqlist), *prev = NULL; + struct request *requeue_list = NULL; + + do { + struct nvme_queue *nvmeq = req->mq_hctx->driver_data; + + if (!nvme_prep_rq_batch(nvmeq, req)) { + /* detach 'req' and add to remainder list */ + if (prev) + prev->rq_next = req->rq_next; + rq_list_add(&requeue_list, req); + } else { + prev = req; + } + + req = rq_list_next(req); + if (!req || (prev && req->mq_hctx != prev->mq_hctx)) { + /* detach rest of list, and submit */ + if (prev) + prev->rq_next = NULL; + nvme_submit_cmds(nvmeq, rqlist); + *rqlist = req; + } + } while (req); + + *rqlist = requeue_list; +} + static __always_inline void nvme_pci_unmap_rq(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -1670,6 +1728,7 @@ static const struct blk_mq_ops nvme_mq_admin_ops = { static const struct blk_mq_ops nvme_mq_ops = { .queue_rq = nvme_queue_rq, + .queue_rqs = nvme_queue_rqs, .complete = nvme_pci_complete_rq, .commit_rqs = nvme_commit_rqs, .init_hctx = nvme_init_hctx,