From patchwork Mon Oct 18 12:49:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12566423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5343EC433FE for ; Mon, 18 Oct 2021 12:49:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 395C5610A1 for ; Mon, 18 Oct 2021 12:49:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231713AbhJRMvu (ORCPT ); Mon, 18 Oct 2021 08:51:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231655AbhJRMvt (ORCPT ); Mon, 18 Oct 2021 08:51:49 -0400 Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC082C06161C for ; Mon, 18 Oct 2021 05:49:38 -0700 (PDT) Received: by mail-io1-xd30.google.com with SMTP id r134so16052441iod.11 for ; Mon, 18 Oct 2021 05:49:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X/I+4ccwjP8RhmzCWaXlqxXJNfMZ+0DfyIWdXDVqb7Q=; b=ixk2v9cfH0i6p+4pK9mFnlLN9NdX1fR3HHLd1KB7nXoeQQ0c7kU3o6Aw9dhQuB/cr1 /nx03bFSRwPhloUc+LUvzTfuL23lOvkEBWTXdA3k5y0YvQGob9Tj+IRdh3wZSHMwlQKf IjyWlKHYx9jsa7ZCirpNnORa8s8OOqp7kv4UYwYTjvsdTVXeGRn0Ru7kw5fKLJIc7zIq roQEpIv4nGQ8NY0We7iKpXi6kcQLuP4yj8Ai6AZoTww5fM5CnVlfVaBNWBmR6fXly3t3 VmbmvwwhWu44I6MAJnbssMMck6h0WcbU1sJZ9xkrj/9ufejw0U1mepdLmz9IN8YrJncI SrHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X/I+4ccwjP8RhmzCWaXlqxXJNfMZ+0DfyIWdXDVqb7Q=; b=XhvDgpSJ6Sz8WJJ3RF+47CCraXu7yS8HV8q8enjcVgmMR1CFVgam1ZtfGTQ+6tdOr6 Ux8EQvWJIHo7X7n18t3AFKt973dDLG9FmRaVej7mTeTf+Ahgm2+3p81+GtBm/PM8nxA4 kRvoQAFtrdEGUklMY5L7TruAEOmcPFUQTEGd4S+CV21m8Su1ZZrqt9r1beXYP58xz87A pGJfnnXIqDFss5GmAgzkRjZnnXYHjXUPlAf4Mc22xn0tlqmpiY0eQP6em/9W5Ax0kWYu qnrrbLLw9YEI/sAcIQ+mytz4F3I+2ZNNOFLDMhV9hbTgpqTDKdlM3rn6V2rdUTyOyA1d J7Tw== X-Gm-Message-State: AOAM531tDj9wvlPMfQWSdxb0jqTpa2GUkdEFzUJ/BFtZ/a/tkMUHLOtP 0iHzOpLgaZ8ewLNea650qml2C9mnjHsK2g== X-Google-Smtp-Source: ABdhPJxX3v0BcpqIEype8haPnxUHNoiCHzqVNcjO/sOY9lSlwFejDGGV/Rq8HWUdXxaDamn/5nOryw== X-Received: by 2002:a6b:ee0d:: with SMTP id i13mr14062191ioh.166.1634561378245; Mon, 18 Oct 2021 05:49:38 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id z26sm6780081ioe.9.2021.10.18.05.49.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 05:49:37 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 1/2] nvme: move command clear into the various setup helpers Date: Mon, 18 Oct 2021 06:49:33 -0600 Message-Id: <20211018124934.235658-2-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018124934.235658-1-axboe@kernel.dk> References: <20211018124934.235658-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We don't have to worry about doing extra memsets by moving it outside the protection of RQF_DONTPREP, as nvme doesn't do partial completions. This is in preparation for making the read/write fast path not do a full memset of the command. Signed-off-by: Jens Axboe --- drivers/nvme/host/core.c | 11 ++++++++--- drivers/nvme/host/zns.c | 2 ++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index ae15cb714596..7944ad52f213 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -874,6 +874,7 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, return BLK_STS_IOERR; } + memset(cmnd, 0, sizeof(*cmnd)); cmnd->dsm.opcode = nvme_cmd_dsm; cmnd->dsm.nsid = cpu_to_le32(ns->head->ns_id); cmnd->dsm.nr = cpu_to_le32(segments - 1); @@ -890,6 +891,8 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, static inline blk_status_t nvme_setup_write_zeroes(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { + memset(cmnd, 0, sizeof(*cmnd)); + if (ns->ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES) return nvme_setup_discard(ns, req, cmnd); @@ -914,6 +917,8 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, u16 control = 0; u32 dsmgmt = 0; + memset(cmnd, 0, sizeof(*cmnd)); + if (req->cmd_flags & REQ_FUA) control |= NVME_RW_FUA; if (req->cmd_flags & (REQ_FAILFAST_DEV | REQ_RAHEAD)) @@ -982,17 +987,17 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) struct nvme_ctrl *ctrl = nvme_req(req)->ctrl; blk_status_t ret = BLK_STS_OK; - if (!(req->rq_flags & RQF_DONTPREP)) { + if (!(req->rq_flags & RQF_DONTPREP)) nvme_clear_nvme_request(req); - memset(cmd, 0, sizeof(*cmd)); - } switch (req_op(req)) { case REQ_OP_DRV_IN: case REQ_OP_DRV_OUT: /* these are setup prior to execution in nvme_init_request() */ + memset(cmd, 0, sizeof(*cmd)); break; case REQ_OP_FLUSH: + memset(cmd, 0, sizeof(*cmd)); nvme_setup_flush(ns, cmd); break; case REQ_OP_ZONE_RESET_ALL: diff --git a/drivers/nvme/host/zns.c b/drivers/nvme/host/zns.c index d95010481fce..bfc259e0d7b8 100644 --- a/drivers/nvme/host/zns.c +++ b/drivers/nvme/host/zns.c @@ -233,6 +233,8 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector, blk_status_t nvme_setup_zone_mgmt_send(struct nvme_ns *ns, struct request *req, struct nvme_command *c, enum nvme_zone_mgmt_action action) { + memset(c, 0, sizeof(*c)); + c->zms.opcode = nvme_cmd_zone_mgmt_send; c->zms.nsid = cpu_to_le32(ns->head->ns_id); c->zms.slba = cpu_to_le64(nvme_sect_to_lba(ns, blk_rq_pos(req))); From patchwork Mon Oct 18 12:49:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12566425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F3BCC433EF for ; Mon, 18 Oct 2021 12:49:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 03BFF610A1 for ; Mon, 18 Oct 2021 12:49:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231712AbhJRMvv (ORCPT ); Mon, 18 Oct 2021 08:51:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231655AbhJRMvu (ORCPT ); Mon, 18 Oct 2021 08:51:50 -0400 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEE22C06161C for ; Mon, 18 Oct 2021 05:49:39 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id y67so16028293iof.10 for ; Mon, 18 Oct 2021 05:49:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yml7Ptd8JydBCH2SbrqO9bF1kUkIRHbJ+XeaaVED5To=; b=4VIvy5swV+y4YUcs/tqI50vjVuCtOnsrvby3VbHcH/lg7iQkEOFhV4Js1j40BSKG5n ACLv0jmBNWw6rfvvmrWqyzvi7+FF7E5yuGdgp9RUvAbclh0ODJVu/QKy+BocFYpT0dXV RP5xWts9hb+kpYQN52LfvfzRujH2lAD5uRLKJSZcTv8hfEKcaisxV4RRWbjH0Cak7NEH LXk/xyR8h15PyrDSf0iNuzNphBNgwTSV10BQR8BFn0H36jKoL4WBRgKj2Q2j+aNJ14vL q8PC37KV7wtw1koL5iNkpF2KzgDs0qob003VPtaxD4eU3Wuz5koyiqgf9YLayZ1RdlSB JZpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yml7Ptd8JydBCH2SbrqO9bF1kUkIRHbJ+XeaaVED5To=; b=VaFVF0bwU+2eDVqr8s+v+W7Poty41JhiDGcgWEM88+s2DaROEm8kK/9UCJcM9/HbXx OiMttKOdtcLKGKxWaQNBvWPCW6lKJxkhkk1CutFmmJKN85q3k6Dsg4pqxMXU/rNPDe2S Dte0YfG77vKKLcp1VmRDAxzOZaLcMPV1VcXyz31NOJH3bNAvED9MTkEyiN7eHI26WkI4 7q0t6sxFoAdmTNelbv1V9H3kq8RzT7rjbTOxP4ZOCo8NFNuQBCk01Lp61Qwam99AO2Eg +fuAjhG0HjjhVBVmM20O/H5noq2xLlyj9/Wr3FpJlYMxoGOFL388yuo8F+7OBx847QK5 +ziQ== X-Gm-Message-State: AOAM531ctbmCrPV0yS8JLC1j3EmzqyqRFhPVlMWsTC8KlOo/0J1ojlN/ tC7uof7q4pJ1olQnYhgWrcYFF63F+aPMUQ== X-Google-Smtp-Source: ABdhPJzl2oC6JWYUugXRX9NCp32xwd+TjJejvJrKCNTIXW1QMY7ZTJTYx9zdDLzzoVF71XauagSUrw== X-Received: by 2002:a05:6602:1343:: with SMTP id i3mr14189514iov.58.1634561379101; Mon, 18 Oct 2021 05:49:39 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id z26sm6780081ioe.9.2021.10.18.05.49.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 05:49:38 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe , Keith Busch Subject: [PATCH 2/2] nvme: don't memset() the normal read/write command Date: Mon, 18 Oct 2021 06:49:34 -0600 Message-Id: <20211018124934.235658-3-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018124934.235658-1-axboe@kernel.dk> References: <20211018124934.235658-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This memset in the fast path costs a lot of cycles on my setup. Here's a top-of-profile of doing ~6.7M IOPS: + 5.90% io_uring [nvme] [k] nvme_queue_rq + 5.32% io_uring [nvme_core] [k] nvme_setup_cmd + 5.17% io_uring [kernel.vmlinux] [k] io_submit_sqes + 4.97% io_uring [kernel.vmlinux] [k] blkdev_direct_IO and a perf diff with this patch: 0.92% +4.40% [nvme_core] [k] nvme_setup_cmd reducing it from 5.3% to only 0.9%. This takes it from the 2nd most cycle consumer to something that's mostly irrelevant. Acked-by: Keith Busch Signed-off-by: Jens Axboe Reviewed-by: Keith Busch --- drivers/nvme/host/core.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 7944ad52f213..3e691354598c 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -917,8 +917,6 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, u16 control = 0; u32 dsmgmt = 0; - memset(cmnd, 0, sizeof(*cmnd)); - if (req->cmd_flags & REQ_FUA) control |= NVME_RW_FUA; if (req->cmd_flags & (REQ_FAILFAST_DEV | REQ_RAHEAD)) @@ -928,9 +926,15 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, dsmgmt |= NVME_RW_DSM_FREQ_PREFETCH; cmnd->rw.opcode = op; + cmnd->rw.flags = 0; cmnd->rw.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->rw.rsvd2 = 0; + cmnd->rw.metadata = 0; cmnd->rw.slba = cpu_to_le64(nvme_sect_to_lba(ns, blk_rq_pos(req))); cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1); + cmnd->rw.reftag = 0; + cmnd->rw.apptag = 0; + cmnd->rw.appmask = 0; if (req_op(req) == REQ_OP_WRITE && ctrl->nr_streams) nvme_assign_write_stream(ctrl, req, &control, &dsmgmt);