From patchwork Sun Oct 27 14:21:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13852666 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55FF0D13562 for ; Sun, 27 Oct 2024 14:23:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8CF58D000C; Sun, 27 Oct 2024 10:23:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D3CC28D0001; Sun, 27 Oct 2024 10:23:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB44B8D000C; Sun, 27 Oct 2024 10:23:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9B7A38D0001 for ; Sun, 27 Oct 2024 10:23:11 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E54C9A1C58 for ; Sun, 27 Oct 2024 14:22:31 +0000 (UTC) X-FDA: 82719598290.03.9F7E1CA Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf13.hostedemail.com (Postfix) with ESMTP id EE2C82000F for ; Sun, 27 Oct 2024 14:22:46 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=eiJIF0ZF; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730038936; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hnHLxYQ+J1zQEl3MZT/YTSEtWDueiC94+si4jpjdUes=; b=qtX9nEheB7E0NmTQna+rA9HxvZ6u3MFuF3DrgmfzJxgl8x1CHWzzI6s3c2UpQLApb0iZPn 8kf+jlwOyf1BBHZsVJSx0WqnqqILFNKN7xibgN2qnaA6XDlwv1kCGnvAXP7+/1BZME+MmP nonBn5eFKVGUDERD+5H0E+qPJQt+V+o= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=eiJIF0ZF; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730038936; a=rsa-sha256; cv=none; b=PFz7VeCZIKV+Jgz0ov+bVonKOWoiayrw9ZpC5lA3QLj9lFtR0of6s+ptNZo74qJq+sY34J +KuxrW5iUCKy3m4nGeo1eZjJjpGn0Lud4MwO9TQL8z5KB2IbXPRYJBDtpYcJwId8vSSDni +CWHNOWsAu7Q6PtX+WqMPVNXh2Cf6dg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 25DD4A40FB2; Sun, 27 Oct 2024 14:21:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29F16C4CEC3; Sun, 27 Oct 2024 14:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730038989; bh=ApIelQA2oAfFsx+G9CAFnUZptSSapywa7pTJgnwTMSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eiJIF0ZFLHGKP8f4jSGly33bDKFCG2di1eSRLyav3A3c0jDbi3wFuI/4gTIyAJqsP 6BsfaifyFf9MsSzeY+5fbfpU0b6hcssFg4v/xfeYYw6VwQWGLLeo3B/keDpUA4VwIn +NmMzdDiyGF3OZZgTiwOFjAj7pV52518FgsFTHu7WSP5J++G+dgRues27OYr/5xhtM pnBlX75XHyWUZvAl9+fd09fuOyo/S0uvjf0rdnRkSmwZeoJZcK471IFO5l/4Q3+qwV p+epEDOkoend5m1LL4+a9d5gYc5UJxwnsS3xd1VeOpI9zP/h1Z21Z2tQwfEqqppaO5 FYRlcndyyO9gg== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 6/7] nvme-pci: use a better encoding for small prp pool allocations Date: Sun, 27 Oct 2024 16:21:59 +0200 Message-ID: X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: h97fauiznnhnc4cn5ac6jxjajtt4dern X-Rspamd-Queue-Id: EE2C82000F X-Rspamd-Server: rspam11 X-HE-Tag: 1730038966-425337 X-HE-Meta: U2FsdGVkX1/u71FUa0L+iNnphOTB1YG9DH3Gp9rjJvaCQEhtQtjwa2FquRjoMa3B4xTrqyC+D68rdAE4E06rHGaJbERflZTKHjN41R+6wxD+IM++N2boWqsG+28D90gDbLl/SdvZAJgPckXLN1RoQEgATei1l+7eprYhNt1v/SBzizCueeovO0Oxp91gbdOPTtPeSknyQYuNjWERZ0CI3vQ9Prce/4DyQiw9D3LQ4u4zHFe5vuWf6SiahePqj+Ys7oqygPpeIJZGqmTTH70kTiWlBU48gN2Pf9yRTX/nMIF7yxIku2T7PKY73RvU6E0eS1SLjiKPfEr6cxwuDl8CYQ0rXgORzrkLEVVDTeltWbWIGgFal4yjsV8Cs1gTUj/otrKmrjMdR26YE2Of4O9aofQS4eA4KOjM/wXfQEoVfDr/ZZmReBW0FfoErxNYvoS9qbDDepqPhVvN8Vg7MiHJiBkdmy0fxY6ZpeAqhfU1mJ8HqTdjO6eSOgjPR0nEyYQTGLuwDKo/Xbj9JovQQO0FL+q0CE6fi38kdonz9SgAk0knkrcb/xqKKV3w3EOwDhKyFnSYwOB4sY3LF7vdUuKlGkX/QmfAC+vAepJZmEMQnkyOcrkPAOnRKb/CuszgGeIXiiE99WwcbKvLloy7mOsqRsi9JoauXGMFONRhwe+ZCBWIgZOnwLbQlZDbn/WoueVI6fpSmfvb57pjDEqw1KoqnYf+uuRuDIIMxliN9Mx2MY/avHF/FRdEuCtWR6BPAWXCNKK9BaP1NGn+5BXNtcLXll4RxwTwMnlGg667Uz9Do62Y9jULVOWVm+N4pHCwnp4naTGLugkfuxeT1uIjlR40UBDffL5o+vHjUamkugKVaO9G1wTp5EmUvOF47QV/t5ZYnzrqPPX92cFoCnEl2bzzQtwjzI7ThSc1gjIZDoIa8QSeCv49e3lewPgetsI5jphzPkb62aS4/RXhAGgXsOB yCGTpy53 G+hBFi6MgBsl2DRR9m6E47NLf+cv3yqhCFn+LbYLYVABvJwIls4LREfzsUULVKwhJ4S1ebPZ52eUAhiZY0HHpRtApJLd7KlwtoyDx+1koshnFYgYhPCNhIR6pGKU7egUxPAe9JUvfrkAVGZ7J6D8iaKLxvC+CvJJZd4UXZ6qR90F1eS/gNYGgBPUPRHzohscBjpeIYonE/Q/Xjx7JVJ4lJEIixdSVXdQ4pUR1u7OjjjGEl4o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig There is plenty of unused space in the iod next to nr_descriptors. Add a separate bool (which could be condensed to a single bit once we start running out of space) to encode that the transfer is using the full page sized pool, and use a normal 0..n count for the number of descriptors. Signed-off-by: Christoph Hellwig Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 85 +++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 45 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index ba077a42cbba..79cd65a5f311 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -45,6 +45,7 @@ #define NVME_MAX_SEGS 128 #define NVME_MAX_NR_DESCRIPTORS 5 +#define NVME_SMALL_DESCRIPTOR_SIZE 256 static int use_threaded_interrupts; module_param(use_threaded_interrupts, int, 0444); @@ -224,8 +225,8 @@ struct nvme_iod { struct nvme_request req; struct nvme_command cmd; bool aborted; - /* # of PRP/SGL descriptors: (0 for small pool) */ - s8 nr_descriptors; + u8 nr_descriptors; /* # of PRP/SGL descriptors */ + bool large_descriptors; /* uses the full page sized descriptor pool */ unsigned int dma_len; /* length of single DMA segment mapping */ dma_addr_t first_dma; dma_addr_t meta_dma; @@ -514,13 +515,27 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, return true; } -static void nvme_free_prps(struct nvme_dev *dev, struct request *req) +static inline struct dma_pool *nvme_dma_pool(struct nvme_dev *dev, + struct nvme_iod *iod) +{ + if (iod->large_descriptors) + return dev->prp_page_pool; + return dev->prp_small_pool; +} + +static void nvme_free_descriptors(struct nvme_dev *dev, struct request *req) { const int last_prp = NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; struct nvme_iod *iod = blk_mq_rq_to_pdu(req); dma_addr_t dma_addr = iod->first_dma; int i; + if (iod->nr_descriptors == 1) { + dma_pool_free(nvme_dma_pool(dev, iod), iod->descriptors[0], + dma_addr); + return; + } + for (i = 0; i < iod->nr_descriptors; i++) { __le64 *prp_list = iod->descriptors[i]; dma_addr_t next_dma_addr = le64_to_cpu(prp_list[last_prp]); @@ -543,15 +558,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) WARN_ON_ONCE(!iod->sgt.nents); dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); - - if (iod->nr_descriptors == 0) - dma_pool_free(dev->prp_small_pool, iod->descriptors[0], - iod->first_dma); - else if (iod->nr_descriptors == 1) - dma_pool_free(dev->prp_page_pool, iod->descriptors[0], - iod->first_dma); - else - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); mempool_free(iod->sgt.sgl, dev->iod_mempool); } @@ -573,7 +580,6 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, struct request *req, struct nvme_rw_command *cmnd) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - struct dma_pool *pool; int length = blk_rq_payload_bytes(req); struct scatterlist *sg = iod->sgt.sgl; int dma_len = sg_dma_len(sg); @@ -581,7 +587,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, int offset = dma_addr & (NVME_CTRL_PAGE_SIZE - 1); __le64 *prp_list; dma_addr_t prp_dma; - int nprps, i; + int i; length -= (NVME_CTRL_PAGE_SIZE - offset); if (length <= 0) { @@ -603,27 +609,23 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, goto done; } - nprps = DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE); - if (nprps <= (256 / 8)) { - pool = dev->prp_small_pool; - iod->nr_descriptors = 0; - } else { - pool = dev->prp_page_pool; - iod->nr_descriptors = 1; - } + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) > + NVME_SMALL_DESCRIPTOR_SIZE / sizeof(__le64)) + iod->large_descriptors = true; - prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); - if (!prp_list) { - iod->nr_descriptors = -1; + prp_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) return BLK_STS_RESOURCE; - } - iod->descriptors[0] = prp_list; + iod->descriptors[iod->nr_descriptors++] = prp_list; iod->first_dma = prp_dma; i = 0; for (;;) { if (i == NVME_CTRL_PAGE_SIZE >> 3) { __le64 *old_prp_list = prp_list; - prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); + + prp_list = dma_pool_alloc(dev->prp_page_pool, + GFP_ATOMIC, &prp_dma); if (!prp_list) goto free_prps; iod->descriptors[iod->nr_descriptors++] = prp_list; @@ -650,7 +652,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma); return BLK_STS_OK; free_prps: - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); return BLK_STS_RESOURCE; bad_sgl: WARN(DO_ONCE(nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents), @@ -679,7 +681,6 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, struct request *req, struct nvme_rw_command *cmd) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - struct dma_pool *pool; struct nvme_sgl_desc *sg_list; struct scatterlist *sg = iod->sgt.sgl; unsigned int entries = iod->sgt.nents; @@ -694,21 +695,13 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, return BLK_STS_OK; } - if (entries <= (256 / sizeof(struct nvme_sgl_desc))) { - pool = dev->prp_small_pool; - iod->nr_descriptors = 0; - } else { - pool = dev->prp_page_pool; - iod->nr_descriptors = 1; - } + if (entries > NVME_SMALL_DESCRIPTOR_SIZE / sizeof(*sg_list)) + iod->large_descriptors = true; - sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); - if (!sg_list) { - iod->nr_descriptors = -1; + sg_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, &sgl_dma); + if (!sg_list) return BLK_STS_RESOURCE; - } - - iod->descriptors[0] = sg_list; + iod->descriptors[iod->nr_descriptors++] = sg_list; iod->first_dma = sgl_dma; nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries); @@ -834,7 +827,8 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) blk_status_t ret; iod->aborted = false; - iod->nr_descriptors = -1; + iod->nr_descriptors = 0; + iod->large_descriptors = false; iod->sgt.nents = 0; ret = nvme_setup_cmd(req->q->queuedata, req); @@ -2694,7 +2688,8 @@ static int nvme_setup_prp_pools(struct nvme_dev *dev) /* Optimisation for I/Os between 4k and 128k */ dev->prp_small_pool = dma_pool_create("prp list 256", dev->dev, - 256, 256, 0); + NVME_SMALL_DESCRIPTOR_SIZE, + NVME_SMALL_DESCRIPTOR_SIZE, 0); if (!dev->prp_small_pool) { dma_pool_destroy(dev->prp_page_pool); return -ENOMEM;