From patchwork Fri Nov 22 21:11:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Johannesmeyer X-Patchwork-Id: 13883633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89975E69193 for ; Fri, 22 Nov 2024 21:11:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE1AA6B0085; Fri, 22 Nov 2024 16:11:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A91C76B0088; Fri, 22 Nov 2024 16:11:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 959606B0089; Fri, 22 Nov 2024 16:11:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 77A3F6B0085 for ; Fri, 22 Nov 2024 16:11:49 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1A197C1528 for ; Fri, 22 Nov 2024 21:11:49 +0000 (UTC) X-FDA: 82814976384.12.BB8635B Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by imf09.hostedemail.com (Postfix) with ESMTP id A1E2F140003 for ; Fri, 22 Nov 2024 21:11:09 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iVAqQPZp; spf=pass (imf09.hostedemail.com: domain of bjohannesmeyer@gmail.com designates 209.85.208.48 as permitted sender) smtp.mailfrom=bjohannesmeyer@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732309658; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=u+tcqNNysQkTc2dEl6EX4aObuUUjMDrPgVEtTOtJS3M=; b=ajky7b9qUiQzWhEJ/eXPoSLC+UT2A55fcazIkVAVTeuBXg0CePSROSFMzprNhxY1RGV351 Qz9QcdLawTRg/Wz3dkoTOwe8vgVP+xgVr4R9j1SbbwQztIGtinKmsIi2Nx0ziU7Zhzf8v+ AMhxEOEBoNR5e6bfECrol8kshH7V2/I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iVAqQPZp; spf=pass (imf09.hostedemail.com: domain of bjohannesmeyer@gmail.com designates 209.85.208.48 as permitted sender) smtp.mailfrom=bjohannesmeyer@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732309658; a=rsa-sha256; cv=none; b=6Ai4Ee2nTdl3+Mz/6uTjoYNTzo2BzWpBnkTRmH0mcdV3uuJPCosZWd3GEkQf8j8GmtsgfZ bmMdlu9CgOh/kJUqtjGq2K0ocE1+Oar9wZng1Xylc/EfNNB9gTG0A2oai9u7ioAkQU8Dgl 2KKavgkBL2x6hZpFwhsxDQZ9zyHgU2M= Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-5cfa1ec3b94so2988495a12.2 for ; Fri, 22 Nov 2024 13:11:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732309906; x=1732914706; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=u+tcqNNysQkTc2dEl6EX4aObuUUjMDrPgVEtTOtJS3M=; b=iVAqQPZp0jByUVYHvv9G8oi/W9Kr/UzePBVaCboyDUioyqx+Ow0vjHW9v11CCIQuAA bhMkGUUgeG5duYa/2u5DvmQCFGEYXWXnBdhw0QFwM4xfpkGKIvWIg02ODkaGPgNaW63P liUeWjDueVNPYDVX0B/9ezEfex7VC8hCnuXUEFoEiCesgnEW9ari3OwwMqYNdAa/XmFK OI4+mLaDfwTuR9FR4mKAilVuD+pX1/iHTN+WGPTJAyDr0+2uvkPV3DVRSOtKyXrenXLq YfXR3lV0GDHjT6yAMLbVE6Bwsmcs7unT4PLiFheHuj9YWjd1IYaLr0FI1dFO1TjedmhH NYQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732309906; x=1732914706; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=u+tcqNNysQkTc2dEl6EX4aObuUUjMDrPgVEtTOtJS3M=; b=QBb1fytgxNBR+d2zVJA968vIcZYZ05D0maz2ROn79FCbQ+Xpw+ehYgLpqL6/2a/ASc TZ0dFHEl8Y1mKgPA8cxbILZQEPZQ+5Gaa7Pz0APIzRUHgjBO/k1GmYK2eW6dNUcjIBzm yyCgz3+tsO8OOmM2wQslsu3x62bTiV0h243Ao4FahefMwBS2kvHJwzFrxds2WuYaZauj apKVXkO4T57B+ubg/jiZ1gSzIfn9hfzCFE+qp04TOH7AmEpShfGrm8kRLwpHC+PWe3Gh YX7jiwqy6VjuwvzJrV7Gk+MXJBkak+Xjis6FJS/n9l1ReHZ1FUuT6hqSTfV0NOlqb/HQ o6MA== X-Forwarded-Encrypted: i=1; AJvYcCXxzBDZciWu1Ew0FtfFH+ehAbX1JVdxiiKvD9jJ66rruYIiC+eFIHlCfHAAbDUiEXpW0ul25FFP+g==@kvack.org X-Gm-Message-State: AOJu0YyMHcrKcvbE+1L6PEBk9Z/McpOKDC3Itavd5iu1rjemmzNI/Heg 9GradR+3XLWfrmjpbXkhhwNadxvfOY3bOJlvPa3dcZ25nSdozBWB X-Gm-Gg: ASbGncvWYotIggiECSlcFJham1FXDHgniJrDDSd1fN+DoCAH94YlL8psOj1D9HQtUj9 JtUkqAMEBtwKJoisrzHjV064MtOkHpUHmbTQaVi7w6i65pYlJo7N649WF82AU5UY+QRTWODPsWz +Nd+MwGm9rC4PsiYsp0mgad/r2006cGTbYGlwYJHMAtdBUQcD/Ki0HQkUtCQn2ceCk0M1rC7Z/t AQjA3nKZycGyTXT9J9ntAFQthwbIzD4otIxObXKPcPNwqUE0QFGJy0Z9VbHf2gkplMAN1Ec6GtV 4HI= X-Google-Smtp-Source: AGHT+IGEeoVa+pVRYbHHLYkUM3zI+FvCZ/MKAJa4ua0+3IKCCAjUzBlQr/GgpjHTIBVkdjwu7BjCMw== X-Received: by 2002:a50:ef18:0:b0:5d0:224b:d585 with SMTP id 4fb4d7f45d1cf-5d0224bd6b7mr2968510a12.31.1732309905574; Fri, 22 Nov 2024 13:11:45 -0800 (PST) Received: from rex.hwlab.vusec.net (lab-4.lab.cs.vu.nl. [192.33.36.4]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5d01d3a3d77sm1276636a12.7.2024.11.22.13.11.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Nov 2024 13:11:45 -0800 (PST) From: Brian Johannesmeyer To: Keith Busch , Christoph Hellwig , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Cc: Brian Johannesmeyer , Raphael Isemann , Cristiano Giuffrida , Herbert Bos , Greg KH Subject: [PATCH v3 3/3] dmapool: Use xarray for vaddr-to-block lookup Date: Fri, 22 Nov 2024 22:11:41 +0100 Message-Id: <20241122211144.4186080-1-bjohannesmeyer@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A1E2F140003 X-Stat-Signature: fkhmqgzfy8yddfgjehowaiyuqnuqd8bm X-Rspam-User: X-HE-Tag: 1732309869-352310 X-HE-Meta: U2FsdGVkX1/ILNid6mgf1StDgh6Tp+81ywumpVoNxMxDL1xoS3cgmrrqppkguz8HOqGneeBH+wAU1/S7fxNyiTYNA6ggs3WD+V7sBx8GVFe54qPKtqRKHxbRnS70tvhGxZy6Lm48caREfc4RtbQi7x6G8hPUhWaTzUzqi9m5SActlQb6rHdRb4+H23J2rdwt6LCtzuAi24V2e7p5VUwzN5TXND6rOjTsq52mQW6k/dhKbLZVVTCeDJVek0RrbRpnvfnkzHgYbzvpViSqlsrdTtApgUhWDu8Z3cqKi7biZ8PmSHqGX67hBPHLfO6WzwvyRyQisPhN6DBpAD54M9QD2/axZjiV//LDOwmi2p5qgeB0BfeRnxtgTkkpCxDDYUWiD4nHOif4SC4QsDw4nah2zmF3oLUprK0HgHqTmtX0o+UdfglCThEH/rgow+uoiZBygJwDJub4bjjNy/OqfvgZGlRR9WA0a0Wrur0HoHnAbIbO/bJUaj2fZ4l43s2wqVtWe7bgsSJqHMcmFYtMdlTV82uN5yuU0t4qPI+TofEIj6x6pMl6OfFxUpEaxCjZ1QLl8SEiqQFvagdQ7gGyAz9v+7WYI8r807edH228BQgnBuNaInWALHAWQAM7OoPtyG1z4wEYszOPcCx2LTsU2DlV9iuAHGO8fef7/cOQ39BTTvz2x02rN4o9Go6eAIrg5r/dcw1szlxaSf3VnZUUSXJs3GIh+gZoFjalVM6octt3I2L/fMprfVWO+jydGUATTxz4NKNv2t83tog9luyQtUMMO/94Hf5izu6AwQ9XPWezECLAQ+PtieMiI15eXCwg13actUuE5bFirFAvCjVcTJbXK4fXlYpJ8WM0numjCgrwmVkr9VSA+06efZ9i48dzxeGRXQooXITZiEaHwHSN37Qih5SFIgr8GRgs8nX58jkVeCrQfF8W+7lx1pYhCxnWx2oAajc3o/75l6NUZxtItOs GtIWr+Pt TQJH+gUaolJjECB92hAWOg4U6keDVf6AqGIG94JSP9sWr70/mJeGYjRt0tdsU38J40SugA2tfFiI0X4AMmyPuaNe+mJkbMVppuMmJU9qmsLCTqOwuKtUTwlgjmHjGkC5dhgXfSL1vpnGXOETFTu3KIgpygmFGxAxsBlieYHi+tnAy/GGmQkq8sfnnVoNt1Ukxr7rFn3cw1/dnCdKOU90wkbaI3sQSu1qjs/bpQQzVDIFsf8Z+7qJOt7wNvljZ5HMH8diB4nuFpd48IZbS4Utv7DxdtfsK9VL8KGj82zxCLbZMvMp8LJxo4LbJOKLx4VQnlNmGwZdHcBc9mnKHeboHHN/g6IOOP2NF9y6p4XXiMdUC6Qy4Xu/a89I2Gf58+fC2WXx7bxO7TcSe72NJf9swBLyIKO6HWsS7AEww X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Optimize the performance of `dma_pool_free()` by implementing an xarray to map a `vaddr` to its corresponding `block`. This eliminates the need to iterate through the entire `page_list` for vaddr-to-block translation, thereby improving performance. Performance results from the `DMAPOOL_TEST` test show the improvement. Before the patch: ``` dmapool test: size:16 align:16 blocks:8192 time:34432 dmapool test: size:64 align:64 blocks:8192 time:62262 dmapool test: size:256 align:256 blocks:8192 time:238137 dmapool test: size:1024 align:1024 blocks:2048 time:61386 dmapool test: size:4096 align:4096 blocks:1024 time:75342 dmapool test: size:68 align:32 blocks:8192 time:88243 ``` After the patch: ``` dmapool test: size:16 align:16 blocks:8192 time:37954 dmapool test: size:64 align:64 blocks:8192 time:40036 dmapool test: size:256 align:256 blocks:8192 time:41942 dmapool test: size:1024 align:1024 blocks:2048 time:10964 dmapool test: size:4096 align:4096 blocks:1024 time:6101 dmapool test: size:68 align:32 blocks:8192 time:41307 ``` This change reduces the runtime overhead, particularly for larger block sizes. Co-developed-by: Raphael Isemann Signed-off-by: Raphael Isemann Signed-off-by: Brian Johannesmeyer --- mm/dmapool.c | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index f2b96be25412..1cc2cc87ab93 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -35,6 +35,7 @@ #include #include #include +#include #ifdef CONFIG_SLUB_DEBUG_ON #define DMAPOOL_DEBUG 1 @@ -59,6 +60,7 @@ struct dma_pool { /* the pool */ unsigned int boundary; char name[32]; struct list_head pools; + struct xarray block_map; }; struct dma_page { /* cacheable header for 'allocation' bytes */ @@ -96,23 +98,7 @@ static DEVICE_ATTR_RO(pools); static struct dma_block *pool_find_block(struct dma_pool *pool, void *vaddr) { - struct dma_page *page; - size_t offset, index; - - list_for_each_entry(page, &pool->page_list, page_list) { - if (vaddr < page->vaddr) - continue; - offset = vaddr - page->vaddr; - if (offset >= pool->allocation) - continue; - - index = offset / pool->size; - if (index >= page->blocks_per_page) - return NULL; - - return &page->blocks[index]; - } - return NULL; + return xa_load(&pool->block_map, (unsigned long)vaddr); } #ifdef DMAPOOL_DEBUG @@ -273,6 +259,7 @@ struct dma_pool *dma_pool_create(const char *name, struct device *dev, retval->boundary = boundary; retval->allocation = allocation; INIT_LIST_HEAD(&retval->pools); + xa_init(&retval->block_map); /* * pools_lock ensures that the ->dma_pools list does not get corrupted. @@ -324,6 +311,12 @@ static void pool_initialise_page(struct dma_pool *pool, struct dma_page *page) block->dma = page->dma + offset; block->next_block = NULL; + if (xa_err(xa_store(&pool->block_map, (unsigned long)block->vaddr, + block, GFP_KERNEL))) { + pr_err("dma_pool: Failed to store block in xarray\n"); + return; + } + if (last) last->next_block = block; else @@ -385,6 +378,7 @@ void dma_pool_destroy(struct dma_pool *pool) if (unlikely(!pool)) return; + xa_destroy(&pool->block_map); mutex_lock(&pools_reg_lock); mutex_lock(&pools_lock); list_del(&pool->pools);