From patchwork Wed Oct 5 18:03:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12999544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E16F8C4332F for ; Wed, 5 Oct 2022 18:03:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D3E16B0073; Wed, 5 Oct 2022 14:03:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7839E6B0074; Wed, 5 Oct 2022 14:03:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64C066B0075; Wed, 5 Oct 2022 14:03:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 549296B0073 for ; Wed, 5 Oct 2022 14:03:56 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0F899160FD7 for ; Wed, 5 Oct 2022 18:03:56 +0000 (UTC) X-FDA: 79987669272.10.EEE40BA Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf13.hostedemail.com (Postfix) with ESMTP id 4A2262002E for ; Wed, 5 Oct 2022 18:03:54 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id pq16so1016944pjb.2 for ; Wed, 05 Oct 2022 11:03:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iroUAKdZ7R5Gx/z/KE5xhjM/Ew8lqKEyV5onJIu1MHY=; b=PIZwg3d0nAqXGCHGip9wpna0F2Rq/W9uZxN7XaeU73FBRauDrwoRGXRkihEuzQ2Fmj +8lBXmjefIoKl6B1WKFOLC1dAS3BqACPb+YgeUCRA5PYSduG8UnZBOjq2+jh1rVRc+hh csj0HpN3+6Fy9tmHtYNYQBKgSf/uTRmMOUWsDLvFQ+MQRA1psom1e8m54Vy6VYYdqC8A S3Bv5I6D3Z9jOE+HSoLYHjLwmIC4QZ3VNvh7AVl2SQC64Atje1J1AdlpNCfpW7liChLB 8HUzvPnEkYb+RyJrdCMB1YJCUfUNLD6IKt2XY73Fk5NCCYaia2pHQJOxD6LCExHNm4rJ vEAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iroUAKdZ7R5Gx/z/KE5xhjM/Ew8lqKEyV5onJIu1MHY=; b=Fi8fcXe5j2ZrEc2VlP553CqT8LyBjNEcRvNVM3teZ+zmBR5D4bmJNkLNMBZ73rjA6k qanqgKrBzc9tJiWdwO8VMt7gMGlkOYJx/NWM1FBm4lHxL/dV2N9EWQlFd08COY3RiyQr 5u0CqvvJ8Gx8eOwaYeTjbF9Hgc52frdh7XrZNQtmeKrUYDPSSBSf+DKbakvshCdAn3JA HQX570+fNSxq7J2T7NUQFkoOuRjocoPhaWs5zOgO0EdWiLS8HU4FmvoYG6JfcWYbL9eQ 6SOillJ1IDU6y6RUfu7LoXMb2GT4i4s6H+eezSHuri/ZMH1sBggqmSDtNYW8IdEZAQnb c+HQ== X-Gm-Message-State: ACrzQf2eST0Fo5S924dNodak/MhihIYi11Rcv86pjRcz1RVQClhwGnBl ZzLR7nN7kYx/0ieRSyZt2Gk= X-Google-Smtp-Source: AMsMyM7sKKoKZPF7OajApXBcH9gnY/xelKkIALaQi5/UQI5v92qoBDhlVKNCJ1myXqDizlCR9tzFmw== X-Received: by 2002:a17:90b:38cb:b0:203:100:bb53 with SMTP id nn11-20020a17090b38cb00b002030100bb53mr944531pjb.107.1664993034172; Wed, 05 Oct 2022 11:03:54 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id y17-20020a170903011100b001788494b764sm10674639plc.231.2022.10.05.11.03.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:03:53 -0700 (PDT) From: Yang Shi To: mgorman@techsingularity.net, agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] mm: mempool: extract common initialization code Date: Wed, 5 Oct 2022 11:03:38 -0700 Message-Id: <20221005180341.1738796-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20221005180341.1738796-1-shy828301@gmail.com> References: <20221005180341.1738796-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664993035; a=rsa-sha256; cv=none; b=4wn2Jm8wRpjfVvVPjeKkomv6HgpLw7uG6gsrM3Ttr2hX7B396Xejkwml+JY3yanoFS9Ozx pKoI29yiOktx/0p6+FonMBWMDpjn3V1HEUHGoGuuVpyPuJV7jfG7OyViYiNq7e7g+lCOI4 18dqixK6WVsGjENs02wN2tZZgXl5Rbo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PIZwg3d0; spf=pass (imf13.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664993035; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iroUAKdZ7R5Gx/z/KE5xhjM/Ew8lqKEyV5onJIu1MHY=; b=0su1aLWJyFu+AlH/mUzvY2npdRTvmgtMZxgzXYjvn03iPEh2tU4SGVfhY4b9CLYQqCdwzS wwOwaXSPRvjCG1nR0ddWp47BERRK4Cs4nnv4CGMv1BO5o6JCquObrbA8C3aGIej65erUmb tqPmeliSrZB1FfGUNs4Ujsqks92A7lM= X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PIZwg3d0; spf=pass (imf13.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: d9ephk9oc4n6kpuaj8p43p6x8r8zzjsa X-Rspamd-Queue-Id: 4A2262002E X-Rspamd-Server: rspam01 X-HE-Tag: 1664993034-616749 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extract the common initialization code to __mempool_init() and __mempool_create(). This will make adding mempool bulk init code easier. Signed-off-by: Yang Shi --- mm/mempool.c | 57 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/mm/mempool.c b/mm/mempool.c index 96488b13a1ef..ba32151f3843 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -176,9 +176,10 @@ void mempool_destroy(mempool_t *pool) } EXPORT_SYMBOL(mempool_destroy); -int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +static inline int __mempool_init(mempool_t *pool, int min_nr, + mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { spin_lock_init(&pool->lock); pool->min_nr = min_nr; @@ -208,6 +209,14 @@ int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, return 0; } + +int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + return __mempool_init(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id); +} EXPORT_SYMBOL(mempool_init_node); /** @@ -227,12 +236,31 @@ EXPORT_SYMBOL(mempool_init_node); int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data) { - return mempool_init_node(pool, min_nr, alloc_fn, free_fn, - pool_data, GFP_KERNEL, NUMA_NO_NODE); + return __mempool_init(pool, min_nr, alloc_fn, free_fn, + pool_data, GFP_KERNEL, NUMA_NO_NODE); } EXPORT_SYMBOL(mempool_init); +static mempool_t *__mempool_create(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) +{ + mempool_t *pool; + + pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); + if (!pool) + return NULL; + + if (__mempool_init(pool, min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id)) { + kfree(pool); + return NULL; + } + + return pool; +} + /** * mempool_create - create a memory pool * @min_nr: the minimum number of elements guaranteed to be @@ -252,8 +280,8 @@ EXPORT_SYMBOL(mempool_init); mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data) { - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); + return __mempool_create(min_nr, alloc_fn, free_fn, pool_data, + GFP_KERNEL, NUMA_NO_NODE); } EXPORT_SYMBOL(mempool_create); @@ -261,19 +289,8 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) { - mempool_t *pool; - - pool = kzalloc_node(sizeof(*pool), gfp_mask, node_id); - if (!pool) - return NULL; - - if (mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, - gfp_mask, node_id)) { - kfree(pool); - return NULL; - } - - return pool; + return __mempool_create(min_nr, alloc_fn, free_fn, pool_data, + gfp_mask, node_id); } EXPORT_SYMBOL(mempool_create_node); From patchwork Wed Oct 5 18:03:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12999545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 552D8C43219 for ; Wed, 5 Oct 2022 18:03:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8A896B0074; Wed, 5 Oct 2022 14:03:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E151C6B0075; Wed, 5 Oct 2022 14:03:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB36F8E0001; Wed, 5 Oct 2022 14:03:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B5BB46B0074 for ; Wed, 5 Oct 2022 14:03:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7F28BAB5B4 for ; Wed, 5 Oct 2022 18:03:57 +0000 (UTC) X-FDA: 79987669314.13.F2DC20D Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf11.hostedemail.com (Postfix) with ESMTP id E090640021 for ; Wed, 5 Oct 2022 18:03:56 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id t10-20020a17090a4e4a00b0020af4bcae10so1655665pjl.3 for ; Wed, 05 Oct 2022 11:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XVCvABAzZpcB1Yd0f9owRm5bhoDgwilteM8uMFdB0/0=; b=mpUGQ5wAmEoiNp0TsmAkDQGf+3eb7aR2cKLQ2LZhfehStaXh3JnCppkRW8LZPCe+by e5zvBFzm7soKVLdanhXKl6KQkT5/DjXZ3Cj87nV2zgPNOlq4Ce/iBJ2Jx+nVtl46XtgO kvOOEfGTXRbBxt3RFHLzzbUkPtY0x8lrK9bfEW8oClsisY3LJz8TAGkG8ECg8oGuqLug MwOm7PHTqecK5bdLmTNbGvcUHt8VRTZ7ZO7Di2OEgigK/uj59HJJjBNLD730OLo24uVk cL6i3sX2VgLkHJ/H/o6n+XOrMls9fxAGxsz8xuQw+BOiyoEjMpwpRgo4fXMfF2On929R oJHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XVCvABAzZpcB1Yd0f9owRm5bhoDgwilteM8uMFdB0/0=; b=cLtuGvnWX0XbRtI9PcUsPLYa8UJDgIWYTxhuZGLym0E95gZusKhB0iXYugb2qyMWzr L0bsLEJF97cOyALx8OQzr6+ihRRsEfOc2OgdqqB5aIhlgMK8lbB6DsnHz818gbihS/nJ Ff+uMFslHuGhEEHvkZhX9CbIqJ11sFFiIgCC3cDKZdiV2vf84Hx4NBRgbL5Cjn2SGwJW VGboOCIZl2De5yM5F37oo8/uHWtbxIz9eDQEGN5z6wCgoGRo0x2NCms42Z1v7nQYOmTB EwkzDiaSHqLsTPCqwLzWCgu1Eq9rfPlHGgQS4d9in93L1ag43VLTjMwd2evNFcN27HG9 szkA== X-Gm-Message-State: ACrzQf2gOuPkK7HSy5wLdHdyZCxcjSWsUTvifnpBHR5zJRJOdGyfNLnl kIFds0wxHxnyWpPSZstsm0w= X-Google-Smtp-Source: AMsMyM76elMyGdYLqmYEzlb0EvZzQrrcq16s35J/Z6liGP+p+uL1N0ov5WsEO+IIPHGlxZIw64UlmQ== X-Received: by 2002:a17:902:e952:b0:17c:2eee:c0ce with SMTP id b18-20020a170902e95200b0017c2eeec0cemr676521pll.145.1664993035919; Wed, 05 Oct 2022 11:03:55 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id y17-20020a170903011100b001788494b764sm10674639plc.231.2022.10.05.11.03.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:03:55 -0700 (PDT) From: Yang Shi To: mgorman@techsingularity.net, agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] mm: mempool: introduce page bulk allocator Date: Wed, 5 Oct 2022 11:03:39 -0700 Message-Id: <20221005180341.1738796-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20221005180341.1738796-1-shy828301@gmail.com> References: <20221005180341.1738796-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664993037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XVCvABAzZpcB1Yd0f9owRm5bhoDgwilteM8uMFdB0/0=; b=ojic9gz8QvaifwC0mjbbpFwXGcvKNaDwum9vaq4iFwejEVW8KQJaMjdKR01F5Q96sWob/g bi8d69IILLA5Mb6U0HybfddeBpqutYxKNTTiV8mFNR98WwLUi1bqVYnYx1EOPlIJUEbU5j JVJDhxcSIBYfql1vd/dI7yZJNuNTDUo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mpUGQ5wA; spf=pass (imf11.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664993037; a=rsa-sha256; cv=none; b=J1Wl3XvA3QYQ2VxYb+FSJZ5Yi5CcrIhr7rsH5QMhcDpG+z9/klNu85KWNO7vqjou3Bdm6o r97uHme5ViTbnjNDdX3dtlkizcPcPRok1HS/OkIPed1f/f2azzUSYY1vVx9aIFX/p8OSw7 OgXdWGqhalCOPIf1KPHwVNR8mgyd1sg= Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mpUGQ5wA; spf=pass (imf11.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E090640021 X-Rspam-User: X-Stat-Signature: dxapoczixzkqqjm18hdt5tw6s38i5pnf X-HE-Tag: 1664993036-283426 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since v5.13 the page bulk allocator was introduced to allocate order-0 pages in bulk. There are a few mempool allocator callers which does order-0 page allocation in a loop, for example, dm-crypt, f2fs compress, etc. A mempool page bulk allocator seems useful. So introduce the mempool page bulk allocator. It introduces the below APIs: - mempool_init_pages_bulk() - mempool_create_pages_bulk() They initialize the mempool for page bulk allocator. The pool is filled by alloc_page() in a loop. - mempool_alloc_pages_bulk_list() - mempool_alloc_pages_bulk_array() They do bulk allocation from mempool. They do the below conceptually: 1. Call bulk page allocator 2. If the allocation is fulfilled then return otherwise try to allocate the remaining pages from the mempool 3. If it is fulfilled then return otherwise retry from #1 with sleepable gfp 4. If it is still failed, sleep for a while to wait for the mempool is refilled, then retry from #1 The populated pages will stay on the list or array until the callers consume them or free them. Since mempool allocator is guaranteed to success in the sleepable context, so the two APIs return true for success or false for fail. It is the caller's responsibility to handle failure case (partial allocation), just like the page bulk allocator. The mempool typically is an object agnostic allocator, but bulk allocation is only supported by pages, so the mempool bulk allocator is for page allocation only as well. Signed-off-by: Yang Shi --- include/linux/mempool.h | 19 ++++ mm/mempool.c | 188 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 197 insertions(+), 10 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 0c964ac107c2..8bad28bceaa8 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -13,6 +13,11 @@ struct kmem_cache; typedef void * (mempool_alloc_t)(gfp_t gfp_mask, void *pool_data); typedef void (mempool_free_t)(void *element, void *pool_data); +typedef unsigned int (mempool_alloc_pages_bulk_t)(gfp_t gfp_mask, + unsigned int nr, void *pool_data, + struct list_head *page_list, + struct page **page_array); + typedef struct mempool_s { spinlock_t lock; int min_nr; /* nr of elements at *elements */ @@ -22,6 +27,7 @@ typedef struct mempool_s { void *pool_data; mempool_alloc_t *alloc; mempool_free_t *free; + mempool_alloc_pages_bulk_t *alloc_pages_bulk; wait_queue_head_t wait; } mempool_t; @@ -36,18 +42,31 @@ int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, gfp_t gfp_mask, int node_id); int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +int mempool_init_pages_bulk(mempool_t *pool, int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, + mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +extern mempool_t *mempool_create_pages_bulk(int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, + mempool_free_t *free_fn, void *pool_data); extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; extern void mempool_free(void *element, mempool_t *pool); +extern bool mempool_alloc_pages_bulk_list(mempool_t *pool, gfp_t gfp_mask, + unsigned int nr, + struct list_head *page_list); +extern bool mempool_alloc_pages_bulk_array(mempool_t *pool, gfp_t gfp_mask, + unsigned int nr, + struct page **page_array); + /* * A mempool_alloc_t and mempool_free_t that get the memory from * a slab cache that is passed in through pool_data. diff --git a/mm/mempool.c b/mm/mempool.c index ba32151f3843..7711ca2e6d66 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -177,6 +177,7 @@ void mempool_destroy(mempool_t *pool) EXPORT_SYMBOL(mempool_destroy); static inline int __mempool_init(mempool_t *pool, int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) @@ -186,8 +187,11 @@ static inline int __mempool_init(mempool_t *pool, int min_nr, pool->pool_data = pool_data; pool->alloc = alloc_fn; pool->free = free_fn; + pool->alloc_pages_bulk = alloc_pages_bulk_fn; init_waitqueue_head(&pool->wait); + WARN_ON_ONCE(alloc_pages_bulk_fn && alloc_fn); + pool->elements = kmalloc_array_node(min_nr, sizeof(void *), gfp_mask, node_id); if (!pool->elements) @@ -199,7 +203,10 @@ static inline int __mempool_init(mempool_t *pool, int min_nr, while (pool->curr_nr < pool->min_nr) { void *element; - element = pool->alloc(gfp_mask, pool->pool_data); + if (pool->alloc_pages_bulk) + element = alloc_page(gfp_mask); + else + element = pool->alloc(gfp_mask, pool->pool_data); if (unlikely(!element)) { mempool_exit(pool); return -ENOMEM; @@ -214,7 +221,7 @@ int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) { - return __mempool_init(pool, min_nr, alloc_fn, free_fn, pool_data, + return __mempool_init(pool, min_nr, NULL, alloc_fn, free_fn, pool_data, gfp_mask, node_id); } EXPORT_SYMBOL(mempool_init_node); @@ -236,15 +243,40 @@ EXPORT_SYMBOL(mempool_init_node); int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data) { - return __mempool_init(pool, min_nr, alloc_fn, free_fn, + return __mempool_init(pool, min_nr, NULL, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } EXPORT_SYMBOL(mempool_init); -static mempool_t *__mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +/** + * mempool_init_pages_bulk - initialize a pages pool for bulk allocator + * @pool: pointer to the memory pool that should be initialized + * @min_nr: the minimum number of elements guaranteed to be + * allocated for this pool. + * @alloc_pages_bulk_fn: user-defined pages bulk allocation function. + * @free_fn: user-defined element-freeing function. + * @pool_data: optional private data available to the user-defined functions. + * + * Like mempool_create(), but initializes the pool in (i.e. embedded in another + * structure). + * + * Return: %0 on success, negative error code otherwise. + */ +int mempool_init_pages_bulk(mempool_t *pool, int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return __mempool_init(pool, min_nr, alloc_pages_bulk_fn, NULL, + free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); +} +EXPORT_SYMBOL(mempool_init_pages_bulk); + +static mempool_t *__mempool_create(int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, + mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -252,8 +284,8 @@ static mempool_t *__mempool_create(int min_nr, mempool_alloc_t *alloc_fn, if (!pool) return NULL; - if (__mempool_init(pool, min_nr, alloc_fn, free_fn, pool_data, - gfp_mask, node_id)) { + if (__mempool_init(pool, min_nr, alloc_pages_bulk_fn, alloc_fn, + free_fn, pool_data, gfp_mask, node_id)) { kfree(pool); return NULL; } @@ -280,7 +312,7 @@ static mempool_t *__mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data) { - return __mempool_create(min_nr, alloc_fn, free_fn, pool_data, + return __mempool_create(min_nr, NULL, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } EXPORT_SYMBOL(mempool_create); @@ -289,11 +321,21 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) { - return __mempool_create(min_nr, alloc_fn, free_fn, pool_data, + return __mempool_create(min_nr, NULL, alloc_fn, free_fn, pool_data, gfp_mask, node_id); } EXPORT_SYMBOL(mempool_create_node); +mempool_t* mempool_create_pages_bulk(int min_nr, + mempool_alloc_pages_bulk_t *alloc_pages_bulk_fn, + mempool_free_t *free_fn, void *pool_data) +{ + return __mempool_create(min_nr, alloc_pages_bulk_fn, NULL, + free_fn, pool_data, GFP_KERNEL, + NUMA_NO_NODE); +} +EXPORT_SYMBOL(mempool_create_pages_bulk); + /** * mempool_resize - resize an existing memory pool * @pool: pointer to the memory pool which was allocated via @@ -457,6 +499,132 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) } EXPORT_SYMBOL(mempool_alloc); +/** + * mempool_alloc_pages_bulk - allocate a bulk of pagesfrom a specific + * memory pool + * @pool: pointer to the memory pool which was allocated via + * mempool_create(). + * @gfp_mask: the usual allocation bitmask. + * @nr: the number of requested pages. + * @page_list: the list the pages will be added to. + * @page_array: the array the pages will be added to. + * + * this function only sleeps if the alloc_pages_bulk_fn() function sleeps + * or the allocation can not be satisfied even though the mempool is depleted. + * Note that due to preallocation, this function *never* fails when called + * from process contexts. (it might fail if called from an IRQ context.) + * Note: using __GFP_ZERO is not supported. And the caller should not pass + * in both valid page_list and page_array. + * + * Return: true when nr pages are allocated or false if not. It is the + * caller's responsibility to free the partial allocated pages. + */ +static bool mempool_alloc_pages_bulk(mempool_t *pool, gfp_t gfp_mask, + unsigned int nr, + struct list_head *page_list, + struct page **page_array) +{ + unsigned long flags; + wait_queue_entry_t wait; + gfp_t gfp_temp; + int i; + unsigned int ret, nr_remaining; + struct page *page; + + VM_WARN_ON_ONCE(gfp_mask & __GFP_ZERO); + might_alloc(gfp_mask); + + gfp_mask |= __GFP_NOMEMALLOC; /* don't allocate emergency reserves */ + gfp_mask |= __GFP_NORETRY; /* don't loop in __alloc_pages */ + gfp_mask |= __GFP_NOWARN; /* failures are OK */ + + gfp_temp = gfp_mask & ~(__GFP_DIRECT_RECLAIM|__GFP_IO); + +repeat_alloc: + i = 0; + ret = pool->alloc_pages_bulk(gfp_temp, nr, pool->pool_data, page_list, + page_array); + + if (ret == nr) + return true; + + nr_remaining = nr - ret; + + spin_lock_irqsave(&pool->lock, flags); + /* Allocate page from the pool and add to the list or array */ + while (pool->curr_nr && (nr_remaining > 0)) { + page = remove_element(pool); + spin_unlock_irqrestore(&pool->lock, flags); + smp_wmb(); + + kmemleak_update_trace((void *)page); + + if (page_list) + list_add(&page->lru, page_list); + else + page_array[ret + i] = page; + + i++; + nr_remaining--; + + spin_lock_irqsave(&pool->lock, flags); + } + + spin_unlock_irqrestore(&pool->lock, flags); + + if (!nr_remaining) + return true; + + /* + * The bulk allocator counts in the populated pages for array, + * but don't do it for list. + */ + if (page_list) + nr = nr_remaining; + + /* + * We use gfp mask w/o direct reclaim or IO for the first round. If + * alloc failed with that and @pool was empty, retry immediately. + */ + if (gfp_temp != gfp_mask) { + gfp_temp = gfp_mask; + goto repeat_alloc; + } + + /* We must not sleep if !__GFP_DIRECT_RECLAIM */ + if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) + return false; + + /* Let's wait for someone else to return an element to @pool */ + init_wait(&wait); + prepare_to_wait(&pool->wait, &wait, TASK_UNINTERRUPTIBLE); + + /* + * FIXME: this should be io_schedule(). The timeout is there as a + * workaround for some DM problems in 2.6.18. + */ + io_schedule_timeout(5*HZ); + + finish_wait(&pool->wait, &wait); + goto repeat_alloc; +} + +bool mempool_alloc_pages_bulk_list(mempool_t *pool, gfp_t gfp_mask, + unsigned int nr, + struct list_head *page_list) +{ + return mempool_alloc_pages_bulk(pool, gfp_mask, nr, page_list, NULL); +} +EXPORT_SYMBOL(mempool_alloc_pages_bulk_list); + +bool mempool_alloc_pages_bulk_array(mempool_t *pool, gfp_t gfp_mask, + unsigned int nr, + struct page **page_array) +{ + return mempool_alloc_pages_bulk(pool, gfp_mask, nr, NULL, page_array); +} +EXPORT_SYMBOL(mempool_alloc_pages_bulk_array); + /** * mempool_free - return an element to the pool. * @element: pool element pointer. From patchwork Wed Oct 5 18:03:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12999546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1E6CC433F5 for ; Wed, 5 Oct 2022 18:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 600CD6B0075; Wed, 5 Oct 2022 14:03:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 587656B0078; Wed, 5 Oct 2022 14:03:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B2B66B007B; Wed, 5 Oct 2022 14:03:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 28E396B0075 for ; Wed, 5 Oct 2022 14:03:59 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E09F080FF9 for ; Wed, 5 Oct 2022 18:03:58 +0000 (UTC) X-FDA: 79987669356.17.423237A Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf29.hostedemail.com (Postfix) with ESMTP id 8934B120022 for ; Wed, 5 Oct 2022 18:03:58 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id i7-20020a17090a65c700b0020ad9666a86so2102883pjs.0 for ; Wed, 05 Oct 2022 11:03:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j+XlP7a4AbJnWW0mvkyBXtZwa/N1Rx2itITpw13S/3I=; b=irYlT/WC2ltGpTTQ+/ZIRFgbI4FjjgprU0/J4MQTS+f+PGfYZscMdNRDCWR1t0WIVx c3AYZNDHPpy416tfUu1G8zyv+V5cHRVfZUkxDgecHomcrjiQOfdqM8UBMQ45cbWzCGWg zazDiDbvm/NDMfZ5l46WEPP5xf3ZJEE02khGUMoA0I+Farvi51O+6Fjs67U3wM2t0sYP UvaWxSVyLhvlrXHXhkJaaiWeJP08MiOSdk9+b0BT/JDvEaH9PkTyGBOtI2scLGj6ZLUw C7BNEi67o0WAh7eP3XlaqlT/lfu1VH6+MEYj+lmbNrPqbklHTYrA4qcuJHuwBRb7NHlR s4RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j+XlP7a4AbJnWW0mvkyBXtZwa/N1Rx2itITpw13S/3I=; b=5R6hiehh06SDNl5eA4rw0jbOP9ibzOCYQXLZ0PHt3gyLr/3ee/v6DwhMzUvN6VyTM7 Aupvr+4rrdjyNo6T4xnMHZgcFtTli7575+qd+GlXqUNRV5ktOb5TwgYQxVbQyoJmJdzZ 94YA7NTTbCUX0EvnRDfwc0UBODGvpdtz7A4dS32ezC5lh5Z1qxnipqx9SnfReONbbnwy cRjA9mfAClLGoQ+u5Alcmh5Wdxs34y9ZCgn9Jx9o4+/f3w4bpeV9n8YZ5cuH8Z8Ap9wK layXRdIHtR6cjVcNsaazm158d9D8tkzpjuDUtRf7bydLnhS2fcK8+Yj2KiR/j0c2f33h K8zQ== X-Gm-Message-State: ACrzQf1/v2PYN+yKfXMHMgbqYpkCUTXdqVYQjX+d06Jzl4pwtI41hIgA bIuB2ggTfaD313Ixsw3iwnI= X-Google-Smtp-Source: AMsMyM7R2vC6Qn5IEDHgIQVREy/czJWxFA0KW9eqPauuVPkuyU3SUmM3xBqqup3gi735DjAFHPyrow== X-Received: by 2002:a17:90b:3e83:b0:20a:f3dd:3e2e with SMTP id rj3-20020a17090b3e8300b0020af3dd3e2emr3106527pjb.191.1664993037711; Wed, 05 Oct 2022 11:03:57 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id y17-20020a170903011100b001788494b764sm10674639plc.231.2022.10.05.11.03.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:03:56 -0700 (PDT) From: Yang Shi To: mgorman@techsingularity.net, agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] md: dm-crypt: move crypt_free_buffer_pages ahead Date: Wed, 5 Oct 2022 11:03:40 -0700 Message-Id: <20221005180341.1738796-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20221005180341.1738796-1-shy828301@gmail.com> References: <20221005180341.1738796-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664993038; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j+XlP7a4AbJnWW0mvkyBXtZwa/N1Rx2itITpw13S/3I=; b=MUTePWrJqBLgNSCUnraCu2LnSp1shaDsvnq5yATrIzItYZG8sKOyO8C9Appgf3sgN3z7ST TZKWq6lyHtRlanDerqUweB394VQxXuRdqI9duR6HJ2N8zSm5/HtaV58mah/bkZVfeWBa5R myY0vjUcRbm3GlqPlbm7otugPXWFcc4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="irYlT/WC"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664993038; a=rsa-sha256; cv=none; b=nZKxKL+u/JfY33F4otqagBF07bmMdQG++bo5wiGm2bNDBiLpTdv1hX91JE1ec6l2fLXD/E InCxZXafflMe86q4s4gc6KAVe4DXaz4cHgNUR35xIfeKUiJSPoLT1o8OpNeAqgMM07C5qv VMaRStnVoqgo9QsR7CJb44iu5KpT69k= X-Rspamd-Queue-Id: 8934B120022 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="irYlT/WC"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: q8fe6windze4gy75fq5n5589cu5yb4kx X-HE-Tag: 1664993038-764174 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000024, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With moving crypt_free_buffer_pages() before crypt_alloc_buffer(), we don't need an extra declaration anymore. Signed-off-by: Yang Shi --- drivers/md/dm-crypt.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 159c6806c19b..85ac1f9b37ae 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1639,7 +1639,16 @@ static blk_status_t crypt_convert(struct crypt_config *cc, return 0; } -static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone); +static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone) +{ + struct bio_vec *bv; + struct bvec_iter_all iter_all; + + bio_for_each_segment_all(bv, clone, iter_all) { + BUG_ON(!bv->bv_page); + mempool_free(bv->bv_page, &cc->page_pool); + } +} /* * Generate a new unfragmented bio with the given size @@ -1707,17 +1716,6 @@ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned size) return clone; } -static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone) -{ - struct bio_vec *bv; - struct bvec_iter_all iter_all; - - bio_for_each_segment_all(bv, clone, iter_all) { - BUG_ON(!bv->bv_page); - mempool_free(bv->bv_page, &cc->page_pool); - } -} - static void crypt_io_init(struct dm_crypt_io *io, struct crypt_config *cc, struct bio *bio, sector_t sector) { From patchwork Wed Oct 5 18:03:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12999547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49E4DC433FE for ; Wed, 5 Oct 2022 18:04:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7D516B0078; Wed, 5 Oct 2022 14:04:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2CB08E0001; Wed, 5 Oct 2022 14:04:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A63C6B007D; Wed, 5 Oct 2022 14:04:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 896AB6B0078 for ; Wed, 5 Oct 2022 14:04:01 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 34F2AC1006 for ; Wed, 5 Oct 2022 18:04:01 +0000 (UTC) X-FDA: 79987669482.08.850EFAB Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf25.hostedemail.com (Postfix) with ESMTP id 78759A0012 for ; Wed, 5 Oct 2022 18:04:00 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id x32-20020a17090a38a300b00209dced49cfso2524873pjb.0 for ; Wed, 05 Oct 2022 11:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z8tNeWv+0nv7FXmMat9FJ50mQ7l5yun8pgPApcsh1yw=; b=iGiCtwUJuErnNlAIo4FT/4bcJD5Wks2spXOcIZtRA/WUiXPtpwF/DU5z9tyTsk5CB/ 39szPETZHXlAC1SqY0naRLLzzPFCToSvDBN0mO1i3PJR2Lru373sQYYQ7KEVita4fbjM nA0YLO/oHE/71/iwUn4rZQerJ29H/Vsm8cHA0UEn9iejDGqhX3xB6P/DeCSkzPKF+zn4 pEwnsbWbJU/i0oRUD/bAECH7khbJqtfJCz0eDaR6Aujbd+xpWbhz0Dw4DdVAin2UKOry 61e05LpZ/BuZaQ6F8fIfNKl1RhxSoQ7g6KwE1485jwEeXgwCsh95cFW0KlPwrBJ0xhbf QUEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z8tNeWv+0nv7FXmMat9FJ50mQ7l5yun8pgPApcsh1yw=; b=66cDrAQ3mYZBQO2KlS1LhMFfOPQnOQrfuEaUqA8qdFEtXBS8fn+7Lq3X4iKn/wAx+8 B7SXcFVDsQPmF6Rdl/SV6Vl6qQw36CuPQPwOUyerXM0W31JItNxDI4sD0M2hLJP7NKe6 EeQB9of9n6ppgK4zt2zo0UsQpwv8F7CGhURvVvD3DwJKBepSaMQYLAbcY3hpaK/sX9iN bLQ12rrQRKdzUyMey1UVi/8HuzWfYoXt7ywOYM109dSEybm5Bs0OWDzQfq6k5sprxLiU oKwh15bmL8ScAfX7TrBch9EiZ5Q1dXRJB3DYlxoobPJ1pFH6SXu/FW4EEjPgaD1MD5M+ Uhrg== X-Gm-Message-State: ACrzQf0RLxks18h5Kjrho3doHA4D/g+axTAfej3oZniswC5NVNGjSFmg Ruula668ju0y+lBS6AdvdfA= X-Google-Smtp-Source: AMsMyM6dffG6twubdAg50b0ZfzNL/4dO6jBfiQTtKARmAZ1J5Sjn4eFIdDTcXNZxz+o/OJWGLmOG5g== X-Received: by 2002:a17:902:d2d2:b0:177:4940:cc0f with SMTP id n18-20020a170902d2d200b001774940cc0fmr907820plc.4.1664993039312; Wed, 05 Oct 2022 11:03:59 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id y17-20020a170903011100b001788494b764sm10674639plc.231.2022.10.05.11.03.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:03:58 -0700 (PDT) From: Yang Shi To: mgorman@techsingularity.net, agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] md: dm-crypt: use mempool page bulk allocator Date: Wed, 5 Oct 2022 11:03:41 -0700 Message-Id: <20221005180341.1738796-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20221005180341.1738796-1-shy828301@gmail.com> References: <20221005180341.1738796-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664993040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8tNeWv+0nv7FXmMat9FJ50mQ7l5yun8pgPApcsh1yw=; b=pcB7ber6wja/HNQNKjuyqs9byvwJM30jlJAdr2gVKPdqEcTsN4xG/3tEs6zQB0r9A1/hfA 3MbeflsNfrs2/ByAIrB09ZpEjV2OasjhJX2/Pi3sJFm7H1esuGYGWFADAiC0f0X5ZxaSY9 TPSkBIxOVoRW0B1gmI8AfXeCLpWbHSw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iGiCtwUJ; spf=pass (imf25.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664993040; a=rsa-sha256; cv=none; b=0zIontlpBFFRTo3AwoOPb4ykIU1Gh7cAu1EjgGKmLyG9TP+RQG9ZQPgtC3aO9LdR+0u3K1 mA3+O2bBKLbH5/g95jMaX/zC8NW6/aDKGrMdsNn1PEvP6hSEhyLfCSnhKsxvrMdcYeN18Z 5yCQJWSKTOEP5BLcSiMeVsOSIi/u4YA= X-Stat-Signature: kg65ufqxjij9doirp44br1w3fikreaoz X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 78759A0012 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iGiCtwUJ; spf=pass (imf25.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-HE-Tag: 1664993040-162751 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When using dm-crypt for full disk encryption, dm-crypt would allocate an out bio and allocate the same amount of pages as in bio for encryption. It currently allocates one page at a time in a loop. This is not efficient. So using mempool page bulk allocator instead of allocating one page at a time. The mempool page bulk allocator would improve the IOPS with 1M I/O by approxiamately 6%. The test is done on a VM with 80 vCPU and 64GB memory with an encrypted ram device (the impact from storage hardware could be minimized so that we could benchmark the dm-crypt layer more accurately). Before the patch: Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=402MiB/s][r=0,w=402 IOPS][eta 00m:00s] crypt: (groupid=0, jobs=1): err= 0: pid=233950: Thu Sep 15 16:23:10 2022 write: IOPS=402, BW=403MiB/s (423MB/s)(23.6GiB/60002msec) slat (usec): min=2425, max=3819, avg=2480.84, stdev=34.00 clat (usec): min=7, max=165751, avg=156398.72, stdev=4691.03 lat (msec): min=2, max=168, avg=158.88, stdev= 4.69 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 167] bw ( KiB/s): min=405504, max=413696, per=99.71%, avg=411845.53, stdev=1155.04, samples=120 iops : min= 396, max= 404, avg=402.17, stdev= 1.15, samples=120 lat (usec) : 10=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.05%, 100=0.08% lat (msec) : 250=100.09% cpu : usr=3.74%, sys=95.66%, ctx=27, majf=0, minf=4 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,24138,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=403MiB/s (423MB/s), 403MiB/s-403MiB/s (423MB/s-423MB/s), io=23.6GiB (25.4GB), run=60002-60002msec After the patch: Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=430MiB/s][r=0,w=430 IOPS][eta 00m:00s] crypt: (groupid=0, jobs=1): err= 0: pid=288730: Thu Sep 15 16:25:39 2022 write: IOPS=430, BW=431MiB/s (452MB/s)(25.3GiB/60002msec) slat (usec): min=2253, max=3213, avg=2319.49, stdev=34.29 clat (usec): min=6, max=149337, avg=146257.68, stdev=4239.52 lat (msec): min=2, max=151, avg=148.58, stdev= 4.24 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min=438272, max=442368, per=99.73%, avg=440463.57, stdev=1305.60, samples=120 iops : min= 428, max= 432, avg=430.12, stdev= 1.28, samples=120 lat (usec) : 10=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.05%, 100=0.09% lat (msec) : 250=100.07% cpu : usr=3.78%, sys=95.37%, ctx=12778, majf=0, minf=4 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,25814,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=431MiB/s (452MB/s), 431MiB/s-431MiB/s (452MB/s-452MB/s), io=25.3GiB (27.1GB), run=60002-60002msec The function tracing also shows the time consumed by page allocations is reduced significantly. The test allocated 1M (256 pages) bio in the same environment. Before the patch: It took approximately 600us by excluding the bio_add_page() calls. 2720.630754 | 56) xfs_io-38859 | 2.571 us | mempool_alloc(); 2720.630757 | 56) xfs_io-38859 | 0.937 us | bio_add_page(); 2720.630758 | 56) xfs_io-38859 | 1.772 us | mempool_alloc(); 2720.630760 | 56) xfs_io-38859 | 0.852 us | bio_add_page(); …. 2720.631559 | 56) xfs_io-38859 | 2.058 us | mempool_alloc(); 2720.631561 | 56) xfs_io-38859 | 0.717 us | bio_add_page(); 2720.631562 | 56) xfs_io-38859 | 2.014 us | mempool_alloc(); 2720.631564 | 56) xfs_io-38859 | 0.620 us | bio_add_page(); After the patch: It took approxiamately 30us. 11564.266385 | 22) xfs_io-136183 | + 30.551 us | __alloc_pages_bulk(); Page allocations overhead is around 6% (600us/9853us) in dm-crypt layer shown by function trace. The data also matches the IOPS data shown by fio. And the benchmark with 4K size I/O doesn't show measurable regression. Signed-off-by: Yang Shi --- drivers/md/dm-crypt.c | 70 +++++++++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 29 deletions(-) diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 85ac1f9b37ae..c86bd4af4d75 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1673,34 +1673,37 @@ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned size) struct bio *clone; unsigned int nr_iovecs = (size + PAGE_SIZE - 1) >> PAGE_SHIFT; gfp_t gfp_mask = GFP_NOWAIT | __GFP_HIGHMEM; - unsigned i, len, remaining_size; + unsigned len; struct page *page; - -retry: - if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) - mutex_lock(&cc->bio_alloc_lock); + LIST_HEAD(page_list); clone = bio_alloc_bioset(cc->dev->bdev, nr_iovecs, io->base_bio->bi_opf, GFP_NOIO, &cc->bs); clone->bi_private = io; clone->bi_end_io = crypt_endio; - remaining_size = size; +retry: + if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) + mutex_lock(&cc->bio_alloc_lock); - for (i = 0; i < nr_iovecs; i++) { - page = mempool_alloc(&cc->page_pool, gfp_mask); - if (!page) { - crypt_free_buffer_pages(cc, clone); - bio_put(clone); - gfp_mask |= __GFP_DIRECT_RECLAIM; - goto retry; + if (mempool_alloc_pages_bulk_list(&cc->page_pool, gfp_mask, nr_iovecs, + &page_list)) { + while (!list_empty(&page_list)) { + page = lru_to_page(&page_list); + list_del_init(&page->lru); + len = (size > PAGE_SIZE) ? PAGE_SIZE : size; + bio_add_page(clone, page, len, 0); + size -= len; + } + } else { + while (!list_empty(&page_list)) { + page = lru_to_page(&page_list); + list_del_init(&page->lru); + mempool_free(page, &cc->page_pool); } - len = (remaining_size > PAGE_SIZE) ? PAGE_SIZE : remaining_size; - - bio_add_page(clone, page, len, 0); - - remaining_size -= len; + gfp_mask |= __GFP_DIRECT_RECLAIM; + goto retry; } /* Allocate space for integrity tags */ @@ -2654,10 +2657,13 @@ static void crypt_calculate_pages_per_client(void) dm_crypt_pages_per_client = pages; } -static void *crypt_page_alloc(gfp_t gfp_mask, void *pool_data) +static unsigned int crypt_alloc_pages_bulk(gfp_t gfp_mask, unsigned int nr, + void *pool_data, + struct list_head *page_list, + struct page **page_array) { struct crypt_config *cc = pool_data; - struct page *page; + unsigned int ret; /* * Note, percpu_counter_read_positive() may over (and under) estimate @@ -2666,13 +2672,13 @@ static void *crypt_page_alloc(gfp_t gfp_mask, void *pool_data) */ if (unlikely(percpu_counter_read_positive(&cc->n_allocated_pages) >= dm_crypt_pages_per_client) && likely(gfp_mask & __GFP_NORETRY)) - return NULL; + return 0; + + ret = alloc_pages_bulk_list(gfp_mask, nr, page_list); - page = alloc_page(gfp_mask); - if (likely(page != NULL)) - percpu_counter_add(&cc->n_allocated_pages, 1); + percpu_counter_add(&cc->n_allocated_pages, ret); - return page; + return ret; } static void crypt_page_free(void *page, void *pool_data) @@ -2704,13 +2710,17 @@ static void crypt_dtr(struct dm_target *ti) bioset_exit(&cc->bs); + /* + * With mempool bulk allocator the pages in the pool are not + * counted in n_allocated_pages. + */ + WARN_ON(percpu_counter_sum(&cc->n_allocated_pages) != 0); + percpu_counter_destroy(&cc->n_allocated_pages); + mempool_exit(&cc->page_pool); mempool_exit(&cc->req_pool); mempool_exit(&cc->tag_pool); - WARN_ON(percpu_counter_sum(&cc->n_allocated_pages) != 0); - percpu_counter_destroy(&cc->n_allocated_pages); - if (cc->iv_gen_ops && cc->iv_gen_ops->dtr) cc->iv_gen_ops->dtr(cc); @@ -3250,7 +3260,9 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv) ALIGN(sizeof(struct dm_crypt_io) + cc->dmreq_start + additional_req_size, ARCH_KMALLOC_MINALIGN); - ret = mempool_init(&cc->page_pool, BIO_MAX_VECS, crypt_page_alloc, crypt_page_free, cc); + ret = mempool_init_pages_bulk(&cc->page_pool, BIO_MAX_VECS, + crypt_alloc_pages_bulk, crypt_page_free, + cc); if (ret) { ti->error = "Cannot allocate page mempool"; goto bad;