From patchwork Mon Dec 18 11:50:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 184EAC35274 for ; Mon, 18 Dec 2023 11:50:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B544E8D0011; Mon, 18 Dec 2023 06:50:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADDAB8D0001; Mon, 18 Dec 2023 06:50:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E1148D0011; Mon, 18 Dec 2023 06:50:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6B2168D0001 for ; Mon, 18 Dec 2023 06:50:56 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36490120A55 for ; Mon, 18 Dec 2023 11:50:56 +0000 (UTC) X-FDA: 81579772512.06.EEFB5B2 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) by imf11.hostedemail.com (Postfix) with ESMTP id 4554040026 for ; Mon, 18 Dec 2023 11:50:54 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf11.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JE7KLoKyK1KRdi42drS6a//u5Xi4f54POK2HQQg/pPo=; b=KT7XZc6pNh30i+HCeoxiy717UBXMelD4myNF4NMAkShUCg2ke7XZZtIscAiZGn3Z/yZ1kg qgEJ4kHD0T/UlQ7rNUbSyaXJhoX4EpZyGx8XBZDmLTKGtl7d4i+43jEhhcdwHIgNJCPBUp HPHEnft/UduATPk0GAHs0aKiTPgTMwI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf11.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900254; a=rsa-sha256; cv=none; b=Y0F7dXn7T1FrT1BaSn3RvmIKBO8eFoj2zQrynv5AZ77U3oGh3JfQkJu5VglQ8IeRUMpkIF 1K9tV7m8TsSQosQ6hJQsUGKqd0GgcRCTKStvoHPVd4yNuPyTeXC6COV5zvNoMwFiCWVWSC 0HVH1f8rYWPf0NRZkbnZc79UkOgQ/VY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:36 +0000 Subject: [PATCH v3 6/6] mm/zswap: directly use percpu mutex and buffer in load/store MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-6-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=6471; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=culrrHCm/Hf4dwHwtvZ06lQ5NsEncVU/QGy2Bp4v+0o=; b=b7n0OpxYpR3XA5GiwrtGYchVJRrAlhz5jqYK7l+kMC6lZWSSrzA4mZoc1L8QALpl7yA1Z8BpB gUTc5av/dY8AUdT5lUxvytYE9E+8PlB9bUSPPKrHM97afZKBM0ohjLG X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 4554040026 X-Rspamd-Server: rspam05 X-Stat-Signature: gdk7yiicdkeo6iq1ircyscx6mnzgdmg9 X-Rspam: Yes X-HE-Tag: 1702900254-622992 X-HE-Meta: U2FsdGVkX1/zT6qVbFqtfSR3+KhcOeTLnmMiIxq0rBbEZKax6KFdzsSvpdhsLCC1Y85RCNTsineREWbAAeYFUaLSacgU9JCQU187GiLWcVK9E4IXzWtUBcU0LDHdJtZRIpz23hwv7aTBrMJxrsXmZedw7+Q5Of0SEOlTXx6lVzDb+H+k6dxqzh8A0gYHYXn0dC+rKY5bJjgvpVQ6hr52niqT8OkEcnxPMBLAOARhTSB6D4c4ZhUO4Ot46zWdoPcZv9jYVqBpkwDnRbsXInpQ5SuKONgV/6eSUNnZfnsh2n8QEsfyiTnDfz8UNOLcxXwG9Ziwar1k/OurO/XfxOcsX4Lem8ZfL2GUU5zOEYLiXDwUAKEkDCwevK6SIh56UIRKT8U2rAp0ldp09Ry0Wl3hp/D1368z7EE0HCQ1Bm/TdRafcHUm9FNxiHghBOn33MwdkesGT5oWHyH9kA/E+bGkcMGK8TY9swxbDcHj8do2nTNrqIcVjAy2VvZSNQAogB73jqxmpgZakeath1UljBNp4rn/6ISf+dPe9+rsu1lbY801iTAPUXFuxdUibgPw0LcfKQYIdeJ2sOLfBvtx+Om+gMoXdRUAMQGelCqqWSgFaKX7xFSiXjo2u0IxBLBWPUI9CFOnc4XiJuGOPBKh1sYqNfOamNFbTcj6lRyGsRyJYCqL2p9KPLI26V6+/FMdjwvQtIC/DjdmFJd/NTqfAQAP3hxk/Njk0cWosulcZvCFUwl9AYf7GhBAKOCyl1kVk4NcpiFrp/u6GQfPoXGG+VuOxeT967OI9ULstGSC/+t5gWmh4H5r074Cpya++mX4fZ9iFrlB1MDsSy10EzmW8tluEtg452zWL/IqAVIZL6tXQPF1fydvdQNC6CdWtk+9F5Pf1U54rAdgZSrBHIbBkva3nalhvcdDe07ZnPXba155uNRpAWOmRuFifMhoJ6pY3ZZdUdDZUx2X6cnUIVHdWjt d/yqpMpd ikC0ClLsuzoyzJfL7JTfhS+ZOt/eFBIa8jTBKkl0aGDeAPH5vWUGwUjqS+qWQgJpmkOTw0WMii530qJJtsVNQJEWp/zAsi8XbncgnOd/meKY2sdAMz4NvW1MPw+JkiT6g+bz7u6YPGzwwaXA2o8c7Iz2ANIKPr32nPC+yDwnVSEOzCd/UHsVmMen8ryZ6rGvKtRy+X5XyQ1mEonPDSCqf+IRn6jFRzzRkl5Tsc7zebULd8ll++H8fPidytJ24gQzwVjJ+xV36dqYEiYkKgm+SHFUB28XP4cNn6u7j7+muQ+fNZCGO+WdfAWAUarINHAtFOgD5dQ8KKT2bporcXcEzNcQ2BYEeQmjk07J/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since the introduce of reusing the dstmem in the load path, it seems confusing that we are now using acomp_ctx->dstmem and acomp_ctx->mutex now for purposes other than what the naming suggests. Yosry suggested removing these two fields from acomp_ctx, and directly using zswap_dstmem and zswap_mutex in both the load and store paths, rename them, and add proper comments above their definitions that they are for generic percpu buffering on the load and store paths. So this patch remove dstmem and mutex from acomp_ctx, and rename the zswap_dstmem to zswap_buffer, using the percpu mutex and buffer on the load and store paths. Suggested-by: Yosry Ahmed Signed-off-by: Chengming Zhou Signed-off-by: Chengming Zhou Acked-by: Chris Li (Google) --- mm/zswap.c | 69 +++++++++++++++++++++++++++++++++----------------------------- 1 file changed, 37 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 2c349fd88904..71bdcd552e5b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,6 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; }; /* @@ -694,7 +692,7 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); +static DEFINE_PER_CPU(u8 *, zswap_buffer); /* * If users dynamically change the zpool type and compressor at runtime, i.e. * zswap is running, zswap can have more than one zpool on one cpu, but they @@ -702,39 +700,39 @@ static DEFINE_PER_CPU(u8 *, zswap_dstmem); */ static DEFINE_PER_CPU(struct mutex *, zswap_mutex); -static int zswap_dstmem_prepare(unsigned int cpu) +static int zswap_buffer_prepare(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; - dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) + buf = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); + if (!buf) return -ENOMEM; mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); if (!mutex) { - kfree(dst); + kfree(buf); return -ENOMEM; } mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; + per_cpu(zswap_buffer, cpu) = buf; per_cpu(zswap_mutex, cpu) = mutex; return 0; } -static int zswap_dstmem_dead(unsigned int cpu) +static int zswap_buffer_dead(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; mutex = per_cpu(zswap_mutex, cpu); kfree(mutex); per_cpu(zswap_mutex, cpu) = NULL; - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; + buf = per_cpu(zswap_buffer, cpu); + kfree(buf); + per_cpu(zswap_buffer, cpu) = NULL; return 0; } @@ -772,9 +770,6 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; } @@ -1397,15 +1392,21 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) struct zpool *zpool = zswap_find_zpool(entry); struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src; + u8 *src, *buf; + int cpu; + struct mutex *mutex; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); + + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + buf = per_cpu(zswap_buffer, cpu); + memcpy(buf, src, entry->length); + src = buf; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1416,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1551,6 +1552,8 @@ bool zswap_store(struct folio *folio) u8 *src, *dst; gfp_t gfp; int ret; + int cpu; + struct mutex *mutex; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1636,11 +1639,13 @@ bool zswap_store(struct folio *folio) } /* compress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); - mutex_lock(acomp_ctx->mutex); + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); + dst = per_cpu(zswap_buffer, cpu); - dst = acomp_ctx->dstmem; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -1683,7 +1688,7 @@ bool zswap_store(struct folio *folio) buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); memcpy(buf, dst, dlen); zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); /* populate entry */ entry->swpentry = swp_entry(type, offset); @@ -1726,7 +1731,7 @@ bool zswap_store(struct folio *folio) return true; put_dstmem: - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1902,10 +1907,10 @@ static int zswap_setup(void) } ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); + zswap_buffer_prepare, zswap_buffer_dead); if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; + pr_err("buffer alloc failed\n"); + goto buffer_fail; } ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, @@ -1940,7 +1945,7 @@ static int zswap_setup(void) zswap_pool_destroy(pool); hp_fail: cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: +buffer_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */