From patchwork Thu Dec 28 09:45:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13505768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4DBBC3DA6E for ; Thu, 28 Dec 2023 09:46:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C3F96B00BA; Thu, 28 Dec 2023 04:46:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64D058D0010; Thu, 28 Dec 2023 04:46:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 514FF8D000F; Thu, 28 Dec 2023 04:46:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 42FF36B00BA for ; Thu, 28 Dec 2023 04:46:38 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1D9BC40896 for ; Thu, 28 Dec 2023 09:46:38 +0000 (UTC) X-FDA: 81615747276.17.FB333CA Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) by imf19.hostedemail.com (Postfix) with ESMTP id 32F731A0011 for ; Thu, 28 Dec 2023 09:46:35 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703756796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8nL6Q1WjhPPL8W6G4tIFkYM/l+TyZuse48lKBGP8lwc=; b=hpRXcwPtT609icMunVtzQZBE0hkoKiQah2+oZzV4xFnUufQl2oaRDP+5vf90tyP0A8ups8 gmx+bj6yt/Poia1sygZ/GjI0KCADSvU+tzjod5k+n1nWUoRqu60BRSFnrhDmYArU5G3dUO +Gc+6LNvpHI+yXBfG47rYRYz0Czoc/8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703756796; a=rsa-sha256; cv=none; b=BDHdqKEDscqejFrN92ATc8W3HL4gOi04g03iP+78A8C+qLDmBOYcGkPRYkx09RIP/W4H3i Sutl9wYFwZV4sRpJM/QY3E9QThkU17RtMIsIXNVnXn4qieV/92z44AkZRTZnNICrKLPRaE bKC1pcyverzqmAFv6o3KUCvI9p2eAWc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 28 Dec 2023 09:45:46 +0000 Subject: [PATCH v5 5/5] mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v5-5-9382162bbf05@bytedance.com> References: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v5-0-9382162bbf05@bytedance.com> To: Barry Song <21cnbao@gmail.com>, Yosry Ahmed , Nhat Pham , Andrew Morton , Dan Streetman , Vitaly Wool , Johannes Weiner , Chris Li , Seth Jennings Cc: Yosry Ahmed , Nhat Pham , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1703756775; l=7667; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=4WwCjcTM4cvOikkhRW0eIrRaRjoJsRX6DvdiNXR+aoM=; b=zjkyeZ8GJvLmdKBjLPG2tkPu01IHk9nTR5X4BWfcrNSIJNqRsEvvUkf2037UBEjkA7VN2xRdg U2WvnDymn6ZCVI0TllIFnqKc5NdiqSdYpssVP3etAh5tKo+MGJxgduq X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 32F731A0011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qb7e6e1dokiqyu863eekxwjsd4q1wksc X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1703756795-402674 X-HE-Meta: U2FsdGVkX1/JxKU1RoLlBGjMRu0yrKlV5QLnfnB0ut4EOeGlfAxDud4S2ImrUMCDXHmQ+dz+NyYuBbTY/fEpd/5B8C24GP58Tq237D5b5whzq+Uoy/B8OuIVVBnaarh/BWu0aQudka3ZlZyaj2XaNSbR7fe5T9rculYyGwKMI6Qe89gv6pX2gthaao2Jx6daT2kcEK5CCqWQ1hZZu/BLLMT+jnrIfSrc9fgyw08/P6BALu0SdE1cHLfewqwRCHIZ5JY3hA2PhMqTdMSRZ5EqjA6TfGzz00ta7Kro2VGVAse367euW9DOvQvlr+jPP8MtGvNQqRK6Qkg3MOZHGcdd8UWZAMIvDoEkbvvK5oVWiGJDBUSRpZCl5Aio9t7+unMcur+ETJCAPoeIL4ouo0PBYYSDPftr75MNqxmB44A/7GqE9XTXo5+XPqnRWcUovmNr4W/ZmammAuNuGk2pj+KHSgVHu25+cXTR+zLyfJ1FqXg3TO2mLIQrf3REpbp7L1hTZEIL1lUPYA9Arg2a0W8hIHNtdyBy5okAvpGFIs4tXT6toycnWSpcHOiuEQQnplPFCZsAlf5x1JrGv6AjmPTK3X9FpA8Q/KFWcWEXrcDA3i4pGQ6XOGUY/G4Z/wsp1XfU4oAfcAaD9NdO7N1GJmehPSCnpGFqS+qzDMGxn1sGgfYyhvN2oiG4Bq/h0i9lZ4hf4E1IQYHPOeviypYyMSL+k0W0Fj2LHWomZCjKnc5mP2aYdzjxM9m0j3Ha9kzyJGyOAS5ENxDGPJfzhAn7U2HZU25YLdMps7Q257rvXAEfAW4DTMvtbaGeOC9PQfKXmcZkJESXDSTYQ54x5si+8r69+4/NtfUerPcO0GK4Da54UtSdBvdNHPMOptntXiAYKyRP3hS1WykDDDyC4jQcGzKh5/zzmJ5lOx09NNAVTZA+fdR8z2Re0GVlVDixnfzcnzlgSlsuHyYiVvOroCcP/Wl 5wzjY9dc jdK6Mrx7pkr9RWBf2KMO2JZF2FRFe966CpM9eYGUa3mI43SkCywMV/Ee8p+tE+jxvPMpCJRrTdOxtSLnLmpD+I/U3ZCucq8HiucXqD1qH7kT4lQEs/Emj/HJP4O3DtXYP4+4XzqHV5WAOIaeXcDUmT/h7/FNcmsZVvwrTcIym+sL5TIr1pNM3X+bmlrbeG9RpyBa4bOKw7gWSHk/yXR/7rCxqpzlQ04EeMbd4wH8P/JHPZnIIGKo7nKpBlKcQkh218smXkY9rWR+yougudYbHY17oWpN8ldr6j0dos4GVNeskknf4vi2jVtc62ouqUdkpeHYZ3OBB0Yb2o/x5I9WZ7MVa9WQ8B62iFkrsfmb0tdW8ZU/yYX6fAFTBRfBKd+kSUpmWadEndNjrx03xu1+yK8RAjRB5+AyGjPmguJZZU7voEtnr8lzWdiOuu4B6/3WhyqSnrXlxSc5tpbM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: First of all, we need to rename acomp_ctx->dstmem field to buffer, since we are now using for purposes other than compression. Then we change per-cpu mutex and buffer to per-acomp_ctx, since them belong to the acomp_ctx and are necessary parts when used in the compress/decompress contexts. So we can remove the old per-cpu mutex and dstmem. Acked-by: Chris Li (Google) Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou --- include/linux/cpuhotplug.h | 1 - mm/zswap.c | 104 ++++++++++++++------------------------------- 2 files changed, 33 insertions(+), 72 deletions(-) diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index efc0c0b07efb..c3e06e21766a 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -124,7 +124,6 @@ enum cpuhp_state { CPUHP_ARM_BL_PREPARE, CPUHP_TRACE_RB_PREPARE, CPUHP_MM_ZS_PREPARE, - CPUHP_MM_ZSWP_MEM_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, CPUHP_ZCOMP_PREPARE, diff --git a/mm/zswap.c b/mm/zswap.c index e756b2af838b..7c18755c6e1c 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,8 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; + u8 *buffer; + struct mutex mutex; }; /* @@ -694,63 +694,26 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); -/* - * If users dynamically change the zpool type and compressor at runtime, i.e. - * zswap is running, zswap can have more than one zpool on one cpu, but they - * are sharing dtsmem. So we need this mutex to be per-cpu. - */ -static DEFINE_PER_CPU(struct mutex *, zswap_mutex); - -static int zswap_dstmem_prepare(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) - return -ENOMEM; - - mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); - if (!mutex) { - kfree(dst); - return -ENOMEM; - } - - mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; - per_cpu(zswap_mutex, cpu) = mutex; - return 0; -} - -static int zswap_dstmem_dead(unsigned int cpu) -{ - struct mutex *mutex; - u8 *dst; - - mutex = per_cpu(zswap_mutex, cpu); - kfree(mutex); - per_cpu(zswap_mutex, cpu) = NULL; - - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; - - return 0; -} - static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); struct crypto_acomp *acomp; struct acomp_req *req; + int ret; + + mutex_init(&acomp_ctx->mutex); + + acomp_ctx->buffer = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->buffer) + return -ENOMEM; acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu)); if (IS_ERR(acomp)) { pr_err("could not alloc crypto acomp %s : %ld\n", pool->tfm_name, PTR_ERR(acomp)); - return PTR_ERR(acomp); + ret = PTR_ERR(acomp); + goto acomp_fail; } acomp_ctx->acomp = acomp; @@ -758,8 +721,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) if (!req) { pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name); - crypto_free_acomp(acomp_ctx->acomp); - return -ENOMEM; + ret = -ENOMEM; + goto req_fail; } acomp_ctx->req = req; @@ -772,10 +735,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; + +req_fail: + crypto_free_acomp(acomp_ctx->acomp); +acomp_fail: + kfree(acomp_ctx->buffer); + return ret; } static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) @@ -788,6 +754,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) acomp_request_free(acomp_ctx->req); if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx->buffer); } return 0; @@ -1400,12 +1367,12 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) u8 *src; acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + memcpy(acomp_ctx->buffer, src, entry->length); + src = acomp_ctx->buffer; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1382,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1636,13 +1603,17 @@ bool zswap_store(struct folio *folio) /* compress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + mutex_lock(&acomp_ctx->mutex); - dst = acomp_ctx->dstmem; + dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ + /* + * We need PAGE_SIZE * 2 here since there maybe over-compression case, + * and hardware-accelerators may won't check the dst buffer size, so + * giving the dst buffer with enough length to avoid buffer overflow. + */ sg_init_one(&output, dst, PAGE_SIZE * 2); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* @@ -1682,7 +1653,7 @@ bool zswap_store(struct folio *folio) buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); memcpy(buf, dst, dlen); zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); /* populate entry */ entry->swpentry = swp_entry(type, offset); @@ -1725,7 +1696,7 @@ bool zswap_store(struct folio *folio) return true; put_dstmem: - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1900,13 +1871,6 @@ static int zswap_setup(void) goto cache_fail; } - ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); - if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; - } - ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, "mm/zswap_pool:prepare", zswap_cpu_comp_prepare, @@ -1938,8 +1902,6 @@ static int zswap_setup(void) if (pool) zswap_pool_destroy(pool); hp_fail: - cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */