From patchwork Mon Dec 18 11:50:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEB7AC46CCD for ; Mon, 18 Dec 2023 11:50:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 479F48D000C; Mon, 18 Dec 2023 06:50:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FE768D0001; Mon, 18 Dec 2023 06:50:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 251F48D000C; Mon, 18 Dec 2023 06:50:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 06FE48D0001 for ; Mon, 18 Dec 2023 06:50:46 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 887911210DA for ; Mon, 18 Dec 2023 11:50:45 +0000 (UTC) X-FDA: 81579772050.07.6DF57CF Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) by imf17.hostedemail.com (Postfix) with ESMTP id A041F40009 for ; Mon, 18 Dec 2023 11:50:43 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf17.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UFtS65bQK5h5w9cm85RbKzaORRgbOq0Q5FLRevBhnWM=; b=OiWOnMqQPKSmqeltikLZN2fHtlTqgfDM2QvY0dHMRhFGbuXZnswNqDhEfgL0Tx/Oj7rhLd +gvPGXkn6IkaeRlkH3ZW+Fh9A2gbZkQLafx8/Vfv5m6Hi7+3UHdmuwG4mztMeDUoyR/jeT kaOsk8BR1dJTmb5DFDWhVWLTlxDRiL0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf17.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900243; a=rsa-sha256; cv=none; b=kQYM4uNGAJrBmD6GVw8AgrI8Uh7nOOtgl4bgR8kW7tx5K5yAUdjEgnXiSmEk5hqCyoGBOk /6WbOW4mGOEP2ogi7IaHPrntQJTl9wK25jee39t6KDS0UOTKSPFHRjsNulixf1vkDQfBzj ShyFNf8+KKlNwcdiuR51hdbjZHLGWSE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:31 +0000 Subject: [PATCH v3 1/6] mm/zswap: change dstmem size to one page MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-1-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=2012; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=knbPWkl1sBJOnq5pAv9yoWZfZi+/5lFRxFAsoTFiAeI=; b=Skf5BWIwB7K3APRmPY9IP+iewRS012Hs6COguVdU5PIlg7L+x04yE80HEjpOimgwkl6cW+uik 96/4hLYrUTUCrBSCuMhN8meBpFS79yrWTNl5P7OeI56K+kia94TKabn X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: 1pddom7q7tfk7uccdn4d9j1fs6zhi8xx X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A041F40009 X-Rspam: Yes X-HE-Tag: 1702900243-69912 X-HE-Meta: U2FsdGVkX1+zAvg3NFNupycCBVAlBnBkSbhN/xNcLM72XTfvq1zLdlc4aWPzP4YMMKOI+a2HhjSIyBAWG8+QWAnMDMyFvQjsYj+q0m8myy/v8OlgoQf6RIY0b3oK4YI8uvGmE6vm9TQ/bML29votECsuz7ASSTUamTims19hbejIf0vfVsKG4nEasON/urF4KLLVCKsb6nEKUWCkxbQGN5W5+VaZZ/p7Z3AcQwVGW82ZQJyV1ZbIiw15EVHYFxU2vm5TX9nEV2IXuBdTDiiBbxEN3YBWRCAHCwi5snV5vJIQ71Pb5/durprYbQ4EoyVT0ychVfPDLTXmcU51JHLeavkiOXsENtQnnrRVKo3iZevrWDvCOxRI9aSJttOwZ89jCYhumpdpaRBw1NFWPcIH+j0A8KxYk4jUA0ur826eeqDgjishxRUWv45swSTrIA2jTqBeL278GxPy6ISpN19osxruwsfNehD4oVScV4wZuQ5gvJL5Mrwck6TIhK0cInAE/RuZy/RhwmgJaKlIYx2bAYF9F8xleKKbbVIBbJcMmn/FRcpOt7gMInFgSyDMciwlh+tuDSKccVvQmkfyMFdOGRZlcKqTEMbWw0HIU3Vi2qOLIXf1e6fYKtVKWdrYZcMcb1pd/y9PboTaDruWhMX3d6lD+5/nodmvldxlJSAGpIpCeeLtsBSYrXzhVTePvWh+vtpCJA1UOr/3wAfPFq4LiDPoLLnQk46xTKzgYSVK9UXb8Zriq/8/Sb2/DBwvKLeC/0wvCNQPWH9dtCKFh12g1eFVXDWWOaWyhDo7Mlg2BTJGVkf+U8ZaptTTQam2KeqMZmNaFYU9V4+Izhf6UxXpVnMB7syS3Txmcc9N3ruKkLnDmmU3goASNA37Z0oNmLusWOWVFqDdoQv1MsINRfFum493RDJNF6PVlEOamzlrmDlq5Bv1w37TKq44BwmfvEyjtrzUTia2Yaa8W2GtuT+ L35F9OeU J3JKXmXWYVkkBmcG/S47iGBxruxZfcGpTlP+/XjfRiZpj6pdnziPEL1y65nJgTFFg9rifQ6IpKtrBTZoIYLOJmuqS5KS25yeVY03wl+9vKNPFL/4f3hHRq4UOVYPD4b8nZzZn2UPLa6jUYSwvw3o5Btn7/8xwzuHfl/c4C7FYgZhly9EDEaOJ8Nk47IQlXEdWeYU90lGHjjHR9N8CXbBZ+UajrRU1t5tgRKnG63zrKzH+0wwq2GVU2LGnIk1GC1x4v9RrHIs7XFpl9U8d5ST7PxULLMjTl55KOfQRjDNsIdkyyELk8JwrTIEsDIZd6+P7qlaSdDPCnOZVL7IBEt0IVNnuPfyW+kuNSnkxJ9jxExyixAkwu5TAsS+O8Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the dstmem size from 2 * PAGE_SIZE to only one page since we only need at most one page when compress, and the "dlen" is also PAGE_SIZE in acomp_request_set_params(). If the output size > PAGE_SIZE we don't wanna store the output in zswap anyway. So change it to one page, and delete the stale comment. There is no any history about the reason why we needed 2 pages, it has been 2 * PAGE_SIZE since the time zswap was first merged. According to Yosry and Nhat, one potential reason is that we used to store a zswap header containing the swap entry in the compressed page for writeback purposes, but we don't do that anymore. This patch works good in kernel build testing even when the input data doesn't compress at all (i.e. dlen == PAGE_SIZE), which we can see from the bpftrace tool: bpftrace -e 'k:zpool_malloc {@[(uint32)arg1==4096]=count()}' @[1]: 2 @[0]: 12011430 Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou --- mm/zswap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 7ee54a3d8281..976f278aa507 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -707,7 +707,7 @@ static int zswap_dstmem_prepare(unsigned int cpu) struct mutex *mutex; u8 *dst; - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); if (!dst) return -ENOMEM; @@ -1662,8 +1662,7 @@ bool zswap_store(struct folio *folio) sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ - sg_init_one(&output, dst, PAGE_SIZE * 2); + sg_init_one(&output, dst, PAGE_SIZE); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, From patchwork Mon Dec 18 11:50:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9740FC35274 for ; Mon, 18 Dec 2023 11:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A6748D000D; Mon, 18 Dec 2023 06:50:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 00AF38D0001; Mon, 18 Dec 2023 06:50:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC2AB8D000D; Mon, 18 Dec 2023 06:50:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C37098D0001 for ; Mon, 18 Dec 2023 06:50:47 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 98A17A2061 for ; Mon, 18 Dec 2023 11:50:47 +0000 (UTC) X-FDA: 81579772134.13.FE8FF2D Received: from out-176.mta0.migadu.com (out-176.mta0.migadu.com [91.218.175.176]) by imf13.hostedemail.com (Postfix) with ESMTP id A1C9320008 for ; Mon, 18 Dec 2023 11:50:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.176 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XP4AUvt47Yj/TKGHm/+RxHfD94HrqsyeRyV0iF80ZQY=; b=Ah7Py1Cr7TQ8kPQHr2lQ+L+BK9BKvtfyC74mxSmOfGmZmXG/a0pPy78p2h7fP7DEy2FZu/ KQi3f5YSV+N1wSZngsLE8GPFid0Tv3OVGVT7HDKOwntpKrczpIydVGEO5unqMNH9KrjxE2 ca0Vyoa6sCrnjR6LDktzqUUQ7OCAZYE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.176 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900245; a=rsa-sha256; cv=none; b=W4LyxyplgX0lacDOY8Wjdv1Bfrb+J9Q2Tkq16M9q5wbeEFnZIGwZPF/eR373VPUUfp/rm0 cfPNGf4ucQ4MQ7FjsrtvtcR7AwdLRky4Ga69lmYuPxcZPxwBW+y7pcy5XSXQG+KVxxlBYv dLmZath3IIh00ui9MEoq+1oelAHHEcU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:32 +0000 Subject: [PATCH v3 2/6] mm/zswap: reuse dstmem when decompress MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-2-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=4279; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=JxHludCxMK0OPSXqNpXYMeMWjmEk17jj62CkEYJJ0Kc=; b=FkiCcWV+1XpQFAG5Ryg3DEVWXLZeHzNrlB7Z4iSS32PbvemI7sdMZ3kL9sbWdE9EZPQ8OiHKX xI4/eIK1X/tCF7SpnRPu4ma50Jp3RTDNqWgnkzjDcnf1OYQ4y7gNKV3 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: A1C9320008 X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: zowqhyda1kf8x7g6jheamxp6wrf83fxw X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1702900245-217174 X-HE-Meta: U2FsdGVkX185kLcC+vQ0lSICjwa9GDiiReE+g7xRfU+5wxk0dJIYF3C6eX6TZ3sYSC71OPv76ouEexWSr05oUDZQ4OpKZOrO/YU7Kw5WHb3XhZFZG0OqLwTlYdE3dW3XabhHUsJMs6cqLmv7RIYZwKIufXENjTSB1/do5iZmv5MYtf9Jcek9j9+n/k1BlfmcZZIgDRYFaGD3i3sDKNjxDnAB+yekSkl6Nn2o8Oj+Hg1uMJvYFgzVkZbVFTI3DuON81HO3jWxwcWMCYiOvw9nidCis+whdeFEKuZdF8tdoBLvDpD3TkS7OwyV3O70i6y/1FIrDk14nr2YSW97OkA5L13no7qiIx87rSLav7iiA+gfZyqmbfgtJB0n4kX3wtn7GEIHYn5z0zrV/rR9T+GgYf8KIiNTPBat63ZkFhzggAF1Pl8aWNjOR+Djm/OFdbZYUEuwDwUYxHPV6bnA6+xudaapsFyPFUErGL0R6Mpdj0JGiALVzBSGm7nsj25EiEdqp7B+XFMR8a/Phvm8t5yn+WGEL3xEsNDDEMXgSrg3i6k0QnnekYxWP4C8cuUvQ4in5KTyzEhLA0RXI4ar28YLsCwU4au7lgLgxBcYbqrxOoMoz6DSlqC+ej9cxFAPVQWDSdiNffjvgI0b6Kn5JfV3GRi70/jiFu4aJFjPvKmkz9T8f2fGCF14d6ralcl8l9foWrdTlhentvk+qVIi11ndzO7jd8q0p46b4Er2wKuGAQvrye8/P0QmLSFh2Txrf6QdoTjKC87ZczFAKhPR66T0EOAgUXBTUkqygMSSOOFjrG/PHDpEV6+PneLd7IFmIuyi0Pe2GbR4P15jty8Rm7lzAe2PN6lLlmoFYadPnmmsC9eImczhAMzd6bIlyXH8T/PbYR33eK6B7tQIMN6lLOaUKGP0sXJK6lWN7OWSOvQADO5Cvv6uZgx+AjS1GgZRzaiFPkd8UpUvemZ46KsdePk L1X6fHgG hWM1q4zmbBnT77WAWDX8SieeqlTgIw9NXlmdU3Ql6ZDlt+qfOn152f/4kMSGv1bvJaJEIKqlI4kNAtBVw9vcwJkQjnRAcZKd8yiaYVlq/8bLb0gfnOJ39umYO3cIs7745bpXPZWVRvgp8kYG0fdTWW54ARAhICwSCEw8/xTcJ/EIWRAKsC1F5UDjAmfARXbSmLIh3hfzLXA4UXjKvzC3ERc6YCRupyZtn3l7IyGYNHxaCbupxjnImkbO4eDhfKRjUHQVj4JzvegeYCoUDIIjmyt6s2TZBLCUuGy5g3rOeWt9WsOOL7lhuXGJrRR9yfVjXJyV/agkmEdHfaoKd0hBKmIiFxC9+aN2i256s20bSCk/E7ZPrbRtNeUcFDCy/NARmjHh11NL08PFakFAOx/8NTTzI2nWiGkxRLkd14dZHtF5z7yEtha1+4Qnjxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the !zpool_can_sleep_mapped() case such as zsmalloc, we need to first copy the entry->handle memory to a temporary memory, which is allocated using kmalloc. Obviously we can reuse the per-compressor dstmem to avoid allocating every time, since it's percpu-compressor and protected in percpu mutex. Reviewed-by: Nhat Pham Acked-by: Chris Li Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou --- mm/zswap.c | 44 ++++++++++++-------------------------------- 1 file changed, 12 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 976f278aa507..6b872744e962 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1417,19 +1417,13 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct crypto_acomp_ctx *acomp_ctx; struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src, *tmp = NULL; + u8 *src; unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; - if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!tmp) - return -ENOMEM; - } - /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, @@ -1465,15 +1459,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* decompress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + mutex_lock(acomp_ctx->mutex); src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(pool, entry->handle); } - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1482,9 +1476,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else + if (zpool_can_sleep_mapped(pool)) zpool_unmap_handle(pool, entry->handle); BUG_ON(ret); @@ -1508,9 +1500,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return ret; fail: - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - /* * If we get here because the page is already in swapcache, a * load may be happening concurrently. It is safe and okay to @@ -1771,7 +1760,7 @@ bool zswap_load(struct folio *folio) struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst, *tmp; + u8 *src, *dst; struct zpool *zpool; unsigned int dlen; bool ret; @@ -1796,26 +1785,19 @@ bool zswap_load(struct folio *folio) } zpool = zswap_find_zpool(entry); - if (!zpool_can_sleep_mapped(zpool)) { - tmp = kmalloc(entry->length, GFP_KERNEL); - if (!tmp) { - ret = false; - goto freeentry; - } - } /* decompress */ dlen = PAGE_SIZE; - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(zpool, entry->handle); } - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1826,15 +1808,13 @@ bool zswap_load(struct folio *folio) if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); - else - kfree(tmp); ret = true; stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); -freeentry: + spin_lock(&tree->lock); if (ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); From patchwork Mon Dec 18 11:50:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF95C35274 for ; Mon, 18 Dec 2023 11:50:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FEEB8D000E; Mon, 18 Dec 2023 06:50:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 538618D0001; Mon, 18 Dec 2023 06:50:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38A5B8D000E; Mon, 18 Dec 2023 06:50:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 16D9C8D0001 for ; Mon, 18 Dec 2023 06:50:50 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BDDC5161101 for ; Mon, 18 Dec 2023 11:50:49 +0000 (UTC) X-FDA: 81579772218.14.9AC069D Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf08.hostedemail.com (Postfix) with ESMTP id C5F29160017 for ; Mon, 18 Dec 2023 11:50:47 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900248; a=rsa-sha256; cv=none; b=jolkInwJ2RoFGIJiJ7m0QakTGwhEYTMJ8RS0bBZ6Dl3wFaIxDnW6xQt0WnrBYTMIiTkz8w hLBNH47my6x1y7u13frtKWmVvxpBn1LnQdQ2s1YNUslHK5b0M1yfc22VSIHSFW7bTaJrWQ /T4GVixCMwjHKMjdmlDUIIT8hv2apow= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900248; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5XHGau3pXpBIRdgWSOxIy56lBlKtyLT1nxaNBp7TWqM=; b=1FnStE3jcHp9mulBAoFr0iM5DqDf8qM45r/bdEytsVkS+vYqu/+bXmHTZvTxpwSzlUrcNo SjVZZWLdNURs9KK3J1OUz1TQZY4Yk9UYBIUaxQG5QolMsetilCTAnuA+xsE0lt1TgbvCwS 873N0RrlE5v29+v32y2jznUPueD3fnQ= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:33 +0000 Subject: [PATCH v3 3/6] mm/zswap: refactor out __zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-3-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=4703; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=B99wC6AH9+R5m6Djw0+W/AJA7ag/bUbsLCKvYrrlo8Q=; b=ild77L3TDGLGQu8Eca6JaXi/fIaO5CaZVqyj/qCaIHMIACqX9eH+pkWshvQ4Y9TZWp3ytCZsC rYJYgewIBt6BWiDIhPEOZvJMVJypVdwjYuUQ0PbiowCtIZh16LIxHdH X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C5F29160017 X-Stat-Signature: j7eht3b66gmc4ntu6nmsbbs9hsqt8b5c X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1702900247-593297 X-HE-Meta: U2FsdGVkX1/ysTU2jnLWitHbG2/+C1qaMy8ANE4JXL60Twveg6xuqo5ZyV1m7ZShesKQEMC8TyRUwakSu9dRNYDvjo2YAepvZMm6QXO9lHXt9XZr85JikZgwSCLD+EIbiF/L7/avgSc2HNe7KTeKms5FcXroYu47Jn3kSVa9n0aQ66IEv59tXAWvzVEjgO1v/MZcR3lwCUhvw5dpv5eXGRHjfX28a6IjwVxqsSRAaoxAPVHiL/1ipgwNeIziAiV48KSTJFaEJEHsgvFDC9do1viWQizJrIrtPFji4ZFu36G8kltQLX9FyGTSG4N4xgWmDjJQmxGbfF6k3K5KbNh12JQBOp/er+b67N1WdU2owGRSA28wVMqw95YBETrg72M2DpF4mJ7K7dqDWGvADbNKKh/erA00Mv5WxKq6C1cfT6twPB1J1Zw12Y+aUzWqecYtBhPOHfyVUxyjl4JOkxjuS+Pn145zb8W3YA2hiVDDBVJTO0mVmZqQhSCfdZORtf50e8XJ7NT30RHNfwlDRDQX5PvzjSO5C4igQsHPIP78IV6wTT4B4LewaKKmSU0axq7keS+qUb67Wt8UG6WEeyEs9h0EL3wSilJCius/RhvXi0rK/fKxwpyaw6IxpmE++Dg8DWtOk8uJzwG7H9gvTiG26AFeX0zAI9xytgrsQjJbDH8vruAtD7e9pfAsYMZaKDPmAoHCkh1RBn8ZbtP79IZFCxo6Ei4tXlkPAUliMT3BDpmXUhvCKBc/6IAuGf3ejWh/xJaXLQUAPaDMN68OYoZaconGExEeyBUzA0Eer9hbEAIfR/j5b4r01D5h3lqV97qIgnhv6m0IcuvRbLSGOMVWuhIwXi6VzDuE8YS5yQWrJogBEdRlxEIOXyMK3Mzf6FKo1epEzRYd0qWLrDF8HZ35lUUA8fOTwoHL3ugW3BF7MgASZSlHQiRfvmkHk6pbDCtQ0ujk2sQRQy9FK9XgQV7 lmNPc36h 4VBNE5B3Z1+Mprg2T4i1FY7upK0X798hHHN/pF6Rd1nVsDSXoeasxBv/eY4YCaMP6eWR3lGTC+cWlbGyJFM1jfAlsHz+rV56n0SXC1dnKnbYYRXs9ekCGLjg5aao+PCxQGMiXc6Yjv9oV/RxUVwWfDB22Xw78pL6uoFdUa6Su2PagUSyOiLeYl0ToSS8Dt67+6WiEvvKOEAdKNg8Izh822idpkJKLoTvakLqVoPMaR8HVJ9aXSLeU+eHgvIWZnOqzw+jn3/j+ZJDoZd07O1oOzrVMpno+7ZUG4AZbVp7HHaIk2SJ+UVHTpP3c1KeeTaazKoABu4BIWDDAuogzeLaOeEFTGkwymaCfjlpZS+levTP6UXVAFktJLtijAXMnC6i8s6N7BiWPWaictdHyWuuDDvT1BIrAjYKdTLf4pkQhX3ceZLU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The zswap_load() and zswap_writeback_entry() have the same part that decompress the data from zswap_entry to page, so refactor out the common part as __zswap_load(entry, page). Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou Acked-by: Chris Li (Google) --- mm/zswap.c | 92 ++++++++++++++++++++++---------------------------------------- 1 file changed, 32 insertions(+), 60 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6b872744e962..3433bd6b3cef 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1392,6 +1392,35 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static void __zswap_load(struct zswap_entry *entry, struct page *page) +{ + struct zpool *zpool = zswap_find_zpool(entry); + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src; + + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); + BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); + mutex_unlock(acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); +} + /********************************* * writeback code **********************************/ @@ -1413,12 +1442,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry = entry->swpentry; struct page *page; struct mempolicy *mpol; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src; - unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, @@ -1456,31 +1480,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(pool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, entry->handle); - - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + __zswap_load(entry, page); /* page is up to date */ SetPageUptodate(page); @@ -1758,11 +1758,7 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst; - struct zpool *zpool; - unsigned int dlen; + u8 *dst; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1784,31 +1780,7 @@ bool zswap_load(struct folio *folio) goto stats; } - zpool = zswap_find_zpool(entry); - - /* decompress */ - dlen = PAGE_SIZE; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); - - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - if (crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)) - WARN_ON(1); - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); - + __zswap_load(entry, page); ret = true; stats: count_vm_event(ZSWPIN); From patchwork Mon Dec 18 11:50:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 997CEC35274 for ; Mon, 18 Dec 2023 11:50:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FB3F8D000F; Mon, 18 Dec 2023 06:50:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7810B8D0001; Mon, 18 Dec 2023 06:50:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58B268D000F; Mon, 18 Dec 2023 06:50:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3E5578D0001 for ; Mon, 18 Dec 2023 06:50:52 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1D745160247 for ; Mon, 18 Dec 2023 11:50:52 +0000 (UTC) X-FDA: 81579772344.21.CBD7783 Received: from out-178.mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) by imf04.hostedemail.com (Postfix) with ESMTP id 2E66C4000F for ; Mon, 18 Dec 2023 11:50:49 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zJqTjnZTPjIHItkHechsbAK+vOUzSf9pl11pwS87u1M=; b=5SXcAqZiHXlzG9tXYE+O+nJHbyiOQzpMVL4DKIjzsM66a1E3Mzt2iNMnoXUpZKAl06+ydW 1s58KqrbZIjuoJKIh7g8gdDfCsjMd8J1XhsHWBfHwIeL3E5OuTLyVeiu4gYkTnxMFX13kc z5xYB6k0fhNKyD0w1p2jg3pZHIBhIM8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900250; a=rsa-sha256; cv=none; b=pizniACf318WatVkl1zN7Ru4Xa/Z20FDif1j2984d48x/xXo6WYvFtYVb7b7UgejsiMJ+S HkETCiRVBZiLwsIEv1axLHxWXKh8wfDt/uBxVNW+3+pCMFRgGcbvtlQUNhhAg+0OypcwTb aUD4i6eWQOfgzdYJkM1Jvauf9SnPuyA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:34 +0000 Subject: [PATCH v3 4/6] mm/zswap: cleanup zswap_load() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-4-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=1470; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=1eOM2WlTMMQMFmZcsE6sU450y2r77yOSYBMqqAB7HFE=; b=mA1a+oM0pO2x4ySl9ZU6T4A1b2XOYkGKZpXj4gIvy56X1aoJsXnojsb8m/11LLzDFIBsJEhQx i+goKQs7a1CAJLnNJrSVIiCo2UaAEaJ/sfFe3MjPRQ5aHJnbPdiJDGR X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 2E66C4000F X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: iu6me38tp79wnf5kh96hzmuaji7fspha X-Rspamd-Server: rspam03 X-Rspam: Yes X-HE-Tag: 1702900249-466608 X-HE-Meta: U2FsdGVkX1+AVf4Vvep4YOW+CYYqt8QPRZfsS/8b3Zpa5PFAyxgQuEE5toCSIGc0zM4yoHfufZAKF/HxK+Cm3awHBhd9JcaSLKjonzFcJJwNjZhDomfDjmLFCtRaDtX5rdY0aXW4f+PblRHFAwf7draEG3ixLO3vV/LBMyO1wJCqUHgA4QL+Lq27br1X8lQ7mEQufjP1bbsNvHLP8dfXR5EcjeOrtoSSi9fVwtOwGkqGl8V7hwd7YX2Bq15bfCW+RphXtsxp4xNn/9HhofMXrV0IXcSw48f2Cngl2SdBuEVG8GI4SPYnXEgkw9hQ/Za1ym2aE4myHTjLIaWz3tUWzcyz48E9/OjXJiuOwDX9jih+/AIkTYnIHhFIIHjBYNfqhppD+IVW12abV615dJxihjOKYvJjpy5sHwRKBejGudlA1oyOVI5ZxSw/QiRRjyWq4Z16txtU93JUOTR643/2l4p0sZ+Ae/+o6s3dxYBXV3f1qUamdy98mOlv5bmyaNxcxHaMpvWYmpP+M607RvBeXkUCjHP4cWuY9hwJhHR+YkLB1sjARJ2VIU8geta+E4aS1RKZa6n9xVXqU5hCNVrsyhBPjaNNoS4yETNcEh0dHRQvQt1WmqaILZKFLbQXqw66DuaopRMDqL/fMz866fVv7f82SjqKycTFp7N8hu+Z/GoCNRmEoreu/R6/ULVra79j/CmCvCX0G+83Tgc+hcRyMmIzYCqo08rgxpBVPFyCv9i32boqcHv2LAcwQ2cCQkuc+Z0+3WXz4eiTZLq77SuLKFThgs0QNgpxMEVrrbTTAxG2wHfEnmTdcLPnNchjvv+nfuoIOlVvmvtgx/VFnhgS79t9FH8lRCArqox3NYyPiufeir/ivQ8gSu7sSDfbjRaY3kGr+8tRllL/szgS9nV5jsvKaOFnyIZf2vZ6A/D0n+VUTSSeLAcppVY8WJG2rs1MGG/0BxT5M7QSVVH6EIj K6bOfTF1 rtvGaIPZAeWU8BUvmtfugVVtZmCLRRcLIaefkVoQ7nTuIu2nsp5KUL8MvGE9y1Ca9pfB/51l4yIVQIQiT45N61ixtqcfXxV+Hn4xDbivDtR6sC9WzE3wpmvWuIiLUgnK5W948k0Fr4Mnol3yIuTXkDkfiNSiqpOKvphIXkDRK/RWV2R+TtGmMmpIiFaI3tcfMvFdtcxths4xAPI4HNUiUwd4aWydDMfWgiZjjd3/HmPNOO/+yq2jru4ZbgAP7rmQlDWleYGcZJ0QfOmaWCRnnqvATX2A4WmDde2Ti6PWEEUlC35nceEvNZ069OZO1Nd9CJdKUYvgMQ2uklvT5EZa3yKwINH3iY6ffL7D+wQEo//nHL5SAsbgGxMKhbgQEpwPp2BMTJNgYkWSelpo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the common decompress part goes to __zswap_load(), we can cleanup the zswap_load() a little. Reviewed-by: Yosry Ahmed Signed-off-by: Chengming Zhou Acked-by: Chis Li (Google) --- mm/zswap.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 3433bd6b3cef..86886276cb81 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1759,7 +1759,6 @@ bool zswap_load(struct folio *folio) struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; u8 *dst; - bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1776,19 +1775,16 @@ bool zswap_load(struct folio *folio) dst = kmap_local_page(page); zswap_fill_page(dst, entry->value); kunmap_local(dst); - ret = true; - goto stats; + } else { + __zswap_load(entry, page); } - __zswap_load(entry, page); - ret = true; -stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (ret && zswap_exclusive_loads_enabled) { + if (zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { @@ -1798,7 +1794,7 @@ bool zswap_load(struct folio *folio) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - return ret; + return true; } void zswap_invalidate(int type, pgoff_t offset) From patchwork Mon Dec 18 11:50:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1C2BC46CCD for ; Mon, 18 Dec 2023 11:50:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C0C48D0010; Mon, 18 Dec 2023 06:50:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FB9B8D0001; Mon, 18 Dec 2023 06:50:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D9CF8D0010; Mon, 18 Dec 2023 06:50:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1BD7F8D0001 for ; Mon, 18 Dec 2023 06:50:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E209741147 for ; Mon, 18 Dec 2023 11:50:54 +0000 (UTC) X-FDA: 81579772428.20.548C6B2 Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf05.hostedemail.com (Postfix) with ESMTP id 1503510001A for ; Mon, 18 Dec 2023 11:50:51 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf05.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900252; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7HR/MLv2aBLIXlJNr8b9Lqs7UJ1j9JsFWT1nUw7L6eQ=; b=4iHJqcrKNyIRSVJwS8hrSlcoX7qAnPq24xnxNsNi1JeMAsgnL1y0l/OApEi9c0+UKZtI/f rSJ7VOfT1xSNIsgYORFluTAY5nGYuAo8UmH/2hcWkr1KlZ1pB0lT11rVrWI2Qc1yplH5Vh x6KT3Aq7bSA5yVTFCGITsu9W5Eoau+k= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf05.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900252; a=rsa-sha256; cv=none; b=jDuzalDt0Bv752RiHa0eD/u8YTZro01C0FSJ0wHOeLy/JuiEjakurWgEFq6zyNMy4LjzUr Q6JPIkUOYBqvzN/Y3z3+0JRi4CGde299G/uzu+lOKiK6uN0AcxmBALi8ADMeYt5LUenYvF twTnTtlQFOquCLOHsOPgudZRVNCM+/M= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:35 +0000 Subject: [PATCH v3 5/6] mm/zswap: cleanup zswap_writeback_entry() MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-5-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=2215; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=u9D8B3ksO9G67avNjANTAO3qS+/SASwDD1QW2eXuPmA=; b=bX9QDtAp6LhqXJx8Yr0LWdcF0z06lztvYdnRzlwugcBS4LaRmP3c9JNAMHRIFxNxSVfv4KXJP 2yn6oW6AM20DLSOlXaQvsN42SgrRlNXTLt3ZWYzqDYQTNtDhy74I4xe X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: 75wychpsodb7tcit5yikocrnbirbegcp X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1503510001A X-Rspam: Yes X-HE-Tag: 1702900251-916467 X-HE-Meta: U2FsdGVkX1/sPXxmuW4xArzsDpwySKHn3WeM+CjdY8xNfhwOkyE+PoLWqMyO2vPxyxD9xFXQefmccOBRdoWO1dCBWjCBr7/y02DB4eVYgHKdDxBD4z7ixMowfZ7kW9bL6oqCQz4x59R2o0LHbNxScm/dINh+CHERtJGXfGcejhJyWoQiEuUgDbnDdahmOJSfwEEngWS0br1b5nibDrGyU7eGIWejBDtcRwR+6q06rEmAGACjWZnaE13SvsB7rYFfOZm+zEbYt4xiMA6pSSgZbQglYwXCh1dYkDUpJeHaDenJgTsNZ08layqtfhH2jSCyxy3VRGo+JAJzkGQbQIeveBuUCPxPyB+EdFxM8hIAqPL0g46f6T2xruoCTZspemYLKa6fiL1einA0ChjT4wPXcxFIII32BoESlYH3Z8vABMOTOfKRrEg3OAin3OJifSE9ib2+TelUqIQJd4cfX96cQHtMpXQIpNKnS2H2yASw+WYDpGRcYuRn76z0NhrLvVhjmljOaYpn1qxiR8ZarecE//YpMxeZNbvNNhv6prIVKw1t8nYaTWMo/e1YZo5tBUrj/nuX7ii5NlmjARLiWn1pGgogKT20ohKa7Q2/WM5Czb74Ruy4+syGCvwClU6hpVo+qtyCZ05okIVvZkSTH1igHRcZapbDKsvMfRuoYAi0PRWbV5AqtxVNY5FmOlNbGGtD9OmhMdWP6Nb+tJE+bc8SMYT6RjQ9cEA8rjYV9F6GadTuO7Cb8UC997SbcMLkBFEn49iEklIAD6e4MkAOuIJFsg0lwMccWcYFKcN284CtcYGHjpqBsMlUnifdc8gQtrgPKlXyApQfcPxv6915pAZ2TEYtOgvd2qRpnfLj9+o9vk3g//nE3Buy/zwdTeWKFnqGCW3NVMpwpbsyF2UMopBbKNT7Enr5mBOPoY+t8BtuvFnMXIGtZKweO1tDNlI2CuL6N4YHmcnBSQxzb/1R7Op nTifhgdp usLLFyM0RHSU/GO1oGyOa6UgeJD5mAvooufTUitCK9NwZUjKCJ/pTGJsvLQicprjRxwwqBn05E63lGBl+5VYe2CC1Ylbj75sEg4Xk0CWNvSIKmKlDBTRexTNPWxHMch6Dg5SEQFAf8EEqOmydt7y12nxidSX5YrUSf+CaEn/vxwACGEU+Frsws8tHPPCexSdvfhT4pcsp7OP9PPh7sjS62efnV2Q55yhmK1cNcMq5Z6ByZri61+XJvGzObzV5hlyW1dbG9pow9ht2jUyrH+zcf53KCPHLYyqpqVCZQyAOrbSgxaUof//xZD75NlpQLkCvYsOb5R+L0vRUYFYKO7uJff1V18WzyTYbjflMPc+cri+lbot2REudnjN6ugCgMdju5C49osv/NCxP8b8wLCrTSTzUaGD+0qQp+9vpqsHOztOITSE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Also after the common decompress part goes to __zswap_load(), we can cleanup the zswap_writeback_entry() a little. Reviewed-by: Yosry Ahmed Reviewed-by: Nhat Pham Signed-off-by: Chengming Zhou Acked-by: Chris Li (Google) --- mm/zswap.c | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 86886276cb81..2c349fd88904 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1443,7 +1443,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct page *page; struct mempolicy *mpol; bool page_was_allocated; - int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; @@ -1453,15 +1452,18 @@ static int zswap_writeback_entry(struct zswap_entry *entry, page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); if (!page) { - ret = -ENOMEM; - goto fail; + /* + * If we get here because the page is already in swapcache, a + * load may be happening concurrently. It is safe and okay to + * not free the entry. It is also okay to return !0. + */ + return -ENOMEM; } /* Found an existing page, we raced with load/swapin */ if (!page_was_allocated) { put_page(page); - ret = -EEXIST; - goto fail; + return -EEXIST; } /* @@ -1475,8 +1477,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; + return -ENOMEM; } spin_unlock(&tree->lock); @@ -1497,15 +1498,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, __swap_writepage(page, &wbc); put_page(page); - return ret; - -fail: - /* - * If we get here because the page is already in swapcache, a - * load may be happening concurrently. It is safe and okay to - * not free the entry. It is also okay to return !0. - */ - return ret; + return 0; } static int zswap_is_page_same_filled(void *ptr, unsigned long *value) From patchwork Mon Dec 18 11:50:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13496763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 184EAC35274 for ; Mon, 18 Dec 2023 11:50:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B544E8D0011; Mon, 18 Dec 2023 06:50:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADDAB8D0001; Mon, 18 Dec 2023 06:50:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E1148D0011; Mon, 18 Dec 2023 06:50:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6B2168D0001 for ; Mon, 18 Dec 2023 06:50:56 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36490120A55 for ; Mon, 18 Dec 2023 11:50:56 +0000 (UTC) X-FDA: 81579772512.06.EEFB5B2 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) by imf11.hostedemail.com (Postfix) with ESMTP id 4554040026 for ; Mon, 18 Dec 2023 11:50:54 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf11.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702900254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JE7KLoKyK1KRdi42drS6a//u5Xi4f54POK2HQQg/pPo=; b=KT7XZc6pNh30i+HCeoxiy717UBXMelD4myNF4NMAkShUCg2ke7XZZtIscAiZGn3Z/yZ1kg qgEJ4kHD0T/UlQ7rNUbSyaXJhoX4EpZyGx8XBZDmLTKGtl7d4i+43jEhhcdwHIgNJCPBUp HPHEnft/UduATPk0GAHs0aKiTPgTMwI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf11.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.180 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702900254; a=rsa-sha256; cv=none; b=Y0F7dXn7T1FrT1BaSn3RvmIKBO8eFoj2zQrynv5AZ77U3oGh3JfQkJu5VglQ8IeRUMpkIF 1K9tV7m8TsSQosQ6hJQsUGKqd0GgcRCTKStvoHPVd4yNuPyTeXC6COV5zvNoMwFiCWVWSC 0HVH1f8rYWPf0NRZkbnZc79UkOgQ/VY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Mon, 18 Dec 2023 11:50:36 +0000 Subject: [PATCH v3 6/6] mm/zswap: directly use percpu mutex and buffer in load/store MIME-Version: 1.0 Message-Id: <20231213-zswap-dstmem-v3-6-4eac09b94ece@bytedance.com> References: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> In-Reply-To: <20231213-zswap-dstmem-v3-0-4eac09b94ece@bytedance.com> To: Seth Jennings , Yosry Ahmed , Vitaly Wool , Dan Streetman , Johannes Weiner , Chris Li , Andrew Morton , Nhat Pham Cc: Chris Li , Yosry Ahmed , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, Nhat Pham X-Developer-Signature: v=1; a=ed25519-sha256; t=1702900234; l=6471; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=culrrHCm/Hf4dwHwtvZ06lQ5NsEncVU/QGy2Bp4v+0o=; b=b7n0OpxYpR3XA5GiwrtGYchVJRrAlhz5jqYK7l+kMC6lZWSSrzA4mZoc1L8QALpl7yA1Z8BpB gUTc5av/dY8AUdT5lUxvytYE9E+8PlB9bUSPPKrHM97afZKBM0ohjLG X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 4554040026 X-Rspamd-Server: rspam05 X-Stat-Signature: gdk7yiicdkeo6iq1ircyscx6mnzgdmg9 X-Rspam: Yes X-HE-Tag: 1702900254-622992 X-HE-Meta: U2FsdGVkX1/zT6qVbFqtfSR3+KhcOeTLnmMiIxq0rBbEZKax6KFdzsSvpdhsLCC1Y85RCNTsineREWbAAeYFUaLSacgU9JCQU187GiLWcVK9E4IXzWtUBcU0LDHdJtZRIpz23hwv7aTBrMJxrsXmZedw7+Q5Of0SEOlTXx6lVzDb+H+k6dxqzh8A0gYHYXn0dC+rKY5bJjgvpVQ6hr52niqT8OkEcnxPMBLAOARhTSB6D4c4ZhUO4Ot46zWdoPcZv9jYVqBpkwDnRbsXInpQ5SuKONgV/6eSUNnZfnsh2n8QEsfyiTnDfz8UNOLcxXwG9Ziwar1k/OurO/XfxOcsX4Lem8ZfL2GUU5zOEYLiXDwUAKEkDCwevK6SIh56UIRKT8U2rAp0ldp09Ry0Wl3hp/D1368z7EE0HCQ1Bm/TdRafcHUm9FNxiHghBOn33MwdkesGT5oWHyH9kA/E+bGkcMGK8TY9swxbDcHj8do2nTNrqIcVjAy2VvZSNQAogB73jqxmpgZakeath1UljBNp4rn/6ISf+dPe9+rsu1lbY801iTAPUXFuxdUibgPw0LcfKQYIdeJ2sOLfBvtx+Om+gMoXdRUAMQGelCqqWSgFaKX7xFSiXjo2u0IxBLBWPUI9CFOnc4XiJuGOPBKh1sYqNfOamNFbTcj6lRyGsRyJYCqL2p9KPLI26V6+/FMdjwvQtIC/DjdmFJd/NTqfAQAP3hxk/Njk0cWosulcZvCFUwl9AYf7GhBAKOCyl1kVk4NcpiFrp/u6GQfPoXGG+VuOxeT967OI9ULstGSC/+t5gWmh4H5r074Cpya++mX4fZ9iFrlB1MDsSy10EzmW8tluEtg452zWL/IqAVIZL6tXQPF1fydvdQNC6CdWtk+9F5Pf1U54rAdgZSrBHIbBkva3nalhvcdDe07ZnPXba155uNRpAWOmRuFifMhoJ6pY3ZZdUdDZUx2X6cnUIVHdWjt d/yqpMpd ikC0ClLsuzoyzJfL7JTfhS+ZOt/eFBIa8jTBKkl0aGDeAPH5vWUGwUjqS+qWQgJpmkOTw0WMii530qJJtsVNQJEWp/zAsi8XbncgnOd/meKY2sdAMz4NvW1MPw+JkiT6g+bz7u6YPGzwwaXA2o8c7Iz2ANIKPr32nPC+yDwnVSEOzCd/UHsVmMen8ryZ6rGvKtRy+X5XyQ1mEonPDSCqf+IRn6jFRzzRkl5Tsc7zebULd8ll++H8fPidytJ24gQzwVjJ+xV36dqYEiYkKgm+SHFUB28XP4cNn6u7j7+muQ+fNZCGO+WdfAWAUarINHAtFOgD5dQ8KKT2bporcXcEzNcQ2BYEeQmjk07J/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since the introduce of reusing the dstmem in the load path, it seems confusing that we are now using acomp_ctx->dstmem and acomp_ctx->mutex now for purposes other than what the naming suggests. Yosry suggested removing these two fields from acomp_ctx, and directly using zswap_dstmem and zswap_mutex in both the load and store paths, rename them, and add proper comments above their definitions that they are for generic percpu buffering on the load and store paths. So this patch remove dstmem and mutex from acomp_ctx, and rename the zswap_dstmem to zswap_buffer, using the percpu mutex and buffer on the load and store paths. Suggested-by: Yosry Ahmed Signed-off-by: Chengming Zhou Signed-off-by: Chengming Zhou Acked-by: Chris Li (Google) --- mm/zswap.c | 69 +++++++++++++++++++++++++++++++++----------------------------- 1 file changed, 37 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 2c349fd88904..71bdcd552e5b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -166,8 +166,6 @@ struct crypto_acomp_ctx { struct crypto_acomp *acomp; struct acomp_req *req; struct crypto_wait wait; - u8 *dstmem; - struct mutex *mutex; }; /* @@ -694,7 +692,7 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) /********************************* * per-cpu code **********************************/ -static DEFINE_PER_CPU(u8 *, zswap_dstmem); +static DEFINE_PER_CPU(u8 *, zswap_buffer); /* * If users dynamically change the zpool type and compressor at runtime, i.e. * zswap is running, zswap can have more than one zpool on one cpu, but they @@ -702,39 +700,39 @@ static DEFINE_PER_CPU(u8 *, zswap_dstmem); */ static DEFINE_PER_CPU(struct mutex *, zswap_mutex); -static int zswap_dstmem_prepare(unsigned int cpu) +static int zswap_buffer_prepare(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; - dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); - if (!dst) + buf = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); + if (!buf) return -ENOMEM; mutex = kmalloc_node(sizeof(*mutex), GFP_KERNEL, cpu_to_node(cpu)); if (!mutex) { - kfree(dst); + kfree(buf); return -ENOMEM; } mutex_init(mutex); - per_cpu(zswap_dstmem, cpu) = dst; + per_cpu(zswap_buffer, cpu) = buf; per_cpu(zswap_mutex, cpu) = mutex; return 0; } -static int zswap_dstmem_dead(unsigned int cpu) +static int zswap_buffer_dead(unsigned int cpu) { struct mutex *mutex; - u8 *dst; + u8 *buf; mutex = per_cpu(zswap_mutex, cpu); kfree(mutex); per_cpu(zswap_mutex, cpu) = NULL; - dst = per_cpu(zswap_dstmem, cpu); - kfree(dst); - per_cpu(zswap_dstmem, cpu) = NULL; + buf = per_cpu(zswap_buffer, cpu); + kfree(buf); + per_cpu(zswap_buffer, cpu) = NULL; return 0; } @@ -772,9 +770,6 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, crypto_req_done, &acomp_ctx->wait); - acomp_ctx->mutex = per_cpu(zswap_mutex, cpu); - acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); - return 0; } @@ -1397,15 +1392,21 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) struct zpool *zpool = zswap_find_zpool(entry); struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src; + u8 *src, *buf; + int cpu; + struct mutex *mutex; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); + + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; + buf = per_cpu(zswap_buffer, cpu); + memcpy(buf, src, entry->length); + src = buf; zpool_unmap_handle(zpool, entry->handle); } @@ -1415,7 +1416,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); @@ -1551,6 +1552,8 @@ bool zswap_store(struct folio *folio) u8 *src, *dst; gfp_t gfp; int ret; + int cpu; + struct mutex *mutex; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1636,11 +1639,13 @@ bool zswap_store(struct folio *folio) } /* compress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + cpu = raw_smp_processor_id(); + mutex = per_cpu(zswap_mutex, cpu); + mutex_lock(mutex); - mutex_lock(acomp_ctx->mutex); + acomp_ctx = per_cpu_ptr(entry->pool->acomp_ctx, cpu); + dst = per_cpu(zswap_buffer, cpu); - dst = acomp_ctx->dstmem; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -1683,7 +1688,7 @@ bool zswap_store(struct folio *folio) buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); memcpy(buf, dst, dlen); zpool_unmap_handle(zpool, handle); - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); /* populate entry */ entry->swpentry = swp_entry(type, offset); @@ -1726,7 +1731,7 @@ bool zswap_store(struct folio *folio) return true; put_dstmem: - mutex_unlock(acomp_ctx->mutex); + mutex_unlock(mutex); put_pool: zswap_pool_put(entry->pool); freepage: @@ -1902,10 +1907,10 @@ static int zswap_setup(void) } ret = cpuhp_setup_state(CPUHP_MM_ZSWP_MEM_PREPARE, "mm/zswap:prepare", - zswap_dstmem_prepare, zswap_dstmem_dead); + zswap_buffer_prepare, zswap_buffer_dead); if (ret) { - pr_err("dstmem alloc failed\n"); - goto dstmem_fail; + pr_err("buffer alloc failed\n"); + goto buffer_fail; } ret = cpuhp_setup_state_multi(CPUHP_MM_ZSWP_POOL_PREPARE, @@ -1940,7 +1945,7 @@ static int zswap_setup(void) zswap_pool_destroy(pool); hp_fail: cpuhp_remove_state(CPUHP_MM_ZSWP_MEM_PREPARE); -dstmem_fail: +buffer_fail: kmem_cache_destroy(zswap_entry_cache); cache_fail: /* if built-in, we aren't unloaded on failure; don't allow use */