From patchwork Wed Dec 6 09:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9F68C10DC1 for ; Wed, 6 Dec 2023 09:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 311656B00A0; Wed, 6 Dec 2023 04:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C1EB6B00A1; Wed, 6 Dec 2023 04:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 115896B00A2; Wed, 6 Dec 2023 04:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 03C966B00A0 for ; Wed, 6 Dec 2023 04:46:37 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D13FE4017F for ; Wed, 6 Dec 2023 09:46:36 +0000 (UTC) X-FDA: 81535913592.13.3F5BEAB Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf28.hostedemail.com (Postfix) with ESMTP id E3B57C001D for ; Wed, 6 Dec 2023 09:46:34 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701855995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jkKMup39apxbLnzkAyTXUI4I35+uiNZ5r/9DdtLwYPQ=; b=DcsEUihx/zXQgJfnJzvb9779biNyBZ9tTvJEHWJr13CC5BPbxde5e4pCM9LtohHL7aB6ch SUwEdYnLbSQdalG+492TwqvFt5/GVvREiBDwHDOr5lE5xCjajTOoxF333QTlNKPp29dL39 wT+4cm4A20XKbHIFic1D093kKG4wNuk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701855995; a=rsa-sha256; cv=none; b=TMuGHhzbC3QK+9/Zo1K4XMJdBtWjmgUTS2SvjPguqLaPJLOSDIgo96HCh4DQlixDQ9Q4ZZ s8Mr2x2hCtHs7hCXPQVL0l6jM+17UOy6Ig/0dypA+JPlRgSxi9LP/v0Dzr/HcWghNi0YpY j59Qku56nQox/Ia/OCNVBZXteKC+aLU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:24 +0000 Subject: [PATCH 1/7] mm/zswap: make sure each swapfile always have zswap rb-tree MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-1-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=3698; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=PZN0xLXpa6FJXo43SkR7sRqIVKPQ5jd6uVBrSVx9EAE=; b=/IE8MsA85GAZ6MZCk9fTWWd8zckSLavOzPjJFd5hfxDZQxaW88TjMxNyw9krvi0fXcdtJlBTb 6fS45gqAMpsAiMJo+PpOcT2QfWAipRf/YXZ1gMVtSGgZ7M7zveNY0XP X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: E3B57C001D X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Stat-Signature: i6tj5omsyi7d6o9rotxyneft1bi9bewo X-Rspamd-Server: rspam01 X-Rspam: Yes X-HE-Tag: 1701855994-729064 X-HE-Meta: U2FsdGVkX1/D2uPe4x0iRkAK6c+NfFvF13DFKl8FRVRRzQxjSP8K147NW1SSkvOV+y21IPi1RKZHslfoQAsqJq/mWRjDG/4Sr8UdQoRm7A7hV1riyygVuc5YdtbDtv8bupqbzhRb4m8PhikpOniSlx9rO2sDjpx2nY4ayrdsCW6UGl5c/RybNXJrvDCL0W87g1l8tKlVviWFc4Zyg2CPIXvmandcfE9FEFpBM6EMcxECDSo23XdBJAQVwHrDdY5AL+2x/xMWWzh2Oi/6IrUDHNH8sPowg8cjzUfVbfOdPMA2rp+DmZ/6Ca4m7YS9UcCa5udh2OHKsZSexBKQL/s8bGXAbWR94cvjSeYd1pEv3fXTkLPyEVgoaYI1+kuWsdqy4trBiWWpQyRiNNJxZkdS2nWF6pAV9GDpf5/TrsmrPUVB5qZXQ9ucgSo84lz/233VYRwo85vyqFNyKa6K7YAYt/qIy0A1HeqAA56SHbMSWqeQrI56vAEM4RI56AsnJDoe9R3zC7Bm+0HfP10yR4ZIBt1VtX7BfSdFGDC9bh0Wjq2gxneM5VgyyAU12vzLUQEa42PazP9/9ANgdhlt0iDSUtWySwt04ds9k0gunl44ntDUTds61HnJHe1O0HAYatQIF5KBZQkwVRY8mEdvZgD+kWckIy916OH42/UfDTKACkd0pr6JTMkZpM279tjxoLKeNjOrpKOL0pSqKbR2W/WzbWb/Gcva9hk5MOdfYvz87yrq/cb5hsvvaCcUmwp+P8kjFW1IoReFVZ6aKfD94NDil/v77ER9w96fZtL+JAGgol1TFHx98z/xWxyV3pgmGqPBhP+zkmdMB/d1yBnPzs6pk1w0F/f5KMf3r3JWtqaThh5NXv/nSIW2N43HBPxfNeNQUSTxFkd0vTVNYcGYaovkDY5WrKO5WFcDOhdFK86m/wsAIiiczQR6S7pmUH+ssXSdzK0TcMy5kQ2z6kw4VhT qA8ZdvcK oa7jymU1X5y6zc+oYT/ceyoDugw2XEiC3hW0iKXkudNlCgSijgdbdM6ULW4e/KfoKLBoHCyXMgvlvgd11nHt1t8DKt5Gf5EDAcunKJ++6K7UzeuoKLACl+P8KiWAgJGDe2dUqHVa/RuOs0kuiAxlOQ8Y2Z+lQ451exuPmud4zy4JMMGATahvmwOGTswzoPh6UlPX38RSX7QNl46K2h8d2AGKB5YWCtfqgB0WC5P6rBUIqM+gfLKklJMY9aq2F6O2/UOtTgjMkitFpiWLWvZtz+AUDwux/9gOTXiurilkbczK6ATv5TGsT/k2pSnWsTsPaoQdigGpneyuNRIM4L7dPxFvcLr7voggHxhNY8MBfWe21YwWTBBvlOmoWag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Not all zswap interfaces can handle the absence of the zswap rb-tree, actually only zswap_store() has handled it for now. To make things simple, we make sure each swapfile always have the zswap rb-tree prepared before being enabled and used. The preparation is unlikely to fail in practice, this patch just make it explicit. Signed-off-by: Chengming Zhou --- include/linux/zswap.h | 4 ++-- mm/swapfile.c | 10 +++++++--- mm/zswap.c | 7 ++++--- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 08c240e16a01..7cccc02cb9e9 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -30,7 +30,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); void zswap_invalidate(int type, pgoff_t offset); -void zswap_swapon(int type); +int zswap_swapon(int type); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); void zswap_lruvec_state_init(struct lruvec *lruvec); @@ -50,7 +50,7 @@ static inline bool zswap_load(struct folio *folio) } static inline void zswap_invalidate(int type, pgoff_t offset) {} -static inline void zswap_swapon(int type) {} +static inline int zswap_swapon(int type) {} static inline void zswap_swapoff(int type) {} static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {} static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {} diff --git a/mm/swapfile.c b/mm/swapfile.c index 8be70912e298..939e7590feda 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2346,8 +2346,6 @@ static void enable_swap_info(struct swap_info_struct *p, int prio, unsigned char *swap_map, struct swap_cluster_info *cluster_info) { - zswap_swapon(p->type); - spin_lock(&swap_lock); spin_lock(&p->lock); setup_swap_info(p, prio, swap_map, cluster_info); @@ -3165,6 +3163,10 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (error) goto bad_swap_unlock_inode; + error = zswap_swapon(p->type); + if (error) + goto free_swap_address_space; + /* * Flush any pending IO and dirty mappings before we start using this * swap device. @@ -3173,7 +3175,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) error = inode_drain_writes(inode); if (error) { inode->i_flags &= ~S_SWAPFILE; - goto free_swap_address_space; + goto free_swap_zswap; } mutex_lock(&swapon_mutex); @@ -3197,6 +3199,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) error = 0; goto out; +free_swap_zswap: + zswap_swapoff(p->type); free_swap_address_space: exit_swap_address_space(p->type); bad_swap_unlock_inode: diff --git a/mm/zswap.c b/mm/zswap.c index 0f086ffd7b6a..5e2b8d5ee33b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1560,7 +1560,7 @@ bool zswap_store(struct folio *folio) if (folio_test_large(folio)) return false; - if (!zswap_enabled || !tree) + if (!zswap_enabled) return false; /* @@ -1850,19 +1850,20 @@ void zswap_invalidate(int type, pgoff_t offset) spin_unlock(&tree->lock); } -void zswap_swapon(int type) +int zswap_swapon(int type) { struct zswap_tree *tree; tree = kzalloc(sizeof(*tree), GFP_KERNEL); if (!tree) { pr_err("alloc failed, zswap disabled for swap type %d\n", type); - return; + return -ENOMEM; } tree->rbroot = RB_ROOT; spin_lock_init(&tree->lock); zswap_trees[type] = tree; + return 0; } void zswap_swapoff(int type) From patchwork Wed Dec 6 09:46:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BBC7C4167B for ; Wed, 6 Dec 2023 09:46:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5ACB46B00A1; Wed, 6 Dec 2023 04:46:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 55BD56B00A2; Wed, 6 Dec 2023 04:46:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FBDC6B00A3; Wed, 6 Dec 2023 04:46:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 31D9C6B00A1 for ; Wed, 6 Dec 2023 04:46:39 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 00B381C030D for ; Wed, 6 Dec 2023 09:46:38 +0000 (UTC) X-FDA: 81535913718.20.B065816 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf10.hostedemail.com (Postfix) with ESMTP id 20575C001F for ; Wed, 6 Dec 2023 09:46:36 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701855997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xjCI9Y+WpwNUiPvyb+PfGaW2yDIpazi6J+7q5rujkjY=; b=d6dFfhdxJD/XCCq9tr1aHwXtd2Oxb90hDXN8LsDZxFofjSJsHwyVMI8LeeYbvggBQP2MB5 O1V3iFVJ0gpSf2+Ee2Kt8aKMKrtwsppQ62AK65VErn0TsVgzza3B/E/R8nVclnGYDd5ylR EA93V/QgyCkLWdtWB9Np+kzdrHmvOeY= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701855997; a=rsa-sha256; cv=none; b=1t1SLyPnGz12wTd5Hej95YVA5rcFZMaY1A+sFkUojUhCVXoyJNRm5SqcjvBZhZMKcxHxrG cXXNI5ZzYQvhhTdLfL5dOWc0PKFun4TJEyv62f+5DnmmOhwrgoRCsJ+UhHuYn52CU9LRA/ Sd0MI0+sHGkBY/A4F0ynH6QDfar9XJY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:25 +0000 Subject: [PATCH 2/7] mm/zswap: split zswap rb-tree MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-2-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=6878; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=91S6pF6My/bUROdLpQALha8Vf8LlqYuhabIDQmuzuWI=; b=wPMG7SuyQz02Gmf6/zoPkNXypnlp9pzAs3LaGP2c2Co5uxHZeHfDwL1yFMzUvPUEcTV7ChSyQ ajsHFI83VQLCQGcg29JkEtY4nc6utk5FaXQG6gU7Y5SrzXyxOn7RFyY X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 20575C001F X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Stat-Signature: tsigm7yazni5rrz574fs9ciit46bxfnr X-Rspamd-Server: rspam01 X-Rspam: Yes X-HE-Tag: 1701855996-73975 X-HE-Meta: U2FsdGVkX19esoJjFzvysCvMqlWyhrMzEsli0Q812z2/k2fJ5T8yzGCGLw/YPv3AJJko+7WORwztozRvamvTbqf4YAAKeVFcLIhp2W6tTeAcs+4/toImxHVDfpbzh6N7tsoudamyTas51bGMayblkvNy1EZoTvf+r1uCg546RrBj3S6qPWTyEJADCfkJAwLoJNY4OsONKQsnziS97sQy5Y7yvgoASSHs7tNPNGyaKbK+saPIHvvXiz8HkAzzT+Cclpj+gTCWJmuoM79Nb6aV5yQz15zVb2+gS1AgnUY5Tumej6kgpjHlHb8HoXQAAUuBJc7sxTRwk2AzxenZ1tvME0e7BeY5LjkVyGDgTxZIQQHqVmhMJ73Xh8Y4/J2CPvnRvom2boRu2ScUHg5PsP8RfbfkQ2mO9ucyLvEp0gve+9GDcik7Gn2p54CbD3ZbTvCzeTwk+7WsEF1lWfZGDOccPc2ZRiV4jl+pkTSwcKnP3H8ZUTsKYAgfTQpWOc6jDRdmhv4hhWWglapz/Q2w+FBWpS6rhdUUFLeRhAXHb6SA2zzSPGEuI4jbjFZ1C6j2j6pKyht/sJpzbWg5XwD2AiHFSb2oaEui1I6wji5yjlW/pqXqED577r1WUKsmh+xbIswaGL2SL1xblx8Ywgx1MXG+xbWC4meatVvJ+Btp/UY31woWLcCoDQebGLl0aNPHBpbXa92P2+iH7dafuDz9WHwwrjCvyzqxouBL6WkUeQRD/Gq5oNyd43IDedXdRNqbsHp4Hrcb0JyMlc+fxeay0V0uF0X+dOednRUoVHmC5bWTcWAH5Ei7XADSgZQtSIySU9Q9V9hBD8qkwdzpA52v/ID0AFAHWTL03aw9r9xxuPYOrWW/gnfpNgkHKaqrbwE9Kxfk2X19B3dQsLD/LoqkgwSIenSwevac2Zldi96AIuhKML2JHoB73TG6TjZ7QYc6LIG6ceOHWbfBGJ3LVjRWjlo jMrZFhE1 RFW8nO7poZjbsdGZTVCZgN6p+Hya0iKHtGzG04mGvpP44OOiPCNs6oIun6vAnX5CsvMtgkQao9Weg8u4ayoQnfWKMb43tjayyeB3RrS1688bnD6yfn90bAgGkWwP79x961xwSD89D/hGof5ah/BNSEHr6Gco735PqNR2Yo/KkbcpdjpsFcsH6r52E8NwjOuevTl5jBJ22UU0dvkwm9c0G1vj41G4jPN+JH/FTRSNqQxcptpoWIP0JN8qYweiTVkS57OK72se/bguvKdYv8HmX0lHu6b6mr5jWx9gV1erL000AnsgRCsVHAQ6zCfAkHdd65h+1bdkN5cH0UV0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Each swapfile has one rb-tree to search the mapping of swp_entry_t to zswap_entry, that use a spinlock to protect, which can cause heavy lock contention if multiple tasks zswap_store/load concurrently. Optimize the scalability problem by splitting the zswap rb-tree into multiple rb-trees, each corresponds to SWAP_ADDRESS_SPACE_PAGES (64M), just like we did in the swap cache address_space splitting. Signed-off-by: Chengming Zhou --- include/linux/zswap.h | 4 +-- mm/swapfile.c | 2 +- mm/zswap.c | 69 ++++++++++++++++++++++++++++++++------------------- 3 files changed, 47 insertions(+), 28 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 7cccc02cb9e9..d3a8bc300b70 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -30,7 +30,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); void zswap_invalidate(int type, pgoff_t offset); -int zswap_swapon(int type); +int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); void zswap_lruvec_state_init(struct lruvec *lruvec); @@ -50,7 +50,7 @@ static inline bool zswap_load(struct folio *folio) } static inline void zswap_invalidate(int type, pgoff_t offset) {} -static inline int zswap_swapon(int type) {} +static inline int zswap_swapon(int type, unsigned long nr_pages) {} static inline void zswap_swapoff(int type) {} static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {} static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {} diff --git a/mm/swapfile.c b/mm/swapfile.c index 939e7590feda..da8367a3e076 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3163,7 +3163,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (error) goto bad_swap_unlock_inode; - error = zswap_swapon(p->type); + error = zswap_swapon(p->type, maxpages); if (error) goto free_swap_address_space; diff --git a/mm/zswap.c b/mm/zswap.c index 5e2b8d5ee33b..a6b4859a0164 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -234,6 +234,7 @@ struct zswap_tree { }; static struct zswap_tree *zswap_trees[MAX_SWAPFILES]; +static unsigned int nr_zswap_trees[MAX_SWAPFILES]; /* RCU-protected iteration */ static LIST_HEAD(zswap_pools); @@ -260,6 +261,10 @@ static bool zswap_has_pool; * helpers and fwd declarations **********************************/ +#define swap_zswap_tree(entry) \ + (&zswap_trees[swp_type(entry)][swp_offset(entry) \ + >> SWAP_ADDRESS_SPACE_SHIFT]) + #define zswap_pool_debug(msg, p) \ pr_debug("%s pool %s/%s\n", msg, (p)->tfm_name, \ zpool_get_type((p)->zpools[0])) @@ -885,7 +890,7 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o * until the entry is verified to still be alive in the tree. */ swpoffset = swp_offset(entry->swpentry); - tree = zswap_trees[swp_type(entry->swpentry)]; + tree = swap_zswap_tree(entry->swpentry); list_lru_isolate(l, item); /* * It's safe to drop the lock here because we return either @@ -1535,10 +1540,9 @@ static void zswap_fill_page(void *ptr, unsigned long value) bool zswap_store(struct folio *folio) { swp_entry_t swp = folio->swap; - int type = swp_type(swp); pgoff_t offset = swp_offset(swp); struct page *page = &folio->page; - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry, *dupentry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; @@ -1610,7 +1614,7 @@ bool zswap_store(struct folio *folio) src = kmap_local_page(page); if (zswap_is_page_same_filled(src, &value)) { kunmap_local(src); - entry->swpentry = swp_entry(type, offset); + entry->swpentry = swp; entry->length = 0; entry->value = value; atomic_inc(&zswap_same_filled_pages); @@ -1688,7 +1692,7 @@ bool zswap_store(struct folio *folio) mutex_unlock(acomp_ctx->mutex); /* populate entry */ - entry->swpentry = swp_entry(type, offset); + entry->swpentry = swp; entry->handle = handle; entry->length = dlen; @@ -1748,10 +1752,9 @@ bool zswap_store(struct folio *folio) bool zswap_load(struct folio *folio) { swp_entry_t swp = folio->swap; - int type = swp_type(swp); pgoff_t offset = swp_offset(swp); struct page *page = &folio->page; - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; @@ -1835,7 +1838,7 @@ bool zswap_load(struct folio *folio) void zswap_invalidate(int type, pgoff_t offset) { - struct zswap_tree *tree = zswap_trees[type]; + struct zswap_tree *tree = swap_zswap_tree(swp_entry(type, offset)); struct zswap_entry *entry; /* find */ @@ -1850,37 +1853,53 @@ void zswap_invalidate(int type, pgoff_t offset) spin_unlock(&tree->lock); } -int zswap_swapon(int type) +int zswap_swapon(int type, unsigned long nr_pages) { - struct zswap_tree *tree; + struct zswap_tree *trees, *tree; + unsigned int nr, i; - tree = kzalloc(sizeof(*tree), GFP_KERNEL); - if (!tree) { + nr = DIV_ROUND_UP(nr_pages, SWAP_ADDRESS_SPACE_PAGES); + trees = kvcalloc(nr, sizeof(*tree), GFP_KERNEL); + if (!trees) { pr_err("alloc failed, zswap disabled for swap type %d\n", type); return -ENOMEM; } - tree->rbroot = RB_ROOT; - spin_lock_init(&tree->lock); - zswap_trees[type] = tree; + for (i = 0; i < nr; i++) { + tree = trees + i; + tree->rbroot = RB_ROOT; + spin_lock_init(&tree->lock); + } + + nr_zswap_trees[type] = nr; + zswap_trees[type] = trees; return 0; } void zswap_swapoff(int type) { - struct zswap_tree *tree = zswap_trees[type]; - struct zswap_entry *entry, *n; + struct zswap_tree *trees = zswap_trees[type]; + unsigned int i; - if (!tree) + if (!trees) return; - /* walk the tree and free everything */ - spin_lock(&tree->lock); - rbtree_postorder_for_each_entry_safe(entry, n, &tree->rbroot, rbnode) - zswap_free_entry(entry); - tree->rbroot = RB_ROOT; - spin_unlock(&tree->lock); - kfree(tree); + for (i = 0; i < nr_zswap_trees[type]; i++) { + struct zswap_tree *tree = trees + i; + struct zswap_entry *entry, *n; + + /* walk the tree and free everything */ + spin_lock(&tree->lock); + rbtree_postorder_for_each_entry_safe(entry, n, + &tree->rbroot, + rbnode) + zswap_free_entry(entry); + tree->rbroot = RB_ROOT; + spin_unlock(&tree->lock); + } + + kvfree(trees); + nr_zswap_trees[type] = 0; zswap_trees[type] = NULL; } From patchwork Wed Dec 6 09:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A6BCC10F04 for ; Wed, 6 Dec 2023 09:46:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E6C06B00A2; Wed, 6 Dec 2023 04:46:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4972F6B00A3; Wed, 6 Dec 2023 04:46:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 386616B00A4; Wed, 6 Dec 2023 04:46:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2ADBD6B00A2 for ; Wed, 6 Dec 2023 04:46:41 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0562E1A012F for ; Wed, 6 Dec 2023 09:46:41 +0000 (UTC) X-FDA: 81535913802.02.B730B88 Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 232D7180023 for ; Wed, 6 Dec 2023 09:46:38 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701855999; a=rsa-sha256; cv=none; b=fvTR9Lj8Qea/ls4L+QtDlsPFk3xK0L+uwAUOA4coQPDS1NLlmWjg/DQI1iM2Rb6KPN/TL6 9iImcrXHvUNkWKHMlUGwKJU5AB2UqWVkM8qYUGNrbzCDbtY+HFfsRWPGHlS6ZzasQpSyeF AldHLE0xmi7lIgc4CxqAGvkkvQgGfVk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf24.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.172 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701855999; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vGUImAAKdFNwNoK42Ha+6s53PcbZS8gtKpqCNEPV03Q=; b=ITUgXk9dfnDZ2DYZMwBygZUMsYB2VJMDX/NVM/90UcT3r+JKpiykcDuEIlkn07GW+8ruFY kj97F68+/LBAbyYVhxjyT2h6lfWXyRCP+N1aP70Mu37mj9FEO36fQU6dT+VYq0pqcMJLFn +pNTBLuVVH76J7F+m4IU1GAZEirUPYU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:26 +0000 Subject: [PATCH 3/7] mm/zswap: reuse dstmem when decompress MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-3-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=2376; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=fpCgC2uqFVZCuogrKgOI9EeZNXHQYwaI+M0J57wRCA4=; b=wudSyMvTqyzG7OZuJjTkGo+D9zCpS8Tofe/rODy3CP0YFRdLMSz4sWGiyGOubX4khTnGFFSyX Zm343RFwsfDDmmErMRr+FCJT56Wg4VhAosu9lLVrHVfRY+DTTGtIZ9b X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 232D7180023 X-Stat-Signature: yfzzrkwnma1ijfrwr3tyb1gacpdfx9bn X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1701855998-982497 X-HE-Meta: U2FsdGVkX19d0HTExebnmayb0VUmArbbW4j9sgqc1u9MJ03Rx6k2tPjRnXWfj19DBcdGdl3uOSBSIkTwEclN6ICoA6K8AiOyn1kNSglZaOkvWC78oHMozsQG3MKD86924Agwrrh/HfyonSdqzRCzlX4efZ6TmeQifInOy7p9y73fwiqp0Vsx9rFxvCehqoZWdSTmLa0ElqAn2fQLbj8o25S6+OA9lvFAsqUG6MUhEf4Txg4mFpIUHjSJF9Chxorj7D5jGVztNbU6dTEf5bG6DKxory0EYK/oqcUfint6aO+X6G+a9Wl+6qbWr14Xcjj3SfqoB38/J5RnNVU/gSIKnu652SHpvvYwsvq3jzzReurEzQKmQNpJX9QP9QOW6tQsbff2fgAUPRodT1BRNOuQ7xhZ3ClRH+Cs1QsSdqLeCusHHAJl093KiIl+9Y8H+o0DqEOgmla8k8K6Nm7zd3CxUCiZP2MCMdXJsgHWAkxq3YBWZLhMorwTN1H31m9Dq3eqbStqFktC7ShmM86/1uXIB4c8HztQCAsiJyalaFSfJqRGsJau7NWwW9IGQgvqkLz4OyWjheEkrmnckDVw4U3b4kWCbY9yFaczy1jjBEOF2BzBMaNMhElJPnpq9w6Xha7jU/D9rXDNHkj78bUNXT6FCBVqreobaCh7fscmhMWH/3LjTFkVT4cEgiI+Xux+grLivehxP5nxp/2g8Tj3yrPnqDOA6Iqexo8SjIuuiBAxyubZrzphH7dYKWlVvSjUnArLnw0b2yJNbLCLD/Rm92omVCg48BIpEgYIqvF7aykaB2sxHuJJN4IEbwYHyyn4X34+wNQViJuMjmig7ClnsptHJFccDSFGWClY6e1G5XDjfRQDaF617g05mRhlNpmNgap2NigLjkOEVmiAUjw8qwdvHI1WBLPz3erJHy2QnMvVDPh9t6Bkc8fQaW/q9MVJDFvhDAsA4ZtmKBHWaGPPYur 0MwMfD9z rNDQdg1FPpBiJvAJkzDhO0hay6zPjQ0YgYimLPax03fVdM1Bek8Y4sZotL/Aynr8LXGhwp6mcd9GEfzcmOjeyzBYbEBr24+GHK0BLR2nHTd02YN3YBAjWeN9ry3S9cwcLHV0v0u/h9E5qS8K1qur+RgURaYoE8Wg6onrjiJIiieE0HgteJ/HhbhO5S3N4tGsLsbNJ8XsfnPeJpxbntKCwnXN3kM79f7XzMUzqPlYIaZZ2wZHjI4sMgbkediUGA76AgJQi76Wp3bysXUqJNCkGdepfSSjJX/A6jKjGkqLKWjvNEiioyezRcl53bt9KCtbJ+Y1SE5NgPGaVYuM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the !zpool_can_sleep_mapped() case such as zsmalloc, we need to first copy the entry->handle memory to a temporary memory, which is allocated using kmalloc. Obviously we can reuse the per-compressor dstmem to avoid allocating every time, since it's percpu-compressor and protected in mutex. Signed-off-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index a6b4859a0164..d93a7b58b5af 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1758,9 +1758,9 @@ bool zswap_load(struct folio *folio) struct zswap_entry *entry; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src, *dst, *tmp; + unsigned int dlen = PAGE_SIZE; + u8 *src, *dst; struct zpool *zpool; - unsigned int dlen; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1782,27 +1782,18 @@ bool zswap_load(struct folio *folio) goto stats; } - zpool = zswap_find_zpool(entry); - if (!zpool_can_sleep_mapped(zpool)) { - tmp = kmalloc(entry->length, GFP_KERNEL); - if (!tmp) { - ret = false; - goto freeentry; - } - } - /* decompress */ - dlen = PAGE_SIZE; - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + zpool = zswap_find_zpool(entry); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); if (!zpool_can_sleep_mapped(zpool)) { - memcpy(tmp, src, entry->length); - src = tmp; + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; zpool_unmap_handle(zpool, entry->handle); } - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); sg_set_page(&output, page, PAGE_SIZE, 0); @@ -1813,15 +1804,13 @@ bool zswap_load(struct folio *folio) if (zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); - else - kfree(tmp); ret = true; stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); -freeentry: + spin_lock(&tree->lock); if (ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); From patchwork Wed Dec 6 09:46:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A7A0C4167B for ; Wed, 6 Dec 2023 09:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 954A86B00A3; Wed, 6 Dec 2023 04:46:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 866C66B00A4; Wed, 6 Dec 2023 04:46:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 707C66B00A5; Wed, 6 Dec 2023 04:46:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5B8E46B00A3 for ; Wed, 6 Dec 2023 04:46:43 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3528812015D for ; Wed, 6 Dec 2023 09:46:43 +0000 (UTC) X-FDA: 81535913886.15.B93319F Received: from out-189.mta1.migadu.com (out-189.mta1.migadu.com [95.215.58.189]) by imf29.hostedemail.com (Postfix) with ESMTP id 51FC4120027 for ; Wed, 6 Dec 2023 09:46:41 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701856001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d6GiiHeHVcS7kukNT2gh1iXVRWGkIYl7tyaTuuFP/Ak=; b=eSCE/I0GjhJx9eMfRNPAh0xBjsLCMEflVXwmOKBJWF4oChHQ14HJjv0uoV11VLr2SyHjcR y3+UDqvvPNUgyAKPX6+HBWjVAswEhJnO+doxSeOvDa+vGL7Ri/rjQTZFHGgcg2n53fAB6v CCN31/9LvDafz9aKYJuxfayZsqQguCk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.189 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701856001; a=rsa-sha256; cv=none; b=YOpjFQWdfvHTk9K9PWU8LFihuP5nFIsB3gTh4oKNqzPa0AmcK0JvsvwXA7qafV5eW47jD2 T+J9dK+nyaigBvkmzF+UBG4RFwk9qm/6N3uwckxwXsGVttVUl0MweG2NvXOfdpEUZk9WaV fhblw0v8u7tBpO/IxAoBWHzDfn16Tj0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:27 +0000 Subject: [PATCH 4/7] mm/zswap: change dstmem size to one page MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-4-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=1265; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=HqwNJM8tE/N/sls10yfGIW7SQYFQNs7CH/U23vKZJJA=; b=vLSavMUZY2NbU7Ht3zXNimzw+b1It8sOxPqmY7gDO7tM7+foam8clg5CtKs4bOcNaCYz3xPso WsTwNYwXlA9BQKJg3zkaJ53yg9R0yBfDoIjVp0o2j0FWuC1shqBwv4r X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 51FC4120027 X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: 7mgio54ywh6ipqsc8zt4osca1ds71ys9 X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1701856001-54096 X-HE-Meta: U2FsdGVkX18eLkuBYMlu51rODMDBzkAFRinfN/eqMtkeBd0UrXAO52WkW162L8Zu6MF9YdWUDoms7HdlZb7xrmYe3ycALAZYT9QZ79FJMSkt/FoTiHqV3G3/qOpu0jWeaScT4+9k9istNbZKj65Egr0sN2VzqfqE0FBlUVitUoznIh3VySvVOXKA2PloiM49kPQZ44/LX82P/zsgJfGplmCJmg4mL+YrbnOBUdfHPO1iTVTa5i69Qi6p/rrlARt+RAZGc2kcf6Xh4AI9VRJU+gUALDP35gLyv6xItNdW4d6ZjiGpC1OhUYs9fzusIXZ2i3fPbBQ4C5H/exSbOgBTHKUeI6Bm6s53z3JmQi7NeoEtPUFq7ss3+CS9wRYHlxCT2JrqaVWC2XaS4jgIUYo1/LITEXuao4tNToqKEMZC+0SNTBcJVLxVwwWDVZG1NthfRU13qMHYnJ1bRtnIlxW9oiXrsEnwdJ3CgUA+NbrN0qKP2zx028FKwBJCE8IKYA844sDg7g94gH1jbVomwBBR549nYlZf/wNrZvyGfuNZLzCygm0Ev9+c0NLaCqRzL2Af3dpq4odWKMywacV+yqaskCqyIu6VvvwJhZXn3KV9StT9bb6kCgP2+7IRXat3k+JsgG4cI/Fs5WX83Bzg0Vo6OP2+wRLa21D9TEhN8EkFVtx5UY1C04c0eUrGiqh9NmA6JMhy0KA7RmQG5HwhLJMp0YhH5+uuxw8kgj8spVUL3A3vSm+Ok4n9XseOlQcALQPbC8RZAkyoObvxdRrtDfyUUkwhbkVDqIsXdcY8mcxhYCFd9sSvb5iyeKWoY/rOiluYlSyQGf/yk1ZLEhOW5tEbuhzdMaAGFRg6rFXRN3tGbGghqErjHtBtbZqOo7N8d10pG6qXRQ7/c6MmD7yf75VgKSuSPxr2KJdHTF8XjupOSYKXbRQFuVPkUvJpBGgLak0w1bP4oWn4e5e5oGsdnqI kUG9HF21 lPuCNdo1y/+dFCc2vdGHFzs83rPEvAzcPHs4UwR2al8brmkwPUFfm4bATPuT+wP/714txpxnyK+NZuuFlWSJcWmuQY5DGZuMwZfsTD2pChtEyzLI1Jejkh+jMkSGIhsNRjVSy/HjgE6ioIDT6pLcKqDxBb8VIkZ+7QO1dDC5pe05IzLlBmbY/C78eBTiPGnNd0njdsEXuDl6BvoOvF9uBHf3qymkJ/p/TuJGVWRZ1ioNxl1bjabJxWODn+cK4N5FvIX8XUTantFogkmzQAx4C7px8OJ23MHUr+cgote3w2bPuLNOsUDM39ybdnH8+hPKLWjKlkrUWuggs4Q4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Maybe I missed something, but the dstmem size of 2 * PAGE_SIZE is very confusing, since we only need at most one page when compress, and the "dlen" is also PAGE_SIZE in acomp_request_set_params(). So change it to one page, and fix the comments. Signed-off-by: Chengming Zhou --- mm/zswap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index d93a7b58b5af..999671dcb469 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -699,7 +699,7 @@ static int zswap_dstmem_prepare(unsigned int cpu) struct mutex *mutex; u8 *dst; - dst = kmalloc_node(PAGE_SIZE * 2, GFP_KERNEL, cpu_to_node(cpu)); + dst = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); if (!dst) return -ENOMEM; @@ -1649,8 +1649,7 @@ bool zswap_store(struct folio *folio) sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); - /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ - sg_init_one(&output, dst, PAGE_SIZE * 2); + sg_init_one(&output, dst, PAGE_SIZE); acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, From patchwork Wed Dec 6 09:46:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C6A8C10F07 for ; Wed, 6 Dec 2023 09:46:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3F9C6B00A4; Wed, 6 Dec 2023 04:46:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 904956B00A5; Wed, 6 Dec 2023 04:46:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7571B6B00A6; Wed, 6 Dec 2023 04:46:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5F1536B00A4 for ; Wed, 6 Dec 2023 04:46:45 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 37AEBC0131 for ; Wed, 6 Dec 2023 09:46:45 +0000 (UTC) X-FDA: 81535913970.21.69509F5 Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) by imf25.hostedemail.com (Postfix) with ESMTP id 4C876A0020 for ; Wed, 6 Dec 2023 09:46:43 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701856003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rtbcaV9uZyvJ49g+RnmIBpB2G5m2DALCMWgXlaPRAwo=; b=KJfHRThBqUDIzxJp7JsRCfCSKEo6/bEUPCDn7cjqN8EQZKxCz9m8t58tIBhkIe/cXT+urU yPlkMh5LdCjkWrmQgJxxctakQcyNUw/vR2GIQt6TJwuC+aVPdbXUeRru1ra3eix01iS96u rWlZgVUx9QVJYdheV3v9qrK/8/L5M40= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701856003; a=rsa-sha256; cv=none; b=h6oCHueXdalPNLOVX74Yz+wwy72easkex7aJzXxlTVjLxzzkGo2mANG0d8bL+RiDHQ/NrK fn6MZuYkNQ+4gALYlDVVvkaNNs2gtVnMpRC9J1khvSWcymMaQqzKGHfeZ1J/7wyKor+qbl JDoFi6FjjJ/fxSSKA3HaJUHy88LcB2o= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.178 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:28 +0000 Subject: [PATCH 5/7] mm/zswap: refactor out __zswap_load() MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-5-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=5290; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=ju3QBW8cgbkzkTilyJdKpdv6IgD35PF5PphWVt2TBXg=; b=sN2QzBgyvVSvfWoqMPvo+BA+Z/wX8hCyJ3NhGmP4yMNcUemm3UiYwXWjJiunDYw+3nKWn0X+s um4PqJ1hXePCeO2kEV8Gev6JvZihkII6L9Ti8GacRbiWlvx29DlAv96 X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 4C876A0020 X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Stat-Signature: j15nsiszx5pyj7i93d4k8u6x9xujhjyi X-Rspamd-Server: rspam03 X-Rspam: Yes X-HE-Tag: 1701856003-765298 X-HE-Meta: U2FsdGVkX1+ZdLTWGpCa2kw3ie9ONqm+pWDQt1nt6crf+fN3XkMR8U4/P0a25+1gmydvjOSDz4Do+b4QLVFVwuO8Oz9/UrgR5UsIDKw/Ie3rz6OA3YG+IuKWXZATraXNjPJ+xKupzzYq0CmsRFdNXGmAA47U2VNJsGrOkxTluHJKVda6aKLXh7hYYhUWKnBd/W2eFs3i5rGSCknHKpXlgF/g29q+wew2wTJ5+kHbhFqqTQpeaVKdo+ykT9jsnQ0r/pQIfV9fFLHBp9vpPyvHiK2ditoRaUXg5oCRr7kpBQRojNvt0kBIV7o18CIURQWWnQfBqWxlvmN0J0fr3XA4r/4orzp4jVV1sAOI5urjIURBHDkQvPsrSqAOJZHgoAqVi1rqI9BlftEBKACOwJLKdGAa60Fag+CPgjBYvnrWhXKqr1tEtbG5LxpKVpUlH5BAqaCxXgkrffpDMHpac+nifHC+L6+xKh/PBoEoKqRk/T2sStzNNqDjp6OjgPbwklC2G9ZKGc64avJUKWQ4KPhJ2lzup9HN5PH2HN/dShuiL0lwlxypMe0G6XEtp6Y7r1bkYfxxwgyaPAHCf0N54OwSJMy7rInZ2jHjTgExHpCH5+H/5U9QNATbdXeuAG1rlcCmhMycmhiJsXrFjHymcsEj6A35BKTg5X0Hj+jRtfsG7KxUWPh26IhIh4fFvfQlfmpqWAfl6v/HX77l8SUqBOkRVwICpcLXafDIAiMoQQu1hfZwk3oMGiwG4Jtr9ahmM7UQqM2gBIM8IqI1LCtTW8HGk65AZMRVlmvYobwH029wRkAhKiZIc4fd/kpbUjfa0ylXRB9L8EMq5e0rEgIByGuJwHilHyZMeMcxh5zLDyepsO1Asg6ZfCVQiw4GCp0j5rdmMiW3rxrum1HhM5bqWmQQ4Tno6gZq9/r6sfVOq6fW41WXqJwUlKEjp1Ej2p/fT0WX8AWo0HOof4Lxjq4NuPI yth6C6MH pqxXk1mkykGKvb0fyrOSeSmHp94c2LScFuGLp7uYr5DTTif7Arr0bbH6czfwLx4N9859u9yWTH21SsMci+ek456aLVrUnpjFrcG23CchMf229ceWf+VbOMeCH8sqxZZGLAwsi/GcRNqVfIib7TTxixZRrC7EfCEdP9++s7nZG9diCOgUKWTV403pi9U+bdcuEXzF+H0SxpSYIZ+P0hPMDi6jqbJgTb9mwP+j9RDP1c1uooKS5FpuH/vQisBFQnUu6ai0WOVpLecySHztL+bDBM22OpDakXj9nA8u/F2MF6fY4JqTehVEGw6qNMhHa5ztpFKGAQ1OneQLhdyo+UxGxP+k0HiQbVMPnd7/oh4yZB1bF1467QhVEfDTYoQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The zswap_load() and zswap_writeback_entry() have the same part that decompress the data from zswap_entry to page, so refactor out the common part as __zswap_load(entry, page). Signed-off-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 108 ++++++++++++++++++++++--------------------------------------- 1 file changed, 39 insertions(+), 69 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 999671dcb469..667b66a3911b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1380,6 +1380,42 @@ static int zswap_enabled_param_set(const char *val, return ret; } +static void __zswap_load(struct zswap_entry *entry, struct page *page) +{ + struct scatterlist input, output; + unsigned int dlen = PAGE_SIZE; + struct crypto_acomp_ctx *acomp_ctx; + struct zpool *zpool; + u8 *src; + int ret; + + /* decompress */ + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(acomp_ctx->mutex); + + zpool = zswap_find_zpool(entry); + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(zpool)) { + memcpy(acomp_ctx->dstmem, src, entry->length); + src = acomp_ctx->dstmem; + zpool_unmap_handle(zpool, entry->handle); + } + + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); + ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + mutex_unlock(acomp_ctx->mutex); + + if (zpool_can_sleep_mapped(zpool)) + zpool_unmap_handle(zpool, entry->handle); + + BUG_ON(ret); + BUG_ON(dlen != PAGE_SIZE); +} + /********************************* * writeback code **********************************/ @@ -1401,23 +1437,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry = entry->swpentry; struct page *page; struct mempolicy *mpol; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - struct zpool *pool = zswap_find_zpool(entry); bool page_was_allocated; - u8 *src, *tmp = NULL; - unsigned int dlen; int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; - if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!tmp) - return -ENOMEM; - } - /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, @@ -1450,33 +1475,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; - - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; - zpool_unmap_handle(pool, entry->handle); - } - - mutex_lock(acomp_ctx->mutex); - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else - zpool_unmap_handle(pool, entry->handle); - - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + __zswap_load(entry, page); /* page is up to date */ SetPageUptodate(page); @@ -1496,9 +1495,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return ret; fail: - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - /* * If we get here because the page is already in swapcache, a * load may be happening concurrently. It is safe and okay to @@ -1755,11 +1751,7 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; - struct scatterlist input, output; - struct crypto_acomp_ctx *acomp_ctx; - unsigned int dlen = PAGE_SIZE; - u8 *src, *dst; - struct zpool *zpool; + u8 *dst; bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1781,29 +1773,7 @@ bool zswap_load(struct folio *folio) goto stats; } - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - mutex_lock(acomp_ctx->mutex); - - zpool = zswap_find_zpool(entry); - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { - memcpy(acomp_ctx->dstmem, src, entry->length); - src = acomp_ctx->dstmem; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - if (crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)) - WARN_ON(1); - mutex_unlock(acomp_ctx->mutex); - - if (zpool_can_sleep_mapped(zpool)) - zpool_unmap_handle(zpool, entry->handle); - + __zswap_load(entry, page); ret = true; stats: count_vm_event(ZSWPIN); From patchwork Wed Dec 6 09:46:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D8B2C4167B for ; Wed, 6 Dec 2023 09:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 937F56B0081; Wed, 6 Dec 2023 04:46:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E0846B00A6; Wed, 6 Dec 2023 04:46:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77D456B00A7; Wed, 6 Dec 2023 04:46:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 682096B0081 for ; Wed, 6 Dec 2023 04:46:47 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3FC42A064A for ; Wed, 6 Dec 2023 09:46:47 +0000 (UTC) X-FDA: 81535914054.07.4D0C66F Received: from out-175.mta1.migadu.com (out-175.mta1.migadu.com [95.215.58.175]) by imf01.hostedemail.com (Postfix) with ESMTP id 5E38C40017 for ; Wed, 6 Dec 2023 09:46:45 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701856005; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BO8VowIXT7NgKBMLiN+PSMhPCGB2s3h8IYWpMrQuxwQ=; b=Spkc75sXV2zKPagSL0UE6J0lZQ5/6jRepKYWbkSM+RFtpD2neMp0ob2ZIKxA9sZJ6mNgof C0cmtElQDcVnG9tkQGq0Z1af2H9b0F7li8JDKoU1CY4cyaS0za2tm/wFmO3wHiB08OAb4e v8zjBwCzDmmkmt7GCSyE/5CyHu8FZ+8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701856005; a=rsa-sha256; cv=none; b=drHigeKOAVTpW3fDC7DWTf7n3mpHV3wMEtPQ/G8HCNvwuTS9TQRpADVfrC6X5KGgsc8DLE nSc649zmw2+4Hpa/NJM6jnq8/yo1B/3+F6bZCXuYhA+0qSadU5gdsWXmD7r9iM67d+CEPE 9G1FvmX6W/G0IOAL1DKkw7N52R08Ag4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:29 +0000 Subject: [PATCH 6/7] mm/zswap: cleanup zswap_load() MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-6-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=1423; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=4dWwyqNx8c2zAiSbKiIuSiLwoPGaeKufk6rm1H1OjyU=; b=mZ9Iu4zkHefQk19MezNP9JCSi8K14mS0atJNAlahjaBU1aLDqTWfFZSjHbYanEfMEBZ7TUpMw gqC7DA3KfbRCEXYKYzA63WfdprZF2BjpqhLC6VPExYC46NfHkEsxMYf X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: 5E38C40017 X-Rspamd-Server: rspam12 X-Stat-Signature: 5itmjuk4bgtp7nr3aw1qh69cinmw414p X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1701856005-378208 X-HE-Meta: U2FsdGVkX1/ul6FV9Zx3zIBMIuaThUIdyQbgGBgPAvuxB9LYg4CQ155kyZkTDJhMwJ8ypKPRCMnc7fLdhaqvyzhOBZDTBE/9/qGVFRrQgdtKC5wJiv+8bO01ijqIe5xn7CaUjTptd/yP16fdY0ZbgnnnS9RR2sgvXfNweD/MSyhUSHWQ+fslzd81o1ad/FWGOsY0NjaWsaTlW8alSaWurAMDVhRNKBxvnKG64INL4gwgPd0UXrUWHcjoa88Sc+OyxeK5BV2ErLCUVlcqNfJh1YJHgYzAOk6VNT6inQk2mCAjiA7sg0bE9/ogYu/CQpZ7cHdJCtPQO3RCxJcGEZcd1HTMLbEITV69YO4Cd3k8H1UyJere8Az3549PhF/GFR6FTqsFpjMfJSptOBrBH7WKEJbI72CQ4a/9LUcZHtwSm7u+fsTwnIPq2gx9KIPNtxU6vmpsnfzy/iV8fQg2o3gFr6QRe/pKkWuq4/msrdYDdcaY7a/iQkD7VxhAVwopWE+qJ6+bAyiutsr4w6Tmd1gdk76gotKlPmjB/XEcSxdD+5hIPyQ9jCDLpM/Sw9y6tue9O8Ity++jxehyF/KN/vELM/0UMPqx4IFt1O0zIrkniYoJQjzz9n1JTwY+lvcx+DKwW1D55g1PBLWgrMM4lJX7RTWaxCzMSV4B5AWtP/lpox8Rlk5D+fk37YQDUpAyVyG/mUssBjsOIMGcirfq6uGoC8V3YYgmVm3X8uD1N7EE2U3VW4hwkt7C9cKz3ekDfRcSolBBiwcEbWw+1dNlLYMf56qF8zq3zvOm1skOTPkF6rRB0KiirX0bZ2aHF5RLRUavmLzCKSL2uWAqERfVUZfU4atso02IUsNFIc+V0OA45bm7dNUa3HGywivK6nXs9zVWtEthJCx4KqYnm0hYEVLaiCgK72aOBRmxFc8sEUxU7CLHyyzBTuFBobO7nDanhhVWifM4n9hwcVUOV+b5ieb k+ShLHuc yY+ofTTzBNK1GR6zlVBqHs7zMNLJaLAabuwAWvmda3/52sWMYmk9egn8Ofu5LR6PslLBZNnucKZQaqajYzGsrMR2PeeTqp0cb0dUBabtk9h8Svz1geLuSEbKxEKN28/k/4fYwS1lILPaUdAihIySJVnCHLvLGo0uTaunjI8leMqVc7LLQ7XK2nU/hUbrvevKr/hDnM+qheTOQ0D6a5yQiksJVm/2hsq8niHQIfwht6PHLwhOqVhAxrHyxO8q5PsrwCu9FD9n4QwW6du3JJ3v7pJG1SWhFrRFrdcPpcqsbsb9Z1Mzy5cHW4M3lMVv26LE0now6ZqmI1/P12CTMBhtFZLhH/qpFhxkXcWGwxxxN6WZDF9AKsDW/Xz1AMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the common decompress part goes to __zswap_load(), we can cleanup the zswap_load() a little. Signed-off-by: Chengming Zhou --- mm/zswap.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 667b66a3911b..50405811cd7b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1752,7 +1752,6 @@ bool zswap_load(struct folio *folio) struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; u8 *dst; - bool ret; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1769,19 +1768,16 @@ bool zswap_load(struct folio *folio) dst = kmap_local_page(page); zswap_fill_page(dst, entry->value); kunmap_local(dst); - ret = true; - goto stats; + } else { + __zswap_load(entry, page); } - __zswap_load(entry, page); - ret = true; -stats: count_vm_event(ZSWPIN); if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (ret && zswap_exclusive_loads_enabled) { + if (zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { @@ -1791,7 +1787,7 @@ bool zswap_load(struct folio *folio) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - return ret; + return true; } void zswap_invalidate(int type, pgoff_t offset) From patchwork Wed Dec 6 09:46:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13481324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53F03C10DC1 for ; Wed, 6 Dec 2023 09:46:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC29D6B009E; Wed, 6 Dec 2023 04:46:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D4BB36B00A6; Wed, 6 Dec 2023 04:46:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B01766B00A7; Wed, 6 Dec 2023 04:46:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9DCC26B009E for ; Wed, 6 Dec 2023 04:46:49 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 75B4114015E for ; Wed, 6 Dec 2023 09:46:49 +0000 (UTC) X-FDA: 81535914138.16.B85B049 Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) by imf14.hostedemail.com (Postfix) with ESMTP id 88F52100005 for ; Wed, 6 Dec 2023 09:46:47 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.187 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701856007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h8eN1DUckOMA/YVkW+LuDv2KkgdMbh5axm547Y4P2HY=; b=eWqoJSqG7Ey+TuvG1s+lC04+ICLkTcyrpOl63poPUiMfsDK/vBKL0V6xgBeEgGKSrimNBf M/ZyFdPZKyPiOl7DnrMUaYnjWNwH4v29QNHOzZ7Tjqw/JVl9s+sb79iG6RapZNqDDZeYvK QrJtIcQVImQAGc6QMXWp/dPcYh9uDy8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.187 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701856007; a=rsa-sha256; cv=none; b=n/DvSJhTDH0OR2xuA7Xfh7Dr4dsIGcYCD3b2NghchmAvwJyHqWCU/EW4lvA8id8AHQBOmf w8myrFMTT3lISG/rylmEJkaG7w0EKsEi3lQAZ/+hZHxb1MKDyfG0UqUP3vU5xyxjUCM9qu /CsHZdX9ZItBkRDvlvzkSi+RYdZHhCw= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 06 Dec 2023 09:46:30 +0000 Subject: [PATCH 7/7] mm/zswap: cleanup zswap_reclaim_entry() MIME-Version: 1.0 Message-Id: <20231206-zswap-lock-optimize-v1-7-e25b059f9c3a@bytedance.com> References: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> In-Reply-To: <20231206-zswap-lock-optimize-v1-0-e25b059f9c3a@bytedance.com> To: Vitaly Wool , Nhat Pham , Johannes Weiner , Michal Hocko , Seth Jennings , Dan Streetman , Andrew Morton , Yosry Ahmed Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou X-Developer-Signature: v=1; a=ed25519-sha256; t=1701855988; l=1963; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=WQp3TO7eGK4NEJ2KSQiW8NgcHiI75uDxqzzNLxV8SyA=; b=cX50JTACBKj/y4OSJ3fqN9XlNGNG8ePq/CbHqPflo46e6ZMG5upWYYRApku021YWMN1SgS3i9 B/xOj7S/bMGBm4R4og7jAUlPoBl15i7PcZIVp5JnUC1yxy4hhNQBj7H X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 88F52100005 X-Rspamd-Server: rspam05 X-Stat-Signature: 7gpd8z7zu1p7npxfsiyyiiffxf5uetuc X-Rspam: Yes X-HE-Tag: 1701856007-978318 X-HE-Meta: U2FsdGVkX19yK8e8Bh2J2NTK9YT8SsU0hf8p7MR5GTn47wYk8u9DM6wVBYp/zynhV3eYeqxcnpWahHsxYTLD6Xch3tKkiZpNcVOkSIYQGwCwXOlxxrWZiyoXnOxdONOXToF/oZCo7zBCEyjP3DxtSlj6vMy6h5E8UK+83d6FnZHLVZEaX/EQh74eQqj+b4lCTbQwiCuOe9jyL+EIZY7cxtZvL92N2FJLy7ag9qP+kRV9caJWdf8fLQZYHxcX4kzAQbLQiqqw8VkYOoPI3Nnm7OMQplQKHbbi2k5QAltRQ0RuJZega8P4bEJQXowMWj89mXtUM2oboODw4Ruvj00SCVdQxheZWumoIVuR+BWtvtFFH2X5vgCekQ1d3Y0kWV+AGya+4Ec9PQ8VwBiw2rvMsRBzk6EkBelg1Ao0kqPQgDWkUqReL45SBwjVGReCOnPpSERxjYaK8YJhu5GyQ81KitQD+QwiJ9Ud4rIcNwXn1wMU3fFDl1KRhM0kCNubIjwQgWf7RN2WbIBbSn68TcShoi16D/PAWwrcdp/B8nCUuDp5BQeD1jBCn+ZubIrbo59EX3XlT1aCnDYUX7GORzklwkJTLzY9JECyBDFmDroT4H45kA1oT9mE160HLS8XjW+Em6B76o5Gk8viHvvrDRM9dw1QxDMXQlc0nTB1GwQQSL2Jr9MK07KO8zswOULq/uRnuBZ5GgbJ0iT8MtWzq6IeQDK4h+wIaejeejJQFsd6QitBtVLAmqTsb+LUIRtWOx1zZ9j2053qp6TNCSzVtxumP7eu3cehLi6Z9ZtDbP0GVoomiBBdWCz2aLvvwPTykTGTZk5dsTsDbM1Js7MlYj4P95XkpPetbk+51Llatd9GMwB57o/zPxPMQtV8ddHjPht536O/QxEQKUj7ijg95T7/R6RAIsIgctBvnT7O1Ts5msjFMLjYHY/yMFVJM2xm85l5YnjNJrzCjKrzjS8Z4Tn pGNmgXc7 N2dv3vMZ/PSsAirTyQDzIFeCz4p1RFCXJa3eMgqGvW7lzSkfSKm3AawjKSWZY5VLiVBsJ/LOBj251fVyH2kQ208M7Mil5j/WaImBoq3G5EaOk6sfFBUQIzYY+MivrnT/Wj+FslcOu+pIgN7ieD6mKsTQUzhuBzyHQLrpI1tBxMwIItI0vEzntlod9evZFKs+33hEyQWYSDTxE70aPHrWJhvez6P46/QvbfYqC89UbcGegbDQDA0CdTR/aUDw+MFQ4vPqeUIipKYJg4yuUwM7bpxzy/DL/GBlB4BHib11cOsEPTuhiff4xq+9CaJkXPOutWry0ItLBIL8d5QRfJNNvBNTZ7rznEZF5bbMOj2zuL37mX8mmKNMtG/9o0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Also after the common decompress part goes to __zswap_load(), we can cleanup the zswap_reclaim_entry() a little. Signed-off-by: Chengming Zhou --- mm/zswap.c | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 50405811cd7b..d3fedda0d774 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1438,7 +1438,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct page *page; struct mempolicy *mpol; bool page_was_allocated; - int ret; struct writeback_control wbc = { .sync_mode = WB_SYNC_NONE, }; @@ -1447,16 +1446,13 @@ static int zswap_writeback_entry(struct zswap_entry *entry, mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); - if (!page) { - ret = -ENOMEM; - goto fail; - } + if (!page) + return -ENOMEM; /* Found an existing page, we raced with load/swapin */ if (!page_was_allocated) { put_page(page); - ret = -EEXIST; - goto fail; + return -EEXIST; } /* @@ -1470,8 +1466,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; + return -ENOMEM; } spin_unlock(&tree->lock); @@ -1492,15 +1487,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, __swap_writepage(page, &wbc); put_page(page); - return ret; - -fail: - /* - * If we get here because the page is already in swapcache, a - * load may be happening concurrently. It is safe and okay to - * not free the entry. It is also okay to return !0. - */ - return ret; + return 0; } static int zswap_is_page_same_filled(void *ptr, unsigned long *value)