From patchwork Tue Jun 18 06:54:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13701841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEA00C2BA15 for ; Tue, 18 Jun 2024 06:54:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80A806B0154; Tue, 18 Jun 2024 02:54:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B78B6B0155; Tue, 18 Jun 2024 02:54:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 631D76B0157; Tue, 18 Jun 2024 02:54:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 413E76B0154 for ; Tue, 18 Jun 2024 02:54:40 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BFFADA0496 for ; Tue, 18 Jun 2024 06:54:39 +0000 (UTC) X-FDA: 82243096278.05.638C693 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) by imf09.hostedemail.com (Postfix) with ESMTP id AF7A914000D for ; Tue, 18 Jun 2024 06:54:37 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=Udpbvqa8; spf=pass (imf09.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718693673; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=koLTHCsWXBAa3N27f7lrGDpaxVfvYWdW4ehMIBy6BWA=; b=dxGfe6wl+cEMKGgnYRnVxTYr+UJAYS+rRxx1HHeqdAtI2YM81pbQCoxX0btvn1JpakCoST cMXhEE0Bem9NLA+vMBobW56/BeMVgPdw8NPe1rOE4mfx6/+bvHEw0O3ntXn6l4vftDTpx7 F14m6zNqRDyJxxznxek/A6eRFpv3vE0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=Udpbvqa8; spf=pass (imf09.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.100 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718693673; a=rsa-sha256; cv=none; b=MXUoyOyctbJUGRqtnKEEKbmNXrQ0JRKmB5JBpQdFcbrLtDjRcFtqcaDXFHN5IRVv4qzOen UFgviahliJF0eHwFn0ypezQn+TO38DYxx/uoC/5eOEMhMPHQONYIIdM1y5g+wfdxyZ6bx5 dJE7uZGPWIIiR7FgKBweSudPVodbxRA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718693675; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=koLTHCsWXBAa3N27f7lrGDpaxVfvYWdW4ehMIBy6BWA=; b=Udpbvqa8C9ywNR4Yd0WabC/RIfeEcOx2FL0N5jolBWFkzUk+OmGThUyCtZS05ueDOQBB6eFmtN8m82W9tO8H3jmWA3WXyrYhWB6n3J6QokY6xreIxVq1Xf8BdWjVkEYE5bm9Nn4RLcKV8SlML61FYBl2Xg7RJgUtryQdtEqqqfM= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045046011;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=17;SR=0;TI=SMTPD_---0W8jKUZG_1718693674; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W8jKUZG_1718693674) by smtp.aliyun-inc.com; Tue, 18 Jun 2024 14:54:34 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, chrisl@kernel.org, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, ioworker0@gmail.com, da.gomez@samsung.com, p.raghav@samsung.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 7/9] mm: shmem: support large folio allocation for shmem_replace_folio() Date: Tue, 18 Jun 2024 14:54:19 +0800 Message-Id: <75d1c1f2fbc88dce476a53fa90f085b3bebb83c5.1718690645.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: jsukdhgiwx56pnu5t36emzeku9yy8oyi X-Rspam-User: X-Rspamd-Queue-Id: AF7A914000D X-Rspamd-Server: rspam02 X-HE-Tag: 1718693677-315579 X-HE-Meta: U2FsdGVkX1+Monm45quVjmmIXiwxYuL/XDjtI/P8UKwoQ998hWUDHMya65zuFbkuzVYv5avyzccWMwPecdv9bOY1pyz6BuDlXQT/pp6FnBquRFgUlPUv6MBfU+Z6BzJczTPe8Rzem/9fN5IkOOPrjXCEgqqMCaDzIMYtorDuTPq+ykRskGYb1rsWu5E3uFz+lnTmnqoTlXPLKSuSnW0f70AnsMLzZ/HK15X/OzMg3HYuxeYG23IWY31PSgcJRCkjqn3E+yyD68zFn+UA2gZUQuOD8CRzwB7zxz1ilx5+UKbUZOSNx+PGiUM6+2/YHTYaQ2SGQQ0x+IbyTe9Ruu/wL/XFrJ5Bs++kBr+nTGL19cHSoYezzDnR0yELyuS1aRAfIFcuhYpRf9qcsw5BqMFSKO6Z+wUnQqLqBQJ6MNDBSk6cXQjT4Guuq0vQjhGoJvaPC9YtRUxSZoPVoBs0zjG/28dxc6vKIMvv1C62dOBBWwZsGECPoaXTyaePoqbGRxaTQDwqVCzm0S6v5/4tWd20zcIrNRiMXj8aJKeYIMzGQ8IuHs9VkM5W2q58+U2Efm32tIZo0rIJi/pKIC5RwkTfkbpaAPthgARZxruG96frpJR0SbVGpHidKkDduUSgOoF69mIBNXv+LdPLG814aK3dLwiwHSWAEi2svU99ysm2x86ODw6p3CJIf8WPPmLVra4uL3wBFpjWHOIQNcOZHGY89fTnpVBBlzKHPXAsNMpBAJ1mB12Rrj73NAtu3485gDGbR/Qm432/evOYKWus8Db/sDmwJHEMYrjfQqJsJx8C0igrArSvGjOkuSJMGNZlgl/KCE00H5VP5G8+L7H22ubVOMYyBygYY9+g3ZSZws/2S07uprNQj9aUcoVkzjuXzFI9NoGZ8wvQ3E7aRsZKINooSEeYaizDW1FMuEh0D6gU+qWi7ZqFEcDAuitqQWoboKHqbLEgFHfhWgkezazAFld NXFcsn4t 1jcHEsAYdJ7GX2eqp9pn60gZ7xmAOXvGLp+B84x25RK4XxMKrxk2Hd4bEzXU5zF/nSzeF7L/uBuw66o1rnMmrTmXLdHQHc+nWC1b0x0A51mJ1tkxCQGvtbW9XMvtpZQIFku/QdWsM66FLjysxYwYCtozz6BNg2pq08zU49ASeaE8rQ/WhTw7IQ/2Trqg27XSFhABtC1ulpM6bE5+R09wwxwBkekspvufkmcr2o7Y4PsANCEPmq5vK/IhMGanx6loavshV0l+LORjXT8w78WtQvDAYHZfbkLMzeIlh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support large folio swapin for shmem in the following patches, add large folio allocation for the new replacement folio in shmem_replace_folio(). Moreover large folios occupy N consecutive entries in the swap cache instead of using multi-index entries like the page cache, therefore we should replace each consecutive entries in the swap cache instead of using the shmem_replace_entry(). As well as updating statistics and folio reference count using the number of pages in the folio. Signed-off-by: Baolin Wang --- mm/shmem.c | 53 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 4d7996962388..c0a9253f3a99 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1878,28 +1878,24 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct folio *old, *new; - struct address_space *swap_mapping; - swp_entry_t entry; - pgoff_t swap_index; - int error; - - old = *foliop; - entry = old->swap; - swap_index = swap_cache_index(entry); - swap_mapping = swap_address_space(entry); + struct folio *new, *old = *foliop; + swp_entry_t entry = old->swap; + struct address_space *swap_mapping = swap_address_space(entry); + pgoff_t swap_index = swap_cache_index(entry); + XA_STATE(xas, &swap_mapping->i_pages, swap_index); + int nr_pages = folio_nr_pages(old); + int error = 0, i; /* * We have arrived here because our zones are constrained, so don't * limit chance of success by further cpuset and node constraints. */ gfp &= ~GFP_CONSTRAINT_MASK; - VM_BUG_ON_FOLIO(folio_test_large(old), old); - new = shmem_alloc_folio(gfp, 0, info, index); + new = shmem_alloc_folio(gfp, folio_order(old), info, index); if (!new) return -ENOMEM; - folio_get(new); + folio_ref_add(new, nr_pages); folio_copy(new, old); flush_dcache_folio(new); @@ -1909,18 +1905,24 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, new->swap = entry; folio_set_swapcache(new); - /* - * Our caller will very soon move newpage out of swapcache, but it's - * a nice clean interface for us to replace oldpage by newpage there. - */ + /* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq(&swap_mapping->i_pages); - error = shmem_replace_entry(swap_mapping, swap_index, old, new); + for (i = 0; i < nr_pages; i++) { + void *item = xas_store(&xas, new); + + if (item != old) { + error = -ENOENT; + break; + } + + xas_next(&xas); + } if (!error) { mem_cgroup_replace_folio(old, new); - __lruvec_stat_mod_folio(new, NR_FILE_PAGES, 1); - __lruvec_stat_mod_folio(new, NR_SHMEM, 1); - __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -1); - __lruvec_stat_mod_folio(old, NR_SHMEM, -1); + __lruvec_stat_mod_folio(new, NR_FILE_PAGES, nr_pages); + __lruvec_stat_mod_folio(new, NR_SHMEM, nr_pages); + __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -nr_pages); + __lruvec_stat_mod_folio(old, NR_SHMEM, -nr_pages); } xa_unlock_irq(&swap_mapping->i_pages); @@ -1940,7 +1942,12 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, old->private = NULL; folio_unlock(old); - folio_put_refs(old, 2); + /* + * The old folio are removed from swap cache, drop the 'nr_pages' + * reference, as well as one temporary reference getting from swap + * cache. + */ + folio_put_refs(old, nr_pages + 1); return error; }