From patchwork Tue Jan 2 17:53:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4637FC47073 for ; Tue, 2 Jan 2024 17:54:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB96B6B0155; Tue, 2 Jan 2024 12:54:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D69B26B0158; Tue, 2 Jan 2024 12:54:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0B106B015B; Tue, 2 Jan 2024 12:54:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AD9196B0155 for ; Tue, 2 Jan 2024 12:54:24 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 94A87A0864 for ; Tue, 2 Jan 2024 17:54:24 +0000 (UTC) X-FDA: 81635120448.01.96A147B Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf03.hostedemail.com (Postfix) with ESMTP id B575020013 for ; Tue, 2 Jan 2024 17:54:22 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gyxTJ2DM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218062; a=rsa-sha256; cv=none; b=0AVKUpKnwB+sBHgkO7mipqTpSgjM2Fl41VVxG8fiEzJFVcHNY/q/uWzbaAYbJfOLuTiGWQ beChp8+oxqA1TfTR+YGz9C/T9ENS17Gx97Iv+R5CckusczAT2qks/CtYL4MTSmn1LNxWOe 8dUN7MhQTBOibHqCjrjhDKydvjc26wY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gyxTJ2DM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218062; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=Kaib8Qf/YQPwqh/BT9IZSDRIZm40csx5NxygoqMm0LHRhG5KqPz81uOK7L9/EceXVi4wn/ 4VLFWyHMpzqFTRJ6hr3TYP1Yq8wvDjE8Ozn6wXe8fhZIhGC02tOj9xEL36fjl9dpiuVSHh fmQHWYaFREaY1E3Cp7V2FUR1I8yCXQg= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1d3f2985425so34728175ad.3 for ; Tue, 02 Jan 2024 09:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218060; x=1704822860; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=gyxTJ2DMcPqfkpRg4OIRUKzhsS0/RpWDDGQRPcodFE95Y31ycGknQL2sl5mHkhqn9t aN1Mg/Pi138qohhUTxwh3z7cyfPJCnC8roBmAvIazNwtT7bTbw3yD+ZGLlsCOsDBL7Q2 mQl45v3v3NAkx8bzLbUCn+ldLrBFU2LCiPmgP2pMlRTGK+yJn4Ke3cxuNp3leBh+SMM5 uE+hmuT+PeSOwYrNI/WuOnXNO3j6OUVYYXw5DhatNCsqJRAnDzfhX9X+CooWhtGXUUVv V3Sv3dl3bZ2u+krokKBfe8PUxrUrE4mThPTHlI8ERY4G4QXltDXsKEcdUr17vbBFymVM BZmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218060; x=1704822860; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=ef5UkpvTroQGvpP1WCCsCatFvZ1U+Noe7jMyejh3XL4mvsM7PFVqxPAIfhq99/EJ9c e/7862mddwipECQFJZOWWYjugGmtbCILiCMDh4xXVZhnzmEIf1mPjjpmRRAGJZzksDZV XCqUgZ5zMRCZTQgQbPD6aU2bny0P171n+G+YCjjH4ha8DKLXQopulZGLOdyieUjEXjgn TXVwK96XA3/QeXLvPDTL0WzS/L/EgfR3SblyTvHmcKVtmbcm8BgA2m4bkn2spaDyqD1Y fwzE8Fxj3PbVoFJBgJTeR7m7Aug0L+1u1zgrmjxhbjAsWy2Va/tZdKgo0ocMZIX0QRZK hz6Q== X-Gm-Message-State: AOJu0YyuJdHzdR6mC+qXnSRuEsfE8HtLXU3mpvhCpsBPcms5DYR//mqb cdjNhnyLrvbesCG4LOUU31Nn+w1HxrXMgR/r X-Google-Smtp-Source: AGHT+IGr1fEmKCpFGmKewU8/hVrmqMb8AM0dTCE+QEyY4/Rq+9azV/ZjBTyHv60Nu5X7VVBSfSChoA== X-Received: by 2002:a17:903:110d:b0:1d4:cc31:71e with SMTP id n13-20020a170903110d00b001d4cc31071emr483019plh.67.1704218060519; Tue, 02 Jan 2024 09:54:20 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:19 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 8/9] mm/swap: introduce a helper for swapin without vmfault Date: Wed, 3 Jan 2024 01:53:37 +0800 Message-ID: <20240102175338.62012-9-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B575020013 X-Stat-Signature: ghk7gczj3p959kap65xq6djp48hqnoka X-HE-Tag: 1704218062-983891 X-HE-Meta: U2FsdGVkX18H2pwm9xof9iWsCyZm2ReYA7dyBwXzUxitXSqTIjgClI2oISmH9y3p6uoEq1RxRkbisanCU37D6G051WN8KauufG8zhGytEq+ocIAZKub7lgHPkwBJLL1dOc6zNVMgWF3+peX4eWCtkfNfBF52KX5NJTqBLGiRvp6Rq+pDlaj6lgTmNuCatR9GCRY9bUTq4XvmjYLtiNj6SiP/33OKJIZSBYxICLvty9P4EsR0F1Vo7SJelYrXOZpQ9z6agCPO7e5zOGN6y/fpsoPEh4nA9FbN4zVLVDUq8ewQwJU5oc1ulZMly3MgcF1rwgI6VeiQSD/1KfcvlIKUKutYBf8/MHFYARxH0B0oGuS3CpyxDTMyGdR6oLpd9STugOBDp6F0KkhwJTMu8/eXbiouhkOpdaqCNnFXlsMCtoZsvPGsSbVaHtIuaclRZKAHlkVC9ONSQZ31+YRtXx5VE6Re66lDRv71rxQ98D6BEgrLLlOqCxVgC/TqajhJP91Ba5N2fWtVxcLlPeODVd5zPT3aRLUwhY3MkReYj/XSH8691aS4E8536bkqSlY6j7p89sfugmY7q+9QZ4KahoRgzumGDOli03ESaMCvM2+h4gZpX3gim4Bd2F8QR515bcAsWzrFwOMoB3Q1uFaz3pBjClDQYfyn1eh3Tow8YiRxcevNJ5sjweNXc/ARIWeb6pDtUHknAmm96TpBORsAbJwt6x3cmJhCPaoWFPpwq98zigl0D9wA8OJyRd4jgPAPQ7Uvwlffab61uGmkgAyR6b+N8gqYTD2VPAsCfrUACUH284fKAFaS/RWm/8tj4J5nQpee0JYPtDEXSJnndswB3n7I9oXbA6FDHV87nM6MH1YMgCdCzkr3q5LZBNW+uUHxJUncqoEul7t63dp7qvG/Rlv/iDNpYj+yG0nCywAo6Rxl9iqqO/Zx0l5hU9WdBmVgQEbFZIkDHDWeD4Qg1LAF8of ewqUh2js g5LiuD5kCWZKpLrWHz48diNvXcoD5jqG7bljVVI49+t04LZT60Z1N32f/667hP447srgJD3UjnM/wUQpiWitZ7SzG30OBG74XAI6BLzTI7QfbVM/bblGqxD1p5SUz/tN9y9GlooprIFE+Ff0uITFD4DGmNxyQXjViERh/iFFyPfhBJewaQ8jzAjdYPa0oueXErVri2pB3DfJ9LkctKOrjBvMSIDDfxl8qd5rsOPKCgf7PtXnnvjTllN5c2bLMmSuQhz7MdDFn+Ezus0RKXHV9LI2MfoChGTn3NNjQsbchJuzA9BFX9+9eRoigf4cfCp0T08wrtmYQr/rcI3NPVJS+5BTUeco6I2PNuoE7fc2TfsX3hMm9DE+wPtCtRYetvi30L2AcRlZbtfYW9x1ZODrsdSm9hp535HK2IwODxCiF68xQziYBbYgSuqG/b14huBTMp2GJDpf2n6e3EFS9Qg+v4skYf9pIsdUU0oGRvOQkiYdGIuYclwlHfs2mMjaPdPrPY2i9OGqEvGNoXOEQTV4YpZ7KXFGnT6TAaBImCXF0QK5LcHZZy3HFNjY40D1YqFVudSF6vlfD6GdaA1BJ21AtdFwpGGzVqmMgPhJu/aBFug1TPu3867UtrzCGVJUtM0HMgjFP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song There are two places where swapin is not caused by direct anon page fault: - shmem swapin, invoked indirectly through shmem mapping - swapoff They used to construct a pseudo vmfault struct for swapin function. Shmem has dropped the pseudo vmfault recently in commit ddc1a5cbc05d ("mempolicy: alloc_pages_mpol() for NUMA policy without vma"). Swapoff path is still using one. Introduce a helper for them both, this help save stack usage for swapoff path, and help apply a unified swapin cache and readahead policy check. Due to missing vmfault info, the caller have to pass in mempolicy explicitly, make it different from swapin_entry and name it swapin_entry_mpol. This commit convert swapoff to use this helper, follow-up commits will convert shmem to use it too. Signed-off-by: Kairui Song --- mm/swap.h | 9 +++++++++ mm/swap_state.c | 40 ++++++++++++++++++++++++++++++++-------- mm/swapfile.c | 15 ++++++--------- 3 files changed, 47 insertions(+), 17 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 9180411afcfe..8f790a67b948 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -73,6 +73,9 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf, enum swap_cache_result *result); +struct folio *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, + enum swap_cache_result *result); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -109,6 +112,12 @@ static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, return NULL; } +static inline struct page *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, enum swap_cache_result *result) +{ + return NULL; +} + static inline int swap_writepage(struct page *p, struct writeback_control *wbc) { return 0; diff --git a/mm/swap_state.c b/mm/swap_state.c index 21badd4f0fc7..3edf4b63158d 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -880,14 +880,13 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * in. */ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf, void *shadow) + struct mempolicy *mpol, pgoff_t ilx, + void *shadow) { - struct vm_area_struct *vma = vmf->vma; struct folio *folio; - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); + folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); if (folio) { if (mem_cgroup_swapin_charge_folio(folio, NULL, GFP_KERNEL, entry)) { @@ -943,18 +942,18 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, goto done; } + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_no_readahead(swp_swap_info(entry), entry)) { - folio = swapin_direct(entry, gfp_mask, vmf, shadow); + folio = swapin_direct(entry, gfp_mask, mpol, ilx, shadow); cache_result = SWAP_CACHE_BYPASS; } else { - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_vma_readahead()) folio = swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf); else folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); - mpol_cond_put(mpol); cache_result = SWAP_CACHE_MISS; } + mpol_cond_put(mpol); done: if (result) *result = cache_result; @@ -962,6 +961,31 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, return folio; } +struct folio *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, + enum swap_cache_result *result) +{ + enum swap_cache_result cache_result; + void *shadow = NULL; + struct folio *folio; + + folio = swap_cache_get_folio(entry, NULL, 0, &shadow); + if (folio) { + cache_result = SWAP_CACHE_HIT; + } else if (swap_use_no_readahead(swp_swap_info(entry), entry)) { + folio = swapin_direct(entry, gfp_mask, mpol, ilx, shadow); + cache_result = SWAP_CACHE_BYPASS; + } else { + folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + cache_result = SWAP_CACHE_MISS; + } + + if (result) + *result = cache_result; + + return folio; +} + #ifdef CONFIG_SYSFS static ssize_t vma_ra_enabled_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) diff --git a/mm/swapfile.c b/mm/swapfile.c index 5aa44de11edc..2f77bf143af8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1840,18 +1840,13 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, do { struct folio *folio; unsigned long offset; + struct mempolicy *mpol; unsigned char swp_count; swp_entry_t entry; + pgoff_t ilx; int ret; pte_t ptent; - struct vm_fault vmf = { - .vma = vma, - .address = addr, - .real_address = addr, - .pmd = pmd, - }; - if (!pte++) { pte = pte_offset_map(pmd, addr); if (!pte) @@ -1871,8 +1866,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - &vmf, NULL); + mpol = get_vma_policy(vma, addr, 0, &ilx); + folio = swapin_entry_mpol(entry, GFP_HIGHUSER_MOVABLE, + mpol, ilx, NULL); + mpol_cond_put(mpol); if (!folio) { /* * The entry could have been freed, and will not