From patchwork Tue Jan 2 17:53:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29DECC46CD2 for ; Tue, 2 Jan 2024 17:54:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B489F6B0151; Tue, 2 Jan 2024 12:54:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD0A16B0152; Tue, 2 Jan 2024 12:54:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 999176B0153; Tue, 2 Jan 2024 12:54:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 82CDB6B0151 for ; Tue, 2 Jan 2024 12:54:17 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 65B75A085F for ; Tue, 2 Jan 2024 17:54:17 +0000 (UTC) X-FDA: 81635120154.01.18F7D0D Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 96A0AC0008 for ; Tue, 2 Jan 2024 17:54:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eLDjO8BG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218055; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=M6zIkgCMxkWWhd4LpQ5ScTsgktJ8VuaPe8qEbJ8TRiRKNmeWDUoLi3lqbcOPwOxg1YC49/ nKTieK6tXXzZyRLtP8WXh+LZ/qht1wV4735xojFOA/Y6TO2iAWOCX6aYS0VE6VgHVVyIb0 2gKuIe9VclS6Al+aO2BAE73Rfc6MiX4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eLDjO8BG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218055; a=rsa-sha256; cv=none; b=8ZLN4iX6sBze+wJ9HaA/6zIjI9fdYatTQ5mbVrH79f6rngJnWZ2ri+eKSDoPxhvNN9HHQw NG74DZoaSEtldfR4i/KuE+InTcwCIQ+lNMfnh75OCwC9mQtPi2l6d/ubYFdgO6rPRLzod2 ekRQbNfEGSTWPkydoA+UpS3547aN1TU= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1d3f3ee00a2so33824755ad.3 for ; Tue, 02 Jan 2024 09:54:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218053; x=1704822853; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=eLDjO8BGK2wP9aETufJqbqm9V2S4GnUurHCmtdI5G41RJS6XLtnm8lMSiTL4w6Q4dk BPETMPnYvJ3Vyq+6hoXwfXsOarT4V2uB9oO0eMXnJWbiQoAA8xYVCXcCeHp9hPpp1rd1 dGG7nSk55jJeIFyhohFzrLU1wKvCDmGdNlMfqzWz12pDXd3e+r5sgiXuS61NjLLhaDep UxmtvIjQa2k9JJAXOlB35u2Muyjf7ugpirohSJ+JZqcEyRaxENUWji42uGbyVJJB/dal vQHOcxrVVxOKfAIs5vuGRsmnj4i1mRsWW2DUmAghxhMPoUvzF1kFeT2zUkD5gsRNzBMv aveQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218053; x=1704822853; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=NKQNBt+uGtLZr3CgjqP4sbcQdKp4VrfCimbg7oJFf+mkCdVE4Pz9OVcXiXw2bHKE18 8BzA/24Ubobd0v8gbSoC3wP0IevIZaOptrOZW1fRK9Z20jQErFGwKPdRchlkD5ssCO0L 8aqE53orw7+InN2TjwGkXgqPE4FUKiCxTQwf69ps+uILJ3Nt+f2LbgvX1Qa5m/ESK6cV foMmLZL2TBKCAiPs/9s3q8L4oo/ReEOcm+xa+p6xSsx+1jKntQPf7lNvs0NlnhXHvMPz KPQumt/453ZH/FEH5CY7XDHygJCmNt2vNGV44H4a/uoYKuRSrjd73tH3tGMoZU5ZBA+G nBvA== X-Gm-Message-State: AOJu0Yy46aySSYbUKiq+Qu9byX2vtfwXL3Jruk1Frg06wfffwlBeWg1/ JnMAGCpNPc0tnpaJssh8tCzcYdGDnRcFkA2M X-Google-Smtp-Source: AGHT+IEqRxzrscJ5aDzXjoCzOKW/mGxYYasxFtxeT189T8sXpnySEuTBNSzOKUPECfaBTd/IXyLMhw== X-Received: by 2002:a17:903:2352:b0:1d4:c1b3:c6c with SMTP id c18-20020a170903235200b001d4c1b30c6cmr1260292plh.51.1704218053424; Tue, 02 Jan 2024 09:54:13 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:12 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 6/9] mm/swap: handle swapcache lookup in swapin_entry Date: Wed, 3 Jan 2024 01:53:35 +0800 Message-ID: <20240102175338.62012-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 96A0AC0008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: rntuf6y9r3kbn4tktm7sthdjuehj7m54 X-HE-Tag: 1704218055-625523 X-HE-Meta: U2FsdGVkX19AksxKBOcHdtQMV1UwoO023joaHgUQHD8mSMWZGXWaIr+2V5Nrq7whkb/pOcgdl51YXwF0pqJUb7Ex+ETK+EYipm7DVytzf2U5IWmYrWN+U1i6qQuZYpRqHaBEls8DAYolYML+7XHRpDPuMUyW5HyIZQqXtMe/STZ6YyTGA2rK9voxc1zjaqWXl7c2gBWYHbNH4QGBRZ5zz55fPpzUnFvcRoogFKAiYXS1SXVUk3ZqlvMgHnqtIkE50Sfy4c9SFB5Gbp3RXQtH7DvKanEGpUGdC37jH6Lwij0RzhKEk6ce8CbybwtEm06/7B25idSk89O7nwe/3Ea+2C+pbI/F2Na3FpeAjrKeLDfyXli9pmZOa2Sup2rC3/YH2SzZOFkr9QJe8H4ejsZYuZouQAPt6PDSBAshI8u0LTB+xKRAR5oUyxNU+pNCWlMPNSgKx0UA18UqfWA9xjv7tb0ZPl6vkkGfkr12P+ZZ4RgvmvHfQG0FyyyCADFoTimkMO8X9CrkSWkTT2GbjtTWtGfDa2NY/t6Gg6ONIjvHxLS90USm6uzKHHGUQe445sb7gSO9f5U31cCVS3OK9Z9K/hWyhxfzdi+AtvPLd3IO684YPZeThP2TDZKA637zk9OhRUMLHQjJsAMJ/9SbPNYNGz4rfbag8glWZuoITc1rYsUDS19vxnUsle39vp67YVYzo4+egxr5jYDhi2dDX9KqWp6zIwstLhnzEw2FE2SHRXurxecanDprFKGYcSv1/Se7yYjJYKRwGnafiNJHy1Rn01cYRO3ozivNMsY+VB3ekjkQIrIs+8G9wZYonz3fUpEgPWuzt9XpPKXAXUfJ7dtpVZ4SLwhjlZ3AiOTlACjuOMKx52N811VeC2t9FoOo0g/JDTB/3y1TNCtMHZ6Y7+2iG+fnJD30d8gUlnH9MkYDr/PY7/gkQqppUJnY3QB0JI0dobPkB6TlCJ3tHvn5ZEr W+C+9GoG KsNViIClETq9wfXdazVwPeONOECztQKZq7ubR9S6oSBoxkzhbPAFaJiPxBTSMEgy5GFIxFXQ8ECBeyLGqQdofm67FLXLpw0nolCRyDFO+bveRAIJWwD5MAzcyw20sbv6DGHsQ4kwOK0SL9Dvvz/t5d8tU47q9GGLgf6JKUZv2lLAEsfNS6tQEo6vS4xRdlCXkgltVVncUi9wnVoq/K9yXI+qG4fHcq3pdgYoMzOLBNv7Q+zQTCh3MRy0G4/ZAwVf+tVllAnhUDMHw4wIeRNVBXQmu5CuNzhNi2zyKh93XnIFxXZ1HTsqPhjoY7gjtthnevGoD/c2tCE8yYGI98eK79Q2YgnzGZwZyAabGwv9aEwpNeUweabxHneg+xC4p1OUR7bVqqYqLi/sZ5JCvjB1WC0aSbQ9nGAQZ+7NZ8SjV/CrTWa/f14P/2igxB3ijHdhTjh4YRHvaqpBA3ZSKszkMPE0w2VUJBY1WJda4ShdS+rrdofgbCnaIa4tF0jFU/GQEAnyTwFDUSHWSSDFX2vjbZwAYuMIkA++/U5575Q9DsS6WcsqzzLIVX82ZigAmPv+x35jv0XA1kXKrLKT0WUDVesZ1Q+/PsW318Mw/1/+7SswSO2IqyT/dnl0pv5tjWam51OAu33nWo7MGfs8Z5q1n9FUb1b/4WYarjbUfWOfCWQB62D4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Since all callers of swapin_entry need to check the swap cache first, we can merge this common routine into swapin_entry, so it can be shared and optimized later. Also introduce a enum to better represent possible swap cache usage, and add some comments about it, make the usage of swap cache easier to understand. Signed-off-by: Kairui Song --- mm/memory.c | 45 ++++++++++++++++++++------------------------- mm/swap.h | 20 ++++++++++++++++++-- mm/swap_state.c | 22 ++++++++++++++-------- mm/swapfile.c | 21 +++++++++------------ 4 files changed, 61 insertions(+), 47 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index b56254a875f8..ab6e76c95632 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3795,13 +3795,13 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct folio *swapcache, *folio = NULL; + struct folio *swapcache = NULL, *folio; + enum swap_cache_result cache_result; struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; swp_entry_t entry; - bool swapcached; pte_t pte; vm_fault_t ret = 0; @@ -3859,31 +3859,26 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - folio = swap_cache_get_folio(entry, vma, vmf->address); - if (folio) + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + vmf, &cache_result); + if (folio) { page = folio_file_page(folio, swp_offset(entry)); - swapcache = folio; - - if (!folio) { - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - vmf, &swapcached); - if (folio) { - page = folio_file_page(folio, swp_offset(entry)); - if (swapcached) - swapcache = folio; - } else { - /* - * Back out if somebody else faulted in this pte - * while we released the pte lock. - */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (likely(vmf->pte && - pte_same(ptep_get(vmf->pte), vmf->orig_pte))) - ret = VM_FAULT_OOM; - goto unlock; - } + if (cache_result != SWAP_CACHE_BYPASS) + swapcache = folio; + } else { + /* + * Back out if somebody else faulted in this pte + * while we released the pte lock. + */ + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (likely(vmf->pte && + pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + ret = VM_FAULT_OOM; + goto unlock; + } + if (cache_result != SWAP_CACHE_HIT) { /* Had to read the page from swap area: Major fault */ ret = VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); diff --git a/mm/swap.h b/mm/swap.h index 502a2801f817..1f4cdb324bf0 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -4,6 +4,22 @@ struct mempolicy; +/* + * Caller of swapin_entry may need to know the cache lookup result: + * + * SWAP_CACHE_HIT: cache hit, cached folio is retured. + * SWAP_CACHE_MISS: cache miss, folio is allocated, read from swap device + * and adde to swap cache, but still may return a cached + * folio if raced (check __read_swap_cache_async). + * SWAP_CACHE_BYPASS: cache miss, folio is new allocated and read + * from swap device bypassing the cache. + */ +enum swap_cache_result { + SWAP_CACHE_HIT, + SWAP_CACHE_MISS, + SWAP_CACHE_BYPASS, +}; + #ifdef CONFIG_SWAP #include /* for bio_end_io_t */ @@ -55,7 +71,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf, bool *swapcached); + struct vm_fault *vmf, enum swap_cache_result *result); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -87,7 +103,7 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, } static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf, bool *swapcached) + struct vm_fault *vmf, enum swap_cache_result *result) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index 66ff187aa5d3..f6f1e6f5d782 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -917,8 +917,7 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information - * @swapcached: pointer to a bool used as indicator if the - * page is swapped in through swapcache. + * @result: a return value to indicate swap cache usage. * * Returns the struct page for entry and addr, after queueing swapin. * @@ -928,16 +927,22 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * or skip the readahead (ie, ramdisk based swap device). */ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf, bool *swapcached) + struct vm_fault *vmf, enum swap_cache_result *result) { + enum swap_cache_result cache_result; struct mempolicy *mpol; struct folio *folio; pgoff_t ilx; - bool cached; + + folio = swap_cache_get_folio(entry, vmf->vma, vmf->address); + if (folio) { + cache_result = SWAP_CACHE_HIT; + goto done; + } if (swap_use_no_readahead(swp_swap_info(entry), entry)) { folio = swapin_direct(entry, gfp_mask, vmf); - cached = false; + cache_result = SWAP_CACHE_BYPASS; } else { mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_vma_readahead()) @@ -945,11 +950,12 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, else folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); mpol_cond_put(mpol); - cached = true; + cache_result = SWAP_CACHE_MISS; } - if (swapcached) - *swapcached = cached; +done: + if (result) + *result = cache_result; return folio; } diff --git a/mm/swapfile.c b/mm/swapfile.c index ce4e6c10dce7..5aa44de11edc 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1845,6 +1845,13 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, int ret; pte_t ptent; + struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; + if (!pte++) { pte = pte_offset_map(pmd, addr); if (!pte) @@ -1864,18 +1871,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; - folio = swap_cache_get_folio(entry, vma, addr); - if (!folio) { - struct vm_fault vmf = { - .vma = vma, - .address = addr, - .real_address = addr, - .pmd = pmd, - }; - - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - &vmf, NULL); - } + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + &vmf, NULL); if (!folio) { /* * The entry could have been freed, and will not