From patchwork Tue Mar 26 18:50:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F83C6FD1F for ; Tue, 26 Mar 2024 19:04:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BAE96B008C; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 743846B0093; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 570AD6B0095; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3D6C96B008C for ; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9DC0E120E4D for ; Tue, 26 Mar 2024 19:04:29 +0000 (UTC) X-FDA: 81940116258.03.D383107 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf28.hostedemail.com (Postfix) with ESMTP id A6EA4C0012 for ; Tue, 26 Mar 2024 19:04:27 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lSqk6lOB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479867; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=Q2p2a/reY/vt3qmdAb6ahWHnQ6jwxo/KagnmYIp+jGZyvo4GNDf8VFlA66KlSpnm9xR4vg bsSF1mHQONhk2j4CWRZfSQUnFSHoRPz/h9tJeMXDSljMpR39qDmnVNacPI9L46DfKHAiui NujZEgAiMM+8q39puCdqCriwPhu4SQ4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lSqk6lOB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479867; a=rsa-sha256; cv=none; b=uj5RLEVLOL4d3k7vmW0bkqedvb3IlNdkNsaAC3g45hn081kSl21gMBQ7F2cWt6e3lzf8BQ 5jakKbqAgSmvPgwSGYhfpiuM2RdRNcfFVL5RzQRuYZBClmBX35opC3cXEfflu8EGPAfswB PNc1538QbPeuUEqlqN1vc1p/TMHZR/k= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e8f51d0bf0so4729223b3a.3 for ; Tue, 26 Mar 2024 12:04:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479865; x=1712084665; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=lSqk6lOBePgsU61sfZby5ESI85aiNkpBxI+ReDxtSAWVNpAOpCEl2S3UIQDyAwo+ms MJR4V4WorxraU3E/BAdVjR08efCMrKopT4hZ0UvN7o3lNOvUKFYrcC2c+7GuEn09e0CJ khf7yMEoUQsjZZrDHaC0HzmKRUTuL5Xto6Deem5nsMbmMc8c/e1UOPB73P9uAYZ+MmXQ bgLzpiTEqCez2ksQSQYGI7P3WybGWKr0/r3PnBvJ5TNjEPMKDzFQJCRIYxVmknm3+u16 qu8Pzw0aeSCLl+2+906Gy5MPfwM/IW7s4PTQgVT6+KRiQDI1xWjHIx3nlootDyOyeVbR 5+GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479865; x=1712084665; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=Vli/P9OxMqsaTBePsQKGjWDcHEep6bpQYpmH0UvEK60Zhbsi5dkP3fVuq52yQF6wHu WEf3eaWp7HLqNWuS1KJnzATxG71/UkuHPdorDcXWdkC9iliN/4/fmcsYKHU4syKQ5bKg uLvvlp1+qIGPunU6kVkxVOCzDfcDE2t9XO5WLIMdVZmOAsaWjxPFKkQyelgfbenFuNHR uglCaG1br2IMj06hvWDTe1pLitDtFomBymeawQy7B2u4XdZWLFB+BxCSBRKnm27yEPO3 4QoFj0P2wqVIPIxAN9gsdAr/1Dx4ysDQuCMgIN4qlhTsyHT7/xrc19urYb/zOR+5jDbr 5uZQ== X-Gm-Message-State: AOJu0Yy45iCgJfmxj+hzxeyqUx1l9dUJBRtMUfa6EW3TkoC+m/KvW5Nu ew749pnpNgch0Y7y8vseneN9lJR+DBQ5q2fAWyJ3WdwCQMcMrAupItxKJ1JZqSEaQPba X-Google-Smtp-Source: AGHT+IHasTHngVbHXAAz60DxTdM0n3aSiwZ9Kg40P8vfF4TYZxDB4AbSEHDL00iy0NSqMDT5XDhRaw== X-Received: by 2002:a05:6a00:21c9:b0:6e9:a70:aa7a with SMTP id t9-20020a056a0021c900b006e90a70aa7amr3745251pfj.19.1711479865567; Tue, 26 Mar 2024 12:04:25 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:24 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 02/10] mm/swap: move no readahead swapin code to a stand-alone helper Date: Wed, 27 Mar 2024 02:50:24 +0800 Message-ID: <20240326185032.72159-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: A6EA4C0012 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 4q34yemxqtp9cngmspettebdwe7kfkuc X-HE-Tag: 1711479867-171001 X-HE-Meta: U2FsdGVkX18DD+39LHp4uSB/J1qilViTPUnmRf4ydfKQ+JcEIyQbgUzUNlsk8WU0Go4fxtEGl9pY/g8uHfkbex/Oxw9x4ktCscqNv/r3xYLT8V1rnSPJPjtMMPQ4zU22GhBbWbaQcbl5R1Oefy4peGwMqZQ0p4ae87dxwVC7Ce3yPiSmtFCxVa8swL7edW0fNstL0flTXFyayLa4ewcyoNAs7ZpS7gV6njMy7UBWLKEbmm4ajVse5Cz8hG0VLTjeIh1vSd/0PyOW7M4bBIRRNCxyx6AG8IGvfsy4hd0B7Lr7bj59kb0lNnZQwMj+UfJN+9UE7a3cgANihRKmRTGogEY6KKzqs2dsFVZg3jc0rlkMSIPahTXA8Uulq5qps+BwQ2fIfuEXq0hpdi1C4Lta15QA/JunjsudFqNZxiQNyhrLe1tFukyQN6HhYv7x4b+clyxTuR+tQADpTQqHEBwmiKSdV/Hl96DAc3QBZ/0Agi40LJUdJs9+pjZGgR0pQnGiioivm3TB/JrCHQd0VUGIVyCAq1KM2faJXx7Cju7pZ3gsC/yC6eZAlvzXfFk4gv9NXfWifQrxlUkS//pwxGPx6Ly73zBp5Y8l/Sif68HsDjPcwMdSrYBGFqqEqt9/4Y0qjt2hleVSxx9zTe0B2MbUy0ywi0jBTpgiDI5oxhgERrANxoixlFiMbJ5J6b7laDQ4SGEF3D9yVO8AxssazKf5qfrAHQojBA6HsIZThrT3KsAr3qhr4HJwRDZC7etil7vtBYrKydJYAXu/o8DOFF2LIhGEcfyYcTdz9is+I9Xo29oIcwRmPz1ldSyRNUBJQul5qzQPcNJ/bw0WmUGOo855FWwPjiDf3vqWh/Nu9tycmchsHDITIwSAOI7C+4JVS94RKLIGQeiDk0Wo3xgaXRYoLxFPEiwfqe4oZp1tslICpdnMLYy66zyWLC7afEAk2EFMt4swnJ8L+Otetu/Bp61 Wwhvw5Ry gI4RM6gqhWYJ7GiCGYW2UClFlF51KWHR/TPFFwlNwG8B2ybqJFuiiwWpqKbpbzwNb3CLUjINdQnuufc7IrDRQE56j9c42YxbXleiE8hYyjCZgoNL8i2GzxY73rBnyLkttNBVPPLZT5NrXZBZ5+ckKHiShWr6dZflMaOW86qMxscHmOSLu1ZTqfSj7ywxfkyoWt2qOCQvcP9gCFzfnIhpqoGoenR+OjZfRVB9I5t1sup6Yp/0/tg98IEPJzPWyzb+qmMMy5BIsRao11tqI/AUx1olYoHmcfBdSu4lbDTFKyjNlQ9HBobeXqdEcWY1XrXHP7ccW4XwQQxrHEUZ+ssSRBfGSxtQXpU4DhP4s/H42N71OyLmzvuq6WRvzrA0MY0ToPzv8xfjCqrYxqcmYiUa/l3ekyZnP+jOgAFjrdNdr0y+jtACE8wh55k4L/I1lS5Oufpp5251p5sOdYbrI9/Ndlj3DBK3lRNBTKcTa5bAoq3gYBV5MRW95gDAsBffdAdP3eYlP6cOmXaMOLPeuFhikZzIiOx+yyXLv9REnByoUP/092gHTCNVTqiWCSgclE5wUdswrHJCqSqyo7bTB4yIL1EZYo184Q3AMJnvJaQZJRDaI/VE84e1Oz3SGsm2iPtGdIdhR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Simply move the routine to a standalone function, having a cleaner split and avoid helpers being referenced corss multiple files. Basically no feature change, but the error path is very slightly different. Previously a mem_cgroup_swapin_charge_folio fail will cause direct OOM, now we go through the error checking path in do_swap_pte, if the page is already there, just return as the page fault was handled. Signed-off-by: Kairui Song --- mm/memory.c | 42 +++------------------------------- mm/swap.h | 8 +++++++ mm/swap_state.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 39 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f2bc6dd15eb8..e42fadc25268 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3937,7 +3937,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swp_entry_t entry; pte_t pte; vm_fault_t ret = 0; - void *shadow = NULL; if (!pte_unmap_same(vmf)) goto out; @@ -4001,47 +4000,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { - /* - * Prevent parallel swapin from proceeding with - * the cache flag. Otherwise, another thread may - * finish swapin first, free the entry, and swapout - * reusing the same entry. It's undetectable as - * pte_same() returns true due to entry reuse. - */ - if (swapcache_prepare(entry)) { - /* Relax a bit to prevent rapid repeated page faults */ - schedule_timeout_uninterruptible(1); + /* skip swapcache and readahead */ + folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); + if (PTR_ERR(folio) == -EBUSY) goto out; - } need_clear_cache = true; - - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); page = &folio->page; - if (folio) { - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - - if (mem_cgroup_swapin_charge_folio(folio, - vma->vm_mm, GFP_KERNEL, - entry)) { - ret = VM_FAULT_OOM; - goto out_page; - } - mem_cgroup_swapin_uncharge_swap(entry); - - shadow = get_shadow_from_swap_cache(entry); - if (shadow) - workingset_refault(folio, shadow); - - folio_add_lru(folio); - - /* To provide entry to swap_read_folio() */ - folio->swap = entry; - swap_read_folio(folio, true, NULL); - folio->private = NULL; - } } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); diff --git a/mm/swap.h b/mm/swap.h index fc2f6ade7f80..40e902812cc5 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -55,6 +55,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); +struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); @@ -87,6 +89,12 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } +static inline struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf) +{ + return NULL; +} + static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, struct vm_fault *vmf) { diff --git a/mm/swap_state.c b/mm/swap_state.c index bfc7e8c58a6d..0a3fa48b3893 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -879,6 +879,66 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, return folio; } +/** + * swapin_direct - swap in folios skipping swap cache and readahead + * @entry: swap entry of this memory + * @gfp_mask: memory allocation flags + * @vmf: fault information + * + * Returns the struct folio for entry and addr after the swap entry is read + * in. + */ +struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct folio *folio; + void *shadow = NULL; + + /* + * Prevent parallel swapin from proceeding with + * the cache flag. Otherwise, another thread may + * finish swapin first, free the entry, and swapout + * reusing the same entry. It's undetectable as + * pte_same() returns true due to entry reuse. + */ + if (swapcache_prepare(entry)) { + /* Relax a bit to prevent rapid repeated page faults */ + schedule_timeout_uninterruptible(1); + return ERR_PTR(-EBUSY); + } + + /* skip swapcache */ + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, + vma, vmf->address, false); + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + + if (mem_cgroup_swapin_charge_folio(folio, + vma->vm_mm, GFP_KERNEL, + entry)) { + folio_unlock(folio); + folio_put(folio); + return NULL; + } + mem_cgroup_swapin_uncharge_swap(entry); + + shadow = get_shadow_from_swap_cache(entry); + if (shadow) + workingset_refault(folio, shadow); + + folio_add_lru(folio); + + /* To provide entry to swap_read_folio() */ + folio->swap = entry; + swap_read_folio(folio, true, NULL); + folio->private = NULL; + } + + return folio; +} + /** * swapin_readahead - swap in pages in hope we need them soon * @entry: swap entry of this memory