From patchwork Mon Jan 29 17:54:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13536158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9220EC47DDB for ; Mon, 29 Jan 2024 17:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 214EF6B0082; Mon, 29 Jan 2024 12:55:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19CF16B0083; Mon, 29 Jan 2024 12:55:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2B76B0085; Mon, 29 Jan 2024 12:55:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D3DF26B0082 for ; Mon, 29 Jan 2024 12:55:24 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B0C1A807A8 for ; Mon, 29 Jan 2024 17:55:24 +0000 (UTC) X-FDA: 81733100568.28.43769A8 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by imf04.hostedemail.com (Postfix) with ESMTP id D471040021 for ; Mon, 29 Jan 2024 17:55:22 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TKk26o8W; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706550922; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y3NzQYMmpvHTXBLoyc702W5Zl5rwxoKujBWnFPZfTzI=; b=5gs+MWJE6Fkic3Ox2TK441V6hit8XoWVPKkUhj7c85v2WufOW1Hv9ZndL6k3Zx0iX9rP3X n00zjbSrbxFFobnBBCZaNkUylDGcIKIxMrMtgYql4+r9LlCg7jytp04HJmp2gf/e3iJFEI eUaX6pr8xO5A2gSBy7NZmJjcSXaceEU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TKk26o8W; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706550922; a=rsa-sha256; cv=none; b=HDmj/5p//TSzZLXDOjjByJ7plh/L9lsceJ1MFVvuH5Y26nDMqUxrG8ZXMMgTqeXKG9n8nW bD0lWNwthsdIXHtV8+CseTdwfW/oxfq0HeAAEDa8/YuhzKHD8Be5Ei9Mw6x70HcWWQKLs7 aHknKbMAZP0jHdIievKLQGs4XMtK2sk= Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3bbbc6b4ed1so2321398b6e.2 for ; Mon, 29 Jan 2024 09:55:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706550921; x=1707155721; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Y3NzQYMmpvHTXBLoyc702W5Zl5rwxoKujBWnFPZfTzI=; b=TKk26o8W4sKYJuS1a++4irMMs8axDJxBEkNHtHU725Uo7En41jzAe7ErCL15Nrxy79 bvCbayzSGRZNOZxp1YtAAQwXh//x51qckwp4/AUfTfpHoy565uMljGqzeS4k85SD6vwR BXjUyCCXjl3FSs+WZmmJvRXpzWKyV0M/826IWCGG5zM/fPyol56G/4XaJNHjo8j53GhC 29GUt/LOA3vz7fZc68tWk1rgPI83kiziGWoLTAjdKfqnNYOD0s5KVkW47jVz3HxeJHn3 yOctdMJGDXV5PmOqM3EeM3oWlrVzwqEVSeShuxUMFwWAQJrBhpJEj1sKRizqvgxRZY3Y rXpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706550921; x=1707155721; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Y3NzQYMmpvHTXBLoyc702W5Zl5rwxoKujBWnFPZfTzI=; b=EIh9XdEGwVYRai3NQuCfnKlG+9hHzRTvXyGCP0dC7XYjWH0p5pH1R65xjNQF54rINo THtmbWgdnE5CzI56UhmsAc3t7p3OutNQvGnmTB9cbRTTCHj/WpQBoxvFz/l3zboJO4Lm P1KGbAlxYxtsqeS34G/5gyHWFOhnZEKpN/z5MP8lEyzlrNPMsOtWmGJ92aqa0iyaSmd/ LYDACXv49HrI82iNW5NI2DckepgTCy7T7sft+JSl5Sd082cfVbGUZJqUlUSwWjtbrmmK x6HAn+SV1ThDePYMLdOlhVr9L8hk6dkHkppxUHmnrN590C3apO8A6hukDfeuO8Kj8z1b JBRw== X-Gm-Message-State: AOJu0Yyrik7CRwW+2VtwTYsEpkDXpCL6yXEBNQ3ri4XsnEEEEZ0fYBxf wXPGzDvfa+JaFUD0UX0laopJDcHytNJdgTUnaMeWBljflarbBMqcETTYlKaVaTJ65A== X-Google-Smtp-Source: AGHT+IHKBzL9b+ijQ48IAq8a2mBY5ABGwlr8jPlejfJOdScwkjPjpxglTY+jaaywvyWnKFqMNfV0Xg== X-Received: by 2002:a05:6808:13cf:b0:3be:5998:2cd4 with SMTP id d15-20020a05680813cf00b003be59982cd4mr2884547oiw.54.1706550921256; Mon, 29 Jan 2024 09:55:21 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id h8-20020aa79f48000000b006ddcadb1e2csm6116676pfr.29.2024.01.29.09.55.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Jan 2024 09:55:20 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 5/7] mm/swap: avoid a duplicated swap cache lookup for SWP_SYNCHRONOUS_IO Date: Tue, 30 Jan 2024 01:54:20 +0800 Message-ID: <20240129175423.1987-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129175423.1987-1-ryncsn@gmail.com> References: <20240129175423.1987-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: D471040021 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: tgnzh8abmj5a4mq3dptxz4eiy3ukh9oy X-HE-Tag: 1706550922-392171 X-HE-Meta: U2FsdGVkX1/Nz/MFw6MYpbyu1rp+46p2As7Pt0tu56iqW3rfv6NkvcHWFtUZJvgs5s7mhS0/Bgkl4DhaYM5e5rjrvn/QVjYVqdcIb7S180zogUXI3cm/ieREOkm5wCnVplH1va3CFz6e4CB+Aouxo84ghLxdgOhiXgc67dtCOsYspUcTpE3BFhx9ASoOoZe5BiG3evnpAZ6KyvFE7b7SC0k8761cdkjlysfpV8h66B63SCDzUn+68nd4Ft0lMqWpJSFNj8dOMVDBD86tQ451IH4WPYzgUDzMakNNn6h+ZtW/whiWRUEyepxt9t6oIgd592r7wuCRZpcRBO7hoi8L5r2x8S30vVkspo59jE1FMSf/+e4s07QIMOSlynbJ+SrSRyJljd5/8unMsqufjhPJo35jV4F2+iRzqxq93hqm3cW0DoSIOW8cHxY8xf7iCcrPHd6/u1I1WN030RM701qi7Wv0dujiNcxq5RYgeXNx8wIqGWc87D0qyECC4AfHZEN8v/ZXu2+XdrnxjFht1xGe7livdbQNjwQ/biRSMWPVc12BgllR2qCMOc45CcO14BfovM9A0USuE+tDJ8TL5Ym7BCqR+HrcMIy0MQKxZ4Ujp1jJo/Xq6ESXJdcZYEJVgyCWcqU4KRBB+Q+uC+qcaz2b9KkAVKVzP/nIkSa74vf6aAadoCgQsZDrHwGTr5L//M3tEui2tfoa5jMvK9/z3awdRzZJvZ6vtyEA3NFqPemID3RTkckXgzcJojbKOFzzNgH5cf/Cu0bq8hXxnDiTsNKrXfRgzRgwuz1RBfyp6pb0zO540rI4crN7b/Fu50tZL7+06grTDEIYJU1cIyn3PTzHpzdXHAntOxTPgw62MtX8YiokRf3pg4ItCYJz7REDYCy/GKyz4fpBnuuphS6gyLJwNrHI74zn7k/gJfgFuO/ukCy2i0QUaPPc5Gk1dBunPcwIPTqG6NUunGc4/uopno4 Z6PCiUJ3 cPtZMf0Oo9PRF2R3abfi0kklG2BXkSR8MEu9MWUakbXIaK81HjXL73T5LFMEbjDoEFhPlHtPSWCI1gXBUlcFGT8ZsvddwhIoZozUOQMhthGv9dXsUFAYaI0eYWdUwemuHS+9vguYOBB0+W20R2bQ8FH4/RuHmu6Jb05Ajs6SdGx4YIM/eSYKZ3uIKEh2HVCWhGr2oQjhBPmGWkjo2GFRHYdKF/qAMt5nOt/FLFe+nKEa7wCbKBp05sIxSYJcZi7ZOriaNhr7jO6unfPwWFBihtONHPvK2UeRrvZpNYW9GhaVpD0rCSJUjgDSvXN0f8ZBnGfgp9YnIOaF0TQxIQurwd5AfFRrpyYiXegF/wmkFr72Aol6u3BvLSBiZJG7TvUTlGYTt9N5INTcDjVsVBqR4BQF2MBwuOIxv1XAJZv8kNQ7stPrWqj0BLsM2Ltsv/yiUenCqpj8yVERkr7wYrCvpIpwnPOtUh4GJ/JIqVXoX5tHSRiIEF9NgsnZ3/3p9kkn0x3LdgHYYci2gSOZepD+k7D3woDI4s8x6Udr9mxpSQm5KKn0NvlCoCrKwRcoaj6Ih5ME3UxVdBtn2wgXBAk/YKQBXXucbKbPh0dQusxYGTfVe+n/wBY2Q+wbG+/4K2V+HXgiF2ZaWyteeEEmJwqzn6b5RbkMpvnN8eMHHxEKbyO8I7Zg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When a xa_value is returned by the cache lookup, keep it to be used later for workingset refault check instead of doing the looking up again in swapin_no_readahead. Shadow look up and workingset check is skipped for swapoff to reduce overhead, workingset checking for anon pages upon swapoff is not helpful, simply consider all pages as inactive make more sense since swapoff doesn't mean pages is being accessed. After this commit, swappin is about 4% faster for ZRAM, micro benchmark result which use madvise to swap out 10G zero-filled data to ZRAM then read them in: Before: 11143285 us After: 10692644 us (+4.1%) Signed-off-by: Kairui Song Reviewed-by: "Huang, Ying" --- mm/memory.c | 5 +++-- mm/shmem.c | 2 +- mm/swap.h | 11 ++++++----- mm/swap_state.c | 23 +++++++++++++---------- mm/swapfile.c | 4 ++-- 5 files changed, 25 insertions(+), 20 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8711f8a07039..349946899f8d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3800,6 +3800,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; + void *shadow = NULL; swp_entry_t entry; pte_t pte; vm_fault_t ret = 0; @@ -3858,14 +3859,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - folio = swap_cache_get_folio(entry, vma, vmf->address); + folio = swap_cache_get_folio(entry, vma, vmf->address, &shadow); if (folio) page = folio_file_page(folio, swp_offset(entry)); swapcache = folio; if (!folio) { folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - vmf, &swapcache); + vmf, &swapcache, shadow); if (!folio) { /* * Back out if somebody else faulted in this pte diff --git a/mm/shmem.c b/mm/shmem.c index d7c84ff62186..698a31bf7baa 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1873,7 +1873,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* Look it up and read it in.. */ - folio = swap_cache_get_folio(swap, NULL, 0); + folio = swap_cache_get_folio(swap, NULL, 0, NULL); if (!folio) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap.h b/mm/swap.h index 8f8185d3865c..ca9cb472a263 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -42,7 +42,8 @@ void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr); + struct vm_area_struct *vma, unsigned long addr, + void **shadowp); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); @@ -54,8 +55,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); -struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf, struct folio **swapcached); +struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf, + struct folio **swapcached, void *shadow); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -87,7 +88,7 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, } static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf, struct folio **swapcached) + struct vm_fault *vmf, struct folio **swapcached, void *shadow) { return NULL; } @@ -98,7 +99,7 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc) } static inline struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, void **shadowp) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index 5e06b2e140d4..e41a137a6123 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -330,12 +330,18 @@ static inline bool swap_use_vma_readahead(void) * Caller must lock the swap device or hold a reference to keep it valid. */ struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, void **shadowp) { struct folio *folio; - folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); - if (!IS_ERR(folio)) { + folio = filemap_get_entry(swap_address_space(entry), swp_offset(entry)); + if (xa_is_value(folio)) { + if (shadowp) + *shadowp = folio; + return NULL; + } + + if (folio) { bool vma_ra = swap_use_vma_readahead(); bool readahead; @@ -365,8 +371,6 @@ struct folio *swap_cache_get_folio(swp_entry_t entry, if (!vma || !vma_ra) atomic_inc(&swapin_readahead_hits); } - } else { - folio = NULL; } return folio; @@ -866,16 +870,16 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information + * @shadow: workingset shadow corresponding to entry * * Returns the struct folio for entry and addr after the swap entry is read * in. */ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct vm_fault *vmf, void *shadow) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; - void *shadow = NULL; /* skip swapcache */ folio = vma_alloc_folio(gfp_mask, 0, @@ -892,7 +896,6 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, } mem_cgroup_swapin_uncharge_swap(entry); - shadow = get_shadow_from_swap_cache(entry); if (shadow) workingset_refault(folio, shadow); @@ -922,7 +925,7 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * or skip the readahead(ie, ramdisk based swap device). */ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf, struct folio **swapcache) + struct vm_fault *vmf, struct folio **swapcache, void *shadow) { struct mempolicy *mpol; struct folio *folio; @@ -930,7 +933,7 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, if (data_race(swp_swap_info(entry)->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { - folio = swapin_direct(entry, gfp_mask, vmf); + folio = swapin_direct(entry, gfp_mask, vmf, shadow); } else { mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_vma_readahead()) diff --git a/mm/swapfile.c b/mm/swapfile.c index 1cf7e72e19e3..aac26f5a6cec 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1865,7 +1865,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; - folio = swap_cache_get_folio(entry, vma, addr); + folio = swap_cache_get_folio(entry, vma, addr, NULL); if (!folio) { struct vm_fault vmf = { .vma = vma, @@ -1875,7 +1875,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, }; folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - &vmf, NULL); + &vmf, NULL, NULL); } if (!folio) { /*