From patchwork Tue Jan 2 17:53:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 665B8C47073 for ; Tue, 2 Jan 2024 17:54:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F41CE6B0149; Tue, 2 Jan 2024 12:54:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFA306B014A; Tue, 2 Jan 2024 12:54:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1C2D6B014B; Tue, 2 Jan 2024 12:54:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BBFDB6B0149 for ; Tue, 2 Jan 2024 12:54:13 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 911AC1A07CE for ; Tue, 2 Jan 2024 17:54:13 +0000 (UTC) X-FDA: 81635119986.30.43221DA Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf05.hostedemail.com (Postfix) with ESMTP id B4459100005 for ; Tue, 2 Jan 2024 17:54:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gKwvfrOv; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218051; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=WWP+Lre2PiCZiSyMkJfNav4ZES+vvzQhGHNs7FqZ47AXiyXu1SFIAte+J60slB0CTioEab iqDOKZ2Q1/m7H+PSXVwmu5RwDGhOjbyFlbdILDpPNZA27brsJS3+OhB5eTFpoCa0nDY3Om jFfvZhs20T+gj/JKLoZ3kJlB/laFN5o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218051; a=rsa-sha256; cv=none; b=S05Qmk4FKrbuTsz/LDSqd1RgsXAKYD5v8aa4Y4LxTfyJo+gsG7qFKkCOIH8jdMI83/wp1i gV7VOq/W8a96R6d8driHNte80bHG98IhWcvzmLKicwBfjunCB3FQGS6oKu/UyLdFlnLdkn 9CiKPD/MrKb/3Mj5C7gj3+9r8NIsiVc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gKwvfrOv; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d3aa0321b5so76952855ad.2 for ; Tue, 02 Jan 2024 09:54:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218050; x=1704822850; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=gKwvfrOvIB/5zisa1TuPIqLNz0Pms5BbtXv+1zjbyv8mWJsbE5LVvB3D/1trXVIs/p Zb2V4rWfOilX302Wa++XZiTp+kYsnnx8IqAx48sImfn4qNAqr99z+Ehmp3fdufPTFtQ4 6plsuu0wOpzCWlKmVR7RWC+AvO0sULdJ6f9CUt+ZlvIZrO8bXHos39TNYz6EKjAUsFlu +DUWjY6iF0SJJKY7uoFVsxM3UNXwNWjx6MSy52HJeFfozqy5sLXFy3mR6vch1R8BOA+g J6cCLx+Fmj2e15Kew5H6PetWC6S39BMHKT5fxkxKa5LkfO8+helpG3xmuuJV26uMGNqC pvlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218050; x=1704822850; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=ms2YUfl99GcjNm1vpnPSMQalDWPFKPX1hbYbh//6qeEFJQcHiWegGB2zZI/7kZn1qh Pd/xgTfEQxJeg5SRABLGEMqQuxfpToLF0YDo16psxko2PRGagcT+sUqz/9aqpOHICLO+ JYjIcNQvKFkvn5MLLkahjdnnVTTNigD/QO5k6mEJilZlcxOlyxxTjXkrP94hJfXdAaeE tNFTF8H+G1nURAFxZ7e0ftTlbIFGlk8iZobXipkZhYZvGRH1i92g716dcD5ONWhxOvgO qNabjGSK+rQxBEwiPKM0in/G5BtlSdqCs8TBvpJIhehPzk9DwUlQM7HK41dMhtos2j85 8Hmg== X-Gm-Message-State: AOJu0Yxi3tCkRzDHU2/weW+v85Hpt88LTCg3H08hp12uuBkAaxqD/Oyi SgorRyme524ySgsx7XyTY8LCgJx6I9ckO+x8 X-Google-Smtp-Source: AGHT+IHg3EaXPsfJ3BsQyYM+9C4NVV8c46gR3f+IfIC1eTcpnGbk9EIEm5Ji/30WqG1cQYmyBx9asQ== X-Received: by 2002:a17:903:41d1:b0:1d4:3b72:b294 with SMTP id u17-20020a17090341d100b001d43b72b294mr20703980ple.88.1704218049886; Tue, 02 Jan 2024 09:54:09 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:09 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 5/9] mm/swap: introduce swapin_entry for unified readahead policy Date: Wed, 3 Jan 2024 01:53:34 +0800 Message-ID: <20240102175338.62012-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: B4459100005 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: u1up63owzr1snup5ugaotiik6qu3fbnz X-HE-Tag: 1704218051-779397 X-HE-Meta: U2FsdGVkX1878dycg5sIRuLEwITMVhLNRFtZAKAFWe3S5CnDpvBbXPXOzOKeEp8shYlbBP1L6X/KsQv5/rPMXCCrp07HEQq+iBfIduyWC7vekYfn0XZVkjrV21WOt3dZH4+XIBc7Ec8p/jN/YLOKZluaIJmAxKZJnscDGRfHUA7LuFlqhWmTGbiE0jLlNY4J3Ovbv5sQvnoCkO3c9Y8P2xDnGeqWMEtoNUY5eEMYnng8Z4jxUvX66YPWZaG+MAAKRSZG6pmWMfyy7K7QZVf6EMXrbbFe8tHS9EyBNLneSdRFlcWpRK+CWVPG9B51Uw6HKCnHOap+8+KaP8BqWrk04wP5koPn7Hm2j6XGDPffcR8B3WdyRGQKJy+UtGE676Lkcj9uRlxH0y6SMQqgCkejj0bKWde5dZlJKnGKC+WMCUE8Bg/OoFL29DW3t6yFhKhNrZNZlvsXYfbM1Gj7tQk10sOwExQE8GEl0gXhA1zvKF359xf7FtY3UgOUQErKKQNndNDGWw1213fKjla8iovxIU55K/MPIvnzvZY2X29KwHaEjdMR6BrlHtdGK3WoHKfUIfXV86mZkYjcYw5ime9I/hDVrliLvNCpuvg/aaHsqlVF46+D5B9Q6lHZ7FbjZXqWz/9jlsgM+0zDh72uF+RaLT1YH2g+iu4oi7Ic9ams6W3DEHFtqyXYMXYqEoArGsFQwL6I/oQrWK3Iz+Mao9sAlA0FGkgSqVjpovecOXvMq8PU0aOY/+/lG3mVWNGHi77JyCOLbOf6OwiVroTIKz3vzN83Lhtdxw+FI8cZZ2Vd0Q7+JnjVjfpGO0OVZYnpYflC7Td76/Y5xHTcmIAvc1wwjaDPiNCa4WwOFuOWrJm/9BcyPDpOFqj86rgokWbdB3DaV9uz74t54Jz+P6tFEuSF5qTtmy+Efo+PfRi6w5TmBtIGHOybVrXCGsMxQugt2HdUAIjmiM2vfGyCigpcAnk Ju2csSU6 7NQIGuQdxumHEUN4XNux06nepey68aNa1m6mi+of3xT+j5zruefeo5/TZYQFgW4Eai29JDK8Pdt0ve7NKSY7Zb4Lbw5W5WCjpBmFr+gmferExc5gOA3W9hw2hl2TvtoY62FTp1C4dm3pqrOWXajhLtyhihYzLKLwegCBJPrgokthn/+g9vZgRqH11GZLtN3IvECEdBRf6Ugu6II1McixxMGpUtGC3DXAJgdGuWzp2TSzAK5XTo5c/37bP+daI7MeY+K6DacKfZnL4MPChy7E5KTrr3lwyfpyPfz96JnzmpUjsx9ZDLwRjcKgZGB/1fTkB1rn144+q2Q2AUzHdzrQ5mKDiC0PkuGENAl8nAIQhNNSp+Nen3LkeuO0rf9HPilkaD0NK/WwR2TXgPeJjBD2CzR2wnbN+W5b0gK+cwoX/ur2gbuo5lHigFq9CGZt+PjYSiUbMXhbeRKypoyXTZyzYAKcpFo3Y72vclBz46vFyVeWaLGa/eqi7PENSkNWa6l8H6YPePA3L3ptaD4V9Cwzyz2vOhUwqHm1MfnBfrEszyR5g4LRPtctTqNgB8wM4pnKOCHoG5CFee4EF7gYrkHql9byFFPjE0DCr7ZFbViL4mFlbsVaqyCJ7QEDVuyRMDNUmljQveL3cvzWuRv+8Oxrf70HaXHNz1RUdHksgqXOcobTavqw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Introduce swapin_entry which merges swapin_readahead and swapin_direct making it the main entry for swapin pages, and use a unified swapin policy. This commit makes swapoff make use of this new helper and now swapping off a 10G ZRAM (lzo-rle) is faster since readahead is skipped. Before: time swapoff /dev/zram0 real 0m12.337s user 0m0.001s sys 0m12.329s After: time swapoff /dev/zram0 real 0m9.728s user 0m0.001s sys 0m9.719s Signed-off-by: Kairui Song --- mm/memory.c | 21 +++++++-------------- mm/swap.h | 16 ++++------------ mm/swap_state.c | 49 +++++++++++++++++++++++++++++++++---------------- mm/swapfile.c | 7 ++----- 4 files changed, 46 insertions(+), 47 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0165c8cad489..b56254a875f8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3801,6 +3801,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; swp_entry_t entry; + bool swapcached; pte_t pte; vm_fault_t ret = 0; @@ -3864,21 +3865,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = folio; if (!folio) { - if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && - __swap_count(entry) == 1) { - /* skip swapcache and readahead */ - folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); - if (folio) - page = &folio->page; + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + vmf, &swapcached); + if (folio) { + page = folio_file_page(folio, swp_offset(entry)); + if (swapcached) + swapcache = folio; } else { - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - vmf); - if (page) - folio = page_folio(page); - swapcache = folio; - } - - if (!folio) { /* * Back out if somebody else faulted in this pte * while we released the pte lock. diff --git a/mm/swap.h b/mm/swap.h index 83eab7b67e77..502a2801f817 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -54,10 +54,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); -struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); +struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf, bool *swapcached); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -88,14 +86,8 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf) -{ - return NULL; -} - -static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf) +static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, + struct vm_fault *vmf, bool *swapcached) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index d39c5369da21..66ff187aa5d3 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -316,6 +316,11 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) release_pages(pages, nr); } +static inline bool swap_use_no_readahead(struct swap_info_struct *si, swp_entry_t entry) +{ + return data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1; +} + static inline bool swap_use_vma_readahead(void) { return READ_ONCE(enable_vma_readahead) && !atomic_read(&nr_rotate_swap); @@ -870,8 +875,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * Returns the struct folio for entry and addr after the swap entry is read * in. */ -struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; @@ -908,33 +913,45 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, } /** - * swapin_readahead - swap in pages in hope we need them soon + * swapin_entry - swap in a page from swap entry * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information + * @swapcached: pointer to a bool used as indicator if the + * page is swapped in through swapcache. * * Returns the struct page for entry and addr, after queueing swapin. * - * It's a main entry function for swap readahead. By the configuration, + * It's a main entry function for swap in. By the configuration, * it will read ahead blocks by cluster-based(ie, physical disk based) - * or vma-based(ie, virtual address based on faulty address) readahead. + * or vma-based(ie, virtual address based on faulty address) readahead, + * or skip the readahead (ie, ramdisk based swap device). */ -struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf, bool *swapcached) { struct mempolicy *mpol; - pgoff_t ilx; struct folio *folio; + pgoff_t ilx; + bool cached; - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = swap_use_vma_readahead() ? - swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : - swap_cluster_readahead(entry, gfp_mask, mpol, ilx); - mpol_cond_put(mpol); + if (swap_use_no_readahead(swp_swap_info(entry), entry)) { + folio = swapin_direct(entry, gfp_mask, vmf); + cached = false; + } else { + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); + if (swap_use_vma_readahead()) + folio = swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf); + else + folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + mpol_cond_put(mpol); + cached = true; + } - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); + if (swapcached) + *swapcached = cached; + + return folio; } #ifdef CONFIG_SYSFS diff --git a/mm/swapfile.c b/mm/swapfile.c index f7271504aa0a..ce4e6c10dce7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1866,7 +1866,6 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { - struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1874,10 +1873,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, .pmd = pmd, }; - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - &vmf); - if (page) - folio = page_folio(page); + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + &vmf, NULL); } if (!folio) { /*