From patchwork Fri Jun 9 01:52:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E27CC7EE23 for ; Fri, 9 Jun 2023 01:52:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A71FC8E0002; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A227E8E0001; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4BD8E0002; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7C6AC8E0001 for ; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 50FA280380 for ; Fri, 9 Jun 2023 01:52:25 +0000 (UTC) X-FDA: 80881534650.24.32A0133 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf10.hostedemail.com (Postfix) with ESMTP id 754DCC0015 for ; Fri, 9 Jun 2023 01:52:23 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PYctsQ9b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275543; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=juzIS613ypA/N0ju0hwO2HxwiqebYZUp5aJmms9/6GU3/npXi3gYP1xXXA8Ijmg/rJ3Dk/ 1iZRfqd5zzHSn+MBx2CeA4AJ6ie8dUem7f2qe+YDpon6yZeBt+Wy1IE/kPptWRl0iayDER 6PCf1nE3J8Pxpm/bO00GgbdHk50LN6w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PYctsQ9b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275543; a=rsa-sha256; cv=none; b=IZ66ELnkUnjDPBGxaQbbJBOSUYiHmlIrpsd5bAdwvBfnYDRI4PGOXSU6ZdPHbPHkjmuOO1 5mJZw240EXCIt9fxEGOaZBjK1dXxEeP6pzZkPg8BeVawUu/Sx5jZsEmdClAvfqi4J0t4T5 SAPuXCVoMURUA4cl2CvdDLsCNiFHN54= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-ba829e17aacso1384567276.0 for ; Thu, 08 Jun 2023 18:52:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275542; x=1688867542; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=PYctsQ9b4HrQorKzO7zX+fdJyjXUiN8i4W2q9bIpl7bYnCcy9KOZVUhjq8Som0kotl 8hN3DheRVe/kgHsmckj1HIHUS7QkhDrduKAtxjWrJ7uPd3Z1kPi0K/a74/drhVMrEv5N iyFn0O/U0PF+LJj7jEIWaYX3IwAMgIU4MQFfWd5rKkg5CQiQ4VOa9f2TnjccivpRjNpe MLfjrjWUH5cF/6qTPBYKfnjLb/mQJxgyL/APasFw6ccuLxdD/9qF0oXkbHPENouaIFKL go5KxwidvL12GJkGeTVlF7tb8Sm8kZvi4Yzd/FTjpbEMUsi8U+HBiLJXyA+0bfuLk/Ed ypDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275542; x=1688867542; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=V0g+6yk2bdfqBfFca9n8K/ZPRbY+RK8IgB4WXmTm+Cds3nmaFxq2lXmGxcVLFbqrzS JiXMZrDyL0+Pteo/ufj1lp85H99/K+0rL6vwbHpHfFcUvKURbT0ITDmhtwd1DX39K6Y8 Hi0QWB6jKExcsmpC9HkF4dYEjQ8jg87CyaXVoM3azxrCxlrqHa3eguDd7M/6p6e3wNXj LQrd8lgOwEWwoyV0ZYTnuPFtmQMCgjYdsfSShkCunP4GusOKFpJEBRPc4lpSbwE8U6t8 FMFY3I4W+G0mPqqHZjYfeHx8bG366Wa4MiiBjUDOGhwVzC0zOYreQuTam44LA6F8xZCk fnjg== X-Gm-Message-State: AC+VfDzb6WNxCwAO9Y0C+epJwcOqfQ4sOuSworXjLJsZ/lROPza6IaET B9lCKnUq/9UTaT6EK1G7Jv21+g== X-Google-Smtp-Source: ACHHUZ4s2iHF1O1eWsJO+RR6O7kpy/Z7YTwwVNBnE8hanvEwdCoqR2iged2HkuPKJixJaGakJ6Kn7w== X-Received: by 2002:a0d:cf86:0:b0:559:ed0a:96c4 with SMTP id r128-20020a0dcf86000000b00559ed0a96c4mr1168635ywd.44.1686275542179; Thu, 08 Jun 2023 18:52:22 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j81-20020a816e54000000b00569e7cbcd56sm300407ywc.69.2023.06.08.18.52.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:52:21 -0700 (PDT) Date: Thu, 8 Jun 2023 18:52:17 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 31/32] mm/swap: swap_vma_readahead() do the pte_offset_map() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 754DCC0015 X-Stat-Signature: 45n7r1gbpnwc5mmgqeja77xnxdm6u8kb X-Rspam-User: X-HE-Tag: 1686275543-579074 X-HE-Meta: U2FsdGVkX19N1G86UaswR3/Tlnv58Y8CZO2w5o1wJd4z0HSEbcwRcSJBljcDEqA9WimfnSX9156eFY8MzAdsbUYxrz9J8dVYRAiDVjPneJoij6PK1SrMIKs9UBeRvePi+Et+MuPan6hN5/zz+iJhF+03XAZ2qbstBs6Yekz6V/6qG/g/ZfawGPW/YP51diziiGasDcn6a52zdbokTkT8MmVwMzcofXrz54WdwHrboINjp9yhNMzAqyjGRzm9JCeGDFQFRKgLg1OkchVUSYLgCQmylqNj5QgwaYwyVuYErwApyJe9Rhl5Az3v2yrmIWOi9129HWLTaiuU1a+QDTVvcVoXZcFz6kqrGpV2xuA2YyM1w8PiumgZbgsRgaah8Izu8Ri4usjtAh5UCDC5qes2Cg3i7aExm5UnfuAEmppyJEX+9MhO+okG0x4s84b37ViZ7u5Na7cTP6SoPvv50HFhGziMezr07zwVXmvZHQ38qzVmYhZFO71jcZCDauryjj+hNB5gcV6MRM9WxRxi150u1toHsi50fXJlnSVB9hHt9bQl4yr2QcK8kb+vApv6rSbqq95Jec4cTfF+kobOTbc5L6g9ci117UJdEorjJ4srPoEorqdn4ein0g8iZGP02PZX3y3GRHQ35nwTaRu3y4EAnWHED9HpDmIdOHFltdsqHtP0qWIw/LOn2Sz8y3xRVAUDXqCy73Lx+wAkW+sALkZY8USFcpSscSjm41Xx2vMMD7udYP+TWwJ4q5nULivR5+E/GtsJlFxxo/o3FudWvx/oTdgnIuE7QVSv1NlvUhv/ZvE+IWx6bid6c4iB1fBw7qIPyMlPVX+YxFGYmrj3WQqOIO+in6uy6gW3qVAR5Ks2aQ4mOyvYKJ7yIkmx1pRw1zwlelM6L72FZ0T+50PBQu53yej3OKFRqofNR7n8/jTVKNVLsc3iFGAndAFFg653CkamQ+khgOvYEtqSYQfLiTA Fh3xu18W sT0xnuCt91OtNLh1QA8g/j5EQ5KMAOKQj9OIgjAHdQMMrQfjkL2bga0A0WVW4a/yEBX/tMT+opgTFjmfkLBsMvM2dpqpjO3/OUyjQ64azutbf0QJbCRznVFd/b9mRBNLZ1CkHUjAtlDAo+u1GALWASAjOfM6fheHr3YhPhu+kKS0eGpnmjeVJOwMMN3YF4cWbvTkS3CYH87jfdOHWHq1bIuT/gZGzEn+5SUab1Mh9Aa7k0Ew+CJAyaR+/WFJrMuWo2Dl6UUBOg8vnjL91HuUpsL1sWTvrV2AzNZWvLwHDBGOaLG5td5j8mwnKr2w170MV8GxkxMP23pa38iPaYNRvcCJwGjkZ+kAQJXj+pboRRU+i2RzplP1BxIX7dN+fDIUQeUPc/AqJ00kuFlXLwsegj97x0TxX13MJ5vG4+yK6yNaSkRPD0hih4zNV9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: swap_vma_readahead() has been proceeding in an unconventional way, its preliminary swap_ra_info() doing the pte_offset_map() and pte_unmap(), then relying on that pte pointer even after the pte_unmap() - in its CONFIG_64BIT case (I think !CONFIG_HIGHPTE was intended; whereas 32-bit copied ptes to stack while they were mapped, but had to limit how many). Though it would be difficult to construct a failing testcase, accessing page table after pte_unmap() will become bad practice, even on 64-bit: an rcu_read_unlock() in pte_unmap() will allow page table to be freed. Move relevant definitions from include/linux/swap.h to mm/swap_state.c, nothing else used them. Delete the CONFIG_64BIT distinction and buffer, delete all reference to ptes from swap_ra_info(), use pte_offset_map() repeatedly in swap_vma_readahead(), breaking from the loop if it fails. (Will the repeated "map" and "unmap" show up as a slowdown anywhere? If so, maybe modify __read_swap_cache_async() to do the pte_unmap() only when it does not find the page already in the swapcache.) Use ptep_get_lockless(), mainly for its READ_ONCE(). Correctly advance the address passed down to each call of __read__swap_cache_async(). Signed-off-by: Hugh Dickins Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 19 ------------------- mm/swap_state.c | 45 +++++++++++++++++++++++--------------------- 2 files changed, 24 insertions(+), 40 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 3c69cb653cb9..1b9f2d92fc10 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -337,25 +337,6 @@ struct swap_info_struct { */ }; -#ifdef CONFIG_64BIT -#define SWAP_RA_ORDER_CEILING 5 -#else -/* Avoid stack overflow, because we need to save part of page table */ -#define SWAP_RA_ORDER_CEILING 3 -#define SWAP_RA_PTE_CACHE_SIZE (1 << SWAP_RA_ORDER_CEILING) -#endif - -struct vma_swap_readahead { - unsigned short win; - unsigned short offset; - unsigned short nr_pte; -#ifdef CONFIG_64BIT - pte_t *ptes; -#else - pte_t ptes[SWAP_RA_PTE_CACHE_SIZE]; -#endif -}; - static inline swp_entry_t folio_swap_entry(struct folio *folio) { swp_entry_t entry = { .val = page_private(&folio->page) }; diff --git a/mm/swap_state.c b/mm/swap_state.c index b76a65ac28b3..a43b41975da2 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -698,6 +698,14 @@ void exit_swap_address_space(unsigned int type) swapper_spaces[type] = NULL; } +#define SWAP_RA_ORDER_CEILING 5 + +struct vma_swap_readahead { + unsigned short win; + unsigned short offset; + unsigned short nr_pte; +}; + static void swap_ra_info(struct vm_fault *vmf, struct vma_swap_readahead *ra_info) { @@ -705,11 +713,7 @@ static void swap_ra_info(struct vm_fault *vmf, unsigned long ra_val; unsigned long faddr, pfn, fpfn, lpfn, rpfn; unsigned long start, end; - pte_t *pte, *orig_pte; unsigned int max_win, hits, prev_win, win; -#ifndef CONFIG_64BIT - pte_t *tpte; -#endif max_win = 1 << min_t(unsigned int, READ_ONCE(page_cluster), SWAP_RA_ORDER_CEILING); @@ -728,12 +732,9 @@ static void swap_ra_info(struct vm_fault *vmf, max_win, prev_win); atomic_long_set(&vma->swap_readahead_info, SWAP_RA_VAL(faddr, win, 0)); - if (win == 1) return; - /* Copy the PTEs because the page table may be unmapped */ - orig_pte = pte = pte_offset_map(vmf->pmd, faddr); if (fpfn == pfn + 1) { lpfn = fpfn; rpfn = fpfn + win; @@ -753,15 +754,6 @@ static void swap_ra_info(struct vm_fault *vmf, ra_info->nr_pte = end - start; ra_info->offset = fpfn - start; - pte -= ra_info->offset; -#ifdef CONFIG_64BIT - ra_info->ptes = pte; -#else - tpte = ra_info->ptes; - for (pfn = start; pfn != end; pfn++) - *tpte++ = *pte++; -#endif - pte_unmap(orig_pte); } /** @@ -785,7 +777,8 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, struct swap_iocb *splug = NULL; struct vm_area_struct *vma = vmf->vma; struct page *page; - pte_t *pte, pentry; + pte_t *pte = NULL, pentry; + unsigned long addr; swp_entry_t entry; unsigned int i; bool page_allocated; @@ -797,17 +790,25 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, if (ra_info.win == 1) goto skip; + addr = vmf->address - (ra_info.offset * PAGE_SIZE); + blk_start_plug(&plug); - for (i = 0, pte = ra_info.ptes; i < ra_info.nr_pte; - i++, pte++) { - pentry = *pte; + for (i = 0; i < ra_info.nr_pte; i++, addr += PAGE_SIZE) { + if (!pte++) { + pte = pte_offset_map(vmf->pmd, addr); + if (!pte) + break; + } + pentry = ptep_get_lockless(pte); if (!is_swap_pte(pentry)) continue; entry = pte_to_swp_entry(pentry); if (unlikely(non_swap_entry(entry))) continue; + pte_unmap(pte); + pte = NULL; page = __read_swap_cache_async(entry, gfp_mask, vma, - vmf->address, &page_allocated); + addr, &page_allocated); if (!page) continue; if (page_allocated) { @@ -819,6 +820,8 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, } put_page(page); } + if (pte) + pte_unmap(pte); blk_finish_plug(&plug); swap_read_unplug(splug); lru_add_drain();