From patchwork Fri Aug 25 13:59:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13365800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9FBC7EE2C for ; Fri, 25 Aug 2023 13:59:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFDD42800BB; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EF422800C0; Fri, 25 Aug 2023 09:59:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF5692800C6; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4F2F32800C3 for ; Fri, 25 Aug 2023 09:59:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 288661606C9 for ; Fri, 25 Aug 2023 13:59:26 +0000 (UTC) X-FDA: 81162784332.11.BC378B4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 662F0120019 for ; Fri, 25 Aug 2023 13:59:24 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Vov6BRAe; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692971964; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qiLtYIqOEP7dbrRgh13/wEWojRMPASON5YTG2W4NUG4=; b=dk9gYzbhGtp2jx76oIIbRHuenadrVQ1qEwmRXmH+QcpwoRwOQDTeNfVB+5QP6DbbbFCfXA GyO/TfP5bZWKzZFGexuqgDr+kZMMAj0sbdm59juOWDkhNdZS8Lf2ZZGNFfy7/KNmQ5d5hk s+FYwRPTtlRTE4g7LkPOSHFCy1jMxCQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Vov6BRAe; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692971964; a=rsa-sha256; cv=none; b=jo0mycn3/pYMgcsFWxFQi105SYDcvf2Blh79XCtJl0zxfNPXdeSvhsCHG45uJbw4yi7Ldq XagqrUfNBaOe4uMGY7ce1nzPIsl5GbWXLJ9HZZ7+o5bDaXTVCF0vrlJx+csyuCsl/iRiS4 0dLrlX/OwB2mlk/TCwsBAVVooU1lz+8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=qiLtYIqOEP7dbrRgh13/wEWojRMPASON5YTG2W4NUG4=; b=Vov6BRAeQ0t1i7+WGecmjU3fDN fUuJdMgQOeCqUIbr8aX/RWKSWNCRs2f/C9GUnhBJkpiyFMhv6y7Ss7K/P0Bim6s/tYKRoqZIBkPjW EEE/fiFCL/M+0QmHws30ohZIKYlAttKwPZXVksChznGh65TGxsRlEMLhbsjw97IPpVcsYJkmo0fvU xqQbYeLVQMByAuIxFg7YUDyyTvsOo9VjrfcvXYTTKkPGaeAkVV6ZKo/owum2u3lbUrvJ4kD1ZIV5Q zdJ8twTTGkbiaZizJCgM9O1LQObWJVL+puCLaGvcn2p7+jbVJNyccKaSmUop2wJA0do1TzFRHAP8D ZMyIF9mg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qZXLN-00HTR3-RD; Fri, 25 Aug 2023 13:59:21 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [RFC PATCH 08/14] mm: use __page_cache_release() in folios_put() Date: Fri, 25 Aug 2023 14:59:12 +0100 Message-Id: <20230825135918.4164671-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230825135918.4164671-1-willy@infradead.org> References: <20230825135918.4164671-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 662F0120019 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: qbad9kqcshkaj4tjzc46618u17t8h1uk X-HE-Tag: 1692971964-943798 X-HE-Meta: U2FsdGVkX1+Vi00OAmgg7zIj6X9PQBOBDDlX3MxERLrv28hNktI30CoweQRu4JHE+lWa9a8c3Z0ebTNMhzcSFV55ZAUWCDGawYEmSzQ8RPH3gHj7rWaeFdygxDU7fzb1LSjMsRd7T3MVYwvkn97zjMqhAHy959myWs2bEfzxzji77oD4parncfrOBnb7xeuDLa0cni8U1y4iTEFjIX+abzU092eVRsw2pV4YNBH9g7/Bl995ZQ413KUgkVztAxRAEbWJSJ583zN01J73X5MkxO54ORsj+0SmeKSfCdCua9AaiVkq/4vSRFCZBbBWXV+nDV3G0xhpod3UQ6/aKXZvZenMZZV849mnbqgNjUq8F9Lj09mzwxfQmPbfZmm4XnIlhznSSFNnsTmblvnPb6uMYhQe32yXJgZdDlD3tVN0wPZ195pVdHvcFPzqXhVMoEDU9zHYrTJFjkaUHtLbP7HDRAiShSAdnPlcOPmFoQWBLp0GnzKDDEYjgp4+RDsgtSZs3xCS0IhOiB88DzuW7V22vTsJNHl/jzBz9jA+CTcUqRdaGyWClrxXLlPfImv5VaoS9/Q7R+Yi0dK8RC6DKOqzC3Mghy6Ds0hagu5qdoda/I+1rSnzkHI1c68K6np63j463lgxALjMww004dv80zwGXE0CE/9iC1//EOx4ife1EkaCCBI8pqBkiuUKW3JXItidjrGsQ8IWfLSR6H4g+PK9i6m5EWX+cp1PWJf77JWeUm/HAQTI1CPaXLGp27sJoqwlppDa5Qextm/RsWiMoogvwdRbitqZAhMnz+1fdINwN1wcLGeWE/oQAIxo+QjuzrZAMTf5098kNqrhhitIKlm6yk2U1VxuEz8YLHUoqxhD9QC38ntvYhVNjdM6e4531D9ugOYxGJpwlaNshm4UMSbmb7tE8YrE5Xy680JVp7+FbahtTRCWAocIHauTihp4R7FGRT/T8evCgd9SeBR+OOt L6uGKatf WA481pt3+lneQYd6KJrzhgBZabxD96cfCGxezwpm1GWMxyPaoxWZlHhvwvUPtRbGcvPyKW7xPHyYCKsN+FboK4EyZtEX6u+a1p2cSHm2RzlcxdMjIYRGJiVX7bw4uWBAprFFGlAh0A4dfnkr2vRnHB6l5PhJ2cEXEB4czaxNj58IOl97M9kv0VB6vWHgRoDPTnen3LCtZGSd3410cTV3HocCaMOwi/xjqAUsrlXEr+Ki0Na29CxSpF05HB3VImxDdMueElQ4ldbBqF78= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d0caeb1c1136..44d3e459d16f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1666,18 +1666,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index fffdb48cfbf9..21c2df0f7928 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -996,24 +1009,7 @@ void folios_put(struct folio_batch *folios) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio;