From patchwork Thu Apr 13 10:40:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13210091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9716C77B61 for ; Thu, 13 Apr 2023 10:40:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47CED6B0072; Thu, 13 Apr 2023 06:40:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 404DF900004; Thu, 13 Apr 2023 06:40:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F468900003; Thu, 13 Apr 2023 06:40:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2249B6B0072 for ; Thu, 13 Apr 2023 06:40:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D3BC71A02BE for ; Thu, 13 Apr 2023 10:40:42 +0000 (UTC) X-FDA: 80676024324.17.D185203 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf23.hostedemail.com (Postfix) with ESMTP id E11B1140014 for ; Thu, 13 Apr 2023 10:40:39 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7DaELoYb; spf=pass (imf23.hostedemail.com: domain of 3Jtw3ZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Jtw3ZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681382440; a=rsa-sha256; cv=none; b=EDGQY7P6ixZn2mHDfGWifZCDiYXlamjK/S/DcSMoFVyeCm1WDK+rNQlzJ/a3UPe1JcmHe0 Q5hftvzvvSRNRbdgCmOD9JXvPDg/9A0D3SJKrxrrFsyNJ8TTV5EUsjKgz0A2JBTR0EvZA1 gsihLbT2kFtLmNjsepY0boCwOetQhCs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7DaELoYb; spf=pass (imf23.hostedemail.com: domain of 3Jtw3ZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3Jtw3ZAoKCOUfVZYfHOTLKNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681382440; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xGnocY4pIBEOyCy1J5p5K30k0wCvlbaawbxDP+ACQr8=; b=el5HEjUODKDTgjQY3FZintVlRiVRxTCXYIPy3xxngOrWj6CBmmlyt0CGE0lwUdeFqZ3pHs 2O26yH02vNhthEuVJKtdKwta5+YB6lrabFHOwz9wFqCNaV4K4pAcsv4o05rhsG+4xcvQb8 Nlhm/ZPJ9Lkeqs773O/PA/uzce8jqsY= Received: by mail-pg1-f201.google.com with SMTP id r78-20020a632b51000000b00513d1de5204so11884957pgr.15 for ; Thu, 13 Apr 2023 03:40:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1681382439; x=1683974439; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xGnocY4pIBEOyCy1J5p5K30k0wCvlbaawbxDP+ACQr8=; b=7DaELoYbwDJCrNtZlmfg9Wt1StcR4uFr/DcPD/2pe3cLhlNGHN0Vo/cQ7EdPJo7up0 8bmCI0JQEmCL566Vot7ySRXaEZ0Q9wHFt3Sh9NvXD+gSBVNkrXxQorzAPrPz3sL21Jvg UkLNapMN9bhRXj3lO4/bDPz8UGmnAz0jlApmptNFgfyGVpKdYVctpDwfppNvGRrl4WXx hDn5QBpGHzMAOlfKXhd4xxjuLNGoae6rpvwcD9uDypxIntJ4NI1CR6tegrXbWE9t7vA4 LHSI7L2dX5euMSDWQaeHCt4S6p/b4GKxns2a9UeAtjH7MQxy0QVqfkDsaHQMGS5XAPuo 907w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681382439; x=1683974439; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xGnocY4pIBEOyCy1J5p5K30k0wCvlbaawbxDP+ACQr8=; b=VJXwTp7jNe5GU32VkeOMP1+RLUqgkRg+2fM2zwwyfBFlosep3dks57cHdHrY5ATldg 502AANOhWuceqelACO9MUUJDQfVQfw+ognXKnL6I3X+RHdDk95ZPpGb8SOaUvOnkLGep YWHCFZcoA4w5MiogmwdvvxUTDAeHhoNFvFQkSmeJGQ7RdV8TwgyAgeV+MlPvgd3bNr2L SE52moORpsQ4mKxU1lG/ACJuOMiv1YRgA5kjvpceB5DrCC8uSSVei7kXJERdnkGPdOw5 rwr4bYFl7ECNPnG4Sopazya2qTVwGk0sMXkdsZHBpOniK1CD5SoDCLKINPgMO+HLwZCo gCww== X-Gm-Message-State: AAQBX9fT7QZ44v8JXonGrvNEVVeGJUpGFKWhACO9gGqfY8OHYiz5klx3 x3Nhf0lRkLxf6bjVZjULQCcYVD0lVvh5pF51 X-Google-Smtp-Source: AKy350a/CmtWydI2qt+GCIKlEW3m+RGe6OazzLUzw02C2znW4S1CHo0XCY6i4wYnjhyCBrNF8AH/yKwjvGLojZVt X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a65:51c7:0:b0:51a:f873:2645 with SMTP id i7-20020a6551c7000000b0051af8732645mr331971pgq.9.1681382438784; Thu, 13 Apr 2023 03:40:38 -0700 (PDT) Date: Thu, 13 Apr 2023 10:40:32 +0000 In-Reply-To: <20230413104034.1086717-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230413104034.1086717-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230413104034.1086717-2-yosryahmed@google.com> Subject: [PATCH v6 1/3] mm: vmscan: ignore non-LRU-based reclaim in memcg reclaim From: Yosry Ahmed To: Andrew Morton , Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko , Yu Zhao , Dave Chinner , Tim Chen Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Queue-Id: E11B1140014 X-Rspamd-Server: rspam01 X-Stat-Signature: o6wi375bxb6jr3qr6c1kuer6kdkp4b35 X-HE-Tag: 1681382439-198226 X-HE-Meta: U2FsdGVkX182DizDLetY9zqRoCTW4EdQaAIKCw61AGgvbFRcZ5lkz024V03idBMa0E6HAr6fWT6+Pg/1/Hb5vI/r4BNaBdO7Ndz8PT679Jiank559uv5SRDn1MJWkZOauHIXL4dkqS7rbjbd+dj8so4r5/6032zAdFkW7i3BooxJQsvWVQRwUfNZkUppqNeuZ685dDYR6JgF+07cZIxFCsK7KwfUAViryYcAXXYTapMva++jq38OeUuLdEus+WgFwBRVMgiTtmPiKvy1k0nVSwqlgy1/Hz/AM54jxPLHd2Bln5tyyas9HDKCc0Ulm8sD0t+Mru4Npu4YEk0AVQc3CrIc1KpvEy1dJ7aRZR2vikxhgLJ+YvhtSgbIaEEAeQ9QK/g8poqieZt2CUwmU35AimFKF4F3b1fwluQUvu6LWBoJ+S77fyaBz/VrWoKtY79TBhH5uFTsw6vv3WDlvLqb+KvQxvM4vEYgwyQ4hqC9d9a7Uan1FtlDvyKbk8usAurPT1GNMZWDCtP1IPDdp0VSwqL2UtmOUgHhAkY/uhb2odT8iRCeT1hpFMu4Fj+jP7l6ELF7UAhWSNjq4FdfmxSOzA/78P2HjfZ1XxnfeaG8oGcNX+Y4uKYdxFnO7lpaysPO+r2+BLRTjxDpeLR6qyXVDti8LDPWARry4bmXP5YMjM66II4SOd9bZMKlyIqdCG0g9vxYIxm6PYz5JpusJR3jrnnh8u2p2BCXRPH30xIhhmXwP5FF5tzenN0Lg6S/YeJdZAMASK2qcG8we38hP9IW7RjFAchi/4u/vom2NELQoMeG6HQipCGkEH0Y57QElEEHldazzx6rCfXzqyjSr4Vegzbsy7dPaFyLT3z4OWThvJrOoBHOe2U/ilkqhP6iM/hM2YjXK2OyyaK7QQlIegvS1SikuTq7E3k+M+1zPKueUssYZRiCi0OGkJgnjIXOxMXvpA5K4gX+zYP6z2b4gyA 1YNiOJoK 1YNWmxpqbIt5+wdKixzS+aVu35aNehY9iVnLPfP7Cf7sIiI40ToRN6IrLHjgoXVNHNX81/rg1haeVgES+NyRetW9f1OjzQdVMbFZeXN0ICiC9avO1fzGn+QEkJbtJQDrkVz6kzzv2gtX9cqiTe02OlEO661gikdMXRbKCZR5pOjeBCPHOoTJF2G5OL2A3t55n1U8wui9UJg9CpWcT5FPBgUa24utW1bQIfevW5nrIs1WYEL1pGtk1UpfAdKrtANAuCZUEIPsDziwihNmFZh/udRSWodL1kkBDzqE/UVXB8hzjSkPKWKTIAghs+GgSmM1DAVs3R6llLOK/d5ASdy2PngVaVSlXi04G7YJxMLPPeWygmPFU1LfeeOAnGRAbZm217Ivv4cBTA7hhOND5CQSTkplW1iNSKAGQ8SnrnpP8fU3TD9iN5ngU8caMqD6fpBEua/HYNHh+gDTfAx7ZFgc/UpDATNWf8fwMOCTNQSRO0q/ZTYqfxLPtJXZQzoHKmX5EmaBcnJ0ZlU/DqN13awQ5lvehDVv8OphdzajpM4YV/UUKoUC63RrN+YBsUje9cMNCw7EmFjGRPfutqAluwRlcxstYnKDE+VlgS1k7HEYOI8i9tQGGhna4Z3k3dyJgo2DHIwrv8sCi3rLk0sByW5fCPYcDEW4fLPTl6XP/pho/Lg4/WdKol7k4MRHY4Hh6tePYYCG7933yiofuVFBr/vxe8YC2x5uQPHN87T7P X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We keep track of different types of reclaimed pages through reclaim_state->reclaimed_slab, and we add them to the reported number of reclaimed pages. For non-memcg reclaim, this makes sense. For memcg reclaim, we have no clue if those pages are charged to the memcg under reclaim. Slab pages are shared by different memcgs, so a freed slab page may have only been partially charged to the memcg under reclaim. The same goes for clean file pages from pruned inodes (on highmem systems) or xfs buffer pages, there is no simple way to currently link them to the memcg under reclaim. Stop reporting those freed pages as reclaimed pages during memcg reclaim. This should make the return value of writing to memory.reclaim, and may help reduce unnecessary reclaim retries during memcg charging. Writing to memory.reclaim on the root memcg is considered as cgroup_reclaim(), but for this case we want to include any freed pages, so use the global_reclaim() check instead of !cgroup_reclaim(). Generally, this should make the return value of try_to_free_mem_cgroup_pages() more accurate. In some limited cases (e.g. freed a slab page that was mostly charged to the memcg under reclaim), the return value of try_to_free_mem_cgroup_pages() can be underestimated, but this should be fine. The freed pages will be uncharged anyway, and we can charge the memcg the next time around as we usually do memcg reclaim in a retry loop. Fixes: f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects instead of pages") Signed-off-by: Yosry Ahmed Acked-by: David Hildenbrand Acked-by: Michal Hocko --- mm/vmscan.c | 49 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..be657832be48 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -511,6 +511,46 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +/* + * flush_reclaim_state(): add pages reclaimed outside of LRU-based reclaim to + * scan_control->nr_reclaimed. + */ +static void flush_reclaim_state(struct scan_control *sc) +{ + /* + * Currently, reclaim_state->reclaimed includes three types of pages + * freed outside of vmscan: + * (1) Slab pages. + * (2) Clean file pages from pruned inodes (on highmem systems). + * (3) XFS freed buffer pages. + * + * For all of these cases, we cannot universally link the pages to a + * single memcg. For example, a memcg-aware shrinker can free one object + * charged to the target memcg, causing an entire page to be freed. + * If we count the entire page as reclaimed from the memcg, we end up + * overestimating the reclaimed amount (potentially under-reclaiming). + * + * Only count such pages for global reclaim to prevent under-reclaiming + * from the target memcg; preventing unnecessary retries during memcg + * charging and false positives from proactive reclaim. + * + * For uncommon cases where the freed pages were actually mostly + * charged to the target memcg, we end up underestimating the reclaimed + * amount. This should be fine. The freed pages will be uncharged + * anyway, even if they are not counted here properly, and we will be + * able to make forward progress in charging (which is usually in a + * retry loop). + * + * We can go one step further, and report the uncharged objcg pages in + * memcg reclaim, to make reporting more accurate and reduce + * underestimation, but it's probably not worth the complexity for now. + */ + if (current->reclaim_state && global_reclaim(sc)) { + sc->nr_reclaimed += current->reclaim_state->reclaimed; + current->reclaim_state->reclaimed = 0; + } +} + static long xchg_nr_deferred(struct shrinker *shrinker, struct shrink_control *sc) { @@ -5346,8 +5386,7 @@ static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) vmpressure(sc->gfp_mask, memcg, false, sc->nr_scanned - scanned, sc->nr_reclaimed - reclaimed); - sc->nr_reclaimed += current->reclaim_state->reclaimed_slab; - current->reclaim_state->reclaimed_slab = 0; + flush_reclaim_state(sc); return success ? MEMCG_LRU_YOUNG : 0; } @@ -6450,7 +6489,6 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) { - struct reclaim_state *reclaim_state = current->reclaim_state; unsigned long nr_reclaimed, nr_scanned; struct lruvec *target_lruvec; bool reclaimable = false; @@ -6472,10 +6510,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) shrink_node_memcgs(pgdat, sc); - if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab = 0; - } + flush_reclaim_state(sc); /* Record the subtree's reclaim efficiency */ if (!sc->proactive) From patchwork Thu Apr 13 10:40:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13210092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C397CC77B6E for ; Thu, 13 Apr 2023 10:40:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5690D900003; Thu, 13 Apr 2023 06:40:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EFD46B0075; Thu, 13 Apr 2023 06:40:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4075F900003; Thu, 13 Apr 2023 06:40:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 281E16B0074 for ; Thu, 13 Apr 2023 06:40:44 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E00BFC02FE for ; Thu, 13 Apr 2023 10:40:43 +0000 (UTC) X-FDA: 80676024366.08.246ECEA Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf01.hostedemail.com (Postfix) with ESMTP id C1BC540024 for ; Thu, 13 Apr 2023 10:40:41 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G81CdUEm; spf=pass (imf01.hostedemail.com: domain of 3KNw3ZAoKCOchXbahJQVNMPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3KNw3ZAoKCOchXbahJQVNMPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681382441; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I6Nt0ZH454lLp6ex7hg7kbI+ex/azfnI8thXAXjWVq8=; b=hkwAj5XGZ+Tki2hLFPXbWwPxP894ne7AhYfk0I1jeiHWKn2mDzTuiPsIC5sH3kETx1/ZsI IyX8jyTW6Q/LgcVOwuwURLRd14RF4O5KkkxiwGQgmBx52m3bFeUEqwIx2Jk0hHXxyzt6so WlA0GdbKPoMNgvWIn+Otd7CDNG2yOrY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G81CdUEm; spf=pass (imf01.hostedemail.com: domain of 3KNw3ZAoKCOchXbahJQVNMPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3KNw3ZAoKCOchXbahJQVNMPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681382441; a=rsa-sha256; cv=none; b=W3ov+KM9b4YQFex88qjtYCGdCj7sTrS21f9Vegtj20hJ7/eFLXcK3ZaUZImUShIA+ga1vE 3NTUCanaYK2o4jeRuEY27MUs2zGe1CfioFEV3dSoH1JmxvJ6CYKItDevuUQyhctqekj0Cb rzsoVKzZKPyykXLp87FrNA9cvUSKJmI= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-6323e92b084so1022499b3a.1 for ; Thu, 13 Apr 2023 03:40:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1681382440; x=1683974440; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=I6Nt0ZH454lLp6ex7hg7kbI+ex/azfnI8thXAXjWVq8=; b=G81CdUEmMWAR7s/uUh1o00+dki8yY8e8KOGduTMJ10Ih63YAx82qvcDq20SAZSx+g/ ueZ3Msj52pKD1lqBImwulf/tYkqr/iT+UOeW4fd3KBZv547W+MwD6xwGD67DaW0HpcOa cz9n6Ru3its9gZbaaSThe+VV0hYHOvVD9yp2j0yltd9IKliEUrjXqAr+OK7m03l2qytF xr+P3ML7/b+lezaU3DwMQukf+O+TPg5UG2i07gjQlaxwJnTBDRcJhvP64M/wiFfrpZzz h7ozlZBA7r8DTpqJhi25AoIblQhs+3bOV3uTp+JguZijEY3iI74Y975Ed/lYiOS4TnGE iVnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681382440; x=1683974440; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=I6Nt0ZH454lLp6ex7hg7kbI+ex/azfnI8thXAXjWVq8=; b=So5UM5kPz+tBFxswa5oI1l7v8Fw3aw/LOjwhY3wvhoJglasUQXwoEXSj72Z2cclL2r Q9aPWqFAVzHineRegcGPSTMo7Z7uWIjgt22Ky3PApaI8c0EO4nj9b+UCICPK6hbAE9H7 YZSb5z1aJMDSZmKPzHb4itBRJBJsUV1360Ke+3GGqtQbn1eWXCz9Ojw3qFFrLxCiwuB2 NOTjXEvDYhbjb3fJGNrTjRu23HGWmv0d8NgdSt9o9PYzIoHrwoOTwmzNP35Q9NXFj199 W4ZR7wYXpnelErkCMBaIXf0LK90A4HVfX2CUwpUvq5qhWnGveC0LILEVCyp59LDICr8N Mtjw== X-Gm-Message-State: AAQBX9dhSkji7OHw9C2UQo2uZJGezCdOgT8NZ2Uf0nHkZ+6U6oULxyZy vc21YuXKyyYaWbf7gEwNgpEz7TP4+UX6mrih X-Google-Smtp-Source: AKy350b4iSx5ozD7gi39VrWatA78FUtgnIqT2lDjsg7piXFtlDqLoC4lH3GhIMzW/GNCYSdmkQiW7wqZ38a9M/36 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2e1d:b0:63b:234e:d641 with SMTP id fc29-20020a056a002e1d00b0063b234ed641mr975833pfb.4.1681382440554; Thu, 13 Apr 2023 03:40:40 -0700 (PDT) Date: Thu, 13 Apr 2023 10:40:33 +0000 In-Reply-To: <20230413104034.1086717-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230413104034.1086717-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230413104034.1086717-3-yosryahmed@google.com> Subject: [PATCH v6 2/3] mm: vmscan: move set_task_reclaim_state() near flush_reclaim_state() From: Yosry Ahmed To: Andrew Morton , Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko , Yu Zhao , Dave Chinner , Tim Chen Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspamd-Queue-Id: C1BC540024 X-Stat-Signature: u7zdr6pehq5i7xgxfjspgqy3mmszb8sa X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1681382441-56822 X-HE-Meta: U2FsdGVkX1/aF52Ru/SB48G9dxS22BACr6DZUS6sDuRmMoF3aTNpQF1ObgWobx6D3r77/lLf80xXdrUvOaGvZSBwX2MXfBfJqysITJsgoEdtZoFr3ZGSWCvNdwpfQir4MKx0dO6LoU2fDBatn+f00gnalbkPIi7WPFTu02IRU2jIgDSV7h0frtnLaVX6svcbBlwd42GGAe4LSuwPkBCPwzdw6UO6SXg2Ij90CbBuAiL0Xdc2LmScFMR2MRWYk7Qi4K682SAUOKOAFCOiSSzEb8+PVQsOuHUHa+fTadDR73tYGg6ZKNfs0Idtx6ZlpdpAy3MohEK2aOnPiH9QHeabXgvMFSXVR1QS8E62EBaXiq0jeJyl4c+OxEvG/NqGdKuRLhuMJFEcMLE5fd/ABiB5XX14Xmn4hVWZN67+iPue4OjnqKl+sXrOKF9pesh+vIJnqx6nv5C2vU/hdsUgQbynnz6cnXsE2k+g6EfnheXP4SeQ0gu/25v9wENZL+Rn7k9SSPTS4qaSzimi6Hbvqfoh0viPUSl/hIqxz7DIIu/k+wfISwnMu0GDG8+f4dB8YQb/e7MlviliixVWv+BBGbs67KqzLvGiLAH6awatGGWtR0a950h0AQRDV6uAIY/TsiIQi9Pse9hIDI6tzeRVId4rMaK/tTDRIDgbTzekWwkT64QnbRi3aEQBzCaVpkl2YpVsf3F1tnZfFZ2A6F6Ug3GACWbyA2Xm788SdQrnZf1bEPz323Jd99NaX7lDBfgUdgdYW5O1++NHvi4G6FiP2Ct62Hgb+LPebWZE5TelENGoc12n1PxgsxufcVDWhU3mwTQfQRw7ORti/6S6XYnl1FQsi7EXzaGiBtgWh6U64wbSs4u+tN0K+QYC0ySXqNYNLKEg8S9a/BS5NunWB3E2HcPeGWPM2aAesIUMZE8LxrOu3JpMeh4NhPi3M8DKZRlnpMchl/XiQCQ5orK0rkbNWam /CDOsw2R IoIUhsEospxJcP9MBGm/00ayDVL11wEI/Rti8kiz7YpfJi6Kn/Obh1O8uPalXYSvLxbBC7R5j3Zon22q0WuzqYUDAbmjv/gMQISjCMotG2s7YPuuA+8dYVG5RAZYD5YlJImU9tqd5uQHIKJ376ANvSZygP3nbo+atcK0nex1cG5RDh08LMekWrZ8KU2cc4kxUfmuUU6fYyPu+XnB+DhtjlbPF2yZtf441QDdA8FOAX1OVNJIZ6jC1gAHw0RgGfDHH3cflKjI07BjA/Y/V1ITRK5EYjo9wqpNYYHITA7LsbvzX4ou8BLS51poPUxmaseQ5VuddIfh6/EFm/qsvuU70Hk14zY3alJ+6EeUgeaXJPhVxJzaYh252KYwXqijHXqoLPGRolMz1mYUwMyCYBkjrOEkqCypBiyoN1YcRtrvxUsTdnFHy4ypOllgFwOB1o9zaPHGtJFLGxNuXhx1VzXu2593zaJNYwmj0HFpg0x3Z5fz2aQaPtRiRU4xUIShJaLN4MFUdP7XdwFj+g6qe3R3Ssdii3hcAAFZPTCJDOwBFeVt27njXzH+yCTZxRfuVpwKmNVYUAd9DwmZsDau+SjwA97lGPUhK+gD+dG1MSJPDaVjEY5/UnEX8gO72uVGQpUX5iUvktDrdlMzqmlwEiFp6OgTXTx6qsZWhv5g+MyAJpYrhFjxmQrBvjNmQhOjUD6MQTBW2MJJq8ofF8dGU7R/zah0fveBAGJ1mFbbKvUV9zU9/gqDcFlE1mnsoFytNViQqzk6RGVBAYGHzn9Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move set_task_reclaim_state() near flush_reclaim_state() so that all helpers manipulating reclaim_state are in close proximity. Signed-off-by: Yosry Ahmed Acked-by: Michal Hocko --- mm/vmscan.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index be657832be48..cb7d5a17c2b2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -188,18 +188,6 @@ struct scan_control { */ int vm_swappiness = 60; -static void set_task_reclaim_state(struct task_struct *task, - struct reclaim_state *rs) -{ - /* Check for an overwrite */ - WARN_ON_ONCE(rs && task->reclaim_state); - - /* Check for the nulling of an already-nulled member */ - WARN_ON_ONCE(!rs && !task->reclaim_state); - - task->reclaim_state = rs; -} - LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); @@ -511,6 +499,18 @@ static bool writeback_throttling_sane(struct scan_control *sc) } #endif +static void set_task_reclaim_state(struct task_struct *task, + struct reclaim_state *rs) +{ + /* Check for an overwrite */ + WARN_ON_ONCE(rs && task->reclaim_state); + + /* Check for the nulling of an already-nulled member */ + WARN_ON_ONCE(!rs && !task->reclaim_state); + + task->reclaim_state = rs; +} + /* * flush_reclaim_state(): add pages reclaimed outside of LRU-based reclaim to * scan_control->nr_reclaimed. From patchwork Thu Apr 13 10:40:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13210093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B97FC77B76 for ; Thu, 13 Apr 2023 10:40:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25D97900004; Thu, 13 Apr 2023 06:40:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 146CD6B0075; Thu, 13 Apr 2023 06:40:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29DE900004; Thu, 13 Apr 2023 06:40:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E63616B0074 for ; Thu, 13 Apr 2023 06:40:45 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B0F7C40352 for ; Thu, 13 Apr 2023 10:40:45 +0000 (UTC) X-FDA: 80676024450.27.E704352 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id EC3A316000A for ; Thu, 13 Apr 2023 10:40:42 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tNgjpd2v; spf=pass (imf08.hostedemail.com: domain of 3Ktw3ZAoKCOkjZdcjLSXPORZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ktw3ZAoKCOkjZdcjLSXPORZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681382443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5OCAYSthWCELWDD+mYmXa9NvWHtbEH3UT/IFpudVmH4=; b=XZ3yQW3tMnl3B3V1vKEz5wd4nNWxOgFpFolFN8AD9fPMsCxIaZLtqyJEmGwFu9m4kidnyr TpHwD2GjFfOwTLuq9lS4p+ay1K0Sog8yAivNs/iF0vXAUo/epBPbg0h3kBdqa89fcgfSTj K2VBJ5HqjbbSGCi6Uf9nJF5jbtpdWMU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tNgjpd2v; spf=pass (imf08.hostedemail.com: domain of 3Ktw3ZAoKCOkjZdcjLSXPORZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ktw3ZAoKCOkjZdcjLSXPORZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681382443; a=rsa-sha256; cv=none; b=wZAoPd+myINrC8YJwLeLvIm+wPpvs2pGjjf/onpSZHL4SVqjtZq+0DTd1PGIceJc0C5/RL dlyonZ72jVe97+zBwfMRO3Mpd1Ul/bWAh7Ng4YdHPvAaayj2+jtHDNuduPIi7V9f9R5OQ8 DS6UMbtYF307KdxY+hF/GwTWUf2WNas= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-54fbb2ee579so18356907b3.14 for ; Thu, 13 Apr 2023 03:40:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1681382442; x=1683974442; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5OCAYSthWCELWDD+mYmXa9NvWHtbEH3UT/IFpudVmH4=; b=tNgjpd2vJ5sYdihexJ27wp/pJdtBdkGoDWlKPI+Qxip+uc94atiC4v8qSoBRSDpXnF 2H1eFR4AE43zDnf3vSAC6PM4Yzv5eOl4FnSZ87ySrNsQr00bC+1pP0q2NisEbs11XCfK wfmIAOuWTU7ukaHpLHwHLoguS2c0haEBvsc03OpVoC1wgoCMAvA/kO8UJucX9FKOKJPQ FUpRKC72wJ0M5aFsenyPyMH3rc7A+TUAdQgnGrgJs85bN23klJx4wsrqR1AZvj2NJaOn RSlnCkB73DeUMUXx2hJ8njcRovku/drtS2o23klZqkp9a4zr1WrXp8WQJA7orQxM0t6d M7ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681382442; x=1683974442; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5OCAYSthWCELWDD+mYmXa9NvWHtbEH3UT/IFpudVmH4=; b=G2SMtSWfBrl5DfON3Po4oIIlS76v3PtflfENQxQ3Z+b8NlxB+8gsolvvpO51ibOY1b kfYf5bOeAE5jcoH9Ff19A5NbTGwEqWZ7A4NAnsIVy7LU2YTNrX4HWJpd/WoU4p3GxqSh eYukyDZb+m9VrN4JGTyUITr9FlqRaospW2In2B1C4eStknZAl1VPq0QuN1VEYJHxya6B qDIxL+z2ExKvEHgXpK0oSATvKPXcmxnuTPTSSxVWYE7olri1cxH02lMFSWcBfXIYSjkn +46rbizHI0X/UwUtZOtiOgISjFGF10WD6VRJLAl1gJShdKw9KUnPvalX0tHQE5rHtZzb lyoQ== X-Gm-Message-State: AAQBX9cxUrAwwf6WKIV+V4XHUs9gjL8a9upU3AuOKAJib0OTrRL5Myek nM7ugOD1/zn6U/0yuu+td3a0No4kZZjqHOzy X-Google-Smtp-Source: AKy350ato23F52bhKncBfpzqlbDSBjiTYHoOr/++AUIP1InXFnH0R1Cd3N+S/780qX9+qoWZd0rLQvVMvNIOlXzG X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:dad2:0:b0:ac9:cb97:bd0e with SMTP id n201-20020a25dad2000000b00ac9cb97bd0emr904171ybf.5.1681382442136; Thu, 13 Apr 2023 03:40:42 -0700 (PDT) Date: Thu, 13 Apr 2023 10:40:34 +0000 In-Reply-To: <20230413104034.1086717-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230413104034.1086717-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.577.gac1e443424-goog Message-ID: <20230413104034.1086717-4-yosryahmed@google.com> Subject: [PATCH v6 3/3] mm: vmscan: refactor updating current->reclaim_state From: Yosry Ahmed To: Andrew Morton , Alexander Viro , "Darrick J. Wong" , Christoph Lameter , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Shakeel Butt , Michal Hocko , Yu Zhao , Dave Chinner , Tim Chen Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EC3A316000A X-Rspam-User: X-Stat-Signature: xumjmg1k833ppxruxwf6rpwq31iqija7 X-HE-Tag: 1681382442-570948 X-HE-Meta: U2FsdGVkX1/DUOiwgx0RYsDXIAZIa/gSK2dW1wMZKNmnOVQ1Ju2pmdd3RQb3mqTKKuNVz5HFZypo8WCF3qiL/eDxztVZqBa3cDcpdCOtLHsof9d5A8ORG4eLuT4IsT1Mehuy5ajT+GFmmF/tBVCsw70L8R9NHFuJNH3A996wQiYGv9j7PnG0Nac7GQ6vokanksOIVgHX0gPeG1c4PpVDn+b7stbK14SimVnaU8f2Lt0QA8z9r0Q/oA6kS7pF5CJP/odDZb8FcYRwHJUlrifzTmFgzayYjJyYuGfeIcx35S4ZMLvWYfmSkcz2p2XYmfpGJj10Gol2lDy/asug+82AihxmjTLL5p2uycJMgEGjjLxNI2EYRrE9EbfiaGkojiTVwQ+g310SAzAta20KF2ax88lmXK35hNZet9Cva+C3NUHsBY/tAI7e6Ar21st8IUhYZlaxll+eNSVw9uozwjXxXK5VkL+OfQuDVbeCv9eNb42Tv9IMPkZMpDJfLsI4k/8MKRvSJ5oNkIz8Onyhyhc8r0FBcmZGRqpQuzQHokqHaWG9Xliqf/YNCe3l1LHISJmRXmniH9T+qES8+VYJRT0chAnV2sEK0QugYn3/DTYoVR5tt1kgKB3WVR4poCROZ2BL9GiPeT9/eTF7hkzUuqgsqPCODxABaSAtxR3fXua1adeBue7/AfTRmA/xRDFW5Rcta7Io0A6vZNAobTvUiL2Z3XnYEQn3ckLDPa+OYp8+ohbohnH6WACByl2N5VfXkD+4xpTkg0GWEX6FmpM6ZQEdRId/PnQSJP7ly+zpmJUCPQI3xl+3UOApAmQi5cT3fR+NyWMJolVBykg9mQ+iWEQQSroGryCvhlhQgu7c8XMtd62a4IvOb6d/ZLp4Pl5G3JB+qME9jAzMeDvE8Q29djH7yOFIkEQijsL5A0QMOlrshbh1a3mlUhb3RHH38B8ChFjxfgOzubR/ze3kP50jbTV LKb2e+8Z okU5wGACo2KGYE/4S8eaCYD4MUQDWyC4jTovxhQ+ZHRZkqG3ltXW7x+5Gt0YxynSW2NXtAh02QnElWVdRMrTaExr6N6AGrd+cYqJCvCVnxG9x7FirF5lvXD0vOOqhxiQtoi18ssxIZwgioN6SmcuuNcytN6vHYt349H/ZKkrECxFHC80+XD0kPhRkeEkI6+Mgul2C9nTHsifdyKLu0ZYxK98b3KXgier0J530HYRTmsEOMm/aF8srnvoN77Ki6CUqJL9dLl0K1gzA4u54MloIkiKeH6PZ8TBQ9+VJlu1JsKn6BjDLifAFsp3zCVeLLpa17zQTQ/W7ZPb7FOKcLpmHxEhKVxb0wwFrzZxjboCASgeEryWb4FWNeqo+mOZEJjXrvvC34vwzz5tyYlDITKhEqyjWhYKVyZ5owmM3jxcyf4zFXdQr8Yt3el7pxMJtPY+JaP/OyJSBynuHGRcjiQv2VPY0tw7nD1PxHJLFdMsgZpY4Xv28QAnRxLSabO6B2YG0oWD39kaXGXgQH3ClCZE/53+7UeMvW8rFsRQwxyHmGvNDhftw7BFROnPfZMQFpVhDyUTBypDkcbOsqb0TrbntaGr63ToIWr0DeqWDoeIKCdLFSN0D+XXi3kwiwWZRKPjWVWQ4NCPA5XJ6ioNMpT81VQa1FuPNYFtZM1pwrb4NiqTFOKglDR7jr1l36B1G5pxtHnVUH5KLH7M8afJtmQ36Pi2Eg4hUsM7Fg51nbwQ8tXiyEmKsi8F1fecJJNj6u24y8q23e5C15qRnBN742RAtb0AgcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During reclaim, we keep track of pages reclaimed from other means than LRU-based reclaim through scan_control->reclaim_state->reclaimed_slab, which we stash a pointer to in current task_struct. However, we keep track of more than just reclaimed slab pages through this. We also use it for clean file pages dropped through pruned inodes, and xfs buffer pages freed. Rename reclaimed_slab to reclaimed, and add a helper function that wraps updating it through current, so that future changes to this logic are contained within include/linux/swap.h. Signed-off-by: Yosry Ahmed Acked-by: Michal Hocko --- fs/inode.c | 3 +-- fs/xfs/xfs_buf.c | 3 +-- include/linux/swap.h | 17 ++++++++++++++++- mm/slab.c | 3 +-- mm/slob.c | 6 ++---- mm/slub.c | 5 ++--- 6 files changed, 23 insertions(+), 14 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 4558dc2f1355..e60fcc41faf1 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -864,8 +864,7 @@ static enum lru_status inode_lru_isolate(struct list_head *item, __count_vm_events(KSWAPD_INODESTEAL, reap); else __count_vm_events(PGINODESTEAL, reap); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += reap; + mm_account_reclaimed_pages(reap); } iput(inode); spin_lock(lru_lock); diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 54c774af6e1c..15d1e5a7c2d3 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -286,8 +286,7 @@ xfs_buf_free_pages( if (bp->b_pages[i]) __free_page(bp->b_pages[i]); } - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += bp->b_page_count; + mm_account_reclaimed_pages(bp->b_page_count); if (bp->b_pages != bp->b_page_array) kmem_free(bp->b_pages); diff --git a/include/linux/swap.h b/include/linux/swap.h index 209a425739a9..e131ac155fb9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -153,13 +153,28 @@ union swap_header { * memory reclaim */ struct reclaim_state { - unsigned long reclaimed_slab; + /* pages reclaimed outside of LRU-based reclaim */ + unsigned long reclaimed; #ifdef CONFIG_LRU_GEN /* per-thread mm walk data */ struct lru_gen_mm_walk *mm_walk; #endif }; +/* + * mm_account_reclaimed_pages(): account reclaimed pages outside of LRU-based + * reclaim + * @pages: number of pages reclaimed + * + * If the current process is undergoing a reclaim operation, increment the + * number of reclaimed pages by @pages. + */ +static inline void mm_account_reclaimed_pages(unsigned long pages) +{ + if (current->reclaim_state) + current->reclaim_state->reclaimed += pages; +} + #ifdef __KERNEL__ struct address_space; diff --git a/mm/slab.c b/mm/slab.c index dabc2a671fc6..64bf1de817b2 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1392,8 +1392,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab) smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; + mm_account_reclaimed_pages(1 << order); unaccount_slab(slab, order, cachep); __free_pages(&folio->page, order); } diff --git a/mm/slob.c b/mm/slob.c index fe567fcfa3a3..79cc8680c973 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -61,7 +61,7 @@ #include #include -#include /* struct reclaim_state */ +#include /* mm_account_reclaimed_pages() */ #include #include #include @@ -211,9 +211,7 @@ static void slob_free_pages(void *b, int order) { struct page *sp = virt_to_page(b); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += 1 << order; - + mm_account_reclaimed_pages(1 << order); mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); __free_pages(sp, order); diff --git a/mm/slub.c b/mm/slub.c index 39327e98fce3..7aa30eef8235 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -11,7 +11,7 @@ */ #include -#include /* struct reclaim_state */ +#include /* mm_account_reclaimed_pages() */ #include #include #include @@ -2063,8 +2063,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) /* Make the mapping reset visible before clearing the flag */ smp_wmb(); __folio_clear_slab(folio); - if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += pages; + mm_account_reclaimed_pages(pages); unaccount_slab(slab, order, s); __free_pages(&folio->page, order); }