From patchwork Wed Sep 20 19:02:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13393227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26AF7C04FEB for ; Wed, 20 Sep 2023 19:03:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA21A6B01A0; Wed, 20 Sep 2023 15:03:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A52F26B01A1; Wed, 20 Sep 2023 15:03:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A45D6B01A2; Wed, 20 Sep 2023 15:03:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 780026B01A0 for ; Wed, 20 Sep 2023 15:03:23 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 59FB71402CC for ; Wed, 20 Sep 2023 19:03:23 +0000 (UTC) X-FDA: 81257899086.09.163981F Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf28.hostedemail.com (Postfix) with ESMTP id 6240FC000A for ; Wed, 20 Sep 2023 19:03:21 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Ts80XX9/"; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695236601; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zmtAWomE9UHhxu4UCSwc3QxDlD+DtTBiD3PMp93+3PE=; b=HYcgGkPrRZ0RLG9mxbeRT4stVwPJNXkPNHE7+a5hoJU1/lPfOznxOz8n6Ao/ew4wRyn+Tp MtdWGRjhSpV6+5Vh4/57POQgKm+cpvfO3fWLz9Z1r0vwuepLIbrIMQRdTo+2d6OGpKwFSk rEIp5VsmhR+sAYi0gwyUJTk48i1La1g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695236601; a=rsa-sha256; cv=none; b=lNBfBV0BFqMG6X25gIF2i0sYD5/QhVu+E1KnPmkSYgHWY84NO7S/Xxzb9r0NEa6OBSPLfV DTaDtzVc08gWFw2sWbJPMpiE0Ri8LNUr7ieqSxP9PmZtmTjoRJyWhmWXRpxSXWzKM7M3eR 26n9U3ikdLqsaK61iPdU4lIP4CBSCwg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="Ts80XX9/"; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-68fdd6011f2so91547b3a.3 for ; Wed, 20 Sep 2023 12:03:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695236600; x=1695841400; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=zmtAWomE9UHhxu4UCSwc3QxDlD+DtTBiD3PMp93+3PE=; b=Ts80XX9/KPQAiMk6FYwDX/0vy3XsHn87PBwrNBYZW/qtKMT4PX3A31Vh2Eta5F2hOS I5CfPKKrXRDKfuUKsxllBHtKkfmwMUPOj7Fb2nQsz3dAAqEXtbVca8OJWbQ8vfzGHNwG WAq3spQ0LgLBVm7d00MVte2ZVqKeC2j3LWUF9bpok/Rd/+dXscgWXl9424sXt3LzPbwo EmlTUXj1isfmEaws6x13xxVcv+LUdCc0nLJ+yfuwmYRVMCFkv0ieBny//db8iV+41sql 06CNE3UpI6MFjH83B8CiLvqSjPmJmZTDlpdcK66JmOa54LgK0X+5CVhThByXCbxUI4zi Aa9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695236600; x=1695841400; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zmtAWomE9UHhxu4UCSwc3QxDlD+DtTBiD3PMp93+3PE=; b=THf27LiBM9sIB5Z3m8inLD+93Hgd9CeDCP8opuwbR+jXZKQlma8bmVvKyexIqvRZzD 0yv2JKqDbArgKumjzu3ofOgAsJR1ddGTkq18ppbgb8DQaZzoWy7/045ojNSFOp0YjHdv ItF7Dn37ytsBlGkri13Ua1UsUTfvPJaCu7O9Ao6js2DcJEb3GhBinpVYGu+XUkFvhlZT XOnhhKkLovZ6xLVf6pLCNTsxhTv6Jo8PiflNR65wkKEzj7SrxuSiPjxw0PNOHOu1ABLr Ml6tAfDNsU2iEcS2FZaXi1f9leIdLK6JIUluCEBNShXDCKxrcdp7+SeUBhB6qhmXWNdH cL7A== X-Gm-Message-State: AOJu0Yw/h235ARr7DUghP4pVKAGO6HaTgzJJbOTBPBx3dBdkaZRGoJ6w FBapIKbfrMX3Af2mM244OG8r1c2tAV1SY3wJ X-Google-Smtp-Source: AGHT+IG0njOPD64WNQhILSI7Yy7mRF1jtyrvGzCcw3tGXClzEqYzeGMpn4pGWqrWE4Nnji19RDQnKw== X-Received: by 2002:a05:6a20:138c:b0:145:6857:457a with SMTP id hn12-20020a056a20138c00b001456857457amr3193200pzc.4.1695236599694; Wed, 20 Sep 2023 12:03:19 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([124.127.145.18]) by smtp.gmail.com with ESMTPSA id m5-20020aa78a05000000b006871fdde2c7sm423935pfa.110.2023.09.20.12.03.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 20 Sep 2023 12:03:19 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , Roman Gushchin , Johannes Weiner , Michal Hocko , Hugh Dickins , Nhat Pham , Yuanchu Xie , Kalesh Singh , Suren Baghdasaryan , "T . J . Mercier" , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH v3 5/6] mm, lru_gen: convert avg_total and avg_refaulted to atomic Date: Thu, 21 Sep 2023 03:02:43 +0800 Message-ID: <20230920190244.16839-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230920190244.16839-1-ryncsn@gmail.com> References: <20230920190244.16839-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: wk5cjgiuzf6jr3p78h5kdi851x41xtmx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6240FC000A X-Rspam-User: X-HE-Tag: 1695236601-147839 X-HE-Meta: U2FsdGVkX1+LTTXmM+XzAKZrtiJNLa4kmiGT9+IenpUsQJUhrp36E4C7/khm5Yt2TBEWhQitvdJQlNKAMedT7+Qn6oZp4Dr+NRr/5UGmID6PWhUw6ucnmyiYlSReQhiPC9l1yT7HPlwHZR61XhiLNR2DBjopem3MjsbFjc+9L1v1GTzRgpAdxUdP4eth8GhOHG93Bn0coc2uh1iim5vba+m627qjUuK8uFPHPNSOpMAfn1535K4svzNpVRtU3kJ649833dyZc4tLHz2JQmkBqTNsoOewOV59Z+gkynq3CscKvAn9+lGp2wB1bjsHfI1kzfLvcHLalk5qOuy5omWqPfOgS+I1VMafjW36sk1wLl0hWsxBIg3wZkUVYwCtfb3Lbg2AVyI4701w4YzpyOFoysrNr2ZIWmaqPrN45guZklie1eNpGUuxcwG4XGglPE6fUYbiFREYR7gslcFNEriJSaH1v7hppYoevDawIqcWeILGUQC/Yi4ji1MXmzEF8qtN9SxzPk0v/3k7o3kiEMnOlfWTM2pAes+VpgkWGjVjU5m8Z1b/sd0KkqNxniJXmeE4cXQff40N8KsdDMJRGSDM2TFUysyIHtxJaPW7RdYj7V0C7o8fNeUCt46jTT/K6zYW/zZwPC0PApjmNl1SpSZLsMHk5rIruRxOlDmakM1EhSwXTtW0Qmlb4QdivpQ7YGA48sSIekN5gKiJjIMLdYyhsoaKAOelBgUBGhjARk/WL+d/B6z9XhsK8FWmbfS7UfGwtmRz8wLkkfjhR7zZLeGwUqD4b6Gq3ouC+xxuE+ArRtiNGXU7a99jlM/ao6jAGnXqIeJ/WMuAiye9mlgMO4XzJhvHaVsIHxDou9kWYf7PYZIPfRJRic8s2X6oBzNeiZT2+4b87NXdaRuiw74+TPF/ZHNxdTDNGNycrQYR9BNr5oPfXLVoKG8E6zknZQPHWQvCp+dEqpDR9Y3XOYNth0b yb1DJZpC V5ceJQmHi3u8K1E/KEIBmLmFTnuO0BJq8FymRfYohefxvuRUNnJBYrKam86V6vj5f8ZKV2G8Acp/UTCNihHrDe1EdBuOvBGGmObpNtFtyUcrnSr6rhrpAW/lkJU9WTB1Gabd/h0wfy845v4nNuYvV+hv7Jr8jucBEyrQP8+VpROnFzoO4pj5JK+rboRBGfI9Wb9KGGq3MKqU2S/0W0JFolv4vop5dDdAo7ioFTyHvN5mQiHzutuKN5mGvGZDyGb5n3arj5TyvQOlxH0WSleSg/bQEqoN+bUNLciR0aipCXL91eGXRfKgRrGiAWmBCpBRcifGejJbKGCdhZBuJPmnoih513CPjzc+/uSJ4WkJtW+XZASnlwbtfT0si+RWVweQnsH/ugUtyzydY/lW9iaQ6j+pTIm7uJFXJBySI3gj6JEtNGER/oC/PWLqTUNAilta0GwGpntvD9Vl4LACQwgdm4W7R5HqluqqxUgk3hm0bvmcaR1Tgrxs8yxySCzDXzObcJqBCFQKeBtMuslpiYixQT9kJAlQf/Ru/EtoQiKDoY457U4nKH4/XxzDGgDuMo8AD2OGqmom/h4uycK9lujGs4yWxRHPJmwv5L6PQjOnfyDlA4Dg97jfh93W4Jp5iXjEjx8dqIGjb5J1/z3Tat3CmykwLeeOS4JrQrXWKAHfqybFCBYs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001575, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song No feature change, prepare for later patch. Signed-off-by: Kairui Song --- include/linux/mmzone.h | 4 ++-- mm/vmscan.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 4106fbc5b4b3..d944987b67d3 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -425,9 +425,9 @@ struct lru_gen_folio { /* the multi-gen LRU sizes, eventually consistent */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the exponential moving average of refaulted */ - unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; + atomic_long_t avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; /* the exponential moving average of evicted+protected */ - unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS]; + atomic_long_t avg_total[ANON_AND_FILE][MAX_NR_TIERS]; /* the first tier doesn't need protection, hence the minus one */ unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS - 1]; /* can be modified without holding the LRU lock */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 3f4de75e5186..82acc1934c86 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3705,9 +3705,9 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, struct lru_gen_folio *lrugen = &lruvec->lrugen; int hist = lru_hist_from_seq(lrugen->min_seq[type]); - pos->refaulted = lrugen->avg_refaulted[type][tier] + + pos->refaulted = atomic_long_read(&lrugen->avg_refaulted[type][tier]) + atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + + pos->total = atomic_long_read(&lrugen->avg_total[type][tier]) + atomic_long_read(&lrugen->evicted[hist][type][tier]); if (tier) pos->total += lrugen->protected[hist][type][tier - 1]; @@ -3732,15 +3732,15 @@ static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) if (carryover) { unsigned long sum; - sum = lrugen->avg_refaulted[type][tier] + + sum = atomic_long_read(&lrugen->avg_refaulted[type][tier]) + atomic_long_read(&lrugen->refaulted[hist][type][tier]); - WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); + atomic_long_set(&lrugen->avg_refaulted[type][tier], sum / 2); - sum = lrugen->avg_total[type][tier] + + sum = atomic_long_read(&lrugen->avg_total[type][tier]) + atomic_long_read(&lrugen->evicted[hist][type][tier]); if (tier) sum += lrugen->protected[hist][type][tier - 1]; - WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); + atomic_long_set(&lrugen->avg_total[type][tier], sum / 2); } if (clear) { @@ -5885,8 +5885,8 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec, if (seq == max_seq) { s = "RT "; - n[0] = READ_ONCE(lrugen->avg_refaulted[type][tier]); - n[1] = READ_ONCE(lrugen->avg_total[type][tier]); + n[0] = atomic_long_read(&lrugen->avg_refaulted[type][tier]); + n[1] = atomic_long_read(&lrugen->avg_total[type][tier]); } else if (seq == min_seq[type] || NR_HIST_GENS > 1) { s = "rep"; n[0] = atomic_long_read(&lrugen->refaulted[hist][type][tier]);