From patchwork Fri Dec 22 10:22:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13503165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37A4EC41535 for ; Fri, 22 Dec 2023 10:24:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2AFF6B0081; Fri, 22 Dec 2023 05:24:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADAB96B0082; Fri, 22 Dec 2023 05:24:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97ACD6B0085; Fri, 22 Dec 2023 05:24:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 890B66B0081 for ; Fri, 22 Dec 2023 05:24:13 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 465CD407CF for ; Fri, 22 Dec 2023 10:24:13 +0000 (UTC) X-FDA: 81594069186.26.97E4709 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf19.hostedemail.com (Postfix) with ESMTP id 698551A0011 for ; Fri, 22 Dec 2023 10:24:11 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EMI8DMvz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703240651; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=1W7Lck6wmP/Zy6pX5lSZ65UNfHwoj1wTm8LTGVplhh2Qbeh5px5rSU3b0MclcUZaF8vsm9 e1WgGdHVaSJUifAuvPoVDODmUBsGxJcVX+BVHcgQwFHzn9FauENe6Wox0uUHiG75iMJI9w eFLNbEfmR0McWjc5HGvF48DjalQTnME= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EMI8DMvz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703240651; a=rsa-sha256; cv=none; b=Q2iAV71gp19rmcPx+4H7RiZ1uVpEUFIAcVpQCxB3x65JmD1uW1vDq/Ggx/VnSAa/BUW34r He9MuoeG/3L/4y4acNWitDGEzn8Cm7+n8fdw3SI3XpcdsT+cHG36cYo4DZOcgX8oY/X11e od30zqUViKRbzQHpFHi+NYF7tfsx92w= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6d6d48354ccso1046410b3a.1 for ; Fri, 22 Dec 2023 02:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240649; x=1703845449; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=EMI8DMvzUzOzSrZE0K+nhTGUxYyZGaK+xHleqjWjHYTviRvkHjWxSRAYZ2V11UFot/ aR3q0mgEgcAeum6BYAOEhTleRiZFC/sMvGDiw1s7NWMp84LwdW5qihna3zzHwu3riBlj KlZhgkmKF83G5nLGBjkHZa8GrPfgDjae9J/f+jUJ6zf0KGL2G+sBcHlkauzvGkBvKjIh 1FU0wNQXZuVmZJppGZm7WkEPZLXl/CxxgMZP5tKe/4XCCQaFMRTm922yAs2E2BHeunGz /bHjIroodL0LIXnf1BfWCAo441v4nT/qJK1Y+mpAp/CBWVCOfOKn23+Ei3kuNNR+IwTs A7sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240649; x=1703845449; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=b6xvm1tFl75eASaCTyhovkvIQLAncamJMHhKpXmXQOvYpQ5+w3C3luFKjJYisDZ2Hh HPr1jeQfEvxse5Xe6sVj42d9c8enwFN8cQiWUZ4hqiMjU8yjjHI/XxRwqpyjKz25yvuT RtX0FngxDwLRnehUHNixwtaXyH81w20XahRlWjpkUZoXlkbV24Jh0IOoau5jLzpvTSFv 0AuOMWrpQ7No1DSVwTGpWgkQ31RgnsO5IVl2f+FdBWQW8/tP1XLWqZ9CZ8Vw/8HoOJx4 wShEDbkhEPTI6eh3b79VWvIcplVx3vGmC+SkyeN5lY2XR/KwFnYGSO812ZKyxJyDRjLc k/Kw== X-Gm-Message-State: AOJu0YzBDPQkieDII4iYPgtMViWNkXVIe7n6/lLQbDaTGO8pE+xbvnvV rQMwUKXgJKisktfsXrhwAAl2wjMqjz2G+qL8 X-Google-Smtp-Source: AGHT+IFqSFyTsdaDJbnz34Tju5Rl8kxf/zGRHLlyWWOVYHMpESFATB5jFyXEKy/Qqs+cjCDyhnGHDQ== X-Received: by 2002:aa7:8554:0:b0:6d8:bef3:4182 with SMTP id y20-20020aa78554000000b006d8bef34182mr721760pfn.1.1703240648841; Fri, 22 Dec 2023 02:24:08 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:08 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/3] mm, lru_gen: batch update counters on againg Date: Fri, 22 Dec 2023 18:22:53 +0800 Message-ID: <20231222102255.56993-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 698551A0011 X-Stat-Signature: w986ukd9afrfkohrhpjqxaj9imuqwyaa X-HE-Tag: 1703240651-69308 X-HE-Meta: U2FsdGVkX18DJBV/bAi3XZYeGU0bgaeXErFaCYTKNzbb9p4nduGfHPIbPQ2onju24sPcDSlY3m+gmmRe81SbyOwwKtdY/ySj3ZSqX3m633nx4ClEs2a2fHoMYgSRwC74aT7+WDmC/wY4NG1gDaoxx1dYWMnNIHuBF+IV+L/nXe/IsN8pUVBVy78pjbA/Yk6rNidnEvVIK28zZP44DXRQvyV1asBkEg0C5fct7x4UCSKooZ0aYQCLKtzcq1NHJ3Rky5w0m83FpPHAG7p3U8iVAF46P4ehj5tIju6bGGHX8jABIKWOHsITQ4QtOQeU35BxuuspPGyb2qdC22Kf1aRGez5F6uPimpOEIR3mgwleSVSc98zGZPgAnOays9J+NoWB0fmkDKAOK8ssrOJzyEJSyiSG9weXC2suGmLGp0ND4+lHlTzeGA9SwPyJfjvspKuL/PBKukeXUzJBMY5xZ3xVmxHmmQvZk/jgGBQ51G+uS6yZsCeGC7/rk2bNtjCPmQZnbxV5p1RhOih4ZGfjc2lX8omuOqvBR9fhSjnx6uuKBeVyrO/oYF2+rS/bXAVsdyzgYetWPeHmxBuRTMxoW5LNCd70JmU8zVS9wbM+uBPcqyuMka82Bm0Ub7zmEejLkptKesf01zJHDOmrl2Lb7SJsrFTfHADAbmzENcQawTe6kP5QDcf3e6WhIlkH4oTgGs0DSwqgmY8krCwjP9u4QPTEJHwNX2IpoWNV3dtUEL12WMcXLoMZ2wuQ7IGYMpdve1jxZsbxfDXLRTVrY8TVmV6kNk6m86iz/9beKL1X+BK5nshHd9vb0bhMYiUG/nxQ9t2TeN8iq0YVsLnkn5nrcVOritkw8LoMkvsTJpb9e+1RiIV8roN68JrWjT5f64s8MbjZmD/vP6vWF0cxSbCl2UFdWI8dEk+vcw/frxK8TxJRL7Pt5AGD9ucPopfSIbWR5b9sx9iBiObZVBGnydryADx xGYdwk1v BemuOtZGZU8pemZxOYmaDxMDbDahYXiBBanqUYaCTQ7CmOm9ICWzy7bk4Fm8pBL6yzL80naCc0ASdUqPEW045T2S1zca7BXQABbw3Dn3BVnT41cjA+bhKzIoolwhSIJxPGWmjP79mk6gkWpmJ/ALEbjCFFq92jYX0DFxVbLGgUIMGbQmSeT5jPyX8/wUBoazBvrk+29lWpPMvE5omc/3Vq8zHg9pzF9AkMEWzz8dIH6rquoVhAabYj96MpbyXJL4b2R2Gsa/E/cibpxHU5LY4IJS54A9PW6EFNqyPZHrjM56mOuG8x9UB4RW6r1KxdZ8WNv1eQhf2ZYLerKcBzeCRPaRRFhq2ynRd7Z2uVW+E9LtZqCaa48BqeCsN4remoyy6lryxDhrr2Wg00vQNjzm+sqtW8Chlgr31qjQPgiLcdEBjD0mDwtnRRsEt0QLqRGW3ljn8Xw7n0/NaXAK/Q1Ru8RaJyetzG552WpP2gRuaW1nrCFtINsMUgS+GISbBhJAQP7b5jXW+iISx8eIcAawSrraDssG3WgEpnijlWd1p9y/iGcxg805vPK8eGaA//jRNZNJ1XYAayQk9dqgQywT0+GxUpig+/8QM+zoqtDP8q83gD5VT6gOaLG1AeThkZaIvovt2cMpKwJ87JPmKhqeTnkjUy7KG3LzOzTe4ekD7v8DM8qY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.062660, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When lru_gen is aging, it will update mm counters page by page, which causes a higher overhead if age happens frequently or there are a lot of pages in one generation getting moved. Optimize this by doing the counter update in batch. Although most __mod_*_state has its own caches the overhead is still observable. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After: 44687.08 Ops/sec (+1.5%) Signed-off-by: Kairui Song --- mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b4ca3563bcf4..e3b4797b9729 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3095,9 +3095,47 @@ static int folio_update_gen(struct folio *folio, int gen) return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; } +/* + * Update LRU gen in batch for each lru_gen LRU list. The batch is limited to + * each gen / type / zone level LRU. Batch is applied after finished or aborted + * scanning one LRU list. + */ +struct gen_update_batch { + int delta[MAX_NR_GENS]; +}; + +static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zone, + struct gen_update_batch *batch) +{ + int gen; + int promoted = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + enum lru_list lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + + for (gen = 0; gen < MAX_NR_GENS; gen++) { + int delta = batch->delta[gen]; + + if (!delta) + continue; + + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + promoted += delta; + } + + if (promoted) { + __update_lru_size(lruvec, lru, zone, -promoted); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, promoted); + } +} + /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, + bool reclaiming, struct gen_update_batch *batch) { + int delta = folio_nr_pages(folio); int type = folio_is_file_lru(folio); struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3120,7 +3158,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai new_flags |= BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + batch->delta[old_gen] -= delta; + batch->delta[new_gen] += delta; return new_gen; } @@ -3663,6 +3702,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; int remaining = MAX_LRU_BATCH; + struct gen_update_batch batch = { }; struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3681,12 +3721,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - new_gen = folio_inc_gen(lruvec, folio, false); + new_gen = folio_inc_gen(lruvec, folio, false, &batch); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (!--remaining) + if (!--remaining) { + lru_gen_update_batch(lruvec, type, zone, &batch); return false; + } } + lru_gen_update_batch(lruvec, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4197,7 +4240,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) ******************************************************************************/ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, - int tier_idx) + int tier_idx, struct gen_update_batch *batch) { bool success; int gen = folio_lru_gen(folio); @@ -4239,7 +4282,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c if (tier > tier_idx || refs == BIT(LRU_REFS_WIDTH)) { int hist = lru_hist_from_seq(lrugen->min_seq[type]); - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], @@ -4249,7 +4292,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4257,7 +4300,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* waiting for writeback */ if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { - gen = folio_inc_gen(lruvec, folio, true); + gen = folio_inc_gen(lruvec, folio, true, batch); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4323,6 +4366,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, for (i = MAX_NR_ZONES; i > 0; i--) { LIST_HEAD(moved); int skipped_zone = 0; + struct gen_update_batch batch = { }; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; @@ -4337,7 +4381,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, scanned += delta; - if (sort_folio(lruvec, folio, sc, tier)) + if (sort_folio(lruvec, folio, sc, tier, &batch)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4357,6 +4401,8 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, skipped += skipped_zone; } + lru_gen_update_batch(lruvec, type, zone, &batch); + if (!remaining || isolated >= MIN_LRU_BATCH) break; }