From patchwork Fri Dec 22 10:22:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13503165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37A4EC41535 for ; Fri, 22 Dec 2023 10:24:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2AFF6B0081; Fri, 22 Dec 2023 05:24:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADAB96B0082; Fri, 22 Dec 2023 05:24:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97ACD6B0085; Fri, 22 Dec 2023 05:24:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 890B66B0081 for ; Fri, 22 Dec 2023 05:24:13 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 465CD407CF for ; Fri, 22 Dec 2023 10:24:13 +0000 (UTC) X-FDA: 81594069186.26.97E4709 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf19.hostedemail.com (Postfix) with ESMTP id 698551A0011 for ; Fri, 22 Dec 2023 10:24:11 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EMI8DMvz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703240651; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=1W7Lck6wmP/Zy6pX5lSZ65UNfHwoj1wTm8LTGVplhh2Qbeh5px5rSU3b0MclcUZaF8vsm9 e1WgGdHVaSJUifAuvPoVDODmUBsGxJcVX+BVHcgQwFHzn9FauENe6Wox0uUHiG75iMJI9w eFLNbEfmR0McWjc5HGvF48DjalQTnME= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EMI8DMvz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703240651; a=rsa-sha256; cv=none; b=Q2iAV71gp19rmcPx+4H7RiZ1uVpEUFIAcVpQCxB3x65JmD1uW1vDq/Ggx/VnSAa/BUW34r He9MuoeG/3L/4y4acNWitDGEzn8Cm7+n8fdw3SI3XpcdsT+cHG36cYo4DZOcgX8oY/X11e od30zqUViKRbzQHpFHi+NYF7tfsx92w= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6d6d48354ccso1046410b3a.1 for ; Fri, 22 Dec 2023 02:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240649; x=1703845449; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=EMI8DMvzUzOzSrZE0K+nhTGUxYyZGaK+xHleqjWjHYTviRvkHjWxSRAYZ2V11UFot/ aR3q0mgEgcAeum6BYAOEhTleRiZFC/sMvGDiw1s7NWMp84LwdW5qihna3zzHwu3riBlj KlZhgkmKF83G5nLGBjkHZa8GrPfgDjae9J/f+jUJ6zf0KGL2G+sBcHlkauzvGkBvKjIh 1FU0wNQXZuVmZJppGZm7WkEPZLXl/CxxgMZP5tKe/4XCCQaFMRTm922yAs2E2BHeunGz /bHjIroodL0LIXnf1BfWCAo441v4nT/qJK1Y+mpAp/CBWVCOfOKn23+Ei3kuNNR+IwTs A7sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240649; x=1703845449; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=b6xvm1tFl75eASaCTyhovkvIQLAncamJMHhKpXmXQOvYpQ5+w3C3luFKjJYisDZ2Hh HPr1jeQfEvxse5Xe6sVj42d9c8enwFN8cQiWUZ4hqiMjU8yjjHI/XxRwqpyjKz25yvuT RtX0FngxDwLRnehUHNixwtaXyH81w20XahRlWjpkUZoXlkbV24Jh0IOoau5jLzpvTSFv 0AuOMWrpQ7No1DSVwTGpWgkQ31RgnsO5IVl2f+FdBWQW8/tP1XLWqZ9CZ8Vw/8HoOJx4 wShEDbkhEPTI6eh3b79VWvIcplVx3vGmC+SkyeN5lY2XR/KwFnYGSO812ZKyxJyDRjLc k/Kw== X-Gm-Message-State: AOJu0YzBDPQkieDII4iYPgtMViWNkXVIe7n6/lLQbDaTGO8pE+xbvnvV rQMwUKXgJKisktfsXrhwAAl2wjMqjz2G+qL8 X-Google-Smtp-Source: AGHT+IFqSFyTsdaDJbnz34Tju5Rl8kxf/zGRHLlyWWOVYHMpESFATB5jFyXEKy/Qqs+cjCDyhnGHDQ== X-Received: by 2002:aa7:8554:0:b0:6d8:bef3:4182 with SMTP id y20-20020aa78554000000b006d8bef34182mr721760pfn.1.1703240648841; Fri, 22 Dec 2023 02:24:08 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:08 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/3] mm, lru_gen: batch update counters on againg Date: Fri, 22 Dec 2023 18:22:53 +0800 Message-ID: <20231222102255.56993-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 698551A0011 X-Stat-Signature: w986ukd9afrfkohrhpjqxaj9imuqwyaa X-HE-Tag: 1703240651-69308 X-HE-Meta: U2FsdGVkX18DJBV/bAi3XZYeGU0bgaeXErFaCYTKNzbb9p4nduGfHPIbPQ2onju24sPcDSlY3m+gmmRe81SbyOwwKtdY/ySj3ZSqX3m633nx4ClEs2a2fHoMYgSRwC74aT7+WDmC/wY4NG1gDaoxx1dYWMnNIHuBF+IV+L/nXe/IsN8pUVBVy78pjbA/Yk6rNidnEvVIK28zZP44DXRQvyV1asBkEg0C5fct7x4UCSKooZ0aYQCLKtzcq1NHJ3Rky5w0m83FpPHAG7p3U8iVAF46P4ehj5tIju6bGGHX8jABIKWOHsITQ4QtOQeU35BxuuspPGyb2qdC22Kf1aRGez5F6uPimpOEIR3mgwleSVSc98zGZPgAnOays9J+NoWB0fmkDKAOK8ssrOJzyEJSyiSG9weXC2suGmLGp0ND4+lHlTzeGA9SwPyJfjvspKuL/PBKukeXUzJBMY5xZ3xVmxHmmQvZk/jgGBQ51G+uS6yZsCeGC7/rk2bNtjCPmQZnbxV5p1RhOih4ZGfjc2lX8omuOqvBR9fhSjnx6uuKBeVyrO/oYF2+rS/bXAVsdyzgYetWPeHmxBuRTMxoW5LNCd70JmU8zVS9wbM+uBPcqyuMka82Bm0Ub7zmEejLkptKesf01zJHDOmrl2Lb7SJsrFTfHADAbmzENcQawTe6kP5QDcf3e6WhIlkH4oTgGs0DSwqgmY8krCwjP9u4QPTEJHwNX2IpoWNV3dtUEL12WMcXLoMZ2wuQ7IGYMpdve1jxZsbxfDXLRTVrY8TVmV6kNk6m86iz/9beKL1X+BK5nshHd9vb0bhMYiUG/nxQ9t2TeN8iq0YVsLnkn5nrcVOritkw8LoMkvsTJpb9e+1RiIV8roN68JrWjT5f64s8MbjZmD/vP6vWF0cxSbCl2UFdWI8dEk+vcw/frxK8TxJRL7Pt5AGD9ucPopfSIbWR5b9sx9iBiObZVBGnydryADx xGYdwk1v BemuOtZGZU8pemZxOYmaDxMDbDahYXiBBanqUYaCTQ7CmOm9ICWzy7bk4Fm8pBL6yzL80naCc0ASdUqPEW045T2S1zca7BXQABbw3Dn3BVnT41cjA+bhKzIoolwhSIJxPGWmjP79mk6gkWpmJ/ALEbjCFFq92jYX0DFxVbLGgUIMGbQmSeT5jPyX8/wUBoazBvrk+29lWpPMvE5omc/3Vq8zHg9pzF9AkMEWzz8dIH6rquoVhAabYj96MpbyXJL4b2R2Gsa/E/cibpxHU5LY4IJS54A9PW6EFNqyPZHrjM56mOuG8x9UB4RW6r1KxdZ8WNv1eQhf2ZYLerKcBzeCRPaRRFhq2ynRd7Z2uVW+E9LtZqCaa48BqeCsN4remoyy6lryxDhrr2Wg00vQNjzm+sqtW8Chlgr31qjQPgiLcdEBjD0mDwtnRRsEt0QLqRGW3ljn8Xw7n0/NaXAK/Q1Ru8RaJyetzG552WpP2gRuaW1nrCFtINsMUgS+GISbBhJAQP7b5jXW+iISx8eIcAawSrraDssG3WgEpnijlWd1p9y/iGcxg805vPK8eGaA//jRNZNJ1XYAayQk9dqgQywT0+GxUpig+/8QM+zoqtDP8q83gD5VT6gOaLG1AeThkZaIvovt2cMpKwJ87JPmKhqeTnkjUy7KG3LzOzTe4ekD7v8DM8qY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.062660, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When lru_gen is aging, it will update mm counters page by page, which causes a higher overhead if age happens frequently or there are a lot of pages in one generation getting moved. Optimize this by doing the counter update in batch. Although most __mod_*_state has its own caches the overhead is still observable. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After: 44687.08 Ops/sec (+1.5%) Signed-off-by: Kairui Song --- mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b4ca3563bcf4..e3b4797b9729 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3095,9 +3095,47 @@ static int folio_update_gen(struct folio *folio, int gen) return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; } +/* + * Update LRU gen in batch for each lru_gen LRU list. The batch is limited to + * each gen / type / zone level LRU. Batch is applied after finished or aborted + * scanning one LRU list. + */ +struct gen_update_batch { + int delta[MAX_NR_GENS]; +}; + +static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zone, + struct gen_update_batch *batch) +{ + int gen; + int promoted = 0; + struct lru_gen_folio *lrugen = &lruvec->lrugen; + enum lru_list lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + + for (gen = 0; gen < MAX_NR_GENS; gen++) { + int delta = batch->delta[gen]; + + if (!delta) + continue; + + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + promoted += delta; + } + + if (promoted) { + __update_lru_size(lruvec, lru, zone, -promoted); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, promoted); + } +} + /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclaiming) +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, + bool reclaiming, struct gen_update_batch *batch) { + int delta = folio_nr_pages(folio); int type = folio_is_file_lru(folio); struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3120,7 +3158,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool reclai new_flags |= BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + batch->delta[old_gen] -= delta; + batch->delta[new_gen] += delta; return new_gen; } @@ -3663,6 +3702,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { int zone; int remaining = MAX_LRU_BATCH; + struct gen_update_batch batch = { }; struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); @@ -3681,12 +3721,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - new_gen = folio_inc_gen(lruvec, folio, false); + new_gen = folio_inc_gen(lruvec, folio, false, &batch); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (!--remaining) + if (!--remaining) { + lru_gen_update_batch(lruvec, type, zone, &batch); return false; + } } + lru_gen_update_batch(lruvec, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4197,7 +4240,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) ******************************************************************************/ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, - int tier_idx) + int tier_idx, struct gen_update_batch *batch) { bool success; int gen = folio_lru_gen(folio); @@ -4239,7 +4282,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c if (tier > tier_idx || refs == BIT(LRU_REFS_WIDTH)) { int hist = lru_hist_from_seq(lrugen->min_seq[type]); - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], @@ -4249,7 +4292,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { - gen = folio_inc_gen(lruvec, folio, false); + gen = folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4257,7 +4300,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* waiting for writeback */ if (folio_test_locked(folio) || folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { - gen = folio_inc_gen(lruvec, folio, true); + gen = folio_inc_gen(lruvec, folio, true, batch); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4323,6 +4366,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, for (i = MAX_NR_ZONES; i > 0; i--) { LIST_HEAD(moved); int skipped_zone = 0; + struct gen_update_batch batch = { }; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; @@ -4337,7 +4381,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, scanned += delta; - if (sort_folio(lruvec, folio, sc, tier)) + if (sort_folio(lruvec, folio, sc, tier, &batch)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4357,6 +4401,8 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, skipped += skipped_zone; } + lru_gen_update_batch(lruvec, type, zone, &batch); + if (!remaining || isolated >= MIN_LRU_BATCH) break; } From patchwork Fri Dec 22 10:22:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13503166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B014C4706C for ; Fri, 22 Dec 2023 10:24:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2533B6B0082; Fri, 22 Dec 2023 05:24:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 202796B0085; Fri, 22 Dec 2023 05:24:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A3576B0087; Fri, 22 Dec 2023 05:24:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E7E1D6B0082 for ; Fri, 22 Dec 2023 05:24:15 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C2F5E40DF7 for ; Fri, 22 Dec 2023 10:24:15 +0000 (UTC) X-FDA: 81594069270.17.4D9965F Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) by imf22.hostedemail.com (Postfix) with ESMTP id DA167C0006 for ; Fri, 22 Dec 2023 10:24:13 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BmVSWgPN; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.160.48 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703240653; a=rsa-sha256; cv=none; b=7an0guRZ1mxUcMIwzDwTHRU8fdWgEqj+8E8Vqr6hkJ0IMPT63ag60zOu+YZiTVcEjIG5JM OxTDy+/+TcRYaVA26JF5AWhBnue0nj2yEzpypuZPnhBuYMwEArDlkj3XHl76c/LWKeJu7O pwzC8Bai6b/Q0Xr6cHoEq8foO3yty0I= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BmVSWgPN; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.160.48 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703240653; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gsAcvd86KuP2ISw06xLpP7WDqEO9VY9n54CovJHHIls=; b=HGAErM74X/gABq5ZKXUiJsYgCvbULsqttrIKMFMr2/sKMn/gtPLjqTheAviRcVqB3UK1Iq 8mksHgxIpzG4r5sh6iNk2QwzRCQkfAvAAZ5ie6glnJEuxe/mG3BQhjE/ZFdFd8DYWBhCq9 5vAwOfc0RsUSw740NBjLAE4b0SuFU/c= Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-203fed05a31so1116478fac.0 for ; Fri, 22 Dec 2023 02:24:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240652; x=1703845452; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=gsAcvd86KuP2ISw06xLpP7WDqEO9VY9n54CovJHHIls=; b=BmVSWgPN0/DEW25nrzZj6sLv/kR8haMwoWJFMTZTSWByA9ndmX+3HEyzaNeg7NuZac t5MZtcIvvo4Vtzbx78m79IdUxpISCJBt3bRtCqF6sCfxHDaiepOhln18WMlWVWo8gmMw 5bnqvKsodCaHiNMtmmJSGsk6nuQ/cPzGg76GHM9cSTQxxx0VxAAlrNI9Os9UiyM22VlK +NAL5XaG4JjC9gxKb+tbk1RSlDk9snUvyTG84zoaAN7tdMQ6UyVj9jPJw51EnTDLxZGb g7et9VP0pKW8BhppjodxIRW+dbzd36TvwYlkrjCjFHBhkNY/TL1b3ghJGlTY/Qcz4rSU 9bsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240652; x=1703845452; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gsAcvd86KuP2ISw06xLpP7WDqEO9VY9n54CovJHHIls=; b=npc7MshweZNjqb2g+v0gH9YCKCrQjktEgtSuOvt8WeGVWjsAy9cqxqTZqwZqtQ1FkS mP4zFHIYZXNJZ9Y5l7L1WWkcHguP7G/T0Hbk6oisf/jTAfVujbiBq0sJSXBXAwLMDuW5 +um1Xi+M9Ov0f2F1VElgNx2iphRv0atLqQb0AIC+nA1BgXqLdASlfQfkyd4nnxmZBAVf 8EMJGqW1014q55/XUT9SB/qT1uTd9RMClh4yM+Vcm9oGvzc0vTeXdzZ8kR+8Iaq3Zgbo 3Sg+nHabNw98qkx0JmzfT59Kjm6kMvIqcn+nY4NowI4K2PwK7LakZPWxXvRULGo0ycY/ vzYw== X-Gm-Message-State: AOJu0YxVMN+A/uYJYlD3AE/lGdoiiGLVkgUa7KyIIELVvjQV7MYqGKWu /u2itlW+5uXTaD+X4GG9sTwnaHs+lefXesnN X-Google-Smtp-Source: AGHT+IFM08JVkja56Id51LMPVYml7/oIhHyVGlp94LC9R8zx3T0/LUwfVjdY7Ob42uuEZ9jE1hgJMg== X-Received: by 2002:a05:6870:ce0b:b0:204:5455:c44b with SMTP id xc11-20020a056870ce0b00b002045455c44bmr41106oab.107.1703240651957; Fri, 22 Dec 2023 02:24:11 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:11 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 2/3] mm, lru_gen: move pages in bulk when aging Date: Fri, 22 Dec 2023 18:22:54 +0800 Message-ID: <20231222102255.56993-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DA167C0006 X-Stat-Signature: add4sq7r47i69dyt56qyr3obtfieb9zs X-Rspam-User: X-HE-Tag: 1703240653-600732 X-HE-Meta: U2FsdGVkX19KbEj1ncYdPgcYl1vPCn1/O4C81/WuMg61BMOVq/3YRmga9iy2KPSfWmtJqGWgWKHV4gN4tQFTydpZc9ZUFBUlKp9AwjOtOyfcyTXcZWZngpZsrKAyjl9Ujuf7BnBphvVjlOLbilsNOpApFqJ3I5KfODT0vlPmcP1SmzWTSMW8ZyG6JpQtlm28scleH+iHLvDTSq3Easoga0Sxj6gMSZr4nKJ5rR3Fz669yuJ/dArqrkuNuQ4DOBV1e2UGqX7R/6KVug/Aq60fSus6olCqcGVBTjbt07Rof1CNUbTevBRCr9+nkFERnnsUbOfwozMQsXw6XGawAwNXFfSdwytdWB5j1HNkLZRHDdd88IrwU6Z4HCMW0P7C56xIz5AqMOg2NYQ6o/rzZqaZI6m5T/DotZkcJx2dTHuriG3JMqiSnvqJ4YkziGgO65V35uA/2PW5EG/Yw2GXeeOMWvrDK7Q+209sUisneGtuCxjsgJIt6t46OCYfN5hi0fUebQc3tjLpz84hMikmHGEXdMKtGJ57/RkoExTvv8YjGDQC9R8a9FaZduIFs8ricryqO5Vb4qrC+TCRS/JiILSzvYqX8DEzJxhE8INW0KfVzlTiyazIBvx6GgcM83dUBoGwo2EdlnHINYhu4CuDZDXgnQaPEy79FU/6h8Ifr27cHpw2q/fGkFViFP3p8+onivYNepW2QvhJIw4gW1kwdxyxicKBBvc6dYBeztLn6vXY8EuEvJ88mZlaKhQATlqrFPUgpGhIvkuQ315dB/vNwJuLjHEJN090b5OZMLrpgAZwSxaKhaJpjpXPv9oyzqaoo8d7ZQaOaB5qDgrYLXu0oiHQBbFPODqToTAf3wLrsNqBnxKBNBQs5kSKlg8938/3nFdZ+JPzBPdTiJ5OLi0m8OL1NLZq4kP8FAJ0OZ4eQvwGYVGgRiNoub0GIw+X3Fr9VrvXi/SqUpCFDE9nyTy+E6I cgG/nbdC gexEz7DL/dhoLbALKHldVQLFGhzuidnX7tjJdz1R1ZuzzFcmbw9P8tmLSgeEoR8LLvpMo1p6eO3DvTHjZRIyd4vekgczP/5gWE7LZyZFlTwwD3HFaAGE2IrXmnsWl4Ccdi0u3dJXIgTfmO3jO6YHIRb6SDDJTuKR6kFCBpKeEYqwb5E6nNTPrHmGmOct0R2k2vxfxp5u35DOLORXomv3B4wA1txKuxConc6MXkgN1bdwojj0ndHGBv8unfYtENbIh+WKTXKRzt5Js3uvi1gTy5jYvP4OBwQKsN4X85emvmbqK0BFrJBPRseuryxoi2/dgJZKp0aHTiPx+xB1kf7jydCkNrXOmCgjA8k6+S/y54MFm82IrzsO39ytaAKMktlQ11PJ4DDo742n1j23EXWex6LV58mYvUu48Ji4iDOEy3YMcJdMY0hM/3mYTcEuFKDFw/eFnyg4r5b/BB821PWRA0CJh2D982Gy2nccn7WFU/6jBY91DszP60JJGemzYO1xVBcSMvMMHO/1gGIx8xG0HmaJnb2luHRh3RiJ4pJs6eukoy4KRl5AjcU6pNqPl69DDd08YL4GBwvfTD+PuDXSNmarBFApI7Mo3yVSqqD38BQl/IsqFG9d5xLx4bDaQD3FKi8iSMOaPQlfxwySB3hk6R5PyYOumLQELOufV1lBViozxJjg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.433821, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Another overhead of aging is page moving. Actually, in most cases, pages are being moved to the same gen after folio_inc_gen is called, especially the protected pages. So it's better to move them in bulk. This also has a good effect on LRU ordering. Currently when MGLRU ages, it walks the LRU backward, and the protected pages are moved to the tail of newer gen one by one, which reverses the order of pages in LRU. Moving them in batches can help keep their order, only in a small scope though due to the scan limit of MAX_LRU_BATCH pages. After this commit, we can see a performance gain: Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-2: 44810.01 Ops/sec (+1.8%) Signed-off-by: Kairui Song --- mm/vmscan.c | 84 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 71 insertions(+), 13 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e3b4797b9729..af1266129c1b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3102,9 +3102,46 @@ static int folio_update_gen(struct folio *folio, int gen) */ struct gen_update_batch { int delta[MAX_NR_GENS]; + struct folio *head, *tail; }; -static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zone, +static void inline lru_gen_inc_bulk_finish(struct lru_gen_folio *lrugen, + int bulk_gen, bool type, int zone, + struct gen_update_batch *batch) +{ + if (!batch->head) + return; + + list_bulk_move_tail(&lrugen->folios[bulk_gen][type][zone], + &batch->head->lru, + &batch->tail->lru); + + batch->head = NULL; +} + +/* + * When aging, protected pages will go to the tail of the same higher + * gen, so the can be moved in batches. Besides reduced overhead, this + * also avoids changing their LRU order in a small scope. + */ +static void inline lru_gen_try_inc_bulk(struct lru_gen_folio *lrugen, struct folio *folio, + int bulk_gen, int gen, bool type, int zone, + struct gen_update_batch *batch) +{ + /* + * If folio not moving to the bulk_gen, it's raced with promotion + * so it need to go to the head of another LRU. + */ + if (bulk_gen != gen) + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); + + if (!batch->head) + batch->tail = folio; + + batch->head = folio; +} + +static void lru_gen_update_batch(struct lruvec *lruvec, int bulk_gen, bool type, int zone, struct gen_update_batch *batch) { int gen; @@ -3112,6 +3149,8 @@ static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zone, struct lru_gen_folio *lrugen = &lruvec->lrugen; enum lru_list lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + lru_gen_inc_bulk_finish(lrugen, bulk_gen, type, zone, batch); + for (gen = 0; gen < MAX_NR_GENS; gen++) { int delta = batch->delta[gen]; @@ -3705,6 +3744,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) struct gen_update_batch batch = { }; struct lru_gen_folio *lrugen = &lruvec->lrugen; int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]); + int bulk_gen = (old_gen + 1) % MAX_NR_GENS; if (type == LRU_GEN_ANON && !can_swap) goto done; @@ -3712,24 +3752,33 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone = 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head = &lrugen->folios[old_gen][type][zone]; + struct folio *prev = NULL; - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); + if (!list_empty(head)) + prev = lru_to_folio(head); + while (prev) { + struct folio *folio = prev; VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); + if (unlikely(list_is_first(&folio->lru, head))) + prev = NULL; + else + prev = lru_to_folio(&folio->lru); + new_gen = folio_inc_gen(lruvec, folio, false, &batch); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &batch); if (!--remaining) { - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); return false; } } - lru_gen_update_batch(lruvec, type, zone, &batch); + + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4240,7 +4289,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) ******************************************************************************/ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, - int tier_idx, struct gen_update_batch *batch) + int tier_idx, int bulk_gen, struct gen_update_batch *batch) { bool success; int gen = folio_lru_gen(folio); @@ -4283,7 +4332,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c int hist = lru_hist_from_seq(lrugen->min_seq[type]); gen = folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4293,7 +4342,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { gen = folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); return true; } @@ -4367,11 +4416,16 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, LIST_HEAD(moved); int skipped_zone = 0; struct gen_update_batch batch = { }; + int bulk_gen = (gen + 1) % MAX_NR_GENS; int zone = (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head = &lrugen->folios[gen][type][zone]; + struct folio *prev = NULL; - while (!list_empty(head)) { - struct folio *folio = lru_to_folio(head); + if (!list_empty(head)) + prev = lru_to_folio(head); + + while (prev) { + struct folio *folio = prev; int delta = folio_nr_pages(folio); VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); @@ -4380,8 +4434,12 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); scanned += delta; + if (unlikely(list_is_first(&folio->lru, head))) + prev = NULL; + else + prev = lru_to_folio(&folio->lru); - if (sort_folio(lruvec, folio, sc, tier, &batch)) + if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted += delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4401,7 +4459,7 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, skipped += skipped_zone; } - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); if (!remaining || isolated >= MIN_LRU_BATCH) break; From patchwork Fri Dec 22 10:22:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13503167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19C6BC46CD8 for ; Fri, 22 Dec 2023 10:24:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A91B16B0087; Fri, 22 Dec 2023 05:24:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3E2E6B0088; Fri, 22 Dec 2023 05:24:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DF2F6B0089; Fri, 22 Dec 2023 05:24:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 761F26B0087 for ; Fri, 22 Dec 2023 05:24:19 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 497B0407C2 for ; Fri, 22 Dec 2023 10:24:19 +0000 (UTC) X-FDA: 81594069438.24.90B0288 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf07.hostedemail.com (Postfix) with ESMTP id 7603D40005 for ; Fri, 22 Dec 2023 10:24:17 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RyuoSLFp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703240657; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZVtAklavGTH0ChBjbCr10B7CRN0g8Fxr7pwyrunV9Wg=; b=HT5YOa39raYz3UMFQSvusWZCUh3FIqXVXmJHW792t3uDYxKAxhRWLCNIGTgLu6bVPRmgKk yi+Q68apyf6n00JPNmeQF6Da5EqbjimGiPuPPakbP64k4m/luNbsVUnbzrPBEPxXrWtUnq aZLuFyr0iof/LnZejIxwpI4Obe4Uqyc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RyuoSLFp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703240657; a=rsa-sha256; cv=none; b=JPjLcM36z7uwlqeL4YOlhI7Wocl95FUi13+mx6tAXFaGWrLY3LHjr7rbh6nHhOfZJTOAVr oj5V7aazdOmcYOSpU1rKsshpvYUJRulzZa+2rPeAvAZfKvs458ayP2ol2kZHeOfz+xvVfK fNcv+yLYPTcWe0OP5OmxEvGUVRwaEgA= Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-28bcc273833so1488250a91.1 for ; Fri, 22 Dec 2023 02:24:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240655; x=1703845455; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZVtAklavGTH0ChBjbCr10B7CRN0g8Fxr7pwyrunV9Wg=; b=RyuoSLFp9suW2h7+RV3rhsukSiuNrv4PxVZ+774QrdAiVF/o9kda1i2ig4dPpAhwwu 47UA7nnOwZYDrsXZ7HgNzaTV5KpvkfaGxWSAwoqsqlYVaTvJlzb0tY3DE4iTkGlACYi2 7BgfH9e5ld/kllA0Fs18pAjRjf7xyOsxeqrTycTy3bYLjbX9CQBnrc2pfL3azpStMwWX KeD0SOLS2pNTs5kOHdNnyLvK5kTKgJ1I792lfuHMSaEXp9JKQugKH5F12iqHEmdiV+YP AXV0oecyo5qAfxreeMT62gsvqO/YcGXFJPO7ZEvxcS8UJLH5pzvrZ5SbHWM6YeDH5iax ndyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240655; x=1703845455; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZVtAklavGTH0ChBjbCr10B7CRN0g8Fxr7pwyrunV9Wg=; b=Ogc2fAQDQBFB46xLVvVlio8ABHpf/JJR65bi9CM7q2QCgvHNKbnBwWyAW6FsoEPfpf 4fxtWLEy4PMvabHLJcvAq8folEimcUMV00C7uB/wNlOa1AFx38015dVYooUMu4/ajUIC Ytsmx7vJ4AHoxFmtdJyaIZe+oJSnVUEnTy2ZT4cF4B6rUM+cw+LZ6QVS5PN5q8QsqzZC ZiT+yy1Kez7KsU0mZBh6xmsITgQlMxXEfW83GNUtaf+HyQP0PP/0LiiVlk3kNm5qaVlx ycimiJc8qydRxU781Wke1UPg9ngIKl3XhJc6ZoFrNujQ3g96k7Z/S78YrzU2IbsZhgQS wPeQ== X-Gm-Message-State: AOJu0YxWtxkqlcvxOxqDZixzhddBuAQ8h+VwLREhKkESmVBihLNbSDRs K/F4mq3X+gAjlWMH6NLqB2L0/zd3Jy6pW6Nj X-Google-Smtp-Source: AGHT+IFG6ajXG6vALhMWO57ARn52j3Kb6LdriotQIo1jxt6hAcExEanAl4pobD+1sBFGtt0fLivXqg== X-Received: by 2002:a05:6a20:12c9:b0:190:85d9:9d4a with SMTP id v9-20020a056a2012c900b0019085d99d4amr1304221pzg.97.1703240655043; Fri, 22 Dec 2023 02:24:15 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:14 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 3/3] mm, lru_gen: try to prefetch next page when canning LRU Date: Fri, 22 Dec 2023 18:22:55 +0800 Message-ID: <20231222102255.56993-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7603D40005 X-Stat-Signature: 9fueftp8iq8mhnixbxoh9xznku693cw9 X-Rspam-User: X-HE-Tag: 1703240657-17535 X-HE-Meta: U2FsdGVkX1/JiIvKD+M60D9hngq4spDrtOjwdx5GrZbQXy/8KU4wKqPtONB3vSFp2IA11BG1ZI0GjQTJoEa+a5WbHS8fiRANCPMvvWw6z42qP4Sdni0nSxmAY1jnwoZ/0nBT4xegnVm2+d7VDTZS9CnBfqBNuHLLW3lubuKfDH4+kwuTy4P/WtIRZXgr0ZtYkinYcbqQM5fQ3+QaVpl+tJK4fkCuZGh0ufaKoJtO4aUQglBXgPrJQMjLAMFqKhlfdxsz70TmBCfd7/+BmBDSRLXgnaLErMw2QupjLSq05Sqzc1sgjxycLYqgZutGaE5f3kb70+YJt08yJLC6uUeJrqe61TUgWxT7gsP1R7X+3ufL6ryapGN81YLqvW+AKTXR/efvLgvG43rml5QalTHvTu6j7ULIjz3LCnreSKUNRMU1CB/1vQ+Hzh7wETbsncKvQ/bYxSOSB8zT0DBslW8A2OIEcgjxgbYpnGo49xtsW8vZNj4kdN7bCJPiGmuEjaP34ivHPby6o1ptIlt6JO7SZZ1cC4bh6I0vfjKooKqj7kDCV61R/iV4SLGGZVxjFN9UrXB+FnjIIvj9yEzWMtlX2qXb7PTtgxhyCC8vOPFzTnIEdEH0cd+PqSsqlNlYV+WKZCq7VOgbNXYjDm7jXuJXkqFW1+u5jc0qdnZZncY8nu3GoWPyZ5FxGXBmW6sXNr6Azm/CT/O/l3YADQe7WWHZix7P6UWttqtfcEvWfr81z18LskkNS+uDUkgfOVjFUYHkitroQoqNzV3tELDqTNB8vsPuMhb1vcwZ0WfG2LufbPI04Ai0i3YG1i/MMdphgtVkcAouPMVaGUVd5GifvfoAJkdGo+7PxRZ2Bk1wlH2HBm2ZwuNs7UMs6SQ94f7kUNNHjp3itusoNj0mR3AflvfxmRGpSK7l4W/L1QUeZRCHG8W1GWZdcX3WhPQxfLohIyB3bEU/8i14f2aD+Nzd/Lf Xe2kf31E y4t2exNGTXfpVMA1dHi0SJHLxiEGRK0/NZYNrujDTIa1l4z09LyMOgc01LtFNtgH/wN/LJ26ePx9HoGHQp8aUtvcW66AcSipWAiDrYgI/ga4qnEEzPEnk5JgLyATJ2uJl2OVSeuoCEmfXAE7KHIkpJCbi8Ohkr2Jtk70oZzfO/1G7UbHrqhwwsA7nz4hRvqC7lnlUMTx1KNXfIcEJd8iq07ChUQtpXMWydLSJ/l5zMIBMLZbN/wPkcsix0JTGQwMf0e25Gg8OsD0Li016dYi5uuSvlS2Jamy5URNWwUKjSsPrdRT7/lyFNwPFvgqH32AnlPGni2q1E5UHuPkqETTPTGgCDuybx5wKq1QKhbqMWUE5fYKwwL6Rrl3RclxH8leujq8+dsg2E2l/vFA/K0Wi9An7Rn7GeUsTWUAj9Z54fSPoRUBR3as8KVzsOYswv41OFRc0T3HcC8Khp02LiydsCai8ydMliEB3aFHcEQClSey8MFF2Sk52dlP7iwScKbMwI0bB5Qgxlh+1/wcQfZpFTZunqdjxbqZo5SKStcm/AsShQ6lBf0R+zHdGti7dL72JCHHzI1jFl+2g747FowbWTJnVWrL37C1GMk1bEDMwA0iDByAoqehBDTOwfgoUx6lPiN6e0M4xcA30VGYL+Ozni70RNcgvphI07lowmgCTvf+woeo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.013936, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Prefetch for inactive/active LRU have been long exiting, apply the same optimization for MGLRU. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=1 --key-maximum=16000000 -d 1024 \ --ratio=1:0 --key-pattern=P:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-3: 44890.50 Ops/sec (+1.8%) Signed-off-by: Kairui Song --- mm/vmscan.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index af1266129c1b..1e9d69e18443 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3764,10 +3764,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) != type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev = NULL; - else + } else { prev = lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } new_gen = folio_inc_gen(lruvec, folio, false, &batch); lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &batch); @@ -4434,10 +4436,12 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) != zone, folio); scanned += delta; - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev = NULL; - else + } else { prev = lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted += delta;