From patchwork Sat Dec 7 22:15:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13898425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9A91E7717D for ; Sat, 7 Dec 2024 22:15:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A8A16B0368; Sat, 7 Dec 2024 17:15:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 456176B036A; Sat, 7 Dec 2024 17:15:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F59C6B036B; Sat, 7 Dec 2024 17:15:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 067DD6B0368 for ; Sat, 7 Dec 2024 17:15:40 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9E4731A026F for ; Sat, 7 Dec 2024 22:15:39 +0000 (UTC) X-FDA: 82869570210.10.B6B6377 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf15.hostedemail.com (Postfix) with ESMTP id D3590A0011 for ; Sat, 7 Dec 2024 22:15:18 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZUpreLSG; spf=pass (imf15.hostedemail.com: domain of 3CMlUZwYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3CMlUZwYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733609725; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h1mvjlX5nT1rLjjtVWZn8iR4hG3sSfAtSE80GuqS3ww=; b=IUo62l8pt4He1wfEccUTPDno7Pv79GEtDd+CMmcyqwM7oYde5mhfJ4V99ad9cqfRQQJLec kkfmYDIUs0qZ6mMCPnffpL7RhOwMwuRkoW7Mi5aH2FNoEBKSg73TzUa8FE5w/4pRxERt5t /LfgLX2oZFZ20/D7vA1e0Pgfx8fa5mw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733609725; a=rsa-sha256; cv=none; b=4PnEMRxbgRJLyYT+cey74VAAQ9Ex4A7Ak6K+uB6jqEqwbsxMKnvjQQMDvcpLSeAs9/i+w7 t8lLbyl17NHTjBhX24rimKtoCcb6sl/hzeQDqW7wrlDRtzTkrpJe739hTmjIsPUQ+lCaML fLM6KPzfTBUB1+MoUMx9dvMDOyrSMOI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZUpreLSG; spf=pass (imf15.hostedemail.com: domain of 3CMlUZwYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3CMlUZwYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ee8ced572eso3449817a91.0 for ; Sat, 07 Dec 2024 14:15:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733609736; x=1734214536; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=h1mvjlX5nT1rLjjtVWZn8iR4hG3sSfAtSE80GuqS3ww=; b=ZUpreLSGqkSEjtDjzteOiarask+Qd0XvjreiREpT8nDGnqXXuDCwFB8Vf2OiBh+oEB qzoAvOUHvRc2hdUUg5jy7b2VYRP7DMn094950o3zAySIwKClJpz5++D/E76b6hO/Qsia aDTPW1DS9KRxaoUaqxw2moTw3zZ5OCr8rqK6cJfDTfHBU43OHtKJ7k29O4Z2zxFc3zx6 PusfOtDKKDfu9FEG77Iyjcul6tReljeISZpuqnPCpMKgFjP1e4TQvYdTVAZEvIR9/5Sd gz3OPJRJc7xU6U65O8UKHpIVFfymaUsITF6F8dle/dX3Q6yhVArus1LnfRpcL4O911ov iEMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733609736; x=1734214536; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h1mvjlX5nT1rLjjtVWZn8iR4hG3sSfAtSE80GuqS3ww=; b=nGyvtC5e46dmos8M5n/p7/l9ah06XlzaUjVkPF/DzFZOzhIsDCbyK55YrfbLWwRZ0L t73PK6sS49s4OtyYYGDQzj+/ZM9KTseKruPda8bJ5EJ2uqKYQAVcNsPMsxoYf+GiFF3o xJJNYKjhckcOnSr22JKbeRfPjF8GHFiPHE3KKAt0mdvYXO9qcXXbi83+hri5fWQuLh1v hbF/e3dZnmE6o2FioqB4FU9kyqQIhPI4i0z9OG6w2cOQOvfkdwci/rn9Yqd3dLKqcf+A OZthhSnotgyADBRryrMX+5JUO6tlhRcvnPfzrxFQbKchBV+2YbZTOfl8KhE1RIDU1mEN PAvA== X-Gm-Message-State: AOJu0YzwsRyMkxWMazSHyeiU5EQYwvAOKvlxPTCOmZ7/ZLnBBqnjZrvP QX4YHAP0zq+m5Vbd17it7rmyV316E0grhFGy6vLxqYM7LwU573s95LdLfu2Gvmpl5fCY7kZtfni 6wA== X-Google-Smtp-Source: AGHT+IE1cK9olUWYZZEb01UAnSqV/H0pU5jqrRIyuw+rrX9BP0rn/osA43XuWh8gcnpTcn+qx/sjV1NZo0M= X-Received: from pjm5.prod.google.com ([2002:a17:90b:2fc5:b0:2ea:448a:8cd1]) (user=yuzhao job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:558f:b0:2ee:4b8f:a5b1 with SMTP id 98e67ed59e1d1-2ef6aadb4cbmr13013029a91.24.1733609736695; Sat, 07 Dec 2024 14:15:36 -0800 (PST) Date: Sat, 7 Dec 2024 15:15:20 -0700 In-Reply-To: <20241207221522.2250311-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241207221522.2250311-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241207221522.2250311-5-yuzhao@google.com> Subject: [PATCH mm-unstable v3 4/6] mm/mglru: rework type selection From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , David Stevens , Kalesh Singh X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D3590A0011 X-Stat-Signature: ybfoeebm55qniux7eyiex7qmmfwozs91 X-Rspam-User: X-HE-Tag: 1733609718-186325 X-HE-Meta: U2FsdGVkX1/jlH0CtVzdkouIHHSYffP3nCCv6IlPE/A8PG2vpRkbT2G2w2oAqL7KV7KDwDWqzdo7nK6NxZr6jp5sS7e8Xv9jK7+TB0tK57WiHNP8zzoWHtOqDKdjvV702MD8D92wLifD6LV01Xg82aEiQpUVRcYsBWBqBD/cuHIHRQCLi8lv2AqlDd1eVsr5Q1NZ3oU056Zf8YWQWMplVctt8cFHG+n1GW4F1RHBnJIfPZhndOsV2yxR+6NRyWlfHnx/wUbEuSPjj8Fqvv9aLQperfCGr06xGDMIYUNHHs5w2iYVKsLXQHUX/g2JKXVAFNG8ZveXsPJIDtE58r/u4sbmLMTIzRHim4OHnCNIH98fo2zRBVFa4bt4ZjgARFZO5vlKxv2uw34ggnepZwgftV0y+SvX6qXma6AtFzCZwaDxOLyFj0U/Uari9ytn6iZ060NKXNSY9toZxlZRP+Do9vE9XhWJHhOrw7l7Q3uFTEDlVbsOP3Gu4u8T2SK9yawiDl+iqBhvI79iVZ3F3I/Q5Xb1HYXxApjIg0dbKIkp8g1WML75YoE9DDU7nomSgTF4+fJ1Mk/4F4II/bJ2J8cfP4clAS/iMrxBdpepJ6fI/0zcCrRm86O5Xqarj2wU8ert+CZU+pGJoRba8x3LPo23j1u1vRMB4dxSJVflPBpZxQMYPelcnFJM5/VHnk8UssRntqMQBYoa7VZcNya3YksLqZoMmCC9shB2epRnINCpEigszWt90M0aUApI2jWYfU+MXgE8eBr2T4vzHUVCu8cgmliG0lm3n7jbNby2APHbWAXekOXNX22EtQTFUZrBPQyEIAwCnqi2V5Ygf3FIQePihUy0GENwAvvtqCFXFYzrG00NI8sXV74qyCne3TGOIqIyUFV/rzElmLPzIBcZ/D1ciM0u0HapfVjqWZGt05N3yeLdHUK2PPbgZmCFPfgyl2dzc4lPhmbm0IfT5gzMJi8 AMKc1Pvx XGyo7kLwVsmIaEWxU7TimnHZr85KoNLrscua31UVVCCfHm+ySqKQUV+g9X5r2PYi6Kiey9Viaknu+Gw1rwV+/vibyxIiZE0EZXhj/ShBAl/EmP0nekfWoIboUQM5E048f/JSP8Gl67vQiDI4r0L5iRMa3el212qkUs5B01NRrOQlUryf8eiEgQ+2JkObXzBuR4feF9hFFD+MOtQFnSoEdKyZzhBU9WCc+Fh5bOJIiEEPP65E6Ie6tKiYu/QWCbRlnlO9/kCI9q5RvKbfRyoO193wwf95FOjiqq5Fnp/Qen1EM2/tiF5kkKYCLONoDDoFaQE8pYFQ6VE33ul0E8LbhoOQ79sTKw3DTEiyuUk3VrKdNXi52LccFIz6vVkMH9TJNEV2IAVkhi1NkBlURzZgLF8k8A4STr0utDcyyyPNmKIRR3Z3+hJT9+u29TRmMNn8PzJ/8t6CjU/xddGDXNwqcjuKPp/XkgIOU4amkOONQkrn3T3pU6+rrIq02lknVhWJeJjUtgWUK8OOy/kdX8J4CY2EKYnQRyeWFxvBpAotWgCoa1GT8QG8OgKA40g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.105525, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With anon and file min_seq being able to move independently, rework type selection so that it is based on the total refaults from all tiers of each type. Also allow a type to be selected until that type reaches MIN_NR_GENS, and therefore abs_diff(min_seq[0],min_seq[1]) now can be 2 (MAX_NR_GENS-MIN_NR_GENS) instead of 1. Since some tiers of a selected type can have higher refaults than the first tier of the other type, use a less larger gain factor 2:3 instead of 1:2, in order for those tiers in the selected type to be better protected. As an intermediate step to the final optimization, this change by itself should not have userspace-visiable effects beyond performance. Reported-by: David Stevens Signed-off-by: Yu Zhao Tested-by: Kalesh Singh --- mm/vmscan.c | 82 +++++++++++++++++------------------------------------ 1 file changed, 26 insertions(+), 56 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 00a5aff3db42..02b01ae2bdbb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3089,15 +3089,20 @@ struct ctrl_pos { static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain, struct ctrl_pos *pos) { + int i; struct lru_gen_folio *lrugen = &lruvec->lrugen; int hist = lru_hist_from_seq(lrugen->min_seq[type]); - pos->refaulted = lrugen->avg_refaulted[type][tier] + - atomic_long_read(&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + - lrugen->protected[hist][type][tier] + - atomic_long_read(&lrugen->evicted[hist][type][tier]); pos->gain = gain; + pos->refaulted = pos->total = 0; + + for (i = tier % MAX_NR_TIERS; i <= min(tier, MAX_NR_TIERS - 1); i++) { + pos->refaulted += lrugen->avg_refaulted[type][i] + + atomic_long_read(&lrugen->refaulted[hist][type][i]); + pos->total += lrugen->avg_total[type][i] + + lrugen->protected[hist][type][i] + + atomic_long_read(&lrugen->evicted[hist][type][i]); + } } static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) @@ -4480,13 +4485,13 @@ static int get_tier_idx(struct lruvec *lruvec, int type) struct ctrl_pos sp, pv; /* - * To leave a margin for fluctuations, use a larger gain factor (1:2). + * To leave a margin for fluctuations, use a larger gain factor (2:3). * This value is chosen because any other tier would have at least twice * as many refaults as the first tier. */ - read_ctrl_pos(lruvec, type, 0, 1, &sp); + read_ctrl_pos(lruvec, type, 0, 2, &sp); for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, 2, &pv); + read_ctrl_pos(lruvec, type, tier, 3, &pv); if (!positive_ctrl_err(&sp, &pv)) break; } @@ -4494,68 +4499,34 @@ static int get_tier_idx(struct lruvec *lruvec, int type) return tier - 1; } -static int get_type_to_scan(struct lruvec *lruvec, int swappiness, int *tier_idx) +static int get_type_to_scan(struct lruvec *lruvec, int swappiness) { - int type, tier; struct ctrl_pos sp, pv; - int gain[ANON_AND_FILE] = { swappiness, MAX_SWAPPINESS - swappiness }; + if (!swappiness) + return LRU_GEN_FILE; + + if (swappiness == MAX_SWAPPINESS) + return LRU_GEN_ANON; /* - * Compare the first tier of anon with that of file to determine which - * type to scan. Also need to compare other tiers of the selected type - * with the first tier of the other type to determine the last tier (of - * the selected type) to evict. + * Compare the sum of all tiers of anon with that of file to determine + * which type to scan. */ - read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, gain[LRU_GEN_ANON], &sp); - read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, gain[LRU_GEN_FILE], &pv); - type = positive_ctrl_err(&sp, &pv); + read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappiness, &pv); - read_ctrl_pos(lruvec, !type, 0, gain[!type], &sp); - for (tier = 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, gain[type], &pv); - if (!positive_ctrl_err(&sp, &pv)) - break; - } - - *tier_idx = tier - 1; - - return type; + return positive_ctrl_err(&sp, &pv); } static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, int *type_scanned, struct list_head *list) { int i; - int type; - int tier = -1; - DEFINE_MIN_SEQ(lruvec); - - /* - * Try to make the obvious choice first, and if anon and file are both - * available from the same generation, - * 1. Interpret swappiness 1 as file first and MAX_SWAPPINESS as anon - * first. - * 2. If !__GFP_IO, file first since clean pagecache is more likely to - * exist than clean swapcache. - */ - if (!swappiness) - type = LRU_GEN_FILE; - else if (min_seq[LRU_GEN_ANON] < min_seq[LRU_GEN_FILE]) - type = LRU_GEN_ANON; - else if (swappiness == 1) - type = LRU_GEN_FILE; - else if (swappiness == MAX_SWAPPINESS) - type = LRU_GEN_ANON; - else if (!(sc->gfp_mask & __GFP_IO)) - type = LRU_GEN_FILE; - else - type = get_type_to_scan(lruvec, swappiness, &tier); + int type = get_type_to_scan(lruvec, swappiness); for_each_evictable_type(i, swappiness) { int scanned; - - if (tier < 0) - tier = get_tier_idx(lruvec, type); + int tier = get_tier_idx(lruvec, type); *type_scanned = type; @@ -4564,7 +4535,6 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; type = !type; - tier = -1; } return 0;