From patchwork Wed Dec 21 00:12:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13078276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7978C4167B for ; Wed, 21 Dec 2022 00:12:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DDC78E0008; Tue, 20 Dec 2022 19:12:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8904A8E0007; Tue, 20 Dec 2022 19:12:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 708948E0008; Tue, 20 Dec 2022 19:12:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 57D208E0007 for ; Tue, 20 Dec 2022 19:12:42 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 22B33A0255 for ; Wed, 21 Dec 2022 00:12:42 +0000 (UTC) X-FDA: 80264387364.05.A319E01 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 873F080006 for ; Wed, 21 Dec 2022 00:12:40 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GlPiUBTF; spf=pass (imf30.hostedemail.com: domain of 3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671581560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=OGylBfRrFzStTyj0AlzS/TcDDW4JEwewZst2XJHnRxC/edqW2FnlgAEwkRvzKmI9dgW5K+ RcYfa+yy6iums/qZ5nN6HgXnceDk0don5ZOY3kJX1qpsRKgyqFjXQaayT9/jUl9M5XA9+X LO1Afrorc5HAQA2BwGNqTERmFwKhfYI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GlPiUBTF; spf=pass (imf30.hostedemail.com: domain of 3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d0-iYwYKCMI627piwowwotm.kwutqv25-uus3iks.wzo@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671581560; a=rsa-sha256; cv=none; b=TS9o9aWhsmzR9PQA+ftVZqswsaYplpmoKOsNJxrGWCbElGV43P8PUVYCqLYGtdnzq/UH6l hnqJILPX43f2k5g0jNPQbluFYZTaSzpajGZUrWOOqBShBe1FpECj4ZqPtkSd9SENYuqt/D Z6mE0J2Vcusi+qNpfGx5cuttFhhoEp8= Received: by mail-yb1-f201.google.com with SMTP id g9-20020a25bdc9000000b0073727a20239so15741819ybk.4 for ; Tue, 20 Dec 2022 16:12:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=GlPiUBTF+eOAUqdbrju+L1+Kh2wEO8CK+cMwhM7K6Y1IUVWqKXq6Srl0dPkE6NZEHM NvnzPFwFsuQL7HH8D4QeVAltwJBOXQAXNpJV2tlNz2kzmGWq9+RWxmJgjzKJ7BDlJTx8 BqlrCKIq6REydAxMtmiIG1Wk7yRm/0MIuq/sGKpU/qThTOiFIb8txt0eAaelMAUKVpmt xnWEPSQG7j+mL+CH22sZaicYWYgO9eV7VsqFV55qrg9aMH2v/OL9Z3KNrB4iBvpBGcTS 1auKzt2mFvRdMGB4k6qPXceTvJwVKzP3vKVe4g+/uaB/gSrnSFHF0RKsFAdGd7PY39hz SECQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y306PH8Q0yp+xWizSY/ux8ywn6bxHcRrOk54D5HB2aU=; b=YSWea5/BcDwPJEcTkwlyFccIKnqc+NSmLEqq21gGSBdI+B13iWWdxhBVVhVDg2l/wE iFLRI9TrT5cZX2KjlwJxnmtEXAd2Tnw+/Pumd5HaESjod3zpUC7Q8//KZZF1zHg+NWUo aEg+I7Z9py+qweSLGM3cN9RZPNxloygo0NaGFt3QPozj1IOl1mx9WfSBNzL0uRaPPvEw tOuCbQZQghlQO84od/fkG4E35xuoVDHqO0L27F4oixUyOKxsVXLIQmCfXVJyxLEf7T7E 1S8bu9NnmPaqQQOXt3iebbAj8ZmuJ29n/YkUIW26W9mefh5fYsnD82caU+FmDtxsIfMG Xl0Q== X-Gm-Message-State: ANoB5pnGGUinmJU+FvcD/ZdHiOca0Hv8MEeHknMBGY00/TSB4FzyW+sD d57UI+LgX/ahmCA5SytSU9QxkzGB6wA= X-Google-Smtp-Source: AA0mqf4kmUm3Oml4SQcszfQyWi0BIkPzgCXt6tJS052cJtMI6HpR2ijMRjA//FRRHQ1i3YBOkd6DUZjJlrY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:a589:30f2:2daa:4ab7]) (user=yuzhao job=sendgmr) by 2002:a05:690c:903:b0:3b7:e501:90cf with SMTP id cb3-20020a05690c090300b003b7e50190cfmr5497843ywb.501.1671581559741; Tue, 20 Dec 2022 16:12:39 -0800 (PST) Date: Tue, 20 Dec 2022 17:12:03 -0700 In-Reply-To: <20221221001207.1376119-1-yuzhao@google.com> Message-Id: <20221221001207.1376119-4-yuzhao@google.com> Mime-Version: 1.0 References: <20221221001207.1376119-1-yuzhao@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Subject: [PATCH mm-unstable v2 3/8] mm: multi-gen LRU: remove eviction fairness safeguard From: Yu Zhao To: Andrew Morton Cc: Johannes Weiner , Jonathan Corbet , Michael Larabel , Michal Hocko , Mike Rapoport , Roman Gushchin , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 873F080006 X-Stat-Signature: 1auejh8wj36w6psjprdd6m4nsjaxenhe X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1671581560-140165 X-HE-Meta: U2FsdGVkX19Dp2Wd45cW7ixvr9CX/4vMjlHy6hrGLsS76npTVImPmA7YtNbbrJswKXKIapL4qSwA1U8euU07W9uVK5tZayrU8IiIvKj6AztNXTBe7kumRceOfkK+A4h99wk+FUIMv+EE5Ff7+5IoXH3paPa2ZqDvDRsxtuXDNG8BQdZPP7O2Cpdm1l5JlDlgt9hqpHOyB3/qOWNGK+HUod6KPJTIWTyXbFou9mFnGNdN7XvNBjwznn3UadghGOAv5Mz6ZfosPrnPAUR/CabRMRa1K0GRlPRJQLg+4X+ZYYdfhqNwx7HG7ZAanqruWfjdH8rmW9oK6mbJGe7zXA3jHUrvcpNRALLQc5U9C8qjyIwAtQMkw5wphwW1fuTrVSWYir4r/9LSffvmhoUopVdGY2i5F7yaO9o1lW/4l/id/np04l64PD+T6sDy0NejkzU0duCnIxB6Z9hvb23x2rdsgNx7VuAMnWdwXt7q1wWg/Tb4qrPzpNnOAWMnuNKXJo5uZH5aJBrcpa+NBDWE5lT7sJV/cKkRU8RVRVGRgZzFmPuex1ZH0XdFYu2TBF6lJh27JiGRgw1abOz0obZDH5vtt3XLEHbvI8Iy4sMV2P0+8/zZgxkJz0J+ndmA0fAmPXu25Z4XNHhwGKnZT7YU3ZAiT59X/eMgYE5Y0Nbc0+5jmhelW3iMqcmaEyvLJucLGicQBkXZRC6nyIFyIx7RiNqJ7t866RW1yj6GyFZMt2sD3o+1Eh5hdiuL/qJocsdYxj7aM2hkZeLvI7Ou1wd5cnKpQcoBILJefTrNs9EosAImVX9beQRJ72gy/f+zZzpgr15Xs9JxASqJ2rCWBiUgMU65qqOa2NWECJrS9A3VY+DXNh6aDWgG8lZpfW+Xs0c7SQot1C+FdF4ATJo0zAEW78ZWv7BpxxemPguztVF14qaMhtWIURpDHuGuPrD9CRuOoiMM/EyMkAshEAYc7k6JqJD knxRDg/g +oG7chJByrXib8u9rE/oNoNuvk/YlnARcmMao6ECt6nSkOiN8ZiBqAl+Bzckh9YjiQoVNZxStZqW4cP0h5v1Q5sVoMV8200H8+7g6RiA9mNwQm6jzDYD1Qj8JO/L3sgdqyripdeHWexdvW4Rf/Fr/jdWCUW45LuEGQ87k2ZH1P6bx+XEeWKCaQV68u+xOwesA+0BbGdi3zkbvGnFjlwni1qWUnfQ/6p2ZdNEeW8dJ+1lcILqGgfGiMNT7zTUDgjvZs51l6JW6ihtcQqEMW9VO/ExPdrt6eRKUzP/zOqYhOpHAOwHjFv0I45ujS9Fllzo3ID8H4mOv+fcGHUKufECEf1pZUeSKmDSAOitm2TQJij1/jOw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recall that the eviction consumes the oldest generation: first it bucket-sorts folios whose gen counters were updated by the aging and reclaims the rest; then it increments lrugen->min_seq. The current eviction fairness safeguard for global reclaim has a dilemma: when there are multiple eligible memcgs, should it continue or stop upon meeting the reclaim goal? If it continues, it overshoots and increases direct reclaim latency; if it stops, it loses fairness between memcgs it has taken memory away from and those it has yet to. With memcg LRU, the eviction, while ensuring eventual fairness, will stop upon meeting its goal. Therefore the current eviction fairness safeguard for global reclaim will not be needed. Note that memcg LRU only applies to global reclaim. For memcg reclaim, the eviction will continue, even if it is overshooting. This becomes unconditional due to code simplification. Signed-off-by: Yu Zhao Change-Id: Ieb2a658c8d441ec11348d2985c7c4d6b72106f6a --- mm/vmscan.c | 81 +++++++++++++++-------------------------------------- 1 file changed, 23 insertions(+), 58 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 94477894b226..9655b3b3a95e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -449,6 +449,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return sc->target_mem_cgroup; } +static bool global_reclaim(struct scan_control *sc) +{ + return !sc->target_mem_cgroup || mem_cgroup_is_root(sc->target_mem_cgroup); +} + /** * writeback_throttling_sane - is the usual dirty throttling mechanism available? * @sc: scan_control in question @@ -499,6 +504,11 @@ static bool cgroup_reclaim(struct scan_control *sc) return false; } +static bool global_reclaim(struct scan_control *sc) +{ + return true; +} + static bool writeback_throttling_sane(struct scan_control *sc) { return true; @@ -5009,8 +5019,7 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int sw return scanned; } -static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness, - bool *need_swapping) +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swappiness) { int type; int scanned; @@ -5099,9 +5108,6 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap goto retry; } - if (need_swapping && type == LRU_GEN_ANON) - *need_swapping = true; - return scanned; } @@ -5141,67 +5147,26 @@ static unsigned long get_nr_to_scan(struct lruvec *lruvec, struct scan_control * return min_seq[!can_swap] + MIN_NR_GENS <= max_seq ? nr_to_scan : 0; } -static bool should_abort_scan(struct lruvec *lruvec, unsigned long seq, - struct scan_control *sc, bool need_swapping) +static unsigned long get_nr_to_reclaim(struct scan_control *sc) { - int i; - DEFINE_MAX_SEQ(lruvec); + /* don't abort memcg reclaim to ensure fairness */ + if (!global_reclaim(sc)) + return -1; - if (!current_is_kswapd()) { - /* age each memcg at most once to ensure fairness */ - if (max_seq - seq > 1) - return true; + /* discount the previous progress for kswapd */ + if (current_is_kswapd()) + return sc->nr_to_reclaim + sc->last_reclaimed; - /* over-swapping can increase allocation latency */ - if (sc->nr_reclaimed >= sc->nr_to_reclaim && need_swapping) - return true; - - /* give this thread a chance to exit and free its memory */ - if (fatal_signal_pending(current)) { - sc->nr_reclaimed += MIN_LRU_BATCH; - return true; - } - - if (cgroup_reclaim(sc)) - return false; - } else if (sc->nr_reclaimed - sc->last_reclaimed < sc->nr_to_reclaim) - return false; - - /* keep scanning at low priorities to ensure fairness */ - if (sc->priority > DEF_PRIORITY - 2) - return false; - - /* - * A minimum amount of work was done under global memory pressure. For - * kswapd, it may be overshooting. For direct reclaim, the allocation - * may succeed if all suitable zones are somewhat safe. In either case, - * it's better to stop now, and restart later if necessary. - */ - for (i = 0; i <= sc->reclaim_idx; i++) { - unsigned long wmark; - struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; - - if (!managed_zone(zone)) - continue; - - wmark = current_is_kswapd() ? high_wmark_pages(zone) : low_wmark_pages(zone); - if (wmark > zone_page_state(zone, NR_FREE_PAGES)) - return false; - } - - sc->nr_reclaimed += MIN_LRU_BATCH; - - return true; + return max(sc->nr_to_reclaim, compact_gap(sc->order)); } static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { struct blk_plug plug; bool need_aging = false; - bool need_swapping = false; unsigned long scanned = 0; unsigned long reclaimed = sc->nr_reclaimed; - DEFINE_MAX_SEQ(lruvec); + unsigned long nr_to_reclaim = get_nr_to_reclaim(sc); lru_add_drain(); @@ -5225,7 +5190,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (!nr_to_scan) goto done; - delta = evict_folios(lruvec, sc, swappiness, &need_swapping); + delta = evict_folios(lruvec, sc, swappiness); if (!delta) goto done; @@ -5233,7 +5198,7 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc if (scanned >= nr_to_scan) break; - if (should_abort_scan(lruvec, max_seq, sc, need_swapping)) + if (sc->nr_reclaimed >= nr_to_reclaim) break; cond_resched(); @@ -5680,7 +5645,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co if (sc->nr_reclaimed >= nr_to_reclaim) return 0; - if (!evict_folios(lruvec, sc, swappiness, NULL)) + if (!evict_folios(lruvec, sc, swappiness)) return 0; cond_resched();