From patchwork Wed Nov 16 01:38:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13044322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0907AC4332F for ; Wed, 16 Nov 2022 01:39:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97A6C6B0071; Tue, 15 Nov 2022 20:39:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 929FF6B0072; Tue, 15 Nov 2022 20:39:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F3F36B0073; Tue, 15 Nov 2022 20:39:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 708416B0071 for ; Tue, 15 Nov 2022 20:39:02 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4399180399 for ; Wed, 16 Nov 2022 01:39:02 +0000 (UTC) X-FDA: 80137596924.25.A337561 Received: from mail-io1-f73.google.com (mail-io1-f73.google.com [209.85.166.73]) by imf25.hostedemail.com (Postfix) with ESMTP id D9A63A000B for ; Wed, 16 Nov 2022 01:39:01 +0000 (UTC) Received: by mail-io1-f73.google.com with SMTP id bx19-20020a056602419300b006bcbf3b91fdso8063138iob.13 for ; Tue, 15 Nov 2022 17:39:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:mime-version:message-id:date:from:to:cc:subject :date:message-id:reply-to; bh=+3H1y/n9oiHD2X4G7xUx8lXHM5xhQora50FOjVN+xhQ=; b=fwi7f8Z+uWmxlEYaf+rAPrJKJ1nNHEwMxBptKoB24RiVdFaQf/UuDA0K35R5vT6bde fSJHlpvPyccSBZmklkceR567YtW4U8tfCgUC+JKpQOCE/fN121M6miEvDKqnam8Iowd0 NfdZTjE4wzRH57xLqoP70EFrc3Pm8P95T2Xmel8UzwnoCYX/ogcNj+xdxLbdWB+6t5O/ g5J3E9u3CBkf5v/Mr8UywLUPVwlElMwuoIDoK7CoPVUshxdeU/1utYOgTBRChNRYP+q1 h1JUJ20kb0H5jWHyhDfQ8JdPzqkC9x6nU6pZ6PUJ8SHzylt2J+VQjqE0aXWjMVTY+X1P Wvbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:mime-version:message-id:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+3H1y/n9oiHD2X4G7xUx8lXHM5xhQora50FOjVN+xhQ=; b=c3wyyq/AIUwOh6BUyZN7kXsWJNaF51mT+uGPUxJ4XqH2XmimhLg+MDKw5NkL73mXAE /wKinapfnvDFUuo+wKz2fDXju+6V3FOUEQ+gnrVEvdoTP8NMXnQWhv6sQjNZB6b0piI2 dJ3rKYxCaRYqTUEl9ht2VElB3Z+9mC+JLL15pS+tlL4QvF9JM4MZBs8+DZ9TwSnYgtQx mWb1PQ1YVACJ05EY42LmApvJHF62zZc4bsqKSoKiob2vhLDiRLkyPxkNc6mzd3G6UhQc LgcztjkLgZidhv050c5K8x9oVSy6bt2q6zuXR5skibGjSRS2zovcR7M1ACZOu4kvO/IH IIkg== X-Gm-Message-State: ANoB5pn07MMEVqj4JeYmuB1E7K9BiJqrYXQqAbyOZA7Ch55Z/Ba6nJjl WgDc/HE8oey6+XBJzBu1vSHtYtZD5Tc= X-Google-Smtp-Source: AA0mqf6//MZV3Nzh6Z1f2Ar1/z45HIFQ68Aegy6PpnxzY9LKdl2JfiFcEeVTmgUqE1x7oHjAZ3W6qOposA8= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:1c9f:5aae:a1b2:1ae0]) (user=yuzhao job=sendgmr) by 2002:a5d:93d4:0:b0:6c3:ddc8:e42a with SMTP id j20-20020a5d93d4000000b006c3ddc8e42amr9055247ioo.76.1668562741083; Tue, 15 Nov 2022 17:39:01 -0800 (PST) Date: Tue, 15 Nov 2022 18:38:07 -0700 Message-Id: <20221116013808.3995280-1-yuzhao@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Subject: [PATCH 1/2] mm: multi-gen LRU: retry folios written back while isolated From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668562741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=+3H1y/n9oiHD2X4G7xUx8lXHM5xhQora50FOjVN+xhQ=; b=6p+7TozPA2GW1VJyxcRZqu+Bx54twcuf6jzozRZQzBb+twgPK719xMYYIlcXKCivgvmjFd iaBGlGk0WIwLY1mAiTwn/66SV+WxAkLg2dV5M8n5mej3AuYC6SG5YLVAu3GT/wbHUeUl7r oJxtmpa/rH7eXglrhv7POFFXOhM5tkc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fwi7f8Z+; spf=pass (imf25.hostedemail.com: domain of 3NT90YwYKCKYeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=3NT90YwYKCKYeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668562741; a=rsa-sha256; cv=none; b=K6v9CF0k1dtejsiNRdwlhRAh6JeC71gIDDea7dNxLdMCaIQm/zJ13lnRWzPdpkYauQ4ffR cA4/28YuJsy49Xc/y3D5LUsSO554NpQtVLh2f+hMocUoAwjjbmeqf3WV7t4oJS2kLy5XZP PCEePgQ+wRVhVimj9lHuT6lev+YPVBE= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: D9A63A000B Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fwi7f8Z+; spf=pass (imf25.hostedemail.com: domain of 3NT90YwYKCKYeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com designates 209.85.166.73 as permitted sender) smtp.mailfrom=3NT90YwYKCKYeafNGUMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 4aet6f4nso64pmh6yabuapmusq17cqnq X-HE-Tag: 1668562741-559831 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page reclaim isolates a batch of folios from the tail of one of the LRU lists and works on those folios one by one. For a suitable swap-backed folio, if the swap device is async, it queues that folio for writeback. After the page reclaim finishes an entire batch, it puts back the folios it queued for writeback to the head of the original LRU list. In the meantime, the page writeback flushes the queued folios also by batches. Its batching logic is independent from that of the page reclaim. For each of the folios it writes back, the page writeback calls folio_rotate_reclaimable() which tries to rotate a folio to the tail. folio_rotate_reclaimable() only works for a folio after the page reclaim has put it back. If an async swap device is fast enough, the page writeback can finish with that folio while the page reclaim is still working on the rest of the batch containing it. In this case, that folio will remain at the head and the page reclaim will not retry it before reaching there. This patch adds a retry to evict_folios(). After evict_folios() has finished an entire batch and before it puts back folios it cannot free immediately, it retries those that may have missed the rotation. Before this patch, ~60% of folios swapped to an Intel Optane missed folio_rotate_reclaimable(). After this patch, ~99% of missed folios were reclaimed upon retry. This problem affects relatively slow async swap devices like Samsung 980 Pro much less and does not affect sync swap devices like zram or zswap at all. Fixes: ac35a4902374 ("mm: multi-gen LRU: minimal implementation") Signed-off-by: Yu Zhao --- mm/vmscan.c | 48 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 37 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 04d8b88e5216..dc6ebafa0a37 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4971,10 +4971,13 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap int scanned; int reclaimed; LIST_HEAD(list); + LIST_HEAD(clean); struct folio *folio; + struct folio *next; enum vm_event_item item; struct reclaim_stat stat; struct lru_gen_mm_walk *walk; + bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4991,20 +4994,37 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap if (list_empty(&list)) return scanned; - +retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false); + sc->nr_reclaimed += reclaimed; - list_for_each_entry(folio, &list, lru) { - /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ - if (folio_test_workingset(folio)) - folio_set_referenced(folio); + list_for_each_entry_safe_reverse(folio, next, &list, lru) { + if (!folio_evictable(folio)) { + list_del(&folio->lru); + folio_putback_lru(folio); + continue; + } - /* don't add rejected pages to the oldest generation */ if (folio_test_reclaim(folio) && - (folio_test_dirty(folio) || folio_test_writeback(folio))) - folio_clear_active(folio); - else - folio_set_active(folio); + (folio_test_dirty(folio) || folio_test_writeback(folio))) { + /* restore LRU_REFS_FLAGS cleared by isolate_folio() */ + if (folio_test_workingset(folio)) + folio_set_referenced(folio); + continue; + } + + if (skip_retry || folio_test_active(folio) || folio_test_referenced(folio) || + folio_mapped(folio) || folio_test_locked(folio) || + folio_test_dirty(folio) || folio_test_writeback(folio)) { + /* don't add rejected folios to the oldest generation */ + set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, + BIT(PG_active)); + continue; + } + + /* retry folios that may have missed folio_rotate_reclaimable() */ + list_move(&folio->lru, &clean); + sc->nr_scanned -= folio_nr_pages(folio); } spin_lock_irq(&lruvec->lru_lock); @@ -5026,7 +5046,13 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap mem_cgroup_uncharge_list(&list); free_unref_page_list(&list); - sc->nr_reclaimed += reclaimed; + INIT_LIST_HEAD(&list); + list_splice_init(&clean, &list); + + if (!list_empty(&list)) { + skip_retry = true; + goto retry; + } if (need_swapping && type == LRU_GEN_ANON) *need_swapping = true; From patchwork Wed Nov 16 01:38:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13044323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C976AC433FE for ; Wed, 16 Nov 2022 01:39:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED20B6B0072; Tue, 15 Nov 2022 20:39:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E826D6B0073; Tue, 15 Nov 2022 20:39:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D49226B0074; Tue, 15 Nov 2022 20:39:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C836B6B0072 for ; Tue, 15 Nov 2022 20:39:03 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F8F1AB85E for ; Wed, 16 Nov 2022 01:39:03 +0000 (UTC) X-FDA: 80137596966.14.12ED49B Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 5100B40004 for ; Wed, 16 Nov 2022 01:39:03 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id j73-20020a25d24c000000b006dca101748bso14762549ybg.14 for ; Tue, 15 Nov 2022 17:39:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8qam/FxTvQ3VR0qeGq43naiB//AdnyQdmZle38abayk=; b=jGqN8aefFs6zNLkGLnpSlnnXZi9tjRjpMUeCGOHy8aO3vjCDZnRxuMfnt9ear0Jb+m 1a4asto4isLp6IL2/q37OWUfm7mnoGRjw3PJQ2ntFs+jIMz1i74mov/nV3S56BjhvKQj SguB7uoHXxkWbtdRXuBMXDa1YBsUolmIWjO1XpxmaaO6Pmj4CT/81+jgUEk6AexMjO9u rLGUzSgNr1VK32Az1Z0HRjEFo+1cFI8U/sWg4tzfb8ZHCxu4UbCxsmjNBXtdv0tXLXzz ARvgaRrbKwraUOuQBx+Z8BrcmcPUfW9lFbwF1+uxO9iNyPU1A7Q6qyjoxNfejqJjBrHT 62yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8qam/FxTvQ3VR0qeGq43naiB//AdnyQdmZle38abayk=; b=a8cg8XVjt7ESCB3q8MqoUf7rg1c/Tl+YCC65wVPIyr06w0kaHxZTXnF6bPWZV+KrwT PIRbITmr0FQ6hfCR93OXIZX8dCUuXNwOswYIGZkmXbM18h/616KY5gAhpIa4rNac/K35 jkxodFRnaHFnyq9MLACQgWsOHMdqW78BhppElRDvtYPIZWRumhdiSFGLADIQU0KlSPB0 XzCk17c8Whv0/Jn2q6dxrFBES/pA8CAW7M1xkx51FiwBJRyM5mm10dUc9KwYaCax9X0s 5rzLB8eNBTh8J1TOFQzkQvXTb/XdQ2YP+5/rNSx4t/VSXCbwUU9D7D17fHcdrtOMjbxA WonA== X-Gm-Message-State: ANoB5pncN4qi+appni1HHt82PBi30DXcSuel302feypnrUzBJzM1W6RS 9smZY14Vd3zx9VZNmMBF+VmDGEclFU4= X-Google-Smtp-Source: AA0mqf5WS+A1TvwtfuOYScKCu90IN4Zy853rnSmsas5yBOQl1zbQMy6CSlUp7gbTrVwKOXyIGofGlVwwp4c= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:1c9f:5aae:a1b2:1ae0]) (user=yuzhao job=sendgmr) by 2002:a0d:d596:0:b0:36a:bc93:587c with SMTP id x144-20020a0dd596000000b0036abc93587cmr20750607ywd.59.1668562742534; Tue, 15 Nov 2022 17:39:02 -0800 (PST) Date: Tue, 15 Nov 2022 18:38:08 -0700 In-Reply-To: <20221116013808.3995280-1-yuzhao@google.com> Message-Id: <20221116013808.3995280-2-yuzhao@google.com> Mime-Version: 1.0 References: <20221116013808.3995280-1-yuzhao@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Subject: [PATCH 2/2] mm: multi-gen LRU: remove NULL checks on NODE_DATA() From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, Yu Zhao ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668562743; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8qam/FxTvQ3VR0qeGq43naiB//AdnyQdmZle38abayk=; b=2VfgIvY7ml7OMT2o3uOYNoeshjzoIVK+evAeM7EOWigVbKN4l/tHF9hSlxbmiLaix2Rs2G 8dj/au1jPH4YmMCaQigK1Nk1JXs17ZSPz9QvpQTgD9qabFKqqGBXcgvVqMsWpox67UJHSn BpB6ELtJJCmF1k70RtdJXttLpsZ19lY= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jGqN8aef; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3Nj90YwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Nj90YwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668562743; a=rsa-sha256; cv=none; b=SHfwybHIC13aNI/B9r+8nJvQXjZ+nSTPgIyUtEb1Pf8nLS4NNZWo5IVVoVZsPshQQBjEoG lYwhHYBVIsLe/LgfIiUd+1aDNqlvxCjWVKSNKfW3ymuWosrfgLqsBdE5OBX8czbaSAtdRm pKSfeDoiCUt4Id1aG6eLa5kOH69sSGU= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5100B40004 X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jGqN8aef; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3Nj90YwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Nj90YwYKCKcfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com X-Stat-Signature: dhwu3dcse4drmbm433jo1i1wnhi1abqk X-HE-Tag: 1668562743-919528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: NODE_DATA() is preallocated for all possible nodes after commit 09f49dca570a ("mm: handle uninitialized numa nodes gracefully"). Checking its return value against NULL is now unnecessary. Signed-off-by: Yu Zhao --- mm/vmscan.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index dc6ebafa0a37..f7c02faf3a07 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3133,7 +3133,7 @@ static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) if (memcg) { struct lruvec *lruvec = &memcg->nodeinfo[nid]->lruvec; - /* for hotadd_new_pgdat() */ + /* see the comment in mem_cgroup_lruvec() */ if (!lruvec->pgdat) lruvec->pgdat = pgdat; @@ -3142,7 +3142,7 @@ static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) #endif VM_WARN_ON_ONCE(!mem_cgroup_disabled()); - return pgdat ? &pgdat->__lruvec : NULL; + return &pgdat->__lruvec; } static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) @@ -3206,9 +3206,6 @@ void lru_gen_add_mm(struct mm_struct *mm) for_each_node_state(nid, N_MEMORY) { struct lruvec *lruvec = get_lruvec(memcg, nid); - if (!lruvec) - continue; - /* the first addition since the last iteration */ if (lruvec->mm_state.tail == &mm_list->fifo) lruvec->mm_state.tail = &mm->lru_gen.list; @@ -3238,9 +3235,6 @@ void lru_gen_del_mm(struct mm_struct *mm) for_each_node(nid) { struct lruvec *lruvec = get_lruvec(memcg, nid); - if (!lruvec) - continue; - /* where the last iteration ended (exclusive) */ if (lruvec->mm_state.tail == &mm->lru_gen.list) lruvec->mm_state.tail = lruvec->mm_state.tail->next; @@ -5315,9 +5309,6 @@ static void lru_gen_change_state(bool enabled) for_each_node(nid) { struct lruvec *lruvec = get_lruvec(memcg, nid); - if (!lruvec) - continue; - spin_lock_irq(&lruvec->lru_lock); VM_WARN_ON_ONCE(!seq_is_valid(lruvec));