From patchwork Mon Nov 13 23:34:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13454609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B59CBC4332F for ; Mon, 13 Nov 2023 23:44:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C60116B018D; Mon, 13 Nov 2023 18:44:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE9086B0206; Mon, 13 Nov 2023 18:44:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A89BA6B0207; Mon, 13 Nov 2023 18:44:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9341F6B018D for ; Mon, 13 Nov 2023 18:44:49 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 677451403AD for ; Mon, 13 Nov 2023 23:44:49 +0000 (UTC) X-FDA: 81454563498.17.0B4BFE3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id C4ADFA0006 for ; Mon, 13 Nov 2023 23:44:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IAV9ZlI3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699919087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:dkim-signature; bh=7tJADaKroqOjkrAmUfZQotFKNEMB2n4vY0DbW8JJQxo=; b=ji0k+abqFheGmwQPUyGBDqOn4zFlE+9xiQVwrS/RASs7JSzSW3NM1KrmrRc6IXvQtl3+8d 6hACwJ9gQm1BgugRFMycyagAqz6Qi8YbfsaE1Fv9NPKnDvZgeZownp0qemeKGedoY4PV6e bV2UMhAvRIOGpOYOjdzMvmoeGVdh3wQ= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IAV9ZlI3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699919087; a=rsa-sha256; cv=none; b=0CoNIXLIEkjEjku2kf52dc5sJQXM7gDM78lHvbefL7CkzQQiEwHFOMrgHSSPEyZGcL6wS1 a9r5X4x5oJBl/6jcbrdFGL7dUTQ2unWxjSFviuXFQuNLtPU2imAHpF12MsLRnXddjbPo5w QlC9h49YN3cg83YaNJNvruqUXZjC3Ok= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1699919087; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=7tJADaKroqOjkrAmUfZQotFKNEMB2n4vY0DbW8JJQxo=; b=IAV9ZlI3zxK0BMtAvLCJwhsNyQGOThM62Nr0YeDdMhQsKsc75FiLa4Wy72pNouB4SkAHbS MXcYxdykOkdBcpVqMHVeRlRKUGMZ++EmktvH2YWtmoJj7A8zW7ulStFinoHSGUfC/u4XTs h/uXtqMw8O4X3INaBWpv4kbNjvoicEI= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-649-6N-CpV6PPMyT08pW-sOidQ-1; Mon, 13 Nov 2023 18:44:42 -0500 X-MC-Unique: 6N-CpV6PPMyT08pW-sOidQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0D90B1C05146; Mon, 13 Nov 2023 23:44:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B87261121307; Mon, 13 Nov 2023 23:44:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 28959409B55B4; Mon, 13 Nov 2023 20:35:57 -0300 (-03) Message-ID: <20231113233420.446465795@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Nov 2023 20:34:20 -0300 From: Marcelo Tosatti To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Vlastimil Babka , Andrew Morton , David Hildenbrand , Peter Xu Subject: [patch 0/2] mm: too_many_isolated can stall due to out of sync VM counters X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C4ADFA0006 X-Stat-Signature: c6zurfhc5db6pthdezykuk136ppho1r9 X-HE-Tag: 1699919087-204491 X-HE-Meta: U2FsdGVkX19U25YHIyDDl4KPS2AMSGBdQB1W6h0zvcrG1d2zHA72ReVylvV6LXKKachFiaI1TuXVbLkuPK4wKqJS8MSlHHbmItI0wrqHBHmgfiRCVvdGWWHY2S3FJqubyk06QGdfut7JAYMJD0CuNj6k5S9v0L/Tyg3jodtv7Ku4BGl48Ntajy1d+PJORR2yHpBHDPvjGwwcxB47a3upNBcD+rUbQDTCSMYe8Ixy1TRcPmOJ/0njFIqx5ZvSVoQqu4brfqNnHA6tSlqkRI4fiFaq2iieeEGNvphVzRdHoDRaVS6q+Rsw2JIc5Mh5BddFpdy7LPmnZ/jw0Xc4r5sN0cTmOhn+6RaaCiiVYInX977IvxXngnpLyVbCdA+HyG7gTIPljgvjNQNGbXLB3V6NmPiMHoVnSs7uPZLy59wVy2gIW9T+2aGRUi3nULsKVfCwnIhRsEADqI/4gJ5CmXWQDfNlA8vTc7FX3eeYb//uQRHhtbWE+YcuMWOBevueaNVH1Cab4qilwx8iggV1P9+ZBb8IMabRxH/IUmnm1E7IEw5+ebCiJ+Fd3iPrhHwqKvauh5hgePXV0Ae0tjEIBMdIDl1KxTM4GiXTGN4/rXIvAJllIeaQjM1iUmOKWPTDZbkoGN5hw8ppQw6PHa+VSh/Dv/4hQ2ISvtNN0hVg5EY+pdLxSbkZ37iWsSySSxvHQh8pXT6T5dPkaOtX6I/kr+UKsaDnigTtVnT2lF4VZ7rB5o4si2VC15icEkMr504O0v2L1XH9hFhmR63BDKRS6zE8eQmzscmItt9dt7gilhIBKkwv/FmtWJzFv6lEx8yjbTkSREEQtP5d0Wim+iWxWZptjl5ToDmY9UfoWhiB/gcOzfSJhVit4nvrgE+Pdc89JKKn2ZQmexIMyXY/9hjp3W96ujko/7HHLaW7Wvpn/A25fz5w3/OYCvyiOGweGXaK//6glCXFaRotRdPv4LmIqUR 9XZzT2fa IQQn82QOCbwQgmPE4f62d4usrOffevpTdF/XlYyZNzSW6Z/4+0BtiPwu7J1f0HiS7KUS9eWf4nLP+fl57JmuYRZ2XhNN57NBdE5+dwRIvn7n0VSYed2+hfBkCG3T8yThbmcOj1bwnLBRLNCojkfEI4byQqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A customer reported seeing processes hung at too_many_isolated, while analysis indicated that the problem occurred due to out of sync per-CPU stats (see below). Fix is to use node_page_state_snapshot to avoid the out of stale values. 2136 static unsigned long 2137 shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, 2138 struct scan_control *sc, enum lru_list lru) 2139 { : 2145 bool file = is_file_lru(lru); : 2147 struct pglist_data *pgdat = lruvec_pgdat(lruvec); : 2150 while (unlikely(too_many_isolated(pgdat, file, sc))) { 2151 if (stalled) 2152 return 0; 2153 2154 /* wait a bit for the reclaimer. */ 2155 msleep(100); <--- some processes were sleeping here, with pending SIGKILL. 2156 stalled = true; 2157 2158 /* We are about to die and free our memory. Return now. */ 2159 if (fatal_signal_pending(current)) 2160 return SWAP_CLUSTER_MAX; 2161 } msleep() must be called only when there are too many isolated pages: 2019 static int too_many_isolated(struct pglist_data *pgdat, int file, 2020 struct scan_control *sc) 2021 { : 2030 if (file) { 2031 inactive = node_page_state(pgdat, NR_INACTIVE_FILE); 2032 isolated = node_page_state(pgdat, NR_ISOLATED_FILE); 2033 } else { : 2046 return isolated > inactive; The return value was true since: crash> p ((struct pglist_data *) 0xffff00817fffe580)->vm_stat[NR_INACTIVE_FILE] $8 = { counter = 1 } crash> p ((struct pglist_data *) 0xffff00817fffe580)->vm_stat[NR_ISOLATED_FILE] $9 = { counter = 2 while per_cpu stats had: crash> p ((struct pglist_data *) 0xffff00817fffe580)->per_cpu_nodestats $85 = (struct per_cpu_nodestat *) 0xffff8000118832e0 crash> p/x 0xffff8000118832e0 + __per_cpu_offset[42] $86 = 0xffff00917fcc32e0 crash> p ((struct per_cpu_nodestat *) 0xffff00917fcc32e0)->vm_node_stat_diff[NR_ISOLATED_FILE] $87 = -1 '\377' crash> p/x 0xffff8000118832e0 + __per_cpu_offset[44] $89 = 0xffff00917fe032e0 crash> p ((struct per_cpu_nodestat *) 0xffff00917fe032e0)->vm_node_stat_diff[NR_ISOLATED_FILE] $91 = -1 '\377' It seems that processes were trapped in direct reclaim/compaction loop because these nodes had few free pages lower than watermark min. crash> kmem -z | grep -A 3 Normal : NODE: 4 ZONE: 1 ADDR: ffff00817fffec40 NAME: "Normal" SIZE: 8454144 PRESENT: 98304 MIN/LOW/HIGH: 68/166/264 VM_STAT: NR_FREE_PAGES: 68 -- NODE: 5 ZONE: 1 ADDR: ffff00897fffec40 NAME: "Normal" SIZE: 118784 MIN/LOW/HIGH: 82/200/318 VM_STAT: NR_FREE_PAGES: 45 -- NODE: 6 ZONE: 1 ADDR: ffff00917fffec40 NAME: "Normal" SIZE: 118784 MIN/LOW/HIGH: 82/200/318 VM_STAT: NR_FREE_PAGES: 53 -- NODE: 7 ZONE: 1 ADDR: ffff00997fbbec40 NAME: "Normal" SIZE: 118784 MIN/LOW/HIGH: 82/200/318 VM_STAT: NR_FREE_PAGES: 52 --- include/linux/vmstat.h | 4 ++++ mm/compaction.c | 6 +++--- mm/vmscan.c | 8 ++++---- mm/vmstat.c | 28 ++++++++++++++++++++++++++++ 4 files changed, 39 insertions(+), 7 deletions(-)