From patchwork Wed Feb 23 19:40:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12757468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B29EC433F5 for ; Wed, 23 Feb 2022 19:40:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B1B98D004F; Wed, 23 Feb 2022 14:40:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73A938D0011; Wed, 23 Feb 2022 14:40:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B33E8D004F; Wed, 23 Feb 2022 14:40:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 2EDBE8D0011 for ; Wed, 23 Feb 2022 14:40:25 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BC99095AF8 for ; Wed, 23 Feb 2022 19:40:24 +0000 (UTC) X-FDA: 79175061168.13.C43B8E0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf12.hostedemail.com (Postfix) with ESMTP id BD94F40008 for ; Wed, 23 Feb 2022 19:40:23 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-2d07ae11460so707737b3.7 for ; Wed, 23 Feb 2022 11:40:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=3O2OEu4eUhR3YtOUni3RQudnfEWV+vLSeR/9NTfWNyU=; b=jzC9fz4TzuA54IP+E5K9YaKUQaPNmQ/lvm3NX2pb3OKYpT8t0JaPn6bAt2BddJkswc CaDH8o0HuYAtisKR5ccNJFhqnT9crftdUBByiSgZ28y3XbzO3EdQaGScOTjoQ8alXMwB JBrL6fQH+JWTaChQwvObiF7Wze08of9eeQzNJBGqa58pXjb8fI0wFM01INm2APxTcq6d 0/z1QuNzzNqvlMbZmvPurZn/8lu9y9/XNtMjiBk+FAPQ0V4eXsbYDNvjkoDuhePHNvGO UffScw890HXGZdnsEUbyecWHYB9Sntgl8fkx0DrcJkg0q5QFExj/O81celFzUTPqIBKJ 3tzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=3O2OEu4eUhR3YtOUni3RQudnfEWV+vLSeR/9NTfWNyU=; b=MGMFuIdXudpuygxT74RBr1iYSJn25tHwMT3Ke90vbd+TaTPhy+3LmzfmYhFoBWTsqk XXTtRJXBbjcScrhWnBAmOFJ1KJ3C0qusvDLTTATv74DCBc6QcqhJy2/1J9Z7184tivQK IgMvkWI9WahiATY0Z5DITwlk1CUK3CY3ZD2IFdzAdz67qX8PMngdUnzXPLMCpws+xu4z gs5YKhS9Ph8DFH0ZcOXmZ66yR8OUANRrrxLLTLwP1N1EzyuaDowyJ0sBR9wapyftFY/t PjWb1X8spsL8uPsddox6lKkHHpR9pYwon3GGSx40mo0QmLH/DpZ7Ck/ialM8HoEGLTPJ hShQ== X-Gm-Message-State: AOAM530nH4XxGyvnSzSUbZsnaLhaIb+fBHqiWVCnNYtQAdgwKKXr4/WD KH3b21Tr8y3YDwgczmkBX71Z+rSl83E= X-Google-Smtp-Source: ABdhPJw7+FhbLt4ftGqCh3/S46XA43RnDHK0Jo9UuVmyeO3SvFEivpVsPC0K8bjqax6iJuYtGWwMzLT1f5k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:5093:9fb5:d0ba:a5f]) (user=surenb job=sendgmr) by 2002:a25:db8d:0:b0:624:5e99:1665 with SMTP id g135-20020a25db8d000000b006245e991665mr1151004ybf.524.1645645223005; Wed, 23 Feb 2022 11:40:23 -0800 (PST) Date: Wed, 23 Feb 2022 11:40:18 -0800 Message-Id: <20220223194018.1296629-1-surenb@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v2 1/1] mm: count time in drain_all_pages during direct reclaim as memory pressure From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, mhocko@suse.com, pmladek@suse.com, peterz@infradead.org, guro@fb.com, shakeelb@google.com, minchan@kernel.org, timmurray@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jzC9fz4T; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3p40WYgYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3p40WYgYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BD94F40008 X-Stat-Signature: ugo95pqd65rb5yay3j5w7aj4rasiierj X-HE-Tag: 1645645223-866085 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page allocation in direct reclaim path fails, the system will make one attempt to shrink per-cpu page lists and free pages from high alloc reserves. Draining per-cpu pages into buddy allocator can be a very slow operation because it's done using workqueues and the task in direct reclaim waits for all of them to finish before proceeding. Currently this time is not accounted as psi memory stall. While testing mobile devices under extreme memory pressure, when allocations are failing during direct reclaim, we notices that psi events which would be expected in such conditions were not triggered. After profiling these cases it was determined that the reason for missing psi events was that a big chunk of time spent in direct reclaim is not accounted as memory stall, therefore psi would not reach the levels at which an event is generated. Further investigation revealed that the bulk of that unaccounted time was spent inside drain_all_pages call. A typical captured case when drain_all_pages path gets activated: __alloc_pages_slowpath took 44.644.613ns __perform_reclaim took 751.668ns (1.7%) drain_all_pages took 43.887.167ns (98.3%) PSI in this case records the time spent in __perform_reclaim but ignores drain_all_pages, IOW it misses 98.3% of the time spent in __alloc_pages_slowpath. Annotate __alloc_pages_direct_reclaim in its entirety so that delays from handling page allocation failure in the direct reclaim path are accounted as memory stall. Reported-by: Tim Murray Signed-off-by: Suren Baghdasaryan Acked-by: Johannes Weiner --- changes in v2: - Added captured sample case to show the delay numbers, per Michal Hocko - Moved annotation from __perform_reclaim into __alloc_pages_direct_reclaim, per Minchan Kim mm/page_alloc.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3589febc6d31..2e9fbf28938f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4595,13 +4595,12 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, const struct alloc_context *ac) { unsigned int noreclaim_flag; - unsigned long pflags, progress; + unsigned long progress; cond_resched(); /* We now go into synchronous reclaim */ cpuset_memory_pressure_bump(); - psi_memstall_enter(&pflags); fs_reclaim_acquire(gfp_mask); noreclaim_flag = memalloc_noreclaim_save(); @@ -4610,7 +4609,6 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, memalloc_noreclaim_restore(noreclaim_flag); fs_reclaim_release(gfp_mask); - psi_memstall_leave(&pflags); cond_resched(); @@ -4624,11 +4622,13 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, unsigned long *did_some_progress) { struct page *page = NULL; + unsigned long pflags; bool drained = false; + psi_memstall_enter(&pflags); *did_some_progress = __perform_reclaim(gfp_mask, order, ac); if (unlikely(!(*did_some_progress))) - return NULL; + goto out; retry: page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); @@ -4644,7 +4644,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } - + psi_memstall_leave(&pflags); +out: return page; }