From patchwork Tue May 30 14:52:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13260309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6FF6C7EE32 for ; Tue, 30 May 2023 14:57:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2E356B007E; Tue, 30 May 2023 10:57:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB544900002; Tue, 30 May 2023 10:57:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 908876B0081; Tue, 30 May 2023 10:57:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7EB316B007E for ; Tue, 30 May 2023 10:57:08 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 174E980274 for ; Tue, 30 May 2023 14:57:08 +0000 (UTC) X-FDA: 80847224136.18.518853D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 2713B180016 for ; Tue, 30 May 2023 14:57:02 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="d9I8Jw/U"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685458626; a=rsa-sha256; cv=none; b=rvuWuhZJyLcMysVltbBHZ1e1TULTydbD1ekOqFXSNzZWwghKx4dDwOYoPABsf36BoNAHjf h5Mm0nzSv/fgiMJZta+hXKSCp0f4+QS4GLqyJvwZlGH9md25a5Hda5dldmQHoxwnjn85MF PV+uWd42Fdt5B8aRSR0+Y3nnhz3vtRE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="d9I8Jw/U"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685458626; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=N0hJfqn4EoxiBlmJOP3aHXOKCKD0R3BQJl7dwB2jg74=; b=W0R0LBbXzHx66wSfqog9XPj56Zq1NhMGau3IGfalXGvMge3Ul3ulKlgqwj6VIobV6dc2fX nKuuA+51nRynGu/aAf1zBjGWIDJeSdXOI02SuWeJCX3Y/Dwe/Tm9V5ZwuzxaDNalgvRTNs inlqBXN56B/hwzISTP8E8lCFQDTUNhA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685458622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=N0hJfqn4EoxiBlmJOP3aHXOKCKD0R3BQJl7dwB2jg74=; b=d9I8Jw/UzrGDyTzlGwzRj8r/HbLPtJTadiO86tlOb0MUaey/a5PCV/fh0yCYmDc/Zpr0IK PJplipWI8RilpwxMDP7yxe9SCPfWY8v2Lpf/Ew2pewlL63/PxdYWD2hCdCX7NiE63NlM8k YevwUXV8cu/rlsq2kiazyXLSUKkznCM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-326-uE-WUPsyOkefq9LDnYZpnw-1; Tue, 30 May 2023 10:56:56 -0400 X-MC-Unique: uE-WUPsyOkefq9LDnYZpnw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 74F20802E58; Tue, 30 May 2023 14:56:51 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0F1B62166B25; Tue, 30 May 2023 14:56:51 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 49105401CE3ED; Tue, 30 May 2023 11:56:33 -0300 (-03) Message-ID: <20230530145335.677325196@redhat.com> User-Agent: quilt/0.67 Date: Tue, 30 May 2023 11:52:35 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH 1/4] vmstat: allow_direct_reclaim should use zone_page_state_snapshot References: <20230530145234.968927611@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2713B180016 X-Stat-Signature: c38tnfk7tshj681yatthsxsutmbmgzbq X-HE-Tag: 1685458622-590624 X-HE-Meta: U2FsdGVkX1+d2UY+7kLjWEo7wumN7ss8RnxHvA6vL4L7Qe6dsE27PU/6BJc+x6uaxvmrEO3jIy3OTgTerlWB+LGzVAW0zjtnnb2utZExJ7z1Ki4ZP8IyEiPl3xfACC7T6A2+Gy+IY5HEz5kfnMJgDoPvGJ9787stZDEJsD7l3vvb66R37s40DaqAm5UACS2tVF2oAYHyCwWAwfnLdjYj0Pu43Da/u9Mg5Go0fwMa6zz0MurwRXLjdmHdUojULbXzehXnxbJlw1F9tYQcK9KSx0+uyN+2mUledAmUOIrVws5i9b+/fxwZLbg0Kb2MmwrXuANFiQ4WkUGJizJ+fBahI+2dz5tRZMuL8fSV/UglLn0OdUrw7MvdgoWBjMcjvJmDzr0wmFdotEupga3rkrtPQsdBb5/c8omDCVPr7rs1Ui4vVz2ARH9cLjPPwWhjfTkegUfFP4fEavx0IIX13VNouFtDC2mUHSocGTgvLb7d0YtfqWkwev066+UOlQW10bdr8w/phuM7VyiDWalSBHGLJgJgKxImmbulOYwqVhA8xCI6x41CZ9Qosa9+XE0tvuFEhx1XnJ83d02DaW1ny5rnP1CtmzgYziI6ym/3nuaGNOwYdVdHoBLJS7yihdJjyNFdOxCaBzV3w5eYsBDVntTaz3Kq7ePC+wiOd8yYhpEZibr6Nu6BYnHBaxgRArt5iQuJ8Z3lLkEaj5Jj35AsidzZsoOqip0Dt2unNuyfTS5lAcIeHwNSzmQdNPPrbfUGAp/IwOHMO/jr1zcIkk+UwHqUf7OQFo+moGumSxi2Tgpys0X+f1EK/+3SdEC1st4OSxalIf97QlMkZjnCzYsurYvcWJ77XkEb478dxFiosgI/nQOs8tWm6Oxz9+xmI6pfgOaqy1xkl88h5vvcBvNaQO6mqD+fGoOOvs42uOzB0DObrLHdYONJWGZoI5v6hDjeilh5FjxEwz1wxU8GBnEF2Fv EbqFjjv3 1STUeJF4MWE3Pay9wSVAjkAOK4b7U5vSfGkxxIKfeNSb8EigYKdTyYoyLF4FWcylIZlWv48Mr+/tMCTHKItlhqdcRe74OINMNHUrSo/ik4xSOdKtu6DkdqRaXqAV8aVAptg3Le+9shHq8dHuzXErFA0NPzl0rIwx6phBew3SER8Mgog+wdIHkRCqy2eAqCWkt3hStX11UrxwHY7o+e9/1UBDghddIjoQ+lJXpLZFBkmJ0Lz9WY4vu1reDbymZ8jZxgI6UShGGQqPyqOWsAGejmqfBFKfKTCiFP/WVyfzzkey/KK/TtPkDJisA1BBBdFOxgErjTp5GicB9fCirk3Xjlg9V8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A customer provided evidence indicating that a process was stalled in direct reclaim: - The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES]. - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty. - This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely. I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet. - In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat. Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark. Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used. allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path. Testing: Due to difficulties accessing the system, it has not been possible for the reproducer to test the patch (however its clear from available data and analysis that it should fix it). Reviewed-by: Michal Hocko Reviewed-by: Aaron Tomlin Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmscan.c =================================================================== --- linux-vmstat-remote.orig/mm/vmscan.c +++ linux-vmstat-remote/mm/vmscan.c @@ -6887,7 +6887,7 @@ static bool allow_direct_reclaim(pg_data continue; pfmemalloc_reserve += min_wmark_pages(zone); - free_pages += zone_page_state(zone, NR_FREE_PAGES); + free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES); } /* If there are no reserves (unexpected config) then do not throttle */ From patchwork Tue May 30 14:52:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13260305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6921CC77B73 for ; Tue, 30 May 2023 14:56:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2D576B0074; Tue, 30 May 2023 10:56:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC0E36B0075; Tue, 30 May 2023 10:56:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 955F8900002; Tue, 30 May 2023 10:56:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 824016B0074 for ; Tue, 30 May 2023 10:56:56 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3135A140227 for ; Tue, 30 May 2023 14:56:56 +0000 (UTC) X-FDA: 80847223632.14.5D94F0F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 160BE40006 for ; Tue, 30 May 2023 14:56:53 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I+am0RZN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685458614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=B/GSKf6UEn0c42XXKcBB/6CIce6RG7UgO/Nmikr6pJA=; b=myATcJ4wHrpm41msnXfEY87/aS3ML2/3F58sNH35MuCSiUhUu1lpADrqfJJdtIwA+Av+Dd PV2LNk+0QBeUqgSql6DgaG+HYIebme3NVRCdF8QihfVRzULNKLXItL78yogQ97loRb5IT9 qA+rebbF7vv38yyFqGRSgk2zd4M+8do= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I+am0RZN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685458614; a=rsa-sha256; cv=none; b=445W4ZJZ538GlN4xzVxX87bwpaINF9GBHAjzlCSri5v6OxM4LICLfVPxwcw+BQHM4sfvK4 8oqGNQlAOa7lYgcMtlZ/WimF08rqiyAcxUQuZVO2wS1bFgJ426yQP1nZddTRwfvo6Wwu7K WKEQf2lNgkcx3GOGC0W77oQ/BE+kK1Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685458613; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=B/GSKf6UEn0c42XXKcBB/6CIce6RG7UgO/Nmikr6pJA=; b=I+am0RZNTTd+4SZTHjd0uxpBOau02f2Y/oDeEpOC1YGJqPr7QDe8mjHymkHHEdytoQytGf u4ImP301WtV+oexy4+n/0GsISrinafw5Xj9HGAhl3rlusMzxx33NIMqUy65zAMunhF7XkF beRWUfg0kBsVQAkp0F2PCHg/i9YKROI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-42-qhAbuYgmMEKTml1kAmu7gA-1; Tue, 30 May 2023 10:56:51 -0400 X-MC-Unique: qhAbuYgmMEKTml1kAmu7gA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 28452185A792; Tue, 30 May 2023 14:56:51 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E7445420A8; Tue, 30 May 2023 14:56:50 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 4E9DA401CFE3F; Tue, 30 May 2023 11:56:33 -0300 (-03) Message-ID: <20230530145335.828634764@redhat.com> User-Agent: quilt/0.67 Date: Tue, 30 May 2023 11:52:36 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH 2/4] vmstat: skip periodic vmstat update for nohz full CPUs References: <20230530145234.968927611@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 160BE40006 X-Stat-Signature: 46sfo8noz1afzc9r4twr7smuzqned8hd X-Rspam-User: X-HE-Tag: 1685458613-30625 X-HE-Meta: U2FsdGVkX1/7/ZV2HytVpdZ4MLtO8x/HqNuDtlFCpnxQrq2LvBCBp2ZThepON/a5kbtc1+sA1YWBmeCRK5BR/hIMeRl0Xu53uAWgqwSLlkGWPQzE4vrE7VXlVx/Z8cjEQKTWQp32sHyJvL/LA9/iSKH1qVZSjqsaSKhHPHkxQb/3xfIprZSkJ+t4P5bX823T7VY6Mx7Ssn5MF8JjJuQn7YtFYos7oUjsLSzX7J9dsz3O1YSNozF3n5QtCw3rrKD9Rsovuw0h9/mAjC+33/wgxb+7cP+xV5gPsg3RwwDtPE2NiVDoXcvyO1YNSTmqR9gA9q62FWvq0HrWCM2oZhjJCpCGLwtlTj4XK+eSvk32ObF3BmFToio3qswtDKwySdDe3FFbXebaUjHJ3cT0Bh+5sdWuwh9BoapDYVvcMoFXiqMYR1pSAtqlCXLsGGDBk2PeZ/HWoDCGAoAST2ZwNSz19bbmB6U2XMlzREKYktQZwMPF5oPCeyGwZsU7QxCxtBii2u5XGpjkOrcVhz/mMqgCsOgvAJP4Y3D6qIaJC7l7YEdeIXpzPmKzXl43tZbPEa0Vpm1CGXXsF+xBr+taCFXjfBKHU/TA6AWzeo+6kPwq9dV12NSaOHDEXTARXIGyv8iIUS6ukl9TkZOqjqe1xc1w3Z/VyqgisiJE76sKVWr3DztyuNkqMfNIwewe1Mb/iuM6m0+yEbb4yoWSCLop0i/RTfNqBt9JhKCHd5OiGq7y/QeYMGF29oEU/6rVeGWB6TA42IS3pveGcPHtUOaQJ2SFQ0W6DVJzooXC8yF3H+2wXbtH7Tx8GVBNM1NnYK34yxG3RoLRTm22v4zS+4/zm+2rw15PgNc/YBDhYbSDZbLGP/j6J6UNhtPPot0zbUBeyf8Tm4JRVDYXQgAMh8MEoIM0/tvXWI/FEKDzqk3xr4QIYYF6g5OCndxOTDHlFCQPCIU6tyMzO7PbiHS6P0QH2qZ kp5nqNRL bMakywFLv/lKA9S8L8VUOll7t2YWERfjJ5cgD9s+n0VYX7jU0jxLr1+q6d2XXg4a6wSRUn9Snj+Kg8fdjjk/4XarO3ilUxHQ+PE2KxaFXCaS5psA9KjZSe6eIGp9j/NsUOKBsYm5okqRH+zZY+f+j4o7TSt7W83QkIP2EJFKEmpivsBwA0/hlJt+wMpQrhB7Rbnu1nsYO9SoQUBL4SpFSJlqU+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The interruption caused by vmstat_update is undesirable for certain aplications: oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... The example above shows an additional 7us for the oslat -> kworker -> oslat switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold. Skip periodic updates for nohz full CPUs. Any callers who need precise values should use a snapshot of the per-CPU counters, or use the global counters with measures to handle errors up to thresholds (see calculate_normal_threshold). Suggested by Michal Hocko. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -28,6 +28,7 @@ #include #include #include +#include #include "internal.h" @@ -2022,6 +2023,16 @@ static void vmstat_shepherd(struct work_ for_each_online_cpu(cpu) { struct delayed_work *dw = &per_cpu(vmstat_work, cpu); + /* + * Skip periodic updates for nohz full CPUs. + * Any callers who need precise values should use + * a snapshot of the per-CPU counters, or use the global + * counters with measures to handle errors up to + * thresholds (see calculate_normal_threshold). + */ + if (tick_nohz_full_cpu(cpu)) + continue; + if (!delayed_work_pending(dw) && need_update(cpu)) queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); From patchwork Tue May 30 14:52:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13260307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDFEEC7EE2C for ; Tue, 30 May 2023 14:57:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 478516B0078; Tue, 30 May 2023 10:56:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4046D6B007B; Tue, 30 May 2023 10:56:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A60E900002; Tue, 30 May 2023 10:56:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0C6A16B0078 for ; Tue, 30 May 2023 10:56:59 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BBE28ADDDE for ; Tue, 30 May 2023 14:56:58 +0000 (UTC) X-FDA: 80847223716.19.F8FC6AE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id C2E2210001C for ; Tue, 30 May 2023 14:56:56 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QOrbMJjH; spf=pass (imf14.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685458616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=zB34+gA9q6N2mJEfFmSt2POPWMX6epdGtnjhsa8vBnc=; b=n/c7pNxUvAzMlpbgSEg+nN61ya61CiYwJVxZ57oL4WRTsM/fN1OksdnUP/8g4kCJSE/GnS 77K8p9xTG33XVjBm5qwE0930USmoe3EGFPRUIgxyD4XDz0SDGGHjGGHhKQsrowKsmGrwvg qVhs3HmyEzPqOmEn8mpbwMO7UFSGLCA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QOrbMJjH; spf=pass (imf14.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685458616; a=rsa-sha256; cv=none; b=hHdwpcoW7Co/Ri+YUNcEUx1meGROqY3WKTY1ghy8ByOVotJoSyedGSpI3hVUhtu/2DduUt WBhkqAE/eGL0MMFita960wSuvSiRyXTWOCAIi0HibtqpcY9iFiKNJp1AUYzEqyMev1jiiS Y8wuIKme6bJ/aPzrBlWG/M+rn3TjEjM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685458616; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=zB34+gA9q6N2mJEfFmSt2POPWMX6epdGtnjhsa8vBnc=; b=QOrbMJjHRirQBvc0Bl8Wx0h//B4ALgfRuBsm0i/8uuAvuzMvVQAlf9mCTnaK+CDpHOPEK5 unbYCeVIzZdQmFXCnav1Kbgd09a0CpdNQHpHCT6KnhpeJPjQ5zFFsHBb2VRjFyK3MZQlFl sAU/4urzbQnLuQkyYAwcNpzE7pgA1jA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-471-M5R71YDnOC6zHMxN_4ZF3Q-1; Tue, 30 May 2023 10:56:51 -0400 X-MC-Unique: M5R71YDnOC6zHMxN_4ZF3Q-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 25935101AA68; Tue, 30 May 2023 14:56:51 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E9F62C154D3; Tue, 30 May 2023 14:56:50 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 57A59401E150C; Tue, 30 May 2023 11:56:33 -0300 (-03) Message-ID: <20230530145335.930262644@redhat.com> User-Agent: quilt/0.67 Date: Tue, 30 May 2023 11:52:37 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper References: <20230530145234.968927611@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: C2E2210001C X-Rspam-User: X-Stat-Signature: rxtue9pw681ossj4jzj4btrnreocuteu X-Rspamd-Server: rspam01 X-HE-Tag: 1685458616-186931 X-HE-Meta: U2FsdGVkX18y+Rq9PoroQDplrg7mP221L4UvgJm/MtaKYqNOwixifg1GWg1LM+yfYdxeJnO4Apn9MfHcJA/j0BI+3CVuo+ppEKxH0hiTBrn82Yw+ILiiWJ8PsxhnDvzzuOwrb6WtmXwDFsw70oXF27WfdSjV/a8quI5s651nrJQx+Nt2HJyy6ztg7U21lavqzlVzw0JxpdAkU1zOGdkJoyecA/HA5ySvQ++ghipBK4Po3Ao0h3v9kIWDp80HX1lkYKMWisRPryCJ/489HB/kXRQOIwy59eZvG3DVQBOiGjfjJRnVQUhdzGHG9EaKebfF+94DczgOfCPatVNdpFQVC3PKfXr0YCGzqqFxpdw6Uspo0MKRFhlPmYD8T5PEWBLazqa43xueIGkYzhm2O10zmmi3fiw+Rkjf3XOPyTJAE1OYTUiJgdgJibbNRwVaV6M/qdWKqANZ8NBv/CQiFa0+DhsQ09gBayF8a/P1tt0ScJ1TmrTqKXzcr1TRgIEGF43sHFMKJNc2jmKWSB37hBEfL1yNByb5VepS9f82cAW7MWjUNXKtjeFhUscDFMf7z80CCWVMKWUCd0/H5m0C8FJzsDk7ObQAboUoZ3V7xFcAgQm5RIAGxq3Uca07Oo+rUrBEf7mPgQs4fPM4A7O/Wrx1Ug2nYl3uWvxpapRQEOY/a2T14ZIDUlTSgw21gUhLK+wLPpMQ18fKBAxIMgREz7bVQEUGnmjstYma8CvBAVfSX/neQQThVLx0/QsoGVYH8ij3JYBh4xk0hVE0Pey+JBQi4s33CMIfvFTDzUoRTCUy8nheQ8YHczMMfA28CxPLx2VeMzhkj0jq9qdcWs6pxwv49IjIChlpsky4+KzJOFq6hc0AfG2DLI28Mf8SfprsHrQnNwW4Y2/22TwPH3AhI7zpNjFGDkQLhLy4oK0FhlnVOF0+cqoqhbtK7ww0Da+88Os3EKfNTKofXWqbQfuahal bWoHF3pY lJIJdkhvy24nWXx7FZM1TY5qyH8CSfSRKRi700A8/9J//E3sH/ubJBlKerNGScth9mo2mi3ZdMgJSUhTKmwXRFv/XjUxSoIozU+Zv5gxHR9cHsOOTzJPa+wU8ZHfpFADy3E0EQVrm2+9tTmeuvScduee5vo0Dbk68WX8ZFN8np9vnYesiCu25bWJGPukmnLQQmrcjvzCyPZWbI/aBPsGvDl/eD6N+YEOvNuEz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/workqueue.c =================================================================== --- linux-vmstat-remote.orig/kernel/workqueue.c +++ linux-vmstat-remote/kernel/workqueue.c @@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun return 0; } + +/** + * schedule_on_each_cpumask - execute a function synchronously on each + * CPU in "cpumask", for those which are online. + * + * @func: the function to call + * @mask: the CPUs which to call function on + * + * schedule_on_each_cpu() executes @func on each specified CPU that is online, + * using the system workqueue and blocks until all such CPUs have completed. + * schedule_on_each_cpu() is very slow. + * + * Return: + * 0 on success, -errno on failure. + */ +int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask) +{ + int cpu; + struct work_struct __percpu *works; + cpumask_var_t effmask; + + works = alloc_percpu(struct work_struct); + if (!works) + return -ENOMEM; + + if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) { + free_percpu(works); + return -ENOMEM; + } + + cpumask_and(effmask, cpumask, cpu_online_mask); + + cpus_read_lock(); + + for_each_cpu(cpu, effmask) { + struct work_struct *work = per_cpu_ptr(works, cpu); + + INIT_WORK(work, func); + schedule_work_on(cpu, work); + } + + for_each_cpu(cpu, effmask) + flush_work(per_cpu_ptr(works, cpu)); + + cpus_read_unlock(); + free_percpu(works); + free_cpumask_var(effmask); + return 0; +} + /** * execute_in_process_context - reliably execute the routine with user context * @fn: the function to execute Index: linux-vmstat-remote/include/linux/workqueue.h =================================================================== --- linux-vmstat-remote.orig/include/linux/workqueue.h +++ linux-vmstat-remote/include/linux/workqueue.h @@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor extern void drain_workqueue(struct workqueue_struct *wq); extern int schedule_on_each_cpu(work_func_t func); +extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask); int execute_in_process_context(work_func_t fn, struct execute_work *); From patchwork Tue May 30 14:52:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13260306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BFE7C7EE2F for ; Tue, 30 May 2023 14:56:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 184CD6B0075; Tue, 30 May 2023 10:56:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16B526B007D; Tue, 30 May 2023 10:56:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAD7C6B007B; Tue, 30 May 2023 10:56:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DCE0D6B0075 for ; Tue, 30 May 2023 10:56:58 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6EC65ADFEB for ; Tue, 30 May 2023 14:56:58 +0000 (UTC) X-FDA: 80847223716.22.DFE4883 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id AB909C001E for ; Tue, 30 May 2023 14:56:56 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KK44kWQt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685458616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Zp+6wKN5pw6h0b4P0Mp9cZhOYKe0APjLNDcaJDBGqc0=; b=YK0oC+fQigN0XoXwMqn/RyAQjtIL+e4CzHeF0b6OZ4093GzSkMmbvamJfFt4lIfIzU8/Kh npIRvqM1Z1PpAaEsvhJT2q1KgB4VW0cLWCKCJOaFMYJN6a0O3Sj8VmNRLB73I0Zcu6F9gG bnCDZhcfGzJ21ItP1pAbpeFUY+perPU= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KK44kWQt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685458616; a=rsa-sha256; cv=none; b=X6stKSvuKS0DfNKigelyheOGDBdvzLAmkyQCBt3pU/UUnZdmJu8C856cnViC2gM6+5QVqX CZQ+6PeVBDvlWOVEmMBgAtc9DVY79JFoByhEGMztO10ozANMiz6kQaVKQfd68XTKxISBIY VG1Cj9S69EbQ6Y1Y5FfoyMUQ3q8/+pk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685458615; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Zp+6wKN5pw6h0b4P0Mp9cZhOYKe0APjLNDcaJDBGqc0=; b=KK44kWQtOd6KwlTRTlwFn57e9430/h84pRMj/h5h7i0+uMBUrpwtXlPyGAdQ4hm25IcuAv Un3PZnuk0CFwM9N9we09QX5zwLIeX7J9m7kwueECKzJti3fcQnYUxSaCHEBKSUP9tyRdi5 nIL+wIuvLNGKaySrXBa4qS5jzvxFqnE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-609-WcYEjc5XOeSOkwHfXbc04Q-1; Tue, 30 May 2023 10:56:53 -0400 X-MC-Unique: WcYEjc5XOeSOkwHfXbc04Q-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 241BB101A53B; Tue, 30 May 2023 14:56:51 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E742BC154D2; Tue, 30 May 2023 14:56:50 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 5C58B401E1522; Tue, 30 May 2023 11:56:33 -0300 (-03) Message-ID: <20230530145336.155097348@redhat.com> User-Agent: quilt/0.67 Date: Tue, 30 May 2023 11:52:38 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH 4/4] mm/vmstat: do not refresh stats for nohz_full CPUs References: <20230530145234.968927611@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Queue-Id: AB909C001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: to6i7t3s8s7uxkg9ge5a1yswsfrsoh5r X-HE-Tag: 1685458616-663774 X-HE-Meta: U2FsdGVkX1/EUAT4VLsENvTBb1yQnHt3pvheCGZRJpg9exeLalUGbLV+JPHvUce33frJnJ4ss4qC1xz2jFL23ffOupos2vB8Kg8ojIbsJfEDoaHmhBQn0slwDv1RruSHuWgaCXs5t4Vy8eCszb28YXSSio6noIumUymh3Hc9+lpTSGc8II8Xxy5LNvlQVCGA8tWS737NiYVBHhZX1W7+M6a++9QwLTnOcTLAo8+ALSJ12Bnk7DwGW08Vi14nd2kyPZ32yd9FRgYX9U2nykntj2zDBBNpXrlnTDhzz60AE9MiNJz2Hyt1FNlTUZKOBtqvvMuI3kHU9/PGETft3Z5wUtgI4AvwcFIuHF2qWiMlfiXWsiqpAAbSGmkFDHDLwq2RPw8yroOu/3LMQX/H4TU9KTALnzFjMGp0Ch+3OX6roDnCMY2cXPkiNad0Micx2YFBsFbjgUYIG8PU25t9sR8WLQ5Z2z4wfuX65YVwUc81NMuoAAJlHEG6rk0V31bI7/KuLXZQP/DbHy/834d2UhdWDbyGppHTviTU5ayRY+dbxkfzRsmlWBGu9LuhcyHOTjokDfP8qbO1Yt445+LNwVN2l+gjvs8Qrlfqz21BmDLsgS77wYrM7xhrSuBCFKc9eBDpLZQ5IKpPDZbV98ZiSRHFyjNgJtYoC0UL+OIInAZtaATSAgmlohQTWjMQw8MVz32fLMpRzdp2XqPBhXTcs3lKF2S5C0x9IP7CQbUHp1C6+YLhbwOegMqkowaXSp44NfIZZ4ALIYtJTjRhHLfkOHseOuKj5leuwi3nxDfBxiOi6UXEh2UfYNd7+jnmP0AXt0Xjxooxu85CgGRVTjTh29OmY0d7XZGseK+e10PKRGNokeCnZGw9ti+r2o+OiyWiCjdLPcsM1E48cEJYbDmX6Df7Op4ddne0NGQVKO78goe+O1LhWICjr/SIO2MkJAg4W/SQ8v5YDiNNqMPNUUmCRAY MZP/7LVe 3vr9csgrVUiWTa2T7SDnmmcp97JMFq6StInQNNisWcDw4e9x8zXSNik47wSgQTsof93vxOeY5Y7tXW+H2P4iRWUSkHiXN+EEqsf8B6HN6BQqgHggwzXt1LFy9+73/am0e0SHGS4Rx7ezsKEicBhZ/DxAGTu3EQQxXVCWFma59U9w1MJVwblqCgc7aqxX103Y4FyafQ0krlh0NFgw+POmczS5VFZQXL2hohefx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The interruption caused by queueing work on nohz_full CPUs is undesirable for certain aplications. Fix by not refreshing per-CPU stats of nohz_full CPUs. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1877,12 +1877,31 @@ static void refresh_vm_stats(struct work refresh_cpu_vm_stats(true); } +#ifdef CONFIG_NO_HZ_FULL +static inline const cpumask_t *tickless_cpumask(void) +{ + return tick_nohz_full_mask; +} +#else +static cpumask_t empty_cpumask; +static inline const cpumask_t *tickless_cpumask(void) +{ + return &empty_cpumask; +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { long val; int err; int i; + cpumask_var_t dstmask; + + if (!alloc_cpumask_var(&dstmask, GFP_KERNEL)) + return -ENOMEM; + + cpumask_andnot(dstmask, cpu_possible_mask, tickless_cpumask()); /* * The regular update, every sysctl_stat_interval, may come later @@ -1896,7 +1915,9 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = schedule_on_each_cpumask(refresh_vm_stats, dstmask); + free_cpumask_var(dstmask); + if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) {