From patchwork Thu Feb 9 15:01:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2EEDC61DA4 for ; Thu, 9 Feb 2023 15:33:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 862B76B007B; Thu, 9 Feb 2023 10:33:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 555DB6B007E; Thu, 9 Feb 2023 10:33:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA02B6B007E; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 85E2B6B0074 for ; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5A63216095F for ; Thu, 9 Feb 2023 15:33:24 +0000 (UTC) X-FDA: 80448147528.17.EA44765 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 74B5B20025 for ; Thu, 9 Feb 2023 15:33:22 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QivdhWtA; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=MUuTZIs6BsFRBaRhUnBdlbmP5t5Pk9DvEsB65DY/sFEfUQi/nGVz47YOku7i2qBvRvFYsR yVtW0Ct4KdpKuhOHui28SNeFCrpQuanKYO/ynOfHUVB+jIGXcmhKVecEJ2CmHnUsSrq7DH UjHhMQ5ZgCiWnI3OGKoof4TbUslVah0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QivdhWtA; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=7C1hmy1E7+0MQkK/mZzmUZ/gKoFVIBd7d2YZY/3tmcQ=; b=m1VqNzAYXe45NVPDiyuTDxwntmEB+jnMDBfCGLtrJFepvarTGiwI9cIC9tXuTPprkb+y0m 8/4C7TG6vwT7sSbz6ni82VSPGxzBb5poUpB6vXnclCFwFhgU+lmn0mPopei2eO1citv2Y7 MpT5Y/mRMOa+tauylYvSQQ1RpQM1nCM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=7C1hmy1E7+0MQkK/mZzmUZ/gKoFVIBd7d2YZY/3tmcQ=; b=QivdhWtAjo+dpj9QfLMVel84Y3Y9CGg1CN97RKTOhkoDU8AK1uQQCCdJ55iooAtZIhKwFU GObu7ZAiQ4NCUxUHqZGbZyL29yOaMLfv0SuNSSB1LxvjCnSDIg1Kl0rfctevYXOFLMe3k7 /Ox8NL/qypZ78WCauU2hT0+kbsRlm4Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-532-6n2BmqIWMciXOAiOSVn0Kg-1; Thu, 09 Feb 2023 10:33:17 -0500 X-MC-Unique: 6n2BmqIWMciXOAiOSVn0Kg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C0E6E802C1D; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8DF9A2026D4B; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6390E403CC06B; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.656996515@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:51 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 01/11] mm/vmstat: remove remote node draining References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: 74B5B20025 X-Rspamd-Server: rspam01 X-Stat-Signature: 5hhn6m9s8sidasgtrh5ntoodwd6iyzpa X-HE-Tag: 1675956802-643740 X-HE-Meta: U2FsdGVkX1+wOYU7p25dNF9JvMLhynJSoZkNXTRsu4TusRNmWggVO8HvtXGYPSuExtqDNjDEVH7bvc1yUrvNSQk0NWY450XsVHV0I4NAbugpc3X/fsJWp+nXEY45U4NHEtLYgetcXa59atHntBt5uLyzXZ3DvnJCRveqi8xYEEdYQy/bUE3BOyi/IUMpce8eAtmGVaOrDfOMij9MOcLUCH8Y8YNJnFRrfA9+0q6dbeCUTdKeHtYjZX7e97oQ+n1oZI448dU1nycGumBJRxnTfEKfV/XcGRiBj0Hcr6WP5plD+JFvOjqYXgRS/8+0oid0j9GoVOSlc+C/26lZ3G3JKl6i8yHWiJqsKaw2dS0QqRVGXyNYP0oFBz6BMM+T4ArqvS7tD9T5ZukiyVMqUPoY3umpxVc+tbyTioeYSpHfr9hUznvPBD+eqaVvCvZYStP7J2tF7F0d2ZJOn1bCvP+0PPqlCoLFUpUxMlr0lBUtqLzmOkSkzVVr9iXCz1PoenpFSA9wI80GwqsfAXuj++ko4x7yIDDb9VmQMpjE92AqyfQRSqEw4LCSsQqs72BYATKGZtuLg1syOyh3uGrckNl7miKDUTph9Lt2yF6DlT/hVl8vgDopIzSphHTaK6r7Irz5VEgdjDPUzym6xfie1YtM7COFRn2HavmYZxqPu3gmzIibofMEydOq08ayHGKZKwMlOvtqZ95PktrTs4Tukwf6yJg6ybyCNbJXn3FsHFbPhM+71CVz/VrRMAusNhp0wmOWdimBRuoouXhpA5cSHLm9wRias6neazUWygTq2El63Zg3VV3bMlhtyTNYI2S4cRPhfVO2oUE5vjho+4i7cg5Se6cUGH9NvTuF7cGtwvj99p0vqggUMezuj01tB/jP8Q28uLVAwNihXJ3/0vaOxXMQzyQ2hOOKldp2DnJtfggBb3yMDIdJBkRZLfd9WBuclk0ze3Zd1c2mPQdafDyY8kF g7uDkafu vFyRmaSvvyi6sePnz/B9uGAfvQCpO09TiNCg5R8f6jaAgpftiVY4ETKxFDFMl4bX0BzKzV86WHsJzO3DDSeut+VTpvLle8bdDNcnQRBgctKHJVol0vqkNv05wYGQ8AqJn9C6nqII9JUMm8ODhuR0nEro63J1ITFLtq+Wev+GQciIRj5c76jJeHK9NssDYqK0Ow9sQSDlZTaSZ0+DwmK1/V/lureuue4p5RgTg6TY4qg4GSP0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Draining of pages from the local pcp for a remote zone was necessary since: "Note that remote node draining is a somewhat esoteric feature that is required on large NUMA systems because otherwise significant portions of system memory can become trapped in pcp queues. The number of pcp is determined by the number of processors and nodes in a system. A system with 4 processors and 2 nodes has 8 pcps which is okay. But a system with 1024 processors and 512 nodes has 512k pcps with a high potential for large amount of memory being caught in them." Since commit 443c2accd1b6679a1320167f8f56eed6536b806e ("mm/page_alloc: remotely drain per-cpu lists"), drain_all_pages() is able to remotely free those pages when necessary. Signed-off-by: Marcelo Tosatti Acked-by: David Hildenbrand Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -577,9 +577,6 @@ struct per_cpu_pages { int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ short free_factor; /* batch scaling factor during free */ -#ifdef CONFIG_NUMA - short expire; /* When 0, remote pagesets are drained */ -#endif /* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[NR_PCP_LISTS]; Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -803,7 +803,7 @@ static int fold_diff(int *zone_diff, int * * The function returns the number of global counters updated. */ -static int refresh_cpu_vm_stats(bool do_pagesets) +static int refresh_cpu_vm_stats(void) { struct pglist_data *pgdat; struct zone *zone; @@ -814,9 +814,6 @@ static int refresh_cpu_vm_stats(bool do_ for_each_populated_zone(zone) { struct per_cpu_zonestat __percpu *pzstats = zone->per_cpu_zonestats; -#ifdef CONFIG_NUMA - struct per_cpu_pages __percpu *pcp = zone->per_cpu_pageset; -#endif for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { int v; @@ -826,44 +823,8 @@ static int refresh_cpu_vm_stats(bool do_ atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; -#ifdef CONFIG_NUMA - /* 3 seconds idle till flush */ - __this_cpu_write(pcp->expire, 3); -#endif } } -#ifdef CONFIG_NUMA - - if (do_pagesets) { - cond_resched(); - /* - * Deal with draining the remote pageset of this - * processor - * - * Check if there are pages remaining in this pageset - * if not then there is nothing to expire. - */ - if (!__this_cpu_read(pcp->expire) || - !__this_cpu_read(pcp->count)) - continue; - - /* - * We never drain zones local to this processor. - */ - if (zone_to_nid(zone) == numa_node_id()) { - __this_cpu_write(pcp->expire, 0); - continue; - } - - if (__this_cpu_dec_return(pcp->expire)) - continue; - - if (__this_cpu_read(pcp->count)) { - drain_zone_pages(zone, this_cpu_ptr(pcp)); - changes++; - } - } -#endif } for_each_online_pgdat(pgdat) { @@ -1864,7 +1825,7 @@ int sysctl_stat_interval __read_mostly = #ifdef CONFIG_PROC_FS static void refresh_vm_stats(struct work_struct *work) { - refresh_cpu_vm_stats(true); + refresh_cpu_vm_stats(); } int vmstat_refresh(struct ctl_table *table, int write, @@ -1928,7 +1889,7 @@ int vmstat_refresh(struct ctl_table *tab static void vmstat_update(struct work_struct *w) { - if (refresh_cpu_vm_stats(true)) { + if (refresh_cpu_vm_stats()) { /* * Counters were updated so we expect more updates * to occur in the future. Keep on running the @@ -1991,7 +1952,7 @@ void quiet_vmstat(void) * it would be too expensive from this path. * vmstat_shepherd will take care about that for us. */ - refresh_cpu_vm_stats(false); + refresh_cpu_vm_stats(); } /* From patchwork Thu Feb 9 15:01:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81F2BC636D6 for ; Thu, 9 Feb 2023 15:33:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24B4E6B007D; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FC9D6B007B; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1DA26B007D; Thu, 9 Feb 2023 10:33:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C77186B0074 for ; Thu, 9 Feb 2023 10:33:23 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 34A11120FF9 for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) X-FDA: 80448147486.03.EC5EC1A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 35FDB40006 for ; Thu, 9 Feb 2023 15:33:20 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="RSiF/vwJ"; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956801; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=Pr6E1KYScttaLJBKuFCJZg45mWV9oYykzlNJcW8JP7Ndu2XbTlHtzNpE2ze5ErsAgZCf3x h/q8oi4telgzpLEP3zfCfKkN79tDh+8BfR+7GbSOe4NBiE1K6vxFDCir7qA4eNtGFAuM5J wXc2uEp2YMyMaCQZnz3Opa4LJr6xWCI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="RSiF/vwJ"; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956801; a=rsa-sha256; cv=none; b=5Fet5+owqVP3FjdRtybiu59t0tY9LEXlwgWl1nlXyrLMSHBrQnuj480s56UMW7fWuxm8WB 3n6b32CPtjYae6swraLlfrdckbnItKEu80rB2mBGgal0rR0sYhU/FcMoVkEAKbHa4wWX/z C6vSI1sDV9xQel3MyO8KYavVnmZHgys= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=RSiF/vwJQxexL4E1hJnJrAHtMPnusT9pM9jyAsg8JvTvqEVtTQomoEs2dPAUW6y07nHacH i9Jaejm2Es2Q/DM9Eb1zXw/Z5lwyPFJTI08m8c2WWoAtKsL3TVAxtYSf9UuJirLcauAeCg lfYpl7J4XK2Hv+LM732PnK+f6SEcD+8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-556-Qjur93AHP26bSv2h0FSKcg-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: Qjur93AHP26bSv2h0FSKcg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CC7EC85CCE9; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A2D7E40398A2; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6718A403CC06F; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.683821550@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:52 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 02/11] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 35FDB40006 X-Rspam-User: X-Stat-Signature: diidrq67e49wf6iuop5sotu6bp6bypu7 X-HE-Tag: 1675956800-856941 X-HE-Meta: U2FsdGVkX1+VjmKjmVVLZv9AVOSF0UX1ypEWAuPnLRmlqqlToCX0g8nQfAw9BxtT7gjld4w38NLlXWvqay3x4aYeftTtJUyn4mpKCHHmzyH69H20UoQpfwa/nPI/l/WHFaFlL2KJG+25eFCbOEWDW3vc9rLvUcgC9KZF+gH1diFK8f5J5Y/b2tOw2eD1xpT8TRXPMJEjPo+6tZHDFqFGAJaPtQijo1rgz/e6s0tcEDDMgBabqG6BlbcRO1RXkfwzsKd+he3ffE7E8+5bpfMjFgXg4pg1TuZJOjg2cIqMolL7RThiKlZTWRDPtXCBmcXYJ9gX51/ClYwfSQGAukEG7tn3AxPat17ZDbdF1pa+MAvsY8cF59dVtueMpSPBd4KM3ItAsK52592JitufRuPrE3dzL8qAKNM3zzbPWWtOCW4mEWh8KFp+POIfoZz49B1EeW+QBCQYjxDkssZSiuGbatI2pPpQZ1MgbawLLlCsIvwRHH10ZgZ6Di/8/DbtTsuwCjEYLyNaIhL0gKL7qdvxjg4EID6+GbMXIv8M6otNbAF7gOfOUf21RXKS7BiVHeaVBRALlAKyBdx4Jc2/8sSFU3w1wCg1I7IyrRNxr+q+l8dIeVAzAyE/xkFfkXWaDoT1gFwKDT9i+kpBlQuFhRh0Z6bgiJZyz//nXe6TOBxX9SOIdypYyso8Na1ypHhTfp93xrBTV0FIF4uFfPzk+AV0xrtaChjXFd8g88X+aP2mTCwdR++fDDQzY87LUsvL0pe15+W9kXyvKlw5tIsmtciXLl6KYaMdqovveBSMlHPDGxCfIkNhVE94jQUifgqNAhKHZIC30S6mmXhEy4rvW+Ty6/BI9lXT0wnmJpuTU6fKe7svCyA/8CuvMz6nbVUUrb8Z6RCuNT5SCbktRIefKo+xjQAfxm/EExgaBEVunMmdaFPpx8el3WiVzvxQAp63BaNVMLOXOyxs8qPik3yVMbf 7addFEUu sVDE77jLpID6ZRjq7FKem/UGk0VU7f/vc2jCif8ambA+1pc+KjnCHWwI7kM3FSalaP0N/UNVvyVq37P4QXv7JbgSN/pCjxMAxZ3P6Ho53adPgJCAV9duH77GZDx3j8cxjfqyPqp3fi6xCG8Azrj+sBBX4BF7HIVFWlf7zjFXFDsN+W4ptvR3Roi4cLVeHJ8XauWretkuqmxeu6NzuMbjzaB+Rj6baLCbv1XNdQ6nWXxhhfdVjEfSZlyniUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Thu Feb 9 15:01:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C842AC636D7 for ; Thu, 9 Feb 2023 15:33:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FDB06B0078; Thu, 9 Feb 2023 10:33:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F53C6B007B; Thu, 9 Feb 2023 10:33:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04D186B007B; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B67566B0083 for ; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 43A5D8095C for ; Thu, 9 Feb 2023 15:33:24 +0000 (UTC) X-FDA: 80448147528.21.9F054FA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 4F04840013 for ; Thu, 9 Feb 2023 15:33:22 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O9IyU574; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=HSrDTGe+tHr/EaN7L8yWyxkihJnDwSibFP0d4lVhdaEk/EQRUM6KuFpbFY0tN3Rgkgiysh Roc0moWvc1O82qVrFhkWQv1eTK2jTP896js9Z4iZUjY7UId2uu7POHpEcIGTMpSOudxCdq 7HHFWz3d0VmSBBIo50g2RhOlOVNRpv8= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=O9IyU574; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=QxWjsRDARezzXU+35xcGNqhG5YWKtFcm9+q/QKT65X2LosKwVh9feBJuPhPUzrpIihbU3g 0ul+BJFKcn5PgutIbuURmkchb/a+6Vr8V1DciWKPTR3X11lYl/qJfkHDm2lkjXS2/epN0O s+nQy6BZQaWMSWYdKTda8npn02Er1n4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=O9IyU574Fv4knMpc7dsljjE9ZyEP581ejvihoL3kF99WyEJqOQ3qBcbWlCccyfoC0KNjmF mEp7GrkzzMr5Oi+V9p72IrW8Lac0c9PkkLL1VcUT600EBYzmymt3t2piJa576KBMx7t4AL fXGXJSzfyWIPBNA7GiG2/64uO/FNE5I= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-556-gM0BUYDHOl-4dDHK3bUZLg-1; Thu, 09 Feb 2023 10:33:17 -0500 X-MC-Unique: gM0BUYDHOl-4dDHK3bUZLg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D00932807D6A; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9FE17440BC; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6B645403CC071; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.711672794@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:53 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 03/11] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4F04840013 X-Stat-Signature: gomxg1z7zffjsp43jd8ieb6y1kjyte9x X-HE-Tag: 1675956802-762641 X-HE-Meta: U2FsdGVkX1+Q5c7mDdx8pzsk3Q97kcjLg+fE9RcxsBPtpZj2sFoykXMFIaXfueJxonZWjxrVdPdXjYcizIYtHgzJvF6gxZGlfEfszMwtfiClOFZz0EFoz2RimXvifTVApB9AmlXIk3IPlSbySk/FREAhNdxFY961/s6+RkhAWyzIxY+2npm5FgsX6taQts6AQ4Typ0FmXUrfWCbczZuHLjO3b/ySHfFqqp8qHGXMM1ZHBNVLUSYPZQRonaa/vgzAu4m8H6mvD2E8oyDL0Vq7asoPVrT5ggrWbJAtCt3kqPixtPc4uVHG0913L3Hf3xkZ5Td8MPYhGDXC5UNAj5Nf/z0Nvr0E3o2jXMMIs2lFsaQMV3/gcm1hat+jH2ccT+7PEc7BppVzejVdhTmn+lRpSk4s9yyFBmrJ8uwSaOHZ2iyw0GXwq2T/aIFLrsrJwO5DvJwn2vzku3z7NHhgMpQyYEZyqvyYuGh8YOqR5Yc/+sIBPEvENtI9lPYWOkNbMw8WeOKu0f9XfipUq9GNvBu21l+c/B9OsGkbyil2B//uhZoZNrnlnbPWeaWknjxl1HmqaP5NRQfCm7xI6xskteuDbOJcZxOwqHKpcXzYXcTqS0yY1By5xhevPPU5l1za681pxRYp8jBWpuP53sQW4BH98YvQSGCsD3xEndgekVj0MEDeV86xTTEUfklXsqhaOCAffb9GIsh8VdbNQsqrB8W3bzxkHaBXACdBwi+BGoCWXXReWIKFadufCTaGI+yRByX+7qiDU6Q7LmrSe4NY4Ft7idzBxZyjO+sClmv6Y83bDSKaujo3fjFPGkexyksR70pHW4o1e0f3XiA/xWdRDaVOptxux9XEhed6dnMJe+s1fVmjWrER/QiVocIyLhU688Wy5Sb7GtqpID4u384UGAJFo3wTYUiZagNgBBKTkYlAgc/TUQuBO1HVN8bCpyKd8nNfKK7AXK0mSusWTq5nJ3K 6ZNk9meM 4V7lHIkPhtbI3wgEkgi1RTILyZk0Z/I8HX1T5Jc2kFtUIun/SBtDOAD4VYCLyMOpdaKcXptHdwGXY7/p74J2gyE+fM5wkAhrCjbwzfU15LPzp9W19Tv4cP55GHjpIV5IncuVxQv6Cv7LVxTiqwgvXSCKoIK7XqZAxqiCEsjWdrqHtSud62qtPd6LjC/qEpcT41ZSNn48yOAmp+34z/aEnqonbFK7p8tjpxE/M46hj8q0MkiCZJuLlw2HTGA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Thu Feb 9 15:01:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C73E7C636D7 for ; Thu, 9 Feb 2023 15:33:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C0906B0081; Thu, 9 Feb 2023 10:33:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26FB56B0085; Thu, 9 Feb 2023 10:33:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05DEA6B0083; Thu, 9 Feb 2023 10:33:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9B1916B0085 for ; Thu, 9 Feb 2023 10:33:25 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4226EA0F5F for ; Thu, 9 Feb 2023 15:33:25 +0000 (UTC) X-FDA: 80448147570.07.EF4E6FD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 687EC4000F for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ci8qSlE6; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956803; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=spFyYZ23wqS6MC1RxdeQC9sXRSuPtqG2yfAIa15tCgeQ8zh+78rpKg8cOkGnrQXy9jCqDh cotwb+iFCagO0iwKi/cELdF7RYvwLDxsagmo+GMk2rYE8HAl29QVRPWdZ6R6zUDy3gxx7g C1WjNMQurR0uueAd9uAWFRXmC5BRBIk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ci8qSlE6; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956803; a=rsa-sha256; cv=none; b=iA6/jq4BYlgyWOY6CWcNr523W25kpawjbW8CzcBJmss3oI5MU5FKOuJu47QpeRlVbw/4O5 zeLKmywZ8XhZRKxKKFztvhBq17Y1xbD7nafs1ApAc3D3MElInvlKHmLZxBQ3eJ7VgoC4tL 4/t/8XNwRH+IdRqhlcCj7nGHAuS8zG8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956802; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=ci8qSlE6n6XYlekD8zeuGNIyqMOP6qWUY75bEb9oYsrAR5PQG6eQrNdyGoulBabFLvcPhm tb58E7PALBvOpzd9+d2MlJhdRv/pef+IQKgYcOsyv8UfRBsV/L/ckpmNIO9aRPHQSy9IKs 9Fks4Ldrf6euzWZArrTytXF2xe5HpiQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-467-f84dd5UROTGCJx2buXnEzg-1; Thu, 09 Feb 2023 10:33:20 -0500 X-MC-Unique: f84dd5UROTGCJx2buXnEzg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CFC68104F0D0; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A28292026D76; Thu, 9 Feb 2023 15:33:14 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6F68A403CC073; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.737772842@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:54 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 04/11] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 687EC4000F X-Rspam-User: X-Stat-Signature: 7s3qffh9yh1sj9hindngnynz7yejnehf X-HE-Tag: 1675956803-181889 X-HE-Meta: U2FsdGVkX1+tDhJqCs8PvPJBGx9hXw9BsyWZm1UiPTYHpZcKkhQ4LWtRw0eXh5gTCWlWH0V1ECYVMhS09Jk7NqFmNwzXVMxnduYFp5hqHOZzezAsaN37ebHodtEI3we48QBImKNR/MBtAJ4DKCFGb6GXpVmsKcUXrofrel851ELzM2Jxrg3RlkoLYGLy6IBNraPMC2KVfQ1ofoNNdYkPvkUO6yF0NjgKplBPGc4c8gK1EJ7yxnW/UcbtvCY3n+mHr+b1SG2q29dWw8dJBQQYKb0yh+fYlZ/Q8DhaSpktLcOt4/lEFEEwoJQQTNaA5yEeU588naWN6AHUsw7zF7LbIz3f2nDwyXyhM7ZB7P+qJYZzyi5jdzEEH5gJhiR0yCobM49BpT7xFZvvfon1bYlY2upNiBmKlGze+hA+WLlXtRnrFp31/Asb/Z3QB7EM9LGKhHMly7abxBWJrhY4ROVQ+ruJybTR4CY4CsTmqTFFQsQZBLswCucTpacpNjVQlmbrOIltWhK/y+Vkz6PtUP1HE/bkcXu7GsTbpv3VoiE7R5Qr5TYrVqPRxx6xMMwPXPWzM5oiQ2jXjUPVgfsYwN/osumDI2rM/rbuY5vrmU7izd6/tpnvbDYK/7G5XLBSwk9XbTlKsERv+Rf/FrhAgaXxcXONiVTnZ/fvhueXEGneX7G3L2ip8R2Um0EKySbbEvq+1ohTnJHGEiNwU+ThUGfuO+sDTjSpFXpB84nPcSDAVkxWEzzcP5VIpg8++Cvb2dD6FLa6X8EThMSZ/L+AWoI3pR1yBALYG6btpGC46EGN0Nd8OjQGOxj1mHG6VuggENT1ft1crGTNk/jJYvB7/7NB9P/Or8MbyumQCCdd1OuEe49GBkq61vhIoSNPYxv202KEGeZPtupOL23INTkcifFTWm181E39LBsw9/ixEwchjr9ka9ELHmKkP/cnYxHPaI6cvFknrDrm34ymCL/YK5E 8vz6iIyW /cw7HNxM92ioluOrrhQ8GS0G/Wi4Q0fdV3wnbtmGYe+UYyBfV3kibjJ8O04zT/ArvznCaUmh+LKlB1dxk0kj5moKRqkLnb+GAKxL9sNV30j63SrFOeomQTEfgXqedX+3AiYXZcpeyeVV58bFGDrGJpDfd807OIkjLHA/xILSmmK0U0xJ4poRj6m8ipC9hjS9HsSHfPck/6eAtdB2ba25NMx8ThQE5/ip3NZ3ooYijotfcUV3PncJQs94fvg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Thu Feb 9 15:01:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFF67C61DA4 for ; Thu, 9 Feb 2023 15:33:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30F746B0082; Thu, 9 Feb 2023 10:33:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AE156B0078; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A10A16B0081; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7EDA06B0078 for ; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4D2FA40814 for ; Thu, 9 Feb 2023 15:33:24 +0000 (UTC) X-FDA: 80448147528.18.07D87F6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 775291A0017 for ; Thu, 9 Feb 2023 15:33:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G6qcjjaj; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=VGaxrHSHIWDZ9wNuzcFJJG7J1CJshnD7QKbHX2GovRBKMAjNwv8fgxlIjlGaN0jvJMaq/b JVZX+cxl+0rWIz4RjKHYvwmh6a3qKDwg6VFqC+JdU6zg9yXditwZe90gAsoDv6axZz7pae 7gFPBJ0zxNzWBMHk1liT2cjPzms0tdM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G6qcjjaj; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=BjuVO7OpQhJwNuI7B77OnNOvZA/4zevcFyn95HN+fW455PbnupHSwVFWAlot6hPuONM2iO Soz4nJxX9fuslWM0Oj1zS+VFW0lTXiWTc2J+AWR0Wx/lgSJ4jwEEEYwBt6sgWOujOAlg+r sfugT48F5SzKu/H13kzG3YpNFNW6KAQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=G6qcjjajdVmVDzxZqne2LsGRPlIf+l8TNlOG7Y30iM3hvqULTMLiqQ5QrxRvdpE1OBfzzJ kUOyDwy7sonVNsw0x7Msv8C8uagxvow5rj9//6W8H37q7HoT7l4eEOXBmXdV24vIFIEgYf nVP1DS2jApkOXNyOcB84t2B3Bmwiy0g= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-311-unXOuOmGPT2X0gACJlUVeQ-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: unXOuOmGPT2X0gACJlUVeQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 98BE93C10EDA; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4973F40398A0; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 72D73403CC077; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.765612949@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:55 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 05/11] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 775291A0017 X-Stat-Signature: bhweurukghh7ebfdn5coossdzdjcbanm X-HE-Tag: 1675956802-67319 X-HE-Meta: U2FsdGVkX18oUtt0g0ktrvDhv04zy6kMBgnlBzbkAu2Tw6O32YbDqzUN+kl+U4ymNEa2+qmLDaGA9dKOZK43MpQFK83vgXWvCg5KigxCLCMabaEmGEdCaq5e0zAfNQmuXPeIGToP2Hn6uvcZ7AtAjSzpOLefatW7Pu8G/2dxCNrdZ6yDHH2Kiv0uV+MCED70BlSMhKK46A6iqwTHhA/L8HYVjNIffka6iwRtx53Dp94ilBm0ulKMj0ksOFo1eVeRWywzP9pWObvXHOedFYtl+T7zuhlCXZ12ABlip/2VWvMmh+dD3hGi41H6Ls9acMyWO5UY5cXXVSslaRjO2539G0V5iFF+v1fiCqsue6kPp9Yz3mxLpgNuFqXuntG58nTjIUbrRFOenU7TLMV/JaxXTsL0uM+4k4xYpQv1VOu/4UPi7iMS+kVf45t8ZtRsV1dKFCdRbmfCS210l5zm63TEJkeISz80G9aHUChnvJNfxXF8KcJCrnBXCpchvi0N8FSi6jq+iqCBDFziAGl6FgdCoo0GFodQ1yMBEcYjyf7pfcSavgbwOPPy5Fik0BvRloUq/8brUJ9ecYD7RpbdAex+FXgpAh7iVrg8IVWpta44TXPhpFm7UaUT1uvaRkC8H3ONAAdh4OkEStDCMmSyRLtrQ0ubNeGnlMFjfBneOkh8y5HyUsKKQifX/oSkvRnXELA9uv9zfzsDew7D4oRfhOy69mRLH8YvA5Xi9k1bo0WKxMHXxBlzidkzVE9hj+ZVz1MxIfYYQHa0RzgdUB+VkABDHfIeihKyr1MttTeV7r+yBBGHiF5jpFWxDbfEw9Euv8oi7RCo9TB4mYNehWUbYH5En7RP0j1+UxTWDBVjyfiUD1sscK2VS1Tjy7Udr8aoroeIWrwqRfISUziY3zkP++YC/hYIAFOGe7E+jRRro7faC70NXQmIL15mcFZLLldgGX4dhqJfFqsSgveHTdYP7j7 8k8ZHOTI rVt8Jh3XGLz+7Gefr9JEULVIj8W7da8ZIZHmrK64G3VuIdStOg0r8PyN0Bee13e8DBOx1yKplTUAsYbOKVdN5eRKrL5UCiSiS9Wz/Ho4tMBxB0MQc5c/TrxB7hXQQpl2dc5rkyS0fUhs7UKl0rNGwzYMIWL7s3mbxCgPlwdKMILpCulPoEj/fgqvZJ09Cu3ZLVsFFpZGzhKQNdvEETxx11+h6tBppdby+Ym/0OfrcQvxJFtjtyke2VHEpQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Thu Feb 9 15:01:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80A76C636D6 for ; Thu, 9 Feb 2023 15:33:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C55246B007E; Thu, 9 Feb 2023 10:33:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F1746B0074; Thu, 9 Feb 2023 10:33:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30E126B0074; Thu, 9 Feb 2023 10:33:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B61F96B0082 for ; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 48B14AAF01 for ; Thu, 9 Feb 2023 15:33:24 +0000 (UTC) X-FDA: 80448147528.17.280960C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 703C41C0019 for ; Thu, 9 Feb 2023 15:33:22 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VUbEwTbq; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=xttH7836oXgQ2gV7qXbyD6NGWlFVj+CDeiD3JQB+5E4=; b=FX32Qm1o4HqrSIOVFDO+6/eQr7W9ASn50a39fZcURiEaHVga9WHLTKf4Cq6PJA6bUp5ADy P0zk7Xp9GMsdECmpLrxEccQ9fP4fDMIsRI8ZUINsaJOpAhqUwk8dXgJ4n8BfGhZz2lPdPC +Vf0zTzcmABM6FrOuGH8lNlTEg1E8XQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VUbEwTbq; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=2aFKonNTRnBw3xVp3a+zALGkN8DWJrikUrYfcQwKJMSyuYkTI10ktAtSIjZ417j3efmrRJ hkforfSlUox2yX+jEn3cVNGSxHZoIJK2FG2epVzT7/22roNl5CgmpwFDxxMfFANCD4OcSm EgSlI0opi4SU8lyYMvhr2QWDWJ2rgYs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=xttH7836oXgQ2gV7qXbyD6NGWlFVj+CDeiD3JQB+5E4=; b=VUbEwTbq2UObIG4CHg4wlLVgxr8/amvl3aOU8tb0z5RGuUbueIETdcFCoKeKxTvXV7C57Y /ykt3gal6iKncmBEm57gMCx1Zh+OkRJotOzKXd+jy/1PBJD/6CJ+4uB1bxX0rM453IZXz1 G3aOnAhZ0jCOjzehyCARFn8u0kvXzog= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-590-HxuIk-9sP12Q212hs8jtxA-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: HxuIk-9sP12Q212hs8jtxA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 937E12807D6E; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6690740398A2; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 77799403CC079; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.791818567@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:56 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 06/11] this_cpu_cmpxchg: asm-generic: switch this_cpu_cmpxchg to locked, add _local function References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 703C41C0019 X-Stat-Signature: ygwdk6n3jd4r35zwre3gkdpwcmh4ch8c X-Rspam-User: X-HE-Tag: 1675956802-693857 X-HE-Meta: U2FsdGVkX192l1JckkI4XxZI59MtcdRQt7oubQLGV144a/1X4wM5kkexoLK72F4xSiDZUPUVWom1Y+MV5Q9RFj4jp1HVPvhFOKLaua7OtG9HDEbzS0r3jAHw9bl6DxiHjV38ItOsxBbeIdLyEAs+X5slS6xbpcEFIk8aICw1vpGVjXTkQjYCu60KCf43tXoBqIoaHWGLVrURj2ThX1SQ5QJ3EIIXrXefzuS90nWmZ4c3mY7G/YZugI0Dj2hNx+QAQ2WbPvowBQH5WQdCCQgWt9+5rkywDVhsh/ziSWB2vshbF1VwdlKCfo8g9T+xN/n/vGeM2h4vAyWUC5u3ugiYoB8IFoa9rPYrHDVWkcccruzBd8xFJkvZde53BtWtpyAstt9P9ScZgwg8+F3AIE46cl6QRQVj2LWUwPDyejOoXwBsQHTmR1IjSOBVfYsWhgCstabhTxpb6lK+H43tuEb6wkPcQZXxNMY99/9A67uJQFjhss2cExd58VcrVrlQtPpm7RgROZsGt+fbuFxe2p2Ki6JpNuG/YvIR/Ia/Fz6+x1OnQu4YTgtJgs0zi3RfxSnGAk5xz5qW0qCp0c5az3OXeM1huVHGINpSJaPBUSAJwm9pxVDYCFtJxyHJnKHo6LFh/3lq1n2kA7IETpeLmHzBPTLh2F2ZSTV6FLQTe0ihC0/rsYOavTb/Gx9bY2fTlG31U9y39xGICZxCHvoBkLqfwFb85q6zAevzuOyG0lpC/sHsAx6LVOhSkJIiCmjOo0z/8ZFa7hmh2FzMbiIbDfGxX6EwfADYNtOvgME8UO35FKmiMs+W/Wt9+Xn65ILDEqBeEsE1ijfvYJmR9Z/OLApu46WZjLfgU6BcmYWgcPLuZLRXwxw2RaQWCLLnzfsWE26OI/07cJsQ60xfdXFk/a86C/DzAHwYEUJxLGNUTZ14WrTyWsvYDSB4/l2IogEy7yxZf03LzVuIsYtQF683agJ 0GfZU4ZV 0nmbhNM9He9C5MdvsNuLBYZH+DefA0tBzDHj2WP4lsh3g3dw87BTr6UlTtdtrVWOEBu/CWE48qrGBVDdXyrGh5rLMabT4fIoKD6+O4W/6JLZrUkNLtP1DzvlTdsK7CBi9saRgVO2Eggye4Ms5kqn4Ui/pGY2UINmQwUDT8vGr6IW2Vo3V8vKOOrmxYukJo12y9DucCw+UieoWOEqxjOuiqbbqmvFN2V1hw98CxHlKeA4W+9AOpD67LsUdmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Thu Feb 9 15:01:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92E29C61DA4 for ; Thu, 9 Feb 2023 15:33:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99DA46B0083; Thu, 9 Feb 2023 10:33:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 94F1E6B0085; Thu, 9 Feb 2023 10:33:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EDDF6B0087; Thu, 9 Feb 2023 10:33:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 691076B0083 for ; Thu, 9 Feb 2023 10:33:29 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 44F2E8095C for ; Thu, 9 Feb 2023 15:33:29 +0000 (UTC) X-FDA: 80448147738.15.81418B8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 5A6F61A001D for ; Thu, 9 Feb 2023 15:33:27 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RrRcKlxs; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956807; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ofXkymEHUHbiccFSxfE3RKhTo0FLU9PUEYKbx6pPdBY=; b=wC3sT9jySzIysaMumFNJYHQy8yd7CI109vIAGxreojV76OwdC0qp7J2L4OUfbLqGdcn+ti UywJ/dHPDuFfOUOEqJllNr10vXD5DAg2/6Po8not6LyNPzgVIc89HNPtsBMKn/6Mzk+I1Z mfqz1z/+NiVtNVjQeNvmYOOShy1Yy4Q= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RrRcKlxs; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956807; a=rsa-sha256; cv=none; b=LdrpuSSDS4XuiukfZO8VbGC5kYrnVZR3ZVZrhmFACFj558WzenupvZuZeXmcdxyWB3s5+q +dhiiXEjKyV4bpDMrP2EPHzlo+LVjHQtQ+XNPkRMxzazKINyTIGwbNsOeKVPTutsUlbria p6rvEKrf5SKEq5E+ZngN37d3DLNojX4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ofXkymEHUHbiccFSxfE3RKhTo0FLU9PUEYKbx6pPdBY=; b=RrRcKlxsnPdV/OEnVVKV48PKnGf0yZs34qcxxBh69qVPX8LijfyynheyWQxMttl+Kh/xFk neeo2q3H5GXquerHpQK8qiubUoiVw/4YkzmZIvAfBYmWU+4CxcfrXMUr63PdPiurKoNggs m8L7vWXi2pr0tbPg0o6lqGHtUJe6Wjw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-311-w-j-N1OhMzS0AJa82rbJ1A-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: w-j-N1OhMzS0AJa82rbJ1A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 799E13C10ED3; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4C3921121314; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 7BB57403CC07A; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.818774692@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:57 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 07/11] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5A6F61A001D X-Stat-Signature: zpbidxhk4gq53qzx8561uce1ygjnftts X-HE-Tag: 1675956807-111503 X-HE-Meta: U2FsdGVkX18uSbpPp8ZGfcE6dd27T593cGDAMyJps/tAsGoPdUxRQky4wZazI+sPim/q2g+kGhIXiFZ6kUAQ1u/ttTDN/L6HyefaWMbfh9V1Ile4Jy6ojnoCDN704cEJFFSZpT0YWE0lVt0MQ19q2qQcC11tw7KSDz+Hixtu7IaEMrab98Jfa0bYAwSgy4Q2gXo7ExJrz8q1Z+gMGltMSbSD1sAypUBlNjZS2B7okKcx1md5JJOdjThwYtGJ66y7EMgspI3skv0djmzbcP+CnkmzdcUb/7GXo9+ScaMHiGBvZ9DJ/A877B5tkOsAWZKn03WDT+QTZ59GEekAjvxjrIGJ9ySP3gTLcow9FtNIzD5gRyzDIRcbHw9BwY+mb+G7xWk2wzGke1UnWj5ZvNVt/YhwaLioXcRrXc9CCo66xZGPBGdbANwxXzuUsGcVobijBmNATA4VUwQWD4TWodaadGCmdwPB0ifBwurVyLJQvS42oxO9uwQD/CbU3BziwOfB0XXFFO6DWIOpwrqTgVBrSae6qfA4Iv3Pe6yjrJVdlKtSbgi40wctON6ZsF0Oqu/es/nPZ+DmoNR+gZIav+RFJ/yXn95gOx3BK1tS8zL854b4v0FM+IK6tC8jNeAP7CPnffb+C/uwJZ+bw1JQGRyk1Fd92t4iMRM8IrRnQmfkF1Y/jKgNh4Vbu6xevTORLxZdgGK/eoy1Cb1IyFjQ5NlGrCkwelURwx8k6WBiIbezWRScNnqaj/0CL5ljKn63u1LBo4Lio8lugXkotgdO1+kFj0X7OwmG8pZC7dsiwlmOkbmMMTTIyb7PcLqLYmc/1YlT+yEnsg5qxYUgKCcokf1yYRzREFxHH6PPfWkP5URZ+4kHL4lfCk4bw4GMPtmxGtKr0vSePnh9ov++sx++F+aW7xGn0Nwa7m5fWiZmQi/8Dbx8FA48owTfdBx/rVhU9VdKyYAu5vDggjqWuV25afW XgHFXyIh 8fJza3c0uUMWwjKTXEkkHDYmiVk0hT3/Ms+k4hoX4H4ldGToBiY7iPmgaiG3g1slCkHkGiWQl3aBzY+ND4HFCQxyIDpyy4UmqDdIZS27l2HGpKGywqw/kTBLyFSkJ6FK5j347vsObtBd1gNSkK4fpsWoHGKp5ILCoJEa/P7zSsoDJQ8urstxifXYpVCDno7usX/NWr3kiLBJAYng= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Signed-off-by: Marcelo Tosatti Acked-by: Peter Xu Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -79,7 +79,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Thu Feb 9 15:01:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D86F1C61DA4 for ; Thu, 9 Feb 2023 15:33:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA7416B0080; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 758366B007B; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 507676B0078; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2E3056B007E for ; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F163C80F22 for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) X-FDA: 80448147486.22.BC1234E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.145.221.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 037BF160023 for ; Thu, 9 Feb 2023 15:33:21 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C2Ktff2w; dmarc=pass (policy=none) header.from=redhat.com; spf=fail (imf08.hostedemail.com: domain of mtosatti@redhat.com does not designate 216.145.221.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Ij2M2yRJo325miWQNsr652YwVVg5UqnJuterQU1PBfY=; b=BpxMMFXSh9SMMljHzsPCY+IKc3OX1fzsYYzMolBi0d2dhfpHJgNKEPl4TOM7javYhbefwW Zo/ecXscnnVYIe0C/B6CBoGyihp1KlCYTbuut+TxqoQUoFs4KG+NFkE79elKMXivu1++xE ZLe3hh19gKDef8JHlS8LJZwn8Vq/Foo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C2Ktff2w; dmarc=pass (policy=none) header.from=redhat.com; spf=fail (imf08.hostedemail.com: domain of mtosatti@redhat.com does not designate 216.145.221.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=kOjQzF0K9MYFkJ/mDTuOR1nuXDPzJEI97pqz499ud+akHNItlb2wycEINsK8EP+RljNBpb fIHM+PHToZwFwLqpJN4oc3rQaxB1yXNsrL2zw1Oe8NGYmRZeSBVPCe9vA8VPhg5z9mvxzg //Zgpxw6n5gpi1umfPmHGGpzWWIau1I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Ij2M2yRJo325miWQNsr652YwVVg5UqnJuterQU1PBfY=; b=C2Ktff2wQPC+Fxh8y7NNg3PqgKD8SLFgDzEs+7Eu2J60PEjMFWd3n8LHWGKy4O+4M/6trF RhD8dVtrVoJBosIGPjacpvO6tZs6zNHDD+7Y9IpW3Fgohrm3IFkjSWWclWaeUar6RMXEn9 HLUz4FdaJHli60YekoeC2vd9Yk9VLZw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-2-l-XLnhCQM8S0UIYM4PgIYQ-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: l-XLnhCQM8S0UIYM4PgIYQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 95BA5104F0A8; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4556518EC5; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 7ECF5403CC07B; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.846239718@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:58 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 08/11] mm/vmstat: switch counter modification to cmpxchg References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Queue-Id: 037BF160023 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qm8t4hbhr9xg74hshe4e8zxq5t8onast X-HE-Tag: 1675956801-128170 X-HE-Meta: U2FsdGVkX1/ux8hAM70/TwdsLgiPx3hr6ChdvZtZHCtivXG2gwTB10I0rXOithK6IablMolbQxPxE+DxObq/V5IQU3CDxzysnW1RrJGMm2S4TF2u4UuZKfm0csdnEIt+0abk3n6ggiruoOMDMy6/h7iqez7Gn9p5D70Sa1xDkanSAAldUU3Rpzhbtfgtca4BFP+sSZJWE4EVOpzEhjaRb9q7BlwF54yLC5dbfoAmWDZtWpgzL/uHNLIlAcdLaZ1ApeiFlPX9QdgnTSoUZqRDN4U+/EAYJiGwwCssjxRfugxy0Yd+uhXtWekKWScnWD0fzxz3iyLBwy39K4UZ7FoziIe0IXzA+ne65JcT7IfzVSHrTPiqTmgXFK60tptboLQm0O2Ydf7I2SR0dv2kUPZvAHTm/jAnZpPXtOhGQWBsoA1WJpO9Kl4kIk+2ijpuLObJjj7VBngkecElfJgB1X4YJs5l9APaOOdmdwJ6AXusKuuIrgcTjGvgpMGKR+fimWMj2xNavFxeSMQdVpJTd5bPITMNfPGdTL0wdgJeE8cy+0EJ+YkLPb+6ZXQdaSYAd0LWj/Al1ftoSCt/CYcpRi+BK2qGVJjTD0sJk10NDnrUKi9pJvRLZ5kVbnf4EwvgTxvVN2hO+SLYk+i+tXPTxLpPZMqWtKIAIIOL0QhqLe/uLdu6eqZXCCCwqHugiKM6CHjC223FjAZh6sUzpdP4H3hNnhYQGWBbuLwtB7m/LUz/C8Bhdr3ZpKWvUazct1WABKHqnhxFHXBdDzf8WW1zzCSu93mYOYjm5l2EptMujuY3FesLmJnpBmg54zz4OfRdb3aLwfgHC65/au3pkRp3vE7KzZzmj4lRgTJ8q6em8qGY9YOingzc1Dlut2sBEPlRnjsETL5kmqBdCOErwzKIRLC7LX2BlBjQkjHpXkSZTwDwjAVB4bkRVnmh76WeMOEXCA3zAb6JKoz61iTBxaDSAZl mRqNytlB GBNSZ7Hzu3IdLbIkKDIH15fr8EnLmr72sMdrjsWgoesNEWYWhUMACe97AJBaZnT91NubN2BWb7WKB39Px/2QQmzhAe04OR9gLrC+HfXaNxt8Z/u6D3yE7XWn876Pghgo/WviuiiB12ItA7GNCdqaprGmhtAzBnn5gPz5jHcTkhhXESsLXH6qg3Zebsi6hLoViaH84+75LDDT67uj2u/h9SxWTfnNeXtelRrY6iYDPB0wqkgLbfggvXk2Wmh407WTBjDi73UkFIxsP42dJYAOJg0fjpTU996gecryuXAxNnFqcRfDZWmh4qAcHQmp8eJpsbFrs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, switch all functions that modify the counters to use cmpxchg. To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 159 165 3.7% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ From patchwork Thu Feb 9 15:01:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79F50C61DA4 for ; Thu, 9 Feb 2023 15:33:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAA126B0071; Thu, 9 Feb 2023 10:33:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C59C96B0072; Thu, 9 Feb 2023 10:33:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B48BE6B0074; Thu, 9 Feb 2023 10:33:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A153D6B0071 for ; Thu, 9 Feb 2023 10:33:23 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6F6A740F19 for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) X-FDA: 80448147486.29.37FA271 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 9693940009 for ; Thu, 9 Feb 2023 15:33:21 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="f/RbsuIs"; spf=pass (imf11.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956801; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=LGWVlVnJpdpZsY64OC4/IsULi3iGJlzsOgwi06Ffb5I=; b=R9Ek4T/5nqH2pb+RlnBi1x7OQ7xfkG1tJv22Z4lNGLK2b4O8Z6f+AYQ0owVwi0LmuoBgXQ FOw7tDcAfLeLOi/YUPyDTkmY0Vn08C9o4oiQW2VmQeY8LA4mf59Zc6Q6HOgHKStfcWh+35 PCoTqNCKVeTGRURLDQtG+dV5aMp45tg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="f/RbsuIs"; spf=pass (imf11.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956801; a=rsa-sha256; cv=none; b=lOXelw2iLwf3jmDUXVCk3rdbR/2voyW/ok9qo3tqli3qN+rZvM2Im6Fa6lKQsLL3vU+5P+ VpuoOMzGqbuAF3ZDeHVRIILKKTVb4AbKXnU2gD1ZMVCd0G4lFG17nSK12/mLwfqZY7e1/K mSmJ7q+LqXJ5hgycRd4co0lMRntTOkQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=LGWVlVnJpdpZsY64OC4/IsULi3iGJlzsOgwi06Ffb5I=; b=f/RbsuIs488HKP153mWQOox/Q8210pglPiRKZcqEhL5IjoLU9Bzb9Srz9pmgJEty4mC6Gx jkqvux6LKSSuw/acmtosR/vmbRKmTRP5tmTJ3gY3nL36/eF1PKXOakXyO0K3XQbtt6u2fq EiExcC2ELDyi3mwZomXhTfkOFWA2Onw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-205-LPxsHX8gOi6p7L8Li5eumw-1; Thu, 09 Feb 2023 10:33:18 -0500 X-MC-Unique: LPxsHX8gOi6p7L8Li5eumw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 902DD1C29D4F; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 625EB40C83B6; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 86BD4400E5701; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.873999366@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:01:59 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 09/11] mm/vmstat: use cmpxchg loop in cpu_vm_stats_fold References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9693940009 X-Rspam-User: X-Stat-Signature: q6n9o88ggo9bfez5fiypqrbfq49hwoif X-HE-Tag: 1675956801-674984 X-HE-Meta: U2FsdGVkX19yUjb76McjvSqPUbbgo02hMs+atEinCrTLWfQmR52+hE1g0xCvrEvLW6cIlMaoOz+rFD1/ZUne46cTUX0FyX6G07znvOxDr6EULLBMn3/YGnge4YDodYu+qrx7h2HSFTl+w5Zqt8EdqinMDJarYkcOLQc6jMnhV5ShUpWPDnGZmYamKetkXOoT1cbUHpAPBmD64eo1YrF0/dW49GyeIo5Nt+cK0Im/zZs63NXHOpnfBFHQbN8AmUlrc2/Z3TQ7rhVKDtLv/F0hFL8/Wi6bVHzcSOWlKa9WIeb1jiDYNcQ2wO8y4GMH0NpcrTkep1qkg13YgJ9VEutAoUu84p0jWqaR4eBlvrOtLBGxw96yyQBSBjarHljsmqOEBbF3mRkzGKDDvVGhgZKSVgnagVjqxhvzMZfD3lg00mxFI9CzO71SheXdNO7XU9EwmCcpInnHMhFOVBRTQXPas/MD2n5pEto/6evU6LWbEI1cLwRfZUvd8LTEiawAw64Y+6s/Jt544c6ZnJ6MtCFTMTpOz7BQkLJQFlnRUabppbFkBdBNdqiEQbLXNi91Cc7DOkugTRr5LiUj+XVhIXW3M9tPm8FvPm6VUhhNSwuX5cvBO7RBLgnqi8ppDlEL+2hChXBPtJVXChOhDHT1Ebl+rsEjGHZCX3DKy+8c7s7YyCpoRwT3y4uAeJ26ykbzvO9dmSHphKY+Fdqip5p044zGXzMYFCloHhVKOvJCSIn0rTK5xYtLB1ASKBHJs4Dst1bp4sVfJAJaDhOcpQWQs6a6lKbiuC2dsl1u6l+EijVc3cwz425EZNOJJHxrR/1RkmaOfiq1pehjgMXrZDqbzdJh33gywWdc6HaZIi7SUi+Y5D6wKZu6/aMy7R9uRbx9VoJEv2/vFiIfQRlhaAT/2xbHfF43Bjg9VnilhDbfsqNXsKcyssn40fzM8sEjCT1KIEzK5idOWPtKOik0HZrV2hF 9hs27kcv QfFZYdjJTqoy4Ghr/MVTmqm2fISp94xPTifE8NEEERMRGLeX9OKie/HQGE0xqqB37OIpb9GL5bCcf/l3s8kyOjuYtAjJ5abzSCif5Eca8F7nH4P39WdIr0MvQvPuLznbHV8PV/HliiiiUDttMog0AsvEmhJdweumEkA62ZN/E23DoYprpWu1w0tjlrwfoPXCuJCNf/lod8sEsc9kagxuDl4F7FMZ0d01BCg2l0+1AciIgKg4Z8bFd+S185CXzIW+EMdRshZFjWMIKcFk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use a cmpxchg loop instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-2.6/mm/vmstat.c =================================================================== --- linux-2.6.orig/mm/vmstat.c +++ linux-2.6/mm/vmstat.c @@ -885,7 +885,7 @@ static int refresh_cpu_vm_stats(void) } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -906,8 +906,9 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + do { + v = pzstats->vm_stat_diff[i]; + } while (!try_cmpxchg(&pzstats->vm_stat_diff[i], &v, 0)); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -917,8 +918,9 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + do { + v = pzstats->vm_numa_event[i]; + } while (!try_cmpxchg(&pzstats->vm_numa_event[i], &v, 0)); zone_numa_event_add(v, zone, i); } } @@ -934,8 +936,9 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + do { + v = p->vm_node_stat_diff[i]; + } while (!try_cmpxchg(&p->vm_node_stat_diff[i], &v, 0)); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Thu Feb 9 15:02:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19496C636D7 for ; Thu, 9 Feb 2023 15:33:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A57C6B0072; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A8DA6B0075; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 184CB6B0074; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E395F6B0075 for ; Thu, 9 Feb 2023 10:33:23 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4F40A120FFB for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) X-FDA: 80448147486.05.7B2437D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id B6BD4180024 for ; Thu, 9 Feb 2023 15:33:20 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="cV/Hwo44"; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956801; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Djq1nafk+T9HY7/d7T0y7Sk074zesK3iaSlQcrfIOBk=; b=k+cZ3TDagVgdW0ty4TZnjwkYUaWknHpgNi1xr52YA7KenVxrltfHxT22sPXBWD6fLUQLLF 815UYuvWyBVbjhTFpT9EZeqyHUCsdey6UH9O351HF0N9DtVlaPe3wtePAhcSW+o8lZO18v +e4WnDEt6UPMfh00j6+ZsHDMHMe4XWo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="cV/Hwo44"; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956801; a=rsa-sha256; cv=none; b=Y/L+keQWVKB83ZDnRYc25QHjGYuaR9PeuvdNxZwIyVcVT58itaXrfplKMy9ulTvpFpRe04 iTmCyGGFOki9/vcvyo28t6QmsOjRjZqPoUaS6vG79quruDu5IRvlHj3oHRBILVyoohK/Rg MnYlwKptR77orjLiBYkcRXd1DlO/oz8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956799; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Djq1nafk+T9HY7/d7T0y7Sk074zesK3iaSlQcrfIOBk=; b=cV/Hwo44u6YXekoUiUU2St1FfZ6Fm+ts1zorxxBdILv1sIVHqszH88+SgOJslpNbjSKP5X CO8k6NHFGbY+9y0AT2sEueV5xYFd7QCE3F+HpG452pKvcin7jJoFrvphrhb4J5Njyn7/zD Y0hYAAHYyml0CEQoPKjpMYMhjAgW6uU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-49-YtM2vNv5O6i_tviMBYIBvA-1; Thu, 09 Feb 2023 10:33:17 -0500 X-MC-Unique: YtM2vNv5O6i_tviMBYIBvA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 94220858F0E; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 668772026D4B; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 8B2A4403CC07D; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.901518530@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:02:00 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 10/11] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: B6BD4180024 X-Stat-Signature: 35upib1cpt6ghbcbq7e3pek9td6bubc6 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1675956800-810007 X-HE-Meta: U2FsdGVkX19stfBzQPP/4R07A27puNwPxjXIwrXKgU80co26M3ZqP9zRglj2VUB//2hcdtsBYxaxYpr2oBW4iEbZO7tjsdweEXWp0LmbfPinXWuX8VdOK4EzJLYbKfR9afInZwbeZhXZvPud7N4IYWQnGfNYGZ/4yAPFN+7TIdBOuOXDcU2/OpwOHeHwBeOgkPTc2C3+B/F2RsoF7VZag08b/oTHSmdaAp1Pnq4XWiaKQGfTnoUfQZvC11PEMKzn45Mu7+sW0+IXzptZiR4y28mU5+BPktjFF0KV6wmLVYc/ogNQoYgCP3W9ZzZbUTHBkUDzjDoBqYj1w3at2ESSC4UNxnUwROvS7LdWrWQBYraKJ7vLajerAcq/7RHzlajCMieKvxGACrNgUGtEU7ciASaZQwLWjnzjB2FcV9Mf5kXyWYY6lIIyjntTDTQ+btEQn2uyThXTn1d/6xTVWhSbJbgylIJ1qNho1PIB/OI4vCi2HT0jnXcFDCkHOj47NNj9i5NEczCUcWIvzzhmejY5Iy/H4PSB99YjHsoDiGHPBE0DdPricFbFnEfb6DOVp0Im67RQEWqnUZ3sr9tz2yR1mAHdtfvjr7GJKs6E0bZoYXcpL63+OuOX3Qmyh6lZ2mc3d+mXNWgCZpu4OqGPc5Lx+TK6zxM32lKzp6DPGkD+niluO6JIEr2guRU5x0jGehcmfYhiS4KWlqrGx0xOeSRqmrGBTro+6aCa1sRrGOBLUPJU6+dWeOyvCLfE0n3lSRkR3AsSCgFk/UkfkA4Z6cdfTmyjuFxvU9RcExep7KAGazhPZHcCCfZT8u8lmmjXNQrxfCrBM2fuuXRyppsVUG1qbT7XZXwXrvX/Y0hUtmLhsOO6Lp5CjT7XGiPxhgcOyGWNLQ5OesCkxOLcy7X8OyRpjZyOk6FsU3te0dyHZ365ki8aKtfwX7XRDzBKCIB3rBl15BLN0TR23PRN1ZsOiXk 9xDz31vh ZPpuIBITdd9yRbtlyoZniCz0p+1abdUm4Jb5hiKkwl707/MDWeaOhMfA4E9o4XMQfbYBIMvg9iBxK2iVAYzSupnKXwfLoQ3CkwUez70Sridh1FevDZnqr2dxUg8vW4yAmnKltpb2J6xfJdpbAk3P6hHPiyQbRQh3e+TTVqUAmTYdGMKNml6LQo2yvWbX+Ojndf82oXRQ02RQaqfk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. This fixes the following two problems: 1. A customer provided some evidence which indicates that the idle tick was stopped; albeit, CPU-specific vmstat counters still remained populated. Thus one can only assume quiet_vmstat() was not invoked on return to the idle loop. If I understand correctly, I suspect this divergence might erroneously prevent a reclaim attempt by kswapd. If the number of zone specific free pages are below their per-cpu drift value then zone_page_state_snapshot() is used to compute a more accurate view of the aforementioned statistic. Thus any task blocked on the NUMA node specific pfmemalloc_wait queue will be unable to make significant progress via direct reclaim unless it is killed after being woken up by kswapd (see throttle_direct_reclaim()) 2. With a SCHED_FIFO task that busy loops on a given CPU, and kworker for that CPU at SCHED_OTHER priority, queuing work to sync per-vmstats will either cause that work to never execute, or stalld (i.e. stall daemon) boosts kworker priority which causes a latency violation Signed-off-by: Marcelo Tosatti Index: linux-2.6/mm/vmstat.c =================================================================== --- linux-2.6.orig/mm/vmstat.c +++ linux-2.6/mm/vmstat.c @@ -2007,6 +2007,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2026,6 +2043,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Thu Feb 9 15:02:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13134753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4B5AC636D6 for ; Thu, 9 Feb 2023 15:33:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 905CA6B0075; Thu, 9 Feb 2023 10:33:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 52DDD6B0080; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2490D6B0072; Thu, 9 Feb 2023 10:33:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E81016B0078 for ; Thu, 9 Feb 2023 10:33:23 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B3A7C8095C for ; Thu, 9 Feb 2023 15:33:23 +0000 (UTC) X-FDA: 80448147486.28.201F3C6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id DB934A002C for ; Thu, 9 Feb 2023 15:33:21 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RxBizTIp; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675956802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=0xpraGtvv+g3M3VM6y/rVH56xF8i4HGSirYhvlOx/70=; b=5cY8+IGfWa5TNrGP9W2xbE5ggiM61ohN0ssz7MdroztAJg+JRqIdh+pTo3BMdH1hmkZNzx hSTjhlz+l3sh0HrZ1l1AcCKZZnzE5J10jyGWigSBLQFEfDPqF7cWUatmSOCJO37W80m5h5 7RKDT9l4TixSCgxf8oa6QP+sA0i/j0A= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RxBizTIp; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675956802; a=rsa-sha256; cv=none; b=L4pn504VIFkc8952+EuvCcFPYwe+AY9gdBUmF3Jnf6I/m5eoC9bOy1dzmtOfyivs4Rtx+i 5jHB2KIP2hDllQGa6MyLaC6HEktUfs1XV3nR6CxeVLmNEYhp7JckZQ0PdFdHIInbRL/obR W04UK0GEIr3TqktD0Ltn+gMagB/ggwQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675956801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0xpraGtvv+g3M3VM6y/rVH56xF8i4HGSirYhvlOx/70=; b=RxBizTIpl7rceYtIkGhMYsVEGeSmyELZFivXAFD14UNiVGGDsFrbKvW0UUEeBsyT+XE8fh 8nRsGICKwhl1vZtXVYjST9T6r35ycJLQ3eLh0mJivSIzTGI10FSXSpK1N7wJNaUxT4Vbze G7PnG1xGON+H/mA6Lo1zydsja2SX29c= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-311-HYnLzgtyPPihlf6ibXjY4Q-1; Thu, 09 Feb 2023 10:33:19 -0500 X-MC-Unique: HYnLzgtyPPihlf6ibXjY4Q-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 963A72807D72; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-3.gru2.redhat.com [10.97.112.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6C910492B00; Thu, 9 Feb 2023 15:33:16 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 90B32403CC181; Thu, 9 Feb 2023 12:32:51 -0300 (-03) Message-ID: <20230209153204.929333982@redhat.com> User-Agent: quilt/0.67 Date: Thu, 09 Feb 2023 12:02:01 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Marcelo Tosatti Subject: [PATCH v2 11/11] mm/vmstat: refresh stats remotely instead of via work item References: <20230209150150.380060673@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DB934A002C X-Stat-Signature: eskmyqynzsk3xu63mdec6wpmmy3zgo6i X-HE-Tag: 1675956801-165118 X-HE-Meta: U2FsdGVkX1/7WGft5lIhNftxw4rB6Ep8XDJmwo9E9T/QU+OMlesQEOMeV1J0Yb/6SCrJVmzTD+BsFxv2W98+Wdak/R9SyggvD10ISIkcPh+1Uf7XABhsr8P1JoDFbuIa1nCUKfPPk4+xy/nrrP7ZbeOwKDQZF38OWyJdrOs1tJYKkJAWGD5zS9nA1F5lCQzRFxFRtimU0lMqzTeHT/A5dmVuMpki0ZYk6FMpRRuzsK9TvjXyGtYCorjYHz/N56K3gpTXXUePdSdpNLnzVKqOK0NwAW8WXZ6cA64Gi98Orl/57qW1PhvHrMiRYOBCVEfDYamiLrs8/6qmUME53W0JIqSSNgaNtUNp+5gOo713SqYAiiDiyTLdujpbCA49v14Xz7C3Ue7JpY+46jKbPxSYWsWaiEzGU466bzTebPngvhfW+2967xHCAMRzDIUYwLQXHdwQOu8GQgK2UstnpafjPR20/4c9aINeAOb/4z4axhAK5sJn7+lhTO7oyiqA0JZGFmpqACbwrnj8Ab2EBlY6pkC/iBZw2cAg/bHbvjxXYL8F15P3aFSpjSjdrQH+JXgOhrI0oogdv018DdV6GHWheUvWnZ9n9fFEYXO5J14EbvO9PE2hczvPSVsgVki57FnAvitmHu0ce7/6mDkEXNluXiHGhn6LR38ubKfgjDe7BM5L8yq5EQUOIGrc5gqZ2T9p1B54GxDnvVDmC8R6ZT3Y7HmMsEraG6ueh3UUqPtIF8tHbey64i7hWqpzv9QWOv4cbWUKclZEaYlNV/iUipi09Uj6rlyk+t97Bkf9MrIilPicn7Nds8axocmH7HV03fvKmSdGoNrylSWF0jrGLXCVZbZRSyl9bEfHk7sO8l3n+57HwOKdD5hsQF8EImq+ZCBr3juFQOS/eWPo/9xqjrnj79zejFY951TpHzbX74hLkPI4C4MJeqnZJscm73OWWU8z39M/tz1xTk5pznNcRfM duuvmvdz sCvITxQnP6+bwWdEe4w53g3k+52aZYtKOL7amoVlaDnIEDpnNcSF9n4shYmv2ke1NcxRLiVzn8UmnHGsyFa50Pa6AVxMxoCft4MAzBEXMZj4/anAoP4LKKEU++3tRPRy33ttLPF8vX3eT5L2ocDiegC3d72kKJPohv5ReJy3Buiao2PVRC0QtDFx44RrpIJwAGdn4S9I1W24OGCo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-2.6/mm/vmstat.c =================================================================== --- linux-2.6.orig/mm/vmstat.c +++ linux-2.6/mm/vmstat.c @@ -1865,11 +1865,21 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS + +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1889,7 +1899,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2009,7 +2019,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2019,7 +2029,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); }