From patchwork Sun Mar 5 13:36:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFD97C6FA9E for ; Sun, 5 Mar 2023 13:41:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5DF86B0074; Sun, 5 Mar 2023 08:41:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE7916B0075; Sun, 5 Mar 2023 08:41:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B38AD6B0078; Sun, 5 Mar 2023 08:41:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A229C6B0074 for ; Sun, 5 Mar 2023 08:41:49 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 71232C063A for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) X-FDA: 80534957538.17.ED9872F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id B1A1E180015 for ; Sun, 5 Mar 2023 13:41:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MylP7h/E"; spf=pass (imf06.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023707; a=rsa-sha256; cv=none; b=OsZ8h+UQwpCmDd7wl73o/U8Hc7Ag9LleuhO9n10RsEVtiGPOGxCV6BipVFPKs12zRP5z3h VUC5/3QhDkDQ4Y7BfVn8xLJoFXell04e9yLBOpLHYxh3lu+P9nfwjJDbmtCTDDDD1qxn6n VxOvzGrHgqYGNwFgGWIKmFbQJH1ulMA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MylP7h/E"; spf=pass (imf06.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=gz8YmAWe77MPk6KLnyeDeBjg9daGz/JZkyszI8+E0yI=; b=mQYjJEhA+wGhbJDDOcium4L2OTPL0gqucK+Q9LqtsddaUGr/3mB41aHVxL9bAlHrQ8/E4G 5NEApPCAf28JbP+KsA6pb1jz6XKj7cTOSqo4wn7wC6hPBmGtm3OSNjeaj3dEcCLLqmIcZ3 7s8UO3/aUxX9BS90hzyeQoSRvapu21E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023707; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=gz8YmAWe77MPk6KLnyeDeBjg9daGz/JZkyszI8+E0yI=; b=MylP7h/EPQHbDp5gvbsdvWRLGUAx+dwTz7co/y6+oCG//mU2xi/yoJZVyszidBG4HQSimc GVevrYjnV1T8rnBkR/QeM9nyZRFrGRIR+zcrnYI0uLQjLS/IkcZQyuAXv/6qPOzkEo5/nR rU+m0d5XlZQoBONMiWZ4EQ7YOewZXD4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-394-d6kOwlJUP_-kTTRd0MJH1g-1; Sun, 05 Mar 2023 08:41:43 -0500 X-MC-Unique: d6kOwlJUP_-kTTRd0MJH1g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3236C185A7A4; Sun, 5 Mar 2023 13:41:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A78002166B26; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 17F0D401344BD; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.436759976@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:36:58 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, David Hildenbrand , Marcelo Tosatti Subject: [PATCH v4 01/12] mm/vmstat: remove remote node draining References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Queue-Id: B1A1E180015 X-Rspamd-Server: rspam01 X-Stat-Signature: 6cse859ajajspa4966wi7k6iykyq4kdt X-HE-Tag: 1678023707-326374 X-HE-Meta: U2FsdGVkX1+tG/msCYSUNS2Y85Bs7xpXgrJQIK10oqcf1cBCNStRpE0DQjiNctAW3TVzE9i9FC4ZUroPePFIPEpUWh7KEmpiZK0L98pGwmaNd8vqejUqXfR/FYQdDBTmU7HtHDD6X1abucGAZodsqHOMYr0URMuXsfIo8FcvpgaPKc3ogoUI4bBtur5YpsvDGRyVbchSgd52ysxn/mzG3+D/2neBvyKyidSpawSjTaDzGvTlnTIQ3jkq7TEHCP6ZpePrscTxMqRF1fJAGIq2da7/XX3Fh2kjZ/RZCqB8nPQnt8hqgI2oIkTUOLP261n7S5h28PzekMf3p8J+QEuG3WhnSW2j198CU1zeKfhPIsVxbH4Zm8+bjloVor9AWZYjUYV2RAjc52vDM5uZBEZeap/dRUkQu6GiDPt3AN9c1NID/HAaaew1PP8KRhG28k2irArKSI5ojNTiT7M+VeDWp+x35b75iPnhfux2niZT4akJwV138h6BCgTLrf6UTsgBmHPygFxgxbS9o+RK+mFj8KPMJuYv/9p0q/RsrZzSkRUCBh/ovJK8nd95LBACBeu8So3J4659znADHcpXoJcQFzYu+1sg53eQxfLb0Sbr+azzYO0iLpgN43WzPRaqXlBZdj1EA8WNdCXPfzdzGpSxjK9XRG8qsGS/31x9gnimGoPXuv3PZFHBAmjkKMcDtwLVyuQaQDeTmPBxLNZ8bG1MoHrajkgOiLJXr6Vitd97AYuayG7zk609Fu5KxgjGk+1p2EA2223GGxbhQg8kkYlCHP27aN3tluVh87EYGwODOgkqsC4+slJtzt8WKdmH8mnrIKaZxDlMJYpnsDe1Du9hhnidJto/Y9/UZ+CRRl4XyDtjvYDaepI7buKNAuVSUF2I6eJi/7ENC2nKYcP57nUCRHRKC/y2sfMrauMeX2jnOyJIe1dBFs2hJEQnFwF8MjoCwnZ/srS3rZtkpBU4mC8 5KNCns2a YS0SS11mpAwoNqJ61RScL6TIUfGLullKiyJPaCmSPAuQ/uwWSeXykUqhzABkE+V2+ex+LSlxEXSJ7LSIEpPDQ0IsHMd2QNfd0OFT0u4p5i2KFEaGefp44cJTj3xbCMvGBYuOmmrXWztzvz5x6BihkrN2y+Zzr20q3UfFPajQ99E1tZMjPYu2uGWARo6KWG2CdZr2i4CJ2ZlMETvRK1PzUnbnvEw9wnMRwhBhlomNXwiwn84I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Draining of pages from the local pcp for a remote zone should not be necessary, since once the system is low on memory (or compaction on a zone is in effect), drain_all_pages should be called freeing any unused pcps. For reference, the original commit which introduces remote node draining is 4037d452202e34214e8a939fa5621b2b3bbb45b7. Acked-by: David Hildenbrand Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -679,9 +679,6 @@ struct per_cpu_pages { int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ short free_factor; /* batch scaling factor during free */ -#ifdef CONFIG_NUMA - short expire; /* When 0, remote pagesets are drained */ -#endif /* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[NR_PCP_LISTS]; Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -803,20 +803,16 @@ static int fold_diff(int *zone_diff, int * * The function returns the number of global counters updated. */ -static int refresh_cpu_vm_stats(bool do_pagesets) +static int refresh_cpu_vm_stats(void) { struct pglist_data *pgdat; struct zone *zone; int i; int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, }; int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, }; - int changes = 0; for_each_populated_zone(zone) { struct per_cpu_zonestat __percpu *pzstats = zone->per_cpu_zonestats; -#ifdef CONFIG_NUMA - struct per_cpu_pages __percpu *pcp = zone->per_cpu_pageset; -#endif for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { int v; @@ -826,44 +822,8 @@ static int refresh_cpu_vm_stats(bool do_ atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; -#ifdef CONFIG_NUMA - /* 3 seconds idle till flush */ - __this_cpu_write(pcp->expire, 3); -#endif } } -#ifdef CONFIG_NUMA - - if (do_pagesets) { - cond_resched(); - /* - * Deal with draining the remote pageset of this - * processor - * - * Check if there are pages remaining in this pageset - * if not then there is nothing to expire. - */ - if (!__this_cpu_read(pcp->expire) || - !__this_cpu_read(pcp->count)) - continue; - - /* - * We never drain zones local to this processor. - */ - if (zone_to_nid(zone) == numa_node_id()) { - __this_cpu_write(pcp->expire, 0); - continue; - } - - if (__this_cpu_dec_return(pcp->expire)) - continue; - - if (__this_cpu_read(pcp->count)) { - drain_zone_pages(zone, this_cpu_ptr(pcp)); - changes++; - } - } -#endif } for_each_online_pgdat(pgdat) { @@ -880,8 +840,7 @@ static int refresh_cpu_vm_stats(bool do_ } } - changes += fold_diff(global_zone_diff, global_node_diff); - return changes; + return fold_diff(global_zone_diff, global_node_diff); } /* @@ -1867,7 +1826,7 @@ int sysctl_stat_interval __read_mostly = #ifdef CONFIG_PROC_FS static void refresh_vm_stats(struct work_struct *work) { - refresh_cpu_vm_stats(true); + refresh_cpu_vm_stats(); } int vmstat_refresh(struct ctl_table *table, int write, @@ -1877,6 +1836,8 @@ int vmstat_refresh(struct ctl_table *tab int err; int i; + drain_all_pages(NULL); + /* * The regular update, every sysctl_stat_interval, may come later * than expected: leaving a significant amount in per_cpu buckets. @@ -1931,7 +1892,7 @@ int vmstat_refresh(struct ctl_table *tab static void vmstat_update(struct work_struct *w) { - if (refresh_cpu_vm_stats(true)) { + if (refresh_cpu_vm_stats()) { /* * Counters were updated so we expect more updates * to occur in the future. Keep on running the @@ -1994,7 +1955,7 @@ void quiet_vmstat(void) * it would be too expensive from this path. * vmstat_shepherd will take care about that for us. */ - refresh_cpu_vm_stats(false); + refresh_cpu_vm_stats(); } /* Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -3176,26 +3176,6 @@ static int rmqueue_bulk(struct zone *zon return allocated; } -#ifdef CONFIG_NUMA -/* - * Called from the vmstat counter updater to drain pagesets of this - * currently executing processor on remote nodes after they have - * expired. - */ -void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) -{ - int to_drain, batch; - - batch = READ_ONCE(pcp->batch); - to_drain = min(pcp->count, batch); - if (to_drain > 0) { - spin_lock(&pcp->lock); - free_pcppages_bulk(zone, to_drain, pcp, 0); - spin_unlock(&pcp->lock); - } -} -#endif - /* * Drain pcplists of the indicated processor and zone. */ From patchwork Sun Mar 5 13:36:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FF21C678DB for ; Sun, 5 Mar 2023 13:42:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F2566B0088; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 523BB6B0087; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AE506B0085; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 044AC6B0080 for ; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6289B409E3 for ; Sun, 5 Mar 2023 13:41:51 +0000 (UTC) X-FDA: 80534957622.03.0C7F666 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id A6646140003 for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="I0/S9n/e"; spf=pass (imf23.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=1mrHCtSeApdR0VGw+VP+C/SC7l1xWr+vWSDfRQv6CBBZaHqTw9fvmJnsbotQxLyTKGtXJD TAXjt6gcRQeucp8URKwTh3fDIzYyp7DfxsjoUbb1ZyBhFK11sq++Wb0yqxUwz45LNsGO+t wB3pWzh4JKzy6f2XdR49WWuP6vT6tzY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="I0/S9n/e"; spf=pass (imf23.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023709; a=rsa-sha256; cv=none; b=58Ph+Lemp2ZS3zDnUkGSEhVXxkpf3RSsUcVqrSw3ZMnFP2gu+r2s1lmKW82Er95+jCZjFq dijUWKFIIzHhBPLquytU3sMcHPWtgxnKH/YqJDh6unV3Qw3VF5HYP8bfzvibXmMECkmceR 5554PgRxBHjqn+KSADNIeavlRG/l3+M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023709; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=I0/S9n/e/8zlsfK+kGCAa22zoAeh+l5TArC6sw2Lr94jgjTEnDwMxgMpz3gdNNUD7MHDdS ECiKBrcrJQi8LUnMSmXGkoNA5E/JWpg7tA/w2di+qexhfI+EkkqnmRCTV6x3jUxzHxt6ng ELy8jWsfDchd0Ry9m3+8J0CeTCY2d94= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-660-XsuFXv00PfuNlCNNpBNWnw-1; Sun, 05 Mar 2023 08:41:43 -0500 X-MC-Unique: XsuFXv00PfuNlCNNpBNWnw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ED221101A521; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A78AA175AD; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 1A6C8401346E0; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.461554895@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:36:59 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 02/12] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A6646140003 X-Stat-Signature: er8b9gzp69kyouk3zjs3nnjd1w61c958 X-HE-Tag: 1678023709-694046 X-HE-Meta: U2FsdGVkX1/YOhl6UFYk8p31gCE8tvtTzYa/qzPUmC0SHM6fF/4S4Vo4XsQmyNmQLZpXMhmDEyAH07Bqie/Mlsu8yxkFNpdXrOu7ANoEqAVErM09fL2F84wZ9lU0I9x+Htwt662dHLAeMpOGZK+IjoLvrRSm0wfy63iE4fYq4hseLBRjwn9Cd5ELxJ/hGaeUJTj9JfbN5V/9F42WSrG5HS9QkHP2bIH5eJSkVo6TfeqT3c8tf3aQdphbrEXP9mv6585pFeTQr0K2N/iKUk5/FB0dLVBhoq8cUiDzNhlsvFx2wjQ2qnEpXDbWap+6YcVi0pOsr3iEf1f9VfR5koy/RYImm4tbz/C8cVNbmGLg0BV7gq2g03Ff69461ixGJB3hTLYyJwwermJC62LTcOUolPxjrMKJIxVHb0heTgKG97QhD7ZAaMnmcDW+mgHZHWdQ47xM2rHPk6s/mrO/ubOTbxT0/pHXPaNSJMWWdcDrOP5xGJk+YkCbifYV5Dknq9X3rT36v7eQUC0B8kEP1o/o+xhpMUl+44EGQURNfU+/X/SsRuOttQ5y6pSgFgZH6TJlaI38Mv4glKsrsp4L3DqB4HUCekLxSoINyxu/gtOD8OV+pNfImgoyYN5Nkt5Tr2ZkPeI2xNXicavGjP5GiRYEmighxFINlQLjepbhbePo5tJaBZtKpoyvU5niXc7EVUmWiUI50poK3kfa+LF+VJuOaCy4ojar+ph1bcHRDFjosDDMKZvVhU0T2rUiqHAM/b/ckEMAU1yy0zTPVkSvHmZ17A6NDBhd6BOB3wAn2smqe171tV74ATh68x62nkBQMz0NQ/fjAZ2mGfwjrQVzKgpHlZSPDsTYxDAxDc8zHZW7z/3MZCtSgBvnEK14aXgkVHoLBWBvk+KvgDi8sJLVdBRbkjHHMVY3D4Xme1XPxn89/R6oAxET83yYmzq8W7qU+vIfaPB2lpfeQ7GPmeZljs7 rnPAuXXU LhPvvNzLP0D1xpa6qe76Nle1J9igDoNEr1qJaWPHuTEsXOo2dMnGB8Cj/JMfzFlPYZimGdaDGs+ssCq39T0jNzZYwchwyQWoRfOv9kygXvx+mdXwNEAcvN9cRSFU4ieghlE0ajCoppzCSep27kf2jPMnH+NLNMslW33fH0QcONychwL+doc9d03mG6cqh7uz4WiQq7xmodM03n6bvWYrtp3P9ouyhQUfIJ8HVNY50fQNJ3BPBnlzlbpL0CQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Sun Mar 5 13:37:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A852C678DB for ; Sun, 5 Mar 2023 13:41:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23BAE6B0078; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 165266B0081; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D35916B0078; Sun, 5 Mar 2023 08:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BA4A06B007D for ; Sun, 5 Mar 2023 08:41:50 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 90E47140652 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) X-FDA: 80534957580.09.E35CA6D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id D15C140005 for ; Sun, 5 Mar 2023 13:41:48 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dWvOnrnY; spf=pass (imf17.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023708; a=rsa-sha256; cv=none; b=Uo49Q6eWq+Xrq7VwQzBWN2ydUdImjfMPaYVnF5KJIbnQC/YzRCqR8aPrjUwk0QG/3nhUte LvPmOgxPwGdtZ2bsfxYF127IBHVq9gkbF+gHefBc3plAKBTppcVk5Q+k+97XrpEI8XlCRK qOJ6johB6PgrDZ4QE/dQjuNPpTYhSrc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dWvOnrnY; spf=pass (imf17.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=AQyj0glQbyFUpUMe4rCoErn/QQ1lVXwlgW8bQ5kagVV6mLa64jLhxR031QP/smFUv0ZmSz y6q4NqPIw0cfGxGUkXF5RXBiFkBl+LZLMLQutwUrHm3A0JPfJj+49FKArmaziqE88VfxzP GAN1XReY41Sli/OLyL8Ry43Ci9l4e2c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=dWvOnrnY/4g2dCB19Eb0m8ih4pub903WylLu5eFef71B7wlUXaYYjsdLIEQsX7qlTBIurW iagl+LCkZp5FoqyYLmNRW5VBw9/uG3TnGU9JifknubCtU91Qxitj9ZX3NWitQE1WI83sXI cl8DVfboMMlLl2FVTrY4UY6dWgwIFb4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-wkTFG0cEPRCz3GrbmFjzyg-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: wkTFG0cEPRCz3GrbmFjzyg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A6E6B3C00088; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6E02A2026D4B; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 1E6FB401A0792; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.487486353@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:00 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 03/12] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: D15C140005 X-Rspamd-Server: rspam01 X-Stat-Signature: wiw4amcxbu54em4ref66169c5kzojzym X-HE-Tag: 1678023708-608671 X-HE-Meta: U2FsdGVkX1/0Jhb/oE5pB8Yw0CnJAIG0KUMZjHyEMWxirQd0Jx/D896r7SYe5xbnJenWawsY6NFOPNP9YbzFzSpuSjsovPDYMS0X8Nz3ak0SRxwbp1s5HkEBKucli7WL8pvLic+LlkBd2JBYMrYeQshweHj7ONpOHQRYEn8D27mrB0/AjtMPxZrNGIymSrpyYCaAM7/GSSwxJRHJs3o43Y7S1DvttoCCD+4MlWxUYKqaM38UscS4o0kZdTKBL1K5Gg3FjCtq74WfU6KJWl1cBDmSRTvWfE6pBpN4aMSrVuE+ABbJnCvuoMBrGE+QBaD4UvjS8ezmo2jeEmNYvJNLKNuaYyE6Afwima+5TqZjsaRptx5HW1NcFa0PcuhumOqmhEU7dTmM1kDeWIu03i85Ty6UCovKJI6Fii7xKWKH2QOl7/+BTe8p4uClSZqLJkvRsNbdMwxS6VXV+R2QUTA4lPdn1oYZsD4rwCUXs6LG06PfvlF3lYZJxFjVE9U17If8XPwljEvhQNHXkp3fI0fNs6bobUtqP5qGUkzg5Xf9pb7ywxgfpyuFXBY+0TiSrd1W1W0HHIXMIi8kqD2coOynWTQe0XkjKXS1ewuV1AVkrp2e+OO5CNAEwWV3Aq6Hf7pPoT37DST9u8SbF6RM3x4dtk29NtEOw9eIAu0nkqd4Zq65IOOiSNBhW+IE0V7sKYnlb3k1mgXNWi1J4iZWxIvcDpnCUcxI7CRGejeBGy6085HpKBIL0phqIkosR3gYwSOKg4PFTi9tCKlG478Z0wxExoBJqO9fIZSvuzY4AbuJkhlWJBDFzfyc3Rpx0NPmYR7adSOKO7xZFJQhuV7wiKFnWPjDvrvQX7WhkRHIL1VYTasRG+3DzKowhxbHRr28kfJbaOlpKL3OHylCBRCwht4aNW/fLhIf856IdcAvJOb1vytS6LMs6Z891tlkEHUEE1JMHbJU0O8grryFPk1JoCS D7w0+WKs 8mwOcn8mebkOWKRyT25bwoKmNuZh5nSAZHv6ea1tXJgP4nlDIzHE99Ahws1aI1eGaFG8qfAKst3+JDmZ8vqZtEYpB1WLRPBZ7nXj4bgZc9FtrnaMrj/2EU0XADpsf4a4tSuaLzVLgQ+s7Rp7LHbHU2e4UYqpzAqlSAlcXRah3Or+4zZir1VI23AyNklr0hjRQb9j2uzYMio54AVRCF6v6+AWhv06E2q8Od2dIqftyrFuxWh7ygAEvSUFcVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Sun Mar 5 13:37:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C13CC61DA4 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B9CB6B0073; Sun, 5 Mar 2023 08:41:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A5706B0074; Sun, 5 Mar 2023 08:41:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F96E6B0075; Sun, 5 Mar 2023 08:41:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3EB2B6B0073 for ; Sun, 5 Mar 2023 08:41:49 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 13FC31203F1 for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) X-FDA: 80534957538.18.EE9BBF3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 561A72000F for ; Sun, 5 Mar 2023 13:41:47 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ckrOSTEU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=vqIXHdtEh8A04gYtfN3OkYzeF10b0IIC6dPjlFrQC5e+sk8r0YHTG/sELU4NYvKERMM/3y Fs63biW4xmjmYtDlRycfMsHUCagF9i6OyhZKhGRAvbKKsEpJIhKQDQ4JkKCkcJi0H9uGsq tANIuNFR+KKtiVsZgm5k+Q8Xyu7/Ea8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ckrOSTEU; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023707; a=rsa-sha256; cv=none; b=eDcQKDpt8WUS+UM+z9d4kCyupkzf3p/4Bthiv6CTT91zgMCx7L6Q7dxFME8Qng9RhLDBCe nx+zYmNUKn8KleNZjenwPm31q9pWlfKuHUDcmBhdubrcKbDL6Kx/qAu44wtUu9XBdh+Aet UM1y/TXBO8Woh6ynhWyd+pa/hx+yStY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=ckrOSTEULFNGx/4XFixAsaC+i6eQ11uraKUjf2aHJyf/D6WmaaD+x5zUq77DIEGBMQZPPk K7o84HOeeEbm6zhjZLBWK6KkrmuIBox6XBcdah+tUf4Nav+xIuc34Q6cDGpxHEfmejVXig YW2FRO2Sn0slD6Wsixv7xqgL+UPPsSM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-fuXqWRBUPB6hB7qRysWiEA-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: fuXqWRBUPB6hB7qRysWiEA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8BAD85CBE0; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7B888492C14; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 20DCB401A07AA; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.513805071@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:01 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 04/12] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 561A72000F X-Stat-Signature: ky3j44zeiq17sndxeybq8jusj9tg8gem X-HE-Tag: 1678023707-982042 X-HE-Meta: U2FsdGVkX19jG5LaoAA4wVG+JrjPBnfUqT/9iXuzm+PZAZj/5Wg4w8ItJKWu6wCLKJemQLfpNJraO5qw97WBlqS1OBaHuWZ6Oeiq1RcYhxa2uoeXS2hXkn0XSh+u9ehyzjxQIBug5Gkt/EYpFfzQsfUhAYEcA7abYaBZHDO2R3BvsCbL2vPkMxww9raPMDkGrBjX4y+KqmSxB90yg838Uhx7le/wfh2krZWBLy0dLxrJIaPdpyDdw5r0j9UhtBrbtBW26BuZHmkeUlIyTYpwKNRrwEUlY5xpB2Lw0C/s2dPaT7IWU094T5QnDl0W3WtqKgqWJ1Yo9xTJCvPYzoaBdCItOXxE6xegdxhsdY3fsbjhUqvYx7NrxqDSq1OWcZBrcBg2JAmV49pkyo3mtoqYNGtGFaCC32tXGRqh68aZFMkwWopfCu1RWjXHNbf39Aa3r0Z2niYqL+mU6ZXfW6rYRSo9lgiAAMtasBiJOpHu2uaCeJnIyAbZ/yb8hdn8/R9cVyCvXv9tUDmqG7kQeHqGXNBNP/hEc30K4j5Juw5c7967dgU/BBDIpqzWCLhJFoae+NIKT1f8e5JUefbzksf5FMp6a4NEupJ8OgKMmOtQeI0lY9b+9J8IOzdfmyYMi/b+Mla/1ikd3uudB5oJvktAeU9vVJy16e/8QP/Ljcq4U0KAlgtJz/uWJVRbrJNVtVXn9FJ+Wj6Qc0ddTXsJe08xVwkjaL/7vrq7IYgd6I78ZfY89Iir23173+jAuYPhfwRc9V5xTuEFCmK9z4xrWcRZgnXS0dGq3J+vb5Xei0+eIH31VcIDZaglbcP0qEWSQ/rxHwzzIoka9a/wZqbMzPuGfiQgGkEi0Pbu0u9Hije0dEzaYOkempTQJ1Pavi3ZMM5d8cgp2b8TpnAJN/DGx3fHrgipVvRCXqh2fIwjPdOgCujPSbNLD1UmBHPZeT7B4b5GHkgCojxFZbZYTGdOUwl JBNddrh0 YfGyGtM9BZSSFNlBS7OJEDDuRx2qM4RQgi8x1a8MooLvnihaKkUPY3UBrU2+0daW+ZGMDcNI4j0q+YRGcCWm7dErIrmh6G3gAzSmUd06UXg6d0chCT8lmWCoAk1UOPGY+5mJcZ4PTXZVjPJ8cgsYkF1Fa2rahAy69n0aPVwgBKnfm1/tzKI8rEUZfZD+NoCk3R6IlIUAlnzrCIkd9b+NjbqFKlhHImR8nHWg60bda2vXn+emhJDHMZXm/2Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Sun Mar 5 13:37:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7ECDC6FA8E for ; Sun, 5 Mar 2023 13:42:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87EAC6B0083; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7778C6B0080; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E8846B0083; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 044D16B0081 for ; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D244F1203F1 for ; Sun, 5 Mar 2023 13:41:51 +0000 (UTC) X-FDA: 80534957622.02.069859C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 256C518000C for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wcjs9ZPK; spf=pass (imf16.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=t4393LbQ2M89qG+LgE6E/CDfUtKd2OarHqChtgOB9r7khIz4o3Qu9IMiNj7ryIVeoicTqB PEPWlQs1D6Y0Zd1wtkk+2MydPQQ64OUHaxRgCOVtSrk0NfFhoko8RFzxRevUvDFJEiJxVu gMqlqcJ/zmFMR9IvBCaZ2IzwOcMzZok= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wcjs9ZPK; spf=pass (imf16.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023710; a=rsa-sha256; cv=none; b=XZO40jGQrYfPLvRB5SpxKqxkKv4hSCTUFIckngg0VuKq1vKPyJP5Z3jUBVzWC/tpaPW+E9 /dP6lV5H2UcUdnZ/ngqWNbIK7kGnm6ZcKUWQEECkxwviUkPdlmTpjX/GcS9Fs1ZDEmmRmS 8EqoqMZy0BAdlF9dPqcexvMqAhx1nzw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023709; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=Wcjs9ZPKoJlSCePSU5BJ+aLysCaE8pneu4ugVsJ71zgV9+C2hf0TRdl2jla0mc0+paO2RP nmz1r7X2SNP60fG3mrGgQ7s7VkKMLO6u+HrBbhgg3isK2QsMJB4JnAE/zGLHMT8tb0BQrx 388yQxgiy8W5D6HgI7DWKlPOzYDOI/o= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-1DIF2uMFNvavzzL6C5C9fQ-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: 1DIF2uMFNvavzzL6C5C9fQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D13F41C0513C; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7E9984014D1D; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 26977401A0B0B; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.537803923@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:02 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 05/12] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: nb7uwpskzdgr9ejsb1ekg69e7m1kfay1 X-Rspam-User: X-Rspamd-Queue-Id: 256C518000C X-Rspamd-Server: rspam06 X-HE-Tag: 1678023709-264719 X-HE-Meta: U2FsdGVkX18ySEtbsb4S+QOyMb4hXqbBx+NntCGusNqLx1SGLKUyaNyT4ZE41MgotVcS5JjR8eJnXfXtWaMk7wh1ec5k+zF01e5GZQNwb1gk7tC/dzCniFnSR5WLuD38L7hTrWDZ2GH1sZeqOVw3RpGSaOPNbuws9EuFNAqAKa367ywiLBhBCyg852p/u463pRME858Cd98nIuMfNTVxYo184gYBR54nSZmO3UMF0DTcKLgmERoKH6I2ybArusE0IlrBRpLBBmbTLinEmL3W4ElWe05tDzGkynYSrOQmdj4+ADbpRlnoG4+SmEXrALrH9TuCpsp53QhCASgrVPoo3lQaHriHtZNUiKYvRfXa/4b9qL7T2bYz6RLxfh2QmYgDBxWcesVAJR75I1PF453cgLai2DPeCg477/MUOPXsJW9EnHsRaW5czK+S+8Jmb5P8sJmZ1ZwaEv+05paYiB22PEYA0zlrOhXQSqQCa9Vo1cpUZCYVMiya4lxkj5Zo+mSzyfUuVn/q/RS2GncrwN0Ru7VbU0SK6XNjeDfwhhN9ftNiSexznHeD+2nyFK3oR1sA10j3kPFRwLDTya8uzzYYImr7OfmoxDDLwmiwE4IT4L3rC3h+A73M1CjbLl67WhD1cSBR9bTZ3nT3wb94UXhF8AtzDsWpX9QmJSqGgUYvW817afDz/jL2t69iJkPXymWjKBILH3/tvL7Oi6q/zLPED13s1sIxk14HMg8uiJvr3c1STJX/v5/PQivRRjznUlpVaSrn9rjU/4uB4omHGxwKVmar889jb3LiblCGFMTRFmMrP7OQwkKB08URZskq0IqaWKdI+oC4+m1ntOcdQ0JWWm0o6nNRjFTW/eOkeRCC8frX+F6hv9FO5ntf66hFrY4CsEGUqBPYyu57kfgetdTApHPe3vKMhKmINu5H6xUvHTsWtLU6lddZk/Vd6VdH9zMlfNJChZvBel1TbSeTdDe 5+m8OpBb Hpv5ggLw3DUW1jEWfW374ms8MiYMMUpF7oaSZEvwE06IDEaPEalCH3c56n2MYhh4B3y/+j/Xpfby+CkVFK1nOndhgcxIiYXrmIJbo+I+70d2NumPLgy2btoPwj3go5F9sNFFbd9db2ntGqRp2wiQTTZWwPKgDD59pFrTm0bMT2zut9I3bC6spFpB+W2o9LlPHOt/TKuRLHeQ0e6uRP7/k3c2vulqk28mboe9xrzrZb6Vsy8+Zn3DaV6LDQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Sun Mar 5 13:37:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50799C6FA9E for ; Sun, 5 Mar 2023 13:42:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAF6D6B007E; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1EF76B0083; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E0E16B007E; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 59E706B007E for ; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 29E8CAABBE for ; Sun, 5 Mar 2023 13:41:51 +0000 (UTC) X-FDA: 80534957622.06.27511B6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 6E87F40003 for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EpgRVJQR; spf=pass (imf11.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=jXjuJTA3/sbb1cq89QpeQPj9v2ty6oO3MlRqvIDw9I76Jk92RsBD2cJQn1z2ZWgL9Riwj2 ahlJ+XaJhF36RbRHhHjAbmVDQLG6xV/4PmggyNYeCiFZqWp08SipmH9EjPo9dsEtac5YZw oFlekCSIQQIpr7IOrfXI6bfdAbiaXww= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EpgRVJQR; spf=pass (imf11.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023709; a=rsa-sha256; cv=none; b=N8+VQBbjl43feTLMMW4+NT8yXOoJpq8AhZQEw8WSPWwQ7SViEmkj1/vRaSKZ3VysDajq4Y 2L/xILLbceig7/+cYoogkrj+8W0JKagTEz58QcUg+Nsf1/BHHFQQrJHigqgBegmskVWTx7 BjLI+UBi7Kc44/eLcQL3jFuA1uLTrOQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=EpgRVJQRYy1jHsb3Fl1VCaVd8MfIWV51kkPDkAVV2gYbc/YTBOFoWnSwViQQGuGaXgafN9 BD0nyTAZb3U4dtN6t6T6Ls7h+2RbkX8LmsQy/uE5eTuucfs/AH3o6PjerzRQVRW2isMgX9 PppGV6HLyUUa0c1+3ATWeo7b2yyjoxc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-tu_aXllPNFqq4K3hRNnk7A-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: tu_aXllPNFqq4K3hRNnk7A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F15BC1875041; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8EEAC2166B26; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 2AAE0401A0FCD; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.563070832@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:03 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 06/12] add this_cpu_cmpxchg_local and asm-generic definitions References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: d65okrsogw7qhq3st4tp5rtb8bkkzng7 X-Rspam-User: X-Rspamd-Queue-Id: 6E87F40003 X-Rspamd-Server: rspam06 X-HE-Tag: 1678023709-968753 X-HE-Meta: U2FsdGVkX19BgZFMnUJzJ2zYIsIqEO7Yzes+yguY47/9/HnjEW2lgZZpgc7WIxi135SxJOdDp1RWpZ7XvryhUwbUJdjZez3/JzTaW6jg7uNdQWtIlWlf/VIiBbYe22K78DbtWxutvnqa4TB+w2HJxeyFa3sKWY0VWQ9lUmIPSR7tHKfVUTSUV2++gqNDC7/tzItBAhPo319zvJw10mufCdEw15XBl71zGQTmwozao/2mJV0oau8j53mlXpEF1Z1nFGen6gLiZtwZzVfIiZYG4GT9wBB5XcGGir7pXEhc1aaLBugK06B+fLTm5tUl+UsbAq5gKudHnqtrfgc+9/Am/x8H3T0Aaz6epaHxUlgD6opGGjQ21T0u7+93CzhpWx3OCLLJ5jHpENVyv7Y7akEei5tThaUgUrhI+xd5l6VqatScIcMtnUYeA2/zHDQZ4RCnsGyL5oyrVXIAC0AhMFpY6jCbvLOoUSWR+i7VP/pHWEFoJKgReJYDgoNcZGfZbTC8j/0kLNaOqQMtHOpj1jo+041tNgs8rW5GG0BV9KZK+qyBL6z3ORehapbrUgDYIXZVwnoNN4yj/0Q5Udug72tJo+slBTK/ezhjigOfGBeG4iXxCIctDcel66Vcp9BnEsTcXGKVf7U4LEOoQPIFs24QkIOHMcfBX63f02XBP6NS+V1HeICCqbqNpPqihrWrwJoDfXFRxef0rGGiSB5qak0tBfrBAR6ofpxzvu/H46f36a+5u78i0A+KIceYx8t7btSu0SsLZf0WIjRyALyOQyyHpbN3WEotRdPFulb2+bxZpgXHnnWVGzuVsn2Zb8rqPG7fj5o3NqcXXBEQFilDNbPtm2D7Ql3ayyg9oeVRvQdGblrF+y7/erlUFsoFZqAiBwR81RZ7EaXY/7Y0kUMP0IA8GJu3YPK/n+ag3Xw05MJz1YcNS8nl3SWztHoz6OJWNL6qnEcwGjKdpsdPT3RjaND 2rnNymep FYtuZjptpsa+1gDNVsuXrTsSHg6EpSGOpyEe70FiCAWKtj68BHvr1ex5n/HKnW0vIcwBQIrn5pN9+kbw3grQP5fyc8Q53aZhwGPZ0TWUgr/K+0+topNFQtxsh41GRIUSqJzVBb+O7nUB5upmFvN9IZWqWJMRun9bsD8xe7n8NOrfsnANAChr0hIfCXHOc04HcdXSneyyr2aTEjUphtDUdcCaA0+GwTfMSK0iL7vynbzssN+IBvzLDx6fG+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) Index: linux-vmstat-remote/include/linux/percpu-defs.h =================================================================== --- linux-vmstat-remote.orig/include/linux/percpu-defs.h +++ linux-vmstat-remote/include/linux/percpu-defs.h @@ -513,6 +513,8 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) +#define this_cpu_cmpxchg_local(pcp, oval, nval) \ + __pcpu_size_call_return2(this_cpu_cmpxchg_local_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Sun Mar 5 13:37:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A910C6FA8E for ; Sun, 5 Mar 2023 13:42:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4D076B007D; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B881A6B0081; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B7246B0080; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3358B6B007D for ; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F1D6EAABA3 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) X-FDA: 80534957580.11.69E6E61 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 51F4720006 for ; Sun, 5 Mar 2023 13:41:49 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dJ4oCxam; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=4iPuBVChDm+48S6jOuQroGvXxk7UdRxMt67FdkHO31qDpGcheBSy/F/VK2ruckBwMiovKE BrONVBOe9DxpeM3KDrnU9bkilk/9F5FHHnzrHmj7JJTttrvzQFQbqiO6cR9rLa3elgTMxb PAYRPjrFj/JugUgGbSa8X6IofL6Dcl0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dJ4oCxam; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023709; a=rsa-sha256; cv=none; b=ZGc0nLGO7QYsflVg+eUHmuX3mRVslEfEyYu+gxojOsAXyCb6aJGSIajCv9Io/QFzClGQrv B+P/XvrMnW8xaG34yUjT7cLc4yzm8OrUFCRSi9tv5ub3fh3SEysz2BLX5zgW96eXuR0eFI NEEh5SFZUQWHdxBlD+0+9Mp4TrDX6ac= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=dJ4oCxamnSSAJS5qySE2URIZktV88L3KwJ2Z5EL8F9hQDGAa7JeN7syCktRTRXqN/04AWq RT0j+jrQZQcK5/asdvRXWPHN7g67uBN/VAVCh0QiI1gSFta5lDeD1WFUTlL860t8R8mViM gdaXPs2SLLZOANXV4cdj3gzZZrSlhBg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-562-8iJu-gaUM3O6uf58ubxlWA-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: 8iJu-gaUM3O6uf58ubxlWA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D9AD7101A52E; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A647D14171C3; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 2F3B7401A0FEE; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.587741722@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:04 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Peter Xu , Marcelo Tosatti Subject: [PATCH v4 07/12] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: 51F4720006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: mtoqt1wupz1wy9wg8zar4e1pkcti8q74 X-HE-Tag: 1678023709-665016 X-HE-Meta: U2FsdGVkX1/GE4V6Nw1yzWQvO97wOj05vYWr+w7B3EVg09yIRiwnbY+tblAWVaE41xEVzeIHFfsvqZNdxhXvQuvyNvkEJkzZ3giTH7VpHeJWO9I1KmNPmeMHztabUWUW4DnPlYO15/XbwNDMFAalmsXJO2J4r65AnTGvtYf3uppIROo9NcQqd6GIy9kVpjVVJMZeUHvtiBkNMeMQ436DClJcs3nJHrTu6G7BtozXM2DsXtQhQPTSWQQqd5XDKPRA2i463s9upYXtlZip32NiPuvyuVmCxTEKcM2uzDgAZ2JX1L9xGV2LKu9+kHh+T6c8BmUlQr+4XTbO5I56A5lPH5pzQBWH5TOtSwJHQWvIX52zoS2J7vfE/dcoEScg07UJB2zV/jH8PtR5cpV6p2oq1K2bcTeoF2ZYLY3+EnEaxutPXQLJeY3+Jz9LFNA7itZDWxk6hMgxTeB1L9LnOX9J69KQy3r4JOz++SN2up6yn66AClOFkxWMZUtgGt9p+y+WFOdG1xeOdzg0WY6ahNYdz/34iDq5OP+Iwy8YPjOFskJbq9JIAkme1tpkVXnMtpu0ZdVVkiq/w/hOLFUZkiCQ+ekioq343P8BvnrzT83fkb1u4O73Vm/sapKjhWaqgrHtJnoFKgV+bPFg6Tbk3ZPpMbouaMR/RWwxLuy8EezQNR0qvzhcc2PXZe8fgb9HY6PuI0UwZPheDjf414CPY7WmTT3xIu1WKMc35J0GbbpWUn/gi+wMbmi/wsPwQGiJIY6ERDe5qBngM4k87u3iKobBnWh/Ppq5NLHH5uOrVbxmlOzhR7/sOO89SnZI1zoqAQ1cPXnrUkr6/u5VHDa5dQ2xWVtI1h7BAR0Lvh36GUmU62nBK3xMFHq/U9yGE40kry6ZxnF5NDyvLVoBTbI8pU7ptDdeBr7nsbwdO/YOcEY9lA1rKS8Ly2S9D/gu+xqw1ImTKwCgO9+0u1kwuIHG+s9 OksUrKV9 PyF2S50ONbnUoNb+DkN8wPUpPqtd5OWlb3xKURNxyqdMoVMaRIA979ayr7+lbCvjEMiESqJVvV1q6xaY7AkUaIyJkgBNdSNsUHj9nYfxtTLXPPpgGpEAPHQBnHLLAKoXmkVvHapte+yysrdka3O8zhfUVf421BLs+pdo35N4OhyOulVk5bPj9G3KXRYkCUAreP/Gn0i093e8CPHlLRBm3tF0S8k9qT7zv31s16cY8xbse5fXI/Gnf2Gz5Jw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Acked-by: Peter Xu Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -83,7 +83,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Sun Mar 5 13:37:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0888C6FA8E for ; Sun, 5 Mar 2023 13:42:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9193F6B0080; Sun, 5 Mar 2023 08:41:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 87B596B0085; Sun, 5 Mar 2023 08:41:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 631B06B0087; Sun, 5 Mar 2023 08:41:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4AF356B0080 for ; Sun, 5 Mar 2023 08:41:53 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 13653160B34 for ; Sun, 5 Mar 2023 13:41:53 +0000 (UTC) X-FDA: 80534957706.20.667B8E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 4E44820006 for ; Sun, 5 Mar 2023 13:41:51 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YsU5N2r1; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=kck/KScqtu56KKjBhC7jhwkN1m2f1dyAwfBXk4OMNDU=; b=ni5QirTVOEYiksdDIVcd4eeU36ard6/59HZytMlEp7WHMoZiggYNczKCOwnSDoHltzEWvB flrO3KGxO+XxkQjMhKaGU4hrTT4h/T9xfaDBvHSa1FRzTXqrDYIgB2cAVbSlzhqZEqYDI6 aWP6jBc7Sq0jzTTzNnVVO79LEaGQ3Uo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YsU5N2r1; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023711; a=rsa-sha256; cv=none; b=iDYfKfYIeQjfirSDLO3SLl+EBnXfAYaDiWiERQkI5wWlQEO7ZYF+tytuTdmSVtdNAOt5Fr 8wZBK2bDEQ7x5W0MbIx3ew7yXxZ68BGpqBGJNI1KXzds1ysr6PZtUgEXqmdbTVwXFsgtXX 1JLLKHhScnmrAms2sibOTr2EL4a3hG4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023710; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=kck/KScqtu56KKjBhC7jhwkN1m2f1dyAwfBXk4OMNDU=; b=YsU5N2r1giWHTXox5cx6pob5685RGAIk8gAmAQQkGhW2I/usojoh61P6X3qQK0ydUURGli 0WmmQqw+z0qS8isMg7FF+ywHYTJuYsOJ2NvqrJpp8u1Ak91yLzEEIaN6X8ASP1NP478ksS KCb9NVykIPLX2qGhgfB+xjv51q+jD2k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-83-KyT7yN_yPAKJGKes4K7Ojw-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: KyT7yN_yPAKJGKes4K7Ojw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB67F85A588; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7A4E8492C3E; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 33509401A1912; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.612408297@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:05 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 08/12] mm/vmstat: switch counter modification to cmpxchg References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Stat-Signature: t33p9wajqydrccs31d689ghx8hc5a9j9 X-Rspam-User: X-Rspamd-Queue-Id: 4E44820006 X-Rspamd-Server: rspam06 X-HE-Tag: 1678023711-162916 X-HE-Meta: U2FsdGVkX1+tTWC/D2LMLT4M9EGS8bHYRcSgyIdHIe2ys4ESyG7p2Low2xjHp08QSW72b+F7EwLHZn3AjYO8Etel2klE/7p3vLrVL0tO5WmiBUhxJ9gs/as/Bu8ntlKy6ngTuGQ2c+MK+07SeyST1vsjJ0IkXVEETO4+tb6Rvh0px7+ffg/rimy/8YHBbBpMOGNFEji18qAqwKY+ExkpOZecVvR+6DvCPxTJ7r6RtsmmryA/s5traO011vTROqKTx/DMqhnDYh5qx3yrcHKgTXrWOBGWLeyNIjKTViyIpmtqEW/Hdee/bW3nbkhVFD/74TJsNSRj2ous0TP23m+OrkE5HKLJ5lVoBuqJUaJUy40MkAV9DtEIVbk/FQSn5hQ+797FoYHES01K9iRoGCQd9BqtYA1Wx9ERerEzJPKNWGFX0i6lkQ1YzNRDykiI2hMYM+Xm8i70evOfRj4g2DxHzg5ayefTaN9GOWRC/2va7msqzcEXZ1A50VIVz/J0DPIP6OuVx2xLGsWAYGWolAdUd7DOGuJNaSxMECMJQLqqdJzzuiMa84PI8f3e1mNtNZ578Dmx6fVzDB/vVTABknJhQlQfcQgvggJTFHN1+5pMIRyv7Vwh7ROz4m/4+5GLgnOU9p5zqn/1jbV4SLqpirllxJ6kcl9oou2GyUzUJhlpebiwToC4cxO7VaA0XPCkZtyFfU+ne/37Qjf4VN1031fs7IbSX7K2RK/tW1XDSI994jv6tMydbynMmPf0gKSyFKtrtufyB/KuYRoKuneQL10Lgzu+CSHC8UU4vUuWuC/whM+cZQE8qkFXIO1thbWyhQKTOJqSPv+3Xh7qvMnMqcJoOBVdB/HrIF15uUyp0/fTH5PPGgjAAV/wY3ErmAePbXB58J/z1zClSKTB+ghgUt6YOiL3TGyt8FX7mkAyreel2Kns5zVRYkaoqHEY06bUeO9VD4ou63A8y7hRZYkhO1P J3i2ZZ/e woZJSsOm0a94FRfCN4AlGRT2Ss6Hpb+gIrDcmx8KDDg5V9p053YA1QTA2fCF0xCIwXBOwuQgl+UFk9spsRc7Jn8GRtSAlXDmLIe6TzTnryVhi7SD8ruVWuK8de7eh20yxIyDqN/WdIN1wdKkgJQ3H9S11oTe3qc5IJv+Vs64Ysx7LUTt1R6mtvqo7lJsovQ0Ex11klC7U/JXIhlwK20VwlLEWAc9NoylhwUw7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for switching vmstat shepherd to flush per-CPU counters remotely, switch the __{mod,inc,dec} functions that modify the counters to use cmpxchg. To facilitate reviewing, functions are ordered in the text file, as: __{mod,inc,dec}_{zone,node}_page_state #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state #endif This patch defines the __ versions for the CONFIG_HAVE_CMPXCHG_LOCAL case to be their non-"__" counterparts: #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state = {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state #endif To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 115.25 117 1.4% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8608,9 +8608,6 @@ static int page_alloc_cpu_dead(unsigned /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. - * - * This is only okay since the processor is dead and cannot - * race with what we are doing. */ cpu_vm_stats_fold(cpu); From patchwork Sun Mar 5 13:37:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1A92C61DA4 for ; Sun, 5 Mar 2023 13:41:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BB056B0082; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A6F96B0083; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 437576B0082; Sun, 5 Mar 2023 08:41:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0B5E26B0080 for ; Sun, 5 Mar 2023 08:41:51 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CFB71160B32 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) X-FDA: 80534957580.08.77C8A7F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 1FDA9140003 for ; Sun, 5 Mar 2023 13:41:48 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SNpZIfIW; spf=pass (imf26.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=cF55NXqs40Wyum0JJGwIV1sBIg7Uydwtt1diuU4lxVY=; b=h51/LwIQgVyOmh+95pZTDaFHTOYIAcMNifxfGPnLWQlu5jST5QEL5IN6g2EKwWIQNHseuI Rhma8hv39v7IxbH2mmhK0M7BNBujksPHEHozkkKhBJqdmDf7iQA/4AYZ/VGHZVyeXP+tpU jF3FOxQR2n4wL1KKSq8pZOCfVB/vIVc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SNpZIfIW; spf=pass (imf26.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023709; a=rsa-sha256; cv=none; b=no3TNe0r21GTBJMR1iHRXbrqdkmfLh5MUqzTsV8Ch+yZmUDq80Sg7+oSAHf9xCSYjXFQMK SCu+MJcXt0xMLnj8wl9qnjeI9emQ7WDJ/aMCbeKkKSRnP7xvwmRm9/QqQ3d2ppKxmpNwVD JfqgC7XCmvECdSiaXj64pe6gDBC9RP8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=cF55NXqs40Wyum0JJGwIV1sBIg7Uydwtt1diuU4lxVY=; b=SNpZIfIWxd2qLWNQeU0E450wZfOxOCtyt7jUJAJcq+0NJtpl3QBNS/MTKjBFPWFDKspWRg F/vQww1N+SgPLGsiZTfpFcWQjOoT4YrF6IHFwewNTKtp3LsoiVRdS+dEPhn5CutBfRBlC9 QhMMIgIFfYlhjilt1nmtzxgvtdD1guo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-p4wUfcAPNJCRL6VWbVgU_A-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: p4wUfcAPNJCRL6VWbVgU_A-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB6C885A5A3; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7B64C492B00; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 38FBB401A1935; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.638239404@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:06 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 09/12] vmstat: switch per-cpu vmstat counters to 32-bits References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1FDA9140003 X-Stat-Signature: wpw8ar1msbfepsmjbh4fx5r6fmrdjgmf X-Rspam-User: X-HE-Tag: 1678023708-923340 X-HE-Meta: U2FsdGVkX18r6A9tv9mzpLKJh0QKhLlO9+wNGPjNZH4N7COQjmpsaNN7Ng+nT00WpaOkrI92nqR/bYGvg91TSZz6SntiYfo8YhUmI6x8Phb0nyxIG2orX0hD82RrmJ6IL4Z/6rhNaPmgTEgac9v5BBQyPtpDxd5otXq5XrkqzxQORuHMQgIXBwa/Qg6WsgUv0IHu03//8s2khePUD7KFByQPDIGinID92X06EPpHE845M1gz0r6BUwcQywH2adP8XxQUbgvvsyxiEfPmYIm+L5cK+6MT9mHUT15gyT9vq5ZLX3ZLBPlqOUT6cSjay1cM11cR2YQ39PpCaNrhzdg6+KSocXxEjs50ZFYCCHf0Fxvo1ifRYsQKUIGOVYBFR6er9/Ssnq6nWuzbxhsQl3GH4zXC1qG/nCUxPUyIZs5krBjXxRzUjDBE5wGinl5w7zcCwwJAfT2PBQKIr5zVEo5pJfIK9kWXMDH6Z1rRJ9/Le3nxnNSyCh7/Z7Q2N30i4H9Ji/BYs3J4rH9X1i6Xjd6+TeGCXQ2dRr/4+LQ+xXSynYIdh3hHX9j1Uq23Iha/1mHL4CHwrE5XVAOsmIrH+uKpbfC+4S6Skf1OR4Ic06yAiQxesCKneA0EDw0+YPGNRt/wYoJog8y57bTwPe0s8rcLdW28r/j7EagDdX7qtI+ICiADVOe/3+AVtEhijtOjc9ak2So7uB81UsjaDadyfgYsSBT20hXQoRhQMD0mb9xmM4/Ctc/8REY1cC9m2vDZRZzidyPFGf8niujuq9QoCj+ka9WQPsbxfPHQQAvv040lEiQ5m9QPLQei8q/FkjN67LC10Ut20NZLRmHZzzV1NqJiypoqROi13Zk8/G9movV+xQeCMDRvXgAEUXRH0gcl98bC6dKykyWY1SckXRT8zA0lyxxn6zOG5a1v+UKJbWPLg4o1LESxS34j2RUhhVu25SLa9iw9HFWt5jsMS+8IF+Q drAUN5xS LdxbSyKpkmJwZkYV5mcCCFRiTyjtxnFuYT9rOSlj1i4Id6Tl4H0Xo3xk2LUkKTOcAUCv7mb83anM5IMGDrvteSN/HGFWbyiQYQrarJ3XZXe+nMD20z7JLvfYI9uZyd4HlVW2VftFMlNoyaIsPdVFripbOJBbVRXg3q0FUWdp1hUYtW+WZ68ORTlbq+3j5z14vS2Zcwcete3LKK2k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures only provide xchg/cmpxchg in 32/64-bit quantities. Since the next patch is about to use xchg on per-CPU vmstat counters, switch them to s32. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -686,8 +686,8 @@ struct per_cpu_pages { struct per_cpu_zonestat { #ifdef CONFIG_SMP - s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; - s8 stat_threshold; + s32 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; + s32 stat_threshold; #endif #ifdef CONFIG_NUMA /* @@ -700,8 +700,8 @@ struct per_cpu_zonestat { }; struct per_cpu_nodestat { - s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 stat_threshold; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -351,7 +351,7 @@ static inline void mod_zone_state(struct long delta, int overstep_mode) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long o, n, t, z; do { @@ -428,7 +428,7 @@ static inline void mod_node_state(struct int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; if (vmstat_item_in_bytes(item)) { @@ -525,7 +525,7 @@ void __mod_zone_page_state(struct zone * long delta) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long x; long t; @@ -556,7 +556,7 @@ void __mod_node_page_state(struct pglist long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -614,8 +614,8 @@ EXPORT_SYMBOL(__mod_node_page_state); void __inc_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -623,7 +623,7 @@ void __inc_zone_state(struct zone *zone, v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v + overstep, zone, item); __this_cpu_write(*p, -overstep); @@ -635,8 +635,8 @@ void __inc_zone_state(struct zone *zone, void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -646,7 +646,7 @@ void __inc_node_state(struct pglist_data v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -670,8 +670,8 @@ EXPORT_SYMBOL(__inc_node_page_state); void __dec_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -679,7 +679,7 @@ void __dec_zone_state(struct zone *zone, v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v - overstep, zone, item); __this_cpu_write(*p, overstep); @@ -691,8 +691,8 @@ void __dec_zone_state(struct zone *zone, void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -702,7 +702,7 @@ void __dec_node_state(struct pglist_data v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); From patchwork Sun Mar 5 13:37:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E2E8C61DA4 for ; Sun, 5 Mar 2023 13:42:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB2FB6B0081; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B19566B0080; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F2966B0081; Sun, 5 Mar 2023 08:41:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2C3006B0080 for ; Sun, 5 Mar 2023 08:41:52 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 06CBA40A76 for ; Sun, 5 Mar 2023 13:41:52 +0000 (UTC) X-FDA: 80534957664.29.EAF37BA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 4849F40012 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jFizVWPG; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=YjZzxvwbzUGJAOWGom5FEayhMjnpZ78I4D4VtdieNs8=; b=jB4HjHlgiBdXSmS/ZICw/DYiAv3k/YOe6YUKnfmKEpUSBO1nLkMiwgA4YysReKp/+e7pEB pCJ4zwIQEkvFk+EiWyHEVpL/FobHKrpk/lsmxNu7BPyKU9LYtGIumaUv8pnYBQgFjYiUhM BoVt11wwDGaBNNcYvtvEmLVnjleaJl8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jFizVWPG; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023710; a=rsa-sha256; cv=none; b=kqI+Ok1o9ap4tRdjGbz+bdSAYtAld+DjIw9/ZI7et2BzRxiv0ypqOLPGrxN04zKNk2O9Ka MBPMrT1plbqgv+eGAjoTDZMDkV6GM18wSgF2vxqkGLETbm60MwW2kyqwP/91rF3Q7hHppn IECr4KN0WHKRrMjo8gj/AHf1bIye63o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023709; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=YjZzxvwbzUGJAOWGom5FEayhMjnpZ78I4D4VtdieNs8=; b=jFizVWPGjLl8vvfZmpdf4fOaS5BoHtHMGqlis/eiRW8Y1bJ3979mTgzvG+2BdWxdxeDYeY LUfVmmobSEG9snyro/+uewPTNPoVrcPRcpNndjCRhKt+Bm4fWeOfYpEddtHiAkPRZFcuYt oMikB2ZFJDXScjdofHv6ZrPufmIOyao= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-83-LrdFy_6IOgSylGfx69LC-A-1; Sun, 05 Mar 2023 08:41:45 -0500 X-MC-Unique: LrdFy_6IOgSylGfx69LC-A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9D8C29AA39A; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7B943400F8FA; Sun, 5 Mar 2023 13:41:44 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 3EE58401A193B; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.663216906@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:07 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 10/12] mm/vmstat: use xchg in cpu_vm_stats_fold References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 4849F40012 X-Stat-Signature: zhp1xfrgh6wjuzgg417joouaatf1g43r X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678023710-184842 X-HE-Meta: U2FsdGVkX18oVMb+MlufIEit6S9aHW9LPIaWv5kbohLFfSxUU++IHa2RaZyzJeYQ5wSezpAo61aQbYKNiphTxyImhFJhau4HPmNydy44vA7G20R2kFlejTlsT5QAFVnECP3T4CdZqnprM/dcJBqackevmEVZEIbQElouGwsLF+V9xl+Wk0dG6c/0qMDUi4ZpOdm783W/9zcpRMltqBq1I3k/91XMcNBzKX8jb+KWN/1T6Lb7z/2+eNPxy2ftgHlostjwU6y/84uRdwfdYlCudIRAF0oISNyzatnWiD8b2m8z67VYwj5pr2Tl497rIvuVjdo1IiBXo4ejph9KJ19RsuiWjqPMxhirL0WUep7HboOWXwI1BNb4R/j4O91R41T5knYcs1i+KPu2PiCWhqiNOTTExroB7APrT0hWttUXLae8TvRXuFyWKODImUfuU4GqQuzrLOmCjkRW90Q6MA8Gng68vzH/QJVMceAiI1+pS0FL8GaiSD7KBvyOPy3cxjDEWlupoBNhTTXR8CkjD57G1hgPsQQSvvjAyHzLAJ1Bg9+MnVLVF+zid666N2gO6ucqSb5MDYZlN2bMmvdIv4vFwKXgyU+SfE0CeG4VI9V6X5bZrznXnLwEqZ3P7sj+kFXy+m+ABUOzIG1TFJg5y3ppd77xfCzDeebSrUNWBTsPRAAr38D/Sql9OwcWwK1d1Sce2i5aH1dPHNQF4tRSrXYWyjRXPPGCnMy48N25yps5qpf51q1DejZZaOucCSWtGPS21nyBceln9ov01aDSHJXXtdhCcKI3PVKYMIDQCSC5FRPTwouycbTrcN8q79B4IAr1JgPT5VxlF7jDa9yTIj1Asfn0I7xgkm9x+wiyjO01S3j0k3ZLd69ZcYB7ljvuR4FXNcM5+sLeuo4GCrTJVoqIVlEnarY+THl327YNCnkrk1nkDg91Sbd4PW02n78Gey0Cp/frj/pjg+hAF++eFOp F5xJeRKa B2dWdzjqDSih4Itiv7d7mG+aMfDUd0yGJFMk65yZggJEfzdRvsIDf7d5Eqw17vMgf12ooqkdlpq1w7hW4a9Fv4rT+PtxG4LGMOkTrV1k2oGTbIKGhbwvDT0Y+YHvVs7b5jSsQuOgNQJLRW+X9es2nW+bXkdfBeXLidFyp8eAyirjsPzPvkLh4Si/bQXyGc7Ej3Rk1jwKAmiz4qjlVVGyScdDGOKsDzcz0I4MzNqMjFVVTuUF1dN6pY7ixFJiZys4Kf6TV8Vh4DSNjZ0s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use xchg instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -883,7 +883,7 @@ static int refresh_cpu_vm_stats(void) } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -904,8 +904,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -915,8 +914,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + v = xchg(&pzstats->vm_numa_event[i], 0); zone_numa_event_add(v, zone, i); } } @@ -932,8 +930,7 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + v = xchg(&p->vm_node_stat_diff[i], 0); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Sun Mar 5 13:37:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E462C61DA4 for ; Sun, 5 Mar 2023 13:41:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBD956B007B; Sun, 5 Mar 2023 08:41:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E45A76B007D; Sun, 5 Mar 2023 08:41:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C99C96B007E; Sun, 5 Mar 2023 08:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8B2A26B007B for ; Sun, 5 Mar 2023 08:41:50 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6070BAABA3 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) X-FDA: 80534957580.11.A3C87A5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 91DA2140015 for ; Sun, 5 Mar 2023 13:41:48 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hJowvzyq; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=tHAQYo4aXRrl160KH7pJqqqPegg14fiAPRCJhIKS8fE=; b=eciaBXeo6dYkIZdDIr7YKFVIKpOlJba4iKmHTBa3zMm3zhKNUBCxYCX4eYvfOaJCq08ZgL jRk/UIDOqURWDNCmGY6iaO7OoQym5hMy8pgNWdPZyP9tsnheUXJkiwbjOM4sR0GiCYCA4O epsycz9xLLxNo5PPCBvBbzR6r9fgzow= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hJowvzyq; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023708; a=rsa-sha256; cv=none; b=bAsQc8hJUJrXq8Sh7pgRF+SNeYqkK1XBb5EDF/rUESI8kMJePMMb3hGag3LWgRMQ1SMp7s 1HOtubmriqKjZqV5j599VS5wljOsq+Aoj4Kvkel183I45zqvQmlvbMDfeYIYVMpMffULTh Rc/tUM7ayxXyvrcNbqPSt5c8bfhh6qQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023707; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=tHAQYo4aXRrl160KH7pJqqqPegg14fiAPRCJhIKS8fE=; b=hJowvzyqJzoEpBxeQZzvTXkgR+wNqqKAFZSsUe8bJ5Z8N8A+9NSTKktRkIAbQzbbx7VdkT PvM1VpQUyNQ9fcUDpjMUOJKIKPh4gz26p7JRG0GvDlY32U1bbDWtluFpNeIeV2vnfodkSY L28XGAm2nuHxGQFwssIDIDKI7P/EMnk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-Aba2Ptl8N9iKTrjZggZs1g-1; Sun, 05 Mar 2023 08:41:43 -0500 X-MC-Unique: Aba2Ptl8N9iKTrjZggZs1g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ECE0885A588; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A78F618EC2; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 434A7401A193C; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.687304254@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:08 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 11/12] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: x497siahc7waetxx5hwbgomooq1jynxq X-Rspamd-Queue-Id: 91DA2140015 X-HE-Tag: 1678023708-663866 X-HE-Meta: U2FsdGVkX19V39aA2zpdM9xsr2mqM35MtwXwAFaU/FEFpSTcf/wS1odlv6fX4VHe61qXZBbA30y3AR656K/j5/5IHvy2ido3xmCzDcvkQz1K6IZw4Ak+er8LN3Wr0FAB7XMwrjyyHvUJoPTYO/5HlHzkT4ScrvYfEBUOsyTE0136jTeTcHXJSv+mdp3eepBJap983UIgbzw7THIiUB32Z+PiXoKapaQDJon7ogt6p6zqm/DmnW9WJLi58sdSo6uxqF+xDQeQL5x14RIfrcdfL7SpkM28od0Ng74FnpyGdtHxfkgRTRIvw+5YK8rruXeNs8cRjC/NDh21b/g9xqbcisOca6JS8/KdXaKKNr8mAKCyQmHMEIYAxwzuuoYFbUneK3B6TbJlLF6UFhE2J2bX6r2zkJPn9XIaS8o//GUoI2eKPe0qzxoWaO4Gi5f5eOCmfv4QFhQktL0+6wPfQM2veQ9x40E63te+itfbRY4gJTOXlXhkTeMVxbsaXQGshYhrtV4HEy3Rj00qojshcGO+gaFRjAGNGoc9+ojIP1ZT6dyEJqh7EPFUcbg6+tXwDNx7i9i3hBUL4YHHaJGsJQwh+nmZssJF92qUP0AHnxWHX9otsSayPL9MRv5l1JFmePyNSGu8p85UDtMXve/oLZuI3KyBXUinpsuWmgDlZ4tKH3mYMSP45VuPEgcgQcw597lrkyUDsQGkWYl2O2dyp278yJW1yM0LoziMy0cXr7uevGFlryv5FM9kvllZ2DxqBqwxa+UIXuUNtZD+N/y3Fv+k5NO302hiHCf6pSAeHFOoZBN5j3bQWVmjto1BHFij7heuzRBHV5bizi2DC9UsNYM4mmE92tdyov3WDZv/zhV541/DXXvfNpBT99smtae4EuGSaxBGk/VD1h213WO81+YOUur0DSpEvmkBGDeer7simqF3bDyPoSbDp+Bw1PyboGLyyTCeqducXdSx8lB8yVC gFZEcBPu YIV7yiJWkjyoF/19XZjfL0BHE5G+ADUVBuIWr9l3jxdjik5rfuRq8NuhgtL5s4TQr0KIW7Rgxdqv7+wu+rDf5F6TTQiHGkHi9XJC16xrin7VBuLvag7bFU4lJwQ439gpOGHZ85Zni6lDAVc4Q+leducZRLluJdP0qKgQPkOHunX73yRejHI810V/UzzLg9czijGpBHbclLghxPoE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. This fixes the following two problems: 1. A customer provided some evidence which indicates that the idle tick was stopped; albeit, CPU-specific vmstat counters still remained populated. Thus one can only assume quiet_vmstat() was not invoked on return to the idle loop. If I understand correctly, I suspect this divergence might erroneously prevent a reclaim attempt by kswapd. If the number of zone specific free pages are below their per-cpu drift value then zone_page_state_snapshot() is used to compute a more accurate view of the aforementioned statistic. Thus any task blocked on the NUMA node specific pfmemalloc_wait queue will be unable to make significant progress via direct reclaim unless it is killed after being woken up by kswapd (see throttle_direct_reclaim()) 2. With a SCHED_FIFO task that busy loops on a given CPU, and kworker for that CPU at SCHED_OTHER priority, queuing work to sync per-vmstats will either cause that work to never execute, or stalld (i.e. stall daemon) boosts kworker priority which causes a latency violation Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2004,6 +2004,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2023,6 +2040,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Sun Mar 5 13:37:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13160143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA640C678DB for ; Sun, 5 Mar 2023 13:41:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E0C86B0075; Sun, 5 Mar 2023 08:41:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FF8A6B0078; Sun, 5 Mar 2023 08:41:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D01F6B007B; Sun, 5 Mar 2023 08:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B3A66B0075 for ; Sun, 5 Mar 2023 08:41:50 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 134921207F7 for ; Sun, 5 Mar 2023 13:41:50 +0000 (UTC) X-FDA: 80534957580.16.079F12E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 59CFAC000B for ; Sun, 5 Mar 2023 13:41:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JOCV+AHS; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678023708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=HEd4Z20qHewy4816S06cVIYXHbFUi/GbntWDJMKAj40=; b=VJS5NoCq9ngKuDsiXWdHk/7Us9CSoTdYeT0eYHyOhOCcV9RbQDQg1Y+rWWkJ+INLu61bLW rkT6/DAjcXlqNrViUHV3vZoZHp6WPnOJZ53zd8piVJDPcF0A2hrxjpfIfEox8y3vwxdm7R 7Ky7TiV63g/H+I3BgAVkZN+AxRWZPo4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JOCV+AHS; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678023708; a=rsa-sha256; cv=none; b=3xF1gPbMK3Grj3KNOUe7K3Q3GqhYAtrCepD5FRXp1txvgNzli9/MKTSEBsI1PtPYT+jqLE /+YdYQ0QK2BZw0eLTQU25ASJmhvcr5VBLL1dvOme7vTWHQfnIf4ETA0XH+SZERHD0CMeMN TTb0xQEEQEG+lpx1BwHgcxoHZLpU7Ps= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678023707; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=HEd4Z20qHewy4816S06cVIYXHbFUi/GbntWDJMKAj40=; b=JOCV+AHSLn/PbV5GAQO+G7z7GjYTf01EMv6nY+SL2+87eArxBCGtWGS3zF58lYCXeFkVlH LnmLjAPnOUZZMwcUGbBTCZUqpivGt6iAmWj5uoo0tQAtXycvvi8edlTVFnvAa8BfZgN4dE sLFdc8/PusctUtCRnrEypGUJ4LXENXE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558--pV2x-S5NLK2R9p0DNh-wg-1; Sun, 05 Mar 2023 08:41:43 -0500 X-MC-Unique: -pV2x-S5NLK2R9p0DNh-wg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EEB34800B23; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B673D401B290; Sun, 5 Mar 2023 13:41:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 48C48401A1D9F; Sun, 5 Mar 2023 10:41:23 -0300 (-03) Message-ID: <20230305134053.711640642@redhat.com> User-Agent: quilt/0.67 Date: Sun, 05 Mar 2023 10:37:09 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Marcelo Tosatti Subject: [PATCH v4 12/12] mm/vmstat: refresh stats remotely instead of via work item References: <20230305133657.255737580@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 59CFAC000B X-Stat-Signature: yd5t4kmzi791w6394dsjm3zb4imy4sha X-Rspam-User: X-HE-Tag: 1678023708-481250 X-HE-Meta: U2FsdGVkX1/j0v7L8n7r2N8w5rYbgGjAOQSSeGOgpdXc2IUvQiX3CB5gIGGvOVwFbvmfu/mjWNz6N5g7aMlk4WEpxfdzZ85hLjLyy29gNeXsGehnEdjzIqxLP/iKzImPKgp9ea3+OoizTgtzUIQFZJEgSzmoUeL0oLIf5CSBP8uQ6M/J0AgF72CiugYwVf/FSRyNk/sEJbCuW8R6V0mgFen+361bEWHA6Jg5LkNlWZzTOpcnul8ZgX7oOSYQfOc9Pzoig3rNwUdPQFL3PmYK3k+kkdC0HYulLP9ykfZmHAQQ/4Oe7/YA0FgRgX2TKjC/S12DhaUsGp3AdQfgAvU97oTnyWqAHrnOjPpiqb4brJiLe1PfM8ZZpjJGO2UGcW3hsgsL+f6KKDgsLIGkPt63wuPbVix1gN+IrclSMa9rjnJ/dh46+j1FhJKqxvYeFegTuhzZX1jxbVZsQsyuushogmecSlGnv9dhpIJvP7d+xWu2cG+qc2Ex/ZAOdQVUiv4C/awFLGBA3wm7XHfki6IrSSO4jk9doGp79SJiqYdOQOI+YPWkSkSXsM+h9Fru6dSHpphjjhhTgynJ1vpmYghGsaRehX2oiJuXC4tTI9sV04iYo/ikh4j0XEjzTOhSSz3HfgbXeONRU4Kd4yvW0gpVOR+6Y8UHZqekqRDF79t0TXn3jO+aw0n9/t0hQpaBZSa1u4T0UNuJniTpmYo9kCdI8MCs2R8uKt5xqNLhf8z5xFka2vhrzL7F+jOWiYOUa3pVwvQGlyLW9Jzj2Qe3877NI0BrEhJ/DIUTNVes38R/qy7xjSiCvpTE8yLuHHsvycsxARQbb+ZnnNGcVm81fQbrlmwrMUCIlKhQzmvsKk8JUh7Xq612Sv60/+tnsJSOqgQrq5Agc2MrvMaIPbpBAMhK8ZtK98EKxQ8VAtCRVHZCTQxiSnbkYb7rcUJEIPAuFUSoYYxDIDSfzBo5DTFYvF3 XYIKZ41z 83cgmsUtD9J6n49wwYFAZoW9puyxoqrJgO0Nak6ffh6irFNL/8audlFl28UVXg9kclL6ZpxSmRiLJ/fuS2bjI6KRQl+vacX+2VIr+A2LzzyrR0jgLoaHngz7RSbWDqk4wiTQIYQBmKttCgJHkiU6GKwZ9vFyiPBKWStm4BCVxy+etqchCwM5KGiJydqhncqHs9MLp/eDL198IBNA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1860,11 +1860,21 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS + +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1886,7 +1896,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2006,7 +2016,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2016,7 +2026,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); }