From patchwork Mon May 15 18:00:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8CE3C7EE24 for ; Mon, 15 May 2023 18:04:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74902900011; Mon, 15 May 2023 14:04:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB0A90000D; Mon, 15 May 2023 14:04:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59B9C900011; Mon, 15 May 2023 14:04:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 421B690000D for ; Mon, 15 May 2023 14:04:08 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1175EA112F for ; Mon, 15 May 2023 18:04:08 +0000 (UTC) X-FDA: 80793263376.08.584EF96 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id C63BD1A004C for ; Mon, 15 May 2023 18:03:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LPwbOa+F; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=g+P2pLjwIHGTBC73R7sGDBbFogsWx67Df0W2nS+tUE4=; b=G2JKKWNDmtLivdPEpGfq7+QlijhXFHZsYf8zk6mELyZ2k4PUcIYcyOJtQcME0F+wDODzf3 Ko9jNQMp1pjtX06X2sYaFZTNAp7FUoEv6P3OK7VoZHPvO1QxG2CDead6pSc/xntO5QtEoe erZARw+M6yAB7dtP3wUp2H9tPRUuq8M= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LPwbOa+F; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173826; a=rsa-sha256; cv=none; b=PYZCdckQgd9zhVMPFO7zn2uYMHIXUBBBBoVC4CxvcGfnjniiYraRiWLEclWEf4B3uC7rzp KR99b2ga57qNeRCKa2X0I8p+5+tSiKQTuBtEWycY7IFK9nOVncGuOInU+KJrMZmZ2TBxAA bLRXiDDRqP/c7qa8oFbJvMz49OzQdPM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=g+P2pLjwIHGTBC73R7sGDBbFogsWx67Df0W2nS+tUE4=; b=LPwbOa+FUHtn5dV0EVawj0fifNw7JpIAoitJ2UHiMyeGaeKJf7KcKgEpRrkn1+LfFLd7C0 ctaphAaz/P07I+aHRO6n58/etDGoURVIXopA6jCPNbUwbxPu9rq9bcBfUpFPNpusfH7JBk WU1RKOnEMtW16pDxu/GRr8kquCZDcYQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-3Deovc6xNXishRmnvXFshg-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: 3Deovc6xNXishRmnvXFshg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2844F87082B; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EFB9040C6EC4; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 40D304161E50E; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.442505633@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:16 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 01/13] vmstat: allow_direct_reclaim should use zone_page_state_snapshot References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C63BD1A004C X-Stat-Signature: eaear97rz6xng3xqn3q1jx5ompqdebgm X-HE-Tag: 1684173825-527701 X-HE-Meta: U2FsdGVkX19xZJwEYsefc4vAWsBngi6bqxm/IxOMl3tz65D9GLMJYtzb8M361QonabiONngFPhfhCZvnwgSV50s8SlvoLvNOCRV9jv2/nGw5ks3SkBtc05AYk7qXfnqssIRmbJ/iWivgX0mdMey+Ru7XykFPal8p2vqLTcPvvS2ZwogU5f0eJi8nC5sZXNvBS1ZbBGRwGJIfwaYGEFcGNZ8MMQf2TiGKApStsOf0S3gv+a779OUsDIcqnN2zIwr4xX4q0ZhwC0H7O4Wy2l7XLYXjvpIeTRoxbEvE/DyRkxnZHK5iFIB4IuKoHZGvV290xVXJqW7Jx23cCi86FHtXC8NlL8P3aOXEQozs3JWTlHkBR9wfNVomRbuvANNuW0Ho768MfTMyKVUIQwOtfvWHZ/+cJrg3FfWOGs2Z+17e2XMsV02N8X9H2YdDtJkPm4zEo6uRtupIWufNJe/SufymtahJ868/1u7nztAoFKdzbm3rCoD8R5ZKmWg6VTyOfSGtsuJFNIhE1vUSeGiiaSe9CeFv8sYeoL/JDaleXWU8AmPIPhaqziklLEmzBMLKSuffkd3e91SlQsJYU1QnwOrlOYiQX3UuYLV5ieoWXCrtk4Hl13exK/h4bbSyc0jeUa/TnHu+yWvHwzmjFKKoCnab9Xbn3EvVMziSPaswzR0qcar1KbOgVi+eS7U6jZtaUkli6dN5TulWk/XrTFCQa/JGxe1IlQwSxrjImoQnFGHoqK00JawdK8cWqKCRV0GijT7/uej0AYEc6N52m4gCXt8jDOtKjc8rHgyVM2eNmM3ewCRwrFGULLnDnJDOqmOzUybSJRmZS4kTIywiKS7L+umxmv+fqRBcZ825Gtzu3VpjBvnVf1rNQo2TLD1FQLqFf1j777VWqU88zFH+ROsftBX+6fD/mhl5W6vUu6oQVpOvms5kyNc6ZLESs4Due90PfC12dbC20tVCcPfmFS8AH4w fT9Mq4E/ XC5r1s/vhuVk4xeGNIyvJ+TOlCqlfzPk07wI5FlQ2c8iV+/j4JHSmjxgKN+he+XitOVo0ufnxseLJDtYm+L+wyNNnSdcY/MWZIbc1ICnZIApomwMcCf9/0QhOvwG5Fisk1wIWwpqydPL/FOPIjWkVDMjQ+Knqw9GNaHJP/yEiKyeGw7VYcwW+Oml+5Ssyi7JPxJMdWT/znRSpmgxkHlfgQvf1c8So2lVDEefYQ24RsYV75UpbQFc2W+cJ4+rB4W9Td1VZioOkkm4ZJfQuA9TcFu4pC0cI1LTkGHT1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A customer provided evidence indicating that a process was stalled in direct reclaim: - The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES]. - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty. - This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely. I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet. - In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat. Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark. Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used. allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path. Suggested-by: Michal Hocko Signed-off-by: Marcelo Tosatti Acked-by: Michal Hocko Reviewed-by: Aaron Tomlin Index: linux-vmstat-remote/mm/vmscan.c =================================================================== --- linux-vmstat-remote.orig/mm/vmscan.c +++ linux-vmstat-remote/mm/vmscan.c @@ -6886,7 +6886,7 @@ static bool allow_direct_reclaim(pg_data continue; pfmemalloc_reserve += min_wmark_pages(zone); - free_pages += zone_page_state(zone, NR_FREE_PAGES); + free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES); } /* If there are no reserves (unexpected config) then do not throttle */ From patchwork Mon May 15 18:00:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FD11C7EE25 for ; Mon, 15 May 2023 18:04:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6851900012; Mon, 15 May 2023 14:04:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF0D890000D; Mon, 15 May 2023 14:04:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96AB7900012; Mon, 15 May 2023 14:04:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 85D8490000D for ; Mon, 15 May 2023 14:04:10 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 777DF813D8 for ; Mon, 15 May 2023 18:04:09 +0000 (UTC) X-FDA: 80793263418.11.142B6E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 4F3C4140092 for ; Mon, 15 May 2023 18:03:48 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YQqwe93Q; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=418Es5nqdsliVmHbTuXzLvQQdVHfWzYdcac5+iWJU4U=; b=Q6nYAE3NfL6kWqYxUvODA4oqFXHzqY8r1rJnxKzk/Xss1c1nGBY4qBhUU+76DyszU+bIvI OChT23USdp/Ox2oHNbcr9J5FUn8RITbO2F+XxZq7PA+GApMCKW09gmJucUX2QF3oaApbug orFGrCA9gbFoVOvexyQ1nHMU1ik+DUY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173829; a=rsa-sha256; cv=none; b=rZLMuHPBOQoCqYmA9JYqwruF67ZP7KyX1CtIdgWPzw74vvH4NoKYO0fkJzJ284jcooCzSJ g/F7WGsdm0AOW4R48B9tu3RD8HdaQS7Z3lI+mDtVTdWLy8fErtkABWlaRv8rOWJtTfLq+y L/VlnnwL1Fr4yBDkaABFGD5D/fxjqNA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YQqwe93Q; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=418Es5nqdsliVmHbTuXzLvQQdVHfWzYdcac5+iWJU4U=; b=YQqwe93QNXKVLVR2pog8ka5UKRU78mtPDcz3WjMlCum1gCzJmBHvr4ZudfmYQEoI5qXBqD mQ2nBGkbJDLFPfk6ErK1zhMTaJLsVICWDYExrwx3hZdMKTuv2qn0dRtmtqOHX9LR4ubBc8 VutM1xAyE5ozn63FZYtsGkmRo613fHw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-351-yD2GRHLLNGCGNdDrFnss0w-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: yD2GRHLLNGCGNdDrFnss0w-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 27ED93C0ED4A; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F1701492B00; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 44A8A4161E523; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.467548391@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:17 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 02/13] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: 4F3C4140092 X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: g4khag43kntt7yzdnwszu774o8cmbewh X-HE-Tag: 1684173828-111940 X-HE-Meta: U2FsdGVkX18wVYi0t5XVgncbuBKGbcUYzBNDDYj8KuL3hBzNutYncTGZYS6/j9MMUsANcagtFiOE2iYTps5Tfso+8nI1FpLw9Ktn1DmSF2rlwkTQxoToSv2iQDr2jIXwlvRtiiv+cS2ajixY81f2nMTZg8yps6RU6mN352DZBAAugCGMTnmS8EB6pUqZwuWmkRRIbnwnCkOykF2jw2ctO9VCJ6e4tpYeV5HXURP0qHIAYJQexAq0aDatvBKd7npAICxBhtnwiObWQozO2dYEeUhq8DijYt8eEP7QsaiDtCZTTm2Xn8C47qUihm6Crrwk1WKrzZ/8ae+O1Qqn5U504AadUScoAhj+9muTuF7YtR8k6NmdiObt+tBq1Ujm8kyOrjR3dR8p35kpx1KvcA5mP3XrWIQoK12iGh7iheBci8EPyuLzNDQ6tJzmoYjCdDNpFYPRkSQlZDWEbRfKiZohZus/YwUwW5Ht/NhYjTwCc54Xl2u0Wk9xNZ6X9hI3RuCCVODddD2APmHb3bKundzPkdtX2EGlKVxN9XxW2GCf+OHmsqk7frWtZkMTMs0REpQgr9hTcdhA46OA74tkrSJw3i6IEP3uNuCIxY3v9VxZ7xrP9UOt+JMOd8r2CDRf/7GRRmzxPtB6TKcpEXxsC9Zfh94OtjOe9ZjLo8MDlpkm3LVoYnuOnLLhEIabBX49FKWfQ/QvKc/eNlFMIWS0LegbMwW7EawkrgcKkwwpip5uMTWBKq50e1IrbZign0im0bpDdCwIdNBEs06JpX7ffRAejCNljBXJxw5EztCa3+eKs8Cr55Z+nCDJc/lJvm+RhZnP/jIRGU623XSjHwcsasWrKtx7H4akp6IL71fP6NSoqm7OLZOEd/oMeWjDy3OWdEZrZIIAdTQZVw9bV+ngl2jS6PwfApk/gMpb7xN0KxgiW5Gb/Yo40FxGJ4sUnpRTd6f4XWinsfzidMFZCJ6929+ fwFAVr7y INNwYy4S/v2VrMjG6iQWj0JY8pQKs0kK44f2Il/TZQ03QGM86igvQEZqYVn7J6k5QPxIfeZB77S0rJX8Fv7zq3hvbi652jGmc9IJDXs38FbWkQNZ5HsCu59shNTJjtDgFdDKJWPI15PXpLB5Ruj/jByoznvnX9t7f4qxlFvwMfsbMsgv9gYWCnsx8m2B0efPdJH/HDzDnaNGxjbBh45MKT6moy7DpRSE1yxpPQ88eFUnW7VQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Mon May 15 18:00:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD236C77B75 for ; Mon, 15 May 2023 18:03:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEE9E900009; Mon, 15 May 2023 14:03:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9F2E900006; Mon, 15 May 2023 14:03:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6636900009; Mon, 15 May 2023 14:03:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A3A28900006 for ; Mon, 15 May 2023 14:03:51 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7B61F1C8CB6 for ; Mon, 15 May 2023 18:03:51 +0000 (UTC) X-FDA: 80793262662.05.DB22321 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 7289FC000C for ; Mon, 15 May 2023 18:03:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="PPx6R/Lb"; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=6QfpHdKEKCW5B9XIpfLylgQS/cYcg8nRN+BFpTs113M=; b=vFVOE8ayU1mDeVtyRPF4B3ol5Q7A33ktnzDfnnNQrNTlyef2pqzfKL2qK5gXKRkD7Uktw3 8IGfATwY0Io4+l9SyIBHJCNttEGw6PqhuDrEvyxgD096sxeIQSW7sig87uKejQZt6f+IN+ LXqJFfG/OgOlaIRdjKqEUMfH5rg+bx4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173829; a=rsa-sha256; cv=none; b=RQaROmpnIrhNqboPRAkzJmG9Y1NNbHaklbLr43JZrTvhc2y7QGN2sSHGll5NymEIiSylN9 GBppJagkVtc09vBYgd4RvHaHLNwWH8AXFFkkpYLPpH6iWBXRW8wZyPj3n+IIIBxpM5s65F vXIfRd+0YCEJUl4NRXDa1LMtPU5poEo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="PPx6R/Lb"; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=6QfpHdKEKCW5B9XIpfLylgQS/cYcg8nRN+BFpTs113M=; b=PPx6R/LbCGgDu9GbUiPFFvvnXBM9HNG3QL9J+snzXoRnbFRFuM4TrmJibHNGlT4PxrRZ1w dbTwgoPppIDYmWHfdSAlDW26T5/wXu1TCvnhGrzRz5w6wNG+nRnBLlVl5DZXpWkNGA4BOd ykby/Uav8t9Yb9IPL3IZqPWhbEWUwSg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-222-4rDDS1EbMJedd7bEsqx9QA-1; Mon, 15 May 2023 14:03:44 -0400 X-MC-Unique: 4rDDS1EbMJedd7bEsqx9QA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2A8703C0ED50; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EF5352026D16; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 48EDC4161E525; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.491271416@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:18 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 03/13] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 7289FC000C X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: ycegfkqm9xod55gfrbtgqtom45rmfguk X-HE-Tag: 1684173829-707679 X-HE-Meta: U2FsdGVkX1+a8aYyuGV0Msmx6mSQu7CU9CstfzGoDBhvV7sTrU1aQY2NkjdHIgCseogV0bZ3XBs+va9cmkCRwYieOpQU3DfxTFSjP1UelNA/sIRx4BJcmwnVEolsaRw0mQnSZGo2TOBdvalbWBEBdhW5sOi1t5OkID2euhSOVS24r5d3+AqiWhrSFy+k5G+CmGCVxttNjujbFrK2ar48b9Ogo6LH0eJMXzR1U9irhkaKDYVe8ARtZBNCMs9p184ALdCwr9RkrJrnI72BVfdIBXLXpqvknihVvsaeEb00kCnAFv5lm8XJyVqzVMFTm9+mjO9VFDevuNaiXsg50ntS9d02JPYQxe0sLSy42A7JuIW/HO6apHh8PRlqS4eHIEsoLDP8VOxGZNOLe2v2cJ02xsfas6HdMcqKJoFphE5Vc7TWGHtFmL2b7/asY1YX8tg+2Gr3r7MQW9zasm7Bg9alGMxomNEB2cejaLjPx+pXztuE56WalxNkKZkoqJGOVkaOj1zzTKkyZ7cCzP7qHqhoJXe3VAma7IzIUoIalzpAIBlP4e1d0s+XrYBykoidEYPBglCKcFQrIsLfa0QMVSLEor7cYzxwZwCb85vYYfqYjKTLjFKqGV/QGWAO5iz56AAyvp0tLuU2ZpTzzmPYB6esCXjaTS+msTWRmjClQUvwtifTRkZE5+WObjmWEQWYT1SwLaK3yQX6E/ulD8hMAO49dRZBFxEgUckIS7k3r8PD08z78aZOTarbprVjNhYu3HFkKtH4aJU4aHl2RIN1VasuwRyxLwe3qvX+A1vupx8zdrUzyfejZKPlJB1mFmZzJoRK1cyqpMFuHBzebBImVF5m7c3VLXS1O6vBl5ymmhtNNJjHVkkyHe0HOWo049gkK43A9OthKuT4pWYcnMBTPQHUgSDR0eo+kbgmZihytos5aWMkjDWrQHDuAFXM2jgLWADpLrz+6JQRUL9Gak4AxG6 okahgcxn kBqvDaoVkgbNV7Q7iy3Ay1lrHL5k6A5jKPk+Efaj2vE1RN8MSQHVJ07H/W0OvqW9YpAp3E1SFeMoM8Z6E9DzzTcrJXcJMVAzCujtNKRR2iEeaiTLBZb5O2EAhg7H73p9yKnyrpTYygRkesYMl4rSX2IMgRXNfgRc7k5cjAWDjtxSVQcb2+gZQBI5ImfcZ3Z7XT0gsyOlt8vpCHL9muQrSFT1eZJSFY64HZNe24se2tpCqNuQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Mon May 15 18:00:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D0EC77B75 for ; Mon, 15 May 2023 18:04:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 243E090000E; Mon, 15 May 2023 14:03:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17F6A90000D; Mon, 15 May 2023 14:03:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE85F90000E; Mon, 15 May 2023 14:03:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D788B90000A for ; Mon, 15 May 2023 14:03:54 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7D06F1A13D8 for ; Mon, 15 May 2023 18:03:54 +0000 (UTC) X-FDA: 80793262788.05.09A5A32 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 5BE2B40036 for ; Mon, 15 May 2023 18:03:48 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KW7Wzmka; spf=pass (imf01.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ZYdI/NkwLfDGmXJs7KnOK9vem40tm0vaN9MiHTPCcnM=; b=Uxnm8raaIkMtOT22oJOkvo/icjBLqbhLRSkVaTzJC3YBe0wFopc6jHkmcfZojfia6QVTGt WMQ/fE6uwz+JikoBoxjjMJ2rTBQ6X4kEXqFI0f7q6fHH1AoG1VBRY8DvShtJNZ2lENBiGE N8yq0iguUB+JG2c7Bkc7jTuSlonOlnM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KW7Wzmka; spf=pass (imf01.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173828; a=rsa-sha256; cv=none; b=isV9rwEDrgahxUI5mEjq88dVMngDRrXVRx0W7xgoAwWUbs2qOYa6+oBkF8ombcFBkXr7rW /AnSk0ObCouh6uUU1AxMCxLgIlIZK30wD70pDipHZjNFgVp4z3U/YEvRBxTpF2ulEldl7N J1a14l4tjFHXCvOjgnQcUHjdG7NDAyw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ZYdI/NkwLfDGmXJs7KnOK9vem40tm0vaN9MiHTPCcnM=; b=KW7WzmkakLnsP7kmyU8q2PuZbyHaVpkwb697741UpWch0mkG/ruP7stxRss8SeCm4Nl4Fj ycnD+imBQKP8otBOKW5+ljWh77NgVkeN3WSZOy8YETyEZMK5ojH9qoELwMQusn6zzHA5GH EpgpGDbG9z+20N+o3nZ8QwS6+lrjTnI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-lkqpmNCLPSK5Vqgm07jqoA-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: lkqpmNCLPSK5Vqgm07jqoA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 258BF382C96A; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EF61A14152F6; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 4D38A4161E526; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.516507215@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:19 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 04/13] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5BE2B40036 X-Stat-Signature: fr5enpmmbr3u5gsagt43rhjfdjb8qsa6 X-HE-Tag: 1684173828-445526 X-HE-Meta: U2FsdGVkX1/DoJsDHvhOy4opqWl187xp+ks0khUzIoGkEWPpsdzoLwTrfXSBNhPEmGf58i7tGlDo7v0/CLttZvfZTLjmo06zbFVunxW90G+zBsBEk0BabiEVsvYqfyakW7phlKO8F2zJ5PB9vbBMbqHFEYmu8UEwd0XK5HRqaItnHVrglVKnwbeseI13rRoduGMLjjA0QwB20eSUBnyDAmqehNuQe2hj4ZC/SZZhm7RNGqv2bgMMZ9dVNnQGk+IyJc5H/nDvSk0uZ3q4lqxK8G732/+vAiXL9HxZDuczaWovZYwUry07qKaphVQsUh/E44Fvq7t93pJbmHFJusmE75P1TFMtMJGxMbaV4kGqFcDV9XJze4Bwbyyn/iUBelqxCkmWEq2Ghwpthw08Lm7lKnpZv5IpQsW0dSBnmzAFIKP03oV+eM6/1XoE0chVcFjKuQOFkzqEA00gAuKke2yuPetAZTxnqCqppjkYz8YbRZ/EKGgJ/5cdSN4YtcIP6KWx0X4zTjxSRGa7Rn3RZRsOqdNfZkR1hDxWB+cr09mkpPSCcPJigUnlyW3fcOjkqQvhR+LSYEytV7y7qYUYV5H8FC1x4BYGRfnAfCTGV97zUd4QzA77f/QxrmebQN2m0QZKSWYPJVzOz/PiTciEVz1iY+l4EDceBtLtRSNwL+hw9Kn8tQuMVwCIZ6BnF4sr923mI7YCbuk46SbrddIXOYtPkhCHnf3JNXIy1/rvYac671UMZ3gUSJu3bBPMjdtRHSPfkTkOk9QQhLC85tbTCsVtU+y49DfvaV01HFZPvkUNONIQy6HUaQqB8nTOqdh/cJ2P6VWfReZOVDGm0TTtH5IPhwas5p9a279h5H76WrCB9s+wtuFY346MBKHZhfKzUmZNAPR0bGzZJQ9vOOzVe/P0AkyBP1xSHyRJKG000K91Owjao2q7B8A5JI6Fd1IS95YQkpxmmz7AYy2Hq/5eKP/ qqnEtreW Rwpi8AkVtX4zzgUbtpIYRqih/a2QDyyghs0nvAvre7vhx4ibxMGdUJ6sQjFiV8gwWOV+FH6zsnlrksnRdgfqBisK9XZsNau6FTrZ+TFOP+/Fsa4L50buZkfQGmZAwBemwou12BartQM3Byrw9TtJFlfra2jJeGcWtXZYDXM3rt02FailJ+HyIkYehXhUJC/HrkiYGUyJFBQT4ZvbndMSTBcocw12qhpEuNVa3dygh4dc/AcU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Mon May 15 18:00:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22E7DC7EE25 for ; Mon, 15 May 2023 18:03:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B40FC900005; Mon, 15 May 2023 14:03:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AFC04900002; Mon, 15 May 2023 14:03:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B8ED900005; Mon, 15 May 2023 14:03:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8D8DD900002 for ; Mon, 15 May 2023 14:03:49 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 588A9AF50B for ; Mon, 15 May 2023 18:03:49 +0000 (UTC) X-FDA: 80793262578.17.986AE87 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 49A0340007 for ; Mon, 15 May 2023 18:03:46 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g3vAmX6c; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173827; a=rsa-sha256; cv=none; b=VIdK7gSqerS2loEcxLXJL1xVIo8Vnxobb/uyvt19GePuUiYWjGsGwYSuAMZFqbPnQi2CM2 wWGSDMeeip8sSDwRFg4VA0ZpDMT/+aO2erZCydHfZBazfQixKZjD0rYwIJHbIBGve6btKa /9JaYF7G8sWDE0gu+5W+CpO3DWfmYFw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g3vAmX6c; spf=pass (imf07.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=F5ZFnFWd0bfXvbTVZzhDKmwHy0T6x+spMFYT61y9M6E=; b=kHZdlWls0X/VuiBzNRQvhVNo7fmoPnh6TlGPUeUtFEKgkxp9RTIGIBlM2fP/434/VjPvXV OurtJFNwF09EEdFbsVs4G4W357GngW1CJHglZIi5xcwx9Ff7fsCyIaTCMA8cANFEgUbRkn c+uZgbSC6lwObQXy+Fwsd2PsII//Bpg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=F5ZFnFWd0bfXvbTVZzhDKmwHy0T6x+spMFYT61y9M6E=; b=g3vAmX6c3AFB9Rrmyat6tJSIckmT2gusprlx0AqtVuoDmaPBoI00F3AYPYM/EOVSpYZuAc wlPVA2L+FypOVn6UPH+LjbB3BNIWXOy2lSi5/p0kBT4tJ2fi4UEVmr7ce0R8GnHWu/wxPH oGea7ik6qAvW7msFR43Da6Fp8I0C2h0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-s_j6uHycNbCQd9e2JrYNNA-1; Mon, 15 May 2023 14:03:42 -0400 X-MC-Unique: s_j6uHycNbCQd9e2JrYNNA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 618F63C0ED41; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 04617492B00; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 54F094161E527; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.541141182@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:20 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 05/13] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 49A0340007 X-Stat-Signature: 4j8w8brhhonifjx4bbpu8bqeurbx8dfd X-HE-Tag: 1684173826-322350 X-HE-Meta: U2FsdGVkX18nTj4nab7COaP7EaO/GqTz8UyRxdcmY2mq+DA5+9KNDtuA8ltUrl9N4z/zaORx01Y0GapjqSVUw9ud5q4OtfSp2oUCyyxObSyFfLxshuYqdQXMQ/qqYL5zEaUZvZ2IvDHP3FPu5EQLN3rJaJHOXnwmCFsePD5wt6sXOkNaoOyWFUffRrOqGACXYNzEb9WiQjsYK3O21kTrmlNhs07NzubVEUNj6ytYEOjFYH6RHMjp/K+Wlq1vq+bB5cFGkxW7CJgTXmvtRglSt95RLSFJAXjT4ulK43q8OU7fEglKnUfYglr5VlJed9++kQdFWI5bUYPOJHJQ5ozXIpy50LVsQwA+WwjIZt5kLzqpCqBwDDgfPhJ6hZZllL87d8/WtFww0xb+CVRBABRB49tsSUADHx1zyPI/PdT467q0/rCbOMW0f1gzudT8+VXVdIQ+WfIxcim6XhtYLko//fScGnOrvM+g972IjktLKs+iE+zOF5gwhzl36PQSDaUL2fhwRQBAnvyh0P6VFVkupP5GsyTlX5z7NdObh+LPufVsbJFWmHGsc3kH19sibOuSFzLolv4yfdc8QVYfpatN465M99ErFmz+gdWW5yngruoPpoDykHuQpRA6EOous24ScqJUIVal/p63qe36+3Cwti/hNRld9g2UN70cH3V58MY5jT9cCxD+eiz35SzVKtcMMaK713ZkqC0mvHrvw4TLQ2OClfTg+PJMJI+lHuv4Bl3XJiafXutV52dwdqblfVzCzrSqXic3wBAbmwvQ0oIruGS8O0guv23g+2GFFxT5zwzhhfU9Buj6RCFC8jQwIUY6CWldOJYQdpBkfYe3eoOHV+h1gaCX/PliLySMjJZqJ39xfuJZqP3KfGjCxzdoE/GMS403Ke/fPYghpu3DFL5/hISrzOTLEcywZzejg33mC/4g27aftVNXZPRqJAYBw2Y/+qPFw/anes9KcgS0lZO KXNZj0jS s9HJSzDRag11oGh3wHg8sGloKURgSHvdkqb50Hg9B6QmPJi9MC8x25uU5MUg0JA600J3Szf7ma9qyhINyR9vJaQa+Sm3Ny+uTw2GCGeugoVVCLXXQ/jjiPqtPs0gmiDM/nstYes5DU0/DTnWUK229pKeSXP9QQ/hJ5DSqsp729smjCiUr8qAxNIzN7XAv7ugJpY5+AzroHp2mwbS7BJzVK7c8/57350wXz0fyc0cTIUYH4PI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Mon May 15 18:00:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2B46C77B75 for ; Mon, 15 May 2023 18:03:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C71BF900002; Mon, 15 May 2023 14:03:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1B43900008; Mon, 15 May 2023 14:03:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2EA9900006; Mon, 15 May 2023 14:03:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 725E4900006 for ; Mon, 15 May 2023 14:03:50 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 418EE141455 for ; Mon, 15 May 2023 18:03:50 +0000 (UTC) X-FDA: 80793262620.29.1E9B188 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 20D8A1C0013 for ; Mon, 15 May 2023 18:03:46 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YSy5ojfd; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=kXDHBNZWD1x4YyqOA6ZVeranXWbQxiEjKgF55Tb+rLk=; b=a94r4AENExJkRuzy7tUNjpJxiBaQD3FYUW7dk6FaKb4fOvVb42bdgHawcPayUU+UkO6DCY wLKmok+uaPkPTuvyYS5uyswA0TGkcl38L/MQXWCdHcl13el4lg9Rh4cllt2xbCadKFTPuP Vyh9B3QmINWvqFKMNeDzhMPzNphy9Tg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173827; a=rsa-sha256; cv=none; b=b1lFOGb59UVJRFaQmcQgH/03ujlVNkvzhn64p/vu56zA4ucihrNdmC0PsyS2+CttJGvbA6 Amd4rS78LWDKxujxb1ewz9AE0r19keCBRH9oQoSdwft1ksbrKMyXC6UApy8EvkvWlMB+rv Qx6KFXMMmWjYOPY5hxHPwzojjUshA4I= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YSy5ojfd; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=kXDHBNZWD1x4YyqOA6ZVeranXWbQxiEjKgF55Tb+rLk=; b=YSy5ojfdbe0fB/AJ5P9zYgd/ZPN6jPCBYYBDfP+2b/OF0q15pazTEugCjyKuPj3zgOPAxY gpErAvzxbUctc+5CW7lbd2UUsE9de18p0gZJeU1t6LNFDSIHMXA3A+vM/2JIuDRH+1pgiR BAcaGVAybSCGeJgGjLLU8YwwfBueqcY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-oDr3EEaKO8-jBZS-XI6F8g-1; Mon, 15 May 2023 14:03:41 -0400 X-MC-Unique: oDr3EEaKO8-jBZS-XI6F8g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 41F9586C60B; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 013F840C2063; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 5E4704161E528; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.566700916@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:21 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 06/13] add this_cpu_cmpxchg_local and asm-generic definitions References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 20D8A1C0013 X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: tcutqxkqnfisnt8qy85wqiidbirg7mrh X-HE-Tag: 1684173826-554795 X-HE-Meta: U2FsdGVkX1/AZ7URmQ0oE8ML/X6IE3XjXInI4J5B8oytpCcqOkinF2uv1XGIQmQpLUhtKxJS0QLV3oPYxMcofgWcsKqZza/nMd2WU3+SEQ1xiW5WpfzKBslZDcNRPVqK7VAJjbD0dyBuG/UyFUCf0+dQExMxdim744HGc27sdr61p94Cs9wCdsJq8dSB1XZXt6OTD/hb7hSnfBrQSpeNZfBckmjw8LzIVNlVVz7Lc0X7ph/8pn/aJAafDDHna+GcqoWVJE3sGSM2VBO5LPHYLiycPfDkHMvjsZKAG+Tw6siiDpUjzhzdBCH8O2zpvPeMcrYSq8ylrumsx5lcRbgNsGEMEMQgNWFh1nWoxEYknkX1IwIR9G+TDXiE6Jq4DLIrNShwjTnc/R8oS5VOusyMdXctNX4Mo0tMXjd9lcPu1k+XGwv0mJiyzzFUdNOePqJeYDWu0iFlk3cOKYXFteC/sam4aPKw59LuJLABa2LX56GoxZMM5w/buc0ytrZs+cdm7xBopAw0hSDLpfGCDYcn9mjQkYUYwHir4bSweW17y5h8I9Vw2bWlhMuo171zcc/Nzlm4rFrAdOJ3GdcDQHHbWDJ9niNQ4nhyanUE3a6uSDEFLvKTT4PyLjenFNcNA48DZNQFsGtCl3stQPruyrKW1X6RJ+Q20KwTvnbIcewc0riOPfLznPnx+kDLTdzOd8ARn5S87hvqm5qNDOrQVmYSCBJuCw9KB//4+PRX7SXqbPItahaNNrgGVrjv28N3aRA3m4E5FY3XRu52mf4uSszTT9iaRt4bQG2yT19RPvwHwUZKmKi/w/5GTY2/pxmDJ23/10o04RgrnQ9MpXLJ38f6N0Z9x7Mua3OqUpGjhrboBKc2G7rMEmbVhbQVCSiMx3NnAtNE/uipswB+LaRG04ZkGDCnGVTeK8/qaaGE3QA/LxM0cW7/2BzHfd5bI2zOeQO95pGZCIUFBL+slah7fKu CcgTWyGY 2q6u180tmqANJE23NGPECXelhRhJa7cZUy8uD85NSKfkOsJ908941tn6rhEMNRJ7SlGamU0UB0J/x/A/jSSNjgGlyFdIa8YQZcbLDAvtgtz96fGr2gOBLXqCJlFPX19SlYhofYH1yNLCCk+Dgjkkf91QqIStWzNwo60T8A3v7eqhBzIS/NGAm7OtFIYbVpnDbBpinulBoS2ytoxbSc63dMXvuH6eijmvSyY9+tXg4XbeJ1sE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) Index: linux-vmstat-remote/include/linux/percpu-defs.h =================================================================== --- linux-vmstat-remote.orig/include/linux/percpu-defs.h +++ linux-vmstat-remote/include/linux/percpu-defs.h @@ -513,6 +513,8 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) +#define this_cpu_cmpxchg_local(pcp, oval, nval) \ + __pcpu_size_call_return2(this_cpu_cmpxchg_local_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Mon May 15 18:00:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CF8EC7EE24 for ; Mon, 15 May 2023 18:04:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5054790000A; Mon, 15 May 2023 14:03:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EEC990000F; Mon, 15 May 2023 14:03:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0427990000A; Mon, 15 May 2023 14:03:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D90C790000D for ; Mon, 15 May 2023 14:03:54 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AFC82121406 for ; Mon, 15 May 2023 18:03:54 +0000 (UTC) X-FDA: 80793262788.22.BB432AD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 4100210007E for ; Mon, 15 May 2023 18:03:46 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f8Orl0+r; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173826; a=rsa-sha256; cv=none; b=O32c8koWd347rRcNPhpsisUD2z81F1Jj5Y5HY9xH7E/LwbSHSM5GQAIZumIVg2hT1nnckM omVf6MgmZjLtIpfUnUBsSPSo2FKOLCFbnWkxwcFVuXw8BBjGv0ujhqWwdcyb3PZ4eSvJRW koxKsN9y1fGRnnWmlAZdeBO5YCWjSTQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f8Orl0+r; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=BJ6i2oDSiY3YdcuHRj+4qVRIpmZay1wZiVJ33jyxj48=; b=pgfBeOjltkXodJH2FLWsZvVBqV+ZuhG/SfUPdSIjjZuHzcTDuupzK7fTRwzXVa19wZ9U9C fmVUTE7Zv3JZLjKhc7cJA163oPqCG9U6CVJbiIv9+VMo/fmTUxWaogmjaiTzEp8Rnla4l2 WxxDImaU7Ob/aoKWWhyOHck5RiuwERI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=BJ6i2oDSiY3YdcuHRj+4qVRIpmZay1wZiVJ33jyxj48=; b=f8Orl0+rFcesGu6vUIhvHbqaBgUgMuKnBi+9MLYFNL/hFVVHPvK2EKZHH3DbggEMghFyAQ QpSi+C0pJJsN6fDoCa2UMhiXPcOd27AMCJTowd8ImykHPa9gmgYCpNfsFJFTtfy8KvWGWm hGSdsQSgFEQD27wPR/HYOO7W/WMN8Nw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-HcyxXZ_fOKim-3NV21rk8g-1; Mon, 15 May 2023 14:03:42 -0400 X-MC-Unique: HcyxXZ_fOKim-3NV21rk8g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 64E8A382C96A; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2E3102026D25; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 65C8A4161E529; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.591710284@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:22 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Peter Xu , Marcelo Tosatti Subject: [PATCH v8 07/13] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Stat-Signature: juyn9sqsqmze5m16kjwkqh5r17i3aoxc X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4100210007E X-HE-Tag: 1684173826-710406 X-HE-Meta: U2FsdGVkX1+Yds0TzK87ktFz6CaMdaVy1I4UUcWwwt43wmgOtILS/nHHpId/PlBwmHIte7rDSv5hwEUN4YGm9qu3wGt7ZZ0tJIIpSHnTH9zozIXqvbuS0EZaSHXM46f6PJiDCgqDkZ63ETGUcgxmy/+TMMxz7bQussQSQgQslpuU7kanbPDbdvQFJoJSXBoR9sJ3a0btAQIr4tNle2W0nTZMgSyUBdmbt6kMDPlqtxDtEjnVDve4aIRCJItrN3y+ZI5ocaXU3nNvMJK7rrTyv3M43fSnJQjJgfP1kgw4vYkigzF5sbsDSRKnWfo1LtgQ4/LT1Fv3GnChAUdkslgI90ZPkWcbVSaW8h4kK/6rvNpD/YQnuX2orZksJp4Iq/Q8gtQrusHttdjZF07J2/kcX8n7tFgOCHVWtouCwlN1jVmVasFWjlUMjpLAXv5fpwBqnF0sJlRZRzZXttcSB7Lbyv1HzWRiyHkUCwhqIPJXq8QLVVIXFZhHaSORfwiSWUsLx7ytgskByu8UtAQlOphWE3nKh0te0kO6k+9eiKE5GJiIcgCZcEYl35PA69jHJEj6d2WGNb8cf5lkqDPbhzyhCeZrrWWz6KKAps4+5LxmuH+zj3UfJA/3rzyglaIX7QWj5i/C66PrbUjQn3BW/hNbSAV4oFIK/lq2XUZToCb6M9kfzmtFualUho75XFurEODl/mZs5Oikot1D6zEBBa5qfE64tgbxeOjAmgNrGKGszDHhDDNGKCzFGWCMvHAGI1Gnw0pBCbbvitab3VIK5CPcJ7jXnUujYtrnXx8mOMv59hXoxTaTBrhTH6LACZD7WxteXFBVSC+/5jhAhj5zX1LveYu1xRU+WA4M/MjOrN53AcStb1CPx+iVwkhvrJ5ujmg4e5iqwFjOPNOvqiQNqp8E05yePfg1rbMCDzc/qSUQ6VTbmVtOOz3Cd5f+cwsTmnWopbWs0GKDc6OcjmcDD/F JDWFybvP 8ZJFz4VMiJFf4pzu+eQCxGxokiuSwyeCXP5PwetiX7QEPrkvyPm/5rmqqTynr0xLZ2uxD6jA0Q+w9THgSFhvbuKkxMRPr+pdUMwUs6gMmWXPfnYU0UyHAXil99SevKQcUR8IBlzDneuN6UJyrsHVt8TR5AdrASaY2/7ATbK6dVS5WWNWMbLZv75AOeTPhaJZUGrdWdxueIFK4m1tF65X9zTbSOIZjaN0R68wnFL2KUc4JqFs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Acked-by: Peter Xu Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -205,7 +205,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -83,7 +83,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Mon May 15 18:00:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A90C4C7EE24 for ; Mon, 15 May 2023 18:04:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87AA2900006; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 812CE90000C; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57335900006; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3F41C90000A for ; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0985E1613FF for ; Mon, 15 May 2023 18:03:52 +0000 (UTC) X-FDA: 80793262704.21.0A1CFFD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 5259B40013 for ; Mon, 15 May 2023 18:03:46 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hZx4pEfJ; spf=pass (imf01.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=yXMqTzjpRwgaI/E9PhdUEcRLH1Fh3WNfw2d4qEVmv+k=; b=iRthyzmQVhUEZy4MAMg42r5AkWl3GDIlfxWptFxRGu4K5NVAizoX79gkTUc1NDveon/LSN W080nWAJkDvvfXvGyAJipgoJTG7RKMsQQ3mHPuyP4D9Zs+LrGnzznadGwLyd22IZFcQRoJ EnPFldKts7uEW8VPbfA09vbJxFFQ8e0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hZx4pEfJ; spf=pass (imf01.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173826; a=rsa-sha256; cv=none; b=eCCpudc6sJ2y32uvBFaPAgrP3AoIr4PnxDizv/CnNIGOHUHrgS1vVAf9NGl1mqZthF1Xfo 1jpH0EE4qBmFkMsYD76UbFp06zbNGIIrH3H9t0XO9RTJbQGaFa66QfkmouMWvjasKZ8Lzo vV571oepAgWqZj2Eql6OIRRdg8VA+ms= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=yXMqTzjpRwgaI/E9PhdUEcRLH1Fh3WNfw2d4qEVmv+k=; b=hZx4pEfJXosw1etoJhSEDn7l+IJRppOowSmzPpcrRsbItxyWyvxyb8A1J+CgdXdvTi1fj+ UDnLtHFiZr8KYvVo+RwlPN2yr9fupVMGcT1GZ6sme2CsXPP2dAhRW1YEjxMGqtEu8PQMYm dOEUKcTEFN/+MnTMsqzozJ3QtNuPLQA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-653-YaqFy-ZENrSao26e_j8GBw-1; Mon, 15 May 2023 14:03:42 -0400 X-MC-Unique: YaqFy-ZENrSao26e_j8GBw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 68332101A553; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 096C914152F6; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6C1034161E52B; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.617500477@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:23 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 08/13] mm/vmstat: switch counter modification to cmpxchg References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Stat-Signature: 8dtpd6zsb7bapo6fbf35bwrz719ie748 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5259B40013 X-Rspam-User: X-HE-Tag: 1684173826-531191 X-HE-Meta: U2FsdGVkX18m4VmaMDPeUnc3MFW3VZR15lcrok98adM3RzEX6lFWkymsb0cVsz8MFjBxoOUYCK8p4H7OuoFeayzgg5RhuEzRBwVkZNOj+odQlEqYyCaAXxdjOqG0xV7HCMBDDnflJzNXXGSJGI8WZ+ZzOoZMrQfn/8YvtnkAKOzIZFNTVMP/dx5Rf1M2nDdQttjlaKrYFFF3SkbvgkFQEV7yfUoGqS+ZTFxmffnIXBt95rk1/nosxHQSrAVgkZYaP180fCnES3Ha0qkv347eaPrSAmA8P0NgYGB6rgy8aqdE9xqGeu62pfkLcCO+VScaPRof7fwvzs9Mmaz0D+crCPQYvz58wi4L2mYy+KrPX02U+NGAUl06+iQlJ54DmwQgcsMXTXcsP9Xv9+NwWmUBKsGLLYY5aGVWyGYpCg6xFi8FDGS2+L6cUtsO2fzlrul28tBZ0iFRtjYaaXfae/cSISqbzt5lbkcC4UiDl86KLldjf8TfMWimmN5QJFPX3gJslYwShb+ILGbAZ9gWfKqUkuvlOct2Puq3CIOuVZkvTK5OhipN6OBTKGTya/nqICXtOS3cDzR/t4q+DEFfqtudHMP5U7SUpbrsTcfkST1p5i2jCI19HPp2OKvPrPMZ01aWGnReuTQwi9NcO+Lspsz2WiJDNy+sRzC9O0kJg4IbozR0tgdr6UFSBWuje7gX8VjPUlebUUYvlaYuCMlK6bWvarV5pLwKj6B6vRM0IMkqyBFjmbLvwQGFOo169dW2wfLOjBBV7sjzQpgAefvH7BVUcyYW2GRL7ES5ARwePkUSb0dyxFXAu0QQ5y/dSLyjnISoKS5W2OJ7tPwmMz6HMDlH5e7g8KzBzLcPtg603WqAq18N4Zf8K1EG5wCmJx+a4URhio8aVn17hPC0qPLpRe9ejcppMLr0zEolCa7xcI7Lf6h4e/th7UDcxOXMmVDNFTLfuui0bqNCLohZUvU76cj BAt01LXa RZMOVu3ddPB/SZOSK/s0wmzItMVJr9knBxavtJxRElTtBxAAWldSoxo+Sf9uPdHtqF1RcPQ5ca5ht3BvM1YtyKOQZRtjJs4sncS0X9edDnk48/Cm+3UmYqszTlcb8OghR+a/WpUDYEMR/kue3ryQKFHw92DbTSGNVVu5nI2HqBlmpxqRgRdL3B8WVVjIQqk7+Rk9/BeVdGewDhjU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for switching vmstat shepherd to flush per-CPU counters remotely, switch the __{mod,inc,dec} functions that modify the counters to use cmpxchg. To facilitate reviewing, functions are ordered in the text file, as: __{mod,inc,dec}_{zone,node}_page_state #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state #endif This patch defines the __ versions for the CONFIG_HAVE_CMPXCHG_LOCAL case to be their non-"__" counterparts: #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state = {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state #endif To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 115.25 117 1.4% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -6249,9 +6249,6 @@ static int page_alloc_cpu_dead(unsigned /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. - * - * This is only okay since the processor is dead and cannot - * race with what we are doing. */ cpu_vm_stats_fold(cpu); From patchwork Mon May 15 18:00:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25EE6C7EE24 for ; Mon, 15 May 2023 18:03:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7B7E900007; Mon, 15 May 2023 14:03:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96BB3900002; Mon, 15 May 2023 14:03:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8335C900007; Mon, 15 May 2023 14:03:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6A9DC900002 for ; Mon, 15 May 2023 14:03:50 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1E81B141424 for ; Mon, 15 May 2023 18:03:50 +0000 (UTC) X-FDA: 80793262620.29.ACE0892 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 3245718000A for ; Mon, 15 May 2023 18:03:47 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aXSRNsRV; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173828; a=rsa-sha256; cv=none; b=NMY9gsg9vuXTccPnj5OCg7EEw5EF2BzKs6LshB1qSJfKJsqwqTrnDAPth32Zm6mWMZys3t gfK+YqjZNMhxRa6gijOY1nUb6MBUOsaI6J8vMMnyrvTDIcJLndGa0h8IRxjXyBn3ESoMuQ NVfdoqt1jtDAsoR+P3ytyWnz9ZQopQ8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aXSRNsRV; spf=pass (imf24.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=17wik0CX4Ctne5k/8Sx2ByUswspOZ54CsQaysWp8u+M=; b=NaVOV1OA9Z5YjVsRvlt3hCIJiWRwC+YYjILzzMrI5/g1/8ALjzia0xXaCUQWFy4+LaTnw2 0iuys+ZoeoPJGSgDxDEc4DiyOcsqUUtpa3ZLMq/y9cimExnuoHYDQWwDLEuwM60Cq9mn4h 1dCoT2t4UItR23haUUcelJIHDEFq0WI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=17wik0CX4Ctne5k/8Sx2ByUswspOZ54CsQaysWp8u+M=; b=aXSRNsRV/Pe7c167mA2DapEm91nJVlLw0DViLXfeFE9/yw/Nk6vanTebsnfwegq9unWfOe l2bU5VP/y8g0WXAHlXyUNLkztfh+iU8F+sYVVUCUiPtjm/k8lB/CnElsEuTZdVTic2Xz66 pS7RVml58IlDgpRGVg6ubW6aerXsdw8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-133-OxnDqdCGMg-hW8ImTw0DVg-1; Mon, 15 May 2023 14:03:42 -0400 X-MC-Unique: OxnDqdCGMg-hW8ImTw0DVg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6554286C60D; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 047482026D16; Mon, 15 May 2023 18:03:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 74B2A4161E52C; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.642582847@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:24 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 09/13] vmstat: switch per-cpu vmstat counters to 32-bits References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3245718000A X-Stat-Signature: 5iwqqwso4pd9om1a1anx9s4x8m3qoyof X-HE-Tag: 1684173827-665201 X-HE-Meta: U2FsdGVkX18jG8XLYQe3FSrCXlIyEwDutJU8SCqtsmW0XJFQAxjauaLDKNcP8Sg7jTr7Luaea07wDAYEimiJWZFzH0mWWyDhNMvItxQmK9Z1i8sQFx+dmxpk1n+wrMFEPQ/okepr0XqBv3/NZafYZLij15uNIuk7Xx/jhnEdsrNvHS4NnfT3eOLIEAsyVcQq3QO37vFzjiIT1BU0xQ43nIC6I/AhxTKlPdT4vf1hfRvQVbZHOg5dyCSPty2jxeTR2h3PHFDcZu6yIsVo3eOaTCWLnyP9t3wcl98g5oYG402RICgrrdXYOdB6RI7zOeXVnSdGeWBtVdfG7QW6Q1NLFCqJ2cGM0ytSllvDSlK+pM74EUHfebIb7MEnnvYDnp1uQ/PtXCV6uPHzWoA+UIcwJiZZz7UKNWNhsUqN75Wl30XB7xOz6O4C1EkV13xLCjJyQYLNOfXP2AJV1MDTeAJPjOIWKSJFCOH9HOw8ul1IGrG0b7n4Pl6axtqb8VaywjMyPTlZUv9fUnniU2aD5wcs/dQN4nNg0zDDG1GqG4HIbUquwb0Tp97hmXkYMNbAHWxkMQxZghLKXpKQLhXlIuekdaZ5GH5qdscOsfghA63FgBqU/glKCMifOmal7Zg/MdwVgyF/w7WYfRZ4/1TqvOL5v5M7IhjZdw4F6siFR5L3UpRgV7QwEkAyTGrWIP76nnI+7+8QZjyGRsYou5dT9UR1+N0UQ9RsSfZZ8zBvcurJgLeyDfxxvH37oX6NlkyIAj4nVf7zk97umvroaHfPYXEkks0nwUB2kicLBrB6A5aB6q4kdx+qU0L/hwKK/eWcF7J4qbNQ07ABYa42R8kbZPM6ZCei+e3TX2nP17Wi/XnyjKVIvpEGAIUPWnh+SD7sFLto/yblNJ5ztjquDGAMvljpBI/fG//nwnMakRrnK64KkCJbI7Xf7j1ED7OL6Leoelr3YPs2QrjUJhOuOKNPxQZ 7T7tiTC0 QrMK4FsTs0H899I+XFfSL8Rz78OWt7lzuRbrfTv+tamrUlxYeu61T1b1i8WSo19C2KTj8f96fGWrubkRVwCMk0gTt1+HdRFL4jXrd4OGfZtBrqSxuU0If27YV22sqKDafz04HZjRxPiMgau972pCgyJSd6kU8YIwgB4782DG5jaa9pUiyRLbeUjgbQg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures only provide xchg/cmpxchg in 32/64-bit quantities. Since the next patch is about to use xchg on per-CPU vmstat counters, switch them to s32. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -675,8 +675,8 @@ struct per_cpu_pages { struct per_cpu_zonestat { #ifdef CONFIG_SMP - s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; - s8 stat_threshold; + s32 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; + s32 stat_threshold; #endif #ifdef CONFIG_NUMA /* @@ -689,8 +689,8 @@ struct per_cpu_zonestat { }; struct per_cpu_nodestat { - s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 stat_threshold; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -351,7 +351,7 @@ static inline void mod_zone_state(struct long delta, int overstep_mode) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long o, n, t, z; do { @@ -428,7 +428,7 @@ static inline void mod_node_state(struct int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; if (vmstat_item_in_bytes(item)) { @@ -525,7 +525,7 @@ void __mod_zone_page_state(struct zone * long delta) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long x; long t; @@ -556,7 +556,7 @@ void __mod_node_page_state(struct pglist long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -614,8 +614,8 @@ EXPORT_SYMBOL(__mod_node_page_state); void __inc_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -623,7 +623,7 @@ void __inc_zone_state(struct zone *zone, v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v + overstep, zone, item); __this_cpu_write(*p, -overstep); @@ -635,8 +635,8 @@ void __inc_zone_state(struct zone *zone, void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -646,7 +646,7 @@ void __inc_node_state(struct pglist_data v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -670,8 +670,8 @@ EXPORT_SYMBOL(__inc_node_page_state); void __dec_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -679,7 +679,7 @@ void __dec_zone_state(struct zone *zone, v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v - overstep, zone, item); __this_cpu_write(*p, overstep); @@ -691,8 +691,8 @@ void __dec_zone_state(struct zone *zone, void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -702,7 +702,7 @@ void __dec_node_state(struct pglist_data v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); From patchwork Mon May 15 18:00:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D33A5C7EE24 for ; Mon, 15 May 2023 18:04:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 685D090000F; Mon, 15 May 2023 14:04:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 639CF90000D; Mon, 15 May 2023 14:04:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39EE390000F; Mon, 15 May 2023 14:04:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 23CEF90000D for ; Mon, 15 May 2023 14:04:03 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D87D441432 for ; Mon, 15 May 2023 18:04:02 +0000 (UTC) X-FDA: 80793263124.13.F247982 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id EEE5F160043 for ; Mon, 15 May 2023 18:03:49 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="M/Drxb1/"; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Hcv+7yXWUE6dz6IV3pdXnHOeNZqOsphpnY/hyL0+XpA=; b=mRGX8W22J+5SiPmWtg+dszZKzJ8bylXv47agQjPVPFeTLKtOJJaBatpB6r3btqdjiZ4ugV R3x/sWrvIzs+9Iuk9eNLZ07/DEzcNoeCMOiCp9PMd7QHLgPKKGyFrFgnspZnmal0Q894Rv hO0inrzBbLlpbivgZwsKuTYX4Gh8Pnc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="M/Drxb1/"; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173830; a=rsa-sha256; cv=none; b=Oak/00cjbhxtf0fiwDdK5UpQkTAItiY/7bMMmCsqESFu/yWy3UfRQYZW7uv8uf3REa3TrO 0HKcO5CPEz5NTA3fI4B/9lfnnb1/J73tyih/mdrghgb5uvzCvhrBJYiAE5dvH8r1VZh4cB bSluq2fl6D+nz+Vzq+ORh2NAIow1WzA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Hcv+7yXWUE6dz6IV3pdXnHOeNZqOsphpnY/hyL0+XpA=; b=M/Drxb1/aAle5djKgVVX8K2IpxERk6fQNT3pY0mOzN6FpOJh7IhSdUnTBlOvr/zlpOox8k MXqJ3vC9obHmcFjUetmEcBByC4YjaPWziWxWOCdDcFzOwg2W2E71ek3s2PMGWEGc34NCUD 15gjkLCKNymGOzLoxkc+g3lhQqLFYHw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-411-lHcv7IUkO2a5FK4nyxKaQw-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: lHcv7IUkO2a5FK4nyxKaQw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0080E88CC41; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C62E92166B26; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 7B6454133CFAA; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.668066640@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:25 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 10/13] mm/vmstat: use xchg in cpu_vm_stats_fold References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: e7meun9e3d8sof4gpre1pqu7aqryh93x X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EEE5F160043 X-Rspam-User: X-HE-Tag: 1684173829-180301 X-HE-Meta: U2FsdGVkX1/2yee71Mm95++SqKW7Wk8Aw4dsZq/CH7qiqzq2QBockc5yW5xhVa9YvFiyzX/QboNu3FhNGQmkqsaO/4O0UOGxrLiuhKwGALn4Ld3Pz4IcizSgMH3MzY/3l9+Dz0HxU+GJiE1MxVm9VY6WlKPIJ0+7K8DbJ+eQ9Ri6ni5U3Go87k/wKmUNlaMHJK5DbFuujIu1f/Ms+XCbPIkM/YZfrG4ljwdB6YngxUHdIlGhDerZ0Jt3e64Xvb6+LLGaVaNoOvUOlMxE4rYYd0FDTwfHKc5biMcICEKxdT5GOXi7Q6RZX34Rr+t2LqEpsifG2J+g29ow9pyIgXaBCsLCG7B38RtfONlCzl5FFUGMXviUif9Jg8uHyM86pwSq2wVMVvAbJ1nFD9drYy+WaTF/nmBgcjI13Gl5kokhOWG2d7ektsBXHMMsIFTJPc1XbP6vZa5G37WSyJdUS/50l2MQeFHGnaw7u1SR2yrV6WEdJyXUImo2BYf1yeZ0FB2psBJp3TOWEBTSRNhmIrcTQpdbP8lbBsy013Pp5qzdQRE6HE1Y0+5wNIuoc6JijRmKcRa28yrXvr1iulavl0bNFbxmmS48exy4rg0evfLQVUIRm/25nVPVw63x02tbwng4mdn2D021d2ExYAsOaY+zZaFqQdf2dRaCVnqLkesATR8myNXIKNXrW7YRApVOA7u0ITiPcvYhuTHcfV+5VZojJRbdtRCzS1ylMm2s64k0qoOMr6SCJCTcTZZeGxA9aIXa6P+j6JnYDHF8vgtJgxxx2wZNkAM6ovNhlfxdemGYOQy9dRrOwX0MVCVUpDUPfHsDRquLYWXN2u/xVc3sFPhvBSA45DtnhMiX1qgxs7Jq4xgFgz2eSREtgNqdSq3hvQpbfI3hnoxQXq9u65qLgZD+xvz5JxU28aIDaho1nJMTQzKneXW0eDUJkVHD04o58ajhSVlGR+bQamUpwOBye1m dpM56YbU tOQyDQ9pgQ/GfcMiqNrMH79nYD0QSZc81/z9TzZD3boCvGEYfqV4hQrkX4+mJ0y+9j8PRCI3skU2SYytuYzDyHNG+bZvcJonPl1LSGN2CBD+4QcgqIp50I/tTkb/PQZwAyhC9It0DvQ7Jq8NusmZW79x65JpSQLgIPv/OmFfP5DX7xWOthulW00P6vp9luuh+4E10iQA6ELiAKpisJPwvmEbtogDNwLBPvRXa9yRntg1qrZLxdQX+soihRy28nzySky7h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use xchg instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -924,7 +924,7 @@ static int refresh_cpu_vm_stats(bool do_ } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -945,8 +945,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -956,8 +955,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + v = xchg(&pzstats->vm_numa_event[i], 0); zone_numa_event_add(v, zone, i); } } @@ -973,8 +971,7 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + v = xchg(&p->vm_node_stat_diff[i], 0); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Mon May 15 18:00:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2CA7C77B7D for ; Mon, 15 May 2023 18:03:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65E9590000B; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60F0F90000A; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B29590000B; Mon, 15 May 2023 14:03:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 360E6900006 for ; Mon, 15 May 2023 14:03:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0722A1613EE for ; Mon, 15 May 2023 18:03:52 +0000 (UTC) X-FDA: 80793262704.22.581468C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id CB2E11C003C for ; Mon, 15 May 2023 18:03:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V0z1gkZ3; spf=pass (imf20.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=jkb4dlPC4zyep6CJ8cMLWwNNaJdusa90HtfOUuDaHEw=; b=ZxHovGdNGoBf3eq6Ed+mhbcrNu+jgNDBS909wGJekkekHfQLNDsOmotb+PJGsT+sEmAg3a 2vH7pHhahRVWtq0z2YDvIVnEEUO+YrFX6leiXxlB+pv6SFnFJVwy3zLKi4+Cbrv4yVo7Lp AqhjMHs+FpkgHOMPvRDs4QfEJ1DHi2U= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V0z1gkZ3; spf=pass (imf20.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173828; a=rsa-sha256; cv=none; b=AagfsiybRqupYc7kPFC/NK9Z7vW3Fc8E2S0VG8TYF4s1Bq9idVbq3IUXogDtt6x4bWvtVP R3nJC9FCaAJthGRquJ4CR+bM3T5/8TYrS1Ud4MVyt/HifzHZwFqMe+2ITbME3AjL0KucTd Y8fucudx9JwYLqWddLTZRxz+6UpboCE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=jkb4dlPC4zyep6CJ8cMLWwNNaJdusa90HtfOUuDaHEw=; b=V0z1gkZ3uh3k9q0poPJcxm5RP+UILH9flOUGPR6wa1ksv7L8pLflJm5jcOC97mlruRUUDd QXF5E8wH/qHxa7DJs3dwqHsbcq2EjGBip8+J+Rb3C0bVu00hmr5BJYbP/3lFRatGPltKNP XgmO1GcFa2pM6dcOShPZzhXpJAFu1GQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-gp1bspFaPzeFbxO02PyHjg-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: gp1bspFaPzeFbxO02PyHjg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 27D73185A79C; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F1C7340C2063; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 81D6B4161E52E; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.692835888@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:26 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 11/13] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Stat-Signature: uo4ps5kq5689oeno63aespod5onbhkiz X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CB2E11C003C X-Rspam-User: X-HE-Tag: 1684173828-829019 X-HE-Meta: U2FsdGVkX1/ZH7+HkeRfM2+cvbHIdUUxKkLA0E++cx6+CCBXMTSxNyybPZGgURusMdxRTrmk57GF8ibfc74vbrFCS+y0cHTMV8+4aGi8wVpZKoiWsC0K2eHFAizqzdN5w9ic+/dfmG+WCDbPGb5u51rK3jCOIObfQTMWxExtnTeOYzVjpknBT6+t748bR48JgwqVMixT4jiWKeJ6d0VZt6vvBIRC5tAouqiQvI+8nZ5NrY2dRee2Dno0jw302HkDP8e3ssD+6H13RpkWh7HM1x8hQrpRc6nDde3M47ZWAABpWjZcdG+mTl4pwBlGn9+7/IZ/A1xVoQ89wg1UtPccVaflMbKUrHGFmqo8SURtkP95QwYFqj4JGb3zzuA0jDKbbZ3cR2I9ONgS7pUn9q4O0mgiwyE3BoyVy45AwOTkh82DPxrTuaIvlfpYp+z2+YGlHDRNu5SBSzZfXOFrPtRNuCaPxl8HTh/knt97NJCUzyBanPvncWXSdV7xueAKBJRcKKufK3Hfb0H8N59YfOoVlsRg45bX1PasXVRXKMvTbhH05IaHsOi72s2NiconFXgYa2u3hz7PXuFYrV8dpXmYojoQdNzZRtkp+lzXdbAGNowMeXmNByC+htYaBUsF6ChkFlmY9UphPcaJOguPqE00zG7WE8aWLJqS+tzrNLf0HCW3mlzcGfNTaSxemF/GgFVuOs4A6k8DLlNOiceBFMl237MKvxSYcvBSw3qEfWsLslHzS/7jmCESFcp4PUVZ3fjHXwi1DbIWSPnuKfle6fNkG1vetMuAhqM5/h2KbWfbLiAutvPjiHsfQdHZVv7oHIfJ8Vm7wSCBGI/cJ204nqATrrU7ggvUbdY0qbsfUQPnhiTXHtb+fQUGVQD2ib2SfBf92dsgH4cMg+/p4Q6yCYNJv4OCNicEMp16PTO8fYbo1EUv09Z0zd8IxpzjiDvhjY75ce3lLr68y0c7gmSuyM2 1PlzdwnW 14O3zuAor2SJgfYcpq1Xs4rWxdaVh5iyFfILGZt8PfNN+B2SRhZEg6jIzqJQV2vwf58LbBrSRBvEQ+bKnQEA6YO2abWy73Kz3ssElcm70cVyJ9x00Roe7g7kI44lUxfPW1EQQFJgKRjh2zZiqSHYNV51ooNKBiUevWSaoXkXxpbPxFguDJIcK0BE6+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications. Performance details for the kworker interruption: oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... The example above shows an additional 7us for the oslat -> kworker -> oslat switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold for certain applications. To fix this, now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2049,6 +2049,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2068,6 +2085,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Mon May 15 18:00:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBD96C77B7D for ; Mon, 15 May 2023 18:04:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6C7290000C; Mon, 15 May 2023 14:03:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A547C90000A; Mon, 15 May 2023 14:03:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A87190000C; Mon, 15 May 2023 14:03:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 79D6390000A for ; Mon, 15 May 2023 14:03:54 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 17979813C6 for ; Mon, 15 May 2023 18:03:54 +0000 (UTC) X-FDA: 80793262788.04.9F9AE19 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id CE449120020 for ; Mon, 15 May 2023 18:03:47 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OnLt/hlo"; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=0t7wAXZL5USXQxpN4vCajFBy/hf1+LlNt2JScd0fQmU=; b=tp+KXl/Ro1Lo+d+lxW7mhP/AVQnC49wuRtjQvj4ij3OUsxjPiekj0ELtV87EilMCJEKoEr FSuKP127YldDufBKW24G24hRZoVut50XkhbpOx4HKCJFEAy64lsWVjFRywTOZ1azj9aCE8 +wxIGesoeB8HzRZngGDZKDHKAzxF+7g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173827; a=rsa-sha256; cv=none; b=ck5kjB1EM+RturTwEsRrIWF4UGWix7sGlNDXADVBe2+81JxYBXP3lUBliw5qtJM5uQLFzd hguA0wyt4ktDo6P3DXtr0gIAnklhDOxwJAgjxVIi2If+Lo6YcVctQiSqy2ip1BxODhZ2uU BfkBAMThbxbCugZyOqqGtS6LhWGyOY0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OnLt/hlo"; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=0t7wAXZL5USXQxpN4vCajFBy/hf1+LlNt2JScd0fQmU=; b=OnLt/hloAQqN9/oh6gQ4tlV/lMNvm5vrUTemEnGhoaBvUASdWNTwFV8VD93TdbQekb1XJA q8ac+myzt3FPmV5dHjULMBytsYHQ5R80GsF56eTD4ecI6OOfB6sMKvlHSIH+F+ZjpvShNM FLuuW7BmSnP3B3jjiPyzRwq8e57pqnc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-411-S8EJZ_4aPK6en7Q5TLj-6A-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: S8EJZ_4aPK6en7Q5TLj-6A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0C0671C0F2E3; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D41A42166B29; Mon, 15 May 2023 18:03:42 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 890914161E531; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.717342763@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:27 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 12/13] mm/vmstat: refresh stats remotely instead of via work item References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Queue-Id: CE449120020 X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: w8upo9686hafu35px6eontmit34m8kxs X-HE-Tag: 1684173827-77598 X-HE-Meta: U2FsdGVkX18jAVik6jFH+emgXP4RQ9RH3ZZYa7LZyPPs7ENthRdrofQiFzhQbC4KyEr3ZgFiabbyvCvmV0onl9PMQZNKlUQRqqWFauK6muO9FDZ5JA3m0iUf4Y0mrrd95nqRS4ngIDIfR/hQogroQnyISeltaTq28k6JrcIIRxHTKMvBT/4v7ZKZD7NIVqP5M7TK5HyUMTHMpNREnOg1QMr/TM1tJb9TftOsxczO8lI39ovc+YNatdW9fSafqVay0P484ps4IJs+NrTBW4/hFQJNjcDVsQB+invuJCj0Cafa7hvQaO1OaL0WPKd4xFzwnwilFsrKWLjd6H0wbXhQwNNu06eh+iZIRZROzJPVPiOpfxbNQ3z+gZWeGqxF9AoDuwCynutuMu0DMYj4KxQMd3ICnRs23Op9F5Tv0HKm7szVYRhkDsxpAPSmPxbelGv/GYimoIZCZtBnDn1pF6rAxk7ZDWmwslNsuo/itE8k3V8BWQXX2DHqP/7R9TkSj2/xCQ+W1Cx5PHbsORwWqDyge/+BbYfdIcuEhYrRnF9BmxIX/5g8ph79Cd/RmNG0w32uVSAFV3MAHj0ttQuEcFE60WOAS7LP9ieqFRbZcUC/Z+fnIAtdoMvKWaczueRvtNQk1iiS7p2RXOGjrqD9Bqc6CbBwaQ58/nMU/Oy8HSxoAESV1VefONoZSkNspd/LfMFF6dqlGS6upjlRPCqi1xkDMKHWQbG9tQn/QRTptCyHOF5XrTXFPyDkrXvyYFMCUW9RKtxDSi8yeNmR0KJDDCKiOjfQ3hI3rFZsqmSSpTs1EeQ3mh0KreytHlSYirpXTHslVLqXrse8vsm3J54VYg7LTDHijm2niYMQ3tFQhb2Y05mhZ3KT6JLarQSqpnOc+AGCz8BWqTNjol0a/M861Lj7CtEVCs7m4zPPoURvRHlo2djg0V3CMZjxSgejGoP15rYfA9DB4P1fmHII7pudAB7 1UgrxBfN rU3U7lc8xYeIQZR4LX1EGhf8SXzFPtn00vNPjdJtS0GKf/Ccflfp6d66cL2Me0PGoQTnVwgZB3Qt4qGY2IP5IHHA2qgkfaDr2uU+NYbk09cmmh5eRYRH4BBhx2yyIwR7xnmg9MEJ7EeVzXU1v18kSOmrT1AIsuAntw+o6HUuFR8v/QM6eUFD6w5UlmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1907,11 +1907,20 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(true); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1931,7 +1940,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2051,7 +2060,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2061,7 +2070,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } From patchwork Mon May 15 18:00:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13242012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1308C77B7D for ; Mon, 15 May 2023 18:03:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EF34900008; Mon, 15 May 2023 14:03:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 39E05900006; Mon, 15 May 2023 14:03:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23F26900008; Mon, 15 May 2023 14:03:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 141E0900006 for ; Mon, 15 May 2023 14:03:51 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C17F3AF4DE for ; Mon, 15 May 2023 18:03:50 +0000 (UTC) X-FDA: 80793262620.24.EDA8642 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id B31991C0013 for ; Mon, 15 May 2023 18:03:47 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f19mH4sz; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684173827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=RFL7dZ0ubDZf1tDjIuRkhAMaPrNe1DHD6pj/+UPZboc=; b=nliwmWfr7aK1aZbzOX6fJOpY0RxnKiapTBW632jbS5gHhIMCOPygU8r3vEESd6C8SRjkzf H6KepQGE34teZhTX/l481DSXWhrxq91fQWGEJdItbXtx66e8ok08IiyJPPkrlsL6HzagXB 0d4lEWBXBdXNvLg5/HOvUqP4entG72Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684173827; a=rsa-sha256; cv=none; b=2bBwAt8mMgoIOOSartNQfs7iDLliOjiZAOPHUDAQecgeOTc1zBj+BzGdHo7NO6p7YI0l4D vgB3z9JUm0QcSFe+dnhztdyajCVYb75VJDltaXEtyEnhdAxQw1x54Xy3DF1is6kDG/RkxA US3SIsqvSfGPLGgBfahrBXsuEi4f0sM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f19mH4sz; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684173827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=RFL7dZ0ubDZf1tDjIuRkhAMaPrNe1DHD6pj/+UPZboc=; b=f19mH4sz1LB90IqxffZnn79SvZat44U/TXUGe1uc7x04DDPCQsLXrVnQ5jR4Wf5JZ8VS+i OLpp87/urzjqEhjAL3+57xO+ne/+UotF1lNb2X76SBUDVNOZyc2o1iF5Av34GGlO+Ra5uL elStIW+zmxEIyRDkcBHS3rbEFH5v2rs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-pdz-PnGCOay5sxJCDmG24w-1; Mon, 15 May 2023 14:03:43 -0400 X-MC-Unique: pdz-PnGCOay5sxJCDmG24w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3A68188CC44; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-4.gru2.redhat.com [10.97.112.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 033C935453; Mon, 15 May 2023 18:03:43 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 8FD874161E532; Mon, 15 May 2023 15:02:17 -0300 (-03) Message-ID: <20230515180138.742158693@redhat.com> User-Agent: quilt/0.67 Date: Mon, 15 May 2023 15:00:28 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v8 13/13] vmstat: add pcp remote node draining via cpu_vm_stats_fold References: <20230515180015.016409657@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Stat-Signature: 4mexaif6nw771t93k47u3c5587wiaide X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B31991C0013 X-Rspam-User: X-HE-Tag: 1684173827-69748 X-HE-Meta: U2FsdGVkX19xYFjtJGfJunIr82LXRZbqD3OkVG+q+aE2fcC+jNlwE8K3WyjVK//7V0oON66GIuP8QqpoNv0XzskdvMtMt9w+dc5ABnlKzQk8XYM8OjzNkQpTYumbQsKplvWZyeWTrfMcDPGrSoxC+UabyItIzQX0ZhBC5r66n4nHduV8kA50ULgeps8RQAzNuw1jMPaJfeRHZ9YgVeY+o3j2oa6d+caFHrKt28GDATZsRPJtyiv6wk8mCm+qv0oJlgZ4kHhcoqK6ei4fObIaGwnfyG2CLD+S27o7LnqdzGPHlS1df4aahH6Blbq5TPUdZPVFxpk+3eMpmcX1B+SbmRGOh8PJt8dvoUWIEvmZ52FMBavXQd4wAL1JRQ3vqtjKASZkmomZfKXV+jDNxm5s4VV5DdAba8c6kClZ+GLG2a+DXTeIXdPJy1vGFOh9dBywuuBL9jysBTG3S8qZFmaUNxijOKI6wAOL+vMzEN2ZKPWBCy1dl4ZC2uh1zp/rtG8MKon10VdaLzJQiwerHXeUdg9FsHo1RKlDXao1a2in/U4Q8XdY08bWciC8YGRxG5niBoMlzpDYPtBQvbddPr3+rogsGGX3bw3CPDJOSlmcVUpWs1LK0Z0JuhZtCD/Wvx0XKlH5o34rwEKI9hUxGO3NqRSZ+MScjfnCRLiyV9y8LpTgl8LOHom2F3+aujUHWgYOwhddeHLAjSxzWlyqYXAj2wAkYQZDOpxOW6YFSWnea1dKdGsm0Syyg1488tuvvb+u/dt86rZ/+thN76BMcWE9zlFL+fqYmJeZI9MUC6zMJj5vBr1vu7qk+IhmlWcxR9TQpJ4Bq/JM1tp50bWv4E/J2joTVGNnmXhEWBggyY8BPBsDkcg4mLNVZIlBrf9o1kTfk+W/TSip+yfVnR0G3K4fKCLHdJ4JCIxQFVsAAK9Uw6XYR5+YPBB7gSCAqcOUruZd44qbsxvZYIdfIky9YbD Sfo0he98 17oEAci0uq7AINdWcOQjAio8wu/S5cCXILCvTueZr/1c9fmNo53uxVAJS+z43+YU3SuBXoZcVNwZ5g4tWXC+a2FgGOicKJKE00FJSrNA/QmzONohjnqRXUOXBkUo5RK4S/iIcf6A0jmqJkWJdapNNtBt0CZIK5EjN2QK7cLflccbYSfnhIUd09b4R8ZwfUqm1ld5d8SHk5/EXY5JUxyKrUGL+aPHfVCpxmqek0wTOXHtk7DE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Large NUMA systems might have significant portions of system memory to be trapped in pcp queues. The number of pcp is determined by the number of processors and nodes in a system. A system with 4 processors and 2 nodes has 8 pcps which is okay. But a system with 1024 processors and 512 nodes has 512k pcps with a high potential for large amount of memory being caught in them. Enable remote node draining for the CONFIG_HAVE_CMPXCHG_LOCAL case, where vmstat_shepherd will perform the aging and draining via cpu_vm_stats_fold. Suggested-by: Vlastimil Babka Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -928,7 +928,7 @@ static int refresh_cpu_vm_stats(bool do_ * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ -void cpu_vm_stats_fold(int cpu) +void cpu_vm_stats_fold(int cpu, bool do_pagesets) { struct pglist_data *pgdat; struct zone *zone; @@ -938,6 +938,9 @@ void cpu_vm_stats_fold(int cpu) for_each_populated_zone(zone) { struct per_cpu_zonestat *pzstats; +#ifdef CONFIG_NUMA + struct per_cpu_pages *pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); +#endif pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); @@ -948,6 +951,11 @@ void cpu_vm_stats_fold(int cpu) v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; +#ifdef CONFIG_NUMA + /* 3 seconds idle till flush */ + if (do_pagesets) + pcp->expire = 3; +#endif } } #ifdef CONFIG_NUMA @@ -959,6 +967,38 @@ void cpu_vm_stats_fold(int cpu) zone_numa_event_add(v, zone, i); } } + + if (do_pagesets) { + cond_resched(); + /* + * Deal with draining the remote pageset of a + * processor + * + * Check if there are pages remaining in this pageset + * if not then there is nothing to expire. + */ + if (!pcp->expire || !pcp->count) + continue; + + /* + * We never drain zones local to this processor. + */ + if (zone_to_nid(zone) == cpu_to_node(cpu)) { + pcp->expire = 0; + continue; + } + + WARN_ON(pcp->expire < 0); + /* + * pcp->expire is only accessed from vmstat_shepherd context, + * therefore no locking is required. + */ + if (--pcp->expire) + continue; + + if (pcp->count) + drain_zone_pages(zone, pcp); + } #endif } @@ -2066,7 +2106,7 @@ static int refresh_all_vm_stats(void) cpus_read_lock(); for_each_online_cpu(cpu) { - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, true); cond_resched(); } cpus_read_unlock(); Index: linux-vmstat-remote/include/linux/vmstat.h =================================================================== --- linux-vmstat-remote.orig/include/linux/vmstat.h +++ linux-vmstat-remote/include/linux/vmstat.h @@ -297,7 +297,7 @@ extern void __dec_zone_state(struct zone extern void __dec_node_state(struct pglist_data *, enum node_stat_item); void quiet_vmstat(void); -void cpu_vm_stats_fold(int cpu); +void cpu_vm_stats_fold(int cpu, bool do_pagesets); void refresh_zone_stat_thresholds(void); struct ctl_table; Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -6250,7 +6250,7 @@ static int page_alloc_cpu_dead(unsigned * Zero the differential counters of the dead processor * so that the vm statistics are consistent. */ - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, false); for_each_populated_zone(zone) zone_pcp_update(zone, 0);