From patchwork Mon Mar 13 16:25:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81EB6C61DA4 for ; Mon, 13 Mar 2023 16:27:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D15886B0085; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9BD56B0089; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 960AF6B008A; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5B93D6B0085 for ; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B05AA120C0C for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) X-FDA: 80564406258.10.8C43D2C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id E25CA80017 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R9glNPjQ; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=4oQtwdfhDtmRySahNTjHTZp/pLgf5DxrRbskxg1WxT2L+7Grp3jpwmtQ3lBATfiCvAGo7T GANQV6jtaUp2nGmbgojJ99ncrkfqxyl8gC1VSdii630HOCGcigS9mAL2u3YpKcsIib9SgV 0yhqe0qsYfEZPcIpQIxSlgtFVRqykZY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R9glNPjQ; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724867; a=rsa-sha256; cv=none; b=hsehvyXCoKj9UBtYhkyVloQnTaO3CdfySf741LBaLhjhqbxEKrny90zwxMWSiKkzTua5qS jycnhLa6Tji4RntuIO3lRU1MYbLPxm6XO6+TqvO0LZC55tr0yd2igQUSkluLNRut+P6UJ2 5oSRd+MFDlDoZhouE9tmD6yzsOeFDAw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724867; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=R9glNPjQ9JH41ZUUhZm78C36m6D7U/izmO00owo2Xjm/m6eHJapvMEmTvr7RTjHBlhfW8t anaNQC4wC79oXfRssqtONlYHzYmI93UG72fyYBcaoflIG61SKF7qsuD5RrPVUE8pkaawRt cY1gHCzGHxrYsg2LMxtiI46QmzaDhI0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-256-_i-nT0qoM-Wt4M1D21UMfQ-1; Mon, 13 Mar 2023 12:27:40 -0400 X-MC-Unique: _i-nT0qoM-Wt4M1D21UMfQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 32BC618E0044; Mon, 13 Mar 2023 16:27:40 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E93CA1121318; Mon, 13 Mar 2023 16:27:39 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 11E84401344A4; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.283504192@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:08 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 01/12] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E25CA80017 X-Stat-Signature: 59hmarrzf513pjt3hg3j5nimqz5drbsr X-HE-Tag: 1678724867-536217 X-HE-Meta: U2FsdGVkX18yIpAQ5oFmGsr1AnOAsYklQxGNHKdc6f5BOt86aU68B/O9rpUKVgVVichDom9Rt05H1NFvc+2SBKwvZw+NzNSlnHuD6nEDxIz/4GSug8qVUPwg2vbn9+aBZ8hmWkqvOND0Hj4zBsthtLDgG3J120qmc8svKh5Lif1og3ENlwkcAgpXZ1FQ/1pmUMVMQHJHlVhkTIM66zfzyiQC+pkh2d1XEGjOd4+pkztWTeYQidENHiEcIc8RbaecJYvOX+3z0qiMCfX9pZ2CY0EDNVQdsdmF1mNL3SbDCAJ2xp+0zc9fmVTC1fpA7SJBKRDokDef8V9DvlUhHLBryQ3zCyzJWf2WNYB7NIkIm+RygDF7TqyI5UR6fHu1w+Ct89xJH1Zi7lqcdbT3l2yDK6d+zjTMahK8ovVZ+pVzWixAf7gkPgWIoxy6rrusXNPuP0G5eZ5SYns8BeNxl+JbiiH0X4f9vGeUSkKJOGcolBy5HyL/4EnoYhtumUUtfM6JZ2Muk8Ou9CLBDEqwxBSxns/v9+06in00TL8O7PY7UnNZe+DkLFeDHCgyJJl9yw70mJP7Wu+Fk3wcDYWAFBgWgyCOgcjHUpyEwivsgs4f6N3Ar5yER3Rz9APooQXBX8EGYbfWZIe6lR51ltwP9IwcfT/XWafj2yPtr5qTdWVL/RsXBcHi5Qr2DS4d8hk3Yjw4MnZF8v6XNxPhd3MemhlAczIo//bF2UqTXa7AYP4b30S9HNeNyQf9ZMFG3XPAVFQNNcrqq/XxThbK0dmZIjvVNJjHKGVkPgbD5e4juVU5Pk0YDloSHCUyGYNSG4E+Esmhp6YH8TrSvFSnQ4Ek+cgZfCMH0jHpiQiuZtlCcHWhjKIIdaOwu4kj+BHKqr0JF9gaUOuIN9Qqdo9Q2ShP/hPXC2jhq0Ts1GOesgbY2/G7/+AW+gJsQ+BoyvhscjwPHgDc55TdM/dVmldgpuXarvh m2NclvMm ENzRp4HJW49xfwmpWpHB8xrRscHPONexrAhUGgv4gmKo+3meVi8tZogHVqH8vOSiml8tjdYgDx5R8Y7v4/BrJ5yIN9kiohQC+rM6ezNqVJN+pl/5SbaOYSDMeF7ZB+d1cT7UnT9HnpU4BoC2VkYRmdoWy8nWllewS34W9BKGgw6VUuLPy1USbZjvB43zqcRgBhO32k9VnJnLQcXoSgXfeg7HspAfVctTIzAtl6TmzqNwQYwnYdwVjjGntZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Mon Mar 13 16:25:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 352A1C61DA4 for ; Mon, 13 Mar 2023 16:27:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17FC16B007E; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0607E6B0082; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D327F6B0081; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B54206B007D for ; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8CEA9AB0D5 for ; Mon, 13 Mar 2023 16:27:48 +0000 (UTC) X-FDA: 80564406216.16.388DBF0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id BAE201C000E for ; Mon, 13 Mar 2023 16:27:46 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AwMGNxuL; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724866; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=M86qbDBArWhe2XMAjlPp4DL+LxOl8N66efKDtvynYHQa1HN9lZT1QEj2VCnCmeJfIttknX vuxZ2/eQWGm7kp5LVr+szsKMgbgRZbX+Ig8QMxGJ++X8TFxdaQA6vJGZVZzZbvUPPt8iRq hdFqUp887dhbhRwJsyH6JsJHk7QlaWU= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AwMGNxuL; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724866; a=rsa-sha256; cv=none; b=t0OWFs3b33EVRlHOcb+hGRyHrKhFdKYp1cmCUITV84PI9U5cJwUVWNXy18nqzl9oS69Ty3 aHfIIzGiEPIw6D4O9lkJHuAsMa13drCBijyfLwp/8Ea/GeEnMzig7oRlp3sep2eKyU9DnD i8Q43bEX7ZG++6wg0RRLX5xLZfZTcR0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=AwMGNxuLKVSaBiNlIyxU0qNHq1G6nH3pjjT2RXTiA/3NkKnLWe1eEnTFfb7t4PvvlB2JNe 5ZIWP//DNcEbVgBa9+bQCzdsOUf2V+t0eR4rUCUvsuQQHU+0b2LscicKIXzrCHDZBWjRgf htGyx+C0Gm/t/9MINJozO/KtWzx1kAk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-131-IclwUQ-EOE-jUgKOuaRX4Q-1; Mon, 13 Mar 2023 12:27:42 -0400 X-MC-Unique: IclwUQ-EOE-jUgKOuaRX4Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CAABD3C0D18B; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9A1CD140E991; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 156F940153588; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.309252817@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:09 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 02/12] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BAE201C000E X-Rspam-User: X-Stat-Signature: p1jm4cxgw6uxco1zfikibwef1t8yutkn X-HE-Tag: 1678724866-395030 X-HE-Meta: U2FsdGVkX1/YBzIoplJKuu8PoY8impPheJscgcTIKGD6hE0Vo+/napVsV9wKFSWX3hCEG0jcYzjDUxwoXjq3x2LnZSZmERZUs1y59cXEY/n1G3C9u+XB8XD7G+srICnVF3fo+P/CcarpJakbgnuPpzW+lihp49YFxUpgrton334jsH88f1zNl4J4u+jwAFgL0ZdLL4i48c9+Skv38y4dAqhZVJZEFilNqNuNLBUlK2VXxLaBQ6Jn0MmJNzC1MXTqXY7ZQy2+VlJE1tLPjGN2RDZAPfViaSDpkzwqVOdOWgYVfBAs5Jk2fdc/5Yim+rv9QxP6h9aUsWAJKYkIp+UlZz43Y8ZtFk/og8aMUa2XyxFP4ZKkCkQxeyz8eDz08MbG/4TT52MmQaAJvLJbJMdOiKh1pE1ZIe7vyRfQqr5uGkMozgDQ34fzL9Gu3iAiv5P7LslFvyGyTKKQpiUGcE1C64or2w7OTmD+EXZ6eUSqAxCJzyzgrmNimxHtjeERB39JkNv4menhuKWXaE6EYg12lpOxvAI5wYYjSEYJs/br1cIgY6CmUsPJU9vacJViro0U5y3Pqf1IRZz1iOemPwN1qiyh8vH1od/2+gaoFGVT31N4PWbqnt6KiUs6O2hgYGAY8neGNUXkF0CHHQ1atp/gdfF2i6ObMYLP1fB1S4Qz84KRRfPleAXwqPxIrP4bPV9O8un18wjcX+VdfU580eLP5ONqiKK7NxgxvjkbOeeUWhNhMK8myi93aGqFZtcZeshrRLSjSOUNkyQJLL6OOcf8pP89hodmoZguT3iE+7D6Gk1iidnXbebaUmG7QYyQVJUnFHiWVnQfihX4m/CQmq/plrbQithILHc06cpQgJgFjD6riMPBM2frP4BROkU/6h5Ts2uvRAHWxClgOLVWbGXoXrgwdsmCDiAlzJYxTdORkOYsbvKi41HdWEV9ZbDp6OL06sds5ssmlsl/s1DoCAf wyy+rN9F y90wA5/3CQEu+YvK29RaHJgQfisQLPMlhcwpbCbvNBwt4FDh2xAjiEJo7gcULoltJ8YjkHmzzZGmsPCFiiHLxNCcVIGXZakDN8pvSNOVkoLpqMrHL5wu37050pDvneZazI/JyFYH65rlUR4MMCU/bSkBK+0lDQ5ZzMiSvmfJSvqU48JUODeR/0EJ+jyXl4RqhrgX1TAEdTS6NTK+EQqEhHd0R3f7Wnt2RueblrHyHiNcUlTNyu1cthKBFEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Mon Mar 13 16:25:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E34ADC6FD19 for ; Mon, 13 Mar 2023 16:27:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1D2F6B007D; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA6516B007E; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B376D6B0080; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 93A3A6B007D for ; Mon, 13 Mar 2023 12:27:48 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 381D31A0C13 for ; Mon, 13 Mar 2023 16:27:48 +0000 (UTC) X-FDA: 80564406216.22.C29DC0E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 6C3F9C0004 for ; Mon, 13 Mar 2023 16:27:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7u4V0dh; spf=pass (imf28.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724866; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=YraQDrjxzlivlfHvqLEKHFt4+TJUm4UhF3XUTe+IRidl7Od44cvKnr/TiUjoKFCN4JW0oE jVY+ZZDFbf0bKAjUvTTRYAhHy1nZpY6yKX851QrJ8QuthSMp79yTCHRlziL3xhyBIsxWe8 RJqkVWWtW3cMhT76MgG4yCc/c9gH7dE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7u4V0dh; spf=pass (imf28.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724866; a=rsa-sha256; cv=none; b=VXE8NO4i8NWdwFCPmpAhNkIQxSPwEQZTWCvRoiycCoUCcCmnF3Uq/anNDMEBoeCtPTnU5Z 9qe965x1VlxxgpKlNa6GmNAkkD9mD0iEttciW+cDi5vlOa96Aqsd/KqlReWTPVNJwBcyf5 KSdLvxABY2JdC09UAvIYwxXRtNaklMI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724865; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=d7u4V0dhmvMLFdpQnLP7FiRD4kdqa9PFJBwGZB0zYNZT2vf0a5DT00YXp20nTxBoxmczcU 6dcymya+7RjsX9Lh+1ZWrH51YSZ1GLSOqHMH6itc0aJWvPao6TwuqebmRjOaGCmrT3Sair dGT/i/rLw/XDFbkcTepQtggKxfcfIyg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-336-rgJUFS3UNpO8OZyHkd8jqg-1; Mon, 13 Mar 2023 12:27:42 -0400 X-MC-Unique: rgJUFS3UNpO8OZyHkd8jqg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D28E680280D; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A603C140E96A; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 17C3B401CB959; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.333370216@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:10 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 03/12] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: 6C3F9C0004 X-Stat-Signature: woaif456bidzzd3yk78e44tuz9yh5i7q X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678724866-905730 X-HE-Meta: U2FsdGVkX1/v1jlduXcvLC2CAbMVOmzYlnm+lyA1HErj65cCzwU4lpzLGyx0x9bgsWHWxafQzkcvUhbSzB++lnved0Z3I7sW19+2VxRgl6E3yHcweJr2f8xBsviBWCHZhFNm+qKhGtdV1d4F04veLWcXqRNlzETnh/m6HMG4y6mCupzrJqd2JejYvFyBvrwCxl0kZHNWp9XP+pdvnwspWfSWYC0SYiLAbk/8FOxMqAssmaqLR3SPw8oR64fHKs0cm/7VoR80QBIgJvxXADbR9BtBi8jWWS+a3nyVs98JWE7RZdnMXh1Mwf46IQPmpgSwDAerdxsr316R5MA1kmw3fFxTPqUuHiq054m5Zc3Ge68FaZ6M0vcUxcfe9PYSSpgGgjFJFtLcLS8LJjA9PJ2mwM2n4e1qYeLr684JrmqE0/7+F45DNkBv47QqmxtzFmV0rP93fBqjueFmZzad6dVnwEOPwD2Jsok7ol+u4oz5iAHW1ktYjE0wCeJK+jS45lcn/BUO1+GIFVkB2dAQI2PXErvBWO3TovrAdkQyMsWms9i7//aa+ImsZxwzPbgcqlnX5K87x9H1zxSaRN2XUocnePR0RCwLUNja2oBSh+gpTjuuEuqw6/o3aUXBLilO4qDMStTcgXXqS0KntUIDUNOrptHQ9mIggxhhc+rqlu27A+clX/uKo9LZwTN5K1cDT0AWU8bDfogxE5JRfSPKvyLdvyXFKQ+JH5qzXxe7U5FrlW0G7bjtGhIcqRXY8Rr4x5z0+s24+gEOegxiW3tA8UiomRlsIsWSnhCLgbnArW4zsBD3bLRQIJouRh79jtQnPufxH0DFjZUvs3PSQPLxTJUMv/sw5z1BljIrZ2+CTHAK2gqkXn0n/kYSYX/RbfuUMegtpKkdCuZUF0jycj7tNgSzkDVxlQdDtdyZ/dD/PIcIki0H24BelxTi4tr3TGijeAy/TDIKi72DRULTNrJOT+p OtAljqFg 7vcMkSmFEs6qLFsge8IUAMcpKr7zL84tTCAIj3Ju4Tm86XwWWYi5HteVLVbNx4ZaYfJViAUmpohgzQDv9q5HHj18th7JTPAqg8yJU43u/+y8NICI5vJAXEh9feZ16F9zMwxf+n+izAwDx8sDRjJ7pV+XwRwLx05UkO5QTwvJ5OTR3+XscB/N8lZo4Bd/2bxgW8QzNlSIots/LqCl/34YBJDflAdjo4LBM+EmKJW/2uZAJw2Ns+UQLzCaWjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Mon Mar 13 16:25:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B113FC74A4B for ; Mon, 13 Mar 2023 16:28:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D34756B0088; Mon, 13 Mar 2023 12:27:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBAC26B0089; Mon, 13 Mar 2023 12:27:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B34C96B008A; Mon, 13 Mar 2023 12:27:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A152E6B0088 for ; Mon, 13 Mar 2023 12:27:51 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2186940C55 for ; Mon, 13 Mar 2023 16:27:51 +0000 (UTC) X-FDA: 80564406342.25.B05354A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 63678100010 for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ks9A+Y+U; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724869; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=i+J9VG0KtXMnwgDyWdy6NY5iJ7kUAK0rl/V2ECq3RDov9rIo4AwoMU9gg4Ydqghaykb1d2 HrhmFD7dwsZ6jKqT6mtTGOuiDWFr+hTYPdXfAStGTbWyGrHUJmkNCB7iHpox8LMbiOng+P zovsDrTJAdn03qffBDAqIP4f35gTjDQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ks9A+Y+U; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724869; a=rsa-sha256; cv=none; b=T7kXuP2dLuzVZu6AN2T6p7UVL/xcpu88NK6735RZR55SKldFcOSxLUxLMI80W5xxKzBZfE pbkVQ8IvbUEGbYk7vGS/uccJN5JELJx74yEEyxqDubUfoaPOJQL2muQ/SNY3pWSoeiDiGq Mz+GOq0hNcfxurSFWcX3k/RCLycYrMI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=Ks9A+Y+Ue+vODelrx5yLeCPN5qbfxcn+9y7bXNXzDbns+pov0IIQAnLXNktfGVF2kTlpxP 4GmP/ijj//psPkWO9X1cglGez9YUfHIbbRxhOgMmKysEW1j+1F2KB/pxyv6LsHyr3BWtEl oD0iE/GN4wO104ZclxPRIS/SofCtnEs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-391-Zn_ZEv9WPYGg8gX5FrCE0Q-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: Zn_ZEv9WPYGg8gX5FrCE0Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0723D85CBE9; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A8FAF40C6E67; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 1B95E4038A1E1; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.359258017@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:11 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 04/12] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: r7zt4nw67p3atamhhfyt83xguqykk311 X-Rspam-User: X-Rspamd-Queue-Id: 63678100010 X-Rspamd-Server: rspam06 X-HE-Tag: 1678724869-851029 X-HE-Meta: U2FsdGVkX1/h92bF3RwbyAx/xEnoKE2/j/JmRUpXFs09woAj88D10dnz9xInR8o5PLMajSgdU8EqwL+CrhQ1BMpsJuL4oWBtwyBDB7rlKmBJdFNvGnhvXLr9mwEC0Zje9BsWRgP/+etwiW9QgM1Oq+1BcD2QoJXL1ECBcqsxex4mXXc/bDi00MJGT1dQEsliTwAsT5MkGpDU2bQVDVo/hOR4jqfmgugnJ/IWAqwmpuFdD7RVAkRVCUBD4kUQ3xmn3rcpwJwGwNPYBTFBLAy3psWnOmG84Sm9duYU8oITkaTrMSQwDrcsK0x4YL++z3/V/NctVa5XFMJYVatfZDubqqMa26ylHuONspRP27EpO6on+SizeKMKh/zbcBzuYcyFV/YHOmWaE76QFPle+Doye4S9M8AAGG5DyRkxB1NGP49+soD3vX3A8h2y9xP6NlOrWRdnDoNvtdKdskVcDTzyh3IKWh04YE2tORpAm664T7sL37IcfGyNRhMSSY6U2+kG3HQwtCAD51Z/kwFR7tuP1BUrNPVNzOK342eT35UmBSZ5YD5eaSM5i1FLwNkGRz61ELzzGobYqTOpqYAreIrywPqTfiyB8BL1hpOLD7YWaddDGykZyBjGt6FsxlErOkBEfSgNkH6f4PqP34IaUuIUqoFfK7i2gphtFouUqPP+WFv8lz623vS+FUbkIwiwZGOfBewjaAJgDjSEWtDhYf2oobsHckwCxrpMTG8bxMVa8XJs2qIGogya9L2HS59f27I38InGQn8Io7fTRWKuTnZPxZYF3O9oplMY97SIgzI7ICJ2Ir00F6qxyqyAgZa+rD+LAtqPHiXmUr8nCq6jVvooNRHapjuuhRWi5IjaIsFTJ3pdVJMmuyQ6IAYFcHzteL/WKZtfE7Vz/hfSGC1jY5E/Ouo70I3D8aAYS0Cu0Up+lZca8l+1dPf9TgqwERdxyrWhexQQQw+Y/SVDckNQZPF kPnWQ2d/ ak+YZ1yjh+EPY669apzst2fT+D0wYOt1IgrMoqbYXY+Af5qxEg1tTLmPJVoVJIBI/lvnySGsWd0R1w2oI01/F/dmj50/3ZpVeZNLDH0NxcCLl+jfDc/mOospQVd4Cf1Y7o1ePOlbXBO9iallmkIw2VgpaFUQKbwRAxFNaxibOYbgnjPtGA2MwFaa5mrw/zrsbWRNdIPf5e3o7ZOIzSFLXSrb8SpmdTsYIXrrfkN9v0jLqRCoL608JutvWlA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Mon Mar 13 16:25:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36265C61DA4 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B35976B0075; Mon, 13 Mar 2023 12:27:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE5E86B0078; Mon, 13 Mar 2023 12:27:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D4BD6B007B; Mon, 13 Mar 2023 12:27:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8CE026B0075 for ; Mon, 13 Mar 2023 12:27:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6344780C54 for ; Mon, 13 Mar 2023 16:27:46 +0000 (UTC) X-FDA: 80564406132.06.AEE6008 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 8955F1C0010 for ; Mon, 13 Mar 2023 16:27:44 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KP0bl8SM; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=kPryCB3NuXaeBT3zV6tLgUf8id6plPOacGgcwVMtdLJq4qEEiF2K4XE6iIzFV4xX1eFqpV uSMyvhqgxpLzZA09Nm4nZovlANHYEfH9M5bRQoH737X8metV/wjHbHtDxNV3IUz/uncAuF DVUzItk2+aCSIvO6jQwsr9aZN531QpE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KP0bl8SM; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724864; a=rsa-sha256; cv=none; b=AJw9OEOsDFLM/WYjbo1UPzx0q4u8hobH/7BcvFTOitSmgjvk5PP4g7J5UjfhLbccNQIU5H fhh7e1mvuYO+lkmMTD7fU3lcQDpM6OzXQ2fuT3yMwbcHl4Agm0iCdbEU965Tve0fDNrysa fDMLgKG2PfSSkD8jt1zaVB9MxBpiQ1Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=KP0bl8SMX45Pjp83k0TT00YggphA/ZBIKYUsqMyiQCOjZOcu+56LxMt0Ti2k8ynvJNQQOg 3ACwC1KkRwimUdesvaZ9huHihhe/LaD+D13f3PJr3nZtRnjslVs8GN5myev1YfROLcPHQ2 6N5Rc30Xaz5ASXA4l7FO9z/yCbbh5Eo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-336-mFgJK7bbMs6HbOsxZVqtVA-1; Mon, 13 Mar 2023 12:27:42 -0400 X-MC-Unique: mFgJK7bbMs6HbOsxZVqtVA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DA28F3814947; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AB14C40C6E68; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 1EF8E4038A1E6; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.383217649@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:12 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 05/12] add this_cpu_cmpxchg_local and asm-generic definitions References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: me7goafijafk1y34ubidaiw8txdh8qyp X-Rspam-User: X-Rspamd-Queue-Id: 8955F1C0010 X-Rspamd-Server: rspam06 X-HE-Tag: 1678724864-575947 X-HE-Meta: U2FsdGVkX1/FVY/3YCFtLNHkPKz0tCGQA2rdi6WHyddQXCvLsi5teXbEOCXFrPxWRR2NoxGZWNV3ylSN4QnPLHy1Z4fITW4xr/2Mh3jR64Pg7UizKji9/yPCMzM6xdjjM4GZYfGYeCbCLlwcjs/Ka6Nc6kUUa99+T/xTjgvV9Bjwkn2kCkob9KGSydmQ5XL0BoWqwsX9ch2q0jYQUDIEv4j8E0Oa4zfb4A40vDKh8x0j67dxlmx2iTV7LTx2C2TItb27OLBYM145YxoTbGYCK8Vmih38F7BlQ8v94KEw2ijAFSqgsdxKwmTYdaX31d3BUIRA8stySLiy4ELIHK+9XSOPlsU7vVKXSTghaNVoV+/8zBImdXSY7d50MhjDly25FG4VSaT1I4384F0pgkSfNeuh2iCYXVENHGaq59kNvX0ug1FCuKx9vfCIgYskK2tUhnEvfVNBcJWWKZJ3Znd7Sf3edJzXNaX8ZMnICQAUez7SLuBCM++z1cAVhChNuMGpRSMvuwO6xPSePTM8vrUGRqhXPp01eg/9+HxpN6TL3vHZs8sybZbd6nGEiAo7hdOaneNrbCTGoTCFqVtUOs188+E9c1848xRgsKCc2dUKYQBCjdgnyG1ii8mO2dWg0uNvjgj5vHJ93Y2gmmffeUg/JsfogQSTZa0QEcM/AXY7aB2r7rdBZHto6XIsfChFaUALK+hgXOCrpLAyIrUHY8+M/FOuDr79twXn4WEIcG9pPH4t2IILSTd8/lvt/M4d+TnogyNTcC1HvVe6t5VlRT+yEOIedtPIRUFTlovpFbVSGonAXRyRqBev/aTShtliSVB45xVm22ujZjgpnyZhZ8fyZhcxe9gqKuUpt4Oh9XZ4tHuym7P9O2W/drN8jcRZyQikoE1VXyUDudyvX7DMlfbs8nt0vXi7HL5KJZX5zdaM42pUWgqLcDucgp03cb4xiV6qQtEm/NPwQlJUnisP3iZ SR7p6MTu 1GpAjnKwenedWlYlO7P9MQxb8NuZ5X1h4B4M3ReipL4ucM28DxCCk0p8GFapILcLD23+TsyLFGrk2I8lwq/E2MrnamNhTlfzqanPzAJXLXr0iD3v6FhSi40oVK+nhCJTBWiMuwTb6gsUncUey8WazeNHL+G1JNKUZKaPPL3Y5smhpbVVpOPgERz3WUadFsYAwZZg4AGTKgPJHiQIyPYT5erLq0nB7yE0rflGZkmpaQGMvk1phjVRpATQaYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) Index: linux-vmstat-remote/include/linux/percpu-defs.h =================================================================== --- linux-vmstat-remote.orig/include/linux/percpu-defs.h +++ linux-vmstat-remote/include/linux/percpu-defs.h @@ -513,6 +513,8 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) +#define this_cpu_cmpxchg_local(pcp, oval, nval) \ + __pcpu_size_call_return2(this_cpu_cmpxchg_local_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Mon Mar 13 16:25:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7570EC6FD1F for ; Mon, 13 Mar 2023 16:27:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 915DB6B0080; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 878ED6B0081; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 679A16B0082; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 524B86B0080 for ; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 141C4C0C8C for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) X-FDA: 80564406258.17.BAB57F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 1F6E780019 for ; Mon, 13 Mar 2023 16:27:46 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TjaKDR7s; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724867; a=rsa-sha256; cv=none; b=pF+JqLMz4DOGP0WdxeFy8dQtLRCfGG76fwlcLo98D6xI32yV+nZWNcSLXgLHb6RqUj/V9N bPyhAWbt+9qV7MLXYl2Ee+J7jDbpM/FXpCIqr7X+IskpYvZM0UyYXxb+UdJs4vJf6lzclT BSqO1PjiU/G8VaS7yQJwOlWZO0trxQs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TjaKDR7s; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=xYYURvVTeRItJiXmxF0x1ZNPZ6MC9qJm6Ydvr/XVqJl82SWDnr3pOXrkw4mblTuQbNFrDc M+8jXmYuMayexpYJr2BgaWEN/MOXGBNFsaQOAgFCMy93RtqFhES+MQRtPsmg/96az2YHdk IFT7KL3pXTdKDcN/FtJS7yRSrik0SwM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=TjaKDR7shOeXASwjakBW6TbzDUHvhcZd6HyuVAwkqONMSq+qY0UeXQjVeq5s+yGIYRcVwv ZFSmU8MFGav9wDF8Kvbvdm7MLpfrXIQrdqqrbIAnWg6M5/EN/GtLbGB2vdrzlK9T6hbz/g dq1fNlfziG2BCofZuwlqv8n2ovMXtGk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-33-kVC4bivEN-u7qO5pSUEO8w-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: kVC4bivEN-u7qO5pSUEO8w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 322263C0D1AE; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F07B23542A; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 25FC74038A1F2; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.408562151@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:13 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Peter Xu , Marcelo Tosatti Subject: [PATCH v5 06/12] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Queue-Id: 1F6E780019 X-Rspamd-Server: rspam01 X-Stat-Signature: d3zwarqfr5ikf44z9mfpa879myhri4da X-HE-Tag: 1678724866-37374 X-HE-Meta: U2FsdGVkX1/qrtHMetg2e2nTqLBx4nE7U3GSJknAqyjr2FrAmLoY8e2/SXLML9SO2DDkduS2uHzPSmMTKXPBnGBWWKhXjZCbhqz5TNUSaK7AFL3Io08f4Pl5uERQtqSIg4lgZZ4sa7uN07qY+K4vpqmXQ32IgcmL8BZiU7tFxlXhm0YoHJQ6s2TUabSWmxSUQDvk3+T1ieaauvaQ/RB0eBB2csyaj7tBUmm9vJiVw1jC7Bnq7dQ6XIVCxe3sYjsKxGaflfvlCrAVOZqUEgXcd8ppDsbt+1hNlmv1fIBpV6c4bPR3kY6svXGheeVlB/S4ecUbCPegXg2SR1yOFAEyRq7R+088SLZ+OdS5SLolZGVrhX9cAIWMiKQGf+mFkr8xheMiLp9XaVGPswj2dJmrNFn27V9XMsR3Yga6pSXWli15dV6HczuXcEThF9kpaate0tuHTnDh4uGzp7tu8t2x3JwO3fw7KnuPbK6Tixvd2oqd5eZ1hqkAJbwUO6VrEJcclUTtrtmDzotFTlYXZJRT9O6Pub7d7GE8Qv0yCXOvG6J7R9cmA12UPpZLFIQmsq7RokhLhfvWwnXXQbZ+PULOKzYZMZcoQ8BLuliMjSoiaMvMRyqzDjgawc6+K5aAMKZQdfPpOr/s5fuAnjytOzobnA+LzeGkNYri/Jkxikv6xkv7iBlr5VvkYXE2IXLFhaX/5/erZPNQ4RkbmEdO7kFDNHtduYk+IaaJ5BFbdcrTw9D4mbiVQo39JhI3KGMoVEAKOqYJIxSsIsJuVcWE48Q13sA2IJoDZ6wDIyBy9t9wIvIAQ6gMJ2UG17SPAN3RKSx9W8Q8P3OZT9XITuYrkzbLqA7xo9vZsV2qF59bMjdWTAfX3ukRhrtRJik8UxI7au9ANe3vL9ZHTVv2PaiBIF+9rlkf95DORKGV7WkG8jUOwcOwIAIfqujnpDVp+Y4JI79GMulSVshavP/JEzLdIdQ quxeZUtZ 18a9AhOKnysvgt8fG6DT+5EaqFqPVwM797g+h/pEbFLozUJvEwSXKf8OzhtYCeJr+jy4fh0bhCf7NY0IEJ8DLGeXIlDVFxhqQxx0saFKEIXd+fG8imTYyop7TmFSlE+SwAbYjCy10QvGRzoT/Dfsn2ZvRsbhereC7ZEs6ldv++p/LKEEnAzQnnz2po6sWesOZT8Jzshuq+KBSS4X0aINrf4McVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Acked-by: Peter Xu Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -83,7 +83,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Mon Mar 13 16:25:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E874AC6FD1F for ; Mon, 13 Mar 2023 16:27:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 525236B0082; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 439FC6B0088; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 100A56B0089; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E91F16B0082 for ; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9E2CCC0C8E for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) X-FDA: 80564406258.09.0759D6D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id D186F80016 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Dz/0gAI8"; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=HRaKPhoc2o/dyfmAGwpM6cYdTDahEMUEDBWaTmQOBWk=; b=Y98BzGh906ZfcdWe12XEJ5igoROr9T6Inu/3YXH6CgVSkYRxdbUtgsL0tDrJ0Q6P8LgFjd fFKa4J0MbQ8DkqkvTX4yB1b9SS/P5Nym9EKiB6pkeRGak3n+qhcftJIBnL85wVcjg6w9kI piItwDCZjWBniOvLcwkvTLxidBgzHn8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Dz/0gAI8"; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724867; a=rsa-sha256; cv=none; b=0WYIszpuj3uUgorfW/5IslwLG5djnq3d3/WcUW/zQAHuWfZ+5o5J9NknaN5wZ6agfNlqzb Q2tp7PbOKtp8xWLy5Nk8Hjm+Vj0W3mKhQcTXAiW4q0xa+iTZR0iOEBwp6b+svW8Bw6VNNt L4cEZPsekduaJRlgwE0ie8Rrb84pTQY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724867; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=HRaKPhoc2o/dyfmAGwpM6cYdTDahEMUEDBWaTmQOBWk=; b=Dz/0gAI8paHTQ/E8bs/pLVXSYC00W0SwtbM08A2MTPRXWPO0KLKL0rm4lj7ZSr2CGRc5fs empC2CftoGIWCkx+BR4SWu27pRdlRphsGdwggs1LnF5P1hz0z7auO//ODtg+PdQZFUI4Ph PcG98H36XqZ189DaEA88h6CDRNke0oI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-447-m-vIZTaiPO-3HnJowFEtqw-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: m-vIZTaiPO-3HnJowFEtqw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 52AC985CBEB; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F0662492C13; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 29EC94038A1F7; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.433875205@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:14 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 07/12] mm/vmstat: switch counter modification to cmpxchg References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D186F80016 X-Stat-Signature: nw7sid7zy9ndi9ard1g58hubyzfo3x4e X-Rspam-User: X-HE-Tag: 1678724867-839205 X-HE-Meta: U2FsdGVkX1+bgm4V/m73gvxouE2/Tt68/Ynffo7ewI9ykSX6N2uT6JlTedFPWJPZFXPmoLN9QlfcUpzzMERhdfVIabuDKo3xWlw3fRwdtb4DEKWkzRFX8EuHifkHFsSy1k7yyQQlJuSnE+QBrY9QRvs65Y22cuQNB1f0iLE/APTnLyEzsgEIcPTHc14DjBc981SKAErEa/ATGDUbwWPl8K/QuWSVEngnStcGgj7vzJK15mqkvTtpsn1y2kD0OoW3QxvIOsrxUB7LMKbnmNFo+Wrc2tJdcAc/ejTuI6Hxtptwe+6Ahk4X5aGKCQ1+krSaP2SA+WRQxgqkRlfgYTAVJtfT6+M7FhUDc/YdZSL07lxwEUSlRdeIRBlJA70stq+aHgrk3HvFLW9/ctGeNu1zcI3cKpOfknmcHL5J/LEPxArsIBhCC7UPz5dnTRwlGIgweudP8GKyuvzuntaYxwnFyzQfrdBBYtIillpK5ZYy7hSWNvnGV2qMc6SLWYz/gpCrxzIi4Zebq9BW2DcfTfcdZTIV69cIbMIsPJwkqIp1IZKFBCbfQM+rBm0F3KfhwCcdqwcfHU/dXHO6MS8x73uvrO9VFEvuOx4nsixVk7JpCBCphwoXTx9SSg1kUnWaezgl0ePFBfyzwXhav3Bhgw3KVRqfQhLZIYM0DXgaJgxM9X9B+VergnGoNd/AE0jdoHK6PvUFCS9RJQ8wlJ/F+qs8WzKKNLURut1hRZbSmxlF8BOjjsYqvsYvOp9aDdd3ifvj+cJuE2jUBwufYgsDorYqOaAS1RYciqjet1uRLMmrfRvnU+BpwX6pQxYsimX7Eo/1d1h7X8qiV1svR+vi4R+bJk5j1fahY6IHxhePYGDJKqesG/JUg+QDpoTlKop1UNZRJsJ98pdheqCr0XZfWeYd0UIbNBuKFcys9sdUsmh2vcNmyfHUYAxaIMW4g9/eoyZPXlGusvQ3C6zZxZsKOgK vlFc4fgZ W0HMmy4o8j1fwrZBpqzqr2dUZEdL8vbIVS9xdovQoKe2J7qYK4VXGk7e4LYSphCDP5LyizBqfy7BS4FVOtDapYKeTlQstw4yzQp5Q4bgZdW4PEDwW39aMlKiPOQ5BpmIiL4rmU5pTMNhBsDXkjiBi8R495vZ2aDVDM7KETWb7dROhJSOsXlPvAT8vEZV3VX7IdoN/E7nQwxn+hFZ6qerFvs7S1+td7EccKODY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for switching vmstat shepherd to flush per-CPU counters remotely, switch the __{mod,inc,dec} functions that modify the counters to use cmpxchg. To facilitate reviewing, functions are ordered in the text file, as: __{mod,inc,dec}_{zone,node}_page_state #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state #endif This patch defines the __ versions for the CONFIG_HAVE_CMPXCHG_LOCAL case to be their non-"__" counterparts: #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state = {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state #endif To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 115.25 117 1.4% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8628,9 +8628,6 @@ static int page_alloc_cpu_dead(unsigned /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. - * - * This is only okay since the processor is dead and cannot - * race with what we are doing. */ cpu_vm_stats_fold(cpu); From patchwork Mon Mar 13 16:25:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8724C61DA4 for ; Mon, 13 Mar 2023 16:27:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA8616B0081; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D35596B0083; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B325F6B0085; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9D1AD6B0082 for ; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 57B2080769 for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) X-FDA: 80564406258.15.16CE234 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 9315A180008 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MqP5YR+e; spf=pass (imf06.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=eQHLLQ5PDSU2xDgYL5oReXWI8s5CvUHAEWJjGfc5xCc=; b=U8/iUEYWkDc73SeEy67CxC2ms6MQ8xR83d9C+MuA0dXhe5Krs6BLTizXe3EXMefKuYYljo lpn4EZjeIJ2yx7KBtxAuMEyJPFubd8y++LXA7jHFfW9L7VvsC+aYtWnnYlEbZJfq2bQDS4 VSvhSopt+OgubwhtDUfbTOYS4MoBIYM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MqP5YR+e; spf=pass (imf06.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724867; a=rsa-sha256; cv=none; b=woAE1sQ5BY6sOgI12cgjxCxpzEOcYNqFX1Y/OlALLjUEPsdyIDEg/kkbubXbweyRxTlXYr u/zbR0FrvXWsIxr8Vi9Zo33giG2qsBpicadxDzUKJmRPBMgZIv974T3RJrDrtUuWCyIr3I 62UYDh/QJbryHTGTkB2kxah+0wLejtE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=eQHLLQ5PDSU2xDgYL5oReXWI8s5CvUHAEWJjGfc5xCc=; b=MqP5YR+eW+rbc6qGvs2sJS7Q/OhgMxFwoAWG8kKMIsiAttq1YqF6+2wwdSbejVd7adIjmF uIcHR+lqYbWcPEbdOoMRn194u6MD3IKcBkW/ifUW08Uj7c2irM3t7kQuIhVBG7CZE70i1O vAttIkeQaYmuvDG4smOK9mq+grIDLLY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-447-fKD2GMTTMDiEEVthat5a4Q-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: fKD2GMTTMDiEEVthat5a4Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2294D811E6E; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B775E202701E; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 305264038A1FF; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.458451052@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:15 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 08/12] vmstat: switch per-cpu vmstat counters to 32-bits References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: grmsyhds7trhztta641s6eiyost8exam X-Rspam-User: X-Rspamd-Queue-Id: 9315A180008 X-Rspamd-Server: rspam06 X-HE-Tag: 1678724867-554504 X-HE-Meta: U2FsdGVkX19kECQQT4mSlUDw/qDwti01iWmtez30iilL00x8LZ5nmfjtmcpcppbBUjZYiViWX/DmnkkG1Bd2vDuSfV615FGRn1Lt/sk92mipptUd9dW0HHnnFrwvRaArw6VZFkbjaZttxJhmUHGJxy8Ln7nBT/kWHZxbSYHL+fzGDbq6c53rCNORA9PaWuKQhpVMzZlzR4MrUvRE74nLrsRNpCJaoeZ8irWln5OYVXXUjfLi3F6DLQviIGivMZW9uh/Rc2KX4CMQRqB5Jb5tTjr5vxCgHfrfTd/s5YIG4mUPWrk5OaxWKKJzBblaznbxLvgOLdWTeJma4aUF2kZUDitUUJE6LXDsh3rIAMw9424ypB2jOPen4jyYJkOB+Y7NzSzEI+98qPaKkcegztNMmA5V6mdXG9Bt3pJsOnH0NxssTgrNJ1Im+pzdRMc7d3LjH/e1YF417uPIuZtxg4l0Lh+05ZGa3fACIuemIwAl/8Pi9TdeZqSQcX7JKfmtm0D7+MfIywQ6HE07nRCB0YBc3iLKYvdqDf2mrpd6SOaXcSe0B0bOTx/UDOC8vKeDnmKiWa3GMs+BzeR7bGa+5VTRCIz32eBdpG+87e7WDpqf7Jo9oLvIaPkjL3qBdyP29HZ+Q45uj6x1M82ZTlHO4yuqQNkPEsB4N5mFCKTWBCcuuccUK4XuxuKWmBOfRkII+Lh211gr35ty51yoBF/jBIfdABUwlpQIVnKeebDSZV1aFwzAGSIJmkdLROl19Wqr6dngZM4CjHnnbKlVamrv4EIiEPMFzuHmBiVJA54E+ESMgRAPLOfCB7CrMUWvPKl0Vs3AyNLzv4/OWRPloK6P9fO/qIbLoccFUOdeBDkLsa7bnBJDUBOWbuYYAMpRVvKIRxM18bV47lT2Js2+OpRphuVqu2j+QjDZVXCDyWVxTCu/JJwI54EGpdwY52UlMaAvlxhTr7XX+NQfxef/WJn70z/ vsz6Vo33 uYPEnkq+nHpzeIuql4kKjMrQMM2mblkS7aADWGNW5xx4dVwNHNeNIzHY7B0TIl1MZL+osyJP9xc/wt8izno3atnMIbtCfTmXOWFxG6r6r2WWzPbVCwRD1O5Ud4BNpGM1QtfYypgNntn9gKEKI0IJKzK808qDQyfeEfjOeTUwqMjFSuQEg6ktyRlWaJON8fzfhpVunga4y6gQs8Q4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures only provide xchg/cmpxchg in 32/64-bit quantities. Since the next patch is about to use xchg on per-CPU vmstat counters, switch them to s32. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -689,8 +689,8 @@ struct per_cpu_pages { struct per_cpu_zonestat { #ifdef CONFIG_SMP - s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; - s8 stat_threshold; + s32 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; + s32 stat_threshold; #endif #ifdef CONFIG_NUMA /* @@ -703,8 +703,8 @@ struct per_cpu_zonestat { }; struct per_cpu_nodestat { - s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 stat_threshold; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -351,7 +351,7 @@ static inline void mod_zone_state(struct long delta, int overstep_mode) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long o, n, t, z; do { @@ -428,7 +428,7 @@ static inline void mod_node_state(struct int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; if (vmstat_item_in_bytes(item)) { @@ -525,7 +525,7 @@ void __mod_zone_page_state(struct zone * long delta) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long x; long t; @@ -556,7 +556,7 @@ void __mod_node_page_state(struct pglist long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -614,8 +614,8 @@ EXPORT_SYMBOL(__mod_node_page_state); void __inc_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -623,7 +623,7 @@ void __inc_zone_state(struct zone *zone, v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v + overstep, zone, item); __this_cpu_write(*p, -overstep); @@ -635,8 +635,8 @@ void __inc_zone_state(struct zone *zone, void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -646,7 +646,7 @@ void __inc_node_state(struct pglist_data v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -670,8 +670,8 @@ EXPORT_SYMBOL(__inc_node_page_state); void __dec_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -679,7 +679,7 @@ void __dec_zone_state(struct zone *zone, v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v - overstep, zone, item); __this_cpu_write(*p, overstep); @@ -691,8 +691,8 @@ void __dec_zone_state(struct zone *zone, void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -702,7 +702,7 @@ void __dec_node_state(struct pglist_data v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); From patchwork Mon Mar 13 16:25:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DFF9C6FD19 for ; Mon, 13 Mar 2023 16:27:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A738E6B0087; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6DC6B0085; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 749C56B0088; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 431C86B0087 for ; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 06DA61609C5 for ; Mon, 13 Mar 2023 16:27:50 +0000 (UTC) X-FDA: 80564406300.08.454AC1C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 3FD531C001A for ; Mon, 13 Mar 2023 16:27:48 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H9S0VYSU; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724868; a=rsa-sha256; cv=none; b=rUad3RyFrC9IdRla2cL2i2Vkabnl1W/OYMYDsHu84obX75EpCK3C4120PfV1DA1WMMdiV4 79NmH9D1dGq6hjqydgfMmbfZhk7hckkOV70P1W7CUsSZx1mhBf7BSG+Jp14mFW0w4sO0xH +921p+z3W7TGKk/esWkeBBky0kV0hL8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H9S0VYSU; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724868; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=/KQqYUQDwTZyYbpRZbwn7x1mgss03C//4Fvg72sL2vY=; b=dYrFEZnqT85h0fTUZxI2oV3WL+toL+WRbNZPBJk3CwPnelR4Pwzqe0X2lVB01VFbMWVDyD WG9NbAKC5AwkyFrYj3RHfRdPtfi82v/9rjStJkj2Sm7gvHxNOqHx4rdLgsVbz+yK787dDQ AeNXjgl+mZodZ2vaA3fV+5XXttfXZ8I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724867; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=/KQqYUQDwTZyYbpRZbwn7x1mgss03C//4Fvg72sL2vY=; b=H9S0VYSU8tZO5hcZveJ9+GtNnLoncamRvm1simoZPcL2HZgWbjXvz9fw/7yKl6+l4f+lau UoMG2/YqqX1IFovZxYPnX5kg+nVSt/ybXKv2GKqA4jJHnjTxIjujtBTT2DygNl0pcm++KQ 73XJhmpXUUcnW11YCqziip8eDebJxdI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-391-DQQRGpshNVW6Jcd0FnfZLw-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: DQQRGpshNVW6Jcd0FnfZLw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0474C803481; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C9959400F4F; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 348BF4038AB89; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.482796394@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:16 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 09/12] mm/vmstat: use xchg in cpu_vm_stats_fold References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Queue-Id: 3FD531C001A X-Rspamd-Server: rspam01 X-Stat-Signature: ii3a4ac6rj59huuj6q1qs19gidmzwynh X-HE-Tag: 1678724868-409556 X-HE-Meta: U2FsdGVkX19SWSc8GCoL+S0ZP+8UZq6VvxSBVfMjbL/Y7BMuKs6/EfhnNj4ODI5p/tEP8hcs1c+gxI7A7zi4Lo03ZBcz7sEqIrSJIf79TFEN28fhexBoUATBR5qq9aHeNroWdjpU/4ldBUas41Gik5Xu6lCVY9gBSf5odpEEpsIlLeaCkn8LDee7m0YXZCr727jnJVKY7fqj5DMml0yI0qf6syAXfk1+EFu93kvQfmLONmO+P0RZ0Ar5L6dLhYfJ35o5HerpV8UgIJnK3LQySL9nXfnZlGpHtKCP3hQzrF6V4GD5fRyrFgUgJEhKpzlSYQGEuz4u7nWg7/6+72UmC5lJdeioRvWxRvibPDOb2bXVN0XNtXNsCFzbqh1sB3Lw11ekYVkxPKHvoNGJYxXoQlXHs3Km8oxzhl1ziMLS7onY39FF7Zi3ZGwb+LDCC95pntIfuQtM5VeTHP2v8H7qk3RzCiUjjvADriFljce+eId5KAq5IJgZurkpnJvwXDbkjkqe3/hRUvNOd1n250yNLEuXtnQPenadjInYalcAxNvcCIFdgBj4BW/RrONbUuB7XbRKHMCrVbeBPM58oYg5HpLbnMOAcC6eFwMnlTTeaefzuZaqhC5582p0HFwDtWhRq4qvzAoBHgKNHwBApYdwzMVmHyMYh41BaNFNTbXWE6nC2m8YKjujw1YxQUaOWM6VBP9Go3z7QugZhpnK2gdteO1kq4p6WEtwhKiHGd9dmG55DnsUn1Y718tidU+Q5iBzEce3qqIVJxCglttT1wS2YFkHWRM38nD+obXiPM637IG2enyFpFadEu/HeZyhCw4EAY2NaUpBCcUzOVfJXEYpSrbVqpqvuQXh9kPAaKj82t1lWsne92WaqQPSQCekcGR8ZvBHh2aqL5w0oFOaejIjUXomfKkBnzaf8TLEL4Lo7Yjpay6BGD8zD8ED1UfLVnVkFioIeYmNdExCmLic5sq s2IL0d0w BOQ7JHvqcxpR00dmXm1ngoVaTWxC4mJvovW1bLaYhxeJsAk6mQChq4Fsi7yORPnuMIXP7AAUj6ot6hsb8sY0zyrBZ7JBpocNuaXGAKZb3QOs481Ztv3P3GhIvzJALJo74oIRaPLiPosDF78W4utZZrWjK2DzKihe8jQGiysdKOSqtxG4xtBB6xEm7qxuYyROXkboU7xoaT5Sq51TPpOVTv8OlQ53ZsDjNl++oxCUL/W3K7PttVrxtRyIIA0pw6O6mJSN2Va5fFzI1kOvNOjOeYVpiZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use xchg instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -924,7 +924,7 @@ static int refresh_cpu_vm_stats(bool do_ } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -945,8 +945,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -956,8 +955,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + v = xchg(&pzstats->vm_numa_event[i], 0); zone_numa_event_add(v, zone, i); } } @@ -973,8 +971,7 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + v = xchg(&p->vm_node_stat_diff[i], 0); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Mon Mar 13 16:25:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6ED4C74A4B for ; Mon, 13 Mar 2023 16:27:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E7EF6B0083; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1990B6B0082; Mon, 13 Mar 2023 12:27:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF37E6B0087; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CAD786B0082 for ; Mon, 13 Mar 2023 12:27:49 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A53CB1C640E for ; Mon, 13 Mar 2023 16:27:49 +0000 (UTC) X-FDA: 80564406258.11.9285519 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id ED18E20013 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eEBEbGe3; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724868; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=tZRdfptRFY5Vlp762UsZ6/anX9gl/kIpoSRft9xlHEw=; b=8FshL8+1EPxugY1bJ1Lp0sg8uUEz91W4vI+gjvmS4vKkNs7EZeCGR5Nq7WYAVh6uvjg3hY 010m8CmLJoCE2YHWCVqDlrZIzG3h8JyDZdxk+t1DE12N8vkLK1EluPOcgj1iRfh0E5iJep IOt8VfkSqrhZGmiRtjBAzS/IoTck7XA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eEBEbGe3; spf=pass (imf03.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724868; a=rsa-sha256; cv=none; b=lA77vvUO7bzmNFg2QxuCSf0PiXY13KW4J4FOwZBsB3SovVQrdID9xEi6z98GZK6hUkO7PU k2VqM73tV8fUvwSQer0GLmcfbMzBIfMrt5fqTZtqHxxeiJvuWkyg6LlYVEvEQou2pC5QQH GTm74gEc11sGWSewElAsar/OF8X6zdk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724867; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=tZRdfptRFY5Vlp762UsZ6/anX9gl/kIpoSRft9xlHEw=; b=eEBEbGe3YxquFhDpHqybxCrg6Y4Zhd2kEZqdfD12QjhZkrGh9il/u86kOK48NLK0a0i9+6 sV3CYF7pKsj9+fAHILn6u1ZTDfvJoMMlNtpEFuCiQXFYMj07S9Jyr4tfkHm+PXpMI0wAAI KVu/tvWvwkkywdLQ0QF+h0Ite5CgqOg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-126-Z4BMmYaWPeK5kERbohyfSA-1; Mon, 13 Mar 2023 12:27:43 -0400 X-MC-Unique: Z4BMmYaWPeK5kERbohyfSA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0022B3814949; Mon, 13 Mar 2023 16:27:42 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C6F47C164E7; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 38FD44038AB8A; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.509267576@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:17 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 10/12] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ED18E20013 X-Stat-Signature: d9mjkqhuniis35ywkjp4akzy9qbf7cdp X-Rspam-User: X-HE-Tag: 1678724867-476571 X-HE-Meta: U2FsdGVkX18WMsdB3Q/49SxqeM0uDHqF7bwv/kCYdVOnoFvGggc0d/iL6wXzQABQUzou1y7gzpI31U2DPOz77xr6mN9bSJ8ASwqd3RYfd80KT9oZwEJu7Y2tOVnt/g781h1bk3strPctx2eyZitYY4R9SItGP/o/pmXk4ols878pwnO4+5t4KXRWPsjIbgZz3s10k1lsQ8n/Zq05SdX5rzpnLuTebHrdnuJ6bdn99gKm0K3tHS3PyvPvNoOVqHIvdLFk6CCqDFlxoewMhdBmVEZw/j7kj7AO1QtPJ/adCTlE3aeK5iSmimecguQCseVZUeHnw5UTUcurblPFZw96rr880WvVwtmIftel2ataazKCwt9o4lWH786i4akM/gHo0CsFgnjaceP/JxMk+UazxwablzBIKZA2+T+exrVPRBOSia67e7cVtvRbqwXa+WTpuAW1MeWto/eiTPI+ldc+GzDnuf0+YDyuPlEjuvhDkLfVpy/ksvx4JTTuQIjMH75ZG95ackZN4wsdZtAOLQMaSrqHubJhgPXFJZZsv+ThXMr9Tn/G+j4RB/cO/y2haulUWGEKsSNlPJw7bTFdiG/QKdAmmWWHHAf/bY9OhqYGmaXO9/KYE0pOhxRTCBYvRSqHWDKLOwAU01a1JQWKl2mdLgTSzmNKIzYJEYYFrbdGciAS+mFlbQlvUL5d4F0RwitaQK1qN7SKx+sZDoqv48zxtZYjT51hVs2CB8jUS38p0d4FSYbBAnhv6bMxptBw7PuFYQ42Ox24irUtprBj5v3NBirpxqvBlWimaPhXqFeN2tPRdvZMcB3v7s/HmQUTSQ1rR/wtQYvnBCm8W5+y+0gWs6mnzlRHfDPwTJ/XcTvvM26dx90b78orEOrPzKfn305CeKPs60YSwEM1oY0BkTDsznbdo0rImjgORN9uIONtg3cIpZv0RSFrq0MVpUml7zEYNTXLviNeaUpC25al88X yABN1VFP 4RDMGORqdDDFGytOAuUeMbgBFFzAuvOaa1Ob/+BFzK1GVvRtAIuZ6hkcISXllutFnStqks8vumF/la4qswoY6MINYx586zFcjzCrDTu/n5yDSWB5h51GFNFol+RM2NTmWKWEHtoIubKAp/nlC0NogDocIF+EA4GLmp2HApKGavICkKbBLJRJTmqMjPa26UQAzBusZ/lfJdC+YDlQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. This fixes the following two problems: 1. A customer provided some evidence which indicates that the idle tick was stopped; albeit, CPU-specific vmstat counters still remained populated. Thus one can only assume quiet_vmstat() was not invoked on return to the idle loop. If I understand correctly, I suspect this divergence might erroneously prevent a reclaim attempt by kswapd. If the number of zone specific free pages are below their per-cpu drift value then zone_page_state_snapshot() is used to compute a more accurate view of the aforementioned statistic. Thus any task blocked on the NUMA node specific pfmemalloc_wait queue will be unable to make significant progress via direct reclaim unless it is killed after being woken up by kswapd (see throttle_direct_reclaim()) 2. With a SCHED_FIFO task that busy loops on a given CPU, and kworker for that CPU at SCHED_OTHER priority, queuing work to sync per-vmstats will either cause that work to never execute, or stalld (i.e. stall daemon) boosts kworker priority which causes a latency violation Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2043,6 +2043,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2062,6 +2079,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Mon Mar 13 16:25:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4CB4C6FD1F for ; Mon, 13 Mar 2023 16:28:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A6F86B008C; Mon, 13 Mar 2023 12:27:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40DC76B008A; Mon, 13 Mar 2023 12:27:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 219BA6B008C; Mon, 13 Mar 2023 12:27:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 082B26B0089 for ; Mon, 13 Mar 2023 12:27:55 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B15B4120C35 for ; Mon, 13 Mar 2023 16:27:54 +0000 (UTC) X-FDA: 80564406468.20.797A4D7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id C449D1C000D for ; Mon, 13 Mar 2023 16:27:52 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KWaJI1aL; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=BjYqqe7MvmRoNdsZ3mRmEvGJpIWPHc0RoB3EQFJD7pE=; b=4vVNFhQypUcsjWAGQx857lMSvQde/H1MEO6NHW8JHFyAm18C+A4v4YcK0y66uEYZySkuo+ E7SpcMTyKtJmoIZqERbegpSkFILYzGf+kOHu9VIXpXU2aKNnKzscHKIYlE1msqW4SitvcJ Xpu0Hz5hl7D5FcMassYGNY4dAfWJULw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KWaJI1aL; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724872; a=rsa-sha256; cv=none; b=EwqW5fExz07t/MgrlwDkGJRLRz88BjUaSikPV4Eh8wwPuDy3MOFHn8aBUiTv94uOWg8eSt 9qe101yu+F1KPeOaIvThAPfJCSTzw4L/jldeR1YXki7Vpw75c6TINarpSRDl+aUq0HaETd YybQOSt0LjQkmCo37VYAUdLksEOZSx0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=BjYqqe7MvmRoNdsZ3mRmEvGJpIWPHc0RoB3EQFJD7pE=; b=KWaJI1aLPamKnFGLKcQBFW5OM6sN43ZFI3xSGbQLCNqpSZ5zHqQEv2bIIwoiDlGfdSxnVe nEJe31I32yncCkBTquR1fqKGAQMOKCY8nsH1SozfF1m+q0TDZQehRdqtBiPP+mANiTVVF4 83oh2g6/GK9iQIPOvWm/KJ37id1PLEk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-391-PN2FRtEPOj6OfPOguGKUAg-1; Mon, 13 Mar 2023 12:27:42 -0400 X-MC-Unique: PN2FRtEPOj6OfPOguGKUAg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F15C23C0D1AC; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C736440C6E69; Mon, 13 Mar 2023 16:27:41 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 3EBD84038AB8B; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.535939407@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:18 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 11/12] mm/vmstat: refresh stats remotely instead of via work item References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C449D1C000D X-Rspam-User: X-Stat-Signature: kz167nxuco4416hckgd8fk4zxbgabqs6 X-HE-Tag: 1678724872-695050 X-HE-Meta: U2FsdGVkX19PqTYkPOIBG++5FBkjJ6V3XjD+jRZH2w2u5pAaQtFhxHVOmmoiUq0FTtbdmPQiotWufxOAQVsonWZTwGhURmtvDD0cXWvLd+EGeGArXSc/IktV2ruqpxL1nMroqoXuQmf7bsK2QdoUejMHYsjvHaN5x1r2klt4Ht555A7SulyIhBR2FF71IADLTWuxJN6pShDo867YLUWzp/Eaw7t98ezh4Qx+ygV37OMDuyCf7JRbdsDzSj333RkRGdkqXuhoJqN9zkHlJQyZWIfJ47RDTgkKzAhN7ACl48c2oF5yYUXBXf4P6W70R+UcadApwtg79fi1+BSrBKj2MOgIBXDRILv2Nq1c6T1PV62pQqhdW3AvA2OAtGlB7Ts5ns0mREgYbzFxQmSlFbMvvPjLoSpduixC4b0icxY0wJjxJ7IEVHRZLU8Gxd7Hl5KoSwvOUYXde2ZPfaGJ5cZkR/5DyyUjkjaVQuNi+TLqayoD73DzeJSudLSWxEWcsx3FzIzQgrlCGUa9V1H/Jme6pxMT7anO1bKfdjKW5IzHBYuaOJ44jUpPrrLvCI4npRJDWSwIEQ5xMALtBWTNnfToybgm4X/r5CcRmGaDezlg/Ymdv3wrYPArSERIlZvthNBt4ie4RWGKial6DIS7TApnJkKOfb3/2ZEOUTKsO1PibThD/ljJKRNskxACbkEWmqfQckqll0evvbMqyCpYfy8ZY1FLmnqd8x7e1+mFsjRgVTlqaw3HvXGVu7HVsGLoqrZG5PUqVQJiiDB8S5wZJM23amrMKQ7Fgc9zC9JM9YHm/CwbMwT9mxMy0WNynWt7OUacsss+qfhK4zr6EIi75C6RnWTq2BRatJhLWzUyiWs7HauRF9FFMKtzcERyggksMFgQoskuuJF5CaCqQor01uIR7x4SIm8v1r34jV6IoQc96mmH+z3HLLRtu49xRnsLu+vAPWK5iMMpiwpj/KN1B+9 uA8DagEf AWL3aEPeQleTd1jMen1RmQhFDy9zyjGhQBC6Kr1jfu4csvMcUqgHTt1s3R2rNMYkQIDUqIoTNPCLRCPuouDh9BiKHBsMduDeOMOk1A4rlru7ySJTcCLwmkJ87eLgrpEkpbAqbkov4K47925HFQ1TymChgWMCQc87wcUJ9wwkn8pYvhSGJdc7Xh/B6HJy8O0nRY/a/FKqbEfHfxJI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1901,11 +1901,20 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(true); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1925,7 +1934,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2045,7 +2054,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2055,7 +2064,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } From patchwork Mon Mar 13 16:25:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13172848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7AC5C6FD19 for ; Mon, 13 Mar 2023 16:27:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 654836B0078; Mon, 13 Mar 2023 12:27:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DCE66B007B; Mon, 13 Mar 2023 12:27:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 431106B007D; Mon, 13 Mar 2023 12:27:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 310076B0078 for ; Mon, 13 Mar 2023 12:27:47 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0367E1C63EA for ; Mon, 13 Mar 2023 16:27:46 +0000 (UTC) X-FDA: 80564406174.30.34F9149 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 442B8C0014 for ; Mon, 13 Mar 2023 16:27:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RfO0qS+x; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678724865; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=s8sJ/cpBUh6LGMZ+lcAYSBuSqrBbcVYVRwc+yqWmsjw=; b=BNznObni2cY4HvTzl9jFk6gSb0H0GLIUK48wB6ynw2qy41P+V1MzOs8XlD6IdD6Zyu1OQm NZJi/Z6caKNcZM748atyaNv8vp3RmuJQPg0WaBdMhwTsdwCVN/9RbsL/K9DKHStrBkYmcT GZ62yvn6umgKpf5+wMKYAhpz6/tTRhk= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RfO0qS+x; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678724865; a=rsa-sha256; cv=none; b=EKmrPj2AfdYPEE8MvxpRFA2ZgirDp56+1fE9O0+YoExWukx+oNpvXTzzA63DsSdrMCBP1G i5D/cGwGpWvqbouQ9E3W2bWktV6pPLGfune5UorZrqUo57xeGpWi3XcDS96bvMtNVP7G37 VoAnT66PDRcLmBnRs6ZPlnFWDdfH3Q0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678724864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=s8sJ/cpBUh6LGMZ+lcAYSBuSqrBbcVYVRwc+yqWmsjw=; b=RfO0qS+xV5J38XqTfnlp/Js/l7i23ggstsp6THXDGpNf1+oJIDlDYopHew6fdqpogSSe0s BdSESmtrmjG1KOYVJj+FUahoYdqSVWi3cg8TqX/gPNm0s4r36GgvDq/heex9ESKt7skBIT aCRg4TM9V3ZjGxwYeWTqwtF1bPgaLsM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-ZMuT39A6OHKSVs7hfn-Tvw-1; Mon, 13 Mar 2023 12:27:41 -0400 X-MC-Unique: ZMuT39A6OHKSVs7hfn-Tvw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4280818E0045; Mon, 13 Mar 2023 16:27:40 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 15EC3492C14; Mon, 13 Mar 2023 16:27:40 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 467DB4038AB8F; Mon, 13 Mar 2023 13:26:44 -0300 (-03) Message-ID: <20230313162634.561005765@redhat.com> User-Agent: quilt/0.67 Date: Mon, 13 Mar 2023 13:25:19 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Marcelo Tosatti Subject: [PATCH v5 12/12] vmstat: add pcp remote node draining via cpu_vm_stats_fold References: <20230313162507.032200398@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 442B8C0014 X-Stat-Signature: tcdinb7i1gxx5r1hphmgadyeohn8wzfg X-HE-Tag: 1678724865-895946 X-HE-Meta: U2FsdGVkX1+BNQBYsu1GtZB3/jOPOZZJlRtqAaJpnpg3VGi3CbtZE4gWaa93e78w37CvlXoG81ezvujSILinSAn6FgjUl7o4JdIZQh7BzX9ab9N2t9scfw9pK6Aap5sP26r/yxA7uA6nTvhs0Iwma+xrqyWTpCtzsekJVRJj2SoieVf/C+/DML6YagwqhVUsXxi3jFUGE0FYHtdnCMJFAiit2dFtsdEVOG0unN4JZDwEGiP6H1RCVjEnaU524R31bTZ6vdUhugja5MZO9DbB0xl5HkXZW3JIT5z1iQLiq+5HfezdquB8p112SYHOTam+zxMCIfnnHO9vYw9Kf50JFvo4ju+DUWLxf1ymi0TtTyacXFkw7SHhpnuzMjYFckcrnKvttVzXVu8psNoma2gWb4sxbVcS0P8TbF6aNGmPJWWHKVvaVtZR6fZp2cge3NvRi1x3yrrW3wa8NixI8htC/hj5cdqjHo6Vw5htXrZuPhKFv3Zo8gVxr8dnOJ/Foumx7re2xa6fZtQtP+D/z7XkHhcf5/5NcZkhACxpOGFD4tJ9pD2H5wmkXemfaGJ2vTV7HT5mVXtMeO4TdPNbfEWyJeGtrBr+Q9gieoRY1O1JY5CUwr9YyTw2Wswecnh95KKJJT94ZD4CpNAUx87/uDOGFaEkvhGflr/UON8ldl6sonOlEsbAJeprJCRn+vBUh3S+jeA585eLaDzZaoicIE6kZx+ityD8+VfqBz/zY/orq8vQzNt0TefLS59poBN6H/V3yPctgNM0SZmSiHiqZJHaix6gzktHgdyy5BOaK6cQv6sGYaUJONZXTCJDAJOwUR1TDGOsQejtQgayeRzLWoYpes2Y183HJf6J33Mc1MrVMvkl2XxHYB5w5RZhA7rNpK8feI5LOQGbAFjyCR8EolGE5Z9mk9x7NZLRCbOmE8PuzfHOqSfoJOfUmZIFsvMYP6Ml42xCtXWLH2AByfiI43O cK93CRN7 rxE1cOT9MEPBIzGk3zbsCx8HTp8/99G7PKt+yKpFHY8TUItm9QjQu8lQAxNyd2p2K1xkXZdeJVELHEfoH1YBFqGebKWTPxL3ThGxCCY/7b9okyiqKdoFYwEEC8MoluLBGDZJ7i3tr9kPcdtzn52CUYdhVlv9u8WsW8Cs57TmLSfwOqbdkAMSnG66Gz6MUoNhk1wwXpt0V91+NJ7HzLFtrEHcyOeuXkXTVU6SkTnsPs5P7t0aFuvalJ3ayoA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Large NUMA systems might have significant portions of system memory to be trapped in pcp queues. The number of pcp is determined by the number of processors and nodes in a system. A system with 4 processors and 2 nodes has 8 pcps which is okay. But a system with 1024 processors and 512 nodes has 512k pcps with a high potential for large amount of memory being caught in them. Enable remote node draining for the CONFIG_HAVE_CMPXCHG_LOCAL case, where vmstat_shepherd will perform the aging and draining via cpu_vm_stats_fold. Suggested-by: Vlastimil Babka Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -928,7 +928,7 @@ static int refresh_cpu_vm_stats(bool do_ * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ -void cpu_vm_stats_fold(int cpu) +void cpu_vm_stats_fold(int cpu, bool do_pagesets) { struct pglist_data *pgdat; struct zone *zone; @@ -938,6 +938,9 @@ void cpu_vm_stats_fold(int cpu) for_each_populated_zone(zone) { struct per_cpu_zonestat *pzstats; +#ifdef CONFIG_NUMA + struct per_cpu_pages *pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); +#endif pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); @@ -948,6 +951,11 @@ void cpu_vm_stats_fold(int cpu) v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; +#ifdef CONFIG_NUMA + /* 3 seconds idle till flush */ + if (do_pagesets) + pcp->expire = 3; +#endif } } #ifdef CONFIG_NUMA @@ -959,6 +967,38 @@ void cpu_vm_stats_fold(int cpu) zone_numa_event_add(v, zone, i); } } + + if (do_pagesets) { + cond_resched(); + /* + * Deal with draining the remote pageset of a + * processor + * + * Check if there are pages remaining in this pageset + * if not then there is nothing to expire. + */ + if (!pcp->expire || !pcp->count) + continue; + + /* + * We never drain zones local to this processor. + */ + if (zone_to_nid(zone) == cpu_to_node(cpu)) { + pcp->expire = 0; + continue; + } + + WARN_ON(pcp->expire < 0); + /* + * pcp->expire is only accessed from vmstat_shepherd context, + * therefore no locking is required. + */ + if (--pcp->expire) + continue; + + if (pcp->count) + drain_zone_pages(zone, pcp); + } #endif } @@ -2060,7 +2100,7 @@ static int refresh_all_vm_stats(void) cpus_read_lock(); for_each_online_cpu(cpu) { - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, true); cond_resched(); } cpus_read_unlock(); Index: linux-vmstat-remote/include/linux/vmstat.h =================================================================== --- linux-vmstat-remote.orig/include/linux/vmstat.h +++ linux-vmstat-remote/include/linux/vmstat.h @@ -291,7 +291,7 @@ extern void __dec_zone_state(struct zone extern void __dec_node_state(struct pglist_data *, enum node_stat_item); void quiet_vmstat(void); -void cpu_vm_stats_fold(int cpu); +void cpu_vm_stats_fold(int cpu, bool do_pagesets); void refresh_zone_stat_thresholds(void); struct ctl_table; Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8629,7 +8629,7 @@ static int page_alloc_cpu_dead(unsigned * Zero the differential counters of the dead processor * so that the vm statistics are consistent. */ - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, false); for_each_populated_zone(zone) zone_pcp_update(zone, 0);