From patchwork Mon Nov 4 10:50:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13861232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5EA1D132AC for ; Mon, 4 Nov 2024 10:58:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B5A06B0088; Mon, 4 Nov 2024 05:58:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 665086B008A; Mon, 4 Nov 2024 05:58:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52DC46B008C; Mon, 4 Nov 2024 05:58:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 373A56B0088 for ; Mon, 4 Nov 2024 05:58:28 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4EE95406FD for ; Mon, 4 Nov 2024 10:51:07 +0000 (UTC) X-FDA: 82748094618.15.147CEC0 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf26.hostedemail.com (Postfix) with ESMTP id 8AB6714000D for ; Mon, 4 Nov 2024 10:50:41 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=bqSJbLly; dkim=pass header.d=linutronix.de header.s=2020e header.b=m8UVPQXo; spf=pass (imf26.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730717417; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ktKI//sMmnnCjs54hJYaz5s5ts5YF9AmvSNM3G4N7f4=; b=o4zd+eVplU3brRes4F0Q00fjKT8BYX29FHSMlBXtJSkDyqMZmR0bTAJrdygiRF3f6Rpq2Q I9RRz4ccoJK1+iw3wzn3OUx9/nAPl7IKdYfJBMPh8U7VAHkLO1ru6BzDr29Y8C8bOo/2nS HTI99ZzK7kk67cC0fNx4FMmzmGhbDq8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730717417; a=rsa-sha256; cv=none; b=1Kjwh4WB+KSg9N5PM0UJ8M1vbvtQ6NutJXWMAQrU4lj78ttGT7NNUQ7EgeKbLMJiNszJF2 FNABLg/j1qLQ9/9oI/VkEPAFz8h3su4Nk6cBwgIEZhpprRsE7f/fH9kNUxhdX99dEOBrvc OjZzb4I4Y07ApMystesfW+mqr4DuO+A= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=bqSJbLly; dkim=pass header.d=linutronix.de header.s=2020e header.b=m8UVPQXo; spf=pass (imf26.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1730717463; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ktKI//sMmnnCjs54hJYaz5s5ts5YF9AmvSNM3G4N7f4=; b=bqSJbLlylEQ2AcG6qDh3kEVWTHGus5P6fdfo3Jrlw4xmNNbnm4xzJICwI8aKw9mgkdy6eC PNT4RHu3ve/GNLutCIO+wHuBdEdnSPgb9t4YGWxXb4IKEhiLRtTxDI5AkPiyx6u7tYJs5J Amya4bviFotCVNltb4rQE9pfPy+NHyLzDM0ecLR2KmJEleJklq2Y4rqN8efsC4peNZrswZ OA3104g/Z6icuUhCawFpcORc0JBCH1Z8qsqCRzkjzprpzvv+oARYJ8TKPR9Zm/DLHRnFpJ VImRB7WxDIIv+zrNOUKnzVdnoPZnquOcihv0hevTweB8DcpiYOMFKtGMBZtorw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1730717463; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ktKI//sMmnnCjs54hJYaz5s5ts5YF9AmvSNM3G4N7f4=; b=m8UVPQXoc+16YCShWzyN8Fe07NpM+kXCd6OWtfkqM2d5/axty9p0fFzsP0yXXHo85GL0i3 R4jFWGF7jhrPunAw== To: "Paul E. McKenney" Cc: Boqun Feng , Vlastimil Babka , Marco Elver , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, sfr@canb.auug.org.au, longman@redhat.com, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, Tomas Gleixner , Peter Zijlstra , Sebastian Andrzej Siewior Subject: [PATCH 1/2] scftorture: Move memory allocation outside of preempt_disable region. Date: Mon, 4 Nov 2024 11:50:52 +0100 Message-ID: <20241104105053.2182833-1-bigeasy@linutronix.de> In-Reply-To: <88694240-1eea-4f4c-bb7b-80de25f252e7@paulmck-laptop> References: <88694240-1eea-4f4c-bb7b-80de25f252e7@paulmck-laptop> MIME-Version: 1.0 X-Stat-Signature: cytmzcpuowx1kr5w74up5u3o64axuuup X-Rspam-User: X-Rspamd-Queue-Id: 8AB6714000D X-Rspamd-Server: rspam02 X-HE-Tag: 1730717441-927027 X-HE-Meta: U2FsdGVkX1/1JULs0sW+YfRb3nhjLAxXY5+erCOqwwVEQLl46r9opKwdVDjIDkSMT6vlD21Y7SWvWUlC+EPGmoR88slPV0OoFChba7LBT2P/zf8VdgQWn2bB+30nnIG46f4+95HVzjp0/EiOshJ3Rh/9JhUsN/xIlBjYnuqmHu1WgCRSrL3UctNVdJ8zBfj6SpXzyOA0IXbI9Am41mn5vYRMIWyjpXE84LbREtTh4kWgqrXUQ5FqN5Yo097Ef4ZWby8M2Ejz/uClG0rwG/aAWYbI6ivHRTtI0V/upc9j8cegx/HXhwgcMso0tE75sFGvt0qg3+plt2ifQOLhXpFWHIxb77Aum7iOHXXhxvRUOJxJ1AeofO/48Yi+8RE7adSDPDhq5ASL+/Lusdozn5jTnHX6tCIMq/8uu4pNfSdyAfC05wC+p/OHe45kYaTThHKLAWKtI3/RIx2exhmSRl11XCr6n+Y6xP8e2Yj80R5giZBSER/z9nOxPcYim/YW5m5MV3s3ytkA5hRtwBaF60Gw2Gj9mLDzc1gHus9pmZYWW/gnzLQn/IWF99KQtm9emDCvu7PEGKI+06bYSMbmmDPf/m+noyKDvKvufa3EcBwzvL473PxgA8YhBhQc82vdeuIlYSKYD7HtoLQDY5A14Lwjp7rv2UXrsIANkNW9FInx8IOaCcKFbbAMaZ3Hz1AVF7mQOhqhOyQtTHQM/atZsWulSivXMBIf8J2yGMEI72BGWj1GsVf795rqB0E8sQPn+uPxRJ73mM6lCuUG/R1MYbzZ3J+Kfc7nDhXYZT6u3dIaTi2C5fSYnHn/tXt4bq9KuSsUW+95mVPnbuI438yETo0NjUWq0tc0BlLwTBRYb8pzXbqlliOwxYOkHe6IG74BAepZ8N4XHOWL63t15/Xbfjl5Hjbc41SetlFiKETJrqhDsd7ajGUTEq5u8474AsMOOs2DECQM0rlZqUVghL77eTN vhoFfOOI utJ+1zWxEgqGJSEsXIFbZWK8S5SBwlv+4bLo+aCbQestVWB3FE3VeXCzOF/HwKWi+js40jvSNWHv2aiIPol0tNRwUGq0rc5CLIlYNPEAmIdOBOaIamxEc8A/5SOnbtqIpj96Iz18qKMHfDMvQStDpQXpQ74ibeF9kUmElGjoJyruGCX0n0YglHK+9VVVxQDNzQPbME/cQQ163oaJxiJs4iXLKm38PeB3+NCEquw+5tSvjtkPxGECWTGKfGd8eJ7Y4vImdDEsCUlizkFY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Memory allocations can not happen within regions with explicit disabled preemption PREEMPT_RT. The problem is that the locking structures underneath are sleeping locks. Move the memory allocation outside of the preempt-disabled section. Keep the GFP_ATOMIC for the allocation to behave like a "ememergncy allocation". Signed-off-by: Sebastian Andrzej Siewior --- kernel/scftorture.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index 44e83a6462647..e5546fe256329 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -320,10 +320,6 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra struct scf_check *scfcp = NULL; struct scf_selector *scfsp = scf_sel_rand(trsp); - if (use_cpus_read_lock) - cpus_read_lock(); - else - preempt_disable(); if (scfsp->scfs_prim == SCF_PRIM_SINGLE || scfsp->scfs_wait) { scfcp = kmalloc(sizeof(*scfcp), GFP_ATOMIC); if (!scfcp) { @@ -337,6 +333,10 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra scfcp->scfc_rpc = false; } } + if (use_cpus_read_lock) + cpus_read_lock(); + else + preempt_disable(); switch (scfsp->scfs_prim) { case SCF_PRIM_RESCHED: if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST)) { From patchwork Mon Nov 4 10:50:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13861225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2F94D132AB for ; Mon, 4 Nov 2024 10:56:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 702066B0083; Mon, 4 Nov 2024 05:56:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B18C6B0085; Mon, 4 Nov 2024 05:56:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 552DA6B0088; Mon, 4 Nov 2024 05:56:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3AB066B0083 for ; Mon, 4 Nov 2024 05:56:37 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 36F2F140745 for ; Mon, 4 Nov 2024 10:51:08 +0000 (UTC) X-FDA: 82748094660.15.28F0391 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf02.hostedemail.com (Postfix) with ESMTP id 5A43580009 for ; Mon, 4 Nov 2024 10:50:06 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=SukzN7T5; dkim=pass header.d=linutronix.de header.s=2020e header.b=fCl8CyFO; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf02.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730717408; a=rsa-sha256; cv=none; b=R9W95GGIkwJQo8T1yE8Mg1qPxjLjiDeu/e26bmzGmJtsWj+x7znUsRFwf0lMMRT5CnzeER ZtMa5QfMdSTBL2NAWGeD8u3xPa9mItGEpq6BW/lff/9J8mmQCg+4ybbKCrcHwJJaAZf1AV 9pGCVRNCYhQu/irBXjSka5Op/HDqMno= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=SukzN7T5; dkim=pass header.d=linutronix.de header.s=2020e header.b=fCl8CyFO; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf02.hostedemail.com: domain of bigeasy@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=bigeasy@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730717408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GcwZ4Huf0ok76UUKQ+LdAyCTe3RrRb5PN85tcNXwRJk=; b=Kle/8XVwwh1AxGTUYV8yRoHGlM6Lyd7jtymr1tBSS/tr/DtYsnYD6vp1jENppgmPXiuunL +9lrWxnxtjZknT9NHvyYg6K66us0vzJAVMdyP7g5vB5OOStJkNaItp2CwQDnfzZN3rHThG PS56G6scVOlI+ZIRLU5dpnuj346QBX8= From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1730717465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GcwZ4Huf0ok76UUKQ+LdAyCTe3RrRb5PN85tcNXwRJk=; b=SukzN7T56lKOuc5BCaY4Ynl/qdZao8MieTHRiEXijI2MnIrgIToBYEpOHHXsdcO++nkZZF 9TMuGCJdxfx63iMHNfe749R/jnPifDtMQtqKl7TjVh1nnayxLmyRaaxcI38GaNalNd7muB CMi5uYuAfjaYWlVOKvfdgrN6iuJ5Ii0NeqHH5jB85aRS2RTHF4xw9pblNqz5da1oxC1Gar VUhb1oQh0w1qibmatIVC0Fa13+XbpEPIxw/BnLXWB5/i7MHfGN1o06C8fp0qXSGJBW3b43 9QvA81w7yiZwQxjM+OWR981o9LNTBGxH8XsUjyc9pa/zJzKMGDx1kMqYDwFGEQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1730717465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GcwZ4Huf0ok76UUKQ+LdAyCTe3RrRb5PN85tcNXwRJk=; b=fCl8CyFOfq10gmJzfuJPKQEqviDpShgUqL5Ok5IGRTcxCmYKQWLlV2l8VujV+hSx0WzucP iP+UhqhAZb/D62AA== To: "Paul E. McKenney" Cc: Boqun Feng , Vlastimil Babka , Marco Elver , linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, sfr@canb.auug.org.au, longman@redhat.com, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, Tomas Gleixner , Peter Zijlstra , Sebastian Andrzej Siewior Subject: [PATCH 2/2] scftorture: Use a lock-less list to free memory. Date: Mon, 4 Nov 2024 11:50:53 +0100 Message-ID: <20241104105053.2182833-2-bigeasy@linutronix.de> In-Reply-To: <20241104105053.2182833-1-bigeasy@linutronix.de> References: <88694240-1eea-4f4c-bb7b-80de25f252e7@paulmck-laptop> <20241104105053.2182833-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 5A43580009 X-Rspamd-Server: rspam01 X-Stat-Signature: 7zofweou6n1oeq96cc5qczeieo6f5gtx X-HE-Tag: 1730717406-895968 X-HE-Meta: U2FsdGVkX19nIWnXi4reM/90C5kH/dq0OVaj5T6xdj05xSxn7F5RO/KsqGCqflkiUJvj7+HMOmlQa6/kut3EauM4VP+78cKFqFqL6dXCOk9W02Xg0MtPsTnaPsYiU8TkYHTC2yips8KsMsug/TA5zmSWZSOjqLKFTjFTHajFWZ50ABrMN/g5nyA4Bj56izXY0NLKLhx+DnHXaoeitYHt+jaITSDzkQS85K4oVEeq+p7LnL07LVuDX8gHjYd6zSshmu8GzhdcbCl/FywdfNiLBhmRW3pK3J2/EzEbMwESF7fL2MoMMNbK1ywbmKTyABaMwqtzY0Iyklr493ACXlH0xvlwR6vUzE60utFRKZIx5wEEFtZNwT9hPi82eFdOJiQx/D80TRHoT5+0CAZM/d6VnxrVLVCucx24BGkhNBRuzZ2oX7XxMbJhdcUvmy4asK6x83nh4ZO3pwWFV/ycUoXD299dIAO8hI3CwkgR0twnTf4c8t9XM3HHZyeTR7yWO+3j6XkpaBf4I6U2oKIdQw0P/G2W606guzWR0nxsMNS+uWH07iirDQiqUi4DF4QqKfENw67mnlNC98SL3/bCi412p33Mk0VYvmI72+GOA3cwOaCvtt6jq+CLrpHqmNGtblwhgUS4DsvGzGi2aTN+Ef8vz6svZqKY1+AzGVVjpo4jGqJ1X2iPLfC4d+EulsKpLbXtLd106MEBtrBR/9KRctdpni9vLBkvTYwWb0Q2YMwk/eLXfGTTE8vwrIBMICM5MobDfYMtCihWf7rSMyypWCirFtxG1FT9ZgFyRXqulnhBQ+JjCHMk43m0Jezn0ESMjpUjETRvjPPtWAvb7wdjj5jD0nJQk3nR2xGSLCHhBOdOjTTnw9jC12Z1JAwx58k6NY7RZEB5mKwRhlNA9LSQ3Dd4YNrtQXa4SNpVhloYVwIh08Jq+ubKKyff0glcWBw72ZjE0y3Khj0/qddrUufsGP5 mEvhSGRb 6krhzQIrFA1D/YeVpmym9SHbM6/wQUTkZA0q7t0DlYibvFNOVaXCNDXL/9IgicMUoHh/arcFdTKoSsDtNPHHa6qJ+LJ1j20bpAunca5RZJM4yKYrPrU+snG2XB7P/J5Cj1EESTocIhKnWoA+vjRUt0MA1lj7C75EXHLXvxc1qAWxau6vsfHKKtDrdlloceICzR9GR1CmL1rpOOSqdZHTQ/KyZIi5jnt592kNZplYEmqOUcFXUB9pY16DXVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: scf_handler() is used as a SMP function call. This function is always invoked in IRQ-context even with forced-threading enabled. This function frees memory which not allowed on PREEMPT_RT because the locking underneath is using sleeping locks. Add a per-CPU scf_free_pool where each SMP functions adds its memory to be freed. This memory is then freed by scftorture_invoker() on each iteration. On the majority of invocations the number of items is less than five. If the thread sleeps/ gets delayed the number exceed 350 but did not reach 400 in testing. These were the spikes during testing. The bulk free of 64 pointers at once should improve the give-back if the list grows. The list size is ~1.3 items per invocations. Having one global scf_free_pool with one cleaning thread let the list grow to over 10.000 items with 32 CPUs (again, spikes not the average) especially if the CPU went to sleep. The per-CPU part looks like a good compromise. Reported-by: "Paul E. McKenney" Closes: https://lore.kernel.org/lkml/41619255-cdc2-4573-a360-7794fc3614f7@paulmck-laptop/ Signed-off-by: Sebastian Andrzej Siewior --- kernel/scftorture.c | 47 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/kernel/scftorture.c b/kernel/scftorture.c index e5546fe256329..ba9f1125821b8 100644 --- a/kernel/scftorture.c +++ b/kernel/scftorture.c @@ -97,6 +97,7 @@ struct scf_statistics { static struct scf_statistics *scf_stats_p; static struct task_struct *scf_torture_stats_task; static DEFINE_PER_CPU(long long, scf_invoked_count); +static DEFINE_PER_CPU(struct llist_head, scf_free_pool); // Data for random primitive selection #define SCF_PRIM_RESCHED 0 @@ -133,6 +134,7 @@ struct scf_check { bool scfc_wait; bool scfc_rpc; struct completion scfc_completion; + struct llist_node scf_node; }; // Use to wait for all threads to start. @@ -148,6 +150,40 @@ static DEFINE_TORTURE_RANDOM_PERCPU(scf_torture_rand); extern void resched_cpu(int cpu); // An alternative IPI vector. +static void scf_add_to_free_list(struct scf_check *scfcp) +{ + struct llist_head *pool; + unsigned int cpu; + + cpu = raw_smp_processor_id() % nthreads; + pool = &per_cpu(scf_free_pool, cpu); + llist_add(&scfcp->scf_node, pool); +} + +static void scf_cleanup_free_list(unsigned int cpu) +{ + struct llist_head *pool; + struct llist_node *node; + struct scf_check *scfcp; + unsigned int slot = 0; + void *free_pool[64]; + + pool = &per_cpu(scf_free_pool, cpu); + node = llist_del_all(pool); + while (node) { + scfcp = llist_entry(node, struct scf_check, scf_node); + node = node->next; + free_pool[slot] = scfcp; + slot++; + if (slot == ARRAY_SIZE(free_pool)) { + kfree_bulk(slot, free_pool); + slot = 0; + } + } + if (slot) + kfree_bulk(slot, free_pool); +} + // Print torture statistics. Caller must ensure serialization. static void scf_torture_stats_print(void) { @@ -296,7 +332,7 @@ static void scf_handler(void *scfc_in) if (scfcp->scfc_rpc) complete(&scfcp->scfc_completion); } else { - kfree(scfcp); + scf_add_to_free_list(scfcp); } } @@ -363,7 +399,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra scfp->n_single_wait_ofl++; else scfp->n_single_ofl++; - kfree(scfcp); + scf_add_to_free_list(scfcp); scfcp = NULL; } break; @@ -391,7 +427,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra preempt_disable(); } else { scfp->n_single_rpc_ofl++; - kfree(scfcp); + scf_add_to_free_list(scfcp); scfcp = NULL; } break; @@ -428,7 +464,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra pr_warn("%s: Memory-ordering failure, scfs_prim: %d.\n", __func__, scfsp->scfs_prim); atomic_inc(&n_mb_out_errs); // Leak rather than trash! } else { - kfree(scfcp); + scf_add_to_free_list(scfcp); } barrier(); // Prevent race-reduction compiler optimizations. } @@ -442,6 +478,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra schedule_timeout_uninterruptible(1); } + // SCF test kthread. Repeatedly does calls to members of the // smp_call_function() family of functions. static int scftorture_invoker(void *arg) @@ -479,6 +516,8 @@ static int scftorture_invoker(void *arg) VERBOSE_SCFTORTOUT("scftorture_invoker %d started", scfp->cpu); do { + scf_cleanup_free_list(scfp->cpu); + scftorture_invoke_one(scfp, &rand); while (cpu_is_offline(cpu) && !torture_must_stop()) { schedule_timeout_interruptible(HZ / 5);