From patchwork Wed Dec 18 13:04:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95420E77188 for ; Wed, 18 Dec 2024 13:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DE906B0085; Wed, 18 Dec 2024 08:05:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 28F516B0088; Wed, 18 Dec 2024 08:05:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12DDD6B0089; Wed, 18 Dec 2024 08:05:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E4F3A6B0085 for ; Wed, 18 Dec 2024 08:05:17 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AC1861C7D7D for ; Wed, 18 Dec 2024 13:05:17 +0000 (UTC) X-FDA: 82908099414.06.EDE1649 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf05.hostedemail.com (Postfix) with ESMTP id EFDE0100018 for ; Wed, 18 Dec 2024 13:04:14 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KFZ0pK9y; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=37d3136MH/Lj3ZUUrUTIyNwrWsJuTyPww/s07XJQrz8=; b=5b/zKLvWe8rSn3P/6aaI6UQ7YhubQXTPnyGrFfkHcavg4R4tnoI0uIYmrX9zOhDNGFTna9 j0iGfLk5T/vyULtsLUbrWGv7PwLNSwXidHIdEwtEkJlOUry2lIDj0byqIy2mZmeCPY64zh kJruErJmnYwoVmluTU3JJJ2rKJJJkzs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KFZ0pK9y; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527094; a=rsa-sha256; cv=none; b=gxNJS8a2bw1agASkrwHkyfuDeXwQu4VSWjm3KB6v3LiSb1eNy70IEEh8ggDpR+DUljyuH4 MqPFgNYXxvbCI/WBRZ/X37pdILjeke6Sbh+dg/WVrq709Xi/H15wcTVOPPUyoil8JuvXn2 LW5agIUALj+HSYgdUMZpypPUllUI9bA= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2165448243fso63062045ad.1 for ; Wed, 18 Dec 2024 05:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527114; x=1735131914; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=37d3136MH/Lj3ZUUrUTIyNwrWsJuTyPww/s07XJQrz8=; b=KFZ0pK9yykL8IrCU4q/f0ijWRzWVuNsFbeZxm7rV9/YqTmNuhMGrCL7qzITJkNqC++ ZI5lytMnzeIyXtJgnUpM+p+CR4fxEbeG2TFabgboy/CFIBYI/kDsM9XM56TEG4VdJLLi WBPLlQOyZ2ilJb0YryZLYFV2Nh2MWGzzffcO04wdLZhS6rIfQEjBuUODjz1ONuYmJZfq 7UVc09UOsU+veWk4uUo6xUbHxCPERBAOEY2POV3WaYBN4XPtZhIT9eb11L5EGtXY4/iA 8HU0ZRCHIhWxu7bi2AeSVexueblfNJ79pgsNiurRfLr9m71emYWDtttcdO7azOJ84Jlf VmAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527114; x=1735131914; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=37d3136MH/Lj3ZUUrUTIyNwrWsJuTyPww/s07XJQrz8=; b=OqF87JkgCqsdGSJ5pqMrA1lb9opbWjo747xBZr/EknFPNSvLTWSb7+GobxsH8af6Yf 3g63rKy5R9VPNAolBHrzaszV6AdoZjeE3wKP4qZq7r0G/1D/pCQQ7anN9o7SL/eJ125H nSmgbP6jIZflWMNCA6L7yvIehm2dvvx5kgh4cG5lf0biYy9z4D4pIVbrSbGWgs8iBHzG ZdfeD/kGKJM1zc3cWitWN17Qp7YLzqCKgiaP/6aiYlv+/g1Lf952dixmBDq7vhviOX0d zCC+MQZzOqffm8WiLGhwlyz7pB52V16WQ07bzcCG1HGDamnJQITgqChjPrez6C+UOi9X aCJA== X-Gm-Message-State: AOJu0Yx7lByeFFeECqybrNA2nk7HwCBFdPunyaiFjmcPC6UJHusuxJMX e6bDWoif6AVLphFrgQJSvy0sLiBmYQtovBuAa5/TJnGI+AZObqJ2z420IYXSqHQ= X-Gm-Gg: ASbGncsXdA8O+bDvvfT5hZekfKEDHbkO8a2Pg/b3MAMzi1nZIz8C+Z04/YjTeTsQHGD gSu8rSnfJFfR4FyKwgzSAYhfwjCO1BXvQVcOdCHI2RgPnTq1ftqTRW1AgARaGT0jpJaUgc/RRGW XZ6r1JhIZlXP9ddSWY3kHLMSORjU5nOBUzsSIPiUT6lBJ6kDCPTt3K6QKEtItTrA16sF7MyDSXo CyNXmfTqJWF/dKyTpAIJ1rbdchub/1SH8OKeVoMphJ7TaxgUuDIqbIdUJj5eK4rwPxGd62ficH8 bOcagXewvj2CLhJ15USu1g== X-Google-Smtp-Source: AGHT+IGVLgY8vFLZqjIL3oQPZehgU5XP1d4YxVu/+EapWtzKcY/+w347Rqmd9s+hVdjxdsfNE1spFQ== X-Received: by 2002:a17:902:ef0f:b0:216:6855:164 with SMTP id d9443c01a7336-218d7252ae5mr34612065ad.39.1734527114526; Wed, 18 Dec 2024 05:05:14 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:13 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 01/15] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Wed, 18 Dec 2024 21:04:37 +0800 Message-Id: <914f1b9359d5547c08a30908e8c970fdf463e210.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: k99zqogzrxzxp9t5f4bbpdjb9ezzxr58 X-Rspamd-Queue-Id: EFDE0100018 X-Rspam-User: X-HE-Tag: 1734527054-162897 X-HE-Meta: U2FsdGVkX19vAs7XEc6TAoamJuH8RT+l37kOHuk2+M+tsU6a/BsxkmLvHSUPBMzG7KudGAift0sJBWVmqnmJQKJ2Lq69TGy3f+cF19vhJZwsKeanq0lJD9Ng9NMFH/sDlTOyObojZOgirnFzBc09CgsczY5U/HiEatk3pktNxUFTWPbBkKwDD1CF0Tf4/kTjAZ7zn8DWKFBA7DwX9sSfMbd7QNkYG4knVvAZrruXSDLiBVqttkd6MHLULMzkYLjrpkaKgV4D5REwH1IQDrX+NHSVX6+BClC2xRjboRjwUyocPTWnojnTlhbaDihdAl40e7fWAbwx9StUBFsveytGqni0CPWQr2bzju8Jff+M1Z7GI8ekJh40OHNFglvXV/ZMbO0enFS5jb0OwD+RitLf4x/CR3+HkI9SiOFbcxla8teNF/L2WPg+WjtykASkdd59TNZS0JJRqD/bHCyCjw9cd4LhRfa7S3q9Q+ithbbuyqicfSWSbf7F/FS248IPwedl2I2Sc7escZEmKHdXCsGici2WUBTIz9RBB0fCaxL8LV7DR0edF/YObNk546LQ7rFbiF2KcPpXz/roeII1WyU32ZFOxKNDu8F9ntj4dLq2qUfjd55XfN7FqFYGj+XWIslTZgUtudwSBW/hOiaSeNf8C6k163QZ8M45wnKHSuo1JkpDJS/050ajzk6Kimgmx9tgXb89s7cpgteIFPgU20t+g7X4NtS8n+i1LApETMJbKyPtDVEkaLQWauOLsbHw9TukSlXVJ2wtWvJjnJgtMw+6z6qyWZva8kJ3gFYL0T4ipCUX0RuV272xdUlYD0H3UxdkiYvzmWzhLSPpG19s34/I6J1FIcTPlnR/muaQWL6vA+UEi+dRJbuZ7mjk9uU+GmdZo+WlzmBaEiOy+SR8W4bY3ocOGF358upyIBHEt4HD1z9ySLN+CsVsBjwVfmjPIEbdX4ErQOL6XZtbDKgqhhq EcKik9Ya zWJ2KJS5AI32yMgkjH7bLx8WhI51sjdhXVhF7thLvpWPMuSle2/qbmahr85/RTlmVjHD4jDk3i+kdHY5r03J29YvEBn1q7wsR8pmkielN2dMac0rIRWGl5J6lCtrjJ00jl+DSgR8UTXwCzxaV1W3GnH2/AyhO40j8Z/rQ0CN/AKkwBtLn7eYYTlQ+pGPvg5fgu74jWE1GPQ91MI0pC4/o+vRegUW1NSuxWzxCYryKSXkmH4bIYnyPjAZ9ZqsiYzzZzYn1Z3en4poqadaUCtIbwKqzOE2vbBc4o+IetqshAfImS523o4RyMxRWUCtNPHBrJ8GVfU76yibFMzO83ONE/wUAPZOOE5cKerh7X/NPM4mAbVreiAAKxHJdGoN9MqcIZPIU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000045, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 024bfb3ca9152..9d58f89e9643e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3019,7 +3019,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index f73645627fd92..e2139af51e050 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,13 +435,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -490,7 +483,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index fed097ebedcbc..e064f5edb7c56 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7087,34 +7087,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Wed Dec 18 13:04:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70424E77188 for ; Wed, 18 Dec 2024 13:05:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E776F6B0089; Wed, 18 Dec 2024 08:05:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E288B6B008A; Wed, 18 Dec 2024 08:05:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA15A6B008C; Wed, 18 Dec 2024 08:05:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 747CF6B0089 for ; Wed, 18 Dec 2024 08:05:26 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0654744B7B for ; Wed, 18 Dec 2024 13:05:26 +0000 (UTC) X-FDA: 82908098784.17.935510E Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf01.hostedemail.com (Postfix) with ESMTP id BA28440020 for ; Wed, 18 Dec 2024 13:04:59 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bSRY8JED; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527109; a=rsa-sha256; cv=none; b=pEJof8WoKiJ+1+RDP+ncPEocOZbBvg7JO8EIDSCr3Ew8K1mxD/xx5UBAvg2hEq1bmMSmK0 4c3L4v/3hXdyglIisvj4XLIVYcgGiTmgbuhj538SB9lN8/G/EJ38DSJ1QVvTf539hWYogn A/SzPtKlZ02bHJlPdw5wLg+UP/8MAqA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bSRY8JED; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=N9tnOfMNAyKlwt7tLvm2UTXKxz5i7XrY+re9Me1pdlHgCrYV4bH+cZfjS9BjYUPL5+Cn5c FD4fhDjHQC3KjPxsK8CDAJ8Si1OB1oQ55cKq424qpClce6f7i//Yz+MTD3bOx6F5fItb6Y 8cfc8xOwMMM0r6ghYJd56z9su1lSrP0= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-215770613dbso43925135ad.2 for ; Wed, 18 Dec 2024 05:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527123; x=1735131923; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=bSRY8JEDczSTeZWaVlY7dkVISVW5Nkx3tF4YWtDGyGd1V0VYNwa9V3kvelbW8anbCY mPcJQOmsnfkqXNyfgaq9dX0kj6tpO+6HX5605NuwqykheZUNZzPENXMH19kgS4xI243q Oub9AVxdJpX63fwbao5Lj43dDkzod7s6eIeKdBL2mlG2wzn5aiU4wIiyPT03nxs/Z+nA JsklgcVdpu3sFbxjbwus08b87wIE3NIulVUq/vTqATR3hpm+m2OwvvSJVfviAHfugQOo ms0g6yuYKYxV5A6A6TCO7A04ON4x7Qg7S7dngFHGWg2QUP41TrAl137RfTmcZxAWyb0C CEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527123; x=1735131923; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=Uv7TgQS7LJZk+AvpJTPEcMkQCu2+S9gXcdsyDPZxxuBXO2YoVCSBw0TTp1ndpwIQNj 1LE2N7T6gLFe1CWunH40Q3CORs9ogo0Yh7cmSG0XHnHQ+i0LfDGHmVCCnueg2NNPf+Ee FB/2mn/naQ+YjymCzITvqBBRNgTLWNEYb6fxvzk4RsAuQGZIcRmiMucKF993i9NCSboK TkOSqkhSLS50mplaq3Gr/T4279zZu6xcUimNseDot0lMZ0QnUke9BY+yNYL0eOI/SWDi kaG5bgIZTeqDd5m/4wThjhXnErwK2FLnxohPHRLiriYXfS5JpSSZf7uyzM3KuybanKnI 0DJw== X-Gm-Message-State: AOJu0YyowPUyLUrt09qSt3lZFme/iu1vl6qXJw08skHUX7Y5Y9o/1RLu DFekpK+60ArFcvk3AAJ0x47GJ4hvWlwcoA+iqJSW3iFmTdVJ7jQ42KwaT9YKouQ= X-Gm-Gg: ASbGnctjDxOGdn+oUYE3ETdoIkqk8KTMNjzEXNjm3SSrMNQfKbREAOYByCTF6o9+2BS JF8SMF1nn/vTbF3JZyPUJwsIQdFB4OPHzm/jesiutdcEym7mOt3LOWDNu4G2/00db2kv6oDxDlR JgnsR9rGcJWdAUIGV9MGBjAsBwLRWMgF+5cudS2dIPohUXygvGDnl7JzvxMOYQK8DFQUszN389r Mhfg30Vkx7NsdPA4fkV6GTZu6GGy0rzutfG8POJVHFT79Cntvw2zC4jd4WtSE9h1Kpu3IF2AGVg K0uh5q60gHZPlR5socpSbw== X-Google-Smtp-Source: AGHT+IHf2m598v0e4o8VjiXswnpFdHX9IrqnwvlyprzzHZEN7MlhmaMV6JPA0vL7pLwTvqcblsUT+g== X-Received: by 2002:a17:902:fac5:b0:216:6769:9ed7 with SMTP id d9443c01a7336-218d724960fmr35428695ad.40.1734527122502; Wed, 18 Dec 2024 05:05:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:22 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 02/15] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() Date: Wed, 18 Dec 2024 21:04:38 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BA28440020 X-Stat-Signature: kgdfczndjqzg7uspqqsezxxdbs4z1ibb X-HE-Tag: 1734527099-951714 X-HE-Meta: U2FsdGVkX18cWN4UoatP8r3ZXVZtR2qej5XMEnqbWGF3aVXYsR3/IfS/tlHfOjU9gBrHQU2uKgT26o6irAFAAZE/of2iolNPyKqTjfxzdz0TKBlizwQ5R+lBLAz2Jxl5Mo3DbqrKiXtbkyKvqHKmw3szJ/DdeehV8lpfl2knZOjOqkXnNnnxLd6vG0ZVrqHmkEJseo07vEOZqs12J+bAZda9BGDYHoQcCl6JusrSZmwYuWzWFZOJySWUza2exgWuWvoShO4JSburX9cGYcEm2ebfa8CuzynhX0FLoAHS22R2ZgiJ/qE7VLwEzjSTzD3DWjb3VHxXRMUBx1bYP4IhTqG8GBACFo7KsGaW08H6Ea5Wa4HO2TG6QjNZpUeq09PEz4EzEDZVQVF+GIvYHeK86X77hXbpEHjRP723hHkdCQpqLoFJMwWjIzBmXY/gnCXbB6FFRsRUlftvBUDtB36VfKdfq7NPZLr6FO6ZFjLZgLadE6WSNN3WodOq+MoqTgKiOL90HVJcP87ZhFf3Tm0pQeEOAryy2pb4nJN3jMCVQCLRmx3xhOPUnT2s3HVV1ZjE3ErrVofltPjrWzxajMP8/eGKeqCWsIAsJDRVpsIr/uIvxN7AY7GsRKQGKjA8FkCGfYZ/m2IaGjiWayMgk/6ISCJPGoasXMEPM+9ZtO3e+2yXX+Jk/Er5mrs7FpbbLgmUqsJNJ2sEoStZEax9oZqXrnPXaeN4bzWaLJtYkd0JlzMhg6viVzxCDDuZR2T6xX6lD5/YHrOwfuA5tnmZolWWvwCToxPUf55M0akpnSTqgS7vWlzrkldlZ3fYVM2DxT1yBMrU5ANEv5qsikQNagkKQj/3u/qXfuekdQR44j2yQXGhdUUKxi2LhXDCVWA7bkxUETNWleYSt1wG7Ga4g9Av9ixgrFEolKNbG4w8aN2+vZfZiBDXYyPhqHAItnUiWsohzXfD/rRotEauIiNY4Ej NL/GKvv+ 0UFmrL9zAzOf/DG21wok/3La7CXY2+PRY9PDFDu0stz+AGyoBnkwgoVKQlAJcjsNphEK4V/LRXnd2B3VN+m6c/sBoZpNvcidFsZhTkfw5cLxfFboFUzLrxKMirG/uLaSUQfSwZbtSF+GNdKJCSQO5/+aY8Wto0z2MQp3apiu0zZFxVjMkBcVDhe3s7UZ8gBFzJsJa2zTIUDnO+xk8JCje8xzXVrzyXDYQOHHVLgNdrqjlujr0Hf2gzqWBk249BsqJNHLP9z2YCWfuVBxlInrGXE9P6xAi+lJ/tsvKyV4GbF2pOzNcSkTHDlAmGELjtEzwSB95ofCswdunn4XinqwMSItCkhITU0xIB8u+z/5XEAjphKAKbJDYWs1THwsEK7iaUqxRqYZg6aAcTSZWhakdTGDwL4qljiY86JfYBftIQ9P8VKKLtvMXoidcblX9tyPcSdP50JkK3lBqLF0pqXunT2ZDsWZ4kpyl2Ci7plW3BuCE/wc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a wrapper for get_zeroed_page() and p4d_free() as a wrapper for free_page(). For these architectures, provide a generic implementation in asm-generic/pgalloc.h and convert them to use it. And like other levels of page tables, add statistics for P4D level page table. For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its own logic, so skip it. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/pgalloc.h | 15 ++++----- arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- arch/x86/include/asm/pgalloc.h | 16 ++++------ arch/x86/mm/pgtable.c | 3 ++ include/asm-generic/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 16 ++++++++++ 6 files changed, 98 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..679c530549327 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,8 @@ #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_PGTABLE_USER; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + __p4d_free(mm, p4d); } #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..bb6e1c5f1fb19 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -14,6 +14,8 @@ #ifdef CONFIG_MMU #define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + if (!pgtable_l5_enabled) + return NULL; - return NULL; -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + return __p4d_alloc_one(mm, addr); } #define p4d_free p4d_free @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..d9bc6cae77c9e 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -8,6 +8,8 @@ #define __HAVE_ARCH_PTE_ALLOC_ONE #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_KERNEL_ACCOUNT; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + return __p4d_free(mm, p4d); } extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..dbf61819b3581 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + pagetable_p4d_ctor(ptdesc); + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +/** + * p4d_alloc_one - allocate memory for a P4D-level page table + * @mm: the mm_struct of the current context + * + * Allocate memory for a page table using %GFP_PGTABLE_USER for user context + * and %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) +{ + __p4d_free(mm, p4d); +} +#endif + +#endif + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 9d58f89e9643e..7168f46ad4770 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3269,6 +3269,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Wed Dec 18 13:04:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FE23E77188 for ; Wed, 18 Dec 2024 13:05:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC4E06B008C; Wed, 18 Dec 2024 08:05:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E75876B0092; Wed, 18 Dec 2024 08:05:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D156C6B0093; Wed, 18 Dec 2024 08:05:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ADA956B008C for ; Wed, 18 Dec 2024 08:05:33 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 782FCADD2C for ; Wed, 18 Dec 2024 13:05:33 +0000 (UTC) X-FDA: 82908099078.17.E3E3EBE Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf10.hostedemail.com (Postfix) with ESMTP id B0E40C0019 for ; Wed, 18 Dec 2024 13:05:17 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=O7ybMgrN; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527101; a=rsa-sha256; cv=none; b=WEJxkKa64T1wi1Rwf9ZzaHnGcPQd9m2XuNwBoafxa0RLt/6SbYWPPJm+J2by4YL/ovOi2H zXDTxEH2aV7528nnfnyPoEthKXMIiufRazN3vzqfkSthOyo+wONYQZwZEP8/Iq/L3cZNx2 3VfzJFI9/QhtyuSsYk2V5rdkd3Fvlx0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=O7ybMgrN; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527101; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hRj0wkIcDyLmKPONLgLEnODGJVOQ/lxkeDhrLguGMRU=; b=YwJwpB/2pgVVNylSQULPwlnI7ZnjMANpqmIotir0OVf0TV5rEV7+3QOe0SyTA7Uh6rS8KC cr7oKcxTYU+gi2BpzqcPw2qWxvm52Z2WywF1zYMD+4PJbjUr8SvPKi9xU4dCCTfx27QtTL UHmxY9MODlDG/jCqxos1rBlv5F5aR3I= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21661be2c2dso47941675ad.1 for ; Wed, 18 Dec 2024 05:05:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527130; x=1735131930; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hRj0wkIcDyLmKPONLgLEnODGJVOQ/lxkeDhrLguGMRU=; b=O7ybMgrNTe+IxaL2eRc4HxwnZWNitNtoc3f1lDKiiv6IiD7NBt8jhILVo2wsTDrI5Q 8C0AqpVX78Gz92gNH44eWEj5BRvNC5uBys5PFLoiINl0Uk7UnTi9d7tRxruiIkgaI7ot d/0WjFu+Eys6HHmCyHq+I4RlhocRWV0fqF7QpSzLHK4Rjg4+tT5svv8kQNfVfJeXqRj9 Ii4bXQisGJx31D41IShFMB125VBYcezY9eBrmQY5SbVAikL4Ak3Tndy+4AN4ZY12nTZM 8DP4NKFJMAgu8pWYV9ktqHl35gizIJdTpeFPuezJF3KoM5mIEF4b8nt7zvwmAFFJ7Rs3 MnOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527130; x=1735131930; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hRj0wkIcDyLmKPONLgLEnODGJVOQ/lxkeDhrLguGMRU=; b=ijgxS7h2WnQ/+TCI+NQ0YObpdlLc4bhfpUBySHcEPkIgsdLdz2PqluzUStFoIyf613 eq6MxNiB13SSTLmjStrHfeMuoLyqpJjoTgtwGvazwhExfMGqZ4CFsXdPwC4GJ98ZFfq3 fmEZRiz29yqXkDBCY8ud2p6xctWzbVX+uE9XNiSsEay8jSiqzhi3j7cu+hAiCLRZR3Br BgCkN0YSF3aScNdhMA3eotznnDtjUVrYWdn82dzoahLTtJB5DLIqkYWjGNpxDUz/qEfF GiU1QzunHJgmritGKeEQ+O8ehc2bPRymzclenZTLHmvho9x0NV+bqep78yYcWbgLc1sL PsjA== X-Gm-Message-State: AOJu0YyqCydfs2RODzszyBDH7rW1zlkzzLv2FsZMY/14/R4jznvMJy2M W4eRiJPSTrbeL+dxq27U5myjoUi/AM0BvYuxCNIlqCHfVaV3VQW0B5ApZV9EL7Q= X-Gm-Gg: ASbGncsufjZf52Pl0saINqsDr2+8kw/shG+ZAQRHbus2D4716pe5D2X/bwTxNEWnlgd +XGa+D+j87yN3jNmd8AK71OhDJs7/FX0XGWX9EKKIif2n8SCp/rjA5JJEz6AzPs/5ce4jvPBGt0 eOs+GejRZ6J7HCWw83F8/UG+hio+gjCEKWVJBhgxmfZhi62ZA8AHEIwCZVtmjb2COgQ/L3oTC1S hyacnbO/18XKNUDyylbPdC8N2sRvOUYs0GJYXHNXXzscs2552vEW1rx7Nb2Ahfy5ZcLpDAxk4/4 UKPssShnI9WW6bTDXkHJ1A== X-Google-Smtp-Source: AGHT+IEbMQUiOWkQtVsukFUm2ZLuTFY97EKfs/PZuqMM+WxRQxlasXxPYQ7Z03KXqwqNdfJrdRWLVA== X-Received: by 2002:a17:902:e5ca:b0:215:4394:40b5 with SMTP id d9443c01a7336-218d724abe7mr34796235ad.43.1734527130325; Wed, 18 Dec 2024 05:05:30 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:29 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 03/15] arm64: pgtable: use mmu gather to free p4d level page table Date: Wed, 18 Dec 2024 21:04:39 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 4wg4yzmo6qq443irh9ssoyrsgmpf7cd9 X-Rspam-User: X-Rspamd-Queue-Id: B0E40C0019 X-Rspamd-Server: rspam08 X-HE-Tag: 1734527117-281024 X-HE-Meta: U2FsdGVkX18bjeSEw62mzPzSIKhPDFiCMxLNUfYALX5aukiPVLI7gPrJpT1jGmrh/UnA/9e5VeH3gVpHsFQ9kHpXgOgE13EyfoUdPx1/o/U/OF82mI9IAVpAvxw9Z9d7ERepljZBX2WIJTYr14Rul3lHSzo41bXEO7Y/lUIkK2kspGRfI4TmRQwVsUTasYsK0C/RvFCDYUCm7PCHGSk8tgCUgdWdGX3b5g08BbEUN2XYNIyIPqz7Hn4vRfe0f5LN4Sm0GLNS/kGnQgFmo0tTCu4exRU+i/Zx8oa7wNY50pKw1kw7Du1AQVsnGcuk7Ac4eEwXjXp/f6YJkSYJWdtDWaz8x0M7X6kyA2kPjw8QukMMo/x5E2QovrMQyvAQFx9V2pbAqbwKT8Pna+o8DGxANqdW6rPCyPfiNTxQ/d1s9frhYVlDyGHjYv+KEIN0Z9QewRyJ4AB6p48qLh1X8PmxZ67Q065dKBsgcy9nd6sY9YOINrVgH5VR+U9Cx+H2hzXZo5v1E2mezK6WVvruXgoUuXQplOiv1j/DMTB+mOd0g0MV1stUY8ALRomQ8G9bLCwAlWub8rNojFIYJ/BvOlLZKP26P0sSu+JbQaa5NtFI04MUyhPAGdWs8mfN9fhbLpPIqxf0U5ju0UFfyYaJRCwjwl6aG6AtLvdncNWfsiLuKH2shS4J59PSmG1hAN4i2olstXuISh8RQeyc6/6QkSDL5xb43o+QeP70IwvWK4jxis9CZ34wnPGWUnaIEX9gEoXZWE3Cqt+GkShY9OMcp6Po0LSwwjaRf8zr03zPTq8k/AJo3NmZIqp8BwQ3/dzN4+0CiCoL3gohnXJMW4kba3+my7s4aQJNEizMX3FoUPNIFBo69zKIm9JG+vFCAwV8q6Meg3Wbq7miwC5VmT/lImI1GfuFky5ae4WxnCqbbBtUeTddz3J79yG78+HiNCfWPX1ClscXRAlE6XtLdN5Z1cE +emL0tVD KMSdZbYXfKyuBC0MDT3yrMVBPqKuqhaHlZVddhTqBwjeDYO6WAA5lw2rhkic8An83/gaAOc9g2huzVBacIrFGtVIEpr/dbFcrDCPMAfRiiXhdlEwdxBJwykDqIjNlHX3Lyf6/Huu8/1F3ZfaYbUtypRrOD10SY94RnOSLChNbCZ0k6dIl8oUw4Aj7dyFUqbuygMSaqWguANS5ZZfwvl4r8DEyzbM6X7ogD+Mc6hRpELmeH7NW/rNiDhbI1vAVpHGBzzi3aOioTwUgSGGtXkOg16Tun554jfP0mB5KikbvBTBTr+FuSkDiQjaYHAA6bXvCyhwnY05uA8bDCnr4o7nZm7uhE1U8/FPFsULPHiTsCgH7GTJPj39HRaHpO9uwL6ZYvPeyCFFzQJ8UG5dtjDR0YPLaWb3A2KVxs/mKPzTOQbNZ/0QyE6psh+p4OADxZA5AI/8n X-Bogosity: Ham, tests=bogofilter, spamicity=0.002910, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 2 -- arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 679c530549327..58cc260b6bee8 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,8 +100,6 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (pgtable_l5_enabled()) __p4d_free(mm, p4d); } - -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Wed Dec 18 13:04:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19984E77188 for ; Wed, 18 Dec 2024 13:05:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A31226B0093; Wed, 18 Dec 2024 08:05:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E0D26B0095; Wed, 18 Dec 2024 08:05:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 881426B0096; Wed, 18 Dec 2024 08:05:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 652236B0093 for ; Wed, 18 Dec 2024 08:05:42 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DF22D1A0DFB for ; Wed, 18 Dec 2024 13:05:41 +0000 (UTC) X-FDA: 82908100212.08.5A2EDDB Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf15.hostedemail.com (Postfix) with ESMTP id 55B96A001C for ; Wed, 18 Dec 2024 13:04:53 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T6JY6OOJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527110; a=rsa-sha256; cv=none; b=eIyFkRjtDgSbaLcdnc+7lvs2tdd9I3vfHwmj1D7Zp6+ccRAoKwRFyfrS5CZD8G3mtGYNT6 jPByu9+1Q4YEWJLsK5LiNLiq67VechrpUTlk6jrh7asWyD7z6Y4ZOz5TT7UvAnz4PDs1R+ rCn5/bqThWNIwDko3w63UEB/ItMlK1Y= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T6JY6OOJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=hM2SPRzlcH/sd96KOaA26kXbxPjSpPc80adHGPO3OVAen8Dr/0v+tDJtK8hd5fK8/mLG4f M9x4LzBOEL9hSEyzezolvS9L5bS+JfIrjQgRnrS/jeY7oXTgeu3dJFf/1kBC4+bMkHoquC S2AChLnkwzhYzxAsdhXw2XjM9Gzf5w8= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21644aca3a0so76879985ad.3 for ; Wed, 18 Dec 2024 05:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527139; x=1735131939; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=T6JY6OOJVEQXU6v3DU6YdOuMzArKj4BhQxg044V5ZCjuo9rw9KKuBnJvUhmFwBmKSB MyDaPMbTjiYtpu56aLDf8dt69jVZ+nZBd6GkaDG/awJlxDaXxBs/2WjBvbT/g2eBxmqG G1tr7NzBqLoS/JEUtaKzfupf842z6D0iLTCovCCDjYWdtKlZH2zWar6XxGtPxYFVXHS8 2OLP7RI28c3+1iov5vUqRucHBAMhPFLIMnKJxkU2npH4rFm9Pynoe8IM0NmHsvodg9mv WNBR4oJ/GE7lkuxfUDYKgSclXuA9KbtBNEcpqRrKdtsM9cgttExVOHBKQJceCohRHssA FXGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527139; x=1735131939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=uZIqDd8/REPrpjs7CXnaHX/jJw8+EQQ029rtTIzj/q8WIeZS8A/k43x2SUgkpgtUBB +CX+2qwY8ZmJAC62fOOFPiYzbAIP3DNYSczQMAEtSUwOjPKyJSIH8IpTWeRRkhLmDA18 tdevPcpBaUQ9jOTjdlCQeC3LnHjrS1G35GfpUYHWWJojAAET9ZHDeKTau0TPClDnQRqY i6m/aaRtvYlol1+miyMs5CJdEvb7Z5BnrZziKIjdzB8gSCWer9f85Y1BSFskZan1GfzF SDBq3ZqJ5ZGh1n0oTW1VcIWDtDquXjwJoi+NNggKpwhFO/2QbhQleQ28mhIl/dzo9vDh O+Zg== X-Gm-Message-State: AOJu0YzWk3PJTBjX5Uhj687jjdq0YDywhDWj5JleUtWB3Y/elkeJQrAD q90w7zJZrRzERIXMJL8h+nPNXH6LWYYok95Qa1V7eAUiY7fWAhyICNpk55rNvhM= X-Gm-Gg: ASbGncvEWO4WgiMdPgTSdfY5WHzsNkKMo47M1bMYjsyD34yvkdjzeTWmA08WKn1FTH8 70TAVXsPnJ+Yz/FBmdYBQcSEwJ7MSPKTgU/FEpzVhnA2zfoGd6pVmk52mocEGaWzky8IOFO0JIP CY+Ctkba1xZBftyy1EXyzwzjHGiuAvSx9fsZ69XiW5yWLPF5FTGLGSSrgDu+zHON/d0slNI7OtS qe97CMsfMVkfHnaQ9UYMZ3sREADwIbZ/Xo6+kKEQVFLmJxKUXrEu0ZOgl0llPHePGZKVcCnp6ra 3Vpe3UqsNI6ADvowZhLElQ== X-Google-Smtp-Source: AGHT+IEBex3adiI4KuoLgCHeY1D81j9Q6JTE0T6gwcWqf4Lz8nOg8PmUUMxAQoz8nfmNTBwOZklCxg== X-Received: by 2002:a17:903:2448:b0:216:6769:9eea with SMTP id d9443c01a7336-218d72477d0mr41450855ad.37.1734527138569; Wed, 18 Dec 2024 05:05:38 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:37 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH v2 04/15] s390: pgtable: add statistics for PUD and P4D level page table Date: Wed, 18 Dec 2024 21:04:40 +0800 Message-Id: <64401d867f3a965f60e9c8d09099b42f21749a47.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: e61wpinbertddfayzafrykeh9rszmbyz X-Rspam-User: X-Rspamd-Queue-Id: 55B96A001C X-Rspamd-Server: rspam08 X-HE-Tag: 1734527093-738591 X-HE-Meta: U2FsdGVkX195cfZ9V00UkjJzxMtYJkuW6P56evSHOX7YZjxlgDqzyyMBadq32p51AaWwPe6x5GSKQqKrhwITbA/lMYm7bFiZPMjoygePirh2iPjKUNHxTl2jVZ7QafGDMbhjUsSySe4dPeAJz7pr1Av9nZhxAgEdn1FDdAXolyEXp/Q3r0iMBn0u/TSIhwTn3YqKCf3x2K5u8U/t3MdI7tHWEjPp1a2gmFmd/GcCqgNfizwmVY27ajGE2jds8eEvMG6e+ajpuPXoDxK901vQr22qyLehanzV8b/q8pRhKYzmLFfyK2ew2ouEDkcwrUCHeQfhh1k8Rs7i0b3g9O+Ju9kfqAFunChbB9R07GqExMCYPx+Zbn22Vl2FE2r+/x2xY8MNlz18+xpdYqD7J3hkz9ySa0YjtMVvywtuthvLB3yfH9klf45U/GgaGgk8NOvTOEs/ARDvCrFeYKUYD+imhzG2JIKnGxfh+zyG8rVniZMoPZsZv32SKv9N1SqGqNNWDyGN572pV9d+8JlXG2jc0irebUB3vPe3dM6iq80nO2JxO/+Xqosfh+0kqQ1t2EbRhjblet40vl+DiYnb05HPyBZiOHCih3kKyo7K6YPXYl95OqOegbS7clxh2PRDtx8XPfQ0pS7qVVY+0kesZUXRyYV6GFWy+wRlf5QVbqzcwkMXijMz132qjHIWR6/iqD/Tk0X0U3745d8ff3i5Wfmt2kPoR7P+NPgpYXmog5iXfC42L0ALiAv/Hzsi5hbbBHOjIw5nwII1CdpF3+rv7sCSsUNuQIJhkpeLeU9TFQZsX0TaQMzHuG9gZCjsbJJGlhUX3vZm3WiDgsIxwD7FAthi3sRXupxayIfvlnSB4r35inVIVggw3zTPkBxmrcuN4NfCUiphps+vEZc8FNsG6opmtdGOMopF9rO8b8QEtT0OBj/8/fItegsOcLKjfMqUIDFgifXG2iymKfPdH4Xw807 qEzV2pVw qMspL0QsW1G8ZJom7huG9QHFSBsedJvGvqNQTmD0yyBsb7JUL7PUMeNOS8tfBsMyslN61GVkGcNHuU1kw5z9bOfFf5hf7HuhBPK6Lba/ri3R4zaRh8LaBLfmSdsQhvcFZ+sl4JoUU3YKLN1psdDv+UwvMGO33B1hBnAXw14LsMfzyWqbnp03GIAFZ9fpQ7WruNbDpV6MEk8Pj55rW7KF8fcG3kB4eGZcD8rQXvo6xS6OMImtqlZA43ucLlxWNUPcoTCQRH4oMr23ZDKLgU78W0pESzYom7eeB1Ar7AKTnUfKv3Mieh5AXSWORx60l7fsApYW/ZWanxI9LdiUfYNbZnsj2IuiftEHKxjSZFvkLjvK4yD1miN3je4MFsgXL3sifWV5rm99E7w9z/IxqQVBylSmjSAunJusUmJicmFgJhy4Anv1KrEKo4JYjZxflaJm1y0R+ghlg5k53Ir4ATLiEEyp3uw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */ From patchwork Wed Dec 18 13:04:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97739E77187 for ; Wed, 18 Dec 2024 13:05:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 268576B0096; Wed, 18 Dec 2024 08:05:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 218F56B0098; Wed, 18 Dec 2024 08:05:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 043CD6B0099; Wed, 18 Dec 2024 08:05:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D4AD76B0096 for ; Wed, 18 Dec 2024 08:05:49 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8397E120D9B for ; Wed, 18 Dec 2024 13:05:49 +0000 (UTC) X-FDA: 82908101052.10.A78A10F Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf28.hostedemail.com (Postfix) with ESMTP id 3DE92C0029 for ; Wed, 18 Dec 2024 13:05:12 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=klH6HxYi; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=caVYIWOx6/5hl1UH4GU5wSnkiaHbuqRYtNqTt93ePYQ=; b=gNe3ekYMMoBzBw8dSB//fIrJ6Eiw2yxbEn35qyGH6k3YWW2XI9GFLh904Uu6OONFMgKX3T tuvM+26rr59GjyPKhn9DlUITJ447apNyMmmlJT1rlQbGjFEwaWJoID1yJEtZ7UHtGfIXwj EMfJ5dsPHD3TojCFNgvAOC5fqUBLefU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527112; a=rsa-sha256; cv=none; b=6p3CX5Hl0Pqm1vLve+w72DKdHV+Er4K733ivos12aRVmwNb73rhDBd2m65ZYw/eqNJTRU7 RO2krLZpSQDRgksgFBzeEdQCckpMmdmYCrKZqhYu+kOJ2ol2fX7cs4GxaLy7iW+9aJlJDK FNCeafTR4AI2spq0MyKMrZcFCLOJ+iY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=klH6HxYi; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-216401de828so51710675ad.3 for ; Wed, 18 Dec 2024 05:05:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527146; x=1735131946; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=caVYIWOx6/5hl1UH4GU5wSnkiaHbuqRYtNqTt93ePYQ=; b=klH6HxYiSC9ZPzO0gLcSs//JBfLsrxj2nz+KKBboZn6FTbJ/r+rrF3oVX4Iao/grKM aVVXxG0t9DvHGqx3WvhILcuo0fdipRUQDGz2VH8nDUcPouKpWjvNjmxymxJJ4mZMJuzx DMK7m001pj0q2uFqRYcfQYfAkr84MB39UGu4AvMSC2b9WTgcH3N53zVvCTSesG43/l3C eLKsvORmwzlgj1AjPVEiJBbHj7fX/Avu1MFVrh3iT/t80Epr5cbQK1T6AwGHMgeyh0DQ zBOt9WpIh/vEh9f7sqsfo5m4OCsrh+BpkvvMq2k5BfrF26D2syFOtVvohSPJxaCIXfKf MqLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527146; x=1735131946; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=caVYIWOx6/5hl1UH4GU5wSnkiaHbuqRYtNqTt93ePYQ=; b=RluxGM6Zdepwf1QSYlO8FodPokz3ty7QRZTzxSLhsOBH2CDi+0VpAAix1dGGDWmi7L sQsePqnd/yTcqGou6Bmht5g/vp4XBRPtziQ00KRdAYw8oEu3lcVEzC5tZEk9U04rIC/A o4ZW/fO47U8DcCEom1s2HBXjCKCvOwOjjj4pUjjYMk7lksavzm0lRjKn907NzdAc1qrD xc0xHDT7pNKzbEMPFSTL4KMs/rPuy+j7/FiWY39/ZVh6EZOzdEc4JeXhHxnSYAIhUgl3 TUNyCN7YvEzHgYLCNGX/pznasJvJFDbalJxQDYCuIZdTO7RK6ijs1pO3Uc32o6nznV2r +syw== X-Gm-Message-State: AOJu0YzWOL6CiCYRV7ZDz3OnNGChMeqnxDbZ8dWILiXo6hoSzZovnUTe hPICc3qIMpEaX+uvfyQ5WNsNrsV3VX4w4d2XWHlbDe7HHLPaOfeL3CXKHgQhOFM= X-Gm-Gg: ASbGncufVLQb7KMcjPNwwDDtd8cOGQFqrhNRfXu+c4wqvL4tIS9mTXC1JVXjB+oabPj rMGzItyr9g41moPnnwKGg9tYvkXEH7nrO3JV2LzOkRQRfS+Y05xdBCUVcIP4dTbNHY8wg5m3L+J pz3rj5X8/QAO1RCBN5w1kEYgvsUFUPOwwPg6LmzffXJGVl8L9//bAgrxRCSqsWPn2kMumO4g990 bRWjvRYZ1LU11joKbtvU2Y15xJqL+RLntnpT06/Wk2HL9pMIUW3hOltNpmEvF7c5GqgKXCylrNs GMW4mb9KRmLsU50XRbkd0w== X-Google-Smtp-Source: AGHT+IHEov6aeh/QZkGUNU3nLpbBQcGuDGLxiZOzaTrM6oHehxavOqsLv6xgwuZxwOrOKdwEXRtt5g== X-Received: by 2002:a17:902:da8e:b0:215:6f5d:b756 with SMTP id d9443c01a7336-218d6fbe530mr37726365ad.7.1734527145692; Wed, 18 Dec 2024 05:05:45 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 05/15] mm: pgtable: introduce pagetable_dtor() Date: Wed, 18 Dec 2024 21:04:41 +0800 Message-Id: <0c00264b336eef6052c229b90bc9e6a32052ebff.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 3DE92C0029 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: qnx3wuwrfzaqwfhcpnqgrntnnnrsutdy X-HE-Tag: 1734527112-429393 X-HE-Meta: U2FsdGVkX1+fj4hhJvOeQWzV4TOt4UVELCi/iByEk2vsidqPyZcs68Bee5hcHmZlvaT5LTMtbMZ5tZa5nKUc4g/OLYQa+IaWc9UjPu5c56hhRgw14E37onG71PoFB45g3+kWsbwgvm4AJW12qv5Bl1flY3XqaJbHVwlQVVOwbpCcB/DGgYZSSBYfG3IrOBDcBvH9OwitgjOPiW4n1kI9fB+kY8AuGKgP/ErVzIko0PoQWA5NjOlp9hGOADHXGjljdtwYtsLBvD/oc+uhGabHn59t1GZpUiZuuopXl5pVYzg4N/PZs2ZsR/sgUdBnOTv7+LgvEopduKPPX4n372kuJnNteZjmlkxwmmIA+R0eOai/9m+TSi1EJ94KoLSQQmN1u0sDrotmXkU6DjYyaWH7dao5oFV0svt1y1n9txoZMSBLzw+4KV12OWDmYO6OZ5nxz67ngTOdRqxrw5FEOTSUdaQ1mh//jbL54IoYbbEk0vRloaJGR650QF0ChcgdkkJF/QRVmXZVYNgVC+Qrx1U1XBVQZaJypm0rTlwFEmVoLs3FL8NG1eYGFRD4nUhY07eWvm53D5ZnEBcODLks0RnX6uGyomNyXb60NQE+xfLiMirnokFsugkkyR/7srcVPXfwmXKsRbSU3GEgK+xY7OmlKUVr5+9Kb2gsxyPGR1G2GQTkgM+PsaKiCL0b0xDin6G1JToqlp/ZarLjWyNwYAoxNkX791HBC7sizUz4NPv8YKGi+Q4XAuBqbiMCkbA+Ha8eTOp+OKXGrRbfrfLhWQ0ZoNS5mFnm849ywyWGjSWhmESv/X3Q0Ma71sudELcE85r6EZhR9e4WlacM9g35IUqBz1yj8QFtR3JnttvjM/DZkIFeBUvgv4hzl2T3aSOGpPx317YT1fZV03TaX/k89MkLZO0Hny0eO9eJVeUOFOFjpIgHXdPtlp4V/4PWJJ0NdGudQIjmg2ZXgnJ2+1zrXUX GWI9t63z 22yg0HEDJDWyFCVLtSoyiXNd9Q00PgB34kOsF77Pq+3Qwj7X6QnfCszzCDARQCcTfySectblq9cpBQTRC6fQoVWuDv4vhuApd0RiI05/roynvdLU06DmdkWObmZIszXDBOG6zGEXeduIplVrf/tELHw+Q0qjpZCJHv4QothpD35sGKK2crfavLHrw/Xa6s5jeDBam3PVi36GmrVLKuPPEv6QWHAxe91O2j8DAPh20NleSqK/N11E/PBICoNlXlSAhr0ZVDWD44GiU73p7TC3LbjH1ED4cXoEOxn+mVhJ0SwIkk/JnEHXoLO6HOyHHy8M7vLfaluthW25mT83TdWQ1549eOTzYk2aWI6WaN+7Hc58GaxHZDG7RU7M6jk9SyozsBukyHs6CegbGRBjKKLFOHR1GwgW086EBar52nT3k1rWhk4AZUZu5ecmr0YFVenXPVGeKGLIYyteSkt6/SFbKcCwmD4XhcBE6WidIpv0efXbHtMw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index bb6e1c5f1fb19..09c2eff571a49 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -112,7 +112,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,7 +139,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -172,7 +172,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -183,7 +183,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index dbf61819b3581..3673e9c29504e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -257,7 +257,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 7168f46ad4770..736215c428293 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3086,6 +3086,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3097,15 +3106,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3182,14 +3182,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3200,7 +3192,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3225,15 +3216,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3261,14 +3243,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3277,14 +3251,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index e064f5edb7c56..e93ad42806478 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7104,7 +7104,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Wed Dec 18 13:04:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A41FE77187 for ; Wed, 18 Dec 2024 13:05:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CCEA6B0099; Wed, 18 Dec 2024 08:05:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 161546B009A; Wed, 18 Dec 2024 08:05:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F11E36B009B; Wed, 18 Dec 2024 08:05:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CC0AA6B0099 for ; Wed, 18 Dec 2024 08:05:55 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 68BE4C0DC3 for ; Wed, 18 Dec 2024 13:05:55 +0000 (UTC) X-FDA: 82908100800.19.F670F81 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf09.hostedemail.com (Postfix) with ESMTP id C53CE14001A for ; Wed, 18 Dec 2024 13:05:31 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IwHLwzm0; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=kXM6JASTBaz3Yd6pfzfqxgbifp65LuwSI8OChPEVjYBPKRSr19gPwlJ3CSZfUV4GFa16pd 0rpjiD3SRdCCAifY2hXFDR5lt3cvOKHMNSiEaE6eAwokocbJspvWSZA+CmMmoL/1brew8I EJIdqGWH/YNurWYQJwlccPS9ha4dMVM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IwHLwzm0; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527129; a=rsa-sha256; cv=none; b=GzdC0gOZtsMBWKut6gx5/wJuglJwJHVRWDiluMvYJeO1qlSL3Lj67nJqvL60LT21eC909Y O8L/wKZ30ljzw7mREjZw7iptYIykfeLYLE1LgfppE01CO3tmcvRSEesrgqYuH51633twUs D2GmEvRT3+GhtxPbLoQA2dCdGK7V6FA= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2163b0c09afso57214295ad.0 for ; Wed, 18 Dec 2024 05:05:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527152; x=1735131952; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=IwHLwzm0yQ8V+eDK4LYBT1dPs2QgxPMbTgRIb+sZGERcZN5eYakVf21S3+iE/CRQ9b 9MDfWkfpQ7l/8q21zrv8wj0FeQConjjQcXJY4VeXn9RKDDbAYkIDLZUNhs/Mcze5JaWo t3nSVPmQmjf1v6Hsv4MFkdDvbIEs/EtvwRK9CZVA3ez22FDw/2YGghQt7INxRJKGJI5r TsAyXgdG6bm/C5FuPnlrr7iYWX3gVApAgO+sOFsikpGTr1zV3D8IkV9bStOkM3bEIy+a zsIQTPFPt+UdJnyp8TngsINSJ6OfoHAloZdFnUGLZw3J0HPBwjD39ISnXzfUwvS9xcsw aqkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527152; x=1735131952; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=CEwrwn7R8itkOpBHyV6xDc8MtsyqH+XIP7SxV0kWBeKLU6Dt8rvFWVu3LMshrCNbpR Cd0VsuDeDbAJ1bazNw3FMUbFyRYMysu7BWVTGavnDhGt/K2kXMEHA1nLvT5vLVOxZ/t4 v99efd/Oowk5QVGA4Ahew2ItV0M77hNkvUu83bsAtw4WSkOi50VpmgpKJc+8WZsMg2WK pW+K8opM0bMmm0cKj8m4ohPTP3FmnjYp5vhZggZKM3O/g/uGXmn0yOKYcXZ1Vaak3R/5 tIHYVyr1LYwYDsRdl5vWPkxvMrWyDKUxrFZbithFksIuFSD0rJS7nVmk6mRq08HmlrWw AYUg== X-Gm-Message-State: AOJu0YyXOsfpC3jGrcjrAqcjEUznODh0VitlGCZWe1X0ar3tW49E5mEQ mZkrNc/hdPUdoJ5jdgxBROD5XUxzNciyJ2uZHKsHQkD7flkMPjYsaBduYvE5xLA= X-Gm-Gg: ASbGnct2Bv2fGWohNUfS60KyAdMJVC5CZVHWw+Al6wUqJNatamqTYX8kFjKGkE4o1+N Zr20SGHVSz/3O4DvR5kZkLSXJOk3HTQI+Zq2/vqF8+oP/X/7gb8bNz2zl7JV6pYiWKTKRW4qh/X p93S9BLsX0wGgietxGml67PGvuMikgHzcdellDV+RqHnW8vCOUc/Ot8jzk2A5DK4w/8872qNVWJ cOFyR3S81lz6OVh4bewob/Igk2LMBa6uoMzhZcuSu5S1yodv+Z1SWiUilwtmxSvKJXAZMiIvcvS 3Ro97wK7ydBJIsddoz9QHw== X-Google-Smtp-Source: AGHT+IFdeNYkaToIEDdUvkBTd8lP/0QrRZpL4HARu9WlaN/FvfyhXaaWSW9JhSu0jLr2tRkQ6EUdVw== X-Received: by 2002:a17:902:f68d:b0:215:b058:289c with SMTP id d9443c01a7336-218d70fe050mr36430515ad.8.1734527152123; Wed, 18 Dec 2024 05:05:52 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:51 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 06/15] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:42 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: C53CE14001A X-Rspamd-Server: rspam12 X-Stat-Signature: fx5m4pgmqodoyry7pjnhkg5iobkmnur8 X-Rspam-User: X-HE-Tag: 1734527131-950737 X-HE-Meta: U2FsdGVkX1+DIyXWFtkCB3THN0/X68PgROS3CWv9vs8xvjMvGqkNPhvCdEPJsUKwFFKzBjiFWadcfWUCovqgHqnPjd2c6JPxHxDxqU1aPJss5PKazt+EOoLqB64092lU8X/z0HhCkgBo7UWSbQ77oN4AWPrMkDneS1AbMpMGYSE9/FdTQw1ZqhT7RU0go+kup/7qL+yeMsYorM1b1zm5Qan6tQRgUKmN79/P8k75PaBnA5fD9HVxOaqWZXpdFTgkYLtveySEusc/dpLuKRL+5VbKEJfeks0WeRP29SN7WNO8qpHpDeH81NbONmtSHcooUEtGMlKFdvBZc/cBxDGDWIdICZIOqqXlKgciPviK2e9B2g9MmGYEoymHhf/AK/qDpoBmeV5zgYV/09tWIWduapmB7L1+rUWNTRdi20/Ctkg/4fBONOyda1T84hCewCwdLZMhGQKnHGHZnAk6+FG/9JHUwCbQGMiIgEgAUowVJRBhWa2Eic+SiObnhNZmkJ10awASEgztV5D2DWRg1VtNbo1FOYsinfOHnCEaDlXNNysJQks0+h75ws+mOEW16lhOejn0s7CKsvUZHSe4H06S0JydHjDzXuV3y2XKMJ4u6v2tNfkY9IYsCN3FXj4dTyRydFmI223Vsiz8Fy9YY3n135XygPSlqvkgxdaP3G83x1cUMZm5BrlKn6Utvyr3lb9DMbfPM6eLRP75KSjUFKgGbGtCmhcyCgINOngo9n6ZwhqlFNOoAoODEJZdKRi7IOKk1B7tKrvrNU1Hyw6URtHqYKE7LxXXf2YheDqfUBTNpyJU6S3uw2Jh7mBdHv5e8AOWiNLS3rO10W42N9B6Xy3dtjq8bTja1iA5WyUu5DigB40YD6//xIrIhR0AOQ5KffDMDH145gWSryuaLUYS+os+bHlxr6NO+3yE8W6YDkxyk5ew+5/MouWGp1PDunmikpI2vABWdln6skBwJ/WFMHY 0nXxjsaV F54aGU2r/r7VgSfjRXC6txG1GbYTTj1+EDPkD/nKB9HvlilnEHV+Zy7gYP+QVseHfZefcQrT353JjJm+SOz8Qvf8/fyvjk6TLY99DWTmMN28f+K464Ga2Badhxk/rpGHwuXMPsKj1SqZafOVkDg1t+Op4WpQAzvrTjn0vvBKk7CAlxpG9tRGSYVtsqt4XtUqTuqaSYPXQ8wRvTJvjSonvaszeNBXd85QxnMqiXPMpgj1H1E5tQNVQ13koneNN8qOlqwd2U9sznf02/TJecJ/LH8LoXRzWTB1KRsnIBhn/a0C8LWLkkQ8JnQrVQ77rcg+f4XnZeE09zp1PO4q1BgZlFIdBz04oGSY7g/TUyc25OjFC/GhkucO8brn9oUHmrVjHj1QvHUr0NYAq58qwwu2q51zvlQlt+iq1+PY4DDye8SKJFSVAzdhxyjyH0G98D0smsHc+8mHEfGardJUN8iviLTBtzA94jtcFK2W+8HFvRTKWXNo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Wed Dec 18 13:04:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E80E0E77188 for ; Wed, 18 Dec 2024 13:06:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BF466B009B; Wed, 18 Dec 2024 08:06:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7716D6B009C; Wed, 18 Dec 2024 08:06:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 637546B009D; Wed, 18 Dec 2024 08:06:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 44DBC6B009B for ; Wed, 18 Dec 2024 08:06:02 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F1EE41A0DF4 for ; Wed, 18 Dec 2024 13:06:01 +0000 (UTC) X-FDA: 82908101598.26.64A41D5 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf06.hostedemail.com (Postfix) with ESMTP id 9B3BE180020 for ; Wed, 18 Dec 2024 13:05:37 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Jfc3Ci6w; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527144; a=rsa-sha256; cv=none; b=7CH311fyoTvQimLtFpDcC1CQvcOY0DpAT0iEF5d6hSVrwFooNO4Pz+mwxK6+vy3J867jGx W7vTb2kH7GO9ulvc5Kd7dfdYKejHZvYSOzQsChWS/bR5DVQa69HaIBrIWLLRmRkzMHOqdF s8XBSWwB8jvT1wJdV+1znBS78J2O72g= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Jfc3Ci6w; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527144; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=q6rpj5tOUadcdxb/ebFIm+k/AwiHULmZDCDwSUNsmGPf20hzUP2C+2FfFvtlrNYkIJnCN4 0ovwrDgLq2f0JYQ3GYrT1+tFEHar1Imqqa4XaN+ps2bVbyVtj8tJmN98fpyahbWRt6ioPr G/vWJpKf4YI6AkNZKgl7Dc6CeWXwqk0= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2164b1f05caso59754215ad.3 for ; Wed, 18 Dec 2024 05:05:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527159; x=1735131959; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=Jfc3Ci6wSEL8FZ55c+t9fBBxQes8GUkXt3DIvrDVK3J1IWQQfzDk1o9v97ROA1tpqO q/GF0Si0QuTEm52tnMM5fuGWDrDu+o9NgVfISBaXnTe6by93ljDurXuYBikyN6LkbQbM DK3QDzWGPs3RavUbrjObAWeVzFc7U85lP5A0KYwaGKSK/aNGItFcUqj59FSjmhrpc49W BFU9TeoNssCUHad1ew63uGRdQzUZpw5/KwFzrOoIzbkchjhJGaYHNg5Ez3sFzqh7P7gB y5a/UzaABXayxQU4Sm5UtAyzF09kmMUlE/wgJ4+e8Ek9pwji/UZKegVPyt9sPI4crS78 0t7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527159; x=1735131959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=mvSLJuiQZC+q6uH2faF4HGB1lsaPHC3mpjsKgTl9zB7tbvVWQnzho/YvpLAgjD+n6t JAOi4IGZhPEHxjQ8zNrMOvtNleQBILitjpHyugRuC2rT54ePe1r6EjzwO8NZsn81iha6 d0dq2Piv7LuQo6mOicOi0CdYJZuav0kHWrn5/h8wy3dDCbuptMWlxO9FMaj4j2z5qfmv bLqb/YE3zazg3YNSdFbu3nIyaa8Kwwb5sADfOXrECpOancPDoYGp+FH0zpPEIW+HKV19 SoEK6nUjdsI7MxkWEppoNx+dpnJgT5Ayv+I11Ct/W0DjcTaK2kEGvoV+xeIMIyi+zWfN jnsA== X-Gm-Message-State: AOJu0YzQOfwnKT56TLqVTZ9dJt7mp41Tt7SmyHMApwxgBOqU6RkfpC+x pI/rflOmpQu4OtNgNssxGwunIAc5Sj4gou5Zn25AP5S+GgvHEm0NKo2Yukvxhyw= X-Gm-Gg: ASbGncuK+L9+8mcHn3WKaOADjyRss7AIPf0XvaL7dNbHmpv7L4ShQU4Jv4JYHVmpKB2 iT3XxS2bNTTTYbcNvkreyQkDvKwQtdmm2HSu/H5LciAN6K6ABEIeBJZabina/9A8nRjpcufBacw GGiQD+SW7oPAs+wCJ0vf/RrlHvhsvrk+2r39M6fLHYFT1T/hKYmBrpw4+0fO3Quc/18FWOVyQOK pxBisVWFJU1vLDfSS346TZX3qovj3hTiWYp9X9ix2o6mIHVEtmRjYDRjO0VsUz6P0XpVJnGyitx DqeNXwX7ZjCsq+fkNYqNgQ== X-Google-Smtp-Source: AGHT+IFH9P+x/Mfn/YDKtSKvkH1s3GazvpYiwIZf55I8HG6lzpeJuaV9oSxqUwHwuTwe8mBCoFXtDw== X-Received: by 2002:a17:903:41cf:b0:216:4dfe:3ebd with SMTP id d9443c01a7336-218d725ebd2mr32859975ad.50.1734527158730; Wed, 18 Dec 2024 05:05:58 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:58 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 07/15] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:43 +0800 Message-Id: <634f0236b4b4effc528b06e108b7bda51cd0ea2c.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9B3BE180020 X-Stat-Signature: ir8zz7quz8aed9t38wtpexihgn6q7e8o X-HE-Tag: 1734527137-106323 X-HE-Meta: U2FsdGVkX1+kyEvgYZcwUoC997YbuTcuhz7UZi+yUyTNt+26Sxydn+DrMB418cmTQiJoOY1GwEd3zIF5Is3r6bI/vj8I0r50NPLyFju+7f7ArbsJEAhlp9FbF+PfZZB8sgu65xtiU50PJmKV2cewRJlx2HX74MY5fvCVOdcM5NiyV2tOJHpzaQUTap+d4fkJq+J9KVsXnFsI8OAuZTm0QXP4zZeu2iX9oxfJHciQIIxAD7LMvCbM0DjSy+qZiNL/8mJvDP3N2aOKPbVEc0xDUauO0mR1aN15ip6awPKg5wi+H66S3cZkJCpWZApwuGBp5vYs999TZFgxiM9xW01A+uSc7UN5Po/dqxxqreZr/XsNHCZ0/jHfkO/uiDW3FTD8DZtVM13fsmFxHbdX/eHvdfLswoR9YeplbRXEXoOQ71qpjP1A4ixhbsk/cpwiE5hTGMbUjeEu8VsBc6pfGd4S9D6gz+CHMNpWOr1kplhv9PKPhbGejAZgWzMsNdySBbbawYgfhWwVJ7mLz8yL+S/jngzgK/GLdbJUFF/P4CfX/GpUUujKwkTPDOgN6BbfBZTbrpeKaO7PnrMubLI4OyuLJ9jwOgPh9JiIPBj9gvmhrrUQlbUwU+8ujFXjXprg/02y2lwlQa0TV0rfD2W+/G7/J7/TfuPnEDCRXfatLoYGsi0MvbqkBk8+rtWNR+zjXYk6c/1mOP37QSjlm8nyq5tTh/iVu4e/JYy9GWFXItpDS8nn4S6/HjP35umXKi/0bgzESRRukQGqC7OQ2R9DWGw2BwRArQyiuRpbZ5WKbUQ4tevdXckoYQzsxHl8NN1wgLnWTjlJMXG1LhY1MT2YM7kpbRBMLbSRr9pBfCNTOPkqn8/zRdSoFW2YMuKKd2t5SYVcC42H4QzaTTQgPiNly4XNgsr9wc5+VI0VNI2wP2W3CB3yjTAhc9LeJ1nDUWUEtMGCo3B7eEnGzQ7KzOhpi5L xoBuTo2J HkHOXa8vVJxPMD1f7zjPMPkBc9MPayYMt5aBRd9598KeJsxt5Xo3jopxqmVoOHmhDAGDfDS8y+ZP6P7nfbKANGfXCcRVTUpvaiok/WiJ6JFzKQVN/8vvCW4hbhk79d2bWTn68Rjitt+bt9oIF2AHN7Ka1iCXNclZwAAYMIPB3MegAaH0I/HNhodVwzFbTxWLCOZZj8L2fJTxnycpS08UWYUQb0Wq7k+jEmxbXMOi5oGju0sRjLgctM/lBcDkFWpYCtCMqJY4kI0ONKy+4bhKUppe0ttYauRl9shEv7DEFWQC4/e5uFMbx6zjYHPrbLO1+jI3xsK7RdxbM9gBKKPaG2ZswpfCpKRCj7zoEN7wppwtzkgk3Og538Y6VsTYoAK9pnlZdmPBRduy3eB3kWVY0M85Vvu7+Kmu216zdtq7HIbz7a0ogN8N5xxqsZRvUC/Msyr+BBMvGT1g0Il/hivXetkq8cY3ngcBkuMmqf4eObuGUQ3g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000249, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Wed Dec 18 13:04:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBA26E77188 for ; Wed, 18 Dec 2024 13:06:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 798E66B009D; Wed, 18 Dec 2024 08:06:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 748516B009E; Wed, 18 Dec 2024 08:06:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C2FD6B009F; Wed, 18 Dec 2024 08:06:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3DD986B009D for ; Wed, 18 Dec 2024 08:06:09 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D62C180E0D for ; Wed, 18 Dec 2024 13:06:08 +0000 (UTC) X-FDA: 82908102228.04.4ED1080 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf26.hostedemail.com (Postfix) with ESMTP id 745D314001B for ; Wed, 18 Dec 2024 13:05:42 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ca9+qcNA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527136; a=rsa-sha256; cv=none; b=fuZ62W6GplNKrWig7Un7gUs6A3/svziSYry8F3VcrKCDDGa5Z3flo5ei31LQxjgL3Rcu8m Cw1deZauPUZRNZ/paSTUJB3cieUzd9YCDV0xhlMZGb9aqFbiDx5w0vuaoum/+nYBnGZxZI 3ks5uavCJlxSFDPNliYBvfoqtms5jbA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ca9+qcNA; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527136; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QtLDYyJyRe/cOzqFg0wwVyja+MW3+zRiiWZkoTiwjPI=; b=FnRQQWYdVMLWW0Ou0q22vp2kLIwkbtq0iTufJRSZCHT4yFqM7nw41QvEb29EO5jINxWGE8 ZO/h9x+KfMOgApKHfHYYGQe/rx09N6QyByjaQ5gR002sFhCDkWWLKY8Molu4NoeI1JtylN 6Rx5RPQlkKailHO2nZwvLhkUkAhKpBg= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2164b662090so51898115ad.1 for ; Wed, 18 Dec 2024 05:06:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527165; x=1735131965; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QtLDYyJyRe/cOzqFg0wwVyja+MW3+zRiiWZkoTiwjPI=; b=ca9+qcNACGlwFTuzSdet28VMCCw3GnC80NnW3QFbgkXvEk4s5BWp7aHNnfvqu9tIt7 NLV6ouNAAjeqi2JIU8ToltaW+EaJyOmkXEU1mTjg5qUElpbhQPhICgXD3Fqaoh62GWba PFP3GAT1EG6Paz5ziV2XO7XgMhPY6pKe4hheDZPcFc+0pBeAxoaYAJ4FpNZak1hPKxKR lvNUXiqiiG3GVgs7Ph5W/LXaDgwGzjHdcsix5Fi0A16o26Zhs9GUQKpm67Yf1MMfu3yO BlYUGWEVMxiw6z4Uqj8RENRe55soYYXySQswSa2XrOxVTljqmKX8qcoTfTEbzNY9pJQS Ie1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527165; x=1735131965; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QtLDYyJyRe/cOzqFg0wwVyja+MW3+zRiiWZkoTiwjPI=; b=fxZEChuS6cwv9Wlkol6toODOmp1cY0GJ9QEeUhtHQlQC6SJoatVm1Z1AEzpEsiRn/I Hq8Tih5Y+ErBBW2DNRTz30HmVElJ29COBVDCCeBXATu0AcqiTKOWfLP8f4VMH1tbWFcL io4q0LhllHwedSQBrDCJ27Vspu+CIVNy/ol8vzcXOt6xr6edKEzjWHvYr+0foAuj0KDT md2qi9Q0lZLvVNnVSoMzRticb71uLfy4kOoolxbZG2QNRpVFXE0HIM0jD8SetldsgvFj P+4PaqOGofmIpagPHRwRCJ4/XnPyPxUPA/BWMQHzXSkcXidnrYSc+FVbF1b6w3uS6htO m+wA== X-Gm-Message-State: AOJu0YzmKrQTZlQGibjVx4Ja/M+/c5eGWTaEHNek1mmCM3W6fGGiNIan Ewd3NIEiS7elzingoDy7F4U/4SNlkyA/QTH1mMa3msVd6XvoKVVMBGPreKKE30k= X-Gm-Gg: ASbGnctNpnPKJ9Cu3uoezz5bLlrxYZP9GbzsDISPkzLaXnTe9nKHONoA9ffZtV2W5YI 1+iTCVRSnIn1X+X7kL7z4rrGJEpuQOTZaqpueWzPZVPQkXAS7g0zoyMqOi2goMbTvPFoC6Cnu0d 9iwdPugKRDDYP5hW3BzptbVUkcm18XZyk/5dB/Eb04qg+uLse+uxsYqg50JdGgukauktNry2/D8 e7FVXgk8G7WQJ8CoLtOq23e22FBCqe0Felmyu7EG8oUfdHQ1xbfqbhRlQNSDouAMRDCWHB0TBCp 17JGFuUkLlP/dobYJJP6/Q== X-Google-Smtp-Source: AGHT+IFCnYr+lY/2cKu1iNZU0iqph0CddR5mvkArPJVeCIaobzeDvVooRJFIjfOJ9AL+45aiVbBYIg== X-Received: by 2002:a17:902:c40d:b0:215:a05d:fb05 with SMTP id d9443c01a7336-218d72161camr43962385ad.32.1734527165431; Wed, 18 Dec 2024 05:06:05 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-riscv@lists.infradead.org Subject: [PATCH v2 08/15] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: mpsciahrdx954dr1zgpi359uzmq1es9z X-Rspam-User: X-Rspamd-Queue-Id: 745D314001B X-Rspamd-Server: rspam08 X-HE-Tag: 1734527142-460155 X-HE-Meta: U2FsdGVkX1+MF8spNkTNUzLV5T1r59dDl1NR7fVA8qkWtWUVUigH3sK7QYpFDE2qkZ9rc+LNZX/kKcDN/0li1avWZ+D7Q2CKd65gUl8bmIB85jW5vpez6zswUDH2Hha6xv4JkOJNKPbftyo5W4KUFla6L/poyjziJLwQjEMWWd8b4AJ0CIXItcAYmzx8FQRjlkq16F85TBneVzJRMS5qr+TnhXOHdefLZalII2BDdhTltNGW9c51O94D0LEUpia+hCL14NnLGYM3EL01gPGAWh+QcRJMvShqihg3yl+2dFPM1e6384nUmLBCG+uV9F4x++ZjPNXlJyqBTLBC5bCk+rXdjFROaudlpaM70yzXD9K6W1hj+H4AvC7oe5V/v39OEQb0FgPX0qbLaUZ/hMgh2FWoha7/FAOjZRGM+5U9br5IxUSZWLn7rqIgI0HkXwIdhbxPHoHgKQ2DaFhgouWAbi9mxJd8P/BR95LkmYkLT3Elrv1sXLBECDbXMd9yzYe3iei/JeWe4toS+M5il3vJUG894/QtJPvF4vX0c9pi9YICF+wCzIstuJJ7lHhxx+49jW3H7nnG1x+wOTTDL93Cd4vvTVk00nQ8fZE921XAqHhSoNfIu8Tl8p0+zMSDog7LG+VbZqUbYI54JLspVuaEoRsluQo0cdcDgeKC3rYsdn5VtZsA4vHl/z9B3klrogk9wwVDPdoTbYK7RGyc1qb9MdC4GBZIPukqXYmKSVRMWWeimW4VuD2Ii+cr/Ayqmh2kcz9XYDtm6bwvwXTJLKvoj4xBzfWyQ/YEP/UkAPSHdrRDU37nZwNHOIulhLxipYLqSsEEuOZmm+8uwJQW7+CNKcbtSLEzOpd0/TPjPWjoaumiKhcVmrPh1Hcf4jBXWvQqXesvEyI8AROuy7tivH6bKiyaJdag8GUdRERk3iP9ZBfGepxAMjKFdtn+zRsnDRBMqMoOcLZFdo99Ssnms0v 1ehd8t3h 5k2TODpNA/tJn3eeEWPpTFzDXhYTRd/ua93+/s9E/jBO34maoBFQthA414/FXHRt8t0nNVupv1vXd3I00XN83LPqu/8l6h+DklLET6CDITBrQaiIZecn69E5FvrN7zYSUbkHO8DNhorW7OLZMnjNwPZO8pErF03YHkSpTrXUtTOAY53RSRgafTob1sQPNDtMcRyxE1yZ+RlyA/+dpnHZ6M82i34GGR7iB7BMKPO5W9x7YMdtZlM396pjBDDVJpo44Xp39Pe3S0JF+e088Xv8Q9+mTgmwUtDGBtvXbR1w/0Pdl7FlMF7XO0WuEfpEGJpJcr1qgxwGYW3aeTOTZq8BWdQTvNbRKYeF3x0j4CmhFTnP+rwVxQgxHoEHiIiDpxUYdPfm2njTP9Yt0zzgywS6fGQH7Td489sqk7Xspq3r9sY6PFGLWZ93FHj4zp+rIzQ7Lt+0qRJZxixZdMECmyrmFBijMIGr63yYrxMzohrA6nM95zZM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000261, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 09c2eff571a49..fc50d14010246 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -18,12 +18,22 @@ #define __HAVE_ARCH_P4D_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -109,12 +119,8 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } #define p4d_alloc_one p4d_alloc_one @@ -136,12 +142,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -170,10 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -181,10 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Wed Dec 18 13:04:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C918E77188 for ; Wed, 18 Dec 2024 13:06:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF07D6B009F; Wed, 18 Dec 2024 08:06:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA04C6B00A0; Wed, 18 Dec 2024 08:06:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1D3A6B00A1; Wed, 18 Dec 2024 08:06:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8D3676B009F for ; Wed, 18 Dec 2024 08:06:15 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 097C9B00BA for ; Wed, 18 Dec 2024 13:06:15 +0000 (UTC) X-FDA: 82908101598.08.15988E3 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf19.hostedemail.com (Postfix) with ESMTP id B1FB31A0015 for ; Wed, 18 Dec 2024 13:05:39 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LS0E4H3o; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527143; a=rsa-sha256; cv=none; b=vpyYFr6Y9CzSuyJ/gtzsrZL1HUtkO42SFHWD0HcvbPSmBFixrMKeVpVsGvCpKeIz6g+sXv 6GmCzOdxwASEM1AnMrl/I3hkwTQLgeleBBm4TgchZtxKtGG+jt/jB4vX30cS9OVlqHlAdP V1VTYhoP2L8lISwMonJ/haDw44GMu9k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LS0E4H3o; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527143; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=DHiC+YJHlnZGG/YLMqrpiwIlsgmDsJlm+LOywCd/mRNW/HM862nRQB17Vzts29otg+5Jcl qZpZZMkKgjf/RpWmLD+a0iA+LW62bkOn9OeTJCRWTJS7SHaBsV6FBZcAjWSunzL8J64dbe i7WCc5ulEnZqE5pQWXEj+CQ2yq2xHhs= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-216634dd574so37778305ad.2 for ; Wed, 18 Dec 2024 05:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527172; x=1735131972; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=LS0E4H3oihi74qCvCb7FgztFV80hzMNZUVhd9gkSJDXQ7LIWAaipcKZdwyNwTSY6Gb TcAqdnaccESEHDbJqkbS8ExLCM5gO1UqbPX3Ec9iqUAiTI4RS5qltMDgPRpm/+OYI/WO d4uQ1KdhfCiYa2u4yO42PtNN1+Kb7mFIR9iUJ8G6sD8NxWmTZq7ZvAS+nlczSlrbWFQy drIAAjIxDUysdOfMgM/IthUKn0ANbI0U8ahWpgId/mXdbze/qYOGfWZJNI+mx7RCFPkn EudsphI7K2ESDenJEOmkNNaNLQ4bJEJePwhJw7u3S3eXLeE6L95LtjFswsXLNy1dhnH2 ypcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527172; x=1735131972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=wGe6ZVQLwXuoh5PcL9M7LDZU/Nkze+H5Zk3kTOoCB83mIoHdAkRKTByAP3R+p/qbJb VfF9JZAMQdGw8m1WR5MdgPQAQS/sFJg2nbB3SSlWsRA0w4TOap7KHjDMhKljQyAYR2tu 44jNjB/IBks/HrFiVot2Hi32xKYKVAZ4J3ILwNbXuJqXsJgaXT9vAATKyt0DweS5pF1i ACRn968ibmGvL67+nTbcM8UYdUhx7D4dBwP72u0Wvvtv28Luk09oyFSi9hL9Q2FHYUAL wyWhjvo0o4LyaBSN6Ah9OAN45SDyAwKaTbo75YbJfAvfThjCEgeOm2zrm939YXsT62bC oxaQ== X-Gm-Message-State: AOJu0YxJWF/Va8eXoNlHmnDosMQhpRVz9WqC664F6Ww5nntk5/ZoHWQU iitP9aK6Q/2EuWa47wMiDHQwefebE6MBd09BeJnWqsMlHPZFngaUuKB8b6ItmkU= X-Gm-Gg: ASbGnctwB5GYwJ9O9VpBHBGdLZQ6poTerzkhK2aqpvx0vApfmodsPyt6sjNdlsOQn6y Ehs3vNKe7//Vu4cJMf/fyS8PYOHrnqHIvfmRs19IlwAEPeQO9gU8441NizEuU+jgGbHWgIPE/ZM 8HAuGTjXqDv2zTfZp5qF1HdwgETlWtLW84dGBw6irbzW55MzlyEOq3kLpBNfhu8d3A1S7GgJchc wD4ZxDVjS/1T9TNfOswR9Ml8JqUNQn1gDBJH3odHRAgCC0BS3qHBPr81q1aDFbGeL3ed5KUzA0v hBxe5v3cxcXGwGogx2OT7g== X-Google-Smtp-Source: AGHT+IEwCZCDbOJ1SqMAwJwdz2DwBH3whjOpJ7w3Lr+F6G2curZkg5Sqfpo28wJQmsWqUUWcuoXMow== X-Received: by 2002:a17:902:cec3:b0:212:6011:594a with SMTP id d9443c01a7336-218d6fbe5b2mr36617165ad.3.1734527171662; Wed, 18 Dec 2024 05:06:11 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:11 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , x86@kernel.org Subject: [PATCH v2 09/15] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:45 +0800 Message-Id: <8c0d97d2a538cafb94cf30f5c94a84a0f93f2f4f.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: B1FB31A0015 X-Stat-Signature: 13gkb8q4tm8pqa4pj1dmw96qm474k6ik X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734527139-245218 X-HE-Meta: U2FsdGVkX18Z82Pcr+zI8pcoiVRbtjj7aaa7ruicjOm10FB0m5cGJqiDLmOMJUTy9cwyD+6fWphnsPhb+ffo7ziEtW149xCmMP1fUmB1kLCUfS+Hd7TFUaRQVnqTyqOc5btNJxXKbfzaAv28AUaxOTlXVJHdemVXMr7sDSYF+fhfw+VnHWiHgFNwtHWY6s63RorkqFrwbvpdqqDO0VOj3oqkLGCjYwUknl3IZ6nQ/EZNDODjJ5vUUyARXOUB0JY/tc1lRbsx84eC0nOvSsgET4oGI6kI+dtNTNV1R+FZjV9fQpcXbNe60QXEgY1BmawNGbW2ccCAMAE1weXzPE56ohlcJSoRknn1pClu+NItqoYDYBaTjl4A7dv/BbY9ex0pUvTOK0Dj6rcb3nfoQaDHgQw49aFZip3syfAOB8/QcY76l69/4aZVuTzmk5LeioITSCzlHyA2UZ7gICYpqCqpAVafBHf+tRxa4WPZWKgkPq18/ZpXtroJuwZWkwgfUQBJ4pPUdFo3lpuVhpciOdpAtx9E9wunNlvzlSf5ooiELlAqMdiMUZf3zqKKaw+5cv+0mpmL2WrL8RRuFwYHWOc4TJ66mPrYEcKZ1DnorJnAbTTdYagP6QNKbfgb2Engykvl5nnH7uGEoo0/0g/gjcOuIVMNSffwuKewzfYRJcY/fRbEr8SMmEIvev1kVo02GJW4kpBdl4jB6GGrM3eNYrEQJCfQ5CJ1M6g07yiCe2aFr5tEy1yihIerUYW3ftcFi8S3BLnkpmbo7AELUAgMEssBbb+PAV7FdRfS5UTJ5lCsc44ZHfMaK+FGQfQbWQTNnz7bxCCBrGpVSRexhOQBT58PpKqyUnGFRQc/vFvaYML6I4G3owcxFj1GRxijblfnWTg2mJeFcyQY0rKzcthTXkF8QjYR6NJaYWnxLMG15Y0Xx+ZqeSZyE1Xwl3fiIUVaShs3r+1Juc/6rwDEL9TSWfr 4Tvx3Inu RKSx3HujxJyon9nPH8JgtaZ7dVqV8m+RtcvTAhRGnluUZilyk+3ZuTm91b0QZOGJ2drPvsvipHOCkr2Hju88jaXD95593gnCyQ7hUWT4cY1oMx05WeTBa/ZhcQZMzfMfHOTL7CsukuGv7YGiP3ueB7pGMN0oFeQMiNghvanbefq27PpVl6B+p2b2XPuIvpY7LckIOpE1qisVYbkRIcZeNd5bzhrwIaRJIwiq33swBaWHTObDEhwiqct7Hno+N/sd4ywDi8Vw0v9pDakz2RbH06rOIc1ZVpmNNWhV4q0PhaldCp13LKAJvIqRtcWv70jlCpEsZAwISvmncSHu/ygSQJa3bcJRvpgS4y6VRIp4kQu8Juhs7agBesTv/3hOrIGnzNm6aEnsA9W77V41E3kupvC0dp9HepUfbcTllpdlV2RfCgJsTUSlVB3zD1MlLxHgL9m46fGvlyp0ye4Qsk12FjNgdYcNy8xJkbPVxQWrVXSikU/4wI1Js/5YnwsO4uG9CLK5m X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } From patchwork Wed Dec 18 13:04:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DA29E7718A for ; Wed, 18 Dec 2024 13:06:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA68C6B00A1; Wed, 18 Dec 2024 08:06:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A568B6B00A2; Wed, 18 Dec 2024 08:06:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D0696B00A3; Wed, 18 Dec 2024 08:06:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6DEFF6B00A1 for ; Wed, 18 Dec 2024 08:06:22 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 20595120DBA for ; Wed, 18 Dec 2024 13:06:22 +0000 (UTC) X-FDA: 82908102690.05.8A5AD29 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf17.hostedemail.com (Postfix) with ESMTP id E0CB14000F for ; Wed, 18 Dec 2024 13:05:55 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DItOWe00; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527150; a=rsa-sha256; cv=none; b=cR0Vi9PPPEizBMMqXD4rS9OEkvSsAaAfKFwFV4hPpfF5wx14Wh+p6hKFrMGaIDAhEV1xoM i4QdQ0dbWKOlUcbDYFAlmr943cUb/BCjoAFDXNfBQhdWJsfjTf4LNXVKTv4QTGgG8XCnJf loaGan5wHx6f0IVvLAYesXZ1n2AEqYQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=DItOWe00; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527150; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Ahoctd41Xu7L+7olWYCaTwfCpap8BqbvCHiRFQm2vo=; b=pSHFJ2LEH2BdkVNVh2UwpsbikYlDUyfPlsi2DpRYl+bbmKtuzEnAFRlc6ErHJmV+ViB6EY SbUQvmFw1kprVUFlXduZfC5DZ/+DCD1Ku4pYl6iNOgBn/cagzKlvMPqiiM6d17ZFs9Wo48 Gdi1wSk9c3OXT8PEVsQGY5xZGj1DdTg= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2163dc5155fso55970815ad.0 for ; Wed, 18 Dec 2024 05:06:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527179; x=1735131979; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Ahoctd41Xu7L+7olWYCaTwfCpap8BqbvCHiRFQm2vo=; b=DItOWe00Pgtn/JfwO1IHf/Ue+K0Noaq2L1AJKbaHwNjA4YnDKetok+8Syabl4WNSh0 He+ZfTEn0IVJQpq9+oI+k718o0BGmaasl1vcbGTtropCk/B5F7T7Y6odYOlcB0owRGAV RIrNy9Tejk3SjAr5OtP5bDxqiZkKPOMtjSNjk1h4R7aD4tvazP7YWwy7rgay3k5vulk2 SjQA6VVW+Jt/OYom8ikYoVE/bTNsmJHBbMsN8XYx9Hw1QnhxQV7XlwFu1QgUa2/508f4 5nz0lQVvLIeBgNGrKWACJ1s4a30kGHStM4XRcUKH83uQMrK0Czbl+ChD51rB5mgcoKWf qryg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527179; x=1735131979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Ahoctd41Xu7L+7olWYCaTwfCpap8BqbvCHiRFQm2vo=; b=WeO5D2Du4JRwM85Nk09BihtXy+Ndxp36X3AAPXp2VNJcAmWp1Ofsk9eMtmoXIMxp9J 8G1H3EU9zNkyNn0Ms9amUkvgs23R6/gMU2uPgJvMyndsUNFovz1ou/kXe03RBOCP5kWo ec3Dd1YE49yDWF7b64WQ/p4gGhdLWj8JvQUKAx7xJyHhHciHZB5m+vCAt/Bc9jIiegSb qmhp2LUmFRyPlnPWIihPI5ZbFlI6eX0R9GSeDnGqxticegHcMdeCq0Hv4gqYaqCf5kXi Zl/07lg0EV3kqlduYLTlKBTnmERhUdCkRAOmXvjpY3qeYENP/2X97pdxl5raiIwcXMoU 0clA== X-Gm-Message-State: AOJu0Yy7Q4aUPrEn0b+tIMZTrc0f6TV858HZaSPddEwDxq1bnG7enRrM 8sWVeztDSNoZmvEFHyzXcplt2PD+8ntdz1VmNuRfAWUPOkrChX5fqXI65Z7SL9Y= X-Gm-Gg: ASbGncui3tgP2W4RXrS4U5m7FcG9LVN8Hfng59NJ4IHRWFitbZqYHBCw/RMQ6DRS5pf FYbEVHOKs2zR5Z2p+pDSlpaK2uER+VXu0tBn1NdVGcvy7QXSeXGePYUeiPO94c/T6ZZGLHA65wx JhLBzByp4c26QE9nNCaTsTQ0ogiYU0ZXSwZ6Jay0sR0TmMC9cCuONoGi+50mnpbCKJnWsRLZ+lw 321sibh/ZpOCcDHUGypiUNlls8PVsiEqt2kGMhdijFEwQ3D0HP/r3ybLn0shkZXK5p5CvUgRb17 LpOXt/TFGiDUkOHKPNxrVQ== X-Google-Smtp-Source: AGHT+IErwrhCem5fzBI6a6wl06mNTQ9yKVlkmz7hDOEqq4HxfcsAp66u7+2R+nQ/CreR8BGTw2JG1g== X-Received: by 2002:a17:902:f646:b0:216:393b:23c9 with SMTP id d9443c01a7336-218d71fd765mr39074785ad.29.1734527178847; Wed, 18 Dec 2024 05:06:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH v2 10/15] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:46 +0800 Message-Id: <0a6caf8305409173b5d41dccb6ecd46460cf9c1c.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E0CB14000F X-Stat-Signature: jxp5za13o3kjhy5ap5xz6ia6nms6xhco X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734527155-157230 X-HE-Meta: U2FsdGVkX19iJC2kuU5NhwUIVySOsXyvUO/Y0WFg9j8lRoR1Vqo8oFMxG7t/AjY4LOMbEk4Z9uOYOAWSXRJqzXSZKJHicr7d1FLX28sZuDpKzhm4GtSXLFGmNh9TnT3MhGSM8oE9jVpOpOznf8sp63J/HJf/PMyVYwSgvvIyveduQm/unR42LAFhSXr2C5dKXyoRpkYdETnGMAjm7hHOZopl1t/Tv4wTo/5cc0gBrQHhvnv6OlvgUFP1ubT973a4n8n3CeK5YAWuel+TKH5/9IXCfFMALxUTkkqhhVKPil2MAg+L7iKqeB0BQvRfCyWdlXolccu5xqEHf9cTqcH9PjOtbtVXnKdmZHtNFqY90fDiarF2APzNmcwd+3Y7/axGI9MGnhP/0bQk8TvpBQqINC12B83uYZlL0dxFBFqb3g744M8fmiZFXv3EvN2nGv7oHi//KCDrB8FbxgNZZTwUQqtDJ1DbwNvl7laK0rZzyWpc4UGNLhMuYAhqBPvQ0C0Ywk2N45PlICX6IkG2IvG6httog1OwqdF7hoEQAGupgwxC9tNeOUMsRUY6bXZhxQUzB9mPzw7JWzYnkhuKm4bktA7bU3syD+Wmy+G4+cUpWo2tYXrU6GJfhphMIbF/aELKMDHde8jkpNSJ5HR90lYhyOjVIppvltLfj5Fy1Nq8lsqEiQLYdCzh7A7kGp0bGVh7F5OsltFx6/b3CR4efAs3VAe9mWQY6d40dyCloXh6VK6wIktI2i+wcGBSEn0LAko3wSci+LhntmeqRZG1k4/wt2GrZOF3CcjX/L/msgC6+dhZEodxDWd3fPuRn6LoH6gPg09dNLQGz0W4bJ7A+eVlzOz3QLgAYYZse80Dgh/Eq0GK1aBqvBcrwkxL1viSD6aBNCLOQxqM6AyampEhtduhbLDhtH1LBzg/q68QkNEuERCgNRiflqIijN4tHVaAO8Z3OgQ8m4MrdF8++C716dk zitx3BSp 98wPd9V4ONSyFWc3q2vVgC39d2iCRdSfSf/qduDEqVtY1cwqAVsjGluzjRl21RyfrqYA7zkV6DNP2+9HWCiKTDVeFTGxEJk1bTfo51mviCmBRCtTRwHs2p2EKkEMaes3wDqw/hot871fvYJyAMg+vf63l8GmC+mJPlYWXvlzkGWG8IfDXnqQxhT+GHFuIBScZp36wnUnmun6k7HSKemiu3h9VwYIxo2SsIO3rIG0kaUeZaGK8+w7YhOtJEVYIShk/gXaq/w0MrWvGc2IyP+RaDgVKRcKMIyFG9LVfdXv6RonhhqmYKTctzlhyeCdMqcZAxAm3LiZ+TWKpPPJBVGdf/KII+ktQSira32mlR7q8M0uHg1sIpAM86U84CgRPgCt1QjJzgROH7PQ9jrWyQd/v3L0kVSAWNvS8KNP8Qs0Irpd5/MG8oNj8bnPI84MzYVZVRNknc7CGJ2QijlA9e/ImD33h2DnxQNlAOcXVNDBYfNGqI0bpGPGa8b5yjscRUWXRDnPr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000814, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In s390, the pagetable_dtor() of PTE has long been moved to __tlb_remove_table(). So similarly, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. By the way, rename pagetable_pte_dtor_free() to pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Wed Dec 18 13:04:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40615E77187 for ; Wed, 18 Dec 2024 13:06:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2D7B6B00A3; Wed, 18 Dec 2024 08:06:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDCC46B00A4; Wed, 18 Dec 2024 08:06:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A56E36B00A5; Wed, 18 Dec 2024 08:06:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 859406B00A3 for ; Wed, 18 Dec 2024 08:06:28 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4216BAEAE9 for ; Wed, 18 Dec 2024 13:06:28 +0000 (UTC) X-FDA: 82908102186.08.1FB8A22 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf14.hostedemail.com (Postfix) with ESMTP id 32301100018 for ; Wed, 18 Dec 2024 13:05:50 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cAkCU5VO; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527151; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8jrhRkH9wk2k8zCLxFmzwNce2I+voVuTloSZdpUGAUc=; b=mKbE/kmLq4v6gNt2sZpjX82p2w2vT3oTUI7MyOyILRJMFdeMyEHnhjjYYMLVmRllWBAnxh qAJs7KMGBsyqlAv6QkcYbGKebT/vFMgMbdA3woIUrGkuqAhQhvQj7rwxHCUgfTCDCccKJS t5ERYzDZcs8nzmHJEf5QQ8XZxkK4/Yc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527151; a=rsa-sha256; cv=none; b=VWqa6OmEZAlNC5Gt0mUs4YTQqivOGlLJTUNSLZzpw/0qo95SdHY2sBM5RQNz0h59zeQooT UtLf6SGHIY/5hqYbqMaTIXRlKe3jkJjn1oxzBsvCSeuJVm1B7PvIbgU4JN4PUsjC+s066D HNveCo+a8FipUe19exvCQNoYRds1JYk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cAkCU5VO; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21670dce0a7so71224075ad.1 for ; Wed, 18 Dec 2024 05:06:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527185; x=1735131985; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8jrhRkH9wk2k8zCLxFmzwNce2I+voVuTloSZdpUGAUc=; b=cAkCU5VOlwetxhCDm3sUnfzA27zreh1uALPTRuDuqiHhknsd9yuVYDl4TzlkHe7kj0 lrqKgOtHdJqIkkOna3QlPJD0bxbJZQVVkDWb3dRKNETXXZ/ikRLyDzKUHuPDGNGlunia Ou6IbPh81MayxQojeopfWetTL+51mq7epg1MEFlLBYFUf3cOneHouQ4uzkMKexV6l5sO oeuKzWGgk5BSNzHeADgrhWuQJF6wOSaGp6UUsJpZLGQeg9aAYlAQV3Mus6pAYfDhAm4r Q7Dpr2kfewqdYo/yZEguEWvlce5ErGkmWUfN6ls1zAqqJWyQMQYkaR/bssmlCETe4zUS TYHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527185; x=1735131985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8jrhRkH9wk2k8zCLxFmzwNce2I+voVuTloSZdpUGAUc=; b=NgIoI+zvFJA4VFOXnZIBTROEsPAjjUssC+2DMCOxCkBdnqYzTJ5iCsb8XKFmVGOOvJ oJhUcUDHRPGrTa1InZ6VVVpR6D1TcmbQ+bSLm8zk2E5rwk74XtzwXCHumCycJhw3MLBV T9PQsjkT6C2rI2SGP7oaXyru8TXlTQ04boqHNgG7uzVcVeeJGte3nYBoL4LmYVh25aAV 2CxOvGmwpfxg0kgohoMjcrRJiFSsN2s9afu2xpp/qOQM8JT6gY8v38p9lIhFmH6Zc9dp 3yPtjX0Aw1FJqsu37om1DU+iuDJskJ02fWxrIe3QGUlIlWtlwjUXIzaqnpjvQib6xzS1 PE8w== X-Gm-Message-State: AOJu0Yxp4SwniDuv+T3LWsiDT5+dwowy2gggFk3o6eJelRWPeE14LkS/ qswho4fmFpXKkd7fViWrq83zzJDJ0PO7EdECRAFmgO0Aq9p5UsKx9YfI5fwkn80= X-Gm-Gg: ASbGncvlhmwmKgrbXbTtcaD3XRFBwGtjP/UOLvRpKGCoE9IDgf/4M5sI+SOkdrAp2Um D2ggN24tw2LL3AF9FC/CGRYgwf0ldTlcwhtabzPOKvGEKG2Cyn7QzGIZM3FC9A3fgywh6PZJQyt l/pxEoIkRMSvRSOil8OYFZWZvGDPJkmwXngyTqAm6lRz6cPX02VBBErSRL9WwNiQG2FkDz5tioY Z/CTbaV+DkLTOWvMvB87pDdOGtPuCoORb6CGUqLcJOdsjmE+KAMt6YJkW0DxeNR1tmZW4t+ogX8 W/zSkKsfGEiywVDJyMH9eQ== X-Google-Smtp-Source: AGHT+IG/JUNAlV4GKtrGOWzVqsVky2vkpSpLF0iAE0qU7NCcidXao7F2hMgn6AIs6nUdS2ae8ITLQQ== X-Received: by 2002:a17:903:1105:b0:216:6769:9ee7 with SMTP id d9443c01a7336-218d7247707mr40539125ad.41.1734527185019; Wed, 18 Dec 2024 05:06:25 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:24 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 11/15] mm: pgtable: introduce generic __tlb_remove_table() Date: Wed, 18 Dec 2024 21:04:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 32301100018 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 4xpibazqcycjezgccym1brqnq48o7pj8 X-HE-Tag: 1734527150-935905 X-HE-Meta: U2FsdGVkX19nBaNZaXb09HX652QOSZ3f9MTbQiBbY4SO0pMwil8ssZ8cpEHkznUP6oiLwWGwsiosezMw0CxV0L10jWJ3mi+lumI0SovjHhdqMAHkCdBYdUpvGJ5/BO9eztR9q/vlcrie8slIF8N1ENI/lsrgCDH+ldBhyUl+AljUeC6ftve02+D942XzRkxO7RquB3Dtp+FSfB2FUMI2daf7pqRgRiRh4SzZ/VDm4fcitMUoCqF/Z/OcIy7AFC0dMuXAWiwaNcIKsEi9B/GzxkvU2mHgxe6+5s4ffsmd7UOmQHrEntmSbTVNTH8VtGnlsq+z7VXpk/Xmm4pkLC8Da1ultFj3hIvIzkOZlO0ihgwvorDYPePVFpV8avo8e12/e4cLWhdO7AsU0tJk0egA+H7loVqE6o0l5DBtJHFKVCcl48pI8iv1pn75OkJEE5f3NLMkq0aVqiJEuzb1Ki/XJj5kM9BcIqGhtgCVE+6Rq5hl2vyB/sV4hop+dErV4nFriFE5uUV2Pc9ginHfJbAy5GbegvgjmMwoX0WTnx54ZpbOa9BjlUZlEOtUwrhcbLye0BCsfZHX946RPjGth4PIiNeeUtpB40fVrFaOfICVXq18YjfKhL9nxoVgMoU7W3al0FODLjcuvvNRV0sLED4SsOrZkMmWASkS+UyRwr2hzsJEMLVDxKRrGYBFiRbTZQ90UpKPvQ6mr9ZgC7CuV20MMDK1JkyS+tpYIOiT3N/nVUAS+wR0k0sSQExl65KDQTZzDJU6tQLFwx/n7dShzzMFejX8UtHQyJnyuLGGPGnR744gDvhP7usUBjCa/Xcz7oquz5C+h0pQ4TweLWIkLFzxHAEXAXOG0ttB4pEVp0h9AOwP5wZQVs1FTOfLvmX3XPeFFlxiYgf1nkWLnYXN6OEi/lANpYWFqWSVJQwGE8IDBzrFp2oM6q7r4DdjLLWvKjXfA4dE6YrHiG22379BDYT w/9F5zZ9 7SOq7K80DKwYaacLwdGSPwv0/oP4/hjlYy9xv+09ZH4iDsJ99TtZQ3Fzw4z7BQ00tezv53QEu2lrbYprzG7XiMiDY2LOXUkIy7kKp0P+O93y+EV1NgzCpCOny+HQFJlJkRiVOrLZ0P+iNTGWcy2xkELOmcT8s2BaWN8h5KooTsuUs6zSZ/fDRryyOfXQ3+jYKsUgDcOoqE82GMdPWdOayGLiX+LP5DX4lUhFKwnqsR6Qk7ZI+QiIrm28JZQTu7HoDYb3cZZkn+7q6d0dtPP21IDYkwMxanEBpzk2Rj26U3FqOhktko62NeLwF2A68ZHpbOH4tPQMipFhyeieG3FqYLtRfBQf5vzIZmA8AXG7lmj+vr0fMLZIY9zQtXJt+xWC6AdhpaZOdVDfCB4U4QxUVooYFftZYWoxUJ8XoF5MpXsf0dpZjJViGOHWyfppik7AT8H5P2T5YTyUWTJryG79G5fzeiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm, arm64, riscv, s390 and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. Signed-off-by: Qi Zheng --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 1 - arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 16 insertions(+), 55 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..7052780740349 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_32.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Wed Dec 18 13:04:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B838E77187 for ; Wed, 18 Dec 2024 13:06:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3DF36B00A5; Wed, 18 Dec 2024 08:06:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CEEC16B00A6; Wed, 18 Dec 2024 08:06:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B696B6B00A7; Wed, 18 Dec 2024 08:06:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 952506B00A5 for ; Wed, 18 Dec 2024 08:06:34 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 46E97A0E23 for ; Wed, 18 Dec 2024 13:06:34 +0000 (UTC) X-FDA: 82908102984.12.BC34BEF Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf16.hostedemail.com (Postfix) with ESMTP id D4649180017 for ; Wed, 18 Dec 2024 13:05:59 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hopCTN7E; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=NRXUSpAUc3BUCzvix549ou7fUMZEFghU7u57+NCiQatZbm7EhrMvjhu1s2TTSb8grSUcEi 33vkvsg+9JORG87K7BfhsdaZUIJ77iv+UUqsE9T64rjph8G3R0f3pMTw/UW/tTZxA5XW7N A+lj5Y6I0bSC5ERyPW7kYnRA9uKWrac= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527171; a=rsa-sha256; cv=none; b=cM6P+9q2vbdco+KBtz3zWvmdKo51QiLX0ebut7ZrGWEdTgasxTTpe7EjtpaNaH0bq7HJaP K4orTIPUri/j7TA8zivJ3hTQbuNS0FwFfR54+Zv3C7oizFe51p4l4FG1DFjSHFfwgKslw5 rgGuuv3t612tNuAdPosWFnmY2euyJTA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hopCTN7E; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21634338cfdso77582775ad.2 for ; Wed, 18 Dec 2024 05:06:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527191; x=1735131991; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=hopCTN7EbW3XOHt4nZN9y2u+kAJuhJfYlg50MGYQXqZxAf+qpGq8+aO5bHmT4Gxlbw 6kG6hIhb5UFmQP3osy28w7gMagb+90DI7hrZAijrc5V25vnHr1kgaEQv7XMJUwp/4Eon 5pG4wokWdU2LV080z7Lc5EtpiYR5/D1ZTn1KMHmPavl5VvWbdDtigEdGzgV07B/mn0oh Eps5XOEVCHcTICz/6nhULC4cnSZxTfABXhYhuTf6tiGsiV/8xthzOc6x/KG6QfsOj4K9 0hAnfeBVGKd9M4iXVH1k2nNrTzvGIe+0wdio6DaD/4Vp8gmL7NqWMZPqer/S87u7PvLM Wc7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527191; x=1735131991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=qX/6HFsxr+n+tjtakoi2owHUK0Ko1tOAvV6Tbe3o6ddTaIzRD90h2au5C5xGLXMeia dIQAjdv1t4g2UcAX7NJ5KzyKhVeeMB6oZp/SVtAzbtSWaH8ZlZbD+d35VuBn4Wh34MK3 yWO7yig7cKHUPtIc3IiwDy4JRy/JTjJBZC1MGj4VohIvtAOL4WAebYQBlB9SEdexTZin 36wQiEgRTUUojdUrDMVAHHUjWk2isl1EUm2EFTgK3WODYAzQ6/RI/tPchGqhldtnZ0iK Z6s3z/CsIcV3rlWk2Z+h7yCRjJNs6SMeGhygcAtVxAF6tmPsbliRnlUlmctQXZLAI45L YPWg== X-Gm-Message-State: AOJu0YxN42dF3ykXnP4dVWTHqnR+6bVluxHUUrS59Z3bhU9nrCiuQJ7h e7RlZjp0JzJtEShcyjEy6gvuigt5ZLfZ8UioqhyC15b1mHv4NaMoyyBI5wzyaaE= X-Gm-Gg: ASbGncv0kNo34TPLrVxlACWkdhi49hizkSi+tT1GEwMtC7c4ggUPPOlXBc1OawA/88l qWfUkMGqju3Ei9jT+/p7xmkKwGq9bKNMGTidqzXf8bjJ/bLlJfzsr5HwQybmp3DdFg+CauLuyA5 jPN8E07eXsF0bUHewpMPHAP5wm2W233yWWqdEQAFAFbFRDBJvdq1wXP8Pb0ZieeYZbl25sGKvqT JEPOgJiuFSoOw2lKmB8RJcBeZsBq6ll5YP5IrjEIn0Im1p0cxbPzMLsWhUWmV6XCMgL0GGGnHQw zUlfINarLJwUkOs7+Navhg== X-Google-Smtp-Source: AGHT+IF0te8v/jB1TE9z7MkQrDAS+cXnBWDMEESZdiy37OaR/n/IJPlBOW2ORibeoxTxN5xCd8QOLA== X-Received: by 2002:a17:902:fa4e:b0:216:3732:ade3 with SMTP id d9443c01a7336-218d7268fcfmr30133425ad.35.1734527191112; Wed, 18 Dec 2024 05:06:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 12/15] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Wed, 18 Dec 2024 21:04:48 +0800 Message-Id: <5129586354b398c26c80165dc9ebc7321ecd7a51.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D4649180017 X-Stat-Signature: qgktozroxi5s6ds9gtbg9ih7f7dhq955 X-Rspam-User: X-HE-Tag: 1734527159-523506 X-HE-Meta: U2FsdGVkX1/Qk2lneYgmPBiXUOxS+ZbOqWKBaB35tYsuq5EevaqwsjOdRSi1OIRPQxquqCX95OPPwVJbHBDLw+NskLdkmGXYScf4zKW24pgqapNxTVj2PeimJUWdH2f6rYq+xxgRHO10YQoQq83iII3VTxkifFWUAcbzR8682DQHxK4mztFokHF+by8ZFHC+AVqq+aEZVr6ISfGqXaGSBWHgxEsDg7w972MPpuX7Ee2Gj1C0IIEn2bD0c5Hn948Z3bR2EyEX/bpC2M1PEPnhImb4J4F/3zgpBZM97uCo881+bbNZiAuOZcWDJ/MrslIrMVrrdOv+MLxrqOZ+qgXQMMlJRU+xGobCBlfJeL1WiAImIJlbzM5JQZG5UM+9G0+rq37KKOPojRMPUuLq1J0MrRXhKw9MSRiHSJTGIg+QIb+5SJf766oBIw9rR6JNKenoBNnXVtaP/Vk16s2Cva62t1j2Ve5Ne6ufia03Xf/kszS9eUD+Fqu1RlL2rtcjnLpATvT9bh99M4PqECL0w8puT2o2epCYpi8vyc1BBHEWvGO4RcDiaRrtCtrPtpL+pJZCyi1gRr8WvDaGFjrzNrgb3yj2cKaT9bC7JYAfQHKv/PuUZpz1CVSOZvp3z7IXFb7QAOjmLnSzt8E02Cfktg+pRFsp5X8UiXw2hvsOckACjlxiYp3rBkorZi7xXo/kHri33adPLsaVDh7Ay5nSJ87Hx4cNGmucgAFiZye5N/RYjSzbfTFV72R5Cacl7DM6KHKorYa/C039q1ytb8Lwl1PYCtj+6yJXCg+m9JUvPHS2eDNlZCOZCDb0oT96SCmd5xhewxD1MjEWaCXJK0G7JfTM9yZ1MdlIWdyFrcaq3sVwdXc1Fo9ADNNukEehVidSXDSmAs6+eoNel4VbnNDeK19YItYFjetlFJ+dtDzby6y9pisD/kyhqadO3IZLmDi4S7fMmlN9MG5QS5zr7WeIMNg uxM4sefs wtZc8+/2pT+bo92QAyCIiPLL7y29qvb8WKw7AG0e6eXjFyPaFYk5VzekIFtNOoiYpeCBvP5EPOPlbOa/pyw4LaUFtnpd9rRnHlMOJoTIdAikk6xU7R+NLrTQp/RTk7ORnWJ3P7VQVMkeRAmYIHpdTMlO9mrbZiXkTifJ/JKnxznZFj1nV3Dwp2wI6u1ve2Mkq0FOLTMLSgiHRi6vdc87EdH+IZ40j08vYF+2V5e00kRd/07GvQ28mM2XGn6ND0uFMD6kPqau+STENo9OqvEayLTz+igT3iLiKNTu3xU1BoE5+6/6/FDE+rXNgIaUG2zaOHKM21Ja5ekiQXSP/rNfHWYEE1hpfv0ZrdSB6deDjIX/RxwZL1n4Wx4dnaqI+glRH6D3Df0tUfuwSRZIqo9aKnlZt/8YfY7jypWtO4PHspIZyzrvIZBQOJ1sRWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.026022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) { From patchwork Wed Dec 18 13:04:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60191E77187 for ; Wed, 18 Dec 2024 13:06:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E024C6B00A7; Wed, 18 Dec 2024 08:06:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB1506B00A8; Wed, 18 Dec 2024 08:06:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2B676B00A9; Wed, 18 Dec 2024 08:06:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A37366B00A7 for ; Wed, 18 Dec 2024 08:06:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4ABB680DF9 for ; Wed, 18 Dec 2024 13:06:40 +0000 (UTC) X-FDA: 82908103530.11.A647DBC Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf02.hostedemail.com (Postfix) with ESMTP id 911B280026 for ; Wed, 18 Dec 2024 13:05:37 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lLBXLN6L; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qjP41dwIWVz3voDw1szUdJpl6GJGCX1n8EUO4uIA81k=; b=7f7wThVSjUgAPs17NHy198tP2Gfux6Z4M5m9NdzzuuJ1sR9l9CGVrhOy6844oZ9oudSoY9 8eiFUD6losuLH1jtfPw8gm5KEDfO3EuTSQ5zdumahuQYL2zC2TRqc0GVQ39WIcxWLWN/jS HTF/yFhN+FSRkxSUiQh/tMmX/tOLs08= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527163; a=rsa-sha256; cv=none; b=EMfCQTRQKFAEYi4FElMBhD8EC7MHRgYJjYuuRmMwKAy/MyHqnJQKqVYPQLZW5aKwP70Ow/ b1akc6eGUjnu+RDE0fn1NmGERrgWk27YfTWOWLA+t8Sk003SxhfZBEIIfztD3ZTietX9H2 PHrtnmhL71a/RNwC0LOJwJPr0rF3oaI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lLBXLN6L; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2165448243fso63081925ad.1 for ; Wed, 18 Dec 2024 05:06:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527197; x=1735131997; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qjP41dwIWVz3voDw1szUdJpl6GJGCX1n8EUO4uIA81k=; b=lLBXLN6LTr8LjHSGRXwNDhg5HxUbpjIQfSpI5RWO4GB6rZp8M4kvkKTpw6GoDJb1/5 sThUGyCW0Un79l86O9zQL9TWzDRj+iNGLU75DhjPh4dux+xRcu6zNtpauU7RpJ9opyov Fx0V9/wAogjMnWvXzR1rcRxaHuuJqs3nUEMWlF/tgwuf015M9/aegDeUitVgAyDJvtpr K4zhAss20QkOsFLSmiD6/kXPourN7smrgwUQOrWtQfxfmX1JPxuFiH1bia8P+QyUz0kM 5DWVzmXnbSP42KaDoSCjThcqsVUXB7YluEWrlegA8L9uMTaGVyj57BIv0MvOjlX3gEQR ZeKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527197; x=1735131997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qjP41dwIWVz3voDw1szUdJpl6GJGCX1n8EUO4uIA81k=; b=kLY3nl+zzAbz23eyVCUlU64ldwkE7cmpQvDefc+A70WMtD9cVLHo83tne5siiU5pIs w3pGkX6u1MVU0H4XW3tHdFuRlYk0GK2ylQyklBNGksjUMp+E1AqCDLvjYbgvTMX5Z6dt pJtEire79M0jewB1Jofza1v5+ag9JymFCpCDMGjcJ1wXFuYZ0cm0xTOoKddOUNQ58vD4 TeMkizEmmv0WG7rxi6pzN4Igij3cumqth7PChTTr051cbfQUDMTNu4TK1sNAYcR0TCWh zD8igu+Zq12226sr/Zy/GmRY9egjZ7wXvfR8cCADFlWzw5cMkXe6Ta4ncF4iht3YYZFU gDUQ== X-Gm-Message-State: AOJu0YwO9Yfc+chHAc0drR88dfl0ZWOBQA7oJ6VSqLO3HNPSI+yBxkX6 8NR66ts9Xe8Q/zG6KSa4V4zUj92r/TRfvG2+YRGDoY4hgfRVR0aXHb7saYwNhA8= X-Gm-Gg: ASbGncstxmreU8qzoH57vc9IvcNIGA0LCFWXVvGhYN7RzeE5G2HovEhzAMZYNidI1v3 rJ611z3FEcVHQA5KQqID8jHC6xTEzP1qgMyiJ2J17N/Uo9ez9NWEtE3j05W/DwCAPrIu/zTXI++ 5v60TEqtGRUTA5Sepa43H8anjAiPYR9/whxKv7QHKm4TJuB7oaPxOj9YSJaAkx4OkB5yxoRsafp loTf3JaKL/fXFcJ9EHcFVu8pZFWIA4aCzRRhQSNLZQSGC1spCNZBEn+tM9L4JahTA+PXGsWzYbu 1So1ccNjMVxFXt9CbcOIpQ== X-Google-Smtp-Source: AGHT+IGZLlII1E3uD2eJMNDRJL7Et07dogsfo2GqYIDSPjE3P+HLWVdTN80snitMlVb3E/0oGm+VVg== X-Received: by 2002:a17:902:e542:b0:216:7d7e:c010 with SMTP id d9443c01a7336-218d7252c97mr36826655ad.37.1734527197173; Wed, 18 Dec 2024 05:06:37 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 13/15] mm: pgtable: remove tlb_remove_page_ptdesc() Date: Wed, 18 Dec 2024 21:04:49 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 911B280026 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: qbmb7saeaqfnfkehdh8wabgkzrw3em38 X-HE-Tag: 1734527137-128776 X-HE-Meta: U2FsdGVkX181pAvrCOF6DAG76OP3nimNRGgC5KTiDbTGUa08VRpM1+Y5rT9kUfeoKMEZqb9pcpJXwBoJF2RKyq2NuQa10UPXTuAAAoRw6LnAQpMxmJMQR+PjMcWFY9I/2PxtwKbgCk21vpPeMcrOPfT8lylp4U4kmwqj4TADPFlJt22WjFRyhzlMVPliNEVo+IaqpjxWPo7TBsLsBMzL3Qylv8RnxG8fYnj9WgP8hfI84GOoGwdRuaqtTiIDq/LPEUY/EgNaTksL5W++PR3hw8Mz2SQvt47k8fJmo9QcNyYYUyxfmwgDUcIImeYGwr6pSLAlhbzMCea2ZwDe7Fh5uV6VQmaA5S6zhRPgkgSSC/CSZH8PiFhWJOnV4Jga3U1it8EWVsm95Vvdl4cBAIr5YuiFfBC7raFG7pEN/pHy31ceDco/1HKTOoOvsyPzHyAZreMOrpT0Zq6sPIMRzhFj4OAOFXKOtIfUPg2mozuh0uGGNTZ4rv30oohRugsIdKvveUYqM3+QdILdJHtOj8Z2CJUkAjyeXwyCpTH6H1/94wcE0IDBKRXT2ysX5cB+dSUrM7dxEuazEda8y8T04FWgp8K8QW0RTN2JtHxZsKfw4n+OzTdxI4Ume5Np3tO+F3+oIuyhhw+asdz1Egs8/HUQn9Hy8693l0PD9nTB+7aI7AUs03cCKoElukpGKSSunqky6paDES/9bParc1hOjZX0A98yXWNsQGByBMXp0PCE7LBr3xzrdlUaHj3oOWCQbsqCg/vErDXpx+dOhtWCEBols1t8ml5lQ1KTgdNghtVHwuuqYktQy6Beyvcj7Dc1BRrv2czOjrRdlg/0ssHGEV0LLySaSwcvSsH7bHnlAv3YUOMUBqcgSSi7xeI9UMJNqS6F/UXvLjlwfCMYkjso8v6Z/weVeire51OYKkv0WB+Bq5fmN24eOwVNpJT8F2u9xXaNQmxfxKFsYl6T3Wi3dEn YJCd4O/P i4EjXqui0KT7ga4J1tsnSnAOsRCNAAAx0DfexLIFJf3dKvhHsX401gHZGoudGCLJ3ADiHbFwcAbMxRtnmPQADI+l2e2nr/8aRTTzkv2mFbEwqw2UkNd3ai1fuuqXr1Cd1Ou+4VGtsHlQpAQznoLn0Mkgn+QCSeBc3MykJGytgARb2KRrCTlwsne6kT5fWnUc8rRMzJfwBKXCTU72f96rxha6wkHJUXWG6mdj9ZA7M67zCjfIWWtWwb4sehDhajyDCJNkutFlwPFWEyST3A/Rem591khh6WS8T/GAoHYH9vxHknGkIjE/yr6o2EEQ1U5Q+K9hSMJc+5LErN4+gCzv25Tu/nUJqjFxeVXriUQU4TAonkl4v81m6k9Px3ucTJipa/7Gt/7tjHEW8AX/gKCmbY1mg9A87UyOCPCzDURWgqGyj8784Qa9+BHovDwZp+maUG2HKUq693HL5/gqtJgLPy7d6PfYHAGVVB3HqTLCwAkjourc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.005715, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Here we are explicitly dealing with struct page, and the following logic semms strange: tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); tlb_remove_page_ptdesc --> tlb_remove_page(tlb, ptdesc_page(pt)); So remove tlb_remove_page_ptdesc() and make callers call tlb_remove_page() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/riscv/include/asm/pgalloc.h | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/um/include/asm/pgalloc.h | 8 ++++---- include/asm-generic/tlb.h | 6 ------ 11 files changed, 13 insertions(+), 19 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index f1ce5b7b28f22..936a43a49e704 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -64,7 +64,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ + tlb_remove_page(tlb, (pte)); \ } while (0) extern void pagetable_init(void); diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 40e42a0e71673..8b1550498f1bf 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -90,7 +90,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor((page_ptdesc(pte))); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 7211dff8c969e..5a4f22aeb6189 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -58,7 +58,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 2b626cb3ad0ae..63d9f95f5e3dd 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -20,7 +20,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 36d9805033c4b..bbee21345154b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index 12a536b7bfbd4..641cec8fb2a22 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -31,7 +31,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* _ASM_NIOS2_PGALLOC_H */ diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index 596e2355824e3..e9b9bc53ece0b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -69,7 +69,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index fc50d14010246..da782a80f8aad 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -32,7 +32,7 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_ptdesc(tlb, pt); } else { pagetable_dtor(pt); - tlb_remove_page_ptdesc(tlb, pt); + tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); } } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 96d938fdf2244..43812b2363efd 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -35,7 +35,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* __ASM_SH_PGALLOC_H */ diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index f0af23c3aeb2b..98190c318a8e9 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -28,7 +28,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 2 @@ -36,15 +36,15 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ pagetable_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ + tlb_remove_page((tlb), virt_to_page(pmd)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 3 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page((tlb), virt_to_page(pud)); \ } while (0) #endif diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..8d6cfe5058543 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -504,12 +504,6 @@ static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_table(tlb, pt); } -/* Like tlb_remove_ptdesc, but for page-like page directories. */ -static inline void tlb_remove_page_ptdesc(struct mmu_gather *tlb, struct ptdesc *pt) -{ - tlb_remove_page(tlb, ptdesc_page(pt)); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { From patchwork Wed Dec 18 13:04:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93637E77188 for ; Wed, 18 Dec 2024 13:06:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CDED6B00A9; Wed, 18 Dec 2024 08:06:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 17EBB6B00AA; Wed, 18 Dec 2024 08:06:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F39EC6B00AB; Wed, 18 Dec 2024 08:06:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D3BE26B00A9 for ; Wed, 18 Dec 2024 08:06:46 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7B4C41C5DB0 for ; Wed, 18 Dec 2024 13:06:46 +0000 (UTC) X-FDA: 82908104034.02.DE33854 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf26.hostedemail.com (Postfix) with ESMTP id 5D841140016 for ; Wed, 18 Dec 2024 13:06:20 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FLBLP5PP; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527190; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oQG7I8CTZ7TxV9gz83QsafJIUScPDY5LnGxZvWTV1Wk=; b=VKF3dZDpy6gs8jUPntEI++C80M0sV8J8GpfuVc6dYozmnioqbfVBmOvF4Hno4UNEFGBWgS TQ7c8APkfj7C0p4hPGoRhs3ChACM6AY1BPqWfLwZdqeFJxB1OJcWW5pDs7ZhDQ5ELBwj+t aXsfjQ6gWQvv1hnPXi+yecV2/pcz2uA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=FLBLP5PP; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527190; a=rsa-sha256; cv=none; b=AAPGi45sqC6Ozntum1rYf15hRcC9YvA7naiT4S/7OVjNO5a1qp4evisEDFcjvWYUvTGvX1 isn7HfyJDtw2liQJogmWVS0vJPXLnf5LCFGvT0qHGK/Ir8AkyZNiltxG7MV2Gj1kDvBVHT hr7iTwB9z+zanxqUPVvyMhCp/64/Ii4= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2164b662090so51907355ad.1 for ; Wed, 18 Dec 2024 05:06:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527203; x=1735132003; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oQG7I8CTZ7TxV9gz83QsafJIUScPDY5LnGxZvWTV1Wk=; b=FLBLP5PPhXLVSGSQvOEPuUsBqqyWf71n86MmDzYYcqh2rI/u5b4sv/nRK9OcxewwKs 765HsEeh3u+DuBcAd0OGqIyy2xLTTvSxk35Q9URIbwg/m52exahVIgKk60+D9yLxYl4P Z4Y5VusqFnfGHXnAtF5AwK3nVEFJqHwM/t/klTr81Db7/g55yvOTjPESf3hBINIucaqa 3PMvrNW5h7LTreYXmdQcgF8Px/WI48Z+yghtUV3+qeEv0WKxIqC/YfJUC8J7PkEqBe/j pooA5a50IedqGU5ptdEyf6GtheHCuKy6FqYFzEQTPBWKlFPROeNvOu5+nEzePjUJj3xT 2vyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527203; x=1735132003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oQG7I8CTZ7TxV9gz83QsafJIUScPDY5LnGxZvWTV1Wk=; b=uY7h+QQfMPZfENbX7dd6ktfuVsG9rm4aaHTsRLkM+ReMr73Ua9JNNZZJl2cio0ntVH LTf98CT2N/XsZcFvGnasg49SOnF44Slcv7iibSQb1t+hLxDmUIIpv58ri3mzwTvtwA4x dS9EkR2PI7e9I0MAN25N0dEyDjFy5GnE8iOkW9zQiGvzGcG8VVuW17NTkRrNfvRfUsdg eDRSHD6p/O7SuJ08TJwSkgTASLPFTHb2UimytqtHnR77J2RVe83gJGZgvWnCyRBHc+vw 6ATpN8i48E0lbrlXt9Ka+pjE/Jci4HSEk8mWD2a/cJKqLDkgkdCrngZLSGvswuZwFR43 CXAg== X-Gm-Message-State: AOJu0Yy25QhLXqaD/qDiOKyeTj9fHFxFr8Vh6XYmdxz7fv/NgLIFnPl4 4sSdYb+v1a4DPB8MERyeyLEB0lhgLV9FFliR3Y3RjXz+u8VUqpqGnn5R/lBiKRc= X-Gm-Gg: ASbGnctGj+yg1VRo1nX3b6F+U6mQEejVcdPCtY/4y0R3eADjFisiEqueJWvvtW6+oyK oKyO7wS9Ke/+5k9CMbyN8vTtc9hhM0AkhfIuMtlyVh1Jp7yt9nuPHtZOiTi+73HLNBHEPUsawxJ 3x4pW7aePtyxm2POc9DRv4IVA8G3kmoEThOcUAHmk7VppMvVSlU1o/5qGxKxFBKIu91RAHQz8iG DxCUjLB/iNzUBPip4XeuV6rDRj9CgfEOpx4WvsjF7vjo4hcmKHYkuvdxS+ho2MpQ0bxrHrjFgBx sqJ9uE4RY8EVr7OhjuEJyQ== X-Google-Smtp-Source: AGHT+IGqOVtOaXTsC11RmGC0pnVBsbhpRr7fSYpWbIozZcJ8TvZ8U4y91o/Td1tfWFL+H7ThjVOaNw== X-Received: by 2002:a17:902:d50b:b0:215:6816:632e with SMTP id d9443c01a7336-218d725bf5dmr42047505ad.48.1734527203288; Wed, 18 Dec 2024 05:06:43 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:42 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 14/15] mm: pgtable: remove tlb_remove_ptdesc() Date: Wed, 18 Dec 2024 21:04:50 +0800 Message-Id: <79b20df78eb81a9cd07e3048f2d4d63d98759b4a.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 5D841140016 X-Stat-Signature: schm7aqp7jymn8kukitgg9prk66n9zed X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734527180-472046 X-HE-Meta: U2FsdGVkX18cWo2OC6DiP9rVL6h1sgVmIOMxLdzhqYvrmZYQKvV4Xnxh2fujeE3yHZyHrugleIu4t8F9hcSDrd1RcGtXS42ypcOob4N+ShESwbLPWThLxM3UJLmOYoALz7/L7jV87j8y71rLXiynFWiPaQsjFY6LnuN4wv6aT3fx2uvNqA+E9mUpQpk0SOK42VfMcB0O486kXiPQy1DGOCTBd+49S3aG/3114Iczl+dzn/T+3WiPA/x/OyAFPq90vIdhSFseil2GZx8Jpg6gyJAcCVCQLKM4436L1+shPxNgbkCdWO2lqragDQyl2l3wRNW8bakXr6FsKGFwqArxuUwqT61f//JBGZKma4Ntbl027fED/iC0FuRIl4MinLD483ITFOs8hMqe+bQSvQFY6LeznH14K8l47NMjA/hxoKOEt0xsXs5IZU+rzYoLYh4v9E7IiuajRUNujeaggS36G/r1FY37arFgIWyrORVpPSG4YKARWrq5MfmpByZNbE4ggNEokr6foaQmUOB8BzeGMwuDSmHrEkQAjGiHqfDuqY8PzGJdvhaZpXJAeh7xuKUem8PaS6IFRt3NBQJNZUo8yVu9DlIA45bx3GNd5leW4qtK4liDMhzoihyw48nCCNO7Fv3wG+wez5zJdWcTqffluAFr4RxVVwxbFLmW3DmD4ilxL8viJVTiXs/EEIJNHq5byR71IiJRYSqTVe4nui4grZToBZPKF67peEkctyN9Ya7r5XG+smf0ibnl3DrPX6CdvYDk5meDV1j6RRvCYVsVFoJD7Woc2yu5lyOUTSskCetpeU5PpfExP9Jj8Ta2HHopXAcFppu4BOqADSLhwJxn6YVou9YSfufQdHRPjpUqXIxhk6YRBzq6iEmVhs7IdqdL02qNMfdj6Eo2uqjp7p6/vkHwUIRUXCm5jqpLfcuQsRkxeualeHI0HOtfTPzistLv9CFDPaMwDeaE1vFTcJt R9CwJpcY WpwRh4iBkzEBXJlrWP7KzyH+I0JEfXk2qu6ec04QFbnzdNXkxYRmFbHga3nPQyhmHtldJjYg8CvTOff61Votbt+jnlHRzaaxp5Ut3AXbHDT1zGBKbTWRxosq9jcJqZ4soFVWTmjtkhAx6lOOMSn0QqeGCEIrLuBX9UCwlFoUQqVLPIsSP2YuveQl6lDXvS0mb/RKyCue+FMBLc8SrI5+Ct03L9bbwCh9XJBCQInL3AH+e2ggw/GcGtNqdyFPbDftJE2MRXWaZd+FeeI/K0g7+qG581klJya+WTN/i3PFMXkJIi1gm94OZ8L5ccQ1RngS9rvqgrlP60Gpd3GHzVJcHzKE4Qx0Nwl6jtdJCk+pIYKp9dk0MLs7XyAMFQcR8MbZUv7tOQ01AJB1t2RhqzlubdY8JLVaJ5MZ81/s3ooY+KwhRyB7ZDaheIFdIi2IOt8wOEoCV0UVpSmFHo7dXZT7jfzlUxUzsMl1owoXseJRlppUiSg4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Just like removing tlb_remove_page_ptdesc(), remove tlb_remove_ptdesc() as well, and make callers call tlb_remove_table() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 8 ++------ arch/arm64/include/asm/tlb.h | 16 ++++------------ arch/riscv/include/asm/pgalloc.h | 14 +++++++------- arch/s390/include/asm/tlb.h | 8 ++++---- include/asm-generic/tlb.h | 7 +------ mm/mmu_gather.c | 11 +++++------ 6 files changed, 23 insertions(+), 41 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ea4fbe7b17f6f..ac3881ec342f1 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -32,8 +32,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -43,16 +41,14 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) __tlb_adjust_range(tlb, addr - PAGE_SIZE, 2 * PAGE_SIZE); #endif - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { #ifdef CONFIG_ARM_LPAE - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 8d762607285cc..4a60569fed696 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -75,18 +75,14 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -94,12 +90,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pudp); - if (!pgtable_l4_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pudp)); } #endif @@ -107,12 +101,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); - if (!pgtable_l5_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(p4dp)); } #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index da782a80f8aad..f319e2c6776e4 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -29,10 +29,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { if (riscv_use_sbi_for_rfence()) { - tlb_remove_ptdesc(tlb, pt); + tlb_remove_table(tlb, pt); } else { - pagetable_dtor(pt); - tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); + pagetable_dtor(page_ptdesc((struct page *)pt)); + tlb_remove_page(tlb, pt); } } @@ -120,7 +120,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { if (pgtable_l4_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pud)); } #define p4d_alloc_one p4d_alloc_one @@ -143,7 +143,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -172,7 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); + riscv_tlb_remove_ptdesc(tlb, pte); } #endif /* CONFIG_MMU */ diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 7052780740349..fba11949dd2ea 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -86,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_table(tlb, pte); } /* @@ -105,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_table(tlb, pmd); } /* @@ -123,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_table(tlb, pud); } /* @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_table(tlb, p4d); } #endif /* _S390_TLB_H */ diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 8d6cfe5058543..583e95568f52b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,7 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = (struct ptdesc *)table; + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -499,11 +499,6 @@ static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *page) return tlb_remove_page_size(tlb, page, PAGE_SIZE); } -static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) -{ - tlb_remove_table(tlb, pt); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 7aa6f18c500b2..c58ce4539c56f 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -314,18 +314,17 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct ptdesc *ptdesc; + struct page *page; - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); + page = container_of(head, struct page, rcu_head); + __tlb_remove_table(page); } static inline void __tlb_remove_table_one(void *table) { - struct ptdesc *ptdesc; + struct page *page = (struct page *)table; - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); } #else static inline void __tlb_remove_table_one(void *table) From patchwork Wed Dec 18 13:04:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31B3EE77187 for ; Wed, 18 Dec 2024 13:06:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B43B66B0083; Wed, 18 Dec 2024 08:06:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ACBF66B00AB; Wed, 18 Dec 2024 08:06:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91E6C6B00AC; Wed, 18 Dec 2024 08:06:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6FAC46B0083 for ; Wed, 18 Dec 2024 08:06:54 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9FCDF44B9D for ; Wed, 18 Dec 2024 13:06:52 +0000 (UTC) X-FDA: 82908104286.28.817401F Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf24.hostedemail.com (Postfix) with ESMTP id 8D9B7180010 for ; Wed, 18 Dec 2024 13:06:46 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MsDXbFrM; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KC7Ql8GT7M6xqj/drFleO18sYzKRSHwzPDd2hD5Jtik=; b=18eOKYnNI5zyUsneFnC3/6qDszK6P8tDSyXjhzY0rFAn5Iv0sJXsln78dRNX1oKiGKouIW RI0xEst/8cq3TPDg5ovj+Gv9DuUI6nBUAXvjPCqOFzltOIyZVmzqN04ddgcEJ9xIb+sytN k9H28E69f0fm4cYG22xZCoSTlFhSUUA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MsDXbFrM; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527177; a=rsa-sha256; cv=none; b=Dm6Ot8f4gZAS9l17Y8hvbI6TvMhSqMnP2cHjTHgR2ApaeOSvrwGIzbZ1rllz+jdJyMBK3g ho61sMUdwvJXvn6ynbxcMaY3myKm5LFmgHry2FJPJ3MWfiGFxTORAkhCrOzFaqfdYG07fi KwcR3t5zz0i6MsxmNCNxTv89Jg78w+4= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2164b1f05caso59770785ad.3 for ; Wed, 18 Dec 2024 05:06:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527209; x=1735132009; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KC7Ql8GT7M6xqj/drFleO18sYzKRSHwzPDd2hD5Jtik=; b=MsDXbFrMoom7THgyuKytj7yBn+4Pj+K9oqW3e8ZC/6uHzYNZFf6iZkChhjla46PPp4 Vt4dxxAi8sWUQUgAUOWf4bgGkPL+6Sb9QC1bMA9+WDxzsFGOVXaifs5SYY0CzRvHelkJ Rub/NfXYX2n3n9OkHOPya2MtPVrlIxzaSHIJcXJYrju/dRbsg84Hs1dRgNCMp9T3bBGj mYmMZLHE7onViJG7g3kT9odwuQPODHhM+USyh7lKq9JmriBy1Z0Q1mcg8nLlrtYs4wma 3h4cmwxk8LG7/sRCwCO4zkSkKYW5tvmvDSl+XpCeEN9XrjHc34Yol3UGfLKsXx2wEm3x lLFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527209; x=1735132009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KC7Ql8GT7M6xqj/drFleO18sYzKRSHwzPDd2hD5Jtik=; b=RjOcM1r4XW4pGFiy6lH+wdxfbi9NNhQhQtKwN62iQV8NXE+vpfgtb6V0vv1HbyYzV1 EiHk3cLN6zx5gTIFQyjDcBkmVh8FcnS6NyYLj2dY6ESsZFbVmnHMO1H8bbSmccLawsDu ekYNX9ePhX922A2xZPSILVrrDZpMnJVRacxTRK6h7brA6OjTL33JmPG/h0y028t155J3 BbfosZZ0bqLZldmWP27Ixi0aLWHDwOh1RRNZVmLjgKOaByM8ZP5yCoMmnNq/1ISYw610 Zjv/3TiHYdVsvAIkfn4vx0FDScGsoRq1sMgQNwHZmtzP8JIRa0ZRQ+LXFXdzMFLh/7av aTtQ== X-Gm-Message-State: AOJu0YzoQyhk+F/zo0wRHVRMmWLBWNzFR1Z8QYbEWdxpazyUdXad6Y1C 3tbbPluPWX+UZLCySPKnGvyCKF5gmfiGul6dhqfV65sNOm5XPyq64NYk4L6JcaI= X-Gm-Gg: ASbGnctyIw55GQngIV2ZS9K86hq3n5NXywAVr8RwBHA1eBUFOn8sEduH7+oLgHah1D8 juzIZTcFoQmgPWhJjaug5K3zapk2HxVDQ41hsAbDxsrE8ZY4y9Dqwf1RJ9Ds1h8r2KgLteij1jd AmBcfkZ9huVLGJCfULwEVJ7OFQlGsOHX+ZtDoMgUPbeZBpZC6yjKd4zWChOJL2oH2Hal4j10mO3 O8vaLxYZbwxW2nGuvDxXvxrp4JtmzGN1o+a6GGHGVbqcaLyENs8dIueDptjwY5Zp6yjB+wtyAa2 BPlyaH4v5raUr03qlTJXEA== X-Google-Smtp-Source: AGHT+IFeQHKjf5x1j6HplQpPLqOMHr9gUGeYMwElc8Se13maebUbMGrfsZ8FVIlBe6kXPLktzNwjjA== X-Received: by 2002:a17:902:cccd:b0:215:6cb2:7877 with SMTP id d9443c01a7336-218d6fcc257mr40842835ad.4.1734527209409; Wed, 18 Dec 2024 05:06:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.06.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:06:49 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 15/15] mm: pgtable: introduce generic pagetable_dtor_free() Date: Wed, 18 Dec 2024 21:04:51 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8D9B7180010 X-Rspam-User: X-Stat-Signature: k4hijcdr9da5ajpbsw5r8wmrr17namyp X-HE-Tag: 1734527206-845339 X-HE-Meta: U2FsdGVkX1/Zwe2VyMnxmHDxp6FGGyauPfqg9G659C+ShkMTttThtPxl3iaDy3YVZ80wWnPKPlRNjx33ufIAfBjvI3NbKVx5TJfckgYjXvquKo1LGg/5TXJJfE40r4i51APh13qtZEp41qO6bOhDTsp0q5Yj7apXkyqvJ02pq8syHsPcUG7USjEIMZBU5/vr7w23cdlSbhU6Svv9U4ijybnMGiXSafF4FojAkrv5egaVZ/Vhbylhs2vdRJNI0c5AiBfW/VHQq62XgFQ93MyXf7Grw3XQTe832mkMil8PHC8pVnjBezWnnj03Z1OSU8zrUbBe+2GwaJG9pXuN9U90t3hlfj+6eK83AIAQ+ZH7HW9p4YXAZT09f0h2fDjSxcktjQ5rtAWmwslQA/3kc2GW6oDNRr6/tqP4HhmS32Mt2HVFnMmrSfMma/umCHIgJpMLtm9BcM/cXLupkCMQLOhgoH3/LrjJIL2XXfwKMZOMSXLBI4MrC9V4oO2own8nkooTGLzWJgf3cVp6ZcZXNjV2jCBOiMCHtScnUWMcFfaD1doU+oYdik38ynPg2jPvh8KHElVFX+IopnPkQttHjZMSHwyCeC1G+qVJIJ3qkprH/HPrhJMxmDJrlAG2l+1HeY/gZip8nhyj7tgag3ObnyxEEimIPgfcq/vlSanmJ2awT7rfAMAHpPkH1zyz1vGAoYE7nsFMGHwGHPr0qsbcleKGfp8NA/ZPjOqYmPB+ppNn0rp0ENSdTP6gqFf6fqqPjWUzk8MtuNnYI9v+Z29qwpijrDTWaIn3PxzBWOhlSWbstvpIjNaz7CJrsQ46lNowHYEsof20PgVPZRg+UfgtNk+16uIEtDzzLC9nYsBsHWc0mDVlR3xbtetBqB3/4w777CFgR5/DetgXJ80O9mMzSXdsgtExyo7gpLi8kcEgv1OQIKXuXVVIEqUXk+vAi2S7dPPex5mnzkTyZzw145cY1yC H6X7tgDo Q5fipGNWQQ9+am6mBLqC90341OINURfZnJkzGAvqtgfFPcFMGKw7saeB/W0X/6ATUJIz4NLjiAS4H9orUcvpDCL/1T7r/Rw4FGuZNePUMG3rchdCjoo6HAVMcD2rb7QPXC5RmL7zGXjRvViSmZfpQJ92Jo1T+o04u1JmrjtpThfbD1AGy29IGrCF/j8DxUFEBTM1vphkn7+MApe//btXQK7DgamY55amcOa4Q8LGoke2ieHX7r8b6VJNjJQ0sPrpgEY2CzMM4VPzlJ3qvilqcvY7Q+u1WDuiibb2rTS3VKU2VgeIZ1mfbic2wk3QF07Zzqn4j+VQGFmgHd1OKe+rnSClSH8UKDDdcN0KsdssZS3ciTGcWngSQ4MDnbJfkPO105h7tgH1m4dHWMImyd7lFZZ0+0xh5XBqKA/m34g5yEg0TZPqxFuLXVJj1W5TDp53cVNWd2kgpWFcVuj2i1m2g5DDbvR8LD9b1oXbqcKOO9lPnoB4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000770, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are actually the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 actually does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/s390/mm/pgalloc.c | 18 ++++++------------ include/asm-generic/pgalloc.h | 23 ++++------------------- include/asm-generic/tlb.h | 5 +---- include/linux/mm.h | 8 ++++++++ 4 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..1e0727be48eaf 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,32 +180,26 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { - struct ptdesc *ptdesc = virt_to_ptdesc(table); + struct page *page = virt_to_page(table); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { - struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + struct page *page = container_of(head, struct page, rcu_head); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) { - struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); + struct page *page = virt_to_page(pgtable); - call_rcu(&ptdesc->pt_rcu_head, pte_free_now); + call_rcu(&page->rcu_head, pte_free_now); /* * THPs are not allowed for KVM guests. Warn if pgste ever reaches here. * Turn to the generic pte_free_defer() version once gmap is removed. diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 3673e9c29504e..370f5b579ff88 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -107,10 +107,7 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm) */ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { - struct ptdesc *ptdesc = page_ptdesc(pte_page); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(pte_page); } @@ -150,11 +147,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad #ifndef __HAVE_ARCH_PMD_FREE static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pmd)); } #endif @@ -199,11 +192,7 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pud)); } #ifndef __HAVE_ARCH_PUD_FREE @@ -254,11 +243,7 @@ static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(p4d)); } #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 583e95568f52b..ef25169523602 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,10 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = page_ptdesc((struct page *)table); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(table); } #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index 736215c428293..f2d04eec66076 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3095,6 +3095,14 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_dtor_free(void *table) +{ + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc);