From patchwork Mon Dec 30 09:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 032BEE77188 for ; Mon, 30 Dec 2024 09:08:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CE6E6B0085; Mon, 30 Dec 2024 04:08:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 557946B0088; Mon, 30 Dec 2024 04:08:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AA5F6B0089; Mon, 30 Dec 2024 04:08:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1A8986B0085 for ; Mon, 30 Dec 2024 04:08:56 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 63CBAA1A21 for ; Mon, 30 Dec 2024 09:08:55 +0000 (UTC) X-FDA: 82951050210.29.41DF154 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf13.hostedemail.com (Postfix) with ESMTP id 05FA120003 for ; Mon, 30 Dec 2024 09:08:08 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AZf9oKdM; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=elMnB6WmMU9vrVrxHi6wUaFuGLmqPc63bRxYG+n2AGLbfQjHZr4PlLYAOOU4zLkGM4bDqP uRSRdzV/76NElHNywcu7wMj3Cj+MN0knsfTpJug9m/1Xp2VPP+sRUPscGAAeMboYzDtZDI osGEN5QyQJQgWGp5MpyVwiznfEpaNJY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AZf9oKdM; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549685; a=rsa-sha256; cv=none; b=xfsuAkthO8XdqOeSH58EZWKoyvlbe5AYtpgXGVMDJsJBTPQ1nr2EpzFnMoTp0CTT6mu79B UrHbnEXRm+vG6V2Zio1uNqWoletyzlf5Tf57JmvUMZR0K/skaA9s6GwTGL8mNw1m9Bsq6O BlkKO9LqrsI389wQpF5V2+A+UfegX8c= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2162c0f6a39so123632265ad.0 for ; Mon, 30 Dec 2024 01:08:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549732; x=1736154532; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=AZf9oKdMfJ9fOeKWr3m8yBSVhgTQJ+dELeRGvukrHknNbGHr6PnQCpcySQ/JfFU239 hskYo8zbIyRCcYbR8kVwwxeQDdEeyt9ukDfbFoy2WSRZPfC8vU5qB45Mr44RUbpKukS7 WUVQO6N7pr4G5XSPKilhvizx5xfTqYal7P9DZEGAR65zytCjjSLfMAEjIpzvBcwC4AV5 pKhU/PGKJLZnncmn1grzUiM9+BFokczgkqv6yZ5cz9gybq8WyHWxp0O6eHP7CCGSWEH+ KtAHxKqjrZYBuuqAIwkalVPB1kO84jfjG1TROxHeyhZBRL0chg+AFcBVVjUfob5IQfgS QB2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549732; x=1736154532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=RrNRpy2ZMWz2x7FTSnSlbeenD92/7Q7bu5Qbm3xvsXDsGGUGPlTuj8E9Vz80Frn2Gq JSuMXezlj3/t7cL/e8zOTdRPWe70RDrZSisC5oBmq4UdHDUwJQmq7x+H1d7P5dKtt7Gc MUHTj9/d2HaMeMWYDpzSS6P0RZ+q3rxVh06bt9N1VTmxSoxpo7OehCjUyfJUHNFmrice m+OgMY2HP+z58klEtCa8tYQEw34UVgV+QJc9us8cjLoRpiVXJp6TRQGb1tvKOTdNfm65 D5FtkOBmPuXPiVGq3gYIbPTJpwyqsaOrdQkxXLxMl141iq344qoHbd4CoToW5dj4+CWt jHew== X-Gm-Message-State: AOJu0YyMZBw8kJKd2QFq9Og+AzNlu0IyLA5z7fnOOy4Dk+++SePSwkTm PSAL2WS5kF+btsZQdeFtEXtW+wdc9we8ymjsUbSwu+YrE/j9IghW0lKStlcRCog= X-Gm-Gg: ASbGnctYC4k3DHcOX5TqlveShbEaXxnSVKApR5vz6prE5I20LJPF5YiOCX2vnOfNByr 2rEHypWdEOkkOntbEfU6PjQqBZLFFTrrUmF+gNNQPWw3N5Xqa/NOsoIsOCYROtQkU93qENn61OX W80JWf0g/Z1hbbbE8ebhbBFlpyZq5gQT3aNo6E1pN7NIAK+Gn8t1v1Tw0EKcJCGrO4tCbJ8KZaW kENFCfbTTfJqjoLtOQJ5hiHQCp7lR7JyOCXmwE4P44P7EZ8ny2bulb0wHPfdGATPPW2mcuV9rfW /x18VIEBvxOOAQiER1GDYA== X-Google-Smtp-Source: AGHT+IEXAwBIqSClrkb4EYoc2OhmhpgilmCz/IsEh6Chc/FsWIzsRB3p/FpLbUSGkXAy0ZMKjVVExg== X-Received: by 2002:a05:6a20:4320:b0:1e6:50a0:f982 with SMTP id adf61e73a8af0-1e650a0fa30mr7283235637.20.1735549732420; Mon, 30 Dec 2024 01:08:52 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:08:51 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 01/15] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Mon, 30 Dec 2024 17:07:36 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 05FA120003 X-Rspam-User: X-Stat-Signature: sqznaq85czrx5qpbrnpi7dekdsdwo3qa X-HE-Tag: 1735549688-442744 X-HE-Meta: U2FsdGVkX19ZZudJDioSL/nCL9pmWdrQL5m+HbbDuj7uXluO0HHy23AtHKHZFexZmkwo+2YV8j7M5Gwb1Ki/h8P/LlA9bXZXgtYdph/dZ1b3vJ7/GXw7h6eXnbdBMpIvG0uryjK6XYVoOx0EMLnR4Y1Hbou/Xh0DSRP+aZzivzL3gYr7zLdcJE9cXGYBRqRjd3HiyWfvHWypxqrLjhB11rjw/bdvak8z8G8Zzc10NZSknKhlwGzdQ8xuiFQkxsVbGQopa2RDiTh11HzBnXxeGI/+u1mN/Ju6F6jmBHDiWnCch+Zh+tR4hKMWFwirGkgKbBH5xLI719IHp9KwWjf8UT8/oAyYKkHeCwlIvw0bHjWyQmEqS4uxCGVOZmVTP6cvNkZo1Cc5dWn9lhTDDJTl1UCH+P+oBqMDfTaheh9zy6GdrDCmGr3weaD7dQOLkApT+/XLeOBoLn74ZbfPzEffGgSJAtxjjWhP1CVShZmTrSoUTEAvzMnLQfrREcRr8ZKyaXKzecGkApHrOfZ1NgWFjZtzSx/QVpUIp/H7s3tiyQaABGy8NdaXvYjqSgAbTBc8Ub6yifut4pIvD99jIsLst2LP3j0nlr7i46I2TDZC5vFdrhDCNTHJihVJod8xwnJ9ADZhQ6SkcqZBO9uTLCrHamiP3T8FQ9SvHouwNfOA2LvOdNLSy1z9eSGWASNHOzVf2nIvp/pppd1PaIsRW+UbdGo8J2SKeflc7Vijx301IA+Wvs3lFI2YMY6RFOFWXfIXcdlXKfy2igWe9jsNX3PfpXfINkLbmaq+4IfIG4pg1Wy0f43TKKSo6adN2w3u9N+FlLUNCkSu8KNsi1jfNpepe77D86hwnuGZ+2wR2BcucbI6HNy+cHe1wo5Lg/BuVlmcRFNSuirk/uS4EXfd6j7R6aUXu6iGfdoqWi712cpOoLEIipO/GoJe6ovNdPof+ahTkEKfXhMKpPlxKQl81ZI 9l7G4ek+ BOX1mVoc6BrX9vDJZlpmPjYY8vLe3ClMy9UQTHtnrYOhspbKSoGrlRZdhISCVyoOP4RTaueWPf0TN5xnWCY3DX/eBYZtW9wp8B94q2sAudz/3jETmVGyvXXGGtg51hhlL2fXyAtXc/FqSqUCFsYHgqhrFNmgOiVDxdyPM8eXEx01CzpKYAhDTfGTA5zsd/18VafGvmHHLOJFYssTXVUehxG7CG4aTuRQkGSzvS4HqpN8D3zqWSkRJtrcKKXzzdx3DkLXZHndacsvIhYr5CRt4RuJNIepkrtbh6wIub69ye/Jmd02OgLhkU2yB2RUN0FUd5VP0PaLrzWc0PeqJdwLrDyTXwlQuBwVdkEcWpluoLHpZ4IgcGMlgQo3hJn+6LH/IfxS4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 30 09:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB43FE77194 for ; Mon, 30 Dec 2024 09:09:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30FA16B0089; Mon, 30 Dec 2024 04:09:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 298806B008A; Mon, 30 Dec 2024 04:09:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EAF26B008C; Mon, 30 Dec 2024 04:09:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E47B96B0089 for ; Mon, 30 Dec 2024 04:09:08 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8D7531408BC for ; Mon, 30 Dec 2024 09:09:08 +0000 (UTC) X-FDA: 82951050126.05.038B9B0 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf24.hostedemail.com (Postfix) with ESMTP id 8788A180014 for ; Mon, 30 Dec 2024 09:09:00 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="W/di8uxu"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549725; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mHzGZQny5JmimEm0kycepUlCOKttN6NXbKtm7VNKgbw=; b=DuD35DHZTUUhkLJpPqNclrFfhP3RZOB6cIqrFdPMJ9eLbeAr79CP1iEcXfFhVX2eVoEHZo 6khYf3bQadio3kfVVx7MK5N9E2E50aUzTaxcKBWdoO/iguVWbHl/xLvc190SnlzR7jqMix CpYJAHRo5F/q6ervTpDshx/uSWT8eFQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549725; a=rsa-sha256; cv=none; b=3mWNFteFjfRpfTnaGSW4wflArDv2jWvKuC0UAgu7iiU2USXucm73s9yc4IsxtGnEV0q5Yb CJPSP5Iz2ZhJ3v/9Tz4TpTqVW3bdLmLgAG4aLsG57VGfBs7JmYaYpzZ5+FZ7DM6i6LLYt2 HRFmgY0YG4scx+sRitQXUBZsVf++aBA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="W/di8uxu"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21683192bf9so116587725ad.3 for ; Mon, 30 Dec 2024 01:09:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549745; x=1736154545; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mHzGZQny5JmimEm0kycepUlCOKttN6NXbKtm7VNKgbw=; b=W/di8uxumOtkgfFJEkGmqygARwSMQsW/aU4S86083VEoxpRi9+SFQUgb3tp7DKYLR2 gLjJjSjAdCotH4vEFtE/6HNis0LjFzhYaZ08dgHhB9ikI/G/WPQLdQRfU07I+xlZKejS SRlF8T080AJp06CIM03tZtfPqMlbryKVoj12ACeOxiYmv1Gaugp0wvTTt91O0iKgpc3I BHWnolcrB3Z73tQsmB3pvTuJh7bBLBAn32c/cYU6zNbApeE6usz2flkZWXuaGBm6Mxew pKvnx2ss5RtZNpGC4hK15TdsVCMcpeVt88hFdnzE7ZMS4PYwrv2FKlu/DQbq1E/VZkR+ EVwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549745; x=1736154545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mHzGZQny5JmimEm0kycepUlCOKttN6NXbKtm7VNKgbw=; b=JxOw6aLKEQbjvn6BWLmF1SY5BnChxpbiwurBmTiOSfgp+KFWypN79LtJ0FilPaxNsZ DF4v3z91Ou5a/UCQMKhkyQJqIlhIkWVJKBAFPvV2pBA6DWW0TP9eYWi3Oy7kK/Ur4yy2 f0brHazcz/VpPzx+794a9V5QoNKRetmLnomR3SF7ZNrumPvgyRMVenWasQ2RtNL1hTnf O27GH8WUGIHl2r8ZpuLHA7UnxCqM24oaFm3G7pQ0bXitNLA1RLTUxu7dFGKDIHBlPBDn DES+Qxv4GJu39UGhsPlNf9+jp46a5u0HgtfC9jzxkZ1YPl5gSkofWYKfpRwntqJlxkN5 91qQ== X-Gm-Message-State: AOJu0YwnCWu9k6p84cHsFCPTaSeHh4I0Xniyd2AINI/jdbdcfWDWGjvG VSnP0upw4Ch7JNb6oX4PJhg8NdtWST7eExRBY8yKdw0HmtYxNe92a4mYjD60mTQ= X-Gm-Gg: ASbGnct+o/vq+Ji/ZKLxzgR4ZVVYgOteZmf6ey6OAiRfKjr7qnk86Yj2OEVoSA6xtRE 6BwE2+ADhdCSpW7CbEvVd5AFpS9KimBVrgAILiXCz0TF0UY3ncCW6jwrV+7iv6WM/a/CFrlU9du aZA8GOXcUdoui+yu72XVpuS3K8yCs5ZmjDuTM83oJPdvtwmUJ1DFY5xDYWzO6HXaJPBWqyumbBK uRJtgT0KH5bjl5YRBhRAwWmV+HujtChcynNWt/N1F/bi5Q+KIhET+huzCUZbb/4MY7yvzCzbyqQ bGwbZIe0JTwpL4BIM1fZVw== X-Google-Smtp-Source: AGHT+IEdmvGR6U/qwRYH5ruA02a+sFRctKWchidIfEBAoumo12DzBepIE5K3QuQVJJ60xG/AvigJ4Q== X-Received: by 2002:a05:6a20:a127:b0:1db:ddba:8795 with SMTP id adf61e73a8af0-1e5e08028cfmr53967973637.36.1735549745320; Mon, 30 Dec 2024 01:09:05 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng , Palmer Dabbelt Subject: [PATCH v4 02/15] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Mon, 30 Dec 2024 17:07:37 +0800 Message-Id: <84ddf857508b98a195a790bc6ff6ab8849b44633.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 9urxigxbb6f4e85117pscabhkzf9c3f9 X-Rspamd-Queue-Id: 8788A180014 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1735549740-440514 X-HE-Meta: U2FsdGVkX1/c3ioLPri9QjxFnp0xiCxYXEBIvA+ZthzhkP5OuF2UQnnxGkE1gUVGMzxCShY+l99mwNbE+nmJdMf+ScbgkcZlxgS1FNHbHr6IQZEd2Konwu4pClBm4WmBMjtNK4Z5xwh4EqSj++9UjhdewZPqG9WSKM+3g7+PxOnRPhhyPWOyE0jauyn76DnSAo8CljPhlJJsi03LAhbArJoFl2KJWoOI8JVxqFV5jqlGDKr+lPWrNppcx4m7uG3GMgGKvrl191KQT8WMQiNYtnZnPKZ0Llwq79xaO5qEkM1dFN7X40cB5Wtk2fIknDjVn+klWYCjpm4Y6xT/Mkssj2sI/I36wh+OQMhxLx8DvPdHDvRlwx/TOtIf7bOnUPDH0zo4fb0tET8gu8/iovnDqzG4Q7hUkHYQgAdqgNaBktYUhYc739S1Hz4+uyJokZNKbkPv0VIxNPCqjor6OOFD06OfFzLRVQ+zawFX5XUF9djwNye9sDkjBVwmd57ZG6B69/JhsU4R4JQTvfV18APcVBCNFSmpTmhTarsvM2DjL5SKRMi1pH3n+X8XE78hfjGb3iNdRCn4FzLoZdZkpqY3746rpINFMqx/S4PLqrGLHkq2u+fZuQLSsioG14lc7g5IKmRpnrdJC+9/hwyXXQMKwGmAHbfXD325uu0SSVWZoo1hd5IkXAvLsQI8CxPjrABeGfPBarmSVdxAabcKWv1C2G7DSRAcSiUOe45T9v9dMi2+EeSrXJasZJFM//XvDcOp/B8fx9wC+XEPBo8kCXC3SOpzZEyV4O58Y5FinWIGxk52sQwU9VLQ6cfz8KUXPVXNEsyzDgJZbT537+MVe0AYGPNZhnEwXYKsEqOHzgpICDm4aE4qpprIiasnIFIc17wDqGn4vFghM5sSt6xT9E1O4JvNhoj2vvEa7a8fS6gJH1RdVu5Kvf2cHuy4hYQcu5cFzbv7XykimGajP507C7f qCc+a27f WW2jXOpeR19i5Cu50HzlDDu3ma9j0AZuPut9u5Ce50Yq91vlgRLOFiMEx0dqPZ8gJX9NC/zs1DI5Ae2Rnh6ddNEi7zn4/BXl6hFOd3fBAwpad52kQWLeKH5sa7NibOHc8d3NezS2fYLH3S1GerUMs68IaGtAiDr3lUjsF2F6M2gnqE4Di8WSRtwzXM4lms32CMDI8oJfl4VfKRKgO0G2IYBnckxK8tHcf8YWSYlxWMjYPYggrj0NxbjSTgtLfE3YdrPgrj/MG0dxPY2yhGEnvWFGIRn26Qm+QbuA8woRwyBAUnBDcvLVHdmUy7f9im+FggH+v5GfYOLqVRiy2egaJbLLeIc6NMHpycl3W8GtdL2LZaMoY5IujAd6X2W74j5ZplunJnYTJmECZQes8cm2WopZRVU4+GFeZgnHfa6scyeUbAwNt/kaqGttzLZcWyHWw74lgVE8ago8qH7rzrTOekSXC8WPOfe+1Gau3TAQetlqH4Bntls1x8Nl2e5QmGBtj12TBPNAyh0Ga00QcrN9byXx1mH8l5ZhpUCRZZCIVtshhWrPoi+9YdfOGgTzZfW8sW0CP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng Acked-by: Palmer Dabbelt Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp = GFP_PGTABLE_USER; - return NULL; + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) From patchwork Mon Dec 30 09:07:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EACCE77188 for ; Mon, 30 Dec 2024 09:09:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C382E6B008C; Mon, 30 Dec 2024 04:09:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC02E6B0092; Mon, 30 Dec 2024 04:09:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EBE06B0093; Mon, 30 Dec 2024 04:09:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7DCCE6B008C for ; Mon, 30 Dec 2024 04:09:21 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 097141604AA for ; Mon, 30 Dec 2024 09:09:21 +0000 (UTC) X-FDA: 82951050252.30.7B537B3 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf05.hostedemail.com (Postfix) with ESMTP id 105BB100008 for ; Mon, 30 Dec 2024 09:07:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kCfkCxL1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549715; a=rsa-sha256; cv=none; b=hF8YfseBLD2MBnLI9CWD5JnMjrBR+T8SmqySPGGHg2LjCWE+fg+2qY9lmIAtV2dRdcbuHj VDzhXYqlK5DiMCPAm4OBAyyLsK63aTgJyy7OzAlTJ3cREn3kWD7Aqp7xC6bWhaIGOByVDD wRPSBoV8Nh8paCKbMckwD1bLXy9UFmg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kCfkCxL1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=3ACCZJqJnRoTmnMFbrgH2Z9QhqQD9QSgc2p14mIbkmbnRLoGWnZQgJjNVFZmwSvYdxfV+F ukm2iOdrCjcleggQO6eWA4YHC3BpIaKIg8yIawo5pIf+mszpF/qtgUpO+YtKilwspjmqpz H1KLpHCaoTgmIcY+YN5VfASzcWsIzfI= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2163bd70069so13701615ad.0 for ; Mon, 30 Dec 2024 01:09:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549758; x=1736154558; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=kCfkCxL1KQbKHj5A3YhqFqOZ5cSuk8bsK+5yFIxqpkKJtBjmzOkOHPlTZigfLkXh0u +6BNPheyn3h19WMHr1aBfFaoJcUzMHfRevT+fovy0JQ5U0OAwMXUpI5nfdOeE4fSjt7N RkHb504HJ8hGWz/tHTTwffdyVgU2TK97vfSkBtt4Pjwaj+Hhwu8ktfvUE+2jINzrF396 YgO4lN7ObV2o+0xkINtp2JlXKh+65MDnu7IgLrpf8fFaU6Qbk4NWOlY748W1dpqZ3RGc tOEkQqR6ZQJqMJBG2aH7RYZrFqnhsz0Xf5n6yVTvSWwB9hfmNANBo8+F2XV0rcIo91+i wJNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549758; x=1736154558; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=IbXQYleNsX1QeWR+vm51BC1SiC+wrFaEPw8vO23K2vlY9s/GHvFNOS3e5VfLPpe8J0 NKmhM6kUTGluvt03butpPQ62/MMaPIKsBU/QS/a7TWLYDA2QkmgXIcF2gjU/DWreCor2 dsYd4kd5UloOgeFn6VAmocaV1Qf+n7SO7C0y8URQBpnp5AVZZ6s6BXFtvkr5UqRKak5c ZWGZTWBhLahX946hzpS2zQcZUvvYZ3mr2qCnsZP2q6mQht1T4xRkgZot1oYtHQKqE7vD 6ZUg8omSG31pq+hxNSlWG5AahcnAafTOielupj5ia/4gvZLABOnKGAH3hZZBl7nhIrax OX2g== X-Gm-Message-State: AOJu0YyASsorDZ8ktEzkRnOFqhw/BzDMKvCDIoWp2nG9UUrP0okpZz7m rIC1o3p6aDSypGBIwvwEaJsL1ccJKbBly6LjNiJ9oHwTDKzQqQ7H4EAxPJ7Ffw0= X-Gm-Gg: ASbGnct7IyerKdUh9cHj3oYd/vOm2fZjapuThuCEr6LraM73MwLydIVPXHLeFU2kuFN y0ecKDS+cp826V4S0NKB1nGi+jUHJY+VzmS2/bRJPEiI1Wjc9DEvwRcwhPGH+NcW3mY9Zz9cUuS PP/Mg7WXsJTDaBKkdjcSq8lYI2BiUPYjtdodu7M7/927fJOWH4ICf1bMaGDKYt6gah5jCScZSIl TM7RsOk2Z1Mwz0yWilF8Y12knP3rXxi88OhP9VHpvNihvw0vUaaxSBQXTilUeGl9tiWg9t4S1uP mJk2oBswcrZsAQQw/eTlcw== X-Google-Smtp-Source: AGHT+IHmMXvIfmrfsWwSSN68U3q7TJmVN0XL9C78+ztBjPSSvkZSld2lKuaEpxPQ8tk0dZT2wkXZcA== X-Received: by 2002:a17:902:d488:b0:212:4ac2:4919 with SMTP id d9443c01a7336-219e6e9f9aemr366243385ad.17.1735549758001; Mon, 30 Dec 2024 01:09:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 03/15] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Mon, 30 Dec 2024 17:07:38 +0800 Message-Id: <4c4bcc1aa565c6252183553aecd5e5cbd1a0f6ea.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: b9eoj8dhbhot8m7xpshuhjzh1tkcysdt X-Rspam-User: X-Rspamd-Queue-Id: 105BB100008 X-Rspamd-Server: rspam08 X-HE-Tag: 1735549669-225212 X-HE-Meta: U2FsdGVkX1+rliVP5urKi17183XgovWT4hMX4PPvEYaAHiRi1zEGKDV2FLw+ECqBkE3ca3CDs0iFH/IUM9iaI0tq644dvyYOqgYzbGlAl/LmFNzaDDZWSaHNjS9YZIz6bcthPxs/MtV/4k4HFMSh1y1Od5dSpIaf9iMCXoMcn4OYOrKHdrRrI5opOvg0PWGjUUjRgOcj3TxgQ8FRX7BbKygL9wXNqqSkOMfpZ6cE4/o2CRmZd7TK19dR4fSxwvKtCMzWXzUqN8vMXhn8czTDDvcPzILEEJ7rGzAiz5O7GtZt2L0IoBgQt1n9KwyCrXskVTC7Mr6xgyebaKJOCVM2ekaHi9OtkdlhBOi0CXZfLypPOtpgsXx00KWAnjubaXt584LuCVo0MV8N1k526qfnALmGJln8gpSglrmsR5NQ6iZMjgFVtQ2NHGjoimqAwU5/WVwU91U0X0vcueVfpucAASwC0uBRSx8wtZ4H0UyLqdr6Wjug3R4zSVAJ9U7/nlO1hsxA6x4CpdLV6PVeVSAf4FcHhB+n0YLLNU9NePRu0c0P6IPNRHjcR2CnYOwXeCYV7xFmBGx4ZHAjySsrFgonwmQlo6NWfQkKdJxCU9RBAc+XxL4t5d0/ODU0UQoIOtG/W/1AEHMnoZ3OaMtMqgD5DTaPG807RyFST6q/llNOtr1xhWngrugbjhSPc+HQIV7WSfEK+6stTlAYz3WPNNIfe4QVY9mtj2Jc2qajtEGUy1t+c4WrL/e39f/7xLTBtNXlZexHKaiQ+Ytfb+/L3n7+o1x4RbBedBtPgmpwwVl7iW0xWcS7cdNHiCIf74gh88fRzHVrlZR0rL5rlIbOn54gAtY/Ei6DJ8RR1olKwQVPBpmeGwaZ/dF+P0n2VnEqlfgULhJISRmYQQVsiiHHm+fA4wd7xo+yYgs+oIJqWli8f8U8+ff9M/QaPsjEY3GxLliM6EaSo5qFehifPcYvrtR hxvgxfsS EgECeVoYoMiUaeN+RDGcnpGXEiHRG4aMP6lYjSOhmR+asYdoEoKQUXcOL5fGLINzuxoG9dJmNxpfXbSm8Nof17LY+UvKKXhuOoBEQ4UgSZP1R5g3kGSACkfxP0bBio2FIxtLyMu+TrLllgV0zjvmx4qztz4WqOw6d6mM2qxvVmAtCnHlUwuIhH/K5Lv8wbHJPFjBA72ZMAmEG9hAws5oThOjhYejOtZCuGZqPQ0dDPN0YSGBLwwIjLSflLmkA+2Oa9CFshaLFj+eLOj1FlIqmVG2H66FE5No2Nzg6iuMFY4j5oM3vn9UQQHIb5RdQPmpL00/0lhsUzCgejxjZ7VWoEWCB3/Sl/62lOY3JYxvAoDYRUNAlnoX+6RhL2yYvYUPuJAXxNXyymbE22Q7SLfc3fLKa2k2lPPf3GjpT38Wd9imApWMF4Y2WquV9luMg5h8ZDy7Whmvx2a4pZaaxkngrS4Sn4/a76kIvgsitgrm/Ocv0jEqxSVeHO2AIk+zSlXYKklE3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { From patchwork Mon Dec 30 09:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D325BE7718F for ; Mon, 30 Dec 2024 09:09:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4974A6B0093; Mon, 30 Dec 2024 04:09:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41F566B0095; Mon, 30 Dec 2024 04:09:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 274386B0096; Mon, 30 Dec 2024 04:09:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 05C106B0093 for ; Mon, 30 Dec 2024 04:09:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 894B8121A06 for ; Mon, 30 Dec 2024 09:09:33 +0000 (UTC) X-FDA: 82951049916.03.0536FE6 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf03.hostedemail.com (Postfix) with ESMTP id 9247620008 for ; Mon, 30 Dec 2024 09:09:10 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Mpx6zM9h; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=Kj+VrEMcUS0Le/QM8DZgswnuOq5x2JlioMNfgKLVmRU8vLPEmZyxxb2kM7Ha41b9Y/DqxF kzxVYsyFwCGBUSgsj73bwNLhCTPNnuEfBTOumIji2tHZ/lC/QVIYj2xOG7yGqejT0apo5e jqbRefWl1hOA687wwbTu19/Rxh2GW9Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549717; a=rsa-sha256; cv=none; b=t0F8vJlTQJ1/q+zy3h0atkdmCf/d/W8laRWr1xM5gqVEQ9E8R5Ln6JEF5n9D1JhRuYAx1w oFXwkbMRJl4DQdB9JlZJOy+UqGzTpcE73wUgM7keBBlep4NoKKyDetsB+LBZWhhVvk/VjD ZO7Zx8srglMGhnPAhYkd8C/guVdFHlI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Mpx6zM9h; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21636268e43so17179015ad.2 for ; Mon, 30 Dec 2024 01:09:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549770; x=1736154570; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=Mpx6zM9hH0RDfC3gtgMb2wkkO0jkSQoOA0NiBmOfWx6UJpejgrmh93GWnVYZImLtFL qtt3AVaHJEqwEPC6/hn3sHnoH+BYkLpOgoFSq00WVcieuCyFk6rHbF0y02Z2TaUAfGAw ryMxeJ0nTIxmw7glZn0pWvrbRDRFOrG05OKJaIFB9CGf9I8d7hYa3usN59vDz9oQSvyp DVXvlmYkRy9mjmxoV+/8reJqPq9X0gJqXoApb+hh5IWd1ZcvlCmO4+8N1CnBjQ/tvf6c qWTju7GIUuOcVRvioDF1RL/VQ20dU9Ma2QneN9IiWSGaWBZ+7zjJ5gvDBmgjJaTblrMq lVrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549770; x=1736154570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=f6jIZH1fza9cDfv+/jIv2PIDf2hRsA6Tp9kOGFMKyIq8NrbQrcbjbMSVeKVfFmZx9p X1STTYNpS09v6jD74PfxqVfXGUnPd4rrx8/PJTCvXE5KGDWdFf2L0UtMTtN1g+ROV59+ 3QXcd+gauegiBvA04g1Dw4aACZ1XKqwqTqkxO3okag/zZXxJ/o5+DRs7eiaDhyCFE75L AHuuIIe95l2NSiXbcI9sPWq6YgnrezL4yo54IxRwVU5MQc6S4UERvcV8VkBk82A5L6P8 KoJA0HHjEqpegDujXJ1H2GAhaQtBYzEK49IJCWTVGHVWLkcKEO7QNQMmF0w9oyxdh0YO x3IQ== X-Gm-Message-State: AOJu0Yx3Asi2/gCPbE7TtJre/WoTyXQm60J9X0TTULiJV601y3AsIplw Zi7HydgRVdkIqZLZyWkx4qgjVxsSzKWxXNeH8NAXMv0jNTPZPh2V42haRRcmyflNlOfZlfAhDPx 3 X-Gm-Gg: ASbGnctn3v2Nwqr3dLeozO/wNN21PoILJRZI34mUqgIg8wdwVQAz5EkoRfvpmDRab2K h8S53yLWs4amE0nzdbpeDqdiSFwIMq8lgr2m8LkPN9k36VPBCdYCN0HWbww/eIDjXVslfJvTRXp 3j24uex6B6DtMuHzIqAlwdJLrzpSwbtDyy41r2zx8yR+GcY8cr0EnLVTchDJCmMb9bmNnfbRHI5 ldkuAvmJuW1oSYOXCwlcWSkdJv1Gw81vxzyMvYv315byMv391BMyEZPurjzUDrkUOopfrMvS29t 6HdgE/impmkFq7zXrPI89g== X-Google-Smtp-Source: AGHT+IFhgZxpyzdRj713xWVIn5lnq5dMIdipHJPkhLMIEEbSziCo236oEOGt7kn6Wpg3lXZNnJqwCQ== X-Received: by 2002:a05:6a21:1517:b0:1e1:adb9:d1a6 with SMTP id adf61e73a8af0-1e5e08439e8mr55857078637.41.1735549770532; Mon, 30 Dec 2024 01:09:30 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 04/15] mm: pgtable: add statistics for P4D level page table Date: Mon, 30 Dec 2024 17:07:39 +0800 Message-Id: <2fa644e37ab917292f5c342e40fa805aa91afbbd.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9247620008 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: fk14ahoxwu4w1whc45ymmsrwdfbfhr6b X-HE-Tag: 1735549750-568224 X-HE-Meta: U2FsdGVkX18G7fu3a3L12gBJNbDS5zD7PH6huIelmVCO+e036lmZk2BNVdPZO/WuDyYn01XttZxjrDifCHLrwfkJDIgWi0iHgwrMnuMYWI5gMUHEHgS6YPEPZWrTyp87cw0irF2vHUOmwkJVC6vmduClkjykXpzl8uiRNQkxIEp1znKqFbVILjbPKFIccKXUk44F2t4SwdGN0Pzn0HtKf+kXb6JP801wuDj+vPhHMg4t4Qrc2xbxkZk2wU6wWAZQY0B/WoAZyV/inj2x+0TYCGwxOFC+TSElXan1K1CMH+9u8WL7EORDOe29TNvlXOhMLZplWlPM/qt4OyZ+sXAOt0/DJ71wcsOh1x3kbgyjdK4If0jKLyIUK6y+NVpQjoJN8ZReyYWiX67SZzy3McD7+inWi4pOuQvIkHhimgXCLGQ9pmNGjCJbB7bop/7pcS7AuBKK48v0EZkb1Dj4eZxUAKi1unDcTux1F0bYulat2I69yzAX95ws2BOXrSt0d0IQ7olnlcVCcynrmzK8EaZWXqMmrbFvzqmAvU3udcolpTuwHWcOrPsSH9tIHPaTlLymMCuJgxX0zAs8zYnZw8KxNZOXS5WTf20QvalEsqyyNHRcvvSf98f7XTtQFMIq1BWo4gFKKGlhv5PaLXu2wnC9nCPCvcivKBz0oT4irlM6UFN86GLLRthKgAuOwzQqh95vZOJQhrG7oUU0tu+jKlLbUPy+rhsXiH4yZsw920IoyTbMX4rZCcKX7hZCddURZa29e/Mm4Yg/G6o6mwE7EAoUX4AwwlZQQDW6Nx1t/rp1z0E/aSv6m4B/cFF1ZU5Kh3ax5KjPZ1jjiK88mjZ5UNfDcNF0WbcKdXcCAJoLiKLUtAOT5ZmwQO4DlSLrNL+IFNShAgXZSvHleS29+HqJVYzy+GM9+jhmhZQasoM0GTSjnyEUeog9zq82Mbqpe7ujXJnTIYguEZoiyaHztaYWSND fJfXPce3 QIi7QiWJ4Zc28QbItFELNr9Q6OxQd4Za8U52oLqsWSA4Pq8iJv2feZSkwwXLSsLOqK2bJhb1gqrpRkXTqo71mUHn5TOzqUVxfpjVprGyNkKdm8XBWR846YOWYEQ+1FxlbB9dazVjfDmKTYqTfeLJAyH5n1qz2xFnXpc6m1DnVS2A40Aje/XdXigh51mm+/uFE/S45cU2YOafmtO1pKNooP++Gnb0y+rVq9u0W8ZyBWB4TUJZV1ui6uh9/9oYDzTgzIeGvc1U0NJCEbOvJFGQEX1CIFYS3vV58j6HXvUh4G+Ai4aQX8tEMuAvTcP7BQKrGj7rj7X0+4CkZIuW/XQevGPYR3X4LSUDoDq/ZniMmgXxim6nPWMlKCngfGOyhI7h8dpxa6jWR80oMBanIAFwulhKBwiK2cPzsuUuLlWAX0xI+BIo22QQLkaufsra882d85VVM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Mon Dec 30 09:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FE97E77188 for ; Mon, 30 Dec 2024 09:09:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8A1C6B0096; Mon, 30 Dec 2024 04:09:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A12FF6B0098; Mon, 30 Dec 2024 04:09:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8656D6B0099; Mon, 30 Dec 2024 04:09:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 647666B0096 for ; Mon, 30 Dec 2024 04:09:46 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E6A3780356 for ; Mon, 30 Dec 2024 09:09:45 +0000 (UTC) X-FDA: 82951050420.03.375D57B Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf05.hostedemail.com (Postfix) with ESMTP id EBDD7100011 for ; Mon, 30 Dec 2024 09:08:14 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bWlxbRaU; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549740; a=rsa-sha256; cv=none; b=Eqetu8A99SDeTfrOOMO85ic9HCFyXz6SSUvZ1jA4/2kZFO/566iFY1oYug9ED4eM7Ps6mm gQV5dbhZBbSW9eRuo/DrA4mNLsXSaFHvKob9vmLN2fxr4BbRSFBHTlRgc0I5rYgzYZgSZm M4npFuryjfqMhbiV3grdNuN0raZOnig= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bWlxbRaU; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=D957MqyP9+wM7rHqOinKD/i1TDWLZL3ap9zCIXsB5pgYsWXKY/778c03hTNeRmjrdAemN6 hhSHORUZFw52M8TQEE9+g81dEl8yHrlnrqAYj6wG1yPyDdZQaeiHCGdf6ViRWN6MXnTJvi e3sgXGXhvswyxGxrR6e3tuDB2KcUnJA= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2167141dfa1so112898275ad.1 for ; Mon, 30 Dec 2024 01:09:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549783; x=1736154583; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=bWlxbRaUJKeHzerBoa8lkK6Ylc33lpAPVshC6ppVPNT+uP9Gzxgxsob48S/Uigew+w wiKxuHQ0puDFE+5u2XBwBHi0skuyjCHZ54Qxrz2lyyVFjyUgXCpA6Eb9h7JaXfGiL2VY NGGOd3k3lfLGC3chQFO4tCbdilpeNwVIFBuRxPK58j0slqaolKvCgVj5TLUGX8uv3BOg nWQ4FDGYpIDazzpnMSSZOknXtaZUI5nFrdH38wR3Cu2xzkcu994vaHUHsViWaLtKG0f0 vjCRHg8Uhf04duC6uHvCmj4EneimXER3IvsGb4bleOwa0ZOTte9nVfxP29qHEJOBIhb3 BWhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549783; x=1736154583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=ESCEi+X/p/O+LOKw9sqOonYI4rDLd+6IQfU2aDQM1nNkZl7DL2CeEFf4zaEpFHttZa gx2qVtxD75I/tdDsBbZnIUL2YqVdX0YJ0koSKmBtbD0dOYgRog5HaEjKqMbKE4imzQaE uQhzR2CPxIXScxOcPNYSTImFFMcODEO8XquvtZFUgmYxUlhbUltq9FvpkymbYE0oiaBH VNWtYRKtWJgJABy3wyvaBSF3tO4vs3J2sHL6G0Qj9RUXdNL1olK2UQI2Zkeh+4XnTZst 8tBO+8V+5px5d69pJaMBwJ5MZY3GMdqQUTJHp69sB2bYCMcjswIybRkTy7e8s+lBjygM 0ACg== X-Gm-Message-State: AOJu0YzEqn6ifymd4vVhOuQMRDwQEPvA0BYp9iYcrA9IDv6Z1n9Nov8r nBSfqaYrE7u9zWmy7Q9SVNBf+qXnxrDSScKCT9UYB+4C1tyLoUizIKQucXxprok= X-Gm-Gg: ASbGncsFApVAewF8A4eApHmy8wHD9oeKZigOgIan8NnNIlQ3NEBptQTKsAqtncJ8uK4 PzDmWD+eZWGgAH1MRDu5kNsv7cvh4zBaWw7liooRU7m8b8sUjHCW2YgrxD5nR+qC3GWpXCE3TAW mkKX/WLc35slaP0xFmbD5W7Gxb7e0eKJigLkeOkEHyOOZLqswv//GNIa6HGle7bS9wUrhdkUFNq xeID/XdOFqLGkttZsOVU+/gc8W0Oc56mCvJE1N7QMBP1SMXY/Ucw02RkkY9RDgR+AvX0MrzX/IA oaUW06JoXTmA705wKVoGLQ== X-Google-Smtp-Source: AGHT+IFA+DEWtAOVikEXL1Ne96u+hKYy4bbBxlgPSiON1NV20vNo54NDlhKj2iEhoHPJgNNx6Etn8w== X-Received: by 2002:a17:903:2a87:b0:216:3f6e:fabd with SMTP id d9443c01a7336-219da5cc2d0mr588821545ad.7.1735549782882; Mon, 30 Dec 2024 01:09:42 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:42 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 05/15] arm64: pgtable: use mmu gather to free p4d level page table Date: Mon, 30 Dec 2024 17:07:40 +0800 Message-Id: <7c12112047ac230809aacd0379259414b9b0d3a3.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EBDD7100011 X-Stat-Signature: qbrxc5qrybii5y7qzxjqwcbfukewqghm X-Rspam-User: X-HE-Tag: 1735549694-90682 X-HE-Meta: U2FsdGVkX1+ljoLn/VvdToaSFswhLXqXtoxbQWAlc4tK0GsFRf5FBSC5rjN1D6iPoS2jrAenBZhgdr0WGIJWSx5t91OhnPAIkVb+2TigWVKlHVSK+WRy7Rgj+CPHGxKNscLnTenEjUs2ndsXCHyqE+PdsSSxV5W5BV1cXESrpyxTUR+mJUluR5ckOBtzw3oJuP3AxemvADU/6gQx6TNTX7vRvXdB37uVyNlfY6Sb/rsJSi1/HiyeJ+F7+scZ/Yka6OT3L4q3Q0mX3wtNnaUoZ8t7MBj8cV6J6v/16I6V/M32O5VGkL3qFzGc5Lz8KAHe1GWH689zAv05tpLIDzfzPcKn7k3ERsIpyFktCx1is7sBvm1cWhbBrqswOkjjlBa4JMDlIm5UqySTtY8C3dkWA1j2FNL6ZlBToHUavnLft0jfQ5TCpyH5NZBNaT656DlmlHOpzCjFbbbGmEmT0E1MjTlPWt7iaT5LSTzqR5ksS0nwKSv7l/iEp9Oho6B0nWE1rrK4ZC3DJpzgXq7tzFCfP8fbPbWoeCImRGs33jwzNBI2zU0ZIkrplGoPEvswNb9dtUpeEwGBhP3kLjTzxmrTq7OrLjTsv7i2k62ojiieBOMFG5z8yd4Sq5/WtD7OmswiRORbuZDzPGBaSfSl7cQHqWgd133Msqd8vGKJiW1jG1C8EgaAw8I8EvE+eLMm6G2GuxegUolgiiGzW4m7nREKAV5f1ONwAptk/Jog0OKPcnPBuzfUOxfWV7fe4wgZBbixubdVCnA7zVJ2WOtcZ8X9HMaiV1WUObj+LCODJrJ/ywUJtOsJA/yUP/08dhyvzDlnc9pLx5kjiLRwQNCOJa4Ns+0hVLzXlZt75r9uz2gS9KVfUReTYRn/B6SD/kEQdy1fQgzaXEJAiXTWVsgKWiELpozrfX+VkcMIhNVv6oO0MhtfUgEHhthUfuCQR9F4MVnnNklv9t6gSbpoteGL1R1 NVYglSWV Xywv08vV3OKRMoRkHbWrFFZrGjf/dCfsVEOX+goJg6J2DuQSDnhiORxGu//qw98UJp9FwUnl2VThfxuSqOrkf2CLWJMjccaXS20p3n+hXojxR1V23aNT1/G2kvxIR8WKJcTPI+sCK607yWSyaFcg6FS+5RBSCnCRwTFbwdUsw+jZVAoJW3plSGSXQ0xZwNa7KXpxUVE/k3cVrEcaRZSnVL6Xsp73igqdcdmryzMgq++HoxivkeVonyRij2+dbfC0JD7MNJSF6YFj9xGA5qjR3kq3dLWD7gV3vxRP6oTU35J6yvHkBdqrg8kgXRc+ko1PvZGq6K+3tPVB/meGvVRUckJpbk4wuiJbeTNzT5KiCjgXXnkwdhvVGUIgIe9UDdKrHMr45BoAfahGuBYGwgjRmoX1ImgM7UH0Jl/J31lS8sLmML/ZUajVbB2zPA6mWryBFqzG8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Mon Dec 30 09:07:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD6D3E77188 for ; Mon, 30 Dec 2024 09:09:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D5996B0099; Mon, 30 Dec 2024 04:09:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35D0A6B009A; Mon, 30 Dec 2024 04:09:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AED16B009B; Mon, 30 Dec 2024 04:09:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F186B6B0099 for ; Mon, 30 Dec 2024 04:09:58 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7A230A044E for ; Mon, 30 Dec 2024 09:09:58 +0000 (UTC) X-FDA: 82951051806.30.9D61FE5 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf22.hostedemail.com (Postfix) with ESMTP id 74203C0008 for ; Mon, 30 Dec 2024 09:09:08 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=k3gbn61E; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=zjzAd/pmZMopQJURPYvBbSyuYnZ/4yzH6a2gz25O3MrGUfcECscjn/biqSLOhd5dNBHcGh 50PDQmCvruh7+Ed4A2Yzlq6qPh5pWCB/O+P+jbjXT/AcZUkCmoMEcczlkk9BW162MDsMZ8 7Ftg5XZC23UYeLoNCLx3crB3XPn21J4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=k3gbn61E; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549761; a=rsa-sha256; cv=none; b=2e74YA7vebIREKfeyUnETtn/VbNbQTQlr7jhmyuMO1d/O0Vq+07gyv6/E+pNtq/qJtMopc E+CgVCcuKkuWxDEEU+pWTmtD/Vegj93wTJWJUm2eKqZrlN5/Js1PGLNTr8a3CEJJVsTxy4 k9yW0A19JUOc7C65pXZOmOwnKK3LPdA= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-216728b1836so101117295ad.0 for ; Mon, 30 Dec 2024 01:09:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549795; x=1736154595; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=k3gbn61EmVwCHYy/+rkIYLA3AD3fSKh2vsJgHTRYxG6dm9jfy2+IyflTfAKzJUbqaJ y6HLxwnZzzb1Q4NyC/+5NOLxLd0+SwsUjpKJ7l8igVAZOKMR2r+xWxQAlL/GZI+oSyAF XsrXDAvyqf4pwxTIcBjBVddT24TvJcSr4FiV6tBNPpZI+ukHaPouZWh/+66GFXYCN9T8 KTQGOA9nL9TPrls7pBYbsdVng2BiAa5lBUR0/k4duLpmCQ0sVuVmnaRNaJbdWkVu9fIx 6xolhg7YKEhxExldm4C/pyYQ9sivbiBtiZqIabtUGQX+sBe7Td2/4P/1ShObsDh/9W49 xJyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549795; x=1736154595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=Ydu6SFr6KU3aRmfW+BLwQ0BYL/S7Vc4LG1riosWJAczf4BH2pQCHmJ764EkKeNWEu8 ND2tSP+q/hQ8wJChpvbVc7Iuayn6E1/Q8vQBRSQ/P1aVOFk6JHPhpJqbTTaoyqn2h820 foenKvlp9jUKQk9hbEvb9ssTcD27RwR9dD4FtJinZampdE11GxtfJ+Jy4UxoY1cskcMT 2MgprHaDEQDWfKjEGTmgysSjm2lwkj1pQFZ1/Ill21jBxEEYYwwZWGmj2XeFYzw6SVd9 fmLlQfWBrsNvl3+Hvi9tXDStqbp8xTjxgJX3pF1Ik6ATtQLIiqC5GZny+Zr1KaLzwTr6 +6cA== X-Gm-Message-State: AOJu0YwPyfL9piHyV08vxvUVmwLXH7dBDfzIfLL2rUdAApLpbP5RJ0Nx e1m61ZBYQW9YZQJhEzdI6YS8tLWYLHOMERWTIa6z5PKDeJpruOIJiLxKmqroC6A= X-Gm-Gg: ASbGncvnzgzPTwdyFZ/E2HJRh90EBTp3TDr5O1Sh0AxqBD1ruQ2T4t7JViB0Sg5TG3Z XpjZ97aic+Cj8x4wEvH+oCyWKMWkMuyFViWhU6+ho/gP+myPFE+UNKLKX7NoEf9CjtsX3OirZHw /9/RuXNXcTSmgRgAnZLdA1epXpKIJY7/PoUJh/IiLWApmX5JS6llef6FswrNjRnx9aYxhy88qOu STHg3yZuZhMJNtQL4cRKAEFrZhS1m2bPGzJIZip/inRMx3LudYDRdcL0NcXQz7ynFvC4qoN5mvk CQGsRqpJbTjkzHzpEfUp1Q== X-Google-Smtp-Source: AGHT+IGnRshrEAoIvJb/W9ouyd0xHkDgdKkbYKOXyUcaItniQPw/Z8/M3j6tdqefIia4rt8bUMG2kg== X-Received: by 2002:a05:6a21:328d:b0:1db:f68a:d943 with SMTP id adf61e73a8af0-1e5e0461b11mr51051580637.17.1735549795427; Mon, 30 Dec 2024 01:09:55 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:09:54 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 06/15] s390: pgtable: add statistics for PUD and P4D level page table Date: Mon, 30 Dec 2024 17:07:41 +0800 Message-Id: <35be22a2b1666df729a9fc108c2da5cce266e4be.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 74203C0008 X-Rspamd-Server: rspam12 X-Stat-Signature: 1x78xzikgauy66fo9p41wcmws7n7umt3 X-Rspam-User: X-HE-Tag: 1735549748-702874 X-HE-Meta: U2FsdGVkX188KTbAwWUNzsGWwykJc83M9RfTKfMPHOxnvWrmRTYZrWArMrMjxNunEp3QOqueM8zEkwR2tNh7yWovb+dNfcLMPcTWXPjPiwb8A0GSINHDv8ek3G/CPCj7hZa3YoH6G6DqYXWDPisTkU+vNyKjmRBIVPGbMpt6TZreC+MTyeFjhIedKQkCqtXqcCffcSFIJL51D7UjVKJlPfQvRy1CdVIDPC0tS38fDa05n0GYNuPWKSERtf8LNRCnI09X0cEnivAeGevLw5t3voN7zbqGSZu5S/YAmT/jzHeEGTOm1VOrcg92kvQhYW6ZReO508SQ/dicf/8XEnMRIhDyldS4KsUwfWAmEMtcmp9ZOetl1YpJG7nhVpAfSm2wO4hyQTKTd5QE3enCMtHTMzOwIsbPGu5JuPir/OrJpKbhWLo8whUxs2+U8isdOrkmeKaq1VxAuxnxnBM2GzMuPJE3CIwsB+lVKXu5mRtQpUvcDb9O/DMXmZHG7CWx8Rw12BI5yXNA0vWk4G5tGFBhDXvoPKb8XomtnjC5efuC8kF0pNP2mx9LqQkdkvbhvMURasrf5dxpP8/BQixA2S71VO9AbXZ1B2khXOvkC+gWwSj/zMsLsZW2niHB2ZSjqD4jlPmW/R4y/WDpJdBk7sjaTnYXfZjoD3GzHXSUG7A89VisCjQg3x1Y7U8HoGVNfDjsgMbN/FI3wx73/ZmfolKxFj7siSA3kxF0P4V0I6zlEoFuKP/9dE5shPX1mdAecnmHBcm4goOCc2z68Ew0tapbG1lREJ4WaDOMVr4zLdINuKpYx4ZV3P123gxxaPJxaiJjZfiQB9IRmyPIvOIfGbBV6EM+/r1DBxvqsQXJx78XGkdqVsJ3UWqpvmth00zvryi1NYJUZkZ8OG6AYPrX0+9woQopJF5wgbp/gGCntV7fQ+nictZIKHu4sdiA3TE5dhvbuAAPwDt7SpSj/aQVRzF iJwjGUW1 4KSHMFq9NIGej5EVy39x+DlcD9Gk9/qp00lMlUbJmwPm63KOXPMdlRHv9hVXVZ3PN93r6oibFA2X6W0yOTQAa6XfduZjsDx1bKA1Y0o/Lroq9E2BSIbVYnvysU0+a7hudXWXMOjSzsZmVrR1iL7hNtkbh8bOGTxRRlZPHyNtnZZefYAYPWY/cp2s6kQObogMIpY1dCz99NHADCKA4uEngfwnGkeRVVTz43fCIjKG5uPK1qsex+Qipu5zjJ3RDYoZEmt4ia9Vk7Rh+VgYlqxxxM27XOAdlCiQiwHuggvMft1b2Jp5Su0vviH9wplA+HSdkB72Trm2sXll1yqQNUyulTMETbhpi3Etn6S3Zbv+NEBHgEepmv/VS9Qz2hvaFxwf9N8tR04AGxrOyAljhX2WwV+oct9H6ULQwvMfb/RJeGIVJglXyLklMQkXX+Muz4kddgnvjSDAwFtcOInd41Q62NDBMlNK2gkY3L5BUodUpSf7IGqOHpqJ9w2Ypv7WvAi56gPYwlgBvafieDpp5CHaa4u2Vm3gwtgSz8Xbp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */ From patchwork Mon Dec 30 09:07:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B7E3E7718F for ; Mon, 30 Dec 2024 09:10:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0C366B009B; Mon, 30 Dec 2024 04:10:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBDB36B009C; Mon, 30 Dec 2024 04:10:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE7DE6B009D; Mon, 30 Dec 2024 04:10:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 983C06B009B for ; Mon, 30 Dec 2024 04:10:11 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4FE35160517 for ; Mon, 30 Dec 2024 09:10:11 +0000 (UTC) X-FDA: 82951052436.26.CF613FE Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf18.hostedemail.com (Postfix) with ESMTP id 2544B1C0011 for ; Mon, 30 Dec 2024 09:09:47 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VDw0Pk+y; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=N+k6zZ9bhPl7QoKNI5Bt5mCM4Vrr2m+fNEkfmSTMDTXeFdXFwJpiCGmbBmpteWp17IVafv 6HYQbTz4Hc9nnCfQqFIWNvgxXr98IVszPOUDtovbbRVctPO6kRPqkojEbLpJFphx4U0wVC UQ1FWXFe1MvPSzSbadnR6D0R0vIsNUI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VDw0Pk+y; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549777; a=rsa-sha256; cv=none; b=uWd2h/HzwqkS/0GIp7V6IplyaFmY91+pqlEdw6QtP+A8SazeVcL8p1ISfPJw/VefUQmR3b 5zjc+kmVYCfIZFITWjScQZL8mC7kuCajVkzsoa7WaBQXr4ap4mCHHHqXTxwMow61G8f4VL HcilmGMndBqGqIKY187OZDGRkN3VKMg= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2156e078563so101236515ad.2 for ; Mon, 30 Dec 2024 01:10:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549808; x=1736154608; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=VDw0Pk+yazLgveQ44dNz6LZ4HIGz1Npbxsmc7aGR9c76oQyxBxe/PLopdomTT00dAS ZMQo1ysjvDLlw+xfJYMQ8rnjIaaUOssUPqttCjzMkPJOF48vFR0pNHMb4gc0AedCl+T9 rTKbMruD6o5/8R3ilkOSk7GdEGbBOoM/oblCopeWwhf67VUaL2xa3Rq2bB4oYH0fcGCp cpmdiH3nG+f75eM7XJFiwVBHL2NmA07cO0j0rnBZyXj51RVzsMExd3TeKSxpaTjof8OY c9tSI+G+s2uPjhUokH81pgh0akdEhyMjrrVkwcIofgUtIZV9Pp12DfG/7XzODVSBBlYX hRYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549808; x=1736154608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=VYxwVx85ovwjvxj24AQVC561hkjRywRNReT78ywUaxIJaRyC4y8akCoz5YzTNvedDK oa7O5w/PGGCpCwJsxM4hhRf68xIuJ+EPBxuEnZ57Ynh1VANqlSY8uLLyHY7lMK7hb6rA NVtdkw7lA2VNg281TDgTvm1WK3b6EFKVzEeXheFoPdy4UbpHjo3neAVZ28dl08THN/3j ElCBJNIFL7p4ipsVoWz7lj5g7mgFiiI7zNvi+NjQ3jQtMqcmYvGOGozNP5tF1Vcxc10Q boPCXoX7i2tNTuNEbYhQcQV+niBDc4Ps38lRjCF6GipIUH2/2gd/c+/xg6cdhjtZxF7V xbBg== X-Gm-Message-State: AOJu0YwjcHBusGQ8WWHvzU4rfmAG7QTPijUbq/LfryFItroDGNrxQJjj FXwRZiQ0+fmyR5tMqw+EKlpXmWxh6qCYMD0dLguvhNKGkvHC9W8WXD26UXBfbHQ= X-Gm-Gg: ASbGncsEDL7Klz8EMwsV34Lldd5Thn1OGrMjCRbhq9IiBqiMgjHbUhuR04WUMU0UDmh Pd1WLlVdJ0WLyID1t65GfXpxK5Mh2DjQvVgx5OlKQXhrIWute59w3/7DiyQzeggHl99kB1p33If 2ANWFOsS/hbGwAtQJdV7l4w7fhR23wIpO1rJrJDvlYF7F6HB7HurkWGePZLr5VuR9SPuSghBZVd 4vIgn4EPqgO4B1DyA4xD1D/TkHGOR44TqVLxE2rvwsZIVZvqISQq1sjsgeiUkJJjGo5nfdch3QV CDF1bk3WBBEPVuHEEPN+2g== X-Google-Smtp-Source: AGHT+IG+hQGJZEvqJjf0GGYYEDMNOYDO3VKGK36HZYKhi78RAlEt/pRApnM1X9rb4tsPHPP4epGREg== X-Received: by 2002:a05:6a20:431d:b0:1e1:bf3d:a190 with SMTP id adf61e73a8af0-1e5e080c83fmr53344548637.30.1735549807869; Mon, 30 Dec 2024 01:10:07 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.09.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:07 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 07/15] mm: pgtable: introduce pagetable_dtor() Date: Mon, 30 Dec 2024 17:07:42 +0800 Message-Id: <8ada95453180c71b7fca92b9a9f11fa0f92d45a6.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 3tfccqe9nasyc39ie9upwjcqi4bi3ytg X-Rspamd-Queue-Id: 2544B1C0011 X-Rspam-User: X-HE-Tag: 1735549787-382457 X-HE-Meta: U2FsdGVkX1/YeRF3mwya/1OyGSX23qPzmZCo5X5SHRGs63m0hfhsNvm32bK7qQVX/f3KpbWkSbh15ZY/o7EW2Lpiics+kE9OOW2qlS0mbCJYylDmRaGXLElvnijCb/XOwqJCGmO78frcq2wmACLyii4SIC0sx2rXegmuh6J6rzMWHT8Eq5i06qgxuJAgIKjxFwNcr8yQ0737MZ8EfpIMi6mzu3+yU9nX2iNF5BnUb6GlC44+iJJAhE9cdhNdIxSnK1Tyb0dDzeeVf9RyDc8HnWFnUAC4Xge7AhgMzyDcOPPDoeWfX7N5ikAQ/B1F7zDMixXAemg2uS26P+HUya/htMDkds2+AcDaL8EER8D+G+/2YM8+SwZzdogOdo2cYsQT+rQnMudgLrLfB/sIjJv+y9PCFf/bEWePSVkL50Ynhxc/eoY4Y6CVdzEL+6SoeSYborZPmAr+KdtVBd3Rt02xpzFG9OvqeRh9Po8/AZ7st/g/KWgsGwH9tqscPqz6Rk0vNghomKWeqFFCjGi4XyvifvqUPHEhqTThmCXVR1BvupZ4x0qUZ7NLqIRTEv5qVy9uH4QDM3/e9QEuS7j/kqZWW74Ore8LOC0z+vzqmJvowqvbX0k60AdyMF1kuQmQ82zDdg4Fy46DeHv0sDRG822zK61pJpDDzHjnMGvTh168WQirKB/Mr0IwOdOtnouIc61gDI2jGrNV3JMxeaH+g4vA3x7vADqKWPN4ixCSpGtYV5PDk9bp+15Y6Hl2WdzMfAK1WODayBWm/FLCNWaEAzrkm/8OdDFn6x9wGTmgZomUSW4v/x5JwApfpFbn63kK89N3ACEWCV3arF5fBlRBJpgMlTG5qYGeKQ1LsO60/U1Hwwv80YcFmZquaWovPrfMH+t81P/9w5AClbt1E2DY4qFsvE+ecDj/RybkZDLZZJDo1AX24bi9j5dmV1NhUVEKH/r84YIuDG25xEWtG48DJES hXuNka4n hE4ssiTiDNwM5YMVq1wNSiCG4xHKB6K8gcwrYdtNDBSzjS5zxBZzI0CNPPug00KEy2T4gb/MVY7pOVI51h0gKQQVzCA/Ic/m+jrMQ4y4/XUVWpnyUzYu06cXFStuJm7WNebZLeAqGRlvy6n0Eh29oxdXFqnPvXeHTM7Ppn3CSzhQCTzHX4kY/8yVzYfdBKMMQlSkimMTy0tVpeJNgtLPlJUhIbNiA/AwCdO8LXPPCvEHL4syYskPoCy5ZXbqsLLhSoBp5ny4koNJSZC8fToIeMPMHMdY75b4OgWtznkz7mthbpgvv1eU45QTBoYUSkH5wNX8z8o37nsD0ND979o2ExnM2u+vPhB8dZvNLwdckr9cB5amWhRZWEQWN4Y2sElpI+0lPV+ywPcd/sV0gieJMVktcNqtfrCzmy/EYygeRAuTwEiVc5o30RbIZyq49BmkxkU679tAIpbyLBTV3EC4KSACswCOm+ZaFf1QoVBNYNoA3/xLL239zBYBKCQB/Ej043fbRpgZZFd/B0i6S/HjTBXb/rIJy/po99gQVAqJI+NzaFoGx3exm3G5BIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 30 09:07:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F7CDE77194 for ; Mon, 30 Dec 2024 09:10:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE5C16B009D; Mon, 30 Dec 2024 04:10:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D6F896B009E; Mon, 30 Dec 2024 04:10:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE94D6B009F; Mon, 30 Dec 2024 04:10:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9D0C86B009D for ; Mon, 30 Dec 2024 04:10:23 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4C40E804BB for ; Mon, 30 Dec 2024 09:10:23 +0000 (UTC) X-FDA: 82951052982.12.5249077 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf07.hostedemail.com (Postfix) with ESMTP id D2E2E40018 for ; Mon, 30 Dec 2024 09:08:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="TuxIPe8/"; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=YbsADdT8UX7pF6xuL4d5QLIk/8SEGhvNFgUKgxfODdadyaPE8cr6NRM9KwiZqYOPfJLfSZ xFf2F2dL3NskVSyd1nYvqf2RPk4W/GoCkgxQFmLNBMIkt66fNqh1SpJ90Ah41YIZMTOx4D zYN4OiIc1YzC4h2hG3F24k2VbVt5Amg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="TuxIPe8/"; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549789; a=rsa-sha256; cv=none; b=O1KiU74KemWYB0oPO8N19iOEujioLo+3Cluz7aLvtpf31nsnl8K2qOAOYIjMCpZLWJ/DS8 OpnA3073zWTRpUPBD5wApGvanmnByPX4kF5jjx7I1zFlhTvituETPuECIlCHj6ME8saXNt 3vGEOGAb9voa7LIb01Ree6m6p3AV5Ig= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21661be2c2dso104540365ad.1 for ; Mon, 30 Dec 2024 01:10:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549820; x=1736154620; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=TuxIPe8/g9J6YPaj8qY6wB/xiVy1WMu0bcxtLzHGhF+wAIcKzKoMjY0bGRZ3BqQapL vIeFbVNyFbv1npAWXppoNDkOwVWVkloFEQtkM5dIhnOaO1w942dmZJcfsfZ+gBlS820s 08S1jLbMntmjCmMFuJRtH51NuL0qtBUXkVo8BCJwA12lA8gBIPtq2NS2jAlgwejmjUY0 AFJuKep2GI+5MHWeHd2hJqkoti1RkhUMfofjB2q+/q04cKxWaJWwCIIV+Vc88IpZY2SJ rkrnb/5Y/Vr4FothE+8f04lU0j7JGrzPqPbbnBE/qNHGzJBq61krU27ITXIoKIlhWDu5 B0HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549820; x=1736154620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=bmtbTSOqR1/8ZgsiYxShjBxVVNhl+bKSxbYfok9u5MsvtYcOrEWtQvc+Joh4zO7NZu 6F3EnWZoX2xQlDogcjmFXrpT/d5nOB/t68oEK+qCoCnu+dmFLtuN/jROWYMCfaodWtJ0 1kFqbvmpHpXbiqpZETsxxhiJBOqcKhMnqGYYD1GAwdS4iX17YwpyO5c99mm0RyEZJ/8Q 5PAtDng3nxYcUO0T3AwRM4I1vIJoLzYhyCKdQf7xJ33qXN57J5p/UPsV7Nzk14rwcP7E 1KIyJpaSRkpNygohKCTDIHWnRTDMLRakpOvg48LnMxZp5Sh33jj85phgGit6n7/55GFh poOA== X-Gm-Message-State: AOJu0Yx8e3Mey/pMJ2GRsvxC8ZXckArqzaOMnrS/JBQesR3NRi3QOd2H uygVR6igWhYVTfudeOIkchyap70mXbsbxevrgNSJona/hwc/eKz331aUeR3jSdw= X-Gm-Gg: ASbGnctxFoTpBW7uJ5zDQkC4/tTAsy7z4hIap5/cGxWCaFCpIxqjnJ6JObVNB+vfkm8 VHsbm3AZ5CZtvGj2ocEL1KDIAxvQvoXkjCIQJrFAKYtLqCUmjrzB17jnc+HsyCY+l7tGkGp9fHw OyQZEBxwzJKC0qwvNAH0/TG0w31pqe2OSYZwCpZbq/fjYbUgm4DrLS/uKRXBKJ/agXIrFA0898y AS40uKpAmwZ07NBlgjWo6XSxxI03M7ICqIAmHIC8LPt4ckxJ4b/QMotnb7P7ruzGAYuCbimpW9z YUARo8xFqYlbzGJSxmgRRA== X-Google-Smtp-Source: AGHT+IFX/3Df8AQRXbgF3uoP6h0ZSX98E/foDuPWt5IdgjLk00IpLZqU8w+lhMMb/JJOCncpjT7B0g== X-Received: by 2002:a05:6a21:670b:b0:1dc:7907:6d67 with SMTP id adf61e73a8af0-1e5e081c8d0mr60502051637.40.1735549820172; Mon, 30 Dec 2024 01:10:20 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 08/15] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:43 +0800 Message-Id: <955162bfbbcd9fbb3b074e1fe2aef4f64b61d6f9.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: hm3w5z5n9sdjw1jpzemccaycfxhdyaap X-Rspamd-Queue-Id: D2E2E40018 X-Rspam-User: X-HE-Tag: 1735549738-101104 X-HE-Meta: U2FsdGVkX1+lYwuSj0yZb27EqXTK4UxPL0qGcNOKUwznanPHPmGKYR5+Pfuio9xxCDzmF+79ox3aPhm+f8aI9pUDlJSGBptLfp7bZ3Wk2movNmnr9BYNExZmRosfXmts8POY51y/+g+V9p5U3oP6p54M3vj19KDTRYZJlZP0iP1YAH+Eqv0a8fmz1D1lc0Kqvxo0luOT1D26kic244wm8gYvrGBPqoj9ih61d9lPpk6KBE4dc+HoUZzlbDZ/r7FaFbVYe2xwWB0cppUpZqyj70P9UOdW/4AgvX7q9QtS9kQgHOjFgz1/Ld/PIQptTeIU8AEnciWFO0rghN0o8EfEB32hLjX1PwwkOU8m5LMxJ0tg45la5KB8pBVAucfuS1ouOSpbVd9NYlH/ekDiqSfEaqKwhhQSTD+WxAA2VKKZ0YLq3HQZg4ggjljrgJYDyyl1SlY25T25Y0qCAD4YZK+aVXtzrhtywwta/AniBF8JeQ1fy3TM/xFO/9XyAnoTsklxl12X4BzB/mtPGdiQaZAMmelVRDZHCGmwXmydqCyIFmJleO9KqWpNWEaORZ/RxM9yBYYC6EVrIhGhyifkvW4zob1zl/jhWSrvFQkJWkTNYCL/lcsuIz6lAAFkSixeyXDQ0Pqei98ghrFLBnnZ+c08SWxCenIhyzugF3jBCbZkv/m1uX0/1plqlnDMFkuBUDEZC1Sygm5krALnGelyquWqq9yPmGSwd6BumEMrQMwkgROgYyqa3KJ1mrdSZwpFBjBXFODaiATyxqi6AUAgcHPIYFMWC9sDYGCqpPQBSwpZqdVAyUsRDqq8AZ65ZkZQueoAaXHc/pxVJ0R8Yziy1byUkpPZz+RuFzPbjWh68s7ifVdy4LuKxemnT/XgH/l2aATRvQzMhSZlxRdp0WjGFSgItsqrRCjlDJvxFaOgFBb2131DUeGhuQkB4xKTdrrNcx37NtcJtQL9b1YUPyuh65s AlCcvW0c HZr/eCSIOTRjE2M5dGq6izdd0c+4/HFDr78jgi/Zajeuuh1cNI+0+8tOQi0eHrZSmYdhPy0xLQupJGGu4zJ4y8XEVhavNCkwQFXzIwM45P6P6hPxAEM7aaVq10Ua8NfCi1W9uzF/ucPLwVSacdf7Ru7TbhTO8dpwyJeabo8PXbS1IfMrMNjeAvS+8vBpogDIwUzni7Mh9Hu30fL/LHvtvz4g4DH79nKfdu6MaIf/iFpzqqAkZJe9RqUtZ6XKpbOqnAMaVLkymTzV8HwWnQb+Vgylyvye6IXxM2ZW3CFeHGjo+ufHZE2fr2+fEaut410LjlcIeNcRrOjtT1k+Z7IRTE9oEAoHzD3BfPJ8csBLWUEK7Qu96v7ouj3r7r04jwqFJazrr1xzAaG+yAFYm0wcsA4APuSTIsEbg6x1Pa4Vm9XgVJXEJzhAItbdFuOY9o8H+2evT5fqF+AEKTBczlwlzOe1cb1ZB20xE4B34U1hRkiGDCbD3/kcQcYbSLsAZgeIV/9tMSWihbjFrl7t0aFCiI6W76TFQ38+pSGOuSneA5AWuFgZCRq/gXO+1TA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Mon Dec 30 09:07:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F6C9E7718F for ; Mon, 30 Dec 2024 09:10:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 225B26B009F; Mon, 30 Dec 2024 04:10:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AF256B00A0; Mon, 30 Dec 2024 04:10:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 029746B00A1; Mon, 30 Dec 2024 04:10:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D85196B009F for ; Mon, 30 Dec 2024 04:10:35 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 95165A04DD for ; Mon, 30 Dec 2024 09:10:35 +0000 (UTC) X-FDA: 82951053696.15.1B3EF50 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf18.hostedemail.com (Postfix) with ESMTP id A07591C0007 for ; Mon, 30 Dec 2024 09:10:12 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Gr3tDr8E; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549790; a=rsa-sha256; cv=none; b=x9+wJUn3FfkIPXs0EkGlCnwGP2Dm/nShw9kLePzQAnMuVIiTnfdHi5UW3BeqQjZBJR4Xqr SGMz8r2L4n3M+8J5IBgm4xvHBmUCxQ/0xh7mYRODporgasOKiu4G6cSQPuFl2mSJwUtQgj pIq+gpNZoWoA6V51JluVBCYqugRUFMg= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Gr3tDr8E; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549790; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=EeDlI2zqXh8v9G/+5O7SMTn0KLabnSAVQ3UuSswWM/uRT9snM2mxDRGCKNzHagp7PTtkKm Wb6Czax1fK0qgjrKpRqvEvIdxviCbYX9kEHG6tWZoICJt8NWsFFqj/gegKr1l3Kh2CfBAc uxBbc61HOeh9oRD+QIrTETBd2PzZL+0= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2165cb60719so116242045ad.0 for ; Mon, 30 Dec 2024 01:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549832; x=1736154632; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=Gr3tDr8E9UhqbUHNLTsEwDz+ajtLOmq1O5/Lvcdb1wygErDSNfi5KDQSeKPaR9HPMF HXrH23YdIdrywoqd2+S7XUf9td+yieaCrXYDWf9wykd7FaDZzTb5QWOgv0P1ind+aEtD P6CtMRVfGgACIqcNA4JHhyNIYQXBM5Xqdc3QqF9OYDnB24M3XKdIzBxpaX8r2LJcDe/G QQz/iOtmdH645NPsD6b/GOcYBxt3aliuce29rkfdOkhoGPe61Guu6B0l2xqHDvOGHgH1 6U8Kg7dgJ+k7lplHeN/3CR2/AO49M6u/RO6XUExsIekgQU0P43xu8e2zpKOlzh6TPkHV 00zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549832; x=1736154632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=BItr31GFcaSwIIAeoPq0dPq5dlFRlJA1c0bbuHIKx1YuKZNf7U4oNjMGou8Gye6nuN qMqsbUGSQgtq9dFOg3f69+/YlNSrsJHdEzy/W4gKPSkW8OvnHzkOBWgEJsdEyawRqN2M DQlvuLcw2MkQ7DPHk9k0vSw4m7ixLokxyW0kkge9oCVahLLNBgCPQvnxm2ji/+vOdHn5 u6UbNK9vRsmTECZ2nt/yDGKJYv0OcwHaCbprVje3O/C3JNNaKtDDCU/MRFe4Y1Mydy11 T5Bc/uKivqQS/b22e+cE/ReRVmPzJX2fQG2RokUBG87zIfsba0/90Eaf0Sqgnd/2ehie wEpg== X-Gm-Message-State: AOJu0Yx3WPps+DIquxUlOPZnjLRaGjTpwLanRdXXfXbjzkvl/ArXrA5t 5dgWEcweD0h7er5nzYbwG2wFbpe4ee2zfOZOzG4zrK9JW4VcdBReqkU4lZHeQuU= X-Gm-Gg: ASbGncuyNHpdJ2yjMpCDf2UfJI4Y5+gxzut2oUPmfhQL+IFbFv8j+p9CeEEEsyWjn3m ctxpNlxqfJ8b0zbovLysGLSX47OLSHTSknZJYdb4N1BuH7T+3zTxGo5ETNWk501mIm0PCoBTtJz KjY/famjY5SIQVaq03LIkxnT0RoVOSwchzFQv9+n+noe3kZ76EN5tFHmfYCCtuinl+iYBDIIA0T c8wRGGt74SSznJPf8UqdG+E2R5EHVuTkb6qKNrGnlAJHL8X2zjpGaWELiuaq0nnmTRJCm2qmkMU X7xB3rOrdzl111xkN3lIig== X-Google-Smtp-Source: AGHT+IGwnjGmNRtsc6wvIZAIM8FV7PRxiURIME1KaVRQ0EplG8wu3sXxm46oGULh37KJjV0x+YOVDQ== X-Received: by 2002:a05:6a00:2181:b0:725:8c0f:6fa3 with SMTP id d2e1a72fcca58-72abdebb85dmr42631070b3a.22.1735549832564; Mon, 30 Dec 2024 01:10:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:32 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 09/15] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A07591C0007 X-Stat-Signature: xeu51dqp5q16d84xjjb4wrjjap8j67sh X-Rspam-User: X-HE-Tag: 1735549812-463572 X-HE-Meta: U2FsdGVkX1/jNAb/XbeC8uGVeaEcnkHz5E8d6U00bYuoMmwWA0CDhsJdciWlYQpmKpplbWU8FOEdPaJOapWr9v1/1hNaec9dbL0MoFiZZygA/c8yha1VPA20vHP9n43zC7t343MYZKOa9jz+V30RFhjYKBs923BzbDN4d7r/IrU7XJtlJ6n6JKWh8pKNhw7uGctDRGFUIfou04UdFQzDWz2/zN0sBFvwjcu0XDSsn++8VIM9qAVP5rNRv73Smbt7NOyew/XHi70QfRA76Cgq6bgLxTwFbzvBgH/eu4p4yJ2/u4QQh3Lo9bS2E/xQNfYJobNN7SdJmfygE8hb5fKA9QONe48SMmWZLcV3R2Mso300GEZHQcopG1VFo2OFb+z9rqGreefZ7tggeJhSv5yxIuFZV4H7BZ8RIHGirbY/s2doFMrK6UV4/DodAYVJ3aL8sQdktARt+52z5Sg+kgGaH7qhwBTWLEfLJMfiVOUDsIIq2raDYQLLSAzCebZ9LDEBD+Art+RW4oX9kqCyaiLsjTDkd83KctJ0+OS7XGdlzILAMXgHri2fi4ELWHKu+U9sU2fw12oITTivAH1liMRkrRXgU7Fm7Kwbnzk/UHf+eCkgcQWHO1hF54MSeTupGyPyh/LAWdHimeKgygkB+KfvOsz/x1WOw3hgOtyIZdjWOTMtGpNQKg6EvheI3vcM9vKb+L5arvO61bKr2wDkJMwOkxzz9Lt+KSzknz4cIIeTiH6D7i8R55b19+5Q6U1A6ihatoKtUu8s7KDA0W/2xYlaPW+ysUMgQ+ST8Q6Pc8EHeJTD3Ve3xLVoR4OA90TIk5S9mabA0CGFf8s7Z/6gLEhW9edKiVA9TR0RNsm/QImNCEcRUNIzezbI/bFCYTVOmIW2p5hNQxrIaGkOz1d5XwXt7bY0ny3T2U5EUjV7M5CkMVtDoXKNGDrUmlaosKdjya5Djb0hFC7UPI5lziMQSgV datbRTp8 0oSa72u5GObS7APAuaoDTI1KJO5/H6Oty40hp0LygsSLt0nDH+uE5TaqDHm8GD/4nXy3Vxrq3x0pNqfPMQCD0FUXR1sCX6s3grMgUJ3nXk9iCM7KQegMRfD9mZHgZBIaTKanhAtp8Kf4qwnm+bSBUlqCig9GK+8diP7KZtiB1k6WVtHHQHee4/KfNwdwedjZqNdQfZyLGn2+YoHd+/pgvG+w5sH8nc7mXKe6pC5E8bJZHDW4ymoAXv7X/k96w5b98MdIaAGd5scYvaLq3knxnY2VXycSYdUwfLfC1GfFPiIwhkR+loa+AqC4UZdhaxqQhyjBRq183IbH6UGXPoXLydM7ZxoylLJF+lYuLV2EzYff7Z+NBgpy9vgehmdA07W7A5jlT2Bd0Ps1tjWc49TKa4XVvq6sbsZHFLfoWI6OpJAcpdFFtifDCQPJm0bPrHTZwn0uLuphRexV/IMZHg7okPEihZsvQhmKVa2vOsdl+lwtEHerKwNeVOL2fIhU0GbwFKY01LxGqBztwa4/DTUfjbbafG+vY3mO5l2iL45fHjTeQnC639Az+DXUliQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Mon Dec 30 09:07:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28520E77188 for ; Mon, 30 Dec 2024 09:10:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EEA86B00A1; Mon, 30 Dec 2024 04:10:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 977666B00A2; Mon, 30 Dec 2024 04:10:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C9B56B00A3; Mon, 30 Dec 2024 04:10:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5A5B56B00A1 for ; Mon, 30 Dec 2024 04:10:48 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 184861C84C9 for ; Mon, 30 Dec 2024 09:10:48 +0000 (UTC) X-FDA: 82951054914.14.1270C0D Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf03.hostedemail.com (Postfix) with ESMTP id 1D02920005 for ; Mon, 30 Dec 2024 09:10:24 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gU551rU5; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=B4Oysvygwzql5Qg7Yjxbq+vHj4pSORmi+pPDhindFQFf14vHLyqrFuDBW8m02PZ505FSml qafPFkNF0qybEvnSa7ClXW8K+aVIH2rir7CSIzGlaMF+e9ytVy5T1J7i53b3rZZYK8MHvi FLgLq51+nq1sz06UuZicagdE7M/fncQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549815; a=rsa-sha256; cv=none; b=VMPiG9NPgR5jD9YuYzpmQOhUpNOUsLLrpKn2+9o6MTGVMjIs0Rqla3t04JvQYTwjMuPSrr XV1coV/+dNdXkZhBnoUS0/WSrrMfV1t4oSYFAPnHeln1UKHrl0pZSNLMgJtjr+nsAB0zae 9ANyvaD+3Dzf4fyXMT8feEyorZCbbrI= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gU551rU5; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21644aca3a0so17748735ad.3 for ; Mon, 30 Dec 2024 01:10:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549845; x=1736154645; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=gU551rU5Ets2/p3S8ICDE2nt3vveC8+/u5rePYTcNGnQWjJRDZmUI7WKP0NtJNV6Tp txfnORGw9Gmy1IoG6vM9f/2v4V2NFFnLy0vbAaRKTs1Nt/ldH2wQvKOdD88zUCRj5Ywc ypclbr7HL2Aw2RLzP6YWI+ATxDC7vBZJxBAF96hGvmYydc+Qj5di0E1mCzj/Hj9Z96Wv h5VNruJJXb72Dx9WrNlyo1HAyheGomSXIEd4VgFhjDhBrr6IQVCTwQjohIcCeut/4e2I KRZY/CExaCXyeiCqp7u2XUrrp305i8H7yOwPfCI6L+eRQeXK3RPRj0ZGtdBaoXgLkjlq iaTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549845; x=1736154645; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=vFD957g0UIG669vMuuum7QAokB3W3B4go4NR56siAB75wMIU/BDbbsz1IyzHwOXW4X Jhrb0l3TyyfRcftpLhy6hiyfusZIWI7wu+a8rARke0WZKAxzJWz0Gg/O1ahM0K22uDJ3 P9xksXAT7WVoennoUygUlKVSI9bSyQ4d7zWQG+hMpIylMlMWTfn/cIoBWQTYuShxOPl0 iTZZe5PQ21Xy3IGF8KarqY9aCqPNKBUSmUMb+xfIuPmLFTd/IMMMzpXIfdyOm/iJaL2I P96DAPRtpFnNPj0YZT0lcK6JKVCwI8sJWe1FBjObZlCU/jofbAk3dGlfM5ELeIkOcQ// QOkA== X-Gm-Message-State: AOJu0YxXbNiF/TJB+9tqqGlynwf7aARkf7QmtD1lduTepFHsPdsL+dDd g7Hv6oqUe3EtqmXZKHksU0GNsK/aT+82fbosBDBzKt29JX2ltJoXyDWSvNAKxJ8= X-Gm-Gg: ASbGncvLS/Tx1P7kGR+w3lAr3HWV8YS3OofkRMSjJvHMrhzCzEyJz3pgBR+NP14Ztmp QTGro1e/TVyKAFLs2LwbrnvRxLtEsnaCPTKGX22QALppgHFYGEka8dqqFBUWvMnYlN1KHsh1e1z pvJcuk6QZuFoIOETvo0O3ScCnWySvxfROBUBEgjAvXpAiTCwCQwxqFad9zWynFxWEu9NQ6DU5dz HVeTSQHCd0AeIpZKKFuyjq9zqqnx4IxYyVLzl9XsIu56B6DRJdIAbGpXz9SFaotCkSsYuuErTem iXFWZETF+hQ+IOw7bXE+uQ== X-Google-Smtp-Source: AGHT+IE9evGbD+yEfyO2G41S4amTNhC2vfWaq9UYV9WQJrC+a4l06PyKmdWx4KtX6MVeEx+cVngAAA== X-Received: by 2002:a05:6a20:c91b:b0:1e1:d22d:cf38 with SMTP id adf61e73a8af0-1e5e05b06bbmr51533885637.21.1735549844993; Mon, 30 Dec 2024 01:10:44 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:44 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 10/15] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:45 +0800 Message-Id: <0e8f0b3835c15e99145e0006ac1020ae45a2b166.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1D02920005 X-Stat-Signature: s9huadoa8koamm6e8cmfpmu45bij9h1t X-Rspam-User: X-HE-Tag: 1735549824-112885 X-HE-Meta: U2FsdGVkX19LOik/Uz05ze1kMCqKFxnpzoJAT1nlGvnBwzY64rb4n4UyE6//K+4JCwvK8bsH3OXYQVoJyll+nWl1Aa3T2Dg5O7xTNWOQKSfbNs4tdZ5fXBERfl84hHUJOBBHUROeah8rNI6K4VYgNqbMTYLO1P/aySA3StImaXmtC9OZKLJJmczL3sVe62Y4zuOlKute3U3GNY9KPVX04QxbSWygfLYlmT1bM40L7U5MQJxkPBu5/fhJ3wArN9/5C06qcgSIA2488R3nrn91ASVrLja1vTOcKOWk1JGksn5mO0qQQVShCQ1Q56pm9Rfdtza4VGCfAJ/uhsLcSFunSH9hWhwe9TB7nqGBtYo4llqqVWbBlKfAyWnucidPfg8LP25x1+076osc2ftEn58qyO8ksQurO6SctE4BHsRWYr3BDkOtMlm2yJ7rftQMFgC0eLmc8QEWo8+iFLhW6OfOExA1jONwccxHVK9+0BKm3qv7V6te3+MzUkKWJD3gH1DvmTidB8V8YfRGmBBImmFqDS9nZtemXkYjN2o0CMzsOTvbV4MS+2MQIr+Jn6DxTGxgfxqw5xHT4pSQoXTpNjj3XUKJ7b8VRyIANswE2kvWr6gZswKjN8BymJ/TrRDaFMDd6EApj0AyfN2oWtIle13LsuZK+Y7FzNKG63oWzRwLf3ngAKSOYPlZC0hG6ig9m8okBCo7V8RexcE2IhocRbNDLeHp8r1J5MOfLZDmZ8/gy9b9YZwwWtD8XM1Ppy0DAW8ENb/eUOHI0Bh1FlsROUpI7idp7XNrSumfFVGrLfdSi7GD/ORJJ8FrsOlfhqLosgGKxwFVlBKbn2BBKk1u5bZjEubqZlQEmRT8emKYnWji2nAcQHfoWMZ0w7HpdI2e3gT/1TEd0NwzywlWSTfXFfPhNOCioyvpGwVYd90SP56QeLQjzlQe4XqyzElPnQNt95eNRtpuoV59BPbihkFxnmL it4Q8OKd yW2w3wZ7N06xdTKGEbpP99ADOBykh+3CnW9gyWG1T9Pt/NDw5ZY3rwkpXy6GH7OuZzCH8krZtbY01r7XCa+XtW2/Ewg+74Rt8llJH6p2HxZOf6/NhmJyLBMh/6mzNiC4RhEpZkqCMEjpaXwSXItzJPKJ69TDZAIoxwB6jogiQX0Dq1PT2FJkMqt2X2Gs7kqAYMfYg4T4mfhydhZPkTDhSD352ndYOBOLljUSjYVUzSQ3Fv4l/utRisXtsy8J/Ob1AFLbIqGIOZArOhC+stz2iMAbhV6hGYnjqK+EiFxe5O1EQcN82tcMmLTd+C0n9Hz0bEwzkaMsCRIEi+rw1Q/WeQs/ODoPG/IkEMUVIv3+4kPCAisf0rntbWx1cJG6Qv4yQvDhmd7Lzbr5lGx3AMe5xPF5RNPu8nI1NWyJeS1o5ccwdqNQChIBN/1e87ZNwLfS646oE4u+RnaFxMS1YuHFB4myQq/QHvKlWhEfpjCy9FgKyg5o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Mon Dec 30 09:07:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F3AAE7718F for ; Mon, 30 Dec 2024 09:11:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02AE56B0092; Mon, 30 Dec 2024 04:11:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EF7316B00A3; Mon, 30 Dec 2024 04:11:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFA3C6B00A4; Mon, 30 Dec 2024 04:11:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B07D46B0092 for ; Mon, 30 Dec 2024 04:11:00 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 740301C8378 for ; Mon, 30 Dec 2024 09:11:00 +0000 (UTC) X-FDA: 82951052604.17.27040C0 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf20.hostedemail.com (Postfix) with ESMTP id 8CFCD1C0016 for ; Mon, 30 Dec 2024 09:10:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T+XnYqe8; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549804; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=eEztf6yKOwIrjCZ8tq2tiikzBbj5Gy1DctcV9NE6RPPsNrpvYaT4SdC3cSuh3W6PqaXU4B 6giOBAgFF4Old2y1Q+mV4qO2MrjdoVNYkcbJJzYXbaX09GNNRSO238Upt12ttDIIIwo+3Q jF6tZuN+XRReyGxO0DknMMp9gWsviJ8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549804; a=rsa-sha256; cv=none; b=Q1WkECU9NTgtPZEiXF8sUFL0iQoNsK1hQKFkgJ66t1kjJql2vm50FTRMOO0lnw25n9NwB/ JWsxD3LljxEJ+EkPjNKocfCsOljkTTS9jZfsCUMgl1BletXKdI7alKfX9xJeJKfbkwQtFU LPKlM06QRe5ZiAratvWZaZdaM6zVWA8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T+XnYqe8; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21634338cfdso167233385ad.2 for ; Mon, 30 Dec 2024 01:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549857; x=1736154657; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=T+XnYqe8XJyiSHwkVsBLbi3N3bCBHoUjy/y0PB1btrT9SYFOoguPljFYIu+UPwTooE +FI8hJRF3BCv07ga7K+cXrJkiBnS3IndAKcbfmp7UoE+R9iTDzyPColQiVtBDDkeE8da qIZH9U9mKgfV31EV7FeEmvBUrc/lUPggtGsfV0MTfn9aQSu2eN0Q6CTZeGTh8XUluvoj JEUcPY/iFmIddALKgMxyy5FKTXr8FJcZ1IbwmO69NIi/rSPg8P+gJrJHqTkUXR+IKFh/ KbI/oSd7zhkcuGjY+ysPSdwRCPOki5xr1tex2p3ChoEjux9fldwrIZ1F1hHBL7jgqAJ5 CnTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549857; x=1736154657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=HR480fBrjeZZWD8UJWyCMcyikal3ZCbTDE63W8a8lmRs7//8SWFwiauHAuQtWioFE/ E1jlj+ZguWFmr6YWWfLv860avX1W2fDck7ueLqEBHVwcwMunPwU/odQLgsAohY8bgQfW 2sGdWuQcTdMdabV9I2ShRImA+pU/c0f4Kcw6hZSkhxiAoMByZ1wXP2OVQgAVy7PAaHWj ULou7qMYTwFPq2zugmCLhR/cPsRJiLWvAwf3SGna3xKGI9YBxrr+tyOdHsbIJW6P01lu G5UKtMXDsJsMpG5NOg9AF8lbW78GngxrUiU+ixqzhcufSBejqJWYWEBjNO/pWkh8HOq9 LO9w== X-Gm-Message-State: AOJu0YzogQ6tZ7+rwakVMBATImKG3d4zIjbeIQj+nT6rR10UIrYbtmQV mehzDEanxSmcHDBHe++UyqEAIJ5OBNTaTt0Q1DUw9dG/08mdyjVWVwSJ0DjN7OQ= X-Gm-Gg: ASbGncsIR74t+E3ywN419EpJ1YTYe9WAIdsJTUHnoAnUKXuEc0eeVO3AJT23UGHUnDI fPQsAX5qbDypZBatghY9U4+vP9+wJXGUxxof5LnQakzvhzA0a72xXMl5hAb79EcJ4qpVm0GWHP5 bmdDtoGRQDbAzPbcCasBsShw+Xsm5nCyDhxp/5jInTXGUuAQ0aHkma0xUg5Mzl+VyDfQIiHkmzP V/s+O5tuCej9Qnoc7/PvVugk0ggPvD6XnItD9eMAgo0Iik3EahI+I8qNfdMUe0AlvZ5LPEwQYn3 J/B6/SJbvShH+x4dx28ysA== X-Google-Smtp-Source: AGHT+IGZk471dqtvPjYzzhjuNb2uivM1pgmRrcDve4hlPbWGQce+RD+FtIOnCoiYNP05sabQ4PQAQA== X-Received: by 2002:a05:6a21:8cc2:b0:1e1:f281:8d36 with SMTP id adf61e73a8af0-1e5e0458dbdmr51544115637.10.1735549857384; Mon, 30 Dec 2024 01:10:57 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:10:56 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 11/15] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:46 +0800 Message-Id: <0dc5a3bf5a692e24379c1d3b879a6d4396f0dbbd.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 8CFCD1C0016 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: npaumsiqodbw335195k5f3q36xoczhfj X-HE-Tag: 1735549810-774678 X-HE-Meta: U2FsdGVkX185xqLQMWRU7EB4dR2FmwapSByWqst25jm2T8bLQo+DY/UidogeZsYLcypQgmsgNEn6vqmWIkCzqiq5N73yE9t3ijcXYokYVtCritdCMem0pA8DC51G0+ahgnLRQurk7nhN2sP0GTl8RnrxyMHeNXRgaPPs6lFrEA4A0V6wRsPIog/9Sqmm7p/rQDxa6PDeihg6MhGFU/zFlbu8T3JYxuK6AluX42mL+wybfIkyPaabWI+IF2oCCC49MweprronpRriPP093YtblHKU+E1H2qRA4qWc4bv4HeN0NGcaSK/+Usjq595UfiT4ZIT677Lv3NbCXEC3+tbDY6JuGoco0Kk8qCGeWUvYgwQO+PwSTM7EWQcbPkUNVxVCtFiohnPpnCzyiGPyTL8YVxLjgjG7Rw0/enuIaeMmGS9ZSPKHQbQinrczxyW+GjNe2hk+v45dFab91+ymaZkGlnTyod2jLoHuU3uaOfXUVgOlzhjKmGofR9zJ2bX/rE/6/bgy1WpJnqI59ZzRAMq8c/yO2Az8u1eA/etZoLHe2doI5P7Hp+xVROv93PLeVJPJahkO14+lT/9wkbehDjf3r+U0J7bI+Y6TZgyUGryWCI0XdpyLjMA4gH2BhlwyrWHq5YvXAMYBHjvBbaDa1TTDLtVAGl4r9GKe14xJvQM3WZvRp/OVEnklU2WB5qmBSiADNgslG7Qrid9Z0V2V8cTdiX9rKJYS7LIMacuRXu4OIKczvOJFtOE3FaBlAsoFODyInDA5rRUbggV1ozhXxxDvRyF7b2w67/yo+4jceEBb2X7NDyUT7iZcYirKe0QbKpswpmguYOewrLOF3UBwHpD14itR0AJNtynpiivFl0DotKYGj6WJEH3l9Y1zIptNRtSkovsA/mw6ySw9oUPLukthrcZkZjln1xY50Rq97yhkiTxIxg20apHLKluzHQExuGDdK49YHLCu03T527vwBbO 5u5qRdy5 aM+DxIy/FMiiaMShl/YKx7vBjcWlrKSt22xhcooNn2kMCCdg6pC6xt3iMz4pfM4qlfcSCYvn6ChqtcvoihkdiUH/CHl5LHguwGyDCLanU9otCba1off+O8Kv+cTawiSFecpRIf+C1Qa067PFxE7bsfxeehCfV09vqDhnzPIOsmF/1RLZASjL47lpmugM7AkuOk6mBdhvsc//idrLKupngR2zTLfVkB137296ZRUAQDfyA/brwhz5U9TAZV1htlgXOJ6XXTV+3dq5pzvw1VOT3XGr40Q/CONiHXX9M+dDA60GlVX8+rvq9sD39eTWj7A+ovvwaq10FMlxaUtgV3/0u7ScDqM8NzJZxV9pRAYspnikb7tD5FtoIacmxlfQPoKOcsSi7G5DBa8StWx2Y3XLAGn5EVgaAPrHCHwRhKQPBybzEGC8aUyysTqt8HMpygmPJ/8HZrCG9hCKOlEaN662QXbOCfLLtdJmrY8mjR/gAMKCLpJRI3f7Dqkdo5yCYDKmBBBWvLzJp1mN+zxY5mXDuPKqchcEiweTVI2z7TFb+5LYeV9seCOOlB4reXqVZpCnSvc5BhnEBTQztn1c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } From patchwork Mon Dec 30 09:07:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5949E77188 for ; Mon, 30 Dec 2024 09:11:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CBFD6B00A5; Mon, 30 Dec 2024 04:11:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7544E6B00A6; Mon, 30 Dec 2024 04:11:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F5056B00A7; Mon, 30 Dec 2024 04:11:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3EED66B00A5 for ; Mon, 30 Dec 2024 04:11:13 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 002041A0552 for ; Mon, 30 Dec 2024 09:11:12 +0000 (UTC) X-FDA: 82951054956.10.AC3F457 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf21.hostedemail.com (Postfix) with ESMTP id F39881C0004 for ; Mon, 30 Dec 2024 09:09:41 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MrYqCF61; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=UAiD6mcZgwqQzEHy4/BbhW8xsPRM8JrjEdYw0oXsu4E/tyAXhgh7bDdSRnvtAsj4c98Vw6 3Aw673S4Vegmb8Ea2OE6gsYEXlmBPqUq84fYa0YBVgO/BlXYwN0CtpDYfwkprVwhZPGf6h EOKsLOpfpOInvYrcDlHZHy8tuNDZlrc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549849; a=rsa-sha256; cv=none; b=o+tVuzsRG+GWTD/l53Vh/S3r5crHB0P9BdxpmcwO56sS6654OwMVoq+B+N1Ry/4uFoIU1k uQAKFy+lAL7Y6+L6lMn2RWRrsmZAZuEEqc7TiJrCOSDDGGjnseUjxXQ93+m5IaTtEwR09z rTZUTzD396c7dbmW9cPeYuS5OaXBZAA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MrYqCF61; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2165448243fso138922315ad.1 for ; Mon, 30 Dec 2024 01:11:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549870; x=1736154670; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=MrYqCF61Aq4VJM9IWW3+hlH8UAg5t/UlkGbPaag+y0mjGLYxUtseU/WOnpuvVp6pLi +6BZUrGcIxStxNjJn3tEjoSdZINJz0sXFw5q4e+LC1HoABejVT3+hbXn3IAPfFkZZCXf VXwY/3eFPdqbARfmBAcp8uGyftn8/EfQGGq5KCqKEQrkHjmljnsC/qD9MmYDQ6LpSuwu Rk8w/jbtetW3yGSzVf1HMSs4+8Q+5I32AnL1PFmyjWCsAT1GZdul87qeSQgMEQZN+Eyf t185qvjDb9YqXtt5jPdEquKSLG67MCo1sMI6Ovr6N5bP0HQGrY4Ydz70gykh2XcjK0qI aM6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549870; x=1736154670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=NMREoIT5Wzzp2v6CWNxQ4SveDEK/ZPQeyNNGFIhtswKbhfB9chw4Vrq8TpNHRTwVbO Zu0RHChAuQausGqwYwsob3lkGcxV48VMpsLIXGrZ6tGdKgVtBvUD1hbXdjbyrJ1dOGY1 kHrk4ExL8L+NKL7/vE3NeDmSXQyogbIQAY93JUpL3DwbeHOQLo1yjm813KCcE78qyPLr 9lQ65OeCDolJ2Zgrv9ULk+Q2JAQxAC7ub+urecEtLBluk70/bnE0AzlDTUulG4V+t9Ax y6roYiShsbbp7DYK3nCco34fi8w40RpzJjTiZrDQh8fxAtwHJAVKySS/zIDW/dXZ83lK tq7g== X-Gm-Message-State: AOJu0YwOATWan3wdEsNpJRRwnhigmxqKG5PLelPw/xz8RHT27so3/56A eacnkcZUbi0A4kswUtpxWfcb8ZLDokzyQpgOjPE0VCkr45tUJiFVJ9QhB0D+1PE= X-Gm-Gg: ASbGncupJJe8Y94HMy+7rDSbUxelHYcxqpr074OK45MrtW7fHgSCOjBwIZrz7Tze6fp swCWQqeojxtA2pl1HYGyne87oLSGMrbFpk2aFnGqGoeLrf9BI/xmJP2qo/9AxdtkO8Q7VdU+6Ag 8SV9NPfY+yEZ8W/n8uno3RrgnJioEBJbn9qPt946Qi6ffWu6qXSu/Sjcy6Qoh6ky5v8gsjFOhOY +d1z49o3IUu9ka96JwZYrsLpilipIfCGrZuLo7pVgckxrfO1T4tSZaaGpp3vQzepaVJqx30fFLG cwCcVDzy3GgdKh6sun2IJg== X-Google-Smtp-Source: AGHT+IHPMMlkiFtH678yJhisFXpRpeLbEmYMWQPlXwvGfvt1NUv1TR7XGzXk5FcrgJpzIcDaQ0LX6w== X-Received: by 2002:a05:6a20:7351:b0:1e0:cadd:f670 with SMTP id adf61e73a8af0-1e5e044c8ebmr53183808637.5.1735549869780; Mon, 30 Dec 2024 01:11:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:09 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 12/15] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: F39881C0004 X-Stat-Signature: y94xpuaru3i8e4nrarhyg8po3neunr38 X-Rspam-User: X-HE-Tag: 1735549781-206217 X-HE-Meta: U2FsdGVkX1/2r/mcYZ3dZH64xZl7tufciTg/UOHCiB77HPtDCUXA3SKIjyEaT2ds2oz0vPt3lQHW4dJTVJxihekbODwRrKPAcYU9TvdexAvLqd6N0oKI0RszTVn1FcJFl7BwN/rB8eg8nlyLJNXR5qEKDECf6hwqSQ/U6xkXbFpoNqn+UXZ4C+J5eg5y99cgbYPWtLBPc9ycERMjjJZQfdHEyWcozEgIqUTJjhi03ACcwbach1bNpC68vr8M/aQf4DEHQnDygBQXZj8QUIXEWix3pUOL9uvt6sUKgqzu4G50iO/aDw/H2oGZnHLqEhq3++FvtUwlWG1DivXKBUbMAjJJX4SHm+MWMdc0/zwpQV0kOCCnM9uG7zT6P6pYoN2gisnB6scgR7E1qna3CZ6SekIpbCzmpunApDpVG85FTCVqpmjsjQZbMjocDAN2oYqVycq4XWyaww8lUctrD+SBg2eQRgnML1tGkxL4nG+mvcJgRsAETg6kcVjAhUpgT3yTR/ylxlO7Za4GF+Z6z8Yb8ng92lbkph95az+Fv5AFC7J+xjbDNQUxmP3hJ8jSp9sDaIHumy7mSk39SwvKcVdP9Zm5hvht2fNEf0u3UmsCD/CtGnmf1/OFUFntDDXYFAoirLdYFszD+MlqiRuGbg7KM/b6q4na4Adxx3OPYiLTy9mKYIzlQdjaky09uRJEe4WVI+oCzR/kbH7c8PMi+GoghvDk9IDY4EoyOLzEjkj8yhb/oZzln6hR5E6TFm0RibBeLT3jzgvZwa4ehJV/AB/C+2g+dmI1/Arch3TSkM8zVJIMOPTtOhf+FVWaJrJgXI39CiupKXMROpDZXdBO2eNQLUYH2RWYhGcQbzu9jn9YkxrZwyW6P8uhy9m2YN8PS7YC7WbKYukA00Ou2VuLNvgKQO0aOW7e6RAiiauNXBN0n1GoLTmd2tTrsq4jQwEvFXcjMT4Qd+vDtqkyNkCnvFu pT7rsu8o CIpVjMLfDc9BBG0KKBZH2Mo2Zu2YlB5nuIGQtEPxtH652RWkiTrLDh60YQM/BsizrqM72gFn8bq+7x4WkowRhciGEa6fOhO671+mFcVktUw+TwoUf3GY1w5unUZAEktVLKKZ4by+z7TiwpIv4ZAzUtVmB196zOaVt3ldxZqnNuAnr1GySCCsnQyWZihMIiqhvvmMekLgN2Kf01dbDVFxjMt1SiOJ39y8Hm3bfLh00HYZJjweU7+I+ai8sKC7G7gJLIMRfu95KZW6I2HYsLwydnITXyxUAAKrgdKuAslxt4b6bJinhFsn52QdwuWWU7keaJBfqaAs9Zor9fSPShqjEgp1EZArOhs3S5LVfOCSw1TpmPu/xyldj+MF7BJMv8KlYW4o+SaMEOLzRO0ibgTm7oMB+yhCh3+88/3CaCYP5uaQBrtb4VTQCbc0TDPpaLJw06rDDE1x0Z6CK/ezJ/wPs7O/B0Ug1AymAxalw2D5lls2qgl60DaWbHHTXUWcxH2vtuDST X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To unify the PxD and PTE TLB free path, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Mon Dec 30 09:07:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73B72E7718F for ; Mon, 30 Dec 2024 09:11:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 003C16B00A7; Mon, 30 Dec 2024 04:11:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECEE86B00A8; Mon, 30 Dec 2024 04:11:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFA886B00A9; Mon, 30 Dec 2024 04:11:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AEFFA6B00A7 for ; Mon, 30 Dec 2024 04:11:25 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5FBEE121889 for ; Mon, 30 Dec 2024 09:11:25 +0000 (UTC) X-FDA: 82951055124.13.5382F1D Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf26.hostedemail.com (Postfix) with ESMTP id 1D309140005 for ; Mon, 30 Dec 2024 09:10:47 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IZMpHBiW; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=Yf6i0NpjWt4fa9qKzfvOReXJpKGdkClD3h4rGU6Gjv3i7QqBpdwiFCjBkq4de+S2OvMKzB eSk86A7oxSfDFQ9ICM6BWEH9n9evRHuAdBFg5xmhnmN4x1MBcxc/YNHrTaufYR08d9zM7b FRsoyoD7td55X5P9i9GCcazCb0hyQMQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549829; a=rsa-sha256; cv=none; b=cvpgioKI6p7/uqG4KhKPU3gLELAk/Gi14pmnAzvDp5l7UhbT1JrrZb+02NDJrdylWNOaan QRxDaJucHVC1+UxmZ4ASBIWJJFVh3QBz/J5exMO7uj79v87TAB6eVbWi1YudunQkkHPaVq tHoyRBBsypTryFLnitNyBouh1IG+xoA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=IZMpHBiW; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-218c8aca5f1so142100355ad.0 for ; Mon, 30 Dec 2024 01:11:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549882; x=1736154682; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=IZMpHBiW14GeXPaBToR6tYZDExa9hcQMi47fEtwCjc0BSTrOxE8nGfE+1YgbfC9320 4nybumBjvUhLC3cmjP7uE+uIDCoaUXbobMaJk6uNI1rW5IJUqjSHIIBJiHPwqCflSI6C UFdxttLWKeApRmGDPcUJKo+Xh4PtMZ02JytzNcKSvVluk+d6fX2d1u4qoht3abf1BEhV COo0HapHD71xe798zO7tY/8RQfofydOp1HCtyHNiTVyXLe0XywDiimsaN8PCEhVueg6S K903hlXvoam0KJxb3eSql8v1frie8tV1mdGux1fL9oHK8/cDzej9zyDemK5IZHQBs/xd 8N8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549882; x=1736154682; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=kKI3P6rwHeBMppu+1BxUCxZI9I8bIJ9hIjqkiwiFwThS2UFgfk4mufNCBJLcn0H2H+ LGTkfVugxPZZczNiejmffkr+IeWy5n9dRxqrKh+LDTJpJMiu/VrO0pAwmC2BA7YreZgK za9e4sqyog5/KBOPeSMQ3mt5tJYagSDZxBONHr0J+BX/Th2ktYn6YcVbHCackCRxZC0Y jaoE4znRqQBSUbmUO1aRxPAgn2dQC8SJN/gELk1Hh1DQVPPvkqT3oS72GPf7EtHe7TVy DH6erXug89iL3a/UrrIjtIavTUMW3APzgBT3SrLI0E95RLImBN2aidbj4mZ/8o63gVzA fT8g== X-Gm-Message-State: AOJu0YzlYG3HuCvba24GkQG5ToCUcJHmQiHZvKoZZTDyC7Asfex06bYf X6mzuEZWRRjYpwILtFKUZ4GZErppendwmHeE1a333K6p8riadAP0l5SZj9Ujqnc= X-Gm-Gg: ASbGncuwVZHHM/ftnwsa3Js7m6u3RPzEB0EhdyzanLMQDyb4J5fot3KRTkVM1AseAmS AszvRA3aSC5CT0vc1L8lT8Dw/xDolDSNVxZ3A6lt1yOAD/STaHEPgBWo2ruLwkol93wTHRIQpIn rMSi86QPx429GhMJatSM/e8bdBigit2p7EEBVm2lrelH8zTL70O9CByOQ6m1gC23JG9l2KYCEDE aQG4dQ2r1u6T5EnavR4KB9vYNKpM7bsgmBF/ZBVXNSILrwaDfJ7Ktfk6YzylMbwbOaYSQbG8olW 0IK8L4xlfNBD0TOLLx6Z2A== X-Google-Smtp-Source: AGHT+IH93LLg/oVZ0Av8lq2Z9OPVl8eXrLxQFXcb3ZPfLzaeZOV1YT5zFyKOR468gBW6qJHDbn+qAQ== X-Received: by 2002:a17:903:947:b0:216:48f4:4f3d with SMTP id d9443c01a7336-219e6e9e004mr571689445ad.13.1735549882158; Mon, 30 Dec 2024 01:11:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 13/15] mm: pgtable: introduce generic __tlb_remove_table() Date: Mon, 30 Dec 2024 17:07:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 1D309140005 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: q98ro5azeexzfywtajw5iabc7dj99qki X-HE-Tag: 1735549847-902402 X-HE-Meta: U2FsdGVkX19OdNjiGi8otfIgRaVXtbaZfcIyeWLDxeFgjDhdRDrFE2H/BGmG2vnUhom4hq3V4ypwgeGG+ANpPKP7PKq7qtEbcgFQAZDtK5JuWyif4aT+sgfCMpfN4K9SyoM0HuQNYP7l3m6d1oKx6C6hO54ZiFtQFTWZuBzPDSoVty9R9eTFBoJpUOombMsY0rxgkOTpmw+DqXr+0ESWdLsY99F4/d/3Yf08Lb+iLd59lhZJs9t4oeH0IFJUZ2d1qKOUO/mMzWmfRUMIHdE2083wGg0KxTJokre06OOMdO+txx4Tht4sKbh7ZVO1n+VGr80CsIGL1s1wP0OhkcaiiQNje4wCuBBpbVNIubz6ZNx01BjleJ2OozhbCHupUPD/BI0MPd8JDvfh1DQC1L2FKUT3Sbejmsp7kKkLocwV1zPX2hSV5pdNCg0wjvyoIWzWfKW9XlBhhLXVGkd6bme/pJeNupXPSV/+SYeRT2zAmFv+SvVDXcvKRxfI4AaWVfSiRqOzPnKu0IWFwLPheML6uTSAL7IOi0eytMZcmhW/c3FR9SmwOClq8ray+Va+7E286KlVkO2hGs7U+UeSjwRHClQpd17jUqCHgP+6Cjxf+dv8bNQW1ln9tN+YcElcJkHeGvSwPCXGWAnO8W+Rz21Il4s9C4s1Uxa344WjVlA6HUOXK6NEJHbKmen8sEVpth2+BItvQ6dVJ4Zrug+2kCy4CNjG7VlOsh9yiTPzZf0lobaTcdJ4Ac5S4QzKTPfCe8VJPETx0Ue6iDFzK5O5JBBYS/Z70pRLNiOfZQOXAcke4io1camio1K3rxMV8PKe66mYOUAZrdPtloN38CAlgtj5i5tOCdTxHRS6F6iDgKPVyFYjG5p3BfIj3ABVg5whmUUb53qw6ifNg3utSKz3DFbkwhdjCAGFqYOujb0VGa5pFmaPe6tZXvgrBacwJmobjspqDdRQ2lor9p9xAdJJH9M Mp/BRi6c XIgwCYhfcsUpNM8LFAFxuV3Jcb8KAXvDsiY7aP+uZn5TzODEbPu4NSKZHuWOlTZkWgK9/t+usQ7NJpb7KIu7KQjIjF+J6+wmgC9wljyY93Uc3JfvV+EmtD9uu+yKJqbPOtovr18B8YbgXKpiR6hlM4cDh5Gk+4ipYSJAieJYTBHvz4JNrOlaWkPaGvB+qmAfQT2KagSqlwu19BG3gQA3FkAHHZuXEKAL8LNLqebbgPhxfVErEMI5Eyp+73paw2C9w5BWYyNIwyBsbRELu97dL8M46YZAIEPNUyfcAKEE0YUqaeDRs+6qDMhZhKSc7krH8MyOdp74i6vhCFURuloTifxhM6a6/4djVttckF9Si7o9QckXjipQK2MNhRIV7PT/xJSDqSoR5HK3cxWC4EpgH3SXTwmdbhBcZ1qbMFieRIMSU4etyzPMVdD8Y06RTtJrZ2eOOxhQf8J6rfLNbzIK+vNfWwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng Acked-by: Andreas Larsson # sparc Acked-by: Alexander Gordeev --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 20 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..da4a7d175f69c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } /* @@ -124,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } /* @@ -142,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* _S390_TLB_H */ diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_32.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Mon Dec 30 09:07:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA3BBE77194 for ; Mon, 30 Dec 2024 09:11:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471596B00A9; Mon, 30 Dec 2024 04:11:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FA2A6B00AA; Mon, 30 Dec 2024 04:11:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 273D46B00AB; Mon, 30 Dec 2024 04:11:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 04F0B6B00A9 for ; Mon, 30 Dec 2024 04:11:37 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8B00FAF3DC for ; Mon, 30 Dec 2024 09:11:37 +0000 (UTC) X-FDA: 82951056384.05.4F2C11B Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf15.hostedemail.com (Postfix) with ESMTP id 2F1F0A0002 for ; Mon, 30 Dec 2024 09:10:12 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fpBMXRRG; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=bxBaoe5t2qRgSXU8GtfChAxkEKi0kVZTfECSNLpv/iYthWi1nfFdVtiI3W1kcWmxal/I7q TwLxXJ9ZSh7GbdhQtl1UFCpAPtKZr0gMGmjd+QwwVzMP7dDfcV7rXH3oLNSqSqGFRFzQUr NAmmAR48J4wQYp0oTw1ygdgMipxu1C8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549841; a=rsa-sha256; cv=none; b=YcNG2r+HBB8rPYivF/l5pKu8XGG4q8Ev6/hfERVTfLcNO4TydVLcItsp6ShIcRir4tjIVA x5ftXTdqIIvpq3O1QqfsntpIjia5/CRjifNqCm/iAu6F+8W2+OoVU9Q28/o9579rv1kswq y3vl4koAF9GQB/lmdRhGMgkxXsv1a8Y= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fpBMXRRG; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-216728b1836so101133285ad.0 for ; Mon, 30 Dec 2024 01:11:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549894; x=1736154694; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=fpBMXRRG/7jNM0k4u5BBBcVvGYA5W3a8genwlsojb9PE2gDSO/3Glf4Ynu4TofUD8p 0ayn0uHAHwjoxYTmeKN+Fdm58QWwfwP3e5r4sS27OuiqDx85ktZlt0GEjcnIvL4H6y4T vwaShG9JWH5CYhyRS+Z3c45Tbb+oJfjUH37c6cdy6/U7KJTb7QgxHyHj6d5JPr/4799p cZxLtZZ5gKlLBS582OWXZXDUp3snn4epoLAsfQk/GhYNzjkcYJzOxsmWQU6e3MWlwlmK mL4FzDlMKygSv+tRhxP0FNqx6rrl5lrLf1lUhcgjBfalj2bqDrfLgRZYO8bZo6h92RGh 3vog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549894; x=1736154694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=g917JPxMMTwfNS2Y3R1kN4wx4Fiy8y9yd3Udm6bi2bt6zS565dVYIIV+Zo6D0SC+C2 jkIQbhenWduTvl0YLj14qB9kiNJ59Snu1y/ip5I1SBKTPTk5wQ5HyVM+W9WkhM3kseCS F4qNNJjNUKk83eyrE8GMXHIAUm6gp7IbOalAY2QiztYQOGNGx/wtlbAzFSlDD0MT5cZr FlDmV35pnEhj/eveR3ZSMSonlfjH6yfpUxr0PBxQjVgoI0PYKetaCrrwGddCXWdJYQRi bBmB+KHYvKR5PDqnHO0V7MAG/EZ4y2+4bcMlQVQeQZ8HsFHO6ZSm0aeVR3ipUe+zI7S5 PzQw== X-Gm-Message-State: AOJu0Ywm52FIxg9VUNgwqJHceDikCL1aLLMD2w2Gzj4pWQ02NMGsp1wv /gxyG0UWeB50poB9V7I5bZUg1Kigluy/a3fZujSS1qLlFOH5XiZxDxX281vjmHQ= X-Gm-Gg: ASbGncv3NGUAX3DXWujPLvJrJtkm/cNmNpW7lJjOz95ATge3XfETUdtGh3gMd8+EQv0 XesX1xQAaabKEStAGsqU9qruJUowgCTCoGG25yfeKOmYN80CDzsRFFXH/EICZrfEJDfpNMqSF0/ KbMdjFnITenwk33q8qlDcHhswjSM5R7Nbrw9RlXz8Hu15PGLEzQAGH2qs5wMGNrTqFFAGAUFCpR LDgFPQfgnSZXbicjmZNliFJ7DBTcjrCLAGph/NZ3GMMSxLxI8ypCd7cKh0IIJw5EspHViTvhOcP jj2FnBodTVB2VC09uc2Pew== X-Google-Smtp-Source: AGHT+IF2zawjY8HUYTCaMJHzWmoJ2TNFStaqczmOTvWfDCo9S9fZfm/YNlaFMm14SUWbTnX0T2oRFg== X-Received: by 2002:a05:6a00:600c:b0:728:b601:86ee with SMTP id d2e1a72fcca58-72abde82a17mr50548930b3a.16.1735549894520; Mon, 30 Dec 2024 01:11:34 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 14/15] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Mon, 30 Dec 2024 17:07:49 +0800 Message-Id: <286e9777dd266dc610de20120fae453b84d3a868.1735549103.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 2F1F0A0002 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: be5yggp34a7yikc8hayzesuyepu784n1 X-HE-Tag: 1735549812-708933 X-HE-Meta: U2FsdGVkX1+WTRis4OYTxwEMWY21P1CCpCYSZB9IgTG/wVHUu+wx0qJzzWpDjMa3fKcDy9Kx4K3IsfgXEunoA1lk5icS17GuORGKnRH81lXpYZBSxlW3ATXcWBF/ymb6RutgDWtrugRYTV2U1luS1ieU+JdC2vzJ0DbGkMBND+Ma7qD/SBgOWiZfRn3jeVAzMrq8th8TR1GoZrhFYejfvKp1O3CPLDimrP5RzKHTFH4fd1phQk6kqADaD70HBL/gwP1ATLUzWY33o5P2CjTluel9IXkPLOte60PPBoEvemKXwRWftcON4yGLAdlxzeDin8km/NmykJkZAfmQLwlzVls8nw98ZvUUUXi9QuITFuVALxd+4KWNkUfjbZi4hkGmj1BKRc8s8DoaY/wbyxn3L4rfcwrW2xtJK6mYaTfsPtcFq1LtFC/emRx7sYErJSqu0U2nlc/ea6lIqyxx2sEcq8tNTfF04qzGT/eW/ypF72mhjLP/ugI4Yn9EbsxJ4betYtZXFb+COqwczgs2mqAIt9fAGe8EgiavlAhJNf3HTAZDuSem4egrYNkHe6VrUdyEMCnd78LGNkhdV9deVhUwuPFx2p0WkA4ygR2juJmZAHJ3fx2O4eXmcP+mLOmp1A+XYNADZhCzLM7P6bt70+n98kPm8N+FqkCbPBssIAzauHDf6+PP3OTSuCFYJyxe99REdpCIyISOf7LY+Ql44Fk7d+3+G9DLFX4+ScmwXakrU/0JgfrMdqQ65rnZ/H/vXbyWO1PsoMo/iVpAv9nZ+K9qRLTFHu1kpe/vlRE82toX0xDzbdC0R85eNZTNwv/s3TyS/HNY53h77BjWjXpEYRFpXsY7hRQMePE7P9MEDMMnl2ghBAvU9bs4WEtxxPdUqgtuIeXXYq2ZRAJsjIZd5wC6kGuwOe98ZNkHDJd9jq2jv5uZ7lrjC/9bRega8fwPYh5g8boypWpQS4HLYHqgybO 84H1Ai18 feqXCAmnu/19v1s/HrwfxUi2hHlrjxjiDHeDSlrlN4fKb87G0P2EkXkhh5IVJh2UTc8wFp76Pjk4Q1Zb9PpYDaUR8KQMaK8kFZ9jjlrJB3Y/xg+K+0u37VtmkxRct0y+bpoFTiWshOMfe2HBGphPAuz0/BrWO+lQ6TvMDlPvgibcw1TIBRCr5hWz4PQmwW0OMLXraRuPnoauxkjuL9Twi0tLmSEABWdPowQll8eKb9Tr3nS+5j0myaNxJLHKUZqO2XSIY49+BP8OHaUhgLDaC2/7oON/5TjesXRY0JhUWmY9temsw0AMKpbNp1jJTAXpRT6wlfDVgg4inOnfk+4nunKrHzJv2qUwN032tqAvk/7h/OghicJyUZihBOh7VlDRDE3xGn3IzzpaDJ0Rf1MPKqzCTvdhp2p5FpwOvDxD3Ba3Pn9N/oXrKDT4B6mq372mvdMwJHQOe1c1WkO7crvZj5YmWVbb5b6wcIyZY4+0lCwN2LKkYPp4l5M5TiKl8YMxoDZZ7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) { From patchwork Mon Dec 30 09:07:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13923141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1D15E77188 for ; Mon, 30 Dec 2024 09:11:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 819BF6B009C; Mon, 30 Dec 2024 04:11:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CA1A6B00AB; Mon, 30 Dec 2024 04:11:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61CE56B00AC; Mon, 30 Dec 2024 04:11:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 44A456B009C for ; Mon, 30 Dec 2024 04:11:50 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0725E1A12A4 for ; Mon, 30 Dec 2024 09:11:50 +0000 (UTC) X-FDA: 82951054704.17.9BFE97A Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf17.hostedemail.com (Postfix) with ESMTP id C5A2D40008 for ; Mon, 30 Dec 2024 09:11:12 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YkuFxYDn; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735549887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Uxx1VFA8X6F1Vml9+szJuo78n9B3sJITH2LTRfnVDm8=; b=4FN0OVJFB8inIKtR4X8jKiE5g9rfdlrZJnfUBTDJ83RM7RjwqPllWImGv8f2kw7bsgyEYa gMi5zJR1YRblRrHuDi8HHsZVWc3Mab2/E7H98KMm0DhIcf+fQBi0m2gpTOmGUV5NaUhL1n 2CK9A7w+RhTB19Kde+7XS/f+7yEiZXQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YkuFxYDn; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735549887; a=rsa-sha256; cv=none; b=aDUuGeC9CuQ0Tpmrqiv+D0NrH7wnn2PzkJqniS7XVBoxzeTzbYQPIMfojovyx/l7/ewitl LpVV8gbbaNit8wW7wlTF5H1JxK/ccCgO0yUTMSmYGtAbGdqs4D/gqO6iyYDRKCkcYT2Cxn Gp2TnqHs0+z+lYgJGo2OZaCa/jM6hio= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21636268e43so17207085ad.2 for ; Mon, 30 Dec 2024 01:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1735549907; x=1736154707; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Uxx1VFA8X6F1Vml9+szJuo78n9B3sJITH2LTRfnVDm8=; b=YkuFxYDn2hW10XzdLCWHTRFwpv0RwEOYX1QwHl9hwPwHFmBXpeyQ4t1KNBNmv53MSt SsotFRjlP+rL6sbYMe5ACm2YPRjVqpAMkTdXItOiCKlfGss0xUawnhLiStXeFBOohqYt zdEchGoe/jRxciwkzKX1wVbdkpJkJ7NuFVn9slHh2KxlqcdlW00Mept9aAVJ0ZQRSWaY 6a+MFE+BjcyE23iMigvQ8TSTXY6rnAAiTDSDdKp3SNJm9NDcB5Af/AcmDqVLoBB6DjhR eDr92j+SfQ1nCAlg0Mblv/65PrfRfjQynby9XiqDJJculyu+mdeRXt7+ARFdRyV1bXUt xNEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735549907; x=1736154707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Uxx1VFA8X6F1Vml9+szJuo78n9B3sJITH2LTRfnVDm8=; b=L8kA+xP1RR3vbjZ/BS1JVkRu82CVwf0mZHybH1datyvsTUNwecrToR1NYeudtotB6g UcfB/jws/S6j+HSwjMtSaatHiD+h6x5SNZOB+fhLiERkyjWmRcs1QhbExJic70OQk6J7 IibzB9qP6BMan2O4c0WeJfR7r7rDq1RKX56kV6kHsi3XUVVM88ogCkys2p8urDIY9uCI o7LhwDL9vHRyF8UPJaCi7UL+REv7/mMqnK/CC+TiLORvHTWsNKMIzGjQGF1QgbroSLb5 q5N7Dc+LHPBdQ/aD9vYRZvp0C4qKwtXx1x7jQQLL1679kMpq01nnk6uX9piGgW3ro9+R M/qA== X-Gm-Message-State: AOJu0YwanBQpSCwLEXh3r+2nM8+9jQjb/XD3zVQ38aMVElEXHumpTZlB TGAJozQJqo0eR3bYHpDYNMlh/BDjdso5NA6LGzi/aX7n6agStebJjm6sSkqEqfc= X-Gm-Gg: ASbGncvv9nv+rAKCbpApEkXxyQ8N7psxRCmXTRv0zn1DcFzArKhV3mD9J0hODP5QzYM JFjL2VxBxzQk2PyLYP4qmMVg8lYsXFs1PzhTCmy/ADpGUBgz6QPpKwO+1PlyTgdmTdZIl5yMQYD b8W5JEZlS/xVmcsl5bRCOdtawLFk++FIopPQpAzwfxX30KBD/3erBzNFj8L60s/i/0OcxLpUW4g nC1XzSwhfMUk5guQ2gULU2Qq6dW+eFlF5vFHPDEYqpy34zQqrD2oMK2wkVX+xQqD4fPiXaXYLnZ 3Q66X7FEIw6R7QN2QDqaHA== X-Google-Smtp-Source: AGHT+IH6enLQBRxhPiyC+b4rbFoH6ZPXIb+9j2XLkfNoZdfNkkGtY/WgMqtF7HfldrPWQopcc9iyNw== X-Received: by 2002:a05:6a20:4308:b0:1e0:c8d9:3382 with SMTP id adf61e73a8af0-1e5e0847084mr55644198637.45.1735549906952; Mon, 30 Dec 2024 01:11:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842aba72f7csm17057841a12.4.2024.12.30.01.11.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:11:46 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v4 15/15] mm: pgtable: introduce generic pagetable_dtor_free() Date: Mon, 30 Dec 2024 17:07:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: C5A2D40008 X-Stat-Signature: rxxn3zapk3w5utxk1xes4drx6skhfp4k X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735549872-299817 X-HE-Meta: U2FsdGVkX19HPvjQmdA5E/5cEXg6au+kcu0li9ozrNxbv6Y0aK7itonr9JhnyG3eY6te4pJeVdPv5YSimyOxOwtiSJmEStrDYi45/rrzdlAdDiN+9epoAqLny1p/G6q4ZPVyDy3Y7wfr6sHUAtF2bF08X8KJrIy8WTRABn9ZTihvybgZx1ASTBU6MwgZlnYdJxTDn7MMONf+Es85BaJF+hSOwyx7Y/JI2aLmXzX68Qn+bmitnZAMrvn939UJsD40NvfCm4GRA+FOso28Ym8lt7tGVtLmVsuPY4TAiY8MbibDvzYl3aqY25aR8WGi57FK/tGaGziUWx9+x3KOLFrjwMzJdGFFrH0P8zWr2pyGZtTl3atPjHTpMJ78XH63m1Xp/DgxztU5dbroFspR6IRLMXD6eepZ1PPqRvRTDrFwePqu27llY1xWwv13eV2VPjwfR+1Yd7ShaSnXwG5ew9WqNKbaU6L+fcQ/uvwpQD0BMrC2pBOu+dAlED7UlvUbVttV1keeh1kYtzfDIYxpQeL+eHqyK2pmVwrzdNB/36CojlFvt5+fkeqSwJ3dhJZxe+ATsfB5r+OEB4Q6/8cvuXKu9qrx27JVG5pRLVlQ8z9cmL5zZKe8q+619/gRCJ71qXW7Lg1DIk4vU6ZLgkdkQu66Gd92mxZZbStApQ5mv1Zj8+2IEXK3czdhqk6n3WCgWrlRDqX2428tVoR+k086glnKc7as3cpVTl2Aq/iZHPvJWtdWfC9mC4asmfItKY+Qjql/8Pmb37w0XsAsUbWMaPyRz8SIC2/HDTH68Es9h0PdK2lvDkpWBeyhoHRPn+tvgKcLnPIPz5WKM2PzPdrlqFFN1SGbRld6PUsvLByIpNDKyyP3PffPiBWBtzexojGAICWivXpqwj0CzKcs231PDwClrkCqRybU7j6LxTVnqU45nRhwtdLYQxqcQnJuZin8Qnp8rbbe5iqPxUGA26zQ7r5 Wt9yasBE iw1aX8YOWvZSKoFhWFTqF+Q9Wfot3ke+OuLlQdx15agBSYXS1K7La4ke5MAZ7OH6k5SuXE46pMLgI/pvZzlwKtuIAe76pdSmSSY7mdFxS3Q2s4zXJVWX1Ek7wBJLxvuBDJ1zeVlccPRuypJY0cgNNeDBDPVCB3ZbiBzhsFHso8TZAHRU0vnn1MOIT8RcpjEz7zUmV96nBI4+LRORiGO7frBB0ztcqg0DruNw3vOxij9BAcGirm1lGdJuWqhOjqS9ri6FXqVrKJiMuZeliqKW4mZlR+ajPHzs8Sg+4ZxMC+DxZ91Gjv5PVW7hKDf15BkDKrJn8JMyL2atrohL/xMvebPcZwM+ZMzLyQPpNf/aEhWTtmrd1h5IaDtLxsHeB0ekwl5hdEmO9UXK+wM8qcraSTy8J++QAdn8t8eMv0I0Z3KMhYyOCnpqeMJeMvF5TsbjZ+GxM6HYTBK2E08TYKPv1+oZgCP4Mt+7+x1uORrGkdI1QkDk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Acked-by: Alexander Gordeev --- arch/s390/mm/pgalloc.c | 6 ------ include/asm-generic/pgalloc.h | 12 ++++-------- include/asm-generic/tlb.h | 3 +-- include/linux/mm.h | 6 ++++++ 4 files changed, 11 insertions(+), 16 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..a4e7619020931 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,12 +180,6 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..e3977ddca15e4 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,8 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } @@ -153,8 +152,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif @@ -202,8 +200,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifndef __HAVE_ARCH_PUD_FREE @@ -248,8 +245,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..a96d4b440f3da 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -213,8 +213,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = (struct ptdesc *)table; - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..94078c488e904 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,12 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_dtor_free(struct ptdesc *ptdesc) +{ + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc);