From patchwork Mon Dec 23 09:40:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46F0AE7718E for ; Mon, 23 Dec 2024 09:43:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEB8F6B0089; Mon, 23 Dec 2024 04:43:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B9B686B008A; Mon, 23 Dec 2024 04:43:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3CB96B008C; Mon, 23 Dec 2024 04:43:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8569B6B0089 for ; Mon, 23 Dec 2024 04:43:21 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EF84C1A12FE for ; Mon, 23 Dec 2024 09:43:20 +0000 (UTC) X-FDA: 82925735046.16.B096414 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf20.hostedemail.com (Postfix) with ESMTP id C13F31C0011 for ; Mon, 23 Dec 2024 09:42:39 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=aftQnd9y; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734946963; a=rsa-sha256; cv=none; b=OLkxKyxCN4K/SQvv18OzoOQlyEcoQpyUWRMiR9+pspn5jH/3I8ObFDPKOe1Yi6HJbslYxk Sda0Xs60huQa6u6FU9hTHpk/tNyRfAFyBmBuL/xCgCaSZtSe/XKfjfFoRhtoA9ICKibTrU gYiHJ6beyi+y5rQgZa7kDoG23mvaPlg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=aftQnd9y; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734946963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=hLzEMK3Ygz5XKspz3vzcNMDpRIxLGcxPW4pvh+FNmWvwTebvICczCHrpIG5zHYEP2i29/G UBla1QOpW3VvCNlNwTHUn3GnhzsNkspcUN7mKCGf/C/nfpz++GVh9QL1sEQRX7wj3GGW40 kGZq8eI6I8xxx50Y/Fj39AYnHT0d2pY= Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7fcfb7db9bfso2727466a12.1 for ; Mon, 23 Dec 2024 01:43:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734946997; x=1735551797; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=aftQnd9yIyNi4bUqNdGkW3xtWWMl511IdBwRf1kA1GOc8DLtC1FigDlPRaCC3vsDbV qCUUwAOyWhpuI3JnQcz7g7N2P7hZW7+BL8OZ/nuSa4j3mSCzc43ZC8vB+ba/CDr2zRvS XhzEsVazCdwHJ+ZFm5upN/6HO5ZG7PoiKTTwuqzQf3Riip6nuyXGnNrVjpBCSxc2kMYo kj3VVAEatzR6oCKVK0ngjdetgvafOiqUpbdP+fQ+dZKe2fXeKPohEScRk2GSrTLH7TBR 0rloSsNN67pbkttlzDWM97z16Gx39HvoMScgAXW42EFk6CqrL/H0cNwYeIXkrHIrYItT 1aWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734946997; x=1735551797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SyzxI6H4ZAISLyWZ1lhQ49SW3wSD1SDR6GIh5AHA3wQ=; b=TeEGQgRSvTmC4nhWYRAuPc2oUTEP4xOn9m//3cOZ4qPfeTIlMUzCR8QEbojEhdugjC 2dcWqscrtF4I5VJrILMHyFdrdrNcgNkyREuI0tIg6IjrLEoy76XFwgzPCVbZhpE9+6fM OxYzeJOZRI9MqAmJHu4kMX9MCNLx/RBm5fbWb0KAZUXX/avllotP/zdOiNV8jAkOlhBE 7jq2ieorgVeWwfZUdrWx9FpgvB16gQMRTHPmCozIwQHFecjTMT6Xwche+rDmo1xCtzrn xUThXA6+wE4pkG34dUF63HfMlZLQwTLSSymSe9RpMrDpOpSlU3kCnt9aW4WHhqisTj7o T9Vw== X-Gm-Message-State: AOJu0Yw/Kz6Ol3rhUxK4Ktg2KSa1tOlgWzZ9PeFZ9XvZlaq+LhPvxgA3 JHDKj7Wo1X8PG5xspv8Lpi7tqF9iXmd22W1eGw0TW7ZxfBrzfJ+tO3ORLS1aUj4= X-Gm-Gg: ASbGnct4pjVxjnhI+AkQ0SpVKpxuAjRr62pljpBswQbY2PBleVTTgo2yVqJhfNIknjT a5Mp4DaDSaLMSoVJW8Zfs98RZCC+cXLMC1UlPLzngoxVjXGC5yec2CYwnj2HSNO4OwcaXePFRCI 487FluLFxvcAuSa7dOJny6I/7HL4taMwe2x/hG1MNBdU24JyXitp1xroGdLEq84u2JZZIh/yrGP z1fhokWsdAPYS9oyuvAl0AR75Km3k/Ql80rhlCJ9K7358veub5ErwHno1kzoBc5QsZEOfBoQKuA xE/rpW4ZrT3QH77lyH0MlA== X-Google-Smtp-Source: AGHT+IETS5CqVMOZwuklYBKyw2d6q7hBKdfBjuJ9ZQ+tZcvMnTE5pZk49SgpvcM01OaAXbZV7hq4vg== X-Received: by 2002:a05:6a21:6da9:b0:1db:e338:ab0a with SMTP id adf61e73a8af0-1e5e0447f5bmr22664720637.8.1734946996090; Mon, 23 Dec 2024 01:43:16 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:15 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 01/17] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Mon, 23 Dec 2024 17:40:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: C13F31C0011 X-Stat-Signature: f8aobcanao3b5pxx8mhjfkps6xr1onxk X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734946959-590350 X-HE-Meta: U2FsdGVkX19+g3D1wiNEUm6KJybOHFucjuEFysvlv0PH5u+qapQf14Sl0yph6dmwDhf0oBB7adygkiH8CHnLM3EjYzMZzYBX5k6DmRwfpf9R3s0zHpVUtj8UrdhVAt0T4hplvkHbegAQGgE0AhLRAo+dbygJBq7YltfjBs4FU9Q600e1CJz2eQDWGdkrlyGcSgU0q9VzusEseHLf4+xgix7cYz4LGuevokGxNFueya43DcTKldoj5ZnmUOicZaHE65+0eti2TWU4aCmFB0RBavqN4XpD8t/pQy/6o0dVQKKnXfkOmbkWLBm31X4wrVYFZuxRqlW/VbDk+uCzz8yFsbKIKG7/dfZRltrCWZcRO5gZGGSqELTgfHRnti+hpb9YAT4RmSmZHiXKy7tEwXdak4X6hVbyinWAwjQ7Q/Elwa0GQbvMOTmIIwI1PaJK1fnIu0dbpk4ixlUGxMYQrgvDH4A4553tDCtEKLdkqemTzuwwXZIuPGjiUY/+FDoXc3IBmu0g//halVp3Om7GJWgHwJdXrDS/RUxTo9kMlT16mhuiAXAI32Hf506WfWHJDjygCq32B4MUppzelxzTXElfld6joJ7cCWT9Pi+qP5GtE2V0i7Bt1pJkA/8o3ApzRp63km3rUd0oG3ttGNsvwBagcu7vH64a+0TN2H+Y4i9kDVVwZw757l4h9N5RH8+s4J928ymMiu9MgUFjYwT+y07qynwmiD+JadXo4qvsSR9upOarZ4Rg350pJs/XWeM372qJQq2h5ginN6f8nROZCMBUE8ZjUi6uUWE2HJZ+VIO+r0CzBcvwBD/zySntVd/JJ1+U/0mEFXCOTTJyFAn2a1oA3pF29Y6ZcE7kBJ9FYTHbviQZ9T5VoBx1cCodx2EnRjlYyCETC2stwwDbnz+Y6FP1qqrJRtIUKmLnIxsGcU0vqaDV2GlAONYabp47AoBtL+ua52WjoPv4s+epFcew+uB QqfurSNN jPEdH8m+K3lwWXCE215v2QbDTHd9W7lXxDxS5oZ31W9mPoBE/2QEOLiaWCwE9NhR3fHH3dNxFSgpai4kDwJhHxl5zwN39sEtIShSD2GO8V5rI1EfO5YYM3Ojx//ROIgOlTSFbf0GmLWOCsjn1iiyKNLle/pEZ2ZFHBy5S/IJdRNdyEcvWi8MFndZ1ceHW+Ej4ktMYeR4AQnPSo2RjiU55hQfhADLA2Y0rw4fknS9Z7LIYp5DzeF/3K+YN7Y4eusvoyd7meNfiQkJO6F0peRB3y2NHsD+ON9kbGU/uaFfwwIT7W3LbKAOEalsYT6xC6t4eGqholJ18VPzgcGqOoCWZQ1ZCSTkF9fyDeGQoDn6gi6nrFOTX0am5FET+BEdk8tiN3sFM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 23 09:40:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40B94E7718B for ; Mon, 23 Dec 2024 09:43:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB0076B008C; Mon, 23 Dec 2024 04:43:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5F606B0092; Mon, 23 Dec 2024 04:43:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD91E6B0093; Mon, 23 Dec 2024 04:43:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8DB4A6B008C for ; Mon, 23 Dec 2024 04:43:32 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0C867C1577 for ; Mon, 23 Dec 2024 09:43:32 +0000 (UTC) X-FDA: 82925735130.24.E89B71E Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf11.hostedemail.com (Postfix) with ESMTP id 84E3040017 for ; Mon, 23 Dec 2024 09:42:55 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PO6+u1LE; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734946974; a=rsa-sha256; cv=none; b=O16l7vYVeT2IW7t6yEAaZQX08AO/Ay4o7GeOUSc62FloKTwCkWZORm3/juBTDTv/XlESvc kpW/bkEqwaliDCdXInNMtSkVYyagksJkx4IOu5Mu1HRznuhhoQbEM7JlUlMADIoybMX3iu bvAQB2YQJXbSj3CXm/m4lTWyNxuUAQE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=PO6+u1LE; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734946974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B/tJ15+vmer/NbqXoUznjybhQp/MgC6GOdT/GuQUXpM=; b=xXUUTssef09jGiFE0wVitdOKcQuzLHs9L8euA0FVwKR+bMxdidHAroHiU0oQcB8ImvdEsC c6R0WzQFjWZqwy4Gds8nlbQJ2MNI52ggooLSqWB6AtD4Iu3Q6KYbVpLab9/al3Io8wRBLN QR5El9h2gDkVD67jY2nexyUWQHz6n9g= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-725c86bbae7so3476783b3a.3 for ; Mon, 23 Dec 2024 01:43:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947009; x=1735551809; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B/tJ15+vmer/NbqXoUznjybhQp/MgC6GOdT/GuQUXpM=; b=PO6+u1LECPhz/UXO0OjVXHaT8dbK/81S8ZJyR1ptrMY3u5hoIAgA9awTSvS5EsH3c1 BgAwIGWSXHSnCKbzm34hko3v3JNSMtVKeNdq6BjAwihc0OSrzKY3V/6biUVxKZL9/vSg BNkSqFRUkdOgvLOwL1GaKvO+k26svz1YDN9FSCCeZ3UyyfFWqpTBRFiXjX4m2+nehPEb a5o5F8vcUHekp8kazqzp80k+eFrxBIXhp4+6qXne6SHblZi77+78eXeEZJuC26Tqwuls tRTSX0JZuOChv7GIUsSVMrlzLGkkcMYav+o91yxr5v7TclDaxElS9hK0V5GV5PtpdFv+ e2Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947009; x=1735551809; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B/tJ15+vmer/NbqXoUznjybhQp/MgC6GOdT/GuQUXpM=; b=KUq8ESvKeiDZpOQQebubjENFHAF88U4p9rWnjUh8/gnAeYCSde/hd4maZ/Fe1yDZDE ILn0rNo1HadGIMgFDG+pb5fCjZCB2Zl1G6wxF1prvF9jJxIcELRinI76IFcQUKZ/52jF jwbc0rVo+0UGiT2pgolK+zkmBo/fiUjzN2Va18CMTAwv0uaTOndERKNN1iUwkdbUTinO Ajwf0QDs9G59E1r0oCgubxs81KWxNJ1EpQ79aeTZvxRa6yrwmS01a5/TQ1rTgokR854h saVD5YuRjblTNkEpHwhIlN5oKVpZGFqn/Kh64L0mo/jV6MruD3YGKPyKalZgb1akfJc3 fqDQ== X-Gm-Message-State: AOJu0YxNraIPV3PLO14Qx19fDY0ZJW+t9ytSuUB82S02jnpsNMg3mZC4 egIGWldPKorGY3J638u+SVDXD3hJNj9XNm8cZAnWuDBI+1sEt1LxvnLkX1tV2SA= X-Gm-Gg: ASbGncsW8zhaEjp86oABOD9TOy+dmfFhzDmfpSQQxa+w5GzCjTJOFsnE6QxByc+6h6e KKIq7XIC9Nu8liolIGliROnEVZ/IRQ938boLokR3nvC1jGL5loEDP4HtmRQSSvge7nKvY3zzZ88 zAQ2MbuHOxdFPp2Sd0bWMWPMvKoiyDoeSOaF418kyCtRofRRJV9rys+1goHq+6Grb9JsKedns1d 8CaKAlfsXKOUqmWhpzTtLhB1ji77L1LJE5rN77wJhqH48IKfRAp2Jw4zsve7ZRg+IS8aPRHGrfx 0pHJ0jOgKlO9awbgvnEiEQ== X-Google-Smtp-Source: AGHT+IGn0WgNlH4n6jFU+xV15Lu8l6dnMV2FFWuW4zvXTMVJTCCaTT+ZxewqFKpQcJgKvmYU0RJrVw== X-Received: by 2002:a05:6a00:35ca:b0:724:f86e:e3d9 with SMTP id d2e1a72fcca58-72abdecbdb4mr16058641b3a.14.1734947008718; Mon, 23 Dec 2024 01:43:28 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:28 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 02/17] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Mon, 23 Dec 2024 17:40:48 +0800 Message-Id: <84ddf857508b98a195a790bc6ff6ab8849b44633.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 84E3040017 X-Stat-Signature: 5cmfu4znopzqc66kzjcqzhexut68henx X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734946975-926064 X-HE-Meta: U2FsdGVkX19vBN4jTl56mEQxmHUDG/wfrWw86h8of/CU1rhhJ15JTugM8Rk8dvkug1zeTRaPMcfwZh49o+gzXMM9xAh0Hb3Y1XpYIXQhDR3TgZRLO64oHjvGS/OpfCFgikHiFwSjfcHBpXEQ5zg2FD7VExXhvgRPT6Xy1qlwMybQAVfll//tnJOJ1JEVOZPZp56ajxjFD3zZJgQ/4T+puImlQaz6MZpUSj9N0zWNELxjoJqE6LUqLAvOp23m7Jj8YTkTTYmd3OtYvC2yPKTVLXRF+m8pA8VLgNIbD3HtnYPWGzqRTO/SY1xJ21EI4B+sC3gwlIT6SnJQ/d5QUCHAJqIZcMur8hKBuJYXLFC0cx1Woi5nyZvMBbqf3DBCzih1XlP+dm4d8Hs0GHYyHlfkMR0so6/eq1ZYlAsuUAHui55N+4hUVWoOW4VI9cVd3DaMp9w4WcdjyzwaiPS2CFS6IJTp6Dfc8E+YXSuyFUnqr7spsVuuzWb8yuIAsdSo52odriTwWnwN5J8tn4ae8zpbCrp3dHxb9sq7qtvSoLKKR5ESlbcOI77t7/ts8WZDyjvgbsAVRlAjP4N/a+jX5FxAv8Fg/m3iTpw7azmJMa9JNh8WAwt1Shdsmavpb8+coi4NAaAyBZRwL+d+xVeVqeCBblmwE2ZCA6tuk9gLCfQgxdF6VHFgzN598zN/aMJ9isM34YKdD0HhO1M92E2tqSlZbQNePBeN1MtsEgewDNzJ9iVKkW1LFPxaRTazrh8hVXi2csiYXji0MbflwYKv/WHaIsbZYPiZzs9p+rIvvvJguuEb3+00ld6NmCey0yhZpW0cqRFQuD1FRdazF/Wutcm7G44onByJOvbtPdZTOdFI7xoa/UXWXOVMJXHcCK8ENvpwjCljoxkDAexiW//GY8EsaByiRpaGoFQ+HQEdsA/aXSdIR6BJjB8TZVTUkz8kV0+xc4caq7s+XMtCuFn+HNk I7WROHec BJXGuzeNq/xHcxj/lplGpJ7mfn3/zAhB1LQHSdIyIZ8Zgx5X+wxMesXbzOyD/+N1dDjGiCSPKmseuLaOI3lxc8kkizoPktE0xfb/fO8+/bdxVK+ilN/RvURKcxBz+FKkNrK5LcbTxmy4X7vnUWG5VjMsIYEPaXssxjeu/IW12W+0QBPxUU2awOCnivT5adz7j97T2iTEiLCNkArnOlziIcginjC6MsDXKBXivupYaPw8yfZitqS7EaUOsYNpeQGTmBeWTEqzZUK1yVjCimpGXrXS2TU27j120kHFS8RdsoMlCZkq2DWClnUcDILR/P1FYft1Id/MHk6d/ecqH9C6ulCoA0UT+4YbeF0CsaNuGExaFMSaHgvZ3ipHdhnSNgIuGVybmP3lZbX11RqY2CjA+Hci49ZGR/ZAz6/YNxsbwhU2SFUO5soiUPL5PV3h9DPj1fXhmVPD7+rv9V4Z06RKQ2PfgAobLBev5rzLfyI9AEcy8y7Zk9HTx5EtZ0a6d9VS5rfit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp = GFP_PGTABLE_USER; - return NULL; + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) From patchwork Mon Dec 23 09:40:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4A83E7718B for ; Mon, 23 Dec 2024 09:43:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BF4A6B0093; Mon, 23 Dec 2024 04:43:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 46F6D6B0095; Mon, 23 Dec 2024 04:43:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 312026B0096; Mon, 23 Dec 2024 04:43:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12A106B0093 for ; Mon, 23 Dec 2024 04:43:45 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 86FDB80C5E for ; Mon, 23 Dec 2024 09:43:44 +0000 (UTC) X-FDA: 82925736432.29.9BDBDFB Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf02.hostedemail.com (Postfix) with ESMTP id 45C008000E for ; Mon, 23 Dec 2024 09:42:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gMBZ7jP7; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734946994; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=nNP61y0Ujn6NSy9418DvAPU0iqWgc7FA5CXmQY+NH74NMVzxpSSgHpu9n8hgjCrmLgmqWR oRVw5h55bGbyP3lcKjfWTxwgIjOUtFrpPlcLvdPmgvRXSksBdBlYu4AOgs7gTsZK6tzGZo 4xED/FdDQp17KL98wnZdG0WnmvOHzQY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gMBZ7jP7; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734946994; a=rsa-sha256; cv=none; b=oVrOGqliiSIo/NM8nQOgD/FepRCDrwnAD9O2EYnlKh2jCxHLQwYRy5szyJW4dZM+JAAKxm L1yrGvc+3rujTXwLaMLg/BiCywMO/kyjhPjhIjwdZFzQSsw/nKFIGoaXLPHDTtGFSxew6t 53mdwRG5ZnYcWBw1tB7eLsYG34eHph0= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-728f28744c5so3822247b3a.1 for ; Mon, 23 Dec 2024 01:43:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947021; x=1735551821; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=gMBZ7jP7iv4hSzqhk2ML5MI5kE/yFXTyXi+9f3Sgy5Jh/shps/NxOO6+vFP/lTItIh eikoXO/IRIkZ+C/iFYVwdIXXwQmsIIPOvrmSP6rtx6ZyjexP9YEtNkd3A2q2CrBMF/Aj wE/BW7m/IsuQczz636XXnKth3qYsDmZZ2Ke4tbC6qNhTZMzmEP6m0ULUVm36ak9gThQb 3U0pW6nrFsqW28cT4Udp6fmIWq7cq3Kkt0Kz7TrRCKL3Ewugo3c3kH2XuwGVBJiHbSd1 apfRHK0gOrApOcImWYfAOzBGICYFTeQaqY2CgtmjhK+U+gK/Mxn5za5cMKznq3YXSwzY Hbkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947021; x=1735551821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=BCSO3qKGZxvds6iNjtg4XzlzZx5D3Lg/ERtETGZl5rTkS36FNV+dxoR8in75Amudek RLUFTsOVt9LIn5LREX3E3gdBHm6LJ2EXWb54aGG6tLLGDIOoaLdbNdCtGQndKYt+ZFGu JIW1JlS78tpYF1dmVu1vpkvDwnnXhT+Zs7/Rju7Vkt51SUiL0peTwkFC+2NxVkznzdE3 9NY/pai8ajgz6SVmVnMqFIC6XbciViX6K2TJtahy54qsh/HIVjnDipJwTdbIZUp8vFSE RJUOIsYQcv6AbTgPY+e49YDbvtwNVho98k31miz4/qimMLyu8+8o1OztPUdG1Gi0Vtzp 1qsA== X-Gm-Message-State: AOJu0Yz9+MKLCDcN7U6gQj3ATJZvLWKNf9r0HEXCtTDFletP3QSOPSF3 UJxZwnRBofv1knLDYbe4CeFBeuWDgdjvb+3lFCLmgUYg70xjoBDQ1ObpNbRjYPA= X-Gm-Gg: ASbGncuFq2BAk4PMP4eeIDHQrxsqv8Yv/elywOcOeTnSRR8aMQUX48kxdV23AoOWaUG JC4leU1wN63oi9eL0LkMR/NHreyIZ+aYYdrVcbUZ/sKZKOtKKPKoBJESeT+Cwr2PsnTTJpJINBL fmxKXekUrVyqvcC6d4yIadgo5Pz/2MNl2H15uDUPAb1ysOFt5xsOCf4zhRDSX90Ck+0oiT5LEtf WfQITeICIOBjOwOSGdJnf9uYZWG1japq3/eHAFyAP4fqQEvjflQT65VopGJSzOGljDsxXeuBS25 MQ9XCeFIMfQkQcmyk0lrFg== X-Google-Smtp-Source: AGHT+IHP/sWNqLjbiRRokya3RRzP3A7lT6GiLvDE2E3HVCuxopdI/+yKYyLcU7/i5H4uYS+gGEKb/A== X-Received: by 2002:a05:6a21:9017:b0:1e5:ddac:1eff with SMTP id adf61e73a8af0-1e5e04a0c7cmr19747203637.20.1734947021430; Mon, 23 Dec 2024 01:43:41 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 03/17] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Mon, 23 Dec 2024 17:40:49 +0800 Message-Id: <4c4bcc1aa565c6252183553aecd5e5cbd1a0f6ea.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 45C008000E X-Rspamd-Server: rspam12 X-Stat-Signature: sbuigff56nfduuuei8its1fp5q84jq6f X-Rspam-User: X-HE-Tag: 1734946950-614877 X-HE-Meta: U2FsdGVkX19OaQ74OPQqlcdAq8221ItwkMyfGFj6l+rc2Fe4j2rD8+BHui89TzIRVyYQf2VnlNDPVW2z4uaV6DA0W70qCRfXOBYJ4TTAjz/HZKoxtU132YCcdMLBjnGtPqVVwhzkMFxnWwNXTY5bOfu3UfUFR/TTZLmnlSCKEOPu5hgLavRo17asyKD9F65FdTyYV2RrYVdb50Zs/Z7SRrz/Q9lGMDqalGwrYqLB/hs4XsYriUZzSC/2C7cIRDQCwaGh5avHcUPR0IUJ1jnSiG7JLcKen3mtwAJWPSaWWRJ51kifGOAYSBq+XtCO/gXzo6i2wKdRORIWuNhnb3ekS+wBRZ5vpzc6H5wU8S9U51nZqbPP4bXr2DepI9DSzHJbKPKF3jkaBahssJSZ2AV9JMgihVoGz608SBy5oiVTNkwBK10dfw0EXRQ6D5l3p1wmYKfpkirYKmr5mgt5VG2632hwcJNls4SUk7nfPkoe12qYvK3EQAwcirs7eDIwXZtK9QcFs503MSSRtm6mkArUveLy4BO2Zl2ZS4dEbPP84j0DRH7V/IiJu3WP3CooM7ABVbsLCd9wa0YeNqKClYmCapEOHUxgyAuXSP4RmgC/I4V4lmtKboVFy6e9mAH0qwbHJ7DDH8aJ1S1qFCZsXM9KC0cRkEGz7tWHZc2ITRBVS+Q1BoJCs9/NwTQC/p7/tAM7L1gJoK9Qc0vdNWcaRlXdrTpzHJga3WSCq4CZIBo6FCOG6WY/mtCGr+D8k2VYr8o0v0WfMo4GWJQZOmGj0ZLLqghim/vbYiSnbADRZpDylLJ2gGoE7vJ1eCjOGOtTvbjpPykwBSIvhXOqjdN+xNwzxLgXRTvdAq1ByE/53tDoe3pH64Uvoz9WpNraLo6UwXWB/5vnHg1qEBg6W66QvBqAeImyKIMxxLuP9MTPI85O1NMr7gJog0jlAgxJ42XU8V7/2QtXI5TEsZitbDMQY+W Oug8tykd QJCykJUOTQjHLnomUNwhdW47/r24cjd8SDHwsf3Vc4HT8P4zbTCzGoFpKRUfM1OlYeKdjW/RQnHx/shvmrVJllDedVEozWFwK23JOjl369V9PXUqjgIl7SwsqTc/TgpIvARFc4BycwPlxr6ITq6C/IYoRHrcUaAbdTu8Pe70x7P5OCDIynMqHa62VDS+xaIdh48LtQ8kZLCObZ8C09Vh3MwPB1mmw+HC/HYNUN8tOYE4Kgra4F9xlIyFNIKIQu8f/tcLn7BoAILt/NWKVUH7Qbft7WVzWkUq/zpu/MSO8/7eR4Wbn2UoZgZ7jlxW5QJ5Z8y/R9we2WGn2j5T25I/8sIHoybw3Gq/1Z+I0egMDHliBv/EVDpHbUfHzRdamZgYv0RxytszzqLopk+/3EdYtetSOEP8Ozk8NuvF1Wiv7DypB+34+PY708bUMcyabvu475eKCCx8sAhNLFF6Xjozwq5f5n8pkvQqA/97yjldKWMB7OGDJLyv1nGockifyYEe6n4t6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000323, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { From patchwork Mon Dec 23 09:40:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 224D0E7718B for ; Mon, 23 Dec 2024 09:43:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8CE16B0096; Mon, 23 Dec 2024 04:43:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3BFC6B0098; Mon, 23 Dec 2024 04:43:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B69C6B0099; Mon, 23 Dec 2024 04:43:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6A5FE6B0096 for ; Mon, 23 Dec 2024 04:43:57 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 252AE46586 for ; Mon, 23 Dec 2024 09:43:57 +0000 (UTC) X-FDA: 82925736180.26.A1CF135 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf21.hostedemail.com (Postfix) with ESMTP id DB8241C000D for ; Mon, 23 Dec 2024 09:42:42 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KJHfofVk; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=r7lTdCgVXCKsD+X7046fjFwB8eaIffnfaYCxnuN+UqkfzZ7QqOHMilveYkASHUNiEIXMuw rReCvlHbHMzKZewT0tLk7Mj2l/J/VZbOEkYU+nxILP7Q91uiArBsRU/OdFA6yK/2rRp3mN 0ghWNobq9UnCmf7Favvu3KcAcjV/b50= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KJHfofVk; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947006; a=rsa-sha256; cv=none; b=lUc56VX/eUU8O2sBp2+FuILPn9BfVhE9X1cAEW9SYTi4MRcGtjAXfx3dPLaJhBg0VqgMNr RZEAC92S8FXbzD2DHR7Hp7vleaFvZ+qKnQ7ff8I1e1wUycvBkZ25mhNJhA42yKmlNirwpJ pZa8MVQk/ofQiFfY2e4voPFQ/+dVZy8= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-7242f559a9fso4801211b3a.1 for ; Mon, 23 Dec 2024 01:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947034; x=1735551834; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=KJHfofVkbxghUIAL7pfF0EHoTGLTsMGOp3xTwrkaTFwa35ODDlzNUhtUpVjNkVjxCV URCxayvBaQRtdKD59gWPmP1ZiClIe3/DwPd0nhhYHVjhymnueI8PUtvkk55HZuPAPEWU clEWIpQ11CdZv6GhCa9Scn9SN9KWvX438i+vBDhVV/3uYX5QZxqM+bb4zPrd8RirM01U C9piBDkxfPdUadalA1pVFAjfySKQTSl2zImcXy7NMqXBZQBkeIZLENg1j7FI0vTjnv+l c8RoJ0zRj36e7w3kDRvlu6heqSf957N8o+Px+8Y1haZGz667bgGaO8ebuIPWi4KTgpaV 6p9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947034; x=1735551834; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t6vBY23hDAeU19z0pXyQ9vuAeZavJyfGUN8npoqwlxU=; b=W9EzlTxuJgH5VB/ZERn4oWn85MdZXQBApLzFt+WSNcShv6flNU5QrnJ64hvAQt4gJd LP9MwhxJiHlpUjjUFZabCEoUvrWTSgg+gGXlJFhoRVWiKqIcD528BGqH0YXf18Vlm3wg HNgVMoskLsL9IRNoudHGRYgwoTHJeHSchs52dWqqBg4d7Qc1o9yf9QzJa/PmrHdNeFM5 05ms8046YMtTIHGxxrW6DcjG8byrdpnia6KRYhzkv4LQ8gMvW9kXQkDSFzXf96j1f6UQ Wcmv2/CRLho8+4y12ReDxngnO8oLfUx8NRipx6i7DJx9WoauLh9BHo5KowaCiz+vnufl mCbg== X-Gm-Message-State: AOJu0Ywm+P2Y2lGfE1c5QBH+zKwq9F2xNspRDvtRYN/Y4LC1p0BZQPdk iTS5HiHZbu87EMy80afJ1ujlSmTH2ANFMAYlNVFYWtOMaAomHYMxQ2ycr5GdbYE= X-Gm-Gg: ASbGncvOeohtg8oWteKg4CtsbdXVPsyQpLIqBdlu2aN/z16Y+qrYKDE4u7RdFlCJJOF oss7fNvIWN8btffSvzxtybBE+GQA1TlKPB+rabx/l8cCFbpTkCfdNufj6qGJehPMey+P/l/p1EN 2gFdjuAUv6vbXEi0ClDbX3GiVDEq2B3VhLpDUbxCphGw3qODAPp+bnyifzLuKJLfdX4qLGf2NNI C7cvGJXFq1Yd+qGGCBkCU3lwnoGoFeExEDBAH7x0DyVzC8IlggzJTBrvX/qcW0UN91yZhDNc46i 02gGkngYqhlShDYfX7ON+Q== X-Google-Smtp-Source: AGHT+IGO0+mWINscMdGqF/L7XDe6Dv/dfGXrIYILRYMXJn04aY3ZCG1rf5V2YGuLlSic6wG5VWjqsA== X-Received: by 2002:a05:6a00:4090:b0:72a:a7a4:b4cd with SMTP id d2e1a72fcca58-72abdeb85b1mr18289604b3a.21.1734947034102; Mon, 23 Dec 2024 01:43:54 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:43:53 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 04/17] mm: pgtable: add statistics for P4D level page table Date: Mon, 23 Dec 2024 17:40:50 +0800 Message-Id: <2fa644e37ab917292f5c342e40fa805aa91afbbd.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: DB8241C000D X-Rspamd-Server: rspam12 X-Stat-Signature: 8g47u6ghty5wgrxiq7xd9tkj6ta6iknq X-Rspam-User: X-HE-Tag: 1734946962-140771 X-HE-Meta: U2FsdGVkX19AW+lw0DmyxBZj4iNwCsSxUDkvos3Pijt0E0gi/pckUtjv0y7z75eAPw0oBex9JzRLVFsvAT+yceypjgJl/j75hM0byiIjcXsEpz3vofs8zjdUJXYmAmeD7Swx0mbaXxOgcBW9pFv1jDpSqnbzii5Fy43VeHGD0smmQjOK2yMrjiqlGWIm/yeD5VgzDI4jTvy02XNwsQfV0rVigNghIjm6393mepG6f5S36le3jQy0eKCBk7wsi27H6HKc5izvE0uOnVW4SM8aturO2Yk6NAbrHYoQST/dNXP7cClZhtNeJjnKF8679gZJShsTG8XAwRPyHRlB2JgUPlIHZnM8rfVGGcAg9nBTtaLQKdleOw2fDUj/HkeFipNCp+l8fvFW4caKta6ZiDqyquN0n2ecdwzTVmnALIVRPJRDyMmvxSaS2fZqP+06rjkwXWIYhxqS5BgY6vrrA44UrtOsIqZWtQMvQE43YSgtR81FhvuH0VpeKkKA5dFmLH++z3v+Op6sbntpW81XbrjVwRrk2HTSGmZXIWHOMpBAR9nJ1WRk9uH4wGNoG//0bD++DHWEeZvcq1ZCNlbSeaMdDSJ/wFJlkMuwy8vRzAvEjKwuFJDlWJqDgUiC9J6dH85i8LhHCIFb7yybFEE0cEQfrwYiwzbi7mZdpCtfxSTdD/affTc4XQStEHNDmgGku8rjg9wVJzFVdiN0/NGAmceNCf5hHppv9aoiFmY3aHEeuNwCRpAuIcOvnuOgP6fn+P+cwDYf7D2OeEuVwiJ1wx32t1ZNpJ/o76xysY4thnvIj3U65ICaQJLE8D4II6b+NtWq3FbBrJQkNHXhFoNxBrlQ8XrUc/G5z2Fqz9zOvfHRebJ1u3Y1FL1y1oQx3P4V/FRTsaac9ESToKkmCmEB+VvQaz4LBKeZxD4PUTELO9ORShNerchhC5506w1J4oTFp7XvF11edSGe3u3ewBr7v+u r/o6eErS DFFK0560JiMfjSN13P/1NLaib0PHwJWy9BBamKtmCTiqCBA9mtzJN4n8kJUKenp1ZO+ezWUWLO2UEps9IraXONbUQEWPHB4XiafBTcloKV+7sWousbZ/o+i+KDkWZY2Jznoo8Zim7yFX8K6aplvpYqCSfacK7BfRj2iW2lHRg+dS7o5NWr7bacY1YuJUdI+4Y7Qp8oP8xuIqiaCv3TDwdRs79KmS+7quOcdKrRNIm0qB+Gx2awkEqQaXgm1WqRzjpY4h1Fi7FzTmILujiP8LS7y/kcl6rxq8utTAP0iXR1EGbNbdnxFv459S1OSzQ1q4xe16IvJKJW8cGX6qKTezx/4KAN47ndujp5vN/isiZvJtGpxWHchSTx6nFJpGsyHOiofFva8oKZ8U4LttNj6n8Be7Zmx7TupjRI6p2aH84tk2yeAkWY948b1Vsfn02RT/U+pEo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Mon Dec 23 09:40:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA7DFE7718B for ; Mon, 23 Dec 2024 09:44:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 757FC6B0099; Mon, 23 Dec 2024 04:44:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 707746B009A; Mon, 23 Dec 2024 04:44:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A9626B009B; Mon, 23 Dec 2024 04:44:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3A50A6B0099 for ; Mon, 23 Dec 2024 04:44:10 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BCBF08107D for ; Mon, 23 Dec 2024 09:44:09 +0000 (UTC) X-FDA: 82925737104.16.614BBEB Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf10.hostedemail.com (Postfix) with ESMTP id 5DAC6C0002 for ; Mon, 23 Dec 2024 09:43:51 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fLDg3Ytl; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947012; a=rsa-sha256; cv=none; b=qFvpg5gSMEpwPTnwPgg4Uzgaq5/kmY0ZnhM53mB85QOkyX6KNaZY8S2SnwNABu076of+cF t7dyKeWmYfFAmGi5i+LE79owpnnxmlpnCN0WAG56D8KL+IBPqBus9+K+hQQzXUogX/6bXD Cqc7/0WUnQuWPZPPksYWU7NaHUHEkug= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fLDg3Ytl; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=aouRqdGP514gg/3A82li6DxlEN4YmfB5bAULvZztX/BxU0nHYA6wqfJg/PWItuUShwQaEI 7dptwb+PIxOTCWBbFxtJUfkcs3A7E7QfPpgGgdYBDItDXdOaqjXtVSxHPpfVhba7qKlkC0 gh4BLdiDgM7hvz9y8v+AVB0iW/7B7nw= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fd526d4d9eso3050886a12.2 for ; Mon, 23 Dec 2024 01:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947047; x=1735551847; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=fLDg3Ytl905tWOAqwyvidw+NuT607izuH04XFclQF6m1vA9u2YdoPBJRUw/wW34Ag0 VnnZfO/wdCj5MwRfGXBvONCvGpMfHstSkA8tbK5U5o7t7GwxI4bfSdbpfoKX9ebASPum rGCo9IYMKfsiqGHSPZNckquNESace4gfsM0DOmdpwyAIkX2JNupjL8oeceExGS/BKDKI 30j1sY6Ww8DWKJM/aIHGV2xpbCFJXvGUvSq827nDRcKaC/2prOVi8tvc98QiKJNiF9Bi 0mMkHU19vYbmmysHwLu/29AuiFLwgxbzGns1uBMTvWBGeHUmgs6IYOV6DrqqE9H6zlyq GIUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947047; x=1735551847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eYmLHVK0AtU6ilI7ucnXt5jLNeYE7S1uMrvnVL/VK/o=; b=eIz4B+/zZrqfJbh6jW03xFos1FbFPAOz77Gc6NHvd/GwEswCD19nLPw4ypmlZyakgD GLtGJENzkips0PF8k00ivXX7pWDQJYjHBo9GN+J7Uh9CeCAbVxB62o6F6EayqpopOcOf qt1trqJPQW9Ur99C9pFZvErfxSae7qh27DeaQDRqiZVRSJef+dCSeMGqbBVy0U0Gr99y jyWYXaHxwFKIZlV5anj6cYn6QLlnL6dYsvGEE09awHms9t571d9rowQZwi4bsIQtEDOZ +XWqKq0QmvZFzTr25Zi/ccWlr32pAjiplTfHZprbQ+lBBts/gXnS453Lm+bknCupnB2D Hs7w== X-Gm-Message-State: AOJu0YxjRamEvhb8mXQ/cHJL46XVFCSdR5xkY7VlsJlWNioK4ako1Hkv m1Kr0EL3X7o5P7mAKpOeQ0L7VL+h4CtPU/9eCq6EgUk+37FvYtkgdEb9qz5S9WM= X-Gm-Gg: ASbGncucAZL73AIS3JUqRV6EzIFuoYvcXQsTDjHy0Jx39ZlfAL+73xY6afW0YJUEkt5 /xZmbpnuGhtJV2zTuy5wWNEFFRFt2/LCoHfaNs+/NiVLmNevJyxKqduPtJZtgGsSGDjQqkn55uY k/KTtmjcc/aOy4beLPjfX/6PNqeoQBm4+Z1Tr4yzVBzMMAoSB/vNJQAWWhRWPfFTfMF/st3XzDd SjQHbFu26EGsYO2g5g3T5M9HsCnKuNnti3Zft0BSbBfaThqIUbjOgnOL1TC4E3SLyTEwlI6nkeZ CdGMdPLtr4F4rrSzMD1Jng== X-Google-Smtp-Source: AGHT+IEE3DAo82DBKWUM6FD2Qxwd8GTGkBziQ/dfcNtaRbY+efEiB7FXaTM2aY8g2l19VVq60zkLag== X-Received: by 2002:a05:6a20:7f82:b0:1e0:d575:8d4f with SMTP id adf61e73a8af0-1e5e081c5acmr19810935637.37.1734947046731; Mon, 23 Dec 2024 01:44:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:06 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 05/17] arm64: pgtable: use mmu gather to free p4d level page table Date: Mon, 23 Dec 2024 17:40:51 +0800 Message-Id: <7c12112047ac230809aacd0379259414b9b0d3a3.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 5DAC6C0002 X-Stat-Signature: qhmejf5epe4bjy63yhrdfiriu4fkza79 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734947031-918296 X-HE-Meta: U2FsdGVkX19DnBxp3F5E7YGHYUjihQbEB8iPqdiTYNyy/BFZRbMOVm1uPPurza4YGMnHqt3fTfXjF6IrRy7r9nJcFUYevr/kEW627k1WFnjcFoDJ/N3zVkscjT8hNW2Srko9dxJbXPoy5INmNQckVzs/Ypqn7V04MScOYRDHlC8KJNoOuGkcCtNpnx7cBhIhXsdQWm8DJow6CtGhGfz+e2I1sD2tzL3OopcFEHHQmHy9teEvuNxKz6yb3T9FQ4uqLuIxkSrqu1aPrKP1eDHgeUVf+rK72tZ8vSrOljXuribmoHN/uqtt2IqRMulGt7FjkkRDaT6OGhYNH8fxqXuMGSoXg5vjM2M2Kji5cbVQ3EBFkw9kXCoAGCV41bAbC/+A8aLJiDgU0EpBOAZsnszRHHytN35dLjReRZQNZf1JgqHiFgG6GAyVKCNFsI42apdj4NpSwK9LnWjHVjO0LmLiy0nkcNX4fdLI6vgX+tq2XYiiZlcqq+4Gan7aC9PD/BfenEXF7WhlymQnSZ7r9LZnWWku5lixVCaNQtBWF2jRa9Y0pUIbB5E6Vouiu04+Wxm2xc5j75ySvMRReb9dopxSpSMfcsNk+6MOpGnmUN+0jlYJwJJiruZD6gz3b5xdH9D+XiRSqG9xaIk/DZCjPQPnCKNiq7LvaUOn/6iEtM5vUA5Dm7qAbeg+kgmK1iOaty3juBUwu9vSR4lfcKE6D0EpV4qvKnCpRNIqDjGYge3TqiLl92sNhKBUmYw9nMtMNvpe/diqmuC0pVOCuGpwIIcjb6EuA2c6e7MdjeltjJuJD+YCsji66xqSpui55Iz1tUKoCK5Og8Wwlfg2cKv0Hhg3LNgFSJ8u3fyq15jMrEUUg8/qnmA/mifngC93hnXubkwH96bwCorNflj3NsZtlqf6xiob1ESNZdcJ9L0e+T3Nc25L0RU7xmCX3GaZcfKlXHR03DUh6/2M8yLSTRorCBQ xNw3jv8M E6ypdbZu8lssYcafCo0mUNoppRG6NEXSrHaxsaXwCNlwC70AbTicJxK4sy2/V3T1wBuHX6DyriTVQNmDuS63gZh4G3RmqzS4YJRFz7JIb8oPpp+E3jBxp+IaTFLRAfPmF8+43cBeHtQwnMc7v/cs1NEUxiCtpCKfauzC/b49xbTxEQ7U7+YfD1WK5WwqEOd6xROgSdzVGaaTq3gKPJTfbT1lNHHy3M7X0In2VA7nkWcOMTfGE1wu70pmEOQu8ELfnJKdy+wBVoN6yeV1kEn//UAMC2sOKnVPSRiqD2ENePtGM81wlu9yFA4tfFjXNVe5vvwdJ5r1vM84icfuKrO5G9SEV+7BHzYVabPVitvw4U0c+Ze/Y1DDLzc5B47uDbwdMMmsM5psaaxrkw1fCdSlQaf6Y61RGa/801Jsllcs+nF77nM4UgyqYnJVPvAanyNIdVNvo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000463, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Mon Dec 23 09:40:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 018CAE7718B for ; Mon, 23 Dec 2024 09:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 906086B009B; Mon, 23 Dec 2024 04:44:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B5C66B009C; Mon, 23 Dec 2024 04:44:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7585B6B009D; Mon, 23 Dec 2024 04:44:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5747B6B009B for ; Mon, 23 Dec 2024 04:44:23 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F39F1C0526 for ; Mon, 23 Dec 2024 09:44:22 +0000 (UTC) X-FDA: 82925737650.16.18A12EE Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf01.hostedemail.com (Postfix) with ESMTP id 5E1CA40017 for ; Mon, 23 Dec 2024 09:43:52 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=dTxtKPH6; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=79R8awp2CZ50a4THD8L7005w1u8k+K6ZmytWCjZGjlPnkxehx2jIEl/VF1L5MlWQnaidxC 73+6QImL1s4XVZoxqFfC4Y4sbN4QyMpyymD/lyKlLuQ3sl/OZ+nsqdDAxbu5fDphAUS2nM wLvLaGnJsqMM8LwnbHvQWxlOCsXoRMg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=dTxtKPH6; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947021; a=rsa-sha256; cv=none; b=6Kuy8ACEibrjsphiVlelQ9IS4EmNFZvS7GisHmBnruo8lgPV/b6wRRmbHL6ve2qB3AY8YP BLCFoTzrDYaNLi3WEe/VNlo2upV7XXCHOvCut0yVLeD3ea2XSTe2iRFTnlhnPEAWfJB43I DSFa+d+UAcGKiYeO8NO8vVY5+vn5TR0= Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-7ea9739647bso2563603a12.0 for ; Mon, 23 Dec 2024 01:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947060; x=1735551860; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=dTxtKPH6qVUaUi/wWWd9ZPwogxL1MGRxqLi0YCSJz28w2g5gn61u49kZ2xtw0RGFbb JEDwHLj9Z7Pct9NdaATmX5dJ+gGEMPvRq4uSaWhsa5N3XYb1eE5znRQ2eff+Q2Va7uuk d6zVM2dbptINW0xZ+F4+lsRbj0d4E7dUFV6RVRSVYEqlBCBosPHKKoZT22KxLbkuo7EQ clnksrT0iMzR7hCI2YPTSNbsoRGG+SrjWGDU+XIktQcVpCmRa6qPYQNxzD43Iyky8EOs m8UA2hGr5Z24VL49Krwm3KnPFaFivt0m2iCRF/mZrSqelb5Tz8siWyg6VO4r3AHfgvuG cMvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947060; x=1735551860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=dW0nQhn/J25Wc+iq3DfDnsqRM2LlvhhJFLFQ20oJrv3d2O2bWGakz9PKTgZklVwOjH YZcxDW0jhbSDK8yaGAypM0fiFIJhLGz4OrhZBFQ3TyQwLuoB704ZPQd0nwtDHMva54S6 ldy7DJ4OV0tF7illUcFbC+yXlrHukn8ohXge6BqAW0m2Cr6BYWDRX8a7CCCGM8vC1jra Gg7YoNyVIVZ4ZIHiQTI/VFrNkYsjTMzDCL7QEjSz8KMjjcjT1/w6aK8ooNn5iAgXA6CO QeGbXVflYczAul+qKEEVfyoqLZ2OlYoAvyaOvgxP+h04pOhBerBvHEbb1SQaUk6cMBwX KIHw== X-Gm-Message-State: AOJu0YwKp/njDvosZzjSlyFdW7hKwBcUvbvNJq4jAXGoz9OSkb87BbVZ KFQRIwW46TDWoCCd+ydU6QtT3OzWmn9Rmp7crmbhcPxiMe7TaiiAgGRq8X/4rtk= X-Gm-Gg: ASbGncs4Cd558SZtJga2ze7thWQ48yI5wEvJhFy6MrQyoOEIx65TsDA5cVxy7MPTFnV u9ytJO9wSMQV/SXF8sUCeqRTKTwFBlWpXFnkNhZDuZyXeook04yHHiERtG1wOmkRxu2qpvDeiXG hKluhtNO7uMQMuzw7yqSw+6rw4MAO/Z75uU/5t9jYNm2Te+acKDHFPpxCLtBTG7sAOTAnR5Jm1L FxWkwD0ngNZMID5JQJgLVMmcxVPprdOyW+yXR1gGAhhW/YRoKqhvHFo7r99Hb4Vj1+azkTTFx7D JGLq3o/ZpxiVG00Ehned3w== X-Google-Smtp-Source: AGHT+IF7O/ZMCvOOtywTANPL1zwoTq06yXXDMhnBrPgrfov3E/r1DzhOhOYD8wImAojzeP7TdceoFw== X-Received: by 2002:a05:6a21:3285:b0:1e1:aad7:d50d with SMTP id adf61e73a8af0-1e5e084b681mr22018514637.46.1734947059944; Mon, 23 Dec 2024 01:44:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 06/17] s390: pgtable: add statistics for PUD and P4D level page table Date: Mon, 23 Dec 2024 17:40:52 +0800 Message-Id: <35be22a2b1666df729a9fc108c2da5cce266e4be.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5E1CA40017 X-Rspam-User: X-Stat-Signature: 1pd7y1rkja9fdnaszcafpe196fbgg7ka X-HE-Tag: 1734947032-297404 X-HE-Meta: U2FsdGVkX18FsQJ1TwON+8zF2nj9VHeVCkF8VQtSKonnK/fuTCwnAVkQsjfoqkvBvqghf0c4jPSJhPa9OermrK+ZtaOvRUaxOhOI8gc9Xm/fPz5chUQaB4E89yfrzSmfKymOBBw0YERIuhqPv2+5hlHS8I7cFp/9Leb9TiOZQgzYfjfIIY48HXJKPo1rYUxjgF0PiH3GdveWquMzB6O7+O4yhZLosXpGXC6VK3/Of8VeJXKZN/ll72YBWGUpHbfouppTfz+to9F9MY7GrRKALPzz6NLWDBZNaU/eUX6e5jS6Abm9Hyy7Pq8XdeKr9bxW6bpa0kHCQ36y8NWJuLrtPZItmK8RcoQL4q/IUZlz6qKTXeI+ETh+seq5c751WCvQT8xaw3uF7d5Xn723kvxb5brwucwhmvb5IYeoAJfJGPqfP+8ruy4+MUktil+QgrMuBM5nbn8vsPrA0HLYytL9qPclGxXdwaMN+IQn+KmdV/vKVsefzb5vT/U/NToisYDQUyd4iNoWM75tC36goJcNVO9mdSNBWQ6zQHWHPMoepIDqiLgOrNMMDcXDGV9tqRB2Ny6AcPtIpNBP8uao/rnK5J7IqOdMmcZ01D+kYgByo7l1AyxWjfPj6U5W0TesR5dCHwiH7kU4qujMmpZEwbVKiixrmhBKFd3cAjZqiUfODS9Zme302U57B7BDrlMidDXmg6jv2+bOmgmiBB2cjC+INMW12HQdUJNt7WR9t3GbErDhVcDCKrfqBmY/1sdwcMc9hZKSSi+ZQDe2Vh4RWz46P/QlbKcnIBQHOXdQOHlGEeQaAzhD7VtZQKMWWXCrx8xkx4kE/BCPrJrkxIauFCkVehk5/xc5hSasFjdNh2h4wyVzQI6VDCPH2bbEZdFgnXDE51/x76YKPlo+gOit3sUDZ/qHfMmpsAtpYcfDJHraTdAjpIOHHcsheVfRlHnrdzyQk2KXs3crzI55xB6uj9l ALmFogBT WozZ8a86kpxY0wLI2vdKFvhQiyHMOl2nn97hn1XmrrwEveLg2cAQXjzdlNxQGYIJ61T8Z/b8etxeSjvZRPuCSv40g6ZR0VWBSzXOqkSxgt5EdlGzCVz/1pqKrz/CjMaP+m5MCrQ9q1rdZAxlZNI7Sh3tM3yrdtGjmUW/WOw06MCWhQdFBtYHaifZBfyvTjydk624h/eZmy/KWAE33x76p5le7XYY3Qiudw15Yhh+0wMpMA1/OzbnXfdubKkYzLUAUeIDfexHSR9XgoKPurvi8/P6b2nHuH59/p7AEOzZ8xchCmtfnZzVLxt/qx8iTMDarZ1/NiSXk8higefJjIlgsLZ7c++qemgUCNa2przI9cUEBJaMxRZ0bgOxDJpq+p/ZFRMubSUy+VgW8I0R3+hy/Q84S406cOB2ZzNG0dHyQe2U+w6v+qq004D1XnR2dXsSSkL8BAx5RrORDbdaHMWR/2mLuzw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000017, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */ From patchwork Mon Dec 23 09:40:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29218E7718B for ; Mon, 23 Dec 2024 09:44:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B285B6B009D; Mon, 23 Dec 2024 04:44:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD8046B009E; Mon, 23 Dec 2024 04:44:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92C136B009F; Mon, 23 Dec 2024 04:44:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F1DB6B009D for ; Mon, 23 Dec 2024 04:44:36 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 24B89B039A for ; Mon, 23 Dec 2024 09:44:36 +0000 (UTC) X-FDA: 82925738490.02.BF46F0D Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf12.hostedemail.com (Postfix) with ESMTP id 901AA40007 for ; Mon, 23 Dec 2024 09:44:17 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AK8knqmQ; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=GJ31PYNerUxpOzHA2G1HFqW/sl+3oGszRzKqIUd5prAkJzWdjgbKzGgie7PIOYdz6TAeNo PojfpnwYdr021lKOgjnHFNiaY033BVTY6fAH1/PcNXB8vBUKMZ7Isjqb5PYHAQ5O1L5vHs 1AnloJSczaxR7l3wJNE8A30cnlu7QbM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=AK8knqmQ; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947048; a=rsa-sha256; cv=none; b=qceokS+xq7D9AQsv0k97gnmCwAFflpsPxMnzeqNiF6qSyVu00ybXQDHCeil7qSS0G3JlXK 1eMxXxE4z5OQW6P9i6TS86FuQtLiKnRNRCbPk4AtJLDD7XtBOj91RGvBHwhqKta5jaUdOW 1XH6rkSgIPLsp4owC8e4qQQdFeqi6wY= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-725dbdf380aso3138502b3a.3 for ; Mon, 23 Dec 2024 01:44:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947073; x=1735551873; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=AK8knqmQJ8QTMqiYgNwsPoUenMj73gxlSIP5RswoQsn0AyNsWzn6aTn42wO6FXk/vM jrfskPgS/bnLv6q5UlhdJUGJQnnljacyz/21HAMn2Yytq6HqtvPbhLz0nwhDmh2o0Fbh jxhg5D9uZ3NSGqo/zWi2NBYvF5hLyzXjn15x+uGDF+hR49KGHsWCvaiJ4uVdpVXHouRa e8mAsf1Lsv6Cinpm1e+7tZLcMzd2Nv3MDWhAuTU+obCz7AHE/LaBXbwweovhBPboKcS9 hUB9YiUgefPRXzCXfCbJVG1RZiTmE9iAehFXx1p9ASnzahDtrZnl3/aVAsdnDdTa7QIR 6KgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947073; x=1735551873; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=if0a5qgfKyLZEs2UFcbOJ2pRLoezhNdJsDQpJdHhbkw=; b=ActyPe4J+1udcLsH422rCCs2OjOV/PfwQrpwds32ycVf2KZuEn6s9+pVVkJ84IwFb5 nqU3vnN4Adr/Ct+KiviFFupA+aDsdivvYqbzp71gF9jVBfTi3LI8vBYxsi5nmHy3Asyp pR23yrHl73OoLuG3oVJzQALmWxzfyUuV7g7R3FW4Ro5kRVviFgdSXqupX28r8ui3LDKp bpiPIrbKamKrms5TlKSgpj0CU+g8k3pDvGgiq+xIm8ow5Y+EgCWXJ1QEfFXkX85JhMWV ou0Gexg1W9RhOpKx8JTxkrl/UPWEV+ix6vRfbtXUKp5p5VvNzl1R6K9X4ZoaK2S8sZ8d tliQ== X-Gm-Message-State: AOJu0Yzr7CKucBK8UrypMgWDAqDkTLvnnIjOWoO6DTt7sdQ3ZG7S6+zG wkuB6PuaviyaGvkmMt+jCcJFPYRvLqntdmD/D6Zd1zA9jysQPnaTA7HgppMDcQ4= X-Gm-Gg: ASbGncuYO4b09NUlCN8Jju5/bM7mALpZbPBQUZuAPdcORqiYOs/0um1jkzsvubs5qBg pPE8SxwAevIc3W8adC4v4Amq9q3FirGoSX/E4UDYOO+AmKsaMJ48IWuXjen+axgO2XLcj0SkZbx 5agicwkM1RvEDoYPttcYqOL6C0C2GZCxo+m0ctrGoc489VQkjLlyRYED4dL4uWBqYcy010ivEcy 33Z0CpVzZ7yCy/gELhtWngOkCci5zsZUuP/05E7dhF22+QIx8GE//eX1GCxmAtq2COj9V/hJR2O 2lK6JYv7AhCLfI7cUnyolg== X-Google-Smtp-Source: AGHT+IHNcG21n8t2Tfuvp8LnDtpoSzOkwaxGY/BugzvHG6e31xXuyXvNhN+O2xx2uo+0V3sMDp7pew== X-Received: by 2002:a05:6a21:6f02:b0:1e5:b082:e38f with SMTP id adf61e73a8af0-1e5e081d16cmr18629210637.45.1734947072789; Mon, 23 Dec 2024 01:44:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:32 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 07/17] mm: pgtable: introduce pagetable_dtor() Date: Mon, 23 Dec 2024 17:40:53 +0800 Message-Id: <8ada95453180c71b7fca92b9a9f11fa0f92d45a6.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: reid9mzctpx5ytgaeegfggt6npdoofw4 X-Rspamd-Queue-Id: 901AA40007 X-Rspam-User: X-HE-Tag: 1734947057-752871 X-HE-Meta: U2FsdGVkX1/41XnwEEVc6jCZpqwIXACo/902/Rgk4EjgAOpoEWP9E3i3+/+itTiPqcOaCYmy6wC8wLQWsHun1gFcbVxB7yBWys5/jOABT+VdYDZXx7VHN6SfPJ36CR+x0tNB7thdW9WuvOBRShzWK4w/0NbM6TzAP9vLunK095W+pAEsO4hPcO/UK9Nb3P+qSHg4DmgDjOnIw1g/pyVnBtdeJuhqKCew0XvPuChvEYNmy2Be6IIxfdgjpMJrYyknbQjxMfcZLwI5LY2z1To66TPPfXZRbKGuwkFF9WWhFNCfDXWyzrOGG+XNTLwJu8CcTA6Nrz8CbtKK7jqNiZyGUCDxe/IBE79dcGvFncaBk+Rexd+ThBv/9MxaPSTaidQf6fwjxo5SzBJHJ1SaPf7z8aYjIXLJQ8bGL3IP7Cvpn1xg+EAmJP6ERo0gIRB09UpHAM9I6qafhKfLUZYCH8IXcT3clXaMckgUtd+aoq0nzUvioCnJuzoKnMpDpdHiGu91SSE3TGSTExTRWJ5rrvHJRE3UbXpHAaqOYGYySMvev1VWbTzBcaVheS9KwrRm1W4jbyd5uTVyZY07Ak7x9IG3TSV1FZJ0kDwKtzV9dCtLX3BPNWVsMMzU6FfVBzChpn5M7w0kubrAHCED7ZyLfUus5sKCyGD8jgD/oswhjrWaTsWWR0Fs+6z2hUHRSBoR1R16YYWVlcWFEYiTQ3GaSJf0WbUiwa4EpoAVEr/ogGmmG7Fhw+5f2XJIuFmcd0xTheeloMB/tNqhPljy8mlV4CTqqgvm4jUPH++/mpeQxHLuuewDgAMtFRjY5fsaIfSetlGBCF/dZ7qaKhXObFMmHScTMzrRevOb7xFhkDdw6Itrv1TgsvZgfNOPLoCzAZi3xeDDRT9/Wb2Dw7ohANbWSGIfO+nsJWAqxvAuRODblsofj8Gb3c3876voAXQH5gpkS9VyA36PlTf0aReQCo9+Pib CwU+q09h SEeGTR0pntXRwnBHrhQczOglzVrNEga7kJfm8pinjYdaDTJVOGlMqD3Holfx/phqaivRsc++psZPelq5yHmJmaYbCUpAOXftr/h15RIJ+L0sLA0DC8UpQYU3NYFIXHJT4KCKsmV7YfrXqrOSehdars+9jTR3NsMa41yfiQxWCJgAzugflO2lq5bfTr+0oOw8grOrHBGa4hBYBAGy7DWdh1fIFy5y7HaV7KJoB/YkXwqy5pon6LNe0YbOjdtTa2XTMWfMR5B9nD2b7JKNqP2r7aLFvsyO17UVPdGARxz40jKrAskE1leINugeO3W9NGZvmz7hWkH2LltskJ8LuK6XgDXi6XM7JfojowwhlyTt9CyrPXVJW+qhrLWNXkNbtRA5pRSlsGGAZERSnJZidLlaYz2ys/tTQfStEBe0yZMBU2BiWNySGX0sqgY912GX+5gh5SD/CaXVwRPgeaQdNupKVXtXE43cA9sstyhriP2FCaSS+epA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Mon Dec 23 09:40:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5646E7718B for ; Mon, 23 Dec 2024 09:44:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 201BC6B00A1; Mon, 23 Dec 2024 04:44:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B0E06B00A0; Mon, 23 Dec 2024 04:44:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 079F46B00A1; Mon, 23 Dec 2024 04:44:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DD21D6B009F for ; Mon, 23 Dec 2024 04:44:48 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9098080FF7 for ; Mon, 23 Dec 2024 09:44:48 +0000 (UTC) X-FDA: 82925738364.24.D4E5B43 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf02.hostedemail.com (Postfix) with ESMTP id 4E2C38000E for ; Mon, 23 Dec 2024 09:43:34 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kt7P59zg; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947051; a=rsa-sha256; cv=none; b=XN51c5+AvH9FROu2mGac6U+pEeyV6uZGJHGziqSqtD6y2cJ9RLaLdoA70e/Gy59u1YwjDI fRr1zk9+2/13X/UFoxjinMW3b21cdaFvU/liaVuqdNd6sscQM7aWNI4oW3mKllg0brLUXP OgryTRMN3u3LbkByOUNEEbqyvic9tcc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kt7P59zg; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=7hhKEcBRevZhCyWJwIEQjiHnCWYbxMQSgHdeHGc40Y1MavnTsFiCdg9R8calJyLvPFn6qj HeUuAq5Hot8aGQ4pphSu+Xq/NC0kG09ZHDC2DC59Zq8AE9A6Y6SAuNyDzsfr392hkfmeC2 BiHSj1xjYsMvIezrNYjQmBlFjWnSWTM= Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-72909c459c4so3085607b3a.1 for ; Mon, 23 Dec 2024 01:44:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947085; x=1735551885; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=kt7P59zgI/Svyo/DsYyuln1aLVBFIe8ZZVWjFw5i38n84v6Xld44IupjGeS5Wckb10 /y5Csq1RfaG/ZhQvX9UW8nQ53Dffcg7XpUdZHErJu62kho1yWlJseETVULepRKb6juYS 53ShYvn89E4uatohj8Pzn6AJUYPDkMZNi+2r+0eQq1W0P1LqpihJrmXCJcVS+gie4IWQ hKZnKnJuD97qBQpls+TQvPpCIxh2G5SKfoqsOVMY9lPTBNzTwx4pX3K06yYXbXdZSN6b 9mMKHSWaPhaHwyzFoQuup7IlRh2SWKmvyROTcYZdxakH172QpYXdl+dc6g1a7IpbErJF R9bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947085; x=1735551885; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qA0mhWnXo6upWemI3POPEc6VJyAkyxFJggkOB6WNq2k=; b=BuZ+dg2EWyAyZxGiqHghs62LmfUv2TNrlX+MVtvi5FYR9ZE0pcvGm9BukA3lMR2xDj XagmEIMieCWrOp7c12MSUZjEiN7zEy/NKVU2hddCcd5qkqJGiOkE0fSkmHFlFDulDVRW Q5fug6yrWq8QYq8Y8Em3biXt8G+7n4nS6//6YcKPFvl6/dqxQnr/3gn3GlPQ99+JV5Zs 8xthHEcQYqC3j4e5D4uk66HSsRAyyIouleeiUDBWbdohykQ+k8ao+T14eLjKOkvndV23 EZwgqZ0a6yMMYKPVlUWZ76Z/upxGnykhPIqS3/0tbn/jAids4vptznn37CO+CpHtRgXU uQCA== X-Gm-Message-State: AOJu0Yzm53dKXUni7vcRHBTjaV6lbjJZ9xsA5VkilKYjhukp1WqBGRnm IpVb6NlfOgyM3EMNR5ETxUj7KuWSHWPOJ4XaT1BJ4qByquCCjCw3oI0seOKKN0c= X-Gm-Gg: ASbGnctR5DLn+WVUqtfDc41mOZg0UXYmADFNY3hiw9xHsNTX8oiEr+CkJqKQ++kvROH /qPUcVq/nTECZqT5NY18uJOtY9F16HjEt212NrZOeWmOedydXMPUftiVHRcZ7j6pp2R3U/CEpTq k7Nle4gxGAGMDsRrELuyFpBrgwDPPTFUaczpnGlEwihMaF+n1sbzWtNrwVFOpAVlT1lezbPivVf Mt02PH8eHqT+oZ53nWrkcxvJZdtCdnvy/9kKmZoMtZeFe/0A67F3vrWtoe9Za2WqKApC7TBi+t2 kYeV4cdWWmfcwJd0FFRvlg== X-Google-Smtp-Source: AGHT+IEwnxNRxjpY566VNLJW6lRIrSoOOApqv/9DKS3yUbVNE+/391+nOnqyFIwO+T3gR5XgQach6A== X-Received: by 2002:a05:6a00:330b:b0:728:e382:5f14 with SMTP id d2e1a72fcca58-72abdd7bae3mr14443469b3a.9.1734947085429; Mon, 23 Dec 2024 01:44:45 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 08/17] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:54 +0800 Message-Id: <955162bfbbcd9fbb3b074e1fe2aef4f64b61d6f9.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4E2C38000E X-Stat-Signature: 79bg58jnrazh4kfweqyhpnerawkg6zau X-Rspam-User: X-HE-Tag: 1734947014-501639 X-HE-Meta: U2FsdGVkX1/TbFYCtNhGXEjezx6JAsXhrWR4nqZhWn3Xy5DkClRcvwUSJVFgw+fPqLVwKz8basAuT4d34hL8k/lshslqS28Y4bhViuMtOSDOYIihZDOTeZWRkSe/TtdjCYZlcBL7+WKFpp2/+f49sm3m6QDu44TVl3euKSA29QEzSHAz4d+YQgI3fPlETwQOVPGadAWDRMmDgVoYUH6Vl45lveFJAjoKQRynMRG6I0RN3U/ARNfurbFHnxGMuLKKoNpEN4ePUw0FLg4AkgJLcQXjlEkG/9USrH9FEcEUS32Mx4eNk0HY9jS5E6wMpGKD9EZWx87RM6NK/go/JdGi0eais/Vj3RXiRTstKvyKKtt5s6TtQZMmZghddhKfLH8VVGCE6k+UvscKJjA7B2dYuRvzZbPkpNjoPilkzmnE0j+pns28BD1nNy6tBk879pJg9tYCE0H0Cle/W9dhx86Tak7h3AHXXKXWGvaDnWaj1BACRpz80uwgMBcxayyn/NMTqwll0+504zWJkB0UhOgm2qBIn/dNpD6VkdhSOj8iV1m3Q4zV+4bEureJVANUDrE0tMFPPrYYd0P0UzkMd1Oy7HubMjmirffCuKYCQgYjK4dYw/j7m3FgYQdwh2fzc4gc0e83Kmbmtvsg6TVL34wh90DxNgf/0bau5KSqqn+oIKxou/Zr10h/OkMMKRaxYT7JTcjzNlLt6ADNO1fMIYaN8mE07+yihVtcQ9E/qJ4bF2GGvamPvd1J0zWzcl95bZYX8nYfk14GthYTvRwXnKWqIPCoYgMuaGZwjLevPI4BYbzkMXB/O0xgW9BoIhnxamUqgAfEGSyrdQiC7Q9zWrGaZxSn5lUP1SleZL2VOL6Rr2aDZc0nmr0CBRRCRiS43FbYwYz/UPIb2jfZcMTFNbo1xlE+N8fY3LPxoO22ZM4tLgatAVzWQi9quEPGOPNYC9Bcn7o/B7CgzPhwJrMto+O ETv11EzW fcKClN/SUXYboOCkp/g41YenE/oG8vqZrJCvQShox/KWtV0OVgnWwH6D7R0YEhU8WHShretTlONMIfdn4+jZz/Z4Q4nhqYcL/en+j4YnmYxbpVRAhjG9x9OE7bI4bU4GFazPuMxJlrZo+xdeF8/DToNUBgQEHbxeVPco0iD+TZGu9gbiqSSxg7KVeodUD//PeM6psEuYd4BzIUjJ01utybc2+xi9OdgfiR57SrM7WoDgLuSvIFP6xrBLEUx8+oZKNx8mymtvPuxpMm5tnzksTsDcMQocG9WqWHEv/Xwd3zFMMxjbwhcxNBmVhjkSKt2a+8f/niwtlJdiBPcuepN2/ShiBLPjPVTqjdVruerYiO5pttjDGHvXV3ZqbpRwWQPY4L+dSSPSxXTjMLb6acCDSKDs5moZK/e/PcZmnLQUJ7dMSraTrnqK+gnMF0WvXdEtLsAenhPkqg2Z6N1Cd5JjFrJrFw4B1enw0SlxLNYtLwgIaG+4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Mon Dec 23 09:40:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12A7EE7718D for ; Mon, 23 Dec 2024 09:45:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C8D56B008A; Mon, 23 Dec 2024 04:45:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 979306B00A0; Mon, 23 Dec 2024 04:45:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8423C6B00A2; Mon, 23 Dec 2024 04:45:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6580D6B008A for ; Mon, 23 Dec 2024 04:45:01 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2599CA0CC4 for ; Mon, 23 Dec 2024 09:45:01 +0000 (UTC) X-FDA: 82925738910.12.3756D94 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf03.hostedemail.com (Postfix) with ESMTP id 1CCFE20003 for ; Mon, 23 Dec 2024 09:44:41 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="LFZkQN/E"; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=k7lK+4q2aMEmrEGrxMnkYN8ug33z+aYfebCHNb4CnrdL5Q2WgJFsSKzCGGuMvRBNO2r6Fq TJ24cRnMf2NhTBE6C5CWUjsibjrf0kWIZDkqHAHd7um6CTuhoLR196KAEIi6PV49qa2myK CyImy+0M17mYaHNG4pdEXG63YOklmcc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947056; a=rsa-sha256; cv=none; b=AKYVxhDUWbPYojuhU153p49oapBCv+NArN+nd7r8qe86oRwoBtrrkw/YgMQRwFgX8u97zr yXTRDOiDJgXHh90zTDVG881AIP3o2Na54PAWumJIoBisva6Wb1BKr4ZbDFKFvPoAFBLn9B koseOUsnn/VWSIumBVdddeovqi3viwY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="LFZkQN/E"; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-728ea1e0bdbso3575261b3a.0 for ; Mon, 23 Dec 2024 01:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947098; x=1735551898; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=LFZkQN/Ef+KU933qTOaxaLcJDHPPWEFqnjY3cSjkKyu1NiqlAwMtAHbYFYdmMkKhTM QR33DsbT2Rb4Fnvyt+YzjLp+mlky/431iLJ6NGYKbaiB2vzAbmPWjbOksGScQBxz1Aaw 0G2K7k+/z5ZfNz91UTWlMThv/pVGtWkjj56MJ+W3iaWp767neDOry3rB0zEJAthyMqe3 e4cc0ST0M47rWXBKZN2/663dgFFyCEYxdMwE0XNplg6HaT19g57gAVuMYsNmMghtRndq oVup+BPnBAq052boLeszDLItSAsvlzV+33D4TAuSWmT/6qkYqe5qLUk5nTtVZ0t+17/h RfuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947098; x=1735551898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nJ0bdEaVLB0+13gaCQHdoBfFkuXXT3Pqta7GOjv8Bo=; b=OesTi+RSSIrOnbTG4bpHRZCN5hQbl+/LvrPhIA70aNawhQqbXaoKhBKUgCQlpcIBHD NyoRgtjfhxIUs+W98jJ4Y+6LVkejkfe3EHEm/46s42rRlyHRTZnF9I5RDGDh0gfn6D8R Caj5v6FE60RQKdkp18aKwG+5zYfTWM3vMfEtaaQS8ZLEb/4wOVUNfH60cx8+elKZrM48 f8zcH6+XFHdRXWYdmQu176ICRxwAbALJ7RkWyoRZb+2zxKTOocZHksSgrn2Tidx23xIW xRcKfmpMeZM9fyK+7uchTZn/Xf6wql/hcDAQFQv+9l9pXwivh64Ug2EYcIsIZYoSNcFi +HTA== X-Gm-Message-State: AOJu0YyUmyvxegJ+gx9lUFtPVZON6NU4zln279Or8IaQYoIaHvR0j/sZ kIcnMB9sY81dZjGweDkKnrqAS9ibUfsnooptZ02LaAY5hciV2jkpo0mfLI2dGiE= X-Gm-Gg: ASbGncux//J+vv14R8zpKwbFe9nzI8ICaKYwps2q8UOCJM0xCxlKL14jJaxOeh01Q+s cXwzRs7i+VDpkaCbfH6eG3oCkJA4JPyaQXc1d7coUX6osHScONz4SMdAci5r7v5HC9RjAsCtZ3t 8dw6zX+9r6/uizY1YtDTsygptXchghwglFQWfJTFW9aAxDJcH3LPGXdcIzEfFPWkE9+BRgYYppJ Txl8Xt77uMgLXRuHix6UamARQbIROGSdEwagxJ3fo7I7/TpvNisKV0DSL3EBDm9ELQp65f1vDqD 8md0KdluKClpMH4lR8Fk1g== X-Google-Smtp-Source: AGHT+IH27a1l8kT42WYYi2KTFhW6smXjjCcrkYYj/EF7HLDGUXcMfllZ3ayHCCSFU+IsHMUxZvkRwQ== X-Received: by 2002:a05:6a21:6f87:b0:1db:c20f:2c4d with SMTP id adf61e73a8af0-1e5e044b1c1mr19165305637.2.1734947098042; Mon, 23 Dec 2024 01:44:58 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:44:57 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 09/17] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:55 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 1CCFE20003 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: hyf4qw1fj9w36pmzmtg66piqh1m3qc4y X-HE-Tag: 1734947081-673895 X-HE-Meta: U2FsdGVkX1+ocleiPhmRqwHYAY2/+2emRxs28tEX6Fv7gehZL+PYWr9ZNRIa1GgdIpXu3jtSGNsAXJ9clKtbjCLz4AHPkRQWE/hZ5ik5t4phGWXXat2FiGSfEMO3DtEG3VXBo/TXgmKGevEHfbL8mnE0ifZhNUC5pIxFk4SJ6e2OVMU6LLDQ9o9n0mrChPerXEDn4tD99v5LEnFoYtYBenL9id8eypp/luPxHfzLkPEJBfwTjZhBvr0BfR8AWnQyYeNPDWda6GTj+P0lfy29ixQ4kUS3g9w2NPkg7r63vc6FF8zZWPfAzzs76sTdN6DTmImEeAszYyQxOKSexS8q0AcjB6lgIr+kw+/Vl9GIC3GviVltzuzPqj/sitSXAWFBDZ1GZ73Oz2pxXZMfSLc01dLA8l0dOrAdXsu9eYQnxRKfkwdPCFh0kz8QRI322Ijrjj6F2QwMvN+0sbPX8uMPmFrfOjQuhkiQSws1/xcCvsYT/35/x8lUw84TOWHQbDuQJ/TKLpTO+oT89yo2nZVVit3uGOeKESJlmFf5zW/67qK+CHfokWfciJLkRjccOy88hlgdcZE9kW9QgUEeDIWLJ677zEEemhtDn+dEjWZ5I3cxMABrSNDIT06/8k061Vhd5ax4KSHws5cfgBFz0GKkclj7hnQ0u408mtjfUFDLaF6rXHLZ21t3ko8zZ9dzowDmkJ7osjqSBUMfqqMpEa5FGyWbBMCpwE/TYnKyKY+RKvMtPnQRIWuBGLkxZ7STHeqmpU53K14b0Cc0wIGEDM8A99YYtoDw82EXS90GgauGeFtVY/Ep/2He44UaGjE9QqwBs30ilaImWFhSYbW/9QOoV8UyecKgzTQ8/m3jEOpn5L9Qka66xdtQPSotFYPt9KSE+ADxXJvxMN6RIYh5Ts2nFhhA99ZciG3uYPR8FAHfMp3K7jNYBT+Euce71wFU7TwEtMNj1cM6htHxhAcynUZ XBwKFEoa EQM3ulrCgB8TEODQNOQf5EU/PE5Lunpv6AXWWeTwLm0TMZB0m7FIm1YmZ6Ej5rY72gPXBI0+MKAkxuViZVgFaJmYz7//q4iqXot/T8qJFk0mFNhyOG1TPJi3UxMRY9ep34RuKlVDp9hiUHzOsuCrm8TsLE8yBFJgexoP8Vgpm3zyA69CJ04tJN5xFRgefIGef3wLmjDHJZRlB2/TPBYs30mzrOZxPAtfSVcW5Zy5XM3SgfQFH/O1ejaU9bX0rHk98tinuppLVfRNlRu3C9bROY/rZ/y0BjZrzyeJ3mZT2INC8EXcSKsOaVorfiWwKTyeqK+rCZK0wGJwwf1sX9e3r2pRBFujRXSE6Hv/MzSt0JeKznYSjIWKd4evSBnaY01I1ST8NiMB7SfRzmleItyqUXLCwZgvmtZOJkX46Zi/+/bqTCSDIb/z2Bg0VOc+lvE/5GOd32ru+22OQnBdsjoGfcr3Att5w1MDEVDdGpgTlgiifMkwX5p1hQH59g8sWGM2V1jNbhkNIVz5U1zdei/H0I5AENpVpiDf+PmoI9kRYFkMIsFX4as1AU4RGaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Mon Dec 23 09:40:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 397DFE7718B for ; Mon, 23 Dec 2024 09:45:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C88DC6B0092; Mon, 23 Dec 2024 04:45:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C386C6B0095; Mon, 23 Dec 2024 04:45:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB3446B00A3; Mon, 23 Dec 2024 04:45:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8D0FF6B0092 for ; Mon, 23 Dec 2024 04:45:14 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 500801A111B for ; Mon, 23 Dec 2024 09:45:14 +0000 (UTC) X-FDA: 82925740128.28.29770AF Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf08.hostedemail.com (Postfix) with ESMTP id 3594E160013 for ; Mon, 23 Dec 2024 09:44:46 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=l4gSxPRh; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=qq4l+5Qn432t+J3AZZfr3yOOT8rh76isOc6aLGOpKeBE11XiViO2MTVP0i/3Ej1Ncw6TL5 YJNDqPmSffWV6DLLX5iWN7BIvza/OCSLU07Uy282f7yIbBRqEZ2fUX7fgHjbjmPjtAhilH HKuzPZp5RA7hxG+WIOSPrlu4sXU6ksI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947095; a=rsa-sha256; cv=none; b=r8wto/U6uFCF9I+2L0uyHnXlldK4LwCbrh4xsQQKGCaDV/iHPg3wqJi4zy/pYjKvKd9R2X q3CKbZxBrbC2paP6cqHRIm4+Q0CheGrYDkV29fP+m6Xt/rGSuH2A5Bi6Lxc1ML4yCDPRHu 79MjxLVoAr17uN0OuIAF2xnHrrnrsCE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=l4gSxPRh; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-725dc290c00so3781373b3a.0 for ; Mon, 23 Dec 2024 01:45:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947111; x=1735551911; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=l4gSxPRhZKC8nb3zwN5DsmqLMtJUCtiifkAUBdhfApgInNdbRtx6nMDE+afeUChcMN uuk8AtAiWvyM6xBr5Zlfa/8pg+frfCobNGa8mL4pWF44Rj1sX+Begs4BHGNH5YRwchqG b84+pRN1RTaV/za5UeXwnMe+bLHYg+M/XtMQRq/MtDFgZQHzd7+VveqzrXQcJYRYKQ5q LG4FgEJZfZtu5+NJiIHJlYIrIsiLzvtMmtUg9921ElBskbDPaXwhHswuG78RGgzglFr5 oPMU8swoCC0MrsK1wZaQDve1IOnR2EkUCkdSSDOxP+Epwf9VWvHsVQyBwYO8AFQZyjLC Ns6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947111; x=1735551911; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76YtbImY0jPwdXnQQVs9oYKl35yZZ81uc4A2YpilpOk=; b=KCfvq7u+ml+tZtJfXPMPV2BuzgXaMbw2fkk/9BaNX9AnKEX2/pUokvpB9NTrKd9v1n tCcCGQO3i3A/tXAbaPg8f5BTqonZ6X5siP7iJZmbQnGje0Lk9BoNj5jsBmCxBDyq9fS2 7sMbGsXwPAS/Of/amWRFYnz16cBHRcEXseNjRWXhIJQBSc9+ENRC+BuiDlLSpzCsS5/t EWM5Ys3z/FvhJmhl7jb2B0YRnxTmXGrDcXZPwxp4McUmh/7K8cTDym21jVV5Xt+9ah6w 99py6rPL9e9nSyfyrZbKJL1oaVoezUX392uqpd4Z2wBsqq0SBsOzcMpfR4r0y/7b4Mbi a0sA== X-Gm-Message-State: AOJu0YwpB90Y1ffPuPz0SkzRKPJxK92jAbPIR5e6+141/1KVFlqxi04N +D6AyX5lDxHv84ZU9Jy1WqTbBQJqaKtBSe+L4456VwH953RI9Pbqgd7jMPptCj4= X-Gm-Gg: ASbGnctPj4HWOz8tcDjndMShM+h4uix7haspTN3oTjPzdzyYRJaI6OCfurHXjPgyorY FJWJ7JgEnU/qpNdg8Zn6puuGpGmibBBJSwp+pJfEF5lUw+qaYRom99sAethsKJpyVVJaNxIsj/p IVYqRchX8IR8FDAFKZg5JkXZE0iP4JyCtzkQ6WEO5cK1QKq29AJY9Ti1ReEF/rtuSO1Kw2b1x5n mNFyvWXo0VB9+EpvRMf3pYxPcd7pdbh99/GYu7e9smE4gHyE0rwZqDAG1iMi9ia6gBMpHyTAkbI I0OReLH/P7m91fyJU+mW/A== X-Google-Smtp-Source: AGHT+IFO8x3oURIM+5JbbGjULQ8gSrXSXvX9ZJiJMgilE8gLJBNqFvKuKmavbyH4ASOcC8EYHQP7xQ== X-Received: by 2002:a05:6a20:1593:b0:1e1:ffec:b1a9 with SMTP id adf61e73a8af0-1e5c6ec6f11mr24851906637.3.1734947111156; Mon, 23 Dec 2024 01:45:11 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.44.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:10 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 10/17] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:56 +0800 Message-Id: <0e8f0b3835c15e99145e0006ac1020ae45a2b166.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: gupgethoicbsx7hbamzkuaxxpdgrcgdo X-Rspamd-Queue-Id: 3594E160013 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734947086-228854 X-HE-Meta: U2FsdGVkX18MzK12J9c4wT8A9afcgngn1JukSuhboZR3Tm3WiYjPrarL6jI4XM1nnNFWUEEe98vv3Er8K5+ye0HUV483mNfojyHDpxs6oU8Uaws6vkatMN5aWn4UAhLUr8PlEm61stgFJlYsHGH1DmcBFBlsbE9o8vO4ZJq0WKmyxOVMDw5fCcWi6eadhDnb6TRNRskYXEhHcLWMU2QU7tkJfSDs/aWVEUXLBYKhBJANU6wOgHidUpaFIR3Ots767b3k9Y3Vtu67qqBgjI509/9jsAdlc7X1r+QXVPdUDNA23KXLMINw+OfXm4G6MCGEfMto3pW44uzaH/cUpWYJ/K0e+2+2kHqkZdBsqrC1R4NRYwTZKa/peAQbskId/0d5AM/0EUtHiTWmK3rmyQpMkMFcL/kSY32/azRHv9ZonQGgv0FNd6OCxWmbZsHb7y6ciq5oCYQo3S+7AyTw6rR+xDHMQaadN6gMKsakBqSZgO2BlUXU4EUeI+Ui3ASz/IA01qFCe/5F6ODDqwUZ4NFmApdoCV79wCehu0DJIoFsuyRdeQ461Sgk6iOJOG+/7apJ81Me2svzuAFagR1X30jRxyvVfoQ/y7ZQH4UB+KruqkB2ScguLU65XLg6g7jBoU+cXPCimA9gFA9FnMtzNtL8jltzqybNwHkNU0uL2ha2olZk5ohvqp+sM5oBl7bE5C4pTfOx1STF3k5d66y+9WXXHCHZ3HeESqrEu7htOotzzD1AwDEsttXuJijQt0gFtNaTaENeu52zWuaTsFeUnQP7U9DxL1vF/keo8NovLO+F0MoZU1hgfi3ClYUrQwKmVXPGhQT173G5IPwQhYL1TcoUElJXqwks5GPIkdaDfOLKsRuzjJGYxSTMcdqSUt4NHABCYpNYXWyT88Q7mKFnIIOzCaptpo0WT0/C8jZNjBAfWXeOhnMZdsLjMYud+lYB5jP/iSFohoaIzdnPt3tdLxv asJDbl76 jKd8gWdgTGj0lGEEHO+kTgVpide0PEmTRVBlA787BJQ3UWASLGR94lrziF/b3e4rWeJ2B4L3AmDxxQK5/d+Ull6w8z5OLxY6inwlnSeARSl2I1CxwL86hFW16zwtEOaUegnYd9SMqAeOi5QJdESmlJViGqLZ8kHZ98YoWnXuJVdeVat1lNkA6WvN/XT6zsXiIGiL/7UW7bYSIr9nB0SMT4rilCbEdJF1+of6wgj79tr5XmL5H3lqdi/c2qO6nVCYYA9DEgjGw2+Nm7yHJLbQ43PfIEI+w1NgT/F+S53iDUMNdBm7Lw1Va/9BHTP5vqadODdCAl1g7oPh36Ceade+o45YnWHH7qW4NqCGkVDWSzi0+Gl7GjMe1zA+GuZXqR5WkBanjlI5l2+LhkQg5JEx8vHs1xEas+T7nczEG1boX23zdIFrNqlrhlpWvRU1drBqI4Zh3cFgkdcMxyBUHLcVlrOROM+XjFK7ARpo0FwVSSyITZpk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Mon Dec 23 09:40:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 079E0E7718D for ; Mon, 23 Dec 2024 09:45:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92DCB6B00A5; Mon, 23 Dec 2024 04:45:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DD406B00A6; Mon, 23 Dec 2024 04:45:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 730466B00A7; Mon, 23 Dec 2024 04:45:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 51E046B00A5 for ; Mon, 23 Dec 2024 04:45:27 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1E76CA1072 for ; Mon, 23 Dec 2024 09:45:27 +0000 (UTC) X-FDA: 82925739876.23.82CB794 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf03.hostedemail.com (Postfix) with ESMTP id 0DE2F20008 for ; Mon, 23 Dec 2024 09:45:07 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=GFu7kHaj; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=PKXmgk14q5iTVVS1mr3/Vjb82WJ7YhIS4Il9PFOabEQl4bAUzrZpNKIqgP3BPmhbR7fAki 8hn3D8a3mx3gHFTytij230nMH2sset+u6YoFlS7wQKwT+C3v6ccFoBR1odMKHqKzhkq5HY X1wGtPjGMSPiVY/bEAGDFaVkEtiDUnw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=GFu7kHaj; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947096; a=rsa-sha256; cv=none; b=3RduaYYs/f9WWj9Y5hMTC+gvMddMaqpG3WuzjJ+usdGniDYarDJI0SDD5wrV1q4GrhoRax o38BcgNBFA0D+LNC4ZOIwL/KmpXPIYPwypysH203xoaKpluB85x3IS2EitsC/yrUTWSE4P D6PgeYEnc/p1LbU7qg2gzacRlRg7x5I= Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-728ea1573c0so3238072b3a.0 for ; Mon, 23 Dec 2024 01:45:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947124; x=1735551924; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=GFu7kHajqALzYZmCPeeZofS7k7vAFvU3YWQlwA5tveCAJyya/z8uEvbg4/Z0JB/uO0 ipVnL9No+pIKbU5uwCQQ3lnBRY5G4pKxwTmYx6G4OHfoq7wPwiD5LAT6PspqMFnO0Ixx uJmvtRpDw3lSxTIVthub7y02in94CL2s2MqGPqP8fMT1FOWNpUW9CNOorBfTINn2v8kg JdZcrhKkZYDkdEnyyDotZs1YoZSgT0J6syBGHMSbQ0fkmk/BV1Eh0xFsSi1X3QTylW8a /31BfEzl8Bf7Wce5MhbPiI8WZTkzoJSMy+O9kUghMKQlWAR3fhWzAtjsGEcAoQmPAYQD TERw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947124; x=1735551924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rs6Lkouarv58ZoKb+WtHLVo9G63kikGgJGrD7ursbxc=; b=rfJvtKeArJuKO80gUS7GCKoqgXl5iiaqYtMAw8iip3yb6JInyk/9YQ3xk0yK4WQYXe 54U/gY74UnrfBuX7QvUWOElvOFquzrs9GvEiF/rqvkrh4WB+CW5RU6iM+OanO4TXf4qS NpIEvmJBggkybqneJPn4mTQ+j6LNhZg2g4dzb6PpJcZX22doVnosEuPprv1cMemmpg9E C0AwzCyzzTmRNZIPyPPiNKVjVrMyGvFyAyEmBsuxyGuxkhGYjdkpjONVUeEv7jq90t9d fG/LW+LhQL2JOST6+WXJduvylc+E94wYfu8myFaIJHG+yb3rPSdE5I74ohb2pz3MhOvb 3bdQ== X-Gm-Message-State: AOJu0YzDS8Wmd3Yd0zGNSJ4UlQLBNAVVe6uWyZMgKxISMvxo0gxOPcbt 5ACvC6dm0f0VOOTvOmOERwC4zD/hpl7hQKeqLjMblC2oEDdoD3Sgy0gcLD89T8Q= X-Gm-Gg: ASbGncuZD8HA7eVzaJz/yf6AI1C9QQEYYl+1CTIgsQkxddA7u0PcHxKrPAMWb7FTAQv idvEX31rp2W3nyBXf84CdMI6W6iNDbxnZ7L4uT3X6PIrtmOfF2OWiazuYzcypASgGtvLKkOWzbY BhILjSpsNED60NGylAJK5eceoeEm41Nngk4wwysS6iJHX2F8WVT2YZPLrZcu4B7BpjHkGz09rme zLqGNMgQ00pMFG+YV1mg/saEXzsjpU05uMRXDjQ+bfD3aUDi9mWpHyfJCCaaDxJ6WC4OTOtelyk BLNCAaV9sCrxI0TUA2simQ== X-Google-Smtp-Source: AGHT+IEsUkA9RBo0Vmvp6m94/gMVFQ4PHl6SoN42Z+rLx+q01B5oEKa1sTTPOWHs1mFCNObFPbuRxA== X-Received: by 2002:a05:6a20:6a25:b0:1d9:18af:d150 with SMTP id adf61e73a8af0-1e5e05a9e39mr19676430637.21.1734947123919; Mon, 23 Dec 2024 01:45:23 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:23 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 11/17] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:57 +0800 Message-Id: <0dc5a3bf5a692e24379c1d3b879a6d4396f0dbbd.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 0DE2F20008 X-Rspamd-Server: rspam12 X-Stat-Signature: 1reohge6eff48fyyzodieuqfxwucc9i8 X-Rspam-User: X-HE-Tag: 1734947107-361413 X-HE-Meta: U2FsdGVkX19jsjEtu9GXvwqIt1/kXcfdxZjotbebG8MS9kToSKKKvdFmuV0j/wsFTeLNn4He+6eXmXCHDHSnSIQTp3FD6PdzP8yNxoiqcp5M1KNJOl+saUtts1WaHWG9Y/B5V9D0ftbkDtZRWeSOB4/C6aMweUhM7DCEk0xPW0qcj4DujSzdQuwPTNPzjfUJURWuMeCa/xz0Ww/LAo5PMiTRhv3o7HHDdgdeKyEbN5nmuZs5yZDqoyBUarzynlHAO4+OUIveRjVRq22vY83gqF9I8PsmkoNKlsPTV78ZIa2d195EOIY130f4VOJ4Nt8qJCx08a65FoT+GnWRg+13Zh6IKRKkOsUslcl6cOj9qub8Th4wxnBo+bhJORrHnd7+r9kMeL7bLsqA26gVdOp5dSRCPtBgHr06Ye+k1WQ5l9VuHYa1tY3fiGisckxC40ak7lAI5nNCUnT8t+y0odS4tcJTsVihD4gi1te+lBomJlGmpHfUqsbbM9xeiDzUB+xzSRyozhyLrCsjAaHO+zenD/Rd3v+lBE2xchI8eRCpGDx5rlQw4BqfLrz9I+uvFS/CMQJMn2lnJp9Mv6+uMzcyKbyTzy0wGmMWJhOpLPtTGvgfLoyFZEUzmAY4gws1kn8X62zU4PnFhGQz/ryPK4SZ2jJ1XPDqLGB+7BYw6oKYJli8k+UaiHI2iAWFuJYlSLdj4edS3jybqL3gavS/kF8C0aZINVZuq0ZV+VLblt2YF6nu1U25CyqhFwyIwmm9oLRl6ybjZMHWfwBoL3fHTel2nK3utUC1htH/cE5NH6sk9X0qqBl9rRENBHjP7Ga1VBrECdEwUT0dnwI1UdVtul+5y3wWhAHrqZGC82eDvYeKjJ/9vaj/JDZzYew1/3IXlc6Uq+I97Oa0L+GDjSqnFcRbbE3+Oagr4k+TAqDaWjDiUEZ2yzLZSvLXUnO2mGd4QVXppVn2ZAJn8AXBqxHc0SA DgP3Dd3w HuH6ybT/exdbIjWNF0oeyD+EbF3uiobiiaKkgjVIQPEODfmiUU6rSTewVjEDBtKqm0VM57F/bA8W9N7HCCbtUixonacaWdU6DU1tqbyqIyOcglm44lFL3SQ5yILOyndbdCWsPFWkRpTfxHFKY2GCe5tVqrMB5YT1gyF6AMlYI1pZu1Z8Sokoe1l9AgER4XRhTyCRsaP/akqZPbAM5J/9ODYbGYn9/Z5JOvVlgBnbvnhSTYVW+LYi1kUPcxlU/u8r52cHoGJdaZiPsI1w71rZLppBukXlWmFTcnD13FgiSC2voADLJhLDecHqu0ufiGxGg3622Zx/NJUm8L04qz/Jen7rOLB7zvssTuldXQe5Gq5WqOIMkmccPc6faF2DYOilaMWRArMyXxQL0nv8EImVYpDlz/JTiq+9mKZqY54Pre508LU5eeMVF8NioNcx4T7yBov6LAENgfncQht/HfE+iDh1CkVn08nn3Q45RBo2E7sQDtAOnJGJcxyo9bYFuxNbvpMxq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } From patchwork Mon Dec 23 09:40:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADBA9E7718B for ; Mon, 23 Dec 2024 09:45:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44A2E6B00A7; Mon, 23 Dec 2024 04:45:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FB5C6B00A8; Mon, 23 Dec 2024 04:45:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24C9C6B00A9; Mon, 23 Dec 2024 04:45:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 043306B00A7 for ; Mon, 23 Dec 2024 04:45:39 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9FA5B140CE6 for ; Mon, 23 Dec 2024 09:45:39 +0000 (UTC) X-FDA: 82925740422.30.A74A28C Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf24.hostedemail.com (Postfix) with ESMTP id D2AA018000F for ; Mon, 23 Dec 2024 09:45:32 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hx2+FlXC; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947120; a=rsa-sha256; cv=none; b=wTjtio90SQUo1e8k6NU22B0ZJk8xRIhUeUXbKcsJxxsoiJ88a0sB8aueiCSHwVM/C6o6Fy mSXq9rMiPWciapSVyrsN/xMMFQ+G6BLVGge1Co/IAP14kxRbRvEqvstwhlXPhRU0DVj8mE ZuyY2Tfg6Q/l+wx6vvawETRrkUwdzn8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hx2+FlXC; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947120; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=OzGT7C4qX0qKiTubMHW7pxuVQ7tQ16QeB8BywbWUCCyd/WZGFVdbYJCXvka7QclykYBdbT y18rIWAGCDWCkEl4yaAvhd7lNpHLSN1460FfvjlQpR+1GM6zkqdogLd10lT7XSS3SkGBYP aaaOIyHU4ZI3Tnx5rScwja/M46Hzl58= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-725ce7b82cbso4567427b3a.0 for ; Mon, 23 Dec 2024 01:45:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947136; x=1735551936; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=hx2+FlXC2WLX5GBGqSRmOW2p+SumbSiwoYStpELSD8Ebp86Gwebjlxfic8QkmEHxvM 4zCQt1KcGBLqD8krcd5TcSHceMjq99hA0uVQUnBZENuAs/sUtGtEkBD6fkobnwr0j/6D QYk994q0URrOWLn+pw6wiGEAg+zjsGVUEugWQg4QMOEaMtWBOdwg5KwuVGOIxA6l9+Ob RZ6pLvYhnn7erW57xpZNcXOvAjsI0pqXvSxIP/jt97SPVocbm1qcIVoKU7lkapWxng2K HNXB6Dpo80nDIF1I0fncU43mocZdo7BVpjf4XBg6Qxm1uSEsiPwVKQVSiaa/Sq4srW0Z FW/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947136; x=1735551936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jyEzElX2/Cl2Hb0vACePBo9dDp7jwROhL3d0Mq9f7ig=; b=YKyKle0i8hesXDGzmSvxmpVRxBT6ijStQbTG4rIWHg7vc/k7ApnNr2BfoLd3trZtnh XsbHTz6DCqqQHM6wd/RiYfNHei1YFOLo851MKs0fqbsCJdXrHKFK/cLpSsp4RIk6Sm9D 3XHAbXxvx/vwPVrsolRvvv/o2zmWHzGcf9B+4q96prPLqyMG9gD9nWf2PgK1nHkyBVsR iKAw4hfwixQ9Rc0IPFCsnXbhG3lKvs2V9MXAEd2ess1ZDrKQOT0IdXiybA0EuWe6SHRw mR71+ao4X4IHixq2/YnFatIYFTmwoLHB1iXG6h31GisZTfYRIqJFTrN8hGMI5Z+O/Kf2 fBDA== X-Gm-Message-State: AOJu0YyxICnanPGxspW0QGIjgd25SWcxyZXPAw27XiK5OhaFNM+Hm3fc Ls0vsx+g9/rrf4tQOQ+4qoqB2QqKyk/y6e0uDtIUIGwFEUS9yiKyGsE3pJx8EKg= X-Gm-Gg: ASbGnct9fGjPQ18ouhlXhPbu1MkJLy6glBrLvDK1VZV8a7iCpavU8kJtZiCU3Kjth8R 0mOEI0JOqq23A5KqF/N9XCFzrQ9jT/T5zhL5ZDswQCXZsTQKeg23TFFj57t5Cr/EJ98dx6TKiqQ UAMYwTCdjrgvcpb2jkSQQuggnqXRxAStx77Eo54nu7DrWwfhb0ooh8IsGqqWTFFzdO9d6Wuglu6 vYsJcVJnYzmhH1uvUZhTaNXhYscFvTapziy3MUJJFzuLT0GPsHWcbn38jREum6FwiOq8OG3UxuF Z/WwIn79tgw1wEn6XLC+6A== X-Google-Smtp-Source: AGHT+IFFxsa0Y8D8QMIcsR8+VA1yaCfovsoDpuS31clip0UDjebZ+1TP8q/TeJDt5eTrdc0wzKH4iQ== X-Received: by 2002:a05:6a00:399a:b0:725:b7dd:e668 with SMTP id d2e1a72fcca58-72abdebb868mr13954007b3a.17.1734947136545; Mon, 23 Dec 2024 01:45:36 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 12/17] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D2AA018000F X-Stat-Signature: mpj38i5jdboefnmoistx38oign4g8s4d X-HE-Tag: 1734947132-888765 X-HE-Meta: U2FsdGVkX1/NVw3sDPiZAHfZkDi+9agWXsSINfW1Tn0bRdNIH1fK/sMs/UDs7qny1647JizU7L1sECg7ei31lJxkEnhtMz5gjW+68ctPNGBMMwGQ1YMw52kZV9rx9JssxHAiLxT4jeqauCGxp+gfyIynCCUh6ddKLZyZO8JEdCh0U12drnS2kheUesOXr48Q5izFTG/wd5tXrC3V1gYFil1WDtIL9CCsxGkrMr1SZXmEFSqtcsFVjs/A34Dl0IZfhB133DlZkUQSC936cY6UHno6TjmS71TdyQkEvXUnwyNOsAnDi5LiwTcC3144CWrsAA+nRSPHUxF+JwKwwGF9tAX8kWfrxqyCIMZoRx7jSyhgo6kO33q+KOf4cVQeVLcnIuJhCyCZaXqUHDwAD+705M26ZnjeLPXxM6VEMNZMWo23vDYfOOLPoa6p8ykjlzD7GHkP57c8yU503Par7tL39NpV8T6/fojYJNVjzxauCEwfrdM/UbRQ3n7FHkdBFf1mVfGIZDgMhIL2sn82kPt1HqgEYRXutKGJWIsTZbZlQiHptcytFq08GYgFK3pRG83DPxoUsjJBnvAkmquQux+ZBlOU6Hj6+qj7XvjGc7U53VA40DTgcjURun+sAOlXAWSQ+7PdHQljtfk7U049qcjLorRRxVxZys8hcq9e55aJS2OP6y0ckE93yMiSB44Cw2GBBeqOtyVj7KcXAnLcrkmYjv3kBsjSay/pZwoqqPYKnlWfBRNDUR1ptkQCOSi8ve26HVfzBdIFHujyBBdZq9Q62NYrTKsyVcx5y1D3vgSmQuupkgeO/RotprGXDztrA8feL2ScdGaKUVguK6FD6u5vmgPLdgxS2OKeD7GJOtM8ishGXC7sw0ZAHs618FsINAsbhizWdn7hFSi8hIj6ppGmXf3nkwl/3Ya1z0Ge0uW+iwhynvMNTBwNBIWsQhBb+VD+v7ZpugtliuTZ5qjrqmy 5n8BaU7f nvQpwr4aGTtwY/dxyvYE80coPwOCbTvynBvsFDu7GruCsTHYXjKYm1oT+3pAa9KhiThtXQZLs22kDBd0Tl/Z8MigDHhIOPNHymHpC40nSdYT3QZp1at1e7tkDv4e8NplAhPFmhsQ9Jgv1RObGeN7HsmKVVV06CmSiLALU4cq6TxFe4A3Vb4SNFpA7EfD+RGLmDeacr79ppgy9w0ttRw+nKIeRrVDe5+mr8iyFXRPseZwkB9vx25ZEuEJl7hFpYJQtbSey5pLJ60UyXprw3CejHL71G+qdh1+ogl7t3Es05TWdMzMWJPfpvxlofjFKuSjDfMBpePWfOUqkC7bn/x+tpZSTyOvu8p8tidRb59ecCLPsrHps3PzxrY5NG6cDfHjMA8uWVpzVzf6FMLM4Lg++m/P8WOK9OIU3yYpKd6k0Pd6/yr4LMfI5FcirjKeV1RlsAU8SfqnF+oTSE3hv/kMO+voaJB8+PmBhrXvhGEYcJCzvdWAtBBYUYVMHtMeFZA0jY4vS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To unify the PxD and PTE TLB free path, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Mon Dec 23 09:40:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C2FAE7718E for ; Mon, 23 Dec 2024 09:45:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02C2C6B00A9; Mon, 23 Dec 2024 04:45:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F1E036B00AA; Mon, 23 Dec 2024 04:45:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D705D6B00AB; Mon, 23 Dec 2024 04:45:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B81666B00A9 for ; Mon, 23 Dec 2024 04:45:52 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 73575B0E4E for ; Mon, 23 Dec 2024 09:45:52 +0000 (UTC) X-FDA: 82925741304.05.552E6D0 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf16.hostedemail.com (Postfix) with ESMTP id 08F70180005 for ; Mon, 23 Dec 2024 09:45:11 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j6AcRjsT; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=cdquzrE/2FidJLZM0Nv8p6+Nks5h0y4ogRPz22Ck/xTA+rypYWe6NrmvcmghVjq8DwP4Ce DXCA9hc04Feq/0zQRBp9Q7zDwGavNXtHx6M5jL5EXgV3xEeRg1W3ruLMtWk9f7WQ4dBnqN O1jzYo15RgfOxMQ2rfWQFCejVVBAuAY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947107; a=rsa-sha256; cv=none; b=HSZesyqxNkUnU9Hcf1Kc02uMJNxuw+sFmpB/s0d7l/E6rkQKgy1k6v2TDsZON7xDQ8SHot aUD062Ph5r9T9l9QnV8EI6o65aQzyLzJZgv7VILh/jP8ioPHdhX58PLjdTHwvYGkRjJmgP oQsK4lqjsggtsnP9z5Fxc+Vp2oT1h0E= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=j6AcRjsT; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-725abf74334so3355155b3a.3 for ; Mon, 23 Dec 2024 01:45:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947149; x=1735551949; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=j6AcRjsTn9rFqV2PiJleqg5FPeoMP/eDhe0W07PJvVLYIOgcpM8cIvWAAOxCwkWmZK aQOkp+kqqPVpV2BnfYlt4ASt7UBokbJ0XH1qbeRfTg3xW/bwQOCey8nyLEkfP8qVPFcD 6u/DFiCbQPm87df/VyRrzK1M3gM5aWXTZnpnUt08ppN5cGpB5MO4ZFQpYyOlUtGit+2j G+GgBuVkZLDPG+bmlc1m0EmbIsm11C/YfOSbJm/g1ry5lFprSmQhbRIWZC6scCoPu4wA EQYfLmqyyq82CBaW8t71Yv4Dt/QzsKYeCE01/rkUdy9Nj6/9CRAB1VzeaOpNxqECi84W j/UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947149; x=1735551949; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m/+0uIvp7xUeTeFwO63TH1y++vtASeDM7X35qnuktEM=; b=HNJhqQqqgbb02mxdCGKtkKoeGDOWaEk2kFf7idOGrKwTUGDAy2IYQXFEmzs5uvRhx9 MshKB2oytK+CHP8Ob/nJhSKN9nhtHtxX7vwNSDsTrKSeia8KDkJPvDRItZS6IPq49Qjz F1leCu9mh5jSKQy/xKc+MgK/f2PxuLymuIV+HUG4udXiE9Uk1zh9QOkjeCsGRZ1DOUMd EhMOm77aXNAcIoaYhExkl8hoiJV8imrNc+2e6HRgILHssMkg5PCh0ETsIFtaddU66Uzm QWn9dpFWV6ijyS0s3u328Y1U4trzSARCEuDMFIjGqRCpBCai8ftL2/OT/z4MgehGLJ3D 7Wpg== X-Gm-Message-State: AOJu0Yyexh9TRg8ux3SC7UX66xOUxZRBLDIbr0mE1dmDEIhNXYD8EqTZ 7/SxZ+ZdxPGEwxJ3/ztFjkBW/eIUNlWQEOpeI3hu5n6IvS5Sp5ny/JB9JAYvpv4= X-Gm-Gg: ASbGncvvvLYA/VBfJdg6AX9b3EIiepGv/HaOJhQPLmCQUryKpLrHlM4HuaBUMfz/JGm JZdQe6ir9xkp1eylBfeEOsmVQ6OJRUjak8r+V6va+d2SYHFNvUigzS/9l75NfuZSFKBh1gUPqBm jIJ8le0Vt6EZXnG5cVuZQV/OuisHSgJH0L4wbayywMQKMCp6NyiaGisMtQAFepO89/YaiNOzj1T CxpIJICLQT2UuvwTtfIUvGnk6nPH3BMK8cKpTma1+lkhDUkzNmoY6jsdQcYXlyE4HSsXRAKn+fb SpCbolDeW0JjYJIrh7GFOw== X-Google-Smtp-Source: AGHT+IHlutX536vagq2BMCvNjirBm1uJlF1dzaH3GgOc0ZHj3lAXm7kK/AuGYqDlsmkXqpl9hOqJ3g== X-Received: by 2002:a05:6a00:92a4:b0:728:e1f9:b680 with SMTP id d2e1a72fcca58-72abdd7ac89mr17103004b3a.6.1734947149240; Mon, 23 Dec 2024 01:45:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:45:48 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 13/17] mm: pgtable: introduce generic __tlb_remove_table() Date: Mon, 23 Dec 2024 17:40:59 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 08F70180005 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: y4qdgguiu6tz8a4zzs3fza3gx19c9rxy X-HE-Tag: 1734947111-469715 X-HE-Meta: U2FsdGVkX1/XT2GGsRy9gwTnO5UVAUO4Ax5neAYgsyUXN1OEFPTDfyhRRus+NNMnfN2f4xlmk0WZQseFqCliU/hnCd/SugI12sfpkEJiueCOa5/+cuUKv/y17QuxO0Lj0SK8cOASorVeG7QP2oZsi51FF0YZoBAJTmwWIAnaM6w+sZnja626ppmoEf4xA2pjchqsdKVEOcChjV43R5n2Bl3TRRahUkakfvRqQ1PLczG13vJege1IUVOokpb8V4CDXgBdsKbT9DZTWJt/BwEAcH8omQj3G9DWDRrMQxe+zO0CvPqHel8mxxF0OXICsikgQglbdCVQMD4E6sK4y9yYd+Mm5P4qnrvs/L4rGB14pGQVGWliWnW68Wa4aKDxV0a8aUpaH6ZmJZ7Mqnd74EBwSRo+K3ddV9NIBjn0OvfKu96Q1+gaOrEbq0ThvqDUrWyP03J4QZjYxPwt3Y80xhk2D3ayfyTjaF5bQn/QjjQuWEPmOsUbM/ziQlsy8AGRb4seIoOjtqk77VCp1YfMm9mYhhlJmRGuIx8rEbsXuil3w/ANasNA6Ko/jcJ6o1DHM1PCEg51GTZ7xiGYRLEZLhDv89vOTTzXSquEu9vIrP0/1hgYEFDvp8Wwb5xRXLZ6JhTzDETcz8V15WELPztXzbPPfkMNPl6haruxgo8hk4jj3notIU1UK1b/qqPHsSjPAxaGstaR+vt4PkNEkTI0yPGjXjCNx0JYoSaVggVbYFVoeLfvax6XBwG9vwl5tEIlhVEa3/AJVXtPWnTR7ZjTOgFydmGjUy/4exS+odGZ2n5jjEUQrs3JsxPJTi5cvKy6FoG56ZsBCEltHESU5P2aAYSMqBWwjLnXVSFV6jMXi7+92w/+wTLaWcLs6jx7lKcJ/pBNgRFkm/8pJxmwpt+hQXEXJdNpi26bRNMp4cDxp11gHE6yOJnjxGqvTjOiZIpdoAH38f8DcQCbzO1tTmj3fRd fFzoDCyZ SGm2OM0E+CIhr9FMskBf564KxDxnEyk88JT0WdD36mH0NQfTccyjYxE40ddLd8tHS135980n491DNgjXcnXh4wpD1H4ga3z3/buOZAlpf9pdudoVMS1RO06UGTi2mSj/NYmzZ8kQUOLaRqMMoBpuBLNcKcSHZ5jzGw64WmIyxBUYZd8mBUO6IdIqZDq/m/C8CPqGOk4cjWlfOp02FyHo37Z6i+7/bAWbx4KU7or3X80lhilp6QrW6rem2iRZsB4ouo3sFF/e16j1Xn6alqlv1uLjAU09Y31DDYwmhULBTdbtdxWVFBx0EEKzPRz2tDjNj0AIek4phCzO7hmi0jTa85QTtrQAIQoMRn9M3bMWlvD9dAAUicUmeJoAzhVUweD5txs1zvxUlJNqQn9Zca1fDvL7CDJQ+xgIaKkMK5qfnp4lk8RFm2JV8SghxQ2wV6mxvFzHCCNiDG+K5nG9rD+AanEL18A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 20 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..da4a7d175f69c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } /* @@ -124,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } /* @@ -142,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* _S390_TLB_H */ diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_32.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Mon Dec 23 09:41:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E249DE7718E for ; Mon, 23 Dec 2024 09:46:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 694AB6B00AB; Mon, 23 Dec 2024 04:46:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6457D6B00AC; Mon, 23 Dec 2024 04:46:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50C086B00AD; Mon, 23 Dec 2024 04:46:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2F1CE6B00AB for ; Mon, 23 Dec 2024 04:46:05 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DD1501A1532 for ; Mon, 23 Dec 2024 09:46:04 +0000 (UTC) X-FDA: 82925741472.23.DEA2131 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf03.hostedemail.com (Postfix) with ESMTP id D8DD020004 for ; Mon, 23 Dec 2024 09:45:45 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QAuqMUNh; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947145; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=PU/F9WXmLQtMOL4qjE/djBfAGfMfwhWqWhJYMfBii/mZoVDHEh6VfOe2ouSTUZBunDl5r5 NlHG/AzxhbHnqVtwPssQdma3EcZ3TJrX5zcW3I6ZLxO/cIUdjKZtZ+am7CeXLIsZ8LghP6 M+VQJFDR13daD6YJkX3COMMk+MSIyrU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947145; a=rsa-sha256; cv=none; b=kdiHxFdkyok5YwYRWpLdTGFfn3H+moS10zPIfq3Y5BxwWioXnD454PuvkXx/uUOF8FDLzH bK9E7NdR6cuJccgzk6L5P6S/EytwEFXUK5fPFUJVs4yZ+ppuHQ2d6t904yWm/XeBHyLoez QgWf8xkLMAdoW/HODdK3l5wKkj/8phg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QAuqMUNh; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fd581c2bf4so2341689a12.3 for ; Mon, 23 Dec 2024 01:46:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947162; x=1735551962; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=QAuqMUNhGySltT09LXbUEasrpKS5CXBYDkZImn/LIOtWbdu5ZBsmyiMF/3Jx0YDhng 6D0Sa13QGgc00Wr23rb536lPFB3knuvYnWLWJ6g8xSgIN5rZFqI4uG/6OgI1w3LZ8iMA N1OTsBgLfpEzW1luXZyXHxxl9M+9HpZH/f4GtOZcM96O0q5DljK0iBBh6b3JDyAvIWeK 3xiY2R8AX+T9eZkS94GiILdSrrhNSnCMXsIuoAveS6C3Cc+bNP0Tv0bdJXPvvEilzpIT D1LyNfevBIcoVUkaWqmiFxE/9MiQu4Kc2MhxMN8Hs/xjmdY6KielkevtLMZflUIhioFW l39A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947162; x=1735551962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=K3yMJpdv0rs5+k3iHlnzz1RJEA3S1VyEMQLLWXiV+knGq+EdDQMNdfOxAo32Ej4EZB qCW51bOY18HeGwtc+zjARY9w9tPs4msqSWlaWIZz0WrrmZOiMkaJ/azMgcL1WHtUk4Yf r4cMoFnU29Dn6QLTSe1a2ClQV6z/Kyxcl758/5Q8oMZ9btaYe1ClQWJ+f+xgy8VMr8Wf YtwnRpo5RjjvRk3JmDUdueckptDELI8bf1GuBLVhuJHt3YHmPNB14r/A3rLT0qkwidB0 GJcT+3B1tfCHQEPRBecOgUH4DaZNp2xq/UsWmvSwTLnlBjrx2k1J1I23w2bYAGl3YSwb N+NQ== X-Gm-Message-State: AOJu0YyC7lPCs5C4AX3bWrOg4pG7NFhYNmxlNn8DW8/c5TzVKlhkLIJO lQevQtaN1h35FlZ3MDVfzROMHUwjbDhQNvS2Rz/iuXEJLT5PUGPTew611TLIpx0= X-Gm-Gg: ASbGnctoKDPQjiw+Da7+y2GLTt4+OZGzvJg5sEtXueyIFs5ZohidqWO8tGcDTFRBF/A gunDp82MtwwwaAQB/kK7s6IGfUCwyIB3pxMZmtpAqNLJHJBRTC4zOD9sirCx7YJlwY0gUqDFTRW 41B5XYz/erzm9Q/xWLtjTmGYAfuve58ZcCRH396ig+3mHziBpUttjmdlqn+mWiSXQ/vY6fxMDCI vElOhXzrYPJF4HcBjeLBvHN9piKpFcPdiNDnwVONCZeO0Lhs2cLgFWXtfWon3ApS/8aif6ya1Dx p0DPc0/xhrl3ZEtNvLVyqg== X-Google-Smtp-Source: AGHT+IEiKQiNoCztX+vEmXJ78+SVJHcoRjQcReSYKDpgexw8hG6RQW+fGx0o5GN+OmUz8JwpFgmgkg== X-Received: by 2002:a05:6a20:c88f:b0:1d5:10d6:92b9 with SMTP id adf61e73a8af0-1e5e07f88demr18787949637.30.1734947161832; Mon, 23 Dec 2024 01:46:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.45.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:01 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 14/17] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Mon, 23 Dec 2024 17:41:00 +0800 Message-Id: <286e9777dd266dc610de20120fae453b84d3a868.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D8DD020004 X-Stat-Signature: f1frtzgfcupwoi99tmspq7j4odikinwz X-Rspam-User: X-HE-Tag: 1734947145-560866 X-HE-Meta: U2FsdGVkX180TbTQJamGJL0D+Z68RiDdcE7Zn/KNZliakVvv5mPTBysjYEVmQek2RqGxEGkenZX7ZKq5clD7U8nSuNSSYpjPMNSDvwbDRLD/0ke7r5HRs7zhI3+6EpGKI1BSZgnOe95s4gTnycAkEBSdPezg1Vg0d239Ul/pCnPp8CSi60uZ42iFzWKL77dpckb4BNwqh98G8Zk+DpRjt4f8RwzmU9g4MeMPcWvQ32OZRsHWSgDJLu5i06W/n0M7KhYN0F7PUmuBfZSZoEPMGmb3B2W9UhCH8U1+LaVw1ZeSJAdHCYeUdwZX4578GXNiQzsSSkGa8tJ+UK5rw6kxypr1Th0TsL8O0qBI/LKoviN57uUqQjUVhF+rWLAw1bOiHJatu8YacmavBej3DiJMNU19CUCPPqmfcFoBkWdZMaWbqBezjFE00ZmnmI1IRHlw79cebTr+TnzV2aKTluedHjM2ag1zL7NZzlJ+gKqBvIBe0dqnLTFSIbGkKMWVAJIdGcPry/K+c6KBnLCfPtRzbiJYlr7QyrIORwLH6p5yhOsR1Ztju67l97X0UUrTiGQS9ZYgXXabOtqzPNNZKn9bM8ZkSis3z64FMbBFeZaMukUX5CeC5yf2jXZshJBR2DhS9vxQLtzCPLRBtE1WrwI0NSHQSSqeGQcDL4Wx+jFPSQGi52pGlO6Bz4jaycPIefmIyB7oTYANN7U6IKalCsnyytVBN4m4BKztN/7GAQdfB2ko5nXFhf3F0vNTKQgn/I/wFw8nn4hO68mMrwByJyal6J9adSHmXQ62yZWyxkOQy4awfKdsAKOzOuO6JmH3UB4RHJAhPbA1T2XQ2udaGQN4NNrCrIqZQolyOBM6pbE8ypNkBoEmqlP9WfPbvJRWOdOBqLwXdWqmnvElAy6EXGPtauvpeUA10raDbzALFvBtq2Ducq8nb6FHhkCI86kLy4QK2ZD2QPpGpB3+o0OJpBy bB2TivdQ klCw5GjCo78zH5tlKEasmQvnR2eNNt8D4qvAGjpbnicgAydonpH9zSfAO00cV2uI8WwFw4eNUE06w+aP1uAoObJTNPq5uXacqz3vTOPkvwhL6AkrgJVk3kiGefrFnr6U850myxXVAsAaIq0Mw+RblfT0DD2Wxx3QGtPDGrQgxcte2bP4oZQuGuCJRKc4ba+BzevmTgqgdiBkUQW4oPfdrTzJN83rxJ70ldQb+6+L4QrkuSdCSx7T9ClRhrjA4Utkhow+9lwkgVqINIm9ksfSFTbwFjkpyq6PcX2MX2mTr7jti9XqKee5L2h91W4lBVFoX7AADpT68BuwAw9D3ZO7DMvPICRgg9xsW0Vka026spqgX4/kIgm8edl2MMTKAwrtRtHOs0fiNVgeFSPVF9l8Jb7fZADPESmBySPr9dybzghOE1HQ+3DPm7wt92g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.015531, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) { From patchwork Mon Dec 23 09:41:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E390E7718D for ; Mon, 23 Dec 2024 09:46:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4B6A6B009E; Mon, 23 Dec 2024 04:46:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CFB216B00AD; Mon, 23 Dec 2024 04:46:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B74E86B00AE; Mon, 23 Dec 2024 04:46:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 989986B009E for ; Mon, 23 Dec 2024 04:46:18 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3E3EB1A111B for ; Mon, 23 Dec 2024 09:46:18 +0000 (UTC) X-FDA: 82925742858.14.47B13FF Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf19.hostedemail.com (Postfix) with ESMTP id E6CB81A0016 for ; Mon, 23 Dec 2024 09:45:36 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hVoCompM; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FItpERzyh1wDMphf2PSqriZlspFkmpQlPhA2jGTCtDU=; b=UM5BFORbmtDNgUrKsa/2UeRhYAsbWfH0IaDGRyWtZaiNjyTqm2CyvTmxNeTrATOxkjSxgn RIjtS7c+ZlZ0yi5zZJbjQ0W6sYAA0Pyw+oc+lSYmgM7Oso/azEnrGsxpTpgegoSlmjwofq +hXBJTW5lVf+G+6wrAnruQnPr6BebCs= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hVoCompM; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947159; a=rsa-sha256; cv=none; b=Od140yefAk1M8WNDQ0Oo1aXTeAyKQ5HzFEJ/avlCFkz99POozWeYdH6c60bZ9HnJ6nCqBp TzbaUtXqQwON27qcc8C5c3/sP71wgdRR8GF1iD3OXi6phToSs4VXmQV35AI8yr2NRwzm5q KxdppNWFHptzdKMknj2IaRptZBvW41o= Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-725ee6f56b4so3378735b3a.3 for ; Mon, 23 Dec 2024 01:46:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947175; x=1735551975; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FItpERzyh1wDMphf2PSqriZlspFkmpQlPhA2jGTCtDU=; b=hVoCompM+2//nNel3xjaNGArI6lU6ZiT4uDvCVwo+zTDbXS4sQFcR07b7z+b0DzrJ7 ni+dasN7h8imwVA6xQVV1OyA+jrDQLoFryQ8cZzyuW3QNyHp8hHuw7L0FWJUOS2qpOFO E+1DJ0syAypAJqVo96lAZNzJDTRyvyWzU9KLv0a/BUWdAmgHULIVL/Oqintwf7QT0I8G mcbhjSzfVbQbaPH2TacANhxKaCP4GvWBYdT5sEvaDZdM3sGtr11k72rAuCKF8YlUaiRq IFuaiHia+0U5Ynh3eiOouS6zOZzAyKg59v6JaL+harwD82Jnqx26raRt1Yi5i2Nnr26W LEcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947175; x=1735551975; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FItpERzyh1wDMphf2PSqriZlspFkmpQlPhA2jGTCtDU=; b=W73DWV+kvGkFR8N6bh/hm4/mGmo2MzxQ/KcrqEveM+D0TNz8tGNNCWr3Qq+2ol6AG8 OsCmI7YqEiuKn6iBaOkDvIRGbwBIiF1ddTapsKjpXDZgS83oII3ioGxdgwLnbafi9ZMS GSfH1coWByGZ0YLM4BPHWF9nt8yxlaKzLooWuWtD5I2UJoeeXUIJse++gGTnTSMhg6yY GpPhEXtN7Jo3EyWXBtCNTdAJUWPYQGBNKgLxjx133Hfa1aXZ5hR2sMq8rNuRY2DYOQY7 03rV/NdHx9eURwoH/8wqxErK/DhJiB/fere60v2SG9b5JrIN1CbYyE4bvA4d2qaHqvB1 ViuQ== X-Gm-Message-State: AOJu0YxTZhCD75ZdJyr/OjDfDPCjd7I+l8mpj8eo8Zc3CLiZBlFiHhRb drsVO+ghDCWS1OcE2XiUnt7URFH0imGxiGgLv8ODjoCAwDVyMFIMbJKfYgVZx10= X-Gm-Gg: ASbGncsE5fwVbbPsQk/X2Nlk/j3j9hNuUTkmrUuXq76ewZajH7flL87OsHRpREDpHkk DZPSP9x3e5wUEt7C7G5KYQm3i9SnFEv+jMKM6A1Hri5HSh7pDVGngQC2zUO9wTZAN1l5dd04qLB gkZC7/OQRKl92H2yo/N62jRu0CZz3RrgRfuDAiGNE0PyVuIWYgY3m7RSNgUD+pD1fevR9yDvOkW 2WVb/zthrLhuWkWYBSF5U5vUT0rCjPpbG+xll5d6IIfeeqrbG1uRBAj3HRx/xiJRtblT6KfP4da N3/W0ruZaQLWCSbNA9Ed7g== X-Google-Smtp-Source: AGHT+IF9RJvbCaq38Te31r1RKGqLS5POHI3EPqaSbNpVA67C9keEJqX0jQ2rnqy+EikMhfv/vLwTVQ== X-Received: by 2002:a05:6a20:12d2:b0:1e0:c0fa:e088 with SMTP id adf61e73a8af0-1e5e059b19cmr20289317637.18.1734947175103; Mon, 23 Dec 2024 01:46:15 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:14 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 15/17] mm: pgtable: remove tlb_remove_page_ptdesc() Date: Mon, 23 Dec 2024 17:41:01 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E6CB81A0016 X-Stat-Signature: c6wnr1omktednyeeieg4hftmeu8h87tk X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734947136-181438 X-HE-Meta: U2FsdGVkX1+Zo3Sa1qWAipNIuMPXkk/l+r4K+JSQrI9Uqb1X7oS7otcA87cHazAjovC2K7K1Uo/1T8XhIH6FOZJ8xYg7dirPyDlVOSFS20clLHl02dwRGeBTODCrN0akUhxQBei+BgwYqNmh0fDxzXj2rXF5GBG3M+vG5zhts0jXdh5ptqy55JArp1f94rFOgnvJBuWPl6AdXtkMxBFRdUl0i8dvvckjXhCAVIEpYd1AhqA3Ee9hXUL2ClQ8pvYLiVml+76g1N7SUIBoNnSbMmh3/r5hiGcPUulN6pdSLiYgVe6cFOQVNV8Rzm/EJwRIKlpopdqRcaj1vSuDynmyr1eQxDykq684zE72SGL5KLHAKh+WMXb8ONWu5nf8g03Eo8Q90zH4zHpmivShGNloEZMyLj5FcFnJ12FRxWI9U2Hh5icwfHX3+Ab2/27/vPiRzDfMEyQEz4ckAxOHLfXK57CO03yWD5HEQXNB8wUhE2KopKiJE02LLIYkYmsP9QJ4bMWFErGwIYP8qbSYQL3kJNszbjCEU/V8vZPTF2pi+8ctiE3aQ4nX8IyusRKEUZO2x3PrCC+9uDdcypkWd2O8xVOO71WrQtdSw8NS9dV6dgbwlJS4YDgcxLMlL3aKuYWEYnN5DLxMXbxF+NTx3XC6L4MgxV5gpyVkeqUT0MZ8DyPQ7Jt/3dEzq1CPd/Fv4JBH5ukyMG1EM9yilKhJw95EuKYmYN3KyWiHsYTl0iWyX/zdW57c0rz8cTbbg4TyYKE8IS+KWvT1cilbBYhm+ZGmoRF6BVTFYS0mCTWb8KYOkLliTOVj8Ww1YGSOiQRS4g6sA9PS2vGRy++BXJadro0cjlmEjaHWST2x0XJO7XGqPTBU1Y91/wOMebU6X9SQNTACmlG3gdQzYmo5fBhs7phk6PK6ADHN9bpRoy7BGiSMKOnsZecJu+XuLHd/9aYHr4hm5Pw39HIcIoGr01xrRKF xV4VdaAV oXn8EFcKnJrG9ix2gBK1Rxx0n8cVDuNIQfjywxJ/jUOoE9e1w55M1xnuqrovwy9ppE21X+FggdU2QNopw47zjbWxXKX3EA//znxR5H5G02I0xwF1r+vxERr0++7V2VBhVliYFaH8WZ/TuxgnWyKghsq2Yk08q5BJO+e7sAoi/D26+zMxUNXvgng/3meWCv1MGvnlXIWv+M7u6Zb6sEyL6bm5wtfsWk16v8yNKQyMoVRq9koaNjeawsj3C3GaeYCwZDFO9ak8cxxWgV2ec7DGvMpnicBH6d0sdQrRfhjcr2ez4paOCLGGlfwOT9OYxwy8Mnr9EIgbnrNAoh/qyYbVDzKeKH2SROrfajHMNf84zYLROcxnHIsVSRuHaUlO49wRAjZ/KqfBALF942kLQzdnrRse/u/lv51y29o8xRcCTblERHrpYZUTawLpibcO8JGbagnBJVJxM7aUJjyotLjCFvUUHmpWNghUOPOh27qCxquBWI+I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Here we are explicitly dealing with struct page, and the following logic semms strange: tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); tlb_remove_page_ptdesc --> tlb_remove_page(tlb, ptdesc_page(pt)); So remove tlb_remove_page_ptdesc() and make callers call tlb_remove_page() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/riscv/include/asm/pgalloc.h | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/um/include/asm/pgalloc.h | 8 ++++---- include/asm-generic/tlb.h | 6 ------ 11 files changed, 13 insertions(+), 19 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index f1ce5b7b28f22..936a43a49e704 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -64,7 +64,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ + tlb_remove_page(tlb, (pte)); \ } while (0) extern void pagetable_init(void); diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 40e42a0e71673..8b1550498f1bf 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -90,7 +90,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor((page_ptdesc(pte))); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 7211dff8c969e..5a4f22aeb6189 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -58,7 +58,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 2b626cb3ad0ae..63d9f95f5e3dd 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -20,7 +20,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 36d9805033c4b..bbee21345154b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #ifndef __PAGETABLE_PMD_FOLDED diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index 12a536b7bfbd4..641cec8fb2a22 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -31,7 +31,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* _ASM_NIOS2_PGALLOC_H */ diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index 596e2355824e3..e9b9bc53ece0b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -69,7 +69,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index c8907b8317115..ab4f9b2cf9e11 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_ptdesc(tlb, pt); } else { pagetable_dtor(pt); - tlb_remove_page_ptdesc(tlb, pt); + tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); } } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 96d938fdf2244..43812b2363efd 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -35,7 +35,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #endif /* __ASM_SH_PGALLOC_H */ diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index f0af23c3aeb2b..98190c318a8e9 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -28,7 +28,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ + tlb_remove_page((tlb), (pte)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 2 @@ -36,15 +36,15 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ pagetable_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ + tlb_remove_page((tlb), virt_to_page(pmd)); \ } while (0) #if CONFIG_PGTABLE_LEVELS > 3 #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page((tlb), virt_to_page(pud)); \ } while (0) #endif diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..8d6cfe5058543 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -504,12 +504,6 @@ static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) tlb_remove_table(tlb, pt); } -/* Like tlb_remove_ptdesc, but for page-like page directories. */ -static inline void tlb_remove_page_ptdesc(struct mmu_gather *tlb, struct ptdesc *pt) -{ - tlb_remove_page(tlb, ptdesc_page(pt)); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { From patchwork Mon Dec 23 09:41:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1182BE7718D for ; Mon, 23 Dec 2024 09:46:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81E856B009F; Mon, 23 Dec 2024 04:46:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CC4E6B00AF; Mon, 23 Dec 2024 04:46:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61EC36B00B0; Mon, 23 Dec 2024 04:46:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 41E846B009F for ; Mon, 23 Dec 2024 04:46:31 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EC281141700 for ; Mon, 23 Dec 2024 09:46:30 +0000 (UTC) X-FDA: 82925742858.15.06D62D8 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf23.hostedemail.com (Postfix) with ESMTP id DE404140007 for ; Mon, 23 Dec 2024 09:46:03 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Bi4Nnr0n; spf=pass (imf23.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6vzAxma0H0oGKVF+0yjfVv8ni44DJQgUTcLWSpNFGuY=; b=1lSjKYRZV1xS3hDTjrDJ8kY9YoBzFF2UskbCZSZAvI3A+zNx6Xxkw2jJunBWgVGhvyJIku 8oAyYHK8TPPqdVo5jeOI0+qDcN1tsUTS4VQPjh7CNPoTv8VguptQCzn4+ievwsHVwTnCXa bF4MnzT2QLVu6Jb9vI5lUaoQdlpAvOs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947171; a=rsa-sha256; cv=none; b=TwXWob7lgHp8p9S+f2t9qBlW1aFicYyXUUEb63mEjQmLcCIkmHBC4Kafd7DCO8JczkDIoR IJ5HVbmTOYwX6Hn15i/GZAZ2WO91LsY0u/8HryrX1+7yyN/9rtEDJYSPRK9G/UZhDjJGMo Rh3sy9kb3w2AL9IJlzypTS2wsanSOW0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Bi4Nnr0n; spf=pass (imf23.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-725f3594965so3283143b3a.3 for ; Mon, 23 Dec 2024 01:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947188; x=1735551988; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6vzAxma0H0oGKVF+0yjfVv8ni44DJQgUTcLWSpNFGuY=; b=Bi4Nnr0nGTpwL2toKuYkSbY8+l2M5rm3g+R1b1+BN+7brLWwEunGiFIgGNk+dBaS1z UayI9w/LNFNZhfIr0OtXSbSSk1w9wArfKBWDnIRUTAkSgnuS9osDyEVPGFqO2Em7yKgI rt9hxk/6Fh7V+Ev3O0UvK3h9YpllnZoYfZqRcdc8kPfNz/+AcUYf2retoNEZYDIENfTl 93eZld3sX/dmu6CYgagRaCWnBwM0gi2bQgMr4iWBI47Zn+vQRDznE7FtXY10gHNO+Dy8 1L1cv5aAXd9OipbR1V6RYwwq0zk50QQsQ5vz/iEW42qENUaoBJZr/o1a46Jx33pO3NQK oW1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947188; x=1735551988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6vzAxma0H0oGKVF+0yjfVv8ni44DJQgUTcLWSpNFGuY=; b=RRKEA26ZSoFQ+g9np5pBMwK/o9SpNgP868W+hWccv/xfmbQZAMo3GliibZqWbTFDzE P1G6NBMw5GrQGovxGqUEr5xJxyLRl61ZbuimdDggXqwYZcStmEGXQhiNjdSkBt11wkQJ gfkhrBPFVYmQFm/7DE9AQfWVKG396dZM1GygwH2hDicut181Un9T2z8Q3ixDkf+RRgTT lgTw40yPWRD6EqyQCRGqYJvKle/bFvpyXzaKDxws8kFKZLGounZa0dQmOXa5w9SVEI0j PKlEW9DwGsw+cdD6LYmbTAUbMyeGE/E/ZfabeGO/qfuIBv+UfsKNZstv+E7oeqcgVnQP 05vw== X-Gm-Message-State: AOJu0YySiTfS48ig3dA0kyM1zauvdGnaTH2Qyr4Os2M9rYD3ecpO9X5T pSWIu04fzvvU+psFg4MXc/Rtsvqcc7In3vY/aPRQdg6M0/pullzP9X9cA7B9Fdw= X-Gm-Gg: ASbGncsWOGI8cV9fqO/EcR5RN+wbQAP927dwY0u4UARXsnKKnza3uoqDdpLA4cPwEHN NzV1Yq0XT25dEFF9x0Unnb7rILlAFcjcAjrFjTU2QTUTE0I7Pmz5PGrNxuk6PDtP10V089u1037 yCTSG09oAWV5KilB8lRYoY2y1ZonaJsV+B0RQzASqae1PTygIKMhdsDk6fVc83KXv0nSr0xhTaL nyqJpoVReYpgXbyRL43O+xFd9lUyci0CKYzXjI93FSoQbkLsJho6NgfHiYk7U7eowslhEIAK6R8 76oPhAnuAQ9TbE1hyyJOjA== X-Google-Smtp-Source: AGHT+IGqxRLXXsgzbTBUt4P8wMbb/CAKHFN0NH0iCzAzNT/ELUdunTp3cz3Ieuj7L8l/gUjxiL3RMw== X-Received: by 2002:aa7:888c:0:b0:72a:a9b5:ed91 with SMTP id d2e1a72fcca58-72abde0e6b7mr17804649b3a.13.1734947187839; Mon, 23 Dec 2024 01:46:27 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:27 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 16/17] mm: pgtable: remove tlb_remove_ptdesc() Date: Mon, 23 Dec 2024 17:41:02 +0800 Message-Id: <93cce93bf8be04f3a5cd828cc0a48750fb90af44.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DE404140007 X-Stat-Signature: n8axtdprm3qrxup11arie9g87giixg9q X-Rspam-User: X-HE-Tag: 1734947163-966085 X-HE-Meta: U2FsdGVkX1+Eco8JJ5zaP4PIq7gwhOSGf9l1gda/3DLuhhKXTDtFDccpcnl2Z5JrbK3sY5AKl0l9hDaYd36JU0yTnBD/gcF+n1WDe5AT5uzeQz8LuO0A12uskY3xK7jiXdND0pBo8y34VlbTJjXXJVgRnTRBBAhq0iax982VF6KqvvKIolkDiKGq1yrBE7KjeBFTH4N6tlG8nkE+uvZLWrLbOcFATelGqpJirV8QF7ciga5YwYPda0Enn+d0dT1aCvjYivRk+NHH6itY/Zp85VDf4tFad7XJGWNkzDyfr/lYDEC9MVHEmdX1ddfyKxXsjOeOlnixxALMLJyqxcJYh2PiGQVtLhbMfRIsRieWe30f/RvpwJthvvJmFG+DbCDY7jDvV/7vMlpHAykUoyUJ6LrL5cPE8Yymibu9YNaVvtbT3q8SSnNkEE2oRVeG3YCK5xtTbd+ZFjfsbeWS+U6GjX8W1CD/NN0acwyw7T52lGL/qBGCFQ+H/uOenkEQ8lNmWzLoXloZMaalgJbKzZ7jAvjZliTj+ETTZ2FYHjMz8h4RrxZSTcmZmTfggdgbegFa55QaIqfi/drF/yY6cEliwjDDIsG73IV869q3aK2Xx2LjVODKj1TwOqbf3BcQ5FkLw5DHiApfDRvMM4RKWh8wrDfUbhLvhQJT57mYH4hefZoDsq136mQObiiJbePozL94aFMnr6OfP+sKlp8oGwZqEva+aNswMjO+1/VDLLvA7fMPu0KMPip3NgVKiN72TJXJ9LvviXqDeMtWwBAZo46OFeWqi2o36CjRIS8HffaxTmKPwxgtjdGtdP2boRMC+r8XftYfVRKdK2McN1+LExiPrCdiILZ8LOBIUlejyDdoyxGBQ2cugryNEcK3g6dqReyr6f5ImaZbIoVramCVGHlM1mS96exlJbPODY6iJlX0HtJT8gKXhx8S27B1bSc8ZcOs2NIBClbJXNUlu66QigQ IPPZLC1x 0FmJuMEffX46R7jvvpev8/NnBgtADhF6Tnc4zRCQjyxcPPNdb/JEDU4GoZ0exaRQL7aOh56zxBnybHGY1kJ9nvoBqOGpMw/mDyPTbAQ9ssqinDljMBkaSQhZ9aQhuFV/4Am+hP67+XLE1S6ZQpiIsdxEsxcuIPeSkEcJHUlpcKrawlsXETh6f92DndmSuGSH4EoiDslW8m/JNgTUXJw98RKxmuEYnovpw+uZH9wi98xQ8YuJupa3ffr0NtURkMI09OxBLwrBql3OmZtRJq+D4qrvixrckKkyTDlPZTkcQNN+9+DWc9qotDGBFmHW3x3PxvaY6Mpkj8DRA3RAx89NJv/pNA4PLtdBOJZvUR4ciD3796vgFUvfsK0xNnBtwUvtKKe/NaecIZu3eQYYP0EShSJt+7TS4zeY5emK0SfqNH/vCADNAZEI/Zyac7Rr8kKUBCbuKb6aMfJX2pH31cQ9OPoOZgH91bxJdmp59qqb+DFef9lE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Just like removing tlb_remove_page_ptdesc(), remove tlb_remove_ptdesc() as well, and make callers call tlb_remove_table() directly. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm/include/asm/tlb.h | 8 ++------ arch/arm64/include/asm/tlb.h | 16 ++++------------ arch/riscv/include/asm/pgalloc.h | 14 +++++++------- arch/s390/include/asm/tlb.h | 8 ++++---- include/asm-generic/tlb.h | 7 +------ mm/mmu_gather.c | 11 +++++------ 6 files changed, 23 insertions(+), 41 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ea4fbe7b17f6f..ac3881ec342f1 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -32,8 +32,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -43,16 +41,14 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) __tlb_adjust_range(tlb, addr - PAGE_SIZE, 2 * PAGE_SIZE); #endif - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { #ifdef CONFIG_ARM_LPAE - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 8d762607285cc..4a60569fed696 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -75,18 +75,14 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, pte); } #if CONFIG_PGTABLE_LEVELS > 2 static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -94,12 +90,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pudp); - if (!pgtable_l4_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(pudp)); } #endif @@ -107,12 +101,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); - if (!pgtable_l5_enabled()) return; - tlb_remove_ptdesc(tlb, ptdesc); + tlb_remove_table(tlb, virt_to_page(p4dp)); } #endif diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index ab4f9b2cf9e11..25c2e2f262810 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -26,10 +26,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { if (riscv_use_sbi_for_rfence()) { - tlb_remove_ptdesc(tlb, pt); + tlb_remove_table(tlb, pt); } else { - pagetable_dtor(pt); - tlb_remove_page(tlb, ptdesc_page((struct ptdesc *)pt)); + pagetable_dtor(page_ptdesc((struct page *)pt)); + tlb_remove_page(tlb, pt); } } @@ -108,14 +108,14 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { if (pgtable_l4_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -144,7 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + riscv_tlb_remove_ptdesc(tlb, virt_to_page(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -152,7 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); + riscv_tlb_remove_ptdesc(tlb, pte); } #endif /* CONFIG_MMU */ diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index da4a7d175f69c..5eed6300f3d72 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -86,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); + tlb_remove_table(tlb, virt_to_page(pte)); } /* @@ -105,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); + tlb_remove_table(tlb, virt_to_page(pmd)); } /* @@ -123,7 +123,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); + tlb_remove_table(tlb, virt_to_page(pud)); } /* @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + tlb_remove_table(tlb, virt_to_page(p4d)); } #endif /* _S390_TLB_H */ diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 8d6cfe5058543..583e95568f52b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,7 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = (struct ptdesc *)table; + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -499,11 +499,6 @@ static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *page) return tlb_remove_page_size(tlb, page, PAGE_SIZE); } -static inline void tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) -{ - tlb_remove_table(tlb, pt); -} - static inline void tlb_change_page_size(struct mmu_gather *tlb, unsigned int page_size) { diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 7aa6f18c500b2..c58ce4539c56f 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -314,18 +314,17 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct ptdesc *ptdesc; + struct page *page; - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); + page = container_of(head, struct page, rcu_head); + __tlb_remove_table(page); } static inline void __tlb_remove_table_one(void *table) { - struct ptdesc *ptdesc; + struct page *page = (struct page *)table; - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); + call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); } #else static inline void __tlb_remove_table_one(void *table) From patchwork Mon Dec 23 09:41:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13918682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9999AE7718B for ; Mon, 23 Dec 2024 09:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C5146B00B1; Mon, 23 Dec 2024 04:46:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 274126B00B2; Mon, 23 Dec 2024 04:46:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EDCB6B00B3; Mon, 23 Dec 2024 04:46:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E007D6B00B1 for ; Mon, 23 Dec 2024 04:46:43 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 90A6980FF7 for ; Mon, 23 Dec 2024 09:46:43 +0000 (UTC) X-FDA: 82925742438.01.36F7874 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf11.hostedemail.com (Postfix) with ESMTP id 0AF0E40008 for ; Mon, 23 Dec 2024 09:46:06 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Khlz2XCR; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734947184; a=rsa-sha256; cv=none; b=JrAWM7YI627Q+UYL1CHEEtVA/Rit+prNwrBeEpYxGWQU2ynqrhmh4RvJl1Y/0UF3j1yB0U FGQ391dDp6SULfyCpT9rWrOokT2XsgLuOnGbmdaPHVwf38oKNgMhS8piP6OvpFIUclJj5p 6Y7Q74tgpFhkoutrTyoGKbBAMhG0H+Q= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Khlz2XCR; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734947184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=WlJVcxjP3q9YgD0uk1gwarSWwZx1ARfYLjLUHcqRzQkFTy6GYQmaPKL5vXubj5lCQNNN/y 55XycPXLYTttWay0UlT9S7DFr2S10ckYujpcZFyT1JNpT9MsA8jnmuPRdtIXCAH/ZtyVuT QOoapbrPeR5UJlKbuYnBRE3cuTVvwmk= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-725f2f79ed9so3110666b3a.2 for ; Mon, 23 Dec 2024 01:46:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734947200; x=1735552000; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=Khlz2XCRf3nkaxHH8ouK5t8qZHaEIiygKtMtUEAOug/mM9Y+lMtqky45dxir3e+Lzl yGLiwr5A6zGSWVD1LfOb29DifgQTU3fKUX09ze95uM3MlKkhAu2OLNDZPjHVLF8z6Ql4 P63gM9cntJUY8DUMZId4jv21feAdgXKTsDd7dRyiWlLOOvbPzV6pmjaVRkHYtZnZxZGX 9sd3VcHYp4JbOVicUICaGB4C4Jct3tLWv6xsteWSiYRqMYF+EY4crU0+S3fJaR+gsmSs mLA6yUzme1zumWpZVvaqHXre7o5FFqZC6FtZoWysyW8CtOMegcjjCMZs9FHdyF0QyehE 7RPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734947200; x=1735552000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Vy7m3PQdTp9s3cHn2TFUEZi29ZbV6wwwxQw/bfJltk=; b=hn0dpV+cGd5PG/PSlBwgjkg+OoEOJUc9w+WaVNvcV1iiQPdCNAdXQifEloGJ1EtodO jvoiWV11GClv+Bwr3Zm5TKLqtbr/Htk8YoOPVP26XwU7MG4B1yD2ZqR0VhTxfeIR+Gt/ 5Hd8WFAR88R1QqYYRBIjo2B8efsulfJx39fN5yhLy4XdT7cbt9RGi4gQG0MOdUb/TLXH BGltrvaS5aScvbPx06uHPd3kuPsmco/tHUGYRSrhQeBKxzkizgsW7aHNNQybbTRnkjgk z1DAQAduTgObztfd84HxJjagxuLnh3NkRrOZI/bLtfsqGVyOLhGHrvMKqXFabIN/YyBj HtgQ== X-Gm-Message-State: AOJu0YxSNKaeab4kxf+fQWxAWLjI2G0xl0hnks0e6f+IL3sL/3ooQL87 qanC+jS2NovemQMHwtl4eEEXYucruJBDKEMmtIhxOeCImfgThnh+ymwyNqNG0uA= X-Gm-Gg: ASbGncs89KuQgOzfRtDtNtLeHTAYu5vf6WTNz5Hx0B2FBfUVsze3ERAzyGt2v5xx8Oa TN/K4jtQiE7R0/exQs1Tj547MKEW+LR9rje/B4tjKpUulDSDDLUNWANTLJppUVRpIvjcvMt8ZAp tRGEfjso0AJBHwuPJtsFaMLzc3OmTT64q+rTrQ/jwc7RFf602bjCw/cPLkxq0k02sjRrBTCRhe7 wZ6ryYIB4sdmbLgvSnlNaA8bqwscMVhDIKeQFCgop++kfaSCLa8zDdscA0e1FhuO4KswTKfsaBo 3bYHd9SqvHgXqlFAdUPPiA== X-Google-Smtp-Source: AGHT+IHTlP+hesMysY/dnyw4k1T/aSGpJ0FSxzWG+ektln9VxnC2FaHlf9zeRwOgabkwOIXU2HSkPg== X-Received: by 2002:a05:6a00:2407:b0:729:597:4faa with SMTP id d2e1a72fcca58-72abde9c004mr15027146b3a.16.1734947200520; Mon, 23 Dec 2024 01:46:40 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8486c6sm7468309b3a.85.2024.12.23.01.46.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Dec 2024 01:46:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v3 17/17] mm: pgtable: introduce generic pagetable_dtor_free() Date: Mon, 23 Dec 2024 17:41:03 +0800 Message-Id: <3ade33c5049f465dc2f0b95edc2d68c80f2048c9.1734945104.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 0AF0E40008 X-Stat-Signature: ttsoogj3pwcxhokibta9n7hy6bcc8a8z X-HE-Tag: 1734947166-282867 X-HE-Meta: U2FsdGVkX1/n2SHVUs+xoIQxrYdZPmIOgLopFmK/8a5cuYKKFUe4h5H0zD/7lJy9qN5g9LRJX72n6NcJGmQgzoseNDbWWl6dkoH5YigRyHznQr4BjalpaEyPSbogMCHQYdMq+7KQTiUm+IIX1g3sKMjU0jWlXa55QmzJ9kVto1qj7sS8rXFYx5EWlRuRFwMXztNV5nnXZqKwVo5k7QqKaoxh/PT2GA0xeOb6Ns7ZiWyYp5GEEUESiJQ3Z7IDFFR+tN4J7ScRR73lhjScxqEiIQxdUtGCcF+zk8HHQt19MYwDG4SZd1CaaBzTa065u1x/gpHuMsy6dMXzSYMAYeXrMWnouSOQ79l3639VtcvuSs3biLDjILwT9bdbkcBYkt1ycvzLwjujCFG91+ZLFlP8erFNQCJn7LmyAv4/ObaKYzPB2gHoRhksKlrfQB6lsT7dSfBxfAC35krRruhCFCCKaaaPfWyJ00HeVBi2L3zpxUfCG/x6JLpRDgiaAKRfVdeC+kp2AuKWn7tFuR84rUTaKAtUrQ87+H/74KViyRfP1OvWsT6a2/xg3NZ/RfY0Uzm05A05ps4RMxWUqeYk7piXSWbdWoom8TMUtRoXLdn/Px26tlLR5gSh5OAYuD3AlYv9o+G8G0dOZZANtMp2X28dfV8JxbCKVDeRYM0JC5d+4kE6qKdFfHQxsmQ4aftxcOcYDTUsfwCHHXlEPyOJezh8XkPkIFQtYA0MPB/u/+csxilOy9FBcXN9gdC1HI4aJ7RvBx9AM3DyyF136yZKfUneYh3wtCPLn8Y8fkC19zWb10BSv1QUrn8u3lAMAM1v4o25+V6oygAj+tGxno9uE26cONDcWbAp+tkXIVd7JvKny35Bao3G9G9aDX8flatUtyyEd9vcokg5hI86PkoDd3iN3c+5KCMevysxHC9k1iwnwT9GUeIbd2jetZQ+EUxjp1oOVziPXN44/BjrwPxlvzz ZhI3usxI Hv9KcXqfrjtkw7imiEJb1BcTCmmesjggKT6u3VT24SDRWAOsfzAOYj+bOasSfUahFxygTanoh2MRV3cBesljf7zLVUlDuKTmfAUHML4oIAImlJdF/JD9wBJtNnSIDNMpvijLhyE03Bv/Pb1ooukvOhRPsKknosUpdB6vTeMzmOrVnqEpY5/VwvR+jMIrTd7Dvr2fxeoY1MSh267zegptp6jZf8C/yupasotZfySCguUS0WkgpnoL3jLQuFB0Tm40lK0cNpv/WIf/ZIMgTMvwlAceYhqLMkciaSlH80kDeNZpm04eh7iDXL5hXOEVnMWxvKI0dyOkJjo28xqEBBVIkc/leMWJr56fnsK5C1Eo3Et41U5VtrygK1la4TKPdR8FpUjtiUAUSlJ99kIeRFk+mAu1O7CTM+g9GMNybJGSr9Pu4f51oM7/jMVCzkXRFiIpjLdSqjYS2+qU867QTMdLK40raSJVCiT7kg1hq+oC+T9pktcvpUFJo8ly1JpYuhEjpzIOwXi2+FbvaF0zHrdrej59ySaTVcYBgAXdt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 actually does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) --- arch/s390/mm/pgalloc.c | 18 ++++++------------ include/asm-generic/pgalloc.h | 23 ++++------------------- include/asm-generic/tlb.h | 5 +---- include/linux/mm.h | 8 ++++++++ 4 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..1e0727be48eaf 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,32 +180,26 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { - struct ptdesc *ptdesc = virt_to_ptdesc(table); + struct page *page = virt_to_page(table); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { - struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + struct page *page = container_of(head, struct page, rcu_head); - pagetable_dtor_free(ptdesc); + pagetable_dtor_free(page); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) { - struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); + struct page *page = virt_to_page(pgtable); - call_rcu(&ptdesc->pt_rcu_head, pte_free_now); + call_rcu(&page->rcu_head, pte_free_now); /* * THPs are not allowed for KVM guests. Warn if pgste ever reaches here. * Turn to the generic pte_free_defer() version once gmap is removed. diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..7d327889df306 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -107,10 +107,7 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm) */ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { - struct ptdesc *ptdesc = page_ptdesc(pte_page); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(pte_page); } @@ -150,11 +147,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long ad #ifndef __HAVE_ARCH_PMD_FREE static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pmd)); } #endif @@ -199,11 +192,7 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(pud)); } #ifndef __HAVE_ARCH_PUD_FREE @@ -245,11 +234,7 @@ static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long ad static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(virt_to_page(p4d)); } #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 583e95568f52b..ef25169523602 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -211,10 +211,7 @@ struct mmu_table_batch { #ifndef __HAVE_ARCH_TLB_REMOVE_TABLE static inline void __tlb_remove_table(void *table) { - struct ptdesc *ptdesc = page_ptdesc((struct page *)table); - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(table); } #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..cd078d51f47c7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,14 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_dtor_free(void *table) +{ + struct ptdesc *ptdesc = page_ptdesc((struct page *)table); + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc);