From patchwork Sat Dec 14 09:02:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0441E7716A for ; Sat, 14 Dec 2024 09:03:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 851726B0085; Sat, 14 Dec 2024 04:03:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DB286B0088; Sat, 14 Dec 2024 04:03:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62CE86B0089; Sat, 14 Dec 2024 04:03:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4163C6B0085 for ; Sat, 14 Dec 2024 04:03:50 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B1D8E161C6E for ; Sat, 14 Dec 2024 09:03:49 +0000 (UTC) X-FDA: 82892976684.02.F274B33 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf19.hostedemail.com (Postfix) with ESMTP id C83421A0003 for ; Sat, 14 Dec 2024 09:03:19 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QJcH2e1y; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167003; a=rsa-sha256; cv=none; b=FkKjNlWRS6T84T1WtHdY3Hsg9qBxSBhOrl6iqNQzaAuRBULR+XNoL1j8vxxVgsXj2v/MGL 1sJWVtR2hPTi0ULfLUFCDDYRfpxdp8iY6AFxpJfPOqPLCR0XOx/rtHqipMzwGVZARiG5rf L+tEbQ1ubfniQaRoUSl3whyTCjIjyNc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QJcH2e1y; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/W0KMvrB9Jil9zRQ2f7XfdU3YONHwHkiAoiFljp+njo=; b=NZB1TROrPUO2AfbVIB5JVInCjdxIpQSvq0+2RqnyYpoVJcV2H80rmuClLCWd50A5/Zpi92 b6wQ8zqomi6TfKUWdnwyXuuFI5Z31jpp+T0s1fyzuJxD9ZcYSz9w80U4wsQ2Z0drdCn7uS o+euWFn+Gu0njxlGV/8doUga/EOfLHc= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-728e3826211so2067039b3a.0 for ; Sat, 14 Dec 2024 01:03:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167026; x=1734771826; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/W0KMvrB9Jil9zRQ2f7XfdU3YONHwHkiAoiFljp+njo=; b=QJcH2e1yCfU8BaYky5MqmWHgzmeKIS6cgtkrxFNudAKgyjgWwVMhqteqqYhgmOw2iS UqPtQj0D9pN4dQxI+gaynR2yNtP7/S5nYHWeIfA3qHZsGP60h2JxeynefZLFlkNbO6cm MhfW9+PYRs08GJqADZ/qYCoBAZgCmz6YMVsS6kYup8wOIRCFTTCsJTwN9my/U+ZJ7JwP YMEa4zEd6Q/G0SE7Zd7iGkaRwoeuJ4oZD97HYRPMW/UMUB++BvDtCyuT90r+sRehwby7 Tl7FiBcrBtvU3pSMBlHO8mvEzrplQrgqlNA7F/BK4sa5Xja7hG9vEvU31GWAW7SY0mxl tawQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167026; x=1734771826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/W0KMvrB9Jil9zRQ2f7XfdU3YONHwHkiAoiFljp+njo=; b=SR8MfM3imWzrFajhKc18+5ztaQDZIqwW5Wdj7Q69lSCxcB1qxhnYJ/8xN47DdWNN8N 4ziZN1pzfxiExrnvJ6TDlXCwYi/zNaujIHesYgdQbdaWkGG0YVFIVIHca4XsBqqrXGQW g+wZdtmU4eKW1vSuU3FKEbM/SIHhZf4cZwsJabn2XvjYZgrfxBDunvReUw8FmrpbkMYJ tA60zR1FkOiyYvhKMNhCQA1zOD8iIuhsqMIugpvG1l06RWYesC2GsZ/2gfvnpLQf4sF2 OAueP906fZllM+4eUYo1poDuN3YRQu28vaFCkLXUvMpMRj1CIWE3fH3yfR8bk2Lv5T2n mB1g== X-Gm-Message-State: AOJu0YyvhL2Sl9hbdKPjb+obMKefssw1I/uDtaXHZU4lFPTzAP+r5BSx sbzlHa3tfZp1lW610t70uBO8JQKRfyEbVn9vo6kILTkHe+QIvJt+xlPjaqxPIbI= X-Gm-Gg: ASbGnctM+BZv9xX9ELwXq7t/riJ3wpIBSHHALKiHg5ikQtWPgQ9CG0yEy0EcphOrfTW J/S3sj+G9683YlO88aQzvyTT/x0NhfazqE5n4Of8s5kIJt1KKPdCEXADIgs3zDR0kMXkO/7maVA pzTZm+QxpX8hrGSEenkfJ3izN5P2YAhQilk2sm5itxxZ29iddufG07b9V28o/yFOjsbF/HJQ4Ia /haixB18aYVyJM5w/j/b8Uf/Yo4nEd0n2N/o8ccGLCARjNryOwMUFCojihcESopTC9rnWysmY5X kVQFI5SgAREXgu0tSRDCdntSPqGlew== X-Google-Smtp-Source: AGHT+IFq7u9PWr4pIiPNWcOSLFRHni+qPQ4ESKUOWoSzAEQHr7L/XMr4iW3HNmhbkmKWJmb1Wl5pRQ== X-Received: by 2002:a05:6a21:a247:b0:1e1:afa9:d39b with SMTP id adf61e73a8af0-1e1dfc12584mr8746124637.7.1734167026495; Sat, 14 Dec 2024 01:03:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:03:46 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 01/12] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Sat, 14 Dec 2024 17:02:47 +0800 Message-Id: <1fdb3ee32e6958ad82229941b2213ef76b7c4705.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C83421A0003 X-Stat-Signature: td6fgbt33priaahy1kn1ujkwqe7ic5q8 X-Rspam-User: X-HE-Tag: 1734166999-851377 X-HE-Meta: U2FsdGVkX18hHjcHQkEXlpo4YfstCDRvaX0ttaOGjGwHbm9bn5eAIC/b7rDXJCyHz2lJhr4k+g9ca5hJVpZ7RxhXxTwjJhRkgUvB6yxvIh/U4E0doTfKHsqtN/QwU3VeOjEZdDODEFnkbzvqq4Z1IJ97aH4/au5K8tyK7wGIlyPIrLPrJavh9CrQzI1IM5+rsfdQXiFxVuX8XLLS1c4lUTXHamR0YfrXZUYkujaeOXqXikr/nXrNJnpzoxdNSfJm6kLTKaOjxNSo8kjwjEdFhnYqHAvsZJMvnS2j6rL60Y1kJZNeICio7E3DUcoqMnzenQJ3FUuB7aHpa+Hf9p0MWbNdKGI4Esev2Muj6rhIDyivsC8SDLp8AfRtjEIj4Uuyph9TRH/HUROXysoSPkJYLA/e1lmu55L6TvuX7yUMDujrOUBYhvi8pX1RYmlavCa8K2ov6Oe1MfL6d887bVoH5g5rxjbA8A+nzaOHZ3uS+94EyCBWG0Nqk0fuLYRBOcpIKLYbptr6e8eMEO1Pfzmarn0f7f39FehrTaE7LSkHgZI+aoUxdNfeTQYMbixKC0UcUAvyPK1sv2OorqJP0mJAnGgCrWaeMFod6wKNTIF4GEIDfTaVBF6l+G4D1Vsj1ExguNJo9phNWLjiA/A7tHMDFmvfV+fVGt3S+qecP5H8eYtue+XSsqdoChe3rSMBAo+yP12YZ7b0PRdUHAQzMI0inVCY44ad9bry7zz7Egn0iiT90B1ltYZbhJPUioConxx9V5GpHIQqYYMZPBPYnIqYOm9RKbWS/DUO8lBa1WPOoLKBKFSVSkJ2lxAVwJGG9mVXGnj/6ZCsP6esK0cewzXtzjjIIQMO5iHtUM+9SrODXKqMi9hGDoLQXh4OAsXwLQ1wnIqXmVNsQgDGwXa0rBRgaR44Lk85dA9E01pSgUiECT1vhdpxkL37C+5lWl5XuxzBHtaQOJuGNMnQE9EOCdB rjRvvl0y 694PPCJkoEprIC9cyspBCkJcx41JGgzi+iPdlalLrWwkzulL2lt8VERifanzLUdYATbcuAciFhSEopOIRTNADWOrx3nRbBLQWyfvMdcrA4G7deOGmpE+YurP1RNz95uHiBX5tq9e+/LNDfes2XW9NOedUkzmY8HxGyCEX7hcDsbOwSCfIqkNmAWQ+MNoN3870gTaETnXN9buGzkoSdzRtc3hS/HaxH+fC8mxWI1qkMcSCjXgQTpyTHDN9XZUMgd0W1fOqYiOQe/pxucxUKMl71JOk6LiWtv0SAaWeZ4l6hVIcn7fEfIm57VrQUNaeyDiMZuAh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000156, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng Acked-by: Yu Zhao --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e7902980439cc..5e73e53c34e9e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2988,7 +2988,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index df8f5152644ec..5d8779997266e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -485,7 +478,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index d9af83dd86bbf..83765632e20b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7041,34 +7041,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Sat Dec 14 09:02:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 588BBE7716A for ; Sat, 14 Dec 2024 09:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1DEB6B0089; Sat, 14 Dec 2024 04:03:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA6816B008A; Sat, 14 Dec 2024 04:03:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFA686B008C; Sat, 14 Dec 2024 04:03:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9EB0C6B0089 for ; Sat, 14 Dec 2024 04:03:58 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 53116C1C5C for ; Sat, 14 Dec 2024 09:03:58 +0000 (UTC) X-FDA: 82892976600.12.CC0E380 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf19.hostedemail.com (Postfix) with ESMTP id 58CA91A0011 for ; Sat, 14 Dec 2024 09:03:28 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lfE0godM; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167018; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ogKjScRTvWjbxx3zCDWukawvbFfMJH57eI/cAcIUhW0=; b=SgjAbuuuVbqI3MAtKiNuFYBf5ZCEN9jGHeXTRv1vaKwilRbO++Ndx6+cpbCPk6OpZ+Oe6q rVClZC1NERz4MinCA8ZcN3RAI8R3qvWuNc35HTiAXwAPTsQmueUBp0dd527l5573To/qyz r0OkWZyw7Ufi3gDuZI/Of9Fsw4eQ1rw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lfE0godM; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167018; a=rsa-sha256; cv=none; b=wwQCGgHs5RN5CKre07xxsMWJYAsJWBsMeIjJ599CKRU3U7HMKRV1fUQ393XtEp3BI0WPq1 ERgrENCNFIsQWqd9sQIxPiPc5EvTCKPzo3kANw4KBy5VEyomzmN4SWIfSdIZxhtpKQrf/f bWACtvNiFBQXf+dgP4H5rsWqK1K2lXg= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-725ed193c9eso2117034b3a.1 for ; Sat, 14 Dec 2024 01:03:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167035; x=1734771835; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ogKjScRTvWjbxx3zCDWukawvbFfMJH57eI/cAcIUhW0=; b=lfE0godMtaU0M0CsHVO1pKL/g7Me2tjqoHoe/miX4UECY39B8qDPv1WzThOJBRrsPj KQxbEcpThZAV1Evd5voL6K2bbTT/ThzB/vC11GRfjzqYVax9zJTBlXk6HnwXrkSGyLtv Q4vJ2lhtgAc2BAqubApqrFI/3w4C/CoTboyIFxaHtfjJvDyjtThSbKJIdKj8lmVW40gk bazwCnRT551pwV/btiWe2vf0pEyu3eegWPW6uy4qX+t2jq0eM1yp+Cnizsb3BS2mmT3a iP/P/Sq65+f+ExkJJyHvdgEcf7H1Ut4JdjA7IS4btGcyl4scx5kUi/4c9/1XFeQuT3KZ P8tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167035; x=1734771835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ogKjScRTvWjbxx3zCDWukawvbFfMJH57eI/cAcIUhW0=; b=ZeSKVC09A79QlDqBgBIeyOp+gXDk1rtR5OPh+7+Ep1ym+FEdi1D5HF+0OrN6yq9R0k 0Kc6pmDIsa7pTauhINr3fXWw+r7BqaFJwwULvo62JZm5+hdvrwQMMQhzB1RA3atZHnOt QgupD4fmM7Jn0UImn++VY1x/DRnm2zkld8DyaNkw/byPDOp05DG8AtBQbvWU+BcYqXZA oAxyP++tH5+NlZAjlhxqUd0t5q0Y95A4t+pS0+DssJkbNYoVDbdgf6PdKM5DaF6Mix6q eKzrdPb/Fd6BJ4eNnFxywqNN3iW5JLdm0Qaq+LbUrpj/fJCUJgPQAPJeRNgetBanZPo5 w/7Q== X-Gm-Message-State: AOJu0YzsPCtjdjhM6i6AJil63eCapESoUyp+X9P+z8H1sw89NmGt8HV8 n6M+N16QmJp4/CBbmLMxd2F007fw1OfDsaWVwqNdz62qd8TBa8thcz1wyMGS4mg= X-Gm-Gg: ASbGnct6Ig2QrN1UJhX97vVSxC9AKRCcRQvak51wvVbtvMIGN4uyX2T0msDQxvJ+ELh jYuPPkLDCE1QTq++FENXu2KPdl9nOAwTWNJ/UJcqaRznfpR7bpEHYIJoOH2p5h6GXnVaKpvjHE+ 61lu3wqun6G7m+TUeZCNQ8R0WCYhPp1Ol1YQjhlbv/Xtdweg8ONx/ZuXKEXkx+4efXCtwp8FcHa LXPPDQ1hKdTm3MGLBpL3XnrdPjADPt5R9paBiZkNhMJL/nl90ltdVOk4M4NjmDpolS7UTWTRF/W 2TFOkcXKkL0V1Dby0i784s/TNYjtFw== X-Google-Smtp-Source: AGHT+IH8cEofDdPO5AW+ExE/DB+fdraWijYPkixsUFC1j8JWg1c9+2I6Y5RryDFAZbl9xgKYU+lFhw== X-Received: by 2002:a05:6a20:9f05:b0:1d9:6c9c:75ea with SMTP id adf61e73a8af0-1e1dfbfbe78mr8233213637.5.1734167035315; Sat, 14 Dec 2024 01:03:55 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:03:54 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 02/12] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() Date: Sat, 14 Dec 2024 17:02:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: y4af1oxf8ffx6o1p34mmk7pfwwrt4oj8 X-Rspamd-Queue-Id: 58CA91A0011 X-Rspam-User: X-HE-Tag: 1734167008-22082 X-HE-Meta: U2FsdGVkX18iH6WEvwyvd82EDzuWnnOlkqF/pb+7i14K6w5tWr1TBp8FCwd3f1YUE7LD5lvBuDCZdNAZbqaix9bDgXpZqZLOysC6Oe7SU7se98kVubEYV6PAfEUrhNhg4W76KP96I9gyWcIp9llUcj+NatjFguLCvxz3Au4/fLEeVCd0q35pARf0vshh51tGHFliEIa96a+JO10g4io+Bq5bkfQyPpvdqEZDe0el3vPLEQdE30dy9QsLzC/X6vO4k2lt8IfsUJBAgB98PC4yAa9gmUs0dgy6gxM1qcfBzHd0YtVq0wHlVLAcZmi3pm4QUyz/4CJFy3XMXE/SQJVvk3w/y+Mj9L3xgEw8nGRzp/WiX0VzEyC8FvZ27/k6ArJv2Z8l7BXn2OytTfXjUmljvbINP2tDC4c4GJTlCjCEPHVpPS7DeN7AuLiStfVAwBbMLVSzcwlPeJyO7CTSOlHiacID8GjCNcD+7PPl8Zj//OcjTef+ILSj90PADCL3duEwK9M/7cfhq3Tp2VRL2sQYkstW++z7dhqeSsmedeAQvlg56sLyyq97LB0ilruCO5HKI8VEsWYbgJqEMh4hJfRSF+ZNrvd2weYTwiQbOia2FUY9jOe5kwsCm7dFYb6xMRWKFgYqYhvZ/CCzKelCH/MX7s5YOwfmh2lfmV2Jpalvc8tmsOLx5OoitTrAHZ9y8gDmMuXAZmb8DSdYCh9S/hFZ013SdiYR05ElNq15rsh2zBbAx7j3yKkGbA721X2Wr+/xtovU0rzFwpoQhOXDQxzfFESXiVfCBd71ETmdeinCVVT6OqABINhssiEwa+Eo9V6fWMuM856sFSOjKM239CFgdSvBgrkM+7iyW/qwS9U/6Oyk24I0a29HbvperFmyRYdzjCzm5wlnlpIRV+kDKZc+m6pGKEOT1LDCYFViCCyXN59Ro58YDsSVMdjWxjI9n2cPgk1U1VHTQOOpdklZGed lkbgmIuh wjFvkZqY2O3gqVjGA6sEe9MyKcuIMU77jxMzHio9OcfydC8WYt0DemXaRy88QVt6wSiiisdE3DrYiJ4BRSldpyx1ElyDi1GMkbOUwM181apHYU65GyKtDTsN2FGPDk6VfzCui4WnhT/1rsBFZ0oe/uQkG+9qFifwF5/Ynk7qHTg0du4T98E0Xw3gZxftNWqVYFRLXBUjlvH0mVI4P0W1T3mn3Q9KdegSsY2L4NP71hcnUrw1bHcdbIG5aSbG66grC0vuk9vreur8m9eENqxlQMuuZH4yb3zX0qpppc6xn9cLcuccWEq/D4szJvO6RM8D9UuUZGhhb1w1KpvF+oovLCbMD4Fakl4s/y6ui4iyNKSK3zVp0oIwGupeDiQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a wrapper for get_zeroed_page() and p4d_free() as a wrapper for free_page(). For these architectures, provide a generic implementation in asm-generic/pgalloc.h and convert them to use it. And like other levels of page tables, add statistics for P4D level page table. For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its own logic, so skip it. Signed-off-by: Qi Zheng --- arch/arm64/include/asm/pgalloc.h | 15 ++++----- arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- arch/x86/include/asm/pgalloc.h | 16 ++++------ arch/x86/mm/pgtable.c | 3 ++ include/asm-generic/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 16 ++++++++++ 6 files changed, 98 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..679c530549327 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,8 @@ #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_PGTABLE_USER; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + __p4d_free(mm, p4d); } #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..bb6e1c5f1fb19 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -14,6 +14,8 @@ #ifdef CONFIG_MMU #define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + if (!pgtable_l5_enabled) + return NULL; - return NULL; -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + return __p4d_alloc_one(mm, addr); } #define p4d_free p4d_free @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..d9bc6cae77c9e 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -8,6 +8,8 @@ #define __HAVE_ARCH_PTE_ALLOC_ONE #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_KERNEL_ACCOUNT; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + return __p4d_free(mm, p4d); } extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..dbf61819b3581 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + pagetable_p4d_ctor(ptdesc); + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +/** + * p4d_alloc_one - allocate memory for a P4D-level page table + * @mm: the mm_struct of the current context + * + * Allocate memory for a page table using %GFP_PGTABLE_USER for user context + * and %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) +{ + __p4d_free(mm, p4d); +} +#endif + +#endif + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 5e73e53c34e9e..807a12ed8ec96 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3237,6 +3237,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Sat Dec 14 09:02:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BDE0E77180 for ; Sat, 14 Dec 2024 09:04:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C21F6B008C; Sat, 14 Dec 2024 04:04:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 14BB36B0092; Sat, 14 Dec 2024 04:04:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F06E96B0093; Sat, 14 Dec 2024 04:04:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CBA9D6B008C for ; Sat, 14 Dec 2024 04:04:07 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4B23BC1C5B for ; Sat, 14 Dec 2024 09:04:07 +0000 (UTC) X-FDA: 82892976474.19.4884018 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf02.hostedemail.com (Postfix) with ESMTP id 9984C8000E for ; Sat, 14 Dec 2024 09:03:14 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=WHr52y9B; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pyTYWh3zj2l92o0qEOJPcU0U+om/laakNd04Rx8p+XM=; b=sfS8za8yYyrDaGxIqPrdRR5eKPQ8mcJMwKi9GGdOZn2YqJ3ALW6kPrgOUl2JpIVNnL2Kzv 7b3uWQ302gRsREhUO/g5IrHwO23iDhg6vLY09C2vs6NG1bYm/rXXKWcfwB1HE39TT0wGcU 3/+u99ypxFOkCqXgA8AKClFU/hcJuoQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167027; a=rsa-sha256; cv=none; b=UAEMpudGSwdNc4/1fqr0NZ1Ay9CHd9QbYx8QNVTlMcQaiIFHQTCJsMMiGC9e8zXm7CxfcS MsIOpfaoTk7vgxKOh6JBrSDwBT51+aoijd96IhUUXVV1Wt/8pYyIh4jDuSANogp8PGHstJ ACaH++TGHLXOF/ad0ITDtLziojM156w= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=WHr52y9B; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2166022c5caso20752665ad.2 for ; Sat, 14 Dec 2024 01:04:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167044; x=1734771844; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pyTYWh3zj2l92o0qEOJPcU0U+om/laakNd04Rx8p+XM=; b=WHr52y9BclhaBz2vvpP7Eyk3s36FgffaQZAQ1Am2MemAQlBFHcVj2L3pSuKyGCXEvD O6+o3/iSQvj5iE0V+4Xav5VuTO1uOx4QaMn1PRq5AE5kq9B5J/11S6PbQccxE6vnlti7 N+nVg4eZEt/CA0FsCDAxhFJigLnlL9935FtkIHLE6O6bT3UQ+llggdIpct17dYNV/Ifx ytvOkBf2HxY7sCvi/0wX7d7fmRnaQVRHlIlhvIN6aSuFPSW3yu/z/9nV2jwU9pvRgVF6 /pCrnMmo99t71z/u665189HckiqMv3Za642RJHPcmcIWmWd/1m7vwZ+2hwNHAgdboL7y 2L1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167044; x=1734771844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pyTYWh3zj2l92o0qEOJPcU0U+om/laakNd04Rx8p+XM=; b=W1Hdsp3Scz8NrdrFHU+ZJKNdXsX1Q3AmUydx9MMLHwDN/vNztjKSWQF0FL+qnt6Qs6 IbUirUBgcBDkrwvPIj70PfnISq2KeS4zQrI1f8Z/mffGVfSk8EoSdEAesbtOOJN50a60 gF4HkKne2ayZgbkM5iEyf3847uYSuccKr54sSYNjic4Y837FjPh82i0g2dQ9IHBv8/JJ +UeT1rwrBKyd1/btqM4KLGT5E3MIksb335gN+y6zBcNAdTfotLYX8yRZ2Nhc/Dp4ggnZ BO3Le13hnsjJyOs8qywuEAUEqN2jDseK0JJ0ho23RD8/lvz+xgpBAsiINWYD6Cwc72le JX1w== X-Gm-Message-State: AOJu0YyFjT3eVU5JdKoR7Sv7S9aZ1HvJ4ViHdp5P4fcPLtdIwQyIKfds NgtRHZB/Vp9iV7yrMDokZIDAQ00PXtxxNvYe58eoZbWATUOB7s0jG/DlCGBPKgk= X-Gm-Gg: ASbGncshUqeGFMVnHFWKONqDgPAjgOqCecdTOZida12yMVInKHHeeNB+8kBtxageCWM wsrwQfC4STiMUataLLSeliwvkyvXX3BfykW0XRdYf7lWOPtKsX8ZDqb2ojpwoTWARmwZ6MjX0A5 xPO7oWjTGjfFxVvrVUhqtP8H/Cx3/F+r0p9znY/HNnA25qWIoItszY3X+FxE1nw+TZ+ZEwIloPx Czq6S2Bt2bRkxhkmyYiYWI/UdIWe2IjQcDerXqZALuji32qiBIrreQZK5vDid9bsnYAyQsX0Rhu ECaeI/2WyQcXBJsC4jFGtAp+zaH2zQ== X-Google-Smtp-Source: AGHT+IFlpyRvtlHG2M4nIIBLJzVMLLLjWf8IFr835Xb0HMOnd3F4yHyM51MIPZOReaS2iWrLwbmGTw== X-Received: by 2002:a17:902:d481:b0:216:4b66:1f1f with SMTP id d9443c01a7336-21892a7c538mr78037165ad.54.1734167044262; Sat, 14 Dec 2024 01:04:04 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:03 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 03/12] arm64: pgtable: use mmu gather to free p4d level page table Date: Sat, 14 Dec 2024 17:02:49 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9984C8000E X-Stat-Signature: 9hu36g8mjgxefafibubcnnijybmngorn X-Rspam-User: X-HE-Tag: 1734166994-963515 X-HE-Meta: U2FsdGVkX1/ekUK25mW6R8WHK7BlabQYDKwMXOmVBdGZlvV1WGifqbhMoek5V3bmc6Hp5k8RSnaIEf45vT0NcYm/tZZi9ojeIOt2m0Dvs5KFhtX/RRfnko7Bs/9GiIJL0LgnE5AXCbPg1pxNKcUb9T0xwtnhkLiINttFBTHSYCsXv5cJ/fcEvjLSclRBqAj6I76ScRMpimpq1bZoqlDquQh+ImUURPdJwqpObKtCqgEA4VEcUovikja7hX08V6lBa9Jdz8yV5BuHCEu9LiClAWTsjNJ661/DNKy10NjtR8ueEhZBQKzv8MBKICm+8NYgffZFiieIZUNzFQEbPTlQInxJqyl1uG+Syn+IwQ+VY3TvdrCCeQi1rD/UQtzAAaxIvuaewa7gfHndYtcAnvBU00iGWTfFfpCJMXgC4eG8Hecy7YZ7Iq1bWubbM50fGmYI3URi2e3l2q8+ZQm7zpM7HN5l+g2T9P0eJBtLSTwpYsp5kygIZ1I5JfPHdCXr3p8II7G5ZIIMQSXO3O3IpMaCFQRMmAxv8IBMOuWc7+OI2Pu+G7j6QU9cPYoFtgQaGM5C7Ec5/PfZArp/6311xCk8dHx0sOATKx46ntc7pP8b1StR6N0Upeiu2Y9KhXncONU4FVP9bOESSQGJeVQ7txwjy8qNQN9LPQwV4O6npiDKA0hhuMUMbo6dCG5DXZHwmYv80LFqDkxflN5CJCNcqrhc1JAiQmYBwHaIBz/45gI66mwT/eeU63n2i1X0Tk6ueTOcrkWIlU1sJmxbsIzNDC5gCESQbmkHIHZNf95lUjQh6dIEPV+4jMlwJ9si4SWGUvmdnlWUyG76q4KB5Ko+AwxHYXfoJ/ayScERiKbwFelLLZXSw1aAHqhhFhPZCAnrNJGuhVq6a9X3LU7i8Y5ovdN8vzFPaKBUG/WraTBPtjv6+fvd3ZPC29H3lZFZVCwfsURxnnRHPuOYihXuKAeruaR CktuPw8f PY+T/3slT1Ass9riV02RCD5S4+Z7A8IwhnOJYEyyBHGZCksnhQeZSwJrTjjuTpB6+16k63PNgX0JNsBSQbbaAzEqD0QkImCFpVWcnpkzNwaO9v/7k02fBmE66ikLmlttBCsPL7SkgX0JxNBJAqcTQRy5Jf0zOYu7er7izXBeqEvGo+uGIcqliwzx0V2tTOXYSYjm4tlNyMlY824rOuThBaSCCnWXpBh17PBM/RCLtRGdOGRga8OxCsIbhyvx5UnEFdlvjspaE05cBRb0I6Jo1IQ+0n3IPS1Gu1BnTy2T7xO5oD2G+DV3d9/Tzp6mNOhSes9lh+p28vmNgbCnKy0vLgpWrtPa/zUEIDEPXHmXmX98+UZs9ZVstSAHO0g6M+9lrZYQE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000848, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 2 -- arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 679c530549327..58cc260b6bee8 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,8 +100,6 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (pgtable_l5_enabled()) __p4d_free(mm, p4d); } - -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Sat Dec 14 09:02:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C972E77180 for ; Sat, 14 Dec 2024 09:04:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EAE96B0093; Sat, 14 Dec 2024 04:04:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 173AF6B0095; Sat, 14 Dec 2024 04:04:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F06A86B0096; Sat, 14 Dec 2024 04:04:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CC04A6B0093 for ; Sat, 14 Dec 2024 04:04:16 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5FBE3B0F9A for ; Sat, 14 Dec 2024 09:04:16 +0000 (UTC) X-FDA: 82892977104.07.63529E3 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf05.hostedemail.com (Postfix) with ESMTP id A5081100019 for ; Sat, 14 Dec 2024 09:03:23 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=U6rnIxeP; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PLGKRzIB8HXo0VMNVO8Y1nbnLVdnH3jBTxV4FtFLzdM=; b=P+HHLbwgnGTi0j2ZSTp9QPiYE/WBECUMa/hlhjiCTsLfI3J8qXEP52xYeQWbRYGIoeSny4 tCSCHTMsXlf93NgwRIJQ28YOvzTI0R5bHChJ76JQ/KVG818xFO63ITaOUstMVQ68wCmhNv pO09AjaG42AVcNg7PXf05oomB8iIjhk= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=U6rnIxeP; spf=pass (imf05.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167034; a=rsa-sha256; cv=none; b=2i2bO5twtFGN5zrRUDcnMARo55irCiV5LBXTAu3nY0oA4bwo2zh3oAqtrMMrVAf0b1lgEJ 87TOGxnDs0Kx7l0K0dS6wIkrLIxrCssbWVQAbUzEmANV1VylxaxGXziAVVmqzy2VKYZwqz rQqng84bOLAsi9KbFVZUG+UtZeea0iY= Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2ef70c7efa5so1715404a91.2 for ; Sat, 14 Dec 2024 01:04:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167053; x=1734771853; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PLGKRzIB8HXo0VMNVO8Y1nbnLVdnH3jBTxV4FtFLzdM=; b=U6rnIxePfvfQHAroRyZ1q4GytjuozRzXhzx51RxwAwEbUKnAExPPCxKmZChZ/GsJk4 0/JqpyuYZzS2bHDQto2pjZV5QvdG0sxGg6zxWZZd8MyvnDDjN1qTBlzBGAhggwpLY3tz 7k++RczcuILW9eVt8QegVWDjHvckCPvyAU7rFc+H+7CtrNkyMhqYAV0086ufuwCixMXw U1qYdu7UgAIlGm2zfM0Pbx+NU2XST/FWb1r9uwo40BKRVNXcxuNdSdRV2AcPCraldpcS DE/B0Fd9k4CTC9Al0Fi4z7XPGr3j576lAwMggFzMQWK8RRwhwpSvECL4lhA74f9/T5M+ 0Ohw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167053; x=1734771853; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PLGKRzIB8HXo0VMNVO8Y1nbnLVdnH3jBTxV4FtFLzdM=; b=KYTEotQuukA1hMI/O8yd6SloBcTgHItZM9/kzGK1qFxGWDwuCiHIZBqEze/hX4ClhQ amevVzgqT1kibPc4c8kRWUbBgQZvW9nFb54+NZwvUui03H76LBANLBJoRCVxKUGAzk+2 OQlPSunqkR787pgKYo6z21wI3y98UqYZ/pWVF2VfA2d14juvbZ4t8Iz7NX9LHDJ53CgI H6h8hPiQNMW7Kx+uYaJJVBLu/GEzaRDy39ezzntrnSs3q6UGG10uX4uqTDI2NaTcUsCa avGSMRvCcUCHYx9EWxeMg5AiGzcpbEHygucjiCu8ugpKyLA4+vzSYgtONR44Gcwyd47F /yTg== X-Gm-Message-State: AOJu0YwR+srAkEAmvV7gWXpButB4juhIs2GjGVPrMxswPNzsz2mm+oc3 BcvQ5QUXrJLTgQykB5wRPaLTZPSCUMXpq9Y3tKaAX0uWbKkzkeo/RRRETeJtWFc= X-Gm-Gg: ASbGncsSvds855/jAJ69G2Y6Oehsacv++yQddkVK/pad7VNpi4gWteyiOSwK5bZqA+0 j7Luwofz4k021Ce7NIyFj7uqhnnZlwZ6wxrlpMJN/zhF2EgwrK82ldlHltjfplImN7PQnJsNtj/ XkJcMsqMsRP0Z3+n4I47/T5s16SDNOrsj43taNgZ7gN8S8TbqMzFOSUUhXdJWFLdegnPv7UzFve noCPPrfYNEDHzsn7K4IjTTNiTMSnbr6XY4pNKyac87J2W2F555bK13X14AdioPUpoUXnqADsdUB DaxdwU0h8syeCaPH4Rx6bjMtJSqnbA== X-Google-Smtp-Source: AGHT+IEUuhlzMDGPbhHcy72Znz6SKiudYIj5PWGCvj9lWM7sC/HnXgPB6JSnynFl46+kyeoAQCdvNg== X-Received: by 2002:a17:90b:3848:b0:2ee:d797:408b with SMTP id 98e67ed59e1d1-2f28fa55c45mr8606530a91.2.1734167053262; Sat, 14 Dec 2024 01:04:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:12 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH 04/12] s390: pgtable: add statistics for PUD and P4D level page table Date: Sat, 14 Dec 2024 17:02:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: A5081100019 X-Rspamd-Server: rspam12 X-Stat-Signature: 5y775sgfs31zxj7xwi99nj4p1ixzzj7g X-Rspam-User: X-HE-Tag: 1734167003-634067 X-HE-Meta: U2FsdGVkX18+EEDzi8ejeO25Cj+SCImhG82burs7DpUFK9EKvZKxPzWXoAYJbVX2yXqlFz4E5tvZhX3fETgpWZZD+cQpKNNt7iCAt6kWFIgo6froIAR3HPt6ShNGkXOUYkvMFAuOzKOkQJtArAfYSkz2pS1BRG00/mco6A5aa4R6nftBTC3zfY0ocOCy93AMdO8AAn9e8USsdWHiK4CRvdo9QGSa12RqV/RxUI329U3KNBkFsrOHTaeplaHLQWwzqjl8Hq3vYf9o39ytRluIx+MIweRDJ3pkiUZR5iuMIfVImBSos4rrpfdSzHY5V52q+BjEBk6sqosIlhlxdrTrmS5X5/mgbAwiMEH6Uxn400GJXBzsYqh87EHlksZYGlvnwrduCTFuBHfg0F4zI7yBYCDd7FWBBHgsRBrl5LJPAAKDVzxx7XSVLUdIA3hWyX5t2MZLAGoxvA578qlibB7mnySGiAQdWlddlNkoIRficrc/IRbEgJ4MfvFn2DrJzi/yJZpll50ZN6By1kBfHA871Xzsy1gNJQUgjrJgazGfcb2Sug9wXKXeKe7u8I2zxsqDvxHC+/4c+TwcK5EoSCGn3nEPUaohFqxQfhvpaoyA5Z1BlZwH+vRnSsSV9F7PJPdxNhRhMj48n8uvuBVF9tdT7LOplzP7iszlkUNyR84XEC4Y4rIptXhL0GGNnMs/A4G0YGEt6bozEDu8Ekb720fvZemhVXMpsT9wQg8gLVuOQWEHQCSfPqT2kiZztbZ4IyotzE9zDNniIEvH1ye5CJNcW1g9Y94j5Ry4MF7QyipFypD6dJ+8dOKm34hQeer4h70QV/H2D1Pa80a0Xu9J2hNmSGPUaINfMHSq0HDfGW+F96L9Aluo3YlJDj8z3D7Ca7ekO6cojKk3lEFhB32yMAwHavOKxFDnRuc0h2/wwh8ROMj5JoQZunE0jq9zMB3bZVi0hIODtl4hFllHEvoZwwE L8xmfmpK veDtzEANDB2PkvdM42zOkLrbsi7yivY2/Ed7yLYdKo9uW18QYCtqYEHeDYLcLhhhW5r37Wu259hUEkE4Q5LkobRslCU1FXsT7oetORmduWi4dnJ7V3ZpeXC2kBkS165aUarWIHD9tPwKy2aQnsQqUMvS3ICjxpLaKBWePtFqOqHNbJCdrDZQdFRBcw7coUxdYcrQJWXWeoI3+0YlzJdl32jw5AqQd5K8+aq3Og4lW0kPus5X3Y1va0FPa11ecQ5gjUOZoQhdwo2UN8sEaNf4erln+5WktMVoAiPP5NCtfcMV+yqJEbqxO4dI0o8BD8jqY059/3i2kth0tv/1qRYOQQTNy4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.011612, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */ From patchwork Sat Dec 14 09:02:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6299CE7716A for ; Sat, 14 Dec 2024 09:04:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1DF96B0096; Sat, 14 Dec 2024 04:04:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DAB206B0098; Sat, 14 Dec 2024 04:04:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAF806B0099; Sat, 14 Dec 2024 04:04:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8CEAC6B0096 for ; Sat, 14 Dec 2024 04:04:26 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 422A31202E5 for ; Sat, 14 Dec 2024 09:04:26 +0000 (UTC) X-FDA: 82892977692.23.193B482 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf21.hostedemail.com (Postfix) with ESMTP id 81E3B1C0003 for ; Sat, 14 Dec 2024 09:03:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cNpgb8wj; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B33p7H15CJn/2cs/WET0YT0xzjvXDMsJMJEC97/vu9s=; b=U1aUNNll8BrNISUY8lIN0ciUQDt9jv7dTUuYAKwpANAlrOaEqfnjkwnPkYV67R3tumkSbR JuX4eTy+GOW2pNoDSni4TAEDdPNLqgGOgcKgbEbptAMRNmbDL+RB94/LWHGV4qJVW/Yxog qh9PGM0Zpls5gLfSMwHlOR00bjNi4rw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cNpgb8wj; spf=pass (imf21.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167036; a=rsa-sha256; cv=none; b=vtbx5kyhcuwIK6hhJbEt9saVEXRPx9WiWotrpj7Hst6JIKJyfhUorihco+tov4iljjLoNQ r4JTqixG2SPEMhAn0koNPCG4uz47na806KhduR7LZ7Dk02MDjW/cCPomcunZRvTLXC0t/e MoI2WD9BLl063JM1WeU+X7bzrtNieGA= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-725e71a11f7so2783977b3a.1 for ; Sat, 14 Dec 2024 01:04:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167063; x=1734771863; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B33p7H15CJn/2cs/WET0YT0xzjvXDMsJMJEC97/vu9s=; b=cNpgb8wjqZ1PXuMGip23WGCoLBNcOE93Dha7caZazQ48NQQEGbGIuV9Gzhw9jxT8dP U8MSZVvgDidwzZSHsiTRfEGNhaIBXS4kghiGTzUpEo78b2n6LxDcRl2HcoOPX6J3dVlh Mz5w7wQqH+WS7naNzWbdGJi4+90rtWeboxGari8Ku3/699ZrC4aqq0hieXt87AOMD41e j/+Kk6L07dPgfcmF6XyKtkd7Ga83zTnzbp+TQJ8bqSaFPKY46MeZVi6p6eigdytXC9W9 wdTsYr75GlccIKl5PbN6oT11G36T0R3Q02giyNXlSl4nhAImiP5eIm1BXNmHZx4sQNYc M1TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167063; x=1734771863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B33p7H15CJn/2cs/WET0YT0xzjvXDMsJMJEC97/vu9s=; b=LHZTucdi5EbynW1WYIvyBHev0LdNdjPgZv/VlY82RmfKdw3bOSNM+ywimjrKxk4JxL ttVl++oL1Rkoq0hAocy9Pp/StRmgR1R1EyQblnQ0nCyc+PK4bG76mczmEJOw4FC+tsnV XRWYcAX41Z9R8DS+hZz8HOpCj3L8tdUjcgLAmp1hY12sBKxv6sSYr100KB9EJTtgHdQU ZMchj8AkXDstFlLO34OHrZ6KNsjVVn20W3YloX1BwWL4FTGmTFc8VdLUihLcfrZLPuY3 rdipljAJuytQLylyhdQqvhTMFp3E0BxnrsYZN5HC9g68UdfXgwu0/viuMRWUitBfaWF7 GGLg== X-Gm-Message-State: AOJu0YxWC3mip+IKh89V/d5uTTPanzVSV5jWmGlvrcU6AhrihNls3b6g 2m6rIP4uL17RdjXZ9FMLNqGru+v5bY8+9+t91oQUTgsbLb4Zf0n46rBtpYAXHHo= X-Gm-Gg: ASbGnctIL0tWIJZlt63z1eviWJOtAfJtvpuDJUKbIOsJrSbDvb8CDHOKcTV7xxrmx02 gu0FHtJOiSqoOexAfonvZko6YdqO3bdrZSmdEAmDTJ/gHsoLW8FSw5Bi2945LDK9f1lSA8mJHtS 3kdjtdHJ+9UF8veI0qtl2swk+E+EyevBygXulyvlgkGbkI6T6vkt39Nnjg+owLqTlngmdHPU3lW u1W0hhrLmA81XmeCjMNt8MFrV3+dPhZ5EJbXn5k2Z6pa9hqHXQ+a/l8binU/xZPlpePGQxBqXcC weFYnWE4MfH3FTgFJsoggJ+3Y0SVHg== X-Google-Smtp-Source: AGHT+IGHGsg1Le1TWBXvKS/11BgmC8ZOwqij0qU2dY3aJLeleELqBt0qnn6Oaqqw31tJpFU41bENng== X-Received: by 2002:a05:6a20:d80f:b0:1e1:b220:63db with SMTP id adf61e73a8af0-1e1e00879d9mr8179608637.16.1734167062929; Sat, 14 Dec 2024 01:04:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 05/12] mm: pgtable: introduce pagetable_dtor() Date: Sat, 14 Dec 2024 17:02:51 +0800 Message-Id: <7b17aa89b669d715b48e94ead1717c20b805074f.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 81E3B1C0003 X-Rspam-User: X-Stat-Signature: hwe4m5ur4r89htky3jgbcy93q6m7wd9w X-HE-Tag: 1734167013-883774 X-HE-Meta: U2FsdGVkX1+8EhBuaNx2xTjO8TBRfXvMDuOmqIllLv9hYa0dBqLteKXx2uR7dCSskrPiWMYdoBXXNHemyevj27HJY2PSyZiidhyUBiYlnEtcQAM3V05FtgiqFtLybQ0x6mN7J8o1JQjDy8z/X6RnAhvAFZW7fRYCyuU2olqxBVXfKc7+vCio5vw97+FUt+c2GNYs1nVr4NxSUkHH1R1AHTqfw7j6tjyGRaybeuxkuRyq2NRmdTua9yBgJcWP9PbpwhB8yARj1dmkmVPGdKrvLrhvsn6yFljcZdGnc7RtAiZC25hRTaM79DK4NKrfPmGwDO1ZP68F7xSeL5P35L4WaM9milsOJm20eNRBZZXzUCIpRu9egkLazU6+CUg0t4uWZvp83b9h4zGLTOoOeTwqlQOe2Wl+HgVQsixjx5AyZWWVUphElUgThZnUyC30lE3lahQ8l88BRTvvCvdTgxJ0He1R0lxHdjp0bbg+2wLAhYUPOuZEemgLXy4gA2M6ZGv6lBHuYVJKvuXgdkPWDYn0mcy3ovPwSGx4pskDhKzyd9TGC3QN7foTzptjJoyYlryP2p9ZEy7OM1aLDMEI9CyPeMAO89NKeS1I+AnJlu4NJHKUcCTVnH+8uTd1exr1cmnza04wYUGldMPB0MMJvUaQuFMQUge79HOum3OdqOrjOVyJMiqlI2z4uAtkCmdOSHpiChvY5EdaZRoLH8PAcetZXiHutc9PLKr4R561Ku720FcbvfhrlmnM9zmsZGE4sGp1WyroIne314vGaol7kjue4Yt1iCwyuX+xzA6eeD+tFtOYCbL6UBPZyYWyp4M1W9Y6QEWqe8U1z875zTaPLBofE7mYvQCtECs3T5zH/tKWPLvuVKB3T5eGiTJ8SP9MuDIVDosMEAPMKjc9VbQsU0gKwxJtpoYzNS8i9BwI0vndhLU1zryo1AGv/YsGkHpqB2d5pqebUPc4yabw1CL0UKd 5dLaXosk Av0QkwpAy11r1ZZofdONGEsgrPZjfCBiWlSk0GJn5C0X1b3p4dmWK5A0A626sJTX4ICHkE2k95clM8/oTgojv65yTIX37tGbEUVjnTCPISajs3kIvjXILSHgrgkXGI4GOu3OjLkOyeLVlYBY+b6Scm/PwUBqG1C4x/izbtkpXUE0au7iaSyD8f9cIesmu/JnjzP4AMIPSi8aY+wGpNJIuHK/S/etqJCuRg2PXEeyr2BHN+17+Nvp5TV0+QZqcnYD7HiOpTQnds1SfKOxyAevirwDziM1qTHEqRwAql3COgYu4AOWgv8XO6e3dqT4tPHHaBGWbdBuN1M9W5S2vngYro+/Jb8jaYJcPyPD1tX1trhsrIMs9R16/aGWWFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index bb6e1c5f1fb19..09c2eff571a49 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -112,7 +112,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,7 +139,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -172,7 +172,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -183,7 +183,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index b946964afce8e..74b6fba4c2ee3 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +141,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index dbf61819b3581..3673e9c29504e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -257,7 +257,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 807a12ed8ec96..497035a78849b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3055,6 +3055,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3066,15 +3075,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3151,14 +3151,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3169,7 +3161,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3193,15 +3184,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3229,14 +3211,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3245,14 +3219,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index 83765632e20b0..c00fa95464b19 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7058,7 +7058,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Sat Dec 14 09:02:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0226E7716A for ; Sat, 14 Dec 2024 09:04:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40B646B0099; Sat, 14 Dec 2024 04:04:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BD9D6B009A; Sat, 14 Dec 2024 04:04:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23C1C6B009B; Sat, 14 Dec 2024 04:04:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 012846B0099 for ; Sat, 14 Dec 2024 04:04:34 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id AF4A44544B for ; Sat, 14 Dec 2024 09:04:34 +0000 (UTC) X-FDA: 82892978112.24.D8EED19 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf20.hostedemail.com (Postfix) with ESMTP id E56781C0004 for ; Sat, 14 Dec 2024 09:04:04 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="NL8vPA6/"; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=12IiSMzMVIlfwdGc3SnaXSwRS1fKZAvHGusKFMV1b3k=; b=HXmPMOCliMpduGsIcmPThYmsHdzbWyXZpQFY0UMt68G9L5juYUF+6eUIkjgV0SzOzXheCz FzSf4f4sr6u7ppj13Omo8ghQn4T0/p29+as5GTO+bIuMgUqOtuW83QfQXazAzSzveuJKgR YlMSVVMK692bJ0Z1YCxDlR3q3pFTDtk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="NL8vPA6/"; spf=pass (imf20.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167052; a=rsa-sha256; cv=none; b=Id7PfEiFdcMPNH4u7dvEmA7AmnOIXXbEFjjU+QHn726Ll7Up6wvqlIVPBOGch3+PjUMaFh dx2HZU25mUKAZZGL5bDxSBXR7/IW7HsF+kbLHcnmqCRe2BeskeaBjM1WBGeI+ISTOfFnJA vPz5xIrzSE/sTnn+cBDOYczJ7JUvfLM= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-72739105e02so2791527b3a.0 for ; Sat, 14 Dec 2024 01:04:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167072; x=1734771872; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=12IiSMzMVIlfwdGc3SnaXSwRS1fKZAvHGusKFMV1b3k=; b=NL8vPA6/SotHPoWdVX/wBdgx87rt74vnuIUE7aHH/6NuU8C2lD97oftTzHkF26A5GA ajvvHkYUIKAY2u1n8AlxE/UGriJk3SYugYIcq6/iB/KPulxSU5DtKMFlgfaHiAzsVV2i anGblsrRp+vDq7giXBcbKMZcbsjiTk40Ye4uff1g8CKPWf9dP7fXxGmoe1DeohbxP9BC bxQeLRZhVQyMpTbJXdosMNEWDb8+/wacvqPJZZjFM3Vz2rnR1gfjoom4QcgwHaopb+NS 9oZgtB19uDG7+Pb+sDagKJOBeXjOIZudbi1HEbqoTH6mYUF6pwlskkZSA3wEidhJ2afa +dZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167072; x=1734771872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=12IiSMzMVIlfwdGc3SnaXSwRS1fKZAvHGusKFMV1b3k=; b=IeRJ+BiyumCU6X6Z4gQDzTqNDhMHDQZb/PYW8tKTg7RqWipo4OPoPc8bzzWNVw8poN 3skEAVvtBsKR2w3VYY4VHQx77A7HUU9YSeiG5QYlUGtR+Os0/iP3myYS+Ct3CgDE8xmR jW1jCvzataJ6Z+iVQrTYO9GTe56JuU/GmeYwk0x63OIfff1Tgs/ngunPyb0BSTFKSUOp hER/cZ9wqkeZrYVhB7mUhImCmhMp2vxfxWATiwDdeLiQHE2OXaUO5jskqND/vQdpf8cC b+SxOBz/ydKrHmFCjrmJnOPPKNdhd5uPwQYoZcUSnJVaK00SedaakrtKBAZaDnXqJLBd HIaw== X-Gm-Message-State: AOJu0YxyDAyHwriEfGe8hI29hfNzPqnKGIFN9hWvWm2mFNpJ6AERHjs6 3wqOnHjKxfGVBC8kcnQa6la45Z4gQtKVn4VCYkU2R1kDHBh2iw56GXr06seoKw4= X-Gm-Gg: ASbGncsjqIPkbg5SOv0r59Eo5jJI8RVBLYJF1sdiM5obI4wvtmXwNX5VHi29WSIj9JA S/4Zil2TQVVWSmXnlalcvzU1ePoEUV4mmchryq7Xa/coEV7uexBn+9XZxrFc/KUiNfVHJ94a7jA ZWSuB8ymlmDdwfB2mzJYJ+yxbhhE/J/L81zUKx3qZJViPrqFx2Vz/UkkKP9JVPQBjEtcXK1F1E4 u6jm7ijyftetlBwUeLN+Xx5i+HBSBwexQNJSC5e22zMo+iDxdTzHmCmpeEQ7QEnHKi51IpSJX9D LK8ywXLyYP/TiXBMHRBcknwPJPGxRA== X-Google-Smtp-Source: AGHT+IFvnXDbPrP66UsHi9s8CFtvVszYyEfjge+aAOFm/P/L2638kVYMzofXCSp9Xlrusy3CzYMSYA== X-Received: by 2002:a05:6a21:33a0:b0:1e0:d796:b079 with SMTP id adf61e73a8af0-1e1dfd55614mr8533024637.17.1734167071820; Sat, 14 Dec 2024 01:04:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:31 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 06/12] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:52 +0800 Message-Id: <59fd7e0364e8dd293132d81b91b28694e70263e2.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E56781C0004 X-Rspamd-Server: rspam12 X-Stat-Signature: geekwqjjwc5aggis8rwa4ec1pa33qhnq X-Rspam-User: X-HE-Tag: 1734167044-174685 X-HE-Meta: U2FsdGVkX1+++60c8M0KbL10x+K1tIWKfoBbLDCcnKNFR1Hhg4jjd+mDMt10RpxRniO+OsjxtlJep1S0COuNWyChOqXBTtbWnxHF9cg6/kRej+5o091CusyfDRtJ5H61Q/2t9ldPiLqg6Uz1su7vdzpQfgMk+4tLwGpGE/RNg6v3Wmoq5PUtNfTCGPsG2QNvz0gcxk/thkz6+xOe9AMAOXX+sbViPHbi3aj/8cyL5W3vop3MHYxDNpg0Ku8cO3wzLictOZiG3lNd+LyWVvQijxLpFl+p44rnQUrTLeJaepv399Sx8wntkk00n685drvs5RzItsjvi9l6COWNfvQ6BO7XmuDOzsTWNVL7oPNuVCqOTz4MKC76KGpeCpzmIsXL9l8qXvU+zPIRi9VR+GMAbhDs7KI7dAU+yv6z7w/OyemtDKYPdFXYmvo6Nly5sPuTNtSGux7TqYFckSC9/8yAZNm06f+/eIY+fR9L0ep4REELsGWuEF4zmW4KFxRsBuHnroROim9YQcdNOf6lg+PIzGbYiMXVW9IjVs8x9s6aW+cewwpbpkOlHJ9WTlE9N8jB0i/dtltYBYiDruzUZEju1HsY00rDJmYFq5TF5pt8isLTLjMbPOL6E8vDnshpSMbqQEa7Gcoap/eMVbh27z1B7OOdNz2sbXm2c//fpLl6Lqko9PwRNcyQC5OY/gbWVpyfCv6JriUe2VNNgBpouiCrIUYW/Dnon68R/p+QP/WpqfwN1Unqd+vUNEYguQMAO+PXQ3APtJxc4VJESaeUuom/+DxO40QGNZjq4hux9OMrgA2SMsPkSVQPnJcj0EexMqg8fgFv0KTsDj77V8RL2WLiCQYwDKxGxqCWrA8mc1oy6jEBh6GrQzrAJpjbcp8GYuj3LyxsE2c2Z0ByNosGqfggCvmKB9qfQ0Q5K2GJkd0/wOh/AUpEoOXNNEnO2G8Fz0xKjd5z/uf8sIFRRCYao/4 BsWjf2w1 58PxYeMsw2Nifqow8dSiI+RhVYxiylbVaFVPGZAHKilQcYYEfRmsKPtjIxojzW8l8ELtg8w1M/gru5f1+S6YM1H2odO2qTVL1mqsPbHpWDBCj+IfOzIRyTxOLUM1HOdk9jGtjirmUF8a8XA+1OFlyEAhNycw3897BkhnCIOvnbvY/SAxAsVvd0WKYnplkryYcgjyIkr9jIQWIx6kvXo9nQCql+pzckLyfOwNxN0jxgqBwZfrxrnxQEbMOl8epsa63Bv9vgdBexk/McOUSnczwrzjwSgTdu9U4R61ekwemsNUB3Nt3XRygIEjPufqT1kaFcQcwf5yGN/vHgQvZhkP2BBgoJv04++DrcXS4jb1dFseGtJYmG2FHr6yy5fV4+WnEhdgIdYWmzwEu4ZlbPglRgxkYnWoOIjcodGuh97tYhr76vmI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001711, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..264ab635e807a 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,8 +43,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd @@ -61,7 +61,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Sat Dec 14 09:02:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 958A7E7716A for ; Sat, 14 Dec 2024 09:04:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 285346B009B; Sat, 14 Dec 2024 04:04:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 23E6D6B009C; Sat, 14 Dec 2024 04:04:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FE6F6B009D; Sat, 14 Dec 2024 04:04:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DDBC16B009B for ; Sat, 14 Dec 2024 04:04:44 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A3824B0F9F for ; Sat, 14 Dec 2024 09:04:44 +0000 (UTC) X-FDA: 82892978784.16.F0B6DC4 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf03.hostedemail.com (Postfix) with ESMTP id BB0A82000C for ; Sat, 14 Dec 2024 09:04:30 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ERDOtL5M; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167070; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+CB+suh8vMLKY78fggEtIk5gD9ivloFZvOHbrupXaAs=; b=cI/XL3clMP3gG4q5IFuOP0BJl2fdloecV1f3vU5bQSmTduQhZ1Z53yxI776yfvWnfDwpdY wKsraMSs1uqaG//CRqPfo1tREo0MQZNcPsVDKqQmw4Im9mVV/F990MLlfLEfojLAH4zYKw P/r8PYnvwXONrR3cwuQb64dkpBsH+ks= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167070; a=rsa-sha256; cv=none; b=l8Fev3/Mj+fAIkUGAI2XidzeGnghJ5twE3bNHbEFH6ukfi2/vkm4jrYOs/dl7DXD7lmWc2 sYVw1FFJfvh0YkYCCrJWO3VshPHjfBoR432Rrh/eTzr0cYyi1GzymYH0XVI8Fr+BlNzkuW DYzkA1rkKgZqjtMIdd5UJZKVuV+yWYg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ERDOtL5M; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2164b662090so20836925ad.1 for ; Sat, 14 Dec 2024 01:04:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167081; x=1734771881; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+CB+suh8vMLKY78fggEtIk5gD9ivloFZvOHbrupXaAs=; b=ERDOtL5MbyH/mxE4JMNpszg5DRzhii4CDFbYJfUJiFiQQD42UFTyezu/nAQ0/rNMpQ NevV2meabLnGPct+qEoQrtrsGOD8Dw7XDLfVks4lbxNLRuyYekTytX8nrE7ETseotjJo 5+IdhwJ0cW8kszhGp+UoSjvFcpNqocu1P0/0EzgL4N0oHhm4NxUOIi+ydmamQHyY41jg bUoX3lUmIyPjN5E0e1QadG0VhxTFr3JMwZqnGt9O6cid8YEuNphxl6jJk7e5IMRTBEfM uL8d37fEiJwnyY4OR/lsbtosVsm4VGrJyWmCGfnTMk37UdkObETScpmuA5upEZaHOLHA Lk+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167081; x=1734771881; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+CB+suh8vMLKY78fggEtIk5gD9ivloFZvOHbrupXaAs=; b=djK1DiyduZKvVU+g+BSU9mxeQ3ibXUp5He7znVPwukndlStNF3jet1ihprxt2RkcHl 5g7jN59miKcdHWiP8LteyWwx5AJtYLwzwJFSbIMG+BWVWDvUkJH/tQ8C9j/Uz8tNzr1C 4jJNArcJmg4jJfz9PBf8UZyAFNCIb594Ocp832n0QfcTEjPEa7N0QoksSvQHpoel9u2I GH8Vqk3aZ/qxn2GSCaRorfvVi6mZllth7yfHLfGRLgMbivb25jY1hY2mNa7a1RspIIJQ Um4i1CyeRrAYJs0fOn/U2EmX1kZHL1LDGQG64M/WrOpeOiptLvMSveh6QdElQXap9H5M F8Ow== X-Gm-Message-State: AOJu0Yxkjt+itrFIgH+9mi/7zmGb8uotrQQPtjBFIIGEW0Lxrwdq+VAs +cjY/f/yscX/mJLuaRmMwibiN9X1gtgDJlSEp9P2CBL6iMby+dzcLwyLpewCObI= X-Gm-Gg: ASbGncsLqiDEBFKHlD+MKK6nGkoSJyAkJOMMql2NdrhhnTFa6mpOt/vV+rFGHL1+5MM wJ4yHx6zMFxLePMp/wYx5e0fKNNN4t/LXFXpxDgJi4wV26OJ0++uKwE49FegY/VKlOdcqQSkZlH DFYknwUHCyYOc28BEXY3RoLUMg+205OF+mweJgIs3/h5ptrPUEpf2K8V0PPYbNo3AHCsLXOqCPZ g/mqSY2NNd6UhMl6buIgQUca6RSDbnlEA6mgxnjaTVXGzwTe/94wQC+RKiAiArA1hjId0Jd1WB4 +GNKQ+G4u9SCQMPm/zn16bEvooKm6g== X-Google-Smtp-Source: AGHT+IE0QPwbAOQZHxlo/n3+wbqMrZuKXsuKTSsN4acRfuabgW+JUZvSc3827znPlMKVF1C6eV/6aw== X-Received: by 2002:a17:902:dac6:b0:215:8847:435c with SMTP id d9443c01a7336-21892982123mr70599045ad.12.1734167081724; Sat, 14 Dec 2024 01:04:41 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:40 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-arm-kernel@lists.infradead.org Subject: [PATCH 07/12] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:53 +0800 Message-Id: <1781d3bab60aade7d64fbeb6682b9e73bd673d3c.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BB0A82000C X-Stat-Signature: s46nicnuyi1m5kn5zpgdwtkfg4ef3p1s X-Rspam-User: X-HE-Tag: 1734167070-597027 X-HE-Meta: U2FsdGVkX1/zEXalu8Slr55I1DzSb/cIanm67t6QdPnxhBjxiVA5QVHWLBTEsj/NAsSx0USMeOf/cvWNz62B/7Yxy6ZhGdLpUdWxxmBBawaxX3nHsXwfR9N/v0i74x2VIVlqwu5WyZMqbrvtnewCEHJvJswN0EESbI7iiMZiV2yJAmETV8jNmJv8CwJDYq/2BCX2FhyF5BM2E9tKOz7cBCe6RjT9GCksr3XYytjXBA2x2b8IwwKEN17VFpwWEXhYUFFZKu9RboT9Y7BsaxwRlDCJHhXj+ZkE9bmR4SZtmcP5dZvdsLIudUh+qeJEHIpI6cDoht+RNyKVuBlWAS3rYzfvt/OIdsIf7BOzEPkw9oslnQK2edRy6VzNmKnYjvkM8ckB+zi4XdPkl3e8aW32LLffO3XftKi0UedXWgpogQygfbScrCif3anLlfh2N0pRzotMUdRDeqFIj4em1lvLm80bfP6tTtlm7wa8A4fwIp4lJeokcGJXoUQwwX4vAnvT8Pv6rgRZSNIuPY6kz6/V124yC6PenqVEYsMBkub+YhAnRp8lV5L5+GcTudkpqbcJe61hNfHhMWMw3kMaQI0G3K3hpPM3McNnr+ck07swdO7FPoe1o9mcwqTWISIR0S7ncBgl6iT7gF4OKr7bG009mtoVYf6G8K/2X8d88BUtthEekX0Ko3egPZ7DnnjnlK35sWpq7joolf8f+g21G2D8KaOBGqciVccKZgBqIM/E5OaYKtKVugxUwaO82n17edJsp1o+ZmTuH0GY9ol87msvagiUDWDZPtEfCp3mX9SxaSpWePBLju9s0QWqWrCHHqgJ1Kma6crEwIqfMYyDr0P9GrddhLR/sx7qBuAdsl30ovQKuDMe1BLpUQp7aC8+JMjPqnSbeP0XC0yKFEspyW0c2yXPIc+jpGQ5DXBJKQTICq7WcyNhkB1uz/kT59iwnUPd49SdYbVXe4Drt2HlsML UcgzskkY D/Kv0geOHNzGRyqHx8uSMc1i6uPfNVV7CRtWHOOBsfjf5yWJJkbD7Gh8Skt2btM3goTwWQZgpxbKGHuVNsXsExsSJxuUVeESQ7WMHgMlDLWyNHXGEciG/4+AK6LR/Gtq9khEOJnH0i/JadB7zK6Jvay5xRTaD91j6agGma8ZdkUuuCv1cXRqG8cv7h9jMrMm81lKNpZu7UMez2/w8l1DXu1xRs5K5WzG6I8D15rkLisok1HqWUT+CxLWkX06X0G3QL4JvwxcPeia9gTIGHTTJDGQQSilaSD4Qhcd/RUduRwxq793czsDbKuoQvOD4AHRrga3A5ujg77sAkyXnSiAKS/985XRgTMejI3kFkj576E/nXv8G0HMxRGuVe8nE8LVa9/Bq44SoPztBJ5T2Nsm1wHLlhYRQVTyVYkGbbHdQVzjjcLw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001425, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Sat Dec 14 09:02:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1919E77180 for ; Sat, 14 Dec 2024 09:04:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56B9F6B009D; Sat, 14 Dec 2024 04:04:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 542CE6B009E; Sat, 14 Dec 2024 04:04:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40A566B009F; Sat, 14 Dec 2024 04:04:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1D4DE6B009D for ; Sat, 14 Dec 2024 04:04:54 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9CF84A1CBC for ; Sat, 14 Dec 2024 09:04:53 +0000 (UTC) X-FDA: 82892979414.14.0A66CB6 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf16.hostedemail.com (Postfix) with ESMTP id 988AC180007 for ; Sat, 14 Dec 2024 09:04:24 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Dl3HHgra; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167066; a=rsa-sha256; cv=none; b=A5S/olfzUw3znC1hpfBP7VQ5NPYuRPBYcdvgSZ7mYk0KSot3Nvtsx8sDRPvkO8Vfl7zXgT jL3s/jsW5KonCCb7NGabsmmUMfsa1FOJGuUIxkdgTUNLZfy+ceTrRTDsLon8DLarUmDRHw zDqyqMnLPuHnxH1fD/hICMDuq62O0ps= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Dl3HHgra; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167066; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kxcjg+YJiLOjdKwLGOoVz8JWl+VadyRpr3tadtBHjWA=; b=bJgDFVXGmHPlcSLt/xqvyt+mWGW/Ah0Q9eY2TiIdGczQXC7fEQLcbek80RBre47KEUpKJg Mv0HGci7DIJjNwX2iF8gsit8tMBTqWmvhELXCN6edre+cHgjaUrHI8Xm/SnITEamluIu7l iknQDQvgxWS29xWrEkEkhpfc3jrGTJA= Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2ee786b3277so1658834a91.1 for ; Sat, 14 Dec 2024 01:04:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167090; x=1734771890; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxcjg+YJiLOjdKwLGOoVz8JWl+VadyRpr3tadtBHjWA=; b=Dl3HHgra7cfcfIEnY3ojCZapxW8DdRaKYDxCSuEpH5NAfXLRZHwIHQM7kJvWthpf0M If7ASUlgV9ydAGwgqD8X3ZFuwae05NVkEgHLPJNw9DVS5hejMrXwKa/8eyewa0qjbz1I k0R/eAfdvGNsd/IaEEI55MNKTLoXYDUz0rl4ygCmtVg2Yd86E/QsetPz2rLUEAVY5hzJ QqHBdkasHjIbbvP0GtJF4E3aE2JCrcDqjRsSKvTMf1qzUR/IwoVpKjxGoU/dtwT5R9LW al3Zx3yuEY5yrrJsoxBYaYiubWam3YAYUa49z6SdMDq4lXxlkFioluQlwTUhNLuEk+PF nr6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167090; x=1734771890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxcjg+YJiLOjdKwLGOoVz8JWl+VadyRpr3tadtBHjWA=; b=YnbrbM6hIladWIO1RX5kZH6CfX03efIbLGAv3izCFXkdPnw/JFnakSoBhVFBRI6fb1 4IJ8oCQoVEksS6HfcYSDl8/CzmHu/X3Uk0HdSMFy/x/e+zsK4R6J2x8v9GWXzYOJv+Fx pNk8zpedNDHWIInN2G27lg/43vJR6TF7NPyvdDzY53NN0S8+mIIs9efw1WiD2P4wNWgx BO3Mwmt+hmg/P/ikfCahbBX1xPbq2B9DySB0PQ2eIbBxQpf4zJqrPeqJka4LbvWwgeGA Ef0TLIjTypk8t9eqIM4U1sauBEuITHwa5N710Sb77Zge/8LXN3loCPMlQOW6LhUJSi+i esPw== X-Gm-Message-State: AOJu0Yy5QuoaJ0gPdJBkW0Wj43K0UgALk4s3lxnu7TuZcCgEzv1tOUtt iTdNRuwTGZhILKPiNZ4zpkkAtsGgCa9bkDB2NeYPXvKuQH0NU+qahC6saWZ18/A= X-Gm-Gg: ASbGncv9pyuF7Pr4/sp/54OPlwvfZiJ02/4jaYQw1P5k1u9OXEPHDalBJe67RBDPHjn WnjjFEF3tv3oktdEljLsK5e4Q6R8EDz2gKz07KmZMjXMijYADhXXkTLNOarB/wVk4zr676aHsdg pihaGAjUdh8ynoQljeyIwmfReNE45dE48DihvrFGMmdxMm4sIZFVH6WYgcZ3JbfgRI9gu9Dx1vE KvIKOdvZcIiyt1EoTWhOumSkcE2EslmslXTPoWtOHynUAbbopwJhXSbHaanpVwFr6TUuBAiUgJ3 y11jSzOUEdxxiR/Q1ky2q4s4vO/n1A== X-Google-Smtp-Source: AGHT+IGmWNhert5ebnmEBEANehnSTDBWCLVNkF+841k1TH0tjB/D3zS21KwZ8s5YJg+vyRMQfoaR6w== X-Received: by 2002:a17:90b:3b52:b0:2ef:2d9f:8e55 with SMTP id 98e67ed59e1d1-2f28fd66b3dmr9591223a91.17.1734167090585; Sat, 14 Dec 2024 01:04:50 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:50 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-riscv@lists.infradead.org Subject: [PATCH 08/12] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:54 +0800 Message-Id: <1e73cf87193d838c64e402896b4252875c212986.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: sh53s5ed7jqqs4zee846p1joboga1k68 X-Rspam-User: X-Rspamd-Queue-Id: 988AC180007 X-Rspamd-Server: rspam08 X-HE-Tag: 1734167064-216656 X-HE-Meta: U2FsdGVkX19yA6ZVo+/29rJ0UMkyEZQhK07dFkAEG4BsCszKBqcgjWt18dn8DBMQiZs6OV+uRdF9PhzxrbORbmIiDgxcvi8O8o4KVuT3LahbqUAR2cfXwKE0aFQBCqfErFGuJIH9Ualoc5dd0JpmndxuXPsd9M11aWK/qldeRPyGU2eank4eqUYkuZ2+XxC1x6neF7F8eQUXgpseCxz7iukWebGZOl1fVTC04PP45yK7UX4YerwkeNwB3BvoHErc59Oo0nHdH2rodHG1eG+8KqadgeqUsBASmYLuQ4cI1bO/bWw/IPPWLXYe0u+7TMGxVyljJiueNabP9XoijH/4cZDGwjBxdUUQH85Scd8K29jL1X+x+COJb5MxOk2WYkI+QEZpysiLjwQsVAVxC3VjVwf/6Hp3eguzovkHaLe6fRTzPwWCVyVEbtgnGeSE/tWmei9ZdxNHeNWhjONhVFUmA4/oRU7OvDKGKx9FW3apruQS7j6FN94xZLt4z8F3EFTaEeuMAAoBF3+HwBYfVzKjdm8yAgNihj9RbZx6d5gNCEub34eK5BS5PSnfzoa7ZtE1rVtvqNmCKqXwKc96C52ost9qQUvp9eLCby9sXnUuVa0LnY8oiqx+2OOFNAIKzRvGxFu82Hdmny+n68pVRlLgijibxqlxdytirnEvqWUImRRPbc3SZ8cdsdKWtvENy1cipFnTe+7tD0UEo2rTgfmpE24FMXRFu192vrHecLVK/ct4eijxyauUYQH3HCaMIbG/lybeQztCyfXkydk6+azLa55akY4zJO14L3BFHNXFZScMYnWByTrEoHEEQbL8TOcrWTCL0ItTTx9TsjTeSv1baQIMGIbaNIdLa97yaUxjc6ZoiXfkBtDtZc42OFpeoOPd8Osom2hlRIcZy3olf4oj5+G7YBo9rOwMLK3PDIPr0Wn4jBbE9dnTvynl/fsVQm7J72h8vWacd/wHuCORz8M v0T1imad pemFIj2Tb7RtO+8KRBhFpOvgdVep2rqK+oXkIkj6rzyeLjaney8SCWCgK5ebAY5V0od4B4r87wuvTe2R570S3N667Wnv57KpkYjMlKG0JhxE/ws1acaU9T9I2sm0eQcyCl7hJ83BEt28ken0aSVelkyl5RubvT4pwHx98svgheYyGqkqsE93bZ5UMSG2z+MZd40X73YhNkIJvFKzfZD9t4hclb+6FJEptls2vGRDj2H7nC3/OjjK4Nga16tsBLo5AHGn8+zKI9KEu+VQbVssh+TegLqGpMnrFaDiCQk/1/74O16/cctp8LbhtSiKJ5YqVMx+wkv6SAU/jWklYpiZuLpjmAZmzbZHwbm1clADtsm/JSvPZWXwSy9aUGMtc/ck7CflbLk369s+azhcuGleEYh+cmX++TGSVWRqUP99v5sOCkDU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000834, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 09c2eff571a49..fc50d14010246 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -18,12 +18,22 @@ #define __HAVE_ARCH_P4D_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -109,12 +119,8 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } #define p4d_alloc_one p4d_alloc_one @@ -136,12 +142,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -170,10 +172,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -181,10 +180,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Sat Dec 14 09:02:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D44B7E77180 for ; Sat, 14 Dec 2024 09:05:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A4B66B00A0; Sat, 14 Dec 2024 04:05:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 654F06B00A1; Sat, 14 Dec 2024 04:05:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F4DB6B00A2; Sat, 14 Dec 2024 04:05:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 28D366B00A0 for ; Sat, 14 Dec 2024 04:05:03 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BE6731C6CAB for ; Sat, 14 Dec 2024 09:05:02 +0000 (UTC) X-FDA: 82892979792.29.FD869E7 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf16.hostedemail.com (Postfix) with ESMTP id A21F4180005 for ; Sat, 14 Dec 2024 09:04:33 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=THeJCdVb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167076; a=rsa-sha256; cv=none; b=c4l3SZELN5eU83i52/HQ4yE5Z8jS4DLN7TUu7SQu1A3GuGTePQZ8uFJkHnQ+iHi7PiWNYU yrjHwVVGZnDy1ROgEXevqW/MmK8QDkiE4tMnHnYar1T53bcsRZrGzyE0yKXmwmlBJrpBFo tlK4OorRCvaJ7Hi6LJVaVwOrHglfk/I= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=THeJCdVb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RMCudVMtlfI7I8C+EWzj20f+Ae2SGvA6pNu/2M7K/mw=; b=m16WMGFAKdV717uk3SEuhkybtRpwcIjeGWtLgMxihTK1iup6U2JTkwY6+HHMj2JQIHMIPD Tfn9gHfsOHfcdzFSg03FjAxuDadGid+JORklQWcBiSJKBTJzUUMLp6oYaROSQqYCS8f370 VeU7oc1AyoXf+/znB+CfVGtpUnhqmJE= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-728e1799d95so3059221b3a.2 for ; Sat, 14 Dec 2024 01:05:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167099; x=1734771899; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RMCudVMtlfI7I8C+EWzj20f+Ae2SGvA6pNu/2M7K/mw=; b=THeJCdVbwCvsy+ZnLSfI4huQmpybCMZHwoqseyMcrfCiWQPB//QJyHAJpG5Yt5ffJm zlewrmh/iTRT3ZKOgQPqzBEalJUvbYsYPrQAd9aFpPldI8x90wrbgSGScCyEvr6GASYg px1lTtm6sYHdj52NgYZzZm9FWjMIYQ5EqcbRukEAodAhQRw4Wh6vEA7aIUyuYwZMw9JE 9vI3cQzHH04KJBIJtQWneOPqUEq16FvItAaHlgjDDz5V2zXDd1KVTSPtfV/hx0/jL1dl 6lVVGi++2O+AyBU777ICHEJQOE9bNQjfLieXjTEgpHH6uEP6CELVNythbydw0Zy4wxR9 4NfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167099; x=1734771899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RMCudVMtlfI7I8C+EWzj20f+Ae2SGvA6pNu/2M7K/mw=; b=J1ZajFQkGAdLc2z8ia5zNh0YRTIhWHqRaZ2/AJj9Iz84264ifiEX+CV4fRnNFjihEe WZbbl73Jy3v2uhY80Ulto3qjG7Kzsv9F9NQwTJYcdjViHPosXMym4ES2XWFJyslZurAy bMvo1jOG3th6KO13LefFBS21gVXZR3Lppk4SACX5DxIwOE4U7mBwSl718W+jWtLce6Up ZgY47TgVKLFqXSu87+vd59rx3dpMNsw+AZ7NA10XZq7+fa496EJMhfKPOK/ZfEAXBdzh z+zfHxWssNVWtFvXCz/zlb7y1lg2j975V5XRcMvyhWetgmwLWO2MvVwTmOp3WmB9MyUI inqg== X-Gm-Message-State: AOJu0YyzVh4vhV0f+nAPumMbqmgPpPmXjS35VSJGzVe48JpqaS62TUZK FoKfTqIEqPJMjwb9oS+senrk/3kbiFiXRGXxRuaBnS1+wl4BIN8RQSh54kINphs= X-Gm-Gg: ASbGncvfiGeDg+ON9v3rdGFCOb20K+AMURNP/TPToW6fs/cXVqYhQTFqZQvg+58/V6B wSTmOBhicetG+IuoroFiyClguZH8TkCTePMk27m0D7ADFXG4Ctx8qbn8Yc63+4wMkHCGyX7OH8a fcN9w1rqjZcPD3LDkqw7QF95ScJEh5HGOIVXgF2MBEIKsJmQ6ctU1R50fh/p3KXrKW7NbC0f3im 1b6Ss6mryRkPI3dnCFvREuuIFxmFSyeVwVqc8waWCowAglvG5Ix3Kht8PXq2eNDTOv1F0DtSJAp XI0/uOzGhWV19djIk3YKUTOM4dg+ig== X-Google-Smtp-Source: AGHT+IHNImhO/lIwU1CxZtuKJT3uG8XCXYTC5vUDdEevSxX9VuCSFNAuRKGmdb9G/Urv3IN3FNQNtw== X-Received: by 2002:a05:6a20:d808:b0:1e1:c0d7:7502 with SMTP id adf61e73a8af0-1e1dfe1d265mr6847659637.37.1734167099625; Sat, 14 Dec 2024 01:04:59 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:04:59 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , x86@kernel.org Subject: [PATCH 09/12] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:55 +0800 Message-Id: <01a0ce994f5d475230117b2991566cd7e2ff1dcf.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: xhqh8iri9kd8cswt7bpkn4u6h1xfmj73 X-Rspam-User: X-Rspamd-Queue-Id: A21F4180005 X-Rspamd-Server: rspam08 X-HE-Tag: 1734167073-739830 X-HE-Meta: U2FsdGVkX19g5lOonXewMG7kQv31wIA5uRSXaoj3thpaWQ/3A2CbE8JVIsgWIToacKqAc8+PzeDu7iP/ffcuicqlGq9vnXsMs0ZxiPZHxWHFzCuu3FCXiuBVkaK4Cl594nBZ9P+7/4yUKR4en8MAAP4gDK3/YNDFg9v/nb+RV7RwZBWhSD192Vuwio5h0fU4I+4nba6rGoF8yT06fR5wGG9+75E/oe6JdmxrjLy+5EDY4nvhn6a55cfG42AqFGuVOHm3fPLBpOqF3VXielY8U2zyEIT4zHhlxueWEczdfPHrCYpYT+IP0iShVSGMoE37I1l4dsF9e5dXntazxXwpJvOPUEo6TdIPB7a7/nQp31CcpGulQ1JSVrUT7toFNWy+elOiQV9OXFXpRBOfTHXeM2GgRXRYyvf1+YBqC+PmAEar9f8ydOSR+BYVVqXJDuHAN1oO9wSlXS3bPvZxbmvTBudo5PvC474yzI2ywi87LTOWrGQBvq1AvS9fpoVBuTtcdFq7IWxk90MvRn2ijsY2lA89p9SUuBlxLLxpin+8/xoDx5WVQcJSx9XQKileTdEvKVj+zzcla7xXw2R+Er84Lywp7lc8rJL8pWRxHBuUGk3nzto3RB6oKAoywjmM//+/4NGNDV6TsTinotgA3AeOe+2NQbWDMLMFtQxd58y8vGlLhjVFRoSYRmXBCgA/CgyJD6OvDMOTtSMjqQJQhcc1r5SY5ctubcIVq/f+bvCBKCNMHP3EUghFZRAM6S1Mk/hesJ/wwTQBvit3FegsXyWURWP7l8h8ekT7WgKUtSRe1fjyPXON+NuowPnPu/3EV3DrORx5LIjpGDzIR5RnqoOuBcQvOmmRINnSyBMNF03CH3Q/anrHM7+/W12kZX51z5efcC4O2WkQvg0gaGJEOYTyL+K/1Uzf7w5ZLNTKmVGHy823bK2+BtVekvGyStliYC6atgMYCS9/YzSvpbH+Ya7 9maad7xM H8YBhxVEp841tFz8J8zNb3yT0J4PovqSbKS+uZrblNDjuAbsg+MFy2VcCkUqz4Kq0nr7a3mJ0EjAcye9ZscqB0UhtkFdMPVTmXIsU4MwJECqyc9xOlm/pYe9s9ifJlDIk09p/4MjBafSKupQBlTegf7aVpVJrKpxo4h/kE2XnqSC3T/nrucwggJz0X/bpcd1hYRvFHoip41wRCqA2oueJ+//1cvHmshe5ZLkjqqYUfC8LIp+XfLUtr0TpoNUIZbMN+v+2S5Kt44sYG4l6YCFb3gXNoRj0uSMOqMB3lsCj8xm/Iuw3D6gT+VYRMAGtZM3GKAomwNABhLVdLNDHyUxZupqquDrUkrJb9T/cUmUDLXSB3TQni7GPU9OLGxoCYBFsIRgZr+bnYmbCQlw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 17 ++++++++++------- arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 12 ++---------- 3 files changed, 13 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,27 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..46d5d325483b0 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,6 +62,7 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..a0b0e501ba663 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,6 +23,7 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { + pagetable_dtor(table); tlb_remove_page(tlb, table); } #else @@ -60,7 +61,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -68,7 +68,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -77,16 +76,12 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, virt_to_page(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -94,9 +89,6 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } From patchwork Sat Dec 14 09:02:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ABAFE7716A for ; Sat, 14 Dec 2024 09:05:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 125996B00A2; Sat, 14 Dec 2024 04:05:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FEF36B00A3; Sat, 14 Dec 2024 04:05:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F085D6B00A4; Sat, 14 Dec 2024 04:05:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D07716B00A2 for ; Sat, 14 Dec 2024 04:05:12 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 88399161989 for ; Sat, 14 Dec 2024 09:05:12 +0000 (UTC) X-FDA: 82892979666.22.212D075 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf01.hostedemail.com (Postfix) with ESMTP id 79CB640007 for ; Sat, 14 Dec 2024 09:04:50 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RPEIBTOh; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eMqrm4pkcrZJ7adK/0JGqqs1dXiKNR49XOWaF/ar7JI=; b=pS0jwXQkq7pGzeuA6zxtd4BXgGEHm82ZM4dllaPIRlYQeJ4tDuytUpbeFD8Fg9KqyFTmd5 VxcHuFjf0VDmGNvjz9Eyhhf/JXCzgkeaPcoV9kNIZlKtFk/fDWpOd4NNAwkAUBFB8k3XGB gidBEfIMMC4wfCnbWyqi7QJ3oBSFu4g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167082; a=rsa-sha256; cv=none; b=aFNGIn/mLE9W1BqmVj+cyBojdjH73C+HezOc8H08B7y368kit5Want+wH9CGxnvnnnk0XI Z3fI9tC2jmDKOs/gGXVpueHRGASd+hOJ+gFv65PZtLBK2KGHeoCW8z/mOfO+eaUUE8CNIK 6vJXvqpwQdsCYBZtfmLHDrfEAUZ1+h8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RPEIBTOh; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-725dac69699so2193755b3a.0 for ; Sat, 14 Dec 2024 01:05:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167109; x=1734771909; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eMqrm4pkcrZJ7adK/0JGqqs1dXiKNR49XOWaF/ar7JI=; b=RPEIBTOhCNZJ3DeRtgiTdcA0JvaZ52asqSAbZ6slDLNprGO4GjhR9p7LQv/zjbKRfs BO6m4fSp/Hb5ikG1JNSufRvl4vLiNLOw1rLukMTmaxiXNZ6ozfn0akUzsf8iFM/8dQ2v rMzf9n1KeWnTa5Tw4WHyD9geBVkNBeuDNl0fsNxRPiNUzmbyZPEf8PIFUl6WBg+l1ZF/ T5xQueWcSZ1eOxOWuAR5vf1KzQneWAQtmMQJRrNKhHqiIAdkHVgK0+zkBMA1ZbzfFEnF cgOo0ag7FhKs2LoHe4ZP3Ni8FOwZJ4RT/UmZISYeYb5bq4Ks4tR3nsH0NK5S25jj9rUl A/Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167109; x=1734771909; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eMqrm4pkcrZJ7adK/0JGqqs1dXiKNR49XOWaF/ar7JI=; b=XwNnRwid/N7qZGDLK8HzshP6UsTLg+l3Kt2jcpoh+P2vcVdgLIH+jHsq44jVo2x5MH N7Owm18IE8t7w/nOynQj300UxPuisBhd6J19FYePe2E9XKTvC04L9+czpVAUhdyHhikS ErdQt7AU14bS7vzn3MYst7MGsnVy7FlnX0LDgu0nCCR/PNa/fzo5/7mUq9vHwSMsJj5J vZ//IZ5md75imp3RqOQYS4iM65AoxKSmOdO2RVJv3A0jHuiY8yjZzydeS+jM0YiYVUtP BHEgmgYATI6xOLzkWteR4vDqhXmAb1zOJaXhxxFMW/tfITjwwQCZCoKMm21ZC3i3Y3lM 3o/A== X-Gm-Message-State: AOJu0Yz7kpWuaEBFutkhJus6j08BXY9DfU0jpm5aVRViDHuCJQhVrJhy Dfoyf0Bmf66giQ7VoXsoE1iicmUviWvJu8fdBpR5nUJCkY9QfKPSeC1Y+hsAsXQ= X-Gm-Gg: ASbGncsgfTWp2fewnA6PEzfQtxG8KwS1PWV27AiXBTdkeEmv+1Q6n7TdFacRlehVIcg jxgVZ4H/rG5XDeQ9jLX4enxWYU4C/QYl+U4AGpNZama2DP9hGieuPRO9DkzdU3/N3XOOFrHAFpJ 4B+SSYJybOm+u2VTPlRBXbWDRWIyl0U6dK5s2ou9QxzihlwVb4TizaVSv/zcCWhjFlCPXDLlksM dCFfO+hs30n0YMKYPq4y0WFOwGEfJzendMcSis56E39eQUsWqIQV0ALd6gaH3lVux5xqZiQSu3q Mvg71dN9mB3r6CLIT1cZeoi+Pjeszg== X-Google-Smtp-Source: AGHT+IE/1mbbP4uNRetgOIypjg2bvSDj3GWH/cGEc/xqzPLinfL2kdCDxCigFtQu4L9Hg/0wQ4NEdw== X-Received: by 2002:a05:6a20:a123:b0:1e1:96d9:a7db with SMTP id adf61e73a8af0-1e1dfbfc129mr8345681637.4.1734167109554; Sat, 14 Dec 2024 01:05:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:08 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH 10/12] s390: pgtable: also move pagetable_dtor() of PxD to __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:56 +0800 Message-Id: <1e87efe43e9b8b679384e947dd8503715dd9faf0.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 79CB640007 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: mezk5egpftxei9i8o9k617hiwmnmn711 X-HE-Tag: 1734167090-497541 X-HE-Meta: U2FsdGVkX1+pM1WW2WB/u3PJOGMDNsMNjuzyNfM83Oek4ywExnwekCrgAm4GrPqr5DyB3vrgR5107NizNNN84QfU0JCEpfQC8vahXtiJAlWCVuVR8q1cr97UcNd+l9x4WlKk9emmo8nSC40GXN9DpFGRbv3Fg2Spl3iK8Wn45ubsd6rzAwDd5FT3RhKE0g8amnta1qzSRBZInJLmHl2ejQbYB6lJEPH4d1UEpWkNM9+LOdOUupyZCu76JFjNF9LX+zHtPOwLCRCtw29ox+6S73H7D+fMqJANz6XTGmDBHlQCwkki/80hoyV5DdGB67QtkCayiCgvTlYgoLNf2ejS35Y+6rOxoWldzjWTVmaB8jVO/sxkpNb++9WO5D9+bVbRmLTMvJc1baLUq9hqVZCxZWEVGr5mWPNHW4zz6vtB/usk74SYz9pzSVZBYF1NK9vnuhwF4whkin/xCYCvYJvZClf/Ho+cGUSqjYGkvmcvY+Kr13HWvxkdKRhEtEyHktPSLV8x4Os5SzENhKYAJGsSw6p30eb5EXFv8aM4R3ScRTxVUh31Nhwtm23o7sRDtu1FVgSFHd47cb1/3OCYeyWTiJXJKuRwhGTduD+LBQWywEgXO70ioZMZmer2FCZ1d9ffDdhjGEwsd0MeYv1Kbipbo9bPex+7k8WMDE8jMSFPHFDAJqPdqrRt8yui+xfjvzhrsJgTACKDvlZyr9pWhOD4DigK1uNGCPDrmGVMVvuJ+EW2o2VzMHCkJACrqGO+45Q9Vr7heeOBCqokJCQCEhRGNxutAaFvnNaPIWEoiqblBMxDl9Ve/4U67QCw7jnxJ1cImJ9G9kan7sswdLQaDlZqdV2gELqoViYfNypCN9PU6EofAeo3gxiORKZ4XjVIemutVFLA2aBX728CGmAjtRxTWDEmPYBpAw7ywmm9VJ5qdsOlLhqv2y2tIpTCS0j2Dk45/O0cnDkeTY+DUhL+iEA zJotF70Z Cj1TfmsmsiaBLSjRks5LGwqdOrvD1DgrPcl4JyLiF3K2FeT8Uin5Dxhn/g82m++SwHynl24pXTIG70VysjvLpnrmFmBizZgWdlE6L1gSHsdfPbQr1a+LNR7H+XifErlsKIk7N05Z5tBJS0V/lDeBGX7aO/UUnWRhuS+1zXeKo5CN9hmSt5r6+QkjJLWTIYtJLbsYXBS0bSmqbvIRZadcaYdXjBUl/UoZpLZTL7+qi3scuVNOMVg73ht8AHSBh2ZO1EZ+ssfMRfkWsWCZNR4QqLNlBRZBgNiOn+3Iza9NkpxZ+4jhVBzAqV2fpDR2t6SSncPitk9VD5Oyat7uD/ZCsm1wpCrYwpx2R4uqGbP+zku1OLonyv1jREsaCTOxmELrbRG0FxWm+YgpheO8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000066, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In s390, the pagetable_dtor() of PTE has long been moved to __tlb_remove_table(). So similarly, also move the pagetable_dtor() of PMD|PUD|P4D to __tlb_remove_table(). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. By the way, rename pagetable_pte_dtor_free() to pagetable_dtor_free(). Signed-off-by: Qi Zheng Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 74b6fba4c2ee3..79df7c0932c56 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; @@ -141,7 +139,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Sat Dec 14 09:02:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23E8EE77180 for ; Sat, 14 Dec 2024 09:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B12586B00A4; Sat, 14 Dec 2024 04:05:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC13E6B00A5; Sat, 14 Dec 2024 04:05:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 989786B00A6; Sat, 14 Dec 2024 04:05:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A12C6B00A4 for ; Sat, 14 Dec 2024 04:05:21 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 340AD121CEB for ; Sat, 14 Dec 2024 09:05:21 +0000 (UTC) X-FDA: 82892979792.06.9614977 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf28.hostedemail.com (Postfix) with ESMTP id 5985DC000C for ; Sat, 14 Dec 2024 09:04:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VWjaccTh; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tGOtYoMgWgj4YJiYefYK2aPYu+m8suzmBGDTS71l9lQ=; b=n24ZNhRI1q8R8DgeJEWDGhY89wtmeu02TXO/uiXuraMQB4zpaXtrqE6FFydYzecHyicQg4 lR/b6BmQa/55TotCzTLowbXfsEaZ6v82sVc9p/cHw+mg5JCIh9gqV8BRiwnsyDTRL7nOhC fDvslmF1JfDdVxQ1iT9vJCWbdfDk9I4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VWjaccTh; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167091; a=rsa-sha256; cv=none; b=cx9Ys1lSH8k/5sucMVkC2uRBMaBVx3t7mbs9j74j/qNicImhWdzs1IODEMEVDhtSD07g7p 6ThzMKMC0mRysSrGbGj7xGcv9xIWWAkkOpIv+yu1+p0pjW6t5tzCu8X4Y13FErgUtlBRHB sYjfxaP/3GCzhp4/yQvIiZy1snWRBXI= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2166022c5caso20755565ad.2 for ; Sat, 14 Dec 2024 01:05:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167118; x=1734771918; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tGOtYoMgWgj4YJiYefYK2aPYu+m8suzmBGDTS71l9lQ=; b=VWjaccThzyC8QDvOjlc8o0T7TEPSf+LhcSNDYWH5CaN7d0r+lzLvXQH2SFnDgYqTzP d9ZQw84EQiBVPKohdSfpTNtrADBBolWMxDp8CIGl7yxa/VDhqgCxkWvQpz0o3+A0mYx3 g4aUJ0ztbJX1gv30nB0aQ8ay172cCJImK7or2V1I8I1NNLH2vSiuC5WQzXwdafgiyow6 VFKU/g9B9DMUWAa/ACa1vDisqNMVeOL/91fkewu5fVsqu7njUoZKZ3/B+1STDM0ciQze cIGMG+wUaqFWldfrIPMj3YUWs6wMNbSHqSMesFBsmhmqYMtfmLOc35bcyhFygbMxYOD9 Ws+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167118; x=1734771918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tGOtYoMgWgj4YJiYefYK2aPYu+m8suzmBGDTS71l9lQ=; b=nSVCsad/A0EbhfC9MLrki1gBV3EjyxNuLZnthz4wyN/Ib/OB4UwKOEu892qj3rId5d iEu+DgK3VlD/R4J7o2O3U8KTI0lcYgX25H3LOm3Q+XBz+Lk94brNfUdlkoYqyADhj7Yf 4MuJw5jS4jbWv8cDL5YAITE7nXP7Nq0VSpZBnfwUWGtl2wUe6psxx4C5W3s97RqQ2Rus qQ1yvnwoE0L2Ap7sKKgbfLELEKgpteU7eoBFWd+RuuKr2F4+aYUUxtoCZiryZaQ1qPd1 WphKAOBF9s6AN1a9f2mqquSjPmQiTa8qEy0BsJCPFxVxAjCHkKam3Bli4xRBOgI+EiJg KjCg== X-Gm-Message-State: AOJu0YwzF9QD9+kMJp3IxlJuM1zgixAeyXxSJuBHML6GwZUFV5+pUG5Z 3JGLN2R6Ty/mPejdP2cNPAPqwd+eua1c29yS8iSecpIWn1T3poKRxZI2xhfyQ8w= X-Gm-Gg: ASbGncvhK2PlPH2ZvUaZ4NTjetPhcIhmzjZLf4np0gD7wQKaLTRpcFqnKJXFtr/UO4F eVrvbY/JqJIUhLMElk6FBTvSYLfXe2+4uN09PrKSYe7FmxKtisZjSHlr3GKzeF1xkixAXC0sp5p LI7qht0QVMrhcgMPbn/cvNR3M/axY98GM0JIjrlribdhcUs/4rjccErnbZ6VREj7yBlRkhInEh9 HsCHwnoRDiZy0E8PgFoYgRZYruOgE3dfkfR66gqcBVvdtN1kcNzqAgMVVlWdEjkpjPSLrEOHemN iyApIFiJle0kMUGUlimNpTrrFZV1uw== X-Google-Smtp-Source: AGHT+IGZK6CYdPWEqKPfkUj35U2xxEx7kipqiMJDua7xnmtA6POZpy/3kuQaFLc+awGAy7moPmF5qQ== X-Received: by 2002:a17:902:d481:b0:216:4b66:1f1f with SMTP id d9443c01a7336-21892a7c538mr78074805ad.54.1734167118105; Sat, 14 Dec 2024 01:05:18 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:17 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 11/12] mm: pgtable: introduce generic __tlb_remove_table() Date: Sat, 14 Dec 2024 17:02:57 +0800 Message-Id: <271e58cd4ab808c4f402539b76d5916924e2bc6f.1734164094.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5985DC000C X-Rspam-User: X-Stat-Signature: yzwn6c61ca9u4b58jt168h74ez5dt9j6 X-HE-Tag: 1734167089-245093 X-HE-Meta: U2FsdGVkX1/uMAWM+CWExRp2akakDJ/p4ZFtxiGTImTYXr2pDCKyE27eTfehx2mBbEbXNA5QHS1IYjg8pvtK4BSwfIbWFVOScuyLu7r6I9H/RDoOBoC81Vel2Wtb9HlZMrRnrZG9BzlAocP+nyDTKJuXe5J5sjoCaGCveOYPJJJMxieuIfR2cMRLJtLrs2aLlMISwojQhJYNvdOXEqkBpNv8LOKYv3QrcDrRZeH+G03acmauKJRzWhzqGJVmF6jjj9LDiHfhvhiikcLRpKukgBCaWpPjwWv6iG5KafBdEl2LaX783LCjmayucJ9z70HErV2c+oL/YzrQM7AzXydQpmMYuOlNzg7HkcTvJchqWfYWvzNWNjoth/9rbIsorG5nyG9WBc+sO/U8qRWX1LZQYPk6bXSAiDWVFX+fmi8u8iX9+WjMXFUbS9sV0NR8eFSnYM3y/dqH2eZzorgfVxDv/Mt7Zel/o0UsBKcSveh6wFohNufUonwf91qrz2TLeFPwtWG4NeqbTM4iN+oC5Dzq9QzM4WuNKpkrZPrjiqX/itttGQ/wrgqiFSBVF/9fHWXmBpRPJFp8pQj7wT9uSMDMzPg/o/2wbGmkJHSgKUIX8bFf5qKx7pn551Y2ef3jHYt7uvO7l7sgCAl7MrCJVCI6hwnkcrBTGH+SzAs/JF55tP5qgRbgJRSI/UZJlwUK1NpCmE6t8e0F84yu3ksOyDVRiwwTZ+1rdqnzu1B076OhwVKH06UWI9K0kxzsgp+GdSsTrsMQI5WfLX5pdE6jsBdtO9jQbmImG6bSTxRS6e3Al0g2a0zJ7LXW5FDZsrzUoFt3xDl4oBdV1VeAFOK28qNI5QC8Yf4PH3PN4mPyjRjBmW8SLaUKIpCdPiU6Yo936y8Kf36ojAmyQpR1hnVsDvE3HD8F8wIRAgDfiUQx3zkDDSYdP+dJZjSsgcV4tigcaKGH1QQUG45AxKL7qTjgjKG N6Q4yx2C bqRDsr2IlANQPdInId6Or3zFFrBPQHfO4CIz6e803wB37aCYrV6ObFfjo7lshOYwIQJEEnxVlNze/9WXISx/uyHefQMttjQodOjJsI/MkADFIRbuClHFdE03Ye5NvJm7b1rnShCdR9jrqBucc0oeEEXrlv0oNBGjLY07HdKvA2GWL/+TWEGYNadHxLa33Yxqr9RdRk5SmatdlJw0kTSn48x9JwFVwlnsP39HFanKbdG13nkw8kN/LnXTW6OBHFgYzFe8vIfCYuIojeAnjT2IMrczfvYHhNhxcQeTwlsEbtictG0Ou6vgdUQ3DCVN1Cb/IEqfnxHUGY3JwSNmIkfodPu0KhxB7Jtj3tZqhWMpdcHVvJMbLkBWJ3mCSG5ZYqBi2+KF5OrLboElUop8/KV9eXO2tMw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm, arm64, riscv, s390 and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. Signed-off-by: Qi Zheng --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 1 - arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_32.h | 1 + arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 10 files changed, 16 insertions(+), 55 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 264ab635e807a..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 79df7c0932c56..7052780740349 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_32.h b/arch/sparc/include/asm/tlb_32.h index 5cd28a8793e39..910254867dfbd 100644 --- a/arch/sparc/include/asm/tlb_32.h +++ b/arch/sparc/include/asm/tlb_32.h @@ -2,6 +2,7 @@ #ifndef _SPARC_TLB_H #define _SPARC_TLB_H +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC_TLB_H */ diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..939a813023d7e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *_table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Sat Dec 14 09:02:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13908437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C09AE77180 for ; Sat, 14 Dec 2024 09:05:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A2786B00A6; Sat, 14 Dec 2024 04:05:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 850EC6B00A7; Sat, 14 Dec 2024 04:05:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CB006B00A8; Sat, 14 Dec 2024 04:05:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4A7E06B00A6 for ; Sat, 14 Dec 2024 04:05:31 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B571E42921 for ; Sat, 14 Dec 2024 09:05:30 +0000 (UTC) X-FDA: 82892980464.22.0B53955 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf28.hostedemail.com (Postfix) with ESMTP id ED17FC0009 for ; Sat, 14 Dec 2024 09:04:58 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gRH3IFyi; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734167111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=Vsk9S+xtGChOliqDoLnSkhLADA01uASwm4V6kTdHJADgeP2fqVwQjDwV2yKJAgPEEVwiSY bXW2/GxGQp5q7z4NBKuir6DRNCOHELE502Rbx5GtkFyUSuWp4Hx8D/ZxnpvS581aLJT9SH /ofPCu5YLmB0Aoh4uVmEF24pOtkimws= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734167111; a=rsa-sha256; cv=none; b=oCeaBOfPXkkntCZW3G9WNYIj5lSvUWDxF/CfaKCq/Mc7hknPxEm7ALdn1pSraUWnGU+PRL SaCFBOjUHt7tlBpXi3RnS2Tns32pIJnaRbwqZv63Z0NHmLgzBztB8vPpQl7jWxHmPgJVf0 Y7BWdEyQF0dJS2h8vbbLA1ZQJek6wkY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gRH3IFyi; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-728f1e66418so2185629b3a.2 for ; Sat, 14 Dec 2024 01:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734167128; x=1734771928; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=gRH3IFyiM9JRze8fmJbQDcFb+hLkcWBdVUrw7m6XIZ5YiwJgzbEs+PaYgxkuW1Grjj W+E26gUhX0tMI5pjGrPOHrLk/BrOOMc0+JSqoRKeWwdy+mpo7wuotYm7hAIZ0Xp9kd+2 PhZINBHQWgHK9e5H5eDjxemxfekRwgLtTaPnn6lAwucFOhA8wjE2zEVB4vDZkT1a3Yf2 9DraEY0mHFlMuBKTn1jIHirClPHDWzTI2bpNz62wyMZJGFYuU6G2EfcXMGN64h35DRIu wv8VZ8n0asJCY+AGIlZN9e7Un5ODhev+paiAUvI1FCpGOZs78d1A5pO2MDMf0Nggeaoj HF5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734167128; x=1734771928; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAJKsDdGNb+ALwQvMYs9Oz9Vq8VGrLJsA8GxB+kZmZc=; b=ZLOKqfEZRaGk42Z4UBAGDDiVRGhq2DQAWQ6AKhOLjKdA9GGxJvgN7s7e4GA5R7TaoH jWs5t57q4puHAbccAcsSHwXi6F5t4Vb5XdL06J4YbPbAqKAMkB9aR1SHpicPAGG0bP3M px2S698gZva17gFXZ/Bmca7p9pF5Ii4UzFy6yS+i8An5UNY/jeDOc+3p/JbkbGU5jWtD /oxK3lPFrS8wfs+dRgUno84rV2gqhXtEZk6mIlCYsJGWUqGnvMO63X/4cQCxqUEpr1rh zjM/3xOUvpARJ2RThF4JLa9S4VW0S/ghl1bSM38QupIwnLYzJLv4Zy+eS0sDYR2uCzwk F/yw== X-Gm-Message-State: AOJu0YwQpKvOjJv3+TT7z8HqJ005YJ8rZ9w/QM0ak708ujWCC4LeCIN1 1MZ00uJVg2xfOvukQhc9ZZg9jDO2POvz7qKk9i6lagg9TzfqMDZMoaBdQ87VmZM= X-Gm-Gg: ASbGnctrMUvaSMpw078GxqP9H1p7G9/tbxSOAmDHcn8lwnWNMFWfmhXdbGq/F6lS0Hb Gs2uOl5itYGTdQfiIOMv76XPOqRLiNaFzhdn10F/aMlENeqI2mA6wub5UZb4yJdSNYaKEF7PlpY RMcu3i+YGyDacmV4JpbgFQ9k1exCybNbzZ5f6u7DFqcDkF5yKdWF/SELOzgy26kBmNeSKF6e2Gm XQivRTXZJeSaY3Qq3VcQJ1UhUhFrIp3kojPUg17JdarijGe4mOFcmPOirftvwqaL7rib4ZSv0LF 3zU8Bhb2LmtQCLGbRubbvQGYTCrf+Q== X-Google-Smtp-Source: AGHT+IFivM0byq9BUMDoda6VIidtD6lpjalZ4ywP7t2KopoDAAS1RcnNTtREGYaK+hQvD7u6okSdbQ== X-Received: by 2002:a05:6a20:1588:b0:1db:e536:158e with SMTP id adf61e73a8af0-1e1dfd91a2emr6805131637.22.1734167127753; Sat, 14 Dec 2024 01:05:27 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([2409:8a28:f4f:a9a4:5d46:4ef0:daa9:1db8]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5a90ba0sm905731a12.16.2024.12.14.01.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2024 01:05:26 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 12/12] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Sat, 14 Dec 2024 17:02:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ED17FC0009 X-Stat-Signature: cgahbdcmp5oiqfmhwxiaqqoesyqnt338 X-Rspam-User: X-HE-Tag: 1734167098-420660 X-HE-Meta: U2FsdGVkX18ZMF3u9qwGDSBjuEig9Q+f6jhZzJqwj9Bh19jhPXLaFZpyUshTRj+S63l239F5HjkDF8v0qicXpvr1n3mcB9IbvMobpD4C2N9pbmQ6P0XYS7PDud0mOH2GgXNc3olr8hYxEiiQ/fiuuoFlQxFc4sG0cDp9Iu+OpBeBJj2Ez6qiBANbNJfCosBSZhpW3BnY3ZwFWlG/55aOyOvPLZ9q+gI2bMm0iXbNST/Hk1IZs5mrLsCFlT10u/KFWT/p1nx3+kJMfawtYX4WVaI7bIatvckG3lEoGQGjJX6TcV15gD1qfrQfT64Xu+GNg4y5CWN6W1YxycyKRp2Z9jkQEn5VqI07TgiuqR54Qm4T+gw2B38S/w+m/cPHyTnUeGu3DX5ETr/w6D8xSWpW7z5R+Y2XIxwkgHT02LcXRRuHJAMHzrtiapTQzW1TkQZkPdzvzc3/WEV6uFszf19rv7jezlTh2V2+JWMY6ON6lU1obiD/jh4/mo/D+K0gzkue8fCcXTc7SfSRshmc+WVkKY24O/U80GaKXwbKFc0epIdbMc74hcMhfNOwDJaB5lkbQsoeUiZWauDQJ4AKKhTZ2/1vH9b8NKyBRUmMIJnKzej0t7VJ19FK3LTSfLGOxkiSLbqnzJszSlO6qL0aGeZ5+eJagKTNrjoV2TbmSHF3zGws2/HduYSq92o8m41pGMRpdYEicd3Wv3sG6/Q4j6ueT+84E7fVw7zoaYywrr9+4SeCtouf6gai39PB6f9WkobS3CMXABzjtGuGUo0d+FiOW1WchQTuLrBnnDGlFx0c/4ZXTljMdk1HC5A5M5Jafp/w+hjH96QQgpfc6hqevkdosrjHuIq2cdd3R3G+rb5RJdbcrgphwUYkey+eFHJidPDjq8hDx68lh2ZNOCUKD0x2JB13NeH9Mu0f8/ER5C2WD76rzo/UAVxGHZBXrmP2AidGXgo9nw18eFqLgMsIfbf S+yRvJMy XmlsQbj4ETljIHgElCKOwfVa8iQr+yi67ESPwPPFuTBF5p1Cu5E4jH6MS41JUJ8oLRz8JRLgypufnbo2nWGevac+mcAKURDVcWg25b/hVSpqtAcUSujAjGgyiFvL6BJRpvSrHgHcEQBYhwxDPVYlDrDt3LpU9ZDcysnpE7odXX4saGeUbluUYDLq5VW3g9BM/VHjeyJ5WCRMs6/2XPAK+5NF11L2+2zS4xVQcmiMlJB6HP4thbo5YAeqhm/7bppGciqZ68RKI+bmhgwA7P6bNbMv5wIRqKv5O/GuAJrXKYm1TdeRSj6bGxGhQGsiogurZDg/J9DjgB4mpw2JkA+vv+ljxu79/9UXjD+f5GNBrtOHJhRYUgLiYhGBeLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.005605, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) {