From patchwork Wed Jan 8 06:57:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BB95E77188 for ; Wed, 8 Jan 2025 06:59:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3B056B008A; Wed, 8 Jan 2025 01:59:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEB3C6B008C; Wed, 8 Jan 2025 01:59:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8B5E6B0093; Wed, 8 Jan 2025 01:59:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A995A6B008A for ; Wed, 8 Jan 2025 01:59:34 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 36953B0524 for ; Wed, 8 Jan 2025 06:59:34 +0000 (UTC) X-FDA: 82983383868.20.7F6384B Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf02.hostedemail.com (Postfix) with ESMTP id 66AAA80004 for ; Wed, 8 Jan 2025 06:59:32 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=I8MwC6rz; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319572; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qK7UmD8A9xmARwj2p/zsqM766rIMmkR9OD65YlbhPf0=; b=6KFg1AlJMvBhZp2QxNxTsCR0KLZ8J98P+B9qotTHaA0navzYEYIrd2WAap+T1R1u+XP/6w XUXvJ1rpvQtkgcv0ooOL4UiMc9BWg6X2MSn53nGS1Qj1wnZr/PQD4M68STzr6yTaVTiHJd uZQcHQgB/m8P2HjcdIKzBhUUDcoI8mU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319572; a=rsa-sha256; cv=none; b=i3Fk8S2oFdFrL5B0Ozbxuk5unJCoZYWIy+w2EPnm5tZRZ7PYji47Ywy1hqmDr5x3+BxCdl gnlyQf9g4nZnKoQzdC9gPSNYuNTgX5mwGUa9PboE78gM5IucdMQ7A6+GkSw3CKu9LuU44j XgVs0irGcm8VKr8APU64cuODxa4urmE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=I8MwC6rz; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21a7ed0155cso22810315ad.3 for ; Tue, 07 Jan 2025 22:59:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319571; x=1736924371; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qK7UmD8A9xmARwj2p/zsqM766rIMmkR9OD65YlbhPf0=; b=I8MwC6rzjdrnS52laBMcfGAT3FaXrBxW117ffQWitPnFUqwV5D4XDz0bEyrIndtEE6 YgACaj9Xh1pW5dit/tijK47nIpOI5LGJlCc1gXTXIxr0F89j8vrF4CgU/1z3DIRW/J8m djjoZ29/MvqLZmupE0W5sOsCzPa7d/OQP0YiHbm9Ft2nM9sNK2Ihv+SYJUEXSfTuyCH6 l95bf6CJGI/gDM4HVAO+JblGiUyEuoVrZEKM+jh8BoGMSS57fbJ6zRoEPOIIIYyLZyjk D/IX8djpDKGpJup3/pmiL9MrVvfJkGmyoc908h/sYfdx9VAziUTIJnDXlO37cosKRv0g dR6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319571; x=1736924371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qK7UmD8A9xmARwj2p/zsqM766rIMmkR9OD65YlbhPf0=; b=JxM5IUd4DmeLQ3epUCruNwdPPSjW3l8CBLV/7GwmN/3RMwY8RkfXGWVgE8y8+AmcHa wzOR8jWO24I2Szj+cvzSfzZdm7Dy1sOmL6Pv/D2fDXL0lP//xeFvf3JPErBWwiPxZzaG i2WtyeOujAE5x6zi4MiUOtsmkPYtPpYYqRPXM5pB+ltdKnrFS55KABUKZo73xDrw0MZz VuAF578BJL6s3qbUEIJZSHDr3VTOmkmqE39N7lOKY7xoyHCKGJLNQUDtzMikIaU1D0CF Sevz45RpFdS4RJD/rX+/1wshDuwwyxTQI+KBr36dPaFvQ3lS7vWOQusBezy+jBQKSnNn Kynw== X-Gm-Message-State: AOJu0YzYTtZkWJleacHbWqc+eSqJdu8YERWv5Z8VYYabj60KuxVwFUuR J0DjENLtR7DKElBOQc5GmNYWDOmI0MxuVVGVHMYkG9HE+ZN2lxPHVQE0XartUQA= X-Gm-Gg: ASbGncvRbTQSjrMmJhaumGMFoK8ZBSnv0UAtA1YNDz3HawVr6Q2h3F/+vprVmHlHsK1 UDww+NTdozztc15RLNDUBE91lc82pLrHG7uVtdzgt1SHz8fEtqTx2axx+tXoefj/Fl2PzL1E7Ye JcQTov5vOMDbg7OBWGG21OY6jOTBktdfpajGkYJF6c5qUUNP9OyqEVd4VNM28/R8IC7HJsP/d0j 4FEUleAvc9HLeuLNy92BFuI9ITHJ0Nl1Abt6hAF0JsPrEWNTZ7s6uRAyktQAvQiCenFeunuBEvQ A20CuXEcxWntSOyLA08KerYFUZA= X-Google-Smtp-Source: AGHT+IHPv/02/1SYwL+5wvpHkGRfZSWRMZ218tgsJxwry8Gh8x8rNq2cUbRiuSy0SgfT7Mb+CNVIfg== X-Received: by 2002:a17:903:186:b0:216:34e5:6e49 with SMTP id d9443c01a7336-21a84001250mr22590365ad.57.1736319571238; Tue, 07 Jan 2025 22:59:31 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 22:59:30 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 01/17] Revert "mm: pgtable: make ptlock be freed by RCU" Date: Wed, 8 Jan 2025 14:57:17 +0800 Message-Id: <366002e0af83f0d5cad3f356db036cb6447492f7.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 66AAA80004 X-Stat-Signature: 4uryyrx6afwzdb9d9ui5rnokaqtpoord X-Rspam-User: X-HE-Tag: 1736319572-523486 X-HE-Meta: U2FsdGVkX1/gKzlQrCPyjEoMQjOJUM/yJpHNihBIEn6kwJ3yJRK2bAS/hw3azWRrdUzv8fRv3IbFrgwgZDmCLfGbLE4F0mE2NYUvbDrZluWPh1Lzk4UKOnH25CryfnDkeOVoQExqcgXsIPBOVnEK88ccn9XxkchndguIMvG5+MrmcTYCEBdOXAj5AwvEmDiG0mN6v7+VL09S7ZGBpHEkuQPUXvWVik5X0pSNHPBOuaGDPxT/3tiobHS5xKuCBu3B5ZzmmCoRdao0iULEY9NertHzkcchNBy8r6rbCBo5K6eONczINUDcvBDgnQhG8FWn+BFy0pB/01lEHWkcuHPcrtk0+BW01iDj/oK/plfrSF8OX9B1NeWnpju/Scxwb4GwnUq7jA9FobH9rj1yP9wxDtWvBo0Z8m90r7LQOLziftzDn60g0JmJ6ymuORivgluqwkHiczWqjqELuYkta6f7Ik3VtexVXIAPsgnqHmFs7zRrHLZM9l9MCHBbjYI7v/pquU90kZZGoOfzcaq+iSV1k9dSejVS7ClYYMCZcQgQm0h+kC00ZYzU7xmDQbE6yXBsex5HBwlEViazUPaUMeXk9q/CgCmH4JiNwxRooY4iXc+2mRycPwBdvNVVqesjY9BR0A/fr2C/bbH+41TJV5oT41S5lgp3h2rvKB6O53aO06IpIl+SasuLqljD03aej9AAzorW367ZnzI8nURGyZpXvNKL+yIIj0wJSE+ctJABCvEfQF0u8Ng4eJRuULBWD1g6RqlWPLrG5SmeR8N0rh3KOcZAPNYw1x6556oU3NGLN/qy7l/YLZBXVCRrmuEctAOZeDKkWZkrEL3dTryt40rXmLnAIp0EGffWiY3S3+15OnTa3sVg/pd62BwYg9DoBNCJaoY9AgPvCU0JGdyOmhFTCg+ZyT6yfLzWGMz/8UpXJMfO1M3AyJgfk878cS/VG8JkeO+nIifhRbOQEyDJiMt Xi9uJl7C AYCokvlOksn64IrbRHuWyyFLWV9h8dTmCNgGXmXF6wcRvbaP8Ap6Z4598Mbw+OrhvqxQm0qq/eqKkZY13/ZwMAiKvOp9KmXJUY8rbO9u8KOb9A8d6ZS/nXhnpFHphVT9AM2eyTfrFF41cB0FOpJ5Lo/0odKowJj5lcNw8+A7jJcQeEqONcskaG8HbvTOTDyxFcfpTFazeVcLDsq2OilrPBTU4SVIL+qCyBugNSTIWKZtKhkNhIjcPwdBGrVSo/MGqnJajuGODMS/DT26frYQomZ5kKngUeEYqVe6oQLHcOXsCXigaUopGha0zAMLLBcdVl68Vmdq67OIgb9TomC5FHC4v5GqlpqPPqY9bnqnXaTaN1jGwBNvSkp6aZEU5p6CiByoKrrj/kBHNbt9o1+O+hSYGHEeTqZxf3qXtkicC6RoG26PK64y4sxeoLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reverts commit 2f3443770437e49abc39af26962d293851cbab6d. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d61b9c7a3a7b0..c49bc7b764535 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc); static inline spinlock_t *ptlock_ptr(struct ptdesc *ptdesc) { - return &(ptdesc->ptl->ptl); + return ptdesc->ptl; } #else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init(void) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 90ab8293d714a..6b27db7f94963 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a); FOLIO_MATCH(compound_head, _head_2a); #undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { - spinlock_t ptl; - struct rcu_head rcu; -}; -#endif - /** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only. @@ -489,7 +482,7 @@ struct ptdesc { union { unsigned long _pt_pad_2; #if ALLOC_SPLIT_PTLOCKS - struct pt_lock *ptl; + spinlock_t *ptl; #else spinlock_t ptl; #endif diff --git a/mm/memory.c b/mm/memory.c index b9b05c3f93f11..9423967b24180 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep; void __init ptlock_cache_init(void) { - page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(struct pt_lock), 0, + page_ptl_cachep = kmem_cache_create("page->ptl", sizeof(spinlock_t), 0, SLAB_PANIC, NULL); } bool ptlock_alloc(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock; + spinlock_t *ptl; - pt_lock = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); - if (!pt_lock) + ptl = kmem_cache_alloc(page_ptl_cachep, GFP_KERNEL); + if (!ptl) return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl; return true; } -static void ptlock_free_rcu(struct rcu_head *head) -{ - struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); - kmem_cache_free(page_ptl_cachep, pt_lock); -} - void ptlock_free(struct ptdesc *ptdesc) { - struct pt_lock *pt_lock = ptdesc->ptl; - - call_rcu(&pt_lock->rcu, ptlock_free_rcu); + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Wed Jan 8 06:57:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFBC8E77188 for ; Wed, 8 Jan 2025 06:59:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49E386B0093; Wed, 8 Jan 2025 01:59:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 474856B0095; Wed, 8 Jan 2025 01:59:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33D006B0096; Wed, 8 Jan 2025 01:59:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 16BFE6B0093 for ; Wed, 8 Jan 2025 01:59:50 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 87B931603F4 for ; Wed, 8 Jan 2025 06:59:49 +0000 (UTC) X-FDA: 82983384498.02.D54932B Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf10.hostedemail.com (Postfix) with ESMTP id B11B4C0006 for ; Wed, 8 Jan 2025 06:59:47 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Td1UYSaq; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WgpljFzcR37DW+k0izImY02NXJU+2N4CamZhPf6BMmU=; b=Em8BnpumJ3D5lxU7VbUKWr/plDiSvoqS02ig4J7u2IVjlENaxDxJvL9vInChkq5lKF+djW QaPO/knbPbM+1sbrXHMqrFmvXS1FKyCXwwm8MCkMWxD/u1g/J8HLl4tCruG81m+SqxbbKA UCqBCzoOYiW4wqh3hv7goNA87drxIiU= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Td1UYSaq; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319587; a=rsa-sha256; cv=none; b=hMD+8+Eb7owHyvpuluceuB8YMEOuQ3xAXdjlg3PgQKOf4jSEORUfeSlBeIzhN62KHg6fDw 4/2M1XX1qgPbItCDBy59U3kZn1yjW6D/8bY1GT1E6DRoo/70hRb5HHvG/mxOVqiRR5sil8 TCZdpfrBD0Nu5LlboqJg19bEGz08fy8= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-216395e151bso7101965ad.0 for ; Tue, 07 Jan 2025 22:59:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319586; x=1736924386; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WgpljFzcR37DW+k0izImY02NXJU+2N4CamZhPf6BMmU=; b=Td1UYSaqbS6rbCNKSTnjEtXnRKlDzYjcD6VoxvGnwN9wcYmbcJNPWrkDe9/3/TsU9J VJmDPjeXJkzu4iglVky8qz5VJnktdh/e3Ti9J8rW1rNN1hR7gvMVDhM/UkPdhSxkEDAV ByXW0RfkSB15XEuTJ9rGNTEVKRBoRN5DsmK1aP3/Lrt6XdHSxDZC478pmF/PA6YNxotH xdX2UJkoJWmBV5iyajMxrT0OEeU2C3CHkfZKAiqse9r4mTyIcQw+TjGfN5IVhm1VrHrA eqPPs4kgkYJBeHq1XQv+5zGdHNk5vib3KnxGJx/0T4FxfferTPaBT65UROlLGVuH+2sV xRpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319586; x=1736924386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WgpljFzcR37DW+k0izImY02NXJU+2N4CamZhPf6BMmU=; b=jT87tfsHiuwa7mO7o5M8+4SIUR8wYB6HoqUtQFmzVNSp2DBD56NPKCo1lIqmmQ7YIJ yPTVxb3DdTo5g+qWAg4MTV0X3xqkQbK5vyk7P/4lSF8CVWL2o5h9DtlaADWsxSJwKj4S aVR33dPNL00SGmiuN7xf/0+8XmMp9jNjBiqlm/ipkb3UcbEGmu6VQYgkUb1LcSnfv+Vf KmhN8q+QerY+et4HumCul99zI3tfm8UX4rpADdOv6WapA21+Sz8UzE3Nl00tKsvTqb2N euKm7NVbFOMJjjuG2/T38EDs2pGdm40/MFTn+Jxh75RUZoJS70bDxF0WfaaZ/YkiI3RP 2Dsg== X-Gm-Message-State: AOJu0Yz/QuMaSl4Ag2tF/VjIY7aG0l7c729BK3fEAcr1+jwnhszpQFWd TwmvboLyxH8CBls+RW437ILZzEZ/IXSYNA+VQKhn+xwk/x7fquCkyh06y0bi4Go= X-Gm-Gg: ASbGncu7O3W7G4uoUEPt4wpOrOxHWIPbkYHl0f3v7tPlBY55fKYyETa9aklGZ5PqJUm QWd8tJgjd6ssMQjNBCz+oP6qSleLo/mIBaIchGyLTkjvSYqbq8LNNFElZh/d1CZhgEqBsyBXog7 LgGtc5xIcFThuuVmDBZy3jLpy8Kt/YEvXPn1+67FgqADwPc4QXyorOb7w25PrY/X45U5dtMeXuG h2HPaAEJNWqy73//RXSbqqYzaaz6zIAWsqFmM4QaGb6VEP/Zf5NSbOjyglxhB38Ft/mumcJBZE4 vqhOKtCUzJE7uxyyxYd9m3vsS70= X-Google-Smtp-Source: AGHT+IEXmx6aGyXUatDDUSC3BIwwvJRBe2xxgGmNdG6bwRG2a4u7PgHO01+qr8GDOf9tBGHx3RRkKg== X-Received: by 2002:a17:902:fc4d:b0:216:5db1:5dc1 with SMTP id d9443c01a7336-21a83bb9991mr32206305ad.1.1736319586298; Tue, 07 Jan 2025 22:59:46 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 22:59:45 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng , Alexandre Ghiti Subject: [PATCH v5 02/17] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Wed, 8 Jan 2025 14:57:18 +0800 Message-Id: <93a1c6bddc0ded9f1a9f15658c1e4af5c93d1194.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: B11B4C0006 X-Stat-Signature: t98jirqnt8k1t8odtah6sg8s7nrgipjr X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736319587-692136 X-HE-Meta: U2FsdGVkX19huti7etzDaRyWXH+A50a310+MqnGRiKRM6u5IE88bcU1JAS/XHz1fPrTqxFqypQ5Sx2UYYp0O5ojFIYEwkeTuv7HN2OdCrnGcgaJnG33WBjZxWIZP6OiTGPPQ0k6fXjFQ8mK7EOkuWc/sZlbMzNSCPjaJWukRgo9s0AauQI5f5Ca2+6M9Duo/eptKqow/sPfoE5JRFZZ3/rrjsMlfVv8CDiSX4KTDMdIzUV4TY14c2EZn3Bg0hzOqgaA8UZudHKNtsR1MoOp6DqTzhPMZk3Vh7FRDCdVPxHThO/sUDOg8eiWRP/SnaQrnM4il0LCW6PtCNtybUL8v13dNAHIMZ8+NvenzZ0zMcbNWhypsp95YacWZDalamMk2+E7VCIUpoY6m9ksnhwIWEmuZyI0Br/Vx1pDFW9ZA/+tn0m4ERaCBsY77tqJlwaME+9pUvU8NyVEQZBh9LIlE1xh3B6jEpDG+snzmeMvQp3OjpPr8m7WCgfIhNMpxVE6D2lPZLjkJm769a7Wy03eYfC5JcOdkX9h+goTu0hLPcp7ECo2ik06drxyGIl/UGsYepgUbQPSJCvGmOigJPNuQ2VMoPakJswpZOZ+9xnXAku1rWT8oQfkFPsUSh+n59vD9i6Tj09lpFlIoAbbDZxwE0h3+3Fi0tQW4wlSF4FAFFwWC8PT65rQ5G/LpZTLbD5aySrXNJGe+FKS+XC/U6H8Pks7Zd3wNQOcUrBVGYiW2JpZvRJCxfzwsaPz75VP4sC77f2MuK4AMCIiOkWHhtyR4KHypCbtLkzjZy5y1Wa9LVmLSzP6TQ5rcKQlV3IVHeigwSP36cYz/YUpzOeIBn9xEz+VdSYQMzjFnufHXqB3tBEVNgnnYyIisRrjtvonoElPNJ/x2KBIpRdIIAEt/IQGQAjmuGscdBbrtkqKrjaZdTjH1tCFwkt5ayDsIDsXaqWxz10mRTPAz71HB1f36jkM +wj8lTkG C9uF3Eat5V5zmoEHd62Pm+dQ9bWmjF8eO6u2exLfW/RJpXd1wTs7CvNH/t0LLwhEsBbWwRVYQMuuxueRFme3p6S7T1H9GRn0EwFaSVe0H72ivpRlSwmk6W3dX7Vq0AHZ93HP/5BboNTUIxdrngV7IzhS5RJaIQWukkYo2IGK9iqYqBKj7YhmdMIsBpKDrKakMtZImoby4qnouM1PVtWJ2hqRcLa6BacvOKWAXlzuqTisAmKybkmmZsnW2B82RuQT0J5SvTS+YHYiK1YPXWcS0rlXuyFu2evcUfnDbwwL0Yxy9fiO8U4/mlLh6B72hZbySiDQSubBT+oOHeBNPCnXu6bJXP3r9/WB1/+I8DJMy1qsYFxSRClOy1CxUpAO6N/FdHcU7iVJUe1sJ9mmCufpX3ccMmeCqlZYLZu6Q9/3NuFQofzqh1D443ICo6JYYrHIzi0Jox99YGiOS4nvh1lpAGEFnJ8I8ouziWFmWUrVAycZW8/zR/Fo0P3abPj0AiiDnUu/4/903FDi09BcWaMLKayNQIhQUh0Z5ASCOle4HhoDr3PClvOLR2fwCMNVWJQl90yU7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky Signed-off-by: Qi Zheng Acked-by: Dave Hansen Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..8ad0bbe838a24 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp = GFP_PGTABLE_USER; - return NULL; + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) From patchwork Wed Jan 8 06:57:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06A6E77188 for ; Wed, 8 Jan 2025 07:00:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F9A46B0096; Wed, 8 Jan 2025 02:00:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9AC6B0098; Wed, 8 Jan 2025 02:00:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6229B6B0099; Wed, 8 Jan 2025 02:00:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4200E6B0096 for ; Wed, 8 Jan 2025 02:00:05 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EED1FA0F6F for ; Wed, 8 Jan 2025 07:00:04 +0000 (UTC) X-FDA: 82983385128.10.C88706E Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf07.hostedemail.com (Postfix) with ESMTP id 15CFF40012 for ; Wed, 8 Jan 2025 07:00:02 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iBUk+ftZ; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=3Mvfo+AyOpau8tTPkrP7bPy0fbff2lT0y6p0xWInTzq69wb2FeaXj758uTnH8XUzLlGiFI ndsrqSrT0SytPEYPq2VS/5M+j3IWNnWT5TfjUQSn0ai425nwX9wcYpZfSLpc/wtAiXBPf+ A9jtYNlR3sGZI1/Ag2svY8vGXUiSRiU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319603; a=rsa-sha256; cv=none; b=1F2450S+fcEAwFLP1ZZ6KagR07pAFDSPrwpAPvSBimAbb3NIvt/V3vFjbMFV9LOraeslsw g5zsKoWLQVa8ShqR+z/4wn/qiDuPyuI9UOBKUevqo7FLdnI2jLWY96gh0KKlUZIy775yoN uXa6CohvCbkg2EiYL0P3Pj4BbbY8iec= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iBUk+ftZ; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2f4409fc8fdso829796a91.1 for ; Tue, 07 Jan 2025 23:00:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319601; x=1736924401; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=iBUk+ftZtfIeAxrbLt/7+8AkLxPKExeXQK9GEOcwXFQVJJHxy75uD13OGfOvZMPfP8 hSUciaJFmIOBqHUqLbW9dNYlzgrnSqikWZInoZdfBuQSWtjTi9L0CLCKI/mY5wWnVZio wxMHw+z+8Ti8O/kB/CIFZuJTWV9iGXT+zJGyVGPzG99riVfrX1ZXoc3MskGM8/5tWhDK 1cKZe2BxqKA490cPFv6R0OdSLXJqunZbQZ9GsCfs+XbajQ09yZ6j531af076r+36J+Il xhNahoP7po8iCJATFFcuq2m34u8GJ2v134vs5QyHhDGkjucpYNVzVjPA8ppw7WfBplC3 fTVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319601; x=1736924401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hi4+N7L/R07Fq6g8WRpMxop4cD3t01a1BHl7awbkQjI=; b=NTm2IN+iDqqoNWjKEDAT+XOjobPue0lJpB6gYbe4aSXBYHgJ0kwxJdn6tEhiPlUDlD N3+21xJ0OccEvYOS3OYjbL1ACrWVesc7XkR6jNbzKorWK2jCS3o6p8fYetFSAoCiVepG sIkKjJqCOkcxYaueQK0EbdaoIAMWi7w4JdHBKobTlDw+/6RgqrM1+ID2KgrkTStueVmo SD6CrYjY7C8Ff9AphqM8VZpVppXAj99Sq/RwUvTFCT12R5n6rotQEcBcb5ZEhJCPILG+ b4VRo/dmJAfe3zLRAJxABB2JZNdUO9miBXFlntVVC2WtFz3SnFUmCXZT65yOj9DnH6f1 hxpA== X-Gm-Message-State: AOJu0YxqsZpCTsOTjF0br6MxS8mGQ4fIhYbCrBM5bhaiPpzHoCCtQBsg Z5tIpML+L8kpfvQcWlmrlIjppbMRsgcUwhcskLQH/B1m5lqBqf+yha6ioqNHFE8= X-Gm-Gg: ASbGnctBAPUGbbZJ+4vUtrFRh8okQAmOAO148T1HyCqhCsUwlbRw5nODZJpP15pSHti IRU1kwV1pNRZpUSWj+JHTVUIxglofAkFOsgWnW7CZLaeIq8xeSiWd+iwm0QMxjrUiItyFlc5l1c ZuSud6rZC/lM5aXv3uyqJcYjpCPijdtKGa0xAmL/9fZ5TCDahAbgWXKimwVvUyeUzIkPM46lPgR 24conwV35IVeENuyH0cK/p+bYoE/SpHWK/YWyLu9dk6mYS47SfXLl/i5teSOktlAJkSwdkJq8dp WnD6nxnuOghaebXbMJMVEhIMHBo= X-Google-Smtp-Source: AGHT+IGjcc9iZhJOiA9Wc8tHUVktzF7/LrG3apZd5+oOKb6c7hgnTezIflh22cvX8oiOIwiosByWAA== X-Received: by 2002:a17:90b:4d09:b0:2ef:2980:4411 with SMTP id 98e67ed59e1d1-2f53cbd9b2fmr9342892a91.9.1736319601534; Tue, 07 Jan 2025 23:00:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.22.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:01 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 03/17] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Wed, 8 Jan 2025 14:57:19 +0800 Message-Id: <26d69c74a29183ecc335b9b407040d8e4cd70c6a.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 15CFF40012 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: bzbxqoeyaw7f5r3wn3iatqwm8xi4d6yb X-HE-Tag: 1736319602-684854 X-HE-Meta: U2FsdGVkX1+yrt3GRJkW2mbTi4xAtG4GJSUDJHZ578zTdDlbcNe2Ygbdy7oeP3zMEwNFUm1c4HSX+0aY8FALD8uDEB0EQIImHEYi2OTJ55oDViZUe/eZ4fZUiXZmHm9O45jeRhfudl7xHyAnowMh9KDCdpxfjkLJrY92bywFLd4U6EfZmpwvoRSVggKSvXT/u1FPlHRj+HalvYDWtRqLhr+v/sdpA0YPNyeolFejTKevV+kv/zsYigfQ9UeT20CNFdLM2xiItSKxz3vP4oalUwXwj3eL5GbQtaucojXtKL8DEwdmaMOtp4i8WqCvcQqko5Rb3OYiMVH91/LF4t87oxnlRzFOQuWXsOJaLk2redEzEY8EgDCxs/ps0i7fOHihlucevFa7zMJsRpf718+nM5q5tJ51cpqCA/1Glb7Yt0XpnGnvtR0qV/H3KGWY6ge77FDJazrUrea/y0wwLGlDeBT6PB5KM5mJQvcTWOy2RBTVRXW0+KQKaZmKKjpr7y6PQQnhkkzjOOwTM0oakzFjvtrOthGBDiQI/OEuLmN7XDfaUroShnTVsrJmoNVncG1Hj7zRdAQ29mLgGlXBIH0FtoWg1AQcUxTbI62r9T2tDCIR9Vt7KM0thE3iuu/TjBOBIsewheW/IO+/f1ZdiFcEulV59NYlznqGOAP0k9/HcSgFSWPM3kZehV4+lMt4NQqgb1G/RP7HVXuzyapkJvGNl9QV/1Ok+/NN3qcq3PDkIqzZxXwBU7DpoiQwZkWIcDFOiucwIlTkN5bPXToB7Tc404eJ4ewnUPJx8GSyS0PCX5KIH6s1UsyWbRB7IIzDaNnJr0J+NjUixV/V5A70UX2ZPHeHuUcODfd8msmfJRXKQ/i+Hm8C5YpDa8e7zD98OgIBW/3Oa3TtsncE7S0Jy7FDTJblnYuumu2/NaJbylSKHArDw1TzdWuAU8rOwGrPngi95nWwaz3Kat2cjrBfGHP WndZvTLn 4qNUUJO/Y2JCZ9PccBNsD4KrMjXqLEqREtx6ASKAmtT9sgr7oAsw9Td3SjxOQsg8hs1KijRKZs25BmhLrJXL1kOhWXgKDtPEIU1Oju0J6ZcdY1yyb4mNnRtYwPvOy/TZQ9F2zgmHhL2L3hivghbc0N8/utj9gFdQ0JdZkke5A86XwAvC9CdTanbvDWh7glqZtndVB+u7/PlSyrDWLEXpSzMwb/aR6Igb+fdSxuFGgvlQnrLNdcJriH30fDsQN21X+L1BdH555Xq3XeP1yBXP2eC/bphnqPHbvbfmtUff+6eHMEQ/2tIT2tkcJW2F/Ckb2C9sPKWSLYHRUawylcSBf8YGx0bThJCEFB5b+f+ypgB/OBWrkIc49EjS6DphqSRFGFl+MBcGx3OBceOjijQudwWB/RwuVyagDUwJb7nAnxI2Whps5q/YIUBxzP+BegXMN3H7bCnoSuSXOSbKWx0mrNv0scaeRNhmowB3YDQFdoTxzdI9fMpAUZs6csoemTSPuqOTPFfPo8gqgQUYFy8tPDQCEGIjj793K6I2KV6u9fFWJtjiVaZQWnzHkDGdDlqoEqWFqN4x9LxnbJI0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kevin Brodsky Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky Acked-by: Dave Hansen Signed-off-by: Qi Zheng Acked-by: Arnd Bergmann # asm-generic --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..2965f5a7e39e3 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 8ad0bbe838a24..551d614d3369c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..dd4841231bb9f 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..59131629ac9cc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { From patchwork Wed Jan 8 06:57:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D98D5E77199 for ; Wed, 8 Jan 2025 07:00:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69A436B009B; Wed, 8 Jan 2025 02:00:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6495A6B009C; Wed, 8 Jan 2025 02:00:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5119F6B009D; Wed, 8 Jan 2025 02:00:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3462F6B009B for ; Wed, 8 Jan 2025 02:00:23 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D13731209DC for ; Wed, 8 Jan 2025 07:00:22 +0000 (UTC) X-FDA: 82983385884.29.4CB6830 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf12.hostedemail.com (Postfix) with ESMTP id EFBE84000D for ; Wed, 8 Jan 2025 07:00:20 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lWqD+JsM; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319621; a=rsa-sha256; cv=none; b=6zDtM2Oi7orLUhxBquD2RPOhJDHmsBPkXmhu6giwi/Ao5X+Y8PjHPCPRPYOlaB3vlWLDmq 3eIGaL360MT0pKnxbxKlxpHCXcOSXpKdALRk3zkZSXOs8w9FgyFsyqL/6PT+CLXysykLxn IuzQsB+ti8y7/4/V83d5He0H6Q6hSCk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lWqD+JsM; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319621; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/cYfjcYM2i48jLO7NuvQJ4j04U6+7fBnrQjgIuvbLyg=; b=IJOMDTv9JYrKLNqhS931em7SrrfMkU1KHaNABbbkLx/c4RqjIDs2FGMTG396GGUMxoIp7L 8QJLjHjeEC/tAqCRRTAwVvMPBO1Z5O/PhnDydvdWlfUPXtfvc8crJI0cxAgD6mnHwxNgVG Bx0KaUbpXJ3hLwWhFisFNR8W511ennQ= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-219f8263ae0so193715245ad.0 for ; Tue, 07 Jan 2025 23:00:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319619; x=1736924419; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/cYfjcYM2i48jLO7NuvQJ4j04U6+7fBnrQjgIuvbLyg=; b=lWqD+JsMk1haG/2Qb7RnAGv/jNY2crK3Eud9isz5yR7U3GABuWJRkUZsGiXvOQkTbQ bw3bdDzzE92J15IVWg/yGxomPwKGNUuRWvF4Jc3EuqfIjoKhdQbiD/PSGQc6P9Qgn1/6 yp1X5tMuYL0xv2wfA560fuh72dFUyG+bzR1d78IH7cBnLviQTve5qhkG6JK4bFa23OGA KM9dcdF0YHaUIdrJAY6oHyeKMwm7XjbGNkvufgSEtlHc1SStitN8QjBFk7dggJKLERB0 cZ+2SgSOGiesfGW7LvjYuTY7G4Od2pl3gmgezF99g5AOXSBtRpWWz/EE9yvIpAENuHXK 0D8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319619; x=1736924419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/cYfjcYM2i48jLO7NuvQJ4j04U6+7fBnrQjgIuvbLyg=; b=r7yv1Ixl6IdqELSLAqF0MZ8KcYuTyvLoKSgWkx2V8JrYkfPypok5p2/I9Jvb7gKYh6 PmHOljlF55fbs/BYUsA6TOIzAFm/uzL0lmdWFirVf7Iwecoyxh8Ch7XUW2EIoGou1KfA gTZcpdgwhx3Byr8/3XGdEDAHIss4PblSV0a/tSbFd53MfNJXB2tLnGGQi7or2rFMKnfR 8oXivzulMfyPzBqJB8LOOamAnyZE8a3/AR7/suAI0k26iWmKWs3Z4mPPozBuQJaaPFFk ghnyaS6GPzFjrNBWEAqrcmdd0ah4xnmWeUDrA+ili6WeC0ltxVSYVx8m6iQy1OmV/gYk uz/A== X-Gm-Message-State: AOJu0YyyQlHY0A9naiC1P6Jp6J3NMPEDLZuD249QrNFKjOXskfIPD6TA kupCSPJXwxUMRGyZORJa5B7XP1Gh1oEMFMJc66YH+Vm1vVpEoCF6mkRDv5lzXgY= X-Gm-Gg: ASbGnct82kOI6l5pgcID7DwWiIOGieUuRMEu0RgCJ0assLrAtqaFgXgjZLO7+TqZy8z jiiOy7o74QFCKNGsCm9So4QAr8wRNBD8yM45EWHOySlflQoiwjFXl5pXInnTUnzokxDmmaUuuap pj/u9w0nlA0Jrb33tgLMM++PhtXfjGaqslwxamM6iFm9kzy/nYkpI7bkngNHT5QC9broGowTa8B dUMgnbO4SnROQsXTcopc5IRWvjblhR8tTrDwzaNGmdriqSDoHRCaKNyqCcoXtCy3qZSRn6pkYwb 2LwCldFkJj05BYmV8RgeQBJZ3Qg= X-Google-Smtp-Source: AGHT+IExFPt1bVFS1B+HfK/rni8xCakw0mpYmNpiHCGWtvviMVtfvk5MaDovSIdy5xSOdSeWSopXEQ== X-Received: by 2002:a17:903:2311:b0:210:fce4:11ec with SMTP id d9443c01a7336-21a83f42687mr30246255ad.1.1736319619595; Tue, 07 Jan 2025 23:00:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 04/17] mm: pgtable: add statistics for P4D level page table Date: Wed, 8 Jan 2025 14:57:20 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EFBE84000D X-Stat-Signature: n7zj6gr7i1p57q1t7zbt67ncnz8g7u4n X-Rspam-User: X-HE-Tag: 1736319620-498370 X-HE-Meta: U2FsdGVkX1+KrlXWNtOQ9APli3sIEL/019ihrTIMje0iBu7irS2VI2Zr4BTpnvTPrSdKW0xjud3kv5LUq5kT5IMCv4gOvpj9JmUMaUmu7hXrbOytSnHKPLFNE2MkT+p9gbqDegFMV7iB/tDhM5A2Feh3xcLDF7FgeWAvS8vq6o/Pel74XbIFJC6plbqyek/jltndYl34PauSpwaSubDxI9iYhM2iteIln/GNCuvVtI/lHgqvPGn6b8b2sM8YU2VNVWyBPXLq4EGlMgbTxWDuEZ5OCruDaOwUd/aWszmlg9mqNYGxmHJUIps/XzNTL9uExdwaKKU+bzb7iJs0HoGRqS1pAD0raOglWEfH0w06z6q3DglRwe2yuwpvDg2GUZa1EZNZXo+TFG85bJktLbrIdkm5zzYxN9ZHKZQkGSoYHqI3JrRaV+zL3dYZprSsq1t9+HW1a4dj33iXbx0Jsxwa78kBlGsPoAJIlE08Y+nCN6BBOTg1n5YBCwLAxanvv9pNsZlzSRojX8lWEB795c7SmXUmF+6BRVTf2kH40tUsg5ZmkqFgxfYy9vSAqf9tPehcu2BfskKSfFtjhX6RfVihHYS4B1LFH6qD0+hnQs2lwej+u+6EiIf4+KZPscQfzfJQmeRZtbh1iVeQPHYQO1W0BgP9yu4+zSZGWZKFuck0mXXjNflO7gQdxNSxRxyjkvdSmNd8dcelZUEOc95jqReWAScsDllcNllscz9U7F8/h2Z5CqSYNXlziJZaNa4JhW6EiOXHM6lR1XBGcu52DOTdxwqLRo7fLPQVfByeJ+bryVtvzCF/MLEy8uYO72aJl0B+4GlEoULzilXeR7EExkbPI0+/ah2Ym8dZvou727pxch7BgXyFBkcafq0sOP5pZUFZVzlGJ3PgJtLUllgp+h1BTGlX7FqMp6PFvR3PgBQ1ihBbdOD+NhPUG2S+2SdtjWhCmjyeJpBoujf20DBoF7+ vdOqMN9U IH/8t+omVpzmHe8hUAAMQHsOtSBB4ltmi97i9cNH28VyY3kMRpn4b5vUQAi0pBFLQn6MbzsKhJCkYttxLpL3GtgAjjeufRCo1z4vQfstabhiVz8IItX8WS00LUS9vKwoQcL71AL7c732d2AOVd1cjXwdOK2dpKDtZrkM1KB1fxKjAUBB1LXt5hWwz/fHpgAdIJVPh0n3bhdOWA9MOhraEkrW37SMgEQc1qAYn2RBwjmRrYOApJTEOovEjQU8ACeGWwcMYnX0FOVL7na3JmPlZe6SjDQzw2QW4YAHI6AsdgAF0y1TNlQk7764Wa+gqUGdLna/kA6rcWXSEbIGSyGVtVWBfVVj+UFqrFyjr/wizgTSs9wDgDQoH6GOPE/TrEav9JGe6qALetWB2kvjtQe+e2qGAOj+P/8lKiStq09VLseTPD4IuWhjDtxKiOL9gRCtlEKi7PE5mAfhx7IrrWlx0iHHj9e+6nQ3XVRzXqrS56Mq+fd/ZCQm2GHRg1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, add statistics for P4D level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369c..3466fbe2e508d 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9cc..bb482eeca0c3e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index c49bc7b764535..5d82f42ddd5cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Wed Jan 8 06:57:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77E7EE7719A for ; Wed, 8 Jan 2025 07:00:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFEB36B009D; Wed, 8 Jan 2025 02:00:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EAE386B009E; Wed, 8 Jan 2025 02:00:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D281A6B009F; Wed, 8 Jan 2025 02:00:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 700B06B009D for ; Wed, 8 Jan 2025 02:00:38 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2A7BB422F9 for ; Wed, 8 Jan 2025 07:00:38 +0000 (UTC) X-FDA: 82983386556.30.8B13DAE Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 3C43240013 for ; Wed, 8 Jan 2025 07:00:36 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fRpDMN4u; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s3ajwCEJzuBWHLhfIS3uOIg9mbtSga1CIaOnP3nLDr0=; b=s9Uomz9tlxN38zwtb7vDGHTShzg/AnogDeUjA6QAkmTJL+sj7bPMpWmksdC68UqmQH80p9 st1djVKDWPh50+4Yk9skU47j3EwN5C/W3x+MRgU5mYN2OCDoc/hQa/SzxMi9yT8+6qgTm1 Acmc2yJ6O/JDo7yVKkx6JTSkzSNPOn8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319636; a=rsa-sha256; cv=none; b=CkZ4pEiPTpB9NPl3n4P+AkjR3zPCar0f8DaGaSbMJ2QJkv7+b4vtqzU0UqBehQ1uMLctzr NjnhEBNaO2oaJfiNjwQO/dR3KSf1Qv4ooB5hg9NVaElWyhjqzHGMF2n0fVwZWl+t4zJ3zw pnc+qB/EUCe83b0x3zsHEiFV4BV7DJo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fRpDMN4u; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2166022c5caso215981835ad.2 for ; Tue, 07 Jan 2025 23:00:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319635; x=1736924435; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s3ajwCEJzuBWHLhfIS3uOIg9mbtSga1CIaOnP3nLDr0=; b=fRpDMN4ulKGN415gjvYDMvAW2J2hZ6ZyaF6kMFk9o4Mwq07/JXv1qOku7lwQ/MhkbP NAih8zs4EnNX5GHRNm81FOYqL84mrNYfGpOHyxXzooQU91p2bTRnrnDmJuaw1jsWdb5S uFP2jr+1Fq1NPXhrJovIJnfqDKupoMDq612w3JjvmsBYob62HZl3TKqoq5NaEzLFCkvP NptzEPPmHzlcF1xUKmN42hr8YkK1Tt7MNe7Ao0nFkwdh4XT9aCok/+Xs7k+7dA7QrSvn Q0bbgPmK6bn4s1r8ja4E8CbVEl7xHs+gWLxxaf3GMrlCren0pXlyslEZmO2kJufq5Qtc NKaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319635; x=1736924435; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s3ajwCEJzuBWHLhfIS3uOIg9mbtSga1CIaOnP3nLDr0=; b=jDyCIt4JzlTYiSlN87rF9033BXK/d72mdZ7ehnmPZd/cPFlnd7j7GO6xQH12WyLrZz WmDTsGRy7mMBS4QYGnU6cO7wzqgpsM/dfaZWasR37zb668QtAQk7MyriGd/ZAy6NqmuO mpzzvbB5297G+VaXFrgmVXRyP01JBgtfJWFQMyRWO1PuvOmCD+jSv57yMW6n+k5BX8GZ c3tWWuU29ku3+xuUQP08k1esng3yY130e7GhqpvoSnJ/vlvMyNYbsFNE8LafQHldu82e HYJsCIQzUEh1W4cMmFYFQ7qklTTPZaSDZMchXftXB3+WaKtaut2SgHDEfM3gOM/4Jdax o8Zg== X-Gm-Message-State: AOJu0YxkkG7+HQQuOlyY8E7cFrPLsMDbjrK6rLlHZVZF1zgNu17ZiMm3 ifUOrusfRvzu4CnyPcfuXbSJ66QHNTNYZBBm1vtqUm/woJQpX5pXP3fbxvey0mE= X-Gm-Gg: ASbGnct6IyYhawTUC27zkYxVlWfsxvufOIycwXebF3dcdvIk9ntf7cGGPyywKgYSLyn thu2R6ZKjXdPX9DA21TIKjAkoG6IyAjArTSnSnzQHGqo6HsWo7pKSqm87p+mxiFx+GmBLV048Z/ 5RH4tAFQ/fhY+up7VxM2q/SY0HOpU5faq1hbvGWhB91hganopDIooGZBGL6woEc/X8o6DI6rhNb eEg3Kpv/ssxjlhOnYr15bg9OwubznBpp04heXnU+PgRc+4ckg33X6JKBIlh6OvWrfi1qYSjXeyn JXI9VPY+KRnB1CqwpUBGWFYg5/Y= X-Google-Smtp-Source: AGHT+IFQatZJEh1vMaYN2M2lTtxd4TB3DgwS5fDlIHVUeP0cQjVCzCSP2sGAFE7qJT7hKS5hKn1bKA== X-Received: by 2002:a17:902:d4d2:b0:215:9642:4d6d with SMTP id d9443c01a7336-21a83f55157mr23731385ad.17.1736319635015; Tue, 07 Jan 2025 23:00:35 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 05/17] arm64: pgtable: use mmu gather to free p4d level page table Date: Wed, 8 Jan 2025 14:57:21 +0800 Message-Id: <3fd48525397b34a64f7c0eb76746da30814dc941.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3C43240013 X-Stat-Signature: q8y6e114bghqw93rcb4mtpbfsoi3no5o X-Rspam-User: X-HE-Tag: 1736319636-267282 X-HE-Meta: U2FsdGVkX1+Rn9/nxSF3HmxETDacwN5XzxpLDxN72r0ccQ81YKsAazcNS30cnZHfpzlqCJBQ2BTTXVm/q+PF1mjG9BW7pw90CwbrkLMDpceSgyEGUAzm/r11cteoAAEpzwL6dH7wMr9HMMZfX58qWzxJquBrP5lG/P7tzd4DoF67gOwlYA0igOmsXwsYLo/CWWQmBMNfBABiiYbKCKijpwc6dGMGsIy1ZvqXZheVLNfTdshdHkZiJkfCCywdN1mk/goQHM0k3p8w8YLXs/SAXGOFgFw9N6/I90sjOONWoAlNLirI81Y8u3QPtS4l0sWE/v8xfbB9ELzgv0DddYHDrfqzdyfQCXbDSsNkxIOJDdcYexRGROv8bPSGgNHYQuFaDZ097iMvhoofVw7QPDegut8tfhTB0CTaJdJLVJsprJwdXZ6BNcAWKK0GCzSoBMvYKEpFXEZl5P8wzIxHEgIsaWE1bAdqN2dmE9+pMdbBXKvQS3krGj1i8akbyZcZhNMY79gcvTtLW+NwbfC3ePBeAXR5e//JUVysa2XZ0JW4TJ2b5PSRVKZNFlLPYgYSjE/pKMgmRKlgfmXhQ6NC9nf/sat4f99ey88tlyu5V28BiFd5NQ4U1sbe6LvNXvpfLVii/YvzTKmBJaEOYSjsAfT6r7350u9s61NF6vdsrQmKvOrMaPmKiBYRDkPUu8L1ev1nqnADPhQkU0mXTBqvBoda5orsk91GUNU7FQ9B/K9rT2Ha/WHm88NtDin6vNtouUU7MhUdyV2beGWKMWj2msYsQOxOfZ6yEJuEkuyQNkyA9IR0yf0YQIRJ7/99p5E1eFw349D5QrvoKqTBLFvxUPd8BGHBeVWxIKBof0GZbLhtP/dgSCQ4NuMm9yi3JxoJpgvpcMWB5ztTcXutKLtkLi0jCITTKOXV1W9sH6LvslTqTe0U5vhVUzPqIu/5glzJC9i5MD5Ry8Ylvsx0i1k6vjF Yo5DtpBH AotJDrMwmz7/cJX+lbbh1o8M4hdoxFBPR488/AokGYxcSjI4cIKDllZPLvT/VlVHdWcswUwd+dMi3+50aQV1Ya8D8abpp+gMixcEVwgPvv6NU7wYiN54hyYIcZo6vH1WH5DpGtr3gPhz9OuXlu2Fe1FaFCE8euJrkQpDB79HrhkPoWxppxnqWi5PqbAIJ9xFb8LulHdwJra144jKmLMM9LxtfqkzIFvzmbdTqBmen1NCRu7hFK1okpNsJE/khzaIw+s+ygSllQLjrDDioQDXNA7v3/p/X4zi13v20pNli3mS056OQVjGxCByDwTlrjzHvV/u8cacYTfHRyuLlVR+8palqtmEFTBallprhZgv3YMrcbda0WKLxiWfh244geemt/pdbTRM48kyPjASovY2kYEUTCQFILby6uUmPWReYvKe92n3iA0e65R306PgfLIvwDFiJlq1onj3KT90SJX36BTFNR56ZmQRvZQ+umCJMafaEbZb/H+OC1k75eA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like other levels of page tables, also use mmu gather mechanism to free p4d level page table. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2965f5a7e39e3..1b4509d3382c6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) { diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index a947c6e784ed2..445282cde9afb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, } #endif +#if CONFIG_PGTABLE_LEVELS > 4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4dp); + + if (!pgtable_l5_enabled()) + return; + + pagetable_p4d_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); +} +#endif + #endif From patchwork Wed Jan 8 06:57:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA122E77188 for ; Wed, 8 Jan 2025 07:00:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C08E6B0083; Wed, 8 Jan 2025 02:00:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 670FF6B009F; Wed, 8 Jan 2025 02:00:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 511C56B00A0; Wed, 8 Jan 2025 02:00:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2C9D96B0083 for ; Wed, 8 Jan 2025 02:00:53 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D7686C0F93 for ; Wed, 8 Jan 2025 07:00:52 +0000 (UTC) X-FDA: 82983387144.13.958DC85 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf03.hostedemail.com (Postfix) with ESMTP id 01ECC20009 for ; Wed, 8 Jan 2025 07:00:50 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hH1WCXIo; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319651; a=rsa-sha256; cv=none; b=OQRN/gjo6+ppvhaD8NjTD7zMujMcgXy0jCXnfawfsnD14eaXpLh9I01caxh7bZzYMBEiCt +Zj0QiHpsOEfLJUUol6D10w2IR9u8EC8bPq3kNDzYXVoafTfcGHwtWtoGZw9HlyjZMGrvi AbFV1uJnZqTZ6/KGvzZj5FZy1vbp2k8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hH1WCXIo; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319651; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=evJHXxUH8wpRsLAfhvt7S2AIimf3Ggw2vmlVHfN1TNM=; b=0XY4xFQ9V6kNNB2IQGMKS0NuMbrG7r2FtmHxk9bTtO/nom53hxltU8b0Z5vNxBheBRVvtu 1u7wbfo5Ytuzz4/2VNmtbiw7FuIPBEnn1vN5lvK7eEzY/3gAiMvaOaoG2h8/amOFFgBVn1 N77V3rDqfdei1nZxeAAdd0esiINtp+E= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-215770613dbso181722965ad.2 for ; Tue, 07 Jan 2025 23:00:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319650; x=1736924450; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=evJHXxUH8wpRsLAfhvt7S2AIimf3Ggw2vmlVHfN1TNM=; b=hH1WCXIoSiUnOcwEKnHtLbX/z9IT9M33cTUtS1NmGa2n7b1qYXi1TM5zS8xHYAnRsd pOq9MYnLX0HJJhvqkfmbGy0cEiGiUFFsKFFnDW3/UzvHxXisweLhHBHRAxD0TBku2AOd tHrCaREhoBTCy7VbUIi0+FwrvlxFaVyaumImiMbas91/0nnedAu+/nlAnEdcrLW24XIL 60G3UUIqbnCIh2iOlJUuy3KH+vUEga0kc30mqToGRc8yyqxI7y1CFc3vUmr4+zzWv0xf 8nskoPpBQmm3x151QCtBBQFqsL0qcUCGGvxgxRO0faD2ZsgH+EZhSVO9rcH7kPZfzycC xr+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319650; x=1736924450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=evJHXxUH8wpRsLAfhvt7S2AIimf3Ggw2vmlVHfN1TNM=; b=OzK55u7lBvtaqfheDOLdG0ikUZzGt5n0ecY6bwA8XSjWSfHGQ7SFTICBclpL3qnoWB TbXlFMXrAvxEZA2bsXsFgpLJql6RcuWWOg01sQqM+S/wp0ipKKydJbYD7ubzwyBCvX0x IO5vq6cV58fbMwO3HzXtt+iP8p4Otltw6tgmkCNKLDkkdMP6aZm4/f7g7myUUIvfC+ZW o+l5Tz6z0jbXSL8o4gttEwTta5HamSo9N5H7VYm+UMHn+cJLxesIeIg4KyrsiV08QvCw G4c540HldO7NopB2cQ+vtmWdVYzBXf3vOPTQe5aLlZIjy78HB709XXpL+noY8smVWu1u Ohzw== X-Gm-Message-State: AOJu0YwFTLAE/nqx6vc+oqxMiFDtQlPZ1UhDjPk2am0mKjDgO8unyjY0 Ktl+CXBbiPaqsNAz3ptCoLhmidZYGNnZdfNcWNJYbceONY4lebPtUtAVNFGMh7g= X-Gm-Gg: ASbGncvnB3ieaQjgLlXK8/tuWzAzHgwPEtK75+j1F69ga5mtPXfVQV+aUUz4+yA0lUL H4vKnAXM8KBG44WETVw/0ssrKFWoXBZXKRiZiDQZUKHKPfJOhAqpUiQS0b0PesP66dviKRM5EU+ sGGrmrAnVV97ssJsRHWcEI3T5Ss1P8VHVfMO2XL5Lm7Q+jKqJsKlbAEiCZsmfzEGBSeXcRO2Ntl FyZVJRp20yYavZidfCWDtMqyTs+eOIDC/+eiAZChv5nHPufxNH9RmX9RO8yXZ+C+d6gmwWM/I/v laBlF/9R9oHIw8Vb2VR5XQVFHkk= X-Google-Smtp-Source: AGHT+IF9X6lXCsguwyHO5KdCHubvdTIDd4Y888fAuDMWr6lGszjnzuDzetJYvdD8sXl2Z0Y5NvXbNg== X-Received: by 2002:a17:902:cec3:b0:216:7ee9:220b with SMTP id d9443c01a7336-21a83f4f43amr30110425ad.22.1736319649715; Tue, 07 Jan 2025 23:00:49 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:00:49 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 06/17] s390: pgtable: add statistics for PUD and P4D level page table Date: Wed, 8 Jan 2025 14:57:22 +0800 Message-Id: <4707dffce228ccec5c6662810566dd12b5741c4b.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: tjxahspz5djycach9g5ddf33upt1aqd9 X-Rspam-User: X-Rspamd-Queue-Id: 01ECC20009 X-Rspamd-Server: rspam08 X-HE-Tag: 1736319650-106221 X-HE-Meta: U2FsdGVkX19IRRtfwReC+KKkchqaL8HQrPaO0vf2xPUY7JDtVa7LH+gtBLQZcTA7xlqDxa5sSUY8453Euu4NdGVJaPA8pHLClEFW5D3my6qw8ZIk7hu1QOlMvF+p1uEF1LWUriE+cJ+q57Kc977EjZAbgTnMrmmXhYpfvatJVIrzhXQmKy5Op4Gji2pU3CrYvLJrEvz0R2zVpiSBEpASVJyVChEIZzV/h5CFdw3+k1eBqT/03vqM6jIs090z1yNDqZP1bq/sxRB2CUEYrgkOvd4e0aBsu2RTnqF8ePFTvubrjpf+vXWxblm2naauusIKJoSRt+Vax4SHthPrw1zI94Fhy+24CHJrejkR702ibG8h87Nchckz02OWW09+r7mijyEGXB+mYlVzOjvnzKsnA1CE76/tSUbxI+QPSHyC+HNgW6u3WCBP2rY9Zt8pyPVvWw1ATjOapUlPQ6iwlMuQX5H+eSPph2YQEIDEIE1GXsgEX1Tsm0UXBF7ceFNL2laYZYqrooUCAqBy4Nxg8MGdtfIM1GUg9go28YnwaaB8yNQcBpxQF00E8wCuae/jR+k9PR4zPV7hm1ZPTRo5gs5HmTsL399RqYVTMrPotKzyUTHbV/6YZRXE87vmz8TyE6wEWiAQuB+WFc+Stzb9HsWfEMliIBqNPsuXzRlYT1Ec0qyjFePFh8nxctG+NW/iwtx1HIE+MIngSlhTEr5OMcNazTTa5X2I5HymIinLhD676/KRNdkVcpA5/E2kjfdVqsSLgP0O8eOWT11C1NIy62O0ug3STlNUCpVs4+5QmHErBDzSmfAsfUn7q//ZQVXCT/7WwyjF3UOw/T/ihbWnsJFmUs4oDm5Ajp74RJR4HhmTRcL0+oFpGLNkE3lbqdvtjvrKWkXXhFA9V0UStuxydPh4YllAvj5sHwWtotJICVqVsw+r/HC+1SyQzy+/ptk6RtbDGzwq50Cffs3ESrRg9Y/ a2i5luK7 XdvP4oWbEpBbRobKxoQi+M+EQBgCl0HYrP4d8hMFxhOmrYr8TLW95qcgVi5L3ZqJtgLBh3LN2PVlN7UJn+97nx9UhFhlHq1NmwxgWfuLB9AC5Yi7s5LXS9px4oeh3HG6vYkiooSN0H7lKznmuIqcM2QhKlvt0WwtzoSOBQFfE7BR9j+GU8u1ubhK0QUn5DmdENaaHz914RbQSNINeXVZ2Xao9DDBsznbP2PkaWBIjFKP/4PFONExt1l++Gd85V65Z2bT6Euk1zemfn9xAgkSA9GTcuhz2ypN5jUi660KnEObAamNQmXcdX4tdxjXXNoVAozYwVcIxVsyGWr2yF2vS8Hj3Pnp2AKw0A+9FHr4EuE9V3uOZi0Tik4zC+XJTL32lDfQt2ehP/pgBgSvDapvpHQxvNtzABjCUnmMZBsOyQ16So0ak3D3Dmq4JXzdXlP2iIE8z5qC2KXZ34ctSPEtVkxREy9oqbsTb882Jgscs1b51id5XxGY6F2aYJKPP/N3sWOlJT9xY3QD9mVg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-s390@vger.kernel.org Acked-by: Alexander Gordeev --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++++-------- arch/s390/include/asm/tlb.h | 2 ++ 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..907d57a68145c 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -122,6 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -140,6 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; From patchwork Wed Jan 8 06:57:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30567E7719A for ; Wed, 8 Jan 2025 07:01:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B56866B00A1; Wed, 8 Jan 2025 02:01:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B04746B00A2; Wed, 8 Jan 2025 02:01:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 957C86B00A3; Wed, 8 Jan 2025 02:01:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7575C6B00A1 for ; Wed, 8 Jan 2025 02:01:08 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 21B5080FC7 for ; Wed, 8 Jan 2025 07:01:08 +0000 (UTC) X-FDA: 82983387816.19.2F40B77 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf10.hostedemail.com (Postfix) with ESMTP id 29B44C001B for ; Wed, 8 Jan 2025 07:01:05 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=NNws45by; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3i/HRrd9DcQR4JIafWbJ+2XekX+xgLIenO/u5trArdk=; b=51xrlHL3AKVkGO4QpZCjIVjnnlMfOJm8ocpg7ruzGINBluYprZ4k2+yypWT9x6HD0dWWqK WQVr3TnZG8r+QvgwnnVHtaWNEW0Qt4Exi8ArZDq0Mm8IMjq1t/G1K2/CTMoXQUEEqMBMuo PPbvXJXv6Eq4KqjE075uj7jlZdqRAlA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=NNws45by; spf=pass (imf10.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319666; a=rsa-sha256; cv=none; b=RFSiMK5GO0cUVtl7v6ixseCJLNAL/aTI67J1gQP2XSnsfLpEimz9idJPKaZhOcY+y1D+Zp HowBknoSCFWpsODRVYaCwynI396z7AN+rDS8edmobC48Z46Skf3KF090RL18KjHf6iyvTP scGpcyMpmCzWkayoz4AiD9vcFBMXpXQ= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-218c8aca5f1so53025335ad.0 for ; Tue, 07 Jan 2025 23:01:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319665; x=1736924465; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3i/HRrd9DcQR4JIafWbJ+2XekX+xgLIenO/u5trArdk=; b=NNws45byAFl/WEuMqv4mugYQVe0tTm83p/yKMPT/xmPX7sCQKSVMaiHoEMhVeY2VWP r1I7wzKYqYIb8sE1udvmxVye7kQxG5pv8W9JCjHQp+j28SzHfUvgmDk1GQokKwRM8xzc SooVOz1ulFkuvX+JeBgDrdf0CcprwKQ8CyVKEzV68WqWlxArscGhFoempbuAXovQhb6U WQDgxZhJpsZ2cG4DxdY22tgX+epAcYPW8f+MSaDWB4Z5gaqLbF1hrZfDY4A1zz+HPmL1 Z5qcZ5XHvfhMMCcYG7PO6HP3AduQbXwnN/9wGrTsCUYhiCQ8fOLBBreQW7eofe8LjCyC Mkiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319665; x=1736924465; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3i/HRrd9DcQR4JIafWbJ+2XekX+xgLIenO/u5trArdk=; b=hNxMfaEVORwHe4AmHFzafyQx6MhCDbmzhh1FJtEedVJr0SNxaZc+/imqaDw/jG1MsJ nOlIUiYiXLaZ9avzs9hQDUpELKTANmtUrhYBd+4hj9v8g6qYUJaSJm0ZoqrSddY5Y0qQ REoTV/quCrwuuwvX096aaWUWtX8Qpt/nEwzZtUmkjA14HIA0bCn+gc3/dcei+I1su8ps k1NDONb2jQBz3Y72iIvz8/CbCLuSThAYcY70KU/eY3/XrgvrwOroQ4phCMiLPeUCuXRM BXUYsqlw675wlV/LHjzxtjyW8SU3AtSBG+q/kShPJHKozg5Ws/Q40tNUV012+k07LF38 d/gw== X-Gm-Message-State: AOJu0YwVMntfVwe/+EzNXE3CzqnsCqVmOOW60DL4StsM4vDzCKGOaOcF LvFaUEo+2CV7ex7i0mJ8WdZYovl9wtazd7v0M1+9i0FbmjgFlrZDZ7cHMxVb8DQ= X-Gm-Gg: ASbGnctOVEDc4wRsQg0u0FKitq8r3cEYRg23Gqoh0BizicNv9jn2HLiNw7GjIN0luxx DKQzBGba1b10qUHDdhVpSDc//dj//i/codT+O+FF/TvVRl8qWwwxwn7zw9zT8qQhYgtrHnmGADp hRIio8f5D30m5886Ksj07XPgXUPkRAiru7a5X45ejcX++yh3unC/nsgoBER2h5XcMPlYeyRIO50 3G2nSSjQu2INScEvi6aHGHmZkebm/PhSjLqewtNQEE/qwGMTCZxuGnkPMPXjKq0cvHxf0EQNOA4 u9/pyc97cR5JokWIvaM4F9VO9RM= X-Google-Smtp-Source: AGHT+IEn/Q+3hOSV0Xf6HR7089ZKBzaD+1qnLFodeVKvlP0f0VEvA1wx4/vhMKgRrGPfbs0rdLpQmg== X-Received: by 2002:a17:903:a4f:b0:219:d28a:ca23 with SMTP id d9443c01a7336-21a83fe4c55mr23137765ad.36.1736319664669; Tue, 07 Jan 2025 23:01:04 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.00.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:04 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 07/17] mm: pgtable: introduce pagetable_dtor() Date: Wed, 8 Jan 2025 14:57:23 +0800 Message-Id: <47f44fff9dc68d9d9e9a0d6c036df275f820598a.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 31z6xuoz4d76jehrdxbzst8a49zeiasd X-Rspamd-Queue-Id: 29B44C001B X-Rspam-User: X-HE-Tag: 1736319665-302976 X-HE-Meta: U2FsdGVkX1+PA7lwRUenL7X9woXHgZlYctOzyQ3k1RJsYpDlVKZ/qwEGccFBp8Q2e3PUx3MFcZfAi+m8JsWjp+rJvVbIG8jHoypo25WHNGSRJjfHG+ex5SmhHUJuv6kxY7RbdCQsgyiDj2pAC7AfBRHPZ39o/6RDXSMEnZIN7EwFVd11fLTjSaywMc/RZmUfQsEZJhNvzWKZHLEpqrm/DFt9EvlwGHYIKENyyg9RN5BJI2kYYPD3VE2p0yp14Ts8UPheyOrb4ia0J3Pu6GnyhQDwzUCrHIvJ46EYIn78dNi7TYdBWll21WjHlPtM6VMYtm54saYYGwMPTG0lk2YV1Cz2cIbt3A285z/2sBzFb5oTbTBXfPnD/MHYI8o5q8Lur7NaNAkWX8zrsANeZT2j8xB0cR4l9O83Fw+erQBPJXsn1L88bJhGPFlVdcByYAuqS+uMtoSrRRuySmRSRkGYIt5A6TmpgJnInpqeUpZ0tLVNkB25wEjwCHuZzyiTO0GWOiLqm3BHfKhQOifFP4VJ7z7yd05+sll5wEN9SvbxIGv/dX3F1R83gm8dtbB2+PNZpRH9qB4FJV44vzwgKLkruBkEA8hKaQoA3R10Nlr3Y1GhXHNdkp7p/ESNCNEVZNKvpZgxsFbQfLxBVjc2wLvkn1JktOEhqtqJgN/FJwPMoUqupBLGCWLljIqA+or7ArCqh9bqnEJMinXB9myl6w2ExM6zocImaBgQntCZkoaVNZl4i9bm1/EtVSlJI3URBv5pZj8XFdZ2fx9E+Cb0S7xvveh4RwdbgyVu9xumuzBTG2t/d6Y2Btr61Du743/LCnXAfaJ0VEAQE+TN0Beq0Hi3bsvJXYPKCg5aK762+t7hplgDzDnXC/SpeVgODFIUuQk+GZff+iG0aqFkm0Y9INhYO7CvJ7rzCJlvgtELLHtc1/EWhARA3BETv/WiCKID1YrI8F2TCm96ngDV/EJhTn1 Chqtu2YC Vb3/5Grw37e3BfTK2dNGdlrB5+/oSHGD77hvUI5GFx+PUnYMhC0JjM+M/5iyRvfTotm/TOCXszs5Li2phGah3QSJpAzhsxKLKMCUIZsbWdC2/BwMu7/s6TMnTlvRuOTKgM6COCgWvBQuFF6RPtkn6bOWyek37EOtZgc2GSlV54I+kkzHo13zBtCmTbXJkaoJpQHUF9ebyiM0lZQ6AaNq1GjPRgXGd1B62+oGTxga+uvqovLnjGzz7veOXZ99sc94ur+Sc4Ch+tBYKYOVfYNhTmb2NMufrAQC+ZHwRrHXA1lx8+6q8UiOpF8A9THUHEBvEJPQ1MqL32Us22mVJqwfA4LcuD4EoygplCekmCgIDKDLkUlY0F/i6UJ/04+u0jZTHPi9c3LbW/EeI8XtZRQQ4SLk83bdQf+tx/kAv0bPk3YXcceJijoQyHFRDP1Cq4BdxxtqjpbOPgLS7GGg4xir4O3BajO2sxVVrC8VuDxsvetg2t9OubopxaK7AZ1N4rNobyKKHqr9usRZuFcm4bGWw/YPS2jllZYgnio9AWMcqqFVvwBw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this. Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Acked-by: Alexander Gordeev --- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 581446d4a4eba..8e1ceb0a6619a 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table @@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index f40d06ad5d2a3..ef79bf1e8563f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 445282cde9afb..408d0f36a8a8f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e534..f1ce5b7b28f22 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0) diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fbc..40e42a0e71673 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index a7b9c9e73593d..7211dff8c969e 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179e..22d6c1fcabfb4 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) { struct ptdesc *ptdesc = virt_to_ptdesc(pgtable); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe4..2b626cb3ad0ae 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -19,7 +19,7 @@ extern const char bad_pmd_string[]; #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc61..81715cece70c6 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type) list_del(dp); mmu_page_dtor((void *)page); if (type == TABLE_PTE) - pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe2..36d9805033c4b 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index ce6bb8e74271f..12a536b7bfbd4 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a5466..596e2355824e3 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c index 1715b07c630c9..4e1e45420bd49 100644 --- a/arch/powerpc/mm/book3s64/mmu_context.c +++ b/arch/powerpc/mm/book3s64/mmu_context.c @@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 3745425280808..3f28e4acd920b 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd) BUG_ON(atomic_read(&ptdesc->pt_frag_refcount) <= 0); if (atomic_dec_and_test(&ptdesc->pt_frag_refcount)) { - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index e89f64a0f24ae..713268ccb1a0e 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } } @@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head) struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3466fbe2e508d..b6793c5c99296 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, if (pgtable_l4_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, if (pgtable_l5_enabled) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } } @@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } @@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fc53ce748c804..8d703fb51b1dc 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) return; } - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else @@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm } if (!is_vmemmap) - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); if (PageReserved(page)) free_reserved_page(page); else diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index a0c1ca5d8423c..5fced6d3c36b0 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) if (mm_p4d_folded(mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); crst_table_free(mm, (unsigned long *) p4d); } @@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) if (mm_pud_folded(mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); crst_table_free(mm, (unsigned long *) pud); } @@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { if (mm_pmd_folded(mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); crst_table_free(mm, (unsigned long *) pmd); } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 907d57a68145c..dde847a5be545 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -141,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_pud_dtor(virt_to_ptdesc(pud)); + pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 58696a0c4e4ac..569de24d33761 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) { - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab15911..96d938fdf2244 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define __pte_free_tlb(tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 21f8cbbd0581c..05882bca5b732 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) { struct ptdesc *ptdesc = virt_to_ptdesc(pte); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index 9df51a62333d6..e3a72c884b867 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT); spin_lock(&mm->page_table_lock); if (page_ref_dec_return(page) == 1) - pagetable_pte_dtor(page_ptdesc(page)); + pagetable_dtor(page_ptdesc(page)); spin_unlock(&mm->page_table_lock); srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE); diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index 04fb4e6969a46..f0af23c3aeb2b 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *); #define __pte_free_tlb(tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0) @@ -35,7 +35,7 @@ do { \ #define __pmd_free_tlb(tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0) @@ -43,7 +43,7 @@ do { \ #define __pud_free_tlb(tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3d6e84da45b24..a6cd9660e29ec 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -60,7 +60,7 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_pte_dtor(page_ptdesc(pte)); + pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, pte); } @@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); } @@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { struct ptdesc *ptdesc = virt_to_ptdesc(pud); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } @@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } @@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) if (pmds[i]) { ptdesc = virt_to_ptdesc(pmds[i]); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); mm_dec_nr_pmds(mm); } @@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) free_page((unsigned long)pmd_sv); - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); + pagetable_dtor(virt_to_ptdesc(pmd)); free_page((unsigned long)pmd); return 1; diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3e..4afb346eae255 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_pte_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_pmd_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } #endif @@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_pud_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } @@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_p4d_dtor(ptdesc); + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d82f42ddd5cc..cad11fa10c192 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2992,6 +2992,15 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } static inline void ptlock_free(struct ptdesc *ptdesc) {} #endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ +static inline void pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + ptlock_free(ptdesc); + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3003,15 +3012,6 @@ static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - pte_t *___pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); static inline pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) @@ -3088,14 +3088,6 @@ static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) return ptlock_init(ptdesc); } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - VM_BUG_ON_PAGE(ptdesc->pmd_huge_pte, ptdesc_page(ptdesc)); -#endif - ptlock_free(ptdesc); -} - #define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) #else @@ -3106,7 +3098,6 @@ static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) } static inline bool pmd_ptlock_init(struct ptdesc *ptdesc) { return true; } -static inline void pmd_ptlock_free(struct ptdesc *ptdesc) {} #define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) @@ -3131,15 +3122,6 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) return true; } -static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - /* * No scalability reason to split PUD locks yet, but follow the same pattern * as the PMD locks to make it easier if we decide to. The VM should not be @@ -3167,14 +3149,6 @@ static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc); @@ -3183,14 +3157,6 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) lruvec_stat_add_folio(folio, NR_PAGETABLE); } -static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) -{ - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); -} - extern void __init pagecache_init(void); extern void free_initmem(void); diff --git a/mm/memory.c b/mm/memory.c index 9423967b24180..ad871e564568b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc) void ptlock_free(struct ptdesc *ptdesc) { - kmem_cache_free(page_ptl_cachep, ptdesc->ptl); + if (ptdesc->ptl) + kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif From patchwork Wed Jan 8 06:57:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B14EE77199 for ; Wed, 8 Jan 2025 07:01:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E01A6B0089; Wed, 8 Jan 2025 02:01:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 190596B00A6; Wed, 8 Jan 2025 02:01:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFE026B00A7; Wed, 8 Jan 2025 02:01:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 81B786B0089 for ; Wed, 8 Jan 2025 02:01:40 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3AD21140FCA for ; Wed, 8 Jan 2025 07:01:40 +0000 (UTC) X-FDA: 82983389160.23.0DFD5EE Received: from relay.hostedemail.com (unirelay09 [10.200.18.72]) by imf09.hostedemail.com (Postfix) with ESMTP id 04926140012 for ; Wed, 8 Jan 2025 07:01:37 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; arc=pass ("hostedemail.com:s=arc-20220608:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=7z3GO83jCA0AXO8KqhXGA9Kns6YRbdfBqi24olaa8MdIFLl+R/N2dh/ymDl5EurnbpypAr u0VCoQoduqAe35AANfjCc5auqcCjD+/GTKGx8RgEEDmHUR7BSkgXH6vepUUhR3F0iEPaUP pPTfCbAxDZtlP0leE7m6fMm9gxfRw6w= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1736319698; a=rsa-sha256; cv=pass; b=V01vo8NjMcQ4xc2Eb5RgzgVOOoMqRt0drbJTlMKTi9SfJmzaYAE4fPXeiuV9BLjSSuZn9t RojQK0jeHpLNigT1O3+y3hBsTukzAQStdy3wdZgBcicqiQmqG4P03l5ztR4DpO0q7i4s3u hlf+Z9OCH7p2z6NZAZfxV587++9lHAg= ARC-Authentication-Results: i=2; imf09.hostedemail.com; arc=pass ("hostedemail.com:s=arc-20220608:i=1") Received: from relay.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 87F8380F27 for ; Wed, 8 Jan 2025 07:01:37 +0000 (UTC) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5303716059C for ; Wed, 8 Jan 2025 07:01:37 +0000 (UTC) X-FDA: 82983389034.11.C742A43 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 1F4FEC0014 for ; Wed, 8 Jan 2025 07:01:20 +0000 (UTC) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319681; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=4MHi23wsKyl6H0Q1She232XG/3/Si5T2miCi9hNacAh7m5I031xJKQt+aFzEZ4CfgSWmja hA3NS+AM4CFqv1SuDAIwqUJqMnXHMWmx128YumephsSyK9Brk3dPqKajlxE9D1qPVTvBMv hdc6HukXKVynqLzzq8tD6CipkR5dPkY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319681; a=rsa-sha256; cv=none; b=NJK04NM94cSsg/ljZxhKkoEBJawjh1K/77t397UkSWCkiG8xqbOHIdcT13Hcv7l15Cs/Um l2bFBU8C0qfjWUs2Mm2bpScm3xxx5MGGj1X62InVypJRr1yUqqj8JSjPLWC8XBPYICa77C CKe8AH2xOU73TxAYrWh9KL22yAjuwbo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=X9S8jE5T; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2164b662090so213580825ad.1 for ; Tue, 07 Jan 2025 23:01:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319680; x=1736924480; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=X9S8jE5TpDvt3kiGDvAxoHHJdO70E5ViVsPfRtmjVBMFgDq/am0+YCY59cQKHgeM0O VDVClY71Dyrdy075I+u67wFgpwpJi43m5JiABb6WCj9lBXDvWG4Dli8ucNQKqC/zCFM0 6Oh8GlU1SbXI7OMKWkZXdxS5jOhaAqtMRFcuckKewQhruISBoIMRIfOWXWk5GDgIr3cz 1oFxK9pCxyjRTImYRsNo86WhvL/R5fKWiaP+xWNjHI1vsGHKOH5GQ0DSNLO+3GJ+yu3o 9FPud/iVXtRDWMl07/BcwaJ1G7wWUsJfi9KnTs1FZTETYUwiDHspnyEhjp4FR96AjMuk 3yaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319680; x=1736924480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xRYydr57EpvJJkVepk/VS3bq1fMuYMijkZdKL5zejaA=; b=i7+Nk5yN+Pwk+jl0P09eIjJJh1jJf0A7EXkEpjRnN9HKWTmD6iP7Bpp6LmxZyW9oSe AqgUjHzsU1Z4duIX5W1swexXhdrTQ36i/Ix5AT91GMIbxgIOXyk0kYxP+6HA2ZWxSyjx kBNg4sSxH//ViLG+N+5GYfpjx/4c1FVAKJ0eZwT3UIaSKE2+Gyyn3nyGRL7tcbdCTUlM 2T5IKGc+8yFBS03ogs1oPRj6jPIINjNor3JR0yZ01fEAjZoQTWff1JoUm1zYx26biCpF 9fz0+s6sOn2fQ8J00ZBy3vnk53ssRHXpeG89JOYWpUvcOVfe7lNRVrYk8U30vrBgth2i OciQ== X-Gm-Message-State: AOJu0YymL3rTdehkTqKBENm22yIcJNGMyiSYL9viMrunXi6TTlIJm/Am TWoXEQzfj+9JHHqZiPt0KIo7ipuVubFktFuRl9jdNkbmQvPdYCvaVaHTMLMUQm8= X-Gm-Gg: ASbGncu4W5JLednCPya7oweXWZk8CD8qFk1fWXu9vHoV9z8LQV2EwcpaKS1VRBSZHZ8 vPgiGosPLJXdqL5YmDWvfwBjbnOv2zv0k+r4XJM6LeUJCQWS1FOEjqjcW/DcP1zDpDKeBDG0PlC SB+5BB6MolWbdbLoIk4wXIATt30kQwmxELl28vvgk/kiIZlBDSCzzGzhPHolBXvVfFKdePpCqZm 0RD2awBpD02t3rHNcwXwrbkSpGtCL1gyfYgvwKkRN/KS+X9RSRv4Xtu8Z2qinCwgxxS/1sncwcM LjQmBCBCzDAybsSJKEgigYvr5QU= X-Google-Smtp-Source: AGHT+IGA+MEO+KKVHOehVcxKdAaP4kVFc+4x/MGW5FidU7ZYZhG+ejtfRbHdQiRj7WdzfqmV0LUjgQ== X-Received: by 2002:a17:902:d352:b0:215:8847:435c with SMTP id d9443c01a7336-21a83f4b133mr25527805ad.12.1736319679689; Tue, 07 Jan 2025 23:01:19 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:19 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 08/17] arm: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:24 +0800 Message-Id: <327b4b8990729edd4ce97d9d5acbdaff2d9fa1d1.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-HE-Meta: U2FsdGVkX1+mnjlw4bB8FoAyqSEkOpQG4ye59DJ0uVkIoqfJ6AJbshSJKiWBFYaNA17Nuv5QpbLhFOJzW9b7ReC5OO6od6mou58SuBt7OXD54EUZplEeSGvd0z0/CvF8Dk+prQExBA/e5B6X0Gr4eUtdJM7seG2sjpW9VVHDDSmnTFW1Gd/lZHWzxKARQB71sFXckS+2WPbNe/G1KO1Xic0wznbtbZtntGO7O5HLW+6lVE2TF/ZFk08oQrQZnXGrL+IOL98UzpYMMFttb/QnRpgHX8gvoMXKlvhRh+vwTRvR+IEbae8iHewRP8BKP4K1CmuiAKAQttIFOKvzZQaaNBvtnjlbyzU62gHASv3bquqTWJul66S7+qcfB0KJ732RpjZHWnCIL6lDnrFKq5vkPTeXfn6oSK+Du0LFzCDZCynvuH4cJDeUlBVJXnBh2GOAJyHSlBXVMroQg6K7lhLq4WUo0TRu7z7qJBaQ7HCfu0vIXdjfwbGr8/ELCUGKhEOPUQMphyLXc8FZTw6XSf2UNSAv7ia59hB8JznK7j+U/souyU8c5P9LeG2PYgaBEsDH54rYjd0S9hVKLEOqBKYWkFkgB2I7pQNonkU4vV72tZCvbNyFBtnxbTH2BGK5adEhuuUDTg16LgBnAHZTIIfCm+BTDCOW/wN1eidWNf0yxUPl9EARFUQsgsvw2JBJyYQCBNne9lFGzyaF8lCl6YGABJh+xJlxzn4Ptqk+5m8XK4ODVAZ+pZWsBrgp287JjR8Xd8yhOF8lFoBsBwVExqxuEgwnim94jdVzW6b/j0Y8J5RkGht1epHkPEAZvOuoNU2HE16bh859FnYSaTh13zgMbedFaFBrfkZAKX/CTV3eQl82BsBq/K8Go7TdtnXiuDe9+QG9hVx1zNy/enhK9B/fpPkIpqdag0LxJBmHBiWkJ0+buBY6mAl9Ke4sC6Jkf3dnCf8rFxYW+BmBEwjRVbX txLrVhKI 15m0qwYqmV1s2gE9i/Y9CJ8uaPsnJSiDKhAqxjJpImYD2l3qQ+jiqUbXevcqwEFVwQVa/z25A9ib3Ru8Y8EC7c0M8wDePyLOGj7g0VNJKS7SL8wZsOZx5g9a3rEEE+lnd6GZ8PxeJIj9ZduHVeXLSyqA4xW/67EA+0P+kToT54SLIj/H/rbLl5ycCCUrQz1LSZVoV1GbXpY7yzLZoxdpg5mNjp3aPjkJvCm1s4HKdld+7w2ZBbmS9FFI142BmIRNoU35fLuZqqFhevgtynRhXvxeDmbJQp0SCdVMo0b0FB8jwfvgTAWvYhS5dDzOVWqJRVvlTz9eXOH2/NPfWcjUUWEnRCAIkpmW/QoH8KgO/2725HJDmVO8IYSoMoYv5gWbZx+03Os1SjBc/y2t+kB0UmBJhyRBrUGSw47wtqtI7XMvMPSQhSDPeoSQQS6TaQjBBN2r3BKv1xNr0AGDrlov/cPh1SUVdGadiVH3lVkNc4M6Ps9Oh1+0bjUYlgwGH4C3eRXIF8BMHccodvriJ7hh2/HFiiAQ0Uqe+AASCEYU8kkPVtBk= X-Rspamd-Queue-Id: 04926140012 X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag-Orig: 1736319680-316222 X-Stat-Signature: nn9ybkctw44syjztq9e3n51b4segxatc X-HE-Tag: 1736319697-559775 X-HE-Meta: U2FsdGVkX1+eWZ86fiZ7NrPJ5eEqUv8cN+klQM3kQOBi7+CPNG7nZzv5i8BEoEziX6snafyZfPmN78MvzSSdBKl2d59HC8m3K1KyyXbMtM0FrYQ5uEzK+CIpgVNGQySQuYdMZOlKuOKj1rQ35FgbbwIDiKstVwFbdaPifcgVFFGZiJfN0dOmcQ3hI4q2HmS7p2YA8O0FRfHQW0aoF6c2YQp9VxPqVLJphK6mAGKUG7ZpwzcT/xnJDbfU8ue0nrW0Zav4jJPZuTjgEpHRMwSdMqYMqGKxIvitisGNw+68IsznSQGiRo4yZRcJxXznw2nZeCUs4TYmh2SQdaVu6OX04EBnJLYKVg6y+u7KsKwDDe8OfzgzWnntaLSKDB4DtVNmu7n5RYn0P6rw8bQZw/FsEJ456ys4j0yD16G7wwuFClqfxIFYcFUPQqwT7UxWGX0NmjX2OyhjwL9954kbhQuIcBg/M2igIcedAEHY9G3GK947gp+and70yOuDzUEFq1DdZPeKHBh7EzP6kwKzvRxhlLy24RyBXzBnqV60fiK9ZsDlhduC0q5+pdWbwtu/sXsIeqkemuXocz8OEZ0hTCFjFjFEDgU4upk2DpWIahAzut/FohyTn7J1Z7Tosd71ZH2BOu72nErv1Knzt/LwjpdS+p9DfS8c3Tzzsdna8McM+tOMjTrnAIEhvZOVQmIbZs/vV51awILxWcxgkTXg/wBKNhFG96wDgpJ2Ft7Uebu7AF3ZpcVJo2VCRYIrP91DylNqa97DuythcPaAy3eYKuWs44UKup5RXMO96mP1XmnaXW7mHbvmOUJEC4xKKZU3l+eghC+nv3GHP4WeoruUzW8O/KQD05XXTDGo9QqPNU0hEx6sE9KQOaHEKVn6a1WoiBFQsBHv7LHMd1i0TzSddZvRcBywEToZIlQAXmDHd5vLdbz5z+t4ANwzgZecwST28AeBPau76rP2mBvoicl6t7b Xm17IPW1 M2Du3seTxNhEDfAyYCN3lChAuVnySKEryPrg4PghBE32nIF/0mJ/fDqepOYcMWt7TNBzanImo9PEQ+TsO8UUi1ltKJ6dO6jYbw8RaTuD84QiJ3D5ubT6p4+4+3Xy+iPq9uusIjGo61KKZeYmSH9QoHcCtz+UP8LB7fiHGYsuNha/MyUHaifxVoc5zvZEWERLvvf3DbMSmEMeWv13RA5WtJvj2sWxwhE3myHDcSLgmjf9Z4BL8rPpotMs9XQrG+D5Ul4jCoSchDWIR5o/slBbYOWIUVilDYKPZqfRjiXi5n+bdiw9IJdXrAXVb3wQH0KeFIhob1Dns5HxsRq9jVL7XpsArMbrAckwtgv5p7vQlIa3bPNPjtgvuz8vZhR0Vqans7954+ecHsUPhG+XF4z9nfW2uUGvPv8lLVYgy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org --- arch/arm/include/asm/tlb.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index ef79bf1e8563f..59854c6b97bc5 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -26,12 +26,14 @@ #else /* !CONFIG_MMU */ -#include #include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #include @@ -41,7 +43,9 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); +#ifndef CONFIG_MMU_GATHER_TABLE_FREE pagetable_dtor(ptdesc); +#endif #ifndef CONFIG_ARM_LPAE /* @@ -61,7 +65,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) #ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); #endif } From patchwork Wed Jan 8 06:57:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0C11E77188 for ; Wed, 8 Jan 2025 07:01:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 895CD6B00A3; Wed, 8 Jan 2025 02:01:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 845CF6B00A4; Wed, 8 Jan 2025 02:01:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70DEB6B00A5; Wed, 8 Jan 2025 02:01:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4CC406B00A3 for ; Wed, 8 Jan 2025 02:01:38 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 09A2280F9C for ; Wed, 8 Jan 2025 07:01:38 +0000 (UTC) X-FDA: 82983389076.17.6D7202B Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf19.hostedemail.com (Postfix) with ESMTP id 44A8D1A000A for ; Wed, 8 Jan 2025 07:01:36 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XR1GTo7E; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HOmYFC5rDCHPndkDeYhMazAtFRSG5eQpr+EJP/iMn+M=; b=VJ6gCP6ZnMZygQKqTvqPpr2juFTZPyNtR4PDdQ8PabdC3lpSsgzfI3R8UrA3yK72f0xB3k PqiRO+OjPkiSK5BXRIHw18XMmTFB/siEx10w6rZ+oWGOKBGAU0Ujr/SQaq4/NjV2la4BhQ aVt9xYhiFx1xTUfX2mkakvuIpnnKsVE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=XR1GTo7E; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319696; a=rsa-sha256; cv=none; b=6L1PDrecIfEx2cuaftgqizRW/vvCNgfp64aQmnNp4lnejh1he2lUOs2+Nd3RIPDeZuoStJ bWb2vqwunUH6sxmhkmMG8QbebbAnw8LBhQHe47H+NE8GcDU6HQ52hoqvetuS8vQZW4tuCy gxEMmcoPgj4Jh9lf/iP1OPbb+S28UXQ= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21649a7bcdcso234385585ad.1 for ; Tue, 07 Jan 2025 23:01:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319695; x=1736924495; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HOmYFC5rDCHPndkDeYhMazAtFRSG5eQpr+EJP/iMn+M=; b=XR1GTo7EZqh+IHK2zp4fFysBlYRcwwa7UYbtH9rlkfqqYS7XReNgjHn+bn4OA5nQkL Mo15352hszJfSkux8DGXiSn2d9A4LwoesHADSdss8VO4PtOLKKlniXJ1Y5O42iqRvRXW 1NLQa357AuG1nX9sPTj2R4u2gWmRPbYbpkhNggHZknYQxJM4//F01/bd4VHl/ecyzs0b Sdjz0NXKrq+RDJkXfYzijXbUuE53lu3v2kP0YSVv9FnxkcRxflulF7RogiaTXsCPekT3 b7GtXsYjhuiBVBt7zF8M6EWHdxkc7UP2f2Tzj1kh//fWPEVpQDTn/H7p4/5R4UhY3eui h5UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319695; x=1736924495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HOmYFC5rDCHPndkDeYhMazAtFRSG5eQpr+EJP/iMn+M=; b=qjHsHNrMwSLceSGxBbGNszIc77lYu8uC2XfHp4dsy9+l6cV0nPtvmDJiJglmP1jrol FO6fpUHbX3hHWgz28JIGh4DKYLhBnPOtnGDBjlYNxEp5NpL3zElOisO4DqXHmwVK0aFf RJzyABGZtnPZXqt60vYURG2ZkrvR4Yoi5iwbCuE9yvXk8GSocJktp+97jIVZl1yNX47N IH2iQ7dIJ2Gwuc/PwWDCWuzlPo257yz/nO0g8ZhgUrVb+fHLCGY064E/9/GUoUBu36cC GaoDhwRKLEGnCzz56z9PUm++qxV1DDaBCK4mUL43BeG3nxowpJe+eJ7MLDDa2b6UPnFV Lxjg== X-Gm-Message-State: AOJu0YyuBAJOPepqveeC8sK/JwCfAnXCBvwn4XfdxZaPpvQL+HmVb/34 uW6UEa8CddaP5Xd8jQAv3ce5fL63gYR4UgBS4jfoLFsm7nGn6xeqpEwvjmV3GBE= X-Gm-Gg: ASbGncuYU0zlxaTQpP+MgpgwOsQiPva90u7ZcC7mzPPUQ3/Y9YiTJ2wFO5ThmZuHX3n JdkjA7vAnVaglNRdYoYB7MrYR93MfnoKXkLUrMw2DjIQnrp898+1H0CT/UGYD+UfecfHXsPDqI+ g2Zv5BORT6zAcAzl1ateL0o+1kYNSwabOgsAnlZXoHjY1Wxgi+zjxYLB2VDyGcZG+BBvU+lNDMW vK/Y+qMWvfwLsZSjjoAyTWzOs1xBV22vBXlficBxNzdNE/PFIxs9/gM37lC056EoKChMTiLOmO1 sRDATV5M3Iw4Ypb1j1LPXWHfhOI= X-Google-Smtp-Source: AGHT+IG2O0LK8ylHfVSG3OwO7HgAN7h4l7Hu7AhiZODZJ2StNi5+XwyqKLd4EhZPQDdK0Gl46ps1VQ== X-Received: by 2002:a17:902:d50d:b0:215:7421:27c with SMTP id d9443c01a7336-21a83f696a9mr29561025ad.29.1736319695027; Tue, 07 Jan 2025 23:01:35 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:34 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 09/17] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 44A8D1A000A X-Stat-Signature: 41xa3nxwdeib8kjtbs8m7jx5rxbnabpr X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736319696-750108 X-HE-Meta: U2FsdGVkX18B7/bzngh/wXD0I72GOCCVrdpa3d2p+zWGMiQv+Pl0f1EQ6WnlL38bqTPWqnRvyls8e6MtESp5vCdGAKwidaGMBarfFzRaFy9FFp4PzpOsdTuQyCi93s/H5L268CmTiK+oWfRGSt/hHhwggYi0iCOlF7FYJPV7Lvp4IjF03vl1lT6HiTWJTRZoDE7PxFNuRrtiiHTZgbRvF+qYJ1xkUDDm71U4cMxVglEV1FUn3THQaFDuYyLhYNnA5S+gpiMnO9OG7/skNtVXJAdP0J4mUZ6CyDvrAnBjCgo5wa/x3jm3kxRK33F/7eArleHGxDqchuPTxY2Sz6dRYJEzwBl6fgyorbraO3SqMttNUjD/3+aK2l5S6FFKnAX9psLYuk/kouHIkOP1TvXqCZMESCMZ/+JUqgg/ncmTf5+k85/ls4avzj4J1gYmU+AfooHtS++WnEpaKYfYduiTVV9jrkDg1F9JNcW6VAaO0hLC1Sf0LbCkcSEmxq2ZghpeISRvT9WDh+5ssbiSV9kLpa1drb1lodAOzWjPSbBmOqPZx7sFLYG42v9b1lezjqhj70lHpNQKNRfGYF1aMFCsCA2Kgfs6/xLDPEzv6CNUKfX1FvY9SqbMVGH8i67YBM9iXTPMzLISCUirrVjjs81VfDEUPbW0m+VhdS5Le4ETF+Az5j7vVpiYSSeUuyb2PKWNoOYOvEiG9rmS0Kk+O52yjy0OtrLhwm8pWNKBimOM6Hyz6UlUt1nCwIkU56gFN5dDBZ9njVKMraRI4gQLu+IaLYZr/dclknL6bi/TQw0xi6PVwAw85h4IYtsQXk1QCQz/GRBHCYtV4gS1Ogz1k4eut8/bImXuicC7RLhWuqXlx7fIalPOjY1WtYv926f5v16pLItpHFxeZanUWW4gNmChDJ71X4kL15bYmfhyXbJ5DrHS9SSc8EvgHbKPQnFgpZY7gmL/goTzAKP4BEwq9kt LdAtKjBL NWkcNRLo8g84hdvBgDvFF+Oii/UG0IQfrf9ctv0nt1hdyLNexs1AVZN+iBWQzGzNRI+NvkZsvhrN9UFFukz939eVlirxl3nQhbFRtVFz4tyNff3SswpIgAOHea6hY/Z+DKjdb+qX2g3lVMUcUvTQxU3n6G4tCt7K1AiICab5DYDsms98LlgoeoMylf14+dK9aQZXfqLUZ4Rh9A33P4HMdXZCrFK3R6tU1zQ76+1iAaS0hv3eGyrYL8KfdJCT395M0gnc+c5gP/K/1QuQZ1Y+jUAj4+ofeIJr2QqqKZrGnpY46fluxH24nRMZaSKWdhZaUpdhCo5jzQgfqf20lHgMEt/jMXyaa5M4xNdL46ck4CSSqkQq2jjerLryeLsopF6mkmDCgR/D71fguH8IkB7Dw8rajtU9SRZap8gOkZMQC6AvLOrfwp/UCK3Bi80pJKgK49cSnl3CDAu87y91512wELwFL7gzx75kmloU4GD0L3EDiElp6RhXVdp7LWUjovnpqLngCOoaQixq8jJKdAWYIfpA8NV1J0QfjTMSIaNA5KQaiCNo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 408d0f36a8a8f..93591a80b5bfb 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -9,11 +9,13 @@ #define __ASM_TLB_H #include -#include static inline void __tlb_remove_table(void *_table) { - free_page_and_swap_cache((struct page *)_table); + struct ptdesc *ptdesc = (struct ptdesc *)_table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #define tlb_flush tlb_flush @@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, { struct ptdesc *ptdesc = page_ptdesc(pte); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } @@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, { struct ptdesc *ptdesc = virt_to_ptdesc(pmdp); - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, if (!pgtable_l4_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif @@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp, if (!pgtable_l5_enabled()) return; - pagetable_dtor(ptdesc); tlb_remove_ptdesc(tlb, ptdesc); } #endif From patchwork Wed Jan 8 06:57:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4299EE77199 for ; Wed, 8 Jan 2025 07:01:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C19D18D0002; Wed, 8 Jan 2025 02:01:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ADF408D0001; Wed, 8 Jan 2025 02:01:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BC888D0002; Wed, 8 Jan 2025 02:01:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 670F88D0001 for ; Wed, 8 Jan 2025 02:01:54 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2FB4CB0827 for ; Wed, 8 Jan 2025 07:01:54 +0000 (UTC) X-FDA: 82983389748.13.6B04F2E Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf13.hostedemail.com (Postfix) with ESMTP id 5A2502000B for ; Wed, 8 Jan 2025 07:01:52 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bxdpY66b; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319712; a=rsa-sha256; cv=none; b=1HYzvHLninl3SVwXnO4T/h3mtZA2r+t/hDOU2SpHSERpHKFn3C5+6sCFZxuCZjTgotes2p Z25SBrj2W+AdUQiaLb84qRrUX6urP49Mjej6BQ2JmiAXWBhgMUxsrBEA2LPpSi+/71B6mV 6iCm1KWZENv7sbPFNZUGUoYbwuGkuN4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bxdpY66b; spf=pass (imf13.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xJFt+T0ZGRLwl7qj4QUye9dvBL3lzNJEqvkgjrbvemg=; b=ChUmoXf0JEgb5q/sUkQpolsd/bBTfEtDGhpRMOtfuH52MKju0CMRcLdhtJ45iy2QX8nBcJ yRoffeEDpFgJZ94oG/dEFn4qF47lcK2C7YXpkkHmWQOnrJcMqYin6BRSp2NBNwBsoRlwEL bPtcFTs6dW9o9zhP3cLySpprcgPr5CA= Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2ee9a780de4so17922299a91.3 for ; Tue, 07 Jan 2025 23:01:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319711; x=1736924511; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xJFt+T0ZGRLwl7qj4QUye9dvBL3lzNJEqvkgjrbvemg=; b=bxdpY66b7INs/MU2+ej0PHeeJ8qrGlkQWzqimUNOrIu54/sXeD9vcmD1Sp4An0qoBw 7mblXIQuEByaevfNPyOw9/ZtPD5EN3hMb197PWbAE+2Xmb9dHfxXNGCJ3hfmyqRs6lF6 Ws24mnnEjkSRX5THOW0zhibrZgDYC6OhFD5AW2OXAzeMav8CKzit0ez3z0+myiXgKtLH tiizKID25QlVr09G/RM7sIRSQRZDF0kQoH3x0k2wyYQ5Axs+7WsTfCzVDipPlYp+qILs VPyTSmB0mEblCJ6XYJ2+OKOSwl8vqjA6kqWqaXN1IeDEOxUyr+aL/rQoBNyDiu4z+Qmw ljzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319711; x=1736924511; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xJFt+T0ZGRLwl7qj4QUye9dvBL3lzNJEqvkgjrbvemg=; b=I+N5xu9l6o2e8NujnKEsVw5EsTN3ffTfqQzE9W1Tw2IIlqLVOky5zI4/7UX0GdFtu9 f7jMHKswNkj8u2YzGR8V2Bgb3IGeMb54l+auAVRhNlrB2qQN4v0Uhfr+PDyjLyPwImNh in5VbKEgnd1K2YcLmMy4PhH6kcAfnOi9YcAYTxktEjRWVnJua7Gt4ybIilJzZ/E+l14O 8Z6Q/OO2HOIKhS8RAu1l6wK6WFVbRPNTqQeiYDLtsgNBjEdlMZ2uDBeYwAARe9jDgUVT 1RxhIobUZVN2iWlpc8zEG8+MlBBfOGaJsTO7HOrPgRKXLeF6moNEcIOTgp/ewAXA+3/6 cGQQ== X-Gm-Message-State: AOJu0YwkTqB4UQLY5CEJ+qNoEHdajC/0VI2Vi3oloDPDsE2M6FfVps5K t632SshcbNAzg3QRCJthLGF3Ghs3swxQvansjKL0mfexpqThQbGy0oduCUlA72I= X-Gm-Gg: ASbGncvYcStM7ZyJEHni6QwJnNUOFzxoqkRVsy5xUjWZLuQi9fp5zfRP9jXMoRthl99 NKsiXolqF7q9pTWQaScAif2NqE0o/8mYnd/4zHlvPKkgWpUsFDdcMHUKS+0RjfyoJL79xBePQ5X +p8qzZGFicEb28YQU9I+LbF12dQwI2qe/YDKOX1EHB9r6aBt5f0qXHZ4mPRjQyPyc+sBHO7hmEA o5NX1rV6Z/lElkeSOfPZZc9O9qnoNULOA10r0WqVj1vfE47UE436rX1JkJ8psTunKsjCkZGZngw 4W/O1soX54CHWixnExjqmJrn3UQ= X-Google-Smtp-Source: AGHT+IHbJSdEFlMcYFI5dK7LOGyjdYP9+xvlrgbj38Eli2e60ze0npY5GY+Stk/tuNQHFcVXscZRAQ== X-Received: by 2002:a17:90b:2dc2:b0:2ee:863e:9ffc with SMTP id 98e67ed59e1d1-2f548ee5378mr2382141a91.21.1736319711135; Tue, 07 Jan 2025 23:01:51 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:01:50 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 10/17] riscv: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A2502000B X-Stat-Signature: q9aeucx54j461s4ordk5das9hs6j9g83 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1736319712-84376 X-HE-Meta: U2FsdGVkX18HEsEkiqEKbXqZes3T3OQ6pIqhi86LC6dZsIalM6mNpT2yNbSjGJ29coxfwGUhTZXYmgHuYx7ubdnMVJR9Yz5yMrbw+NZrLOJYTe0Q0NIRAntCL7qCHv2uDz+e0Oqxo72kBCzqmx2gGVTtKD4Y8yT2bXnhYJY8L7QdrhcuWpFiA0VWg94G7olEi8mUvl3whzIwSa5ZoXpDDTclWiNDDinlb0W7xJjLjL4CUil6hvviCY+uxpIThzsfEG8RXX/AM6R7QZl8gEQQXr4UPiIs7110VCMRMFwCKQpwq0mOI5BJm5pylsTlneIuKMpNCUZMacahpVyTKuAdV/9Nyyv6neXkyYqZGNEatOa6QjtaKJsixE/X9WYGg4srFsRkxrn2vAx5KFeFyhaK2zGDkVM+pxlZCztiwNCihDcVj36TYRwn06o+k+M8HrytSvazVanBiWp6I6uMEVJu2AqtpvRbUkPyIfm0rCzg5pTbFv8AOSyBSEbLh4WTlibspbwWrST3L8T5iEiHxTb9o60jB4HlGlAKcU7TTXdbuWoADcGqL4BDphEdxU5mGMwIDk7GEsciCSZAO9sMaS2L5shmNEY3T7rrcOC0HgRd2F63tqMsPQfY4Xt1+R0maK4EiacZIER9HpoyXvZU1oQv3nQaOjU52F8T8VwBy0eBDiY5YDRw/WF8VCCx3fd2J5Pm4tl6iLKK0Y6Y6TOG307ApicQKNPyT80HDtG6Pezr1KY5Evb8VGO7w/E/01DWShHfk9rTBb2Oww+iL1pMxCcKKSTbwS5Z0N58fXA+cqc43igcC3MQ8aZ5lpPM6cYSjYOIIOIM049LtsvlYEm9niK0O3PGEWuF+cCp1hYmkR/fiz1sXEnNigaG4BYJWQP3yQ7BWCX+bEmwOvRqXRMO7+IFpHpm2/Qm13CKhtEhXcMLDk/AEWdIzes4Mjo+5gTdseSHV1Zl+iX2K+/GaUdi+ww jPJ0qvJc 6qRwn3MRyX89a1iL2xr81k6ugheBkm76baADdn7pSKFqeMo7Kqaw6Sot/KBNgwzys1qTjIiFGKEQHuPT8VOAP5JfahUP8WX/SSExUKJAYQj3glsBTjd+9ygQkOWZAf/hZ7w7hm8TVNPC8Ih6JrGH3fK+lWAK8CEMFveW4p5FHn9e9bChmZNztMmYlEmVtxP/ZBb/rT9HinWrgOnhom5+pb2OsIvI1F3VFA+tQb4VHYzBizKWSLYVvt6NRwOL+mgFDfqrOu7TmVWiRWAVNGY0nGgWWZOz8U3iNM+Fy7B3IS2i159VAveyrgOHk/nhnlEpN02+Ps7rpCn6qG7ljsrm0nZdJ+iij50b9EU9wETG54usV77FzBfwZ5gIaFLmFTAJoNEI0PehmGm9LQV/iouXB7TNGo+CVcp5/3+QfbH0z89LnCVAsEhb+Cx9sFEG66FHyJvv/K0XiniewLPxIzzkTMvV8/UmEyEUkd/fV7Doha4p3XaS0rzSlmkJEWpLETkgvdegVo3gv4TvbcUrmnAYzMa7JCKzrmWcumiI4tnjXiWucqTk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-riscv@lists.infradead.org --- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b6793c5c99296..c8907b8317115 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,12 +15,22 @@ #define __HAVE_ARCH_PUD_FREE #include +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_sbi_for_rfence()) + if (riscv_use_sbi_for_rfence()) { tlb_remove_ptdesc(tlb, pt); - else + } else { + pagetable_dtor(pt); tlb_remove_page_ptdesc(tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm, @@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, unsigned long addr) { - if (pgtable_l4_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); - } + if (pgtable_l4_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); + if (pgtable_l5_enabled) riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - struct ptdesc *ptdesc = page_ptdesc(pte); - - pagetable_dtor(ptdesc); - riscv_tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, page_ptdesc(pte)); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1f6c38420d8e0..ded8724b3c4f7 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -11,19 +11,13 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_MMU -#include -/* - * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to - * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use - * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this - * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the - * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); } #endif /* CONFIG_MMU */ From patchwork Wed Jan 8 06:57:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41F7DE77188 for ; Wed, 8 Jan 2025 07:02:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB6AE6B00A6; Wed, 8 Jan 2025 02:02:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C40196B00A7; Wed, 8 Jan 2025 02:02:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1DDE6B00A8; Wed, 8 Jan 2025 02:02:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7BA196B00A6 for ; Wed, 8 Jan 2025 02:02:10 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F2DE01C7F5E for ; Wed, 8 Jan 2025 07:02:09 +0000 (UTC) X-FDA: 82983390378.07.6B44222 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf17.hostedemail.com (Postfix) with ESMTP id 0D7CB4000A for ; Wed, 8 Jan 2025 07:02:07 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cGtZjN5f; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0duyzCo9dAwX01bCrGbAFGgpbJyQOIdvGclplTLFMvs=; b=Gu3SV1tH2DKpsxH56IzVhxQYLHRWY6sJzoPF9RPTQXa6b6vBOQLxI/bJk645AtJpBLnkE7 95tKxeV6Kz5rMxj7rbo6XB5oiNIVdoMrKng8Y6wyW57hMLeSCbUlksl7cxIkqHNeo31/dR 3MbTyyVTwJYfti/QdfLNhGMlhe3yJJw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cGtZjN5f; spf=pass (imf17.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319728; a=rsa-sha256; cv=none; b=GVuDuEgrnCWsRiswZhIDxHGtkVXpCZDPBxMGxD72eNQfLUaJjcmq5QUlycgLA1cueXSBeK 8JDCH46Gcodaq6+tAX+v9KI0mUlxyhxsUU8qdE/YP9pltOkVYMNg43rFusVo45qWMs2yeO axg/axbj571B9+95N9F44SDmSQqVQhE= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21634338cfdso29972945ad.2 for ; Tue, 07 Jan 2025 23:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319727; x=1736924527; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0duyzCo9dAwX01bCrGbAFGgpbJyQOIdvGclplTLFMvs=; b=cGtZjN5fyc6fQPN8ZMP6cskBdhWFRp0AUOeqgibwg6CfH7vg7+7cnOkxpWN79KTdwR sQxnaLSYqx5FWuV9pjQD0jMLQaN01+7CEt3U7i6hNoMS0wuXFNS5mwYKIOWenLaLG+Ay /HrGxORP2TUKBDS6ogLkySVc5PJr7+iigHdgHC/ByNzDnrD4DL/uUbciw0J8KjT5LAvs S/Fr/oo18AXw0hiUBjuSxhH7sLgb2agANrKcYzdQbJKK4slddtvIednMjk4ZqSqO9cde EsAvxbMRsN+/tkBAjECRHwn957k1bpqe0O+rB9IoQXY8X4pmTtbFB6DoijmikAlXYy8f ejCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319727; x=1736924527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0duyzCo9dAwX01bCrGbAFGgpbJyQOIdvGclplTLFMvs=; b=aHbG6vYWuHLvhb7btirI7nECWIA0veh/cVHsaIeKNKPWvcAag92IjjsAHT36/5Ljvh Ui3tuY08GLB56zKFJ+dSI47/l+MFYAlayea1E7Gku8FWllHSLRotemPQKxTyqYQGR2p/ APc/F61mYS9t6R30L1o6G9IMUI0rsqNrWz2r1FPyrWgiGFJFqHL+gIpYFZYIB3slq/8D awCyWvAMxWFkKw9iDX++P3aMWGsjbvRLlZUAJ5LIYA1/IfNUye/DfzlJthgm8FKXUR8j 1vN/kzAkGb1LCFERlb94WhNDfw2cyPWcP8tfDuCey9P5jM6HPe3TgPps7uBynh0cTGc8 WtKQ== X-Gm-Message-State: AOJu0YyuMU6YFRpEHS2JYjIuaTKQC/sjV7B8Xa1k856xc1GHvwSYXPoa f/+1qKSq2Zh/PIzb8Fv72aXvmJAkyVSrFq8dC5dowemoNWak8SOavyCZwcw8Pf0= X-Gm-Gg: ASbGnctIa7WkDdaYDhK+L7WpcPqONs0J26xuTGnjwNTgPrNRMwGj4Pk3PLuJu+LzjZK OhR+WbF4lPxmRTfyPxD16nurWpJCesvKtG9TXlIWeVObQTg1m45wCbcxhnh9jcNXvJGz8wFYyQJ bFPZ0FvJZfSONoorcGVwAuXbk1P5Sxw69R6KOQWCk2g/YNebYYQcIuCsfVVIPYyn9nffGcI50fX 0kPAgiBZ9NQS7QbxqWPclhyTaazudTwiN8WX72AcuoMbPP7aETdysWYT0BFeMU/GEqgJZuXWFBr S08bL18BzO72s48w0vLHBZ4W1ow= X-Google-Smtp-Source: AGHT+IFHWir1ocwTjoDZn2JicpigLOS4g472SxX6v3Yl/J2b2LHmsbTHHj3CMM+1OfXcdbPj30xuWQ== X-Received: by 2002:a17:902:c406:b0:216:7ee9:21ff with SMTP id d9443c01a7336-21a83fdf307mr27262725ad.49.1736319726731; Tue, 07 Jan 2025 23:02:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:06 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 11/17] x86: pgtable: convert __tlb_remove_table() to use struct ptdesc Date: Wed, 8 Jan 2025 14:57:27 +0800 Message-Id: <39f60f93143ff77cf5d6b3c3e75af0ffc1480adb.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 0D7CB4000A X-Rspamd-Server: rspam12 X-Stat-Signature: mkgjky5dszzzi67ku6budxkc8n65nbbd X-Rspam-User: X-HE-Tag: 1736319727-402577 X-HE-Meta: U2FsdGVkX1+vni5AnuPZpsdHlwEsFNkqqglcul8pAYXC314Dev33BDt37X8IYKSsp1Zmh0dF4FBN/fI50VKx6nImXL2QWT5cWYJsN88lGpI57rYMs6Dh2tMCdNPVvUCTgjTwZX+F7nYbkWabPjrtLFCdmV1t0PO3eKrrIe2YC487E2I6ChvyLFOKeomi+X8mHxarh19hLiMAwPBSeC0ynI8SinmbuE7wfEG+wfF9lE4mQsvXwe/uNsNDZ6ho0LA7n4QTcBU73wnNR+gMsc0ltCi+eqSlHoHtM97Ox+adi9pJuNI4JWiCi0AEiRJcvk4HsG52LRmRY+r5imYPCg7q6wSbMfvunu3M50qIQQEoOMJzCmX7QkTEhbbdPVSABXI2eccyBEhpKKpjq3caL+/Ndy9bOkDOk6873q4XT20z0uUHSTxHkKJySQVEOLgxUUGXGNrgpxsPoWnGcUW0XhSFzPIZ3BRMqx1Slbk3HMOXowtYDynLWK4GlgxdPK5Efcn2Eb8YOqB8Ru+jUxywhik8Ty3/IlulyHZU4YDC0NKA1I3h+cncCBijblrt9Ib9r32tcq0LqKZonYEoxEk9vrdQGuLxm7RPS2Q3wNujWMG9V5vVk3zuKf0Od4GbX6fM2mwPfU+jphbTfQylKu+FNybAzkKhsN7XFtZquUx5eNtLED9cRzLm3NKlojefdc51yo30vbd2jqSf/mNKepcc7Z22JGoTPUSzr2EPbb6yR/cie1hxRWqC5pQ5gUlkGlwqMPRsaj82xedbhWMESAGIbnZ2z9EDpfhfeapym5ylvCT8ZLn7vWV3HoMOr2kLF2XsCpUe3dPSsm4ZEQHYcps3TknA68YJlepbbTTPxlz2BHbvYy7a1/B/sTGYfSCYgyzkn5zUhOXlP9mkoJnd9flT4IejIO1ODzW1qZBBIIikv0IVKy/b+mXFWeh0brgfWuuyFGpOJ3dMHaEX+yOweNUcZJC /y+B+3RE JSsC1DbvZdANaBAjv7gLPLlQsMVuWiCHwyMwneMfPMm81kJRsHxTEubINMbUeATtfKNwCIvZylBdlKZICL9UbNNNC3tfx/oR0TlQvzOqV7Zxy4nyEEqa4sdewC1EU6sQo+3NmtgQl0cJxVcnxoiJUSS3YgrgZj2JRIYDKavSwhvnRorXlusmYvSMTA/OsmytV6QijVTUrawJ6P2hP4TukGNYpdnTqS+BdolMZ8dFppKbU7L2nuV1/G8FjKAA/zvclEj6c70cmKjXQar4CthFu5B171ClA+2e3MBdxaErgefNwDU17RKxqTfW6fAFhcfQebBvrD0UqX2VhNECcQulCzNZ2ZxcbsHOnjJFPHGAhMjEHa7pF/rYYutClIFtnXmb3FwrlX1F/V8Vr+wDUvhkooULAqQLfMDFzR9thJMSg/Ko+ImR5I96rAf9jWA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert __tlb_remove_table() to use struct ptdesc, which will help to move pagetable_dtor() to __tlb_remove_table(). And page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 16 +++++++++------- arch/x86/kernel/paravirt.c | 4 +++- arch/x86/mm/pgtable.c | 12 +++++++----- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 73f0786181cc9..680ec3d47915a 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -31,24 +31,26 @@ static inline void tlb_flush(struct mmu_gather *tlb) */ static inline void __tlb_remove_table(void *table) { - free_page_and_swap_cache(table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_free(ptdesc); } #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { - struct page *page; + struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); - put_page(page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); } static inline void __tlb_remove_table_one(void *table) { - struct page *page; + struct ptdesc *ptdesc; - page = table; - call_rcu(&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); } #define __tlb_remove_table_one __tlb_remove_table_one #endif /* CONFIG_PT_RECLAIM */ diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 7bdcf152778c0..dc1ab9301d2c6 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -62,7 +62,9 @@ void __init native_pv_lock_init(void) #ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { - tlb_remove_page(tlb, table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a6cd9660e29ec..f9516024cbe5d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,7 +23,9 @@ EXPORT_SYMBOL(physical_mask); static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { - tlb_remove_page(tlb, table); + struct ptdesc *ptdesc = (struct ptdesc *)table; + + tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else static inline @@ -62,7 +64,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); - paravirt_tlb_remove_table(tlb, pte); + paravirt_tlb_remove_table(tlb, page_ptdesc(pte)); } #if CONFIG_PGTABLE_LEVELS > 2 @@ -78,7 +80,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all = 1; #endif pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc)); + paravirt_tlb_remove_table(tlb, ptdesc); } #if CONFIG_PGTABLE_LEVELS > 3 @@ -88,7 +90,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, ptdesc); } #if CONFIG_PGTABLE_LEVELS > 4 @@ -98,7 +100,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); + paravirt_tlb_remove_table(tlb, ptdesc); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */ From patchwork Wed Jan 8 06:57:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 303FFE77199 for ; Wed, 8 Jan 2025 07:02:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7E176B00A8; Wed, 8 Jan 2025 02:02:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B05C76B00A9; Wed, 8 Jan 2025 02:02:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97F856B00AA; Wed, 8 Jan 2025 02:02:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 38A5A6B00A8 for ; Wed, 8 Jan 2025 02:02:25 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E917E16059C for ; Wed, 8 Jan 2025 07:02:24 +0000 (UTC) X-FDA: 82983391008.22.494CFBA Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf11.hostedemail.com (Postfix) with ESMTP id 1AAD24000E for ; Wed, 8 Jan 2025 07:02:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="j3/kd9iU"; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319743; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DRkfj+TNyCYVuLoiDWLKznOk7pFXzbzTll42IhJYX5s=; b=m53Cxup8KX+6QqvXPOF63FEHqIPiQgG2sduZpjKLGU/qB7Te0JSOaeRBZMjPey5RscdLXZ n6kf9VnYcm2IzuC4TDeo1j6bRtT41vSeDx/nFuNln4TtYyJH7wqwS+WLuAZqGesEjyc9sP 8Fsr1nzgE4QUJqmbHscDKadIiFBBIQE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="j3/kd9iU"; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319743; a=rsa-sha256; cv=none; b=C8ODbD+aiZoxaC1tM+CTQhAmiM8XBXenx2boCujQEAoHpgqR0TADZEc+6OziAXABl/voDl hQRijaPfl4lo2F8sMDy0UijjSwNdpz4YZ9ktAv/96KhezL5GWLVWzAAd+1/i81URAV8kQp y024J8SM26r03ebNSZnZUIbqkZW65OI= Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2ee50ffcf14so841090a91.0 for ; Tue, 07 Jan 2025 23:02:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319742; x=1736924542; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DRkfj+TNyCYVuLoiDWLKznOk7pFXzbzTll42IhJYX5s=; b=j3/kd9iU4IuMGNnap01jSPfZ6N/wJsKru4YY9BLSOtccTrWVNyJeFDdWw4OwfN15qd n5zr5OH3ZjQLK7ouTBmDCuX8vF20cRTBvJBiQuKJ5BAYdC3I65CtfkHfCa1YDNzrbgcv SiBipnYJxpmmsu+gds6wJ797JdePi6fkrH5KeydTf8uYxk70MuilSPX7vvtoODgi+eTf 4Qx2t4JXEveHA/r1fOIPJ7Ivd+uROnQ++/lraQisDV1/nJDIWPUgu4vfQNG/KbuAyHwb SjxH3ojqrLfmPRAxgcquRyi+RHRPdX/GwNjF6ug9NCjHAWmktNABp67v5aq6rX2SMUsI Rhjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319742; x=1736924542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DRkfj+TNyCYVuLoiDWLKznOk7pFXzbzTll42IhJYX5s=; b=T7SXPhcS3KQPHj2mUvB5XqvF8c1SziICtvoOn0KUQn7mggGaLdr5yhVtiwpQZ2sOCU GHyAAoOO+Kcrnay3XUXMtZ+E45E2GGJR1jlTwBS202vudw0nwKtoEt8iNZPbrJcjUnyP CLbbWm2X/NSSPOAwkChlJ5oQzwTlmao+V6aFCgExYWXt6Uj6n7AKDvYN/+u1NBFzz/R4 q/RP6rQaRyfIhAd4Mkb3YVn4lc9ADGPnKzehqKdpW/IKJr0DHa6GIZhjGtBSbj3H6fcE TZzvG274P3tXdpk5FB7cJ6H+odAN7s7FuKjoJVeMVIrCLKSZ/onV94Dd9vPh+giNV+C1 37YA== X-Gm-Message-State: AOJu0YxwohUbbtZTO7r1/PtqMEJkiV2tRI6wln/tKo58mO188eU1ICVE EDYqcgulPFdTDaJ9bZDsO2ukMrAL9Bmpdnp6qmSAknLx40+W+3sizr6x0pKegf4= X-Gm-Gg: ASbGncsO9Cq2+zwGW4rWmk6Cg0RUi3H81sf7686zgtakkZOcps3MBQ+zy8QNaoS0odW gICW0CHE0n/UVOv/r8ETvFWSfJQeFhao2poIQ3Kwg3WffRxNi/hrMwlrWbKLHIhmVqf1QODgvdy Ac1fstJ7MECSYrNjN3KuIgDDZ6ivSrHMx3TOi9wDJmvF/u5xVPQihdFpVMWSitT8Vy4qTAfdcSb lXMnpK+LXNqW5BYKHVbDpnb5z+iSXkl1JmFsI2ZqVYRz8ND1YpafIdU2UZmqiXKbmo2RBtORjYP 5l+d5GyIlje57j8YL29+qIzUJxA= X-Google-Smtp-Source: AGHT+IEyar1kSnzqQimnz8qjA8akoGYyLgYubsk1qekCNbk2Sp0If2dwICCDFtvO4jCrehuyEyMuwg== X-Received: by 2002:a17:90a:dfc4:b0:2ee:8cbb:de28 with SMTP id 98e67ed59e1d1-2f548602842mr3182102a91.8.1736319741885; Tue, 07 Jan 2025 23:02:21 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:21 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 12/17] x86: pgtable: move pagetable_dtor() to __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:28 +0800 Message-Id: <27b3cdc8786bebd4f748380bf82f796482718504.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 1AAD24000E X-Stat-Signature: ngy96sit6qo85f5h7e7rs9ywgwxanwbq X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736319742-805075 X-HE-Meta: U2FsdGVkX1+aEpoUmdp2MTSjVFackfKgsaPGpeR4XufcG+XWeibU419Dbv7jCM7oa4wNnW+EEx9KAognp31eYCJo4E4mjqrQJustXpfVdpUqWtqw6QQ9e5mxxM20dCSIRApZBpUzPnBbnAYnE96SFb1vfc9mHkDAj+xni7/afnCOP/I9hhtUJT2hmbTe1/Wm5KgBH3rZRdZv9DQp40DgYcyjH66pgKZbDVlgkJIE89UwSDhwhtgohvbAwLAYwWP1bmEbAX/qt7g/YcD5VaIunVwpFZz0iJUDVECHciF1HVZ13KFNwMRtO8yn1W6Rj01fAieJdNadM+G/vUiGtMQLLlfZetm/x/D8pLgGMpkJCPvO5ShTTWFJXzWib5EN9lMlMq8GKlj5uWXurhkeKRcVGhiSIfmWomuQ3FYlNZn8eHXsyDs1CJzCpQB/ZFNlvBFdxzXLaGSmB+jjJoJ1CD/w7NTcMPSfKTcmdKXuiBc4Tdua9rOKBJ7T3qSf/sAH249PxyrIj3v2uwptAszwsIZxF/5QY7rtZZK0QKZwYHM0lfPjYXxQhXuTQ5b9bc0ZfuGrAKv6V4mBE3HsO5Zr5cmW/l2aHfqTpELlS6/5j9dzleP1S8Ih0vUacRyAzIhVufgOW9h7Y8+rLNLVaqwLXasuOM9ni07uEYFcK24Et0Jyj0uKEwVdMgTBsxDeiVxLhaGJGMmZKcz423f/DgiosZMXQ7PLRiuk0ZhpCj2s7udoJ3+ybCIeHsGd4vcUJXICWK30guD/t4+qVYHurAJmH2m5vnmH0S7d6l4AIpcAvA4UCFATF92LrXkT0rnAdMUM/eB04WSnlLhVfnWUU6QGAERVENYpI7iQe94N3/BeJAydQGbGhUi6OCD4bAO9Y779t78yP/fBASaB6lFcIwwF68gvCa6h6CZJSsizwyyv6gwcp+aP9mPcixlQRiZgp30kVxfTMrKUuhLfiXbGiD8msPN F4pqbuCi 8k7/vNGjkn6CuX2SfdtZ9vY44RFKlzgrJc9+yShS9chvZ/faMotHZK+bOr01wLPW4btK8ct+2BLOJQUL7GhT+UAvFb+HT1zOpAP5dAHnrskM77zoQ96yhHP5lnha6OyBWe2crTvre3KkvVqqWiSjoZ+S4RDSWdC+wY9HldRxxUz0XhG6An62FTa0GNwCoipzK9kSvRG01mjf7zwKRpUVMc/zsXGKpVnmcRCzGz97cJxl1I6CYOuNmp3+7fiBOIFB83Q2o/qQetRtjY2dHZtI7flgL82GggMwweiXllvBI2fFir+vT1dLWZ04i+zO+O+16bt7KVQ67BTPvLWrgaEQoE8hkvL56eQkEE+XcE0hRTRInoi/J1O9V0GQAKizJOyS7ySTmPHl0uht9VkUNhiUOCQA8o00xAPVU7hhgf3fLdS5JpQosNt4rNFo7JaeOavEuvvwlfhgjAz34pfHQh66sbA9qzPGHG9Wf59A7LV2F1UDfVg5bFKQ8b7vOIzCZH+f7VoP0Q/mgAd62TECTCd2kWYT6vZ/eePXG4vNRL1vfuQ+tx57HfPgLXd+A8BWrsHtLdiT8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: x86@kernel.org --- arch/x86/include/asm/tlb.h | 1 + arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 16 ++++------------ 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 680ec3d47915a..f64730be5ad67 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -33,6 +33,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = (struct ptdesc *)table; + pagetable_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index dc1ab9301d2c6..1ccaa3397a670 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -64,6 +64,7 @@ static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) { struct ptdesc *ptdesc = (struct ptdesc *)table; + pagetable_dtor(ptdesc); tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f9516024cbe5d..c02aa0427a6a5 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -25,6 +25,7 @@ void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) { struct ptdesc *ptdesc = (struct ptdesc *)table; + pagetable_dtor(ptdesc); tlb_remove_page(tlb, ptdesc_page(ptdesc)); } #else @@ -62,7 +63,6 @@ early_param("userpte", setup_userpte); void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { - pagetable_dtor(page_ptdesc(pte)); paravirt_release_pte(page_to_pfn(pte)); paravirt_tlb_remove_table(tlb, page_ptdesc(pte)); } @@ -70,7 +70,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) #if CONFIG_PGTABLE_LEVELS > 2 void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) { - struct ptdesc *ptdesc = virt_to_ptdesc(pmd); paravirt_release_pmd(__pa(pmd) >> PAGE_SHIFT); /* * NOTE! For PAE, any changes to the top page-directory-pointer-table @@ -79,28 +78,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #ifdef CONFIG_X86_PAE tlb->need_flush_all = 1; #endif - pagetable_dtor(ptdesc); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(pmd)); } #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { - struct ptdesc *ptdesc = virt_to_ptdesc(pud); - - pagetable_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(pud)); } #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { - struct ptdesc *ptdesc = virt_to_ptdesc(p4d); - - pagetable_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, ptdesc); + paravirt_tlb_remove_table(tlb, virt_to_ptdesc(p4d)); } #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */ From patchwork Wed Jan 8 06:57:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53D9EE77188 for ; Wed, 8 Jan 2025 07:02:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E63FD6B00AA; Wed, 8 Jan 2025 02:02:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DFBA76B00AB; Wed, 8 Jan 2025 02:02:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18D16B00AC; Wed, 8 Jan 2025 02:02:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9F27A6B00AA for ; Wed, 8 Jan 2025 02:02:40 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 616BDC05F0 for ; Wed, 8 Jan 2025 07:02:40 +0000 (UTC) X-FDA: 82983391680.24.E03751F Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf09.hostedemail.com (Postfix) with ESMTP id 746E1140008 for ; Wed, 8 Jan 2025 07:02:38 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KZtHNFAs; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oFL9SGbXxYqRlw6G0JwTVWRijc5/6EOrJfB/UywRwlQ=; b=CmB5Q7bxogNcADrhy8ZtCOEZKGR35gdX3hLaaBVyc4reQ9UnZ1B7HV00r6CVlDxFvdaITM Tdv9f3qeo4b78JAtI6dwPxu89Bwl+iIetx0hzaPcr3DXaFVxQubMFAi0BJULFDoh7DmdDM KEkm/CPYomb9GfH9/SVjJFKgl8AkEZU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319758; a=rsa-sha256; cv=none; b=hpzXdzTf8L5Rc8L3m13AgzcEb1C8v34Vsp9DyY9X/tEnc65bUJUkwK9kB+eXqSNGR2PK5F /h6rgnsceXjwdTluDhdQtWWnQuZaB7WGI3W0gWbc12Cxw42U4U6Ffrivbe7mimoeZINjE7 IFNDI5jm5Y21uKtiwNzu47P6TTL2/Yc= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KZtHNFAs; spf=pass (imf09.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21636268e43so68037755ad.2 for ; Tue, 07 Jan 2025 23:02:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319757; x=1736924557; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oFL9SGbXxYqRlw6G0JwTVWRijc5/6EOrJfB/UywRwlQ=; b=KZtHNFAss0K6DS2eHTORNgb79l2bTKqi3AjExEkNTn1BwBUckYz1mAucfYigTixZ58 asLQK5uKvE29/IDrGsKHTvMv4pJdNdZ5qx6HevrFw4GxrwokFgBtKIWQeqsIkO7P3tBy pNC9vruCOV2Tb1ePggYYejUT3WwEdZcINM7KdM9Juh8d5My+dn3M9fr9PC3TkO7ydRAc vC6WXavkxveVUc+bgp9rMBeOY/opddGAwBongW3B94WSgemBkm8IWpxslMkqYzi1n2cC 8FVkzUhdK4zwz9XJxNmBfWpUckuyuLcFA+rBiYn7TpDaX7wzLRysp1VvRGy40A65iibR aMnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319757; x=1736924557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oFL9SGbXxYqRlw6G0JwTVWRijc5/6EOrJfB/UywRwlQ=; b=xVGIWglkMd2iSSacJQfI65GFfmLkVlCnvBrJARrsKgvYQT9hFSi+FUrKyy+MUcOJa7 gq5c2jjptc5ei3vDPJD+kLQbA0Y/5vTa/oWxR/LyMZvfFrMPA/y9AZIu1shPcOH2xhdD 4cJuprreltGPUpYTAM0eG0Dn6IHIsA7djKjiLuyhylLt402HWf6DGBzK7/Z7peeZfut/ JhoOW3pgUz5bPiZfukUEsb8BGIY5bDvqcq651nGu5NpmLaPzrKgBfnIDL6MvnY4LvZ2K Sz3BvcT6I5R6X4seX9V0rrpFGoyzH955EH9cB9oss2Y68PpkmPWVqvuK+Cbg7OKBzXBY Ap2Q== X-Gm-Message-State: AOJu0YytvDgrcsHrzgJyV2L9vU0QkGDMzsg+tF3acUWjqK/b/84AeSUi G93JPEEy+wWFKJDSvk/VeLbiXR7N9QHAO6OZpvZf8HiSuyYASJsr3ad/NfaqJz0= X-Gm-Gg: ASbGnctHWFkGkXak71fxUfdI94LrVtRmCLdMiFfbKTfxnpI4TKMlntLXgG+9ULueLhR LnF+WQAam5+qCQ4+lDeta+YkHuZQQxVxCOxDQgdBZD62u17LP3STKAv3hTBPIWeeluTsH8GdGrT gpeKtlNEjwQDEGB4bozOK6PMwK3ShLKy5zZJjYZCsaLFd7lkzBp1freI9UWuoWtDQWmJ993TjY+ 5jXEoLmNQyP628CYbgzzZZjxejv2Nu4+tvdDVzSwPWvKNm/ZQo1f9lgnqme4F2Vk8yrayX/m7C8 eaGvEQnTAObRWR4CxYBt1XSonp0= X-Google-Smtp-Source: AGHT+IFBkkoQabtOk36FtfqYc8ZUQnhLYoik+Qreb3ophWoDY5tLIXttH+D63mfzQSCD0xoXOJPU5A== X-Received: by 2002:a17:902:dac8:b0:215:773a:c168 with SMTP id d9443c01a7336-21a83f48cf9mr30903435ad.1.1736319757144; Tue, 07 Jan 2025 23:02:37 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:36 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 13/17] s390: pgtable: consolidate PxD and PTE TLB free paths Date: Wed, 8 Jan 2025 14:57:29 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 746E1140008 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: hp3jme9uts5qwnti7hjcakpz96sqwzbw X-HE-Tag: 1736319758-913625 X-HE-Meta: U2FsdGVkX18GZoTdIOiQxENO6rWwF7cIDf1U4315CXI0lYXt+wGVMwjGKFqcTDrd0NVoOpUIMyAUFt/qHRz9j08bGBavJEwHStucwe+qPJdvGy79CIoONnUXJcmmkvnKpnlGYA3L/sNsRsYSwltdWpejC7xne5cp9eXVQmK9Ta6A2d6SiBCRSsAc6WZ4zdjohQBOUBt1GkClOvJ9QshtE4llUCQiSonJZtckxNezag4OLchvFZj/JNEWL1kR4e+Io72jlYaAHHshKa4P30FpeTJsJA9rfwppEqYPx0HzFPH7PMF56rNXyjuZ4ioYSkyYtBSudnW6K/9ao83Ub9sPzWG5rxA1Yyk02z+EWk3V2CFsLLT4GatzePGadRageZmDmFWqt1XVAmBOKhuyA+g5qet5dC00Z6fwhwSSMzd/YlhvjbULckHwcxhSdZwfDDFIFKoAgcM8pOU0b38gbNziw1PMVlG6CIYtnX9SKINiPOGn3YPuEA1WXwnL5XJqxNq6KVGE8Jc7ZhivdpvNOkl4zh6Jaq59JJMjd1W6+e5pl5QpMjYixlQvmAzliY2+ecHF0oPkI5mDhMBllNXqCj8mCNM6OBFkZO551HlxtQAbIlJXLkoajCyopLiwtqrO4dHtzFQrsGaTF7he/IFNg/9OHrryUkRD3jGZpNZx6SVCkCu5hnnd8HBM5FC/We+OS7ld9CoW82CUsQTVgK1P+pNub0hCuPlcetNABkr2k0cM6dVcNfcQtt7axaBmeWole3G2gm9qC9kFQ2Spqz9bImyw9kXrQIQDuKSpRGiZn4l5JXGu8E/CkU/2X488CJpMaD1qap4pCeNLlKVxgAQ/hu7AH63G6YbV7PyL+096E/VgXvlLxcPsjrG76H6O7CZIJc2LLBcy4ZAbjFJ56NQtc90l4+GL29OYLOpWrFBRNeNFDTwa1Y8Iyz8oJommPROpvjrTtT+XQxEDiBtQfhcD9N/ GSJGbvNM XKaHwPGMqq3BF8cDzVWjdHUfM2o22nLHz1p74tgF4y8M7VscKhlqr1WgEIlsWULiClKGmr60H+v4HuUTxBsgyIsoUnOPMUapyUF2CD84RNPw3nVzWj5336pVmhgSb4+8k/3U1Xowzmoor10Tw14qVei3xoGsrl2RO90AtsnuqIl1m4HmNYZFdSP04wEW1xpbxDZV+KWr1sLjw5glSj5QRdL/4zZSQqezshSHmxHJw2dA7/b81YR9T3LlWdEfp55ZkQtgVvgtDjQW+Q5NwG9KwKueWbBW90OaTnOWTkLyFc+2y7OxEFiM+K9GnS3rHZgsvY6GPkP/f4RNn0eOn3ZlrL4/klLui9ZYYHhlvWr7jkuxzkKH+gy7BnpUtyn/4sb+xtv76OOOTHtu8P0DywnBBVDMonQC+H1jM4KTJ74tA+U14FIGBKdgQtG5brSA8dDh/vI8YpRAKT6tL33qHUJL0apj3mjFeJr4bAEWyq6C8uevb2n8LIFB0HRDvX6ESp1QfZGzAJhwJW64SrdE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Call pagetable_dtor() for PMD|PUD|P4D tables just before ptdesc is freed - same as it is done for PTE tables. That allows consolidating TLB free paths for all table types. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: linux-s390@vger.kernel.org Acked-by: Alexander Gordeev --- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index dde847a5be545..d5b27a2445c96 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, { if (mm_pmd_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pmd)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -122,7 +121,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, { if (mm_p4d_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(p4d)); __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; @@ -141,7 +139,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; - pagetable_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 569de24d33761..c73b89811a264 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_pte_dtor_free(struct ptdesc *ptdesc) +static void pagetable_dtor_free(struct ptdesc *ptdesc) { pagetable_dtor(ptdesc); pagetable_free(ptdesc); @@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); - struct page *page = ptdesc_page(ptdesc); - if (compound_order(page) == CRST_ALLOC_ORDER) { - /* pmd, pud, or p4d */ - pagetable_free(ptdesc); - return; - } - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - pagetable_pte_dtor_free(ptdesc); + pagetable_dtor_free(ptdesc); } void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable) From patchwork Wed Jan 8 06:57:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81AC8E77188 for ; Wed, 8 Jan 2025 07:02:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 206E66B00AC; Wed, 8 Jan 2025 02:02:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 190B76B00AD; Wed, 8 Jan 2025 02:02:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F24FD6B00AE; Wed, 8 Jan 2025 02:02:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C4EB66B00AC for ; Wed, 8 Jan 2025 02:02:56 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7D1F81C6375 for ; Wed, 8 Jan 2025 07:02:56 +0000 (UTC) X-FDA: 82983392352.02.EFC7213 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf29.hostedemail.com (Postfix) with ESMTP id 9BB0D120005 for ; Wed, 8 Jan 2025 07:02:54 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iPm1oOcP; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DKWQZ+PEOlL8JpVZj7ISn1dZhyFPgkd436O3CmTW6X0=; b=46zjkv2VkNPmgMH1x7N+T/be1iMOI/xkepwZUBgzlqdiAPgGMJpjRKdbl/JWSun89/TJUm v3uqUGdW6SXDRLPB/YYUMWHLKQHkQzBbxGAg8AwbLsQdA+evgnoGnfEYcpB794zgLRAxUD OxmyfI+cLlNw3gEeUDGyNmr/T12a8Ms= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319774; a=rsa-sha256; cv=none; b=yQdgRF1+Bs/v3LWEhKSXmr3uG1yo1b204fW0tpK1FbuEEw4aLHGF1Es+17ZHg0BaLJxk4Q v8m20XXkgVMqgv8q6jANngoJ14c87M8rI4Btm7cORMpkGwCRUTRXJDxVxdPA7+F68MSV+4 Wj8iTT2wSBukOWn39xHs4IvdBo/YkaM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=iPm1oOcP; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21628b3fe7dso235491125ad.3 for ; Tue, 07 Jan 2025 23:02:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319773; x=1736924573; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DKWQZ+PEOlL8JpVZj7ISn1dZhyFPgkd436O3CmTW6X0=; b=iPm1oOcPUUFsTsRXhia/aDGFdaOQNrecIL9AhdM3fEemFxHgjOkVvKwm2qVx0DLWde 5vS2NiIcYtPmPP38HzUmcbBwnYxmydozjlUj+C+gJUa6sz5i/zip0nLUmagjFZnmZiJF e29TsxGTckxvLNfkOQ9Q7MrJVKgkRU9tl0WNNyfQ+zNmLj+lF53xS5g9OHHqxsAILz2E FiZ/uczKDCNRNs25apvvjDgHcmeZdEM5vlbitH/QTAIGnhqyVTjZ4xsBjAurDN2W/wEH KbqMcU0AzmV+l2BI4cKOa0W9XnAC/lpZlNPlEDyu1VOnsLA9TWu+jgY6y+ncUQ4p+Zzd JW2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319773; x=1736924573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DKWQZ+PEOlL8JpVZj7ISn1dZhyFPgkd436O3CmTW6X0=; b=f0pl5Rsm0zycmHasg0mekKLni2QHLVA9NkAXScmnAU+5iU3Ogw7DPNRJ7JIm9ImP9t l8KcsAOOyvVSgnfTk9OUXJ8D/i166bSURjhNNAmvHznZ4YglNhj8yi2oXMkChVmvwHQ8 XBTIc6HAvmIWYfYF4uIXK6zpBXAV8emSMGCLBbzjpeMg4aeCTUau3fXE42SbR/6zcZto R7vqec2jqWP7rujXJTniaz6vdF4idXd07sbWuhJ5Jfymxi4mqYqMlpqLDTK+vuzvwvHr t2CtjK6ADSBWXt//rum48qEb0Bf6bICCi3ioiRBzUbykwtLpQuSBnmpepj8SaVm7Mrtb EgBw== X-Gm-Message-State: AOJu0YzbHN32y11l3iS8g3s6w6YFo+Np1KUqRs7dBlmSgID3QM6Ajwko g7dr10crwqCoUWx8z0VhQHfBPAlFezWHx8LyERX6k3rk2fLOsIon1n205U1soVM= X-Gm-Gg: ASbGncvND0wolmE8wgomsS2jpGe0PExV1T+21YKoLA+xIABRGqNX30gvGhMwFrgqknL ofG0nv96aR1r8jBz0Bo5NGeesgPZosOQSoJ3rUHwXUzw+kztglbzQj1bNZY1Fa0qyqH0+DyLcIX VM8Zq3ib2iaesrMg1YxK1G/0tbO0J94lYZlkNMjtJ1okt4qu947Zwhk8KGp1/vHP0qZvZnU/hXE bWI2kSpwnveqUoJzVLhvvJrsUn6SCPzsmymJ5by3khHOMtWNtvGRbtYrCC+a4HfOduSUAVcLOeV BTJEbIUKIfRi3qE1lfn1eRyQBBg= X-Google-Smtp-Source: AGHT+IHGzG078/xZKjf5ni4u4VqQSEqesUDiW4uhWhxryubYr9zfzyAKAXvb8eE6PvXFj9EqVM5aSQ== X-Received: by 2002:a17:903:244a:b0:211:fcad:d6ea with SMTP id d9443c01a7336-21a83fcf7a9mr28371525ad.45.1736319773330; Tue, 07 Jan 2025 23:02:53 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:02:52 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 14/17] mm: pgtable: introduce generic __tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:30 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 9BB0D120005 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: ezibb9wqc6utp9bbqnibun5f3k1ix4i4 X-HE-Tag: 1736319774-705189 X-HE-Meta: U2FsdGVkX19rVotVmL+KNF637Ve/Dc5oO+LKqpBbNHd1SQbXXxCVR1Rj62HGtyFO4z5Dku3MwB6xDpwbZegEEB0djGIHuKSryR4/Ig+z8IzKYtM1onQTJn1upRskyXqZPjZ2FX9QWxErBqSEKEo97iJSg79W9YNFHObMjKJsfRPXY+1Fx37ElLHLYK2c8KO465KSv4jbywgjv9jGZNizbLWr4+jgSY7DF1VGD22Glq8nSAgIsk8UPqkddMmxrk2Q0KuxbROW1m09JDszz9eh+/LVjmVL24tWq0WPBnAyg7CIOQJYNPIieYlZVjfNX7VHl6UjfZSkhb1vKUBwABbpLZoFEkI6ZqerLdMsL4aSm8s5c3f+rvjzPU+/eJhNg8RUWP4TuTAQKDsQccIKwUTSRx2fwP1bHZqDcHO13jzsC+4UjARaq+0sHm9sehr9wBUsxmDJeB0MCCpt7vwiNVDHXYFcQ3Fb1FRfqiJ/cU5UQRL4CBOX93vQ4kk2kIHVvFtUyKUilh9gR5rCAWI43hkjB5Q6paLQkGTINUSOlCleuXl/DzV2hhXkQbtSNhaJzC8rKrKaWpt0O680sdbgbR8cUpXoktfuMiOogo1ZjE4rF3Ukjr3dxwDsIowq/4aPPrSuB7x+B2IcnVVy6zrLxDKnbWaPeSh7A0tElWeC8VFl5Vj7uhN4Zn6NBUwGD/1uhGBSTvBc+GBdqrok0ZwTxipyBXTdf2PAVZ/hBFNxWi7DveBeDKEa4RmKc69vdGkMVhI0T/tP6ilENUbJteI64M5MdkWwEqHFLYOyDpi8cH7SWV4kj1goFgD95/j0rnZJBnMumeYsTu4GTNw2cOPxixxi5NbPhA+0+EpjenLzoeq3gRM+0EVZDi4+m2D+ZsAe1bVgtXp637NemgdfMpr+uHe7tH+3afd+x2pgdGlZxyMn06HavbRfaFcdPeznzDr5go1GFPLXXUJNrM5CHBFrDC1 g0j9TilI 088lVeAua3KatiFRd9nrOomJzX+VfXSU4iC09U1+JKPsc4Q2P4RAZgLFf/61u7W0sF7/hzOMH2Dcedi1CEki9RwebwkkCQmc3YmpsRTRcDywA/jj7EAkIIpDqcBywvBNg36PFJPFArOD/3qx2irxiESv6y4kscs+2yCG+M2dgRv6sPf0x1tQzMbsVmwsNzJUi/1d3EOOLGG2mRIB2mWpKyUpoKZfeBckjcMe5rs9H5LaRB73latttJnzWpk/A9omJLFkod8C9OCjNDwvvxjHkNRNuNogIVgvE3+31Bj6fpJ9xZFfE1i5KpwyAa7TDwNsYpCDHTANG3M4G1evLSXmx5j7MQgrFeWhoGki+i/BKfLtfvUpT29tWiZncosp/q564CWFkZugqIBKXmgWC9BubjrPdH45UZohHjUwdUJUxxWHOvaqhdYizPDTeYPhyIGzstzRSCmPRms15K0RTgsN4N8Z7EjSd+Pa6LCHx9VAjVtGJ7y3qH+16Uf68zqc+4EbgKnFIiInAaDr+8kZJLrfW4z/00ScDM22wi877yTLg1jGji0/eJKPGGqBl0GZo/LRRMBZg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications. The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky Acked-by: Andreas Larsson # sparc Acked-by: Alexander Gordeev # s390 Acked-by: Arnd Bergmann # asm-generic --- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 9 files changed, 19 insertions(+), 59 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index 59854c6b97bc5..b8eebdb598631 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -27,15 +27,6 @@ #else /* !CONFIG_MMU */ #include - -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #include static inline void diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 93591a80b5bfb..8d762607285cc 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -10,13 +10,6 @@ #include -static inline void __tlb_remove_table(void *_table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)_table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} #define tlb_flush tlb_flush static void tlb_flush(struct mmu_gather *tlb); diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 1ca7d4c4b90db..2058e8d3e0138 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb); */ #define tlb_needs_table_invalidate() radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE /* Get the generic bits... */ #include diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index ded8724b3c4f7..50b63b5c15bd8 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,18 +10,6 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - -#endif /* CONFIG_MMU */ - #define tlb_flush tlb_flush #include diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index d5b27a2445c96..f39f8c4723f15 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -22,7 +22,6 @@ * Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); @@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds = 1; if (mm_alloc_pgste(tlb->mm)) gmap_unlink(tlb->mm, (unsigned long *)pte, address); - tlb_remove_ptdesc(tlb, pte); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pte)); } /* @@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_puds = 1; - tlb_remove_ptdesc(tlb, pmd); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pmd)); } /* @@ -124,7 +123,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, __tlb_adjust_range(tlb, address, PAGE_SIZE); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } /* @@ -142,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; - tlb_remove_ptdesc(tlb, pud); + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pud)); } diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index c73b89811a264..3e002dea6278f 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) pagetable_dtor_free(ptdesc); } -void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = virt_to_ptdesc(table); - - pagetable_dtor_free(ptdesc); -} - #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now(struct rcu_head *head) { diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index 3037187482db7..1a6e694418e39 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -33,6 +33,7 @@ void flush_tlb_pending(void); #define tlb_needs_table_invalidate() (false) #endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE #include #endif /* _SPARC64_TLB_H */ diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index f64730be5ad67..3858dbf75880e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* - * While x86 architecture in general requires an IPI to perform TLB - * shootdown, enablement code for several hypervisors overrides - * .flush_tlb_others hook in pv_mmu_ops and implements it by issuing - * a hypercall. To keep software pagetable walkers safe in this case we - * switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment - * below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h - * for more details. - */ -static inline void __tlb_remove_table(void *table) -{ - struct ptdesc *ptdesc = (struct ptdesc *)table; - - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - #ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 709830274b756..69de47c7ef3c5 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -153,8 +153,9 @@ * * Useful if your architecture has non-page page directories. * - * When used, an architecture is expected to provide __tlb_remove_table() - * which does the actual freeing of these pages. + * When used, an architecture is expected to provide __tlb_remove_table() or + * use the generic __tlb_remove_table(), which does the actual freeing of these + * pages. * * MMU_GATHER_RCU_TABLE_FREE * @@ -207,6 +208,16 @@ struct mmu_table_batch { #define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table(void *table) +{ + struct ptdesc *ptdesc = (struct ptdesc *)table; + + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} +#endif + extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ From patchwork Wed Jan 8 06:57:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4B57E77199 for ; Wed, 8 Jan 2025 07:03:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3DBC46B00AE; Wed, 8 Jan 2025 02:03:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3662F6B00AF; Wed, 8 Jan 2025 02:03:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DF036B00B0; Wed, 8 Jan 2025 02:03:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EC3016B00AE for ; Wed, 8 Jan 2025 02:03:12 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ACFC3120A7F for ; Wed, 8 Jan 2025 07:03:12 +0000 (UTC) X-FDA: 82983393024.14.C7FE02A Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf26.hostedemail.com (Postfix) with ESMTP id DA72714000A for ; Wed, 8 Jan 2025 07:03:10 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZSy+53s0; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319790; a=rsa-sha256; cv=none; b=t9HsTtQjuFisds1JtSbZSM5gcjnP95s8pZjz5gqfY2hwpQEBXlBXmRy0GoxRH0S71iMhiw imso5vDCF8Ebfv76c5635QHWxHW8fws6vw+Ejy1+2dHcWbvNUOOrKpfHtIdJ8EhhDNMXLa Hnt95naY8t2EP3F25u0J/AfeHIXecXo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZSy+53s0; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319790; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SzpcYcAmR7wIjJMr9qjJ0ueu9TAADg8s+gPSbWL3su4=; b=Iw2SqCG/r05ByrwKP1XCk9PZQLKPLMQ7xlVaTJFGgvMtq0X+LLXEb3I9FSCIoGrBB5iImq yoBqUuvxMqiuBM03pIx2GJ/skjY2XCxXVdGRb6gnn0rp/y80OuJ2ALXn9gFNbk9r/L0JRc DXTBnyH8BgD1PYd26lJh8Ho6YFiHUdw= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21634338cfdso29994685ad.2 for ; Tue, 07 Jan 2025 23:03:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319790; x=1736924590; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SzpcYcAmR7wIjJMr9qjJ0ueu9TAADg8s+gPSbWL3su4=; b=ZSy+53s0uxkUwFVHqblr491ASJ2hK4rq7JHQ8hRwkfY7R4expld1NoP96JcuMF69JQ QltYCD1ZTK3yjB4H6BWJQUQMZgyDXJYkJQq7GmyS+TcQiDjYS3hQd52lZp81j6XPe+KW OMNG5lguvz4WWRTwLZJJCoKNKAOEBWbed6PR8MvbX5hy6DI1a8QxEuAxYN1/cs3bjObd 2NSFNfy/9tuB2ztJ1HLlhq1xqKPB1GfVfvyxJ32F8B3++o4yxi/5NL1aws5vkjm38FBc 170rtCfoWBbvDPSG9fGsD1g9RB0h/mCrfFheN+D+28eWOFhevnu6B8kobcx41JZuC6oK dplA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319790; x=1736924590; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SzpcYcAmR7wIjJMr9qjJ0ueu9TAADg8s+gPSbWL3su4=; b=T25enMEcnGcV9dQi7sBd2w/L02dGV1n97Eeh3xnxZgmT+zCiNRmCe1LhmN2dLnNiBM UjzBA5C7yJ3wCeLvZehwsPrwxLsLiX0tIO23Iw7S/aXz+8hUic2QivX4zD4ky12jtGs8 sBUJDFjwYUTleJPUObxr74r6nlAQ0maT6v/7icQ2lzJX5i9lBDFzh0dJK2RtoxzV5sZD flmDYwS08Ac9gAI0ylOuQxiOirttM3w37UsAV2VpLb22QBVLP1LwXfTBvsvxmmj+SXV2 PTAw32njXvdDmd3w96fYBcrU95Ylv9lbXBsFCd6HJbcAyZ3sdaj+TB86RbGiKtoXU2jO jSUw== X-Gm-Message-State: AOJu0YyeDwuwyvBTFuY+xjHMXFQ7346OXyosWnUqA5/T2jD9RQRNEq26 ifgCqWPo2/47jETD5VtVL9rSlwaPUd9DW3OD4NMRjRzEOyiCy22MzGNXtKZ5msI= X-Gm-Gg: ASbGncuIu45nnbvIr10nwOiBrDdh9Ux+XgnmzZgRBfED2bDhD9HWWNx/fb41efnloKs Y258fUhQkMyO7eRODRe2UY7iXAcWYYfa1as8BJpHX7pzEHhob1TlrOnT6tFIZdgxbq9F9C2xBgx zh6DgE1Rz4YNWKVuUkmXRgNfRtbYIMxujzpNkRzUFW9nvaiH+N75VFGO/VUSUt5CH0lEhpSa2vq ZzbJn8jLGbZLgV29qt3acPgM/W0zRb8QwZSzmapPzONAiihINN2VyicZMY3dIvtY6KBREnSIhVp FowyQc2EMrCUVkAM6sF7v0hrdHo= X-Google-Smtp-Source: AGHT+IGLH2wnugoAqXpKOjRMOcNm05e1d3DjACkQOqMsLyNNJS4LCZNqibWg6/9XtyaXTuj3byMh6A== X-Received: by 2002:a17:902:e74f:b0:216:7cde:523 with SMTP id d9443c01a7336-21a83f6710dmr31869555ad.32.1736319789694; Tue, 07 Jan 2025 23:03:09 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.02.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:09 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 15/17] mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table() Date: Wed, 8 Jan 2025 14:57:31 +0800 Message-Id: <0c733ac867b287ec08190676496d1decebf49da2.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DA72714000A X-Stat-Signature: owpqr8r7ifuktyzqth9k11grgnes8uwt X-Rspam-User: X-HE-Tag: 1736319790-696590 X-HE-Meta: U2FsdGVkX188NQpnnUEUN8h4cCXdFUceSxLq4Whgz32jVPNJurrzBuTylqNKEWpTZYdLh3XnGF6FjdyBvlEAI1cMcj4i0iSltda9EtYwKk+Pbw/g8MJFCqHbbhn22t0KaFJ5E3HHio+Tej0baghC0HolNQH6XpanhpHAelX9EM+WzK2EFeWvqhnv6uGb1KV2FFlbJrJE28y9xFriEeG9F+HNSuis2lYGM3jLg1eLJh9NdplzmuST7Ho8WUtkcxerIOU2WIy/+QXiiWPhA4DYRaAzPtBQcdwOi8gf4++x4i5oBgd8Q2B4Em4jI5g1gjnP+3qgWzxU8QGuu7qUoM3sv/rvZyT+J7y6a395WgjhzrxmDUSw3Sw8vrZFM9IX7GreZNMZeT+fCQzw0KGxj4o+N/PeW0j327mJ+GKCJilWM+J+9xfzdt31XUjL4wv7CQobl8MSnD76r6/4/xXkyzuy7Tb6DMUuscWWan5tJupeAUC+L56xLr8VbaiE5pkBysADDkSWD6tcNPcUSlpivJX+YVaX4SI0+5ty+XKMvfZPAWKm3FUiF4o7yuRVFP8mlaLwOQAwJpUK34wTZ0tMU5C8KAIghKZbEZgHmyYHdQ0KZdHi11mLfZSaod3wwHtk6W/ge/yEdfo4PzrD7K3cVDOJqi14872WQ9Db8igFEAjEwC67GrgOVtebLl1xOLMJzElEzsTXUZYj+/jHqNSAD27EA0894QCxIilUTHsJ0S2H3iHt+CEmE6AZBT/dfVB9vs9V2RAyP/hRkHuTNeBz0SOamYPlMUS8RE8Q8Ov+GaixP3FyLwvyDvwZyl9AiSho+9kKfqY8pFu2jEe/20YyWdGCwkZctfxPjOtoyvZ+sHOKmqrvmtkoE6Npq5jklM5IR/f879kOGt8BozFGMbnLCf5j1ZjAVaPukCEoZrrN6VRiKuR/XSpo40T3ZAkqDrJ0UQa3UEh6OQmVuq0cqJzlIIl 69+L6f9Z QpVJQuZmZ5PxSIAVtn/erV3sSLxG4XzMDMIYSb8h3TbDzsnosI+rOiBEypUm5h6xDqh07/oz18y7cMjeupoUEKu8G8LbJSAceWQLdFRKDA+isYl7SYHwtV6ss5XFfCql5rpqA+kwcOulW7F93yziIwdINOXzkNHTaO2c7h1JEYybEGtOLdv7IRfEhOnsHj+Kxwr/jkASk3RgHdvfjSCHnVCmTmfqqs15vLdBDGjvy8lqdyN5zFlnSdpKA5VveWGqHp9THon4fCcEJtESYSuAdoh1eZUk9pP8y7poGiTW202oAI+13k4KI6sMGrrGnhc5KwtBQbTqht5WpKC7hOzn2VOHV/Q1lQZHLqSkjfEZ6VEFpNjT86COE/ncWcO3iRhP/Og1zpqbd8dHqKU1NMHZv27zgb+206YH3QamCZ1ATWS+C7JyiOIsbRsXJiXdvzmIeFBsEBrB/n7ptp5zCVCPAa98+Mv4Ur5+zkMNFy7opae0/9qw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For the generic tlb_remove_table(), it is implemented in the following two forms: 1) CONFIG_MMU_GATHER_TABLE_FREE is enabled tlb_remove_table --> generic __tlb_remove_table() 2) CONFIG_MMU_GATHER_TABLE_FREE is disabled tlb_remove_table --> tlb_remove_page For case 1), the pagetable_dtor() has already been moved to generic __tlb_remove_table(). For case 2), now only arm will call tlb_remove_table()/tlb_remove_ptdesc() when CONFIG_MMU_GATHER_TABLE_FREE is disabled. Let's move pagetable_dtor() completely to generic tlb_remove_table(), so that the architectures can follow more easily. Signed-off-by: Qi Zheng Suggested-by: Kevin Brodsky Reviewed-by: Kevin Brodsky --- arch/arm/include/asm/tlb.h | 4 ---- include/asm-generic/tlb.h | 10 ++++++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h index b8eebdb598631..ea4fbe7b17f6f 100644 --- a/arch/arm/include/asm/tlb.h +++ b/arch/arm/include/asm/tlb.h @@ -34,10 +34,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { struct ptdesc *ptdesc = page_ptdesc(pte); -#ifndef CONFIG_MMU_GATHER_TABLE_FREE - pagetable_dtor(ptdesc); -#endif - #ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 69de47c7ef3c5..53ae7748f555b 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -220,14 +220,20 @@ static inline void __tlb_remove_table(void *table) extern void tlb_remove_table(struct mmu_gather *tlb, void *table); -#else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ +#else /* !CONFIG_MMU_GATHER_TABLE_FREE */ +static inline void tlb_remove_page(struct mmu_gather *tlb, struct page *page); /* * Without MMU_GATHER_TABLE_FREE the architecture is assumed to have page based * page directories and we can use the normal page batching to free them. */ -#define tlb_remove_table(tlb, page) tlb_remove_page((tlb), (page)) +static inline void tlb_remove_table(struct mmu_gather *tlb, void *table) +{ + struct page *page = (struct page *)table; + pagetable_dtor(page_ptdesc(page)); + tlb_remove_page(tlb, page); +} #endif /* CONFIG_MMU_GATHER_TABLE_FREE */ #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE From patchwork Wed Jan 8 06:57:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87066E77188 for ; Wed, 8 Jan 2025 07:03:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B9726B00B0; Wed, 8 Jan 2025 02:03:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 142F76B00B1; Wed, 8 Jan 2025 02:03:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED7326B00B2; Wed, 8 Jan 2025 02:03:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CCAC26B00B0 for ; Wed, 8 Jan 2025 02:03:27 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 797EF160953 for ; Wed, 8 Jan 2025 07:03:27 +0000 (UTC) X-FDA: 82983393654.05.D6853D2 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf26.hostedemail.com (Postfix) with ESMTP id A383F14000E for ; Wed, 8 Jan 2025 07:03:25 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QdKucnse; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319805; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x40aCfPUIT+7f+SOGlgpQ1mZ5+3GERu3xGzgSvQpPds=; b=1eRr9z/N8ZDHQfyEqamExaZz4rHWFl7w6qjQAlQgOdxDKMXTBMMtKeDq61WXEIOyFnoBFt T6d2NdGfaomHLgh+9+rkfgCBfrMvVi+JzxNBbQrEhvHnWyXWwLl+Nn5m/CSXjA3K/camNV VSvodnXOPSNfEpZiCtqKSXMazV9rPDg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319805; a=rsa-sha256; cv=none; b=X84zHJ2zwdPpJ4+z0WH3NqMkEPVdYLr2ZVKTaU00UHdHpN1oHN9PTPOYl3Y4Cgz9apdaPb i4A+g209jkIDq5uxaWAJekuxPDH9oPKgUVKz874NxahD6oPbXSOhPwE9ZXnhdOj0mQ/UXR u8Pb04OvtLWHg+lfpmtZQQTu1UhMUpI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=QdKucnse; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21628b3fe7dso235497315ad.3 for ; Tue, 07 Jan 2025 23:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319804; x=1736924604; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x40aCfPUIT+7f+SOGlgpQ1mZ5+3GERu3xGzgSvQpPds=; b=QdKucnselaOe4Q7uU6l0wWKJw2p2Wyst1Y3yiEvpF8VSmI0sWDWJSW9l3J1r1SKI49 caArw6SArPiZpTI9PJC72T6ZL4Q8WaUhBiNighwzAiVmD8MnVXzxBeMTo8Hjmga6RBHG ffSxGpTVLtOa7VbQiwFN2LwmZRwWzEX9eMI0PxLwsGTxCikOMXGnVPEpzjlgngU1tryp sr/m5RWNSbMCXwk73K3Xh+4nRR/V583muOXTUzdjvab1G0jEgEB8Ugcg8bQSQkP1XV1q FNIcYjRxsEtVao8oH3+brgQkEVZFd36sFbZNbNTSSBK4sv1yVJeawD9Cz2AA+266YO40 w9Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319804; x=1736924604; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x40aCfPUIT+7f+SOGlgpQ1mZ5+3GERu3xGzgSvQpPds=; b=bWRNhySCCr8iOSFEhfw9mgpomQDf2/W1za4gsX31CRCDDOoy3IN7c+lUXbnZhIvgnd vT3d0wSj5W8tqvM9XWd67XfRuucvErmL1sA6arRiUttIAyfj8w6W/EFirYuGUi82lS3B 0FEs5yriOOe11EmeyIQctcbpSPeok1ZhgAFlc6tBOOADMWAlG7+AYkHpWpGCiPrUf0BP vcEVCLY0VaNO2jLHwt5qfbBXNUamjU4/O+dd9FVxD0CAhZQ4teUNJPgp7bEQpP2zcQZX 3kGdEkNz5G5hKwbWky6Q412mJD9trfxdtiU/OTjd5qpj5DFlnANdGhBKrX7y9cm/5S05 WkgQ== X-Gm-Message-State: AOJu0YxCJFkeGhA6wPRO+5/b40dpvwuriO3pqUISarm33Div4EKkNelL f6orbMaebKRe3aQUcrVtONPLNtHrHhG3+cQZ8Oy2ppvMz6k+7rRwMtUtf3Wr7uQ= X-Gm-Gg: ASbGnctf4RHq8LV0NvOmoUiHFQpz0T/yJN7UtukDV3jjCyAr2kAuNwviH0jJWt4o7Bf RCjE5E3Q9muw+NRzZmHa1abjAd+YCSWoPh4ROvHlN2BJe/lPxtEim+03jSoLDUCOsSTx1tSiG0G gYFbqhXhehA1iwYgEXD7zMgi/MomblB9i8r6tsbf0d8ftcJRkh/Dh249VsdILdj2Xt0gfXnzYrX l2KLVXieOjjk7RwR9KpQDsOLW8X40h4o26IXMzqSWz6eSpQoKg4P/vmprSU9eGEV8YS+Dr2MihK GnfF7ZCSwah8ub9jdnbKxLGCcRI= X-Google-Smtp-Source: AGHT+IH8IKObAIjKqBGW3XQDBdiOawtDgYf1Tuj1urKOF9Kp4Be/TSELj5l2Ma8a+iBaK6i0ld/PMQ== X-Received: by 2002:a17:902:da8e:b0:216:410d:4c67 with SMTP id d9443c01a7336-21a83fb1395mr28350885ad.41.1736319804422; Tue, 07 Jan 2025 23:03:24 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.03.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:23 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 16/17] mm: pgtable: move __tlb_remove_table_one() in x86 to generic file Date: Wed, 8 Jan 2025 14:57:32 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A383F14000E X-Stat-Signature: 55d37kichpi8tgpwzkhgj4odrzbojhg8 X-Rspam-User: X-HE-Tag: 1736319805-906218 X-HE-Meta: U2FsdGVkX18S+hLHIj7Lo+HThrDkQNN8U0sr2AfbcoCfBifsIN+vS0SggTc4ShionuarGGl5hYoVZF2lljp3QQudvtcV4VL4ic0sycpp8BqBKBUc+3f3HCwm8Y5Jv8TF2gjgDdKoJzM5PG6ZNe+UOiTQlELwdCB6IMkbqg+aOQW5i3ksLXrry514g0kUXKG8PuoXZpzlwCZ669XSSOHer+X+hYKunyp4gwes9TOMQNj1YCYdybAekfFJbWiWOkgPQPAwmtR+54ZBTCKgpSew8CUTwbca4unHI5vHaqnuBRnSWjRyV6myk+DPz6vLhFCKkJgL/3hFgzBg2DM8VSVp0QWR0hsDerrqAx/YhucfM11enheatwRvyROV+oemv2srOxFDZygrS6jL2uIqLiGdZuVbzn7RBGgXupw9tSE7Uzn8n2RNRPUgPTu/aPpLwt7oaexE9ek5yNFqhAD69QT+FU4ReFlwii0lsDJJYby38G2Y1ivvb8L5bOvh2gxUcBWp1tZiqq3DCC9D93GUMO6pcfKfZLGtt2lNtWsdngqZCLKf2P8TJFz3lm5si0LHFS0HN1MsuxJZNLaceM+ZPZJT/BTWC1h9n6U/J1/LqzxzH0mlKKjmokOqA+HbVM8fsABMuWGPixI9fufvc8aJ5aAhE7RFmQWsxBwgmNSbemzPdMpaw067bsdSZgBEqGs9ben9ARDfhRglLl8XTLGO/eIokn9h3q75MC5U1DKjuTJSQ3OFrrxXL4Dsvd4nt9r5AMtltUYxMlFFhLEK7QjKbncAFYk2sau0MTcrcs5RriFdkaV7YX6xZO6TXfu6p6v7Vph3jlLGjK6H+5MyM64hFG5qhBayyDFVg/XW33GdF4titPtXsgOTIVdKLr3JGBpd0w00AR3SSURVwPdlge0n+2+ctD9BKCytsnfotKBGqduFBUBuxvX2SstsZa90Icea7WXM1Z/HsqtF0H4z6q0wWHs JlBhrUmg 2YDxvP3ubk5hibWcMxi+sROsJrZL6pFhIddbAjPwA8hnze1HMenkJ+gbm1FGB65cU6Wecmeq9D0m4Lrhjz3ggp40OjCJKw0w9Ttci+vYM9ZIlykpVnYqTWemxaYRkRpSLLuYKO+Iqqvoy/8Vn8ZUGLXR9RanTM8DIVmdcz4W/4WW5AfImqMU4R08clvaqru/gvv540LlrDlzEPMth1jUR6aap3Q/ChvMWyyKfAAoTwD9bvNrA3Vj3k5Ku7qkHNCgdPnMj8eY/s7ihyiHhuBCpvW0b2NwqeRf3vPlzN0Vj625yKHRLx3nE5gfeyvqJgtDRyH8b2A8j7tucpFkYcchrojJauz8qL7iWKmchVx5V61UfPKk6mlh1HnCW0RoxgmE/0XfsIqHwojwdApQ8FXimufYb4XkgJ1Dy/YzaV4q29T8acGH9SRcS9twFyAQak4j5Zk8aOpouik25DlXod6uo2QhL28sPmkyzEMgU7qEQEwAdOsA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file. Signed-off-by: Qi Zheng Reviewed-by: Kevin Brodsky --- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 3858dbf75880e..77f52bc1578a7 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) -{ - struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); - __tlb_remove_table(ptdesc); -} - -static inline void __tlb_remove_table_one(void *table) -{ - struct ptdesc *ptdesc; - - ptdesc = table; - call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ - static inline void invlpg(unsigned long addr) { asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 1e21022bcf339..7aa6f18c500b2 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu(struct rcu_head *head) +{ + struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); + __tlb_remove_table(ptdesc); +} + +static inline void __tlb_remove_table_one(void *table) +{ + struct ptdesc *ptdesc; + + ptdesc = table; + call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one(void *table) { tlb_remove_table_sync_one(); __tlb_remove_table(table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one(void *table) { From patchwork Wed Jan 8 06:57:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13930208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 503E8E77188 for ; Wed, 8 Jan 2025 07:03:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEE1E6B00B2; Wed, 8 Jan 2025 02:03:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D77386B00B3; Wed, 8 Jan 2025 02:03:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC9FB6B00B4; Wed, 8 Jan 2025 02:03:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9490E6B00B2 for ; Wed, 8 Jan 2025 02:03:43 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6242F1C7E5F for ; Wed, 8 Jan 2025 07:03:43 +0000 (UTC) X-FDA: 82983394326.06.4292F94 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 845ADC000A for ; Wed, 8 Jan 2025 07:03:41 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jmKIbT4L; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736319821; a=rsa-sha256; cv=none; b=WvdH5T/5zBnY7VnpSTRsBRtntAUxi7OvwCX40dPUgKeRltBUgir6G5C96NPbEoLsGl2F0x hrjMynT6SGuH7QKqiMeiZyoQ/NR0HO5I6sNEUNK2Y2saMvWrabBRR9HsuvVFyXU9ViDEfW 96gsb+bUyJvD4RLoleyGK0cRTXNP7jY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jmKIbT4L; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736319821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SmV2C/123ilQKoYc6h41pKDzG2aiaIfZ7nQkaBcWxp0=; b=Zfo2y9p24VC26TiqkRYO4Nb/YRt6uJl8c1OPDkkef/qV6bUEt1JcYjvo7dfN64TW5DIoFF TAZRf41Rhqczxk1pRq33W/UnZRu2wJ6oJmbq4qc6FALBgiBN7nJxF2QZ/Vm3b7ukyDqJ0R 0XxEYiVudEEXCF/87CnW3QzKbs9H0Ds= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2167141dfa1so9535535ad.1 for ; Tue, 07 Jan 2025 23:03:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1736319820; x=1736924620; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SmV2C/123ilQKoYc6h41pKDzG2aiaIfZ7nQkaBcWxp0=; b=jmKIbT4LT4Rpz2ffRDBRSXZV1+SZbGu/RG8jpBiiP5iap9z7loXBI0DMv6MoT4NeR4 roTP422aq5jRwsJURqifTzxLIFdMUh5gQoFhnzIed2ZBN1FVDH7EHS3W91GoE+DZPrhv p5cPGX1foFWOvkGzdIR3lBZfcNsrd3Jle9lDjW8OrzMboMHmNWlC7h/JMznQ0it+s0sr sGqr3tUPJjjPJImD1BysKTFEP3QLAMydhItoz3AHLrkrFxD72x+MiqW39NbLDe+F00XU lqeeEpGHFTV+wBuROev+Dx+j/5Hw3zuWbkOjxtDu8Lp5lBFZcaZ4DhaYbkiD8bMDKT4D CmBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736319820; x=1736924620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SmV2C/123ilQKoYc6h41pKDzG2aiaIfZ7nQkaBcWxp0=; b=hR/ikIaxUDv+kksdBAzzEEnZjgHulM+h9CEksT/zzZkaq7wfIJiMyflYSYqC7hodez E4kZQtErG084ew2SF7sDF4wzUKzYHjF8m7DkoJNad/+/30UU+ynX/LFVCsQ9QESVC4wZ QDQlDENCwTVNwl1mXIqETKl12kAdnQh2aF9MhIeEAiRzJ0Cfwf/a4teU1aX/t1sWiLzi /3ukDzRYegi69IoSVPSrXDgIrIF1rVDXA8+MmjP9AibKQuIlanpyytZ6cDSKJNAqsAdm Wq3I3qxQDNkaerzvdcRW+yy8UNQ3Ee3+Ug7DrRRHFB4pEewbi7JPOdShA097zEdAISZD GMCg== X-Gm-Message-State: AOJu0Yzz6IYwYo+qudEInDv1vt5ukXVwPrFkWVVML48LMATI4hBuFagl RPw82wJ+tCrrkj7MAmxmi88XT1mAQMchPD4lLWT2Sd+9FW6oHUO4O5aLdKrHT0w= X-Gm-Gg: ASbGncsMBpkgbAzn/DKlbHOBVDs7gIZLKEOx4E5u228Kdb45CSBqgpaRNYwchXvxQC7 1aFvSj0NDiRe9piVUE9uu23TWZrgnwgaMh6nUYtudbswGJ+xP8RcszO7BsyIRCRb/yPCd+odwkh W1G9GolTim2OT6cGshvpbbvnQyWLtv+fye6Cf+ji3y1qZ5vT/3Hg7B/JBj4fs+KkyRFjSrZo0D1 77lpqGXlx6hVE8KyDj+mCfK84PkoITSEcSLTy+OXJZt0tRTEPdy91iFQBSUO83sZewc2dMFMxXf Ohq9xlXJlPSDXG3iV06xn1hXV/w= X-Google-Smtp-Source: AGHT+IFPhscn4ro6sBJsBOR2a8pRlJGCRsiSDouRo+43OGNlwIzbjZLHQRkjQ8iB7a2FmZbbQjyk4w== X-Received: by 2002:a17:902:fc46:b0:216:4fad:35d0 with SMTP id d9443c01a7336-21a7a1d4731mr100818035ad.9.1736319820276; Tue, 07 Jan 2025 23:03:40 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dca023a3sm320067275ad.250.2025.01.07.23.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jan 2025 23:03:39 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, agordeev@linux.ibm.com, kevin.brodsky@arm.com, alex@ghiti.fr, andreas@gaisler.com, palmer@dabbelt.com, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com, arnd@arndb.de, will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, dave.hansen@linux.intel.com, rppt@kernel.org, ryan.roberts@arm.com Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-sh@vger.kernel.org, linux-um@lists.infradead.org, Qi Zheng Subject: [PATCH v5 17/17] mm: pgtable: introduce generic pagetable_dtor_free() Date: Wed, 8 Jan 2025 14:57:33 +0800 Message-Id: <1663a0565aca881d1338ceb7d1db4aa9c333abd6.1736317725.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 845ADC000A X-Stat-Signature: cnstqhhsnikjwn6a3pd7khgej81dcupj X-HE-Tag: 1736319821-732799 X-HE-Meta: U2FsdGVkX19xrbgMQjmv4QB0e9BDMkoi7t7G2ygwg3rYuKnrwow7T5G4xIQtoyAmZftJRfjRYqUPGcKzan4gtAYs4xSDwUytz5694agS8GU9r2RPKf+DIl10LN6TrmJW9y4l2yZEpC3AItEEct/Rl0QQ78n2yhvu0rI878IN2pcSb6qP7qCuuxM06CMJERRkuo4uVJmgEAwGsAJrEWVB+8U3Je/N+WOVLhw//5bVMzvH9qf9pjmuwSzC8Jhh/eLQudgTDRVbjMeBIWVVfLNp20cdQsKmphmD0DbxswYQRF6f2od1oiLVRZtLRw5sOz+brk/2SZCvIKs8M/w9KPQkYgvE4y3em5Sny+8npcDmbLoJN5BxXIZgGRgaGfimMB1XiNJw0XfN4p9Fqkd7bfCENMbBsVBtkDc1fXBLX1twXDdAs8VKP7nMwx2+MEHDfNE8rNojPVa2Qt7XzBrPsPDRL9NNDMzO+FEFBC5abuxEVli2arLpQ0wnFaFvwnizQ9VG+PTmocBZ1UHJKy15TZPrJWwWL5zmXOpkxMOCb2UG/kc1LWjoMRfgG3q4Eu7XdS5a7e8IJGN+usTXyvkRq5pkqaiiB7B+ITu1DknYDOlJ2OF0vKZkakC9a9mwnRd7t4uTd1QRUL9pKqGFA2GCUtKlJllV8SB55cYWkIp65J9k3GyzXfS6OLjLOPwK/75Dw7GNGVIOiAEWyLbGYoFaB4Brn3KGaWpnQNWG1a6+cN6adkGFyu1kG6DTR/CFrzbxsO0mklv0gI2RxmYdFQGSyzddRM6HWgeeh19Jk9E16AS3ytGiDm8si9MgM3T8/lpCXpb4lwnlKkf58hFFXuOrCFFJKU2Gs+/TdJoUyldogPAZcyIRfiDhKpm/UuzF8o7e0QqZSvJDN+wPSBUkvsPm7iJtCRdo7BqI1wkkDd84pELMDtOMak1nH6LyVTg7M6Sm2hLWq6GNIpCV3xLYWXFHiHv J4qKVTeE OZkQ6ON8/s0hf1T4X5wIZW357K+gBZg4lhlu38gzuMQU29+1Pv85XeGedDf/erQQneG9/IMd6NjN3+DZUb4mZu2DvL1TMt/PUN275m90vk6oWKtcuRzx/WuShC2HdEDkLhnNempeA575IAW2gCwGjcVHsRRkZM2O9+eSaAOZ8ecAPmFeARWW6fWSBiqEsZgJolmeygXD58a1zeJFsU4CsS9jOOd4CRUYfF1Mo5MVWHEhTJNGBgGd7ukcmOD40vqlpv0O6qySlvK2wvELdZPm0nSA8fpcNWG7/umMJj8MHMelAnGkVnlinMcQgGqrjliTYMide5qFJRtbgFYwnzzQPPGHLyXyt5fDCA4clY75rBtJ5JwpZlvnEKAPVZxd0VmFYMi9T3CyNNVWNEj+O4U3J6RJF46C438KaqN+N5j+pC/irkEPQcQdYovAJM5RirnSsCwE6Qc/l94642Jg26Rrhmplqrl55xT5lqUKt6X8Z8LnFVS2qzXT5uYZFUJLOZkn9miYpRP07VPeDJSOXit8mJXFb1SfJk7rmXmsi5WoY4Q7BligLzap/Qew9HUcUfH5DhXUq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them. In addition, the pagetable_dtor_free() in s390 does the same thing, so let's s390 also calls generic pagetable_dtor_free(). Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Acked-by: Alexander Gordeev # s390 --- arch/s390/mm/pgalloc.c | 6 ------ include/asm-generic/pgalloc.h | 12 ++++-------- include/asm-generic/tlb.h | 3 +-- include/linux/mm.h | 6 ++++++ 4 files changed, 11 insertions(+), 16 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 3e002dea6278f..a4e7619020931 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -180,12 +180,6 @@ unsigned long *page_table_alloc(struct mm_struct *mm) return table; } -static void pagetable_dtor_free(struct ptdesc *ptdesc) -{ - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); -} - void page_table_free(struct mm_struct *mm, unsigned long *table) { struct ptdesc *ptdesc = virt_to_ptdesc(table); diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 4afb346eae255..e3977ddca15e4 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -109,8 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) { struct ptdesc *ptdesc = page_ptdesc(pte_page); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } @@ -153,8 +152,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) struct ptdesc *ptdesc = virt_to_ptdesc(pmd); BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif @@ -202,8 +200,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud) struct ptdesc *ptdesc = virt_to_ptdesc(pud); BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifndef __HAVE_ARCH_PUD_FREE @@ -248,8 +245,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifndef __HAVE_ARCH_P4D_FREE diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 53ae7748f555b..e402aef79c93e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -213,8 +213,7 @@ static inline void __tlb_remove_table(void *table) { struct ptdesc *ptdesc = (struct ptdesc *)table; - pagetable_dtor(ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index cad11fa10c192..94078c488e904 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3001,6 +3001,12 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_dtor_free(struct ptdesc *ptdesc) +{ + pagetable_dtor(ptdesc); + pagetable_free(ptdesc); +} + static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { struct folio *folio = ptdesc_folio(ptdesc);