From patchwork Wed Jan 26 06:05:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12724613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 910B8C28CF5 for ; Wed, 26 Jan 2022 06:05:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B03296B0078; Wed, 26 Jan 2022 01:05:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A894B6B007D; Wed, 26 Jan 2022 01:05:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 936456B007B; Wed, 26 Jan 2022 01:05:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 802DF6B0074 for ; Wed, 26 Jan 2022 01:05:19 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3CE8B8E578 for ; Wed, 26 Jan 2022 06:05:19 +0000 (UTC) X-FDA: 79071400758.28.4EE3F71 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf23.hostedemail.com (Postfix) with ESMTP id CAADA140012 for ; Wed, 26 Jan 2022 06:05:18 +0000 (UTC) Received: by mail-qv1-f46.google.com with SMTP id d8so7287948qvv.2 for ; Tue, 25 Jan 2022 22:05:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=aWNRX9QsBPHLZOFVrwNXiWXc+ZQSkTRboCbVzmJISWM=; b=LlMcMekRBawFA/p+rIOXAZAiAa1k3SdlK7Hkpzh6hMhMDHMLaRQKA8ML9fwHV6llAJ Bdxeiz9HYmUpAchr0GfAWwJBjAWkCI7hp2dT6wD+ckNBhnwWq6ACHfH7aUSgQdr2rkI6 mpHA6t/nLQ+gI7EEx382Dl6maFaESiD2TqeWZhSl0Wp5MWF7QFaUj2Mq/Xv0SxWLoibg 4/IQbSGz3QzAafFPlJu6lx2YmPs8ClFWtFFBtMKmssnPMca5EWGQVPAvZhtySQ4pIdN/ 28UuDSoc27Z6S7UpTlIDARr3keJuGWON+9QTLfGn8icD8SJS61b3RMeKZK/dfDT8mLNu 9KFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aWNRX9QsBPHLZOFVrwNXiWXc+ZQSkTRboCbVzmJISWM=; b=v+7iJr6mHIpsaYIjAHbGk1LYksNVoG7EBTfFHoVBAsi7/3+VilOGkhQ1gyeofUNlAu 6KdgjkezFBkKAAukYv7QV5h3y4+J/HGHRDk7QNLmAdzcWAvj27cGyMLSyVgHTTWLuzWP SRTdIaANux2nBqxDwWHHKqKSwjTZuHWRczt3OAGDkgTkoTp4HzgdyzF2xweQ8TRa7uJ+ K4R2QWpgCfOYryIuiH8afIcmGsdOKCKUgxuUQZpx3bXX2OUSGIgbDeTSB1yixUMRNzYw J0xNDTpLTymXQnDO2LAdEIOMIDJ/IZP5kr44fNLBSKPYtsL/7uAAhZgSr1/L7+lVWg2K V9yw== X-Gm-Message-State: AOAM531O6G3Mpi3337CXlgCmEPwAiiczKCG5JVpT3JFxW5HAOXLieJ0l aDUDSrMqZECMCTfaCwc17Ik43Q== X-Google-Smtp-Source: ABdhPJwVByTrnm101Bf8YAmr4+nFjtChqbu8bu6/f6QCYcLdWej0+Cltb2+DvquGjJnoBQkLcKwHiA== X-Received: by 2002:a05:6214:1cca:: with SMTP id g10mr22397753qvd.36.1643177118106; Tue, 25 Jan 2022 22:05:18 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id o19sm9856699qta.40.2022.01.25.22.05.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jan 2022 22:05:17 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v3 1/4] mm/debug_vm_pgtable: remove pte entry from the page table Date: Wed, 26 Jan 2022 06:05:11 +0000 Message-Id: <20220126060514.1574935-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126060514.1574935-1-pasha.tatashin@soleen.com> References: <20220126060514.1574935-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspam-User: nil X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CAADA140012 X-Stat-Signature: 3wsf4n7em7fsm5nbyq61hjx86ao5ny1n Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=LlMcMekR; dmarc=none; spf=pass (imf23.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.46 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1643177118-423701 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The pte entry that is used in pte_advanced_tests() is never removed from the page table at the end of the test. The issue is detected by page_table_check, to repro compile kernel with the following configs: CONFIG_DEBUG_VM_PGTABLE=y CONFIG_PAGE_TABLE_CHECK=y CONFIG_PAGE_TABLE_CHECK_ENFORCED=y During the boot the following BUG is printed: [ 2.262821] debug_vm_pgtable: [debug_vm_pgtable ]: Validating architecture page table helpers [ 2.276826] ------------[ cut here ]------------ [ 2.280426] kernel BUG at mm/page_table_check.c:162! [ 2.284118] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 2.287787] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.16.0-11413-g2c271fe77d52 #3 [ 2.293226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014 ... The entry should be properly removed from the page table before the page is released to the free list. Fixes: a5c3b9ffb0f4 ("mm/debug_vm_pgtable: add tests validating advanced arch page table helpers") Signed-off-by: Pasha Tatashin Reviewed-by: Zi Yan Tested-by: Zi Yan Acked-by: David Rientjes --- mm/debug_vm_pgtable.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index a7ac97c76762..db2abd9e415b 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -171,6 +171,8 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) ptep_test_and_clear_young(args->vma, args->vaddr, args->ptep); pte = ptep_get(args->ptep); WARN_ON(pte_young(pte)); + + ptep_get_and_clear_full(args->mm, args->vaddr, args->ptep, 1); } static void __init pte_savedwrite_tests(struct pgtable_debug_args *args) From patchwork Wed Jan 26 06:05:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12724615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61F91C3526D for ; Wed, 26 Jan 2022 06:05:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D1156B007B; Wed, 26 Jan 2022 01:05:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 580966B0075; Wed, 26 Jan 2022 01:05:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29C506B007D; Wed, 26 Jan 2022 01:05:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 14F816B0075 for ; Wed, 26 Jan 2022 01:05:21 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 011179519A for ; Wed, 26 Jan 2022 06:05:20 +0000 (UTC) X-FDA: 79071400800.06.FEBA1C3 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf03.hostedemail.com (Postfix) with ESMTP id B042920010 for ; Wed, 26 Jan 2022 06:05:19 +0000 (UTC) Received: by mail-qt1-f181.google.com with SMTP id c15so13749378qtv.1 for ; Tue, 25 Jan 2022 22:05:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=R7V05PwaNjZnBTJ0+d7ZFZvO5dL9q1xRZty8VkA9hmQ=; b=bYJUNPpR1cZ5cpCeIRGCfDxZJpYmpPoh66ZJK77ukWwzsfKXMtKjFR0oyDl2PrVsRW IjZbI251p7s2yxGbyhaCpLtUYBe6eQ/H9o9djosXMfNCEvGwGBN+R0lcfbPMC5Cksznp HzEoAy39YDsLI/M5Ax6D0T9dXeKhcoYrDGg62jHcw+RFE+cRQEohFqYIRggXfbAR09jh c2sbLnBREwPZM/oV0w9qlL4dm7bHOUr+bSxvw/pnn/B3Aa/Q5IwY6gYNswVBk4dgWm/4 tybcJbilgAp6um5hCw1b5mXjM2VS1BA4dZTRuLQV21VwHvpW96HOLQcJFswBr1TlyuL5 O08Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R7V05PwaNjZnBTJ0+d7ZFZvO5dL9q1xRZty8VkA9hmQ=; b=o1khbxfcFlQXUAHnk01HWhd4vrcEPeeCOKx8f5lD+r5ZyOmZN6M1apPJZO8pjGxnuz sjyuj+2hxAHBbLjA0YaQCaushFz7LpzYGRe2KOnyuFwX1oCaRi5W0ieeBTd6LEaRFkdq z+NaKbrpr5H1kjBY670Ac6tZ0TGg+eIjqU3mkRvyz9AtL6QhKs7WQetlQpeu80RoUu90 G6p8Lb158DOywWU8LzVl4mGqJJkayqiaO14oVJw20HHnOJYn6Qo6bgFe+FMalhq/LTNa PHpVtHpjHUwAZDsEegg35tiVqkb3L+ZubetJ5RJEnCPyV5uyZrUkiiYB5cAn0qpSVKW2 b48A== X-Gm-Message-State: AOAM533W4ROeP7zTWbOKAMKlwu8zZx4H4TIceLLZ95gpMQkwdFSzaSmb N138ByZPO21kH5uRp9W67K9VgQ== X-Google-Smtp-Source: ABdhPJyGJu1wLWlwWE4gJcN7imPL+nBoU18Asvh3ix2RWYM+mVME0JOpFCDhcqGNLxudLxhV3f6eqg== X-Received: by 2002:a05:622a:43:: with SMTP id y3mr19448205qtw.347.1643177119022; Tue, 25 Jan 2022 22:05:19 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id o19sm9856699qta.40.2022.01.25.22.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jan 2022 22:05:18 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v3 2/4] mm/page_table_check: use unsigned long for page counters and cleanup Date: Wed, 26 Jan 2022 06:05:12 +0000 Message-Id: <20220126060514.1574935-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126060514.1574935-1-pasha.tatashin@soleen.com> References: <20220126060514.1574935-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B042920010 X-Stat-Signature: g9h54xg8a6wmsf4phq3btp3dkbmn4xg4 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=bYJUNPpR; dmarc=none; spf=pass (imf03.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Rspam-User: nil X-HE-Tag: 1643177119-59261 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For the consistency, use "unsigned long" for all page counters. Also, reduce code duplication by calling __page_table_check_*_clear() from __page_table_check_*_set() functions. Signed-off-by: Pasha Tatashin Reviewed-by: Wei Xu Acked-by: David Rientjes --- mm/page_table_check.c | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 7504e7caa2a1..c61d7ebe13b1 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -86,8 +86,8 @@ static void page_table_check_clear(struct mm_struct *mm, unsigned long addr, { struct page_ext *page_ext; struct page *page; + unsigned long i; bool anon; - int i; if (!pfn_valid(pfn)) return; @@ -121,8 +121,8 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, { struct page_ext *page_ext; struct page *page; + unsigned long i; bool anon; - int i; if (!pfn_valid(pfn)) return; @@ -152,10 +152,10 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, void __page_table_check_zero(struct page *page, unsigned int order) { struct page_ext *page_ext = lookup_page_ext(page); - int i; + unsigned long i; BUG_ON(!page_ext); - for (i = 0; i < (1 << order); i++) { + for (i = 0; i < (1ul << order); i++) { struct page_table_check *ptc = get_page_table_check(page_ext); BUG_ON(atomic_read(&ptc->anon_map_count)); @@ -206,17 +206,10 @@ EXPORT_SYMBOL(__page_table_check_pud_clear); void __page_table_check_pte_set(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { - pte_t old_pte; - if (&init_mm == mm) return; - old_pte = *ptep; - if (pte_user_accessible_page(old_pte)) { - page_table_check_clear(mm, addr, pte_pfn(old_pte), - PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pte_clear(mm, addr, *ptep); if (pte_user_accessible_page(pte)) { page_table_check_set(mm, addr, pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT, @@ -228,17 +221,10 @@ EXPORT_SYMBOL(__page_table_check_pte_set); void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) { - pmd_t old_pmd; - if (&init_mm == mm) return; - old_pmd = *pmdp; - if (pmd_user_accessible_page(old_pmd)) { - page_table_check_clear(mm, addr, pmd_pfn(old_pmd), - PMD_PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pmd_clear(mm, addr, *pmdp); if (pmd_user_accessible_page(pmd)) { page_table_check_set(mm, addr, pmd_pfn(pmd), PMD_PAGE_SIZE >> PAGE_SHIFT, @@ -250,17 +236,10 @@ EXPORT_SYMBOL(__page_table_check_pmd_set); void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud) { - pud_t old_pud; - if (&init_mm == mm) return; - old_pud = *pudp; - if (pud_user_accessible_page(old_pud)) { - page_table_check_clear(mm, addr, pud_pfn(old_pud), - PUD_PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pud_clear(mm, addr, *pudp); if (pud_user_accessible_page(pud)) { page_table_check_set(mm, addr, pud_pfn(pud), PUD_PAGE_SIZE >> PAGE_SHIFT, From patchwork Wed Jan 26 06:05:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12724616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29489C28CF5 for ; Wed, 26 Jan 2022 06:05:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 809736B0075; Wed, 26 Jan 2022 01:05:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 780776B0080; Wed, 26 Jan 2022 01:05:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 531ED6B0081; Wed, 26 Jan 2022 01:05:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id 356646B0075 for ; Wed, 26 Jan 2022 01:05:21 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E5CE7183380A7 for ; Wed, 26 Jan 2022 06:05:20 +0000 (UTC) X-FDA: 79071400800.29.F0A50AF Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf26.hostedemail.com (Postfix) with ESMTP id 9143914000D for ; Wed, 26 Jan 2022 06:05:20 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id i4so11772245qtr.0 for ; Tue, 25 Jan 2022 22:05:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=HOl6dnjRSlkzFWIwAEZQBaaOxfPRDLrxvpvmNPVil3I=; b=jwFuMnHWqoNk8uXeaYsGmMgbTXTo5euHb6/esSzRi8Hx0i9wGtS6JDG3u8LGq6AmU4 eFdQHw+wFC2IAD7gAkutikIJJdEwF7jOdPz3HURvZDFMbhCd/ezSO9xcpJINsMjiXzM/ ux3ThHF0mwzLaaRsxHG+POu362xUihatB6ZiznsjtQ4fvhmMxxbd8YAbcJLjLQXGr+jW KGJA/EIVl7kOjJRSIaFq3sieVyL0O4O64ErJ4CMN2OkScAJfguQeZMBZ0kmND05wNMNY hqv9KmXfIvyrrebEnHTN4PTse2Py7UlyFStajrNbvPv+9NnRYJab498z0qxxx6Ztv/bY Sltw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HOl6dnjRSlkzFWIwAEZQBaaOxfPRDLrxvpvmNPVil3I=; b=VvF5qxyuV9q7y/Pa2HMB00hXPUnF6rCwcPB9QqvFR4OS+/XIjf6qGyo1Z1LbRiQ7xB UvQ0l/GegGeXPFd/OWnJXGE81Nd7XdrUjMgJSLs/IuBkgIiPulps/tHpHyxjiGFGBblQ kQNBV2TQty91P2qYv5adqezepTeVIS+vfRz19stPznMbgPPjk0Fo3LyOfM4lHp1oqoJk ZcVUwFOXv5Lsn/EC4Tnb2j4w2lJBbVKIFiRWwXs5QQ77HKki1IulyLwkG1XUxIFjSG5W L2Yy+VefCXtNIN8XKwua3HEqcvI0nlD6CspD63Qc+yb1eKPHJZabvEXDgU0tnNQuTOb3 iDNw== X-Gm-Message-State: AOAM5338B3DJMkA9sNAaXksmIwEJ1hpFfVcLs9gE4qVAZKE2IZyDRT8t JEIuXPN9/yOdO9lXLUyDnBRLr3aLyPKUuw== X-Google-Smtp-Source: ABdhPJxawlYifreSuisEPDkI9IQt+vr1qpvjrerAOfVLQ1Jx0TsXCwSpoyobPIpCZnLAgMF70g7lVg== X-Received: by 2002:ac8:5b01:: with SMTP id m1mr19121273qtw.360.1643177119921; Tue, 25 Jan 2022 22:05:19 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id o19sm9856699qta.40.2022.01.25.22.05.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jan 2022 22:05:19 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v3 3/4] mm/khugepaged: unify collapse pmd clear, flush and free Date: Wed, 26 Jan 2022 06:05:13 +0000 Message-Id: <20220126060514.1574935-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126060514.1574935-1-pasha.tatashin@soleen.com> References: <20220126060514.1574935-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=jwFuMnHW; spf=pass (imf26.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspam-User: nil X-Rspamd-Queue-Id: 9143914000D X-Stat-Signature: qpsnsb5n7kbutfch3k4mezazcet9xxyq X-Rspamd-Server: rspam12 X-HE-Tag: 1643177120-443575 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unify the code that flushes, clears pmd entry, and frees the PTE table level into a new function collapse_and_free_pmd(). This clean-up is useful as in the next patch we will add another call to this function to iterate through PTE prior to freeing the level for page table check. Signed-off-by: Pasha Tatashin Acked-by: David Rientjes --- mm/khugepaged.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 35f14d0a00a6..440112355ffe 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1416,6 +1416,17 @@ static int khugepaged_add_pte_mapped_thp(struct mm_struct *mm, return 0; } +static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + spinlock_t *ptl = pmd_lock(vma->vm_mm, pmdp); + pmd_t pmd = pmdp_collapse_flush(vma, addr, pmdp); + + spin_unlock(ptl); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(pmd)); +} + /** * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at * address haddr. @@ -1433,7 +1444,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) struct vm_area_struct *vma = find_vma(mm, haddr); struct page *hpage; pte_t *start_pte, *pte; - pmd_t *pmd, _pmd; + pmd_t *pmd; spinlock_t *ptl; int count = 0; int i; @@ -1509,12 +1520,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) } /* step 4: collapse pmd */ - ptl = pmd_lock(vma->vm_mm, pmd); - _pmd = pmdp_collapse_flush(vma, haddr, pmd); - spin_unlock(ptl); - mm_dec_nr_ptes(mm); - pte_free(mm, pmd_pgtable(_pmd)); - + collapse_and_free_pmd(mm, vma, haddr, pmd); drop_hpage: unlock_page(hpage); put_page(hpage); @@ -1552,7 +1558,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) struct vm_area_struct *vma; struct mm_struct *mm; unsigned long addr; - pmd_t *pmd, _pmd; + pmd_t *pmd; i_mmap_lock_write(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) { @@ -1591,14 +1597,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) * reverse order. Trylock is a way to avoid deadlock. */ if (mmap_write_trylock(mm)) { - if (!khugepaged_test_exit(mm)) { - spinlock_t *ptl = pmd_lock(mm, pmd); - /* assume page table is clear */ - _pmd = pmdp_collapse_flush(vma, addr, pmd); - spin_unlock(ptl); - mm_dec_nr_ptes(mm); - pte_free(mm, pmd_pgtable(_pmd)); - } + if (!khugepaged_test_exit(mm)) + collapse_and_free_pmd(mm, vma, addr, pmd); mmap_write_unlock(mm); } else { /* Try again later */ From patchwork Wed Jan 26 06:05:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12724617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9E2AC3526D for ; Wed, 26 Jan 2022 06:05:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B05896B007D; Wed, 26 Jan 2022 01:05:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8E486B0080; Wed, 26 Jan 2022 01:05:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DEB96B0081; Wed, 26 Jan 2022 01:05:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id 6E2286B007D for ; Wed, 26 Jan 2022 01:05:22 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 20B028249980 for ; Wed, 26 Jan 2022 06:05:22 +0000 (UTC) X-FDA: 79071400884.12.2BEBA23 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf18.hostedemail.com (Postfix) with ESMTP id 9AB351C0019 for ; Wed, 26 Jan 2022 06:05:21 +0000 (UTC) Received: by mail-qv1-f50.google.com with SMTP id k4so27928855qvt.6 for ; Tue, 25 Jan 2022 22:05:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=NVid4lwIGU/SbGxET47lcYa9PMp6oZkK6MJSbQCBUP8=; b=VHecgKNFLp6+K8o+yvV6rnlGSNdQlBeJONWAcseLydDnJtbkgoNf0Rw/8Lz/nVAGfV d+hM2tCC86PkP2UUDjdPUnhcUKD1ufG4VzYPbBCK0KNotR7lUwePnhxeYXoFEzh7mcQe w0WJTijjWGUmQZAnUkLRrtcvcEqtQwWd18lPktYpYgcpP67/DcJrLIWRUZUfjeSlMG4Q yx3YUewSwqiwIAjSNilaNWM274WwyWPNRzWFJdLX1tW6rvdv5XrCcv5yq+rBJ44oG2gV VKkBDgU8utXoZDbGBLgYdEGpFCAlDm5eojRZ9+CCDFsXjMdvwtWKsvoy86o9aHx/FLn4 lt+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NVid4lwIGU/SbGxET47lcYa9PMp6oZkK6MJSbQCBUP8=; b=qNJQed1ZnyVmrbS+gvJNIVnAqrmMaevnlQN4XSsgmCDkh4oX2FFTNzZ+wJhevg/OPd JhemTtmLT3DaswHboCrtfK+SepHa3NHjP5j5/v6QTK8JWHiKVY14qY2vohDOvcasRbks lSq/fBG37ie+sbibPNSkmyDsd0EhnyVNAc5nMGBWpt9ClF554XQMOiEIROmYIa3/rwmD Ze9KxfHQwUvCrKBR4wcPuhF26bjM4GKyjoaQxVLxG9akDdTrodJM9nMZ27L87geoTGJ3 nJ5U876G9LBXGtTqqCk2/H+EqH36uVkNGO50PkftOY4uE9q1ySHm4GJwKYL4yqfozvtu ABtw== X-Gm-Message-State: AOAM532GkhHl/wpEMd/hTz0JiOhnQn2d3G1+CQ+TcEOQhFmVLdfahnY+ xWAWGyKw2/fVvBYyQqnpzcpU9w== X-Google-Smtp-Source: ABdhPJzd2K1nQEX6RdRaNxBEBXX9UK+PMvC+jASppza1eRowqKv+fF5BXklIge08ffTBviDpvDBTEg== X-Received: by 2002:a05:6214:2302:: with SMTP id gc2mr22470192qvb.126.1643177120814; Tue, 25 Jan 2022 22:05:20 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id o19sm9856699qta.40.2022.01.25.22.05.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jan 2022 22:05:20 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v3 4/4] mm/page_table_check: check entries at pmd levels Date: Wed, 26 Jan 2022 06:05:14 +0000 Message-Id: <20220126060514.1574935-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126060514.1574935-1-pasha.tatashin@soleen.com> References: <20220126060514.1574935-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9AB351C0019 X-Rspam-User: nil Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=VHecgKNF; dmarc=none; spf=pass (imf18.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.50 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Stat-Signature: ainecje5xwghmpaknz8sy3or6nmhruam X-HE-Tag: 1643177121-533008 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: syzbot detected a case where the page table counters were not properly updated. syzkaller login: ------------[ cut here ]------------ kernel BUG at mm/page_table_check.c:162! invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 3099 Comm: pasha Not tainted 5.16.0+ #48 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIO4 RIP: 0010:__page_table_check_zero+0x159/0x1a0 Code: 7d 3a b2 ff 45 39 f5 74 2a e8 43 38 b2 ff 4d 85 e4 01 RSP: 0018:ffff888010667418 EFLAGS: 00010293 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000 RDX: ffff88800cea8680 RSI: ffffffff81becaf9 RDI: 0000000003 RBP: ffff888010667450 R08: 0000000000000001 R09: 0000000000 R10: ffffffff81becaab R11: 0000000000000001 R12: ffff888008 R13: 0000000000000001 R14: 0000000000000200 R15: dffffc0000 FS: 0000000000000000(0000) GS:ffff888035e00000(0000) knlG0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffd875cad00 CR3: 00000000094ce000 CR4: 0000000000 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000 Call Trace: free_pcp_prepare+0x3be/0xaa0 free_unref_page+0x1c/0x650 ? trace_hardirqs_on+0x6a/0x1d0 free_compound_page+0xec/0x130 free_transhuge_page+0x1be/0x260 __put_compound_page+0x90/0xd0 release_pages+0x54c/0x1060 ? filemap_remove_folio+0x161/0x210 ? lock_downgrade+0x720/0x720 ? __put_page+0x150/0x150 ? filemap_free_folio+0x164/0x350 __pagevec_release+0x7c/0x110 shmem_undo_range+0x85e/0x1250 ... The repro involved having a huge page that is split due to uprobe event temporarily replacing one of the pages in the huge page. Later the huge page was combined again, but the counters were off, as the PTE level was not properly updated. Make sure that when PMD is cleared and prior to freeing the level the PTEs are updated. Fixes: df4e817b7108 ("mm: page table check") Signed-off-by: Pasha Tatashin --- include/linux/page_table_check.h | 18 ++++++++++++++++++ mm/khugepaged.c | 2 ++ mm/page_table_check.c | 21 +++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/include/linux/page_table_check.h b/include/linux/page_table_check.h index 38cace1da7b6..e88bbe37727b 100644 --- a/include/linux/page_table_check.h +++ b/include/linux/page_table_check.h @@ -26,6 +26,8 @@ void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud); +void __page_table_check_pmd_clear_full(struct mm_struct *mm, unsigned long addr, + pmd_t pmd); static inline void page_table_check_alloc(struct page *page, unsigned int order) { @@ -100,6 +102,16 @@ static inline void page_table_check_pud_set(struct mm_struct *mm, __page_table_check_pud_set(mm, addr, pudp, pud); } +static inline void page_table_check_pmd_clear_full(struct mm_struct *mm, + unsigned long addr, + pmd_t pmd) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pmd_clear_full(mm, addr, pmd); +} + #else static inline void page_table_check_alloc(struct page *page, unsigned int order) @@ -143,5 +155,11 @@ static inline void page_table_check_pud_set(struct mm_struct *mm, { } +static inline void page_table_check_pmd_clear_full(struct mm_struct *mm, + unsigned long addr, + pmd_t pmd) +{ +} + #endif /* CONFIG_PAGE_TABLE_CHECK */ #endif /* __LINUX_PAGE_TABLE_CHECK_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 440112355ffe..eefe3706f6c2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -1424,6 +1425,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v spin_unlock(ptl); mm_dec_nr_ptes(mm); + page_table_check_pmd_clear_full(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); } diff --git a/mm/page_table_check.c b/mm/page_table_check.c index c61d7ebe13b1..251f95a808b4 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -247,3 +247,24 @@ void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, } } EXPORT_SYMBOL(__page_table_check_pud_set); + +void __page_table_check_pmd_clear_full(struct mm_struct *mm, unsigned long addr, + pmd_t pmd) +{ + if (&init_mm == mm) + return; + + if (!pmd_bad(pmd) && !pmd_leaf(pmd)) { + pte_t *ptep = pte_offset_map(&pmd, addr); + unsigned long i; + + pte_unmap(ptep); + for (i = 0; i < PTRS_PER_PTE; i++) { + __page_table_check_pte_clear(mm, addr, *ptep); + addr += PAGE_SIZE; + ptep++; + } + } else { + __page_table_check_pmd_clear(mm, addr, pmd); + } +}