From patchwork Wed Jan 26 18:36:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12725595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A90DC63682 for ; Wed, 26 Jan 2022 18:36:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 228436B0073; Wed, 26 Jan 2022 13:36:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 198966B008A; Wed, 26 Jan 2022 13:36:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0B06B0074; Wed, 26 Jan 2022 13:36:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id C26F26B0073 for ; Wed, 26 Jan 2022 13:36:41 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 89F9386E81 for ; Wed, 26 Jan 2022 18:36:41 +0000 (UTC) X-FDA: 79073294202.25.049314F Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf31.hostedemail.com (Postfix) with ESMTP id 416EB20030 for ; Wed, 26 Jan 2022 18:36:41 +0000 (UTC) Received: by mail-qv1-f48.google.com with SMTP id hu2so615753qvb.8 for ; Wed, 26 Jan 2022 10:36:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=gwQJ38KtV0HXsmYJc+K1cECJTZ3g+2Vx7ehA2fokXc0=; b=QcWdQlJI68RhXl0oTXAQOXqwPV7EOeuJVrY/beLTxsFBpQZ/nyKXBBMIIRIxp3UZb6 RE0qIRmmDPvKHGdBBPccD0mrIgW7zATHPRADWCTEb7XaEUGVnBm9GJfhl8LYGahP8bRU habISB7anoDtdxvsn9BMFWlNI/HO3zm4p1rtKlge3r3TqmSlDKaEnx/0D1getJ3c1I80 z0cbTvjVWolLQAkLRBL3tcgH38g6KeNmYOxgMgSytQ4Hklca1ZLKBTHYgpWiWTLjQKh2 lz0Hk2/F2AzAgIWChOtEJStlib6W2XqYpPkdHK/penKyAh12R+x6E1iZ8reMOKdyS5NP BhHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gwQJ38KtV0HXsmYJc+K1cECJTZ3g+2Vx7ehA2fokXc0=; b=s1+/qnAwZL6wnBkMpenUDET/ehkBcrCnrl0ta88rDoPQPMdWYbynpMtOhvDVwV3FzU POttaDJji2MHwOszS786hUxUIckk3ZEXzgbw38TvSn04ZV4Sl8JoneemZ0yIDFfS4Fsa 4VGenEB0DbIuY9HwnvWHeAi8tL135WiVN4mu6JN1C/PRKSvX9aGkJozQWZr9rnfGToEg E64d1BEzn6nejuUs0MMnwXquP47xNwsss83wPsuTHtKpVszb6ixqWl+8RrYoYNcntZVc WEZegTz7mP1g7/Y+6eLlk2yKxYPA5d5rEBZurvvmL1PBLtYrnV1/AJPcn022s0CN7C1V kqrw== X-Gm-Message-State: AOAM532efYipV/+J+NivYuMIyONDneqRY0FK9H6Yiuhevvyz3Ksx4z6J ylzi0AN/RsSZDoKI0Ywy3/VXwg== X-Google-Smtp-Source: ABdhPJzHYYWtkn3tMOAcHosHOkT8eLk8rK3linsKwbJSVMvq/WP3ZiLoTcazMadoRkZvSZnoFSFZPQ== X-Received: by 2002:a05:6214:2a83:: with SMTP id jr3mr32858qvb.68.1643222200531; Wed, 26 Jan 2022 10:36:40 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id h9sm26226qkp.75.2022.01.26.10.36.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:36:40 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v4 1/4] mm/debug_vm_pgtable: remove pte entry from the page table Date: Wed, 26 Jan 2022 18:36:34 +0000 Message-Id: <20220126183637.1840960-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183637.1840960-1-pasha.tatashin@soleen.com> References: <20220126183637.1840960-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 416EB20030 X-Stat-Signature: 4zr8f1qtdbcyf3rh6cfsqay86556jtjo Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=QcWdQlJI; dmarc=none; spf=pass (imf31.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.48 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Rspam-User: nil X-HE-Tag: 1643222201-199967 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The pte entry that is used in pte_advanced_tests() is never removed from the page table at the end of the test. The issue is detected by page_table_check, to repro compile kernel with the following configs: CONFIG_DEBUG_VM_PGTABLE=y CONFIG_PAGE_TABLE_CHECK=y CONFIG_PAGE_TABLE_CHECK_ENFORCED=y During the boot the following BUG is printed: [ 2.262821] debug_vm_pgtable: [debug_vm_pgtable ]: Validating architecture page table helpers [ 2.276826] ------------[ cut here ]------------ [ 2.280426] kernel BUG at mm/page_table_check.c:162! [ 2.284118] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 2.287787] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.16.0-11413-g2c271fe77d52 #3 [ 2.293226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014 ... The entry should be properly removed from the page table before the page is released to the free list. Fixes: a5c3b9ffb0f4 ("mm/debug_vm_pgtable: add tests validating advanced arch page table helpers") Cc: stable@vger.kernel.org # 5.9+ Signed-off-by: Pasha Tatashin Reviewed-by: Zi Yan Tested-by: Zi Yan Acked-by: David Rientjes --- mm/debug_vm_pgtable.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index a7ac97c76762..db2abd9e415b 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -171,6 +171,8 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) ptep_test_and_clear_young(args->vma, args->vaddr, args->ptep); pte = ptep_get(args->ptep); WARN_ON(pte_young(pte)); + + ptep_get_and_clear_full(args->mm, args->vaddr, args->ptep, 1); } static void __init pte_savedwrite_tests(struct pgtable_debug_args *args) From patchwork Wed Jan 26 18:36:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12725596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2034C2BA4C for ; Wed, 26 Jan 2022 18:36:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8CD76B0074; Wed, 26 Jan 2022 13:36:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E3C2D6B0075; Wed, 26 Jan 2022 13:36:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C19806B008A; Wed, 26 Jan 2022 13:36:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id ADAD26B0074 for ; Wed, 26 Jan 2022 13:36:42 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 641DA8249980 for ; Wed, 26 Jan 2022 18:36:42 +0000 (UTC) X-FDA: 79073294244.18.499D165 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf11.hostedemail.com (Postfix) with ESMTP id 0230C4003B for ; Wed, 26 Jan 2022 18:36:41 +0000 (UTC) Received: by mail-qv1-f54.google.com with SMTP id t7so685322qvj.0 for ; Wed, 26 Jan 2022 10:36:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=NIQdhozYZQYwJuXanYTpSkQiAWCcpJhyfoG58pHCslA=; b=Bo3Fdqy4H5CvO8zl5pomEkR1gp3x8MwUzxy7Z5HKmiSyjdlsmqjAhR00ywO5pAO3x4 uNdiT3FyK6gp2JqbGiSd/Q2PthIYWdhPSdAn5TOjaihu46l72mfeiNMyhA7dutVKxnFC wA+rf72VjD8j3THQUhwX2QykoPzF+4MFSz1Y3IID6oxp2xlAXoMzoIpb0MJF9DKU6SnM xOgftlP8LGcFpqOtobCRLgW+R/1Ngani8djbZ77zWs9MEs9OHHcuEOJ5C1cSxMkRCd16 eGxndmGbnEooWd/kaYhc+on4vEHAeG6wRjeebLrkxHZo73GVcyyxKk2j8j8Ue86rPdzi 9RKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NIQdhozYZQYwJuXanYTpSkQiAWCcpJhyfoG58pHCslA=; b=u6R0oUMsejJB6lV3CmXIUzqZh6AN0AkdXCyCQI2aexSYV2yG7QFtinMfCuwVBRAaRK wMP6A/znuvfAU+jDVr++M/20XwKpkHEP4utsPWzpkR6XRBUdP/o9MICggwbBCIm7X6FK F7N7zIZPOS9zV+B0gXBa0iazf1dVz/48DIYjBLz0bF0S6GfE4CstSHUBZQpCu74I4PGn NymcfdPgiA4uDXsr4ymFnM+PwoHXhU74HMyd1PIskJFeKAifAC353rfPOpOYkrlBrXjm QYi1WWlO2JjHyoDTlIdfI4BfsH+qcpPnnHOOdyMAWq47fPIJmk47oYPbs3hfLyQFgzYh nx6g== X-Gm-Message-State: AOAM532+uH+ecel5t77H0qJpoDv2NU8NyEpMIlss9iN9t9LL/JFPXtck nsxlAQUeaGp6QXcZoJY6Hgk9Fg== X-Google-Smtp-Source: ABdhPJyuNmP3Wf47MQITz51h0tip2WfX44XPte2EplKCXHLqfSHCHJVpHfQJ5NtwUc2OxyfaeImh2w== X-Received: by 2002:a05:6214:1874:: with SMTP id eh20mr1050qvb.13.1643222201356; Wed, 26 Jan 2022 10:36:41 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id h9sm26226qkp.75.2022.01.26.10.36.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:36:40 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v4 2/4] mm/page_table_check: use unsigned long for page counters and cleanup Date: Wed, 26 Jan 2022 18:36:35 +0000 Message-Id: <20220126183637.1840960-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183637.1840960-1-pasha.tatashin@soleen.com> References: <20220126183637.1840960-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 0230C4003B X-Stat-Signature: p3dog6xf59k4kad6fz5gqwr6d78373t4 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=Bo3Fdqy4; dmarc=none; spf=pass (imf11.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.54 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Rspam-User: nil X-HE-Tag: 1643222201-33536 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For the consistency, use "unsigned long" for all page counters. Also, reduce code duplication by calling __page_table_check_*_clear() from __page_table_check_*_set() functions. Signed-off-by: Pasha Tatashin Reviewed-by: Wei Xu Acked-by: David Rientjes --- mm/page_table_check.c | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 7504e7caa2a1..c61d7ebe13b1 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -86,8 +86,8 @@ static void page_table_check_clear(struct mm_struct *mm, unsigned long addr, { struct page_ext *page_ext; struct page *page; + unsigned long i; bool anon; - int i; if (!pfn_valid(pfn)) return; @@ -121,8 +121,8 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, { struct page_ext *page_ext; struct page *page; + unsigned long i; bool anon; - int i; if (!pfn_valid(pfn)) return; @@ -152,10 +152,10 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, void __page_table_check_zero(struct page *page, unsigned int order) { struct page_ext *page_ext = lookup_page_ext(page); - int i; + unsigned long i; BUG_ON(!page_ext); - for (i = 0; i < (1 << order); i++) { + for (i = 0; i < (1ul << order); i++) { struct page_table_check *ptc = get_page_table_check(page_ext); BUG_ON(atomic_read(&ptc->anon_map_count)); @@ -206,17 +206,10 @@ EXPORT_SYMBOL(__page_table_check_pud_clear); void __page_table_check_pte_set(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { - pte_t old_pte; - if (&init_mm == mm) return; - old_pte = *ptep; - if (pte_user_accessible_page(old_pte)) { - page_table_check_clear(mm, addr, pte_pfn(old_pte), - PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pte_clear(mm, addr, *ptep); if (pte_user_accessible_page(pte)) { page_table_check_set(mm, addr, pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT, @@ -228,17 +221,10 @@ EXPORT_SYMBOL(__page_table_check_pte_set); void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) { - pmd_t old_pmd; - if (&init_mm == mm) return; - old_pmd = *pmdp; - if (pmd_user_accessible_page(old_pmd)) { - page_table_check_clear(mm, addr, pmd_pfn(old_pmd), - PMD_PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pmd_clear(mm, addr, *pmdp); if (pmd_user_accessible_page(pmd)) { page_table_check_set(mm, addr, pmd_pfn(pmd), PMD_PAGE_SIZE >> PAGE_SHIFT, @@ -250,17 +236,10 @@ EXPORT_SYMBOL(__page_table_check_pmd_set); void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud) { - pud_t old_pud; - if (&init_mm == mm) return; - old_pud = *pudp; - if (pud_user_accessible_page(old_pud)) { - page_table_check_clear(mm, addr, pud_pfn(old_pud), - PUD_PAGE_SIZE >> PAGE_SHIFT); - } - + __page_table_check_pud_clear(mm, addr, *pudp); if (pud_user_accessible_page(pud)) { page_table_check_set(mm, addr, pud_pfn(pud), PUD_PAGE_SIZE >> PAGE_SHIFT, From patchwork Wed Jan 26 18:36:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12725597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A282C28CF5 for ; Wed, 26 Jan 2022 18:36:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3E5B6B0075; Wed, 26 Jan 2022 13:36:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEFBF6B008A; Wed, 26 Jan 2022 13:36:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C91DB6B008C; Wed, 26 Jan 2022 13:36:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id A8FFD6B0075 for ; Wed, 26 Jan 2022 13:36:43 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 67C328F6E4 for ; Wed, 26 Jan 2022 18:36:43 +0000 (UTC) X-FDA: 79073294286.08.B6E064B Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf21.hostedemail.com (Postfix) with ESMTP id 051B71C0026 for ; Wed, 26 Jan 2022 18:36:42 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id 71so400579qkf.4 for ; Wed, 26 Jan 2022 10:36:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=a93X1YuUUYjeQ1DmHZzgEs8n16C54TBZM8wi1s2vhvo=; b=N1Vt6B9Sta1FwZczZQvEpySLcRnZhDydeiS6pntiC+WThA+3ZreI3+60RQfO5kdCVX PYbT9ygJ+RpShOQSUmBUF4LeD0OMOmi4BVXaeieVPEDI7qjBopBvxedBObpiCvVEXM19 vIaReulPyOxwwY/oUq54wlge34q8x/ppQVF/brbp0pf/bFE+mP23rJDsUoy6LY1nNdvs h56xvgdY/V+epLYxhiPz25bNyIF3I1Jo2UFm/dTmOhdMEIvwyXt9q4boHCTlKTfyDPb/ FQBTIf+mHIo+LKv2ZXUV+XiQw9lfKjQr1nPDLZz4qcagM/2eqA8M54EazwRtE/p3p7qW J1lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a93X1YuUUYjeQ1DmHZzgEs8n16C54TBZM8wi1s2vhvo=; b=PLAOIe4BK7572Wn0q9dPm81ipVdAOqThBmnHpAMnPmrQksaakRYfVIoE0l4a+LN5BX ZiGCLSaJ0+nYh2U5E+0IhIu4ufHE6r9mrIWMrt4XTgrnpEFUoxpoAfxwrjyxu8/3KG3v 1gtPshG7y3V7WEO9wiWets1sDpcPezKo1fvF+/6b6zVLG3OKKA4kZ03R2lJCo+x4nsEq 9g3VfYbgcaCUtKy8RQasecK5LkYcA4v7WNYupk4v1G2SPQfkoA9gEQi/Z9Yx1QDDfPk4 oANOv7b7DKSxsRVdohu7MgVbv9T4OamNtLvPEPxnso9GIbzbY0mufnJHLDxfGev9iuCu H73Q== X-Gm-Message-State: AOAM532mhXdeIRKy+3TRFxWTujgaSob6Iqr9qWLxvN5BLoyBjT+f1ZSB 9pzOsG1wSOLWoFz4I1St8s0DaQ== X-Google-Smtp-Source: ABdhPJw0PTloyF5Qbm9D+I2W7WmpXBIhuugeFvsdlL3/szCTDEds6C8kR7dEnSFl+43JC4D5IJPKGg== X-Received: by 2002:a05:620a:31a3:: with SMTP id bi35mr100511qkb.198.1643222202238; Wed, 26 Jan 2022 10:36:42 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id h9sm26226qkp.75.2022.01.26.10.36.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:36:41 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v4 3/4] mm/khugepaged: unify collapse pmd clear, flush and free Date: Wed, 26 Jan 2022 18:36:36 +0000 Message-Id: <20220126183637.1840960-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183637.1840960-1-pasha.tatashin@soleen.com> References: <20220126183637.1840960-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 051B71C0026 X-Stat-Signature: 9mrs3dthdwuzrawt5tqr5tu5r59etg68 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=N1Vt6B9S; dmarc=none; spf=pass (imf21.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Rspam-User: nil X-HE-Tag: 1643222202-203579 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unify the code that flushes, clears pmd entry, and frees the PTE table level into a new function collapse_and_free_pmd(). This clean-up is useful as in the next patch we will add another call to this function to iterate through PTE prior to freeing the level for page table check. Signed-off-by: Pasha Tatashin Acked-by: David Rientjes --- mm/khugepaged.c | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 35f14d0a00a6..30e59e4af272 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1416,6 +1416,19 @@ static int khugepaged_add_pte_mapped_thp(struct mm_struct *mm, return 0; } +static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + spinlock_t *ptl; + pmd_t pmd; + + ptl = pmd_lock(vma->vm_mm, pmdp); + pmd = pmdp_collapse_flush(vma, addr, pmdp); + spin_unlock(ptl); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(pmd)); +} + /** * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at * address haddr. @@ -1433,7 +1446,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) struct vm_area_struct *vma = find_vma(mm, haddr); struct page *hpage; pte_t *start_pte, *pte; - pmd_t *pmd, _pmd; + pmd_t *pmd; spinlock_t *ptl; int count = 0; int i; @@ -1509,12 +1522,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) } /* step 4: collapse pmd */ - ptl = pmd_lock(vma->vm_mm, pmd); - _pmd = pmdp_collapse_flush(vma, haddr, pmd); - spin_unlock(ptl); - mm_dec_nr_ptes(mm); - pte_free(mm, pmd_pgtable(_pmd)); - + collapse_and_free_pmd(mm, vma, haddr, pmd); drop_hpage: unlock_page(hpage); put_page(hpage); @@ -1552,7 +1560,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) struct vm_area_struct *vma; struct mm_struct *mm; unsigned long addr; - pmd_t *pmd, _pmd; + pmd_t *pmd; i_mmap_lock_write(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) { @@ -1591,14 +1599,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) * reverse order. Trylock is a way to avoid deadlock. */ if (mmap_write_trylock(mm)) { - if (!khugepaged_test_exit(mm)) { - spinlock_t *ptl = pmd_lock(mm, pmd); - /* assume page table is clear */ - _pmd = pmdp_collapse_flush(vma, addr, pmd); - spin_unlock(ptl); - mm_dec_nr_ptes(mm); - pte_free(mm, pmd_pgtable(_pmd)); - } + if (!khugepaged_test_exit(mm)) + collapse_and_free_pmd(mm, vma, addr, pmd); mmap_write_unlock(mm); } else { /* Try again later */ From patchwork Wed Jan 26 18:36:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12725598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0295AC2BA4C for ; Wed, 26 Jan 2022 18:36:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1983D6B008A; Wed, 26 Jan 2022 13:36:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 11F9A6B008C; Wed, 26 Jan 2022 13:36:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED8F56B0092; Wed, 26 Jan 2022 13:36:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id D0A506B008A for ; Wed, 26 Jan 2022 13:36:45 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9481491E2A for ; Wed, 26 Jan 2022 18:36:45 +0000 (UTC) X-FDA: 79073294370.24.C977D84 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf03.hostedemail.com (Postfix) with ESMTP id C09FE20037 for ; Wed, 26 Jan 2022 18:36:43 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id w8so383245qkw.8 for ; Wed, 26 Jan 2022 10:36:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UkC+OFwWk3t7ZVs9bgfgTxL7WVNxv4IAEUpD9718gw8=; b=NuQmPwn9ODC2HrsAp/Fbc+4/QtEhN2zRjerYLrui6bUFh1RWkI7nzbm+smnYo7LWeB 2dNlkMC+FM17xTVcaRXVJa3qQN37qUBGEYuuHMCT0jJAdyCS+1xVMS+jDOSJTKesYi20 um1TQGUvf4zRX0ILDdsgDMoBxG7bPZqsGMcDuka0gdbtzgAr2KlM3lJtNHCgq+1TNNL8 b5hEJG9BKbjM/6uQVOkHVTjkLg9CVQoV+ytlotm1wrEIB6oFTc1MiZKybpTeMUq6mYIi EWhWv1fayjNbSLbJz8ugZgU7nSXLQiLfKkoptdkxlxSmEXack3JpAWmv0ICdY4lS2svL i0nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UkC+OFwWk3t7ZVs9bgfgTxL7WVNxv4IAEUpD9718gw8=; b=e83LBAdvSZ/QzF2ivvf+VmSZ3wLgzGke5ccXkpJYVSozegJqT5sjbAjwGzj4v8a9MV /Pf9yxSc8SW8E2otTY4JUME7dpfnjqhsozK0XrG2dY+4wekHtNe1TOyc+68NkuDQwRnw I60CWCARyKv7IiYlNljRrL+ccvPBMSbEPLPLu7/lgLidxRGRPxr7Qbs44TApp/Eo9hdk 2TKdSRg7q6cLrWUU5QtIGn4RPoSwJoPONgMEvGyita1tN7K6w4+mFUGZNapVAdwK/WIL cOVc8sAYyBiSlQQU/kVKNRV/UWTlk2N1v/mBoN/CYxEDO8PVrs075U2wJACa0K6CXyl/ k+UQ== X-Gm-Message-State: AOAM532G4r4dAYU6Ac+RWD1+gJmZe5Opd1NgJUArcZSO+/NOqgdKr1pH +O1aM2/lgH9GSFZ/ROI94f8hww== X-Google-Smtp-Source: ABdhPJz0DCm314qYSJO3d12j8XktOUREfwthUeQQKZNtRzqebLxqNzzu0tZ+we7pmNupJmyGGAPyiQ== X-Received: by 2002:a05:620a:103c:: with SMTP id a28mr21371qkk.441.1643222202808; Wed, 26 Jan 2022 10:36:42 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id h9sm26226qkp.75.2022.01.26.10.36.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jan 2022 10:36:42 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH v4 4/4] mm/page_table_check: check entries at pmd levels Date: Wed, 26 Jan 2022 18:36:37 +0000 Message-Id: <20220126183637.1840960-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog In-Reply-To: <20220126183637.1840960-1-pasha.tatashin@soleen.com> References: <20220126183637.1840960-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C09FE20037 X-Stat-Signature: 8zdrfkn68braozmj3c1g4un3jaq95ggw Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=NuQmPwn9; dmarc=none; spf=pass (imf03.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-Rspam-User: nil X-HE-Tag: 1643222203-309441 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: syzbot detected a case where the page table counters were not properly updated. syzkaller login: ------------[ cut here ]------------ kernel BUG at mm/page_table_check.c:162! invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 3099 Comm: pasha Not tainted 5.16.0+ #48 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIO4 RIP: 0010:__page_table_check_zero+0x159/0x1a0 Code: 7d 3a b2 ff 45 39 f5 74 2a e8 43 38 b2 ff 4d 85 e4 01 RSP: 0018:ffff888010667418 EFLAGS: 00010293 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000 RDX: ffff88800cea8680 RSI: ffffffff81becaf9 RDI: 0000000003 RBP: ffff888010667450 R08: 0000000000000001 R09: 0000000000 R10: ffffffff81becaab R11: 0000000000000001 R12: ffff888008 R13: 0000000000000001 R14: 0000000000000200 R15: dffffc0000 FS: 0000000000000000(0000) GS:ffff888035e00000(0000) knlG0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffd875cad00 CR3: 00000000094ce000 CR4: 0000000000 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000 Call Trace: free_pcp_prepare+0x3be/0xaa0 free_unref_page+0x1c/0x650 ? trace_hardirqs_on+0x6a/0x1d0 free_compound_page+0xec/0x130 free_transhuge_page+0x1be/0x260 __put_compound_page+0x90/0xd0 release_pages+0x54c/0x1060 ? filemap_remove_folio+0x161/0x210 ? lock_downgrade+0x720/0x720 ? __put_page+0x150/0x150 ? filemap_free_folio+0x164/0x350 __pagevec_release+0x7c/0x110 shmem_undo_range+0x85e/0x1250 ... The repro involved having a huge page that is split due to uprobe event temporarily replacing one of the pages in the huge page. Later the huge page was combined again, but the counters were off, as the PTE level was not properly updated. Make sure that when PMD is cleared and prior to freeing the level the PTEs are updated. Fixes: df4e817b7108 ("mm: page table check") Signed-off-by: Pasha Tatashin Acked-by: David Rientjes --- include/linux/page_table_check.h | 18 ++++++++++++++++++ mm/khugepaged.c | 3 +++ mm/page_table_check.c | 21 +++++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/include/linux/page_table_check.h b/include/linux/page_table_check.h index 38cace1da7b6..e88bbe37727b 100644 --- a/include/linux/page_table_check.h +++ b/include/linux/page_table_check.h @@ -26,6 +26,8 @@ void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud); +void __page_table_check_pmd_clear_full(struct mm_struct *mm, unsigned long addr, + pmd_t pmd); static inline void page_table_check_alloc(struct page *page, unsigned int order) { @@ -100,6 +102,16 @@ static inline void page_table_check_pud_set(struct mm_struct *mm, __page_table_check_pud_set(mm, addr, pudp, pud); } +static inline void page_table_check_pmd_clear_full(struct mm_struct *mm, + unsigned long addr, + pmd_t pmd) +{ + if (static_branch_likely(&page_table_check_disabled)) + return; + + __page_table_check_pmd_clear_full(mm, addr, pmd); +} + #else static inline void page_table_check_alloc(struct page *page, unsigned int order) @@ -143,5 +155,11 @@ static inline void page_table_check_pud_set(struct mm_struct *mm, { } +static inline void page_table_check_pmd_clear_full(struct mm_struct *mm, + unsigned long addr, + pmd_t pmd) +{ +} + #endif /* CONFIG_PAGE_TABLE_CHECK */ #endif /* __LINUX_PAGE_TABLE_CHECK_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 30e59e4af272..d84977c6dc0d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -1422,10 +1423,12 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v spinlock_t *ptl; pmd_t pmd; + mmap_assert_write_locked(mm); ptl = pmd_lock(vma->vm_mm, pmdp); pmd = pmdp_collapse_flush(vma, addr, pmdp); spin_unlock(ptl); mm_dec_nr_ptes(mm); + page_table_check_pmd_clear_full(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); } diff --git a/mm/page_table_check.c b/mm/page_table_check.c index c61d7ebe13b1..251f95a808b4 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -247,3 +247,24 @@ void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, } } EXPORT_SYMBOL(__page_table_check_pud_set); + +void __page_table_check_pmd_clear_full(struct mm_struct *mm, unsigned long addr, + pmd_t pmd) +{ + if (&init_mm == mm) + return; + + if (!pmd_bad(pmd) && !pmd_leaf(pmd)) { + pte_t *ptep = pte_offset_map(&pmd, addr); + unsigned long i; + + pte_unmap(ptep); + for (i = 0; i < PTRS_PER_PTE; i++) { + __page_table_check_pte_clear(mm, addr, *ptep); + addr += PAGE_SIZE; + ptep++; + } + } else { + __page_table_check_pmd_clear(mm, addr, pmd); + } +}