From patchwork Thu Jan 20 04:25:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12718246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79BFAC4332F for ; Thu, 20 Jan 2022 04:25:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE1516B00C5; Wed, 19 Jan 2022 23:25:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B43726B00C4; Wed, 19 Jan 2022 23:25:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96E296B00C5; Wed, 19 Jan 2022 23:25:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id 878F16B00C3 for ; Wed, 19 Jan 2022 23:25:19 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3D1358170961 for ; Thu, 20 Jan 2022 04:25:19 +0000 (UTC) X-FDA: 79049375958.22.0F271C8 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf22.hostedemail.com (Postfix) with ESMTP id D15ECC0013 for ; Thu, 20 Jan 2022 04:25:18 +0000 (UTC) Received: by mail-qt1-f174.google.com with SMTP id c3so2553022qtc.8 for ; Wed, 19 Jan 2022 20:25:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=nKZbvnKqUABkXpOqTg5mpojiJ4GqEoBKOOiPWM73Qxw=; b=bu7EWjQ32HnOd5DIeVyrxkiwHfYyZeU1n3WqXyfs6V8TzykoyKOhzXQUF5Xj35f3td vJ91pwx58PU7M7jxX2L7kJlm3pjn9WI6OhAlG5I+jBj/mKTjIs4F8byCHLNodvmqPI23 KDga4e2ajDKFSN+ZzIs6fs6gtUJ527vE3zNBvOC/Jm6AAYkgtH8HNrd88igosq9JGtHK 8MTKLhi13hLvpsrhKUwTl8Q/JpgX1qSYMFIZo8XR0kL8fjGiFiGf0p/vK8obLPk+CPm3 cE0aQEFlpuCQ6RDom4qL0xWd1opvo7KHBpFQOeDaD2j0ddMVRnkE1Pdo15RLGxD+k7ap +4Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nKZbvnKqUABkXpOqTg5mpojiJ4GqEoBKOOiPWM73Qxw=; b=Ra71XQo2AZj+FpNp3BP1UFTf3KGBE8fr7TpbcCIw1gS6BW6HavjtX6YB6MDJTzzVLG 3RVVBOc34972zgRUcqseBrgUh1yTqHnMcoR+oIiADY7UVLis0BXfiURaQB+2Cy3Ed3Iy Mn+nbS7GYd5w52ILNZYqjai6GJ6fk3IU44kJzQ5yPYJcR0Hx+vK03w0HQyo5cwPLe8N+ W2A07aPvePJhvZ6b4KM5GSpS978RbfBw9UyyAKMYG6ViYPssk18DQ8u6Vzggdnf0AYMv IzEzT38ID7pf3nOB3kze00/QMHqpxFNqpi/5RU0zhrXgosrsjj+ChrUv+/uxeRWE28WN mg/w== X-Gm-Message-State: AOAM530hetSQreGn/nqm342rr31My0DgwtlA7dnysT6/sbtjW9YdlfNV CRnh/XCcU2BF/fUDNQRBWfY25g== X-Google-Smtp-Source: ABdhPJwDDA5NIRNSeOLvTEcxsAy5nszcqa0qbDh0l+wfx933/17zCvV7fpoerVnAqcsbC4bItVnryw== X-Received: by 2002:ac8:574e:: with SMTP id 14mr12632481qtx.626.1642652718132; Wed, 19 Jan 2022 20:25:18 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id n22sm927228qka.46.2022.01.19.20.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 20:25:17 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, will@kernel.org, rppt@kernel.org, dave.hansen@linux.intel.com, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com, hughd@google.com, ziy@nvidia.com, anshuman.khandual@arm.com Subject: [PATCH 2/3] mm/page_table_check: check entries at pud and pmd levels Date: Thu, 20 Jan 2022 04:25:12 +0000 Message-Id: <20220120042513.1648831-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.703.g22d0c6ccf7-goog In-Reply-To: <20220120042513.1648831-1-pasha.tatashin@soleen.com> References: <20220120042513.1648831-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D15ECC0013 X-Stat-Signature: ckm98f4t8c7o3qjmpzkfoz7b7h6wpwrz Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=bu7EWjQ3; spf=pass (imf22.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.174 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam07 X-HE-Tag: 1642652718-498452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: syzbot detected a case where the page table counters were not properly updated. syzkaller login: ------------[ cut here ]------------ kernel BUG at mm/page_table_check.c:162! invalid opcode: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 3099 Comm: pasha Not tainted 5.16.0+ #48 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIO4 RIP: 0010:__page_table_check_zero+0x159/0x1a0 Code: 7d 3a b2 ff 45 39 f5 74 2a e8 43 38 b2 ff 4d 85 e4 01 RSP: 0018:ffff888010667418 EFLAGS: 00010293 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000 RDX: ffff88800cea8680 RSI: ffffffff81becaf9 RDI: 0000000003 RBP: ffff888010667450 R08: 0000000000000001 R09: 0000000000 R10: ffffffff81becaab R11: 0000000000000001 R12: ffff888008 R13: 0000000000000001 R14: 0000000000000200 R15: dffffc0000 FS: 0000000000000000(0000) GS:ffff888035e00000(0000) knlG0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffd875cad00 CR3: 00000000094ce000 CR4: 0000000000 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000 Call Trace: free_pcp_prepare+0x3be/0xaa0 free_unref_page+0x1c/0x650 ? trace_hardirqs_on+0x6a/0x1d0 free_compound_page+0xec/0x130 free_transhuge_page+0x1be/0x260 __put_compound_page+0x90/0xd0 release_pages+0x54c/0x1060 ? filemap_remove_folio+0x161/0x210 ? lock_downgrade+0x720/0x720 ? __put_page+0x150/0x150 ? filemap_free_folio+0x164/0x350 __pagevec_release+0x7c/0x110 shmem_undo_range+0x85e/0x1250 ... The repro involved having a huge page that is split due to uprobe event temporarily replacing one of the pages in the huge page. Later the huge page was combined again, but the counters were off, as the PTE level was not properly updated. Make sure that not only huge page but also small pages are updated when a new entry is set or cleared. Fixes: df4e817b7108 ("mm: page table check") Signed-off-by: Pasha Tatashin --- mm/page_table_check.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 7504e7caa2a1..2341ac382cd5 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -145,6 +145,40 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, } } +static void pte_clear_level(struct mm_struct *mm, unsigned long addr, + pte_t *ptep) +{ + unsigned long i; + + for (i = 0; i < PTRS_PER_PTE; i++) { + __page_table_check_pte_clear(mm, addr, *ptep); + addr += PAGE_SIZE; + ptep++; + } +} + +static void pmd_clear_level(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp) +{ + unsigned long i; + + for (i = 0; i < PTRS_PER_PMD; i++) { + pmd_t old_pmd = *pmdp; + + if (pmd_user_accessible_page(old_pmd)) { + page_table_check_clear(mm, addr, pmd_pfn(old_pmd), + PMD_PAGE_SIZE >> PAGE_SHIFT); + } else if (!pmd_bad(old_pmd) && !pmd_leaf(old_pmd)) { + pte_t *ptep = pte_offset_map(&old_pmd, addr); + + pte_clear_level(mm, addr, ptep); + pte_unmap(ptep); + } + addr += PMD_PAGE_SIZE; + pmdp++; + } +} + /* * page is on free list, or is being allocated, verify that counters are zeroes * crash if they are not. @@ -186,6 +220,11 @@ void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, if (pmd_user_accessible_page(pmd)) { page_table_check_clear(mm, addr, pmd_pfn(pmd), PMD_PAGE_SIZE >> PAGE_SHIFT); + } else if (!pmd_bad(pmd) && !pmd_leaf(pmd)) { + pte_t *ptep = pte_offset_map(&pmd, addr); + + pte_clear_level(mm, addr, ptep); + pte_unmap(ptep); } } EXPORT_SYMBOL(__page_table_check_pmd_clear); @@ -199,6 +238,10 @@ void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr, if (pud_user_accessible_page(pud)) { page_table_check_clear(mm, addr, pud_pfn(pud), PUD_PAGE_SIZE >> PAGE_SHIFT); + } else if (!pud_bad(pud) && !pud_leaf(pud)) { + pmd_t *pmdp = pmd_offset(&pud, addr); + + pmd_clear_level(mm, addr, pmdp); } } EXPORT_SYMBOL(__page_table_check_pud_clear); @@ -237,6 +280,11 @@ void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr, if (pmd_user_accessible_page(old_pmd)) { page_table_check_clear(mm, addr, pmd_pfn(old_pmd), PMD_PAGE_SIZE >> PAGE_SHIFT); + } else if (!pmd_bad(old_pmd) && !pmd_leaf(old_pmd)) { + pte_t *ptep = pte_offset_map(&old_pmd, addr); + + pte_clear_level(mm, addr, ptep); + pte_unmap(ptep); } if (pmd_user_accessible_page(pmd)) { @@ -259,6 +307,10 @@ void __page_table_check_pud_set(struct mm_struct *mm, unsigned long addr, if (pud_user_accessible_page(old_pud)) { page_table_check_clear(mm, addr, pud_pfn(old_pud), PUD_PAGE_SIZE >> PAGE_SHIFT); + } else if (!pud_bad(old_pud) && !pud_leaf(old_pud)) { + pmd_t *pmdp = pmd_offset(&old_pud, addr); + + pmd_clear_level(mm, addr, pmdp); } if (pud_user_accessible_page(pud)) {