From patchwork Sat Dec 4 18:23:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12656747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1FC1C433F5 for ; Sat, 4 Dec 2021 18:25:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3326C6B0083; Sat, 4 Dec 2021 13:23:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B9266B0085; Sat, 4 Dec 2021 13:23:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10C756B0087; Sat, 4 Dec 2021 13:23:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id F40D86B0083 for ; Sat, 4 Dec 2021 13:23:31 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id ADCE98249980 for ; Sat, 4 Dec 2021 18:23:21 +0000 (UTC) X-FDA: 78880934202.20.8B375C6 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf06.hostedemail.com (Postfix) with ESMTP id 57D60801A88D for ; Sat, 4 Dec 2021 18:23:21 +0000 (UTC) Received: by mail-qv1-f48.google.com with SMTP id i13so6108718qvm.1 for ; Sat, 04 Dec 2021 10:23:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=JjYKEJcTyq7gxxwF/qsfUV8xV15YsFb8MoAiHyojDyw=; b=fxFPKGKGmjFVeeQhXbIUZYdRSXvIWE/CzCjUnvZrLiETxDBowMyU/cmr4fjdBfMxoj IfE5nyLBXTp10mSWnOmKjr+8PSJqHahFmmhLQrO6bV0tnTKYZaOXITNLJF8UmTUu6i+5 QTKH/xSpLDBkEHwemKUTiLxwrQKxUxbedRLWGYFeFz9wLJVtCqQebPorPKgigVsj/qzo MRUVGDlGzUzfbvxMqGrmPLD3GiwhD3ewCPTtPSDVRFDlaMiyXwFHFcrhlw6BDYDTIqhH MqnHoVvmqaL/uFqOIsLIW17hJHEyl36jkrqr6vkJzjrHfuc1chqtRQTCFW8GcD6apMDW M2pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JjYKEJcTyq7gxxwF/qsfUV8xV15YsFb8MoAiHyojDyw=; b=H6EPMS5JPwCO6CnMi5u/4L7ElJl7TM5maUzMPQ1QFv4MRoNrN4slBmd8EZcR59e2I2 t9/ewGjYyND3cs3UG/+1Ru2VV9eqIFihljHXFWVKzDORGL2vGyqNRZUvjnWkAKagGMQG u2jKok14CSb8f+7P6ElLuBLMu5UH6CDZq8CUFrGHyZtg03l3yLh/Lbwj3IsC4I+pIyVC 3I1QUmj5Vimjyd2KJT6j9GdOOqKJekd5R1NyCVCQYXVpFFcpWny7xjMhw6482xyzaVy1 b6wTfZeXwu9/ITktofYFyV9jYs4EIPueQxoEcLBiPDsyjG76o1NoniSuTOuMHVwFQ0pk CZMg== X-Gm-Message-State: AOAM533J6wo3hkIJBKdVb/siJtTNOGKFXLwPCv3J7/Qa4BHIzZuhvlip lmwSQFIGCcss/FnS9zPTLRX5mg== X-Google-Smtp-Source: ABdhPJxIq+kzgXQQCk00YqMBJ4oav+C29EWpXkfE4hU1sZ72bgR5ZiRVt5i/2I3KRZFKaZcjkTvslA== X-Received: by 2002:ad4:4452:: with SMTP id l18mr26855725qvt.8.1638642200754; Sat, 04 Dec 2021 10:23:20 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id a24sm4394728qtp.95.2021.12.04.10.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 04 Dec 2021 10:23:20 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, corbet@lwn.net, will@kernel.org, rppt@kernel.org, keescook@chromium.org, tglx@linutronix.de, peterz@infradead.org, masahiroy@kernel.org, samitolvanen@google.com, dave.hansen@linux.intel.com, x86@kernel.org, frederic@kernel.org, hpa@zytor.com, aneesh.kumar@linux.ibm.com, jirislaby@kernel.org, songmuchun@bytedance.com, qydwhotmail@gmail.com Subject: [PATCH v2 4/4] x86: mm: add x86_64 support for page table check Date: Sat, 4 Dec 2021 18:23:14 +0000 Message-Id: <20211204182314.1470076-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.1.400.ga245620fadb-goog In-Reply-To: <20211204182314.1470076-1-pasha.tatashin@soleen.com> References: <20211204182314.1470076-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 57D60801A88D Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=fxFPKGKG; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.219.48 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none X-Rspamd-Server: rspam04 X-Stat-Signature: 575amsta719y34kwog8xsyczzyrkpp84 X-HE-Tag: 1638642201-433097 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add page table check hooks into routines that modify user page tables. Signed-off-by: Pasha Tatashin --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5e16393d9988..7636ea400a71 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -104,6 +104,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_DEBUG_PAGEALLOC + select ARCH_SUPPORTS_PAGE_TABLE_CHECK if X86_64 select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_SUPPORTS_LTO_CLANG diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 448cd01eb3ec..ae34614b7e8d 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -26,6 +26,7 @@ #include #include #include +#include extern pgd_t early_top_pgt[PTRS_PER_PGD]; bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd); @@ -1006,18 +1007,21 @@ static inline pud_t native_local_pudp_get_and_clear(pud_t *pudp) static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { + page_table_check_pte_set(mm, addr, ptep, pte); set_pte(ptep, pte); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) { + page_table_check_pmd_set(mm, addr, pmdp, pmd); set_pmd(pmdp, pmd); } static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud) { + page_table_check_pud_set(mm, addr, pudp, pud); native_set_pud(pudp, pud); } @@ -1048,6 +1052,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { pte_t pte = native_ptep_get_and_clear(ptep); + page_table_check_pte_clear(mm, addr, pte); return pte; } @@ -1063,12 +1068,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, * care about updates and native needs no locking */ pte = native_local_ptep_get_and_clear(ptep); + page_table_check_pte_clear(mm, addr, pte); } else { pte = ptep_get_and_clear(mm, addr, ptep); } return pte; } +#define __HAVE_ARCH_PTEP_CLEAR +static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep) +{ + if (IS_ENABLED(CONFIG_PAGE_TABLE_CHECK)) + ptep_get_and_clear(mm, addr, ptep); + else + pte_clear(mm, addr, ptep); +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1109,14 +1125,22 @@ static inline int pmd_write(pmd_t pmd) static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { - return native_pmdp_get_and_clear(pmdp); + pmd_t pmd = native_pmdp_get_and_clear(pmdp); + + page_table_check_pmd_clear(mm, addr, pmd); + + return pmd; } #define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pud_t *pudp) { - return native_pudp_get_and_clear(pudp); + pud_t pud = native_pudp_get_and_clear(pudp); + + page_table_check_pud_clear(mm, addr, pud); + + return pud; } #define __HAVE_ARCH_PMDP_SET_WRPROTECT @@ -1137,6 +1161,7 @@ static inline int pud_write(pud_t pud) static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { + page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); if (IS_ENABLED(CONFIG_SMP)) { return xchg(pmdp, pmd); } else {