From patchwork Tue Nov 16 22:00:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12623233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD87BC433F5 for ; Tue, 16 Nov 2021 22:02:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 75B766140A for ; Tue, 16 Nov 2021 22:02:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 75B766140A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 449416B0074; Tue, 16 Nov 2021 17:00:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F8A96B007B; Tue, 16 Nov 2021 17:00:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29AC46B007D; Tue, 16 Nov 2021 17:00:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id 1DCF76B0074 for ; Tue, 16 Nov 2021 17:00:57 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D6A0787709 for ; Tue, 16 Nov 2021 22:00:46 +0000 (UTC) X-FDA: 78816163776.01.E324349 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) by imf24.hostedemail.com (Postfix) with ESMTP id 753AFB000812 for ; Tue, 16 Nov 2021 22:00:46 +0000 (UTC) Received: by mail-qk1-f169.google.com with SMTP id q64so441761qkd.5 for ; Tue, 16 Nov 2021 14:00:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Ze7s4dFasnQ7mCcq4UJB+UXGJMvUv9ROGRjXSi8JeCc=; b=PYE0eOTn57jJrGuOBttcs67wGW7i0vcFv3xF+yOX31TfP6tpTd1zRJXTDQjZFdiPPD RrbZ/Z5k8B83x9p7iF0+V4MirVepNc6MCeQVHFNr7ZSNWFFsMh5lsWFR+mDev7MrNwXG 4VukV/ioB5jiT5SqOz9JcgvLzDN6KZGkDTMzI8NZgsrVY0iK92vAiLe0uOwRZGPDPkQk AOi/JGr/vMcPWLEhLCPeXsSXlzNwTbi6bzi6sur61FQyPiizJr5GNXzAjzakt5AxXNs8 Qg1I5qwpsOzWzxT2lKQrUShMtDVPVBuBX9FqQwalrmppV+gKuPLzIUj1bOzxtg1OsSoj bt4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ze7s4dFasnQ7mCcq4UJB+UXGJMvUv9ROGRjXSi8JeCc=; b=kUYqeiD5dKHlUuAOW5NkpPgZtSVGvADDrGLN4BLPvDx3HLOpgobRkj7xo0S+HdexkB FIj2wx6zQtr0khqS0l55YXTU/unC9BjI6u38kRqI2/VKsukJfBUirq7YCdZVG4390KkI s6yVKPoAbjccZ+958C7UVqVlszWAsQtp7r+I/tAY2vuVjv3EvFM8/Kf5l1gQXsP0zWDo +C8pt+VQGpeLX39C4PG2c1AnWAxHAaiSJ2zXqCQaaQ23yyt7w3j9uc3QWe4v6blwxgTU 5ZSwDpz3RYkm6UBuyUCY4WkojLOxDNvZIZ48ppBKC9+0bAlJzrFG4fZO+CK7H6cRsna0 FHfw== X-Gm-Message-State: AOAM530EC4B/lvdTQEj3A1DZyGq9/tRWrSqwl0wvgULl+o9vQy57MIkN fI2Y88YYTuDWuW6gJw0GBKQAkQ== X-Google-Smtp-Source: ABdhPJzL/avgjc1rIMxoXxUMQHZqqX1CzOhqmE/cIRTCc4PO7/fmsSAXcnMtZuv8or6XmwAWEJf+dQ== X-Received: by 2002:a05:620a:4ca:: with SMTP id 10mr9271903qks.194.1637100045836; Tue, 16 Nov 2021 14:00:45 -0800 (PST) Received: from soleen.c.googlers.com.com (189.216.85.34.bc.googleusercontent.com. [34.85.216.189]) by smtp.gmail.com with ESMTPSA id i11sm2975655qko.116.2021.11.16.14.00.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 14:00:45 -0800 (PST) From: Pasha Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, akpm@linux-foundation.org, rientjes@google.com, pjt@google.com, weixugc@google.com, gthelen@google.com, mingo@redhat.com, corbet@lwn.net, will@kernel.org, rppt@kernel.org, keescook@chromium.org, tglx@linutronix.de, peterz@infradead.org, masahiroy@kernel.org, samitolvanen@google.com, dave.hansen@linux.intel.com, x86@kernel.org, frederic@kernel.org, hpa@zytor.com, aneesh.kumar@linux.ibm.com Subject: [RFC 3/3] x86: mm: add x86_64 support for page table check Date: Tue, 16 Nov 2021 22:00:38 +0000 Message-Id: <20211116220038.116484-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211116220038.116484-1-pasha.tatashin@soleen.com> References: <20211116220038.116484-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 753AFB000812 X-Stat-Signature: xq7n3c4oqzzr9dkrenii9edp1nxpym5b Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=PYE0eOTn; dmarc=none; spf=pass (imf24.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com X-HE-Tag: 1637100046-259213 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add page table check hooks into routines that modify user page tables. Signed-off-by: Pasha Tatashin --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 27 +++++++++++++++++++++++++-- 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b1d4b481fcdd..9d28f2ac85ff 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -104,6 +104,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_DEBUG_PAGEALLOC + select ARCH_SUPPORTS_PAGE_TABLE_CHECK if X86_64 select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_SUPPORTS_LTO_CLANG diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 448cd01eb3ec..46f0112f0303 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -26,6 +26,7 @@ #include #include #include +#include extern pgd_t early_top_pgt[PTRS_PER_PGD]; bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd); @@ -1006,18 +1007,21 @@ static inline pud_t native_local_pudp_get_and_clear(pud_t *pudp) static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { + page_table_check_pte_set(mm, addr, ptep, pte); set_pte(ptep, pte); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd) { + page_table_check_pmd_set(mm, addr, pmdp, pmd); set_pmd(pmdp, pmd); } static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, pud_t *pudp, pud_t pud) { + page_table_check_pud_set(mm, addr, pudp, pud); native_set_pud(pudp, pud); } @@ -1048,6 +1052,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { pte_t pte = native_ptep_get_and_clear(ptep); + page_table_check_pte_clear(mm, addr, pte); return pte; } @@ -1063,12 +1068,21 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, * care about updates and native needs no locking */ pte = native_local_ptep_get_and_clear(ptep); + page_table_check_pte_clear(mm, addr, pte); } else { pte = ptep_get_and_clear(mm, addr, ptep); } return pte; } +#define __HAVE_ARCH_PTEP_CLEAR +static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep) +{ + page_table_check_pte_clear(mm, addr, *ptep); + pte_clear(mm, addr, ptep); +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1109,14 +1123,22 @@ static inline int pmd_write(pmd_t pmd) static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { - return native_pmdp_get_and_clear(pmdp); + pmd_t pmd = native_pmdp_get_and_clear(pmdp); + + page_table_check_pmd_clear(mm, addr, pmd); + + return pmd; } #define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pud_t *pudp) { - return native_pudp_get_and_clear(pudp); + pud_t pud = native_pudp_get_and_clear(pudp); + + page_table_check_pud_clear(mm, addr, pud); + + return pud; } #define __HAVE_ARCH_PMDP_SET_WRPROTECT @@ -1137,6 +1159,7 @@ static inline int pud_write(pud_t pud) static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { + page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); if (IS_ENABLED(CONFIG_SMP)) { return xchg(pmdp, pmd); } else {