From patchwork Thu Jul 4 04:31:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13723195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B314C3065C for ; Thu, 4 Jul 2024 04:32:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 881F56B00A6; Thu, 4 Jul 2024 00:32:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 808A86B00A7; Thu, 4 Jul 2024 00:32:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6604A6B00A8; Thu, 4 Jul 2024 00:32:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3EAAE6B00A6 for ; Thu, 4 Jul 2024 00:32:15 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id ECB80160FC2 for ; Thu, 4 Jul 2024 04:32:14 +0000 (UTC) X-FDA: 82300798188.12.E74BE91 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf23.hostedemail.com (Postfix) with ESMTP id B6117140005 for ; Thu, 4 Jul 2024 04:32:12 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=rGzyLD7L; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=8nHTN6MV; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=rGzyLD7L; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=8nHTN6MV; spf=pass (imf23.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720067508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fKLC6fxewOrLf0i98Rd25YFQam44kkf2M+DOutpQYdQ=; b=Nfskna/yHITUE/HjyApQr/HZFHRjCiX+m+iKGk3JbxuzHzdFm5nXIbkHhyRjMrCHxAUsK1 8VJjQ3ilP1Ru4wqUhYomyTojE5w9gkBnHchOtrUeKyS3WDj2uY2IOpOr8gIcGUg1JNtR0k V0CchcVx2Bo+Go2NTPd/Bh3ft7Recws= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720067508; a=rsa-sha256; cv=none; b=Nk9fVQM3lE49FbPPHs+iVuKg81LWTyJq5dRDheUNszGoqQXvDyeAmE/F3W9s9NZz8E+Hh4 Wd198GuOm3Pdk2zMt+RG7Dsc7MV9yePiRdQZMeP8nWBc3szq2v0D03MVWAZHpAK8jbivdX QhHqiaGi7ivpEKoySeGq+n+/PEzrnME= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=rGzyLD7L; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=8nHTN6MV; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=rGzyLD7L; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=8nHTN6MV; spf=pass (imf23.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 717E31FCF8; Thu, 4 Jul 2024 04:32:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1720067531; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKLC6fxewOrLf0i98Rd25YFQam44kkf2M+DOutpQYdQ=; b=rGzyLD7LkYyMAvmIw4xFUDH+buX9d/II8oeFi+WRfNzUSU6mBH1Xd+8r3gYOEPGL9dYDCy IlLqdjWxRy5mraTCwA+0ut4zZnng0xAJIvDVvopn+3dMXHd6APSdUZmCaNvfWzCqwkjiNn u/St7F29ojB/6wl0e44bDnr1D1/7TB8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1720067531; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKLC6fxewOrLf0i98Rd25YFQam44kkf2M+DOutpQYdQ=; b=8nHTN6MVi8gMxqJWpwXDGmQUc1P5i9EMQ+Rwyw40Sbfz/xQvdeEiZOiulE9osEkcCpeY4k 3YaniTexv02tM6Cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1720067531; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKLC6fxewOrLf0i98Rd25YFQam44kkf2M+DOutpQYdQ=; b=rGzyLD7LkYyMAvmIw4xFUDH+buX9d/II8oeFi+WRfNzUSU6mBH1Xd+8r3gYOEPGL9dYDCy IlLqdjWxRy5mraTCwA+0ut4zZnng0xAJIvDVvopn+3dMXHd6APSdUZmCaNvfWzCqwkjiNn u/St7F29ojB/6wl0e44bDnr1D1/7TB8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1720067531; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKLC6fxewOrLf0i98Rd25YFQam44kkf2M+DOutpQYdQ=; b=8nHTN6MVi8gMxqJWpwXDGmQUc1P5i9EMQ+Rwyw40Sbfz/xQvdeEiZOiulE9osEkcCpeY4k 3YaniTexv02tM6Cg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4E2E313889; Thu, 4 Jul 2024 04:32:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id UP40DsolhmbnDAAAD6G6ig (envelope-from ); Thu, 04 Jul 2024 04:32:10 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Peter Xu , Muchun Song , David Hildenbrand , SeongJae Park , Miaohe Lin , Michal Hocko , Matthew Wilcox , Christophe Leroy , Oscar Salvador Subject: [PATCH 17/45] mm: Implement pud-version for pud_mkinvalid and pudp_establish Date: Thu, 4 Jul 2024 06:31:04 +0200 Message-ID: <20240704043132.28501-18-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240704043132.28501-1-osalvador@suse.de> References: <20240704043132.28501-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B6117140005 X-Stat-Signature: eqgfwtg7fyytzs1x86ihni4kyh4fypkq X-HE-Tag: 1720067532-236303 X-HE-Meta: U2FsdGVkX1888D+iyzwkpV90QtXXNjTlF9/WmQO0tV0S2SWTQftH4I3TG8VoCKOMxPqN+WCnMAxD4wJaMgJIJqlTIB/JILzVIdNpu0KHHb5vLB8hPrbSG5ENkmB/o8quzwWsorvUsxb+KTTIruyUhgQPMGxH5vKY9l8w+YlqpNHF6Xu4iWWD3tpK3usf/QITGp51q8mhwSNr92ExMhJzHjIjJfF+c9KI9vaJP0e/6LZwONWVH1XiChcixYiijH+wWNoRjOBNNks/go60IReKNEbzGwFt+tVeJONCsKxGyY7CCcq6LsNeCwnXv6FKkoFuPGGHD0TcO/sv2CFmtHDYQW2KoTyhl/S/RJuVzIYNnTXlf6CtFo5Kcn1Av9Qe4fvJm5VDy0JJ7YvCjIGRMHxCpzBsotiudXFGDMBrTV7sl966qk406FYg6xSX49KADllIf0BdDkFbdBR19Lkuh3Vh5w5g8nUQqHaf52fjmWne+R6wPDHvnugwe/i9ALwkMez0kh3VZWoWfAG14oZnQVyAwyXPZXYY0qHAQSbhwc+RlpBHUVy/QTyuKEwOuQ2RfC4NusPoTUB2uRrGm0dyzN5Eh3xOeU3oBz+HPuPuJtarDnCwDp8+Fau5Ka51lQLQiAZmdKIOW5plQ2dVnyYXHhvv8vVvaOMSG3kato+ieLbUfW6mW3Z3RyrI5jyPkIe8L9EQSg9Bylc2NN7xUPXhVLDbsPNQyfATWWBnXp4gwzVVNbvKNbviJhjcmfV6DyFAt0/9XM53ZIT/naBuUULsy+1DX3gMAgA+HpqNN8BKkW7BagxdHKEo0fw7UMndbGA2A7rLJAoEYAftbN7Wt4Csrlmy4d0hBrIE1qwaDgyqmHw4U59X/hduQN5NFHXT6QAySOWChm95iYXKMoaZ44q/45Tp/H5moVOz0pWzfrgudxFIjwoR0vPQyOL9fwzLuBfftJB9DHc1UvmxRah4Q5Q2dXR CTDV27uj WvTpU1wQVYBrMJfTJntg7OyAwiZlO63TVzbUxCTW3EbjHWtAvZF4HSZryp30lL8yph3sb9xmt5QoUSZhql3sCBcJAS84psORzgGbhCw6bHf4aoymecLviKkRmuZtNo2GWOamtbemNnFQEOOdg1LRIJ9h/IBgiT7POEQTVXinooQmJL67N9P4Xv7DW0270U70Nj11VPqhELCyQkuuQyGr4lxrg+dtg+rcCM1f0ahj94a2YV8/8Mbnk1NM3BAZvvYfVJ38HyhdX2RsNYb8QHpmNsAoMxFcnusLHTXaR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: HugeTLB pages will be handled on pud level as well, so we need to implement pud-versions of pud_mkinvalid and pudp_establish. Signed-off-by: Oscar Salvador --- arch/arm64/include/asm/pgtable.h | 11 ++++++ arch/loongarch/include/asm/pgtable.h | 8 ++++ arch/mips/include/asm/pgtable.h | 7 ++++ arch/powerpc/include/asm/book3s/64/pgtable.h | 7 +++- arch/powerpc/mm/book3s64/pgtable.c | 15 ++++++- arch/riscv/include/asm/pgtable.h | 15 +++++++ arch/x86/include/asm/pgtable.h | 31 ++++++++++++++- include/linux/pgtable.h | 41 +++++++++++++++++++- mm/pgtable-generic.c | 21 ++++++++++ 9 files changed, 150 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 936ed3a915a3..5e26e63b1012 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -595,6 +595,7 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_write(pud) pte_write(pud_pte(pud)) #define pud_mkhuge(pud) (__pud(pud_val(pud) & ~PUD_TABLE_BIT)) +#define pud_mkinvalid(pud) pte_pud(pte_mkinvalid(pud_pte(pud))) #define __pud_to_phys(pud) __pte_to_phys(pud_pte(pud)) #define __phys_to_pud_val(phys) __phys_to_pte_val(phys) @@ -1344,6 +1345,16 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +#ifdef CONFIG_HUGETLB_PAGE +#define pudp_establish pudp_establish +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + return __pud(xchg_relaxed(&pud_val(*pudp), pud_val(pud))); +} +#endif + /* * Encode and decode a swap entry: * bits 0-1: present (must be zero) diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index 161dd6e10479..cf73c2f2da2c 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -581,6 +581,14 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) return pmd; } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + pud_val(pud) |= _PAGE_PRESENT_INVALID; + pud_val(pud) &= ~(_PAGE_PRESENT | _PAGE_VALID | _PAGE_DIRTY | _PAGE_PROTNONE); + + return pud; +} + /* * The generic version pmdp_huge_get_and_clear uses a version of pmd_clear() with a * different prototype. diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index c29a551eb0ca..390a2f022147 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -736,6 +736,13 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) return pmd; } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + pud_val(pud) &= ~(_PAGE_PRESENT | _PAGE_VALID | _PAGE_DIRTY); + + return pud; +} + /* * The generic version pmdp_huge_get_and_clear uses a version of pmd_clear() with a * different prototype. diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index fa4bb8d6356f..f95ac2a87548 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1085,7 +1085,8 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd) #define pmd_mksoft_dirty(pmd) pte_pmd(pte_mksoft_dirty(pmd_pte(pmd))) #define pmd_clear_soft_dirty(pmd) pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd))) -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#if defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) || defined(CONFIG_HUGETLB_PAGE) +#define pud_swp_soft_dirty(pud) pte_swp_soft_dirty(pud_pte(pud)) #define pmd_swp_mksoft_dirty(pmd) pte_pmd(pte_swp_mksoft_dirty(pmd_pte(pmd))) #define pmd_swp_soft_dirty(pmd) pte_swp_soft_dirty(pmd_pte(pmd)) #define pmd_swp_clear_soft_dirty(pmd) pte_pmd(pte_swp_clear_soft_dirty(pmd_pte(pmd))) @@ -1386,6 +1387,10 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_INVALIDATE +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; extern int pmd_move_must_withdraw(struct spinlock *new_pmd_ptl, diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index f4d8d3c40e5c..1b6ae7898f99 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -37,7 +37,7 @@ EXPORT_SYMBOL(__pmd_frag_nr); unsigned long __pmd_frag_size_shift; EXPORT_SYMBOL(__pmd_frag_size_shift); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES /* * This is called when relaxing access to a hugepage. It's also called in the page * fault path when we don't hit any of the major fault cases, ie, a minor @@ -259,7 +259,18 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pud_present(*pudp)); + old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return __pud(old_pud); +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ /* For use by kexec, called with MMU off */ notrace void mmu_cleanup_all(void) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ebfe8faafb79..51600afa203c 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -657,6 +657,11 @@ static inline unsigned long pud_pfn(pud_t pud) return ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return __pud(pud_val(pud) & ~(_PAGE_PRESENT|_PAGE_PROT_NONE)); +} + static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); @@ -804,6 +809,16 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifdef CONFIG_HUGETLB_PAGE +#define pudp_establish pudp_establish +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + return __pud(atomic_long_xchg((atomic_long_t *)pudp, pud_val(pud))); +} +#endif + /* * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that * are !pte_none() && !pte_present(). diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 640edc31962f..572458a106e9 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -783,6 +783,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -1353,6 +1359,23 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, return pud; } +#ifndef pudp_establish +#define pudp_establish pudp_establish +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + + WRITE_ONCE(*pudp, pud); + return old; + } +} +#endif + #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1389,7 +1412,6 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); - /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. @@ -1541,7 +1563,12 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte) return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY); } -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +#if defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) || defined(CONFIG_HUGETLB_PAGE) +static inline int pud_swp_soft_dirty(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SWP_SOFT_DIRTY; +} + static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_SWP_SOFT_DIRTY); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a2e2ebb93f21..458e3cbc96b2 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -956,6 +956,11 @@ extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); #endif +#ifndef __HAVE_ARCH_PUDP_INVALIDATE +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); +#endif + #ifndef __HAVE_ARCH_PMDP_INVALIDATE_AD /* @@ -976,6 +981,26 @@ extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); #endif +#ifndef __HAVE_ARCH_PUDP_INVALIDATE_AD + +/* + * pudp_invalidate_ad() invalidates the PMD while changing a hugetlb mapping in + * the page tables. This function is similar to pudp_invalidate(), but should + * only be used if the access and dirty bits would not be cleared by the software + * in the new PUD value. The function ensures that hardware changes of the access + * and dirty bits updates would not be lost. + * + * Doing so can allow in certain architectures to avoid a TLB flush in most + * cases. Yet, another TLB flush might be necessary later if the PUD update + * itself requires such flush (e.g., if protection was set to be stricter). Yet, + * even when a TLB flush is needed because of the update, the caller may be able + * to batch these TLB flushing operations, so fewer TLB flush operations are + * needed. + */ +extern pud_t pudp_invalidate_ad(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp); +#endif + #ifndef __HAVE_ARCH_PTE_SAME static inline int pte_same(pte_t pte_a, pte_t pte_b) { @@ -1406,7 +1431,16 @@ static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) #endif #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY -#ifndef CONFIG_ARCH_ENABLE_THP_MIGRATION +static inline int pud_soft_dirty(pud_t pud) +{ + return 0; +} +#if !defined(CONFIG_ARCH_ENABLE_THP_MIGRATION) && !defined(CONFIG_HUGETLB_PAGE) +static inline int pud_swp_soft_dirty(pud_t pud) +{ + return 0; +} + static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd) { return pmd; @@ -1487,6 +1521,11 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) { return pmd; } + +static inline int pud_swp_soft_dirty(pud_t pud) +{ + return 0; +} #endif #ifndef __HAVE_PFNMAP_TRACKING diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711a..e11ad8663903 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -194,6 +194,27 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) } #endif +#ifndef __HAVE_ARCH_PUDP_INVALIDATE +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + + flush_pud_tlb_range(vma, address, address + PUD_SIZE); + return old; +} +#endif + +#ifndef __HAVE_ARCH_PUDP_INVALIDATE_AD +pud_t pudp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + return pudp_invalidate(vma, address, pudp); +} +#endif + #ifndef __HAVE_ARCH_PMDP_INVALIDATE pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp)