| Message ID | 20221203003606.6838-25-rick.p.edgecombe@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Shadow stacks for userspace | expand |
On Fri, Dec 02, 2022 at 04:35:51PM -0800, Rick Edgecombe wrote: > The x86 Control-flow Enforcement Technology (CET) feature includes a new > type of memory called shadow stack. This shadow stack memory has some > unusual properties, which requires some core mm changes to function > properly. > > One sharp edge is that PTEs that are both Write=0 and Dirty=1 are > treated as shadow by the CPU, but this combination used to be created by > the kernel on x86. Previous patches have changed the kernel to now avoid > creating these PTEs unless they are for shadow stack memory. In case any > missed corners of the kernel are still creating PTEs like this for > non-shadow stack memory, and to catch any re-introductions of the logic, > warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow > stack VMAs when they are being zapped. This won't catch transient cases > but should have decent coverage. It will be compiled out when shadow > stack is not configured. > > In order to check if a pte is shadow stack in core mm code, add default > implmentations for pte_shstk() and pmd_shstk(). > > Tested-by: Pengfei Xu <pengfei.xu@intel.com> > Tested-by: John Allen <john.allen@amd.com> > Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com> Reviewed-by: Kees Cook <keescook@chromium.org>
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 945d58681a87..519a266ace10 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -129,6 +129,7 @@ static inline bool pte_dirty(pte_t pte) return pte_flags(pte) & _PAGE_DIRTY_BITS; } +#define pte_shstk pte_shstk static inline bool pte_shstk(pte_t pte) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) @@ -147,6 +148,7 @@ static inline bool pmd_dirty(pmd_t pmd) return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } +#define pmd_shstk pmd_shstk static inline bool pmd_shstk(pmd_t pmd) { if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b4a9d9936463..34aa57941d57 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -531,6 +531,20 @@ static inline pte_t pte_mkwrite_shstk(pte_t pte) } #endif +#ifndef pte_shstk +static inline bool pte_shstk(pte_t pte) +{ + return false; +} +#endif + +#ifndef pmd_shstk +static inline bool pmd_shstk(pmd_t pte) +{ + return false; +} +#endif + #ifndef pte_clear_savedwrite #define pte_clear_savedwrite pte_wrprotect #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b6294c4ad471..64d81aa97a61 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1656,6 +1656,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, */ orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pmd_shstk(orig_pmd)); tlb_remove_pmd_tlb_entry(tlb, pmd, addr); if (vma_is_special_huge(vma)) { if (arch_needs_pgtable_deposit()) diff --git a/mm/memory.c b/mm/memory.c index c02b6421241d..bae62b2d6696 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1437,6 +1437,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pte_shstk(ptent)); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent);