| Message ID | 152800340082.17112.1154560126059273408.stgit@dwillia2-desk3.amr.corp.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Sat, Jun 02, 2018 at 10:23:20PM -0700, Dan Williams wrote: > +static inline int set_mce_nospec(unsigned long pfn) > +{ > + int rc; > + > + rc = set_memory_uc((unsigned long) __va(PFN_PHYS(pfn)), 1); You should really do the decoy_addr thing here that I had in mce_unmap_kpfn(). Putting the virtual address of the page you mustn't accidentally prefetch from into a register is a pretty good way to make sure that the processor does do a prefetch. -Tony
On Mon, Jun 4, 2018 at 10:08 AM, Luck, Tony <tony.luck@intel.com> wrote: > On Sat, Jun 02, 2018 at 10:23:20PM -0700, Dan Williams wrote: >> +static inline int set_mce_nospec(unsigned long pfn) >> +{ >> + int rc; >> + >> + rc = set_memory_uc((unsigned long) __va(PFN_PHYS(pfn)), 1); > > You should really do the decoy_addr thing here that I had in mce_unmap_kpfn(). > Putting the virtual address of the page you mustn't accidentally prefetch > from into a register is a pretty good way to make sure that the processor > does do a prefetch. Maybe I'm misreading, but doesn't that make the page completely inaccessible? We still want to read pmem through the driver and the linear mapping with memcpy_mcsafe(). Alternatively I could just drop this patch and setup a private / alias mapping for the pmem driver to use. It seems aliased mappings would be the safer option, but I want to make sure I've comprehended your suggestion correctly?
On Mon, Jun 04, 2018 at 10:39:48AM -0700, Dan Williams wrote: > On Mon, Jun 4, 2018 at 10:08 AM, Luck, Tony <tony.luck@intel.com> wrote: > > On Sat, Jun 02, 2018 at 10:23:20PM -0700, Dan Williams wrote: > >> +static inline int set_mce_nospec(unsigned long pfn) > >> +{ > >> + int rc; > >> + > >> + rc = set_memory_uc((unsigned long) __va(PFN_PHYS(pfn)), 1); > > > > You should really do the decoy_addr thing here that I had in mce_unmap_kpfn(). > > Putting the virtual address of the page you mustn't accidentally prefetch > > from into a register is a pretty good way to make sure that the processor > > does do a prefetch. > > Maybe I'm misreading, but doesn't that make the page completely > inaccessible? We still want to read pmem through the driver and the > linear mapping with memcpy_mcsafe(). Alternatively I could just drop > this patch and setup a private / alias mapping for the pmem driver to > use. It seems aliased mappings would be the safer option, but I want > to make sure I've comprehended your suggestion correctly? I'm OK with the call to set_memory_uc() to make this uncacheable instead of set_memory_np() to make it inaccessible. The problem is how to achieve that. The result of __va(PFN_PHYS(pfn) is the virtual address where the poison page is currently mapped into the kernel. That value gets put into register %rdi to make the call to set_memory_uc() (which goes on to call a bunch of other functions passing the virtual address along the way). Now imagine an impatient super-speculative processor is waiting for some result to decide where to jump next, and picks a path that isn't going to be taken ... out in the weeds somewhere it runs into: movzbl (%rdi), %eax Oops ... now you just read from the address you were trying to avoid. So we log an error. Eventually the speculation gets sorted out and the processor knows not to signal a machine check. But the log is sitting in a machine check bank waiting to cause an overflow if we try to log a second error. The decoy_addr trick in mce_unmap_kpfn() throws in the high bit to the address passed. The set_memory_np() code (and I assume the set_memory_uc()) code ignores it, but it means any stray speculative access won't point at the poison page. -Tony Note: this is *mostly* a problem if the poison is in the first cache line of the page. But you could hit other lines if the instruction you speculatively ran into had the right offset. E.g. to hit the third line: movzbl 128(%rdi), %eax
On Mon, Jun 4, 2018 at 11:08 AM, Luck, Tony <tony.luck@intel.com> wrote: > On Mon, Jun 04, 2018 at 10:39:48AM -0700, Dan Williams wrote: >> On Mon, Jun 4, 2018 at 10:08 AM, Luck, Tony <tony.luck@intel.com> wrote: >> > On Sat, Jun 02, 2018 at 10:23:20PM -0700, Dan Williams wrote: >> >> +static inline int set_mce_nospec(unsigned long pfn) >> >> +{ >> >> + int rc; >> >> + >> >> + rc = set_memory_uc((unsigned long) __va(PFN_PHYS(pfn)), 1); >> > >> > You should really do the decoy_addr thing here that I had in mce_unmap_kpfn(). >> > Putting the virtual address of the page you mustn't accidentally prefetch >> > from into a register is a pretty good way to make sure that the processor >> > does do a prefetch. >> >> Maybe I'm misreading, but doesn't that make the page completely >> inaccessible? We still want to read pmem through the driver and the >> linear mapping with memcpy_mcsafe(). Alternatively I could just drop >> this patch and setup a private / alias mapping for the pmem driver to >> use. It seems aliased mappings would be the safer option, but I want >> to make sure I've comprehended your suggestion correctly? > > I'm OK with the call to set_memory_uc() to make this uncacheable > instead of set_memory_np() to make it inaccessible. > > The problem is how to achieve that. > > The result of __va(PFN_PHYS(pfn) is the virtual address where the poison > page is currently mapped into the kernel. That value gets put into > register %rdi to make the call to set_memory_uc() (which goes on to > call a bunch of other functions passing the virtual address along > the way). > > Now imagine an impatient super-speculative processor is waiting for > some result to decide where to jump next, and picks a path that isn't > going to be taken ... out in the weeds somewhere it runs into: > > movzbl (%rdi), %eax > > Oops ... now you just read from the address you were trying to > avoid. So we log an error. Eventually the speculation gets sorted > out and the processor knows not to signal a machine check. But the > log is sitting in a machine check bank waiting to cause an overflow > if we try to log a second error. > > The decoy_addr trick in mce_unmap_kpfn() throws in the high bit > to the address passed. The set_memory_np() code (and I assume the > set_memory_uc()) code ignores it, but it means any stray speculative > access won't point at the poison page. > > -Tony > > Note: this is *mostly* a problem if the poison is in the first > cache line of the page. But you could hit other lines if the > instruction you speculatively ran into had the right offset. E.g. > to hit the third line: > > movzbl 128(%rdi), %eax Ok, makes sense and I do see now that this decoy resolves to the same physical address once PTE_PFN_MASK is applied when we start messing with page tables. However, set_memory_uc() is currently not prepared for this trick as it specifies the unmasked physical address to reserve_memtype(). ret = reserve_memtype(__pa(addr), __pa(addr) + numpages * PAGE_SIZE, _PAGE_CACHE_MODE_UC_MINUS, NULL); ...compared to set_memory_np() which does not manipulate the memtype tracking. I'll fix up reserve_memtype() and free_memtype() to be prepared for decoy addresses.
diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index bd090367236c..debc1fee1457 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -88,4 +88,33 @@ extern int kernel_set_to_readonly; void set_kernel_text_rw(void); void set_kernel_text_ro(void); +#ifdef CONFIG_X86_64 +/* + * Mark the linear address as UC to disable speculative pre-fetches into + * potentially poisoned memory. + */ +static inline int set_mce_nospec(unsigned long pfn) +{ + int rc; + + rc = set_memory_uc((unsigned long) __va(PFN_PHYS(pfn)), 1); + if (rc) + pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); + return rc; +} +#define set_mce_nospec set_mce_nospec + +/* Restore full speculative operation to the pfn. */ +static inline int clear_mce_nospec(unsigned long pfn) +{ + return set_memory_wb((unsigned long) __va(PFN_PHYS(pfn)), 1); +} +#define clear_mce_nospec clear_mce_nospec +#else +/* + * Few people would run a 32-bit kernel on a machine that supports + * recoverable errors because they have too much memory to boot 32-bit. + */ +#endif + #endif /* _ASM_X86_SET_MEMORY_H */ diff --git a/arch/x86/kernel/cpu/mcheck/mce-internal.h b/arch/x86/kernel/cpu/mcheck/mce-internal.h index 374d1aa66952..ceb67cd5918f 100644 --- a/arch/x86/kernel/cpu/mcheck/mce-internal.h +++ b/arch/x86/kernel/cpu/mcheck/mce-internal.h @@ -113,21 +113,6 @@ static inline void mce_register_injector_chain(struct notifier_block *nb) { } static inline void mce_unregister_injector_chain(struct notifier_block *nb) { } #endif -#ifndef CONFIG_X86_64 -/* - * On 32-bit systems it would be difficult to safely unmap a poison page - * from the kernel 1:1 map because there are no non-canonical addresses that - * we can use to refer to the address without risking a speculative access. - * However, this isn't much of an issue because: - * 1) Few unmappable pages are in the 1:1 map. Most are in HIGHMEM which - * are only mapped into the kernel as needed - * 2) Few people would run a 32-bit kernel on a machine that supports - * recoverable errors because they have too much memory to boot 32-bit. - */ -static inline void mce_unmap_kpfn(unsigned long pfn) {} -#define mce_unmap_kpfn mce_unmap_kpfn -#endif - struct mca_config { bool dont_log_ce; bool cmci_disabled; diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c index 42cf2880d0ed..a0fbf0a8b7e6 100644 --- a/arch/x86/kernel/cpu/mcheck/mce.c +++ b/arch/x86/kernel/cpu/mcheck/mce.c @@ -42,6 +42,7 @@ #include <linux/irq_work.h> #include <linux/export.h> #include <linux/jump_label.h> +#include <linux/set_memory.h> #include <asm/intel-family.h> #include <asm/processor.h> @@ -50,7 +51,6 @@ #include <asm/mce.h> #include <asm/msr.h> #include <asm/reboot.h> -#include <asm/set_memory.h> #include "mce-internal.h" @@ -108,10 +108,6 @@ static struct irq_work mce_irq_work; static void (*quirk_no_way_out)(int bank, struct mce *m, struct pt_regs *regs); -#ifndef mce_unmap_kpfn -static void mce_unmap_kpfn(unsigned long pfn); -#endif - /* * CPU/chipset specific EDAC code can register a notifier call here to print * MCE errors in a human-readable form. @@ -602,7 +598,7 @@ static int srao_decode_notifier(struct notifier_block *nb, unsigned long val, if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { pfn = mce->addr >> PAGE_SHIFT; if (!memory_failure(pfn, 0)) - mce_unmap_kpfn(pfn); + set_mce_nospec(pfn); } return NOTIFY_OK; @@ -1070,38 +1066,10 @@ static int do_memory_failure(struct mce *m) if (ret) pr_err("Memory error not recovered"); else - mce_unmap_kpfn(m->addr >> PAGE_SHIFT); + set_mce_nospec(m->addr >> PAGE_SHIFT); return ret; } -#ifndef mce_unmap_kpfn -static void mce_unmap_kpfn(unsigned long pfn) -{ - unsigned long decoy_addr; - - /* - * Unmap this page from the kernel 1:1 mappings to make sure - * we don't log more errors because of speculative access to - * the page. - * We would like to just call: - * set_memory_np((unsigned long)pfn_to_kaddr(pfn), 1); - * but doing that would radically increase the odds of a - * speculative access to the poison page because we'd have - * the virtual address of the kernel 1:1 mapping sitting - * around in registers. - * Instead we get tricky. We create a non-canonical address - * that looks just like the one we want, but has bit 63 flipped. - * This relies on set_memory_np() not checking whether we passed - * a legal address. - */ - - decoy_addr = (pfn << PAGE_SHIFT) + (PAGE_OFFSET ^ BIT(63)); - - if (set_memory_np(decoy_addr, 1)) - pr_warn("Could not invalidate pfn=0x%lx from 1:1 map\n", pfn); -} -#endif - /* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18. diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index da5178216da5..2a986d282a97 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -17,6 +17,20 @@ static inline int set_memory_x(unsigned long addr, int numpages) { return 0; } static inline int set_memory_nx(unsigned long addr, int numpages) { return 0; } #endif +#ifndef set_mce_nospec +static inline int set_mce_nospec(unsigned long pfn) +{ + return 0; +} +#endif + +#ifndef clear_mce_nospec +static inline int clear_mce_nospec(unsigned long pfn) +{ + return 0; +} +#endif + #ifndef CONFIG_ARCH_HAS_MEM_ENCRYPT static inline int set_memory_encrypted(unsigned long addr, int numpages) {
Currently memory_failure() returns zero if the error was handled. On that result mce_unmap_kpfn() is called to zap the page out of the kernel linear mapping to prevent speculative fetches of potentially poisoned memory. However, in the case of dax mapped devmap pages the page may be in active permanent use by the device driver, so it cannot be unmapped from the kernel. Instead of marking the page not present, marking the page UC should be sufficient for preventing poison from being pre-fetched into the cache. Convert mce_unmap_pfn() to set_mce_nospec() remapping the page as UC, to hide it from speculative accesses. Given that that persistent memory errors can be cleared by the driver, include a facility to restore the page to cacheable operation, clear_mce_nospec(). Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@redhat.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Tony Luck <tony.luck@intel.com> Cc: Borislav Petkov <bp@alien8.de> Cc: <linux-edac@vger.kernel.org> Cc: <x86@kernel.org> Signed-off-by: Dan Williams <dan.j.williams@intel.com> --- arch/x86/include/asm/set_memory.h | 29 ++++++++++++++++++++++ arch/x86/kernel/cpu/mcheck/mce-internal.h | 15 ----------- arch/x86/kernel/cpu/mcheck/mce.c | 38 ++--------------------------- include/linux/set_memory.h | 14 +++++++++++ 4 files changed, 46 insertions(+), 50 deletions(-)