| Message ID | 20200827163205.23096-1-tony.luck@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RFD] x86/mce: Make sure to send SIGBUS even after losing the race to poison a page | expand |
Hi, On Thu, Aug 27, 2020 at 09:32:05AM -0700, Tony Luck wrote: > For discussion ... I'm 100% sure the patch below is the wrong way to > fix this ... for one thing it doesn't provide the virtual address of > the error to the user signal handler. For another it just looks like > a hack. I'm just not sure whether to delve deep into the > memory_failure() path to make sure the signal is sent to the current > process in the SRAR case. Or just to do a better job at an error return > value and make the X86 specific code send the signal with the address. I think that both options could work. memory_failure() now sends SIGBUS only to current process for SRAR. Or relying on kill_me_now() is fine. > > I've also got a feeling that this issue has been discussed before, but > can't remember how that turned out. > > First few paragraphs describe the problem (and I think are OK). Things > go off the rails with the fix. > > -Tony > > X86 hardware may provide two indications that a page has poison. First > the memory controller that detects the failed ECC check may log a UCNA > (uncorrected no action) signature in one machine check bank and signal > the OS with a CMCI (corrected machine check interrupt ... historical name > did not get updated for this case). Next the processor core may log a > SRAR (software recoverable action required) signature in another bank > and signal with #MC. > > The #MC used to win that race and the page was taken offline and SIGBUS > sent to the task. > > Changes to how Linux processes machine checks now mean that: > a) Linux will offline the page based on the UCNA siganture in the > CMCI handler. > b) The machine check handler defers processing using task_work_add() which > can happen after the CMCI is processed. > > memory_failure() avoids races with multiple callers reporting the same > page with an atomic test and set operation to mark the page as poisoned. > > The net result of all of the above is that when a task consumes poison > the page is taken offline by the UCNA/CMCI path, and the SRAR/#MC path > takes an early return without sending a SIGBUS. Yes, in this case with the above changes we will fail to send SIGBUS for SRAR, which is not good. We should always send SIGBUS for SRAR whether the page is already poisoned or not. > > Fix by changing memory_failure() to return -EEXIST in the case where > the page is already poisoned and make the machine check code path check > for this error and force a SIGBUS. Simply returning -EEXIST maybe breaks the behavior for SRAO, so could you add checking MF_ACTION_REQUIRED in this path? Thanks, Naoya Horiguchi > > Note that -EBUSY might have been a more logical error code, but that is > already used for many other error cases from memory_failure(). > > Signed-off-by: Tony Luck <tony.luck@intel.com> > --- > arch/x86/kernel/cpu/mce/core.c | 7 +++++-- > mm/memory-failure.c | 2 +- > 2 files changed, 6 insertions(+), 3 deletions(-) > > diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c > index fb6b5f64f7e6..8515809e0472 100644 > --- a/arch/x86/kernel/cpu/mce/core.c > +++ b/arch/x86/kernel/cpu/mce/core.c > @@ -1182,18 +1182,21 @@ static void kill_me_maybe(struct callback_head *cb) > { > struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me); > int flags = MF_ACTION_REQUIRED; > + int ret; > > pr_err("Uncorrected hardware memory error in user-access at %llx", p->mce_addr); > > if (!p->mce_ripv) > flags |= MF_MUST_KILL; > > - if (!memory_failure(p->mce_addr >> PAGE_SHIFT, flags)) { > + ret = memory_failure(p->mce_addr >> PAGE_SHIFT, flags); > + if (!ret) { > set_mce_nospec(p->mce_addr >> PAGE_SHIFT, p->mce_whole_page); > return; > } > > - pr_err("Memory error not recovered"); > + if (ret != -EEXIST) > + pr_err("Memory error not recovered"); > kill_me_now(cb); > } > > diff --git a/mm/memory-failure.c b/mm/memory-failure.c > index f1aa6433f404..e0486c4e0130 100644 > --- a/mm/memory-failure.c > +++ b/mm/memory-failure.c > @@ -1298,7 +1298,7 @@ int memory_failure(unsigned long pfn, int flags) > if (TestSetPageHWPoison(p)) { > pr_err("Memory failure: %#lx: already hardware poisoned\n", > pfn); > - return 0; > + return -EEXIST; > } > > orig_head = hpage = compound_head(p); > -- > 2.21.1 >
On Thu, Aug 27, 2020 at 09:32:05AM -0700, Tony Luck wrote: > The #MC used to win that race and the page was taken offline and SIGBUS > sent to the task. > > Changes to how Linux processes machine checks now mean that: > a) Linux will offline the page based on the UCNA siganture in the > CMCI handler. Let's see if that logic makes sense: if #MC offlines the page and sends SIGBUS but CMCI only offlines the page, isn't it only logical for the CMCI to *also* send the SIGBUS too, after having offlined the page? I.e., both should do the proper and full recovery action. Just sayin...
> Let's see if that logic makes sense: if #MC offlines the page and sends > SIGBUS but CMCI only offlines the page, isn't it only logical for the > CMCI to *also* send the SIGBUS too, after having offlined the page? > > I.e., both should do the proper and full recovery action. Just sayin... It made sense, and seemed to explain an issue I was seeing, when I wrote it. But some stress testing of that patch showed that it introduces some problems and instability. Without the patch I can inject 10,000 errors and have every one of them complete correctly (process gets a SIGBUS with the address of the error). With my patch around 0.4% of injections fail to provide the address to the SIGBUS handler, worse the test gets a fatal error every 600-700 injections. So, I'm abandoning that patch. -Tony
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index fb6b5f64f7e6..8515809e0472 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -1182,18 +1182,21 @@ static void kill_me_maybe(struct callback_head *cb) { struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me); int flags = MF_ACTION_REQUIRED; + int ret; pr_err("Uncorrected hardware memory error in user-access at %llx", p->mce_addr); if (!p->mce_ripv) flags |= MF_MUST_KILL; - if (!memory_failure(p->mce_addr >> PAGE_SHIFT, flags)) { + ret = memory_failure(p->mce_addr >> PAGE_SHIFT, flags); + if (!ret) { set_mce_nospec(p->mce_addr >> PAGE_SHIFT, p->mce_whole_page); return; } - pr_err("Memory error not recovered"); + if (ret != -EEXIST) + pr_err("Memory error not recovered"); kill_me_now(cb); } diff --git a/mm/memory-failure.c b/mm/memory-failure.c index f1aa6433f404..e0486c4e0130 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1298,7 +1298,7 @@ int memory_failure(unsigned long pfn, int flags) if (TestSetPageHWPoison(p)) { pr_err("Memory failure: %#lx: already hardware poisoned\n", pfn); - return 0; + return -EEXIST; } orig_head = hpage = compound_head(p);
For discussion ... I'm 100% sure the patch below is the wrong way to fix this ... for one thing it doesn't provide the virtual address of the error to the user signal handler. For another it just looks like a hack. I'm just not sure whether to delve deep into the memory_failure() path to make sure the signal is sent to the current process in the SRAR case. Or just to do a better job at an error return value and make the X86 specific code send the signal with the address. I've also got a feeling that this issue has been discussed before, but can't remember how that turned out. First few paragraphs describe the problem (and I think are OK). Things go off the rails with the fix. -Tony X86 hardware may provide two indications that a page has poison. First the memory controller that detects the failed ECC check may log a UCNA (uncorrected no action) signature in one machine check bank and signal the OS with a CMCI (corrected machine check interrupt ... historical name did not get updated for this case). Next the processor core may log a SRAR (software recoverable action required) signature in another bank and signal with #MC. The #MC used to win that race and the page was taken offline and SIGBUS sent to the task. Changes to how Linux processes machine checks now mean that: a) Linux will offline the page based on the UCNA siganture in the CMCI handler. b) The machine check handler defers processing using task_work_add() which can happen after the CMCI is processed. memory_failure() avoids races with multiple callers reporting the same page with an atomic test and set operation to mark the page as poisoned. The net result of all of the above is that when a task consumes poison the page is taken offline by the UCNA/CMCI path, and the SRAR/#MC path takes an early return without sending a SIGBUS. Fix by changing memory_failure() to return -EEXIST in the case where the page is already poisoned and make the machine check code path check for this error and force a SIGBUS. Note that -EBUSY might have been a more logical error code, but that is already used for many other error cases from memory_failure(). Signed-off-by: Tony Luck <tony.luck@intel.com> --- arch/x86/kernel/cpu/mce/core.c | 7 +++++-- mm/memory-failure.c | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-)