| Message ID | 20250320082057.622983-1-pandoh@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Rate limit AER logs | expand |
On Thu, Mar 20, 2025 at 01:20:50AM -0700, Jon Pan-Doh wrote:
> Several OCP members
The fact that some users are part of a borderline illegal purchasing
cartel should not in any way matter for a linux patch submission.
Please work on your commit messages.
On Thu, Mar 20, 2025 at 01:20:50AM -0700, Jon Pan-Doh wrote: > Proposal > ======== > > When using native AER, spammy devices can flood kernel logs with AER errors > and slow/stall execution. Add per-device per-error-severity ratelimits > for more robust error logging. Allow userspace to configure ratelimits > via sysfs knobs. > > Motivation > ========== > > Several OCP members have issues with inconsistent PCIe error handling, > exacerbated at datacenter scale (myriad of devices). > OCP HW/Fault Management subproject set out to solve this by > standardizing industry: > > - PCIe error handling best practices > - Fault Management/RAS (incl. PCIe errors) > > Exposing PCIe errors/debug info in-band for a userspace daemon (e.g. > rasdaemon) to collect/pass on to repairability services is part of the > roadmap. For the series: Reported-by: Sargun Dhillon <sargun@meta.com> Acked-by: Paul E. McKenney <paulmck@kernel.org> And thank you! I did throw together a quick hack to get things going internally on our fleet, but this series is way better. Thanx, Paul > Background > ========== > > AER error spam has been observed many times, both publicly (e.g. [1], [2], > [3]) and privately. While it usually occurs with correctable errors, it can > happen with uncorrectable errors (e.g. during new HW bringup). > > There have been previous attempts to add ratelimits to AER logs ([4], > [5]). The most recent attempt[5] has many similarities with the proposed > approach. > > Patch organization > ================== > 1-4 AER logging cleanup > 5-7 Ratelimits and sysfs knobs > > Outstanding work > ================ > Cleanup: > - Consolidate aer_print_error() and pci_print_error() path > > Roadmap: > - IRQ ratelimiting > > v4: > - Fix bug where trace not emitted with malformed aer_err_info > - Extend ratelimit to malformed aer_err_info > - Update commit messages with patch motivation > - Squash AER sysfs filename change (Patch 8) > v3: > - Ratelimit aer_print_port_info() (drop Patch 1) > - Add ratelimit enable toggle > - Move trace outside of ratelimit > - Split log level (Patch 2) into two > - More descriptive documentation/sysfs naming > v2: > - Rebased on top of pci/aer (6.14.rc-1) > - Split series into log and IRQ ratelimits (defer patch 5) > - Dropped patch 8 (Move AER sysfs) > - Added log level cleanup patch[6] from Karolina's series > - Fixed bug where dpc errors didn't increment counters > - "X callbacks suppressed" message on ratelimit release -> immediately > - Separate documentation into own patch > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=215027 > [2] https://bugzilla.kernel.org/show_bug.cgi?id=201517 > [3] https://bugzilla.kernel.org/show_bug.cgi?id=196183 > [4] https://lore.kernel.org/linux-pci/20230606035442.2886343-2-grundler@chromium.org/ > [5] https://lore.kernel.org/linux-pci/cover.1736341506.git.karolina.stolarek@oracle.com/ > [6] https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.karolina.stolarek@oracle.com/ > > Jon Pan-Doh (5): > PCI/AER: Move AER stat collection out of __aer_print_error() > PCI/AER: Rename struct aer_stats to aer_report > PCI/AER: Introduce ratelimit for error logs > PCI/AER: Add ratelimits to PCI AER Documentation > PCI/AER: Add sysfs attributes for log ratelimits > > Karolina Stolarek (2): > PCI/AER: Check log level once and propagate down > PCI/AER: Make all pci_print_aer() log levels depend on error type > > ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++ > Documentation/PCI/pcieaer-howto.rst | 16 +- > drivers/pci/pci-sysfs.c | 1 + > drivers/pci/pci.h | 4 +- > drivers/pci/pcie/aer.c | 271 +++++++++++++----- > drivers/pci/pcie/dpc.c | 3 +- > include/linux/pci.h | 2 +- > 7 files changed, 260 insertions(+), 71 deletions(-) > rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%) > > -- > 2.49.0.rc1.451.g8f38331e32-goog >
Proposal ======== When using native AER, spammy devices can flood kernel logs with AER errors and slow/stall execution. Add per-device per-error-severity ratelimits for more robust error logging. Allow userspace to configure ratelimits via sysfs knobs. Motivation ========== Several OCP members have issues with inconsistent PCIe error handling, exacerbated at datacenter scale (myriad of devices). OCP HW/Fault Management subproject set out to solve this by standardizing industry: - PCIe error handling best practices - Fault Management/RAS (incl. PCIe errors) Exposing PCIe errors/debug info in-band for a userspace daemon (e.g. rasdaemon) to collect/pass on to repairability services is part of the roadmap. Background ========== AER error spam has been observed many times, both publicly (e.g. [1], [2], [3]) and privately. While it usually occurs with correctable errors, it can happen with uncorrectable errors (e.g. during new HW bringup). There have been previous attempts to add ratelimits to AER logs ([4], [5]). The most recent attempt[5] has many similarities with the proposed approach. Patch organization ================== 1-4 AER logging cleanup 5-7 Ratelimits and sysfs knobs Outstanding work ================ Cleanup: - Consolidate aer_print_error() and pci_print_error() path Roadmap: - IRQ ratelimiting v4: - Fix bug where trace not emitted with malformed aer_err_info - Extend ratelimit to malformed aer_err_info - Update commit messages with patch motivation - Squash AER sysfs filename change (Patch 8) v3: - Ratelimit aer_print_port_info() (drop Patch 1) - Add ratelimit enable toggle - Move trace outside of ratelimit - Split log level (Patch 2) into two - More descriptive documentation/sysfs naming v2: - Rebased on top of pci/aer (6.14.rc-1) - Split series into log and IRQ ratelimits (defer patch 5) - Dropped patch 8 (Move AER sysfs) - Added log level cleanup patch[6] from Karolina's series - Fixed bug where dpc errors didn't increment counters - "X callbacks suppressed" message on ratelimit release -> immediately - Separate documentation into own patch [1] https://bugzilla.kernel.org/show_bug.cgi?id=215027 [2] https://bugzilla.kernel.org/show_bug.cgi?id=201517 [3] https://bugzilla.kernel.org/show_bug.cgi?id=196183 [4] https://lore.kernel.org/linux-pci/20230606035442.2886343-2-grundler@chromium.org/ [5] https://lore.kernel.org/linux-pci/cover.1736341506.git.karolina.stolarek@oracle.com/ [6] https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.karolina.stolarek@oracle.com/ Jon Pan-Doh (5): PCI/AER: Move AER stat collection out of __aer_print_error() PCI/AER: Rename struct aer_stats to aer_report PCI/AER: Introduce ratelimit for error logs PCI/AER: Add ratelimits to PCI AER Documentation PCI/AER: Add sysfs attributes for log ratelimits Karolina Stolarek (2): PCI/AER: Check log level once and propagate down PCI/AER: Make all pci_print_aer() log levels depend on error type ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++ Documentation/PCI/pcieaer-howto.rst | 16 +- drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 4 +- drivers/pci/pcie/aer.c | 271 +++++++++++++----- drivers/pci/pcie/dpc.c | 3 +- include/linux/pci.h | 2 +- 7 files changed, 260 insertions(+), 71 deletions(-) rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%)