| Message ID | 20221117201440.21183-2-gerhard@engleder-embedded.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | tsnep: Throttle irq, rotten pkts, RX buffer alloc and ethtool_get_channels() | expand |
| Context | Check | Description |
|---|---|---|
| netdev/tree_selection | success | Clearly marked for net-next |
| netdev/fixes_present | success | Fixes tag not required for -next series |
| netdev/subject_prefix | success | Link |
| netdev/cover_letter | success | Series has a cover letter |
| netdev/patch_count | success | Link |
| netdev/header_inline | success | No static functions without inline keyword in header files |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/cc_maintainers | success | CCed 6 of 6 maintainers |
| netdev/build_clang | success | Errors and warnings before: 0 this patch: 0 |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Signed-off-by tag matches author and committer |
| netdev/check_selftest | success | No net selftest shell script |
| netdev/verify_fixes | success | No Fixes tag |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 43 lines checked |
| netdev/kdoc | success | Errors and warnings before: 0 this patch: 0 |
| netdev/source_inline | success | Was 0 now: 0 |
On Thu, Nov 17, 2022 at 09:14:37PM +0100, Gerhard Engleder wrote: > Without interrupt throttling, iperf server mode generates a CPU load of > 100% (A53 1.2GHz). Also the throughput suffers with less than 900Mbit/s > on a 1Gbit/s link. The reason is a high interrupt load with interrupts > every ~20us. Not my area of expertise, but is NAPI working correctly? It should be that you disable interrupts while NAPI is polling, and only re-enable interrupts when polling has stopped. If you are receiving at near line rate at 100% load, i would of thought that NAPI would be polling most of the time and interrupts would be mostly disabled? Interrupt coalescence makes a lot of sense thought, so the patch itself is useful. Andrew
On 17.11.22 21:33, Andrew Lunn wrote: > On Thu, Nov 17, 2022 at 09:14:37PM +0100, Gerhard Engleder wrote: >> Without interrupt throttling, iperf server mode generates a CPU load of >> 100% (A53 1.2GHz). Also the throughput suffers with less than 900Mbit/s >> on a 1Gbit/s link. The reason is a high interrupt load with interrupts >> every ~20us. > > Not my area of expertise, but is NAPI working correctly? It should be > that you disable interrupts while NAPI is polling, and only re-enable > interrupts when polling has stopped. If you are receiving at near line > rate at 100% load, i would of thought that NAPI would be polling most > of the time and interrupts would be mostly disabled? I disable and re-enable interrupts in the driver, but I don't know exactly under which conditions interrupts will be disabled mostly. At least during my XDP tests with minimum size packets the interrupt rate dropped significantly. Here the CPU had also 100% load but packet rate was much higher with ~880000 pkt/s. I compared the performance and the interrupt rate with the macb driver on the same CPU and the results were similar. > Interrupt coalescence makes a lot of sense thought, so the patch > itself is useful. Thanks! Gerhard
On Thu, 17 Nov 2022 21:14:37 +0100 Gerhard Engleder wrote: > Without interrupt throttling, iperf server mode generates a CPU load of > 100% (A53 1.2GHz). Also the throughput suffers with less than 900Mbit/s > on a 1Gbit/s link. The reason is a high interrupt load with interrupts > every ~20us. > > Reduce interrupt load by throttling of interrupts. Interrupt delay is > configured statically to 64us. For iperf server mode the CPU load is > significantly reduced to ~20% and the throughput reaches the maximum of > 941MBit/s. Interrupts are generated every ~140us. User should be able to control these settings, please implement ethtool::get_coalesce / set_coalesce.
On 19.11.22 02:24, Jakub Kicinski wrote: > On Thu, 17 Nov 2022 21:14:37 +0100 Gerhard Engleder wrote: >> Without interrupt throttling, iperf server mode generates a CPU load of >> 100% (A53 1.2GHz). Also the throughput suffers with less than 900Mbit/s >> on a 1Gbit/s link. The reason is a high interrupt load with interrupts >> every ~20us. >> >> Reduce interrupt load by throttling of interrupts. Interrupt delay is >> configured statically to 64us. For iperf server mode the CPU load is >> significantly reduced to ~20% and the throughput reaches the maximum of >> 941MBit/s. Interrupts are generated every ~140us. > > User should be able to control these settings, please implement > ethtool::get_coalesce / set_coalesce. I will implement it.
diff --git a/drivers/net/ethernet/engleder/tsnep_hw.h b/drivers/net/ethernet/engleder/tsnep_hw.h index 315dada75323..55e1caf193a6 100644 --- a/drivers/net/ethernet/engleder/tsnep_hw.h +++ b/drivers/net/ethernet/engleder/tsnep_hw.h @@ -48,6 +48,13 @@ #define ECM_COUNTER_LOW 0x0028 #define ECM_COUNTER_HIGH 0x002C +/* interrupt delay */ +#define ECM_INT_DELAY 0x0030 +#define ECM_INT_DELAY_MASK 0xF0 +#define ECM_INT_DELAY_SHIFT 4 +#define ECM_INT_DELAY_BASE_US 16 +#define ECM_INT_DELAY_OFFSET 1 + /* control and status */ #define ECM_STATUS 0x0080 #define ECM_LINK_MODE_OFF 0x01000000 diff --git a/drivers/net/ethernet/engleder/tsnep_main.c b/drivers/net/ethernet/engleder/tsnep_main.c index 48fb391951dd..a99320e03279 100644 --- a/drivers/net/ethernet/engleder/tsnep_main.c +++ b/drivers/net/ethernet/engleder/tsnep_main.c @@ -39,6 +39,8 @@ #endif #define DMA_ADDR_LOW(dma_addr) ((u32)((dma_addr) & 0xFFFFFFFF)) +#define INT_DELAY (64 / ECM_INT_DELAY_BASE_US) + static void tsnep_enable_irq(struct tsnep_adapter *adapter, u32 mask) { iowrite32(mask, adapter->addr + ECM_INT_ENABLE); @@ -1298,6 +1300,7 @@ static int tsnep_phy_init(struct tsnep_adapter *adapter) static int tsnep_queue_init(struct tsnep_adapter *adapter, int queue_count) { u32 irq_mask = ECM_INT_TX_0 | ECM_INT_RX_0; + u8 irq_delay = (INT_DELAY << ECM_INT_DELAY_SHIFT) & ECM_INT_DELAY_MASK; char name[8]; int i; int retval; @@ -1316,6 +1319,7 @@ static int tsnep_queue_init(struct tsnep_adapter *adapter, int queue_count) adapter->queue[0].tx = &adapter->tx[0]; adapter->queue[0].rx = &adapter->rx[0]; adapter->queue[0].irq_mask = irq_mask; + iowrite8(irq_delay, adapter->addr + ECM_INT_DELAY); adapter->netdev->irq = adapter->queue[0].irq; @@ -1336,6 +1340,8 @@ static int tsnep_queue_init(struct tsnep_adapter *adapter, int queue_count) adapter->queue[i].rx = &adapter->rx[i]; adapter->queue[i].irq_mask = irq_mask << (ECM_INT_TXRX_SHIFT * i); + iowrite8(irq_delay, adapter->addr + ECM_INT_DELAY + + ECM_INT_DELAY_OFFSET * i); } return 0;
Without interrupt throttling, iperf server mode generates a CPU load of 100% (A53 1.2GHz). Also the throughput suffers with less than 900Mbit/s on a 1Gbit/s link. The reason is a high interrupt load with interrupts every ~20us. Reduce interrupt load by throttling of interrupts. Interrupt delay is configured statically to 64us. For iperf server mode the CPU load is significantly reduced to ~20% and the throughput reaches the maximum of 941MBit/s. Interrupts are generated every ~140us. Signed-off-by: Gerhard Engleder <gerhard@engleder-embedded.com> --- drivers/net/ethernet/engleder/tsnep_hw.h | 7 +++++++ drivers/net/ethernet/engleder/tsnep_main.c | 6 ++++++ 2 files changed, 13 insertions(+)