From patchwork Tue Sep 14 19:25:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Linus_L=C3=BCssing?= X-Patchwork-Id: 12494519 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8754C433EF for ; Tue, 14 Sep 2021 19:33:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D32F761175 for ; Tue, 14 Sep 2021 19:33:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233206AbhINTe5 (ORCPT ); Tue, 14 Sep 2021 15:34:57 -0400 Received: from mail.aperture-lab.de ([116.203.183.178]:44902 "EHLO mail.aperture-lab.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230061AbhINTey (ORCPT ); Tue, 14 Sep 2021 15:34:54 -0400 X-Greylist: delayed 482 seconds by postgrey-1.27 at vger.kernel.org; Tue, 14 Sep 2021 15:34:53 EDT Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 173B841012; Tue, 14 Sep 2021 21:25:33 +0200 (CEST) From: =?utf-8?q?Linus_L=C3=BCssing?= To: Kalle Valo , Felix Fietkau , Sujith Manoharan , ath9k-devel@qca.qualcomm.com Cc: linux-wireless@vger.kernel.org, "David S . Miller" , Jakub Kicinski , "John W . Linville" , Felix Fietkau , Simon Wunderlich , Sven Eckelmann , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, =?utf-8?q?Linus_L=C3=BCssing?= , =?utf-8?q?Linus_L?= =?utf-8?q?=C3=BCssing?= Subject: [PATCH 2/3] ath9k: Fix potential interrupt storm on queue reset Date: Tue, 14 Sep 2021 21:25:14 +0200 Message-Id: <20210914192515.9273-3-linus.luessing@c0d3.blue> In-Reply-To: <20210914192515.9273-1-linus.luessing@c0d3.blue> References: <20210914192515.9273-1-linus.luessing@c0d3.blue> MIME-Version: 1.0 X-Last-TLS-Session-Version: TLSv1.3 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Linus Lüssing In tests with two Lima boards from 8devices (QCA4531 based) on OpenWrt 19.07 we could force a silent restart of a device with no serial output when we were sending a high amount of UDP traffic (iperf3 at 80 MBit/s in both directions from external hosts, saturating the wifi and causing a load of about 4.5 to 6) and were then triggering an ath9k_queue_reset(). Further debugging showed that the restart was caused by the ath79 watchdog. With disabled watchdog we could observe that the device was constantly going into ath_isr() interrupt handler and was returning early after the ATH_OP_HW_RESET flag test, without clearing any interrupts. Even though ath9k_queue_reset() calls ath9k_hw_kill_interrupts(). With JTAG we could observe the following race condition: 1) ath9k_queue_reset() ... -> ath9k_hw_kill_interrupts() -> set_bit(ATH_OP_HW_RESET, &common->op_flags); ... <- returns 2) ath9k_tasklet() ... -> ath9k_hw_resume_interrupts() ... <- returns 3) loops around: ... handle_int() -> ath_isr() ... -> if (test_bit(ATH_OP_HW_RESET, &common->op_flags)) return IRQ_HANDLED; x) ath_reset_internal(): => never reached <= And in ath_isr() we would typically see the following interrupts / interrupt causes: * status: 0x00111030 or 0x00110030 * async_cause: 2 (AR_INTR_MAC_IPQ) * sync_cause: 0 So the ath9k_tasklet() reenables the ath9k interrupts through ath9k_hw_resume_interrupts() which ath9k_queue_reset() had just disabled. And ath_isr() then keeps firing because it returns IRQ_HANDLED without actually clearing the interrupt. To fix this IRQ storm also clear/disable the interrupts again when we are in reset state. Cc: Sven Eckelmann Cc: Simon Wunderlich Cc: Linus Lüssing Fixes: 872b5d814f99 ("ath9k: do not access hardware on IRQs during reset") Signed-off-by: Linus Lüssing --- drivers/net/wireless/ath/ath9k/main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/net/wireless/ath/ath9k/main.c b/drivers/net/wireless/ath/ath9k/main.c index 139831539da3..98090e40e1cf 100644 --- a/drivers/net/wireless/ath/ath9k/main.c +++ b/drivers/net/wireless/ath/ath9k/main.c @@ -533,8 +533,10 @@ irqreturn_t ath_isr(int irq, void *dev) ath9k_debug_sync_cause(sc, sync_cause); status &= ah->imask; /* discard unasked-for bits */ - if (test_bit(ATH_OP_HW_RESET, &common->op_flags)) + if (test_bit(ATH_OP_HW_RESET, &common->op_flags)) { + ath9k_hw_kill_interrupts(sc->sc_ah); return IRQ_HANDLED; + } /* * If there are no status bits set, then this interrupt was not