| Message ID | 20220425021442.1.I650b809482e1af8d0156ed88b5dc2677a0711d46@changeid (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | ath10k: skip ath10k_halt during suspend for driver state RESTARTING | expand |
| Context | Check | Description |
|---|---|---|
| netdev/tree_selection | success | Not a local patch |
Abhishek Kumar <kuabhs@chromium.org> writes: > Double free crash is observed when FW recovery(caused by wmi > timeout/crash) is followed by immediate suspend event. The FW recovery > is triggered by ath10k_core_restart() which calls driver clean up via > ath10k_halt(). When the suspend event occurs between the FW recovery, > the restart worker thread is put into frozen state until suspend completes. > The suspend event triggers ath10k_stop() which again triggers ath10k_halt() > The double invocation of ath10k_halt() causes ath10k_htt_rx_free() to be > called twice(Note: ath10k_htt_rx_alloc was not called by restart worker > thread because of its frozen state), causing the crash. > > To fix this, during the suspend flow, skip call to ath10k_halt() in > ath10k_stop() when the current driver state is ATH10K_STATE_RESTARTING. > Also, for driver state ATH10K_STATE_RESTARTING, call > ath10k_wait_for_suspend() in ath10k_stop(). This is because call to > ath10k_wait_for_suspend() is skipped later in > [ath10k_halt() > ath10k_core_stop()] for the driver state > ATH10K_STATE_RESTARTING. > > The frozen restart worker thread will be cancelled during resume when the > device comes out of suspend. > > Below is the crash stack for reference: > > [ 428.469167] ------------[ cut here ]------------ > [ 428.469180] kernel BUG at mm/slub.c:4150! > [ 428.469193] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI > [ 428.469219] Workqueue: events_unbound async_run_entry_fn > [ 428.469230] RIP: 0010:kfree+0x319/0x31b > [ 428.469241] RSP: 0018:ffffa1fac015fc30 EFLAGS: 00010246 > [ 428.469247] RAX: ffffedb10419d108 RBX: ffff8c05262b0000 > [ 428.469252] RDX: ffff8c04a8c07000 RSI: 0000000000000000 > [ 428.469256] RBP: ffffa1fac015fc78 R08: 0000000000000000 > [ 428.469276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 428.469285] Call Trace: > [ 428.469295] ? dma_free_attrs+0x5f/0x7d > [ 428.469320] ath10k_core_stop+0x5b/0x6f > [ 428.469336] ath10k_halt+0x126/0x177 > [ 428.469352] ath10k_stop+0x41/0x7e > [ 428.469387] drv_stop+0x88/0x10e > [ 428.469410] __ieee80211_suspend+0x297/0x411 > [ 428.469441] rdev_suspend+0x6e/0xd0 > [ 428.469462] wiphy_suspend+0xb1/0x105 > [ 428.469483] ? name_show+0x2d/0x2d > [ 428.469490] dpm_run_callback+0x8c/0x126 > [ 428.469511] ? name_show+0x2d/0x2d > [ 428.469517] __device_suspend+0x2e7/0x41b > [ 428.469523] async_suspend+0x1f/0x93 > [ 428.469529] async_run_entry_fn+0x3d/0xd1 > [ 428.469535] process_one_work+0x1b1/0x329 > [ 428.469541] worker_thread+0x213/0x372 > [ 428.469547] kthread+0x150/0x15f > [ 428.469552] ? pr_cont_work+0x58/0x58 > [ 428.469558] ? kthread_blkcg+0x31/0x31 > > Signed-off-by: Abhishek Kumar <kuabhs@chromium.org> > Co-developed-by: Wen Gong <quic_wgong@quicinc.com> > Signed-off-by: Wen Gong <quic_wgong@quicinc.com> Tested-on tag missing, but I can add it if you provide it. https://wireless.wiki.kernel.org/en/users/drivers/ath10k/submittingpatches#tested-on_tag
Thanks Kalle for having a look and adding this on behalf of me. Here is the Tested-on tag, Tested-on: QCA6174 hw3.2 PCI WLAN.RM.4.4.1-00288-QCARMSWPZ-1 Thanks Abhishek On Sun, Apr 24, 2022 at 11:14 PM Kalle Valo <kvalo@kernel.org> wrote: > > Abhishek Kumar <kuabhs@chromium.org> writes: > > > Double free crash is observed when FW recovery(caused by wmi > > timeout/crash) is followed by immediate suspend event. The FW recovery > > is triggered by ath10k_core_restart() which calls driver clean up via > > ath10k_halt(). When the suspend event occurs between the FW recovery, > > the restart worker thread is put into frozen state until suspend completes. > > The suspend event triggers ath10k_stop() which again triggers ath10k_halt() > > The double invocation of ath10k_halt() causes ath10k_htt_rx_free() to be > > called twice(Note: ath10k_htt_rx_alloc was not called by restart worker > > thread because of its frozen state), causing the crash. > > > > To fix this, during the suspend flow, skip call to ath10k_halt() in > > ath10k_stop() when the current driver state is ATH10K_STATE_RESTARTING. > > Also, for driver state ATH10K_STATE_RESTARTING, call > > ath10k_wait_for_suspend() in ath10k_stop(). This is because call to > > ath10k_wait_for_suspend() is skipped later in > > [ath10k_halt() > ath10k_core_stop()] for the driver state > > ATH10K_STATE_RESTARTING. > > > > The frozen restart worker thread will be cancelled during resume when the > > device comes out of suspend. > > > > Below is the crash stack for reference: > > > > [ 428.469167] ------------[ cut here ]------------ > > [ 428.469180] kernel BUG at mm/slub.c:4150! > > [ 428.469193] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI > > [ 428.469219] Workqueue: events_unbound async_run_entry_fn > > [ 428.469230] RIP: 0010:kfree+0x319/0x31b > > [ 428.469241] RSP: 0018:ffffa1fac015fc30 EFLAGS: 00010246 > > [ 428.469247] RAX: ffffedb10419d108 RBX: ffff8c05262b0000 > > [ 428.469252] RDX: ffff8c04a8c07000 RSI: 0000000000000000 > > [ 428.469256] RBP: ffffa1fac015fc78 R08: 0000000000000000 > > [ 428.469276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 428.469285] Call Trace: > > [ 428.469295] ? dma_free_attrs+0x5f/0x7d > > [ 428.469320] ath10k_core_stop+0x5b/0x6f > > [ 428.469336] ath10k_halt+0x126/0x177 > > [ 428.469352] ath10k_stop+0x41/0x7e > > [ 428.469387] drv_stop+0x88/0x10e > > [ 428.469410] __ieee80211_suspend+0x297/0x411 > > [ 428.469441] rdev_suspend+0x6e/0xd0 > > [ 428.469462] wiphy_suspend+0xb1/0x105 > > [ 428.469483] ? name_show+0x2d/0x2d > > [ 428.469490] dpm_run_callback+0x8c/0x126 > > [ 428.469511] ? name_show+0x2d/0x2d > > [ 428.469517] __device_suspend+0x2e7/0x41b > > [ 428.469523] async_suspend+0x1f/0x93 > > [ 428.469529] async_run_entry_fn+0x3d/0xd1 > > [ 428.469535] process_one_work+0x1b1/0x329 > > [ 428.469541] worker_thread+0x213/0x372 > > [ 428.469547] kthread+0x150/0x15f > > [ 428.469552] ? pr_cont_work+0x58/0x58 > > [ 428.469558] ? kthread_blkcg+0x31/0x31 > > > > Signed-off-by: Abhishek Kumar <kuabhs@chromium.org> > > Co-developed-by: Wen Gong <quic_wgong@quicinc.com> > > Signed-off-by: Wen Gong <quic_wgong@quicinc.com> > > Tested-on tag missing, but I can add it if you provide it. > > https://wireless.wiki.kernel.org/en/users/drivers/ath10k/submittingpatches#tested-on_tag > > -- > https://patchwork.kernel.org/project/linux-wireless/list/ > > https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
On Mon, Apr 25, 2022 at 02:15:20AM +0000, Abhishek Kumar wrote: > --- a/drivers/net/wireless/ath/ath10k/mac.c > +++ b/drivers/net/wireless/ath/ath10k/mac.c > @@ -5345,8 +5345,22 @@ static void ath10k_stop(struct ieee80211_hw *hw) > > mutex_lock(&ar->conf_mutex); > if (ar->state != ATH10K_STATE_OFF) { > - if (!ar->hw_rfkill_on) > - ath10k_halt(ar); > + if (!ar->hw_rfkill_on) { > + /* If the current driver state is RESTARTING but not yet > + * fully RESTARTED because of incoming suspend event, > + * then ath11k_halt is already called via s/ath11k/ath10k/ I know ath11k is the hot new thing, but this is ath10k ;) > + * ath10k_core_restart and should not be called here. s/ath10k/ath11k/ > + */ > + if (ar->state != ATH10K_STATE_RESTARTING) > + ath10k_halt(ar); > + else > + /* Suspending here, because when in RESTARTING > + * state, ath11k_core_stop skips s/ath10k/ath11k/ > + * ath10k_wait_for_suspend. > + */ > + ath10k_wait_for_suspend(ar, > + WMI_PDEV_SUSPEND_AND_DISABLE_INTR); > + } > ar->state = ATH10K_STATE_OFF; > } > mutex_unlock(&ar->conf_mutex); Otherwise, I believe this is the right solution to the problem pointed out in the commit message: Reviewed-by: Brian Norris <briannorris@chromium.org>
On Mon, Apr 25, 2022 at 04:11:44PM -0700, Brian Norris wrote: > On Mon, Apr 25, 2022 at 02:15:20AM +0000, Abhishek Kumar wrote: > > --- a/drivers/net/wireless/ath/ath10k/mac.c > > +++ b/drivers/net/wireless/ath/ath10k/mac.c > > @@ -5345,8 +5345,22 @@ static void ath10k_stop(struct ieee80211_hw *hw) ... > > + /* If the current driver state is RESTARTING but not yet > > + * fully RESTARTED because of incoming suspend event, > > + * then ath11k_halt is already called via > > s/ath11k/ath10k/ > > I know ath11k is the hot new thing, but this is ath10k ;) > > > + * ath10k_core_restart and should not be called here. > > s/ath10k/ath11k/ Oh boy, I got *that* backwards! Should be this, obviously: s/ath11k/ath10k/ > > + */ > > + if (ar->state != ATH10K_STATE_RESTARTING) > > + ath10k_halt(ar); > > + else > > + /* Suspending here, because when in RESTARTING > > + * state, ath11k_core_stop skips > > s/ath10k/ath11k/ Same. Brian
On Mon, Apr 25, 2022 at 4:13 PM Brian Norris <briannorris@chromium.org> wrote: > > On Mon, Apr 25, 2022 at 04:11:44PM -0700, Brian Norris wrote: > > On Mon, Apr 25, 2022 at 02:15:20AM +0000, Abhishek Kumar wrote: > > > --- a/drivers/net/wireless/ath/ath10k/mac.c > > > +++ b/drivers/net/wireless/ath/ath10k/mac.c > > > @@ -5345,8 +5345,22 @@ static void ath10k_stop(struct ieee80211_hw *hw) > ... > > > + /* If the current driver state is RESTARTING but not yet > > > + * fully RESTARTED because of incoming suspend event, > > > + * then ath11k_halt is already called via > > > > s/ath11k/ath10k/ > > > > I know ath11k is the hot new thing, but this is ath10k ;) > > > > > + * ath10k_core_restart and should not be called here. > > > > s/ath10k/ath11k/ > > Oh boy, I got *that* backwards! Should be this, obviously: > > s/ath11k/ath10k/ > > > > + */ > > > + if (ar->state != ATH10K_STATE_RESTARTING) > > > + ath10k_halt(ar); > > > + else > > > + /* Suspending here, because when in RESTARTING > > > + * state, ath11k_core_stop skips > > > > s/ath10k/ath11k/ > > Same. Oh!, lately working on ath11k mostly and the muscle memory kicked in ... sorry about this, will fix in next revision. Kale whenever you get time, let me know if the logic looks good to you and if ath11k/ath10k typo fix is the only comment. Thanks Abhishek > > Brian
On 4/24/2022 7:15 PM, Abhishek Kumar wrote: [...snip...] > > Signed-off-by: Abhishek Kumar <kuabhs@chromium.org> Your S-O-B should be last? > Co-developed-by: Wen Gong <quic_wgong@quicinc.com> > Signed-off-by: Wen Gong <quic_wgong@quicinc.com>
On 4/24/2022 7:15 PM, Abhishek Kumar wrote: > Double free crash is observed when FW recovery(caused by wmi > timeout/crash) is followed by immediate suspend event. The FW recovery > is triggered by ath10k_core_restart() which calls driver clean up via > ath10k_halt(). When the suspend event occurs between the FW recovery, > the restart worker thread is put into frozen state until suspend completes. > The suspend event triggers ath10k_stop() which again triggers ath10k_halt() > The double invocation of ath10k_halt() causes ath10k_htt_rx_free() to be > called twice(Note: ath10k_htt_rx_alloc was not called by restart worker > thread because of its frozen state), causing the crash. > > To fix this, during the suspend flow, skip call to ath10k_halt() in > ath10k_stop() when the current driver state is ATH10K_STATE_RESTARTING. > Also, for driver state ATH10K_STATE_RESTARTING, call > ath10k_wait_for_suspend() in ath10k_stop(). This is because call to > ath10k_wait_for_suspend() is skipped later in > [ath10k_halt() > ath10k_core_stop()] for the driver state > ATH10K_STATE_RESTARTING. > > The frozen restart worker thread will be cancelled during resume when the > device comes out of suspend. > > Below is the crash stack for reference: > > [ 428.469167] ------------[ cut here ]------------ > [ 428.469180] kernel BUG at mm/slub.c:4150! > [ 428.469193] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI > [ 428.469219] Workqueue: events_unbound async_run_entry_fn > [ 428.469230] RIP: 0010:kfree+0x319/0x31b > [ 428.469241] RSP: 0018:ffffa1fac015fc30 EFLAGS: 00010246 > [ 428.469247] RAX: ffffedb10419d108 RBX: ffff8c05262b0000 > [ 428.469252] RDX: ffff8c04a8c07000 RSI: 0000000000000000 > [ 428.469256] RBP: ffffa1fac015fc78 R08: 0000000000000000 > [ 428.469276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 428.469285] Call Trace: > [ 428.469295] ? dma_free_attrs+0x5f/0x7d > [ 428.469320] ath10k_core_stop+0x5b/0x6f > [ 428.469336] ath10k_halt+0x126/0x177 > [ 428.469352] ath10k_stop+0x41/0x7e > [ 428.469387] drv_stop+0x88/0x10e > [ 428.469410] __ieee80211_suspend+0x297/0x411 > [ 428.469441] rdev_suspend+0x6e/0xd0 > [ 428.469462] wiphy_suspend+0xb1/0x105 > [ 428.469483] ? name_show+0x2d/0x2d > [ 428.469490] dpm_run_callback+0x8c/0x126 > [ 428.469511] ? name_show+0x2d/0x2d > [ 428.469517] __device_suspend+0x2e7/0x41b > [ 428.469523] async_suspend+0x1f/0x93 > [ 428.469529] async_run_entry_fn+0x3d/0xd1 > [ 428.469535] process_one_work+0x1b1/0x329 > [ 428.469541] worker_thread+0x213/0x372 > [ 428.469547] kthread+0x150/0x15f > [ 428.469552] ? pr_cont_work+0x58/0x58 > [ 428.469558] ? kthread_blkcg+0x31/0x31 > > Signed-off-by: Abhishek Kumar <kuabhs@chromium.org> > Co-developed-by: Wen Gong <quic_wgong@quicinc.com> > Signed-off-by: Wen Gong <quic_wgong@quicinc.com> > --- > > drivers/net/wireless/ath/ath10k/mac.c | 18 ++++++++++++++++-- > 1 file changed, 16 insertions(+), 2 deletions(-) > > diff --git a/drivers/net/wireless/ath/ath10k/mac.c b/drivers/net/wireless/ath/ath10k/mac.c > index d804e19a742a..57ba27c46371 100644 > --- a/drivers/net/wireless/ath/ath10k/mac.c > +++ b/drivers/net/wireless/ath/ath10k/mac.c > @@ -5345,8 +5345,22 @@ static void ath10k_stop(struct ieee80211_hw *hw) > > mutex_lock(&ar->conf_mutex); > if (ar->state != ATH10K_STATE_OFF) { > - if (!ar->hw_rfkill_on) > - ath10k_halt(ar); > + if (!ar->hw_rfkill_on) { > + /* If the current driver state is RESTARTING but not yet > + * fully RESTARTED because of incoming suspend event, > + * then ath11k_halt is already called via > + * ath10k_core_restart and should not be called here. > + */ > + if (ar->state != ATH10K_STATE_RESTARTING) > + ath10k_halt(ar); > + else > + /* Suspending here, because when in RESTARTING > + * state, ath11k_core_stop skips > + * ath10k_wait_for_suspend. > + */ > + ath10k_wait_for_suspend(ar, > + WMI_PDEV_SUSPEND_AND_DISABLE_INTR); > + } > ar->state = ATH10K_STATE_OFF; > } > mutex_unlock(&ar->conf_mutex);
(sorry for the 2nd message with no content -- operator error)
Addressed all the above comments and pushed out a v2 patch. Fixes in V2: - Fixed typo, replaced ath11k by ath10k in comments. - Adjusted the position of the S-O-B tag. - Added tested on tag. Thanks Abhishek On Tue, Apr 26, 2022 at 9:23 AM Jeff Johnson <quic_jjohnson@quicinc.com> wrote: > > (sorry for the 2nd message with no content -- operator error)
diff --git a/drivers/net/wireless/ath/ath10k/mac.c b/drivers/net/wireless/ath/ath10k/mac.c index d804e19a742a..57ba27c46371 100644 --- a/drivers/net/wireless/ath/ath10k/mac.c +++ b/drivers/net/wireless/ath/ath10k/mac.c @@ -5345,8 +5345,22 @@ static void ath10k_stop(struct ieee80211_hw *hw) mutex_lock(&ar->conf_mutex); if (ar->state != ATH10K_STATE_OFF) { - if (!ar->hw_rfkill_on) - ath10k_halt(ar); + if (!ar->hw_rfkill_on) { + /* If the current driver state is RESTARTING but not yet + * fully RESTARTED because of incoming suspend event, + * then ath11k_halt is already called via + * ath10k_core_restart and should not be called here. + */ + if (ar->state != ATH10K_STATE_RESTARTING) + ath10k_halt(ar); + else + /* Suspending here, because when in RESTARTING + * state, ath11k_core_stop skips + * ath10k_wait_for_suspend. + */ + ath10k_wait_for_suspend(ar, + WMI_PDEV_SUSPEND_AND_DISABLE_INTR); + } ar->state = ATH10K_STATE_OFF; } mutex_unlock(&ar->conf_mutex);