[net,v2] net/smc: Fix hung_task when removing SMC-R devices

Commit Message

Wen Gu Jan. 16, 2022, 7:43 a.m. UTC

A hung_task is observed when removing SMC-R devices. Suppose that
a link group has two active links(lnk_A, lnk_B) associated with two
different SMC-R devices(dev_A, dev_B). When dev_A is removed, the
link group will be removed from smc_lgr_list and added into
lgr_linkdown_list. lnk_A will be cleared and smcibdev(A)->lnk_cnt
will reach to zero. However, when dev_B is removed then, the link
group can't be found in smc_lgr_list and lnk_B won't be cleared,
making smcibdev->lnk_cnt never reaches zero, which causes a hung_task.

This patch fixes this issue by restoring the implementation of
smc_smcr_terminate_all() to what it was before commit 349d43127dac
("net/smc: fix kernel panic caused by race of smc_sock"). The original
implementation also satisfies the intention that make sure QP destroy
earlier than CQ destroy because we will always wait for smcibdev->lnk_cnt
reaches zero, which guarantees QP has been destroyed.

Fixes: 349d43127dac ("net/smc: fix kernel panic caused by race of smc_sock")
Signed-off-by: Wen Gu <[e-mail]>
---
v1 -> v2:
- Remove some comments.
---
 net/smc/smc_core.c | 17 +----------------
 1 file changed, 1 insertion(+), 16 deletions(-)

Comments

[e-mail] Jan. 16, 2022, 12:40 p.m. UTC | #1

Hello:

This patch was applied to netdev/net.git (master)
by David S. Miller <[e-mail]>:

On Sun, 16 Jan 2022 15:43:42 +0800 you wrote:
> A hung_task is observed when removing SMC-R devices. Suppose that
> a link group has two active links(lnk_A, lnk_B) associated with two
> different SMC-R devices(dev_A, dev_B). When dev_A is removed, the
> link group will be removed from smc_lgr_list and added into
> lgr_linkdown_list. lnk_A will be cleared and smcibdev(A)->lnk_cnt
> will reach to zero. However, when dev_B is removed then, the link
> group can't be found in smc_lgr_list and lnk_B won't be cleared,
> making smcibdev->lnk_cnt never reaches zero, which causes a hung_task.
> 
> [...]

Here is the summary with links:
  - [net,v2] net/smc: Fix hung_task when removing SMC-R devices
    https://git.kernel.org/netdev/net/c/56d99e81ecbc

You are awesome, thank you!

Patch

diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index b19c0aa..5fc84f0 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1524,16 +1524,11 @@  void smc_smcd_terminate_all(struct smcd_dev *smcd)
 /* Called when an SMCR device is removed or the smc module is unloaded.
  * If smcibdev is given, all SMCR link groups using this device are terminated.
  * If smcibdev is NULL, all SMCR link groups are terminated.
- *
- * We must wait here for QPs been destroyed before we destroy the CQs,
- * or we won't received any CQEs and cdc_pend_tx_wr cannot reach 0 thus
- * smc_sock cannot be released.
  */
 void smc_smcr_terminate_all(struct smc_ib_device *smcibdev)
 {
 	struct smc_link_group *lgr, *lg;
 	LIST_HEAD(lgr_free_list);
-	LIST_HEAD(lgr_linkdown_list);
 	int i;
 
 	spin_lock_bh(&smc_lgr_list.lock);
@@ -1545,7 +1540,7 @@  void smc_smcr_terminate_all(struct smc_ib_device *smcibdev)
 		list_for_each_entry_safe(lgr, lg, &smc_lgr_list.list, list) {
 			for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
 				if (lgr->lnk[i].smcibdev == smcibdev)
-					list_move_tail(&lgr->list, &lgr_linkdown_list);
+					smcr_link_down_cond_sched(&lgr->lnk[i]);
 			}
 		}
 	}
@@ -1557,16 +1552,6 @@  void smc_smcr_terminate_all(struct smc_ib_device *smcibdev)
 		__smc_lgr_terminate(lgr, false);
 	}
 
-	list_for_each_entry_safe(lgr, lg, &lgr_linkdown_list, list) {
-		for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
-			if (lgr->lnk[i].smcibdev == smcibdev) {
-				mutex_lock(&lgr->llc_conf_mutex);
-				smcr_link_down_cond(&lgr->lnk[i]);
-				mutex_unlock(&lgr->llc_conf_mutex);
-			}
-		}
-	}
-
 	if (smcibdev) {
 		if (atomic_read(&smcibdev->lnk_cnt))
 			wait_event(smcibdev->lnks_deleted,