From patchwork Sat Feb 6 13:50:41 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Estrin, Alex" X-Patchwork-Id: 8242541 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 65FE19F37A for ; Sat, 6 Feb 2016 13:51:03 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7AC2C20212 for ; Sat, 6 Feb 2016 13:51:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 61C032015E for ; Sat, 6 Feb 2016 13:51:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751206AbcBFNup (ORCPT ); Sat, 6 Feb 2016 08:50:45 -0500 Received: from mga04.intel.com ([192.55.52.120]:55959 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751174AbcBFNuo (ORCPT ); Sat, 6 Feb 2016 08:50:44 -0500 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP; 06 Feb 2016 05:50:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,405,1449561600"; d="scan'208";a="906852734" Received: from sedona.ch.intel.com ([143.182.228.65]) by orsmga002.jf.intel.com with ESMTP; 06 Feb 2016 05:50:43 -0800 Received: from phlsvlogin03.ph.intel.com (phlsvlogin03.ph.intel.com [10.228.195.69]) by sedona.ch.intel.com (8.13.6/8.14.3/Standard MailSET/Hub) with ESMTP id u16DogjM011826; Sat, 6 Feb 2016 06:50:42 -0700 Received: from phlsvlogin03.ph.intel.com (localhost [127.0.0.1]) by phlsvlogin03.ph.intel.com with ESMTP id u16Dofpc011652; Sat, 6 Feb 2016 08:50:41 -0500 Subject: [PATCH v2] IB/ipoib: fix for rare multicast join race condition. To: linux-rdma@vger.kernel.org, Doug Ledford From: Alex Estrin Date: Sat, 06 Feb 2016 08:50:41 -0500 Message-ID: <20160206135041.11630.77019.stgit@phlsvlogin03.ph.intel.com> User-Agent: StGit/0.16 MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A narrow window for race condition still exist between multicast join thread and *dev_flush workers. A kernel crash caused by prolong erratic link state changes was observed (most likely a faulty cabling): [167275.656270] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 [167275.665973] IP: [] ipoib_mcast_join+0xae/0x1d0 [ib_ipoib] [167275.674443] PGD 0 [167275.677373] Oops: 0000 [#1] SMP ... [167275.977530] Call Trace: [167275.982225] [] ? ipoib_mcast_free+0x200/0x200 [ib_ipoib] [167275.992024] [] ipoib_mcast_join_task+0x2a7/0x490 [ib_ipoib] [167276.002149] [] process_one_work+0x17b/0x470 [167276.010754] [] worker_thread+0x11b/0x400 [167276.019088] [] ? rescuer_thread+0x400/0x400 [167276.027737] [] kthread+0xcf/0xe0 Here was a hit spot: ipoib_mcast_join() { .............. rec.qkey = priv->broadcast->mcmember.qkey; ^^^^^^^ ..... } Proposed patch should prevent multicast join task to continue if link state change is detected. Signed-off-by: Alex Estrin --- Changes from v1: No need to lock again if error detected. --- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 14 ++++++++++++-- 1 files changed, 12 insertions(+), 2 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c index 050dfa1..3ce2f0a 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c @@ -456,7 +456,7 @@ out_locked: return status; } -static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast) +static int ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast) { struct ipoib_dev_priv *priv = netdev_priv(dev); struct ib_sa_multicast *multicast; @@ -466,6 +466,9 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast) ib_sa_comp_mask comp_mask; int ret = 0; + if (!priv->broadcast) + return -EINVAL; + ipoib_dbg_mcast(priv, "joining MGID %pI6\n", mcast->mcmember.mgid.raw); rec.mgid = mcast->mcmember.mgid; @@ -539,6 +542,7 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast) spin_unlock_irq(&priv->lock); complete(&mcast->done); } + return 0; } void ipoib_mcast_join_task(struct work_struct *work) @@ -611,6 +615,11 @@ void ipoib_mcast_join_task(struct work_struct *work) * and attached */ list_for_each_entry(mcast, &priv->multicast_list, list) { + if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags)) { + mcast = NULL; + delay_until = 0; + goto out; + } if (IS_ERR_OR_NULL(mcast->mc) && !test_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags) && (!test_bit(IPOIB_MCAST_FLAG_SENDONLY, &mcast->flags) || @@ -621,7 +630,8 @@ void ipoib_mcast_join_task(struct work_struct *work) init_completion(&mcast->done); set_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags); spin_unlock_irq(&priv->lock); - ipoib_mcast_join(dev, mcast); + if (ipoib_mcast_join(dev, mcast) != 0) + return; spin_lock_irq(&priv->lock); } else if (!delay_until || time_before(mcast->delay_until, delay_until))