From patchwork Tue Dec 11 22:15:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10724935 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80EBF14E2 for ; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 705DF2AA69 for ; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 64A372AAF5; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E46CE2AA69 for ; Tue, 11 Dec 2018 22:25:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726314AbeLKWZR (ORCPT ); Tue, 11 Dec 2018 17:25:17 -0500 Received: from alln-iport-6.cisco.com ([173.37.142.93]:57843 "EHLO alln-iport-6.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726355AbeLKWZR (ORCPT ); Tue, 11 Dec 2018 17:25:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5662; q=dns/txt; s=iport; t=1544567116; x=1545776716; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JtQAsYr6gLdxlTVrL1jUAoziSGZeQE/FfXC9KkFQHEE=; b=O+alGSviwbZXlag3xKW7OC215iJpFO2v4zwi7l14brZO8ndvgxmFF94N Fvppj0D+PlxQJmTzBqx9I9uXLPVcTHGcWcrShgA/e+xwi5B1ULjPkJTWR PJhbmfkDnEptg3Cfc0StcGjJIwN8IuRrmfqbKsXMc0jvqXdNLoXXyuIPK E=; X-IronPort-AV: E=Sophos;i="5.56,343,1539648000"; d="scan'208";a="211088518" Received: from alln-core-2.cisco.com ([173.36.13.135]) by alln-iport-6.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 22:15:46 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-2.cisco.com (8.15.2/8.15.2) with ESMTP id wBBMFkEV018885; Tue, 11 Dec 2018 22:15:46 GMT Received: by cisco.com (Postfix, from userid 478433) id 5E4F420F2001; Tue, 11 Dec 2018 14:15:46 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: pkaustub@cisco.com, benve@cisco.com, gvaradar@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, ravianan@cisco.com, jsquyres@cisco.com Subject: [PATCH for-next 1/3] IB/usnic: fix deadlock Date: Tue, 11 Dec 2018 14:15:41 -0800 Message-Id: <1544566543-3395-2-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> References: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: alln-core-2.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Govindarajulu Varadarajan There is a dead lock in usnic ib_register() and netdev_notify() path. usnic_ib_discover_pf() | mutex_lock(&usnic_ib_ibdev_list_lock); | usnic_ib_device_add(); | ib_register_device() | usnic_ib_query_port() | mutex_lock(&us_ibdev->usdev_lock); | ib_get_eth_speed() | rtnl_lock() Order of locking: &usnic_ib_ibdev_list_lock -> usdev_lock -> rtnl_lock rtnl_lock() | usnic_ib_netdevice_event() | mutex_lock(&usnic_ib_ibdev_list_lock); Order of locking: rtnl_lock -> &usnic_ib_ibdev_list_lock The solution is to not handle usnic_ib_netdevice_event() with the rtnl lock held. This commit creates a single threaded workqueue to defer the handling of netdev/inet events. Signed-off-by: Govindarajulu Varadarajan Signed-off-by: Parvi Kaustubhi --- drivers/infiniband/hw/usnic/usnic_ib.h | 6 +++ drivers/infiniband/hw/usnic/usnic_ib_main.c | 60 ++++++++++++++++++++++++----- 2 files changed, 56 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_ib.h b/drivers/infiniband/hw/usnic/usnic_ib.h index 525bf27..aff4eb4 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib.h +++ b/drivers/infiniband/hw/usnic/usnic_ib.h @@ -93,6 +93,12 @@ struct usnic_ib_vf { struct list_head link; }; +struct usnic_work { + struct work_struct work; + unsigned long event; + void *ptr; +}; + static inline struct usnic_ib_dev *to_usdev(struct ib_device *ibdev) { diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c index 413fa57..dd28442 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c @@ -71,6 +71,7 @@ static const char usnic_version[] = static DEFINE_MUTEX(usnic_ib_ibdev_list_lock); static LIST_HEAD(usnic_ib_ibdev_list); +static struct workqueue_struct *usnic_notify_work; /* Callback dump funcs */ static int usnic_ib_dump_vf_hdr(void *obj, char *buf, int buf_sz) @@ -212,21 +213,35 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev, mutex_unlock(&us_ibdev->usdev_lock); } -static int usnic_ib_netdevice_event(struct notifier_block *notifier, - unsigned long event, void *ptr) +static void usnic_ib_netdevice_work(struct work_struct *work) { + struct usnic_work *uswork = container_of(work, struct usnic_work, work); + struct net_device *netdev = netdev_notifier_info_to_dev(uswork->ptr); struct usnic_ib_dev *us_ibdev; - struct net_device *netdev = netdev_notifier_info_to_dev(ptr); - mutex_lock(&usnic_ib_ibdev_list_lock); list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { if (us_ibdev->netdev == netdev) { - usnic_ib_handle_usdev_event(us_ibdev, event); + usnic_ib_handle_usdev_event(us_ibdev, uswork->event); break; } } mutex_unlock(&usnic_ib_ibdev_list_lock); + kfree(uswork); +} + +static int usnic_ib_netdevice_event(struct notifier_block *nblock, + unsigned long event, void *ptr) +{ + struct usnic_work *uswork; + + uswork = kzalloc(sizeof(*uswork), GFP_ATOMIC); + if (uswork) { + uswork->event = event; + uswork->ptr = ptr; + INIT_WORK(&uswork->work, usnic_ib_netdevice_work); + queue_work(usnic_notify_work, &uswork->work); + } return NOTIFY_DONE; } @@ -276,24 +291,42 @@ static int usnic_ib_handle_inet_event(struct usnic_ib_dev *us_ibdev, return NOTIFY_DONE; } -static int usnic_ib_inetaddr_event(struct notifier_block *notifier, - unsigned long event, void *ptr) +static void usnic_ib_inetaddr_work(struct work_struct *work) { + struct usnic_work *uswork = container_of(work, struct usnic_work, work); struct usnic_ib_dev *us_ibdev; - struct in_ifaddr *ifa = ptr; + struct in_ifaddr *ifa = uswork->ptr; struct net_device *netdev = ifa->ifa_dev->dev; mutex_lock(&usnic_ib_ibdev_list_lock); list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { if (us_ibdev->netdev == netdev) { - usnic_ib_handle_inet_event(us_ibdev, event, ptr); + usnic_ib_handle_inet_event(us_ibdev, uswork->event, + uswork->ptr); break; } } mutex_unlock(&usnic_ib_ibdev_list_lock); + kfree(uswork); +} + +static int usnic_ib_inetaddr_event(struct notifier_block *nblock, + unsigned long event, void *ptr) +{ + struct usnic_work *uswork; + + uswork = kzalloc(sizeof(*uswork), GFP_ATOMIC); + + if (uswork) { + uswork->ptr = ptr; + uswork->event = event; + INIT_WORK(&uswork->work, usnic_ib_inetaddr_work); + queue_work(usnic_notify_work, &uswork->work); + } return NOTIFY_DONE; } + static struct notifier_block usnic_ib_inetaddr_notifier = { .notifier_call = usnic_ib_inetaddr_event }; @@ -653,10 +686,15 @@ static int __init usnic_ib_init(void) return err; } + usnic_notify_work = create_singlethread_workqueue("usnic_notify_work"); + if (!usnic_notify_work) { + usnic_err("Failed to create notify workqueue"); + goto out_umem_fini; + } err = pci_register_driver(&usnic_ib_pci_driver); if (err) { usnic_err("Unable to register with PCI\n"); - goto out_umem_fini; + goto out_notify_work; } err = register_netdevice_notifier(&usnic_ib_netdevice_notifier); @@ -687,6 +725,8 @@ static int __init usnic_ib_init(void) unregister_netdevice_notifier(&usnic_ib_netdevice_notifier); out_pci_unreg: pci_unregister_driver(&usnic_ib_pci_driver); +out_notify_work: + destroy_workqueue(usnic_notify_work); out_umem_fini: usnic_uiom_fini(); From patchwork Tue Dec 11 22:15:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10724931 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC3DD1869 for ; Tue, 11 Dec 2018 22:25:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 985F32AA69 for ; Tue, 11 Dec 2018 22:25:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 898792AAF5; Tue, 11 Dec 2018 22:25:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 392752AA69 for ; Tue, 11 Dec 2018 22:25:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726314AbeLKWZF (ORCPT ); Tue, 11 Dec 2018 17:25:05 -0500 Received: from alln-iport-3.cisco.com ([173.37.142.90]:30078 "EHLO alln-iport-3.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726183AbeLKWZF (ORCPT ); Tue, 11 Dec 2018 17:25:05 -0500 X-Greylist: delayed 557 seconds by postgrey-1.27 at vger.kernel.org; Tue, 11 Dec 2018 17:25:04 EST DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=1649; q=dns/txt; s=iport; t=1544567104; x=1545776704; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QB/pe2R2yy5XuCpnaM8kOsCkqLAcoNa4j4jceXQC6wU=; b=BYyG0iCvkbdnC3mHwGac2BDovS/ljyQ/xTvz+l4RjJFtOlqMA4ddRQ+U UoFXAnt/sGGtH00NngbCpIqKABS/A0Oi5kvi3LSxU3IXq4DQkeoHQzqdU 4hRsBaWW9OOjh4lw7/UMJLYpw3OaD5rXFwVK5tEGlktPiCq9ZD17JI8ba E=; X-IronPort-AV: E=Sophos;i="5.56,343,1539648000"; d="scan'208";a="211960244" Received: from rcdn-core-6.cisco.com ([173.37.93.157]) by alln-iport-3.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 22:15:48 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-core-6.cisco.com (8.15.2/8.15.2) with ESMTP id wBBMFlXe004112; Tue, 11 Dec 2018 22:15:47 GMT Received: by cisco.com (Postfix, from userid 478433) id 8638720F2001; Tue, 11 Dec 2018 14:15:47 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: pkaustub@cisco.com, benve@cisco.com, gvaradar@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, ravianan@cisco.com, jsquyres@cisco.com Subject: [PATCH for-next 2/3] IB/usnic: fix potential deadlock Date: Tue, 11 Dec 2018 14:15:42 -0800 Message-Id: <1544566543-3395-3-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> References: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-core-6.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Acquiring the rtnl lock while holding usdev_lock could result in a deadlock. For example: usnic_ib_query_port() | mutex_lock(&us_ibdev->usdev_lock) | ib_get_eth_speed() | rtnl_lock() rtnl_lock() | usnic_ib_netdevice_event() | mutex_lock(&us_ibdev->usdev_lock) This commit moves the usdev_lock acquisition after the rtnl lock has been released. This is safe to do because usdev_lock is not protecting anything being accessed in ib_get_eth_speed(). Hence, the correct order of holding locks (rtnl -> usdev_lock) is not violated. Signed-off-by: Parvi Kaustubhi --- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c index 0b91ff3..598e23c 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c @@ -336,13 +336,16 @@ int usnic_ib_query_port(struct ib_device *ibdev, u8 port, usnic_dbg("\n"); - mutex_lock(&us_ibdev->usdev_lock); if (ib_get_eth_speed(ibdev, port, &props->active_speed, - &props->active_width)) { - mutex_unlock(&us_ibdev->usdev_lock); + &props->active_width)) return -EINVAL; - } + /* + * usdev_lock is acquired after (and not before) ib_get_eth_speed call + * because acquiring rtnl_lock in ib_get_eth_speed, while holding + * usdev_lock could lead to a deadlock. + */ + mutex_lock(&us_ibdev->usdev_lock); /* props being zeroed by the caller, avoid zeroing it here */ props->lid = 0; From patchwork Tue Dec 11 22:15:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10724933 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2ADD214E2 for ; Tue, 11 Dec 2018 22:25:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A5F52AA69 for ; Tue, 11 Dec 2018 22:25:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E5152AAF5; Tue, 11 Dec 2018 22:25:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B69502AA69 for ; Tue, 11 Dec 2018 22:25:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726416AbeLKWZQ (ORCPT ); Tue, 11 Dec 2018 17:25:16 -0500 Received: from alln-iport-1.cisco.com ([173.37.142.88]:50409 "EHLO alln-iport-1.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726215AbeLKWZQ (ORCPT ); Tue, 11 Dec 2018 17:25:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=592; q=dns/txt; s=iport; t=1544567115; x=1545776715; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=HRp67aWZVJYbL5vdK7kUK6fKDShgef6azO8bCxRy3h8=; b=Kr2nKwnlift3edWRlfutICH8sO1cfrG8iSz0Rdj8vj+YNcq2P25ZUtXn epS3PNIXatfuUgdqzcm4cRE9mfYdB/D8uX6fjoEadMH2kRpqH/qwBKAkd g//vp/kmsXHnWNP5wMhQh+jPHoUZoyWxbrJEKcrX0nVGELseoqJNt/4Fy s=; X-IronPort-AV: E=Sophos;i="5.56,343,1539648000"; d="scan'208";a="211705509" Received: from rcdn-core-7.cisco.com ([173.37.93.143]) by alln-iport-1.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 22:15:49 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-core-7.cisco.com (8.15.2/8.15.2) with ESMTP id wBBMFmKs007465; Tue, 11 Dec 2018 22:15:48 GMT Received: by cisco.com (Postfix, from userid 478433) id 908F520F2001; Tue, 11 Dec 2018 14:15:48 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: pkaustub@cisco.com, benve@cisco.com, gvaradar@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, ravianan@cisco.com, jsquyres@cisco.com Subject: [PATCH for-next 3/3] MAINTAINERS: update usnic driver maintainers Date: Tue, 11 Dec 2018 14:15:43 -0800 Message-Id: <1544566543-3395-4-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> References: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-core-7.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add Nelson Escobar and myself as maintainers for drivers/infiniband/hw/usnic Signed-off-by: Parvi Kaustubhi --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index f485597..a0035a7 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3591,6 +3591,8 @@ F: drivers/net/ethernet/cisco/enic/ CISCO VIC LOW LATENCY NIC DRIVER M: Christian Benvenuti +M: Nelson Escobar +M: Parvi Kaustubhi S: Supported F: drivers/infiniband/hw/usnic/