From patchwork Wed Mar 5 00:00:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Long Li X-Patchwork-Id: 14001642 X-Patchwork-Delegate: kuba@kernel.org Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5B3F524C09A; Wed, 5 Mar 2025 00:00:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741132809; cv=none; b=abMoiwmWNrmEd+PrgPPpK4AU2RAMIHmYZPmoaebKXGJiqQIMsw9Hsqysct1PxX8BK4Ao5UE8mW6YiKBJQKEp577eJLBq0PGjCEdX+MSz50Oq/G53nUsKD1RwpFuHxt5QVRlSumB1jfN/5Q9DqmpUN7eDG0IMtKWCWGmtBpS8st4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741132809; c=relaxed/simple; bh=maeTuTRd1RxaKuCtTaK1UNUJtNBfFivEbPZ207OaVYM=; h=From:To:Cc:Subject:Date:Message-Id; b=GMYL8VFvaKjl1t77Ifc6TgjYM2greaU2kWSY2zf0ND18Cs5zh+WFl3lHXNGa49fMP9mU4fH4yJOf7jkIfmHMV7OMFfpInEJ1xxU9x91OZ6SCh0ZuA3J/0koFcClUDzlAvtUS6EAsZ0EVYgUY6BGd9FUl+u7+F9RgJwNaI8ct9mU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b=hA3/+aEQ; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b="hA3/+aEQ" Received: by linux.microsoft.com (Postfix, from userid 1202) id D22CE2112503; Tue, 4 Mar 2025 16:00:06 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com D22CE2112503 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com; s=default; t=1741132806; bh=YQAURlkfG3zyvMgNUuVAdt7wlORGt/c6p0druqSXPmM=; h=From:To:Cc:Subject:Date:From; b=hA3/+aEQsCaaOvHufY5KTMCArhTJFvIdHfJ7u4joBgsDWBpjZpqLE1O2uVdpDxd6A MkTAcW6ND7hADwClrgS7UTZ6CZYvVWuEnCpzuFwNSBq93ar4PUzOzBeeFYMcOIwxzy xamNw2rSq33mKeODFampNIZaBiYIxxOhzz98HhPQ= From: longli@linuxonhyperv.com To: Jason Gunthorpe , Leon Romanovsky , Konstantin Taranov , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org, Long Li Subject: [patch rdma-next v3 1/2] net: mana: Change the function signature of mana_get_primary_netdev_rcu Date: Tue, 4 Mar 2025 16:00:01 -0800 Message-Id: <1741132802-26795-1-git-send-email-longli@linuxonhyperv.com> X-Mailer: git-send-email 1.8.3.1 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-Patchwork-Delegate: kuba@kernel.org From: Long Li Change mana_get_primary_netdev_rcu() to mana_get_primary_netdev(), and return the ndev with refcount held. The caller is responsible for dropping the refcount by calling dev_put(). Signed-off-by: Long Li Reviewed-by: Jason Gunthorpe --- drivers/infiniband/hw/mana/device.c | 7 +++---- drivers/net/ethernet/microsoft/mana/mana_en.c | 11 +++++++---- include/net/mana/mana.h | 2 +- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index 3416a85f8738..afe3d8d20b3b 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -84,10 +84,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, dev->ib_dev.num_comp_vectors = mdev->gdma_context->max_num_queues; dev->ib_dev.dev.parent = mdev->gdma_context->dev; - rcu_read_lock(); /* required to get primary netdev */ - ndev = mana_get_primary_netdev_rcu(mc, 0); + ndev = mana_get_primary_netdev(mc, 0); if (!ndev) { - rcu_read_unlock(); ret = -ENODEV; ibdev_err(&dev->ib_dev, "Failed to get netdev for IB port 1"); goto free_ib_device; @@ -95,7 +93,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, ether_addr_copy(mac_addr, ndev->dev_addr); addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, ndev->dev_addr); ret = ib_device_set_netdev(&dev->ib_dev, ndev, 1); - rcu_read_unlock(); + /* mana_get_primary_netdev() returns ndev with refcount held */ + dev_put(ndev); if (ret) { ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret); goto free_ib_device; diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c index aa1e47233fe5..cfa664d5a137 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -3131,21 +3131,24 @@ void mana_remove(struct gdma_dev *gd, bool suspending) kfree(ac); } -struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index) +struct net_device *mana_get_primary_netdev(struct mana_context *ac, u32 port_index) { struct net_device *ndev; - RCU_LOCKDEP_WARN(!rcu_read_lock_held(), - "Taking primary netdev without holding the RCU read lock"); if (port_index >= ac->num_ports) return NULL; + rcu_read_lock(); + /* When mana is used in netvsc, the upper netdevice should be returned. */ if (ac->ports[port_index]->flags & IFF_SLAVE) ndev = netdev_master_upper_dev_get_rcu(ac->ports[port_index]); else ndev = ac->ports[port_index]; + dev_hold(ndev); + rcu_read_unlock(); + return ndev; } -EXPORT_SYMBOL_NS(mana_get_primary_netdev_rcu, "NET_MANA"); +EXPORT_SYMBOL_NS(mana_get_primary_netdev, "NET_MANA"); diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h index 0d00b24eacaf..770359ce7d3b 100644 --- a/include/net/mana/mana.h +++ b/include/net/mana/mana.h @@ -827,5 +827,5 @@ int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id, u32 doorbell_pg_id); void mana_uncfg_vport(struct mana_port_context *apc); -struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index); +struct net_device *mana_get_primary_netdev(struct mana_context *ac, u32 port_index); #endif /* _MANA_H */ From patchwork Wed Mar 5 00:00:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Long Li X-Patchwork-Id: 14001643 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 087C323BCF7; Wed, 5 Mar 2025 00:00:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741132809; cv=none; b=G/rv7HIfOFD2ChyvihdfnJvJn2JSuWixrVHGt82DiR9P4EYE1zmpfVZVQG9ZTU8ggUuf4NRr4hmj+NNHx+J/KxnUuwvrm2EVlyMblrq5D9rfXfZJVMgQlvcDESpeMueKAL/3fwdcaDdJ5B+qNh+PAZkmNC53Mw3qZ7Aa0LJOKcE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741132809; c=relaxed/simple; bh=p7ys8mvToushqQoAbCFX9UMvMk9S3Fd3cwdjFVfRosg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Ov2MHUSc2jq1f8mrUVCC9N6uocau/nz6eZHjKsRdWWRw/jRcCDT9r+Q/g++4yRd2inBGxCv2tpdEN47XUvp2MILzp+7b0odEA/ThafHL6YXNveIT55UD3brwbqmEvD6vfhm5EsEWMXia/TrbY/Q28pjQqD/FWtGIUXznGBN4wMQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b=DM/wmKzU; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b="DM/wmKzU" Received: by linux.microsoft.com (Postfix, from userid 1202) id 9F50A2112504; Tue, 4 Mar 2025 16:00:07 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 9F50A2112504 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com; s=default; t=1741132807; bh=vzn02cWkPx/F8QdVCcYD6nrle8ymPhDePDvSmaJZ+Pg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DM/wmKzU6DQLt4EFXxBTKQ3bMsbSEUSjWLu2HP+Lo2hPz0DY2xFkngmBuSzcs+Jjm rfmEVDDak9rmZJyX10LnCUDmH02pfxr7oxJZCUUi3Vh8ddM318Z+L7nOrLJcwBzQKV RViyP9bRw4ylXWPIyxeeoInuERNP/i8kll7gryB8= From: longli@linuxonhyperv.com To: Jason Gunthorpe , Leon Romanovsky , Konstantin Taranov , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org, Long Li Subject: [patch rdma-next v3 2/2] RDMA/mana_ib: Handle net event for pointing to the current netdev Date: Tue, 4 Mar 2025 16:00:02 -0800 Message-Id: <1741132802-26795-2-git-send-email-longli@linuxonhyperv.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1741132802-26795-1-git-send-email-longli@linuxonhyperv.com> References: <1741132802-26795-1-git-send-email-longli@linuxonhyperv.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Long Li When running under Hyper-V, the master device to the RDMA device is always bonded to this RDMA device. This is not user-configurable. The master device can be unbind/bind from the kernel. During those events, the RDMA device should set to the current netdev to relect the change of master device from those events. Signed-off-by: Long Li --- Changes v2: Add missing error handling when register_netdevice_notifier() fails. v3: Change mana_get_primary_netdev() to return with netdev refcount held. drivers/infiniband/hw/mana/device.c | 46 ++++++++++++++++++++++++++-- drivers/infiniband/hw/mana/mana_ib.h | 1 + 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index afe3d8d20b3b..dc03f2d8c1b6 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -51,6 +51,37 @@ static const struct ib_device_ops mana_ib_dev_ops = { ib_ind_table), }; +static int mana_ib_netdev_event(struct notifier_block *this, + unsigned long event, void *ptr) +{ + struct mana_ib_dev *dev = container_of(this, struct mana_ib_dev, nb); + struct net_device *event_dev = netdev_notifier_info_to_dev(ptr); + struct gdma_context *gc = dev->gdma_dev->gdma_context; + struct mana_context *mc = gc->mana.driver_data; + struct net_device *ndev; + + if (event_dev != mc->ports[0]) + return NOTIFY_DONE; + + switch (event) { + case NETDEV_CHANGEUPPER: + ndev = mana_get_primary_netdev(mc, 0); + /* + * RDMA core will setup GID based on updated netdev. + * It's not possible to race with the core as rtnl lock is being + * held. + */ + ib_device_set_netdev(&dev->ib_dev, ndev, 1); + + /* mana_get_primary_netdev() returns ndev with refcount held */ + dev_put(ndev); + + return NOTIFY_OK; + default: + return NOTIFY_DONE; + } +} + static int mana_ib_probe(struct auxiliary_device *adev, const struct auxiliary_device_id *id) { @@ -108,17 +139,25 @@ static int mana_ib_probe(struct auxiliary_device *adev, } dev->gdma_dev = &mdev->gdma_context->mana_ib; + dev->nb.notifier_call = mana_ib_netdev_event; + ret = register_netdevice_notifier(&dev->nb); + if (ret) { + ibdev_err(&dev->ib_dev, "Failed to register net notifier, %d", + ret); + goto deregister_device; + } + ret = mana_ib_gd_query_adapter_caps(dev); if (ret) { ibdev_err(&dev->ib_dev, "Failed to query device caps, ret %d", ret); - goto deregister_device; + goto deregister_net_notifier; } ret = mana_ib_create_eqs(dev); if (ret) { ibdev_err(&dev->ib_dev, "Failed to create EQs, ret %d", ret); - goto deregister_device; + goto deregister_net_notifier; } ret = mana_ib_gd_create_rnic_adapter(dev); @@ -147,6 +186,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, mana_ib_gd_destroy_rnic_adapter(dev); destroy_eqs: mana_ib_destroy_eqs(dev); +deregister_net_notifier: + unregister_netdevice_notifier(&dev->nb); deregister_device: mana_gd_deregister_device(dev->gdma_dev); free_ib_device: @@ -162,6 +203,7 @@ static void mana_ib_remove(struct auxiliary_device *adev) xa_destroy(&dev->qp_table_wq); mana_ib_gd_destroy_rnic_adapter(dev); mana_ib_destroy_eqs(dev); + unregister_netdevice_notifier(&dev->nb); mana_gd_deregister_device(dev->gdma_dev); ib_dealloc_device(&dev->ib_dev); } diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index b53a5b4de908..d88187072899 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -64,6 +64,7 @@ struct mana_ib_dev { struct gdma_queue **eqs; struct xarray qp_table_wq; struct mana_ib_adapter_caps adapter_caps; + struct notifier_block nb; }; struct mana_ib_wq {