From patchwork Thu Mar 6 19:24:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Long Li X-Patchwork-Id: 14005250 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C0301211476; Thu, 6 Mar 2025 19:24:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741289084; cv=none; b=ZptCuxmCPyCv/L07GeyBewCdZJgx+vFF3wohlT4rwppojeGr/oLOw3Szo8nxfS6KdLPvErvHUxQKdUGWINO7TjrJ1qAS5D9bUTI3bdOx4+FJyQmWW1LhIp9sbRUftq7VV5ddszequD7ylkABrHkEAUzvON7hrLj/QLbyN2gQ0dE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741289084; c=relaxed/simple; bh=nbRLzIOOa7m9nYevtoKhk36VnuLqeUKs5GWulz/zjQo=; h=From:To:Cc:Subject:Date:Message-Id; b=asosVHgk7CH/WzB5b/ParGISO3ojYnN7D1B2c/vV7r0f0dMTh2ulGnWVYhbcrkVHhi3adB4gkIdF0SXGuOO4NSV6gOzbxWyCsVnSOAXDe8+rH6NEKG/WUGFBK9xnnqcbxRvDawIezZJZeY4fKHEiiz8cthIQo9vXEVw0Lm1Qd+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b=DuHzzPAf; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b="DuHzzPAf" Received: by linux.microsoft.com (Postfix, from userid 1202) id 60D7A210EAE5; Thu, 6 Mar 2025 11:24:42 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 60D7A210EAE5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com; s=default; t=1741289082; bh=TvbvlMSyhaZWRXp1gG0nGpdUVO19wq4jNvWj6tm9Igs=; h=From:To:Cc:Subject:Date:From; b=DuHzzPAfqwDKzkp3CnIWwzIjRsdVYWkQ576huWqZNW9H4rGJryVMRxMZ5Vyvx4moz UsOJSMVKSqPOmWHVGPA7zxDxK6LaKZc8+h/XgVeh7OXg1pViy2lgbOTj3ufZ1E3bxW i4ngtLOclpaMkVC/VXUDD2jZNNAbB1Gl1SOQnWjw= From: longli@linuxonhyperv.com To: Jason Gunthorpe , Leon Romanovsky , Konstantin Taranov , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org, Long Li Subject: [patch rdma-next v5 1/2] net: mana: Change the function signature of mana_get_primary_netdev_rcu Date: Thu, 6 Mar 2025 11:24:38 -0800 Message-Id: <1741289079-18744-1-git-send-email-longli@linuxonhyperv.com> X-Mailer: git-send-email 1.8.3.1 Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Long Li Change mana_get_primary_netdev_rcu() to mana_get_primary_netdev(), and return the ndev with refcount held. The caller is responsible for dropping the refcount. Also drop the check for IFF_SLAVE as it is not necessary if the upper device is present. Signed-off-by: Long Li --- Changes v4: use netdev_hold()/netdev_put() and remove the check for IFF_SLAVE v5: use netdevice_tracker in mana_ib_dev for netdev_hold()/netdev_put() drivers/infiniband/hw/mana/device.c | 7 +++--- drivers/infiniband/hw/mana/mana_ib.h | 1 + drivers/net/ethernet/microsoft/mana/mana_en.c | 22 ++++++++++++------- include/net/mana/mana.h | 4 +++- 4 files changed, 21 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index 3416a85f8738..363566095501 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -84,10 +84,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, dev->ib_dev.num_comp_vectors = mdev->gdma_context->max_num_queues; dev->ib_dev.dev.parent = mdev->gdma_context->dev; - rcu_read_lock(); /* required to get primary netdev */ - ndev = mana_get_primary_netdev_rcu(mc, 0); + ndev = mana_get_primary_netdev(mc, 0, &dev->dev_tracker); if (!ndev) { - rcu_read_unlock(); ret = -ENODEV; ibdev_err(&dev->ib_dev, "Failed to get netdev for IB port 1"); goto free_ib_device; @@ -95,7 +93,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, ether_addr_copy(mac_addr, ndev->dev_addr); addrconf_addr_eui48((u8 *)&dev->ib_dev.node_guid, ndev->dev_addr); ret = ib_device_set_netdev(&dev->ib_dev, ndev, 1); - rcu_read_unlock(); + /* mana_get_primary_netdev() returns ndev with refcount held */ + netdev_put(ndev, &dev->dev_tracker); if (ret) { ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret); goto free_ib_device; diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index b53a5b4de908..2638688f2505 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -64,6 +64,7 @@ struct mana_ib_dev { struct gdma_queue **eqs; struct xarray qp_table_wq; struct mana_ib_adapter_caps adapter_caps; + netdevice_tracker dev_tracker; }; struct mana_ib_wq { diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c index aa1e47233fe5..4e870b11f946 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -3131,21 +3131,27 @@ void mana_remove(struct gdma_dev *gd, bool suspending) kfree(ac); } -struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index) +struct net_device *mana_get_primary_netdev(struct mana_context *ac, + u32 port_index, + netdevice_tracker *tracker) { struct net_device *ndev; - RCU_LOCKDEP_WARN(!rcu_read_lock_held(), - "Taking primary netdev without holding the RCU read lock"); if (port_index >= ac->num_ports) return NULL; - /* When mana is used in netvsc, the upper netdevice should be returned. */ - if (ac->ports[port_index]->flags & IFF_SLAVE) - ndev = netdev_master_upper_dev_get_rcu(ac->ports[port_index]); - else + rcu_read_lock(); + + /* If mana is used in netvsc, the upper netdevice should be returned. */ + ndev = netdev_master_upper_dev_get_rcu(ac->ports[port_index]); + + /* If there is no upper device, use the parent Ethernet device */ + if (!ndev) ndev = ac->ports[port_index]; + netdev_hold(ndev, tracker, GFP_ATOMIC); + rcu_read_unlock(); + return ndev; } -EXPORT_SYMBOL_NS(mana_get_primary_netdev_rcu, "NET_MANA"); +EXPORT_SYMBOL_NS(mana_get_primary_netdev, "NET_MANA"); diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h index 0d00b24eacaf..0f78065de8fe 100644 --- a/include/net/mana/mana.h +++ b/include/net/mana/mana.h @@ -827,5 +827,7 @@ int mana_cfg_vport(struct mana_port_context *apc, u32 protection_dom_id, u32 doorbell_pg_id); void mana_uncfg_vport(struct mana_port_context *apc); -struct net_device *mana_get_primary_netdev_rcu(struct mana_context *ac, u32 port_index); +struct net_device *mana_get_primary_netdev(struct mana_context *ac, + u32 port_index, + netdevice_tracker *tracker); #endif /* _MANA_H */ From patchwork Thu Mar 6 19:24:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Long Li X-Patchwork-Id: 14005251 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5F1322702BD; Thu, 6 Mar 2025 19:24:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741289087; cv=none; b=G6kK2JNSrMSE491mztiGOJvJI7WRpxbwl4/3FmD9XnPL7ttTgNF6FX7SKbfCjTA24zDQBtys8PttS6pwISPdZuepAgmBpy7CoA9OdLv4TWLkJrH+PIJfTABy+0re3JafOhlvGdz7YI4xt80X0s31JujOzsWHBeF8+r0ERAlv1Ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741289087; c=relaxed/simple; bh=OcENzM4d2EDUOPyhxguoovCPDUxBTZG2CkJcQI650ww=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=fpiNUvDiu3dvueC1g5Ojeti8x8i55dlqQ6WcWu5GGJ9UvGbpcAnM3QNZ9J+biegSrtx4N9BbymDMpoF0/cFRAqLtLEeCX2SMgIG7wl0TCFdHL3UFL04SrTpc281YQMA4nTRsJhZj+IQ9JQOzQrXGsvt5i+tmwx7j96rp9Jo5zNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b=h34wjSXq; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxonhyperv.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxonhyperv.com header.i=@linuxonhyperv.com header.b="h34wjSXq" Received: by linux.microsoft.com (Postfix, from userid 1202) id 0BC68210EAE5; Thu, 6 Mar 2025 11:24:46 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 0BC68210EAE5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com; s=default; t=1741289086; bh=fvdydONmy1aI+e2a+ZOhB/AI/VHKMKgfGUWSmoRPceM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h34wjSXqi5/3VCp4NjO8xaUsMHbSjp+NEw1WNqbucbk99WIUOtpPO0loXGb6vzDz1 WWZwmrGDcgW+J/mfRiS7UNVYes5rSEFpNSEUgT80QH9eyTrgnPOdTCfYmngBiuhecn 4OLgKUjoZZl/0QZL4hstQgRKgwSXjUgOME0r7sng= From: longli@linuxonhyperv.com To: Jason Gunthorpe , Leon Romanovsky , Konstantin Taranov , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org, Long Li Subject: [patch rdma-next v5 2/2] RDMA/mana_ib: Handle net event for pointing to the current netdev Date: Thu, 6 Mar 2025 11:24:39 -0800 Message-Id: <1741289079-18744-2-git-send-email-longli@linuxonhyperv.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1741289079-18744-1-git-send-email-longli@linuxonhyperv.com> References: <1741289079-18744-1-git-send-email-longli@linuxonhyperv.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Long Li When running under Hyper-V, the master device to the RDMA device is always bonded to this RDMA device. This is not user-configurable. The master device can be unbind/bind from the kernel. During those events, the RDMA device should set to the current netdev to reflect the change of master device from those events. Signed-off-by: Long Li --- Changes v2: Add missing error handling when register_netdevice_notifier() fails. v3: Change mana_get_primary_netdev() to return with netdev refcount held. v4: use netdev_put(). v5: use netdevice_tracker for netdev_hold()/netdev_put(). drivers/infiniband/hw/mana/device.c | 47 ++++++++++++++++++++++++++-- drivers/infiniband/hw/mana/mana_ib.h | 1 + 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index 363566095501..b0b866b574a0 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -51,6 +51,38 @@ static const struct ib_device_ops mana_ib_dev_ops = { ib_ind_table), }; +static int mana_ib_netdev_event(struct notifier_block *this, + unsigned long event, void *ptr) +{ + struct mana_ib_dev *dev = container_of(this, struct mana_ib_dev, nb); + struct net_device *event_dev = netdev_notifier_info_to_dev(ptr); + struct gdma_context *gc = dev->gdma_dev->gdma_context; + struct mana_context *mc = gc->mana.driver_data; + struct net_device *ndev; + + /* Only process events from our parent device */ + if (event_dev != mc->ports[0]) + return NOTIFY_DONE; + + switch (event) { + case NETDEV_CHANGEUPPER: + ndev = mana_get_primary_netdev(mc, 0, &dev->dev_tracker); + /* + * RDMA core will setup GID based on updated netdev. + * It's not possible to race with the core as rtnl lock is being + * held. + */ + ib_device_set_netdev(&dev->ib_dev, ndev, 1); + + /* mana_get_primary_netdev() returns ndev with refcount held */ + netdev_put(ndev, &dev->dev_tracker); + + return NOTIFY_OK; + default: + return NOTIFY_DONE; + } +} + static int mana_ib_probe(struct auxiliary_device *adev, const struct auxiliary_device_id *id) { @@ -108,17 +140,25 @@ static int mana_ib_probe(struct auxiliary_device *adev, } dev->gdma_dev = &mdev->gdma_context->mana_ib; + dev->nb.notifier_call = mana_ib_netdev_event; + ret = register_netdevice_notifier(&dev->nb); + if (ret) { + ibdev_err(&dev->ib_dev, "Failed to register net notifier, %d", + ret); + goto deregister_device; + } + ret = mana_ib_gd_query_adapter_caps(dev); if (ret) { ibdev_err(&dev->ib_dev, "Failed to query device caps, ret %d", ret); - goto deregister_device; + goto deregister_net_notifier; } ret = mana_ib_create_eqs(dev); if (ret) { ibdev_err(&dev->ib_dev, "Failed to create EQs, ret %d", ret); - goto deregister_device; + goto deregister_net_notifier; } ret = mana_ib_gd_create_rnic_adapter(dev); @@ -147,6 +187,8 @@ static int mana_ib_probe(struct auxiliary_device *adev, mana_ib_gd_destroy_rnic_adapter(dev); destroy_eqs: mana_ib_destroy_eqs(dev); +deregister_net_notifier: + unregister_netdevice_notifier(&dev->nb); deregister_device: mana_gd_deregister_device(dev->gdma_dev); free_ib_device: @@ -162,6 +204,7 @@ static void mana_ib_remove(struct auxiliary_device *adev) xa_destroy(&dev->qp_table_wq); mana_ib_gd_destroy_rnic_adapter(dev); mana_ib_destroy_eqs(dev); + unregister_netdevice_notifier(&dev->nb); mana_gd_deregister_device(dev->gdma_dev); ib_dealloc_device(&dev->ib_dev); } diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 2638688f2505..bb9c6b1af24e 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -65,6 +65,7 @@ struct mana_ib_dev { struct xarray qp_table_wq; struct mana_ib_adapter_caps adapter_caps; netdevice_tracker dev_tracker; + struct notifier_block nb; }; struct mana_ib_wq {