From patchwork Tue Aug 15 14:51:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Pirko X-Patchwork-Id: 13353880 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBBA413AE1 for ; Tue, 15 Aug 2023 14:52:05 +0000 (UTC) Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8205719A4 for ; Tue, 15 Aug 2023 07:52:00 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-3fe426b86a8so51362515e9.3 for ; Tue, 15 Aug 2023 07:52:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20221208.gappssmtp.com; s=20221208; t=1692111119; x=1692715919; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LQjn1wcD2TcMPnivS8Au5P/abNagCYVuKRHkyUuFKlE=; b=c1IelibR2BsXg7VepaxkJ0k2E7iAr9O/B+LCW8sGbPcbGFuC3yFX8KBE2NO97JJbFH 8YOo3q6WfJTbddobuMwpKsIZlCAR3HP+HyZEqM1weQcqRbNibIeTm5zxT/PNEPfkLXn/ Ge8YVZQ2xEcl1F2pbLm41llTeC1BQrQxml+AJ/5n9MUvMp9OVkdupDffMSdkT7qwY7GT qk1FzgMXskCi1/GNnjfjsBUN9zZySUggYENUiGviwviE+Qkak7XkRDc8whOCIBJPJRGn NRQOLwMpq5hGEEDOYPCTDKqNUZYx7dzonIYqbxzgvtx5CzFt3WfZcA/NIY1n+R875xIX TU5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692111119; x=1692715919; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LQjn1wcD2TcMPnivS8Au5P/abNagCYVuKRHkyUuFKlE=; b=avpNROsKrlS0JbBKc6oYlTaO5fGVd6BF6OYiyF0F+Kj1pgeor2a0PrzGgfqlg6uVnO zMm1jaksRi0EFasGt8JmfU3RnVb3axh9GkSbSyHDgiZC5MoL9yOrV9WQhiiCxWp4REPe 7WLrrzadIveGXgAQJOhPQl2ZxbOi8Ec4wrNbcUz+ER+vy2+v1OsiwcP7+ovsAckTOgir qDW7KIndl1EyRK+ffjUq3J8CPVqhi/XJ1DELuZPZ2c7mUFLc20j4CNv1fNPOdc7+cKU7 A6cmpgaVQCnBJFQyALZNlxK/Ug9Dyung1du6X/NhoyO2+D4YpjPzspZ3CJ0NT1l0JyeR fuIg== X-Gm-Message-State: AOJu0YxZAt7utgXRRnO6kHkoyOgg1Blaj65qcOlntBK58XyFWuW+nwZk BI7PJsbCT5e9FJZ5gBdbgI9ooI83wVjCNIcNLrgSJUWc X-Google-Smtp-Source: AGHT+IFHX/MPMYp5wXbRO7XttS2jZ5d2+GqaXfanJ94kFOldhDgG4+Alvrd6MGicEzKs6jrwU7xYyw== X-Received: by 2002:a05:600c:ac4:b0:3f5:146a:c79d with SMTP id c4-20020a05600c0ac400b003f5146ac79dmr10108489wmr.15.1692111119129; Tue, 15 Aug 2023 07:51:59 -0700 (PDT) Received: from localhost ([212.23.236.67]) by smtp.gmail.com with ESMTPSA id z15-20020a1c4c0f000000b003fc0505be19sm18070055wmf.37.2023.08.15.07.51.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Aug 2023 07:51:58 -0700 (PDT) From: Jiri Pirko To: netdev@vger.kernel.org Cc: kuba@kernel.org, pabeni@redhat.com, davem@davemloft.net, edumazet@google.com, moshe@nvidia.com, saeedm@nvidia.com, shayd@nvidia.com, leon@kernel.org Subject: [patch net-next 1/4] net/mlx5: Disable eswitch as the first thing in mlx5_unload() Date: Tue, 15 Aug 2023 16:51:52 +0200 Message-ID: <20230815145155.1946926-2-jiri@resnulli.us> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230815145155.1946926-1-jiri@resnulli.us> References: <20230815145155.1946926-1-jiri@resnulli.us> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org From: Jiri Pirko The eswitch disable call does removal of all representors. Do that before clearing the SF device table and maintain the same flow as during SF devlink port removal, where the representor is remove before the actual SF is removed. Signed-off-by: Jiri Pirko --- drivers/net/ethernet/mellanox/mlx5/core/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index f4fe06a5042e..4d36066e2f7a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1406,9 +1406,9 @@ static int mlx5_load(struct mlx5_core_dev *dev) static void mlx5_unload(struct mlx5_core_dev *dev) { + mlx5_eswitch_disable(dev->priv.eswitch); mlx5_devlink_traps_unregister(priv_to_devlink(dev)); mlx5_sf_dev_table_destroy(dev); - mlx5_eswitch_disable(dev->priv.eswitch); mlx5_sriov_detach(dev); mlx5_lag_remove_mdev(dev); mlx5_ec_cleanup(dev); From patchwork Tue Aug 15 14:51:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Pirko X-Patchwork-Id: 13353881 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50FCC14270 for ; Tue, 15 Aug 2023 14:52:09 +0000 (UTC) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 714B219BF for ; Tue, 15 Aug 2023 07:52:02 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-3fbd33a57b6so54896515e9.2 for ; Tue, 15 Aug 2023 07:52:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20221208.gappssmtp.com; s=20221208; t=1692111121; x=1692715921; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f7StQvKqkD/ig3WWtJjQD+rWxMN/0SjBGTnb9BrUQXs=; b=UHfb9+xtlzdV/7pP80ZydLcMitH1rFnlv3QPLPf6RhH3vB9/OwAQPTxi2jBNDv90IM dbIPnCVFoeZ+CjZsQjO05Q9vq9kGwTYvn7L8A++yyxyGcGnRlHCgm3Qe7ryo+3Tk3qaY SW4QtRxE6bpxSLrBBQ0vegvZofpZEbARTSoqIlXL2yiqNi7a0WSwjlcIstGtkHL/U5gT VDvaMonUnlO/qpcv2F4wpr2p/4oBrI7A5/Qo3xsvYW+gYaZxRVrEyO1VkDQwN7BvyYPZ DtpY2aLDte+6ORe6Y8yz7dMCKY9hXIyK0D8oWQWEKB8J0fQosbBULScSpkwX7MyU32oe 0xSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692111121; x=1692715921; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f7StQvKqkD/ig3WWtJjQD+rWxMN/0SjBGTnb9BrUQXs=; b=ipRVHzjU9WYtJK6zAM68YcOfof9V5Grf5AdS2vGPu000VZtya1wkWq1WM1skt/B3if IqswaY+Bkp5M2v0L+ezqXqHlGql6dooTyTgLQ/ro12Zuhec7M7fj7rVv1bKvaX3cV4lv OV1LKgJOtnjAu3Mf6gw3AgEM3943SuKEWzhsNNlh1te7viz9M8Q91F1IAuyiDvveefW1 lb54aSPR+rl5bbftAG2FQJhcCcPojMe86wMguuOtrYnI6EZ/lD2RwqHNteXqM2NCHxHO Yh9+fz0mu5vvnCFJQxmp9VsVEWF4XlT9sCpUhbVjh0jYqp8ZtJODB1KUB/kSKcfnR7nW /VEg== X-Gm-Message-State: AOJu0YxyTUjVsPDsbf4LrKJDDav88OmKRrr2o+VHcGAoXaBWa0Ma8T9O eSWrlpdvqhzykK2akdRhgbdfDhzIGoHkk30EfoFABBHg X-Google-Smtp-Source: AGHT+IH8qwweIIa6kGpsITVVxAczYdG8ux6KM2OO6dFLTZbUR5a3BB/p5ispFfTUgx6Rx0dJKi+RKQ== X-Received: by 2002:adf:ef02:0:b0:317:3c89:7f03 with SMTP id e2-20020adfef02000000b003173c897f03mr10002815wro.5.1692111120736; Tue, 15 Aug 2023 07:52:00 -0700 (PDT) Received: from localhost ([212.23.236.67]) by smtp.gmail.com with ESMTPSA id t25-20020a1c7719000000b003fe4ca8decdsm21022031wmi.31.2023.08.15.07.51.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Aug 2023 07:52:00 -0700 (PDT) From: Jiri Pirko To: netdev@vger.kernel.org Cc: kuba@kernel.org, pabeni@redhat.com, davem@davemloft.net, edumazet@google.com, moshe@nvidia.com, saeedm@nvidia.com, shayd@nvidia.com, leon@kernel.org Subject: [patch net-next 2/4] net/mlx5: Lift reload limitation when SFs are present Date: Tue, 15 Aug 2023 16:51:53 +0200 Message-ID: <20230815145155.1946926-3-jiri@resnulli.us> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230815145155.1946926-1-jiri@resnulli.us> References: <20230815145155.1946926-1-jiri@resnulli.us> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org From: Jiri Pirko Historically, the shared devlink_mutex prevented devlink instances from being registered/unregistered during another devlink instance reload operation. However, devlink_muxex is gone for some time now, this limitation is no longer needed. Lift it. Signed-off-by: Jiri Pirko --- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 3d82ec890666..6eea65686c14 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -138,7 +138,6 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change, { struct mlx5_core_dev *dev = devlink_priv(devlink); struct pci_dev *pdev = dev->pdev; - bool sf_dev_allocated; int ret = 0; if (mlx5_dev_is_lightweight(dev)) { @@ -148,16 +147,6 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change, return 0; } - sf_dev_allocated = mlx5_sf_dev_allocated(dev); - if (sf_dev_allocated) { - /* Reload results in deleting SF device which further results in - * unregistering devlink instance while holding devlink_mutext. - * Hence, do not support reload. - */ - NL_SET_ERR_MSG_MOD(extack, "reload is unsupported when SFs are allocated"); - return -EOPNOTSUPP; - } - if (mlx5_lag_is_active(dev)) { NL_SET_ERR_MSG_MOD(extack, "reload is unsupported in Lag mode"); return -EOPNOTSUPP; From patchwork Tue Aug 15 14:51:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Pirko X-Patchwork-Id: 13353882 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66BDE14270 for ; Tue, 15 Aug 2023 14:52:10 +0000 (UTC) Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6A4E1BC2 for ; Tue, 15 Aug 2023 07:52:03 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-3fe490c05c9so37785815e9.0 for ; Tue, 15 Aug 2023 07:52:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20221208.gappssmtp.com; s=20221208; t=1692111122; x=1692715922; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=96rV7vulxyafEpNNLIvobkaEuUWlM7rFcKC40bxHGLQ=; b=jYCAaFGYinoJ2ZMhrlviCGmLx5m8J+Lasdmci5EmpII1iT1BxYYdnRKWz5F2J2PFd1 fUODtFVht+QnxvUNBCaPv59AgcMuZzWVdfhLRBOAKjlC4YqmuUwAevGACdNuSItG0yx9 RVH4bfANPNg5zzbNDaQWb7Iv2SYRz71D70uwdjG8oR4k33IdTwIasBvJhRaNd1cI2E8Z vvExJqPDGw7V77gyYDVBzZAa++GUl2rpjsVp5x98zc+5WteJXsEZdnsNLFyTMmNa8D0k Fyu1jfmZ1Y5tg6KaV36vZokRUgPdg23/o1WMjoCx0JIwyksGP+uKvK4M/IMpgDc0IP8t v1ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692111122; x=1692715922; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=96rV7vulxyafEpNNLIvobkaEuUWlM7rFcKC40bxHGLQ=; b=HSiZJIHvjNswkfj/jyS/4OpyC08TpxEYuNIvAxhlKe0/ag8QNvbyNP3te7rmDdWubn mg3oEb1nObbZZLMkv7SB3VkLSN40IeWtYeT3DY1Y9Ss9HPvBjEBudPxze0+HMOeJqylW TmXdtT7XXCSs2Q787xjh5ChPbzLOFxt2SjM9cSF3C7hj3dfCrDMAgSrQHKUiRoMA/SIS IgfyKmIxgxy9nVrtFge1p5AfFlZkv36FkuhY2mycbNhL2M54abNiCYE+xM21mPt5qY4b qf7hc/vZ75MP7JqX3e4ultKEKZtkGJqJGI8DK9qrc87lmssNbRYwW7TnImHcpNnnhS07 J4Vg== X-Gm-Message-State: AOJu0YySvBlS6HOPyGfB++JX/mRvWdX+MeSYdQz+k7M8NnxUYa2xBA3V TsGu0hxaTx4eBwgVeY9/VJeUt9cakoeuT9mLX81R9wsQ X-Google-Smtp-Source: AGHT+IGtCNiQEdmOVdrPRLWCRWrunq2n5/kkUZ2SRWeqQSC1/RDwH9OCtm7X+lQiQCB6JnH5cz/uHQ== X-Received: by 2002:a5d:6a45:0:b0:317:52ba:81f2 with SMTP id t5-20020a5d6a45000000b0031752ba81f2mr1686873wrw.16.1692111122383; Tue, 15 Aug 2023 07:52:02 -0700 (PDT) Received: from localhost ([212.23.236.67]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c378d00b003fe2de3f94fsm18011150wmr.12.2023.08.15.07.52.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Aug 2023 07:52:01 -0700 (PDT) From: Jiri Pirko To: netdev@vger.kernel.org Cc: kuba@kernel.org, pabeni@redhat.com, davem@davemloft.net, edumazet@google.com, moshe@nvidia.com, saeedm@nvidia.com, shayd@nvidia.com, leon@kernel.org Subject: [patch net-next 3/4] devlink: expose peer SF devlink instance Date: Tue, 15 Aug 2023 16:51:54 +0200 Message-ID: <20230815145155.1946926-4-jiri@resnulli.us> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230815145155.1946926-1-jiri@resnulli.us> References: <20230815145155.1946926-1-jiri@resnulli.us> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org From: Jiri Pirko Introduce a new helper devl_port_fn_devlink_set() to be used by driver assigning a devlink instance to the peer devlink port function. Expose this to user over new netlink attribute nested under port function nest to expose devlink handle related to the port function. This is particularly helpful for user to understand the relationship between devlink instances created for SFs and the port functions they belong to. Note that caller of devlink_port_notify() needs to hold devlink instance lock, put the assertion to devl_port_fn_devlink_set() to make this requirement explicit. Also note the limitations that only allow to make this assignment for registered objects. Signed-off-by: Jiri Pirko --- include/net/devlink.h | 4 ++++ include/uapi/linux/devlink.h | 1 + net/devlink/leftover.c | 45 +++++++++++++++++++++++++++++++++--- 3 files changed, 47 insertions(+), 3 deletions(-) diff --git a/include/net/devlink.h b/include/net/devlink.h index f7fec0791acc..49662d44471c 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -150,6 +150,8 @@ struct devlink_port { struct devlink_rate *devlink_rate; struct devlink_linecard *linecard; + + struct devlink *fn_devlink; /* Peer function devlink instance */ }; struct devlink_port_new_attrs { @@ -1667,6 +1669,8 @@ void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port, u32 contro void devlink_port_attrs_pci_sf_set(struct devlink_port *devlink_port, u32 controller, u16 pf, u32 sf, bool external); +void devl_port_fn_devlink_set(struct devlink_port *devlink_port, + struct devlink *fn_devlink); struct devlink_rate * devl_rate_node_create(struct devlink *devlink, void *priv, char *node_name, struct devlink_rate *parent); diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 3782d4219ac9..dd96086860ca 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -676,6 +676,7 @@ enum devlink_port_function_attr { DEVLINK_PORT_FN_ATTR_STATE, /* u8 */ DEVLINK_PORT_FN_ATTR_OPSTATE, /* u8 */ DEVLINK_PORT_FN_ATTR_CAPS, /* bitfield32 */ + DEVLINK_PORT_FN_ATTR_DEVLINK, /* nested */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1 diff --git a/net/devlink/leftover.c b/net/devlink/leftover.c index 3883a90d32bb..89172c5e3eaa 100644 --- a/net/devlink/leftover.c +++ b/net/devlink/leftover.c @@ -591,11 +591,12 @@ devlink_region_snapshot_get_by_id(struct devlink_region *region, u32 id) return NULL; } -static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink) +static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink, + int attrtype) { struct nlattr *nested_attr; - nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK); + nested_attr = nla_nest_start(msg, attrtype); if (!nested_attr) return -EMSGSIZE; if (devlink_nl_put_handle(msg, devlink)) @@ -884,6 +885,15 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por if (err) goto out; err = devlink_port_fn_state_fill(port, msg, extack, &msg_updated); + if (err) + goto out; + if (port->fn_devlink) { + err = devlink_nl_put_nested_handle(msg, port->fn_devlink, + DEVLINK_PORT_FN_ATTR_DEVLINK); + if (!err) + msg_updated = true; + } + out: if (err || !msg_updated) nla_nest_cancel(msg, function_attr); @@ -1785,7 +1795,8 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg, } if (linecard->nested_devlink && - devlink_nl_put_nested_handle(msg, linecard->nested_devlink)) + devlink_nl_put_nested_handle(msg, linecard->nested_devlink, + DEVLINK_ATTR_NESTED_DEVLINK)) goto nla_put_failure; genlmsg_end(msg, hdr); @@ -7133,6 +7144,34 @@ void devlink_port_attrs_pci_sf_set(struct devlink_port *devlink_port, u32 contro } EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_sf_set); +/** + * devl_port_fn_devlink_set - Attach/detach peer devlink + * instance to port function. + * @devlink_port: devlink port + * @fn_devlink: devlink instance to attach or NULL to detach + */ +void devl_port_fn_devlink_set(struct devlink_port *devlink_port, + struct devlink *fn_devlink) +{ + lockdep_assert_held(&devlink_port->devlink->lock); + ASSERT_DEVLINK_PORT_REGISTERED(devlink_port); + + if (fn_devlink) + ASSERT_DEVLINK_REGISTERED(fn_devlink); + else if (WARN_ON(!devlink_port->fn_devlink)) + return; + else + ASSERT_DEVLINK_REGISTERED(devlink_port->fn_devlink); + + if (WARN_ON(devlink_port->attrs.flavour != DEVLINK_PORT_FLAVOUR_PCI_SF || + devlink_port->attrs.pci_sf.external)) + return; + + devlink_port->fn_devlink = fn_devlink; + devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW); +} +EXPORT_SYMBOL_GPL(devl_port_fn_devlink_set); + /** * devl_rate_node_create - create devlink rate node * @devlink: devlink instance From patchwork Tue Aug 15 14:51:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Pirko X-Patchwork-Id: 13353883 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D7C914AA4 for ; Tue, 15 Aug 2023 14:52:15 +0000 (UTC) Received: from mail-lj1-x230.google.com (mail-lj1-x230.google.com [IPv6:2a00:1450:4864:20::230]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 432561737 for ; Tue, 15 Aug 2023 07:52:06 -0700 (PDT) Received: by mail-lj1-x230.google.com with SMTP id 38308e7fff4ca-2b9bb097c1bso83467851fa.0 for ; Tue, 15 Aug 2023 07:52:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20221208.gappssmtp.com; s=20221208; t=1692111124; x=1692715924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9Qjd/sm0WDyLfJrhyXPh4PnDUSuxLyKwf3hwfnGNrmA=; b=gBtkVkvAE0RSZER36vp9fogxMRc14mwbUOnvLuAuUv561YS5Cvh9c6+KbXoPAep+EM bb214gKYblEETKbsE8sRvdTmWAa4Yzmc3kXaEf/HZH8YmEapkRJPK9ob0/YhK+1maJxp pYcbKo0Nxft+VvRCUcGbXV2t02FzYWxrh8yDTFCQkyoSG+6VZ3BFnvCNjzcZ/ViJA7jG VPfw8FG7lT9BoVhEW3VQNjWHp5YWRIfxn57x5FI3V/oDeRsieCxNSV8XjqoXFFoC1JJ6 om0b+k0UX1oAcbRLdK1Jjew2+w8kwD3DuahQ0FXUDmCl1DCiTYsQ8rxJyAp/NfqJEewN aRYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692111124; x=1692715924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Qjd/sm0WDyLfJrhyXPh4PnDUSuxLyKwf3hwfnGNrmA=; b=iMx8WtyavLz5PUVUgvUtAUEASDBXs82GvmoQ0xF+Shoa0hYSZyjOKePz2i6QfdGAZU 2V/Xklg65dfsQo7gge93netmqQk3aUVebXgKMvTjaRq753AuYv4ezjT0pCixZjSInppL L93SCJ+LzV9+y1xJRvCW2K0eOadtQ8JW77lquaDNrXLbF7+1lFODyzDQr3xJvN1APi+G UEeVoqk6xuTz8j8nzoM6MKhNWcktskReq+9kIm93pISNbSTEgUF8rQi2DfzYrjefb7ku pbqaH+xaicIiha1BJNxmM9Pmv78uR/McrW5yc/lrVr6uwZFmNboNfHENEEK0Au2MQXc3 dvgg== X-Gm-Message-State: AOJu0YxtQw1JgwEJyOo4AOWhvLvEqGExLut7g5M8Y0RUA5hYUTKmr49K WF7JeaZzYGUKkTpqbtaCt09UE5Uo0dIMqlV8gXtNM+2f X-Google-Smtp-Source: AGHT+IFGbh3ABAoKUgySJhmA4aSYKebk+o4rkZN7OO20eNVciU/Naj9yfgoQ4qZ3NFFaWvyqB3T5hQ== X-Received: by 2002:a2e:9642:0:b0:2b8:39e4:2e2c with SMTP id z2-20020a2e9642000000b002b839e42e2cmr9333494ljh.1.1692111124361; Tue, 15 Aug 2023 07:52:04 -0700 (PDT) Received: from localhost ([212.23.236.67]) by smtp.gmail.com with ESMTPSA id 8-20020a05600c024800b003fd2d33ea53sm17896347wmj.14.2023.08.15.07.52.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Aug 2023 07:52:03 -0700 (PDT) From: Jiri Pirko To: netdev@vger.kernel.org Cc: kuba@kernel.org, pabeni@redhat.com, davem@davemloft.net, edumazet@google.com, moshe@nvidia.com, saeedm@nvidia.com, shayd@nvidia.com, leon@kernel.org Subject: [patch net-next 4/4] net/mlx5: SF, Implement peer devlink set for SF representor devlink port Date: Tue, 15 Aug 2023 16:51:55 +0200 Message-ID: <20230815145155.1946926-5-jiri@resnulli.us> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230815145155.1946926-1-jiri@resnulli.us> References: <20230815145155.1946926-1-jiri@resnulli.us> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org From: Jiri Pirko Benefit from the existence of internal mlx5 notifier and extend it by event MLX5_DRIVER_EVENT_SF_PEER_DEVLINK. Use this event from SF auxiliary device probe/remove functions to pass the registered SF devlink instance to the SF representor. Process the new event in SF representor code and call devl_port_fn_devlink_set() to do the assignments. Implement this in work to avoid possible deadlock when probe/remove function of SF may be called with devlink instance lock held during devlink reload. Signed-off-by: Jiri Pirko --- .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 5 ++ .../mellanox/mlx5/core/sf/dev/driver.c | 14 ++++ .../ethernet/mellanox/mlx5/core/sf/devlink.c | 75 +++++++++++++++++++ include/linux/mlx5/device.h | 1 + 4 files changed, 95 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h index 2a66a427ef15..c2e3277a418a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h @@ -19,6 +19,11 @@ struct mlx5_sf_dev { u16 fn_id; }; +struct mlx5_sf_peer_devlink_event_info { + u16 fn_id; + struct devlink *devlink; +}; + void mlx5_sf_dev_table_create(struct mlx5_core_dev *dev); void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c index 8fe82f1191bb..7e45b338eb54 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c @@ -8,6 +8,18 @@ #include "dev.h" #include "devlink.h" +static void mlx5_core_peer_devlink_set(struct mlx5_sf_dev *sf_dev, struct devlink *devlink) +{ + struct mlx5_sf_peer_devlink_event_info event_info = { + .fn_id = sf_dev->fn_id, + .devlink = devlink, + }; + + mlx5_blocking_notifier_call_chain(sf_dev->parent_mdev, + MLX5_DRIVER_EVENT_SF_PEER_DEVLINK, + &event_info); +} + static int mlx5_sf_dev_probe(struct auxiliary_device *adev, const struct auxiliary_device_id *id) { struct mlx5_sf_dev *sf_dev = container_of(adev, struct mlx5_sf_dev, adev); @@ -55,6 +67,7 @@ static int mlx5_sf_dev_probe(struct auxiliary_device *adev, const struct auxilia goto init_one_err; } devlink_register(devlink); + mlx5_core_peer_devlink_set(sf_dev, devlink); return 0; init_one_err: @@ -72,6 +85,7 @@ static void mlx5_sf_dev_remove(struct auxiliary_device *adev) struct devlink *devlink = priv_to_devlink(sf_dev->mdev); mlx5_drain_health_wq(sf_dev->mdev); + mlx5_core_peer_devlink_set(sf_dev, NULL); devlink_unregister(devlink); if (mlx5_dev_is_lightweight(sf_dev->mdev)) mlx5_uninit_one_light(sf_dev->mdev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c index 6a3fa30b2bf2..06753032a9f1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c @@ -18,6 +18,10 @@ struct mlx5_sf { u16 id; u16 hw_fn_id; u16 hw_state; + struct mlx5_core_dev *dev; + struct work_struct peer_devlink_set_work; + bool peer_devlink_set_work_disabled; + struct devlink *peer_devlink; }; struct mlx5_sf_table { @@ -28,6 +32,7 @@ struct mlx5_sf_table { struct mutex sf_state_lock; /* Serializes sf state among user cmds & vhca event handler. */ struct notifier_block esw_nb; struct notifier_block vhca_nb; + struct notifier_block mdev_nb; }; static struct mlx5_sf * @@ -59,6 +64,36 @@ static void mlx5_sf_id_erase(struct mlx5_sf_table *table, struct mlx5_sf *sf) xa_erase(&table->port_indices, sf->port_index); } +static void mlx5_sf_peer_devlink_set_work_flush(struct mlx5_sf *sf) +{ + flush_work(&sf->peer_devlink_set_work); +} + +static void mlx5_sf_peer_devlink_set_work_queue(struct mlx5_sf *sf) +{ + if (sf->peer_devlink_set_work_disabled) + return; + mlx5_events_work_enqueue(sf->dev, &sf->peer_devlink_set_work); +} + +static void mlx5_sf_peer_devlink_set_work_disable(struct mlx5_sf *sf) +{ + sf->peer_devlink_set_work_disabled = true; + cancel_work_sync(&sf->peer_devlink_set_work); +} + +static void mlx5_sf_peer_devlink_set_work(struct work_struct *work) +{ + struct mlx5_sf *sf = container_of(work, struct mlx5_sf, peer_devlink_set_work); + + if (!devl_trylock(sf->dl_port.devlink)) { + mlx5_sf_peer_devlink_set_work_queue(sf); + return; + } + devl_port_fn_devlink_set(&sf->dl_port, sf->peer_devlink); + devl_unlock(sf->dl_port.devlink); +} + static struct mlx5_sf * mlx5_sf_alloc(struct mlx5_sf_table *table, struct mlx5_eswitch *esw, u32 controller, u32 sfnum, struct netlink_ext_ack *extack) @@ -93,6 +128,9 @@ mlx5_sf_alloc(struct mlx5_sf_table *table, struct mlx5_eswitch *esw, sf->hw_state = MLX5_VHCA_STATE_ALLOCATED; sf->controller = controller; + sf->dev = table->dev; + INIT_WORK(&sf->peer_devlink_set_work, &mlx5_sf_peer_devlink_set_work); + err = mlx5_sf_id_insert(table, sf); if (err) goto insert_err; @@ -296,6 +334,7 @@ static int mlx5_sf_add(struct mlx5_core_dev *dev, struct mlx5_sf_table *table, new_attr->controller, new_attr->sfnum); if (err) goto esw_err; + *dl_port = &sf->dl_port; trace_mlx5_sf_add(dev, sf->port_index, sf->controller, sf->hw_fn_id, new_attr->sfnum); return 0; @@ -400,6 +439,7 @@ int mlx5_devlink_sf_port_del(struct devlink *devlink, goto sf_err; } + mlx5_sf_peer_devlink_set_work_disable(sf); mlx5_esw_offloads_sf_vport_disable(esw, sf->hw_fn_id); mlx5_sf_id_erase(table, sf); @@ -472,6 +512,7 @@ static void mlx5_sf_deactivate_all(struct mlx5_sf_table *table) * arrive. It is safe to destroy all user created SFs. */ xa_for_each(&table->port_indices, index, sf) { + mlx5_sf_peer_devlink_set_work_disable(sf); mlx5_esw_offloads_sf_vport_disable(esw, sf->hw_fn_id); mlx5_sf_id_erase(table, sf); mlx5_sf_dealloc(table, sf); @@ -511,6 +552,36 @@ static int mlx5_sf_esw_event(struct notifier_block *nb, unsigned long event, voi return 0; } +static int mlx5_sf_mdev_event(struct notifier_block *nb, unsigned long event, void *data) +{ + struct mlx5_sf_table *table = container_of(nb, struct mlx5_sf_table, mdev_nb); + struct mlx5_sf_peer_devlink_event_info *event_info = data; + int ret = NOTIFY_DONE; + struct mlx5_sf *sf; + + if (event != MLX5_DRIVER_EVENT_SF_PEER_DEVLINK) + return NOTIFY_DONE; + + table = mlx5_sf_table_try_get(table->dev); + if (!table) + return NOTIFY_DONE; + + mutex_lock(&table->sf_state_lock); + sf = mlx5_sf_lookup_by_function_id(table, event_info->fn_id); + if (!sf) + goto out; + + mlx5_sf_peer_devlink_set_work_flush(sf); + sf->peer_devlink = event_info->devlink; + mlx5_sf_peer_devlink_set_work_queue(sf); + + ret = NOTIFY_OK; +out: + mutex_unlock(&table->sf_state_lock); + mlx5_sf_table_put(table); + return ret; +} + static bool mlx5_sf_table_supported(const struct mlx5_core_dev *dev) { return dev->priv.eswitch && MLX5_ESWITCH_MANAGER(dev) && @@ -544,6 +615,9 @@ int mlx5_sf_table_init(struct mlx5_core_dev *dev) if (err) goto vhca_err; + table->mdev_nb.notifier_call = mlx5_sf_mdev_event; + mlx5_blocking_notifier_register(dev, &table->mdev_nb); + return 0; vhca_err: @@ -562,6 +636,7 @@ void mlx5_sf_table_cleanup(struct mlx5_core_dev *dev) if (!table) return; + mlx5_blocking_notifier_unregister(dev, &table->mdev_nb); mlx5_vhca_event_notifier_unregister(table->dev, &table->vhca_nb); mlx5_esw_event_notifier_unregister(dev->priv.eswitch, &table->esw_nb); WARN_ON(refcount_read(&table->refcount)); diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 80cc12a9a531..2473743f36b2 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -364,6 +364,7 @@ enum mlx5_event { enum mlx5_driver_event { MLX5_DRIVER_EVENT_TYPE_TRAP = 0, MLX5_DRIVER_EVENT_UPLINK_NETDEV, + MLX5_DRIVER_EVENT_SF_PEER_DEVLINK, }; enum {