From patchwork Tue Apr 2 11:25:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 13613761 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2062.outbound.protection.outlook.com [40.107.93.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B2F14CB4A; Tue, 2 Apr 2024 11:26:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712057183; cv=fail; b=ITp1EF56vmlgaSYuQNAcakcc9RmOiYuRSc7yn2E1AvrRo/EFs0ltt2smvlW02k6rGEtwH1kRDS33sVhOAQKkfsJCFsqwhpFvSDWI+4WJh7wO6iSkWz28G16Jf/eSrbHmS82l42DSkXfuG/M1n6RSOOqoSvHNgtcGwcWOuEAOcNA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712057183; c=relaxed/simple; bh=MpokWOljzVHpeyGImRa2p6hUhIrw6JtieaBS4zSoHes=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ALowlaibCLmdiPSC8wxV1fChG7ELwAS8ZrqnHn4S3kdPWvp93Wl/mvzw6o9+AWQx2ch7H8O+op4QRyOGIFSvg44hLMmmkZLiFY2906GFxoTwgJIhmgzzcIAaz6aLXqwMcuVTrtZjZc45dgnqkEW45x2nhJrSFrdUVPy79Rw4qMg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=neb06cUB; arc=fail smtp.client-ip=40.107.93.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="neb06cUB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mObalgLhmjAA81lZd0ELmqQCCykzHotOAiFE+dKFq26II6jjkVfdFpTpTZ2Ul5hKlUnBYAH+CU8vKpJrcnP4psS6n7KG0Mxrd6LnOU9ZDIxo5mpr5X8jWqtI3RiKS4OUM/hfYVxhfK1atmtro+dqngqf8Ha33s6q9s/WuOwuqYLcW0pn7KvykVS2Lw1j4mXHT34t7HHMnhNWSoUGWSbnWTlIyUFNlEQOC9E/a4AwLCRDee+NB3DPOI+2u4OPjL+Rk4+xYHSxpUgbcj+l+G76BPzw1sJU5XtZwkSo6qEANMjexkRaBnY9QyTS+B+qtvULQicQ5RSbqJUzTtwVXcUrlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zNUyV/MGjUBQvkJQCgosiKq1fBm4kagmzRs6s6mUwNk=; b=IoiRTLyz+LsQTrZrIo4ur89weLhc8IO4ogCOJ9Z9ZBy8CpDucFROKFaW1/MxZ2Tm3ok4cS5OeCLFqUqj+XUyNTu2bxxtRsrF0g+4gMAoEsVWIsFNHfWnStWDhHnqdDIDikYJXOEU/WHtjMs3UNgNa0Zrmf/46YLPl6Mp9Ak3zUJymICM6ZCetw92imKEFaYREZod+aTsUH3/I5Xd10c/Jk7qqQ01HbOaGmLxtuZiGrySw3ycOBOjdS1BloR2VpWEoOBAkwVgudzXWNHbOorBe1digkZMtjkyT85psCa8uSfaolvMdd6UVmJcWNPEwsKN3B9uvq+DL9dVkrMdRItUtw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zNUyV/MGjUBQvkJQCgosiKq1fBm4kagmzRs6s6mUwNk=; b=neb06cUBlfDWwR03A0y9i2lZV0O9PlEDRTSRamutoKdLMB/AVBrLulUvc9B2c9vkfjVWdHVqKh6yItyuHaSxrpLI2R9atEcDHiMnjWhuKyPLiKyUpBZUcQTcKBjc3wH91j4omo3D4YsEgQ8du3e+pISWAchC/NuN7B9LcXFxWk6dlUHD4ov+pIMikOzFAH5+M6nAz8ZmD7TwoYyDw96Zzpi4mUAYTfd/mHlkTqMEp/SM52gXUZ6Zt+Xloe4y9ZtSuOnMzve3IJGgpz7DyVf7IjDUYVLPMzFGUwFq9uszZufgzDqFFg0CUqlqbPvyDAjq5iOliFACCd6QTz4XPkprCQ== Received: from SJ0PR03CA0186.namprd03.prod.outlook.com (2603:10b6:a03:2ef::11) by SA1PR12MB6823.namprd12.prod.outlook.com (2603:10b6:806:25e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Tue, 2 Apr 2024 11:26:18 +0000 Received: from SN1PEPF0002529E.namprd05.prod.outlook.com (2603:10b6:a03:2ef:cafe::78) by SJ0PR03CA0186.outlook.office365.com (2603:10b6:a03:2ef::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.34 via Frontend Transport; Tue, 2 Apr 2024 11:26:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002529E.mail.protection.outlook.com (10.167.242.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.22 via Frontend Transport; Tue, 2 Apr 2024 11:26:17 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 2 Apr 2024 04:26:05 -0700 Received: from sw-mtx-036.mtx.labs.mlnx (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Tue, 2 Apr 2024 04:26:03 -0700 From: Parav Pandit To: , , , , , , CC: , , , , , , , , Parav Pandit , Jiri Pirko Subject: [net-next v2 1/2] devlink: Support setting max_io_eqs Date: Tue, 2 Apr 2024 14:25:48 +0300 Message-ID: <20240402112549.598596-2-parav@nvidia.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20240402112549.598596-1-parav@nvidia.com> References: <20240402112549.598596-1-parav@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002529E:EE_|SA1PR12MB6823:EE_ X-MS-Office365-Filtering-Correlation-Id: 47ab9dd4-3f8e-4c91-766b-08dc5307b706 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: y1gHoizDepgQdLgMHBGLSpdV2VtOUexUSiVJYOXOMXaH2pmTDKSiPH/LA+nJeAvYOSlCZZoKSE0sOVeuJF42WR4oa35xwWHUN6VgjNvp5f4qYtLIunbMPt++oNagM0rDv0JqiAk1unHIaot1VMHANGIslSyVR4AN4gjJ1fe5EwRYQcOkv9E2Wp2a/7kr1mqWpnPwxHlKM1A4EbuB0aniNTRUEggG3L6xFqx6NoZXzf6firNX3azvJGDX4vT6aYxgKBWwZ1SV5qttEITaXBSIpPQM3H955SuVQfs/q/RrrEwwzEoxhLJCPx0FMI5V0op/8HIbpselK7NWriXOx2t6RInDYlbAoFkuh3jRxGsvjra1LnLlo/CBV+7zvZ1113dn8u/7IByQCnBi7UmQ9bL29RXypwJK1+gmJQvRbbJZZpioVokssyaTtQWns8m4G/a4DjXAZplVOsRp7vW3ysnbJ9vMbEHQXV825hxIw9F5pQYQ1FTwLdzD/HX/dYNjyM+S6h1S+OQIEdleTIwkI7KpZU6Y3alGT9N6H+U3friwojV0YCTzWZFuFcRS4qjNbs+TZvIZfM65HzagKV1DAVpVL75oQnKl/IFS0LgoIpTqG53skAEHGQdKfMVx+/1y/vdqJt5WNdegKuZ81rgErmMOTr4A7uRCohdbzdMSH6BDrOGa1HqTsgCK5mGh0CcWehH9azvI5tZcI+Gw52MSVMH4B9Xq1u4vl3Qmw9kXJOMTquWyc2bs9FBB2CehEiR6tarS X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(82310400014)(36860700004)(7416005)(376005)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2024 11:26:17.8892 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 47ab9dd4-3f8e-4c91-766b-08dc5307b706 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002529E.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6823 Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues. Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage. example: Get maximum IO event queues of the VF device:: $ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 Set maximum IO event queues of the VF device:: $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 $ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 Reviewed-by: Jiri Pirko Reviewed-by: Shay Drory Signed-off-by: Parav Pandit --- changelog: v1->v2: - limited comment to 80 chars per line in header file --- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 52 +++++++++++++++++++ 4 files changed, 92 insertions(+) diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst index 562f46b41274..451f57393f11 100644 --- a/Documentation/networking/devlink/devlink-port.rst +++ b/Documentation/networking/devlink/devlink-port.rst @@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability of the function using Users may also set the IPsec packet capability of the function using `devlink port function set ipsec_packet` command. +Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes =================== @@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or +a VF, such function driver is limited to consume only enforced +number of IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 + Subfunction ============ diff --git a/include/net/devlink.h b/include/net/devlink.h index 9ac394bdfbe4..bb1af599d101 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink); * capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. + * @port_fn_max_io_eqs_get: Callback used to get port function's maximum number + * of event queues. Should be used by device drivers to + * report the maximum event queues of a function + * managed by the devlink port. + * @port_fn_max_io_eqs_set: Callback used to set port function's maximum number + * of event queues. Should be used by device drivers to + * configure maximum number of event queues + * of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops { int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port, bool enable, struct netlink_ext_ack *extack); + int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, + u32 *max_eqs, + struct netlink_ext_ack *extack); + int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, + u32 max_eqs, + struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 2da0c7eb6710..9401aa343673 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -686,6 +686,7 @@ enum devlink_port_function_attr { DEVLINK_PORT_FN_ATTR_OPSTATE, /* u8 */ DEVLINK_PORT_FN_ATTR_CAPS, /* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK, /* nested */ + DEVLINK_PORT_FN_ATTR_MAX_IO_EQS, /* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1 diff --git a/net/devlink/port.c b/net/devlink/port.c index 118d130d2afd..307bfeedda54 100644 --- a/net/devlink/port.c +++ b/net/devlink/port.c @@ -16,6 +16,7 @@ static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK), + [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, }; #define ASSERT_DEVLINK_PORT_REGISTERED(devlink_port) \ @@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, return 0; } +static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port, + struct sk_buff *msg, + struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u32 max_io_eqs; + int err; + + if (!port->ops->port_fn_max_io_eqs_get) + return 0; + + err = port->ops->port_fn_max_io_eqs_get(port, &max_io_eqs, extack); + if (err) { + if (err == -EOPNOTSUPP) + return 0; + return err; + } + err = nla_put_u32(msg, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS, max_io_eqs); + if (err) + return err; + *msg_updated = true; + return 0; +} + int devlink_nl_port_handle_fill(struct sk_buff *msg, struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port *devlink_port, return 0; } +static int +devlink_port_fn_max_io_eqs_set(struct devlink_port *devlink_port, + const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + u32 max_io_eqs; + + max_io_eqs = nla_get_u32(attr); + return devlink_port->ops->port_fn_max_io_eqs_set(devlink_port, + max_io_eqs, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *port, struct netlink_ext_ack *extack) @@ -428,6 +465,9 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por if (err) goto out; err = devlink_port_fn_state_fill(port, msg, extack, &msg_updated); + if (err) + goto out; + err = devlink_port_fn_max_io_eqs_fill(port, msg, extack, &msg_updated); if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,11 @@ static int devlink_port_function_validate(struct devlink_port *devlink_port, } } } + if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] && !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack, tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS], + "Function does not support max_io_eqs setting"); + return -EOPNOTSUPP; + } return 0; } @@ -761,6 +806,13 @@ static int devlink_port_function_set(struct devlink_port *port, return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS]; + if (attr) { + err = devlink_port_fn_max_io_eqs_set(port, attr, extack); + if (err) + return err; + } + /* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state. From patchwork Tue Apr 2 11:25:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 13613762 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2059.outbound.protection.outlook.com [40.107.220.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9952C627E8; Tue, 2 Apr 2024 11:26:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712057185; cv=fail; b=adQ+Q5iA0A32eYkgw7UOAUIFAaSKBZvWB2GNDGRW05OHbsjhyvbRzC5xAcq9LAiaO4+jYizXDc0/g0oNA0dFL9jKMH+RE71xPOFJD8dOZsX3qjECq/i/3bPD2rACPQ1WX/JV+YPwwcT70zYbWS+iqCCiNOaZIQWEXycRrk5oiNM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712057185; c=relaxed/simple; bh=It/6SHSaixETk5v3CsVqb9Ev8JhOOChMjUsJTFvvnBo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OV15aUyV0vt9ylavmpKtkte4FMrrjviUrhlvwpQ7A2CaIei8wbdVwAoVrqPAmVU2OVA7R//HXzHg59aL3Za0hTaqX7CS5WQcKPC6jLMAuemunkHV6y2fQZQZF0AwAl2SZxd2XRgYm6rgxznGH7XTLopJFm4bSWzJC+23MtcLQvg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=BHoIF3Qw; arc=fail smtp.client-ip=40.107.220.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="BHoIF3Qw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JeTLnN8GE/eSf9vpyjv4y+qv+BK/iKn2C6Zmeb878WZ+tH0ikh8kirsqwx+5QjUgXsrJOaPGZ8nWQX/bm5M6eWvXMaUAR9pGEsH1e7EYuTcsNXGsy+Hvmj2SwGUg8CjT2C7lEOBJTZZWu8UZ3J96cAYTrwmJs20A6QNOLrWOvJg9VB/JwZ8RFjUS0meAmEAx+EzJZBAsAT+aBMCg789Ak15EIOahsC9rR+v51gdMGXp92IgLiQt3pUXgnsLjR+V0A9kXqGoPTxK9WAjmh7lEM1fSFe2GBxezcsqNj76zmwle2GEf/2pHjJgz9PSgOrZPhlUQq/QW6vZSrA0DoqCASQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+u/NLwIobXyARWsSuh6gPKrqiKlbBW1Jyplj3Dv2lzE=; b=frZG66H8QecC0kk2c1b5tQr2AWe41S5sZKbVf9A0OKmXjyo7GiwTTnQAjqCdyiT/1FaSRMiCo3+zqsQrF3rsEx84OKNx/xIMSDXXWZXbr8vQe7qizJPjYv/yMaISHv8qzRE69Kw1Etl+0YMECe1qHDLYltBPc5kV/G247Q/qyRZJOJ1EYfxAK/n823+qE7vAH2bWdN+sf+0v8zTPrGasy65PIUQ3FWch92EnpisT3MQawQE0AvY/t0z6REOFthZk03WVilnTJKDJdeKwQo+MFP7AzLrmiGuDE10Qc02kbjqNp1lDApWZPY17IqEgZ8zQ2W9iKW8wsbzixNdKFfp0oQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+u/NLwIobXyARWsSuh6gPKrqiKlbBW1Jyplj3Dv2lzE=; b=BHoIF3QwM6XzlA6grOsp6P3d/Xq3gPdH9giPuPfd6eS2Y1OkP8IjnG3Kxt6cdgrEU/HC3MQCcrJxxHvgRBmo6oAlgE3c5yqUS2SQ8FnGBWxAXMGmMK1g5Bd5RGqTBMEfKRbdQXmv2Qi9yno2on+l6kZS89WPbImcXDiYwHpS11po0KfxFszD4DRcXxzBTSbrKEFwk+q6E+Uqd23W4u3u55+L/zr3ravSi+e+1my+a8b7d3jA6T2UXkG2oIY8u78O4F7uDjBIsw/1vrC0pXExL5Q5X/n2HRQi8fsNkWifEkmS3WW5KS/nGAt9g+uUYR+/dpjZY2VoFn7ozMIpzyRbJg== Received: from SJ0PR03CA0204.namprd03.prod.outlook.com (2603:10b6:a03:2ef::29) by DS7PR12MB5885.namprd12.prod.outlook.com (2603:10b6:8:78::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Tue, 2 Apr 2024 11:26:20 +0000 Received: from SN1PEPF0002529E.namprd05.prod.outlook.com (2603:10b6:a03:2ef:cafe::cf) by SJ0PR03CA0204.outlook.office365.com (2603:10b6:a03:2ef::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.32 via Frontend Transport; Tue, 2 Apr 2024 11:26:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002529E.mail.protection.outlook.com (10.167.242.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.22 via Frontend Transport; Tue, 2 Apr 2024 11:26:19 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 2 Apr 2024 04:26:07 -0700 Received: from sw-mtx-036.mtx.labs.mlnx (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Tue, 2 Apr 2024 04:26:05 -0700 From: Parav Pandit To: , , , , , , CC: , , , , , , , , Parav Pandit , Jiri Pirko Subject: [net-next v2 2/2] mlx5/core: Support max_io_eqs for a function Date: Tue, 2 Apr 2024 14:25:49 +0300 Message-ID: <20240402112549.598596-3-parav@nvidia.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20240402112549.598596-1-parav@nvidia.com> References: <20240402112549.598596-1-parav@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002529E:EE_|DS7PR12MB5885:EE_ X-MS-Office365-Filtering-Correlation-Id: dd997a52-e657-4002-699d-08dc5307b81d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EMQc+smjnBGV3qpMZK2bIgpubd7HyuWUVcOfzsKUOYAsuw1EjSPSlsAVT2cOrG97GWgtfrWofCDkAXRLESzHBEt6jLhjCOEI2HBle0YtJ4m9+AW2ODrtdxVP5ZYhOt7rC4WNhZaDdtln2UWt3CgVRXmr0MpgDAdJ9/KfnnN0vtH3zZpqeTw3JpoqIGDpwP70weQ7D7DcD6l6lQz2t/dayythPb6vTRcd13PqgcVQwNUhz+2Md7mWoW5EiP5ti8/vr9kw5szs1Tv8HnooGh1sDgro+MQJASq7DvbSIdzToiZvhlFapxLB7WxOeRCdAvneFllOzNWtusLRPck4qwr2sx2X0pA642rPQMEf/BPJjlpFgWGwJgag/VZxT+WsBBf10Lxk5BkpamI4kcui/bJDo7mjHglKnJ8FdVdn2d75areHNP5o7R47AfmOwuKkhQRbU+GR5fyV7qurj3ICC5DkMCQKiiVr5+wvQIbXUX8X+O7Em349jSFY6JeDQYgmhUpCjHgQHYo2icBlwIloEAd9Tpf4qQ9rsujTO4w/jP/3e/wk32BKiH4YpZh5AaAVozXq+KH2rfAhl6sZcaXtapPaLR0a4647rvxBEN8+a4/PRCpeIoW7kwLfSWDAUMMiORNZPgmwTyJEya7WSX/ZHtGZ0/yUeV0mGkYm8pyGW2GRNSgWTYseziBEvI3Gm5jMyV4T9LyAe7uzkh/4rbxCQLCXLmvyyxMHOeDwonhy/giZiKOEV8Y2Cr/H0ke3IR4Nnl6Y X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(7416005)(36860700004)(82310400014)(376005)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2024 11:26:19.7173 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dd997a52-e657-4002-699d-08dc5307b81d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002529E.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5885 Implement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors. Reviewed-by: Jiri Pirko Reviewed-by: Shay Drory Signed-off-by: Parav Pandit --- changelog: v1->v2: - fixed comments from Kalesh - fixed missing kfree in get call - returning error code for get cmd failure - fixed error msg copy paste error in set on cmd failure - limited code to 80 chars limit - fixed set function variables for reverse christmas tree --- .../mellanox/mlx5/core/esw/devlink_port.c | 2 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 106 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c index d8e739cbcbce..76d1ed93c773 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c @@ -98,6 +98,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = { .port_fn_ipsec_packet_get = mlx5_devlink_port_fn_ipsec_packet_get, .port_fn_ipsec_packet_set = mlx5_devlink_port_fn_ipsec_packet_set, #endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index 349e28a6dd8d..50ce1ea20dd4 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(struct devlink_port *port, bool *is_en int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, bool enable, struct netlink_ext_ack *extack); #endif /* CONFIG_XFRM_OFFLOAD */ +int mlx5_devlink_port_fn_max_io_eqs_get(struct devlink_port *port, + u32 *max_io_eqs, + struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_max_io_eqs_set(struct devlink_port *port, + u32 max_io_eqs, + struct netlink_ext_ack *extack); + void *mlx5_eswitch_get_uplink_priv(struct mlx5_eswitch *esw, u8 rep_type); int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index baaae628b0a0..a6d68370d43a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -66,6 +66,8 @@ #define MLX5_ESW_FT_OFFLOADS_DROP_RULE (1) +#define MLX5_ESW_MAX_CTRL_EQS 4 + static struct esw_vport_tbl_namespace mlx5_esw_vport_tbl_mirror_ns = { .max_fte = MLX5_ESW_VPORT_TBL_SIZE, .max_num_groups = MLX5_ESW_VPORT_TBL_NUM_GROUPS, @@ -4557,3 +4559,98 @@ int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, return err; } #endif /* CONFIG_XFRM_OFFLOAD */ + +int +mlx5_devlink_port_fn_max_io_eqs_get(struct devlink_port *port, u32 *max_io_eqs, + struct netlink_ext_ack *extack) +{ + struct mlx5_vport *vport = mlx5_devlink_port_vport_get(port); + int query_out_sz = MLX5_ST_SZ_BYTES(query_hca_cap_out); + u16 vport_num = vport->vport; + struct mlx5_eswitch *esw; + void *query_ctx; + void *hca_caps; + u32 max_eqs; + int err; + + esw = mlx5_devlink_eswitch_nocheck_get(port->devlink); + if (!MLX5_CAP_GEN(esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCA management"); + return -EOPNOTSUPP; + } + + query_ctx = kzalloc(query_out_sz, GFP_KERNEL); + if (!query_ctx) + return -ENOMEM; + + mutex_lock(&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num, query_ctx, + MLX5_CAP_GENERAL); + if (err) { + NL_SET_ERR_MSG_MOD(extack, "Failed getting HCA caps"); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability); + max_eqs = MLX5_GET(cmd_hca_cap, hca_caps, max_num_eqs); + if (max_eqs < MLX5_ESW_MAX_CTRL_EQS) + *max_io_eqs = 0; + else + *max_io_eqs = max_eqs - MLX5_ESW_MAX_CTRL_EQS; +out: + mutex_unlock(&esw->state_lock); + kfree(query_ctx); + return err; +} + +int +mlx5_devlink_port_fn_max_io_eqs_set(struct devlink_port *port, u32 max_io_eqs, + struct netlink_ext_ack *extack) +{ + struct mlx5_vport *vport = mlx5_devlink_port_vport_get(port); + int query_out_sz = MLX5_ST_SZ_BYTES(query_hca_cap_out); + u16 max_eqs = max_io_eqs + MLX5_ESW_MAX_CTRL_EQS; + u16 vport_num = vport->vport; + struct mlx5_eswitch *esw; + void *query_ctx; + void *hca_caps; + int err; + + esw = mlx5_devlink_eswitch_nocheck_get(port->devlink); + if (!MLX5_CAP_GEN(esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCA management"); + return -EOPNOTSUPP; + } + + if (max_io_eqs + MLX5_ESW_MAX_CTRL_EQS > USHRT_MAX) { + NL_SET_ERR_MSG_MOD(extack, "Supplied value out of range"); + return -EINVAL; + } + + query_ctx = kzalloc(query_out_sz, GFP_KERNEL); + if (!query_ctx) + return -ENOMEM; + + mutex_lock(&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num, query_ctx, + MLX5_CAP_GENERAL); + if (err) { + NL_SET_ERR_MSG_MOD(extack, "Failed getting HCA caps"); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability); + MLX5_SET(cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err = mlx5_vport_set_other_func_cap(esw->dev, hca_caps, vport_num, + MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE); + mutex_unlock(&esw->state_lock); + if (err) + NL_SET_ERR_MSG_MOD(extack, "Failed setting HCA caps"); + +out: + kfree(query_ctx); + return err; +}