From patchwork Sun Feb 9 10:17:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966716 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2089.outbound.protection.outlook.com [40.107.96.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0AD7154425; Sun, 9 Feb 2025 10:19:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096344; cv=fail; b=Iv3qt511XEq4GjDGM9oAIDWt/6TB86siKsDBugD3kaxl3hbbHhEG0CnfwtoVE41rkmzHCty0ZX+FYZ/CpZPOqmePqRT9AH5qpw015pbRZbfWrtfv8AHUblUlPjEAxvNxpQ6jNmaqtdVIL7FWPERw/JpmTyQtoXd1/+KWUHnrj8c= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096344; c=relaxed/simple; bh=AE9uREYWWt1E+iCZ61ePGJQWMwgvR5BslZJtc6CSxAc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=faUxtsveykHwfK2FcxcY/xAYWp/XEaxQfKEfmg/m+NB+FgRRMO/C/Z6eAUXKlULvfYQc3pMb2tkfWYpHBSghARQpnDm6WsTesFpVtAMxpNSMnXDtM2CY5IirOO5aQM/fJqR4vm1btjrYce2ivzG4z48qPFOSKe51vVUH36I6kao= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ZzUze2Gz; arc=fail smtp.client-ip=40.107.96.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ZzUze2Gz" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HoGFxJUuWTt5aHKL0ZkVBiQd0CkwGoxY+4etrViN0l4LvVhfXA5f960mBex1dnrDgQa4CkYsNM9X6Vrwf8xpoowXtuGGCs/GHUperm0BxYZmJWEvvwPNpyKYRfhas/z+nLapdr4y3gayEvH6jwK2rFDpwI4/j4D5az22z3AxvKl9mgsV5jwAyNbyzYyZsEZuLATWHx0UmJfz/29dn/xWYOtvBO8QsJhe2S+W7nn9OqPDBiaZ4ppPxOdXKEdN71xOKfGz6Vykvq+Box4yW+Ryw03+ac9Jnsqg9WYrZhAEANcV38OiLCpvpGfUMomDi5YKk3z3iR+H5ksD9HxABHQLOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fBjNnR1zr2j63On85I4sITHK6GqY+hSDh/q+nszAJOs=; b=Bh/yK9T8TF3IapMKexc4Xf4jKqy8uanQvVNZjgHBlTex70A1ydDEuiJDoB0W5VQlM7l2XtTxHKandstexzzlNwHftV1p43c1VBWloXbt09MU32yQeaYMdzjFxC+E5cI0+j/Uba0JLI8pd3JSYBn/d17X1W4LzYTsMTIrvVEBAfFiNowHPUxTTTbNp0Fr5ZARjoxnb700aDTvZhWKqciJEycO7pJ7ylofmD34ib+04dfKtPD+n5VbE06HoA//UoR6IDXAQ24dcse2SSHohP22db2mFWQM+zviAks7Y7/IR1nUWswtQgG1STc4GPYoV5jgsZwvKjUzeC4vaZxJn/hCWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fBjNnR1zr2j63On85I4sITHK6GqY+hSDh/q+nszAJOs=; b=ZzUze2GzSYko7BHKjxcWdveFI2CmfXQc6tKybXUuQFwFxn2diNvZx6/YQcCCQA5LSyy7f46gnuKEWYA+fLuriNo1to66COAk1EknUjG053tv/UTwozj5J7fInElwWS69fnywYyzAiljLkhXY04v2jr6PoxiXpC+x2eHXNwVYk2dP75QyNnvycVwHSNli24Bj3iupCBQ3txG5Tpj98iVZtFd6vhXrsqVldBSW6ZoUWFj/9d11xy09r4dbyWcqMd0n5tf19tUy7oQ3455yn0Xv8dPv7Yl4DeR0NhV4NJjYt3r/GFDNV+Q99Z9U4tj16UbnfcDyhe/5TDRm+3FTsSqUBw== Received: from MW4PR04CA0190.namprd04.prod.outlook.com (2603:10b6:303:86::15) by LV3PR12MB9187.namprd12.prod.outlook.com (2603:10b6:408:194::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.15; Sun, 9 Feb 2025 10:18:56 +0000 Received: from SJ1PEPF00002325.namprd03.prod.outlook.com (2603:10b6:303:86:cafe::ab) by MW4PR04CA0190.outlook.office365.com (2603:10b6:303:86::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:18:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002325.mail.protection.outlook.com (10.167.242.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:18:55 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:18:51 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:18:51 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:18:45 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Carolina Jubran , Cosmin Ratiu , Jiri Pirko Subject: [PATCH net-next 01/15] devlink: Extend devlink rate API with traffic classes bandwidth management Date: Sun, 9 Feb 2025 12:17:02 +0200 Message-ID: <20250209101716.112774-2-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002325:EE_|LV3PR12MB9187:EE_ X-MS-Office365-Filtering-Correlation-Id: 8e4c5d2a-d0da-4c14-596f-08dd48f32897 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: iiM/8Dh1Qwfai6iwtv53AQ3DxXB/uhlpnTRz08vApsX2BdxyhXMqYUdEJRS4rNyi0TYC0J9dsV6qEs7/wz9afrNLBUD2SxVmsL/HwaktIMvarvIG/nI7ezYSjftWiXJBKZCmL/jOpSWYcJjKy+yn3h29NxDf+EikK7t7M9gWyWaK7Ol1aJJEn0xljvRdekhAkUfNJ4VFbNly5e1CYUd168gNxaKrCstphVv4UE3C99h6EThvEM8VnGvCKr1ZTTT3Y2+xUEUOUD2Cg450cPOD3XlULolKP6gPNMhI9BvjHfrJcVc4D1XD9HZJK2vB4iEYjR2THdOD9+50D1LlAySm/XJpDXWY5My6t4VdQ8jx7PJXcS5Q9ksLM9kEeAIe5oAARhhayusf4j99aTQZYTSamvwBSokNI7er2a7PQS0BUd4YdH6f0IG757jiLom7XdFXGGDk4fFGq4oesLb7yabKKcKoMRRNiwS7ljDCiFThAdrqk/hYg1IfYEsHJpj2ZbqE4wKEZyohyvFbj1OXKzZVngW8nmY8w1iQN54Y5DVorJkZ7LL2rTD5EChG4vcxM6PoMTSuJA5RYQrbr9+IiZgYDfQ6WCwvSJciD7SyWl5mlbwDgZhslL0VGUjBoJYdC2VDtuaeAfEYZWLUrqpVFBmeJ86KwEXZizx6BHpchMjLRcBAfWfCQMl2ao6HVDf/WnYVe4qI/K7WvFZKviUEgUARjxGScYRZeWsMe7fKUbY8GOsRTLwrnDhHf9EHg2WBdF+R/e4bqhIv4iyu0HloHP20Iwhh1hD7ZZuDTtUzyw4EpRLgwlQtd0Ddit+F1ZzBbRko9nHJxU9z0BtW/EsH2I/F6i64ef0wwpWzhuc705Iy2gt0fnzTYcDQPIep8ulmIg3AnT3kN+mDUQzdYBYSC8UxoJrhyzrzfrnxlTQ0C29hUuDyCVwCc5Hw43LByMr5u9q4+58+AROIDw+/CJGXSbiJGD7IUKyNl1aE/K1JyQ4HA+CHrSQReZE5rfkfy0pDn7ATYcow/LlIykXIERVcV4k1d85PJLdWPDormzWbU6075VTE1/avTI71mKKuChGvhVsSOTnKGNIuS+Rmq+9v8kCL8cOHToOmZ1xGhlYZUCv6DXrLJm6rY2YRj0fYGw4DF2qbo4I0laRqMihHVlmv+mHUWrZs85kFdyOSped8MWzESSQ5SkuJPyqQAjrJUg9ELDgx95x9cFysmYHt4sT7JdyKpKm8lINzlRI90v4BNASPsXjew1xHB2Nwg3pBlf4uaTDTUzGi1pKkxfApkWlpt0br3O2vdVYZ8XdGRTq2icuKMuUXGPkshNxyoF+/Z30HxXnOmcgYGZfKJzxrvO4F4c1H/AezVNFAwf+8ttFMc1XVpr8YOuSOSH1Heo5zKP5GgriIPmnP0NRGRKW6fgAEai/BchvqF3Zz/cOFTKqOhxJLIBV9L+QDIbJRKFEja+MK1NIzWuVeB7vA2OHoc4YpUS7i9FwBgdUnIkOi8Gg7fI3VfME= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:18:55.1582 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8e4c5d2a-d0da-4c14-596f-08dd48f32897 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002325.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9187 From: Carolina Jubran Introduce support for specifying bandwidth proportions between traffic classes (TC) in the devlink-rate API. This new option allows users to allocate bandwidth across multiple traffic classes in a single command. This feature provides a more granular control over traffic management, especially for scenarios requiring Enhanced Transmission Selection. Users can now define a specific bandwidth share for each traffic class, such as allocating 20% for TC0 (TCP/UDP) and 80% for TC5 (RoCE). Example: DEV=pci/0000:08:00.0 $ devlink port function rate add $DEV/vfs_group tx_share 10Gbit \ tx_max 50Gbit tc-bw 0:20 1:0 2:0 3:0 4:0 5:80 6:0 7:0 $ devlink port function rate set $DEV/vfs_group \ tc-bw 0:20 1:0 2:0 3:0 4:0 5:20 6:60 7:0 Example usage with ynl: ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml \ --do rate-set --json '{ "bus-name": "pci", "dev-name": "0000:08:00.0", "port-index": 1, "rate-tc-bws": [ {"rate-tc-index": 0, "rate-tc-bw": 50}, {"rate-tc-index": 1, "rate-tc-bw": 50}, {"rate-tc-index": 2, "rate-tc-bw": 0}, {"rate-tc-index": 3, "rate-tc-bw": 0}, {"rate-tc-index": 4, "rate-tc-bw": 0}, {"rate-tc-index": 5, "rate-tc-bw": 0}, {"rate-tc-index": 6, "rate-tc-bw": 0}, {"rate-tc-index": 7, "rate-tc-bw": 0} ] }' ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml \ --do rate-get --json '{ "bus-name": "pci", "dev-name": "0000:08:00.0", "port-index": 1 }' output for rate-get: {'bus-name': 'pci', 'dev-name': '0000:08:00.0', 'port-index': 1, 'rate-tc-bws': [{'rate-tc-bw': 50, 'rate-tc-index': 0}, {'rate-tc-bw': 50, 'rate-tc-index': 1}, {'rate-tc-bw': 0, 'rate-tc-index': 2}, {'rate-tc-bw': 0, 'rate-tc-index': 3}, {'rate-tc-bw': 0, 'rate-tc-index': 4}, {'rate-tc-bw': 0, 'rate-tc-index': 5}, {'rate-tc-bw': 0, 'rate-tc-index': 6}, {'rate-tc-bw': 0, 'rate-tc-index': 7}], 'rate-tx-max': 0, 'rate-tx-priority': 0, 'rate-tx-share': 0, 'rate-tx-weight': 0, 'rate-type': 'leaf'} Signed-off-by: Carolina Jubran Reviewed-by: Cosmin Ratiu Reviewed-by: Jiri Pirko Signed-off-by: Tariq Toukan --- Documentation/netlink/specs/devlink.yaml | 36 ++++- .../networking/devlink/devlink-port.rst | 7 + include/net/devlink.h | 9 ++ include/uapi/linux/devlink.h | 4 + net/devlink/netlink_gen.c | 16 ++- net/devlink/netlink_gen.h | 2 + net/devlink/rate.c | 127 ++++++++++++++++++ 7 files changed, 196 insertions(+), 5 deletions(-) diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml index 09fbb4c03fc8..3a6dd62b6684 100644 --- a/Documentation/netlink/specs/devlink.yaml +++ b/Documentation/netlink/specs/devlink.yaml @@ -202,6 +202,11 @@ definitions: name: exception - name: control + - + name: devlink-rate-tc-index-max + header: net/devlink.h + type: const + value: 7 attribute-sets: - @@ -820,7 +825,26 @@ attribute-sets: - name: region-direct type: flag - + - + name: rate-tc-bws + type: nest + multi-attr: true + nested-attributes: dl-rate-tc-bws + - + name: rate-tc-index + type: u8 + checks: + min: 0 + max: devlink-rate-tc-index-max + - + name: rate-tc-bw + type: u32 + doc: | + Specifies the bandwidth allocation for the Traffic Class as a + percentage. + checks: + min: 0 + max: 100 - name: dl-dev-stats subset-of: devlink @@ -1225,6 +1249,14 @@ attribute-sets: - name: flash type: flag + - + name: dl-rate-tc-bws + subset-of: devlink + attributes: + - + name: rate-tc-index + - + name: rate-tc-bw operations: enum-model: directional @@ -2149,6 +2181,7 @@ operations: - rate-tx-priority - rate-tx-weight - rate-parent-node-name + - rate-tc-bws - name: rate-new @@ -2169,6 +2202,7 @@ operations: - rate-tx-priority - rate-tx-weight - rate-parent-node-name + - rate-tc-bws - name: rate-del diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst index 9d22d41a7cd1..bc3b41ac2d51 100644 --- a/Documentation/networking/devlink/devlink-port.rst +++ b/Documentation/networking/devlink/devlink-port.rst @@ -418,6 +418,13 @@ API allows to configure following rate object's parameters: to all node children limits. ``tx_max`` is an upper limit for children. ``tx_share`` is a total bandwidth distributed among children. +``tc_bw`` + Allow users to set the bandwidth allocation per traffic class on rate + objects. This enables fine-grained QoS configurations by assigning specific + bandwidth percentages to different traffic classes. When applied to a + non-leaf node, tc_bw determines how bandwidth is shared among its child + elements. + ``tx_priority`` and ``tx_weight`` can be used simultaneously. In that case nodes with the same priority form a WFQ subgroup in the sibling group and arbitration among them is based on assigned weights. diff --git a/include/net/devlink.h b/include/net/devlink.h index b8783126c1ed..1b7fa11b5841 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -20,6 +20,7 @@ #include #include #include +#include struct devlink; struct devlink_linecard; @@ -99,6 +100,8 @@ struct devlink_port_attrs { }; }; +#define DEVLINK_RATE_TC_INDEX_MAX (IEEE_8021QAZ_MAX_TCS - 1) + struct devlink_rate { struct list_head list; enum devlink_rate_type type; @@ -118,6 +121,8 @@ struct devlink_rate { u32 tx_priority; u32 tx_weight; + + u32 tc_bw[IEEE_8021QAZ_MAX_TCS]; }; struct devlink_port { @@ -1482,6 +1487,8 @@ struct devlink_ops { u32 tx_priority, struct netlink_ext_ack *extack); int (*rate_leaf_tx_weight_set)(struct devlink_rate *devlink_rate, void *priv, u32 tx_weight, struct netlink_ext_ack *extack); + int (*rate_leaf_tc_bw_set)(struct devlink_rate *devlink_rate, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack); int (*rate_node_tx_share_set)(struct devlink_rate *devlink_rate, void *priv, u64 tx_share, struct netlink_ext_ack *extack); int (*rate_node_tx_max_set)(struct devlink_rate *devlink_rate, void *priv, @@ -1490,6 +1497,8 @@ struct devlink_ops { u32 tx_priority, struct netlink_ext_ack *extack); int (*rate_node_tx_weight_set)(struct devlink_rate *devlink_rate, void *priv, u32 tx_weight, struct netlink_ext_ack *extack); + int (*rate_node_tc_bw_set)(struct devlink_rate *devlink_rate, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack); int (*rate_node_new)(struct devlink_rate *rate_node, void **priv, struct netlink_ext_ack *extack); int (*rate_node_del)(struct devlink_rate *rate_node, void *priv, diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 9401aa343673..b3b538c67c34 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -614,6 +614,10 @@ enum devlink_attr { DEVLINK_ATTR_REGION_DIRECT, /* flag */ + DEVLINK_ATTR_RATE_TC_BWS, /* nested */ + DEVLINK_ATTR_RATE_TC_INDEX, /* u8 */ + DEVLINK_ATTR_RATE_TC_BW, /* u32 */ + /* Add new attributes above here, update the spec in * Documentation/netlink/specs/devlink.yaml and re-generate * net/devlink/netlink_gen.c. diff --git a/net/devlink/netlink_gen.c b/net/devlink/netlink_gen.c index f9786d51f68f..186f31522af0 100644 --- a/net/devlink/netlink_gen.c +++ b/net/devlink/netlink_gen.c @@ -9,6 +9,7 @@ #include "netlink_gen.h" #include +#include /* Common nested types */ const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS + 1] = { @@ -18,6 +19,11 @@ const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_ [DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(15), }; +const struct nla_policy devlink_dl_rate_tc_bws_nl_policy[DEVLINK_ATTR_RATE_TC_BW + 1] = { + [DEVLINK_ATTR_RATE_TC_INDEX] = NLA_POLICY_RANGE(NLA_U8, 0, DEVLINK_RATE_TC_INDEX_MAX), + [DEVLINK_ATTR_RATE_TC_BW] = NLA_POLICY_RANGE(NLA_U32, 0, 100), +}; + const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH + 1] = { [DEVLINK_ATTR_SELFTEST_ID_FLASH] = { .type = NLA_FLAG, }, }; @@ -496,7 +502,7 @@ static const struct nla_policy devlink_rate_get_dump_nl_policy[DEVLINK_ATTR_DEV_ }; /* DEVLINK_CMD_RATE_SET - do */ -static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT + 1] = { +static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TC_BWS + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING, }, @@ -505,10 +511,11 @@ static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_W [DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, + [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), }; /* DEVLINK_CMD_RATE_NEW - do */ -static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT + 1] = { +static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TC_BWS + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING, }, @@ -517,6 +524,7 @@ static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_W [DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, }, [DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, }, + [DEVLINK_ATTR_RATE_TC_BWS] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bws_nl_policy), }; /* DEVLINK_CMD_RATE_DEL - do */ @@ -1164,7 +1172,7 @@ const struct genl_split_ops devlink_nl_ops[74] = { .doit = devlink_nl_rate_set_doit, .post_doit = devlink_nl_post_doit, .policy = devlink_rate_set_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT, + .maxattr = DEVLINK_ATTR_RATE_TC_BWS, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, { @@ -1174,7 +1182,7 @@ const struct genl_split_ops devlink_nl_ops[74] = { .doit = devlink_nl_rate_new_doit, .post_doit = devlink_nl_post_doit, .policy = devlink_rate_new_nl_policy, - .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT, + .maxattr = DEVLINK_ATTR_RATE_TC_BWS, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, { diff --git a/net/devlink/netlink_gen.h b/net/devlink/netlink_gen.h index 8f2bd50ddf5e..e3558cf89be4 100644 --- a/net/devlink/netlink_gen.h +++ b/net/devlink/netlink_gen.h @@ -10,9 +10,11 @@ #include #include +#include /* Common nested types */ extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS + 1]; +extern const struct nla_policy devlink_dl_rate_tc_bws_nl_policy[DEVLINK_ATTR_RATE_TC_BW + 1]; extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH + 1]; /* Ops table for devlink */ diff --git a/net/devlink/rate.c b/net/devlink/rate.c index 8828ffaf6cbc..3a046e929594 100644 --- a/net/devlink/rate.c +++ b/net/devlink/rate.c @@ -80,6 +80,29 @@ devlink_rate_get_from_info(struct devlink *devlink, struct genl_info *info) return ERR_PTR(-EINVAL); } +static int devlink_rate_put_tc_bws(struct sk_buff *msg, u32 *tc_bw) +{ + struct nlattr *nla_tc_bw; + int i; + + for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) { + nla_tc_bw = nla_nest_start(msg, DEVLINK_ATTR_RATE_TC_BWS); + if (!nla_tc_bw) + return -EMSGSIZE; + + if (nla_put_u8(msg, DEVLINK_ATTR_RATE_TC_INDEX, i) || + nla_put_u32(msg, DEVLINK_ATTR_RATE_TC_BW, tc_bw[i])) + goto nla_put_failure; + + nla_nest_end(msg, nla_tc_bw); + } + return 0; + +nla_put_failure: + nla_nest_cancel(msg, nla_tc_bw); + return -EMSGSIZE; +} + static int devlink_nl_rate_fill(struct sk_buff *msg, struct devlink_rate *devlink_rate, enum devlink_command cmd, u32 portid, u32 seq, @@ -129,6 +152,9 @@ static int devlink_nl_rate_fill(struct sk_buff *msg, devlink_rate->parent->name)) goto nla_put_failure; + if (devlink_rate_put_tc_bws(msg, devlink_rate->tc_bw)) + goto nla_put_failure; + genlmsg_end(msg, hdr); return 0; @@ -316,6 +342,89 @@ devlink_nl_rate_parent_node_set(struct devlink_rate *devlink_rate, return 0; } +static int devlink_nl_rate_tc_bw_parse(struct nlattr *parent_nest, u32 *tc_bw, + unsigned long *bitmap, struct netlink_ext_ack *extack) +{ + struct nlattr *tb[DEVLINK_ATTR_MAX + 1]; + u8 tc_index; + + nla_parse_nested(tb, DEVLINK_ATTR_MAX, parent_nest, devlink_dl_rate_tc_bws_nl_policy, + extack); + if (!tb[DEVLINK_ATTR_RATE_TC_INDEX]) { + NL_SET_ERR_ATTR_MISS(extack, parent_nest, DEVLINK_ATTR_RATE_TC_INDEX); + return -EINVAL; + } + + tc_index = nla_get_u8(tb[DEVLINK_ATTR_RATE_TC_INDEX]); + + if (!tb[DEVLINK_ATTR_RATE_TC_BW]) { + NL_SET_ERR_ATTR_MISS(extack, parent_nest, DEVLINK_ATTR_RATE_TC_BW); + return -EINVAL; + } + + if (test_and_set_bit(tc_index, bitmap)) { + NL_SET_ERR_MSG_FMT(extack, "Duplicate traffic class index specified (%u)", + tc_index); + return -EINVAL; + } + + tc_bw[tc_index] = nla_get_u32(tb[DEVLINK_ATTR_RATE_TC_BW]); + + return 0; +} + +static int devlink_nl_rate_tc_bw_set(struct devlink_rate *devlink_rate, + struct genl_info *info) +{ + DECLARE_BITMAP(bitmap, IEEE_8021QAZ_MAX_TCS) = {}; + struct devlink *devlink = devlink_rate->devlink; + const struct devlink_ops *ops = devlink->ops; + int rem, err = -EOPNOTSUPP, i, total = 0; + u32 tc_bw[IEEE_8021QAZ_MAX_TCS] = {}; + struct nlattr *attr; + + nla_for_each_attr(attr, genlmsg_data(info->genlhdr), + genlmsg_len(info->genlhdr), rem) { + if (nla_type(attr) == DEVLINK_ATTR_RATE_TC_BWS) { + err = devlink_nl_rate_tc_bw_parse(attr, tc_bw, bitmap, info->extack); + if (err) + return err; + } + } + + for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) { + if (!test_bit(i, bitmap)) { + NL_SET_ERR_MSG_FMT(info->extack, + "Bandwidth values must be specified for all %u traffic classes", + IEEE_8021QAZ_MAX_TCS); + return -EINVAL; + } + + total += tc_bw[i]; + } + + if (total && total != 100) { + NL_SET_ERR_MSG_FMT(info->extack, + "Sum of all traffic class bandwidth share values must be 100, got %u", + total); + return -EINVAL; + } + + if (devlink_rate_is_leaf(devlink_rate)) + err = ops->rate_leaf_tc_bw_set(devlink_rate, devlink_rate->priv, tc_bw, + info->extack); + else if (devlink_rate_is_node(devlink_rate)) + err = ops->rate_node_tc_bw_set(devlink_rate, devlink_rate->priv, tc_bw, + info->extack); + + if (err) + return err; + + memcpy(devlink_rate->tc_bw, tc_bw, sizeof(tc_bw)); + + return 0; +} + static int devlink_nl_rate_set(struct devlink_rate *devlink_rate, const struct devlink_ops *ops, struct genl_info *info) @@ -388,6 +497,12 @@ static int devlink_nl_rate_set(struct devlink_rate *devlink_rate, return err; } + if (attrs[DEVLINK_ATTR_RATE_TC_BWS]) { + err = devlink_nl_rate_tc_bw_set(devlink_rate, info); + if (err) + return err; + } + return 0; } @@ -423,6 +538,12 @@ static bool devlink_rate_set_ops_supported(const struct devlink_ops *ops, "TX weight set isn't supported for the leafs"); return false; } + if (attrs[DEVLINK_ATTR_RATE_TC_BWS] && !ops->rate_leaf_tc_bw_set) { + NL_SET_ERR_MSG_ATTR(info->extack, + attrs[DEVLINK_ATTR_RATE_TC_BWS], + "TC bandwidth set isn't supported for the leafs"); + return false; + } } else if (type == DEVLINK_RATE_TYPE_NODE) { if (attrs[DEVLINK_ATTR_RATE_TX_SHARE] && !ops->rate_node_tx_share_set) { NL_SET_ERR_MSG(info->extack, "TX share set isn't supported for the nodes"); @@ -449,6 +570,12 @@ static bool devlink_rate_set_ops_supported(const struct devlink_ops *ops, "TX weight set isn't supported for the nodes"); return false; } + if (attrs[DEVLINK_ATTR_RATE_TC_BWS] && !ops->rate_node_tc_bw_set) { + NL_SET_ERR_MSG_ATTR(info->extack, + attrs[DEVLINK_ATTR_RATE_TC_BWS], + "TC bandwidth set isn't supported for the nodes"); + return false; + } } else { WARN(1, "Unknown type of rate object"); return false; From patchwork Sun Feb 9 10:17:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966717 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2074.outbound.protection.outlook.com [40.107.96.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADBF1154BE0; Sun, 9 Feb 2025 10:19:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096344; cv=fail; b=innhFAdTCM1qsYn+Lz/oBIwZCZ0fEUS1Oq+AVnvsIn3lEtaBd1FH40R0HlyMkTAr8RJiSTKntj4E0MjMPTPOiUokU7AvQfiT7GdZ97rQc7u6BwAFfEYUJi7Pn/9p7zX0b9G3xXcp98KTE+sXTw6QbOY1+DscLS9TtXRBxrPXdts= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096344; c=relaxed/simple; bh=PZhKfW9LvIzlsLnOmrB4W+C28i6LzyXbnfKYo0jYdOs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=cClUBpU6TJeahu1brYyHnW0d4tn53ya10aw9BLTsEubBYT1HbCuZ9afizIn7EV7sGSvj2/PJ2fBclRdYut+Xd0lyCyGNmcaZk2Erl0GOQzfizmm98c3KAwzFasp1jlHkRzmpu0MJbfkuJk1pGFJp9NGaxUoqzBvxSfmwSzelymo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JZvkWRQ2; arc=fail smtp.client-ip=40.107.96.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JZvkWRQ2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gg5Rl/Kr4cRzLOUHAs2DUcjgxaIyf1OzhhnrS3SXjZy2kHqQAOb1AKPsn9uxpLUKm38Whp4nzLM3TrFyA3m+1mWt0JzGq9gGy+M+N0vQjmrJUYp7D9OQsdIv+tvlWU/a5lGilJtwZKp7m22jf+BbOTsGKdd6K0Zx/0u21Mh/8NsqjAu8hCS/Hi3zMF+b08HH612SWVcj+DS6qi5K9uR88+5Td9Gll/hXsqeVwtGb/yAJHCEZiAVrnli8nzT8JkvQjuNOOopSwf6ktX/dWlf8MAbDOngJlFif3ZpCUKlJLtGQofdpArWM2NCcRNpavyTR1KZd5BTcJzpiQSzCVq6Aiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Cw3laofYjxED2GxVbHJFY7fPZTjPN8ctuWyOhHuF1z8=; b=Te0n+eLITO3ujHlRFYunFzuSOt8WOjt57maK70NXdxqxBee1hsgE8mOwpaAx72276gOHkiCZQkolAOuhOWJlOGaUjLy8j8lxx0j2glhDqSQ8MVYtBU3pJ2k0KD0tGVqNODWX+rCRoDvVl3Shzrcp5rys5OynbzY/CpKaveYaU5kcGDwRZP5PHNv2+uVyQZVuwzdF3MdZ8SggFqU8OQb25gCTYg+W5BoA/brXnt+5Y0zc7yPsMc5WGxI8j9xRSkeFpLJDXdVxtFRZHCHIlmaVjbkpUbsqScoTqlcWfHnJBym4NIhPBLpDm09bwv8uJQAZhm3u+3M1YINboTijTbPKIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Cw3laofYjxED2GxVbHJFY7fPZTjPN8ctuWyOhHuF1z8=; b=JZvkWRQ2eyMwmgobh/RzNxGNMGgLlpQ3HfDFLYL+PziBhIA0n84gIs5UWVaulu8YKEVhM4H5ALfqq3hqnSUR04QYwXPniNFr8+kC2Rfe5Fp7eLEtaOTApvP4AcX/lLw57xAO7OStW4cNmmUwpiuM1PV8ZHipNPhiI45CkDvLmQ6lktl3OjuoqKZpNcuaIzAWHx25LHgdaqyBCZ4tZeQXqepTGxY8f0yOb4QhRFpItRwJ47h8D4ASsKkreJR5/OKloetfhebeX0Ite4XJWr38cUqvGTpUkw6SaM4U4IuzBPxREBklim5Ug/+8eX01PofovVNLds/y3UDTiI1In5fWMA== Received: from PH7P222CA0023.NAMP222.PROD.OUTLOOK.COM (2603:10b6:510:33a::30) by CY8PR12MB8313.namprd12.prod.outlook.com (2603:10b6:930:7d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.16; Sun, 9 Feb 2025 10:18:59 +0000 Received: from SN1PEPF000252A4.namprd05.prod.outlook.com (2603:10b6:510:33a:cafe::c4) by PH7P222CA0023.outlook.office365.com (2603:10b6:510:33a::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:18:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A4.mail.protection.outlook.com (10.167.242.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:18:58 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:18:57 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:18:57 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:18:52 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Carolina Jubran , Cosmin Ratiu Subject: [PATCH net-next 02/15] net/mlx5: Add no-op implementation for setting tc-bw on rate objects Date: Sun, 9 Feb 2025 12:17:03 +0200 Message-ID: <20250209101716.112774-3-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A4:EE_|CY8PR12MB8313:EE_ X-MS-Office365-Filtering-Correlation-Id: 63a8800c-4e31-4667-900e-08dd48f32a7b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: /eANClT/AwRh9hkcAo8krgLn7fggb7HSreRbbaoBZ40SV7zXZGy28AsRHaSp4c3ZbqVgf9iYnTEuRVwLiGsEPg6Z1As0FkCwpbHjlNXf8dQHH9ZbjDHsvzpusRMyEawCMQgBxcJUZ33L0cgEzx2wyJZzdmeqnQTRVnr584FbFWcdCZDcnnrIG8AJmY8obnq1m0klmiDRIJoR/l7E9hGLglWRSrrqw8ltpR/RpQAdH/KpyA8wgR6hV/HpXq6hEzTvc7UhGULYcA9lWOhF73QfaUeBw8lr6ubdFXya73YreBQKzLgB4QTvNssI8//RtA66/7Qg7oDmeafZYbHvuPIUFXc4Bnc4bLUtQsMMfy5fwFSytPfkBtAPWq42gm/XPpjgsJuGOw51Kp+u9zIt73GKUb3RV486UDdee8+PWtG24QHZgK+tTb9f0L1TWVOMpSSWVl9Kh+S+ozn2OUQsJGO07+sfgJHGRbCbpRdWAUvR1nSAvKSswlC3EEDm9tt8vKJX8qH5Jt3qWu82YRInuhyS9upA0xULBL/+1fmXyRmwyY5AdADIWtjBUEXZFzCBl0kHbgx5Lodz9buJ+P7bCRFeqW4hOCQ17dexABrsS/avSDsXlFNSBsYJKv+Er2g1DabyGB/q+EMTR4OfKbByyJ4V8L0kv9lpaPLx+lwRikaY7LFqR88t2iKJ5rKNEtJpKzMrvWL4um+Rm4oHYt9xqFaMNeZHUN6VgyJ09lE+bzeeM0GwlxpaTaIzPwUGT/VD4xfTkcliQxKr3uuAgmW9S3tIeEMcFItW3uVZEE0fR5eI5Xtv3hHofh8d6GQkBjNHQdeiFgS2qlhKXJQ33TC0QFKeJ+qp4vrmyoS2A/t8ltaUYlCWsuDRetRK6RzMzBWLcH0NUU1QsOm0pWJGT7a1WWIPc9XZiHxGpBgW7R9+CFWuY6O3e6WY98N5i+OHp9n1ckRLcOvvAFPBkkJ1qeTyeKUiuY5ZArK6RP20DGiP7bOh4rxWwIs548DbEAiZlSBb3OXhGUbapjsdKgYv99hU1hhxkq80oBG7l8unZdXNBst+YEIZr/elJg7bhG/J9rS8YRYNCOQ1EJr4f7dzOkrZLLS01z07A8k76LcwuiBKBqLo4fPRWAeV7W2aL/mVtxyB9N24V+L17ni7QfSSk5QTBNmuUYcSOg0uLL+Ua08ep9vdO+KVW6vhn7TDwn7Xb9Jw1cCt8qCrEYGt5AGfWM94+klBC3FU0A/bY2swSlZD140fNIeGtLe6x6NVloEnCPs3q6pECAIFXhA/x9mqxkUkl+PiPSKEIML8wl7vgBnyqJCcQcC7w7WCbfhRr/g6ZBiD/xuybmP2KIFHJh9U1t18oEftGpUBzc0+pTmj/umzH3Xz4+f31xO86lvkAY8PKewDQfAfJnPVBRTHPmDak/YHJ8b7g054Bct8e2b37qdWI3ubizh6X9cq5CeFNtSbHuvsA4RPO78O7/3hYA4T8k24OMYz/WABRklnTVbnm9yPcq5Ello= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:18:58.2222 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 63a8800c-4e31-4667-900e-08dd48f32a7b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A4.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8313 From: Carolina Jubran Introduce `mlx5_esw_devlink_rate_node_tc_bw_set()` and `mlx5_esw_devlink_rate_leaf_tc_bw_set()` with no-op logic. Future patches will add support for setting traffic class bandwidth on rate objects. Signed-off-by: Carolina Jubran Reviewed-by: Cosmin Ratiu Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 2 ++ drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c | 14 ++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h | 4 ++++ 3 files changed, 20 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 98d4306929f3..728d5c06d612 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -320,6 +320,8 @@ static const struct devlink_ops mlx5_devlink_ops = { .eswitch_encap_mode_get = mlx5_devlink_eswitch_encap_mode_get, .rate_leaf_tx_share_set = mlx5_esw_devlink_rate_leaf_tx_share_set, .rate_leaf_tx_max_set = mlx5_esw_devlink_rate_leaf_tx_max_set, + .rate_leaf_tc_bw_set = mlx5_esw_devlink_rate_leaf_tc_bw_set, + .rate_node_tc_bw_set = mlx5_esw_devlink_rate_node_tc_bw_set, .rate_node_tx_share_set = mlx5_esw_devlink_rate_node_tx_share_set, .rate_node_tx_max_set = mlx5_esw_devlink_rate_node_tx_max_set, .rate_node_new = mlx5_esw_devlink_rate_node_new, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c index 8b7c843446e1..db112a87b7ee 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c @@ -882,6 +882,20 @@ int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void * return err; } +int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack) +{ + NL_SET_ERR_MSG_MOD(extack, "TC bandwidth shares are not supported on leafs"); + return -EOPNOTSUPP; +} + +int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack) +{ + NL_SET_ERR_MSG_MOD(extack, "TC bandwidth shares are not supported on nodes"); + return -EOPNOTSUPP; +} + int mlx5_esw_devlink_rate_node_tx_share_set(struct devlink_rate *rate_node, void *priv, u64 tx_share, struct netlink_ext_ack *extack) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h index 6eb8f6a648c8..0239f10f95e7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.h @@ -21,6 +21,10 @@ int mlx5_esw_devlink_rate_leaf_tx_share_set(struct devlink_rate *rate_leaf, void u64 tx_share, struct netlink_ext_ack *extack); int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void *priv, u64 tx_max, struct netlink_ext_ack *extack); +int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_node, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack); +int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *priv, + u32 *tc_bw, struct netlink_ext_ack *extack); int mlx5_esw_devlink_rate_node_tx_share_set(struct devlink_rate *rate_node, void *priv, u64 tx_share, struct netlink_ext_ack *extack); int mlx5_esw_devlink_rate_node_tx_max_set(struct devlink_rate *rate_node, void *priv, From patchwork Sun Feb 9 10:17:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966718 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2048.outbound.protection.outlook.com [40.107.102.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A6E71598F4; Sun, 9 Feb 2025 10:19:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096364; cv=fail; b=dmYj5rM5A+1H1Y2Xql1itWrq8CK858qqGXUJqH2Uq0Tq9iGLk+iRYGkkiaTzebixm3elkI3hpvgUSo1/LsgnyjH+2Kn9e6q1I1qFe/PrG1i/vKqwHgR1VxdaMAV0x+qH/wMMvipOyUBsykZT9jq7Fq6YbOJrGrKk1xUiYy0Sjws= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096364; c=relaxed/simple; bh=O9UUxjEdfh/ZujFYDncbW9+opnXzJiOQ12mLCZsxejk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Qo/A63tDlAgB/X7MTh9BZzRHVcuABO4eUa58CtmiX5NxqH4VLpjBBwdXaMia65MHMdl2gKSf7GfwXQNTSMA5e5sieRH1rA0RbNnBp4IopapNV1FDB2Ksovs+5uwpulJ+S2CMI5n+zkryv9sq/wtNuY2NRS+a6rNZOpA01bNdbTY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=dLj4bdKH; arc=fail smtp.client-ip=40.107.102.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="dLj4bdKH" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qSRdnd9roK3+YqCn39HfQ9834WgxXm21ss5bckjfhiviyG/bgn4KxN/btWOcxEPOD/wA8vAapXKgLO+snyASNkFGkNoDo89hflOInBPEj6fpQT6EhNc92RNCdQJ2+gTFJpXlmqiAZoha+a06NjlLaVkjAhKDrv/9LyFOP+omqRNlo8U1rvEx4iYy+74WGQhurvpcB1m4Q/aU9M3PaDU3thNw0hS96kueUpe5qF6qLvlcM+jSxOiKrCVj7YTO+Qth4pZPX5AnuO56yGu74LLE4O3OhuKtJWiZAOOUaZl52QLuTfKHdMDIrPSGXE7FcNgAY9CqL2dok88VA7vW1U68gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CfMVPFWorxjt+1cVGBkPA7uYKrHKsbHNopjQxabBpLc=; b=xNYYU9/0/XuEPfjvinGzmh71AweBF2OFfRyvxxgvchPuFJWocs/8y2HYwT+SqD52QDru/VtmsE4WTHDHakaKwv3nTdDl+u2f+zBMekFaowLkCBfHlfAukjG0psNN5LBZa18ndDsLjCcZfMqn7q0STBf+cJstBE342e0yrSu6UeRJT6TYTRg+K7wml8L5jT2rICeMjGhuSa8GxLj8ej6SYs888MGj6ICuioSmwLbqQYlyCPc+sfZwiAQpdNXNbJFhEfgPshnaztclvZNgJHsajUgD67df8/mFQb8QBAdDB2WbLIZufbpYxuwb5qe4ClwaAkXQYdnA2ux2QtrGm8DnDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CfMVPFWorxjt+1cVGBkPA7uYKrHKsbHNopjQxabBpLc=; b=dLj4bdKHQE+p/tNyqjrEc8ICQrbbW877FBaDHqAY7JZBKQI9Wfm/HbQsTu/WS9J93nQKtvKa0aBM7C7NM6PZD1I41CzfnSxI6uKf84hnwOJPZxg0j3+i5LvjWBzbySpP0V/urq76XISW6la+dssjFtBCV5igPIrj0bn6DzSZYfjFz1kbT563BBmpdLkkwsVnFtpsjnEgaDvy2tQMjHlnbdECd4ct6cqGknYd11z2s0DLJKUvZyhoV08EJgM9I+FZkcqN+na9Zd7K/GSJU29vHNzyXAzHNXom9z4aUE+fxOdVBZZAl3gNEgz4L4JpbBpnLZ8fRqBpVetoMUWGds9B0A== Received: from PH7P220CA0166.NAMP220.PROD.OUTLOOK.COM (2603:10b6:510:33b::13) by MN2PR12MB4422.namprd12.prod.outlook.com (2603:10b6:208:265::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.16; Sun, 9 Feb 2025 10:19:16 +0000 Received: from SN1PEPF000252A3.namprd05.prod.outlook.com (2603:10b6:510:33b:cafe::ba) by PH7P220CA0166.outlook.office365.com (2603:10b6:510:33b::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A3.mail.protection.outlook.com (10.167.242.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:15 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:03 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:03 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:18:57 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Carolina Jubran , Cosmin Ratiu Subject: [PATCH net-next 03/15] net/mlx5: Add support for setting tc-bw on nodes Date: Sun, 9 Feb 2025 12:17:04 +0200 Message-ID: <20250209101716.112774-4-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A3:EE_|MN2PR12MB4422:EE_ X-MS-Office365-Filtering-Correlation-Id: 4617167a-6a22-4633-65bd-08dd48f3347e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?q?REHhC3M/6QZHTWAqa5+NzFoEof5tEHN?= =?utf-8?q?cpZKblh1D+rIL4cQvEB7R5PDzyR+VbkbneUWtkTsG5GLMC83pK3SncmcjCzyILHVG?= =?utf-8?q?Pyt6o0xr3Lox59hDhFUP0RvaoKFUKnLQGjYvPfDRjKHdohp7S950/jWx0/qZspOKT?= =?utf-8?q?SMCv2YyiDw+vb99cVzTjCClkqjhUlmX+lBwEPtSZxV9BvWVrjJYLMXUkU50Cdx2+2?= =?utf-8?q?LQwJdXukIkEeXyyQbBs9LeVi10+W2kssKZDxZKBDAD5+hYu9jpqEUGRsZPPPZhHgJ?= =?utf-8?q?kW4IT0l0kBwUuejn4i+Z6SOtw91EEXlkQ22c0fLiBF8dR0/pKxYzXOBUZaPJCZOni?= =?utf-8?q?KxQkOdmdtbtCbDj+V5jQ0AVKS9dxCybkZt8GnZ74BWs6NDkY5qe5mKfiMyT5HPKEr?= =?utf-8?q?Z63VmKAj8do7PFZ3rEtC7eMJitSusD7NrXiILs1OAn5hxhqdqaQYudhMTmCEzd8hR?= =?utf-8?q?bzaEzllefoMcjs7T7AOu1EbYESgkBrhRy/JHHtUcy1eE475EImtF5yLvuJ0jZ45mV?= =?utf-8?q?GX5K7Nt7wwmZrvN/r6SzNuxHeGTJr8l9NKBr/jXspmjbXCL1rz5632pOzhmScvh9y?= =?utf-8?q?h3l74kE/1DR0oGEYyH/3jTx/WFWUpQx5ehlEj9RMtQwwQprKJoD6YUKzU3d+pIaPG?= =?utf-8?q?Bs90Bvm3P27U3dsIXpvIgv1Kze0/pdouIM4R0vfILLTmIsrJlrGUgf3rvywENrSri?= =?utf-8?q?NTP7RVt/ZnjaqBqfBHOA22B8B7bQdNWOR97yvHmZOScsjdLovVO482UbxJKcA5etK?= =?utf-8?q?DKNAMYPzqNjG3iGHRMqlMbHzbAU+MJ2AgEke4LZkWh2kj/k0ibXtvf7MrPhuB4VzW?= =?utf-8?q?2jYudyWYSCO38UTGR3/Rg5awSagZtDcxcwJPKbONTio1Bj/SgufkdOK5ojoP23F+x?= =?utf-8?q?nElVjG7dyPfQYZNwzNj9EwEHkvUdog1/6sO4wxQJZVW6r21dS8ErzdciTYnrPkh/a?= =?utf-8?q?Z/x3acyQv5Ta/leQSgw+g1cnugoN9m/WgH5SWBg20v/sqnTifRCt/5/uzjm1X0GQ6?= =?utf-8?q?vrNZaCQfERFMWncXSutHIQs+V7tT6o+BW3WzRtCgfaGm3CPg7uVbx7qmI4G4c0J5s?= =?utf-8?q?QwLFplYysiMkRVN8gkd8MWIrFAdEQpFM2SRWUqWgbBMpgqfxV1RLOAFYFELxYGFF/?= =?utf-8?q?vF1kcfS1RbRBwwLfmyIftFty/Alh6S4OIh32bfIJjKL0p5Xzr4Yn81QMk4+fmgMEp?= =?utf-8?q?L/0vJvaeAHXNSpVst7t0Sb+EscyLRs9AptVPxBlE38JvfxXipxCXSEnHcyu8423YB?= =?utf-8?q?LMng7llPFuCM3JmQGA/EwLAndS3m5PTaT/gEaD7l0QkVrVg67GhuNKd5IbK6EamZp?= =?utf-8?q?VYDahguj9rvzhKHfgkFh1cH6un0ojxVFaWs8z1qiIHNEUYLYpg+VJX3TSClxt4xHm?= =?utf-8?q?R1V/PrsiZCFoYooyuOEYbqcOHAoVOdd+GhRAEkrh2QBfILtN/a0/14=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:15.0492 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4617167a-6a22-4633-65bd-08dd48f3347e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A3.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4422 From: Carolina Jubran Introduce support for enabling and disabling Traffic Class (TC) arbitration for existing devlink rate nodes. This patch adds support for a new scheduling node type, `SCHED_NODE_TYPE_TC_ARBITER_TSAR`. Key changes include: - New helper functions for transitioning existing rate nodes to TC arbiter nodes and vice versa. These functions handle the allocation of TC arbiter nodes, copying of child nodes, and restoring vport QoS settings when TC arbitration is disabled. - Implementation of `mlx5_esw_devlink_rate_node_tc_bw_set()` to manage tc-bw configuration on nodes. - Introduced stubs for `esw_qos_tc_arbiter_scheduling_setup()` and `esw_qos_tc_arbiter_scheduling_teardown()`, which will be extended in future patches to provide full support for tc-bw on devlink rate objects. - Validation functions for tc-bw settings, allowing graceful handling of unsupported traffic class bandwidth configurations. - Updated `__esw_qos_alloc_node()` to insert the new node into the parent’s children list only if the parent is not NULL. For the root TSAR, the new node is inserted directly after the allocation call. - Updated esw_qos_create_node_sched_elem to receive max_rate and bw_share values to save the old configuration when changing the group type. This patch lays the groundwork for future support for configuring tc-bw on devlink rate nodes. Although the infrastructure is in place, full support for tc-bw is not yet implemented; attempts to set tc-bw on nodes will return `-EOPNOTSUPP`. No functional changes are introduced at this stage. Signed-off-by: Carolina Jubran Reviewed-by: Cosmin Ratiu Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 279 ++++++++++++++++-- 1 file changed, 262 insertions(+), 17 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c index db112a87b7ee..efcbd3180317 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c @@ -64,11 +64,13 @@ static void esw_qos_domain_release(struct mlx5_eswitch *esw) enum sched_node_type { SCHED_NODE_TYPE_VPORTS_TSAR, SCHED_NODE_TYPE_VPORT, + SCHED_NODE_TYPE_TC_ARBITER_TSAR, }; static const char * const sched_node_type_str[] = { [SCHED_NODE_TYPE_VPORTS_TSAR] = "vports TSAR", [SCHED_NODE_TYPE_VPORT] = "vport", + [SCHED_NODE_TYPE_TC_ARBITER_TSAR] = "TC Arbiter TSAR", }; struct mlx5_esw_sched_node { @@ -92,6 +94,13 @@ struct mlx5_esw_sched_node { struct mlx5_vport *vport; }; +static int esw_qos_num_tcs(struct mlx5_core_dev *dev) +{ + int num_tcs = mlx5_max_tc(dev) + 1; + + return num_tcs < IEEE_8021QAZ_MAX_TCS ? num_tcs : IEEE_8021QAZ_MAX_TCS; +} + static void esw_qos_node_set_parent(struct mlx5_esw_sched_node *node, struct mlx5_esw_sched_node *parent) { @@ -101,6 +110,25 @@ esw_qos_node_set_parent(struct mlx5_esw_sched_node *node, struct mlx5_esw_sched_ node->esw = parent->esw; } +static void +esw_qos_nodes_set_parent(struct list_head *nodes, struct mlx5_esw_sched_node *parent) +{ + struct mlx5_esw_sched_node *node, *tmp; + + list_for_each_entry_safe(node, tmp, nodes, entry) { + esw_qos_node_set_parent(node, parent); + if (!list_empty(&node->children) && + parent->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) { + struct mlx5_esw_sched_node *child; + + list_for_each_entry(child, &node->children, entry) { + if (child->vport) + child->vport->qos.sched_node->parent = parent; + } + } + } +} + void mlx5_esw_qos_vport_qos_free(struct mlx5_vport *vport) { kfree(vport->qos.sched_node); @@ -126,16 +154,23 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport) static void esw_qos_sched_elem_warn(struct mlx5_esw_sched_node *node, int err, const char *op) { - if (node->vport) { + switch (node->type) { + case SCHED_NODE_TYPE_VPORT: esw_warn(node->esw->dev, "E-Switch %s %s scheduling element failed (vport=%d,err=%d)\n", op, sched_node_type_str[node->type], node->vport->vport, err); - return; + break; + case SCHED_NODE_TYPE_TC_ARBITER_TSAR: + case SCHED_NODE_TYPE_VPORTS_TSAR: + esw_warn(node->esw->dev, + "E-Switch %s %s scheduling element failed (err=%d)\n", + op, sched_node_type_str[node->type], err); + break; + default: + esw_warn(node->esw->dev, + "E-Switch %s scheduling element failed (err=%d)\n", op, err); + break; } - - esw_warn(node->esw->dev, - "E-Switch %s %s scheduling element failed (err=%d)\n", - op, sched_node_type_str[node->type], err); } static int esw_qos_node_create_sched_element(struct mlx5_esw_sched_node *node, void *ctx, @@ -306,7 +341,7 @@ static int esw_qos_set_node_min_rate(struct mlx5_esw_sched_node *node, } static int esw_qos_create_node_sched_elem(struct mlx5_core_dev *dev, u32 parent_element_id, - u32 *tsar_ix) + u32 max_rate, u32 bw_share, u32 *tsar_ix) { u32 tsar_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; void *attr; @@ -323,6 +358,8 @@ static int esw_qos_create_node_sched_elem(struct mlx5_core_dev *dev, u32 parent_ SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR); MLX5_SET(scheduling_context, tsar_ctx, parent_element_id, parent_element_id); + MLX5_SET(scheduling_context, tsar_ctx, max_average_bw, max_rate); + MLX5_SET(scheduling_context, tsar_ctx, bw_share, bw_share); attr = MLX5_ADDR_OF(scheduling_context, tsar_ctx, element_attributes); MLX5_SET(tsar_element, attr, tsar_type, TSAR_ELEMENT_TSAR_TYPE_DWRR); @@ -358,7 +395,6 @@ static struct mlx5_esw_sched_node * __esw_qos_alloc_node(struct mlx5_eswitch *esw, u32 tsar_ix, enum sched_node_type type, struct mlx5_esw_sched_node *parent) { - struct list_head *parent_children; struct mlx5_esw_sched_node *node; node = kzalloc(sizeof(*node), GFP_KERNEL); @@ -370,8 +406,10 @@ __esw_qos_alloc_node(struct mlx5_eswitch *esw, u32 tsar_ix, enum sched_node_type node->type = type; node->parent = parent; INIT_LIST_HEAD(&node->children); - parent_children = parent ? &parent->children : &esw->qos.domain->nodes; - list_add_tail(&node->entry, parent_children); + if (parent) + list_add_tail(&node->entry, &parent->children); + else + INIT_LIST_HEAD(&node->entry); return node; } @@ -396,7 +434,7 @@ __esw_qos_create_vports_sched_node(struct mlx5_eswitch *esw, struct mlx5_esw_sch u32 tsar_ix; int err; - err = esw_qos_create_node_sched_elem(esw->dev, esw->qos.root_tsar_ix, &tsar_ix); + err = esw_qos_create_node_sched_elem(esw->dev, esw->qos.root_tsar_ix, 0, 0, &tsar_ix); if (err) { NL_SET_ERR_MSG_MOD(extack, "E-Switch create TSAR for node failed"); return ERR_PTR(err); @@ -409,6 +447,7 @@ __esw_qos_create_vports_sched_node(struct mlx5_eswitch *esw, struct mlx5_esw_sch goto err_alloc_node; } + list_add_tail(&node->entry, &esw->qos.domain->nodes); esw_qos_normalize_min_rate(esw, NULL, extack); trace_mlx5_esw_node_qos_create(esw->dev, node, node->ix); @@ -463,7 +502,7 @@ static int esw_qos_create(struct mlx5_eswitch *esw, struct netlink_ext_ack *exta if (!MLX5_CAP_GEN(dev, qos) || !MLX5_CAP_QOS(dev, esw_scheduling)) return -EOPNOTSUPP; - err = esw_qos_create_node_sched_elem(esw->dev, 0, &esw->qos.root_tsar_ix); + err = esw_qos_create_node_sched_elem(esw->dev, 0, 0, 0, &esw->qos.root_tsar_ix); if (err) { esw_warn(dev, "E-Switch create root TSAR failed (%d)\n", err); return err; @@ -475,11 +514,11 @@ static int esw_qos_create(struct mlx5_eswitch *esw, struct netlink_ext_ack *exta /* The eswitch doesn't support scheduling nodes. * Create a software-only node0 using the root TSAR to attach vport QoS to. */ - if (!__esw_qos_alloc_node(esw, - esw->qos.root_tsar_ix, - SCHED_NODE_TYPE_VPORTS_TSAR, + if (!__esw_qos_alloc_node(esw, esw->qos.root_tsar_ix, SCHED_NODE_TYPE_VPORTS_TSAR, NULL)) esw->qos.node0 = ERR_PTR(-ENOMEM); + else + list_add_tail(&esw->qos.node0->entry, &esw->qos.domain->nodes); } if (IS_ERR(esw->qos.node0)) { err = PTR_ERR(esw->qos.node0); @@ -537,6 +576,17 @@ static void esw_qos_put(struct mlx5_eswitch *esw) esw_qos_destroy(esw); } +static void esw_qos_tc_arbiter_scheduling_teardown(struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{} + +static int esw_qos_tc_arbiter_scheduling_setup(struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + NL_SET_ERR_MSG_MOD(extack, "TC arbiter elements are not supported."); + return -EOPNOTSUPP; +} + static void esw_qos_vport_disable(struct mlx5_vport *vport, struct netlink_ext_ack *extack) { struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; @@ -699,6 +749,158 @@ static int esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw return err; } +static void esw_qos_switch_vport_tcs_to_vport(struct mlx5_esw_sched_node *tc_arbiter_node, + struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vports_tc_node, *vport_tc_node, *tmp; + + vports_tc_node = list_first_entry(&tc_arbiter_node->children, struct mlx5_esw_sched_node, + entry); + + list_for_each_entry_safe(vport_tc_node, tmp, &vports_tc_node->children, entry) + esw_qos_vport_update_parent(vport_tc_node->vport, node, extack); +} + +static int esw_qos_switch_tc_arbiter_node_to_vports(struct mlx5_esw_sched_node *tc_arbiter_node, + struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + u32 parent_tsar_ix = node->parent ? node->parent->ix : node->esw->qos.root_tsar_ix; + int err; + + err = esw_qos_create_node_sched_elem(node->esw->dev, parent_tsar_ix, node->max_rate, + node->bw_share, &node->ix); + if (err) { + NL_SET_ERR_MSG_MOD(extack, + "Failed to create scheduling element for vports node when disabliing vports TC QoS"); + return err; + } + + node->type = SCHED_NODE_TYPE_VPORTS_TSAR; + + /* Disable TC QoS for vports in the arbiter node. */ + esw_qos_switch_vport_tcs_to_vport(tc_arbiter_node, node, extack); + + return 0; +} + +static int esw_qos_switch_vports_node_to_tc_arbiter(struct mlx5_esw_sched_node *node, + struct mlx5_esw_sched_node *tc_arbiter_node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node, *tmp; + struct mlx5_vport *vport; + int err; + + /* Enable TC QoS for each vport in the node. */ + list_for_each_entry_safe(vport_node, tmp, &node->children, entry) { + vport = vport_node->vport; + err = esw_qos_vport_update_parent(vport, tc_arbiter_node, extack); + if (err) + goto err_out; + } + + /* Destroy the current vports node TSAR. */ + err = mlx5_destroy_scheduling_element_cmd(node->esw->dev, SCHEDULING_HIERARCHY_E_SWITCH, + node->ix); + if (err) + goto err_out; + + return 0; +err_out: + /* Restore vports back into the node if an error occurs. */ + esw_qos_switch_vport_tcs_to_vport(tc_arbiter_node, node, NULL); + + return err; +} + +static struct mlx5_esw_sched_node *esw_qos_move_node(struct mlx5_esw_sched_node *curr_node) +{ + struct mlx5_esw_sched_node *new_node; + + new_node = __esw_qos_alloc_node(curr_node->esw, curr_node->ix, curr_node->type, NULL); + if (!IS_ERR(new_node)) + esw_qos_nodes_set_parent(&curr_node->children, new_node); + + return new_node; +} + +static int esw_qos_node_disable_tc_arbitration(struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *curr_node; + int err; + + if (node->type != SCHED_NODE_TYPE_TC_ARBITER_TSAR) + return 0; + + /* Allocate a new rate node to hold the current state, which will allow + * for restoring the vports back to this node after disabling TC arbitration. + */ + curr_node = esw_qos_move_node(node); + if (IS_ERR(curr_node)) { + NL_SET_ERR_MSG_MOD(extack, "Failed setting up vports node"); + return PTR_ERR(curr_node); + } + + /* Disable TC QoS for all vports, and assign them back to the node. */ + err = esw_qos_switch_tc_arbiter_node_to_vports(curr_node, node, extack); + if (err) + goto err_out; + + /* Clean up the TC arbiter node after disabling TC QoS for vports. */ + esw_qos_tc_arbiter_scheduling_teardown(curr_node, extack); + goto out; +err_out: + esw_qos_nodes_set_parent(&curr_node->children, node); +out: + __esw_qos_free_node(curr_node); + return err; +} + +static int esw_qos_node_enable_tc_arbitration(struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *curr_node; + int err; + + if (node->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) + return 0; + + /* Allocate a new node that will store the information of the current node. + * This will be used later to restore the node if necessary. + */ + curr_node = esw_qos_move_node(node); + if (IS_ERR(curr_node)) { + NL_SET_ERR_MSG_MOD(extack, "Failed setting up node TC QoS"); + return PTR_ERR(curr_node); + } + + /* Initialize the TC arbiter node for QoS management. + * This step prepares the node for handling Traffic Class arbitration. + */ + err = esw_qos_tc_arbiter_scheduling_setup(node, extack); + if (err) + goto err_setup; + + /* Enable TC QoS for each vport within the current node. */ + err = esw_qos_switch_vports_node_to_tc_arbiter(curr_node, node, extack); + if (err) + goto err_switch_vports; + goto out; + +err_switch_vports: + esw_qos_tc_arbiter_scheduling_teardown(node, NULL); + node->ix = curr_node->ix; + node->type = curr_node->type; +err_setup: + esw_qos_nodes_set_parent(&curr_node->children, node); +out: + __esw_qos_free_node(curr_node); + return err; +} + static u32 mlx5_esw_qos_lag_link_speed_get_locked(struct mlx5_core_dev *mdev) { struct ethtool_link_ksettings lksettings; @@ -824,6 +1026,30 @@ static int esw_qos_devlink_rate_to_mbps(struct mlx5_core_dev *mdev, const char * return 0; } +static bool esw_qos_validate_unsupported_tc_bw(struct mlx5_eswitch *esw, u32 *tc_bw) +{ + int i, num_tcs = esw_qos_num_tcs(esw->dev); + + for (i = num_tcs; i < IEEE_8021QAZ_MAX_TCS; i++) { + if (tc_bw[i]) + return false; + } + + return true; +} + +static bool esw_qos_tc_bw_disabled(u32 *tc_bw) +{ + int i; + + for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) { + if (tc_bw[i]) + return false; + } + + return true; +} + int mlx5_esw_qos_init(struct mlx5_eswitch *esw) { if (esw->qos.domain) @@ -892,8 +1118,27 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *p int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *priv, u32 *tc_bw, struct netlink_ext_ack *extack) { - NL_SET_ERR_MSG_MOD(extack, "TC bandwidth shares are not supported on nodes"); - return -EOPNOTSUPP; + struct mlx5_esw_sched_node *node = priv; + struct mlx5_eswitch *esw = node->esw; + bool disable; + int err; + + if (!esw_qos_validate_unsupported_tc_bw(esw, tc_bw)) { + NL_SET_ERR_MSG_MOD(extack, "E-Switch traffic classes number is not supported"); + return -EOPNOTSUPP; + } + + disable = esw_qos_tc_bw_disabled(tc_bw); + esw_qos_lock(esw); + if (disable) { + err = esw_qos_node_disable_tc_arbitration(node, extack); + goto unlock; + } + + err = esw_qos_node_enable_tc_arbitration(node, extack); +unlock: + esw_qos_unlock(esw); + return err; } int mlx5_esw_devlink_rate_node_tx_share_set(struct devlink_rate *rate_node, void *priv, From patchwork Sun Feb 9 10:17:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966719 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2041.outbound.protection.outlook.com [40.107.92.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6756215CD52; Sun, 9 Feb 2025 10:19:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096365; cv=fail; b=niupTyOr8cvR/cRqRzfWi0FQ7aguZ8zuzu3ZlvWh7i5P73TbV9EV8THGLbULDkktswNFD/GqwFQr+deke0pH8yt028dR1j8m8flRnk+9O2V3IJ5ftjF/jLhxgU0u+Z299EyLu/IAQGbp4kMCYIG3R1w75aH9RSyz4ksm+5csQ1g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096365; c=relaxed/simple; bh=Rfm7YdffS5CkaXFPSyT66y02kgc4835PZrE1n70Z6Pk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Ss9NxNhxbcbwvhjfSaTgacMtVtUCR0TYqWIV1Xi5uCVA4za1oL20a7F/KSR58UvgPJ/95nbkk2Zy26UU1WN3aN5QGTQeMtY9oBKRqoRgvJxb5cLRlHzucvCzFHhRkKmIoY0chomD1mu4+lyj4BjdJeFjj7qGJ0XtFuFwWi7zr94= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=G87k+fJv; arc=fail smtp.client-ip=40.107.92.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="G87k+fJv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RLQW3O2D4lvvLrZqGxc0uyvCoDE3OA1BXCqO2GvaVmk6/YvqEy8VBOGr/UMzRJKrQkjkVkdh9tWVOKt0GuzZ9vHo90lLsV2JGsgTwidBWRCVw96JbGGqGELBuWIagJ6izoWwHW7rEMyzosv2LIT+u0DziinROrYfy55MLDDy1HQYv+GuW3usnlyV54sh6/Q0z2p3g8NBoapifhnL/1MDUB+NOVrigvy93KaZEQ4OXGlhkvWmRZ+dMf6MXeypG8DTol2F6+lLY76qrZgxkD+Z39HkVVNl4xEOuNLfvgCeT3tccb7g2wmzQVcJBKNmjOr47h3XUVK7XTiW9W1BVIUYRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=K40Blj6Y8NEcT6Y898hnDSOcc5qGjgt3KopFiVnWoXA=; b=sgecejshd4ltWBWfTQ3t/GXe4PK+YcE8mZEbhtWeUGaqXYADWxwnOUURSfbsanIfD9iJZJ23QKvsHr3AxNtdUmCqmB0hAHM9F0Rxn2pZR2j/UP+zhZ7s8es6xNV5N8fEZ/YybMX3+bGujU8NCGqk8PkhyJ9+ESeow5wKZ7w8qLYfvgDYYlr4D+UMZwoQqEfWBNrmQhST85u79dLdA+SyjImEANnTdQmf/oqErSP+JRah0nvCqA1HdYQtiu1L6r5CXZXHctfgqPup8DFpONFms25ScejHeZzinY+nTRaz1Hea+xBf6H7FwTP362tgSxMO3YwdAjI6HvpwD9Gpv5dqBQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K40Blj6Y8NEcT6Y898hnDSOcc5qGjgt3KopFiVnWoXA=; b=G87k+fJvvGUHV+HCwyCrLWO6wdAqKcW8sYzgwojnN0e4es2acnq4VWpWb6Wcd0DEYSQ4Q7LShdDg+Lk9ezEP+RGCeL2zKo3QNiiG4ZxITa0o9YJOfbrdynlE5hw1VJ6CZRJYPzczODoKgCbEM4fhM+90/YZNCvLtLCGQWrJzTqwTt5t0CM6zTL6kd7yaFB8DLFyM8t4MiNEq3tzfAy3nVAyoicOLgwamkcge1X25zqyq6LFL4DhmmmKYgmIuLVAJPo/GjnQbRvke1MBZwunHcG3K1gu1HPu8NkVcl2fy0imv3EgIfus7nslFm8UMNj0r5jRK2mHwpKFEiVWabtjFeQ== Received: from SJ0PR03CA0094.namprd03.prod.outlook.com (2603:10b6:a03:333::9) by PH7PR12MB7454.namprd12.prod.outlook.com (2603:10b6:510:20d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.16; Sun, 9 Feb 2025 10:19:14 +0000 Received: from SJ1PEPF00002322.namprd03.prod.outlook.com (2603:10b6:a03:333:cafe::19) by SJ0PR03CA0094.outlook.office365.com (2603:10b6:a03:333::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.24 via Frontend Transport; Sun, 9 Feb 2025 10:19:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002322.mail.protection.outlook.com (10.167.242.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8398.14 via Frontend Transport; Sun, 9 Feb 2025 10:19:14 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:09 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:08 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:03 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Carolina Jubran , Cosmin Ratiu Subject: [PATCH net-next 04/15] net/mlx5: Add traffic class scheduling support for vport QoS Date: Sun, 9 Feb 2025 12:17:05 +0200 Message-ID: <20250209101716.112774-5-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002322:EE_|PH7PR12MB7454:EE_ X-MS-Office365-Filtering-Correlation-Id: 02a44008-c918-4e19-7e94-08dd48f3341e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: jAWX+I5HBGN5rwmoPWO2kNJGJJ6l85sYLQBmpebYqQZQ8IjZiaWwvqj8brpwLRAfB2vsOztZ3iSPowNUrG5QGi+rQEqmydZOyH/cpBo6dgMuSca6Hj1vUaWR9fYn0bymra9STDKCiAwJ5Rxyon8SO3vX9Mj4EMUk4USGMViW+FjHln1xYKXucB4yzl12Dd0mKHvhYeQYpFv6azzjP7ETHwOyjQ70o8m8fpRQXqxad1gSIZ5V1sa6Hh+A5p6/9hdjUk5raHLqS14F0w6/YnA+retarUPUy3hjrLQbC7N8jaw4o4J3MBPi03JkqbjXcTjYi3ssRNeLQdnjSPtc82tfNNgWoI4nHZvU0cTbrS1gQeZEBJBcbc+Mj9w88ROgy9twcEWC9+g6jvB/1/Mjrcy4FvsOahLvR+mRBkP5f52n0LtfE32InXyWpopsUyV/Wk6Dscee0DpH2vmxTxLV+o+j1hiPnb2FNn854zeeOrnDpARJejvJijs0UffYD8kroaougYc9YUJVtdYGE/nCY7swg9kkeACENsfI1XtpcXOY+ZqRD2i5fAapofHkjLjdquEejJZFdr/9EbPjRtgk8Qe27a7JK6n/TsfL6sCrQtuoiBvVwTMZrlqjwGrxPlb0a/SeX9xiCmnQ3XQU/7NKypg31YBZz514HRvm0gwfkp+f0hb9roiIPN7Wzulg4IT0mgEoq2i3CrFAeAhEH2XBTuTLPa9Uo6cxDENbFwILmXP2BFqf/mwc0T53Uh4kgkn7OBRzUKum1mpAQSPH6HqYr6XLWMsArPHk2Y0qt80mAsqH6wy4FaeVx0mVWv0xq59Px/ZVwTXwf42M2kZrQK5T15rgs2o1M6P+hCGXsRF2YYnOhBIdlhnfpjkF8Zu4VZQeDkom2ZaFprk5wfLJ0NeBsObCTkB0v8zAvCGY+SFJSQyVt11gzs5F9KCOQQ/uiJa679dfbBCfvwiKtXGl85qTRRWo4rDaYHU15NjVfMvarg+mCwk+eArruky5l2t3nnnBuOZh8x5YDuGJuORBS3OdbdHHQ2FKCmJQSL2PdXaO7JG3ruIDmtioTs5+WZ1En4wKGF6r40gFKw+DStdAX5OSW9t7hcPSxoKi/G0CYjbHWtbwuwSDqbUi8jODVe0CJqFOcd2ddfojTe5ksBR4yESR68h44Dt8gnAGNiKRj6V5Y/eS+UVslC+FDM0jkXtXmaWCd3Cg8QOmj4NGc5SRBC4xgFgVUBctz3tkUym4HW8xCVyKbdS/bLCAV57av7zH0Ka9WrMwKIHrbAdWncDB/pHZnSax1uiYvmR4w82YlehPlQaVBcgdG/QoyjyWVCywIxZO8BnMrAkrLMeW5wSH9or4mdNxjmRYVy6JHiW3Spyn9i7L0WECIxCihUDFbpLp/DGZ1gaO0qA5bCPVcMQ+k/qAa96kLO43EwLsZz0vQbe60wu9NMpEB5cq812k5Boft4QEiMp2IoLdJoGR57D0R6rHmq3ymIxA5iwRtEsG0jDAzaCzOW4= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:14.4832 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 02a44008-c918-4e19-7e94-08dd48f3341e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002322.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7454 From: Carolina Jubran Introduce support for traffic class (TC) scheduling on vports by allowing the vport to own multiple TC scheduling nodes. This patch enables more granular control of QoS by defining three distinct QoS states for vports, each providing unique scheduling behavior: 1. Regular QoS: The `sched_node` represents the vport directly, handling QoS as a single scheduling entity. 2. TC QoS on the vport: The `sched_node` acts as a TC arbiter, enabling TC scheduling directly on the vport. 3. TC QoS on the parent node: The `sched_node` functions as a rate limiter, with TC arbitration enabled at the parent level, associating multiple scheduling nodes with each vport. Key changes include: - Added support for new scheduling elements, vport traffic class and rate limiter. - New helper functions for creating, destroying, and restoring vport TC scheduling nodes, handling transitions between regular QoS and TC arbitration states. - Updated `esw_qos_vport_enable()` and `esw_qos_vport_disable()` to support both regular QoS and TC arbitration states, ensuring consistent transitions between scheduling modes. - Introduced a `sched_nodes` array under `vport->qos` to store multiple TC scheduling nodes per vport, enabling finer control over per-TC QoS. - Enhanced `esw_qos_vport_update_parent()` to handle transitions between the three QoS states based on the current and new parent node types. This patch lays the groundwork for future support for configuring tc-bw on vports. Although the infrastructure is in place, full support for tc-bw is not yet implemented; attempts to set tc-bw on vports will return `-EOPNOTSUPP`. No functional changes are introduced at this stage. Signed-off-by: Carolina Jubran Reviewed-by: Cosmin Ratiu Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 361 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 13 +- 2 files changed, 353 insertions(+), 21 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c index efcbd3180317..84f680aecfe2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c @@ -65,12 +65,16 @@ enum sched_node_type { SCHED_NODE_TYPE_VPORTS_TSAR, SCHED_NODE_TYPE_VPORT, SCHED_NODE_TYPE_TC_ARBITER_TSAR, + SCHED_NODE_TYPE_RATE_LIMITER, + SCHED_NODE_TYPE_VPORT_TC, }; static const char * const sched_node_type_str[] = { [SCHED_NODE_TYPE_VPORTS_TSAR] = "vports TSAR", [SCHED_NODE_TYPE_VPORT] = "vport", [SCHED_NODE_TYPE_TC_ARBITER_TSAR] = "TC Arbiter TSAR", + [SCHED_NODE_TYPE_RATE_LIMITER] = "Rate Limiter", + [SCHED_NODE_TYPE_VPORT_TC] = "vport TC", }; struct mlx5_esw_sched_node { @@ -92,6 +96,8 @@ struct mlx5_esw_sched_node { struct list_head children; /* Valid only if this node is associated with a vport. */ struct mlx5_vport *vport; + /* Valid only when this node represents a traffic class. */ + u8 tc; }; static int esw_qos_num_tcs(struct mlx5_core_dev *dev) @@ -131,6 +137,14 @@ esw_qos_nodes_set_parent(struct list_head *nodes, struct mlx5_esw_sched_node *pa void mlx5_esw_qos_vport_qos_free(struct mlx5_vport *vport) { + if (vport->qos.sched_nodes) { + int i, num_tcs = esw_qos_num_tcs(vport->qos.sched_node->esw->dev); + + for (i = 0; i < num_tcs; i++) + kfree(vport->qos.sched_nodes[i]); + kfree(vport->qos.sched_nodes); + } + kfree(vport->qos.sched_node); memset(&vport->qos, 0, sizeof(vport->qos)); } @@ -155,11 +169,17 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport) static void esw_qos_sched_elem_warn(struct mlx5_esw_sched_node *node, int err, const char *op) { switch (node->type) { + case SCHED_NODE_TYPE_VPORT_TC: + esw_warn(node->esw->dev, + "E-Switch %s %s scheduling element failed (vport=%d,tc=%d,err=%d)\n", + op, sched_node_type_str[node->type], node->vport->vport, node->tc, err); + break; case SCHED_NODE_TYPE_VPORT: esw_warn(node->esw->dev, "E-Switch %s %s scheduling element failed (vport=%d,err=%d)\n", op, sched_node_type_str[node->type], node->vport->vport, err); break; + case SCHED_NODE_TYPE_RATE_LIMITER: case SCHED_NODE_TYPE_TC_ARBITER_TSAR: case SCHED_NODE_TYPE_VPORTS_TSAR: esw_warn(node->esw->dev, @@ -253,6 +273,23 @@ static int esw_qos_sched_elem_config(struct mlx5_esw_sched_node *node, u32 max_r return 0; } +static int esw_qos_create_rate_limit_element(struct mlx5_esw_sched_node *node, + struct netlink_ext_ack *extack) +{ + u32 sched_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; + + if (!mlx5_qos_element_type_supported(node->esw->dev, + SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT, + SCHEDULING_HIERARCHY_E_SWITCH)) + return -EOPNOTSUPP; + + MLX5_SET(scheduling_context, sched_ctx, max_average_bw, node->max_rate); + MLX5_SET(scheduling_context, sched_ctx, element_type, + SCHEDULING_CONTEXT_ELEMENT_TYPE_RATE_LIMIT); + + return esw_qos_node_create_sched_element(node, sched_ctx, extack); +} + static u32 esw_qos_calculate_min_rate_divider(struct mlx5_eswitch *esw, struct mlx5_esw_sched_node *parent) { @@ -391,6 +428,31 @@ static int esw_qos_vport_create_sched_element(struct mlx5_esw_sched_node *vport_ return esw_qos_node_create_sched_element(vport_node, sched_ctx, extack); } +static int esw_qos_vport_tc_create_sched_element(struct mlx5_esw_sched_node *vport_tc_node, + u32 rate_limit_elem_ix, + struct netlink_ext_ack *extack) +{ + u32 sched_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; + struct mlx5_core_dev *dev = vport_tc_node->esw->dev; + void *attr; + + if (!mlx5_qos_element_type_supported(dev, + SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC, + SCHEDULING_HIERARCHY_E_SWITCH)) + return -EOPNOTSUPP; + + MLX5_SET(scheduling_context, sched_ctx, element_type, + SCHEDULING_CONTEXT_ELEMENT_TYPE_VPORT_TC); + attr = MLX5_ADDR_OF(scheduling_context, sched_ctx, element_attributes); + MLX5_SET(vport_tc_element, attr, vport_number, vport_tc_node->vport->vport); + MLX5_SET(vport_tc_element, attr, traffic_class, vport_tc_node->tc); + MLX5_SET(scheduling_context, sched_ctx, max_bw_obj_id, rate_limit_elem_ix); + MLX5_SET(scheduling_context, sched_ctx, parent_element_id, vport_tc_node->parent->ix); + MLX5_SET(scheduling_context, sched_ctx, bw_share, vport_tc_node->bw_share); + + return esw_qos_node_create_sched_element(vport_tc_node, sched_ctx, extack); +} + static struct mlx5_esw_sched_node * __esw_qos_alloc_node(struct mlx5_eswitch *esw, u32 tsar_ix, enum sched_node_type type, struct mlx5_esw_sched_node *parent) @@ -587,12 +649,169 @@ static int esw_qos_tc_arbiter_scheduling_setup(struct mlx5_esw_sched_node *node, return -EOPNOTSUPP; } +static int esw_qos_create_vport_tc_sched_node(struct mlx5_vport *vport, + u32 rate_limit_elem_ix, + struct mlx5_esw_sched_node *vports_tc_node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; + struct mlx5_esw_sched_node *vport_tc_node; + u8 tc = vports_tc_node->tc; + int err; + + vport_tc_node = __esw_qos_alloc_node(vport_node->esw, 0, SCHED_NODE_TYPE_VPORT_TC, + vports_tc_node); + if (!vport_tc_node) + return -ENOMEM; + + vport_tc_node->min_rate = vport_node->min_rate; + vport_tc_node->tc = tc; + vport_tc_node->vport = vport; + err = esw_qos_vport_tc_create_sched_element(vport_tc_node, rate_limit_elem_ix, extack); + if (err) + goto err_out; + + vport->qos.sched_nodes[tc] = vport_tc_node; + + return 0; +err_out: + __esw_qos_free_node(vport_tc_node); + return err; +} + +static void esw_qos_destroy_vport_tc_sched_elements(struct mlx5_vport *vport, + struct netlink_ext_ack *extack) +{ + int i, num_tcs = esw_qos_num_tcs(vport->qos.sched_node->esw->dev); + + for (i = 0; i < num_tcs; i++) { + if (vport->qos.sched_nodes[i]) + __esw_qos_destroy_node(vport->qos.sched_nodes[i], extack); + } + + kfree(vport->qos.sched_nodes); + vport->qos.sched_nodes = NULL; +} + +static int esw_qos_create_vport_tc_sched_elements(struct mlx5_vport *vport, + enum sched_node_type type, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; + struct mlx5_esw_sched_node *tc_arbiter_node, *vports_tc_node; + int err, num_tcs = esw_qos_num_tcs(vport_node->esw->dev); + u32 rate_limit_elem_ix; + + vport->qos.sched_nodes = kcalloc(num_tcs, sizeof(struct mlx5_esw_sched_node *), GFP_KERNEL); + if (!vport->qos.sched_nodes) { + NL_SET_ERR_MSG_MOD(extack, "Allocating the vport TC scheduling elements failed."); + return -ENOMEM; + } + + rate_limit_elem_ix = type == SCHED_NODE_TYPE_RATE_LIMITER ? vport_node->ix : 0; + tc_arbiter_node = type == SCHED_NODE_TYPE_RATE_LIMITER ? vport_node->parent : vport_node; + list_for_each_entry(vports_tc_node, &tc_arbiter_node->children, entry) { + err = esw_qos_create_vport_tc_sched_node(vport, rate_limit_elem_ix, vports_tc_node, + extack); + if (err) + goto err_create_vport_tc; + } + + return 0; + +err_create_vport_tc: + esw_qos_destroy_vport_tc_sched_elements(vport, NULL); + + return err; +} + +static int esw_qos_vport_tc_enable(struct mlx5_vport *vport, enum sched_node_type type, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; + int err; + + if (type == SCHED_NODE_TYPE_TC_ARBITER_TSAR && + MLX5_CAP_QOS(vport_node->esw->dev, log_esw_max_sched_depth) < 2) { + NL_SET_ERR_MSG_MOD(extack, "Setting up TC Arbiter for a vport is not supported."); + return -EOPNOTSUPP; + } + + esw_assert_qos_lock_held(vport->dev->priv.eswitch); + + if (type == SCHED_NODE_TYPE_RATE_LIMITER) + err = esw_qos_create_rate_limit_element(vport_node, extack); + else + err = esw_qos_tc_arbiter_scheduling_setup(vport_node, extack); + if (err) + return err; + + /* Rate limiters impact multiple nodes not directly connected to them + * and are not direct members of the QoS hierarchy. + * Unlink it from the parent to reflect that. + */ + if (type == SCHED_NODE_TYPE_RATE_LIMITER) + list_del_init(&vport_node->entry); + + err = esw_qos_create_vport_tc_sched_elements(vport, type, extack); + if (err) + goto err_sched_nodes; + + return 0; + +err_sched_nodes: + if (type == SCHED_NODE_TYPE_RATE_LIMITER) { + esw_qos_node_destroy_sched_element(vport_node, NULL); + list_add_tail(&vport_node->entry, &vport_node->parent->children); + } else { + esw_qos_tc_arbiter_scheduling_teardown(vport_node, NULL); + } + return err; +} + +static void esw_qos_vport_tc_disable(struct mlx5_vport *vport, struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; + enum sched_node_type curr_type = vport_node->type; + + esw_qos_destroy_vport_tc_sched_elements(vport, extack); + + if (curr_type == SCHED_NODE_TYPE_RATE_LIMITER) + esw_qos_node_destroy_sched_element(vport_node, extack); + else + esw_qos_tc_arbiter_scheduling_teardown(vport_node, extack); +} + +static int esw_qos_set_vport_tcs_min_rate(struct mlx5_vport *vport, u32 min_rate, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; + int err, i, num_tcs = esw_qos_num_tcs(vport_node->esw->dev); + + for (i = 0; i < num_tcs; i++) { + err = esw_qos_set_node_min_rate(vport->qos.sched_nodes[i], min_rate, extack); + if (err) + goto err_out; + } + vport_node->min_rate = min_rate; + + return 0; +err_out: + for (--i; i >= 0; i--) + esw_qos_set_node_min_rate(vport->qos.sched_nodes[i], vport_node->min_rate, extack); + return err; +} + static void esw_qos_vport_disable(struct mlx5_vport *vport, struct netlink_ext_ack *extack) { struct mlx5_esw_sched_node *vport_node = vport->qos.sched_node; struct mlx5_esw_sched_node *parent = vport_node->parent; + enum sched_node_type curr_type = vport_node->type; - esw_qos_node_destroy_sched_element(vport_node, extack); + if (curr_type == SCHED_NODE_TYPE_VPORT) + esw_qos_node_destroy_sched_element(vport_node, extack); + else + esw_qos_vport_tc_disable(vport, extack); vport_node->bw_share = 0; list_del_init(&vport_node->entry); @@ -601,7 +820,8 @@ static void esw_qos_vport_disable(struct mlx5_vport *vport, struct netlink_ext_a trace_mlx5_esw_vport_qos_destroy(vport_node->esw->dev, vport); } -static int esw_qos_vport_enable(struct mlx5_vport *vport, struct mlx5_esw_sched_node *parent, +static int esw_qos_vport_enable(struct mlx5_vport *vport, enum sched_node_type type, + struct mlx5_esw_sched_node *parent, struct netlink_ext_ack *extack) { int err; @@ -609,10 +829,14 @@ static int esw_qos_vport_enable(struct mlx5_vport *vport, struct mlx5_esw_sched_ esw_assert_qos_lock_held(vport->dev->priv.eswitch); esw_qos_node_set_parent(vport->qos.sched_node, parent); - err = esw_qos_vport_create_sched_element(vport->qos.sched_node, extack); + if (type == SCHED_NODE_TYPE_VPORT) + err = esw_qos_vport_create_sched_element(vport->qos.sched_node, extack); + else + err = esw_qos_vport_tc_enable(vport, type, extack); if (err) return err; + vport->qos.sched_node->type = type; esw_qos_normalize_min_rate(parent->esw, parent, extack); return 0; @@ -640,7 +864,7 @@ static int mlx5_esw_qos_vport_enable(struct mlx5_vport *vport, enum sched_node_t sched_node->min_rate = min_rate; sched_node->vport = vport; vport->qos.sched_node = sched_node; - err = esw_qos_vport_enable(vport, parent, extack); + err = esw_qos_vport_enable(vport, type, parent, extack); if (err) esw_qos_put(esw); @@ -692,6 +916,8 @@ static int mlx5_esw_qos_set_vport_min_rate(struct mlx5_vport *vport, u32 min_rat if (!vport_node) return mlx5_esw_qos_vport_enable(vport, SCHED_NODE_TYPE_VPORT, NULL, 0, min_rate, extack); + else if (vport_node->type == SCHED_NODE_TYPE_RATE_LIMITER) + return esw_qos_set_vport_tcs_min_rate(vport, min_rate, extack); else return esw_qos_set_node_min_rate(vport_node, min_rate, extack); } @@ -724,12 +950,59 @@ bool mlx5_esw_qos_get_vport_rate(struct mlx5_vport *vport, u32 *max_rate, u32 *m return enabled; } +static int esw_qos_vport_tc_check_type(enum sched_node_type curr_type, + enum sched_node_type new_type, + struct netlink_ext_ack *extack) +{ + if (curr_type == SCHED_NODE_TYPE_TC_ARBITER_TSAR && + new_type == SCHED_NODE_TYPE_RATE_LIMITER) { + NL_SET_ERR_MSG_MOD(extack, + "Cannot switch from vport-level TC arbitration to node-level TC arbitration"); + return -EOPNOTSUPP; + } + + if (curr_type == SCHED_NODE_TYPE_RATE_LIMITER && + new_type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) { + NL_SET_ERR_MSG_MOD(extack, + "Cannot switch from node-level TC arbitration to vport-level TC arbitration"); + return -EOPNOTSUPP; + } + + return 0; +} + +static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type type, + struct mlx5_esw_sched_node *parent, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *curr_parent = vport->qos.sched_node->parent; + enum sched_node_type curr_type = vport->qos.sched_node->type; + int err; + + esw_assert_qos_lock_held(vport->dev->priv.eswitch); + parent = parent ?: curr_parent; + if (curr_type == type && curr_parent == parent) + return 0; + + err = esw_qos_vport_tc_check_type(curr_type, type, extack); + if (err) + return err; + + esw_qos_vport_disable(vport, extack); + + err = esw_qos_vport_enable(vport, type, parent, extack); + if (err) + esw_qos_vport_enable(vport, curr_type, curr_parent, NULL); + + return err; +} + static int esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw_sched_node *parent, struct netlink_ext_ack *extack) { struct mlx5_eswitch *esw = vport->dev->priv.eswitch; struct mlx5_esw_sched_node *curr_parent; - int err; + enum sched_node_type type; esw_assert_qos_lock_held(esw); curr_parent = vport->qos.sched_node->parent; @@ -737,16 +1010,17 @@ static int esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw if (curr_parent == parent) return 0; - esw_qos_vport_disable(vport, extack); - - err = esw_qos_vport_enable(vport, parent, extack); - if (err) { - if (esw_qos_vport_enable(vport, curr_parent, NULL)) - esw_warn(parent->esw->dev, "vport restore QoS failed (vport=%d)\n", - vport->vport); - } + /* Set vport QoS type based on parent node type if different from default QoS; + * otherwise, use the vport's current QoS type. + */ + if (parent->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) + type = SCHED_NODE_TYPE_RATE_LIMITER; + else if (curr_parent->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) + type = SCHED_NODE_TYPE_VPORT; + else + type = vport->qos.sched_node->type; - return err; + return esw_qos_vport_update(vport, type, parent, extack); } static void esw_qos_switch_vport_tcs_to_vport(struct mlx5_esw_sched_node *tc_arbiter_node, @@ -1038,6 +1312,14 @@ static bool esw_qos_validate_unsupported_tc_bw(struct mlx5_eswitch *esw, u32 *tc return true; } +static bool esw_qos_vport_validate_unsupported_tc_bw(struct mlx5_vport *vport, u32 *tc_bw) +{ + struct mlx5_eswitch *esw = vport->qos.sched_node ? + vport->qos.sched_node->parent->esw : vport->dev->priv.eswitch; + + return esw_qos_validate_unsupported_tc_bw(esw, tc_bw); +} + static bool esw_qos_tc_bw_disabled(u32 *tc_bw) { int i; @@ -1111,8 +1393,45 @@ int mlx5_esw_devlink_rate_leaf_tx_max_set(struct devlink_rate *rate_leaf, void * int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *priv, u32 *tc_bw, struct netlink_ext_ack *extack) { - NL_SET_ERR_MSG_MOD(extack, "TC bandwidth shares are not supported on leafs"); - return -EOPNOTSUPP; + struct mlx5_esw_sched_node *vport_node; + struct mlx5_vport *vport = priv; + struct mlx5_eswitch *esw; + bool disable; + int err = 0; + + esw = vport->dev->priv.eswitch; + if (!mlx5_esw_allowed(esw)) + return -EPERM; + + disable = esw_qos_tc_bw_disabled(tc_bw); + esw_qos_lock(esw); + + if (!esw_qos_vport_validate_unsupported_tc_bw(vport, tc_bw)) { + NL_SET_ERR_MSG_MOD(extack, "E-Switch traffic classes number is not supported"); + err = -EOPNOTSUPP; + goto unlock; + } + + vport_node = vport->qos.sched_node; + if (disable && !vport_node) + goto unlock; + + if (disable) { + if (vport_node->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) + err = esw_qos_vport_update(vport, SCHED_NODE_TYPE_VPORT, NULL, extack); + goto unlock; + } + + if (!vport_node) { + err = mlx5_esw_qos_vport_enable(vport, SCHED_NODE_TYPE_TC_ARBITER_TSAR, NULL, 0, 0, + extack); + vport_node = vport->qos.sched_node; + } else { + err = esw_qos_vport_update(vport, SCHED_NODE_TYPE_TC_ARBITER_TSAR, NULL, extack); + } +unlock: + esw_qos_unlock(esw); + return err; } int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *priv, @@ -1231,10 +1550,14 @@ int mlx5_esw_qos_vport_update_parent(struct mlx5_vport *vport, struct mlx5_esw_s } esw_qos_lock(esw); - if (!vport->qos.sched_node && parent) - err = mlx5_esw_qos_vport_enable(vport, SCHED_NODE_TYPE_VPORT, parent, 0, 0, extack); - else if (vport->qos.sched_node) + if (!vport->qos.sched_node && parent) { + enum sched_node_type type = parent->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR ? + SCHED_NODE_TYPE_RATE_LIMITER : SCHED_NODE_TYPE_VPORT; + + err = mlx5_esw_qos_vport_enable(vport, type, parent, 0, 0, extack); + } else if (vport->qos.sched_node) { err = esw_qos_vport_update_parent(vport, parent, extack); + } esw_qos_unlock(esw); return err; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index 8573d36785f4..9cd231a6b924 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -212,10 +212,19 @@ struct mlx5_vport { struct mlx5_vport_info info; - /* Protected with the E-Switch qos domain lock. */ + /* Protected with the E-Switch qos domain lock. The Vport QoS can + * either be disabled (sched_node is NULL) or in one of three states: + * 1. Regular QoS (sched_node is a vport node). + * 2. TC QoS enabled on the vport (sched_node is a TC arbiter). + * 3. TC QoS enabled on the vport's parent node + * (sched_node is a rate limit node). + * When TC is enabled in either mode, the vport owns vport TC scheduling nodes. + */ struct { - /* Vport scheduling element node. */ + /* Vport scheduling node. */ struct mlx5_esw_sched_node *sched_node; + /* Array of vport traffic class scheduling nodes. */ + struct mlx5_esw_sched_node **sched_nodes; } qos; u16 vport; From patchwork Sun Feb 9 10:17:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966720 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2062.outbound.protection.outlook.com [40.107.220.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA4E1195B37; Sun, 9 Feb 2025 10:19:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096368; cv=fail; b=BLxvDLuV2U6bx9xO48dOMjPIFNb+K7qqMLqckYSFfH4i2ytQdK/Ibt+1KGxJJIdAh0idLzmkAB1EtUVA9+3V5WaPO7FdDC/9f6svq4wotq+qwpHDqeDj+xnPk7YhA0Cz0Lm8bXpxrzCS7L3kN55pD72oUtwoaXXGedxfznWtZ/E= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096368; c=relaxed/simple; bh=f0tkYPqNgjlWhutZqG5p0PGxUCDZN6UCpb8nyxIkk2s=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dcTn2dFYmItx5mzi4Hcxr2YvJlmUrDAgmN8fA5OOCzVHArMjq2Va2ML7wZ2pMxsPxB1qxXrroAxo9mEABdMsWFlqnm2XUp8xzECEH9n3VI0U1YfmKJPX4pFSTa4EqVV3ua4SPCQsPatbNs8uybidIYrwB8Mf7RL7WqBu5R2oHrY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=LpyVLQdb; arc=fail smtp.client-ip=40.107.220.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="LpyVLQdb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KzpQmCTnT27St+QZjOo5/sKB7UexfiNp1uEdwyUg6d06q4ZxIAcO2ThssDd+dYH7fS+LQSfjAv/ZVxhbeq7pHLXYU7SBs0Saj2VglhWxr2oiyBrmv2DieRxMp6KjH2QPDCqkIwoCte/mEV9dPGh5IaHRdQyLXsP7PhTkV8DIWTvJfxZbR1/6r2XMqYETUkmMgMlmyR8ZtsrD4FzCx/cECv2/6s2UPyqL86+folsWqoWt7XVMnFa1IyUDMTU/EOh+dRg/1mEHsWPUdl/SHos2Saj7jeLPD3SpOLdDM/Jbab9f0ZZrQm2KctLvfMTGStGw2uAezp0b8mvijIRw2+4Sqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZmVJq97C+Oq339jRPn7JB3ykfpCJEp7E+H+lmhCvXQE=; b=ED64bNzVhs1PXzCt3K+7CVECn8iEN7xXA6tixXkemailkE1wvBi+UIZ+IL90/1KjUBtYw26buY6V3XhHvs7QqkHuF/SwOvoHKUTbAsuae1YNWckOKNhS8NkOmKzqBpWfpOtjqCo2scFT8kGMo5YsvRv8JGVYuaRtv2cslL0P1f9NW8cFKU/qz+HgbzIIw0lwwtqq+x/OBEbM/qD79U6Ip7jqGSbgfD7wujU7PdXzf4fEf/IQZ9aacc+WrfbB1rva2kdamc8wjWQJ9J3ZAvDIuOa1XsnkWqhb0hi5+e3q6h72AG82cdYyjpJDLGM1nioukOrEDz13veaJs659VD+IYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZmVJq97C+Oq339jRPn7JB3ykfpCJEp7E+H+lmhCvXQE=; b=LpyVLQdb9DJBdZBdHYVVq4z65LYNsa2MSbZvFK82tmtvb0T8MEOdNj6VPZaBHNZ7U9o1scY9Yq9/KUd6CTa/FIGtsANyhkeQJY/wNj9EBZ9kKExd559kaBQ/4LLWnVWDExI/A4b4HZZmXMK57l29PsupwEFgNuHA/xvVHfvRnnvjtLUXMf93hVPkyGs8L3j3xV6uCdK7pnsDqproCFDkMrbqhlZZsFa2Tk23vqbKM8/K/70+QRLK23wbcbhldIPkONXxg1C6CpVCND/iPvsh2eC5BrtaXyXlmHQGM43zcGJV0oyZuQmHmMfUwJUbJ6Kn8AEekVPf3fEqCqs47sGBUQ== Received: from BY3PR05CA0001.namprd05.prod.outlook.com (2603:10b6:a03:254::6) by BY5PR12MB4290.namprd12.prod.outlook.com (2603:10b6:a03:20e::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.18; Sun, 9 Feb 2025 10:19:20 +0000 Received: from SJ1PEPF00002320.namprd03.prod.outlook.com (2603:10b6:a03:254:cafe::69) by BY3PR05CA0001.outlook.office365.com (2603:10b6:a03:254::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.25 via Frontend Transport; Sun, 9 Feb 2025 10:19:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002320.mail.protection.outlook.com (10.167.242.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:20 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:15 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:14 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:09 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Carolina Jubran , Cosmin Ratiu Subject: [PATCH net-next 05/15] net/mlx5: Manage TC arbiter nodes and implement full support for tc-bw Date: Sun, 9 Feb 2025 12:17:06 +0200 Message-ID: <20250209101716.112774-6-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002320:EE_|BY5PR12MB4290:EE_ X-MS-Office365-Filtering-Correlation-Id: 32257613-7fde-4aa8-d5fc-08dd48f33772 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: DvqpTZFbII4tqDmPkqahYgO8UAvH5D18gG2AHlRXNG5i7qX//H9qywtuoGBzc2EWkW3nEzJs4kSPqO4DpoOKylG2qE3uaqo24fST49WQ3QaNXaisVQGYXXJypgNnmPJuZTmHyvKXFFUoh9pyIMkIkG9AnpWt0uZczOTo81T3+9UnYhfZCcBNypHo67pl8+SGjOZyhN49WoMrkyxBeRe1eaJdqUhfxO5wjkZV+UeEXMCeOQA1bo5MbjQsdm/d83o/lTaU04BNcpvtMjn5J3j7bnibn0KcW5IyyQx5W4Pn4evvlzVRx61ZP8PMuqoCteD6GGI5XDDJrLa/Se+jQzrc+3pvsq1QuVOwtiQ5mvmLq4hylkAlgQbuSc7Y3rg1BOot2IGbnw1H7W8jkZed2WOJMrou15DJMkA3q+TX2cUplWIcXcKJ9oapqtw2Ea39qn8cLXRNCQXxE/dICTfbbylWTNWyDvTsqyJmF0rLHS1cl5JYCRuBUC3oWJkIFp9sIHODWAJO+5fScyVyIFJgiF4wMPjWWvwrkdM23c+SdooJ8E7rRX3Ots8cY5iymlZ9/Uc7AWp2a5a95Uz7lFGOsI8yyZfT1NEyeUl8Q43G0nMxzklG7ZXArhYNO8YWNtniBdE7VycMozZXudrshKYRaHpav80mPHeavEKHR43OGYnsjVspQUMI9+HexVlQf8mHKMfH8CsbGZ370elSxeauPcfu7uMH2kxtzdm+IT6Qh+8dZrZV2jd8uDOQuqgqWAkGEx9fd2Eip7JLnl5PRtAz2F22MHx+3ORkhXstrGKBFDs2exo0xLQFavAlRWBbxYa6CxSulP0zSnzxquYbCVlsP6/ZBBniqhSsLhN9V3fvCVx+ysK9N/SDU2elocZ0YKXP1SboQ9Dnt5zQJxZfw45NBQ1XBNlJFbxAFcJMlV1SgiR2/LCr7EyTPPFKV9ONNpcdgV09p/meavScow5MUn+gjC6YYsZmN3HrIZIOt8JS9fDIzsYwUcqKnEnk0/u5Rk2aXoFL8Sutm4JQxuBEq1JPrXxOrPrGSzrhv/V214ZzuadNmTrk4jGRiAeU2AArePgwgA1iq2g2/ZSjHjTy80PgnnHIn61gN0+2uTUe97QmqBEpCQC69ZYz8lFtJXJzDFHDijudOeTQbJq/hoXgSAnT01/bBStaNxV/eNN8KtXlrWA8eX7sNil8NtkuZiHPbImbEtfyZV8UrLGQTiI4ft1eRJnFeyiWJAwyl4hQ/iQbqvkNf4FxiPep6SAT1dtXij2JFx4DJoOZ00zX3jsY9rD1PPXaYbVyzc/bczSnaP4iksL9zW7DJnrA4/scabh4YYthaG1cyuq6wR3GNBXKf34G4HIgjQbIPYuFncTHT+11BctF2hAge+E4Azv0SnWr5G4DTFiQ3BTHnMJZB9HFyMuR31iJMx5JT4R4lgMWXnXSGqMk9ZSoAoQoefumyhdq345gY7ggfjOU4iTRfRkoqLQYRodBBo1lxNbtgCD44qf6i/l1dcI= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:20.0535 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32257613-7fde-4aa8-d5fc-08dd48f33772 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002320.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4290 From: Carolina Jubran Introduce support for managing Traffic Class (TC) arbiter nodes and associated vports TC nodes within the E-Switch QoS hierarchy. This patch adds support for the new scheduling node type, `SCHED_NODE_TYPE_VPORTS_TC_TSAR`, and implements full support for setting tc-bw on both vports and nodes. Key changes include: - Introduced the new scheduling node type, `SCHED_NODE_TYPE_VPORTS_TC_TSAR`, for managing vports within the TC arbiter node. - New helper functions for creating and destroying vports TC nodes under the TC arbiter. - Updated the minimum rate normalization function to skip nodes of type `SCHED_NODE_TYPE_VPORTS_TC_TSAR`. Vports TC TSARs have bandwidth shares configured on them but not minimum rates, so their `min_rate` cannot be normalized. - Implementation of `esw_qos_tc_arbiter_scheduling_setup()` and `esw_qos_tc_arbiter_scheduling_teardown()` for initializing and cleaning up TC arbiter scheduling elements. These functions now fully support tc-bw configuration on TC arbiter nodes. - Added `esw_qos_tc_arbiter_get_bw_shares()` and `esw_qos_set_tc_arbiter_bw_shares()` to handle the settings of bandwidth shares for vports traffic class TSARs. - Refactored `mlx5_esw_devlink_rate_node_tc_bw_set()` and `mlx5_esw_devlink_rate_leaf_tc_bw_set()` to fully support configuring tc-bw on devlink rate nodes and vports, respectively. Signed-off-by: Carolina Jubran Reviewed-by: Cosmin Ratiu Signed-off-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 185 +++++++++++++++++- 1 file changed, 180 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c index 84f680aecfe2..097fda4d746b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/qos.c @@ -67,6 +67,7 @@ enum sched_node_type { SCHED_NODE_TYPE_TC_ARBITER_TSAR, SCHED_NODE_TYPE_RATE_LIMITER, SCHED_NODE_TYPE_VPORT_TC, + SCHED_NODE_TYPE_VPORTS_TC_TSAR, }; static const char * const sched_node_type_str[] = { @@ -75,6 +76,7 @@ static const char * const sched_node_type_str[] = { [SCHED_NODE_TYPE_TC_ARBITER_TSAR] = "TC Arbiter TSAR", [SCHED_NODE_TYPE_RATE_LIMITER] = "Rate Limiter", [SCHED_NODE_TYPE_VPORT_TC] = "vport TC", + [SCHED_NODE_TYPE_VPORTS_TC_TSAR] = "vports TC TSAR", }; struct mlx5_esw_sched_node { @@ -169,6 +171,11 @@ mlx5_esw_qos_vport_get_parent(const struct mlx5_vport *vport) static void esw_qos_sched_elem_warn(struct mlx5_esw_sched_node *node, int err, const char *op) { switch (node->type) { + case SCHED_NODE_TYPE_VPORTS_TC_TSAR: + esw_warn(node->esw->dev, + "E-Switch %s %s scheduling element failed (tc=%d,err=%d)\n", + op, sched_node_type_str[node->type], node->tc, err); + break; case SCHED_NODE_TYPE_VPORT_TC: esw_warn(node->esw->dev, "E-Switch %s %s scheduling element failed (vport=%d,tc=%d,err=%d)\n", @@ -354,7 +361,11 @@ static void esw_qos_normalize_min_rate(struct mlx5_eswitch *esw, if (node->esw != esw || node->ix == esw->qos.root_tsar_ix) continue; - esw_qos_update_sched_node_bw_share(node, divider, extack); + /* Vports TC TSARs don't have a minimum rate configured, + * so there's no need to update the bw_share on them. + */ + if (node->type != SCHED_NODE_TYPE_VPORTS_TC_TSAR) + esw_qos_update_sched_node_bw_share(node, divider, extack); if (list_empty(&node->children)) continue; @@ -488,6 +499,129 @@ static void esw_qos_destroy_node(struct mlx5_esw_sched_node *node, struct netlin __esw_qos_free_node(node); } +static int esw_qos_create_vports_tc_node(struct mlx5_esw_sched_node *parent, u8 tc, + struct netlink_ext_ack *extack) +{ + u32 tsar_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; + struct mlx5_core_dev *dev = parent->esw->dev; + struct mlx5_esw_sched_node *vports_tc_node; + void *attr; + int err; + + if (!mlx5_qos_element_type_supported(dev, + SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR, + SCHEDULING_HIERARCHY_E_SWITCH) || + !mlx5_qos_tsar_type_supported(dev, + TSAR_ELEMENT_TSAR_TYPE_DWRR, + SCHEDULING_HIERARCHY_E_SWITCH)) + return -EOPNOTSUPP; + + vports_tc_node = __esw_qos_alloc_node(parent->esw, 0, SCHED_NODE_TYPE_VPORTS_TC_TSAR, + parent); + if (!vports_tc_node) { + NL_SET_ERR_MSG_MOD(extack, "E-Switch alloc node failed"); + esw_warn(dev, "Failed to alloc vports TC node (tc=%d)\n", tc); + return -ENOMEM; + } + + attr = MLX5_ADDR_OF(scheduling_context, tsar_ctx, element_attributes); + MLX5_SET(tsar_element, attr, tsar_type, TSAR_ELEMENT_TSAR_TYPE_DWRR); + MLX5_SET(tsar_element, attr, traffic_class, tc); + MLX5_SET(scheduling_context, tsar_ctx, parent_element_id, parent->ix); + MLX5_SET(scheduling_context, tsar_ctx, element_type, SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR); + + err = esw_qos_node_create_sched_element(vports_tc_node, tsar_ctx, extack); + if (err) + goto err_create_sched_element; + + vports_tc_node->tc = tc; + + return 0; + +err_create_sched_element: + __esw_qos_free_node(vports_tc_node); + return err; +} + +static void +esw_qos_tc_arbiter_get_bw_shares(struct mlx5_esw_sched_node *tc_arbiter_node, u32 *tc_bw) +{ + struct mlx5_esw_sched_node *vports_tc_node; + + list_for_each_entry(vports_tc_node, &tc_arbiter_node->children, entry) + tc_bw[vports_tc_node->tc] = vports_tc_node->bw_share; +} + +static void esw_qos_set_tc_arbiter_bw_shares(struct mlx5_esw_sched_node *tc_arbiter_node, + u32 *tc_bw, struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vports_tc_node; + + list_for_each_entry(vports_tc_node, &tc_arbiter_node->children, entry) { + u32 bw_share; + u8 tc; + + tc = vports_tc_node->tc; + bw_share = tc_bw[tc] ?: MLX5_MIN_BW_SHARE; + esw_qos_sched_elem_config(vports_tc_node, 0, bw_share, extack); + } +} + +static void esw_qos_destroy_vports_tc_nodes(struct mlx5_esw_sched_node *tc_arbiter_node, + struct netlink_ext_ack *extack) +{ + struct mlx5_esw_sched_node *vports_tc_node, *tmp; + + list_for_each_entry_safe(vports_tc_node, tmp, &tc_arbiter_node->children, entry) + esw_qos_destroy_node(vports_tc_node, extack); +} + +static int esw_qos_create_vports_tc_nodes(struct mlx5_esw_sched_node *tc_arbiter_node, + struct netlink_ext_ack *extack) +{ + struct mlx5_eswitch *esw = tc_arbiter_node->esw; + int err, i, num_tcs = esw_qos_num_tcs(esw->dev); + + for (i = 0; i < num_tcs; i++) { + err = esw_qos_create_vports_tc_node(tc_arbiter_node, i, extack); + if (err) + goto err_tc_node_create; + } + + return 0; + +err_tc_node_create: + esw_qos_destroy_vports_tc_nodes(tc_arbiter_node, NULL); + return err; +} + +static int esw_qos_create_tc_arbiter_sched_elem(struct mlx5_esw_sched_node *tc_arbiter_node, + struct netlink_ext_ack *extack) +{ + u32 tsar_ctx[MLX5_ST_SZ_DW(scheduling_context)] = {}; + u32 tsar_parent_ix; + void *attr; + + if (!mlx5_qos_tsar_type_supported(tc_arbiter_node->esw->dev, + TSAR_ELEMENT_TSAR_TYPE_TC_ARB, + SCHEDULING_HIERARCHY_E_SWITCH)) { + NL_SET_ERR_MSG_MOD(extack, + "E-Switch TC Arbiter scheduling element is not supported"); + return -EOPNOTSUPP; + } + + attr = MLX5_ADDR_OF(scheduling_context, tsar_ctx, element_attributes); + MLX5_SET(tsar_element, attr, tsar_type, TSAR_ELEMENT_TSAR_TYPE_TC_ARB); + tsar_parent_ix = tc_arbiter_node->parent ? tc_arbiter_node->parent->ix : + tc_arbiter_node->esw->qos.root_tsar_ix; + MLX5_SET(scheduling_context, tsar_ctx, parent_element_id, tsar_parent_ix); + MLX5_SET(scheduling_context, tsar_ctx, element_type, SCHEDULING_CONTEXT_ELEMENT_TYPE_TSAR); + MLX5_SET(scheduling_context, tsar_ctx, max_average_bw, tc_arbiter_node->max_rate); + MLX5_SET(scheduling_context, tsar_ctx, bw_share, tc_arbiter_node->bw_share); + + return esw_qos_node_create_sched_element(tc_arbiter_node, tsar_ctx, extack); +} + static struct mlx5_esw_sched_node * __esw_qos_create_vports_sched_node(struct mlx5_eswitch *esw, struct mlx5_esw_sched_node *parent, struct netlink_ext_ack *extack) @@ -551,6 +685,9 @@ static void __esw_qos_destroy_node(struct mlx5_esw_sched_node *node, struct netl { struct mlx5_eswitch *esw = node->esw; + if (node->type == SCHED_NODE_TYPE_TC_ARBITER_TSAR) + esw_qos_destroy_vports_tc_nodes(node, extack); + trace_mlx5_esw_node_qos_destroy(esw->dev, node, node->ix); esw_qos_destroy_node(node, extack); esw_qos_normalize_min_rate(esw, NULL, extack); @@ -640,13 +777,38 @@ static void esw_qos_put(struct mlx5_eswitch *esw) static void esw_qos_tc_arbiter_scheduling_teardown(struct mlx5_esw_sched_node *node, struct netlink_ext_ack *extack) -{} +{ + /* Clean up all Vports TC nodes within the TC arbiter node. */ + esw_qos_destroy_vports_tc_nodes(node, extack); + /* Destroy the scheduling element for the TC arbiter node itself. */ + esw_qos_node_destroy_sched_element(node, extack); +} static int esw_qos_tc_arbiter_scheduling_setup(struct mlx5_esw_sched_node *node, struct netlink_ext_ack *extack) { - NL_SET_ERR_MSG_MOD(extack, "TC arbiter elements are not supported."); - return -EOPNOTSUPP; + u32 curr_ix = node->ix; + int err; + + err = esw_qos_create_tc_arbiter_sched_elem(node, extack); + if (err) + return err; + /* Initialize the vports TC nodes within created TC arbiter TSAR. */ + err = esw_qos_create_vports_tc_nodes(node, extack); + if (err) + goto err_vports_tc_nodes; + + node->type = SCHED_NODE_TYPE_TC_ARBITER_TSAR; + + return 0; + +err_vports_tc_nodes: + /* If initialization fails, clean up the scheduling element + * for the TC arbiter node. + */ + esw_qos_node_destroy_sched_element(node, NULL); + node->ix = curr_ix; + return err; } static int esw_qos_create_vport_tc_sched_node(struct mlx5_vport *vport, @@ -977,6 +1139,7 @@ static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type t { struct mlx5_esw_sched_node *curr_parent = vport->qos.sched_node->parent; enum sched_node_type curr_type = vport->qos.sched_node->type; + u32 curr_tc_bw[IEEE_8021QAZ_MAX_TCS] = {0}; int err; esw_assert_qos_lock_held(vport->dev->priv.eswitch); @@ -988,11 +1151,19 @@ static int esw_qos_vport_update(struct mlx5_vport *vport, enum sched_node_type t if (err) return err; + if (curr_type == SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) + esw_qos_tc_arbiter_get_bw_shares(vport->qos.sched_node, curr_tc_bw); + esw_qos_vport_disable(vport, extack); err = esw_qos_vport_enable(vport, type, parent, extack); - if (err) + if (err) { esw_qos_vport_enable(vport, curr_type, curr_parent, NULL); + extack = NULL; + } + + if (curr_type == SCHED_NODE_TYPE_TC_ARBITER_TSAR && curr_type == type) + esw_qos_set_tc_arbiter_bw_shares(vport->qos.sched_node, curr_tc_bw, extack); return err; } @@ -1429,6 +1600,8 @@ int mlx5_esw_devlink_rate_leaf_tc_bw_set(struct devlink_rate *rate_leaf, void *p } else { err = esw_qos_vport_update(vport, SCHED_NODE_TYPE_TC_ARBITER_TSAR, NULL, extack); } + if (!err) + esw_qos_set_tc_arbiter_bw_shares(vport_node, tc_bw, extack); unlock: esw_qos_unlock(esw); return err; @@ -1455,6 +1628,8 @@ int mlx5_esw_devlink_rate_node_tc_bw_set(struct devlink_rate *rate_node, void *p } err = esw_qos_node_enable_tc_arbitration(node, extack); + if (!err) + esw_qos_set_tc_arbiter_bw_shares(node, tc_bw, extack); unlock: esw_qos_unlock(esw); return err; From patchwork Sun Feb 9 10:17:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966735 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2070.outbound.protection.outlook.com [40.107.94.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8805A6F06A; Sun, 9 Feb 2025 10:19:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096377; cv=fail; b=cbOZROvu0iEkI0k4dUdajU6ASNaaIrGceFlPRtFZdcp1e6C8carAzprEayeUOKC+mtei8DsuDMh9/Mtubtnij50hjaZdBWiDdhFLw7QmW47Ood3pPeJS93Snn4CSUVUKnrQEr3NmW/HtS9Jvs8ORU+2GTlyzo9Y7CdOlAy8OFXI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096377; c=relaxed/simple; bh=R+hLaWzESyGlfY+x8EfeS1rJI2l+H5ofHvTtKjCXEC8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DnBmsLy0t59j535fSwbfZ5rZ952ECYOjRN/vhxLrLXsn6NwPBbPwDRIUF83WlcJ92PIAKMLazjg4K8+q84Jvr2Kvu9E6BkUCsl3Lo/JkUdSU0TXpBn8aLPrD2tjeIdJ7+2y7jLvJoEK7iNip2Z6Zl1vXQApQfy3ZCQiiZGfD/Hk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=KGNNsbel; arc=fail smtp.client-ip=40.107.94.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="KGNNsbel" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NgP1GRJ39AdXztSvGC/pO6feHvBLIGopwqH3Rx0NYDvk3ib1Zu9uBdln17nR6E14zXZja0Zla9EZv5JAkRcLdphwoNZ1fSP+y9hFuuBjbLxacNIIMwuMzqhQ9Igm4T3xHQ/sEiY8rlWYip+M9Tv9AbQGeUm1XTpIFvNarWDQZ4pr4FmHnUt/cUZwX9syA0EUGw1gz4Iq2k7fphX9+1nve2rGkAbXzirLkIJjJp1k1SGyCCuMIYMUGKExtGMcvelQ2ofI55GwAu1MSY2xk/oZTcbvR54dT07jR4dly9VNBrAjhGFjThHcL9VUiBfpI7zqpMut+IkyYT1dlU7flAspfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wFllP0J8wk8dCwmsDF+YV2hWQCRMazFe5TQkA0j268s=; b=RCbvkDM8q2NHs7/ikivd7pk8BClLaY8pLpgshfdZHpHCa6j7lMcvgDjbL+ou3H1bw5xpSUEHf8+asNL/D7JQNK2KdMC0WyJIv2DIrp9J8+IAEL9bwuuRQU5FIzEvsTUzc6S2Tws34uflciLvqJAZpO+iqymI4etqADtTBZRhzqa0C7mQbEw7bkWohvhpOtMJsEtRHGvBbsEOmxsoM7NjzlJo48gy6pVfrN31P9by6dwOwbF4QsVPV7buiJnhEyklR2ImGw0wwBuxFzvQnjk5K2yoz5A7g4GYyVFo7VBdkZdIFbpm0/beq8LuB6cwMGrOzLz1x8RXRmqZEWkPRZ/Tuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wFllP0J8wk8dCwmsDF+YV2hWQCRMazFe5TQkA0j268s=; b=KGNNsbelD6kov612kmjPUW64GxPCuYn/EQKvmXrciA7lTeGvvrKcBU+/BSVCwaU0FA5agZlIM2jj14niePWYjWRYvC/Km+NRpKlg1NrRg6L9aYD7AO9w70Uy9cyVjkS5wPwsB0s6lQEqfDkFbXS8+KZ8kMCuFLXJsI+xvJEv2r6+G5yNVsCplcMyzW9ECQOyeq3rZDjH/yzM3b+CAZQKMGTYEq0EOKgmVYdfovCikooYRBkOBezixkoo1Mf84PfoU+Tco6taRaPgA7QL/sDiVmgnRGAmvfIFaGMgtKEB4OB25VlUvCMvw+kYancLicP3ENq/AqdZVKB2otwhn7Qjjg== Received: from SA1P222CA0173.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c3::16) by SA1PR12MB6947.namprd12.prod.outlook.com (2603:10b6:806:24e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.15; Sun, 9 Feb 2025 10:19:30 +0000 Received: from SN1PEPF000252A0.namprd05.prod.outlook.com (2603:10b6:806:3c3:cafe::ff) by SA1P222CA0173.outlook.office365.com (2603:10b6:806:3c3::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A0.mail.protection.outlook.com (10.167.242.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:30 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:20 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:20 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:15 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , William Tu Subject: [PATCH net-next 06/15] net/mlx5e: reduce the max log mpwrq sz for ECPF and reps Date: Sun, 9 Feb 2025 12:17:07 +0200 Message-ID: <20250209101716.112774-7-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A0:EE_|SA1PR12MB6947:EE_ X-MS-Office365-Filtering-Correlation-Id: c6dd96ba-51cd-4c3c-14d3-08dd48f33d9b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024|7416014; X-Microsoft-Antispam-Message-Info: P06M9WNMjhR0qKVbWFq2Tv1ikGy97M+H5co3fDM1kgbbxHln6lsaF2IRMhYIA8P+kQe5Du5azPQC7R8DhoLMxEfQg+bX3xmjFH03KesA3LuK9kJHFco1jvF19Cn3WN63j7mrY8CZ0rdVjkXnlen96f8DyieaJ2mXOrSUIqjCqGqpAyJiiVR+bKZnAk1Y/h57gxr7g6rkpo6vQanyS9+Nu1wDa8Gc7b0GoVluy5yZ0WEmhSE5DU5iBSa55R1we/8It3lBiQL0kMRnuFPnrxbTsDOkr78OMHT4DrSb1iTbIGqBLQUSiTvZ7EV/UrMUQFgv4HSgC793h6ztVrIdB2m/mq/OpqMSF1/rsRMYHGjM2A9db6QnLfH753xMM636N+uMATWAX75uKSrhez6BWomjxUjX6kJfOjIwjYeBDgm9DZaDNhu58hhfi0B6k5e/g47rxYkY4Ae9CB2GwrMvLQhKSD7PKvG8IqkrSTglgW6eKmQ/DBFlykJZoQe9wEfP99gobiXqz2bNKRvMLpAg46v6AP40nhlBva23eMxLR91GbuAAuJmu+jHx16Ijszn2Iny64fjgQnlnFxPpqh3egs5ybNYQ1i2FFCbCFjaMiJWJRb+dEX9z/W9FHjcEhxvicBIYa9wq9E62xzDzGy5hSGpbrA1PpnmEDcBvFM7iS7aeff2PbaEy97yfEOCWAQYXWdNAKAi+2QKw056YXTs8nrTN2rASnMG1Xl1YIhb/z93vjSLVprJ5YwRFcVvvmSGxWbuSu+Q5HxbQFKsqdII2ldn3VQIUyFMzmYyL6h3khWQfZ46Iv22C+z26CAmLd2eaQDOmgi7OWbkCs8RhrkSqCxOXLFPrMRQ1SpIH7DrxaQWdaWG7RkDPu9p9Pw30wI+P2sAzAEks+H7AH/JaGQAUp6W+zvD5P3F7O9GfQ7kFz++s0zxkIeygyFYHdckxIsWmc4gIglCPgCmZQCm/pXe9RJM6PL+dChYq/HBBJ6HH4WfMsq27HJtMIFEAt34rs9uRbEoPwEuj/r9Bu8wvwQLkp6S5+QRRlNJ8X4K8FRrTSGc1kL3b3qGpjsiPutZxwkc6hZwdKCMIHaFxyfsxqt8cNVUDOXQQKk2FwcL+M4R/s2mmC3xEqPQKTD28AYm+22dYn+rGMmdYMFBXT4DKXd9MrnXoNSj0TNFHMf8yW2HB4U92STPZcZ+KiRlshEeUU4EJRgLyfhJ1GGYCROBrXmXbPtxg3gqNpMYvuj7APtYz5xAllEc6guuiERe/sy/MwM5VXTEuKy7UJgXicRTe12F/uk0JZAFTYHHxR1Uw05creBzrd5lLgc/9Rje1tfNcX6KO/kC7PdXGTsVrqP1fCBo1mU5iIZwQiOb8UgYzv7OsXKMrqoaCwYN7e8Nynp5hLNL+q1Vd4mPvXfMH/S+TzldTT6JzD5rOY9pWLs43YTyPmRFfc4Vnlt7TJ3Dqc0TAce35xQibtjzKtgjbJqVDV7k/hAgiEKeK8MMRbXS1zruZJ9GIT8U= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:30.2898 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c6dd96ba-51cd-4c3c-14d3-08dd48f33d9b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A0.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6947 From: William Tu For the ECPF and representors, reduce the max MPWRQ size from 256KB (18) to 128KB (17). This prepares the later patch for saving representor memory. With Striding RQ, there is a minimum of 4 MPWQEs. So with 128KB of max MPWRQ size, the minimal memory is 4 * 128KB = 512KB. When creating page pool, consider 1500 mtu, the minimal page pool size will be 512KB/4KB = 128 pages = 256 rx ring entries (2 entries per page). Before this patch, setting RX ringsize (ethtool -G rx) to 256 causes driver to allocate page pool size more than it needs due to max MPWRQ is 256KB (18). Ex: 4 * 256KB = 1MB, 1MB/4KB = 256 pages, but actually 128 pages is good enough. Reducing the max MPWRQ to 128KB fixes the limitation. Signed-off-by: William Tu Signed-off-by: Tariq Toukan Reviewed-by: Michal Swiatkowski --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 -- .../net/ethernet/mellanox/mlx5/core/en/params.c | 15 +++++++++++---- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 979fc56205e1..534fdd27c8de 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -95,8 +95,6 @@ struct page_pool; #define MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev) \ MLX5_MPWRQ_LOG_STRIDE_SZ(mdev, order_base_2(MLX5E_RX_MAX_HEAD)) -#define MLX5_MPWRQ_MAX_LOG_WQE_SZ 18 - /* Keep in sync with mlx5e_mpwrq_log_wqe_sz. * These are theoretical maximums, which can be further restricted by * capabilities. These values are used for static resource allocations and diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c index 64b62ed17b07..e37d4c202bba 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c @@ -10,6 +10,9 @@ #include #include +#define MLX5_MPWRQ_MAX_LOG_WQE_SZ 18 +#define MLX5_REP_MPWRQ_MAX_LOG_WQE_SZ 17 + static u8 mlx5e_mpwrq_min_page_shift(struct mlx5_core_dev *mdev) { u8 min_page_shift = MLX5_CAP_GEN_2(mdev, log_min_mkey_entity_size); @@ -103,18 +106,22 @@ u8 mlx5e_mpwrq_log_wqe_sz(struct mlx5_core_dev *mdev, u8 page_shift, enum mlx5e_mpwrq_umr_mode umr_mode) { u8 umr_entry_size = mlx5e_mpwrq_umr_entry_size(umr_mode); - u8 max_pages_per_wqe, max_log_mpwqe_size; + u8 max_pages_per_wqe, max_log_wqe_size_calc; + u8 max_log_wqe_size_cap; u16 max_wqe_size; /* Keep in sync with MLX5_MPWRQ_MAX_PAGES_PER_WQE. */ max_wqe_size = mlx5e_get_max_sq_aligned_wqebbs(mdev) * MLX5_SEND_WQE_BB; max_pages_per_wqe = ALIGN_DOWN(max_wqe_size - sizeof(struct mlx5e_umr_wqe), MLX5_UMR_FLEX_ALIGNMENT) / umr_entry_size; - max_log_mpwqe_size = ilog2(max_pages_per_wqe) + page_shift; + max_log_wqe_size_calc = ilog2(max_pages_per_wqe) + page_shift; + + WARN_ON_ONCE(max_log_wqe_size_calc < MLX5E_ORDER2_MAX_PACKET_MTU); - WARN_ON_ONCE(max_log_mpwqe_size < MLX5E_ORDER2_MAX_PACKET_MTU); + max_log_wqe_size_cap = mlx5_core_is_ecpf(mdev) ? + MLX5_REP_MPWRQ_MAX_LOG_WQE_SZ : MLX5_MPWRQ_MAX_LOG_WQE_SZ; - return min_t(u8, max_log_mpwqe_size, MLX5_MPWRQ_MAX_LOG_WQE_SZ); + return min_t(u8, max_log_wqe_size_calc, max_log_wqe_size_cap); } u8 mlx5e_mpwrq_pages_per_wqe(struct mlx5_core_dev *mdev, u8 page_shift, From patchwork Sun Feb 9 10:17:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966736 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2047.outbound.protection.outlook.com [40.107.96.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26EB31B3934; Sun, 9 Feb 2025 10:19:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096378; cv=fail; b=PFB90g41VjDJM10s9sitmFpe1UHP59y+fehy6SBcKueHgYg1RETYWBBzv5QhbKYpf3N6y2ATe/Fwj6Bw7yNDPqOD1yX00jGWcjNGWa16Uhxdh+PbxtKcmwgzhGlrfs2IqegumThFcA2LMGEsAmd12B5Cmaad7JMMT/MJQvmBUe0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096378; c=relaxed/simple; bh=TgQWt33rAOJBDd8PlXsSc4Gh/qR2QvwOgXOLeInwBX4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JQIr0ZsBU7A2Pw8OdBVH13uPq9BePkT/STs6/ASrGInQe8kSMfHo5SROqBhpwwtpUeGvaePiBuwLzlx1l8lE05Q6Gxv0ODAXP5+U0qtJ8j8+2kysL9edJHnZqpwDWemmJF4TAv9Trfbz26fzbdBP0YPv5Vk6EVJNqn/guXA3Sj0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=geAA06AI; arc=fail smtp.client-ip=40.107.96.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="geAA06AI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=C/jBrRSfDvP+or1hqH/RHYgF0tyEWf6g/oXEDtO8GuzEdrR7V8KUx8W0Z9uueqMJ4Vqsf4/4D0REPoIECIEKJ5zITCjHR6niNY8ohEEQvngqm+R2HYlS/pbgeyZpwSIK0ay6mUkIwj0OL30GmhSqWaa4h7IxB9eGu1VmJVayaxn/oHSbvmi4QyEZBH5roeeUpkJjd7YzY8VOtJ88FP3k530MuTXzSLKR+h6/csljhK8jcBaymYRQi6Lp3T0lY+ePAhllYI6DtHzgtfxWzh/o6xHzK+cll8hhJQGXpD8YNLOnVcbD1qcubsnB8HjfKxupHM7I4YV7IpRsavZQ5p+vjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3GWS5JXCsTWSTgDnmfyt4PME8KrCpmeWLMMhx44O+NY=; b=v3+kyk2dFbJOP+W/FOQTenzNndqkAiXdbxnuVJoJjf93DYMlQUXnkGj4iqnV/6AjTA3XGgfYd2WHrfz0KxU677kPCAcATJPTq7eA3gSPk35lKVXYOXsIjOVI250BYGkEUB/mQAK6IBX7SpA1/I01iR1Sdn2EeQ0tDEl/T1QgNhLrl52DSQqFu04LT2sS9Im9p5mX5XoiTb8DtiUP3rjRaAzuojytHKpj/dHAIA1O4k6NQjuQQnLWWG28uGaWDR52W2xCWdJRrN0PvzbtsfBdBYpwbFzmEZpr/Ukk336Yh/99I7hTP9QugsuAKssTsTNpEDEK3e41NzZP3lz+E2ByLQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3GWS5JXCsTWSTgDnmfyt4PME8KrCpmeWLMMhx44O+NY=; b=geAA06AIpcjIGs3KOuaMUMpwe5UhFqQ1lC0gxSlSnVJ1AdKLgQyTughz8YscuycK3G21XXNyinzy7Wdg2DYfJZRPl3wwwo8mLwJEjcgbf149wC0ynaB56FbXgSu9DIGqBQMGS7r9Ua3JoHX+3ooWR0VNobDqvNbdoudWZ3jFXhAIASb68XuA6gJTIyayGPdRNv0rU8+mDfEYUzI19jvBOqfN3vERRM6Ql+x+iQd701ONyKYmWmCrmLt6ECSGY/5v+6KLYLDhWQPJVcA48KFuD15zxGvXuHti24DG+mR+DKze0sp7S8Yh3sSjQQF+8ZZ6YZ09bZtfGFfsyeVEnmMsgg== Received: from SA1P222CA0167.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c3::15) by DM6PR12MB4251.namprd12.prod.outlook.com (2603:10b6:5:21e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.11; Sun, 9 Feb 2025 10:19:34 +0000 Received: from SN1PEPF000252A0.namprd05.prod.outlook.com (2603:10b6:806:3c3:cafe::fd) by SA1P222CA0167.outlook.office365.com (2603:10b6:806:3c3::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A0.mail.protection.outlook.com (10.167.242.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:33 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:26 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:26 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:20 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , William Tu , Bodong Wang Subject: [PATCH net-next 07/15] net/mlx5e: reduce rep rxq depth to 256 for ECPF Date: Sun, 9 Feb 2025 12:17:08 +0200 Message-ID: <20250209101716.112774-8-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A0:EE_|DM6PR12MB4251:EE_ X-MS-Office365-Filtering-Correlation-Id: 652bce22-6188-4e1a-250a-08dd48f33f92 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: gjf8PseNISQ7QzEOXksMWrhZ8tORiueyRyp9nL+cx8RINL0WBaF8Udplmx/+++wVv12uInSpTxPq16FltEftWZfuZJBcQcHvEgNIoAdWJ286gDZ0IlhMQVYp4Glt+NAJE3AVhJlIYD8sXYIFIPAR6evQ2xm2H8jcxB6g/CegXoxR6ImNGeQCKcy1dpFzXCjO/2uLz3vk4Tn/oQEIGBoX+KsN9IukfkRkQynBWVINi3PRbfZRSIntsJ0HNGKoxpJBNWv4pTScFSjxXGleUHbmdB1AiXqB1HakBOZxBYTqqwKSzZKe80IlukyjKm6pOmAnsZxdsHQ3u6ykgzw6DMsm1gD4kmlS7/4lzyAjPuO7yDmsoU07WIBXVabYR5fYI9b/M8N732nNa2QHbZL4oEYLkXc2UiiFUGQ3QOMZ1NkjsWIlQs8xuaLHEeFUA0PjTTpaMXzK3RnkWZ3EoFoWjVfWpt8JmilnSq1/5kSRQ5CJkgOw+8biLiU2HB5l+hUXHXkL4Rai3I8616Jdu7dZK7aA3HSqI+ufSJwdQkfpsG+3GrPqd/r/BAct/3BonUKCI+jvg55z8PjzQ6RjQTA9sHC58Tl+Me9zALCIV5YxmQGQ+u88JfXUNBH2mWMANhNk7JfzK69UBiBujFvuEWdUxgfS9VCwFotWe/gdh1aL4uZIlLI1yxo/M50YXw0LEGk6VggNvrkrpDWFZb85+Ojcdz1oGgC1c2vibrS5RhMYr0CvKHbPQ86RAleR8H16I5A7zA/hMtHryqxnqm8ltumtEqblewXobn3emGaXAnRfSKWEozB2HOJQFnV8q/7abCyKkzYLWp/amKgS35cPj6Vjv8fCHLTyeAbPMs6b/Qj8MhGpeDxGlXY2VJamAI4dhxyfa1XhDRj/Gq/bMSinbLqn/MbdQvlbR5eI0SrwbVA6KazR2/bnPngfQYAYPbSuZFyX/Rw9Mywl4I+rlwnaWDgjGfvw2iKljufw6hZnF6+tAGOPGcokKdehYksM0ZOIbS1bnaPW/vXFULeupUqTVWn+QMV8ZgmZxsRWk9mNrZgxjkZJ1WxjxiFq5ApDWKsTE1YnbRLJZb1ZYlGUYQP5UM4rRxIDUBEiSijt9SUq7LleQqiluVnSU87muvPigGpuswZse2kq9mDz3QPmOHfVfUkdFIKOBIYpumz7o7r7fgWeW4IL7OW5oIN/r6lLDGfm0DFhKu5bUfzv22OW03l04om0afOfkwFNBe2bmtzO7M2gryi3SWf3U5IqXjIJq/utkHtRLekYbvF5rp9eMO7m3miOnV2y/fYXNAwLDuQQ9fIOz61y5iRKPOdlynB1YfD0FM9Xu3VCMa5FoLVrxkyxLU5ZlMC9gQqc63GvG3vq8YiUXMQ9aY6IHvQHXXgYzEShLmgd72BDRze56OAN7ZIjYEkBj2AP23vHR6+rzDP4Zv/wf8rtS8ryBCKbL3+Kai5GO8jb3Tf1CzMkNdYazvqqTA29Y4NMVSYelk63fHyhOfx2ZgC65Ew= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:33.5867 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 652bce22-6188-4e1a-250a-08dd48f33f92 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A0.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4251 From: William Tu By experiments, a single queue representor netdev consumes kernel memory around 2.8MB, and 1.8MB out of the 2.8MB is due to page pool for the RXQ. Scaling to a thousand representors consumes 2.8GB, which becomes a memory pressure issue for embedded devices such as BlueField-2 16GB / BlueField-3 32GB memory. Since representor netdevs mostly handles miss traffic, and ideally, most of the traffic will be offloaded, reduce the default non-uplink rep netdev's RXQ default depth from 1024 to 256 if mdev is ecpf eswitch manager. This saves around 1MB of memory per regular RQ, (1024 - 256) * 2KB, allocated from page pool. With rxq depth of 256, the netlink page pool tool reports $./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --dump page-pool-get {'id': 277, 'ifindex': 9, 'inflight': 128, 'inflight-mem': 786432, 'napi-id': 775}] This is due to mtu 1500 + headroom consumes half pages, so 256 rxq entries consumes around 128 pages (thus create a page pool with size 128), shown above at inflight. Note that each netdev has multiple types of RQs, including Regular RQ, XSK, PTP, Drop, Trap RQ. Since non-uplink representor only supports regular rq, this patch only changes the regular RQ's default depth. Signed-off-by: William Tu Reviewed-by: Bodong Wang Reviewed-by: Saeed Mahameed Signed-off-by: Tariq Toukan Reviewed-by: Michal Swiatkowski --- drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index fdff9fd8a89e..da399adc8854 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -65,6 +65,7 @@ #define MLX5E_REP_PARAMS_DEF_LOG_SQ_SIZE \ max(0x7, MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE) #define MLX5E_REP_PARAMS_DEF_NUM_CHANNELS 1 +#define MLX5E_REP_PARAMS_DEF_LOG_RQ_SIZE 0x8 static const char mlx5e_rep_driver_name[] = "mlx5e_rep"; @@ -855,6 +856,8 @@ static void mlx5e_build_rep_params(struct net_device *netdev) /* RQ */ mlx5e_build_rq_params(mdev, params); + if (!mlx5e_is_uplink_rep(priv) && mlx5_core_is_ecpf(mdev)) + params->log_rq_mtu_frames = MLX5E_REP_PARAMS_DEF_LOG_RQ_SIZE; /* If netdev is already registered (e.g. move from nic profile to uplink, * RTNL lock must be held before triggering netdev notifiers. From patchwork Sun Feb 9 10:17:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966737 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2045.outbound.protection.outlook.com [40.107.100.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 754F11B87C1; Sun, 9 Feb 2025 10:19:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.45 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096381; cv=fail; b=LdaAxcAXI5Sz8ye0YMTcexcNJbvQ3Y0PtkL08Wtu1b1wUSa3R6EGUH0j4m1n8djakn/3TZCQr1ol6wAt52cktbQYuz3gOYxbN+ql6lm603Tn3BDnJIcvwxQyvHDgcfzuxdPqmskwOSObot2S7VdIcFJWLi4MpkfMC3yZLVCnjT0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096381; c=relaxed/simple; bh=8kO8XKdGEPVGNLSGBDtaJ1JX3DyeuEY8RzmTVmaeNe4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qbwPPEdkmYH9Do+YNhJG4zIj3xsY8ktYRgt8N+sjgXUSIRu+wh9Czvw1LdUztD/xDb8jaBWiWZ9Pog6RQW4Fnpa07eQxlCNCvbd2m3FKdZuPZ/HDZkQZ3Bmbz/efVboP3uD6CHa53BjcQ6znQOg3Z7HsULG6GUM/Tj1GIAd6td8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=IZkKoX1/; arc=fail smtp.client-ip=40.107.100.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="IZkKoX1/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wEOu5C9cXFcWLh1j8AAGKG/zBnIdpAy1RFzdMYg03rX83EfSGp1BOUfq+Ij88P4zjsuxwwx+XFm6pN6ES2HJbdy7bhqqcnfkH0+xff5/Q/cd9MAvXCfs6ZUTzGOkf1Z1fMOEP+lSZGog50cGWN4851jyV78GuwSVUwlDuUJ+hVVQpAIBMbY40SXP2C3uFQe/r3RGSiNHN+NErk7AqVErER/EaPkAatbOiFRp8SMTd1wAt7cDFT/GJ25q4DDzVexDE/oI48YgSlzcxa5k31YFrCo8U8B9GkiPoy0uLig+BDrp+QrwLJbhe5lm4WMmk6dBnj7O9z0c9Y41Sj4ztdGjpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TqJa+brBEhU/iw3G1XWFwtfeZAGcMWfr9DBmC7PFrgE=; b=rPnV9HtGkKPp07ZX2ugttLJn5Mw2DUFdbyBywWo8frpchMjPpmnMthR9sGvPvhTcRyXN0TB+lDFmjJjueHcpFTwI1Pg0EFVRaynfv24CudeMS3/HQxLqVEjCVubgN2bHRsYA8yr7mtmTQa9ZwplJ8j50nfCbpWjFs4xsbjCJiWkpGQFK/AEgdOXQ7hLGxsQYiLzYcrN798r7IHHHR4qfd/EzpWHQhLmVFIpIOxSPwdN92uoalcVDslBJNodnA9yc7RrpLB85fRjck2sGtEOKT7O+KkQPw+MGzexoXEA94mGV8BSmHgt2FkSPsw7h2Dt1K11UVK8SL3cklPr/6T3HLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TqJa+brBEhU/iw3G1XWFwtfeZAGcMWfr9DBmC7PFrgE=; b=IZkKoX1/9126JXITQzqRYCoYtdrfyHNJWkshPBDEQfZ/7Tua/LCLouvNDBVfIm6yr0E4miOUPqlnwxsEIfFqArQqdU+ZXIYKnDc0Z+4WyugX7bGVHmsa6Rd2vAs8QXF2+3yD2ors35XQeI4OJnBnV+NXaYMJ5rhV+aYpeTsDfCUj/4M4pydEz/0YVDY64TS1uXDcEj9SNA6USHVxga4pSyH/UoXiC7cpOvJaErUxHVig9GQu/b9fUTaBWZfExLyVl21B+ooh2u0T2WZECETGv+1rQ8cX/U5SyBqnNb4EbJe1Fi8ThewKyOm3Bx8rxC3Y1/oPHf4QV6P5GO+16eVwzA== Received: from BYAPR07CA0011.namprd07.prod.outlook.com (2603:10b6:a02:bc::24) by DM4PR12MB6184.namprd12.prod.outlook.com (2603:10b6:8:a6::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.11; Sun, 9 Feb 2025 10:19:33 +0000 Received: from SJ1PEPF00002323.namprd03.prod.outlook.com (2603:10b6:a02:bc:cafe::92) by BYAPR07CA0011.outlook.office365.com (2603:10b6:a02:bc::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.28 via Frontend Transport; Sun, 9 Feb 2025 10:19:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002323.mail.protection.outlook.com (10.167.242.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:32 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:32 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:32 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:26 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , William Tu , Daniel Jurgens Subject: [PATCH net-next 08/15] net/mlx5e: set the tx_queue_len for pfifo_fast Date: Sun, 9 Feb 2025 12:17:09 +0200 Message-ID: <20250209101716.112774-9-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002323:EE_|DM4PR12MB6184:EE_ X-MS-Office365-Filtering-Correlation-Id: d8fafaef-bb76-43e3-8f55-08dd48f33f0a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: 1PqG/FzE6j9X5ynWrw1cJmSVtTHLhP/FYhZ8I32OvfkV93c+sZK9wY1jx3njRDNAhy03DNtgDP8F47eAE7h8Mx1yqQWKAq1Al8Xs+XpjGxJYpQUyOUke/+TayXNYej5SnOAI+H5YluNpshsEJAC8ZGxpJeRxxYYIxBN/2Q2usfBJMsXBwglIkUdHDQ0BqfCOrauDyvHRfTdbpgmdp6oNIuWcPMqytqtyLbN4EZvDqRsZGSURwJ0S730WKPlcVXcHX7+0ZM9Ou7xHsKrbF48WPstyPtqMkA6rAifPdxTl1dG/iTUgqhscsrcgoo955oKp+JeznRpMQs7Ua+wq5/BPvzG2QRBKGSpbcq6A305IspV87q+pVe44NGELKcC/MUS05ZfDnSfiLojEb+uup1cWDeXzWZn/eVoOYaRYYm5pKl3rn5HZ7QMdheZIqUt3eYp2PQlcEKdRDvx4sX7YJnTRgxWqBafkB8djV5vaDYSfzYtPeSPrtiyCYKn9mA32Z9GXPm/AqmOGDojsbDE6Ns+UoAi79E73hZ1RpEaJvKT5JpWBG3JOW52O1G41h4diSuGdWiD4kFfcEi9h78nsZgi7wSa2aSUjOl78YvBGRip+xNA5lquyOeCr7XScGbO4iksbygPPrMR4aK81VuA8fQIURFdexovomnW+GFSHOvxPEuQL50UZaIugHJwGFa8leax4Csg3jVPf1n5xqfHHqC6n3AXwBh8ZT3Bv9O5ivsN8l1DQ/4ZYfBHLNAy34hSsRP3llPdcx7FgE/OVvBlCUrxwoVsvllwFbtM+D8HaZ6zvbtZ2SOPvrQ3V20YcSzlkChWBhCdRSnKbn8NmTaO0W14wD/wiv0tm7PBOUs7i0MKcZijt7ttTnzNfDReO3m+vWlYaRLGynygZUZsu+dx+DFAzpHGQXhqRQwtZfbr+PZe3WmAF50bOZZ/e4zPnqWcdJwKxMkhJ7wjkuhL2xmP0SCcP8JXLPGqZ81MCNuvpqHps2At6xKXwGFDxjeBqJrt16cT5HZZNymBKKHXQ6ZiDdj/CmlSGfVcGm4XxUPyuxG5MQMzDcCOSZV+0hDjZZdDDaJfcyq1RYzwm9bmpYqlsOA+J8GSuN6AZVEZIKxraS/9K8ZowU9hLuL2PSAo9+3hdT5otrAHIH2hO26j9p/3qy1NxlUb+sJh0vAn4iU+iZTR/zUutjsIVckyYFz2hJv4Kh3YhjcO/EitSl62It6KIJtBJ/ys1A7mdumHd+KHhpT5dybP9vEMtci53lMrEApsAq8JGKcE1I3M0wkqxodKy8rN+kA375o73pqXCTxqKWokBCALZO4Ik0OhAJX1ouU8oICFwd4cl3m5huItwNNv44Zb6Lo+M6PW4qPOM8v9vXWbZ+ZACB4mticeAyAw+rUA3FpDUMpqYvWgW+N7g1JVy7gMyQUkUiTXRXBtK0uYJJreRrmxTCDynHw1cDLPh8WNN4CjfPe5Yat4gAZpvNtSurGV5hoDaSPg3nUFSp0tC+nRvhqI= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:32.8231 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d8fafaef-bb76-43e3-8f55-08dd48f33f0a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002323.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6184 From: William Tu By default, the mq netdev creates a pfifo_fast qdisc. On a system with 16 core, the pfifo_fast with 3 bands consumes 16 * 3 * 8 (size of pointer) * 1024 (default tx queue len) = 393KB. The patch sets the tx qlen to representor default value, 128 (1< Reviewed-by: Daniel Jurgens Signed-off-by: Tariq Toukan Reviewed-by: Michal Swiatkowski --- drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index da399adc8854..07f38f472a27 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -889,6 +889,8 @@ static void mlx5e_build_rep_netdev(struct net_device *netdev, netdev->ethtool_ops = &mlx5e_rep_ethtool_ops; netdev->watchdog_timeo = 15 * HZ; + if (mlx5_core_is_ecpf(mdev)) + netdev->tx_queue_len = 1 << MLX5E_REP_PARAMS_DEF_LOG_SQ_SIZE; #if IS_ENABLED(CONFIG_MLX5_CLS_ACT) netdev->hw_features |= NETIF_F_HW_TC; From patchwork Sun Feb 9 10:17:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966738 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2082.outbound.protection.outlook.com [40.107.237.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7015318E764; Sun, 9 Feb 2025 10:19:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.82 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096387; cv=fail; b=Isck8X7irMpnl68AjJYNH6nCJ2K79K7KJgD2pYMirdsnBG2Y2TpH/L2g83WJ96rkR3KZ8FXN91402z6PBdVcM4GjN/65XnIGu69Ug00b0BHyo1qoJQnSQ8EPMIdtZtry5yUcHwGXSeUQxKMlmFyTOHT5hE9KOSQbCAHOt3LwNgs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096387; c=relaxed/simple; bh=4m4ciW03NwtYOo9BVW9sw4arpkXPJ53uCHW9VTKF5YA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DHcUNfRUZN1mhY/QR0WY/ZUH24SXWQIKi5NJ3AAvAvnQnwZLjo8HZasW4W4+aQ6wZnSWWvxnZnJOM/p5s3pzLA0bEHnErheOlsPkpP+dXwAmGXzSpxWUC4HwBrPXpHNyti+va9mxjrcZB73u6WSj7M8K03hNeM3BtIDW8OmHTBk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=C1jYZJ3L; arc=fail smtp.client-ip=40.107.237.82 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="C1jYZJ3L" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=y5dNM7nWs52+U3JEKR/z6qGIi/rANJM3KhcZ1hsQYRwQY9G1jZPpYkR5gskWXSiUmyfaUz8eOGYgOIPoRpWH3SAlkbRC9jRf/muqH/kFYq5vfv9gaGM2kt8BcYLsU8V38JzLBE3HGl0G9SV05hySiNF4bg/HEmCIxE/aoTCRNjGdkh2YaUsX9yXmcuHhRZOvJeD0BkAfFpy5DN9vx+8Aj9QkWFNOZRI4CpKKEstIUic+pr2gfpjJtSVYuD++PEZwyTu3yl5BewCMQbQoQT0wl0UlpIJEZD9yoThirUMxk41jziWsOHHTM2eGFEsHB4aY5NCOi31KqeUI/lyaRVl1Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jmc470bIMvOgWdUT9yAuFL+wM6HfSBZK/SEGFniHyMg=; b=IfK5yV+spqvGykhIoaaL7Ft3wcj2bgUjoFewHwbx+Nq1J7/C7V2L7xkWUJBsuEn5oBDzYXIl2X4svPZqN/JerE63X6i/nPGd+SoWOA8H1OUlOfi6OWQb9HFs+M9EyvmkCak08eipzWuhHeXNcOotx7L6U7FMmWrnCoAMNGRZQ57usKA6zp6H8HTAv0+/mcJxzZlyrMpQjNN1cTENyYBrDKFtt38/PPcqSew3I1cAQXi6AFtBysqkwTMNMoEUeUXLkjPUQHvH9ud+uQQ6mW0x3VFIFv0NAd5xNObEHF8mlcrwLwOdo3ai0eNJyBkEQY/vKB2ug7qRTJextYXjswEL8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jmc470bIMvOgWdUT9yAuFL+wM6HfSBZK/SEGFniHyMg=; b=C1jYZJ3LxMNMzbkHOjULmtkk2Xiw4E/GJ8bM0JLyCJsQGnadWNLEcj9jaOpW4qCbq6JwmVHyELU5uWiGfvPOG5Oza6l/aKwzc20v8bWJpHRNbHvutODZETkdi5QyKQ37VIQn8VYDeBu1kFEsgoREmbYEyf9U6NFmGseGy1wY55hXzj/V8hb0DAH94mEt94nlzjJZwQ5vO+lGU2P2l9c+3LCBhOBB+SUCoEoPFIjIqe6WfyemlY6Cu88qBfjG+bHtWUyvQvpy0kv977dT3J341lVGvTZ4jysRbRMmDGxZHPDc3qxBvSQ92Y4CDyX49kdwlAzb4ckfXM2Q9JT0cgfdRQ== Received: from SA1P222CA0152.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c3::20) by PH7PR12MB6908.namprd12.prod.outlook.com (2603:10b6:510:1ba::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.15; Sun, 9 Feb 2025 10:19:41 +0000 Received: from SN1PEPF000252A0.namprd05.prod.outlook.com (2603:10b6:806:3c3:cafe::bc) by SA1P222CA0152.outlook.office365.com (2603:10b6:806:3c3::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A0.mail.protection.outlook.com (10.167.242.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:40 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:38 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:37 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:32 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Akiva Goldberger Subject: [PATCH net-next 09/15] net/mlx5: Rename and move mlx5_esw_query_vport_vhca_id Date: Sun, 9 Feb 2025 12:17:10 +0200 Message-ID: <20250209101716.112774-10-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A0:EE_|PH7PR12MB6908:EE_ X-MS-Office365-Filtering-Correlation-Id: ee223b19-e1c2-4cfb-7b20-08dd48f343f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: DmfywPqUAk8UEmDVdXGngoJ8DD3H98gZ1QlUeslnnZeAllj7ANDoVvqLogxTTZcde4tHbHUOq49eXXnI4e+JY591EIIgr9wOoNwtfYaWTSVBtZjma4FC4V3P6CW99wPrWjHvJLdRkNAUGiZDtbEQMt70w9jdoefcZOUQvOazI13I9z2uFYQDaWePjzpjYHzQoeJvKQcp7+g9vdqtJBYnDoizpbvC4BPUdrsnB4+GbEUwKNnWWjwpwc/3+nBJuI8N+xG6wQSm0M+iyxzKi3rgi3zBgPSQnpxs0g/inwV9m0/B9Rx5RI+UK8RbOmOTCpbQYsoHYFVfZZgiQaNZ56iIDeA3OZV6uHTfYgY+BmjcMQK7V1hG+lnYYp97CGKrCEIqmg25lIkv/ibG1mDMPGPOnbhGWpWKvHhVJUP44N+moLgRceX0LgXK5Xc84O305Aio54K8u3L+6gLe5Z1lldwOIfKXBd5IybghAooBzXyYafIR9EL+wwxR2ubOSqxPZ3Nzn75z3OkjowSQReyNXx4Nx8+hlcb3KjettVblPBHYZaw1ZHPttqyhs4O8kTOw+ttrelRUX8CtiBs+iRdyWV1xFMu/x124tanAdSWbfODtvTLQsnEilLpNWJ98AA4tFz9yYSIxV5FvcM4TX8ex9I6i2zwpsUY0Lff6Xu10+PWTMb7bS5ayk3ISlkdBhQrhRXK9t0icd+lJaJaB/Yga64HbqPe1w5OuIBdgrEikp+A6u0a6AgdzyEirseGihwT3JDkpKN3WGsxh7Jsh4/inzf4hswuCpDb6vgsLuwg/SoL8bX0LIjoyHPs0PlteGPk+Kxwouif2VaibK6cXHwseSsyI2bL9Exj53WL/JpTU1hzDMqnTeUd642anhzCjNTSIwC0y/8dV/HdIZ+2RRhc2sAB9uC8tjoydUjuHfvGhRUykGSNThvfRXeHqaLbfFELJTrgRXIbOtIxE3jiEkDmiJsnnC5I8IvJiHngdZIii1TKeuriTarpEvI4NrH7EfYDbLuQt4h7W/5FmZQL63AGqgeRGJS2HW5TF9aSpJ5MI6Wfd1lKMRNLm7uZxEoNv41w0c+UHBPEdaKLoke5ywv3h/V3KGcRTi8wEBJnslswfzdY0blegIteOKneKH6VB85d+DWHHlnSqJhXmSxa0tUKnuc7Fx8aruIwfDXl2N5BahR+S3mD5fuVsvCAdmHYqNPjzg123ug8OS9qFgx9BZqNc4Z9gPpMUyQI/UcpKO+WKv9N1fz9Af7sTNpQWRhH/qpQoIfhaVoUi8LXlov4rSd+BhlTX6lEdkAmK/y8EsxTjy1xE7U1ODLqElQBbtp3DJkPVnpTqY2nX8oj1amkkU/HkvgQurhK5fRbOYUnfD2dB6sBl+IVMg7AKGT8impE3fIaWuARFk1YOfG1foIJbgcUmja0du9uT7Mg3z1SoSGibK0PmPg/VB8T2fD7NASH0m887NKbqRQWBotIfIlPC9zWnkoIbGzByQbNjTjacjFYn3bbXVUI= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:40.9617 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ee223b19-e1c2-4cfb-7b20-08dd48f343f7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A0.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6908 From: Akiva Goldberger Rename mlx5_esw_query_vport_vhca_id to mlx5_vport_get_vhca_id and move it to vport file. Also, add function declaration to mlx5_core header file. This better represents the function's usage and allows for it to be called from other parts of the mlx5_core driver. Signed-off-by: Akiva Goldberger Signed-off-by: Tariq Toukan --- .../mellanox/mlx5/core/eswitch_offloads.c | 29 ++----------------- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 ++ .../net/ethernet/mellanox/mlx5/core/vport.c | 25 ++++++++++++++++ 3 files changed, 29 insertions(+), 27 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index 20cc01ceee8a..0fa0333106a2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -4157,37 +4157,12 @@ u32 mlx5_eswitch_get_vport_metadata_for_match(struct mlx5_eswitch *esw, } EXPORT_SYMBOL(mlx5_eswitch_get_vport_metadata_for_match); -static int mlx5_esw_query_vport_vhca_id(struct mlx5_eswitch *esw, u16 vport_num, u16 *vhca_id) -{ - int query_out_sz = MLX5_ST_SZ_BYTES(query_hca_cap_out); - void *query_ctx; - void *hca_caps; - int err; - - *vhca_id = 0; - - query_ctx = kzalloc(query_out_sz, GFP_KERNEL); - if (!query_ctx) - return -ENOMEM; - - err = mlx5_vport_get_other_func_general_cap(esw->dev, vport_num, query_ctx); - if (err) - goto out_free; - - hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability); - *vhca_id = MLX5_GET(cmd_hca_cap, hca_caps, vhca_id); - -out_free: - kfree(query_ctx); - return err; -} - int mlx5_esw_vport_vhca_id_set(struct mlx5_eswitch *esw, u16 vport_num) { u16 *old_entry, *vhca_map_entry, vhca_id; int err; - err = mlx5_esw_query_vport_vhca_id(esw, vport_num, &vhca_id); + err = mlx5_vport_get_vhca_id(esw->dev, vport_num, &vhca_id); if (err) { esw_warn(esw->dev, "Getting vhca_id for vport failed (vport=%u,err=%d)\n", vport_num, err); @@ -4213,7 +4188,7 @@ void mlx5_esw_vport_vhca_id_clear(struct mlx5_eswitch *esw, u16 vport_num) u16 *vhca_map_entry, vhca_id; int err; - err = mlx5_esw_query_vport_vhca_id(esw, vport_num, &vhca_id); + err = mlx5_vport_get_vhca_id(esw->dev, vport_num, &vhca_id); if (err) esw_warn(esw->dev, "Getting vhca_id for vport failed (vport=%hu,err=%d)\n", vport_num, err); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index 99de67c3aa74..6fef1005c469 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -346,6 +346,8 @@ int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap #define mlx5_vport_get_other_func_general_cap(dev, vport, out) \ mlx5_vport_get_other_func_cap(dev, vport, out, MLX5_CAP_GENERAL) +int mlx5_vport_get_vhca_id(struct mlx5_core_dev *dev, u16 vport, u16 *vhca_id); + static inline u32 mlx5_sriov_get_vf_total_msix(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c index 0d5f750faa45..d10d4c396040 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c @@ -1199,6 +1199,31 @@ int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 vport, void *ou } EXPORT_SYMBOL_GPL(mlx5_vport_get_other_func_cap); +int mlx5_vport_get_vhca_id(struct mlx5_core_dev *dev, u16 vport, u16 *vhca_id) +{ + int query_out_sz = MLX5_ST_SZ_BYTES(query_hca_cap_out); + void *query_ctx; + void *hca_caps; + int err; + + *vhca_id = 0; + + query_ctx = kzalloc(query_out_sz, GFP_KERNEL); + if (!query_ctx) + return -ENOMEM; + + err = mlx5_vport_get_other_func_general_cap(dev, vport, query_ctx); + if (err) + goto out_free; + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability); + *vhca_id = MLX5_GET(cmd_hca_cap, hca_caps, vhca_id); + +out_free: + kfree(query_ctx); + return err; +} + int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap, u16 vport, u16 opmod) { From patchwork Sun Feb 9 10:17:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966739 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2068.outbound.protection.outlook.com [40.107.100.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E3AC1BEF82; Sun, 9 Feb 2025 10:19:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096391; cv=fail; b=gXbPNOrIWB9OFa9LUZQQe8GmtdgOiJaL8TSy0pGi6qWGl5EOaRP2/IgM+Z7Cm0yFP7BQA4jVhPUxQHcpfvhDZyPUZtMzT+9pg6Jb9YZ1YFZLr0xlASv+DsXqhJ3m+XuvgvABJU7pCwwqvFJHuTyJ/0lMXxQtIUMnEPumU1seir4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096391; c=relaxed/simple; bh=5Y4tF82OaX68Bo+FeBuCWaTSOxNkeIEUA3KLdzZjoos=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tgScOz9W+nOYdrZYUtARIlhArlHZ4DKWOlCc2uwN+pM4ba/T62KjEQt1mlxd/+QlIXLe6QGaLjQB5PhYWlAp68BzbCpdBxltf8854XW8kw2+ZmE8wn7bkBaD5Y0ChLUDRCblQxSc+H+uWtSi4Cnm7UnSmjevIlxOE9xxBnssT9g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=oEmXXu8/; arc=fail smtp.client-ip=40.107.100.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="oEmXXu8/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xQEgXIhhn0NxnMoxZXe7hmeHAN4j2dIhrjsliKs6w4FsjXxCfupCxG7JYOJpEIip0xDhvOD5J8B1+o5uFjZw/VYUHpYrhzPHELKWraaYRcaHSdjmcgwHlRYrRagnjo7AhD5jmQRpjFHMuvb4r0HgLiaYi8WfE2GNIG7bvBt2IguiCGof/LLe2sZL8KXSz0UdzzlVpeqgO6rjXYjHCuOvNxJe+uZu8rHWbO3KNhoAJ2L6TgC4f70CMQeOkS4EuyEkRB03A+spJdDkXZtBeEGpwjKyQ0ZmRo6CIqCakuv18MKITkkdxnt7SgoV+Xi5OFEVfgUJ0Bu7wtFCdcS/q9mruA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xkAxzzWjsle1KBQZsvaGlIVAQlIYl6CihvtSpklbu0c=; b=Pv+jjqtpzD5bMmSRawYcu8WnO+xpVxf7gwq+hrdg1zH5XJOw3uhe2Lnm81GW++oU41i2ibOBVNNo2KHF6bJAre5Xf0lrALDdrxuIkMYTmC8nyK582i3+72MtRW5AJKGLzGhjvlT37aTJoeEuFJmwYlATLJKIndxjxoRM9F675iaqeWQmAJmbNKqKn++QZaMdqsrUNUZ+iQW6r95dJVDb/q6EehapnK9Qa5d8jz8/3IIqoxIWqBsTU6ZRIk8BOdYo0OLxxTl+EpmV5MkFPckhg3ZJbGmqdUAoI9YEcThHVzP1s9b9+r7lyI86mEpDOJ9gnzerkRLXVJ7pjr+v5AwBkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xkAxzzWjsle1KBQZsvaGlIVAQlIYl6CihvtSpklbu0c=; b=oEmXXu8/SmypWmNeWC0QkQO6N2ht76ccC3EfcMfI4wUmd/ZiUpT8PVH+QMbDAhdOIDWw/rSkjTM9hIm3bP7gLeu7Jcy4O119TNNy8S4DmWuxzNTwFj4ixe0G5XAj8+3V1VRGsY4nZOTc8gpRS2wQEtlvRXvy2crTR10/jYUsToqWMBQywSAoU5O44XwSHYZdEbm85B5xHnbO9MyZPaW9Qt9498yOOXzbU0e48Ba1crrHIrhU/HOZ2Uc6didPBA/cow5CxERoHyrCRfuBOawqP6GSVmc0skldMXuBWqNcpF1D/M9R8asvgGAJ7ie/vvZ1X4xSO19sRapI+j8obrRr5Q== Received: from SA9PR03CA0009.namprd03.prod.outlook.com (2603:10b6:806:20::14) by DS7PR12MB9528.namprd12.prod.outlook.com (2603:10b6:8:252::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.15; Sun, 9 Feb 2025 10:19:46 +0000 Received: from SN1PEPF0002529D.namprd05.prod.outlook.com (2603:10b6:806:20:cafe::65) by SA9PR03CA0009.outlook.office365.com (2603:10b6:806:20::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF0002529D.mail.protection.outlook.com (10.167.242.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:45 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:44 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:43 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:38 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Akiva Goldberger , "Moshe Shemesh" Subject: [PATCH net-next 10/15] net/mlx5: Expose ICM consumption per function Date: Sun, 9 Feb 2025 12:17:11 +0200 Message-ID: <20250209101716.112774-11-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002529D:EE_|DS7PR12MB9528:EE_ X-MS-Office365-Filtering-Correlation-Id: 259b7231-5ea8-4584-60c3-08dd48f346b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: nIaBCODGO3uEnwnEP2u4T22DB2vU+iY5uMGiqS/62wSF2WnXTB7IfaFgAz2R03TF2bJk6AmCKCaat8uKpXafeo/J/vp+D9fRZo1PZ63wd6Dw9690iBbsIhgDsGfdJLYT9p/XQS2GTFbEUFtDe/K4rn/jHHfullQkkYQ1eq6gizV33pYKpbyiqSxVpL5mPQro3IqIjUTYJmvbw1B4l6Wb8uIDFvbLy8X1fg3Q4myyD/XsH+xikzURHozOPt+A6YUjiYeivPzV9DPHO5kiiviQm8epj8GKj6MnuzAiNOJ21z127vuQHJUJFWhz03ohV2OZRO/jrRwZea27gFFmMWySOC+VzRLBrP3mrz72TziB5gHDxFtSAF8uh5Uqn+5smmdUDRTEaWN5xF6ytxX27IsklrsojLMsYCn1fomm9f+yjEVXdRo/KbeqM+R87YfHStiKK1H6cZVouZgOfKvxTc+xyeY/E0MX5B96S5Wx/yWsckDt19Sxgrq6K1sfS/ryjtxu2HSZ3Ay62LiMpHnL/UaLub3S56PWZs6FC5mkouxbJUKuVH4vEGRYxP+CTlLgiPuCxt9xfjtKBPlOMDoq8ylN2ZR/x0Cyc6R6JVV2+rqPB7MiCHbiT+5D+/1PrcmEcUj73ccIlXVqA+BJkzqJjf48e+LIPu40ZT4laiIhTyG1rO14tLEXE3+P80mmnQvcVv/Y0eky+XqL5kBvH9w8etWg/orrwMifgBUnQCQVUvyIQ99IG6WdEAuHqk+QLBvhIE/MQCCi8NfRwK4W9Hsd5uygQMTU1/8XepEbIUbW6bU+lI53RL067Rx7wXYPHAsUs5KfDGLWaPAhIk1qbxxmBqma2j2JmItExqvo1gJddGM+MLTt6hhDYP4hP4krCn3OX6hX3/YS1mILd8Zvt9YfXVso8KiC3C0YG36AxdSHwOmwRI+Sd9FQ0oYmS5OA6TV3Ra0/hKCKq785F/hXJgmmak8d9JPcGW2b94BblbyfcSXhTbfAVvDPnNLbWiIQSYUnspozpObq/n2bmFr67QPE7rmnGbxnJ7hUqGRgwS5ZWkSc2zhWYYCtr8eguTRlc/MODjB/O1lPyHDuA+P6zz8Z17N2z/j3mKQkNWj5Bf2krJWcuutVYu/j39sEJLdX/HV7FPjcBL68DtUS636apvU+IRjTE7Ic71jgDgAvpvEaJ2BPfXPTxkrK6+7D1/vv8nNVI7t+57YFV9Bs33Unf/zFRsPzcoRxhM/hCL+UOW+WErpP4xEG6V+7WDCopQktk5OOHGk9lHUgVLmmCsxqj1H+MUKrOkSi1+QrooDo0FhDcMOoQulXcqROyIeyowtYPEUpSpo5ZF07Liv1WF+u3uXDeF0CWmb1itQLhcDHEUyWTLSO6BJNw0lN2kTUw4U4EVIebgAkq03FD0iH5EmBj1p0yDtWorWfenG9n0TI1gr/MyuzT7lgFhwiK1Y4LWIEdX6U8OwKHDyD559H6ZaW70fadXLXhvDmJqw8qnKQio2sYLVxStE= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:45.5748 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 259b7231-5ea8-4584-60c3-08dd48f346b7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002529D.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB9528 From: Akiva Goldberger ICM is a portion of the host's memory assigned to a function by the OS through requests made by the NIC's firmware. PF ICM consumption can be accessed directly, while VF/SF ICM consumption can be accessed through their representors in switchdev mode. The value is exposed to the user in granularity of 4KB through the vnic health reporter as follows: $ devlink health diagnose pci/0000:08:00.0 reporter vnic vNIC env counters: total_error_queues: 0 send_queue_priority_update_flow: 0 comp_eq_overrun: 0 async_eq_overrun: 0 cq_overrun: 0 invalid_command: 0 quota_exceeded_command: 0 nic_receive_steering_discard: 0 icm_consumption: 1032 Signed-off-by: Akiva Goldberger Reviewed-by: Moshe Shemesh Signed-off-by: Tariq Toukan --- Documentation/networking/devlink/mlx5.rst | 4 ++ .../mellanox/mlx5/core/diag/reporter_vnic.c | 46 +++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst index 41618538fc70..7febe0aecd53 100644 --- a/Documentation/networking/devlink/mlx5.rst +++ b/Documentation/networking/devlink/mlx5.rst @@ -280,6 +280,10 @@ Description of the vnic counters: number of packets handled by the VNIC experiencing unexpected steering failure (at any point in steering flow owned by the VNIC, including the FDB for the eswitch owner). +- icm_consumption + amount of Interconnect Host Memory (ICM) consumed by the vnic in + granularity of 4KB. ICM is host memory allocated by SW upon HCA request + and is used for storing data structures that control HCA operation. User commands examples: diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c index c7216e84ef8c..86253a89c24c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/reporter_vnic.c @@ -13,6 +13,50 @@ struct mlx5_vnic_diag_stats { __be64 query_vnic_env_out[MLX5_ST_SZ_QW(query_vnic_env_out)]; }; +static void mlx5_reporter_vnic_diagnose_counter_icm(struct mlx5_core_dev *dev, + struct devlink_fmsg *fmsg, + u16 vport_num, bool other_vport) +{ + u32 out_icm_reg[MLX5_ST_SZ_DW(vhca_icm_ctrl_reg)] = {}; + u32 in_icm_reg[MLX5_ST_SZ_DW(vhca_icm_ctrl_reg)] = {}; + u32 out_reg[MLX5_ST_SZ_DW(nic_cap_reg)] = {}; + u32 in_reg[MLX5_ST_SZ_DW(nic_cap_reg)] = {}; + u32 cur_alloc_icm; + int vhca_icm_ctrl; + u16 vhca_id; + int err; + + err = mlx5_core_access_reg(dev, in_reg, sizeof(in_reg), out_reg, + sizeof(out_reg), MLX5_REG_NIC_CAP, 0, 0); + if (err) { + mlx5_core_warn(dev, "Reading nic_cap_reg failed. err = %d\n", err); + return; + } + vhca_icm_ctrl = MLX5_GET(nic_cap_reg, out_reg, vhca_icm_ctrl); + if (!vhca_icm_ctrl) + return; + + MLX5_SET(vhca_icm_ctrl_reg, in_icm_reg, vhca_id_valid, other_vport); + if (other_vport) { + err = mlx5_vport_get_vhca_id(dev, vport_num, &vhca_id); + if (err) { + mlx5_core_warn(dev, "vport to vhca_id failed. vport_num = %d, err = %d\n", + vport_num, err); + return; + } + MLX5_SET(vhca_icm_ctrl_reg, in_icm_reg, vhca_id, vhca_id); + } + err = mlx5_core_access_reg(dev, in_icm_reg, sizeof(in_icm_reg), + out_icm_reg, sizeof(out_icm_reg), + MLX5_REG_VHCA_ICM_CTRL, 0, 0); + if (err) { + mlx5_core_warn(dev, "Reading vhca_icm_ctrl failed. err = %d\n", err); + return; + } + cur_alloc_icm = MLX5_GET(vhca_icm_ctrl_reg, out_icm_reg, cur_alloc_icm); + devlink_fmsg_u32_pair_put(fmsg, "icm_consumption", cur_alloc_icm); +} + void mlx5_reporter_vnic_diagnose_counters(struct mlx5_core_dev *dev, struct devlink_fmsg *fmsg, u16 vport_num, bool other_vport) @@ -59,6 +103,8 @@ void mlx5_reporter_vnic_diagnose_counters(struct mlx5_core_dev *dev, devlink_fmsg_u64_pair_put(fmsg, "handled_pkt_steering_fail", VNIC_ENV_GET64(&vnic, handled_pkt_steering_fail)); } + if (MLX5_CAP_GEN(dev, nic_cap_reg)) + mlx5_reporter_vnic_diagnose_counter_icm(dev, fmsg, vport_num, other_vport); devlink_fmsg_obj_nest_end(fmsg); devlink_fmsg_pair_nest_end(fmsg); From patchwork Sun Feb 9 10:17:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966740 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2088.outbound.protection.outlook.com [40.107.244.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ED241C07F3; Sun, 9 Feb 2025 10:19:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.88 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096396; cv=fail; b=InoBvzCDXLuQu7MrdOgMDabmVOdcVlfWzG91+ekIVpnxvookZJuNVkfWA1pnTxWgdG69Q/x3bcBCWN8FK3IqV1odV2cppuqtMj+Nqwjv74un42XKVkXZHx3p5esU1rLhP4z2zf+Uq4WF+qHpY74+TPr7Cas48HPVpwk5AO2lg4c= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096396; c=relaxed/simple; bh=Rsq8cRevFqQ1dJakWbjsVUxOtgZa1ITmNleLJt3RSck=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=omGHI9dmMUiEuuv9ayuUh4RfvsVB2W6TiaBpQhdIHFjoJXTIk+wWj5xW7ObEIbMN+93UZ2EQpkvyM0Isf4Iy1pypFByPvMC+LwE830jHb3A0rkC6PNQkX4B/VVquvhjUjQjvdNwRB3DkyhH+mq+NVeB4GA5FpJxjTN4wKvpL6wM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=WJoMrI0B; arc=fail smtp.client-ip=40.107.244.88 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="WJoMrI0B" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=deeKuLUExF7y998/VvJmX2VwQugnE653EO6oUfDkoZHPr6oSod4Ka/BwHP42hqP1TDzpBtXwlIUzTz3vm1OnZ91W7WnTuGTK1MBTjxwZILDpJG+V7FEKik8joR1aysEMC0hENvSV9BTHOqrNifLDsJbYcW0D6Krg30avLMlGWw5dQoTiz1JfQAbkQHMZlOt6mrmBWW1vIb9Ih44Hyj/oNLS/1p0bSKnYCt+MWyx+yAClkbhuy4YByPSaEO9D77iM0ZpFQgL1hz9yXYRGYapB6Vh9gXTsz6gRb7sGAJjS1GPIhrUJb6aYQrF72/ZMqAIpJ0s7V9CDoT8rwJF6Nhh8ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OL0VxAaoDRj3cc8vIfPy+cK8eZ5xb2P6nRX4zgoWc5c=; b=o5BqKObhM/fu2hkX1TLSHMOapHQ6794bvJ0J4EtwmH+Ag2kgMKc7KtfBX/TGNpgnzf45fWZQ0dci4ipI7+UmGbQD98tATsyboX3p4pFzl+xSXsC53AXO6X3ieFkTvtG+Xm+sz6pQJvOf+YSLdgj6x5jfvw6ONm09D9kajvR8BxIgJnFllnad4yFNsGw3nljz7fK6txVAIYclBmGFFMpbmzMcEYBPpqI0gL+42Y3QfkPnuhr+g4TuGqKHndRneOa4fDlVWcoPJtp+UU1f6jcp69jr9Ba+XBR4hESt3qknwcJ0Nhh5umsS9jmKlwedBa/tA/O6iVEC88srLCDNUE/qbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OL0VxAaoDRj3cc8vIfPy+cK8eZ5xb2P6nRX4zgoWc5c=; b=WJoMrI0BwV2j844GpgrWou7o5tadn5TfEqwACwgYckWH6yrPuWBOH/5rQfR1VbgT/OarofHaCqL1PzDS/VkZEqGhp5X+9e9u/uEb9edunBDSoFrIPSASMsJhOTDzoKRRcq2ZyUM3v0biKYR/mv00n3M8ZdrdNWYhxUj9I7uk8VufPR+sKuL3MHUbfGClDoe4x1h1Bi6skhPy4JuCDoNbEfN1GP/IE/X4/9butZLWTyJKNvJ64FXCwiAB5HdRpJV8dBLEfDwEbQraIQv4gF+3nfTeQGuGiZHXUtZItPcvz+mjdNXP0RY6mI30h3GHWcL8DSwR2iHI5s2YhmBhCiGU6g== Received: from SJ0PR03CA0103.namprd03.prod.outlook.com (2603:10b6:a03:333::18) by SA3PR12MB9105.namprd12.prod.outlook.com (2603:10b6:806:382::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.16; Sun, 9 Feb 2025 10:19:50 +0000 Received: from SJ1PEPF00002322.namprd03.prod.outlook.com (2603:10b6:a03:333:cafe::46) by SJ0PR03CA0103.outlook.office365.com (2603:10b6:a03:333::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002322.mail.protection.outlook.com (10.167.242.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8398.14 via Frontend Transport; Sun, 9 Feb 2025 10:19:49 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:49 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:49 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:44 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Amir Tzin , Aya Levin Subject: [PATCH net-next 11/15] net/mlx5e: Move RQs diagnose to a dedicated function Date: Sun, 9 Feb 2025 12:17:12 +0200 Message-ID: <20250209101716.112774-12-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002322:EE_|SA3PR12MB9105:EE_ X-MS-Office365-Filtering-Correlation-Id: 4a3711b5-8129-4713-e6f5-08dd48f3493d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: PtFl8iUnfNO+nayXkmYoZZ7vzMnU22+MsgqbiUN09LufV90bW0FxRS74685ddEE9CN5RpM3yNeUu6wY6z0+INA2hrYYTEgpBwPBbKmsJqCxokiQ/GEROy/YK0CZ7mU/WJaU+XxDGbKrA0SEVxQCpQHSN7XuxIZOtudc3ksFCdP0F4QbYXrVn46H4lZHyKFWVBCtsJ4XY201kyqRMraXPQeW7rMkSW0B4qejbS+yUBIFJcbKRj+C9rve+izjGgzGhFpBaxgACKH/bNXNeyCmGYAFkpuBvemhPFMGxs0hWaOGzQWR+nXCc/3xKE+eTE4conXceCfPdLb+kjcqnpbnFgNEpp8+blj8T9M8e6VTChTCetjxhN9qp3b1LOg7nyQrK6mny0Z5hiuwyuodvKh71HAcpvDZG3qRJ/zOKME0qgkW0++PTn1cLVZa6D0XXR4ikrYU9/39xXHbKVaR9zHWP2mkgEpF1qyjIqjFBD7tLCvJ1bGCYiPLXrSOVEhnqVcfHgrT+7kpLGZ4uTiiTA/tyudp6djIwZwbE2RDXZTzEE/AmXbWrXOpngj7UxMvb5tMYKpfPuzUzlZsMANVTvqVabtm7vxOXj8PbSPKpQw96D6uOZsYs7bH2tuAvSl5BwOyYGRNhzn+9yQfJUvKuC8hO5isrAqMlvuI5IPuH4VYcDTCS8MJr/tqNWPWMvUGStUpmeG+K3p5gbb8mjrp3pCtRqa/dYlBepDXgYeom9Z25k9ztNmnOjgNXViJVRsTPwFKSIilhfYmlF5+s6eoML4kel63qPBuw7Zp2UUYiLMaCjRG0ep/83fu+hIu7JQVoUJawXwLF8Whzdm5cgUjfYtYODjTTbM34JFoYQslQ3boyYxY5lgfdWxZZ0g/M06kR+9gN3aQqemNuYLKzur3kMIfAeScHOckHge0xdwmhM436pAmOiupdK3SkJvSe/A8YMWLD3ulcYVlcB6jSsp43eFj4M/IV4g6H/g4yCTjJEPLXUiyGhGgt67BfOq6OyXZcR3gDKykD1iKrTXZRPn7NkIq9tMh6VEd/N4hCXC2PsDlSQUP4egL2OTj3vTwcjMCUQa+GttjJhV2Lu9DoKD9NkYeeRmqIkGTdoTNilV+2QwZMl+VIwD5/jNl/VQyvwxwSQfvdea6hI7LHYncxX2yM3zUwrr9hB9wlz/ZZk6jABdXighmyPKt0zenJqdu6oW9pR523IYM+u0nQdHOGQSllNS93Nu2X3BfvQHQ8G8CzozdF/NmcWBXd5998UoDLCwFuMioheLJlvO2I8j3/mwLrkx5r9ZmzEl44po3OXUc3/3bopm7MB+Rg0Xcx1s7ZikINjba/uD98tcUQY/vKUd8PsHvoraO61GDw5If/eRhjASMuPNFTS1L2Xge/cLmzh62lGCN5/o0mFlRsE+6O1m76XP6ZzqPe2xK8MhDv69EPZpUfxgarzT8aI3zxtubNT6RPnNravZIQhHD+tM/nB/GvbrAQ6SPP71KUMWjgYF5tXwdu1gY= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:49.9368 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4a3711b5-8129-4713-e6f5-08dd48f3493d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002322.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB9105 From: Amir Tzin Move rx reporter RQs diagnose from mlx5e_rx_reporter_diagnose() to a dedicated function. This change is a preparation for the following series which extends diagnose output for the rx reporter. While at it, also pass a mlx5e_priv pointer to mlx5e_rx_reporter_diagnose_common_config() as this is the argument the latter actually needs. Signed-off-by: Amir Tzin Reviewed-by: Aya Levin Signed-off-by: Tariq Toukan --- .../mellanox/mlx5/core/en/reporter_rx.c | 31 +++++++++++-------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c index 25d751eba99b..9255ab662af9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c @@ -317,10 +317,8 @@ mlx5e_rx_reporter_diagnose_common_ptp_config(struct mlx5e_priv *priv, struct mlx } static void -mlx5e_rx_reporter_diagnose_common_config(struct devlink_health_reporter *reporter, - struct devlink_fmsg *fmsg) +mlx5e_rx_reporter_diagnose_common_config(struct mlx5e_priv *priv, struct devlink_fmsg *fmsg) { - struct mlx5e_priv *priv = devlink_health_reporter_priv(reporter); struct mlx5e_rq *generic_rq = &priv->channels.c[0]->rq; struct mlx5e_ptp *ptp_ch = priv->channels.ptp; @@ -340,20 +338,11 @@ static void mlx5e_rx_reporter_build_diagnose_output_ptp_rq(struct mlx5e_rq *rq, devlink_fmsg_obj_nest_end(fmsg); } -static int mlx5e_rx_reporter_diagnose(struct devlink_health_reporter *reporter, - struct devlink_fmsg *fmsg, - struct netlink_ext_ack *extack) +static void mlx5e_rx_reporter_diagnose_rqs(struct mlx5e_priv *priv, struct devlink_fmsg *fmsg) { - struct mlx5e_priv *priv = devlink_health_reporter_priv(reporter); struct mlx5e_ptp *ptp_ch = priv->channels.ptp; int i; - mutex_lock(&priv->state_lock); - - if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) - goto unlock; - - mlx5e_rx_reporter_diagnose_common_config(reporter, fmsg); devlink_fmsg_arr_pair_nest_start(fmsg, "RQs"); for (i = 0; i < priv->channels.num; i++) { @@ -367,7 +356,23 @@ static int mlx5e_rx_reporter_diagnose(struct devlink_health_reporter *reporter, } if (ptp_ch && test_bit(MLX5E_PTP_STATE_RX, ptp_ch->state)) mlx5e_rx_reporter_build_diagnose_output_ptp_rq(&ptp_ch->rq, fmsg); + devlink_fmsg_arr_pair_nest_end(fmsg); +} + +static int mlx5e_rx_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack) +{ + struct mlx5e_priv *priv = devlink_health_reporter_priv(reporter); + + mutex_lock(&priv->state_lock); + + if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) + goto unlock; + + mlx5e_rx_reporter_diagnose_common_config(priv, fmsg); + mlx5e_rx_reporter_diagnose_rqs(priv, fmsg); unlock: mutex_unlock(&priv->state_lock); return 0; From patchwork Sun Feb 9 10:17:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966741 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2041.outbound.protection.outlook.com [40.107.95.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 513071632CA; Sun, 9 Feb 2025 10:19:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096402; cv=fail; b=gGQbqg8tmJ8zmdtvx2tPfoLWhdLkYG8c26nnRvAuJMkPILWvLuZOmj4/k/6Nt84E7EM9Jjwtg8C0/7eXnIQO9NtoqnNWNf0orhlNNgNhNGb/o6/UD7rQK1J5d44SO08yRibOgyGsvQpb84K1FK84+IEVncR2dTsGdUprYujuBko= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096402; c=relaxed/simple; bh=8yxiMWQrM/4YsFWl9P5Cf1xj9cskLAhWehgNwQmkJAs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DO3n2ITd4nq+oWusrTQRMpIpmtwvbaGgdoJueF3P8Zi9APseFbMn3zhi7hTsqintSpoAbqwjsm0j+f4iFQjc+ZzZRMRfsosFbDxBoX9pD2hXaVl8+NkNiyYaBV6t9YXDWfWYsu1EgpPo2FUGa31XZa9DasA4nuPmqfkdzq1hqdY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=iAQTAPTk; arc=fail smtp.client-ip=40.107.95.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="iAQTAPTk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WjheUbLeZWPdFhGafdXpxzyNxr9vSThRgBhxb4mT6stxX1kOM/ANqurVrbJtOIM1QWEmvmSq40E2DKoVAH0gV3UZv1KmNuiS5eGqTXzMOFkqm8lOs5KG37UneJ6Y2smmQdJVzUmEMR5CereLiBRo4o26h00KxwxTlaSc5OeQPovvWtgeajOuAm62kD//sTD2XY54U/wEpwHNAANF5gigseJFwbj7W+DVsQKzSbDukJ8VP78Km9Y4v5fX7S4YoGPw5lgzYpS5j3DBk26Q5Q3lpDnQJlREVRBkjKc8J06+9CDqKhMCl6Ih+dFxrQN19vCY/xFohDO/ZwFRcWd6yKgmkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u+pBSAJWrUyVPddSTLcqC8g6AQVKZkYAzdB5JH7FPZ0=; b=D6bcVpQyyjU6tsk7ana99gWy8ii/if81WJjKiuH5nbLxnllWZDhJnWpph3X+5kSk/yKB9tg8Th9363OFOqax0p9xu35iPBrRgofGykeQBFqU673Y61Ve4CUZVXgNJd8zcTC34v3JaWkDfLPJydpmt/U+uPtKmjdd0P8ZgFC45i0wAjegj/OLLUBV9+0h8bxIW49SkTQd/ymTjSvPX7L6nJvM4tOo/+IxBfrH4qsAUbaaD0PZK4PEkziYnK+6ocQ195Wz7dijcI2URM61CuVGR6OrFXrkL/E40p3qF0ShHUdzYHrOzleTwc30SAhtVez9gu15vZogLuJ8vrzlfBkawA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u+pBSAJWrUyVPddSTLcqC8g6AQVKZkYAzdB5JH7FPZ0=; b=iAQTAPTks+Bo0ODeBfPFig4TfC8xebeN0wnWUWhQfqF7SeIHLCUMpxjvWnExtUi3GTpUZC9oe4mj5JT1dNeGvX/8aO1j6I9XhQ/W6AO+NABC9Ewo+GKpfHjF6e8TerVEwch/0Z1SkLw6oobdoEd0he594sNQB1sUDDlsIbXlYJDKm5wByOtC91qBZGz8QWJR/COCMFgi09mA5EwlmWHgf+cS/6MCSz4bosBwbCeP71pv9IWnHlvVLj96BYzE9OKNhLHVBV9aK0VfqGGPgdacvbH/4t/rFjqlHGEhDkXKTYrtxDVQ2ab8kCBdfY+m2N5TBWcdz6YgWiYyoBdpkqHQkQ== Received: from PH8PR07CA0032.namprd07.prod.outlook.com (2603:10b6:510:2cf::6) by DS0PR12MB9397.namprd12.prod.outlook.com (2603:10b6:8:1bd::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.12; Sun, 9 Feb 2025 10:19:56 +0000 Received: from SN1PEPF0002529E.namprd05.prod.outlook.com (2603:10b6:510:2cf:cafe::d2) by PH8PR07CA0032.outlook.office365.com (2603:10b6:510:2cf::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.31 via Frontend Transport; Sun, 9 Feb 2025 10:19:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF0002529E.mail.protection.outlook.com (10.167.242.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:19:56 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:19:55 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:19:55 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:49 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Amir Tzin , Aya Levin Subject: [PATCH net-next 12/15] net/mlx5e: Add direct TIRs to devlink rx reporter diagnose Date: Sun, 9 Feb 2025 12:17:13 +0200 Message-ID: <20250209101716.112774-13-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002529E:EE_|DS0PR12MB9397:EE_ X-MS-Office365-Filtering-Correlation-Id: 69f9041f-2c68-46d8-a218-08dd48f34cf1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: SsyLwBcfLiIhqceJ1FKZfJA9nPry+nZfJucn1/+EKpyidFZVGmwG+k668AtcWKdLenERtdc1d2ciVXoVn6Jl0lksAB0vVxVIFnfFLzQ5D60RywE54F9Rv7v8HtBxdose49sn66kuYJUpbHa0t/3hEUaHDDsQhTGCoYiviw86/rkA899SaQwlrQRM8SAdUM4wldrDoeDq67LlBSQ5OfAMnhOMQeEK3RgcQxbhGpxW2+MZBpkoFpTD5vKqlkXJMuTDhTxW3wCpxdxATU0OdLg0IXDqqpI39PsTawvROVRwxb01J+SbuoE//jVQjTUSGIknusVRRls6Ubf5zCFUfJX20JNN9N7U9l+029Y7fncDMn/HwrGuulwD7bN3oH/AWBJXS30Zlbbee12Gfr0kkf4n0IH+Yn3Gz/VUQ6mX2z1lIT9dZ9LYc3bNB/y8N5srcMJsNMBqciylRCJE+oDM4k0akNg6YvrEFTpKAWsWtMA790tS5sogAkS0RPWv/SnO1UvCHmVAw7brqxwBwoaXKHfuCInFLQgeEke089Gal+i2EtbMSokFO4fi7Bf071S2hk5GVLAnyVh1bRJJpaApVOL9A9BrgCecDktjWSPoqjW86rPcyMCY9kd7ortLklEqZVUFO0d9hMeLDF/mqeSbz7M3p2yq93PLOxDdmxAgIvWOGvJi6kIvJv9K96SCnEmE9WfbP7QfS7vA9HaNOjV9uQEtNAAtRPoue2m2SJEh2wOldeGVVFK2qu+O+VztdkKx+n6M5MfELj9MhZ6tDgndZ7OrG7O3WmUq6wmCASjnoXrc+/QyP+qtnxoPBKNfUV9qPrMYu7dk1BScV1OgTPn7fY6LdrQN3oFSwUIGp82fcZsuqC6L+KQ8Mu99rCCCG1rdA6yZstSHTUeJp1psFpZI0VZBFGNGETacsoHM084aNF8n2DBR5iitaDyedCFwQMZoV/oCzhtCPoLGhfzkN9laUUTch1U7rOP+fpVGtzJJgSL/zOwpENFBbcRzJs/pSAKW4vYK8XR3ISeV6rza7YjSbF/qbx/YGll1/X+5C+Yqkefywa8uCjnnGYhzAJlPd1j3tdr8e7tWnYmDuay4q1FNzcRLOuzVz3WOxLVY1ExiFgjRvzsIE6tg7pAFj26wZ9Q7h3WRNvWPgBeRZGeWMxU4syrZoaXijoKjwFkJmZsHEY4TkdcN72ARMuLJ2gIpjUYpnI58L/83kJVWaMzWtO5dlcPzQ2U4PSTrQB+6rCwU8aHBNH4QCnoiZu/xxOysI673POBPNaS3ZMxIcJJJ85eKa31iMVfHXBKdBUYB8XBlL2Udw4OlelNc80lR2sPApUSoYQZ9BmezLLjwCyZrFNWATVjJG9v8esXjrVEF/kLMNzwVAz7MgMI5KbU1D6B1v/cKFB/ivB/FY7gQlT68WbSgiTTE4hrFiwwa/Ym2yew3zofWIzcxn1eKWpiIafky/Ir9asfrn4fYfv3OPtXb2/FbGnJMHBLdXlsmQ8dJueFtP9jNThs= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:19:56.0229 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 69f9041f-2c68-46d8-a218-08dd48f34cf1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002529E.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB9397 From: Amir Tzin Add "RX resources" tag to the output of rx reporter diagnose callback. Underneath add tag for direct TIRs, for each TIR expose its tirn and the corresponding rqtn. $ sudo devlink health diagnose auxiliary/mlx5_core.eth.0/65535 reporter rx .... rx resources: Direct TIRs: ix: 0 tirn: 20 rqtn: 1 ix: 1 tirn: 21 rqtn: 2 ix: 2 tirn: 22 rqtn: 3 ix: 3 tirn: 23 rqtn: 4 ix: 4 tirn: 24 rqtn: 5 ix: 5 tirn: 25 rqtn: 6 ix: 6 tirn: 26 rqtn: 7 ix: 7 tirn: 27 rqtn: 8 ix: 8 tirn: 28 rqtn: 9 ix: 9 tirn: 29 rqtn: 10 ix: 10 tirn: 30 rqtn: 11 ix: 11 tirn: 31 rqtn: 12 ix: 12 tirn: 32 rqtn: 13 ix: 13 tirn: 33 rqtn: 14 ix: 14 tirn: 34 rqtn: 15 ix: 15 tirn: 35 rqtn: 16 ix: 16 tirn: 36 rqtn: 17 ix: 17 tirn: 37 rqtn: 18 ix: 18 tirn: 38 rqtn: 19 ix: 19 tirn: 39 rqtn: 20 ix: 20 tirn: 40 rqtn: 21 ix: 21 tirn: 41 rqtn: 22 ix: 22 tirn: 42 rqtn: 23 ix: 23 tirn: 43 rqtn: 24 Signed-off-by: Amir Tzin Reviewed-by: Aya Levin Signed-off-by: Tariq Toukan --- .../mellanox/mlx5/core/en/reporter_rx.c | 32 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/en/rx_res.c | 7 +++- .../ethernet/mellanox/mlx5/core/en/rx_res.h | 2 ++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c index 9255ab662af9..bb513a22dc66 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c @@ -338,6 +338,37 @@ static void mlx5e_rx_reporter_build_diagnose_output_ptp_rq(struct mlx5e_rq *rq, devlink_fmsg_obj_nest_end(fmsg); } +static void mlx5e_rx_reporter_diagnose_rx_res_dir_tirns(struct mlx5e_rx_res *rx_res, + struct devlink_fmsg *fmsg) +{ + unsigned int max_nch = mlx5e_rx_res_get_max_nch(rx_res); + int i; + + devlink_fmsg_arr_pair_nest_start(fmsg, "Direct TIRs"); + + for (i = 0; i < max_nch; i++) { + devlink_fmsg_obj_nest_start(fmsg); + + devlink_fmsg_u32_pair_put(fmsg, "ix", i); + devlink_fmsg_u32_pair_put(fmsg, "tirn", mlx5e_rx_res_get_tirn_direct(rx_res, i)); + devlink_fmsg_u32_pair_put(fmsg, "rqtn", mlx5e_rx_res_get_rqtn_direct(rx_res, i)); + + devlink_fmsg_obj_nest_end(fmsg); + } + + devlink_fmsg_arr_pair_nest_end(fmsg); +} + +static void mlx5e_rx_reporter_diagnose_rx_res(struct mlx5e_priv *priv, + struct devlink_fmsg *fmsg) +{ + struct mlx5e_rx_res *rx_res = priv->rx_res; + + mlx5e_health_fmsg_named_obj_nest_start(fmsg, "RX resources"); + mlx5e_rx_reporter_diagnose_rx_res_dir_tirns(rx_res, fmsg); + mlx5e_health_fmsg_named_obj_nest_end(fmsg); +} + static void mlx5e_rx_reporter_diagnose_rqs(struct mlx5e_priv *priv, struct devlink_fmsg *fmsg) { struct mlx5e_ptp *ptp_ch = priv->channels.ptp; @@ -373,6 +404,7 @@ static int mlx5e_rx_reporter_diagnose(struct devlink_health_reporter *reporter, mlx5e_rx_reporter_diagnose_common_config(priv, fmsg); mlx5e_rx_reporter_diagnose_rqs(priv, fmsg); + mlx5e_rx_reporter_diagnose_rx_res(priv, fmsg); unlock: mutex_unlock(&priv->state_lock); return 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c index a86eade9a9e0..4e301bb5e305 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c @@ -497,6 +497,11 @@ void mlx5e_rx_res_destroy(struct mlx5e_rx_res *res) mlx5e_rx_res_free(res); } +unsigned int mlx5e_rx_res_get_max_nch(struct mlx5e_rx_res *res) +{ + return res->max_nch; +} + u32 mlx5e_rx_res_get_tirn_direct(struct mlx5e_rx_res *res, unsigned int ix) { return mlx5e_tir_get_tirn(&res->channels[ix].direct_tir); @@ -522,7 +527,7 @@ u32 mlx5e_rx_res_get_tirn_ptp(struct mlx5e_rx_res *res) return mlx5e_tir_get_tirn(&res->ptp.tir); } -static u32 mlx5e_rx_res_get_rqtn_direct(struct mlx5e_rx_res *res, unsigned int ix) +u32 mlx5e_rx_res_get_rqtn_direct(struct mlx5e_rx_res *res, unsigned int ix) { return mlx5e_rqt_get_rqtn(&res->channels[ix].direct_rqt); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h index 7b1a9f0f1874..391671b09c91 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h @@ -34,6 +34,8 @@ u32 mlx5e_rx_res_get_tirn_direct(struct mlx5e_rx_res *res, unsigned int ix); u32 mlx5e_rx_res_get_tirn_rss(struct mlx5e_rx_res *res, enum mlx5_traffic_types tt); u32 mlx5e_rx_res_get_tirn_rss_inner(struct mlx5e_rx_res *res, enum mlx5_traffic_types tt); u32 mlx5e_rx_res_get_tirn_ptp(struct mlx5e_rx_res *res); +u32 mlx5e_rx_res_get_rqtn_direct(struct mlx5e_rx_res *res, unsigned int ix); +unsigned int mlx5e_rx_res_get_max_nch(struct mlx5e_rx_res *res); /* Activate/deactivate API */ void mlx5e_rx_res_channels_activate(struct mlx5e_rx_res *res, struct mlx5e_channels *chs); From patchwork Sun Feb 9 10:17:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966742 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2048.outbound.protection.outlook.com [40.107.244.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FAA11C6FE3; Sun, 9 Feb 2025 10:20:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096409; cv=fail; b=VoRspTjo5T9OQOZVH0ktFo2hFuNbhntzMoaMtZDyjv9JaUxdDDdP6IN71m2mFTgsIHvubnKv0y8QkHI0EejN7WvXuJc6701pmxnchMYfJsFbBQcsx3GNBbe2LM2FEO6Fp9yBIG4mbedamTmC0ISPX4tjR4c8z2b2C2PYY0g4aEs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096409; c=relaxed/simple; bh=2Dtq1Z181y6JdpdlxsBVQg3QjvdjZujOMgzGtudtEVc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Olhzl9CFl8aqLeUcUaS1y4ZtS0P5hZnPyvvowTVJuYlN4CGlWAsD6vNmh2ZMg+nRauS32TZlUW7xFkydqdUYayph8tBITjFoYDsfvWF+FJlHH/SctqH9fi8Nw21qIcrO3l894QuRGVKpGmvw6jT7VYXFtMElO/XvgcWNkeJ51pI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Wf7RIapi; arc=fail smtp.client-ip=40.107.244.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Wf7RIapi" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lGEKIX/F2PzQhCfYFutFBanNV1cd3N8Afux5n2TTxOYNG+vn6P9vmFhfMIaPYnycEeBCJuTHXelN5qbWuBiO9s8U2v4qSWclyY16PVLnh0rV5NZPtlh2zKOXeKW1DY1vboDrs+7rO78izo3fbjwz6eKtZNU+aMMQZW0KmF9Y+UoLzK2S/Zi8MH4VoeCxQOR/loJYBVRZ2khfmvgItpJLS2EgjRpHi5e3kBDvf00B22TjnixEx/e8q+ORBKwtr7Jwa4dA5kvJ5y6c7rdigbL0dTquLHq0+GnkCfLzx2+r3b7GHF68bPeXvUE/ne3PJsBqCfeDiPbZuXEMXVjbYvOR7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yy2zmBFY9k2CWHopSktNDw03KUCfjBokgULjr5WIWg8=; b=MGzXc8O6V0iL+/DOHKRZGvrLYuhdsz6ASbPa/Tfdkqy9SBsl3qpQ9L35kFfANm8QRewdjIUOSip0Hhj46YpeJBjRojR5SBsbIWb+6cAd3aYzLSreKE6Q66gJtviaf4WAF9w1BXjTPjXyXG8o6sqT8R32WXIYAYN/yjoop8yZpiPGBWgM7cX33dEAPn/Ie5o0X5mh+0lNZ1K6RwplKOkRTyE0rj7m7rlGk0ydfnv1zCUOmHBfIRbRyQrLcAkpaJfBR2YrFMzKTUyN7VjDcos9gvkiXtzCfnGWjOvchUNbb2evr9tvVrS0B3cGrRlFhCEvMSbjw2wZXHJOxr5k2u6b7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yy2zmBFY9k2CWHopSktNDw03KUCfjBokgULjr5WIWg8=; b=Wf7RIapiaR39rgYMa9e+fjvzsLkaZ+9MZ1v1bHAjEr/b0Pn5c9tiKlLHuC05XikgYN5HOG5JcUx+ftWqIVe4umXryVpCiXcFohoUs5nxT+2tuDTUKPTZlA0X2bSfNy8NUGAioQKOfPI9/wP2Q1mYW450eZdZrNyUApF86/cqGGnkBI4OariMTDzmwqOUneP154Xshl7iU2LcOAnGdE65o5/PNXlplxbRPUWNOyYUAF3Vp6lyc4E/BBJuFpe4pj8IdoUZ9VpKUKmq0dmJpvNYRGnYS2+lX/jQE8gr/2gpQHGHybHLC2lOXNUBKNHJpR7CTl3aK9A4SfRJQu8xkH3rqQ== Received: from BYAPR07CA0003.namprd07.prod.outlook.com (2603:10b6:a02:bc::16) by LV2PR12MB5989.namprd12.prod.outlook.com (2603:10b6:408:171::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.12; Sun, 9 Feb 2025 10:20:02 +0000 Received: from SJ1PEPF00002323.namprd03.prod.outlook.com (2603:10b6:a02:bc:cafe::1b) by BYAPR07CA0003.outlook.office365.com (2603:10b6:a02:bc::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.30 via Frontend Transport; Sun, 9 Feb 2025 10:20:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002323.mail.protection.outlook.com (10.167.242.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:20:02 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:20:01 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:20:00 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:19:55 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Amir Tzin , Aya Levin Subject: [PATCH net-next 13/15] net/mlx5e: Expose RSS via devlink rx reporter diagnose Date: Sun, 9 Feb 2025 12:17:14 +0200 Message-ID: <20250209101716.112774-14-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002323:EE_|LV2PR12MB5989:EE_ X-MS-Office365-Filtering-Correlation-Id: 752b8a40-533f-472a-9717-08dd48f35091 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|7416014|1800799024; X-Microsoft-Antispam-Message-Info: fBAW+SEFNeumi1B8C4P4POwP83xWgJg2cI2NQYQfZyYCGbfZPSewCEwGLqIjeYAbm0WFr9XoqugfMm6bCD1bMiqOhPX2BqSbe8WC07udvlwHiKZuVfrp0K/+FCabq3s5FFBht8UL1BR131ftXzybjWVEQZRWUR4EaGVFl0tmmMKtaVDHfpuDenAzzpCqR/w3VWuNq7yAcyXjGg/XKqu0vC03l0o1UuEtaVv7ucFItqF6wiWbXGeaCe42DHSE1Ga3HA4dTcZdkvr6IfdvEbGG1NcCOPFW18CzyXOGC30CxMJVNz/LAGvuA0OtjmOEgOTOojtKRh48PkrhgQGF6bDOm8ZkGyH5qMEbu+9eZGsFzR2aRLLYOdWW1q5UUAhMGTYqP1B7R2j3DjVCCIdDt7Qb6pQcQgrwqNt51QOWbs2ZmJ4gGcm3Iq2IjIfjbYfxjejU5hsQTD88Zu6U7NXz45cvUGCxn+6pY/6IG8XYIwgAdTWxh0P6pH7BXTgvudTPGrqxjv6WUzQOje0zVZZWOfvP8m7uWFa0/3mTDQi+yfLGCvgj2buc1k9k3SRcG6L6xVSDNbvA818SeeUEIn1WlDj7M4pfc/1JNPOd7ybdPc7f01vxYhqw2LoWBJNoSsozpbXizuhWxq0CPFb+uV4u4uvxXZ9/5+xGcsjYuALvvv+6vKjanOSDrfUOQr4AlWVeNG9uqCVgzvI4QIIC8/cYYCbNOyTrY8LTzsqUQPjL6iuoSELggc32uDGW1NjHaMgGcSFXABnuNyGqCgNOx/umkQqoXAsyEsEC68BcHx1arIVlXSLOdNQYszOMsCsdypiWDxQ7bO3B6WElf8VkPPpoIH6qIKGEGYydjO9slm3dRu+qRPAOtwAF4Vp9EVWzMof3/mH7wiJPwtFwxOJArGkINDQEebgkdZ+zuXQ5z7CZNZESJ8Uhi/LT9HWW70sge/jh/XY6pD/v69S9t9dKrAiuy2P90NoWbXi8quMy5PqEboBiNwbXjtf6752gR/+lAjsdWYkh83CdgJcOO9+D/dKWFRpMS3PyOPPojV4EOYuU1ONCZe79wlS9z59s9kR/GYau2rhFXMNQZT7dtLRuWIT2P4IAxb5zqaTHb8zAd3v3zN7hEtjW2N58BcJDh+1U4c37MB/LeQ5orW2wT2VAkEQQ3A9TgVqCHGwA5eDM9wLM9xhwnWjQ93CXFAgnL/mFbaQmudre1s5BExqEyL8HC+oOZ2TS7kE3wQPMtBMV7u5s+fO3kqACd/s1bzf0IUXYqKTakpl60T9b+Bc4G6ZIzLUINZOywy8JvG2XrpaMKIDAbmX1yQcq+lg5zeiyR8+ubemSI9s2VpFCqojo9AHkGEDz8NpUWOzmYWsRLhtp+KfMHZtHSzZvgwrbOS8URNHxlWTVgiDWhQVCJmup9cF2DI1AZdBj1sxPi9iGSIQCGTGtP729FxPxWng6GBhDqESiXhw4dHhAmQ8zjfftugVjovvzTCXKFiosKW4u20thsiRfiMJ0SzM= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:20:02.2293 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 752b8a40-533f-472a-9717-08dd48f35091 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002323.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5989 From: Amir Tzin Underneath "rx resources" tag expose RSS diagnostic information. For each RSS expose its rqtn, TIRs and inner TIRs. $ devlink health diagnose auxiliary/mlx5_core.eth.0/65535 reporter rx ....... RSS: Index: 0 rqtn: 0 TIRs Numbers: tt: TT_IPV4_TCP tirn: 0 tt: TT_IPV6_TCP tirn: 1 tt: TT_IPV4_UDP tirn: 2 tt: TT_IPV6_UDP tirn: 3 tt: TT_IPV4_IPSEC_AH tirn: 4 tt: TT_IPV6_IPSEC_AH tirn: 5 tt: TT_IPV4_IPSEC_ESP tirn: 6 tt: TT_IPV6_IPSEC_ESP tirn: 7 tt: TT_IPV4 tirn: 8 tt: TT_IPV6 tirn: 9 Inner TIRs Numbers: tt: TT_IPV4_TCP tirn: 10 tt: TT_IPV6_TCP tirn: 11 tt: TT_IPV4_UDP tirn: 12 tt: TT_IPV6_UDP tirn: 13 tt: TT_IPV4_IPSEC_AH tirn: 14 tt: TT_IPV6_IPSEC_AH tirn: 15 tt: TT_IPV4_IPSEC_ESP tirn: 16 tt: TT_IPV6_IPSEC_ESP tirn: 17 tt: TT_IPV4 tirn: 18 tt: TT_IPV6 tirn: 19 Index: 2 rqtn: 27 TIRs Numbers: tt: TT_IPV6_TCP tirn: 46 Signed-off-by: Amir Tzin Reviewed-by: Aya Levin Signed-off-by: Tariq Toukan --- .../mellanox/mlx5/core/en/reporter_rx.c | 58 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/en/rss.c | 15 +++++ .../net/ethernet/mellanox/mlx5/core/en/rss.h | 3 + .../ethernet/mellanox/mlx5/core/en/rx_res.c | 2 - .../ethernet/mellanox/mlx5/core/en/rx_res.h | 3 + .../ethernet/mellanox/mlx5/core/lib/fs_ttc.c | 19 ++++++ .../ethernet/mellanox/mlx5/core/lib/fs_ttc.h | 1 + 7 files changed, 99 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c index bb513a22dc66..e75759533ae0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c @@ -359,6 +359,63 @@ static void mlx5e_rx_reporter_diagnose_rx_res_dir_tirns(struct mlx5e_rx_res *rx_ devlink_fmsg_arr_pair_nest_end(fmsg); } +static void mlx5e_rx_reporter_diagnose_rx_res_rss_tirn(struct mlx5e_rss *rss, bool inner, + struct devlink_fmsg *fmsg) +{ + bool found_valid_tir = false; + int tt; + + for (tt = 0; tt < MLX5E_NUM_INDIR_TIRS; tt++) { + if (!mlx5e_rss_valid_tir(rss, tt, inner)) + continue; + + if (!found_valid_tir) { + char *tir_msg = inner ? "Inner TIRs Numbers" : "TIRs Numbers"; + + found_valid_tir = true; + devlink_fmsg_arr_pair_nest_start(fmsg, tir_msg); + } + + devlink_fmsg_obj_nest_start(fmsg); + devlink_fmsg_string_pair_put(fmsg, "tt", mlx5_ttc_get_name(tt)); + devlink_fmsg_u32_pair_put(fmsg, "tirn", mlx5e_rss_get_tirn(rss, tt, inner)); + devlink_fmsg_obj_nest_end(fmsg); + } + + if (found_valid_tir) + devlink_fmsg_arr_pair_nest_end(fmsg); +} + +static void mlx5e_rx_reporter_diagnose_rx_res_rss_ix(struct mlx5e_rx_res *rx_res, u32 rss_idx, + struct devlink_fmsg *fmsg) +{ + struct mlx5e_rss *rss = mlx5e_rx_res_rss_get(rx_res, rss_idx); + + if (!rss) + return; + + devlink_fmsg_obj_nest_start(fmsg); + + devlink_fmsg_u32_pair_put(fmsg, "Index", rss_idx); + devlink_fmsg_u32_pair_put(fmsg, "rqtn", mlx5e_rss_get_rqtn(rss)); + mlx5e_rx_reporter_diagnose_rx_res_rss_tirn(rss, false, fmsg); + if (mlx5e_rss_get_inner_ft_support(rss)) + mlx5e_rx_reporter_diagnose_rx_res_rss_tirn(rss, true, fmsg); + + devlink_fmsg_obj_nest_end(fmsg); +} + +static void mlx5e_rx_reporter_diagnose_rx_res_rss(struct mlx5e_rx_res *rx_res, + struct devlink_fmsg *fmsg) +{ + int rss_ix; + + devlink_fmsg_arr_pair_nest_start(fmsg, "RSS"); + for (rss_ix = 0; rss_ix < MLX5E_MAX_NUM_RSS; rss_ix++) + mlx5e_rx_reporter_diagnose_rx_res_rss_ix(rx_res, rss_ix, fmsg); + devlink_fmsg_arr_pair_nest_end(fmsg); +} + static void mlx5e_rx_reporter_diagnose_rx_res(struct mlx5e_priv *priv, struct devlink_fmsg *fmsg) { @@ -366,6 +423,7 @@ static void mlx5e_rx_reporter_diagnose_rx_res(struct mlx5e_priv *priv, mlx5e_health_fmsg_named_obj_nest_start(fmsg, "RX resources"); mlx5e_rx_reporter_diagnose_rx_res_dir_tirns(rx_res, fmsg); + mlx5e_rx_reporter_diagnose_rx_res_rss(rx_res, fmsg); mlx5e_health_fmsg_named_obj_nest_end(fmsg); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c index 5f742f896600..0d8ccc7b6c11 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c @@ -81,6 +81,11 @@ struct mlx5e_rss { refcount_t refcnt; }; +bool mlx5e_rss_get_inner_ft_support(struct mlx5e_rss *rss) +{ + return rss->inner_ft_support; +} + void mlx5e_rss_params_indir_modify_actual_size(struct mlx5e_rss *rss, u32 num_channels) { rss->indir.actual_table_size = mlx5e_rqt_size(rss->mdev, num_channels); @@ -449,6 +454,16 @@ u32 mlx5e_rss_get_tirn(struct mlx5e_rss *rss, enum mlx5_traffic_types tt, return mlx5e_tir_get_tirn(tir); } +u32 mlx5e_rss_get_rqtn(struct mlx5e_rss *rss) +{ + return mlx5e_rqt_get_rqtn(&rss->rqt); +} + +bool mlx5e_rss_valid_tir(struct mlx5e_rss *rss, enum mlx5_traffic_types tt, bool inner) +{ + return !!rss_get_tir(rss, tt, inner); +} + /* Fill the "tirn" output parameter. * Create the requested TIR if it's its first usage. */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h index d0df98963c8d..72089f5f473c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h @@ -32,8 +32,11 @@ void mlx5e_rss_refcnt_inc(struct mlx5e_rss *rss); void mlx5e_rss_refcnt_dec(struct mlx5e_rss *rss); unsigned int mlx5e_rss_refcnt_read(struct mlx5e_rss *rss); +bool mlx5e_rss_get_inner_ft_support(struct mlx5e_rss *rss); u32 mlx5e_rss_get_tirn(struct mlx5e_rss *rss, enum mlx5_traffic_types tt, bool inner); +bool mlx5e_rss_valid_tir(struct mlx5e_rss *rss, enum mlx5_traffic_types tt, bool inner); +u32 mlx5e_rss_get_rqtn(struct mlx5e_rss *rss); int mlx5e_rss_obtain_tirn(struct mlx5e_rss *rss, enum mlx5_traffic_types tt, const struct mlx5e_packet_merge_param *init_pkt_merge_param, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c index 4e301bb5e305..9d8b2f5f6c96 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c @@ -5,8 +5,6 @@ #include "channels.h" #include "params.h" -#define MLX5E_MAX_NUM_RSS 16 - struct mlx5e_rx_res { struct mlx5_core_dev *mdev; /* primary */ enum mlx5e_rx_res_features features; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h index 391671b09c91..05b438043bcb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h @@ -10,6 +10,8 @@ #include "fs.h" #include "rss.h" +#define MLX5E_MAX_NUM_RSS 16 + struct mlx5e_rx_res; struct mlx5e_channels; @@ -36,6 +38,7 @@ u32 mlx5e_rx_res_get_tirn_rss_inner(struct mlx5e_rx_res *res, enum mlx5_traffic_ u32 mlx5e_rx_res_get_tirn_ptp(struct mlx5e_rx_res *res); u32 mlx5e_rx_res_get_rqtn_direct(struct mlx5e_rx_res *res, unsigned int ix); unsigned int mlx5e_rx_res_get_max_nch(struct mlx5e_rx_res *res); +bool mlx5_rx_res_rss_inner_ft_support(struct mlx5e_rx_res *res); /* Activate/deactivate API */ void mlx5e_rx_res_channels_activate(struct mlx5e_rx_res *res, struct mlx5e_channels *chs); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.c index 9f13cea16446..eb3bd9c7f66e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.c @@ -61,6 +61,25 @@ static void mlx5_cleanup_ttc_rules(struct mlx5_ttc_table *ttc) } } +static const char *mlx5_traffic_types_names[MLX5_NUM_TT] = { + [MLX5_TT_IPV4_TCP] = "TT_IPV4_TCP", + [MLX5_TT_IPV6_TCP] = "TT_IPV6_TCP", + [MLX5_TT_IPV4_UDP] = "TT_IPV4_UDP", + [MLX5_TT_IPV6_UDP] = "TT_IPV6_UDP", + [MLX5_TT_IPV4_IPSEC_AH] = "TT_IPV4_IPSEC_AH", + [MLX5_TT_IPV6_IPSEC_AH] = "TT_IPV6_IPSEC_AH", + [MLX5_TT_IPV4_IPSEC_ESP] = "TT_IPV4_IPSEC_ESP", + [MLX5_TT_IPV6_IPSEC_ESP] = "TT_IPV6_IPSEC_ESP", + [MLX5_TT_IPV4] = "TT_IPV4", + [MLX5_TT_IPV6] = "TT_IPV6", + [MLX5_TT_ANY] = "TT_ANY" +}; + +const char *mlx5_ttc_get_name(enum mlx5_traffic_types tt) +{ + return mlx5_traffic_types_names[tt]; +} + struct mlx5_etype_proto { u16 etype; u8 proto; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.h index 92eea6bea310..ab9434fe3ae6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/fs_ttc.h @@ -49,6 +49,7 @@ struct ttc_params { struct mlx5_flow_destination tunnel_dests[MLX5_NUM_TUNNEL_TT]; }; +const char *mlx5_ttc_get_name(enum mlx5_traffic_types tt); struct mlx5_flow_table *mlx5_get_ttc_flow_table(struct mlx5_ttc_table *ttc); struct mlx5_ttc_table *mlx5_create_ttc_table(struct mlx5_core_dev *dev, From patchwork Sun Feb 9 10:17:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966743 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2065.outbound.protection.outlook.com [40.107.92.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D6E91991CD; Sun, 9 Feb 2025 10:20:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096420; cv=fail; b=dU6x38bh8By9cviJNVJ/7Q9QAK54xVJtmKUfT3q2YYL9KhslxH6RJkE2UXUxhUdLlfGckgpeWQBq7glztL2zixx6uPyAOBlKx033z0isRqgUfgnDrtv71kQicqarxBEUBiRmfFDJBMEsE+ZgpJiRiQOVwubNG80MiYsPoG0OtUc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096420; c=relaxed/simple; bh=crmBcZNjO/U0ic52M5f4OkDKN1chMTsXSyAuybUGJ3o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=k4mPM+FlqEvh5n/IXafbEA9dXdzxFKO7PwmULxMh0pjp4iZGIJNsDc1GzxM7HSLd/49DF/d2O8H50M4fvCQHWeuH0zDRNX8cY9KpwsV83e1tf5TyLZ0uAreyHbPtGffQaQCLywSoN2lF93CCFspHJ9EsWN8Jqt1nvVgvd6bUv28= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=dTlmyaym; arc=fail smtp.client-ip=40.107.92.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="dTlmyaym" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gMiW9CTltnrc28J1XLyLNadff22i4rjEClDA+jtS65j4YoMeP5gXrSM11RX+Gp3XnQ0MjpuMSoPRceDpMoSISKUSexCNuwGMKgKQyEFvjrLzO34/am6M6z4+FCg20zFoLYxN5FpYLMk2hmYBvsCOH4bJnq2Sff/OVSTyM17rO87Sd/RHtQqPEH1m6PHuG6S5tyaf7ubb5BGE6W7y0DwwbJYCk1hlEfHaiCm17dVwTd0A4VEeJdgiMveOjEXc4Odt4louP5nqgong25UA+WgMLFsthdGYDJuzRee/pmc04nn2h7kYO4Rzdiilpwwmm+6+2ZODTamEW027sMMrDgbI8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a/cG9Vx2V6eiCq31/r+Ssw1l3oTTLglzykKTkqoos8w=; b=KbjeNrXORyKrwCOyWgdsaBVoOH4167i95REDwNLvZeJFXUnpY/O+/fLXAvRwbN5haBMdHQoC/DU3owj8cvzJQm6vmKhHN63NmbLN4ozpI0whcjos6gjaD5jc4KwtAgVkhZVj5mmCl0oITGA/+zZkdCmPvNGyT3SdaN5CFy7bdTSxYPiZm0uGOOAUayPLbxRS5bAvlfc4k3Pd3GXOnBLiVhGyUsFTebIvApNpBa5cnQB8148Cm1/5g3av7ogK1XIuVQL4+TR51tshMtmG68nLpp8Q0BS9Oj3iHy03gKcLuRAhSp4n7cgrQYVkYciEALGNwfiGRjfQe/2e4U0rE38EOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a/cG9Vx2V6eiCq31/r+Ssw1l3oTTLglzykKTkqoos8w=; b=dTlmyaymOacg6WO4A/PJxMbdb3h/QyxHNmKI/WNmaMLmXlZAInSK8zb5JOifyrUDi2sZZTB7q0nIs9OSMomgzJOwV4ds2ka0G4GLbWneY0IQpKbjSXRf2qDOwu0ItCSLtE0O4MfiZxaCb7c7AKfLaPA8drfSMjZRD4GwVSBCCqhKRAI2YQQCP2aQ5xjC5kwt8tEkpSWLLBkQ3V5UqAhZkJtWk1w6waSoVuit0zLEBlQ9muqhJ5QQpeaB3YcqVfQBYXnXsC4CEvNSUJXO0NRfku2qJSIxjRHPeJCK0e328yz3K3imF75tTk0wgBVo0og9bPLsmbt2ePWdfes7bQ9OWw== Received: from SJ0PR05CA0050.namprd05.prod.outlook.com (2603:10b6:a03:33f::25) by PH7PR12MB5594.namprd12.prod.outlook.com (2603:10b6:510:134::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.12; Sun, 9 Feb 2025 10:20:14 +0000 Received: from SN1PEPF000252A2.namprd05.prod.outlook.com (2603:10b6:a03:33f:cafe::eb) by SJ0PR05CA0050.outlook.office365.com (2603:10b6:a03:33f::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8445.6 via Frontend Transport; Sun, 9 Feb 2025 10:20:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF000252A2.mail.protection.outlook.com (10.167.242.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:20:14 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:20:07 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:20:06 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:20:01 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Alexei Lazar , Dragos Tatulea Subject: [PATCH net-next 14/15] net/mlx5: Extend Ethtool loopback selftest to support non-linear SKB Date: Sun, 9 Feb 2025 12:17:15 +0200 Message-ID: <20250209101716.112774-15-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A2:EE_|PH7PR12MB5594:EE_ X-MS-Office365-Filtering-Correlation-Id: bc05cd9c-672e-4038-a5e0-08dd48f357bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: qMnPYYyKGBujYZr/9aWf3jy4xSYaEFaUun1PQBL1HlPOyzxdXjCPDVyFNunOxGeci+yKiJQu7PKLxDu4ESx5mXHfi78NcIMzgMHB+8RAA1Se9yjS7mFn0o+be8df5QWqww2R3Jj0Q2wVg3Z92t25ikN5rCrVE6QVm6Wtp2rhDZRMGm3M5EK+K9mRt2Pr8ZtxsXZh4deDX4eyc34Lb60gPHRomHAYq9c5Z/QnIme61IpGOGkAsnuS5UAO6I8wfKf29rOFwpS6kXM7hJYAxOphxryShaZbPGWnzy3jjnkgRMiNokWwPh1tTsTthZW7oE/xqX29Z4ykht3NQM9yekw26DneSD1DPNxd5/0HTk7Xy0bj8G94XwvrwUhO1Vyn084rJqYAnssc6BVAiD5BxOA+ht60jvdlbTyoGaSwc+IZGFP3EDV6eX/evp6ZR3SPCBQGJrOK7XOZc4ARk9fK5vSZcZZyMuQb3xiE5ROtEks8MgHgIEXJqnIIhJVI3CHMEAtamoujUfedYTeFNSaXKhDYYnQg/D3ZjtxBfQrOvP/0As0mBbgyLwuIRWZiUNXbYZdC0J4w6d9qmPi73Tpj0CuutdYPcInldVJdYc7FYFCChWq2hroJk+YQWfQmxCQ5CfNLJOcGnUiVbG7DkLVkaHsfbZCvyDYNxX/QseMUy2j/cBjUmhf4LMw3vFpL1scexDLgxjL0YSFMFYVUZfgzs+p4o3QT1YZg6MjiqsA4GlwUraFhAn+q9bBgOZKEnNped7KlwQRPT0eUUv13zOGcKEwk6C1IbSCYMaMWzUZkdDZYy4cK0/+Nf9NHNND6VrCPYscXwl2gDV2I4B4SpKjJECDZqfUHfqKWvO4LUgEhgLN6barHNODYEGWAMiPveQapWjdVNb0BEEGVgroxpuHOj/g01xhPZ/XU2XQam9HHDkDuSQkfbfIOYTrsfjZeAlCTh66/Y9qjDI3veimbh2D/ITjjkQZ/nm4BbSYTcylUEWFq9/qQekDPZt/MeSvqw3RAqgHlq7Tv5TE8UAUw6XFmLGjXraz0nPUprZqb4IOiHbdt3mmO8wa9CPKZC3goxseh//VeRAV+KaPpfemAid5JTa8K4xuaKtRI8e6hu9nhCSNL4bbyUXepIjlVlOj7ohWvhSxQDKZB/Odh8/aWQsKz2KoAMNqQJ2wzoT5p8De4dPO3Z0uztwrmstMHKIDneyMQn8SI4nsf7WkuYk/h0zexovUV18ahC9TlxkvkFrofeW+K2XcTBgTdk7KxMyM2JatHGN5KOAlpzenwMuH91AjNoAI4aQiXtOaPxZWx5jRtM/YXTqB9R8lk8EfqIrwWAstDmgrJpOvFIw+DNJlF+6DrJgLlzOLB+Vhzu+hu1AfJXQLN5DH4OPAxvCPnzCtI9elyFYEjShePMtq2ut56uNSv9SQChy/LsgnW8vDsv2y0KVp/gCWu0Lifp3RudJLJ1NtQvZLJhBPECWFwbdjI9w/o/gZnk4DhcsRbPFW8t6WfrHJSwyY= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(7416014)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:20:14.1396 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bc05cd9c-672e-4038-a5e0-08dd48f357bb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A2.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5594 From: Alexei Lazar Current loopback test validation ignores non-linear SKB case in the SKB access, which can lead to failures in scenarios such as when HW GRO is enabled. Linearize the SKB so both cases will be handled. Signed-off-by: Alexei Lazar Reviewed-by: Dragos Tatulea Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c index 1d60465cc2ca..2f7a543feca6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c @@ -166,6 +166,9 @@ mlx5e_test_loopback_validate(struct sk_buff *skb, struct udphdr *udph; struct iphdr *iph; + if (skb_linearize(skb)) + goto out; + /* We are only going to peek, no need to clone the SKB */ if (MLX5E_TEST_PKT_SIZE - ETH_HLEN > skb_headlen(skb)) goto out; From patchwork Sun Feb 9 10:17:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 13966744 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2043.outbound.protection.outlook.com [40.107.94.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84EEF1D5AAD; Sun, 9 Feb 2025 10:20:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096423; cv=fail; b=kxgsmgSLkn7wsi5BVf1xil8KUnlt1cX/TTROql9fyaIjItSVTOrKgeYwglxV4UCDEPFbfAk1vFmkLbPkVMcHxB3XFfFd8bMQXWr3McMKIUXRDluSs8ndcUHsHt1PnXH0SmaRfxg03Cax/LVz2T67wSQPvWYOwFJcL80vbVbGhog= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739096423; c=relaxed/simple; bh=mtG2OBxa5lEk38ygdIVL/B6KTM8gISqWMJiPwtyYS/I=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VO/0puUQwGvSCalcQUf++B90Us6eG8u5S2lblK6TTUmlTjelgc3S/JUNxSBkPrbnY4kDW3UDpDB0eOzBP+J8d5HHArH+uA9Q0SB59uXjjnMDIpyz8qhJGYClQw8KuRj4WstVuioScAplb4QiMQBLYAf7Fy+v5oixP1OJnSk0SGM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ta3Y3B0j; arc=fail smtp.client-ip=40.107.94.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ta3Y3B0j" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ROuI3EXEb/6otIP0Ojf6/+jJTYIJlFqlCmrnh/q3pxJi0aAOKF8kTrHUz66eBCIa7tnyWHT4GN5bBqcxItIc2HC45RZF5klbw7TSNRy8r7tcr5qBrkI3+rmTXVt7hXCYMZYHn6FV+YfVNU30AZivi8HjKwtja7RiuNeSZh+IkbrB5U5fjJggUZfHiPq9Qkz2km4kwNq6aB7tiZpymkP03To0CjhL5zZe4h6wUcGId+aUIoKF/nye67YNdXH5X36Rh/YogxdSosDSCOkirT4YWr0Jw1KC6EQGARPbMlgjAufBAu1aFEoCJYbfOqaUET6pZme8/VFohA6rngVaJxJ2Yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r0gHxefuejxy5/6ZoMCuTrQc38czNSeq20vxSOdC1dQ=; b=mse7O6MrCjKZS1bUPpEBLD5pNuCpQJoKZATAf9u+l57mnzH80FVd52OMIekfG3TIpmMA9fOgR3grV4b3bsUOfTFvbb81GozOlrtpwIs347cGY7gtsNfOVFkeYSKpFq4PeSFxeoKeY9riA5p37zAUvHDr5HfdJQ2Fgt1+x2K/IfB0Pwmt4G6coQvdgRkYkuJYm2dbkQN/Ofp9ZBNg2xtasZ9cX9to7FNw8tJ81CWmh71MWKK1hsYT/wJIj9I+IBWdMVcsTdNA3l1YInhuvXta1hD7mopuh340ftoQsYFl+lPTdKzTbq+sTKgQU4VIDKksMkQimx1SsAgG2zK0YE9irA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=davemloft.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r0gHxefuejxy5/6ZoMCuTrQc38czNSeq20vxSOdC1dQ=; b=ta3Y3B0jKCMrCkzfjjNMDNnQ8evMbEyLUlqKx9r4UC/XSO8Ivsyn79yKfch4rRJBEXph5l4p0EQXSDyUQzZQRF5FrWVYzjeeC3QyiqYZYeZ5/9+xoHqOEXSHYDeVoj93ttwopNTFiw6rRKeWVM3Sctw86bjKA0BJj7TDXy4eDAKAQDPgwsOUQ5iAs2TwK6/RNEx0bhQlGf+49RL1XB9/Wq0ra0oKMvkAyGInX4BfuJN3mZZvVAFvN2TH1Pf0vh15Jqdn492JpCFflsjL2BjG1PpXsT4d0Rut0mH/tVzciKCOehMajHidKJtWjepwpDFDXnfiSM1Qmnryk/qhJroe1g== Received: from BY3PR10CA0021.namprd10.prod.outlook.com (2603:10b6:a03:255::26) by CY5PR12MB6645.namprd12.prod.outlook.com (2603:10b6:930:42::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.16; Sun, 9 Feb 2025 10:20:16 +0000 Received: from SJ1PEPF00002321.namprd03.prod.outlook.com (2603:10b6:a03:255:cafe::1) by BY3PR10CA0021.outlook.office365.com (2603:10b6:a03:255::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8398.24 via Frontend Transport; Sun, 9 Feb 2025 10:20:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by SJ1PEPF00002321.mail.protection.outlook.com (10.167.242.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.10 via Frontend Transport; Sun, 9 Feb 2025 10:20:15 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 9 Feb 2025 02:20:12 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Sun, 9 Feb 2025 02:20:12 -0800 Received: from vdi.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.14 via Frontend Transport; Sun, 9 Feb 2025 02:20:07 -0800 From: Tariq Toukan To: "David S. Miller" , Jakub Kicinski , Paolo Abeni , Eric Dumazet , "Andrew Lunn" CC: , Saeed Mahameed , Gal Pressman , Leon Romanovsky , Simon Horman , Donald Hunter , Jiri Pirko , Jonathan Corbet , Leon Romanovsky , Tariq Toukan , Alexei Starovoitov , Daniel Borkmann , "Jesper Dangaard Brouer" , John Fastabend , "Richard Cochran" , , , , , Alexei Lazar Subject: [PATCH net-next 15/15] net/mlx5: XDP, Enable TX side XDP multi-buffer support Date: Sun, 9 Feb 2025 12:17:16 +0200 Message-ID: <20250209101716.112774-16-tariqt@nvidia.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20250209101716.112774-1-tariqt@nvidia.com> References: <20250209101716.112774-1-tariqt@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002321:EE_|CY5PR12MB6645:EE_ X-MS-Office365-Filtering-Correlation-Id: 71062ce8-be56-4393-3256-08dd48f35870 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: rme5fRCW4Xfl4pem0lPzHy5AF/9qyyA84mcpOJAOO+VV7DgfxGYtgXnKtp/R80YhjbNJ43i+K2m78mH1Mcg1NeEdHmDTVRfVnn8Qc934LHyvErWb9DhQ4wRFrVOD9CHhTC8JDo76Wz79ZruasB8nXu52NRKAkfA+vFWR4aFaTrW6lqSUTAeWYbwe4gn05eTZbONi6n8a+zICRbUT2wAJ2JlVQtPsEWze8jms3mR5FeDYo0xFtwPWMCOb0wDyG/HGwfDxcHCKioP7Z8dLdI3EiufvP9Dwe30txYGdrjztJvZvNxNX3yHWkhWD7qukt46vyPj7YNEAoKkcPH/Cc70qO93wTk/ZyX3cW6INEl25j4HCk1SKahJGyAXsihYRJOnzTrDh9dptG0IHQTGkCotMSmfzaTubPA7EHOYKDESqGXJZ+BRT/TmodzoiGj/aW0mA1cGlZ7PTJKTVqwMN/uJVm35SNXnVnjD8SMbyRk/LIIAcmllgOLFS6E1yVLBmXA9TBZe19ZdYv6PhjpNNisthjib9KqYNfPWAbkUkTcje8dIKopX5Fjy7OSnb7ialCkAD2pzxZvbQD0t9MSjZhyOL4sGEg/Huma6HaOkwxszC5mE6kE2vDkhNrf4eW6AvqfhroA4vJV8K8BTZlM6MMJcguODiviE565LY1me9K+YaCkDK9YwL4CigvT9CTkhTGfryaWgzqjd9tzJQXVY053fTZB2lyZ9Sm5i5SEmS+XRA/n9iv2eY5fEn9Wl2UoZxPuEdmnn5z0D4GvEtLFqYuIqQAUGZsrV3ZFzhPNziWKrHV9sqycoWJ92vX7ZCxMZtBWwzQER5EKOp655l6xg3vETjoOdgrQZq+gy8sjLoQuRfZaFb0XeiOPP783g0P7y1LsxpLT7Oyf2pA5yIyUii/FFr9CqY+d2leNfGwgwu9CvXFA7vMVEnM4pLQIG+JfkFeNuTkx/avUtcbgp7mxlBeKIlxqQMM389ktjAKwSBTwZZM7qTk7dTj23dOIYIsbpd+Tn9mIGCSruQTVEPY6JtIyNL2GezQCZ2Olxu74gMJfpZr+nlhWtiR/0Pdo2RUGh8J1V7KqOooSkGk8dvrNNkv6A0hf2t2/h+ByoFqpKs529b+7+tEJlGmY9LAYSvwTJXvDlTZaRSN7lICPr3fIVySAVz8VhH9wyjIwqrGUrWrltDqYtNGXCb2idcn4C/LXbSDmXKeVbltHP/33ZK79+Mtaf65zoJbCiJ/6Cp1sDZRV5f3Am90pupXI6GBj3KeAwFKCuSAPnRNnef7O0NlIZDJIGp6B/bDZPQtFZ1HoLQ+gJiUiX/FGaBV3dMShT5F2Ft8g7gH6mXnbT+5Oc6sFPbDzPAeNPhZnEPosff6r7DsmUWabX0vXJdD3i3iOEEJPTV5KFX9QBo6xpTQs9liP1lSuZEdVRgnppvT5pma706BfR0O0cZbjBxNWDJfYmVvwwPckNCtlbFzhJNNaLRdUjy9nBcI9KiH/tEVxf8BY2rRaRbLJg= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2025 10:20:15.4166 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 71062ce8-be56-4393-3256-08dd48f35870 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002321.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6645 From: Alexei Lazar In XDP scenarios, fragmented packets can occur if the MTU is larger than the page size, even when the packet size fits within the linear part. If XDP multi-buffer support is disabled, the fragmented part won't be handled in the TX flow, leading to packet drops. Since XDP multi-buffer support is always available, this commit removes the conditional check for enabling it. This ensures that XDP multi-buffer support is always enabled, regardless of the `is_xdp_mb` parameter, and guarantees the handling of fragmented packets in such scenarios. Signed-off-by: Alexei Lazar Signed-off-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 - .../ethernet/mellanox/mlx5/core/en/params.c | 1 - .../ethernet/mellanox/mlx5/core/en/params.h | 1 - .../mellanox/mlx5/core/en/reporter_tx.c | 1 - .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 49 ++++++++----------- .../net/ethernet/mellanox/mlx5/core/en_main.c | 29 ----------- 6 files changed, 21 insertions(+), 61 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 534fdd27c8de..769e683f2488 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -384,7 +384,6 @@ enum { MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE, MLX5E_SQ_STATE_PENDING_XSK_TX, MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC, - MLX5E_SQ_STATE_XDP_MULTIBUF, MLX5E_NUM_SQ_STATES, /* Must be kept last */ }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c index e37d4c202bba..aa36670d9a36 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c @@ -1247,7 +1247,6 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev, mlx5e_build_sq_param_common(mdev, param); MLX5_SET(wq, wq, log_wq_sz, params->log_sq_size); param->is_mpw = MLX5E_GET_PFLAG(params, MLX5E_PFLAG_XDP_TX_MPWQE); - param->is_xdp_mb = !mlx5e_rx_is_linear_skb(mdev, params, xsk); mlx5e_build_tx_cq_param(mdev, params, ¶m->cqp); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h index 3f8986f9d862..bd5877acc5b1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h @@ -33,7 +33,6 @@ struct mlx5e_sq_param { struct mlx5_wq_param wq; bool is_mpw; bool is_tls; - bool is_xdp_mb; u16 stop_room; }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c index 09433b91be17..532c7fa94d17 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c @@ -16,7 +16,6 @@ static const char * const sq_sw_state_type_name[] = { [MLX5E_SQ_STATE_VLAN_NEED_L2_INLINE] = "vlan_need_l2_inline", [MLX5E_SQ_STATE_PENDING_XSK_TX] = "pending_xsk_tx", [MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC] = "pending_tls_rx_resync", - [MLX5E_SQ_STATE_XDP_MULTIBUF] = "xdp_multibuf", }; static int mlx5e_wait_for_sq_flush(struct mlx5e_txqsq *sq) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index 3cc4d55613bf..6f3094a479e1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -546,6 +546,7 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, bool inline_ok; bool linear; u16 pi; + int i; struct mlx5e_xdpsq_stats *stats = sq->stats; @@ -612,41 +613,33 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_SEND); - if (test_bit(MLX5E_SQ_STATE_XDP_MULTIBUF, &sq->state)) { - int i; - - memset(&cseg->trailer, 0, sizeof(cseg->trailer)); - memset(eseg, 0, sizeof(*eseg) - sizeof(eseg->trailer)); - - eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz); + memset(&cseg->trailer, 0, sizeof(cseg->trailer)); + memset(eseg, 0, sizeof(*eseg) - sizeof(eseg->trailer)); - for (i = 0; i < num_frags; i++) { - skb_frag_t *frag = &xdptxdf->sinfo->frags[i]; - dma_addr_t addr; + eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz); - addr = xdptxdf->dma_arr ? xdptxdf->dma_arr[i] : - page_pool_get_dma_addr(skb_frag_page(frag)) + - skb_frag_off(frag); + for (i = 0; i < num_frags; i++) { + skb_frag_t *frag = &xdptxdf->sinfo->frags[i]; + dma_addr_t addr; - dseg->addr = cpu_to_be64(addr); - dseg->byte_count = cpu_to_be32(skb_frag_size(frag)); - dseg->lkey = sq->mkey_be; - dseg++; - } + addr = xdptxdf->dma_arr ? xdptxdf->dma_arr[i] : + page_pool_get_dma_addr(skb_frag_page(frag)) + + skb_frag_off(frag); - cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt); + dseg->addr = cpu_to_be64(addr); + dseg->byte_count = cpu_to_be32(skb_frag_size(frag)); + dseg->lkey = sq->mkey_be; + dseg++; + } - sq->db.wqe_info[pi] = (struct mlx5e_xdp_wqe_info) { - .num_wqebbs = num_wqebbs, - .num_pkts = 1, - }; + cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt); - sq->pc += num_wqebbs; - } else { - cseg->fm_ce_se = 0; + sq->db.wqe_info[pi] = (struct mlx5e_xdp_wqe_info) { + .num_wqebbs = num_wqebbs, + .num_pkts = 1, + }; - sq->pc++; - } + sq->pc += num_wqebbs; xsk_tx_metadata_request(meta, &mlx5e_xsk_tx_metadata_ops, eseg); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 2fdc86432ac0..5d5e7b19c396 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -2023,41 +2023,12 @@ int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params, csp.min_inline_mode = sq->min_inline_mode; set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state); - if (param->is_xdp_mb) - set_bit(MLX5E_SQ_STATE_XDP_MULTIBUF, &sq->state); - err = mlx5e_create_sq_rdy(c->mdev, param, &csp, 0, &sq->sqn); if (err) goto err_free_xdpsq; mlx5e_set_xmit_fp(sq, param->is_mpw); - if (!param->is_mpw && !test_bit(MLX5E_SQ_STATE_XDP_MULTIBUF, &sq->state)) { - unsigned int ds_cnt = MLX5E_TX_WQE_EMPTY_DS_COUNT + 1; - unsigned int inline_hdr_sz = 0; - int i; - - if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) { - inline_hdr_sz = MLX5E_XDP_MIN_INLINE; - ds_cnt++; - } - - /* Pre initialize fixed WQE fields */ - for (i = 0; i < mlx5_wq_cyc_get_size(&sq->wq); i++) { - struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(&sq->wq, i); - struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl; - struct mlx5_wqe_eth_seg *eseg = &wqe->eth; - - sq->db.wqe_info[i] = (struct mlx5e_xdp_wqe_info) { - .num_wqebbs = 1, - .num_pkts = 1, - }; - - cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt); - eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz); - } - } - return 0; err_free_xdpsq: