From patchwork Mon Feb 6 15:36:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Horman X-Patchwork-Id: 13130186 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8169AC05027 for ; Mon, 6 Feb 2023 15:40:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230203AbjBFPjt (ORCPT ); Mon, 6 Feb 2023 10:39:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230309AbjBFPjg (ORCPT ); Mon, 6 Feb 2023 10:39:36 -0500 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2116.outbound.protection.outlook.com [40.107.92.116]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19618A5E3 for ; Mon, 6 Feb 2023 07:38:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ly6HNpjjQdqRIjoQj3NYS0c3/SFMgJ7qDFmjWlovTEJr8ZyBhAyJxvAg1FK0OZ0QdpU1Y59Ur7z143Uo8YOzMvrnFlLGHshsQdgwGWZD2QF6ygMzhwA89aa4ZQA7vE9JvhhZvFYnveT+fl3Erv4Oa1DynpnM+etGGU8jBrSVoEvS9PE7obPmq/DjNcuU3sgEZo4CEGofHMq7aYsgP/Ij1UmVZkpvhtrA0H1TUq7Fop0fFUUBcxZYKznyvBrABo/4T7v+4y90cr7xWcG1+qIQUpX7gvZKxW5A/0+pN7MEgWIcmFUyGX/7lLAqpEy3qIdoRyjFaiCkrJkAnP09etTGVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kc0+NPqnKyMVCWVyOOyMFsJjWBC+FhpOZqTNhNQU4z4=; b=LaB9G5tLQpnGEmvhrG5KhunLQjwH9TOCPfGSeW9s1NV6dCl1o19dNtJ8u4ADm6kYSRkAwIKbXSr8o86EwGSWZZXQMM+6nb9OUOqd3y0Q37OXpPcUzDgArVLh1qzcHLXMAVU8fXN8xGMRen/aToAgxCmCynpONmt2nmMcZZUq6l24IY5KsB5IT9kD3s2VuAhOOPiO91aygHdSh0TEkTPHeCEe4bqmhlBzLWmEk30yZi/ryxAlNbnaOk59SXcdBb+TFnLbuwMExWxnqFDVfiMbgLcTGU2yPgnvhZnH4sEDLGfUUOm94ZsBrpHnn9JioxXzfPYTNNnTHt9pb3AqvT1X0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kc0+NPqnKyMVCWVyOOyMFsJjWBC+FhpOZqTNhNQU4z4=; b=Ujmg/JZefsIam3KU1NmoAHtRj9u6V7LZIAELMNizha05reuvPtPec9I0/IXYvY1u/CBZirUIpGc2SM7oxktvKUfRfM/AAgqv3fRXEW7mgtFv2LnloL21dTtfJIzR96avzGgZH1N29011mB/j9uJZ5u99JhxdcmLRGvFeCE5pHxg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) by SA1PR13MB4912.namprd13.prod.outlook.com (2603:10b6:806:184::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 15:37:20 +0000 Received: from PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c]) by PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c%3]) with mapi id 15.20.6064.034; Mon, 6 Feb 2023 15:37:20 +0000 From: Simon Horman To: David Miller , Jakub Kicinski , Paolo Abeni Cc: Michael Chan , Andy Gospodarek , Gal Pressman , Saeed Mahameed , Jesse Brandeburg , Tony Nguyen , Edward Cree , Vladimir Oltean , Andrew Lunn , Fei Qin , netdev@vger.kernel.org, oss-drivers@corigine.com Subject: [PATCH/RFC net-next 1/2] devlink: expose port function commands to assign VFs to multiple netdevs Date: Mon, 6 Feb 2023 16:36:02 +0100 Message-Id: <20230206153603.2801791-2-simon.horman@corigine.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206153603.2801791-1-simon.horman@corigine.com> References: <20230206153603.2801791-1-simon.horman@corigine.com> X-ClientProxiedBy: AM4PR0202CA0013.eurprd02.prod.outlook.com (2603:10a6:200:89::23) To PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB4842:EE_|SA1PR13MB4912:EE_ X-MS-Office365-Filtering-Correlation-Id: 198314a2-f979-4cdb-edf3-08db0858088f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h0Czpa2tBpUFhCF0t4GyHbGZb3SG6fY8L17QAihHNRdyJleyPmvQJKpKhtwi6PZgt3aD+OOezLP6cbSfgT1uhxrnEuWigi8Ra3XepLL5KxOrDe3QBy27+YdlkxdotTUte2fJxxIO9lPWJVWyGj+ekcMq5p5erYcR+R4KtWqqTNrAg80CAcO56jUWG3/Lr3sLUhNkN87TOe3y36sLh6md/tHbeEoTSlpK1S9HOev7iBtalSOqOqmqWP8oRyyi1W8rw3AkEQ1CoS3duh2849IcqB4sMnvNBtXvYMf4/LtJshN6rEQHrSpDAhIQlzHVRwEZPCSVhZehpGHrkYoV9PKfrcHfcM/w67d76P0GnBqdDOzrG37/tzBjb/gpUMpl3JnVH6eMCguULYD647OFp3Ax+B2vkHjGUV0ZSJmswfY98mOiOE4P3Mt2WTSn1WhGMXCezvXbvEAkZobuRxwQiKR7bFCm/weWNRkrKmvor4Aoz6bCNxWnN3hDf6Xk6a1j6RF1Ksq2+OH0cGuTCqa4fkjT9caEFikuntICXOeCKAN0RkZJh+JXqZAB8Kx2jz+L8DBp/yQHBNZEmcKxvfX8S9bhNuUGQMozXEceW8q0CmOJ58cuRoTerHS4/OQ3c8y/vgfCt6YKqLcrTBwklLp46Bx+8g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR13MB4842.namprd13.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(136003)(396003)(39840400004)(346002)(376002)(451199018)(478600001)(2906002)(6486002)(66946007)(186003)(6512007)(66476007)(6666004)(107886003)(41300700001)(1076003)(6506007)(8676002)(66556008)(52116002)(4326008)(7416002)(316002)(54906003)(5660300002)(38100700002)(110136005)(8936002)(44832011)(86362001)(36756003)(83380400001)(2616005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?ZqnJfmbn/SUsCYwoC6ey7G+H5MSK?= =?utf-8?q?gWZzf/dvAhmik/Z057Bx2Vc2Z+vEmRBaB3Dvq6vnmtflGDSfy9qfCgDjY2iclxImE?= =?utf-8?q?+nLGRXow8n2KsIxDcBvLqybniQ3wTCQgm+veE1b78jBtNzMdRyREGZ9qMAfPrtguE?= =?utf-8?q?G//AhSDst2xoS2FniEEug7LGDdgnrrE1WhL6NOCVAxypvjqry6flEbL1msQhXyqEG?= =?utf-8?q?mzVJsuIT42z7geMMiMBce2yVnn64ejDKjPx5hdmxb7TikwH/UxakjzyYQYwho+M/o?= =?utf-8?q?7+zyQwbSxG3kcXtJm+hNIXf+dGAfZz5/mbg8JGnZiKCuZsnqdumkaSicv3p7LXWDO?= =?utf-8?q?8VNEtk/pOl/WKLEeGoYH9+6SNKecm83IIXgR114ezU/mQo5lHVDyGW1QKHkXq9jck?= =?utf-8?q?aC3SK0doxJMlH0+0bmBMw6I6wFlT9F9p/OxWQEYVo49LynmU83dW7b5NrKhYsQqWJ?= =?utf-8?q?PqzI519iGYe7o43EPKiqO533V+GrezeWXg23r5tuuoqYBVwo4nSboYvX7de7VIUW1?= =?utf-8?q?dSDRvGbjsbrM4i+vzF3nM3RJYG4E9q9vHHAjtSpCYDpKVfVRrhTybmay6fPoVQ36T?= =?utf-8?q?6JGb/Vj6vVxvXTUXpi+9OZGQ6PhajDYZ7SSwxWRYtlJpiHEw1hg46gF9y2Srb1ZIH?= =?utf-8?q?aHk2fE6pcIzDUvha9UPWSqKzIYwmuY3nNeIzfDaRPR435UceWzu2BayDnh6YpVzVz?= =?utf-8?q?TlaMs9ND5fMmC3+GDVhSBJlYrNr6fVNnhTZjYJPmpjUVSGAZuz+Yjf1XKgI5ppJ74?= =?utf-8?q?ilEhk3MEJIHIsMM3DzXjMtaA52nS3syo58osBFy3HFLTcUY5OuZKZX9aAIfR4/xOG?= =?utf-8?q?4nlxKsDM/TqaRG9ILU5v9MhYjGcWEpYQxb6MwF2TFn7GJN//boVwIzXonENhEyR6f?= =?utf-8?q?ahlB6S2RKIscMsFqQEzE41yNcGZbPhgBhbiPCUzKtxlDBmN34hyMU+U2S9h/eMh5J?= =?utf-8?q?DJH5Oi5wnagwGflpv9+AuG9vEh5WY+Hx8asBQ/L7ZUmqY1+nx4E8K7zTIOSn6JgaD?= =?utf-8?q?jm4wh3WCJGojG2lvw0XwXiBL66ke/v33xBW826Dt/s1DxnyTLYga24lyfRWcLA8jE?= =?utf-8?q?oCROnMd0iRTelcGlUA+Pq4SHPkdVzvdOzf9gmIgTNCWOGTWQc2hzlWtcPIejYV6CD?= =?utf-8?q?C8BRem7VToIL9iikR5q8a/63he7jK34yudNbRm0AcY+JRNYUnhhsQkJSNxClB4vlY?= =?utf-8?q?uhjegpcBzHdtyWTs4hdJDIpiN+ietkKzr/DOJj7froIpxAA4B7j3iJpaquyp8Khuz?= =?utf-8?q?2+1untPr1Pjt6M244ieRQqRKg0G5z28kIJTfYlG3yem65/bCSA+6R2EweL9G6lpbc?= =?utf-8?q?rJlVq8d4f9PI5vKsP1BnlVbt2dz8JsRCIK29sodWMdyCdrVcPLSaY8jK5rawUWJay?= =?utf-8?q?qKoOq4gcmcZmTtyteyGeZUlb7y0ZPyStwPOipvkzKq61Q0Px0ocw5MTeRJ8u1TN72?= =?utf-8?q?axAEterrubYCTxeMl6rwA5cAgNRqQmsqoTOJcMmm/Gqa/918OB7otPIh8MhUrb/+v?= =?utf-8?q?VMHMhP1GdL56TYn9iicl5yp0JEOJnHHrqv064y9ZmfeZK9ndSnPwptRFkoSxhxtEX?= =?utf-8?q?bEyn2DQNk639AHaf4AGf6EHWba4satuAg=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 198314a2-f979-4cdb-edf3-08db0858088f X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB4842.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 15:37:19.9038 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cIxz3TTDHiwBVuY86Mjh0PlQqc6uLtyUy1vgNcBEruCMFfL8edxJknwSs1ZfWQBr/W0q593JrAOVsksRU4Y86nMOk0w1F4lgVU78htCjfq0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR13MB4912 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC From: Fei Qin Multiple physical ports of the same NIC may share the single PCI address. In some cases, assigning VFs to different physical ports can be demanded, especially under high-traffic scenario. Load balancing can be realized in virtualised use¬cases through distributing packets between different physical ports with LAGs of VFs which are assigned to those physical ports. This patch adds new attribute "vf_count" to 'devlink port function' API which only can be shown and configured under devlink ports with flavor "DEVLINK_PORT_FLAVOUR_PHYSICAL". e.g. $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 0 $ devlink port function set pci/0000:82:00.0/0 vf_count 3 $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 3 Signed-off-by: Fei Qin Signed-off-by: Simon Horman --- .../networking/devlink/devlink-port.rst | 24 +++++++ include/net/devlink.h | 21 ++++++ include/uapi/linux/devlink.h | 1 + net/devlink/leftover.c | 65 +++++++++++++++++++ 4 files changed, 111 insertions(+) diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst index 3da590953ce8..5c3996bce6d9 100644 --- a/Documentation/networking/devlink/devlink-port.rst +++ b/Documentation/networking/devlink/devlink-port.rst @@ -128,6 +128,9 @@ Users may also set the RoCE capability of the function using Users may also set the function as migratable using 'devlink port function set migratable' command. +Users may also assign VFs to physical ports using +'devlink port function set vf_count' command. + Function attributes =================== @@ -240,6 +243,27 @@ Attach VF to the VM. Start the VM. Perform live migration. + +VF assignment setup +--------------------------- +In some cases, NICs could have multiple physical ports per PF. Users can assign VFs to +different ports. + +- Get count of VFs assigned to physical port:: + + $ devlink port show pci/0000:82:00.0/0 + pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 + function: + vf_count 0 + +- Set count of VFs assigned to physical port:: + $ devlink port function set pci/0000:82:00.0/0 vf_count 3 + + $ devlink port show pci/0000:82:00.0/0 + pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 + function: + vf_count 3 + Subfunction ============ diff --git a/include/net/devlink.h b/include/net/devlink.h index 2e85a5970a32..3e98fa3d251f 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -1491,6 +1491,27 @@ struct devlink_ops { int (*port_fn_migratable_set)(struct devlink_port *devlink_port, bool enable, struct netlink_ext_ack *extack); + + /** + * @port_fn_vf_count_get: Port function's VF count get function + * + * Get assigned VF count of a function managed by the devlink port, + * should only be used for DEVLINK_PORT_FLAVOUR_PHYSICAL. + * Return -EOPNOTSUPP if port function vf_count setup is not supported. + */ + int (*port_fn_vf_count_get)(struct devlink_port *port, u16 *vf_count, + struct netlink_ext_ack *extack); + + /** + * @port_fn_vf_count_set: Port function's VF count set function + * + * Set assigned VF count of a function managed by the devlink port, + * should only be used for DEVLINK_PORT_FLAVOUR_PHYSICAL. + * Return -EOPNOTSUPP if port function vf_count setup is not supported. + */ + int (*port_fn_vf_count_set)(struct devlink_port *port, u16 vf_count, + struct netlink_ext_ack *extack); + /** * port_new() - Add a new port function of a specified flavor * @devlink: Devlink instance diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 3782d4219ac9..774e17f6100b 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -676,6 +676,7 @@ enum devlink_port_function_attr { DEVLINK_PORT_FN_ATTR_STATE, /* u8 */ DEVLINK_PORT_FN_ATTR_OPSTATE, /* u8 */ DEVLINK_PORT_FN_ATTR_CAPS, /* bitfield32 */ + DEVLINK_PORT_FN_ATTR_VF_COUNT, /* u16 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1 diff --git a/net/devlink/leftover.c b/net/devlink/leftover.c index 97d30ea98b00..6dac8b562232 100644 --- a/net/devlink/leftover.c +++ b/net/devlink/leftover.c @@ -141,6 +141,7 @@ static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK), + [DEVLINK_PORT_FN_ATTR_VF_COUNT] = { .type = NLA_U16 }, }; #define ASSERT_DEVLINK_PORT_REGISTERED(devlink_port) \ @@ -520,6 +521,35 @@ static int devlink_port_fn_caps_fill(const struct devlink_ops *ops, return 0; } +static int devlink_port_fn_vf_count_fill(const struct devlink_ops *ops, + struct devlink_port *devlink_port, + struct sk_buff *msg, + struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u16 vf_count; + int err; + + if (!ops->port_fn_vf_count_get || + devlink_port->attrs.flavour != DEVLINK_PORT_FLAVOUR_PHYSICAL) + return 0; + + err = ops->port_fn_vf_count_get(devlink_port, &vf_count, extack); + if (err) { + if (err == -EOPNOTSUPP) + return 0; + return err; + } + + err = nla_put_u16(msg, DEVLINK_PORT_FN_ATTR_VF_COUNT, vf_count); + if (err) + return err; + + *msg_updated = true; + + return 0; +} + static int devlink_sb_tc_index_get_from_info(struct devlink_sb *devlink_sb, struct genl_info *info, @@ -871,6 +901,16 @@ static int devlink_port_fn_caps_set(struct devlink_port *devlink_port, return 0; } +static int devlink_port_fn_vf_count_set(struct devlink_port *devlink_port, + const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const struct devlink_ops *ops = devlink_port->devlink->ops; + u16 vf_count = nla_get_u16(attr); + + return ops->port_fn_vf_count_set(devlink_port, vf_count, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *port, struct netlink_ext_ack *extack) @@ -893,6 +933,11 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por &msg_updated); if (err) goto out; + + err = devlink_port_fn_vf_count_fill(ops, port, msg, extack, &msg_updated); + if (err) + goto out; + err = devlink_port_fn_state_fill(ops, port, msg, extack, &msg_updated); out: if (err || !msg_updated) @@ -1219,6 +1264,19 @@ static int devlink_port_function_validate(struct devlink_port *devlink_port, "Function does not support state setting"); return -EOPNOTSUPP; } + attr = tb[DEVLINK_PORT_FN_ATTR_VF_COUNT]; + if (attr) { + if (!ops->port_fn_vf_count_set) { + NL_SET_ERR_MSG_ATTR(extack, attr, + "Function doesn't support VF assignment"); + return -EOPNOTSUPP; + } + if (devlink_port->attrs.flavour != DEVLINK_PORT_FLAVOUR_PHYSICAL) { + NL_SET_ERR_MSG_ATTR(extack, attr, + "VFs assignment supported for physical ports only"); + return -EOPNOTSUPP; + } + } attr = tb[DEVLINK_PORT_FN_ATTR_CAPS]; if (attr) { struct nla_bitfield32 caps; @@ -1278,6 +1336,13 @@ static int devlink_port_function_set(struct devlink_port *port, return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_VF_COUNT]; + if (attr) { + err = devlink_port_fn_vf_count_set(port, attr, extack); + if (err) + return err; + } + /* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.