From patchwork Mon Feb 6 15:36:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Simon Horman X-Patchwork-Id: 13130186 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8169AC05027 for ; Mon, 6 Feb 2023 15:40:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230203AbjBFPjt (ORCPT ); Mon, 6 Feb 2023 10:39:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230309AbjBFPjg (ORCPT ); Mon, 6 Feb 2023 10:39:36 -0500 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2116.outbound.protection.outlook.com [40.107.92.116]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19618A5E3 for ; Mon, 6 Feb 2023 07:38:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ly6HNpjjQdqRIjoQj3NYS0c3/SFMgJ7qDFmjWlovTEJr8ZyBhAyJxvAg1FK0OZ0QdpU1Y59Ur7z143Uo8YOzMvrnFlLGHshsQdgwGWZD2QF6ygMzhwA89aa4ZQA7vE9JvhhZvFYnveT+fl3Erv4Oa1DynpnM+etGGU8jBrSVoEvS9PE7obPmq/DjNcuU3sgEZo4CEGofHMq7aYsgP/Ij1UmVZkpvhtrA0H1TUq7Fop0fFUUBcxZYKznyvBrABo/4T7v+4y90cr7xWcG1+qIQUpX7gvZKxW5A/0+pN7MEgWIcmFUyGX/7lLAqpEy3qIdoRyjFaiCkrJkAnP09etTGVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kc0+NPqnKyMVCWVyOOyMFsJjWBC+FhpOZqTNhNQU4z4=; b=LaB9G5tLQpnGEmvhrG5KhunLQjwH9TOCPfGSeW9s1NV6dCl1o19dNtJ8u4ADm6kYSRkAwIKbXSr8o86EwGSWZZXQMM+6nb9OUOqd3y0Q37OXpPcUzDgArVLh1qzcHLXMAVU8fXN8xGMRen/aToAgxCmCynpONmt2nmMcZZUq6l24IY5KsB5IT9kD3s2VuAhOOPiO91aygHdSh0TEkTPHeCEe4bqmhlBzLWmEk30yZi/ryxAlNbnaOk59SXcdBb+TFnLbuwMExWxnqFDVfiMbgLcTGU2yPgnvhZnH4sEDLGfUUOm94ZsBrpHnn9JioxXzfPYTNNnTHt9pb3AqvT1X0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kc0+NPqnKyMVCWVyOOyMFsJjWBC+FhpOZqTNhNQU4z4=; b=Ujmg/JZefsIam3KU1NmoAHtRj9u6V7LZIAELMNizha05reuvPtPec9I0/IXYvY1u/CBZirUIpGc2SM7oxktvKUfRfM/AAgqv3fRXEW7mgtFv2LnloL21dTtfJIzR96avzGgZH1N29011mB/j9uJZ5u99JhxdcmLRGvFeCE5pHxg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) by SA1PR13MB4912.namprd13.prod.outlook.com (2603:10b6:806:184::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 15:37:20 +0000 Received: from PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c]) by PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c%3]) with mapi id 15.20.6064.034; Mon, 6 Feb 2023 15:37:20 +0000 From: Simon Horman To: David Miller , Jakub Kicinski , Paolo Abeni Cc: Michael Chan , Andy Gospodarek , Gal Pressman , Saeed Mahameed , Jesse Brandeburg , Tony Nguyen , Edward Cree , Vladimir Oltean , Andrew Lunn , Fei Qin , netdev@vger.kernel.org, oss-drivers@corigine.com Subject: [PATCH/RFC net-next 1/2] devlink: expose port function commands to assign VFs to multiple netdevs Date: Mon, 6 Feb 2023 16:36:02 +0100 Message-Id: <20230206153603.2801791-2-simon.horman@corigine.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206153603.2801791-1-simon.horman@corigine.com> References: <20230206153603.2801791-1-simon.horman@corigine.com> X-ClientProxiedBy: AM4PR0202CA0013.eurprd02.prod.outlook.com (2603:10a6:200:89::23) To PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB4842:EE_|SA1PR13MB4912:EE_ X-MS-Office365-Filtering-Correlation-Id: 198314a2-f979-4cdb-edf3-08db0858088f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h0Czpa2tBpUFhCF0t4GyHbGZb3SG6fY8L17QAihHNRdyJleyPmvQJKpKhtwi6PZgt3aD+OOezLP6cbSfgT1uhxrnEuWigi8Ra3XepLL5KxOrDe3QBy27+YdlkxdotTUte2fJxxIO9lPWJVWyGj+ekcMq5p5erYcR+R4KtWqqTNrAg80CAcO56jUWG3/Lr3sLUhNkN87TOe3y36sLh6md/tHbeEoTSlpK1S9HOev7iBtalSOqOqmqWP8oRyyi1W8rw3AkEQ1CoS3duh2849IcqB4sMnvNBtXvYMf4/LtJshN6rEQHrSpDAhIQlzHVRwEZPCSVhZehpGHrkYoV9PKfrcHfcM/w67d76P0GnBqdDOzrG37/tzBjb/gpUMpl3JnVH6eMCguULYD647OFp3Ax+B2vkHjGUV0ZSJmswfY98mOiOE4P3Mt2WTSn1WhGMXCezvXbvEAkZobuRxwQiKR7bFCm/weWNRkrKmvor4Aoz6bCNxWnN3hDf6Xk6a1j6RF1Ksq2+OH0cGuTCqa4fkjT9caEFikuntICXOeCKAN0RkZJh+JXqZAB8Kx2jz+L8DBp/yQHBNZEmcKxvfX8S9bhNuUGQMozXEceW8q0CmOJ58cuRoTerHS4/OQ3c8y/vgfCt6YKqLcrTBwklLp46Bx+8g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR13MB4842.namprd13.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(136003)(396003)(39840400004)(346002)(376002)(451199018)(478600001)(2906002)(6486002)(66946007)(186003)(6512007)(66476007)(6666004)(107886003)(41300700001)(1076003)(6506007)(8676002)(66556008)(52116002)(4326008)(7416002)(316002)(54906003)(5660300002)(38100700002)(110136005)(8936002)(44832011)(86362001)(36756003)(83380400001)(2616005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?ZqnJfmbn/SUsCYwoC6ey7G+H5MSK?= =?utf-8?q?gWZzf/dvAhmik/Z057Bx2Vc2Z+vEmRBaB3Dvq6vnmtflGDSfy9qfCgDjY2iclxImE?= =?utf-8?q?+nLGRXow8n2KsIxDcBvLqybniQ3wTCQgm+veE1b78jBtNzMdRyREGZ9qMAfPrtguE?= =?utf-8?q?G//AhSDst2xoS2FniEEug7LGDdgnrrE1WhL6NOCVAxypvjqry6flEbL1msQhXyqEG?= =?utf-8?q?mzVJsuIT42z7geMMiMBce2yVnn64ejDKjPx5hdmxb7TikwH/UxakjzyYQYwho+M/o?= =?utf-8?q?7+zyQwbSxG3kcXtJm+hNIXf+dGAfZz5/mbg8JGnZiKCuZsnqdumkaSicv3p7LXWDO?= =?utf-8?q?8VNEtk/pOl/WKLEeGoYH9+6SNKecm83IIXgR114ezU/mQo5lHVDyGW1QKHkXq9jck?= =?utf-8?q?aC3SK0doxJMlH0+0bmBMw6I6wFlT9F9p/OxWQEYVo49LynmU83dW7b5NrKhYsQqWJ?= =?utf-8?q?PqzI519iGYe7o43EPKiqO533V+GrezeWXg23r5tuuoqYBVwo4nSboYvX7de7VIUW1?= =?utf-8?q?dSDRvGbjsbrM4i+vzF3nM3RJYG4E9q9vHHAjtSpCYDpKVfVRrhTybmay6fPoVQ36T?= =?utf-8?q?6JGb/Vj6vVxvXTUXpi+9OZGQ6PhajDYZ7SSwxWRYtlJpiHEw1hg46gF9y2Srb1ZIH?= =?utf-8?q?aHk2fE6pcIzDUvha9UPWSqKzIYwmuY3nNeIzfDaRPR435UceWzu2BayDnh6YpVzVz?= =?utf-8?q?TlaMs9ND5fMmC3+GDVhSBJlYrNr6fVNnhTZjYJPmpjUVSGAZuz+Yjf1XKgI5ppJ74?= =?utf-8?q?ilEhk3MEJIHIsMM3DzXjMtaA52nS3syo58osBFy3HFLTcUY5OuZKZX9aAIfR4/xOG?= =?utf-8?q?4nlxKsDM/TqaRG9ILU5v9MhYjGcWEpYQxb6MwF2TFn7GJN//boVwIzXonENhEyR6f?= =?utf-8?q?ahlB6S2RKIscMsFqQEzE41yNcGZbPhgBhbiPCUzKtxlDBmN34hyMU+U2S9h/eMh5J?= =?utf-8?q?DJH5Oi5wnagwGflpv9+AuG9vEh5WY+Hx8asBQ/L7ZUmqY1+nx4E8K7zTIOSn6JgaD?= =?utf-8?q?jm4wh3WCJGojG2lvw0XwXiBL66ke/v33xBW826Dt/s1DxnyTLYga24lyfRWcLA8jE?= =?utf-8?q?oCROnMd0iRTelcGlUA+Pq4SHPkdVzvdOzf9gmIgTNCWOGTWQc2hzlWtcPIejYV6CD?= =?utf-8?q?C8BRem7VToIL9iikR5q8a/63he7jK34yudNbRm0AcY+JRNYUnhhsQkJSNxClB4vlY?= =?utf-8?q?uhjegpcBzHdtyWTs4hdJDIpiN+ietkKzr/DOJj7froIpxAA4B7j3iJpaquyp8Khuz?= =?utf-8?q?2+1untPr1Pjt6M244ieRQqRKg0G5z28kIJTfYlG3yem65/bCSA+6R2EweL9G6lpbc?= =?utf-8?q?rJlVq8d4f9PI5vKsP1BnlVbt2dz8JsRCIK29sodWMdyCdrVcPLSaY8jK5rawUWJay?= =?utf-8?q?qKoOq4gcmcZmTtyteyGeZUlb7y0ZPyStwPOipvkzKq61Q0Px0ocw5MTeRJ8u1TN72?= =?utf-8?q?axAEterrubYCTxeMl6rwA5cAgNRqQmsqoTOJcMmm/Gqa/918OB7otPIh8MhUrb/+v?= =?utf-8?q?VMHMhP1GdL56TYn9iicl5yp0JEOJnHHrqv064y9ZmfeZK9ndSnPwptRFkoSxhxtEX?= =?utf-8?q?bEyn2DQNk639AHaf4AGf6EHWba4satuAg=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 198314a2-f979-4cdb-edf3-08db0858088f X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB4842.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 15:37:19.9038 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cIxz3TTDHiwBVuY86Mjh0PlQqc6uLtyUy1vgNcBEruCMFfL8edxJknwSs1ZfWQBr/W0q593JrAOVsksRU4Y86nMOk0w1F4lgVU78htCjfq0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR13MB4912 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC From: Fei Qin Multiple physical ports of the same NIC may share the single PCI address. In some cases, assigning VFs to different physical ports can be demanded, especially under high-traffic scenario. Load balancing can be realized in virtualised use¬cases through distributing packets between different physical ports with LAGs of VFs which are assigned to those physical ports. This patch adds new attribute "vf_count" to 'devlink port function' API which only can be shown and configured under devlink ports with flavor "DEVLINK_PORT_FLAVOUR_PHYSICAL". e.g. $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 0 $ devlink port function set pci/0000:82:00.0/0 vf_count 3 $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 3 Signed-off-by: Fei Qin Signed-off-by: Simon Horman --- .../networking/devlink/devlink-port.rst | 24 +++++++ include/net/devlink.h | 21 ++++++ include/uapi/linux/devlink.h | 1 + net/devlink/leftover.c | 65 +++++++++++++++++++ 4 files changed, 111 insertions(+) diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst index 3da590953ce8..5c3996bce6d9 100644 --- a/Documentation/networking/devlink/devlink-port.rst +++ b/Documentation/networking/devlink/devlink-port.rst @@ -128,6 +128,9 @@ Users may also set the RoCE capability of the function using Users may also set the function as migratable using 'devlink port function set migratable' command. +Users may also assign VFs to physical ports using +'devlink port function set vf_count' command. + Function attributes =================== @@ -240,6 +243,27 @@ Attach VF to the VM. Start the VM. Perform live migration. + +VF assignment setup +--------------------------- +In some cases, NICs could have multiple physical ports per PF. Users can assign VFs to +different ports. + +- Get count of VFs assigned to physical port:: + + $ devlink port show pci/0000:82:00.0/0 + pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 + function: + vf_count 0 + +- Set count of VFs assigned to physical port:: + $ devlink port function set pci/0000:82:00.0/0 vf_count 3 + + $ devlink port show pci/0000:82:00.0/0 + pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 + function: + vf_count 3 + Subfunction ============ diff --git a/include/net/devlink.h b/include/net/devlink.h index 2e85a5970a32..3e98fa3d251f 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -1491,6 +1491,27 @@ struct devlink_ops { int (*port_fn_migratable_set)(struct devlink_port *devlink_port, bool enable, struct netlink_ext_ack *extack); + + /** + * @port_fn_vf_count_get: Port function's VF count get function + * + * Get assigned VF count of a function managed by the devlink port, + * should only be used for DEVLINK_PORT_FLAVOUR_PHYSICAL. + * Return -EOPNOTSUPP if port function vf_count setup is not supported. + */ + int (*port_fn_vf_count_get)(struct devlink_port *port, u16 *vf_count, + struct netlink_ext_ack *extack); + + /** + * @port_fn_vf_count_set: Port function's VF count set function + * + * Set assigned VF count of a function managed by the devlink port, + * should only be used for DEVLINK_PORT_FLAVOUR_PHYSICAL. + * Return -EOPNOTSUPP if port function vf_count setup is not supported. + */ + int (*port_fn_vf_count_set)(struct devlink_port *port, u16 vf_count, + struct netlink_ext_ack *extack); + /** * port_new() - Add a new port function of a specified flavor * @devlink: Devlink instance diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 3782d4219ac9..774e17f6100b 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -676,6 +676,7 @@ enum devlink_port_function_attr { DEVLINK_PORT_FN_ATTR_STATE, /* u8 */ DEVLINK_PORT_FN_ATTR_OPSTATE, /* u8 */ DEVLINK_PORT_FN_ATTR_CAPS, /* bitfield32 */ + DEVLINK_PORT_FN_ATTR_VF_COUNT, /* u16 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1 diff --git a/net/devlink/leftover.c b/net/devlink/leftover.c index 97d30ea98b00..6dac8b562232 100644 --- a/net/devlink/leftover.c +++ b/net/devlink/leftover.c @@ -141,6 +141,7 @@ static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK), + [DEVLINK_PORT_FN_ATTR_VF_COUNT] = { .type = NLA_U16 }, }; #define ASSERT_DEVLINK_PORT_REGISTERED(devlink_port) \ @@ -520,6 +521,35 @@ static int devlink_port_fn_caps_fill(const struct devlink_ops *ops, return 0; } +static int devlink_port_fn_vf_count_fill(const struct devlink_ops *ops, + struct devlink_port *devlink_port, + struct sk_buff *msg, + struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u16 vf_count; + int err; + + if (!ops->port_fn_vf_count_get || + devlink_port->attrs.flavour != DEVLINK_PORT_FLAVOUR_PHYSICAL) + return 0; + + err = ops->port_fn_vf_count_get(devlink_port, &vf_count, extack); + if (err) { + if (err == -EOPNOTSUPP) + return 0; + return err; + } + + err = nla_put_u16(msg, DEVLINK_PORT_FN_ATTR_VF_COUNT, vf_count); + if (err) + return err; + + *msg_updated = true; + + return 0; +} + static int devlink_sb_tc_index_get_from_info(struct devlink_sb *devlink_sb, struct genl_info *info, @@ -871,6 +901,16 @@ static int devlink_port_fn_caps_set(struct devlink_port *devlink_port, return 0; } +static int devlink_port_fn_vf_count_set(struct devlink_port *devlink_port, + const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const struct devlink_ops *ops = devlink_port->devlink->ops; + u16 vf_count = nla_get_u16(attr); + + return ops->port_fn_vf_count_set(devlink_port, vf_count, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *port, struct netlink_ext_ack *extack) @@ -893,6 +933,11 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por &msg_updated); if (err) goto out; + + err = devlink_port_fn_vf_count_fill(ops, port, msg, extack, &msg_updated); + if (err) + goto out; + err = devlink_port_fn_state_fill(ops, port, msg, extack, &msg_updated); out: if (err || !msg_updated) @@ -1219,6 +1264,19 @@ static int devlink_port_function_validate(struct devlink_port *devlink_port, "Function does not support state setting"); return -EOPNOTSUPP; } + attr = tb[DEVLINK_PORT_FN_ATTR_VF_COUNT]; + if (attr) { + if (!ops->port_fn_vf_count_set) { + NL_SET_ERR_MSG_ATTR(extack, attr, + "Function doesn't support VF assignment"); + return -EOPNOTSUPP; + } + if (devlink_port->attrs.flavour != DEVLINK_PORT_FLAVOUR_PHYSICAL) { + NL_SET_ERR_MSG_ATTR(extack, attr, + "VFs assignment supported for physical ports only"); + return -EOPNOTSUPP; + } + } attr = tb[DEVLINK_PORT_FN_ATTR_CAPS]; if (attr) { struct nla_bitfield32 caps; @@ -1278,6 +1336,13 @@ static int devlink_port_function_set(struct devlink_port *port, return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_VF_COUNT]; + if (attr) { + err = devlink_port_fn_vf_count_set(port, attr, extack); + if (err) + return err; + } + /* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state. From patchwork Mon Feb 6 15:36:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Horman X-Patchwork-Id: 13130187 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A57E1C05027 for ; Mon, 6 Feb 2023 15:40:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229501AbjBFPkn (ORCPT ); Mon, 6 Feb 2023 10:40:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231698AbjBFPj4 (ORCPT ); Mon, 6 Feb 2023 10:39:56 -0500 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on20720.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e8a::720]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6ABEA2BEEB for ; Mon, 6 Feb 2023 07:39:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LODu54GJn/x6R4IRem+xbHyFic10FPy5A3OZC/98PnaqMTghR+Ekv8PhO17zXFpQZuuRmaZrutE8QVJeXt9yElk+0sND4p1gG1V+NanlGEg71Yv7TTucHIjE6cdQa3QJ9VljUOSkUeIgz/mMRBO+yaQsaxlaWf1n95xc35ktrQVmafoPnqGHSKf+T2qt2e5kvTPuaKn4hNGY71ZMzq2hLM/NALN76jU4sn4MjuftHupzXYmcgvSGa74i9XS0YB9HYozid/k+ZZgYFLRt0kjwpl25e75HQVON3JIwFwqOpbbbcKshEWk2h2ByuhthXuhgkSSdee1dXDj50JTTzu/rDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=krAZ/z9lLu4yftXseWdiMmTVvKil55EoYub4vkVmGhQ=; b=lxWBVA3t0nLiPqmj5udUlZZuNUiChq/YcEYRcJZ84tM4mGaGuc+ZUAyYaXelaXhodKBhGEcH9s2gZk9aYKvkR0uDseJ+VmAXEptqcmn3j2b10LPrzMUIc4oeLojEuL0HX6n9Mbv7KmNXjaA4c0Gyd4vt2ZUzO+9OR1BhssnybGR/Np/oRzLtAwUbdvQiOOrUciJJuol5y/5g1LWiLZdTwGT4ZirrScFpI+p6tKlF7GjMU515n2qRm1GgJvpxT1RZqSMaQkH+djs9PwPjY96DjI7ygITfk5zlTV1zqwbzZz25SyIH+FMj/WhSUGXA18JjxsEU+ELgFOd1quvD3+hzaQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=krAZ/z9lLu4yftXseWdiMmTVvKil55EoYub4vkVmGhQ=; b=sO1BclWGLqIxZ3rCTAZM9Enwmv6KhE4EE9FlLdO8HbHEJLVie8KU7f3rqny8N8y5RdVBb5+Mz3QpHbYq75xSEcVFExRV2otd219Eh58BCk3QBVv50SGTtZwbUtCKQ3rVs17Z6xcD6M4qoZ5+pY8lC9lfqx6mDvGeOaIEO02JJqc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) by SA1PR13MB4912.namprd13.prod.outlook.com (2603:10b6:806:184::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 15:37:22 +0000 Received: from PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c]) by PH0PR13MB4842.namprd13.prod.outlook.com ([fe80::85f5:bdb:fb9e:294c%3]) with mapi id 15.20.6064.034; Mon, 6 Feb 2023 15:37:22 +0000 From: Simon Horman To: David Miller , Jakub Kicinski , Paolo Abeni Cc: Michael Chan , Andy Gospodarek , Gal Pressman , Saeed Mahameed , Jesse Brandeburg , Tony Nguyen , Edward Cree , Vladimir Oltean , Andrew Lunn , Fei Qin , netdev@vger.kernel.org, oss-drivers@corigine.com Subject: [PATCH/RFC net-next 2/2] nfp: add support for assigning VFs to different physical ports Date: Mon, 6 Feb 2023 16:36:03 +0100 Message-Id: <20230206153603.2801791-3-simon.horman@corigine.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206153603.2801791-1-simon.horman@corigine.com> References: <20230206153603.2801791-1-simon.horman@corigine.com> X-ClientProxiedBy: AM4PR0202CA0013.eurprd02.prod.outlook.com (2603:10a6:200:89::23) To PH0PR13MB4842.namprd13.prod.outlook.com (2603:10b6:510:78::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB4842:EE_|SA1PR13MB4912:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b0f75e4-b81f-4e32-3ebe-08db08580a3d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LQ/wUzuBifSrts57zIIfvZucHPseRoxICpUsnBTb19CDa7GKnkK+TVpFQSLyPwkfR91XDp6+z00AkSSKAUrL84G3kWKRszoFM9YDKLwOJ8QV1hyP2yxX2hqSDGaaLMaqyPfJb0D5KwcaqRLP4ZLlWWFQSYyDsASZ5i4Z3B56z4wZw6k1amyuKuLdEjXdJzfuVy89tIbsf6nRSmN+t4NwI1Vs4RnLPngTz3JIZq/YAm/BgX+WMW+LJgUAyaOfx2QJwx5LkHfZ9x+1cMPIPQWsygBjX+vkWRe9yw9vIBBe52ORSne306GW4FGBbAMXdb7YJlXCyghnfbNVwpxOXEPiPE44cBC/4NwOqqdBk5aQmzl/pZF6XheUbeYBWGMXn2LA5I9ycELXqHEqIr9oAkxiuFOu7ZSPwrOkIxIeA0u/woYMe2l8uhg7qqksgWfoxhs5Up6M8iwF4uDQHv1oZFszpj3XypQG9tfb+iyRHX1yl4X8Yk6bB7ilXf7SzDZqidFjKzCgu4gjNZ5K7tTJkgwuO/PVPCu7FMTkCMjAmVMqtAKnZH1LTsdjTqtIpYecZy9E8XfPidmXSZyXDUmBNcGW1nOZnazcTXf2y/l+tZH1szeCdy35Jcuq7D/nyRFvZIHuEs0rVN3ieYNLFDt6f+YaAw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR13MB4842.namprd13.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(136003)(396003)(39840400004)(346002)(376002)(451199018)(478600001)(2906002)(6486002)(66946007)(186003)(6512007)(66476007)(6666004)(107886003)(41300700001)(1076003)(6506007)(8676002)(66556008)(52116002)(4326008)(30864003)(7416002)(316002)(54906003)(5660300002)(38100700002)(110136005)(8936002)(44832011)(86362001)(36756003)(83380400001)(2616005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rBPkx1B833YMeS0k7kWCFFHshnG1Q6CcFUGcttVmuIch1D7QENTO/OunG5jfkCkrwiBFcD/f8LQ2woBC5lga45rSSTA/q0pDVpFjks4Oet6sYit/1yNui+ggUeneAvjvrBD009/ohBUCTdAlmZs5a2hkCK+i6NScvVW3NJNHbiI9EE2Q++vujfgHmTis5h6vz20Yc0eZW/w7c/aDkiXcmLoEYo+kChVAFGgTw7gkAWtlWF8XVdrqdhbIsovPm59ccynL6YyxLIUHQkzl4tDIx+w9erni41z8ub/qxTU5z53MxEMZJnNdpzmAxZgFPWteBVM/4cfROda5hd6qbPn9Uf8mlpCQC7QoIkv3SuyJmM5LCV7F9WjwG+mHs1CLH7ICx1PlORl5ELp/KlKW/qOuTtxJr8dVg864iN8xBQr/dDEQOsaMEcAJMcp97x82k50JZJLJjAN1zuXYh12sJFQJINm4+2SUB1Q30DtsiaMmGmfo3Ko3ynqxOP/R7FCVFltoq2tF9LEUwvd8RE9UWG8IFzls6v2f1/vP4jFV5fxn9ej2ib5v6JKKuu9Dk7hYFCPV1m30C5pVWDccrX0qkHzusnXmEa3Yfl2pDctLA0cUKPTzk7crmJSSCcUkUXxorwFYvIPQJ/0ypxlfGxdoOZbIG+/dm7MtfI33Gl2na7ogymR40B9jz3Wk3XRzal4Y3JehOSoKSVQa8pp+b9FEkFXrttXscUGyA2Z9/8okcm37oeBPMVw0SWIIfPx2vFjzp3UR0ia699YCynR09A7l+9GPF6RlQvCgT2z6i1LgeL6j/CsyAIND2m7w2JWoJV+QQZYV2+nkbMX386CvmLlLtfjfMECQSUt+QoEp3hXYYQiFzdRx8FG8XjESrrSSUvXKOkHSFKoLRhHw+/sdlbLBFrMoPeO2d702MvoKhx4BIYeXZd1d5t5KfGxLOF8LafT7fKptRuT2VT18Lqe3na9whIfdSSNt/SyMTUYy877y79pfacOnK8EtCIWrs1g5fQC+ANLceMJkhaBLD7m3u3fEyW3kNw7JgDw6gVJRphe3Tml82HvY1y5aRkzNOopdmsT8lOhgrlo5xSAvztBuserSW8KSvKGBs3hfo4Czq4tbCSgHH0rLJv3+eZaSTVj+99+uPRF4mecThvs4aw+EsRa+hCM8rAlY8yGqLnY0BiXVGcwNQ6G7dTX78yZNBixTLCJbq7dZ1gVoEhg3hd7ynMQtR2mhFISqYJOLY/kQR3Lia6Jg5++dTzQRhs2gGOswpam7ZMnPiHL2J614Kg5dL2gIK/9NUHMD7dqhYT2EqfExVM4NjW45wmRyfLpCIDvLSVpoiFV29z+xJenRm9OtZEFo4f8NR0fs9HzZKWUSm5mkvDt/E4JrYX8xZfz+8DWoutabkG2BDTFp6l+q+fKJnA9crOYDjc/g1g2iFFg2R9+JzOGot1Eck4dQuaK0ARp135i0XF0CBjTTL4sTUq030M4dzHEsSvRvg+PI+QO153K+6uqT/YFTI3NM53Xv+zKvlOBCc8wOGK0wF9idADUqBA5docJLdP2uFVnr8p3z8TaOjFt1KTF45SwKmgZUbqZc6MaGgP+qCdkmZVSRCusyZiF/t2btnalwPeg8U/I/HS2aw8CkrBvQK3cQV3D/oCXlrchSI8R9Xq8xHE57ufZSzo4kzwWXQQ== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7b0f75e4-b81f-4e32-3ebe-08db08580a3d X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB4842.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 15:37:22.6875 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7zngTnMWwkWA9z7Nqi/5nPcjRNRkyiTW/UMjY5MaobWZnDZHbt8uuqtNE5XtvAuj25zJEsriYGSBeUhHU4MNm/2v9dF9V8zNIT8HSdNqYDg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR13MB4912 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org X-Patchwork-State: RFC From: Fei Qin Currently, all the VFs are bridged to port 0 for the device. Assigning VFs to different ports is required to realize traffic load balancing. Add implement for "devlink port function set vf_count " command to support assigning vfs to different physical ports. e.g. $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 0 $ devlink port function set pci/0000:82:00.0/0 vf_count 3 $ devlink port show pci/0000:82:00.0/0 pci/0000:82:00.0/0: type eth netdev enp130s0np0 flavour physical port 0 splittable true lanes 4 function: vf_count 3 Signed-off-by: Fei Qin Signed-off-by: Simon Horman --- .../net/ethernet/netronome/nfp/nfp_devlink.c | 71 +++++++++ drivers/net/ethernet/netronome/nfp/nfp_main.c | 5 + drivers/net/ethernet/netronome/nfp/nfp_main.h | 4 + .../ethernet/netronome/nfp/nfp_net_sriov.c | 147 +++++++++++++++++- .../ethernet/netronome/nfp/nfp_net_sriov.h | 6 + 5 files changed, 225 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c index bf6bae557158..9a6b4d6c28ca 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c @@ -9,6 +9,7 @@ #include "nfp_app.h" #include "nfp_main.h" #include "nfp_port.h" +#include "nfp_net_sriov.h" static int nfp_devlink_fill_eth_port(struct nfp_port *port, @@ -310,6 +311,74 @@ nfp_devlink_flash_update(struct devlink *devlink, return nfp_flash_update_common(devlink_priv(devlink), params->fw, extack); } +static int +nfp_devlink_vf_count_get_port_id(struct devlink_port *dl_port) +{ + struct nfp_pf *pf = devlink_priv(dl_port->devlink); + struct nfp_eth_table_port *eth_port; + struct nfp_port *port; + int err; + + err = nfp_net_sriov_check(pf->app, 0, NFP_NET_VF_CFG_MB_CAP_VF_ASSIGNMENT, + "vf_assignment", false); + if (err) + return err; + + port = container_of(dl_port, struct nfp_port, dl_port); + eth_port = __nfp_port_get_eth_port(port); + if (!eth_port) + return -EOPNOTSUPP; + + return eth_port - pf->eth_tbl->ports; +} + +static int +nfp_devlink_vf_count_get(struct devlink_port *dl_port, u16 *vf_count, + struct netlink_ext_ack *extack) +{ + int port_id = nfp_devlink_vf_count_get_port_id(dl_port); + struct nfp_pf *pf = devlink_priv(dl_port->devlink); + + if (port_id < 0 || port_id >= NFP_VF_ASSIGNMENT_PORT_COUNT) + return -EOPNOTSUPP; + + *vf_count = pf->num_assigned_vfs[port_id]; + + return 0; +} + +static int +nfp_devlink_vf_count_set(struct devlink_port *dl_port, u16 vf_count, + struct netlink_ext_ack *extack) +{ + int port_id = nfp_devlink_vf_count_get_port_id(dl_port); + struct nfp_pf *pf = devlink_priv(dl_port->devlink); + int total_num_ports = pf->eth_tbl->count; + int total_num_vfs = 0; + unsigned int i; + + if (port_id < 0 || port_id >= NFP_VF_ASSIGNMENT_PORT_COUNT) + return -EOPNOTSUPP; + + for (i = 0; i < total_num_ports; i++) { + if (i != port_id) + total_num_vfs += pf->num_assigned_vfs[i]; + else + total_num_vfs += vf_count; + + if (total_num_vfs > pf->limit_vfs) + goto exit_out_of_range; + } + + pf->num_assigned_vfs[port_id] = vf_count; + + return 0; + +exit_out_of_range: + NL_SET_ERR_MSG_MOD(extack, "Parameter out of range"); + return -EINVAL; +} + const struct devlink_ops nfp_devlink_ops = { .port_split = nfp_devlink_port_split, .port_unsplit = nfp_devlink_port_unsplit, @@ -319,6 +388,8 @@ const struct devlink_ops nfp_devlink_ops = { .eswitch_mode_set = nfp_devlink_eswitch_mode_set, .info_get = nfp_devlink_info_get, .flash_update = nfp_devlink_flash_update, + .port_fn_vf_count_get = nfp_devlink_vf_count_get, + .port_fn_vf_count_set = nfp_devlink_vf_count_set, }; int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c index 71301dbd8fb5..29663d2562aa 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_main.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c @@ -29,6 +29,7 @@ #include "nfp_app.h" #include "nfp_main.h" #include "nfp_net.h" +#include "nfp_net_sriov.h" static const char nfp_driver_name[] = "nfp"; @@ -252,6 +253,10 @@ static int nfp_pcie_sriov_enable(struct pci_dev *pdev, int num_vfs) return -EINVAL; } + err = nfp_configure_assign_vf(pdev, num_vfs); + if (err) + return err; + err = pci_enable_sriov(pdev, num_vfs); if (err) { dev_warn(&pdev->dev, "Failed to enable PCI SR-IOV: %d\n", err); diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h index 14a751bfe1fe..67692fcf1201 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_main.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h @@ -16,6 +16,8 @@ #include #include +#define NFP_VF_ASSIGNMENT_PORT_COUNT 8 + struct dentry; struct device; struct pci_dev; @@ -63,6 +65,7 @@ struct nfp_dumpspec { * @irq_entries: Array of MSI-X entries for all vNICs * @limit_vfs: Number of VFs supported by firmware (~0 for PCI limit) * @num_vfs: Number of SR-IOV VFs enabled + * @num_assigned_vfs: Number of VFs assigned to different physical ports * @fw_loaded: Is the firmware loaded? * @unload_fw_on_remove:Do we need to unload firmware on driver removal? * @ctrl_vnic: Pointer to the control vNIC if available @@ -111,6 +114,7 @@ struct nfp_pf { unsigned int limit_vfs; unsigned int num_vfs; + u8 num_assigned_vfs[NFP_VF_ASSIGNMENT_PORT_COUNT]; bool fw_loaded; bool unload_fw_on_remove; diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c index 6eeeb0fda91f..873c6d707c0e 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.c @@ -13,9 +13,13 @@ #include "nfp_net_ctrl.h" #include "nfp_net.h" #include "nfp_net_sriov.h" +#include "nfp_port.h" +#include "nfpcore/nfp_nsp.h" -static int -nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool warn) +/* Capability of VF pre-configuration */ +#define NFP_NET_VF_PRE_CONFIG NFP_NET_VF_CFG_MB_CAP_VF_ASSIGNMENT + +int nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool warn) { u16 cap_vf; @@ -29,6 +33,9 @@ nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool return -EOPNOTSUPP; } + if (cap & NFP_NET_VF_PRE_CONFIG) + return 0; + if (vf < 0 || vf >= app->pf->num_vfs) { if (warn) nfp_warn(app->pf->cpp, "invalid VF id %d\n", vf); @@ -38,17 +45,125 @@ nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, bool return 0; } +/* VFs can be shown and configured through each physical port even with VF + * assignment enabled. FW requires configurations to be sent through the port + * that the VF assigned to. Driver may need to get the port id and judge if the + * current netdev is the one that the VF assigned to. + */ +static int nfp_vf_assignment_get_port_id(struct nfp_app *app, int vf) +{ + struct nfp_pf *pf = app->pf; + unsigned int i, start_vf; + + for (start_vf = 0, i = 0; i < ARRAY_SIZE(pf->num_assigned_vfs); i++) { + if (vf >= start_vf && vf < (start_vf + pf->num_assigned_vfs[i])) + return i; + start_vf += pf->num_assigned_vfs[i]; + } + + /* If VF assignment is disabled, all the VFs are assigned to port 0 */ + return 0; +} + +static bool nfp_vf_assignment_assigned_to_cur_port(struct net_device *netdev, int vf) +{ + struct nfp_app *app = nfp_app_from_netdev(netdev); + struct nfp_eth_table_port *eth_port; + struct nfp_pf *pf = app->pf; + int assigned_port_id, err; + struct nfp_port *port; + + /* If firmware doesn't support vf_assignment, each VF should be shown under + * each physical port or PF normally. + */ + err = nfp_net_sriov_check(pf->app, 0, NFP_NET_VF_CFG_MB_CAP_VF_ASSIGNMENT, + "vf_assignment", false); + if (err < 0) + return true; + + port = nfp_port_from_netdev(netdev); + eth_port = nfp_port_get_eth_port(port); + if (!eth_port) + return true; + + assigned_port_id = nfp_vf_assignment_get_port_id(app, vf); + return eth_port == &pf->eth_tbl->ports[assigned_port_id]; +} + +int nfp_configure_assign_vf(struct pci_dev *pdev, int num_vfs) +{ + struct nfp_pf *pf = pci_get_drvdata(pdev); + int err, total_num_vfs = 0, idx = 0; + struct nfp_net *nn; + unsigned int i; + + err = nfp_net_sriov_check(pf->app, 0, NFP_NET_VF_CFG_MB_CAP_VF_ASSIGNMENT, + "vf_assignment", false); + if (err) + return 0; + + for (i = 0; i < ARRAY_SIZE(pf->num_assigned_vfs); i++) + total_num_vfs += pf->num_assigned_vfs[i]; + + /* When the total_num_vfs is nonzero, the VF assignment is enabled. + * The total number of created VFs is required to be consistent with + * the one set in VF assignment. + */ + if (total_num_vfs && num_vfs != total_num_vfs) { + dev_err(&pdev->dev, + "Trying to create %d VFs not satisfy the configuration of VF assignment\n", + num_vfs); + return -EINVAL; + } + + /* When VF assignment is disabled, all the VFs are allocated to port 0 */ + list_for_each_entry(nn, &pf->vnics, vnic_list) { + u8 num_assigned_vfs = ((idx == 0) && !total_num_vfs) ? + pf->limit_vfs : pf->num_assigned_vfs[idx]; + idx++; + + writeb(num_assigned_vfs, pf->vfcfg_tbl2 + NFP_NET_VF_CFG_VF_ASSIGNMENT); + writew(NFP_NET_VF_CFG_MB_UPD_VF_ASSIGNMENT, pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_UPD); + + err = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_VF); + if (err) + return err; + + err = readw(pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_RET); + if (err) { + dev_info(&pdev->dev, + "FW refused VF assignment update with errno: %d\n", err); + return -err; + } + } + + return 0; +} + static int nfp_net_sriov_update(struct nfp_app *app, int vf, u16 update, const char *msg) { + int ret, assigned_port_id, cnt = 0; + struct nfp_net *nn_iter; struct nfp_net *nn; - int ret; /* Write update info to mailbox in VF config symbol */ writeb(vf, app->pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_VF_NUM); writew(update, app->pf->vfcfg_tbl2 + NFP_NET_VF_CFG_MB_UPD); nn = list_first_entry(&app->pf->vnics, struct nfp_net, vnic_list); + + /* If VF assignment is enabled, reconfig of VF should be set through "nn" + * of corresponding physical port + */ + assigned_port_id = nfp_vf_assignment_get_port_id(app, vf); + list_for_each_entry(nn_iter, &app->pf->vnics, vnic_list) { + if (cnt++ == assigned_port_id) { + nn = nn_iter; + break; + } + } + /* Signal VF reconfiguration */ ret = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_VF); if (ret) @@ -257,11 +372,18 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf, "link state"); } +/* VFs can be shown under each physical port. When the VF is + * not assigned to the physical port, hardcode its mac to + * ff:ff:ff:ff:ff:ff to distinguish. The changes of VFs' + * configurations can be only seen under the corresponding + * physical ports. + */ int nfp_app_get_vf_config(struct net_device *netdev, int vf, struct ifla_vf_info *ivi) { struct nfp_app *app = nfp_app_from_netdev(netdev); u32 vf_offset, mac_hi, rate; + bool is_assigned = true; u32 vlan_tag; u16 mac_lo; u8 flags; @@ -271,13 +393,19 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf, if (err) return err; - vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ; + is_assigned = nfp_vf_assignment_assigned_to_cur_port(netdev, vf); - mac_hi = readl(app->pf->vfcfg_tbl2 + vf_offset); - mac_lo = readw(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO); + vf_offset = NFP_NET_VF_CFG_MB_SZ + vf * NFP_NET_VF_CFG_SZ; - flags = readb(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_CTRL); - vlan_tag = readl(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_VLAN); + if (is_assigned) { + mac_hi = readl(app->pf->vfcfg_tbl2 + vf_offset); + mac_lo = readw(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_MAC_LO); + flags = readb(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_CTRL); + vlan_tag = readl(app->pf->vfcfg_tbl2 + vf_offset + NFP_NET_VF_CFG_VLAN); + } else { + mac_hi = 0xffffffff; + mac_lo = 0xffff; + } memset(ivi, 0, sizeof(*ivi)); ivi->vf = vf; @@ -285,6 +413,9 @@ int nfp_app_get_vf_config(struct net_device *netdev, int vf, put_unaligned_be32(mac_hi, &ivi->mac[0]); put_unaligned_be16(mac_lo, &ivi->mac[4]); + if (!is_assigned) + return 0; + ivi->vlan = FIELD_GET(NFP_NET_VF_CFG_VLAN_VID, vlan_tag); ivi->qos = FIELD_GET(NFP_NET_VF_CFG_VLAN_QOS, vlan_tag); if (!nfp_net_sriov_check(app, vf, NFP_NET_VF_CFG_MB_CAP_VLAN_PROTO, "vlan_proto", false)) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h index 2d445fa199dc..fdf429f60d34 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h +++ b/drivers/net/ethernet/netronome/nfp/nfp_net_sriov.h @@ -21,6 +21,7 @@ #define NFP_NET_VF_CFG_MB_CAP_TRUST (0x1 << 4) #define NFP_NET_VF_CFG_MB_CAP_VLAN_PROTO (0x1 << 5) #define NFP_NET_VF_CFG_MB_CAP_RATE (0x1 << 6) +#define NFP_NET_VF_CFG_MB_CAP_VF_ASSIGNMENT (0x1 << 8) #define NFP_NET_VF_CFG_MB_RET 0x2 #define NFP_NET_VF_CFG_MB_UPD 0x4 #define NFP_NET_VF_CFG_MB_UPD_MAC (0x1 << 0) @@ -30,6 +31,8 @@ #define NFP_NET_VF_CFG_MB_UPD_TRUST (0x1 << 4) #define NFP_NET_VF_CFG_MB_UPD_VLAN_PROTO (0x1 << 5) #define NFP_NET_VF_CFG_MB_UPD_RATE (0x1 << 6) +#define NFP_NET_VF_CFG_MB_UPD_VF_ASSIGNMENT (0x1 << 8) +#define NFP_NET_VF_CFG_VF_ASSIGNMENT 0x6 #define NFP_NET_VF_CFG_MB_VF_NUM 0x7 /* VF config entry @@ -67,5 +70,8 @@ int nfp_app_set_vf_link_state(struct net_device *netdev, int vf, int link_state); int nfp_app_get_vf_config(struct net_device *netdev, int vf, struct ifla_vf_info *ivi); +int nfp_configure_assign_vf(struct pci_dev *pdev, int num_vfs); +int nfp_net_sriov_check(struct nfp_app *app, int vf, u16 cap, const char *msg, + bool warn); #endif /* _NFP_NET_SRIOV_H_ */