From patchwork Fri Nov 8 21:47:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868989 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-3.cisco.com (alln-iport-3.cisco.com [173.37.142.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EB0C1DFE1D; Fri, 8 Nov 2024 21:50:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.90 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102624; cv=none; b=szksfpiixLznlx+sSq57308uEd3pqspqalQf7olUeAAV+cw8shrpzVwDtrSvyz1fu6mefLX4lel0+uGs7V53FqAejpR27Omtp5xQKVpCUqY9Z3j1XfXuqR5v3m0lA9BydXenSbuVGmR5GjzZm3dC1fFQd6ncQbD9HxfpyPHz1EE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102624; c=relaxed/simple; bh=SPkGkJwYMTv83BMMgZFjzq2iJG1yLa4FoLBit5GloU8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=idbbcn41dqJ2GAfxyoVcmUvI2l50aap/ScSh4+jry0cRLcMpUHCqJ2UjXnut5hJ8zjI1wT156iQlWZ6D6LuJsD+O8efljX+UMlT0s76snqzS2WxoZ2OX1aHb/pGEHqdglMNEIMdr8v7mZy3fVKuXL6xqnPMof7oypOUUb8dMmXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=Rhnst6Sh; arc=none smtp.client-ip=173.37.142.90 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="Rhnst6Sh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=18944; q=dns/txt; s=iport; t=1731102621; x=1732312221; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=UFqXy0J1OEIolKkPnDz9umk/wJmr2e/IfWQp+BXm8WM=; b=Rhnst6ShFpW+SPbn9zDEmBVOujO9FQU17UfGjaMUdgr3IsGT7BrQAK+N fHUhF+VNYCXr55DhtEPAoSJsh1Mv1XvmrBZB8LmBbTrh8EVANUmG+oLAr OXkSGo9aVPgdKVdbQskG/NLPW5BrS0XYiDodxJCzlWMmqtGkTU4cUdCtn E=; X-CSE-ConnectionGUID: /QysbxoYRY+RqyYYImuMUQ== X-CSE-MsgGUID: Z5era5MYRU+lPuTuIHi6fA== X-IPAS-Result: A0AHAACvhi5n/4//Ja1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYJKgVBCSIRViB2HMIIhi3WSIxSBEQNWDwEBAQ9EBAEBhQcCijoCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFgQ4ThgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBden8zgQGEe9k4gW2BGi4BiEsBgWyDfTuEPCcbgUlEglCCLYQqg3SCaQSDQoJYUnYlgRMCAgIHAodzmFIJP4EFHANZIREBVRMNCgsHBWNYPgMib2lceiuBDoEXOkOBO4EiLxshC1yBOIEaFAYVBIEOQT+CSmlLNwINAjaCJCRZgk+FHYRvhGiCEh1AAwsYDUgRLDUGDhsGPQFuB54pRoMtexR/gRQsBSY4kloJAQeDaY09gh+fTIQkoVkzqk0eEJhJIqQbhGaBZzyBWTMaCBsVgyJSGQ+OKgMWFpMAAbVAQzU7AgcLAQEDCYZLim5gAQE IronPort-Data: A9a23:6zQLPqtH0uONpgruSROQ0AGnzOfnVIRfMUV32f8akzHdYApBsoF/q tZmKTqEbPmJZWCkeo90aYnk9xlV6sDVmoQwQQU/rSoyEnkQgMeUXt7xwmUckM+xwmwvaGo9s q3yv/GZdJhcokf0/0nrav656yEhjclkf5KkYMbcICd9WAR4fykojBNnioYRj5Vh6TSDK1vlV eja/YuGYjdJ5xYuajhIsvvb+Esz1BjPkGpwUmIWNKgjUGD2zxH5PLpHTYmtIn3xRJVjH+LSb 47r0LGj82rFyAwmA9Wjn6yTWhVirmn6ZFXmZtJ+AsBOszAazsAA+v9T2Mk0NS+7vw60c+VZk 72hg3AfpTABZcUgkMxFO/VR/roX0aduoNcrKlDn2SCfItGvn3bEm51T4E8K0YIwvd9mGmte8 s0iGXM3dwGytsOz3ZeGc7w57igjBJGD0II3oHpsy3TdSP0hW52GG/uM7t5D1zB2jcdLdRrcT 5NGMnw0M1KaPkAJYwtMYH49tL/Aan3XdTBVs1mSr6Mf6GnIxws327/oWDbQUofaFZ8ExBvA+ goq+UzLXxwAPYG6+AC4sXaUm9PSvAP6Bow7QejQGvlCxQf7KnYoIBEfS1a+ifWwlEO7X9VRN woS9zZGhaU+6UmiXNThdxK/p3GAs1gXXN84O+098gSW4qnZ+QCUAi4DVDEpQN87vsYeRjEw0 FKN2dTzClRHubuZU3+CtbGZsT+/JwARMGkEIyQEJSMd6tPupoAblB/DTt9/VqWyi7XdHTD23 iDPryUkgbgXpdAE2r/9/l3dhT+o4J/TQWYd4AjLUm+7xh12aZTjZIGy71Xfq/FaI+6kokKpp nMInY2aqesJF5zIzHLLS+QWF7bv7PGAWNHBvWNS81Aa32zF0xaekUp4uVmS+G8B3h44RALU IronPort-HdrOrdr: A9a23:UZ2NOqhVcRl6fB6By8RaajiqPnBQXhcji2hC6mlwRA09TyVXrb HIoB1973/JYVcqOU3I9urtBEDtex7hHNtOkOws1NSZMjUOxlHYT72KhLGKq1aLJ8S9zJ8+6U 4KScdD4bPLfDxHpPe/zQWmH9Mn2dWdtIKllY7lvg9QZDAvRq1+4wJ+EwqBVnd3Sg5PGIYjGP Onl7N6TkKbCBIqhgDRPAh+YwAFzOe7767bXQ== X-Talos-CUID: 9a23:0WAc+m8bVjDbD5Sz+MSVvwktC9svYnrs9kaTGVapOFdITaCwdGbFrQ== X-Talos-MUID: 9a23:2RBpag2nJNYPKx0jDI5U6Vo4GTUj0fmjEklQtKQ8hoqADTFMJha9jWq5a9py X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="388292504" Received: from rcdn-l-core-06.cisco.com ([173.37.255.143]) by alln-iport-3.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-06.cisco.com (Postfix) with ESMTPS id 798C718000252; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id F17D1CC128E; Fri, 8 Nov 2024 21:49:11 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:47 +0000 Subject: [PATCH net-next v3 1/7] enic: Create enic_wq/rq structures to bundle per wq/rq data Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-1-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=19489; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=SPkGkJwYMTv83BMMgZFjzq2iJG1yLa4FoLBit5GloU8=; b=jvM7gjbSFQeiWiuSkEX925wET1bFLwsVqBdixZusITO+YZiBJWdxg33UFnyjI9ydGqMMTRLTr nv4DnSF+dSpD839JwTimNyntJNYN2rclaVMxNsbcSQ9gcgGQRwcw+6I X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: rcdn-l-core-06.cisco.com X-Patchwork-Delegate: kuba@kernel.org Bundling the wq/rq specific data into dedicated enic_wq/rq structures cleans up the enic structure and simplifies future changes related to wq/rq. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 18 ++-- drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 +- drivers/net/ethernet/cisco/enic/enic_main.c | 120 ++++++++++++------------- drivers/net/ethernet/cisco/enic/enic_res.c | 12 +-- 4 files changed, 81 insertions(+), 73 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 0cc3644ee8554f52401a0be7f44a1475ab2ea2b9..e6edb43515b97feeb21a9b55a1eeaa9b9381183f 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -162,6 +162,17 @@ struct enic_rq_stats { u64 desc_skip; /* Rx pkt went into later buffer */ }; +struct enic_wq { + struct vnic_wq vwq; + struct enic_wq_stats stats; + spinlock_t lock; /* spinlock for wq */ +}; + +struct enic_rq { + struct vnic_rq vrq; + struct enic_rq_stats stats; +}; + /* Per-instance private data structure */ struct enic { struct net_device *netdev; @@ -194,16 +205,13 @@ struct enic { struct enic_port_profile *pp; /* work queue cache line section */ - ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX]; - spinlock_t wq_lock[ENIC_WQ_MAX]; - struct enic_wq_stats wq_stats[ENIC_WQ_MAX]; + ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ - ____cacheline_aligned struct vnic_rq rq[ENIC_RQ_MAX]; - struct enic_rq_stats rq_stats[ENIC_RQ_MAX]; + ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index f7986f2b6a1794144909ffad9e3e09c32ea44c93..909d6f7000e160cf2e15de4660c1034cad7d51ba 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -337,7 +337,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, for (i = 0; i < NUM_ENIC_GEN_STATS; i++) *(data++) = ((u64 *)&enic->gen_stats)[enic_gen_stats[i].index]; for (i = 0; i < enic->rq_count; i++) { - struct enic_rq_stats *rqstats = &enic->rq_stats[i]; + struct enic_rq_stats *rqstats = &enic->rq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_RQ_STATS; j++) { @@ -346,7 +346,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, } } for (i = 0; i < enic->wq_count; i++) { - struct enic_wq_stats *wqstats = &enic->wq_stats[i]; + struct enic_wq_stats *wqstats = &enic->wq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_WQ_STATS; j++) { diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index ffed14b63d41d1737c577fe1662eb1c2c8aea808..eb00058b6c68ec5c1ac433b54b5bc6f3fb613777 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -342,8 +342,8 @@ static void enic_wq_free_buf(struct vnic_wq *wq, { struct enic *enic = vnic_dev_priv(wq->vdev); - enic->wq_stats[wq->index].cq_work++; - enic->wq_stats[wq->index].cq_bytes += buf->len; + enic->wq[wq->index].stats.cq_work++; + enic->wq[wq->index].stats.cq_bytes += buf->len; enic_free_wq_buf(wq, buf); } @@ -352,20 +352,20 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - spin_lock(&enic->wq_lock[q_number]); + spin_lock(&enic->wq[q_number].lock); - vnic_wq_service(&enic->wq[q_number], cq_desc, + vnic_wq_service(&enic->wq[q_number].vwq, cq_desc, completed_index, enic_wq_free_buf, opaque); if (netif_tx_queue_stopped(netdev_get_tx_queue(enic->netdev, q_number)) && - vnic_wq_desc_avail(&enic->wq[q_number]) >= + vnic_wq_desc_avail(&enic->wq[q_number].vwq) >= (MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS)) { netif_wake_subqueue(enic->netdev, q_number); - enic->wq_stats[q_number].wake++; + enic->wq[q_number].stats.wake++; } - spin_unlock(&enic->wq_lock[q_number]); + spin_unlock(&enic->wq[q_number].lock); return 0; } @@ -377,7 +377,7 @@ static bool enic_log_q_error(struct enic *enic) bool err = false; for (i = 0; i < enic->wq_count; i++) { - error_status = vnic_wq_error_status(&enic->wq[i]); + error_status = vnic_wq_error_status(&enic->wq[i].vwq); err |= error_status; if (error_status) netdev_err(enic->netdev, "WQ[%d] error_status %d\n", @@ -385,7 +385,7 @@ static bool enic_log_q_error(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - error_status = vnic_rq_error_status(&enic->rq[i]); + error_status = vnic_rq_error_status(&enic->rq[i].vrq); err |= error_status; if (error_status) netdev_err(enic->netdev, "RQ[%d] error_status %d\n", @@ -598,9 +598,9 @@ static int enic_queue_wq_skb_vlan(struct enic *enic, struct vnic_wq *wq, err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); /* The enic_queue_wq_desc() above does not do HW checksum */ - enic->wq_stats[wq->index].csum_none++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_none++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -634,9 +634,9 @@ static int enic_queue_wq_skb_csum_l4(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].csum_partial++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_partial++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -699,11 +699,11 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, if (skb->encapsulation) { hdr_len = skb_inner_tcp_all_headers(skb); enic_preload_tcp_csum_encap(skb); - enic->wq_stats[wq->index].encap_tso++; + enic->wq[wq->index].stats.encap_tso++; } else { hdr_len = skb_tcp_all_headers(skb); enic_preload_tcp_csum(skb); - enic->wq_stats[wq->index].tso++; + enic->wq[wq->index].stats.tso++; } /* Queue WQ_ENET_MAX_DESC_LEN length descriptors @@ -757,8 +757,8 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, pkts = len / mss; if ((len % mss) > 0) pkts++; - enic->wq_stats[wq->index].packets += pkts; - enic->wq_stats[wq->index].bytes += (len + (pkts * hdr_len)); + enic->wq[wq->index].stats.packets += pkts; + enic->wq[wq->index].stats.bytes += (len + (pkts * hdr_len)); return 0; } @@ -792,9 +792,9 @@ static inline int enic_queue_wq_skb_encap(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].encap_csum++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.encap_csum++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -812,7 +812,7 @@ static inline int enic_queue_wq_skb(struct enic *enic, /* VLAN tag from trunking driver */ vlan_tag_insert = 1; vlan_tag = skb_vlan_tag_get(skb); - enic->wq_stats[wq->index].add_vlan++; + enic->wq[wq->index].stats.add_vlan++; } else if (enic->loop_enable) { vlan_tag = enic->loop_tag; loopback = 1; @@ -859,11 +859,11 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, struct netdev_queue *txq; txq_map = skb_get_queue_mapping(skb) % enic->wq_count; - wq = &enic->wq[txq_map]; + wq = &enic->wq[txq_map].vwq; if (skb->len <= 0) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].null_pkt++; + enic->wq[wq->index].stats.null_pkt++; return NETDEV_TX_OK; } @@ -878,19 +878,19 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, skb_shinfo(skb)->nr_frags + 1 > ENIC_NON_TSO_MAX_DESC && skb_linearize(skb)) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].skb_linear_fail++; + enic->wq[wq->index].stats.skb_linear_fail++; return NETDEV_TX_OK; } - spin_lock(&enic->wq_lock[txq_map]); + spin_lock(&enic->wq[txq_map].lock); if (vnic_wq_desc_avail(wq) < skb_shinfo(skb)->nr_frags + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); /* This is a hard error, log it */ netdev_err(netdev, "BUG! Tx ring full when queue awake!\n"); - spin_unlock(&enic->wq_lock[txq_map]); - enic->wq_stats[wq->index].desc_full_awake++; + spin_unlock(&enic->wq[txq_map].lock); + enic->wq[wq->index].stats.desc_full_awake++; return NETDEV_TX_BUSY; } @@ -899,14 +899,14 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, if (vnic_wq_desc_avail(wq) < MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); - enic->wq_stats[wq->index].stopped++; + enic->wq[wq->index].stats.stopped++; } skb_tx_timestamp(skb); if (!netdev_xmit_more() || netif_xmit_stopped(txq)) vnic_wq_doorbell(wq); error: - spin_unlock(&enic->wq_lock[txq_map]); + spin_unlock(&enic->wq[txq_map].lock); return NETDEV_TX_OK; } @@ -941,9 +941,9 @@ static void enic_get_stats(struct net_device *netdev, net_stats->multicast = stats->rx.rx_multicast_frames_ok; for (i = 0; i < ENIC_RQ_MAX; i++) { - struct enic_rq_stats *rqs = &enic->rq_stats[i]; + struct enic_rq_stats *rqs = &enic->rq[i].stats; - if (!enic->rq->ctrl) + if (!enic->rq[i].vrq.ctrl) break; pkt_truncated += rqs->pkt_truncated; bad_fcs += rqs->bad_fcs; @@ -1313,7 +1313,7 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq) } skb = netdev_alloc_skb_ip_align(netdev, len); if (!skb) { - enic->rq_stats[rq->index].no_skb++; + enic->rq[rq->index].stats.no_skb++; return -ENOMEM; } @@ -1366,7 +1366,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq, struct net_device *netdev = enic->netdev; struct sk_buff *skb; struct vnic_cq *cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - struct enic_rq_stats *rqstats = &enic->rq_stats[rq->index]; + struct enic_rq_stats *rqstats = &enic->rq[rq->index].stats; u8 type, color, eop, sop, ingress_port, vlan_stripped; u8 fcoe, fcoe_sof, fcoe_fc_crc_ok, fcoe_enc_error, fcoe_eof; @@ -1512,7 +1512,7 @@ static int enic_rq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - vnic_rq_service(&enic->rq[q_number], cq_desc, + vnic_rq_service(&enic->rq[q_number].vrq, cq_desc, completed_index, VNIC_RQ_RETURN_DESC, enic_rq_indicate_buf, opaque); @@ -1609,7 +1609,7 @@ static int enic_poll(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[0].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling * mode so we can try to fill the ring again. @@ -1621,7 +1621,7 @@ static int enic_poll(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[0]); + enic_calc_int_moderation(enic, &enic->rq[0].vrq); if ((rq_work_done < budget) && napi_complete_done(napi, rq_work_done)) { @@ -1630,11 +1630,11 @@ static int enic_poll(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[0]); + enic_set_int_moderation(enic, &enic->rq[0].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[0].napi_complete++; + enic->rq[0].stats.napi_complete++; } else { - enic->rq_stats[0].napi_repoll++; + enic->rq[0].stats.napi_repoll++; } return rq_work_done; @@ -1683,7 +1683,7 @@ static int enic_poll_msix_wq(struct napi_struct *napi, int budget) struct net_device *netdev = napi->dev; struct enic *enic = netdev_priv(netdev); unsigned int wq_index = (napi - &enic->napi[0]) - enic->rq_count; - struct vnic_wq *wq = &enic->wq[wq_index]; + struct vnic_wq *wq = &enic->wq[wq_index].vwq; unsigned int cq; unsigned int intr; unsigned int wq_work_to_do = ENIC_WQ_NAPI_BUDGET; @@ -1737,7 +1737,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[rq], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[rq].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling mode * so we can try to fill the ring again. @@ -1749,7 +1749,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[rq]); + enic_calc_int_moderation(enic, &enic->rq[rq].vrq); if ((work_done < budget) && napi_complete_done(napi, work_done)) { @@ -1758,11 +1758,11 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[rq]); + enic_set_int_moderation(enic, &enic->rq[rq].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[rq].napi_complete++; + enic->rq[rq].stats.napi_complete++; } else { - enic->rq_stats[rq].napi_repoll++; + enic->rq[rq].stats.napi_repoll++; } return work_done; @@ -1989,10 +1989,10 @@ static int enic_open(struct net_device *netdev) for (i = 0; i < enic->rq_count; i++) { /* enable rq before updating rq desc */ - vnic_rq_enable(&enic->rq[i]); - vnic_rq_fill(&enic->rq[i], enic_rq_alloc_buf); + vnic_rq_enable(&enic->rq[i].vrq); + vnic_rq_fill(&enic->rq[i].vrq, enic_rq_alloc_buf); /* Need at least one buffer on ring to get going */ - if (vnic_rq_desc_used(&enic->rq[i]) == 0) { + if (vnic_rq_desc_used(&enic->rq[i].vrq) == 0) { netdev_err(netdev, "Unable to alloc receive buffers\n"); err = -ENOMEM; goto err_out_free_rq; @@ -2000,7 +2000,7 @@ static int enic_open(struct net_device *netdev) } for (i = 0; i < enic->wq_count; i++) - vnic_wq_enable(&enic->wq[i]); + vnic_wq_enable(&enic->wq[i].vwq); if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic)) enic_dev_add_station_addr(enic); @@ -2027,9 +2027,9 @@ static int enic_open(struct net_device *netdev) err_out_free_rq: for (i = 0; i < enic->rq_count; i++) { - ret = vnic_rq_disable(&enic->rq[i]); + ret = vnic_rq_disable(&enic->rq[i].vrq); if (!ret) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); } enic_dev_notify_unset(enic); err_out_free_intr: @@ -2071,12 +2071,12 @@ static int enic_stop(struct net_device *netdev) enic_dev_del_station_addr(enic); for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_disable(&enic->wq[i]); + err = vnic_wq_disable(&enic->wq[i].vwq); if (err) return err; } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_disable(&enic->rq[i]); + err = vnic_rq_disable(&enic->rq[i].vrq); if (err) return err; } @@ -2086,9 +2086,9 @@ static int enic_stop(struct net_device *netdev) enic_free_intr(enic); for (i = 0; i < enic->wq_count; i++) - vnic_wq_clean(&enic->wq[i], enic_free_wq_buf); + vnic_wq_clean(&enic->wq[i].vwq, enic_free_wq_buf); for (i = 0; i < enic->rq_count; i++) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); for (i = 0; i < enic->cq_count; i++) vnic_cq_clean(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -2576,7 +2576,7 @@ static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { struct enic *enic = netdev_priv(dev); - struct enic_rq_stats *rqstats = &enic->rq_stats[idx]; + struct enic_rq_stats *rqstats = &enic->rq[idx].stats; rxs->bytes = rqstats->bytes; rxs->packets = rqstats->packets; @@ -2590,7 +2590,7 @@ static void enic_get_queue_stats_tx(struct net_device *dev, int idx, struct netdev_queue_stats_tx *txs) { struct enic *enic = netdev_priv(dev); - struct enic_wq_stats *wqstats = &enic->wq_stats[idx]; + struct enic_wq_stats *wqstats = &enic->wq[idx].stats; txs->bytes = wqstats->bytes; txs->packets = wqstats->packets; @@ -2993,7 +2993,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_WORK(&enic->change_mtu_work, enic_change_mtu_work); for (i = 0; i < enic->wq_count; i++) - spin_lock_init(&enic->wq_lock[i]); + spin_lock_init(&enic->wq[i].lock); /* Register net device */ diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1c48aebdbab02b88293544dcabda2d90d1a71a70..60be09acb9fd56b642b7cabc77fac01f526b29a2 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -176,9 +176,9 @@ void enic_free_vnic_resources(struct enic *enic) unsigned int i; for (i = 0; i < enic->wq_count; i++) - vnic_wq_free(&enic->wq[i]); + vnic_wq_free(&enic->wq[i].vwq); for (i = 0; i < enic->rq_count; i++) - vnic_rq_free(&enic->rq[i]); + vnic_rq_free(&enic->rq[i].vrq); for (i = 0; i < enic->cq_count; i++) vnic_cq_free(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -233,7 +233,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->rq_count; i++) { cq_index = i; - vnic_rq_init(&enic->rq[i], + vnic_rq_init(&enic->rq[i].vrq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -241,7 +241,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->wq_count; i++) { cq_index = enic->rq_count + i; - vnic_wq_init(&enic->wq[i], + vnic_wq_init(&enic->wq[i].vwq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -322,7 +322,7 @@ int enic_alloc_vnic_resources(struct enic *enic) */ for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_alloc(enic->vdev, &enic->wq[i], i, + err = vnic_wq_alloc(enic->vdev, &enic->wq[i].vwq, i, enic->config.wq_desc_count, sizeof(struct wq_enet_desc)); if (err) @@ -330,7 +330,7 @@ int enic_alloc_vnic_resources(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_alloc(enic->vdev, &enic->rq[i], i, + err = vnic_rq_alloc(enic->vdev, &enic->rq[i].vrq, i, enic->config.rq_desc_count, sizeof(struct rq_enet_desc)); if (err) From patchwork Fri Nov 8 21:47:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868986 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-8.cisco.com (rcdn-iport-8.cisco.com [173.37.86.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 327501EF0BD; Fri, 8 Nov 2024 21:50:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.79 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102623; cv=none; b=JGQeZIgVv6yDvaVwvq+VrXYKFe4NKcs58QioIJ5HijO/aZ6ROFK5eosQsrMHVHT/uH/JAJ5EE77Q2wqf2LFx7rw7u0M3hfPdlJ57oz+/3rbHgEeC1DcCH9J34RSYIAa29F8U0OTvIMjFA9xyqGGkBdOoUm0GRMcznrvV3HIVp4Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102623; c=relaxed/simple; bh=XQ3jHf1N8vFfVXlcACUMbfR0xqkNz6oH2BhWU4pzZ/0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=diH1IfXy3ZpMvP3mutfoEy7JX1QdPkUDi0YzAVZH3YkkSEtdnE3FNFLnXRxae5OEiCo/WQ5A+QemETLlIZfsZ6g98lk0wOFTpKvafgMkZq/DBEqgjRUXRvqItROssIB0KrVIz2fGXF0ZUexIWT73xDZXf9izEfDbMwG0x1+gRK0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=M9asuuQS; arc=none smtp.client-ip=173.37.86.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="M9asuuQS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=3516; q=dns/txt; s=iport; t=1731102621; x=1732312221; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=/2cgGfNC+AxfAfMGCkaQxOgbTkx4rZWj1kqd0GxZ8Wo=; b=M9asuuQS5BqUUbI1/DPNvQcq+ShKAWTkAe/BHmM5GCOG7S1Ha196YG96 1gesMv1SEKq5ZA1bsFWtmZaan04Ie3Ja0i1N9F7U8iYLm1lTOAfntc7v0 LWXUGGFFmIdcqrhrnuzunlCoscS3OabLhmzJcY21vs1fRIGlUcbVOeZlU I=; X-CSE-ConnectionGUID: QPMTbMA6QSSOTS6w0Yh93w== X-CSE-MsgGUID: NA2JPclQQa+5ytNLi30wFQ== X-IPAS-Result: A0AHAAA5hi5n/43/Ja1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYJKgVBCSIRViB2HMIIhi3WSI4ElA1YPAQEBD0QEAQGFBwKKOgImNAkOAQIEAQEBAQMCAwEBAQEBAQEBAQEBCwEBBQEBAQIBBwWBDhOGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsFp6fzOBAYR72TiBbYEaLgGISwGBbIN9O4Q8JxuBSUSBFAGDaIgegmkEh2IliRWYUgk/gQUcA1khEQFVEw0KCwcFY1g+AyJvaVx6K4EOgRc6Q4E7gSIvGyELXIE4gRoUBhUEgQ5BP4JKaUs3Ag0CNoIkJFmCT4UdhG+EaIISHUADCxgNSBEsNQYOGwY9AW4HnilGgyYHexSBLgJAP5dHjT2ha4QkoVkzqk2YdyKkG4RmgWc8gVkzGggbFYMiUhkPji0WFpMAAbU6QzU7AgcLAQEDCZIZAQE IronPort-Data: A9a23:5lIL56xyXLDUNzO7QVp6t+f3xyrEfRIJ4+MujC+fZmUNrF6WrkVUy mYfUWDVPKqDYGbzfdlyPI/i9UgDv5WAmNNmTldlpFhgHilAwSbn6Xt1DatR0we6dJCroJdPt p1GAjX4BJlqCCea/lH1b+CJQUBUjcmgXqD7BPPPJhd/TAplTDZJoR94kobVuKYw6TSCK13L4 ImaT/H3Ygf/h2ctajJMt8pvlTs21BjMkGJA1rABTagjUG/2zxE9EJ8ZLKetGHr0KqE8NvK6X evK0Iai9Wrf+Ro3Yvv9+losWhRXKlJ6FVHmZkt+A8BOsDAbzsAB+vpT2M4nVKtio27hc+adZ zl6ncfYpQ8BZsUgkQmGOvVSO3kW0aZuoNcrLZUj2CCe5xWuTpfi/xlhJHgOGLES388pOFp10 sAjdi9VXxDS3P3jldpXSsE07igiBNPgMIVavjRryivUSK56B5vCWK7No9Rf2V/chOgXQq2YP JVfM2cyKk2cOnWjOX9PYH46tOGli2L0dDdRgFmUvqEwpWPUyWSd1ZC3b4eEJ4TaHpw9ckCwg Euc3VzyBigjPZ/C2Caj1nP33vDhpHauMG4VPPjinhJwu3Wfz3IeDTUaXEW2pP2+hFL4Xd9DQ 2QZ9jcrpLo/6GSkSd7yWxD+q3mB1jYfRtBZO+438geAzuzT+QnxLmECQiRMd58gudM6SCIC0 kKPmZXiBVRHqLSfRHSc3q2ZoTO7JW4eKmpqTSsFSxYVptruuoc+ijrRQdt5Vq24lNv4HXf32 T/ihC4zm7kek+YV2Kihu1PKmTShot7OVAFd2+nMdniu4gU8YMuuYJalrACHq/1BN42eCFKGu RDohvSj0QzHNrnV/ATlfQnHNOjBCyqtWNEEvWNSIg== IronPort-HdrOrdr: A9a23:Km9ghqrggEVQ8UPYhyqcT9AaV5r0eYIsimQD101hICG9vPbo8P xG+8566faUslcssR4b9exoVJPsfZqYz+8R3WBzB9mftXfdyQiVxehZhOOIqQEIWReOlNK1vp 0OT0ERMqyVMXFKyev3/wW8Fc8t252k/LDAv5an815dCSxndK1k6R50EUKgEkNwTBRbHpZRLu vk2iM+nUvHRUgq X-Talos-CUID: 9a23:eY6UGmGO2zUPbUEPqmJezGsoEOUMLETm91DsGBGnUVdpGeaKHAo= X-Talos-MUID: 9a23:BO8QUw7/rZl//fIoQJEAH/rOxoxOyv6MMEsJqKwP+JSVKQpUOmzFkxqOF9o= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="275950831" Received: from rcdn-l-core-04.cisco.com ([173.37.255.141]) by rcdn-iport-8.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:12 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-04.cisco.com (Postfix) with ESMTPS id 81F9B1800019C; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 0488BCC128F; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:48 +0000 Subject: [PATCH net-next v3 2/7] enic: Make MSI-X I/O interrupts come after the other required ones Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-2-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=3573; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=XQ3jHf1N8vFfVXlcACUMbfR0xqkNz6oH2BhWU4pzZ/0=; b=z0jDXtD93+vpE09VO+RoS0uNefrG35ZmyKaYkFc0+QTtsWmvNkPiXSVJSUz6Md6lUUoLk9m5O lKjUhdSt4qxC4jE9x+O4jVtkD/vh0OoGemyl7PRtVDZ3Ef8+QFrwbME X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Patchwork-Delegate: kuba@kernel.org The VIC hardware has a constraint that the MSIX interrupt used for errors be specified as a 7 bit number. Before this patch, it was allocated after the I/O interrupts, which would cause a problem if 128 or more I/O interrupts are in use. So make the required interrupts come before the I/O interrupts to guarantee the error interrupt offset never exceeds 7 bits. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 20 +++++++++++++++----- drivers/net/ethernet/cisco/enic/enic_res.c | 11 +++++++---- 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index e6edb43515b97feeb21a9b55a1eeaa9b9381183f..ac7236f76a51bf32e7060ee0482b41fe82b60b44 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -280,18 +280,28 @@ static inline unsigned int enic_msix_wq_intr(struct enic *enic, return enic->cq[enic_cq_wq(enic, wq)].interrupt_offset; } -static inline unsigned int enic_msix_err_intr(struct enic *enic) -{ - return enic->rq_count + enic->wq_count; -} +/* MSIX interrupts are organized as the error interrupt, then the notify + * interrupt followed by all the I/O interrupts. The error interrupt needs + * to fit in 7 bits due to hardware constraints + */ +#define ENIC_MSIX_RESERVED_INTR 2 +#define ENIC_MSIX_ERR_INTR 0 +#define ENIC_MSIX_NOTIFY_INTR 1 +#define ENIC_MSIX_IO_INTR_BASE ENIC_MSIX_RESERVED_INTR +#define ENIC_MSIX_MIN_INTR (ENIC_MSIX_RESERVED_INTR + 2) #define ENIC_LEGACY_IO_INTR 0 #define ENIC_LEGACY_ERR_INTR 1 #define ENIC_LEGACY_NOTIFY_INTR 2 +static inline unsigned int enic_msix_err_intr(struct enic *enic) +{ + return ENIC_MSIX_ERR_INTR; +} + static inline unsigned int enic_msix_notify_intr(struct enic *enic) { - return enic->rq_count + enic->wq_count + 1; + return ENIC_MSIX_NOTIFY_INTR; } static inline bool enic_is_err_intr(struct enic *enic, int intr) diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 60be09acb9fd56b642b7cabc77fac01f526b29a2..72b51e9d8d1a26a2cd18df9c9d702e5b11993b70 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -221,9 +221,12 @@ void enic_init_vnic_resources(struct enic *enic) switch (intr_mode) { case VNIC_DEV_INTR_MODE_INTX: + error_interrupt_enable = 1; + error_interrupt_offset = ENIC_LEGACY_ERR_INTR; + break; case VNIC_DEV_INTR_MODE_MSIX: error_interrupt_enable = 1; - error_interrupt_offset = enic->intr_count - 2; + error_interrupt_offset = enic_msix_err_intr(enic); break; default: error_interrupt_enable = 0; @@ -249,15 +252,15 @@ void enic_init_vnic_resources(struct enic *enic) /* Init CQ resources * - * CQ[0 - n+m-1] point to INTR[0] for INTx, MSI - * CQ[0 - n+m-1] point to INTR[0 - n+m-1] for MSI-X + * All CQs point to INTR[0] for INTx, MSI + * CQ[i] point to INTR[ENIC_MSIX_IO_INTR_BASE + i] for MSI-X */ for (i = 0; i < enic->cq_count; i++) { switch (intr_mode) { case VNIC_DEV_INTR_MODE_MSIX: - interrupt_offset = i; + interrupt_offset = ENIC_MSIX_IO_INTR_BASE + i; break; default: interrupt_offset = 0; From patchwork Fri Nov 8 21:47:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868985 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-4.cisco.com (alln-iport-4.cisco.com [173.37.142.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 023E81E47CB; Fri, 8 Nov 2024 21:50:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.91 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102622; cv=none; b=ZzFBWAkj/jskDkm2ri4ldt4XrW3P0v5/cqfhhsDnCIwhpA0EY7vYzfF0DYV64d4xq+RZ8bVlIgIHcKbbENFpApeSkyM/+oVMP6BCdWnhUTAuPfs2uB2YEoyZFC/JsWJKfxUufkE9ecJN+Numkq8yjHoSNpfjvwGoRpMblD1Xj34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102622; c=relaxed/simple; bh=iwLqBIaYDt+zF0jzy60eoatbh4XPWQr7YNCqVTD0bdY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=AdgjexRf+BW+Hg8TW97ziSYKtc6Afr8wy9yVaIo7EWJxXAql2fdu4vfP7MTQLjDGmWzF/WX2LDHVgH901Kv1d7Xpp36ZjWEZP4e0uQxdNahARbuKWOfd1wg3lIVhFxoGcUg5ef+XJRYhhIpedjAzqrzNcHaar8x8ieJ692W/V7A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=iBmrHDM9; arc=none smtp.client-ip=173.37.142.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="iBmrHDM9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=4143; q=dns/txt; s=iport; t=1731102621; x=1732312221; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=Dl7oapZht/FmgDjgm74J9jb0/gMpPf4+RKRAlR78OqA=; b=iBmrHDM9NVM8UTitKNXE7/72xsRsSgrk50y6uPYmF/C/sFUcxWBa3+lP q2ua2YC1zfoAA4wv6ZP/E74bJfrhuGsRjkbm9/XiLk32fxtD9QEDEPCca 6DmL7OI7EpXjKnVnmj5EDH+TUminKTVVsi4dxU9IjHbag/tOZFmYKnfpt I=; X-CSE-ConnectionGUID: 9Yu1TuSsQeGZohNrjEzaBA== X-CSE-MsgGUID: C9RQHqr2SSuHHrrYmanyRw== X-IPAS-Result: A0AUAAA5hi5nj4z/Ja1aHAEBAQEBAQcBARIBAQQEAQFAgT8HAQELAYQaQkiEVYgdhzCCIYt1kiOBJQNWDwEBAQ9EBAEBhQcCijoCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFFAEBAQEBATkFSYYIhlsCAQMjBFIQJQImAgIrGxAGARKDAYJlAgGwWnp/M4EBhHvZOIFtgRouAYhLAYFsg307hDwnG4FJRIJQgT5viB6CaQSGbHaJOphSCT+BBRwDWSERAVUTDQoLBwVjWD4DIm9pXHorgQ6BFzpDgTuBIi8bIQtcgTiBGhQGFQSBDkE/gkppSzcCDQI2giQkWYJPhR2Eb4RoghIdQAMLGA1IESw1Bg4bBj0BbgeeKUaDJgeBD4ITLGOWVI09oWuEJKFZM4QElAGSSJh3IqQbhGaBZzqBWzMaCBsVgyJSGQ+OLQ0JkxYBtTpDNTsCBwsBAQMJkhkBAQ IronPort-Data: A9a23:NP4Y8KDXqCkx3RVW/73jw5YqxClBgxIJ4kV8jS/XYbTApDMggjIAn WMeCGiBb/3cNzTxKognOY+yoE8CvJ+AzYA2OVdlrnsFo1CmBibm6XV1Cm+qYkt+++WaFBoPA /02M4eGdIZsCCeB/n9BC5C5xVFkz6aEW7HgP+DNPyF1VGdMRTwo4f5Zs7ZRbrVA357gWWthh fuo+5eDYQb9i2YoWo4pw/vrRC1H7ayaVAww5jTSVdgT1HfCmn8cCo4oJK3ZBxPQXolOE+emc P3Ixbe/83mx109F5gSNy+uTnuUiG9Y+DCDW4pZkc/HKbitq+kTe5p0G2M80Mi+7vdkmc+dZk 72hvbToIesg0zaldO41C3G0GAkmVUFKFSOuzXWX6aSuI0P6n3TEksRzVX0rBpYh98lFBmFo7 /41OjknR0XW7w626OrTpuhEnM8vKozveYgYoHwllWqfBvc9SpeFSKLPjTNa9G5v3YYVQrCEO pdfMGY3BPjDS0Un1lM/Dp8zh+yvjHDXeDxDo1XTrq0yi4TW5FcujumyboOMK7RmQ+1+z3yhh nvcpV/CHzE6LsauzDOD40iz07qncSTTA99KS+biqZaGmma7ymUNBRg+WVKlrPy9jUCiHdRSN yQ89yYzqKEg+VCDQd76UBm15nWDu3Y0W91KFvYS6wyXzKfQpQGDCQAsTSNLYfQludUwSDhs0 UWG9/vgAzB1vaLWT3+B+rqKhTevPy5TJm9qTTcNRwYD4vH5rY0zhw6JRdFmeIaxj9voCXTzz iqMoSwWmboel4gI2r+98FSBhCijzrDPQxI56xv/QG2o9EV6aZSjaoju7kLUhcusN66DRVWH+ XxBkM+E4aVWVteGlTeGR6MGG7TBC+u53CP00G9yRqU8zC+U9lGReoB2+jNTNk5DG5NREdP2W 3P7tQRU7Z5VGXKla65rfo68Y/jGK4C+SLwJsdiKMrJzjohNSeORwM15iaetM4HRfKoEzfBX1 XSzKJrE4ZMm5UJPl2XeqwA1iuFD+8zG7TmPLa0XNjz+uVZkWFabSK0eLHyFZf0j4aWPrW39q okEbpPSl08OCL2mP0E7FLL/y3hUdRDX4rir+6RqmhKre1cO9JwJUqWImOhwIeSJYYwPy72So xlRpXO0OHKk2CWYcl/VApySQLjuRp145WkqJjAhOE3g2n4oJ+6SAFQ3KfMKkU0c3LU7l5ZcF qBdE+3ZW6QnYmqcoVw1M8KixLGOgTz33mpiyQL5O2BnJ/aNhmXhprfZQ+cY3HJeVHPr7JBm8 +PIO8GyacNrejmOxf3+MJqHp25dd1BE8A6udyMk+uVuRXg= IronPort-HdrOrdr: A9a23:OxE5i6m4Ga/mgFKAEUJ/pfW4FVrpDfId3DAbv31ZSRFFG/Fw9v rCoB1/73SftN9/YgBEpTn+AtjlfZq+z/JICOsqTNWftWDd0QOVxYhZnOzf6jfrchefygb6uJ 0PT4FkBMT0HRxmi6/BkWqFOsw9y9qK+r3Av4jj5mpqJDsKV51d X-Talos-CUID: 9a23:rc3nZmE0q2lgnd5MqmI2xVNNJt0eaUTQ40XXI2miC0t7QZe8HAo= X-Talos-MUID: 9a23:zAh7bQUNmWGaCl3q/BS9vghCLd0y2J+zKGAXi6QDss+VDzMlbg== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="376976119" Received: from rcdn-l-core-03.cisco.com ([173.37.255.140]) by alln-iport-4.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-03.cisco.com (Postfix) with ESMTPS id 8445C180001EF; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 0A6A7CC1290; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:49 +0000 Subject: [PATCH net-next v3 3/7] enic: Save resource counts we read from HW Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-3-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=4213; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=iwLqBIaYDt+zF0jzy60eoatbh4XPWQr7YNCqVTD0bdY=; b=AgPz3O7EytTb8uH92kySJ21e+kkNE0RqwJ6lnirKH9lDMrzeSU8L2Y75GHmphSldJa1u43/bw 90Ehdc3/qCpCAF8D5FjsX6iFYkRT0NENN6Qf/ePFnRBHCAd+1m3IYrK X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: rcdn-l-core-03.cisco.com X-Patchwork-Delegate: kuba@kernel.org Save the resources counts for wq,rq,cq, and interrupts in *_avail variables so that we don't lose the information when adjusting the counts we are actually using. Report the wq_avail and rq_avail as the channel maximums in 'ethtool -l' output. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 4 ++++ drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 ++-- drivers/net/ethernet/cisco/enic/enic_res.c | 19 ++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index ac7236f76a51bf32e7060ee0482b41fe82b60b44..1f32413a8f7c690060fe385b50f7447943e72596 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -206,23 +206,27 @@ struct enic { /* work queue cache line section */ ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; /* interrupt resource cache line section */ ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; u32 rx_copybreak; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index 909d6f7000e160cf2e15de4660c1034cad7d51ba..d607b4f0542ceaef09e9528a591ca27177986143 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -695,8 +695,8 @@ static void enic_get_channels(struct net_device *netdev, switch (vnic_dev_get_intr_mode(enic->vdev)) { case VNIC_DEV_INTR_MODE_MSIX: - channels->max_rx = ENIC_RQ_MAX; - channels->max_tx = ENIC_WQ_MAX; + channels->max_rx = min(enic->rq_avail, ENIC_RQ_MAX); + channels->max_tx = min(enic->wq_avail, ENIC_WQ_MAX); channels->rx_count = enic->rq_count; channels->tx_count = enic->wq_count; break; diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 72b51e9d8d1a26a2cd18df9c9d702e5b11993b70..1261251998330c8b8363c4dd2db1ccc25847476c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -187,16 +187,21 @@ void enic_free_vnic_resources(struct enic *enic) void enic_get_res_counts(struct enic *enic) { - enic->wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); - enic->rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); - enic->cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); - enic->intr_count = vnic_dev_get_res_count(enic->vdev, - RES_TYPE_INTR_CTRL); + enic->wq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); + enic->rq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); + enic->cq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); + enic->intr_avail = vnic_dev_get_res_count(enic->vdev, + RES_TYPE_INTR_CTRL); + + enic->wq_count = enic->wq_avail; + enic->rq_count = enic->rq_avail; + enic->cq_count = enic->cq_avail; + enic->intr_count = enic->intr_avail; dev_info(enic_get_dev(enic), "vNIC resources avail: wq %d rq %d cq %d intr %d\n", - enic->wq_count, enic->rq_count, - enic->cq_count, enic->intr_count); + enic->wq_avail, enic->rq_avail, + enic->cq_avail, enic->intr_avail); } void enic_init_vnic_resources(struct enic *enic) From patchwork Fri Nov 8 21:47:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868984 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D85B819A28D; Fri, 8 Nov 2024 21:49:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102562; cv=none; b=JFB0nBN220wZqLQFD7WjJ11nkgKJG+sPEjeYHxhuKho6D+L43IO5zjSI+hrCBjH2oh2dvYEgmq+BiQP4W0UFXAwU/U3K9muOTcox7TnCmRAzG8KTO/Cq2mFEZ0SUu3BJno5wgPDW3ZiLBqA9UZk4oVLRHSCedeINLcfBzdOxWbc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102562; c=relaxed/simple; bh=zG0pnFfmCDMAJ+eK4a9AAcJEzHFseuHYHQmT/1eBiyY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=nM+vQG0vVU/t5UXhQmgYrVrKnVLncYGWYJB2ldfZRv3OjTVJvGpQ6VLpDlAuYPfGBISzrZKogfafP3uZSWjqHDHuUiD14qccitv42KOP61eAhd7DYoTmzmSQaaVmzTA/gySmFEfvbdidX6Pmvi/HM8QLTujXuzvsII07guWlGDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=GFcrYCKq; arc=none smtp.client-ip=173.37.86.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="GFcrYCKq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=6960; q=dns/txt; s=iport; t=1731102560; x=1732312160; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=R2A01fl2qoObns+t9iSIOEepQAEKaqoMSZq4LZWXw+Y=; b=GFcrYCKq4bIvByDjmRYKM0SXvH8qfH3SVxIToG3ehVhuvCl7MVwTGMUd u00/fJPV//G6ilDA9dM9HeZIp2Jz0i2gBS2SHwwNKttNjJLSuZuBl/ndb NS8jNw2G86N9a0m5KWHNhwe4GsnaTKWH9wYAlQBZvs9wP6oaNXejKn9Wq E=; X-CSE-ConnectionGUID: oMyOmn41R2uAvifhKTx5vw== X-CSE-MsgGUID: T5Fapq/nQMSOwOv9yGwxjQ== X-IPAS-Result: A0AHAACvhi5n/4v/Ja1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYJKgVBCSIRViB2HMIIhi3WSIxSBEQNWDwEBAQ9EBAEBhQcCijoCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFgQ4ThgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBden8zgQGEe9k4gW2BGi4BiEsBgWyDfTuEPCcbgUlEgRWBO4E+b4Qqg3SCaQSCRoNUUnYlgRMCAgIHAodzW5d3CT+BBRwDWSERAVUTDQoLBwVjWD4DIm9pXHorgQ6BFzpDgTuBIi8bIQtcgTiBGQEUBhUEgQ5BP4JKaUs3Ag0CNoIkJFmCT4UdhG+EaIISHUADCxgNSBEsNQYOGwY9AW4HnilGgyYHgQ+BJgcDYyxjkloKg3CNPYIfn0yEJKFZM6pNmHcio01OhGaBZzyBWTMaCBsVgyJSGQ+OLRYWkwABtUBDNTsCBwsBAQMJkhkBAQ IronPort-Data: A9a23:SU3Z06y0f38awrJWrWp6t+flxyrEfRIJ4+MujC+fZmUNrF6WrkVWy mMbXDvTPv6IM2KhKoh1Poiy9R4C65CBytQ3SgNtqFhgHilAwSbn6Xt1DatR0we6dJCroJdPt p1GAjX4BJlqCCea/lH1b+CJQUBUjcmgXqD7BPPPJhd/TAplTDZJoR94kobVuKYw6TSCK13L4 ImaT/H3Ygf/h2ctajJMt8pvlTs21BjMkGJA1rABTagjUG/2zxE9EJ8ZLKetGHr0KqE8NvK6X evK0Iai9Wrf+Ro3Yvv9+losWhRXKlJ6FVHmZkt+A8BOsDAbzsAB+vpT2M4nVKtio27hc+adZ zl6ncfYpQ8BZsUgkQmGOvVSO3kW0aZuoNcrLZUj2CCe5xWuTpfi/xlhJEYTLKod/LouOztT3 OIZJjAuUgDemf3jldpXSsE07igiBNPgMIVavjRryivUSK58B5vCWK7No9Rf2V/chOgXQq2YP JVfM2cyKk2bMnWjOX9PYH46tOGli2L0dDdRgFmUvqEwpWPUyWSd1ZC3b4KOJYTQFZw9ckCwm j/o9nzgRRcjEee41h2bo2ODmf/NpHauMG4VPPjinhJwu3Wfz3IeDTUaXEW2pP2+hFL4Xd9DQ 2QZ9jcrpLo/6GSkSd7yWxD+q3mB1jYcXMBVCMU55RuLx66S5ByWbkAHUzRIQN8rrsk7QXotz FDht9rvCSZir/6TRG6R+6m8qS60P24eLQcqfSYOQA0Ey8PurIE6klTESdMLOKq0iMDlXDL92 TaHqAAgiLgJy80GzaO2+RbAmT3Ejp7EUgI4+C3JUW+/qAB0foioY8qv81ez0BpbBJySQl/Eu D0PnNKTqbhUS5qMjyeKBu4KGdlF+sq4DdEVunY3d7FJythn0yfLkVx4iN2mGHpUDw== IronPort-HdrOrdr: A9a23:SSMo/a8nhLv3Vl6NBb5uk+AVI+orL9Y04lQ7vn2ZhyYlFfBw8P rPoB17737JYVkqNE3I9ersBEDEewK4yXcX2+cs1MmZLWrbUQKTRekIh7cKgQeQeREWndQz6U 4PSdkbNPTASXV3ksr+5hC1CJIDzMnvytHRuc7ui1pgUg1ubbht9ENCCgidGlBrXwUuP+tBKH Pl3Lsgm9JlEk5nFPhSwRI+LpP+m+E= X-Talos-CUID: 9a23:I/LW32mT7mbl8UxKqyJ3t34uO2/XOXr43VDVJRPhMzx0YaOUR1+zxLM5jdU7zg== X-Talos-MUID: 9a23:GCK79w3XFc3VZzLSbHnJr8o8hjUj3/mAAVgU1pM/nvKjKgtuaiaykyS1e9py X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="270370281" Received: from rcdn-l-core-02.cisco.com ([173.37.255.139]) by rcdn-iport-2.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-02.cisco.com (Postfix) with ESMTPS id 88D7918000222; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 101FBCC12A5; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:50 +0000 Subject: [PATCH net-next v3 4/7] enic: Allocate arrays in enic struct based on VIC config Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-4-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=7119; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=zG0pnFfmCDMAJ+eK4a9AAcJEzHFseuHYHQmT/1eBiyY=; b=SRxdr4iquTxQOmdNqsM2pV80RA6xBTGJQsUEhfZJNhq4tcvimE3INgnAQIsSHrO9JGUL/Gh2H GNrdd7nCBDnAxNZMettCdDp+JQFnPxkuKnnvJYdeJsGj7OE1q9lxF4M X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: rcdn-l-core-02.cisco.com X-Patchwork-Delegate: kuba@kernel.org Allocate wq, rq, cq, intr, and napi arrays based on the number of resources configured in the VIC. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 24 ++++----- drivers/net/ethernet/cisco/enic/enic_main.c | 84 ++++++++++++++++++++++++++--- 2 files changed, 87 insertions(+), 21 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 1f32413a8f7c690060fe385b50f7447943e72596..cfb4667953de2c578911aad138a1d392fa9d3bdc 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -23,10 +23,8 @@ #define ENIC_BARS_MAX 6 -#define ENIC_WQ_MAX 8 -#define ENIC_RQ_MAX 8 -#define ENIC_CQ_MAX (ENIC_WQ_MAX + ENIC_RQ_MAX) -#define ENIC_INTR_MAX (ENIC_CQ_MAX + 2) +#define ENIC_WQ_MAX 256 +#define ENIC_RQ_MAX 256 #define ENIC_WQ_NAPI_BUDGET 256 @@ -184,8 +182,8 @@ struct enic { struct work_struct reset; struct work_struct tx_hang_reset; struct work_struct change_mtu_work; - struct msix_entry msix_entry[ENIC_INTR_MAX]; - struct enic_msix_entry msix[ENIC_INTR_MAX]; + struct msix_entry *msix_entry; + struct enic_msix_entry *msix; u32 msg_enable; spinlock_t devcmd_lock; u8 mac_addr[ETH_ALEN]; @@ -204,28 +202,24 @@ struct enic { bool enic_api_busy; struct enic_port_profile *pp; - /* work queue cache line section */ - ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + struct enic_wq *wq; unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; - /* receive queue cache line section */ - ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + struct enic_rq *rq; unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; - struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; + struct napi_struct *napi; - /* interrupt resource cache line section */ - ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + struct vnic_intr *intr; unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ - /* completion queue cache line section */ - ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + struct vnic_cq *cq; unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index eb00058b6c68ec5c1ac433b54b5bc6f3fb613777..564202e81a711a6791bef7e848627f0a439cc6f3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -940,7 +940,7 @@ static void enic_get_stats(struct net_device *netdev, net_stats->rx_errors = stats->rx.rx_errors; net_stats->multicast = stats->rx.rx_multicast_frames_ok; - for (i = 0; i < ENIC_RQ_MAX; i++) { + for (i = 0; i < enic->rq_count; i++) { struct enic_rq_stats *rqs = &enic->rq[i].stats; if (!enic->rq[i].vrq.ctrl) @@ -1792,7 +1792,7 @@ static void enic_free_intr(struct enic *enic) free_irq(enic->pdev->irq, enic); break; case VNIC_DEV_INTR_MODE_MSIX: - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) if (enic->msix[i].requested) free_irq(enic->msix_entry[i].vector, enic->msix[i].devid); @@ -1859,7 +1859,7 @@ static int enic_request_intr(struct enic *enic) enic->msix[intr].isr = enic_isr_msix_notify; enic->msix[intr].devid = enic; - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) enic->msix[i].requested = 0; for (i = 0; i < enic->intr_count; i++) { @@ -2456,8 +2456,7 @@ static int enic_set_intr_mode(struct enic *enic) * (the last INTR is used for notifications) */ - BUG_ON(ARRAY_SIZE(enic->msix_entry) < n + m + 2); - for (i = 0; i < n + m + 2; i++) + for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; /* Use multiple RQs if RSS is enabled @@ -2674,6 +2673,71 @@ static const struct netdev_stat_ops enic_netdev_stat_ops = { .get_base_stats = enic_get_base_stats, }; +static void enic_free_enic_resources(struct enic *enic) +{ + kfree(enic->wq); + enic->wq = NULL; + + kfree(enic->rq); + enic->rq = NULL; + + kfree(enic->cq); + enic->cq = NULL; + + kfree(enic->napi); + enic->napi = NULL; + + kfree(enic->msix_entry); + enic->msix_entry = NULL; + + kfree(enic->msix); + enic->msix = NULL; + + kfree(enic->intr); + enic->intr = NULL; +} + +static int enic_alloc_enic_resources(struct enic *enic) +{ + enic->wq = kcalloc(enic->wq_avail, sizeof(struct enic_wq), GFP_KERNEL); + if (!enic->wq) + goto free_queues; + + enic->rq = kcalloc(enic->rq_avail, sizeof(struct enic_rq), GFP_KERNEL); + if (!enic->rq) + goto free_queues; + + enic->cq = kcalloc(enic->cq_avail, sizeof(struct vnic_cq), GFP_KERNEL); + if (!enic->cq) + goto free_queues; + + enic->napi = kcalloc(enic->wq_avail + enic->rq_avail, + sizeof(struct napi_struct), GFP_KERNEL); + if (!enic->napi) + goto free_queues; + + enic->msix_entry = kcalloc(enic->intr_avail, sizeof(struct msix_entry), + GFP_KERNEL); + if (!enic->msix_entry) + goto free_queues; + + enic->msix = kcalloc(enic->intr_avail, sizeof(struct enic_msix_entry), + GFP_KERNEL); + if (!enic->msix) + goto free_queues; + + enic->intr = kcalloc(enic->intr_avail, sizeof(struct vnic_intr), + GFP_KERNEL); + if (!enic->intr) + goto free_queues; + + return 0; + +free_queues: + enic_free_enic_resources(enic); + return -ENOMEM; +} + static void enic_dev_deinit(struct enic *enic) { unsigned int i; @@ -2691,6 +2755,7 @@ static void enic_dev_deinit(struct enic *enic) enic_free_vnic_resources(enic); enic_clear_intr_mode(enic); enic_free_affinity_hint(enic); + enic_free_enic_resources(enic); } static void enic_kdump_kernel_config(struct enic *enic) @@ -2734,6 +2799,12 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + err = enic_alloc_enic_resources(enic); + if (err) { + dev_err(dev, "Failed to allocate enic resources\n"); + return err; + } + /* modify resource count if we are in kdump_kernel */ enic_kdump_kernel_config(enic); @@ -2746,7 +2817,7 @@ static int enic_dev_init(struct enic *enic) if (err) { dev_err(dev, "Failed to set intr mode based on resource " "counts and system capabilities, aborting\n"); - return err; + goto err_out_free_vnic_resources; } /* Allocate and configure vNIC resources @@ -2788,6 +2859,7 @@ static int enic_dev_init(struct enic *enic) enic_free_affinity_hint(enic); enic_clear_intr_mode(enic); enic_free_vnic_resources(enic); + enic_free_enic_resources(enic); return err; } From patchwork Fri Nov 8 21:47:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868991 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-4.cisco.com (alln-iport-4.cisco.com [173.37.142.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 248061A9B33; Fri, 8 Nov 2024 21:50:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.91 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102625; cv=none; b=UutahZTwZqAmIQ8alZ0OrREdleCV9jGFuA3BT08k5HkGuySI8DVKECMpIy9qIhFWLj5e58u1d7wbs8rJuUQ2p+VbaKGd11ApFBskMHDITJ53rO1xBT4SfoBXy1upvJ9442Nsbg942r+v41rJh4wsp8MR6f8AIefy0K4L/azgoWM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102625; c=relaxed/simple; bh=mninbMArocvS6AE+lwVCGhfK4ejOfQE8gCkKXB/dNtQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=G29uzmu10Uf4yH4IuY8PC6lvCEO3kNkrEz5qgjqrMKDKjhUDvj1tRahHhd74s9Cdbz36D7qYSbcVg0QKea6G+ayRJa0ZoeY+kefgvsjqAyqVvnxKbHpcodV28QS/tvvrYG4VOXUauiEMK9S1XHKvXlFmpe+oEgwQ71AVvSA4K2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=FKbzNgeY; arc=none smtp.client-ip=173.37.142.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="FKbzNgeY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=5848; q=dns/txt; s=iport; t=1731102623; x=1732312223; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=I7OTkuoFHF/S5MDa31Q8DP6vHJIajVzvqL+oz+KRYPM=; b=FKbzNgeYmQdeBb+M5pSTZzrMPVdRzbV49Xumwja0x4GX4qZFb7Lp3+3g /7aPvOcfKT2aQty7ogJIQWVBpnWTogoR3hxVyWH5heUWEIDSrwcRZIY+C SPPmA79iQR8kOPK75S4p8wCb8xii0/VuMZ+Jo74HeOqRHour+FCbPuO6q s=; X-CSE-ConnectionGUID: Lkfx0kXGQ8KIV4nzxTet2g== X-CSE-MsgGUID: mhhIWeriTIiA/O7A37fHWg== X-IPAS-Result: A0AGAAA5hi5nj4//Ja1aGwEBAQEBAQEBBQEBARIBAQEDAwEBAUCBPwYBAQELAYQaQkiEVYgdhzCOFpIjgSUDVg8BAQEPRAQBAYUHAoo6AiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBAQELAQEFAQEBAgEHBRQBAQEBAQE5BUmGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsFp6fzOBAd4zgW2BGi4BiEsBgWyDfTuEPCcbgUlEgRQBg2iEG4QDgmkEhDEDgjh2JYkVW5d3CT+BBRwDWSERAVUTDQoLBwVjWD4DIm9pXHorgQ6BFzpDgTuBIi8bIQtcgTiBGhQGFQSBDkE/gkppSzcCDQI2giQkWYJPhR2Eb4RoghIdQAMLGA1IESw1Bg4bBj0BbgeeKUaDLYEOgScHgXWSa4NpjT2ha4QkoVkzqk2YdyKkG4RmgWc6gVszGggbFYMiUhkPji0NCRaTAAG1OkM1OwIHCwEBAwmQHYF8AQE IronPort-Data: A9a23:rpd7CKLBSrY9dpIcFE+RK5UlxSXFcZb7ZxGr2PjKsXjdYENS1mcBm DZMWTiObqneY2T0e4siOorl9B5XucDVzdRgSFAd+CA2RRqmiyZq6fd1j6vUF3nPRiEWZBs/t 63yUvGZcoZsCCea/kr1WlTYhSEU/bmSQbbhA/LzNCl0RAt1IA8skhsLd9QR2uaEuvDnRVrX0 T/Oi5eHYgP8gmYkaj58B5+r8XuDgtyj4Fv0gXRmDRx7lAe2v2UYCpsZOZawIxPQKqFIHvS3T vr017qw+GXU5X8FUrtJRZ6iLyXm6paLVeS/oiI+t5qK23CulQRuukoPD8fwXG8M49m/c3+d/ /0W3XC4YV9B0qQhA43xWTEAe811FfUuFLMqvRFTvOTLp3AqfUcAzN1KD08mZY4o1d8nIkBwq foKOAwAckqM0rfeLLKTEoGAh+w5J8XteYdasXZ6wHSBU7AtQIvIROPB4towMDUY358VW62BI ZBENHw2ME+ojx5nYj/7DLo4keqzjX71ehVTqUmeouw85G27IAlZi+ezYYWLI4fSLSlTtkOYp kP58n74PjAbHZu4lhPerXyMueCayEsXX6pJSeXnraQ16LGJ/UQfAQMbUHO3qOe0j0q5Vc4ZL UEIkgIjobU3/V6mUvHyWBq3pHPCtRkZM/JTDuczwAKA0KzZ50CeHGdsZjdHZMYrq4wwSCAm2 0Ghm87vA3pksNW9UXuX+7GVhSm/NSgcMSkJYipsZQ8E5cTz5YI+lBTCSv59H6OvyN74Azf9x 3aNtidWr7MekcIGyY2l8l3dxTGhvJ7ESkgy/Aq/Y46+xhlyaIjgY8mj7kLWqK4eao2YVVKG+ nMDnqBy8dziE7mplyKdGOY9M4j5xNe7bDvOkWJiA8MuomHFF2GYQahc5zR3JUFMO8kCeCP0b EK7he+3zMEOVJdNRfErC79dG/gXIb7c+cMJv804j+aigLAtJWdrHwk3OSZ8OlwBdmB3z8nT3 r/AL66R4Y4yU/gP8dZPb751PUUX7i4/33jPYpvw0g6q17GTDFbMFuxcaAPSNLFhvPvZyOkwz zq5H5XSo/m4eLCuChQ7DaZKcDjm0FBiX8mv8J0NHgJ9ClA7Qjt8YxMu/V/RU9c4x/sOzLigE oCVUU5Dw123nmzcNQiPcThibripNauTXlplVRHAyW2AgiB5Ca72tf93X8JuLdEPqrc5pdYqF KZtRil1KqgUItgx02hGNcGlxGGjHTz37T+z092NO2hlJsI5HFSZpLcJvGLHrUEzM8Z+juNmy 5XI6+8RacNrq9hKZCoOVM+S8g== IronPort-HdrOrdr: A9a23:wuFh66rRTPx6lEB2rSf7kDMaV5oaeYIsimQD101hICG9E/b0qy nKpp9w6faaskd1ZJheo6HlBEDiex/hHPxOkOss1N6ZNWHbUWKTXeZfxIHpqgeBJ8VZm9Qy6Y 56f7F6GJnsCzFB/KXHyROlGNUtysTvys+VrPrZpk0NcT1X X-Talos-CUID: 9a23:KcIu5GgHygZ0482E4tK5Dq0MLzJuVn7T3C/TE2CBWUVIduGTW0+s9b8+nJ87 X-Talos-MUID: 9a23:ZX/yLQ4JaH7JI81D2CBoHxhoxoxt46SPN00XkKknuuDDEnNZYGiTrAu4F9o= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="376976130" Received: from rcdn-l-core-06.cisco.com ([173.37.255.143]) by alln-iport-4.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-06.cisco.com (Postfix) with ESMTPS id 19D451800025E; Fri, 8 Nov 2024 21:49:13 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 1831ECC12AC; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:51 +0000 Subject: [PATCH net-next v3 5/7] enic: Adjust used MSI-X wq/rq/cq/interrupt resources in a more robust way Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-5-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=6178; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=mninbMArocvS6AE+lwVCGhfK4ejOfQE8gCkKXB/dNtQ=; b=+yzgWnRSapdfT2l++y6UIMmE3mJebyXBv5qnv4mTkZMCih1qaaQcB7id4AzRO2vAFsQcf8Ffg HBolPohbrD8CDnOKcEKNGolYk9TlPr+G9/GwJ5rCCPW3dBfsx4H6tnv X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: rcdn-l-core-06.cisco.com X-Patchwork-Delegate: kuba@kernel.org Instead of failing to use MSI-X if resources aren't configured exactly right, use the resources we do have. Since we could start using large numbers of rq resources, we do limit the rq count to what netif_get_num_default_rss_queues() recommends. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic_main.c | 116 ++++++++++++++-------------- 1 file changed, 57 insertions(+), 59 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 564202e81a711a6791bef7e848627f0a439cc6f3..8b07899462d0671843579d16c8c935d9ebbe447b 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2442,85 +2442,86 @@ static void enic_tx_hang_reset(struct work_struct *work) static int enic_set_intr_mode(struct enic *enic) { - unsigned int n = min_t(unsigned int, enic->rq_count, ENIC_RQ_MAX); - unsigned int m = min_t(unsigned int, enic->wq_count, ENIC_WQ_MAX); unsigned int i; + int num_intr; /* Set interrupt mode (INTx, MSI, MSI-X) depending * on system capabilities. * - * Try MSI-X first + * We need a minimum of 1 RQ, 1 WQ, and 2 CQs * - * We need n RQs, m WQs, n+m CQs, and n+m+2 INTRs - * (the second to last INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) */ - for (i = 0; i < enic->intr_avail; i++) - enic->msix_entry[i].entry = i; - - /* Use multiple RQs if RSS is enabled - */ - - if (ENIC_SETTING(enic, RSS) && - enic->config.intr_mode < 1 && - enic->rq_count >= n && - enic->wq_count >= m && - enic->cq_count >= n + m && - enic->intr_count >= n + m + 2) { - - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - n + m + 2, n + m + 2) > 0) { - - enic->rq_count = n; - enic->wq_count = m; - enic->cq_count = n + m; - enic->intr_count = n + m + 2; - - vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - - return 0; - } + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; } + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; + + /* Try MSI-X first */ if (enic->config.intr_mode < 1 && - enic->rq_count >= 1 && - enic->wq_count >= m && - enic->cq_count >= 1 + m && - enic->intr_count >= 1 + m + 2) { - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - 1 + m + 2, 1 + m + 2) > 0) { - - enic->rq_count = 1; - enic->wq_count = m; - enic->cq_count = 1 + m; - enic->intr_count = 1 + m + 2; + enic->intr_avail >= ENIC_MSIX_MIN_INTR) { + unsigned int max_queues; + unsigned int rq_default; + unsigned int rq_avail; + unsigned int wq_avail; + + for (i = 0; i < enic->intr_avail; i++) + enic->msix_entry[i].entry = i; + + num_intr = pci_enable_msix_range(enic->pdev, enic->msix_entry, + ENIC_MSIX_MIN_INTR, + enic->intr_avail); + if (num_intr > 0) { + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_default = netif_get_num_default_rss_queues(); + rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + + if (wq_avail + rq_avail <= max_queues) { + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - + VNIC_DEV_INTR_MODE_MSIX); + enic->intr_avail = num_intr; return 0; } } /* Next try MSI * - * We need 1 RQ, 1 WQ, 2 CQs, and 1 INTR + * We need 1 INTR */ if (enic->config.intr_mode < 2 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 1 && + enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { enic->rq_count = 1; enic->wq_count = 1; enic->cq_count = 2; enic->intr_count = 1; - + enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); return 0; @@ -2528,23 +2529,20 @@ static int enic_set_intr_mode(struct enic *enic) /* Next try INTx * - * We need 1 RQ, 1 WQ, 2 CQs, and 3 INTRs + * We need 3 INTRs * (the first INTR is used for WQ/RQ) * (the second INTR is used for WQ/RQ errors) * (the last INTR is used for notifications) */ if (enic->config.intr_mode < 3 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 3) { + enic->intr_avail >= 3) { enic->rq_count = 1; enic->wq_count = 1; enic->cq_count = 2; enic->intr_count = 3; - + enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); return 0; @@ -2762,8 +2760,8 @@ static void enic_kdump_kernel_config(struct enic *enic) { if (is_kdump_kernel()) { dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_count = 1; - enic->wq_count = 1; + enic->rq_avail = 1; + enic->wq_avail = 1; enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; enic->config.mtu = min_t(u16, 1500, enic->config.mtu); From patchwork Fri Nov 8 21:47:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868988 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-8.cisco.com (alln-iport-8.cisco.com [173.37.142.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D25131F26CD; Fri, 8 Nov 2024 21:50:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.95 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102623; cv=none; b=G0142/dvinu3tHvDpiHHG8RjV4kAEhHLX7TfW7vbWzy+FTa2JR5Vf+DdVcsuXu6s+cPeJWzkDRBpoX8fU57+SyV+Z7YEkWGJKeYEoOtT6gK/sRlBZk+pQeZqTOmozIv/58IuZHQtF6IPS5zz6LH+nKZ0Yu8t2YHtPClUO/NyvSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102623; c=relaxed/simple; bh=QtIVFbVGBYj5JREn7vN0UZu7VcWRmQYTsiHdSKZGm+M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MuIqQwa5mizBfKBzp4uzdKi7hmg2JnQwtZrPrQrdcQ/TVN1AchR7p8XQHgQmK05RRFU3f29DGA1DLHZX0PTJ7LbPhmR2WCEA03yXvqNfeoI847vHSAbLg2+vvS5xtmmAMWCUZikRCr8V8yvsaLCTtP4mQQCN9KWptjO+oESwTQU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=lErp51ra; arc=none smtp.client-ip=173.37.142.95 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="lErp51ra" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=6191; q=dns/txt; s=iport; t=1731102622; x=1732312222; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=PyWtOUrxpB55r8TtBgwj7aFJpNi+G9Fo6NJ2RK4oWeo=; b=lErp51rasrO2cZsDr1Gji4q8fr0T4lq+3DWbjI2zxvqfWAZX7yK3U04E tENf18jo315ZhnERBZTPQGY4eETMZRzzIu/xHJ4tQgptlGjUmEDLNANx1 Lx+AvqeUx53ece5TtsjS7BuPotLs0p9jGra9P/TQZjxB67724nUc4WC5n 8=; X-CSE-ConnectionGUID: reZh0nw+RU2XBF6fwJDOpQ== X-CSE-MsgGUID: Wg4NDScEQ56GiIhbCB96Vw== X-IPAS-Result: A0ACAAA5hi5nj47/Ja1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYQaQkiEVYgdhzCOFpIjgSUDVg8BAQEPRAQBAYUHAoo6AiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBAQELAQEFAQEBAgEHBRQBAQEBAQE5BUmGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsFp6fzOBAYR72TiBbYEaLgGISwGBbIN9O4Q8JxuBSUSBFINpiB6CaQSHYiWJFZhSCT+BBRwDWSERAVUTDQoLBwVjWD4DIm9pXHorgQ6BFzpDgTuBIi8bIQtcgTiBGQEUBhUEgQ5BP4JKaUs3Ag0CNoIkJFmCT4UdhG+EaIISHUADCxgNSBEsNQYOGwY9AW4HnilGgyYHgQ6BMZRMEYNpjT2CH59MhCShWTOqTZh3IqQbhGaBZzqBWzMaCBsVgyJSGQ+OLQ0JFpMAAbU6QzU7AgcLAQEDCZIZAQE IronPort-Data: A9a23:KF21Iq/Pln6xmARr5pOSDrUDen6TJUtcMsCJ2f8bNWPcYEJGY0x3z mNLD2zUa/qIYGbxcoonPNu0pktUsJaByIIyGQtr/H9EQiMRo6IpJzg2wmQcns+2BpeeJK6yx 5xGMrEsFOhtEDmE4E/rauW5xZVF/fngbqLmD+LZMTxGSwZhSSMw4TpugOdRbrRA2bBVOCvT/ 4qoyyHjEAX9gWItazpEs/7rRC5H5ZwehhtJ5jTSWtgT1LPuvyF9JI4SI6i3M0z5TuF8dsamR /zOxa2O5WjQ+REgELuNyt4XpWVTH9Y+lSDX4pZnc/DKbipq/0Te4Y5nXBYoUnq7vh3S9zxHJ HqhgrTrIeshFvWkdO3wyHC0GQkmVUFN0OevzXRSLaV/wmWeG0YAzcmCA2ksML8YouBoQltzr 8wyBm4DcTKvoPqPlefTpulE3qzPLeHxN48Z/3UlxjbDALN+HNbIQr7B4plT2zJYasJmRKmFI ZFGL2AyMVKZP0cn1lQ/UPrSmM+ui3TkeDpSoXqepLE85C7YywkZPL3Fa4WKIYDTHZkK9qqej l3Jo0LVORo3DYHB5juk8niHrL6RozyuDer+E5Xjq6Y12wfMroAJMzUaXEW2pNG1g1CzXtZYJ VBS/CcyxYA/+FGuR8vwQzW3p3mLuhNaUN1Ve8U59QuE4qnZ+QCUAi4DVDEpQNUguNU7Wn8s2 0OFks3BASFptvueSRq17r6eoDWzETIYIW8LeWkPSg5ty93ippwjyxHCVNBuFIargdDvXzL92 TaHqG45nbp7pcgGy6m243jZjD+24JvEVAg44kPQRG3N0+9iTJSua4rt7R3Q6uxNad7ECFKAp 3MD3cOZ6YjiEK1higSXTdcNRLe2pMy4ag32rwBhALsN0CyUrivLkZ9r3BlyI0JgM8AhcDDvY VPOtQ452HO1FCX3BUOQS9ztY/nG3ZTd+cLZuuc4h+eig6SdlifapEmChmbJgwgBdXTAd4lka f93lu73Ux4n5VxPlmbeegvk+eZDKtoC7W3SX4vn6B+szKCTYnWYIZ9cbwDXNrhltvLc8F2Im zq6Cyds40sHOAEZSnSHmbP/0XhTcBDX+Liv8ZUOLb/ZSuaYMDt8UaWMqV/eR2CVt/8IzriTp C7Vtr5ww1vkjnqPMhSRdn1mc/vuW505xU/XzgRyVWtEL0MLON71hI9GLsNfVeB+pIRLk6UuJ 9FbIJroPxi6Ymick9jrRcWm9NQ6HPlq7CrSVxeYjM8XJM46HlCQoYe0IGMCNkAmV0KKiCf3m JX4viuzfHbJb14K4Br+AB53826MgA== IronPort-HdrOrdr: A9a23:l09c/6BFYkHZ+FblHemd55DYdb4zR+YMi2TDtnoddfUxSKfzqy nApoV/6faKskd0ZJhCo7y90cu7MBHhHPdOiOEs1L6ZLW7bkWGjRbsD0WNFrgePJwTk+vdZxe N8dcFFeb7NJEJnhsX36hTQKbkd6cSAmZrIudvj X-Talos-CUID: 9a23:IKFN5mAsSTxtuDr6EzNqxk5MHN9mSECH/F3+fUGnK0pOUITAHA== X-Talos-MUID: 9a23:KxsisAtyv1NA/vEdAc2nuz4+EutT6ImXVR4ikLQGseKuZHJOJGLI X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="377341047" Received: from rcdn-l-core-05.cisco.com ([173.37.255.142]) by alln-iport-8.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-05.cisco.com (Postfix) with ESMTPS id 1921C1800022F; Fri, 8 Nov 2024 21:49:13 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 1EEEECC12AD; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:52 +0000 Subject: [PATCH net-next v3 6/7] enic: Move enic resource adjustments to separate function Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-6-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=6449; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=QtIVFbVGBYj5JREn7vN0UZu7VcWRmQYTsiHdSKZGm+M=; b=XuxHNRo6IxLMgAmjKzcgjAstDXlIOQliLABTauG6dkMfNOd+G3BExcHiWcHrIesERB8scLzWI UlsVcJID4a5D/6zoM8caD+at4Ca1NviY2Lp4BpkRmW2/WpV7ENU/pWv X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Patchwork-Delegate: kuba@kernel.org Move the enic resource adjustments out of enic_set_intr_mode() and into its own function, enic_adjust_resources(). Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic_main.c | 127 +++++++++++++++------------- 1 file changed, 70 insertions(+), 57 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 8b07899462d0671843579d16c8c935d9ebbe447b..84e85c9e2bf52f0089c0a8d03fb6d22ada4d086c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2448,30 +2448,11 @@ static int enic_set_intr_mode(struct enic *enic) /* Set interrupt mode (INTx, MSI, MSI-X) depending * on system capabilities. * - * We need a minimum of 1 RQ, 1 WQ, and 2 CQs - * + * Try MSI-X first */ - if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { - dev_err(enic_get_dev(enic), - "Not enough resources available rq: %d wq: %d cq: %d\n", - enic->rq_avail, enic->wq_avail, - enic->cq_avail); - return -ENOSPC; - } - - /* if RSS isn't set, then we can only use one RQ */ - if (!ENIC_SETTING(enic, RSS)) - enic->rq_avail = 1; - - /* Try MSI-X first */ if (enic->config.intr_mode < 1 && enic->intr_avail >= ENIC_MSIX_MIN_INTR) { - unsigned int max_queues; - unsigned int rq_default; - unsigned int rq_avail; - unsigned int wq_avail; - for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; @@ -2479,28 +2460,6 @@ static int enic_set_intr_mode(struct enic *enic) ENIC_MSIX_MIN_INTR, enic->intr_avail); if (num_intr > 0) { - wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); - rq_default = netif_get_num_default_rss_queues(); - rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); - max_queues = min(enic->cq_avail, - enic->intr_avail - ENIC_MSIX_RESERVED_INTR); - - if (wq_avail + rq_avail <= max_queues) { - enic->rq_count = rq_avail; - enic->wq_count = wq_avail; - } else { - /* recalculate wq/rq count */ - if (rq_avail < wq_avail) { - enic->rq_count = min(rq_avail, max_queues / 2); - enic->wq_count = max_queues - enic->rq_count; - } else { - enic->wq_count = min(wq_avail, max_queues / 2); - enic->rq_count = max_queues - enic->wq_count; - } - } - enic->cq_count = enic->rq_count + enic->wq_count; - enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; - vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSIX); enic->intr_avail = num_intr; @@ -2516,14 +2475,8 @@ static int enic_set_intr_mode(struct enic *enic) if (enic->config.intr_mode < 2 && enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 1; enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); - return 0; } @@ -2537,14 +2490,8 @@ static int enic_set_intr_mode(struct enic *enic) if (enic->config.intr_mode < 3 && enic->intr_avail >= 3) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 3; enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); - return 0; } @@ -2569,6 +2516,67 @@ static void enic_clear_intr_mode(struct enic *enic) vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_UNKNOWN); } +static int enic_adjust_resources(struct enic *enic) +{ + unsigned int max_queues; + unsigned int rq_default; + unsigned int rq_avail; + unsigned int wq_avail; + + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; + } + + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; + + switch (vnic_dev_get_intr_mode(enic->vdev)) { + case VNIC_DEV_INTR_MODE_INTX: + case VNIC_DEV_INTR_MODE_MSI: + enic->rq_count = 1; + enic->wq_count = 1; + enic->cq_count = 2; + enic->intr_count = enic->intr_avail; + break; + case VNIC_DEV_INTR_MODE_MSIX: + /* Adjust the number of wqs/rqs/cqs/interrupts that will be + * used based on which resource is the most constrained + */ + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_default = netif_get_num_default_rss_queues(); + rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + if (wq_avail + rq_avail <= max_queues) { + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; + + break; + default: + dev_err(enic_get_dev(enic), "Unknown interrupt mode\n"); + return -EINVAL; + } + + return 0; +} + static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { @@ -2807,9 +2815,7 @@ static int enic_dev_init(struct enic *enic) */ enic_kdump_kernel_config(enic); - /* Set interrupt mode based on resource counts and system - * capabilities - */ + /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic); if (err) { @@ -2818,6 +2824,13 @@ static int enic_dev_init(struct enic *enic) goto err_out_free_vnic_resources; } + /* Adjust resource counts based on most constrained resources */ + err = enic_adjust_resources(enic); + if (err) { + dev_err(dev, "Failed to adjust resources\n"); + goto err_out_free_vnic_resources; + } + /* Allocate and configure vNIC resources */ From patchwork Fri Nov 8 21:47:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13868990 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-6.cisco.com (alln-iport-6.cisco.com [173.37.142.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F1E71F26C3; Fri, 8 Nov 2024 21:50:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.93 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102624; cv=none; b=Eiv0eD0HKP0tgan6BWxT9Dz0GcJhqIDwyXE/QfbLy9MJPJX3o5bKr6ETH8uOVxpjmXS3mX8rN1DFBqgzXPmjWg3IqJwchH56rCYYUfuS9NfUXdxMP9A1PZ21348Qm1vbf29FIEg6ym79mmcfAmfzkZFYKcN8yIvJDTfmVrVfQkQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731102624; c=relaxed/simple; bh=xatTfDDvuha6eMR9JG9Hd2IzFzAHGTmF0QhSbBpuJrI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=UFXWK50YpW7PG3LDrJK3fmCauKUPvR62VDegxGQ/1V9Dr7HLwvcEoXvh/+97wQFi72O3pfFxZIlPYJ5kFmF+bO0v3OPGtRgq82vJAgvwIWVhLxnPfUSYpOaLL4OonOBWWYGkwskAopF51wM4BtZP3w4QQ+3IVe8G7IaObot8Ea8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=HU1Liym6; arc=none smtp.client-ip=173.37.142.93 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="HU1Liym6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=2260; q=dns/txt; s=iport; t=1731102622; x=1732312222; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=tHLwdiP31Rtapi/mWtIkSPTBuH5ItPWzaydhVQl5G3Y=; b=HU1Liym6ltFXlIyPAK1lV8ysAlUhj0UqvQRaAJPnODNSyykMuvO6Wv9C vnU6Vy22T29+8DDXaUM5xpN6bBVFV8AVexEhlJbgVgZwk1wtbWbiv0FcR wcckCZbOiqHA8B/23lEdccCRgQJrKJirMCFkoeXL4m5kuYvdFIpYFzF5p U=; X-CSE-ConnectionGUID: nOAVb4vCRomNsiUiTuDJvA== X-CSE-MsgGUID: XGXdF/6kQfeeb3pHOIJyVg== X-IPAS-Result: A0AUAACvhi5nj5P/Ja1aHAEBAQEBAQcBARIBAQQEAQFAgT8HAQELAYQaQkiEVYgdhzCOFoxHhVyBJQNWDwEBAQ9EBAEBhQcCijoCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFFAEBAQEBATkFSYYIhlsCAQMjBFIQJQImAgIrGxAGARKDAYJlAgGwXXp/M4EBhHvZOIFtgRouAYhLAYFsg2QZO4Q8JxuBSUSEfYgegmkEhDEDgy6JOphSCT+BBRwDWSERAVUTDQoLBwVjWD4DIm9pXHorgQ6BFzpDgTuBIi8bIQtcgTiBGhQGFQSBDkE/gkppSzcCDQI2giQkWYJPhR2Eb4RoghIdQAMLGA1IESw1Bg4bBj0BbgeeKUaDLQGBDZYOg2mNPaFrhCShWTOqTS6HZJBlIqQbhGaBZzqBWzMaCBsVgyJSGQ+OLQ0JkxYBtUBDNTsCBwsBAQMJkhkBAQ IronPort-Data: A9a23:gPUPK6AJqjvVBBVW/9Pjw5YqxClBgxIJ4kV8jS/XYbTApGt3gWZTz DEZUGuCM/6KZGqhfohyPo2x80MOu5/RzNcyOVdlrnsFo1CmBibm6XV1Cm+qYkt+++WaFBoPA /02M4eGdIZsCCeB/n9BC5C5xVFkz6aEW7HgP+DNPyF1VGdMRTwo4f5Zs7ZRbrVA357gWWthh fuo+5eDYQb9i2YoWo4pw/vrRC1H7ayaVAww5jTSVdgT1HfCmn8cCo4oJK3ZBxPQXolOE+emc P3Ixbe/83mx109F5gSNy+uTnuUiG9Y+DCDW4pZkc/HKbitq+kTe5p0G2M80Mi+7vdkmc+dZk 72hvbToIesg0zaldO41C3G0GAkmVUFKFSOuzXWX6aSuI0P6n3TExdJHMktqI6Yi/eNRGmBz9 MAFBBITR0XW7w626OrTpuhEnM8vKozveYgYoHwllGmfBvc9SpeFSKLPjTNa9G5v3YYVQrCEO pdfMGYwBPjDS0Un1lM/Dp8zh+yvjHDXeDxDo1XTrq0yi4TW5FcujuKyaoqKIrRmQ+1lsXaUi CWY/F33DxwQb8S/7hun40yj07qncSTTA99KS+biqZaGmma7ymUNBRg+WVKlrPy9jUCiHdRSN yQ89yYzqKEg+VCDQd76UBm15nWDu3Y0WMdaGsU55RuLx66S5ByWbkANSDJbZcNlssIqSTE0/ luUmdWvDjwHmKWcQ3+b95+OoD+yMDRTJmgHDQcATA0Y85zgrZs1gxbnUNluCui2g8fzFDW2x CqFxAA4iqkfgNAjyaq25xbEjiiqq5yPSRQ6jjg7RUq/5Q9/IYrgbIuy5B2CtbBLLZ2SSR+Ku 31sd9WiAP4mC7akqibRXs43G7SP/Pe4MQHBrXxxNsx0n9iywEKLcYdV6TB4AU5mNMcYZDPkC HM/XysPufe/21P0McdKj5KNNig88UT3+T3YuhHogjhmPskZmOyvpX0GiausM4bFyxZEfUYXY snzTCpUJSxGYZmLNRLvLwvn7Zclxzol2UTYTo3hwhKs3NK2PSHOEO9dbAHfNrBhsctoRTk5F f4Bb6NmLD0CAIXDjtX/qNN7wa0idCJiXMun8aS7iMbZf1A8RQnN9MM9MZt6JtQ6xP4K/gs51 nq8QURfgEHunmHKLB7Ca3ZoLtvSsWVX8xoG0dgXFQ/wgRALON/3hI9GLstfVed8roRLk6UrJ 8Tpju3cWZyjvByboGxFNfEQbeVKKHyWuO55F3H1OmFjJsU4F1KhFx2NVlKHyRTixxGf7aMWy 4BMHCuCKXbfb2yO1PrrVc8= IronPort-HdrOrdr: A9a23:Bi5cFKyRciYeZNUg6hoWKrPw871zdoMgy1knxilNoNJuA6ulfq eV/MjztCWUtN9/Yh0dcLy7VZVoBEmskKKdgrN+AV7dZniEhILAFugLhuHfKn/bak/DH4Vmup uIHZIObOHYPBxWgdn35Q+gH81l4tWWmZrY/dv2/jNBQR5nbqd44xw8MAaUFUVqWBJLbKBJba Z0nvA3wQadRQ== X-Talos-CUID: 9a23:8TAtjWjX2gm9qjhLxyj9h7Up5jJudFr08n2MPWSCDmNPU7yoSGON1IF6up87 X-Talos-MUID: 9a23:C35z3w5aJfslaqC5t6VxjhzExoxYzJSiJgdTkqwItsSObjRQHBa/nQq4F9o= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,139,1728950400"; d="scan'208";a="376272870" Received: from rcdn-l-core-10.cisco.com ([173.37.255.147]) by alln-iport-6.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 08 Nov 2024 21:49:13 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by rcdn-l-core-10.cisco.com (Postfix) with ESMTPS id 896F218000274; Fri, 8 Nov 2024 21:49:13 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id 26CE0CC12B6; Fri, 8 Nov 2024 21:49:12 +0000 (GMT) From: Nelson Escobar Date: Fri, 08 Nov 2024 21:47:53 +0000 Subject: [PATCH net-next v3 7/7] enic: Move kdump check into enic_adjust_resources() Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241108-remove_vic_resource_limits-v3-7-3ba8123bcffc@cisco.com> References: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> In-Reply-To: <20241108-remove_vic_resource_limits-v3-0-3ba8123bcffc@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731102551; l=2292; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=xatTfDDvuha6eMR9JG9Hd2IzFzAHGTmF0QhSbBpuJrI=; b=+UaZwuNQtMzNVshj56EsIrPn3MWPIFxheCCcSJ5ascHIIKxWH+Krl98RTnWFa7zYf6Ifpt7wz /ajvRtsg1iPBIT/+zg9o2NAUQQFEdZvH3dp7gnqGnGVv9xZQ+9zQWhx X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Patchwork-Delegate: kuba@kernel.org Move the kdump check into enic_adjust_resources() so that everything that modifies resources is in the same function. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic_main.c | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 84e85c9e2bf52f0089c0a8d03fb6d22ada4d086c..9913952ccb42f2017037a81a8e2c42daa7b53ec3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2531,6 +2531,15 @@ static int enic_adjust_resources(struct enic *enic) return -ENOSPC; } + if (is_kdump_kernel()) { + dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); + enic->rq_avail = 1; + enic->wq_avail = 1; + enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; + enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; + enic->config.mtu = min_t(u16, 1500, enic->config.mtu); + } + /* if RSS isn't set, then we can only use one RQ */ if (!ENIC_SETTING(enic, RSS)) enic->rq_avail = 1; @@ -2764,18 +2773,6 @@ static void enic_dev_deinit(struct enic *enic) enic_free_enic_resources(enic); } -static void enic_kdump_kernel_config(struct enic *enic) -{ - if (is_kdump_kernel()) { - dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_avail = 1; - enic->wq_avail = 1; - enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; - enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; - enic->config.mtu = min_t(u16, 1500, enic->config.mtu); - } -} - static int enic_dev_init(struct enic *enic) { struct device *dev = enic_get_dev(enic); @@ -2811,10 +2808,6 @@ static int enic_dev_init(struct enic *enic) return err; } - /* modify resource count if we are in kdump_kernel - */ - enic_kdump_kernel_config(enic); - /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic);