From patchwork Tue Oct 22 04:17:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13845087 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-4.cisco.com (rcdn-iport-4.cisco.com [173.37.86.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B44CA79B8E for ; Tue, 22 Oct 2024 04:17:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.75 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570682; cv=none; b=mpxqVXAHKet0n3ZEPOr6aCN0o2Ks7MKZIwPYUa1rnEmJVv5D34G5FmxwUyV9DY39a82Av0K2aEt2ca8ItC3HUTfruFHGoujIRBGEZ8tIzmWNdQf9aIHys44JR9SRQVEvOaP5NEWNsYxywozpNz2/KUnZANOvudilwluJtwHb/a8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570682; c=relaxed/simple; bh=1pwxduZoVN5bdAHKw6awCIT5VHEEoIswP1XXYlrl3L0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nFj5c2TTwgWhl+f95l4bGilXOgee6eDrXeuOTEe1q1U1djBkXTFvLWKmG5jEm9PfplnQInNbqtfVtdFEj9/H16WDzDDg/cktEwurxNXlwzayn3u7yQKsVDITNIokGnJ/y+Ppqrz+Viig3L313RMRc+Z5xGuPQ++qCaUYa7nOvwY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=mRSF1mKU; arc=none smtp.client-ip=173.37.86.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="mRSF1mKU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=18556; q=dns/txt; s=iport; t=1729570679; x=1730780279; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QdXa6QPTyKWowknidBxofnQuP0Hla8Vbs+iSsCUbm3A=; b=mRSF1mKU/80VQ0DX27tC6o33aow21Hbm6s7rcM1ePiY41vKrxQ+osnEw NcqD6gq8e2/TzxqIHEVY4KhfZ9YopXITA1G6OEX0MsjlscynfuaagTj+K VeNXE2f1R2h/F9whEtyLkH3hlB8+TmaPhhwv60ldTd9JgRn1OcecJdlVZ g=; X-CSE-ConnectionGUID: f4AufLEdTpufz8qx+NjbEg== X-CSE-MsgGUID: 838GCxIfTF6S24Ljh4lr+A== X-IPAS-Result: A0ANAAC6Jhdn/4z/Ja1aGwEBAQEBAQEBBQEBARIBAQEDAwEBAYF/BgEBAQsBgkqBT0NIjHJfiHKLdZIiFIERA1YPAQEBD0QEAQGFBwKKIwImNAkOAQIEAQEBAQMCAwEBAQEBAQEBAQ0BAQUBAQECAQcFgQ4ThgiGWwIBAycLAUYQUSsrGYMBgmUDr1uBeTOBAYR72TiBbIFIAY1FcIR3JxuBSUSCUIIthCqGXQSHaIwthQ2BD4FtVXwlgSACAgIHAogOkXZIgSEDWSECEQFVEw0KCwkFiTWDJimBa4EIgwiFJYFnCWGIR4EHLYERgR86ggOBNkqFN0c/gk9qTjcCDQI3giSBAIJRhkdAAwsYDUgRLDUUGwY+bgesekaCZnsUf4EUMSY4klgJAZISgTSfSoQkoT8aM6pMHhCYSaQ6hGaBZzyBWTMaCBsVgyJSGQ+OKgMWzDwmMjsCBwsBAQMJhkuGfWABAQ IronPort-Data: A9a23:KFbXuaP0aQUfB43vrR3ilsFynXyQoLVcMsEvi/4bfWQNrUor1zMCm GRNDGjSa/rbYmHxeYhxO9uy9RxVu8XczINrTHM5pCpnJ55oRWUpJjg4wmPYZX76whjrFRo/h ykmQoCdap1yFDmE/0fF3oHJ9RFUzbuPSqf3FNnKMyVwQR4MYCo6gHqPocZh6mJTqYb/WlrlV e/a+ZWFZAb9gWQsawr41orawP9RlKWq0N8nlgRWicBj5Df2i3QTBZQDEqC9R1OQapVUBOOzW 9HYx7i/+G7Dlz91Yj9yuu+mGqGiaue60Tmm0hK6aYD76vRxjnBaPpIACRYpQRw/ZwNlMDxG4 I4lWZSYEW/FN0BX8QgXe0Ew/ypWZcWq9FJbSJSymZT78qHIT5fj6/syJVw4GaAUxv5qD3Me7 OxbDjIHQx/W0opawJrjIgVtrt4oIM+uOMYUvWttiGmCS/0nWpvEBa7N4Le03h9p2ZsIRqmYP ZdEL2MzNHwsYDUXUrsTIJQzkfyjgXP2WzZZs1mS46Ew5gA/ySQrj+e1bYqLJ4ziqcN9zh3Ii UnW3mrFLE8hboC7yzrZ9mjvr7qa9c/8cMdIfFGizdZsjUGfy3I7FhIbTx24rOO/h0r4XMhQQ 3H44QI0pqQ0sUjuRd7nUljg+ziPvwUXXJxbFOhSBByx95c4Kj2xXgAsJgOtovR83CPqbVTGD mO0ou4= IronPort-HdrOrdr: A9a23:YbfblaA2WQZ12wDlHemr55DYdb4zR+YMi2TDGXofdfUzSL3+qy nAppUmPHPP5Qr5HUtQ++xoW5PwJU80i6QU3WB5B97LN2PbUSmTXeRfBODZrQEIdReTygck79 YCT0C7Y+eAdGSTSq3BkW+FL+o= X-Talos-CUID: 9a23:moMoBmCmIpfzTfL6Eyc73VMaPu0pTmXAwmWAOl/7L2RjZbLAHA== X-Talos-MUID: 9a23:wjul6AmyrYJMEXR5gbFodnoyb5dHyLSPBnwjz65c5+2GNXB0IXS02WE= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.11,222,1725321600"; d="scan'208";a="277058658" Received: from rcdn-l-core-03.cisco.com ([173.37.255.140]) by rcdn-iport-4.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 22 Oct 2024 04:17:58 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-l-core-03.cisco.com (Postfix) with ESMTP id A76CC180001EE; Tue, 22 Oct 2024 04:17:57 +0000 (GMT) Received: by cisco.com (Postfix, from userid 412739) id 7D98E20F2003; Mon, 21 Oct 2024 21:17:57 -0700 (PDT) From: Nelson Escobar To: netdev@vger.kernel.org Cc: satishkh@cisco.com, johndale@cisco.com, Nelson Escobar Subject: [Patch net-next 1/5] enic: Create enic_wq/rq structures to bundle per wq/rq data Date: Mon, 21 Oct 2024 21:17:03 -0700 Message-Id: <20241022041707.27402-2-neescoba@cisco.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20241022041707.27402-1-neescoba@cisco.com> References: <20241022041707.27402-1-neescoba@cisco.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-l-core-03.cisco.com X-Patchwork-Delegate: kuba@kernel.org Bundling the wq/rq specific data into dedicated enic_wq/rq structures cleans up the enic structure and simplifies future changes related to wq/rq. Signed-off-by: Nelson Escobar Signed-off-by: John Daley Signed-off-by: Satish Kharat Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 18 ++- .../net/ethernet/cisco/enic/enic_ethtool.c | 4 +- drivers/net/ethernet/cisco/enic/enic_main.c | 120 +++++++++--------- drivers/net/ethernet/cisco/enic/enic_res.c | 12 +- 4 files changed, 81 insertions(+), 73 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 0cc3644ee855..e6edb43515b9 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -162,6 +162,17 @@ struct enic_rq_stats { u64 desc_skip; /* Rx pkt went into later buffer */ }; +struct enic_wq { + struct vnic_wq vwq; + struct enic_wq_stats stats; + spinlock_t lock; /* spinlock for wq */ +}; + +struct enic_rq { + struct vnic_rq vrq; + struct enic_rq_stats stats; +}; + /* Per-instance private data structure */ struct enic { struct net_device *netdev; @@ -194,16 +205,13 @@ struct enic { struct enic_port_profile *pp; /* work queue cache line section */ - ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX]; - spinlock_t wq_lock[ENIC_WQ_MAX]; - struct enic_wq_stats wq_stats[ENIC_WQ_MAX]; + ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ - ____cacheline_aligned struct vnic_rq rq[ENIC_RQ_MAX]; - struct enic_rq_stats rq_stats[ENIC_RQ_MAX]; + ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index f7986f2b6a17..909d6f7000e1 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -337,7 +337,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, for (i = 0; i < NUM_ENIC_GEN_STATS; i++) *(data++) = ((u64 *)&enic->gen_stats)[enic_gen_stats[i].index]; for (i = 0; i < enic->rq_count; i++) { - struct enic_rq_stats *rqstats = &enic->rq_stats[i]; + struct enic_rq_stats *rqstats = &enic->rq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_RQ_STATS; j++) { @@ -346,7 +346,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, } } for (i = 0; i < enic->wq_count; i++) { - struct enic_wq_stats *wqstats = &enic->wq_stats[i]; + struct enic_wq_stats *wqstats = &enic->wq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_WQ_STATS; j++) { diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index ffed14b63d41..eb00058b6c68 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -342,8 +342,8 @@ static void enic_wq_free_buf(struct vnic_wq *wq, { struct enic *enic = vnic_dev_priv(wq->vdev); - enic->wq_stats[wq->index].cq_work++; - enic->wq_stats[wq->index].cq_bytes += buf->len; + enic->wq[wq->index].stats.cq_work++; + enic->wq[wq->index].stats.cq_bytes += buf->len; enic_free_wq_buf(wq, buf); } @@ -352,20 +352,20 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - spin_lock(&enic->wq_lock[q_number]); + spin_lock(&enic->wq[q_number].lock); - vnic_wq_service(&enic->wq[q_number], cq_desc, + vnic_wq_service(&enic->wq[q_number].vwq, cq_desc, completed_index, enic_wq_free_buf, opaque); if (netif_tx_queue_stopped(netdev_get_tx_queue(enic->netdev, q_number)) && - vnic_wq_desc_avail(&enic->wq[q_number]) >= + vnic_wq_desc_avail(&enic->wq[q_number].vwq) >= (MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS)) { netif_wake_subqueue(enic->netdev, q_number); - enic->wq_stats[q_number].wake++; + enic->wq[q_number].stats.wake++; } - spin_unlock(&enic->wq_lock[q_number]); + spin_unlock(&enic->wq[q_number].lock); return 0; } @@ -377,7 +377,7 @@ static bool enic_log_q_error(struct enic *enic) bool err = false; for (i = 0; i < enic->wq_count; i++) { - error_status = vnic_wq_error_status(&enic->wq[i]); + error_status = vnic_wq_error_status(&enic->wq[i].vwq); err |= error_status; if (error_status) netdev_err(enic->netdev, "WQ[%d] error_status %d\n", @@ -385,7 +385,7 @@ static bool enic_log_q_error(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - error_status = vnic_rq_error_status(&enic->rq[i]); + error_status = vnic_rq_error_status(&enic->rq[i].vrq); err |= error_status; if (error_status) netdev_err(enic->netdev, "RQ[%d] error_status %d\n", @@ -598,9 +598,9 @@ static int enic_queue_wq_skb_vlan(struct enic *enic, struct vnic_wq *wq, err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); /* The enic_queue_wq_desc() above does not do HW checksum */ - enic->wq_stats[wq->index].csum_none++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_none++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -634,9 +634,9 @@ static int enic_queue_wq_skb_csum_l4(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].csum_partial++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_partial++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -699,11 +699,11 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, if (skb->encapsulation) { hdr_len = skb_inner_tcp_all_headers(skb); enic_preload_tcp_csum_encap(skb); - enic->wq_stats[wq->index].encap_tso++; + enic->wq[wq->index].stats.encap_tso++; } else { hdr_len = skb_tcp_all_headers(skb); enic_preload_tcp_csum(skb); - enic->wq_stats[wq->index].tso++; + enic->wq[wq->index].stats.tso++; } /* Queue WQ_ENET_MAX_DESC_LEN length descriptors @@ -757,8 +757,8 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, pkts = len / mss; if ((len % mss) > 0) pkts++; - enic->wq_stats[wq->index].packets += pkts; - enic->wq_stats[wq->index].bytes += (len + (pkts * hdr_len)); + enic->wq[wq->index].stats.packets += pkts; + enic->wq[wq->index].stats.bytes += (len + (pkts * hdr_len)); return 0; } @@ -792,9 +792,9 @@ static inline int enic_queue_wq_skb_encap(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].encap_csum++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.encap_csum++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -812,7 +812,7 @@ static inline int enic_queue_wq_skb(struct enic *enic, /* VLAN tag from trunking driver */ vlan_tag_insert = 1; vlan_tag = skb_vlan_tag_get(skb); - enic->wq_stats[wq->index].add_vlan++; + enic->wq[wq->index].stats.add_vlan++; } else if (enic->loop_enable) { vlan_tag = enic->loop_tag; loopback = 1; @@ -859,11 +859,11 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, struct netdev_queue *txq; txq_map = skb_get_queue_mapping(skb) % enic->wq_count; - wq = &enic->wq[txq_map]; + wq = &enic->wq[txq_map].vwq; if (skb->len <= 0) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].null_pkt++; + enic->wq[wq->index].stats.null_pkt++; return NETDEV_TX_OK; } @@ -878,19 +878,19 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, skb_shinfo(skb)->nr_frags + 1 > ENIC_NON_TSO_MAX_DESC && skb_linearize(skb)) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].skb_linear_fail++; + enic->wq[wq->index].stats.skb_linear_fail++; return NETDEV_TX_OK; } - spin_lock(&enic->wq_lock[txq_map]); + spin_lock(&enic->wq[txq_map].lock); if (vnic_wq_desc_avail(wq) < skb_shinfo(skb)->nr_frags + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); /* This is a hard error, log it */ netdev_err(netdev, "BUG! Tx ring full when queue awake!\n"); - spin_unlock(&enic->wq_lock[txq_map]); - enic->wq_stats[wq->index].desc_full_awake++; + spin_unlock(&enic->wq[txq_map].lock); + enic->wq[wq->index].stats.desc_full_awake++; return NETDEV_TX_BUSY; } @@ -899,14 +899,14 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, if (vnic_wq_desc_avail(wq) < MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); - enic->wq_stats[wq->index].stopped++; + enic->wq[wq->index].stats.stopped++; } skb_tx_timestamp(skb); if (!netdev_xmit_more() || netif_xmit_stopped(txq)) vnic_wq_doorbell(wq); error: - spin_unlock(&enic->wq_lock[txq_map]); + spin_unlock(&enic->wq[txq_map].lock); return NETDEV_TX_OK; } @@ -941,9 +941,9 @@ static void enic_get_stats(struct net_device *netdev, net_stats->multicast = stats->rx.rx_multicast_frames_ok; for (i = 0; i < ENIC_RQ_MAX; i++) { - struct enic_rq_stats *rqs = &enic->rq_stats[i]; + struct enic_rq_stats *rqs = &enic->rq[i].stats; - if (!enic->rq->ctrl) + if (!enic->rq[i].vrq.ctrl) break; pkt_truncated += rqs->pkt_truncated; bad_fcs += rqs->bad_fcs; @@ -1313,7 +1313,7 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq) } skb = netdev_alloc_skb_ip_align(netdev, len); if (!skb) { - enic->rq_stats[rq->index].no_skb++; + enic->rq[rq->index].stats.no_skb++; return -ENOMEM; } @@ -1366,7 +1366,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq, struct net_device *netdev = enic->netdev; struct sk_buff *skb; struct vnic_cq *cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - struct enic_rq_stats *rqstats = &enic->rq_stats[rq->index]; + struct enic_rq_stats *rqstats = &enic->rq[rq->index].stats; u8 type, color, eop, sop, ingress_port, vlan_stripped; u8 fcoe, fcoe_sof, fcoe_fc_crc_ok, fcoe_enc_error, fcoe_eof; @@ -1512,7 +1512,7 @@ static int enic_rq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - vnic_rq_service(&enic->rq[q_number], cq_desc, + vnic_rq_service(&enic->rq[q_number].vrq, cq_desc, completed_index, VNIC_RQ_RETURN_DESC, enic_rq_indicate_buf, opaque); @@ -1609,7 +1609,7 @@ static int enic_poll(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[0].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling * mode so we can try to fill the ring again. @@ -1621,7 +1621,7 @@ static int enic_poll(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[0]); + enic_calc_int_moderation(enic, &enic->rq[0].vrq); if ((rq_work_done < budget) && napi_complete_done(napi, rq_work_done)) { @@ -1630,11 +1630,11 @@ static int enic_poll(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[0]); + enic_set_int_moderation(enic, &enic->rq[0].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[0].napi_complete++; + enic->rq[0].stats.napi_complete++; } else { - enic->rq_stats[0].napi_repoll++; + enic->rq[0].stats.napi_repoll++; } return rq_work_done; @@ -1683,7 +1683,7 @@ static int enic_poll_msix_wq(struct napi_struct *napi, int budget) struct net_device *netdev = napi->dev; struct enic *enic = netdev_priv(netdev); unsigned int wq_index = (napi - &enic->napi[0]) - enic->rq_count; - struct vnic_wq *wq = &enic->wq[wq_index]; + struct vnic_wq *wq = &enic->wq[wq_index].vwq; unsigned int cq; unsigned int intr; unsigned int wq_work_to_do = ENIC_WQ_NAPI_BUDGET; @@ -1737,7 +1737,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[rq], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[rq].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling mode * so we can try to fill the ring again. @@ -1749,7 +1749,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[rq]); + enic_calc_int_moderation(enic, &enic->rq[rq].vrq); if ((work_done < budget) && napi_complete_done(napi, work_done)) { @@ -1758,11 +1758,11 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[rq]); + enic_set_int_moderation(enic, &enic->rq[rq].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[rq].napi_complete++; + enic->rq[rq].stats.napi_complete++; } else { - enic->rq_stats[rq].napi_repoll++; + enic->rq[rq].stats.napi_repoll++; } return work_done; @@ -1989,10 +1989,10 @@ static int enic_open(struct net_device *netdev) for (i = 0; i < enic->rq_count; i++) { /* enable rq before updating rq desc */ - vnic_rq_enable(&enic->rq[i]); - vnic_rq_fill(&enic->rq[i], enic_rq_alloc_buf); + vnic_rq_enable(&enic->rq[i].vrq); + vnic_rq_fill(&enic->rq[i].vrq, enic_rq_alloc_buf); /* Need at least one buffer on ring to get going */ - if (vnic_rq_desc_used(&enic->rq[i]) == 0) { + if (vnic_rq_desc_used(&enic->rq[i].vrq) == 0) { netdev_err(netdev, "Unable to alloc receive buffers\n"); err = -ENOMEM; goto err_out_free_rq; @@ -2000,7 +2000,7 @@ static int enic_open(struct net_device *netdev) } for (i = 0; i < enic->wq_count; i++) - vnic_wq_enable(&enic->wq[i]); + vnic_wq_enable(&enic->wq[i].vwq); if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic)) enic_dev_add_station_addr(enic); @@ -2027,9 +2027,9 @@ static int enic_open(struct net_device *netdev) err_out_free_rq: for (i = 0; i < enic->rq_count; i++) { - ret = vnic_rq_disable(&enic->rq[i]); + ret = vnic_rq_disable(&enic->rq[i].vrq); if (!ret) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); } enic_dev_notify_unset(enic); err_out_free_intr: @@ -2071,12 +2071,12 @@ static int enic_stop(struct net_device *netdev) enic_dev_del_station_addr(enic); for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_disable(&enic->wq[i]); + err = vnic_wq_disable(&enic->wq[i].vwq); if (err) return err; } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_disable(&enic->rq[i]); + err = vnic_rq_disable(&enic->rq[i].vrq); if (err) return err; } @@ -2086,9 +2086,9 @@ static int enic_stop(struct net_device *netdev) enic_free_intr(enic); for (i = 0; i < enic->wq_count; i++) - vnic_wq_clean(&enic->wq[i], enic_free_wq_buf); + vnic_wq_clean(&enic->wq[i].vwq, enic_free_wq_buf); for (i = 0; i < enic->rq_count; i++) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); for (i = 0; i < enic->cq_count; i++) vnic_cq_clean(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -2576,7 +2576,7 @@ static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { struct enic *enic = netdev_priv(dev); - struct enic_rq_stats *rqstats = &enic->rq_stats[idx]; + struct enic_rq_stats *rqstats = &enic->rq[idx].stats; rxs->bytes = rqstats->bytes; rxs->packets = rqstats->packets; @@ -2590,7 +2590,7 @@ static void enic_get_queue_stats_tx(struct net_device *dev, int idx, struct netdev_queue_stats_tx *txs) { struct enic *enic = netdev_priv(dev); - struct enic_wq_stats *wqstats = &enic->wq_stats[idx]; + struct enic_wq_stats *wqstats = &enic->wq[idx].stats; txs->bytes = wqstats->bytes; txs->packets = wqstats->packets; @@ -2993,7 +2993,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_WORK(&enic->change_mtu_work, enic_change_mtu_work); for (i = 0; i < enic->wq_count; i++) - spin_lock_init(&enic->wq_lock[i]); + spin_lock_init(&enic->wq[i].lock); /* Register net device */ diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1c48aebdbab0..60be09acb9fd 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -176,9 +176,9 @@ void enic_free_vnic_resources(struct enic *enic) unsigned int i; for (i = 0; i < enic->wq_count; i++) - vnic_wq_free(&enic->wq[i]); + vnic_wq_free(&enic->wq[i].vwq); for (i = 0; i < enic->rq_count; i++) - vnic_rq_free(&enic->rq[i]); + vnic_rq_free(&enic->rq[i].vrq); for (i = 0; i < enic->cq_count; i++) vnic_cq_free(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -233,7 +233,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->rq_count; i++) { cq_index = i; - vnic_rq_init(&enic->rq[i], + vnic_rq_init(&enic->rq[i].vrq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -241,7 +241,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->wq_count; i++) { cq_index = enic->rq_count + i; - vnic_wq_init(&enic->wq[i], + vnic_wq_init(&enic->wq[i].vwq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -322,7 +322,7 @@ int enic_alloc_vnic_resources(struct enic *enic) */ for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_alloc(enic->vdev, &enic->wq[i], i, + err = vnic_wq_alloc(enic->vdev, &enic->wq[i].vwq, i, enic->config.wq_desc_count, sizeof(struct wq_enet_desc)); if (err) @@ -330,7 +330,7 @@ int enic_alloc_vnic_resources(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_alloc(enic->vdev, &enic->rq[i], i, + err = vnic_rq_alloc(enic->vdev, &enic->rq[i].vrq, i, enic->config.rq_desc_count, sizeof(struct rq_enet_desc)); if (err) From patchwork Tue Oct 22 04:17:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13845088 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-9.cisco.com (rcdn-iport-9.cisco.com [173.37.86.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C328B6EB7C for ; Tue, 22 Oct 2024 04:18:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.80 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570705; cv=none; b=g4fiqDtOWqBuuu1K5O8Q5ijlT6sPKX2VZ0a9kFUNNndQQ3WaAXvl2M76kaKkEXfBp2BOevGjILtQnUvievBW0TwGEHHT/iQQQxMF3xdo4KurcBS7YvBGof5Hsdco044rf3KXbIiQ6C4Y1LWU/MM26yKzi8chNlc2FMwvC3SXjgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570705; c=relaxed/simple; bh=SSVOZUhdiMtDzgvAvLrwSTVWiADUkQOXxXVUQwOEbrc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NzEhAwaCp+L1mBrTh1XZXlmqhYh5l/nexuSrckIWCcxSFPl/GGD8byVyEPPyJux9QhWeT4NoJAZg0FscvpncNj78KDcXe/8XwmnFGonDIArGr/NlqNhOw6/qQG+MK7I/8xLMKlwV8IPU2CPE9MwIkgvyQxT/PYNdkF7wPs6RM5A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=D41BZzgO; arc=none smtp.client-ip=173.37.86.80 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="D41BZzgO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=2564; q=dns/txt; s=iport; t=1729570703; x=1730780303; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HcUKEv2emSx5BAHfHG306BIQJWaOigYcS0Z6xNllAG0=; b=D41BZzgOzs6NG9pRLgWuT48ZjlQEr6bqjLQQI8xZoq9Qz4uJ0g7tTiFh pTbGYySEkyj5nRK9SJBEt2eDPmldmddrUQFg2GNmRvzYnxegQUXmKNvUK ScpngMPNEjF4lj8zdPfqPKTi6+55ZE4kJXp52UlxFOposTssIJzCLpteo 0=; X-CSE-ConnectionGUID: effnDKefTWi2OMGEJTbcKw== X-CSE-MsgGUID: Ev34CNMZReaH7XA6AC7jXg== X-IPAS-Result: A0ApAABGJhdn/5T/Ja1aHAEBAQEBAQcBARIBAQQEAQGBfwcBAQsBgkqBT0NIjHJfiHKLdZIigSUDVg8BAQEPRAQBAYUHAoojAiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBDQEBBQEBAQIBBwWBDhOGCIZbAgEDJwsBRhBRKysZgwGCZQOvW4F5M4EBhHvZOIFsgUgBjUVwhHcnG4FJRIEUAYNohRCFdwSHaIwtiVoliT2RdkiBIQNZIQIRAVUTDQoLCQWJNYMmKYFrgQiDCIUlgWcJYYhHgQctgRGBHzqCA4E2SoU3Rz+CT2pONwINAjeCJIEAglGGR0ADCxgNSBEsNRQbBj5uB6x6RoJfB3sUgS4CQD+lZ6B+hCShPxozqkyYd6Q6hGaBZzyBWTMaCBsVgyJSGQ+OLRYWzCgmMjsCBwsBAQMJjigBAQ IronPort-Data: A9a23:4d2diqhDmWBJ0lJopzVHwfVoX161JxEKZh0ujC45NGQN5FlHY01je htvXWjQbvqLMTOnfYp/aoi+oEpV7MXRn9FlT1Zq+y5kRX5jpJueD7x1DKtf0wB+jyHnZBg6h ynLQoCYdKjYdleF+1HwdOKn9SAsvU2xbuKUIPbePSxsThNTRi4kiBZy88Y0mYcAbeKRW2thg vus5ZSGULOZ82QsaD5Ns/jZ8EoHUMna4Vv0gHRvPZing3eG/5UlJMp3Db28KXL+Xr5VEoaSL 87fzKu093/u5BwkDNWoiN7TKiXmlZaLYGBiIlIPM0STqkAqSh4ai87XB9JAAatjsAhlqvgqo Dl7WTNcfi9yVkHEsLx1vxC1iEiSN4UekFPMCSDXXcB+UyQqflO0q8iCAn3aMqUV9OhxLUJD+ 8AnKWoqSz+upd68y5GkH7wEasQLdKEHPasFsX1miDWcBvE8TNWaGuPB5MRT23E7gcUm8fT2P pVCL2EwKk6dPlsWZgd/5JEWxI9EglH5fjBDo1WfrII84nPYy0p6172F3N/9IIPUG50Kxx3Jz o7A10nnWU0xEvmt8zeM0i2cr7Hmtnz5aI1HQdVU8dYv2jV/3Fc7DhAKWValiee2h1T4WN9FL UEQvC00osAPGFeDVNLxWVi85XWDpBNZAoMWGOwh4wbLwa3Ri+qEOlU5ovd6QIROnKcLqfYCj zdlQ/uB6eRTjYCo IronPort-HdrOrdr: A9a23:rWfXA6C2SlFZxYXlHemr55DYdb4zR+YMi2TDGXofdfUzSL3+qy nAppUmPHPP5Qr5HUtQ++xoW5PwJU80i6QU3WB5B97LN2PbUSmTXeRfBODZrQEIdReTygck79 YCT0C7Y+eAdGSTSq3BkW+FL+o= X-Talos-CUID: 9a23:qfhLJmk2aE/uT9gqG6irCN3ijxDXOXDTwnH8AEK2NVloEpqwZnKyxKlvsNU7zg== X-Talos-MUID: 9a23:Qh/2uQbVKg2ci+BTmiG32BpEEfVS7q2OV3Akz7wt5Mu0Onkl X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.11,222,1725321600"; d="scan'208";a="276607139" Received: from rcdn-l-core-11.cisco.com ([173.37.255.148]) by rcdn-iport-9.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 22 Oct 2024 04:18:17 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-l-core-11.cisco.com (Postfix) with ESMTP id F36821800024F; Tue, 22 Oct 2024 04:18:16 +0000 (GMT) Received: by cisco.com (Postfix, from userid 412739) id BC87320F2003; Mon, 21 Oct 2024 21:18:16 -0700 (PDT) From: Nelson Escobar To: netdev@vger.kernel.org Cc: satishkh@cisco.com, johndale@cisco.com, Nelson Escobar Subject: [Patch net-next 2/5] enic: Make MSI-X I/O interrupts come after the other required ones Date: Mon, 21 Oct 2024 21:17:04 -0700 Message-Id: <20241022041707.27402-3-neescoba@cisco.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20241022041707.27402-1-neescoba@cisco.com> References: <20241022041707.27402-1-neescoba@cisco.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-l-core-11.cisco.com X-Patchwork-Delegate: kuba@kernel.org The VIC hardware has a constraint that the MSIX interrupt used for errors be specified as a 7 bit number. Before this patch, it was allocated after the I/O interrupts, which would cause a problem if 128 or more I/O interrupts are in use. So make the required interrupts come before the I/O interrupts to guarantee the error interrupt offset never exceeds 7 bits. Signed-off-by: Nelson Escobar Signed-off-by: John Daley Signed-off-by: Satish Kharat Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 20 +++++++++++++++----- drivers/net/ethernet/cisco/enic/enic_res.c | 2 +- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index e6edb43515b9..ac7236f76a51 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -280,18 +280,28 @@ static inline unsigned int enic_msix_wq_intr(struct enic *enic, return enic->cq[enic_cq_wq(enic, wq)].interrupt_offset; } -static inline unsigned int enic_msix_err_intr(struct enic *enic) -{ - return enic->rq_count + enic->wq_count; -} +/* MSIX interrupts are organized as the error interrupt, then the notify + * interrupt followed by all the I/O interrupts. The error interrupt needs + * to fit in 7 bits due to hardware constraints + */ +#define ENIC_MSIX_RESERVED_INTR 2 +#define ENIC_MSIX_ERR_INTR 0 +#define ENIC_MSIX_NOTIFY_INTR 1 +#define ENIC_MSIX_IO_INTR_BASE ENIC_MSIX_RESERVED_INTR +#define ENIC_MSIX_MIN_INTR (ENIC_MSIX_RESERVED_INTR + 2) #define ENIC_LEGACY_IO_INTR 0 #define ENIC_LEGACY_ERR_INTR 1 #define ENIC_LEGACY_NOTIFY_INTR 2 +static inline unsigned int enic_msix_err_intr(struct enic *enic) +{ + return ENIC_MSIX_ERR_INTR; +} + static inline unsigned int enic_msix_notify_intr(struct enic *enic) { - return enic->rq_count + enic->wq_count + 1; + return ENIC_MSIX_NOTIFY_INTR; } static inline bool enic_is_err_intr(struct enic *enic, int intr) diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 60be09acb9fd..6910f83185c4 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -257,7 +257,7 @@ void enic_init_vnic_resources(struct enic *enic) switch (intr_mode) { case VNIC_DEV_INTR_MODE_MSIX: - interrupt_offset = i; + interrupt_offset = ENIC_MSIX_IO_INTR_BASE + i; break; default: interrupt_offset = 0; From patchwork Tue Oct 22 04:17:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13845089 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F0004EB38 for ; Tue, 22 Oct 2024 04:18:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570723; cv=none; b=qP6E7l7BEz/xak00i+LWwz+FJtO7aeUycK6MRg+c+pr25FkF7itdF4Yu72pOLlKf92Gd1/TLpTyvSNs07C8eBEkGFgoZ5NgmhBnxzpDeUrtwEYchLxCpGspAbHmUPv109Ar08ox/IKdpPWXcagzFUGeB61cSnwbP1IlA1vSyssE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570723; c=relaxed/simple; bh=09cAVEe9j8qGzQFQF8pxQvRMeI2pYl1vhYnW2Al656c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MYuzZL4nxBTjSznhgSS1O3a32Em2Hnb0c61v1mLe45bPu+nm5g0hagnxkK+9bvfDsn5+FCQmqgxY1Jb3t//vFOHn6hCF7DddjC2lMql7mI3dGbVrF+FM4haf+rbbI/MRe3MmMITYNlKtSZwpJxOAu6kjsuXcBhaTZI9dxeSMnFs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=kSdDpDsf; arc=none smtp.client-ip=173.37.86.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="kSdDpDsf" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=3820; q=dns/txt; s=iport; t=1729570722; x=1730780322; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RacxNF7c4jaMAjFcLtK9Bt2ej642MBqc/gXWznPvjSI=; b=kSdDpDsf5fi/C/JJAPKtsb8Ilw1yrWJMeVBLDZ2GiI6PhhBMUtYueN6l ISJ3tQ8DrDQV4Lk21DjrlRYn0mmYI9xqHmw9TYZmkSYst2BQVaDuOevGS c4DRZAXy1qGNwaml6hcqPCk6aHKE2K9Dwtc/WOZRlUzOcpZhikm9qNWMe c=; X-CSE-ConnectionGUID: LcHaTsvOS8eU7KRe0F6/ew== X-CSE-MsgGUID: qE09txZGRsC/QxEPwWG9Dw== X-IPAS-Result: A0A8AABGJhdn/5X/Ja1aHQEBAQEJARIBBQUBgX8IAQsBgkqBT0NIjHJflGeSIoElA1YPAQEBD0QEAQGFBwKKIwImNAkOAQIEAQEBAQMCAwEBAQEBAQEBAQ0BAQUBAQECAQcFgQ4ThgiGWwIBAycLAUYQUSsrGYMBgmUDr1uBeTOBAYR72TiBbIFIAY1FcIR3JxuBSUSCUIE+b4sHBJxzfIlikXZIgSEDWSECEQFVEw0KCwkFgUyHaYMmKYFrgQiDCIUlgWcJYYhHgQctgRGBHzqCA4E2SoU3Rz+CT2pONwINAjeCJIEAglGGR0ADCxgNSBEsNRQbBj5uB6x6RoJfB4EPghOBD6R0oH6EJKE/GjOEBZQBkkaYd6Q6hGaBZzyBWTMaCBsVgyJSGQ+OLRbMPiYyOwIHCwEBAwmOKAEB IronPort-Data: A9a23:0ucvRa0JyVQSdC+bdvbD5ZRwkn2cJEfYwER7XKvMYLTBsI5bpzxRn GRMDGuBb6neZjPwe9l0bt/ipx8BsJHUnNJgSwNl3Hw8FHgiRegpqji6wuYcGwvIc6UvmWo+t 512huHodZ1yEzmG4E/0YtANlFEkvYmQXL3wFeXYDS54QA5gWU8JhAlq8wIDqtYAbeORXUXU4 Lsen+WFYAX5gmYuaDpNg06+gEoHUMra6WtwUmMWPZinjHeG/1EJAZQWI72GLneQauF8Au6gS u/f+6qy92Xf8g1FIovNfmHTKxBirhb6ZGBiu1IOM0SQqkEqSh8ajs7XAMEhhXJ/0F1lqTzeJ OJl7vRcQS9xVkHFdX90vxNwS0mSNoUekFPLzOTWXcG7lyX7n3XQL/pGKEoWNLED5MFMUWxzq t8iJWgjMFeOmLfjqF67YrEEasULNsLnOsYb/3pn1zycVKxgSpHYSKKM7thdtNsyrpkRRrCFO IxDNGcpNUiRC/FMEg9/5JYWmuqlnXL4eTRwo1OOrq1x6G/WpOB0+OKzaoqKJYfaH625mG6Bi zqX43TLLSsBH5um2Re97U2D18PQyHaTtIU6UefQGuRRqFue2mAeFjUIWlah5/q0kEizX5RYM UN8x8Y1hbI5+EruSpz2WAe15Sfe+BUdQNFXVeY97Wlh15bp3upQPUBcJhYpVTDsnJZeqeACv rNRo+7UOA== IronPort-HdrOrdr: A9a23:3yXiwKmjqMAgHAOvAaDfKrf/MZ/pDfIr3DAbv31ZSRFFG/FwWf rAoB19726StN9/YhAdcLy7VZVoBEmsl6KdgrNhWYtKIjOHhILAFugLhuHfKn/bakjDH4Vmu5 uIHZITNDSJNykYsS4/izPIaurJB7K8gcaVuds= X-Talos-CUID: 9a23:DI05+Wji33jYeD6paqPkisnEczJuVnb9xUvLPWiDWWtUF56vRAe+puRpjJ87 X-Talos-MUID: 9a23:FBsItA8Slx359TJl1+k8+PiQf95Fu/WyGHIiq44t/PCUGBRvBBjMoCviFw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.11,222,1725321600"; d="scan'208";a="262877382" Received: from rcdn-l-core-12.cisco.com ([173.37.255.149]) by rcdn-iport-2.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 22 Oct 2024 04:18:34 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-l-core-12.cisco.com (Postfix) with ESMTP id 5D4661800026A; Tue, 22 Oct 2024 04:18:34 +0000 (GMT) Received: by cisco.com (Postfix, from userid 412739) id 3213D20F2003; Mon, 21 Oct 2024 21:18:34 -0700 (PDT) From: Nelson Escobar To: netdev@vger.kernel.org Cc: satishkh@cisco.com, johndale@cisco.com, Nelson Escobar Subject: [Patch net-next 3/5] enic: Save resource counts we read from HW Date: Mon, 21 Oct 2024 21:17:05 -0700 Message-Id: <20241022041707.27402-4-neescoba@cisco.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20241022041707.27402-1-neescoba@cisco.com> References: <20241022041707.27402-1-neescoba@cisco.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-l-core-12.cisco.com X-Patchwork-Delegate: kuba@kernel.org Save the resources counts for wq,rq,cq, and interrupts in *_avail variables so that we don't lose the information when adjusting the counts we are actually using. Report the wq_avail and rq_avail as the channel maximums in 'ethtool -l' output. Signed-off-by: Nelson Escobar Signed-off-by: John Daley Signed-off-by: Satish Kharat Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 4 ++++ .../net/ethernet/cisco/enic/enic_ethtool.c | 4 ++-- drivers/net/ethernet/cisco/enic/enic_res.c | 19 ++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index ac7236f76a51..1f32413a8f7c 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -206,23 +206,27 @@ struct enic { /* work queue cache line section */ ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; /* interrupt resource cache line section */ ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; u32 rx_copybreak; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index 909d6f7000e1..d607b4f0542c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -695,8 +695,8 @@ static void enic_get_channels(struct net_device *netdev, switch (vnic_dev_get_intr_mode(enic->vdev)) { case VNIC_DEV_INTR_MODE_MSIX: - channels->max_rx = ENIC_RQ_MAX; - channels->max_tx = ENIC_WQ_MAX; + channels->max_rx = min(enic->rq_avail, ENIC_RQ_MAX); + channels->max_tx = min(enic->wq_avail, ENIC_WQ_MAX); channels->rx_count = enic->rq_count; channels->tx_count = enic->wq_count; break; diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 6910f83185c4..e26e43dbfa9c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -187,16 +187,21 @@ void enic_free_vnic_resources(struct enic *enic) void enic_get_res_counts(struct enic *enic) { - enic->wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); - enic->rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); - enic->cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); - enic->intr_count = vnic_dev_get_res_count(enic->vdev, - RES_TYPE_INTR_CTRL); + enic->wq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); + enic->rq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); + enic->cq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); + enic->intr_avail = vnic_dev_get_res_count(enic->vdev, + RES_TYPE_INTR_CTRL); + + enic->wq_count = enic->wq_avail; + enic->rq_count = enic->rq_avail; + enic->cq_count = enic->cq_avail; + enic->intr_count = enic->intr_avail; dev_info(enic_get_dev(enic), "vNIC resources avail: wq %d rq %d cq %d intr %d\n", - enic->wq_count, enic->rq_count, - enic->cq_count, enic->intr_count); + enic->wq_avail, enic->rq_avail, + enic->cq_avail, enic->intr_avail); } void enic_init_vnic_resources(struct enic *enic) From patchwork Tue Oct 22 04:17:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13845090 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-1.cisco.com (rcdn-iport-1.cisco.com [173.37.86.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B25D17441A for ; Tue, 22 Oct 2024 04:18:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570741; cv=none; b=DK0glb+DSxs0JZdMLIQGo9mGFVBKyEu8Uvl5JKi6jhd6W6Mo6tSy4J+sHJiL3fwxXS4MGgHxJmaGJ3Ojj6wNBo8w5xfpC/TtoQ9oAvdUmkOa2MIrQHCGNoNi7ZDo0OGqVxSeHGQWO5vCsPAic7t8stQVS5+4Z9RpnwJXhIdsRqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570741; c=relaxed/simple; bh=sZahI1qyZm1y50stL+D+KfTyu7n4sgpFOu1qmm+kIn4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dYbr8SpFvJbbmcHXs84AJWSt7/Kt7DKCg1dagLfp+DhtlQ0M15B64vk4dUz7ZgI/e+GBtSBtPS/gkhrXU0Dq2ED2MzVUlxXSx1ytUvZgiFDlOJ6fSnUjb9tjTSelPIOLXOW6w8mf0keiQsoSFj2ELfbyWBhjHTr47sYI8ut57uI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=hG3T7ZqP; arc=none smtp.client-ip=173.37.86.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="hG3T7ZqP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=7013; q=dns/txt; s=iport; t=1729570739; x=1730780339; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WA8ghEKOr5jAfNLvGFwfJvJqlzmi9aenQqbE8to8Slc=; b=hG3T7ZqPS8KcpOxFQ1CcSG89nZ2b+0/SmLdwPgPc06kvpBDtcsgRKnm9 66YB98X0eBINbUtNhRQ0cqRGDnaeWmiqUC/uz/qSZXY3xj/vb4fSlobrK pewllbZaZJMfnCgZ6nU6ZcpBXJDuTOcXmXNML7djtDuxkRVzjqTGDSsnL A=; X-CSE-ConnectionGUID: oJpzYh9EQVS2eC88ueloEg== X-CSE-MsgGUID: LFdPiHwPS4qSZvsNO3o82A== X-IPAS-Result: A0ANAAC6Jhdn/4r/Ja1aGwEBAQEBAQEBBQEBARIBAQEDAwEBAYF/BgEBAQsBgkqBT0NIjHJfiHKLdZIiFIERA1YPAQEBD0QEAQGFBwKKIwImNAkOAQIEAQEBAQMCAwEBAQEBAQEBAQ0BAQUBAQECAQcFgQ4ThgiGWwIBAycLAUYQUSsrGYMBgmUDr1uBeTOBAYR72TiBbIFIAY1FcIR3JxuBSUSBFYE7gT5vhCqGXQSUFYYcgW1VfCWBIAICAgcCiA5bkRtIgSEDWSECEQFVEw0KCwkFiTWDJimBa4EIgwiFJYFnCWGIR4EHLYERgR86ggOBNkqDaIFPRz+CT2pONwINAjeCJIEAglGGR0ADCxgNSBEsNRQbBj5uB6x6RoJfB4EPgSYHA2OBD5JYCpISgTSfSoQkoT8aM6pMmHejbU2EZoFnPIFZMxoIGxWDIlIZD44tFsw8JjI7AgcLAQEDCY4oAQE IronPort-Data: A9a23:XQ/BMKqhcp3eLvlMGEolN6Qdc4BeBmJfZBIvgKrLsJaIsI4StFCzt garIBnUPfiNYmD0KtF2PoW/900BvJPWydZhTQRopXwyQXgRouPIVI+TRqvS04x+DSFioGZPt Zh2hgzodZhsJpPkjk7wdOWn9z8kjPHgqoPUUIbsIjp2SRJvVBAvgBdin/9RqoNziLBVOSvV0 T/Ji5OZYQbNNwJcaDpOt/vb8kk35ZwehRtB1rAATaET1LPhvyF94KI3fcmZM3b+S49IKe+2L 86r5K255G7Q4yA2AdqjlLvhGmVSKlIFFVHT4pb+c/HKbilq/kTe4I5iXBYvQRs/ZwGyojxE4 I4lWapc5useFvakdOw1C3G0GszlVEFM0OevzXOX6aR/w6BaGpfh660GMa04AWEX0uFVGXN/9 a01EQ42NB2o17Krzba8Z+Y506zPLOGzVG8ekmtrwTecCbMtRorOBv2To9RZxzw3wMtJGJ4yZ eJANmEpN0uGOUASfA5MWPrSn8/w7pX7WztVpUmeoqA+y2PS1wd2lrPqNbI5f/TQGpsFxhvB9 zOuE2LRJj8WDN2/1gK+rmuXtPHUuwSgX9JIPejtnhJtqBjJroAJMzUQWEe3rOeRlEGzQZRcJ lYS9y5oqrI9nHFHVfHnVBG+5XrBtRkGVp8ISqsx6RqGzezf5APx6nU4cwOtoecO7KceLQHGH HfQ9z81LVSDaIGodE8= IronPort-HdrOrdr: A9a23:JRykh6xOQjTwWnXCAh3LKrPwK71zdoMgy1knxilNoNJuHfBw8P re+8jzuiWUtN98YhwdcJW7Scu9qBDnhPpICPcqXYtKNTOO0ADDEGgh1/qG/9SKIUPDH4BmuZ uIC5IOa+EZyTNB/L/HCM7SKadH/OW6 X-Talos-CUID: 9a23:3RuB0GoaI1V497+azP6UgZTmUccHam/szCb9GnGDU2tCGO2IRFuX1awxxg== X-Talos-MUID: 9a23:D8EBOgTaUlwYekEdRXTIqj1aNed325/yEXs/j44gnJKDZXZZbmI= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.11,222,1725321600"; d="scan'208";a="276805199" Received: from rcdn-l-core-01.cisco.com ([173.37.255.138]) by rcdn-iport-1.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 22 Oct 2024 04:18:52 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-l-core-01.cisco.com (Postfix) with ESMTP id 6D8B818000295; Tue, 22 Oct 2024 04:18:51 +0000 (GMT) Received: by cisco.com (Postfix, from userid 412739) id 43B3A20F2003; Mon, 21 Oct 2024 21:18:51 -0700 (PDT) From: Nelson Escobar To: netdev@vger.kernel.org Cc: satishkh@cisco.com, johndale@cisco.com, Nelson Escobar Subject: [Patch net-next 4/5] enic: Allocate arrays in enic struct based on VIC config Date: Mon, 21 Oct 2024 21:17:06 -0700 Message-Id: <20241022041707.27402-5-neescoba@cisco.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20241022041707.27402-1-neescoba@cisco.com> References: <20241022041707.27402-1-neescoba@cisco.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-l-core-01.cisco.com X-Patchwork-Delegate: kuba@kernel.org Allocate wq, rq, cq, intr, and napi arrays based on the number of resources configured in the VIC. Signed-off-by: Nelson Escobar Signed-off-by: John Daley Signed-off-by: Satish Kharat Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 24 ++--- drivers/net/ethernet/cisco/enic/enic_main.c | 102 ++++++++++++++++++-- 2 files changed, 105 insertions(+), 21 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 1f32413a8f7c..cfb4667953de 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -23,10 +23,8 @@ #define ENIC_BARS_MAX 6 -#define ENIC_WQ_MAX 8 -#define ENIC_RQ_MAX 8 -#define ENIC_CQ_MAX (ENIC_WQ_MAX + ENIC_RQ_MAX) -#define ENIC_INTR_MAX (ENIC_CQ_MAX + 2) +#define ENIC_WQ_MAX 256 +#define ENIC_RQ_MAX 256 #define ENIC_WQ_NAPI_BUDGET 256 @@ -184,8 +182,8 @@ struct enic { struct work_struct reset; struct work_struct tx_hang_reset; struct work_struct change_mtu_work; - struct msix_entry msix_entry[ENIC_INTR_MAX]; - struct enic_msix_entry msix[ENIC_INTR_MAX]; + struct msix_entry *msix_entry; + struct enic_msix_entry *msix; u32 msg_enable; spinlock_t devcmd_lock; u8 mac_addr[ETH_ALEN]; @@ -204,28 +202,24 @@ struct enic { bool enic_api_busy; struct enic_port_profile *pp; - /* work queue cache line section */ - ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + struct enic_wq *wq; unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; - /* receive queue cache line section */ - ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + struct enic_rq *rq; unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; - struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; + struct napi_struct *napi; - /* interrupt resource cache line section */ - ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + struct vnic_intr *intr; unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ - /* completion queue cache line section */ - ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + struct vnic_cq *cq; unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index eb00058b6c68..a5d607be66b7 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -940,7 +940,7 @@ static void enic_get_stats(struct net_device *netdev, net_stats->rx_errors = stats->rx.rx_errors; net_stats->multicast = stats->rx.rx_multicast_frames_ok; - for (i = 0; i < ENIC_RQ_MAX; i++) { + for (i = 0; i < enic->rq_count; i++) { struct enic_rq_stats *rqs = &enic->rq[i].stats; if (!enic->rq[i].vrq.ctrl) @@ -1792,7 +1792,7 @@ static void enic_free_intr(struct enic *enic) free_irq(enic->pdev->irq, enic); break; case VNIC_DEV_INTR_MODE_MSIX: - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) if (enic->msix[i].requested) free_irq(enic->msix_entry[i].vector, enic->msix[i].devid); @@ -1859,7 +1859,7 @@ static int enic_request_intr(struct enic *enic) enic->msix[intr].isr = enic_isr_msix_notify; enic->msix[intr].devid = enic; - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) enic->msix[i].requested = 0; for (i = 0; i < enic->intr_count; i++) { @@ -2456,8 +2456,7 @@ static int enic_set_intr_mode(struct enic *enic) * (the last INTR is used for notifications) */ - BUG_ON(ARRAY_SIZE(enic->msix_entry) < n + m + 2); - for (i = 0; i < n + m + 2; i++) + for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; /* Use multiple RQs if RSS is enabled @@ -2674,6 +2673,89 @@ static const struct netdev_stat_ops enic_netdev_stat_ops = { .get_base_stats = enic_get_base_stats, }; +static void enic_free_enic_resources(struct enic *enic) +{ + kfree(enic->wq); + enic->wq = NULL; + + kfree(enic->rq); + enic->rq = NULL; + + kfree(enic->cq); + enic->cq = NULL; + + kfree(enic->napi); + enic->napi = NULL; + + kfree(enic->msix_entry); + enic->msix_entry = NULL; + + kfree(enic->msix); + enic->msix = NULL; + + kfree(enic->intr); + enic->intr = NULL; +} + +static int enic_alloc_enic_resources(struct enic *enic) +{ + int ret; + + enic->wq = NULL; + enic->rq = NULL; + enic->cq = NULL; + enic->napi = NULL; + enic->msix_entry = NULL; + enic->msix = NULL; + enic->intr = NULL; + + enic->wq = kcalloc(enic->wq_avail, sizeof(struct enic_wq), GFP_KERNEL); + if (!enic->wq) { + ret = -ENOMEM; + goto free_queues; + } + enic->rq = kcalloc(enic->rq_avail, sizeof(struct enic_rq), GFP_KERNEL); + if (!enic->rq) { + ret = -ENOMEM; + goto free_queues; + } + enic->cq = kcalloc(enic->cq_avail, sizeof(struct vnic_cq), GFP_KERNEL); + if (!enic->cq) { + ret = -ENOMEM; + goto free_queues; + } + enic->napi = kcalloc(enic->wq_avail + enic->rq_avail, + sizeof(struct napi_struct), GFP_KERNEL); + if (!enic->napi) { + ret = -ENOMEM; + goto free_queues; + } + enic->msix_entry = kcalloc(enic->intr_avail, sizeof(struct msix_entry), + GFP_KERNEL); + if (!enic->msix_entry) { + ret = -ENOMEM; + goto free_queues; + } + enic->msix = kcalloc(enic->intr_avail, sizeof(struct enic_msix_entry), + GFP_KERNEL); + if (!enic->msix) { + ret = -ENOMEM; + goto free_queues; + } + enic->intr = kcalloc(enic->intr_avail, sizeof(struct vnic_intr), + GFP_KERNEL); + if (!enic->intr) { + ret = -ENOMEM; + goto free_queues; + } + + return 0; + +free_queues: + enic_free_enic_resources(enic); + return ret; +} + static void enic_dev_deinit(struct enic *enic) { unsigned int i; @@ -2691,6 +2773,7 @@ static void enic_dev_deinit(struct enic *enic) enic_free_vnic_resources(enic); enic_clear_intr_mode(enic); enic_free_affinity_hint(enic); + enic_free_enic_resources(enic); } static void enic_kdump_kernel_config(struct enic *enic) @@ -2734,6 +2817,12 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + err = enic_alloc_enic_resources(enic); + if (err) { + dev_err(dev, "Failed to allocate queues, aborting\n"); + return err; + } + /* modify resource count if we are in kdump_kernel */ enic_kdump_kernel_config(enic); @@ -2746,7 +2835,7 @@ static int enic_dev_init(struct enic *enic) if (err) { dev_err(dev, "Failed to set intr mode based on resource " "counts and system capabilities, aborting\n"); - return err; + goto err_out_free_vnic_resources; } /* Allocate and configure vNIC resources @@ -2788,6 +2877,7 @@ static int enic_dev_init(struct enic *enic) enic_free_affinity_hint(enic); enic_clear_intr_mode(enic); enic_free_vnic_resources(enic); + enic_free_enic_resources(enic); return err; } From patchwork Tue Oct 22 04:17:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13845091 X-Patchwork-Delegate: kuba@kernel.org Received: from rcdn-iport-6.cisco.com (rcdn-iport-6.cisco.com [173.37.86.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A709E824AD for ; Tue, 22 Oct 2024 04:19:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.86.77 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570756; cv=none; b=KZJ2lHv582XBi6TzYbMyKV3UzO6ntLKXq1TNTKzNRfvJS2RGxxMlOKa0rGm5dFLaM9sfkkoZfy2Kw+BmD8YSkD87ylatu241uN2uCuMLjqiRGmI7FDgKTk20teIWhbyRhTqBDIg2kT4xu4y1frH9zqhHZ1Ey5TCZq0zPD6j23PA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729570756; c=relaxed/simple; bh=wyERo6IQcqy01fB6Ygzvjz29FTVXeiVgStzGDnlhayo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Exgl05OFORGzX7HpEDWoLuIq7i73QukXHKYTKJLegz9QTAQTKVhL6zIaiyfJKSBg9gi2VSEg8IRrj7lG8wdFz1dVwp2BgNbvpVwTGgZdZf72ThowbP+a8A74a5jZW/aBcuxI8ehMS5UOHky1ALuFgzLOIq/axO7WSuiKQmRHxCM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=KfhmdS1S; arc=none smtp.client-ip=173.37.86.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="KfhmdS1S" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=7865; q=dns/txt; s=iport; t=1729570754; x=1730780354; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=euHFmU7yDdilMuv1aryHVx1KbTA00sOv9ElUY5NBvcQ=; b=KfhmdS1SWFv/udANUMJz8xDWZ38m7PfRvVzDCXDUPqKfs1BwE5dL9pMz zX43l1FMtzm+GkH52lOUq5JJo/Cv4b01JLh5J9uoQzIdhjeBYLuJ4Gc3d ey+bB+8+w5fUMB6lgd8p1SDCy69/V+lK+n3AGZcC+ENYMC4MtdT6Eg+l2 A=; X-CSE-ConnectionGUID: hqWQ2Hw2RVm6YyVu+9I73w== X-CSE-MsgGUID: uzdVCsbeQzindMJtemvZXw== X-IPAS-Result: A0ApAABGJhdn/5D/Ja1aHAEBAQEBAQcBARIBAQQEAQGBfwcBAQsBgkqBT0NIjHJfiHKLdZIigSUDVg8BAQEPRAQBAYUHAoojAiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBDQEBBQEBAQIBBwWBDhOGCIZbAgEDJwsBRhBRKysZgwGCZQOvW4F5M4EB3jOBbIFIAY1FcIR3JxuBSUSBFAGDaIsHBI0qhmuGDgOCTXwliT1bkRtIgSEDWSECEQFVEw0KCwkFiTWCA4EjKYFrgQiDCIUlgWcJYYhHgQctgRGBHzqCA4E2SoU3Rz+CT2pONwINAjeCJIEAglGGR0ADCxgNSBEsNRQbBj5uB6x6RoJfB4EOei0HA4FyklgRkguBNJ9KhCShPxozqkyYd6Q6hGaBZzyBWTMaCBsVgyJSGQ+OLRYWzCgmMjsCBwsBAQMJjigBAQ IronPort-Data: A9a23:FZPx8KjfQDN3Fvrq+ZpJIsiMX161JxEKZh0ujC45NGQN5FlHY01je htvXD2GOq7eYzT1LYgna9vgo0hT6JKEzYcyS1Zs+Cs3FyNjpJueD7x1DKtf0wB+jyHnZBg6h ynLQoCYdKjYdleF+1HwdOKn9SAsvU2xbuKUIPbePSxsThNTRi4kiBZy88Y0mYcAbeKRW2thg vus5ZSGULOZ82QsaD5Ns/jZ8EoHUMna4Vv0gHRvPZing3eG/5UlJMp3Db28KXL+Xr5VEoaSL 87fzKu093/u5BwkDNWoiN7TKiXmlZaLYGBiIlIPM0STqkAqSh4ai87XB9JAAatjsAhlqvgqo Dl7WTNcfi9yVkHEsLx1vxC1iEiSN4UekFPMCSDXXcB+UyQqflO0q8iCAn3aMqUY49lOW0FRr 8YmLW48fy6exLOXn+mkH7wEasQLdKEHPasFsX1miDWcBvE8TNWbHOPB5MRT23E7gcUm8fT2P pVCL2EwKk6dPlsWZgt/5JEWxI9EglH5fjBDo1WfrII84nPYy0p6172F3N/9IIPVHJkPzhjBz o7A13nrDi8xL9aS8CiE9H6JpPPCmD3JaJ1HQdVU8dYv2jV/3Fc7DhAKWValiee2h1T4WN9FL UEQvC00osAPGFeDVNLxWVi85XWDpBNZA4UWGOwh4wbLwa3Ri+qEOlU5ovd6QIROnKcLqfYCj DdlQ/uB6eRTjYCo IronPort-HdrOrdr: A9a23:Tq/s76AJ7OFz/QvlHemr55DYdb4zR+YMi2TDGXofdfUzSL3+qy nAppUmPHPP5Qr5HUtQ++xoW5PwJU80i6QU3WB5B97LN2PbUSmTXeRfBODZrQEIdReTygck79 YCT0C7Y+eAdGSTSq3BkW+FL+o= X-Talos-CUID: 9a23:v5xsAWBNx+BdLpr6Eyl+y3dOOt9/SEPYkkbBHVL7WEc2E6LAHA== X-Talos-MUID: 9a23:Pzf2DQ3TJJmgD39Jl+NeMWrv9DUj/raiNmsWm5g/qciCOQlSJzqYkgaHXdpy X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.11,222,1725321600"; d="scan'208";a="277142861" Received: from rcdn-l-core-07.cisco.com ([173.37.255.144]) by rcdn-iport-6.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 22 Oct 2024 04:19:08 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-l-core-07.cisco.com (Postfix) with ESMTP id 9249A1800022B; Tue, 22 Oct 2024 04:19:07 +0000 (GMT) Received: by cisco.com (Postfix, from userid 412739) id 6739A20F2003; Mon, 21 Oct 2024 21:19:07 -0700 (PDT) From: Nelson Escobar To: netdev@vger.kernel.org Cc: satishkh@cisco.com, johndale@cisco.com, Nelson Escobar Subject: [Patch net-next 5/5] enic: Adjust used MSI-X wq/rq/cq/interrupt resources in a more robust way Date: Mon, 21 Oct 2024 21:17:07 -0700 Message-Id: <20241022041707.27402-6-neescoba@cisco.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20241022041707.27402-1-neescoba@cisco.com> References: <20241022041707.27402-1-neescoba@cisco.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: rcdn-l-core-07.cisco.com X-Patchwork-Delegate: kuba@kernel.org Instead of erroring out on probe if the resources are not configured exactly right in hardware, try to make due with the resources we do have. To accomplish this do the following: - Make enic_set_intr_mode() only set up interrupt related stuff. - Move resource adjustment out of enic_set_intr_mode() into its own function, and basing the resources used on the most constrained resource. - Move the kdump resources limitations into the new function too. Signed-off-by: Nelson Escobar Signed-off-by: John Daley Signed-off-by: Satish Kharat Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic_main.c | 197 ++++++++++---------- 1 file changed, 96 insertions(+), 101 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index a5d607be66b7..094112ab5e4a 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2440,113 +2440,120 @@ static void enic_tx_hang_reset(struct work_struct *work) rtnl_unlock(); } -static int enic_set_intr_mode(struct enic *enic) +static int enic_adjust_resources(struct enic *enic) { - unsigned int n = min_t(unsigned int, enic->rq_count, ENIC_RQ_MAX); - unsigned int m = min_t(unsigned int, enic->wq_count, ENIC_WQ_MAX); - unsigned int i; + unsigned int max_queues; + unsigned int rq_avail; + unsigned int wq_avail; - /* Set interrupt mode (INTx, MSI, MSI-X) depending - * on system capabilities. - * - * Try MSI-X first - * - * We need n RQs, m WQs, n+m CQs, and n+m+2 INTRs - * (the second to last INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) - */ + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; + } - for (i = 0; i < enic->intr_avail; i++) - enic->msix_entry[i].entry = i; + if (is_kdump_kernel()) { + dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); + enic->rq_avail = 1; + enic->wq_avail = 1; + enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; + enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; + enic->config.mtu = min_t(u16, 1500, enic->config.mtu); + } - /* Use multiple RQs if RSS is enabled - */ + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; - if (ENIC_SETTING(enic, RSS) && - enic->config.intr_mode < 1 && - enic->rq_count >= n && - enic->wq_count >= m && - enic->cq_count >= n + m && - enic->intr_count >= n + m + 2) { + switch (vnic_dev_get_intr_mode(enic->vdev)) { + case VNIC_DEV_INTR_MODE_INTX: + case VNIC_DEV_INTR_MODE_MSI: + enic->rq_count = 1; + enic->wq_count = 1; + enic->cq_count = 2; + enic->intr_count = enic->intr_avail; + break; + case VNIC_DEV_INTR_MODE_MSIX: + /* Adjust the number of wqs/rqs/cqs/interrupts that will be + * used based on which resource is the most constrained + */ + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_avail = min(enic->rq_avail, ENIC_RQ_MAX); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + if (wq_avail + rq_avail <= max_queues) { + /* we have enough cq and interrupt resources to cover + * the number of wqs and rqs + */ + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - n + m + 2, n + m + 2) > 0) { + break; + default: + dev_err(enic_get_dev(enic), "Unknown interrupt mode\n"); + return -EINVAL; + } - enic->rq_count = n; - enic->wq_count = m; - enic->cq_count = n + m; - enic->intr_count = n + m + 2; + return 0; +} - vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); +static int enic_set_intr_mode(struct enic *enic) +{ + unsigned int i; + int num_intr; - return 0; - } - } + /* Set interrupt mode (INTx, MSI, MSI-X) depending + * on system capabilities. + * + * Try MSI-X first + */ if (enic->config.intr_mode < 1 && - enic->rq_count >= 1 && - enic->wq_count >= m && - enic->cq_count >= 1 + m && - enic->intr_count >= 1 + m + 2) { - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - 1 + m + 2, 1 + m + 2) > 0) { - - enic->rq_count = 1; - enic->wq_count = m; - enic->cq_count = 1 + m; - enic->intr_count = 1 + m + 2; - + enic->intr_avail >= ENIC_MSIX_MIN_INTR) { + for (i = 0; i < enic->intr_avail; i++) + enic->msix_entry[i].entry = i; + + num_intr = pci_enable_msix_range(enic->pdev, enic->msix_entry, + ENIC_MSIX_MIN_INTR, + enic->intr_avail); + if (num_intr > 0) { vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - + VNIC_DEV_INTR_MODE_MSIX); + enic->intr_avail = num_intr; return 0; } } - /* Next try MSI - * - * We need 1 RQ, 1 WQ, 2 CQs, and 1 INTR - */ + /* Next try MSI */ if (enic->config.intr_mode < 2 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 1 && + enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 1; - + enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); - return 0; } - /* Next try INTx - * - * We need 1 RQ, 1 WQ, 2 CQs, and 3 INTRs - * (the first INTR is used for WQ/RQ) - * (the second INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) - */ + /* Next try INTx */ if (enic->config.intr_mode < 3 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 3) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 3; - + enic->intr_avail >= 3) { + enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); - return 0; } @@ -2776,18 +2783,6 @@ static void enic_dev_deinit(struct enic *enic) enic_free_enic_resources(enic); } -static void enic_kdump_kernel_config(struct enic *enic) -{ - if (is_kdump_kernel()) { - dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_count = 1; - enic->wq_count = 1; - enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; - enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; - enic->config.mtu = min_t(u16, 1500, enic->config.mtu); - } -} - static int enic_dev_init(struct enic *enic) { struct device *dev = enic_get_dev(enic); @@ -2823,14 +2818,7 @@ static int enic_dev_init(struct enic *enic) return err; } - /* modify resource count if we are in kdump_kernel - */ - enic_kdump_kernel_config(enic); - - /* Set interrupt mode based on resource counts and system - * capabilities - */ - + /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic); if (err) { dev_err(dev, "Failed to set intr mode based on resource " @@ -2838,6 +2826,13 @@ static int enic_dev_init(struct enic *enic) goto err_out_free_vnic_resources; } + /* Adjust resource counts based on most constrained resources */ + err = enic_adjust_resources(enic); + if (err) { + dev_err(dev, "Failed to adjust resources\n"); + goto err_out_free_vnic_resources; + } + /* Allocate and configure vNIC resources */