From patchwork Fri Oct 27 23:22:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Chan X-Patchwork-Id: 13439309 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24DA24121C for ; Fri, 27 Oct 2023 23:23:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="HujTh2i/" Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7AFDC2 for ; Fri, 27 Oct 2023 16:23:47 -0700 (PDT) Received: by mail-qk1-x72e.google.com with SMTP id af79cd13be357-7789577b53fso184148785a.3 for ; Fri, 27 Oct 2023 16:23:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1698449027; x=1699053827; darn=vger.kernel.org; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=dl5XqIHLRaRqK1o4O9iEgVJ4uyv/31TdjCZfA7YeN0s=; b=HujTh2i/Tr6NeTvPxj2u2xEGXXMo/cu5eqN2zJGLadI4NgPyMoDlHStbJ0Q4ZDGMqX b8U8xv1dGPhKUS6/7NZZj7x3E9zCAX4Ois0xljPDtG0wQqiJ0R+FNkHyitnHKRV+LVuO oJb8IE0QJyzJSw0CC6QJxIsge2tMI48z7PLOw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698449027; x=1699053827; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dl5XqIHLRaRqK1o4O9iEgVJ4uyv/31TdjCZfA7YeN0s=; b=RzOuMP/BffUohBAM5Xv4i6+FMcrBrEm9boVK/oRMYz2cq7w8OTgD6bwtOl+9dPRI5i y4oy955lzWSpTz7jvjXRRVIINOqvKcL4iOzOOgUM6EXCkPMzJDpvBLCiy56CZYTMViye jt0wthBwyf+Jr5nE5RDd8kokHlz1DZaBEIID84PwEdsj4cgyAgzZ4yKhtjszurCGnUyV 0uAwVwy8j82R0FHs/BGOR6NBjzAc9eFZv8+ZdUrQApFqf1F5bg1ZlVXVBLeEptKgyRdf fOtWPA3WQhXbxpUTw6DQoX5CY1M3jPpqWeq9vYFYpI0ZlvMRwcexce7YWP4zkDW8dwMh 1GsQ== X-Gm-Message-State: AOJu0YympPxC5GAefuG+L87Q3cW3HSLO0NsEMHtRGZZ6x/MHuJzf+Xer nbNmNf0by+5wtP6WYv6vi+6jcA== X-Google-Smtp-Source: AGHT+IFeFW/rDOzySMTKo6fp5HYvPLKI38C6fMhlT9b4zz/0j8jCmRw0S90yyXUzH6oIWR7Pm/5j7g== X-Received: by 2002:a05:620a:4c4:b0:774:2113:7432 with SMTP id 4-20020a05620a04c400b0077421137432mr3950841qks.67.1698449026453; Fri, 27 Oct 2023 16:23:46 -0700 (PDT) Received: from lvnvda5233.lvn.broadcom.net ([192.19.161.250]) by smtp.gmail.com with ESMTPSA id y27-20020a05620a09db00b007742ad3047asm984169qky.54.2023.10.27.16.23.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Oct 2023 16:23:46 -0700 (PDT) From: Michael Chan To: davem@davemloft.net Cc: netdev@vger.kernel.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, gospo@broadcom.com, Andy Gospodarek Subject: [PATCH net-next 10/13] bnxt_en: Add helper to get the number of CP rings required for TX rings Date: Fri, 27 Oct 2023 16:22:49 -0700 Message-Id: <20231027232252.36111-11-michael.chan@broadcom.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20231027232252.36111-1-michael.chan@broadcom.com> References: <20231027232252.36111-1-michael.chan@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Up until now, each TX ring always requires a completion ring/NQ/MSIX. bnxt_trim_rings() and the assignment of bp->cp_nr_rings always make this assumption. This will no longer be true in the next patches, so we refactor and add helper functions to determine the proper relationship between TX rings and the required completion ring/NQ/MSIX. This patch does not change the 1:1 relationship yet. Note that on P5 chips, each RX and TX ring still requires a completion ring. Only the number of NQs has been reduced. We should no longer call bnxt_trim_rings() to adjust the RX and TX rings on P5 chips. Replace with simple logic to check that RX + TX < CP and adjust accordingly. bnxt_check_rings() should call _bnxt_get_max_rings() to get the raw number of rings instead of bnxt_get_max_rings(). If we are about to create TCs, bnxt_get_max_rings() would not be able to calculate the max rings correctly. Reviewed-by: Andy Gospodarek Signed-off-by: Michael Chan --- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 95 +++++++++++++++---- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 + .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 6 +- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 5 +- 4 files changed, 82 insertions(+), 25 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 6002b834e898..7c1a3db651f5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -6284,7 +6284,8 @@ static int bnxt_hwrm_get_rings(struct bnxt *bp) if (bp->flags & BNXT_FLAG_AGG_RINGS) rx >>= 1; if (cp < (rx + tx)) { - bnxt_trim_rings(bp, &rx, &tx, cp, false); + rx = cp / 2; + tx = rx; if (bp->flags & BNXT_FLAG_AGG_RINGS) rx <<= 1; hw_resc->resv_rx_rings = rx; @@ -6585,6 +6586,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp) int grp, rx_rings, rc; int vnic = 1, stat; bool sh = false; + int tx_cp; if (!bnxt_need_reserve_rings(bp)) return 0; @@ -6634,7 +6636,8 @@ static int __bnxt_reserve_rings(struct bnxt *bp) rc = bnxt_trim_rings(bp, &rx_rings, &tx, cp, sh); if (bp->flags & BNXT_FLAG_AGG_RINGS) rx = rx_rings << 1; - cp = sh ? max_t(int, tx, rx_rings) : tx + rx_rings; + tx_cp = bnxt_num_tx_to_cp(bp, tx); + cp = sh ? max_t(int, tx_cp, rx_rings) : tx_cp + rx_rings; bp->tx_nr_rings = tx; /* If we cannot reserve all the RX rings, reset the RSS map only @@ -9061,8 +9064,8 @@ static int bnxt_set_real_num_queues(struct bnxt *bp) return rc; } -static int bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max, - bool shared) +static int __bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max, + bool shared) { int _rx = *rx, _tx = *tx; @@ -9085,6 +9088,46 @@ static int bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max, return 0; } +static int __bnxt_num_tx_to_cp(struct bnxt *bp, int tx, int tx_sets, int tx_xdp) +{ + return tx; +} + +int bnxt_num_tx_to_cp(struct bnxt *bp, int tx) +{ + int tcs = netdev_get_num_tc(bp->dev); + + if (!tcs) + tcs = 1; + return __bnxt_num_tx_to_cp(bp, tx, tcs, bp->tx_nr_rings_xdp); +} + +static int bnxt_num_cp_to_tx(struct bnxt *bp, int tx_cp) +{ + int tcs = netdev_get_num_tc(bp->dev); + + return (tx_cp - bp->tx_nr_rings_xdp) * tcs + + bp->tx_nr_rings_xdp; +} + +static int bnxt_trim_rings(struct bnxt *bp, int *rx, int *tx, int max, + bool sh) +{ + int tx_cp = bnxt_num_tx_to_cp(bp, *tx); + + if (tx_cp != *tx) { + int tx_saved = tx_cp, rc; + + rc = __bnxt_trim_rings(bp, rx, &tx_cp, max, sh); + if (rc) + return rc; + if (tx_cp != tx_saved) + *tx = bnxt_num_cp_to_tx(bp, tx_cp); + return 0; + } + return __bnxt_trim_rings(bp, rx, tx, max, sh); +} + static void bnxt_setup_msix(struct bnxt *bp) { const int len = sizeof(bp->irq_tbl[0].name); @@ -9247,7 +9290,7 @@ static int bnxt_get_num_msix(struct bnxt *bp) static int bnxt_init_msix(struct bnxt *bp) { - int i, total_vecs, max, rc = 0, min = 1, ulp_msix; + int i, total_vecs, max, rc = 0, min = 1, ulp_msix, tx_cp; struct msix_entry *msix_ent; total_vecs = bnxt_get_num_msix(bp); @@ -9289,9 +9332,10 @@ static int bnxt_init_msix(struct bnxt *bp) if (rc) goto msix_setup_exit; + tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings); bp->cp_nr_rings = (min == 1) ? - max_t(int, bp->tx_nr_rings, bp->rx_nr_rings) : - bp->tx_nr_rings + bp->rx_nr_rings; + max_t(int, tx_cp, bp->rx_nr_rings) : + tx_cp + bp->rx_nr_rings; } else { rc = -ENOMEM; @@ -12186,23 +12230,27 @@ static void bnxt_sp_task(struct work_struct *work) clear_bit(BNXT_STATE_IN_SP_TASK, &bp->state); } +static void _bnxt_get_max_rings(struct bnxt *bp, int *max_rx, int *max_tx, + int *max_cp); + /* Under rtnl_lock */ int bnxt_check_rings(struct bnxt *bp, int tx, int rx, bool sh, int tcs, int tx_xdp) { - int max_rx, max_tx, tx_sets = 1; + int max_rx, max_tx, max_cp, tx_sets = 1, tx_cp; int tx_rings_needed, stats; int rx_rings = rx; - int cp, vnics, rc; + int cp, vnics; if (tcs) tx_sets = tcs; - rc = bnxt_get_max_rings(bp, &max_rx, &max_tx, sh); - if (rc) - return rc; + if (bp->flags & BNXT_FLAG_AGG_RINGS) + rx_rings <<= 1; - if (max_rx < rx) + _bnxt_get_max_rings(bp, &max_rx, &max_tx, &max_cp); + + if (max_rx < rx_rings) return -ENOMEM; tx_rings_needed = tx * tx_sets + tx_xdp; @@ -12211,11 +12259,12 @@ int bnxt_check_rings(struct bnxt *bp, int tx, int rx, bool sh, int tcs, vnics = 1; if ((bp->flags & (BNXT_FLAG_RFS | BNXT_FLAG_CHIP_P5)) == BNXT_FLAG_RFS) - vnics += rx_rings; + vnics += rx; - if (bp->flags & BNXT_FLAG_AGG_RINGS) - rx_rings <<= 1; - cp = sh ? max_t(int, tx_rings_needed, rx) : tx_rings_needed + rx; + tx_cp = __bnxt_num_tx_to_cp(bp, tx_rings_needed, tx_sets, tx_xdp); + cp = sh ? max_t(int, tx_cp, rx) : tx_cp + rx; + if (max_cp < cp) + return -ENOMEM; stats = cp; if (BNXT_NEW_RM(bp)) { cp += bnxt_get_ulp_msix_num(bp); @@ -12849,7 +12898,7 @@ int bnxt_setup_mq_tc(struct net_device *dev, u8 tc) { struct bnxt *bp = netdev_priv(dev); bool sh = false; - int rc; + int rc, tx_cp; if (tc > bp->max_tc) { netdev_err(dev, "Too many traffic classes requested: %d. Max supported is %d.\n", @@ -12880,8 +12929,9 @@ int bnxt_setup_mq_tc(struct net_device *dev, u8 tc) netdev_reset_tc(dev); } bp->tx_nr_rings += bp->tx_nr_rings_xdp; - bp->cp_nr_rings = sh ? max_t(int, bp->tx_nr_rings, bp->rx_nr_rings) : - bp->tx_nr_rings + bp->rx_nr_rings; + tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings); + bp->cp_nr_rings = sh ? max_t(int, tx_cp, bp->rx_nr_rings) : + tx_cp + bp->rx_nr_rings; if (netif_running(bp->dev)) return bnxt_open_nic(bp, true, false); @@ -13360,7 +13410,10 @@ static void _bnxt_get_max_rings(struct bnxt *bp, int *max_rx, int *max_tx, if (bp->flags & BNXT_FLAG_AGG_RINGS) *max_rx >>= 1; if (bp->flags & BNXT_FLAG_CHIP_P5) { - bnxt_trim_rings(bp, max_rx, max_tx, *max_cp, false); + if (*max_cp < (*max_rx + *max_tx)) { + *max_rx = *max_cp / 2; + *max_tx = *max_rx; + } /* On P5 chips, max_cp output param should be available NQs */ *max_cp = max_irq; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 2028233c0561..4ce993943924 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -2393,6 +2393,7 @@ int __bnxt_hwrm_get_tx_rings(struct bnxt *bp, u16 fid, int *tx_rings); int bnxt_nq_rings_in_use(struct bnxt *bp); int bnxt_hwrm_set_coal(struct bnxt *); void bnxt_free_ctx_mem(struct bnxt *bp); +int bnxt_num_tx_to_cp(struct bnxt *bp, int tx); unsigned int bnxt_get_max_func_stat_ctxs(struct bnxt *bp); unsigned int bnxt_get_avail_stat_ctxs_for_en(struct bnxt *bp); unsigned int bnxt_get_max_func_cp_rings(struct bnxt *bp); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c index 18c06158fead..76f2eab52ce7 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c @@ -923,6 +923,7 @@ static int bnxt_set_channels(struct net_device *dev, bool sh = false; int tx_xdp = 0; int rc = 0; + int tx_cp; if (channel->other_count) return -EINVAL; @@ -994,8 +995,9 @@ static int bnxt_set_channels(struct net_device *dev, if (tcs > 1) bp->tx_nr_rings = bp->tx_nr_rings_per_tc * tcs + tx_xdp; - bp->cp_nr_rings = sh ? max_t(int, bp->tx_nr_rings, bp->rx_nr_rings) : - bp->tx_nr_rings + bp->rx_nr_rings; + tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings); + bp->cp_nr_rings = sh ? max_t(int, tx_cp, bp->rx_nr_rings) : + tx_cp + bp->rx_nr_rings; /* After changing number of rx channels, update NTUPLE feature. */ netdev_update_features(dev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c index 52b75108e130..9d428eb3fdb9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c @@ -398,7 +398,7 @@ int bnxt_xdp_xmit(struct net_device *dev, int num_frames, static int bnxt_xdp_set(struct bnxt *bp, struct bpf_prog *prog) { struct net_device *dev = bp->dev; - int tx_xdp = 0, rc, tc; + int tx_xdp = 0, tx_cp, rc, tc; struct bpf_prog *old; if (prog && !prog->aux->xdp_has_frags && @@ -446,7 +446,8 @@ static int bnxt_xdp_set(struct bnxt *bp, struct bpf_prog *prog) } bp->tx_nr_rings_xdp = tx_xdp; bp->tx_nr_rings = bp->tx_nr_rings_per_tc * tc + tx_xdp; - bp->cp_nr_rings = max_t(int, bp->tx_nr_rings, bp->rx_nr_rings); + tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings); + bp->cp_nr_rings = max_t(int, tx_cp, bp->rx_nr_rings); bnxt_set_tpa_flags(bp); bnxt_set_ring_params(bp);