From patchwork Tue Apr 30 22:44:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Chan X-Patchwork-Id: 13650009 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF0381E86E for ; Tue, 30 Apr 2024 22:46:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714517181; cv=none; b=K+gCWyITm8Wv2Z5IgcMWUIe5SKQhhDtlpDRqStfMr5OV3UjPV+PDU250axgiOsqwCiNcOIUMOaikcIr12dUdYuW8dbibOnnXKe0CykZp5lxQM1GHtZRI0vl2J+Mc1mcEWOEa8oizPLUCWf9ugw9rAukvqPbV7V/GdjvX3ZjxZSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714517181; c=relaxed/simple; bh=kdzlwUF/UHFb2w0rbYfNxBu/C4MVgbSz5wt6LIrJALY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=LJ9nk0d34kNEHGUxt42HvmlH0B0Is7sAA1saJjzvlJi+7UIqSf+s5wMaW+HMDpAxGGZ766dHADlQA0vobe/CdKoTVUHFnjIcoAJcqYQCKa3PEmnKVun6CPRvAm6B7yM9gpzwG6M0it/jbtoxqFWkAoMnnmvMQrsJs7q7vrJeUMw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=TCiLGaSu; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="TCiLGaSu" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1ec41d82b8bso12336055ad.2 for ; Tue, 30 Apr 2024 15:46:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1714517179; x=1715121979; darn=vger.kernel.org; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=ow6G6TnDEsRyRmPppeXnL6X2ewoO6C6MophJ7XX5+0M=; b=TCiLGaSuc0NQ0CoDUFwauAauIUaBhBw4EvCFd2liX0EWIC3EQXPBbRTuA0LgEC1bYP kp1YFs5AJqT9PupaLsvpbFcx/8qiFgqhZn7Q+FmJoeyLRuCZswQLH4x9iXMyIY7n4Dse 7EFhhD0A+m7ynumBeU0pxMrLbFhJCnd3fij2g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714517179; x=1715121979; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ow6G6TnDEsRyRmPppeXnL6X2ewoO6C6MophJ7XX5+0M=; b=srSIvePd49OERfYRyT2LUytxW/jiv0ycE/lqEeIQLB9kX4bmpynGzcyVeQTSgcfe1u Bgd3eaW2Yw6Vb5HAZGC1lVAq7ZMcMESLk2oECR5HN7JOw3pU1yYBJihUfJs2MsJAz6zm PMRiniEhR4oTJYQjuyqYpLUDn6H/9tiyS4t48/P5vVT6lMedDCiTD6BLTut/RpfQty7B 7J71OJBpCWfNW3UW9h5WOjFBLe4V/6eveGTd2WUjurADJcXK/nJRyNvH+4izqXaO6U44 4PfUk3O0Mc0LPqy2ua9H95rKrevzYyU+5FQcGn6tnaCkuP7uV5hXvnMjzJzoF+d+Qezc ZYCg== X-Gm-Message-State: AOJu0Yzul6QgRMGWPLPGDMowVXZ+EJmBEBoX+ZTd0a0zRylzv8MgUZA7 9aYQ631Sm/4H2n57BS/r5OVI42pSCf4DAw7HcvUh9YosEeOxYWbFKG4cZ6czqQ== X-Google-Smtp-Source: AGHT+IHQErMnaGaL2b5tBUz4PWQL1B5EpRf0oBVkCmpLE8AvF6U1yvnOYxmeLMvxQs739UyciKemHg== X-Received: by 2002:a17:902:c403:b0:1e0:ca47:4d96 with SMTP id k3-20020a170902c40300b001e0ca474d96mr811625plk.3.1714517178668; Tue, 30 Apr 2024 15:46:18 -0700 (PDT) Received: from lvnvda5233.lvn.broadcom.net ([192.19.161.250]) by smtp.gmail.com with ESMTPSA id s11-20020a170902a50b00b001eb2fb28eabsm7836476plq.227.2024.04.30.15.46.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Apr 2024 15:46:17 -0700 (PDT) From: Michael Chan To: davem@davemloft.net Cc: netdev@vger.kernel.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, andrew.gospodarek@broadcom.com, Kalesh AP , Selvin Thyparampil Xavier , Vikas Gupta , Pavan Chebbi Subject: [PATCH net-next 5/7] bnxt_en: Add a mutex to synchronize ULP operations Date: Tue, 30 Apr 2024 15:44:36 -0700 Message-Id: <20240430224438.91494-6-michael.chan@broadcom.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20240430224438.91494-1-michael.chan@broadcom.com> References: <20240430224438.91494-1-michael.chan@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Kalesh AP The current scheme relies heavily on the RTNL lock for all ULP operations between the L2 and the RoCE driver. Add a new en_dev_lock mutex so that the asynchronous ULP_STOP and ULP_START operations can be serialized with bnxt_register_dev() and bnxt_unregister_dev() calls without relying on the RTNL lock. The next patch will remove the RTNL lock from the ULP_STOP and ULP_START calls. Reviewed-by: Selvin Thyparampil Xavier Reviewed-by: Vikas Gupta Reviewed-by: Pavan Chebbi Signed-off-by: Kalesh AP Signed-off-by: Michael Chan --- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 20 ++++++++++++++++++- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 +++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index d8927838f1cf..ba3fa1c2e5d9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -113,6 +113,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, int rc = 0; rtnl_lock(); + mutex_lock(&edev->en_dev_lock); if (!bp->irq_tbl) { rc = -ENODEV; goto exit; @@ -136,6 +137,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, bnxt_fill_msix_vecs(bp, bp->edev->msix_entries); edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; exit: + mutex_unlock(&edev->en_dev_lock); rtnl_unlock(); return rc; } @@ -150,6 +152,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; rtnl_lock(); + mutex_lock(&edev->en_dev_lock); if (ulp->msix_requested) edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; edev->ulp_tbl->msix_requested = 0; @@ -165,6 +168,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) msleep(100); i++; } + mutex_unlock(&edev->en_dev_lock); rtnl_unlock(); return; } @@ -223,6 +227,12 @@ void bnxt_ulp_stop(struct bnxt *bp) if (!edev) return; + mutex_lock(&edev->en_dev_lock); + if (!bnxt_ulp_registered(edev)) { + mutex_unlock(&edev->en_dev_lock); + return; + } + edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; if (aux_priv) { struct auxiliary_device *adev; @@ -237,6 +247,7 @@ void bnxt_ulp_stop(struct bnxt *bp) adrv->suspend(adev, pm); } } + mutex_unlock(&edev->en_dev_lock); } void bnxt_ulp_start(struct bnxt *bp, int err) @@ -252,6 +263,12 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; + mutex_lock(&edev->en_dev_lock); + if (!bnxt_ulp_registered(edev)) { + mutex_unlock(&edev->en_dev_lock); + return; + } + if (edev->ulp_tbl->msix_requested) bnxt_fill_msix_vecs(bp, edev->msix_entries); @@ -267,7 +284,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err) adrv->resume(adev); } } - + mutex_unlock(&edev->en_dev_lock); } void bnxt_ulp_irq_stop(struct bnxt *bp) @@ -383,6 +400,7 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->l2_db_size = bp->db_size; edev->l2_db_size_nc = bp->db_size; edev->l2_db_offset = bp->db_offset; + mutex_init(&edev->en_dev_lock); if (bp->flags & BNXT_FLAG_ROCEV1_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index b86baf901a5d..4eafe6ec0abf 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -88,6 +88,9 @@ struct bnxt_en_dev { u16 ulp_num_msix_vec; u16 ulp_num_ctxs; + + /* serialize ulp operations */ + struct mutex en_dev_lock; }; static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev)