From patchwork Wed May 1 00:30:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Chan X-Patchwork-Id: 13650147 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B59F4C97 for ; Wed, 1 May 2024 00:31:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714523507; cv=none; b=btEja2F6G7UR5+Vhw6KlYG0vadY76JivQgxUS5fgHwLxUXp/bg2zbUNk7uyof4xObgspM+DdiAYKOKFP8g4tBWItwHIrdWNZlnXg+2MdRz6DS2v2Yvc6zGbRMzdHzw2Y0SyCM3KFT+1Mvjq+j2+BelSNoSn/rMvzWjOe9EfdayA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714523507; c=relaxed/simple; bh=JxAzk8KWG8zeHpSDKHDEp4s3uYVE1c7g6bC1RRBHNus=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=WGPz5xS4BD3HpcbS3EbGSGrXpUWM+ByaM60GCHFdURh/kCdV76UpqczqL9Qqqrih1kRJ/Ymb3nzKpsNpaP72C8pgDXk2tfWDa/2h+UdDSsb5NUqHLYpjFW0+2xB8jqfzq3WXImInf4Tt3DOLz5G+RuJqBYTu4rUF1Yhisp4iAXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=S/gSUmWW; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="S/gSUmWW" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-61b68644ab4so61509297b3.0 for ; Tue, 30 Apr 2024 17:31:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1714523504; x=1715128304; darn=vger.kernel.org; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=tHSb8FkF23eWmFvu5gHoSQDY/BD3mX4bf7+L4wZr8jw=; b=S/gSUmWWp7Qu1c7O5RnpkhOTlG+0W8wjuWK75PRP9EIYaOs2kVGooLAbsU2D7A6GWs 02ed3wpSWnSPUTydJPPDZqM3q0T2GP0iE2knR88SERsS1kkkzPFTe0xxWD+9dMQxhYlp rsiqQhOnunLQj8vI5U5T1zPq1ATUdpfjAk92Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714523504; x=1715128304; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tHSb8FkF23eWmFvu5gHoSQDY/BD3mX4bf7+L4wZr8jw=; b=CusZ7pZmcvm9ttyWKsNhXz0cXioqEwU+i+XQLKA+X1aQ4BoL6k1kW0NKBxyCIrSk/4 YzEJlHEPl1qxUYWDMNcHgeFOZfcekEymGMjyvg8kFSVy6Acm3Z/DUrj2FkZAWaDV2zRV Z0DFU6S2wDjq3o9tPu65jb4NxbeTFKu6SPm7vzWFjVw3Z89DeoKzmFwnbpfHv4zA4oIL eeBAuazfp1qOZLS2LSHTSXrjMjqDY5qVpJCelYASeS8LhZiy/HgCE2Fkb9WMXc7WGx5K TK2iQAZ15gok1NqKmTyowZHqfE+X0RS0x8YUsiYLyv2fdRC3N7DNv2uIJTTc2OTZyDQz dEjA== X-Gm-Message-State: AOJu0YxFw7WlcsoKggCyuLznN+AClusX+83OxTeKNdquIndk7bFJdI5U CyBbN9dH41PIFFTlPvfDDz5z7X/Xzsc6xfLPjX80HyYvr9dpEi0sGKCkpAzqTPZxTPZymi2w/ho = X-Google-Smtp-Source: AGHT+IER+z2fs5aQbGc+ieEXgZSqNS5U09BdyWigMbvyt/Jlau17pHTF34p3TcrljpPWa5j4A2kV8Q== X-Received: by 2002:a05:690c:39b:b0:61a:e856:85f1 with SMTP id bh27-20020a05690c039b00b0061ae85685f1mr1170827ywb.13.1714523504118; Tue, 30 Apr 2024 17:31:44 -0700 (PDT) Received: from lvnvda5233.lvn.broadcom.net ([192.19.161.250]) by smtp.gmail.com with ESMTPSA id f15-20020ac8470f000000b0043a7cb47069sm4337935qtp.9.2024.04.30.17.31.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Apr 2024 17:31:43 -0700 (PDT) From: Michael Chan To: davem@davemloft.net Cc: netdev@vger.kernel.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, andrew.gospodarek@broadcom.com, Kalesh AP , Selvin Thyparampil Xavier , Vikas Gupta , Pavan Chebbi Subject: [PATCH net-next v2 4/6] bnxt_en: Add a mutex to synchronize ULP operations Date: Tue, 30 Apr 2024 17:30:54 -0700 Message-Id: <20240501003056.100607-5-michael.chan@broadcom.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20240501003056.100607-1-michael.chan@broadcom.com> References: <20240501003056.100607-1-michael.chan@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Kalesh AP The current scheme relies heavily on the RTNL lock for all ULP operations between the L2 and the RoCE driver. Add a new en_dev_lock mutex so that the asynchronous ULP_STOP and ULP_START operations can be serialized with bnxt_register_dev() and bnxt_unregister_dev() calls without relying on the RTNL lock. The next patch will remove the RTNL lock from the ULP_STOP and ULP_START calls. Reviewed-by: Selvin Thyparampil Xavier Reviewed-by: Vikas Gupta Reviewed-by: Pavan Chebbi Signed-off-by: Kalesh AP Signed-off-by: Michael Chan Reviewed-by: Simon Horman --- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 20 ++++++++++++++++++- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 +++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index d8927838f1cf..ba3fa1c2e5d9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -113,6 +113,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, int rc = 0; rtnl_lock(); + mutex_lock(&edev->en_dev_lock); if (!bp->irq_tbl) { rc = -ENODEV; goto exit; @@ -136,6 +137,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, bnxt_fill_msix_vecs(bp, bp->edev->msix_entries); edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; exit: + mutex_unlock(&edev->en_dev_lock); rtnl_unlock(); return rc; } @@ -150,6 +152,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; rtnl_lock(); + mutex_lock(&edev->en_dev_lock); if (ulp->msix_requested) edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; edev->ulp_tbl->msix_requested = 0; @@ -165,6 +168,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) msleep(100); i++; } + mutex_unlock(&edev->en_dev_lock); rtnl_unlock(); return; } @@ -223,6 +227,12 @@ void bnxt_ulp_stop(struct bnxt *bp) if (!edev) return; + mutex_lock(&edev->en_dev_lock); + if (!bnxt_ulp_registered(edev)) { + mutex_unlock(&edev->en_dev_lock); + return; + } + edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; if (aux_priv) { struct auxiliary_device *adev; @@ -237,6 +247,7 @@ void bnxt_ulp_stop(struct bnxt *bp) adrv->suspend(adev, pm); } } + mutex_unlock(&edev->en_dev_lock); } void bnxt_ulp_start(struct bnxt *bp, int err) @@ -252,6 +263,12 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; + mutex_lock(&edev->en_dev_lock); + if (!bnxt_ulp_registered(edev)) { + mutex_unlock(&edev->en_dev_lock); + return; + } + if (edev->ulp_tbl->msix_requested) bnxt_fill_msix_vecs(bp, edev->msix_entries); @@ -267,7 +284,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err) adrv->resume(adev); } } - + mutex_unlock(&edev->en_dev_lock); } void bnxt_ulp_irq_stop(struct bnxt *bp) @@ -383,6 +400,7 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->l2_db_size = bp->db_size; edev->l2_db_size_nc = bp->db_size; edev->l2_db_offset = bp->db_offset; + mutex_init(&edev->en_dev_lock); if (bp->flags & BNXT_FLAG_ROCEV1_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index b86baf901a5d..4eafe6ec0abf 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -88,6 +88,9 @@ struct bnxt_en_dev { u16 ulp_num_msix_vec; u16 ulp_num_ctxs; + + /* serialize ulp operations */ + struct mutex en_dev_lock; }; static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev)