From patchwork Thu Feb 20 18:34:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984401 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCF93264619 for ; Thu, 20 Feb 2025 18:55:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077755; cv=none; b=nKnSTmjHV8zkvvAkmwCEMZkJPMXe5a61K1NszBqcKTHBAw8vwbMSvvX2P/gHv0k+mPbW2r1vcaN4Xwl0mLsTPzqCmCxYQ10454xrOFcl9im4EqjKergeGVz+xkz1pQ0csjqt3rna8qYcgawJB8NG+aI2JkNhYMep5u11CNM9kNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077755; c=relaxed/simple; bh=DRyXIrtbadlAOTbSf4eaomN59p5ysVYTfj15XxsNg3g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=uCl9Ln8hKtKgLthdT+C/EwkqIYb8+ZktE4eJ52VOqHC/w0SMF5huav8tpYCkbtOqC95Mwnf/3d+x79WvEtyV0b1rnjcN82EBudaz1fGI3Te1+TpLVBUMAAOZdvOVrtjFfUV6hIBUyJFFd5l8vmQtgPHvPV2ycGvN5ac5XhDzhg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=RZYB1PR6; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="RZYB1PR6" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220e83d65e5so25573315ad.1 for ; Thu, 20 Feb 2025 10:55:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077752; x=1740682552; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=10BDGIGdjI+2I+pAbbcrEY6QirgZ2x3nU1sHIH+QS5U=; b=RZYB1PR6ijthAtsMhmqlMzLE83mJx0DGZRroLbuD73yvXyF0mar2TdIGBlB3lx1ynh k1K0yiK5inm9H6NKUc0pBSdBBHPeDuRyCLh2dIauccOpSPbQrCkmmF99qO/KAfMLCjwh c8LSkOgvlmIiDQoOnX0eKXGyJBuZsTRgmOv3I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077752; x=1740682552; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=10BDGIGdjI+2I+pAbbcrEY6QirgZ2x3nU1sHIH+QS5U=; b=fpfGpPRdPLY8GVWgKuxI6ravk32cEQDnyG1WTe+VTpRwSI9P/SKVP6QMIvR0eM6Gov 3AMCnALz+WAhwlEdegQpmOghXfMs4Zpc99yrgbd9+pldwD6XgBRU/VhmGF3sYDKsUHLH EaLGcHyxtAOOmiWOrO4/E7jNU8qL+M7ZlqVWILqpUQM2SxKYvWAvTygH5DSqPT9tStuS jWu4DrlxRyWh5ttmMJzeocUxYoLOSO7k337WOW1yt2gO8eXhPVvVgkDNyyKAj6SK4NpQ LtF2++9b0Zs24aVqjrrculbHf1OzUEV5BZIUvgN6NVX56B9XVR3+Sz4yAs+gGMb2vAXu V8Xg== X-Forwarded-Encrypted: i=1; AJvYcCUPLfTn/emqCd3FV6E4G3vm+/8M6/wOmiHub1tVJ5E/0Ue4NTvjymWUC2H5pIWYrtdhtV6Fx9M=@vger.kernel.org X-Gm-Message-State: AOJu0YzQiwZMS+hltdEhuRmJp1wZYgxwEhFEC5MUa4R0wsZKAlEg5LF5 IFxioqiKMK7zkxr9jXC0zK0s3/zkGnDQsL4RKHjj48XKPo1wyL8AntofzF0qRA== X-Gm-Gg: ASbGncvA0ujVTyccxL0+TvDbKdFF3YjuXW7Ad2zw3CRteLGxFcwY8zqSOIaLwNkBlis YJ/8D+Cdpk8WEvGrxImHotvwM62gExhys80hRHtuDznNVEKOyoFtW+QQPWgelq2CVGrCUG7hkSM dSa18LeAcO7wGDo8t8Qfvipa9LaUkjVwUDFPqMxlpHL5Itmsmz2LQuxc3rAakglWVgYXn8F6xw4 J/bYvjJk7oLFvA0tF6DMiYtT4CTvqdocGCLpxybJ3V54TtFaHuXNo2ZHZ1mSPuTEn5oWnzY2q2n DQzfOL2Gk/ekNdXIAugXIYoDNOhoty2NEJRVxoiXVstMh6Do0g4xi4e/4g7OLU6pfMKFgNs= X-Google-Smtp-Source: AGHT+IFjAlHdQjg7jtPWPxhoIoMWi6AdSAqqq3IVW4vbmaqpvymP+c1FKAid6KJD2t7F+UXIzoE+KQ== X-Received: by 2002:a05:6a20:9184:b0:1ee:ce5b:853d with SMTP id adf61e73a8af0-1eef3dcc063mr381560637.39.1740077752042; Thu, 20 Feb 2025 10:55:52 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.55.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:55:51 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Saravanan Vajravel , Selvin Xavier Subject: [PATCH rdma-next 1/9] RDMA/bnxt_re: Add support for collecting the Queue dumps Date: Thu, 20 Feb 2025 10:34:48 -0800 Message-Id: <1740076496-14227-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Saravanan Vajravel As part of enhancing the debug data collection, allocate few data structures to hold the resources after the queues are destroyed. Initialize the data structures to capture the data. By default, driver will cache the info of the QPs that are in error state. The dump levels can be changed from debugfs hook in a later patch. Driver caches the info of the last 1024 entries only. Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 64 +++++++++++++++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/main.c | 18 ++++++++++ 2 files changed, 82 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index b33b04e..5818db1 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -173,6 +173,67 @@ static inline bool bnxt_re_chip_gen_p7(u16 chip_num) chip_num == CHIP_NUM_57608); } +#define BNXT_RE_MAX_QDUMP_ENTRIES 1024 + +struct qdump_qpinfo { + u32 id; + u32 dest_qpid; + u64 qp_handle; + u32 mtu; + u8 type; + u8 wqe_mode; + u8 state; + u8 is_user; + u64 scq_handle; + u64 rcq_handle; + u32 scq_id; + u32 rcq_id; +}; + +struct qdump_mrinfo { + int type; + u32 lkey; + u32 rkey; + u64 total_size; + u64 mr_handle; +}; + +struct qdump_element { + struct bnxt_qplib_pbl pbl[PBL_LVL_MAX]; + enum bnxt_qplib_pbl_lvl level; + struct bnxt_qplib_hwq *hwq; + struct bnxt_re_dev *rdev; + struct ib_umem *umem; + bool is_user_qp; + char des[32]; + char *buf; + size_t len; + u16 stride; + u32 prod; + u32 cons; +}; + +struct qdump_array { + struct qdump_qpinfo qpinfo; + struct qdump_mrinfo mrinfo; + bool valid; + bool is_mr; +}; + +struct bnxt_re_qdump_head { + struct qdump_array *qdump; + u32 max_elements; + struct mutex lock; /* lock qdump array elements */ + u32 index; +}; + +enum { + BNXT_RE_SNAPDUMP_NONE = 0, + BNXT_RE_SNAPDUMP_ERR, + /* Add new entry before this */ + BNXT_RE_SNAPDUMP_ALL +}; + struct bnxt_re_dev { struct ib_device ibdev; struct list_head list; @@ -232,6 +293,9 @@ struct bnxt_re_dev { unsigned long event_bitmap; struct bnxt_qplib_cc_param cc_param; struct workqueue_struct *dcb_wq; + /* Head to track all QP dump */ + struct bnxt_re_qdump_head qdump_head; + u8 snapdump_dbg_lvl; struct dentry *cc_config; struct bnxt_re_dbg_cc_config_params *cc_config_params; }; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index e9e4da4..87fdf69 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -2008,6 +2008,11 @@ static void bnxt_re_free_nqr_mem(struct bnxt_re_dev *rdev) rdev->nqr = NULL; } +static void bnxt_re_clean_qdump(struct bnxt_re_dev *rdev) +{ + vfree(rdev->qdump_head.qdump); +} + static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) { u8 type; @@ -2018,6 +2023,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type) bnxt_re_net_unregister_async_event(rdev); bnxt_re_uninit_dcb_wq(rdev); + bnxt_re_clean_qdump(rdev); if (test_and_clear_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags)) cancel_delayed_work_sync(&rdev->worker); @@ -2063,6 +2069,16 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } +static void bnxt_re_init_qdump(struct bnxt_re_dev *rdev) +{ + rdev->qdump_head.max_elements = BNXT_RE_MAX_QDUMP_ENTRIES; + rdev->qdump_head.index = 0; + rdev->snapdump_dbg_lvl = BNXT_RE_SNAPDUMP_ERR; + mutex_init(&rdev->qdump_head.lock); + rdev->qdump_head.qdump = vzalloc(rdev->qdump_head.max_elements * + sizeof(struct qdump_array)); +} + static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) { struct bnxt_re_ring_attr rattr = {}; @@ -2235,6 +2251,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) hash_init(rdev->srq_hash); bnxt_re_debugfs_add_pdev(rdev); + if (bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx)) + bnxt_re_init_qdump(rdev); bnxt_re_init_dcb_wq(rdev); bnxt_re_net_register_async_event(rdev); From patchwork Thu Feb 20 18:34:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984402 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC866264F8A for ; Thu, 20 Feb 2025 18:55:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077758; cv=none; b=crJfw/MK49Gyr5U2rAkuFSHmylLdmVYyZBtHK8XaJ4JhkMD/cBv3jLZ8W9UDqKeGk3iUMo2K+XD0z6Y5z06gmmB3uDhXy65UZpf7Ai29TOvwZX84ZTV9PDKv7ChUZDg99Apu4ZJFjNesY1Eb9G9eZj2C+D2A4Xhmtx3kU86F7YI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077758; c=relaxed/simple; bh=KFiItcYA1+vfCd+hinIriGeUe9fKsH3FKL1ViQr6mGI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=Uo+b6LLNWiQ5XyzvUDb1zj/xCv35jRTSI8DWFJUB0NXRgM6emGfSa2ROHG52cQlnw27hz19M4F3NsgL/jclnthD+R+S/oHMUjbwDXguD+M8MfMlJQH+N9Ss5/4nTthPjwaOEjsT5dtlqt2vPH/fxlPEL2zQ69uHcQgNu1pfIPjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=InWgO/iL; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="InWgO/iL" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-220e6028214so28841085ad.0 for ; Thu, 20 Feb 2025 10:55:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077756; x=1740682556; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Zr+FohAS+5s358/XL5DNb3kyfI1vHkhFvhvpgEYGLio=; b=InWgO/iLJemDlouldjP+zHXUSKGAXRN4owAQBVQXKHIwjYo8SjJ2pbZHfqVX2hO/4K v0BDTLPyCt5LqXfKGsg/TWPbIZeBtwuDPNJuTPyPGUXKbS97FkIjs6GkHZxpYpHxH/D9 E9SZu4MUxMjindphCFZgts8EPw4RFDujLyJZY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077756; x=1740682556; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zr+FohAS+5s358/XL5DNb3kyfI1vHkhFvhvpgEYGLio=; b=oMj+EaYdhTLa2REyf8yCyKDjhbx+wRvs+Y1lSHXPdJ5cWTqk4ssRUKaCDxADbyo7AY TmPWzYXJTgotGKgastBw9t7hdDkMKF36kpycRcQGTDtRr3QcJ0g+Ka1Tc2gq75YJWuVz zmarXZxaPcNhcwCRorPElnVjI4UQ86gAHBEdP4FnRcdiS7QOazwodBgO5MhbamgoEmvG 63z0SkPnSJb06N/o40L8+Zky7NqPuYKkHCpy+4tDvrhXHM2HjwqnqnjusOXkoqaYDf/D saabd+0nFq8Odm4HGWLjikqmNw0o2wtkUjjvIp9knFsZwx/13RwMkCe0GQCXPNl7QoRB hV1g== X-Forwarded-Encrypted: i=1; AJvYcCUlM1fuyYma/qAOEduV8U1wmw6Kbph5lFS6HQfNKF4M5Zc75RJT5MepRUurAjUgMyPvNzO9Ui0=@vger.kernel.org X-Gm-Message-State: AOJu0YwcOHdohxEhvWTUwS8p6WxrNlhEsuoubRHUorkPWSZorUq1e3ep R3QMMfWD5AGr8ty9zzw+oDsNTiE3VISUEIUGjqMIuKdoe4aHbdTJvjYghSCd7A== X-Gm-Gg: ASbGncuXc4WhcbDrLT8yCmeZ7NHSTF8b4jZDksmndcBKUQaIkNWySglFLI6Zg1znreS NAhbhiskq1DnM1N/xlA4Nnc4oOHcow8EQ4CJC48AeOkGevuSUe+VVCT7jACrh2FXbG0NPMJ6ATC FpRL5PN1AphjXGfgdLoHBcKgorpIYcs26Q8jbcIrAqGs8kvy6Cy/ddvfuynLL/D0nHqWqH/QERB mnG0L17kmgGExhTMH6Iu2XCL4jAYHdFFqnQEt8BYvD8pL05elhwHFAUJ19/DV0d2IEc2tKg1wln LW5bD5AspEKsQ4sHPQQCJAOHxv/gF/TRsa6yM7pdYE+45xwl7iXRf4z5+nL2LB3r3drUpqk= X-Google-Smtp-Source: AGHT+IG5H60oDdyi7JNgfkxGe8sLuWExbjYsWJu3/e2SJDRMel5HMa3BX5oHa9sFdtx0xPH3Zg+4fA== X-Received: by 2002:a05:6a20:9144:b0:1ee:e24d:8fdc with SMTP id adf61e73a8af0-1eef3c883f7mr379399637.10.1740077756102; Thu, 20 Feb 2025 10:55:56 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.55.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:55:55 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Saravanan Vajravel , Selvin Xavier Subject: [PATCH rdma-next 2/9] RDMA/bnxt_re: Cache the QP information Date: Thu, 20 Feb 2025 10:34:49 -0800 Message-Id: <1740076496-14227-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Saravanan Vajravel Add routines to cache the information about the QP before destroying. This can be retrieved and used for further debugging. The default behavior is to capture the QPs that are moved to error. The driver sets the flags to capture the dump from the error asynchronous event. Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 67 ++++++++++++++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/ib_verbs.h | 42 ++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/main.c | 8 +++- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 7 ++++ 4 files changed, 123 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 2de101d..060143e 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -954,6 +955,71 @@ static int bnxt_re_destroy_gsi_sqp(struct bnxt_re_qp *qp) return rc; } +static struct qdump_array *bnxt_re_get_next_qpdump(struct bnxt_re_dev *rdev) +{ + struct qdump_array *qdump; + u32 index; + + index = rdev->qdump_head.index; + qdump = &rdev->qdump_head.qdump[index]; + memset(qdump, 0, sizeof(*qdump)); + + index++; + index %= rdev->qdump_head.max_elements; + rdev->qdump_head.index = index; + + return qdump; +} + +/* + * bnxt_re_capture_qpdump - Capture snapshot of various queues of a QP. + * @qp - Pointer to QP for which data has to be collected + * + * This function will capture info about SQ/RQ/SCQ/RCQ of a QP which + * can be used to debug any issue + * + */ +void bnxt_re_capture_qpdump(struct bnxt_re_qp *qp) +{ + struct bnxt_qplib_qp *qpl = &qp->qplib_qp; + struct bnxt_re_dev *rdev = qp->rdev; + struct qdump_qpinfo *qpinfo; + struct qdump_array *qdump; + bool capture_snapdump; + + if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_NONE) + return; + + capture_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qpl->flags); + if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ERR && + !capture_snapdump) + return; + + if (qp->is_snapdump_captured || !rdev->qdump_head.qdump) + return; + + mutex_lock(&rdev->qdump_head.lock); + qdump = bnxt_re_get_next_qpdump(rdev); + + qpinfo = &qdump->qpinfo; + qpinfo->id = qpl->id; + qpinfo->dest_qpid = qpl->dest_qpn; + qpinfo->is_user = qpl->is_user; + qpinfo->mtu = qpl->mtu; + qpinfo->state = qpl->state; + qpinfo->type = qpl->type; + qpinfo->wqe_mode = qpl->wqe_mode; + qpinfo->qp_handle = qpl->qp_handle; + qpinfo->scq_handle = qp->scq->qplib_cq.cq_handle; + qpinfo->rcq_handle = qp->rcq->qplib_cq.cq_handle; + qpinfo->scq_id = qp->scq->qplib_cq.id; + qpinfo->rcq_id = qp->rcq->qplib_cq.id; + + qdump->valid = true; + qp->is_snapdump_captured = true; + mutex_unlock(&rdev->qdump_head.lock); +} + /* Queue Pairs */ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) { @@ -965,6 +1031,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) unsigned int flags; int rc; + bnxt_re_capture_qpdump(qp); bnxt_re_debug_rem_qpinfo(rdev, qp); bnxt_qplib_flush_cqn_wq(&qp->qplib_qp); diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index fbb16a4..8d82066 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -96,6 +96,7 @@ struct bnxt_re_qp { struct bnxt_re_cq *scq; struct bnxt_re_cq *rcq; struct dentry *dentry; + bool is_snapdump_captured; }; struct bnxt_re_cq { @@ -113,6 +114,7 @@ struct bnxt_re_cq { int resize_cqe; void *uctx_cq_page; struct hlist_node hash_entry; + bool is_snapdump_captured; }; struct bnxt_re_mr { @@ -267,6 +269,46 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata); void bnxt_re_dealloc_ucontext(struct ib_ucontext *context); int bnxt_re_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry); +void bnxt_re_capture_qpdump(struct bnxt_re_qp *qp); + +static inline const char *__to_qp_type_str(u8 type) +{ + switch (type) { + case CMDQ_CREATE_QP1_TYPE_GSI: + case CMDQ_CREATE_QP_TYPE_GSI: + return "GSI"; + case CMDQ_CREATE_QP_TYPE_RC: + return "RC"; + case CMDQ_CREATE_QP_TYPE_UD: + return "UD"; + case CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE: + return "RAW_ETH"; + default: + return "NotSupp"; + } +} + +static inline const char *__to_qp_state_str(u8 state) +{ + switch (state) { + case CMDQ_MODIFY_QP_NEW_STATE_RESET: + return "RESET"; + case CMDQ_MODIFY_QP_NEW_STATE_INIT: + return "INIT"; + case CMDQ_MODIFY_QP_NEW_STATE_RTR: + return "RTR"; + case CMDQ_MODIFY_QP_NEW_STATE_RTS: + return "RTS"; + case CMDQ_MODIFY_QP_NEW_STATE_SQD: + return "SQD"; + case CMDQ_MODIFY_QP_NEW_STATE_SQE: + return "SQE"; + case CMDQ_MODIFY_QP_NEW_STATE_ERR: + return "ERR"; + default: + return "NotSupp"; + } +} static inline u32 __to_ib_port_num(u16 port_id) { diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 87fdf69..67d2bf0 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -1420,9 +1420,15 @@ static int bnxt_re_handle_qp_async_event(struct creq_qp_event *qp_event, struct ib_event event = {}; unsigned int flags; - if (qp->qplib_qp.srq) + if (qp->qplib_qp.srq) { srq = container_of(qp->qplib_qp.srq, struct bnxt_re_srq, qplib_srq); + set_bit(SRQ_FLAGS_CAPTURE_SNAPDUMP, &srq->qplib_srq.flags); + } + + set_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qp->qplib_qp.flags); + set_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &qp->scq->qplib_cq.flags); + set_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &qp->rcq->qplib_cq.flags); if (qp->qplib_qp.state == CMDQ_MODIFY_QP_NEW_STATE_ERR && rdma_is_kernel_res(&qp->ib_qp.res)) { diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 0d9487c..d1acb01 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -106,6 +106,8 @@ struct bnxt_qplib_srq { u16 eventq_hw_ring_id; spinlock_t lock; /* protect SRQE link list */ u8 toggle; + unsigned long flags; +#define SRQ_FLAGS_CAPTURE_SNAPDUMP 1 }; struct bnxt_qplib_sge { @@ -279,6 +281,8 @@ struct bnxt_qplib_qp { u8 wqe_mode; u8 state; u8 cur_qp_state; + u16 ctx_size_sb; + u8 is_user; u64 modify_flags; u32 max_inline_data; u32 mtu; @@ -344,6 +348,8 @@ struct bnxt_qplib_qp { u32 msn_tbl_sz; bool is_host_msn_tbl; u8 tos_dscp; + unsigned long flags; +#define QP_FLAGS_CAPTURE_SNAPDUMP 1 }; #define BNXT_QPLIB_MAX_CQE_ENTRY_SIZE sizeof(struct cq_base) @@ -448,6 +454,7 @@ struct bnxt_qplib_cq { #define CQ_RESIZE_WAIT_TIME_MS 500 unsigned long flags; #define CQ_FLAGS_RESIZE_IN_PROG 1 +#define CQ_FLAGS_CAPTURE_SNAPDUMP 2 wait_queue_head_t waitq; struct list_head sqf_head, rqf_head; atomic_t arm_state; From patchwork Thu Feb 20 18:34:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984403 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B58E3264630 for ; Thu, 20 Feb 2025 18:56:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077762; cv=none; b=fnniODKINRLELVvSlrt0XCmb+gLON5wyC6BldCRXzu1NRlvYnyvmidF7V2LuL+c3rpa1TGilwQL1660iwkG5RuDQSXIa5KVucRAXjCMmGwb+Zh/kS/lviS60xQu8o+8rpMtpFEEeKPGjmFj5Qb8OYoQlqX+N94qiXi5ZrRFmZUA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077762; c=relaxed/simple; bh=OO35DOOYWqXdZ4O8hec5kL6h6m3RU0cEIxNWZ1peCks=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ZYUm2es5s1QtIMF7zRSzSDiiVXbYLS7A/kQAnSmycHKo4FtopQPP/tGLVe2ErrTU5CwFLS3dACHqIS333RElTCo5JrD8QaliukXYjLOzLXGfr8xGjf3Buu2FcAZ/Y9rpRKXOsjexFf5Hv7+v/3Ab0Hcfnel1PYH22CiXF3Wtf5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=CqV5TGlh; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="CqV5TGlh" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-220f048c038so23609315ad.2 for ; Thu, 20 Feb 2025 10:56:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077760; x=1740682560; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=AmBAXHsxWXsrXYjGGcmVq0b9Wu0WtSKDOr6er0hlrAA=; b=CqV5TGlh4p2MJMYKQNuZ7iLdmkqcP7mzR4vcAaxFq05Z97B3KF271+KIDYJWOVyT73 3f7XzcL9yFUQ85FNeceHxWy2z+sY9OsuFRNRqatR558ITaFFz/j+hxvdT1WO45D5aPWp THyQ4wo2jLnrPft0sAT+7IyoPSdK1grQ9W2EI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077760; x=1740682560; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AmBAXHsxWXsrXYjGGcmVq0b9Wu0WtSKDOr6er0hlrAA=; b=TRDbs8mllUaREUIfOidHwXsoT4Wnjao/qhJD/s29Uo1lnPcrizP74HRpjEsoLwVaez BvQ98qQq5RH0etLIFw9+N2iVMrfZCuE56fk3fnSuUu4B1t2nx6/WmTiiWTh1ifmPGmSR S50YbNlhZ33bNI9PZJPNLYwdinLYiDqMhLES7sfRa3SvPFuzHLE707Q3QRoAaHBNu75x ZtNk29edfpYXbmO6yDOGJpnLW2VDRClqsOe+Wy7Slz/fnGqMDfvZR5wNc/ZNoT9tKv9g zc5KkDHawTWXz/pBipcwhl9BgwbZiPKw0CVucBZIG+zJYbV95lRs/qG3Cam2Zb6/GGGn JE5g== X-Forwarded-Encrypted: i=1; AJvYcCUSaUXcXHXd4xtVurLU4eE/3aIfUaTNYIgl2Qju1Hwg4nrFmEacWtWnQOUQ+40g69PRg0eqmDc=@vger.kernel.org X-Gm-Message-State: AOJu0YxdWnr7AybsIAc/NaZAFNokHkpJ/MmvKk6wVzaCFwbe1Vl/IZ33 iTRoEQ3N2LiM7z4uKWhUgKDx6SLddH1w9emEQc4bAtRZRAU4F5d7R8L1h3WzFw== X-Gm-Gg: ASbGncuf7a8QCXws2FhUhxfJa1v6t0sTSCBfMTaMkk1rf6Ka1fvBOsJh3feGWmULxzo qxY3dKfXu3UdO5vWyNI41ViaER6xei2UyLRHT0iRr5FqLDo3++eq5MWhwvVSfFtZCztI++rngES H9ip3ejzj1/QCQb/TtspIvJuxxFxayMisLVjOZ9u8xNo3SNUz/4JyyOzzXXVHRrjHIEZQERJD5t nsmeS5Ue9oF12xi50eAygNca/qfch6N0pIW+O81R/8RLZmWEufC5UJcxIrehAEoVvZRH4Pars8+ tENmgq6DH+MsbTvyeAagJNZaGGaKXY75WLDthEnr0g/onfH4mP4VlRQsSkSAdZGpbqkajNk= X-Google-Smtp-Source: AGHT+IEvYFGe/Oy8MoaSy7XrmhDwHcku3iPlGrY6y4GGF0RxJLx00BwkgxaeUR0BhRu6J4lU7uyVIg== X-Received: by 2002:a05:6a21:9215:b0:1ee:e4f0:629 with SMTP id adf61e73a8af0-1eef3ca07b3mr368607637.18.1740077760124; Thu, 20 Feb 2025 10:56:00 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.55.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:55:59 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH rdma-next 3/9] RDMA/bnxt_re : Initialize the HW context dump collection Date: Thu, 20 Feb 2025 10:34:50 -0800 Message-Id: <1740076496-14227-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kashyap Desai HW context of the destroyed resources are returned during the destroy operation of the resource. Initialize the data structures to collect the HW contexts. Check if the FW supports the context dump and initialize the FW to send the dump to host. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 32 ++++++++++++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_rcfw.c | 2 ++ drivers/infiniband/hw/bnxt_re/qplib_rcfw.h | 12 +++++++++++ drivers/infiniband/hw/bnxt_re/qplib_res.h | 14 +++++++++++++ drivers/infiniband/hw/bnxt_re/roce_hsi.h | 2 ++ 5 files changed, 62 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 67d2bf0..6b5a169 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -2016,6 +2016,12 @@ static void bnxt_re_free_nqr_mem(struct bnxt_re_dev *rdev) static void bnxt_re_clean_qdump(struct bnxt_re_dev *rdev) { + struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; + + vfree(rcfw->qp_ctxm_data); + vfree(rcfw->cq_ctxm_data); + vfree(rcfw->srq_ctxm_data); + vfree(rcfw->mrw_ctxm_data); vfree(rdev->qdump_head.qdump); } @@ -2075,14 +2081,40 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } +static void bnxt_re_init_ctxm_size(struct bnxt_re_dev *rdev) +{ + struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; + + if (bnxt_qplib_is_chip_gen_p7(rdev->chip_ctx)) { + rcfw->qp_ctxm_size = BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P7; + rcfw->cq_ctxm_size = BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P7; + rcfw->srq_ctxm_size = BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7; + rcfw->mrw_ctxm_size = BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7; + + } else if (bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx)) { + rcfw->qp_ctxm_size = BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P5; + rcfw->cq_ctxm_size = BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P5; + rcfw->srq_ctxm_size = BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P5; + rcfw->mrw_ctxm_size = BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P5; + } +} + static void bnxt_re_init_qdump(struct bnxt_re_dev *rdev) { + struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; + rdev->qdump_head.max_elements = BNXT_RE_MAX_QDUMP_ENTRIES; rdev->qdump_head.index = 0; rdev->snapdump_dbg_lvl = BNXT_RE_SNAPDUMP_ERR; mutex_init(&rdev->qdump_head.lock); rdev->qdump_head.qdump = vzalloc(rdev->qdump_head.max_elements * sizeof(struct qdump_array)); + /* Setup Context cache information */ + bnxt_re_init_ctxm_size(rdev); + rcfw->qp_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->qp_ctxm_size); + rcfw->cq_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->cq_ctxm_size); + rcfw->srq_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->srq_ctxm_size); + rcfw->mrw_ctxm_data = vzalloc(BNXT_RE_MAX_QDUMP_ENTRIES * rcfw->mrw_ctxm_size); } static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 op_type) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c index 17e62f2..435b01f 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.c @@ -903,6 +903,8 @@ int bnxt_qplib_init_rcfw(struct bnxt_qplib_rcfw *rcfw, flags |= CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED; if (rcfw->res->en_dev->flags & BNXT_EN_FLAG_ROCE_VF_RES_MGMT) flags |= CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT; + if (is_roce_context_destroy_sb_enabled(rcfw->res->dattr->dev_cap_flags2)) + flags |= CMDQ_INITIALIZE_FW_FLAGS_DESTROY_CONTEXT_SB_SUPPORTED; req.flags |= cpu_to_le16(flags); req.stat_ctx_id = cpu_to_le32(ctx->stats.fw_id); bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h index 88814cb..155e24f 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_rcfw.h @@ -228,6 +228,18 @@ struct bnxt_qplib_rcfw { struct bnxt_qplib_qp_node *qp_tbl; /* To synchronize the qp-handle hash table */ spinlock_t tbl_lock; + u32 qp_ctxm_data_index; + u32 cq_ctxm_data_index; + u32 mrw_ctxm_data_index; + u32 srq_ctxm_data_index; + u16 qp_ctxm_size; + u16 cq_ctxm_size; + u16 mrw_ctxm_size; + u16 srq_ctxm_size; + void *qp_ctxm_data; + void *cq_ctxm_data; + void *mrw_ctxm_data; + void *srq_ctxm_data; u64 oos_prev; u32 init_oos_stats; u32 cmdq_depth; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index be5d907..93200ea 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -592,4 +592,18 @@ static inline bool _is_cq_coalescing_supported(u16 dev_cap_ext_flags2) return dev_cap_ext_flags2 & CREQ_QUERY_FUNC_RESP_SB_CQ_COALESCING_SUPPORTED; } +static inline bool is_roce_context_destroy_sb_enabled(u16 dev_cap_ext_flags2) +{ + return !!(dev_cap_ext_flags2 & CREQ_QUERY_FUNC_RESP_SB_DESTROY_CONTEXT_SB_SUPPORTED); +} + +#define BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P5 1088 +#define BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P5 128 +#define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P5 128 +#define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P5 192 + +#define BNXT_RE_CONTEXT_TYPE_QPC_SIZE_P7 1088 +#define BNXT_RE_CONTEXT_TYPE_CQ_SIZE_P7 192 +#define BNXT_RE_CONTEXT_TYPE_MRW_SIZE_P7 192 +#define BNXT_RE_CONTEXT_TYPE_SRQ_SIZE_P7 192 #endif /* __BNXT_QPLIB_RES_H__ */ diff --git a/drivers/infiniband/hw/bnxt_re/roce_hsi.h b/drivers/infiniband/hw/bnxt_re/roce_hsi.h index 0ee60fd..041e9ac 100644 --- a/drivers/infiniband/hw/bnxt_re/roce_hsi.h +++ b/drivers/infiniband/hw/bnxt_re/roce_hsi.h @@ -218,6 +218,7 @@ struct cmdq_initialize_fw { #define CMDQ_INITIALIZE_FW_FLAGS_HW_REQUESTER_RETX_SUPPORTED 0x2UL #define CMDQ_INITIALIZE_FW_FLAGS_OPTIMIZE_MODIFY_QP_SUPPORTED 0x8UL #define CMDQ_INITIALIZE_FW_FLAGS_L2_VF_RESOURCE_MGMT 0x10UL + #define CMDQ_INITIALIZE_FW_FLAGS_DESTROY_CONTEXT_SB_SUPPORTED 0x20UL __le16 cookie; u8 resp_size; u8 reserved8; @@ -2215,6 +2216,7 @@ struct creq_query_func_resp_sb { #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE (0x2UL << 4) #define CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_LAST \ CREQ_QUERY_FUNC_RESP_SB_REQ_RETRANSMISSION_SUPPORT_IQM_MSN_TABLE + #define CREQ_QUERY_FUNC_RESP_SB_DESTROY_CONTEXT_SB_SUPPORTED 0x100UL #define CREQ_QUERY_FUNC_RESP_SB_MIN_RNR_RTR_RTS_OPT_SUPPORTED 0x1000UL __le16 max_xp_qp_size; __le16 create_qp_batch_size; From patchwork Thu Feb 20 18:34:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984404 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72C4626560D for ; Thu, 20 Feb 2025 18:56:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077768; cv=none; b=fkSLA0+TUhxjWwLUJepyAddKVMGbnoGEO4aDmM3176dq9e5Q2efwwASRTJb9rg7mlEtjK35cwEw/l87JYULoyr4zDVgDQhdqXJxMZb/T2v47sgTsWy1M1Yn72pTyOTdoHfHAu6rKgRTjztYEqPfTgmacFkP4JZxdEfaZD/rcClE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077768; c=relaxed/simple; bh=e39Ye7f115z5V+apYQtx/SpSC66HkndM95PkDhQUplA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=LRX75m6zAKf1KWNmqAZk8R8xGW2AELf2JBUjQRLQpRcTqlRjghN7EnWKQov09vYcicS3+bfKoaH7H7TCHaNO2k0V4eE/e9Nm4qXQYjbj5lWVd0W/t5pT9ExroAYK5SUNjpFKT9N9MdWz8PTqRPe6qqSvPtGCc3+X27se/hXyEtY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=eK5IZSoe; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="eK5IZSoe" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-220f048c038so23610405ad.2 for ; Thu, 20 Feb 2025 10:56:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077765; x=1740682565; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=PBW5jpO95lyNQP0CO3WA3VL0tUEtqSN8vPLv1rH3qtg=; b=eK5IZSoemjr4SSUgJaZ+Uc700VsnHd2/2hKolark4FtoClu4vHkE2sQk5fvYGPd5kH hvDdjfhNbv1NvJjYZTFChzORUWSlFESwG0R6Shej8TzBpzDIf3FGHwvwjKYdfl2l8v8T sl+ArTwudajLuYNvUUFoKQq+qHdijVlb/nsj8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077765; x=1740682565; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PBW5jpO95lyNQP0CO3WA3VL0tUEtqSN8vPLv1rH3qtg=; b=JjopA6r3W1F8lfVw6lmfLosnl4lHOju6mKFlOWNzdTgyPbvCfKqVn66pHl9cT1i+jK adna8kaCF3gWFNzR6DOx23J1pwJtJ30b3QPiOCoOrnjzx3WIQjOIx9gceNp6DQlNxigm JcJlZXFEBzCsSXxIRi6rgNlyzf1pCpfP2fYUtKDrdJ2KWK/Mu+oJUU1E1mgSmJgsYlZ4 Ti3+GR22A3f8OJM/PKNRvGTsUTOEIgdUT8OHZYkxOpN05GEAIqlWViKK+IvUVVUrzESV kDCndsze5R6S2Y56g/SyMy5dkBZ7YqvvEtcg/0r3NNmLr5zETzl91IbZxdS9XQ3g+rmK 0XMA== X-Forwarded-Encrypted: i=1; AJvYcCX/K3QDfaG5wvFA2mYTgZnxaUuYg8UkDfJKmh4euWo8gsY4DAfxy/WAxvd2EAYqsYYPHix9Ras=@vger.kernel.org X-Gm-Message-State: AOJu0Yzu2A6AUtUHRbJ0Q6DkBkJlD9sclbbWMmEQrlAbpDYbQ+I0KFkN fbbmcGx2wvBtPdUhW06+HyIv3EfWO1juOGg6z2wNnKUjJMS7oIjgtrRpCQ3ewQ== X-Gm-Gg: ASbGnctMtrtWy739bqEHTaGG6hVrKA49wxkzK12+bx/3BVfNGQuDdD6c/nNwUgCQWIn 0g7yL6DqF45OrQFHc2q3d+ZPdvmhUydCm2Vgqh+ELNFLFAr9MtyVFCsE1A5WUsG2ox5MST+RsAD 7wZPOzj5WjBsOmQq2+WQ8/oKskc71Evp/4uHMp0Mx0tjbkpDLrGzKgpJWXi+klPFBoJalh5UrcH OxYMMcJubjfpvmfvdMo9eEDHb++jtxqqGLHV7/jbC+asEYmVV5+/udpFiXEwN489fT7dRiKFCHd v5deHTZryODVVtpDqepWR5bWuW7fYemexbhYPwZSV0UhJgXhsWSQuG622stz7l+juRB35N0= X-Google-Smtp-Source: AGHT+IGgxyoRm0rro/+RGU0DQSiNh0KMz3mFvEijqmEYGx7+Kd8cXubMPj5jjq6aQuAQ/+11wvdvFQ== X-Received: by 2002:a05:6a21:3949:b0:1ee:c7c8:cad with SMTP id adf61e73a8af0-1eef3dc67c0mr335619637.28.1740077764264; Thu, 20 Feb 2025 10:56:04 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:03 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH rdma-next 4/9] RDMA/bnxt_re: Get the resource contexts from the HW Date: Thu, 20 Feb 2025 10:34:51 -0800 Message-Id: <1740076496-14227-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kashyap Desai While destroying the resources like QP/CQ/SRQ/MRs, FW sends the contexts to the host. Driver caches these information to be dumped later for debugging. Driver can hold the information of the last 1024 entries of each resources. Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 9 +++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 120 +++++++++++++++++++++++++++---- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 47 +++++++++++- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 9 ++- drivers/infiniband/hw/bnxt_re/qplib_sp.c | 15 +++- drivers/infiniband/hw/bnxt_re/qplib_sp.h | 3 +- 6 files changed, 181 insertions(+), 22 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 5818db1..45601635 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -175,6 +175,15 @@ static inline bool bnxt_re_chip_gen_p7(u16 chip_num) #define BNXT_RE_MAX_QDUMP_ENTRIES 1024 +enum { + BNXT_RE_RES_TYPE_CQ = 0, + BNXT_RE_RES_TYPE_UCTX, + BNXT_RE_RES_TYPE_QP, + BNXT_RE_RES_TYPE_SRQ, + BNXT_RE_RES_TYPE_MR, + BNXT_RE_RES_TYPE_MAX +}; + struct qdump_qpinfo { u32 id; u32 dest_qpid; diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 060143e..a1ee6ca 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -537,7 +537,7 @@ static void bnxt_re_destroy_fence_mr(struct bnxt_re_pd *pd) bnxt_qplib_dereg_mrw(&rdev->qplib_res, &mr->qplib_mr, true); if (mr->ib_mr.lkey) - bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); + bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL); kfree(mr); fence->mr = NULL; } @@ -886,6 +886,55 @@ int bnxt_re_query_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr) return 0; } +static inline void bnxt_re_save_resource_context(struct bnxt_re_dev *rdev, + void *ctx_sb_data, + u8 res_type, bool do_snapdump) +{ + void *drv_ctx_data; + u32 *ctx_index; + u16 ctx_size; + + if (!ctx_sb_data) + return; + + switch (res_type) { + case BNXT_RE_RES_TYPE_QP: + drv_ctx_data = rdev->rcfw.qp_ctxm_data; + ctx_index = &rdev->rcfw.qp_ctxm_data_index; + ctx_size = rdev->rcfw.qp_ctxm_size; + break; + case BNXT_RE_RES_TYPE_CQ: + drv_ctx_data = rdev->rcfw.cq_ctxm_data; + ctx_index = &rdev->rcfw.cq_ctxm_data_index; + ctx_size = rdev->rcfw.cq_ctxm_size; + break; + case BNXT_RE_RES_TYPE_MR: + drv_ctx_data = rdev->rcfw.mrw_ctxm_data; + ctx_index = &rdev->rcfw.mrw_ctxm_data_index; + ctx_size = rdev->rcfw.mrw_ctxm_size; + break; + case BNXT_RE_RES_TYPE_SRQ: + drv_ctx_data = rdev->rcfw.srq_ctxm_data; + ctx_index = &rdev->rcfw.srq_ctxm_data_index; + ctx_size = rdev->rcfw.srq_ctxm_size; + break; + default: + return; + } + + if (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ALL || + (rdev->snapdump_dbg_lvl == BNXT_RE_SNAPDUMP_ERR && do_snapdump)) { + memcpy(drv_ctx_data + (*ctx_index * ctx_size), + ctx_sb_data, ctx_size); + *ctx_index = *ctx_index + 1; + *ctx_index = *ctx_index % BNXT_RE_MAX_QDUMP_ENTRIES; + dev_dbg(rdev_to_dev(rdev), + "%s : res_type %d ctx_index %d 0x%lx\n", __func__, + res_type, *ctx_index, + (unsigned long)drv_ctx_data + (*ctx_index * ctx_size)); + } +} + unsigned long bnxt_re_lock_cqs(struct bnxt_re_qp *qp) __acquires(&qp->scq->cq_lock) __acquires(&qp->rcq->cq_lock) { @@ -930,7 +979,7 @@ static int bnxt_re_destroy_gsi_sqp(struct bnxt_re_qp *qp) bnxt_qplib_clean_qp(&qp->qplib_qp); ibdev_dbg(&rdev->ibdev, "Destroy the shadow QP\n"); - rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &gsi_sqp->qplib_qp); + rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &gsi_sqp->qplib_qp, 0, NULL); if (rc) { ibdev_err(&rdev->ibdev, "Destroy Shadow QP failed"); goto fail; @@ -1029,17 +1078,28 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) struct bnxt_qplib_nq *scq_nq = NULL; struct bnxt_qplib_nq *rcq_nq = NULL; unsigned int flags; + void *ctx_sb_data = NULL; + bool do_snapdump; + u16 ctx_size; int rc; bnxt_re_capture_qpdump(qp); bnxt_re_debug_rem_qpinfo(rdev, qp); bnxt_qplib_flush_cqn_wq(&qp->qplib_qp); + ctx_size = qplib_qp->ctx_size_sb; + if (ctx_size) + ctx_sb_data = vzalloc(ctx_size); + do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags); - rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); + rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, ctx_size, ctx_sb_data); if (rc) ibdev_err(&rdev->ibdev, "Failed to destroy HW QP"); + else + bnxt_re_save_resource_context(rdev, ctx_sb_data, + BNXT_RE_RES_TYPE_QP, do_snapdump); + vfree(ctx_sb_data); if (rdma_is_kernel_res(&qp->ib_qp.res)) { flags = bnxt_re_lock_cqs(qp); bnxt_qplib_clean_qp(&qp->qplib_qp); @@ -1599,7 +1659,7 @@ static int bnxt_re_create_shadow_gsi(struct bnxt_re_qp *qp, &qp->qplib_qp); if (!sah) { bnxt_qplib_destroy_qp(&rdev->qplib_res, - &sqp->qplib_qp); + &sqp->qplib_qp, 0, NULL); rc = -ENODEV; ibdev_err(&rdev->ibdev, "Failed to create AH entry for ShadowQP"); @@ -1747,7 +1807,7 @@ int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, return 0; qp_destroy: - bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp); + bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, 0, NULL); free_umem: ib_umem_release(qp->rumem); ib_umem_release(qp->sumem); @@ -1841,6 +1901,9 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata) struct bnxt_re_dev *rdev = srq->rdev; struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq; struct bnxt_qplib_nq *nq = NULL; + void *ctx_sb_data = NULL; + bool do_snapdump; + u16 ctx_size; if (qplib_srq->cq) nq = qplib_srq->cq->nq; @@ -1848,7 +1911,18 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata) free_page((unsigned long)srq->uctx_srq_page); hash_del(&srq->hash_entry); } - bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq); + ctx_size = rdev->rcfw.srq_ctxm_size; + if (ctx_size) + ctx_sb_data = vzalloc(ctx_size); + + bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq, ctx_size, ctx_sb_data); + + do_snapdump = test_bit(SRQ_FLAGS_CAPTURE_SNAPDUMP, &qplib_srq->flags); + bnxt_re_save_resource_context(rdev, ctx_sb_data, + BNXT_RE_RES_TYPE_SRQ, do_snapdump); + + vfree(ctx_sb_data); + ib_umem_release(srq->umem); atomic_dec(&rdev->stats.res.srq_count); if (nq) @@ -1968,7 +2042,7 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq, if (rc) { ibdev_err(&rdev->ibdev, "SRQ copy to udata failed!"); bnxt_qplib_destroy_srq(&rdev->qplib_res, - &srq->qplib_srq); + &srq->qplib_srq, 0, NULL); goto fail; } } @@ -3127,6 +3201,9 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata) struct bnxt_qplib_nq *nq; struct bnxt_re_dev *rdev; struct bnxt_re_cq *cq; + void *ctx_sb_data = NULL; + bool do_snapdump; + u16 ctx_size; cq = container_of(ib_cq, struct bnxt_re_cq, ib_cq); rdev = cq->rdev; @@ -3137,7 +3214,15 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata) free_page((unsigned long)cq->uctx_cq_page); hash_del(&cq->hash_entry); } - bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq); + ctx_size = rdev->rcfw.cq_ctxm_size; + if (ctx_size) + ctx_sb_data = vzalloc(ctx_size); + bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq, ctx_size, ctx_sb_data); + do_snapdump = test_bit(CQ_FLAGS_CAPTURE_SNAPDUMP, &cq->qplib_cq.flags); + bnxt_re_save_resource_context(rdev, ctx_sb_data, + BNXT_RE_RES_TYPE_CQ, do_snapdump); + + vfree(ctx_sb_data); bnxt_re_put_nq(rdev, nq); ib_umem_release(cq->umem); @@ -3247,7 +3332,7 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, rc = ib_copy_to_udata(udata, &resp, min(sizeof(resp), udata->outlen)); if (rc) { ibdev_err(&rdev->ibdev, "Failed to copy CQ udata"); - bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq); + bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq, 0, NULL); goto free_mem; } } @@ -4070,7 +4155,7 @@ struct ib_mr *bnxt_re_get_dma_mr(struct ib_pd *ib_pd, int mr_access_flags) return &mr->ib_mr; fail_mr: - bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); + bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL); fail: kfree(mr); return ERR_PTR(rc); @@ -4080,13 +4165,20 @@ int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata) { struct bnxt_re_mr *mr = container_of(ib_mr, struct bnxt_re_mr, ib_mr); struct bnxt_re_dev *rdev = mr->rdev; + void *ctx_sb_data = NULL; + u16 ctx_size; int rc; - rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); + ctx_size = rdev->rcfw.mrw_ctxm_size; + if (ctx_size) + ctx_sb_data = vzalloc(ctx_size); + rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, ctx_size, ctx_sb_data); if (rc) { ibdev_err(&rdev->ibdev, "Dereg MR failed: %#x\n", rc); return rc; } + bnxt_re_save_resource_context(rdev, ctx_sb_data, BNXT_RE_RES_TYPE_MR, 0); + vfree(ctx_sb_data); if (mr->pages) { rc = bnxt_qplib_free_fast_reg_page_list(&rdev->qplib_res, @@ -4175,7 +4267,7 @@ struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type type, fail_mr: kfree(mr->pages); fail: - bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); + bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL); bail: kfree(mr); return ERR_PTR(rc); @@ -4222,7 +4314,7 @@ int bnxt_re_dealloc_mw(struct ib_mw *ib_mw) struct bnxt_re_dev *rdev = mw->rdev; int rc; - rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mw->qplib_mw); + rc = bnxt_qplib_free_mrw(&rdev->qplib_res, &mw->qplib_mw, 0, NULL); if (rc) { ibdev_err(&rdev->ibdev, "Free MW failed: %#x\n", rc); return rc; @@ -4301,7 +4393,7 @@ static struct ib_mr *__bnxt_re_user_reg_mr(struct ib_pd *ib_pd, u64 length, u64 return &mr->ib_mr; free_mrw: - bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr); + bnxt_qplib_free_mrw(&rdev->qplib_res, &mr->qplib_mr, 0, NULL); free_mr: kfree(mr); return ERR_PTR(rc); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 5336f74..fa221a7 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -616,14 +616,21 @@ int bnxt_qplib_alloc_nq(struct bnxt_qplib_res *res, struct bnxt_qplib_nq *nq) /* SRQ */ void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res, - struct bnxt_qplib_srq *srq) + struct bnxt_qplib_srq *srq, u16 sb_resp_size, void *sb_resp_va) { struct bnxt_qplib_rcfw *rcfw = res->rcfw; struct creq_destroy_srq_resp resp = {}; + struct bnxt_qplib_rcfw_sbuf sbuf = {}; struct bnxt_qplib_cmdqmsg msg = {}; struct cmdq_destroy_srq req = {}; int rc; + if (sb_resp_size && sb_resp_va) { + sbuf.size = sb_resp_size; + sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size, + &sbuf.dma_addr, GFP_KERNEL); + } + bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, CMDQ_BASE_OPCODE_DESTROY_SRQ, sizeof(req)); @@ -632,7 +639,13 @@ void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res, req.srq_cid = cpu_to_le32(srq->id); bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); + req.resp_addr = cpu_to_le64(sbuf.dma_addr); + req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS; rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); + if (sbuf.sb) { + memcpy(sb_resp_va, sbuf.sb, sb_resp_size); + dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); + } kfree(srq->swq); if (rc) return; @@ -1593,15 +1606,22 @@ static void __clean_cq(struct bnxt_qplib_cq *cq, u64 qp) } int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, - struct bnxt_qplib_qp *qp) + struct bnxt_qplib_qp *qp, + u16 sb_resp_size, void *sb_resp_va) { struct bnxt_qplib_rcfw *rcfw = res->rcfw; + struct bnxt_qplib_rcfw_sbuf sbuf = {}; struct creq_destroy_qp_resp resp = {}; struct bnxt_qplib_cmdqmsg msg = {}; struct cmdq_destroy_qp req = {}; u32 tbl_indx; int rc; + if (sb_resp_size && sb_resp_va) { + sbuf.size = sb_resp_size; + sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size, + &sbuf.dma_addr, GFP_KERNEL); + } spin_lock_bh(&rcfw->tbl_lock); tbl_indx = map_qp_id_to_tbl_indx(qp->id, rcfw); rcfw->qp_tbl[tbl_indx].qp_id = BNXT_QPLIB_QP_ID_INVALID; @@ -1613,6 +1633,8 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, sizeof(req)); req.qp_cid = cpu_to_le32(qp->id); + req.resp_addr = cpu_to_le64(sbuf.dma_addr); + req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS; bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); @@ -1623,6 +1645,10 @@ int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, spin_unlock_bh(&rcfw->tbl_lock); return rc; } + if (sbuf.sb) { + memcpy(sb_resp_va, sbuf.sb, sb_resp_size); + dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); + } return 0; } @@ -2355,25 +2381,40 @@ int bnxt_qplib_resize_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq, return rc; } -int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq) +int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq, + u16 sb_resp_size, void *sb_resp_va) { struct bnxt_qplib_rcfw *rcfw = res->rcfw; + struct bnxt_qplib_rcfw_sbuf sbuf = {}; struct creq_destroy_cq_resp resp = {}; struct bnxt_qplib_cmdqmsg msg = {}; struct cmdq_destroy_cq req = {}; u16 total_cnq_events; int rc; + if (sb_resp_size && sb_resp_va) { + sbuf.size = sb_resp_size; + sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size, + &sbuf.dma_addr, GFP_KERNEL); + } + bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, CMDQ_BASE_OPCODE_DESTROY_CQ, sizeof(req)); + req.resp_addr = cpu_to_le64(sbuf.dma_addr); + req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS; + req.cq_cid = cpu_to_le32(cq->id); bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); if (rc) return rc; + if (sbuf.sb) { + memcpy(sb_resp_va, sbuf.sb, sb_resp_size); + dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); + } total_cnq_events = le16_to_cpu(resp.total_cnq_events); __wait_for_all_nqes(cq, total_cnq_events); bnxt_qplib_free_hwq(res, &cq->hwq); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index d1acb01..1710b59 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -555,14 +555,16 @@ int bnxt_qplib_modify_srq(struct bnxt_qplib_res *res, int bnxt_qplib_query_srq(struct bnxt_qplib_res *res, struct bnxt_qplib_srq *srq); void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res, - struct bnxt_qplib_srq *srq); + struct bnxt_qplib_srq *srq, + u16 sb_resp_size, void *sb_resp_va); int bnxt_qplib_post_srq_recv(struct bnxt_qplib_srq *srq, struct bnxt_qplib_swqe *wqe); int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); -int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); +int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp, + u16 ctx_size, void *ctx_sb_data); void bnxt_qplib_clean_qp(struct bnxt_qplib_qp *qp); void bnxt_qplib_free_qp_res(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp); @@ -584,7 +586,8 @@ int bnxt_qplib_resize_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq, int new_cqes); void bnxt_qplib_resize_cq_complete(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq); -int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq); +int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq, + u16 sb_resp_size, void *sb_resp_v); int bnxt_qplib_poll_cq(struct bnxt_qplib_cq *cq, struct bnxt_qplib_cqe *cqe, int num, struct bnxt_qplib_qp **qp); bool bnxt_qplib_is_cq_empty(struct bnxt_qplib_cq *cq); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c index 4ccd440..d2c519d 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c @@ -504,10 +504,12 @@ int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah, } /* MRW */ -int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw) +int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw, + u16 sb_resp_size, void *sb_resp_va) { struct creq_deallocate_key_resp resp = {}; struct bnxt_qplib_rcfw *rcfw = res->rcfw; + struct bnxt_qplib_rcfw_sbuf sbuf = {}; struct cmdq_deallocate_key req = {}; struct bnxt_qplib_cmdqmsg msg = {}; int rc; @@ -517,6 +519,11 @@ int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw) return 0; } + if (sb_resp_size && sb_resp_va) { + sbuf.size = sb_resp_size; + sbuf.sb = dma_alloc_coherent(&rcfw->pdev->dev, sbuf.size, + &sbuf.dma_addr, GFP_KERNEL); + } bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, CMDQ_BASE_OPCODE_DEALLOCATE_KEY, sizeof(req)); @@ -530,12 +537,18 @@ int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw) else req.key = cpu_to_le32(mrw->lkey); + req.resp_addr = cpu_to_le64(sbuf.dma_addr); + req.resp_size = sb_resp_size / BNXT_QPLIB_CMDQE_UNITS; bnxt_qplib_fill_cmdqmsg(&msg, &req, &resp, NULL, sizeof(req), sizeof(resp), 0); rc = bnxt_qplib_rcfw_send_message(rcfw, &msg); if (rc) return rc; + if (sbuf.sb) { + memcpy(sb_resp_va, sbuf.sb, sb_resp_size); + dma_free_coherent(&rcfw->pdev->dev, sbuf.size, sbuf.sb, sbuf.dma_addr); + } /* Free the qplib's MRW memory */ if (mrw->hwq.max_elements) bnxt_qplib_free_hwq(res, &mrw->hwq); diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h index e626b05..728b1f1 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h @@ -340,7 +340,8 @@ int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw, bool block); int bnxt_qplib_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, struct ib_umem *umem, int num_pbls, u32 buf_pg_size); -int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr); +int bnxt_qplib_free_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, + u16 ctx_size, void *ctx_sb_data); int bnxt_qplib_alloc_fast_reg_mr(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mr, int max); int bnxt_qplib_alloc_fast_reg_page_list(struct bnxt_qplib_res *res, From patchwork Thu Feb 20 18:34:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984405 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08CFB265622 for ; Thu, 20 Feb 2025 18:56:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077770; cv=none; b=TWYimRCM19tTI0lbVy4H3FX4uwz5ZvN2OoLI2XzP3IEBsN8iRSHz7madx7smgrdkWxMcOa5z+5lWyNvTByCCqKdr7whrEkSCXEtKDqYh+wGNzSkMjrSHq5HLodgEudf6JupHJtBWK8O76oC1GJiWaBSOcDqcI0nqO4hgFRaZtME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077770; c=relaxed/simple; bh=D8r3NMs/v/gdOgXckK137771Z7EmXwYtM9x6L549VTk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=AIui9ZRhXfIzsgJO7b0g8+rD6AG7nWDmqNrEPZjHKJZpzrHDwHi3hYUdfG9sFwqzw5u8aNHlGwCzl/4CoAuluVtCleNuq/BboIwgi/sL9dtWVIXRvRlWgazL0TqCOVopy9yPqLY8cxGmTbOoaA0rSrCYMswKqZ3dNByDQ2s1k/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=c5twu1Ee; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="c5twu1Ee" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-22185cddbffso35409825ad.1 for ; Thu, 20 Feb 2025 10:56:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077768; x=1740682568; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=DDGE3L4+QtgAAh6Ko802g9HGK0Xw8N2Rxx0aARspUp4=; b=c5twu1Ee+b62oEfz88Cq4R02kV0MES30ivKaXpVXLT3NAdG1xyEMmWFAWiiMEmO5WL jVlLE0KOMTbDgb3brE2VvDmzDP1CtXDijhn64CuzYCbRFBIlDl7Jwcf7+NpHAJ6gPiDj w7a85qfRjusiAP5jKkBjUc/9FL2eLSInR+UdU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077768; x=1740682568; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DDGE3L4+QtgAAh6Ko802g9HGK0Xw8N2Rxx0aARspUp4=; b=TjDXtrxrfMgNl5Yz3Nf/1Zk2K6kuAQWnc+thYDfSDHhYUpxk3zKEaNL7hX/RT+A7dc DDLlKzL6HlPfcFrXTQuBap7vCuMIM11gjTm80xBGfnOHqmVQwFDB3qZmrMOZQI74VKyx T+Y11tAi93bkcrnk4AKSySzzYEqZ/C0QdAr/24Tjreuj4aobZtJzyPFkXloBtwASDr0s 8hFOgXiqyH+elJixsIWA4NLsL9qsNqmWaugQrXo8wFPKu0xL/jLEb0z/kcpAz7p4nPzy +QU8g/wGoSx6FAXGPwCHm5a4MOGa6C0QJj47bLFWAUiB/VXxyR2HLpt32IYHWiOjTWgu pfjw== X-Forwarded-Encrypted: i=1; AJvYcCUuq1XWRrY+LUQD5g8IYQcywbRQrkHrAdVppwTeP4hVaRTR9+0P1IXbDcx3mqDFEcDovSpL5RM=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5MxxaFphSkvWWRXlCDIXSqbPffybJx7ileNGpA9Z7TE/EcTDo FPmHiuqVZ7MWZXURHnfNV69IgeYnB7NEkY2BTVD4RGA+l4ybZNBl6ajwbmcr1w== X-Gm-Gg: ASbGncvHGwCVUzWvPLsRYS+pAwYUXbqnII5247uGliSPMgp8DpvBo9hj5zqZoCWUOjq M1cO4Io/HczPBvWCvkLipKTAi/0tVJZ3gwjlmAKv9Cgt7hXCI2o+qTA8Ma9ycYC+5hVE1Kk9e3F f/5URJ9Z0WbNo2UpWJaIee2z9b3I7FDLLMoXlSd6nAtDtcl+NNFFJUwgonfAR7Jf2Ssq6ja4kgU 5kNNiAOsnm8yADPtqOTIfsorb7FxOzMf9me5ZV8lMZF2tm235c2P4XW+KT9mD8fbatrvLNKGOfw 2qmxHPIlDqzgo8U7LVEvy4+ujSQdybTJgwmKlC7NDZI/cDA/vNY9kMYlWjqcYsRl/peaGmc= X-Google-Smtp-Source: AGHT+IGk3I5xMJOQybBNubZesmP0ZDBZjg0y/axwJ+QSrgEcCl4h1AEI/eEbdqV5zZZFRZPg+G0BBQ== X-Received: by 2002:a05:6a21:6d88:b0:1e1:a789:1b4d with SMTP id adf61e73a8af0-1eef3d9f0cfmr321809637.15.1740077768126; Thu, 20 Feb 2025 10:56:08 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.04 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:07 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next 5/9] bnxt_en: Introduce ULP coredump callbacks Date: Thu, 20 Feb 2025 10:34:52 -0800 Message-Id: <1740076496-14227-6-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: X-Patchwork-Delegate: kuba@kernel.org From: Michael Chan Add .ulp_get_dump_info() and .ulp_get_dump_data() callbacks to struct bnxt_ulp_ops. When ethtool -w is invoked to get the coredump, these 2 callbacks to the bnxt_re auxbus driver will be called if they are populated by the bnxt_re driver. The first callback gets the number of coredump segments and the size of each coredump segment. The 2nd callback copies the coredump data for each segment. Signed-off-by: Michael Chan Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Signed-off-by: Selvin Xavier Acked-by: Jakub Kicinski --- drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c | 12 ++++- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 57 ++++++++++++++++++++++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 22 +++++++++ 3 files changed, 89 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c index 7236d8e..2106d0d 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_coredump.c @@ -14,6 +14,7 @@ #include "bnxt.h" #include "bnxt_hwrm.h" #include "bnxt_coredump.h" +#include "bnxt_ulp.h" static const u16 bnxt_bstore_to_seg_id[] = { [BNXT_CTX_QP] = BNXT_CTX_MEM_SEG_QP, @@ -414,13 +415,20 @@ static int __bnxt_get_coredump(struct bnxt *bp, u16 dump_type, void *buf, } if (dump_type == BNXT_DUMP_DRIVER) { - u32 drv_len, segs = 0; + u32 drv_len, drv_segs, segs = 0; + void *drv_buf = NULL; drv_len = bnxt_get_ctx_coredump(bp, buf, offset, &segs); + drv_segs = segs; + segs = 0; + if (buf) + drv_buf = buf + offset + drv_len; + drv_len += bnxt_get_ulp_dump(bp, dump_type, drv_buf, &segs); + drv_segs += segs; *dump_len += drv_len; offset += drv_len; if (buf) - coredump.total_segs += segs; + coredump.total_segs += drv_segs; goto err; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index e4a7f37..6f640b2 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -25,6 +25,7 @@ #include "bnxt.h" #include "bnxt_hwrm.h" #include "bnxt_ulp.h" +#include "bnxt_coredump.h" static DEFINE_IDA(bnxt_aux_dev_ids); @@ -393,6 +394,62 @@ void bnxt_register_async_events(struct bnxt_en_dev *edev, } EXPORT_SYMBOL(bnxt_register_async_events); +static void bnxt_ulp_fill_dump_hdr(struct bnxt *bp, void *buf, u32 seg_id, + u32 seg_len) +{ + struct bnxt_coredump_segment_hdr seg_hdr; + + bnxt_fill_coredump_seg_hdr(bp, &seg_hdr, NULL, seg_len, 0, 0, 0, + BNXT_DRV_COMP_ID, seg_id); + memcpy(buf, &seg_hdr, sizeof(seg_hdr)); +} + +u32 bnxt_get_ulp_dump(struct bnxt *bp, u32 dump_flag, void *buf, u32 *segs) +{ + struct bnxt_en_dev *edev = bp->edev; + struct bnxt_ulp_dump *dump; + struct bnxt_ulp_ops *ops; + struct bnxt_ulp *ulp; + u32 i, dump_len = 0; + + *segs = 0; + if (!edev || !bnxt_ulp_registered(edev)) + return 0; + + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_get_dump_info || !ops->ulp_get_dump_data) + return 0; + + dump = &ulp->ulp_dump; + if (!buf) { + memset(dump, 0, sizeof(*dump)); + ops->ulp_get_dump_info(ulp->handle, dump_flag, dump); + if (dump->segs > BNXT_ULP_MAX_DUMP_SEGS) + return 0; + for (i = 0; i < dump->segs; i++) { + dump_len += dump->seg_tbl[i].seg_len; + dump_len += BNXT_SEG_HDR_LEN; + } + } else { + for (i = 0; i < dump->segs; i++) { + struct bnxt_ulp_dump_tbl *tbl = &dump->seg_tbl[i]; + u32 seg_len = tbl->seg_len; + u32 seg_id = tbl->seg_id; + + bnxt_ulp_fill_dump_hdr(bp, buf, seg_id, seg_len); + buf += BNXT_SEG_HDR_LEN; + dump_len += BNXT_SEG_HDR_LEN; + ops->ulp_get_dump_data(ulp->handle, seg_id, buf, + seg_len); + buf += seg_len; + dump_len += seg_len; + } + } + *segs = dump->segs; + return dump_len; +} + void bnxt_rdma_aux_device_uninit(struct bnxt *bp) { struct bnxt_aux_priv *aux_priv; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 7fa3b8d..9298589 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -29,11 +29,31 @@ struct bnxt_msix_entry { u32 db_offset; }; +#define BNXT_ULP_MAX_DUMP_SEGS 8 + +/** + * struct bnxt_ulp_dump - bnxt ULP aux device coredump info + * @segs: number of coredump segments with info in the seg_tbl + * @seg_tbl: coredump segment table + * @seg_tbl.seg_id: coredump segment ID + * @seg_tbl.seg_len: coredump segment len + */ +struct bnxt_ulp_dump { + u32 segs; + struct bnxt_ulp_dump_tbl { + u32 seg_id; + u32 seg_len; + } seg_tbl[BNXT_ULP_MAX_DUMP_SEGS]; +}; + struct bnxt_ulp_ops { /* async_notifier() cannot sleep (in BH context) */ void (*ulp_async_notifier)(void *, struct hwrm_async_event_cmpl *); void (*ulp_irq_stop)(void *, bool); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); + void (*ulp_get_dump_info)(void *handle, u32 dump_flags, + struct bnxt_ulp_dump *dump); + void (*ulp_get_dump_data)(void *handle, u32 seg_id, void *buf, u32 len); }; struct bnxt_fw_msg { @@ -51,6 +71,7 @@ struct bnxt_ulp { u16 max_async_event_id; u16 msix_requested; atomic_t ref_count; + struct bnxt_ulp_dump ulp_dump; }; struct bnxt_en_dev { @@ -119,6 +140,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err); void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs); void bnxt_ulp_irq_stop(struct bnxt *bp); void bnxt_ulp_irq_restart(struct bnxt *bp, int err); +u32 bnxt_get_ulp_dump(struct bnxt *bp, u32 dump_flag, void *buf, u32 *segs); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_rdma_aux_device_uninit(struct bnxt *bp); void bnxt_rdma_aux_device_del(struct bnxt *bp); From patchwork Thu Feb 20 18:34:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984406 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B892A2638B6 for ; Thu, 20 Feb 2025 18:56:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077774; cv=none; b=MOAg4usnKohxZqEt9+q2C1x2D7ZSwtDCPVT4AZ1npdpyg2eNqxBbLRMTnZbx5DshqNgxReinFVwtj/ogDnK738VqnUdM6PKTyd9Xk5YRUAoEhreZ74BkCvQ/whs9fwKkm1lK2bUpVqOVFts2eZzTHL7ZmHn6YV38akXpjKmu+GU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077774; c=relaxed/simple; bh=TmVHhw+YyfHJXm1Tk11ATr1f5Ybq7UumTRqLxFLOUkE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=jzGgz3QRqgqP83Udyb5C56r8hW3mBLcNOvAqYxren5YTqkWVroJLRp1ZnC/m8/6V6uBHWQBN+5qAFCPYDCJa+d7k7nep27t2s/BcIS6QwQ8F4+CA0Ec4ELs5JjCFqBQU5dmUcpyMwmDfQHjj/Kh0cTdB4rIzURl2hO//6sBhxPc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=A4P0ySsn; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="A4P0ySsn" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-220dc3831e3so37839195ad.0 for ; Thu, 20 Feb 2025 10:56:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077772; x=1740682572; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=wbEO2ZvYX8S6gkbKt5XzFSL0VmDi/kPQc2XpzR/hKDw=; b=A4P0ySsnPzFxVRD6vE8EOCLWwoe7fKSSyqjuK2KBBM8fHFuJlvZvkCwkT4sLlZhhxW EU2ZjwDDw5SwLnwyhj3fF/AnYVUYH7b6h8NIlzUIGdQ/3h+j8vcNnFXA4+aruqTH1vyU LpU52g45bPbV330Je7L38TcenQp0CKnUMKZ54= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077772; x=1740682572; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wbEO2ZvYX8S6gkbKt5XzFSL0VmDi/kPQc2XpzR/hKDw=; b=Ac7Pj3KLEf1D+nsTO3OlzlDmdLCZIHCo7U+ozZgf4PnuVwPWQVfqud2oCwGotCizVs EmcQmO6RwjABEnODTRDXa9JCXY9HyHgwsYLG9M9OpHMeYx8pa9dfvTUk0VS8y2D5gyJL /tL5P8z96bOZDMn9777ufH+IAG1HhaJ2KXhBMBYIjfuGVdw21gBoKJevT47pfHDzcP5y 5oLFfPXVveoQfxad2ch/UyoMeNPuetky6R+TMroNL8Yvl3jrcWYB9L1Co86cu38guzWh NeeduLFgBdYwe7KloaTmwrErPzawsaTUwroxLwbaJwReWcYGgDXLsy1xgBpBxVyDE90J nzuw== X-Forwarded-Encrypted: i=1; AJvYcCUWD3I0BynMfDP17NwFIHZMwBbRxqzlaCaO9Z+RtQWYLTfRklScI2R5X5UpDpgfMauKu6EgSHs=@vger.kernel.org X-Gm-Message-State: AOJu0YwUIkyxjkK7bSBKg9VPdx+YmldP0UifrvGb5HOJ5Chuwg7XgT3C gALeeQNvYVKS+zj39nKE4oT63vPh3ydXVKXgcJANxGXz8+Kf6jV0luE7ZoR4BA== X-Gm-Gg: ASbGnctucvQ9rN22pNbpfuBra4w+A7D2yqxnuZr8AyJtAU2kKcHLiN+UU0xj4P83IKY uqRi4nv4b+OT9c8FMZd+jRJtawLyUs2V2aAkxTMt7tgd+YCU5Z2RrELMfB/1CCLNmcBfZyS+mhP l3IYD92mo/bhqrehEGh2Dwq2OHcpaCx4J6h9waya/0KdMoiHRRnpaUgI9XI0uoYbWoUCpQx/cbk +5Q2EVPupPqe4faNPHeTAW1YD5TOuxQE2cgOdyGz205FVHCoBXYIN3XhzzBjnGxdKTkwfrgdGjQ cJEMmiyPAkylARYdOKQXRzBxJtgz8fc4m3PJBRUfGid5iBoL4kJLirF34uq3Azxi/eoMVDc= X-Google-Smtp-Source: AGHT+IHvxObTMqiJriOaCFcBkp4iNsiG0ly710DfyEz61MF6mHwZNyTEkxPtzIfoqhMu9KW7oIb1mA== X-Received: by 2002:a05:6a20:7f9c:b0:1ee:a410:4aa5 with SMTP id adf61e73a8af0-1eef3dd0e8bmr314966637.17.1740077771979; Thu, 20 Feb 2025 10:56:11 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:11 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next 6/9] RDMA/bnxt_re: Support the dump infrastructure Date: Thu, 20 Feb 2025 10:34:53 -0800 Message-Id: <1740076496-14227-7-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Michael Chan Adds the stubs for the debug dump in bnxt_re driver. Each segment of the dump data holds a particalar resource type. get_dump_info returns the number of segments and the size of each segments. Driver currently supports segments for QP/CQ/SRQ/MR and for the generic host logs. While taking the dump, get_dump_data will be invoked for each of the segments and bnxt_re driver returns the data associated with each segment. Signed-off-by: Michael Chan Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 85 +++++++++++++++++++++++++++++++++++- 1 file changed, 84 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 6b5a169..76dd0fa 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -497,10 +497,93 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } } +#define BNXT_SEGMENT_ROCE 255 +#define BNXT_SEGMENT_QP_CTX 256 +#define BNXT_SEGMENT_SRQ_CTX 257 +#define BNXT_SEGMENT_CQ_CTX 258 +#define BNXT_SEGMENT_MR_CTX 270 + +static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf, + u32 buf_len) +{ +} + +/* bnxt_re_snapdump - Collect RoCE debug data for coredump. + * @rdev - rdma device instance + * @buf - Pointer to dump buffer + * @buf_len - Buffer length + * + * This function will dump RoCE debug data to the coredump. + * + * Returns: Nothing + * + */ +static void bnxt_re_snapdump(struct bnxt_re_dev *rdev, void *buf, u32 buf_len) +{ +} + +#define BNXT_RE_TRACE_DUMP_SIZE 0x2000000 + +/* bnxt_re_get_dump_info - ULP callback from L2 driver to collect dump info + * @handle - en_dev information. L2 and RoCE device information + * @dump_flags - ethtool dump flags + * @dump - ulp structure containing all coredump segment info + * + * This function is the callback from the L2 driver to provide the list of + * dump segments for the ethtool coredump. + * + * Returns: Nothing + * + */ +static void bnxt_re_get_dump_info(void *handle, u32 dump_flags, + struct bnxt_ulp_dump *dump) +{ + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle); + struct bnxt_ulp_dump_tbl *tbl = dump->seg_tbl; + struct bnxt_re_dev *rdev = en_info->rdev; + + dump->segs = 5; + tbl[0].seg_id = BNXT_SEGMENT_QP_CTX; + tbl[0].seg_len = rdev->rcfw.qp_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES; + tbl[1].seg_id = BNXT_SEGMENT_CQ_CTX; + tbl[1].seg_len = rdev->rcfw.cq_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES; + tbl[2].seg_id = BNXT_SEGMENT_MR_CTX; + tbl[2].seg_len = rdev->rcfw.mrw_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES; + tbl[3].seg_id = BNXT_SEGMENT_SRQ_CTX; + tbl[3].seg_len = rdev->rcfw.srq_ctxm_size * BNXT_RE_MAX_QDUMP_ENTRIES; + tbl[4].seg_id = BNXT_SEGMENT_ROCE; + tbl[4].seg_len = BNXT_RE_TRACE_DUMP_SIZE; +} + +/* bnxt_re_get_dump_data - ULP callback from L2 driver to collect dump data + * @handle - en_dev information. L2 and RoCE device information + * @seg_id - segment ID of the dump + * @buf - dump buffer pointer + * @len - length of the buffer + * + * This function is the callback from the L2 driver to fill the buffer with + * coredump data for each segment. + * + * Returns: Nothing + * + */ +static void bnxt_re_get_dump_data(void *handle, u32 seg_id, void *buf, u32 len) +{ + struct bnxt_re_en_dev_info *en_info = auxiliary_get_drvdata(handle); + struct bnxt_re_dev *rdev = en_info->rdev; + + if (seg_id == BNXT_SEGMENT_ROCE) + return bnxt_re_snapdump(rdev, buf, len); + + bnxt_re_dump_ctx(rdev, seg_id, buf, len); +} + static struct bnxt_ulp_ops bnxt_re_ulp_ops = { .ulp_async_notifier = bnxt_re_async_notifier, .ulp_irq_stop = bnxt_re_stop_irq, - .ulp_irq_restart = bnxt_re_start_irq + .ulp_irq_restart = bnxt_re_start_irq, + .ulp_get_dump_info = bnxt_re_get_dump_info, + .ulp_get_dump_data = bnxt_re_get_dump_data, }; /* RoCE -> Net driver */ From patchwork Thu Feb 20 18:34:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984407 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C03752641C9 for ; Thu, 20 Feb 2025 18:56:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077778; cv=none; b=KBJWKsXpawrkbT+dbyZy0C5JFq1z9qqnDQbUW6jutvFfK3QEdI51raVtMG0NRpzOV0AQSyLrJf0dLQbP6faTIhHrLijZpZvRuCQi81aFKGB/vtWLSn/7l2kRTQ9fYzFYQTOXWvQ8gBnAoFih+1ooJFozb9qm6Mq6P3dRlOAbUPw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077778; c=relaxed/simple; bh=qQmOt74s8s7ggF+1kRfXXp1O+t46qBBm4OZfmbYtmr0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=HASXP0GgFoTkyKW+3l4OFJaHfebYP3Dt18EeZZ/HBN5hxb1Xp34A7ZqQwLco1HXEg07L+TmEI/WXJT8kKkzUr/C4DvTSGHJAHtwdBSdX4iUcAO82gUP0SBp5yBMxA/OyP1OIS/ciwN0pNHon4hup2QA13oReBHinYnPLFKmTOWU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=g8WdLfBS; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="g8WdLfBS" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-219f8263ae0so28600455ad.0 for ; Thu, 20 Feb 2025 10:56:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077776; x=1740682576; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=62gcd1c9eE38mVAGq05LCRcGvvb8fU0AO/xGYZf2u2I=; b=g8WdLfBSnGkXhRa46Gf+ocEWvUmmbxaAwBXO94k4mjRdWnGkklA70e7cTXCqco8ZNq LHveT4YHnR9e3wj6sbBsw8+2mQgfnqNvP8JNC8CLc8XJGw3qh+hJp2t3dQIIef6/tNHy SuQueltiNBy5U6Yf8YaYc6dpsi1fGWlhjdQPs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077776; x=1740682576; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=62gcd1c9eE38mVAGq05LCRcGvvb8fU0AO/xGYZf2u2I=; b=qn2SBb/unGjn5l8PN8cgepCLfhfccOG9TuNtP+X1cwDITXWp5thDUEzUAHOdlozwPL xJc1awvuvf8VlmTqJCH3av02JEkqXfswBHa5PkqsO7nM5frB04R3Zl64DXTgQ1KE7iho LkbIkk99r6RS4LCn+8WqVBlFQBbYKEuDsK4F5DNLRzjG6u01N7BojAc9hziNS1MFEa3/ PN2vSRwX0G17P4RkqcWVzLpkB+0tpLsRBcaI9dcihUAF3d09H1ZiII/twBdU4CQgKyB2 aiauH6gaK9xjYMuYd7yDMTeQiRbCNJrluDwnJ1n466OFdfbyH+J0AcY7w1NKEAFj/hkD Zbfg== X-Forwarded-Encrypted: i=1; AJvYcCUWPuhgh11QiE8pnU45Omw+JwqE1q+6crPRIAKC9HfKIb5sTg1HleDPDg5pFj9Ta4djfIoo6Ls=@vger.kernel.org X-Gm-Message-State: AOJu0YwIwWRdSEwJDCCw8Znkh12PG6S1e4dw1y++NoCcbJdx29QNxlUA N2RUgIbhh59ibu2a6tof68cVo9wcP+kG4QE8RmmFlehJXEwMVqZGzu7nehr0OQ== X-Gm-Gg: ASbGncuA/3hJaKuikhQ0R8b9tWmkPXZWivrCrsTunB0DBM0sSjGorfQVuq8QqM+Tx5a x1LCT5ZWoWTsWtvAOWkPIfZ7rRcumX9KHqoI48/L7GmTeROgbtiWEKKqMKukCfVHguPfLjRpMn3 LuTFRURHG7yaMEc+B1nyJ9dJJupB2R3xdRlN/C+dvtmbwooO6u2N/59NTIUHuzzt5DiXDGHSA// 3fU1pWVXBfecopBm6EgPhDQHZAqLX/mRz+x3TVa3rEHTBxRypKvsvrjH734Z+F8ANCtWIwPj0pM 6HBHyBJi2HtoUl6aJ5gSJhOXe3DEAecXcyJSAURgsIIibPbqNNNH/tuDTy0RQRYe/pR1r8c= X-Google-Smtp-Source: AGHT+IEqTR1Tj8xQJb09Um5L3+18Y3HeubsWUM4BCPraVepPaC1Lc0gQGeDiUUcGWKdFr4E2tzR1LQ== X-Received: by 2002:a05:6a21:6b0f:b0:1ee:e24d:8fd2 with SMTP id adf61e73a8af0-1eef3cbdfbdmr519522637.25.1740077776052; Thu, 20 Feb 2025 10:56:16 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:15 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Saravanan Vajravel , Selvin Xavier Subject: [PATCH rdma-next 7/9] RDMA/bnxt_re: Dump the debug information in snapdump Date: Thu, 20 Feb 2025 10:34:54 -0800 Message-Id: <1740076496-14227-8-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Saravanan Vajravel Dump the cached debug information when the L2 driver invokes the get_dump_data hook. Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/main.c | 109 +++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 76dd0fa..afde0ef 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -508,6 +508,106 @@ static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf, { } +/* + * bnxt_re_snapdump_qp - Log QP dump to the coredump buffer + * @rdev - Pointer to RoCE device instance + * @buf - Pointer to dump buffer + * @buf_len - Buffer length + * + * This function will invoke ULP logger to capture snapshot of + * SQ/RQ/SCQ/RCQ of a QP in ASCII format. + * + * Returns: Buffer length + */ +static u32 bnxt_re_snapdump_qp(struct bnxt_re_dev *rdev, void *buf, u32 buf_len) +{ + u32 index, len = 0, count = 0; + struct qdump_qpinfo *qpinfo; + struct qdump_array *qdump; + + if (!rdev->qdump_head.qdump) + return 0; + + mutex_lock(&rdev->qdump_head.lock); + index = rdev->qdump_head.index; + while (count < rdev->qdump_head.max_elements) { + count++; + index = (!index) ? (rdev->qdump_head.max_elements - 1) : (index - 1); + qdump = &rdev->qdump_head.qdump[index]; + if (!qdump->valid || qdump->is_mr) + continue; + + qpinfo = &qdump->qpinfo; + len += snprintf(buf + len, buf_len - len, + "qp_handle 0x%.8x 0x%.8x (xid %d) dest_qp %d type %s state %s ", + (u32)(qpinfo->qp_handle >> 32), + (u32)(qpinfo->qp_handle & 0xFFFFFFFF), + qpinfo->id, qpinfo->dest_qpid, + __to_qp_type_str(qpinfo->type), + __to_qp_state_str(qpinfo->state)); + len += snprintf(buf + len, buf_len - len, + "is_usr %d scq_handle 0x%llx rcq_handle 0x%llx", + qpinfo->is_user, qpinfo->scq_handle, qpinfo->rcq_handle); + len += snprintf(buf + len, buf_len - len, + "scq_id %d rcq_id %d\n", + qpinfo->scq_id, qpinfo->rcq_id); + if (len >= buf_len) + goto dump_full; + + if (len >= buf_len) + goto dump_full; + } + mutex_unlock(&rdev->qdump_head.lock); + return len; + +dump_full: + mutex_unlock(&rdev->qdump_head.lock); + return buf_len; +} + +/* + * bnxt_re_snapdump_mr - Log PBL of MR to the coredump buffer + * @rdev - Pointer to RoCE device instance + * @buf - Pointer to dump buffer + * @buf_len - Buffer length + * + * This function will invoke ULP logger to capture PBL list of + * Memory Region in ASCII format. + * + * Returns: Buffer length + */ +static u32 bnxt_re_snapdump_mr(struct bnxt_re_dev *rdev, void *buf, u32 buf_len) +{ + u32 index, count = 0, len = 0; + struct qdump_array *qdump; + + if (!rdev->qdump_head.qdump) + return 0; + + mutex_lock(&rdev->qdump_head.lock); + index = rdev->qdump_head.index; + while (count < rdev->qdump_head.max_elements) { + count++; + index = (!index) ? (rdev->qdump_head.max_elements - 1) : (index - 1); + qdump = &rdev->qdump_head.qdump[index]; + if (!qdump->valid || !qdump->is_mr) + continue; + + len += snprintf(buf + len, buf_len - len, + "**MemoryRegion: type %d lkey 0x%x rkey 0x%x tot_sz %llu **\n", + qdump->mrinfo.type, qdump->mrinfo.lkey, + qdump->mrinfo.rkey, qdump->mrinfo.total_size); + if (len >= buf_len) + goto dump_full; + } + mutex_unlock(&rdev->qdump_head.lock); + return len; + +dump_full: + mutex_unlock(&rdev->qdump_head.lock); + return buf_len; +} + /* bnxt_re_snapdump - Collect RoCE debug data for coredump. * @rdev - rdma device instance * @buf - Pointer to dump buffer @@ -520,6 +620,15 @@ static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf, */ static void bnxt_re_snapdump(struct bnxt_re_dev *rdev, void *buf, u32 buf_len) { + u32 len = 0; + + len += bnxt_re_snapdump_qp(rdev, buf + len, buf_len - len); + if (len >= buf_len) + return; + + len += bnxt_re_snapdump_mr(rdev, buf + len, buf_len - len); + if (len >= buf_len) + return; } #define BNXT_RE_TRACE_DUMP_SIZE 0x2000000 From patchwork Thu Feb 20 18:34:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984408 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE678266585 for ; Thu, 20 Feb 2025 18:56:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077782; cv=none; b=OMzh5wX6QQOxXQn/2TKVKALIhQ8waqgZ1Bpf2xjZhzkXu7BzUGK9eSUQ24ase3XzNA4KqLBlERVzaw0TrGmp05MoPAwe3u+r6CZXZ4mhRQ9iElqBnSABmbJGa2JV6pBi47M7LpctA/DHJjPNY8SVZh5idV5jXz1/hgZrDJmgPQw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077782; c=relaxed/simple; bh=ps9j0Wix5FxOg6J4lOpwhIo0pwUPZtapGIIm3Ker8C8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=XBMASTO+9IiRWbYeVkfbvLwb1toZq7PfZqn2Jw+AGSYAPuR+T9T/9yLvidpZRZHvwblLZyvi+q2vF2xcAxBxoZt1zCUZA2/jXWqaWEoBdTAdxdDbraZrnkIvm3kVUsbHNszX88xyX+0xr5NxEmmrZbMdmvkP+JODJ83BC+jc3E8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=bUGHchtp; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="bUGHchtp" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-220c4159f87so18527155ad.0 for ; Thu, 20 Feb 2025 10:56:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077780; x=1740682580; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=tif6h+vIQ6/biBxnmUJxR84piPWX45JJ651sYhwJ4MA=; b=bUGHchtplBXLKm8Rk3lJo84LP25Yj4pfjWc7sMUIBtgO8w7ZhoV0/di4HrhBAt6qbD LwE4rcicn4EQnf/tndGKWC3h357DwLjSRyrKKz0LZ/r4XgxrD28jzDLtySI1YL8+/9ha yWqmbC9q7KkVOCIu4wdA91gsxB7gjZsb6Re4w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077780; x=1740682580; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tif6h+vIQ6/biBxnmUJxR84piPWX45JJ651sYhwJ4MA=; b=f6KT9PT++U2z9xboCGn6vF1DtWerX505Eo+zpdWZ+qc3B4C8CRNju9MO5AFnTtG3iJ qpSOXrN8FM3t4A8TDqWndfQnhHOwq+byGGvsKdww/IpBZrSp9PTQ91Mihk7x8LReK3Qu +dRipChwzrCfpqGIP7Bns3L9zkFk0UAU1Nr4619fZzTNHj7x8bn+E55Jsjt1LbaZmNXB NNNg+uN8G+PanCTNXBHVdH2+wSh+bbrXzgUz6X3k8JKle5i+Ee0ROfZAsOe/J3+tynS/ pQPnzgKu6FldON0xt1VCMe7KxTgB91hPXKktWhMtCiYqrNoYBg7aSpdt/ZydIYKSpWR+ Tpug== X-Forwarded-Encrypted: i=1; AJvYcCXM97yY5hPMZN/3bJUshSjrYrjwtGo8MmUVoOdy8fOJEFG6ODngUInUXRaB+tdoi4H/T6aGvMQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwO5FHFfGdR5Wop8OdW3LwDauIgeLnCE6/0o9inodWaM7u6mGtB Rq5uaqPFrOV2IpL9e8UdVibk8NyxmNhyT/kH+qIw2IizbW6NUdK6JijuLHHyQw== X-Gm-Gg: ASbGncuyceABtz9QHh9qNH4mkVbfuCsfHORAmHvEZp2DknNCyMcMjnRfXCzrO9fNN0R GTRNgswRip3GgmTqwUoDu05G4K41QOAJzEqhnxRiKLtL/wBh6J7q1xO7UHuUPSZ5LUZmEg8rKF3 KpSpYWgy0f54l695pzbKTdei15nq5aJx+1ljPk8ukb2CTOq7tgsJkdzSWI/UNKXFMFmxM0rI/fq qtFu77weY/74Or31q59iK9sNe94a4hZZj5aO7DzLotw8m67KtVZKfPPZHjkFg5SGeJ2qhHP2KEB Yy5qjjRreZC7QpjStD5PLrnKRwsgK4Sr545zQRD8G90/EKC43ovR6uItSJX4Ymn1jz+6gOo= X-Google-Smtp-Source: AGHT+IFw4WLUXvlZeVZ7DNjlcwoAVhBgMp/91u56hwsVYXz4XkQ5BCYpHuWdsyyiEkUv1s+GRqSPCw== X-Received: by 2002:a05:6a21:6b0f:b0:1ee:c093:e23c with SMTP id adf61e73a8af0-1eef3db70e6mr504945637.41.1740077780108; Thu, 20 Feb 2025 10:56:20 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:19 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Kashyap Desai , Selvin Xavier Subject: [PATCH rdma-next 8/9] RDMA/bnxt_re: Dump the HW context information Date: Thu, 20 Feb 2025 10:34:55 -0800 Message-Id: <1740076496-14227-9-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Kashyap Desai Dump the cached HW context information for QP/CQ/SRQ/MRs when the L2 driver queries using get_dump_data Signed-off-by: Kashyap Desai Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 10 ++++----- drivers/infiniband/hw/bnxt_re/main.c | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index a1ee6ca..dca435e 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1079,7 +1079,7 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) struct bnxt_qplib_nq *rcq_nq = NULL; unsigned int flags; void *ctx_sb_data = NULL; - bool do_snapdump; + bool do_snapdump = false; u16 ctx_size; int rc; @@ -1087,17 +1087,15 @@ int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata) bnxt_re_debug_rem_qpinfo(rdev, qp); bnxt_qplib_flush_cqn_wq(&qp->qplib_qp); - ctx_size = qplib_qp->ctx_size_sb; + ctx_size = rdev->rcfw.qp_ctxm_size; if (ctx_size) ctx_sb_data = vzalloc(ctx_size); - do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags); rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp, ctx_size, ctx_sb_data); if (rc) ibdev_err(&rdev->ibdev, "Failed to destroy HW QP"); - else - bnxt_re_save_resource_context(rdev, ctx_sb_data, - BNXT_RE_RES_TYPE_QP, do_snapdump); + do_snapdump = test_bit(QP_FLAGS_CAPTURE_SNAPDUMP, &qplib_qp->flags); + bnxt_re_save_resource_context(rdev, ctx_sb_data, BNXT_RE_RES_TYPE_QP, do_snapdump); vfree(ctx_sb_data); if (rdma_is_kernel_res(&qp->ib_qp.res)) { diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index afde0ef..b2bf0d0 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -506,6 +506,44 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) static void bnxt_re_dump_ctx(struct bnxt_re_dev *rdev, u32 seg_id, void *buf, u32 buf_len) { + int ctx_index, i; + void *ctx_data; + u16 ctx_size; + + switch (seg_id) { + case BNXT_SEGMENT_QP_CTX: + ctx_data = rdev->rcfw.qp_ctxm_data; + ctx_size = rdev->rcfw.qp_ctxm_size; + ctx_index = rdev->rcfw.qp_ctxm_data_index; + break; + case BNXT_SEGMENT_CQ_CTX: + ctx_data = rdev->rcfw.cq_ctxm_data; + ctx_size = rdev->rcfw.cq_ctxm_size; + ctx_index = rdev->rcfw.cq_ctxm_data_index; + break; + case BNXT_SEGMENT_MR_CTX: + ctx_data = rdev->rcfw.mrw_ctxm_data; + ctx_size = rdev->rcfw.mrw_ctxm_size; + ctx_index = rdev->rcfw.mrw_ctxm_data_index; + break; + case BNXT_SEGMENT_SRQ_CTX: + ctx_data = rdev->rcfw.srq_ctxm_data; + ctx_size = rdev->rcfw.srq_ctxm_size; + ctx_index = rdev->rcfw.srq_ctxm_data_index; + break; + default: + return; + } + + if (!ctx_data || (ctx_size * BNXT_RE_MAX_QDUMP_ENTRIES) > buf_len) + return; + + for (i = ctx_index; i < BNXT_RE_MAX_QDUMP_ENTRIES + ctx_index; i++) { + memcpy(buf, ctx_data + ((i % BNXT_RE_MAX_QDUMP_ENTRIES) * ctx_size), + ctx_size); + buf += ctx_size; + } + } /* From patchwork Thu Feb 20 18:34:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13984409 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE1F5266585 for ; Thu, 20 Feb 2025 18:56:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077786; cv=none; b=Se8qvpeg7x1c7Is/SBqZHvkvVZIKC2+P+wJxF2cb/L19Q699HpwCAoY8FdDuYOA8qiOur1vBQr09+XPO3FUk5z0oZ1KSaDlyE9UbPjYr4e2TwDj1NhtQ0hLgewDw4UBQuFCCvYc9l4Syjtn3J+3yxdohsu3VxHGVnTY8dayXZ0c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740077786; c=relaxed/simple; bh=4pokzCpm/lUAfMKuosqyIEkYPbh7I36XpLT3BcQvRag=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=CNgwsHciAhkUEfSm8hldWIIgQX/He+4F1m1g/AHwla0Kw1RYNZx7tFeIOYrAheuU8b7IAzVB9sOGmrvn21VpOkrCudTQxJtPjh+HTSABtfyhjaOYzql6NTbTo7ttKxGnPYVmJ7n9oc8bFbz/ROITDHKvb0tRs304fWGCAUj9r3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=EmQo9PuX; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="EmQo9PuX" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-221206dbd7eso25880895ad.2 for ; Thu, 20 Feb 2025 10:56:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1740077784; x=1740682584; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=WE82c+aiEEEUeKpSOXCYU/WI+Y4+5eTNi2kEnvfIaks=; b=EmQo9PuXIak/V3eZp/81+nrMNRvZYjhLMhXsQnRdGJMr2k0PpMfD0wlzOk5KW0I8lV jqemdsPa3M2F83mpCZ2EkCyQKaOpiQwuyFkqqIe5tQcAI2I1lD9cu/L7IA/HFmu7ydBo TxuWo5khV6fsvbP4K3USLEvTYlsNmmSX4wg0Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740077784; x=1740682584; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WE82c+aiEEEUeKpSOXCYU/WI+Y4+5eTNi2kEnvfIaks=; b=Mu++OV6naBe0nHnIIHO5mNExKwJHubDbRXXy6bumSktNuNYT+HkDOYs9PJn+n7CKT3 nwOQ2FKdELMJzw36rs/YSXYRmnDhtiY3a0vAE3LVElcMojSgND+aaGljgK6wURGT3wln 2Be2Q+Vwn8K8U3lMI1c6qLrUr7ADUFe+Q02rFLJrHHqJo3/74+tqZy55d24WViBnOLVA CboczcSXJnmHtvdVVZfTJ50x97XTRdQZSyU5CcO8JMDNCDGDdUOPbICHnC/xp5qGKiB8 WBDl7uYaq1ZSORNsN4Ix+kEXDmP+/S0FjJTWUxKgJEpHMheKsdD5jtXKT4gCivqDNtDf 3uOQ== X-Forwarded-Encrypted: i=1; AJvYcCV3yxQq9Uz+DBgactzn4kt9HXCVxJTK3Uc+QLU8+dYiZ10JzfVtiwnnymK72SGjxe0+pm4gimc=@vger.kernel.org X-Gm-Message-State: AOJu0YyuWkS+oDiF9QUFbmwGW+Uxg8ZDNL8AwsQOxO136mgycB7WDqC0 uhqMybvKVKs416kPoFs62f5vDJ16mOgLHKd4U19HC1f/0yYtH8U24zHPhogRIw== X-Gm-Gg: ASbGncvi8rznRvfispFx4mWRYF2bGkalT62dwmYb3sbcfxq93G2K2k1HcIgvIPQkorg Y2Hv/HOiS77JakRoxI3HiKVMLlbEpy1jHXpG6ZQh1w+3iW6ONNqwohXCvsXEP4bUJmi7cmbFoD/ 4vUaRmNh4/3fm0dKHPxudOqGn4YB33g1yccZnBVzbta+sEQkx8S4lMnAOD3O/yCqv9vXxX6QlSx AUkz5nSw/J6dkRH7/wyfyxTszh9892kxBmF/aTUJCFY0OCithXz0LR2DaAhAqEtSzv01rB7uf9c 2ZuvT4lDEPWJH0/lkwdkYvrc1sGwbA6f5iNwQ8EU3Rna6eN6He2fKg7z7grmlsU//5AblsQ= X-Google-Smtp-Source: AGHT+IGEA8WDnrG+ENLZvMkfcpeufwauXW2db4AOLTcqYxDOdLCVeXrL0ymSpYLCNBgTU7qcybsNdw== X-Received: by 2002:a05:6a20:7f91:b0:1ee:8435:6b69 with SMTP id adf61e73a8af0-1eef3c4908dmr498654637.1.1740077784029; Thu, 20 Feb 2025 10:56:24 -0800 (PST) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-addee79a984sm9572262a12.32.2025.02.20.10.56.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Feb 2025 10:56:23 -0800 (PST) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, kalesh-anakkur.purayil@broadcom.com, netdev@vger.kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, abeni@redhat.com, horms@kernel.org, michael.chan@broadcom.com, Selvin Xavier Subject: [PATCH rdma-next 9/9] RDMA/bnxt_re: Add support for changing the snap dump level Date: Thu, 20 Feb 2025 10:34:56 -0800 Message-Id: <1740076496-14227-10-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> References: <1740076496-14227-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Add support for changing the default debug dump level. Reviewed-by: Saravanan Vajravel Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/debugfs.c | 49 +++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 45601635..45d62f6 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -304,6 +304,8 @@ struct bnxt_re_dev { struct workqueue_struct *dcb_wq; /* Head to track all QP dump */ struct bnxt_re_qdump_head qdump_head; + struct dentry *tunables; + struct dentry *snapdump_dbg; u8 snapdump_dbg_lvl; struct dentry *cc_config; struct bnxt_re_dbg_cc_config_params *cc_config_params; diff --git a/drivers/infiniband/hw/bnxt_re/debugfs.c b/drivers/infiniband/hw/bnxt_re/debugfs.c index af91d16..e3fba53 100644 --- a/drivers/infiniband/hw/bnxt_re/debugfs.c +++ b/drivers/infiniband/hw/bnxt_re/debugfs.c @@ -306,6 +306,52 @@ static const struct file_operations bnxt_re_cc_config_ops = { .write = bnxt_re_cc_config_set, }; +static ssize_t bnxt_re_snapdump_dbg_lvl_set(struct file *filp, const char __user *buffer, + size_t count, loff_t *ppos) +{ + struct bnxt_re_dev *rdev = filp->private_data; + char buf[16]; + u32 val; + + if (count >= sizeof(buf)) + return -EINVAL; + + if (copy_from_user(buf, buffer, count)) + return -EFAULT; + + buf[count] = '\0'; + if (kstrtou32(buf, 0, &val)) + return -EINVAL; + + if (val > BNXT_RE_SNAPDUMP_ALL) + return -EINVAL; + + rdev->snapdump_dbg_lvl = val; + + return count; +} + +static ssize_t bnxt_re_snapdump_dbg_lvl_get(struct file *filp, char __user *buffer, + size_t usr_buf_len, loff_t *ppos) +{ + struct bnxt_re_dev *rdev = filp->private_data; + char buf[16]; + int rc; + + rc = snprintf(buf, sizeof(buf), "%d\n", rdev->snapdump_dbg_lvl); + if (rc < 0) + return rc; + + return simple_read_from_buffer(buffer, usr_buf_len, ppos, (u8 *)(buf), rc); +} + +static const struct file_operations bnxt_re_snapdump_dbg_lvl = { + .owner = THIS_MODULE, + .open = simple_open, + .read = bnxt_re_snapdump_dbg_lvl_get, + .write = bnxt_re_snapdump_dbg_lvl_set, +}; + void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) { struct pci_dev *pdev = rdev->en_dev->pdev; @@ -329,6 +375,9 @@ void bnxt_re_debugfs_add_pdev(struct bnxt_re_dev *rdev) rdev->cc_config, tmp_params, &bnxt_re_cc_config_ops); } + rdev->tunables = debugfs_create_dir("tunables", rdev->dbg_root); + rdev->snapdump_dbg = debugfs_create_file("snapdump_dbg_lvl", 0400, rdev->tunables, rdev, + &bnxt_re_snapdump_dbg_lvl); } void bnxt_re_debugfs_rem_pdev(struct bnxt_re_dev *rdev)