From patchwork Tue Dec 17 21:08:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912507 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18C1F1F8AF9 for ; Tue, 17 Dec 2024 21:08:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469733; cv=none; b=WL+4r8CY7cEZcRBEtErmYfU4pT3vxsJ6/m/tv+h0qymTQ9Ch8MbCh/Q4fAsDkSsL9t+UQWGrvyMqzbu/pFDHQr0yJEfHGsHCKX/EsoUcJoN4cAPLqANyyIFbOSdbSvLY81PEHUllmDt+yXke+u0Ifx+xfq+TjYcudSpTMfj2LQ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469733; c=relaxed/simple; bh=bKTUhGqIJAv0rmqbIdvWk+NxA3Y7E9yK5xKdfShBEGQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QVW0ba28tZJXHTpWMq0w7KC9ysWqeYMGWHGF2VGcK1I0xqxkJ7qWTOXqBPKLfz4JptgsSHMfnlTJ2wVkVOGdvurO5kD7g92YHZvO3hVUzFvo4hTnHma/zPxiZUIgejf33+CQT/PoKrWoGg62cT5IVjikYhz562f/tNyiMxjT1Do= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=eeVSBVC7; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="eeVSBVC7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469731; x=1766005731; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bKTUhGqIJAv0rmqbIdvWk+NxA3Y7E9yK5xKdfShBEGQ=; b=eeVSBVC7ncG+/SCFZ746Td8PCIJVtfCSurNmtOAXbkNdeQy5qbr4c5tK gSNMvI3QTrJSrftCd3uuLnLEEFU3AQEnYP4HWftWCu/5HTP2Z5UW+70+/ RgZGmE82fmCy8apNl0OAx37rKDVZpUM2En9kRW61HuiqMe5aad0Df4EaV +tyTivWl5Oi16G4E4J8OouOtzxG5ZSfANIkQJDrcNgSY99pBSAp2vgXTW 351HguRF10Gmc/mrm0ZNoe1fTiHrIgb83FbBzrMSq/xLymgmxyb9mCHc4 xQ4x1MUtGxwl2Ta6XFE8ZTS2O5D8qTPLfR5dqwv6OznGAd1JOGgfAq0aB Q==; X-CSE-ConnectionGUID: 3giKfYcPRzGYxXeH990iXA== X-CSE-MsgGUID: r+Rzm6QrTl+xM3IROirVCg== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794824" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794824" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:49 -0800 X-CSE-ConnectionGUID: iQCudGe/Ra2n8C8QWNFs0g== X-CSE-MsgGUID: LsQasKyeRwiC+Ps4X9SokA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436297" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:48 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Przemek Kitszel , anthony.l.nguyen@intel.com, mateusz.polchlopek@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com Subject: [PATCH net-next v2 1/6] checkpatch: don't complain on _Generic() use Date: Tue, 17 Dec 2024 13:08:28 -0800 Message-ID: <20241217210835.3702003-2-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Przemek Kitszel Improve CamelCase recognition logic to avoid reporting on _Generic() use. Other C keywords, such as _Bool, are intentionally omitted, as those should be rather avoided in new source code. Reviewed-by: Wojciech Drewek Reviewed-by: Simon Horman Signed-off-by: Mateusz Polchlopek Acked-by: Joe Perches Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- scripts/checkpatch.pl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index 9eed3683ad76..a2066a6c9dd8 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -5843,6 +5843,8 @@ sub process { #CamelCase if ($var !~ /^$Constant$/ && $var =~ /[A-Z][a-z]|[a-z][A-Z]/ && +#Ignore C keywords + $var !~ /^_Generic$/ && #Ignore some autogenerated defines and enum values $var !~ /^(?:[A-Z]+_){1,5}[A-Z]{1,3}[a-z]/ && #Ignore Page variants From patchwork Tue Dec 17 21:08:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912505 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9BB91F9EAA for ; Tue, 17 Dec 2024 21:08:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469732; cv=none; b=AEXhHSJRSWU+euxVIJN7cuoLv1wI18/gO4ZroWfB+0qsTsvouL5yVjhfS1+pHnmuwv0NBBKXWrYTtoisljwg2YNnAzh8b+uzyc5IH/YI2G0tLdwr2PdAloTQX9+Q+j/mYyDtljnABgzCwjEgZFxaZA92zxLyIOixe3K1eqGTZ4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469732; c=relaxed/simple; bh=IXkXg69wbt2m9zSjVm/D2OAKlEca6Qb6vTLu4GFD83k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QOqi5GP+kG6vthfLwMtFVHy43t2LMQTDBXiHrWQnfvvqTv8Y++RK38xgDA2IzZcIgWezivrusy+yKJUUofrv+swOCdA39+YfaEHNAC/17bh5ioXVUPjE4OS/6HgXn57roCM+JJtTZdPvDQElP2ykThNC7griEhqaa/QlFZw3Hnk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=c2kUwDnh; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="c2kUwDnh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469731; x=1766005731; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IXkXg69wbt2m9zSjVm/D2OAKlEca6Qb6vTLu4GFD83k=; b=c2kUwDnhBX7btN8wqnb9CPbi1S74RQIGR+VHoJlAWvDKiAZuVA9ec+e5 HlmHJmg/EODBk9kKj8kLooLsmiLuBXNRDzYOOHYH0eDQHB9xEHdT72ZlT 0iuQrhBrFAZH3burywrxs9uAvWLUqC+rKzC+UAbfIMnuc3unS3L3FFdJz BU5elKpUv9UVoBX5keHzTw1U42qXbedCY2Rh0BBAtZF8QJBJi1dcU+v8i Tqz+PUCjWAujdCxJQ+aNkRd0G0F/UOwHid5jhE2mJ1kNGw4v6xAmV41yy eL+W3EkaWGZcyBPghL+RIuSn6MBZgEgpjxKdCAXiQVbwsrSFmS0nBDgPp w==; X-CSE-ConnectionGUID: hmFRRdW4R2mptRmB4Cpzyw== X-CSE-MsgGUID: 8nWA5YXmRoi++vLdn/VB1g== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794834" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794834" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:50 -0800 X-CSE-ConnectionGUID: XyYvSRBfSim3eTfAT55Gog== X-CSE-MsgGUID: hDAUtNPaTm2DqByu4zlDIQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436303" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:48 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Przemek Kitszel , anthony.l.nguyen@intel.com, mateusz.polchlopek@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com, Pucha Himasekhar Reddy Subject: [PATCH net-next v2 2/6] devlink: add devlink_fmsg_put() macro Date: Tue, 17 Dec 2024 13:08:29 -0800 Message-ID: <20241217210835.3702003-3-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Przemek Kitszel Add devlink_fmsg_put() that dispatches based on the type of the value to put, example: bool -> devlink_fmsg_bool_pair_put(). Reviewed-by: Wojciech Drewek Reviewed-by: Simon Horman Signed-off-by: Mateusz Polchlopek Tested-by: Pucha Himasekhar Reddy (A Contingent worker at Intel) Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- include/net/devlink.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/net/devlink.h b/include/net/devlink.h index fbb9a2668e24..b5e1427ea4d7 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -1261,6 +1261,17 @@ enum devlink_trap_group_generic_id { .min_burst = _min_burst, \ } +#define devlink_fmsg_put(fmsg, name, value) ( \ + _Generic((value), \ + bool : devlink_fmsg_bool_pair_put, \ + u8 : devlink_fmsg_u8_pair_put, \ + u16 : devlink_fmsg_u32_pair_put, \ + u32 : devlink_fmsg_u32_pair_put, \ + u64 : devlink_fmsg_u64_pair_put, \ + char * : devlink_fmsg_string_pair_put, \ + const char * : devlink_fmsg_string_pair_put) \ + (fmsg, name, (value))) + enum { /* device supports reload operations */ DEVLINK_F_RELOAD = 1UL << 0, From patchwork Tue Dec 17 21:08:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912506 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 285EB1F9ED1 for ; Tue, 17 Dec 2024 21:08:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469732; cv=none; b=AI7eY50XIe0rYaN9oYeLaLLbmLmTr3wirWEB1ahrHOXopAJMY0a2SY02zqd8NgtTbP1MNNPWKsL5191EzBNoRiuIs16yNQ+u97dhxdJpO3ze1g02eF5MUUBAaYs4M/1dR3TwuempWTGKJNgjjDXLBxMpenghHe5q8oC1A5+YhwQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469732; c=relaxed/simple; bh=/bhlUdgvNeBmpHrj0Y1PbRHanK8lIFLQo7+aRsAKnCA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KLybh45Z6oL/WBZGX85xFI5v19SDW/uknJKBDawzRO7O15Gn7MP/myjmz/pl8naSTznxjKD028/y5owLB+qkrXZQBMTGAIAVUHze2sLaePj2fsslujojcNiOnAuKNi/T3fGnab2HCKdt8nwuxU3apVBHkBV/2h/Dd0ej+WUXj5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Nk6g99T2; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Nk6g99T2" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469732; x=1766005732; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/bhlUdgvNeBmpHrj0Y1PbRHanK8lIFLQo7+aRsAKnCA=; b=Nk6g99T2h0Iga73lHL5arwg8PYt5snkR6d261tByAp6ISvPIabf1+NET 5SPM49zq1AGxfvVbRD+ggttca7XqQmo4RptA05Neq7Jql6FjRcyzWkL4B qMYOpoX4xbr+kiunDKcn14Vue9toJxN6MBS2FvWg6NizGTVzLlAu+pOl7 zlqCaKcE0FTyhsk/JkL577PwFROnj83TQ5qAgWnqksbSsw+6g5BtsWkAW v11uiuL1Pfv24G0D9QNcXqxxDqxS/Y4tuNSmN5+u6FPEmDqXyuQAFkRGr YiovIP+0k9ZqHiWM7KljM7xyiekvPESNBdviwB9unMZ446EgeVAE7ten4 w==; X-CSE-ConnectionGUID: ayfABbuRTzelBHaMBkXpoA== X-CSE-MsgGUID: c+y9bFG1SH69C7uox0I4Tw== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794844" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794844" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:50 -0800 X-CSE-ConnectionGUID: aSauyNM1QMOE+E/H+BTYOQ== X-CSE-MsgGUID: 27F2ZUYWQambR1f1+nNfyA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436324" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:49 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Mateusz Polchlopek , anthony.l.nguyen@intel.com, przemyslaw.kitszel@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com, Pucha Himasekhar Reddy Subject: [PATCH net-next v2 3/6] devlink: add devlink_fmsg_dump_skb() function Date: Tue, 17 Dec 2024 13:08:30 -0800 Message-ID: <20241217210835.3702003-4-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Mateusz Polchlopek Add devlink_fmsg_dump_skb() function that adds some diagnostic information about skb (like length, pkt type, MAC, etc) to devlink fmsg mechanism using bunch of devlink_fmsg_put() function calls. Signed-off-by: Mateusz Polchlopek Tested-by: Pucha Himasekhar Reddy (A Contingent worker at Intel) Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- include/net/devlink.h | 2 ++ net/devlink/health.c | 67 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/include/net/devlink.h b/include/net/devlink.h index b5e1427ea4d7..58e33959c852 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -1268,6 +1268,7 @@ enum devlink_trap_group_generic_id { u16 : devlink_fmsg_u32_pair_put, \ u32 : devlink_fmsg_u32_pair_put, \ u64 : devlink_fmsg_u64_pair_put, \ + int : devlink_fmsg_u32_pair_put, \ char * : devlink_fmsg_string_pair_put, \ const char * : devlink_fmsg_string_pair_put) \ (fmsg, name, (value))) @@ -2005,6 +2006,7 @@ int devlink_compat_switch_id_get(struct net_device *dev, int devlink_nl_port_handle_fill(struct sk_buff *msg, struct devlink_port *devlink_port); size_t devlink_nl_port_handle_size(struct devlink_port *devlink_port); +void devlink_fmsg_dump_skb(struct devlink_fmsg *fmsg, const struct sk_buff *skb); #else diff --git a/net/devlink/health.c b/net/devlink/health.c index b8d3084e6fe0..57db6799722a 100644 --- a/net/devlink/health.c +++ b/net/devlink/health.c @@ -1238,3 +1238,70 @@ int devlink_nl_health_reporter_test_doit(struct sk_buff *skb, return reporter->ops->test(reporter, info->extack); } + +/** + * devlink_fmsg_dump_skb - Dump sk_buffer structure + * @fmsg: devlink formatted message pointer + * @skb: pointer to skb + * + * Dump diagnostic information about sk_buff structure, like headroom, length, + * tailroom, MAC, etc. + */ +void devlink_fmsg_dump_skb(struct devlink_fmsg *fmsg, const struct sk_buff *skb) +{ + struct skb_shared_info *sh = skb_shinfo(skb); + struct sock *sk = skb->sk; + bool has_mac, has_trans; + + has_mac = skb_mac_header_was_set(skb); + has_trans = skb_transport_header_was_set(skb); + + devlink_fmsg_pair_nest_start(fmsg, "skb"); + devlink_fmsg_obj_nest_start(fmsg); + devlink_fmsg_put(fmsg, "actual len", skb->len); + devlink_fmsg_put(fmsg, "head len", skb_headlen(skb)); + devlink_fmsg_put(fmsg, "data len", skb->data_len); + devlink_fmsg_put(fmsg, "tail len", skb_tailroom(skb)); + devlink_fmsg_put(fmsg, "MAC", has_mac ? skb->mac_header : -1); + devlink_fmsg_put(fmsg, "MAC len", + has_mac ? skb_mac_header_len(skb) : -1); + devlink_fmsg_put(fmsg, "network hdr", skb->network_header); + devlink_fmsg_put(fmsg, "network hdr len", + has_trans ? skb_network_header_len(skb) : -1); + devlink_fmsg_put(fmsg, "transport hdr", + has_trans ? skb->transport_header : -1); + devlink_fmsg_put(fmsg, "csum", (__force u32)skb->csum); + devlink_fmsg_put(fmsg, "csum_ip_summed", (u8)skb->ip_summed); + devlink_fmsg_put(fmsg, "csum_complete_sw", !!skb->csum_complete_sw); + devlink_fmsg_put(fmsg, "csum_valid", !!skb->csum_valid); + devlink_fmsg_put(fmsg, "csum_level", (u8)skb->csum_level); + devlink_fmsg_put(fmsg, "sw_hash", !!skb->sw_hash); + devlink_fmsg_put(fmsg, "l4_hash", !!skb->l4_hash); + devlink_fmsg_put(fmsg, "proto", ntohs(skb->protocol)); + devlink_fmsg_put(fmsg, "pkt_type", (u8)skb->pkt_type); + devlink_fmsg_put(fmsg, "iif", skb->skb_iif); + + if (sk) { + devlink_fmsg_pair_nest_start(fmsg, "sk"); + devlink_fmsg_obj_nest_start(fmsg); + devlink_fmsg_put(fmsg, "family", sk->sk_type); + devlink_fmsg_put(fmsg, "type", sk->sk_type); + devlink_fmsg_put(fmsg, "proto", sk->sk_protocol); + devlink_fmsg_obj_nest_end(fmsg); + devlink_fmsg_pair_nest_end(fmsg); + } + + devlink_fmsg_obj_nest_end(fmsg); + devlink_fmsg_pair_nest_end(fmsg); + + devlink_fmsg_pair_nest_start(fmsg, "shinfo"); + devlink_fmsg_obj_nest_start(fmsg); + devlink_fmsg_put(fmsg, "tx_flags", sh->tx_flags); + devlink_fmsg_put(fmsg, "nr_frags", sh->nr_frags); + devlink_fmsg_put(fmsg, "gso_size", sh->gso_size); + devlink_fmsg_put(fmsg, "gso_type", sh->gso_type); + devlink_fmsg_put(fmsg, "gso_segs", sh->gso_segs); + devlink_fmsg_obj_nest_end(fmsg); + devlink_fmsg_pair_nest_end(fmsg); +} +EXPORT_SYMBOL_GPL(devlink_fmsg_dump_skb); From patchwork Tue Dec 17 21:08:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912508 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BC181F9AAC for ; Tue, 17 Dec 2024 21:08:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469734; cv=none; b=SbrPA02laOHzZkICa8qyri/n4pvGZUaQJ15vrMQpYba6QNmL4dWY+sbS2Ocm10/MPUXZXbZv9blO+e0YK0Qt4UDg+Kp/KKT6vXB9HhNaX4jrvBznwnxUxTtfWshPNXRrSNK52MrSNDkiRb8IvMaSJXYZyRyIsjZdvbF1jlkxxHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469734; c=relaxed/simple; bh=XW8piwVRvvLzZTxoL1Req3q4u+I6rbvTV8h0/eudJm4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GQrxEHQOkQ142K7CXeQE4g5bkPMSI/o/ApQwE4hp22km3Qx2kyY/KGGR9H6eB0aZTWLckBPO2CIe259V3P4YYazSjd5lh14lLETxaMAEflZlS2/Wd0bKHZnEaCHgvzVWDP62RF6c1ePp/L+Udl24CuN5PMXR1MG/Rx8nJHOdep4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=kcGovcaV; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kcGovcaV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469733; x=1766005733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XW8piwVRvvLzZTxoL1Req3q4u+I6rbvTV8h0/eudJm4=; b=kcGovcaVKTbTmLrK7yneEF6Y4ufER4k661C/eQqqp9CxK5N/V+XQE5lr fIUUZMPTI29fSf/t9AWjgJ6Bbo/AQgi7unpx4QgvFY0u7ZcTcRVIhVXlL 3NBmrfKRj420ah69uvyzVsEB4jFt7lnJTpPrKl/JroI0U/6E4DI/AbnsH B8M1FhOrRZIFq3efgUyPk7unAZnsO/VELyZt8ncPdpcfmEhWb31yR7qdx fGxhtkt5qsCJbEWjOdPuJGRSGY9e/VkAAEMP7JW5uWHXtYRSNsfrAOrFs 8KFHMuI5DbRrsNqlBhYx+VUrEh9HAwHXfvFDUL8c22wuOGwRHVwmpa/XF w==; X-CSE-ConnectionGUID: 3/yGN18oSBqImCoxRhOY0A== X-CSE-MsgGUID: damk1oXQSNKXgiNT4o9TDA== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794854" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794854" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:51 -0800 X-CSE-ConnectionGUID: 98azbp7jTZa0RmFnNXYerg== X-CSE-MsgGUID: aTR4LGYwT26SnMenqF8mlg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436331" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:50 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Przemek Kitszel , anthony.l.nguyen@intel.com, mateusz.polchlopek@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com, Kalesh AP Subject: [PATCH net-next v2 4/6] ice: rename devlink_port.[ch] to port.[ch] Date: Tue, 17 Dec 2024 13:08:31 -0800 Message-ID: <20241217210835.3702003-5-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Przemek Kitszel Drop "devlink_" prefix from files that sit in devlink/. I'm going to add more files there, and repeating "devlink" does not feel good. This is also the scheme used in most other places, most notably the devlink core files are named like that. devlink.[ch] stays as is. Reviewed-by: Kalesh AP Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/ice/Makefile | 2 +- drivers/net/ethernet/intel/ice/devlink/devlink.c | 2 +- .../net/ethernet/intel/ice/devlink/{devlink_port.c => port.c} | 2 +- .../net/ethernet/intel/ice/devlink/{devlink_port.h => port.h} | 0 drivers/net/ethernet/intel/ice/ice_eswitch.h | 2 +- drivers/net/ethernet/intel/ice/ice_main.c | 2 +- drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- drivers/net/ethernet/intel/ice/ice_sf_eth.c | 2 +- 8 files changed, 7 insertions(+), 7 deletions(-) rename drivers/net/ethernet/intel/ice/devlink/{devlink_port.c => port.c} (99%) rename drivers/net/ethernet/intel/ice/devlink/{devlink_port.h => port.h} (100%) diff --git a/drivers/net/ethernet/intel/ice/Makefile b/drivers/net/ethernet/intel/ice/Makefile index 3307d551f431..56aa23aee472 100644 --- a/drivers/net/ethernet/intel/ice/Makefile +++ b/drivers/net/ethernet/intel/ice/Makefile @@ -32,7 +32,7 @@ ice-y := ice_main.o \ ice_parser_rt.o \ ice_idc.o \ devlink/devlink.o \ - devlink/devlink_port.o \ + devlink/port.o \ ice_sf_eth.o \ ice_sf_vsi_vlan_ops.o \ ice_ddp.o \ diff --git a/drivers/net/ethernet/intel/ice/devlink/devlink.c b/drivers/net/ethernet/intel/ice/devlink/devlink.c index 415445cefdb2..1b10682c00b8 100644 --- a/drivers/net/ethernet/intel/ice/devlink/devlink.c +++ b/drivers/net/ethernet/intel/ice/devlink/devlink.c @@ -6,7 +6,7 @@ #include "ice.h" #include "ice_lib.h" #include "devlink.h" -#include "devlink_port.h" +#include "port.h" #include "ice_eswitch.h" #include "ice_fw_update.h" #include "ice_dcb_lib.h" diff --git a/drivers/net/ethernet/intel/ice/devlink/devlink_port.c b/drivers/net/ethernet/intel/ice/devlink/port.c similarity index 99% rename from drivers/net/ethernet/intel/ice/devlink/devlink_port.c rename to drivers/net/ethernet/intel/ice/devlink/port.c index c6779d9dffff..767419a67fef 100644 --- a/drivers/net/ethernet/intel/ice/devlink/devlink_port.c +++ b/drivers/net/ethernet/intel/ice/devlink/port.c @@ -5,7 +5,7 @@ #include "ice.h" #include "devlink.h" -#include "devlink_port.h" +#include "port.h" #include "ice_lib.h" #include "ice_fltr.h" diff --git a/drivers/net/ethernet/intel/ice/devlink/devlink_port.h b/drivers/net/ethernet/intel/ice/devlink/port.h similarity index 100% rename from drivers/net/ethernet/intel/ice/devlink/devlink_port.h rename to drivers/net/ethernet/intel/ice/devlink/port.h diff --git a/drivers/net/ethernet/intel/ice/ice_eswitch.h b/drivers/net/ethernet/intel/ice/ice_eswitch.h index ac7db100e2cd..5c7dcf21b222 100644 --- a/drivers/net/ethernet/intel/ice/ice_eswitch.h +++ b/drivers/net/ethernet/intel/ice/ice_eswitch.h @@ -5,7 +5,7 @@ #define _ICE_ESWITCH_H_ #include -#include "devlink/devlink_port.h" +#include "devlink/port.h" #ifdef CONFIG_ICE_SWITCHDEV void ice_eswitch_detach_vf(struct ice_pf *pf, struct ice_vf *vf); diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index 0ab35607e5d5..d641dd8b8184 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -14,7 +14,7 @@ #include "ice_dcb_lib.h" #include "ice_dcb_nl.h" #include "devlink/devlink.h" -#include "devlink/devlink_port.h" +#include "devlink/port.h" #include "ice_sf_eth.h" #include "ice_hwmon.h" /* Including ice_trace.h with CREATE_TRACE_POINTS defined will generate the diff --git a/drivers/net/ethernet/intel/ice/ice_repr.c b/drivers/net/ethernet/intel/ice/ice_repr.c index 970a99a52bf1..fb7a1b9a4313 100644 --- a/drivers/net/ethernet/intel/ice/ice_repr.c +++ b/drivers/net/ethernet/intel/ice/ice_repr.c @@ -4,7 +4,7 @@ #include "ice.h" #include "ice_eswitch.h" #include "devlink/devlink.h" -#include "devlink/devlink_port.h" +#include "devlink/port.h" #include "ice_sriov.h" #include "ice_tc_lib.h" #include "ice_dcb_lib.h" diff --git a/drivers/net/ethernet/intel/ice/ice_sf_eth.c b/drivers/net/ethernet/intel/ice/ice_sf_eth.c index 75d7147e1c01..1a2c94375ca7 100644 --- a/drivers/net/ethernet/intel/ice/ice_sf_eth.c +++ b/drivers/net/ethernet/intel/ice/ice_sf_eth.c @@ -5,8 +5,8 @@ #include "ice_txrx.h" #include "ice_fltr.h" #include "ice_sf_eth.h" -#include "devlink/devlink_port.h" #include "devlink/devlink.h" +#include "devlink/port.h" static const struct net_device_ops ice_sf_netdev_ops = { .ndo_open = ice_open, From patchwork Tue Dec 17 21:08:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912509 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C2EE1F9F6F for ; Tue, 17 Dec 2024 21:08:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469735; cv=none; b=SvqzQ/1bgeuo11pwfnFvOAD5wzsoAiYwOZCJwpqMRUeDtsAaKFcdghtgV8wTcnrwShiO11jTLk2Yh+5lyhVDcgKyIJUEhW7juyNgltumq0x6xxKSEKHh1BiDDBrdSqeumzoRE33X8YDiFvoUrJeLr8/WETKFVw9imLB+ZEkEdPI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469735; c=relaxed/simple; bh=2tytuUZvm3nJXddfq6bF/ZkXswf1hBc3FocAMhjrtaE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TNYAX2bp26fbd8mNrQyQOKjdhBNYbYce2igmVYYvdsqsoxgJ5wKWocu7txOd04lZuwXPbrqMnNmTtbDDLyNnEi8lnKg3NAQqB7m3exeBxYJc0ZyFYhbiCG0G2E6ygJ62bt4YFOT1QByUiEKS2AoUsy1f98dvag5pPbsDEHDmZsc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XrgU5/5X; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XrgU5/5X" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469733; x=1766005733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2tytuUZvm3nJXddfq6bF/ZkXswf1hBc3FocAMhjrtaE=; b=XrgU5/5X+crkyqSqkGGV5/43Rser+u2R6Qa4wA1KoHexeTb5OglmK10m ottMLgfOFAjpiOYcB/kPurqz3pjUmIeTd6xEOpfk8XlQ0unIEgUGv4lQE XRgigJysHvNwRcFofO+TresWKo1XhE3MJ13cvXsuI1jF6lP/Yb9g/rIge 7Gr9pYxEjuqJXBpfdnEadbqiFn/4RipQZ1Sw944+OOcPH9iPPFykvomvT p1zGlIzJn4imef/Lro3t97cbyxBdlTd8u9MAYY38Q+zDuaZLbadZ2mLn4 3N7QMMAN4UxLChO/AoqLc2nNOwqlx/PzXlgeLCjzwxXsu6KSFZumPkXQK A==; X-CSE-ConnectionGUID: LwvOUmMGTJSwzWGFYZlXeQ== X-CSE-MsgGUID: WexEN+ybSZavS3VNo7s7FA== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794865" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794865" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:52 -0800 X-CSE-ConnectionGUID: n6KJEVyeSe+jUf1BUUKCcA== X-CSE-MsgGUID: jpLzBKsFQZqfPXLrivxbRw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436339" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:50 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Przemek Kitszel , anthony.l.nguyen@intel.com, mateusz.polchlopek@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com, Igor Bagnucki , Pucha Himasekhar Reddy Subject: [PATCH net-next v2 5/6] ice: add Tx hang devlink health reporter Date: Tue, 17 Dec 2024 13:08:32 -0800 Message-ID: <20241217210835.3702003-6-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Przemek Kitszel Add Tx hang devlink health reporter, see struct ice_tx_hang_event to see what exactly is reported. For now dump descriptors with little metadata and skb diagnostic information. Reviewed-by: Igor Bagnucki Reviewed-by: Wojciech Drewek Co-developed-by: Mateusz Polchlopek Signed-off-by: Mateusz Polchlopek Tested-by: Pucha Himasekhar Reddy (A Contingent worker at Intel) Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/ice/Makefile | 1 + .../net/ethernet/intel/ice/devlink/health.c | 192 ++++++++++++++++++ .../net/ethernet/intel/ice/devlink/health.h | 47 +++++ drivers/net/ethernet/intel/ice/ice.h | 2 + drivers/net/ethernet/intel/ice/ice_main.c | 18 +- 5 files changed, 255 insertions(+), 5 deletions(-) create mode 100644 drivers/net/ethernet/intel/ice/devlink/health.c create mode 100644 drivers/net/ethernet/intel/ice/devlink/health.h diff --git a/drivers/net/ethernet/intel/ice/Makefile b/drivers/net/ethernet/intel/ice/Makefile index 56aa23aee472..9e0d9f710441 100644 --- a/drivers/net/ethernet/intel/ice/Makefile +++ b/drivers/net/ethernet/intel/ice/Makefile @@ -32,6 +32,7 @@ ice-y := ice_main.o \ ice_parser_rt.o \ ice_idc.o \ devlink/devlink.o \ + devlink/health.o \ devlink/port.o \ ice_sf_eth.o \ ice_sf_vsi_vlan_ops.o \ diff --git a/drivers/net/ethernet/intel/ice/devlink/health.c b/drivers/net/ethernet/intel/ice/devlink/health.c new file mode 100644 index 000000000000..984d910fc41d --- /dev/null +++ b/drivers/net/ethernet/intel/ice/devlink/health.c @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024, Intel Corporation. */ + +#include "health.h" +#include "ice.h" + +#define ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, obj, name) \ + devlink_fmsg_put(fmsg, #name, (obj)->name) + +/** + * ice_devlink_health_report - boilerplate to call given @reporter + * + * @reporter: devlink health reporter to call, do nothing on NULL + * @msg: message to pass up, "event name" is fine + * @priv_ctx: typically some event struct + */ +static void ice_devlink_health_report(struct devlink_health_reporter *reporter, + const char *msg, void *priv_ctx) +{ + if (!reporter) + return; + + /* We do not do auto recovering, so return value of the below function + * will always be 0, thus we do ignore it. + */ + devlink_health_report(reporter, msg, priv_ctx); +} + +/** + * ice_fmsg_put_ptr - put hex value of pointer into fmsg + * + * @fmsg: devlink fmsg under construction + * @name: name to pass + * @ptr: 64 bit value to print as hex and put into fmsg + */ +static void ice_fmsg_put_ptr(struct devlink_fmsg *fmsg, const char *name, + void *ptr) +{ + char buf[sizeof(ptr) * 3]; + + sprintf(buf, "%p", ptr); + devlink_fmsg_put(fmsg, name, buf); +} + +struct ice_tx_hang_event { + u32 head; + u32 intr; + u16 vsi_num; + u16 queue; + u16 next_to_clean; + u16 next_to_use; + struct ice_tx_ring *tx_ring; +}; + +static int ice_tx_hang_reporter_dump(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, void *priv_ctx, + struct netlink_ext_ack *extack) +{ + struct ice_tx_hang_event *event = priv_ctx; + struct sk_buff *skb; + + if (!event) + return 0; + + skb = event->tx_ring->tx_buf->skb; + devlink_fmsg_obj_nest_start(fmsg); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, head); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, intr); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, vsi_num); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, queue); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, next_to_clean); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, event, next_to_use); + devlink_fmsg_put(fmsg, "irq-mapping", event->tx_ring->q_vector->name); + ice_fmsg_put_ptr(fmsg, "desc-ptr", event->tx_ring->desc); + ice_fmsg_put_ptr(fmsg, "dma-ptr", (void *)(long)event->tx_ring->dma); + ice_fmsg_put_ptr(fmsg, "skb-ptr", skb); + devlink_fmsg_binary_pair_put(fmsg, "desc", event->tx_ring->desc, + event->tx_ring->count * sizeof(struct ice_tx_desc)); + devlink_fmsg_dump_skb(fmsg, skb); + devlink_fmsg_obj_nest_end(fmsg); + + return 0; +} + +void ice_prep_tx_hang_report(struct ice_pf *pf, struct ice_tx_ring *tx_ring, + u16 vsi_num, u32 head, u32 intr) +{ + struct ice_health_tx_hang_buf *buf = &pf->health_reporters.tx_hang_buf; + + buf->tx_ring = tx_ring; + buf->vsi_num = vsi_num; + buf->head = head; + buf->intr = intr; +} + +void ice_report_tx_hang(struct ice_pf *pf) +{ + struct ice_health_tx_hang_buf *buf = &pf->health_reporters.tx_hang_buf; + struct ice_tx_ring *tx_ring = buf->tx_ring; + + struct ice_tx_hang_event ev = { + .head = buf->head, + .intr = buf->intr, + .vsi_num = buf->vsi_num, + .queue = tx_ring->q_index, + .next_to_clean = tx_ring->next_to_clean, + .next_to_use = tx_ring->next_to_use, + .tx_ring = tx_ring, + }; + + ice_devlink_health_report(pf->health_reporters.tx_hang, "Tx hang", &ev); +} + +static struct devlink_health_reporter * +ice_init_devlink_rep(struct ice_pf *pf, + const struct devlink_health_reporter_ops *ops) +{ + struct devlink *devlink = priv_to_devlink(pf); + struct devlink_health_reporter *rep; + const u64 graceful_period = 0; + + rep = devl_health_reporter_create(devlink, ops, graceful_period, pf); + if (IS_ERR(rep)) { + struct device *dev = ice_pf_to_dev(pf); + + dev_err(dev, "failed to create devlink %s health report er", + ops->name); + return NULL; + } + return rep; +} + +#define ICE_DEFINE_HEALTH_REPORTER_OPS(_name) \ + static const struct devlink_health_reporter_ops ice_ ## _name ## _reporter_ops = { \ + .name = #_name, \ + .dump = ice_ ## _name ## _reporter_dump, \ +} + +ICE_DEFINE_HEALTH_REPORTER_OPS(tx_hang); + +/** + * ice_health_init - allocate and init all ice devlink health reporters and + * accompanied data + * + * @pf: PF struct + */ +void ice_health_init(struct ice_pf *pf) +{ + struct ice_health *reps = &pf->health_reporters; + + reps->tx_hang = ice_init_devlink_rep(pf, &ice_tx_hang_reporter_ops); +} + +/** + * ice_deinit_devl_reporter - destroy given devlink health reporter + * @reporter: reporter to destroy + */ +static void ice_deinit_devl_reporter(struct devlink_health_reporter *reporter) +{ + if (reporter) + devl_health_reporter_destroy(reporter); +} + +/** + * ice_health_deinit - deallocate all ice devlink health reporters and + * accompanied data + * + * @pf: PF struct + */ +void ice_health_deinit(struct ice_pf *pf) +{ + ice_deinit_devl_reporter(pf->health_reporters.tx_hang); +} + +static +void ice_health_assign_healthy_state(struct devlink_health_reporter *reporter) +{ + if (reporter) + devlink_health_reporter_state_update(reporter, + DEVLINK_HEALTH_REPORTER_STATE_HEALTHY); +} + +/** + * ice_health_clear - clear devlink health issues after a reset + * @pf: the PF device structure + * + * Mark the PF in healthy state again after a reset has completed. + */ +void ice_health_clear(struct ice_pf *pf) +{ + ice_health_assign_healthy_state(pf->health_reporters.tx_hang); +} diff --git a/drivers/net/ethernet/intel/ice/devlink/health.h b/drivers/net/ethernet/intel/ice/devlink/health.h new file mode 100644 index 000000000000..5ce601227acb --- /dev/null +++ b/drivers/net/ethernet/intel/ice/devlink/health.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2024, Intel Corporation. */ + +#ifndef _HEALTH_H_ +#define _HEALTH_H_ + +#include + +/** + * DOC: health.h + * + * This header file stores everything that is needed for broadly understood + * devlink health mechanism for ice driver. + */ + +struct ice_pf; +struct ice_tx_ring; + +/** + * struct ice_health - stores ice devlink health reporters and accompanied data + * @tx_hang: devlink health reporter for tx_hang event + * @tx_hang_buf: pre-allocated place to put info for Tx hang reporter from + * non-sleeping context + * @tx_ring: ring that the hang occurred on + * @head: descriptor head + * @intr: interrupt register value + * @vsi_num: VSI owning the queue that the hang occurred on + */ +struct ice_health { + struct devlink_health_reporter *tx_hang; + struct_group_tagged(ice_health_tx_hang_buf, tx_hang_buf, + struct ice_tx_ring *tx_ring; + u32 head; + u32 intr; + u16 vsi_num; + ); +}; + +void ice_health_init(struct ice_pf *pf); +void ice_health_deinit(struct ice_pf *pf); +void ice_health_clear(struct ice_pf *pf); + +void ice_prep_tx_hang_report(struct ice_pf *pf, struct ice_tx_ring *tx_ring, + u16 vsi_num, u32 head, u32 intr); +void ice_report_tx_hang(struct ice_pf *pf); + +#endif /* _HEALTH_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h index 2f5d6f974185..71e05d30f0fd 100644 --- a/drivers/net/ethernet/intel/ice/ice.h +++ b/drivers/net/ethernet/intel/ice/ice.h @@ -78,6 +78,7 @@ #include "ice_irq.h" #include "ice_dpll.h" #include "ice_adapter.h" +#include "devlink/health.h" #define ICE_BAR0 0 #define ICE_REQ_DESC_MULTIPLE 32 @@ -665,6 +666,7 @@ struct ice_pf { struct ice_agg_node vf_agg_node[ICE_MAX_VF_AGG_NODES]; struct ice_dplls dplls; struct device *hwmon_dev; + struct ice_health health_reporters; u8 num_quanta_prof_used; }; diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index d641dd8b8184..316f5109bd3f 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -2364,9 +2364,11 @@ static void ice_service_task(struct work_struct *work) struct ice_pf *pf = container_of(work, struct ice_pf, serv_task); unsigned long start_time = jiffies; - /* subtasks */ + if (pf->health_reporters.tx_hang_buf.tx_ring) { + ice_report_tx_hang(pf); + pf->health_reporters.tx_hang_buf.tx_ring = NULL; + } - /* process reset requests first */ ice_reset_subtask(pf); /* bail if a reset/recovery cycle is pending or rebuild failed */ @@ -5087,6 +5089,7 @@ static int ice_init_devlink(struct ice_pf *pf) return err; ice_devlink_init_regions(pf); + ice_health_init(pf); ice_devlink_register(pf); return 0; @@ -5095,6 +5098,7 @@ static int ice_init_devlink(struct ice_pf *pf) static void ice_deinit_devlink(struct ice_pf *pf) { ice_devlink_unregister(pf); + ice_health_deinit(pf); ice_devlink_destroy_regions(pf); ice_devlink_unregister_params(pf); } @@ -7793,6 +7797,8 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) /* if we get here, reset flow is successful */ clear_bit(ICE_RESET_FAILED, pf->state); + ice_health_clear(pf); + ice_plug_aux_dev(pf); if (ice_is_feature_supported(pf, ICE_F_SRIOV_LAG)) ice_lag_rebuild(pf); @@ -8283,16 +8289,18 @@ void ice_tx_timeout(struct net_device *netdev, unsigned int txqueue) if (tx_ring) { struct ice_hw *hw = &pf->hw; - u32 head, val = 0; + u32 head, intr = 0; head = FIELD_GET(QTX_COMM_HEAD_HEAD_M, rd32(hw, QTX_COMM_HEAD(vsi->txq_map[txqueue]))); /* Read interrupt register */ - val = rd32(hw, GLINT_DYN_CTL(tx_ring->q_vector->reg_idx)); + intr = rd32(hw, GLINT_DYN_CTL(tx_ring->q_vector->reg_idx)); netdev_info(netdev, "tx_timeout: VSI_num: %d, Q %u, NTC: 0x%x, HW_HEAD: 0x%x, NTU: 0x%x, INT: 0x%x\n", vsi->vsi_num, txqueue, tx_ring->next_to_clean, - head, tx_ring->next_to_use, val); + head, tx_ring->next_to_use, intr); + + ice_prep_tx_hang_report(pf, tx_ring, vsi->vsi_num, head, intr); } pf->tx_timeout_last_recovery = jiffies; From patchwork Tue Dec 17 21:08:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 13912510 X-Patchwork-Delegate: kuba@kernel.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DB541FA14D for ; Tue, 17 Dec 2024 21:08:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469735; cv=none; b=QpY3h+Ry6/A+hYv5Xrh8TRa6ruw/krXVyzG2+QxWf7LzPvnrNTsgRK/C75I8EOEZVKHeilMOMjTbOFFbnPklVj5NovWDaqKntdtrhII2277rJZ4W251b7qg9D7I/KVPli9OHhHuLNpsEFdKHXUvxE26z770xW1MyGsADmJ5hej8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734469735; c=relaxed/simple; bh=6baHemqpIGcdwzsVEbWbAm0Oj+lgxTkHJL7Q50m+wLs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mxEH3ky6ZcG1DV+OvQQcVNrUFXnOOF0Fm/WGng/LUxq73+lWouPMCqaypDe6J+5r36A7lIzMs0vNUxg/HnkXltrbiAEQtvxLGeRdg1lKrWY/n8fg3q2MJ1ad2CtN9yMBStwBEwfcdhNEzsdXYKxlaGXizZ+AQ2pLcT29FvMhSEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mJFq14C0; arc=none smtp.client-ip=198.175.65.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mJFq14C0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1734469734; x=1766005734; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6baHemqpIGcdwzsVEbWbAm0Oj+lgxTkHJL7Q50m+wLs=; b=mJFq14C0qKWSUiflQpMpi5oawEfqOWo/e/7FnLNs5hcFIDveyK4GgHeS 8/DSUxXjlHt5te7Tmb5UM7QFH5GrOUqfuXgsIaQx9nY7rR/7QMt03zMA+ i6PaFbppN403mbHur1uAR6HhIIWNZFOAek6Dzv1QXtiCeYWZAvvpdXmPm hc9OjdfDpxhO13CIuJqbvLqoh7k72wW9VnJUwvGYA5XfR76FcKKzsBYw+ qK8a3v3QztwUduEdxrMh5DxKEX2vfzJrQ0UNZpQ3xzaIoWvbDTU3WoqVO pbEzmozZNVOARSl/fWknxZhTFvTKsr5Je5OvOMMNXvGuhJVCgDCa5bI/Q A==; X-CSE-ConnectionGUID: kS+JgNpSR9ynClpy36Hy/w== X-CSE-MsgGUID: ThJf3bypR+eCzKYo8p3XNg== X-IronPort-AV: E=McAfee;i="6700,10204,11289"; a="34794876" X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="34794876" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2024 13:08:52 -0800 X-CSE-ConnectionGUID: eLLGfPt2QimNURFlDvYsQQ== X-CSE-MsgGUID: 9zRQPXuTSG+NsTyjZDvaCg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,242,1728975600"; d="scan'208";a="97436345" Received: from anguy11-upstream.jf.intel.com ([10.166.9.133]) by fmviesa006.fm.intel.com with ESMTP; 17 Dec 2024 13:08:51 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, edumazet@google.com, andrew+netdev@lunn.ch, netdev@vger.kernel.org Cc: Ben Shelton , anthony.l.nguyen@intel.com, przemyslaw.kitszel@intel.com, mateusz.polchlopek@intel.com, joe@perches.com, horms@kernel.org, jiri@resnulli.us, apw@canonical.com, lukas.bulwahn@gmail.com, dwaipayanray1@gmail.com, Igor Bagnucki , Pucha Himasekhar Reddy Subject: [PATCH net-next v2 6/6] ice: Add MDD logging via devlink health Date: Tue, 17 Dec 2024 13:08:33 -0800 Message-ID: <20241217210835.3702003-7-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> References: <20241217210835.3702003-1-anthony.l.nguyen@intel.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Ben Shelton Add a devlink health reporter for MDD events. The 'dump' handler will return the information captured in each call to ice_handle_mdd_event(). A device reset (CORER/PFR) will put the reporter back in healthy state. Signed-off-by: Ben Shelton Reviewed-by: Igor Bagnucki Reviewed-by: Wojciech Drewek Reviewed-by: Simon Horman Signed-off-by: Mateusz Polchlopek Tested-by: Pucha Himasekhar Reddy (A Contingent worker at Intel) Co-developed-by: Przemek Kitszel Signed-off-by: Przemek Kitszel Signed-off-by: Tony Nguyen --- .../net/ethernet/intel/ice/devlink/health.c | 77 +++++++++++++++++++ .../net/ethernet/intel/ice/devlink/health.h | 11 +++ drivers/net/ethernet/intel/ice/ice_main.c | 6 ++ 3 files changed, 94 insertions(+) diff --git a/drivers/net/ethernet/intel/ice/devlink/health.c b/drivers/net/ethernet/intel/ice/devlink/health.c index 984d910fc41d..d23ae3aafaa7 100644 --- a/drivers/net/ethernet/intel/ice/devlink/health.c +++ b/drivers/net/ethernet/intel/ice/devlink/health.c @@ -26,6 +26,79 @@ static void ice_devlink_health_report(struct devlink_health_reporter *reporter, devlink_health_report(reporter, msg, priv_ctx); } +struct ice_mdd_event { + enum ice_mdd_src src; + u16 vf_num; + u16 queue; + u8 pf_num; + u8 event; +}; + +static const char *ice_mdd_src_to_str(enum ice_mdd_src src) +{ + switch (src) { + case ICE_MDD_SRC_TX_PQM: + return "tx_pqm"; + case ICE_MDD_SRC_TX_TCLAN: + return "tx_tclan"; + case ICE_MDD_SRC_TX_TDPU: + return "tx_tdpu"; + case ICE_MDD_SRC_RX: + return "rx"; + default: + return "invalid"; + } +} + +static int +ice_mdd_reporter_dump(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, void *priv_ctx, + struct netlink_ext_ack *extack) +{ + struct ice_mdd_event *mdd_event = priv_ctx; + const char *src; + + if (!mdd_event) + return 0; + + src = ice_mdd_src_to_str(mdd_event->src); + + devlink_fmsg_obj_nest_start(fmsg); + devlink_fmsg_put(fmsg, "src", src); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, mdd_event, pf_num); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, mdd_event, vf_num); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, mdd_event, event); + ICE_DEVLINK_FMSG_PUT_FIELD(fmsg, mdd_event, queue); + devlink_fmsg_obj_nest_end(fmsg); + + return 0; +} + +/** + * ice_report_mdd_event - Report an MDD event through devlink health + * @pf: the PF device structure + * @src: the HW block that was the source of this MDD event + * @pf_num: the pf_num on which the MDD event occurred + * @vf_num: the vf_num on which the MDD event occurred + * @event: the event type of the MDD event + * @queue: the queue on which the MDD event occurred + * + * Report an MDD event that has occurred on this PF. + */ +void ice_report_mdd_event(struct ice_pf *pf, enum ice_mdd_src src, u8 pf_num, + u16 vf_num, u8 event, u16 queue) +{ + struct ice_mdd_event ev = { + .src = src, + .pf_num = pf_num, + .vf_num = vf_num, + .event = event, + .queue = queue, + }; + + ice_devlink_health_report(pf->health_reporters.mdd, "MDD event", &ev); +} + /** * ice_fmsg_put_ptr - put hex value of pointer into fmsg * @@ -136,6 +209,7 @@ ice_init_devlink_rep(struct ice_pf *pf, .dump = ice_ ## _name ## _reporter_dump, \ } +ICE_DEFINE_HEALTH_REPORTER_OPS(mdd); ICE_DEFINE_HEALTH_REPORTER_OPS(tx_hang); /** @@ -148,6 +222,7 @@ void ice_health_init(struct ice_pf *pf) { struct ice_health *reps = &pf->health_reporters; + reps->mdd = ice_init_devlink_rep(pf, &ice_mdd_reporter_ops); reps->tx_hang = ice_init_devlink_rep(pf, &ice_tx_hang_reporter_ops); } @@ -169,6 +244,7 @@ static void ice_deinit_devl_reporter(struct devlink_health_reporter *reporter) */ void ice_health_deinit(struct ice_pf *pf) { + ice_deinit_devl_reporter(pf->health_reporters.mdd); ice_deinit_devl_reporter(pf->health_reporters.tx_hang); } @@ -188,5 +264,6 @@ void ice_health_assign_healthy_state(struct devlink_health_reporter *reporter) */ void ice_health_clear(struct ice_pf *pf) { + ice_health_assign_healthy_state(pf->health_reporters.mdd); ice_health_assign_healthy_state(pf->health_reporters.tx_hang); } diff --git a/drivers/net/ethernet/intel/ice/devlink/health.h b/drivers/net/ethernet/intel/ice/devlink/health.h index 5ce601227acb..532277fc57d7 100644 --- a/drivers/net/ethernet/intel/ice/devlink/health.h +++ b/drivers/net/ethernet/intel/ice/devlink/health.h @@ -16,9 +16,17 @@ struct ice_pf; struct ice_tx_ring; +enum ice_mdd_src { + ICE_MDD_SRC_TX_PQM, + ICE_MDD_SRC_TX_TCLAN, + ICE_MDD_SRC_TX_TDPU, + ICE_MDD_SRC_RX, +}; + /** * struct ice_health - stores ice devlink health reporters and accompanied data * @tx_hang: devlink health reporter for tx_hang event + * @mdd: devlink health reporter for MDD detection event * @tx_hang_buf: pre-allocated place to put info for Tx hang reporter from * non-sleeping context * @tx_ring: ring that the hang occurred on @@ -27,6 +35,7 @@ struct ice_tx_ring; * @vsi_num: VSI owning the queue that the hang occurred on */ struct ice_health { + struct devlink_health_reporter *mdd; struct devlink_health_reporter *tx_hang; struct_group_tagged(ice_health_tx_hang_buf, tx_hang_buf, struct ice_tx_ring *tx_ring; @@ -42,6 +51,8 @@ void ice_health_clear(struct ice_pf *pf); void ice_prep_tx_hang_report(struct ice_pf *pf, struct ice_tx_ring *tx_ring, u16 vsi_num, u32 head, u32 intr); +void ice_report_mdd_event(struct ice_pf *pf, enum ice_mdd_src src, u8 pf_num, + u16 vf_num, u8 event, u16 queue); void ice_report_tx_hang(struct ice_pf *pf); #endif /* _HEALTH_H_ */ diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index 316f5109bd3f..1701f7143f24 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -1816,6 +1816,8 @@ static void ice_handle_mdd_event(struct ice_pf *pf) if (netif_msg_tx_err(pf)) dev_info(dev, "Malicious Driver Detection event %d on TX queue %d PF# %d VF# %d\n", event, queue, pf_num, vf_num); + ice_report_mdd_event(pf, ICE_MDD_SRC_TX_PQM, pf_num, vf_num, + event, queue); wr32(hw, GL_MDET_TX_PQM, 0xffffffff); } @@ -1829,6 +1831,8 @@ static void ice_handle_mdd_event(struct ice_pf *pf) if (netif_msg_tx_err(pf)) dev_info(dev, "Malicious Driver Detection event %d on TX queue %d PF# %d VF# %d\n", event, queue, pf_num, vf_num); + ice_report_mdd_event(pf, ICE_MDD_SRC_TX_TCLAN, pf_num, vf_num, + event, queue); wr32(hw, GL_MDET_TX_TCLAN_BY_MAC(hw), U32_MAX); } @@ -1842,6 +1846,8 @@ static void ice_handle_mdd_event(struct ice_pf *pf) if (netif_msg_rx_err(pf)) dev_info(dev, "Malicious Driver Detection event %d on RX queue %d PF# %d VF# %d\n", event, queue, pf_num, vf_num); + ice_report_mdd_event(pf, ICE_MDD_SRC_RX, pf_num, vf_num, event, + queue); wr32(hw, GL_MDET_RX, 0xffffffff); }