From patchwork Thu Oct 14 18:19:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 12559215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B047C4332F for ; Thu, 14 Oct 2021 18:21:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71F4B611C5 for ; Thu, 14 Oct 2021 18:21:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232336AbhJNSXu (ORCPT ); Thu, 14 Oct 2021 14:23:50 -0400 Received: from mga04.intel.com ([192.55.52.120]:62163 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230422AbhJNSXu (ORCPT ); Thu, 14 Oct 2021 14:23:50 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="226531105" X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="226531105" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2021 11:21:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="717815812" Received: from anguy11-desk2.jf.intel.com ([10.166.244.147]) by fmsmga005.fm.intel.com with ESMTP; 14 Oct 2021 11:21:43 -0700 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org Cc: Brett Creeley , netdev@vger.kernel.org, anthony.l.nguyen@intel.com, linux-rdma@vger.kernel.org, shiraz.saleem@intel.com, Jerzy Wiktor Jurkowski Subject: [PATCH net 1/4] ice: Fix failure to re-add LAN/RDMA Tx queues Date: Thu, 14 Oct 2021 11:19:50 -0700 Message-Id: <20211014181953.3538330-2-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> References: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Brett Creeley Currently if the VSI is rebuilt/removed and the RDMA PF driver is active the RDMA Tx queue scheduler node configuration will not be cleaned up. This will cause the rebuild/re-add of the VSI to fail due to the software structures not being correctly cleaned up for the VSI index. Fix this by always calling ice_rm_vsi_rdma_cfg() for all VSI. If there are no RDMA scheduler nodes created, then there is no harm in calling ice_rm_vsi_rdma_cfg(). This change applies to all VSI types, so if RDMA support is added for other VSI types they will also get this change. Fixes: 348048e724a0 ("ice: Implement iidc operations") Signed-off-by: Brett Creeley Tested-by: Jerzy Wiktor Jurkowski Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/ice/ice_lib.c | 9 +++++++++ drivers/net/ethernet/intel/ice/ice_sched.c | 13 +++++++++++++ drivers/net/ethernet/intel/ice/ice_sched.h | 1 + 3 files changed, 23 insertions(+) diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c index dde9802c6c72..b718e196af2a 100644 --- a/drivers/net/ethernet/intel/ice/ice_lib.c +++ b/drivers/net/ethernet/intel/ice/ice_lib.c @@ -2841,6 +2841,7 @@ void ice_napi_del(struct ice_vsi *vsi) */ int ice_vsi_release(struct ice_vsi *vsi) { + enum ice_status err; struct ice_pf *pf; if (!vsi->back) @@ -2912,6 +2913,10 @@ int ice_vsi_release(struct ice_vsi *vsi) ice_fltr_remove_all(vsi); ice_rm_vsi_lan_cfg(vsi->port_info, vsi->idx); + err = ice_rm_vsi_rdma_cfg(vsi->port_info, vsi->idx); + if (err) + dev_err(ice_pf_to_dev(vsi->back), "Failed to remove RDMA scheduler config for VSI %u, err %d\n", + vsi->vsi_num, err); ice_vsi_delete(vsi); ice_vsi_free_q_vectors(vsi); @@ -3092,6 +3097,10 @@ int ice_vsi_rebuild(struct ice_vsi *vsi, bool init_vsi) prev_num_q_vectors = ice_vsi_rebuild_get_coalesce(vsi, coalesce); ice_rm_vsi_lan_cfg(vsi->port_info, vsi->idx); + ret = ice_rm_vsi_rdma_cfg(vsi->port_info, vsi->idx); + if (ret) + dev_err(ice_pf_to_dev(vsi->back), "Failed to remove RDMA scheduler config for VSI %u, err %d\n", + vsi->vsi_num, ret); ice_vsi_free_q_vectors(vsi); /* SR-IOV determines needed MSIX resources all at once instead of per diff --git a/drivers/net/ethernet/intel/ice/ice_sched.c b/drivers/net/ethernet/intel/ice/ice_sched.c index 9f07b6641705..2d9b10277186 100644 --- a/drivers/net/ethernet/intel/ice/ice_sched.c +++ b/drivers/net/ethernet/intel/ice/ice_sched.c @@ -2070,6 +2070,19 @@ enum ice_status ice_rm_vsi_lan_cfg(struct ice_port_info *pi, u16 vsi_handle) return ice_sched_rm_vsi_cfg(pi, vsi_handle, ICE_SCHED_NODE_OWNER_LAN); } +/** + * ice_rm_vsi_rdma_cfg - remove VSI and its RDMA children nodes + * @pi: port information structure + * @vsi_handle: software VSI handle + * + * This function clears the VSI and its RDMA children nodes from scheduler tree + * for all TCs. + */ +enum ice_status ice_rm_vsi_rdma_cfg(struct ice_port_info *pi, u16 vsi_handle) +{ + return ice_sched_rm_vsi_cfg(pi, vsi_handle, ICE_SCHED_NODE_OWNER_RDMA); +} + /** * ice_get_agg_info - get the aggregator ID * @hw: pointer to the hardware structure diff --git a/drivers/net/ethernet/intel/ice/ice_sched.h b/drivers/net/ethernet/intel/ice/ice_sched.h index 9beef8f0ec76..fdf7a5882f07 100644 --- a/drivers/net/ethernet/intel/ice/ice_sched.h +++ b/drivers/net/ethernet/intel/ice/ice_sched.h @@ -89,6 +89,7 @@ enum ice_status ice_sched_cfg_vsi(struct ice_port_info *pi, u16 vsi_handle, u8 tc, u16 maxqs, u8 owner, bool enable); enum ice_status ice_rm_vsi_lan_cfg(struct ice_port_info *pi, u16 vsi_handle); +enum ice_status ice_rm_vsi_rdma_cfg(struct ice_port_info *pi, u16 vsi_handle); /* Tx scheduler rate limiter functions */ enum ice_status From patchwork Thu Oct 14 18:19:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 12559213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43D05C43217 for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CED7611B0 for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231148AbhJNSXv (ORCPT ); Thu, 14 Oct 2021 14:23:51 -0400 Received: from mga04.intel.com ([192.55.52.120]:62163 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232425AbhJNSXu (ORCPT ); Thu, 14 Oct 2021 14:23:50 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="226531107" X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="226531107" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2021 11:21:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="717815815" Received: from anguy11-desk2.jf.intel.com ([10.166.244.147]) by fmsmga005.fm.intel.com with ESMTP; 14 Oct 2021 11:21:44 -0700 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org Cc: Dave Ertman , netdev@vger.kernel.org, anthony.l.nguyen@intel.com, linux-rdma@vger.kernel.org, shiraz.saleem@intel.com, Jun Miao , Jesse Brandeburg Subject: [PATCH net 2/4] ice: Avoid crash from unnecessary IDA free Date: Thu, 14 Oct 2021 11:19:51 -0700 Message-Id: <20211014181953.3538330-3-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> References: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Dave Ertman In the remove path, there is an attempt to free the aux_idx IDA whether it was allocated or not. This can potentially cause a crash when unloading the driver on systems that do not initialize support for RDMA. But, this free cannot be gated by the status bit for RDMA, since it is allocated if the driver detects support for RDMA at probe time, but the driver can enter into a state where RDMA is not supported after the IDA has been allocated at probe time and this would lead to a memory leak. Initialize aux_idx to an invalid value and check for a valid value when unloading to determine if an IDA free is necessary. Fixes: d25a0fc41c1f ("ice: Initialize RDMA support") Reported-by: Jun Miao Signed-off-by: Dave Ertman Tested-by: Jesse Brandeburg Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/ice/ice_main.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index 0d6c143f6653..94037881bfd8 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -4224,6 +4224,9 @@ ice_probe(struct pci_dev *pdev, const struct pci_device_id __always_unused *ent) if (!pf) return -ENOMEM; + /* initialize Auxiliary index to invalid value */ + pf->aux_idx = -1; + /* set up for high or low DMA */ err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); if (err) @@ -4615,7 +4618,8 @@ static void ice_remove(struct pci_dev *pdev) ice_aq_cancel_waiting_tasks(pf); ice_unplug_aux_dev(pf); - ida_free(&ice_aux_ida, pf->aux_idx); + if (pf->aux_idx >= 0) + ida_free(&ice_aux_ida, pf->aux_idx); set_bit(ICE_DOWN, pf->state); mutex_destroy(&(&pf->hw)->fdir_fltr_lock); From patchwork Thu Oct 14 18:19:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 12559217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66ACFC4167B for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 415E2611CC for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233908AbhJNSXv (ORCPT ); Thu, 14 Oct 2021 14:23:51 -0400 Received: from mga04.intel.com ([192.55.52.120]:62163 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230422AbhJNSXv (ORCPT ); Thu, 14 Oct 2021 14:23:51 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="226531109" X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="226531109" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2021 11:21:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="717815819" Received: from anguy11-desk2.jf.intel.com ([10.166.244.147]) by fmsmga005.fm.intel.com with ESMTP; 14 Oct 2021 11:21:44 -0700 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org Cc: Michal Swiatkowski , netdev@vger.kernel.org, anthony.l.nguyen@intel.com, linux-rdma@vger.kernel.org, shiraz.saleem@intel.com, Gurucharan G Subject: [PATCH net 3/4] ice: fix getting UDP tunnel entry Date: Thu, 14 Oct 2021 11:19:52 -0700 Message-Id: <20211014181953.3538330-4-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> References: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Michal Swiatkowski Correct parameters order in call to ice_tunnel_idx_to_entry function. Entry in sparse port table is correct when the idx is 0. For idx 1 one correct entry should be skipped, for idx 2 two of them should be skipped etc. Change if condition to be true when idx is 0, which means that previous valid entry of this tunnel type were skipped. Fixes: b20e6c17c468 ("ice: convert to new udp_tunnel infrastructure") Signed-off-by: Michal Swiatkowski Tested-by: Gurucharan G Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/ice/ice_flex_pipe.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/ice/ice_flex_pipe.c b/drivers/net/ethernet/intel/ice/ice_flex_pipe.c index 06ac9badee77..1ac96dc66d0d 100644 --- a/drivers/net/ethernet/intel/ice/ice_flex_pipe.c +++ b/drivers/net/ethernet/intel/ice/ice_flex_pipe.c @@ -1668,7 +1668,7 @@ static u16 ice_tunnel_idx_to_entry(struct ice_hw *hw, enum ice_tunnel_type type, for (i = 0; i < hw->tnl.count && i < ICE_TUNNEL_MAX_ENTRIES; i++) if (hw->tnl.tbl[i].valid && hw->tnl.tbl[i].type == type && - idx--) + idx-- == 0) return i; WARN_ON_ONCE(1); @@ -1828,7 +1828,7 @@ int ice_udp_tunnel_set_port(struct net_device *netdev, unsigned int table, u16 index; tnl_type = ti->type == UDP_TUNNEL_TYPE_VXLAN ? TNL_VXLAN : TNL_GENEVE; - index = ice_tunnel_idx_to_entry(&pf->hw, idx, tnl_type); + index = ice_tunnel_idx_to_entry(&pf->hw, tnl_type, idx); status = ice_create_tunnel(&pf->hw, index, tnl_type, ntohs(ti->port)); if (status) { From patchwork Thu Oct 14 18:19:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 12559219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF5B4C4321E for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BB45D611C5 for ; Thu, 14 Oct 2021 18:21:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232425AbhJNSXw (ORCPT ); Thu, 14 Oct 2021 14:23:52 -0400 Received: from mga04.intel.com ([192.55.52.120]:62163 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233901AbhJNSXv (ORCPT ); Thu, 14 Oct 2021 14:23:51 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="226531110" X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="226531110" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2021 11:21:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,373,1624345200"; d="scan'208";a="717815822" Received: from anguy11-desk2.jf.intel.com ([10.166.244.147]) by fmsmga005.fm.intel.com with ESMTP; 14 Oct 2021 11:21:44 -0700 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org Cc: Brett Creeley , netdev@vger.kernel.org, anthony.l.nguyen@intel.com, linux-rdma@vger.kernel.org, shiraz.saleem@intel.com, Gurucharan G Subject: [PATCH net 4/4] ice: Print the api_patch as part of the fw.mgmt.api Date: Thu, 14 Oct 2021 11:19:53 -0700 Message-Id: <20211014181953.3538330-5-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> References: <20211014181953.3538330-1-anthony.l.nguyen@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Brett Creeley Currently when a user uses "devlink dev info", the fw.mgmt.api will be the major.minor numbers as shown below: devlink dev info pci/0000:3b:00.0 pci/0000:3b:00.0: driver ice serial_number 00-01-00-ff-ff-00-00-00 versions: fixed: board.id K91258-000 running: fw.mgmt 6.1.2 fw.mgmt.api 1.7 <--- No patch number included fw.mgmt.build 0xd75e7d06 fw.mgmt.srev 5 fw.undi 1.2992.0 fw.undi.srev 5 fw.psid.api 3.10 fw.bundle_id 0x800085cc fw.app.name ICE OS Default Package fw.app 1.3.27.0 fw.app.bundle_id 0xc0000001 fw.netlist 3.10.2000-3.1e.0 fw.netlist.build 0x2a76e110 stored: fw.mgmt.srev 5 fw.undi 1.2992.0 fw.undi.srev 5 fw.psid.api 3.10 fw.bundle_id 0x800085cc fw.netlist 3.10.2000-3.1e.0 fw.netlist.build 0x2a76e110 There are many features in the driver that depend on the major, minor, and patch version of the FW. Without the patch number in the output for fw.mgmt.api debugging issues related to the FW API version is difficult. Also, using major.minor.patch aligns with the existing firmware version which uses a 3 digit value. Fix this by making the fw.mgmt.api print the major.minor.patch versions. Shown below is the result: devlink dev info pci/0000:3b:00.0 pci/0000:3b:00.0: driver ice serial_number 00-01-00-ff-ff-00-00-00 versions: fixed: board.id K91258-000 running: fw.mgmt 6.1.2 fw.mgmt.api 1.7.9 <--- patch number included fw.mgmt.build 0xd75e7d06 fw.mgmt.srev 5 fw.undi 1.2992.0 fw.undi.srev 5 fw.psid.api 3.10 fw.bundle_id 0x800085cc fw.app.name ICE OS Default Package fw.app 1.3.27.0 fw.app.bundle_id 0xc0000001 fw.netlist 3.10.2000-3.1e.0 fw.netlist.build 0x2a76e110 stored: fw.mgmt.srev 5 fw.undi 1.2992.0 fw.undi.srev 5 fw.psid.api 3.10 fw.bundle_id 0x800085cc fw.netlist 3.10.2000-3.1e.0 fw.netlist.build 0x2a76e110 Fixes: ff2e5c700e08 ("ice: add basic handler for devlink .info_get") Signed-off-by: Brett Creeley Tested-by: Gurucharan G Signed-off-by: Tony Nguyen --- Documentation/networking/devlink/ice.rst | 9 +++++---- drivers/net/ethernet/intel/ice/ice_devlink.c | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Documentation/networking/devlink/ice.rst b/Documentation/networking/devlink/ice.rst index a432dc419fa4..5d97cee9457b 100644 --- a/Documentation/networking/devlink/ice.rst +++ b/Documentation/networking/devlink/ice.rst @@ -30,10 +30,11 @@ The ``ice`` driver reports the following versions PHY, link, etc. * - ``fw.mgmt.api`` - running - - 1.5 - - 2-digit version number of the API exported over the AdminQ by the - management firmware. Used by the driver to identify what commands - are supported. + - 1.5.1 + - 3-digit version number (major.minor.patch) of the API exported over + the AdminQ by the management firmware. Used by the driver to + identify what commands are supported. Historical versions of the + kernel only displayed a 2-digit version number (major.minor). * - ``fw.mgmt.build`` - running - 0x305d955f diff --git a/drivers/net/ethernet/intel/ice/ice_devlink.c b/drivers/net/ethernet/intel/ice/ice_devlink.c index 14afce82ef63..da7288bdc9a3 100644 --- a/drivers/net/ethernet/intel/ice/ice_devlink.c +++ b/drivers/net/ethernet/intel/ice/ice_devlink.c @@ -63,7 +63,8 @@ static int ice_info_fw_api(struct ice_pf *pf, struct ice_info_ctx *ctx) { struct ice_hw *hw = &pf->hw; - snprintf(ctx->buf, sizeof(ctx->buf), "%u.%u", hw->api_maj_ver, hw->api_min_ver); + snprintf(ctx->buf, sizeof(ctx->buf), "%u.%u.%u", hw->api_maj_ver, + hw->api_min_ver, hw->api_patch); return 0; }