From patchwork Wed Oct 9 11:16:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anastasia Kovaleva X-Patchwork-Id: 13828178 Received: from mta-03.yadro.com (mta-03.yadro.com [89.207.88.253]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0C5C191F85; Wed, 9 Oct 2024 11:17:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=89.207.88.253 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472669; cv=none; b=CqFIhtrqzI5P7qLNrdPem5kFtvFrU419zHTfCvLDrRids4ZnmMaQzwflfDUz5ZXrC1BoFog+Zu3Fjmvs4F97Uix5ZXHyWj1s49SwyjDwj1j21adCFBp4uO0ureCi8ct80GtzKDz5chpLwHcdBCuggK2/6GlEQJCqwgTYh2r6mR4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472669; c=relaxed/simple; bh=5aHOeAtBoQ4fy9LIVowu40CIt8+E+nZORgo1JCwLaMs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ZzL5quo8adsC8oGNTfapFjRmP6SIwxSULMWvpiXIda8wptaDJNu2kgptgvEqFBxGASFLBCnS0vffS8dv6P1KZdjNbR2da6p/ag6QPbN0TYOrooPbrLzOnbEKncQuorid4I2UDn6GWjzcWmSkY3gKFeVOW2EwzOOpSRLhN9yLVmg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com; spf=pass smtp.mailfrom=yadro.com; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=MFkklhY7; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=mz5uD94U; arc=none smtp.client-ip=89.207.88.253 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="MFkklhY7"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="mz5uD94U" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-03.yadro.com D3ED9E0010 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1728472661; bh=sB43ziImdTPLgJU6g+q4wMVcdAGDARS25Yq8tvz7Y9s=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=MFkklhY7m6kgOTJesJMrQW2kHYM4UV4AticrjWsJMp0w3IgaSftEryyfrvF3fVucn TpkvPfzu4PQQov2McvHoLy4XgasZOddRnKsDx6Zjvdfh1lTe/xcZJ7/6juDF2MKPTo +KRn6oit0nWpR4yZeW8IclIGfZgXWMwIOOy5BQrLv6tclEECnjJ7c4iLdkZo3Blm27 oyjLGrxyCvoY+2fzC9FBZsY2Om1T5ql5WHkxRFGnGE/viCTvAKYaFiIjjHJbZ6azfH jHwRxcHePCRxPPh85kC5Mw31YwPLZu8FP/oirPIsysCEEjq8KpD2p4aa6BMV03GZCL dQaDUGlfp/Cgg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1728472661; bh=sB43ziImdTPLgJU6g+q4wMVcdAGDARS25Yq8tvz7Y9s=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=mz5uD94UePKwv5w1OgF8sDDxDD+DnzmCYCTFvJfCDIjYhA/+Lq3/cWucyKkycMFmT rFvx6nB+gO+IFYTK1jEsVE0ZBfQXqcr+t0FmTQPIH+6x4GnIQGEnqkKo4w9WEFnnUv pI7o7X1QOVQnqndkMG7q9ph8U/jO98jU2daRKZlS+8cHYST9k9KMoCDtLjGQJO4/u2 ySH3zwrO1TrchELcXXgikxkC7bf9dG3WTbN1IhtOP0j5w4PXiupeIwgyrv8+WETHGb zWD0JXQJccu4X11rfQnWvW0FdNuGca0TGaj/1GQp0nUD724sgto4fib5uMfV6M8CYw /efkezeodzwDg== From: Anastasia Kovaleva To: CC: , , , , , , , , , , , Subject: [PATCH v2 1/3] scsi: qla2xxx: Drop starvation counter on success Date: Wed, 9 Oct 2024 14:16:52 +0300 Message-ID: <20241009111654.4697-2-a.kovaleva@yadro.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009111654.4697-1-a.kovaleva@yadro.com> References: <20241009111654.4697-1-a.kovaleva@yadro.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-Exch-05.corp.yadro.com (172.17.10.109) To T-EXCH-09.corp.yadro.com (172.17.11.59) Long-lived sessions under high load can accumulate a starvation counter, and the current implementation does not allow this counter to be reset during an active session. If HBA sends correct ATIO IOCB, then it has enough resources to process commands and we should not call ISP recovery. Cc: stable@vger.kernel.org Fixes: ead038556f64 ("qla2xxx: Add Dual mode support in the driver") Signed-off-by: Anastasia Kovaleva Reviewed-by: Dmitry Bogdanov --- drivers/scsi/qla2xxx/qla_isr.c | 4 ++++ drivers/scsi/qla2xxx/qla_target.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c index fe98c76e9be3..5234ce0985e0 100644 --- a/drivers/scsi/qla2xxx/qla_isr.c +++ b/drivers/scsi/qla2xxx/qla_isr.c @@ -1959,6 +1959,10 @@ qla2x00_async_event(scsi_qla_host_t *vha, struct rsp_que *rsp, uint16_t *mb) ql_dbg(ql_dbg_async, vha, 0x5091, "Transceiver Removal\n"); break; + case MBA_REJECTED_FCP_CMD: + ql_dbg(ql_dbg_async, vha, 0x5092, "LS_RJT was sent. No resources to process the ELS request.\n"); + break; + default: ql_dbg(ql_dbg_async, vha, 0x5057, "Unknown AEN:%04x %04x %04x %04x\n", diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c index d7551b1443e4..bc6b014eb422 100644 --- a/drivers/scsi/qla2xxx/qla_target.c +++ b/drivers/scsi/qla2xxx/qla_target.c @@ -6826,6 +6826,12 @@ qlt_24xx_process_atio_queue(struct scsi_qla_host *vha, uint8_t ha_locked) qlt_send_term_exchange(ha->base_qpair, NULL, pkt, ha_locked, 0); } else { + /* + * If we get correct ATIO, then HBA had enough memory + * to proceed without reset. + */ + WRITE_ONCE(&vha->hw->exch_starvation, 0); + qlt_24xx_atio_pkt_all_vps(vha, (struct atio_from_isp *)pkt, ha_locked); } From patchwork Wed Oct 9 11:16:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anastasia Kovaleva X-Patchwork-Id: 13828180 Received: from mta-03.yadro.com (mta-03.yadro.com [89.207.88.253]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FF19192D6C; Wed, 9 Oct 2024 11:17:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=89.207.88.253 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472671; cv=none; b=VcqNi03Nv7KY3U3f68ZXVqxj+q5axAx1i79HQkt8GL/u0riC9TgLrDsOYRi4H/vPQ6DfYlmimfho6P8sNcLKU8HqRZYVtlOcx297bl/5uGS5P+NGcWoSdWAaRFnzrdJXMeUQ218UEJqxFsHtM6c+thyea8ScvXyk4WYuJr1dGdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472671; c=relaxed/simple; bh=L3xl7Y45JnJJvfxHv9fI4yzTm9K8ZaJXo9/cApROCTo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eJuq9Eow7ZtrcqNbMKGmQx3b/OBQS2JhUZyVMj5UUc91yugDy7rSIBfp5XXnTb5BL2x9YCYsRSG/rxfy481VNSknqzOXaSjqhMF/nOLtF8HsntJnW/2Rak8qkEpY0ja4HjzHV8NHSjHhnwbraqKMpCfi92rti8RnPNBygbKOIhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com; spf=pass smtp.mailfrom=yadro.com; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=nxht8hmS; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=EUWTp1Jr; arc=none smtp.client-ip=89.207.88.253 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="nxht8hmS"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="EUWTp1Jr" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-03.yadro.com 74B68E0015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1728472667; bh=cWg6uv3EM0obnieXASGAm1ATa7D8mOki8svgc34Ic6c=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=nxht8hmSBW3UmeW9u5Kw7fwlr8wUT02Z5lm+Lm8gIdAmYC29+2MUhPZjvltOQm0qt T3Fowhe6OYmXB5kgHK2Br/wrGEeTHohDV0UVItLCyHPjyL879Jmp/kJvptRhyDziwT QVI9DcrfzVfG08J0PqqDmVVn0JMj82wnFoqDuTNigCyt4TYIgOb1BaUCRbPVH4NRVl h5vAAh1eRANSr9aZk6SP9mHc90K0ZSPSsKfzXUlgrfJAYpZsoH90F2Mocba0GtuyOl MXmKuUVzVdheiCca7+ej7A9SYV0Y6TtptffRAZwNARXtykV3kTJiSBkcm2AYN7UkkP 7IqJI7C60pFOA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1728472667; bh=cWg6uv3EM0obnieXASGAm1ATa7D8mOki8svgc34Ic6c=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=EUWTp1Jrndwijvbu06GzBtfcaG6sthgKuIGVhdXfMvMdt8tzyfmd6J/6KdAjlfrzZ 6jF1Ew6edOmn7nu5FNZcNd3J8U9OhUVtTaOET7H35XST2Rnvkithg6GQ3OanozlFmQ BCRVviZzYtKnSFQGlsCgpk1GFcWRg0Kg/rSKAllnTiFqmmAvsQ9dzXVoRmYUqiviXl E6pD8Gqq3h80Aau2z3z6QqwVuUaQCzp1aCXmvtuIHZFR3nJc1QsxZT+pPMKKOQ4mjz iP4OyYYSamF2POWDeI3ylBYbGw9xALHNMwSIteLd6OgBS8yQwvcdyeWHkF8b4Jsi22 3sbSePMc0amhA== From: Anastasia Kovaleva To: CC: , , , , , , , , , , , Subject: [PATCH v2 2/3] scsi: qla2xxx: Make target send correct LOGO Date: Wed, 9 Oct 2024 14:16:53 +0300 Message-ID: <20241009111654.4697-3-a.kovaleva@yadro.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009111654.4697-1-a.kovaleva@yadro.com> References: <20241009111654.4697-1-a.kovaleva@yadro.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-06.corp.yadro.com (172.17.10.110) To T-EXCH-09.corp.yadro.com (172.17.11.59) Upon removing the ACL from the target, it sends a LOGO command to the initiator to break the connection. But HBA fills port_name and port_id of the LOGO command with all zeroes, which is not valid. The initiator sends a reject for this command, but it is not being processed on the target, since it assumes LOGO can never fail. This leaves a system in a state where the initiator thinks it is still logged in to the target and can send commands to it, but the target ignores all incoming commands from this initiator. If, in such a situation, the initiator sends some command (e.g. during a scan), after not receiving a response for a timeout duration, it sends ABORT for the command. After a timeout on receiving an ABORT response, the initiator sends LOGO to the target. Only after that, the initiator can successfully relogin to the target and restore the connection. In the end, this whole situation hangs the system for approximately a minute. By default, the driver sends a LOGO command to HBA filling only port_id, expecting HBA to match port_id with the correct port_name from it's internal table. HBA doesn't do that, instead filling these fields with all zeroes. This patch makes the driver send a LOGO command to HBA with port_name and port_id in the I/O PARMETER fields. HBA then copies these values to corresponding fields in the LOGO command frame. Signed-off-by: Anastasia Kovaleva Reviewed-by: Dmitry Bogdanov Reviewed-by: Hannes Reinecke --- drivers/scsi/qla2xxx/qla_iocb.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c index 0b41e8a06602..90026fca14dc 100644 --- a/drivers/scsi/qla2xxx/qla_iocb.c +++ b/drivers/scsi/qla2xxx/qla_iocb.c @@ -2486,6 +2486,17 @@ qla24xx_logout_iocb(srb_t *sp, struct logio_entry_24xx *logio) logio->port_id[1] = sp->fcport->d_id.b.area; logio->port_id[2] = sp->fcport->d_id.b.domain; logio->vp_index = sp->vha->vp_idx; + logio->io_parameter[0] = cpu_to_le32(sp->vha->d_id.b.al_pa | + sp->vha->d_id.b.area << 8 | + sp->vha->d_id.b.domain << 16); + logio->io_parameter[1] = cpu_to_le32(sp->vha->port_name[3] | + sp->vha->port_name[2] << 8 | + sp->vha->port_name[1] << 16 | + sp->vha->port_name[0] << 24); + logio->io_parameter[2] = cpu_to_le32(sp->vha->port_name[7] | + sp->vha->port_name[6] << 8 | + sp->vha->port_name[5] << 16 | + sp->vha->port_name[4] << 24); } static void From patchwork Wed Oct 9 11:16:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anastasia Kovaleva X-Patchwork-Id: 13828179 Received: from mta-03.yadro.com (mta-03.yadro.com [89.207.88.253]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FF6A192D6D; Wed, 9 Oct 2024 11:17:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=89.207.88.253 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472671; cv=none; b=tBv0E7F0oijCzOm765SIdqSdSw63BOuG+efn0lcUtLNslFGSxhlOi8ip4fmiHf+aNsSttR3ZGK2nkjZqmjY6qcPNmT98V35pSYEAC4iIoStbGs6cFY7SlERNSaOeKcBqhn7BwQAg7AISDbbVxbjybXwJ9yTfJcQjr9HuLr0cFOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728472671; c=relaxed/simple; bh=KIxBJHEXLmkaHDuNyoNCEDlOFyVyaazsAK0OclfC2qM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WZHEx8i13izMSkdU9c58/vPyG7cxSwsxRiYPgdAWUXw2eKQArF2v9XiM7HaW/l9n0IdnThf6sEprt8+KjH8pNHWTH/W87T6zYUwEuENC0Meo99Kx3AATg438x+0BKiH9x6sQktJsDw7yKJcxtPoanI5ZtKrWYxfEn7yN550SUu4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com; spf=pass smtp.mailfrom=yadro.com; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=k4YL0XWN; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b=rBMh1To3; arc=none smtp.client-ip=89.207.88.253 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="k4YL0XWN"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="rBMh1To3" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-03.yadro.com 09BC6E0017 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1728472668; bh=y20oOosez36zWrQ1+4E1TZpFUVdNhnGMc9YF2n3uVr4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=k4YL0XWN9dbw/6xMrJidRW9nu8U1a387extJQfFm3H4pCQ3xBzbdN3q+AlFbsNi1s mVn2aL5KyrIMRDMSJ6E0wlAAYk20DGQix20Oqpsu001gzPuQTG7cYy9LXtH2ldrwT6 kQV2jXTcwFnaG3fv3T+kRvpzrvF+JYYTpCJ5GsMb5PHvNs9SQP7Pn/41Ogvr6lezsL WvLtrJJBp5oLd0eMKGr82B3xu65MRIWrcO1bvDBOWXpc51OMqEErn7TfRpjmANHjd8 p2xWpJQWQ2LVOuOKotZIDj43J/Lgiq7amYbQaQ5MG2vSLver7gSj3j/wg4bmes3rfy 6yIXU+rUXgpNg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1728472668; bh=y20oOosez36zWrQ1+4E1TZpFUVdNhnGMc9YF2n3uVr4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=rBMh1To3soGpmcdyhDIq7lpW/uKlfBjoww6e2GYkRzvOpe7h5azwm7RwskNui5CxQ 9g8vxuJHhsPXJIw9vAfOB6VG/qVKxAuKsd5lJB7JBl6CXfunPARwl3M+GY8iznBXiE MDr6Nkhx0JvD1xxsBlIuahirNdGRN3tatO3ygoPtKHlvp6r6hW9moRBkMhe7f1dg3o wE0wwW9EznuT1r4s4qWuPRvZ1nG05Ulx01E8jKwqwj+79TtANLTH3fpWE2rmmmkWdU YksSemKlvUWmWlDvUMAw9n4hqO4O5n/ieqbqnacqZeGKk9vXOYsb2o2YDQY5ppzrxw GTO2a5wY8Jf5w== From: Anastasia Kovaleva To: CC: , , , , , , , , , , , Subject: [PATCH v2 3/3] scsi: qla2xxx: Remove incorrect trap Date: Wed, 9 Oct 2024 14:16:54 +0300 Message-ID: <20241009111654.4697-4-a.kovaleva@yadro.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20241009111654.4697-1-a.kovaleva@yadro.com> References: <20241009111654.4697-1-a.kovaleva@yadro.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-10.corp.yadro.com (172.17.11.60) To T-EXCH-09.corp.yadro.com (172.17.11.59) This BUG_ON() is triggered when there is no fc_port with a certain loop ID in the scsi host vp_fcports list, but there is one in lport_loopid_map. As these two data structures do not change simultaneously and atomically, such a trap is invalid. Cc: stable@vger.kernel.org Fixes: 726b85487067 ("qla2xxx: Add framework for async fabric discovery") Signed-off-by: Anastasia Kovaleva Reviewed-by: Dmitry Bogdanov Reviewed-by: Hannes Reinecke --- drivers/scsi/qla2xxx/qla_target.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c index bc6b014eb422..492fc1627354 100644 --- a/drivers/scsi/qla2xxx/qla_target.c +++ b/drivers/scsi/qla2xxx/qla_target.c @@ -5190,15 +5190,7 @@ static int qlt_24xx_handle_els(struct scsi_qla_host *vha, ql_dbg(ql_dbg_disc, vha, 0x20fc, "%s: logo %llx res %d sess %p ", __func__, wwn, res, sess); - if (res == 0) { - /* - * cmd went upper layer, look for qlt_xmit_tm_rsp() - * for LOGO_ACK & sess delete - */ - BUG_ON(!sess); - res = 0; - } else { - /* cmd did not go to upper layer. */ + if (res) { if (sess) { qlt_schedule_sess_for_deletion(sess); res = 0;