From patchwork Thu Dec 7 21:59:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rafael Tinoco X-Patchwork-Id: 10101167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 410AD60329 for ; Thu, 7 Dec 2017 21:59:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33F2E209CE for ; Thu, 7 Dec 2017 21:59:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 28CEB2876C; Thu, 7 Dec 2017 21:59:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEEBE209CE for ; Thu, 7 Dec 2017 21:59:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751001AbdLGV7W (ORCPT ); Thu, 7 Dec 2017 16:59:22 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:59833 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750967AbdLGV7V (ORCPT ); Thu, 7 Dec 2017 16:59:21 -0500 Received: from mail-qt0-f200.google.com ([209.85.216.200]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1eN4CO-0007z1-82 for linux-scsi@vger.kernel.org; Thu, 07 Dec 2017 21:59:20 +0000 Received: by mail-qt0-f200.google.com with SMTP id k23so9847536qtc.14 for ; Thu, 07 Dec 2017 13:59:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=LK3aygqGcmkr/XflFFbVfsW4OZ58gpIXLWpWbmZlu+c=; b=Cb4RoCI87Y1+zyr1ZLQLqQAEE2rSkxEC76eVyzVi9FekObzHxqeP6ccCld0whGuHwF TPNsiiSocK0R0R1QXw4ENxUs/1HNV6u/vyUBq2dZ6HToM6aU3xrFJW23nGSgenOAp6aX LHUCR3PswDceQ6qU/3XWn9b/ktLtgmxVE9Q7SQIv7ELFsVZ7TimKTQVjEWaWLhUHsu8x puxlqOYnt2X07EYNUKgwg1gvvppUumL6WW2uuxZEi4HkiEWGLDV3j0p5y+9aG3gfgq2Z rzTVEGB2/Zf0T/CGjktn0yBFZPxV4bGARn2GXXQlJvIUfCPvoldVUqF9LjglVQF5SywR EqtQ== X-Gm-Message-State: AKGB3mLP44sYp8MMBWsNhtmHs7GPuWIMu5tt61/Vuvq2cX3rr+nkFCXV a5U/eKhnvQ7bF2AznrdtH60gOqkXbTi7um+dHGwqacoWqyTW7RKBO6rsvWeRAtLwFeNQ+/+N8E6 /pYo1crQsplhgK1kmeFJnUkwERrl/9qLoZlKwImQ= X-Received: by 10.200.37.125 with SMTP id 58mr11353647qtn.138.1512683958590; Thu, 07 Dec 2017 13:59:18 -0800 (PST) X-Google-Smtp-Source: AGs4zMYs39WIqhesY4slsGI1LRee37q+dzn/XUnx83I8Rq0uawobBx0LVemd6z6BIjDDTwKePqcy2A== X-Received: by 10.200.37.125 with SMTP id 58mr11353626qtn.138.1512683958227; Thu, 07 Dec 2017 13:59:18 -0800 (PST) Received: from localhost.localdomain ([177.215.81.117]) by smtp.gmail.com with ESMTPSA id c50sm4181966qtk.45.2017.12.07.13.59.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Dec 2017 13:59:17 -0800 (PST) From: Rafael David Tinoco To: Lee Duncan , Chris Leech Cc: open-iscsi@googlegroups.com, linux-scsi@vger.kernel.org, Jay Vosburgh , Daniel Axtens , Dragan Stancevic Subject: [PATCH] scsi: libiscsi: Allow sd_shutdown on bad transport Date: Thu, 7 Dec 2017 19:59:13 -0200 Message-Id: <1512683953-5959-1-git-send-email-rafael.tinoco@canonical.com> X-Mailer: git-send-email 2.7.4 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If, for any reason, userland shuts down iscsi transport interfaces before proper logouts - like when logging in to LUNs manually, without logging out on server shutdown, or when automated scripts can't umount/logout from logged LUNs - kernel will hang forever on its sd_sync_cache() logic, after issuing the SYNCHRONIZE_CACHE cmd to all still existent paths. PID: 1 TASK: ffff8801a69b8000 CPU: 1 COMMAND: "systemd-shutdow" #0 [ffff8801a69c3a30] __schedule at ffffffff8183e9ee #1 [ffff8801a69c3a80] schedule at ffffffff8183f0d5 #2 [ffff8801a69c3a98] schedule_timeout at ffffffff81842199 #3 [ffff8801a69c3b40] io_schedule_timeout at ffffffff8183e604 #4 [ffff8801a69c3b70] wait_for_completion_io_timeout at ffffffff8183fc6c #5 [ffff8801a69c3bd0] blk_execute_rq at ffffffff813cfe10 #6 [ffff8801a69c3c88] scsi_execute at ffffffff815c3fc7 #7 [ffff8801a69c3cc8] scsi_execute_req_flags at ffffffff815c60fe #8 [ffff8801a69c3d30] sd_sync_cache at ffffffff815d37d7 #9 [ffff8801a69c3da8] sd_shutdown at ffffffff815d3c3c This happens because iscsi_eh_cmd_timed_out(), the transport layer timeout helper, would tell the queue timeout function (scsi_times_out) to reset the request timer over and over, until the session state is back to logged in state. Unfortunately, during server shutdown, this might never happen again. Other option would be "not to handle" the issue in the transport layer. That would trigger the error handler logic, which would also need the session state to be logged in again. Best option, for such case, is to tell upper layers that the command was handled during the transport layer error handler helper, marking it as DID_NO_CONNECT, which will allow completion and inform about the problem. After the session was marked as ISCSI_STATE_FAILED, due to the first timeout during the server shutdown phase, all subsequent cmds will fail to be queued, allowing upper logic to fail faster. Signed-off-by: Rafael David Tinoco Reviewed-by: Lee Duncan --- drivers/scsi/libiscsi.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c index 9c50d2d9f27c..785d1c55d152 100644 --- a/drivers/scsi/libiscsi.c +++ b/drivers/scsi/libiscsi.c @@ -1696,6 +1696,15 @@ int iscsi_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *sc) */ switch (session->state) { case ISCSI_STATE_FAILED: + /* + * cmds should fail during shutdown, if the session + * state is bad, allowing completion to happen + */ + if (unlikely(system_state != SYSTEM_RUNNING)) { + reason = FAILURE_SESSION_FAILED; + sc->result = DID_NO_CONNECT << 16; + break; + } case ISCSI_STATE_IN_RECOVERY: reason = FAILURE_SESSION_IN_RECOVERY; sc->result = DID_IMM_RETRY << 16; @@ -1978,6 +1987,19 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc) } if (session->state != ISCSI_STATE_LOGGED_IN) { + /* + * During shutdown, if session is prematurely disconnected, + * recovery won't happen and there will be hung cmds. Not + * handling cmds would trigger EH, also bad in this case. + * Instead, handle cmd, allow completion to happen and let + * upper layer to deal with the result. + */ + if (unlikely(system_state != SYSTEM_RUNNING)) { + sc->result = DID_NO_CONNECT << 16; + ISCSI_DBG_EH(session, "sc on shutdown, handled\n"); + rc = BLK_EH_HANDLED; + goto done; + } /* * We are probably in the middle of iscsi recovery so let * that complete and handle the error. @@ -2082,7 +2104,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd *sc) task->last_timeout = jiffies; spin_unlock(&session->frwd_lock); ISCSI_DBG_EH(session, "return %s\n", rc == BLK_EH_RESET_TIMER ? - "timer reset" : "nh"); + "timer reset" : "shutdown or nh"); return rc; } EXPORT_SYMBOL_GPL(iscsi_eh_cmd_timed_out);