diff mbox

[v2] scsi: mpt3sas: fix oops in error handlers after shutdown/unload

Message ID 1517523386-9868-1-git-send-email-mauricfo@linux.vnet.ibm.com (mailing list archive)
State Changes Requested
Headers show

Commit Message

Mauricio Faria de Oliveira Feb. 1, 2018, 10:16 p.m. UTC 
This patch adds checks for 'ioc->remove_host' in the SCSI error
handlers, so not to access pointers/resources potentially freed
in the PCI shutdown/module unload path.  The error handlers may
be invoked after shutdown/unload, depending on other components.

This problem was observed with kexec on a system with a mpt3sas
based adapter and an infiniband adapter which takes long enough
to shutdown:

The mpt3sas driver finished shutting down / disabled interrupt
handling, thus some commands have not finished and timed out.

Since the system was still running (waiting for the infiniband
adapter to shutdown), the scsi error handler for task abort of
mpt3sas was invoked, and hit an oops -- either in scsih_abort()
because 'ioc->scsi_lookup' was NULL (without commit dbec4c9040ed
("scsi: mpt3sas: lockless command submission")), or later up in
scsih_host_reset() (with or without that commit), because it
eventually called mpt3sas_base_get_iocstate().

After that commit, the oops in scsih_abort() does not occur
anymore (_scsih_scsi_lookup_find_by_scmd() is no longer called),
but that commit is too big and out of the scope of linux-stable,
where this patch might help, so still go for the changes.

Also, this might help to prevent similar errors in the future,
in case code changes and possibly tries to access freed stuff.

Note the fix in scsih_host_reset() is still important anyway.

Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
v2:
 - rebase on top of mkp/scsi.git's fixes branch
 - insert check for 'ioc->remove_host' in existing
   checks which already set DID_NO_CONNECT instead
   of duplicating that code. (helps with backports)
 - update commit message about that commit.

 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Comments

Mauricio Faria de Oliveira Feb. 1, 2018, 10:27 p.m. UTC | #1 
The test-case results with PATCH v2.

scsih_abort()
=============

Without patch:

    [  362.669743] setting logging_level(0x10000000)
    [  362.705074] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fd4f2bd40)
    [  363.956579] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  363.956844] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  363.957147] mpt3sas_cm0: sending diag reset !!
    [  365.073568] mpt3sas_cm0: diag reset: SUCCESS
    [  365.090316] mpt3sas_cm0: sleep on shutdown
    [  366.120209] mpt3sas_cm0: sleep on shutdown
    ...
    [  373.419954] sd 16:0:1:0: attempting task abort! scmd(c000001fd4f2bd40)
    ...
    [  373.420256] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.420300] sd 16:0:1:0: task abort: FAILED scmd(c000001fd4f2bd40)
    [  373.459961] sd 16:0:1:0: attempting device reset! scmd(c000001fd4f2bd40)
    ...
    [  373.460271] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.460315] sd 16:0:1:0: device reset: FAILED scmd(c000001fd4f2bd40)
    [  373.460362] scsi target16:0:1: attempting target reset! scmd(c000001fd4f2bd40)
    ...
    [  373.460628] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  373.460673] scsi target16:0:1: target reset: FAILED scmd(c000001fd4f2bd40)
    [  373.460720] mpt3sas_cm0: attempting host reset! scmd(c000001fd4f2bd40)
    [  373.460764] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  373.460821] Unable to handle kernel paging request for data at address 0xd0003800817d0000
    [  373.460876] Faulting instruction address: 0xd000000017b169b8
    [  373.460921] Oops: Kernel access of bad area, sig: 11 [#1]
    ...
    [  373.462028] NIP [d000000017b169b8] mpt3sas_base_get_iocstate+0x38/0xb0 [mpt3sas]
    [  373.462085] LR [d000000017b1a3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  373.462138] Call Trace:
    [  373.462160] [c000001fdf5f7ab0] [d000000017b1a3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  373.462225] [c000001fdf5f7b80] [d000000017b20f6c] scsih_host_reset+0x7c/0x100 [mpt3sas]
    [  373.462281] [c000001fdf5f7c00] [c00000000076f34c] scsi_try_host_reset+0x5c/0x140
    [  373.462335] [c000001fdf5f7c40] [c00000000077107c] scsi_eh_ready_devs+0x73c/0x960
    [  373.462390] [c000001fdf5f7d10] [c000000000772800] scsi_error_handler+0x4c0/0x4e0
    [  373.462444] [c000001fdf5f7dc0] [c000000000107ed4] kthread+0x164/0x1b0
    [  373.462490] [c000001fdf5f7e30] [c00000000000b5a0] ret_from_kernel_thread+0x5c/0xbc
    [  373.468473] Instruction dump:

With patch:

    [  332.994654] setting logging_level(0x10000000)
    [  333.024246] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000003c97348140)
    [  334.232041] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  334.232324] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  334.232598] mpt3sas_cm0: sending diag reset !!
    [  335.352533] mpt3sas_cm0: diag reset: SUCCESS
    [  335.372075] mpt3sas_cm0: sleep on shutdown
    [  336.440544] mpt3sas_cm0: sleep on shutdown
    ...
    [  343.100179] sd 16:0:1:0: attempting task abort! scmd(c000003c97348140)
    ...
    [  343.100488] sd 16:0:1:0: device been deleted! scmd(c000003c97348140)
    [  343.100532] sd 16:0:1:0: task abort: SUCCESS scmd(c000003c97348140)
    [  343.140185] sd 16:0:1:0: [sdf] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [  343.140275] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  343.140338] print_req_error: I/O error, dev sdf, sector 0


scsih_dev/target/host_reset()
===========================

Without patch:

    [  101.077946] setting logging_level(0x11000000)
    [  101.110029] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fb7a2dd40)
    [  102.356108] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [  102.356396] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [  102.356647] mpt3sas_cm0: sending diag reset !!
    [  103.476380] mpt3sas_cm0: diag reset: SUCCESS
    [  103.492696] mpt3sas_cm0: sleep on shutdown
    [  104.512914] mpt3sas_cm0: sleep on shutdown
    ...
    [  111.732912] sd 16:0:1:0: attempting task abort! scmd(c000001fb7a2dd40)
    ...
    [  111.733202] mpt3sas_cm0: fail task abort scmd(c000001fb7a2dd40)
    [  111.733246] sd 16:0:1:0: task abort: FAILED scmd(c000001fb7a2dd40)
    [  111.772919] sd 16:0:1:0: attempting device reset! scmd(c000001fb7a2dd40)
    ...
    [  111.773177] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  111.773222] sd 16:0:1:0: device reset: FAILED scmd(c000001fb7a2dd40)
    [  111.773266] scsi target16:0:1: attempting target reset! scmd(c000001fb7a2dd40)
    ...
    [  111.773528] mpt3sas_scsih_issue_tm: mpt3sas_cm0: host reset in progress!
    [  111.773574] scsi target16:0:1: target reset: FAILED scmd(c000001fb7a2dd40)
    [  111.773617] mpt3sas_cm0: attempting host reset! scmd(c000001fb7a2dd40)
    [  111.773662] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [  111.773721] Unable to handle kernel paging request for data at address 0xd000380081790000
    [  111.773773] Faulting instruction address: 0xd0000000179d69b8
    [  111.773818] Oops: Kernel access of bad area, sig: 11 [#1]
    ...
    [  111.774925] NIP [d0000000179d69b8] mpt3sas_base_get_iocstate+0x38/0xb0 [mpt3sas]
    [  111.774980] LR [d0000000179da3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  111.775032] Call Trace:
    [  111.775054] [c000001fd995bab0] [d0000000179da3f0] mpt3sas_base_hard_reset_handler+0x1a0/0x6d0 [mpt3sas]
    [  111.775118] [c000001fd995bb80] [d0000000179e0f6c] scsih_host_reset+0x7c/0x100 [mpt3sas]
    [  111.775173] [c000001fd995bc00] [c00000000076f34c] scsi_try_host_reset+0x5c/0x140
    [  111.775227] [c000001fd995bc40] [c00000000077107c] scsi_eh_ready_devs+0x73c/0x960
    [  111.775280] [c000001fd995bd10] [c000000000772800] scsi_error_handler+0x4c0/0x4e0
    [  111.775333] [c000001fd995bdc0] [c000000000107ed4] kthread+0x164/0x1b0
    [  111.775378] [c000001fd995be30] [c00000000000b5a0] ret_from_kernel_thread+0x5c/0xbc
    [  111.778393] Instruction dump:

With patch:

    [   56.316855] setting logging_level(0x11000000)
    [   56.347607] mpt3sas_cm0: skip free_smid/scsi_done scmd(c000001fc3afb540)
    [   57.604052] sd 16:0:1:0: [sdf] Synchronizing SCSI cache
    [   57.604319] sd 16:0:0:0: [sde] Synchronizing SCSI cache
    [   57.604615] mpt3sas_cm0: sending diag reset !!
    [   58.725154] mpt3sas_cm0: diag reset: SUCCESS
    [   58.741245] mpt3sas_cm0: sleep on shutdown
    [   59.803191] mpt3sas_cm0: sleep on shutdown
    ...
    [   67.103187] sd 16:0:1:0: attempting task abort! scmd(c000001fc3afb540)
    ...
    [   67.103521] mpt3sas_cm0: fail task abort scmd(c000001fc3afb540)
    [   67.103569] sd 16:0:1:0: task abort: FAILED scmd(c000001fc3afb540)
    [   67.143193] sd 16:0:1:0: attempting device reset! scmd(c000001fc3afb540)
    ...
    [   67.143514] sd 16:0:1:0: device been deleted! scmd(c000001fc3afb540)
    [   67.143564] sd 16:0:1:0: device reset: SUCCESS scmd(c000001fc3afb540)
    [   67.143613] scsi target16:0:1: attempting target reset! scmd(c000001fc3afb540)
    ...
    [   67.143897] scsi target16:0:1: target been deleted! scmd(c000001fc3afb540)
    [   67.143944] scsi target16:0:1: target reset: SUCCESS scmd(c000001fc3afb540)
    [   67.143992] mpt3sas_cm0: attempting host reset! scmd(c000001fc3afb540)
    [   67.144040] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [   67.144096] mpt3sas_cm0: Blocking the host reset
    [   67.144133] mpt3sas_cm0: host reset: FAILED scmd(c000001fc3afb540)
    [   67.144179] sd 16:0:1:0: Device offlined - not ready after error recovery
    [   67.144228] sd 16:0:1:0: [sdf] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
    [   67.144295] sd 16:0:1:0: [sdf] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
    [   67.144351] print_req_error: I/O error, dev sdf, sector 0
Martin K. Petersen Feb. 9, 2018, 11:22 p.m. UTC | #2 
> This patch adds checks for 'ioc->remove_host' in the SCSI error
> handlers, so not to access pointers/resources potentially freed
> in the PCI shutdown/module unload path.  The error handlers may
> be invoked after shutdown/unload, depending on other components.

Broadcom folks: Please review!
diff mbox

Patch

diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 74fca18..5ab3caf 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2835,7 +2835,8 @@  int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2898,7 +2899,8 @@  int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		sdev_printk(KERN_INFO, scmd->device,
 			"device been deleted! scmd(%p)\n", scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -2961,7 +2963,8 @@  int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	_scsih_tm_display_info(ioc, scmd);
 
 	sas_device_priv_data = scmd->device->hostdata;
-	if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
+	if (!sas_device_priv_data || !sas_device_priv_data->sas_target ||
+	    ioc->remove_host) {
 		starget_printk(KERN_INFO, starget, "target been deleted! scmd(%p)\n",
 			scmd);
 		scmd->result = DID_NO_CONNECT << 16;
@@ -3019,7 +3022,7 @@  int mpt3sas_scsih_issue_locked_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle,
 	    ioc->name, scmd);
 	scsi_print_command(scmd);
 
-	if (ioc->is_driver_loading) {
+	if (ioc->is_driver_loading || ioc->remove_host) {
 		pr_info(MPT3SAS_FMT "Blocking the host reset\n",
 		    ioc->name);
 		r = FAILED;