From patchwork Thu Jun 14 10:25:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaitra P B X-Patchwork-Id: 10463763 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 298786020F for ; Thu, 14 Jun 2018 10:25:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19E4528A93 for ; Thu, 14 Jun 2018 10:25:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E42E28A46; Thu, 14 Jun 2018 10:25:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 273E228A89 for ; Thu, 14 Jun 2018 10:25:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754879AbeFNKZy (ORCPT ); Thu, 14 Jun 2018 06:25:54 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:51418 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754772AbeFNKZw (ORCPT ); Thu, 14 Jun 2018 06:25:52 -0400 Received: by mail-wm0-f67.google.com with SMTP id r15-v6so9644267wmc.1 for ; Thu, 14 Jun 2018 03:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id; bh=Z5z40mhhkXnfclODb+prGrS3iyKiFeDg9R+eEhIUyxY=; b=BwJTKYvrvMSE16q35roY/dX14b/xdGDF2GD+RTJLl7wsJ+XPzfYRKMmA8byvCK3T3+ H3p7vxxRD3IgijWPaaz/PsBecHGHeQwZGooLUlu8gMLdxXpznQVx3YnQIFQNnTgycBEl Ncka2D69xKJnBAubfqQAjSSgjGvXa/PFHyNmY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=Z5z40mhhkXnfclODb+prGrS3iyKiFeDg9R+eEhIUyxY=; b=pDJa7riAZLAriomkfqKFZ3Q+oHFmgSSwPzlV8gLY0MnkrdVObsi4kKsMNWCVuSTL/M 3seS21Js3ZnoLeiawuIGt83ylRKDfjlTEwAq+3XeSUcEBUhtmXDtaXIQhMewQDthdT/K 7Q1dEpaUUzpF1Uo2UetpPOmIWgTi6DxfVNPOnHm4+f7VIhG3z8goYDS70U+NWVJNUSIQ GBE3VoOM06T1VlnwB1Vhnk0CNIv8QPo0u7inoV8xbX1ahvEr6vKfNimsMMExteSmWpkQ Cm0KiyBuJ6vkqCEHEt2pljDhAPBwmkdTvn7GAqnk5+Rqe2GIenB0EqWFR2WJkh6nqdHg z5oQ== X-Gm-Message-State: APt69E0nNV64PieoHPWEdZWQz4cu/uQHDzD+E2CrYzj7gzT6zr1Mi8DM SsaXJk6tAB3BywMSJK1ZAwMu3ZWo X-Google-Smtp-Source: ADUXVKIo2L48n7XGyl80Vxdsg7YYskn9mMz2kyaeJJ9Bc8XeU7lYiz20LnziEPUrJGZjwzKPrm4tmA== X-Received: by 2002:a50:d1c1:: with SMTP id i1-v6mr2186182edg.122.1528971951139; Thu, 14 Jun 2018 03:25:51 -0700 (PDT) Received: from localhost.localdomain.localdomain ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id z24-v6sm972772edd.56.2018.06.14.03.25.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 03:25:50 -0700 (PDT) From: Chaitra P B To: linux-scsi@vger.kernel.org Cc: sathya.prakash@broadcom.com, suganath-prabu.subramani@broadcom.com, Chaitra P B , Sreekanth Reddy Subject: [PATCH v1] mpt3sas: Fix calltrace observed while running IO & host reset Date: Thu, 14 Jun 2018 06:25:21 -0400 Message-Id: <1528971921-15216-1-git-send-email-chaitra.basappa@broadcom.com> X-Mailer: git-send-email 1.8.3.1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Below kernel BUG was observed while running IOs with host reset (issued from application), mpt3sas_cm0: diag reset: SUCCESS ------------[ cut here ]------------ WARNING: CPU: 12 PID: 4336 at drivers/scsi/mpt3sas/mpt3sas_base.c:3282 mpt3sas_base_clear_st+0x3d/0x40 [mpt3sas] Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support dcdbas pcspkr joydev ipmi_ssif ses enclosure sg ipmi_devintf acpi_pad ipmi_msghandler acpi_power_meter mei_me lpc_ich wmi mei shpchp ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi uas usb_storage mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mpt3sas libata crct10dif_pclmul crct10dif_common tg3 crc32c_intel i2c_core raid_class ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod CPU: 12 PID: 4336 Comm: python Kdump: loaded Tainted: G W ------------ 3.10.0-875.el7.brdc.x86_64 #1 Hardware name: Dell Inc. PowerEdge R820/0YWR73, BIOS 1.5.0 03/08/2013 Call Trace: [] dump_stack+0x19/0x1b [] __warn+0xd8/0x100 [] warn_slowpath_null+0x1d/0x20 [] mpt3sas_base_clear_st+0x3d/0x40 [mpt3sas] [] _scsih_flush_running_cmds+0x92/0xe0 [mpt3sas] [] mpt3sas_scsih_reset_handler+0x43b/0xaf0 [mpt3sas] [] ? vprintk_default+0x29/0x40 [] ? printk+0x60/0x77 [] ? _base_diag_reset+0x238/0x340 [mpt3sas] [] mpt3sas_base_hard_reset_handler+0x1ad/0x420 [mpt3sas] [] _ctl_ioctl_main.isra.12+0x11b9/0x1200 [mpt3sas] [] ? xfs_file_aio_write+0x155/0x1b0 [xfs] [] ? do_sync_write+0x93/0xe0 [] _ctl_ioctl+0x1a/0x20 [mpt3sas] [] do_vfs_ioctl+0x350/0x560 [] ? __sb_end_write+0x31/0x60 [] SyS_ioctl+0xa1/0xc0 [] ? system_call_after_swapgs+0xa2/0x146 [] system_call_fastpath+0x1c/0x21 [] ? system_call_after_swapgs+0xae/0x146 ---[ end trace 5dac5b98d89aaa3c ]--- ------------[ cut here ]------------ kernel BUG at block/blk-core.c:1476! invalid opcode: 0000 [#1] SMP Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support dcdbas pcspkr joydev ipmi_ssif ses enclosure sg ipmi_devintf acpi_pad ipmi_msghandler acpi_power_meter mei_me lpc_ich wmi mei shpchp ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi uas usb_storage mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mpt3sas libata crct10dif_pclmul crct10dif_common tg3 crc32c_intel i2c_core raid_class ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod CPU: 12 PID: 4336 Comm: python Kdump: loaded Tainted: G W ------------ 3.10.0-875.el7.brdc.x86_64 #1 Hardware name: Dell Inc. PowerEdge R820/0YWR73, BIOS 1.5.0 03/08/2013 task: ffff903fc96e0fd0 ti: ffff903fb1eec000 task.ti: ffff903fb1eec000 RIP: 0010:[] [] blk_requeue_request+0x90/0xa0 RSP: 0018:ffff903c6b783dc0 EFLAGS: 00010087 RAX: ffff903bb67026d0 RBX: ffff903b7d6a6140 RCX: dead000000000200 RDX: ffff903bb67026d0 RSI: ffff903bb6702580 RDI: ffff903bb67026d0 RBP: ffff903c6b783dd8 R08: ffff903bb67026d0 R09: ffffd97e80000000 R10: ffff903c658bac00 R11: 0000000000000000 R12: ffff903bb6702580 R13: ffff903fa9a292f0 R14: 0000000000000246 R15: 0000000000001057 FS: 00007f7026f5b740(0000) GS:ffff903c6b780000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f298877c004 CR3: 00000000caf36000 CR4: 00000000000607e0 Call Trace: [] __scsi_queue_insert+0xbf/0x110 [] scsi_io_completion+0x5da/0x6a0 [] scsi_finish_command+0xdc/0x140 [] scsi_softirq_done+0x132/0x160 [] blk_done_softirq+0x96/0xc0 [] __do_softirq+0xf5/0x280 [] call_softirq+0x1c/0x30 [] do_softirq+0x65/0xa0 [] irq_exit+0x105/0x110 [] smp_apic_timer_interrupt+0x48/0x60 [] apic_timer_interrupt+0x162/0x170 [] ? scsi_done+0x21/0x60 [] ? delay_tsc+0x38/0x60 [] __const_udelay+0x2d/0x30 [] _base_handshake_req_reply_wait+0x8e/0x4a0 [mpt3sas] [] _base_get_ioc_facts+0x123/0x590 [mpt3sas] [] ? _base_diag_reset+0x238/0x340 [mpt3sas] [] mpt3sas_base_hard_reset_handler+0x1f3/0x420 [mpt3sas] [] _ctl_ioctl_main.isra.12+0x11b9/0x1200 [mpt3sas] [] ? xfs_file_aio_write+0x155/0x1b0 [xfs] [] ? do_sync_write+0x93/0xe0 [] _ctl_ioctl+0x1a/0x20 [mpt3sas] [] do_vfs_ioctl+0x350/0x560 [] ? __sb_end_write+0x31/0x60 [] SyS_ioctl+0xa1/0xc0 [] ? system_call_after_swapgs+0xa2/0x146 [] system_call_fastpath+0x1c/0x21 [] ? system_call_after_swapgs+0xae/0x146 Code: 83 c3 10 4c 89 e2 4c 89 ee e8 8d 21 04 00 48 8b 03 48 85 c0 75 e5 41 f6 44 24 4a 10 74 ad 4c 89 e6 4c 89 ef e8 b2 42 00 00 eb a0 <0f> 0b 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 RIP [] blk_requeue_request+0x90/0xa0 RSP As a part of host reset operation, driver will flushout all IOs outstanding at driver level with "DID_RESET" result. To find which are all commands outstanding at the driver level, driver loops with smid starting from one to HBA queue depth and calls mpt3sas_scsih_scsi_lookup_get() to get scmd as shown below for (smid = 1; smid <= ioc->scsiio_depth; smid++) { scmd = mpt3sas_scsih_scsi_lookup_get(ioc, smid); if (!scmd) continue; But in mpt3sas_scsih_scsi_lookup_get() function, driver returns some scsi cmnds which are not outstanding at the driver level (possibly request is constructed at block layer since QUEUE_FLAG_QUIESCED is not set. Even if driver uses scsi_block_requests and scsi_unblock_requests, issue still persists as they will be just blocking further IO from scsi layer and not from block layer) and these commands are flushed with DID_RESET host bytes thus resulting into above kernel BUG. To fix this issue, we have modified the mpt3sas_scsih_scsi_lookup_get() to check for smid equals to zero (note: whenever any scsi cmnd is outstanding at the driver level then smid for that scsi cmnd won't be zero, always it starts from one) before it returns the scmd pointer to the caller. If smid is zero then this function returns scmd pointer as NULL and driver won't flushout those scsi cmnds at driver level with DID_RESET host byte thus this issue will not be observed. Signed-off-by: Chaitra P B Signed-off-by: Sreekanth Reddy --- drivers/scsi/mpt3sas/mpt3sas_base.c | 1 + drivers/scsi/mpt3sas/mpt3sas_scsih.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 668f350..49d92d0 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -3309,6 +3309,7 @@ void mpt3sas_base_clear_st(struct MPT3SAS_ADAPTER *ioc, return; st->cb_idx = 0xFF; st->direct_io = 0; + st->smid = 0; atomic_set(&ioc->chain_lookup[st->smid - 1].chain_offset, 0); } diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c index 23902ad..96e523a 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c +++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c @@ -1489,7 +1489,7 @@ struct scsi_cmnd * scmd = scsi_host_find_tag(ioc->shost, unique_tag); if (scmd) { st = scsi_cmd_priv(scmd); - if (st->cb_idx == 0xFF) + if (st->cb_idx == 0xFF || st->smid == 0) scmd = NULL; } }