From patchwork Wed Jun 6 18:31:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Madhani, Himanshu" X-Patchwork-Id: 10450817 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4FA8260146 for ; Wed, 6 Jun 2018 18:31:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3DAD1298F3 for ; Wed, 6 Jun 2018 18:31:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 30ACF29900; Wed, 6 Jun 2018 18:31:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C1C1298F3 for ; Wed, 6 Jun 2018 18:31:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932138AbeFFSbn (ORCPT ); Wed, 6 Jun 2018 14:31:43 -0400 Received: from mail-dm3nam03on0081.outbound.protection.outlook.com ([104.47.41.81]:13222 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752164AbeFFSbl (ORCPT ); Wed, 6 Jun 2018 14:31:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6IhD81gLfJ2GbiWfV7rSZI4EbwTXTza9+8MJ6v4L4no=; b=jNPoeM1x9G/JCqGG13GT5KkkJxvwEndlZ56/8vpC/v/IYQmnALZ5YIVRnC5uA7OTq7SXfzkrk6XZ6b9hJbaNDRvbX7JlCqQe3sQ/7so3tmwxS63FOcpwpIKpupL7Gvkag2+8e0qrqFKIUC4Ja8yiDNelvyFUpnbLP38M2/LnU7E= Received: from DM5PR0701MB3750.namprd07.prod.outlook.com (10.167.109.166) by DM5PR0701MB3654.namprd07.prod.outlook.com (10.167.109.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.841.14; Wed, 6 Jun 2018 18:31:39 +0000 Received: from DM5PR0701MB3750.namprd07.prod.outlook.com ([fe80::fc45:c1cf:503f:fba6]) by DM5PR0701MB3750.namprd07.prod.outlook.com ([fe80::fc45:c1cf:503f:fba6%4]) with mapi id 15.20.0841.011; Wed, 6 Jun 2018 18:31:39 +0000 From: "Madhani, Himanshu" To: Li Wang CC: "Martin K. Petersen" , "Tran, Quinn" , "William.Kuzeja@stratus.com" , linux-kernel , "linux-scsi@vger.kernel.org" , Laurence Oberman Subject: Re: qla2xxx cause BUG on kernel-4.17-rc6 Thread-Topic: qla2xxx cause BUG on kernel-4.17-rc6 Thread-Index: AQHT+VlWebSsauP2jUycx2vUO1XVIqRTa1NGgAABKYCAACLJgIAABz4A Date: Wed, 6 Jun 2018 18:31:39 +0000 Message-ID: References: <7988FB77-4AE4-4935-B7B7-F7584674981B@cavium.com> <1528308343.17774.3.camel@redhat.com> In-Reply-To: <1528308343.17774.3.camel@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Himanshu.Madhani@cavium.com; x-originating-ip: [173.186.134.106] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DM5PR0701MB3654; 7:kv4B0MmtDQevUzm2wRo7NOxgRdXBVjITFfrBBIYuvvCv3aTFIAHTDVJ/iB/mtfPsPnXDfgEETqaYckJOqSpHLuXcF4eFvbGlk2sPTq4dXy4xtN6MaJuNhxvCci4LDCrbyCUQGMjsMKQVOrCz6WScyQMdpprUJ0OEA3pAqva/CAZD4aKnYYS0P1qKQ5XF2UskYu4oOo47FhN6HpiTe2JHJleE3Yt9PGCVSTHTiGbb3pPckMHHEJKmxYa0qKtrWkp/ x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10009020)(39860400002)(39380400002)(346002)(376002)(366004)(396003)(199004)(189003)(51234002)(99286004)(76176011)(6116002)(3846002)(8656006)(66066001)(72206003)(6506007)(26005)(105586002)(229853002)(59450400001)(5250100002)(305945005)(106356001)(186003)(53546011)(54906003)(8936002)(14454004)(102836004)(97736004)(68736007)(7736002)(6486002)(25786009)(2616005)(53936002)(5660300001)(6916009)(6512007)(93886005)(11346002)(316002)(6436002)(83716003)(6246003)(476003)(4326008)(36756003)(2900100001)(33656002)(446003)(86362001)(486006)(82746002)(3280700002)(3660700001)(478600001)(8676002)(45080400002)(81166006)(81156014)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR0701MB3654; H:DM5PR0701MB3750.namprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DM5PR0701MB3654; x-ms-traffictypediagnostic: DM5PR0701MB3654: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(3231254)(944501410)(52105095)(149027)(150027)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123562045)(20161123558120)(20161123564045)(6072148)(201708071742011)(7699016); SRVR:DM5PR0701MB3654; BCL:0; PCL:0; RULEID:; SRVR:DM5PR0701MB3654; x-forefront-prvs: 06952FC175 received-spf: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: E30gOzccG4/S68VkEQlKTAZgOYweylNSbY5V+TCyrUveLIjJyDj+BOL0OsU3bSB/dWl31CZjQktYrgQd12LGkHItHcwKQmkofuKLtvlZ6Qbc9r+e3JQYx9FEiJSrOtYeVB5i514o0j/0nrRoeWbj8ykXsofeVBVfK8NQmtVHVSQkiPPt2ifDRSMPHNxXlyGZg9icdXuGDrnkxRUikRXEww== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <9807786EC3C8104A80DFFBB101D48A7F@namprd07.prod.outlook.com> MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 99709119-5c8d-4325-1c6c-08d5cbdbbe5e X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-Network-Message-Id: 99709119-5c8d-4325-1c6c-08d5cbdbbe5e X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Jun 2018 18:31:39.4297 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR0701MB3654 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Li, > On Jun 6, 2018, at 11:05 AM, Laurence Oberman wrote: > > On Wed, 2018-06-06 at 16:01 +0000, Madhani, Himanshu wrote: >>> On Jun 6, 2018, at 8:56 AM, Martin K. Petersen >> cle.com> wrote: >>> >>> >>> Himanshu, >>> >>> Ping? >>> >> >> Will look at this one. Sorry, somehow fell thru cracks. >> >> >>>> Hi scsi experts, >>>> >>>> Not sure who is the right person to ask, I just hit this bug on >>>> my HP >>>> DL385 platform, can any one of you take a look? >>>> >>>> system config: >>>> ----------------- >>>> HP ProLiant DL385 G7 >>>> AMD Opteron(TM) Processor 6234 >>>> 16384 MB memory, 369 GB disk space >>>> >>>> >>>> [ 24.539274] qla2xxx [0000:0c:00.7]-500a:5: LOOP UP detected >>>> (10 Gbps). >>>> [ 24.577259] BUG: unable to handle kernel NULL pointer >>>> dereference >>>> at 0000000000000102 >>>> [ 24.623133] PGD 0 P4D 0 >>>> [ 24.636760] Oops: 0000 [#1] SMP NOPTI >>>> [ 24.656942] Modules linked in: i2c_algo_bit drm_kms_helper >>>> sr_mod(+) syscopyarea sysfillrect sysimgblt cdrom fb_sys_fops >>>> ata_generic ttm pata_acpi sd_mod ahci pata_atiixp sfc(+) >>>> qla2xxx(+) >>>> libahci drm qla4xxx(+) nvme_fc hpsa mdio libiscsi qlcnic(+) >>>> nvme_fabrics scsi_transport_sas serio_raw mtd crc32c_intel libata >>>> nvme_core i2c_core scsi_transport_iscsi tg3 scsi_transport_fc >>>> bnx2 >>>> iscsi_boot_sysfs dm_multipath dm_mirror dm_region_hash dm_log >>>> dm_mod >>>> [ 24.887449] CPU: 0 PID: 177 Comm: kworker/0:3 Not tainted >>>> 4.17.0-rc6 #1 >>>> [ 24.925119] Hardware name: HP ProLiant DL385 G7, BIOS A18 >>>> 08/15/2012 >>>> [ 24.962106] Workqueue: events work_for_cpu_fn >>>> [ 24.987098] RIP: 0010:__queue_work+0x1f/0x3a0 >>>> [ 25.011672] RSP: 0018:ffff992642ceba10 EFLAGS: 00010082 >>>> [ 25.042116] RAX: 0000000000000082 RBX: 0000000000000082 RCX: >>>> 0000000000000000 >>>> [ 25.083293] RDX: ffff8cf9abc6d7d0 RSI: 0000000000000000 RDI: >>>> 0000000000002000 >>>> [ 25.123094] RBP: 0000000000000000 R08: 0000000000025a40 R09: >>>> ffff8cf9aade2880 >>>> [ 25.164087] R10: 0000000000000000 R11: ffff992642ceb6f0 R12: >>>> ffff8cf9abc6d7d0 >>>> [ 25.202280] R13: 0000000000002000 R14: ffff8cf9abc6d7b8 R15: >>>> 0000000000002000 >>>> [ 25.242050] FS: 0000000000000000(0000) f9b5c00000(0000) >>>> knlGS:0000000000000000 >>>> [ 25.977565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>> [ 26.010457] CR2: 0000000000000102 CR3: 000000030760a000 CR4: >>>> 00000000000406f0 >>>> [ 26.051048] Call Trace: >>>> [ 26.063572] ? __switch_to_asm+0x34/0x70 >>>> [ 26.086079] queue_work_on+0x24/0x40 >>>> [ 26.107090] qla2x00_post_work+0x81/0xb0 [qla2xxx] >>>> [ 26.133356] qla2x00_async_event+0x1ad/0x1a20 [qla2xxx] >>>> [ 26.164075] ? lock_timer_base+0x67/0x80 >>>> [ 26.186420] ? try_to_del_timer_sync+0x4d/0x80 >>>> [ 26.212284] ? del_timer_sync+0x35/0x40 >>>> [ 26.234080] ? schedule_timeout+0x165/0x2f0 >>>> [ 26.259575] qla82xx_poll+0x13e/0x180 [qla2xxx] >>>> [ 26.285740] qla2x00_mailbox_command+0x74b/0xf50 [qla2xxx] >>>> [ 26.319040] qla82xx_set_driver_version+0x13b/0x1c0 [qla2xxx] >>>> [ 26.352108] ? qla2x00_init_rings+0x206/0x3f0 [qla2xxx] >>>> [ 26.381733] qla2x00_initialize_adapter+0x35c/0x7f0 [qla2xxx] >>>> [ 26.413240] qla2x00_probe_one+0x1479/0x2390 [qla2xxx] >>>> [ 26.442055] local_pci_probe+0x3f/0xa0 >>>> [ 26.463108] work_for_cpu_fn+0x10/0x20 >>>> [ 26.483295] process_one_work+0x152/0x350 >>>> [ 26.505730] worker_thread+0x1cf/0x3e0 >>>> [ 26.527090] kthread+0xf5/0x130 >>>> [ 26.545085] ? max_active_store+0x80/0x80 >>>> [ 26.568085] ? kthread_bind+0x10/0x10 >>>> [ 26.589533] ret_from_fork+0x22/0x40 >>>> [ 26.610192] Code: 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 >>>> 00 >>>> 00 41 57 41 89 ff 41 56 41 55 41 89 fd 41 54 49 89 d4 55 48 89 f5 >>>> 53 >>>> 48 83 ec 0 86 02 01 00 00 01 0f 85 80 02 00 00 49 c7 c6 c0 ec 01 >>>> 00 41 >>>> [ 27.308540] RIP: __queue_work+0x1f/0x3a0 RSP: ffff992642ceba10 >>>> [ 27.341591] CR2: 0000000000000102 >>>> [ 27.360208] ---[ end trace 01b7b7ae2c005cf3 ]--- >>> >>> -- >>> Martin K. Petersen Oracle Linux Engineering >> >> Thanks, >> - Himanshu >> > > I can't find the original message for this that Martin reminded us of. > > To the person who logged this: > How many times has this happened and was it after a kernel update. > What is the history, what is the exact Qlogic card, etc. > Do you have the rest of the log log leading to the invalid pointer > fault > > Thanks > Laurence From the Snippet of Log provided looks like the crash is with 10G FCoE adapter. Can you try this untested diff to see if it resolves issue. Basically we are initializing adapter so driver will start receiving AEN notification but we have not yet allocated work queue for it. ————— ———— host->can_queue, base_vha->req, base_vha->mgmt_svr_loop_id, host->sg_tablesize); INIT_WORK(&base_vha->iocb_work, qla2x00_iocb_work_fn); - ha->wq = alloc_workqueue("qla2xxx_wq", 0, 0); + if (ha->mqenable) { bool mq = false; ————— ———— Thanks, - Himanshu diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index 30bf4b9..462d825 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -3229,6 +3229,8 @@ qla2x00_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) "req->req_q_in=%p req->req_q_out=%p rsp->rsp_q_in=%p rsp->rsp_q_out=%p.\n", req->req_q_in, req->req_q_out, rsp->rsp_q_in, rsp->rsp_q_out); + ha->wq = alloc_workqueue("qla2xxx_wq", 0, 0); + if (ha->isp_ops->initialize_adapter(base_vha)) { ql_log(ql_log_fatal, base_vha, 0x00d6, "Failed to initialize adapter - Adapter flags %x.\n", @@ -3270,7 +3272,7 @@ qla2x00_probe_one(struct pci_dev *pdev, const struct pci_device_id *id)