From patchwork Thu Mar 15 14:21:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dexuan Cui X-Patchwork-Id: 10284549 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AE95B602C2 for ; Thu, 15 Mar 2018 14:21:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A74B28AB9 for ; Thu, 15 Mar 2018 14:21:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98B2428ABE; Thu, 15 Mar 2018 14:21:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB3F028ABA for ; Thu, 15 Mar 2018 14:21:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752477AbeCOOVY (ORCPT ); Thu, 15 Mar 2018 10:21:24 -0400 Received: from mail-pu1apc01on0125.outbound.protection.outlook.com ([104.47.126.125]:29120 "EHLO APC01-PU1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752253AbeCOOVX (ORCPT ); Thu, 15 Mar 2018 10:21:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=kK9RPKqtoc2wLmKsyQ0tQ44zkRhD98QQnhA4wcj8+V8=; b=Zwl6S/l9Y5mVjdK0iqyD5LrI+PQGPVrxNWSNIxDz75U22/4PS/NPQAZJstGT234RWVNI/vYdjkgJGic1jD67AYOMJcsUZD9OqqWfS+Kh+x375WAsa9UvO7B1FgUmdyZmSjNlc0n0PovAh/k7CLuFt1nkQf/tZCdgssuAljLqHiY= Received: from KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM (10.170.167.17) by KL1P15301MB0072.APCP153.PROD.OUTLOOK.COM (10.170.168.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.609.6; Thu, 15 Mar 2018 14:21:08 +0000 Received: from KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM ([10.170.167.17]) by KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM ([10.170.167.17]) with mapi id 15.20.0609.006; Thu, 15 Mar 2018 14:21:08 +0000 From: Dexuan Cui To: "lorenzo.pieralisi@arm.com" , "bhelgaas@google.com" , "linux-pci@vger.kernel.org" , KY Srinivasan , Stephen Hemminger CC: "linux-kernel@vger.kernel.org" , "driverdev-devel@linuxdriverproject.org" , Haiyang Zhang , "olaf@aepfle.de" , "apw@canonical.com" , "jasowang@redhat.com" , "vkuznets@redhat.com" , "marcelo.cerri@canonical.com" , "Michael Kelley (EOSG)" Subject: [PATCH v4 2/2] PCI: hv: Fix 2 hang issues in hv_compose_msi_msg() Thread-Topic: [PATCH v4 2/2] PCI: hv: Fix 2 hang issues in hv_compose_msi_msg() Thread-Index: AdO8O4kEYfjYpPA4SNKhP2VbewlXZA== Date: Thu, 15 Mar 2018 14:21:08 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=decui@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2018-03-15T14:21:05.9351840Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General authentication-results: spf=none (sender IP is ) smtp.mailfrom=decui@microsoft.com; x-originating-ip: [2601:600:a27f:df20:d8b5:442c:6a7d:25c6] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; KL1P15301MB0072; 7:cERJbhrHV3TRuKNPUN7s2fSdYTdSlOfx7K2PcjGjiUGc7zi57F/bKdGJlSOl4k2JncQlVh+qeUDWBW6zqSvZLqCEDvpY3SOPpyaZF9OAL7U5HYsIUPyPLSASuzSqCfm4Qv4+ZwkdKoO8LEPMSUlXJQdEfn4GH/PB//pID7P/yNnBIsqaH9bpNPYDsuH7ZvuSZ5YJfukNIaJSVawDtltx7BBhFcPRCNPzNMG1Bds+pziCPU55hEGZG0bb1nBR52Wr; 20:URytUCp3dmLCfUVa2OftNx29vMjjXlyZvtb8qei3yNGBfE5Bm8+2dQKBcyYrMQyoccLRbnwCP0H3F1tUfHCcEQ2nxq4asfv/jjJjyjpJnQpVMPTUsXiRBdxHdkmTlJ4h7WQF1g5jXy1VbaFSs9bM99qh9TgnlgBe1R38+RKbiH0= x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 5b4db9fb-014d-40a7-c3b2-08d58a7fff12 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7193020); SRVR:KL1P15301MB0072; x-ms-traffictypediagnostic: KL1P15301MB0072: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(89211679590171)(9452136761055); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(61425038)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231221)(944501244)(52105095)(3002001)(10201501046)(6055026)(61426038)(61427038)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123564045)(20161123560045)(20161123562045)(6072148)(201708071742011); SRVR:KL1P15301MB0072; BCL:0; PCL:0; RULEID:; SRVR:KL1P15301MB0072; x-forefront-prvs: 0612E553B4 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(346002)(39860400002)(366004)(396003)(376002)(39380400002)(189003)(199004)(14454004)(7696005)(4326008)(2900100001)(478600001)(5660300001)(68736007)(10290500003)(7416002)(3660700001)(97736004)(110136005)(7736002)(74316002)(107886003)(54906003)(86362001)(316002)(22452003)(25786009)(99286004)(2501003)(2201001)(8936002)(305945005)(6116002)(10090500001)(102836004)(33656002)(2906002)(81166006)(186003)(86612001)(77096007)(53936002)(8990500004)(6636002)(8676002)(105586002)(55016002)(1511001)(6436002)(9686003)(106356001)(81156014)(3280700002)(59450400001)(46003)(6506007); DIR:OUT; SFP:1102; SCL:1; SRVR:KL1P15301MB0072; H:KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: +SXZ7qlHevDTy2ze2G0O8oU8k7y/nBxOXG8oFjdqVo0+ZnE1eEGTV7M8+1wP3JZv0WiVoWLISwYY8CWpAjlqkUc4AteDlpjLbl3HH38SpyZpaJf4wGYjEoswiFeumdHdOcWFlIH5t4LdtEPwcTZnqRrLi5wMejEKrE1lBHIi1/miKT/XjC321dNIi2Y2HjfDfHrnd2X1gsXNXWPXS40ySTjcbYP+q361qJ2qis0MOrCNG9BiLdlhKPK/lFGf3EV9fYD2xOykxhaYOEXWv4g1pm8RJmqHQv/r6RBLEMAdNJOGTaCBY4aVXfah+LTl8l6pnD+SiPpuoUiQINnREz5ZzA== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5b4db9fb-014d-40a7-c3b2-08d58a7fff12 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Mar 2018 14:21:08.2067 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: KL1P15301MB0072 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 1. With the patch "x86/vector/msi: Switch to global reservation mode", the recent v4.15 and newer kernels always hang for 1-vCPU Hyper-V VM with SR-IOV. This is because when we reach hv_compose_msi_msg() by request_irq() -> request_threaded_irq() ->__setup_irq()->irq_startup() -> __irq_startup() -> irq_domain_activate_irq() -> ... -> msi_domain_activate() -> ... -> hv_compose_msi_msg(), local irq is disabled in __setup_irq(). Note: when we reach hv_compose_msi_msg() by another code path: pci_enable_msix_range() -> ... -> irq_domain_activate_irq() -> ... -> hv_compose_msi_msg(), local irq is not disabled. hv_compose_msi_msg() depends on an interrupt from the host. With interrupts disabled, a UP VM always hangs in the busy loop in the function, because the interrupt callback hv_pci_onchannelcallback() can not be called. We can do nothing but work it around by polling the channel. This is ugly, but we don't have any other choice. 2. If the host is ejecting the VF device before we reach hv_compose_msi_msg(), in a UP VM, we can hang in hv_compose_msi_msg() forever, because at this time the host doesn't respond to the CREATE_INTERRUPT request. This issue exists the first day the pci-hyperv driver appears in the kernel. Luckily, this can also by worked around by polling the channel for the PCI_EJECT message and hpdev->state, and by checking the PCI vendor ID. Note: actually the above 2 issues also happen to a SMP VM, if "hbus->hdev->channel->target_cpu == smp_processor_id()" is true. Fixes: 4900be83602b ("x86/vector/msi: Switch to global reservation mode") Tested-by: Adrian Suhov Tested-by: Chris Valean Signed-off-by: Dexuan Cui Reviewed-by: Michael Kelley Acked-by: Haiyang Zhang Cc: stable@vger.kernel.org Cc: Stephen Hemminger Cc: K. Y. Srinivasan Cc: Vitaly Kuznetsov Cc: Jack Morgenstein --- drivers/pci/host/pci-hyperv.c | 58 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c index 5e67dff779a7..acb1941423a9 100644 --- a/drivers/pci/host/pci-hyperv.c +++ b/drivers/pci/host/pci-hyperv.c @@ -522,6 +522,8 @@ struct hv_pci_compl { s32 completion_status; }; +static void hv_pci_onchannelcallback(void *context); + /** * hv_pci_generic_compl() - Invoked for a completion packet * @context: Set up by the sender of the packet. @@ -666,6 +668,31 @@ static void _hv_pcifront_read_config(struct hv_pci_dev *hpdev, int where, } } +static u16 hv_pcifront_get_vendor_id(struct hv_pci_dev *hpdev) +{ + u16 ret; + unsigned long flags; + void __iomem *addr = hpdev->hbus->cfg_addr + CFG_PAGE_OFFSET + + PCI_VENDOR_ID; + + spin_lock_irqsave(&hpdev->hbus->config_lock, flags); + + /* Choose the function to be read. (See comment above) */ + writel(hpdev->desc.win_slot.slot, hpdev->hbus->cfg_addr); + /* Make sure the function was chosen before we start reading. */ + mb(); + /* Read from that function's config space. */ + ret = readw(addr); + /* + * mb() is not required here, because the spin_unlock_irqrestore() + * is a barrier. + */ + + spin_unlock_irqrestore(&hpdev->hbus->config_lock, flags); + + return ret; +} + /** * _hv_pcifront_write_config() - Internal PCI config write * @hpdev: The PCI driver's representation of the device @@ -1108,8 +1135,37 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) * Since this function is called with IRQ locks held, can't * do normal wait for completion; instead poll. */ - while (!try_wait_for_completion(&comp.comp_pkt.host_event)) + while (!try_wait_for_completion(&comp.comp_pkt.host_event)) { + /* 0xFFFF means an invalid PCI VENDOR ID. */ + if (hv_pcifront_get_vendor_id(hpdev) == 0xFFFF) { + dev_err_once(&hbus->hdev->device, + "the device has gone\n"); + goto free_int_desc; + } + + /* + * When the higher level interrupt code calls us with + * interrupt disabled, we must poll the channel by calling + * the channel callback directly when channel->target_cpu is + * the current CPU. When the higher level interrupt code + * calls us with interrupt enabled, let's add the + * local_bh_disable()/enable() to avoid race. + */ + local_bh_disable(); + + if (hbus->hdev->channel->target_cpu == smp_processor_id()) + hv_pci_onchannelcallback(hbus); + + local_bh_enable(); + + if (hpdev->state == hv_pcichild_ejecting) { + dev_err_once(&hbus->hdev->device, + "the device is being ejected\n"); + goto free_int_desc; + } + udelay(100); + } if (comp.comp_pkt.completion_status < 0) { dev_err(&hbus->hdev->device,