From patchwork Fri Aug 4 14:57:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srinath Mannam X-Patchwork-Id: 9881453 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1356A602B8 for ; Fri, 4 Aug 2017 14:57:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04A83289DF for ; Fri, 4 Aug 2017 14:57:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED8D1289E4; Fri, 4 Aug 2017 14:57:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E464289DF for ; Fri, 4 Aug 2017 14:57:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752146AbdHDO5p (ORCPT ); Fri, 4 Aug 2017 10:57:45 -0400 Received: from mail-qt0-f178.google.com ([209.85.216.178]:38400 "EHLO mail-qt0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752109AbdHDO5o (ORCPT ); Fri, 4 Aug 2017 10:57:44 -0400 Received: by mail-qt0-f178.google.com with SMTP id t37so10346714qtg.5 for ; Fri, 04 Aug 2017 07:57:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id; bh=j5anHAmCPBqE+PwxGVFtebe4297p3/DCm2fdQUeS8Ss=; b=SB1ET0rGu6uzKNTUcR9oOQrNywM0PI5YL7USAVFe91VuSZ07Cz8RHv4+aNTfrlK3R/ jGLON079IAPg5tAnC2A7heyA11Tgz1BNPbCfyrC9wqGAcLlLHL5K8kMrpTYuVw8/2h0t osQ7luLA355ntoWEU5DculFIWlj1m09/Qu8Ms= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=j5anHAmCPBqE+PwxGVFtebe4297p3/DCm2fdQUeS8Ss=; b=gpZM8J9/Up3vIBLpcwup1Wf4V0V1F1H4CjI36z1GVaC4kIl1x4WnCynI+PAQG2lI7Z u7u3xXrElGMnd86th8SWp1+OznUEBHmO9Jku8ckRK0a5cKJw1ix4cmcXZQc1+dYz5zQs O+Kwe/bzpMs9msD6aQgxk3GImo2BpBoBoO6CQMmkCiXi+iL6dI6Qk66E7DFDTco5RqKF lSs96xqiVSkAyhrmgTjjj51092jKHs9GxnldNN7o2/PzYbS+y/5nrCq4eibrsvWYY7lL i+deVzwF0k1+cvBaql4xOOUhNGYNYX/lZKjmUnjdUH+cQ1jrI7cy4MSX+pFFoMCFMRS6 TFEQ== X-Gm-Message-State: AHYfb5jIIzZZAwQFlQYHo8VA2AbzphtrVe39lqvmKKUGhj+IDdMoBHHE 3W3Fux4cnJhoajzK X-Received: by 10.200.37.27 with SMTP id 27mr3680760qtm.247.1501858663876; Fri, 04 Aug 2017 07:57:43 -0700 (PDT) Received: from mannams-OptiPlex-7010.dhcp.avagotech.net ([192.19.237.250]) by smtp.gmail.com with ESMTPSA id z1sm1233473qtz.62.2017.08.04.07.57.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 04 Aug 2017 07:57:43 -0700 (PDT) From: Srinath Mannam To: bhelgaas@google.com Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, bcm-kernel-feedback-list@broadcom.com, Srinath Mannam Subject: [RFC PATCH v3] pci: Concurrency issue during pci enable bridge Date: Fri, 4 Aug 2017 20:27:28 +0530 Message-Id: <1501858648-22228-1-git-send-email-srinath.mannam@broadcom.com> X-Mailer: git-send-email 2.7.4 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Concurrency issue is observed during pci enable bridge called for multiple pci devices initialization in SMP system. Setup details: - SMP system with 8 ARMv8 cores running Linux kernel(4.11). - Two EPs are connected to PCIe RC through bridge as shown in the below figure. [RC] | [BRIDGE] | ----------- | | [EP] [EP] Issue description: After PCIe enumeration completed EP driver probe function called for both the devices from two CPUS simultaneously. From EP probe function, pci_enable_device_mem called for both the EPs. This function called pci_enable_bridge enable for all the bridges recursively in the path of EP to RC. Inside pci_enable_bridge function, at two places concurrency issue is observed. Place 1: CPU 0: 1. Done Atomic increment dev->enable_cnt in pci_enable_device_flags 2. Inside pci_enable_resources 3. Completed pci_read_config_word(dev, PCI_COMMAND, &cmd) 4. Ready to set PCI_COMMAND_MEMORY (0x2) in pci_write_config_word(dev, PCI_COMMAND, cmd) CPU 1: 1. Check pci_is_enabled in function pci_enable_bridge and it is true 2. Check (!dev->is_busmaster) also true 3. Gone into pci_set_master 4. Completed pci_read_config_word(dev, PCI_COMMAND, &old_cmd) 5. Ready to set PCI_COMMAND_MASTER (0x4) in pci_write_config_word(dev, PCI_COMMAND, cmd) By the time of last point for both the CPUs are read value 0 and ready to write 2 and 4. After last point final value in PCI_COMMAND register is 4 instead of 6. Place 2: CPU 0: 1. Done Atomic increment dev->enable_cnt in pci_enable_device_flags Signed-off-by: Srinath Mannam --- drivers/pci/pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc34..12721df 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -52,6 +52,7 @@ static void pci_pme_list_scan(struct work_struct *work); static LIST_HEAD(pci_pme_list); static DEFINE_MUTEX(pci_pme_list_mutex); static DECLARE_DELAYED_WORK(pci_pme_work, pci_pme_list_scan); +static DEFINE_MUTEX(pci_bridge_mutex); struct pci_pme_device { struct list_head list; @@ -1348,10 +1349,11 @@ static void pci_enable_bridge(struct pci_dev *dev) if (bridge) pci_enable_bridge(bridge); + mutex_lock(&pci_bridge_mutex); if (pci_is_enabled(dev)) { if (!dev->is_busmaster) pci_set_master(dev); - return; + goto end; } retval = pci_enable_device(dev); @@ -1359,6 +1361,8 @@ static void pci_enable_bridge(struct pci_dev *dev) dev_err(&dev->dev, "Error enabling bridge (%d), continuing\n", retval); pci_set_master(dev); +end: + mutex_unlock(&pci_bridge_mutex); } static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) @@ -1383,7 +1387,7 @@ static int pci_enable_device_flags(struct pci_dev *dev, unsigned long flags) return 0; /* already enabled */ bridge = pci_upstream_bridge(dev); - if (bridge) + if (bridge && !pci_is_enabled(bridge)) pci_enable_bridge(bridge); /* only skip sriov related */