From patchwork Tue Jun 9 00:20:22 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Luis R. Rodriguez" X-Patchwork-Id: 6568921 Return-Path: X-Original-To: patchwork-linux-fbdev@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id B9D2EC0020 for ; Tue, 9 Jun 2015 00:30:10 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 1D687203ED for ; Tue, 9 Jun 2015 00:30:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9EF6D203F1 for ; Tue, 9 Jun 2015 00:29:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753451AbbFIA31 (ORCPT ); Mon, 8 Jun 2015 20:29:27 -0400 Received: from mail-qg0-f42.google.com ([209.85.192.42]:36092 "EHLO mail-qg0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753200AbbFIA3Z (ORCPT ); Mon, 8 Jun 2015 20:29:25 -0400 Received: by qgep100 with SMTP id p100so1094919qge.3; Mon, 08 Jun 2015 17:29:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=BQ5POq/XA2KwE5Qn/CAoU1ppuVcSRNRna2fn7I1C2Hw=; b=uvJilLXlJhvre10qYreH/rwtFtuaXcquJkGjuqwxbxgeAn+AuXK0oBt0rMo2I8lwjS z8Qd+Ef0cjOfM4j1IPiJAZuCMx3nOnB/oCRcQ2xk449vU5Z4QpRvPkKfrZWR0BIZfQgO OHnpuYLtJ5u/Bmk13CRcm0lAuBUEP3/L6X4B7TbDfEzA3TmzN84ZvM278F/RET5o/2D6 HtE+FaeH91eNT3kR7QWaMDMXRicb4E7Bq/Q2TayDVmfNi1Zry0xaAcZApM7pC5NG/60s 151E+gT9niwLnn6l+msLOcR8lZlQRBzQXs9nJkiYs4NBwydCbVJou8VLS3zeRm9mTCLf jJAA== X-Received: by 10.229.222.130 with SMTP id ig2mr17333652qcb.6.1433809764556; Mon, 08 Jun 2015 17:29:24 -0700 (PDT) Received: from mcgrof@gmail.com (c-98-234-145-61.hsd1.ca.comcast.net. [98.234.145.61]) by mx.google.com with ESMTPSA id 63sm1970767qgx.25.2015.06.08.17.29.18 (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 08 Jun 2015 17:29:23 -0700 (PDT) Received: by mcgrof@gmail.com (sSMTP sendmail emulation); Mon, 08 Jun 2015 17:27:10 -0700 From: "Luis R. Rodriguez" To: mchehab@osg.samsung.com, bp@suse.de Cc: tomi.valkeinen@ti.com, bhelgaas@google.com, linux-media@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, "Luis R. Rodriguez" , Doug Ledford , Andy Walls , Hal Rosenstock , Sean Hefty , Suresh Siddha , Rickard Strandqvist , Mike Marciniszyn , Roland Dreier , Andy Lutomirski , Ingo Molnar , Linus Torvalds , Thomas Gleixner , Juergen Gross , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , =?UTF-8?q?Ville=20Syrj=C3=A4l=C3=A4?= , Mel Gorman , Vlastimil Babka , Davidlohr Bueso , Dave Hansen , Arnd Bergmann , "Michael S. Tsirkin" , Stefan Bader , konrad.wilk@oracle.com, ville.syrjala@linux.intel.com, david.vrabel@citrix.com, jbeulich@suse.com, toshi.kani@hp.com, =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , infinipath@intel.com, linux-fbdev@vger.kernel.org, xen-devel@lists.xensource.com Subject: [PATCH v6 3/3] IB/ipath: use arch_phys_wc_add() and require PAT disabled Date: Mon, 8 Jun 2015 17:20:22 -0700 Message-Id: <1433809222-28261-4-git-send-email-mcgrof@do-not-panic.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1433809222-28261-1-git-send-email-mcgrof@do-not-panic.com> References: <1433809222-28261-1-git-send-email-mcgrof@do-not-panic.com> MIME-Version: 1.0 Sender: linux-fbdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fbdev@vger.kernel.org X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, T_DKIM_INVALID, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Luis R. Rodriguez" We are burrying direct access to MTRR code support on x86 in order to take advantage of PAT. In the future we also want to make the default behaviour of ioremap_nocache() to use strong UC, use of mtrr_add() on those systems would make write-combining void. In order to help both enable us to later make strong UC default and in order to phase out direct MTRR access code port the driver over to arch_phys_wc_add() and annotate that the device driver requires systems to boot with PAT disabled, with the nopat kernel parameter. This is a worthy compromise given that the ipath device driver powers the old HTX bus cards that only work in AMD systems, while the newer IB/qib device driver powers all PCI-e cards. The ipath device driver is obsolete, hardware hard to find and because of this this its a reasonable compromise to make to require users of ipath to boot with nopat. Acked-by: Doug Ledford Cc: Doug Ledford Cc: Andy Walls Cc: Hal Rosenstock Cc: Sean Hefty Cc: Suresh Siddha Cc: Rickard Strandqvist Cc: Mike Marciniszyn Cc: Roland Dreier Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Linus Torvalds Cc: Thomas Gleixner Cc: Juergen Gross Cc: Daniel Vetter Cc: Dave Airlie Cc: Bjorn Helgaas Cc: Antonino Daplas Cc: Jean-Christophe Plagniol-Villard Cc: Tomi Valkeinen Cc: Ville Syrjälä Cc: Mel Gorman Cc: Vlastimil Babka Cc: Borislav Petkov Cc: Davidlohr Bueso Cc: Dave Hansen Cc: Arnd Bergmann Cc: Michael S. Tsirkin Cc: Stefan Bader Cc: konrad.wilk@oracle.com Cc: ville.syrjala@linux.intel.com Cc: david.vrabel@citrix.com Cc: jbeulich@suse.com Cc: toshi.kani@hp.com Cc: Roger Pau Monné Cc: infinipath@intel.com Cc: linux-rdma@vger.kernel.org Cc: linux-fbdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: xen-devel@lists.xensource.com Signed-off-by: Luis R. Rodriguez --- drivers/infiniband/hw/ipath/Kconfig | 3 ++ drivers/infiniband/hw/ipath/ipath_driver.c | 18 +++++++---- drivers/infiniband/hw/ipath/ipath_kernel.h | 4 +-- drivers/infiniband/hw/ipath/ipath_wc_x86_64.c | 43 ++++++--------------------- 4 files changed, 26 insertions(+), 42 deletions(-) diff --git a/drivers/infiniband/hw/ipath/Kconfig b/drivers/infiniband/hw/ipath/Kconfig index 1d9bb11..8fe54ff 100644 --- a/drivers/infiniband/hw/ipath/Kconfig +++ b/drivers/infiniband/hw/ipath/Kconfig @@ -9,3 +9,6 @@ config INFINIBAND_IPATH as IP-over-InfiniBand as well as with userspace applications (in conjunction with InfiniBand userspace access). For QLogic PCIe QLE based cards, use the QIB driver instead. + + If you have this hardware you will need to boot with PAT disabled + on your x86-64 systems, use the nopat kernel parameter. diff --git a/drivers/infiniband/hw/ipath/ipath_driver.c b/drivers/infiniband/hw/ipath/ipath_driver.c index bd0caed..441cfe5 100644 --- a/drivers/infiniband/hw/ipath/ipath_driver.c +++ b/drivers/infiniband/hw/ipath/ipath_driver.c @@ -42,6 +42,9 @@ #include #include #include +#ifdef CONFIG_X86_64 +#include +#endif #include "ipath_kernel.h" #include "ipath_verbs.h" @@ -395,6 +398,14 @@ static int ipath_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) unsigned long long addr; u32 bar0 = 0, bar1 = 0; +#ifdef CONFIG_X86_64 + if (WARN(pat_enabled(), + "ipath needs PAT disabled, boot with nopat kernel parameter\n")) { + ret = EINVAL; + goto bail; + } +#endif + dd = ipath_alloc_devdata(pdev); if (IS_ERR(dd)) { ret = PTR_ERR(dd); @@ -542,6 +553,7 @@ static int ipath_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) dd->ipath_kregbase = __ioremap(addr, len, (_PAGE_NO_CACHE|_PAGE_WRITETHRU)); #else + /* XXX: split this properly to enable on PAT */ dd->ipath_kregbase = ioremap_nocache(addr, len); #endif @@ -587,12 +599,8 @@ static int ipath_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) ret = ipath_enable_wc(dd); - if (ret) { - ipath_dev_err(dd, "Write combining not enabled " - "(err %d): performance may be poor\n", - -ret); + if (ret) ret = 0; - } ipath_verify_pioperf(dd); diff --git a/drivers/infiniband/hw/ipath/ipath_kernel.h b/drivers/infiniband/hw/ipath/ipath_kernel.h index e08db70..f0f9471 100644 --- a/drivers/infiniband/hw/ipath/ipath_kernel.h +++ b/drivers/infiniband/hw/ipath/ipath_kernel.h @@ -463,9 +463,7 @@ struct ipath_devdata { /* offset in HT config space of slave/primary interface block */ u8 ipath_ht_slave_off; /* for write combining settings */ - unsigned long ipath_wc_cookie; - unsigned long ipath_wc_base; - unsigned long ipath_wc_len; + int wc_cookie; /* ref count for each pkey */ atomic_t ipath_pkeyrefs[4]; /* shadow copy of struct page *'s for exp tid pages */ diff --git a/drivers/infiniband/hw/ipath/ipath_wc_x86_64.c b/drivers/infiniband/hw/ipath/ipath_wc_x86_64.c index 70c1f3a..7b6e4c8 100644 --- a/drivers/infiniband/hw/ipath/ipath_wc_x86_64.c +++ b/drivers/infiniband/hw/ipath/ipath_wc_x86_64.c @@ -37,7 +37,6 @@ */ #include -#include #include #include "ipath_kernel.h" @@ -122,27 +121,14 @@ int ipath_enable_wc(struct ipath_devdata *dd) } if (!ret) { - int cookie; - ipath_cdbg(VERBOSE, "Setting mtrr for chip to WC " - "(addr %llx, len=0x%llx)\n", - (unsigned long long) pioaddr, - (unsigned long long) piolen); - cookie = mtrr_add(pioaddr, piolen, MTRR_TYPE_WRCOMB, 1); - if (cookie < 0) { - { - dev_info(&dd->pcidev->dev, - "mtrr_add() WC for PIO bufs " - "failed (%d)\n", - cookie); - ret = -EINVAL; - } - } else { - ipath_cdbg(VERBOSE, "Set mtrr for chip to WC, " - "cookie is %d\n", cookie); - dd->ipath_wc_cookie = cookie; - dd->ipath_wc_base = (unsigned long) pioaddr; - dd->ipath_wc_len = (unsigned long) piolen; - } + dd->wc_cookie = arch_phys_wc_add(pioaddr, piolen); + if (dd->wc_cookie < 0) { + ipath_dev_err(dd, "Seting mtrr failed on PIO buffers\n"); + ret = -ENODEV; + } else if (dd->wc_cookie == 0) + ipath_cdbg(VERBOSE, "Set mtrr for chip to WC not needed\n"); + else + ipath_cdbg(VERBOSE, "Set mtrr for chip to WC\n"); } return ret; @@ -154,16 +140,5 @@ int ipath_enable_wc(struct ipath_devdata *dd) */ void ipath_disable_wc(struct ipath_devdata *dd) { - if (dd->ipath_wc_cookie) { - int r; - ipath_cdbg(VERBOSE, "undoing WCCOMB on pio buffers\n"); - r = mtrr_del(dd->ipath_wc_cookie, dd->ipath_wc_base, - dd->ipath_wc_len); - if (r < 0) - dev_info(&dd->pcidev->dev, - "mtrr_del(%lx, %lx, %lx) failed: %d\n", - dd->ipath_wc_cookie, dd->ipath_wc_base, - dd->ipath_wc_len, r); - dd->ipath_wc_cookie = 0; /* even on failure */ - } + arch_phys_wc_del(dd->wc_cookie); }