From patchwork Mon Apr 7 21:58:36 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neil Greatorex X-Patchwork-Id: 3947201 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id F127A9F371 for ; Mon, 7 Apr 2014 21:59:20 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 00706201BA for ; Mon, 7 Apr 2014 21:59:20 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 566992017B for ; Mon, 7 Apr 2014 21:59:18 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXHZR-0000Fp-3H; Mon, 07 Apr 2014 21:59:13 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXHZO-0003WW-PD; Mon, 07 Apr 2014 21:59:10 +0000 Received: from mail-wg0-x22e.google.com ([2a00:1450:400c:c00::22e]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXHZK-0003V0-T7 for linux-arm-kernel@lists.infradead.org; Mon, 07 Apr 2014 21:59:07 +0000 Received: by mail-wg0-f46.google.com with SMTP id b13so61516wgh.29 for ; Mon, 07 Apr 2014 14:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fatboyfat.co.uk; s=google; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version:content-type; bh=yxaspu4sn2Cf05bY6bWlW62P9+OBLzGaHWy3Cw1fyrk=; b=heB4ZMKDCtR3Yra05r4MggaUxEu4DPt1DVyHSvr5RLWz3oLRkx8aqcmQUtX6KCKDfx g7ncvFbwHxy9MQemWWJTTryVwARmA3/IlKj2Ev8g21OQs+CxAacYF0PfL+twr8yJk3pt yUkgTGNswoaOTsdxw22kdLgUOkLLswGLDVzVU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version:content-type; bh=yxaspu4sn2Cf05bY6bWlW62P9+OBLzGaHWy3Cw1fyrk=; b=mZ/cTYoj8KNIKaEkGKCGV2kva6KhGYhYXI7Tj66absbeDzOkvhx2rt90ySFlKYUG8j GuWel2bKUYBvPnaiwxZpzOR2bsa2ZRfLdv11VA1qP7KuZvN9fHEhKEI1sSo0qAXoihbp fTOYwzGw/gd0z9LZSbHE9t0h2ZybHHPJm3s32N5uputmZ+9Gsii2Jh3Kxj8uANYtDzkB nYyaqRzE2LraU/Q0Y0pyM6Djo3Et4lCBZoIc0uJLU8kuhXcETu2iJ503MIfT2wH4BzAf +Pf4p5bKqrx/p7rya+6PCUYs2m4UjGDa1oVeiUJUrnFFMU66vT/4/Sb+FJJarNE59qar xPcg== X-Gm-Message-State: ALoCoQm068F9lTC8nVq/2hQCnfJh8zgMGydvPmTTw6qZVs0VK610EGTuOwbglyJqhqdbQ8krKsaI X-Received: by 10.194.60.37 with SMTP id e5mr34278wjr.32.1396907922384; Mon, 07 Apr 2014 14:58:42 -0700 (PDT) Received: from vroombuntu.local (97e4e50d.skybroadband.com. [151.228.229.13]) by mx.google.com with ESMTPSA id jd2sm848602wic.9.2014.04.07.14.58.40 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 07 Apr 2014 14:58:41 -0700 (PDT) Date: Mon, 7 Apr 2014 22:58:36 +0100 (BST) From: Neil Greatorex X-X-Sender: neil@vroombuntu To: Jason Gunthorpe , Thomas Petazzoni , Willy Tarreau Subject: Re: Intel I350 mini-PCIe card (igb) on Mirabox (mvebu / Armada 370) In-Reply-To: <20140407204817.GB20736@obsidianresearch.com> Message-ID: References: <20140326214259.GA12330@obsidianresearch.com> <20140327044054.GA22681@obsidianresearch.com> <20140406185833.GI29787@1wt.eu> <20140407174106.GD9952@obsidianresearch.com> <20140407204817.GB20736@obsidianresearch.com> User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140407_175907_178300_98296FCF X-CRM114-Status: GOOD ( 29.66 ) X-Spam-Score: -2.0 (--) Cc: Matthew Minter , linux-arm-kernel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Jason, Thomas, Willy, On Mon, 7 Apr 2014, Jason Gunthorpe wrote: >>> HOWEVER, looking now very closely: >>> >>> 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) >>> Memory behind bridge: e0000000-e02fffff >>> 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) >>> Memory behind bridge: e0300000-e03fffff >>> >>> This is certainly wrong, MBUS requires special alignment and sizing. >>> 0x300000 is not a size which is a power of two, and the next window >>> starts right after. > >> Interesting. Does the PCI code provide a way to specify that the sizes >> much be a power of 2? I don't fully understand the implications but >> would it be possible to assign just one MBUS window for the whole of >> the PCIe memory instead? > > I know this has come up before, but I don't recall where things > settled out.. mvebu_pcie_align_resource is the function to look at, > the size at that point needs to be rounded up to a power of two and > communicated back to the caller. > > Alternately, I belive Thomas once discussed using multiple mbus > windows for these non-aligned requests. > >>> Just to confirm, what does something like the below say for you guys? >> >> See https://gist.github.com/ngreatorex/10025253 for the dmesg output. >> I have also included the contents of >> /sys/kernel/debug/mvebu-mbus/devices both before and after the >> modprobe / oops. As you can see I get a total of 3 WARNINGs - one at >> boot for the xHCI controller, and two when inserting igb.ko. Note that >> this time I did this with both ports enabled. > > Yep, that is certainly the root problem - most likely for > everyone. This also explains why Will saw success when he reverted > that unrelated patch. That change altered the allocation pattern of > the BARs and it just so happened to make things fall out properly. > I have finally managed to get the card working on both ports! Of course, to do so I have added some nice kludges to the code that now need to be implemented properly, but it is verification of what the problem is and how to fix it! I have included the patch at the end of this e-mail. It probably won't apply cleanly for you as I have other dev_dbg calls in pci-mvebu.c. What I did was to alter mvebu_pcie_align_resource to make the bridge memory resource aligned to 4M. This had the effect that the 2nd bridge to the xHCI controller was bumped to address 0xe0400000 instead of 0xe0300000. I then also made it so that when we request the MBUS window to be set up we ensure that the size is a power of 2. This has the effect of creating the windows and addresses how we want them: Relevant part of lspci -vvv: 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) Memory behind bridge: e0000000-e02fffff 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) Memory behind bridge: e0400000-e04fffff cat /sys/kernel/debug/mvebu-mbus/devices: [00] 00000000e8010000 - 00000000e8020000 : 0004:00e0 (remap 0000000000010000) [01] disabled [02] disabled [03] disabled [04] disabled [05] disabled [06] disabled [07] disabled [08] 00000000fff00000 - 0000000100000000 : 0001:00e0 [09] 00000000e0400000 - 00000000e0500000 : 0008:00e8 [10] 00000000e0000000 - 00000000e0400000 : 0004:00e8 [11] disabled [12] disabled [13] disabled [14] disabled [15] disabled [16] disabled [17] disabled [18] disabled [19] disabled Now, over to the experts to implement this properly :-) Thanks to Jason, Thomas and Willy for your help with tracking this down. Cheers, Neil From e68870135cd54609d0421746ccdc1cb58ebbd4dd Mon Sep 17 00:00:00 2001 From: Neil Greatorex Date: Mon, 7 Apr 2014 22:33:22 +0100 Subject: [PATCH] pci: host: mvebu - Test fix for Mirabox PCI issues Ensure that bridge MBUS window is aligned at 4M to bump up 2nd bridge to address e0400000 instead of e0300000. Also ensure that when we request the MBUS window, it's size is a power of 2. --- drivers/pci/host/pci-mvebu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/pci/host/pci-mvebu.c b/drivers/pci/host/pci-mvebu.c index afd0dce..5e0144c 100644 --- a/drivers/pci/host/pci-mvebu.c +++ b/drivers/pci/host/pci-mvebu.c @@ -364,6 +364,9 @@ static void mvebu_pcie_handle_membase_change(struct mvebu_pcie_port *port) (((port->bridge.memlimit & 0xFFF0) << 16) | 0xFFFFF) - port->memwin_base; + if (!is_power_of_2(port->memwin_size)) + port->memwin_size = 1 << fls(port->memwin_size); + mvebu_mbus_add_window_by_id(port->mem_target, port->mem_attr, port->memwin_base, port->memwin_size); } @@ -728,7 +731,7 @@ static resource_size_t mvebu_pcie_align_resource(struct pci_dev *dev, if (res->flags & IORESOURCE_IO) return round_up(start, max_t(resource_size_t, SZ_64K, size)); else if (res->flags & IORESOURCE_MEM) - return round_up(start, max_t(resource_size_t, SZ_1M, size)); + return round_up(start, max_t(resource_size_t, SZ_4M, size)); else return start; }