From patchwork Mon Apr 7 17:41:06 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 3946601 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 24EED9F369 for ; Mon, 7 Apr 2014 17:41:49 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2E5C2202E6 for ; Mon, 7 Apr 2014 17:41:48 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 09102202BE for ; Mon, 7 Apr 2014 17:41:47 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXDYA-0005nC-W9; Mon, 07 Apr 2014 17:41:39 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXDY8-0005el-GI; Mon, 07 Apr 2014 17:41:36 +0000 Received: from quartz.orcorp.ca ([184.70.90.242]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WXDY5-0005eB-70 for linux-arm-kernel@lists.infradead.org; Mon, 07 Apr 2014 17:41:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=obsidianresearch.com; s=rsa1; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=IXdIPf0Tv4/EOyMz14n6/Ad5uagEUGPjn6zKyXXdrPk=; b=T46eBG1wnX0cPX/VqOTFj3NDSkDJvh7VmWnXV+qqc7qTUEma5Ikl0MC6hQdCGbRG216PQsi90DJARb5GHwr7tvY1p+uec6J1TNcu4kSFNEwZecFTAbHp6MS04viLLo2Rx/y204usUxICsGS2TVD0sYhmdcXScJBLmYrUiWVr0v0=; Received: from [10.0.0.161] (helo=jggl.edm.orcorp.ca) by quartz.orcorp.ca with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1WXDXf-0000Nu-2Y; Mon, 07 Apr 2014 11:41:07 -0600 Received: from jgg by jggl.edm.orcorp.ca with local (Exim 4.80) (envelope-from ) id 1WXDXe-0004Lr-Mu; Mon, 07 Apr 2014 11:41:06 -0600 Date: Mon, 7 Apr 2014 11:41:06 -0600 From: Jason Gunthorpe To: Neil Greatorex , Matthew Minter Subject: Re: Intel I350 mini-PCIe card (igb) on Mirabox (mvebu / Armada 370) Message-ID: <20140407174106.GD9952@obsidianresearch.com> References: <20140326201243.GA1536@obsidianresearch.com> <20140326214259.GA12330@obsidianresearch.com> <20140327044054.GA22681@obsidianresearch.com> <20140406185833.GI29787@1wt.eu> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.161 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140407_134133_410219_47E0B710 X-CRM114-Status: GOOD ( 25.92 ) X-Spam-Score: -2.0 (--) Cc: Thomas Petazzoni , Willy Tarreau , linux-arm-kernel X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Sun, Apr 06, 2014 at 10:57:40PM +0100, Neil Greatorex wrote: > Willy, > > On Sun, 6 Apr 2014, Willy Tarreau wrote: > > >As you can see, the sequence is exactly the same for both ports. The > >first one has no problem performing the readl(), but the second one > >cannot. Both of them got the memory address returned by a call to > >pci_iomap(dev, 0, 0). I could verify that the pci_resource_len() for > >both is 524288 (0x80000). > > > >The last "hwaddr=f0400000" is printed just before calling rd32() and > >shows that it was still OK there. Since the resource flags are 0x40200, > >we only have IORESOURCE_MEM so pci_iomap() calls ioremap_nocache(). > > > > Good work - I've been doing similar things myself! I can confirm > that I see exactly the same thing with similar printks: > > First port: > [ 1809.452878] igb 0000:01:00.0: enabling bus mastering > [ 1809.453098] igb 0000:01:00.0 (unregistered net_device): hw_addr > is f1000000, start=e0000000, len=80000, flags=40200 > [ 1809.453109] igb 0000:01:00.0 (unregistered net_device): About to > read from offset 18 > [ 1809.453120] igb 0000:01:00.0 (unregistered net_device): Read from > 18 returned 1400c0 > > Second port: > [ 1809.459445] igb 0000:01:00.1: enabling bus mastering > [ 1809.459563] igb 0000:01:00.1 (unregistered net_device): hw_addr is > f1100000, start=e0100000, len=80000, flags=40200 > [ 1809.459573] igb 0000:01:00.1 (unregistered net_device): About to read > from offset 18 > [ 1809.459581] Unhandled fault: external abort on non-linefetch > (0x1008) at 0xf1100018 > > In the output above, the start= part shows the physical address and > hw_addr shows the mapped address. This is very similar to what Matthew Minter is seeing on Hot Plug with AHCI. (See 'Armada XP (mvebu) PCIe memory (BAR/window) re-allocation' thread) That probably says it is somehow mbus related - dumping the mbus registers when the fault happens should clarify that point. The size would a good place to check first. > The physical addresses match those given in the lspci -vvv output > (see https://gist.github.com/ngreatorex/9772195). I don't know > enough about PCIe, the SoC *or* the Intel card to know if these > addresses look correct or even sane! I did wonder if there was some > issue due to the fact that the resources for 01:00.0 and 01:00.1 > overlap, but I would guess(!?) that it's common in hardware that > presents multiple devices. Which overlap? To be very clear, PCI BARs, should never overlap. The bridge windows should fully contain downstream bars: 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) Bus: primary=00, secondary=01, subordinate=02, sec-latency=0 Memory behind bridge: e0000000-e02fffff 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Region 0: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=512K] 01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Region 0: Memory at e0100000 (32-bit, non-prefetchable) [disabled] [size=512K] Looks good to me. HOWEVER, looking now very closely: 00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) Memory behind bridge: e0000000-e02fffff 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6710 (rev 01) (prog-if 00 [Normal decode]) Memory behind bridge: e0300000-e03fffff This is certainly wrong, MBUS requires special alignment and sizing. 0x300000 is not a size which is a power of two, and the next window starts right after. We need to see the first bridge use e0000000-e03fffff Just to confirm, what does something like the below say for you guys? Regards, Jason diff --git a/drivers/bus/mvebu-mbus.c b/drivers/bus/mvebu-mbus.c index 2ac754e..dfeb0dd 100644 --- a/drivers/bus/mvebu-mbus.c +++ b/drivers/bus/mvebu-mbus.c @@ -56,6 +56,7 @@ #include #include #include +#include /* * DDR target is the same on all platforms. @@ -266,6 +267,8 @@ static int mvebu_mbus_setup_window(struct mvebu_mbus_state *mbus, mbus->soc->win_cfg_offset(win); u32 ctrl, remap_addr; + WARN_ON(!is_power_of_2(size)); + ctrl = ((size - 1) & WIN_CTRL_SIZE_MASK) | (attr << WIN_CTRL_ATTR_SHIFT) | (target << WIN_CTRL_TGT_SHIFT) |