From patchwork Thu Nov 21 00:44:30 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Willy Tarreau X-Patchwork-Id: 3216211 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 71C59C045B for ; Thu, 21 Nov 2013 00:45:13 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 1FA0420780 for ; Thu, 21 Nov 2013 00:45:12 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4CE2C2076C for ; Thu, 21 Nov 2013 00:45:10 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjIOI-0002RJ-Em; Thu, 21 Nov 2013 00:45:06 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjIOF-0004a2-RU; Thu, 21 Nov 2013 00:45:03 +0000 Received: from 1wt.eu ([62.212.114.60]) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjIOB-0004Yr-7I for linux-arm-kernel@lists.infradead.org; Thu, 21 Nov 2013 00:45:02 +0000 Received: (from willy@localhost) by mail.home.local (8.14.4/8.14.4/Submit) id rAL0iUnk013882; Thu, 21 Nov 2013 01:44:30 +0100 Date: Thu, 21 Nov 2013 01:44:30 +0100 From: Willy Tarreau To: Arnaud Ebalard Subject: Re: [BUG, REGRESSION?] 3.11.6+, 3.12: GbE iface rate drops to few KB/s Message-ID: <20131121004430.GX8581@1wt.eu> References: <1384710098.8604.58.camel@edumazet-glaptop2.roam.corp.google.com> <87li0kkhzx.fsf@natisbad.org> <1384869194.8604.92.camel@edumazet-glaptop2.roam.corp.google.com> <20131119174323.GH913@1wt.eu> <1384885910.8604.110.camel@edumazet-glaptop2.roam.corp.google.com> <20131119184121.GN913@1wt.eu> <874n780wzc.fsf@natisbad.org> <20131120191145.GP8581@1wt.eu> <87txf692zx.fsf@natisbad.org> <20131120215435.GT8581@1wt.eu> Mime-Version: 1.0 Content-Disposition: inline In-Reply-To: <20131120215435.GT8581@1wt.eu> User-Agent: Mutt/1.4.2.3i X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20131120_194500_666329_B4EF3E99 X-CRM114-Status: GOOD ( 42.33 ) X-Spam-Score: -2.5 (--) Cc: Thomas Petazzoni , Florian Fainelli , simon.guinot@sequanux.org, Eric Dumazet , netdev@vger.kernel.org, edumazet@google.com, Cong Wang , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Arnaud, On Wed, Nov 20, 2013 at 10:54:35PM +0100, Willy Tarreau wrote: > I'm currently trying to implement TX IRQ handling. I found the registers > description in the neta driver that is provided in Marvell's LSP kernel > that is shipped with some devices using their CPUs. This code is utterly > broken (eg: splice fails with -EBADF) but I think the register descriptions > could be trusted. > > I'd rather have real IRQ handling than just relying on mvneta_poll(), so > that we can use it for asymmetric traffic/routing/whatever. OK it paid off. And very well :-) I did it at once and it worked immediately. I generally don't like this because I always fear that some bug was left there hidden in the code. I have only tested it on the Mirabox, so I'll have to try on the OpenBlocks AX3-4 and on the XP-GP board for some SMP stress tests. I upgraded my Mirabox to latest Linus' git (commit 5527d151) and compared with and without the patch. without : - need at least 12 streams to reach gigabit. - 60% of idle CPU remains at 1 Gbps - HTTP connection rate on empty objects is 9950 connections/s - cumulated outgoing traffic on two ports reaches 1.3 Gbps with the patch : - a single stream easily saturates the gigabit - 87% of idle CPU at 1 Gbps (12 streams, 90% idle at 1 stream) - HTTP connection rate on empty objects is 10250 connections/s - I saturate the two gig ports at 99% CPU, so 2 Gbps sustained output. BTW I must say I was impressed to see that big an improvement in CPU usage between 3.10 and 3.13, I suspect some of the Tx queue improvements that Eric has done in between account for this. I cut the patch in 3 parts : - one which reintroduces the hidden bits of the driver - one which replaces the timer with the IRQ - one which changes the default Tx coalesce from 16 to 4 packets (larger was preferred with the timer, but less is better now). I'm attaching them, please test them on your device. Note that this is *not* for inclusion at the moment as it has not been tested on the SMP CPUs. Cheers, Willy From b77b32dbffdfffc6aa21fa230054e09e2a4cd227 Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Wed, 20 Nov 2013 23:58:30 +0100 Subject: net: mvneta: add missing bit descriptions for interrupt masks and causes Marvell has not published the chip's datasheet yet, so it's very hard to find the relevant bits to manipulate to change the IRQ behaviour. Fortunately, these bits are described in the proprietary LSP patch set which is publicly available here : http://www.plugcomputer.org/downloads/mirabox/ So let's put them back in the driver in order to reduce the burden of current and future maintenance. Signed-off-by: Willy Tarreau --- drivers/net/ethernet/marvell/mvneta.c | 44 +++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index b8e232b..6630690 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -101,16 +101,56 @@ #define MVNETA_CPU_RXQ_ACCESS_ALL_MASK 0x000000ff #define MVNETA_CPU_TXQ_ACCESS_ALL_MASK 0x0000ff00 #define MVNETA_RXQ_TIME_COAL_REG(q) (0x2580 + ((q) << 2)) + +/* Exception Interrupt Port/Queue Cause register */ + #define MVNETA_INTR_NEW_CAUSE 0x25a0 -#define MVNETA_RX_INTR_MASK(nr_rxqs) (((1 << nr_rxqs) - 1) << 8) #define MVNETA_INTR_NEW_MASK 0x25a4 + +/* bits 0..7 = TXQ SENT, one bit per queue. + * bits 8..15 = RXQ OCCUP, one bit per queue. + * bits 16..23 = RXQ FREE, one bit per queue. + * bit 29 = OLD_REG_SUM, see old reg ? + * bit 30 = TX_ERR_SUM, one bit for 4 ports + * bit 31 = MISC_SUM, one bit for 4 ports + */ +#define MVNETA_TX_INTR_MASK(nr_txqs) (((1 << nr_txqs) - 1) << 0) +#define MVNETA_TX_INTR_MASK_ALL (0xff << 0) +#define MVNETA_RX_INTR_MASK(nr_rxqs) (((1 << nr_rxqs) - 1) << 8) +#define MVNETA_RX_INTR_MASK_ALL (0xff << 8) + #define MVNETA_INTR_OLD_CAUSE 0x25a8 #define MVNETA_INTR_OLD_MASK 0x25ac + +/* Data Path Port/Queue Cause Register */ #define MVNETA_INTR_MISC_CAUSE 0x25b0 #define MVNETA_INTR_MISC_MASK 0x25b4 + +#define MVNETA_CAUSE_PHY_STATUS_CHANGE BIT(0) +#define MVNETA_CAUSE_LINK_CHANGE BIT(1) +#define MVNETA_CAUSE_PTP BIT(4) + +#define MVNETA_CAUSE_INTERNAL_ADDR_ERR BIT(7) +#define MVNETA_CAUSE_RX_OVERRUN BIT(8) +#define MVNETA_CAUSE_RX_CRC_ERROR BIT(9) +#define MVNETA_CAUSE_RX_LARGE_PKT BIT(10) +#define MVNETA_CAUSE_TX_UNDERUN BIT(11) +#define MVNETA_CAUSE_PRBS_ERR BIT(12) +#define MVNETA_CAUSE_PSC_SYNC_CHANGE BIT(13) +#define MVNETA_CAUSE_SERDES_SYNC_ERR BIT(14) + +#define MVNETA_CAUSE_BMU_ALLOC_ERR_SHIFT 16 +#define MVNETA_CAUSE_BMU_ALLOC_ERR_ALL_MASK (0xF << MVNETA_CAUSE_BMU_ALLOC_ERR_SHIFT) +#define MVNETA_CAUSE_BMU_ALLOC_ERR_MASK(pool) (1 << (MVNETA_CAUSE_BMU_ALLOC_ERR_SHIFT + (pool))) + +#define MVNETA_CAUSE_TXQ_ERROR_SHIFT 24 +#define MVNETA_CAUSE_TXQ_ERROR_ALL_MASK (0xFF << MVNETA_CAUSE_TXQ_ERROR_SHIFT) +#define MVNETA_CAUSE_TXQ_ERROR_MASK(q) (1 << (MVNETA_CAUSE_TXQ_ERROR_SHIFT + (q))) + #define MVNETA_INTR_ENABLE 0x25b8 #define MVNETA_TXQ_INTR_ENABLE_ALL_MASK 0x0000ff00 -#define MVNETA_RXQ_INTR_ENABLE_ALL_MASK 0xff000000 +#define MVNETA_RXQ_INTR_ENABLE_ALL_MASK 0xff000000 // note: neta says it's 0x000000FF + #define MVNETA_RXQ_CMD 0x2680 #define MVNETA_RXQ_DISABLE_SHIFT 8 #define MVNETA_RXQ_ENABLE_MASK 0x000000ff