From patchwork Wed Nov 20 19:11:45 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Willy Tarreau X-Patchwork-Id: 3215001 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id BD1E49F26C for ; Wed, 20 Nov 2013 19:12:24 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 822A4200E3 for ; Wed, 20 Nov 2013 19:12:23 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C9AD220034 for ; Wed, 20 Nov 2013 19:12:21 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjDCC-0000jT-QZ; Wed, 20 Nov 2013 19:12:16 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjDCA-0003Mn-Cg; Wed, 20 Nov 2013 19:12:14 +0000 Received: from 1wt.eu ([62.212.114.60]) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1VjDC7-0003MN-0u for linux-arm-kernel@lists.infradead.org; Wed, 20 Nov 2013 19:12:11 +0000 Received: (from willy@localhost) by mail.home.local (8.14.4/8.14.4/Submit) id rAKJBjdD012182; Wed, 20 Nov 2013 20:11:45 +0100 Date: Wed, 20 Nov 2013 20:11:45 +0100 From: Willy Tarreau To: Arnaud Ebalard Subject: Re: [BUG, REGRESSION?] 3.11.6+, 3.12: GbE iface rate drops to few KB/s Message-ID: <20131120191145.GP8581@1wt.eu> References: <87vbzxd473.fsf@natisbad.org> <20131113072257.GB10591@1wt.eu> <20131117141940.GA18569@1wt.eu> <1384710098.8604.58.camel@edumazet-glaptop2.roam.corp.google.com> <87li0kkhzx.fsf@natisbad.org> <1384869194.8604.92.camel@edumazet-glaptop2.roam.corp.google.com> <20131119174323.GH913@1wt.eu> <1384885910.8604.110.camel@edumazet-glaptop2.roam.corp.google.com> <20131119184121.GN913@1wt.eu> <874n780wzc.fsf@natisbad.org> Mime-Version: 1.0 Content-Disposition: inline In-Reply-To: <874n780wzc.fsf@natisbad.org> User-Agent: Mutt/1.4.2.3i X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20131120_141211_430817_7A63D733 X-CRM114-Status: GOOD ( 23.65 ) X-Spam-Score: -2.5 (--) Cc: Thomas Petazzoni , Florian Fainelli , simon.guinot@sequanux.org, Eric Dumazet , netdev@vger.kernel.org, edumazet@google.com, Cong Wang , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Arnaud, first, thanks for all these tests. On Wed, Nov 20, 2013 at 12:53:43AM +0100, Arnaud Ebalard wrote: (...) > In the end, here are the conclusions *I* draw from this test session, > do not hesitate to correct me: > > - Eric, it seems something changed in linus tree betwen the beginning > of the thread and now, which somehow reduces the effect of the > regression we were seen: I never got back the 256KB/s. > - You revert patch still improves the perf a lot > - It seems reducing MVNETA_TX_DONE_TIMER_PERIOD does not help > - w/ your revert patch, I can confirm that mvneta driver is capable of > doing line rate w/ proper tweak of TCP send window (256KB instead of > 4M) > - It seems I will I have to spend some time on the SATA issues I > previously thought were an artefact of not cleaning my tree during a > debug session [1], i.e. there is IMHO an issue. Could you please try Eric's patch that was just merged into Linus' tree if it was not yet in the kernel you tried : 98e09386c0e tcp: tsq: restore minimal amount of queueing For me it restored the original performance (I saturate the Gbps with about 7 concurrent streams). Further, I wrote the small patch below for mvneta. I'm not sure it's smp-safe but it's a PoC. In mvneta_poll() which currently is only called upon Rx interrupt, it tries to flush all possible remaining Tx descriptors if any. That significantly improved my transfer rate, now I easily achieve 1 Gbps using a single TCP stream on the mirabox. Not tried on the AX3 yet. It also increased the overall connection rate by 10% on empty HTTP responses (small packets), very likely by reducing the dead time between some segments! You'll probably want to give it a try, so here it comes. Cheers, Willy From d1a00e593841223c7d871007b1e1fc528afe8e4d Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Wed, 20 Nov 2013 19:47:11 +0100 Subject: EXP: net: mvneta: try to flush Tx descriptor queue upon Rx interrupts Right now the mvneta driver doesn't handle Tx IRQ, and solely relies on a timer to flush Tx descriptors. This causes jerky output traffic with bursts and pauses, making it difficult to reach line rate with very few streams. This patch tries to improve the situation which is complicated by the lack of public datasheet from Marvell. The workaround consists in trying to flush pending buffers during the Rx polling. The idea is that for symmetric TCP traffic, ACKs received in response to the packets sent will trigger the Rx interrupt and will anticipate the flushing of the descriptors. The results are quite good, a single TCP stream is now capable of saturating a gigabit. This is only a workaround, it doesn't address asymmetric traffic nor datagram based traffic. Signed-off-by: Willy Tarreau --- drivers/net/ethernet/marvell/mvneta.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 5aed8ed..59e1c86 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2013,6 +2013,26 @@ static int mvneta_poll(struct napi_struct *napi, int budget) } pp->cause_rx_tx = cause_rx_tx; + + /* Try to flush pending Tx buffers if any */ + if (test_bit(MVNETA_F_TX_DONE_TIMER_BIT, &pp->flags)) { + int tx_todo = 0; + + mvneta_tx_done_gbe(pp, + (((1 << txq_number) - 1) & + MVNETA_CAUSE_TXQ_SENT_DESC_ALL_MASK), + &tx_todo); + + if (tx_todo > 0) { + mod_timer(&pp->tx_done_timer, + jiffies + msecs_to_jiffies(MVNETA_TX_DONE_TIMER_PERIOD)); + } + else { + clear_bit(MVNETA_F_TX_DONE_TIMER_BIT, &pp->flags); + del_timer(&pp->tx_done_timer); + } + } + return rx_done; }