From patchwork Mon Dec 1 07:28:02 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Willy Tarreau X-Patchwork-Id: 5410291 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id BB7559F1CD for ; Mon, 1 Dec 2014 07:30:50 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 98240202E6 for ; Mon, 1 Dec 2014 07:30:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1E87820268 for ; Mon, 1 Dec 2014 07:30:48 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XvLPL-0003ER-2l; Mon, 01 Dec 2014 07:28:31 +0000 Received: from 1wt.eu ([62.212.114.60]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1XvLPH-0003C2-67 for linux-arm-kernel@lists.infradead.org; Mon, 01 Dec 2014 07:28:28 +0000 Received: (from willy@localhost) by pcw.home.local (8.14.3/8.14.3/Submit) id sB17S2k3021997; Mon, 1 Dec 2014 08:28:02 +0100 Date: Mon, 1 Dec 2014 08:28:02 +0100 From: Willy Tarreau To: Maggie Mae Roxas Subject: Re: Issue found in Armada 370: "No buffer space available" error during continuous ping Message-ID: <20141201072802.GB21731@1wt.eu> References: <20140717081527.GJ14723@1wt.eu> <20140721054405.GK21834@1wt.eu> <20140721070303.GM21834@1wt.eu> <20140723061659.GE30488@1wt.eu> Mime-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20141130_232827_779619_D1ED3613 X-CRM114-Status: GOOD ( 17.00 ) X-Spam-Score: -0.0 (/) Cc: Thomas Petazzoni , Gregory CLEMENT , Maria Cecilia Caluya , Ezequiel Garcia , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Maggie, On Mon, Dec 01, 2014 at 02:26:49PM +0800, Maggie Mae Roxas wrote: > Hi Willy, Thomas. > Good day. > > I am reopening this discussion because we found an unusual behavior > after using this combination that we thought was OK as discussed in > the previous messages of this thread: > > > - use 3.13.9 mvneta.c > > - apply cd71e246c16b30e3f396a85943d5f596202737ba > > - revert 4f3a4f701b59a3e4b5c8503ac3d905c0a326f922 > > Specifically, if we apply above, the "No buffer space available" error > during continuous ping does NOT occur anymore. > # Attached: with_patch_3_13_9_no_buffer_space_solved.txt > > However, after continuous and further testing, we encounter the ff. issues: > 1. Low throughput during iperf when Armada 370 device is set as iperf > client. For example, in 1000Mbits/s, we only get below 140Mbits/s. Yes that was the intent of the original fix. We recently diagnosed the issue related to "no buffer space available". What happens is that the "ping" utility uses a very small socket buffer. It sends a few packets, and the NIC doesn't send interrupts until the TX interrupt count is reached, so the Tx skbs are not freed and the socket buffers remain full. The only solution at the moment is to make the NIC emit an IRQ for each Tx packet. I'm still trying to find a better way to do this (either find a way to make the NIC emit an IRQ once the Tx queue is empty or adjust the IRQ delay when adding more packets, though it creates a race condition). In the mean time you can apply the attached patch. I haven't submitted it yet only by lack of time :-( Best regards, Willy From 01b23da3607dbce1d1abfe5b7f092de11ae327cf Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Sat, 25 Oct 2014 19:12:49 +0200 Subject: net: mvneta: fix TX coalesce interrupt mode The mvneta driver sets the amount of Tx coalesce packets to 16 by default. Normally that does not cause any trouble since the driver uses a much larger Tx ring size (532 packets). But some sockets might run with very small buffers, much smaller than the equivalent of 16 packets. This is what ping is doing for example, by setting SNDBUF to 324 bytes rounded up to 2kB by the kernel. The problem is that there is no documented method to force a specific packet to emit an interrupt (eg: the last of the ring) nor is it possible to make the NIC emit an interrupt after a given delay. In this case, it causes trouble, because when ping sends packets over its raw socket, the few first packets leave the system, and the first 15 packets will be emitted without an IRQ being generated, so without the skbs being freed. And since the socket's buffer is small, there's no way to reach that amount of packets, and the ping ends up with "send: no buffer available" after sending 6 packets. Running with 3 instances of ping in parallel is enough to hide the problem, because with 6 packets per instance, that's 18 packets total, which is enough to grant a Tx interrupt before all are sent. The original driver in the LSP kernel worked around this design flaw by using a software timer to clean up the Tx descriptors. This timer was slow and caused terrible network performance on some Tx-bound workloads (such as routing) but was enough to make tools like ping work correctly. Instead here, we simply set the packet counts before interrupt to 1. This ensures that each packet sent will produce an interrupt. NAPI takes care of coalescing interrupts since the interrupt is disabled once generated. No measurable performance impact nor CPU usage were observed on small nor large packets, including when saturating the link on Tx, and this fixes tools like ping which rely on too small a send buffer. This fix needs to be backported to stable kernels starting with 3.10. Signed-off-by: Willy Tarreau --- drivers/net/ethernet/marvell/mvneta.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 4762994..35bfba7 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -214,7 +214,7 @@ /* Various constants */ /* Coalescing */ -#define MVNETA_TXDONE_COAL_PKTS 16 +#define MVNETA_TXDONE_COAL_PKTS 1 #define MVNETA_RX_COAL_PKTS 32 #define MVNETA_RX_COAL_USEC 100