From patchwork Mon Jan 11 17:30:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Anderson X-Patchwork-Id: 8007191 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 4C2BCBEEED for ; Mon, 11 Jan 2016 17:34:13 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 44E2F20256 for ; Mon, 11 Jan 2016 17:34:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B729E20263 for ; Mon, 11 Jan 2016 17:34:10 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1aIgK6-0007vX-DI; Mon, 11 Jan 2016 17:32:06 +0000 Received: from mail-pf0-x233.google.com ([2607:f8b0:400e:c00::233]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aIgK2-0007jS-Sg for linux-arm-kernel@lists.infradead.org; Mon, 11 Jan 2016 17:32:04 +0000 Received: by mail-pf0-x233.google.com with SMTP id 65so47758599pff.2 for ; Mon, 11 Jan 2016 09:31:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=85BmqCNFCXBgpgKBpShxIQAiJTZgmPdwFP4sXjtPa8o=; b=EGJLPG8Ejo/KDLR0M7ZiVrB3pnZLJk27mZN5CipEopwsW8xkgVvhbx/b6r91+nHzas P3KBoLZKcOXE+azgIVvJzdA/3TNs5TuydevvR7pCAfWvsOEdDRD+gGEPxKRnPZQR4lew f2RUHu4eOqu/CiLRrgXl0+IFRukI9Yt/IlQ5E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=85BmqCNFCXBgpgKBpShxIQAiJTZgmPdwFP4sXjtPa8o=; b=A8yP7LIJMPQgEenu1OE8rUXniq9RW66BTmeBZiDoUQjw5rQFW+TnDcanKT9+zoWB8y amPgXhkD3CtI2vtqLc8HwutKsQ2tbIdj5Ym7W01paE3SnSOfojI5CJtCSUEoFXkzGq0S OoLAry8nGx2HfeXHiEOz4APdB3ABygeWe/mMfwjU5C0Eb6BTQ134mqfprLyjYQnH7bUU 7XUhLaGpW2MBHYTQsSnV+v0OtPQOSQYqd+4GCYgA7L3PTd5GwXs31xD24PThgXjzDRa8 jKR3I6KSfMKluQzBTFcslKIanqdnbYmJgGy9q0Ww0i7c8KYaf43e6P+TCM8nPuJ0PsgS WPpw== X-Gm-Message-State: ALoCoQk1EY8fhUCaDahAftmjqD21fvGFlDrpL+BTVynzhMGawdJXrIxYKneSHAqJFMZai68x6jARb+hcfGX06OSxGq7QrzohQA== X-Received: by 10.98.31.210 with SMTP id l79mr6742553pfj.144.1452533503194; Mon, 11 Jan 2016 09:31:43 -0800 (PST) Received: from tictac.mtv.corp.google.com ([172.22.65.76]) by smtp.gmail.com with ESMTPSA id wa17sm118618460pac.38.2016.01.11.09.31.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 11 Jan 2016 09:31:42 -0800 (PST) From: Douglas Anderson To: linux@arm.linux.org.uk, mchehab@osg.samsung.com, robin.murphy@arm.com, tfiga@chromium.org, m.szyprowski@samsung.com Subject: [PATCH v6 1/5] ARM: dma-mapping: Optimize allocation Date: Mon, 11 Jan 2016 09:30:23 -0800 Message-Id: <1452533428-12762-2-git-send-email-dianders@chromium.org> X-Mailer: git-send-email 2.6.0.rc2.230.g3dd15c0 In-Reply-To: <1452533428-12762-1-git-send-email-dianders@chromium.org> References: <1452533428-12762-1-git-send-email-dianders@chromium.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160111_093203_065155_15F9A345 X-CRM114-Status: GOOD ( 20.86 ) X-Spam-Score: -2.7 (--) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.pinchart+renesas@ideasonboard.com, pawel@osciak.com, mike.looijmans@topic.nl, Dmitry Torokhov , will.deacon@arm.com, Douglas Anderson , linux-kernel@vger.kernel.org, hch@infradead.org, penguin-kernel@i-love.sakura.ne.jp, carlo@caione.org, akpm@linux-foundation.org, dan.j.williams@intel.com, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The __iommu_alloc_buffer() is expected to be called to allocate pretty sizeable buffers. Upon simple tests of video I saw it trying to allocate 4,194,304 bytes. The function tries to allocate large chunks in order to optimize IOMMU TLB usage. The current function is very, very slow. One problem is the way it keeps trying and trying to allocate big chunks. Imagine a very fragmented memory that has 4M free but no contiguous pages at all. Further imagine allocating 4M (1024 pages). We'll do the following memory allocations: - For page 1: - Try to allocate order 10 (no retry) - Try to allocate order 9 (no retry) - ... - Try to allocate order 0 (with retry, but not needed) - For page 2: - Try to allocate order 9 (no retry) - Try to allocate order 8 (no retry) - ... - Try to allocate order 0 (with retry, but not needed) - ... - ... Total number of calls to alloc() calls for this case is: sum(int(math.log(i, 2)) + 1 for i in range(1, 1025)) => 9228 The above is obviously worse case, but given how slow alloc can be we really want to try to avoid even somewhat bad cases. I timed the old code with a device under memory pressure and it wasn't hard to see it take more than 120 seconds to allocate 4 megs of memory! (NOTE: testing was done on kernel 3.14, so possibly mainline would behave differently). A second problem is that allocating big chunks under memory pressure when we don't need them is just not a great idea anyway unless we really need them. We can make due pretty well with smaller chunks so it's probably wise to leave bigger chunks for other users once memory pressure is on. Let's adjust the allocation like this: 1. If a big chunk fails, stop trying to hard and bump down to lower order allocations. 2. Don't try useless orders. The whole point of big chunks is to optimize the TLB and it can really only make use of 2M, 1M, 64K and 4K sizes. We'll still tend to eat up a bunch of big chunks, but that might be the right answer for some users. A future patch could possibly add a new DMA_ATTR that would let the caller decide that TLB optimization isn't important and that we should use smaller chunks. Presumably this would be a sane strategy for some callers. Signed-off-by: Douglas Anderson Acked-by: Marek Szyprowski --- Changes in v6: None Changes in v5: None Changes in v4: - Added Marek's ack Changes in v3: None Changes in v2: - No longer just 1 page at a time, but gives up higher order quickly. - Only tries important higher order allocations that might help us. arch/arm/mm/dma-mapping.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 0eca3812527e..bc9cebfa0891 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1122,6 +1122,9 @@ static inline void __free_iova(struct dma_iommu_mapping *mapping, spin_unlock_irqrestore(&mapping->lock, flags); } +/* We'll try 2M, 1M, 64K, and finally 4K; array must end with 0! */ +static const int iommu_order_array[] = { 9, 8, 4, 0 }; + static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, gfp_t gfp, struct dma_attrs *attrs) { @@ -1129,6 +1132,7 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, int count = size >> PAGE_SHIFT; int array_size = count * sizeof(struct page *); int i = 0; + int order_idx = 0; if (array_size <= PAGE_SIZE) pages = kzalloc(array_size, GFP_KERNEL); @@ -1162,22 +1166,24 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, while (count) { int j, order; - for (order = __fls(count); order > 0; --order) { - /* - * We do not want OOM killer to be invoked as long - * as we can fall back to single pages, so we force - * __GFP_NORETRY for orders higher than zero. - */ - pages[i] = alloc_pages(gfp | __GFP_NORETRY, order); - if (pages[i]) - break; + order = iommu_order_array[order_idx]; + + /* Drop down when we get small */ + if (__fls(count) < order) { + order_idx++; + continue; } - if (!pages[i]) { - /* - * Fall back to single page allocation. - * Might invoke OOM killer as last resort. - */ + if (order) { + /* See if it's easy to allocate a high-order chunk */ + pages[i] = alloc_pages(gfp | __GFP_NORETRY, order); + + /* Go down a notch at first sign of pressure */ + if (!pages[i]) { + order_idx++; + continue; + } + } else { pages[i] = alloc_pages(gfp, 0); if (!pages[i]) goto error;