From patchwork Mon Feb 29 23:17:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laura Abbott X-Patchwork-Id: 8460201 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id BE82AC0553 for ; Mon, 29 Feb 2016 23:20:17 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C594320204 for ; Mon, 29 Feb 2016 23:20:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C1BBC2022A for ; Mon, 29 Feb 2016 23:20:11 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1aaX5U-0008KT-B4; Mon, 29 Feb 2016 23:18:48 +0000 Received: from mx1.redhat.com ([209.132.183.28]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aaX4y-0007a3-UU for linux-arm-kernel@lists.infradead.org; Mon, 29 Feb 2016 23:18:19 +0000 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id C67ED73039; Mon, 29 Feb 2016 23:17:55 +0000 (UTC) Received: from [10.3.112.35] (ovpn-112-35.phx2.redhat.com [10.3.112.35]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u1TNHrkx021649; Mon, 29 Feb 2016 18:17:53 -0500 Subject: Re: [PATCH 0/2] arm64, cma, gicv3-its: Use CMA for allocation of large device tables To: Marc Zyngier , Robert Richter References: <1456398164-16864-1-git-send-email-rrichter@caviumnetworks.com> <56D42199.7040207@arm.com> <20160229122511.GS24726@rric.localdomain> <56D44812.6000309@arm.com> From: Laura Abbott Message-ID: <56D4D1A1.9060305@redhat.com> Date: Mon, 29 Feb 2016 15:17:53 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D44812.6000309@arm.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 29 Feb 2016 23:17:55 +0000 (UTC) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160229_151817_209495_2786145F X-CRM114-Status: GOOD ( 28.73 ) X-Spam-Score: -6.9 (------) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Tirumalesh Chalamarla , Greg Kroah-Hartman , Thomas Gleixner , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 02/29/2016 05:30 AM, Marc Zyngier wrote: > On 29/02/16 12:25, Robert Richter wrote: >> On 29.02.16 10:46:49, Marc Zyngier wrote: >>> On 25/02/16 11:02, Robert Richter wrote: >>>> From: Robert Richter >>>> >>>> This series implements the use of CMA for allocation of large device >>>> tables for the arm64 gicv3 interrupt controller. >>>> >>>> There are 2 patches, the first is for early activation of cma, which >>>> needs to be done before interrupt initialization to make it available >>>> to the gicv3. The second implements the use of CMA to allocate >>>> gicv3-its device tables. >>>> >>>> This solves the problem where mem allocation is limited to 4MB. A >>>> previous patch sent to the list to address this that instead increases >>>> FORCE_MAX_ZONEORDER becomes obsolete. >>> >>> I think you're looking at the problem the wrong way. Instead of going >>> through CMA directly, I'd rather go through the normal DMA API >>> (dma_alloc_coherent), which can itself try CMA (should it be enabled). >>> >>> That will give you all the benefit of the CMA allocation, and also make >>> the driver more robust. I meant to do this for a while, and never found >>> the time. Any chance you could have a look? >> >> I was considering this first, and in fact the backend used is the >> same. The problem is that irq initialization is much more earlier than >> standard device probing. The gic even does not have its own struct >> device and is not initialized like devices are. This makes the whole >> dma_alloc_coherent() approach not feasable, at least this would >> require introducing and using a dev struct for the gic. But still this >> migth not work as it could be too early during boot. I also think >> there were reasons not implementing the gic as a device. >> >> I was following more the approach of iommu/mmu implementations which >> use dma_alloc_from_contiguous() directly. I think this is more close >> to the device tables for its. >> >> Code path of dma_alloc_coherent(): >> >> dma_alloc_coherent() >> v >> dma_alloc_attrs() <---- Requires get_dma_ops(dev) != NULL >> v >> dma_alloc_from_coherent() >> v >> ... >> >> The difference it that dma_alloc_coherent() tries cma first and then >> proceeds with ops->alloc() (which is __dma_alloc() for arm64) if >> dma_alloc_from_coherent() fails. In my implementation I am directly >> using dma_alloc_from_coherent() and only for large mem sizes. >> >> So both approaches uses finally the same allocation, but for gicv3-its >> the generic dma framework is not used since the gic is not implemented >> as a device. > > And that's what I propose we change. > > The core GIC itself indeed isn't a device, and I'm not proposing we make > it a device (yet). But the ITS is only used much later in the game, and > we could move the table allocation to a different time (when the actual > domains are allocated, for example...). Then, we'd have a set of devices > available, and the DMA API is our friend again. > > M. > I did the first drop of CMA in the DMA APIs for arm64. When adding that, it was decided to disallow dma_alloc calls without a valid device pointer (c666e8d5cae7 "arm64: Warn on NULL device structure for dma APIs") so if the GIC code wants to use dma_alloc it _must_ have a proper device. If the device shift still isn't feasible, a better approach might be what powerpc did for kvm (arch/powerpc/kvm/book3s_hv_builtin.c). This calls the cma_alloc functions directly and skips trying to work around the DMA layer. With either option, I don't think the early initialization approach proposed is great. If we want CMA early, it's probably be just to explicitly initialize it early rather than trying to do it from two places. Something like: Thanks, Laura diff --git a/include/linux/cma.h b/include/linux/cma.h index 29f9e77..a26712a 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -28,4 +28,5 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, struct cma **res_cma); extern struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align); extern bool cma_release(struct cma *cma, const struct page *pages, unsigned int count); +extern int __init cma_init_reserved_areas(void); #endif diff --git a/init/main.c b/init/main.c index 58c9e37..a92bdb8 100644 --- a/init/main.c +++ b/init/main.c @@ -81,6 +81,7 @@ #include #include #include +#include #include #include @@ -492,6 +493,7 @@ static void __init mm_init(void) pgtable_init(); vmalloc_init(); ioremap_huge_init(); + cma_init_reserved_areas(); } asmlinkage __visible void __init start_kernel(void) diff --git a/mm/cma.c b/mm/cma.c index ea506eb..42278d4 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -142,7 +142,7 @@ err: return -EINVAL; } -static int __init cma_init_reserved_areas(void) +int __init cma_init_reserved_areas(void) { int i; @@ -155,7 +155,6 @@ static int __init cma_init_reserved_areas(void) return 0; } -core_initcall(cma_init_reserved_areas); /** * cma_init_reserved_mem() - create custom contiguous area from reserved memory