| Message ID | 20200625074330.13668-3-song.bao.hua@hisilicon.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | make dma_alloc_coherent NUMA-aware by per-NUMA CMA | expand |
On 2020-06-25 08:43, Barry Song wrote: > Right now, smmu is using dma_alloc_coherent() to get memory to save queues > and tables. Typically, on ARM64 server, there is a default CMA located at > node0, which could be far away from node2, node3 etc. > with this patch, smmu will get memory from local numa node to save command > queues and page tables. that means dma_unmap latency will be shrunk much. > Meanwhile, when iommu.passthrough is on, device drivers which call dma_ > alloc_coherent() will also get local memory and avoid the travel between > numa nodes. > > Cc: Christoph Hellwig <hch@lst.de> > Cc: Marek Szyprowski <m.szyprowski@samsung.com> > Cc: Will Deacon <will@kernel.org> > Cc: Robin Murphy <robin.murphy@arm.com> > Cc: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de> > Cc: Steve Capper <steve.capper@arm.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Mike Rapoport <rppt@linux.ibm.com> > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> > --- > arch/arm64/mm/init.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > index 1e93cfc7c47a..07d4d1fe7983 100644 > --- a/arch/arm64/mm/init.c > +++ b/arch/arm64/mm/init.c > @@ -420,6 +420,8 @@ void __init bootmem_init(void) > > arm64_numa_init(); > > + dma_pernuma_cma_reserve(); > + It might be worth putting this after the hugetlb_cma_reserve() call for clarity, since the comment below applies equally to this call too. Robin. > /* > * must be done after arm64_numa_init() which calls numa_init() to > * initialize node_online_map that gets used in hugetlb_cma_reserve() >
> -----Original Message----- > From: Robin Murphy [mailto:robin.murphy@arm.com] > Sent: Thursday, June 25, 2020 11:16 PM > To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>; hch@lst.de; > m.szyprowski@samsung.com; will@kernel.org; > ganapatrao.kulkarni@cavium.com; catalin.marinas@arm.com > Cc: iommu@lists.linux-foundation.org; Linuxarm <linuxarm@huawei.com>; > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Nicolas > Saenz Julienne <nsaenzjulienne@suse.de>; Steve Capper > <steve.capper@arm.com>; Andrew Morton <akpm@linux-foundation.org>; > Mike Rapoport <rppt@linux.ibm.com> > Subject: Re: [PATCH v2 2/2] arm64: mm: reserve per-numa CMA after > numa_init > > On 2020-06-25 08:43, Barry Song wrote: > > Right now, smmu is using dma_alloc_coherent() to get memory to save > queues > > and tables. Typically, on ARM64 server, there is a default CMA located at > > node0, which could be far away from node2, node3 etc. > > with this patch, smmu will get memory from local numa node to save > command > > queues and page tables. that means dma_unmap latency will be shrunk > much. > > Meanwhile, when iommu.passthrough is on, device drivers which call dma_ > > alloc_coherent() will also get local memory and avoid the travel between > > numa nodes. > > > > Cc: Christoph Hellwig <hch@lst.de> > > Cc: Marek Szyprowski <m.szyprowski@samsung.com> > > Cc: Will Deacon <will@kernel.org> > > Cc: Robin Murphy <robin.murphy@arm.com> > > Cc: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com> > > Cc: Catalin Marinas <catalin.marinas@arm.com> > > Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de> > > Cc: Steve Capper <steve.capper@arm.com> > > Cc: Andrew Morton <akpm@linux-foundation.org> > > Cc: Mike Rapoport <rppt@linux.ibm.com> > > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> > > --- > > arch/arm64/mm/init.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > > index 1e93cfc7c47a..07d4d1fe7983 100644 > > --- a/arch/arm64/mm/init.c > > +++ b/arch/arm64/mm/init.c > > @@ -420,6 +420,8 @@ void __init bootmem_init(void) > > > > arm64_numa_init(); > > > > + dma_pernuma_cma_reserve(); > > + > > It might be worth putting this after the hugetlb_cma_reserve() call for > clarity, since the comment below applies equally to this call too. Yep, it looks even better though dma_pernuma_cma_reserve() is self-documenting by name. > > Robin. > > > /* > > * must be done after arm64_numa_init() which calls numa_init() to > > * initialize node_online_map that gets used in hugetlb_cma_reserve() > > Thanks Barry
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 1e93cfc7c47a..07d4d1fe7983 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -420,6 +420,8 @@ void __init bootmem_init(void) arm64_numa_init(); + dma_pernuma_cma_reserve(); + /* * must be done after arm64_numa_init() which calls numa_init() to * initialize node_online_map that gets used in hugetlb_cma_reserve()
Right now, smmu is using dma_alloc_coherent() to get memory to save queues and tables. Typically, on ARM64 server, there is a default CMA located at node0, which could be far away from node2, node3 etc. with this patch, smmu will get memory from local numa node to save command queues and page tables. that means dma_unmap latency will be shrunk much. Meanwhile, when iommu.passthrough is on, device drivers which call dma_ alloc_coherent() will also get local memory and avoid the travel between numa nodes. Cc: Christoph Hellwig <hch@lst.de> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Will Deacon <will@kernel.org> Cc: Robin Murphy <robin.murphy@arm.com> Cc: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de> Cc: Steve Capper <steve.capper@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Mike Rapoport <rppt@linux.ibm.com> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com> --- arch/arm64/mm/init.c | 2 ++ 1 file changed, 2 insertions(+)