From patchwork Fri Feb 15 01:00:12 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chanho Park X-Patchwork-Id: 2143971 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by patchwork2.kernel.org (Postfix) with ESMTP id 42CDFDF283 for ; Fri, 15 Feb 2013 01:04:44 +0000 (UTC) Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.76 #1 (Red Hat Linux)) id 1U69ff-0006nl-MR; Fri, 15 Feb 2013 01:00:59 +0000 Received: from mailout1.samsung.com ([203.254.224.24]) by merlin.infradead.org with esmtp (Exim 4.76 #1 (Red Hat Linux)) id 1U69fb-0006m1-A7 for linux-arm-kernel@lists.infradead.org; Fri, 15 Feb 2013 01:00:57 +0000 Received: from epcpsbgr1.samsung.com (u141.gpu120.samsung.co.kr [203.254.230.141]) by mailout1.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTP id <0MI8004LILFTWH70@mailout1.samsung.com> for linux-arm-kernel@lists.infradead.org; Fri, 15 Feb 2013 10:00:48 +0900 (KST) Received: from epcpsbgm1.samsung.com ( [203.254.230.42]) by epcpsbgr1.samsung.com (EPCPMTA) with SMTP id 92.1A.16905.0C88D115; Fri, 15 Feb 2013 10:00:48 +0900 (KST) X-AuditID: cbfee68d-b7f4f6d000004209-19-511d88c0ae68 Received: from epmmp1.local.host ( [203.254.227.16]) by epcpsbgm1.samsung.com (EPCPMTA) with SMTP id 79.C8.03918.FB88D115; Fri, 15 Feb 2013 10:00:47 +0900 (KST) Received: from localhost.localdomain ([10.90.51.45]) by mmp1.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTPA id <0MI800K9HLH5N110@mmp1.samsung.com> for linux-arm-kernel@lists.infradead.org; Fri, 15 Feb 2013 10:00:47 +0900 (KST) From: Chanho Park To: linux@arm.linux.org.uk Subject: [PATCH] arm: mm: lockless get_user_pages_fast Date: Fri, 15 Feb 2013 10:00:12 +0900 Message-id: <1360890012-4684-1-git-send-email-chanho61.park@samsung.com> X-Mailer: git-send-email 1.7.9.5 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrJLMWRmVeSWpSXmKPExsVy+t8zLd0DHbKBBjPOslhsenyN1YHRY/OS +gDGKC6blNSczLLUIn27BK6M8x/+sxbM0a7ovvSHtYGxU6mLkYNDQsBE4t0FzS5GTiBTTOLC vfVsXYxcHEICyxgl3rYdYYJImEgsXH4bKrGIUeLC1dPMEM5aJon/vw4yglSxCehKbHn+CswW EZCSmLFzKTtIEbPAT0aJqb2XmEDWCQuYSVw+FgtSwyKgKvHq1hEWkDCvgIfEjZPREAcpSMyZ ZAOxdwebxI4TJRDVAhLfJh9igSiRldh0gBmiRFLi4IobLBMYBRcwMqxiFE0tSC4oTkovMtQr TswtLs1L10vOz93ECAmm3h2Mtw9YH2JMBho3kVlKNDkfGIx5JfGGxgbGhoaWhmamlqYGpAkr ifPKXZIJFBJITyxJzU5NLUgtii8qzUktPsTIxMEp1cDod0S9b/4UX1mb2mW+4qbs/6+qz078 5yoinJhQ8z7g7Re5tctsgmY1S91wkets7XMX2Ru2e/+81m1LS0+/+HDwQLgFz+RXEyJlDPd4 zEyUCEzjerKgM+iUC9O+lWe/SU8KLs5K3ZXSLXl00mqp8r5wtuezHs6McNnDJ/Jprr3ajhjt H2Yfms2VWIozEg21mIuKEwHyKDoYPAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupgkeLIzCtJLcpLzFFi42I5/e+xgO7+DtlAg7a9JhabHl9jdWD02Lyk PoAxqoHRJiM1MSW1SCE1Lzk/JTMv3VbJOzjeOd7UzMBQ19DSwlxJIS8xN9VWycUnQNctMwdo qpJCWWJOKVAoILG4WEnfDtOE0BA3XQuYxghd35AguB4jAzSQsI4x4/yH/6wFc7Qrui/9YW1g 7FTqYuTkkBAwkVi4/DYbhC0mceHeeiCbi0NIYBGjxIWrp5khnLVMEv9/HWQEqWIT0JXY8vwV mC0iICUxY+dSdpAiZoGfjBJTey8xdTFycAgLmElcPhYLUsMioCrx6tYRFpAwr4CHxI2T0SCm hICCxJxJNhMYuRcwMqxiFE0tSC4oTkrPNdQrTswtLs1L10vOz93ECA7XZ1I7GFc2WBxiFOBg VOLhFZCQDRRiTSwrrsw9xCjBwawkwrsmBCjEm5JYWZValB9fVJqTWnyIMRlo90RmKdHkfGAs 5ZXEGxqbmBlZGpkZm5gbG5MmrCTOy3jqSYCQQHpiSWp2ampBahHMFiYOTqkGxjR9ZqfLnxgu 7PvWafCVlXtTy277fw6VdZZ61h0W2Yn+JpY+Oo9v9ctueXBplXVKWcika4aP35TOuPzu8J+S 81qR3dnPTyTp6fsFtuRNTXP9Nl9l/4vG1z+9DqW7rsuT4ziqvbW6J+ibnsaEqN6mQ4arvnkd eReQ99/6pPdRkbozhhMZ5m8RUmIpzkg01GIuKk4EACNWO22bAgAA DLP-Filter: Pass X-MTR: 20000000000000000@CPGS X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20130214_200055_824531_ACAC2153 X-CRM114-Status: GOOD ( 21.26 ) X-Spam-Score: -7.6 (-------) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-7.6 points) pts rule name description ---- ---------------------- -------------------------------------------------- -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [203.254.224.24 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.7 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: catalin.marinas@arm.com, Inki Dae , linux-mm@kvack.org, Kyungmin Park , Myungjoo Ham , Chanho Park , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: linux-arm-kernel-bounces@lists.infradead.org Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This patch adds get_user_pages_fast(old name is "fast_gup") for ARM. The fast_gup can walk pagetable without taking mmap_sem or any locks. If there is not a pte with the correct permissions for the access, we fall back to slow path(get_user_pages) to get remaining pages. This patch is written on reference the x86's gup implementation. Traversing of hugepages is excluded because ARM haven't supported hugepages yet[1], just only RFC. [1]: http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/126382.html Signed-off-by: Chanho Park Signed-off-by: Kyungmin Park Cc: Myungjoo Ham Cc: Inki Dae --- arch/arm/mm/Makefile | 2 +- arch/arm/mm/gup.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 arch/arm/mm/gup.c diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 8a9c4cb..1f01c0b 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -6,7 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ iomap.o obj-$(CONFIG_MMU) += fault-armv.o flush.o idmap.o ioremap.o \ - mmap.o pgd.o mmu.o vmregion.o + mmap.o pgd.o mmu.o vmregion.o gup.o ifneq ($(CONFIG_MMU),y) obj-y += nommu.o diff --git a/arch/arm/mm/gup.c b/arch/arm/mm/gup.c new file mode 100644 index 0000000..ed54fd8 --- /dev/null +++ b/arch/arm/mm/gup.c @@ -0,0 +1,160 @@ +/* + * linux/arch/arm/mm/gup.c - Lockless get_user_pages_fast for arm + * + * Copyright (c) 2013 Samsung Electronics Co., Ltd. + * http://www.samsung.com + * Author : Chanho Park + * + * This code is written on reference from the x86 and PowerPC versions, by: + * + * Copyright (C) 2008 Nick Piggin + * Copyright (C) 2008 Novell Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + +/* + * The performance critical leaf functions are made noinline otherwise gcc + * inlines everything into a single function which results in too much + * register pressure. + */ +static noinline int gup_pte_range(pmd_t *pmdp, unsigned long addr, + unsigned long end, int write, struct page **pages, int *nr) +{ + pte_t *ptep, pte; + + ptep = pte_offset_kernel(pmdp, addr); + do { + struct page *page; + + pte = *ptep; + smp_rmb(); + + if (!pte_present_user(pte) || (write && !pte_write(pte))) + return 0; + + VM_BUG_ON(!pfn_valid(pte_pfn(pte))); + page = pte_page(pte); + + if (!page_cache_get_speculative(page)) + return 0; + + pages[*nr] = page; + (*nr)++; + + } while (ptep++, addr += PAGE_SIZE, addr != end); + + return 1; +} + +static int gup_pmd_range(pud_t *pudp, unsigned long addr, unsigned long end, + int write, struct page **pages, int *nr) +{ + unsigned long next; + pmd_t *pmdp; + + pmdp = pmd_offset(pudp, addr); + do { + next = pmd_addr_end(addr, end); + if (pmd_none(*pmdp)) + return 0; + else if (!gup_pte_range(pmdp, addr, next, write, pages, nr)) + return 0; + } while (pmdp++, addr = next, addr != end); + + return 1; +} + +static int gup_pud_range(pgd_t *pgdp, unsigned long addr, unsigned long end, + int write, struct page **pages, int *nr) +{ + unsigned long next; + pud_t *pudp; + + pudp = pud_offset(pgdp, addr); + do { + next = pud_addr_end(addr, end); + if (pud_none(*pudp)) + return 0; + else if (!gup_pmd_range(pudp, addr, next, write, pages, nr)) + return 0; + } while (pudp++, addr = next, addr != end); + + return 1; +} + +int get_user_pages_fast(unsigned long start, int nr_pages, int write, + struct page **pages) +{ + struct mm_struct *mm = current->mm; + unsigned long addr, len, end; + unsigned long next; + pgd_t *pgdp; + int nr = 0; + + start &= PAGE_MASK; + addr = start; + len = (unsigned long) nr_pages << PAGE_SHIFT; + end = start + len; + + if (unlikely(!access_ok(write ? VERIFY_WRITE : VERIFY_READ, + start, len))) + goto slow_irqon; + + /* + * This doesn't prevent pagetable teardown, but does prevent + * the pagetables from being freed on arm. + * + * So long as we atomically load page table pointers versus teardown, + * we can follow the address down to the the page and take a ref on it. + */ + local_irq_disable(); + + pgdp = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgdp)) + goto slow; + else if (!gup_pud_range(pgdp, addr, next, write, pages, &nr)) + goto slow; + } while (pgdp++, addr = next, addr != end); + + local_irq_enable(); + + VM_BUG_ON(nr != (end - start) >> PAGE_SHIFT); + return nr; + + { + int ret; + +slow: + local_irq_enable(); +slow_irqon: + /* Try to get the remaining pages with get_user_pages */ + start += nr << PAGE_SHIFT; + pages += nr; + + down_read(&mm->mmap_sem); + ret = get_user_pages(current, mm, start, + (end - start) >> PAGE_SHIFT, write, 0, pages, NULL); + up_read(&mm->mmap_sem); + + /* Have to be a bit careful with return values */ + if (nr > 0) { + if (ret < 0) + ret = nr; + else + ret += nr; + } + + return ret; + } +}