From patchwork Fri Mar 28 15:01:29 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 3903991 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 2D93B9F2B6 for ; Fri, 28 Mar 2014 15:05:34 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 57588202FF for ; Fri, 28 Mar 2014 15:05:33 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6A467202FE for ; Fri, 28 Mar 2014 15:05:32 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WTYKC-0005ry-2t; Fri, 28 Mar 2014 15:04:04 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WTYJX-0008Hh-Bf; Fri, 28 Mar 2014 15:03:23 +0000 Received: from mail-wi0-f180.google.com ([209.85.212.180]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WTYIK-00086F-5X for linux-arm-kernel@lists.infradead.org; Fri, 28 Mar 2014 15:02:27 +0000 Received: by mail-wi0-f180.google.com with SMTP id q5so860407wiv.1 for ; Fri, 28 Mar 2014 08:01:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uIwHQYqhQO7g9zN182v9cLfi+tz5PWv+ZbaQgCIh4y4=; b=dK5j8Ubl0GhqA8Ou13uq6tNU8oLiPEfDRlfO3OIndG5sLA0f4OIVUViPkVXJayQLW6 uNhp/BjL7NdDumKvJL5Au/q+iZfKI1aXV+MbTjWbjuN4JHTqyFFikmW/ocyENPqxv9U7 /WlbIFOVND7rY44NpuCHowPQ8L7NyqeorGuicmlb6o59WsCx5249ck3QZV+jiHMIFGdt sJEZYdx+7erRsCUwnr4z4CWGQQSJY37zPKPMQh2kXMr/biZNLBbqroHSFFaVF3XTNz5/ 15FUd7cAMOfP1XzD7KmWSpw0VDI86Ifp7JmbdyeKOCMMXvtk2nr3N/jvuUnAAVfQ4Bvg XzEw== X-Gm-Message-State: ALoCoQlfxpfB2qrwFf3PBf67Ytl2JS0dpFW+MHk3z1t4oXR30m4X74At6zubcXvZ4yYA9es5K+5D X-Received: by 10.180.188.229 with SMTP id gd5mr48085255wic.54.1396018906430; Fri, 28 Mar 2014 08:01:46 -0700 (PDT) Received: from marmot.wormnet.eu (marmot.wormnet.eu. [188.246.204.87]) by mx.google.com with ESMTPSA id fo6sm8038670wib.7.2014.03.28.08.01.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Mar 2014 08:01:45 -0700 (PDT) From: Steve Capper To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, linux@arm.linux.org.uk, linux-mm@kvack.org, linux-arch@vger.kernel.org Subject: [RFC PATCH V4 4/7] arm: mm: Enable RCU fast_gup Date: Fri, 28 Mar 2014 15:01:29 +0000 Message-Id: <1396018892-6773-5-git-send-email-steve.capper@linaro.org> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1396018892-6773-1-git-send-email-steve.capper@linaro.org> References: <1396018892-6773-1-git-send-email-steve.capper@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140328_110208_738264_978C88E9 X-CRM114-Status: UNSURE ( 9.75 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -2.6 (--) Cc: peterz@infradead.org, gary.robertson@linaro.org, akpm@linux-foundation.org, anders.roxell@linaro.org, Steve Capper X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Activate the RCU fast_gup for ARM. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Signed-off-by: Steve Capper --- arch/arm/Kconfig | 3 +++ arch/arm/include/asm/pgtable-3level.h | 6 ++++++ arch/arm/mm/flush.c | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 7d5340d..3cf589e 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1788,6 +1788,9 @@ config ARCH_SELECT_MEMORY_MODEL config HAVE_ARCH_PFN_VALID def_bool ARCH_HAS_HOLES_MEMORYMODEL || !SPARSEMEM +config HAVE_RCU_GUP + def_bool y + config HIGHMEM bool "High Memory Support" depends on MMU diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index b286ba9..fdc4a4f 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -226,6 +226,12 @@ static inline pte_t pte_mkspecial(pte_t pte) #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT)) #define pmd_trans_splitting(pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING) + +#ifdef CONFIG_HAVE_RCU_TABLE_FREE +#define __HAVE_ARCH_PMDP_SPLITTING_FLUSH +void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmdp); +#endif #endif #define PMD_BIT_FUNC(fn,op) \ diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index 3387e60..91a2b59 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -377,3 +377,22 @@ void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned l */ __cpuc_flush_dcache_area(page_address(page), PAGE_SIZE); } + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#ifdef CONFIG_HAVE_RCU_TABLE_FREE +static void thp_splitting_flush_sync(void *arg) +{ +} + +void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address, + pmd_t *pmdp) +{ + pmd_t pmd = pmd_mksplitting(*pmdp); + VM_BUG_ON(address & ~PMD_MASK); + set_pmd_at(vma->vm_mm, address, pmdp, pmd); + + /* dummy IPI to serialise against fast_gup */ + smp_call_function(thp_splitting_flush_sync, NULL, 1); +} +#endif /* CONFIG_HAVE_RCU_TABLE_FREE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */