From patchwork Tue Jun 21 15:50:30 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 902522 Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p5LFq8h2019854 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 21 Jun 2011 15:52:29 GMT Received: from canuck.infradead.org ([2001:4978:20e::1]) by merlin.infradead.org with esmtps (Exim 4.76 #1 (Red Hat Linux)) id 1QZ3Ey-0006Ay-S7; Tue, 21 Jun 2011 15:51:48 +0000 Received: from localhost ([127.0.0.1] helo=canuck.infradead.org) by canuck.infradead.org with esmtp (Exim 4.76 #1 (Red Hat Linux)) id 1QZ3Ey-0008JD-Fn; Tue, 21 Jun 2011 15:51:48 +0000 Received: from casper.infradead.org ([2001:770:15f::2]) by canuck.infradead.org with esmtps (Exim 4.76 #1 (Red Hat Linux)) id 1QZ3Ev-0008J3-AK for linux-arm-kernel@canuck.infradead.org; Tue, 21 Jun 2011 15:51:45 +0000 Received: from mail-wy0-f177.google.com ([74.125.82.177]) by casper.infradead.org with esmtps (Exim 4.76 #1 (Red Hat Linux)) id 1QZ3Er-00069k-P4 for linux-arm-kernel@lists.infradead.org; Tue, 21 Jun 2011 15:51:43 +0000 Received: by wyf23 with SMTP id 23so3977976wyf.36 for ; Tue, 21 Jun 2011 08:49:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=CcwGpLMHfEu0K2nZG7ct3jVA+PlkDu7y2UFVcmnxUmI=; b=VCtNVse+XGzsDb2yNRBtz+gDqLq/BUhwp3aVwA0Zc6KThO86aUDnohIfK+VfBRVR9p kL6bOzAItUvwp6OLHYZ56GurHIfvQ/TPCudBgyFTf6iNc2aG3o522qpO0Pk6Y+1E/9HD nAPpraLiL5RlYBlJIT7KNHlpJhNfQWmGDCKKY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=NPcfQqTBf6HdFG5iQK4zk/9wwfk9zlIETGufJabVRkayyajeCrlHXIp9isEVX6vQZE yIgZT+dEf82vdKDXKhcX3bDQnb8zrVdzH9/56CqtTtVxGaaZ3DJoYACOmuRIU3HpvOE/ MymlyuqrLNi9h5ZKMcyBLTP8SpXI86vYKQeaQ= Received: by 10.227.177.12 with SMTP id bg12mr1553456wbb.59.1308671375418; Tue, 21 Jun 2011 08:49:35 -0700 (PDT) Received: from 1n450.cable.virginmedia.net (cpc3-cmbg12-0-0-cust651.5-4.cable.virginmedia.com [86.9.126.140]) by mx.google.com with ESMTPS id eq4sm4104694wbb.20.2011.06.21.08.49.33 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 21 Jun 2011 08:49:34 -0700 (PDT) Date: Tue, 21 Jun 2011 16:50:30 +0100 From: Catalin Marinas To: "EXTERNAL Waechtler Peter (Fa. TCP, CM-AI/PJ-CF31)" Subject: Re: parallel load of modules on an ARM multicore Message-ID: <20110621155030.GA21245@1n450.cable.virginmedia.net> References: <274124B9C6907D4B8CE985903EAA19E9185A92923E@SI-MBX06.de.bosch.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <274124B9C6907D4B8CE985903EAA19E9185A92923E@SI-MBX06.de.bosch.com> User-Agent: Mutt/1.5.20 (2009-06-14) X-CRM114-Version: 20090807-BlameThorstenAndJenny ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20110621_165141_891943_E41CFCA6 X-CRM114-Status: GOOD ( 32.89 ) X-Spam-Score: -2.6 (--) X-Spam-Report: SpamAssassin version 3.3.2-r929478 on casper.infradead.org summary: Content analysis details: (-2.6 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [74.125.82.177 listed in list.dnswl.org] 0.0 FREEMAIL_FROM Sender email is freemail (catalin.marinas[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.0 RFC_ABUSE_POST Both abuse and postmaster missing on sender domain Cc: "linux-arm-kernel@lists.infradead.org" X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-arm-kernel-bounces@lists.infradead.org Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Tue, 21 Jun 2011 15:52:29 +0000 (UTC) On Mon, Jun 20, 2011 at 03:43:27PM +0200, EXTERNAL Waechtler Peter (Fa. TCP, CM-AI/PJ-CF31) wrote: > I'm getting unexpected results from loading several modules - some > of them in parallel - on an ARM11 MPcore system. > > The system does not use "single sequential" modprobe commands - instead it > starts several shells doing insmod sequences like this: > > shell A: > insmod a > insmod ab > insmod abc > > shell B: > insmod b > insmod bc > insmod bcd > > shell C: > insmod c > insmod cd > insmod cde > > This is done to crash^H^H^H^H^Hboot faster ;) > > While one insmod operation is protected via the module_mutex - I'm wondering > what happens with the instruction cache invalidation. > AFAICT the flush_icache_range only invalidates the ICache on the running cpu. > The module_mutex is unlocked after _loading_ the module, do_mod_ctors() and > do_one_initcall() are called without that lock - can they run on a different cpu? > It's an preemptible system (SMP PREEMPT armv6l). > > Wouldn't it be required to flush the icache on _all_ cpus? I theory, you would need to flush both the I and D cache on all the CPUs but that's not easily possible (well, I don't think there are many users of flush_icache_range(), so we could make this do an smp_call_function(). I think what happens (I'm looking at a 3.0-rc3 kernel) is that the module code is copied into RAM and the process could migrate to another CPU before flush_icache_range() gets called (this function is called outside the mutex_lock regions). We therefore don't flush the data out of the D-cache that was allocated during copy_from_user(). You can try the patch below (I haven't tested it for some time), it may fix the issue. 8<----------------------------------------------- ARM: Allow lazy cache flushing on ARM11MPCore From: Catalin Marinas The ARM11MPCore doesn't broadcast the cache maintenance operations in hardware, therefore the flush_dcache_page() currently performs the cache flushing non-lazily. But since not all drivers call this function after writing to a page cache page, the kernel needs a different approach like using read-for-ownership on the CPU flushing the cache to force the dirty cache lines migration from other CPUs. This way the cache flushing operation can be done lazily via __sync_icache_dcache(). Since we cannot force read-for-ownership on the I-cache, the patch invalidates the whole I-cache when a thread migrates to another CPU. Signed-off-by: Catalin Marinas --- arch/arm/include/asm/mmu_context.h | 3 ++- arch/arm/mm/cache-v6.S | 8 ++++++++ arch/arm/mm/flush.c | 3 +-- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h index 71605d9..d116a23 100644 --- a/arch/arm/include/asm/mmu_context.h +++ b/arch/arm/include/asm/mmu_context.h @@ -114,7 +114,8 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #ifdef CONFIG_SMP /* check for possible thread migration */ if (!cpumask_empty(mm_cpumask(next)) && - !cpumask_test_cpu(cpu, mm_cpumask(next))) + (cache_ops_need_broadcast() || + !cpumask_test_cpu(cpu, mm_cpumask(next)))) __flush_icache_all(); #endif if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) { diff --git a/arch/arm/mm/cache-v6.S b/arch/arm/mm/cache-v6.S index 73b4a8b..bdc1cc1 100644 --- a/arch/arm/mm/cache-v6.S +++ b/arch/arm/mm/cache-v6.S @@ -133,6 +133,10 @@ ENTRY(v6_coherent_user_range) #ifdef HARVARD_CACHE bic r0, r0, #CACHE_LINE_SIZE - 1 1: +#ifdef CONFIG_SMP + /* no cache maintenance broadcasting on ARM11MPCore */ + USER( ldr r2, [r0] ) @ read for ownership +#endif USER( mcr p15, 0, r0, c7, c10, 1 ) @ clean D line add r0, r0, #CACHE_LINE_SIZE 2: @@ -178,6 +182,10 @@ ENTRY(v6_flush_kern_dcache_area) add r1, r0, r1 bic r0, r0, #D_CACHE_LINE_SIZE - 1 1: +#ifdef CONFIG_SMP + /* no cache maintenance broadcasting on ARM11MPCore */ + ldr r2, [r0] @ read for ownership +#endif #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 1 @ clean & invalidate D line #else diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index 1a8d4aa..72f9333 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -291,8 +291,7 @@ void flush_dcache_page(struct page *page) mapping = page_mapping(page); - if (!cache_ops_need_broadcast() && - mapping && !mapping_mapped(mapping)) + if (mapping && !mapping_mapped(mapping)) clear_bit(PG_dcache_clean, &page->flags); else { __flush_dcache_page(mapping, page);