From patchwork Thu Jun 28 16:46:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 10494519 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 04EE66022E for ; Thu, 28 Jun 2018 16:46:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3620D283CA for ; Thu, 28 Jun 2018 16:46:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 266782A40A; Thu, 28 Jun 2018 16:46:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A5CDB29C3D for ; Thu, 28 Jun 2018 16:46:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sZwXyRo5X5kaUO1dsQumKVHheaHOVTyEAHmDHb52ZSY=; b=R5IXnufUNqQbXj GKBEUUoZa1ZslgtuN4qL5O8PyD0cvmrVvIGA+me3WRvWR4JSC+6LyTeb8UaJLVICs+m1m1Iy8gmja QUcAoKycH/GmZ7f4jzkLepauRr6k4HncR0NGHbulJDc/hoSChmhgXK7aIJ4OLChlsED/F2mkC55CL yRkKyevSfDdHOJQ5XC0lgF1lpQaNt9RG5fa1hudUajQd4nJ/z+81KnsMxy41cFMCwatc0WbtD7Ay/ GrFt8EiBWGg8CrQWWsLN6Qn521cLk6o262OuWZlVoAJHy9JDluXAbhF2QCQw8cDRXFjKh57L8l8mW LNqrqbEVqpVLx7XWiHxg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fYa3t-0000cw-NZ; Thu, 28 Jun 2018 16:46:25 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fYa3q-0000Xc-CF for linux-arm-kernel@lists.infradead.org; Thu, 28 Jun 2018 16:46:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2A00E18A; Thu, 28 Jun 2018 09:46:09 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EED463F318; Thu, 28 Jun 2018 09:46:08 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 233D21AE540D; Thu, 28 Jun 2018 17:46:47 +0100 (BST) Date: Thu, 28 Jun 2018 17:46:47 +0100 From: Will Deacon To: Catalin Marinas Subject: Re: [PATCH v2 3/4] arm64: IPI each CPU after invalidating the I-cache for kernel mappings Message-ID: <20180628164646.GC10751@arm.com> References: <1529656278-878-1-git-send-email-will.deacon@arm.com> <1529656278-878-4-git-send-email-will.deacon@arm.com> <20180628160005.cywb6vn7jl4v7kmb@armageddon.cambridge.arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180628160005.cywb6vn7jl4v7kmb@armageddon.cambridge.arm.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180628_094622_521937_611B6E3D X-CRM114-Status: GOOD ( 25.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, rokhanna@nvidia.com, steve.capper@arm.com, leif.lindholm@linaro.org, avanbrunt@nvidia.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Catalin, On Thu, Jun 28, 2018 at 05:00:05PM +0100, Catalin Marinas wrote: > On Fri, Jun 22, 2018 at 09:31:17AM +0100, Will Deacon wrote: > > When invalidating the instruction cache for a kernel mapping via > > flush_icache_range(), it is also necessary to flush the pipeline for > > other CPUs so that instructions fetched into the pipeline before the > > I-cache invalidation are discarded. For example, if module 'foo' is > > unloaded and then module 'bar' is loaded into the same area of memory, > > a CPU could end up executing instructions from 'foo' when branching into > > 'bar' if these instructions were fetched into the pipeline before 'foo' > > was unloaded. > > > > Whilst this is highly unlikely to occur in practice, particularly as > > any exception acts as a context-synchronizing operation, following the > > letter of the architecture requires us to execute an ISB on each CPU > > in order for the new instruction stream to be visible. > > > > Signed-off-by: Will Deacon > > I hit a warning via kgdb_flush_swbreak_addr() because if IRQs disabled > (and it actually deadlocks for me; running as a guest under KVM on TX1): > > WARNING: CPU: 3 PID: 1 at /home/cmarinas/work/Linux/linux-2.6-aarch64/kernel/smp.c:416 smp_call_function_many+0xd4/0x350 > Modules linked in: > CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.18.0-rc2-00006-ga5cfc8429d36 #97 > Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 > pstate: 200003c5 (nzCv DAIF -PAN -UAO) > pc : smp_call_function_many+0xd4/0x350 > lr : smp_call_function+0x38/0x68 > sp : ffff0000080737f0 > x29: ffff0000080737f0 x28: ffff000009169000 > x27: 0000000000000003 x26: 0000000000000000 > x25: ffff00000815a3d0 x24: 0000000000000001 > x23: 0000000000000000 x22: ffff000008eba730 > x21: ffff000009169940 x20: 0000000000000000 > x19: 0000000000000003 x18: ffffffffffffffff > x17: 0000000000000001 x16: 0000000000000019 > x15: ffff0000091696c8 x14: ffff00008930ef07 > x13: ffff00000930ef1d x12: 0000000000000010 > x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f > x9 : 656565652aff6223 x8 : ffffffffffffffff > x7 : fefefefefefefefe x6 : ffff7dfffe7fe014 > x5 : ffff000009169940 x4 : ffff000009147018 > x3 : 0000000000000001 x2 : 0000000000000000 > x1 : ffff00000815a3d0 x0 : 0000000000000000 > Call trace: > smp_call_function_many+0xd4/0x350 > smp_call_function+0x38/0x68 > kick_all_cpus_sync+0x20/0x28 > kgdb_flush_swbreak_addr+0x14/0x20 > dbg_activate_sw_breakpoints+0x74/0xb0 > gdb_serial_stub+0x6ec/0xc80 > kgdb_cpu_enter+0x36c/0x578 Yuck, this is horrible. We don't actually need the IPI in this case because kgdb is using smp_call_function to roundup the secondary cores, but the core code doesn't have the flexibility for us to hook the operation like that. All it provides is a CACHE_FLUSH_IS_SAFE #define, which you can set to 0 if the maintenance isn't safe in IRQ-disabled context. However, there isn't a fall-back and the maintenance is just not performed at all in that case. If we had a "flush the entire D-cache to PoU" instruction, we could hack something into the backend (MIPS does this) but we don't. The best I can come up with is the nasty hack below. Will --->8 Acked-by: Catalin Marinas diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index a0ec27066e6f..5a15d3ce3f0e 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -89,6 +89,19 @@ static inline void flush_icache_range(unsigned long start, unsigned long end) * IPI all online CPUs so that they undergo a context synchronization * event and are forced to refetch the new instructions. */ +#ifdef CONFIG_KGDB + /* + * KGDB performs cache maintenance with interrupts disabled, so we + * will deadlock trying to IPI the secondary CPUs. In theory, we can + * set CACHE_FLUSH_IS_SAFE to 0 to avoid this known issue, but that + * just means that KGDB will elide the maintenance altogether! As it + * turns out, KGDB uses IPIs to round-up the secondary CPUs during + * the patching operation, so we don't need extra IPIs here anyway. + * In which case, add a KGDB-specific bodge and return early. + */ + if (kgdb_connected && irqs_disabled()) + return; +#endif kick_all_cpus_sync(); }