From patchwork Tue Jun 19 12:48:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 10474249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D794D6029B for ; Tue, 19 Jun 2018 12:52:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C5A1828B49 for ; Tue, 19 Jun 2018 12:52:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B9F5728B97; Tue, 19 Jun 2018 12:52:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2D08A28B49 for ; Tue, 19 Jun 2018 12:52:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nOJlI6iiQuapekgmDdujdmbb1cLSQ2pdFGzdSxStkZo=; b=SRWaAklU8ODE9hN2IRWT2mFEYS H5ai38WfQuPxobvj/UFQiJT5CHgvVFTRYfNGnnb9JRkl1p6eVMD6359cpSJWxa+hY9sBvnm3xzShb MqF0SPrQlKW1tr9lUvfO/lU6qSX8ercUy+VKejZuVOIvpL/PCpDBSkOUi7JXQaxZnQPls+CxZRjXH gc7nGmN+XIgjme3IuI45gYAN5i1DOyZ8WD3VOfiakUWEmGMJqdBO2wxPkj0vYz1UrRyAxxAWXH0/k nacldoz0QaGNYsOsg4dnEnStL2ey31yZKluWyrCGx/Uuijt2tHoQNDR1vxDJ+ilfB+Yj6Y1Vi3T0x 6EA9ZUBw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fVG7b-0003GC-DA; Tue, 19 Jun 2018 12:52:31 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fVG7Y-0003Dy-Ql for linux-arm-kernel@bombadil.infradead.org; Tue, 19 Jun 2018 12:52:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=O83Bz2FtzibkvuwNyDFCzsUvxeTBWcF+xLjZTBFQ0Jc=; b=BqED5IiA4Z4vjsTW2v5zAcrGm //VckIkYRO6oFNsoaeUSC/AML/azDOadIj0CzLojS9KcW6+54x5oOAy3KAxDR5OXfiqykLdxUYt8Z tB5aHY+V77dKhWBve57sj3/916PSvKeBlkXQDOTy+4DJ9TPwQFhMyyM61G+BzTF5AASpYQAQlYgvl /UZu0zZdhx+1hALcoN/xQEhEp3F99z3BEijJBcw03AHuilG3B20gbHPiPcK0oTFLsXlrsaSvbknOg VxG0E/fZD+m5DjF128CyqXDUaSdvg/yFFfRN0JSuiejwhz2UqE0rnV9lQqr18NLFsz0jExAPXwKXe U8+/mDfeQ==; Received: from foss.arm.com ([217.140.101.70]) by casper.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fVG35-0001WL-3D for linux-arm-kernel@lists.infradead.org; Tue, 19 Jun 2018 12:47:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6141915BE; Tue, 19 Jun 2018 05:47:40 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 334073F7F3; Tue, 19 Jun 2018 05:47:40 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id C662B1AE3638; Tue, 19 Jun 2018 13:48:15 +0100 (BST) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 3/3] arm64: IPI each CPU after invalidating the I-cache for kernel mappings Date: Tue, 19 Jun 2018 13:48:15 +0100 Message-Id: <1529412495-17525-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1529412495-17525-1-git-send-email-will.deacon@arm.com> References: <1529412495-17525-1-git-send-email-will.deacon@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180619_134751_415551_117A1A42 X-CRM114-Status: GOOD ( 20.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, rokhanna@nvidia.com, steve.capper@arm.com, catalin.marinas@arm.com, Will Deacon , leif.lindholm@linaro.org, avanbrunt@nvidia.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP When invalidating the instruction cache for a kernel mapping via flush_icache_range(), it is also necessary to flush the pipeline for other CPUs so that instructions fetched into the pipeline before the I-cache invalidation are discarded. For example, if module 'foo' is unloaded and then module 'bar' is loaded into the same area of memory, a CPU could end up executing instructions from 'foo' when branching into 'bar' if these instructions were fetched into the pipeline before 'foo' was unloaded. Whilst this is highly unlikely to occur in practice, particularly as any exception acts as a context-synchronizing operation, following the letter of the architecture requires us to execute an ISB on each CPU in order for the new instruction stream to be visible. Signed-off-by: Will Deacon --- arch/arm64/include/asm/cacheflush.h | 13 ++++++++++++- arch/arm64/kernel/alternative.c | 1 - arch/arm64/kernel/cpu_errata.c | 2 +- arch/arm64/kernel/insn.c | 15 ++------------- arch/arm64/mm/cache.S | 4 ++-- 5 files changed, 17 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index d264a7274811..a0ec27066e6f 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -71,7 +71,7 @@ * - kaddr - page address * - size - region size */ -extern void flush_icache_range(unsigned long start, unsigned long end); +extern void __flush_icache_range(unsigned long start, unsigned long end); extern int invalidate_icache_range(unsigned long start, unsigned long end); extern void __flush_dcache_area(void *addr, size_t len); extern void __inval_dcache_area(void *addr, size_t len); @@ -81,6 +81,17 @@ extern void __clean_dcache_area_pou(void *addr, size_t len); extern long __flush_cache_user_range(unsigned long start, unsigned long end); extern void sync_icache_aliases(void *kaddr, unsigned long len); +static inline void flush_icache_range(unsigned long start, unsigned long end) +{ + __flush_icache_range(start, end); + + /* + * IPI all online CPUs so that they undergo a context synchronization + * event and are forced to refetch the new instructions. + */ + kick_all_cpus_sync(); +} + static inline void flush_cache_mm(struct mm_struct *mm) { } diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 4f3dcc15a5b2..0ac06560c166 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -205,7 +205,6 @@ static int __apply_alternatives_multi_stop(void *unused) if (smp_processor_id()) { while (!READ_ONCE(alternatives_applied)) cpu_relax(); - isb(); } else { BUG_ON(alternatives_applied); __apply_alternatives(®ion, false); diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 1d2b6d768efe..0a338a1cd2d7 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -101,7 +101,7 @@ static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start, for (i = 0; i < SZ_2K; i += 0x80) memcpy(dst + i, hyp_vecs_start, hyp_vecs_end - hyp_vecs_start); - flush_icache_range((uintptr_t)dst, (uintptr_t)dst + SZ_2K); + __flush_icache_range((uintptr_t)dst, (uintptr_t)dst + SZ_2K); } static void __install_bp_hardening_cb(bp_hardening_cb_t fn, diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c index 816d03c4c913..4cc41864f277 100644 --- a/arch/arm64/kernel/insn.c +++ b/arch/arm64/kernel/insn.c @@ -249,7 +249,6 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg) } else { while (atomic_read(&pp->cpu_count) <= num_online_cpus()) cpu_relax(); - isb(); } return ret; @@ -283,18 +282,8 @@ int __kprobes aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt) if (ret) return ret; - if (aarch64_insn_hotpatch_safe(insn, insns[0])) { - /* - * ARMv8 architecture doesn't guarantee all CPUs see - * the new instruction after returning from function - * aarch64_insn_patch_text_nosync(). So send IPIs to - * all other CPUs to achieve instruction - * synchronization. - */ - ret = aarch64_insn_patch_text_nosync(addrs[0], insns[0]); - kick_all_cpus_sync(); - return ret; - } + if (aarch64_insn_hotpatch_safe(insn, insns[0])) + return aarch64_insn_patch_text_nosync(addrs[0], insns[0]); } return aarch64_insn_patch_text_sync(addrs, insns, cnt); diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 30334d81b021..0c22ede52f90 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -35,7 +35,7 @@ * - start - virtual start address of region * - end - virtual end address of region */ -ENTRY(flush_icache_range) +ENTRY(__flush_icache_range) /* FALLTHROUGH */ /* @@ -77,7 +77,7 @@ alternative_else_nop_endif 9: mov x0, #-EFAULT b 1b -ENDPROC(flush_icache_range) +ENDPROC(__flush_icache_range) ENDPROC(__flush_cache_user_range) /*