From patchwork Thu Oct 29 20:59:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 11867527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54D4BC4742C for ; Thu, 29 Oct 2020 21:00:32 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C4E7D20790 for ; Thu, 29 Oct 2020 21:00:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="cdtMWej9"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="nVKt0e3C" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C4E7D20790 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:Mime-Version:Message-Id:Date: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=fNKH2uzAHWNXMBRXoxmL+J5TfEcyT+vJtols8Xhrf6w=; b=cdtMWej9nGoZWlQqIKBxP99SnB qFNBcjmz9xuHMFSwYghmN4xHMh3qC4w5WQn4wD8/H66xousiYIZKv5L+bV0bdkU9CdM1dQAhecf6w 04Ugp3XIKzKOlZsKN+C11MmuMmtUYtEE71t1uTYshFWlvzJW/SRQ9KuREA3TIJfeZQOzyEbi6K6nH 4QI9NFJL562xyfTNkGy6GE2HpmQ0XayTLHKwrfLPaqie6+xY1UGOBLRgXietnpdEQnwOKpHuy3DlB Cej+GjxFJDVGV6jmtrq9Z1dz91OEt8lA6HwwNXr0h/UtUziocMYhAHfoMqq0Mz27SN4cmZ8bh6zvm thmK9MCw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYF1b-0001X2-H8; Thu, 29 Oct 2020 20:59:59 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYF1Z-0001WE-4B for linux-arm-kernel@lists.infradead.org; Thu, 29 Oct 2020 20:59:58 +0000 Received: by mail-pf1-x449.google.com with SMTP id b139so3123622pfb.2 for ; Thu, 29 Oct 2020 13:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc; bh=26rvunJv+3oezks1HwLYtPnr9HpqfX8Du9KCQslB++8=; b=nVKt0e3CP6Xd27VNX8U0PSH1J1AYau/X3RNcNn8KdDslhjCMvka7tVj4McSpAJ6lIt yeLK3rIsXYaaTU/mZLTCNEaIE30GwVVERCdn2nXVK1LD2ojtY31pWHStYWwgnydKRWe+ VDA8UmQJ4V4Aa/LEaRN/qVBAcMghfyh47WmehXs0j5qu9jdIS0+Wox2uAMXnoXK96225 wml4BL+KP+Xwrn1CwYZDDSjMiqdthAL7R2MyfAyvPqn9zShObD+xEuO6X0Lrw+BQ6PgS iBi3oOO4FK4dgelxquU47f/zQ2n2AM8sV0TC3XSYZJXcqLUq59V2RMsB0FHUDArbyTbw z7EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc; bh=26rvunJv+3oezks1HwLYtPnr9HpqfX8Du9KCQslB++8=; b=BeceV5ijYZFvbwP6D2hE5gvpS7f55Mqjx3yG49DO/nO7/jsmfFOEeKBdWuTvGOmUp0 aAtjNIG2AlgiunTHp0sYPuIJTMa3nb1zPpX3UbCvBzdWi4fy+h0wInfN5Bw+LfafcU/D NSGYycr10aNxrZJtCtKpeQRN1K/znif0R01b82kRfJYZWsrouXNikp1XAbd6UU8IqUCl q6Xcff1v6EJiMoDr2kw/EHbpLNApmY5hSnzKTFcc0RbFJKDew4eMAJlfpXKLwsu60xaQ KQdJEEwla3B6fZqGYVurqQktJFpelLpSs1nZ+V9YFwJ1lZSX9gZbfbP+D6GbwGqqq2JK OM0w== X-Gm-Message-State: AOAM530Ke5XGHUmiHyFMpM57tdLJzUNYjOavfNUxp1Wo9cPdgjlCOBM8 C7EkHKuQdn+MopxV+fQBWY0G4DI= X-Google-Smtp-Source: ABdhPJy8gpMhC6eAdRS79wwwRzifbhSF6meqaX58eGXXs1LnujjGJy9fQfr/wPUH6oqSvdtrt5+DQ8o= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:0:7220:84ff:fe09:385a]) (user=pcc job=sendgmr) by 2002:aa7:85c6:0:b029:164:61d:4384 with SMTP id z6-20020aa785c60000b0290164061d4384mr6368504pfn.65.1604005193080; Thu, 29 Oct 2020 13:59:53 -0700 (PDT) Date: Thu, 29 Oct 2020 13:59:44 -0700 Message-Id: <20201029205944.2769-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog Subject: [PATCH] kasan: arm64: support specialized outlined tag mismatch checks From: Peter Collingbourne To: Mark Brown , Mark Rutland , Will Deacon , Catalin Marinas , Andrey Konovalov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201029_165957_220294_F510E72B X-CRM114-Status: GOOD ( 30.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Collingbourne , Linux ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org By using outlined checks we can achieve a significant code size improvement by moving the tag-based ASAN checks into separate functions. Unlike the existing CONFIG_KASAN_OUTLINE mode these functions have a custom calling convention that preserves most registers and is specialized to the register containing the address and the type of access, and as a result we can eliminate the code size and performance overhead of a standard calling convention such as AAPCS for these functions. This change depends on a separate series of changes to Clang [1] to support outlined checks in the kernel, although the change works fine without them (we just don't get outlined checks). This is because the flag -mllvm -hwasan-inline-all-checks=0 has no effect until the Clang changes land. The flag was introduced in the Clang 9.0 timeframe as part of the support for outlined checks in userspace and because our minimum Clang version is 10.0 we can pass it unconditionally. Outlined checks require a new runtime function with a custom calling convention. Add this function to arch/arm64/lib. I measured the code size of defconfig + tag-based KASAN, as well as boot time (i.e. time to init launch) on a DragonBoard 845c with an Android arm64 GKI kernel. The results are below: code size boot time CONFIG_KASAN_INLINE=y before 92824064 6.18s CONFIG_KASAN_INLINE=y after 38822400 6.65s CONFIG_KASAN_OUTLINE=y 39215616 11.48s We can see straight away that specialized outlined checks beat the existing CONFIG_KASAN_OUTLINE=y on both code size and boot time for tag-based ASAN. As for the comparison between CONFIG_KASAN_INLINE=y before and after we saw similar performance numbers in userspace [2] and decided that since the performance overhead is minimal compared to the overhead of tag-based ASAN itself as well as compared to the code size improvements we would just replace the inlined checks with the specialized outlined checks without the option to select between them, and that is what I have implemented in this patch. But we may make a different decision for the kernel such as having CONFIG_KASAN_OUTLINE=y turn on specialized outlined checks if Clang is new enough. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I1a30036c70ab3c3ee78d75ed9b87ef7cdc3fdb76 Link: [1] https://reviews.llvm.org/D90426 Link: [2] https://reviews.llvm.org/D56954 --- arch/arm64/include/asm/asm-prototypes.h | 6 +++ arch/arm64/include/asm/module.lds.h | 17 +++++++- arch/arm64/lib/Makefile | 2 + arch/arm64/lib/kasan_sw_tags.S | 54 +++++++++++++++++++++++++ mm/kasan/tags.c | 7 ++++ scripts/Makefile.kasan | 1 + 6 files changed, 85 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/lib/kasan_sw_tags.S diff --git a/arch/arm64/include/asm/asm-prototypes.h b/arch/arm64/include/asm/asm-prototypes.h index 1c9a3a0c5fa5..ec1d9655f885 100644 --- a/arch/arm64/include/asm/asm-prototypes.h +++ b/arch/arm64/include/asm/asm-prototypes.h @@ -23,4 +23,10 @@ long long __ashlti3(long long a, int b); long long __ashrti3(long long a, int b); long long __lshrti3(long long a, int b); +/* + * This function uses a custom calling convention and cannot be called from C so + * this prototype is not entirely accurate. + */ +void __hwasan_tag_mismatch(unsigned long addr, unsigned long access_info); + #endif /* __ASM_PROTOTYPES_H */ diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h index 691f15af788e..4a6d717f75f3 100644 --- a/arch/arm64/include/asm/module.lds.h +++ b/arch/arm64/include/asm/module.lds.h @@ -1,7 +1,20 @@ -#ifdef CONFIG_ARM64_MODULE_PLTS SECTIONS { +#ifdef CONFIG_ARM64_MODULE_PLTS .plt (NOLOAD) : { BYTE(0) } .init.plt (NOLOAD) : { BYTE(0) } .text.ftrace_trampoline (NOLOAD) : { BYTE(0) } -} #endif + +#ifdef CONFIG_KASAN_SW_TAGS + /* + * Outlined checks go into comdat-deduplicated sections named .text.hot. + * Because they are in comdats they are not combined by the linker and + * we otherwise end up with multiple sections with the same .text.hot + * name in the .ko file. The kernel module loader warns if it sees + * multiple sections with the same name so we use this sections + * directive to force them into a single section and silence the + * warning. + */ + .text.hot : { *(.text.hot) } +#endif +} diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index d31e1169d9b8..8e60d76a1b47 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -18,3 +18,5 @@ obj-$(CONFIG_CRC32) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_ARM64_MTE) += mte.o + +obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/arch/arm64/lib/kasan_sw_tags.S b/arch/arm64/lib/kasan_sw_tags.S new file mode 100644 index 000000000000..f72f0e8e1192 --- /dev/null +++ b/arch/arm64/lib/kasan_sw_tags.S @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + */ + +#include +#include + +/* + * Report a tag mismatch detected by tag-based KASAN. + * + * This function has a custom calling convention in order to minimize the sizes + * of the compiler-generated thunks that call it. All registers except for x16 + * and x17 must be preserved. This includes x0 and x1 which are used by the + * caller to pass arguments, and x29 (the frame pointer register). In order to + * allow these registers to be restored the caller spills them to stack. The 256 + * bytes of stack space allocated by the caller must be deallocated on return. + * + * This function takes care of transitioning to the standard AAPCS calling + * convention and calls the C function kasan_tag_mismatch to report the error. + * + * Parameters: + * x0 - the fault address + * x1 - an encoded description of the faulting access + */ +SYM_FUNC_START(__hwasan_tag_mismatch) + add x29, sp, #232 + stp x2, x3, [sp, #16] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #48] + stp x8, x9, [sp, #64] + stp x10, x11, [sp, #80] + stp x12, x13, [sp, #96] + stp x14, x15, [sp, #112] +#ifndef CONFIG_SHADOW_CALL_STACK + str x18, [sp, #128] +#endif + mov x2, x30 + bl kasan_tag_mismatch + ldp x29, x30, [sp, #232] +#ifndef CONFIG_SHADOW_CALL_STACK + ldr x18, [sp, #128] +#endif + ldp x14, x15, [sp, #112] + ldp x12, x13, [sp, #96] + ldp x10, x11, [sp, #80] + ldp x8, x9, [sp, #64] + ldp x6, x7, [sp, #48] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #16] + ldp x0, x1, [sp], #256 + ret +SYM_FUNC_END(__hwasan_tag_mismatch) +EXPORT_SYMBOL(__hwasan_tag_mismatch) diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index e02a36a51f42..d00613956c79 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -198,3 +198,10 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache, return &alloc_meta->free_track[i]; } + +void kasan_tag_mismatch(unsigned long addr, unsigned long access_info, + unsigned long ret_ip) +{ + kasan_report(addr, 1 << (access_info & 0xf), access_info & 0x10, + ret_ip); +} diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan index 1e000cc2e7b4..1f2cccc264d8 100644 --- a/scripts/Makefile.kasan +++ b/scripts/Makefile.kasan @@ -44,6 +44,7 @@ endif CFLAGS_KASAN := -fsanitize=kernel-hwaddress \ -mllvm -hwasan-instrument-stack=$(CONFIG_KASAN_STACK) \ -mllvm -hwasan-use-short-granules=0 \ + -mllvm -hwasan-inline-all-checks=0 \ $(instrumentation_flags) endif # CONFIG_KASAN_SW_TAGS