From patchwork Thu May 13 02:27:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12255113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19F3BC433ED for ; Thu, 13 May 2021 02:30:03 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7EAA6613FE for ; Thu, 13 May 2021 02:30:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7EAA6613FE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QxNz/a9oQ7zhqh6k1267RfUO5itU7kwLnDoRaUARUq0=; b=BXN15sbdiECmrH K1gxN4yGCmW0rQvb+otrCDi2hReh5Nt/p7ZHWVwJrleexUXt7fWI1Qo870Y7dGEJTFhGZt8f518Y7 wi+XwT8+2MwTsR1CRgRT6swG685HPaIPxkO2Y/5sqAP+3VDY+itiV91vxCIxV/R4ztCYAhI1F0YQm 6zQ9O7zJM2+9Gr1KgMhZpbbcniwH59qRy2qU1Uc5+ax3ihUE9xNLYO3obWEEmYdpXKRpiaEw5rIpL SoJAXTZrpcT9gnekl++qNBs2ngE4SoXO1PsC6dB9klQ4rHEXhb3/iNHYnCSD4Md7TYF12SRkzr5Tz EMPxjZ7VVJf8I0lHFrYA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lh14s-004VRA-5l; Thu, 13 May 2021 02:27:54 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lh14N-004VMj-Eq for linux-arm-kernel@desiato.infradead.org; Thu, 13 May 2021 02:27:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=HDZsh4JM8Q1JtS/KQVFloZzQR7kVxgV3pjliw+t9vEo=; b=4YKvtlPIcso0m8jcqnzSBSIukM pL3W7oAwo/IkZRrgBK/VuKxTmWyzzGlyXgwVO5CaimcUUv445lZ1XbCBlLDxFU0ONNigdgzUzoHc+ XFNgv7n6Xr+Bp7Zk8fiHRuvkmZ5PE3zOt0PIwihWDmpV0DSVbBxqWa4rm2awVTlCZWDY6YFggTUUJ aqB76ivR3Z1mstXEcsBQ+kRJKBOWpPJA+Sugdc0/IO1sHLb8fcY1REoNTqdjdZW2PML+Bq9yISNWl ZK0GcnG+Cggri348vXTGGYkVqzAuXpxth8W3ymYR8PHvhiIz/FMqcbpxnUpu1nTXGysV9pfdvFxt+ aVTfIvow==; Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lh14K-00AyI7-1N for linux-arm-kernel@lists.infradead.org; Thu, 13 May 2021 02:27:22 +0000 Received: by mail-qt1-x84a.google.com with SMTP id g11-20020ac84b6b0000b02901dfe301071fso7239792qts.15 for ; Wed, 12 May 2021 19:27:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HDZsh4JM8Q1JtS/KQVFloZzQR7kVxgV3pjliw+t9vEo=; b=C/vS5rtNDhoc9o60TrKqYRf+GVaDBD2twx780KhJj17gRNoSPMQOWTBtmHNmCKrHAz bQyG+RRNwtcUvrHqoqys5qcbqBK2o7gr6fy8fXmhg2mu3LMmIocmlyLA6eVMD58lNylq zdxrf4atBel1R0bqWJkEpkSgE3uhlW1+X7zqFVtYRo9xdIoLLqEvzQpxwvhNxrmH+1Cv d28R+CWtNsu06qdENKkqwXgnxSaMxEdg1ise3HTAg3yWRH0B8XY4fIrrZlzT1R7j+/Kh 1UGGNsv0lwg2xw1xupWFHNUPsYgwn9bGPFSHP7JkfSJHAaVUeNf/lOY4oczm72nwgrzU cL4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HDZsh4JM8Q1JtS/KQVFloZzQR7kVxgV3pjliw+t9vEo=; b=ZOHFImm+/iUnLx4wFTG9v4zwylOd/xOiiQ7/PTWxYfn6zO3WzKGtM1znYcS1Yf6cFk 8hfbRX13kXNDzSV/aXRuJYaO+QakDCrSbaseFHROpBOw/RoDlGH+8RBlDj+BvWsX10ic 8sc/JO19Cf79pwGX0uXuSL/okhl98+2U4MjEvn4F+LdorTquOfP47mze+8oMtqvsBoZg uHZ8ioq5Gpov1ADQzTrbKLaBtqRUdZlQhq0ZrQiz/q2Ni+rdQer3wJvq4Hhj6ad83YxI etcU6trFIgB4dJfeGBEFR8OzwXcJCZ6J8sh96VVlzSLjH/UJyrfL8WBM7ODl6aPvpDn3 i3ag== X-Gm-Message-State: AOAM533oWQWxa1Wzlsx7/ytwOrbDkkvVmpr9goVrL2I4X4yXkbzPlzAm nmCg/Stt1AUyKYJ5VTlN77/ihDU= X-Google-Smtp-Source: ABdhPJxroPkU3DzoWylF4lUUeDznty04hgi2RZ+RbhcU3OmqrFV92alT9izuMaBz5H3g22u81nu3Ozc= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:d8b:fba6:20f0:cbe3]) (user=pcc job=sendgmr) by 2002:a0c:9b83:: with SMTP id o3mr10647600qve.4.1620872838347; Wed, 12 May 2021 19:27:18 -0700 (PDT) Date: Wed, 12 May 2021 19:27:09 -0700 In-Reply-To: <20210513022709.983982-1-pcc@google.com> Message-Id: <20210513022709.983982-3-pcc@google.com> Mime-Version: 1.0 References: <20210513022709.983982-1-pcc@google.com> X-Mailer: git-send-email 2.31.1.607.g51e8a6a459-goog Subject: [PATCH v4 3/3] kasan: arm64: support specialized outlined tag mismatch checks From: Peter Collingbourne To: Mark Brown , Mark Rutland , Will Deacon , Catalin Marinas , Andrey Konovalov , Evgenii Stepanov , Ard Biesheuvel Cc: Peter Collingbourne , Linux ARM , Andrey Konovalov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210512_192720_109893_620D8032 X-CRM114-Status: GOOD ( 32.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org By using outlined checks we can achieve a significant code size improvement by moving the tag-based ASAN checks into separate functions. Unlike the existing CONFIG_KASAN_OUTLINE mode these functions have a custom calling convention that preserves most registers and is specialized to the register containing the address and the type of access, and as a result we can eliminate the code size and performance overhead of a standard calling convention such as AAPCS for these functions. This change depends on a separate series of changes to Clang [1] to support outlined checks in the kernel, although the change works fine without them (we just don't get outlined checks). This is because the flag -mllvm -hwasan-inline-all-checks=0 has no effect until the Clang changes land. The flag was introduced in the Clang 9.0 timeframe as part of the support for outlined checks in userspace and because our minimum Clang version is 10.0 we can pass it unconditionally. Outlined checks require a new runtime function with a custom calling convention. Add this function to arch/arm64/lib. I measured the code size of defconfig + tag-based KASAN, as well as boot time (i.e. time to init launch) on a DragonBoard 845c with an Android arm64 GKI kernel. The results are below: code size boot time CONFIG_KASAN_INLINE=y before 92824064 6.18s CONFIG_KASAN_INLINE=y after 38822400 6.65s CONFIG_KASAN_OUTLINE=y 39215616 11.48s We can see straight away that specialized outlined checks beat the existing CONFIG_KASAN_OUTLINE=y on both code size and boot time for tag-based ASAN. As for the comparison between CONFIG_KASAN_INLINE=y before and after we saw similar performance numbers in userspace [2] and decided that since the performance overhead is minimal compared to the overhead of tag-based ASAN itself as well as compared to the code size improvements we would just replace the inlined checks with the specialized outlined checks without the option to select between them, and that is what I have implemented in this patch. Signed-off-by: Peter Collingbourne Acked-by: Andrey Konovalov Reviewed-by: Mark Rutland Tested-by: Mark Rutland Link: https://linux-review.googlesource.com/id/I1a30036c70ab3c3ee78d75ed9b87ef7cdc3fdb76 Link: [1] https://reviews.llvm.org/D90426 Link: [2] https://reviews.llvm.org/D56954 --- v4: - rebase to 5.13rc1 v3: - adopt Mark Rutland's suggested changes - move frame record alignment patches behind this one v2: - use calculations in the stack spills and restores - improve the comment at the top of the function - add a BTI instruction arch/arm64/include/asm/asm-prototypes.h | 6 ++ arch/arm64/include/asm/module.lds.h | 17 +++++- arch/arm64/lib/Makefile | 2 + arch/arm64/lib/kasan_sw_tags.S | 76 +++++++++++++++++++++++++ mm/kasan/sw_tags.c | 7 +++ scripts/Makefile.kasan | 1 + 6 files changed, 107 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/lib/kasan_sw_tags.S diff --git a/arch/arm64/include/asm/asm-prototypes.h b/arch/arm64/include/asm/asm-prototypes.h index 1c9a3a0c5fa5..ec1d9655f885 100644 --- a/arch/arm64/include/asm/asm-prototypes.h +++ b/arch/arm64/include/asm/asm-prototypes.h @@ -23,4 +23,10 @@ long long __ashlti3(long long a, int b); long long __ashrti3(long long a, int b); long long __lshrti3(long long a, int b); +/* + * This function uses a custom calling convention and cannot be called from C so + * this prototype is not entirely accurate. + */ +void __hwasan_tag_mismatch(unsigned long addr, unsigned long access_info); + #endif /* __ASM_PROTOTYPES_H */ diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h index 810045628c66..a11ccadd47d2 100644 --- a/arch/arm64/include/asm/module.lds.h +++ b/arch/arm64/include/asm/module.lds.h @@ -1,7 +1,20 @@ -#ifdef CONFIG_ARM64_MODULE_PLTS SECTIONS { +#ifdef CONFIG_ARM64_MODULE_PLTS .plt 0 (NOLOAD) : { BYTE(0) } .init.plt 0 (NOLOAD) : { BYTE(0) } .text.ftrace_trampoline 0 (NOLOAD) : { BYTE(0) } -} #endif + +#ifdef CONFIG_KASAN_SW_TAGS + /* + * Outlined checks go into comdat-deduplicated sections named .text.hot. + * Because they are in comdats they are not combined by the linker and + * we otherwise end up with multiple sections with the same .text.hot + * name in the .ko file. The kernel module loader warns if it sees + * multiple sections with the same name so we use this sections + * directive to force them into a single section and silence the + * warning. + */ + .text.hot : { *(.text.hot) } +#endif +} diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index d31e1169d9b8..8e60d76a1b47 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -18,3 +18,5 @@ obj-$(CONFIG_CRC32) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_ARM64_MTE) += mte.o + +obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/arch/arm64/lib/kasan_sw_tags.S b/arch/arm64/lib/kasan_sw_tags.S new file mode 100644 index 000000000000..5b04464c045e --- /dev/null +++ b/arch/arm64/lib/kasan_sw_tags.S @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + */ + +#include +#include + +/* + * Report a tag mismatch detected by tag-based KASAN. + * + * A compiler-generated thunk calls this with a non-AAPCS calling + * convention. Upon entry to this function, registers are as follows: + * + * x0: fault address (see below for restore) + * x1: fault description (see below for restore) + * x2 to x15: callee-saved + * x16 to x17: safe to clobber + * x18 to x30: callee-saved + * sp: pre-decremented by 256 bytes (see below for restore) + * + * The caller has decremented the SP by 256 bytes, and created a + * structure on the stack as follows: + * + * sp + 0..15: x0 and x1 to be restored + * sp + 16..231: free for use + * sp + 232..247: x29 and x30 (same as in GPRs) + * sp + 248..255: free for use + * + * Note that this is not a struct pt_regs. + * + * To call a regular AAPCS function we must save x2 to x15 (which we can + * store in the gaps), and create a frame record (for which we can use + * x29 and x30 spilled by the caller as those match the GPRs). + * + * The caller expects x0 and x1 to be restored from the structure, and + * for the structure to be removed from the stack (i.e. the SP must be + * incremented by 256 prior to return). + */ +SYM_CODE_START(__hwasan_tag_mismatch) +#ifdef BTI_C + BTI_C +#endif + add x29, sp, #232 + stp x2, x3, [sp, #8 * 2] + stp x4, x5, [sp, #8 * 4] + stp x6, x7, [sp, #8 * 6] + stp x8, x9, [sp, #8 * 8] + stp x10, x11, [sp, #8 * 10] + stp x12, x13, [sp, #8 * 12] + stp x14, x15, [sp, #8 * 14] +#ifndef CONFIG_SHADOW_CALL_STACK + str x18, [sp, #8 * 18] +#endif + + mov x2, x30 + bl kasan_tag_mismatch + + ldp x0, x1, [sp] + ldp x2, x3, [sp, #8 * 2] + ldp x4, x5, [sp, #8 * 4] + ldp x6, x7, [sp, #8 * 6] + ldp x8, x9, [sp, #8 * 8] + ldp x10, x11, [sp, #8 * 10] + ldp x12, x13, [sp, #8 * 12] + ldp x14, x15, [sp, #8 * 14] +#ifndef CONFIG_SHADOW_CALL_STACK + ldr x18, [sp, #8 * 18] +#endif + ldp x29, x30, [sp, #8 * 29] + + /* remove the structure from the stack */ + add sp, sp, #256 + ret +SYM_CODE_END(__hwasan_tag_mismatch) +EXPORT_SYMBOL(__hwasan_tag_mismatch) diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c index 9df8e7f69e87..9362938abbfa 100644 --- a/mm/kasan/sw_tags.c +++ b/mm/kasan/sw_tags.c @@ -207,3 +207,10 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache, return &alloc_meta->free_track[i]; } + +void kasan_tag_mismatch(unsigned long addr, unsigned long access_info, + unsigned long ret_ip) +{ + kasan_report(addr, 1 << (access_info & 0xf), access_info & 0x10, + ret_ip); +} diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan index 3d791908ed36..801c415bac59 100644 --- a/scripts/Makefile.kasan +++ b/scripts/Makefile.kasan @@ -50,6 +50,7 @@ endif CFLAGS_KASAN := -fsanitize=kernel-hwaddress \ $(call cc-param,hwasan-instrument-stack=$(stack_enable)) \ $(call cc-param,hwasan-use-short-granules=0) \ + $(call cc-param,hwasan-inline-all-checks=0) \ $(instrumentation_flags) endif # CONFIG_KASAN_SW_TAGS