From patchwork Wed Sep 28 12:32:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 12992223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D2EAC32771 for ; Wed, 28 Sep 2022 12:32:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14CDD8E0143; Wed, 28 Sep 2022 08:32:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FD8C8E0120; Wed, 28 Sep 2022 08:32:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F05C78E0143; Wed, 28 Sep 2022 08:32:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E08B28E0120 for ; Wed, 28 Sep 2022 08:32:31 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A8D0D1612A0 for ; Wed, 28 Sep 2022 12:32:31 +0000 (UTC) X-FDA: 79961432502.24.FBBEFC6 Received: from mail-lj1-f201.google.com (mail-lj1-f201.google.com [209.85.208.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 8C612180004 for ; Wed, 28 Sep 2022 12:32:29 +0000 (UTC) Received: by mail-lj1-f201.google.com with SMTP id bx10-20020a05651c198a00b0026c1cdb5b4cso3524555ljb.2 for ; Wed, 28 Sep 2022 05:32:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date; bh=JsFIRRYJ2T9TagGvX3xWTjwzlw12xCXMfZ5c7W97p44=; b=FI/rRBInCnWHnuN2dGpvZCvzhz3ex15QmBJNiZqcQewEhE1XXwLDQpMLVBembXx+NZ 8Ksx1qY5ZxNbZnSAEfl5Pefz0U7T4c1XTkNXvIa5kOh7UrbLcELqZZ9DFQbSjNujzoZI DdS0An9I3TzRMDWzrkrLX2GcWvU67F3rXWH4EJ+8Tpbp7krbla92cE49cHIoapPQiP3a UkDvQb3S6zidubvXA3+yiZn9QLddxIqVl/XSWDU1NQWD5L2xHWn8+NAuSrPVD67RxxOf y/17LzkB6IVkbnkaf4v6w4rfJ24jjYLsn3wSxjcJg0maoN4pzwb/6/27rtQvnIVceF4P a5GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date; bh=JsFIRRYJ2T9TagGvX3xWTjwzlw12xCXMfZ5c7W97p44=; b=Q9C7mnUyH7CxEsStqz/LORh/p2SlxOJgA62PHD/Mg4bBgPPZHq8CqPg4S8bpM2JuFn pEgUv/jVqWlZWN5srMoOs7Y3MBC6uAxeBURBkb09JSGzWJzTRbErXSIGF7pmtbdiVr7b fOjW0K9qFQNqcgVoytEg0OqtQXWejxuZbBs8dvt+B/U2HTqnH0d+O9VWytJqrgE12FhH 1Gnxz1SnaCGTK4jcpptkzhv5Tm5FNcYG8w2Aa5/yYqIeKPvF5JUWLV6a7NQRuLrEhScg so7LEh0/IfPc/DZi9eckxFt+EUxWG/ETXe3wLS2cdcR1PF/TK6ckVgqUHnMjvMax6vGW VxtQ== X-Gm-Message-State: ACrzQf0GVQMYlmVjOtMCj7yRv7UxOY1p1szx+Ppm1kjd8qpZTuOMHgnz bqO3j0q+v9PdxaNA6C1ifyJ6guvQ8Hc= X-Google-Smtp-Source: AMsMyM7TdHDyxQ9jhSzFoaghSjQsnbdkOZBFSfedCadqIIjUK0N6a9//RbYrgMikASYo96tMEl2Zqvg9elo= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:6933:6590:440d:394]) (user=glider job=sendgmr) by 2002:a05:6512:23a0:b0:499:4c84:b7a6 with SMTP id c32-20020a05651223a000b004994c84b7a6mr14207019lfv.390.1664368347413; Wed, 28 Sep 2022 05:32:27 -0700 (PDT) Date: Wed, 28 Sep 2022 14:32:19 +0200 Mime-Version: 1.0 X-Mailer: git-send-email 2.37.3.998.g577e59143f-goog Message-ID: <20220928123219.1101883-1-glider@google.com> Subject: [PATCH mm] x86: kmsan: handle CPU entry area From: Alexander Potapenko To: glider@google.com Cc: elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, peterz@infradead.org, luto@kernel.org, dave.hansen@linux.intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664368351; a=rsa-sha256; cv=none; b=WtvEmyqMtR6mY7TgJLlBgBW5FOBzzN77MXqBCjeRL8FkHbi4z1Rayra22NuCQezSe+Pigg XmwI3U3hAT+TlAGW6sRxvlaUHYcQWJg5ON+fPXIrkMw9H6SZoUasV2R6046hb3ISsunpzu hsyJ1W1GvMmLxeJMnAzrpz2cEVwfEDE= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="FI/rRBIn"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 32z40YwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com designates 209.85.208.201 as permitted sender) smtp.mailfrom=32z40YwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664368351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=JsFIRRYJ2T9TagGvX3xWTjwzlw12xCXMfZ5c7W97p44=; b=ioTBKboNNk1oenGWv9tUGZXqe+Uu11QGfU6jmfZlyffUM3/2ONHhvYNhar1oQy3403XiA9 NyVAWd03G7HTzRlUJ/jx+18zC2piFcJFvwdv8fad/uTS/7qSjSbmnp/MXp+6tnCw8uBTYx 5Uc25tISUWygeFX29uCS9MUauSIdxWs= X-Rspam-User: X-Rspamd-Queue-Id: 8C612180004 X-Rspamd-Server: rspam04 X-Stat-Signature: 493znj76os9qqnepgosheetd5pg671rx Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="FI/rRBIn"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 32z40YwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com designates 209.85.208.201 as permitted sender) smtp.mailfrom=32z40YwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com X-HE-Tag: 1664368349-816353 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Among other data, CPU entry area holds exception stacks, so addresses from this area can be passed to kmsan_get_metadata(). This previously led to kmsan_get_metadata() returning NULL, which in turn resulted in a warning that triggered further attempts to call kmsan_get_metadata() in the exception context, which quickly exhausted the exception stack. This patch allocates shadow and origin for the CPU entry area on x86 and introduces arch_kmsan_get_meta_or_null(), which performs arch-specific metadata mapping. Signed-off-by: Alexander Potapenko Fixes: 21d723a7c1409 ("kmsan: add KMSAN runtime core") --- MAINTAINERS | 1 + arch/x86/include/asm/kmsan.h | 32 ++++++++++++++++++++++++++++++++ arch/x86/mm/Makefile | 3 +++ arch/x86/mm/kmsan_shadow.c | 20 ++++++++++++++++++++ mm/kmsan/shadow.c | 6 +++++- 5 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 arch/x86/mm/kmsan_shadow.c diff --git a/MAINTAINERS b/MAINTAINERS index 517e71ea02156..3a39eb1fd3f6f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11381,6 +11381,7 @@ L: kasan-dev@googlegroups.com S: Maintained F: Documentation/dev-tools/kmsan.rst F: arch/*/include/asm/kmsan.h +F: arch/*/mm/kmsan_* F: include/linux/kmsan*.h F: lib/Kconfig.kmsan F: mm/kmsan/ diff --git a/arch/x86/include/asm/kmsan.h b/arch/x86/include/asm/kmsan.h index a790b865d0a68..8fa6ac0e2d766 100644 --- a/arch/x86/include/asm/kmsan.h +++ b/arch/x86/include/asm/kmsan.h @@ -11,9 +11,41 @@ #ifndef MODULE +#include #include #include +DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow); +DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin); + +/* + * Functions below are declared in the header to make sure they are inlined. + * They all are called from kmsan_get_metadata() for every memory access in + * the kernel, so speed is important here. + */ + +/* + * Compute metadata addresses for the CPU entry area on x86. + */ +static inline void *arch_kmsan_get_meta_or_null(void *addr, bool is_origin) +{ + unsigned long addr64 = (unsigned long)addr; + char *metadata_array; + unsigned long off; + int cpu; + + if ((addr64 < CPU_ENTRY_AREA_BASE) || + (addr64 >= (CPU_ENTRY_AREA_BASE + CPU_ENTRY_AREA_MAP_SIZE))) + return NULL; + cpu = (addr64 - CPU_ENTRY_AREA_BASE) / CPU_ENTRY_AREA_SIZE; + off = addr64 - (unsigned long)get_cpu_entry_area(cpu); + if ((off < 0) || (off >= CPU_ENTRY_AREA_SIZE)) + return NULL; + metadata_array = is_origin ? cpu_entry_area_origin : + cpu_entry_area_shadow; + return &per_cpu(metadata_array[off], cpu); +} + /* * Taken from arch/x86/mm/physaddr.h to avoid using an instrumented version. */ diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 39c0700c9955c..76b5a0f7533cc 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -43,6 +43,9 @@ obj-$(CONFIG_HIGHMEM) += highmem_32.o KASAN_SANITIZE_kasan_init_$(BITS).o := n obj-$(CONFIG_KASAN) += kasan_init_$(BITS).o +KMSAN_SANITIZE_kmsan_shadow.o := n +obj-$(CONFIG_KMSAN) += kmsan_shadow.o + obj-$(CONFIG_MMIOTRACE) += mmiotrace.o mmiotrace-y := kmmio.o pf_in.o mmio-mod.o obj-$(CONFIG_MMIOTRACE_TEST) += testmmiotrace.o diff --git a/arch/x86/mm/kmsan_shadow.c b/arch/x86/mm/kmsan_shadow.c new file mode 100644 index 0000000000000..bee2ec4a3bfa8 --- /dev/null +++ b/arch/x86/mm/kmsan_shadow.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * x86-specific bits of KMSAN shadow implementation. + * + * Copyright (C) 2022 Google LLC + * Author: Alexander Potapenko + */ + +#include +#include + +/* + * Addresses within the CPU entry area (including e.g. exception stacks) do not + * have struct page entries corresponding to them, so they need separate + * handling. + * arch_kmsan_get_meta_or_null() (declared in the header) maps the addresses in + * CPU entry area to addresses in cpu_entry_area_shadow/cpu_entry_area_origin. + */ +DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow); +DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin); diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c index 6e90a806a7045..21e3e196ec3cf 100644 --- a/mm/kmsan/shadow.c +++ b/mm/kmsan/shadow.c @@ -12,7 +12,6 @@ #include #include #include -#include #include #include #include @@ -126,6 +125,7 @@ void *kmsan_get_metadata(void *address, bool is_origin) { u64 addr = (u64)address, pad, off; struct page *page; + void *ret; if (is_origin && !IS_ALIGNED(addr, KMSAN_ORIGIN_SIZE)) { pad = addr % KMSAN_ORIGIN_SIZE; @@ -136,6 +136,10 @@ void *kmsan_get_metadata(void *address, bool is_origin) kmsan_internal_is_module_addr(address)) return (void *)vmalloc_meta(address, is_origin); + ret = arch_kmsan_get_meta_or_null(address, is_origin); + if (ret) + return ret; + page = virt_to_page_or_null(address); if (!page) return NULL;