From patchwork Tue Oct 27 14:15:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 11860555 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6310C921 for ; Tue, 27 Oct 2020 14:16:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1AD3E2072D for ; Tue, 27 Oct 2020 14:16:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="lqYJFavI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AD3E2072D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0834A6B0070; Tue, 27 Oct 2020 10:16:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 00A206B0071; Tue, 27 Oct 2020 10:16:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC7D16B0072; Tue, 27 Oct 2020 10:16:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id AD0826B0070 for ; Tue, 27 Oct 2020 10:16:43 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 409E7180AD804 for ; Tue, 27 Oct 2020 14:16:43 +0000 (UTC) X-FDA: 77417906286.22.tree34_44015292727c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id F406718038E68 for ; Tue, 27 Oct 2020 14:16:42 +0000 (UTC) X-Spam-Summary: 1,0,0,8dbdcb7e819236a8,d41d8cd98f00b204,3ysuyxwukciqmt3mzowwotm.kwutqv25-uus3iks.wzo@flex--elver.bounces.google.com,,RULES_HIT:41:152:355:379:421:541:800:960:973:988:989:1260:1277:1313:1314:1345:1359:1431:1437:1516:1518:1535:1544:1593:1594:1605:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2553:2559:2562:2897:2901:2903:3138:3139:3140:3141:3142:3152:3865:3866:3867:3868:3870:3871:3874:4118:4250:4321:4605:5007:6117:6261:6653:6742:6743:7576:7903:9108:9969:10004:11026:11473:11657:11658:11914:12043:12219:12291:12295:12296:12297:12438:12555:12679:12683:12895:12986:13141:13230:14096:14097:14181:14394:14659:14721:21063:21080:21220:21324:21433:21444:21451:21627:21966:21990:30003:30054:30070:30074:30090,0,RBL:209.85.128.74:@flex--elver.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04ygbj6kqfo3z8iw45zdxk5bttewcypa1mw5z3ikwtbet91tnqsdiy3wjayxxmm.86zubsr65q39etwpy4u1hgwk7nkza1yejas4qnwqo9yrfn34iqu638qjgz8ouz8.y-lbl8.mailshell.net-223.238.255.100,Cach eIP:none X-HE-Tag: tree34_44015292727c X-Filterd-Recvd-Size: 7998 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Tue, 27 Oct 2020 14:16:42 +0000 (UTC) Received: by mail-wm1-f74.google.com with SMTP id b23so638972wmj.6 for ; Tue, 27 Oct 2020 07:16:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=amWdP6FV7IwvIOgrtWK6Wa5LLRx0C6kQ2UzTi3lX/Ww=; b=lqYJFavIH+U4gFwR4Ufpqsg9KCYNXbJ3F8gnrRSYEHQI3/A9+te13dBCntzIVhHjPA yM/u2APl511RqstDqN55oOyS5UP4F22H/qyDB5WR7vaX3lk9aNkuR0sQDMgO4HTJoT2P nUddrSTOur3uISgb2z/F6hHZ6MIYXhpIqkCKj2iS3tCmoXeHvH04eMp18wWCBfWo3H9C IWJ3GPgBuk64n7JwdO61szS3AlX7mUf1cu8NrRaBk+3UmrzTr4PXcXohLQCOyybY0a39 ZogLQhigFh5RhWSVOsR8UYLQRHv3SabS6DfurmVtNpuvQc1oCiGBsU8hyksRke4zFsLy e1pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=amWdP6FV7IwvIOgrtWK6Wa5LLRx0C6kQ2UzTi3lX/Ww=; b=HHW5PMRigB2tONKik3htNOLSaXbyRYR2//BK6guqbaEiOgclAdh6Xf0jhlqBW4NHqh tLelF2/Xmy9sPL+HP9Lj/Mdwl+msKRJKVrTieqDCV4M4VGOAR64aP+dK6wcoJtH+Chw2 woMKCpXtgPvyq+71msrf4VZE79zfyMiFu1OeVVfM5aZOkjze/wraxOhIXtvy92I7ZwWS 5cQ0WEoO3PP6HG7S02MuV1ntVAtbdl0ixNf9mzKhYYim7+mLfXlaG/JSLDH8RzFRgLFd pjlb1TyerRMEjvIgjTSxxsWzchnfvOCAqTUke3QjLYihkCvWdTLwAD4DqgO+FqlI4yDo nb0Q== X-Gm-Message-State: AOAM530nYz5Ebg4x7N0WqUegGtgPo7dkzJSQUYtiLi60RGHE+Vm/L6zr EUmn6SFfzJaE0jvNYo0zP9mnBG8apw== X-Google-Smtp-Source: ABdhPJxg2TfSVwFzHU/1bZA22yprPnzmXJsVKmsjs/zjJxuqWYfMpJQfyT+LsH4Tz15aeMtqUAjINyxpZw== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:adf:eac6:: with SMTP id o6mr3348520wrn.117.1603808201007; Tue, 27 Oct 2020 07:16:41 -0700 (PDT) Date: Tue, 27 Oct 2020 15:15:59 +0100 In-Reply-To: <20201027141606.426816-1-elver@google.com> Message-Id: <20201027141606.426816-3-elver@google.com> Mime-Version: 1.0 References: <20201027141606.426816-1-elver@google.com> X-Mailer: git-send-email 2.29.0.rc2.309.g374f81d7ae-goog Subject: [PATCH v5 2/9] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, catalin.marinas@arm.com, cl@linux.com, dave.hansen@linux.intel.com, rientjes@google.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, hdanton@sina.com, mingo@redhat.com, jannh@google.com, Jonathan.Cameron@huawei.com, corbet@lwn.net, iamjoonsoo.kim@lge.com, joern@purestorage.com, keescook@chromium.org, mark.rutland@arm.com, penberg@kernel.org, peterz@infradead.org, sjpark@amazon.com, tglx@linutronix.de, vbabka@suse.cz, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Reviewed-by: Dmitry Vyukov Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- v5: * MAJOR CHANGE: Switch to the memblock_alloc'd pool. Running benchmarks with the newly optimized is_kfence_address(), no difference between baseline and KFENCE is observed. * Suggested by Jann Horn: * Move x86 kfence_handle_page_fault before oops handling. * WARN_ON in kfence_protect_page if non-4K pages. * Better comments for x86 kfence_protect_page. v4: * Define __kfence_pool_attrs. --- arch/x86/Kconfig | 1 + arch/x86/include/asm/kfence.h | 65 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 70 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f6946b81f74a..c9ec6b5ba358 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,7 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..beeac105dae7 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_init_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLB. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (WARN_ON(!pte || level != PG_LEVEL_4K)) + return false; + + /* + * We need to avoid IPIs, as we may get KFENCE allocations or faults + * with interrupts disabled. Therefore, the below is best-effort, and + * does not flush TLBs on all CPUs. We can tolerate some inaccuracy; + * lazy fault handling takes care of faults after the page is PRESENT. + */ + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + /* Flush this CPU's TLB. */ + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 82bf37a5c9ec..380638745f42 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -725,6 +726,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, if (IS_ENABLED(CONFIG_EFI)) efi_recover_from_page_fault(address); + if (kfence_handle_page_fault(address)) + return; + oops: /* * Oops. The kernel tried to access some bad page. We'll have to