From patchwork Mon Sep 21 13:26:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 11789851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD3CE112C for ; Mon, 21 Sep 2020 13:26:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6F00E221EC for ; Mon, 21 Sep 2020 13:26:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="qL2wY6WB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F00E221EC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 684CB90005E; Mon, 21 Sep 2020 09:26:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 633BC900059; Mon, 21 Sep 2020 09:26:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AF2A90005E; Mon, 21 Sep 2020 09:26:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0102.hostedemail.com [216.40.44.102]) by kanga.kvack.org (Postfix) with ESMTP id 1B3D7900059 for ; Mon, 21 Sep 2020 09:26:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CFF43362B for ; Mon, 21 Sep 2020 13:26:42 +0000 (UTC) X-FDA: 77287143444.10.cover70_3b065c727145 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id A7D7516A0B9 for ; Mon, 21 Sep 2020 13:26:42 +0000 (UTC) X-Spam-Summary: 1,0,0,abf5ede7356fb169,d41d8cd98f00b204,3eapoxwukcaoov5o1qyyqvo.mywvsx47-wwu5kmu.y1q@flex--elver.bounces.google.com,,RULES_HIT:41:152:355:379:421:541:800:960:973:988:989:1260:1277:1313:1314:1345:1359:1431:1437:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:2393:2559:2562:2897:2903:3138:3139:3140:3141:3142:3152:3355:3865:3867:3868:3870:3871:3874:4118:4250:4321:4605:5007:6261:6653:6742:6743:7576:9969:10004:10400:11026:11473:11657:11658:11914:12043:12219:12291:12296:12297:12438:12555:12679:12683:12895:12986:13141:13230:14096:14097:14181:14394:14659:14721:21080:21324:21433:21444:21451:21611:21627:21990:30003:30054:30074,0,RBL:209.85.219.201:@flex--elver.bounces.google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04ygrcuiprthq74yff4s94gak8kqbyc8gyuagpjkn6add4zbpopq7tym7ycabh5.gk7qepytjdc9j7gi16g65jpxfbrgiro39mm8s9chzb7yygxmjg9zemipz3pon8w.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SP F:fp,MSB X-HE-Tag: cover70_3b065c727145 X-Filterd-Recvd-Size: 7244 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Mon, 21 Sep 2020 13:26:42 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id z40so12826376ybi.3 for ; Mon, 21 Sep 2020 06:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=wt63fMbRtF63+v7/GuudoZYJcaqBUfqT28LRHuRusC0=; b=qL2wY6WBUQcx5pj52oZp6zYoEVKIicz7S4B4rLCTdAz6PS5lHgO06blQ7f6xllHz2o hagGVVZ/7022UmkPMfpMkPPKNZDUdgN63y58z5Wfn/b4uf508D80p+A1ZuFXfGAkimes i74FDyotzaUDEAHidRxPTcdVhHau2t6wLifADNQe4rPzoE3e+aWQkaHadVV09M08zOf4 oK+GxDQ+bjyvYR/vvbrC2I5GIiNu8ZKvhpcg4822VkRb+LENM1snc9rW613oonoj5WHN 8+tuJRn5I/69FohHkuBzfpzx0pGYBsGBPz7yJAAa6pfKkS/ktySeqDYNwSHOS1IJ7BrI EDJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wt63fMbRtF63+v7/GuudoZYJcaqBUfqT28LRHuRusC0=; b=N0jRMO/UNQnkNfQ6IlGKT9IvpbLKg/7FWQvRBzHo5EjSK1azhiqzhhWzgvYEQkzK0W N5lJAYkA/f5RqqE0ZfuN3thnirfwtoGQLlIuUBhzMwRtsknoV9KZFMcFY5WPQcXE90Y1 0Znkt0ra5stj90/hs7VHN0N6emjF52vjQ5MiGrEmSW7Lm7VDK1KVhsJhjNrk2Fy8qkWW whmZgSGI3u9cUTqyBon+GUjxA4FsmazxIn9JWYHoNWNxFpYFp0d/79H/Q834YOZJ8jZ0 vS0VHK0NABnTZ/r94bCuwUrFpBKWlVUIcgG7dyx7aRo49mV1m9ia8GQfcyG/itByqJKH P4Xw== X-Gm-Message-State: AOAM5321eGce4oyn/h5ZoBr8mYYjzwbyUzr47qrGvjFD9402imGdlrPt uD+O2WjPXclW4cyFgD1O2BjQTKApTA== X-Google-Smtp-Source: ABdhPJzdGcwUC1pBrVmyD+23TVrmfQ31wFp42KfuDHSbJFG8Jn48HEPqt8sIw6l6hPH7bByrLSO6bhg5eg== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a25:69cb:: with SMTP id e194mr49735261ybc.243.1600694801234; Mon, 21 Sep 2020 06:26:41 -0700 (PDT) Date: Mon, 21 Sep 2020 15:26:03 +0200 In-Reply-To: <20200921132611.1700350-1-elver@google.com> Message-Id: <20200921132611.1700350-3-elver@google.com> Mime-Version: 1.0 References: <20200921132611.1700350-1-elver@google.com> X-Mailer: git-send-email 2.28.0.681.g6f77f65b4e-goog Subject: [PATCH v3 02/10] x86, kfence: enable KFENCE for x86 From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com Cc: hpa@zytor.com, paulmck@kernel.org, andreyknvl@google.com, aryabinin@virtuozzo.com, luto@kernel.org, bp@alien8.de, catalin.marinas@arm.com, cl@linux.com, dave.hansen@linux.intel.com, rientjes@google.com, dvyukov@google.com, edumazet@google.com, gregkh@linuxfoundation.org, hdanton@sina.com, mingo@redhat.com, jannh@google.com, Jonathan.Cameron@huawei.com, corbet@lwn.net, iamjoonsoo.kim@lge.com, keescook@chromium.org, mark.rutland@arm.com, penberg@kernel.org, peterz@infradead.org, sjpark@amazon.com, tglx@linutronix.de, vbabka@suse.cz, will@kernel.org, x86@kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Potapenko Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Reviewed-by: Dmitry Vyukov Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- arch/x86/Kconfig | 2 ++ arch/x86/include/asm/kfence.h | 60 +++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 4 +++ 3 files changed, 66 insertions(+) create mode 100644 arch/x86/include/asm/kfence.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..e22dc722698c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -144,6 +144,8 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KFENCE_STATIC_POOL select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h new file mode 100644 index 000000000000..cf09e377faf9 --- /dev/null +++ b/arch/x86/include/asm/kfence.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_X86_KFENCE_H +#define _ASM_X86_KFENCE_H + +#include +#include + +#include +#include +#include +#include + +/* The alignment should be at least a 4K page. */ +#define KFENCE_POOL_ALIGNMENT PAGE_SIZE + +/* + * The page fault handler entry function, up to which the stack trace is + * truncated in reports. + */ +#define KFENCE_SKIP_ARCH_FAULT_HANDLER "asm_exc_page_fault" + +/* Force 4K pages for __kfence_pool. */ +static inline bool arch_kfence_initialize_pool(void) +{ + unsigned long addr; + + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + unsigned int level; + + if (!lookup_address(addr, &level)) + return false; + + if (level != PG_LEVEL_4K) + set_memory_4k(addr, 1); + } + + return true; +} + +/* Protect the given page and flush TLBs. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + unsigned int level; + pte_t *pte = lookup_address(addr, &level); + + if (!pte || level != PG_LEVEL_4K) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_PRESENT)); + else + set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); + + flush_tlb_one_kernel(addr); + return true; +} + +#endif /* _ASM_X86_KFENCE_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 6e3e8a124903..423e15ad5eb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -9,6 +9,7 @@ #include /* oops_begin/end, ... */ #include /* search_exception_tables */ #include /* max_low_pfn */ +#include /* kfence_handle_page_fault */ #include /* NOKPROBE_SYMBOL, ... */ #include /* kmmio_handler, ... */ #include /* perf_sw_event */ @@ -701,6 +702,9 @@ no_context(struct pt_regs *regs, unsigned long error_code, } #endif + if (kfence_handle_page_fault(address)) + return; + /* * 32-bit: *