From patchwork Thu Jul 11 14:25:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040065 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1FF25112C for ; Thu, 11 Jul 2019 14:26:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EA8328AC8 for ; Thu, 11 Jul 2019 14:26:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0246128AD9; Thu, 11 Jul 2019 14:26:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2229E28AC8 for ; Thu, 11 Jul 2019 14:26:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05B9E8E00C3; Thu, 11 Jul 2019 10:26:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 00CB78E00BF; Thu, 11 Jul 2019 10:26:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3BD18E00C3; Thu, 11 Jul 2019 10:26:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by kanga.kvack.org (Postfix) with ESMTP id C3E7C8E00BF for ; Thu, 11 Jul 2019 10:26:12 -0400 (EDT) Received: by mail-io1-f71.google.com with SMTP id q26so6961728ioi.10 for ; Thu, 11 Jul 2019 07:26:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=01fwbgJFKdvEoWu4pwKWewOW1PRaOpulx/v99NC5AeE=; b=UfFL0UELSd75gSeXU2tHnMAyGOdUYwjAPc7KIQxB99ingLGiEnBgkFqLZ51z/ZPFSZ jZKaeXW3YG7P5g/A0q3lvm862FuzcnjsyDaOTNRRGUu7xwf1equIReckQvE1W0O28Cae w/jSm4Vn3vVXxOY7l8h48R+pYbaE2tNBszB5r7cRwB/8c1gMFiHweMQUNqgjbzt344zU 3v4HKa/0MO1VpMWhZBun+C5Ao/Zb810YBJGcMZa3IWxIUA5yCtMSplXZ6hresWPuVM/i xciCfoqmxLxLHPIbVHf92TrVqsz7gtRwUcJXvkwF4PEZy5iZr/o+tA0mNJQx9LMgNxXT xnzw== X-Gm-Message-State: APjAAAXLG2TgDofq2ZTTw/ShNjjxWexssKH49tbomfA/RlH+FfegRgWh KFI9DOKe0ID37xrUdVOn4tvCc19aaaPQETZC7VZNxqgwX0uYnh+dZ6i6kesMXXCdMxSP0Sf8i/K pDkBTOH4/xAw2Q70GLwRSSSllk6DyOY9cYFz1j0xCNLei+/ol8xt04R9+T6R/+f7Zmg== X-Received: by 2002:a6b:790a:: with SMTP id i10mr4382675iop.150.1562855172483; Thu, 11 Jul 2019 07:26:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqx4dXR7dqAr3nSE12kt2GAZVxYONscGB1FlyqevRgupQ8VNv3289VKXroE4s5C6SaoT4HOH X-Received: by 2002:a6b:790a:: with SMTP id i10mr4382570iop.150.1562855171268; Thu, 11 Jul 2019 07:26:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855171; cv=none; d=google.com; s=arc-20160816; b=jQA5Gswh+ZS2UQaz4e5MD7eShyGFpCqP2lKP61G9GNr3RN+h2f0P4J3Q3uTArGce9l upsqe/XqAtj0m4d72hrA7SWJAJ+H8nMoiCKJ0P7WqZ3ASTtVYKn9jumzKEGpNEUZq7I0 nxYY7MNKGIVh1wtNLwB+JYlKtzwE16ZEskMdFduNwhLa3oRx+11sTi2ELNIB3FQuDT/w 9c4aiYrySJr990F/PBP7KiBNmGbC3x9FBrllZUnh0ZD4ju6ALMA7Vt5cSN5ClmjzJW0u 9PxF0p//OYI5BWqVpBilnIXMzVXYQv6IHykf/pmbF5K+7Aa9pkRZlOVdf0XUxPG02NXe 8jlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=01fwbgJFKdvEoWu4pwKWewOW1PRaOpulx/v99NC5AeE=; b=Xtg998v+qUGafnhcC1kDEb5R9YZEFXlxLuNHmnjU/iR8bdbOkzhhIhrHM/L6nDkD8+ 6GQ3rGSqv4tMRves4pfsqOSRAQTrTDF/4DPN56kccDoiG5WsZPZx9juAzBYP4qm55oXc e0QVKvz1zPn+ICXVvF29dO9bCkWVODh2nytIwZObz2YgQjJKJvynpXiMDm858SF71W2v VJhqBN/+ycN5r2zs7ruo55Vu1XEoxWjQoBKbmluqQLwnxz/luDu00M4KHd2e2CWA1w2K w6UDXJRtr2RN+7IEifVi0YrfietBxepjRaECi9ob6OCYCVhPTNV7WA7GcCAaICvmh+VE c02w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=OUZHm+j4; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id t8si11151295jan.0.2019.07.11.07.26.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:11 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=OUZHm+j4; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOEWc100497; Thu, 11 Jul 2019 14:25:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=01fwbgJFKdvEoWu4pwKWewOW1PRaOpulx/v99NC5AeE=; b=OUZHm+j4S+p6/v1st7v0l2XFGb9Yzt9tSB6OEwidnVH8C7ZbftVmyTxkNolkQOvSzhgY cbHKPq4ORwMiLSvoraLOmO32yfqW3gASglaC7FLB9Ef9R707DAjk/G6dzWZVUMkGyvAu iay9bQQTO3wg3AefHAS5a13R3q6omWxKLrNSwhL0s5bdk/AqX+ms0qxcg6eBJqXRKS+p 1w7YM5+MF3gpt84KMBf39eDBip4O9sr14o2Z9tFI78xInQo9JS7l178MkVnONAP6qWu2 2HBBJ2jtAhFVr8aFlUUKhi4FcgaY2f8JyFXyHWl81HeHbgMzGEK96qXqtVniyPis1l+7 jw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0c5k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:25:50 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcts021444; Thu, 11 Jul 2019 14:25:47 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 01/26] mm/x86: Introduce kernel address space isolation Date: Thu, 11 Jul 2019 16:25:13 +0200 Message-Id: <1562855138-19507-2-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce core functions and structures for implementing Address Space Isolation (ASI). Kernel address space isolation provides the ability to run some kernel code with a reduced kernel address space. An address space isolation is defined with a struct asi structure which has its own page-table. While, for now, this page-table is empty, it will eventually be possible to populate it so that it is much smaller than the full kernel page-table. Isolation is entered by calling asi_enter() which switches the kernel page-table to the address space isolation page-table. Isolation is then exited by calling asi_exit() which switches the page-table back to the kernel page-table. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 41 ++++++++++++ arch/x86/mm/Makefile | 2 + arch/x86/mm/asi.c | 152 ++++++++++++++++++++++++++++++++++++++++++++ security/Kconfig | 10 +++ 4 files changed, 205 insertions(+), 0 deletions(-) create mode 100644 arch/x86/include/asm/asi.h create mode 100644 arch/x86/mm/asi.c diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h new file mode 100644 index 0000000..8a13f73 --- /dev/null +++ b/arch/x86/include/asm/asi.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_MM_ASI_H +#define ARCH_X86_MM_ASI_H + +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + +#include +#include + +struct asi { + spinlock_t lock; /* protect all attributes */ + pgd_t *pgd; /* ASI page-table */ +}; + +/* + * An ASI session maintains the state of address state isolation on a + * cpu. There is one ASI session per cpu. There is no lock to protect + * members of the asi_session structure as each cpu is managing its + * own ASI session. + */ + +enum asi_session_state { + ASI_SESSION_STATE_INACTIVE, /* no address space isolation */ + ASI_SESSION_STATE_ACTIVE, /* address space isolation is active */ +}; + +struct asi_session { + struct asi *asi; /* ASI for this session */ + enum asi_session_state state; /* state of ASI session */ + unsigned long original_cr3; /* cr3 before entering ASI */ + struct task_struct *task; /* task during isolation */ +} __aligned(PAGE_SIZE); + +extern struct asi *asi_create(void); +extern void asi_destroy(struct asi *asi); +extern int asi_enter(struct asi *asi); +extern void asi_exit(struct asi *asi); + +#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */ + +#endif diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 84373dc..dae5c8a 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -49,7 +49,9 @@ obj-$(CONFIG_X86_INTEL_MPX) += mpx.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c new file mode 100644 index 0000000..c3993b7 --- /dev/null +++ b/arch/x86/mm/asi.c @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. + * + * Kernel Address Space Isolation (ASI) + */ + +#include +#include +#include +#include +#include + +#include +#include +#include + +/* ASI sessions, one per cpu */ +DEFINE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); + +static int asi_init_mapping(struct asi *asi) +{ + /* + * TODO: Populate the ASI page-table with minimal mappings so + * that we can at least enter isolation and abort. + */ + return 0; +} + +struct asi *asi_create(void) +{ + struct page *page; + struct asi *asi; + int err; + + asi = kzalloc(sizeof(*asi), GFP_KERNEL); + if (!asi) + return NULL; + + page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!page) + goto error; + + asi->pgd = page_address(page); + spin_lock_init(&asi->lock); + + err = asi_init_mapping(asi); + if (err) + goto error; + + return asi; + +error: + asi_destroy(asi); + return NULL; +} +EXPORT_SYMBOL(asi_create); + +void asi_destroy(struct asi *asi) +{ + if (!asi) + return; + + if (asi->pgd) + free_page((unsigned long)asi->pgd); + + kfree(asi); +} +EXPORT_SYMBOL(asi_destroy); + + +/* + * When isolation is active, the address space doesn't necessarily map + * the percpu offset value (this_cpu_off) which is used to get pointers + * to percpu variables. So functions which can be invoked while isolation + * is active shouldn't be getting pointers to percpu variables (i.e. with + * get_cpu_var() or this_cpu_ptr()). Instead percpu variable should be + * directly read or written to (i.e. with this_cpu_read() or + * this_cpu_write()). + */ + +int asi_enter(struct asi *asi) +{ + enum asi_session_state state; + struct asi *current_asi; + struct asi_session *asi_session; + + state = this_cpu_read(cpu_asi_session.state); + /* + * We can re-enter isolation, but only with the same ASI (we don't + * support nesting isolation). Also, if isolation is still active, + * then we should be re-entering with the same task. + */ + if (state == ASI_SESSION_STATE_ACTIVE) { + current_asi = this_cpu_read(cpu_asi_session.asi); + if (current_asi != asi) { + WARN_ON(1); + return -EBUSY; + } + WARN_ON(this_cpu_read(cpu_asi_session.task) != current); + return 0; + } + + /* isolation is not active so we can safely access the percpu pointer */ + asi_session = &get_cpu_var(cpu_asi_session); + asi_session->asi = asi; + asi_session->task = current; + asi_session->original_cr3 = __get_current_cr3_fast(); + if (!asi_session->original_cr3) { + WARN_ON(1); + err = -EINVAL; + goto err_clear_asi; + } + asi_session->state = ASI_SESSION_STATE_ACTIVE; + + load_cr3(asi->pgd); + + return 0; + +err_clear_asi: + asi_session->asi = NULL; + asi_session->task = NULL; + + return err; + +} +EXPORT_SYMBOL(asi_enter); + +void asi_exit(struct asi *asi) +{ + struct asi_session *asi_session; + enum asi_session_state asi_state; + unsigned long original_cr3; + + asi_state = this_cpu_read(cpu_asi_session.state); + if (asi_state == ASI_SESSION_STATE_INACTIVE) + return; + + /* TODO: Kick sibling hyperthread before switching to kernel cr3 */ + original_cr3 = this_cpu_read(cpu_asi_session.original_cr3); + if (original_cr3) + write_cr3(original_cr3); + + /* page-table was switched, we can now access the percpu pointer */ + asi_session = &get_cpu_var(cpu_asi_session); + WARN_ON(asi_session->task != current); + asi_session->state = ASI_SESSION_STATE_INACTIVE; + asi_session->asi = NULL; + asi_session->task = NULL; + asi_session->original_cr3 = 0; +} +EXPORT_SYMBOL(asi_exit); diff --git a/security/Kconfig b/security/Kconfig index 466cc1f..241b9a7 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -65,6 +65,16 @@ config PAGE_TABLE_ISOLATION See Documentation/x86/pti.txt for more details. +config ADDRESS_SPACE_ISOLATION + bool "Allow code to run with a reduced kernel address space" + default y + depends on (X86_64 || X86_PAE) && !UML + help + This feature provides the ability to run some kernel code + with a reduced kernel address space. This can be used to + mitigate speculative execution attacks which are able to + leak data between sibling CPU hyper-threads. + config SECURITY_INFINIBAND bool "Infiniband Security Hooks" depends on SECURITY && INFINIBAND From patchwork Thu Jul 11 14:25:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040073 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99E50112C for ; Thu, 11 Jul 2019 14:26:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8415E28A5A for ; Thu, 11 Jul 2019 14:26:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 711D728AD9; Thu, 11 Jul 2019 14:26:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4767328A5A for ; Thu, 11 Jul 2019 14:26:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47A5E8E00C7; Thu, 11 Jul 2019 10:26:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 452938E00C4; Thu, 11 Jul 2019 10:26:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A4E98E00C7; Thu, 11 Jul 2019 10:26:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by kanga.kvack.org (Postfix) with ESMTP id 098958E00C4 for ; Thu, 11 Jul 2019 10:26:15 -0400 (EDT) Received: by mail-io1-f71.google.com with SMTP id r27so6955035iob.14 for ; Thu, 11 Jul 2019 07:26:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=eEi0ZTZbIEZ47JResTnFegGOS+nY5vwRKMSKjoZA8rw=; b=PCAAXDQjagJk+w6iab3zF5iQ2m6rarqdjpl8O3LdXgGSL030yDS6V5QgehotauMdLt qn9V9IEucNmzGLsRJRXpCeooHPgFIwKR5vh5ff8H9NoicRs8XJu1Gf6dlhBnYa/u2yJR hI2lPdFrWxdElhAsqySIZRHOwEKJncWqCCeLZTsoxfKqReMDYRb3N7Na+fNJPyaz1ZkT yqceFSrCrcaCFA2ZDvYzY80nZy/q/UYYSnYi4j6X6fCJp2GnTP0+r1U4ztGeDkPyh5S7 bjnVPQVTvLedb1VUBdpyN3jEA6/CuxgO9pwmb+Ngucq5Kk9FvtfoBFVjFyoZKIuUKNEX EfIA== X-Gm-Message-State: APjAAAXPs3tHnlCSayYX7qbrnQuzPDYNi2Wf7fOT1BXVlImnUcuQcyQu L7GeirzhlgrIV69+zAVQovqE4ncFEcOFTtTNMmZeTLfH0PnZD0D/mgwxO7zPC5fb6wXQTHjSYex I9sp0Do2VWhUlbKd6fItbn/7vb/VN/toDobScwuIRNZDcJ7hvduhSeQzFyLAQyuck+w== X-Received: by 2002:a6b:ed01:: with SMTP id n1mr1251294iog.255.1562855174780; Thu, 11 Jul 2019 07:26:14 -0700 (PDT) X-Google-Smtp-Source: APXvYqztWe9GdJ/M8YhchDDKwrVp4kWf6ryJAkWVKY1y/0M4oGwRy34iNvI8SXAOBdTFwcnJkvaA X-Received: by 2002:a6b:ed01:: with SMTP id n1mr1251182iog.255.1562855173568; Thu, 11 Jul 2019 07:26:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855173; cv=none; d=google.com; s=arc-20160816; b=MrX3ebL0/ji/VcbZ82IjmlWEqJN7ty9QmRr1L8jvgW6mW5tenWmBNOg2xjKFIDb/z4 pvyw9a+Cd851j7MB+h4IYXbH4fHyouzFQV+5BaVAhOoOngBpeImz0pCZwIXEMhijPYHY MSJTdQuJpgLXkAUo2od7P/1LHsbjVUQ5lFxIS1Rv4UL5+x5SfIEP5uvV8dw8/M/QLRyI Y+L60BaFHigKa+ByjSj0vWj4DE189I7As08uh1Y+PliCFwDrt0KS3EwonjM4XQEgAKys z04yvZ4MQdiPaQZnDKN1LDa/4YAEQYfNWT64xHagGJP40DseOJkHz1/inj5GlVbApvE3 /+VA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eEi0ZTZbIEZ47JResTnFegGOS+nY5vwRKMSKjoZA8rw=; b=HKZYi/nPPoGmNuEcBq1KCYNfKliVjW3ZcwPRcSTkFn9hrFcT85yXdsI1yDkEvjZ8gf AP6NBHB1sc3cKBMrrd9cvnh4C1jnNhSkFQt6Tc4UGc0xxaUB6lIJHTOrCRleLlOZ1GWO Uhwe+HUOGrIzmXoBSC8e7gWQ9XvEB0HttejbSwobcxQu7RJl5CRMrBlibq2svIH+JjgJ V7Zf2ci9sjHtXKUEhArpCLoVRN0/8eA1nuycjH93r54m7mjB9TdlxeF+p/NktMF14+Eb prLK3Cm7tsOjWrJzpmfxMtXtXcWEs3T1GfYUnFs688dBiBcPO65Ecl383LO1tC7Qa5JA fK0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=doVQHd64; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id x22si8828685iob.85.2019.07.11.07.26.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:13 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=doVQHd64; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOGvV100511; Thu, 11 Jul 2019 14:25:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=eEi0ZTZbIEZ47JResTnFegGOS+nY5vwRKMSKjoZA8rw=; b=doVQHd64if/2kjiu3iJcq23b/ysNGaO3zFQ1Fbt9B8gbpk5AqZTdAGLfrkqQEQ1t29L7 vIj+rjQt4iariplk3zmGOGhNmniQGZlOnZRXIEVu/XP2NILI1mTBKcs9y9sxpXNe7zLm WTpN/PNi/5Jc1UHjSLknTN0GAMM8aeWagwwQaQOKeG6DbEY5KTVF1BDM73GoF0OnyPfM bujoi6mhv9a226crDPB4jhcZ5qwIaENDIb5Eyq65a1CYJBrVBQCE3nFbdPTH6gGouBW2 4zVkdrvRe3v52Mrj+Eaw7ZDk5eDA1G0rOGRsj+izv+9ASQxDllLZRPrfz0ej8u3KokCM vA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0c69-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:25:58 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPctt021444; Thu, 11 Jul 2019 14:25:50 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 02/26] mm/asi: Abort isolation on interrupt, exception and context switch Date: Thu, 11 Jul 2019 16:25:14 +0200 Message-Id: <1562855138-19507-3-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Address space isolation should be aborted if there is an interrupt, an exception or a context switch. Interrupt/exception handlers and context switch code need to run with the full kernel address space. Address space isolation is aborted by restoring the original CR3 value used before entering address space isolation. Signed-off-by: Alexandre Chartre --- arch/x86/entry/entry_64.S | 42 ++++++++++- arch/x86/include/asm/asi.h | 114 ++++++++++++++++++++++++++++ arch/x86/kernel/asm-offsets.c | 4 + arch/x86/mm/asi.c | 165 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/core.c | 4 + 5 files changed, 315 insertions(+), 14 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 11aa3b2..3dc6174 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -38,6 +38,7 @@ #include #include #include +#include #include #include "calling.h" @@ -558,8 +559,15 @@ ENTRY(interrupt_entry) TRACE_IRQS_OFF CALL_enter_from_user_mode - +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + jmp 2f +#endif 1: +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* Abort address space isolation if it is active */ + ASI_START_ABORT +2: +#endif ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 /* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF @@ -583,6 +591,9 @@ common_interrupt: call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + ASI_FINISH_ABORT +#endif DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF @@ -947,6 +958,9 @@ ENTRY(\sym) addq $\ist_offset, CPU_TSS_IST(\shift_ist) .endif +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + ASI_FINISH_ABORT +#endif /* these procedures expect "no swapgs" flag in ebx */ .if \paranoid jmp paranoid_exit @@ -1182,6 +1196,16 @@ ENTRY(paranoid_entry) xorl %ebx, %ebx 1: +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* + * If address space isolation is active then abort it and return + * the original kernel CR3 in %r14. + */ + ASI_START_ABORT_ELSE_JUMP 2f + movq %rdi, %r14 + ret +2: +#endif /* * Always stash CR3 in %r14. This value will be restored, * verbatim, at exit. Needed if paranoid_entry interrupted @@ -1265,6 +1289,15 @@ ENTRY(error_entry) CALL_enter_from_user_mode ret +.Lerror_entry_check_address_space_isolation: +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* + * Abort address space isolation if it is active. This will restore + * the original kernel CR3. + */ + ASI_START_ABORT +#endif + .Lerror_entry_done: TRACE_IRQS_OFF ret @@ -1283,7 +1316,7 @@ ENTRY(error_entry) cmpq %rax, RIP+8(%rsp) je .Lbstep_iret cmpq $.Lgs_change, RIP+8(%rsp) - jne .Lerror_entry_done + jne .Lerror_entry_check_address_space_isolation /* * hack: .Lgs_change can fail with user gsbase. If this happens, fix up @@ -1632,7 +1665,10 @@ end_repeat_nmi: movq %rsp, %rdi movq $-1, %rsi call do_nmi - + +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + ASI_FINISH_ABORT +#endif /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14 diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 8a13f73..ff126e1 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -4,6 +4,8 @@ #ifdef CONFIG_ADDRESS_SPACE_ISOLATION +#ifndef __ASSEMBLY__ + #include #include @@ -22,20 +24,132 @@ struct asi { enum asi_session_state { ASI_SESSION_STATE_INACTIVE, /* no address space isolation */ ASI_SESSION_STATE_ACTIVE, /* address space isolation is active */ + ASI_SESSION_STATE_ABORTED, /* isolation has been aborted */ }; struct asi_session { struct asi *asi; /* ASI for this session */ enum asi_session_state state; /* state of ASI session */ + bool retry_abort; /* always retry abort */ + unsigned int abort_depth; /* abort depth */ unsigned long original_cr3; /* cr3 before entering ASI */ struct task_struct *task; /* task during isolation */ } __aligned(PAGE_SIZE); +DECLARE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); + extern struct asi *asi_create(void); extern void asi_destroy(struct asi *asi); extern int asi_enter(struct asi *asi); extern void asi_exit(struct asi *asi); +/* + * Function to exit the current isolation. This is used to abort isolation + * when a task using isolation is scheduled out. + */ +static inline void asi_abort(void) +{ + enum asi_session_state asi_state; + + asi_state = this_cpu_read(cpu_asi_session.state); + if (asi_state == ASI_SESSION_STATE_INACTIVE) + return; + + asi_exit(this_cpu_read(cpu_asi_session.asi)); +} + +/* + * Barriers for code which sets CR3 to use the ASI page-table. That's + * the case, for example, when entering isolation, or during a VMExit if + * isolation was active. If such a code is interrupted before CR3 is + * effectively set, then the interrupt will abort isolation and restore + * the original CR3 value. But then, the code will sets CR3 to use the + * ASI page-table while isolation has been aborted by the interrupt. + * + * To prevent this issue, such a code should call asi_barrier_begin() + * before CR3 gets updated, and asi_barrier_end() after CR3 has been + * updated. + * + * asi_barrier_begin() will set retry_abort to true. This will force + * interrupts to retain the isolation abort state. Then, after the code + * has updated CR3, asi_barrier_end() will be able to check if isolation + * was aborted and effectively abort isolation in that case. Setting + * retry_abort to true will also force all interrupt to restore the + * original CR3; that's in case we have interrupts both before and + * after CR3 is set. + */ +static inline unsigned long asi_restore_cr3(void) +{ + unsigned long original_cr3; + + /* TODO: Kick sibling hyperthread before switching to kernel cr3 */ + original_cr3 = this_cpu_read(cpu_asi_session.original_cr3); + if (original_cr3) + write_cr3(original_cr3); + + return original_cr3; +} + +static inline void asi_barrier_begin(void) +{ + this_cpu_write(cpu_asi_session.retry_abort, true); + mb(); +} + +static inline void asi_barrier_end(void) +{ + enum asi_session_state state; + + this_cpu_write(cpu_asi_session.retry_abort, false); + mb(); + state = this_cpu_read(cpu_asi_session.state); + if (state == ASI_SESSION_STATE_ABORTED) { + (void) asi_restore_cr3(); + asi_abort(); + return; + } + +} + +#else /* __ASSEMBLY__ */ + +/* + * If address space isolation is active, start aborting isolation. + */ +.macro ASI_START_ABORT + movl PER_CPU_VAR(cpu_asi_session + CPU_ASI_SESSION_state), %edi + testl %edi, %edi + jz .Lasi_start_abort_done_\@ + call asi_start_abort +.Lasi_start_abort_done_\@: +.endm + +/* + * If address space isolation is active, finish aborting isolation. + */ +.macro ASI_FINISH_ABORT + movl PER_CPU_VAR(cpu_asi_session + CPU_ASI_SESSION_state), %edi + testl %edi, %edi + jz .Lasi_finish_abort_done_\@ + call asi_finish_abort +.Lasi_finish_abort_done_\@: +.endm + +/* + * If address space isolation is inactive then jump to the specified + * label. Otherwise, start aborting isolation. + */ +.macro ASI_START_ABORT_ELSE_JUMP asi_inactive_label:req + movl PER_CPU_VAR(cpu_asi_session + CPU_ASI_SESSION_state), %edi + testl %edi, %edi + jz \asi_inactive_label + call asi_start_abort + testq %rdi, %rdi + jz \asi_inactive_label +.endm + +#endif /* __ASSEMBLY__ */ + #endif /* CONFIG_ADDRESS_SPACE_ISOLATION */ #endif diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 168543d..395d0c6 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -18,6 +18,7 @@ #include #include #include +#include #ifdef CONFIG_XEN #include @@ -105,4 +106,7 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); + + BLANK(); + OFFSET(CPU_ASI_SESSION_state, asi_session, state); } diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index c3993b7..fabb923 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -84,9 +84,17 @@ int asi_enter(struct asi *asi) enum asi_session_state state; struct asi *current_asi; struct asi_session *asi_session; + unsigned long original_cr3; state = this_cpu_read(cpu_asi_session.state); /* + * The "aborted" state is a transient state used in interrupt and + * exception handlers while aborting isolation. So it shouldn't be + * set when entering isolation. + */ + WARN_ON(state == ASI_SESSION_STATE_ABORTED); + + /* * We can re-enter isolation, but only with the same ASI (we don't * support nesting isolation). Also, if isolation is still active, * then we should be re-entering with the same task. @@ -105,15 +113,44 @@ int asi_enter(struct asi *asi) asi_session = &get_cpu_var(cpu_asi_session); asi_session->asi = asi; asi_session->task = current; - asi_session->original_cr3 = __get_current_cr3_fast(); - if (!asi_session->original_cr3) { + WARN_ON(asi_session->abort_depth > 0); + + /* + * Instructions ordering is important here because we should be + * able to deal with any interrupt/exception which will abort + * the isolation and restore CR3 to its original value: + * + * - asi_session->original_cr3 must be set before the ASI session + * becomes active (i.e. before setting asi_session->state to + * ASI_SESSION_STATE_ACTIVE); + * - the ASI session must be marked as active (i.e. set + * asi_session->state to ASI_SESSION_STATE_ACTIVE) before + * loading the CR3 used during isolation. + * + * Any exception or interrupt occurring after asi_session->state is + * set to ASI_SESSION_STATE_ACTIVE will cause the exception/interrupt + * handler to abort the isolation. The handler will then restore + * cr3 to asi_session->original_cr3 and move asi_session->state to + * ASI_SESSION_STATE_ABORTED. + */ + original_cr3 = __get_current_cr3_fast(); + if (!original_cr3) { WARN_ON(1); err = -EINVAL; goto err_clear_asi; } - asi_session->state = ASI_SESSION_STATE_ACTIVE; + asi_session->original_cr3 = original_cr3; + /* + * Use ASI barrier as we are setting CR3 with the ASI page-table. + * The barrier should begin before setting the state to active as + * any interrupt after the state is active will abort isolation. + */ + asi_barrier_begin(); + asi_session->state = ASI_SESSION_STATE_ACTIVE; + mb(); load_cr3(asi->pgd); + asi_barrier_end(); return 0; @@ -130,23 +167,129 @@ void asi_exit(struct asi *asi) { struct asi_session *asi_session; enum asi_session_state asi_state; - unsigned long original_cr3; asi_state = this_cpu_read(cpu_asi_session.state); - if (asi_state == ASI_SESSION_STATE_INACTIVE) + switch (asi_state) { + case ASI_SESSION_STATE_INACTIVE: return; - - /* TODO: Kick sibling hyperthread before switching to kernel cr3 */ - original_cr3 = this_cpu_read(cpu_asi_session.original_cr3); - if (original_cr3) - write_cr3(original_cr3); + case ASI_SESSION_STATE_ACTIVE: + (void) asi_restore_cr3(); + break; + case ASI_SESSION_STATE_ABORTED: + /* + * No need to restore cr3, this was already done during + * the isolation abort. + */ + break; + } /* page-table was switched, we can now access the percpu pointer */ asi_session = &get_cpu_var(cpu_asi_session); - WARN_ON(asi_session->task != current); + /* + * asi_exit() can be interrupted before setting the state to + * ASI_SESSION_STATE_INACTIVE. In that case, the interrupt will + * exit isolation before we have started the actual exit. So + * check that the session ASI is still set to verify that an + * exit hasn't already be done. + */ asi_session->state = ASI_SESSION_STATE_INACTIVE; + mb(); + if (asi_session->asi == NULL) { + /* exit was already done */ + return; + } + WARN_ON(asi_session->retry_abort); + WARN_ON(asi_session->task != current); asi_session->asi = NULL; asi_session->task = NULL; asi_session->original_cr3 = 0; + + /* + * Reset abort_depth because some interrupt/exception handlers + * (like the user page-fault handler) can schedule us out and so + * exit isolation before abort_depth reaches 0. + */ + asi_session->abort_depth = 0; } EXPORT_SYMBOL(asi_exit); + +/* + * Functions to abort isolation. When address space isolation is active, + * these functions are used by interrupt/exception handlers to abort + * isolation. + * + * Common Case + * ----------- + * asi_start_abort() is invoked at the beginning of the interrupt/exception + * handler. It aborts isolation by restoring the original CR3 value, + * increments the abort count, and move the isolation state to "aborted" + * (ASI_SESSION_STATE_ABORTED). If the interrupt/exception is interrupted + * by another interrupt/exception then the new interrupt/exception will + * just increment the abort count. + * + * asi_finish_abort() is invoked at the end of the interrupt/exception + * handler. It decrements is abort count and if that count reaches zero + * then it invokes asi_exit() to exit isolation. + * + * Special Case When Entering Isolation + * ------------------------------------ + * When entering isolation, asi_enter() will set cpu_asi_session.retry_abort + * while updating CR3 to the ASI page-table. This forces asi_start_abort() + * handlers to abort isolation even if isolation was already aborted. Also + * asi_finish_abort() will retain the aborted state and not exit isolation + * (no call to asi_exit()). + */ +unsigned long asi_start_abort(void) +{ + enum asi_session_state state; + unsigned long original_cr3; + + state = this_cpu_read(cpu_asi_session.state); + + switch (state) { + + case ASI_SESSION_STATE_INACTIVE: + return 0; + + case ASI_SESSION_STATE_ACTIVE: + original_cr3 = asi_restore_cr3(); + this_cpu_write(cpu_asi_session.state, + ASI_SESSION_STATE_ABORTED); + break; + + case ASI_SESSION_STATE_ABORTED: + /* + * In the normal case, if the session was already aborted + * then CR3 has already been restored. However if retry_abort + * is set then we restore CR3 again. + */ + if (this_cpu_read(cpu_asi_session.retry_abort)) + original_cr3 = asi_restore_cr3(); + else + original_cr3 = this_cpu_read( + cpu_asi_session.original_cr3); + break; + } + + this_cpu_inc(cpu_asi_session.abort_depth); + + return original_cr3; +} + +void asi_finish_abort(void) +{ + enum asi_session_state state; + + state = this_cpu_read(cpu_asi_session.state); + if (state == ASI_SESSION_STATE_INACTIVE) + return; + + WARN_ON(state != ASI_SESSION_STATE_ABORTED); + + /* if retry_abort is set then we retain the abort state */ + if (this_cpu_dec_return(cpu_asi_session.abort_depth) > 0 || + this_cpu_read(cpu_asi_session.retry_abort)) + return; + + asi_exit(this_cpu_read(cpu_asi_session.asi)); +} diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 874c427..bb363f3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -14,6 +14,7 @@ #include #include +#include #include "../workqueue_internal.h" #include "../smpboot.h" @@ -2597,6 +2598,9 @@ static inline void finish_lock_switch(struct rq *rq) prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + asi_abort(); +#endif kcov_prepare_switch(prev); sched_info_switch(rq, prev, next); perf_event_task_sched_out(prev, next); From patchwork Thu Jul 11 14:25:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040069 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AF77112C for ; Thu, 11 Jul 2019 14:26:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1974E28A5A for ; Thu, 11 Jul 2019 14:26:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0CC2528AD1; Thu, 11 Jul 2019 14:26:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56D4728A5A for ; Thu, 11 Jul 2019 14:26:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42E498E00C5; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3B5E28E00C6; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A5C88E00C5; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 0844B8E00C4 for ; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id s9so6955630iob.11 for ; Thu, 11 Jul 2019 07:26:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=N1S18AguseqvsteFhWr/8Dc0t6ablZh92+dI5Teb2M0=; b=LQQMnkKjsmL38hNCqO1DaSRBihyR9APsS8Uv9wHGiHCnorR6zWYwa4mH7mei8LFCQ+ LhL9j6UhYbHDOLFZEypWA90RjPUOo6vRrUDzfJ3iZXNWP+cnm4dVCJL5LEppJnTtrx0s 6lbHyaECyyUpnEsQRPMtTipHLGYosJtD2Irz35Y8LW9OIOAREdt8u2G3eUxM3nUo/nD/ 8YexMdsclxWaBb68J2IQBxfGQFfM374VdJR6oejOjZsI3hJMrZyf8CapQVide72u2h7m xrZmUjBxLlBsKZVUkiOmkn8qTwlBF1vZZrJNzfhYOXDePDdalxKIU5RVybLPlgjM62Q4 3TQg== X-Gm-Message-State: APjAAAVYaMzaeRm0rvuO/OOk6voYa+H6DvXVJxP4MiXMK+3UcMzTQ/rz iNMVc8WDZw6Mr8qjs9VDT2NfLAm0miwmLuF4xWZ6RzBm0Mn880mWytUz6utM3fwOXc6cwJpt9Gx xMrSkH62cLi77v+xHXvE/NEhGqbOZ89c7YhBremCffJM3jYmvM0oGG4GnIgqjVaq8Ww== X-Received: by 2002:a6b:d81a:: with SMTP id y26mr4409487iob.126.1562855173797; Thu, 11 Jul 2019 07:26:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqxl03chD0DN0E1nWU1B727EHj+dZI4tyLlJ23fblaF+M/AdEXY3/n4JEStBgfrr44DBxlCJ X-Received: by 2002:a6b:d81a:: with SMTP id y26mr4409419iob.126.1562855173037; Thu, 11 Jul 2019 07:26:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855173; cv=none; d=google.com; s=arc-20160816; b=hlNlQednIz7jWNeNf/nUelAneu7hpmK/Xtvb1HX9y5i8+8eCR/9LTg+FggnpKX9pU6 BlAqNJergVCRjAlYTQObcA2R/yMiIbiff795PKi7iuUSjO60mgCB0mqIglKf5HDQJwVJ coQdmXqKaL1b0qaFRoWU1Byd1Y6q4rpW71xHfiiz5telO6Owrce11JLkFyl85fB4jToq 5InJwgvriaWqjY+dmv5XUK9sJQexLluXiak2wwFNvvMBQ49B377N/lgp0nPKoLRnQdZz kw3o9LxVycz6yXZ863kpKwyPlsZB+fZxGgfUWoK2Lznqn4pv8WOnO10SDOPOWvaxpIjG 1uhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=N1S18AguseqvsteFhWr/8Dc0t6ablZh92+dI5Teb2M0=; b=vvzYYQKcxMY60NVrLZ9LBDLkSVFnk7AeZ+Ebo5Q1QXQ9lhAmAAqMwCH404vtXOBsfV Jwj/DQTl2QUsRLIO62W2fGNK16ZYGNz973MK/d8HypbiXb4V7KhmIRGOeIhJeQ36/w8h E0eIbh8p1PmRb3cYk72RcgI0ROHcF4R7WM79CjQslRlMNC4eB3+CQyTSPinHnr4tpwkG xJVRHE/2Uv4pWSNNXwJD1QL/PmzgGJjTqPf9wK2YW8s2PWtgk34pJ7LXLPEfIsrUnc18 mspDhX+WHekAX+gT7YH/PoRoUyZcnp2V2efIKsD9IUcUq6vd962i6brgBG7D6VDO1Ohe AfcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=aEmCnnwO; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id s20si7926749iom.2.2019.07.11.07.26.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:13 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=aEmCnnwO; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO9M4001474; Thu, 11 Jul 2019 14:26:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=N1S18AguseqvsteFhWr/8Dc0t6ablZh92+dI5Teb2M0=; b=aEmCnnwO0ArceoX+05Si1M+AHvh5m5JJ1bSo9UUo9F7KLd7OqzZym5b7ZPYuJjAW3g4s Gb7xZ3/STbdj4kYw8hKG/DtOtrLsatWwPXpImxRc8wftUMiHP7PR1oUssKSt8pbPiUFk nOqjHRqOXAb4/fQSVgHDj2BF7mvKm78/5UcHY8N02qZKTsjiTAfY8DQq2Lf8iHYOueTq x/h6EnEuXbJlxgYKOboLBJWJ/+nQMtliAUEUPoaDXOXG/UfKOlKbOCOAR2+mYE6dul7p hiUvWM7MnEfAJk+hfdeMiPyCJVFvALeGy7d/JxN9SdvyplS80ju1Ttaf/csNEpaxSMA5 6Q== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dw7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:02 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPctu021444; Thu, 11 Jul 2019 14:25:53 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 03/26] mm/asi: Handle page fault due to address space isolation Date: Thu, 11 Jul 2019 16:25:15 +0200 Message-Id: <1562855138-19507-4-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When address space isolation is active, kernel page faults can occur because data are not mapped in the ASI page-table. In such a case, log information about the fault and report the page fault as handled. As the page fault handler (like any exception handler) aborts isolation and switch back to the full kernel page-table, the faulty instruction will be retried using the full kernel address space. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 7 ++++ arch/x86/mm/asi.c | 68 ++++++++++++++++++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 7 ++++ 3 files changed, 82 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index ff126e1..013d77a 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -9,9 +9,14 @@ #include #include +#define ASI_FAULT_LOG_SIZE 128 + struct asi { spinlock_t lock; /* protect all attributes */ pgd_t *pgd; /* ASI page-table */ + spinlock_t fault_lock; /* protect fault_log */ + unsigned long fault_log[ASI_FAULT_LOG_SIZE]; + bool fault_stack; /* display stack of fault? */ }; /* @@ -42,6 +47,8 @@ struct asi_session { extern void asi_destroy(struct asi *asi); extern int asi_enter(struct asi *asi); extern void asi_exit(struct asi *asi); +extern bool asi_fault(struct pt_regs *regs, unsigned long error_code, + unsigned long address); /* * Function to exit the current isolation. This is used to abort isolation diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index fabb923..717160d 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -18,6 +19,72 @@ /* ASI sessions, one per cpu */ DEFINE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); +static void asi_log_fault(struct asi *asi, struct pt_regs *regs, + unsigned long error_code, unsigned long address) +{ + int i = 0; + + /* + * Log information about the fault only if this is a fault + * we don't know about yet (and the fault log is not full). + */ + spin_lock(&asi->fault_lock); + for (i = 0; i < ASI_FAULT_LOG_SIZE; i++) { + if (asi->fault_log[i] == regs->ip) { + spin_unlock(&asi->fault_lock); + return; + } + if (!asi->fault_log[i]) { + asi->fault_log[i] = regs->ip; + break; + } + } + spin_unlock(&asi->fault_lock); + + if (i >= ASI_FAULT_LOG_SIZE) + pr_warn("ASI %p: fault log buffer is full [%d]\n", asi, i); + + pr_info("ASI %p: PF#%d (%ld) at %pS on %px\n", asi, i, + error_code, (void *)regs->ip, (void *)address); + + if (asi->fault_stack) + show_stack(NULL, (unsigned long *)regs->sp); +} + +bool asi_fault(struct pt_regs *regs, unsigned long error_code, + unsigned long address) +{ + struct asi_session *asi_session; + + /* + * If address space isolation was active when the fault occurred + * then the page fault handler has already aborted the isolation + * (exception handlers abort isolation very early) and switched + * CR3 back to its original value. + */ + + /* + * If address space isolation is not active, or we have a fault + * after isolation was aborted then this is a regular kernel fault, + * and we don't handle it. + */ + asi_session = &get_cpu_var(cpu_asi_session); + if (asi_session->state == ASI_SESSION_STATE_INACTIVE) + return false; + + WARN_ON(asi_session->state != ASI_SESSION_STATE_ABORTED); + WARN_ON(asi_session->abort_depth != 1); + + /* + * We have a fault while the cpu is using address space isolation. + * Log the fault and report that we have handled fault. This way, + * the faulty instruction will be retried with no isolation. + * + */ + asi_log_fault(asi_session->asi, regs, error_code, address); + return true; +} + static int asi_init_mapping(struct asi *asi) { /* @@ -43,6 +110,7 @@ struct asi *asi_create(void) asi->pgd = page_address(page); spin_lock_init(&asi->lock); + spin_lock_init(&asi->fault_lock); err = asi_init_mapping(asi); if (err) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 46df4c6..a405c43 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -29,6 +29,7 @@ #include /* efi_recover_from_page_fault()*/ #include /* store_idt(), ... */ #include /* exception stack */ +#include /* asi_fault() */ #define CREATE_TRACE_POINTS #include @@ -1252,6 +1253,12 @@ static int fault_in_kernel_space(unsigned long address) */ WARN_ON_ONCE(hw_error_code & X86_PF_PK); +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* Check if the fault occurs with address space isolation */ + if (asi_fault(regs, hw_error_code, address)) + return; +#endif + /* * We can fault-in kernel-space virtual memory on-demand. The * 'reference' page table is init_mm.pgd. From patchwork Thu Jul 11 14:25:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040071 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2316A112C for ; Thu, 11 Jul 2019 14:26:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 10DAB28A5A for ; Thu, 11 Jul 2019 14:26:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0402828AD1; Thu, 11 Jul 2019 14:26:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4234828A5A for ; Thu, 11 Jul 2019 14:26:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89C718E00C6; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 821EC8E00C4; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C3908E00C6; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 391E58E00C4 for ; Thu, 11 Jul 2019 10:26:14 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id h3so6948821iob.20 for ; Thu, 11 Jul 2019 07:26:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=SJbrUhc02AgBQXRxdyh8NMsUj+oqXOnPmIbZ5Fc5Oh8=; b=C5TPt4zpCYv7DedysdqCDPFMIOALE9tJrjzEV7QnqnSYhVOFtjHA2jf8rtUtwSzC+c TbG8e+6AximFQRrRe17xe92a3reoKLQNynMqxkR1u2kIP322x3zMjctnW79vRYm/7VQC 9oWdceXYgN1BFKmrwzuItYAuvKweHWCgkcz3YlUv+VSZnpX6hJdcOVRAfUwQsaH5URu9 ZJ8mwS0/49yaE7Lz/SZok1qiNT/1DYEwVGorg04O0XI86jOhbQ1UzIaw3EJPyLCjUgwv 6JqzRsbK10F1KoVJE5YrS8piRKm30WgJgEiaaAjj5zR8EOxbrCAsR/x3kZ9LCWolYaKq 65Kg== X-Gm-Message-State: APjAAAXK6iF3rgLZwc42bNQYPm3MZE5VYcsTpDPUSLxrJ6WpvScxcLok LLZ/W4q4bBJjSgEalU5CpVRVIrrxVAf+ShcAJYWt74TZVgJ71OMyvolUpYHdUek7FqlUi2TUOk/ d+u77rgDvExyKkX7078RGBZdHrytw1H85Yzsf6kSNFXF8x3qn37+RyO7vqzRTUpJ5BQ== X-Received: by 2002:a05:6602:2413:: with SMTP id s19mr4680461ioa.161.1562855173959; Thu, 11 Jul 2019 07:26:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqw6KBKPvTmr4al1nqBzniwWKnXMF9Of97qHxwxigS3XS5Lvm8E0jrdUItMeKwo6upWeHGt6 X-Received: by 2002:a05:6602:2413:: with SMTP id s19mr4680385ioa.161.1562855173075; Thu, 11 Jul 2019 07:26:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855173; cv=none; d=google.com; s=arc-20160816; b=Syv7ajoF6rlxN2l+XNGjcbzXZr57dZV5LesuY9duIrqMX3nbG/TnqLLzXASic2I/JT OdP3XITRzZsWKLQxDo0A24kVaoclKqi7R9qvyBPX0SvjP+yFZ/SwStkGOwGlnRdVCm3L dd9Jme76u4O3y4wrQfAIPY93E9C6dRzK/+UD0KsuBNJGXn7FJvvO7+qgEgdOkrs8HyVp claMI0kMerMGRxL+beGilk1rTWSgHxudrNBagvvTAT4sF0NhoKZrikgCjvtuWeAiPMrh +h5XVhmKrtqPYbNcW9vOLBM85LUlxsDnJjtUiQrL153vXsRyI97Dvh0xGLufRUCb4Yh0 Jfcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SJbrUhc02AgBQXRxdyh8NMsUj+oqXOnPmIbZ5Fc5Oh8=; b=V3HyZAyzlWeC/LtxblIzHjsCo0Ykw4F08A9bgo/ACiC0VcMxtvVupNSgUWMZn+RqTt 6nAU6BUvcUNI7TcaC+yyLvRDIZSO/2yVlHGaAn1E9CVvTK8bFn4S4aTYqQRpXQqDuVJj snEzAmhNZEnQEZChZccLa4My6rKnApbmbQhnuJmevdea7p50zqSgUFz4a9evIaJQxbec TTmGTFx6yimKpi1yzUCG38T5IJiy+LFqSKkw1lpcmaC9P/ygWJY+XGu3uidukRj2oPW/ c1q/Tio8QcslDeilgeaJrfvSr/wDb4vcVRqnQ2tqJvK5vAQeI6xUYvOiIZ4qExZ2E+4y Y4Mw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=xzFz4q2m; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id r1si9932885jac.1.2019.07.11.07.26.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:13 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=xzFz4q2m; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOgjM001960; Thu, 11 Jul 2019 14:26:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=SJbrUhc02AgBQXRxdyh8NMsUj+oqXOnPmIbZ5Fc5Oh8=; b=xzFz4q2m+w7TU5mOs3ALiXB2uWjQppudBO654kJwJ3xcjzMIDadZ8a8b6jz4YyHWy1y3 WQjbtVLYHkN/sQca6bf2zIB7mNhCTUasiGxRSTnzAv+FRaXEzVfw9Y+TtQO4hmByjCup HS11ZsQytlUEzwToD8/1Qka8/FUyswUFS6tpdq5B5ty/H9uU9+lZ/iY44X92sv0bKyZR FyCRbHGO9wKQsS+KaiSfk6fjhgaVbgZ3csy84EpWlVvgjG8YbMp2E1bRONCx7joKhrFT fgbFCcCBEHZLg4ujshUEkzIPwKG7zacpaOBPeP/3uweuUnh4D/616+hC65KocGucwirZ aQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dw2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:00 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPctv021444; Thu, 11 Jul 2019 14:25:56 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 04/26] mm/asi: Functions to track buffers allocated for an ASI page-table Date: Thu, 11 Jul 2019 16:25:16 +0200 Message-Id: <1562855138-19507-5-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add functions to track buffers allocated for an ASI page-table. An ASI page-table can have direct references to the kernel page table, at different levels (PGD, P4D, PUD, PMD). When freeing an ASI page-table, we should make sure that we free parts actually allocated for the ASI page-table, and not parts of the kernel page table referenced from the ASI page-table. To do so, we will keep track of buffers when building the ASI page-table. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 26 +++++++++++ arch/x86/mm/Makefile | 2 +- arch/x86/mm/asi.c | 3 + arch/x86/mm/asi_pagetable.c | 99 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 129 insertions(+), 1 deletions(-) create mode 100644 arch/x86/mm/asi_pagetable.c diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 013d77a..3d965e6 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -8,12 +8,35 @@ #include #include +#include + +enum page_table_level { + PGT_LEVEL_PTE, + PGT_LEVEL_PMD, + PGT_LEVEL_PUD, + PGT_LEVEL_P4D, + PGT_LEVEL_PGD +}; #define ASI_FAULT_LOG_SIZE 128 struct asi { spinlock_t lock; /* protect all attributes */ pgd_t *pgd; /* ASI page-table */ + + /* + * An ASI page-table can have direct references to the full kernel + * page-table, at different levels (PGD, P4D, PUD, PMD). When freeing + * an ASI page-table, we should make sure that we free parts actually + * allocated for the ASI page-table, and not part of the full kernel + * page-table referenced from the ASI page-table. + * + * To do so, the backend_pages XArray is used to keep track of pages + * used for the kernel isolation page-table. + */ + struct xarray backend_pages; /* page-table pages */ + unsigned long backend_pages_count; /* pages count */ + spinlock_t fault_lock; /* protect fault_log */ unsigned long fault_log[ASI_FAULT_LOG_SIZE]; bool fault_stack; /* display stack of fault? */ @@ -43,6 +66,9 @@ struct asi_session { DECLARE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); +void asi_init_backend(struct asi *asi); +void asi_fini_backend(struct asi *asi); + extern struct asi *asi_create(void); extern void asi_destroy(struct asi *asi); extern int asi_enter(struct asi *asi); diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index dae5c8a..b972f0f 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -49,7 +49,7 @@ obj-$(CONFIG_X86_INTEL_MPX) += mpx.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o -obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o +obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o asi_pagetable.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 717160d..dfde245 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -111,6 +111,7 @@ struct asi *asi_create(void) asi->pgd = page_address(page); spin_lock_init(&asi->lock); spin_lock_init(&asi->fault_lock); + asi_init_backend(asi); err = asi_init_mapping(asi); if (err) @@ -132,6 +133,8 @@ void asi_destroy(struct asi *asi) if (asi->pgd) free_page((unsigned long)asi->pgd); + asi_fini_backend(asi); + kfree(asi); } EXPORT_SYMBOL(asi_destroy); diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c new file mode 100644 index 0000000..7a8f791 --- /dev/null +++ b/arch/x86/mm/asi_pagetable.c @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. + * + */ + +#include + +/* + * Get the pointer to the beginning of a page table directory from a page + * table directory entry. + */ +#define ASI_BACKEND_PAGE_ALIGN(entry) \ + ((typeof(entry))(((unsigned long)(entry)) & PAGE_MASK)) + +/* + * Pages used to build the address space isolation page-table are stored + * in the backend_pages XArray. Each entry in the array is a logical OR + * of the page address and the page table level (PTE, PMD, PUD, P4D) this + * page is used for in the address space isolation page-table. + * + * As a page address is aligned with PAGE_SIZE, we have plenty of space + * for storing the page table level (which is a value between 0 and 4) in + * the low bits of the page address. + * + */ + +#define ASI_BACKEND_PAGE_ENTRY(addr, level) \ + ((typeof(addr))(((unsigned long)(addr)) | ((unsigned long)(level)))) +#define ASI_BACKEND_PAGE_ADDR(entry) \ + ((void *)(((unsigned long)(entry)) & PAGE_MASK)) +#define ASI_BACKEND_PAGE_LEVEL(entry) \ + ((enum page_table_level)(((unsigned long)(entry)) & ~PAGE_MASK)) + +static int asi_add_backend_page(struct asi *asi, void *addr, + enum page_table_level level) +{ + unsigned long index; + void *old_entry; + + if ((!addr) || ((unsigned long)addr) & ~PAGE_MASK) + return -EINVAL; + + lockdep_assert_held(&asi->lock); + index = asi->backend_pages_count; + + old_entry = xa_store(&asi->backend_pages, index, + ASI_BACKEND_PAGE_ENTRY(addr, level), + GFP_KERNEL); + if (xa_is_err(old_entry)) + return xa_err(old_entry); + if (old_entry) + return -EBUSY; + + asi->backend_pages_count++; + + return 0; +} + +void asi_init_backend(struct asi *asi) +{ + xa_init(&asi->backend_pages); +} + +void asi_fini_backend(struct asi *asi) +{ + unsigned long index; + void *entry; + + if (asi->backend_pages_count) { + xa_for_each(&asi->backend_pages, index, entry) + free_page((unsigned long)ASI_BACKEND_PAGE_ADDR(entry)); + } +} + +/* + * Check if an offset in the address space isolation page-table is valid, + * i.e. check that the offset is on a page effectively belonging to the + * address space isolation page-table. + */ +static bool asi_valid_offset(struct asi *asi, void *offset) +{ + unsigned long index; + void *addr, *entry; + bool valid; + + addr = ASI_BACKEND_PAGE_ALIGN(offset); + valid = false; + + lockdep_assert_held(&asi->lock); + xa_for_each(&asi->backend_pages, index, entry) { + if (ASI_BACKEND_PAGE_ADDR(entry) == addr) { + valid = true; + break; + } + } + + return valid; +} From patchwork Thu Jul 11 14:25:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 570441395 for ; Thu, 11 Jul 2019 14:26:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4640828A5A for ; Thu, 11 Jul 2019 14:26:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39F3928AD9; Thu, 11 Jul 2019 14:26:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B990928AD1 for ; Thu, 11 Jul 2019 14:26:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA2948E00C9; Thu, 11 Jul 2019 10:26:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2DE78E00C4; Thu, 11 Jul 2019 10:26:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE13E8E00C9; Thu, 11 Jul 2019 10:26:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 834648E00C4 for ; Thu, 11 Jul 2019 10:26:23 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id 132so6987870iou.0 for ; Thu, 11 Jul 2019 07:26:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=gEB6qC/xqft/fsE7GBzDnr3O7NBTUOw27/wZk4edBIk=; b=HrNYCakVj7g0RDuIUi+wzPjxmMcHSabh/CaZMWd58g3oZu08p90feX9MIRgGRrosK7 OhYnshcu3eq0ri4BU12Xp99nwGw7KGQbt/felSw8NRqDZes3lYMqkadrMA8CvUekJLeb HrmrgJoNtTsmUwzElquKDHsoF/8Tu/jcVPnSKcOf/dev/iS12MHtO+5kYkoiRiQNXPvN xtnzMR10e5t8mSV+DhHSGosWaeeKLye4RjJOrfY9vPLiFsoJXRMHVFATrI1iN+v+kpNv 9ttcV30+9CCNNpaI+75PWkEBux9QNSkyPsoZ+/WoUCZEtcbRhaXqxI1tU/t6r/3KoBM7 00ig== X-Gm-Message-State: APjAAAVUbTvul4RRs9ffsCoknAIYzZ3hHItJF/Fvhnr6QKgVzoJRJAPR XyctV6lBVEMWPvOVrQLnMcZ3vrD82Xid3XLLklr1v7Au7PYxtMnHV71XL2H3JphZwz4hajc4aXr 3qlGoXy3LJOqBcXinO5qX1DXaHFrlnf+RFGMNJ4q9S5y34MvPFPdAFogBmq2gnQ042Q== X-Received: by 2002:a6b:f711:: with SMTP id k17mr4406767iog.273.1562855183340; Thu, 11 Jul 2019 07:26:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqyiA0Yi0OaCvJ2hVCCjQwMZ2CZqR01eBztE3bARl3GrY6geL3JC6rD7oDwsE7ByopI0lTpP X-Received: by 2002:a6b:f711:: with SMTP id k17mr4406717iog.273.1562855182755; Thu, 11 Jul 2019 07:26:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855182; cv=none; d=google.com; s=arc-20160816; b=POOUVbjmYQeC+HMaRkGjXFkJHsPCNTg+UqDmZD1E/uuqLqTrfp9c9xBVWyU/Hi5kMF +4cb88b7gah1cxgXQT9H4dTLnNFZjY48RJ1JwQEx25gySHSAu4E6Hm8N8iOkF+pce0jf +pGpAS/Ccwi5rPxRKzOTbCk0QY0zkoHFSmjZlbPfdgBgUhDING0hCPG0ehwk0Zo3cS6C AxenICIwtb+w2hLiDYUBslvIFXWzEQJ3u6wvhe1VHMeDbv9yKp6rS8kEKpXnyFy9U3Uj A/3nmBSOTMSGyCZNuIMYFKtKJKkbod8/O2JPrXVVWRALWjPvWKxng6nYdSQpGeVWhwhJ PZhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gEB6qC/xqft/fsE7GBzDnr3O7NBTUOw27/wZk4edBIk=; b=MMac11r2KPGpI5Z+ttau4XFDA4tg8RnqAAkVGlI00hJK/IC0/p1AEg6HY8Q1TZ4C4d UI3dOBYL+iOMNtAqMLpX5FGxB2OZcYgMA7ZL29RQBNGR2ofjRbgSOQopqoKk94XaUR18 f5QHt5/f6/zCJpQHbmjM9tgnlnPFJZkA6iyLDIwAbDYYSscyNTOBI2djWaz+7w+qjHju JQetyk1U2CsUnoV/BYwpmebDFExN6tvPlaAup1pChzw9WMDmIry9H2dWlUlHxWaJLaiw gPpB5M4KMQXMzmJF2PAbSppgfq/5MUDadVdwuSpAM0osbC0MjUshrX8WN/bHpXVXcukk TgMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=2lbqLqqQ; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id o4si8681990jao.68.2019.07.11.07.26.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:22 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=2lbqLqqQ; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOAv6001518; Thu, 11 Jul 2019 14:26:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=gEB6qC/xqft/fsE7GBzDnr3O7NBTUOw27/wZk4edBIk=; b=2lbqLqqQMw9uxcd2J0e8Od2Fbg4RGmN4xuVesYNv7Y1WX2picK7zuP05DlYEdlTfaf8/ /r7nj8HLahdOMoTA2qZgo+/gfuMNT7Tq+4PK/S1hx7st0ER2GrN8SVMswbugTm+RisIK EY8L1gj7I42ax3Fxdef1h3ib61twTnbM35uhbRL7AAUoZFNDkEIoKv7I4NP1KhgZcU/R jZEVCuE/tyUKHROCoRyc1hZZuUr29PCD1HGeFsTbcm2+VPtYBpOACfriqJHm7UaHZUcG iSP2iP8CAbWrB5YbqwtuxyxbjJXj06/ciMakhxZ1tr5dVf0YbXkVnIwILeJB0axX/ioc dQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dwu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:08 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPctw021444; Thu, 11 Jul 2019 14:25:59 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 05/26] mm/asi: Add ASI page-table entry offset functions Date: Thu, 11 Jul 2019 16:25:17 +0200 Message-Id: <1562855138-19507-6-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=1 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add wrappers around the p4d/pud/pmd/pte offset kernel functions which ensure that page-table pointers are in the specified ASI page-table. Signed-off-by: Alexandre Chartre --- arch/x86/mm/asi_pagetable.c | 62 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 62 insertions(+), 0 deletions(-) diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index 7a8f791..a89e02e 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -97,3 +97,65 @@ static bool asi_valid_offset(struct asi *asi, void *offset) return valid; } + +/* + * asi_pXX_offset() functions are equivalent to kernel pXX_offset() + * functions but, in addition, they ensure that page table pointers + * are in the kernel isolation page table. Otherwise an error is + * returned. + */ + +static pte_t *asi_pte_offset(struct asi *asi, pmd_t *pmd, unsigned long addr) +{ + pte_t *pte; + + pte = pte_offset_map(pmd, addr); + if (!asi_valid_offset(asi, pte)) { + pr_err("ASI %p: PTE %px not found\n", asi, pte); + return ERR_PTR(-EINVAL); + } + + return pte; +} + +static pmd_t *asi_pmd_offset(struct asi *asi, pud_t *pud, unsigned long addr) +{ + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + if (!asi_valid_offset(asi, pmd)) { + pr_err("ASI %p: PMD %px not found\n", asi, pmd); + return ERR_PTR(-EINVAL); + } + + return pmd; +} + +static pud_t *asi_pud_offset(struct asi *asi, p4d_t *p4d, unsigned long addr) +{ + pud_t *pud; + + pud = pud_offset(p4d, addr); + if (!asi_valid_offset(asi, pud)) { + pr_err("ASI %p: PUD %px not found\n", asi, pud); + return ERR_PTR(-EINVAL); + } + + return pud; +} + +static p4d_t *asi_p4d_offset(struct asi *asi, pgd_t *pgd, unsigned long addr) +{ + p4d_t *p4d; + + p4d = p4d_offset(pgd, addr); + /* + * p4d is the same has pgd if we don't have a 5-level page table. + */ + if ((p4d != (p4d_t *)pgd) && !asi_valid_offset(asi, p4d)) { + pr_err("ASI %p: P4D %px not found\n", asi, p4d); + return ERR_PTR(-EINVAL); + } + + return p4d; +} From patchwork Thu Jul 11 14:25:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040079 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32679112C for ; Thu, 11 Jul 2019 14:26:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20F5C28AC8 for ; Thu, 11 Jul 2019 14:26:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1510428AD9; Thu, 11 Jul 2019 14:26:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E98028AC8 for ; Thu, 11 Jul 2019 14:26:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CEB38E00CA; Thu, 11 Jul 2019 10:26:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1E2058E00C4; Thu, 11 Jul 2019 10:26:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDBBA8E00CA; Thu, 11 Jul 2019 10:26:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id B225D8E00C4 for ; Thu, 11 Jul 2019 10:26:25 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id e20so6913645ioe.12 for ; Thu, 11 Jul 2019 07:26:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=lCtyGi9lNXZNoOqW5lwUfawQTHm8Gs8rgqwuOoYVSXg=; b=ZhuRrSFlanx/heka0lEcveK+B3L5m+c3gQVsScc27RX/gScwbqwZXS2WvFbWz5en45 6xbMFWRG4lyVkFOdXg0PMOxG52RAEqRrKR5bW8pSIju2Jwbw3xwRmD2udLhuo/v2s3th FsGoeApCBk10UR+FA6RY1CjBmx8hZT6PdiwaQP0ce6hCf8yH9QPiEIAlMXr76VQA/bwj EWzi5f6rKebQEA0KP4wbXucJtcOJKeoP7IZlKHg24GUUJn0vl8wItJ8BnSW7y5q1gBH6 18BlWSX4wh1DIXzN4cLfh91GpbQuBbLBFl65hSr+2vYjjsQSMsP//MbWSuA02VIYMOWb N2ZA== X-Gm-Message-State: APjAAAX/bDzOi2mcMD/zCLbmKEl83AkZoH/h63nLzoF0yn16kSQISkkH 7oSaKE4dUuyPOrkhclGGP0fDsQ+pkljGgu9OQ1pPZKoqNB30Z7gj8CTBTHKbOwByvK4LTJeyQSJ OW2lMH4tlckecXiwHLhVQAl2Jc3PLpTRVywTvtfMTHyD5iD11X5mrT3EDrszOA+AuXA== X-Received: by 2002:a5d:9618:: with SMTP id w24mr4565730iol.279.1562855185484; Thu, 11 Jul 2019 07:26:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqyyrzmkEJJRjnLreD8mMgEz+wk/BP34MmMCpog6yLC1ygb8MV8E6qEBUktSVhyCDrWLzQuf X-Received: by 2002:a5d:9618:: with SMTP id w24mr4565676iol.279.1562855184865; Thu, 11 Jul 2019 07:26:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855184; cv=none; d=google.com; s=arc-20160816; b=u198FC2sNGT/6W/KHmf4UT1U/vqj4nqHJMvl8Ou5/CWennR66ex8PLjMgGlbhPJcuO Z29qAZVt0fyzYRe1ZT4/FGWRuQlXgFtIOex08qSmozot4A7Zyn7uYiVKb+WYqWfMvLEs NgED/YNNG4CkrTaKvExpd1mhF0zGlcD6MkfhxXPOOJyjgSv3LWe4AxFvC9nJyn8vkjLI ght6gcsfdw+99cA5yc061t2CQr/ttDjNAB55STsetVfHvwvmAIZMENo3gm7Vh/bxzfjk M/L1akGwJvndcyDFqA6fY3tzVHLjC+dfTxlXr7MPxjNR1IngE66vGu8ll5AmCAwgQCQ0 cDyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lCtyGi9lNXZNoOqW5lwUfawQTHm8Gs8rgqwuOoYVSXg=; b=jI+zO5RVEvtiZJNtWjMQLs8gRNthu754PsPgfLmY7XjhpMH8mTr6Ex1CybejkOS4oS sCmK/fo98gwXsxjuGtVqttl8oY1p50RGNMxUMGl7qFCeh6SinOHvyS5AwBoI8RgDZVpu pLeZitAO/ngEjw3AOIIhKuaZKPJIfP2ExAYNxQf5LR95yIjkAXQ57xUrLsLNusWcLt0V UvqW69+8ZxNA9r3UBj1qk4nDqnRkpnwocQ578mjND7ZEiWYyqLZgitfKFEefFsQLg8OI l3tOh6t1bwPZtex0lfB/9Gkt6H5sIjj3tbVv96kLV0O5FsBgOlsv78gG5l32JN/FYrUi 1yfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=lJAKdulh; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id h6si9010051jaa.71.2019.07.11.07.26.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:24 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=lJAKdulh; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO7Xi100417; Thu, 11 Jul 2019 14:26:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=lCtyGi9lNXZNoOqW5lwUfawQTHm8Gs8rgqwuOoYVSXg=; b=lJAKdulhKeHbnYjRsyD5nMQD+gRQsuPOeXQdtSSCbufwUKXgsEfUN2Rz5GYHTyi0V/o7 /7wfZKsgL4wiSMWTnhXm2xWq6UxcJMo9Wcw35HV4xUxxM34clQuAg7xvJmhbR4yOZmCW HtxhTH15oHUeFU7K6yML/HzF46ooAQ0oDiETifZF8/K0Eb+1V8sYTyAXh2PY56Cglqe0 4to8Zt1JMHQxjc6IQGJB++dychz0/PvHCxVVbx1+c6+qGz1P2iNzE3+TWmWcb+jtnmaE 8hDeydy1NGeaojoYQskyTsqpWzZDVRocAyhp4SKVywbNL71DbkjE7rT5ZK2hAsuvr/wq nA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0c74-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:11 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPctx021444; Thu, 11 Jul 2019 14:26:02 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 06/26] mm/asi: Add ASI page-table entry allocation functions Date: Thu, 11 Jul 2019 16:25:18 +0200 Message-Id: <1562855138-19507-7-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add functions to allocate p4d/pud/pmd/pte pages for an ASI page-table and keep track of them. Signed-off-by: Alexandre Chartre --- arch/x86/mm/asi_pagetable.c | 111 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 111 insertions(+), 0 deletions(-) diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index a89e02e..0fc6d59 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -4,6 +4,8 @@ * */ +#include + #include /* @@ -159,3 +161,112 @@ static bool asi_valid_offset(struct asi *asi, void *offset) return p4d; } + +/* + * asi_pXX_alloc() functions are equivalent to kernel pXX_alloc() functions + * but, in addition, they keep track of new pages allocated for the specified + * ASI. + */ + +static pte_t *asi_pte_alloc(struct asi *asi, pmd_t *pmd, unsigned long addr) +{ + struct page *page; + pte_t *pte; + int err; + + if (pmd_none(*pmd)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pte = (pte_t *)page_address(page); + err = asi_add_backend_page(asi, pte, PGT_LEVEL_PTE); + if (err) { + free_page((unsigned long)pte); + return ERR_PTR(err); + } + set_pmd_safe(pmd, __pmd(__pa(pte) | _KERNPG_TABLE)); + pte = pte_offset_map(pmd, addr); + } else { + pte = asi_pte_offset(asi, pmd, addr); + } + + return pte; +} + +static pmd_t *asi_pmd_alloc(struct asi *asi, pud_t *pud, unsigned long addr) +{ + struct page *page; + pmd_t *pmd; + int err; + + if (pud_none(*pud)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pmd = (pmd_t *)page_address(page); + err = asi_add_backend_page(asi, pmd, PGT_LEVEL_PMD); + if (err) { + free_page((unsigned long)pmd); + return ERR_PTR(err); + } + set_pud_safe(pud, __pud(__pa(pmd) | _KERNPG_TABLE)); + pmd = pmd_offset(pud, addr); + } else { + pmd = asi_pmd_offset(asi, pud, addr); + } + + return pmd; +} + +static pud_t *asi_pud_alloc(struct asi *asi, p4d_t *p4d, unsigned long addr) +{ + struct page *page; + pud_t *pud; + int err; + + if (p4d_none(*p4d)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pud = (pud_t *)page_address(page); + err = asi_add_backend_page(asi, pud, PGT_LEVEL_PUD); + if (err) { + free_page((unsigned long)pud); + return ERR_PTR(err); + } + set_p4d_safe(p4d, __p4d(__pa(pud) | _KERNPG_TABLE)); + pud = pud_offset(p4d, addr); + } else { + pud = asi_pud_offset(asi, p4d, addr); + } + + return pud; +} + +static p4d_t *asi_p4d_alloc(struct asi *asi, pgd_t *pgd, unsigned long addr) +{ + struct page *page; + p4d_t *p4d; + int err; + + if (!pgtable_l5_enabled()) + return (p4d_t *)pgd; + + if (pgd_none(*pgd)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + p4d = (p4d_t *)page_address(page); + err = asi_add_backend_page(asi, p4d, PGT_LEVEL_P4D); + if (err) { + free_page((unsigned long)p4d); + return ERR_PTR(err); + } + set_pgd_safe(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE)); + p4d = p4d_offset(pgd, addr); + } else { + p4d = asi_p4d_offset(asi, pgd, addr); + } + + return p4d; +} From patchwork Thu Jul 11 14:25:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040075 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1E401395 for ; Thu, 11 Jul 2019 14:26:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B105E28AC8 for ; Thu, 11 Jul 2019 14:26:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A518928ADC; Thu, 11 Jul 2019 14:26:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14D2128AD1 for ; Thu, 11 Jul 2019 14:26:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F8BB8E00C8; Thu, 11 Jul 2019 10:26:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 081A38E00C4; Thu, 11 Jul 2019 10:26:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E65F98E00C8; Thu, 11 Jul 2019 10:26:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id C38558E00C4 for ; Thu, 11 Jul 2019 10:26:22 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id c5so6904127iom.18 for ; Thu, 11 Jul 2019 07:26:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=385wpu4NDusZzlrvO4wdMoGEKubdnUYw4N6zqr4muDU=; b=cG6fKZ3VeUElWj3A9w1ew9h2UjSs1fvJb/bsxXJohFkjRsSTMPMFihUSxjaiXuxAOi s2IX4lN11e5dIRGxJfD/wbROzqC6zAmgJB33PJFZ2AjZmcP993U2suagiVhBB/XQMrPt B9KpPlp+6cHMxwCSRc2aTvpK/lJSuPbmm3Y/GjbOFvE4MFJVC5kawKm7cRmDIt+PWeeS wYq1dnUek3ROZnyPatXy7PKyClzjsIl7QHjZH8oFFWlFs9nT+HYGRYfpsX3GrgCEFb13 pzbLFgyhmxYs21IyTzuC070oMR5fkvmh4QulE0wRrrWB639sWE9R2cNDc7AdvCNTyNQi UG3Q== X-Gm-Message-State: APjAAAV0xd74y8Grw1eHW6R3DPUr0MJJWVx4Z0xZF/0MddOOQX4lv2IE 0MKCYQYTWaZdi6L8Z7E567JZvSOw/EZy4FAnVLfdr/tzw3qS5KnuC84etdeJEnn5rUh3NS9YvdG FbOd2JnnIsI9qR0y0cYgkXhNm6VFMdmj4TmDr/1YG6IW2ePaWHu8xjsG7/DbhKA17YQ== X-Received: by 2002:a5d:9643:: with SMTP id d3mr4775857ios.227.1562855182587; Thu, 11 Jul 2019 07:26:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqwDexv5X+iHb0htnuAlGjnV0d+rO9gQ5F7JJfMSHftJaVfAK0TACi8VzCKIOmmIyEZxmhav X-Received: by 2002:a5d:9643:: with SMTP id d3mr4775793ios.227.1562855181934; Thu, 11 Jul 2019 07:26:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855181; cv=none; d=google.com; s=arc-20160816; b=PWSNgQObdccPDXnzVrLh+8OxP+95T/4f3IungOaXgkWKpEP4LxGbg8FAsxwnczkNuQ caezh+s/1DkrTvS6twUWXTmxmT1buTduwWb0RP3+V3zHaortcFT7uUvIBaceCe23ni2p R3Hun4T98IwRi0dq5zUnWm1iNUCBMBSxpIAey/AeMx/uLxNdXU+NZ9+68mQuAso4LJWb AMwxI+lhPQfKY7DCd6WluczwCFAQeNj5kiUSpAKs4M7AZyC3j412f/mgRjRvsDy7OJsk 3cS9gkNoQltFBucNG2U4MZnrqo0wN0p8/0ymf+kFBhhMsftqK3F/LRNmyonPO+zSlLDV M+fA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=385wpu4NDusZzlrvO4wdMoGEKubdnUYw4N6zqr4muDU=; b=AuYLT6A0ztFSCFeBYnMC8wiSnEs/O5l+ESsWx7bsVjYWYZDsDwTGsoQ3t6dtm0hox6 Z84keEWmpphCFYEsWxnt0uNETljCmILRSSCnqEs5ek80KGWvK02I0BrWSOuslMLDjtdU JcsC4mZ9D4ibQ/b2oYDYd1oVLDZ6D3Rh6kvVUUypwvvwEf/eoAYW9FpNEGL8vor5461t renr9amnRVP2su6vrfKZH8Do//tMi+LnVUQ5/iLG3UchcSaNyJMJuWl5UVpCEL6AoU7d HtsqYId7bFxU3HEcn0f8NoB6bLoZv0dF0nd0T3sO7oHFDytMATkjcKPst7LOuiEP3dYk v1KA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="j/8IHCr9"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id g12si9223876jao.11.2019.07.11.07.26.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:21 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="j/8IHCr9"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO7tr013226; Thu, 11 Jul 2019 14:26:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=385wpu4NDusZzlrvO4wdMoGEKubdnUYw4N6zqr4muDU=; b=j/8IHCr9S0gUaWnFi3Upjzuz55gy9Ljh6nfaWePnOoytFJ9B7SnJKULz74C3ywY6RXqi e2nlqVa9W7MNYSdP2Vf2C/oYboYtNcwS8EW9VbU8cIsaN6Fl9HN5Irx/xBWUPbwuO3Rb pA8nC/jt3qZSlX5P9mWE6i1BavyZZ35GVGnmvivmjOvqS8a8/Z0v3tfm2TkPowKHagRD jG7N8dpzCgLK0VRzFa0NpGxGayMyo+vj0XV5/GWaoX6BlSffhi7TdWAzW0NBMvhvqMgW Mi7BkTLlSEZlEqWrGTNR/OK6j6uUWeCyQ/O4mFUX/8vA2YAylUPyhsIeKxT9HDn0UNu4 +A== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2tjm9r0bn2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:09 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu0021444; Thu, 11 Jul 2019 14:26:06 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 07/26] mm/asi: Add ASI page-table entry set functions Date: Thu, 11 Jul 2019 16:25:19 +0200 Message-Id: <1562855138-19507-8-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add wrappers around the page table entry (pgd/p4d/pud/pmd) set functions which check that an existing entry is not being overwritten. Signed-off-by: Alexandre Chartre --- arch/x86/mm/asi_pagetable.c | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 124 insertions(+), 0 deletions(-) diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index 0fc6d59..e17af9e 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -270,3 +270,127 @@ static bool asi_valid_offset(struct asi *asi, void *offset) return p4d; } + +/* + * asi_set_pXX() functions are equivalent to kernel set_pXX() functions + * but, in addition, they ensure that they are not overwriting an already + * existing reference in the page table. Otherwise an error is returned. + */ +static int asi_set_pte(struct asi *asi, pte_t *pte, pte_t pte_value) +{ +#ifdef DEBUG + /* + * The pte pointer should come from asi_pte_alloc() or asi_pte_offset() + * both of which check if the pointer is in the kernel isolation page + * table. So this is a paranoid check to ensure the pointer is really + * in the kernel page table. + */ + if (!asi_valid_offset(asi, pte)) { + pr_err("ASI %p: PTE %px not found\n", asi, pte); + return -EINVAL; + } +#endif + set_pte(pte, pte_value); + + return 0; +} + +static int asi_set_pmd(struct asi *asi, pmd_t *pmd, pmd_t pmd_value) +{ +#ifdef DEBUG + /* + * The pmd pointer should come from asi_pmd_alloc() or asi_pmd_offset() + * both of which check if the pointer is in the kernel isolation page + * table. So this is a paranoid check to ensure the pointer is really + * in the kernel page table. + */ + if (!asi_valid_offset(asi, pmd)) { + pr_err("ASI %p: PMD %px not found\n", asi, pmd); + return -EINVAL; + } +#endif + if (pmd_val(*pmd) == pmd_val(pmd_value)) + return 0; + + if (!pmd_none(*pmd)) { + pr_err("ASI %p: PMD %px overwriting %lx with %lx\n", + asi, pmd, pmd_val(*pmd), pmd_val(pmd_value)); + return -EBUSY; + } + + set_pmd(pmd, pmd_value); + + return 0; +} + +static int asi_set_pud(struct asi *asi, pud_t *pud, pud_t pud_value) +{ +#ifdef DEBUG + /* + * The pud pointer should come from asi_pud_alloc() or asi_pud_offset() + * both of which check if the pointer is in the kernel isolation page + * table. So this is a paranoid check to ensure the pointer is really + * in the kernel page table. + */ + if (!asi_valid_offset(asi, pud)) { + pr_err("ASI %p: PUD %px not found\n", asi, pud); + return -EINVAL; + } +#endif + if (pud_val(*pud) == pud_val(pud_value)) + return 0; + + if (!pud_none(*pud)) { + pr_err("ASI %p: PUD %px overwriting %lx with %lx\n", + asi, pud, pud_val(*pud), pud_val(pud_value)); + return -EBUSY; + } + + set_pud(pud, pud_value); + + return 0; +} + +static int asi_set_p4d(struct asi *asi, p4d_t *p4d, p4d_t p4d_value) +{ +#ifdef DEBUG + /* + * The p4d pointer should come from asi_p4d_alloc() or asi_p4d_offset() + * both of which check if the pointer is in the kernel isolation page + * table. So this is a paranoid check to ensure the pointer is really + * in the kernel page table. + */ + if (!asi_valid_offset(asi, p4d)) { + pr_err("ASI %p: P4D %px not found\n", asi, p4d); + return -EINVAL; + } +#endif + if (p4d_val(*p4d) == p4d_val(p4d_value)) + return 0; + + if (!p4d_none(*p4d)) { + pr_err("ASI %p: P4D %px overwriting %lx with %lx\n", + asi, p4d, p4d_val(*p4d), p4d_val(p4d_value)); + return -EBUSY; + } + + set_p4d(p4d, p4d_value); + + return 0; +} + +static int asi_set_pgd(struct asi *asi, pgd_t *pgd, pgd_t pgd_value) +{ + if (pgd_val(*pgd) == pgd_val(pgd_value)) + return 0; + + if (!pgd_none(*pgd)) { + pr_err("ASI %p: PGD %px overwriting %lx with %lx\n", + asi, pgd, pgd_val(*pgd), pgd_val(pgd_value)); + return -EBUSY; + } + + set_pgd(pgd, pgd_value); + + return 0; +} From patchwork Thu Jul 11 14:25:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040081 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42A491395 for ; Thu, 11 Jul 2019 14:26:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3214D28A5A for ; Thu, 11 Jul 2019 14:26:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 253B328AD1; Thu, 11 Jul 2019 14:26:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 635B428AC8 for ; Thu, 11 Jul 2019 14:26:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47C018E00CB; Thu, 11 Jul 2019 10:26:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3684D8E00C4; Thu, 11 Jul 2019 10:26:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20A358E00CB; Thu, 11 Jul 2019 10:26:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by kanga.kvack.org (Postfix) with ESMTP id E20F58E00C4 for ; Thu, 11 Jul 2019 10:26:26 -0400 (EDT) Received: by mail-io1-f71.google.com with SMTP id s9so6956145iob.11 for ; Thu, 11 Jul 2019 07:26:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=y9IAz60J7qXRlFN2NRsHUzd+qvZ9ZunbWoLTMdcwAx8=; b=IxjZRzbuP+atDgfo9Ib/4AaGyDPHVGi79gIIzCLEfmxlI3MXfG/q5kau4K33gNWG6G jXxrXdAjT51dtUWe3/B4H5V5O42txlMZacIlwgYIbkhrNe4zwAIvHx8O+N+BedQDW4nt Os+Qhp24s9Yks2GnzZ7n06DkvHGlRZ0BBKojZ73k7N1zK/ceC6d9nalXfNhTBrbfpkyw LoyK6PuRHJTHdQPoxN226T7mmftWMSrtEm6GGLg44ze57ofCAoUb9ebTajitIhDKEVJ5 n0eUuZN0UbJuMTE9XdDwTz6WlAZofutxqmD+EtUjF90i5RysMJZhXrDZCiWv7udqrxfW IGgg== X-Gm-Message-State: APjAAAWUCnwFlEVWL99mulRn7bORgvsrNZNCpLQVy6e1U6EO5niZ3Tpl mv2iKsJ69nzcqrNXAuXs0OVAzkQhgOudUYoOaIrqrqjK1sOq7xmFjyfduLsc5qqo4p58IS15BnA 1/eR4MYdM41v5X+l952lz/AaNTkFT68RBMIbAjfGxztTOnQpzufxaWKhgM97/nNF7Lw== X-Received: by 2002:a05:6638:149:: with SMTP id y9mr4957121jao.76.1562855186684; Thu, 11 Jul 2019 07:26:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqxuffeQrVjODQEU+acvsnVEem1BLaVQ+GaB+LBCP+bqLxVcBfGDv4D4QQoAuui8di+yALCk X-Received: by 2002:a05:6638:149:: with SMTP id y9mr4957025jao.76.1562855185763; Thu, 11 Jul 2019 07:26:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855185; cv=none; d=google.com; s=arc-20160816; b=uZtLipxMM3leaCavWHFN1KwCUJZ2caBOgJhMSPAaD0V42npqEZe0w9GPSYk/6Yu4Y/ IM9G+P7swW3hq9rzXWBUtVj4SI7AwvnBE6tvAAN3B5hqVOe+jdQJt77p2D0TpR4glssA ENPtv2fdfAPlmByuyIbMWuS8Em1rgWu6ZA+TynKRqWcRyGfUsDxO7Cd15xNu4BRcWul+ auyvDOLo2NcaoqSlSg/Pz1cqZZiKhdnxecql4xC9GK+NSONnbz7lFKZUsGuRwnoqWbmn B1eRRG38QdMfiOcFV8B06+h9PUBmnbW/URWm5aDbMinPN8fgQsEqTtFpWZiL69/Pvo+K 65YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y9IAz60J7qXRlFN2NRsHUzd+qvZ9ZunbWoLTMdcwAx8=; b=j3UYHq+p7Fn3mNOPHTCZDW+ENNR0QtcyoB08jcM5pbmyxTD81y9n0efUjnM1bFji2T OU1ewcikWuiY6s7r5iUI6yyhpH66AXEG7W4U4zXGxRXicP2vIPZmUPN+wWmzNp9ZJof3 BoTWb8CqIMJulpPjKudmWpnOrI1Euy4hVK9bvdYMDgm5KUBU2wvibRIDR9ryoJbLBiXY rFTvmY7osgCS+bmiiTfBUoSuCnM896Pw8e0+dzEvQ42LvuSAWJSIt61bSBX0q41KcHgw qijCBlowL+eymaoeEv/x+gJ72HaqVa53tslJwfRK9dD/0do11CZZvrsX8bye41sJTtSF HhUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SmfOq+K7; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id a17si9049208jap.14.2019.07.11.07.26.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:25 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=SmfOq+K7; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOMPI001606; Thu, 11 Jul 2019 14:26:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=y9IAz60J7qXRlFN2NRsHUzd+qvZ9ZunbWoLTMdcwAx8=; b=SmfOq+K7kq1i3rKVZwlHKx3Hxs01appLJg4uPnU/ZriVoOVHCERIDHALooM0c4fIGSBJ 6sJUwYFar/i1eRJ0bDwA5eXmBBC3pkJst2pWeRiO5XvbO/TYtBqcsljdxwFDY+ygijHk H0MYZu44ra2JBIbub0jXMvwYLNtqfrcVe7JBvlbVaiEu0d5hM3uanYLNWIvfENEj46Wm gz3Qt8PeZrmim/eo4JN+pfX8fKIq4/XNikWq8u1EuUds0UNZgKC6NbZKX2Q0LuHEumqL b5/SpV0/Ww9/ELn0rWmcvfcCc+XgXWqEmOxEsU2TT2yDZ4Zr37YzZrqonMAsPGVng5cD rQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dx7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:12 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu1021444; Thu, 11 Jul 2019 14:26:09 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 08/26] mm/asi: Functions to populate an ASI page-table from a VA range Date: Thu, 11 Jul 2019 16:25:20 +0200 Message-Id: <1562855138-19507-9-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=917 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Provide functions to copy page-table entries from the kernel page-table to an ASI page-table for a specified VA range. These functions are based on the copy_pxx_range() functions defined in mm/memory.c. A difference is that a level parameter can be specified to indicate the page-table level (PGD, P4D, PUD PMD, PTE) at which the copy should be done. Also functions don't rely on mm or vma, and they don't alter the source page-table even if an entry is bad. Also the VA range start and size don't need to be page-aligned. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 4 + arch/x86/mm/asi_pagetable.c | 205 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 209 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 3d965e6..19656aa 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -76,6 +76,10 @@ struct asi_session { extern bool asi_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address); +extern int asi_map_range(struct asi *asi, void *ptr, size_t size, + enum page_table_level level); +extern int asi_map(struct asi *asi, void *ptr, unsigned long size); + /* * Function to exit the current isolation. This is used to abort isolation * when a task using isolation is scheduled out. diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index e17af9e..0169395 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -394,3 +394,208 @@ static int asi_set_pgd(struct asi *asi, pgd_t *pgd, pgd_t pgd_value) return 0; } + +static int asi_copy_pte_range(struct asi *asi, pmd_t *dst_pmd, pmd_t *src_pmd, + unsigned long addr, unsigned long end) +{ + pte_t *src_pte, *dst_pte; + + dst_pte = asi_pte_alloc(asi, dst_pmd, addr); + if (IS_ERR(dst_pte)) + return PTR_ERR(dst_pte); + + addr &= PAGE_MASK; + src_pte = pte_offset_map(src_pmd, addr); + + do { + asi_set_pte(asi, dst_pte, *src_pte); + + } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr < end); + + return 0; +} + +static int asi_copy_pmd_range(struct asi *asi, pud_t *dst_pud, pud_t *src_pud, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pmd_t *src_pmd, *dst_pmd; + unsigned long next; + int err; + + dst_pmd = asi_pmd_alloc(asi, dst_pud, addr); + if (IS_ERR(dst_pmd)) + return PTR_ERR(dst_pmd); + + src_pmd = pmd_offset(src_pud, addr); + + do { + next = pmd_addr_end(addr, end); + if (level == PGT_LEVEL_PMD || pmd_none(*src_pmd) || + pmd_trans_huge(*src_pmd) || pmd_devmap(*src_pmd)) { + err = asi_set_pmd(asi, dst_pmd, *src_pmd); + if (err) + return err; + continue; + } + + if (!pmd_present(*src_pmd)) { + pr_warn("ASI %p: PMD not present for [%lx,%lx]\n", + asi, addr, next - 1); + pmd_clear(dst_pmd); + continue; + } + + err = asi_copy_pte_range(asi, dst_pmd, src_pmd, addr, next); + if (err) { + pr_err("ASI %p: PMD error copying PTE addr=%lx next=%lx\n", + asi, addr, next); + return err; + } + + } while (dst_pmd++, src_pmd++, addr = next, addr < end); + + return 0; +} + +static int asi_copy_pud_range(struct asi *asi, p4d_t *dst_p4d, p4d_t *src_p4d, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pud_t *src_pud, *dst_pud; + unsigned long next; + int err; + + dst_pud = asi_pud_alloc(asi, dst_p4d, addr); + if (IS_ERR(dst_pud)) + return PTR_ERR(dst_pud); + + src_pud = pud_offset(src_p4d, addr); + + do { + next = pud_addr_end(addr, end); + if (level == PGT_LEVEL_PUD || pud_none(*src_pud) || + pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) { + err = asi_set_pud(asi, dst_pud, *src_pud); + if (err) + return err; + continue; + } + + err = asi_copy_pmd_range(asi, dst_pud, src_pud, addr, next, + level); + if (err) { + pr_err("ASI %p: PUD error copying PMD addr=%lx next=%lx\n", + asi, addr, next); + return err; + } + + } while (dst_pud++, src_pud++, addr = next, addr < end); + + return 0; +} + +static int asi_copy_p4d_range(struct asi *asi, pgd_t *dst_pgd, pgd_t *src_pgd, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + p4d_t *src_p4d, *dst_p4d; + unsigned long next; + int err; + + dst_p4d = asi_p4d_alloc(asi, dst_pgd, addr); + if (IS_ERR(dst_p4d)) + return PTR_ERR(dst_p4d); + + src_p4d = p4d_offset(src_pgd, addr); + + do { + next = p4d_addr_end(addr, end); + if (level == PGT_LEVEL_P4D || p4d_none(*src_p4d)) { + err = asi_set_p4d(asi, dst_p4d, *src_p4d); + if (err) + return err; + continue; + } + + err = asi_copy_pud_range(asi, dst_p4d, src_p4d, addr, next, + level); + if (err) { + pr_err("ASI %p: P4D error copying PUD addr=%lx next=%lx\n", + asi, addr, next); + return err; + } + + } while (dst_p4d++, src_p4d++, addr = next, addr < end); + + return 0; +} + +static int asi_copy_pgd_range(struct asi *asi, + pgd_t *dst_pagetable, pgd_t *src_pagetable, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pgd_t *src_pgd, *dst_pgd; + unsigned long next; + int err; + + dst_pgd = pgd_offset_pgd(dst_pagetable, addr); + src_pgd = pgd_offset_pgd(src_pagetable, addr); + + do { + next = pgd_addr_end(addr, end); + if (level == PGT_LEVEL_PGD || pgd_none(*src_pgd)) { + err = asi_set_pgd(asi, dst_pgd, *src_pgd); + if (err) + return err; + continue; + } + + err = asi_copy_p4d_range(asi, dst_pgd, src_pgd, addr, next, + level); + if (err) { + pr_err("ASI %p: PGD error copying P4D addr=%lx next=%lx\n", + asi, addr, next); + return err; + } + + } while (dst_pgd++, src_pgd++, addr = next, addr < end); + + return 0; +} + +/* + * Copy page table entries from the current page table (i.e. from the + * kernel page table) to the specified ASI page-table. The level + * parameter specifies the page-table level (PGD, P4D, PUD PMD, PTE) + * at which the copy should be done. + */ +int asi_map_range(struct asi *asi, void *ptr, size_t size, + enum page_table_level level) +{ + unsigned long addr = (unsigned long)ptr; + unsigned long end = addr + ((unsigned long)size); + unsigned long flags; + int err; + + pr_debug("ASI %p: MAP %px/%lx/%d\n", asi, ptr, size, level); + + spin_lock_irqsave(&asi->lock, flags); + err = asi_copy_pgd_range(asi, asi->pgd, current->mm->pgd, + addr, end, level); + spin_unlock_irqrestore(&asi->lock, flags); + + return err; +} +EXPORT_SYMBOL(asi_map_range); + +/* + * Copy page-table PTE entries from the current page-table to the + * specified ASI page-table. + */ +int asi_map(struct asi *asi, void *ptr, unsigned long size) +{ + return asi_map_range(asi, ptr, size, PGT_LEVEL_PTE); +} +EXPORT_SYMBOL(asi_map); From patchwork Thu Jul 11 14:25:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040083 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4304F1395 for ; Thu, 11 Jul 2019 14:26:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3354028AC8 for ; Thu, 11 Jul 2019 14:26:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 276AD28AD9; Thu, 11 Jul 2019 14:26:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC28428AC8 for ; Thu, 11 Jul 2019 14:26:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDBE08E00CC; Thu, 11 Jul 2019 10:26:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E648E8E00C4; Thu, 11 Jul 2019 10:26:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D071C8E00CC; Thu, 11 Jul 2019 10:26:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id A3C808E00C4 for ; Thu, 11 Jul 2019 10:26:31 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id z19so6970763ioi.15 for ; Thu, 11 Jul 2019 07:26:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=x/IjzSr0NJXbBWkc/GbWFGd8aRvvo+k/bAxRCg2smaA=; b=IYiDx+wxwvVYusXwOuWotZ3CeeOX2YXRapwvpukTN+dmuLtRMvUzlLIMRErFdAqo74 MrKsefNCdKbp+2/vJgSds3lLVlAF8cnJJv84y61QRmffMXURBFWky3mrizLuHj0r1Uvs 5cemMkyPnzadCuVoFNbh0QGmXGnnFUHb4rtonj/lNkXNIZkNXWrsUrq16We/velq5RMB 15f83JffpuSnHz7k+ol5XR6U4uGfNpACZz9OEuzI7X/kaSzfUPO5j7O1GjFx2lwu8GFc HMFrQ7yE95ss/VtkX1ujjS2UwIna/cFhH8A0csFro6XMGYb31YU3RsiNT0WCfug4mPW5 ZVow== X-Gm-Message-State: APjAAAWtc+6XCJf8GKppbp0Zon+yyNM0KzdgojVMtFgJxqzh7NDK+TBq eg00ef8o0obj5ZZ1v0N6E8DjEbZiTkjoKWgRU9n/fji8MO6aJaqFce216x0PI3zZzpN8yQhdDgD 2wQkFmBVoIN2w3qhcLj3kuCGWysGdCHSgMCiOgQm/oye+/WqKNemnS/ALhMxeItZRlQ== X-Received: by 2002:a6b:b3c1:: with SMTP id c184mr4097540iof.222.1562855191487; Thu, 11 Jul 2019 07:26:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqzhOc2nQdfjOJX1g5BJ6PQIEImKrNU2FiTcnmtB6KD3UiTJJo8kzucMCLacFsPM+9/cnEGi X-Received: by 2002:a6b:b3c1:: with SMTP id c184mr4097502iof.222.1562855190959; Thu, 11 Jul 2019 07:26:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855190; cv=none; d=google.com; s=arc-20160816; b=LEf/yqclCkEg39r2t13n0YD80oOc618bIOUokvAaLGbeKGg+pvmNy3aLmiJQMxkAlL IC+rY4Dny0m9jxpOcl56yhKeldfvxDW6jsA3dcnoxUCoAvXoKesNfJDGpLk4EGcbeP1l srhg99a1M+irnoxqVIw2xsORdDDR3UoZMC+Irnb75+Ah8zZE2rMtN7rwJwkCORhT00sX 1Q01sKWPiZEFZ1a5juKEq5p801aiB/tcqUJUq/ucbMuVBL6dpvam8ceJ1tuaol1yLvIf nOF1lO+JqMH9RmQZ6QyhGYwrKODn8ddAYUHmnIoWoQJ2GHOYq9DF8DuJEvNdvhyyem/8 fE7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=x/IjzSr0NJXbBWkc/GbWFGd8aRvvo+k/bAxRCg2smaA=; b=z6IJI2A/fS63bMt7SnivdMQZ+ZXMo4T/xfUq1Ac/YdtBgYQ1mU0GgeO2qCu9CcKycF 2vRwDr7ECCkTFWukouy2dTCKQ2FiFNq4jU/5ioa6Y3HubT+7/Vs8cuA4yNFuMHwLves1 yV6tnNdrvYYZSy+SuQ/fB1U8oRqMLWMlVvoWfTRXapHf8cj0MPnSnxLSbBtY19hXZRGz FjR/LAPTqKBa5SkGg7+knOTWTiW+Fq23H1432UP0RhCsw65BcvTGZUtv9ubKPY7f7b/e fabIvEb42HHbrvIERPNjOQCQz5yaF5y1raexZcIx1bgwzthsNF7UgbAgOWGs2icW4zXb 0jMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=UpGsaKxV; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id r62si8988739iod.101.2019.07.11.07.26.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:30 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=UpGsaKxV; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOE8u013326; Thu, 11 Jul 2019 14:26:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=x/IjzSr0NJXbBWkc/GbWFGd8aRvvo+k/bAxRCg2smaA=; b=UpGsaKxVuo2eIaT3MIyMj6uwvZYdjod6xr9sIWH6GLVIiGJo5xv5khjwyLVyDDuy1GvG yfeeGSjBWTuaMV/ezPA0NULlZQYuPGQftx9frpdHWZCWGt6EzWWMYPjn5VPHT47SjUpI A1Zvxp/4R+waiK8CFltSblp1Oe52CtoJA0LD+OBZSfdqHFgWhRLUnvbf0i9N2AFU9yvt 42HuhPrb/rYgNn3SpXtQLfgcHyAjGcS15pNxoiZQoNGdL0uAmA7B00OD3j6dk0aaUwQn KjZFUxYUuNzdESj/m6+MVakQNxN1UqdtuIgCsY0XLrDA5xHgNra+3zY3ilOMR4T3aYcQ vg== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2tjm9r0bpk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:21 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu2021444; Thu, 11 Jul 2019 14:26:12 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 09/26] mm/asi: Helper functions to map module into ASI Date: Thu, 11 Jul 2019 16:25:21 +0200 Message-Id: <1562855138-19507-10-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=896 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add helper functions to easily map a module into an ASI. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 21 +++++++++++++++++++++ 1 files changed, 21 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 19656aa..b5dbc49 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -6,6 +6,7 @@ #ifndef __ASSEMBLY__ +#include #include #include #include @@ -81,6 +82,26 @@ extern int asi_map_range(struct asi *asi, void *ptr, size_t size, extern int asi_map(struct asi *asi, void *ptr, unsigned long size); /* + * Copy the memory mapping for the current module. This is defined as a + * macro to ensure it is expanded in the module making the call so that + * THIS_MODULE has the correct value. + */ +#define ASI_MAP_THIS_MODULE(asi) \ + (asi_map(asi, THIS_MODULE->core_layout.base, \ + THIS_MODULE->core_layout.size)) + +static inline int asi_map_module(struct asi *asi, char *module_name) +{ + struct module *module; + + module = find_module(module_name); + if (!module) + return -ESRCH; + + return asi_map(asi, module->core_layout.base, module->core_layout.size); +} + +/* * Function to exit the current isolation. This is used to abort isolation * when a task using isolation is scheduled out. */ From patchwork Thu Jul 11 14:25:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD4E2112C for ; Thu, 11 Jul 2019 14:26:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9823928A5A for ; Thu, 11 Jul 2019 14:26:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8C2E928AD1; Thu, 11 Jul 2019 14:26:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE1CD28A5A for ; Thu, 11 Jul 2019 14:26:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9380B8E00CE; Thu, 11 Jul 2019 10:26:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8705C8E00C4; Thu, 11 Jul 2019 10:26:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69E528E00CE; Thu, 11 Jul 2019 10:26:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 410548E00C4 for ; Thu, 11 Jul 2019 10:26:34 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id m26so6944794ioh.17 for ; Thu, 11 Jul 2019 07:26:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=d9A9L3caok6Qomoe3AKAL7r9AlPWZpUNUtRWqLcduag=; b=BRsXY9kBDHERV6pMxPMGsuj4iwbAox9XsuY3UI/OLwLZ3od3+XGmxPh3ySVFi4Lua3 HDcQWt3EGSsmh3H3BLLfU+aChtrDJa2FYEZ/yDcsB17GWuC1kfE5PrfHxWU/7s8nsJmJ AUwQYvJf/DB2Zrd6yskERdjfmffomV01rc6eGlkFF0MvR795bRaoo8JGG1+NoNiDX8jW mL9CiaftuVDbnblZSM33rgDtWyOLInm9EFawvj1m64bNd8d8R8vaTvlPmoKXyyCKmyJC nmyPTK6bFrxKhzAsFT7Coy4+lrZTx6CbwMxOurVHDi1tTAC/Yoqu/O4JnlKCcGiVHYc2 caDg== X-Gm-Message-State: APjAAAUzWNfltG4z+JFK8BAwaGpX5uo5JuuG0hlt49vFiq/FW3onF20T bZ4+QC7nsHOPHPOdTsqVTLBNF9/tY0p1MK4SzaMTCfQQjYxUk+BlFQ/bYFFaZa+Gc+V5FnNQnpJ ylcIohW+6s07LxKo7UGYrz1kB4vmuIe5c9XCVWJ9Mwa82rVfPux5GHdGP5T8wWkhrtQ== X-Received: by 2002:a5d:94d0:: with SMTP id y16mr4260572ior.123.1562855194028; Thu, 11 Jul 2019 07:26:34 -0700 (PDT) X-Google-Smtp-Source: APXvYqwDDl+yxbju+f8xOK1eSi7OK9Wm9bRbZsN4qar0H7pM/cArVt8t1Rk4jxjxSntbqDDkPjFa X-Received: by 2002:a5d:94d0:: with SMTP id y16mr4260482ior.123.1562855193060; Thu, 11 Jul 2019 07:26:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855193; cv=none; d=google.com; s=arc-20160816; b=F4qDNJ2JACur4/gwrG/BMUkOzqn2+XUfWQKFX5VYafVm2c+78E+KhvbB14bTEbVpcT IaOp5Y/7hsPCYn7nSSu3DQFLkcsF7/VeFUHVF++OgP0x9gTHeXloyfOo9xLnwQiPOE/p 8lwOABNeA2lHVtmMXc+jVdtSjtxoG2b0vY1DiILMQ0LyPau5cglQlpziBIHbpekYEVD8 ye20ZPvgIQNzatni66GB9Ir1U2K6F4zX5Dhn9HdZHG0UgC4ak0Buwvdkq5eeefj5Jrfg LTUSlb99Mge/l2pmkgykAHc7jNnnvmg5d2RKMmai/tGtYKVd0s9PPj3EVhNvcTa8jEmS vEGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=d9A9L3caok6Qomoe3AKAL7r9AlPWZpUNUtRWqLcduag=; b=C9w4a2V9X/fzKryFvQ2KkN5CR66/VuEwdsOZoAsFLAFzrtmTLa703RJa42dEpwAb5b bwFXwCY1Viq3+tfu+/m2aKua8F0vgNzn0XNCmYPSx8JCVqDMaV8mveTuXQW4tx5zlZuP 7lEbKDvSiEWzeBzk9079BTiJdt1a/AcLeCB1/u2Cmi/SruA7xAFqYcJFTB94LTZu+JdC ULS/G5osTnqVVW5szZWacbJMhnvRReXpzWYne9Ayj6ZfiSfTTHri54OcOv57uaPc6JyE nhaXEqiN7Jq1TN5xFvpyFciWQwVk0bqz06wo6jpN8r4yznkRXC6E8qPa/sTascnVLjYp BUJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=MARahqNw; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id y8si9041733jae.39.2019.07.11.07.26.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:33 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=MARahqNw; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO7gN100410; Thu, 11 Jul 2019 14:26:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=d9A9L3caok6Qomoe3AKAL7r9AlPWZpUNUtRWqLcduag=; b=MARahqNw+qo/sURXfHbD3S/83gqCOgFJ/swtRFnPmTZXmwoCOVR/N/i1pIKlToYWY/Ri k2aBJrN9/v+khzouzf0G9zHxuoQwHwjL5cB4NqEmK6LZPn4Du9L8R3LvRixNs0Ek2k51 TpfCDdoT/V5OQHZtXE2PjUdEaibVkBeu3OSq6a2iDA4XeaUx9Z+Ps6cN0b9Yifm/yA4j Iy9lSi7ma1G+usDIN0MJA79lNYFYSEqhzzbTW8owPRRK6v2c+xTImcLNpv1ndNvar4Yf gffkGfUVZ70Om45QblhdamFaGWrmGQZSauAg6M3MpWqBnQzN6fxwcMN6Aymg/BBEx4rs vQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0c8w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:23 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu3021444; Thu, 11 Jul 2019 14:26:15 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 10/26] mm/asi: Keep track of VA ranges mapped in ASI page-table Date: Thu, 11 Jul 2019 16:25:22 +0200 Message-Id: <1562855138-19507-11-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=882 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add functions to keep track of VA ranges mapped in an ASI page-table. This will be used when unmapping to ensure the same range is unmapped, at the same page-table level. This is also be used to handle mapping and unmapping of overlapping VA ranges. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 3 ++ arch/x86/mm/asi.c | 3 ++ arch/x86/mm/asi_pagetable.c | 71 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index b5dbc49..be1c190 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -24,6 +24,7 @@ enum page_table_level { struct asi { spinlock_t lock; /* protect all attributes */ pgd_t *pgd; /* ASI page-table */ + struct list_head mapping_list; /* list of VA range mapping */ /* * An ASI page-table can have direct references to the full kernel @@ -69,6 +70,8 @@ struct asi_session { void asi_init_backend(struct asi *asi); void asi_fini_backend(struct asi *asi); +void asi_init_range_mapping(struct asi *asi); +void asi_fini_range_mapping(struct asi *asi); extern struct asi *asi_create(void); extern void asi_destroy(struct asi *asi); diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index dfde245..25633a6 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -104,6 +104,8 @@ struct asi *asi_create(void) if (!asi) return NULL; + asi_init_range_mapping(asi); + page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!page) goto error; @@ -133,6 +135,7 @@ void asi_destroy(struct asi *asi) if (asi->pgd) free_page((unsigned long)asi->pgd); + asi_fini_range_mapping(asi); asi_fini_backend(asi); kfree(asi); diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index 0169395..a09a22d 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -5,10 +5,21 @@ */ #include +#include #include /* + * Structure to keep track of address ranges mapped into an ASI. + */ +struct asi_range_mapping { + struct list_head list; + void *ptr; /* range start address */ + size_t size; /* range size */ + enum page_table_level level; /* mapping level */ +}; + +/* * Get the pointer to the beginning of a page table directory from a page * table directory entry. */ @@ -75,6 +86,39 @@ void asi_fini_backend(struct asi *asi) } } +void asi_init_range_mapping(struct asi *asi) +{ + INIT_LIST_HEAD(&asi->mapping_list); +} + +void asi_fini_range_mapping(struct asi *asi) +{ + struct asi_range_mapping *range, *range_next; + + list_for_each_entry_safe(range, range_next, &asi->mapping_list, list) { + list_del(&range->list); + kfree(range); + } +} + +/* + * Return the range mapping starting at the specified address, or NULL if + * no such range is found. + */ +static struct asi_range_mapping *asi_get_range_mapping(struct asi *asi, + void *ptr) +{ + struct asi_range_mapping *range; + + lockdep_assert_held(&asi->lock); + list_for_each_entry(range, &asi->mapping_list, list) { + if (range->ptr == ptr) + return range; + } + + return NULL; +} + /* * Check if an offset in the address space isolation page-table is valid, * i.e. check that the offset is on a page effectively belonging to the @@ -574,6 +618,7 @@ static int asi_copy_pgd_range(struct asi *asi, int asi_map_range(struct asi *asi, void *ptr, size_t size, enum page_table_level level) { + struct asi_range_mapping *range_mapping; unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); unsigned long flags; @@ -582,8 +627,34 @@ int asi_map_range(struct asi *asi, void *ptr, size_t size, pr_debug("ASI %p: MAP %px/%lx/%d\n", asi, ptr, size, level); spin_lock_irqsave(&asi->lock, flags); + + /* check if the range is already mapped */ + range_mapping = asi_get_range_mapping(asi, ptr); + if (range_mapping) { + pr_debug("ASI %p: MAP %px/%lx/%d already mapped\n", + asi, ptr, size, level); + err = -EBUSY; + goto done; + } + + /* map new range */ + range_mapping = kmalloc(sizeof(*range_mapping), GFP_KERNEL); + if (!range_mapping) { + err = -ENOMEM; + goto done; + } + err = asi_copy_pgd_range(asi, asi->pgd, current->mm->pgd, addr, end, level); + if (err) + goto done; + + INIT_LIST_HEAD(&range_mapping->list); + range_mapping->ptr = ptr; + range_mapping->size = size; + range_mapping->level = level; + list_add(&range_mapping->list, &asi->mapping_list); +done: spin_unlock_irqrestore(&asi->lock, flags); return err; From patchwork Thu Jul 11 14:25:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040085 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A989F112C for ; Thu, 11 Jul 2019 14:26:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9866528A5A for ; Thu, 11 Jul 2019 14:26:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8C76328AD1; Thu, 11 Jul 2019 14:26:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC41528A5A for ; Thu, 11 Jul 2019 14:26:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D57CC8E00CD; Thu, 11 Jul 2019 10:26:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CE4168E00C4; Thu, 11 Jul 2019 10:26:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A963B8E00CD; Thu, 11 Jul 2019 10:26:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 7980C8E00C4 for ; Thu, 11 Jul 2019 10:26:32 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id p12so6933305iog.19 for ; Thu, 11 Jul 2019 07:26:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Ccv45kVv21CrVP2UQjHtQ159yrdddBRkfeerFm1lQZs=; b=RKVKJqrnyGcTkDYad4uF1YBFpKnKXUlSjWA3nEA1BxRk7hH+lvdEO08brX2Nltg+UG ORF6ivdoWyd9tdqjyI96Cqqlarx+q6SpIIKzJ8D/UUotipa2jtpL44lc/fQEnmXqYOqi lG9rK0AhSOrn2MiEA8IUrPTPVCPfsdXHz5pKhEpZ1AAAkN6oCkmPmYkOBtUEqIk/sfGk KRcXjUoZvWEQpLtrJDCAC6Z1icDpo0+xn7GOgv91GdM0CIlIEOSHce4bZVNZBCKmgZie rN/OHHNwN88+yn7bI6diG6tF5kxYOr/iVkEQpWP/P3uayfXHNFMO8/UmnyZrOdP1ALyL DZ4A== X-Gm-Message-State: APjAAAWUZuKMOG2RZLP0W7e8B7bHSca7G98oLz+sHWdpkgWWCl+a3ov2 jy6jHwdeaXTFO7GfLX2fh1cY2CMg4jlMK/9TAUXq2Y5qdm08X+zMfUJ9KvVqgCMNRv2fen3f0tm rLdvcwdUKcPJjlVBnu9hd3IXj6IgfN1QeWY5DKBMw5L/bQXIzXTmQgg37Kt953ejutw== X-Received: by 2002:a6b:5106:: with SMTP id f6mr4265464iob.15.1562855192268; Thu, 11 Jul 2019 07:26:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqyl8g77ejFaGOQqKS2F9s4PPw6clpgurGnNy5KNSFNGa0dHEk9r5FvW3dUOZhlgenLsprzX X-Received: by 2002:a6b:5106:: with SMTP id f6mr4265401iob.15.1562855191577; Thu, 11 Jul 2019 07:26:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855191; cv=none; d=google.com; s=arc-20160816; b=IS3Gx0KwKK5FZngku5uLr4fwIncUh73z3metVCDioOghD7a3RS3kLy9L5s4sXQq9KB NEZF9KW1XiO3oQZp+K2tXeag+h/ePlnjZdMxYBnJfTHW48n2zv6aXRF6S3IRVhzVS8rg OF22xt/lvok5XiyjXX7f9hFFlBL58g7k/C01U00/avVMxOCLsZ7LGC5CuEIiiLzxhOzo MTcCsJkSlYPFz5bmt3QvS4N9vQV5pPSEKEGGhDGNQGy8jZzoRlF9qrbF5/EKoFAccV7K 6Rk+JVVoh1PB6Tn/hRodZ6Q2jJdPkvDe5k2gyY/JpC0uArT1ST5pVgj+JsmFlpYLfQzD Smsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ccv45kVv21CrVP2UQjHtQ159yrdddBRkfeerFm1lQZs=; b=awOsdQKEvk7fRhjbIm2EHq+27dgt8K6HemPzd4/RJ2MczB4WPN3GOZ6hzEVPYzEQqk ka3PSX8++1tT7whUe/jrZCYvmfpiNiFdMFzSx0urtQfbOi9DAze9pgc3jR0WCUV67F0s AoL7qaNEFYEHCLbqdJPnP64H5u6P+RDe6PHW2358YuYsj9nVfrHpOL48XOsDIRUBtMCM FFk4jsmYLfr93YBHupMADf6bagKDAMhSyWgmO0X6T0IPpuhEUyOqJRghHBAp996PwsjH klxAv669w8Fmfq2GxelE4ASfDTZyQx98TxsJBINQkH1RWVyZ0ptywMnDhr7WP/9zINr8 1UsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="HGyms/pI"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id t17si9199384jam.48.2019.07.11.07.26.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:31 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="HGyms/pI"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOabr100891; Thu, 11 Jul 2019 14:26:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Ccv45kVv21CrVP2UQjHtQ159yrdddBRkfeerFm1lQZs=; b=HGyms/pISRX5hcPdr6Z5qUYPoK8BQ52EIauuQPyhPhRre44rLyUHzvnoqOTRvXzrwhCv OTY05ptHcmicwwbGGPPXQN6m4TPthhy/xClxEZa4yF8uHSsKm7ZSH9u/tLfCQHG6hg1K +gEPReBvdFOuXQjB3QhAq8WvFQ4EtwdfdpRMWXWmCpj3REbSSnlT8iB9x+evcMVyxruc OTPrrYVoOXTGYADVN58b8Xu0sMhIRq/VtBIXKZS/5/ya9K9aD3oXpTX8QexDk1Ib94j1 ij8M3PZb4/wsMYxdM8hN/VwLMvqwLeYU8gvZc6wjF0NGEF6IBAOdWQJA2MvUwBG0KojU 6Q== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0c8m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:21 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu4021444; Thu, 11 Jul 2019 14:26:18 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 11/26] mm/asi: Functions to clear ASI page-table entries for a VA range Date: Thu, 11 Jul 2019 16:25:23 +0200 Message-Id: <1562855138-19507-12-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=905 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Provide functions to clear page-table entries in the ASI page-table for a specified VA range. Functions also check that the clearing effectively happens in the ASI page-table and there is no crossing of the ASI page-table boundary (through references to the kernel page table), so that the kernel page table is not modified by mistake. As information (address, size, page-table level) about VA ranges mapped to the ASI page-table is tracked, clearing is done with just specifying the start address of the range. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 1 + arch/x86/mm/asi_pagetable.c | 134 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index be1c190..919129f 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -83,6 +83,7 @@ extern bool asi_fault(struct pt_regs *regs, unsigned long error_code, extern int asi_map_range(struct asi *asi, void *ptr, size_t size, enum page_table_level level); extern int asi_map(struct asi *asi, void *ptr, unsigned long size); +extern void asi_unmap(struct asi *asi, void *ptr); /* * Copy the memory mapping for the current module. This is defined as a diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index a09a22d..7aee236 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -670,3 +670,137 @@ int asi_map(struct asi *asi, void *ptr, unsigned long size) return asi_map_range(asi, ptr, size, PGT_LEVEL_PTE); } EXPORT_SYMBOL(asi_map); + +static void asi_clear_pte_range(struct asi *asi, pmd_t *pmd, + unsigned long addr, unsigned long end) +{ + pte_t *pte; + + pte = asi_pte_offset(asi, pmd, addr); + if (IS_ERR(pte)) + return; + + do { + pte_clear(NULL, addr, pte); + } while (pte++, addr += PAGE_SIZE, addr < end); +} + +static void asi_clear_pmd_range(struct asi *asi, pud_t *pud, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pmd_t *pmd; + + pmd = asi_pmd_offset(asi, pud, addr); + if (IS_ERR(pmd)) + return; + + do { + next = pmd_addr_end(addr, end); + if (pmd_none(*pmd) || pmd_present(*pmd)) + continue; + if (level == PGT_LEVEL_PMD || pmd_trans_huge(*pmd) || + pmd_devmap(*pmd)) { + pmd_clear(pmd); + continue; + } + asi_clear_pte_range(asi, pmd, addr, next); + } while (pmd++, addr = next, addr < end); +} + +static void asi_clear_pud_range(struct asi *asi, p4d_t *p4d, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pud_t *pud; + + pud = asi_pud_offset(asi, p4d, addr); + if (IS_ERR(pud)) + return; + + do { + next = pud_addr_end(addr, end); + if (pud_none(*pud)) + continue; + if (level == PGT_LEVEL_PUD || pud_trans_huge(*pud) || + pud_devmap(*pud)) { + pud_clear(pud); + continue; + } + asi_clear_pmd_range(asi, pud, addr, next, level); + } while (pud++, addr = next, addr < end); +} + +static void asi_clear_p4d_range(struct asi *asi, pgd_t *pgd, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + p4d_t *p4d; + + p4d = asi_p4d_offset(asi, pgd, addr); + if (IS_ERR(p4d)) + return; + + do { + next = p4d_addr_end(addr, end); + if (p4d_none(*p4d)) + continue; + if (level == PGT_LEVEL_P4D) { + p4d_clear(p4d); + continue; + } + asi_clear_pud_range(asi, p4d, addr, next, level); + } while (p4d++, addr = next, addr < end); +} + +static void asi_clear_pgd_range(struct asi *asi, pgd_t *pagetable, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pgd_t *pgd; + + pgd = pgd_offset_pgd(pagetable, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgd)) + continue; + if (level == PGT_LEVEL_PGD) { + pgd_clear(pgd); + continue; + } + asi_clear_p4d_range(asi, pgd, addr, next, level); + } while (pgd++, addr = next, addr < end); +} + +/* + * Clear page table entries in the specified ASI page-table. + */ +void asi_unmap(struct asi *asi, void *ptr) +{ + struct asi_range_mapping *range_mapping; + unsigned long addr, end; + unsigned long flags; + + spin_lock_irqsave(&asi->lock, flags); + + range_mapping = asi_get_range_mapping(asi, ptr); + if (!range_mapping) { + pr_debug("ASI %p: UNMAP %px - not mapped\n", asi, ptr); + goto done; + } + + addr = (unsigned long)range_mapping->ptr; + end = addr + range_mapping->size; + pr_debug("ASI %p: UNMAP %px/%lx/%d\n", asi, ptr, + range_mapping->size, range_mapping->level); + asi_clear_pgd_range(asi, asi->pgd, addr, end, range_mapping->level); + list_del(&range_mapping->list); + kfree(range_mapping); +done: + spin_unlock_irqrestore(&asi->lock, flags); +} +EXPORT_SYMBOL(asi_unmap); From patchwork Thu Jul 11 14:25:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55D451395 for ; Thu, 11 Jul 2019 14:26:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 451EA28A5A for ; Thu, 11 Jul 2019 14:26:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3932428AD1; Thu, 11 Jul 2019 14:26:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B786D28AC8 for ; Thu, 11 Jul 2019 14:26:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84ECC8E00CF; Thu, 11 Jul 2019 10:26:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 786D08E00C4; Thu, 11 Jul 2019 10:26:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6271F8E00CF; Thu, 11 Jul 2019 10:26:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by kanga.kvack.org (Postfix) with ESMTP id 33DE28E00C4 for ; Thu, 11 Jul 2019 10:26:36 -0400 (EDT) Received: by mail-io1-f70.google.com with SMTP id n8so6924565ioo.21 for ; Thu, 11 Jul 2019 07:26:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=gDByh8p+inBGj8eNOkt+0s0ErRCor6nCeXzUp1uJmZ0=; b=C+MGP3jzaNCZ1pNDycejCLKD3HU1TOhXffj8MLl+lZy1rorZsqxtkplQ8w7p9HhMd1 f/8eaKACUKNcAnglJc79t06SNCq2lblu42+QXVMUgwmfk019VJzIYklJDYzBuODqTKtS z0+tHOvLKRlpuWaT5Ty3I9uiFYu153VXP/unHt2NbuCnevvT/EiEwR29BXjJSJFC0aq1 VWznPJM86JiyV9QGz58UcGcLymaPfjDR8CTkxhQmuxdOp9U50mdj059wMphEMOt+vZo6 7ECrjkHUPd+TGFhSw2nkP4zIvwjJkX8looqBUcGRVZnn0Rfm4I7FOCeuhHe7n6q73Rip nFKg== X-Gm-Message-State: APjAAAVidJtDwKJRww0vW+hfHeRbN9OrfHCKfQkfrfLWGUjYDBn4eLk2 zxUe760iLM3+zicovee/frlNiFDI8IGIbg32Jpmb0Ydb+wg13KbnRssy3WJDqzQh7tvB3ImfnkG evoykqpsyuAXTExRC6eEkKn9Hk0vYA+JqEO//0oIkmtn38RMTGv2rskuXvvrejgpsBg== X-Received: by 2002:a5d:928a:: with SMTP id s10mr4647146iom.29.1562855195982; Thu, 11 Jul 2019 07:26:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqwMydXJcs8IqO9pyZAMFPj+cK/87CVIe08oz4/ZRLY+2J06ORNfy12w3qhxvRMkP5rtZqsh X-Received: by 2002:a5d:928a:: with SMTP id s10mr4647092iom.29.1562855195425; Thu, 11 Jul 2019 07:26:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855195; cv=none; d=google.com; s=arc-20160816; b=PbJUbd6V8ubdHJQ00C8M0RATV8ZH9sax5GYgoZmKKGuQzYxiStoZGbLPV+1/yMxSg4 wJDqVBErOheXuORnyNqNuB0elbQ1folGAHTbJiohktLBZUpHCfTVoKOhObhUqFbudRE3 iJFFLlq5eMoucIvA1jEyFogO2hwwvf+bMWsc/ULT2vrqVt4SDd023Fpg5QVPc8JHJM35 BxQkjfMYgk2PsBjZsY97pps8K3liykStfyC8/JNcZ7XSg2X3umh5c2NPpUVro+Vd5UFu S4UuvqFp6WKU6Yk3Vn6V8gK5NkUx6pvqbp+wGej+3JshzfrGZdk87y9E1XbSrQUiqx2X SLxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gDByh8p+inBGj8eNOkt+0s0ErRCor6nCeXzUp1uJmZ0=; b=v+hDurvBXM0HiP0/ZJ1VCCrl8fnHMEFogQbIJeEzottcYdK3wf8T1ME3sTE8hIiIuP 0iL0Hwc4QQp/wFgvZekxAZ0D6txPPCJWZH/Q3t2vl4nAi2dnk2/iTRnu8sogYms7ndaf 5VjyhsuL0fGhCmI7kNGSJT//vGzJsq6zQ62zGP0lD1qaduHJZ+A0D443JaIkEnT0UQ0U xMBCoKo8uieCkWncZR6XzrTfxThqsXhfoJhj06qQsgIGM3/ue75IV+FLPrmWRxqAd3dH KLbCVmit4hk/cLe4GmtlJp+QK/klDjLPZBnf+YIL/GEJbcK32FCbRzSpMM/wNFa3d6fj 9vnQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=uIvYew5i; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id l10si7686574ion.114.2019.07.11.07.26.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:35 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=uIvYew5i; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOKHZ001595; Thu, 11 Jul 2019 14:26:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=gDByh8p+inBGj8eNOkt+0s0ErRCor6nCeXzUp1uJmZ0=; b=uIvYew5i+/N3gKnyr4ociYqq+qt56BKvpfHx2yG6ELbiXTDFF1FnRFdNiTxKLqzKpugT 7rZFZ27tp3XeOzJIDBEQWKZlGMdD04zWtCMR9kz9rtKT/I1eEKjmUoR0EfXOS4LZP41X SaOmMD0h5rmiZKjuWODlCFnTwFv/w3OU84qMp7I8vBQjvkB4kLSw+vU/HUfR9FiS4qmM dt9lswqhmPEYZvrjigJDb4qNQfKlbGFfmUQluSi/uyNCjdt/jH+m3QqAeoRgbBE5SECi sbhB6HhXJDdRZ5C1Vh+uI9HzCIOa4DMf+C+kQvxHt3TG4Fqj9spPHDE/SpvfNokpoD2Z Tw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dyd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:25 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu5021444; Thu, 11 Jul 2019 14:26:21 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 12/26] mm/asi: Function to copy page-table entries for percpu buffer Date: Thu, 11 Jul 2019 16:25:24 +0200 Message-Id: <1562855138-19507-13-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=895 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Provide functions to copy page-table entries from the kernel page-table to an ASI page-table for a percpu buffer. A percpu buffer have a different VA range for each cpu and all them have to be copied. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 6 ++++++ arch/x86/mm/asi_pagetable.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 919129f..912b6a7 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -105,6 +105,12 @@ static inline int asi_map_module(struct asi *asi, char *module_name) return asi_map(asi, module->core_layout.base, module->core_layout.size); } +#define ASI_MAP_CPUVAR(asi, cpuvar) \ + asi_map_percpu(asi, &cpuvar, sizeof(cpuvar)) + +extern int asi_map_percpu(struct asi *asi, void *percpu_ptr, size_t size); +extern void asi_unmap_percpu(struct asi *asi, void *percpu_ptr); + /* * Function to exit the current isolation. This is used to abort isolation * when a task using isolation is scheduled out. diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index 7aee236..a4fe867 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -804,3 +804,41 @@ void asi_unmap(struct asi *asi, void *ptr) spin_unlock_irqrestore(&asi->lock, flags); } EXPORT_SYMBOL(asi_unmap); + +void asi_unmap_percpu(struct asi *asi, void *percpu_ptr) +{ + void *ptr; + int cpu; + + pr_debug("ASI %p: UNMAP PERCPU %px\n", asi, percpu_ptr); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + pr_debug("ASI %p: UNMAP PERCPU%d %px\n", asi, cpu, ptr); + asi_unmap(asi, ptr); + } +} +EXPORT_SYMBOL(asi_unmap_percpu); + +int asi_map_percpu(struct asi *asi, void *percpu_ptr, size_t size) +{ + int cpu, err; + void *ptr; + + pr_debug("ASI %p: MAP PERCPU %px\n", asi, percpu_ptr); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + pr_debug("ASI %p: MAP PERCPU%d %px\n", asi, cpu, ptr); + err = asi_map(asi, ptr, size); + if (err) { + /* + * Need to unmap any percpu mapping which has + * succeeded before the failure. + */ + asi_unmap_percpu(asi, percpu_ptr); + return err; + } + } + + return 0; +} +EXPORT_SYMBOL(asi_map_percpu); From patchwork Thu Jul 11 14:25:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040091 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2784B1395 for ; Thu, 11 Jul 2019 14:26:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1681128A5A for ; Thu, 11 Jul 2019 14:26:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 074A028AD1; Thu, 11 Jul 2019 14:26:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8687B28A5A for ; Thu, 11 Jul 2019 14:26:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9366A8E00D0; Thu, 11 Jul 2019 10:26:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 871DF8E00C4; Thu, 11 Jul 2019 10:26:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EA4C8E00D0; Thu, 11 Jul 2019 10:26:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 4865B8E00C4 for ; Thu, 11 Jul 2019 10:26:39 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id r27so6955921iob.14 for ; Thu, 11 Jul 2019 07:26:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Ag0k+tMlXoCUCjgmP3NxD1Cl4FnNgcaLg1SLsqSglmA=; b=sAJGt/5QBTvFocxxinOowTIVs/HeK+Ofru8WFiVm2G/oINKg7+dvDmBF89SJSRATX4 O32Ly01Kuh8beaQ9aR8KB2BOp0DKwG2ujvV/dGjdVahk9rcxosLsQ0+d5BxMh9Xab5pc mFFGyIMV2Ei2im0D59uhDh3v5wm8L+/u0RqmuVHHYVwcExkxDUDK80dRsfpeWvQjS13V NkQlU6s4qUCpJgqhlCC8yNYHMidvle0hN0c2wSPhd//M/uhwv7DwGEZCgw4KKtfmDwmg iGQdGj2PWUQBTN63gWC+ejP8WfmJ5qVdiyOxIqISMknypnFji0bbuvNB1T60GPPG4C1+ xdDg== X-Gm-Message-State: APjAAAXLjmN/2V33KbMSz1HtHmtKEhTxTBNVqV9sh57SRq579h7vd23Q UeBOs3mCkg100u27buAYkClD+t6Zt10gBdMwFWGnZ6TJXhXBuX6o7yjZ2f9En/3jMmkvcE/5ibt W+oXzzXdCg66iNvGxOVtXPSQaOLND4gEWE2Es819M4qL8qGIBkgt0fYYe9Fc4c/NI2g== X-Received: by 2002:a6b:5115:: with SMTP id f21mr4752023iob.173.1562855199091; Thu, 11 Jul 2019 07:26:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqw8t0ZRyRV502o/BpWpsPvHc0IuQoI0KYc8bn3Y37gjQbnB8wrcrbDQXrgCEIzfwLhObNq1 X-Received: by 2002:a6b:5115:: with SMTP id f21mr4751956iob.173.1562855198449; Thu, 11 Jul 2019 07:26:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855198; cv=none; d=google.com; s=arc-20160816; b=QzXMnhK5FWYqhImBVF+jJHDD/nZTjl565FfhEnpv/XEHIO4igz4KwWJshR7+kgVz5z 35ZFKhXD3pIXEanmGMAtWe14j5LsZ+bPChm6j3R6g/LXVncSbzdvNH4hKas3W5ixLBeN xIFV9rLsyhFIIUbgooBHPMjwf0N/cCkjbh3OPD+kFzSKRxV2YztydRzm+G8zMgtCSw0k I/V/P0EqiyrAcirYbt4GfWEb7z1k50iFKGTHpSbtEfWRV2jOWhd/swFz+sFwuqAfmqF0 TyuQjzEtbGCe8gIPMmSDQRAEp9xOfVZX5N0u4yAKUoiGew7+k1LqK6rvAPKKDq9HMq6r Q1Iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ag0k+tMlXoCUCjgmP3NxD1Cl4FnNgcaLg1SLsqSglmA=; b=QZbV0ei2gVRcX5sUOURklQFzrAVvKnw+yUtAoFfzBwjyi+cHPDPMiGxb0JqMNmMqhr /9YKMV5pafgR2J6uwQMaUPmwxywJmfw+hqDHY7jVrQAMU3bXXdXUEiDxEmAf0WZoic4u gAQch9ti3ltDs90/Rs7q++gLMjObzZAOiN46Kip4JtwwC6D54iuqz7DiLSYdbqWqFgbi uy/9EvX26AQXeTjrKC4CEcEAHufqzgdG/L5QEqp9f0Uh9REuYNHBMK06Es6eYe7RY6pE vMpBy/euXNNaFt7eCUUB9qWtP56e1dr3xVkYVEC5T6Nfye5704khyHnVCzbse7xWqQiw BRpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=keVQsRLh; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id y14si10133684jan.93.2019.07.11.07.26.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:38 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=keVQsRLh; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOAvB001518; Thu, 11 Jul 2019 14:26:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Ag0k+tMlXoCUCjgmP3NxD1Cl4FnNgcaLg1SLsqSglmA=; b=keVQsRLhZyNb/dFJnnzguyu+6Vl7qT1iW80eQJUi6p8HUYNezOn2M6QRB0NcNQhQI7JT 5HACwzQ7vK/fKkmWqNIFxS0mv8uBEA1J5s9pHNXY/wHvMqw/x77ya8pCk5djXXg3w8tA Y+DFHNzDwpT4d81yj2dhIMIFubn8ssOAMPMn8hTz8Cg2l5maUEKHDpqBF2nYVJbVIius zq+q7wJ21XaxrzeREUMqbHN8popuelxmyOtwjWh5plKHWNmsE8C4KEMn9NIt7tf3KMir nsOpTlKGCKH8e0WHGLCVGvgf9UFWaCxJYPAxRKOlsGhf5idexkvlsNaYd43tN8SZVOoQ SA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0dyw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:28 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu6021444; Thu, 11 Jul 2019 14:26:25 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 13/26] mm/asi: Add asi_remap() function Date: Thu, 11 Jul 2019 16:25:25 +0200 Message-Id: <1562855138-19507-14-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=832 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add a function to remap an already mapped buffer with a new address in an ASI page-table: the already mapped buffer is unmapped, and a new mapping is added for the specified new address. This is useful to track and remap a buffer which can be freed and then reallocated. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 1 + arch/x86/mm/asi_pagetable.c | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 912b6a7..cf5d198 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -84,6 +84,7 @@ extern int asi_map_range(struct asi *asi, void *ptr, size_t size, enum page_table_level level); extern int asi_map(struct asi *asi, void *ptr, unsigned long size); extern void asi_unmap(struct asi *asi, void *ptr); +extern int asi_remap(struct asi *asi, void **mapping, void *ptr, size_t size); /* * Copy the memory mapping for the current module. This is defined as a diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index a4fe867..1ff0c47 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -842,3 +842,28 @@ int asi_map_percpu(struct asi *asi, void *percpu_ptr, size_t size) return 0; } EXPORT_SYMBOL(asi_map_percpu); + +int asi_remap(struct asi *asi, void **current_ptrp, void *new_ptr, size_t size) +{ + void *current_ptr = *current_ptrp; + int err; + + if (current_ptr == new_ptr) { + /* no change, already mapped */ + return 0; + } + + if (current_ptr) { + asi_unmap(asi, current_ptr); + *current_ptrp = NULL; + } + + err = asi_map(asi, new_ptr, size); + if (err) + return err; + + *current_ptrp = new_ptr; + + return 0; +} +EXPORT_SYMBOL(asi_remap); From patchwork Thu Jul 11 14:25:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040093 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4D4E112C for ; Thu, 11 Jul 2019 14:26:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B377428AC8 for ; Thu, 11 Jul 2019 14:26:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A787528AD9; Thu, 11 Jul 2019 14:26:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBF7328AC8 for ; Thu, 11 Jul 2019 14:26:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 563638E00D1; Thu, 11 Jul 2019 10:26:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4F1C68E00C4; Thu, 11 Jul 2019 10:26:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F72D8E00D1; Thu, 11 Jul 2019 10:26:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by kanga.kvack.org (Postfix) with ESMTP id 08F9E8E00C4 for ; Thu, 11 Jul 2019 10:26:43 -0400 (EDT) Received: by mail-io1-f70.google.com with SMTP id v11so6958992iop.7 for ; Thu, 11 Jul 2019 07:26:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=/snaNkVEEW4nTR1xAXfvlEoe9rj/2L0Z92puglvXclU=; b=sha/vlkX1IpQ1PmdC3D2JvGUoIpIPNsOA8yPFiGN3HbZ4lzz2gXKPBu/wR0fBlLwSB TpyQ2ALPYgxI9flFhi7iYJeWJUhXKXJxU9hPZtTMtmYPNW08FHJxqIpKNLiB0TeyiQDh CQ0XnpTKOfByEOyZBOJofuAV0TbmYCKhZxm/HmzKSgssVHTZwMeTY4zD+E+GZVvgra92 FhiBxX5XJ/y0u1O9wW8SRLLC8sueJ74WRn+DzY01x89eKa5uSXJAn9g7vGbR81FXx73S evH6SKQ06SgwzhzgPKVWlr4WFjHfgqMUaw1Nw7qHr4uJrd0rFOGDJ5fpIEvtq7n9TVJw XdKA== X-Gm-Message-State: APjAAAUrl4t8Tr+fFhlWj0P3Ar3W9l/XzZOs/kAosjOCHd0osVyimqvp wLf00ZIjJ8rkpuop1hTg7kqvloKH9piBAE0sKYVlwL3Kv67OGp0Y/7U7oS9t08CI14AIYLhg/Ct mkzYWEUfmbZuJEhZvky5K4a7lm5VW2iJRMkScdfEm06to4bYfT01DGEpQZO0SYqRq+A== X-Received: by 2002:a5d:9550:: with SMTP id a16mr4668807ios.106.1562855202786; Thu, 11 Jul 2019 07:26:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqzL1nD4zSuovF2OMieH4L/I7bjx/Kl096XVXbL21vsplsrD82gAoSF4jMI1f/2MpXl6XQe8 X-Received: by 2002:a5d:9550:: with SMTP id a16mr4668702ios.106.1562855201789; Thu, 11 Jul 2019 07:26:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855201; cv=none; d=google.com; s=arc-20160816; b=rp+Xb4+o3I2yaM9ewYoz8XrBLRSv+7qUnVHLLeZR7rJ8uhugAXUUXNiDeHLL1SLg3Y joxGFpJze3mjxp4ATJ1GpEIlfo4sGfEzWAWJdjdDMtCme1Y1BMSVoud31ZFX6g6Oq0V5 LIzCofGr4nREGOkQNdRCniE3hpSGP3qpk/RWJCCSpmfoMKSzQynGfbFRbXPBVTOr4hGd X+vtAF6Qu5SQszjWmxR7v4dihO2stmHqWEYm53nbmFR9oD60/orna73O6psjAZv8ER6k OTokerE6PQXuypBmZI7zQ7oEZC1MiaHLOBxA77/aILtX0y/LfIkrExVxmFdov+9lQoIp 9BEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/snaNkVEEW4nTR1xAXfvlEoe9rj/2L0Z92puglvXclU=; b=CA3wMrtGMNm0OGS4Y6yvgxdkez1xZX0kAPmi2LgMOzIIJDuQ1zUHvkzbevOC4gjQyv jyTKVck0o+kZT1dWIogjdfDUXvHQ+RTP2mdEYrcZpmxvLt5FGnG23HZ33c9/v/6A2GL1 jQl8ua9WoqC4iUmsQxhiSCQpUvgMWfFT/6aQwEILlMmGzxx46VFdOczgrQeBXPR5HWCB 1gZog8ar2SEIupmS5fwUmXt7aU4juKUViSmnudUfA1baWyMCR1n6vijY6dlVksKCof+r MTb2PAmFlgGC/CI7vM8l9RoBq1E4LmKSdPoZQLJ/AdA38tqbakQsJP2AVhJkJvTyescO mutw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=DN3eG8DX; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id m4si9249096iob.58.2019.07.11.07.26.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:41 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=DN3eG8DX; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO9em001500; Thu, 11 Jul 2019 14:26:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=/snaNkVEEW4nTR1xAXfvlEoe9rj/2L0Z92puglvXclU=; b=DN3eG8DXxQ7ozWrPnuAeiliXI+C+IxsV4od5f1TGtSHKulg61bcs9xHwRh7Wbxnghz9I ThGtm4GjaEXofJ/g9FZ3+haVgxr28XtB+ucHwwPynqaJaB3a3cRcsSIJZKS7VGg7EiMw rNBOsKQ9GocLmZ4MB3gTzKkSMvJmagNALGBonkOZRxAN95EUC0/Wbwu+HXXfAxWl18C9 GkNsUrqBGLn4Ir8iOvlMsFHQtMtfTKAD2WUAOBydyAaB9K/Tb/BB4EigTZEu+vozMNyu ysx0uiZcdvKRUDvzf4/aPdWlIS0c9udhkbyOJiJWUg4LAaH97xlf2yC2mUSNfBoqVSwX 4Q== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e09-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:32 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu7021444; Thu, 11 Jul 2019 14:26:28 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 14/26] mm/asi: Handle ASI mapped range leaks and overlaps Date: Thu, 11 Jul 2019 16:25:26 +0200 Message-Id: <1562855138-19507-15-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=857 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When mapping a buffer in an ASI page-table, data around the buffer can also be mapped if the entire buffer is not aligned with the page directory size used for the mapping. So, data can potentially leak into the ASI page-table. In such a case, print a warning that data are leaking. Also data effectively mapped can overlap with an already mapped buffer. This is not an issue when mapping data but, when unmapping, make sure data from another buffer don't get unmapped as a side effect. Signed-off-by: Alexandre Chartre --- arch/x86/mm/asi_pagetable.c | 230 +++++++++++++++++++++++++++++++++++++++---- 1 files changed, 212 insertions(+), 18 deletions(-) diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index 1ff0c47..f1ee65b 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -9,6 +9,14 @@ #include +static unsigned long page_directory_size[] = { + [PGT_LEVEL_PTE] = PAGE_SIZE, + [PGT_LEVEL_PMD] = PMD_SIZE, + [PGT_LEVEL_PUD] = PUD_SIZE, + [PGT_LEVEL_P4D] = P4D_SIZE, + [PGT_LEVEL_PGD] = PGDIR_SIZE, +}; + /* * Structure to keep track of address ranges mapped into an ASI. */ @@ -17,8 +25,16 @@ struct asi_range_mapping { void *ptr; /* range start address */ size_t size; /* range size */ enum page_table_level level; /* mapping level */ + int overlap; /* overlap count */ }; +#define ASI_RANGE_MAP_ADDR(r) \ + round_down((unsigned long)((r)->ptr), page_directory_size[(r)->level]) + +#define ASI_RANGE_MAP_END(r) \ + round_up((unsigned long)((r)->ptr + (r)->size), \ + page_directory_size[(r)->level]) + /* * Get the pointer to the beginning of a page table directory from a page * table directory entry. @@ -609,6 +625,71 @@ static int asi_copy_pgd_range(struct asi *asi, return 0; } + +/* + * Map a VA range, taking into account any overlap with already mapped + * VA ranges. On error, return < 0. Otherwise return the number of + * ranges the specified range is overlapping with. + */ +static int asi_map_overlap(struct asi *asi, void *ptr, size_t size, + enum page_table_level level) +{ + unsigned long map_addr, map_end; + unsigned long addr, end; + struct asi_range_mapping *range; + bool need_mapping; + int err, overlap; + + addr = (unsigned long)ptr; + end = addr + (unsigned long)size; + need_mapping = true; + overlap = 0; + + lockdep_assert_held(&asi->lock); + list_for_each_entry(range, &asi->mapping_list, list) { + + if (range->ptr == ptr && range->size == size) { + /* we are mapping the same range again */ + pr_debug("ASI %p: MAP %px/%lx/%d already mapped\n", + asi, ptr, size, level); + return -EBUSY; + } + + /* check overlap with mapped range */ + map_addr = ASI_RANGE_MAP_ADDR(range); + map_end = ASI_RANGE_MAP_END(range); + if (end <= map_addr || addr >= map_end) { + /* no overlap, continue */ + continue; + } + + pr_debug("ASI %p: MAP %px/%lx/%d overlaps with %px/%lx/%d\n", + asi, ptr, size, level, + range->ptr, range->size, range->level); + range->overlap++; + overlap++; + + /* + * Check if new range is included into an existing range. + * If so then the new range is already entirely mapped. + */ + if (addr >= map_addr && end <= map_end) { + pr_debug("ASI %p: MAP %px/%lx/%d implicitly mapped\n", + asi, ptr, size, level); + need_mapping = false; + } + } + + if (need_mapping) { + err = asi_copy_pgd_range(asi, asi->pgd, current->mm->pgd, + addr, end, level); + if (err) + return err; + } + + return overlap; +} + /* * Copy page table entries from the current page table (i.e. from the * kernel page table) to the specified ASI page-table. The level @@ -619,44 +700,53 @@ int asi_map_range(struct asi *asi, void *ptr, size_t size, enum page_table_level level) { struct asi_range_mapping *range_mapping; + unsigned long page_dir_size = page_directory_size[level]; unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); + unsigned long map_addr, map_end; unsigned long flags; - int err; + int err, overlap; + + map_addr = round_down(addr, page_dir_size); + map_end = round_up(end, page_dir_size); - pr_debug("ASI %p: MAP %px/%lx/%d\n", asi, ptr, size, level); + pr_debug("ASI %p: MAP %px/%lx/%d -> %lx-%lx\n", asi, ptr, size, level, + map_addr, map_end); + if (map_addr < addr) + pr_debug("ASI %p: MAP LEAK %lx-%lx\n", asi, map_addr, addr); + if (map_end > end) + pr_debug("ASI %p: MAP LEAK %lx-%lx\n", asi, end, map_end); spin_lock_irqsave(&asi->lock, flags); - /* check if the range is already mapped */ - range_mapping = asi_get_range_mapping(asi, ptr); - if (range_mapping) { - pr_debug("ASI %p: MAP %px/%lx/%d already mapped\n", - asi, ptr, size, level); - err = -EBUSY; - goto done; + /* + * Map the new range with taking overlap with already mapped ranges + * into account. + */ + overlap = asi_map_overlap(asi, ptr, size, level); + if (overlap < 0) { + err = overlap; + goto error; } - /* map new range */ + /* add new range */ range_mapping = kmalloc(sizeof(*range_mapping), GFP_KERNEL); if (!range_mapping) { err = -ENOMEM; - goto done; + goto error; } - err = asi_copy_pgd_range(asi, asi->pgd, current->mm->pgd, - addr, end, level); - if (err) - goto done; - INIT_LIST_HEAD(&range_mapping->list); range_mapping->ptr = ptr; range_mapping->size = size; range_mapping->level = level; + range_mapping->overlap = overlap; list_add(&range_mapping->list, &asi->mapping_list); -done: spin_unlock_irqrestore(&asi->lock, flags); + return 0; +error: + spin_unlock_irqrestore(&asi->lock, flags); return err; } EXPORT_SYMBOL(asi_map_range); @@ -776,6 +866,110 @@ static void asi_clear_pgd_range(struct asi *asi, pgd_t *pagetable, } while (pgd++, addr = next, addr < end); } + +/* + * Unmap a VA range, taking into account any overlap with other mapped + * VA ranges. This unmaps the specified range then remap any range this + * range was overlapping with. + */ +static void asi_unmap_overlap(struct asi *asi, struct asi_range_mapping *range) +{ + unsigned long map_addr, map_end; + struct asi_range_mapping *r; + unsigned long addr, end; + unsigned long r_addr; + bool need_unmapping; + int err, overlap; + + addr = (unsigned long)range->ptr; + end = addr + (unsigned long)range->size; + overlap = range->overlap; + need_unmapping = true; + + lockdep_assert_held(&asi->lock); + + /* + * Adjust overlap information and check if range effectively needs + * to be unmapped. + */ + list_for_each_entry(r, &asi->mapping_list, list) { + + if (!overlap) { + /* no more overlap */ + break; + } + + WARN_ON(range->ptr == r->ptr && range->size == r->size); + + /* check overlap with other range */ + map_addr = ASI_RANGE_MAP_ADDR(r); + map_end = ASI_RANGE_MAP_END(r); + if (end < map_addr || addr >= map_end) { + /* no overlap, continue */ + continue; + } + + pr_debug("ASI %p: UNMAP %px/%lx/%d overlaps with %px/%lx/%d\n", + asi, range->ptr, range->size, range->level, + r->ptr, r->size, r->level); + r->overlap--; + overlap--; + + /* + * Check if range is included into a remaining mapped range. + * If so then there's no need to unmap. + */ + if (map_addr <= addr && end <= map_end) { + pr_debug("ASI %p: UNMAP %px/%lx/%d still mapped\n", + asi, range->ptr, range->size, range->level); + need_unmapping = false; + } + } + + WARN_ON(overlap); + + if (need_unmapping) { + asi_clear_pgd_range(asi, asi->pgd, addr, end, range->level); + + /* + * Remap all range we overlap with as mapping clearing + * will have unmap the overlap. + */ + overlap = range->overlap; + list_for_each_entry(r, &asi->mapping_list, list) { + if (!overlap) { + /* no more overlap */ + break; + } + + /* check overlap with other range */ + map_addr = ASI_RANGE_MAP_ADDR(r); + map_end = ASI_RANGE_MAP_END(r); + if (end < map_addr || addr >= map_end) { + /* no overlap, continue */ + continue; + } + pr_debug("ASI %p: UNMAP %px/%lx/%d remaps %px/%lx/%d\n", + asi, range->ptr, range->size, range->level, + r->ptr, r->size, r->level); + overlap--; + + r_addr = (unsigned long)r->ptr; + err = asi_copy_pgd_range(asi, asi->pgd, + current->mm->pgd, + r_addr, r_addr + r->size, + r->level); + if (err) { + pr_debug("ASI %p: UNMAP %px/%lx/%d remaps %px/%lx/%d error %d\n", + asi, range->ptr, range->size, + range->level, + r->ptr, r->size, r->level, + err); + } + } + } +} + /* * Clear page table entries in the specified ASI page-table. */ @@ -797,8 +991,8 @@ void asi_unmap(struct asi *asi, void *ptr) end = addr + range_mapping->size; pr_debug("ASI %p: UNMAP %px/%lx/%d\n", asi, ptr, range_mapping->size, range_mapping->level); - asi_clear_pgd_range(asi, asi->pgd, addr, end, range_mapping->level); list_del(&range_mapping->list); + asi_unmap_overlap(asi, range_mapping); kfree(range_mapping); done: spin_unlock_irqrestore(&asi->lock, flags); From patchwork Thu Jul 11 14:25:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1677614E5 for ; Thu, 11 Jul 2019 14:27:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0BFE28A5A for ; Thu, 11 Jul 2019 14:27:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E504928AE0; Thu, 11 Jul 2019 14:27:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D83CB28A5A for ; Thu, 11 Jul 2019 14:27:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D54E48E00D3; Thu, 11 Jul 2019 10:26:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CDD678E00C4; Thu, 11 Jul 2019 10:26:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE0F28E00D3; Thu, 11 Jul 2019 10:26:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) by kanga.kvack.org (Postfix) with ESMTP id 83F5B8E00C4 for ; Thu, 11 Jul 2019 10:26:51 -0400 (EDT) Received: by mail-io1-f70.google.com with SMTP id r27so6956401iob.14 for ; Thu, 11 Jul 2019 07:26:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=tktXu8oSlvrRNi4oCpkMHKHtnQ30RjQ+vGHk9xchmuI=; b=NmgiiCVDY/CXc+dqNO6k1DiChJ0MHMxHRc7at+5fqJSzftUN0CrgBF4DEkJVsSF298 XtKoxfw5k8rwsQ2Ufc6oMkhxdTggDoTdFNO2cl/yHatSiLg6xMGMsXG1tVzgwzLjxxrb ibNDZUUfBklTKiF4KEbKhfznTwk5J3RGObripr5z+sNMEZRKBRG0Ch3pQBNIkj4OMupx DZrLLyaSLvTwmnaOlU2eBR5ZCXnsMcMucvLsaHl/CQ0ym2D3QgRhCkvIbPG0wMoQ8WNk J4Z0dJvmG12QQ2kCt+F4Mt8gh4RthoaeagmkzsWtEVsN9prgLvKkjfa2bh6nRefkR05v af+w== X-Gm-Message-State: APjAAAXBuJZhaScE++KwKhk8zF88mNQ0ml8NdYBeRLICDyo8c2Ynh8G5 fZdU9R/en1N6GKdD46jva/UHgZ3pvtiyWbQ0tEtBni9bThj2YVaormLDdeKg3G2PjEmTgpQDAzY IsiNrQhanVG2OChk85FSOofZ/hui4Jve+IU453PYwbjjvCZ2Lao1xcCW08o7KVGbcYQ== X-Received: by 2002:a6b:5a17:: with SMTP id o23mr1380557iob.41.1562855211306; Thu, 11 Jul 2019 07:26:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqxefB5i5iJd9boZTrhUTRwU6ho7S2GkDPgbrT5ZNEHRzHnsfrSBthvIOXHU8vwyja8bWJZL X-Received: by 2002:a6b:5a17:: with SMTP id o23mr1380480iob.41.1562855210497; Thu, 11 Jul 2019 07:26:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855210; cv=none; d=google.com; s=arc-20160816; b=Qb3cmQvau/o/7bxXksz+8JmUdv7AQ8fYXWtYWFI/aIxUxsGwH7AChvEyEbivxvOXBi Of/VQw71rnlZ3s7BLrEAqjeizsojim8YAKQOj3xDUpBKPK7mdtFveCbcgDJe192mGBkm eAiVoqGxfU9BG6cUudyCFh3I/6O2B4i8tQ2I5mNmPUgZmXtyNEeNjPL7ZH6uo7gD4Cof 70wEM2Use8qEJjnCYJgGdgAs2xGELiAbUxWW2kt+Ku5tkCPPfgpWOVPAEGMhf9GVIg9S OHRelHEZizMZVNsUqvfryw1ahwuOnb+4qhMsJY/NWWlYVqpFyNc0Vp04ioHr45C8EguI +spw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tktXu8oSlvrRNi4oCpkMHKHtnQ30RjQ+vGHk9xchmuI=; b=a5AG9S6RrIyOq01iYu9ueH75xIn/v77hIXPG1b/AGVH27Wz9TuSlx7JtepchLBMKue qkxgD0hUvMAH3/xz1YSDh1oEx+lY1SHqzwJ+IRSymZahtjKxoyIP2haPLAFLTq65g7A2 tsOKIV5wN2CyEQdxOVq1y7bkhFD0rPSB6HDVFogiSN7AIV+eAsCvBzGQg2cj5dLprL2O AoJvelA+PQq9mx1VZIfQwrp2AyTK/ZVOQqi5izqVn0dAbZCOr8PYrVA8DfOg8Sgx78ig kALaEvfXTfGP7Uaki71viAcesSIirk1xN3fmQN0snNueD5uC2RyXg5cnW++NvDFr5qCB +PcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Xw8imofO; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id z9si9291957iof.139.2019.07.11.07.26.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:50 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Xw8imofO; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO7gP100410; Thu, 11 Jul 2019 14:26:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=tktXu8oSlvrRNi4oCpkMHKHtnQ30RjQ+vGHk9xchmuI=; b=Xw8imofO8fKNfBvyII53RRKIWPHJ/UdjgH1ZSaBocnV0EADrjb1GTZYN4HsE7xrVPRc/ TEHR2f4ozNEvHmT4xkzmrRWShe8NJkdGTGU5UyvTwwVLuobGplkQYt22venKGQfjZsCi BuPNZ1Zy1I5Lm3q4cyPwf0rvVmiPcVoh9A/tJ1V5PFUsSDajeMDtZazluCVycnUMAv6u /YyGN39NkO9rtwovlVRiIRORlVYEoHIigSvvQZaOjXC7B6+jVAU9bBj+zUDnf/Rei8Mf yGpCGEFGnOwzX/ATnI3MWljtUlcXz+wMpxZcI3AhvYUC5PX/005SgrPaW6hQhMYqVhz6 ew== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0cat-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:39 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu8021444; Thu, 11 Jul 2019 14:26:31 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 15/26] mm/asi: Initialize the ASI page-table with core mappings Date: Thu, 11 Jul 2019 16:25:27 +0200 Message-Id: <1562855138-19507-16-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Core mappings are the minimal mappings we need to be able to enter isolation and handle an isolation abort or exit. This includes the kernel code, the GDT and the percpu ASI sessions. We also need a stack so we map the current stack when entering isolation and unmap it on exit/abort. Optionally, additional mappins can be added like the stack canary or the percpu offset to be able to use get_cpu_var()/this_cpu_ptr() when isolation is active. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 9 ++++- arch/x86/mm/asi.c | 75 +++++++++++++++++++++++++++++++++++++++--- arch/x86/mm/asi_pagetable.c | 30 ++++++++++++---- 3 files changed, 99 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index cf5d198..1ac8fd3 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -11,6 +11,13 @@ #include #include +/* + * asi_create() map flags. Flags are used to map optional data + * when creating an ASI. + */ +#define ASI_MAP_STACK_CANARY 0x01 /* map stack canary */ +#define ASI_MAP_CPU_PTR 0x02 /* for get_cpu_var()/this_cpu_ptr() */ + enum page_table_level { PGT_LEVEL_PTE, PGT_LEVEL_PMD, @@ -73,7 +80,7 @@ struct asi_session { void asi_init_range_mapping(struct asi *asi); void asi_fini_range_mapping(struct asi *asi); -extern struct asi *asi_create(void); +extern struct asi *asi_create(int map_flags); extern void asi_destroy(struct asi *asi); extern int asi_enter(struct asi *asi); extern void asi_exit(struct asi *asi); diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 25633a6..f049438 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -19,6 +19,17 @@ /* ASI sessions, one per cpu */ DEFINE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); +struct asi_map_option { + int flag; + void *ptr; + size_t size; +}; + +struct asi_map_option asi_map_percpu_options[] = { + { ASI_MAP_STACK_CANARY, &fixed_percpu_data, sizeof(fixed_percpu_data) }, + { ASI_MAP_CPU_PTR, &this_cpu_off, sizeof(this_cpu_off) }, +}; + static void asi_log_fault(struct asi *asi, struct pt_regs *regs, unsigned long error_code, unsigned long address) { @@ -85,16 +96,55 @@ bool asi_fault(struct pt_regs *regs, unsigned long error_code, return true; } -static int asi_init_mapping(struct asi *asi) +static int asi_init_mapping(struct asi *asi, int flags) { + struct asi_map_option *option; + int i, err; + + /* + * Map the kernel. + * + * XXX We should check if we can map only kernel text, i.e. map with + * size = _etext - _text + */ + err = asi_map(asi, (void *)__START_KERNEL_map, KERNEL_IMAGE_SIZE); + if (err) + return err; + /* - * TODO: Populate the ASI page-table with minimal mappings so - * that we can at least enter isolation and abort. + * Map the cpu_entry_area because we need the GDT to be mapped. + * Not sure we need anything else from cpu_entry_area. */ + err = asi_map_range(asi, (void *)CPU_ENTRY_AREA_PER_CPU, P4D_SIZE, + PGT_LEVEL_P4D); + if (err) + return err; + + /* + * Map the percpu ASI sessions. This is used by interrupt handlers + * to figure out if we have entered isolation and switch back to + * the kernel address space. + */ + err = ASI_MAP_CPUVAR(asi, cpu_asi_session); + if (err) + return err; + + /* + * Optional percpu mappings. + */ + for (i = 0; i < ARRAY_SIZE(asi_map_percpu_options); i++) { + option = &asi_map_percpu_options[i]; + if (flags & option->flag) { + err = asi_map_percpu(asi, option->ptr, option->size); + if (err) + return err; + } + } + return 0; } -struct asi *asi_create(void) +struct asi *asi_create(int map_flags) { struct page *page; struct asi *asi; @@ -115,7 +165,7 @@ struct asi *asi_create(void) spin_lock_init(&asi->fault_lock); asi_init_backend(asi); - err = asi_init_mapping(asi); + err = asi_init_mapping(asi, map_flags); if (err) goto error; @@ -159,6 +209,7 @@ int asi_enter(struct asi *asi) struct asi *current_asi; struct asi_session *asi_session; unsigned long original_cr3; + int err; state = this_cpu_read(cpu_asi_session.state); /* @@ -190,6 +241,13 @@ int asi_enter(struct asi *asi) WARN_ON(asi_session->abort_depth > 0); /* + * We need a stack to run with isolation, so map the current stack. + */ + err = asi_map(asi, current->stack, PAGE_SIZE << THREAD_SIZE_ORDER); + if (err) + goto err_clear_asi; + + /* * Instructions ordering is important here because we should be * able to deal with any interrupt/exception which will abort * the isolation and restore CR3 to its original value: @@ -211,7 +269,7 @@ int asi_enter(struct asi *asi) if (!original_cr3) { WARN_ON(1); err = -EINVAL; - goto err_clear_asi; + goto err_unmap_stack; } asi_session->original_cr3 = original_cr3; @@ -228,6 +286,8 @@ int asi_enter(struct asi *asi) return 0; +err_unmap_stack: + asi_unmap(asi, current->stack); err_clear_asi: asi_session->asi = NULL; asi_session->task = NULL; @@ -284,6 +344,9 @@ void asi_exit(struct asi *asi) * exit isolation before abort_depth reaches 0. */ asi_session->abort_depth = 0; + + /* unmap stack */ + asi_unmap(asi, current->stack); } EXPORT_SYMBOL(asi_exit); diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index f1ee65b..bcc95f2 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -710,12 +710,20 @@ int asi_map_range(struct asi *asi, void *ptr, size_t size, map_addr = round_down(addr, page_dir_size); map_end = round_up(end, page_dir_size); - pr_debug("ASI %p: MAP %px/%lx/%d -> %lx-%lx\n", asi, ptr, size, level, - map_addr, map_end); - if (map_addr < addr) - pr_debug("ASI %p: MAP LEAK %lx-%lx\n", asi, map_addr, addr); - if (map_end > end) - pr_debug("ASI %p: MAP LEAK %lx-%lx\n", asi, end, map_end); + /* + * Don't log info the current stack because it is mapped/unmapped + * everytime we enter/exit isolation. + */ + if (ptr != current->stack) { + pr_debug("ASI %p: MAP %px/%lx/%d -> %lx-%lx\n", + asi, ptr, size, level, map_addr, map_end); + if (map_addr < addr) + pr_debug("ASI %p: MAP LEAK %lx-%lx\n", + asi, map_addr, addr); + if (map_end > end) + pr_debug("ASI %p: MAP LEAK %lx-%lx\n", + asi, end, map_end); + } spin_lock_irqsave(&asi->lock, flags); @@ -989,8 +997,14 @@ void asi_unmap(struct asi *asi, void *ptr) addr = (unsigned long)range_mapping->ptr; end = addr + range_mapping->size; - pr_debug("ASI %p: UNMAP %px/%lx/%d\n", asi, ptr, - range_mapping->size, range_mapping->level); + /* + * Don't log info the current stack because it is mapped/unmapped + * everytime we enter/exit isolation. + */ + if (ptr != current->stack) { + pr_debug("ASI %p: UNMAP %px/%lx/%d\n", asi, ptr, + range_mapping->size, range_mapping->level); + } list_del(&range_mapping->list); asi_unmap_overlap(asi, range_mapping); kfree(range_mapping); From patchwork Thu Jul 11 14:25:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E95B9112C for ; Thu, 11 Jul 2019 14:27:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8D6028A5A for ; Thu, 11 Jul 2019 14:27:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC6EB28AD1; Thu, 11 Jul 2019 14:27:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6DC128A5A for ; Thu, 11 Jul 2019 14:27:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84BC38E00D2; Thu, 11 Jul 2019 10:26:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7D87B8E00C4; Thu, 11 Jul 2019 10:26:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 650008E00D2; Thu, 11 Jul 2019 10:26:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 3FAE08E00C4 for ; Thu, 11 Jul 2019 10:26:49 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id u25so6882444iol.23 for ; Thu, 11 Jul 2019 07:26:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=k/9Fug63afNC647KTGwGwhcj7r27BkaO1PdUgfdBHl8=; b=YXAhNpp9YS/+TdqYb5dFYJ9kUruCsihh3CvGHL1pk63VYK0e7Pkra02S0iqjNIkmeL qTM8PV8cY0B6tlk3qtYa1HA77fOo4oGBM0jzKjYiDC8WCrLvi7mb/In12Ybzd3zTqRqP L8GqwRa84IdZoriIRswOxhK6KPdxnQB5kBE72aaqTod1XVjQK375hGT69aeucOT9mexI dPvPUZ6zlWYyTVdkaXGBpBaFXzBllNG/kJPyrCPnJ+N71kMzXVULCY9uIM66h4JYx3c4 CopKmagIUAat8yIpPwfX8tZdVRWZwH1JZ/YCM6WbIw5hqfAUecXhrYlgfaJpfI2SmIEY V/4g== X-Gm-Message-State: APjAAAVFO4i7/qjoANtcBoF96e3Lbuuyykqp4IbOVENuVG6vLR9jeuot pBodtU0CXhsyDqFhzBZ1fidM4jscNtYPNhyfGj4Dg/Zt+2NNshfn4P27eErq8bAve2FR1a/nR1m P5w9J9JOwkqvjrj+3ZtX4nNo83+NKuCJlzzpp1f0VSlj9Ad866v1OsViKOetMhB687A== X-Received: by 2002:a02:bb08:: with SMTP id y8mr4883350jan.51.1562855208991; Thu, 11 Jul 2019 07:26:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqxG5VIsAmzjMukIltJi53jT6H8wyBOi95FPvSbiVkL5j6sBvH6rKtwftuiMSoRZ/RmMsBEx X-Received: by 2002:a02:bb08:: with SMTP id y8mr4883278jan.51.1562855208269; Thu, 11 Jul 2019 07:26:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855208; cv=none; d=google.com; s=arc-20160816; b=WOyMUsXtEfoZ/qFvRPWB26Sq5lYeAWdqCSWi4u1GrtWQ5zoDS5ZzahAfHmDWafAUoB Navq+fZUtef4JVhJmV48TuJz28LIbuM+buCmx+MPbzgZ9qq0MFcprxorBWz8ZlDg6VuV XZJ/fbuO2g3VKZna0a1bTkXlOHMnpRI31rdVf4cVZYaVwtnECCQRylRXG2Nh2vUUpq4o Sh2Ni8nbDOUesIAO/2fOI6YJQlY8z2W+fPD0tbWTcs2/QuANCQwvwexHLl4vr+WVg3NX uXFI2MZ00one140y4ixUf8ashlf1WyCw3sKD/dcPFpw1fFqq9TsLLaCSM4Midu7xT0xA 4StA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=k/9Fug63afNC647KTGwGwhcj7r27BkaO1PdUgfdBHl8=; b=JnqBDikoqJ00ZfueSsTEWiCC9Yqro96zP44evW/llY/f72gz5YV1Y8k9er2f52hoEZ Chqw8YSrvx1TsC/xMczc3QLiLgVnyZu2ZpNXwEuZvjHFzH+S7r3fYJRw86TELj3norJp NNsDsq9MvJBoPvj9eKmNRMhbIzIYFLiefwSDZ8U0uPR/DJqdCmy/HwXJ5Xh8Jzmv8FAm 6q/TldkRpLghxG9JCcWfw0zCzica1THDUh9nHtXqlOlbjiybC2VrXT5pvFVatbW48KVS YcU1ko8dMqcYqFMt9vkDpNEg2p9Ku6KSlP7DH9hk5JckYTeGwOd/tSOFfez7qCsDbpeS w7qQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=5KLn6vAl; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id p7si8150221iob.149.2019.07.11.07.26.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:48 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=5KLn6vAl; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO90p001480; Thu, 11 Jul 2019 14:26:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=k/9Fug63afNC647KTGwGwhcj7r27BkaO1PdUgfdBHl8=; b=5KLn6vAlS8mqB4CpSvgIU+Q+dh/FIPZ30597ShKunxhzPyWufNzv9Z2Cx08UVoBiEll/ JBVMx/vKSYB0C0s+WVyLcIBN58ZXRTbwEhf/IwNYl7zZ7sSsGPjmF/6hyaDRwuF9hk/r Od3KI8aMoJde526KmMFcX2swAUblIJ6IEJr7eaGDmgCNG3lLZyVRqS3W4XYV29OC/ZVc 4mk68QurguhhWeX8FUCmmK47mMaaLROKzVLzTFajITGGIClxoo3c+fKTYG1lt2Dpitky k9uYwYmfAA4lryCWr9Zg3nU9L0blXjaO/G+ZtIQSyB+mD3low+GPiTf4kzuLZVh/PRPA cg== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e0w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:38 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcu9021444; Thu, 11 Jul 2019 14:26:34 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 16/26] mm/asi: Option to map current task into ASI Date: Thu, 11 Jul 2019 16:25:28 +0200 Message-Id: <1562855138-19507-17-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=853 adultscore=26 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add an option to map the current task into an ASI page-table. The task is mapped when entering isolation and unmapped on abort/exit. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 2 ++ arch/x86/mm/asi.c | 25 +++++++++++++++++++++---- arch/x86/mm/asi_pagetable.c | 4 ++-- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 1ac8fd3..a277e43 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -17,6 +17,7 @@ */ #define ASI_MAP_STACK_CANARY 0x01 /* map stack canary */ #define ASI_MAP_CPU_PTR 0x02 /* for get_cpu_var()/this_cpu_ptr() */ +#define ASI_MAP_CURRENT_TASK 0x04 /* map the current task */ enum page_table_level { PGT_LEVEL_PTE, @@ -31,6 +32,7 @@ enum page_table_level { struct asi { spinlock_t lock; /* protect all attributes */ pgd_t *pgd; /* ASI page-table */ + int mapping_flags; /* map flags */ struct list_head mapping_list; /* list of VA range mapping */ /* diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index f049438..acd1135 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -28,6 +28,7 @@ struct asi_map_option { struct asi_map_option asi_map_percpu_options[] = { { ASI_MAP_STACK_CANARY, &fixed_percpu_data, sizeof(fixed_percpu_data) }, { ASI_MAP_CPU_PTR, &this_cpu_off, sizeof(this_cpu_off) }, + { ASI_MAP_CURRENT_TASK, ¤t_task, sizeof(current_task) }, }; static void asi_log_fault(struct asi *asi, struct pt_regs *regs, @@ -96,8 +97,9 @@ bool asi_fault(struct pt_regs *regs, unsigned long error_code, return true; } -static int asi_init_mapping(struct asi *asi, int flags) +static int asi_init_mapping(struct asi *asi) { + int flags = asi->mapping_flags; struct asi_map_option *option; int i, err; @@ -164,8 +166,9 @@ struct asi *asi_create(int map_flags) spin_lock_init(&asi->lock); spin_lock_init(&asi->fault_lock); asi_init_backend(asi); + asi->mapping_flags = map_flags; - err = asi_init_mapping(asi, map_flags); + err = asi_init_mapping(asi); if (err) goto error; @@ -248,6 +251,15 @@ int asi_enter(struct asi *asi) goto err_clear_asi; /* + * Optionally, also map the current task. + */ + if (asi->mapping_flags & ASI_MAP_CURRENT_TASK) { + err = asi_map(asi, current, sizeof(struct task_struct)); + if (err) + goto err_unmap_stack; + } + + /* * Instructions ordering is important here because we should be * able to deal with any interrupt/exception which will abort * the isolation and restore CR3 to its original value: @@ -269,7 +281,7 @@ int asi_enter(struct asi *asi) if (!original_cr3) { WARN_ON(1); err = -EINVAL; - goto err_unmap_stack; + goto err_unmap_task; } asi_session->original_cr3 = original_cr3; @@ -286,6 +298,9 @@ int asi_enter(struct asi *asi) return 0; +err_unmap_task: + if (asi->mapping_flags & ASI_MAP_CURRENT_TASK) + asi_unmap(asi, current); err_unmap_stack: asi_unmap(asi, current->stack); err_clear_asi: @@ -345,8 +360,10 @@ void asi_exit(struct asi *asi) */ asi_session->abort_depth = 0; - /* unmap stack */ + /* unmap stack and task */ asi_unmap(asi, current->stack); + if (asi->mapping_flags & ASI_MAP_CURRENT_TASK) + asi_unmap(asi, current); } EXPORT_SYMBOL(asi_exit); diff --git a/arch/x86/mm/asi_pagetable.c b/arch/x86/mm/asi_pagetable.c index bcc95f2..8076626 100644 --- a/arch/x86/mm/asi_pagetable.c +++ b/arch/x86/mm/asi_pagetable.c @@ -714,7 +714,7 @@ int asi_map_range(struct asi *asi, void *ptr, size_t size, * Don't log info the current stack because it is mapped/unmapped * everytime we enter/exit isolation. */ - if (ptr != current->stack) { + if (ptr != current->stack && ptr != current) { pr_debug("ASI %p: MAP %px/%lx/%d -> %lx-%lx\n", asi, ptr, size, level, map_addr, map_end); if (map_addr < addr) @@ -1001,7 +1001,7 @@ void asi_unmap(struct asi *asi, void *ptr) * Don't log info the current stack because it is mapped/unmapped * everytime we enter/exit isolation. */ - if (ptr != current->stack) { + if (ptr != current->stack && ptr != current) { pr_debug("ASI %p: UNMAP %px/%lx/%d\n", asi, ptr, range_mapping->size, range_mapping->level); } From patchwork Thu Jul 11 14:25:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AB27112C for ; Thu, 11 Jul 2019 14:27:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38A4E28AC8 for ; Thu, 11 Jul 2019 14:27:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2939828ADC; Thu, 11 Jul 2019 14:27:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E7E328AC8 for ; Thu, 11 Jul 2019 14:27:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16E108E00D4; Thu, 11 Jul 2019 10:26:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0858B8E00C4; Thu, 11 Jul 2019 10:26:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3D3C8E00D4; Thu, 11 Jul 2019 10:26:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f71.google.com (mail-io1-f71.google.com [209.85.166.71]) by kanga.kvack.org (Postfix) with ESMTP id BF3BF8E00C4 for ; Thu, 11 Jul 2019 10:26:52 -0400 (EDT) Received: by mail-io1-f71.google.com with SMTP id q26so6963234ioi.10 for ; Thu, 11 Jul 2019 07:26:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=DxKvEFI9OqvnwAFHfsS7VJpKVi8ynRLOZoMuRYswZIw=; b=hzS0W9kl8H4W0BIbCihrcL6W7rq1a5XEo9L1trTWuyzjBWRUzF/MLAfkyAx0kz/SmT I5cHdFW9FlaIsGqaVl5KxYpohwkJ5iMaTIqdwgX6dv3kRPzqjeGCjxXHF3npRoTLUqXv dLIYE5lLvzooad7la6aSwHwjHQ2oRpNPu1MXldGP78cICJWrfiwtudKyotrQAcN6iiiq +XuguJF+TjaTwHVFDycwGlKJXjUZyvQQEshFZX8AQe13z2FYleLsbK2KtsLsba75a/0C P4C0Ehg6L8X7qhGuSy4DHraj4bShN7RRmGsDRRNfwN39OeVVysx3jFmeLyzP/fyJIfG+ RGdQ== X-Gm-Message-State: APjAAAVEMglRZrqgRQp74NFBzeGqqTuNYHLETW6E4xTCKAy7LyXaHomA jnXsRC0VcdyYNy9DGxbWiy25H1/KRHfJxg/D3CT4XR5QLYoGD851tW85RL0AlrV2Ly4J6pAXaSD 8+/4NZtYFolZ+jRMkJrpQWf21ORA+D3X4BTv3BWz5Wc9rS5QGGLlsls5e3JrDAaiU6w== X-Received: by 2002:a5d:88c6:: with SMTP id i6mr4767228iol.107.1562855212566; Thu, 11 Jul 2019 07:26:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqyDWC9i/2FMckqlkPq+wiimV4/zWTsZppuFE1stiXkUlvZh91qHeDJUP/UM3Fbj89EgWJ2T X-Received: by 2002:a5d:88c6:: with SMTP id i6mr4767166iol.107.1562855211889; Thu, 11 Jul 2019 07:26:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855211; cv=none; d=google.com; s=arc-20160816; b=wtLwY0AABEwrBCYgZHGTpcTvnJ1Q2tzh/23EgKq7Q40KoRNTVY3PglZnUrr01F98lX G9ulilWzzqE1QoHYwVTjeqeNKuM3Si/zzxYshylmoEU4aXI5CDPnbFTBH5SJs7FwnU25 sythw8h+GuHt/1pfTH3BE/N1nAEUIgn8HTMdM9LC69p0CujsF4Se7zsMJV/qj0t6R8bH R42aGrGWRE2YK40Up0qOLUnO3k3CPMCs/BVgwZAZ5Cjwu/8SoyUiMUvZ6vvbviOrC42D GLQEtzEp5ocNsjdCyvMlMVyLwIqYTy9p2Y7l3LhXcz/Wo+6dGGOXIzXHMP1gRsM2Vg7i DKuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DxKvEFI9OqvnwAFHfsS7VJpKVi8ynRLOZoMuRYswZIw=; b=ihmbYtISGMoPXsjd766ivTBPxuWx4CVRkasa0Fby4XZ2TV+8bJy0tpxwggf39ftLCe 69A9eB9xPtMTXYoS0HlbxfRj6MaQkWYOW49zM0mfGCbWs0MOOLce+ySRSjZDgWymE87M zMWvNxL5zHicx8tdcKsHkW71zpXyLzh9OU5eRc3PdwZeZRyG317MeJF6Sq+OS3RuRpk1 RROd2PCcLlJWxsiJKr3hwKuBbQ9nVCPeakTukFzD1X2lnX53kh3O7nkfLSCCIAVRZe/z Wa/0GUC6T+ynS44ITzuIFBh/mI50Vp6+hWGM5EMIi1/UmvzA2YhLxZo0hJXsSL/ka9eX QLjg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=k3XXPihT; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id n123si9762546iod.129.2019.07.11.07.26.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:51 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=k3XXPihT; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO75F013247; Thu, 11 Jul 2019 14:26:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=DxKvEFI9OqvnwAFHfsS7VJpKVi8ynRLOZoMuRYswZIw=; b=k3XXPihT1uvy+OLIiRmWMdfDXIT+WHMFAjj45U6uCpQ8rV2jfdSnRJmMQA1mV5ALTQyT PERU4/YPhFf4ckQAGG2rOQDm5vbl52OROkwzviVZDqxaVQLQCpGKejd6n6Gi8g/OW973 b1WRq4uNsMv9cAwP/H6vRscq1e3iobbZ324Tq6PgoCdWFemJ/1YPJhNIMn+keJv5JOQh Ff8q9kmaJ7pYkwC9amWnZvecFXs6OmpibtOtjhm23oXx9IX6dzIXv0YiCsY7b8cavCtn S3SvjejyJfLMAMiB7WwpFazOkLIoHuippfiDTsTbnPYepjTPVENPQMOCjagrpefgERlp TA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2tjm9r0brp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:41 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuA021444; Thu, 11 Jul 2019 14:26:37 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 17/26] rcu: Move tree.h static forward declarations to tree.c Date: Thu, 11 Jul 2019 16:25:29 +0200 Message-Id: <1562855138-19507-18-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP tree.h has static forward declarations for inline function declared in tree_plugin.h and tree_stall.h. These forward declarations prevent including tree.h into a file different from tree.c Signed-off-by: Alexandre Chartre --- kernel/rcu/tree.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++ kernel/rcu/tree.h | 55 +---------------------------------------------------- 2 files changed, 55 insertions(+), 54 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 980ca3c..44dd3b4 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -55,6 +55,60 @@ #include "tree.h" #include "rcu.h" +/* Forward declarations for tree_plugin.h */ +static void rcu_bootup_announce(void); +static void rcu_qs(void); +static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp); +#ifdef CONFIG_HOTPLUG_CPU +static bool rcu_preempt_has_tasks(struct rcu_node *rnp); +#endif /* #ifdef CONFIG_HOTPLUG_CPU */ +static int rcu_print_task_exp_stall(struct rcu_node *rnp); +static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp); +static void rcu_flavor_sched_clock_irq(int user); +static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck); +static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags); +static void rcu_preempt_boost_start_gp(struct rcu_node *rnp); +static void invoke_rcu_callbacks_kthread(void); +static bool rcu_is_callbacks_kthread(void); +static void __init rcu_spawn_boost_kthreads(void); +static void rcu_prepare_kthreads(int cpu); +static void rcu_cleanup_after_idle(void); +static void rcu_prepare_for_idle(void); +static bool rcu_preempt_has_tasks(struct rcu_node *rnp); +static bool rcu_preempt_need_deferred_qs(struct task_struct *t); +static void rcu_preempt_deferred_qs(struct task_struct *t); +static void zero_cpu_stall_ticks(struct rcu_data *rdp); +static bool rcu_nocb_cpu_needs_barrier(int cpu); +static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); +static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); +static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp, + bool lazy, unsigned long flags); +static bool rcu_nocb_adopt_orphan_cbs(struct rcu_data *my_rdp, + struct rcu_data *rdp, + unsigned long flags); +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp); +static void do_nocb_deferred_wakeup(struct rcu_data *rdp); +static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); +static void rcu_spawn_cpu_nocb_kthread(int cpu); +static void __init rcu_spawn_nocb_kthreads(void); +#ifdef CONFIG_RCU_NOCB_CPU +static void __init rcu_organize_nocb_kthreads(void); +#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ +static bool init_nocb_callback_list(struct rcu_data *rdp); +static unsigned long rcu_get_n_cbs_nocb_cpu(struct rcu_data *rdp); +static void rcu_bind_gp_kthread(void); +static bool rcu_nohz_full_cpu(void); +static void rcu_dynticks_task_enter(void); +static void rcu_dynticks_task_exit(void); + +/* Forward declarations for tree_stall.h */ +static void record_gp_stall_check_time(void); +static void rcu_iw_handler(struct irq_work *iwp); +static void check_cpu_stall(struct rcu_data *rdp); +static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp, + const unsigned long gpssdelay); + #ifdef MODULE_PARAM_PREFIX #undef MODULE_PARAM_PREFIX #endif diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e253d11..9790b58 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -392,58 +392,5 @@ struct rcu_state { #endif /* #else #ifdef CONFIG_TRACING */ int rcu_dynticks_snap(struct rcu_data *rdp); - -/* Forward declarations for tree_plugin.h */ -static void rcu_bootup_announce(void); -static void rcu_qs(void); -static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp); -#ifdef CONFIG_HOTPLUG_CPU -static bool rcu_preempt_has_tasks(struct rcu_node *rnp); -#endif /* #ifdef CONFIG_HOTPLUG_CPU */ -static int rcu_print_task_exp_stall(struct rcu_node *rnp); -static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp); -static void rcu_flavor_sched_clock_irq(int user); void call_rcu(struct rcu_head *head, rcu_callback_t func); -static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck); -static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags); -static void rcu_preempt_boost_start_gp(struct rcu_node *rnp); -static void invoke_rcu_callbacks_kthread(void); -static bool rcu_is_callbacks_kthread(void); -static void __init rcu_spawn_boost_kthreads(void); -static void rcu_prepare_kthreads(int cpu); -static void rcu_cleanup_after_idle(void); -static void rcu_prepare_for_idle(void); -static bool rcu_preempt_has_tasks(struct rcu_node *rnp); -static bool rcu_preempt_need_deferred_qs(struct task_struct *t); -static void rcu_preempt_deferred_qs(struct task_struct *t); -static void zero_cpu_stall_ticks(struct rcu_data *rdp); -static bool rcu_nocb_cpu_needs_barrier(int cpu); -static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); -static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); -static void rcu_init_one_nocb(struct rcu_node *rnp); -static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp, - bool lazy, unsigned long flags); -static bool rcu_nocb_adopt_orphan_cbs(struct rcu_data *my_rdp, - struct rcu_data *rdp, - unsigned long flags); -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp); -static void do_nocb_deferred_wakeup(struct rcu_data *rdp); -static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); -static void rcu_spawn_cpu_nocb_kthread(int cpu); -static void __init rcu_spawn_nocb_kthreads(void); -#ifdef CONFIG_RCU_NOCB_CPU -static void __init rcu_organize_nocb_kthreads(void); -#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ -static bool init_nocb_callback_list(struct rcu_data *rdp); -static unsigned long rcu_get_n_cbs_nocb_cpu(struct rcu_data *rdp); -static void rcu_bind_gp_kthread(void); -static bool rcu_nohz_full_cpu(void); -static void rcu_dynticks_task_enter(void); -static void rcu_dynticks_task_exit(void); - -/* Forward declarations for tree_stall.h */ -static void record_gp_stall_check_time(void); -static void rcu_iw_handler(struct irq_work *iwp); -static void check_cpu_stall(struct rcu_data *rdp); -static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp, - const unsigned long gpssdelay); + From patchwork Thu Jul 11 14:25:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040103 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 799641395 for ; Thu, 11 Jul 2019 14:27:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05E5D28A5A for ; Thu, 11 Jul 2019 14:27:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE2C028AC8; Thu, 11 Jul 2019 14:27:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 771D728AD9 for ; Thu, 11 Jul 2019 14:27:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C92298E00D5; Thu, 11 Jul 2019 10:26:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BF5018E00C4; Thu, 11 Jul 2019 10:26:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6FAB8E00D5; Thu, 11 Jul 2019 10:26:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 7F76D8E00C4 for ; Thu, 11 Jul 2019 10:26:55 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id h3so6950446iob.20 for ; Thu, 11 Jul 2019 07:26:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=j5hNYpa6Ps4ofAY49xQlyVzas1I7FTDoeSXSCbctYus=; b=KaP++4x2o5IsXUxZJz/U+OjhpwJH51Oe35ILVu6HJhsBnc2jYUv5BAsBbzRfKWdE6Y aWmz/ue/Ppu0sm6U1NZr34OVjqlOzMuzgdHEkVY4h/9/SojJjdDn4JjvcStpiRlA/WTT 72SJTis9CKx4c1xMNslZXk5yAjmKHAIbcIjdNIuSFeGDO9TLp5Zz6+kjl5qK41pjolhH 9rue8y0hBoVxcgbx3eoAm7V3+Im8zGV55PcXJHqszTx9G7dyJm+bvccUlbVJMRW+yf84 LRCznUdS4TTbtP0Gptd6GkYm9pR10YJGIJT+e5p12ABElXBkdlqPX2IyZvLn1s7I1qqR +OfQ== X-Gm-Message-State: APjAAAVJGjHJf4Y+X66jrYZXKO1qT06BwR94tR7DjbG8bJHbBIM7KkY1 fVR+9DOH+87NiRWzDduRdHx+VH6nNDI4/mzrlkh4m4zJdYbq/u11u0QV4qIgpQhFOqBadTh+iGs H0ifEAGQfo3/1v6cdd392O625lXlbn+abuSuxapsvaUwpw3Q+mGMIbucqBO1svuPiAQ== X-Received: by 2002:a6b:f90f:: with SMTP id j15mr4571740iog.43.1562855215318; Thu, 11 Jul 2019 07:26:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqwvdo5d/WssBNUo3rE+k+Edg/hMx7olZ5D4MyMyXi80azhA2Fyagt9Kclr1PBsQabUKYUm8 X-Received: by 2002:a6b:f90f:: with SMTP id j15mr4571694iog.43.1562855214709; Thu, 11 Jul 2019 07:26:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855214; cv=none; d=google.com; s=arc-20160816; b=bn5HZ/yTsYmskjWRun+HTOiSJdm4OlviNgTI0aOyM/CvP7h3Zbfztr0hRi/KfwROpY 0H1KnLy7As2CJLON7MwUi0KHNk0PAWRStDQRonSxrdxmj8VW06nc4i10oZ2pqAbDrYLF kp9kdnxF0ij1c/ZPId3qSwEHxBRILLoR9L6UPE3l3bSl2ah8e81kp9GFn/lJdPjWBfGS poZhb/qoKPGFBT9VLUcsJUS2fm7OWx+Scdz17u24A8uWF3vzJ4pE5Ze4+GoBbtXrr8Go w9aDiGuU65vpaLTjkhSF8Zm8Sy6LyHLpfQZWkkLm63MfPDnNLUgFWe1fTM0dKcrKaY9L SisA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j5hNYpa6Ps4ofAY49xQlyVzas1I7FTDoeSXSCbctYus=; b=d/sWo3i0dE2G5FxY5+44gW6k3U2bgAb9Bo9U9cGhRpEcDXy+8XD3heJpjTCTtVr+yE 6nNIhPwyXcCtH0QavibcLZTAxCZVrI0Bta7yxVAxonZNjkVQKosvUDYzSlSy/UO73L7T OJTu8/T9yaW7JCNoUr2TtSUJbpbYOAcHVJATrdGi18YhqrO6EOGsM2RuwRbg2Nk3fho8 iCjWsHJjpKe8p1805a4SGPnIs8GbBcDp50qDdpX0t6cQJHAuBMEiijWGHiW/BAN00xy1 WY4i+froI5hp89Roi1/ZN8wkm8+TB8CZnFz2JXDXC1QJKVt+65eMIDfSCkhk6gXBtlg7 SoxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=IZOylVir; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id v3si7768588iot.133.2019.07.11.07.26.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:54 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=IZOylVir; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO8Kr001447; Thu, 11 Jul 2019 14:26:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=j5hNYpa6Ps4ofAY49xQlyVzas1I7FTDoeSXSCbctYus=; b=IZOylVirFWDOmZYSdGDrwFnJwm9CEHLxntE+KdhdKKVzqt1gSAfqZTTsVwqNsGzmUzgE PAj74VbhDOsq4BCutkFXYEtKMfHv6uDQN4EwPs/i8mhcZpVyCpdqZIKiMK3B0iniWbhc nCHcdwRFI9A6XWvL9fDPRjGkqCOObfWGxfS7XmG6Q9mdYcz4LapWwYkhkHzoQvWylY6M ysC+FaRK07kBZ2670+CaYM92TpCurpQy/wzZmTZFtJvwBPFzsFwFvWvwJU5DIkIsFNK3 zM5EMYBeH4z52ZPL8IQ1EjyFUcZnoz5RUscIoFGUEJ9sBLvHsPC7wF+/3bg5aRqA88yQ 5g== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e1m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:44 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuB021444; Thu, 11 Jul 2019 14:26:41 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 18/26] rcu: Make percpu rcu_data non-static Date: Thu, 11 Jul 2019 16:25:30 +0200 Message-Id: <1562855138-19507-19-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Make percpu rcu_data non-static so that it can be mapped into an isolation address space page-table. This will allow address space isolation to use RCU without faulting. Signed-off-by: Alexandre Chartre --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree.h | 1 + 2 files changed, 2 insertions(+), 1 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 44dd3b4..2827b2b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -126,7 +126,7 @@ static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp, #define rcu_eqs_special_exit() do { } while (0) #endif -static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { .dynticks_nesting = 1, .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, .dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR), diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 9790b58..a043fde 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -394,3 +394,4 @@ struct rcu_state { int rcu_dynticks_snap(struct rcu_data *rdp); void call_rcu(struct rcu_head *head, rcu_callback_t func); +DECLARE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data); From patchwork Thu Jul 11 14:25:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B81B814E5 for ; Thu, 11 Jul 2019 14:27:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4CE128A5A for ; Thu, 11 Jul 2019 14:27:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98BFB28AD1; Thu, 11 Jul 2019 14:27:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29F7C28AD9 for ; Thu, 11 Jul 2019 14:27:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC93C8E00D6; Thu, 11 Jul 2019 10:26:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D53A18E00C4; Thu, 11 Jul 2019 10:26:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF6148E00D6; Thu, 11 Jul 2019 10:26:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 8D1F28E00C4 for ; Thu, 11 Jul 2019 10:26:57 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id s83so6957571iod.13 for ; Thu, 11 Jul 2019 07:26:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=2d2Yv2y5zLW9fLZ4ZOO2tadSx6yS9UU2pD1uUkqP2I4=; b=S2koVFlXQLVeeIXDsSk+KOxPkmr7NVC/4/WNIiYcDhMeQ3MopJrwsgwjeQDADtDXxf esT8he/pA8KHS4XjSwr+2qpMwuHtoCvDib4jdyYEInK9swwqWgAqKb8fOBo8bK+f44ov dp2STZ+HffK8/ixjA2TTZnSZB9AEQpxWuWlwNqMq4bzAEvVLhJCvkqR3wlUYSfVYcUUA SVjmUiiVzGZgXARB+qFHYKFiGMCneCkaBe7mWD8setR+KCMYm60O1fkS2HGigFDUN5/R VzeP6gMlinxchl+8USaSfs+8t4RyTPMjN9JkRA8WT+XJGDwrz3jleKPfVR9d9tRybbxp Hc7Q== X-Gm-Message-State: APjAAAUCX16wau4ucfWsVBEEzg9nbKQ3/rI4vIqWZREWWYfB7R8SRUY8 +Y3Lx0MAT1pAZht2oEwp8fu2eL9zNdz4Is9D+OiXSoHy5t8ehFRgb4jKJ+GHJzYJcWhQhHnBhtl EaA20s5fHrjZPaSrbQXRZsgpPUHFnGczKickDjLuYrrKtDyKxDc1VNFj/ztQnAtIkzw== X-Received: by 2002:a02:c65a:: with SMTP id k26mr4946560jan.18.1562855217318; Thu, 11 Jul 2019 07:26:57 -0700 (PDT) X-Google-Smtp-Source: APXvYqwDjUgGt+CeojedCZJ6snqYdC29L32sFFsg7wreWm44hf7GEBmSUd3fW2Wzz6+5qisTGpwZ X-Received: by 2002:a02:c65a:: with SMTP id k26mr4946509jan.18.1562855216724; Thu, 11 Jul 2019 07:26:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855216; cv=none; d=google.com; s=arc-20160816; b=VdwC/nlViLFtdph8m957S7Wuac3KaLVeDNLLbIHTVJbVbYYEJx/SmtXY00JWcwuI0N mnVNO570/eqkAUlhiVEIN1eBkXhBs6s8q3zp+y2/VB61tOkWxvBYk8jOZQ9/wK+oQuyh KpSZGD0rfA9i1R1HsVw1BHtj+PUO8+Oqoo8yh2w5h8BQojV+MdHz7i5OBfY+faYe5xeq P+Ra4rrga4tpXqXOHZR4gyxt4tboV43FKcvWppqU1mryjJ4Peqgruz/0OKZmChpP+1/k wXRIZp6jpa4LOD9o7V4o7JEBz6JCd+bL+XHMASRC9BARCdSccYXXaenXWRH1G9sfKKIB EMFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2d2Yv2y5zLW9fLZ4ZOO2tadSx6yS9UU2pD1uUkqP2I4=; b=GbvKdnfH6A/ZYAsvwlXm80qJKjcn1g591lDoW0oYG4LXY/nOP9SxAvm99oxGzNV3J/ haMZS5XuRVE3BPdC+qdh6K9ZO57mmuLoOrWwe1LL1jELsW8J9K8tdAPSM7+ELFbQTsjX MMrY5UuYtySJ+OBd2Z28/INp8dhPVWfKZLFI3p1joyoKTiP59kDCtq4bR1+nGGA2pVIq YJAUonA0wKaHHJ41mLNF/N6EZ2kVZVpR7bkQhFrQxCK2nGSkuots9PpG/5KBhGX67Yye I5syywcAxi9XBD0qUB6j6haLM2jiXtHF7do7G9iw5jegztd7HOVskg7DDyubMEYqK9o5 nVgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=PEK1VY+3; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id j2si7727545iog.116.2019.07.11.07.26.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:26:56 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=PEK1VY+3; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO8vO001464; Thu, 11 Jul 2019 14:26:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=2d2Yv2y5zLW9fLZ4ZOO2tadSx6yS9UU2pD1uUkqP2I4=; b=PEK1VY+3UVMeYt+RB002bYINLbdCxWtdVaXXuLcUErNmdDVQeXpj0b9OMDpmCIqqZa4Y 9PvB0Lnb9b0uhxGmrMFXcR44+qa8LQlt1HNPX0Dv/WgAESNOrwyxaWcdUWdyw5aYvMo9 6hMEg+A/C1vRmGh2CVcZaeUmibNGRM+Z0ynXpbec7GOZxoWF9wbcV+Bk5OzwKInmLI0/ 2E7YCwLLpvnIvEy49dwDdr51/iqNr/GOk+NSxBuovD4OQOPclLfLK8Qy+EAEEUquuZ8L HADMT7wC0JoiunyhO1kAnDjpVZAnFUxpCSF1OfqJMoGYFOwX+PMYYSzDX1GbCZ5/xKmc PQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e24-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:48 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuC021444; Thu, 11 Jul 2019 14:26:44 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 19/26] mm/asi: Add option to map RCU data Date: Thu, 11 Jul 2019 16:25:31 +0200 Message-Id: <1562855138-19507-20-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add an option to map RCU data when creating an ASI. This will map the percpu rcu_data (which is not exported by the kernel), and allow ASI to use RCU without faulting. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 1 + arch/x86/mm/asi.c | 4 ++++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index a277e43..8199618 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -18,6 +18,7 @@ #define ASI_MAP_STACK_CANARY 0x01 /* map stack canary */ #define ASI_MAP_CPU_PTR 0x02 /* for get_cpu_var()/this_cpu_ptr() */ #define ASI_MAP_CURRENT_TASK 0x04 /* map the current task */ +#define ASI_MAP_RCU_DATA 0x08 /* map rcu data */ enum page_table_level { PGT_LEVEL_PTE, diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index acd1135..20c23dc 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -16,6 +17,8 @@ #include #include +#include "../../../kernel/rcu/tree.h" + /* ASI sessions, one per cpu */ DEFINE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); @@ -29,6 +32,7 @@ struct asi_map_option asi_map_percpu_options[] = { { ASI_MAP_STACK_CANARY, &fixed_percpu_data, sizeof(fixed_percpu_data) }, { ASI_MAP_CPU_PTR, &this_cpu_off, sizeof(this_cpu_off) }, { ASI_MAP_CURRENT_TASK, ¤t_task, sizeof(current_task) }, + { ASI_MAP_RCU_DATA, &rcu_data, sizeof(rcu_data) }, }; static void asi_log_fault(struct asi *asi, struct pt_regs *regs, From patchwork Thu Jul 11 14:25:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040107 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8859D112C for ; Thu, 11 Jul 2019 14:27:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7791028A5A for ; Thu, 11 Jul 2019 14:27:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6B01A28AD1; Thu, 11 Jul 2019 14:27:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED2AB28A5A for ; Thu, 11 Jul 2019 14:27:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BE448E00D7; Thu, 11 Jul 2019 10:27:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0FA8F8E00C4; Thu, 11 Jul 2019 10:27:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEE888E00D7; Thu, 11 Jul 2019 10:27:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id B688A8E00C4 for ; Thu, 11 Jul 2019 10:27:01 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id k21so6990681ioj.3 for ; Thu, 11 Jul 2019 07:27:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=9FGKxWOaR9lg8qKZvVxZYyBDHDZO4/U+GnNunz7EXSk=; b=lJoduAUbnMO/Y+o34z8MD/uFj6b4dikhGeNBTud29bhGHwS0YjxdRSw6w2Khf/srMU LIprHJquOQDhowcRC6li4io5ODtv5bhsG8fV1kckwjsKAqFeN66MEvUqd+pogiEiafqa 9Dz7UCh8d1LoLoCbiTYqySk0Geyng4d5NemTwZ2i2oOYalH2/ohsqc/IGA9E10KKiIJ9 eh+MpIGDa749bMnc3zsHjI83voAWSvgk8seTgxCZPO9HBJmn2+u5RlBmQfKV23BzpXPe TzSyn31mFGtEfyk2nfhplQepdlGh58uq8Qj0U5O71631+pO18ZLIuqbkkHqHXHDcx/3v s3Jw== X-Gm-Message-State: APjAAAVHWsgYI39TmDME4wEq/1wJfg1Tl7dlirgBnUtGRCF/vwWFlwAo Tw4xj/aVXNvybcyHOhemyYDUzmqEaYEopgnEI1e+dyyUsurE+VunX0Zkvl29Rp4KLwpmYKBdjBQ m29dUKyQDVPBnlw/M0WsAZkC2HIitC0oXjMR1WvBLpud8NWHWy5VjWAtM8GwSAn1Etg== X-Received: by 2002:a5d:87da:: with SMTP id q26mr4633698ios.193.1562855221551; Thu, 11 Jul 2019 07:27:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqx3vEubOZkD1MppYky2ndwi18FrQTB6UN1p5v/Wm/e8H7HSOXfnTZkTPI35qPRfQjVNZB9U X-Received: by 2002:a5d:87da:: with SMTP id q26mr4633624ios.193.1562855220681; Thu, 11 Jul 2019 07:27:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855220; cv=none; d=google.com; s=arc-20160816; b=Sk00lhrtIgRxgRFZhQxCzIl44RloMcRvXUX2mNjBpC6tG5Ys24rc86sAR8N7h//P1L L0cLkiVbrQbNzO4WYxzmXmvmO+BskSJSS3VBKjZkd/XVuOQJB1hRwfb1iuTk7WITQS2r YIDRKK4+zIfHnhvHvifv4eFXwXXrOs7/2YGoRw7Pevrm7JKO9H+DJQ0LYOOFUyx5YVWV Bb4CgnC82jYROOMoOQl+n6h/6GRHSv6jqTXu/MI+Sh6I7pd4pmdc7uy/wjXzHs6CtvQW N+Lc96D2813WzMBrnR1C6kEXXmlYxFmb8Hb3NaznOrwwMFVifcJo4kgtOWVgcUgoCDLo tbaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9FGKxWOaR9lg8qKZvVxZYyBDHDZO4/U+GnNunz7EXSk=; b=ZPjdSoZ9c4fVFhiN+KP5FW9VKEo1zys2pV3+djQ2IAFASHPXfKTv9TPLJ24yQ8iWU/ 6Q7ZQ+nLmyYCAfDgZCr0AoFIL1vlhCYAuMgbrctv8Ca2MS1XAcUeav56Zx+J4Bimsms/ j/QAZrEn+X+/Mi7GAfx5674wVTEIouCWZ1a073QBKlYDFbf2pOXgNmraOnMUbAVxiSWo OBD1tw8mBGko3Wua/wSyG72ugNy/dMicqh9IFxVL3+srU9zMI+z/wHSFcqddHMIVA2vg rVEit7y8SIOsJ/vfJzp1ChxOwlGOPEKcqjPg7yRtwil1xOxb5kzTkV7lVSGrXYabUOtC dr3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=lniqBkC9; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id a16si8273870ioo.7.2019.07.11.07.27.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:00 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=lniqBkC9; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOMdI013631; Thu, 11 Jul 2019 14:26:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=9FGKxWOaR9lg8qKZvVxZYyBDHDZO4/U+GnNunz7EXSk=; b=lniqBkC9yuqD/Ju9/6Xyu42PxGGU6Yt/2fjCL1pA8uTadjROIw7CBcc8faiSFRo7Bv3q BPubFEelFsfRjyPHH37WhqoIXQX5UbWj/nD4TDzwZBHQoeubLcnl1AB5vQ/mCucdF+Yu 6nQIlGjLClEFr6VXl3jVjtuCT9T1LE8cucXjWWInMM2yXZ80VjgZXh9Wagaq6vop/Niu wJjXwR5Jbi7oMNqUbY65Is6pPs++Vxlbu62PVImCdj3jQ/wjgIZg0r9DuCe22wMVOgbF ph3T8g1h5qYYjiZctfTIEHMk2DM78dBxXxMV6ZGQQw9l9nE44jpFZXce/yhCQnDc6q9c 6g== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2tjm9r0bsr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:51 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuD021444; Thu, 11 Jul 2019 14:26:47 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 20/26] mm/asi: Add option to map cpu_hw_events Date: Thu, 11 Jul 2019 16:25:32 +0200 Message-Id: <1562855138-19507-21-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add option to map cpu_hw_events in ASI pagetable. Also restructure to select ptions for percpu optional mapping. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 1 + arch/x86/mm/asi.c | 3 +++ 2 files changed, 4 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 8199618..f489551 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -19,6 +19,7 @@ #define ASI_MAP_CPU_PTR 0x02 /* for get_cpu_var()/this_cpu_ptr() */ #define ASI_MAP_CURRENT_TASK 0x04 /* map the current task */ #define ASI_MAP_RCU_DATA 0x08 /* map rcu data */ +#define ASI_MAP_CPU_HW_EVENTS 0x10 /* map cpu hw events */ enum page_table_level { PGT_LEVEL_PTE, diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 20c23dc..d488704 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -17,6 +18,7 @@ #include #include +#include "../events/perf_event.h" #include "../../../kernel/rcu/tree.h" /* ASI sessions, one per cpu */ @@ -33,6 +35,7 @@ struct asi_map_option asi_map_percpu_options[] = { { ASI_MAP_CPU_PTR, &this_cpu_off, sizeof(this_cpu_off) }, { ASI_MAP_CURRENT_TASK, ¤t_task, sizeof(current_task) }, { ASI_MAP_RCU_DATA, &rcu_data, sizeof(rcu_data) }, + { ASI_MAP_CPU_HW_EVENTS, &cpu_hw_events, sizeof(cpu_hw_events) }, }; static void asi_log_fault(struct asi *asi, struct pt_regs *regs, From patchwork Thu Jul 11 14:25:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040109 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 886A71395 for ; Thu, 11 Jul 2019 14:27:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7674028A5A for ; Thu, 11 Jul 2019 14:27:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A49F28AD1; Thu, 11 Jul 2019 14:27:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D249E28A5A for ; Thu, 11 Jul 2019 14:27:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E67A98E00D8; Thu, 11 Jul 2019 10:27:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D79958E00C4; Thu, 11 Jul 2019 10:27:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF4838E00D8; Thu, 11 Jul 2019 10:27:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 9755A8E00C4 for ; Thu, 11 Jul 2019 10:27:05 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id k21so6990854ioj.3 for ; Thu, 11 Jul 2019 07:27:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=/gc70sOJB+vK9w9Yd0bv1jRw7Ga4U07eYdWLQD2PoZs=; b=rq3N//33YUWn7UazwDkM09Q6NTq4Vfvtgk5LuOtucd2ICpqtXy6etisgpwCJ1sHnh/ 36OjvcGoHLoO7AZVrIXqzFnsImAn7mMfCHoSt2/Z4KaL7BpYb40aWtx95kx6LDBCEHmi 03XFiuzOg7KP0mv4sRZubF3uNB6gHJ7zeo9KsMg5w9q6VYcoBMBbu2D0ozjdAR4xdDyZ NUCmW1EHNPrkORSYU+0Rk9tactln26j4M5RX+8Jq909OTNg6uH2tA1F27KkEJBupfl04 /7BMkM5NS6i0+J497mk3i6jpoygaXk8CLf+vdTJlp33mJKKBYYWo/7B+hDAXk869tMxm n2Cg== X-Gm-Message-State: APjAAAWN5Aq9ZI8Rp7c/cqj17HEri4Ym4gZ6rXf4A20DGxfMkJ9oFG7K IDXl0kUiHsxQLJecERdR1jNa79lQkCwd+zh5WvwZfOnVJzj7lD5kRR2OwIJbSyDGD8fN7ecctdp R5gedoBndL2CONYWYbW4D2VQpmbvikRZkiRuErhzoCjOKo8lioF8rsLW1lzlkL9jChA== X-Received: by 2002:a05:6602:219a:: with SMTP id b26mr4702134iob.55.1562855225412; Thu, 11 Jul 2019 07:27:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqz8W2dwSQPiNSE8DAE9H308CD25jD55MvxpK3PHCJ0niK/nrlVNjZWR7lG2+/x4+1snCE3v X-Received: by 2002:a05:6602:219a:: with SMTP id b26mr4702065iob.55.1562855224655; Thu, 11 Jul 2019 07:27:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855224; cv=none; d=google.com; s=arc-20160816; b=TGc/JhYQW+wKJ5joyxHGJ58nSFPU7c27RwuUvqzad3j7aDRdk3DDl4yN/HXNWgdH2L tjNqCIfSRhhvQbcpIS+UjkuDhGH6r4NK5hAo64EoglijNQjz5Tg/QL/OjkLC5KeQ+/EO bSZPBdIfVeuQRXlyBq0eQYDi3dT37igceeikIIeSvFFv1t6MkZuTfhk0SM4YFZrS8Naf +smqvP3Q8V77hsz0gTIuAyxflIjvkBs7BodFujbfWixVk8sjYL9wRH6kJi8yc50L8C0e hztUWfJhyPud7HUnqquZ4EbEO4gngHKOfHG+hOX8h3HjBThOCAGe1KUp82hb9UVxCWPR /O1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/gc70sOJB+vK9w9Yd0bv1jRw7Ga4U07eYdWLQD2PoZs=; b=gM4OaJ9amzON8DmmU1M/MAP2T1fJUjmEFT3sioPq6R0Geyjqk4TYQpHD8vrMyFDHEl UVqMqZmh5QZWko1CBn+/x1HHJiZ+XnyOSErC7SZ3bkrMwYNNmhlyfs6+NC5Hl+l7laa6 I+tUuLJdIWHqQbvkRpQL6ZIxehkMtrHf073A9Lpvudo7hGPgVlk4a/JzajuI1vhsscla Pw9qjF1PZljnmzDWBHpIBprGK+AvwHdjvM2PqSLyACyHVKCna4IEwJgRHTt93t5YaiTS YXOBOeIZbHf3s1+vXntR8Ivjrr8hsMS3oj2gXFjVuSKSlZ1xakL0Pu6K/J/vKyd3ARzo CItQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="EUNz/kVV"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id r17si8721748jao.29.2019.07.11.07.27.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:04 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="EUNz/kVV"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO7iI013253; Thu, 11 Jul 2019 14:26:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=/gc70sOJB+vK9w9Yd0bv1jRw7Ga4U07eYdWLQD2PoZs=; b=EUNz/kVVuwEftp1KU5qURSDwp+TGdX5vvQ5vTvYLS3DXTF1MuLYbqn75wFx41g9hyOGh RM9we7oEvKnvKWG/vpmQwbpaiu20SQ1RssVY5TTvMhrnQ8jKRj/CsRaQfpMKJY/qrm/1 LbhGJOplC+6BG/nJHq9sTPd5TIz1YACiQ9xWzDt1BtXTgliVimphNKiT1UU38OYP7oSW ELLA9E7UVvNKoLW2OQtdqbwZk0J9ufJIiTIIREI+9J5HCJcDTctrZalqmR+lSizxjXTr S4aV0Cb5jllRnc+3jSzNcfsprSQmpSqw2GObGMXoLB3Fng/zJt5MeVYf3wZ0eH8MRSjK eQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2tjm9r0btb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:26:54 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuE021444; Thu, 11 Jul 2019 14:26:50 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 21/26] mm/asi: Make functions to read cr3/cr4 ASI aware Date: Thu, 11 Jul 2019 16:25:33 +0200 Message-Id: <1562855138-19507-22-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When address space isolation is active, cpu_tlbstate isn't necessarily mapped in the ASI page-table, this would cause ASI to fault. Instead of just mapping cpu_tlbstate, update __get_current_cr3_fast() and cr4_read_shadow() by caching the cr3/cr4 values in the ASI session when ASI is active. Note that the cached cr3 value is the ASI cr3 value (i.e. the current CR3 value when ASI is active). The cached cr4 value is the cr4 value when isolation was entered (ASI doesn't change cr4). Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 2 ++ arch/x86/include/asm/mmu_context.h | 20 ++++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 10 ++++++++++ arch/x86/mm/asi.c | 3 +++ 4 files changed, 33 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index f489551..07c2b50 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -73,7 +73,9 @@ struct asi_session { enum asi_session_state state; /* state of ASI session */ bool retry_abort; /* always retry abort */ unsigned int abort_depth; /* abort depth */ + unsigned long isolation_cr3; /* cr3 when ASI is active */ unsigned long original_cr3; /* cr3 before entering ASI */ + unsigned long original_cr4; /* cr4 before entering ASI */ struct task_struct *task; /* task during isolation */ } __aligned(PAGE_SIZE); diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 9024236..8cec983 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -14,6 +14,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -347,8 +348,23 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, */ static inline unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3; + +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* + * If isolation is active, cpu_tlbstate isn't necessarily mapped + * in the ASI page-table (and it doesn't have the current pgd anyway). + * The current CR3 is cached in the CPU ASI session. + */ + if (this_cpu_read(cpu_asi_session.state) == ASI_SESSION_STATE_ACTIVE) + cr3 = this_cpu_read(cpu_asi_session.isolation_cr3); + else + cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid)); +#else + cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid)); +#endif /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index dee3758..917f9a5 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -12,6 +12,7 @@ #include #include #include +#include /* * The x86 feature is called PCID (Process Context IDentifier). It is similar @@ -324,6 +325,15 @@ static inline void cr4_toggle_bits_irqsoff(unsigned long mask) /* Read the CR4 shadow. */ static inline unsigned long cr4_read_shadow(void) { +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + /* + * If isolation is active, cpu_tlbstate isn't necessarily mapped + * in the ASI page-table. The CR4 value is cached in the CPU + * ASI session. + */ + if (this_cpu_read(cpu_asi_session.state) == ASI_SESSION_STATE_ACTIVE) + return this_cpu_read(cpu_asi_session.original_cr4); +#endif return this_cpu_read(cpu_tlbstate.cr4); } diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index d488704..4a5a4ba 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -23,6 +23,7 @@ /* ASI sessions, one per cpu */ DEFINE_PER_CPU_PAGE_ALIGNED(struct asi_session, cpu_asi_session); +EXPORT_SYMBOL(cpu_asi_session); struct asi_map_option { int flag; @@ -291,6 +292,8 @@ int asi_enter(struct asi *asi) goto err_unmap_task; } asi_session->original_cr3 = original_cr3; + asi_session->original_cr4 = cr4_read_shadow(); + asi_session->isolation_cr3 = __sme_pa(asi->pgd); /* * Use ASI barrier as we are setting CR3 with the ASI page-table. From patchwork Thu Jul 11 14:25:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60A6F112C for ; Thu, 11 Jul 2019 14:27:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5019828A5A for ; Thu, 11 Jul 2019 14:27:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43AD928AD1; Thu, 11 Jul 2019 14:27:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B845928A5A for ; Thu, 11 Jul 2019 14:27:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 341B58E00D9; Thu, 11 Jul 2019 10:27:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2CB698E00C4; Thu, 11 Jul 2019 10:27:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16BF98E00D9; Thu, 11 Jul 2019 10:27:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id DEB488E00C4 for ; Thu, 11 Jul 2019 10:27:11 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id h3so6951210iob.20 for ; Thu, 11 Jul 2019 07:27:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=R7Xfip06p699muIVGHbRVSpcUyRHanu29pNDIpaJD54=; b=qp+C1QJ9y1Lg/RytDcG5BgEwRzJRL1RbRbtf3330AH/VjmkGWPWeFVrUkMNipvxK9v UEvpQ5jiztBH2XJbkLVKtxbVEEaQzXz/ptanKEGKjMlF4l8+Ofe1y5SVubhSG7fuLM4f pbl+vnI84KMXw2rSAiJxjvH+Z6ozkDgLfhmh5Gpjc+mAuot7xd1XRovWJLTyzCq8tObU XBA8YgZ7rtnPhwqyIFVyYoDOEXaT1szdusxM9PFWuRhxQbdGxnHjJGlxwwqHCDrIMKdr NVCf6JqZoFj1EAx82eEpPPB5gqM6T7K2NNF1oRoxvOO0ONNP5BQrWU9h4AoNKVcS9MM/ JOOA== X-Gm-Message-State: APjAAAWM33VIuNdH89h84DJN8JF84Jh1gl9o++ajbrqzHZwsEoZHFGPr beRAxI1uURGiHFhY59tnf5EIPUGjyaVgsJkMgga0yaJ95VRT0jfQUC0S3iETq9aim6hYCjVKdyc sQ851qzlIzkk+X3Xsn76fGnVTd83mjFQ4lsdDeiukMOSm7KH7D7+TwVdkLDWrFpjmWA== X-Received: by 2002:a6b:5103:: with SMTP id f3mr587517iob.142.1562855231661; Thu, 11 Jul 2019 07:27:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqw7ECHKQ2f7hZwF0jY9gg0NOF5NdD+0PPtXIE+BNEroqN7fMCZozDADEyX33L3/3/sQfsLb X-Received: by 2002:a6b:5103:: with SMTP id f3mr587449iob.142.1562855230986; Thu, 11 Jul 2019 07:27:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855230; cv=none; d=google.com; s=arc-20160816; b=eKRj/sdoIMCWdm+TtQhioNDmtW0kVBWOiB3w+IVfbaOBMvj2aWt0scG3Jchath9/6X 2JnVadnZyzdMQvCbplDCV3XX8XWxUjeKSEjdBkMPUFOmAgvzmUPstgzDXILxvLOj7eJ4 RHbfQs06qqi1BM6YIQfR/BVJUPnwDmNYSv/pQDoYEYMw7EuTBSroXoxSHneAEcl5Olyz qJL9X3a6uSoOFALIfhDVGN9dutzEKHBMqccu3kYGCyQJ4AlTvG6ZcBSi9TOXhb01XJ7g m4lnw/bU5hkGboTZdeku4025oZg5h2gVKtoxtSyXv1atesaj1todn4xzFeG2q7uuePCu E5fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=R7Xfip06p699muIVGHbRVSpcUyRHanu29pNDIpaJD54=; b=0sOKxK+mQ9aLwJibUkKy5Lc7QN7yyoatt8O1SL0s61k3TAoqxIFnKN3/QkTiB0ykqh P9QYRZiNM1TAgrJr/jxWdJEZEToZQOFox7GdC1vIFXbxf/ET0jiM1jyaRx5JYhmFtJUg rzwAWcTdi3H7YRUfhbvIM500tUaW0ooeKuYsObjvnURvH0afgW/pYaI3jRqREP7D6aZ6 gz1Ofy545QotQrSIhqj8mBOpc5apByRJ8faFsTy1gkz9VCLgJxD4OUBjuPM5/PqNDWlw 585uejGQ9DXqrdbkhFLAR5Rq+kRTW01s2nxYDOVUB0fJyJ7xSIrl7vOIPmn4lPDIs63V tI1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="LYEr+9/u"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id p24si8131931ioj.51.2019.07.11.07.27.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:10 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="LYEr+9/u"; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOGvb100511; Thu, 11 Jul 2019 14:27:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=R7Xfip06p699muIVGHbRVSpcUyRHanu29pNDIpaJD54=; b=LYEr+9/uvSyvI7OVyk9UP5FJcmH4YzsINobd9Ws4d1Sn7zQRqsU665EFWo2tRiz/cBWH TQPi+azMpPQ50ca6ZxpYu/g0a8JuwQaUyBMwUTLzLIprNNem/R910eIUU8P5Bv98N3Zz VQLKGIpKXFE80COh/4g3j2DCRhhYcgvjn2KBcjXSAHnxcqRegfAAZE4hnlxbfYZgQOEd YTBZwZfCqCbfS+eDaW7yYwGctCL21NUDe7ydmDXBTvwj7gQ1BR2oLXZZ5z2nmPV3Av6O uW3d4XWaCBCFsg3LcCI9MxRw/Tgth3Gipeew1Xr8UIl8LL6WLVneXw3ZkgDZG2ZEH0Xh 4A== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0cdp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:27:02 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuF021444; Thu, 11 Jul 2019 14:26:53 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 22/26] KVM: x86/asi: Introduce address_space_isolation module parameter Date: Thu, 11 Jul 2019 16:25:34 +0200 Message-Id: <1562855138-19507-23-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Add the address_space_isolation parameter to the kvm module. When set to true, KVM #VMExit handlers run in isolated address space which maps only KVM required code and per-VM information instead of entire kernel address space. This mechanism is meant to mitigate memory-leak side-channels CPU vulnerabilities (e.g. Spectre, L1TF and etc.) but can also be viewed as security in-depth as it also helps generically against info-leaks vulnerabilities in KVM #VMExit handlers and reduce the available gadgets for ROP attacks. This is set to false by default because it incurs a performance hit which some users will not want to take for security gain. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/Makefile | 3 ++- arch/x86/kvm/vmx/isolation.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletions(-) create mode 100644 arch/x86/kvm/vmx/isolation.c diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 31ecf7a..71579ed 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -12,7 +12,8 @@ kvm-y += x86.o mmu.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ hyperv.o page_track.o debugfs.o -kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o +kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ + vmx/evmcs.o vmx/nested.o vmx/isolation.o kvm-amd-y += svm.o pmu_amd.o obj-$(CONFIG_KVM) += kvm.o diff --git a/arch/x86/kvm/vmx/isolation.c b/arch/x86/kvm/vmx/isolation.c new file mode 100644 index 0000000..e25f663 --- /dev/null +++ b/arch/x86/kvm/vmx/isolation.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. + * + * KVM Address Space Isolation + */ + +#include +#include + +/* + * When set to true, KVM #VMExit handlers run in isolated address space + * which maps only KVM required code and per-VM information instead of + * entire kernel address space. + * + * This mechanism is meant to mitigate memory-leak side-channels CPU + * vulnerabilities (e.g. Spectre, L1TF and etc.) but can also be viewed + * as security in-depth as it also helps generically against info-leaks + * vulnerabilities in KVM #VMExit handlers and reduce the available + * gadgets for ROP attacks. + * + * This is set to false by default because it incurs a performance hit + * which some users will not want to take for security gain. + */ +static bool __read_mostly address_space_isolation; +module_param(address_space_isolation, bool, 0444); From patchwork Thu Jul 11 14:25:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040115 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 40C1E112C for ; Thu, 11 Jul 2019 14:27:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E24428A5A for ; Thu, 11 Jul 2019 14:27:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 213DE28AD1; Thu, 11 Jul 2019 14:27:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A1FD28A5A for ; Thu, 11 Jul 2019 14:27:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E0778E00DE; Thu, 11 Jul 2019 10:27:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F04988E00DD; Thu, 11 Jul 2019 10:27:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3FF08E00DC; Thu, 11 Jul 2019 10:27:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 9B0E38E00C4 for ; Thu, 11 Jul 2019 10:27:16 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id w17so6989802iom.2 for ; Thu, 11 Jul 2019 07:27:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=0gU3+EHAnLmrdQwDq6KHZSyy6MdJI7eVGdpzzFAPgXk=; b=FT1WFsQ4ljkaMhP7fwxoJH57Ti6cVQBc5AzLVaPSdA6fhfm7MKMr5FOKCz6z6xC9wW YEjEtHwoLn8wpezLBjs3efoUbOQ3BI4KvIliwGU/AeSfYbaVwXdvJ5pAX9KHv+sTX1k5 EE4uM/xaGBc0qi+BO3et6Zr2CnAP5lJyIPlIS53jFxY9R6COmknfZP97G5AhhhC/TDYc jVeaSZvHT2dB8VG5U3PoVa9hrc+cC9haqBV8kXiRwB64MaNFKgvkbF4v0Si0QUq36wYe M02hPWAvV/E4zkWZQ2DnPCwkgflCKjaYXxNenSbP+W/1xbN/b5hZRK/xGu0xizK0hM9o m4nA== X-Gm-Message-State: APjAAAWDWnkCjgLNw1W5/dNGGpw5pLZIcGIDa8xIkwpnrJ7sYHccfVLQ Gpe/G5qSCA2+PRlt4suWaFNhJXynkEJ101G55y8rUeaiTFvbJgrSc8OjK8c9ap2eLx4dvM/Fg7J w5yZbVArg7KPVE8O2kgPA6EcdDquHhhtmgPdwFueWg8oOzg6cwg0qPMfQx5w934g+Ag== X-Received: by 2002:a6b:dc13:: with SMTP id s19mr4682193ioc.53.1562855236378; Thu, 11 Jul 2019 07:27:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqx+dg2vp6x5QjuMXysOlicwZ/bRZTT+fUXicQSj1yHU21CsQlh5PwTRgrvhdxUfQwzhtH7L X-Received: by 2002:a6b:dc13:: with SMTP id s19mr4682108ioc.53.1562855235381; Thu, 11 Jul 2019 07:27:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855235; cv=none; d=google.com; s=arc-20160816; b=JZE3FdX1Xq81bwG/L8LAAncxbYmqw8gTnuJW5YMra/jmDK8v0lI5/cIEo7b3NhQWNL KKL+uMj09TzszcHfWaZ8prLAfGcl3y2xaR5LNBBXAfwNyeHIGFa83/E8YZ3ROqKzbuc5 8HU/hyht9MAV2tnl0J/TkYtsU+XqzQ5WvangPBq2Wx4E6hR+u/FMwywT28Q07f4J6Gkd sfMZ/SWE3pxxoObdmv814jMLz9NNdmIUZUvbdBGdqADi61tk81acvy8tqScEIIeu39ew wZY2qgjtOppm28wo8pt/2fsibq/7c+TJy9Ph7kwTOomwSj0KLLd4kbx6Cs1nCxFZVIql DaxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0gU3+EHAnLmrdQwDq6KHZSyy6MdJI7eVGdpzzFAPgXk=; b=Wbo1ut+dX4Tvmz4jqCRSGQ8cgBF8tFzKqW/5RI5WhBoYCi/JT2GR1W678S/p7XJ+7d ecJeEdyYb6d5l5bCf84h+JJmgaA3BzORLYXU+ZpnPMvWticyQEIfDIq1LedIBSQVnS5V OV741xosPP/uzj2wQvEUXZYxJToaxHuc14XYHIBvsnBE2Cx9HarCNpcQBAYNg8no2nRX Jj2tRyjBNnlkoL0UggxWmNiV1+96tf7KEqqj3kmiMqz3oQpSt+xRpdCh5PO/7K89Fosr RlNXB4IBYNLZO4ZXFt8mVuo4uN9+rIlaiZPSn3npg4qclk1GgaECVnYJ8ct9bxjFpWh7 ltwA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=iiyyes3p; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id t2si9936464jam.54.2019.07.11.07.27.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:15 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=iiyyes3p; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO8MS001456; Thu, 11 Jul 2019 14:27:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=0gU3+EHAnLmrdQwDq6KHZSyy6MdJI7eVGdpzzFAPgXk=; b=iiyyes3p413lD4tawJspczGrP6E0lodllk7rkaTrmsT9/Hc2gRBbnS6pJ51rtyC6w6oK s0zXzPmMoMMZfu+NfOM5wgMpW+ZwZb5dNJIjL1sOQ2wAe8I6VnoBZVNNsVBlEfHguvjZ UWzv4mgQ5Z/QbJM2eHak92dK2mAAiscm2gFOv4AEAAm/6hM07TfZIH/hBOE3puhP7Woq 7qMIQ3AxCkBa3FdCnuUeCDTF3Vl7BFaFDKI8v0ZHZA9v/ppmgWW2qg65hjSCEFqy4KMB vWZ/Kr+eO2oRgHWyu5lBLP3blCrBHPO1SGFrwvoYGRq8pBU17+caS24D540bGjrkqGJ+ fA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e4s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:27:05 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuG021444; Thu, 11 Jul 2019 14:26:57 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 23/26] KVM: x86/asi: Introduce KVM address space isolation Date: Thu, 11 Jul 2019 16:25:35 +0200 Message-Id: <1562855138-19507-24-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Create a separate address space for KVM that will be active when KVM #VMExit handlers run. Up until the point which we architectully need to access host (or other VM) sensitive data. This patch just create the address space using address space isolation (asi) but never makes it active yet. This will be done by next commits. Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/vmx/isolation.c | 58 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 7 ++++- arch/x86/kvm/vmx/vmx.h | 3 ++ include/linux/kvm_host.h | 5 +++ 4 files changed, 72 insertions(+), 1 deletions(-) diff --git a/arch/x86/kvm/vmx/isolation.c b/arch/x86/kvm/vmx/isolation.c index e25f663..644d8d3 100644 --- a/arch/x86/kvm/vmx/isolation.c +++ b/arch/x86/kvm/vmx/isolation.c @@ -7,6 +7,15 @@ #include #include +#include +#include +#include + +#include "vmx.h" +#include "x86.h" + +#define VMX_ASI_MAP_FLAGS \ + (ASI_MAP_STACK_CANARY | ASI_MAP_CPU_PTR | ASI_MAP_CURRENT_TASK) /* * When set to true, KVM #VMExit handlers run in isolated address space @@ -24,3 +33,52 @@ */ static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); + +static int vmx_isolation_init_mapping(struct asi *asi, struct vcpu_vmx *vmx) +{ + /* TODO: Populate the KVM ASI page-table */ + + return 0; +} + +int vmx_isolation_init(struct vcpu_vmx *vmx) +{ + struct kvm_vcpu *vcpu = &vmx->vcpu; + struct asi *asi; + int err; + + if (!address_space_isolation) { + vcpu->asi = NULL; + return 0; + } + + asi = asi_create(VMX_ASI_MAP_FLAGS); + if (!asi) { + pr_debug("KVM: x86: Failed to create address space isolation\n"); + return -ENXIO; + } + + err = vmx_isolation_init_mapping(asi, vmx); + if (err) { + vcpu->asi = NULL; + return err; + } + + vcpu->asi = asi; + + pr_info("KVM: x86: Running with isolated address space\n"); + + return 0; +} + +void vmx_isolation_uninit(struct vcpu_vmx *vmx) +{ + struct kvm_vcpu *vcpu = &vmx->vcpu; + + if (!address_space_isolation || !vcpu->asi) + return; + + asi_destroy(vcpu->asi); + vcpu->asi = NULL; + pr_info("KVM: x86: End of isolated address space\n"); +} diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d98eac3..9b92467 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -202,7 +202,7 @@ }; #define L1D_CACHE_ORDER 4 -static void *vmx_l1d_flush_pages; +void *vmx_l1d_flush_pages; static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) { @@ -6561,6 +6561,7 @@ static void vmx_free_vcpu(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); + vmx_isolation_uninit(vmx); if (enable_pml) vmx_destroy_pml_buffer(vmx); free_vpid(vmx->vpid); @@ -6672,6 +6673,10 @@ static void vmx_free_vcpu(struct kvm_vcpu *vcpu) vmx->ept_pointer = INVALID_PAGE; + err = vmx_isolation_init(vmx); + if (err) + goto free_vmcs; + return &vmx->vcpu; free_vmcs: diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 61128b4..09c1593 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -525,4 +525,7 @@ static inline void decache_tsc_multiplier(struct vcpu_vmx *vmx) void dump_vmcs(void); +int vmx_isolation_init(struct vcpu_vmx *vmx); +void vmx_isolation_uninit(struct vcpu_vmx *vmx); + #endif /* __KVM_X86_VMX_H */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index d1ad38a..2a9d073 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -34,6 +34,7 @@ #include #include +#include #ifndef KVM_MAX_VCPU_ID #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS @@ -320,6 +321,10 @@ struct kvm_vcpu { bool preempted; struct kvm_vcpu_arch arch; struct dentry *debugfs_dentry; + +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION + struct asi *asi; +#endif }; static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu) From patchwork Thu Jul 11 14:25:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040113 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79F6B112C for ; Thu, 11 Jul 2019 14:27:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 639FA28AC8 for ; Thu, 11 Jul 2019 14:27:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 570B828A5A; Thu, 11 Jul 2019 14:27:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A61D828A5A for ; Thu, 11 Jul 2019 14:27:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 628F08E00DA; Thu, 11 Jul 2019 10:27:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 58BB18E00C4; Thu, 11 Jul 2019 10:27:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DDF98E00DA; Thu, 11 Jul 2019 10:27:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id 16F048E00C4 for ; Thu, 11 Jul 2019 10:27:15 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id c5so6906174iom.18 for ; Thu, 11 Jul 2019 07:27:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=44bGpNhDn4CQMJWfcA7dlBtv4lKR0GVNyXSLf3OmODU=; b=pCh4uDFC4XTL4WcSU0wuXIeIUkorjXDot+y2+4jEE8RoxkJtT1lJH9juewRxfzrDi3 WuAWabhtXVYbXt9bq0XRRaE9W+Bij5XzTppglk8an3q96agwDGdvTOBiQ8BfTc9Dvrto SKDYZAT1f6mDZWft9RRH18fYsGPLqktQKj3ni2AqFKdmFGD9dyUUnQjsgu5GxkberAgS atjgP0qREbQbIHA8XCF50YJOE/0c2arHf3yetw/dLFl9tzusey+mym69DMVUEHaG1YKa KSa5wa3fSIvSfkzEh3pl1h120DDabY3xqqNhQbEuFf5vanwKz98dlDAVJ5wkUnsgqC0I eldw== X-Gm-Message-State: APjAAAUUrKweYkyKfSclYsBJhZPwTPslkbVa3j1EsCx2k7dn+0HUeCtP w7RAsKWccPWm371Au2xgum0ZTtwJHojlLcsIrPvKVaHE30EPzfYLruLim0tFLjlnUiK5FlcoPC0 ohoj87J6AGZ8avtSu2hPrKhjkjsecTYhMsCs4jHN393g7Vm+ywdy6LPUv2h2MNitd5Q== X-Received: by 2002:a02:3093:: with SMTP id q141mr4960915jaq.128.1562855234866; Thu, 11 Jul 2019 07:27:14 -0700 (PDT) X-Google-Smtp-Source: APXvYqyx3JkDeJKtCxBqVSCeUIqJnOuzov8HrqcpnK9a9zqX+fscGxJvEeAJYuz6e1cSdeWxUTeD X-Received: by 2002:a02:3093:: with SMTP id q141mr4960846jaq.128.1562855234036; Thu, 11 Jul 2019 07:27:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855234; cv=none; d=google.com; s=arc-20160816; b=qqc9rGT3yFowwWs8IFINc9KYAPWwodVCqvyKkoCbcj5BWz2sRG52nOIf9CRUmZHoj4 oAXc5vU0xRGWJasY7P7t81rviuWNL0nXkM7TeMq7EtLPEuIpTxV54/EBhhGWFp7B1Cgb aA4i/0jFjboyqvqlZnusbQWl/4TeWglJGxc0xdfbYyI9YK7aAJPm03cL+/vovz8b6Mc9 8Z9w4e9VjfBr2Unom/dJ+nHTFPSgLt1ydGB+DEIIv0ruQ62qhDf85TZKyI2/ZHvBYbhn ZhY3NUf0LP5H5bXOdiqH/NRWmamg3CN+A+IX0sg1DYZpuRccns5iTiP1PTvQenqmG7td cqZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=44bGpNhDn4CQMJWfcA7dlBtv4lKR0GVNyXSLf3OmODU=; b=r5OMBZ3h5L/OovWYRfJIQ1L7gSyLbqDfmolcOMEA+u+2RJPPT+OTAiH4q1b/GhXY3e FJkHtfp+MwSOshydpHRmZp5noaqNFesaoPDqiWF4WWCSrtK1Pq4qK6TllSyLWnkyZrJv SRs0b4Uu4LnGDT/pmhDtup4e8g/RzoPUX1NdjD16/CbsPNYqrSKUVxKQgxVjlnnw7pNW HrEae8Xb7btL96Ykym0dEygCsNiWzO9nbJ0jKRlxCEz5SB9anazmhoHnEVDanOZSW4g6 zevcsl4u+3pMY+NoCoje467ZC7M6+gBg3jkkECHR6Eu1RzLDtgdHOiOo/8LoU0icFU/k dZ9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=sYC68pAg; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id f1si8316787jaa.44.2019.07.11.07.27.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:14 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=sYC68pAg; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO8ft001446; Thu, 11 Jul 2019 14:27:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=44bGpNhDn4CQMJWfcA7dlBtv4lKR0GVNyXSLf3OmODU=; b=sYC68pAgGYMeZCzOFkUOuj4EApQvKlshFt7mE2WAdPKV4oXjKlwvx1k4YJanPkydZ+Rn nhL5rnoahTfhc8IS1gDxcvz2TkFDuiQbUpBXbrUM3d7hA1hqNT1LXCfxGliFZOD6TdOF /3LBjw+5bGaWzvBtKCMuwVIUzbBatm/N7BwPJzQJCnQfj9Csi3AeniX/ps6uYP+y0BkL obw32Tp/dAVmexQz4kk3ptW1s2ArR2RXMrCD7YlTDm2ZoFxL1jsctlJFVVEs2mdJMt1K kWQize+PYuE3+4ce/ZZAOApqTAW4neDTZZCWGAKbNKb/yPwlRIT1dU+LMDkfey1//U/x dQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e4a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:27:03 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuH021444; Thu, 11 Jul 2019 14:27:00 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 24/26] KVM: x86/asi: Populate the KVM ASI page-table Date: Thu, 11 Jul 2019 16:25:36 +0200 Message-Id: <1562855138-19507-25-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add mappings to the KVM ASI page-table so that KVM can run with its address space isolation without faulting too much. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/vmx/isolation.c | 155 ++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.c | 1 - arch/x86/kvm/vmx/vmx.h | 3 + 3 files changed, 154 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/isolation.c b/arch/x86/kvm/vmx/isolation.c index 644d8d3..d82f6b6 100644 --- a/arch/x86/kvm/vmx/isolation.c +++ b/arch/x86/kvm/vmx/isolation.c @@ -5,7 +5,7 @@ * KVM Address Space Isolation */ -#include +#include #include #include #include @@ -14,8 +14,11 @@ #include "vmx.h" #include "x86.h" -#define VMX_ASI_MAP_FLAGS \ - (ASI_MAP_STACK_CANARY | ASI_MAP_CPU_PTR | ASI_MAP_CURRENT_TASK) +#define VMX_ASI_MAP_FLAGS (ASI_MAP_STACK_CANARY | \ + ASI_MAP_CPU_PTR | \ + ASI_MAP_CURRENT_TASK | \ + ASI_MAP_RCU_DATA | \ + ASI_MAP_CPU_HW_EVENTS) /* * When set to true, KVM #VMExit handlers run in isolated address space @@ -34,9 +37,153 @@ static bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); +/* + * Map various kernel data. + */ +static int vmx_isolation_map_kernel_data(struct asi *asi) +{ + int err; + + /* map context_tracking, used by guest_enter_irqoff() */ + err = ASI_MAP_CPUVAR(asi, context_tracking); + if (err) + return err; + + /* map irq_stat, used by kvm_*_cpu_l1tf_flush_l1d */ + err = ASI_MAP_CPUVAR(asi, irq_stat); + if (err) + return err; + return 0; +} + +/* + * Map kvm module and data from that module. + */ +static int vmx_isolation_map_kvm_data(struct asi *asi, struct kvm *kvm) +{ + int err; + + /* map kvm module */ + err = asi_map_module(asi, "kvm"); + if (err) + return err; + + err = asi_map_percpu(asi, kvm->srcu.sda, + sizeof(struct srcu_data)); + if (err) + return err; + + return 0; +} + +/* + * Map kvm-intel module and generic x86 data. + */ +static int vmx_isolation_map_kvm_x86_data(struct asi *asi) +{ + int err; + + /* map current module (kvm-intel) */ + err = ASI_MAP_THIS_MODULE(asi); + if (err) + return err; + + /* map current_vcpu, used by vcpu_enter_guest() */ + err = ASI_MAP_CPUVAR(asi, current_vcpu); + if (err) + return (err); + + return 0; +} + +/* + * Map vmx data. + */ +static int vmx_isolation_map_kvm_vmx_data(struct asi *asi, struct vcpu_vmx *vmx) +{ + struct kvm_vmx *kvm_vmx; + struct kvm_vcpu *vcpu; + struct kvm *kvm; + int err; + + vcpu = &vmx->vcpu; + kvm = vcpu->kvm; + kvm_vmx = to_kvm_vmx(kvm); + + /* map kvm_vmx (this also maps kvm) */ + err = asi_map(asi, kvm_vmx, sizeof(*kvm_vmx)); + if (err) + return err; + + /* map vmx (this also maps vcpu) */ + err = asi_map(asi, vmx, sizeof(*vmx)); + if (err) + return err; + + /* map vcpu data */ + err = asi_map(asi, vcpu->run, PAGE_SIZE); + if (err) + return err; + + err = asi_map(asi, vcpu->arch.apic, sizeof(struct kvm_lapic)); + if (err) + return err; + + /* + * Map additional vmx data. + */ + + if (vmx_l1d_flush_pages) { + err = asi_map(asi, vmx_l1d_flush_pages, + PAGE_SIZE << L1D_CACHE_ORDER); + if (err) + return err; + } + + if (enable_pml) { + err = asi_map(asi, vmx->pml_pg, sizeof(struct page)); + if (err) + return err; + } + + err = asi_map(asi, vmx->guest_msrs, PAGE_SIZE); + if (err) + return err; + + err = asi_map(asi, vmx->vmcs01.vmcs, PAGE_SIZE << vmcs_config.order); + if (err) + return err; + + err = asi_map(asi, vmx->vmcs01.msr_bitmap, PAGE_SIZE); + if (err) + return err; + + err = asi_map(asi, vmx->vcpu.arch.pio_data, PAGE_SIZE); + if (err) + return err; + + return 0; +} + static int vmx_isolation_init_mapping(struct asi *asi, struct vcpu_vmx *vmx) { - /* TODO: Populate the KVM ASI page-table */ + int err; + + err = vmx_isolation_map_kernel_data(asi); + if (err) + return err; + + err = vmx_isolation_map_kvm_data(asi, vmx->vcpu.kvm); + if (err) + return err; + + err = vmx_isolation_map_kvm_x86_data(asi); + if (err) + return err; + + err = vmx_isolation_map_kvm_vmx_data(asi, vmx); + if (err) + return err; return 0; } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 9b92467..d47f093 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -201,7 +201,6 @@ [VMENTER_L1D_FLUSH_NOT_REQUIRED] = {"not required", false}, }; -#define L1D_CACHE_ORDER 4 void *vmx_l1d_flush_pages; static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 09c1593..e8de23b 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -11,6 +11,9 @@ #include "ops.h" #include "vmcs.h" +#define L1D_CACHE_ORDER 4 +extern void *vmx_l1d_flush_pages; + extern const u32 vmx_msr_index[]; extern u64 host_efer; From patchwork Thu Jul 11 14:25:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040117 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E2EB1395 for ; Thu, 11 Jul 2019 14:27:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C93328A5A for ; Thu, 11 Jul 2019 14:27:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FD8728AD1; Thu, 11 Jul 2019 14:27:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7724B28A5A for ; Thu, 11 Jul 2019 14:27:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BA318E00C4; Thu, 11 Jul 2019 10:27:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D1B48E00DC; Thu, 11 Jul 2019 10:27:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7B398E00C4; Thu, 11 Jul 2019 10:27:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id A41EF8E00DB for ; Thu, 11 Jul 2019 10:27:16 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id u84so7014846iod.1 for ; Thu, 11 Jul 2019 07:27:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=fCmddW3R3yJKPuzx/XlLHeQxEVYNN8yCIHDije5QoA8=; b=Dr4hisdJrNaCqasg73GKBBRyLguFjrdvptRbWksr7QSqMrno4g4zLQYfCBcL5Fi8lr RvlEKqUZCYI4JOmI/lD065I4CPS6YLJMj8T3wqxxyk0+xW83Q6EFjotpWKOjaaQeiGrM y7yuS0lPg/PJDpOKjG/Rf+QRcoM6CAC25aADZKeViAvwof7fxD/1vcj0DD88q4rLMriJ tydj2f3MPv4/3N3v5KhySbIEvq0LnPId31r8evjvjTsYk0+i3T7S4WOD6tYjcZGMTvq9 pQ7A2YTKGwZHEgfk0cVBdTQXHAw/Q3jCkxup+GKu8I6KbtnLrPiTVg/smKkGIpDDIjRV +l2g== X-Gm-Message-State: APjAAAV10+jGsgfUIaR02o/UjW+1ficY3ankOdDFNgt66PKrcIu8Mq5D ouGE5k0LAEjbGJ3zDVqCgrpdlmO6dfsWoNPnOEO2CjqhczOiEMhw0v3vFZzbGI4sygqxieO11hn M3Axa5Xf03SC4+hcO46rxPAI9UWDMB6W4KyBvPwmALbUiuHsL6mhpfu1yfJgt+TC8cQ== X-Received: by 2002:a5d:8ad0:: with SMTP id e16mr4724385iot.262.1562855236413; Thu, 11 Jul 2019 07:27:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqzgNvJ1HNu/ew6Er6f8UJWHpUqlNDLhrIa7gfZ3zBprI2/hDQudi2p+iWiDewW1A78N6yJM X-Received: by 2002:a5d:8ad0:: with SMTP id e16mr4724304iot.262.1562855235513; Thu, 11 Jul 2019 07:27:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855235; cv=none; d=google.com; s=arc-20160816; b=l0eBthBHasR3H1342LJ1PXDeZhUOcQ1w60Rto2ca4jNzKKMwdGi1VnbInPCNE0rJIs /vkpkOx+cWIYJ7aOqkzicU7j1ormHMsrTeL8iV93lTLgvjTQzGKdcT2s3Sq7Z0udNsry +IegyIMr8v6bH271ug5J7gxY9NNBKRkobY7P3n40yP1in88STSXDOQtYhYlGvlhshbp9 YapZy3RHfF+KWZLD3TGPZL8fFUDWQNFBTmvCffDRBGwiaPPWoI7DYuWnZhX2bq8S0+Y8 qp1HXrqhm4Nlsx5+VHgbFf1pNdTb9bnjMXNznh+AAIiMr6FedUErThJqQe0ZI4u12adG hLZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fCmddW3R3yJKPuzx/XlLHeQxEVYNN8yCIHDije5QoA8=; b=W04rpwChe3Y/BkYyKI7y+e01DfPsnyICxhdLx9ybHbkqXswCo8p4LPJOYx/IN/41KW XCCIh1QmXrAJbxxY6UhQ6XEGXcwvHvhSqSd5s/fKeQ6xulY913eJfdXy6s4G0mLk4Gmc QsciaZCpS1Ye15oEU+iTT/gDFkoBPyJFMMSCH4JDorhTIvZybRFjYpfn0d5YuJCAk4wK 44CxkUdsNHgzBzoOpAnH/+oukhCFTsnqXo13O65XXWYIzZs+cAau8sscFUr3i3l1Izuc ij44jnHHXMOaINrm2pmks+ubOXvxb/E0gmSLZwXCydQCoZYrcvXmtHKvHKNv4CfTLzxC aiPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Goc71bNb; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2120.oracle.com (aserp2120.oracle.com. [141.146.126.78]) by mx.google.com with ESMTPS id v10si10067097jar.114.2019.07.11.07.27.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:15 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) client-ip=141.146.126.78; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Goc71bNb; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 141.146.126.78 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEOeBP100905; Thu, 11 Jul 2019 14:27:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=fCmddW3R3yJKPuzx/XlLHeQxEVYNN8yCIHDije5QoA8=; b=Goc71bNby+SDwxhZAZJVL35O0KCgSN6SQ19s01XeIVYA0KpDzxqCOIh+DsMUdWnvypT6 gGIgJRxy1708M9KVMhnYoB2ywb6qTtvwn4CXAcD7cmjlrvcJBS5+6PMDrIoeoEOtcDJK KW4X5dXci15RHrRtDaZcLfMFnXMXtBJVZEJjoExPc4mdpEdW63wB2JO1A+Dh4bIcTLp9 33On0i+8xG4PIxV+nRWqZqXbmizbh6ObtA00K1qWRZ6jEfJsKpM/X75UoB221Hphl68n Yka3rbKbML0ZOqcFcBNFej4+/AWOTKnp/kUU6rp69SbIMOLJvaM5ku0lYaGL5zwa2anP Lg== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2tjkkq0ceg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:27:06 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuI021444; Thu, 11 Jul 2019 14:27:03 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 25/26] KVM: x86/asi: Switch to KVM address space on entry to guest Date: Thu, 11 Jul 2019 16:25:37 +0200 Message-Id: <1562855138-19507-26-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=753 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Liran Alon Switch to KVM address space on entry to guest. Most of KVM #VMExit handlers will run in KVM isolated address space and switch back to host address space only before accessing sensitive data. Sensitive data is defined as either host data or other VM data. Currently, we switch back to the host address space on the following scenarios: 1) When handling guest page-faults: As this will access SPTs which contains host PFNs. 2) On schedule-out of vCPU thread 3) On write to guest virtual memory (kvm_write_guest_virt_system() can pull in tons of pages) 4) On return to userspace (e.g. QEMU) 5) On interrupt or exception Signed-off-by: Liran Alon Signed-off-by: Alexandre Chartre --- arch/x86/kvm/mmu.c | 2 +- arch/x86/kvm/vmx/isolation.c | 2 +- arch/x86/kvm/vmx/vmx.c | 6 ++++++ arch/x86/kvm/vmx/vmx.h | 18 ++++++++++++++++++ arch/x86/kvm/x86.c | 34 +++++++++++++++++++++++++++++++++- arch/x86/kvm/x86.h | 1 + 6 files changed, 60 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 98f6e4f..298f602 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -4067,7 +4067,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, { int r = 1; - vcpu->arch.l1tf_flush_l1d = true; + kvm_may_access_sensitive_data(vcpu); switch (vcpu->arch.apf.host_apf_reason) { default: trace_kvm_page_fault(fault_address, error_code); diff --git a/arch/x86/kvm/vmx/isolation.c b/arch/x86/kvm/vmx/isolation.c index d82f6b6..8f57f10 100644 --- a/arch/x86/kvm/vmx/isolation.c +++ b/arch/x86/kvm/vmx/isolation.c @@ -34,7 +34,7 @@ * This is set to false by default because it incurs a performance hit * which some users will not want to take for security gain. */ -static bool __read_mostly address_space_isolation; +bool __read_mostly address_space_isolation; module_param(address_space_isolation, bool, 0444); /* diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d47f093..b5867cc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6458,8 +6458,14 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu) if (vcpu->arch.cr2 != read_cr2()) write_cr2(vcpu->arch.cr2); + /* + * Use an isolation barrier as VMExit will restore the isolation + * CR3 while interrupts can abort isolation. + */ + vmx_isolation_barrier_begin(vmx); vmx->fail = __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, vmx->loaded_vmcs->launched); + vmx_isolation_barrier_end(vmx); vcpu->arch.cr2 = read_cr2(); diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index e8de23b..b65f059 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -531,4 +531,22 @@ static inline void decache_tsc_multiplier(struct vcpu_vmx *vmx) int vmx_isolation_init(struct vcpu_vmx *vmx); void vmx_isolation_uninit(struct vcpu_vmx *vmx); +extern bool __read_mostly address_space_isolation; + +static inline void vmx_isolation_barrier_begin(struct vcpu_vmx *vmx) +{ + if (!address_space_isolation || !vmx->vcpu.asi) + return; + + asi_barrier_begin(); +} + +static inline void vmx_isolation_barrier_end(struct vcpu_vmx *vmx) +{ + if (!address_space_isolation || !vmx->vcpu.asi) + return; + + asi_barrier_end(); +} + #endif /* __KVM_X86_VMX_H */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9857992..9458413 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3346,6 +3346,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) * guest. do_debug expects dr6 to be cleared after it runs, do the same. */ set_debugreg(0, 6); + + kvm_may_access_sensitive_data(vcpu); } static int kvm_vcpu_ioctl_get_lapic(struct kvm_vcpu *vcpu, @@ -5259,7 +5261,7 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) { /* kvm_write_guest_virt_system can pull in tons of pages. */ - vcpu->arch.l1tf_flush_l1d = true; + kvm_may_access_sensitive_data(vcpu); return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, PFERR_WRITE_MASK, exception); @@ -7744,6 +7746,32 @@ void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_GPL(__kvm_request_immediate_exit); +static void vcpu_isolation_enter(struct kvm_vcpu *vcpu) +{ + int err; + + if (!vcpu->asi) + return; + + err = asi_enter(vcpu->asi); + if (err) + pr_debug("KVM isolation failed: error %d\n", err); +} + +static void vcpu_isolation_exit(struct kvm_vcpu *vcpu) +{ + if (!vcpu->asi) + return; + + asi_exit(vcpu->asi); +} + +void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu) +{ + vcpu->arch.l1tf_flush_l1d = true; + vcpu_isolation_exit(vcpu); +} + /* * Returns 1 to let vcpu_run() continue the guest execution loop without * exiting to the userspace. Otherwise, the value will be returned to the @@ -7944,6 +7972,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto cancel_injection; } + vcpu_isolation_enter(vcpu); + if (req_immediate_exit) { kvm_make_request(KVM_REQ_EVENT, vcpu); kvm_x86_ops->request_immediate_exit(vcpu); @@ -8130,6 +8160,8 @@ static int vcpu_run(struct kvm_vcpu *vcpu) srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx); + kvm_may_access_sensitive_data(vcpu); + return r; } diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index a470ff0..69a7402 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -356,5 +356,6 @@ static inline bool kvm_pat_valid(u64 data) void kvm_load_guest_xcr0(struct kvm_vcpu *vcpu); void kvm_put_guest_xcr0(struct kvm_vcpu *vcpu); +void kvm_may_access_sensitive_data(struct kvm_vcpu *vcpu); #endif From patchwork Thu Jul 11 14:25:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11040119 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 183561395 for ; Thu, 11 Jul 2019 14:27:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0668028AC8 for ; Thu, 11 Jul 2019 14:27:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EEC7328AD9; Thu, 11 Jul 2019 14:27:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E7A028AC8 for ; Thu, 11 Jul 2019 14:27:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D559D8E00DC; Thu, 11 Jul 2019 10:27:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D07BB8E00DB; Thu, 11 Jul 2019 10:27:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2F068E00DC; Thu, 11 Jul 2019 10:27:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 9295D8E00DB for ; Thu, 11 Jul 2019 10:27:25 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id c5so6906522iom.18 for ; Thu, 11 Jul 2019 07:27:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=p0TwseF6Rl2ptej9BTwWkpZCi9Mm6El6sXWBWxSv9Wc=; b=cPgqGNZZMUF1vTM/VI0/cyT200JAzNFB96+gCOVBB6nEe8I3mCZufLB5ykwlQTLbOD XqG8C7VZTcoaBry+cQUcG3bCKHose2x0VDoHRTnGMOcygTBd3z1E7R+lKZGweOjuu25U UfLETeN5s/IhbX3+iRkmcjWNrq3kC5ELUSdKOyG8vtRW6Qn6Lu44PDSeEUqAe6PbOHbs d+3B1ECFux+FKuJaJWFD3hh7FmK5AdPcaV2SeJxqK0Mn0/f1n70aEqo8FnhswEp6GJZg 4lE+caJfF0of86qjriFey2Wzn157+PlDHEUQUup2U23KjWEsj5ec/020l8aW1tLdXYNN Q90w== X-Gm-Message-State: APjAAAXXkDdUd2O1eW+cOIaQchMJIefYXJkf22ZoyR6msTjMTfghD0PX BfIoBzCA11gmtJLllGChTVM0j17Aq3+fpzAWaK4/PYquUV98js8RNIPhE7f/ACI4i/2GqNFRyPa 55rqp4Q+xBwMht5Har+KTTK3M7PiaCXlPYWWcVvAsFenRcCZptjnGMqgWYQcW6NnX9Q== X-Received: by 2002:a02:9004:: with SMTP id w4mr4954615jaf.111.1562855245396; Thu, 11 Jul 2019 07:27:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqxY4/hh5m5SARweNKm68n49sZBK3UFUoSJu4Ps0b1/z9Qt52U09oBe4TGGLy7x6gk01Ajlx X-Received: by 2002:a02:9004:: with SMTP id w4mr4954555jaf.111.1562855244728; Thu, 11 Jul 2019 07:27:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1562855244; cv=none; d=google.com; s=arc-20160816; b=VqH8UFfAkBsxOEzKylrcP9/zZucI073ilvxmjfmxLO4nuHq6K8qwcm2kcEeFf1cLk4 VWSIt1E/z1KtEGxMtRGxEHdKrbfvnwS+12UEd4KfQh2Eg/G3uIIWEXAy9mRzbhcuuDIM iP5A65U+5B5CtsATxNESg14v/m9wUYXNZS+VO0yFrYFgrHqeJUR/o0VpRWkPsBUZqUQE MKdjPcNl0RVC/3QCLH9abY5JKphD27bvRkOmoEu+N7g412qkzLEkqemgxX4LYBaTWSR9 07LKELoxSFEQGduUaVfV5uu9yks7eh5eraAaXFLMTTGcfX6ZxKS/Ftm2tMdW+7RKrqk8 GoDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=p0TwseF6Rl2ptej9BTwWkpZCi9Mm6El6sXWBWxSv9Wc=; b=Cp/bG3U+Ov/O9UNn3ZU8FPPDTS4BJ3TwE9+faplUmK9q239ntbbSbyaL/xZ+rvDYmu ouK5QvIiJ0atARW72MdyuamP5gyrG6dGZ1w0zgopeBIG72n87lZHarVGwclR1Z6u7TdV 9bxumpdAg0tcWwakBkT2VxGTRGkjSNSkYqPGnlO5G20I0fcca7es938ZjkLiVp7l9mtN jhlzZ3TOLO/B5bvg+X1dR1WjnOB9kiwgVreNzoNZtpSL6oX6FoT95TpfTE5aLOGePhdA mM31a6c9nsjM4B/yp3ogfQwjABOJoWZMe5KGA5VTpntmQ1aZheuRkJ3ISHXAkGRzk/xZ SQcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=VxnBJQPS; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id b10si7691987ioq.76.2019.07.11.07.27.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Jul 2019 07:27:24 -0700 (PDT) Received-SPF: pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=VxnBJQPS; spf=pass (google.com: domain of alexandre.chartre@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=alexandre.chartre@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x6BEO8vQ001464; Thu, 11 Jul 2019 14:27:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=p0TwseF6Rl2ptej9BTwWkpZCi9Mm6El6sXWBWxSv9Wc=; b=VxnBJQPSh0LmxFyWqVND6G3imbrgiNDc2XYktPKhPxTH7jeK12iQ07uhiR7KSkYssaXb Pv64TstUQ6TZGJ1PMaSuITPpNq63mpZXM6Iv7MEPC/m1XIUd6uAVwnuCuLzWwEkJpk+d bLhzpPXWzEGxFhxH1pYNQvLU4rahHXqpvu4/IcVK0tDJMHq3A7NSNALQrgdnEe+A5MWw HjWBvSrm/StSA0wUZr133cDwxkrZIL+2nTAatg2QWIUq+pis7baTZiMjbcwkfcdkdyAy 8SJFtfEFVLR+g5pFWeVErgqD0yalSI8+c1PK8OfF5ETzFY1qWu5XfRWDWSejjFNNjfXr 8Q== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2tjk2u0e5t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 11 Jul 2019 14:27:15 +0000 Received: from achartre-desktop.fr.oracle.com (dhcp-10-166-106-34.fr.oracle.com [10.166.106.34]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x6BEPcuJ021444; Thu, 11 Jul 2019 14:27:06 GMT From: Alexandre Chartre To: pbonzini@redhat.com, rkrcmar@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, kvm@vger.kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, jwadams@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, alexandre.chartre@oracle.com Subject: [RFC v2 26/26] KVM: x86/asi: Map KVM memslots and IO buses into KVM ASI Date: Thu, 11 Jul 2019 16:25:38 +0200 Message-Id: <1562855138-19507-27-git-send-email-alexandre.chartre@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> References: <1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9314 signatures=668688 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907110162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Map KVM memslots and IO buses into KVM ASI. Mapping is checking on each KVM ASI enter because they can change. Signed-off-by: Alexandre Chartre --- arch/x86/kvm/x86.c | 36 +++++++++++++++++++++++++++++++++++- include/linux/kvm_host.h | 2 ++ 2 files changed, 37 insertions(+), 1 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9458413..7c52827 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7748,11 +7748,45 @@ void __kvm_request_immediate_exit(struct kvm_vcpu *vcpu) static void vcpu_isolation_enter(struct kvm_vcpu *vcpu) { - int err; + struct kvm *kvm = vcpu->kvm; + struct kvm_io_bus *bus; + int i, err; if (!vcpu->asi) return; + /* + * Check memslots and buses mapping as they tend to change. + */ + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + if (vcpu->asi_memslots[i] == kvm->memslots[i]) + continue; + pr_debug("remapping kvm memslots[%d]: %px -> %px\n", + i, vcpu->asi_memslots[i], kvm->memslots[i]); + err = asi_remap(vcpu->asi, &vcpu->asi_memslots[i], + kvm->memslots[i], sizeof(struct kvm_memslots)); + if (err) { + pr_debug("failed to map kvm memslots[%d]: error %d\n", + i, err); + } + } + + + for (i = 0; i < KVM_NR_BUSES; i++) { + bus = kvm->buses[i]; + if (bus == vcpu->asi_buses[i]) + continue; + pr_debug("remapped kvm buses[%d]: %px -> %px\n", + i, vcpu->asi_buses[i], bus); + err = asi_remap(vcpu->asi, &vcpu->asi_buses[i], bus, + sizeof(*bus) + bus->dev_count * + sizeof(struct kvm_io_range)); + if (err) { + pr_debug("failed to map kvm buses[%d]: error %d\n", + i, err); + } + } + err = asi_enter(vcpu->asi); if (err) pr_debug("KVM isolation failed: error %d\n", err); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 2a9d073..1f82de4 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -324,6 +324,8 @@ struct kvm_vcpu { #ifdef CONFIG_ADDRESS_SPACE_ISOLATION struct asi *asi; + void *asi_memslots[KVM_ADDRESS_SPACE_NUM]; + void *asi_buses[KVM_NR_BUSES]; #endif };