From patchwork Tue Nov 17 18:15:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5065BC63697 for ; Tue, 17 Nov 2020 18:18:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E63D522210 for ; Tue, 17 Nov 2020 18:18:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="PO+/keG4"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="i29eJtch" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E63D522210 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QAbYTMsh3ojyWBK+rL9nduBbNgDVsjiJ8FVYO1V7VdQ=; b=PO+/keG4a/Ox+O+lQwG5FemeB 6h5FAz95ulBJj3y4TCSSDrVSlLcPiK/yuZ+8Vngar6Kt4js/kyvcXC1HH1Ioy7M8lpyR2xCjRjDfo mRtJLhI0+igrzrzsvjnP51lhSkc8PqltF38x4HVpmfr4hM6Lvrs+DJk77nNEaQixC2dgkKmXZ7mPh ze5ZM/Qr45KMwMW6LPhGrO/iHf7vvFHeeyDGMPXy7U6K8+GJIpUDVyjHoXKmBT8uIXY5KLYooxB1u 42IxnWOTdbscKNTbPsi6XxbfXFj6YfJ62A1NXkiYxUDonffPzPNmp2EFRE65fVWIpTHloSmOzfq8o JLO4Fj/qg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Ws-0002Eg-WC; Tue, 17 Nov 2020 18:16:35 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wj-0002A9-3j for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:26 +0000 Received: by mail-wm1-x34a.google.com with SMTP id d2so24591wmd.8 for ; Tue, 17 Nov 2020 10:16:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=wfMub1Dq2/zzWvEJdcb9YkOg3Kk4cyIQtTysz5FPZ9k=; b=i29eJtch8pt4MddWICqOePgkq/MwJTbTc/hOyGFj3909KVEBPm5n0cRj+EplVnonKY WbI373E5+4cYXjRgWXgB7yoEj51rISpjpUCCpNubJ4wb3DMp/jUm/ja856YXPM1roevU DCSoS5kmz/1ross0mm9LO5aO5Df0pZvR0PhjdaR8ldJFyvyqiVcNi/BAVeb15byHXn9C 8IMifCCt418cG8UX7UZpo+rA5552KYu5sl1fjIkWTaOGhboJk+EaVtU77VEJh1fdvKKP 1csKvBXdENtZrlIOODJUyxJZCWUu3IhGdceC5DZsOslbPK2hFkR/d4psXG8fA7BWlsm1 lmjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wfMub1Dq2/zzWvEJdcb9YkOg3Kk4cyIQtTysz5FPZ9k=; b=cq5q3D0q0HrLF1T/kuflIETbn+MKnBiO4tG/L3JlftxEPepw9RxS3ffwqXH90psEro yT2QRCDYdzsNMw9kYCbRXJmpiIe1OTpEhsIV4RvPFYQzhLF2pwyXakINznBedFMeTuly qDJDgimkodzsn3KW1LhedOa2f/FBaIb/YxICTvijI+DRM63LeGwJQ7AXrAHATZKylGff adV4pzaeh6wy1T4flvaLawLEE/ehCOU+NMUh+NZt8JK0Iy64gEtGc3TEATRs0CPlaN8Y k0p39rtW5OwLYwcfxW4d6/xBNPLvk64MSYn2JEmMRqd5wC9ldUHIeWrhQzOXkLwHjTCo remw== X-Gm-Message-State: AOAM530YlTLO2LtG1kV7oyfweWzYwxyTgoR9rglXspmnlkzxCmlznBMs Fs/xjMPIFcP0ZHQFgBR/+JDLRTA4grlZ X-Google-Smtp-Source: ABdhPJwLzQmM3f+/yXxfkHBpPHMIy89xImU6xwZzUPy415BoTaClg6AeXK5YHYslpqgilPliowk09nC1PT5L X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:3d54:: with SMTP id k81mr389168wma.144.1605636981668; Tue, 17 Nov 2020 10:16:21 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:41 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-2-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 01/27] arm64: lib: Annotate {clear, copy}_page() as position-independent From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131625_226303_136F36B7 X-CRM114-Status: GOOD ( 10.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon clear_page() and copy_page() are suitable for use outside of the kernel address space, so annotate them as position-independent code. Signed-off-by: Will Deacon --- arch/arm64/lib/clear_page.S | 4 ++-- arch/arm64/lib/copy_page.S | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S index 073acbf02a7c..b84b179edba3 100644 --- a/arch/arm64/lib/clear_page.S +++ b/arch/arm64/lib/clear_page.S @@ -14,7 +14,7 @@ * Parameters: * x0 - dest */ -SYM_FUNC_START(clear_page) +SYM_FUNC_START_PI(clear_page) mrs x1, dczid_el0 and w1, w1, #0xf mov x2, #4 @@ -25,5 +25,5 @@ SYM_FUNC_START(clear_page) tst x0, #(PAGE_SIZE - 1) b.ne 1b ret -SYM_FUNC_END(clear_page) +SYM_FUNC_END_PI(clear_page) EXPORT_SYMBOL(clear_page) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index e7a793961408..29144f4cd449 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -17,7 +17,7 @@ * x0 - dest * x1 - src */ -SYM_FUNC_START(copy_page) +SYM_FUNC_START_PI(copy_page) alternative_if ARM64_HAS_NO_HW_PREFETCH // Prefetch three cache lines ahead. prfm pldl1strm, [x1, #128] @@ -75,5 +75,5 @@ alternative_else_nop_endif stnp x16, x17, [x0, #112 - 256] ret -SYM_FUNC_END(copy_page) +SYM_FUNC_END_PI(copy_page) EXPORT_SYMBOL(copy_page) From patchwork Tue Nov 17 18:15:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 415C5C64E69 for ; Tue, 17 Nov 2020 18:17:10 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C042024686 for ; Tue, 17 Nov 2020 18:17:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="akycev3n"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Dnk6DEKL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C042024686 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1SurI9ARHhF/5Vjax+6e2gDdGrmObI9y59A8lJUEUq8=; b=akycev3ns8aJFyFA9gOyIBzwc FNS9cw+kom0ogmkLAUHuqpUaIxHUekh/0Wpu0qbvY4ldNuPDEV0AfBJtQ06CiYpVdwqNhFsObhqrm zrgsrAe8aHWexBmuelYLHNv00qUDWC3bUsqymfBStc9pQK/0YHBvu7anYLAiH2yJHGc4fXAgr7CN6 UHOfp56uTS2KnqFDp0UDfQjQgv8aqoX2gqaIUvvfKmyJUrwEnZLD3BovMj+UMXhpXneW5tKTS3lZm t12rPMggzf8AFC4xLfeLpfvzgDX7JsA2NVe88J9EtOzzba9tV7FaAtYBMpTHRbEXJgPOZR257g837 jfeT+OEpQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wx-0002Fv-OY; Tue, 17 Nov 2020 18:16:39 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wl-0002Am-S6 for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:32 +0000 Received: by mail-qt1-x849.google.com with SMTP id g12so13045607qtc.15 for ; Tue, 17 Nov 2020 10:16:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BJadavlyJaEECXi/UfC92ghmWl3ZHk3QkybvjorYO/E=; b=Dnk6DEKLEHkZgTwkTbD6MYFgmWYFpJl2OGFyj4gCuoFYePt7Mtzuqdl8/Pq1pvD11U BWX8ZkLWbm00UHzCwQKUlwmt7cxlXEikfwIeaywi0jAmubiAVQC6UFZRSD64nhv4OajC 8GM+OO1cEmHVMjDnBj1yjfr+CpFs5FuisgZcCrst6Kti8soVn7OtELYLgmoHevMNGj7L Ui7OZS+hF5d9GnfYGHSfJS66fNDOL0URBhlRZdoDaUHxQZNAySSvA8fmrx/lkyK/2roN 5HV6l22AiuuUqQ38FO7K47r4nUDftLLbmFziAUUXdVz5RHqS++6+XcMJfWYt3sxg7qBB Li+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BJadavlyJaEECXi/UfC92ghmWl3ZHk3QkybvjorYO/E=; b=tLGmi+8cA0j6MrMbeRtMW9LU9kjs2rsL0MtB1/2HQv8+maSSt8U5Tcn+AIgrP9izGH FH4OTg225S4KFlRPl9y2wyRaAmJYH08uU34ey0U+1sD13Iq9P23aLQNJTIM8H/r/3s0o /6KnDUv2hdD1jID/6IGG2I2O5KnWFs4Wkgf7xluCIQg3nGqS0aifGwiaMcHEa8/XYmNY es6GQHWYrirgfCUmIJJiY8eVt5BQJZ3pgomk4Ug0F2tOz9Txs5nYR3CHppqrBOvJytXg DZAV6+WnQ7yzTMCN7REp0RASbwBCZJRwfmzoq2itGeX1wsmiHjfZlKEZuO0q+hfedV4t 4bkA== X-Gm-Message-State: AOAM5323yV4yWCL0Kb/GtgB5zn8zPeDlbmBZYQ5+Yb1dEbxjtOrhBGaL nSkMbtp8+QnKzfjsp3DxHA7o6PS0i8ui X-Google-Smtp-Source: ABdhPJwADVZ7KA4lluf+rS88dHojtzpcrwFUrsFZ5Z/Kf+n3XgZdS9/NeT5dfwme4Jn7l/egpgFYp8Qa6c6N X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:cd92:: with SMTP id v18mr884407qvm.47.1605636983901; Tue, 17 Nov 2020 10:16:23 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:42 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-3-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 02/27] KVM: arm64: Link position-independent string routines into .hyp.text From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131628_011289_CA18D2A4 X-CRM114-Status: GOOD ( 12.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon Pull clear_page(), copy_page(), memcpy() and memset() into the nVHE hyp code and ensure that we always execute the '__pi_' entry point on the offchance that it changes in future. [ qperret: Commit title nits ] Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kernel/image-vars.h | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 ++++ 2 files changed, 15 insertions(+) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8539f34d7538..dd8ccc9efb6a 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -105,6 +105,17 @@ KVM_NVHE_ALIAS(__stop___kvm_ex_table); /* Array containing bases of nVHE per-CPU memory regions. */ KVM_NVHE_ALIAS(kvm_arm_hyp_percpu_base); +/* Position-independent library routines */ +__kvm_nvhe_clear_page = __kvm_nvhe___pi_clear_page; +__kvm_nvhe_copy_page = __kvm_nvhe___pi_copy_page; +__kvm_nvhe_memcpy = __kvm_nvhe___pi_memcpy; +__kvm_nvhe_memset = __kvm_nvhe___pi_memset; + +#ifdef CONFIG_KASAN +__kvm_nvhe___memcpy = __kvm_nvhe___pi_memcpy; +__kvm_nvhe___memset = __kvm_nvhe___pi_memset; +#endif + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 1f1e351c5fe2..590fdefb42dd 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -6,10 +6,14 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ +lib-objs := clear_page.o copy_page.o memcpy.o memset.o +lib-objs := $(addprefix ../../../lib/, $(lib-objs)) + obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o +obj-y += $(lib-objs) ## ## Build rules for compiling nVHE hyp code From patchwork Tue Nov 17 18:15:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50891C2D0E4 for ; Tue, 17 Nov 2020 18:19:16 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D1DBD221FD for ; Tue, 17 Nov 2020 18:19:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Lpkezyi3"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="m4MI8sDp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D1DBD221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7pbcgNeCMqY1DreG27J2+hnJrRdl9r5rd6DyE1hns8Q=; b=Lpkezyi389GJyrF/F1POTCnRU oXugegf8sb2ag5/oiySCQ/MV1953RM9edeoLd77wWCbKJpjmRO60jGfmkk9VIIZIEOiFGeWB+WT0m 7ddOMNG3/3M2wyvk/IaXAeO4m7klvIRxsisolMrAWHpZ9Po50PnQljDs90WLZBwKzm9TAFYF5hlIe rs+gRzzkaCAXWmEHOMAqvpawOGSh2Xj6Up2/K2Zl7Db1zUk4r54jfI9DaycTKhJGLRux0r3DLbiQU CY5sVFXjc8whFd3j5MERiyczBGwHvPp+iRY0spGEEXti3k1TxoD9Jx4oaWSILhw/Yi8LbIL1ZcdzX j4Ty7Mnzw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Xp-0002Y3-4g; Tue, 17 Nov 2020 18:17:33 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wn-0002Bc-Mc for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id u9so1907584wmb.2 for ; Tue, 17 Nov 2020 10:16:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=NKT870NUlC/fR/KT4iQDh391rPBCM0t1P6CT6nmxduk=; b=m4MI8sDpmwsqB62UcR7cNmGEWAr1ts32ZyLwXA1tnB9t+yJAMMGil4BHA5iGFJ7isM umGqL0u0iLk+qAebFVa2/UT/SQWeeh7tni7O556SIB6aDKkpJxCYHRGUoUB/25AJyogM m0GtNJ9pve1RrMyvX1nGKvXZMC5UI2L2akL8ld7Q4/y+8HqxmmV9yUUCBjG3B1alizqu 8zjAts8U7BaAYdyVNKihGgQdLMb6oQFGE2FpINJY51qA6o8yP59LmhW7Ib9PbyrhYwXD MkzMfv1BcHdsGpSFaKYYevM+SLgZoP9bPFgKtRVWF2iuGJDAmZDeIv19lHS+uvxwOZfN +4Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NKT870NUlC/fR/KT4iQDh391rPBCM0t1P6CT6nmxduk=; b=tPAWU4MMKxUBGdD1iKZRkKvCh5TrUylNTI2JorhXvpRdPZUelMLGyfvhtm8eZlzAkY x/H/LGYh6K8WXureZu5ME8vl2xU4PfkQu5L5CcSpiy9YgHyTPGxcDF5jgSL9VYCMqC98 DMGxXJfSaRXw3M4pW111Gvl9tC7QLzSJJMeIr1xD8bRzWSaEQLTvVNU/Ze6eGEnczNcS sChnf1URRSbR3TKTbRMBZvp4WNe8mcEUIpmeap+v/p4/kTsWCFmyzX6bGHVF9Ungw2mI uKnhK9CT2aAsIDlEYE06A5lCfJse8gFZ0AeHwyPnRux/E8hgWO3o3FJwIey1zgREO5yh qM0A== X-Gm-Message-State: AOAM533N7w103jrR2aVvLCM2dG7VueqKtlWf4V2f4R0iue0BiRJJrZH0 F4SWHbBYBjIMhYy1ZsQbwL3C1ysIA8uL X-Google-Smtp-Source: ABdhPJwaipw8Yg+hF0IOY//Mk8QfB1T5Et/1Qnq5qCttJhd99HX+QQmEaSPNLhuQ7z/N6ynjdi4lLhEzca6o X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:bc08:: with SMTP id m8mr360694wmf.137.1605636986141; Tue, 17 Nov 2020 10:16:26 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:43 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-4-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 03/27] KVM: arm64: Add standalone ticket spinlock implementation for use at hyp From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131629_907933_38F2D633 X-CRM114-Status: GOOD ( 16.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon We will soon need to synchronise multiple CPUs in the hyp text at EL2. The qspinlock-based locking used by the host is overkill for this purpose and requires a working "percpu" implementation for the MCS nodes. Implement a simple ticket locking scheme based heavily on the code removed by c11090474d70 ("arm64: locking: Replace ticket lock implementation with qspinlock"). [ qperret: removed the __KVM_NVHE_HYPERVISOR__ build-time check from spinlock.h ] Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/spinlock.h | 95 ++++++++++++++++++++++ arch/arm64/kvm/hyp/include/nvhe/util.h | 25 ++++++ 2 files changed, 120 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/spinlock.h create mode 100644 arch/arm64/kvm/hyp/include/nvhe/util.h diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h new file mode 100644 index 000000000000..bbfe2cbd9f62 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * A stand-alone ticket spinlock implementation, primarily for use by the + * non-VHE hypervisor code running at EL2. + * + * Copyright (C) 2020 Google LLC + * Author: Will Deacon + * + * Heavily based on the implementation removed by c11090474d70 which was: + * Copyright (C) 2012 ARM Ltd. + */ + +#ifndef __ARM64_KVM_HYP_SPINLOCK_H__ +#define __ARM64_KVM_HYP_SPINLOCK_H__ + +#include + +typedef union hyp_spinlock { + u32 __val; + struct { +#ifdef __AARCH64EB__ + u16 next, owner; +#else + u16 owner, next; +#endif + }; +} hyp_spinlock_t; + +#define hyp_spin_lock_init(l) \ +do { \ + *(l) = (hyp_spinlock_t){ .__val = 0 }; \ +} while (0) + +static inline void hyp_spin_lock(hyp_spinlock_t *lock) +{ + u32 tmp; + hyp_spinlock_t lockval, newval; + + asm volatile( + /* Atomically increment the next ticket. */ + ALTERNATIVE( + /* LL/SC */ +" prfm pstl1strm, %3\n" +"1: ldaxr %w0, %3\n" +" add %w1, %w0, #(1 << 16)\n" +" stxr %w2, %w1, %3\n" +" cbnz %w2, 1b\n", + /* LSE atomics */ +" .arch_extension lse\n" +" mov %w2, #(1 << 16)\n" +" ldadda %w2, %w0, %3\n" + __nops(3), + ARM64_HAS_LSE_ATOMICS) + + /* Did we get the lock? */ +" eor %w1, %w0, %w0, ror #16\n" +" cbz %w1, 3f\n" + /* + * No: spin on the owner. Send a local event to avoid missing an + * unlock before the exclusive load. + */ +" sevl\n" +"2: wfe\n" +" ldaxrh %w2, %4\n" +" eor %w1, %w2, %w0, lsr #16\n" +" cbnz %w1, 2b\n" + /* We got the lock. Critical section starts here. */ +"3:" + : "=&r" (lockval), "=&r" (newval), "=&r" (tmp), "+Q" (*lock) + : "Q" (lock->owner) + : "memory"); +} + +static inline void hyp_spin_unlock(hyp_spinlock_t *lock) +{ + u64 tmp; + + asm volatile( + ALTERNATIVE( + /* LL/SC */ + " ldrh %w1, %0\n" + " add %w1, %w1, #1\n" + " stlrh %w1, %0", + /* LSE atomics */ + " .arch_extension lse\n" + " mov %w1, #1\n" + " staddlh %w1, %0\n" + __nops(1), + ARM64_HAS_LSE_ATOMICS) + : "=Q" (lock->owner), "=&r" (tmp) + : + : "memory"); +} + +#endif /* __ARM64_KVM_HYP_SPINLOCK_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/util.h b/arch/arm64/kvm/hyp/include/nvhe/util.h new file mode 100644 index 000000000000..9c58cc436a83 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/util.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Standalone re-implementations of kernel interfaces for use at EL2. + * Copyright (C) 2020 Google LLC + * Author: Will Deacon + */ + +#ifndef __KVM_NVHE_HYPERVISOR__ +#error "Attempt to include nVHE code outside of EL2 object" +#endif + +#ifndef __ARM64_KVM_NVHE_UTIL_H__ +#define __ARM64_KVM_NVHE_UTIL_H__ + +/* Locking (hyp_spinlock_t) */ +#include + +#undef spin_lock_init +#define spin_lock_init hyp_spin_lock_init +#undef spin_lock +#define spin_lock hyp_spin_lock +#undef spin_unlock +#define spin_unlock hyp_spin_unlock + +#endif /* __ARM64_KVM_NVHE_UTIL_H__ */ From patchwork Tue Nov 17 18:15:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6084C2D0E4 for ; Tue, 17 Nov 2020 18:18:15 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D69B221FD for ; Tue, 17 Nov 2020 18:18:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="sOsmuL3g"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="vS+rvB1/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D69B221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UauljW88FSPylenYCoZjxF0liHi+v1RU5CimTbEeOHo=; b=sOsmuL3g52LU+h+1tXwARayls YW/a4NXHlfwOmflBIr9/GERXk94HCg1IrbCNff3H/LorrWPqcvv7ZhVnJhAM3ed6imx0WGhKMc9BN yLnh4u8Q2y+0brBwLXkkBTdj+qroA0pw1JcxZWHF/ekRqYHGJMELKGWpAgZT3KDNDT6IL/DTHZqCP vLI0fGwH3uM7CWqOxgWHAM4rLWZyJDWCCnRUIxZlQaeroo8XxOrLmDYP+kYugyIAOE1I/dBISUygM d+6oG+VkcB7lnKhhY5I3XgaxgPYaTrWvXzgc0rMvfYFwjE9BnhLV7DfIGNBq4Ya844Ht4Sws6NAhe xZLRLLVkA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5X7-0002JR-HY; Tue, 17 Nov 2020 18:16:49 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wo-0002Cl-UR for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id d2so24710wmd.8 for ; Tue, 17 Nov 2020 10:16:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=IZC56EUbcZGLv67uWppNeQH9/JsQWBJOLy6w8MLjYOk=; b=vS+rvB1/6KGnGkdgssdAFqr2pdbbPpSdTO3v0TbihY/ScWy+IKWsAhYSb+9u1xIoQY iJ5ArdUb3OBFMAalNNv9B0qc726x3h1bvk8i+Bt1in0DvI/BetdAiJgv2iq6+11P85K3 POOkjSLXMrr5CvPC7Usq08l9ERlJLMKxBdTUq+O3qFMGt1wjdx9cjLeG1FpZm0WbcGgI gWEdRNRf4cc7JUlIQ8kW/6J7K8sBCpWEHsM8GS/TPC0z9RhMv2am4tyeUPp/P/FUsJ0h xuW1/Ugcc9nP8aKCuNn+d2qK3dxsz5m2Qxz8K4SfKxYCDJLLMPLQDJAyyK1BKwJgdl4V VQOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IZC56EUbcZGLv67uWppNeQH9/JsQWBJOLy6w8MLjYOk=; b=ljYMFxYJzE7V1Yi9mGx0M+Lo85CQ40VfpmNgzpa0mX3AK8FZlXadlHl033GQV/lLLj dqUC7WkGBZrVmPjLUrfIuAb+bF86pCzB2PMukXg+IIW1bbS6dS161TEXYq96xzhCePJ6 Wj8qCQ2GSIgANJrxqiHOGdjyO+4IOLiWuO7t/CaEF0fMNMiCXRRDfLROCImbAg6tjZz+ 6JkZBMl4XH2iln78balb1ml+TDNsm4DrV0Sl+mKTmktBVbwTvWTjku1rkjcxShUcfxtE 4fXPg/ij0ekqOz8iCA6k2kV3tH/A2AxwSrl0/XR9DFCHXEvnD60FWkIAURGmT7DDF4GL W/Uw== X-Gm-Message-State: AOAM533HI7/To4sfrHl6n4/pweASX2zkSXu0pr7RpNYDzblQz6AwGnma XZ1Ev4E4QTPHd7bzYFesyyzD1Snmtz/1 X-Google-Smtp-Source: ABdhPJzcNUt5sppuGQvEX0isAFbepTz9C9E9Dic3GRImbsOwvG5qyl5LhxGBa5f3ag7x2AHxrvf7fK/v7tNU X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a7b:cd10:: with SMTP id f16mr348760wmj.69.1605636988252; Tue, 17 Nov 2020 10:16:28 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:44 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-5-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 04/27] KVM: arm64: Initialize kvm_nvhe_init_params early From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131631_061593_157C693F X-CRM114-Status: GOOD ( 15.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the initialization of kvm_nvhe_init_params in a dedicated function that is run early, and only once during KVM init, rather than every time the KVM vectors are set and reset. This also opens the opportunity for the hypervisor to change the init structs during boot, hence simplifying the replacement of host-provided page-tables and stacks by the ones the hypervisor will create for itself. Signed-off-by: Quentin Perret --- arch/arm64/kvm/arm.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index d6d5211653b7..7335eb4fb0bd 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1355,24 +1355,20 @@ static int kvm_init_vector_slots(void) return 0; } -static void cpu_init_hyp_mode(void) +static void cpu_prepare_hyp_mode(int cpu) { - struct kvm_nvhe_init_params *params = this_cpu_ptr_nvhe_sym(kvm_init_params); - struct arm_smccc_res res; - - /* Switch from the HYP stub to our own HYP init vector */ - __hyp_set_vectors(kvm_get_idmap_vector()); + struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); /* * Calculate the raw per-cpu offset without a translation from the * kernel's mapping to the linear mapping, and store it in tpidr_el2 * so that we can use adr_l to access per-cpu variables in EL2. */ - params->tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe_sym(__per_cpu_start) - + params->tpidr_el2 = (unsigned long)per_cpu_ptr_nvhe_sym(__per_cpu_start, cpu) - (unsigned long)kvm_ksym_ref(CHOOSE_NVHE_SYM(__per_cpu_start)); params->vector_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref(__kvm_hyp_host_vector)); - params->stack_hyp_va = kern_hyp_va(__this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE); + params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->entry_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref(__kvm_hyp_psci_cpu_entry)); params->pgd_pa = kvm_mmu_get_httbr(); @@ -1381,6 +1377,15 @@ static void cpu_init_hyp_mode(void) * be read while the MMU is off. */ __flush_dcache_area(params, sizeof(*params)); +} + +static void cpu_init_hyp_mode(void) +{ + struct kvm_nvhe_init_params *params; + struct arm_smccc_res res; + + /* Switch from the HYP stub to our own HYP init vector */ + __hyp_set_vectors(kvm_get_idmap_vector()); /* * Call initialization code, and switch to the full blown HYP code. @@ -1389,6 +1394,7 @@ static void cpu_init_hyp_mode(void) * cpus_have_const_cap() wrapper. */ BUG_ON(!system_capabilities_finalized()); + params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); @@ -1742,6 +1748,12 @@ static int init_hyp_mode(void) init_cpu_logical_map(); init_psci_relay(); + /* + * Prepare the CPU initialization parameters + */ + for_each_possible_cpu(cpu) + cpu_prepare_hyp_mode(cpu); + return 0; out_err: From patchwork Tue Nov 17 18:15:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9955AC2D0E4 for ; Tue, 17 Nov 2020 18:18:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 10BF622210 for ; Tue, 17 Nov 2020 18:18:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Bi89wB2Q"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="lNGeNA+z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10BF622210 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=U/5A+bEQThfyiRKqaeRBoVNAMciCJFVeczGyCG7fF8I=; b=Bi89wB2QtoKsqXZKlC+eVtt7K IMISiJPwCOUYJtgu8wh71q1TD1skaKzEhSFDDBEWX0eiEmkXmQat/tdMg8SQ+kC9eCyv3rIi9DIr7 XbNSSZGc1NdS2gVMo5paL6nLwOfO91TEm/DWsvpbbajUR4fe+HlQHPdHOMcnS94YHXU23dZkODnsu 6PoLCcciUFgFQwlYR1K/jpZyWH9Jv9abWvzxqJqZPhXV5+3ZvmAvP3bs60YfuIwmkY0aIhAyisjM1 2PBWFOYlv/QszV+ksEZFql3Etpmls8vs9kUKPrpBsPjKxVxjlqU+OKyPqsusM3mEoLuIWMf9C75E7 Y8LnXgbOw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Y7-0002jd-HR; Tue, 17 Nov 2020 18:17:51 +0000 Received: from mail-ed1-x54a.google.com ([2a00:1450:4864:20::54a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Ws-0002Dm-7x for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:35 +0000 Received: by mail-ed1-x54a.google.com with SMTP id n25so10080065edr.20 for ; Tue, 17 Nov 2020 10:16:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=NAAv66aNUaltEt0mONTaW9BQUFoOFbMtL1i8hZDZuGs=; b=lNGeNA+zGfHwCaxYy0TrJSE1pl2oq3vvwMkxyPjR74vzMfWQMK1AHp2yCHnWdNCzXT SKJ0EyWSPKtiPU3Wh2G9XDOR5Y1cjmB7NRaJEONE0OXpf5p3XNmXzb90URbBEsu0OiMG KuXcCgSExkU9INz54eIwLV4KcmNLH/I9a0/DDvClup2XnMQzTU+/aAnYh5EhesGt/A7s qFxI65OjN5RHOHpWYJu5e+gN57HAzMgmW6bGR1CZy3/rKzcs8kJUQWw+4bFdSnzI54qo Py6oernt+i8KqdH6xBSdPV2GkNn6JTGC8WeFqhURYfWCHRkbLG4rP+XZYrsjO7KLM2cu CPCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NAAv66aNUaltEt0mONTaW9BQUFoOFbMtL1i8hZDZuGs=; b=l/UuCcrtOr24ou5pHvexaXRN6OOalNT3JvT2yLG97F43Oxjnx1fK8aFgwVybqTcC7+ 018/6TnE1KQLBTUi3XrJIglpX1N50rHIrMuS3omz79vZnb1eD3qZtuDsMzCQIJfIQE+m baeE8A28ohkWTW4VPciv/+Ye5VxB78WSZf78Vjx03wzakfunMOiCmuQwbQlum67rmeZ2 ZBsrLQ/kKm4CwfXXm3YCItKzv0vmpH6ar/t5pRCK+ihgKrPKLVHyAVULJPCCWi+4srDh Eaq2w9Z2dB/D51OIOyse6UwdUsgatqejdM3xjWFUMnMKrmlCUFEk+ZqsXNy8uWuS4/yE bECA== X-Gm-Message-State: AOAM530GwkWpk536PjcB0NNj5L6TyfG3kZVz2H9NNvqsSODoNG95kgRC TF29p/JK9NACnjGNw4lUKOOP3idF3uyr X-Google-Smtp-Source: ABdhPJwRKjykeVPq3BUBGAvQxs6YtxOlWZ1XX97g5ZVMcWNcwBONleeq9otY9dDDVDJIIzj6jTLQHcAJgIdB X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a17:906:6414:: with SMTP id d20mr19849332ejm.82.1605636990564; Tue, 17 Nov 2020 10:16:30 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:45 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-6-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 05/27] KVM: arm64: Avoid free_page() in page-table allocator From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131634_354057_0EE4490F X-CRM114-Status: GOOD ( 14.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the KVM page-table allocator uses a mix of put_page() and free_page() calls depending on the context even though page-allocation is always achieved using variants of __get_free_page(). Make the code consitent by using put_page() throughout, and reduce the memory management API surface used by the page-table code. This will ease factoring out page-alloction from pgtable.c, which is a pre-requisite to creating page-tables at EL2. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 0271b4a3b9fe..d7122c5eac24 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -410,7 +410,7 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); return 0; } @@ -422,7 +422,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - free_page((unsigned long)pgt->pgd); + put_page(virt_to_page(pgt->pgd)); pgt->pgd = NULL; } @@ -551,7 +551,7 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); put_page(virt_to_page(ptep)); if (data->anchor == ptep) { @@ -674,7 +674,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, } if (childp) - free_page((unsigned long)childp); + put_page(virt_to_page(childp)); return 0; } @@ -871,7 +871,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, put_page(virt_to_page(ptep)); if (kvm_pte_table(pte, level)) - free_page((unsigned long)kvm_pte_follow(pte)); + put_page(virt_to_page(kvm_pte_follow(pte))); return 0; } From patchwork Tue Nov 17 18:15:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14D3AC5519F for ; Tue, 17 Nov 2020 18:19:33 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8DFAD22210 for ; Tue, 17 Nov 2020 18:19:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ESa1iJtM"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="H6jcaREl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8DFAD22210 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=v3gTJWpxrfdG1+B55QrCa9WA82eujRFdCsHGE10oiGQ=; b=ESa1iJtMI1cZwU+YA1HhCKqv4 iBMc7ijXT9Op1pM41Ko2sX6NcxRDl7ie6wN5q9Mrw3eHDgw170PYU6PGIoKM3Bt0eoo+KC2GqHMFE cwUwsADOB63uQq1/W+Elvsa3ZlXdrqvwS6nqp//c4KwOFoYGBtyoug/KOlo4SBnDmr3krMV1FKwXz P1P4ta4ofzetQ6FVTOd9CMw9Z4FskYGKz2zcE8h4GoPv87ZSHK7ZaotxLaM/TY3PtEzMxAYqU/0a+ q45Ex7uN4+Sv0MinHyUywv/WhjK9WAQ6rBEcjaYOI41DpiduCeln/9RuLipRJ8DoODJaU+1h1v5SH OhNTHotQA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5YC-0002n3-Cw; Tue, 17 Nov 2020 18:17:56 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wt-0002EJ-VH for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:38 +0000 Received: by mail-wm1-x34a.google.com with SMTP id y187so2137391wmy.3 for ; Tue, 17 Nov 2020 10:16:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=//vCyfZoaOPaFjgyWzys1wYaNowSCAo4t4+N8nYKkZo=; b=H6jcaRElEqs5hXb6bMgihDYyDKrO1+Gy39SzLEUcsQxAEe1/+XJj5jn4B6m6NNiohO HbyYnLLerUZarsKwXbsDAJ4/6M0USS2sso/rDZqDTMN+Baf7agbKN8pu+Lu1SLVwG49I byTOch55idOBIg7fPDTJPo3au0uQts5gMSL5lJmae0aOz2D7+y4ZaVOxUvD1RM6supnq d4BniXHYjBAcN4RlJp64XVz48uEOjEI9UOOe0pKXiQib3vE6jH5c2ofYICMdjuX2sRTV SqZaIE18NDJPLAvCbSn78T1HjmYZT7juswbejz9o8kWzKSJv2vgJ8DHFJAxigSQ4gDh8 ugkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=//vCyfZoaOPaFjgyWzys1wYaNowSCAo4t4+N8nYKkZo=; b=S6ycxsERxY7d9r9rdPLzJmK9h5IVMpBJdnrSJQ/00BmLpUAPAs1FjzgDDFCQXnaZEz 6W1VBdEGm0bDzBdkrKmRRG6OedSiuU2ydC2AFgaudQ5nKyMqAImSn7gTxFcvMOHRe3c1 /nsZAklIS0a5WYz6zjmGAWfGHs180A5U+WBTlXPjdeEqytchVI49a5PI76aGNXOzC7ni SDP+Ko7FX1N6ccB65ldmTO4NOozG3Qs9ZV2hDTpqYuIqNE2UipKYG+9KJFAgp83ITWMv uXX+L6oT3f5ecy0nY5/Rv0eW7ENytA4F/MWGa1OYOURompO0i6WUY/fIkDvQSVEbaBYr ShMw== X-Gm-Message-State: AOAM5335XAVTQZQYKT+yIB3usGok6eoYfM5fTdMCAoHUJCY1td4deYOM 5RFbaWgVelHNlRcFkWKSS97lItYg7KAy X-Google-Smtp-Source: ABdhPJyLtyr+86yek0HKSyIi+DcPvHQUWebfawW5hYO7d4fC4yEczgrhA5sZg6LawzMfNVzNCGXXrJz0FPJ2 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a5d:5291:: with SMTP id c17mr770864wrv.311.1605636992995; Tue, 17 Nov 2020 10:16:32 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:46 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-7-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 06/27] KVM: arm64: Factor memory allocation out of pgtable.c From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131636_102125_B2469BA6 X-CRM114-Status: GOOD ( 19.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for enabling the creation of page-tables at EL2, factor all memory allocation out of the page-table code, hence making it re-usable with any compatible memory allocator. No functional changes intended. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 32 +++++++++- arch/arm64/kvm/hyp/pgtable.c | 90 +++++++++++++++++----------- arch/arm64/kvm/mmu.c | 70 +++++++++++++++++++++- 3 files changed, 154 insertions(+), 38 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 52ab38db04c7..45acc9dc6c45 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -13,17 +13,41 @@ typedef u64 kvm_pte_t; +/** + * struct kvm_pgtable_mm_ops - Memory management callbacks. + * @zalloc_page: Allocate a zeroed memory page. + * @zalloc_pages_exact: Allocate an exact number of zeroed memory pages. + * @free_pages_exact: Free an exact number of memory pages. + * @get_page: Increment the refcount on a page. + * @put_page: Decrement the refcount on a page. + * @page_count: Returns the refcount of a page. + * @phys_to_virt: Convert a physical address into a virtual address. + * @virt_to_phys: Convert a virtual address into a physical address. + */ +struct kvm_pgtable_mm_ops { + void* (*zalloc_page)(void *arg); + void* (*zalloc_pages_exact)(size_t size); + void (*free_pages_exact)(void *addr, size_t size); + void (*get_page)(void *addr); + void (*put_page)(void *addr); + int (*page_count)(void *addr); + void* (*phys_to_virt)(phys_addr_t phys); + phys_addr_t (*virt_to_phys)(void *addr); +}; + /** * struct kvm_pgtable - KVM page-table. * @ia_bits: Maximum input address size, in bits. * @start_level: Level at which the page-table walk starts. * @pgd: Pointer to the first top-level entry of the page-table. + * @mm_ops: Memory management callbacks. * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. */ struct kvm_pgtable { u32 ia_bits; u32 start_level; kvm_pte_t *pgd; + struct kvm_pgtable_mm_ops *mm_ops; /* Stage-2 only */ struct kvm_s2_mmu *mmu; @@ -86,10 +110,12 @@ struct kvm_pgtable_walker { * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table. * @pgt: Uninitialised page-table structure to initialise. * @va_bits: Maximum virtual address bits. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits); +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_hyp_destroy() - Destroy an unused hypervisor stage-1 page-table. @@ -126,10 +152,12 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. * @kvm: KVM structure representing the guest virtual machine. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm); +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index d7122c5eac24..61a8a34ddfdb 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -148,9 +148,9 @@ static kvm_pte_t kvm_phys_to_pte(u64 pa) return pte; } -static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte) +static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) { - return __va(kvm_pte_to_phys(pte)); + return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); } static void kvm_set_invalid_pte(kvm_pte_t *ptep) @@ -159,9 +159,10 @@ static void kvm_set_invalid_pte(kvm_pte_t *ptep) WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID); } -static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp) +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, + struct kvm_pgtable_mm_ops *mm_ops) { - kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(__pa(childp)); + kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE); pte |= KVM_PTE_VALID; @@ -229,7 +230,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, goto out; } - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, data->pgt->mm_ops); ret = __kvm_pgtable_walk(data, childp, level + 1); if (ret) goto out; @@ -304,8 +305,9 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, } struct hyp_map_data { - u64 phys; - kvm_pte_t attr; + u64 phys; + kvm_pte_t attr; + struct kvm_pgtable_mm_ops *mm_ops; }; static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -355,6 +357,8 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { kvm_pte_t *childp; + struct hyp_map_data *data = arg; + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg)) return 0; @@ -362,11 +366,11 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; - childp = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + childp = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!childp) return -ENOMEM; - kvm_set_table_pte(ptep, childp); + kvm_set_table_pte(ptep, childp, mm_ops); return 0; } @@ -376,6 +380,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, int ret; struct hyp_map_data map_data = { .phys = ALIGN_DOWN(phys, PAGE_SIZE), + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = hyp_map_walker, @@ -393,16 +398,18 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, return ret; } -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops) { u64 levels = ARM64_HW_PGTABLE_LEVELS(va_bits); - pgt->pgd = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + pgt->pgd = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = va_bits; pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels; + pgt->mm_ops = mm_ops; pgt->mmu = NULL; return 0; } @@ -410,7 +417,9 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - put_page(virt_to_page(kvm_pte_follow(*ptep))); + struct kvm_pgtable_mm_ops *mm_ops = arg; + + mm_ops->put_page((void *)kvm_pte_follow(*ptep, mm_ops)); return 0; } @@ -419,10 +428,11 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) struct kvm_pgtable_walker walker = { .cb = hyp_free_walker, .flags = KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - put_page(virt_to_page(pgt->pgd)); + pgt->mm_ops->put_page(pgt->pgd); pgt->pgd = NULL; } @@ -434,6 +444,8 @@ struct stage2_map_data { struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; + + struct kvm_pgtable_mm_ops *mm_ops; }; static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -501,12 +513,12 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp, pte = *ptep; - struct page *page = virt_to_page(ptep); if (data->anchor) { if (kvm_pte_valid(pte)) - put_page(page); + mm_ops->put_page(ptep); return 0; } @@ -520,7 +532,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (!data->memcache) return -ENOMEM; - childp = kvm_mmu_memory_cache_alloc(data->memcache); + childp = mm_ops->zalloc_page(data->memcache); if (!childp) return -ENOMEM; @@ -532,13 +544,13 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (kvm_pte_valid(pte)) { kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); - put_page(page); + mm_ops->put_page(ptep); } - kvm_set_table_pte(ptep, childp); + kvm_set_table_pte(ptep, childp, mm_ops); out_get_page: - get_page(page); + mm_ops->get_page(ptep); return 0; } @@ -546,13 +558,14 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; int ret = 0; if (!data->anchor) return 0; - put_page(virt_to_page(kvm_pte_follow(*ptep))); - put_page(virt_to_page(ptep)); + mm_ops->put_page(kvm_pte_follow(*ptep, mm_ops)); + mm_ops->put_page(ptep); if (data->anchor == ptep) { data->anchor = NULL; @@ -607,6 +620,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .phys = ALIGN_DOWN(phys, PAGE_SIZE), .mmu = pgt->mmu, .memcache = mc, + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -643,7 +657,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - struct kvm_s2_mmu *mmu = arg; + struct kvm_pgtable *pgt = arg; + struct kvm_s2_mmu *mmu = pgt->mmu; + struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; kvm_pte_t pte = *ptep, *childp = NULL; bool need_flush = false; @@ -651,9 +667,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, mm_ops); - if (page_count(virt_to_page(childp)) != 1) + if (mm_ops->page_count(childp) != 1) return 0; } else if (stage2_pte_cacheable(pte)) { need_flush = true; @@ -666,15 +682,15 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, */ kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, addr, level); - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (need_flush) { - stage2_flush_dcache(kvm_pte_follow(pte), + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); } if (childp) - put_page(virt_to_page(childp)); + mm_ops->put_page(childp); return 0; } @@ -683,7 +699,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, - .arg = pgt->mmu, + .arg = pgt, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; @@ -815,12 +831,13 @@ static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pte)) return 0; - stage2_flush_dcache(kvm_pte_follow(pte), kvm_granule_size(level)); + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); return 0; } @@ -829,6 +846,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) struct kvm_pgtable_walker walker = { .cb = stage2_flush_walker, .flags = KVM_PGTABLE_WALK_LEAF, + .arg = pgt->mm_ops, }; if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) @@ -837,7 +855,8 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; u64 vtcr = kvm->arch.vtcr; @@ -846,12 +865,13 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; pgd_sz = kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE; - pgt->pgd = alloc_pages_exact(pgd_sz, GFP_KERNEL_ACCOUNT | __GFP_ZERO); + pgt->pgd = mm_ops->zalloc_pages_exact(pgd_sz); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = ia_bits; pgt->start_level = start_level; + pgt->mm_ops = mm_ops; pgt->mmu = &kvm->arch.mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ @@ -863,15 +883,16 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte)) return 0; - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (kvm_pte_table(pte, level)) - put_page(virt_to_page(kvm_pte_follow(pte))); + mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); return 0; } @@ -883,10 +904,11 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) .cb = stage2_free_walker, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; - free_pages_exact(pgt->pgd, pgd_sz); + pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 1f41173e6149..278e163beda4 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -88,6 +88,48 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_valid(pfn); } +static void *stage2_memcache_alloc_page(void *arg) +{ + struct kvm_mmu_memory_cache *mc = arg; + kvm_pte_t *ptep = NULL; + + /* Allocated with GFP_KERNEL_ACCOUNT, so no need to zero */ + if (mc && mc->nobjs) + ptep = mc->objects[--mc->nobjs]; + + return ptep; +} + +static void *kvm_host_zalloc_pages_exact(size_t size) +{ + return alloc_pages_exact(size, GFP_KERNEL_ACCOUNT | __GFP_ZERO); +} + +static void kvm_host_get_page(void *addr) +{ + get_page(virt_to_page(addr)); +} + +static void kvm_host_put_page(void *addr) +{ + put_page(virt_to_page(addr)); +} + +static int kvm_host_page_count(void *addr) +{ + return page_count(virt_to_page(addr)); +} + +static phys_addr_t kvm_host_pa(void *addr) +{ + return __pa(addr); +} + +static void *kvm_host_va(phys_addr_t phys) +{ + return __va(phys); +} + /* * Unmapping vs dcache management: * @@ -351,6 +393,17 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, return 0; } +static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { + .zalloc_page = stage2_memcache_alloc_page, + .zalloc_pages_exact = kvm_host_zalloc_pages_exact, + .free_pages_exact = free_pages_exact, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .page_count = kvm_host_page_count, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + /** * kvm_init_stage2_mmu - Initialise a S2 MMU strucrure * @kvm: The pointer to the KVM structure @@ -374,7 +427,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm); + err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; @@ -1198,6 +1251,19 @@ static int kvm_map_idmap_text(void) return err; } +static void *kvm_hyp_zalloc_page(void *arg) +{ + return (void *)get_zeroed_page(GFP_KERNEL); +} + +static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { + .zalloc_page = kvm_hyp_zalloc_page, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + int kvm_mmu_init(void) { int err; @@ -1241,7 +1307,7 @@ int kvm_mmu_init(void) goto out; } - err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits); + err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits, &kvm_hyp_mm_ops); if (err) goto out_free_pgtable; From patchwork Tue Nov 17 18:15:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0C2BC5519F for ; Tue, 17 Nov 2020 18:18:38 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72CC1221FD for ; Tue, 17 Nov 2020 18:18:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="KJRGoqgY"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="H5BJeQDN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72CC1221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bVnaZk6QgJn7QPUzLVJ1nWAVOzKEi2nXXb9WFwWI4XQ=; b=KJRGoqgYw4NZfy3AS5Tdrm/PD OqSNbGMloxtm/fAPfMyj+m2hEWCJVMLepGKURQUgNb36wmBzkj6aKlL0NxbzybVwhls0Skv6rxKjx IybThp2xIIqIjtIoA3g7sNF7XrzQms/LSv+h0QTgSGxZQsK/jJSQGQMLn7vSOGNSRAYZa1/uMYypm P2ypm5ezy4A/JCsigPGpEZ5PpsbuTZeumT3egOeg6rCHQG1goMsMSrA8TRf80HK+Z+VFaBu2x4cEl swiM/zt8MFJPxxk2mDnTE0yZknGknDVcBtzFZcsahDQFDE8P8b2yqi7y3FWkUW1eipkR1HuX4iuGT ke3iqpSRg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5YJ-0002os-9E; Tue, 17 Nov 2020 18:18:03 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Ww-0002FH-LU for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:39 +0000 Received: by mail-wr1-x44a.google.com with SMTP id y2so13443455wrl.3 for ; Tue, 17 Nov 2020 10:16:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ku0UFgVufHs9v1IJuu4Ni7X8qoCAXYV3NVQ1AKXoaAA=; b=H5BJeQDNLb++2xZvg4tH88cc0a517Nx1EBbTKeYIiv7k2xmQWwSp58eVZTiebUdyMG +g9ZeTElu3osA0nWOAV/aPjF1sobwPu1mjr/Dc4ZR36cFV5rLKkzslvkaXdW4CLYUChE nsu09uy4ADkg28f4TLlCpZGieLbT7PYoNhrfsiVb3HuoZcGEbh4XzBqyFM2bwNG3qIfY hH/rYe/JYRFiC/HTk4i7mRnSpAofJBPD+hamGi87aZT+MwV5MFfE4MgTO4nNxijcnPew C0S1hc2bUUhl2BBBC+fbW8SxHu46dT+sn/YDa0tekeckiqLhsPvIgfkzwciINYE0N9jK cxtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ku0UFgVufHs9v1IJuu4Ni7X8qoCAXYV3NVQ1AKXoaAA=; b=o3c4DZy0ulwiRNcyaMmsq9h/5oPVZCftGmWZ5hl/MXDdlxZ+1/Z3uNbCwWuey/x74V 9G80LvvDmC/SOI4ZSu9LcjBbh//rFcYQK1dfZr404SkxUVfwc3oW6hsB9B7gxNErIm53 BUYVv5Ryf2TVT2myagvIGoXb0pwEBqYBvR6JnjrFmhRbLPYDl2hNUyM82VOlPPkvq66P qEEML/CSaV6rdjBoUNJ5ol/h3m+ej1N74iQVQsqqhpudBEdjrtGpHdowFgs4VYDxyx9j hqT1jTIwMVPUDxVkijljjV+VtCAOBQL+VHjam7m+rtEyl99cMaKwBpALKjiPJcYYghcG O5jw== X-Gm-Message-State: AOAM5310g4P3OtU3FDZ4jHY5BluzP/ZG0lD7gZ/q0RK+0EfwjMxgIUR2 4ubcj83QQ4hy64huJGuBZrKIRgi+PO4a X-Google-Smtp-Source: ABdhPJxGgMTS1HY9eY7+MyhEXSMxE4MyyZGir1Tn8gNiQHb8pv7qDl59xEr1Ob0B7Wzc6Bbww1Gn9i2dBo1H X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:398a:: with SMTP id g132mr368672wma.51.1605636995303; Tue, 17 Nov 2020 10:16:35 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:47 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-8-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 07/27] KVM: arm64: Introduce a BSS section for use at Hyp From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131638_759800_E87B80CD X-CRM114-Status: GOOD ( 16.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the hyp code cannot make full use of a bss, as the kernel section is mapped read-only. While this mapping could simply be changed to read-write, it would intermingle even more the hyp and kernel state than they currently are. Instead, introduce a __hyp_bss section, that uses reserved pages, and create the appropriate RW hyp mappings during KVM init. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 7 +++++++ arch/arm64/kvm/arm.c | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 1 + 4 files changed, 20 insertions(+) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 8ff579361731..f58cf493de16 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -12,6 +12,7 @@ extern char __hibernate_exit_text_start[], __hibernate_exit_text_end[]; extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[]; extern char __hyp_text_start[], __hyp_text_end[]; extern char __hyp_data_ro_after_init_start[], __hyp_data_ro_after_init_end[]; +extern char __hyp_bss_start[], __hyp_bss_end[]; extern char __idmap_text_start[], __idmap_text_end[]; extern char __initdata_begin[], __initdata_end[]; extern char __inittext_begin[], __inittext_end[]; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 4382b5d0645d..ded78a25365d 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -8,6 +8,13 @@ #define RO_EXCEPTION_TABLE_ALIGN 8 #define RUNTIME_DISCARD_EXIT +#define BSS_FIRST_SECTIONS \ + . = ALIGN(PAGE_SIZE); \ + __hyp_bss_start = .; \ + *(.hyp.bss) \ + . = ALIGN(PAGE_SIZE); \ + __hyp_bss_end = .; + #include #include #include diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7335eb4fb0bd..882eb383bd75 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1709,7 +1709,18 @@ static int init_hyp_mode(void) goto out_err; } + /* + * .hyp.bss is placed at the beginning of the .bss section, so map that + * part RW, and the rest RO as the hyp shouldn't be touching it. + */ err = create_hyp_mappings(kvm_ksym_ref(__bss_start), + kvm_ksym_ref(__hyp_bss_end), PAGE_HYP); + if (err) { + kvm_err("Cannot map hyp bss section: %d\n", err); + goto out_err; + } + + err = create_hyp_mappings(kvm_ksym_ref(__hyp_bss_end), kvm_ksym_ref(__bss_stop), PAGE_HYP_RO); if (err) { kvm_err("Cannot map bss section\n"); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index 5d76ff2ba63e..dc281d90063e 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -17,4 +17,5 @@ SECTIONS { PERCPU_INPUT(L1_CACHE_BYTES) } HYP_SECTION(.data..ro_after_init) + HYP_SECTION(.bss) } From patchwork Tue Nov 17 18:15:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE95AC63798 for ; Tue, 17 Nov 2020 18:20:12 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 40F85221FD for ; Tue, 17 Nov 2020 18:20:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="gcY3ehzg"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="nNRoD2M9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40F85221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=w9LUFe7mcPcrtT30uonPdUto3s6zbqUPGTdQM+X/OlM=; b=gcY3ehzgLY1WUG0WYEGzhm6+J QgyID1j2WI08O03iLBVyfelPtgJWoeCKtFhfPAEMWKVIwKH4PMBuEBnPMa5mivpOhcz2PgpEU951z ESrR+KaUnUOl82WplDl+xpnAr1wSwOyA/DmBF5YYJyD/bvh/asuZvf7dqM9JwrlDWyOQ9DU5lRrFG lqKQ2Z0ctecCIRi5cYxwLtouZ1s0/9xA/F/XQ++Sv6TyS9XR6h7LGA6aSzRzbRN52OgqJQXwa4aJn fLyLvAiNwS1aQCo8tUduhWUPWqHD0aL8EvAMz0Yz1MDwq3yQo6EpL6VZGxipvYtg9q91R3Oh5iLXL X9OkhDgww==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Yg-0002wU-3O; Tue, 17 Nov 2020 18:18:26 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Wz-0002Fu-LO for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:42 +0000 Received: by mail-qk1-x74a.google.com with SMTP id 198so13837442qkj.7 for ; Tue, 17 Nov 2020 10:16:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=lh2wfGLmmAYfouTWYrt95KjC1/ZF6WAi7ZBhV5AjDD0=; b=nNRoD2M9XXWoWM8NgpAlnEhXBoYa9xn/smGakcLCZJxBll99tRVH7XlZ6kpAfLmDzS YEBOp/igtTiXRZTz+j56GXi5pzEcTsxtC69P2Bv0KA1AZUyCDWPY853b0xcR0BXqNsjY zzoThHvrbMLTNPm/8mNuW8sATGGuqFD6NUWfoAWRd9Krb9CmjdTe3yBuDe1UP2ALLPRo K9J+hEQ46hZWlOQ5xsj57z6jJFoBPfuEglscL+ZMWfWKy3HK4Aci7rdN/cI80VZX34fB Q8O1Od1+22A+zc26YZLJls0NZORus1Ag2kLC1/y6w0rVECW+qKDUHu2XDuWGww78dUnJ 85Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lh2wfGLmmAYfouTWYrt95KjC1/ZF6WAi7ZBhV5AjDD0=; b=Pz8LzyGvyJBdS+TMCGeR83Mt0hwj9d4ypYOvaDUxCm0Zdj43N2EX+L/aPWV5DdYuX1 D/NG6khHQZrEWyAoDK5x5bDs3g5e3bm4sFQocqFzpzoWxMeMhUCI6kJoQbEP/rlP0TP4 NkEKIqKWGZNTYNxKrrnucsyDEz9eXTNK7vdHDVznUc/P33MQe1subKl+5fGbN8JAUdgA 0D9Fjv7I6UbWUanC7BqEW5MeJfgknhe5QmGWgXJzv2ZMFPFwZQbpoDQQLI5gli9lIlOm PS3+F4L9UlfONsRkzIConPfrVKijNUY9C76iJtXsF+97o6X2sv6LnE+tVEshsW0BnLym IZKQ== X-Gm-Message-State: AOAM532PO3Z1ejtt6/gdVapV004ERwDRJsgiHrY9YNS3zcKJUiA7lzhE DnESfzsmUL/fLBfhvOpzkkXx7yR3LaKM X-Google-Smtp-Source: ABdhPJw9IfFuggDXuw8zeDBboBwHrDiqLV2n/32la8Pmpb9DK5T8ZLl8MJjnvcQGvWKpPM+6LkFNsSmTdWC9 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:80e1:: with SMTP id 88mr995546qvb.10.1605636997650; Tue, 17 Nov 2020 10:16:37 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:48 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-9-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 08/27] KVM: arm64: Make kvm_call_hyp() a function call at Hyp From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131641_767330_3A1E00FB X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kvm_call_hyp() has some logic to issue a function call or a hypercall depending the EL at which the kernel is running. However, all the code compiled under __KVM_NVHE_HYPERVISOR__ is guaranteed to run only at EL2, and in this case a simple function call is needed. Add ifdefery to kvm_host.h to symplify kvm_call_hyp() in .hyp.text. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ac11adab6602..7a5d5f4b3351 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -557,6 +557,7 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva); void kvm_arm_halt_guest(struct kvm *kvm); void kvm_arm_resume_guest(struct kvm *kvm); +#ifndef __KVM_NVHE_HYPERVISOR__ #define kvm_call_hyp_nvhe(f, ...) \ ({ \ struct arm_smccc_res res; \ @@ -596,6 +597,11 @@ void kvm_arm_resume_guest(struct kvm *kvm); \ ret; \ }) +#else /* __KVM_NVHE_HYPERVISOR__ */ +#define kvm_call_hyp(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_ret(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) +#endif /* __KVM_NVHE_HYPERVISOR__ */ void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); From patchwork Tue Nov 17 18:15:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD08BC5519F for ; Tue, 17 Nov 2020 18:19:37 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6BB8A221FD for ; Tue, 17 Nov 2020 18:19:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="te49Ky7I"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="E0fCuokE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6BB8A221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=frPdwMUWFyjNOOqQ2ExoYajbjslU/YtT1Yr/lav81uc=; b=te49Ky7IffYneSiV+x4SVINPn GeWZemBgt7ZGTYNxFZn9BE4FORN24EKsiz4BJ+ljAkA/6IeEYWdyV0KjYENNzLMpnyG/qGesEmJUd 9HJJnwXKcdfg+f+qvGPb5sCSdqTjtqvlvjS2qImaMzJN+VoXX0F8vP8urHgwloTWp6+KWhTbKRoAz HuSAjnQ4qcdh6wPZ32tFl7nReX/dQr4mPPmY9DzUlLCEN0HLWOCyDtBTRjrGly/G3rPjy93UTmZbJ 2FyZJ4pvl/1TX7xL4DsxmoGyIfH4C53IohSn7hZAJIpf2fJpR0qEcUzTx4QY+VVz+PCfZ0b+OH4iG Yg0ko6bZw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Z7-00039l-U4; Tue, 17 Nov 2020 18:18:54 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5X1-0002Gf-C0 for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:44 +0000 Received: by mail-qv1-xf49.google.com with SMTP id u19so13699583qvx.4 for ; Tue, 17 Nov 2020 10:16:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=TDLolWJ6DLDdLZxFLwTPZoAbRqc8wO9K1ZVQwJmhgQI=; b=E0fCuokET/iSPfCmN0hv1bpwVzG19gOi60bqbVg4tbU/MXV8JkAYrWoJTWsyKtcZmb NCzh+HSrGM/VrpPjSSavb+jAMioMF4i51wNcX3ISkV63+fEwxFrZc6vXKw/yNZVQcwCy 05Hvg2hX6rvUiRO1FEufam8DDHknNkWZPYw2fzBkTd8AuB8YloMYzzbBHwuePeZao66I UYUGCo9WEpMJep8NfwFKG2NCoQu+vd/Oqeqg/bAdwkpHhbpnuHl+hVrGxKJS3KlbgYWl e+Gk+eHeUbd/8E9fdjlIquCJgsoKzgzbJUq0eQYXqTuK5nk5LEhfuIzVSQAxYT+jwcTN pZ0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TDLolWJ6DLDdLZxFLwTPZoAbRqc8wO9K1ZVQwJmhgQI=; b=KkYiGJy1ZB3rPmTd3VT0Gs0AFZspdOGXUgYuNeuh1sM7h1RfMDaN/PWXvEFqbvYoj2 +z2BsyLPxsMWqTVWs9HCniqz4HdTCFV6IpVD7/pth90UtcZsSozQBlSTF0yq5HTuH6jf M76SycWOg78K/NATYf7LoLR8Y119Y/U8KdS3pKLAgDMaWBH3gFb3h7gmc8dA3xsSSj6F EFZMjvf4AxkDEoNhWf6gKlk6Kb8Y1JiAtz9imJhqIXRFQZcIs1PsNjWb688P9qYW1NL0 vmMSI72Vkgjsxu7aJPc0ti566IW68DWw1O2oF00HV5q3basWJV0kN4RGW2KSe52Y6SCE fLKg== X-Gm-Message-State: AOAM5316L7EcGH//g8tRVxoyYcwhOSGl+o33gi1fqFGvEVGB1pXy+84y AY49y/u4HGp3bHEcq2JdeI9Lnqt+97eQ X-Google-Smtp-Source: ABdhPJzbODkjl7s4UxQH0AkdtMhlkcj6CRSeDqE2+AHaFV3kAPIwJibRapqdnHmTcqcB2PiXnAFDr9sXfpa9 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:df08:: with SMTP id g8mr854334qvl.17.1605636999772; Tue, 17 Nov 2020 10:16:39 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:49 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-10-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 09/27] KVM: arm64: Allow using kvm_nvhe_sym() in hyp code From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131643_468951_10AFD1EC X-CRM114-Status: GOOD ( 14.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow the usage of code shared by the host and the hyp in static inline library function, allow the usage of kvm_nvhe_sym() at el2 by defaulting to the raw symbol name. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/hyp_image.h | 4 ++++ arch/arm64/include/asm/kvm_asm.h | 4 ++-- arch/arm64/kvm/arm.c | 2 +- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h index daa1a1da539e..8b807b646b8f 100644 --- a/arch/arm64/include/asm/hyp_image.h +++ b/arch/arm64/include/asm/hyp_image.h @@ -7,11 +7,15 @@ #ifndef __ARM64_HYP_IMAGE_H__ #define __ARM64_HYP_IMAGE_H__ +#ifndef __KVM_NVHE_HYPERVISOR__ /* * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, * to separate it from the kernel proper. */ #define kvm_nvhe_sym(sym) __kvm_nvhe_##sym +#else +#define kvm_nvhe_sym(sym) sym +#endif #ifdef LINKER_SCRIPT diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 1a86581e581e..e4934f5e4234 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -173,11 +173,11 @@ struct kvm_s2_mmu; DECLARE_KVM_NVHE_SYM(__kvm_hyp_init); DECLARE_KVM_NVHE_SYM(__kvm_hyp_host_vector); DECLARE_KVM_HYP_SYM(__kvm_hyp_vector); -DECLARE_KVM_NVHE_SYM(__kvm_hyp_psci_cpu_entry); #define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init) #define __kvm_hyp_host_vector CHOOSE_NVHE_SYM(__kvm_hyp_host_vector) #define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector) -#define __kvm_hyp_psci_cpu_entry CHOOSE_NVHE_SYM(__kvm_hyp_psci_cpu_entry) + +void kvm_nvhe_sym(__kvm_hyp_psci_cpu_entry)(void); extern unsigned long kvm_arm_hyp_percpu_base[NR_CPUS]; DECLARE_KVM_NVHE_SYM(__per_cpu_start); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 882eb383bd75..391cf6753a13 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1369,7 +1369,7 @@ static void cpu_prepare_hyp_mode(int cpu) params->vector_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref(__kvm_hyp_host_vector)); params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); - params->entry_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref(__kvm_hyp_psci_cpu_entry)); + params->entry_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref_nvhe(__kvm_hyp_psci_cpu_entry)); params->pgd_pa = kvm_mmu_get_httbr(); /* From patchwork Tue Nov 17 18:15:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDB65C63798 for ; Tue, 17 Nov 2020 18:20:23 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 49262221FD for ; Tue, 17 Nov 2020 18:20:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TVYRo29D"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="gOHYeik0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49262221FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XxPMVLzxArfHMO+DvhCNtGoYUMr0iTRn+M/fv7vmaUI=; b=TVYRo29DmtsnBxWuR+0BCyd1C 4GgW2b8aypxfeWchl9CIwpgickayJQjlZWP2BRl2r5JnyOA7q75et4X+Jx3XB0czyS5IG4Nm3Bgkf L3UD8MzjzMlM9FC7IUbP6p11/HS2xj5sFKS+Z0eQlDlKiGWghsdVo92qztGKv/5LSY7feLXg3w8wY h+BT7aoXt/xYkBxGsl0qrIxMh6nYTTYPF7y70kHth8HPpmzzYjt0KPmR+DomKWklQ2l0L491QI/RN ScXHCBAVFSb7ckpLn8WNf6MMQzQ/gwJ5Sqy1irgVqfZaZsFUmD4b4pRgHHSjqHAxvsbF3o9CJ3Gh5 BnWvnU/DQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Zu-0003SW-3W; Tue, 17 Nov 2020 18:19:42 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5X4-0002Hl-RL for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:48 +0000 Received: by mail-wm1-x34a.google.com with SMTP id g3so1891213wmh.9 for ; Tue, 17 Nov 2020 10:16:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Nx9TzG8OmvjJxKNgiM/4waPsacypWhGMNqjRf5oXlbQ=; b=gOHYeik0Afu79DkLLq77U6M3Z0aWksdQKzAaxc0pEuoi6vq+mZ+7aoUPAYEia6bkBQ 9IrUSm0MEJOXZ7x6ny2axoWaZn00V/b44gHHx63cD9WwcGzTfW5WifeWGoTFivw+WX/P KkfodH14f9sfhSpiwxbImnoa9TFjZnqQOyEpgrf0mwsbW11qOp5BtIrM2nYLu5g9/Ln+ Rz3PmqagXh38rv9cjiqPGgYn2U/plPy31Md0cZbeZKwT+2qZUIeZhE5sc+u2dCpMfpvF gEqfzsNnU4cJx98r2oU7plVAULgQrF0izi2qcXfZ/80LI1tJ7Fsh7hga4tXYeN1pdo3c CU7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Nx9TzG8OmvjJxKNgiM/4waPsacypWhGMNqjRf5oXlbQ=; b=Aedn5J9GikYNF15sPBjRhU3EPvoAPXfWHK6fQDMC+ml8dt2BXzKB2x9jm2fxQPzACQ knpaKGHRg6Nf5i0X8RNn3k32ycA8OnCFjeh39XiQIrvXdZif/rRHYzYi2N0ng4M9iBQg jzTn1o2JDGYU23Joies2XlcHxcgUfE5mGjoU44aqzxOOuYDDiEfSfzRAC5Znw3Ez8pWj QO5dXyhcSKynOnHtWNDPqvuZvnwiP8TRr2OswjOlMoXk/uAGUtwAYnUtPSSm33TYzhW5 0wDTjo/KnHNVhHrJoxSsPNz0yMiLloy7Ztv0PurFZQGpRuQKmPpGnmorU+20+/L/zDmX Bt5A== X-Gm-Message-State: AOAM530+2i3W8Xjgyk2OuFPdj8sbKPKhxe42hpglQ2KKmKIDR+9f6l1v 635Gfeyr7Xj9XioO5qyF1px2WeoEKE/j X-Google-Smtp-Source: ABdhPJyhHRlIsUkfi/eYpmMORHp5D3o0p7xzGK05hQPhQ5wLmsbUlLSI6op+6zLdgW/DN66bUlH0P+0JVW6S X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:adf:ebc5:: with SMTP id v5mr876894wrn.392.1605637003434; Tue, 17 Nov 2020 10:16:43 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:50 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-11-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 10/27] KVM: arm64: Introduce an early Hyp page allocator From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131646_994422_84E5A5C3 X-CRM114-Status: GOOD ( 20.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With nVHE, the host currently creates all s1 hypervisor mappings at EL1 during boot, installs them at EL2, and extends them as required (e.g. when creating a new VM). But in a world where the host is no longer trusted, it cannot have full control over the code mapped in the hypervisor. In preparation for enabling the hypervisor to create its own s1 mappings during boot, introduce an early page allocator, with minimal functionality. This allocator is designed to be used only during early bootstrap of the hyp code when memory protection is enabled, which will then switch to using a full-fledged page allocator after init. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/early_alloc.h | 14 +++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 24 ++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/early_alloc.c | 60 +++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/psci-relay.c | 5 +- 5 files changed, 101 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/early_alloc.h create mode 100644 arch/arm64/kvm/hyp/include/nvhe/memory.h create mode 100644 arch/arm64/kvm/hyp/nvhe/early_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h new file mode 100644 index 000000000000..68ce2bf9a718 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_EARLY_ALLOC_H +#define __KVM_HYP_EARLY_ALLOC_H + +#include + +void hyp_early_alloc_init(void *virt, unsigned long size); +unsigned long hyp_early_alloc_nr_pages(void); +void *hyp_early_alloc_page(void *arg); +void *hyp_early_alloc_contig(unsigned int nr_pages); + +extern struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; + +#endif /* __KVM_HYP_EARLY_ALLOC_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h new file mode 100644 index 000000000000..64c44c142c95 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MEMORY_H +#define __KVM_HYP_MEMORY_H + +#include + +#include + +extern s64 hyp_physvirt_offset; + +#define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) +#define __hyp_va(virt) ((void *)((phys_addr_t)(virt) - hyp_physvirt_offset)) + +static inline void *hyp_phys_to_virt(phys_addr_t phys) +{ + return __hyp_va(phys); +} + +static inline phys_addr_t hyp_virt_to_phys(void *addr) +{ + return __hyp_pa(addr); +} + +#endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 590fdefb42dd..1fc0684a7678 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -10,7 +10,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/early_alloc.c b/arch/arm64/kvm/hyp/nvhe/early_alloc.c new file mode 100644 index 000000000000..de4c45662970 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/early_alloc.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include + +#include + +struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; +s64 __ro_after_init hyp_physvirt_offset; + +static unsigned long base; +static unsigned long end; +static unsigned long cur; + +unsigned long hyp_early_alloc_nr_pages(void) +{ + return (cur - base) >> PAGE_SHIFT; +} + +extern void clear_page(void *to); + +void *hyp_early_alloc_contig(unsigned int nr_pages) +{ + unsigned long ret = cur, i, p; + + if (!nr_pages) + return NULL; + + cur += nr_pages << PAGE_SHIFT; + if (cur > end) { + cur = ret; + return NULL; + } + + for (i = 0; i < nr_pages; i++) { + p = ret + (i << PAGE_SHIFT); + clear_page((void *)(p)); + } + + return (void *)ret; +} + +void *hyp_early_alloc_page(void *arg) +{ + return hyp_early_alloc_contig(1); +} + +void hyp_early_alloc_init(unsigned long virt, unsigned long size) +{ + base = virt; + end = virt + size; + cur = virt; + + hyp_early_alloc_mm_ops.zalloc_page = hyp_early_alloc_page; + hyp_early_alloc_mm_ops.phys_to_virt = hyp_phys_to_virt; + hyp_early_alloc_mm_ops.virt_to_phys = hyp_virt_to_phys; +} diff --git a/arch/arm64/kvm/hyp/nvhe/psci-relay.c b/arch/arm64/kvm/hyp/nvhe/psci-relay.c index 313ef42f0eab..dbe57ae84a0c 100644 --- a/arch/arm64/kvm/hyp/nvhe/psci-relay.c +++ b/arch/arm64/kvm/hyp/nvhe/psci-relay.c @@ -14,6 +14,8 @@ #include #include +#include + #define INVALID_CPU_ID UINT_MAX extern char __kvm_hyp_cpu_entry[]; @@ -21,9 +23,6 @@ extern char __kvm_hyp_cpu_entry[]; /* Config options set by the host. */ u32 __ro_after_init kvm_host_psci_version = PSCI_VERSION(0, 0); u32 __ro_after_init kvm_host_psci_function_id[PSCI_FN_MAX]; -s64 __ro_after_init hyp_physvirt_offset; - -#define __hyp_pa(x) ((phys_addr_t)((x)) + hyp_physvirt_offset) struct kvm_host_psci_state { atomic_t pending_on; From patchwork Tue Nov 17 18:15:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D290C2D0E4 for ; Tue, 17 Nov 2020 18:21:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D929320888 for ; Tue, 17 Nov 2020 18:21:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="pWsiMk0V"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="I3vfgCro" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D929320888 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QRVex9X75EYgqR9LHnc2Fcx20cin8lYGzf9qUaZX6wE=; b=pWsiMk0VIfPr88KHsg398GUYC UogOXol8TmhjVfZTaFJFLq6SZQGsuVfEOYIbTU9hWs/7HoL/RfZH4D5CdH+zLVRalxSu2l4UYN+un RcWg3zzgkQSlKJ6MUPthj4Xf7pAyn3yaGqNVSw15rELF4BMvtoSoFn9q9RAl7/AHGr63zICL6+euX e4t+QPi9E9j2RssC07GZDwb4jWVGnU+YSF6eVTQ0oc3fu+8tbOdGUIN2dIo1K/TOxHcGh4WjPmd7m tSkZX8KgVbpVemQpZE9gnAe/punuAwb7S0LNESyPhI6eBOXSC+xhg7ld6kjlRNG+nYDmDwMmS8bWz mUCjdDL2w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5aQ-0003kA-RO; Tue, 17 Nov 2020 18:20:14 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5X6-0002IG-FW for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:49 +0000 Received: by mail-qk1-x74a.google.com with SMTP id 202so14708049qkl.9 for ; Tue, 17 Nov 2020 10:16:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=43GxmItVf3ENfAwK24h1jbs6QJ7Tr5/fb9a8KdpnALQ=; b=I3vfgCroLZcbxbMBelA50c1N1Y3hD7RO8dK4Yfdg92JJ9mDudKl7ZgVFDEGcLokKqJ fep0U6jqOGtZ9zoGEC9cF+peh0IJyKmC2E8YytUJtdBUfDiS2VrM7RvmCyrkIEOyoCVj Zj5RKXdkNzPbGhMxtY7uW4LVsd/+tcfslNA/BdBW/Ry87jVtrKtdKPqwdrxwWN35EhWb bdEhtmYnellgGGg7lrrS4XNju/dLo6f/36HiRTbFiTj1kTY/q9HvfamGy/O7EQSsS/SA thHp38VOizlOMz3Ke6XtLCtKEpujSjul4xmuEtP/hRSxLWC+cRCGJH7WvDUtgQ736y89 XCuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=43GxmItVf3ENfAwK24h1jbs6QJ7Tr5/fb9a8KdpnALQ=; b=oSc8Yf8UlQyn7ET1czDaNNjpoY570Y8rrxATOXxFwLLvBmwT9Z1mn2mDNtAlgcW4ij xtR8spiiNYT2MRyt/uTEJrhA8bjFYzytrJyPzUewBSf3g4Ks54f2jk3aW/A2i4isX1j+ M5dmCmqCdBktgZd4Yh8lvdnVRIpihZQoyLBzUQm61ilnTA7EIwJNinEVERo5p+bA3cop 0hq5GFQ/jaoiy/wj6UGaT6NzzUhbzDUHNug/a+4QJHXsgyZccND1zoEwMLx/IvOieyMQ 7Lr70QY7Dc8Rw0wm2Y1IhaGuQD9EOwSesaY2jicYxyCjl3Y8sCgd563ShU3bx5ajk8w/ pe8w== X-Gm-Message-State: AOAM532Yj4Vgh9AGjUiCLYp3fuB/F8p7QuA50XL7OVkRb5oyF9KKE1qB 8x1PZsecuGb2O9mHNZ7QxXLM2H/PCuwd X-Google-Smtp-Source: ABdhPJystvOylX+iR1GTTm6D07Kb5FsHciKocNWKW8OwH+Vo7WRMYhFofs8v1FHeyNq/UupTe8A1TLvb0Rxw X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a05:6214:12e8:: with SMTP id w8mr990171qvv.16.1605637005406; Tue, 17 Nov 2020 10:16:45 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:51 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-12-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 11/27] KVM: arm64: Stub CONFIG_DEBUG_LIST at Hyp From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131648_596825_E640C148 X-CRM114-Status: GOOD ( 16.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to use the kernel list library at EL2, introduce stubs for the CONFIG_DEBUG_LIST out-of-lines calls. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/stub.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/stub.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 1fc0684a7678..33bd381d8f73 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -10,7 +10,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/stub.c b/arch/arm64/kvm/hyp/nvhe/stub.c new file mode 100644 index 000000000000..c0aa6bbfd79d --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/stub.c @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Stubs for out-of-line function calls caused by re-using kernel + * infrastructure at EL2. + * + * Copyright (C) 2020 - Google LLC + */ + +#include + +#ifdef CONFIG_DEBUG_LIST +bool __list_add_valid(struct list_head *new, struct list_head *prev, + struct list_head *next) +{ + return true; +} + +bool __list_del_entry_valid(struct list_head *entry) +{ + return true; +} +#endif From patchwork Tue Nov 17 18:15:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 091E5C5519F for ; Tue, 17 Nov 2020 18:21:10 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 799EC22210 for ; Tue, 17 Nov 2020 18:21:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="aGpgtbjf"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="PTs/nMj1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 799EC22210 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mEgbBvSj8U22+fQbDraQiMv4R9uq9Oa6sROJOl7AULM=; b=aGpgtbjf7+uM0/yciNC9MQIKu j2OSHwrStC6EVPBv1oqEtYpleq/vLGXAIAUge9SRAtatVd/BHVD5FtC5TovBa+GXnnKJyV3hulMuD Orx+kjRnuIfkvjKBNcKpaeD5VhjHKTAYzEMRqZYIwP4QabvCOU4Oo9zZRKm9JO8DAjtwg4FZh3Ps5 UVA1ADO+6tOCM3elZQtEk97rWmIMqZ7WonWRZqYdJAmAtLRxyKKegyI0jsS+/Inqq0NBYJY1otE8a W7gZqOk1eo6YBF87P4/QHsZMlQ+fGCE5yKyxQ/aD8bO5IJS4qwWoQ5uxYO7FfstGXCt+7FIE8Aky2 P/MI+F9Gw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5am-0003wW-Ki; Tue, 17 Nov 2020 18:20:36 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5X7-0002JF-Tk for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:51 +0000 Received: by mail-wm1-x349.google.com with SMTP id a134so1896101wmd.8 for ; Tue, 17 Nov 2020 10:16:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5c2126h8hJMJInu5rMyBSqfTxQg6bBsWCBGBv0nWH30=; b=PTs/nMj1rSzwe/wSPkVuPnUS9QAbEZDDVdo4bLPL975yp2xazuIce4Qfib3J/RQPH8 kVNn4zpMRSMWkqdCLJHhjKRj+nTWZs8FnDGnpG8uDJJqAmP8b+3veln7Gx+gdwrrgCg1 l02471ZrmHCWn+hjN8ZTGd3HFLOQkT7RCN1y5kCQALGL4KR+XRAtdti4NGBPkbTeVGgA G7nGSa3IdmlPDf6H8ynGlUoQ+srHsnBn8bp2NGD3ZlNR5Q8n0PUiHmKwPWIDs99tfoDt c92cdqpzGYyO7riUBAghXhLOlW1ksgJCbmKhNd/EZjjC+CcSLdEqNkvmkus8BXZlUS5H b/Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5c2126h8hJMJInu5rMyBSqfTxQg6bBsWCBGBv0nWH30=; b=KbWprPrdzOrPJWghnf9JhEGDQDVbpWL1tmRLb4eTL+vsNyuW2/GdvVcLzJqOEDwaaz qQ20lsLEU6e8OzZ4/932yE4Guww6eR9Dm29GHFqvfg3ZcXUaG30Su+4jpzinXcOWQqh3 BU9KuoxowK2qEfI4FtkXJ+DOXBeQ8n+xKS4wlZzQ7BtoqNaAlqAc4yIAPrhvuvpUsAA8 6iudB48kxrwBi/8t0/uy42Idjp/nGICQXMU0WMRN7OQA5w6QJxkgyA9cCXcNbUJDIR+A zrCMgMSIOCUDjG5CkHKo8L3BHkoY7O4Kn6UVubPFr/tbSL67z3GRzxZc8SSmqy9DI3eQ ykyw== X-Gm-Message-State: AOAM531RPHuFDYnal0+A8j78eYM1/n/eXXQNj+hRQdnQ9GkNIt34xA9K sQCpAdtR87JMn18Q+mXqM5TS8D8s4w9N X-Google-Smtp-Source: ABdhPJwe5lEWWT4w8AnFxEqyPY4I/jybna8PARPCVKG/GicoHwrp8jvJFCyxkci6ZE+268MrhdixJueX3V/k X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:1d82:: with SMTP id d124mr417015wmd.12.1605637007451; Tue, 17 Nov 2020 10:16:47 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:52 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-13-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 12/27] KVM: arm64: Introduce a Hyp buddy page allocator From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131650_096806_E1C81D11 X-CRM114-Status: GOOD ( 28.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the hyp code will require a basic form of memory management in order to allocate and free memory pages at EL2. This is needed for various use-cases, including the creation of hyp mappings or the allocation of stage 2 page tables. To address these use-case, introduce a simple memory allocator in the hyp code. The allocator is designed as a conventional 'buddy allocator', working with a page granularity. It allows to allocate and free physically contiguous pages from memory 'pools', with a guaranteed order alignment in the PA space. Each page in a memory pool is associated with a struct hyp_page which holds the page's metadata, including its refcount, as well as its current order, hence mimicking the kernel's buddy system in the GFP infrastructure. The hyp_page metadata are made accessible through a hyp_vmemmap, following the concept of SPARSE_VMEMMAP in the kernel. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 32 ++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 25 +++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 185 +++++++++++++++++++++++ 4 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/gfp.h create mode 100644 arch/arm64/kvm/hyp/nvhe/page_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h new file mode 100644 index 000000000000..95587faee171 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_GFP_H +#define __KVM_HYP_GFP_H + +#include + +#include +#include + +#define HYP_MAX_ORDER 11U +#define HYP_NO_ORDER UINT_MAX + +struct hyp_pool { + hyp_spinlock_t lock; + struct list_head free_area[HYP_MAX_ORDER + 1]; + phys_addr_t range_start; + phys_addr_t range_end; +}; + +/* GFP flags */ +#define HYP_GFP_NONE 0 +#define HYP_GFP_ZERO 1 + +/* Allocation */ +void *hyp_alloc_pages(struct hyp_pool *pool, gfp_t mask, unsigned int order); +void hyp_get_page(void *addr); +void hyp_put_page(void *addr); + +/* Used pages cannot be freed */ +int hyp_pool_init(struct hyp_pool *pool, phys_addr_t phys, + unsigned int nr_pages, unsigned int used_pages); +#endif /* __KVM_HYP_GFP_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h index 64c44c142c95..ed47674bc988 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/memory.h +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -6,7 +6,17 @@ #include +struct hyp_pool; +struct hyp_page { + unsigned int refcount; + unsigned int order; + struct hyp_pool *pool; + struct list_head node; +}; + extern s64 hyp_physvirt_offset; +extern u64 __hyp_vmemmap; +#define hyp_vmemmap ((struct hyp_page *)__hyp_vmemmap) #define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) #define __hyp_va(virt) ((void *)((phys_addr_t)(virt) - hyp_physvirt_offset)) @@ -21,4 +31,19 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr) return __hyp_pa(addr); } +#define hyp_phys_to_pfn(phys) ((phys) >> PAGE_SHIFT) +#define hyp_phys_to_page(phys) (&hyp_vmemmap[hyp_phys_to_pfn(phys)]) +#define hyp_virt_to_page(virt) hyp_phys_to_page(__hyp_pa(virt)) + +#define hyp_page_to_phys(page) ((phys_addr_t)((page) - hyp_vmemmap) << PAGE_SHIFT) +#define hyp_page_to_virt(page) __hyp_va(hyp_page_to_phys(page)) +#define hyp_page_to_pool(page) (((struct hyp_page *)page)->pool) + +static inline int hyp_page_count(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + return p->refcount; +} + #endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 33bd381d8f73..9e5eacfec6ec 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -10,7 +10,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c new file mode 100644 index 000000000000..6de6515f0432 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -0,0 +1,185 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include + +u64 __hyp_vmemmap; + +/* + * Example buddy-tree for a 4-pages physically contiguous pool: + * + * o : Page 3 + * / + * o-o : Page 2 + * / + * / o : Page 1 + * / / + * o---o-o : Page 0 + * Order 2 1 0 + * + * Example of requests on this zon: + * __find_buddy(pool, page 0, order 0) => page 1 + * __find_buddy(pool, page 0, order 1) => page 2 + * __find_buddy(pool, page 1, order 0) => page 0 + * __find_buddy(pool, page 2, order 0) => page 3 + */ +static struct hyp_page *__find_buddy(struct hyp_pool *pool, struct hyp_page *p, + unsigned int order) +{ + phys_addr_t addr = hyp_page_to_phys(p); + + addr ^= (PAGE_SIZE << order); + if (addr < pool->range_start || addr >= pool->range_end) + return NULL; + + return hyp_phys_to_page(addr); +} + +static void __hyp_attach_page(struct hyp_pool *pool, + struct hyp_page *p) +{ + unsigned int order = p->order; + struct hyp_page *buddy; + + p->order = HYP_NO_ORDER; + for (; order < HYP_MAX_ORDER; order++) { + /* Nothing to do if the buddy isn't in a free-list */ + buddy = __find_buddy(pool, p, order); + if (!buddy || list_empty(&buddy->node) || buddy->order != order) + break; + + /* Otherwise, coalesce the buddies and go one level up */ + list_del_init(&buddy->node); + buddy->order = HYP_NO_ORDER; + p = (p < buddy) ? p : buddy; + } + + p->order = order; + list_add_tail(&p->node, &pool->free_area[order]); +} + +void hyp_put_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + if (!p->refcount) + hyp_panic(); + p->refcount--; + if (!p->refcount) + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); +} + +void hyp_get_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + p->refcount++; + hyp_spin_unlock(&pool->lock); +} + +/* Extract a page from the buddy tree, at a specific order */ +static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + struct hyp_page *buddy; + + if (p->order == HYP_NO_ORDER || p->order < order) + return NULL; + + list_del_init(&p->node); + + /* Split the page in two until reaching the requested order */ + while (p->order > order) { + p->order--; + buddy = __find_buddy(pool, p, p->order); + buddy->order = p->order; + list_add_tail(&buddy->node, &pool->free_area[buddy->order]); + } + + p->refcount = 1; + + return p; +} + +static void clear_hyp_page(struct hyp_page *p) +{ + unsigned long i; + + for (i = 0; i < (1 << p->order); i++) + clear_page(hyp_page_to_virt(p) + (i << PAGE_SHIFT)); +} + +static void *__hyp_alloc_pages(struct hyp_pool *pool, gfp_t mask, + unsigned int order) +{ + unsigned int i = order; + struct hyp_page *p; + + /* Look for a high-enough-order page */ + while (i <= HYP_MAX_ORDER && list_empty(&pool->free_area[i])) + i++; + if (i > HYP_MAX_ORDER) + return NULL; + + /* Extract it from the tree at the right order */ + p = list_first_entry(&pool->free_area[i], struct hyp_page, node); + p = __hyp_extract_page(pool, p, order); + + if (mask & HYP_GFP_ZERO) + clear_hyp_page(p); + + return p; +} + +void *hyp_alloc_pages(struct hyp_pool *pool, gfp_t mask, unsigned int order) +{ + struct hyp_page *p; + + hyp_spin_lock(&pool->lock); + p = __hyp_alloc_pages(pool, mask, order); + hyp_spin_unlock(&pool->lock); + + return p ? hyp_page_to_virt(p) : NULL; +} + +/* hyp_vmemmap must be backed beforehand */ +int hyp_pool_init(struct hyp_pool *pool, phys_addr_t phys, + unsigned int nr_pages, unsigned int used_pages) +{ + struct hyp_page *p; + int i; + + if (phys % PAGE_SIZE) + return -EINVAL; + + hyp_spin_lock_init(&pool->lock); + for (i = 0; i <= HYP_MAX_ORDER; i++) + INIT_LIST_HEAD(&pool->free_area[i]); + pool->range_start = phys; + pool->range_end = phys + (nr_pages << PAGE_SHIFT); + + /* Init the vmemmap portion */ + p = hyp_phys_to_page(phys); + memset(p, 0, sizeof(*p) * nr_pages); + for (i = 0; i < nr_pages; i++, p++) { + p->pool = pool; + INIT_LIST_HEAD(&p->node); + } + + /* Attach the unused pages to the buddy tree */ + p = hyp_phys_to_page(phys + (used_pages << PAGE_SHIFT)); + for (i = used_pages; i < nr_pages; i++, p++) + __hyp_attach_page(pool, p); + + return 0; +} From patchwork Tue Nov 17 18:15:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBC0BC2D0E4 for ; Tue, 17 Nov 2020 18:22:53 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5C38820888 for ; Tue, 17 Nov 2020 18:22:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="aN+NMmkT"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Rw7RcFAX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5C38820888 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DA+kCAZfbNaJm4Bvlev5g3Mj4v3h4IR6/bNBWKDwmf4=; b=aN+NMmkTptdncGjU29twDxNyK Sdq4jE3yI9XnkuH/xvnEOHWg/vMaDDb5GIQnFOSZTsH4THASkGTmwZPQZPEDujE0srhiXjL3Qliwm 6CD+LdERp0KfL3VzUqXLGc3cPdheFc/ZoN/W4dyX6TvaJPinzFDdymKSJTXwVESWgfTaVXyBd/LXR jI+wyrH31y3/ysKAY7AAyr45epGN3GPSgjS80Aoitd1qkhcQsc2Pt2w+OD9kEB0f5F0y5h6UstApE iat5klDKJd6yVs6/tSgsqsX6K08CHbtFA03yOIN6VRMguIaazBYT+rptnpxtGAHFgEOSGaz3FeG9A 7mbTLJSFQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5b3-00047D-65; Tue, 17 Nov 2020 18:20:53 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XA-0002Kc-OY for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:55 +0000 Received: by mail-wm1-x34a.google.com with SMTP id j62so2141178wma.4 for ; Tue, 17 Nov 2020 10:16:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=S6cwVwNRfmqiLf8lpRu/biimRGMThpGBDpyPzm8SfcU=; b=Rw7RcFAXWTa9NM5eUWt9ns6PvMjCLwpQUv7zU5flYQ+vJf2W0LNCLix93I5eiPBNhd aSn/XNz2iCuRt7d9fob7z1iSqBYaHL4jSNIE6w7GIgsZO1tNa4nmEohpgfm+zAH0Wlw9 DVg6qEKI+ump3dkMvgBwDMhZEox7plRC1EVURxfkurDRSnciwtlpumtvCwe1TwDJ/3uH PdUcYZ6f6e6G2VWWwtyBc6DSgoIsDauyd0xqvgYdPPX1RYdtzlcNTAi5Rx3h8USdW9Qj +XXV9dpQsp81t1EzBw5lGKS45aGWAI6aKc6lgHu35S7qhBK3hFUqeBSLNq/FTuY5Z6ci JNbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=S6cwVwNRfmqiLf8lpRu/biimRGMThpGBDpyPzm8SfcU=; b=paNHrqZNwQVYRE6YHXloo2eF0cd1CNHxnezPMpYxBzf2JX3XS7Kvd6qXK3kuGLLI6n wCoaZYvk0e+b3p5FI1MFzQNlAX6p8FiDvnVa5Bx89HWg0OebMo1a0NtmSDqoIhT+AAaU MSGmpCKja1GyXfXRl2ihuO2vGtYNXcR1fu5vW6xSz4wVMP8mqaJIxSfwFg7IT2AvC0K+ 7X/K6buTXEbshifkO77le1kz2xkbw1exhc8obGqogVxFm14YuGIt+s+F9JLMWsihPcL5 p7x+yOB8+JdHgpABRap9zEm38fti6AimOaHxthyvyVFwxfcSE3Xnd6y2b/O4DcTSM+oR vF6w== X-Gm-Message-State: AOAM531FvmKoTpfG0GoNW8RT/d/vI6HR/D+eD9HE3Rknx/RNyUlLFP0G 3GN7T9zQikKyU+DAOqg4jijTPwGZfWyM X-Google-Smtp-Source: ABdhPJyBvcTlVnO9kqVOz3fMTDExVUwGTO7ddJFQeswKQ25tqqOb6DNLvzHX+R28Y8s8EBggqoaXPxIosaD/ X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a7b:c195:: with SMTP id y21mr379137wmi.138.1605637009838; Tue, 17 Nov 2020 10:16:49 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:53 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-14-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 13/27] KVM: arm64: Enable access to sanitized CPU features at EL2 From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131652_942185_1CCEA7FA X-CRM114-Status: GOOD ( 25.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce the infrastructure in KVM enabling to copy CPU feature registers into EL2-owned data-structures, to allow reading sanitised values directly at EL2 in nVHE. Given that only a subset of these features are being read by the hypervisor, the ones that need to be copied are to be listed under together with the name of the nVHE variable that will hold the copy. While at it, introduce the first user of this infrastructure by implementing __flush_dcache_area at EL2, which needs arm64_ftr_reg_ctrel0. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/include/asm/kvm_cpufeature.h | 17 ++++++++++++++ arch/arm64/kernel/cpufeature.c | 12 ++++++++++ arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kvm/arm.c | 31 +++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/cache.S | 13 +++++++++++ arch/arm64/kvm/hyp/nvhe/cpufeature.c | 8 +++++++ 8 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_cpufeature.h create mode 100644 arch/arm64/kvm/hyp/nvhe/cache.S create mode 100644 arch/arm64/kvm/hyp/nvhe/cpufeature.c diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index da250e4741bd..3dfbd76fb647 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -600,6 +600,7 @@ void __init setup_cpu_features(void); void check_local_cpu_capabilities(void); u64 read_sanitised_ftr_reg(u32 id); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst); static inline bool cpu_supports_mixed_endian_el0(void) { diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h new file mode 100644 index 000000000000..d34f85cba358 --- /dev/null +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include + +#ifndef KVM_HYP_CPU_FTR_REG +#if defined(__KVM_NVHE_HYPERVISOR__) +#define KVM_HYP_CPU_FTR_REG(id, name) extern struct arm64_ftr_reg name; +#else +#define KVM_HYP_CPU_FTR_REG(id, name) DECLARE_KVM_NVHE_SYM(name); +#endif +#endif + +KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index dd5bc0f0cf0d..3bc86d1423f8 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1116,6 +1116,18 @@ u64 read_sanitised_ftr_reg(u32 id) } EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst) +{ + struct arm64_ftr_reg *regp = get_arm64_ftr_reg(id); + + if (!regp) + return -EINVAL; + + memcpy(dst, regp, sizeof(*regp)); + + return 0; +} + #define read_sysreg_case(r) \ case r: return read_sysreg_s(r) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index dd8ccc9efb6a..c35d768672eb 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -116,6 +116,8 @@ __kvm_nvhe___memcpy = __kvm_nvhe___pi_memcpy; __kvm_nvhe___memset = __kvm_nvhe___pi_memset; #endif +_kvm_nvhe___flush_dcache_area = __kvm_nvhe___pi___flush_dcache_area; + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 391cf6753a13..c7f8fca97202 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -1636,6 +1637,29 @@ static void teardown_hyp_mode(void) } } +#undef KVM_HYP_CPU_FTR_REG +#define KVM_HYP_CPU_FTR_REG(id, name) \ + { .sys_id = id, .dst = (struct arm64_ftr_reg *)&kvm_nvhe_sym(name) }, +static const struct __ftr_reg_copy_entry { + u32 sys_id; + struct arm64_ftr_reg *dst; +} hyp_ftr_regs[] = { + #include +}; + +static int copy_cpu_ftr_regs(void) +{ + int i, ret; + + for (i = 0; i < ARRAY_SIZE(hyp_ftr_regs); i++) { + ret = copy_ftr_reg(hyp_ftr_regs[i].sys_id, hyp_ftr_regs[i].dst); + if (ret) + return ret; + } + + return 0; +} + /** * Inits Hyp-mode on all online CPUs */ @@ -1644,6 +1668,13 @@ static int init_hyp_mode(void) int cpu; int err = 0; + /* + * Copy the required CPU feature register in their EL2 counterpart + */ + err = copy_cpu_ftr_regs(); + if (err) + return err; + /* * Allocate Hyp PGD and setup Hyp identity mapping */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 9e5eacfec6ec..72cfe53f106f 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -10,7 +10,8 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ + cache.o cpufeature.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/cache.S b/arch/arm64/kvm/hyp/nvhe/cache.S new file mode 100644 index 000000000000..36cef6915428 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cache.S @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Code copied from arch/arm64/mm/cache.S. + */ + +#include +#include +#include + +SYM_FUNC_START_PI(__flush_dcache_area) + dcache_by_line_op civac, sy, x0, x1, x2, x3 + ret +SYM_FUNC_END_PI(__flush_dcache_area) diff --git a/arch/arm64/kvm/hyp/nvhe/cpufeature.c b/arch/arm64/kvm/hyp/nvhe/cpufeature.c new file mode 100644 index 000000000000..a887508f996f --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cpufeature.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#define KVM_HYP_CPU_FTR_REG(id, name) struct arm64_ftr_reg name; +#include From patchwork Tue Nov 17 18:15:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4936BC2D0E4 for ; Tue, 17 Nov 2020 18:21:38 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9931720888 for ; Tue, 17 Nov 2020 18:21:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QmubwNnn"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="ZN7TMzWc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9931720888 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SHNP0OEd4j68YAL8PHiAtdC4Rxh8offkZps6XC845LA=; b=QmubwNnnmSyZxuf+NS49yl0Zp a2utp24Bzb8dLwCJiAqzO8PHgYju3uD8YO4dree6+GlzG050qZBC9sh2X8zfcmYxiB3PA79MPU8ft k4VJXjtgzXc3x36vVslx5nu2dFxu+FOH2BsrUt61PiNC5n9KLbj9N8FFJUX3Th8XHxvdeMGTb0+l3 9UtsG5XhNOWrQTGs6SaT86b3Ggr/gWw6LuLLfg48PwMCtlzlKVZqde5Z6tgwq9oZaNnYwXOCPxpie EHr7xcDSH61rNLlHK8kkGMYwjcV/JXIHfKP+4kQ0OHxBsSyLFK8jSTZpW+NIgTG51g2m6myXvOF6Z m7TOv2IEg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5bB-0004BM-Dy; Tue, 17 Nov 2020 18:21:01 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XD-0002L8-KZ for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:57 +0000 Received: by mail-wr1-x449.google.com with SMTP id 67so13499201wra.2 for ; Tue, 17 Nov 2020 10:16:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=F6/Ku+z/rldXTKyTjaRYVBv4lFiTA1v23xAOFYW1znk=; b=ZN7TMzWcg2kpOSqffDoVhtCt8uuThETRtG/2q52T5qCUyS9wVVeqC8DLp9GHUxjPzl aRJom2yDGA29gjrTZrqqST+sCuyxUep0AcG3zyv30IblaP/4C9KEnPFKrtG+ZDCaIltz G1DBpjrBArDifkZHj2qKeMp5ucykr8vNPn9d5SaiPvot0qDGmYdqV7hchxuHrIYBK3Yt zQezM/vBty6xeYGG3xPKCMDP12KKhXP5Pcw3ErAA+STpsQVaYQ62FE6TFa2w6PAZc4YJ a7cIVY5OSaYEJLpdXMS0djxudcoIzYtYFsaSPvip1YOmJbhIujEddKb27SQvDMsnNhfF PBPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=F6/Ku+z/rldXTKyTjaRYVBv4lFiTA1v23xAOFYW1znk=; b=PLiMb/Ce3pScTnQLG9gS4ogaX+2iholnydmk7lw4NYJEfGX82SQQGDwt1z3sggRKUf 3wUk7jkUT8wbp0fcWONsQ9B6BTXL7JSPNdBIb5WjNfCA/0uvvlz9xBgosMUlKnvKfUiO 7TNqROvB0Gz39QqUOXnQhE62iiqn9tIp959HfYnBu2opqG2b9omisqzwzL4pPi1ICj0i VauKeRA27x8FBWQgbvtZnwHt7W+GbikFBdYN68k7xPLKrez19p7ohx0bHw+/hLEYT+N8 ptqXImEjMI/HFGRAMZM28DHkxcDEvYkBxDonMSBhBEKGaB14DcKSKUbeJCzIKdY1vtA/ nfOA== X-Gm-Message-State: AOAM531RWyNElOkloNLamVFl4fllAoOzrwFIx+Rl9P+xBUIObiqWQEFb BQc68Lze9T6WO3klzZT9ZTMdESTRD1H+ X-Google-Smtp-Source: ABdhPJxPq5M30rdGbubfNCRpT9hqgInad/1CQDFyjy/L+083VvfB25t4bfYqH1wKxMY1b6mlTwrfv3eLZu9/ X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:adf:82f5:: with SMTP id 108mr804663wrc.269.1605637011993; Tue, 17 Nov 2020 10:16:51 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:54 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-15-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 14/27] KVM: arm64: Factor out vector address calculation From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131655_761343_E84236C9 X-CRM114-Status: GOOD ( 14.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-map the guest vectors at EL2 when pKVM is enabled, refactor __kvm_vector_slot2idx() and kvm_init_vector_slot() to move all the address calculation logic in a static inline function. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 8 ++++++++ arch/arm64/kvm/arm.c | 9 +-------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 5168a0c516ae..cb104443d8e4 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -171,6 +171,14 @@ phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(void); +static inline void *__kvm_vector_slot2addr(void *base, + enum arm64_hyp_spectre_vector slot) +{ + int idx = slot - (slot != HYP_VECTOR_DIRECT); + + return base + (idx * SZ_2K); +} + struct kvm; #define kvm_flush_dcache_to_poc(a,l) __flush_dcache_area((a), (l)) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index c7f8fca97202..b1e1747e4bbf 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1318,16 +1318,9 @@ static unsigned long nvhe_percpu_order(void) /* A lookup table holding the hypervisor VA for each vector slot */ static void *hyp_spectre_vector_selector[BP_HARDEN_EL2_SLOTS]; -static int __kvm_vector_slot2idx(enum arm64_hyp_spectre_vector slot) -{ - return slot - (slot != HYP_VECTOR_DIRECT); -} - static void kvm_init_vector_slot(void *base, enum arm64_hyp_spectre_vector slot) { - int idx = __kvm_vector_slot2idx(slot); - - hyp_spectre_vector_selector[slot] = base + (idx * SZ_2K); + hyp_spectre_vector_selector[slot] = __kvm_vector_slot2addr(base, slot); } static int kvm_init_vector_slots(void) From patchwork Tue Nov 17 18:15:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42D49C5519F for ; Tue, 17 Nov 2020 18:23:03 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9873B22210 for ; Tue, 17 Nov 2020 18:23:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="GXtKbZWe"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="JQlCWex6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9873B22210 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=G1H30bnHWPKdGH9tdYvSn408/FW/e3X+eJD9dtP/VEA=; b=GXtKbZWelXhAzL20vsnK8vFOa iqPBchZTFVPjda/VNsxAx5VdkZnyb1HB7IgB5iImu5G9Em23RnsHSWo4gfrm+YjVXJc6xRLdjQS1u JRHK0Vb8ocLOka+sZuJUbEmchIVeeTqehAXu1CxlkvgM3wXDzeY+wJAXK8D2QZwmYqgnhCSrR2Wqw JJWYIyTVUrPAT+QlLU6f5Dd9KPvaLQbcX2xizuIXu2v1lKvhgisyge7DyvALFb5RUOJtX1CMdDhLj gP+RkkPg7Yabnx0JR0X/KUTjCB/nuH2DghallBBJDyYiV7UaIqfCMb0IUq85afZJX2Yrbwe5VHHj4 XtXq/FGHw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5bG-0004EN-H2; Tue, 17 Nov 2020 18:21:06 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XG-0002MT-74 for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:16:59 +0000 Received: by mail-wr1-x449.google.com with SMTP id z13so13418045wrm.19 for ; Tue, 17 Nov 2020 10:16:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=9NJwaj6H9dpl3R5GKvz3RD0FPuXlHhNUuulS5FTh3Hk=; b=JQlCWex6GQEr90aD+Xq83dUt3R1pZAtX7hpfO3tXPKjNHdQDhvuF4w7p9Uq1cYYUYD 0qrdweQ40YIYcJpB0rKkgokebr+IL6PKzeAVgT/WlcVavgVeGC39u48slN1Uf9VaNibF 7mREaFFzfgYyEmzz/EM0L6NqrHnXIhZ0MgyYVApx8H6kpW0muE4kg6NHnwBm0Z89W6HC BnINsKivgdOVXfLE+ZlQWsr02uvo74sizwj9Y/eaZ3zIO9kF8K2u6C6mTeKsGxWotXkK amR0Q5jpJ0Dgqi7rng0ETiiuwY82ca7PMq+gKYVBGnztUjGQQ+TpyrGRWn42mo1p7mPT J0Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9NJwaj6H9dpl3R5GKvz3RD0FPuXlHhNUuulS5FTh3Hk=; b=H90qEoFgDtMrruxXAK4VFvkCW5aTAeqAvE78/BWGd2zw7d8Tjf1eMylGclKvS3q6kG wto8UMFdFxcUcef86ofuZ1Y21touEPiR3JLPFbX16fVlTEGBGyhvjOfp7hjITES8o8wA syHTFSK4S4Y68iKHoSJlynU0JiO6A/etkkDxO5wOOfHgt+optOFCizpLG8GUVil+BqXq 9Va3uiJkf1fsCh1IRkZtdgl9BnjCZEOgfKctSRDAsy09I9uJvci6E21g0YFSVuDlpcA/ cXghk9vzWR0FOWSYa38lu+4Y+lvngiApAMpM+ONk+G80FyCTdB7u1vVNkmaOUcxovTok ljPg== X-Gm-Message-State: AOAM530HMrHv5LB8Jn43DZ1MJnO6pFhURG61+tYv3VB/N8KaPvkB26av uXm0gpYyBg/2lONkx1CdrJ3GJHgbDTBe X-Google-Smtp-Source: ABdhPJyGVJgIcTCT/DdbnY1gsliRaeUzyr2tI4CihTRnunVErPVhmxXy6EImfhV9SGx6S5piiKOb6bkx3WEt X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:44d4:: with SMTP id r203mr394378wma.60.1605637014324; Tue, 17 Nov 2020 10:16:54 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:55 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-16-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 15/27] of/fdt: Introduce early_init_dt_add_memory_hyp() From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131658_365480_7C762ECB X-CRM114-Status: GOOD ( 12.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce early_init_dt_add_memory_hyp() to allow KVM to conserve a copy of the memory regions parsed from DT. This will be needed in the context of the protected nVHE feature of KVM/arm64 where the code running at EL2 will be cleanly separated from the host kernel during boot, and will need its own representation of memory. Signed-off-by: Quentin Perret --- drivers/of/fdt.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c index 4602e467ca8b..af2b5a09c5b4 100644 --- a/drivers/of/fdt.c +++ b/drivers/of/fdt.c @@ -1099,6 +1099,10 @@ int __init early_init_dt_scan_chosen(unsigned long node, const char *uname, #define MAX_MEMBLOCK_ADDR ((phys_addr_t)~0) #endif +void __init __weak early_init_dt_add_memory_hyp(u64 base, u64 size) +{ +} + void __init __weak early_init_dt_add_memory_arch(u64 base, u64 size) { const u64 phys_offset = MIN_MEMBLOCK_ADDR; @@ -1139,6 +1143,7 @@ void __init __weak early_init_dt_add_memory_arch(u64 base, u64 size) base = phys_offset; } memblock_add(base, size); + early_init_dt_add_memory_hyp(base, size); } int __init __weak early_init_dt_mark_hotplug_memory_arch(u64 base, u64 size) From patchwork Tue Nov 17 18:15:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35A27C2D0E4 for ; Tue, 17 Nov 2020 18:23:25 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 65D982467D for ; Tue, 17 Nov 2020 18:23:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="YPXYFhZX"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="vj91rGly" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65D982467D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ttQ3ZgjtIro9hu26CdPQpgJ0TAJiHHQ4dWfVGURM3x8=; b=YPXYFhZXMDLKA47gRlkYfCFFB VkKny7zFTuDiiZr4ZUcn7bVGHjg7QHfDu1V+MmUz9olnofGxNxgNlWrOmzazcO09QV2EAVIdBI1DP oTPBgilpo223oNbH4GHcG347eD0grIpEK87RMOhmfEFoEzRcLFnearln5afEsbbtdlSW0wgVPHWU2 6zbsG3dD4r9cxAfp8HD9Yke0In3ENsGFXvJkf49iO2RzL513DHALyBh63ntfY5KRFHq0UQmQTHj62 b7O/RVi22MB3Rynh7Hl0C8eRVXXaWMp7DORD/Y3ininKzr1cltCuPY/0v/y+zEAcRD/HG12bcOmOf 97iP2p8fQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5bM-0004Hm-1a; Tue, 17 Nov 2020 18:21:12 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XI-0002NT-NU for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:06 +0000 Received: by mail-wm1-x34a.google.com with SMTP id y1so2139015wma.5 for ; Tue, 17 Nov 2020 10:16:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=MgizBMNnQpOXMuPbGh3Gv9pNQJdacDwGgNswbIKCgRM=; b=vj91rGlyBUiDaweBntl+5cvf32le9A/39kXxtbIkljG8ZEKn/l/CPd8HLS2CKj5WZz AnOXzv5VnD2+DDeBd1baBNBOt3sJC1ZZzjXs/+RjCpAxMaxU5MoJ5uLPYCq8so+TzN+m Lk9ci5ay+OQF82KAc7Na5JPfCxpOmzFkgXsc22LYMpnjKGZ1utArWX4ApIRtVwdzXUCw Q8/tshlSiEGdcHIqAYDr6YajRx8x/B+OMGMERGdEUUxq7tTg09BwiD0MowhQtAbEP0Q6 mbqVsNcMNMO1q45lm2lpgzdYzjd/w0Hl4DBRWqJH/E9LRVRdPqy5O8neqkIeKISnsgzf QwcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MgizBMNnQpOXMuPbGh3Gv9pNQJdacDwGgNswbIKCgRM=; b=O5nxhcGnO4T9+VRabAJCG7HowALqQYYK5+E57glGCtbm4lIyPvls5Ff2kZoQj9Tv1S cokOlabkkBUiN9VN7COt3y4l4au6Y1y1vgwTLvXhI+GuOIGaJl9cZzWVBZoOQ8dKgk3r 7m2mn74ZRGCHWl5VB/79f1GZA5ghdXki6U99CkejDMNoZX6Na+uYSGxS4W+5c65xxAYb KCXfdcysSQ0yYWrdueqeeAAgOvHFLwPXQTjslR8HOtp8x4f1tOtdWnG76OmX7ThISc0N rbvCaM9/Wh2GnMDRRVE/0Q34bb5Y7gea+mSzqoeFgXXYWHg/APK1jJnAI0Gv8AT7O0YN /dHA== X-Gm-Message-State: AOAM530JOVsp79LY9cP+c0XVRRURwZ70fezV55570p3qPZks3y/1IX+u j9a2vAuXMaD7DE84CorvPlC3aR/18vM/ X-Google-Smtp-Source: ABdhPJz2eJqLEtX4wrSz5wgqPzcYPIYZ6U19THDj7bMfThHVEgY3yn4Tbc9CIRdWeU2TS/IWvG6covmrMsxf X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:658b:: with SMTP id z133mr353482wmb.1.1605637016704; Tue, 17 Nov 2020 10:16:56 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:56 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-17-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 16/27] KVM: arm64: Prepare Hyp memory protection From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131700_924456_6FE709B3 X-CRM114-Status: GOOD ( 28.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the Hyp code needs the ability to create and manage its own page-table. To do so, introduce a new set of hypercalls to initialize Hyp memory protection. During the init hcall, the hypervisor runs with the host-provided page-table and uses the trivial early page allocator to create its own set of page-tables, using a memory pool that was donated by the host. Specifically, the hypervisor creates its own mappings for __hyp_text, the Hyp memory pool, the __hyp_bss, the portion of hyp_vmemmap corresponding to the Hyp pool, among other things. It then jumps back in the idmap page, switches to use the newly-created pgd (instead of the temporary one provided by the host) and then installs the full-fledged buddy allocator which will then be the only one in used from then on. Note that for the sake of symplifying the review, this only introduces the code doing this operation, without actually being called by anyhing yet. This will be done in a subsequent patch, which will introduce the necessary host kernel changes. Credits to Will for __kvm_init_switch_pgd. Co-authored-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 6 +- arch/arm64/include/asm/kvm_host.h | 8 + arch/arm64/include/asm/kvm_hyp.h | 8 + arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kvm/hyp/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/memory.h | 6 + arch/arm64/kvm/hyp/include/nvhe/mm.h | 79 +++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 30 ++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 44 +++++ arch/arm64/kvm/hyp/nvhe/mm.c | 175 ++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/psci-relay.c | 2 - arch/arm64/kvm/hyp/nvhe/setup.c | 196 +++++++++++++++++++++++ arch/arm64/kvm/hyp/reserved_mem.c | 75 +++++++++ arch/arm64/kvm/mmu.c | 2 +- arch/arm64/mm/init.c | 3 + 17 files changed, 653 insertions(+), 8 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mm.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mm.c create mode 100644 arch/arm64/kvm/hyp/nvhe/setup.c create mode 100644 arch/arm64/kvm/hyp/reserved_mem.c diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index e4934f5e4234..9266b17f8ba9 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -57,6 +57,10 @@ #define __KVM_HOST_SMCCC_FUNC___kvm_get_mdcr_el2 12 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs 13 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs 14 +#define __KVM_HOST_SMCCC_FUNC___kvm_hyp_protect 15 +#define __KVM_HOST_SMCCC_FUNC___hyp_create_mappings 16 +#define __KVM_HOST_SMCCC_FUNC___hyp_create_private_mapping 17 +#define __KVM_HOST_SMCCC_FUNC___hyp_cpu_set_vector 18 #ifndef __ASSEMBLY__ @@ -171,7 +175,7 @@ struct kvm_vcpu; struct kvm_s2_mmu; DECLARE_KVM_NVHE_SYM(__kvm_hyp_init); -DECLARE_KVM_NVHE_SYM(__kvm_hyp_host_vector); +DECLARE_KVM_HYP_SYM(__kvm_hyp_host_vector); DECLARE_KVM_HYP_SYM(__kvm_hyp_vector); #define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init) #define __kvm_hyp_host_vector CHOOSE_NVHE_SYM(__kvm_hyp_host_vector) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7a5d5f4b3351..ee8bb8021637 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -742,4 +742,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); #define kvm_vcpu_has_pmu(vcpu) \ (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features)) +#ifdef CONFIG_KVM +extern phys_addr_t hyp_mem_base; +extern phys_addr_t hyp_mem_size; +void __init reserve_kvm_hyp(void); +#else +static inline void reserve_kvm_hyp(void) { } +#endif + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 95a2bbbcc7e1..dbd2ef86afa9 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -105,5 +105,13 @@ void __noreturn hyp_panic(void); void __noreturn __hyp_do_panic(bool restore_host, u64 spsr, u64 elr, u64 par); #endif +#ifdef __KVM_NVHE_HYPERVISOR__ +void __kvm_init_switch_pgd(phys_addr_t phys, unsigned long size, + phys_addr_t pgd, void *sp, void *cont_fn); +int __kvm_hyp_protect(phys_addr_t phys, unsigned long size, + unsigned long nr_cpus, unsigned long *per_cpu_base); +void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); +#endif + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 3bc86d1423f8..010458f6d799 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1722,7 +1722,7 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap) #endif /* CONFIG_ARM64_MTE */ #ifdef CONFIG_KVM -static bool enable_protected_kvm; +bool enable_protected_kvm; static bool has_protected_kvm(const struct arm64_cpu_capabilities *entry, int __unused) { diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index c35d768672eb..f2d43e6cd86d 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -118,6 +118,25 @@ __kvm_nvhe___memset = __kvm_nvhe___pi_memset; _kvm_nvhe___flush_dcache_area = __kvm_nvhe___pi___flush_dcache_area; +/* Hypevisor VA size */ +KVM_NVHE_ALIAS(hyp_va_bits); + +/* Kernel memory sections */ +KVM_NVHE_ALIAS(__start_rodata); +KVM_NVHE_ALIAS(__end_rodata); +KVM_NVHE_ALIAS(__bss_start); +KVM_NVHE_ALIAS(__bss_stop); + +/* Hyp memory sections */ +KVM_NVHE_ALIAS(__hyp_idmap_text_start); +KVM_NVHE_ALIAS(__hyp_idmap_text_end); +KVM_NVHE_ALIAS(__hyp_text_start); +KVM_NVHE_ALIAS(__hyp_text_end); +KVM_NVHE_ALIAS(__hyp_data_ro_after_init_start); +KVM_NVHE_ALIAS(__hyp_data_ro_after_init_end); +KVM_NVHE_ALIAS(__hyp_bss_start); +KVM_NVHE_ALIAS(__hyp_bss_end); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile index 687598e41b21..b726332eec49 100644 --- a/arch/arm64/kvm/hyp/Makefile +++ b/arch/arm64/kvm/hyp/Makefile @@ -10,4 +10,4 @@ subdir-ccflags-y := -I$(incdir) \ -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o +obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o reserved_mem.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h index ed47674bc988..c8af6fe87bfb 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/memory.h +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -6,6 +6,12 @@ #include +#define HYP_MEMBLOCK_REGIONS 128 +struct hyp_memblock_region { + phys_addr_t start; + phys_addr_t end; +}; + struct hyp_pool; struct hyp_page { unsigned int refcount; diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h new file mode 100644 index 000000000000..5a3ad6f4e5bc --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MM_H +#define __KVM_HYP_MM_H + +#include +#include +#include + +#include +#include + +extern struct hyp_memblock_region kvm_nvhe_sym(hyp_memory)[]; +extern int kvm_nvhe_sym(hyp_memblock_nr); +extern struct kvm_pgtable hyp_pgtable; +extern hyp_spinlock_t __hyp_pgd_lock; +extern struct hyp_pool hpool; +extern u64 __io_map_base; +extern u32 hyp_va_bits; + +int hyp_create_idmap(void); +int hyp_map_vectors(void); +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); +int hyp_cpu_set_vector(enum arm64_hyp_spectre_vector slot); +int hyp_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +int __hyp_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, unsigned long prot); +unsigned long __hyp_create_private_mapping(phys_addr_t phys, size_t size, + unsigned long prot); + +static inline void hyp_vmemmap_range(phys_addr_t phys, unsigned long size, + unsigned long *start, unsigned long *end) +{ + unsigned long nr_pages = size >> PAGE_SHIFT; + struct hyp_page *p = hyp_phys_to_page(phys); + + *start = (unsigned long)p; + *end = *start + nr_pages * sizeof(struct hyp_page); + *start = ALIGN_DOWN(*start, PAGE_SIZE); + *end = ALIGN(*end, PAGE_SIZE); +} + +static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) +{ + unsigned long total = 0, i; + + /* Provision the worst case scenario with 4 levels of page-table */ + for (i = 0; i < 4; i++) { + nr_pages = DIV_ROUND_UP(nr_pages, PTRS_PER_PTE); + total += nr_pages; + } + + return total; +} + +static inline unsigned long hyp_s1_pgtable_size(void) +{ + struct hyp_memblock_region *reg; + unsigned long nr_pages, res = 0; + int i; + + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) + return 0; + + for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) { + reg = &kvm_nvhe_sym(hyp_memory)[i]; + nr_pages = (reg->end - reg->start) >> PAGE_SHIFT; + nr_pages = __hyp_pgtable_max_pages(nr_pages); + res += nr_pages << PAGE_SHIFT; + } + + /* Allow 1 GiB for private mappings */ + nr_pages = (1 << 30) >> PAGE_SHIFT; + nr_pages = __hyp_pgtable_max_pages(nr_pages); + res += nr_pages << PAGE_SHIFT; + + return res; +} + +#endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 72cfe53f106f..d7381a503182 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -11,9 +11,9 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o + cache.o cpufeature.o setup.o mm.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ - ../fpsimd.o ../hyp-entry.o ../exception.o + ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) ## diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index 8f3602f320ac..e2d62297edfe 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -247,4 +247,34 @@ alternative_else_nop_endif SYM_CODE_END(__kvm_handle_stub_hvc) +SYM_FUNC_START(__kvm_init_switch_pgd) + /* Turn the MMU off */ + pre_disable_mmu_workaround + mrs x2, sctlr_el2 + bic x3, x2, #SCTLR_ELx_M + msr sctlr_el2, x3 + isb + + tlbi alle2 + + /* Install the new pgtables */ + ldr x3, [x0, #NVHE_INIT_PGD_PA] + phys_to_ttbr x4, x3 +alternative_if ARM64_HAS_CNP + orr x4, x4, #TTBR_CNP_BIT +alternative_else_nop_endif + msr ttbr0_el2, x4 + + /* Set the new stack pointer */ + ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] + mov sp, x0 + + /* And turn the MMU back on! */ + dsb nsh + isb + msr sctlr_el2, x2 + isb + ret x1 +SYM_FUNC_END(__kvm_init_switch_pgd) + .popsection diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 933329699425..a0bfe0d26da6 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -6,12 +6,15 @@ #include +#include #include #include #include #include #include +#include + DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); #define cpu_reg(ctxt, r) (ctxt)->regs.regs[r] @@ -106,6 +109,43 @@ static void handle___vgic_v3_restore_aprs(struct kvm_cpu_context *host_ctxt) __vgic_v3_restore_aprs(kern_hyp_va(cpu_if)); } +static void handle___kvm_hyp_protect(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, nr_cpus, host_ctxt, 3); + DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 4); + + cpu_reg(host_ctxt, 1) = __kvm_hyp_protect(phys, size, nr_cpus, + per_cpu_base); +} + +static void handle___hyp_cpu_set_vector(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(enum arm64_hyp_spectre_vector, slot, host_ctxt, 1); + + cpu_reg(host_ctxt, 1) = hyp_cpu_set_vector(slot); +} + +static void handle___hyp_create_mappings(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(unsigned long, start, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, phys, host_ctxt, 3); + DECLARE_REG(unsigned long, prot, host_ctxt, 4); + + cpu_reg(host_ctxt, 1) = __hyp_create_mappings(start, size, phys, prot); +} + +static void handle___hyp_create_private_mapping(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(size_t, size, host_ctxt, 2); + DECLARE_REG(unsigned long, prot, host_ctxt, 3); + + cpu_reg(host_ctxt, 1) = __hyp_create_private_mapping(phys, size, prot); +} + typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = kimg_fn_ptr(handle_##x) @@ -125,6 +165,10 @@ static const hcall_t *host_hcall[] = { HANDLE_FUNC(__kvm_get_mdcr_el2), HANDLE_FUNC(__vgic_v3_save_aprs), HANDLE_FUNC(__vgic_v3_restore_aprs), + HANDLE_FUNC(__kvm_hyp_protect), + HANDLE_FUNC(__hyp_cpu_set_vector), + HANDLE_FUNC(__hyp_create_mappings), + HANDLE_FUNC(__hyp_create_private_mapping), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c new file mode 100644 index 000000000000..cad5dae197c6 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct kvm_pgtable hyp_pgtable; + +hyp_spinlock_t __hyp_pgd_lock; +u64 __io_map_base; + +struct hyp_memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS]; +int hyp_memblock_nr; + +int __hyp_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, unsigned long prot) +{ + int err; + + hyp_spin_lock(&__hyp_pgd_lock); + err = kvm_pgtable_hyp_map(&hyp_pgtable, start, size, phys, prot); + hyp_spin_unlock(&__hyp_pgd_lock); + + return err; +} + +unsigned long __hyp_create_private_mapping(phys_addr_t phys, size_t size, + unsigned long prot) +{ + unsigned long addr; + int ret; + + hyp_spin_lock(&__hyp_pgd_lock); + + size = PAGE_ALIGN(size + offset_in_page(phys)); + addr = __io_map_base; + __io_map_base += size; + + /* Are we overflowing on the vmemmap ? */ + if (__io_map_base > __hyp_vmemmap) { + __io_map_base -= size; + addr = 0; + goto out; + } + + ret = kvm_pgtable_hyp_map(&hyp_pgtable, addr, size, phys, prot); + if (ret) { + addr = 0; + goto out; + } + + addr = addr + offset_in_page(phys); +out: + hyp_spin_unlock(&__hyp_pgd_lock); + + return addr; +} + +int hyp_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) +{ + unsigned long start = (unsigned long)from; + unsigned long end = (unsigned long)to; + unsigned long virt_addr; + phys_addr_t phys; + + start = start & PAGE_MASK; + end = PAGE_ALIGN(end); + + for (virt_addr = start; virt_addr < end; virt_addr += PAGE_SIZE) { + int err; + + phys = hyp_virt_to_phys((void *)virt_addr); + err = __hyp_create_mappings(virt_addr, PAGE_SIZE, phys, prot); + if (err) + return err; + } + + return 0; +} + +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back) +{ + unsigned long start, end; + + hyp_vmemmap_range(phys, size, &start, &end); + + return __hyp_create_mappings(start, end - start, back, PAGE_HYP); +} + +static void *__hyp_bp_vect_base; +int hyp_cpu_set_vector(enum arm64_hyp_spectre_vector slot) +{ + void *vector; + + switch (slot) { + case HYP_VECTOR_DIRECT: { + vector = hyp_symbol_addr(__kvm_hyp_vector); + break; + } + case HYP_VECTOR_SPECTRE_DIRECT: { + vector = hyp_symbol_addr(__bp_harden_hyp_vecs); + break; + } + case HYP_VECTOR_INDIRECT: + case HYP_VECTOR_SPECTRE_INDIRECT: { + vector = (void *)__hyp_bp_vect_base; + break; + } + default: + return -EINVAL; + } + + vector = __kvm_vector_slot2addr(vector, slot); + *this_cpu_ptr(&kvm_hyp_vector) = (unsigned long)vector; + + return 0; +} + +int hyp_map_vectors(void) +{ + unsigned long bp_base; + + if (!cpus_have_const_cap(ARM64_SPECTRE_V3A)) + return 0; + + bp_base = (unsigned long)hyp_symbol_addr(__bp_harden_hyp_vecs); + bp_base = __hyp_pa(bp_base); + bp_base = __hyp_create_private_mapping(bp_base, __BP_HARDEN_HYP_VECS_SZ, + PAGE_HYP_EXEC); + if (!bp_base) + return -1; + + __hyp_bp_vect_base = (void *)bp_base; + + return 0; +} + +int hyp_create_idmap(void) +{ + unsigned long start, end; + + start = (unsigned long)hyp_symbol_addr(__hyp_idmap_text_start); + start = hyp_virt_to_phys((void *)start); + start = ALIGN_DOWN(start, PAGE_SIZE); + + end = (unsigned long)hyp_symbol_addr(__hyp_idmap_text_end); + end = hyp_virt_to_phys((void *)end); + end = ALIGN(end, PAGE_SIZE); + + /* + * One half of the VA space is reserved to linearly map portions of + * memory -- see va_layout.c for more details. The other half of the VA + * space contains the trampoline page, and needs some care. Split that + * second half in two and find the quarter of VA space not conflicting + * with the idmap to place the IOs and the vmemmap. IOs use the lower + * half of the quarter and the vmemmap the upper half. + */ + __io_map_base = start & BIT(hyp_va_bits - 2); + __io_map_base ^= BIT(hyp_va_bits - 2); + __hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3); + + return __hyp_create_mappings(start, end - start, start, PAGE_HYP_EXEC); +} diff --git a/arch/arm64/kvm/hyp/nvhe/psci-relay.c b/arch/arm64/kvm/hyp/nvhe/psci-relay.c index dbe57ae84a0c..cfc6dac0f0ac 100644 --- a/arch/arm64/kvm/hyp/nvhe/psci-relay.c +++ b/arch/arm64/kvm/hyp/nvhe/psci-relay.c @@ -193,8 +193,6 @@ static int psci_cpu_on(u64 func_id, struct kvm_cpu_context *host_ctxt) return ret; } -void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); - asmlinkage void __noreturn __kvm_hyp_psci_cpu_entry(void) { struct kvm_host_psci_state *cpu_state = this_cpu_ptr(&kvm_host_psci_state); diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c new file mode 100644 index 000000000000..9679c97b875b --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include + +#include +#include +#include +#include + +struct hyp_pool hpool; +struct kvm_pgtable_mm_ops hyp_pgtable_mm_ops; +unsigned long hyp_nr_cpus; + +#define hyp_percpu_size ((unsigned long)__per_cpu_end - \ + (unsigned long)__per_cpu_start) + +static void *stacks_base; +static void *vmemmap_base; +static void *hyp_pgt_base; + +static int divide_memory_pool(void *virt, unsigned long size) +{ + unsigned long vstart, vend, nr_pages; + + hyp_early_alloc_init(virt, size); + + stacks_base = hyp_early_alloc_contig(hyp_nr_cpus); + if (!stacks_base) + return -ENOMEM; + + hyp_vmemmap_range(__hyp_pa(virt), size, &vstart, &vend); + nr_pages = (vend - vstart) >> PAGE_SHIFT; + vmemmap_base = hyp_early_alloc_contig(nr_pages); + if (!vmemmap_base) + return -ENOMEM; + + nr_pages = hyp_s1_pgtable_size() >> PAGE_SHIFT; + hyp_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!hyp_pgt_base) + return -ENOMEM; + + return 0; +} + +static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, + unsigned long *per_cpu_base) +{ + void *start, *end, *virt = hyp_phys_to_virt(phys); + int ret, i; + + /* Recreate the hyp page-table using the early page allocator */ + hyp_early_alloc_init(hyp_pgt_base, hyp_s1_pgtable_size()); + ret = kvm_pgtable_hyp_init(&hyp_pgtable, hyp_va_bits, + &hyp_early_alloc_mm_ops); + if (ret) + return ret; + + ret = hyp_create_idmap(); + if (ret) + return ret; + + ret = hyp_map_vectors(); + if (ret) + return ret; + + ret = hyp_back_vmemmap(phys, size, hyp_virt_to_phys(vmemmap_base)); + if (ret) + return ret; + + ret = hyp_create_mappings(hyp_symbol_addr(__hyp_text_start), + hyp_symbol_addr(__hyp_text_end), + PAGE_HYP_EXEC); + if (ret) + return ret; + + ret = hyp_create_mappings(hyp_symbol_addr(__start_rodata), + hyp_symbol_addr(__end_rodata), PAGE_HYP_RO); + if (ret) + return ret; + + ret = hyp_create_mappings(hyp_symbol_addr(__hyp_data_ro_after_init_start), + hyp_symbol_addr(__hyp_data_ro_after_init_end), + PAGE_HYP_RO); + if (ret) + return ret; + + ret = hyp_create_mappings(hyp_symbol_addr(__bss_start), + hyp_symbol_addr(__hyp_bss_end), PAGE_HYP); + if (ret) + return ret; + + ret = hyp_create_mappings(hyp_symbol_addr(__hyp_bss_end), + hyp_symbol_addr(__bss_stop), PAGE_HYP_RO); + if (ret) + return ret; + + ret = hyp_create_mappings(virt, virt + size - 1, PAGE_HYP); + if (ret) + return ret; + + for (i = 0; i < hyp_nr_cpus; i++) { + start = (void *)kern_hyp_va(per_cpu_base[i]); + end = start + PAGE_ALIGN(hyp_percpu_size); + ret = hyp_create_mappings(start, end, PAGE_HYP); + if (ret) + return ret; + } + + return 0; +} + +static void update_nvhe_init_params(void) +{ + struct kvm_nvhe_init_params *params; + unsigned long i, stack; + + for (i = 0; i < hyp_nr_cpus; i++) { + stack = (unsigned long)stacks_base + (i << PAGE_SHIFT); + params = per_cpu_ptr(&kvm_init_params, i); + params->stack_hyp_va = stack + PAGE_SIZE; + params->pgd_pa = __hyp_pa(hyp_pgtable.pgd); + __flush_dcache_area(params, sizeof(*params)); + } +} + +static void *hyp_zalloc_hyp_page(void *arg) +{ + return hyp_alloc_pages(&hpool, HYP_GFP_ZERO, 0); +} + +void __noreturn __kvm_hyp_protect_finalise(void) +{ + struct kvm_host_data *host_data = this_cpu_ptr(&kvm_host_data); + struct kvm_cpu_context *host_ctxt = &host_data->host_ctxt; + unsigned long nr_pages, used_pages; + int ret; + + /* Now that the vmemmap is backed, install the full-fledged allocator */ + nr_pages = hyp_s1_pgtable_size() >> PAGE_SHIFT; + used_pages = hyp_early_alloc_nr_pages(); + ret = hyp_pool_init(&hpool, __hyp_pa(hyp_pgt_base), nr_pages, used_pages); + if (ret) + goto out; + + hyp_pgtable_mm_ops.zalloc_page = hyp_zalloc_hyp_page; + hyp_pgtable_mm_ops.phys_to_virt = hyp_phys_to_virt; + hyp_pgtable_mm_ops.virt_to_phys = hyp_virt_to_phys; + hyp_pgtable_mm_ops.get_page = hyp_get_page; + hyp_pgtable_mm_ops.put_page = hyp_put_page; + hyp_pgtable.mm_ops = &hyp_pgtable_mm_ops; + +out: + host_ctxt->regs.regs[0] = SMCCC_RET_SUCCESS; + host_ctxt->regs.regs[1] = ret; + + __host_enter(host_ctxt); +} + +int __kvm_hyp_protect(phys_addr_t phys, unsigned long size, + unsigned long nr_cpus, unsigned long *per_cpu_base) +{ + struct kvm_nvhe_init_params *params; + void *virt = hyp_phys_to_virt(phys); + void (*fn)(phys_addr_t params_pa, void *finalize_fn_va); + int ret; + + if (phys % PAGE_SIZE || size % PAGE_SIZE || (u64)virt % PAGE_SIZE) + return -EINVAL; + + hyp_spin_lock_init(&__hyp_pgd_lock); + hyp_nr_cpus = nr_cpus; + + ret = divide_memory_pool(virt, size); + if (ret) + return ret; + + ret = recreate_hyp_mappings(phys, size, per_cpu_base); + if (ret) + return ret; + + update_nvhe_init_params(); + + /* Jump in the idmap page to switch to the new page-tables */ + params = this_cpu_ptr(&kvm_init_params); + fn = (typeof(fn))__hyp_pa(hyp_symbol_addr(__kvm_init_switch_pgd)); + fn(__hyp_pa(params), hyp_symbol_addr(__kvm_hyp_protect_finalise)); + + unreachable(); +} diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c new file mode 100644 index 000000000000..02b0b18006f5 --- /dev/null +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include +#include + +#include + +#include +#include + +phys_addr_t hyp_mem_base; +phys_addr_t hyp_mem_size; + +void __init early_init_dt_add_memory_hyp(u64 base, u64 size) +{ + struct hyp_memblock_region *reg; + + if (kvm_nvhe_sym(hyp_memblock_nr) >= HYP_MEMBLOCK_REGIONS) + kvm_nvhe_sym(hyp_memblock_nr) = -1; + + if (kvm_nvhe_sym(hyp_memblock_nr) < 0) + return; + + reg = kvm_nvhe_sym(hyp_memory); + reg[kvm_nvhe_sym(hyp_memblock_nr)].start = base; + reg[kvm_nvhe_sym(hyp_memblock_nr)].end = base + size; + kvm_nvhe_sym(hyp_memblock_nr)++; +} + +extern bool enable_protected_kvm; +void __init reserve_kvm_hyp(void) +{ + u64 nr_pages, prev; + + if (!enable_protected_kvm) + return; + + if (!is_hyp_mode_available() || is_kernel_in_hyp_mode()) + return; + + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) + return; + + hyp_mem_size += num_possible_cpus() << PAGE_SHIFT; + hyp_mem_size += hyp_s1_pgtable_size(); + + /* + * The hyp_vmemmap needs to be backed by pages, but these pages + * themselves need to be present in the vmemmap, so compute the number + * of pages needed by looking for a fixed point. + */ + nr_pages = 0; + do { + prev = nr_pages; + nr_pages = (hyp_mem_size >> PAGE_SHIFT) + prev; + nr_pages = DIV_ROUND_UP(nr_pages * sizeof(struct hyp_page), PAGE_SIZE); + nr_pages += __hyp_pgtable_max_pages(nr_pages); + } while (nr_pages != prev); + hyp_mem_size += nr_pages << PAGE_SHIFT; + + hyp_mem_base = memblock_find_in_range(0, memblock_end_of_DRAM(), + hyp_mem_size, SZ_2M); + if (!hyp_mem_base) { + kvm_err("Failed to reserve hyp memory\n"); + return; + } + memblock_reserve(hyp_mem_base, hyp_mem_size); + + kvm_info("Reserved %lld MiB at 0x%llx\n", hyp_mem_size >> 20, + hyp_mem_base); +} diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 278e163beda4..3cf9397dabdb 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1264,10 +1264,10 @@ static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { .virt_to_phys = kvm_host_pa, }; +u32 hyp_va_bits; int kvm_mmu_init(void) { int err; - u32 hyp_va_bits; hyp_idmap_start = __pa_symbol(__hyp_idmap_text_start); hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE); diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 095540667f0f..f81da019b677 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -390,6 +391,8 @@ void __init arm64_memblock_init(void) reserve_elfcorehdr(); + reserve_kvm_hyp(); + high_memory = __va(memblock_end_of_DRAM() - 1) + 1; dma_contiguous_reserve(arm64_dma32_phys_limit); From patchwork Tue Nov 17 18:15:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91F2CC2D0E4 for ; Tue, 17 Nov 2020 18:23:02 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EDCD220888 for ; Tue, 17 Nov 2020 18:23:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ZE/c9gLc"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="NJLgqh5H" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDCD220888 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rthbicB9e1WVJtE6aewyNDxi9eyxqWKSSNXqDESig8U=; b=ZE/c9gLcDA0wgAl8aVFRHIa6y 6UJ1RySRenS8/j2OLlt+TNi/f9WzWKknuyRdiRwTVVfS7H/06J48lektWtH55v85MmlcCcs+onC1L hYWrY5/Q+UQQlqMFybMrnShOObO6eJmrbLXIIEDutd8r2H5lI/MzWC9UHnwvoqvfLZmw8x09icAQd h/dHyPGZYMRW+0GlnsQk56SEKZxfo9rJg3hCFnfVbQmhSOQSWcHHqBvYnARO66pjkBfBzsB9bRi/K xVMz7IHF6XxNPvuCLAGNz+ch/HqnQpxXuzjxnfAAAz0EB6fYJUfyT5zc9lZMGMQvJA2FCwYN+9yK7 ytTlr59YA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5cL-0004ll-Sr; Tue, 17 Nov 2020 18:22:13 +0000 Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XL-0002OT-8s for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:12 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id a1so13600783qvv.18 for ; Tue, 17 Nov 2020 10:17:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Px6WAgjKWF2Q0iqwF7bmFxmseAlEjtKBsgqTnzkkNWs=; b=NJLgqh5HBJngzvYPET5mEaPI73GpSKAf4rJXeE6y8BoRGUgdmGNL8Ri4PkROPi6FUs sBzLkCyeyPxVaN9jtD0MMHQlnrSA76EKS5GArc0RZ48GV2tAms/4qOC/gsJHLw+/8+kQ 27vI8g71hHkjnRxdvbxBG/o7y4vNgVpvBctJdSyUJagWy0Y9IASIv5nV2j7SWOJJjLYx ZUjB4QS1y8BOo6IY5XcN9nGEmicXNxCrp2F0vmDDaqf0TxrEKEzgSOCatomjtBBlRQQH dyeQe39UQ4bZlvedXtUtNM6uc0v3By9WnNozWhxdarrlqjcgPQUu+l4FeYr5smA8ydts GIfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Px6WAgjKWF2Q0iqwF7bmFxmseAlEjtKBsgqTnzkkNWs=; b=Ayo+515cXdjI4ywWlWO5x5D4XxbfxlbLBHd7oXGnnUf60kbEqZQiJFs/7+I3p8C+p0 nNOpwTVUVhBBrdQC+M8R9EpCWAT3kCCkLkh8yQiXzItpfL/RtXcGMdQcj5g1dEYuSlVj 9hk/WRAPuspvKLzC3jRau9Z4b1XoF3uDi0yGUAlLJh3eCRNk4zaZtktQXxBILlkVsTj9 tKTvzqrpuwgkLQr+duUGTPGC/Qc/LgU+NyrGlhj/XEuTKr9bj0SO8fUOv3rL/oMWPcOp cUDslLcjEEnT+j7YE+aM/3erAXLjFguikTTkcPSzeRRg7cvdqObyNQt95ShCJFSrsEmM Z3BA== X-Gm-Message-State: AOAM53120M+48gAF+QC87KDCgiSu7qLbtZxWeDsOzdO58CVYmOLnNuak +8wIjqPmGRhP9NVkX/Abu3LRD9xv5mJh X-Google-Smtp-Source: ABdhPJzrKHjfNzUAFaFELqtQU9Z839XHaSsET7WjpyCI34QBLSq4T6TTN/LNOv3c8qkVPSVywbDlrFz4Lbl0 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:e50a:: with SMTP id l10mr812333qvm.55.1605637019068; Tue, 17 Nov 2020 10:16:59 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:57 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-18-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 17/27] KVM: arm64: Elevate Hyp mappings creation at EL2 From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131703_410798_F9720DC2 X-CRM114-Status: GOOD ( 23.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Previous commits have introduced infrastructure at EL2 to enable the Hyp code to manage its own memory, and more specifically its stage 1 page tables. However, this was preliminary work, and none of it is currently in use. Put all of this together by elevating the hyp mappings creation at EL2 when memory protection is enabled. In this case, the host kernel running at EL1 still creates _temporary_ Hyp mappings, only used while initializing the hypervisor, but frees them right after, and flips a static key marking the new 'protected' mode of operation. As such, all calls to create_hyp_mappings() after kvm init has finished turn into hypercalls, as the host now has no 'legal' way to modify the hypevisor page tables directly. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 1 - arch/arm64/kvm/arm.c | 51 ++++++++++++++++++++++++++++++-- arch/arm64/kvm/mmu.c | 34 +++++++++++++++++++++ 3 files changed, 82 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index cb104443d8e4..bb756757b51c 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -285,6 +285,5 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) */ asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } - #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index b1e1747e4bbf..cfe5cc55b425 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1373,7 +1373,7 @@ static void cpu_prepare_hyp_mode(int cpu) __flush_dcache_area(params, sizeof(*params)); } -static void cpu_init_hyp_mode(void) +static void kvm_set_hyp_vector(void) { struct kvm_nvhe_init_params *params; struct arm_smccc_res res; @@ -1391,6 +1391,11 @@ static void cpu_init_hyp_mode(void) params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); +} + +static void cpu_init_hyp_mode(void) +{ + kvm_set_hyp_vector(); /* * Disabling SSBD on a non-VHE system requires us to enable SSBS @@ -1433,7 +1438,10 @@ static void cpu_set_hyp_vector(void) struct bp_hardening_data *data = this_cpu_ptr(&bp_hardening_data); void *vector = hyp_spectre_vector_selector[data->slot]; - *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + if (!is_protected_kvm_enabled()) + *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + else + kvm_call_hyp_nvhe(__hyp_cpu_set_vector, data->slot); } static void cpu_hyp_reinit(void) @@ -1441,13 +1449,14 @@ static void cpu_hyp_reinit(void) kvm_init_host_cpu_context(&this_cpu_ptr_hyp_sym(kvm_host_data)->host_ctxt); cpu_hyp_reset(); - cpu_set_hyp_vector(); if (is_kernel_in_hyp_mode()) kvm_timer_init_vhe(); else cpu_init_hyp_mode(); + cpu_set_hyp_vector(); + kvm_arm_init_debug(); if (vgic_present) @@ -1653,6 +1662,36 @@ static int copy_cpu_ftr_regs(void) return 0; } +static int kvm_hyp_enable_protection(void) +{ + void *per_cpu_base = kvm_ksym_ref(kvm_arm_hyp_percpu_base); + int ret, cpu; + void *addr; + + if (!is_protected_kvm_enabled()) + return 0; + + if (!hyp_mem_base) + return -ENOMEM; + + addr = phys_to_virt(hyp_mem_base); + ret = create_hyp_mappings(addr, addr + hyp_mem_size - 1, PAGE_HYP); + if (ret) + return ret; + + kvm_set_hyp_vector(); + ret = kvm_call_hyp_nvhe(__kvm_hyp_protect, hyp_mem_base, hyp_mem_size, + num_possible_cpus(), kern_hyp_va(per_cpu_base)); + if (ret) + return ret; + + free_hyp_pgds(); + for_each_possible_cpu(cpu) + free_page(per_cpu(kvm_arm_hyp_stack_page, cpu)); + + return 0; +} + /** * Inits Hyp-mode on all online CPUs */ @@ -1789,6 +1828,12 @@ static int init_hyp_mode(void) for_each_possible_cpu(cpu) cpu_prepare_hyp_mode(cpu); + err = kvm_hyp_enable_protection(); + if (err) { + kvm_err("Failed to enable hyp memory protection: %d\n", err); + goto out_err; + } + return 0; out_err: diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 3cf9397dabdb..5c2e0feb9689 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -225,15 +225,39 @@ void free_hyp_pgds(void) if (hyp_pgtable) { kvm_pgtable_hyp_destroy(hyp_pgtable); kfree(hyp_pgtable); + hyp_pgtable = NULL; } mutex_unlock(&kvm_hyp_pgd_mutex); } +static bool kvm_host_owns_hyp_mappings(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + return false; + + /* + * This can happen at boot time when __create_hyp_mappings() is called + * after the hyp protection has been enabled, but the static key has + * not been flipped yet. + */ + if (!hyp_pgtable && is_protected_kvm_enabled()) + return false; + + BUG_ON(!hyp_pgtable); + + return true; +} + static int __create_hyp_mappings(unsigned long start, unsigned long size, unsigned long phys, enum kvm_pgtable_prot prot) { int err; + if (!kvm_host_owns_hyp_mappings()) { + return kvm_call_hyp_nvhe(__hyp_create_mappings, + start, size, phys, prot); + } + mutex_lock(&kvm_hyp_pgd_mutex); err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot); mutex_unlock(&kvm_hyp_pgd_mutex); @@ -295,6 +319,16 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, unsigned long base; int ret = 0; + if (!kvm_host_owns_hyp_mappings()) { + base = kvm_call_hyp_nvhe(__hyp_create_private_mapping, + phys_addr, size, prot); + if (!base) + return -ENOMEM; + *haddr = base; + + return 0; + } + mutex_lock(&kvm_hyp_pgd_mutex); /* From patchwork Tue Nov 17 18:15:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F2A1C2D0E4 for ; Tue, 17 Nov 2020 18:23:58 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C93572467A for ; Tue, 17 Nov 2020 18:23:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="nf1ZHCwR"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="qMufXY54" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C93572467A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cj6yRN4YXLD/lI086KrGT1ghvD84Qc93Nlc+cxBGKlk=; b=nf1ZHCwR0QzXAVIB9/Iyv6M08 2UR7OTWygS1pyiGu8LQrepZhJ3AITm3TKSsaa4g9i2i7b/SxX02+csZ8d/HOiXhARpt7S0AWa8uMI Ib2Pn0wG3Dm0nbeFOrwEZPJAnExtzMxu78/kRuUn2ij91oXwMomcncjwV6EnmpDHwu0lqoOI5T50D U4mnPXLRaKTyMhGL1uJOh5qJxWaUEwH5dAe2QOa/WrbXVVGl6rSmBazikwPTfLSgPUUE76D9/nfuZ kLs1ccN7ml817vhBw1pMXMVFKU0CQ9/Zk3gM0uG+rzdPn00ZVsRdpoQUEgXOSBzfAXd0jfUusuhbt mAD9Pz0Vw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5bx-0004Wo-FH; Tue, 17 Nov 2020 18:21:49 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XM-0002PU-OM for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:11 +0000 Received: by mail-qv1-xf49.google.com with SMTP id du18so4733484qvb.14 for ; Tue, 17 Nov 2020 10:17:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=EZMTqnbgaQsSDAJvbG1XxJlQ79bagawpZ079qLuIDUc=; b=qMufXY54r9Tw35ml8YyFoPlAwDp0r3TGxMAr86L/dQTFHJDmLAygkajnXncZ6HZQ2k 2xNFCfSX0DlCJ065rQ8Hi/6WXafc8DZpYAgTbrHNf1F1+kiPJ8hWXMEfKskS2nWCZ28N d6WRT972maQTo7niliiLO+H94x2r+g8fAK7s9FPaTTY/iyAoYE1MDxxbQz54qAtI0Y+m znuZs/tR2Yho5nq/SRXBlObO0lJOH1SQLeHUCqGq5p7mvYw44FJLW6qMUDQPvqKKmxBc dSxg2dX0HlCBEbRqk9LUZzg4OxnawJSbHzk/FpR0msYJGzYoMIUPRFdwMuqNJdYFlZhs xGnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=EZMTqnbgaQsSDAJvbG1XxJlQ79bagawpZ079qLuIDUc=; b=DbP5ZSSqt4UwI86FaahsVxTPoe1twI/tLmrPwmA5WgoA/Uys2+5/qPPEbPrGang3fE gdg8pZjXwRhtgTnSZMmBibiRgtGDIFHI0NLhEVuMZU1tpA6jvhJSI5LkD0J5mQwMfPDg C9ugYgB/7miCHqEbRBlzWAExJoFqYNYjrC9H1ISoD6U+IyHp9zbCGY7U2uk5r6+CAwZQ UpYppxh/v0ZuK4yeZc12zgKXZiibrGHRIJmWsL3iTY7TcVHYBeNpl5ewBsa+5l/MMeAi RQTB+yMju0kb04vEYk5W4gdL2xWjODZMfQ0gZJ+xX8UAta2XqpkxFQq+Cj4GpW9VKeA4 70kg== X-Gm-Message-State: AOAM532OFPV3vybkU4ng4YME9+84sAuIPmjWTnDNd+fsHWO3isxR1Lr/ zUA3/0QJRqpI6lH/yeAONK7Twa8P4WTW X-Google-Smtp-Source: ABdhPJxEnDzHoSiNaMlrXnG6cnHtgZaXnUxTO1bmEfh2lGjsMU/qzCf4IQGItC9HE+ZfEJGOV/17H4jHf0yD X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a05:6214:612:: with SMTP id z18mr817548qvw.41.1605637021225; Tue, 17 Nov 2020 10:17:01 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:58 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-19-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 18/27] KVM: arm64: Use kvm_arch for stage 2 pgtable From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131704_888066_35FEF09B X-CRM114-Status: GOOD ( 14.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, use struct kvm_arch in lieu of struct kvm as the host will have the former but not the latter. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 5 +++-- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- arch/arm64/kvm/mmu.c | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 45acc9dc6c45..8e8f1d2c5e0e 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -151,12 +151,13 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, /** * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. - * @kvm: KVM structure representing the guest virtual machine. + * @arch: Arch-specific KVM structure representing the guest virtual + * machine. * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops); /** diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 61a8a34ddfdb..96a25d0b7b6e 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -855,11 +855,11 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; - u64 vtcr = kvm->arch.vtcr; + u64 vtcr = arch->vtcr; u32 ia_bits = VTCR_EL2_IPA(vtcr); u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; @@ -872,7 +872,7 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, pgt->ia_bits = ia_bits; pgt->start_level = start_level; pgt->mm_ops = mm_ops; - pgt->mmu = &kvm->arch.mmu; + pgt->mmu = &arch->mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ dsb(ishst); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 5c2e0feb9689..384f2acc0115 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -461,7 +461,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); + err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; From patchwork Tue Nov 17 18:15:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5776C5519F for ; Tue, 17 Nov 2020 18:25:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1F2732467A for ; Tue, 17 Nov 2020 18:25:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mAUBK2D6"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="N1TibN9j" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1F2732467A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vZgrscFY1AR9fPUrtqUfVQvuVIP8rMx0MZ2CgfLje7Q=; b=mAUBK2D69mKk0pKx7Ub7jEWp9 zsS+uXdZz5297fyvT354u0Z0Wmi3vi5yxQp7resamg+2r2zfj9w4A8ajoMyzlcrR7lePh2PjUTtAs weIbDiSnMZ8CzkRnm0aHrjma/DqakEP+VgAVxy2XXptzy87E4ZopXVeui9fAIsu+f6Wtg1o81uxSk qCmv/9cj3iqUypfrar12IBf9tUBVrCtWqUKSBGOZyApibjJaRq37Cd7XOiabpstd6rOmi6yrDVFYh 1zroSW8ChuF8zLHVf7DrrkY98DEn/VDsvJ1Xpe7/K1TGHupdiChDV+Wmne/YBh+0gJ20qp2TtaiZ8 KuG7tgV0Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5d3-000526-QQ; Tue, 17 Nov 2020 18:22:58 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XO-0002QN-7x for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:11 +0000 Received: by mail-wr1-x449.google.com with SMTP id f4so13375412wru.21 for ; Tue, 17 Nov 2020 10:17:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=nDg4Q5UW5VyG5A48tL8468KlneKnyHjdeOZ0CSmRGUk=; b=N1TibN9j6rqyDAz7w2MvN4SO5GZxQitBMjo5jrhjQsL+CqRT6cSgzwpG8/qIi2cqKl QupXDJkW94uUSAHQ/Avd20xRPPX5Qshr8JcXWEIpweKRZChjRI+YU7261YvmSIgucGbb YGbidLKp2mBFY77m2LsC5eN7onLPhV6anYgipS2Wbz8yGdVJ5EkVuUBY24a4Oo3WPnkf WvgYp/NvweM/l3FStseFKS26JKS9j8Idghu6U/AclJSNJVmYb5G6lNg8AiDGCVrU6TVx AGBetFJ2Kvmc3d57+t1arZwyoK4CzoIJCeQ7BEGcY2Uf29V1bO3LFUb4jjqch68AkY8Y GtOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nDg4Q5UW5VyG5A48tL8468KlneKnyHjdeOZ0CSmRGUk=; b=faFMQZvw9xKup46axxJP4u+MXBz4NRCUJa54ZZpR+OrK8J4EWgY9KWqo3P/8S5eW1o R/n9lAQvMYNHsFaXSYwPaVOuvckz5W+Lv4MnF0MxeYrX2YJWv87yhS8skldbnX3EUGGI IM1ByvAXVOiBLeiw39kAUAEJKRLkMIgmsevuArnOINxe7ZBsZUEqeW/msdfVgHI5LLQE OoFef3J6Na19gLwfMmYbs/LjrR/gmo1NCLr+1F/cukK4W7BplZ1CjtjGl1X54Ci7iL7C tHZAgNV+fJfRGLIlvC0FvNkVcRiY9kE6vg8W0QbMR/Yy0Ry8d56KVfBfTgDquCNQIQsp 00Og== X-Gm-Message-State: AOAM533kckyTfEvLv0CcAeOMsB/ybEtTQw25MRzxs2sTTGwSJ5qISfcI ZDvstGH9B/I0zO1H7iVbt8Nn8R/cnMLJ X-Google-Smtp-Source: ABdhPJzBZwi9tCgTF9ABdJKEIr3LviF1NVAku/55zJOf5XT/50xgW9YDis8O5ppDeir1CJBxP84GEo3WRJol X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:e442:: with SMTP id b63mr418225wmh.10.1605637023474; Tue, 17 Nov 2020 10:17:03 -0800 (PST) Date: Tue, 17 Nov 2020 18:15:59 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-20-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 19/27] KVM: arm64: Use kvm_arch in kvm_s2_mmu From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131706_398233_75631B23 X-CRM114-Status: GOOD ( 16.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, change kvm_s2_mmu to use a kvm_arch pointer in lieu of the kvm pointer, as the host will have the former but not the latter. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/asm/kvm_mmu.h | 7 ++++++- arch/arm64/kvm/mmu.c | 8 ++++---- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ee8bb8021637..53b01d25e7d9 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -86,7 +86,7 @@ struct kvm_s2_mmu { /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; - struct kvm *kvm; + struct kvm_arch *arch; }; struct kvm_arch { diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index bb756757b51c..714357ebd278 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -275,7 +275,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) */ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) { - write_sysreg(kern_hyp_va(mmu->kvm)->arch.vtcr, vtcr_el2); + write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -285,5 +285,10 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) */ asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } + +static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) +{ + return container_of(mmu->arch, struct kvm, arch); +} #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 384f2acc0115..3b1c53e754ee 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -169,7 +169,7 @@ static void *kvm_host_va(phys_addr_t phys) static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size, bool may_block) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); phys_addr_t end = start + size; assert_spin_locked(&kvm->mmu_lock); @@ -474,7 +474,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; - mmu->kvm = kvm; + mmu->arch = &kvm->arch; mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); mmu->vmid.vmid_gen = 0; @@ -556,7 +556,7 @@ void stage2_unmap_vm(struct kvm *kvm) void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt = NULL; spin_lock(&kvm->mmu_lock); @@ -625,7 +625,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, */ static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect); } From patchwork Tue Nov 17 18:16:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54FD8C2D0E4 for ; Tue, 17 Nov 2020 18:24:30 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D89862467A for ; Tue, 17 Nov 2020 18:24:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="jUFLYhYt"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="QOvLP7+8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D89862467A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GIekCXWg0Wp9DAoyDWeJ5qFhE5K58WnQGEGopSooTic=; b=jUFLYhYt/JxbbgV/8HrzpFbsj TSRQypz0x1BlbTHX4ZqSsOnzybp6rfRioujR99WyfKDnCYfefBH4bVcoNp3Sw5RTuPPzdxv5uaKeL 4mJu9F3+i/ooaRkYJJqj4GxAvCKyOLDaIl4NTex09Fxqp4bUdTGdtIoJ0aonv8BxWSoOeiitfTqDN i0uwgoWeDlWoceAEEY/1c5dB9RjPza8RoR388TbXjG8sLmwm7Fx7EnI2p8+3lZt1GkClFxakar9Vb hJP5+8z4VJ8gOHv7rTWdqTStGAFSbHpxodMW08Cz/ZHzQ2Q2hojIiqZm/6m+PT0Oe1/2VnL9Vzgtr lC9jeIg9A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5dS-0005Kj-VG; Tue, 17 Nov 2020 18:23:23 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XR-0002R9-D6 for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:14 +0000 Received: by mail-qt1-x849.google.com with SMTP id h26so13086519qtm.2 for ; Tue, 17 Nov 2020 10:17:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=uWTirSBGllb/UnC1/gfuRUGVZCQysfGVvtfYpfqhwps=; b=QOvLP7+8rJSTpw6cIxRDKU37M+xX2BsqlE+SSoroXw4nil7j3KIE5nlZL9QPhOUnV2 neKN1A7pQ+x+SQWyzb6lI+RG4t3CdSbNFNKbHjPSf/y2Cy7OdeGY+IEdNMjcuceKN2L0 T1sJRvfMBIkD++hzeifuRpVg7u3ErEumK8vjo1uj0nq8BIvfiQAko56WeOPhqZZMID8+ SHUSt/rtqPXiPAZnsC7EX97B/GMHUi2XVojFSZHDFVU95enBs0xE3vp9oc+pprUh9dxA 0RVD59xrq8RJdccXUEgST+e87cFbujgI/4HZeyYKmtJHEk87BTRvkL+2Jbca91yf6Bx+ Bx1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=uWTirSBGllb/UnC1/gfuRUGVZCQysfGVvtfYpfqhwps=; b=rLn5pyqVxhPcdDeH5rCYQpADX3jgw+4OJ/yxGfziEopOH/rW8xXml18q6spuXA3c/v 4JBEFfPkrU3PnBfwxI3/JQu8HZCqhNz0ffo975m6JDIDoYgT7irri1kjVggjvjP4TZzN CvSN5rGBvoDD5b3FVr/VDqRilcAfOJQrRFW02TM1yNKWV2Z+8yRDuXclfZiC4W/n1L46 6fmvRtxLji7R9R6umlZP/4vjvdhlTl6ivgufJf42rpcQy2hwQXp/EPV9O+FTt9IEz/ni Dq41e5dVoqkAAXTOBPB/huoCInetSM7+0ajeQTBlfBOZB1hdkF95X0mIq9iuZXR0h+3f DQ7g== X-Gm-Message-State: AOAM532PuEslEPe8RWgUlAnmj+B1n3I2hEIPFzi9cFvmGGCLBqkAFoKI 26YNYdj6uHPxZCksQ/ewJa1lOMBQJEKJ X-Google-Smtp-Source: ABdhPJwdF2Gg54bc1EmlvAWDzl41GijC84V6agc2skK4FVwntGHb1cSvCccXppxqfrsHcQ1LZjG1wvbDZQfU X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:db11:: with SMTP id d17mr513281qvk.39.1605637025659; Tue, 17 Nov 2020 10:17:05 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:00 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-21-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 20/27] KVM: arm64: Set host stage 2 using kvm_nvhe_init_params From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131709_629642_01803F21 X-CRM114-Status: GOOD ( 15.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the registers relevant to host stage 2 enablement to kvm_nvhe_init_params to prepare the ground for enabling it in later patches. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kvm/arm.c | 5 +++++ arch/arm64/kvm/hyp/nvhe/hyp-init.S | 9 +++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 5 +---- 5 files changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 9266b17f8ba9..089eea6e54fc 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -158,6 +158,9 @@ struct kvm_nvhe_init_params { unsigned long stack_hyp_va; unsigned long entry_hyp_va; phys_addr_t pgd_pa; + unsigned long hcr_el2; + unsigned long vttbr; + unsigned long vtcr; }; /* Translate a kernel address @ptr into its equivalent linear mapping */ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 9752100bf01f..2c3813bff6ea 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -115,6 +115,9 @@ int main(void) DEFINE(NVHE_INIT_STACK_HYP_VA, offsetof(struct kvm_nvhe_init_params, stack_hyp_va)); DEFINE(NVHE_INIT_ENTRY_HYP_VA, offsetof(struct kvm_nvhe_init_params, entry_hyp_va)); DEFINE(NVHE_INIT_PGD_PA, offsetof(struct kvm_nvhe_init_params, pgd_pa)); + DEFINE(NVHE_INIT_HCR_EL2, offsetof(struct kvm_nvhe_init_params, hcr_el2)); + DEFINE(NVHE_INIT_VTTBR, offsetof(struct kvm_nvhe_init_params, vttbr)); + DEFINE(NVHE_INIT_VTCR, offsetof(struct kvm_nvhe_init_params, vtcr)); #endif #ifdef CONFIG_CPU_PM DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp)); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index cfe5cc55b425..e06c95a10dba 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1365,6 +1365,11 @@ static void cpu_prepare_hyp_mode(int cpu) params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->entry_hyp_va = kern_hyp_va((unsigned long)kvm_ksym_ref_nvhe(__kvm_hyp_psci_cpu_entry)); params->pgd_pa = kvm_mmu_get_httbr(); + if (is_protected_kvm_enabled()) + params->hcr_el2 = HCR_HOST_NVHE_PROTECTED_FLAGS; + else + params->hcr_el2 = HCR_HOST_NVHE_FLAGS; + params->vttbr = params->vtcr = 0; /* * Flush the init params from the data cache because the struct will diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index e2d62297edfe..9f3f3098670a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -103,6 +103,15 @@ alternative_else_nop_endif ldr x1, [x0, #NVHE_INIT_STACK_HYP_VA] mov sp, x1 + ldr x1, [x0, #NVHE_INIT_HCR_EL2] + msr hcr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTTBR] + msr vttbr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTCR] + msr vtcr_el2, x1 + ldr x1, [x0, #NVHE_INIT_PGD_PA] phys_to_ttbr x0, x1 alternative_if ARM64_HAS_CNP diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index f3d0e9eca56c..979a76cdf9fb 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -97,10 +97,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT; write_sysreg(mdcr_el2, mdcr_el2); - if (is_protected_kvm_enabled()) - write_sysreg(HCR_HOST_NVHE_PROTECTED_FLAGS, hcr_el2); - else - write_sysreg(HCR_HOST_NVHE_FLAGS, hcr_el2); + write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); write_sysreg(CPTR_EL2_DEFAULT, cptr_el2); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } From patchwork Tue Nov 17 18:16:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DEAFC63697 for ; Tue, 17 Nov 2020 19:38:06 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AC8362463B for ; Tue, 17 Nov 2020 19:38:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Tzf3r3cd"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="u6eskt9l" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AC8362463B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TVDCjaTo2KtEbEAItzuAjDXMI4OXoOAkH84L3L9wlfs=; b=Tzf3r3cd4AhTfoCvigaGEBw4H 1NhdcnuPRNr0NpXZZkUKP8k78mDl8TJKzbJv3HTMmI8e9j3DFsuVmTbiP49dUQPdt/yAl7Xauk48f Y2qlEdK8NmyQeV9ALIfW7Ih3tRhPz02LujrPPbZJ0ugbizQjLj25ScR2BMR28g0zqd6T3z5m9IOZU s9+lysOVWeqyPIK0Mm8SI5o3E1KpzVztOKtvER4kGy2LQkx6G2/PtY+8lZiosRMJjDIRUovUo+PwM eIc2XDrsr9+rUxYwiR5154d602y4jnxPQIeO4ntcGlwrTn4CEbpHctb+zRd9HnphPjhfgdL0QRAlQ auyypqIBw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf6mu-0001Dp-Pm; Tue, 17 Nov 2020 19:37:12 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XV-0002SX-4u for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:16 +0000 Received: by mail-qv1-xf49.google.com with SMTP id x9so11700074qvt.16 for ; Tue, 17 Nov 2020 10:17:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=YRRsngS9OD6tBnxQxVeL/idwsF6yEfmuwUZwYdjr4cI=; b=u6eskt9lKKmGSunnDChD5nx9UAFs54CCl+3VWgy6MyEcVVg8OeGSLy6w+3jrAgg5wo lIQFdYhOvLApYwomcOOT904ylp69grlIlOZX+z4g49TOTxv58iJt30YEMMbaDkgokaiV +kDh/6I2WkKdw1mI/OeSLV7yoQu8bc9QP/M3gsk0AO2v4897Wo4N8zHFREx6/Mmrycej FaBnrvSuEuIl+wMhPSOrswKNQVY/YQYHpDnZtnmHTfXn3RryY102vNLKNzxLh5JQxvcA ikQQMFvZGh6oA+q0po5wCBIIFKVotM758kswEkO/JkHpisi3Q+W2jTRHPSKvegIXFjFe 7iFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YRRsngS9OD6tBnxQxVeL/idwsF6yEfmuwUZwYdjr4cI=; b=DM2Tbwhxht6mDnA/07U3v4unUShooL9/bNpwaNl+/bfQlruBPBMTZfiO127Da2sAyL FGu1bQdOCTzKmLNZuqnb4vEDNGcRe7x0HS3EE3jkDPoPbxJODb3p2iCLwf9/54xp/Art h9MWjEeNJ95a1N1y2uunx6Xzw9KgABEtxFJmJosifxl6dBbLxxK5nf5pEn8i3oPCD0Yc TTCrU3J0mCFU6n2x6hZeMUrzMD/osjoLU9arrTWmN3ov9rXxIcyaC45WS8K/PCZlAEVV vSZWBljsoUlDvfl0y2cp1NHh+PP8JC9zSij6rW9WVdWJoMkUpccr10Ghyz4rP7bC24Pi uWNQ== X-Gm-Message-State: AOAM532xpeCB09YVDBsSUnz1uD3Q3K7pbpVv5Vhvi+DSoU3sViTUhUS6 XYxOw5AjBUyzIIShhDwXi5MM41OUWhHc X-Google-Smtp-Source: ABdhPJx5Z5F0VDtWsI/mKAVXvn+oRTGzalsg8aLfqcFVm74qRo8wjQOMWj3cl7sUTCwGro5vT2NLftM2aBIV X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a0c:e443:: with SMTP id d3mr975170qvm.18.1605637028048; Tue, 17 Nov 2020 10:17:08 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:01 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-22-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 21/27] KVM: arm64: Refactor kvm_arm_setup_stage2() From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131713_323151_F30BC000 X-CRM114-Status: GOOD ( 17.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-use some of the stage 2 setup at EL2, factor parts of kvm_arm_setup_stage2() out into static inline functions. No functional change intended. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 48 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/reset.c | 42 +++------------------------- 2 files changed, 52 insertions(+), 38 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 714357ebd278..5a76358e8c7a 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -256,6 +256,54 @@ static inline int kvm_write_guest_lock(struct kvm *kvm, gpa_t gpa, return ret; } +static inline u64 kvm_get_parange(u64 mmfr0) +{ + u64 parange = cpuid_feature_extract_unsigned_field(mmfr0, + ID_AA64MMFR0_PARANGE_SHIFT); + if (parange > ID_AA64MMFR0_PARANGE_MAX) + parange = ID_AA64MMFR0_PARANGE_MAX; + + return parange; +} + +/* + * The VTCR value is common across all the physical CPUs on the system. + * We use system wide sanitised values to fill in different fields, + * except for Hardware Management of Access Flags. HA Flag is set + * unconditionally on all CPUs, as it is safe to run with or without + * the feature and the bit is RES0 on CPUs that don't support it. + */ +static inline u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) +{ + u64 vtcr = VTCR_EL2_FLAGS; + u8 lvls; + + vtcr |= kvm_get_parange(mmfr0) << VTCR_EL2_PS_SHIFT; + vtcr |= VTCR_EL2_T0SZ(phys_shift); + /* + * Use a minimum 2 level page table to prevent splitting + * host PMD huge pages at stage2. + */ + lvls = stage2_pgtable_levels(phys_shift); + if (lvls < 2) + lvls = 2; + vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); + + /* + * Enable the Hardware Access Flag management, unconditionally + * on all CPUs. The features is RES0 on CPUs without the support + * and must be ignored by the CPUs. + */ + vtcr |= VTCR_EL2_HA; + + /* Set the vmid bits */ + vtcr |= (get_vmid_bits(mmfr1) == 16) ? + VTCR_EL2_VS_16BIT : + VTCR_EL2_VS_8BIT; + + return vtcr; +} + #define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr) static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 3e772ea4e066..074b39dbe539 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -384,19 +384,10 @@ int kvm_set_ipa_limit(void) return 0; } -/* - * Configure the VTCR_EL2 for this VM. The VTCR value is common - * across all the physical CPUs on the system. We use system wide - * sanitised values to fill in different fields, except for Hardware - * Management of Access Flags. HA Flag is set unconditionally on - * all CPUs, as it is safe to run with or without the feature and - * the bit is RES0 on CPUs that don't support it. - */ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) { - u64 vtcr = VTCR_EL2_FLAGS, mmfr0; - u32 parange, phys_shift; - u8 lvls; + u64 mmfr0, mmfr1; + u32 phys_shift; if (type & ~KVM_VM_TYPE_ARM_IPA_SIZE_MASK) return -EINVAL; @@ -411,33 +402,8 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) } mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - parange = cpuid_feature_extract_unsigned_field(mmfr0, - ID_AA64MMFR0_PARANGE_SHIFT); - if (parange > ID_AA64MMFR0_PARANGE_MAX) - parange = ID_AA64MMFR0_PARANGE_MAX; - vtcr |= parange << VTCR_EL2_PS_SHIFT; - - vtcr |= VTCR_EL2_T0SZ(phys_shift); - /* - * Use a minimum 2 level page table to prevent splitting - * host PMD huge pages at stage2. - */ - lvls = stage2_pgtable_levels(phys_shift); - if (lvls < 2) - lvls = 2; - vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); - - /* - * Enable the Hardware Access Flag management, unconditionally - * on all CPUs. The features is RES0 on CPUs without the support - * and must be ignored by the CPUs. - */ - vtcr |= VTCR_EL2_HA; + mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1); + kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); - /* Set the vmid bits */ - vtcr |= (kvm_get_vmid_bits() == 16) ? - VTCR_EL2_VS_16BIT : - VTCR_EL2_VS_8BIT; - kvm->arch.vtcr = vtcr; return 0; } From patchwork Tue Nov 17 18:16:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BE25C2D0E4 for ; Tue, 17 Nov 2020 18:26:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A8B6E2467A for ; Tue, 17 Nov 2020 18:26:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="NiAffHPD"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="q1Y7UuuK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A8B6E2467A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=yJVWhYXIHFnhnCaOgawwMs7Dssn6uCF5KlYQBV4jvms=; b=NiAffHPDsjWpqPsc3FJX+os1y 87Eig5FBbohCWRvZl0lkbUZYdDs6dsnFKuGTj1ofmn7Bp/+/eWR18FI2irhI5Gn1tqCoNkfqScHOW z6L5J4HpkyfH8eWTnme3lfYdf6rV6DHnopTbpaNu9JUc1SpNIDUUdLqGIesMNQa/wbwO8vTAUfPTl hB8ci6GF7IFn0m8L3RNjL0dRNxTCt2iaR1FrkuA5Na9raUwVzRCBA6MDfAt8kE3816su/4RrandME UM1xYfof7r3JaMHh4mg/Sk5g291ey4OU8CnlT1vDW70c8QQOQmItO3BqrDlL0VGnIltszF1wRei6n 9tvi8yadA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5fM-0006Cw-FG; Tue, 17 Nov 2020 18:25:20 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XX-0002Tj-8X for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:19 +0000 Received: by mail-qt1-x849.google.com with SMTP id m17so11855503qtu.11 for ; Tue, 17 Nov 2020 10:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ayRqxL4G0sTBj5uvihVXjKdzy9qeF8DDVF1g7AI5ntU=; b=q1Y7UuuK4agsFAxmEhSW2c7W+ugWmHqTFkFHYU9uxsHsHtJseCbRf0SQ7qQmqgRH0S IJ7WOXz4iTftbE6rNj/KFjacgodQVSpkxUfy87L1meiv0RTeOf2yNb41sch1llaLiN8l QJcVAb8dgtRMBXHcZkNOP6wVtXXW2xYp1z/GD3+GA/XjH6FYMBHU3xPpvtCy0VIJ0BtQ sEGxtvzteDzBTDjnjh0PUSkWyBWrP9eqltpt89XpKbvqZMDGBqolGqoEZZNhbUJBkAnZ jNgMKPq3jNgQD8MMYcnDCDKng1+LvtGR4BfGS2if1J7mwNkJdcBsKq8IyV/2THkKBMO8 9dbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ayRqxL4G0sTBj5uvihVXjKdzy9qeF8DDVF1g7AI5ntU=; b=JWPuRkTOFnGgv0vSlizvhPptwFZff9TTAYNc/GIgUeWEnm24EeC9TPoE8MitdLYI/8 vNFBJI+yjh+DND7Kf4tFsWGJjibq+7PTFKkHq3BlCqL1JXDnSrhWvjv6h5ib5AAMCWsF yFh9xWnTys+4KR9Kdq+zxQ8ihtIeNZ/kksOVpT16AYYP6uQQ/hvK0D0YJDkuOTRfiLov yUynj63ZFIF2gPZptahAwkuNhZG58KYUIkXV/liwShiFpJdeXp5iS8CJLbOWPc95lN+D mRnDaD/+NFumBemIa12zflry9VsN4cNqReuRmINTP5Y4F9xxGR1Vh8JjZEemvZkTZt+B tUZw== X-Gm-Message-State: AOAM533CUsDtCZKfkXpq0rcyKDaOnAthsDnv3r20qMAltwV+kgUMN5i6 yuY64sO9SpeM87anErFoIHWVA5nBlEUc X-Google-Smtp-Source: ABdhPJw6vNiywURQRQTe/Th1gnWP0F7Nn6vyTGTOjCETYnBx8vkYHwtVqdL8HyV0Snr2qBKg07hRw7Ypbkm+ X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a05:6214:b83:: with SMTP id fe3mr531341qvb.24.1605637030158; Tue, 17 Nov 2020 10:17:10 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:02 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-23-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 22/27] KVM: arm64: Refactor __load_guest_stage2() From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131716_360136_737AB5F4 X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __load_guest_stage2() to introduce __load_stage2() which will be re-used when loading the host stage 2. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 5a76358e8c7a..96843b7b6eaa 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -321,9 +321,9 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) * Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu, unsigned long vtcr) { - write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); + write_sysreg(vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -334,6 +334,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +{ + __load_stage2(mmu, kern_hyp_va(mmu->arch)->vtcr); +} + static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) { return container_of(mmu->arch, struct kvm, arch); From patchwork Tue Nov 17 18:16:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04037C2D0E4 for ; Tue, 17 Nov 2020 18:27:51 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 753952462E for ; Tue, 17 Nov 2020 18:27:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="NfK8rIyv"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="GrNemWJ5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 753952462E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=DkYRHdYwVElQQ5g1RbQkQ92UuMeCoexd7UXIEgKFaJ0=; b=NfK8rIyv+uwLl8J8OL0IZHJz0 /9pmYmvz53V3KyHUyx7fLeq0KsV7ZOjrCLrMdCJL12q67PYG5a6//ycEcpq+ChTBZo6OQwtirvwrW 464woyDFaQ+ka6gVvUt3bsgPrwzse+Fl0xnXSICSDXpaBYgL1jSPYOJgvwJ7M+xfb4NIwR1miUggl SnGJqKiwE3/vVqTiBXz0qFN328ppE9eI7gfjCsPpE4D/Gru+/R3+Q9RBKGLIH7ZnzcXdct0L9rH44 QnozNjpWIPQR/egplj0wGlqBsf1dF7t0pBhc2wgJqx5ILe+//w3z60swAOPeYQJJtG9IMeAa2k4KI BcXCydSWQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5fs-0006UQ-UW; Tue, 17 Nov 2020 18:25:53 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XZ-0002Ul-QP for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:21 +0000 Received: by mail-wm1-x349.google.com with SMTP id s3so2134897wmj.6 for ; Tue, 17 Nov 2020 10:17:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=e9MvRELompdcQ9+x82O89Ri9iXp7qNvrmx/wGARfLl4=; b=GrNemWJ571s3IP05lzCIf+o9mQWG20WSzj1XWIbqs4BH51p4FibpD+1iIudjxdsldY VTOoE+KKuVVeeDxGnJBQ6qpGvS4olzVpA4h2kHUJdYePK03MTWF1sVNn29wYr8jCqsP1 zfnuBAtCdwfY4dVJuW13O2zHGn/wvP8TJd+7Gv88DeaAnl6Vr0lt8jupKQCVqBMY6LJu UuVkqgukfwDXxkwuFocX4twgxviyx3guAteMin0WGg+8TngA8wcruUWGUCFFyFlGLDAH GgrgZwJ5cpl+PeisJnHcFbNvplQ+JU+cK6fr/J2jY7HohI0LpW8XQQFBmZ0pMJ5mGWt9 sFyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=e9MvRELompdcQ9+x82O89Ri9iXp7qNvrmx/wGARfLl4=; b=SP+HhiTEDh1aUhwanuFa3FuBYynzNEktizmjdHTAmdWqw/5SUuGjlZJUxewLaMIbZI 5xyVN8IpNjMy7V+R4v3VGf3NDX2AkpokQW7xu6XGhxQjHkTKlb4Kt/W1yz8kBar/OXue LEBdLSwkmw/zWAUNUBMCZZaP4y4BA1opEITEAhatMZzo14RA33MSZtIHa/MVQ80Pqx9A nTWpX/L3WgXAcyGb3vckNxzc+X8ZA3bTGlnkjHKw2TLFjUvo+PLp4s1I38SvrSSh04AX JfW9030thyGYeOkGFRHkvCDlinvYaS2OSiqjhRXAQYvcJyUjPueGQCRi9ACqWHhinX/p wYlg== X-Gm-Message-State: AOAM533+HOiiZso24iGZ8Az82SxRoJH1oxz7oJBW9F5RrIwPhn+jLxnT xVwTKzZRT+dEBZ1FeuX+RwdZTKUYqCuB X-Google-Smtp-Source: ABdhPJz7BNpBZ+O3LELUcm62RbTCKz5SEYVRtuO5fjYtpJDAqHE3syxKSKHYJAgBXthIughVPk93YxkQXa72 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:6405:: with SMTP id y5mr376392wmb.150.1605637032528; Tue, 17 Nov 2020 10:17:12 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:03 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-24-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 23/27] KVM: arm64: Refactor __populate_fault_info() From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131718_160830_78130860 X-CRM114-Status: GOOD ( 13.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __populate_fault_info() to introduce __get_fault_info() which will be used once the host is wrapped in a stage 2. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/hyp/switch.h | 36 +++++++++++++++---------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 84473574c2e7..e9005255d639 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -157,19 +157,9 @@ static inline bool __translate_far_to_hpfar(u64 far, u64 *hpfar) return true; } -static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +static inline bool __get_fault_info(u64 esr, u64 *far, u64 *hpfar) { - u8 ec; - u64 esr; - u64 hpfar, far; - - esr = vcpu->arch.fault.esr_el2; - ec = ESR_ELx_EC(esr); - - if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) - return true; - - far = read_sysreg_el2(SYS_FAR); + *far = read_sysreg_el2(SYS_FAR); /* * The HPFAR can be invalid if the stage 2 fault did not @@ -185,12 +175,30 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) if (!(esr & ESR_ELx_S1PTW) && (cpus_have_final_cap(ARM64_WORKAROUND_834220) || (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) { - if (!__translate_far_to_hpfar(far, &hpfar)) + if (!__translate_far_to_hpfar(*far, hpfar)) return false; } else { - hpfar = read_sysreg(hpfar_el2); + *hpfar = read_sysreg(hpfar_el2); } + return true; +} + +static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +{ + u8 ec; + u64 esr; + u64 hpfar, far; + + esr = vcpu->arch.fault.esr_el2; + ec = ESR_ELx_EC(esr); + + if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) + return true; + + if (!__get_fault_info(esr, &far, &hpfar)) + return false; + vcpu->arch.fault.far_el2 = far; vcpu->arch.fault.hpfar_el2 = hpfar; return true; From patchwork Tue Nov 17 18:16:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D5A9C2D0E4 for ; Tue, 17 Nov 2020 18:26:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C63552222E for ; Tue, 17 Nov 2020 18:26:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="pTNVuxGT"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="h47uA7si" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C63552222E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bzyGHJ5daZWWjnRrkYJw7btN1IFqswhuJ9sz6GpAtKo=; b=pTNVuxGTjKrqda6Uoa9mj9fIr 3sOdgnLvFNpJF12Jld4SaUmZ1iHn0RiTWCRjmR5iyvLlAdAbVxme64RhVlepGIl95+Nu0qsKQU1X6 Xlq5gW6dL6lCRBITc9dLhrM19sYyRqTTJxOqxaMFCJM0mKrOVHY6ahU/I+873AEWXkIeBGTqffk3E F7yGYjfCntEFeMAquExz52QilZp1DzXMhPF2h49iqD3BBX+N2+lu1U4jZgHnt0JCWGLWlYqabqX8u csBTZTbr94oFqiubG9WyoXFxTNJ4fczkCxQj1My80MZGDlqhld2MCRzunv01g5FAFLEm6ZxxW5vZM 3Uw6a469A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5g2-0006dN-WB; Tue, 17 Nov 2020 18:26:03 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5XZ-0002Vc-V2 for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:22 +0000 Received: by mail-wm1-x349.google.com with SMTP id y21so56448wma.6 for ; Tue, 17 Nov 2020 10:17:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=M/E0UhVeGc45Y/IlBGkVkubQWCrfXxEDspmvE6rtTmM=; b=h47uA7siZLtvL8FWrvwGPw/6Gi1mVGK4nZPglvcMDSUbFrhK4w0NcSmRrlHWGouKYX DxAk39sKPKJSnzNr/dvUmYhMjJyvl2biCxHaV350s9h9UrFq+4fYHWb9Zd2whPRKZgOU 5ZYwd8MjgnfmNhSh6qqITtKnGClZ4L4DTmZzyrdLS/8HVtxkmrRXVWfBa3+Eczfby2Bm 138d1axuSmMT3FykbncqSS0cBngX86BAorILyga+A5QsleKuGZTBaPcMlAGTPKQgH7Um 4dDBvYa+ckAypcHKMLm2cTKYqLTYhhSEqdiAjwmGjh9IBYT6G3Dqg+VGKhYNSjQt/DSm 1wFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=M/E0UhVeGc45Y/IlBGkVkubQWCrfXxEDspmvE6rtTmM=; b=I4I4Kuwgwao16y0V6net7Igv6YMFgs29g2yOKOV49NzxTBGTS4yjujC3yXtRGTlk8k GPlDdVULah1JHR2Cp5R7mBZbSqEHPIzI7/n+nJ2b+utKlzVnc03b4Jnc46uRA56R9C7r Wh3r7lTR2S8EPww/4TYFfnnSfuhCIShUEE1v9XhbzVUX5Ivzn2usxhhK5gpDCHbFiuca EwpRoyC4mgYtSTUDwvqnUFlCXN5ulBOKeXfc+5L4LIPOU8R7TxQnYv0S6MoYAK3PbX2l XM24X2DpOb8TGIKNEVcFHZ+mRTBud7Ydxb6pmOYeW4UbWVh0bmSRZxg37Q3g9hCaHrzB fCsw== X-Gm-Message-State: AOAM533XtCp5seJNYG6VSbbEIOT/aQIjbw4Bvj9OhfW4J+7xPNPqPSbx RO+OIGIhUDz7KYpsswOEK4hTs5rMjeL9 X-Google-Smtp-Source: ABdhPJwd0iCsqW2dZ9aYbtfJakkoDunmCxcAn2nFAzyEKoo2mrj/F+fEgofRCHuvOqrcwO9QAcEK/FdZYiNr X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:7418:: with SMTP id p24mr392016wmc.36.1605637034987; Tue, 17 Nov 2020 10:17:14 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:04 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-25-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 24/27] KVM: arm64: Make memcache anonymous in pgtable allocator From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131718_198681_7957B370 X-CRM114-Status: GOOD ( 15.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current stage2 page-table allocator uses a memcache to get pre-allocated pages when it needs any. To allow re-using this code at EL2 which uses a concept of memory pools, make the memcache argument to kvm_pgtable_stage2_map() anonymous. and let the mm_ops zalloc_page() callbacks use it the way they need to. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 6 +++--- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 8e8f1d2c5e0e..d846bc3d3b77 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -176,8 +176,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * @size: Size of the mapping. * @phys: Physical address of the memory to map. * @prot: Permissions and attributes for the mapping. - * @mc: Cache of pre-allocated GFP_PGTABLE_USER memory from which to - * allocate page-table pages. + * @mc: Cache of pre-allocated memory from which to allocate page-table + * pages. * * The offset of @addr within a page is ignored, @size is rounded-up to * the next page boundary and @phys is rounded-down to the previous page @@ -194,7 +194,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc); + void *mc); /** * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 96a25d0b7b6e..5dd1b4978fe8 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -443,7 +443,7 @@ struct stage2_map_data { kvm_pte_t *anchor; struct kvm_s2_mmu *mmu; - struct kvm_mmu_memory_cache *memcache; + void *memcache; struct kvm_pgtable_mm_ops *mm_ops; }; @@ -613,7 +613,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc) + void *mc) { int ret; struct stage2_map_data map_data = { From patchwork Tue Nov 17 18:16:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E8CFC5519F for ; Tue, 17 Nov 2020 18:28:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7566C24686 for ; Tue, 17 Nov 2020 18:28:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="KES/TQUB"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="OEoAZqf/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7566C24686 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pm4KkySdXHRQWn3IAIdcJBG7ocb0loSY/Z9zWaPL2+k=; b=KES/TQUBvfdxfMbv/bUzwyBrg YUExa2yi9pfa6uwd57OXpP6A2gIR4gtIL99DHVFs4vpHmsrg4wk5slaqIlPxahjcS55dq81152hY2 6bRAMJf207Y6jhQHro51Asv9B6t8Ui4ZChyZI8aXpe8PK8rCnYJOr/aWWrT/ZCkjBwxV/TWfT81Rx 3qjgQ1S6Qu2FhHtroNOTgp2DfEQzYAnibtG+LRXZlha9hPKbQd3OlKoPw3wewRFMpgvTKKfOWSY39 skf8yV0GBIxGK6otEOwEn2apnoCsDynWR969MafliUKdO3McUVa2hh1OkbvGIAk8PM1pSc13cHFEU pWjgTR7MQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5gM-0006mo-11; Tue, 17 Nov 2020 18:26:22 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Xe-0002Wo-4V for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:25 +0000 Received: by mail-wm1-x349.google.com with SMTP id u123so1904252wmu.5 for ; Tue, 17 Nov 2020 10:17:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=U9buW3EkB/K6NinBuCPzqj4JKuwVqT/SXDU0SZ8bieQ=; b=OEoAZqf/nKz8y4wwbNx4ce9JZPHSdATgOxmylZEZ63UUl4SLy8rF5lv+5jh8HRhWHV l3jGcsZVeScg2fnHeX8O1ZoHr2neKk6P/nYD6Urly0tw8n40b2B5togib9oJ6S8u3J2G DFJzS6t4+o1+m2Wrt+X107G2mNv5796ZBQ3LThG4GqG3MHwAyh/szE5NsseL5rxg6NbE jradhxYJAJ+gyWiQzQs70x7NpeqFztQHGIra3pXzLaVk+w+fYaQrxiIKC6MLVCNPhS1a WAn/iamjzPdbSB5+T9yPzgX/YFXLTSRG0UxqXOOA97dB1iR3N49f6Pd3LVxRUZ/1Iq49 GbaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=U9buW3EkB/K6NinBuCPzqj4JKuwVqT/SXDU0SZ8bieQ=; b=m2FMg1wzty35DwzryDZKyC3hvO1QAZQJhZdgHIk8Ndexs6BYFiRjMr5lWRvgWYVjU7 IDYUJhxIPqVb/xAsDaean2JodF8MOx9+aY3nDXPCO5c51NPHFawknYsRzaxCU7PypTqr 9KyBxjdEyuRJhQvjSeqvxwRTwpeJ51xRAzVTB6d+sWI0xB0XodH9BBxmhC0Cl0GnI/0g g0jv6UfEeANHSp7cGPULY28A4dSL0KWW+m70HRplffBaMQx8syXTGDOF4ObszKWf4qph mEkbV0mTY5IbnetsqfbqlGd8R48UYAWcmW43w8ebbfmkd1j68Ar41phGWdP7USCzSsqd q/zA== X-Gm-Message-State: AOAM533bxSKSh5QoxyjLjVmlAeIZT92BSreFhOL+NFcVZHvLTsPa2zwF Wa78uoW9zyMF1cxkOqFl/VNf7rMofn2l X-Google-Smtp-Source: ABdhPJxdY+lqNlpPzdnmipW4jOsekrda/kjR1w92gZ+51XZgmNLFCJj9D0pJqUaOPEVX/Dsw6fZ/dN80t1P3 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:adf:a3ca:: with SMTP id m10mr842044wrb.228.1605637037211; Tue, 17 Nov 2020 10:17:17 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:05 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-26-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 25/27] KVM: arm64: Reserve memory for host stage 2 From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131722_289733_B759F227 X-CRM114-Status: GOOD ( 17.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extend the memory pool allocated for the hypervisor to include enough pages to map all of memory at page granularity for the host stage 2. While at it, also reserve some memory for device mappings. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/mm.h | 36 ++++++++++++++++++++++++---- arch/arm64/kvm/hyp/nvhe/setup.c | 12 ++++++++++ arch/arm64/kvm/hyp/reserved_mem.c | 2 ++ 3 files changed, 46 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index 5a3ad6f4e5bc..b79be2580164 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -52,15 +52,12 @@ static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) return total; } -static inline unsigned long hyp_s1_pgtable_size(void) +static inline unsigned long __hyp_pgtable_total_size(void) { struct hyp_memblock_region *reg; unsigned long nr_pages, res = 0; int i; - if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) - return 0; - for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) { reg = &kvm_nvhe_sym(hyp_memory)[i]; nr_pages = (reg->end - reg->start) >> PAGE_SHIFT; @@ -68,6 +65,18 @@ static inline unsigned long hyp_s1_pgtable_size(void) res += nr_pages << PAGE_SHIFT; } + return res; +} + +static inline unsigned long hyp_s1_pgtable_size(void) +{ + unsigned long res, nr_pages; + + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) + return 0; + + res = __hyp_pgtable_total_size(); + /* Allow 1 GiB for private mappings */ nr_pages = (1 << 30) >> PAGE_SHIFT; nr_pages = __hyp_pgtable_max_pages(nr_pages); @@ -76,4 +85,23 @@ static inline unsigned long hyp_s1_pgtable_size(void) return res; } +static inline unsigned long host_s2_mem_pgtable_size(void) +{ + unsigned long max_pgd_sz = 16 << PAGE_SHIFT; + + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) + return 0; + + return __hyp_pgtable_total_size() + max_pgd_sz; +} + +static inline unsigned long host_s2_dev_pgtable_size(void) +{ + if (kvm_nvhe_sym(hyp_memblock_nr) <= 0) + return 0; + + /* Allow 1 GiB for private mappings */ + return __hyp_pgtable_max_pages((1 << 30) >> PAGE_SHIFT) << PAGE_SHIFT; +} + #endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 9679c97b875b..b73e6b08cfba 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -24,6 +24,8 @@ unsigned long hyp_nr_cpus; static void *stacks_base; static void *vmemmap_base; static void *hyp_pgt_base; +static void *host_s2_mem_pgt_base; +static void *host_s2_dev_pgt_base; static int divide_memory_pool(void *virt, unsigned long size) { @@ -46,6 +48,16 @@ static int divide_memory_pool(void *virt, unsigned long size) if (!hyp_pgt_base) return -ENOMEM; + nr_pages = host_s2_mem_pgtable_size() >> PAGE_SHIFT; + host_s2_mem_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_mem_pgt_base) + return -ENOMEM; + + nr_pages = host_s2_dev_pgtable_size() >> PAGE_SHIFT; + host_s2_dev_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_dev_pgt_base) + return -ENOMEM; + return 0; } diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index 02b0b18006f5..c2c0484b6211 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -47,6 +47,8 @@ void __init reserve_kvm_hyp(void) hyp_mem_size += num_possible_cpus() << PAGE_SHIFT; hyp_mem_size += hyp_s1_pgtable_size(); + hyp_mem_size += host_s2_mem_pgtable_size(); + hyp_mem_size += host_s2_dev_pgtable_size(); /* * The hyp_vmemmap needs to be backed by pages, but these pages From patchwork Tue Nov 17 18:16:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9423C2D0E4 for ; Tue, 17 Nov 2020 18:27:12 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5BEF12222E for ; Tue, 17 Nov 2020 18:27:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ho3sr3nl"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="BwMpHLzz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5BEF12222E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=M6ooSv58iF+MEM1qzPvGXUtniQNswZbjIQsW+kPqJR0=; b=ho3sr3nlrGLYryvrPvNiLRg1v Q2Fl3y326GWfDmTNnWPjCoL3KnJKZBk7hqMjNj8uIQ2R2aog6KWKS820M+6u66M07Rhr9mW5/ynQ6 gLWkXbc9LHvuTwbxZYhxVnu+ocpxnY8qLGoUIo1yi1F6bDiuvEEx+z0dLG5zP5Ajo+rPsHOJoLafk 139WvOfFf5uvHFNgLSj0OzDU701pP7zi2ZlwqYFd7wpmdgBbzhKlSApNiioN2wXuqMScEfEf9VGpD uKRX5+6uvry9htsOCIIvF5lR/2XR7tQY+fFzNETzdc0LoHPUd+tQQS0NyW7KGR+TiFsD0S1L+I1om OSW4dv7lA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5gU-0006qH-7B; Tue, 17 Nov 2020 18:26:30 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Xf-0002Xr-RU for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:26 +0000 Received: by mail-qk1-x74a.google.com with SMTP id t64so14724541qkd.5 for ; Tue, 17 Nov 2020 10:17:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=E22eq9BAxEe+AO1nuitRnkTYq4b0+jwZ2QqbOY4WjS4=; b=BwMpHLzzaf2zH+ej4IZ+ZkIk/2jWr7L+FHUmN9hGsyCqOGOmdabExESKfOLgOuQEMx NHx/9YL1ICL/V6NTINPSEuafCnh+jEpymTyAt+ffdheOuo7jOJ8vS/yeL41NvksNjLOE x8t+7P7Bm81+jxi2AQWLRtbYG98F660Av+2v6E1SVR18iUDIbjud6M7WtnnUxBYtf4yg aCkEG1Nbb7InqizJzHfB0W0Noq7MOQtxHUbyYkRV8CEDKX/4SUjh+2KgSkkYcs5KTcOv 9gcPbPB/EtvtRoo0fWhkrwKl79dMwsmHYYo8KVVxYYexbKJ9YZA/vmIK3+v8nlSTkUDs TWaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=E22eq9BAxEe+AO1nuitRnkTYq4b0+jwZ2QqbOY4WjS4=; b=bYfNNao4VdWoYaH16Ao0qKi7Et+ot1k3FvOZl8As24x63fPKk95L1Is9Jug0mIegVv M/qlVZThATK8Ltak5+Ilbp62lMQjGRuYx1my4moBlF41jNTPpaQO8ztTu+E4cczi4xvi V1wtI2Sh50naxxlgu+g++tpGKlOuAaTrk4oDAW8JbYbGTwRzANyCEKnKAH/cfRs6+4Zv Wi6zAOepIGfrHiTf5MeWqAdjllqycqG+erWkL9E+Cw1SkySS8vc/Lv/rQwV1xzSbr73i aTR45TLncRj/vOTXiYPLibMpXQrLM5g5ENgmDqEmKpS0oWSlX8CaaYXZEJd2XutyRf6U fk5w== X-Gm-Message-State: AOAM533CZGxS1uTimTW/D7SHAoXoosuYa4Z/XSXggMJS20NWR25aVB1k w4m96J2ZdYQ3io3ftV3N0ZreNdMPE1nJ X-Google-Smtp-Source: ABdhPJz/mVc1v+qYJqOE8yeahxM0e0LR02PFTS4Ofl1niXl14opy+AyQuh/+jpD0RcsQFfuxJf4epmdjTYs2 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:ad4:53c8:: with SMTP id k8mr531726qvv.40.1605637039404; Tue, 17 Nov 2020 10:17:19 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:06 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-27-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 26/27] KVM: arm64: Sort the memblock regions list From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131723_962405_81549B63 X-CRM114-Status: GOOD ( 14.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hypervisor will need the list of memblock regions sorted by increasing start address to make look-ups more efficient. Make the host do the hard work early while it is still trusted to avoid the need for a sorting library at EL2. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/hyp/reserved_mem.c | 18 ++++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 53b01d25e7d9..ec304a5c728b 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -746,6 +746,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); extern phys_addr_t hyp_mem_base; extern phys_addr_t hyp_mem_size; void __init reserve_kvm_hyp(void); +void kvm_sort_memblock_regions(void); #else static inline void reserve_kvm_hyp(void) { } #endif diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index e06c95a10dba..8160a0d12a58 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1685,6 +1685,7 @@ static int kvm_hyp_enable_protection(void) return ret; kvm_set_hyp_vector(); + kvm_sort_memblock_regions(); ret = kvm_call_hyp_nvhe(__kvm_hyp_protect, hyp_mem_base, hyp_mem_size, num_possible_cpus(), kern_hyp_va(per_cpu_base)); if (ret) diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index c2c0484b6211..7da8e2915c1c 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -6,6 +6,7 @@ #include #include +#include #include @@ -31,6 +32,23 @@ void __init early_init_dt_add_memory_hyp(u64 base, u64 size) kvm_nvhe_sym(hyp_memblock_nr)++; } +static int cmp_hyp_memblock(const void *p1, const void *p2) +{ + const struct hyp_memblock_region *r1 = p1; + const struct hyp_memblock_region *r2 = p2; + + return r1->start < r2->start ? -1 : (r1->start > r2->start); +} + +void kvm_sort_memblock_regions(void) +{ + sort(kvm_nvhe_sym(hyp_memory), + kvm_nvhe_sym(hyp_memblock_nr), + sizeof(struct hyp_memblock_region), + cmp_hyp_memblock, + NULL); +} + extern bool enable_protected_kvm; void __init reserve_kvm_hyp(void) { From patchwork Tue Nov 17 18:16:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 11913231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22A3EC2D0E4 for ; Tue, 17 Nov 2020 18:28:44 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D2062222E for ; Tue, 17 Nov 2020 18:28:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="dasUCaxH"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="Y46xsWr+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D2062222E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=plfCq/Kbz7X+5aEThplO6ZE+FSo0WdgaBgsevsxRQ2Y=; b=dasUCaxH0UQucxzbia3GRXRMw JioEX3KLF6U0uFtWzgIqTNCQ6Gan63bp/VB4KPOZj8WqmuFccKR4q75kspksVWZLUfx/5sgniFRBN iHm72QePbSdvESHQnxmg2fkzvYM5oSxuywLh01fm+nWIdjPmn0Ho4PVOuXSbSjUO4qVhcWWs53m1x lXmviYWCMNw/o5JLQj1jl0YxiWUD8Wi8J1pLbPNO9ZYslitH9aCLuD9OylU7OKV3yy6hiq3F8oqS1 0M5yxCtNQldZqF4Iw3BjyDRy4S/RpIo8aCRFmOrOhdiv0hwE7bgosqQMCL3YyeRpgKF8MpsVhIjQo 548X3z7+Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5hp-0007RD-9G; Tue, 17 Nov 2020 18:27:53 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kf5Xi-0002Yb-TR for linux-arm-kernel@lists.infradead.org; Tue, 17 Nov 2020 18:17:29 +0000 Received: by mail-wr1-x449.google.com with SMTP id x16so8648794wrn.9 for ; Tue, 17 Nov 2020 10:17:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=UnISW/7ysLvEOA4M0KYS2DiSYKHuk/dOZvBw6jTbu2M=; b=Y46xsWr+fpscULTkv6wJmNLh4Rny8VDfXImWw0g4FGuDGcH5Dj65BmOnRZesT6ZFth nzmvzgrCAMT35AMiTIAi2DFYOrh/Aq6nY55zm/m6DPUxjPGsHta+ja7jqxqUfCU4FnQb MHtfB5aoKisNqH6shJKNLDU/NzfYKajoHo+mIn3KhtSthuEGflYJZp2bEpYphw8Ro2WW /CkJ+W+rUFgkwTPjM8pjl1ZQjOacdVnCazJPTUwiatafcU5KRWPDo1VSlIpvGL2urLm2 2XqyygWV30YhIq8Z5vT4EcO3G69skl7KBQjf4G1HIF/tPT4vHuT52snzhwlVedT2HSLA 53Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UnISW/7ysLvEOA4M0KYS2DiSYKHuk/dOZvBw6jTbu2M=; b=bPbTgJVcxBUV5XM5h70bkwzVnS2lNQ4fhyy871viSx3tUuGy8UP9BCtJ5O17tsswhm /+a7nSPBNmBjAVyJxl8aswJr+vtj07pMXu8C2qa5Sa1kLc/5l572ZZmt+1DFilGe8dIG gp/p798kuHDPg/Cyk7D8Neo+bywrS36nkuGiL90W5h8G88w0rBn/sGO7xHVFxveh0vAt uMQWtxJj6nsRTPVqugOxG7PIv/oGv3Vv6hld+WfRvSeVTQHzkQ8u6nZjgMLS43kS6F8V 7cpf6RXs2ac9a9OqabXcb8VxMhi0+HK1ccMiWduwvuh9w70jLJDcQpBWaE8Orz9f2qrX yAJQ== X-Gm-Message-State: AOAM531ztVNPRUo02L2PKcZRg3O0+zjK/moPZqTpI7hhFX/tYG5nC1Al q3pPU+LYFBuQGcxfUEtHEoYSC+KXzFVU X-Google-Smtp-Source: ABdhPJxB+DSZR+N3pyG/NRrgmrD0zn/VZQ6upGsPUcqxEFyzBuPz5gz5FQIvepAWoMo779y7fqqxMM7WrNUQ X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:f693:9fff:fef4:a7ef]) (user=qperret job=sendgmr) by 2002:a1c:9ad3:: with SMTP id c202mr346589wme.43.1605637041767; Tue, 17 Nov 2020 10:17:21 -0800 (PST) Date: Tue, 17 Nov 2020 18:16:07 +0000 In-Reply-To: <20201117181607.1761516-1-qperret@google.com> Message-Id: <20201117181607.1761516-28-qperret@google.com> Mime-Version: 1.0 References: <20201117181607.1761516-1-qperret@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH 27/27] KVM: arm64: Wrap the host with a stage 2 From: Quentin Perret To: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Rob Herring , Frank Rowand X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201117_131727_205292_BB303189 X-CRM114-Status: GOOD ( 29.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE" , Quentin Perret , android-kvm@google.com, open list , kernel-team@android.com, "open list:KERNEL VIRTUAL MACHINE FOR ARM64 \(KVM/arm64\)" , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM runs in protected nVHE mode, make use of a stage 2 page-table to give the hypervisor some control over the host memory accesses. At the moment all memory aborts from the host will be instantly idmapped RWX at stage 2 in a lazy fashion. Later patches will make use of that infrastructure to implement access control restrictions to e.g. protect guest memory from the host. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_cpufeature.h | 2 + arch/arm64/kernel/image-vars.h | 3 + arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 33 +++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 191 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 6 + arch/arm64/kvm/hyp/nvhe/switch.c | 7 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 4 +- 9 files changed, 247 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mem_protect.c diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h index d34f85cba358..74043a149322 100644 --- a/arch/arm64/include/asm/kvm_cpufeature.h +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -15,3 +15,5 @@ #endif KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR0_EL1, arm64_ftr_reg_id_aa64mmfr0_el1) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR1_EL1, arm64_ftr_reg_id_aa64mmfr1_el1) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index f2d43e6cd86d..e2652278dd63 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -137,6 +137,9 @@ KVM_NVHE_ALIAS(__hyp_data_ro_after_init_end); KVM_NVHE_ALIAS(__hyp_bss_start); KVM_NVHE_ALIAS(__hyp_bss_end); +/* pKVM static key */ +KVM_NVHE_ALIAS(kvm_protected_mode_initialized); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h new file mode 100644 index 000000000000..a22ef118a610 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#ifndef __KVM_NVHE_MEM_PROTECT__ +#define __KVM_NVHE_MEM_PROTECT__ +#include +#include +#include +#include +#include + +struct host_kvm { + struct kvm_arch arch; + struct kvm_pgtable pgt; + struct kvm_pgtable_mm_ops mm_ops; + hyp_spinlock_t lock; +}; +extern struct host_kvm host_kvm; + +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool); +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); + +static __always_inline void __load_host_stage2(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + else + write_sysreg(0, vttbr_el2); +} +#endif /* __KVM_NVHE_MEM_PROTECT__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index d7381a503182..c3e2f98555c4 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -11,7 +11,7 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o setup.o mm.o + cache.o cpufeature.o setup.o mm.o mem_protect.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index a0bfe0d26da6..5d0cb17e03a1 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -13,6 +13,7 @@ #include #include +#include #include DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); @@ -234,6 +235,11 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; + case ESR_ELx_EC_IABT_LOW: + fallthrough; + case ESR_ELx_EC_DABT_LOW: + handle_host_mem_abort(host_ctxt); + break; default: hyp_panic(); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c new file mode 100644 index 000000000000..0cd3eb178f3b --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -0,0 +1,191 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include + +extern unsigned long hyp_nr_cpus; +struct host_kvm host_kvm; + +struct hyp_pool host_s2_mem; +struct hyp_pool host_s2_dev; + +static void *host_s2_zalloc_pages_exact(size_t size) +{ + return hyp_alloc_pages(&host_s2_mem, HYP_GFP_ZERO, get_order(size)); +} + +static void *host_s2_zalloc_page(void *pool) +{ + return hyp_alloc_pages(pool, HYP_GFP_ZERO, 0); +} + +static int prepare_s2_pools(void *mem_pgt_pool, void *dev_pgt_pool) +{ + unsigned long nr_pages; + int ret; + + nr_pages = host_s2_mem_pgtable_size() >> PAGE_SHIFT; + ret = hyp_pool_init(&host_s2_mem, __hyp_pa(mem_pgt_pool), nr_pages, 0); + if (ret) + return ret; + + nr_pages = host_s2_dev_pgtable_size() >> PAGE_SHIFT; + ret = hyp_pool_init(&host_s2_dev, __hyp_pa(dev_pgt_pool), nr_pages, 0); + if (ret) + return ret; + + host_kvm.mm_ops.zalloc_pages_exact = host_s2_zalloc_pages_exact; + host_kvm.mm_ops.zalloc_page = host_s2_zalloc_page; + host_kvm.mm_ops.phys_to_virt = hyp_phys_to_virt; + host_kvm.mm_ops.virt_to_phys = hyp_virt_to_phys; + host_kvm.mm_ops.page_count = hyp_page_count; + host_kvm.mm_ops.get_page = hyp_get_page; + host_kvm.mm_ops.put_page = hyp_put_page; + + return 0; +} + +static void prepare_host_vtcr(void) +{ + u32 parange, phys_shift; + u64 mmfr0, mmfr1; + + mmfr0 = arm64_ftr_reg_id_aa64mmfr0_el1.sys_val; + mmfr1 = arm64_ftr_reg_id_aa64mmfr1_el1.sys_val; + + /* The host stage 2 is id-mapped, so use parange for T0SZ */ + parange = kvm_get_parange(mmfr0); + phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange); + + host_kvm.arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); +} + +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool) +{ + struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; + struct kvm_nvhe_init_params *params; + int ret, i; + + prepare_host_vtcr(); + hyp_spin_lock_init(&host_kvm.lock); + + ret = prepare_s2_pools(mem_pgt_pool, dev_pgt_pool); + if (ret) + return ret; + + ret = kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch, + &host_kvm.mm_ops); + if (ret) + return ret; + + mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd); + mmu->arch = &host_kvm.arch; + mmu->pgt = &host_kvm.pgt; + mmu->vmid.vmid_gen = 0; + mmu->vmid.vmid = 0; + + for (i = 0; i < hyp_nr_cpus; i++) { + params = per_cpu_ptr(&kvm_init_params, i); + params->vttbr = kvm_get_vttbr(mmu); + params->vtcr = host_kvm.arch.vtcr; + params->hcr_el2 |= HCR_VM; + __flush_dcache_area(params, sizeof(*params)); + } + + write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + + return 0; +} + +static void host_stage2_unmap_dev_all(void) +{ + struct kvm_pgtable *pgt = &host_kvm.pgt; + struct hyp_memblock_region *reg; + u64 addr = 0; + int i; + + /* Unmap all non-memory regions to recycle the pages */ + for (i = 0; i < hyp_memblock_nr; i++, addr = reg->end) { + reg = &hyp_memory[i]; + kvm_pgtable_stage2_unmap(pgt, addr, reg->start - addr); + } + kvm_pgtable_stage2_unmap(pgt, addr, ULONG_MAX); +} + +static bool ipa_is_memory(u64 ipa) +{ + int cur, left = 0, right = hyp_memblock_nr; + struct hyp_memblock_region *reg; + + /* The list of memblock regions is sorted, binary search it */ + while (left < right) { + cur = (left + right) >> 1; + reg = &hyp_memory[cur]; + if (ipa < reg->start) + right = cur; + else if (ipa >= reg->end) + left = cur + 1; + else + return true; + } + + return false; +} + +static int __host_stage2_map(u64 ipa, u64 size, enum kvm_pgtable_prot prot, + struct hyp_pool *p) +{ + return kvm_pgtable_stage2_map(&host_kvm.pgt, ipa, size, ipa, prot, p); +} + +static int host_stage2_map(u64 ipa, u64 size, enum kvm_pgtable_prot prot) +{ + int ret, is_memory = ipa_is_memory(ipa); + struct hyp_pool *pool; + + pool = is_memory ? &host_s2_mem : &host_s2_dev; + + hyp_spin_lock(&host_kvm.lock); + ret = __host_stage2_map(ipa, size, prot, pool); + if (ret == -ENOMEM && !is_memory) { + host_stage2_unmap_dev_all(); + ret = __host_stage2_map(ipa, size, prot, pool); + } + hyp_spin_unlock(&host_kvm.lock); + + return ret; +} + +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) +{ + enum kvm_pgtable_prot prot; + u64 far, hpfar, esr, ipa; + int ret; + + esr = read_sysreg_el2(SYS_ESR); + if (!__get_fault_info(esr, &far, &hpfar)) + hyp_panic(); + + prot = KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W | KVM_PGTABLE_PROT_X; + ipa = (hpfar & HPFAR_MASK) << 8; + ret = host_stage2_map(ipa, PAGE_SIZE, prot); + if (ret) + hyp_panic(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index b73e6b08cfba..3e079027e608 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -12,6 +12,7 @@ #include #include #include +#include #include struct hyp_pool hpool; @@ -161,6 +162,11 @@ void __noreturn __kvm_hyp_protect_finalise(void) if (ret) goto out; + /* Wrap the host with a stage 2 */ + ret = kvm_host_prepare_stage2(host_s2_mem_pgt_base, host_s2_dev_pgt_base); + if (ret) + goto out; + hyp_pgtable_mm_ops.zalloc_page = hyp_zalloc_hyp_page; hyp_pgtable_mm_ops.phys_to_virt = hyp_phys_to_virt; hyp_pgtable_mm_ops.virt_to_phys = hyp_virt_to_phys; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 979a76cdf9fb..31bc1a843bf8 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -28,6 +28,8 @@ #include #include +#include + /* Non-VHE specific context */ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); @@ -102,11 +104,6 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) write_sysreg(__kvm_hyp_host_vector, vbar_el2); } -static void __load_host_stage2(void) -{ - write_sysreg(0, vttbr_el2); -} - /* Save VGICv3 state on non-VHE systems */ static void __hyp_vgic_save_state(struct kvm_vcpu *vcpu) { diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index fbde89a2c6e8..255a23a1b2db 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -8,6 +8,8 @@ #include #include +#include + struct tlb_inv_context { u64 tcr; }; @@ -43,7 +45,7 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu, static void __tlb_switch_to_host(struct tlb_inv_context *cxt) { - write_sysreg(0, vttbr_el2); + __load_host_stage2(); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { /* Ensure write of the host VMID */