From patchwork Wed Mar 10 17:57:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1D45C433DB for ; Wed, 10 Mar 2021 17:59:50 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4BEB264F3A for ; Wed, 10 Mar 2021 17:59:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4BEB264F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JPm8pIXnsy5+cVHVepX3e8Hg8gpdmUi4UkSHupwVo+Q=; b=DlFpSnbDNXLrdu TXNxVI6ROinWU27zNoGM4v8RnqBUvj896XP6PBpajHhEz1A6WtX3etzqAkYx/08i7FG4rynthMwmB 21e8qaCCALYTtU5yTCdC6J/jbrEiu51iUVThvT632s8onSwp+1EPmh0tTpPB06STjLWZk8eafDo5i SE+fO+SWaEYK277th+wAsVRtLs4N8pdTmCcXTOOW/h6FRjsjWyciUSYlPUrsQSJocFYkqOqrxNBco qz4WVKAWMLXsfa6RUQ5rVMs3j+VokvONyRkZA6HnoduJ2jllZy1Z5BSXEotwXn4RElJmgC431xBDd VgLqGWs5Z2Q/TKnI+ZOg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK362-007OAj-Of; Wed, 10 Mar 2021 17:58:10 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK35r-007O7C-KU for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:01 +0000 Received: by mail-wr1-x44a.google.com with SMTP id g5so8272740wrd.22 for ; Wed, 10 Mar 2021 09:57:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=AbQOhTo5qTBDcGKT4ROquEtU01mLpiQKpICJ+GOnoG0=; b=JzpUXZ4hpdoBvUyEsWDm2q1s1QRxAl2wx4lPyxgUQpUIJ7F0aVV92HnT2t7d2+gB78 bUz/2TCUabh7EGBtNVRZShXzUflsBTbBP8+TtuPJERTGtDDbxvky8XAhZfj+cyxBTIOr VWwXSeTf22kfUPlEI+Gwhr+nqoB0tAKIffsGWkZdhNwdwyheRpKO2ypsVssGnl+iZBcG DoFVBZxFu50SG2HHLuRMrZtkXo9vsu4OjCirEILdzOmzM58CyNEo/DRyzmIvNtEoaf8R gisXQSUy9RC42Ahgk3HJDSsgTcaFEJri7l2kjGdD/YjK4DU4vgnl3B/RH4W0CV1cQkKX HAAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AbQOhTo5qTBDcGKT4ROquEtU01mLpiQKpICJ+GOnoG0=; b=Qd7loxDOpe5NNn2AJflqeoay3H+weG9RM6y+Pd1upu09PT2oSfjKso9+G049js5/WQ /qi9LZdPN/nVf7sILRrMXkbYGwNqZ0UByx1EXEv+PmesXzdt+iueCnbrGMqLbkqAt3tu A1XyniQV1iyzNrQ4sr490Vc4mP8LAIOh16ZZpHoQsZxdMr6ced9wdde3hT4AsSv96fsV V2HKJWnW6IJUGuyyWQdTrvUnAJmz2vKnKZM0L5BzTbnt41D71RK5LWO1frX29eFUpU/v z5Kefsalr6pbmMugMp00yYZ1kAj2RNIR32yCwzBHxXwXAaLsN9rMssd9qTeo97ZgF+70 GGSQ== X-Gm-Message-State: AOAM530O4S4ZodGjtldsGUnFTWKZnGDhc07taVgv7/0yL+axI1PibgSx OjPgBt4W78COiTTC3RSvMC0UjecB2n44 X-Google-Smtp-Source: ABdhPJyoHV7k5VKRC4cMLOuiadpNjmOubi8A8GxCIf77avXZ6hUq1pIFK8fqsdVSlzJaLhL/5cbha4JM2YHV X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:7210:: with SMTP id n16mr4388170wmc.13.1615399076669; Wed, 10 Mar 2021 09:57:56 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:18 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-2-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 01/34] arm64: lib: Annotate {clear, copy}_page() as position-independent From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175759_932979_5317C73B X-CRM114-Status: GOOD ( 10.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon clear_page() and copy_page() are suitable for use outside of the kernel address space, so annotate them as position-independent code. Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/lib/clear_page.S | 4 ++-- arch/arm64/lib/copy_page.S | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S index 073acbf02a7c..b84b179edba3 100644 --- a/arch/arm64/lib/clear_page.S +++ b/arch/arm64/lib/clear_page.S @@ -14,7 +14,7 @@ * Parameters: * x0 - dest */ -SYM_FUNC_START(clear_page) +SYM_FUNC_START_PI(clear_page) mrs x1, dczid_el0 and w1, w1, #0xf mov x2, #4 @@ -25,5 +25,5 @@ SYM_FUNC_START(clear_page) tst x0, #(PAGE_SIZE - 1) b.ne 1b ret -SYM_FUNC_END(clear_page) +SYM_FUNC_END_PI(clear_page) EXPORT_SYMBOL(clear_page) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index e7a793961408..29144f4cd449 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -17,7 +17,7 @@ * x0 - dest * x1 - src */ -SYM_FUNC_START(copy_page) +SYM_FUNC_START_PI(copy_page) alternative_if ARM64_HAS_NO_HW_PREFETCH // Prefetch three cache lines ahead. prfm pldl1strm, [x1, #128] @@ -75,5 +75,5 @@ alternative_else_nop_endif stnp x16, x17, [x0, #112 - 256] ret -SYM_FUNC_END(copy_page) +SYM_FUNC_END_PI(copy_page) EXPORT_SYMBOL(copy_page) From patchwork Wed Mar 10 17:57:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09316C433E0 for ; Wed, 10 Mar 2021 17:59:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 73C5A64FB2 for ; Wed, 10 Mar 2021 17:59:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73C5A64FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UkT4Yq8D/buAXtdCDm3r00hg6YFL05+md6fsA3v/fOU=; b=MizVz4CDm+Yp1b BGnjM9AcwSWxaMJGOf/COR46QG/bAFdjOF92XMDKVuXJZpTZNtxaTyliQhQxD0N6cMTA44BOiYIA5 4cLyZ/HRilgYS++RezdYWJ7q5TtCGenj/J+8SPexakiHitqlQfzjBB4JZA6Lb3UQ/+W+218sORbMa O++Cq4leegbJnoKlHrUEEOWJHeWMQyqOyURC3tfTS55TipnbGNmGrQNr1D9IyDYnynt3dDkjrdsS/ TWb1F/h42LVsIfKZshfTLRtrmyUWUqAJGyOVFKeh2KVcMRPOIDlShicIueD+JQ7ZASLkryiEsZ7pX oHI5Dje6H+hmXbiJ2Deg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK369-007OCr-94; Wed, 10 Mar 2021 17:58:17 +0000 Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK35s-007O7v-5l for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:01 +0000 Received: by mail-qk1-x749.google.com with SMTP id 130so13405292qkm.0 for ; Wed, 10 Mar 2021 09:57:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dxCzN3RMauKWh2fvqFWMKg2laCezvVx9+eyFyeOVXIM=; b=Pb/keeFM6FLTObUZ0vCj8r+u4fGlhfurmJ1BDbmQmSEHz4lE1RmPC9dWhtSUH4ursS dMPYP6E31+WNt7F1w+2XsVL1GTNJkRRfUu7PxKUuIp2JvUceMsSdTUIGueMRxU5SA+K0 53rJE19I5YooOpMQfpqkoOid8LwrEUhstQoTEknAdifmfG417EI+b1y3EhsQfA+WLskR LFQdY140E4mgZqa+6Tuxpk+zuxstuuf4LH97W+qopGdatwH8lkKy3DsR1kN7gDWuldJt 3YAx44ZgdVrQNiMxH0VwJPb3tEtfDnjjZ64pJIWelpAwol/CTOYdpCy8JiYYnvQDewv+ Xliw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dxCzN3RMauKWh2fvqFWMKg2laCezvVx9+eyFyeOVXIM=; b=bNLrgusvurCxSLo2tNyDBfCNuSWALAYhFWCUaSw/zqe40DZKS1HWP24T0YdSROCFcl vc34AcRmS9s1YGcVSzwoBLFUH7mxYLqqzk75DpLoymGmpf3R9I4+h65A6JvxlldsiOs7 l8UdHALry/AIXxS0uZGLC723RhjMCkhkDAbdkE+EWFExCHQLSua3n16OlX3NaEgIrjoj bLQo/ApqM6eKGeSNVmtgy/BAK0Yr3sJvTBUHPhFdBbBZkTYpD7y4reNcpvmCwVXjsaip tJIBcI4gvJGTW2dFt5nWdfBYO+aFpERT3pKHhyvkBbS3NRvtERNmFFv+LyRRmbk5WkzJ VIww== X-Gm-Message-State: AOAM532Uy73hCmcUV6rhg6AztItoTmdElANK6KA+RfF6UUKvGgSSIPy3 U73yefpjt+cqAVMRjauTSouyhPOPtrz/ X-Google-Smtp-Source: ABdhPJzFbcnSuRhMeKDYXy6/C1z2diqQrt+imwRgbptD7Lg8gH79A2Af7xKDqf7ZkVXomhp6ER56ZqVFJa7D X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:ad4:4eae:: with SMTP id ed14mr4180391qvb.52.1615399078778; Wed, 10 Mar 2021 09:57:58 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:19 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-3-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 02/34] KVM: arm64: Link position-independent string routines into .hyp.text From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175800_341089_FB3C7BF4 X-CRM114-Status: GOOD ( 11.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon Pull clear_page(), copy_page(), memcpy() and memset() into the nVHE hyp code and ensure that we always execute the '__pi_' entry point on the offchance that it changes in future. [ qperret: Commit title nits and added linker script alias ] Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/hyp_image.h | 3 +++ arch/arm64/kernel/image-vars.h | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 ++++ 3 files changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h index 737ded6b6d0d..78cd77990c9c 100644 --- a/arch/arm64/include/asm/hyp_image.h +++ b/arch/arm64/include/asm/hyp_image.h @@ -56,6 +56,9 @@ */ #define KVM_NVHE_ALIAS(sym) kvm_nvhe_sym(sym) = sym; +/* Defines a linker script alias for KVM nVHE hyp symbols */ +#define KVM_NVHE_ALIAS_HYP(first, sec) kvm_nvhe_sym(first) = kvm_nvhe_sym(sec); + #endif /* LINKER_SCRIPT */ #endif /* __ARM64_HYP_IMAGE_H__ */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 5aa9ed1e9ec6..4eb7a15c8b60 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -104,6 +104,17 @@ KVM_NVHE_ALIAS(kvm_arm_hyp_percpu_base); /* PMU available static key */ KVM_NVHE_ALIAS(kvm_arm_pmu_available); +/* Position-independent library routines */ +KVM_NVHE_ALIAS_HYP(clear_page, __pi_clear_page); +KVM_NVHE_ALIAS_HYP(copy_page, __pi_copy_page); +KVM_NVHE_ALIAS_HYP(memcpy, __pi_memcpy); +KVM_NVHE_ALIAS_HYP(memset, __pi_memset); + +#ifdef CONFIG_KASAN +KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy); +KVM_NVHE_ALIAS_HYP(__memset, __pi_memset); +#endif + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index a6707df4f6c0..bc98f8e3d1da 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -9,10 +9,14 @@ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -D__DISABLE_EXPORTS hostprogs := gen-hyprel HOST_EXTRACFLAGS += -I$(objtree)/include +lib-objs := clear_page.o copy_page.o memcpy.o memset.o +lib-objs := $(addprefix ../../../lib/, $(lib-objs)) + obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o +obj-y += $(lib-objs) ## ## Build rules for compiling nVHE hyp code From patchwork Wed Mar 10 17:57:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9E1FC433E6 for ; Wed, 10 Mar 2021 17:59:56 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6132F64FAB for ; Wed, 10 Mar 2021 17:59:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6132F64FAB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C93ohdCIj6V6o4O/5y8pR9c4bwR2VoyVKhL4ZwIZqC0=; b=pqcnMzq2mLEh1l JcVleWiULinpw4mDfDsNJNG4jPS6PwSR2nn6GOCOsqxoIWtzJ841w1cGXv0jXic6ZfRYFFNSVG4cZ zZS8Qamqq7/rugh2DMoR7puKl9mgVHW27y9DSWK20j/t+apqe0n67PhtHEQALjM3B98RI8/BM5nNr +cy3RrLTDdvDZutqpLaKsBM0LJfl6FP+2uJynnIADltNK51ZFOfh0MaaIEe3tYbQRQZJbcXJJjx6c iDUu7rmjodDaO+RFuURCBAjk4Bb0roWsy/7F8sKYmCBT+73x1iTIdFkwzMo5sz8CYCGGddQMnu+4m k2OqZ1RZ/IFz28xYJ6mg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK36J-007OGK-9Q; Wed, 10 Mar 2021 17:58:27 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK35t-007O8V-Uy for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:03 +0000 Received: by mail-wm1-x349.google.com with SMTP id l16so1707124wmc.0 for ; Wed, 10 Mar 2021 09:58:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=s/CcypIoKHhWlDuFq90NauZRAhLJN+zOA/2+VbPKEl7bCkkYK5hoLzP/8fcTIFS2Jw Eh2QLV0/3ae/Me9LJg/kSIJHfINCwyUghsvfc3NlyS2+CKouZoPhVRQp4NqkSfiM4elE 0XL7hHifOLC3rTjkxj5OHkLTZ0z70EjGPISclEW/WScJ9OD/opCYqzZwIuC0aHgNKOvH NjdLcmyK/ZC6x/3dY+D9yrroIdQ1WieRSpqp/FkFJD+pZ3GbV/6xBMG32Bm8isSkVG8A X6Tvi+R4lZcBDuZRCRR0fbtq1aFKojnTGRy99uLXC4Cz29gJPuMF1btIyudETPS2gUn8 44Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=t7UmS5jHvGq7IDaC3ju+yB3HK/65vzG5iwK5dyZD/oTlwVY8BAdnIxthh2fOjLTA1t qVmmFqcx+zPqzkm3Dxd6Jqr5zrflnO41xIhz5k8gcHnSbrbyG5bskRpsVv0nXnuYxPnV 36Y1AdRnY14hva8shdy9XpEQmrte7jC/FNdyDExbpl2/y6TIop87lyaFcpFcMnQzuifJ fxNTW72iKi1TeqUb8NS/c185VqM8F2HL+QLBx7dII1d5QInpSEI1K1zB5zthW1UmLjbt PTZYoMuhwfqadRzXplg9HqYKn8BsTOF5BFKfNW1Y6c0Z0ec0NRcpfJFTW1Oanj7GJxMU /ERw== X-Gm-Message-State: AOAM531QLw4hIkv+fteZBGmon48H8WAx/OkxFJYMHAPPgXXsRQyTpzQX CXlojd5BbFMM+8bHLsxqz8jxQqy93Lq/ X-Google-Smtp-Source: ABdhPJy9/R9TGY9ivDqgnnFZkLtnWiNHZ1S7FS08+EvGk2jWUIr8VHHpig3MSO5hiQDgO7P3GGdlNqMBiDpE X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c010:: with SMTP id c16mr4574451wmb.134.1615399080993; Wed, 10 Mar 2021 09:58:00 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:20 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-4-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 03/34] arm64: kvm: Add standalone ticket spinlock implementation for use at hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175802_249863_AE0F2DE4 X-CRM114-Status: GOOD ( 16.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon We will soon need to synchronise multiple CPUs in the hyp text at EL2. The qspinlock-based locking used by the host is overkill for this purpose and relies on the kernel's "percpu" implementation for the MCS nodes. Implement a simple ticket locking scheme based heavily on the code removed by commit c11090474d70 ("arm64: locking: Replace ticket lock implementation with qspinlock"). Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/spinlock.h | 92 ++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/spinlock.h diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h new file mode 100644 index 000000000000..76b537f8d1c6 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * A stand-alone ticket spinlock implementation for use by the non-VHE + * KVM hypervisor code running at EL2. + * + * Copyright (C) 2020 Google LLC + * Author: Will Deacon + * + * Heavily based on the implementation removed by c11090474d70 which was: + * Copyright (C) 2012 ARM Ltd. + */ + +#ifndef __ARM64_KVM_NVHE_SPINLOCK_H__ +#define __ARM64_KVM_NVHE_SPINLOCK_H__ + +#include +#include + +typedef union hyp_spinlock { + u32 __val; + struct { +#ifdef __AARCH64EB__ + u16 next, owner; +#else + u16 owner, next; +#endif + }; +} hyp_spinlock_t; + +#define hyp_spin_lock_init(l) \ +do { \ + *(l) = (hyp_spinlock_t){ .__val = 0 }; \ +} while (0) + +static inline void hyp_spin_lock(hyp_spinlock_t *lock) +{ + u32 tmp; + hyp_spinlock_t lockval, newval; + + asm volatile( + /* Atomically increment the next ticket. */ + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ +" prfm pstl1strm, %3\n" +"1: ldaxr %w0, %3\n" +" add %w1, %w0, #(1 << 16)\n" +" stxr %w2, %w1, %3\n" +" cbnz %w2, 1b\n", + /* LSE atomics */ +" mov %w2, #(1 << 16)\n" +" ldadda %w2, %w0, %3\n" + __nops(3)) + + /* Did we get the lock? */ +" eor %w1, %w0, %w0, ror #16\n" +" cbz %w1, 3f\n" + /* + * No: spin on the owner. Send a local event to avoid missing an + * unlock before the exclusive load. + */ +" sevl\n" +"2: wfe\n" +" ldaxrh %w2, %4\n" +" eor %w1, %w2, %w0, lsr #16\n" +" cbnz %w1, 2b\n" + /* We got the lock. Critical section starts here. */ +"3:" + : "=&r" (lockval), "=&r" (newval), "=&r" (tmp), "+Q" (*lock) + : "Q" (lock->owner) + : "memory"); +} + +static inline void hyp_spin_unlock(hyp_spinlock_t *lock) +{ + u64 tmp; + + asm volatile( + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " ldrh %w1, %0\n" + " add %w1, %w1, #1\n" + " stlrh %w1, %0", + /* LSE atomics */ + " mov %w1, #1\n" + " staddlh %w1, %0\n" + __nops(1)) + : "=Q" (lock->owner), "=&r" (tmp) + : + : "memory"); +} + +#endif /* __ARM64_KVM_NVHE_SPINLOCK_H__ */ From patchwork Wed Mar 10 17:57:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7F64C433DB for ; Wed, 10 Mar 2021 18:00:16 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 11DFB64D73 for ; Wed, 10 Mar 2021 18:00:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 11DFB64D73 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tOSnHz7qD1RAL7vtFfKpbrNO4xTsuzr9WPKMuE0BdXk=; b=XdGwX96qxLhoDF mvsOB7s9L/LVEIgBCbGcaXwITt1tlJ7LVaDAPJzzR6ICRUkYS0x0r0+4QvDdXeARe/lprSpAi6EFZ +xddDH1/vjaCbJTrp9O+/1yktCLAbVz5+KOCTDiicJ454u4yut/urFEXgxc3xzv+fKlruQ7PV7xvn GXV58tPa4xgxpSCBNEHlJpkSUprdk2T3qPDsaelgJLU6SfFBo5OI4gvPYikgQcwPIFFO1TBVVw0ol QMN4aZRZESWlvqupGwGwvShfZ0C1i6BDpjaysSruK6qIRdxg9McrnULmNKQh+TyQg6Xoi8ii2wfXQ WKhTRLsrAJ0yI1zwZggA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK36T-007OKa-GH; Wed, 10 Mar 2021 17:58:40 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK35v-007O8z-St for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:06 +0000 Received: by mail-wr1-x44a.google.com with SMTP id m9so8329886wrx.6 for ; Wed, 10 Mar 2021 09:58:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jVYWjziEKUKc3S+4nq/6jQVOOomaowaXjC9E5zES6pQ=; b=nsvOfXUVW6h8kIQh3yiueCj6FV4dlzSfhRdKHnNGnCkKPZI5fNxOpJJFoBLk7a54Ra 0GPqUFtoTtHggQ+6IiV2+VOOIhT901RRTg5+YuxqE+ARSv1/VUKqHMpwQ1MdLCYl8k2b Xb283tTIwYqpbwCorQjUPznmdKjXxotCgclOs8CvTcamx8etgspyG34WIuYYcE0QWrlr 8bo6+eBnQTPquXEMRDOSwJeb+gbU8uCFsF/Y1VUKs15yWviezlT9lUmCraZz/TI46qlV BTsS2Hw6XyHDuUwibzLiCcEgGmd/mXEW6BxSqwDw6CHUZYbLm1m+KGVFOu7ettih+hXF zIdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jVYWjziEKUKc3S+4nq/6jQVOOomaowaXjC9E5zES6pQ=; b=KlhCQNaNKxpO5FgoYb8FZnqFPEmkdayn0YfajNRwt04wgV61ojS0ANAlve1fZUZ/1x +4XZ4r1vJf2qHyUyWn3zdkQK3KQ/VVEDLd4Fl7i46wUceLhHDfL59pvqF9xC76DjRkKd m/+F27oFsuSfTkdUJXZ2LtGIkasiFhkRwNSnIm2XYF/bBJOf22hOZzpaZsdCRlKn1qgK m5ZiGPSmrWrI5OeQmsvnKCzoF7f34nu+0NjfY85uns7LBFtxaKmrzBIf4v3ozQx66SdA 0nJNJxITwg/D6wu4QEaKK4GIMslf9z1/6L2dq75baOOEqPbtnba/KldVH5O28PO4VdAV v3vg== X-Gm-Message-State: AOAM530KD6ojgCg7p5fD+XNOL5VqAkmlP27MwYaKypcH+iS/TAzbP3D7 BHQ9Q8RUmMkGMKU+AZJBJdkel/DVbawu X-Google-Smtp-Source: ABdhPJyO6+G1NNLekpztDtLp4v2lf7XQ4BBn1PTycPaCeHbJSNnRb9NKPHusbNv9yGG4hMs7VKpqhSmBM23a X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:adf:a406:: with SMTP id d6mr4774378wra.141.1615399083137; Wed, 10 Mar 2021 09:58:03 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:21 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-5-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 04/34] KVM: arm64: Initialize kvm_nvhe_init_params early From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175803_967218_41E9290D X-CRM114-Status: GOOD ( 17.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the initialization of kvm_nvhe_init_params in a dedicated function that is run early, and only once during KVM init, rather than every time the KVM vectors are set and reset. This also opens the opportunity for the hypervisor to change the init structs during boot, hence simplifying the replacement of host-provided page-table by the one the hypervisor will create for itself. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/arm.c | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index fc4c95dd2d26..2d1e7ef69c04 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1383,22 +1383,18 @@ static int kvm_init_vector_slots(void) return 0; } -static void cpu_init_hyp_mode(void) +static void cpu_prepare_hyp_mode(int cpu) { - struct kvm_nvhe_init_params *params = this_cpu_ptr_nvhe_sym(kvm_init_params); - struct arm_smccc_res res; + struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); unsigned long tcr; - /* Switch from the HYP stub to our own HYP init vector */ - __hyp_set_vectors(kvm_get_idmap_vector()); - /* * Calculate the raw per-cpu offset without a translation from the * kernel's mapping to the linear mapping, and store it in tpidr_el2 * so that we can use adr_l to access per-cpu variables in EL2. * Also drop the KASAN tag which gets in the way... */ - params->tpidr_el2 = (unsigned long)kasan_reset_tag(this_cpu_ptr_nvhe_sym(__per_cpu_start)) - + params->tpidr_el2 = (unsigned long)kasan_reset_tag(per_cpu_ptr_nvhe_sym(__per_cpu_start, cpu)) - (unsigned long)kvm_ksym_ref(CHOOSE_NVHE_SYM(__per_cpu_start)); params->mair_el2 = read_sysreg(mair_el1); @@ -1422,7 +1418,7 @@ static void cpu_init_hyp_mode(void) tcr |= (idmap_t0sz & GENMASK(TCR_TxSZ_WIDTH - 1, 0)) << TCR_T0SZ_OFFSET; params->tcr_el2 = tcr; - params->stack_hyp_va = kern_hyp_va(__this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE); + params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); /* @@ -1430,6 +1426,15 @@ static void cpu_init_hyp_mode(void) * be read while the MMU is off. */ kvm_flush_dcache_to_poc(params, sizeof(*params)); +} + +static void cpu_init_hyp_mode(void) +{ + struct kvm_nvhe_init_params *params; + struct arm_smccc_res res; + + /* Switch from the HYP stub to our own HYP init vector */ + __hyp_set_vectors(kvm_get_idmap_vector()); /* * Call initialization code, and switch to the full blown HYP code. @@ -1438,6 +1443,7 @@ static void cpu_init_hyp_mode(void) * cpus_have_const_cap() wrapper. */ BUG_ON(!system_capabilities_finalized()); + params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); @@ -1785,19 +1791,19 @@ static int init_hyp_mode(void) } } - /* - * Map Hyp percpu pages - */ for_each_possible_cpu(cpu) { char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu]; char *percpu_end = percpu_begin + nvhe_percpu_size(); + /* Map Hyp percpu pages */ err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP); - if (err) { kvm_err("Cannot map hyp percpu region\n"); goto out_err; } + + /* Prepare the CPU initialization parameters */ + cpu_prepare_hyp_mode(cpu); } if (is_protected_kvm_enabled()) { From patchwork Wed Mar 10 17:57:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6B90C433DB for ; Wed, 10 Mar 2021 18:01:01 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F10864FB4 for ; Wed, 10 Mar 2021 18:01:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F10864FB4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7H7sQrTyDotrU+b/UR0LE3wVaeYktumM9TUtr8+b7w4=; b=QCXZ0TkF7aqgfS J/AXHIt6Rywi2iqk6PEtrXNeXju3B2n8u1fivrs2o8a1hr6m1qwTltMnDWmmoisNCd6QrlOIUo3dy QLUCb4osJFdOXm2SuQ5Uq906tltsMcFgch0NFroUwFWxbbtqS4ptIbHOnRaO1lt96qSLB2og3HtXk jT9YhgfmsDPVXV6frK5bHeaOQT4JZww9i10DKYmVdqHKWKjQ101mZPabsgbVallF9beMhgku9omCB KCBGQQ5tQymDmbKSLJ559fXRwtx57ptst8u8nd5pH1aY149mP5fE/KbiszcVhiozdhsqY6+ZUjieI Wy3GTblyBXxjX7rEJm5A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK36q-007OVb-Vk; Wed, 10 Mar 2021 17:59:01 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK35z-007O9H-Ih for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:09 +0000 Received: by mail-qv1-xf49.google.com with SMTP id bt20so4357038qvb.0 for ; Wed, 10 Mar 2021 09:58:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lIPthT51J+nIs48nl/QvpI3Ln3AY2S6u6ppS3XkUMwM=; b=fHBfJM8xzLAclGXEjFaYJ5Rhln578wLzMWoSnqk9psHRaDBuLXBcARc1NhvK6zE3B1 cI2osMD4eClmpEaY1HG2sFX5g274O4sHptREG3EHY7jDQs+EYJD+eX0AZ/hpzN3YLGmp Y8QjFoJ1rGBijnkcrGc5QF/tGIGHoU258V1+/DMXbn9XDFr8KvWBiqDs4JlKA1EZ5Teb 9WPr3djabjTTnmuikYjQMgMjSzAxUvjpbqNOPWa0DZKjuxX5BUxiVqEBXB2KVvBeDAsg ffEpVSe+BFI6nvd6tRs+1g4UntXEnAu0/1UiOLszmNy6hb6AU8I4J4elFWEdhdROrc2Y 3yTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lIPthT51J+nIs48nl/QvpI3Ln3AY2S6u6ppS3XkUMwM=; b=r5/gFp+Tfcuf3epgbxYHttKAYcHdotIkFShXiHrWkTLu37p6k93MLcgChDA6ECMksR P9ND3JR76Z2IjxBsFNdWRxSt31I7/oe2x255p94wAbtTl2P2PEnDpsIz89GTlSO0DpJL HLwULAexLLE9ru4cNj0mF05fHxwwzgQ6b9hMH5S1U64pnELG7FYX8ou4Vvw1fPV8hMgs 5+7FfkjSDgKZz+EM4j43tKGt2HD6tBrjHYglUoU8S8Gx6YEEeTQ8fVxC8tZGrXflz2wi jKZqj5P36SHBLopm6ovSBMxCx5bstYer1MrcNdjAff05h4Il+6QFJvfZfDpiGTA8xcQw 9NXA== X-Gm-Message-State: AOAM531UOGGKvXII3SJrYBGx9p7eOjkDQfdlVABLRUTy+h6R1Yg9NpvD FzQocTpuYTMcutHZttNhrHaZjSYUp3YP X-Google-Smtp-Source: ABdhPJz0tdSYBeivnJPeagi2ze/9rhhZHyfPUL0XortI6Q0zqZ15QP96f7jyHF9BhRekOYOtkNSCxsIk+TrK X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:c60b:: with SMTP id v11mr4064868qvi.44.1615399085165; Wed, 10 Mar 2021 09:58:05 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:22 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-6-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 05/34] KVM: arm64: Avoid free_page() in page-table allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175807_707865_F0CB77E9 X-CRM114-Status: GOOD ( 13.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the KVM page-table allocator uses a mix of put_page() and free_page() calls depending on the context even though page-allocation is always achieved using variants of __get_free_page(). Make the code consistent by using put_page() throughout, and reduce the memory management API surface used by the page-table code. This will ease factoring out page-allocation from pgtable.c, which is a pre-requisite to creating page-tables at EL2. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 4d177ce1d536..81fe032f34d1 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -413,7 +413,7 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); return 0; } @@ -425,7 +425,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - free_page((unsigned long)pgt->pgd); + put_page(virt_to_page(pgt->pgd)); pgt->pgd = NULL; } @@ -577,7 +577,7 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); put_page(virt_to_page(ptep)); if (data->anchor == ptep) { @@ -700,7 +700,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, } if (childp) - free_page((unsigned long)childp); + put_page(virt_to_page(childp)); return 0; } @@ -897,7 +897,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, put_page(virt_to_page(ptep)); if (kvm_pte_table(pte, level)) - free_page((unsigned long)kvm_pte_follow(pte)); + put_page(virt_to_page(kvm_pte_follow(pte))); return 0; } From patchwork Wed Mar 10 17:57:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C662CC433DB for ; Wed, 10 Mar 2021 18:01:12 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7A7AB64FAB for ; Wed, 10 Mar 2021 18:01:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A7AB64FAB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CIBV+VXJNGjuqBJJkqY8b476eSPZKGw+fr2UnpopnhE=; b=L/Xi9hGONU561l p/WkVHwKP7DNnSM+KrA1wfrW/0RGVrdbomMipkVABDBEGTBvHvaRF4Gm2a8C51b3OfcnloJyijbnL ypLK3W5iznXg2RxhsOHW+gJxQ2Nj3cO5KAuhwqtCrUqQDNaQEnIVDWou31eYgu/eAyse10nAMpn9E OIe4T0jL+tWOAp/mfIVjYAhMAslfy0HrNENl2NUxQY2Q6lgIELbfdhW8YYe+Kz5sGCblutdS9R7Eb FU3T75LRDY0yIUSnD/6cKt8EeegVvh1ovgDvimh4AmU6bdfm6dy3EXM8g7KjgSFVCPofIVNrq3iCH YVBSEnHnVP9ntZBDmuFQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK372-007OeD-RD; Wed, 10 Mar 2021 17:59:13 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK360-007O9d-H1 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:11 +0000 Received: by mail-wm1-x34a.google.com with SMTP id z26so2983395wml.4 for ; Wed, 10 Mar 2021 09:58:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=rdFGZTWCX0CcxBmhRRLEZcZ3F5Rwyx/Rl1k0s5yGSNE=; b=ECLspN4kNujG0OhMg0MKm4jcmTxH39XpeEmZI9FWVukKIGR0R0yGP5RKIbsIRUg3Ih 7qzdYSPGfu/mkGhqiU7K6zVI5IEKJ5yZBH7CXzSKIE1PX3df/xx3AsbENQWJoEDSD2Mg zN7mBX0M0D0zCtdxMPx6THqV16aWByqQ7WOvUTdXGBWoBGBHdpDpXOxIUv3m2SlqiyOj mW6fKxBKfkxI5mogUDJr/d5QTSOmrqdjtwK/5guc9tRxVv16ikk3Utt0DptbhJP9P4vC Ydur84w4fdEL7J2qWgkVAtE4zzwrItobRnBnPXn4errQxFeXVp8knn2q1NQZv9JXhb4K ayuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=rdFGZTWCX0CcxBmhRRLEZcZ3F5Rwyx/Rl1k0s5yGSNE=; b=pDzgV43hxDXlUuGYd/52yy3pdfxh3nAdfJbkf2cvYoMBmR6QgdBh911UNPSSgMbdjw vGzXNSDKiHJowRQtprcCu2/Eq4zUll2tvTrGFT85KzCUCE45JxBrJdazX5N7d8E6z98z lY6z0Ngw3TwwN4GhDjpiVcAXGCaL3b3VDGx+rzsuUuQD/wAqNzfFM2DPY4OiAVtgPF3Q OrYEMaC38Fa0XAwYoFIDADXZHcrvUKqvwCDvq9J9TZfHo6zT0PO4fYqVBHcF+iDqsbW7 Fz2NoA7TuixHsdPOW4HDsO0eSPVIxD2YnpjHa3CenPs1k4eJ8mN/+a/8BKLmBGTiPoGE Mrkw== X-Gm-Message-State: AOAM530HBPyBt7KmcqwCYoDhbfMvqxaz+0xsw1erM4nWlEINbq216xNV 98OnxC6ynArye2YZA3wj0sXrM6yU3U7M X-Google-Smtp-Source: ABdhPJzj72qrmenAgL4KyeuYj9YMQSMJVSiiit1nti7l1BZ+/xldp5MUczXr+r8dV22p7zAognYZvEX2eftv X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:bc56:: with SMTP id m83mr4566331wmf.174.1615399087350; Wed, 10 Mar 2021 09:58:07 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:23 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-7-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 06/34] KVM: arm64: Factor memory allocation out of pgtable.c From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175808_764821_CD9A3C1A X-CRM114-Status: GOOD ( 22.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for enabling the creation of page-tables at EL2, factor all memory allocation out of the page-table code, hence making it re-usable with any compatible memory allocator. No functional changes intended. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_pgtable.h | 41 +++++++++++- arch/arm64/kvm/hyp/pgtable.c | 98 +++++++++++++++++----------- arch/arm64/kvm/mmu.c | 66 ++++++++++++++++++- 3 files changed, 163 insertions(+), 42 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 8886d43cfb11..bbe840e430cb 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -13,17 +13,50 @@ typedef u64 kvm_pte_t; +/** + * struct kvm_pgtable_mm_ops - Memory management callbacks. + * @zalloc_page: Allocate a single zeroed memory page. The @arg parameter + * can be used by the walker to pass a memcache. The + * initial refcount of the page is 1. + * @zalloc_pages_exact: Allocate an exact number of zeroed memory pages. The + * @size parameter is in bytes, and is rounded-up to the + * next page boundary. The resulting allocation is + * physically contiguous. + * @free_pages_exact: Free an exact number of memory pages previously + * allocated by zalloc_pages_exact. + * @get_page: Increment the refcount on a page. + * @put_page: Decrement the refcount on a page. When the refcount + * reaches 0 the page is automatically freed. + * @page_count: Return the refcount of a page. + * @phys_to_virt: Convert a physical address into a virtual address mapped + * in the current context. + * @virt_to_phys: Convert a virtual address mapped in the current context + * into a physical address. + */ +struct kvm_pgtable_mm_ops { + void* (*zalloc_page)(void *arg); + void* (*zalloc_pages_exact)(size_t size); + void (*free_pages_exact)(void *addr, size_t size); + void (*get_page)(void *addr); + void (*put_page)(void *addr); + int (*page_count)(void *addr); + void* (*phys_to_virt)(phys_addr_t phys); + phys_addr_t (*virt_to_phys)(void *addr); +}; + /** * struct kvm_pgtable - KVM page-table. * @ia_bits: Maximum input address size, in bits. * @start_level: Level at which the page-table walk starts. * @pgd: Pointer to the first top-level entry of the page-table. + * @mm_ops: Memory management callbacks. * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. */ struct kvm_pgtable { u32 ia_bits; u32 start_level; kvm_pte_t *pgd; + struct kvm_pgtable_mm_ops *mm_ops; /* Stage-2 only */ struct kvm_s2_mmu *mmu; @@ -86,10 +119,12 @@ struct kvm_pgtable_walker { * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table. * @pgt: Uninitialised page-table structure to initialise. * @va_bits: Maximum virtual address bits. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits); +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_hyp_destroy() - Destroy an unused hypervisor stage-1 page-table. @@ -126,10 +161,12 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. * @kvm: KVM structure representing the guest virtual machine. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm); +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 81fe032f34d1..b975a67d1f85 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -152,9 +152,9 @@ static kvm_pte_t kvm_phys_to_pte(u64 pa) return pte; } -static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte) +static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) { - return __va(kvm_pte_to_phys(pte)); + return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); } static void kvm_set_invalid_pte(kvm_pte_t *ptep) @@ -163,9 +163,10 @@ static void kvm_set_invalid_pte(kvm_pte_t *ptep) WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID); } -static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp) +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, + struct kvm_pgtable_mm_ops *mm_ops) { - kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(__pa(childp)); + kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE); pte |= KVM_PTE_VALID; @@ -227,7 +228,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, goto out; } - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, data->pgt->mm_ops); ret = __kvm_pgtable_walk(data, childp, level + 1); if (ret) goto out; @@ -302,8 +303,9 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, } struct hyp_map_data { - u64 phys; - kvm_pte_t attr; + u64 phys; + kvm_pte_t attr; + struct kvm_pgtable_mm_ops *mm_ops; }; static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -358,6 +360,8 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { kvm_pte_t *childp; + struct hyp_map_data *data = arg; + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg)) return 0; @@ -365,11 +369,11 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; - childp = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + childp = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!childp) return -ENOMEM; - kvm_set_table_pte(ptep, childp); + kvm_set_table_pte(ptep, childp, mm_ops); return 0; } @@ -379,6 +383,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, int ret; struct hyp_map_data map_data = { .phys = ALIGN_DOWN(phys, PAGE_SIZE), + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = hyp_map_walker, @@ -396,16 +401,18 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, return ret; } -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops) { u64 levels = ARM64_HW_PGTABLE_LEVELS(va_bits); - pgt->pgd = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + pgt->pgd = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = va_bits; pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels; + pgt->mm_ops = mm_ops; pgt->mmu = NULL; return 0; } @@ -413,7 +420,9 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - put_page(virt_to_page(kvm_pte_follow(*ptep))); + struct kvm_pgtable_mm_ops *mm_ops = arg; + + mm_ops->put_page((void *)kvm_pte_follow(*ptep, mm_ops)); return 0; } @@ -422,10 +431,11 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) struct kvm_pgtable_walker walker = { .cb = hyp_free_walker, .flags = KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - put_page(virt_to_page(pgt->pgd)); + pgt->mm_ops->put_page(pgt->pgd); pgt->pgd = NULL; } @@ -437,6 +447,8 @@ struct stage2_map_data { struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; + + struct kvm_pgtable_mm_ops *mm_ops; }; static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -470,7 +482,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, { kvm_pte_t new, old = *ptep; u64 granule = kvm_granule_size(level), phys = data->phys; - struct page *page = virt_to_page(ptep); + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (!kvm_block_mapping_supported(addr, end, phys, level)) return -E2BIG; @@ -492,11 +504,11 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, */ kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); - put_page(page); + mm_ops->put_page(ptep); } smp_store_release(ptep, new); - get_page(page); + mm_ops->get_page(ptep); data->phys += granule; return 0; } @@ -526,13 +538,13 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret; + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp, pte = *ptep; - struct page *page = virt_to_page(ptep); + int ret; if (data->anchor) { if (kvm_pte_valid(pte)) - put_page(page); + mm_ops->put_page(ptep); return 0; } @@ -547,7 +559,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (!data->memcache) return -ENOMEM; - childp = kvm_mmu_memory_cache_alloc(data->memcache); + childp = mm_ops->zalloc_page(data->memcache); if (!childp) return -ENOMEM; @@ -559,11 +571,11 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (kvm_pte_valid(pte)) { kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); - put_page(page); + mm_ops->put_page(ptep); } - kvm_set_table_pte(ptep, childp); - get_page(page); + kvm_set_table_pte(ptep, childp, mm_ops); + mm_ops->get_page(ptep); return 0; } @@ -572,13 +584,14 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; int ret = 0; if (!data->anchor) return 0; - put_page(virt_to_page(kvm_pte_follow(*ptep))); - put_page(virt_to_page(ptep)); + mm_ops->put_page(kvm_pte_follow(*ptep, mm_ops)); + mm_ops->put_page(ptep); if (data->anchor == ptep) { data->anchor = NULL; @@ -633,6 +646,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .phys = ALIGN_DOWN(phys, PAGE_SIZE), .mmu = pgt->mmu, .memcache = mc, + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -669,7 +683,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - struct kvm_s2_mmu *mmu = arg; + struct kvm_pgtable *pgt = arg; + struct kvm_s2_mmu *mmu = pgt->mmu; + struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; kvm_pte_t pte = *ptep, *childp = NULL; bool need_flush = false; @@ -677,9 +693,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, mm_ops); - if (page_count(virt_to_page(childp)) != 1) + if (mm_ops->page_count(childp) != 1) return 0; } else if (stage2_pte_cacheable(pte)) { need_flush = true; @@ -692,15 +708,15 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, */ kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, addr, level); - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (need_flush) { - stage2_flush_dcache(kvm_pte_follow(pte), + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); } if (childp) - put_page(virt_to_page(childp)); + mm_ops->put_page(childp); return 0; } @@ -709,7 +725,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, - .arg = pgt->mmu, + .arg = pgt, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; @@ -841,12 +857,13 @@ static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pte)) return 0; - stage2_flush_dcache(kvm_pte_follow(pte), kvm_granule_size(level)); + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); return 0; } @@ -855,6 +872,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) struct kvm_pgtable_walker walker = { .cb = stage2_flush_walker, .flags = KVM_PGTABLE_WALK_LEAF, + .arg = pgt->mm_ops, }; if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) @@ -863,7 +881,8 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; u64 vtcr = kvm->arch.vtcr; @@ -872,12 +891,13 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; pgd_sz = kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE; - pgt->pgd = alloc_pages_exact(pgd_sz, GFP_KERNEL_ACCOUNT | __GFP_ZERO); + pgt->pgd = mm_ops->zalloc_pages_exact(pgd_sz); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = ia_bits; pgt->start_level = start_level; + pgt->mm_ops = mm_ops; pgt->mmu = &kvm->arch.mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ @@ -889,15 +909,16 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte)) return 0; - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (kvm_pte_table(pte, level)) - put_page(virt_to_page(kvm_pte_follow(pte))); + mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); return 0; } @@ -909,10 +930,11 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) .cb = stage2_free_walker, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; - free_pages_exact(pgt->pgd, pgd_sz); + pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 77cb2d28f2a4..4d41d7838d53 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -88,6 +88,44 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_valid(pfn); } +static void *stage2_memcache_zalloc_page(void *arg) +{ + struct kvm_mmu_memory_cache *mc = arg; + + /* Allocated with __GFP_ZERO, so no need to zero */ + return kvm_mmu_memory_cache_alloc(mc); +} + +static void *kvm_host_zalloc_pages_exact(size_t size) +{ + return alloc_pages_exact(size, GFP_KERNEL_ACCOUNT | __GFP_ZERO); +} + +static void kvm_host_get_page(void *addr) +{ + get_page(virt_to_page(addr)); +} + +static void kvm_host_put_page(void *addr) +{ + put_page(virt_to_page(addr)); +} + +static int kvm_host_page_count(void *addr) +{ + return page_count(virt_to_page(addr)); +} + +static phys_addr_t kvm_host_pa(void *addr) +{ + return __pa(addr); +} + +static void *kvm_host_va(phys_addr_t phys) +{ + return __va(phys); +} + /* * Unmapping vs dcache management: * @@ -351,6 +389,17 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, return 0; } +static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { + .zalloc_page = stage2_memcache_zalloc_page, + .zalloc_pages_exact = kvm_host_zalloc_pages_exact, + .free_pages_exact = free_pages_exact, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .page_count = kvm_host_page_count, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + /** * kvm_init_stage2_mmu - Initialise a S2 MMU strucrure * @kvm: The pointer to the KVM structure @@ -374,7 +423,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm); + err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; @@ -1208,6 +1257,19 @@ static int kvm_map_idmap_text(void) return err; } +static void *kvm_hyp_zalloc_page(void *arg) +{ + return (void *)get_zeroed_page(GFP_KERNEL); +} + +static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { + .zalloc_page = kvm_hyp_zalloc_page, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + int kvm_mmu_init(void) { int err; @@ -1251,7 +1313,7 @@ int kvm_mmu_init(void) goto out; } - err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits); + err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits, &kvm_hyp_mm_ops); if (err) goto out_free_pgtable; From patchwork Wed Mar 10 17:57:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AB5CC433E0 for ; Wed, 10 Mar 2021 18:01:20 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CD74E64F3A for ; Wed, 10 Mar 2021 18:01:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD74E64F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DKE9tlo8M5fwyFz+Kmx+Tn/XAx2dg6JG9sCWW/TtQ+A=; b=jP0Ha/aP7XwvZ2 kAf0N3+c06UW5rhhlp2Ww1WYXb0fQo6admlXMr6d7cbx0vTj8hCOnVorxAHPXsFh1a/hpkwjCrWOv D55CqLjARANgu7H9vHOkY5vfmcA7yEbT9rD5BmJxav2MgnhgAFtkLvTIKRjLqmqk9v/E5qw5UFtyg iEigtPWoIv1dWXRlPwzQAIuKP8dQbX0z5wdMzwbb2iolYuU4M2ULTOH5ZFQ0NsvMV7XtBIHgRVZAx xKmgRb8ZpRATTyfamsJSWgZIhsPZt42+GgncaIVe+Qoki8BZt/Be6qIJ0AUUYEGbyzKkTa3Y6e6VF fqpDQg4ZB4HvvlJDNBTA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK37F-007Ojr-CJ; Wed, 10 Mar 2021 17:59:26 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK362-007OAg-PR for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:12 +0000 Received: by mail-qv1-xf49.google.com with SMTP id u6so4255526qvu.11 for ; Wed, 10 Mar 2021 09:58:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9K/nJk/Yl2gqbmnNeiM3LL/ayLtkmDAblR9vIKiL4JQ=; b=GO/2FBWOCecwwU9cbVkaCLcKOpNoYUEQxlXaOTju5rYBLJe6KVFOjZhrsr50oPi7cJ oixBWT8jN56XkSpGuyKtO7eFphPP+fGTZfyqjhPKFqd5yH5nSuUmj1pyzAf5pzxmu56q A+0qc/vZxgDTaFGM0qtohkzJjFQauPf5/o54cNNBWWToVoSBWWknU86U6JG86OhlV+G/ vZ5/5v1tIaKV0lowHVFm2NL0U6mxRVnpTeAe53hf5LDhQ4RuUfWYqTjD1ujQMQP6H/hw E5lUbIgyH/HaFOtUtqmH3lb3sXaCSAfTmPfh2bO716yMOy+JBoQaPhloemCr0YtCElvV juvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9K/nJk/Yl2gqbmnNeiM3LL/ayLtkmDAblR9vIKiL4JQ=; b=JOoxOebRHRzYDPEOieaFawC5WzQscP4GnBV4Z/MtFUZG6PChSqbXS+iuYdsz5Qd3TD 4wUjqsHsD2xQZk2GbQiQ+27fULlUgRJL5lFr473pEByktiR7T7rd6RLC4RwgX12QQSvV BDbRNaqp58Omd4f1eUzhWqZlblQyRH0vqMvA1c8zeeRR4AC5Ine6INaep9cxHvP0SIbe fDmeyau5QT3HgMWtaS/Mle6L3e//USCmKIIXMKLak9TVQONcpqaVuMiCuoXVFck5d7Ch G1vyqu4p370h2AYWo7YAkM4c20L7bmC+Hmb/CVORzdEcU0PuJtRZpU6fTsBbYNJe5vdE JoHw== X-Gm-Message-State: AOAM533MrIBB0JfpbMtlC2IN0zsopUWvkWdRfBSdqVDzMGzN03TSNSJG T2zMftzPDmE6i+CAYnAnwtyUiO++7m+o X-Google-Smtp-Source: ABdhPJzB9OC6KWOlWJUJdL2k6mNjWxJq7jz6ESdJuBvXdwVSCdMLhWW4203j676QenoaxLPF0EYv6dv6kppm X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:1870:: with SMTP id eh16mr4059145qvb.23.1615399089435; Wed, 10 Mar 2021 09:58:09 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:24 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-8-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 07/34] KVM: arm64: Introduce a BSS section for use at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175810_956126_F0AF594F X-CRM114-Status: GOOD ( 20.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the hyp code cannot make full use of a bss, as the kernel section is mapped read-only. While this mapping could simply be changed to read-write, it would intermingle even more the hyp and kernel state than they currently are. Instead, introduce a __hyp_bss section, that uses reserved pages, and create the appropriate RW hyp mappings during KVM init. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 52 ++++++++++++++++++++----------- arch/arm64/kvm/arm.c | 14 ++++++++- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 1 + 4 files changed, 49 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 2f36b16a5b5d..e4ad9db53af1 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -13,6 +13,7 @@ extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[]; extern char __hyp_text_start[], __hyp_text_end[]; extern char __hyp_rodata_start[], __hyp_rodata_end[]; extern char __hyp_reloc_begin[], __hyp_reloc_end[]; +extern char __hyp_bss_start[], __hyp_bss_end[]; extern char __idmap_text_start[], __idmap_text_end[]; extern char __initdata_begin[], __initdata_end[]; extern char __inittext_begin[], __inittext_end[]; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7eea7888bb02..e96173ce211b 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -5,24 +5,7 @@ * Written by Martin Mares */ -#define RO_EXCEPTION_TABLE_ALIGN 8 -#define RUNTIME_DISCARD_EXIT - -#include -#include #include -#include -#include -#include - -#include "image.h" - -OUTPUT_ARCH(aarch64) -ENTRY(_text) - -jiffies = jiffies_64; - - #ifdef CONFIG_KVM #define HYPERVISOR_EXTABLE \ . = ALIGN(SZ_8); \ @@ -51,13 +34,43 @@ jiffies = jiffies_64; __hyp_reloc_end = .; \ } +#define BSS_FIRST_SECTIONS \ + __hyp_bss_start = .; \ + *(HYP_SECTION_NAME(.bss)) \ + . = ALIGN(PAGE_SIZE); \ + __hyp_bss_end = .; + +/* + * We require that __hyp_bss_start and __bss_start are aligned, and enforce it + * with an assertion. But the BSS_SECTION macro places an empty .sbss section + * between them, which can in some cases cause the linker to misalign them. To + * work around the issue, force a page alignment for __bss_start. + */ +#define SBSS_ALIGN PAGE_SIZE #else /* CONFIG_KVM */ #define HYPERVISOR_EXTABLE #define HYPERVISOR_DATA_SECTIONS #define HYPERVISOR_PERCPU_SECTION #define HYPERVISOR_RELOC_SECTION +#define SBSS_ALIGN 0 #endif +#define RO_EXCEPTION_TABLE_ALIGN 8 +#define RUNTIME_DISCARD_EXIT + +#include +#include +#include +#include +#include + +#include "image.h" + +OUTPUT_ARCH(aarch64) +ENTRY(_text) + +jiffies = jiffies_64; + #define HYPERVISOR_TEXT \ /* \ * Align to 4 KB so that \ @@ -276,7 +289,7 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; - BSS_SECTION(0, 0, 0) + BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; @@ -324,6 +337,9 @@ ASSERT(__hibernate_exit_text_end - (__hibernate_exit_text_start & ~(SZ_4K - 1)) ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE, "Entry trampoline text too big") #endif +#ifdef CONFIG_KVM +ASSERT(__hyp_bss_start == __bss_start, "HYP and Host BSS are misaligned") +#endif /* * If padding is applied before .head.text, virt<->phys conversions will fail. */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 2d1e7ef69c04..3f8bcf8db036 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1770,7 +1770,19 @@ static int init_hyp_mode(void) goto out_err; } - err = create_hyp_mappings(kvm_ksym_ref(__bss_start), + /* + * .hyp.bss is guaranteed to be placed at the beginning of the .bss + * section thanks to an assertion in the linker script. Map it RW and + * the rest of .bss RO. + */ + err = create_hyp_mappings(kvm_ksym_ref(__hyp_bss_start), + kvm_ksym_ref(__hyp_bss_end), PAGE_HYP); + if (err) { + kvm_err("Cannot map hyp bss section: %d\n", err); + goto out_err; + } + + err = create_hyp_mappings(kvm_ksym_ref(__hyp_bss_end), kvm_ksym_ref(__bss_stop), PAGE_HYP_RO); if (err) { kvm_err("Cannot map bss section\n"); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index cd119d82d8e3..f4562f417d3f 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -25,4 +25,5 @@ SECTIONS { BEGIN_HYP_SECTION(.data..percpu) PERCPU_INPUT(L1_CACHE_BYTES) END_HYP_SECTION + HYP_SECTION(.bss) } From patchwork Wed Mar 10 17:57:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D83FC433DB for ; Wed, 10 Mar 2021 18:01:08 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9189364F3A for ; Wed, 10 Mar 2021 18:01:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9189364F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U45u+rdb6t/3/geF3CD8HdXMiWK41NPPcmk7186mgZ4=; b=l/FWpB495sZhAr +TWA4lcN7WhIlsmUFFqI7y40hDmJ4fjXMbijQlkSBAvcCzZ3nh8qqmDRyE2XIBQPFmO8YeK6Mm54Q +LVz/+P311b5Zz02VomQIdZM9Xt+fRgujCOmvzSLegdc4vDcCZEO2CjjiCb81a2UL90wwlWtkG5RZ zkPKG9vn7teAraC12o631+OupGCtXOe2Std642ngjFDmNKDx36veP/tAAHcXu9Jz7UVwQuIqv8JX0 jgJ0YPKhIyx+ccep2jvH3oeOEsuYu90CApuSZ67uGgqj+GOyvsp4WYxG5KnRV8uZTMLQKNkxfX5bm rQzpXtuqNMpUx67vaAdA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK37W-007Oub-0S; Wed, 10 Mar 2021 17:59:42 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK365-007OB9-I1 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:15 +0000 Received: by mail-wr1-x44a.google.com with SMTP id m23so3318351wrh.7 for ; Wed, 10 Mar 2021 09:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=FQ7gk+4i0GZUVGgKW9zQ6uuaT/h986N5brkzpgihVQ4=; b=l2/JAssrCgiSVNWvv4xKwvbtNAJ4uVkYJ8t6QBmtVRzgoG35risu7QZVUsv1ZVEVVc d0cu6c8ao+zRESzgSYR0tOiVun97ubBgTvNljNr07rf0ZaBhRxxm3aeL4YQF0lrAWMd3 J6EHU6UjAWsuDfgdjF2Xy9kBIiLSqFoO/8/8jY1z0jZEnLVRSGHJdrgiLo4rMCXudmEy SmGdNMHEAxYX4BQ0m/Q8CpJ8tyO8EwGVTgfToo90g/wP1g66/b1nh7za3kbk5Dbi1OyZ cHdkyNd2U0aCwKGZLb5VJukCs5LMD1M2DzqKbuxUNl5vA15jwSiQVnBxV4noRW0uEN6X S3WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=FQ7gk+4i0GZUVGgKW9zQ6uuaT/h986N5brkzpgihVQ4=; b=uN7JaApCILvREVtjb8AC+FaT9wFKMHR1bmXeSJq6XRXDCF9t2FYTh0hZem8nViIC6t 3eidswP0N/dUw/3BpYsgxeNpLVrOPgH3kB8nMRcy+BjzjjDOgkUquveeaXzeWIXVSxgp e59Sr1+lwyDdWWRMmu8/QfrpkC/IhuWO9D1+ZTsQT8oVw0hAsNGAaGxc1o4zonM+0weP uVR4IYdsMinM8Al14BCxXH3IqOiKXMM7QDmJBl9UG+XElqOxMjfmtAC4EIGX4mpeIRTv mp0Fa5gN26jFl06eZxubiqNA3xSXRO6LqP49TBvC4mFZEQNJ36p+qphS8lIgd52uoU8H x1hg== X-Gm-Message-State: AOAM532DDosqkftRcVw0ANSdu01fT2jD1Sn1bsaqNW/n5fUbtzUJR21L cmEieNssUSQw8w4Tu8eldbpvHG7qUnPK X-Google-Smtp-Source: ABdhPJw1jFJ2uambxxtcgPuAbKpIituWx/uYJZ6wnBrvmsU1lXHbtQHSpkrolAxbBPbUyRL7xj955m5ZQfKl X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c18e:: with SMTP id y14mr1333539wmi.1.1615399091555; Wed, 10 Mar 2021 09:58:11 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:25 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-9-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 08/34] KVM: arm64: Make kvm_call_hyp() a function call at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175813_884816_229755C3 X-CRM114-Status: GOOD ( 11.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kvm_call_hyp() has some logic to issue a function call or a hypercall depending on the EL at which the kernel is running. However, all the code compiled under __KVM_NVHE_HYPERVISOR__ is guaranteed to only run at EL2 which allows us to simplify. Add ifdefery to kvm_host.h to simplify kvm_call_hyp() in .hyp.text. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3d10e6527f7d..06ca4828005f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -591,6 +591,7 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva); void kvm_arm_halt_guest(struct kvm *kvm); void kvm_arm_resume_guest(struct kvm *kvm); +#ifndef __KVM_NVHE_HYPERVISOR__ #define kvm_call_hyp_nvhe(f, ...) \ ({ \ struct arm_smccc_res res; \ @@ -630,6 +631,11 @@ void kvm_arm_resume_guest(struct kvm *kvm); \ ret; \ }) +#else /* __KVM_NVHE_HYPERVISOR__ */ +#define kvm_call_hyp(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_ret(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) +#endif /* __KVM_NVHE_HYPERVISOR__ */ void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); From patchwork Wed Mar 10 17:57:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FC11C433DB for ; Wed, 10 Mar 2021 18:01:33 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B40C164F3A for ; Wed, 10 Mar 2021 18:01:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B40C164F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3jVJdLiLaLp4Jp7M2IClawu5QXL14UNUlP8IUw0dbLE=; b=eR3b75iJN6H8g3 RM68/huhYtEEdislCNlQCO5ylykPlRMNnSZ/0td2lob2mJ8fi7S4kb2FWk7qfROLtoMYK7iz+DEJd J9wOKFu2+ImuAFzRi4MFxigHyj8x0F6ca1RxVY7XVUAST7hyNoZb9sb4HWolVOM7dfA8CUTXYnj7r rLW2zrOztDfBrQa2l5UdqGtmmPA0CCPZKi0f90T+yp0DgGgA+rUj0az0hPk6bLCatRJbwkk7rCHgv CpEKxhxNGaPEj3HQFaqyeWj+Gd/qk1/qZqjmCaagKgCzQGweOoymHiz9n89AAyUUY7xguuF9MLpW8 0LkKJmL8Bf713Ru+TKUg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK37m-007P3D-HC; Wed, 10 Mar 2021 17:59:58 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK366-007OBp-PU for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:16 +0000 Received: by mail-wr1-x449.google.com with SMTP id l10so8322721wry.16 for ; Wed, 10 Mar 2021 09:58:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=T621CrLzDVvHMT7g1JI2svAdR9qKxB9zRXJEq9CxOEU=; b=Hbvsvxu6NLISPbrvhiFxE7FXEqAU0Mv8t9SPVam60nUlEefn0BFRgckh6OphPN/n34 mMAxbCfp63tyHccY0qa/f3Jpc86iiuTI1qKtqv9p3GgGY0jIRmo2YehbRMB+bwHX+rkd FciY30zEjHtG42uGSiQAnecMzyOeKipqkx91RsUlvsIu/qkCtj/fVmS6EDgNmYhAdE64 CHNIzVWY6jaAli+IqjdG8DYNqiV3kcGDigjdAxVcu9mjVlx/FgYwRvHQ/HN8W8Ksp+tD UGCShyNHhI0MSaxMWGIBO2OuJLu9etYsd6zmBCTOBQlY6AXhHDojB8xnAR2EeHwTlARr WvoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=T621CrLzDVvHMT7g1JI2svAdR9qKxB9zRXJEq9CxOEU=; b=ochBsrpUFM17Q4ts5thDP6o6tpxKU/akPKQlbwBXIfEDn+0ZVuos9ZddJb7bpP38Gy Vhp2yXlQDvWeBeXK9jp8hAR9CfmL+TO5YG0h582qmmHuoij8B1zvWx5Lj2SZFeEesh9L PJ8gYK/P2alJiBaWXakq6dAWFvGJn4XFJXaElk8bmfTCxEpwxbzlAqTe6bEWM38Ddkh1 YDFtR18uz7BiLjFT5UkJsDlm3l3DiJLRaWwYrnOREixrDYdDHJ26waOC02lcBJZrWEhu miCkZEpy+UKiP7/yO5KdUP8D3WDjTVgJ5gdFudH/VJ55+1Jcqdld+y8yezisOaHz2yUx fnFg== X-Gm-Message-State: AOAM532aCHt6TXoa4xLZ0r57S/H5OFvExAFfcAisQpuDz8szbO26l8zk VYS7pEc8URN+g8O22ZSEgpjWt9zQIouN X-Google-Smtp-Source: ABdhPJxqfINpp6EENtOEj1k9D5Kyx0lGUnntW+YAdT5CBabTaE0ejM3PwJv5r9lDZwF+U+rCzWRzTGC22KfH X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:ed13:: with SMTP id l19mr4406714wmh.83.1615399093776; Wed, 10 Mar 2021 09:58:13 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:26 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-10-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 09/34] KVM: arm64: Allow using kvm_nvhe_sym() in hyp code From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175814_925841_8D55687A X-CRM114-Status: GOOD ( 10.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow the usage of code shared by the host and the hyp in static inline library functions, allow the usage of kvm_nvhe_sym() at EL2 by defaulting to the raw symbol name. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/hyp_image.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h index 78cd77990c9c..b4b3076a76fb 100644 --- a/arch/arm64/include/asm/hyp_image.h +++ b/arch/arm64/include/asm/hyp_image.h @@ -10,11 +10,15 @@ #define __HYP_CONCAT(a, b) a ## b #define HYP_CONCAT(a, b) __HYP_CONCAT(a, b) +#ifndef __KVM_NVHE_HYPERVISOR__ /* * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, * to separate it from the kernel proper. */ #define kvm_nvhe_sym(sym) __kvm_nvhe_##sym +#else +#define kvm_nvhe_sym(sym) sym +#endif #ifdef LINKER_SCRIPT From patchwork Wed Mar 10 17:57:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2399C433DB for ; Wed, 10 Mar 2021 18:01:54 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 38EF564F3A for ; Wed, 10 Mar 2021 18:01:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38EF564F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sstrT2DW1Q2IIpgnsLXdQCkc31MQTGjz4p0vILKduN4=; b=pnlci/wUor0Mng ZHpL/H1Baf+mjhIO0+neBdt1kN5nZQQ+iT4tGBBrCOUvwpv7WdBRZj7lS0fm3fbMoj2X35cRhiY2F 5bZISI9Sbv4uPfRtDgKQojJZVpVyi/NBTEKkkMO1YtBanl9qAOFk7KIx2UyODsEFdDlT5+HzqDs8l JixK67rRelB2sTTuKc/r6dJjW4c93C5j/cvB1fINClRLwM+NS4v7BvKwPzLrEOhszTM5zTiGpV8O8 HfFbyo3Sn4OtEq1NmyRXWvJJ6dfXIhJuwJeZCZxgpOaGeBTNmIhWS7r42WWzhXc73o7VB353JQxAl CqrYurBtgaFpDPh8UL+g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK386-007PEQ-WB; Wed, 10 Mar 2021 18:00:19 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK368-007OCJ-Cy for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:18 +0000 Received: by mail-wr1-x44a.google.com with SMTP id n17so8330544wrq.5 for ; Wed, 10 Mar 2021 09:58:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=VP8mf4K4lbKPxoOK4Eqb7s/YbwLsns2Dk6uqu7xpKg4=; b=lii4hE0NwlVugsKpM5UCLRkqnlSVsbBClXn1iNHrkaI3eox7X7nnAghzxeMAc8ylT6 hweHM0U1Etxj/ih8weub/puP6IvFHyRYyneOPy75qjQJQmIYzO+q5IZXtWu7QShmzcgj LnpMztYUkVYMY+erd5BKQGyWcOxR8z3MIO+3vFKDpbzNZ/AUG0aLe0CVeOzNozu4d7at pjBnwOU7ON2vG8KlxWm2BTkbhe4D0R7o98HB5G0ovo+JTwcShKkiZYCX9yv74ebNvhU5 1D9WrNDdAwlsXZ3cufbxN+zv0FpNw1vKBPYmtzrWXKAKm29h5BMjIM9Yrtzjirpao2hE imJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=VP8mf4K4lbKPxoOK4Eqb7s/YbwLsns2Dk6uqu7xpKg4=; b=qR08zVbhdwBXbrkjSxU8mCv9SGlhPsBKmc9A34zT7QfJaxowmrPStF9QwfJiHJtebT yDkwvFPCEuOltCk+6jCBy1lKZzXEn0KUZ5s+srlIit5hGWZpvIqIb+4RzOrTVoMhzosP sQ7Arexl8IHIdk7fWuyZlcydKoC00mPA4JUgzdqK4kLWaXaeFzboEYdjNQ0R2CMHRAEB 3SQDH3kaO1Sp431zBLEvdiSh9IG5jmDR0phLJ0y0F5sVISfzQuW9gfCqZUBpzu4gttkT QJmAmTwaXs+e7ZbFhvH21lLrFDjNeekojoCz22pwLt6eCvSae3BXADHru2ULN7WgKHoY spGA== X-Gm-Message-State: AOAM533KVN5kb1pX3889VN5zYirRhA8qPdU8QgQxp+jxQQc8mz8Qmz8K KBcD9zwF4CNpI9Bu1n6Pwg3faHV6yqLN X-Google-Smtp-Source: ABdhPJypsPzY+twHMqIoIR/dRQTgpcMRT2Q2HXKYRHwF3hnk6gvN33YLE/Tpf/7bdxQkYHUhGZfKNanPRnKI X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:adf:d84d:: with SMTP id k13mr4901235wrl.164.1615399095727; Wed, 10 Mar 2021 09:58:15 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:27 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-11-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 10/34] KVM: arm64: Introduce an early Hyp page allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175816_515577_5D3E68A7 X-CRM114-Status: GOOD ( 20.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With nVHE, the host currently creates all stage 1 hypervisor mappings at EL1 during boot, installs them at EL2, and extends them as required (e.g. when creating a new VM). But in a world where the host is no longer trusted, it cannot have full control over the code mapped in the hypervisor. In preparation for enabling the hypervisor to create its own stage 1 mappings during boot, introduce an early page allocator, with minimal functionality. This allocator is designed to be used only during early bootstrap of the hyp code when memory protection is enabled, which will then switch to using a full-fledged page allocator after init. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/early_alloc.h | 14 +++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 24 +++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/early_alloc.c | 54 +++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/psci-relay.c | 4 +- 5 files changed, 94 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/early_alloc.h create mode 100644 arch/arm64/kvm/hyp/include/nvhe/memory.h create mode 100644 arch/arm64/kvm/hyp/nvhe/early_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h new file mode 100644 index 000000000000..dc61aaa56f31 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_EARLY_ALLOC_H +#define __KVM_HYP_EARLY_ALLOC_H + +#include + +void hyp_early_alloc_init(void *virt, unsigned long size); +unsigned long hyp_early_alloc_nr_used_pages(void); +void *hyp_early_alloc_page(void *arg); +void *hyp_early_alloc_contig(unsigned int nr_pages); + +extern struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; + +#endif /* __KVM_HYP_EARLY_ALLOC_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h new file mode 100644 index 000000000000..3e49eaa7e682 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MEMORY_H +#define __KVM_HYP_MEMORY_H + +#include + +#include + +extern s64 hyp_physvirt_offset; + +#define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) +#define __hyp_va(phys) ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset)) + +static inline void *hyp_phys_to_virt(phys_addr_t phys) +{ + return __hyp_va(phys); +} + +static inline phys_addr_t hyp_virt_to_phys(void *addr) +{ + return __hyp_pa(addr); +} + +#endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index bc98f8e3d1da..24ff99e2eac5 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/early_alloc.c b/arch/arm64/kvm/hyp/nvhe/early_alloc.c new file mode 100644 index 000000000000..1306c430ab87 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/early_alloc.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include + +#include +#include + +struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; +s64 __ro_after_init hyp_physvirt_offset; + +static unsigned long base; +static unsigned long end; +static unsigned long cur; + +unsigned long hyp_early_alloc_nr_used_pages(void) +{ + return (cur - base) >> PAGE_SHIFT; +} + +void *hyp_early_alloc_contig(unsigned int nr_pages) +{ + unsigned long size = (nr_pages << PAGE_SHIFT); + void *ret = (void *)cur; + + if (!nr_pages) + return NULL; + + if (end - cur < size) + return NULL; + + cur += size; + memset(ret, 0, size); + + return ret; +} + +void *hyp_early_alloc_page(void *arg) +{ + return hyp_early_alloc_contig(1); +} + +void hyp_early_alloc_init(void *virt, unsigned long size) +{ + base = cur = (unsigned long)virt; + end = base + size; + + hyp_early_alloc_mm_ops.zalloc_page = hyp_early_alloc_page; + hyp_early_alloc_mm_ops.phys_to_virt = hyp_phys_to_virt; + hyp_early_alloc_mm_ops.virt_to_phys = hyp_virt_to_phys; +} diff --git a/arch/arm64/kvm/hyp/nvhe/psci-relay.c b/arch/arm64/kvm/hyp/nvhe/psci-relay.c index 63de71c0481e..08508783ec3d 100644 --- a/arch/arm64/kvm/hyp/nvhe/psci-relay.c +++ b/arch/arm64/kvm/hyp/nvhe/psci-relay.c @@ -11,6 +11,7 @@ #include #include +#include #include void kvm_hyp_cpu_entry(unsigned long r0); @@ -20,9 +21,6 @@ void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); /* Config options set by the host. */ struct kvm_host_psci_config __ro_after_init kvm_host_psci_config; -s64 __ro_after_init hyp_physvirt_offset; - -#define __hyp_pa(x) ((phys_addr_t)((x)) + hyp_physvirt_offset) #define INVALID_CPU_ID UINT_MAX From patchwork Wed Mar 10 17:57:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F6EC433E0 for ; Wed, 10 Mar 2021 18:02:11 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 92FE764FB3 for ; Wed, 10 Mar 2021 18:02:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 92FE764FB3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X+N3L6GMBiibzVm/VBzh+BCWH4pG8LggoO2vnZl52Go=; b=ARwCMnRQur+w/k ztpidLLFookJaeRHok0W9J1lRPjq28q9HKqoKHXg5JSdvCCkPx6trYuo79ZQ6ofjNDhhwaZKgG3cA b+TzIBm7b84GB9S+fHQxAUUWjVBnvwA9Dc+VuXFIKg+zf6cA8jLfnGBR98IfdJ1oEN1eyFK9fLFTe AmIQWoy3iTTWDJKC81bv8zlSmbwakSZqoSQ/iiodfa9oWN2xCtJQWy7VLOVe9w5tvluWWTor33Ta/ JCQFh0JzhvgHBpstkTY4aSCbIknbS6uZS2b+vYzhKJ2ZkEH7BZpPHR01Q6DewbRzTvJA3V/iyZN57 ttze7glJnHFf7H6a+EDA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK38O-007PN1-LQ; Wed, 10 Mar 2021 18:00:36 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36B-007ODE-KY for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:21 +0000 Received: by mail-qk1-x74a.google.com with SMTP id k68so13395202qke.2 for ; Wed, 10 Mar 2021 09:58:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=twje/T88dKdujyu/Ipi7uGGbAfwKvVyGbqSSEkcY7iA=; b=LQ0ZVrzLKNdU5CFZ/uHgdFHY9JJtbPriswTjvCuNFFdqNAFUOVuVsL11DRafoQlXRs +qA6mOO/XIpZhsDrX4ERKUL831GIYfsbE+YhI/HZL9z2EJgXSh57xNEXRp06rvAklb4F MZgzc6XLsTevrg2CaMb1FFTglqx9ngsFCS7aGzbYfmG/3tMbDI7yw9Bz9V9sE4k9t6H9 9ISno341dtYOyZomYdAIDZVzCfzWji7BZQYuO9WuddDxANZaqnLlb6SP9sEdd+81N0E5 DEiVR5nkt8P07Yw9f/2m/XFboo6ihfT8aJqnIv+81HN+EqmqZGtbe1vRqIAWUTogBnRV 0mMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=twje/T88dKdujyu/Ipi7uGGbAfwKvVyGbqSSEkcY7iA=; b=i0xb9gNOB2gOoc6+9VvclOwMdzc3pxUAnvgfizX483O3uwoL4J4o10npAAw/odqXr8 0MIzylyYvVwTin/NntU/77Fq7V2tPgHO/U2lSDXUmyIkxsIDQSvF1qwG2utSM9QJtVOL rWYoJuKEkyyWlk9g6S1oTbxEPqBPfeYiUzIiBZ0yxP5PdHDxiNxUnAVUdLCMPTCX43N8 eHp3rMGhu9UbmuKiYyJV5Wt1a9lGG6J7ZBZPdlsFQHQLNcwANaeMXuN7ofweyKFUWtVA 0v/MKxMCi7N9ppYCdEi6F1/Btsj/Ta4/ARmhgNSfb2mtb2d9OHFPm+EpXUVrYFtSQEP8 Ehdw== X-Gm-Message-State: AOAM532aeJTFWihg0cDiI4cADGCYSRTMhONWUL3HrZgYACb2o8XqmWkJ k10ip9JhlWRZXYlYXJDj+v8ERv6aYSp2 X-Google-Smtp-Source: ABdhPJzpVCtmRY3IUgz5C3HD74m5dt+Xopm/y+FY/xkaDnXqHj0IY+JJ6z8YgmzfYvx8drfccRDDpQ7iF9lF X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:1484:: with SMTP id bn4mr4267365qvb.8.1615399097698; Wed, 10 Mar 2021 09:58:17 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:28 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-12-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 11/34] KVM: arm64: Stub CONFIG_DEBUG_LIST at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175819_739979_45FAA91C X-CRM114-Status: GOOD ( 14.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to use the kernel list library at EL2, introduce stubs for the CONFIG_DEBUG_LIST out-of-lines calls. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/stub.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/stub.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 24ff99e2eac5..144da72ad510 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/stub.c b/arch/arm64/kvm/hyp/nvhe/stub.c new file mode 100644 index 000000000000..c0aa6bbfd79d --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/stub.c @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Stubs for out-of-line function calls caused by re-using kernel + * infrastructure at EL2. + * + * Copyright (C) 2020 - Google LLC + */ + +#include + +#ifdef CONFIG_DEBUG_LIST +bool __list_add_valid(struct list_head *new, struct list_head *prev, + struct list_head *next) +{ + return true; +} + +bool __list_del_entry_valid(struct list_head *entry) +{ + return true; +} +#endif From patchwork Wed Mar 10 17:57:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B816BC433DB for ; Wed, 10 Mar 2021 18:02:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3083B64F3A for ; Wed, 10 Mar 2021 18:02:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3083B64F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=m6kcOV+qyS1FtLKRwGef0CT8C3d30cFy8rrKCn9Ockg=; b=V7aTueTNZH2jj/ UfJcVjeDIeMlQ4QylD6nhdJ3pjcWrke+U5DwfNYQvEneFYZXZb1qc6crctIHFR9YSzy7BC/KlL7y4 p1vaLOuNSsLl+7Km3mSg/5ed4vrMaXz4kll2c0FpD9rapliKvhcQiw72cd1J4d0X/cMwv4POJyoS3 /Dhd6G+kUyCKNOiGL1dzYuq6uC+8MQjerG7RbLV5VZ0v6jL3P8ekiTRruHdJHKAQurkt9q4Jnhsmw 2DmakSUuJZDMyUNHLeOsISn7DxeTAwbo22ntRdJAu67Vk2hG1cv2SHUJOhd2Mbh3auAx0sRx2LF07 W2fmeAfSrvTSmu2IOttg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK38e-007PUB-7s; Wed, 10 Mar 2021 18:00:52 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36D-007ODw-2d for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:23 +0000 Received: by mail-wr1-x44a.google.com with SMTP id n16so7242386wro.1 for ; Wed, 10 Mar 2021 09:58:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Vko/POQoG1louXL8h4WbFiMfH/TPQZb+EYjyQ3Qtr8o=; b=JZ4vd/uOQtdnCZc4VES1NJ1PCqVnXeUeXqc/NSTjZ7wp0hteQXI7f+7l27xo9jn/4B L0ffS4w4V9QZuPcWm5glyBq7Jnmv+tnvrAlK7GNSL9XYMRx8D0yp05BNXY/iJsin1Vr9 4/npWdKJziQE9P60yPDx88l8jE2G0pwOFxjVdJEULmaoJ7lSHYZvLMJyPTqmDOoD7st1 mK3UyO0puiGbVL0XxnBUl089DoQ6+XKHO1sfzXAFL1ZQ6HBVKASqIAWr7yPE6/siIWXb RASfqoaZYb07YN2Wf2aJ34k4h8v2x9GE4g9mkFpH9mhZb70tNmCnwgHH3kVmHEeXGnbr tL9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Vko/POQoG1louXL8h4WbFiMfH/TPQZb+EYjyQ3Qtr8o=; b=S5hTGE25G7qRqKrFd7m4zb2vwKcAdZRZ/MoE5igSjALmCfbGqVBKFdjrSGd632bJ2P iz9CIiz7xL6FVvy4LXHKD1UEv0V9JMHqBLIl1ZwdLMcT1eibr+zhPopG3F8tHnEKzEeh d3llSM7kQSnKlw1ucjmBsA1NFGHF14PgZexju1b9OytpdPSltbs+bzfTvcEaBCOJ9gXn QS9CGSjJn8GkH4yO9MQgqimMU+krkFKmrFUL0/JDV+SOBxCXNGMxYyKJmKeSEfHuCs8E GPKFqLiZx+W0xeIKWFh8vBuqFQpbFfE9kKaIEjxxqb/HBoLbfkXyRmT/gMtqfaFT2Hs1 ZcZQ== X-Gm-Message-State: AOAM530g/7jS94A3SJIFk/5hld9fO+Qle5w6LXfu6oKiEjJI/wg/axoy UDEj4QBD9r8OXmC6Jk53nCjNRNB7HaJX X-Google-Smtp-Source: ABdhPJxE1hbjD87PPslcXEpix5TcSGI6ccD/wDw4HgvH86Qd17WFX2v/EbQIlsexm/uZqiDVevRwnmO/05tp X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:8041:: with SMTP id b62mr4669106wmd.0.1615399099962; Wed, 10 Mar 2021 09:58:19 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:29 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-13-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 12/34] KVM: arm64: Introduce a Hyp buddy page allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175821_185573_F4B758D3 X-CRM114-Status: GOOD ( 32.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the hyp code will require a basic form of memory management in order to allocate and free memory pages at EL2. This is needed for various use-cases, including the creation of hyp mappings or the allocation of stage 2 page tables. To address these use-case, introduce a simple memory allocator in the hyp code. The allocator is designed as a conventional 'buddy allocator', working with a page granularity. It allows to allocate and free physically contiguous pages from memory 'pools', with a guaranteed order alignment in the PA space. Each page in a memory pool is associated with a struct hyp_page which holds the page's metadata, including its refcount, as well as its current order, hence mimicking the kernel's buddy system in the GFP infrastructure. The hyp_page metadata are made accessible through a hyp_vmemmap, following the concept of SPARSE_VMEMMAP in the kernel. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 68 ++++++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 28 ++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 195 +++++++++++++++++++++++ 4 files changed, 292 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/gfp.h create mode 100644 arch/arm64/kvm/hyp/nvhe/page_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h new file mode 100644 index 000000000000..55b3f0ce5bc8 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_GFP_H +#define __KVM_HYP_GFP_H + +#include + +#include +#include + +#define HYP_NO_ORDER UINT_MAX + +struct hyp_pool { + /* + * Spinlock protecting concurrent changes to the memory pool as well as + * the struct hyp_page of the pool's pages until we have a proper atomic + * API at EL2. + */ + hyp_spinlock_t lock; + struct list_head free_area[MAX_ORDER]; + phys_addr_t range_start; + phys_addr_t range_end; + unsigned int max_order; +}; + +static inline void hyp_page_ref_inc(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + p->refcount++; + hyp_spin_unlock(&pool->lock); +} + +static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + int ret; + + hyp_spin_lock(&pool->lock); + p->refcount--; + ret = (p->refcount == 0); + hyp_spin_unlock(&pool->lock); + + return ret; +} + +static inline void hyp_set_page_refcounted(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + if (p->refcount) { + hyp_spin_unlock(&pool->lock); + hyp_panic(); + } + p->refcount = 1; + hyp_spin_unlock(&pool->lock); +} + +/* Allocation */ +void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order); +void hyp_get_page(void *addr); +void hyp_put_page(void *addr); + +/* Used pages cannot be freed */ +int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages, + unsigned int reserved_pages); +#endif /* __KVM_HYP_GFP_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h index 3e49eaa7e682..d2fb307c5952 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/memory.h +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -6,7 +6,17 @@ #include +struct hyp_pool; +struct hyp_page { + unsigned int refcount; + unsigned int order; + struct hyp_pool *pool; + struct list_head node; +}; + extern s64 hyp_physvirt_offset; +extern u64 __hyp_vmemmap; +#define hyp_vmemmap ((struct hyp_page *)__hyp_vmemmap) #define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) #define __hyp_va(phys) ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset)) @@ -21,4 +31,22 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr) return __hyp_pa(addr); } +#define hyp_phys_to_pfn(phys) ((phys) >> PAGE_SHIFT) +#define hyp_pfn_to_phys(pfn) ((phys_addr_t)((pfn) << PAGE_SHIFT)) +#define hyp_phys_to_page(phys) (&hyp_vmemmap[hyp_phys_to_pfn(phys)]) +#define hyp_virt_to_page(virt) hyp_phys_to_page(__hyp_pa(virt)) +#define hyp_virt_to_pfn(virt) hyp_phys_to_pfn(__hyp_pa(virt)) + +#define hyp_page_to_pfn(page) ((struct hyp_page *)(page) - hyp_vmemmap) +#define hyp_page_to_phys(page) hyp_pfn_to_phys((hyp_page_to_pfn(page))) +#define hyp_page_to_virt(page) __hyp_va(hyp_page_to_phys(page)) +#define hyp_page_to_pool(page) (((struct hyp_page *)page)->pool) + +static inline int hyp_page_count(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + return p->refcount; +} + #endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 144da72ad510..6894a917f290 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c new file mode 100644 index 000000000000..237e03bf0cb1 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include + +u64 __hyp_vmemmap; + +/* + * Index the hyp_vmemmap to find a potential buddy page, but make no assumption + * about its current state. + * + * Example buddy-tree for a 4-pages physically contiguous pool: + * + * o : Page 3 + * / + * o-o : Page 2 + * / + * / o : Page 1 + * / / + * o---o-o : Page 0 + * Order 2 1 0 + * + * Example of requests on this pool: + * __find_buddy_nocheck(pool, page 0, order 0) => page 1 + * __find_buddy_nocheck(pool, page 0, order 1) => page 2 + * __find_buddy_nocheck(pool, page 1, order 0) => page 0 + * __find_buddy_nocheck(pool, page 2, order 0) => page 3 + */ +static struct hyp_page *__find_buddy_nocheck(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + phys_addr_t addr = hyp_page_to_phys(p); + + addr ^= (PAGE_SIZE << order); + + /* + * Don't return a page outside the pool range -- it belongs to + * something else and may not be mapped in hyp_vmemmap. + */ + if (addr < pool->range_start || addr >= pool->range_end) + return NULL; + + return hyp_phys_to_page(addr); +} + +/* Find a buddy page currently available for allocation */ +static struct hyp_page *__find_buddy_avail(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + struct hyp_page *buddy = __find_buddy_nocheck(pool, p, order); + + if (!buddy || buddy->order != order || list_empty(&buddy->node)) + return NULL; + + return buddy; + +} + +static void __hyp_attach_page(struct hyp_pool *pool, + struct hyp_page *p) +{ + unsigned int order = p->order; + struct hyp_page *buddy; + + memset(hyp_page_to_virt(p), 0, PAGE_SIZE << p->order); + + /* + * Only the first struct hyp_page of a high-order page (otherwise known + * as the 'head') should have p->order set. The non-head pages should + * have p->order = HYP_NO_ORDER. Here @p may no longer be the head + * after coallescing, so make sure to mark it HYP_NO_ORDER proactively. + */ + p->order = HYP_NO_ORDER; + for (; (order + 1) < pool->max_order; order++) { + buddy = __find_buddy_avail(pool, p, order); + if (!buddy) + break; + + /* Take the buddy out of its list, and coallesce with @p */ + list_del_init(&buddy->node); + buddy->order = HYP_NO_ORDER; + p = min(p, buddy); + } + + /* Mark the new head, and insert it */ + p->order = order; + list_add_tail(&p->node, &pool->free_area[order]); +} + +static void hyp_attach_page(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); +} + +static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + struct hyp_page *buddy; + + list_del_init(&p->node); + while (p->order > order) { + /* + * The buddy of order n - 1 currently has HYP_NO_ORDER as it + * is covered by a higher-level page (whose head is @p). Use + * __find_buddy_nocheck() to find it and inject it in the + * free_list[n - 1], effectively splitting @p in half. + */ + p->order--; + buddy = __find_buddy_nocheck(pool, p, p->order); + buddy->order = p->order; + list_add_tail(&buddy->node, &pool->free_area[buddy->order]); + } + + return p; +} + +void hyp_put_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + if (hyp_page_ref_dec_and_test(p)) + hyp_attach_page(p); +} + +void hyp_get_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + hyp_page_ref_inc(p); +} + +void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) +{ + unsigned int i = order; + struct hyp_page *p; + + hyp_spin_lock(&pool->lock); + + /* Look for a high-enough-order page */ + while (i < pool->max_order && list_empty(&pool->free_area[i])) + i++; + if (i >= pool->max_order) { + hyp_spin_unlock(&pool->lock); + return NULL; + } + + /* Extract it from the tree at the right order */ + p = list_first_entry(&pool->free_area[i], struct hyp_page, node); + p = __hyp_extract_page(pool, p, order); + + hyp_spin_unlock(&pool->lock); + hyp_set_page_refcounted(p); + + return hyp_page_to_virt(p); +} + +int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages, + unsigned int reserved_pages) +{ + phys_addr_t phys = hyp_pfn_to_phys(pfn); + struct hyp_page *p; + int i; + + hyp_spin_lock_init(&pool->lock); + pool->max_order = min(MAX_ORDER, get_order(nr_pages << PAGE_SHIFT)); + for (i = 0; i < pool->max_order; i++) + INIT_LIST_HEAD(&pool->free_area[i]); + pool->range_start = phys; + pool->range_end = phys + (nr_pages << PAGE_SHIFT); + + /* Init the vmemmap portion */ + p = hyp_phys_to_page(phys); + memset(p, 0, sizeof(*p) * nr_pages); + for (i = 0; i < nr_pages; i++) { + p[i].pool = pool; + INIT_LIST_HEAD(&p[i].node); + } + + /* Attach the unused pages to the buddy tree */ + for (i = reserved_pages; i < nr_pages; i++) + __hyp_attach_page(pool, &p[i]); + + return 0; +} From patchwork Wed Mar 10 17:57:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE054C433E0 for ; Wed, 10 Mar 2021 18:03:16 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B5A4B64FB3 for ; Wed, 10 Mar 2021 18:03:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5A4B64FB3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=K1iVbiL8y4Y8D1zc4mUMQuiMBB//9GcyfmzjFIZapfw=; b=IDLmoNXXboBZiI M+jtrekvhFaretlS9Lp5ebXqN7Gb5s+ntt36hohZGAEUTooSR8kKfuaJEFVTcse5CUo8fVkbFhmFC YnbHIPC++uJn7bWhFekbbsCgugmW4PFuflM7gwMeXa5dExpzIQzi7TKe13YonyAVvNVYajr3GvM7Y crU/4Cv1H4iDbp3IfU7cc3NQY4RAZIGxrIi4JypTDryTMn2r45PRs4ro7A1aEGVL803otBUH3liOT mEAcwj8ypCyLGpmIZzqXFHMAieWbSYV0U6G4dN2+Io5xYt9j0pfhdcKLD0slFDv/Dalnw71i2Yexm NXuRnKv9PG73MYFnzQqg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK39G-007Pn6-UC; Wed, 10 Mar 2021 18:01:31 +0000 Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36F-007OEJ-O3 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:28 +0000 Received: by mail-qk1-x749.google.com with SMTP id k68so13395340qke.2 for ; Wed, 10 Mar 2021 09:58:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Nj4Lfx4RBIkW7BkT83ganNaPcX2b1UPE5UuHOUQIP/k=; b=Lz9VNJNblKnJkWT5K4VU7PyB0b9hN+o7HyyDCG02NgZinEFB3SYgmxmsahuuavWMAi NMSIyNG6PR+7/Dpgi4NqJJ9lMOzxnJUeVu+L2l42/1XOIHgwADvQNCgGozWki+5Udz3G Gb8pHWYK9vUkD4pDjyL8w+KTfG6gC71qKyRpM6M356+sZsrprfr30mKvKz2QJV1mIZC5 atJlNJXZIZxNSv6wMPebfLOfqhAD29Vj17J93VlDbzo02RhuVVlJ5fdknlbakfTvqKpg Caot5MfItt58v0EjXCYfi8xamJejyAMqpXDKR1I+BN54GVAE+f8AE32lOy9at7bE5faB YZiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Nj4Lfx4RBIkW7BkT83ganNaPcX2b1UPE5UuHOUQIP/k=; b=lkyycbInhxMIfNnkqGKPOFxUYKqdwJ6SyWVvOXK9lGqnqeHS1QB4Fvrx630cMEDhkb sY4+hVudnGaVRidiWWj+9k/hXxvi1xp1M2w/Q4NhS5dsm5p78cDiO0DQePUGuHsI09wO APGUwe6gupJmKzZge9xBqKjCnXsgD6SwCIPQTl/RoZp23qs6t0ypPN15F4u/RBb6L2fi RNstc/ZN8iRMaCzYLNUVnoGjCKoU+SHqGwf2M0jCyN99263lQd4InYAa4XRwMGxs6BPN 3c3Fa67GbkLNEe7EfiC8rjcA7ZfF/q7VKbGmM5YnKfvkv8tOw2AGCSDPmK3jp2SRsDnQ Ry4w== X-Gm-Message-State: AOAM530mpWE8NeDpMO9C9Cho5MsQ1D2k+RVPz54xvStLq8bEF6NNUAXB K/JDSPLSfkkGV4kZpLP5YbxXhWJs+Ffi X-Google-Smtp-Source: ABdhPJw5PfRtzS1h/8o/TGI7ehwR04Ncd7VgemXreET/UNEe/4aGwCj/3ct3FaQSNnY4iFj0MwIQXVnNH6gT X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:ad4:44ef:: with SMTP id p15mr4032818qvt.25.1615399101946; Wed, 10 Mar 2021 09:58:21 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:30 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-14-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 13/34] KVM: arm64: Enable access to sanitized CPU features at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175824_798653_4C94324D X-CRM114-Status: GOOD ( 20.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce the infrastructure in KVM enabling to copy CPU feature registers into EL2-owned data-structures, to allow reading sanitised values directly at EL2 in nVHE. Given that only a subset of these features are being read by the hypervisor, the ones that need to be copied are to be listed under together with the name of the nVHE variable that will hold the copy. While at it, introduce the first user of this infrastructure by implementing __flush_dcache_area at EL2, which needs arm64_ftr_reg_ctrel0. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/include/asm/kvm_cpufeature.h | 17 +++++++++++++++++ arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kernel/cpufeature.c | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/cache.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/cpufeature.c | 8 ++++++++ arch/arm64/kvm/sys_regs.c | 21 +++++++++++++++++++++ 8 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_cpufeature.h create mode 100644 arch/arm64/kvm/hyp/nvhe/cache.S create mode 100644 arch/arm64/kvm/hyp/nvhe/cpufeature.c diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 61177bac49fa..a85cea2cac57 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -607,6 +607,7 @@ void check_local_cpu_capabilities(void); u64 read_sanitised_ftr_reg(u32 id); u64 __read_sysreg_by_encoding(u32 sys_id); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst); static inline bool cpu_supports_mixed_endian_el0(void) { diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h new file mode 100644 index 000000000000..d34f85cba358 --- /dev/null +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include + +#ifndef KVM_HYP_CPU_FTR_REG +#if defined(__KVM_NVHE_HYPERVISOR__) +#define KVM_HYP_CPU_FTR_REG(id, name) extern struct arm64_ftr_reg name; +#else +#define KVM_HYP_CPU_FTR_REG(id, name) DECLARE_KVM_NVHE_SYM(name); +#endif +#endif + +KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 06ca4828005f..459ee557f87c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -751,9 +751,13 @@ void kvm_clr_pmu_events(u32 clr); void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu); void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); + +void setup_kvm_el2_caps(void); #else static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {} static inline void kvm_clr_pmu_events(u32 clr) {} + +static inline void setup_kvm_el2_caps(void) {} #endif void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 066030717a4c..f2d8b479ff74 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1154,6 +1154,18 @@ u64 read_sanitised_ftr_reg(u32 id) } EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst) +{ + struct arm64_ftr_reg *regp = get_arm64_ftr_reg(id); + + if (!regp) + return -EINVAL; + + memcpy(dst, regp, sizeof(*regp)); + + return 0; +} + #define read_sysreg_case(r) \ case r: val = read_sysreg_s(r); break; @@ -2773,6 +2785,7 @@ void __init setup_cpu_features(void) setup_system_capabilities(); setup_elf_hwcaps(arm64_elf_hwcaps); + setup_kvm_el2_caps(); if (system_supports_32bit_el0()) setup_elf_hwcaps(compat_elf_hwcaps); diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 6894a917f290..0033591553fc 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,8 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ + cache.o cpufeature.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/cache.S b/arch/arm64/kvm/hyp/nvhe/cache.S new file mode 100644 index 000000000000..36cef6915428 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cache.S @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Code copied from arch/arm64/mm/cache.S. + */ + +#include +#include +#include + +SYM_FUNC_START_PI(__flush_dcache_area) + dcache_by_line_op civac, sy, x0, x1, x2, x3 + ret +SYM_FUNC_END_PI(__flush_dcache_area) diff --git a/arch/arm64/kvm/hyp/nvhe/cpufeature.c b/arch/arm64/kvm/hyp/nvhe/cpufeature.c new file mode 100644 index 000000000000..a887508f996f --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cpufeature.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#define KVM_HYP_CPU_FTR_REG(id, name) struct arm64_ftr_reg name; +#include diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 4f2f1e3145de..84be93df52fa 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -2775,3 +2776,23 @@ void kvm_sys_reg_table_init(void) /* Clear all higher bits. */ cache_levels &= (1 << (i*3))-1; } + +#undef KVM_HYP_CPU_FTR_REG +#define KVM_HYP_CPU_FTR_REG(id, name) \ + { .sys_id = id, .dst = (struct arm64_ftr_reg *)&kvm_nvhe_sym(name) }, +struct __ftr_reg_copy_entry { + u32 sys_id; + struct arm64_ftr_reg *dst; +} hyp_ftr_regs[] __initdata = { + #include +}; + +void __init setup_kvm_el2_caps(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(hyp_ftr_regs); i++) { + WARN(copy_ftr_reg(hyp_ftr_regs[i].sys_id, hyp_ftr_regs[i].dst), + "%u feature register not found\n", hyp_ftr_regs[i].sys_id); + } +} From patchwork Wed Mar 10 17:57:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0260C433E0 for ; Wed, 10 Mar 2021 18:02:49 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1CCEF64F3A for ; Wed, 10 Mar 2021 18:02:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1CCEF64F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0a4WhGor+qH54PVEY+iMLI75WyS/nzsxK6cOcs4RVJM=; b=jRSC1aEBCmLPlK rNkJTm9S5n/FKNvx3Es0xuE77GGk2HTFyNUlXLivvmvFy4vD8xSkH9QsZJeqi6LWEB2y5u1E80/5n +yptc5QRFPKz14giKUtlbedvi+Koz8fGExn1uORJ0LUF8YVKrwxJZwjuji622WYAhUrgdQy9BWL9I rjUYqQ+3Q9cdEKxzN52hWx6OXL+QPHCuxL62wpOSllgOdrE5TnaaSiN3vqxSq2AH//53aypYPCj4M YO96LkJ9f9c5LlL9cCZZF5YSO69XuGL/AevtpUfYyIVbo3L5l/iNe/BKdAkEDD+UXIjTatBg4WJS+ OhoUJYMDLMdl3lHAWllQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK38x-007Pbb-62; Wed, 10 Mar 2021 18:01:11 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36H-007OEl-KT for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:28 +0000 Received: by mail-wr1-x44a.google.com with SMTP id n16so7242461wro.1 for ; Wed, 10 Mar 2021 09:58:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=84bkslXA7Qi7zlX0xHNbjR4XGFTrfSQqHB0l+SkjRTk=; b=gWR4E2H56ggbHXScQX04OGikwUKGFnpT1gmE6DUsaiqo17dy/goaVT6bwUgnhoi2Ju l6LWaN0AfBgHJvRU3Ntr0R6mbn6chazZBSXWjI/zmceN+GqIiziZZG6WvehBBHjFFiCv ff9Sb8VsJ6LT0Kj7avPebY6x8I2r8IJSx0TSinGSZuQBatEs2GvxWIim6kNPaBAilhzc 6wISeQvXokT2A2HYnczpuZZ3DMpgom1eOj/dEnrcQ2uXdttitkZONhv0pPSGLyhCFNNJ 2IpQR0s3Uwh8m+aS8s26mFq9vTzaNVZO59IeMkJaiD7fJ0Ial468WcZluMzWA4tW56/3 EcUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=84bkslXA7Qi7zlX0xHNbjR4XGFTrfSQqHB0l+SkjRTk=; b=JtM1MXrO+yugtwIqZu+J+6F6YuFjG/cqd8CdZ7Mh9LU0TIbgl1oFDfd6uBG7N4F+LE 7OCxX5izsJK1u/zRLo/lBqnGDdgFL/yM35oAurNBPfUx2ZUYl6QlO2z8G/d/iqK+mHr0 lEWm9Q94LpymzTNjuDTbLFoH4aLqSvNjU6ga4yLiddvm2tYD1qMiN2elaBcDMtwArHuD 6nZC/FhgJXwbApfuR/FeUo4vvUoQwT3cvA5QJrOqqgnrFO84Qh0ahenGIoLapORA/Jwb 7pZbDltnElDEmr7Wkf5DFywKosrHktZnHAdv8CIpQsZC9bWfNiXU/FJGOOI/g6J6SBEO GTew== X-Gm-Message-State: AOAM533+AeNo80DQgvR1yWzPt3r24totjsy6+o/OOBPN2rHBqP2wh0ya 8XOYXTahDx983zUA0pggFctIWiE2gFK1 X-Google-Smtp-Source: ABdhPJya6ohuYr2xnAIymmnPbC5dgEguUfV3trhqlVeC84z+zmtstjGngwQdmVgPcn0lHtJOrOlTur1n8Dx9 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a5d:4ec5:: with SMTP id s5mr4826061wrv.168.1615399104184; Wed, 10 Mar 2021 09:58:24 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:31 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-15-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 14/34] KVM: arm64: Factor out vector address calculation From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175825_852535_C69B40EC X-CRM114-Status: GOOD ( 11.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-map the guest vectors at EL2 when pKVM is enabled, refactor __kvm_vector_slot2idx() and kvm_init_vector_slot() to move all the address calculation logic in a static inline function. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 8 ++++++++ arch/arm64/kvm/arm.c | 9 +-------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 90873851f677..5c42ec023cc7 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -168,6 +168,14 @@ phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(void); +static inline void *__kvm_vector_slot2addr(void *base, + enum arm64_hyp_spectre_vector slot) +{ + int idx = slot - (slot != HYP_VECTOR_DIRECT); + + return base + (idx * SZ_2K); +} + struct kvm; #define kvm_flush_dcache_to_poc(a,l) __flush_dcache_area((a), (l)) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 3f8bcf8db036..26e573cdede3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1345,16 +1345,9 @@ static unsigned long nvhe_percpu_order(void) /* A lookup table holding the hypervisor VA for each vector slot */ static void *hyp_spectre_vector_selector[BP_HARDEN_EL2_SLOTS]; -static int __kvm_vector_slot2idx(enum arm64_hyp_spectre_vector slot) -{ - return slot - (slot != HYP_VECTOR_DIRECT); -} - static void kvm_init_vector_slot(void *base, enum arm64_hyp_spectre_vector slot) { - int idx = __kvm_vector_slot2idx(slot); - - hyp_spectre_vector_selector[slot] = base + (idx * SZ_2K); + hyp_spectre_vector_selector[slot] = __kvm_vector_slot2addr(base, slot); } static int kvm_init_vector_slots(void) From patchwork Wed Mar 10 17:57:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA0BEC433E0 for ; Wed, 10 Mar 2021 18:03:45 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5722264FB2 for ; Wed, 10 Mar 2021 18:03:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5722264FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xc52tDmPT7qzOqkgdu4Sck8BDbAGxZ83ywBezTYMIdc=; b=J6t2LsdNEfleUW RvGfAnBWuQEF2rCkL7gI47rGyGClkaFcdWXNpEx2q/rq/PjXdDACcZAG6JKOdvyDyIEE8vq9wJ6Nt xzFAJCkUonKL9WZdWakgq39p3JAX3zeb+ET0v09oFIssdLWm4r2b4KBQLDI/ZJUprsQdG7pqjlVBm IYeNELXBIFDsXnY5U2r7SE6cY4Izl5l5vRVeCH0Qx9ZaB58iXqbs5aNlDpgSH4Kx23mkfsT/2aLZn BQWeNBmJLscrlBmy5jSS7xWLFfbNLVTjggXYCJbP8XNcdOZ1S/DKplPl6yQjerEVN7OUxiCS1M5LM gcmlqbwLd6gQOtWpgFFA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK39j-007Q0L-3h; Wed, 10 Mar 2021 18:01:59 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36J-007OGR-9f for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:30 +0000 Received: by mail-wm1-x34a.google.com with SMTP id l16so1707493wmc.0 for ; Wed, 10 Mar 2021 09:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=7ThSG/RVLlh/DBRky56NDH1ajIaqy925wlZsTYsbYqw=; b=JEuYpJoMd4eciXRQGFuNKzbOmm8jGRXPcoKh/AeoZEfNeJ8pnd2yQE6JjHaW0xAyCa pitaJDGhocf+iS6Ix5s83gf0SdJs18Bt+3rQeBC1COJjUlE/pWqLmZGpYl4Y3Q008Xxz +KdxiZRuiY9IFxXtfadfJLEllXYm2dMSI1zsnCLjKK//Z68dcq7eoQguhGhmu8QBcbSo zzqFqyy0UmFXIphK/sSFENn1hVHMx3C6eW9CGE2mWa11W4GEHDVuzldvjqZ4ksKH3Gay m69aD9AQuVq3EYAST0FylU3iVNlhbgm1upS9OevLd1cZJCy4YWWktRZC6J9ZOco8M96R DPtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=7ThSG/RVLlh/DBRky56NDH1ajIaqy925wlZsTYsbYqw=; b=cVNRFl2vOrfZAaDvjM13j1EBh+zYncrvOheAzEzrXKGoUT8niahUx2QQr9WYGq97lq kVwfiq5aS13315gZFrFr0yCup/O/LtZORKfUTlg+4HsWjBZpTagI83hXrqI2K0SdiZKo uBgPt+Cx65ljcSAE/u1mufOTrs7VLnzFrEBnZvF1e7yVnhWtLR8EI6XYHQnon59/0Cmu MZs3/POAhtNZ4vidSyR6HgyMZwE8X3Vd+mnEeEpJxYEhLBf9XrE6KE1n5qM5fOjPCvnM VU6T0emDNrwGaiVcGJOgd6zFm00P9Bx8nhdE+FOvpbDwcbHLBBJdfTT73Zc1f7mUFEG/ HYGg== X-Gm-Message-State: AOAM531o+6tOKyhaNrGfqpHvpWsn6b9T+K5/VQnzw+Nlup1btHlsnR+W 7TpTPUflag660Uqlf+80FsGypN2feJ0r X-Google-Smtp-Source: ABdhPJyE9VwVGP/miD0PZ3GZDCYZcaB5M6aMqWesnLSzh0vu9I8m1MWCHPX7MchrOCygw+09qd4prsL24INh X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:214d:: with SMTP id v13mr4574201wml.7.1615399106500; Wed, 10 Mar 2021 09:58:26 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:32 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-16-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 15/34] arm64: asm: Provide set_sctlr_el2 macro From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175827_777441_830868E6 X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will soon need to turn the EL2 stage 1 MMU on and off in nVHE protected mode, so refactor the set_sctlr_el1 macro to make it usable for that purpose. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index ca31594d3d6c..fb651c1f26e9 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -676,11 +676,11 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU .endm /* - * Set SCTLR_EL1 to the passed value, and invalidate the local icache + * Set SCTLR_ELx to the @reg value, and invalidate the local icache * in the process. This is called when setting the MMU on. */ -.macro set_sctlr_el1, reg - msr sctlr_el1, \reg +.macro set_sctlr, sreg, reg + msr \sreg, \reg isb /* * Invalidate the local I-cache so that any instructions fetched @@ -692,6 +692,14 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU isb .endm +.macro set_sctlr_el1, reg + set_sctlr sctlr_el1, \reg +.endm + +.macro set_sctlr_el2, reg + set_sctlr sctlr_el2, \reg +.endm + /* * Check whether to yield to another runnable task from kernel mode NEON code * (which runs with preemption disabled). From patchwork Wed Mar 10 17:57:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A4EFC433E0 for ; Wed, 10 Mar 2021 18:04:26 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7039C64FB3 for ; Wed, 10 Mar 2021 18:04:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7039C64FB3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=imUGPEQPBsBbccfIn9tK0LgiX7odnIdYSdrV9+uIfxs=; b=XcRjL+7zhHhvO4 DAQitAD3J3061woGsqKupipZLFfEfosUP6+sxwXSYG8YiiE5/s8iLcX+BjJ1xSlKhzRoyKn7Lj+j+ XXgtaY0QeCFWjqYNtui7Wi5JBbepGAuJeVtxiLFevUyDqaggsY+VPYgnSp94QYim8wN6VHNvptWIZ kaluqV8XNoNoWvOThTzmIrYFp9aZ5+KnoXFpmQJAG5tW2S+wlMALPyoSpxNZAOvbJFQKlBX7w1Kgl S4VHJEpbUxBr26D4y7RcJYGu/W64NjRD0sE/oQJck/m7L9+iX2dXkpFcXS/yqFaA+xKWZqrTnsZBN VxZkY6omwYg9TFuZ5l4g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3AG-007QHz-Gf; Wed, 10 Mar 2021 18:02:32 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36M-007OHY-E3 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:33 +0000 Received: by mail-wr1-x449.google.com with SMTP id v13so8319591wrs.21 for ; Wed, 10 Mar 2021 09:58:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=DvRmrPCKZ4Yr0KwMOlajXkwPvwPmP/BZHVMD3/FAGfs=; b=Q3jntZ19pppYHhY0CroKuZHHBibjeRNH94drFq/3o6nRnCLR1BLP99YiJr8o2+/+WY F06cWspeSHtMClHYvbY2rNxKYRpZKApOpGyVmwTTAkVzwh/lBLPmkBMv9ZfV0EvF7yhs t/V2VCgPMtrGfjRaoWxIY1NqFt999t6abAmrpidzeli5qvZPLLwD+xQQgfyylT/sxbrp EAexnrknJ2h9Uj5EnUga8rWkybmy2X9wlDvHfFqHFbuiAutlHmnF/rmXKZSsjFEEFDLw srvI4UwM2yesoPeT3Fa6YnEDlO8xyHn/jSA23Kv5TycGUXeR4Dh3zbqU8FCTkUbLCZQT HSNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=DvRmrPCKZ4Yr0KwMOlajXkwPvwPmP/BZHVMD3/FAGfs=; b=NuNaw10YWKkRxoNXxv9DK0yob7TCu+VuhJUJ8a4sEWEHHOZNnC02x0xMcmMqVNErra WcczZI+T2YuG0MX4STH0wRCitlK3CMfKxsOKbXrsyCoUZEC+IUBovofiBbFc/7GEgdj1 vWGxtyPr1iRAKhyAmjFZaO8QdmN7x1YVEwBPOgiJ83DvUryCwzE1atqm09en+G3ZSqle nX1/mCKAK4RczKh3fdZC2nDRiOkXQYIWTwK+sPK8hW9vPhiY0YdYQNShhkBof8s4rQbU igw3UebHaosbjHbzCyyr+SlytWgWFSHNx4fAVlz6a4swx8VE5769hjitenedLLoiKXdr AcRg== X-Gm-Message-State: AOAM530BJ+492CEuGjGYsQEKXOv8T02Ej0e3pTtfl1yxM+3rx7mG1qoQ HBgKATUUCEIq81AwCrTvbpou7BWahGzt X-Google-Smtp-Source: ABdhPJw6sQFF8J+n7nJ001abEEuBgHqUZrqni1u2TINpDQDprM6Egl9cPYCEeKlo3Poch8UtgplLSGEl84jK X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c18e:: with SMTP id y14mr1333654wmi.1.1615399108535; Wed, 10 Mar 2021 09:58:28 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:33 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-17-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 16/34] KVM: arm64: Prepare the creation of s1 mappings at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175830_792428_40A4E8B4 X-CRM114-Status: GOOD ( 31.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the EL2 code needs the ability to create and manage its own page-table. To do so, introduce a new set of hypercalls to bootstrap a memory management system at EL2. This leads to the following boot flow in nVHE Protected mode: 1. the host allocates memory for the hypervisor very early on, using the memblock API; 2. the host creates a set of stage 1 page-table for EL2, installs the EL2 vectors, and issues the __pkvm_init hypercall; 3. during __pkvm_init, the hypervisor re-creates its stage 1 page-table and stores it in the memory pool provided by the host; 4. the hypervisor then extends its stage 1 mappings to include a vmemmap in the EL2 VA space, hence allowing to use the buddy allocator introduced in a previous patch; 5. the hypervisor jumps back in the idmap page, switches from the host-provided page-table to the new one, and wraps up its initialization by enabling the new allocator, before returning to the host. 6. the host can free the now unused page-table created for EL2, and will now need to issue hypercalls to make changes to the EL2 stage 1 mappings instead of modifying them directly. Note that for the sake of simplifying the review, this patch focuses on the hypervisor side of things. In other words, this only implements the new hypercalls, but does not make use of them from the host yet. The host-side changes will follow in a subsequent patch. Credits to Will for __pkvm_init_switch_pgd. Co-authored-by: Will Deacon Signed-off-by: Will Deacon Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_host.h | 7 + arch/arm64/include/asm/kvm_hyp.h | 8 ++ arch/arm64/include/asm/kvm_pgtable.h | 2 + arch/arm64/kernel/image-vars.h | 16 +++ arch/arm64/kvm/hyp/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/mm.h | 71 ++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 27 ++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 49 +++++++ arch/arm64/kvm/hyp/nvhe/mm.c | 173 +++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 197 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 2 - arch/arm64/kvm/hyp/reserved_mem.c | 92 +++++++++++++ arch/arm64/mm/init.c | 3 + 15 files changed, 652 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mm.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mm.c create mode 100644 arch/arm64/kvm/hyp/nvhe/setup.c create mode 100644 arch/arm64/kvm/hyp/reserved_mem.c diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 22d933e9b59e..db20a9477870 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -57,6 +57,10 @@ #define __KVM_HOST_SMCCC_FUNC___kvm_get_mdcr_el2 12 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs 13 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs 14 +#define __KVM_HOST_SMCCC_FUNC___pkvm_init 15 +#define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16 +#define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 +#define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 459ee557f87c..b9d45a1f8538 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -781,5 +781,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features)) int kvm_trng_call(struct kvm_vcpu *vcpu); +#ifdef CONFIG_KVM +extern phys_addr_t hyp_mem_base; +extern phys_addr_t hyp_mem_size; +void __init kvm_hyp_reserve(void); +#else +static inline void kvm_hyp_reserve(void) { } +#endif #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index c0450828378b..ae55351b99a4 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -100,4 +100,12 @@ void __noreturn hyp_panic(void); void __noreturn __hyp_do_panic(bool restore_host, u64 spsr, u64 elr, u64 par); #endif +#ifdef __KVM_NVHE_HYPERVISOR__ +void __pkvm_init_switch_pgd(phys_addr_t phys, unsigned long size, + phys_addr_t pgd, void *sp, void *cont_fn); +int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, + unsigned long *per_cpu_base, u32 hyp_va_bits); +void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); +#endif + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index bbe840e430cb..bf7a3cc49420 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -11,6 +11,8 @@ #include #include +#define KVM_PGTABLE_MAX_LEVELS 4U + typedef u64 kvm_pte_t; /** diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 4eb7a15c8b60..940c378fa837 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -115,6 +115,22 @@ KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy); KVM_NVHE_ALIAS_HYP(__memset, __pi_memset); #endif +/* Kernel memory sections */ +KVM_NVHE_ALIAS(__start_rodata); +KVM_NVHE_ALIAS(__end_rodata); +KVM_NVHE_ALIAS(__bss_start); +KVM_NVHE_ALIAS(__bss_stop); + +/* Hyp memory sections */ +KVM_NVHE_ALIAS(__hyp_idmap_text_start); +KVM_NVHE_ALIAS(__hyp_idmap_text_end); +KVM_NVHE_ALIAS(__hyp_text_start); +KVM_NVHE_ALIAS(__hyp_text_end); +KVM_NVHE_ALIAS(__hyp_bss_start); +KVM_NVHE_ALIAS(__hyp_bss_end); +KVM_NVHE_ALIAS(__hyp_rodata_start); +KVM_NVHE_ALIAS(__hyp_rodata_end); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile index 687598e41b21..b726332eec49 100644 --- a/arch/arm64/kvm/hyp/Makefile +++ b/arch/arm64/kvm/hyp/Makefile @@ -10,4 +10,4 @@ subdir-ccflags-y := -I$(incdir) \ -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o +obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o reserved_mem.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h new file mode 100644 index 000000000000..ac0f7fcffd08 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MM_H +#define __KVM_HYP_MM_H + +#include +#include +#include +#include + +#include +#include + +#define HYP_MEMBLOCK_REGIONS 128 +extern struct memblock_region kvm_nvhe_sym(hyp_memory)[]; +extern unsigned int kvm_nvhe_sym(hyp_memblock_nr); +extern struct kvm_pgtable pkvm_pgtable; +extern hyp_spinlock_t pkvm_pgd_lock; +extern struct hyp_pool hpool; +extern u64 __io_map_base; + +int hyp_create_idmap(u32 hyp_va_bits); +int hyp_map_vectors(void); +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); +int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); +int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +int __pkvm_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, enum kvm_pgtable_prot prot); +unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot); + +static inline void hyp_vmemmap_range(phys_addr_t phys, unsigned long size, + unsigned long *start, unsigned long *end) +{ + unsigned long nr_pages = size >> PAGE_SHIFT; + struct hyp_page *p = hyp_phys_to_page(phys); + + *start = (unsigned long)p; + *end = *start + nr_pages * sizeof(struct hyp_page); + *start = ALIGN_DOWN(*start, PAGE_SIZE); + *end = ALIGN(*end, PAGE_SIZE); +} + +static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) +{ + unsigned long total = 0, i; + + /* Provision the worst case scenario */ + for (i = 0; i < KVM_PGTABLE_MAX_LEVELS; i++) { + nr_pages = DIV_ROUND_UP(nr_pages, PTRS_PER_PTE); + total += nr_pages; + } + + return total; +} + +static inline unsigned long hyp_s1_pgtable_pages(void) +{ + unsigned long res = 0, i; + + /* Cover all of memory with page-granularity */ + for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) { + struct memblock_region *reg = &kvm_nvhe_sym(hyp_memory)[i]; + res += __hyp_pgtable_max_pages(reg->size >> PAGE_SHIFT); + } + + /* Allow 1 GiB for private mappings */ + res += __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); + + return res; +} +#endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 0033591553fc..e204ea77ab27 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -14,9 +14,9 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o + cache.o cpufeature.o setup.o mm.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ - ../fpsimd.o ../hyp-entry.o ../exception.o + ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) ## diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index c631e29fb001..a2b8b6a84cbd 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -244,4 +244,31 @@ alternative_else_nop_endif SYM_CODE_END(__kvm_handle_stub_hvc) +SYM_FUNC_START(__pkvm_init_switch_pgd) + /* Turn the MMU off */ + pre_disable_mmu_workaround + mrs x2, sctlr_el2 + bic x3, x2, #SCTLR_ELx_M + msr sctlr_el2, x3 + isb + + tlbi alle2 + + /* Install the new pgtables */ + ldr x3, [x0, #NVHE_INIT_PGD_PA] + phys_to_ttbr x4, x3 +alternative_if ARM64_HAS_CNP + orr x4, x4, #TTBR_CNP_BIT +alternative_else_nop_endif + msr ttbr0_el2, x4 + + /* Set the new stack pointer */ + ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] + mov sp, x0 + + /* And turn the MMU back on! */ + set_sctlr_el2 x2 + ret x1 +SYM_FUNC_END(__pkvm_init_switch_pgd) + .popsection diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f012f8665ecc..ae6503c9be15 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -6,12 +6,14 @@ #include +#include #include #include #include #include #include +#include #include DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); @@ -106,6 +108,49 @@ static void handle___vgic_v3_restore_aprs(struct kvm_cpu_context *host_ctxt) __vgic_v3_restore_aprs(kern_hyp_va(cpu_if)); } +static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, nr_cpus, host_ctxt, 3); + DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 4); + DECLARE_REG(u32, hyp_va_bits, host_ctxt, 5); + + /* + * __pkvm_init() will return only if an error occurred, otherwise it + * will tail-call in __pkvm_init_finalise() which will have to deal + * with the host context directly. + */ + cpu_reg(host_ctxt, 1) = __pkvm_init(phys, size, nr_cpus, per_cpu_base, + hyp_va_bits); +} + +static void handle___pkvm_cpu_set_vector(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(enum arm64_hyp_spectre_vector, slot, host_ctxt, 1); + + cpu_reg(host_ctxt, 1) = pkvm_cpu_set_vector(slot); +} + +static void handle___pkvm_create_mappings(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(unsigned long, start, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, phys, host_ctxt, 3); + DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 4); + + cpu_reg(host_ctxt, 1) = __pkvm_create_mappings(start, size, phys, prot); +} + +static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(size_t, size, host_ctxt, 2); + DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); + + cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); +} + typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -125,6 +170,10 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_get_mdcr_el2), HANDLE_FUNC(__vgic_v3_save_aprs), HANDLE_FUNC(__vgic_v3_restore_aprs), + HANDLE_FUNC(__pkvm_init), + HANDLE_FUNC(__pkvm_cpu_set_vector), + HANDLE_FUNC(__pkvm_create_mappings), + HANDLE_FUNC(__pkvm_create_private_mapping), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c new file mode 100644 index 000000000000..a8efdf0f9003 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct kvm_pgtable pkvm_pgtable; +hyp_spinlock_t pkvm_pgd_lock; +u64 __io_map_base; + +struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS]; +unsigned int hyp_memblock_nr; + +int __pkvm_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, enum kvm_pgtable_prot prot) +{ + int err; + + hyp_spin_lock(&pkvm_pgd_lock); + err = kvm_pgtable_hyp_map(&pkvm_pgtable, start, size, phys, prot); + hyp_spin_unlock(&pkvm_pgd_lock); + + return err; +} + +unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot) +{ + unsigned long addr; + int err; + + hyp_spin_lock(&pkvm_pgd_lock); + + size = PAGE_ALIGN(size + offset_in_page(phys)); + addr = __io_map_base; + __io_map_base += size; + + /* Are we overflowing on the vmemmap ? */ + if (__io_map_base > __hyp_vmemmap) { + __io_map_base -= size; + addr = (unsigned long)ERR_PTR(-ENOMEM); + goto out; + } + + err = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, size, phys, prot); + if (err) { + addr = (unsigned long)ERR_PTR(err); + goto out; + } + + addr = addr + offset_in_page(phys); +out: + hyp_spin_unlock(&pkvm_pgd_lock); + + return addr; +} + +int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) +{ + unsigned long start = (unsigned long)from; + unsigned long end = (unsigned long)to; + unsigned long virt_addr; + phys_addr_t phys; + + start = start & PAGE_MASK; + end = PAGE_ALIGN(end); + + for (virt_addr = start; virt_addr < end; virt_addr += PAGE_SIZE) { + int err; + + phys = hyp_virt_to_phys((void *)virt_addr); + err = __pkvm_create_mappings(virt_addr, PAGE_SIZE, phys, prot); + if (err) + return err; + } + + return 0; +} + +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back) +{ + unsigned long start, end; + + hyp_vmemmap_range(phys, size, &start, &end); + + return __pkvm_create_mappings(start, end - start, back, PAGE_HYP); +} + +static void *__hyp_bp_vect_base; +int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot) +{ + void *vector; + + switch (slot) { + case HYP_VECTOR_DIRECT: { + vector = __kvm_hyp_vector; + break; + } + case HYP_VECTOR_SPECTRE_DIRECT: { + vector = __bp_harden_hyp_vecs; + break; + } + case HYP_VECTOR_INDIRECT: + case HYP_VECTOR_SPECTRE_INDIRECT: { + vector = (void *)__hyp_bp_vect_base; + break; + } + default: + return -EINVAL; + } + + vector = __kvm_vector_slot2addr(vector, slot); + *this_cpu_ptr(&kvm_hyp_vector) = (unsigned long)vector; + + return 0; +} + +int hyp_map_vectors(void) +{ + phys_addr_t phys; + void *bp_base; + + if (!cpus_have_const_cap(ARM64_SPECTRE_V3A)) + return 0; + + phys = __hyp_pa(__bp_harden_hyp_vecs); + bp_base = (void *)__pkvm_create_private_mapping(phys, + __BP_HARDEN_HYP_VECS_SZ, + PAGE_HYP_EXEC); + if (IS_ERR_OR_NULL(bp_base)) + return PTR_ERR(bp_base); + + __hyp_bp_vect_base = bp_base; + + return 0; +} + +int hyp_create_idmap(u32 hyp_va_bits) +{ + unsigned long start, end; + + start = hyp_virt_to_phys((void *)__hyp_idmap_text_start); + start = ALIGN_DOWN(start, PAGE_SIZE); + + end = hyp_virt_to_phys((void *)__hyp_idmap_text_end); + end = ALIGN(end, PAGE_SIZE); + + /* + * One half of the VA space is reserved to linearly map portions of + * memory -- see va_layout.c for more details. The other half of the VA + * space contains the trampoline page, and needs some care. Split that + * second half in two and find the quarter of VA space not conflicting + * with the idmap to place the IOs and the vmemmap. IOs use the lower + * half of the quarter and the vmemmap the upper half. + */ + __io_map_base = start & BIT(hyp_va_bits - 2); + __io_map_base ^= BIT(hyp_va_bits - 2); + __hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3); + + return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c new file mode 100644 index 000000000000..1e8bcd8b0299 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct hyp_pool hpool; +struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; +unsigned long hyp_nr_cpus; + +#define hyp_percpu_size ((unsigned long)__per_cpu_end - \ + (unsigned long)__per_cpu_start) + +static void *vmemmap_base; +static void *hyp_pgt_base; + +static int divide_memory_pool(void *virt, unsigned long size) +{ + unsigned long vstart, vend, nr_pages; + + hyp_early_alloc_init(virt, size); + + hyp_vmemmap_range(__hyp_pa(virt), size, &vstart, &vend); + nr_pages = (vend - vstart) >> PAGE_SHIFT; + vmemmap_base = hyp_early_alloc_contig(nr_pages); + if (!vmemmap_base) + return -ENOMEM; + + nr_pages = hyp_s1_pgtable_pages(); + hyp_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!hyp_pgt_base) + return -ENOMEM; + + return 0; +} + +static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, + unsigned long *per_cpu_base, + u32 hyp_va_bits) +{ + void *start, *end, *virt = hyp_phys_to_virt(phys); + unsigned long pgt_size = hyp_s1_pgtable_pages() << PAGE_SHIFT; + int ret, i; + + /* Recreate the hyp page-table using the early page allocator */ + hyp_early_alloc_init(hyp_pgt_base, pgt_size); + ret = kvm_pgtable_hyp_init(&pkvm_pgtable, hyp_va_bits, + &hyp_early_alloc_mm_ops); + if (ret) + return ret; + + ret = hyp_create_idmap(hyp_va_bits); + if (ret) + return ret; + + ret = hyp_map_vectors(); + if (ret) + return ret; + + ret = hyp_back_vmemmap(phys, size, hyp_virt_to_phys(vmemmap_base)); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_text_start, __hyp_text_end, PAGE_HYP_EXEC); + if (ret) + return ret; + + ret = pkvm_create_mappings(__start_rodata, __end_rodata, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_rodata_start, __hyp_rodata_end, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_bss_start, __hyp_bss_end, PAGE_HYP); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_bss_end, __bss_stop, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(virt, virt + size, PAGE_HYP); + if (ret) + return ret; + + for (i = 0; i < hyp_nr_cpus; i++) { + start = (void *)kern_hyp_va(per_cpu_base[i]); + end = start + PAGE_ALIGN(hyp_percpu_size); + ret = pkvm_create_mappings(start, end, PAGE_HYP); + if (ret) + return ret; + + end = (void *)per_cpu_ptr(&kvm_init_params, i)->stack_hyp_va; + start = end - PAGE_SIZE; + ret = pkvm_create_mappings(start, end, PAGE_HYP); + if (ret) + return ret; + } + + return 0; +} + +static void update_nvhe_init_params(void) +{ + struct kvm_nvhe_init_params *params; + unsigned long i; + + for (i = 0; i < hyp_nr_cpus; i++) { + params = per_cpu_ptr(&kvm_init_params, i); + params->pgd_pa = __hyp_pa(pkvm_pgtable.pgd); + __flush_dcache_area(params, sizeof(*params)); + } +} + +static void *hyp_zalloc_hyp_page(void *arg) +{ + return hyp_alloc_pages(&hpool, 0); +} + +void __noreturn __pkvm_init_finalise(void) +{ + struct kvm_host_data *host_data = this_cpu_ptr(&kvm_host_data); + struct kvm_cpu_context *host_ctxt = &host_data->host_ctxt; + unsigned long nr_pages, reserved_pages, pfn; + int ret; + + /* Now that the vmemmap is backed, install the full-fledged allocator */ + pfn = hyp_virt_to_pfn(hyp_pgt_base); + nr_pages = hyp_s1_pgtable_pages(); + reserved_pages = hyp_early_alloc_nr_used_pages(); + ret = hyp_pool_init(&hpool, pfn, nr_pages, reserved_pages); + if (ret) + goto out; + + pkvm_pgtable_mm_ops = (struct kvm_pgtable_mm_ops) { + .zalloc_page = hyp_zalloc_hyp_page, + .phys_to_virt = hyp_phys_to_virt, + .virt_to_phys = hyp_virt_to_phys, + .get_page = hyp_get_page, + .put_page = hyp_put_page, + }; + pkvm_pgtable.mm_ops = &pkvm_pgtable_mm_ops; + +out: + /* + * We tail-called to here from handle___pkvm_init() and will not return, + * so make sure to propagate the return value to the host. + */ + cpu_reg(host_ctxt, 1) = ret; + + __host_enter(host_ctxt); +} + +int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, + unsigned long *per_cpu_base, u32 hyp_va_bits) +{ + struct kvm_nvhe_init_params *params; + void *virt = hyp_phys_to_virt(phys); + void (*fn)(phys_addr_t params_pa, void *finalize_fn_va); + int ret; + + if (!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size)) + return -EINVAL; + + hyp_spin_lock_init(&pkvm_pgd_lock); + hyp_nr_cpus = nr_cpus; + + ret = divide_memory_pool(virt, size); + if (ret) + return ret; + + ret = recreate_hyp_mappings(phys, size, per_cpu_base, hyp_va_bits); + if (ret) + return ret; + + update_nvhe_init_params(); + + /* Jump in the idmap page to switch to the new page-tables */ + params = this_cpu_ptr(&kvm_init_params); + fn = (typeof(fn))__hyp_pa(__pkvm_init_switch_pgd); + fn(__hyp_pa(params), __pkvm_init_finalise); + + unreachable(); +} diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b975a67d1f85..7ce0969203e8 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -10,8 +10,6 @@ #include #include -#define KVM_PGTABLE_MAX_LEVELS 4U - #define KVM_PTE_VALID BIT(0) #define KVM_PTE_TYPE BIT(1) diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c new file mode 100644 index 000000000000..9bc6a6d27904 --- /dev/null +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include +#include + +#include + +#include +#include + +static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory); +static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr); + +phys_addr_t hyp_mem_base; +phys_addr_t hyp_mem_size; + +static int __init register_memblock_regions(void) +{ + struct memblock_region *reg; + + for_each_mem_region(reg) { + if (*hyp_memblock_nr_ptr >= HYP_MEMBLOCK_REGIONS) + return -ENOMEM; + + hyp_memory[*hyp_memblock_nr_ptr] = *reg; + (*hyp_memblock_nr_ptr)++; + } + + return 0; +} + +void __init kvm_hyp_reserve(void) +{ + u64 nr_pages, prev, hyp_mem_pages = 0; + int ret; + + if (!is_hyp_mode_available() || is_kernel_in_hyp_mode()) + return; + + if (kvm_get_mode() != KVM_MODE_PROTECTED) + return; + + ret = register_memblock_regions(); + if (ret) { + *hyp_memblock_nr_ptr = 0; + kvm_err("Failed to register hyp memblocks: %d\n", ret); + return; + } + + hyp_mem_pages += hyp_s1_pgtable_pages(); + + /* + * The hyp_vmemmap needs to be backed by pages, but these pages + * themselves need to be present in the vmemmap, so compute the number + * of pages needed by looking for a fixed point. + */ + nr_pages = 0; + do { + prev = nr_pages; + nr_pages = hyp_mem_pages + prev; + nr_pages = DIV_ROUND_UP(nr_pages * sizeof(struct hyp_page), PAGE_SIZE); + nr_pages += __hyp_pgtable_max_pages(nr_pages); + } while (nr_pages != prev); + hyp_mem_pages += nr_pages; + + /* + * Try to allocate a PMD-aligned region to reduce TLB pressure once + * this is unmapped from the host stage-2, and fallback to PAGE_SIZE. + */ + hyp_mem_size = hyp_mem_pages << PAGE_SHIFT; + hyp_mem_base = memblock_find_in_range(0, memblock_end_of_DRAM(), + ALIGN(hyp_mem_size, PMD_SIZE), + PMD_SIZE); + if (!hyp_mem_base) + hyp_mem_base = memblock_find_in_range(0, memblock_end_of_DRAM(), + hyp_mem_size, PAGE_SIZE); + else + hyp_mem_size = ALIGN(hyp_mem_size, PMD_SIZE); + + if (!hyp_mem_base) { + kvm_err("Failed to reserve hyp memory\n"); + return; + } + memblock_reserve(hyp_mem_base, hyp_mem_size); + + kvm_info("Reserved %lld MiB at 0x%llx\n", hyp_mem_size >> 20, + hyp_mem_base); +} diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 0ace5e68efba..2b1b41b1ff9b 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -429,6 +430,8 @@ void __init bootmem_init(void) dma_pernuma_cma_reserve(); + kvm_hyp_reserve(); + /* * sparse_init() tries to allocate memory from memblock, so must be * done after the fixed reservations From patchwork Wed Mar 10 17:57:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 210AEC433DB for ; Wed, 10 Mar 2021 18:04:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 573FA64FB2 for ; Wed, 10 Mar 2021 18:04:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 573FA64FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8ymDul3Z2KwPA/daZOoWvEQze+TklF6Dx7S0df9sEec=; b=kQWnUSz9ZFmzQW f78qeY+dGBjGruO7OHjB2xBcyxiPB+uzh8y8iUogPeNGPqg7ZcwXrRMaSeZqjY+QmmD2dWO63kSwH HMtzrY06FXQZLYSh9jDwv0sKcd1YDURHBSEiAfbC/BK+0hMRTH/ZPsgFOCAx/W/8YXDSuz9t8+rlW dW12ez9UR7D+J3/1+7RcQji+V2CrSI4hdablCRMcM4ViiytTNYBC4Kz49ltFV/ITkoM5HVPd7B6Q2 TxonvPVEzO/Zcxhm6EsUplm9TZCPh09z1UmL1VG3kJC1sI+6heMKtXP50iJnUiRAgQer8233eYeTd j/xiSV9g5lB1qJBYAcGg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Ab-007QTU-7G; Wed, 10 Mar 2021 18:02:53 +0000 Received: from mail-ej1-x64a.google.com ([2a00:1450:4864:20::64a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36O-007OIK-Ew for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:34 +0000 Received: by mail-ej1-x64a.google.com with SMTP id v27so2807408ejq.0 for ; Wed, 10 Mar 2021 09:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MDAOMBrujHgZpnE20qSGm1QQUtAE8AkHofkpYK5DKw8=; b=SPKpXNIElrr/kccAA33eiC02lZd/YQo1wMyMgbwKipLQ+IprzwGijG8z/UEp/8m9OC Veedhj366bN5pJ475K//bVmQiIU7RKOllPbey+Ghj/nPBpAKLcxSo3gg6ce2CmkbPcfa 4cO2C4TLjUn7RmzR90g9nGZMaYv5Hb5DMkVAlioaUgTg+QJ7OKoyBeYQf24G90aqnIOA W4z1DhqjHLAQiZH2cITd8qvnmA90p57cPxzcnuSlzTQx4C58FJ28jGV39gFwlPix5G0I yk0GvWB8qWNv9qfZblGybI/B5paabO4TAT+2uEIiDcSjGJ+TRrbqj/eiAyqOmw8oDpZV i4Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MDAOMBrujHgZpnE20qSGm1QQUtAE8AkHofkpYK5DKw8=; b=MTbE3RQrSWxThPmOpfTEWxoqDJZ9xSxiyRcAyCHyLZIIXeo+apmvxuVzoXUmWNdCmW ssy2k/iEb0aiRKmGNtE8dbVt+JytarbbSO432eHl8Cr9lAByxM/Cm1xWtlk54M0GvMer NeWTfY3v+B1sk+5GmrSmGAP1pvpkGRzQ1eMykwkfEgToxPWJVLFe3+SuZQTR31lv7q9M +Icvb/X8o4F//FDMaMkmCEtU3pGBpcx+aiLg4dXSI9OB9boS3/iHaHAuDpVXsQAylPLS +6Fc6R4zUUrIKPNG8tqOQhPuLXJpzIX1vG1eufG52fDVoJqkWJ42/pd+8wj85Y3uHNYj 4xug== X-Gm-Message-State: AOAM532jX31yA8X8Y0DzaPmT0D4j9ooSjtnDQNkMM6awIw0RbqgQwp5F 9Lro1JVG3D/H0yg4U0LR9s8w/whAl7J6 X-Google-Smtp-Source: ABdhPJxtAtrUYpq8wz+am4H4wifwHgXtajAQPy3lRNGJJ51Ane5m5Z0baC9vyZnXI1Mel9wGB6uDORVXo5Bt X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a17:906:607:: with SMTP id s7mr4814258ejb.495.1615399110936; Wed, 10 Mar 2021 09:58:30 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:34 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-18-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 17/34] KVM: arm64: Elevate hypervisor mappings creation at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175832_731657_228BE572 X-CRM114-Status: GOOD ( 23.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Previous commits have introduced infrastructure to enable the EL2 code to manage its own stage 1 mappings. However, this was preliminary work, and none of it is currently in use. Put all of this together by elevating the mapping creation at EL2 when memory protection is enabled. In this case, the host kernel running at EL1 still creates _temporary_ EL2 mappings, only used while initializing the hypervisor, but frees them right after. As such, all calls to create_hyp_mappings() after kvm init has finished turn into hypercalls, as the host now has no 'legal' way to modify the hypevisor page tables directly. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_mmu.h | 2 +- arch/arm64/kvm/arm.c | 87 +++++++++++++++++++++++++++++--- arch/arm64/kvm/mmu.c | 43 ++++++++++++++-- 3 files changed, 120 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 5c42ec023cc7..ce02a4052dcf 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -166,7 +166,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); -int kvm_mmu_init(void); +int kvm_mmu_init(u32 *hyp_va_bits); static inline void *__kvm_vector_slot2addr(void *base, enum arm64_hyp_spectre_vector slot) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 26e573cdede3..7d62211109d9 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1421,7 +1421,7 @@ static void cpu_prepare_hyp_mode(int cpu) kvm_flush_dcache_to_poc(params, sizeof(*params)); } -static void cpu_init_hyp_mode(void) +static void hyp_install_host_vector(void) { struct kvm_nvhe_init_params *params; struct arm_smccc_res res; @@ -1439,6 +1439,11 @@ static void cpu_init_hyp_mode(void) params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); +} + +static void cpu_init_hyp_mode(void) +{ + hyp_install_host_vector(); /* * Disabling SSBD on a non-VHE system requires us to enable SSBS @@ -1481,7 +1486,10 @@ static void cpu_set_hyp_vector(void) struct bp_hardening_data *data = this_cpu_ptr(&bp_hardening_data); void *vector = hyp_spectre_vector_selector[data->slot]; - *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + if (!is_protected_kvm_enabled()) + *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + else + kvm_call_hyp_nvhe(__pkvm_cpu_set_vector, data->slot); } static void cpu_hyp_reinit(void) @@ -1489,13 +1497,14 @@ static void cpu_hyp_reinit(void) kvm_init_host_cpu_context(&this_cpu_ptr_hyp_sym(kvm_host_data)->host_ctxt); cpu_hyp_reset(); - cpu_set_hyp_vector(); if (is_kernel_in_hyp_mode()) kvm_timer_init_vhe(); else cpu_init_hyp_mode(); + cpu_set_hyp_vector(); + kvm_arm_init_debug(); if (vgic_present) @@ -1691,18 +1700,59 @@ static void teardown_hyp_mode(void) } } +static int do_pkvm_init(u32 hyp_va_bits) +{ + void *per_cpu_base = kvm_ksym_ref(kvm_arm_hyp_percpu_base); + int ret; + + preempt_disable(); + hyp_install_host_vector(); + ret = kvm_call_hyp_nvhe(__pkvm_init, hyp_mem_base, hyp_mem_size, + num_possible_cpus(), kern_hyp_va(per_cpu_base), + hyp_va_bits); + preempt_enable(); + + return ret; +} + +static int kvm_hyp_init_protection(u32 hyp_va_bits) +{ + void *addr = phys_to_virt(hyp_mem_base); + int ret; + + ret = create_hyp_mappings(addr, addr + hyp_mem_size, PAGE_HYP); + if (ret) + return ret; + + ret = do_pkvm_init(hyp_va_bits); + if (ret) + return ret; + + free_hyp_pgds(); + + return 0; +} + /** * Inits Hyp-mode on all online CPUs */ static int init_hyp_mode(void) { + u32 hyp_va_bits; int cpu; - int err = 0; + int err = -ENOMEM; + + /* + * The protected Hyp-mode cannot be initialized if the memory pool + * allocation has failed. + */ + if (is_protected_kvm_enabled() && !hyp_mem_base) + goto out_err; /* * Allocate Hyp PGD and setup Hyp identity mapping */ - err = kvm_mmu_init(); + err = kvm_mmu_init(&hyp_va_bits); if (err) goto out_err; @@ -1818,6 +1868,14 @@ static int init_hyp_mode(void) goto out_err; } + if (is_protected_kvm_enabled()) { + err = kvm_hyp_init_protection(hyp_va_bits); + if (err) { + kvm_err("Failed to init hyp memory protection\n"); + goto out_err; + } + } + return 0; out_err: @@ -1826,6 +1884,16 @@ static int init_hyp_mode(void) return err; } +static int finalize_hyp_mode(void) +{ + if (!is_protected_kvm_enabled()) + return 0; + + static_branch_enable(&kvm_protected_mode_initialized); + + return 0; +} + static void check_kvm_target_cpu(void *ret) { *(int *)ret = kvm_target_cpu(); @@ -1942,8 +2010,15 @@ int kvm_arch_init(void *opaque) if (err) goto out_hyp; + if (!in_hyp_mode) { + err = finalize_hyp_mode(); + if (err) { + kvm_err("Failed to finalize Hyp protection\n"); + goto out_hyp; + } + } + if (is_protected_kvm_enabled()) { - static_branch_enable(&kvm_protected_mode_initialized); kvm_info("Protected nVHE mode initialized successfully\n"); } else if (in_hyp_mode) { kvm_info("VHE mode initialized successfully\n"); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 4d41d7838d53..9d331bf262d2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -221,15 +221,39 @@ void free_hyp_pgds(void) if (hyp_pgtable) { kvm_pgtable_hyp_destroy(hyp_pgtable); kfree(hyp_pgtable); + hyp_pgtable = NULL; } mutex_unlock(&kvm_hyp_pgd_mutex); } +static bool kvm_host_owns_hyp_mappings(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + return false; + + /* + * This can happen at boot time when __create_hyp_mappings() is called + * after the hyp protection has been enabled, but the static key has + * not been flipped yet. + */ + if (!hyp_pgtable && is_protected_kvm_enabled()) + return false; + + WARN_ON(!hyp_pgtable); + + return true; +} + static int __create_hyp_mappings(unsigned long start, unsigned long size, unsigned long phys, enum kvm_pgtable_prot prot) { int err; + if (!kvm_host_owns_hyp_mappings()) { + return kvm_call_hyp_nvhe(__pkvm_create_mappings, + start, size, phys, prot); + } + mutex_lock(&kvm_hyp_pgd_mutex); err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot); mutex_unlock(&kvm_hyp_pgd_mutex); @@ -291,6 +315,16 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, unsigned long base; int ret = 0; + if (!kvm_host_owns_hyp_mappings()) { + base = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, + phys_addr, size, prot); + if (IS_ERR_OR_NULL((void *)base)) + return PTR_ERR((void *)base); + *haddr = base; + + return 0; + } + mutex_lock(&kvm_hyp_pgd_mutex); /* @@ -1270,10 +1304,9 @@ static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { .virt_to_phys = kvm_host_pa, }; -int kvm_mmu_init(void) +int kvm_mmu_init(u32 *hyp_va_bits) { int err; - u32 hyp_va_bits; hyp_idmap_start = __pa_symbol(__hyp_idmap_text_start); hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE); @@ -1287,8 +1320,8 @@ int kvm_mmu_init(void) */ BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); - hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); - kvm_debug("Using %u-bit virtual addresses at EL2\n", hyp_va_bits); + *hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + kvm_debug("Using %u-bit virtual addresses at EL2\n", *hyp_va_bits); kvm_debug("IDMAP page: %lx\n", hyp_idmap_start); kvm_debug("HYP VA range: %lx:%lx\n", kern_hyp_va(PAGE_OFFSET), @@ -1313,7 +1346,7 @@ int kvm_mmu_init(void) goto out; } - err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits, &kvm_hyp_mm_ops); + err = kvm_pgtable_hyp_init(hyp_pgtable, *hyp_va_bits, &kvm_hyp_mm_ops); if (err) goto out_free_pgtable; From patchwork Wed Mar 10 17:57:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F33C433E0 for ; Wed, 10 Mar 2021 18:05:15 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67D8764FB2 for ; Wed, 10 Mar 2021 18:05:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67D8764FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HeBzea+OZfggrL+hiWTbKOtwIXF8sSAzq0GrIwWic8Y=; b=GN3jS6uKaSSIqG wAIlXrw23nJIEravMA93+CsZ2nQQLPc+rc/jrDEatfByp9Q4zR7FDLrIuPkCLzY5ztI2P8qNiUAdq YUae+fhfikqfr2T0vSPytiCJo22EIzsoY+CThbXM91lt5aqGuxiPwsWw+Zs+z6PP4eMN1FPfPwEmk sWipwbqe5EjwWWOo8xUp5Vro4m7/ciZHFBnZ++uoZJgWuHHvHjHH2/Oh9ZM2R52RpE3pg0ZQR9oWS e39QBHZ4/LXYbOG1RvpLlROxXcS5sSyoD2T29Wo28DIuaWv4XCThd8MP1Obd3k6Wsd3O3yA7I7QBF GQoqaq6dmvJCitis+tgA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Az-007Qbn-Bk; Wed, 10 Mar 2021 18:03:17 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36Q-007OJ8-Ge for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:36 +0000 Received: by mail-wr1-x449.google.com with SMTP id l10so8323136wry.16 for ; Wed, 10 Mar 2021 09:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=W4XDH/01AoBIjD0ZOe4o0NanGmxh9U/l/TLYQRWS474=; b=rVjU1ZdVfExOrksCAL4Z1GT3PZeMJZ0h7ZAPKRQHJAIJfJW7tDOd2AASJ2mux0qrYN 1rULuskHCPdzDmo6vhlNkqJ/NlPUZ0ntL5GCYmSFrx0kYMwpJauRJZYDlNha8olzUJoz vrlLbUG+29hP5ULy+oYpA47soaUj6+UVCe12tmvosC6BUnX26weFobSb5dSstRjH65vm pNoPXQw3J40BAazYKB+sE+NwpC1+hccrTonWnW+TgO27u5e+KJUPqpIRT8UY7Uq6RA0R dq/NXsI7zetZ9lQVfui0ouUnpss4a1IHVw+0U9BDGZeEmnpbFqBu3D1IWJHbM+U2WvCv ySuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=W4XDH/01AoBIjD0ZOe4o0NanGmxh9U/l/TLYQRWS474=; b=T4BB7UbvoOoa1+WYpNlaL0Iv5Z/z6jrEKnO90UlpDjvK67nj2HUAQo5tSsmZ5laswF Z4cKL5MfMWh2w0TaolMNXVMFV6I0FRO55ouELZE1+Q6ikBraGIuS9cgBGlhwqHCa3nvo BWxwZ2xTCsL1kkqvYtnAdkOy/NYwGiUO+4mRfqWBSyaWi0p8sdgb0GJa5Npb3FtQUle/ 2TPygGJmmcCLa0JagXLO0IUl2DtWOoLkaAlYU7tkQqW0vm8T9qczdkNn2RSMdYCrlzsG UQv0SfCcW20Dkoz5ZfVLQ/BgzEF3MoKM5qTCAJDp+qDK4Dpb4vJ950H2QTJOnHhGdBQZ qZmg== X-Gm-Message-State: AOAM5306n5OqWKmY1brw00fp391F+FGCisKEgos8ev/nRk+35F/n9BAC 6+DvRhBGkWnMTmByQYDVDGuloG5oMHov X-Google-Smtp-Source: ABdhPJw1WVYoOd+S+aYtg9+9QgHIHueXvo6DK8FDPXLM33mRGspP0d/W+dbOqjpMDK4WkTw85hCFJYxXdY/H X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:6243:: with SMTP id w64mr1336047wmb.0.1615399113037; Wed, 10 Mar 2021 09:58:33 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:35 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-19-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 18/34] KVM: arm64: Use kvm_arch for stage 2 pgtable From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175834_685330_7D4A4DE5 X-CRM114-Status: GOOD ( 13.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, use struct kvm_arch in lieu of struct kvm as the host will have the former but not the latter. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 5 +++-- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- arch/arm64/kvm/mmu.c | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index bf7a3cc49420..7945ec87eaec 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -162,12 +162,13 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, /** * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. - * @kvm: KVM structure representing the guest virtual machine. + * @arch: Arch-specific KVM structure representing the guest virtual + * machine. * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops); /** diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 7ce0969203e8..3d79c8094cdd 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -879,11 +879,11 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; - u64 vtcr = kvm->arch.vtcr; + u64 vtcr = arch->vtcr; u32 ia_bits = VTCR_EL2_IPA(vtcr); u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; @@ -896,7 +896,7 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, pgt->ia_bits = ia_bits; pgt->start_level = start_level; pgt->mm_ops = mm_ops; - pgt->mmu = &kvm->arch.mmu; + pgt->mmu = &arch->mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ dsb(ishst); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9d331bf262d2..41f9c03cbcc3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -457,7 +457,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); + err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; From patchwork Wed Mar 10 17:57:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0BA2C433DB for ; Wed, 10 Mar 2021 18:05:35 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 196C864F95 for ; Wed, 10 Mar 2021 18:05:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 196C864F95 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eOli8+ez0tBg8X4Ik3hJXecbQQgvaoFvL9EGRteHWoQ=; b=ljNcz22Ve9l0WR wvV7Y2U+KXv7s7Dk0+tXwIEWS4+1euyHzQEG66gduJir1mUdnQZbBjUdZQlXvCcb7hD6iu4ENRYv3 6UjZoA5LjKtIDexjKpURk6FJ49bpuT+Dzof+OG6tnTsI1e2BS9EipEbGwfsT/5SBDZy2U2s1uYTSp xbE1Fu8X3gQnodsm67N+ElYJhxDQhQWclP56vvbox8Om8heMarD/uKngkqY2ZdH3a8aEdIfZfwuiH nJf1mdKPYajyM/tDamXP8lNuvlRgswCwbsxxXi8TnihaZ6+Jife/sEhXpm/JKvThbP6kRLDUpZlaJ cJJidn2KkvV6dtWpNw6w==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3BV-007QnY-6z; Wed, 10 Mar 2021 18:03:49 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36S-007OKE-9m for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:38 +0000 Received: by mail-wr1-x449.google.com with SMTP id p15so8233118wre.13 for ; Wed, 10 Mar 2021 09:58:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bwje5/mTjAPXp3ucjvZl1d3xYnpG4DOJqI6qSewZTKg=; b=kTiEMz/ZmApZPyPPBuOAVVI1XfEWrDD/FlF9YL2vlvyUk0ZxpWZq3Vi1P8ENfyf0yW D17Yf4vX3j1XGW/K8xI2SLf89zP337IPhHEL/jdhn9M1+VDba9SqNKFobPjoh2NGwWSB RDixKT/u9Jildo/Y/HLiqJZSyI/IWe2xUBO5AJ5JlBNdkrd0Zu0Vf2n7fqVD+LzeY+4x KqhovBFzGi07230Rr2y0RdacrGcazfH8JgFDFtTA2H0NTq1XSLz2+EaWylvc0fhkLnu6 GBglPu+rtSYfnztBzxRMswohSH+NtYVchzFRw//FT8yJ1Yr6M0pZG+/GE+SR8cI9vlPP 4yxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bwje5/mTjAPXp3ucjvZl1d3xYnpG4DOJqI6qSewZTKg=; b=Kf/pRQP0g7mqVtsbPbHFPz5f11fIdrR2pvaay9c7Pu8PFLrRS2ZqYkNzg1lUpPCcaC NW6BE3nYcPUmPibFmQl5w7a2Byx7k3eJ4v7gcEr1ObGq2r5muxX5nhmvzFn443JUCODU uvOFATfdr2RO+m/hV5acNXexISv2PopdpB1AJRwlkQq+E1LCoDhgxcidkByp/7iNUxNA Ao1w4zhfXfRQp6b32xlsqjaEJ7IVLcNpscdHF+d92ezvDvdQCtwq0CAtxtSr0A4KlfGF rwpB+iGSLJSORmsf3i+GBVLWd3Dl65Ev96QZhzgwh2xKxVFBxuznf+YKTNIQ46ah1P/z BN0A== X-Gm-Message-State: AOAM533Ymi1TapqoTvEaa2CL/Un7pgjxYD69xjG4LGrUUC69drncVktv hPeunB3xntC6H7ftBDdBDYWIf1U9n79X X-Google-Smtp-Source: ABdhPJyfj1AGgYu6hP2WiGG5K1co03Lv+9xUcg9I+svgyqAymq4hmK7PO5V/NqSdZBVwz+UtX25b6J8FGq9/ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:4e0f:: with SMTP id g15mr4599326wmh.144.1615399115454; Wed, 10 Mar 2021 09:58:35 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:36 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-20-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 19/34] KVM: arm64: Use kvm_arch in kvm_s2_mmu From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175836_826365_87CEA8FE X-CRM114-Status: GOOD ( 15.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, change kvm_s2_mmu to use a kvm_arch pointer in lieu of the kvm pointer, as the host will have the former but not the latter. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/asm/kvm_mmu.h | 6 +++++- arch/arm64/kvm/mmu.c | 8 ++++---- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index b9d45a1f8538..90565782ce3e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -94,7 +94,7 @@ struct kvm_s2_mmu { /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; - struct kvm *kvm; + struct kvm_arch *arch; }; struct kvm_arch_memory_slot { diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ce02a4052dcf..6f743e20cb06 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -272,7 +272,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) */ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) { - write_sysreg(kern_hyp_va(mmu->kvm)->arch.vtcr, vtcr_el2); + write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -283,5 +283,9 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) +{ + return container_of(mmu->arch, struct kvm, arch); +} #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 41f9c03cbcc3..3257cadfab24 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -165,7 +165,7 @@ static void *kvm_host_va(phys_addr_t phys) static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size, bool may_block) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); phys_addr_t end = start + size; assert_spin_locked(&kvm->mmu_lock); @@ -470,7 +470,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; - mmu->kvm = kvm; + mmu->arch = &kvm->arch; mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); mmu->vmid.vmid_gen = 0; @@ -552,7 +552,7 @@ void stage2_unmap_vm(struct kvm *kvm) void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt = NULL; spin_lock(&kvm->mmu_lock); @@ -621,7 +621,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, */ static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect); } From patchwork Wed Mar 10 17:57:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13565C433E0 for ; Wed, 10 Mar 2021 18:06:18 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DB8A264FB2 for ; Wed, 10 Mar 2021 18:06:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB8A264FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GF1gz80QKBLAfdEdrg3WyjpVo+KVIyY7O0AP8uu7lPA=; b=H5xW6PGxKbqWVg cN2v+VW1pUqip6NLW7qIPBAmNrU+UjezBkJcnBRBBETWAr4jr13IHF7BBS06RweDDpkiP8WcQi4zL GTasfxOVWHJOi9qFC9pzptPY7/SSlcshgXMPagvXiR4gKsgqgAVVUxGyFasB0Pk/nBIdXADL5Bexh d80tGw80fpUeS4MXfaFX/CaAIk3f4ohkh0AS9kZ2C7HJgumyjUKILiJ7T3FuSNVMfkKe6g0e4QHQg RIqomqNeisHR6si9nKnZOpsQxAa5SGg+yXBdZD5pyK+RrmbFk0+9m7BixQiBbaPZda8ApXW/hl/gT 2HCJKC8nYK6AfYbQM4nA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3By-007QxC-Jh; Wed, 10 Mar 2021 18:04:19 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36V-007OL9-Hb for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:44 +0000 Received: by mail-wr1-x449.google.com with SMTP id e13so8314725wrg.4 for ; Wed, 10 Mar 2021 09:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ukntJyqlY8URVrIATiFSUwM/7ndF4QPoRdbFD1D8cLg=; b=lDk0OHk/md/iMsmkAy5iConC2JErXeKcb17H5xQgrPoesMToSTbLM6aPWft+KddUDv xRHMLfzHCwXB02tqj2157yByCFQq6A9y3T4I41Q3Blpg3fBKOx3HaKymOpG/gAIsJi/K iCY7fu7KiB8H1v+pMNSvapgYjEXAkH67GLVwTq1pMBR1BOSROfw0SkCSoPtGnYtn5HNI JJVW//6uDYJwmlyVR8IReX82QnF5sXJ4Nkr7eNLYnv5XgTw7Uv+t+fbEhByMYyWIfgzc pUhXvPwLK5BSrgo17ecInjt6ijW1YaBAxL7Bdx2E/dleEuzvha1lQ1eYzLTtGlxDHV1H 2pig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ukntJyqlY8URVrIATiFSUwM/7ndF4QPoRdbFD1D8cLg=; b=AJ4mGYDYeYu/KHVoz3Dhd4MBhYvJGLYTIyDA+Io1p+sAoxKXHciOtf/MQ85kDg10D8 7gdoOQeNwDXOgnM3MuuCcqk4CUQPfsu9OWjJ6rNs2iNHODjgw04LAQflee7L/bdWsZ1h dbamKPF+Gc7AmPQUn1wVjP/VSom7RNH1cjtbh+ONxQtSdCfrIxcE3LrSjTfDvghvOJAg U22RmXPrUJqkjaB5IkdtLLouTgryG0GoLGmvXpZst3n8orc52NxLu2CRgenKnp+f9qkN RiS2zigxbCdgQvVm8yAuIqzlCMASLfP4bgYy102lnI2+FBw6XHvxI9E5gL3P7Kdcek/0 /SGg== X-Gm-Message-State: AOAM530KAk9K6B2WrGEXYu2TifBP2ltuieR31i9ybB99nc8WHY1ywN2E uO1iwpVoh2eiWevbTQpHO9b6rR35kt3Y X-Google-Smtp-Source: ABdhPJwb2As/znXn4RytVDQMc+kTRchy4s+1YPQC255pRPsVjgsDHstfFQUtsKt7jJ+dLTWNx3JlUPzmMAjj X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:3c8b:: with SMTP id j133mr4505584wma.164.1615399117543; Wed, 10 Mar 2021 09:58:37 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:37 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-21-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 20/34] KVM: arm64: Set host stage 2 using kvm_nvhe_init_params From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175841_958872_31FBC22E X-CRM114-Status: GOOD ( 13.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the registers relevant to host stage 2 enablement to kvm_nvhe_init_params to prepare the ground for enabling it in later patches. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kvm/arm.c | 5 +++++ arch/arm64/kvm/hyp/nvhe/hyp-init.S | 14 +++++++++----- arch/arm64/kvm/hyp/nvhe/switch.c | 5 +---- 5 files changed, 21 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index db20a9477870..6dce860f8bca 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -158,6 +158,9 @@ struct kvm_nvhe_init_params { unsigned long tpidr_el2; unsigned long stack_hyp_va; phys_addr_t pgd_pa; + unsigned long hcr_el2; + unsigned long vttbr; + unsigned long vtcr; }; /* Translate a kernel address @ptr into its equivalent linear mapping */ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index a36e2fc330d4..8930b42f6418 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -120,6 +120,9 @@ int main(void) DEFINE(NVHE_INIT_TPIDR_EL2, offsetof(struct kvm_nvhe_init_params, tpidr_el2)); DEFINE(NVHE_INIT_STACK_HYP_VA, offsetof(struct kvm_nvhe_init_params, stack_hyp_va)); DEFINE(NVHE_INIT_PGD_PA, offsetof(struct kvm_nvhe_init_params, pgd_pa)); + DEFINE(NVHE_INIT_HCR_EL2, offsetof(struct kvm_nvhe_init_params, hcr_el2)); + DEFINE(NVHE_INIT_VTTBR, offsetof(struct kvm_nvhe_init_params, vttbr)); + DEFINE(NVHE_INIT_VTCR, offsetof(struct kvm_nvhe_init_params, vtcr)); #endif #ifdef CONFIG_CPU_PM DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp)); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7d62211109d9..d474eec606a3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1413,6 +1413,11 @@ static void cpu_prepare_hyp_mode(int cpu) params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); + if (is_protected_kvm_enabled()) + params->hcr_el2 = HCR_HOST_NVHE_PROTECTED_FLAGS; + else + params->hcr_el2 = HCR_HOST_NVHE_FLAGS; + params->vttbr = params->vtcr = 0; /* * Flush the init params from the data cache because the struct will diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index a2b8b6a84cbd..a50ad9e9fc05 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -83,11 +83,6 @@ SYM_CODE_END(__kvm_hyp_init) * x0: struct kvm_nvhe_init_params PA */ SYM_CODE_START_LOCAL(___kvm_hyp_init) -alternative_if ARM64_KVM_PROTECTED_MODE - mov_q x1, HCR_HOST_NVHE_PROTECTED_FLAGS - msr hcr_el2, x1 -alternative_else_nop_endif - ldr x1, [x0, #NVHE_INIT_TPIDR_EL2] msr tpidr_el2, x1 @@ -97,6 +92,15 @@ alternative_else_nop_endif ldr x1, [x0, #NVHE_INIT_MAIR_EL2] msr mair_el2, x1 + ldr x1, [x0, #NVHE_INIT_HCR_EL2] + msr hcr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTTBR] + msr vttbr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTCR] + msr vtcr_el2, x1 + ldr x1, [x0, #NVHE_INIT_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index f3d0e9eca56c..979a76cdf9fb 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -97,10 +97,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT; write_sysreg(mdcr_el2, mdcr_el2); - if (is_protected_kvm_enabled()) - write_sysreg(HCR_HOST_NVHE_PROTECTED_FLAGS, hcr_el2); - else - write_sysreg(HCR_HOST_NVHE_FLAGS, hcr_el2); + write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); write_sysreg(CPTR_EL2_DEFAULT, cptr_el2); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } From patchwork Wed Mar 10 17:57:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C4A8C433DB for ; Wed, 10 Mar 2021 18:06:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AAB0C64FB2 for ; Wed, 10 Mar 2021 18:06:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAB0C64FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=etL4Udov181QD8rHuu9o3JyJ1xLWVpzKUiTMdeNiR58=; b=pCOid+vOWdqT55 gP48Qmt/b/fAaqFeu/ZzcbbTRmsPuLDQFrIBP+01lZp/eG99cHfmNXUcve2gA73QnF9yZfpd44BEJ 3ATaAN9LhebjOLkFC/pFMBX8Rj54aFZrMsgDELk8QxnzDQcwGeLDqAg9yYz5Bpm7kbfO2SIBgQotD 1247ZvQxDwbTgIhrIfdPDHehctVdkjfGxxUQUDn/mYnZY3BpZ71alPA0/dc8Lq2PFYVy062NLOuL2 JA/kd7A+s0n1aLfTZ7RZO5PGzLvE/Tl0gQQ9LRZHris+0K//GfKKeJhoXXb24C0LN5ciKLKXfAbbp skpjh9vSmVMqtDm294Lg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3CT-007R7d-4W; Wed, 10 Mar 2021 18:04:49 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36X-007OLS-RG for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:45 +0000 Received: by mail-wm1-x349.google.com with SMTP id l16so1707799wmc.0 for ; Wed, 10 Mar 2021 09:58:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5IlrHyzKGX66i7zKiJ2V9j/u7ly1Dp8Z1hyFGz/2u64=; b=iVAj6JJV6CazuRaEPHki7YV1asV87XjurUX2cUuvZ+I5M5+7StJNzXQ21pGTkVmDPz 2aczpVWNfqn7LkeVOxA4/yeE8nSN0ITE4MVenhoN2DdY+LEMGGlGB6dS2Fm2BhIRJXLD rSiLrmFClj4NldUmlaELdYXImbF1cUAqAObLSzjCF0Z6iB1wX08jVJ8aRTSKpGuuFLXC KLzpf94b1W+zksuPkHDjocduv3XlZIR/2ciHM7qd+bkMsL0xXUzrYwM98UcwaCO40nwX HBdxVkjPx1O/4/9AeMOtCur4ZXKGvLW3PbkmYR/LKB2Hovbu36MfZT+uDT99TDt7FF6c mAbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5IlrHyzKGX66i7zKiJ2V9j/u7ly1Dp8Z1hyFGz/2u64=; b=CuWXypCjhbpmkaCM80kfEHzxDQq254Z03cmGjBHwVxL6isFwmDf+cXQNWaTmIXWc9X 9YLe9RNscV1wqdy/jHF5IOzC/Im2Hheje5I5U5IHUsFYdYKjuhvAmyKZd6m9kcRy4Qs6 Mn0OJAuv+hx+IIj+zMkR4Xlp0WbLiFJXa84uE1PrwtSR+y1Lllmo7PRK+TUtxVGtI5E4 i5sw0Gjf71HFdznmvnVX0nMp0JKDnWu/cwTA1YsP5+fl5YvAxwzFW9wG+4J3hyvwERzW RrrPHfH2wRrdAMMfYUSguOGOj8aR0zZbBHlCGHFjTfJMAEt/TO2PeEQi53FqoS/jvr8Q q/ng== X-Gm-Message-State: AOAM5325qkn1rTeM/CHGSVaR1vrfWnqwVzCdzQCr9js1OGDm1NGLNSz5 ohQgQeHVG6nLqEpEKucCujDzqT8K46N7 X-Google-Smtp-Source: ABdhPJxGHzNd7t0P9DMTeyoCUCJ6fZLjg9pp9aeQh2h0lWdksae5/FeYBt7pbAYrMnYAs7rK3cYXZJFsHPtR X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:6243:: with SMTP id w64mr1336092wmb.0.1615399119789; Wed, 10 Mar 2021 09:58:39 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:38 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-22-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 21/34] KVM: arm64: Refactor kvm_arm_setup_stage2() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175842_145431_1C0C1A0F X-CRM114-Status: GOOD ( 20.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-use some of the stage 2 setup code at EL2, factor parts of kvm_arm_setup_stage2() out into separate functions. No functional change intended. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 26 +++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 32 +++++++++++++++++++++ arch/arm64/kvm/reset.c | 42 +++------------------------- 3 files changed, 62 insertions(+), 38 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 7945ec87eaec..9cdc198ea6b4 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -13,6 +13,16 @@ #define KVM_PGTABLE_MAX_LEVELS 4U +static inline u64 kvm_get_parange(u64 mmfr0) +{ + u64 parange = cpuid_feature_extract_unsigned_field(mmfr0, + ID_AA64MMFR0_PARANGE_SHIFT); + if (parange > ID_AA64MMFR0_PARANGE_MAX) + parange = ID_AA64MMFR0_PARANGE_MAX; + + return parange; +} + typedef u64 kvm_pte_t; /** @@ -159,6 +169,22 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt); int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot); +/** + * kvm_get_vtcr() - Helper to construct VTCR_EL2 + * @mmfr0: Sanitized value of SYS_ID_AA64MMFR0_EL1 register. + * @mmfr1: Sanitized value of SYS_ID_AA64MMFR1_EL1 register. + * @phys_shfit: Value to set in VTCR_EL2.T0SZ. + * + * The VTCR value is common across all the physical CPUs on the system. + * We use system wide sanitised values to fill in different fields, + * except for Hardware Management of Access Flags. HA Flag is set + * unconditionally on all CPUs, as it is safe to run with or without + * the feature and the bit is RES0 on CPUs that don't support it. + * + * Return: VTCR_EL2 value + */ +u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift); + /** * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3d79c8094cdd..296675e5600d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -9,6 +9,7 @@ #include #include +#include #define KVM_PTE_VALID BIT(0) @@ -449,6 +450,37 @@ struct stage2_map_data { struct kvm_pgtable_mm_ops *mm_ops; }; +u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) +{ + u64 vtcr = VTCR_EL2_FLAGS; + u8 lvls; + + vtcr |= kvm_get_parange(mmfr0) << VTCR_EL2_PS_SHIFT; + vtcr |= VTCR_EL2_T0SZ(phys_shift); + /* + * Use a minimum 2 level page table to prevent splitting + * host PMD huge pages at stage2. + */ + lvls = stage2_pgtable_levels(phys_shift); + if (lvls < 2) + lvls = 2; + vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); + + /* + * Enable the Hardware Access Flag management, unconditionally + * on all CPUs. The features is RES0 on CPUs without the support + * and must be ignored by the CPUs. + */ + vtcr |= VTCR_EL2_HA; + + /* Set the vmid bits */ + vtcr |= (get_vmid_bits(mmfr1) == 16) ? + VTCR_EL2_VS_16BIT : + VTCR_EL2_VS_8BIT; + + return vtcr; +} + static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, struct stage2_map_data *data) { diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 47f3f035f3ea..6aae118c960a 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -332,19 +332,10 @@ int kvm_set_ipa_limit(void) return 0; } -/* - * Configure the VTCR_EL2 for this VM. The VTCR value is common - * across all the physical CPUs on the system. We use system wide - * sanitised values to fill in different fields, except for Hardware - * Management of Access Flags. HA Flag is set unconditionally on - * all CPUs, as it is safe to run with or without the feature and - * the bit is RES0 on CPUs that don't support it. - */ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) { - u64 vtcr = VTCR_EL2_FLAGS, mmfr0; - u32 parange, phys_shift; - u8 lvls; + u64 mmfr0, mmfr1; + u32 phys_shift; if (type & ~KVM_VM_TYPE_ARM_IPA_SIZE_MASK) return -EINVAL; @@ -359,33 +350,8 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) } mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - parange = cpuid_feature_extract_unsigned_field(mmfr0, - ID_AA64MMFR0_PARANGE_SHIFT); - if (parange > ID_AA64MMFR0_PARANGE_MAX) - parange = ID_AA64MMFR0_PARANGE_MAX; - vtcr |= parange << VTCR_EL2_PS_SHIFT; - - vtcr |= VTCR_EL2_T0SZ(phys_shift); - /* - * Use a minimum 2 level page table to prevent splitting - * host PMD huge pages at stage2. - */ - lvls = stage2_pgtable_levels(phys_shift); - if (lvls < 2) - lvls = 2; - vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); - - /* - * Enable the Hardware Access Flag management, unconditionally - * on all CPUs. The features is RES0 on CPUs without the support - * and must be ignored by the CPUs. - */ - vtcr |= VTCR_EL2_HA; + mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1); + kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); - /* Set the vmid bits */ - vtcr |= (kvm_get_vmid_bits() == 16) ? - VTCR_EL2_VS_16BIT : - VTCR_EL2_VS_8BIT; - kvm->arch.vtcr = vtcr; return 0; } From patchwork Wed Mar 10 17:57:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 838B7C433E0 for ; Wed, 10 Mar 2021 18:07:10 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 945B264FB2 for ; Wed, 10 Mar 2021 18:07:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 945B264FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/jM+N60xCx+X5CQ8U0/QkoTPm76I6oa9sBdHtH4+4wk=; b=rE4WIwi8yx50Ev /uWMJBXiXarFEN3YYIcMagoHvtaxyLLlczW2de5WlDw2zm8RizIiH/vwCiAVztzXvChqBC0xSJFir WY0YjkhTZraTzwee8Id/aSubAsArJt2lGF8gxNebmKjbwzLRukPHUdNBOZoH31zmqXu02K1w9R+hJ 2G88Xleaf61bJzYP60qEMMdH/UdkZibVCa3ZzGRNAS1YEZHz6UtCCdV5m30Z+niGvcobz5bRzvCeH yos4CJVIaJuapC6a3HMtKrAFrgB0x2bfB4oI+9otadvt1eY6XW8661Jtlc1xT9Xjn2cAW5RT8Kzrp SxrOZd7ve+P6NFp62Icw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Cy-007RIb-Eg; Wed, 10 Mar 2021 18:05:21 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36Z-007OMq-KM for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:46 +0000 Received: by mail-qt1-x849.google.com with SMTP id w2so7730743qts.18 for ; Wed, 10 Mar 2021 09:58:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4oxp+YpplIoIB69u+tbsX2FMX4bca6h5fwiDtQ6/a1E=; b=vfNvL6VYP4cIvis4RUMqKQm+EjWb6Qdd5uxQRDjzD2jvaui8luwSb3kbZaVs7Cun3c I70CqF0WoKxquNq8GHLfoqZouqHwob9jrTw3fol8OJSK/TwkWGztOcvr42WqszU0moEe H1A8F/B1Tfh3dsLtou0ZuncYrXXuZK625m0AzBYrpPpIlL1jDNEpm3fusVFyhxTSEQLr 9SrqgMAJKv566vQ7PZQQcfFufCHMyYUsTWwpbZFUcqzB38QgZf896YlSPDkat+pg+eei OL16OP7Oedq8fTKRKJjFfbS+wpyjBA5hZtamumQ0v6fBmoQZrRQdY6ofweJTdrk80kOH KjfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4oxp+YpplIoIB69u+tbsX2FMX4bca6h5fwiDtQ6/a1E=; b=peqcIuk6TCswugd1mS7HNhMXy/XrNl1ZOsGQqM2dIdShkRr0913v86kSgyjwPBl0CQ Ex8Q1gQmf9N2REtH63nzowJIlCmXzii5C7xow14g9cNmvSBwMUZ3baQwfsW7YUcnNj38 St79rFPXXjj+QzJk+2o8Z6qip4Zrq/USU8Qz8uSdKkhGF6DuZsxoNiT6ghAuOb3yyhVX iaKTlOgAbtVS/o3zNKNUqhyn0fSCSvG0edoLeRcFYup4qxuxBPNgCDEglpsKIvTRz/6O 6/oLjRF8Qz+lNI3HmU5HlrR3K4Qr4PjwKt2i+jG6+GYkMRDFuAdHlK7JHBKMmqgEZSj8 aDbQ== X-Gm-Message-State: AOAM5316lrpTDfQ6dxUqCZ5XEYDx2CtqcZcuzpYMrGx/72T6ShzI8t7I eBrv4lXEm/40K4QE4PMkEOpUwIrixCT6 X-Google-Smtp-Source: ABdhPJysjBX02XgOxXRd70RHGblWwl/bjhm+VVsnnUsNEc3jBBZ20r71lSy0Vi2+2+6+nm09mj60syhCGiEY X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:f9cc:: with SMTP id j12mr3939245qvo.15.1615399122003; Wed, 10 Mar 2021 09:58:42 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:39 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-23-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 22/34] KVM: arm64: Refactor __load_guest_stage2() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175843_811282_8458FD91 X-CRM114-Status: GOOD ( 10.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __load_guest_stage2() to introduce __load_stage2() which will be re-used when loading the host stage 2. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 6f743e20cb06..9d64fa73ee67 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -270,9 +270,9 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) * Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu, unsigned long vtcr) { - write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); + write_sysreg(vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -283,6 +283,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +{ + __load_stage2(mmu, kern_hyp_va(mmu->arch)->vtcr); +} + static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) { return container_of(mmu->arch, struct kvm, arch); From patchwork Wed Mar 10 17:57:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B7BEC433DB for ; Wed, 10 Mar 2021 18:07:54 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F59664F95 for ; Wed, 10 Mar 2021 18:07:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F59664F95 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=S9zLzvvTTDC/M1XL2HzC7OOL5NKLj3lkO988SuUHXQk=; b=T+3iNHsdlOvHu+ tAXWDW6NCK6Lqd8khEil+g6oQT6tl6SbqBuZgDmdipw3fOZIkUlp35P/g13rq4uNApjFgTvw4yqCj WE58YINmLe7X1M7daMljDIHIGeRkqEJn5M64DOyNt9LJsdF7pb2m9Dfpnk4+oEselyXwtu4ASDy0P IKyYo2jluzkNOxicxjo8KTYPof7sT2NMgIe2stJq5Y3t1YAisUQL9kM8tOd11diHl1TiXB9+saXac S8CB5TpbUgxW1/Xi0cabIIDUWlBRRF3TC1S2LWb55mS3PHOpkjGEEN+tfLMGqx4SoPEka1f54hcED JQ0YMN9Qi4Nhv2A73YqQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Dh-007RY9-HG; Wed, 10 Mar 2021 18:06:06 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36e-007OP3-UB for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:53 +0000 Received: by mail-qt1-x84a.google.com with SMTP id k10so13482511qte.17 for ; Wed, 10 Mar 2021 09:58:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mOohSeaIfwaEz8hEwSDUNnz+PysLBU9WmW/RUHzGWms=; b=UpCuP2JVMuI2+hVj+HMC9GepoPsIZM85HpFVq86MfaLWQfpLNX0CKray2/cYIx3hap FulboVVX/ZHpVWShTcu4rkz5liv323G1IaAkq/Q7XxYW+eVy3QOYbeogjkUE553M3PKX biChA2Ckr+PQ2tq4vQH2hsa+yxoCKi4mutHQoucMaEC1c1zTEb4ltW/vBP3L7OdS8mx1 Z+aDeD8BxzSyaR4xMCs8dlEBcK/zLbGW9ng687rwEBkWedmJU5x5I8x5pTDkDJfsOGr3 gh9NFs7XMGoW4QhnPBTufa45Fj4aVoY9kvDtdX6Mm9AoCMnRhBRJtiXHkfmQanMbjAOo Fiog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mOohSeaIfwaEz8hEwSDUNnz+PysLBU9WmW/RUHzGWms=; b=FXHoCcnTycxJszzikfRuWH5apFBbA27gCPWO4ncUUKtDLfBb+8yZVJgNyASQWTVy2v E/j/w8Vf8xVnVCM0WjD+jxdaizJk/A01ZyMy2SC29Y0fkEzzpuLoSpv7ZKe7VYlpJy4Z 14Epz0ZdjGQ6nE2bhxfLTDFCX2TjWkbM/GJXhZnA9y/wdy3b867aW2g0BinDWPckhNfw BOcQ5o8nr3kk609gcLBJiMoDMlxbDZmPtXMtu8fN4gcTx4ZL9mrTOVzIdCxfeCB+QtHA FOl2KBlUJTBbSqv9/RS19H9aJGjQnEwBjOuyZWWL6au6dt4aGNs8GJyb/lz/zKxAOdpF Ugug== X-Gm-Message-State: AOAM530qAYbuL/VKsiDqKeYDYWc4jq9gahZO0fEAcs7DPgFKIVZGzXFi J0L2yGoLgvIUdwfT4Pn9rIySka6Pms37 X-Google-Smtp-Source: ABdhPJwx7QrSBh2ptN4PQ7VNEdb7G5Onq4cJuMCN82nqpZk0yWXTQDUEtyuol5HOBBtLBr8L8iuseENS5Fk+ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:b894:: with SMTP id y20mr4097294qvf.43.1615399125954; Wed, 10 Mar 2021 09:58:45 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:40 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-24-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 23/34] KVM: arm64: Refactor __populate_fault_info() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175849_446913_0488EC9E X-CRM114-Status: GOOD ( 13.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __populate_fault_info() to introduce __get_fault_info() which will be used once the host is wrapped in a stage 2. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/hyp/switch.h | 34 +++++++++++++------------ 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 6c1f51f25eb3..40c274da5a7c 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -160,19 +160,9 @@ static inline bool __translate_far_to_hpfar(u64 far, u64 *hpfar) return true; } -static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +static inline bool __get_fault_info(u64 esr, struct kvm_vcpu_fault_info *fault) { - u8 ec; - u64 esr; - u64 hpfar, far; - - esr = vcpu->arch.fault.esr_el2; - ec = ESR_ELx_EC(esr); - - if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) - return true; - - far = read_sysreg_el2(SYS_FAR); + fault->far_el2 = read_sysreg_el2(SYS_FAR); /* * The HPFAR can be invalid if the stage 2 fault did not @@ -188,17 +178,29 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) if (!(esr & ESR_ELx_S1PTW) && (cpus_have_final_cap(ARM64_WORKAROUND_834220) || (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) { - if (!__translate_far_to_hpfar(far, &hpfar)) + if (!__translate_far_to_hpfar(fault->far_el2, &fault->hpfar_el2)) return false; } else { - hpfar = read_sysreg(hpfar_el2); + fault->hpfar_el2 = read_sysreg(hpfar_el2); } - vcpu->arch.fault.far_el2 = far; - vcpu->arch.fault.hpfar_el2 = hpfar; return true; } +static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +{ + u8 ec; + u64 esr; + + esr = vcpu->arch.fault.esr_el2; + ec = ESR_ELx_EC(esr); + + if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) + return true; + + return __get_fault_info(esr, &vcpu->arch.fault); +} + /* Check for an FPSIMD/SVE trap and handle as appropriate */ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) { From patchwork Wed Mar 10 17:57:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A989C433DB for ; Wed, 10 Mar 2021 18:08:48 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 436D464E20 for ; Wed, 10 Mar 2021 18:08:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 436D464E20 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ONRONUSPpUD9Jl9lNPqzzgnjSkk4yWTWl0refAQEt0Y=; b=goOzds9sQ/bHS4 7Aa4xGyefJmvxjZ/ipdvVSUIp4MV8D6tqw67LJsd26gxMVFFAP+MO0zPndFJfAh45AFkIorPGrp2P Jfg4KBeHPyuBuFFZyAp32jwUbRstQ/qJ8akLoRZuWdX9G2hvTF71Z8gtJz0Lx2LT08ucVcIk2VWFS B7rl6rrlmz28Qmq1UzhLBD5PxNN3mDvQkwrxIjIw4XfNh3FZxUH0ZSTLsbfxIzFPIbs6/QTWbZqDn 4a2lJU8jpQPtc1tU5GeQ3DlyLZUFSfg46FPIlzwfmjP9jKPOgUxBnOeiOJfz2Zv5Et+lnanVcP0hT iE8XJEH3Ubz1hE7rGSIg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3EM-007RnS-NX; Wed, 10 Mar 2021 18:06:46 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36f-007OPI-IC for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:53 +0000 Received: by mail-wr1-x449.google.com with SMTP id h30so8334894wrh.10 for ; Wed, 10 Mar 2021 09:58:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=nO+Dw20r7SUumZfYakzeexkud+udjeRUH05spovGIBc=; b=q98vijkRUeBOdzAWoY6jRSeh2vMwodAX3jBub6VuJmeqX1nog5ludU2AjzqL8PAyuq 3cPVvj3cnJOPB8MRhg4r4XPGpnVaB3OyXeAowiksQMcBVNgSPw1LkT2uJGYyKIgFxcij M/JowpIjqjiIs5oXD3XYIS+g6gTCrKX6YM5/vQZkGhwkzjsiNiVrklSK8eGZPwXSALI6 mjfYDEg94yHnnmrjLyFv4KW5o/KxZZAxhFmGgZWX80bSgINRDmOqSImA4+eg8cM0yeQl VKWh5T18d3rrEZMMW5p0AAVlXtJD4jrKvoyWlmZ1E7RkfQg7ecYwBLpIYjCgOV4lU9Tu jHrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nO+Dw20r7SUumZfYakzeexkud+udjeRUH05spovGIBc=; b=k3IHlg2fClq512Jy/BJaUDeGyM0XEe0ZeHPrsT43Wa6mOLu5dbWnBXy0PGtLnA0EH0 e1Pk0ZSsXRWmG6rKDZgrWBghv0y8dWeQtZC3cVaV9T4x+AOVF43PUlesojUpWbGSn2b0 RmLYhpThwJaeQRLaO4OZM/HYQL7S3ciB/nZrJulZqsRuRJYiJYEKWPdEXpnPKs7N0drI uZyzDnm5pd5MCvi0hNxjbOmlfuGg0LQ5BHRdW0Q7bj/eKOrszLZf+n0HrKtqhEEn2Sz1 KO6i74uS8gmv6Egt+VOM18hvtkY/Pbk1rj1i8/oDgOVqReTtQTpzGeS+HIdU4lXte7QU kpnw== X-Gm-Message-State: AOAM5322++TCvLTw44HZBolwFFbRU5XJEvP6eyNn4vmWVgzi6r8W2fgh UpO98Zuzpor+2Sy3iRrtLnv5O9nOG7fa X-Google-Smtp-Source: ABdhPJwVhnSLfeKiraD5idqzHXTWJrPR2Q57MoB1wpdcj8fUwYlAEfZZUCvSXNNkTOyW9ggJRQmLFzB6VaEi X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:cf16:: with SMTP id l22mr4421313wmg.26.1615399128195; Wed, 10 Mar 2021 09:58:48 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:41 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-25-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 24/34] KVM: arm64: Make memcache anonymous in pgtable allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175850_124964_BAE3C7BE X-CRM114-Status: GOOD ( 15.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current stage2 page-table allocator uses a memcache to get pre-allocated pages when it needs any. To allow re-using this code at EL2 which uses a concept of memory pools, make the memcache argument of kvm_pgtable_stage2_map() anonymous, and let the mm_ops zalloc_page() callbacks use it the way they need to. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 6 +++--- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 9cdc198ea6b4..4ae19247837b 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -213,8 +213,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * @size: Size of the mapping. * @phys: Physical address of the memory to map. * @prot: Permissions and attributes for the mapping. - * @mc: Cache of pre-allocated GFP_PGTABLE_USER memory from which to - * allocate page-table pages. + * @mc: Cache of pre-allocated and zeroed memory from which to allocate + * page-table pages. * * The offset of @addr within a page is ignored, @size is rounded-up to * the next page boundary and @phys is rounded-down to the previous page @@ -236,7 +236,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc); + void *mc); /** * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 296675e5600d..bdd6e3d4eeb6 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -445,7 +445,7 @@ struct stage2_map_data { kvm_pte_t *anchor; struct kvm_s2_mmu *mmu; - struct kvm_mmu_memory_cache *memcache; + void *memcache; struct kvm_pgtable_mm_ops *mm_ops; }; @@ -669,7 +669,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc) + void *mc) { int ret; struct stage2_map_data map_data = { From patchwork Wed Mar 10 17:57:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CDE1C433E0 for ; Wed, 10 Mar 2021 18:09:18 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 868D364FB1 for ; Wed, 10 Mar 2021 18:09:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 868D364FB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T7K1KtXW+vi8FEhIu+c1FJW6CTyimb7+tPRKlG6a65o=; b=p64OtpftdVx3yn CYDGydWd+CKKrgKQhF1kP+iynoKoQL7ZrPk2LW5hDRtaBJFOkT3Q7cY6iiJEHsRgy+2dtv2zUo4Zg hE7JVCRVYPpDB2hqwVvFE2EQgaGQoADzIAhkXM/jriiHTipZyAinQbKakyxhgEY5vPkMSvwxpufHY NBgXPPhYa3vLu/yZ0GgwD2QxuNF0LBNI89eIkcE9e4cbHrnnA4dsJqN8RguuU4gGiX1agkdO0fU4B usmVVZhUTlEVPRVlDL+rZeU4ImWVbMiOCHUP69KCDoeq9wRMpy0fGZRT5PwahOuPnEhTKu+k0NGct cMNAJQVsmGjFMs1FCDJA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Ev-007Rz4-SA; Wed, 10 Mar 2021 18:07:22 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36j-007OQp-3b for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:55 +0000 Received: by mail-qk1-x74a.google.com with SMTP id h126so13386457qkd.4 for ; Wed, 10 Mar 2021 09:58:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mIqe1MGSlcb95mMvYxk2H+NidebDAWpSvKZgF+GIUJ4=; b=vx1GePNKXUsy3lm2tRdqorO6elSbn6hExBP9oaHOyHkER8tNbdC4pg+nDr9AGdaxu/ 3Pewgjk9+slL6lrKsPYNW1OpM3lhTxz2lcsvqeup2Wyxbzq2NXQaVBftczldIThhz7Yp /HAcyfzAp3RoP8J9843KokXr36i92XcvOZWvZo93MZts7mNB1shqN4YA+qaTXAvDoTS6 28UBYIp8LKmk/A0xaarxs0aZTp/RH/iKCJj4GooZLOJUopksawDWvdVuHDx/lrG+zeV5 gDMOvGL4kjJ01oz0/cO/LrK5/pM+kPniLilZvJxb6+MyXtgKLpLtfI2U/M6uuxfIEjR/ 72Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mIqe1MGSlcb95mMvYxk2H+NidebDAWpSvKZgF+GIUJ4=; b=lBYMmla1EXASYppW7bT+5P4zlQod3snovGz4Hl3w8lmxSEkrjkdTomskXF67vxUqmX SLBUZSD5GwlDXn+6MHsVd0svHGHx2cXlt5Mop0scedugSJmVpy9XOHuU3o7SrZKv0V4s DqFstzcyIHWvH5nZGpsVdDTP/0H6GHufzRq0RQs++Fq4Texz2KSVKkEHUCiXAML8o40X 0U5Xb8bADl8l9LYkQuAwVl5UqDDC5PoRU3pIutNBDCLiRWjWjF0mDZGJPs0y2TD+F706 QnCq0qx2jSCBjt3RuCcGmIf+pTN9QkxAL+fGJ49Q81t/pZ+oj7mUKUjfgJWDz7jt1deF ExMw== X-Gm-Message-State: AOAM531AIIgQJDzgVfc3scRONmpGKLSOHaI/o42a8x9kvDFBrrJKB5VK TU5mxGDPfx0uXjsqmdPzJBnnEMNi35gO X-Google-Smtp-Source: ABdhPJy2ywA58O3Ivcmfuywfr7DLLO/RhF30F1qBje3/94+cZdCa6U9wIlM0XrJdUI5Hekvh7JWZdZjaP3b/ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:bec3:: with SMTP id f3mr3180215qvj.49.1615399130193; Wed, 10 Mar 2021 09:58:50 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:42 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-26-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 25/34] KVM: arm64: Reserve memory for host stage 2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175853_608783_0B205DC3 X-CRM114-Status: GOOD ( 16.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extend the memory pool allocated for the hypervisor to include enough pages to map all of memory at page granularity for the host stage 2. While at it, also reserve some memory for device mappings. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/mm.h | 27 ++++++++++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 12 ++++++++++++ arch/arm64/kvm/hyp/reserved_mem.c | 2 ++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index ac0f7fcffd08..0095f6289742 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -53,7 +53,7 @@ static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) return total; } -static inline unsigned long hyp_s1_pgtable_pages(void) +static inline unsigned long __hyp_pgtable_total_pages(void) { unsigned long res = 0, i; @@ -63,9 +63,34 @@ static inline unsigned long hyp_s1_pgtable_pages(void) res += __hyp_pgtable_max_pages(reg->size >> PAGE_SHIFT); } + return res; +} + +static inline unsigned long hyp_s1_pgtable_pages(void) +{ + unsigned long res; + + res = __hyp_pgtable_total_pages(); + /* Allow 1 GiB for private mappings */ res += __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); return res; } + +static inline unsigned long host_s2_mem_pgtable_pages(void) +{ + /* + * Include an extra 16 pages to safely upper-bound the worst case of + * concatenated pgds. + */ + return __hyp_pgtable_total_pages() + 16; +} + +static inline unsigned long host_s2_dev_pgtable_pages(void) +{ + /* Allow 1 GiB for MMIO mappings */ + return __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); +} + #endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 1e8bcd8b0299..c1a3e7e0ebbc 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -24,6 +24,8 @@ unsigned long hyp_nr_cpus; static void *vmemmap_base; static void *hyp_pgt_base; +static void *host_s2_mem_pgt_base; +static void *host_s2_dev_pgt_base; static int divide_memory_pool(void *virt, unsigned long size) { @@ -42,6 +44,16 @@ static int divide_memory_pool(void *virt, unsigned long size) if (!hyp_pgt_base) return -ENOMEM; + nr_pages = host_s2_mem_pgtable_pages(); + host_s2_mem_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_mem_pgt_base) + return -ENOMEM; + + nr_pages = host_s2_dev_pgtable_pages(); + host_s2_dev_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_dev_pgt_base) + return -ENOMEM; + return 0; } diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index 9bc6a6d27904..fd42705a3c26 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -52,6 +52,8 @@ void __init kvm_hyp_reserve(void) } hyp_mem_pages += hyp_s1_pgtable_pages(); + hyp_mem_pages += host_s2_mem_pgtable_pages(); + hyp_mem_pages += host_s2_dev_pgtable_pages(); /* * The hyp_vmemmap needs to be backed by pages, but these pages From patchwork Wed Mar 10 17:57:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C504C433E0 for ; Wed, 10 Mar 2021 18:10:21 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B4E6464FB2 for ; Wed, 10 Mar 2021 18:10:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4E6464FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Kze//MFsyT3Rtg1i+50RQX1iypYzGC7rkSMrsAIfqXM=; b=f+TBLkRzx2injF ILZWm/ihPSPa2Eqyy3T3Pct8jYAOt2bG2cu25/PPO2UnorvNr2XvWvhJpQnO6GFZow6Bw8eN11mpX ww0bccOrYcZKxQmSwXBfbtv5QMUb5rW17q/BhDHQFpA+l2JUyFcVj2K3g2Mpz+AlgJL6ulWmjZmac vgYz2GYEbzGHL7F61C2eIYz1VWbL7ovEmASHXlxzwnRUzfLaKtbu6iKrbQc9IE8oVHUrqrkHcnSeU LCEsAymW3oefFqu+UP79NVMpETMw1lY5H7naaRBZV8FIDKRVMe1HvgmK5kj9DGxfVAUrZU6KDTFs4 sUhoeys+b2Oul6gbzrZw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Fe-007SBI-RK; Wed, 10 Mar 2021 18:08:07 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36j-007ORI-Mv for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:56 +0000 Received: by mail-qt1-x849.google.com with SMTP id l63so10081425qtd.23 for ; Wed, 10 Mar 2021 09:58:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qkIv/Dj8eNtVWTw/hqhsYIXxQswk0/NCtZ2mXt61vt4=; b=Yd6qXdpHhKJCGS2nS6b73itvEp4TXHoB4WqOXle6O5tLkJBJPw/XtiPR/uKeu43swS 2hvS/aF6Anvw1YMOPyb5/0LqiiPvfB0aL64RNhlAz42c75kMdBUkplEzK+yFBXNQ6fyQ J/nbBIcWvsjxT1kGPNG/ru1ol04BtoCNsl4j5yN3RA+BegfK37UCg92KsBMRdyqOwIKd jYHjj6ABnTFkH96MW2j3xNeVLtIeiTKy+FdXKciOBIb7A4k5eusNa3eNL2V3Pe6/NjVY Cfa4vxJHinlwjdhhlyquWiYMJZUzIssWJwFW9dFCLf208MYkzWGv7Z4GyP9hNnulY8Hr GKdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qkIv/Dj8eNtVWTw/hqhsYIXxQswk0/NCtZ2mXt61vt4=; b=s16ozOmYCYNQ+nl6yq2Pzkj9/B3Fj5SH0Tp65qXe3AGLLiVlrdFRNtJVBnd+8VI+66 C0TT3YeJaczurlU5TUtRP3wjP7SLFwv/9bUUiOr5SqBhq2sei3NEHv2TS+AWA8SgdQlY AQeuYymm2/fST66eESJ9nBZUjIfPnQhUn+yFsFB6vJOdl1C/UiDwvkMLM6+n8VjqycBa GDXg5p/Sl+VX4JI3WwKrZm+e420OO8/jVIfrf2pmBJPXtEkRS8haCw+0/saOStdZVVhL fPmaqdLbQg0Xh7tTft8ds+b5D75S+fsWCG9T9LOrAQMU4rScR7AlIGw8xbH6D/X0amZ9 krBA== X-Gm-Message-State: AOAM530bnjERZc/otLlORay14X3+267CRpiR/BcNnm9ZVvyZTWOm3j3T lWrcw1X4Dr8pH4igaYyBzZrZjehbjJad X-Google-Smtp-Source: ABdhPJxlCTx9p7rzRSKY5hPCMXHH9PbCR3/vxNwiQaM6JpLBZ/W3y1yrQSuLj+BwkUwpvNqFCqkE++nkXuKm X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:1909:: with SMTP id er9mr4159347qvb.5.1615399132338; Wed, 10 Mar 2021 09:58:52 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:43 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-27-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 26/34] KVM: arm64: Sort the hypervisor memblocks From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175854_078877_E543F25D X-CRM114-Status: GOOD ( 12.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will soon need to check if a Physical Address belongs to a memblock at EL2, so make sure to sort them so this can be done efficiently. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/reserved_mem.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index fd42705a3c26..83ca23ac259b 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -6,6 +6,7 @@ #include #include +#include #include @@ -18,6 +19,23 @@ static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr); phys_addr_t hyp_mem_base; phys_addr_t hyp_mem_size; +static int cmp_hyp_memblock(const void *p1, const void *p2) +{ + const struct memblock_region *r1 = p1; + const struct memblock_region *r2 = p2; + + return r1->base < r2->base ? -1 : (r1->base > r2->base); +} + +static void __init sort_memblock_regions(void) +{ + sort(hyp_memory, + *hyp_memblock_nr_ptr, + sizeof(struct memblock_region), + cmp_hyp_memblock, + NULL); +} + static int __init register_memblock_regions(void) { struct memblock_region *reg; @@ -29,6 +47,7 @@ static int __init register_memblock_regions(void) hyp_memory[*hyp_memblock_nr_ptr] = *reg; (*hyp_memblock_nr_ptr)++; } + sort_memblock_regions(); return 0; } From patchwork Wed Mar 10 17:57:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E66EC433E6 for ; Wed, 10 Mar 2021 18:11:09 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 696F764FB2 for ; Wed, 10 Mar 2021 18:11:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 696F764FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=70ARFvTlLXIb89fSHwCilwmy2h98/PIqb1CKxI8hbRw=; b=q+z3eD9owucOkH 7p+WR3TLfo+rvP6ONNUXekcrHn5672CxhFKIDejuxELZPDYRrP3o/Zr2x/5GjOoY4Gv3Hv+cbVTNy od++qTrbPD0fe2LoBQiC/mVI17GLqnBNxQHEXQrOqXato4aVc6yVTbORZig3vm8bHFVAe81tU9nii kxeHQHZNEZqfIovi0iVRsW+PVovjtX1e5KeMEJd9N6wkQ9xbyIvS/iva5YrrCJQArHdI4cOxu94OI SavbaM0wJ+Q9VvivlxwBW6NkQV3kg7C8oEiZosk6VMzKHbTJ4mlwNGjP7/5EmAZAfsUap2V9B0xGD DiEF0zlgD7JY4gGVct2Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3GY-007SR8-Fh; Wed, 10 Mar 2021 18:09:03 +0000 Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36m-007OSf-4X for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:58:58 +0000 Received: by mail-qk1-x749.google.com with SMTP id a1so13348404qkn.11 for ; Wed, 10 Mar 2021 09:58:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=KHg8GNHRWwdgHORBJFIM4n3Psw8hXYgV1Q7feze9Wwg=; b=r7hm1X+/A4HHjywquwAv8UbYAC4tXhZrRcjw5KRNMuRuugEIl9mZiVAY/05E4lCJ/r mJTLgUN7/+Hr2pV5bAsaKsvRidg2SEkD9/zXw+s5Ztk6sxyrU8HvKe+vTQLw3LG01fOE JHPpmFLbjbPAN+XwZHagqy2Ooi5//8Rg247YAx2wK+3j2Sft1b5cmSGrw/mSyOcP855p 7UnlC90DSABtGbdUnuIuLTbD1MK9ZKjMlWgNcVtYWTqnnHZ+iLJIOXE53XyztiG7sCQz XkxrZ2zyGBLxkidgk/YANYRo9KEhtYj9pTuZiRLmeb68EDE+Dg9j77mf3Oj8CYFTC8Cy 3qWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KHg8GNHRWwdgHORBJFIM4n3Psw8hXYgV1Q7feze9Wwg=; b=T5E4qp2FPl+3LRQT7UATJfcB/fJlXmuRWh2ZrybcCaJGdIFIh5kMRKESxDqCyD1m7d 0fMQ47O1RXD2YNbDn51NbHSkZsGMZuomimyXxbMaHGLd801SqFdSvsDo6G6SoeyHYB1B ma8Zj18kn2JJ1D2hPah4dRPqv9hI+OdpECsSOdgazHqvlw88bN36Mid3xDn/co5igTBM ohXy3vxvzmv3doA+U47xUpRmF6x2V9ckDcj17888lvCqyzh0TzYtHduh/zJCo0ElzUnB nYTWc3zU5J5W64vP+iqfbOO46opesyD73IqS/RjG7z6kYarG2ZuVKP3t5P08dP0nup4r /UaA== X-Gm-Message-State: AOAM532A0xM7xSbwGomY4KaixbnunVKY+hWCQfM1SdPxrUYEWUtHAO9z qE6OCvTQMp5NwswcPE9VTFpmvYtjDExS X-Google-Smtp-Source: ABdhPJxctmL0qh9AUqySqIh+J7wmsFSyJZCllX/E1sFQxbY+Rs5BdkUUA7jDJCdbb2ggAgOkd5NnOmpGsuXf X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:1305:: with SMTP id a5mr4160961qvv.42.1615399134545; Wed, 10 Mar 2021 09:58:54 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:44 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-28-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 27/34] KVM: arm64: Always zero invalid PTEs From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175856_858768_5EFB1B97 X-CRM114-Status: GOOD ( 14.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kvm_set_invalid_pte() currently only clears bit 0 from a PTE because stage2_map_walk_table_post() needs to be able to follow the anchor. In preparation for re-using bits 63-02 from invalid PTEs, make sure to zero it entirely by ensuring to cache the anchor's child upfront. Suggested-by: Will Deacon Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index bdd6e3d4eeb6..f37b4179b880 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -156,10 +156,9 @@ static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_op return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); } -static void kvm_set_invalid_pte(kvm_pte_t *ptep) +static void kvm_clear_pte(kvm_pte_t *ptep) { - kvm_pte_t pte = *ptep; - WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID); + WRITE_ONCE(*ptep, 0); } static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, @@ -443,6 +442,7 @@ struct stage2_map_data { kvm_pte_t attr; kvm_pte_t *anchor; + kvm_pte_t *childp; struct kvm_s2_mmu *mmu; void *memcache; @@ -532,7 +532,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, * There's an existing different valid leaf entry, so perform * break-before-make. */ - kvm_set_invalid_pte(ptep); + kvm_clear_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); mm_ops->put_page(ptep); } @@ -553,7 +553,8 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; - kvm_set_invalid_pte(ptep); + data->childp = kvm_pte_follow(*ptep, data->mm_ops); + kvm_clear_pte(ptep); /* * Invalidate the whole stage-2, as we may have numerous leaf @@ -599,7 +600,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * will be mapped lazily. */ if (kvm_pte_valid(pte)) { - kvm_set_invalid_pte(ptep); + kvm_clear_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); mm_ops->put_page(ptep); } @@ -615,19 +616,24 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, struct stage2_map_data *data) { struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; + kvm_pte_t *childp; int ret = 0; if (!data->anchor) return 0; - mm_ops->put_page(kvm_pte_follow(*ptep, mm_ops)); - mm_ops->put_page(ptep); - if (data->anchor == ptep) { + childp = data->childp; data->anchor = NULL; + data->childp = NULL; ret = stage2_map_walk_leaf(addr, end, level, ptep, data); + } else { + childp = kvm_pte_follow(*ptep, mm_ops); } + mm_ops->put_page(childp); + mm_ops->put_page(ptep); + return ret; } @@ -736,7 +742,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * block entry and rely on the remaining portions being faulted * back lazily. */ - kvm_set_invalid_pte(ptep); + kvm_clear_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, addr, level); mm_ops->put_page(ptep); From patchwork Wed Mar 10 17:57:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31F62C433DB for ; Wed, 10 Mar 2021 18:12:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2B3FC64E77 for ; Wed, 10 Mar 2021 18:12:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B3FC64E77 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=azgca4pySfO1lwNAU3gJDXhE3EpLmj0EnVQZp2Cph3w=; b=rCJF41IynJi7eI OtuEAJIG6VysgIC60amQqonEWp2vyxfPbYfzMTNkNBsqvrGBJRU0w2rzbF6Ox2ybyK2ms9kvhNHV7 6RX9/+ChnYFBzVWUFGXgYkjGPWhuK6GutuBBP8KuaBnjY0Tu9t2HBN9eGqYvVqorTkxgnAbJgRXSI 5VqEaadyc8lfuURAaxrhIXDrG56K8YGRkXUC3TVKL4h7F16vBtV/HF7W7R1BK0sabD+mmPBFYPtOE 2RpX9UXUV9yQUtCcXPcujAFjJaUaiVNrnGL7PejVbQnXt4G4ga4pkWI2dJb4rrhCfvhKyjTc6APeq wb/J6W4g63LlC6Q93k4Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3HT-007Sj9-75; Wed, 10 Mar 2021 18:09:59 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36o-007OTz-C5 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:00 +0000 Received: by mail-qt1-x849.google.com with SMTP id v19so13522779qtw.19 for ; Wed, 10 Mar 2021 09:58:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=oJAEWGythqEowVh0n8HMKtwcniboC/S1aRtheNM7IbE=; b=cJzFnkrAT9fGjpNBXYhuuJPuJr8xf3G3JewcI+WPvURd6Io9t0i5rlaNRjeWqYPx/Y eka8vAXibXBnvp1mXYExjSyRNZKktSJI4VnhbJcjj7XalIQPeivVnmJRgECvhF7yYktc QH+jTNqGz17MxFaT24apKPJgDPJ84YCqDw0AQKQoJBDrUW4FUxuafKRVaqUASQCJgt4W zDXjXTbKU655Hi06aEooIs+AEcvNKClEJAQFk4k07xLpfajULWPSGUyKHd3q96HcDkfC OiBee+YVNUnM9DkSFpdLAcxla0asRJhapKsob7jGQ++zkQptlutrfxjQG2DUIKPAUHlR +Avw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=oJAEWGythqEowVh0n8HMKtwcniboC/S1aRtheNM7IbE=; b=f9F08UbauTVQarnBG4ny6B7+I3WzvyiWkYeJKVMGMQH7XEt/l3Q6uoICEeVq8mv2DQ RL6YBqKN3CQH221itAN+l6snFnjOyF2299VF4sVCnyKlfoUWZ1ii3baGhCIIMc1g9182 c81Lo+87ZIMJ3pCNf4m2hrwFAsRSoqakgxUoIPGhGCGMx9vwaUBdLoTHpRt9+45D1FW9 DN9nFMrMktDYUG3RBJ0ESsX7ZAh5vjhoOaUi1jxhv+rTgLA4V5hqCYm2a/q2oy3z1321 mTb4CYLXLRRYOPGlqdOINzihJ3SpsMIxvUhPcLIWZCNOLcBgbdtg2bwmaS5EeVzb6jBG eK6w== X-Gm-Message-State: AOAM5300Pz1VDcFJW9X1HplM4AtVz9fYq8uDLvN3Ek05gcmGM9bzr33K KjzI4v0k4RfR5OUZQFxT2VN/ig8RE9ag X-Google-Smtp-Source: ABdhPJzEM5vQ7ta5jwY/6r7uIre+Ch2NIuo2JzdxZFwKwwT7Uiy3AfJO64LToZY/HChaNQ0THAG1P5zpU4GU X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:b92f:: with SMTP id u47mr4092501qvf.8.1615399136610; Wed, 10 Mar 2021 09:58:56 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:45 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-29-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 28/34] KVM: arm64: Use page-table to track page ownership From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175858_785725_C734FE28 X-CRM114-Status: GOOD ( 26.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As the host stage 2 will be identity mapped, all the .hyp memory regions and/or memory pages donated to protected guestis will have to marked invalid in the host stage 2 page-table. At the same time, the hypervisor will need a way to track the ownership of each physical page to ensure memory sharing or donation between entities (host, guests, hypervisor) is legal. In order to enable this tracking at EL2, let's use the host stage 2 page-table itself. The idea is to use the top bits of invalid mappings to store the unique identifier of the page owner. The page-table owner (the host) gets identifier 0 such that, at boot time, it owns the entire IPA space as the pgd starts zeroed. Provide kvm_pgtable_stage2_set_owner() which allows to modify the ownership of pages in the host stage 2. It re-uses most of the map() logic, but ends up creating invalid mappings instead. This impacts how we do refcount as we now need to count invalid mappings when they are used for ownership tracking. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 21 +++++++ arch/arm64/kvm/hyp/pgtable.c | 92 ++++++++++++++++++++++++---- 2 files changed, 101 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 4ae19247837b..b09af4612656 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -238,6 +238,27 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, void *mc); +/** + * kvm_pgtable_stage2_set_owner() - Annotate invalid mappings with metadata + * encoding the ownership of a page in the + * IPA space. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Intermediate physical address at which to place the annotation. + * @size: Size of the IPA range to annotate. + * @mc: Cache of pre-allocated and zeroed memory from which to allocate + * page-table pages. + * @owner_id: Unique identifier for the owner of the page. + * + * The page-table owner has identifier 0. This function can be used to mark + * portions of the IPA space as owned by other entities. When a stage 2 is used + * with identity-mappings, these annotations allow to use the page-table data + * structure as a simple rmap. + * + * Return: 0 on success, negative error code on failure. + */ +int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, + void *mc, u32 owner_id); + /** * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table. * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index f37b4179b880..e4670b639726 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -48,6 +48,8 @@ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) +#define KVM_INVALID_PTE_OWNER_MASK GENMASK(63, 32) + struct kvm_pgtable_walk_data { struct kvm_pgtable *pgt; struct kvm_pgtable_walker *walker; @@ -186,6 +188,11 @@ static kvm_pte_t kvm_init_valid_leaf_pte(u64 pa, kvm_pte_t attr, u32 level) return pte; } +static kvm_pte_t kvm_init_invalid_leaf_owner(u32 owner_id) +{ + return FIELD_PREP(KVM_INVALID_PTE_OWNER_MASK, owner_id); +} + static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag) @@ -440,6 +447,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) struct stage2_map_data { u64 phys; kvm_pte_t attr; + u32 owner_id; kvm_pte_t *anchor; kvm_pte_t *childp; @@ -506,6 +514,24 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, return 0; } +static bool stage2_is_permission_change(kvm_pte_t old, kvm_pte_t new) +{ + if (!kvm_pte_valid(old) || !kvm_pte_valid(new)) + return false; + + return !((old ^ new) & (~KVM_PTE_LEAF_ATTR_S2_PERMS)); +} + +static bool stage2_pte_is_counted(kvm_pte_t pte) +{ + /* + * The refcount tracks valid entries as well as invalid entries if they + * encode ownership of a page to another entity than the page-table + * owner, whose id is 0. + */ + return !!pte; +} + static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) @@ -517,28 +543,36 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, phys, level)) return -E2BIG; - new = kvm_init_valid_leaf_pte(phys, data->attr, level); - if (kvm_pte_valid(old)) { + if (kvm_pte_valid(data->attr)) + new = kvm_init_valid_leaf_pte(phys, data->attr, level); + else + new = kvm_init_invalid_leaf_owner(data->owner_id); + + if (stage2_pte_is_counted(old)) { /* * Skip updating the PTE if we are trying to recreate the exact * same mapping or only change the access permissions. Instead, * the vCPU will exit one more time from guest if still needed * and then go through the path of relaxing permissions. */ - if (!((old ^ new) & (~KVM_PTE_LEAF_ATTR_S2_PERMS))) + if (stage2_is_permission_change(old, new)) return -EAGAIN; /* - * There's an existing different valid leaf entry, so perform - * break-before-make. + * Clear the existing PTE, and perform break-before-make with + * TLB maintenance if it was valid. */ kvm_clear_pte(ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); + if (kvm_pte_valid(old)) { + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, + level); + } mm_ops->put_page(ptep); } smp_store_release(ptep, new); - mm_ops->get_page(ptep); + if (stage2_pte_is_counted(new)) + mm_ops->get_page(ptep); data->phys += granule; return 0; } @@ -574,7 +608,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int ret; if (data->anchor) { - if (kvm_pte_valid(pte)) + if (stage2_pte_is_counted(pte)) mm_ops->put_page(ptep); return 0; @@ -599,9 +633,10 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, * a table. Accesses beyond 'end' that fall within the new table * will be mapped lazily. */ - if (kvm_pte_valid(pte)) { + if (stage2_pte_is_counted(pte)) { kvm_clear_pte(ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); + if (kvm_pte_valid(pte)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); mm_ops->put_page(ptep); } @@ -683,6 +718,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .mmu = pgt->mmu, .memcache = mc, .mm_ops = pgt->mm_ops, + .owner_id = 0, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -696,6 +732,33 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, if (ret) return ret; + /* Set the valid flag to distinguish with the set_owner() path. */ + map_data.attr |= KVM_PTE_VALID; + + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + dsb(ishst); + return ret; +} + +int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, + void *mc, u32 owner_id) +{ + int ret; + struct stage2_map_data map_data = { + .mmu = pgt->mmu, + .memcache = mc, + .mm_ops = pgt->mm_ops, + .owner_id = owner_id, + .attr = 0, + }; + struct kvm_pgtable_walker walker = { + .cb = stage2_map_walker, + .flags = KVM_PGTABLE_WALK_TABLE_PRE | + KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_TABLE_POST, + .arg = &map_data, + }; + ret = kvm_pgtable_walk(pgt, addr, size, &walker); dsb(ishst); return ret; @@ -725,8 +788,13 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, kvm_pte_t pte = *ptep, *childp = NULL; bool need_flush = false; - if (!kvm_pte_valid(pte)) + if (!kvm_pte_valid(pte)) { + if (stage2_pte_is_counted(pte)) { + kvm_clear_pte(ptep); + mm_ops->put_page(ptep); + } return 0; + } if (kvm_pte_table(pte, level)) { childp = kvm_pte_follow(pte, mm_ops); @@ -948,7 +1016,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; - if (!kvm_pte_valid(pte)) + if (!stage2_pte_is_counted(pte)) return 0; mm_ops->put_page(ptep); From patchwork Wed Mar 10 17:57:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F5B0C433E0 for ; Wed, 10 Mar 2021 18:13:00 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72AD464FB2 for ; Wed, 10 Mar 2021 18:12:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72AD464FB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KMtQ7aWH9ipi2ybAK29g9KEUoGd1UxOUcITGjc4O2yg=; b=p7xmSRHxipi6T5 oRS/+NeduQml/KbbgA/GoOB5zKg9D4y9WonIX2irwjYHSsulQxcbOl0uWqhj3M1TKXtiquXQg/5pX Eh+saVLwuzucoGy0/lT0/0OseSpQWRh18ktFffZYqkPSQ+ZLD6S153xNglrZB6cJq03ia4I/mZCnc 5yWcIlwDj1XW6hgfBuHTr1IKq4xG/KmI+C/TSMqVV4oK0f3dYSmM9lWmpgjlJP2SmJvtSFJ/wKN7k LDJB7UQ3ozdkY7VhB4sjw8MwS6Vp/1sRuezWPLT5d1pgcfhX2ief7epBGtvs+ZxZSJ4EEU7huFxmi XU+sSjEfKqnWnChrqp/g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3IM-007SzK-6Y; Wed, 10 Mar 2021 18:10:54 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36q-007OVR-D5 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:02 +0000 Received: by mail-qv1-xf49.google.com with SMTP id u8so13275817qvm.5 for ; Wed, 10 Mar 2021 09:58:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=6iNTe0mF+hjFWLgudHI/zWSf2clb7t9N+GDYgm7EigU=; b=uaCdrsKfFSgEFJjwltC9RmF59Ae7yc016S997ZvqjUKBgk8suIeUI8tNo9lXnjVcFu 2TLmSeOAjSDRESJ/NLQB7NEcltJ8lm7/1lgfPzBhrxE0X30r58dk52bDkKiWrJ0U/ly4 RFLMCW7FhB3lzNXpvkxGbEEWw2VsVX/v89kRWom2NPXRitUUuaSEtxub5xzCfAQj/QEv ipn9/xy2+PZHU7mfi6Eelvf9mWH+vr113PBdfw6Sxgv9/ZaPo0n+ZxYF95KSUq8n96zJ 1xkQHpOlRKYCDEukUW/4tZsL1k+hh9mrB1hZ/5M5eDoytop/zTSDu4VEhsAUZqUHU6W7 SHUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6iNTe0mF+hjFWLgudHI/zWSf2clb7t9N+GDYgm7EigU=; b=WIcFMWmz5+AFEQCPsTyAOOCBmRDtIMISJ8vQUN4b8TLth/X0p6N4MtDKhI4KInkKH3 Mp8xkmgNnY3BD0qnwddgkvsHt//hU+K3ZNYUxWczzuYpxoPpGtCVeqCL/XQ+gmZUjszg /2qXxN0x1Tf7sa5H+FPMXwZG9yxWzj82nBuwsJpcZdQvy77kswiXCZ/p5G6GUcOvLU8x H0zQNM8j23Kazdgfgtd0gXe2f96qaBO2IqzHfkFBsO9S89nP8+XaU3V6tgsVezUsoOak D0VV5W7V44+y0GeZx6MW/CdFlmXOKk8rZKiUPYaBJ094ozVq1Ircy0hROsquisF4kzcw D4zA== X-Gm-Message-State: AOAM530WS9aJ48mRIfLYKQX9yS9D3S1nusj26qaZ+6aTPn0O7EXN9P3k SD+Id3o7S/esbKbbvfm4bs7JktkFkNOz X-Google-Smtp-Source: ABdhPJyDfHqUWFaXueHB0jBN/4Y6pf8dWGEZxgcpIy7CVo3fvkDLvVm5LW+MkE2AJvNQ0cA1KndyOwMjPQzd X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:f092:: with SMTP id g18mr4249565qvk.11.1615399138690; Wed, 10 Mar 2021 09:58:58 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:46 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-30-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 29/34] KVM: arm64: Refactor stage2_map_set_prot_attr() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175900_671693_61C5DD1C X-CRM114-Status: GOOD ( 11.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to ease its re-use in other code paths, refactor stage2_map_set_prot_attr() to not depend on a stage2_map_data struct. No functional change intended. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index e4670b639726..c16e0306dd9a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -489,8 +489,7 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) return vtcr; } -static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, - struct stage2_map_data *data) +static int stage2_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep) { bool device = prot & KVM_PGTABLE_PROT_DEVICE; kvm_pte_t attr = device ? PAGE_S2_MEMATTR(DEVICE_nGnRE) : @@ -510,7 +509,8 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF; - data->attr = attr; + *ptep = attr; + return 0; } @@ -728,7 +728,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .arg = &map_data, }; - ret = stage2_map_set_prot_attr(prot, &map_data); + ret = stage2_set_prot_attr(prot, &map_data.attr); if (ret) return ret; From patchwork Wed Mar 10 17:57:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52E6EC433E0 for ; Wed, 10 Mar 2021 18:13:42 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DEFCF64E77 for ; Wed, 10 Mar 2021 18:13:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DEFCF64E77 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Nb+B8R5CbUdCfAy++2G3dl1THyInrNv15zp7A9WwWLk=; b=YSbRS2f0F2VdXu JiTLYuG+iBIXHgKGkSg5GznRnnVV3F/c7k/yL9VlKrTZiJv0+yA00TLfsOejAibNU9pLm4PwrKL9U 4kl3M2dGzFLuX356Now52JFHmGH8SXv9Q2tCD74MYv8eRyRXM3AKwyhx6DDq4ZGz7Te4231GpzR/A 4MrD29SPlJ65WHJGs8suaoOtQ+bgtULr0PBscdwYq3qsf2rNhdsTLKDi+TdksCrGiXgiXV7USn/Gc eIMRr3Z9jMXMYfummJuPD8EI3FaLDXKEb7iYNN8S9up2GM49k9eQnCnpd9zKdKZn6nVfE9f8eaVbu tfjQzTLVPrVlBCgRVwtw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3J4-007TEz-ON; Wed, 10 Mar 2021 18:11:39 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36s-007OWZ-81 for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:04 +0000 Received: by mail-qt1-x849.google.com with SMTP id m8so11452571qtp.14 for ; Wed, 10 Mar 2021 09:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=v22OPvpSkxqoB2GxDbJ4rnO0RdXvHHXz0HAMkqu2tKU=; b=nYF7az3GofIlNu5zNxCiTcHwvkEdtos7U9v0280LOmKI6mg0GPAeX5KEzv/BtqLLFj 6v+C9HZnCnX0J7yNbu0L+ew4Uncbw3RBqDvnK3sOCGPtm7DRIWLSkId7+QUq/uz9VhL/ BDCplnG3XhLHnkyKSaAFm4ywblHwJZXVVTU8Jk3EcedqOBR8YiWyQpF2qvweyNSolH/A NqsGfVTnQFG/Lb84IJ7Kq3FpYSVVXdrg9uSiu4sZ+FTbfeQpRGUaGrlvtOEAeFojuKLY qh03mESFMb4ZBlUZgOqEbw4/vV3dpokD8PMpLKxrMK/udtM9HG9xXCWlU2COvR1Luin7 gtiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=v22OPvpSkxqoB2GxDbJ4rnO0RdXvHHXz0HAMkqu2tKU=; b=EHls1idqwGvdRRfBIp2XPtoOWoqMGWT3MgzYUoQ8Sg332+TBBHyoQu+3Gtgs2Mnu/v ib34ZeW5FpXvDquCONXWQQkmZlDZDVEbULkLRe7BjqshuHrJOiZgkf3b2MAq+PEEInHH BHx5xgI3U9edYUZQ2pWAuCXZa7dwIVP4mw1/CbAhQg+60LEt/dqc/7erAiKAFeSsQmYj SAhDTczgytkgQB2u6zNQdsbxli85k/JEmY6dRq7c2KTLqED1YFDuv+Qrk5RZIqGAFYd+ 1qxz2/kVDdxwBQ1rJXabhAUnCO07JxlYPclFbCJQV9x8r3B6tI5FD4b87YOMdTyvXqxb Cr/A== X-Gm-Message-State: AOAM531hg/Lu8HtEMoeKXmCQyUEGtVJl2YVckcd38Sa/2Zau66Qfs2k5 tq3zJyYjq9zbu4jyZAxPngYXoPaPaxDW X-Google-Smtp-Source: ABdhPJyp3d83D9IJZw5miKqWsWt+Sc6zcBaNVvFInoqRG/CRUiZ7DZZiE71n712cDvkDiwaqnfGCK7uI+r3s X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:553:: with SMTP id ci19mr4103466qvb.18.1615399140822; Wed, 10 Mar 2021 09:59:00 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:47 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-31-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 30/34] KVM: arm64: Add kvm_pgtable_stage2_find_range() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175902_714931_8105D778 X-CRM114-Status: GOOD ( 20.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since the host stage 2 will be identity mapped, and since it will own most of memory, it would preferable for performance to try and use large block mappings whenever that is possible. To ease this, introduce a new helper in the KVM page-table code which allows to search for large ranges of available IPA space. This will be used in the host memory abort path to greedily idmap large portion of the PA space. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 30 ++++++++++ arch/arm64/kvm/hyp/pgtable.c | 90 +++++++++++++++++++++++++++- 2 files changed, 117 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index b09af4612656..477bf10c48a9 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -94,6 +94,16 @@ enum kvm_pgtable_prot { #define PAGE_HYP_RO (KVM_PGTABLE_PROT_R) #define PAGE_HYP_DEVICE (PAGE_HYP | KVM_PGTABLE_PROT_DEVICE) +/** + * struct kvm_mem_range - Range of Intermediate Physical Addresses + * @start: Start of the range. + * @end: End of the range. + */ +struct kvm_mem_range { + u64 start; + u64 end; +}; + /** * enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk. * @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid @@ -398,4 +408,24 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, struct kvm_pgtable_walker *walker); +/** + * kvm_pgtable_stage2_find_range() - Find a range of Intermediate Physical + * Addresses with compatible permission + * attributes. + * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init(). + * @addr: Address that must be covered by the range. + * @prot: Protection attributes that the range must be compatible with. + * @range: Range structure used to limit the search space at call time and + * that will hold the result. + * + * The offset of @addr within a page is ignored. An existing mapping is defined + * as compatible with @prot if it is invalid and not owned by another entity, or + * if its permission attributes are strictly similar to @prot and it has no + * software bits set. + * + * Return: 0 on success, negative error code on failure. + */ +int kvm_pgtable_stage2_find_range(struct kvm_pgtable *pgt, u64 addr, + enum kvm_pgtable_prot prot, + struct kvm_mem_range *range); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c16e0306dd9a..f20287bb3e41 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -48,6 +48,8 @@ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) +#define KVM_PTE_LEAF_ATTR_S2_RES GENMASK(58, 55) + #define KVM_INVALID_PTE_OWNER_MASK GENMASK(63, 32) struct kvm_pgtable_walk_data { @@ -69,10 +71,8 @@ static u64 kvm_granule_size(u32 level) return BIT(kvm_granule_shift(level)); } -static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level) +static bool kvm_level_support_block_mappings(u32 level) { - u64 granule = kvm_granule_size(level); - /* * Reject invalid block mappings and don't bother with 4TB mappings for * 52-bit PAs. @@ -80,6 +80,16 @@ static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level) if (level == 0 || (PAGE_SIZE != SZ_4K && level == 1)) return false; + return true; +} + +static bool kvm_block_mapping_supported(u64 addr, u64 end, u64 phys, u32 level) +{ + u64 granule = kvm_granule_size(level); + + if (!kvm_level_support_block_mappings(level)) + return false; + if (granule > (end - addr)) return false; @@ -1042,3 +1052,77 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; } + +#define KVM_PTE_LEAF_S2_COMPAT_MASK (KVM_PTE_LEAF_ATTR_S2_PERMS | \ + KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR | \ + KVM_PTE_LEAF_ATTR_S2_RES) + +static int stage2_check_permission_walker(u64 addr, u64 end, u32 level, + kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + kvm_pte_t old_attr, pte = *ptep, *new_attr = arg; + + /* + * Compatible mappings are either invalid and owned by the page-table + * owner (whose id is 0), or valid with matching permission attributes. + */ + if (kvm_pte_valid(pte)) { + old_attr = pte & KVM_PTE_LEAF_S2_COMPAT_MASK; + if (old_attr != *new_attr) + return -EEXIST; + } else if (pte) { + return -EEXIST; + } + + return 0; +} + +int kvm_pgtable_stage2_find_range(struct kvm_pgtable *pgt, u64 addr, + enum kvm_pgtable_prot prot, + struct kvm_mem_range *range) +{ + kvm_pte_t attr; + struct kvm_pgtable_walker check_perm_walker = { + .cb = stage2_check_permission_walker, + .flags = KVM_PGTABLE_WALK_LEAF, + .arg = &attr, + }; + u64 granule, start, end; + u32 level; + int ret; + + ret = stage2_set_prot_attr(prot, &attr); + if (ret) + return ret; + attr &= KVM_PTE_LEAF_S2_COMPAT_MASK; + + for (level = pgt->start_level; level < KVM_PGTABLE_MAX_LEVELS; level++) { + granule = kvm_granule_size(level); + start = ALIGN_DOWN(addr, granule); + end = start + granule; + + if (!kvm_level_support_block_mappings(level)) + continue; + + if (start < range->start || range->end < end) + continue; + + /* + * Check the presence of existing mappings with incompatible + * permissions within the current block range, and try one level + * deeper if one is found. + */ + ret = kvm_pgtable_walk(pgt, start, granule, &check_perm_walker); + if (ret != -EEXIST) + break; + } + + if (!ret) { + range->start = start; + range->end = end; + } + + return ret; +} From patchwork Wed Mar 10 17:57:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC1B9C433DB for ; Wed, 10 Mar 2021 18:14:03 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 887FA64E5C for ; Wed, 10 Mar 2021 18:14:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 887FA64E5C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sIIT3CS2zlnbmlaCiHAi+eesr5P0CXz/oHtLVxqFEWw=; b=lNn9uNg4RcRMSs ijTOSi35vUpcYtrSR23xA/zSj0Ddyd1RmcI/53S9C3lTO1kkUrZxQloiQ5U0vLDEfqBteTxmJJwzr t3iGfgZHq/vOam0mZB7Q7jD/ZKVqrcvh0UE3cD38gXC492LOtSZOnkIU8R3TQXbx9S68ft5f79toZ XTuDnKhj90NNpe/3yI2TnCwBCKnUDkiEElg/lPCiOJTotmPQXpR2VEUD+F6SilMQvyEuVcF/npLYT moXdQxgTfimdk7gCkmw9yV3L6MHGRGFQgi1MG2FuBeCNhF3O98JexsN/i13hYwLqP9jMGERvuAKX0 p5RPZlRMMIEKJryzvA4A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Jc-007TPQ-6U; Wed, 10 Mar 2021 18:12:12 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36u-007OXh-BV for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:07 +0000 Received: by mail-wm1-x349.google.com with SMTP id n2so2992187wmi.2 for ; Wed, 10 Mar 2021 09:59:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CYUgjPCTwAZhevieC0Mxy3/co0eV77+l7/T8QyaZKic=; b=ifxq8HRvBBXoYKSgbPXr1K2yfYOPuL/eXvsiyCqeXHBkB28UWHtx9CdqpC36R6+m8w idCph9BKRw2op3MnltdECYmpnUtlZ77hrccyIM+7OLpQDfQR5HtOpfmzHnemhGhrCI8c mYZHRUBN7yfkZMu8DFZLtLMdT+e2Az7LF0LGclxrPQKNeMyt8Sd2mEHi6OdiTQmTXMeQ nFeAaAQtWv4klLWa12xwT5I5EQx+qVetLZo2jxNVMgyuEM/IcrMpL8MXJIinTns79FJp pwg6RzjyxJp92aeW2amxJz9YSV5nMrtekAkh25T9R11gEiEG1bHQ1xxGgunPb8hL0xa8 jbsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CYUgjPCTwAZhevieC0Mxy3/co0eV77+l7/T8QyaZKic=; b=jz/FV3AhOt1tIqvGUlMhRCTAqA8Ow9WwQ6U2R7kMPAIOyeHTvIoGoTrfp4HKjnNoWd PaeZFdPvj/zhwBst0JQnq8h2hhKcM/sBC+dn6lgkbA5aSErpp5PHaN1aKwr2+OJQTWV1 1/2IzxA2XmbaHAJF80n6Y8gJiGUwaDMQRo2UR2XTQzSR/zjzzg7dCzmAh9lO28RaJ5Sd 8cPIptUpzqsRKhMfPxaGqWFnrXf1qvucqljBW9+RLV/D11VbU8+yPXhaQZWVQGoDPGWL fiHImBflPirDm6u846sQXbp2sugjVwfGjpokyPoYhtBw+d+6F56oU7K2BO51cwqMAuAk I55w== X-Gm-Message-State: AOAM5324kRQS/sKc+xziMQmQwRAT1wJBke4Hpn9kpdPCAK7VjWJRWS8P tPKuHvOL66OSVgx0pxJsgROoTcnQ9+1F X-Google-Smtp-Source: ABdhPJwrylHLc9EjnEo3QMwuGtetyPOrqoXFSRms3vEYNxR/2gbPzd3IMyAxX6mUFbpiO59A/QMaavzWC5bk X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:4ca9:: with SMTP id g41mr4572026wmp.150.1615399142908; Wed, 10 Mar 2021 09:59:02 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:48 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-32-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 31/34] KVM: arm64: Wrap the host with a stage 2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175904_646664_EECA3C85 X-CRM114-Status: GOOD ( 30.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM runs in protected nVHE mode, make use of a stage 2 page-table to give the hypervisor some control over the host memory accesses. The host stage 2 is created lazily using large block mappings if possible, and will default to page mappings in absence of a better solution. From this point on, memory accesses from the host to protected memory regions (e.g. marked PROT_NONE) are fatal and lead to hyp_panic(). Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/include/asm/kvm_cpufeature.h | 2 + arch/arm64/kernel/image-vars.h | 3 + arch/arm64/kvm/arm.c | 10 + arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 34 +++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 1 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 246 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 5 + arch/arm64/kvm/hyp/nvhe/switch.c | 7 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 4 +- 12 files changed, 319 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mem_protect.c diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 6dce860f8bca..b127af02bd45 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -61,6 +61,7 @@ #define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16 #define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 #define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 +#define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h index d34f85cba358..74043a149322 100644 --- a/arch/arm64/include/asm/kvm_cpufeature.h +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -15,3 +15,5 @@ #endif KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR0_EL1, arm64_ftr_reg_id_aa64mmfr0_el1) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR1_EL1, arm64_ftr_reg_id_aa64mmfr1_el1) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 940c378fa837..d5dc2b792651 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -131,6 +131,9 @@ KVM_NVHE_ALIAS(__hyp_bss_end); KVM_NVHE_ALIAS(__hyp_rodata_start); KVM_NVHE_ALIAS(__hyp_rodata_end); +/* pKVM static key */ +KVM_NVHE_ALIAS(kvm_protected_mode_initialized); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index d474eec606a3..7e6a81079652 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1889,12 +1889,22 @@ static int init_hyp_mode(void) return err; } +void _kvm_host_prot_finalize(void *discard) +{ + WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)); +} + static int finalize_hyp_mode(void) { if (!is_protected_kvm_enabled()) return 0; + /* + * Flip the static key upfront as that may no longer be possible + * once the host stage 2 is installed. + */ static_branch_enable(&kvm_protected_mode_initialized); + on_each_cpu(_kvm_host_prot_finalize, NULL, 1); return 0; } diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h new file mode 100644 index 000000000000..d293cb328cc4 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#ifndef __KVM_NVHE_MEM_PROTECT__ +#define __KVM_NVHE_MEM_PROTECT__ +#include +#include +#include +#include +#include + +struct host_kvm { + struct kvm_arch arch; + struct kvm_pgtable pgt; + struct kvm_pgtable_mm_ops mm_ops; + hyp_spinlock_t lock; +}; +extern struct host_kvm host_kvm; + +int __pkvm_prot_finalize(void); +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool); +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); + +static __always_inline void __load_host_stage2(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + else + write_sysreg(0, vttbr_el2); +} +#endif /* __KVM_NVHE_MEM_PROTECT__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index e204ea77ab27..ce49795324a7 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -14,7 +14,7 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o setup.o mm.o + cache.o cpufeature.o setup.o mm.o mem_protect.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index a50ad9e9fc05..c164045af238 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -119,6 +119,7 @@ alternative_else_nop_endif /* Invalidate the stale TLBs from Bootloader */ tlbi alle2 + tlbi vmalls12e1 dsb sy /* diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index ae6503c9be15..f47028d3fd0a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -13,6 +13,7 @@ #include #include +#include #include #include @@ -151,6 +152,10 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); } +static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) +{ + cpu_reg(host_ctxt, 1) = __pkvm_prot_finalize(); +} typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -174,6 +179,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_cpu_set_vector), HANDLE_FUNC(__pkvm_create_mappings), HANDLE_FUNC(__pkvm_create_private_mapping), + HANDLE_FUNC(__pkvm_prot_finalize), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) @@ -226,6 +232,11 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; + case ESR_ELx_EC_IABT_LOW: + fallthrough; + case ESR_ELx_EC_DABT_LOW: + handle_host_mem_abort(host_ctxt); + break; default: hyp_panic(); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c new file mode 100644 index 000000000000..5c88a325e6fc --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -0,0 +1,246 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include + +extern unsigned long hyp_nr_cpus; +struct host_kvm host_kvm; + +struct hyp_pool host_s2_mem; +struct hyp_pool host_s2_dev; + +static void *host_s2_zalloc_pages_exact(size_t size) +{ + return hyp_alloc_pages(&host_s2_mem, get_order(size)); +} + +static void *host_s2_zalloc_page(void *pool) +{ + return hyp_alloc_pages(pool, 0); +} + +static int prepare_s2_pools(void *mem_pgt_pool, void *dev_pgt_pool) +{ + unsigned long nr_pages, pfn; + int ret; + + pfn = hyp_virt_to_pfn(mem_pgt_pool); + nr_pages = host_s2_mem_pgtable_pages(); + ret = hyp_pool_init(&host_s2_mem, pfn, nr_pages, 0); + if (ret) + return ret; + + pfn = hyp_virt_to_pfn(dev_pgt_pool); + nr_pages = host_s2_dev_pgtable_pages(); + ret = hyp_pool_init(&host_s2_dev, pfn, nr_pages, 0); + if (ret) + return ret; + + host_kvm.mm_ops = (struct kvm_pgtable_mm_ops) { + .zalloc_pages_exact = host_s2_zalloc_pages_exact, + .zalloc_page = host_s2_zalloc_page, + .phys_to_virt = hyp_phys_to_virt, + .virt_to_phys = hyp_virt_to_phys, + .page_count = hyp_page_count, + .get_page = hyp_get_page, + .put_page = hyp_put_page, + }; + + return 0; +} + +static void prepare_host_vtcr(void) +{ + u32 parange, phys_shift; + u64 mmfr0, mmfr1; + + mmfr0 = arm64_ftr_reg_id_aa64mmfr0_el1.sys_val; + mmfr1 = arm64_ftr_reg_id_aa64mmfr1_el1.sys_val; + + /* The host stage 2 is id-mapped, so use parange for T0SZ */ + parange = kvm_get_parange(mmfr0); + phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange); + + host_kvm.arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); +} + +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool) +{ + struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; + int ret; + + prepare_host_vtcr(); + hyp_spin_lock_init(&host_kvm.lock); + + ret = prepare_s2_pools(mem_pgt_pool, dev_pgt_pool); + if (ret) + return ret; + + ret = kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch, + &host_kvm.mm_ops); + if (ret) + return ret; + + mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd); + mmu->arch = &host_kvm.arch; + mmu->pgt = &host_kvm.pgt; + mmu->vmid.vmid_gen = 0; + mmu->vmid.vmid = 0; + + return 0; +} + +int __pkvm_prot_finalize(void) +{ + struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; + struct kvm_nvhe_init_params *params = this_cpu_ptr(&kvm_init_params); + + params->vttbr = kvm_get_vttbr(mmu); + params->vtcr = host_kvm.arch.vtcr; + params->hcr_el2 |= HCR_VM; + if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) + params->hcr_el2 |= HCR_FWB; + kvm_flush_dcache_to_poc(params, sizeof(*params)); + + write_sysreg(params->hcr_el2, hcr_el2); + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + + /* + * Make sure to have an ISB before the TLB maintenance below but only + * when __load_stage2() doesn't include one already. + */ + asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT)); + + /* Invalidate stale HCR bits that may be cached in TLBs */ + __tlbi(vmalls12e1); + dsb(nsh); + isb(); + + return 0; +} + +static int host_stage2_unmap_dev_all(void) +{ + struct kvm_pgtable *pgt = &host_kvm.pgt; + struct memblock_region *reg; + u64 addr = 0; + int i, ret; + + /* Unmap all non-memory regions to recycle the pages */ + for (i = 0; i < hyp_memblock_nr; i++, addr = reg->base + reg->size) { + reg = &hyp_memory[i]; + ret = kvm_pgtable_stage2_unmap(pgt, addr, reg->base - addr); + if (ret) + return ret; + } + return kvm_pgtable_stage2_unmap(pgt, addr, BIT(pgt->ia_bits) - addr); +} + +static bool find_mem_range(phys_addr_t addr, struct kvm_mem_range *range) +{ + int cur, left = 0, right = hyp_memblock_nr; + struct memblock_region *reg; + phys_addr_t end; + + range->start = 0; + range->end = ULONG_MAX; + + /* The list of memblock regions is sorted, binary search it */ + while (left < right) { + cur = (left + right) >> 1; + reg = &hyp_memory[cur]; + end = reg->base + reg->size; + if (addr < reg->base) { + right = cur; + range->end = reg->base; + } else if (addr >= end) { + left = cur + 1; + range->start = end; + } else { + range->start = reg->base; + range->end = end; + return true; + } + } + + return false; +} + +static inline int __host_stage2_idmap(u64 start, u64 end, + enum kvm_pgtable_prot prot, + struct hyp_pool *pool) +{ + return kvm_pgtable_stage2_map(&host_kvm.pgt, start, end - start, start, + prot, pool); +} + +static int host_stage2_idmap(u64 addr) +{ + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W; + struct kvm_mem_range range; + bool is_memory = find_mem_range(addr, &range); + struct hyp_pool *pool = is_memory ? &host_s2_mem : &host_s2_dev; + int ret; + + if (is_memory) + prot |= KVM_PGTABLE_PROT_X; + + hyp_spin_lock(&host_kvm.lock); + ret = kvm_pgtable_stage2_find_range(&host_kvm.pgt, addr, prot, &range); + if (ret) + goto unlock; + + ret = __host_stage2_idmap(range.start, range.end, prot, pool); + if (is_memory || ret != -ENOMEM) + goto unlock; + + /* + * host_s2_mem has been provided with enough pages to cover all of + * memory with page granularity, so we should never hit the ENOMEM case. + * However, it is difficult to know how much of the MMIO range we will + * need to cover upfront, so we may need to 'recycle' the pages if we + * run out. + */ + ret = host_stage2_unmap_dev_all(); + if (ret) + goto unlock; + + ret = __host_stage2_idmap(range.start, range.end, prot, pool); + +unlock: + hyp_spin_unlock(&host_kvm.lock); + + return ret; +} + +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) +{ + struct kvm_vcpu_fault_info fault; + u64 esr, addr; + int ret = 0; + + esr = read_sysreg_el2(SYS_ESR); + if (!__get_fault_info(esr, &fault)) + hyp_panic(); + + addr = (fault.hpfar_el2 & HPFAR_MASK) << 8; + ret = host_stage2_idmap(addr); + if (ret && ret != -EAGAIN) + hyp_panic(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index c1a3e7e0ebbc..7488f53b0aa2 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -157,6 +158,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; + ret = kvm_host_prepare_stage2(host_s2_mem_pgt_base, host_s2_dev_pgt_base); + if (ret) + goto out; + pkvm_pgtable_mm_ops = (struct kvm_pgtable_mm_ops) { .zalloc_page = hyp_zalloc_hyp_page, .phys_to_virt = hyp_phys_to_virt, diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 979a76cdf9fb..31bc1a843bf8 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -28,6 +28,8 @@ #include #include +#include + /* Non-VHE specific context */ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); @@ -102,11 +104,6 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) write_sysreg(__kvm_hyp_host_vector, vbar_el2); } -static void __load_host_stage2(void) -{ - write_sysreg(0, vttbr_el2); -} - /* Save VGICv3 state on non-VHE systems */ static void __hyp_vgic_save_state(struct kvm_vcpu *vcpu) { diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index fbde89a2c6e8..255a23a1b2db 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -8,6 +8,8 @@ #include #include +#include + struct tlb_inv_context { u64 tcr; }; @@ -43,7 +45,7 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu, static void __tlb_switch_to_host(struct tlb_inv_context *cxt) { - write_sysreg(0, vttbr_el2); + __load_host_stage2(); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { /* Ensure write of the host VMID */ From patchwork Wed Mar 10 17:57:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADE0CC433E0 for ; Wed, 10 Mar 2021 18:14:56 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2122864DC4 for ; Wed, 10 Mar 2021 18:14:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2122864DC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YRDfI/7pAoPCktHF1isE3BClE1i4wGqKBh+OB/0tSVM=; b=cBnEeYyXaPCbQh VqSJjqOFhHP29kropS/kJq4MzZ/BYkxcG1mBZ35fzz2GWdclnWTPc9BE3rqnFFdZAcRSjbUpG9VmZ KukCx+77icu9tb7sy6INDqwPStGPd/uBMxYHQ7mEGPRRAs1Zxgd/6fkNo2fd4LVVkZfuWaVH3pPce FqxJaawylqUR+2CbFiH023P25yF04efHSwGDX07Sj6LgW1lR/9WInNDioqldDeB1LshLYK2iOgXpe TFQ3j7MWatgHrj53pdyOMnvB8VpvEbjF0GvAc9O/GxxeAiKtkrA9P8uMR5LJg69VN1Wvg/n+6Ccn5 +P5hV/8U3rJxhRwPz6+g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3KM-007Tc4-T8; Wed, 10 Mar 2021 18:12:59 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36w-007OZk-0o for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:08 +0000 Received: by mail-wr1-x449.google.com with SMTP id l10so8323706wry.16 for ; Wed, 10 Mar 2021 09:59:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=gpovMkXHvDEd7Q1vbye6zBnrPUcdwhSYcDTZCWLwbWg=; b=EykAkvQxBXiDzdm6FE4Uy5mf7/ey2u/c+U5eLurLHPbE8jOinaMNiPrzd4uPyZwUZL 2r0ZxYaUbj1XS7bDR/Mb1V8xcCBkPAGfIzGS5aU2R8y9FQN6UiYCnReXBwvS+lmODCoR SuLRCs3v6esu/pGnhhVYIhxo5hM8+cAzHRNj6ARFbIIXvItGTWgs46jw+pcEOvB6mOCk tYi+qFS0/ePTkbt0zYmXp2NVpom3ZglGq//Drt94PX6dESV2KONoPjhoxQjYEj1GZ6sA I7bAth+4o+atC2uZJUJNQSU8KOV+souLg6H4/Yv5ckKbF63NE/By69s4N3aq6kMVvb2t 6FKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gpovMkXHvDEd7Q1vbye6zBnrPUcdwhSYcDTZCWLwbWg=; b=bQbipoobOJv/xNXs1LphyI59RCwAyApvO9HIHVlCtnzojOfL4an6go78RzEIJGxXGO WWoLCJ4BQSRk6qr02+ePWNU1GIz+yFfrKm2QDnMhb/0IGjGj62nkPOi+IfPVH9OV79pp EWsb18qsuM+J156+kFKBQmBvjqlVhOpf/WhVsKJ4Z3l4TPTLe7Osi1FnoCRzO51viCVq a8aS3lMEoiZqHR1chya/zVnPkWBDb9ulOCiNtntWn9NWVPThQsly0C5qvQGk9WOwyNxN 7BH39uIqfHET1YOiHGXPNuEGRBgBcDm2EoCNYmjfAiNGi3h+1qItkt3P0LJ38KFFoxyU JKQg== X-Gm-Message-State: AOAM530zi2UFRFzloycfeP1v5xHTrQzD68jw63zJbyEMwe8oUDPJWuXk vZUmj3TJcg+AYLtPEoRWndEcY1iPPifO X-Google-Smtp-Source: ABdhPJziRnNwKB8Ne6xf3o0qOfV9qCZo6Q3H55q9W1Ku8THlZcg0eIahqVQjh17lDsKSyjkP5RcbAP3UO7Jm X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:2247:: with SMTP id a7mr4544145wmm.131.1615399145092; Wed, 10 Mar 2021 09:59:05 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:49 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-33-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 32/34] KVM: arm64: Page-align the .hyp sections From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175906_355485_5B59577C X-CRM114-Status: GOOD ( 13.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will soon unmap the .hyp sections from the host stage 2 in Protected nVHE mode, which obviously works with at least page granularity, so make sure to align them correctly. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kernel/vmlinux.lds.S | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index e96173ce211b..709d2c433c5e 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -15,9 +15,11 @@ #define HYPERVISOR_DATA_SECTIONS \ HYP_SECTION_NAME(.rodata) : { \ + . = ALIGN(PAGE_SIZE); \ __hyp_rodata_start = .; \ *(HYP_SECTION_NAME(.data..ro_after_init)) \ *(HYP_SECTION_NAME(.rodata)) \ + . = ALIGN(PAGE_SIZE); \ __hyp_rodata_end = .; \ } @@ -72,21 +74,14 @@ ENTRY(_text) jiffies = jiffies_64; #define HYPERVISOR_TEXT \ - /* \ - * Align to 4 KB so that \ - * a) the HYP vector table is at its minimum \ - * alignment of 2048 bytes \ - * b) the HYP init code will not cross a page \ - * boundary if its size does not exceed \ - * 4 KB (see related ASSERT() below) \ - */ \ - . = ALIGN(SZ_4K); \ + . = ALIGN(PAGE_SIZE); \ __hyp_idmap_text_start = .; \ *(.hyp.idmap.text) \ __hyp_idmap_text_end = .; \ __hyp_text_start = .; \ *(.hyp.text) \ HYPERVISOR_EXTABLE \ + . = ALIGN(PAGE_SIZE); \ __hyp_text_end = .; #define IDMAP_TEXT \ @@ -322,11 +317,12 @@ SECTIONS #include "image-vars.h" /* - * The HYP init code and ID map text can't be longer than a page each, - * and should not cross a page boundary. + * The HYP init code and ID map text can't be longer than a page each. The + * former is page-aligned, but the latter may not be with 16K or 64K pages, so + * it should also not cross a page boundary. */ -ASSERT(__hyp_idmap_text_end - (__hyp_idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, - "HYP init code too big or misaligned") +ASSERT(__hyp_idmap_text_end - __hyp_idmap_text_start <= PAGE_SIZE, + "HYP init code too big") ASSERT(__idmap_text_end - (__idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, "ID map text too big or misaligned") #ifdef CONFIG_HIBERNATION From patchwork Wed Mar 10 17:57:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2900CC433DB for ; Wed, 10 Mar 2021 18:15:27 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3E12764DC4 for ; Wed, 10 Mar 2021 18:15:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E12764DC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VRlzB8uBj/qjvPGQm3YzZJxyfyETuwC0pLRvPeTRKSU=; b=okuISzam81KGxc uKPQhUN/CxbmRe8NQSVl0VjVXFiRLYCXEnX+RFdzPc8dn4Pc7v/zrzlxjShVo6VOpLcYHltmzVtq5 36EB++Qf8sNho4whBGEOGyae/kbhVAGLrtuqB/1dIimn9SraKTrLVDKdUKF4oFYU9yyMwzOaMJxF0 jonaoTN2FtmhhWfwt9hZkRgRWJKblysEJ+yQf4svvfdkXuqYGasUtG5AC42AG8qo0d5P32bY+0T8s W+2gApNsSUUKqPAFmaJtsG6RkuHi7TGXc2fvPS+Yj0feKQYO5rpZ7wR6EYOWiPEyjRHizvcmPxUJt Q7l/AxGHBLijhyoXMG1Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3L1-007ToS-KO; Wed, 10 Mar 2021 18:13:39 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK36y-007Oba-Ed for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:11 +0000 Received: by mail-wr1-x44a.google.com with SMTP id z17so8301909wrv.23 for ; Wed, 10 Mar 2021 09:59:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4EIGP2sevMQZpUEhabq0vC76HCNx6QyIyPvKeAoptlw=; b=IUQjACmevhkq9a+GZxx+Ua4aB9ez1Bug28fxYTbFk46qxQU32oTjX7DRLGH9l/Il7X FqP7G7yyiuLPnHPSwII8NLcc8xT3okF4VsOdVhhzD45KhzQKh06Ri+jitNWpfqbLNjbH s7lNBTP16IEnv0g+B92roDljctBUXo4JJN1l3q5jmxXNKvjrw4EX2MUG2YyqiyXTFc0v efw7ExzmKbpVMP1Qf7AqwLiKwmBN9zmwUKW/Xurj/d027vYb3d6dQtC+WGD06I7JSB1Y AFj0gkGVkEn9NnTYZK0YUmw6ye4NL3Qrs3yQ4hcfmWyjyO/wYe696C31It5TCXguX2XH NObg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4EIGP2sevMQZpUEhabq0vC76HCNx6QyIyPvKeAoptlw=; b=COkfR5bt+CDeCQDDboI5ocPLnrH1Bf6F0RW2AuTa9qNFrUN4eKCoAvmg19KPtwrwp9 KFlI3WkFHKSzk7bP5pCd4qL6b+vINmSa7Sojgptsej1gFzyby/JhNX6thTyYq8aN2Cc4 yTizy2/xjcpRY1WYXDzoFuxE6Utjzoj8Ebgpht/NAQmimEfZwhX1xcGOczEzEa/fVS4C GGMmTCmM6iuAdtghzYa9Q7hp5H5E9SG4kILQsCJyjJZ6CTDQvz7ThvJemtP5HVt3FlA2 9MIK79Nqk97tS9wFZq0DR9pYFj94ryAoyqhCOL2GPAKIWfDDmliCJ6aLT5Zh4CURwM5B FiUA== X-Gm-Message-State: AOAM530M/xZZ2IlXHHziZH096YylWyPcWeUNShOy4zWckM0ecpe6h9C6 Th2eyLsFULre3x3Txdua5ATojggHNgbm X-Google-Smtp-Source: ABdhPJxqMXK+TnYtjOGnCmi+0PPkd0E3rU72V4cOYCFhyuU5o0IeGdkRrDs5euTQUF4Pwmy1O7cEd08Y9MWQ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6000:1803:: with SMTP id m3mr4806187wrh.50.1615399147126; Wed, 10 Mar 2021 09:59:07 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:50 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-34-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 33/34] KVM: arm64: Disable PMU support in protected mode From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175908_815056_73BB1BCE X-CRM114-Status: GOOD ( 14.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The host currently writes directly in EL2 per-CPU data sections from the PMU code when running in nVHE. In preparation for unmapping the EL2 sections from the host stage 2, disable PMU support in protected mode as we currently do not have a use-case for it. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/perf.c | 3 ++- arch/arm64/kvm/pmu.c | 8 ++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 739164324afe..8f860ae56bb7 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -55,7 +55,8 @@ int kvm_perf_init(void) * hardware performance counters. This could ensure the presence of * a physical PMU and CONFIG_PERF_EVENT is selected. */ - if (IS_ENABLED(CONFIG_ARM_PMU) && perf_num_counters() > 0) + if (IS_ENABLED(CONFIG_ARM_PMU) && perf_num_counters() > 0 + && !is_protected_kvm_enabled()) static_branch_enable(&kvm_arm_pmu_available); return perf_register_guest_info_callbacks(&kvm_guest_cbs); diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c index faf32a44ba04..03a6c1f4a09a 100644 --- a/arch/arm64/kvm/pmu.c +++ b/arch/arm64/kvm/pmu.c @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp_sym(kvm_host_data); - if (!ctx || !kvm_pmu_switch_needed(attr)) + if (!kvm_arm_support_pmu_v3() || !ctx || !kvm_pmu_switch_needed(attr)) return; if (!attr->exclude_host) @@ -49,7 +49,7 @@ void kvm_clr_pmu_events(u32 clr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp_sym(kvm_host_data); - if (!ctx) + if (!kvm_arm_support_pmu_v3() || !ctx) return; ctx->pmu_events.events_host &= ~clr; @@ -172,7 +172,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu) struct kvm_host_data *host; u32 events_guest, events_host; - if (!has_vhe()) + if (!kvm_arm_support_pmu_v3() || !has_vhe()) return; preempt_disable(); @@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu) struct kvm_host_data *host; u32 events_guest, events_host; - if (!has_vhe()) + if (!kvm_arm_support_pmu_v3() || !has_vhe()) return; host = this_cpu_ptr_hyp_sym(kvm_host_data); From patchwork Wed Mar 10 17:57:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12128965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56801C433E6 for ; Wed, 10 Mar 2021 18:16:32 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DECBE64DC4 for ; Wed, 10 Mar 2021 18:16:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DECBE64DC4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T7Ki5KiUVbbuBPPhBlFnJ6ut0ezRWzyx1gA8v+OIb0w=; b=fGIIMgnq4F4gc3 ZZI82Z/QAfRKYSbjbLm7543GRRW49sdHZ3yz+HF9LR3owwvl7g4OXuCiR3rmKfRPO0zxAJu1M5a9m Iz3HLs6O6RxRECxnU5HRyi6f4knHCAZaTGivM1GI7jwWJfF8z98iEuc6DnFZ8pgJjr28YQ0TJvlg/ k/PoNxy46hL3P2cFn3uuuN6DJgD13s1RrXN2YFdVLlN3SFh/duq8QNSOcbSpOm2Yt8OxZijl0KTP8 NK9V+P30FEp4CcAd/5lmUPxU/OjMWjkmzTVq+0o//CLjhE7waKGvsgKTQj65DkLSgdPHTIf5/fLZw SnJ5YC6SiY+J8qMrWl+Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lK3Lk-007U1z-DV; Wed, 10 Mar 2021 18:14:25 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lK370-007Ocz-Fs for linux-arm-kernel@lists.infradead.org; Wed, 10 Mar 2021 17:59:13 +0000 Received: by mail-wr1-x449.google.com with SMTP id i5so8319016wrp.8 for ; Wed, 10 Mar 2021 09:59:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Izv1B26QXp5fMc/VBnQrm2CNhoL84Ug8LOnqFmCQ1Ig=; b=ItduwNAOHw7sl7SfTdrsmOhuNxkYB+uL69Ld1jgrXtbIP7oT1ltbUj65vkfckRejVo H/GtOSwQfQ4Wr7N4+fDgyoZfBOduWnHmU72oatPYqIPRkc3igSo8/5QhUPxoXKya+XTs 5qhZDbYM7y3pLpa9R6ng1GacjsA/2G8nkofF5b+5rvRhEBHWRYY7NfG6pcuzeWcIYpVA hT08jBVRC3r2CbuMiiiWf8nZJhnCYaTX8l39dXYSBsNtQtK/VDJsb8SRsP+Gt1QwQLfZ HMWYfrRHApwAqtnlfExk0wR/m1YuVB0jdXNr8sy2DyeECS+GZ/c2h+biEiJV3oLKfgkA F+6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Izv1B26QXp5fMc/VBnQrm2CNhoL84Ug8LOnqFmCQ1Ig=; b=YNSErRLAdp48L7+KQ3eY6ZY+NeBXG5bMODyXCnpD14FJ+yY2e2TRabMD7oLJOCFd2C 6VRzkwF3QmEDZYvw1BiaGheVv9RPA1v6iKNvyfgCWz2MDGOvFkQ7kaXy1NTAb/oL80ZO CkftJjlbKFnuXyodZf6DrpnjJWyfqGvvccPvpvhhF6uYbWHgvR0eiU7Do++g1Lekm/gp GyDg+asmTTvCysG57vgzMlVbhwye9IHil3R0sTlAgEOX0Hb7/TAH0mowZpsrB/drgfUH DCo97R9MvBF98nwPP3YOw4PFcq9hTtNQUQYTLxUmZNkYGDRnMgOvQIvX5Wep4L57fNg5 43SA== X-Gm-Message-State: AOAM531jTYedOK5mrA2apliDtPSdFqJR7v5cRxR0jzWhSSGiTVlEOIt0 Eg9tWpRvCojNR/2xG2i9eQBk6IDSRb8u X-Google-Smtp-Source: ABdhPJxrvGTs3vci5lUmhmaPsZPM93YibPtQp837uFnTbTMuR06Qiq5j6ejGsWMlr+vGUqW6o5C4jparLuWA X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:adf:e34f:: with SMTP id n15mr4749877wrj.224.1615399149316; Wed, 10 Mar 2021 09:59:09 -0800 (PST) Date: Wed, 10 Mar 2021 17:57:51 +0000 In-Reply-To: <20210310175751.3320106-1-qperret@google.com> Message-Id: <20210310175751.3320106-35-qperret@google.com> Mime-Version: 1.0 References: <20210310175751.3320106-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v4 34/34] KVM: arm64: Protect the .hyp sections from the host From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org, ardb@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210310_175911_387431_332CB432 X-CRM114-Status: GOOD ( 17.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM runs in nVHE protected mode, use the host stage 2 to unmap the hypervisor sections by marking them as owned by the hypervisor itself. The long-term goal is to ensure the EL2 code can remain robust regardless of the host's state, so this starts by making sure the host cannot e.g. write to the .hyp sections directly. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 46 +++++++++++++++++++ arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 9 ++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 33 +++++++++++++ 5 files changed, 91 insertions(+) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index b127af02bd45..d468c4b37190 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -62,6 +62,7 @@ #define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 #define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 #define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19 +#define __KVM_HOST_SMCCC_FUNC___pkvm_mark_hyp 20 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 7e6a81079652..d6baf76d4747 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1894,11 +1894,57 @@ void _kvm_host_prot_finalize(void *discard) WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)); } +static inline int pkvm_mark_hyp(phys_addr_t start, phys_addr_t end) +{ + return kvm_call_hyp_nvhe(__pkvm_mark_hyp, start, end); +} + +#define pkvm_mark_hyp_section(__section) \ + pkvm_mark_hyp(__pa_symbol(__section##_start), \ + __pa_symbol(__section##_end)) + static int finalize_hyp_mode(void) { + int cpu, ret; + if (!is_protected_kvm_enabled()) return 0; + ret = pkvm_mark_hyp_section(__hyp_idmap_text); + if (ret) + return ret; + + ret = pkvm_mark_hyp_section(__hyp_text); + if (ret) + return ret; + + ret = pkvm_mark_hyp_section(__hyp_rodata); + if (ret) + return ret; + + ret = pkvm_mark_hyp_section(__hyp_bss); + if (ret) + return ret; + + ret = pkvm_mark_hyp(hyp_mem_base, hyp_mem_base + hyp_mem_size); + if (ret) + return ret; + + for_each_possible_cpu(cpu) { + phys_addr_t start = virt_to_phys((void *)kvm_arm_hyp_percpu_base[cpu]); + phys_addr_t end = start + (PAGE_SIZE << nvhe_percpu_order()); + + ret = pkvm_mark_hyp(start, end); + if (ret) + return ret; + + start = virt_to_phys((void *)per_cpu(kvm_arm_hyp_stack_page, cpu)); + end = start + PAGE_SIZE; + ret = pkvm_mark_hyp(start, end); + if (ret) + return ret; + } + /* * Flip the static key upfront as that may no longer be possible * once the host stage 2 is installed. diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h index d293cb328cc4..42d81ec739fa 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -21,6 +21,8 @@ struct host_kvm { extern struct host_kvm host_kvm; int __pkvm_prot_finalize(void); +int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end); + int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool); void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f47028d3fd0a..3df33d4de4a1 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -156,6 +156,14 @@ static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) { cpu_reg(host_ctxt, 1) = __pkvm_prot_finalize(); } + +static void handle___pkvm_mark_hyp(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, start, host_ctxt, 1); + DECLARE_REG(phys_addr_t, end, host_ctxt, 2); + + cpu_reg(host_ctxt, 1) = __pkvm_mark_hyp(start, end); +} typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -180,6 +188,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_create_mappings), HANDLE_FUNC(__pkvm_create_private_mapping), HANDLE_FUNC(__pkvm_prot_finalize), + HANDLE_FUNC(__pkvm_mark_hyp), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 5c88a325e6fc..fffa432ce3eb 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -25,6 +25,8 @@ struct host_kvm host_kvm; struct hyp_pool host_s2_mem; struct hyp_pool host_s2_dev; +static const u32 pkvm_hyp_id = 0xffffffff; + static void *host_s2_zalloc_pages_exact(size_t size) { return hyp_alloc_pages(&host_s2_mem, get_order(size)); @@ -182,6 +184,18 @@ static bool find_mem_range(phys_addr_t addr, struct kvm_mem_range *range) return false; } +static bool range_is_memory(u64 start, u64 end) +{ + struct kvm_mem_range r1, r2; + + if (!find_mem_range(start, &r1) || !find_mem_range(end, &r2)) + return false; + if (r1.start != r2.start) + return false; + + return true; +} + static inline int __host_stage2_idmap(u64 start, u64 end, enum kvm_pgtable_prot prot, struct hyp_pool *pool) @@ -229,6 +243,25 @@ static int host_stage2_idmap(u64 addr) return ret; } +int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end) +{ + int ret; + + /* + * host_stage2_unmap_dev_all() currently relies on MMIO mappings being + * non-persistent, so don't allow changing page ownership in MMIO range. + */ + if (!range_is_memory(start, end)) + return -EINVAL; + + hyp_spin_lock(&host_kvm.lock); + ret = kvm_pgtable_stage2_set_owner(&host_kvm.pgt, start, end - start, + &host_s2_mem, pkvm_hyp_id); + hyp_spin_unlock(&host_kvm.lock); + + return ret != -EAGAIN ? ret : 0; +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault;