From patchwork Tue Mar 2 14:59:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56C54C433E0 for ; Wed, 3 Mar 2021 17:32:52 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 601C064ED7 for ; Wed, 3 Mar 2021 17:32:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 601C064ED7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JPm8pIXnsy5+cVHVepX3e8Hg8gpdmUi4UkSHupwVo+Q=; b=SMAT0RoEhNt18e rXzFu4PXIxySCIl+qTCEgkVSPzEl05SPFpmINmI1+GRFbc4zaqRtePdBPYqOv0N29VxW9ew5H5vzt JROZ7AjFXZb6x3918B/2Xlu0YlTklDaqzsPpnBYC/R0tb+0BjY3KKJxERhFNij96WH2IXiFd9XvNx jFerjasd8s9uaT+VppJFrp41PvEbP6KmgBS8ti0xJvbd4JvMCEybRjBcmJN+x17w2ghqLJXhJw/6n bUGVAnicMtLHOIcdT+hz9BVz0t/sh1x19CDCCrF7LtwV8hAxgS0QnCe22lIbHF3uwQPfa7ep1w/Vf YrGHu5bOAE+Nrou0mFzw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHVKO-005oJN-Vf; Wed, 03 Mar 2021 17:30:29 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSvN-005FWq-9a for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:56:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=AbQOhTo5qTBDcGKT4ROquEtU01mLpiQKpICJ+GOnoG0=; b=Gr/fPGoGuDXLEpiNmqA8hibHqt 7Uo1I+x7fYDB3vG32TrkXvrYbzyc2oGOgLaMwxN6oFVYI0ke5DyFzV8/Iy3Mv0TRgKJQgLBSIBnCo h56SG4CdyC9Xo+6pCZOuahP92Xo5HcHENE6MXSasTpoA7ApzbzjqkIJcxOVgoBAOwz+RaHyfnGnPZ upB6eQBnKT4IBsQLAodKduMtqGFyMpssBlJlSTdkb820kfj5MIv8d1w+L1FUervE2mD+KezdsmxJy NZK6tjmydJkg13SDBDUWEFX6MMXU/ebwXccgVyQaPJ6XsIpmxig4d+knLo06OprxFtKPAJxaqGh4L Z8BdCPvQ==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6WY-00HJ13-GA for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:28 +0000 Received: by mail-wm1-x34a.google.com with SMTP id s192so1310521wme.6 for ; Tue, 02 Mar 2021 07:01:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=AbQOhTo5qTBDcGKT4ROquEtU01mLpiQKpICJ+GOnoG0=; b=MfW/XHtPJX0w2ipZVk2ijDvDAMfL6HCaJrecrAAwfFAlbKa/hgXriwrQC/xOMitILV HdeX+FO8dVOXvJ349UOGKSfzw/G8uO3yAwL6GNM/TEN+7BJOpvN6cPTje25i9rOjEdFo Ze8aV5OUD8zbeIjktjuBv3Fx4bJTcKIUqeaNFPiZzdrljYxVNzrXJrvbLcKWBzbh478I gL8cwdt0E3dCMiyFlf1XGEThwupYU3c9H/mT2wRIck9dvEg3atzKumxpxoJ5g+SNIHGF FMI1FTzWJ4QlMmJ+28ZhCaYFXCUALVIdDv9d3OE6oZERGtFbYN4KyiSg+RbRZnI4i231 Mmfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AbQOhTo5qTBDcGKT4ROquEtU01mLpiQKpICJ+GOnoG0=; b=VOa1wiTdSIq90eqXSV+t+SrasyN3UIGZsi5GXtTo+Bu7PVm8INo4S8GnERpuOut37I +AYU+FlZFQUsXuaOQdpE7bXLsYVSHzR/bHx6XZl+kO6Qpm5GTe5AEg38vnkQ+DuX47hA 54kXKZRsZomC9SdFdf26RRZFvb6edp/M2XHV3R3upBOd/VgvWw8wG5ApR7Zy/UFbk2Iy qjO5TO8LPablfIX8WDgT8SFL5tF03LrpH2kU7mrt2zpmdwvsBG+cQPi6oKLVe3oSiv8y LmM9/FQgPx8otJWjPjJ+egCT6bssLF2YquMeb7VTpq2mDG+6FUUBQVtuTBdKEgAuH45k P3uQ== X-Gm-Message-State: AOAM530IfWzejUs7CYS4CMXTTpOPaC4tc0YwtWYyKP3SWsBmvYngCK+L r5Y1F4jdbLhEsuKBf4et9zcOZPTsKUAM X-Google-Smtp-Source: ABdhPJxKe/abaoaQDAN0ud0u087kNQ1v1cFYLoKd692l087a2pLQ0SHZGffvQJSKkW0kxuZW5igSyzEjrCP4 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:4c11:: with SMTP id d17mr4573540wmp.86.1614697208554; Tue, 02 Mar 2021 07:00:08 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:31 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-2-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 01/32] arm64: lib: Annotate {clear, copy}_page() as position-independent From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150128_554171_D6ABEE26 X-CRM114-Status: UNSURE ( 8.43 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon clear_page() and copy_page() are suitable for use outside of the kernel address space, so annotate them as position-independent code. Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/lib/clear_page.S | 4 ++-- arch/arm64/lib/copy_page.S | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S index 073acbf02a7c..b84b179edba3 100644 --- a/arch/arm64/lib/clear_page.S +++ b/arch/arm64/lib/clear_page.S @@ -14,7 +14,7 @@ * Parameters: * x0 - dest */ -SYM_FUNC_START(clear_page) +SYM_FUNC_START_PI(clear_page) mrs x1, dczid_el0 and w1, w1, #0xf mov x2, #4 @@ -25,5 +25,5 @@ SYM_FUNC_START(clear_page) tst x0, #(PAGE_SIZE - 1) b.ne 1b ret -SYM_FUNC_END(clear_page) +SYM_FUNC_END_PI(clear_page) EXPORT_SYMBOL(clear_page) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index e7a793961408..29144f4cd449 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -17,7 +17,7 @@ * x0 - dest * x1 - src */ -SYM_FUNC_START(copy_page) +SYM_FUNC_START_PI(copy_page) alternative_if ARM64_HAS_NO_HW_PREFETCH // Prefetch three cache lines ahead. prfm pldl1strm, [x1, #128] @@ -75,5 +75,5 @@ alternative_else_nop_endif stnp x16, x17, [x0, #112 - 256] ret -SYM_FUNC_END(copy_page) +SYM_FUNC_END_PI(copy_page) EXPORT_SYMBOL(copy_page) From patchwork Tue Mar 2 14:59:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46DD8C433E0 for ; Wed, 3 Mar 2021 16:19:25 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13F2664ED0 for ; Wed, 3 Mar 2021 16:19:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13F2664ED0 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UkT4Yq8D/buAXtdCDm3r00hg6YFL05+md6fsA3v/fOU=; b=OCDeWIXi/nXUDU /hC7JedNYx1DZunERXUwC3OCh2/JxEdD6fNUiUunlmunTEsVq0Wqq0/bTh/MoDP0+QlAE2M/2J46E yNN0jX48t+XRhLaBEsYxsU5+OyJ2W/yf8q+B9hEWfkFabiUoAx1AFZDxDU1cPUna6TL5H7IcuZPKU UTl61pzKQQ33EtJfBA8Ehn8HJp5tySRpt/c3yGIsIXDi4mCxS4bDVL9dbnbaj7MO0MDcJtkkn1MGd 49whDp3A2jVx4vkAXDS258Z5fI1F5X/foE6GUkKLAwLkXwX0JhjC5Cu2t3qPUAKqlbAZvtMME8EPs AZdlBEep6EZjwSPfW8tQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHUBf-005Z9X-BR; Wed, 03 Mar 2021 16:17:23 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSpS-005DwB-UN for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:50:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=dxCzN3RMauKWh2fvqFWMKg2laCezvVx9+eyFyeOVXIM=; b=J/r+0QoM1mW40flupQz3KL2tSm NHvt024duiU/mUd1nouZM+eW8xyEYN7a3XfKmDD8edyCbQr+ecxUM0lcMI7Pg7n2epoES2VqhS6Jq DMFLcHyDWkCJYvoAbOyRVoKTQDy87IVDuXBNfGTXeXzRkS+FN2F6r/9mAGKQ34DKUXfNFFoEMEkP/ OzyYD8SEAPrUq383By0WynGN856QhixAcV0j3/ly2I0QMv1kmYSFc99VIUXSDzgWUXxDL7APpAYaa ONDSCZnJgoFk/KI0Yqthy5UrLX06sbMM/4Ds7B1otpmKVE/7osFHeQmVZRHSGWNAeoYR0M//S3N7+ LuPb4fnQ==; Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wk-00HJ5S-Ks for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:38 +0000 Received: by mail-qk1-x74a.google.com with SMTP id k68so13305532qke.2 for ; Tue, 02 Mar 2021 07:01:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=dxCzN3RMauKWh2fvqFWMKg2laCezvVx9+eyFyeOVXIM=; b=SYScX5nR7lDOGlIBks1wS5OnBfnruugDhQBoDtlrcbZ3ivNnmk3Bs6FUZozkeKfqUw 8FQnMyan0qCfc0piBSuDGa3ZdS6PWWoNnVtTdxngVrbsCxqT7kUhluPqK1E9J8/SZr+i ymJHr9JIAnzwHKh3RMDwHPcpJ/1u827BNbW4N6hu3aTniww7uqtKpch1dkNZ3MsFC9F9 /G3yHgNC/ffTCpeOUZYbd0UcP/eV2nyrv2S+u2R+ppY+A9DLrs1gH7JOs0evPYZPAljI MLnL8vBSo6ZQhoQFgM9hj1r0lL4DQwJ6HLotKRxxq9PVfZ4bZAkBdIQcoTjnsNDO6jq5 ZNyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dxCzN3RMauKWh2fvqFWMKg2laCezvVx9+eyFyeOVXIM=; b=Zck76HVvpTO5PKQAO9JTcNJW1VL/oVa+t5xWEoAXn9+W8CJZF1llGAfUNAqfhttt3V 1jAJWXclTTM5028lW2Isc4EDOhvQ8fxA/3kZ1Ke4hJmWUiuwjcu3mOyRLd06GBYaYSDo xJAmFXUdtY60U5i2nxiyfuE/76AKKmsavUIJ1MBbpvOz9SC7/kbE30Um+Cr4jG0z+T+n zWtPu2xiQlffYI16eCrYMSh4btNKBn2He7jm25q1v6jG8b4icRc75MRdsZYeR4FMyxoQ c+IOZaj7Y2jp/lTE+DZuOixxkuXSgaRrVNQfSoPgti6mE8Sy7Trg6tPIc1yWchL1VnOZ /hEQ== X-Gm-Message-State: AOAM5325n+H0V++zh691Z0AZPDS7AjnapSrhaqIKaEn1sQvccrToCal8 tb+y3PugVZ1BIeoF8k0r1Xfs08dqGkVi X-Google-Smtp-Source: ABdhPJxJVHSJYp6BIq2xVxKnGl6ariG1yZa/yPQUQ6PMkwXMADmlS67ocdrPIQpjgm15zvCkG4mhGRx9u+Bb X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:c60b:: with SMTP id v11mr19576129qvi.44.1614697210699; Tue, 02 Mar 2021 07:00:10 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:32 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-3-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 02/32] KVM: arm64: Link position-independent string routines into .hyp.text From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150138_319570_BC6DB371 X-CRM114-Status: GOOD ( 10.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon Pull clear_page(), copy_page(), memcpy() and memset() into the nVHE hyp code and ensure that we always execute the '__pi_' entry point on the offchance that it changes in future. [ qperret: Commit title nits and added linker script alias ] Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/hyp_image.h | 3 +++ arch/arm64/kernel/image-vars.h | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 ++++ 3 files changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h index 737ded6b6d0d..78cd77990c9c 100644 --- a/arch/arm64/include/asm/hyp_image.h +++ b/arch/arm64/include/asm/hyp_image.h @@ -56,6 +56,9 @@ */ #define KVM_NVHE_ALIAS(sym) kvm_nvhe_sym(sym) = sym; +/* Defines a linker script alias for KVM nVHE hyp symbols */ +#define KVM_NVHE_ALIAS_HYP(first, sec) kvm_nvhe_sym(first) = kvm_nvhe_sym(sec); + #endif /* LINKER_SCRIPT */ #endif /* __ARM64_HYP_IMAGE_H__ */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 5aa9ed1e9ec6..4eb7a15c8b60 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -104,6 +104,17 @@ KVM_NVHE_ALIAS(kvm_arm_hyp_percpu_base); /* PMU available static key */ KVM_NVHE_ALIAS(kvm_arm_pmu_available); +/* Position-independent library routines */ +KVM_NVHE_ALIAS_HYP(clear_page, __pi_clear_page); +KVM_NVHE_ALIAS_HYP(copy_page, __pi_copy_page); +KVM_NVHE_ALIAS_HYP(memcpy, __pi_memcpy); +KVM_NVHE_ALIAS_HYP(memset, __pi_memset); + +#ifdef CONFIG_KASAN +KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy); +KVM_NVHE_ALIAS_HYP(__memset, __pi_memset); +#endif + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index a6707df4f6c0..bc98f8e3d1da 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -9,10 +9,14 @@ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -D__DISABLE_EXPORTS hostprogs := gen-hyprel HOST_EXTRACFLAGS += -I$(objtree)/include +lib-objs := clear_page.o copy_page.o memcpy.o memset.o +lib-objs := $(addprefix ../../../lib/, $(lib-objs)) + obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o +obj-y += $(lib-objs) ## ## Build rules for compiling nVHE hyp code From patchwork Tue Mar 2 14:59:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60483C433E6 for ; Wed, 3 Mar 2021 16:13:45 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0627E64F07 for ; Wed, 3 Mar 2021 16:13:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0627E64F07 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C93ohdCIj6V6o4O/5y8pR9c4bwR2VoyVKhL4ZwIZqC0=; b=L/YIptt6zGYv/4 fFhx+2g/PJmjjf7yfQp5KoLjqsXkZgwh+GPB/LoUtFZ6OZT6ISfOQUX+55/3nPpF+6G2/63mweU8A fZVrGoDQ8eFG76WFdoqU9TRk1m4gz7xIkgkhh+1nTXCnytY5nAdCtp22aXzTWNm2YGSrikjbInZNf 6eODawLUuD7vMvg7wa8m3FPZkw1ElZCkxJNu7rqKnneXC1EK09Aofb7amv+cDWgw9sw+0NJGfZ4AL eDb6yR39k0N0EzTPeC4JAA5cRDBSYY3UpDjnAk/BacB2kFWLof3KfGomtTfDlUkNuICPkS/R/pBnX yvPTRkVpnOluLG8aioxA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHU68-005Y83-Ot; Wed, 03 Mar 2021 16:11:41 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSp1-005Dmh-62 for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=c79EyduIP7MoVuBONgvo5g7CIT pzZhjzJgCbkFjYl748/7y5RNyhx6HS+/OsPbLoIxOI0D/m3st1H132NfZglbYfJ0M/DuQLrwmk6h5 37VzZs8jRxGUH0hODBpN9rZU6R3efklsNME7Tk+kiJYUFvwlqwor+3tSN/dzhpQlBPj/ost+qdWY/ ig4amP4u6LIt0cENnWv2huQyZ6QBvQbBSVZfG5rxg3IVEyw65/9ILVs4p4CViK+DFHnCxq9jG64mS 9xaBa4k99rZtDcTpF/4xgOiLGdeI9kOYYSr9HdIoeEEU3W/y/BZa0ZI5G3E/OYJJd5ZcaJPqeiLQF VCIdcgjw==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wi-00HJ16-Si for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:36 +0000 Received: by mail-wm1-x349.google.com with SMTP id w10so789449wmk.1 for ; Tue, 02 Mar 2021 07:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=IWWlsrxoA4/VxGyh8bUKvZgzoCrsAILxDoeJNc3FK3cKMETE655OoJhqjhur1cilCo LYMhoMHTidPCmYEdxDntem+etgADvVUzHzOW5PAiTOjO1sxQz4TD7miwsbP4bqNvgZRY GlZRhT8A/bOsfAFAa73IPeR9tNI/bBCdkf7nnRVUDgwoAz6Myk3gOAqETYRQmdfeuwDY 8lBXQN3iPad2h8Ls6GQLyVj6IEr29AuM9kgnxAvmC7NESy44kzshGuXQ3dD7/Q0a9qA3 OFFXEc6wzEJ7OPcIwYWrU4XqEJ8dFiy2sRgH0Sm3Ojhm/myIj/5hcEc+9CMBFz1e24oI 1/Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=rWOIxYxuT5W+kh1aKlDaUIVI9lA5G/rEvBh0LGQjFRqwiGrM8j7nDadN3PkWbNLq+v vfb9rW9XOnhGGXbj6ANDa88X4sBl1YiXm6ONkUsSOcvVTy7ALfp8mFlhDjg8uXvjzHke pL0lCkMirBFwTsaeKmdz9d7Tdv2NVKAWqWdmvtAqy/0nsQaxTj10Cje/4yi5j0X+WdKG nuXyqvxieQs9V7lxFIn079QWw8MZrMH4mhBErM5KTZ1jp/twvYdTrwB/J1rC1o3Sn8sN sUOWDWroEQQ9F8Tz/ow81HQoidmcmnpUPwPqUCX6Djma8ASyJB9g8cyeAr9bkAn4IVTB O/xA== X-Gm-Message-State: AOAM531u3KVH0cU/KxQT6KQWWN/sXaPl7EMEH0EouKLGnK0R72djBAMf z8qfcAHf9KpS3YdL1gupZZLyHBh8Bc/2 X-Google-Smtp-Source: ABdhPJyhRguqkBiAL8rawxftQTL4cQe8tkMBmRvxwBQhjvtLIZuZAUV1pK/Un8+K/eUrAfebq5+z4KnW7CHU X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:5f89:: with SMTP id t131mr4235263wmb.173.1614697212949; Tue, 02 Mar 2021 07:00:12 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:33 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-4-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 03/32] arm64: kvm: Add standalone ticket spinlock implementation for use at hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150136_056085_C5902E5C X-CRM114-Status: GOOD ( 14.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon We will soon need to synchronise multiple CPUs in the hyp text at EL2. The qspinlock-based locking used by the host is overkill for this purpose and relies on the kernel's "percpu" implementation for the MCS nodes. Implement a simple ticket locking scheme based heavily on the code removed by commit c11090474d70 ("arm64: locking: Replace ticket lock implementation with qspinlock"). Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/spinlock.h | 92 ++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/spinlock.h diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h new file mode 100644 index 000000000000..76b537f8d1c6 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * A stand-alone ticket spinlock implementation for use by the non-VHE + * KVM hypervisor code running at EL2. + * + * Copyright (C) 2020 Google LLC + * Author: Will Deacon + * + * Heavily based on the implementation removed by c11090474d70 which was: + * Copyright (C) 2012 ARM Ltd. + */ + +#ifndef __ARM64_KVM_NVHE_SPINLOCK_H__ +#define __ARM64_KVM_NVHE_SPINLOCK_H__ + +#include +#include + +typedef union hyp_spinlock { + u32 __val; + struct { +#ifdef __AARCH64EB__ + u16 next, owner; +#else + u16 owner, next; +#endif + }; +} hyp_spinlock_t; + +#define hyp_spin_lock_init(l) \ +do { \ + *(l) = (hyp_spinlock_t){ .__val = 0 }; \ +} while (0) + +static inline void hyp_spin_lock(hyp_spinlock_t *lock) +{ + u32 tmp; + hyp_spinlock_t lockval, newval; + + asm volatile( + /* Atomically increment the next ticket. */ + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ +" prfm pstl1strm, %3\n" +"1: ldaxr %w0, %3\n" +" add %w1, %w0, #(1 << 16)\n" +" stxr %w2, %w1, %3\n" +" cbnz %w2, 1b\n", + /* LSE atomics */ +" mov %w2, #(1 << 16)\n" +" ldadda %w2, %w0, %3\n" + __nops(3)) + + /* Did we get the lock? */ +" eor %w1, %w0, %w0, ror #16\n" +" cbz %w1, 3f\n" + /* + * No: spin on the owner. Send a local event to avoid missing an + * unlock before the exclusive load. + */ +" sevl\n" +"2: wfe\n" +" ldaxrh %w2, %4\n" +" eor %w1, %w2, %w0, lsr #16\n" +" cbnz %w1, 2b\n" + /* We got the lock. Critical section starts here. */ +"3:" + : "=&r" (lockval), "=&r" (newval), "=&r" (tmp), "+Q" (*lock) + : "Q" (lock->owner) + : "memory"); +} + +static inline void hyp_spin_unlock(hyp_spinlock_t *lock) +{ + u64 tmp; + + asm volatile( + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " ldrh %w1, %0\n" + " add %w1, %w1, #1\n" + " stlrh %w1, %0", + /* LSE atomics */ + " mov %w1, #1\n" + " staddlh %w1, %0\n" + __nops(1)) + : "=Q" (lock->owner), "=&r" (tmp) + : + : "memory"); +} + +#endif /* __ARM64_KVM_NVHE_SPINLOCK_H__ */ From patchwork Tue Mar 2 14:59:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0377EC433DB for ; Wed, 3 Mar 2021 18:23:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C6B164EBD for ; Wed, 3 Mar 2021 18:23:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C6B164EBD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SgS2rvFQkdSdBp7ezEAUChhBl0g0b2XsUlbZXm7aGy8=; b=fM2/URwSoZdIvb sos/FtnDn2WTndDZqAr++Z0AXr8aQGKu3O1Vy/fdMRVfRacj1mYSuiBjgGhZbV/otmGBqv5QkLheV CHZ1oPWphYb9m6I1yhy7mxjqcF2Jmzf6PZYfeh9vRJwuIPMemzTrxyUwppVwgGKtzDBFnSEFY0O3E e2jeCaOZs40mvGdgLadfp1rdpXkCUQhGo0TYgPNHSvMPYLTbxW9epqcPrIt/iw92Yq3aL/UyKWbv6 rrC2LLZaDg+hhmwo00Vg3SwqjwvchyLVy++r0lMSxxdT/8VGFrZUEijDwts4PVcDFgu4+UpHunnrM 6BYy8g/rdfRPHPfJ41Fg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHW7I-005zi0-MX; Wed, 03 Mar 2021 18:21:02 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT0E-005GwD-Ii for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:01:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=CR8slcHJty71kvDCKzriIjKyavoYKi9QlRsnw//QRog=; b=kS/MptQkcOv7gqhmd2ukJ0Z6ZE a6tsNhubkGISmDcpR8OwGWwX7TVLWNRPxJcflpl8SXDzyQVwXf5fBSCDIOqrdI6U5hTo7TYHKYtK/ Jfywi38in6VdRRQS6OW9CKThnGxjYZip7QThQNK40OHpbBPgAhR6JwOOnQuQLoJdBJpm/M/UuEUYd U1+LfHhFrzbNKN16zaVbuSIWicwPbhpVOWfQJwE1SCMa19nmvZLxwXFHFy8nwGMh3EEdVPTDg1qsl hLKb90Tzdkxz0babqy1BNIpsqG3OJLN7KZWH7xAo7r5TV6GNbuLPWKsGmmxplzbpNJrfegTuYZ19w Y0KMuZvQ==; Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wm-00HJ6H-0H for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:40 +0000 Received: by mail-qt1-x849.google.com with SMTP id 4so9874922qtc.13 for ; Tue, 02 Mar 2021 07:01:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=CR8slcHJty71kvDCKzriIjKyavoYKi9QlRsnw//QRog=; b=gpNav6inPzEt9MzUtLCd/NJ9zfZ25Hl8Fy5hqYea9JqzZVivwRYLifUQyzoD2FEiid b2UuiFbsnXWY11BMcHhgQD/oBacoTL5GwDVp1MD/kfNwKNgY0OPtv6voubsIXVzeC+Cd pxHVjVfb/gt1Iz0gyThf70TSrfBa0KpCM7TCt8YoMCZi9AS+YeEr7kzDGNyId75/Xuy9 w2bcoc2cCIaav3tj/U86Nn1iVbzQJDABbjCM/IrZ4bFnt8fv0giaK0P3VD70fYTu+WQV t51XY5o2hrnMbZCiHBAZ5QMj4a60W+acb1k2xSjBvdTOvSbbHdKHP5Gwp0rSJ+CD561U dTqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CR8slcHJty71kvDCKzriIjKyavoYKi9QlRsnw//QRog=; b=ccuqxb2dyVlU9RrzC5Cp65o9lYom84JfMjqB6wy2nYXnQG5hbiZnEL2ORbyYy4z3xq 2s5U7BN4zP/NLPOuOBAakp9p0EowPpKV+nMqXZB9ItDh0h9jGsB4YmNSMubtrzSkR10N qBkHIXvRmN0i6hnxvAG5miqMBAy1wn8MTEOJJ5OXnRWalBpYBo8dKB9IWhExXT7W6ypX ms0xdbQ7NN2m2ZnSwy3CKR4h3vJqmFNWQ0lyPOBg0Ri5enV16p66eU1hPC+wHx0ZtRmH el4EDD+iQycnJ+eApfPCDDX87yfDwRwxVfeSyjXlFDaoQTqqg9WxgGu5l3y2V8Npud+a AKCw== X-Gm-Message-State: AOAM532JALDEA/gnWRrvSDreXLwJvTWhbrQZT2XD7SupOuixa6iRnl96 uBL12tNICWTRUY0TlHwk54Sex0gUf+4L X-Google-Smtp-Source: ABdhPJzFh6M/prxCMqtYunj1NZawACo7xUzCIHsXMY1Qmi0D2izVSSFm5ylNdk9Hc66GGiKSd7N3vkDsOVnd X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:52:: with SMTP id c18mr3713024qvr.54.1614697215183; Tue, 02 Mar 2021 07:00:15 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:34 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-5-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 04/32] KVM: arm64: Initialize kvm_nvhe_init_params early From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150140_624459_76D0C80F X-CRM114-Status: GOOD ( 14.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the initialization of kvm_nvhe_init_params in a dedicated function that is run early, and only once during KVM init, rather than every time the KVM vectors are set and reset. This also opens the opportunity for the hypervisor to change the init structs during boot, hence simplifying the replacement of host-provided page-table by the one the hypervisor will create for itself. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/arm.c | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index fc4c95dd2d26..2d1e7ef69c04 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1383,22 +1383,18 @@ static int kvm_init_vector_slots(void) return 0; } -static void cpu_init_hyp_mode(void) +static void cpu_prepare_hyp_mode(int cpu) { - struct kvm_nvhe_init_params *params = this_cpu_ptr_nvhe_sym(kvm_init_params); - struct arm_smccc_res res; + struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); unsigned long tcr; - /* Switch from the HYP stub to our own HYP init vector */ - __hyp_set_vectors(kvm_get_idmap_vector()); - /* * Calculate the raw per-cpu offset without a translation from the * kernel's mapping to the linear mapping, and store it in tpidr_el2 * so that we can use adr_l to access per-cpu variables in EL2. * Also drop the KASAN tag which gets in the way... */ - params->tpidr_el2 = (unsigned long)kasan_reset_tag(this_cpu_ptr_nvhe_sym(__per_cpu_start)) - + params->tpidr_el2 = (unsigned long)kasan_reset_tag(per_cpu_ptr_nvhe_sym(__per_cpu_start, cpu)) - (unsigned long)kvm_ksym_ref(CHOOSE_NVHE_SYM(__per_cpu_start)); params->mair_el2 = read_sysreg(mair_el1); @@ -1422,7 +1418,7 @@ static void cpu_init_hyp_mode(void) tcr |= (idmap_t0sz & GENMASK(TCR_TxSZ_WIDTH - 1, 0)) << TCR_T0SZ_OFFSET; params->tcr_el2 = tcr; - params->stack_hyp_va = kern_hyp_va(__this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE); + params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); /* @@ -1430,6 +1426,15 @@ static void cpu_init_hyp_mode(void) * be read while the MMU is off. */ kvm_flush_dcache_to_poc(params, sizeof(*params)); +} + +static void cpu_init_hyp_mode(void) +{ + struct kvm_nvhe_init_params *params; + struct arm_smccc_res res; + + /* Switch from the HYP stub to our own HYP init vector */ + __hyp_set_vectors(kvm_get_idmap_vector()); /* * Call initialization code, and switch to the full blown HYP code. @@ -1438,6 +1443,7 @@ static void cpu_init_hyp_mode(void) * cpus_have_const_cap() wrapper. */ BUG_ON(!system_capabilities_finalized()); + params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); @@ -1785,19 +1791,19 @@ static int init_hyp_mode(void) } } - /* - * Map Hyp percpu pages - */ for_each_possible_cpu(cpu) { char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu]; char *percpu_end = percpu_begin + nvhe_percpu_size(); + /* Map Hyp percpu pages */ err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP); - if (err) { kvm_err("Cannot map hyp percpu region\n"); goto out_err; } + + /* Prepare the CPU initialization parameters */ + cpu_prepare_hyp_mode(cpu); } if (is_protected_kvm_enabled()) { From patchwork Tue Mar 2 14:59:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EECCC433E0 for ; Wed, 3 Mar 2021 20:50:26 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CD65F64EF8 for ; Wed, 3 Mar 2021 20:50:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD65F64EF8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7H7sQrTyDotrU+b/UR0LE3wVaeYktumM9TUtr8+b7w4=; b=ErRZfdOQ+Xa60O jwB2N6kWHVlvjgeU8FYocYdH9PihCCe+tPlCS0UdwuMCs9mcR+xNPtT8qpdvUjtZF0m9VmJ0gxQKj 0VmTvP5I2kIQO9V2j4D1Pcsu1M1vMIp9ASeOIHP/BzzFxZNH3+f5ctfA1q3Ao8f6u4/SX3lsbRdgu J/szBsZ5UZseKod13Z8QLlbJVsnmStozWibJ4EqXy4vrJ0V+dkYG9sBoDZbFR5ggGBsH80S55OR5F Ld8aptzUGZC02u2u5CiLNzmBTqqMFPYQHd1prVQHczqnhTIlsryu8p+JzMNKSgk1QyKHdcsDeieFN lBr8NF4w2k8q7uC0415Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYLr-006TGO-Rj; Wed, 03 Mar 2021 20:44:12 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT6n-005JB3-8t for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:08:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=lIPthT51J+nIs48nl/QvpI3Ln3AY2S6u6ppS3XkUMwM=; b=NCBtBRFvZNxwYvs0rrqQVu6QdW ItQaJvpzqoMeG4UwNTHwkVAasKenKVhqhJuFotHrN9HHE50VX8/rq1f6q6k/HeqyXbStQI/ZuzFyz 0nKKn4f0EG5XjrRIeTnExjcw8oDbef0Eh+PvKSlgaR4ntQpjMkMDh3jIX04SpXFxwMef2ziQ0Xuwq 0O40Q5m4GIogovkPGpqx11LPr9eZzAZsr+9/zpicAN8r6m7BKL3Df7YY4+otgcW8ZJIXrn3Je+C3j 3p1uSY5wC67wAkJm0v9qmteIf1SHk+PwLh2iibOA2oZVsKTShBtfpwWBdLOBQNFVKmU8Skhlie2jY GZGqbQRQ==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wc-00HJEA-1h for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:32 +0000 Received: by mail-wr1-x449.google.com with SMTP id h21so6835664wrc.19 for ; Tue, 02 Mar 2021 07:01:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=lIPthT51J+nIs48nl/QvpI3Ln3AY2S6u6ppS3XkUMwM=; b=HY2FQJKTnEHypa62GHDeyX5+DVpyd3kd+DNyVif3kQtZ0Ox8tb7XmQpSllG+iYjI1q 2QUVT2ttxPR69J+1gr+Neb8KNaKUHnwZPw3KUtPEso2PhLlezOzkuodC5vfiw9ttCpDt PRv1H5+P/SxKxew8i6WYse/QbcgSljdRs8FF7QDs49k78eLuvKtMpiJAuzX2X73QLIXU D6R6lQ6XuxLRnxSp+qKmjlwaaJbUgl6MooyomJKPlhfPlxL+ZUe44/BrLtDEXg46c0WL EuYGfH2gL3Od1t6WL3JKq7MFziOBysv7Jm4YOo4Ku6QwkNhrBF/wvIu6ii7T0msYsmIh 2teA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lIPthT51J+nIs48nl/QvpI3Ln3AY2S6u6ppS3XkUMwM=; b=G5iCygypiMJoXnQrCZKE2D+7yxkNZFRsH488brpVPmfBv43unn3ahiwdo9oZydMfzn fJTqVlVx2dv9qe2DtqbX3hmcc+LHjsIXlnN0sA8Ux+abjdrEBJOWMtFHBc5DFYscX0iU A+jG5Th6TCjszTRYqkQEgTL8SZHTdFsBLGd+/TdW39GHkvFglw2jw7ysHvHlRGU9bXFE aaNfE1D8YprfUjeaNv3OKSpgeopPpM8h1e+SPZ1U36amtGPxRNHCpD17bRXzOxgB+Wcc BMSj/cP9XqW9nFHLv3A16J8GlJ4WWxJszzflYiWdOmps6zSRbWlxfIEJu/0yGJW9Ezxp pIQQ== X-Gm-Message-State: AOAM532Y09KIZ+E+o4eaWtfyGuXxgrazA+VEjILQWo4Fd55yF08cXuvj tx1vkEglFKexY/i05puAdEOJtCFIlR4v X-Google-Smtp-Source: ABdhPJwsf7neAAxO1z8gBsowskCoHbhgsUYl3UEiGSThsujkIogwftEA1DRC0UHx1iWXfw2Ebq84LUWTcvS+ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c18e:: with SMTP id y14mr495425wmi.1.1614697217407; Tue, 02 Mar 2021 07:00:17 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:35 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-6-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 05/32] KVM: arm64: Avoid free_page() in page-table allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150132_840390_C780629A X-CRM114-Status: GOOD ( 12.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the KVM page-table allocator uses a mix of put_page() and free_page() calls depending on the context even though page-allocation is always achieved using variants of __get_free_page(). Make the code consistent by using put_page() throughout, and reduce the memory management API surface used by the page-table code. This will ease factoring out page-allocation from pgtable.c, which is a pre-requisite to creating page-tables at EL2. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 4d177ce1d536..81fe032f34d1 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -413,7 +413,7 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); return 0; } @@ -425,7 +425,7 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - free_page((unsigned long)pgt->pgd); + put_page(virt_to_page(pgt->pgd)); pgt->pgd = NULL; } @@ -577,7 +577,7 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, if (!data->anchor) return 0; - free_page((unsigned long)kvm_pte_follow(*ptep)); + put_page(virt_to_page(kvm_pte_follow(*ptep))); put_page(virt_to_page(ptep)); if (data->anchor == ptep) { @@ -700,7 +700,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, } if (childp) - free_page((unsigned long)childp); + put_page(virt_to_page(childp)); return 0; } @@ -897,7 +897,7 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, put_page(virt_to_page(ptep)); if (kvm_pte_table(pte, level)) - free_page((unsigned long)kvm_pte_follow(pte)); + put_page(virt_to_page(kvm_pte_follow(pte))); return 0; } From patchwork Tue Mar 2 14:59:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACD21C433DB for ; Wed, 3 Mar 2021 16:05:19 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3AC9564ED7 for ; Wed, 3 Mar 2021 16:05:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3AC9564ED7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WkZn+4jDLcYrxky93mniGPNgimgXQpLUfVWwT3RtDvQ=; b=Jk+yb1m/eTw3tl kWQrX/32JFf98mRPAPU336VgVnJakjW5GFcMnvtV7aSpBU2Lq3TAvfiyGVBVJq3RH3qfPvy3nacWS 0tBcFR3cGUILX3E2BYQGdc/3TFayEwH4FgHyXGZC7Fqm5uLx0ZoY/LmIm2jBKgOgCG8mxpAPMvehO oELKCZT5cmmPgxsiYngBxW5J4knXvK4Gl2Y7i5kZx9B6j18Y6nXqKx6Dvao+mcz4c81hG4dbvSqHh U7ucuHFQ4fIKy22Xh56vWgcfalitj9C/DrCxdL1ddpeL4IfKsHjzoLNNlAGIVEjilvYDdGwxhHiEX 2IUbkbRrh3vlI7p7/RpQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHTxa-005Vtq-O6; Wed, 03 Mar 2021 16:02:50 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSoD-005DWf-HD for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=3RmLnpy8mDDVYoaJ0E6AZQtKyoBpxUVoRVMExArBfMk=; b=HvwiYyv1SgMWSp7YehhroCuGHa 0cylVGj75yFMljTWHFtVe+X5Y0tmS2+mMK5z7pQCvmr7ztDzWGh/zodbBX4TT9JNi3SYfIX+IHv6Q QooVpp+WzSXgMoYUBxKhsaH9wh4vNZSVFWCu45HlxhZZaQFu8woE0pQGLmXoaR5Dh3wXKjOe+ajQO IvYzlgETQIXyG61ZmHHWPTcg+QadX3wFIkGiLWXPLq1/Pl/eQBM/z644UixOjljmcZ1n75DSIgxPL rpdBoojeDPeGxefxwmS4ba57gEX8O4gTUTgzMWbngjXl5/cUZifm79CoLNBE12rKMkU0jgTl5KgDc PyD/dcLw==; Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wg-00HJEB-Ge for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:34 +0000 Received: by mail-qt1-x849.google.com with SMTP id j25so10144945qtv.10 for ; Tue, 02 Mar 2021 07:01:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=3RmLnpy8mDDVYoaJ0E6AZQtKyoBpxUVoRVMExArBfMk=; b=hCZY1PDWUXlpbOc2OSBcjOyvF/EEHuSw/HsVVIkm1Nl9GUBQJGbd4gMOj9TzZUNMlG 2yMQkowTJjOrgYNwdhnlTF0akyJ6hPZeuu5KWCQaVdLJYKh0QMoodLhTmwhBsiRFbl5B D/XgKtkVF5a/ff9swQnN79WAmbCbtNP4Q0bPOIBQrJLUFOfh/gXTlgJoONJRaTWFE+di 1Et4uT8j4oFUgdtk6xQCQB5abbvfinHcv/dJvt2bbxv8KPWY0ZAww4R7DZzftEZvuKzH PheIsbv3eOCYmEIdiPbxE45nbjaHwpnkLIFMRi0zD9vYH1h0o+ZbJgvdMM8O9oGQ9+UE q4XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=3RmLnpy8mDDVYoaJ0E6AZQtKyoBpxUVoRVMExArBfMk=; b=ZWuihrHZMce6Ju1huvPNrtprw8gr265R4J37Ri65JfOunrud6IIXFPcYgCBEBT9T41 fKygnXUHkY1RGn1/hd4D4fV2xerBGuQ2+NxSRpyFVWxmnwJ3kU5UviTPZxobRsk6ma1k EKwNACWamZIiKD7PSCOCwiqLnnhbNzS2WRVT/qovnU04tKssfulTBk/Bj/rV+wjH0RNE ZerBKVLKl0H3FB8bIPECrcloTSDQOFc7wpt+S5/l/AI+e1/+1Q4Vr4Ri9tfKk6nUclrb IBIlut99SOo/QKGJWKj89No7rt3qTwo4jlIyajMTG8dinl/63Yd0pPY8rv+2BMzHCXka LVag== X-Gm-Message-State: AOAM532dYfhe27KfNa6tXTpUP/Mg1eIGqsU3upyNeJxvBFGb7CGEXHRv yeBFN008gF7GPixuF9/BmN+k9RahzsG2 X-Google-Smtp-Source: ABdhPJx8obl4JR9jS29SfY9cM9WsPeqktW9NZCLyxtk2WCddOylqvlH8OMiKQV3XevJAGSNSfmt5BrKxLP8q X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:aae:: with SMTP id ew14mr3942139qvb.24.1614697219779; Tue, 02 Mar 2021 07:00:19 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:36 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-7-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 06/32] KVM: arm64: Factor memory allocation out of pgtable.c From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150134_787662_52E97077 X-CRM114-Status: GOOD ( 20.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for enabling the creation of page-tables at EL2, factor all memory allocation out of the page-table code, hence making it re-usable with any compatible memory allocator. No functional changes intended. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_pgtable.h | 41 +++++++++++- arch/arm64/kvm/hyp/pgtable.c | 98 +++++++++++++++++----------- arch/arm64/kvm/mmu.c | 66 ++++++++++++++++++- 3 files changed, 163 insertions(+), 42 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 8886d43cfb11..3c306f90f7da 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -13,17 +13,50 @@ typedef u64 kvm_pte_t; +/** + * struct kvm_pgtable_mm_ops - Memory management callbacks. + * @zalloc_page: Allocate a single zeroed memory page. The @arg parameter + * can be used by the walker to pass a memcache. The + * initial refcount of the page is 1. + * @zalloc_pages_exact: Allocate an exact number of zeroed memory pages. The + * @size parameter is in bytes, it is automatically rounded + * to PAGE_SIZE and the resulting allocation is physically + * contiguous. + * @free_pages_exact: Free an exact number of memory pages, to free memory + * allocated with zalloc_pages_exact. + * @get_page: Increment the refcount on a page. + * @put_page: Decrement the refcount on a page. When the refcount + * reaches 0 the page is automatically freed. + * @page_count: Return the refcount of a page. + * @phys_to_virt: Convert a physical address into a virtual address as + * accessible in the current context. + * @virt_to_phys: Convert a virtual address in the current context into a + * physical address. + */ +struct kvm_pgtable_mm_ops { + void* (*zalloc_page)(void *arg); + void* (*zalloc_pages_exact)(size_t size); + void (*free_pages_exact)(void *addr, size_t size); + void (*get_page)(void *addr); + void (*put_page)(void *addr); + int (*page_count)(void *addr); + void* (*phys_to_virt)(phys_addr_t phys); + phys_addr_t (*virt_to_phys)(void *addr); +}; + /** * struct kvm_pgtable - KVM page-table. * @ia_bits: Maximum input address size, in bits. * @start_level: Level at which the page-table walk starts. * @pgd: Pointer to the first top-level entry of the page-table. + * @mm_ops: Memory management callbacks. * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. */ struct kvm_pgtable { u32 ia_bits; u32 start_level; kvm_pte_t *pgd; + struct kvm_pgtable_mm_ops *mm_ops; /* Stage-2 only */ struct kvm_s2_mmu *mmu; @@ -86,10 +119,12 @@ struct kvm_pgtable_walker { * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table. * @pgt: Uninitialised page-table structure to initialise. * @va_bits: Maximum virtual address bits. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits); +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_hyp_destroy() - Destroy an unused hypervisor stage-1 page-table. @@ -126,10 +161,12 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. * @kvm: KVM structure representing the guest virtual machine. + * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm); +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops); /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 81fe032f34d1..b975a67d1f85 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -152,9 +152,9 @@ static kvm_pte_t kvm_phys_to_pte(u64 pa) return pte; } -static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte) +static kvm_pte_t *kvm_pte_follow(kvm_pte_t pte, struct kvm_pgtable_mm_ops *mm_ops) { - return __va(kvm_pte_to_phys(pte)); + return mm_ops->phys_to_virt(kvm_pte_to_phys(pte)); } static void kvm_set_invalid_pte(kvm_pte_t *ptep) @@ -163,9 +163,10 @@ static void kvm_set_invalid_pte(kvm_pte_t *ptep) WRITE_ONCE(*ptep, pte & ~KVM_PTE_VALID); } -static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp) +static void kvm_set_table_pte(kvm_pte_t *ptep, kvm_pte_t *childp, + struct kvm_pgtable_mm_ops *mm_ops) { - kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(__pa(childp)); + kvm_pte_t old = *ptep, pte = kvm_phys_to_pte(mm_ops->virt_to_phys(childp)); pte |= FIELD_PREP(KVM_PTE_TYPE, KVM_PTE_TYPE_TABLE); pte |= KVM_PTE_VALID; @@ -227,7 +228,7 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data, goto out; } - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, data->pgt->mm_ops); ret = __kvm_pgtable_walk(data, childp, level + 1); if (ret) goto out; @@ -302,8 +303,9 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, } struct hyp_map_data { - u64 phys; - kvm_pte_t attr; + u64 phys; + kvm_pte_t attr; + struct kvm_pgtable_mm_ops *mm_ops; }; static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -358,6 +360,8 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { kvm_pte_t *childp; + struct hyp_map_data *data = arg; + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (hyp_map_walker_try_leaf(addr, end, level, ptep, arg)) return 0; @@ -365,11 +369,11 @@ static int hyp_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (WARN_ON(level == KVM_PGTABLE_MAX_LEVELS - 1)) return -EINVAL; - childp = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + childp = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!childp) return -ENOMEM; - kvm_set_table_pte(ptep, childp); + kvm_set_table_pte(ptep, childp, mm_ops); return 0; } @@ -379,6 +383,7 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, int ret; struct hyp_map_data map_data = { .phys = ALIGN_DOWN(phys, PAGE_SIZE), + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = hyp_map_walker, @@ -396,16 +401,18 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, return ret; } -int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) +int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, + struct kvm_pgtable_mm_ops *mm_ops) { u64 levels = ARM64_HW_PGTABLE_LEVELS(va_bits); - pgt->pgd = (kvm_pte_t *)get_zeroed_page(GFP_KERNEL); + pgt->pgd = (kvm_pte_t *)mm_ops->zalloc_page(NULL); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = va_bits; pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels; + pgt->mm_ops = mm_ops; pgt->mmu = NULL; return 0; } @@ -413,7 +420,9 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits) static int hyp_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - put_page(virt_to_page(kvm_pte_follow(*ptep))); + struct kvm_pgtable_mm_ops *mm_ops = arg; + + mm_ops->put_page((void *)kvm_pte_follow(*ptep, mm_ops)); return 0; } @@ -422,10 +431,11 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt) struct kvm_pgtable_walker walker = { .cb = hyp_free_walker, .flags = KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); - put_page(virt_to_page(pgt->pgd)); + pgt->mm_ops->put_page(pgt->pgd); pgt->pgd = NULL; } @@ -437,6 +447,8 @@ struct stage2_map_data { struct kvm_s2_mmu *mmu; struct kvm_mmu_memory_cache *memcache; + + struct kvm_pgtable_mm_ops *mm_ops; }; static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, @@ -470,7 +482,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, { kvm_pte_t new, old = *ptep; u64 granule = kvm_granule_size(level), phys = data->phys; - struct page *page = virt_to_page(ptep); + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; if (!kvm_block_mapping_supported(addr, end, phys, level)) return -E2BIG; @@ -492,11 +504,11 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, */ kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); - put_page(page); + mm_ops->put_page(ptep); } smp_store_release(ptep, new); - get_page(page); + mm_ops->get_page(ptep); data->phys += granule; return 0; } @@ -526,13 +538,13 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { - int ret; + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; kvm_pte_t *childp, pte = *ptep; - struct page *page = virt_to_page(ptep); + int ret; if (data->anchor) { if (kvm_pte_valid(pte)) - put_page(page); + mm_ops->put_page(ptep); return 0; } @@ -547,7 +559,7 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (!data->memcache) return -ENOMEM; - childp = kvm_mmu_memory_cache_alloc(data->memcache); + childp = mm_ops->zalloc_page(data->memcache); if (!childp) return -ENOMEM; @@ -559,11 +571,11 @@ static int stage2_map_walk_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, if (kvm_pte_valid(pte)) { kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, data->mmu, addr, level); - put_page(page); + mm_ops->put_page(ptep); } - kvm_set_table_pte(ptep, childp); - get_page(page); + kvm_set_table_pte(ptep, childp, mm_ops); + mm_ops->get_page(ptep); return 0; } @@ -572,13 +584,14 @@ static int stage2_map_walk_table_post(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct stage2_map_data *data) { + struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; int ret = 0; if (!data->anchor) return 0; - put_page(virt_to_page(kvm_pte_follow(*ptep))); - put_page(virt_to_page(ptep)); + mm_ops->put_page(kvm_pte_follow(*ptep, mm_ops)); + mm_ops->put_page(ptep); if (data->anchor == ptep) { data->anchor = NULL; @@ -633,6 +646,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .phys = ALIGN_DOWN(phys, PAGE_SIZE), .mmu = pgt->mmu, .memcache = mc, + .mm_ops = pgt->mm_ops, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -669,7 +683,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - struct kvm_s2_mmu *mmu = arg; + struct kvm_pgtable *pgt = arg; + struct kvm_s2_mmu *mmu = pgt->mmu; + struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; kvm_pte_t pte = *ptep, *childp = NULL; bool need_flush = false; @@ -677,9 +693,9 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, return 0; if (kvm_pte_table(pte, level)) { - childp = kvm_pte_follow(pte); + childp = kvm_pte_follow(pte, mm_ops); - if (page_count(virt_to_page(childp)) != 1) + if (mm_ops->page_count(childp) != 1) return 0; } else if (stage2_pte_cacheable(pte)) { need_flush = true; @@ -692,15 +708,15 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, */ kvm_set_invalid_pte(ptep); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, addr, level); - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (need_flush) { - stage2_flush_dcache(kvm_pte_follow(pte), + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); } if (childp) - put_page(virt_to_page(childp)); + mm_ops->put_page(childp); return 0; } @@ -709,7 +725,7 @@ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, - .arg = pgt->mmu, + .arg = pgt, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; @@ -841,12 +857,13 @@ static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte) || !stage2_pte_cacheable(pte)) return 0; - stage2_flush_dcache(kvm_pte_follow(pte), kvm_granule_size(level)); + stage2_flush_dcache(kvm_pte_follow(pte, mm_ops), kvm_granule_size(level)); return 0; } @@ -855,6 +872,7 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) struct kvm_pgtable_walker walker = { .cb = stage2_flush_walker, .flags = KVM_PGTABLE_WALK_LEAF, + .arg = pgt->mm_ops, }; if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) @@ -863,7 +881,8 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, + struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; u64 vtcr = kvm->arch.vtcr; @@ -872,12 +891,13 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm) u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; pgd_sz = kvm_pgd_pages(ia_bits, start_level) * PAGE_SIZE; - pgt->pgd = alloc_pages_exact(pgd_sz, GFP_KERNEL_ACCOUNT | __GFP_ZERO); + pgt->pgd = mm_ops->zalloc_pages_exact(pgd_sz); if (!pgt->pgd) return -ENOMEM; pgt->ia_bits = ia_bits; pgt->start_level = start_level; + pgt->mm_ops = mm_ops; pgt->mmu = &kvm->arch.mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ @@ -889,15 +909,16 @@ static int stage2_free_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { + struct kvm_pgtable_mm_ops *mm_ops = arg; kvm_pte_t pte = *ptep; if (!kvm_pte_valid(pte)) return 0; - put_page(virt_to_page(ptep)); + mm_ops->put_page(ptep); if (kvm_pte_table(pte, level)) - put_page(virt_to_page(kvm_pte_follow(pte))); + mm_ops->put_page(kvm_pte_follow(pte, mm_ops)); return 0; } @@ -909,10 +930,11 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) .cb = stage2_free_walker, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, + .arg = pgt->mm_ops, }; WARN_ON(kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker)); pgd_sz = kvm_pgd_pages(pgt->ia_bits, pgt->start_level) * PAGE_SIZE; - free_pages_exact(pgt->pgd, pgd_sz); + pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 77cb2d28f2a4..4d41d7838d53 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -88,6 +88,44 @@ static bool kvm_is_device_pfn(unsigned long pfn) return !pfn_valid(pfn); } +static void *stage2_memcache_zalloc_page(void *arg) +{ + struct kvm_mmu_memory_cache *mc = arg; + + /* Allocated with __GFP_ZERO, so no need to zero */ + return kvm_mmu_memory_cache_alloc(mc); +} + +static void *kvm_host_zalloc_pages_exact(size_t size) +{ + return alloc_pages_exact(size, GFP_KERNEL_ACCOUNT | __GFP_ZERO); +} + +static void kvm_host_get_page(void *addr) +{ + get_page(virt_to_page(addr)); +} + +static void kvm_host_put_page(void *addr) +{ + put_page(virt_to_page(addr)); +} + +static int kvm_host_page_count(void *addr) +{ + return page_count(virt_to_page(addr)); +} + +static phys_addr_t kvm_host_pa(void *addr) +{ + return __pa(addr); +} + +static void *kvm_host_va(phys_addr_t phys) +{ + return __va(phys); +} + /* * Unmapping vs dcache management: * @@ -351,6 +389,17 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, return 0; } +static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { + .zalloc_page = stage2_memcache_zalloc_page, + .zalloc_pages_exact = kvm_host_zalloc_pages_exact, + .free_pages_exact = free_pages_exact, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .page_count = kvm_host_page_count, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + /** * kvm_init_stage2_mmu - Initialise a S2 MMU strucrure * @kvm: The pointer to the KVM structure @@ -374,7 +423,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm); + err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; @@ -1208,6 +1257,19 @@ static int kvm_map_idmap_text(void) return err; } +static void *kvm_hyp_zalloc_page(void *arg) +{ + return (void *)get_zeroed_page(GFP_KERNEL); +} + +static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { + .zalloc_page = kvm_hyp_zalloc_page, + .get_page = kvm_host_get_page, + .put_page = kvm_host_put_page, + .phys_to_virt = kvm_host_va, + .virt_to_phys = kvm_host_pa, +}; + int kvm_mmu_init(void) { int err; @@ -1251,7 +1313,7 @@ int kvm_mmu_init(void) goto out; } - err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits); + err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits, &kvm_hyp_mm_ops); if (err) goto out_free_pgtable; From patchwork Tue Mar 2 14:59:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51B0BC433E0 for ; Wed, 3 Mar 2021 14:09:09 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CB0C064E68 for ; Wed, 3 Mar 2021 14:09:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB0C064E68 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=V0+KKQ6JE7GOgxrQ1WdWhSfkVr0WdGV0ccBHVNaaynA=; b=GsUTCU3TodghJo uWbog+xs5M93NfG8N1QGPtrhBsj4evQcCSqhZXVgwvksM/bqnBWKC+R4JFf0BQ9kltcFQVx2j2gzB 3pBN36X3blSffN997Y3Bw+ls/wmr7uEj+IxOjtpUKD6WNTm8GI1kjkjx50VI3oHRGwCuoPZPA21SY MskfkA1NJzTTcvmi588Qn0LhwHtNoE8c0SBSqf432xmklgyvpxg4LryVLGXQRyv2bL+t2BHdEKjRi vbKKY8RIpu+DizODyaj9U5IDQoP4yKzlGYNZVwBKHYfJCJjLRrD1gcs37pMVgnITByjD9mD0wN8QW pwH0BXd+cy6dzurPTLMQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHS9L-0053ZK-C6; Wed, 03 Mar 2021 14:06:51 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHS4g-00520w-OF for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:02:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=8bbnvPt9CW/Bw+xbxj5kVgoG1b0td8h29usK6cRvX/g=; b=TB3y5MORO077SJ3hkan4g8rcn/ U/h92A2IjQoUYWDwmav3UQfR5xcjF2T5Q7C20Xt3ijY7jlV8ss12vNJfNlHL1WAnG/u7045/vhgeO d1y9MsOErEo5cGpwb6bCl+9v/v9xfHMZFPkQ78nLDh87mscWDDglg8RgXQalikiexlBQP7Bw2shkK 0RM+IoEK+QeZcpT5JaaqbLnKfIJPcR4xtj1BeyWkEk8fKWghKn3iU3B6LB1MT0fsuNlwH75hv1t96 xPJKp1CXIxx2YXkGXTVdgwcgsiNUcDJy3MnB9/QMaK4/G3D/XKaQ1bnZh2/2P3E8SzoO5cRI2Orxi LKUYy+0g==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wo-00HJEn-VD for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:41 +0000 Received: by mail-wr1-x449.google.com with SMTP id e7so4384478wrw.18 for ; Tue, 02 Mar 2021 07:01:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=8bbnvPt9CW/Bw+xbxj5kVgoG1b0td8h29usK6cRvX/g=; b=bGP9S11iy12yWIq+SRzbBLfrqT/KxZQvr6+/zzIfBr0VHxpTLgSqLY3ympRI037cHJ FxOsE9QbT27L1i81ENsqMxD0lPJqOB0UmjGmWO7tge8huWR/oBFRJjyMSiIJc2lR154S 5finKE3AWj9R06EYW87A4qaOVLD5Day1XvI+OagN8jGtEV7m4mJXayfCPPGXOt3aoew/ wsRsVfJg1PP9dlOMKX3SgrvZEZQLqsUFaGe4s0bItLVcgfCwDzCYWj21VoDgYAWcDP6O eqW5Eh2E5q87g4In7/DR+Nre1cgbN3My1wHSZp675LMlravcQ+CGT22oGNOxrV0hCGiR AAEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8bbnvPt9CW/Bw+xbxj5kVgoG1b0td8h29usK6cRvX/g=; b=To7WRXS9f/b5AKXm7VZTZ3WGYmcmkUZnY+lKp3KnpiNytB5fiHCk/+QXvOoZYztT3m vfNQtVwuLRTGcjwaZTwucztx9Yzm0C6fDjx3OjUBjx5FJXJaKMdm2tDk1hyF5ZgXkQjL lZcdqNKNjg4Xnbx9NAqIri/Z7myNw//943StYglVtYjgEua0AqM7bveDLaCseOkiy1IS rrNPPQzmc+232e1NKtoABOS6owsgyxsUfuvurqkkbHXPHFW4E1/IJUOQ5B5aC3U5cpIb fQqGLr0TfZx98yIfhYyjnECUBy2Ky6NMJX/V11xSTaTGlck8kiRboU91XRk+nAd4hq0S bahA== X-Gm-Message-State: AOAM5300ugytMYP8i2dU19HNuqVvPta/PzGCrL4cAlsabvZibeLNhBCY /vTD73eX5gUT2zAmSGO5MsyAKS1aVR/p X-Google-Smtp-Source: ABdhPJyVmg/89pOwnrWFLDSfYM5uH/qd3yEnMJU8ydZC+pYcedTms4XYjP3iwCvWd4XucfLqxTMPiadxX0ey X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c18e:: with SMTP id y14mr495456wmi.1.1614697222067; Tue, 02 Mar 2021 07:00:22 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:37 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-8-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 07/32] KVM: arm64: Introduce a BSS section for use at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150141_893953_F0C1DF58 X-CRM114-Status: GOOD ( 17.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the hyp code cannot make full use of a bss, as the kernel section is mapped read-only. While this mapping could simply be changed to read-write, it would intermingle even more the hyp and kernel state than they currently are. Instead, introduce a __hyp_bss section, that uses reserved pages, and create the appropriate RW hyp mappings during KVM init. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/vmlinux.lds.S | 52 ++++++++++++++++++++----------- arch/arm64/kvm/arm.c | 14 ++++++++- arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 1 + 4 files changed, 49 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 2f36b16a5b5d..e4ad9db53af1 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -13,6 +13,7 @@ extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[]; extern char __hyp_text_start[], __hyp_text_end[]; extern char __hyp_rodata_start[], __hyp_rodata_end[]; extern char __hyp_reloc_begin[], __hyp_reloc_end[]; +extern char __hyp_bss_start[], __hyp_bss_end[]; extern char __idmap_text_start[], __idmap_text_end[]; extern char __initdata_begin[], __initdata_end[]; extern char __inittext_begin[], __inittext_end[]; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7eea7888bb02..e96173ce211b 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -5,24 +5,7 @@ * Written by Martin Mares */ -#define RO_EXCEPTION_TABLE_ALIGN 8 -#define RUNTIME_DISCARD_EXIT - -#include -#include #include -#include -#include -#include - -#include "image.h" - -OUTPUT_ARCH(aarch64) -ENTRY(_text) - -jiffies = jiffies_64; - - #ifdef CONFIG_KVM #define HYPERVISOR_EXTABLE \ . = ALIGN(SZ_8); \ @@ -51,13 +34,43 @@ jiffies = jiffies_64; __hyp_reloc_end = .; \ } +#define BSS_FIRST_SECTIONS \ + __hyp_bss_start = .; \ + *(HYP_SECTION_NAME(.bss)) \ + . = ALIGN(PAGE_SIZE); \ + __hyp_bss_end = .; + +/* + * We require that __hyp_bss_start and __bss_start are aligned, and enforce it + * with an assertion. But the BSS_SECTION macro places an empty .sbss section + * between them, which can in some cases cause the linker to misalign them. To + * work around the issue, force a page alignment for __bss_start. + */ +#define SBSS_ALIGN PAGE_SIZE #else /* CONFIG_KVM */ #define HYPERVISOR_EXTABLE #define HYPERVISOR_DATA_SECTIONS #define HYPERVISOR_PERCPU_SECTION #define HYPERVISOR_RELOC_SECTION +#define SBSS_ALIGN 0 #endif +#define RO_EXCEPTION_TABLE_ALIGN 8 +#define RUNTIME_DISCARD_EXIT + +#include +#include +#include +#include +#include + +#include "image.h" + +OUTPUT_ARCH(aarch64) +ENTRY(_text) + +jiffies = jiffies_64; + #define HYPERVISOR_TEXT \ /* \ * Align to 4 KB so that \ @@ -276,7 +289,7 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; - BSS_SECTION(0, 0, 0) + BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; @@ -324,6 +337,9 @@ ASSERT(__hibernate_exit_text_end - (__hibernate_exit_text_start & ~(SZ_4K - 1)) ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE, "Entry trampoline text too big") #endif +#ifdef CONFIG_KVM +ASSERT(__hyp_bss_start == __bss_start, "HYP and Host BSS are misaligned") +#endif /* * If padding is applied before .head.text, virt<->phys conversions will fail. */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 2d1e7ef69c04..3f8bcf8db036 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1770,7 +1770,19 @@ static int init_hyp_mode(void) goto out_err; } - err = create_hyp_mappings(kvm_ksym_ref(__bss_start), + /* + * .hyp.bss is guaranteed to be placed at the beginning of the .bss + * section thanks to an assertion in the linker script. Map it RW and + * the rest of .bss RO. + */ + err = create_hyp_mappings(kvm_ksym_ref(__hyp_bss_start), + kvm_ksym_ref(__hyp_bss_end), PAGE_HYP); + if (err) { + kvm_err("Cannot map hyp bss section: %d\n", err); + goto out_err; + } + + err = create_hyp_mappings(kvm_ksym_ref(__hyp_bss_end), kvm_ksym_ref(__bss_stop), PAGE_HYP_RO); if (err) { kvm_err("Cannot map bss section\n"); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S index cd119d82d8e3..f4562f417d3f 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S @@ -25,4 +25,5 @@ SECTIONS { BEGIN_HYP_SECTION(.data..percpu) PERCPU_INPUT(L1_CACHE_BYTES) END_HYP_SECTION + HYP_SECTION(.bss) } From patchwork Tue Mar 2 14:59:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0CCEC433DB for ; Wed, 3 Mar 2021 16:02:12 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2797C64EDC for ; Wed, 3 Mar 2021 16:02:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2797C64EDC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U45u+rdb6t/3/geF3CD8HdXMiWK41NPPcmk7186mgZ4=; b=QazZQ0FajG5RyV dPsVhfGy69dLkBjSGvcISvqGHzR7as2ffcrKCVzMkS8rc2NDKmkR5ZnRorYgMV5dHwzV1yO8YVBWH YdYeuZyWhOMfFM9BPs4hEr5IuYLGrn97xE8hLC7TBteegzj2PMPX0lf642jixW9ZvwqBBvyLCwtkE pDy4IGwNjU5VNzAlQZeZ47LXrq6O5TCwWd6PeNvVGdEKjsWOp8qmP1J89gw2tkx/2jPENpr497HRA mZBwfjyuVMI4VDA6YDm5hlRG583cj0ASgS7w8I9E9g29nRITMWyGzZuEEszXZQO7mvOslXzPE4uF3 4khosvE6+3JxyE8mSmUQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHTuQ-005V6n-4G; Wed, 03 Mar 2021 15:59:35 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSoC-005DWh-0W for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=FQ7gk+4i0GZUVGgKW9zQ6uuaT/h986N5brkzpgihVQ4=; b=V2U9T2LHtk1GISg8pzy5ECbceN DfTQ+hagTLGewYPaS13SGmbfJrL45rEdwMf2ZWaGSKG3dfuqULkHo40ST26dvnJnMRA6oyOWFfZX6 K59MgMC3l2dQltn4piy+DaOoS4KgD5wuybDNHphyaTtNezNv9Gg6BWhhcENSnJp731lZzgmQ2p01L JB2+LVpa2691CXIM2kUUnxa9sDLOp9KlUqnvIKiqrQ+tskQ0ClEC22azuhOF3Pdnc4+dp6UXN+sjr dWv7R6FHvBlZcw84nvpk9LoyfvXzCks1YOvYcpx0hYeoSBFPeSBb1zu9gq4x/46ZtzyMWIVg0cCyh HFbP2avw==; Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wv-00HJEt-PT for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:54 +0000 Received: by mail-qk1-x749.google.com with SMTP id d66so17071539qke.9 for ; Tue, 02 Mar 2021 07:01:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=FQ7gk+4i0GZUVGgKW9zQ6uuaT/h986N5brkzpgihVQ4=; b=X/OUs9SG1r0jdb49+8nVXln6PO8Zu5SI0ZdYl8tCmxLjvl2MV2hgALRZh3zow++nD5 LfYOTe7jubKzAq/fe+4mCbvp1kF7X4I3lzigk2zpB6/taDq/VfhukisnZJlcHe6Cw4or Z2DAVqnLaMStLxV/sg78xIsCcf06CcFYlUZgrbGmAtabdpSBs7a/fthy26V76rPX+XbO vfOPmxuoz0iynn8DGaZuNQ8aFdVjHcjEpkeI56RfdvgEMwpiAyJrKCIXlq6vGBTzLF1f QEby0P939iHyfOgGvqBORuYNyZn81dn4C75IXu/p+aWEWyofsIx6RlVZOzkWwW6/g//M vpSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=FQ7gk+4i0GZUVGgKW9zQ6uuaT/h986N5brkzpgihVQ4=; b=KcgqO9nPYiHCWOR+W3n+VJNpZExgS+7LHgi3ILuWbKUZln+cDumvDvk6uflSeUsCPu MW46+RFN0jGidgyn66ylu/prQh+m+UvJZZ8tlrel7qlM5YanwVNvsbVX/ExwOUtLoXrn oWeEE+ZFwQXetWto0Y+pSeWfsZYthed4EMWomClEB1ozbjZs0j2mm5o0ODSMwwV6rpvD OFLwfDOwkRd7Ms7VSveRf+sY6t4bNhmByFprqIg02OHL6g0WVZT3EqNbDOBv4w233HDv Tm7O0N2c0bl4Zq/r0UYJdHz57FBv+SAZrPD53UMhU8w5/kZHip5LlRsVmcpJrAWI8PLe QWew== X-Gm-Message-State: AOAM532oBkgk4NxvuGU1D6orV7fT9x8chp3BObZU+iGXhwnLjjBOoPVv 8bgKWvs+02OUf90fSrYfQolOznD+CxX6 X-Google-Smtp-Source: ABdhPJzhqT1MCWJ5omfxYXzYFsXFvD1UddN55tmUgOSTPj1CkCQRYxZaBVCibdke5an4rJD9Wh3ODO7o5zyX X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6214:76f:: with SMTP id f15mr3892367qvz.56.1614697224470; Tue, 02 Mar 2021 07:00:24 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:38 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-9-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 08/32] KVM: arm64: Make kvm_call_hyp() a function call at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150154_197932_16DE5810 X-CRM114-Status: UNSURE ( 9.88 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org kvm_call_hyp() has some logic to issue a function call or a hypercall depending on the EL at which the kernel is running. However, all the code compiled under __KVM_NVHE_HYPERVISOR__ is guaranteed to only run at EL2 which allows us to simplify. Add ifdefery to kvm_host.h to simplify kvm_call_hyp() in .hyp.text. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3d10e6527f7d..06ca4828005f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -591,6 +591,7 @@ int kvm_test_age_hva(struct kvm *kvm, unsigned long hva); void kvm_arm_halt_guest(struct kvm *kvm); void kvm_arm_resume_guest(struct kvm *kvm); +#ifndef __KVM_NVHE_HYPERVISOR__ #define kvm_call_hyp_nvhe(f, ...) \ ({ \ struct arm_smccc_res res; \ @@ -630,6 +631,11 @@ void kvm_arm_resume_guest(struct kvm *kvm); \ ret; \ }) +#else /* __KVM_NVHE_HYPERVISOR__ */ +#define kvm_call_hyp(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_ret(f, ...) f(__VA_ARGS__) +#define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) +#endif /* __KVM_NVHE_HYPERVISOR__ */ void force_vm_exit(const cpumask_t *mask); void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); From patchwork Tue Mar 2 14:59:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A3CFC433E0 for ; Wed, 3 Mar 2021 14:06:37 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6224C64E68 for ; Wed, 3 Mar 2021 14:06:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6224C64E68 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3jVJdLiLaLp4Jp7M2IClawu5QXL14UNUlP8IUw0dbLE=; b=etQne4KnnL5sZP pNM23RoVjAe3Wf6cQYq/zaLgwBty5248Ji9H5T6IMr+LhgwqzRSwZzYpsUjMzPlJ52HmJInAGyGNr Id853wY3H9ikNeh8gk/U5jYj1DNmKZ8yCTXiv8KiMBEtjVXpE354zWX91qDsPbjxOQC14KtE4LpTc cLa1P/sBbfn8qPhw8F/nlNocGM1DuSiBKlAe+pPC4LO10aVMObC3DSfugvVJCu/qcSOPt6P/s3ZAs n7R+12pmP+4g222UAly2Nf+LvJB5164L+q5iAe7GTmYdxJR+DFWkpeO94WHGEsgevztJIK9LGz4s9 shx4h1jewr3dwRGv2zFA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHS67-0052Pw-Oj; Wed, 03 Mar 2021 14:03:31 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHS3U-0051pr-3M for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:00:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=T621CrLzDVvHMT7g1JI2svAdR9qKxB9zRXJEq9CxOEU=; b=QVS1MBbUtoxFkzjjiSxGob5KoS xvucIrVQZKDhmSif/CFIgSIiNsci2e7rPzdrbKyxegjVqYWyhwuuadQYLpWGqRp7umjxWyC3xn40m G9Ad/ahV/ThXrsU9GK9v2VIgUAP/Q6YtRQGTZAjREgsSMF6eyvEw6bjkG+Ag18EkqlnaCj2ff50Dw 6pseH2geUb8TGnX44bClW1rcAQSfcNayz1RDDal3H7DE6EVGop8nYE4P9E3vcd2YX3tBYNzWyQJ9p +uuARxWsqKEtQ54GuePdtfcoHOxDT/H/mV+iK/uOTyAfke9BBkPgqqLvrTP8uVtNYpZ/kR0DOpIR/ T0+rTBlA==; Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wt-00HJF9-Tp for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:50 +0000 Received: by mail-qt1-x84a.google.com with SMTP id 4so9875353qtc.13 for ; Tue, 02 Mar 2021 07:01:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=T621CrLzDVvHMT7g1JI2svAdR9qKxB9zRXJEq9CxOEU=; b=JhVB1bpGiYn1jRzMxn5C86Cv84iFwqhC6iR5apn3eWaYAhiRMUSeUiA/loGNHC6UEZ WXJ7G+oIo0kRVVksB+BCF8QFW7BhegDukCh8s7n3swxcqA2kN9JLclL0IFaqfGiqGYZw wMwnC0Nm6lKe/+9njbqaVCvnF2gW6ls0LuTrHRUJwjGtQbiS/ziLG8McYhBEndabj5u4 lXqBAEzmTjgHHDB+UnlVBrk5jiudORfTBxu2CTEnE8ITDIwRVAd2UVMEl4YPH6u1i4cw 18SIyImxlr2wYX1yNBmi2YQPDuGfAGwYpUaQcVzbN2FuvEq9+DGEN/acYO9wm5gY3URn WgMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=T621CrLzDVvHMT7g1JI2svAdR9qKxB9zRXJEq9CxOEU=; b=Cvo8XSsacwK6M6kRuRUANGzKTNrnki+0XXGUOY8nMFi+dV46/v7kaZ+cIqcPFMr96i Zec7FKLvkAV2XsKTjs7nAxFKiXXPiuT5cC9lbWF12/LroC+tgQIxvuoU/jict5F+9UeI fgBeGUk++No0ohylfOnCisFEkXtg+0E9klSiyWADc8cl6MOL9xsNH2dWrJIr5Wjgixqa coinElLpudcGagzK9k9CN69U8Q3dWd6aacoIqyPq31Va0Wd2C/IMgZgoBpjCxU0CRDTY uFHB72TmbfhcXkjE6UbAVZBMOX5cphGbTL+DzS69gHkMHOL6OE2yvRk9hILSrJ8Bt7/D 7LNw== X-Gm-Message-State: AOAM533AtuSJcvsrGfo7JnArW7xU+ax22pvqEPgSxfcbgg4sVrXdiMWe xO4nHyjsNWRQ0Hd08AU27rt8j5xOnIG6 X-Google-Smtp-Source: ABdhPJz1Cnzac8s5SYaMmUMvh8VTsVMc6vRSeZjJ+2CAkCroHCAupXgtoMWTtWwTuJftkM7mvaKD3quAWmDb X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:b7a7:: with SMTP id l39mr3898578qve.56.1614697226584; Tue, 02 Mar 2021 07:00:26 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:39 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-10-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 09/32] KVM: arm64: Allow using kvm_nvhe_sym() in hyp code From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150150_464908_D112D248 X-CRM114-Status: UNSURE ( 9.87 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow the usage of code shared by the host and the hyp in static inline library functions, allow the usage of kvm_nvhe_sym() at EL2 by defaulting to the raw symbol name. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/hyp_image.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h index 78cd77990c9c..b4b3076a76fb 100644 --- a/arch/arm64/include/asm/hyp_image.h +++ b/arch/arm64/include/asm/hyp_image.h @@ -10,11 +10,15 @@ #define __HYP_CONCAT(a, b) a ## b #define HYP_CONCAT(a, b) __HYP_CONCAT(a, b) +#ifndef __KVM_NVHE_HYPERVISOR__ /* * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, * to separate it from the kernel proper. */ #define kvm_nvhe_sym(sym) __kvm_nvhe_##sym +#else +#define kvm_nvhe_sym(sym) sym +#endif #ifdef LINKER_SCRIPT From patchwork Tue Mar 2 14:59:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DDCAC433DB for ; Wed, 3 Mar 2021 20:52:08 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C9AE764EF4 for ; Wed, 3 Mar 2021 20:52:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C9AE764EF4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GoKD5SnVJe3H+vtT4wxYTIN3RNPcfFq55N9iqRxb5f4=; b=CtzjhmIqkXdoY7 8r2JRQQsLkAkY1+0jeAB7DaXDP9AZbSE8LhP4J50flYP9fkVNwa3wPqGgHWUjorAFF7dui1q1TfOH 5anWZrC/QVqPi1VfwxQ9J1cOGCd1geZGMqVVhC81ixyBIKzYw4FhS/VQlyn1BrZjprnJITD6EG3wK NP6JOAASjkajfOROrOyqB6+4ApofSXVAKgszImAqjhWLxS0y2YVHPnwMR9fDg4hOYIbJNXUrtk4vW hOfY/fhU4rdUKPXJqU33OWSOTGRkRsWRe0H+m3sTSNroSM/N6Uh3vgr5nj9oFeSi7QZGKwZb+ZwdX AYAyJnXCS1F/E5vyU9mg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYMi-006TSw-RT; Wed, 03 Mar 2021 20:45:34 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT6n-005JAo-1p for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:08:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=02sj0LTqbtTGnTkyaLnw9s9OCSKkaVL1IC7xFtrVu+I=; b=I5TUDhV2BPlqlkAgKcNnJg3t2X O+TlwczIH9UjA2AA5NFuAY/a1SFoIIXbU8TrrHVePeShqIftTt4SwC3iT+gRbgiL85Y1sblp8x489 s5nJtZArJ/ad81+Y3nm+DvpNI5wAd/vDpHW5oXuc2z5llQLdR3h3+jzSEXKNVIH+vJphvjU5Z0fuk G6xcC5uGD9rTPURcoHRGYXJbdAnmYROEH6KYIkMDDPrutEIjJYZoI25O0gV/5+wixRRb74UCJsaUD IZjqhtkEtYm1IgSoVokPyPQ2J6LXJTZT5GhWAJ6fyDAHwZUaDJ8uDxo48BvFaI1KnJJsOG5+KB4rX SNelOSgA==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6X0-00HJFB-Ck for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:58 +0000 Received: by mail-wm1-x349.google.com with SMTP id h16so1302926wmq.8 for ; Tue, 02 Mar 2021 07:01:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=02sj0LTqbtTGnTkyaLnw9s9OCSKkaVL1IC7xFtrVu+I=; b=kMg+U+Xw7UPztIEgGwxp8ej4zksed9nn61/tmYFKd9Kot6UNliiXUFLWe7Hl6eFhKa gFiPBs/PkKuSn8NSR5OXGwDNUSxGD3ygLSPmXaDVSgAXgz9/XwpNPzic2kw3nzgbmt1M iXSbdI+NYj0e5B1CdYiAlvTT26Hhhe1ycMpRXg5DuuRiX8unpL3UF1NowNzIPRrdITEY qIGnWuGclkBbcy/wAbnZpsDgxPmmlHQKQxsm3jgdexP8fS2uWWlX5nyrsIGfmdDen1k+ AQJiTX/I5tiMF2f/6re4RQgccxrMvyigksNPimTItjsJgaMCI8iBrzykdv9dnT+5gCa6 L2Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=02sj0LTqbtTGnTkyaLnw9s9OCSKkaVL1IC7xFtrVu+I=; b=SMy+nglNgzTWIWp7aiaeIXDbF3XtP7RjfRclCQDd5EGjUGe4k7Lw6StKVn8Gzady9P hplpAN8aJtKA1WTx50Mhu1GKCNvcI32GfkCxqtn5Bv8tDvBDADYFr7Ue1vci/BUYilbY Ohz6TbxwyLPTjrQ89h22iC7jE3btVx/U8fqZTannTh4q9VpF/YAiyb24tlexlvCHqcnm SWhSsCnaRbhsC8zDrAy6LDIWFSsAAQ8MfzQxEIwrri+ipgGqb3nvnYV/RVwjzvMq3AhS 0q1PbWskgmEPz87XpX7lXIfquTSnScH9dug1mL5sSvYeSxoCKnMipt3yF+HovEFCEIRV Ctaw== X-Gm-Message-State: AOAM531nY7YLPEvHiBjxRQKAgkYcewm4kfIMyM7BDtdDBJ02+8hwrOfZ dJULw33NqHelVGXGOp7lww4s9XhXO2wh X-Google-Smtp-Source: ABdhPJyUW2GTOBiNzcvUxp3QcWXsso1W4SzADKlj626x0ZTBaB7QQ69cNw4QBANWnStsF+W5woTPkEz9iXQP X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:2e90:: with SMTP id u138mr496449wmu.0.1614697228949; Tue, 02 Mar 2021 07:00:28 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:40 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-11-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 10/32] KVM: arm64: Introduce an early Hyp page allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150158_086004_8367D8EC X-CRM114-Status: GOOD ( 19.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org With nVHE, the host currently creates all stage 1 hypervisor mappings at EL1 during boot, installs them at EL2, and extends them as required (e.g. when creating a new VM). But in a world where the host is no longer trusted, it cannot have full control over the code mapped in the hypervisor. In preparation for enabling the hypervisor to create its own stage 1 mappings during boot, introduce an early page allocator, with minimal functionality. This allocator is designed to be used only during early bootstrap of the hyp code when memory protection is enabled, which will then switch to using a full-fledged page allocator after init. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/include/nvhe/early_alloc.h | 14 +++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 24 +++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/early_alloc.c | 54 +++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/psci-relay.c | 4 +- 5 files changed, 94 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/early_alloc.h create mode 100644 arch/arm64/kvm/hyp/include/nvhe/memory.h create mode 100644 arch/arm64/kvm/hyp/nvhe/early_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h new file mode 100644 index 000000000000..dc61aaa56f31 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/early_alloc.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_EARLY_ALLOC_H +#define __KVM_HYP_EARLY_ALLOC_H + +#include + +void hyp_early_alloc_init(void *virt, unsigned long size); +unsigned long hyp_early_alloc_nr_used_pages(void); +void *hyp_early_alloc_page(void *arg); +void *hyp_early_alloc_contig(unsigned int nr_pages); + +extern struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; + +#endif /* __KVM_HYP_EARLY_ALLOC_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h new file mode 100644 index 000000000000..3e49eaa7e682 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MEMORY_H +#define __KVM_HYP_MEMORY_H + +#include + +#include + +extern s64 hyp_physvirt_offset; + +#define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) +#define __hyp_va(phys) ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset)) + +static inline void *hyp_phys_to_virt(phys_addr_t phys) +{ + return __hyp_va(phys); +} + +static inline phys_addr_t hyp_virt_to_phys(void *addr) +{ + return __hyp_pa(addr); +} + +#endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index bc98f8e3d1da..24ff99e2eac5 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/early_alloc.c b/arch/arm64/kvm/hyp/nvhe/early_alloc.c new file mode 100644 index 000000000000..1306c430ab87 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/early_alloc.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include + +#include +#include + +struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops; +s64 __ro_after_init hyp_physvirt_offset; + +static unsigned long base; +static unsigned long end; +static unsigned long cur; + +unsigned long hyp_early_alloc_nr_used_pages(void) +{ + return (cur - base) >> PAGE_SHIFT; +} + +void *hyp_early_alloc_contig(unsigned int nr_pages) +{ + unsigned long size = (nr_pages << PAGE_SHIFT); + void *ret = (void *)cur; + + if (!nr_pages) + return NULL; + + if (end - cur < size) + return NULL; + + cur += size; + memset(ret, 0, size); + + return ret; +} + +void *hyp_early_alloc_page(void *arg) +{ + return hyp_early_alloc_contig(1); +} + +void hyp_early_alloc_init(void *virt, unsigned long size) +{ + base = cur = (unsigned long)virt; + end = base + size; + + hyp_early_alloc_mm_ops.zalloc_page = hyp_early_alloc_page; + hyp_early_alloc_mm_ops.phys_to_virt = hyp_phys_to_virt; + hyp_early_alloc_mm_ops.virt_to_phys = hyp_virt_to_phys; +} diff --git a/arch/arm64/kvm/hyp/nvhe/psci-relay.c b/arch/arm64/kvm/hyp/nvhe/psci-relay.c index 63de71c0481e..08508783ec3d 100644 --- a/arch/arm64/kvm/hyp/nvhe/psci-relay.c +++ b/arch/arm64/kvm/hyp/nvhe/psci-relay.c @@ -11,6 +11,7 @@ #include #include +#include #include void kvm_hyp_cpu_entry(unsigned long r0); @@ -20,9 +21,6 @@ void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); /* Config options set by the host. */ struct kvm_host_psci_config __ro_after_init kvm_host_psci_config; -s64 __ro_after_init hyp_physvirt_offset; - -#define __hyp_pa(x) ((phys_addr_t)((x)) + hyp_physvirt_offset) #define INVALID_CPU_ID UINT_MAX From patchwork Tue Mar 2 14:59:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93157C433E6 for ; Wed, 3 Mar 2021 21:15:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A966B64EF6 for ; Wed, 3 Mar 2021 21:15:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A966B64EF6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X+N3L6GMBiibzVm/VBzh+BCWH4pG8LggoO2vnZl52Go=; b=Fv6r9uzqMYDv/H myK6nXRoh9gKe2HQuLqLSLVBlvKVgQSsp3ISywCa50HjxdHm9HA+bCvoGRUTd0MIKD0MlEi30ZxgY 3ok9UA/K7YciQVkh+YNLLTLPLY3QCV3PU8c+LEbn/3mdKT6Dxe1fLSfRtrD+vlt0M+SCwHrj18rty KBA9h/UwpoA50h+hPIf4Oy6/CtzJw2IA+K60ud2oHjafJkh3FH/ApezgTCssKwaJQpkxfFowYSzhy JTIeiViSJ1iRf1SsrMoHd8SH0mgnfZdNzS+oIW3z653gNbWoSGOc28iMiiDvr/RmbzwHm4POPuBpi 3fsfunLakbwXTDn8jLxg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYkm-006Y8n-Uk; Wed, 03 Mar 2021 21:09:57 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT7q-005JZH-2k for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:09:22 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=twje/T88dKdujyu/Ipi7uGGbAfwKvVyGbqSSEkcY7iA=; b=INgxrla6fwK6JA6elp/P7RA2Te 5LBnQznCDUDExgW0voGtTROHWhAvLSasFokUHwUC/DNV8Y5smlirlGmaO06qFT7ZFm5BEAJ9k135e svXxpE8hJl04JNEpCUK0+E//Zdp3hNYsahqIbUN7OWjkhWz3+q5LvqWes2oBqTyFUFD1e47CzVhM6 HWuy3mhTbIJJUv5rLzveE0rvHpDjICopnqNI+x0ihCX8rFxzI1j99npN0fzZRHQl5Ey3jFvFsdnhA GdIWVBkOfGA86JQ5HY2gQwmE5ES3T7gBkDrwynpPm4ymjEFQ+JzId2AhO7cDLuM3UoAceBelM37C5 4M2GsKWw==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6X4-00HJFA-Np for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:01 +0000 Received: by mail-wm1-x34a.google.com with SMTP id v5so1305240wml.9 for ; Tue, 02 Mar 2021 07:01:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=twje/T88dKdujyu/Ipi7uGGbAfwKvVyGbqSSEkcY7iA=; b=h3Auuv0gx9w0uL2JejVSJYcj+FN0Xhi6lI9t1VRPl88R9mO9oGEu3dwGf1lxzcNigU pb7GHhkBl5+LSyCDcns3NnUnbBEzGjihkEgB3NuB1WUfpkovjjgoD1wLlCyaw7a2FAzM 29nnxhE3yIYM9mfy3/brP5+xqxG78O0uvZzH9spB/5Xaqhy0K9l6oIN3dAkt8HS9unYe cJSuE1ohG6VZw+l7HWUPmoEy3/b1EPuCPG4Lon8j/wC/fVSMwrEdkqkEVg5Mcc/wWAU0 Q9ZGG7OBUqJ+sWmtLWqyWoMOkrOIwWF+xvd311ON5vNOd3iU6HnxXKcyf3xcKnxcGhGr s9ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=twje/T88dKdujyu/Ipi7uGGbAfwKvVyGbqSSEkcY7iA=; b=YmKzZpKYNIA1JoPebJ8Xgoq3mVOxdQ4KPkQ4sCVjskZS/CWuYpJHTx6IWBGMhSyX8Q UTia926+gVhyhLPz8iqnAdhjBQznbceCxU8GSu/f54pOR+FHBDVS+SA+SCsLbiMG//xp EQHb6ZiIq+9G3Q1IdMpfwOakI5Q2EPnPAZn14Buon/e8T/v92N2CJttC04V/QLHSjFaa lf13GOh0BXdX5TMFMOrbSzm3IwgBE5LRenvW802qOjrVLKEWx/TdeQjyF2R+eoDy1amC iyVaPJiGA+pnT4MZ4Us38jceZjEsB3aWab5+oY5pTpaopuf0XPlG3x86AYL5a/dcBkVh 0T6Q== X-Gm-Message-State: AOAM532cyhLJZXbqfxKXUM1zsmantTluGYAxe8XUesATk+AhKOU3vcYV uHb50uN1IlcQ3zdNMt3TF+zD7uOA2Xo8 X-Google-Smtp-Source: ABdhPJwQwv/y/Y1CqtU5mX7QFPielulT4k9vdUFWl45cr8FFafLXMgEO9ymMubzKsO1L14g5Q1NqlrmfyD6Z X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:de05:: with SMTP id v5mr744456wmg.25.1614697231398; Tue, 02 Mar 2021 07:00:31 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:41 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-12-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 11/32] KVM: arm64: Stub CONFIG_DEBUG_LIST at Hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150201_307757_822B8959 X-CRM114-Status: GOOD ( 13.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to use the kernel list library at EL2, introduce stubs for the CONFIG_DEBUG_LIST out-of-lines calls. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/stub.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/stub.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 24ff99e2eac5..144da72ad510 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/stub.c b/arch/arm64/kvm/hyp/nvhe/stub.c new file mode 100644 index 000000000000..c0aa6bbfd79d --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/stub.c @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Stubs for out-of-line function calls caused by re-using kernel + * infrastructure at EL2. + * + * Copyright (C) 2020 - Google LLC + */ + +#include + +#ifdef CONFIG_DEBUG_LIST +bool __list_add_valid(struct list_head *new, struct list_head *prev, + struct list_head *next) +{ + return true; +} + +bool __list_del_entry_valid(struct list_head *entry) +{ + return true; +} +#endif From patchwork Tue Mar 2 14:59:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B956C433DB for ; Wed, 3 Mar 2021 16:03:22 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 293A964ED7 for ; Wed, 3 Mar 2021 16:03:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 293A964ED7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=g5kIeQmWPGyZTTeB5S3gUWdZfgwr6+W8YfqAmYjTj3I=; b=OtrG74qc2tcdc2 tBhjtoRDc9KN8sQZ44mjkLAYzQ06ZYfCRjN85KduO6MQRXF0ejOvmXDeXdr2wwssCDQm7xpL9DAbf CpltxcHVPe+yz9YIyDBX3dearwojIFrLr602s9dFgLnCeejI7Ai/C5roGBlBwTDsLkELCxrYuj9lx LOmQdqLj0BnpES3OH837grjaAHjSrKNa5tdLTQI/W2UEEO1Z3Yf3YhIYNLJQcSt46kYNtgMM8FEXc roMCiXWlTDuevaEIZbAF3ejNs5JxqvyIArjgVllr3jffeZywucqc+Iyib/wPcMsfN8qVttbsk0/Bk BFWJNixCOTcERq14YlhA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHTvk-005VQN-AM; Wed, 03 Mar 2021 16:00:56 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSoC-005DWo-KV for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=pye8MWijHDuX/6U4nUOVsXqZBW6SBmLY4zPnwQvFKsw=; b=f1rq1NN9N3kJppjbNu+wyp0WA3 o5CFTV/R6qjsAFIQTFLS7JNtRUGSiMn9OnUCJlsZe9JHx6PYz/jeLlVtNHQeLS9pQwcYFhiQ5nFGt H2uvFGYpIbPAy8Q1CMlJz2EOYyh/l8PrcVqSsP7byu51vfnEYY0ZNYJWZ3nReewocqtawzNtO5yLJ UP1SWXVdxfiqtdsZR9ygC+jW4auOk/gx3RN9GyEvgrRt1vvgKRfDpD5olDbHfB18jbJqPQrF6Yqtz O4JGPaLRBBv42Oyj3V049g8Fdf5dRAhDiosKTZr3BPdwzvMWkxRJRMJ08pl/PS8LVwurzLw2WVrXM Rw86oF5A==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wq-00HJIm-Fs for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:46 +0000 Received: by mail-wr1-x449.google.com with SMTP id l10so11228005wry.16 for ; Tue, 02 Mar 2021 07:01:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=pye8MWijHDuX/6U4nUOVsXqZBW6SBmLY4zPnwQvFKsw=; b=epDHIE/L93TKXA7lQa8OgDcHaSB6tPztVKPOaQGywIiQJfROqLgjmNZK9s7BuQEZCQ ulFzjV+XDin1zWhRy8bl5TKBaam+7aCd02v73F1fQuvP6LrLFa+qY6uy0wKCay6qCNNj 5wxAx9if0cS2zloTC9veY+x64MQq4HxvCWH7ZI/jSL60OUuXLCOSQwLVLDNyWT2og9YT vHMc3yMYUucpvG9ksdaVNy2d7/0Sn50mcjTrrBsQCmtkxDRDn4wlY3Iiv6m5oVfp21Fk ND8I3ldY9T+swh8x9j3PvEtyR3ZJM+aprvLfel2/snbaDBuWfmkADrYb3ARs0SOLLuLP uczQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=pye8MWijHDuX/6U4nUOVsXqZBW6SBmLY4zPnwQvFKsw=; b=FqDtU6UAjlYrqMVvjhmr8IDiYgkvdNfZpDGfpWER2xG0N3gbGNUvFWQ0uzb5ImrIpf UqdNPWOwPG11Rc+I0EUj9Fa+46zGHHd6W5aO8gPlTkXQUJm8cpD0xfuHCRR1vPtwAtc7 0XFjwFHm3kEZluaBDHjdikx0Dn2Dueo9C0dFmWrdiZf1TA02sDtfrd59Ya2YZ8In9Hxc iqY9T+ke5r4Q5OdCJyt9nibd4WfgO/b4Hni8YcObxLgTYJeC9Uu5Mytm4aGffVoYGtJB vUt9jgWH+4nKj1sUn3o97gSucUVefnufHz7iGPqqHQpFJicSlOtmiH7hOIuKelZaMdlz +jcw== X-Gm-Message-State: AOAM531u3/kjcApEifRSBAalaH/6ElG0A6qr3Yfo92QXnWD9/KuNdxNC ta+eDs3H45dgrPUMdnAn3bRQHD1+dvWa X-Google-Smtp-Source: ABdhPJyfV1MJIORGvvergETmz4TjrbeLzl0o8XcLTDwcrpsGfbKpZNLJA1ZH5Z44AlNYao0NkvRZHPdZwRj+ X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:adf:e548:: with SMTP id z8mr23029463wrm.246.1614697233474; Tue, 02 Mar 2021 07:00:33 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:42 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-13-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 12/32] KVM: arm64: Introduce a Hyp buddy page allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150145_973149_76DD41E5 X-CRM114-Status: GOOD ( 27.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the hyp code will require a basic form of memory management in order to allocate and free memory pages at EL2. This is needed for various use-cases, including the creation of hyp mappings or the allocation of stage 2 page tables. To address these use-case, introduce a simple memory allocator in the hyp code. The allocator is designed as a conventional 'buddy allocator', working with a page granularity. It allows to allocate and free physically contiguous pages from memory 'pools', with a guaranteed order alignment in the PA space. Each page in a memory pool is associated with a struct hyp_page which holds the page's metadata, including its refcount, as well as its current order, hence mimicking the kernel's buddy system in the GFP infrastructure. The hyp_page metadata are made accessible through a hyp_vmemmap, following the concept of SPARSE_VMEMMAP in the kernel. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 55 +++++++ arch/arm64/kvm/hyp/include/nvhe/memory.h | 28 ++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 195 +++++++++++++++++++++++ 4 files changed, 279 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/gfp.h create mode 100644 arch/arm64/kvm/hyp/nvhe/page_alloc.c diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h new file mode 100644 index 000000000000..d039086d86b5 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_GFP_H +#define __KVM_HYP_GFP_H + +#include + +#include +#include + +#define HYP_NO_ORDER UINT_MAX + +struct hyp_pool { + /* + * Spinlock protecting concurrent changes to the memory pool as well as + * the struct hyp_page of the pool's pages until we have a proper atomic + * API at EL2. + */ + hyp_spinlock_t lock; + struct list_head free_area[MAX_ORDER]; + phys_addr_t range_start; + phys_addr_t range_end; + unsigned int max_order; +}; + +static inline void hyp_page_ref_inc(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + p->refcount++; + hyp_spin_unlock(&pool->lock); +} + +static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + int ret; + + hyp_spin_lock(&pool->lock); + p->refcount--; + ret = (p->refcount == 0); + hyp_spin_unlock(&pool->lock); + + return ret; +} + +/* Allocation */ +void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order); +void hyp_get_page(void *addr); +void hyp_put_page(void *addr); + +/* Used pages cannot be freed */ +int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages, + unsigned int reserved_pages); +#endif /* __KVM_HYP_GFP_H */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/memory.h b/arch/arm64/kvm/hyp/include/nvhe/memory.h index 3e49eaa7e682..d2fb307c5952 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/memory.h +++ b/arch/arm64/kvm/hyp/include/nvhe/memory.h @@ -6,7 +6,17 @@ #include +struct hyp_pool; +struct hyp_page { + unsigned int refcount; + unsigned int order; + struct hyp_pool *pool; + struct list_head node; +}; + extern s64 hyp_physvirt_offset; +extern u64 __hyp_vmemmap; +#define hyp_vmemmap ((struct hyp_page *)__hyp_vmemmap) #define __hyp_pa(virt) ((phys_addr_t)(virt) + hyp_physvirt_offset) #define __hyp_va(phys) ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset)) @@ -21,4 +31,22 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr) return __hyp_pa(addr); } +#define hyp_phys_to_pfn(phys) ((phys) >> PAGE_SHIFT) +#define hyp_pfn_to_phys(pfn) ((phys_addr_t)((pfn) << PAGE_SHIFT)) +#define hyp_phys_to_page(phys) (&hyp_vmemmap[hyp_phys_to_pfn(phys)]) +#define hyp_virt_to_page(virt) hyp_phys_to_page(__hyp_pa(virt)) +#define hyp_virt_to_pfn(virt) hyp_phys_to_pfn(__hyp_pa(virt)) + +#define hyp_page_to_pfn(page) ((struct hyp_page *)(page) - hyp_vmemmap) +#define hyp_page_to_phys(page) hyp_pfn_to_phys((hyp_page_to_pfn(page))) +#define hyp_page_to_virt(page) __hyp_va(hyp_page_to_phys(page)) +#define hyp_page_to_pool(page) (((struct hyp_page *)page)->pool) + +static inline int hyp_page_count(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + return p->refcount; +} + #endif /* __KVM_HYP_MEMORY_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 144da72ad510..6894a917f290 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,7 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c new file mode 100644 index 000000000000..348fc4329e8f --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include + +u64 __hyp_vmemmap; + +/* + * Index the hyp_vmemmap to find a potential buddy page, but make no assumption + * about its current state. + * + * Example buddy-tree for a 4-pages physically contiguous pool: + * + * o : Page 3 + * / + * o-o : Page 2 + * / + * / o : Page 1 + * / / + * o---o-o : Page 0 + * Order 2 1 0 + * + * Example of requests on this pool: + * __find_buddy_nocheck(pool, page 0, order 0) => page 1 + * __find_buddy_nocheck(pool, page 0, order 1) => page 2 + * __find_buddy_nocheck(pool, page 1, order 0) => page 0 + * __find_buddy_nocheck(pool, page 2, order 0) => page 3 + */ +static struct hyp_page *__find_buddy_nocheck(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + phys_addr_t addr = hyp_page_to_phys(p); + + addr ^= (PAGE_SIZE << order); + + /* + * Don't return a page outside the pool range -- it belongs to + * something else and may not be mapped in hyp_vmemmap. + */ + if (addr < pool->range_start || addr >= pool->range_end) + return NULL; + + return hyp_phys_to_page(addr); +} + +/* Find a buddy page currently available for allocation */ +static struct hyp_page *__find_buddy_avail(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + struct hyp_page *buddy = __find_buddy_nocheck(pool, p, order); + + if (!buddy || buddy->order != order || list_empty(&buddy->node)) + return NULL; + + return buddy; + +} + +static void __hyp_attach_page(struct hyp_pool *pool, + struct hyp_page *p) +{ + unsigned int order = p->order; + struct hyp_page *buddy; + + memset(hyp_page_to_virt(p), 0, PAGE_SIZE << p->order); + + /* + * Only the first struct hyp_page of a high-order page (otherwise known + * as the 'head') should have p->order set. The non-head pages should + * have p->order = HYP_NO_ORDER. Here @p may no longer be the head + * after coallescing, so make sure to mark it HYP_NO_ORDER proactively. + */ + p->order = HYP_NO_ORDER; + for (; (order + 1) < pool->max_order; order++) { + buddy = __find_buddy_avail(pool, p, order); + if (!buddy) + break; + + /* Take the buddy out of its list, and coallesce with @p */ + list_del_init(&buddy->node); + buddy->order = HYP_NO_ORDER; + p = (p < buddy) ? p : buddy; + } + + /* Mark the new head, and insert it */ + p->order = order; + list_add_tail(&p->node, &pool->free_area[order]); +} + +static void hyp_attach_page(struct hyp_page *p) +{ + struct hyp_pool *pool = hyp_page_to_pool(p); + + hyp_spin_lock(&pool->lock); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); +} + +static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, + struct hyp_page *p, + unsigned int order) +{ + struct hyp_page *buddy; + + list_del_init(&p->node); + while (p->order > order) { + /* + * The buddy of order n - 1 currently has HYP_NO_ORDER as it + * is covered by a higher-level page (whose head is @p). Use + * __find_buddy_nocheck() to find it and inject it in the + * free_list[n - 1], effectively splitting @p in half. + */ + p->order--; + buddy = __find_buddy_nocheck(pool, p, p->order); + buddy->order = p->order; + list_add_tail(&buddy->node, &pool->free_area[buddy->order]); + } + + return p; +} + +void hyp_put_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + if (hyp_page_ref_dec_and_test(p)) + hyp_attach_page(p); +} + +void hyp_get_page(void *addr) +{ + struct hyp_page *p = hyp_virt_to_page(addr); + + hyp_page_ref_inc(p); +} + +void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) +{ + unsigned int i = order; + struct hyp_page *p; + + hyp_spin_lock(&pool->lock); + + /* Look for a high-enough-order page */ + while (i < pool->max_order && list_empty(&pool->free_area[i])) + i++; + if (i >= pool->max_order) { + hyp_spin_unlock(&pool->lock); + return NULL; + } + + /* Extract it from the tree at the right order */ + p = list_first_entry(&pool->free_area[i], struct hyp_page, node); + p = __hyp_extract_page(pool, p, order); + + hyp_spin_unlock(&pool->lock); + hyp_page_ref_inc(p); + + return p ? hyp_page_to_virt(p) : NULL; +} + +int hyp_pool_init(struct hyp_pool *pool, u64 pfn, unsigned int nr_pages, + unsigned int reserved_pages) +{ + phys_addr_t phys = hyp_pfn_to_phys(pfn); + struct hyp_page *p; + int i; + + hyp_spin_lock_init(&pool->lock); + pool->max_order = min(MAX_ORDER, get_order(nr_pages << PAGE_SHIFT)); + for (i = 0; i < pool->max_order; i++) + INIT_LIST_HEAD(&pool->free_area[i]); + pool->range_start = phys; + pool->range_end = phys + (nr_pages << PAGE_SHIFT); + + /* Init the vmemmap portion */ + p = hyp_phys_to_page(phys); + memset(p, 0, sizeof(*p) * nr_pages); + for (i = 0; i < nr_pages; i++) { + p[i].pool = pool; + INIT_LIST_HEAD(&p[i].node); + } + + /* Attach the unused pages to the buddy tree */ + for (i = reserved_pages; i < nr_pages; i++) + __hyp_attach_page(pool, &p[i]); + + return 0; +} From patchwork Tue Mar 2 14:59:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79987C433DB for ; Wed, 3 Mar 2021 19:46:21 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D96B664ED0 for ; Wed, 3 Mar 2021 19:46:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D96B664ED0 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=K1iVbiL8y4Y8D1zc4mUMQuiMBB//9GcyfmzjFIZapfw=; b=LLrWqnB/8sFf/M Cw1oteQDtWqwgPlAwKKrHP37zidbUVFL52wUBZUQqJrPNH+qR8jgAexL0wq36o8rTYn3SIyLLC1wq 61CMInGKE23iNruvF6qPd0ZUw2Mnpb+oC8LiHXjSTBuyZThuPuMxNKVvVUScrqiMLCP5QIVMrJj1V DL6j0oQhY9oH5A9bc+XBbCIzq8Dt6h/bBgKfgeq8q0WpN9FR50JavdlDLcHU+c/eC+R9PLAH3zgdK ufWVRotJ8aBCKo73RTMw2k9q6GSq5h6ERl0llYCnFl4xQh1RPBykc65XBpSEy6t8o4PcrbH8p1C+6 fGjKwuLJb0KEVFTVPCVg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHXP4-006GgT-2G; Wed, 03 Mar 2021 19:43:26 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT46-005IFu-Ex for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:05:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Nj4Lfx4RBIkW7BkT83ganNaPcX2b1UPE5UuHOUQIP/k=; b=KruiqkJbw98oPPjGSmN9mgSFOH 1VxQEcSl4VViqF443697owOzjRzJSVvX6TR7SOw3Ex9D+Ndq+jE8KN0wDK/FAxp8nk3LT22rCrQRL Xm5TL19BaTj9C4g7d2AnkMU1GCEcOI3QcmzgbgHo04nn9iHrJ1rayeTzjUbfkMGapguZqG2h/YWJm LkX8NTJZGMGWlVvkrWvODJaP2a4U69BQ3cYWUR6MDLOTo5SPx70QIZ27OemVu5TDvzK1WQgST9wSZ ocJl2iu7slInzqyWRj5CgWN4KyWAdQyAZfaPlyI/vI5d/K547N/TgPMpE0O1tSk12KPMyqeXF8uRA +t3XaiOA==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XD-00HJJF-1Y for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:06 +0000 Received: by mail-wm1-x349.google.com with SMTP id h16so1303100wmq.8 for ; Tue, 02 Mar 2021 07:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Nj4Lfx4RBIkW7BkT83ganNaPcX2b1UPE5UuHOUQIP/k=; b=tmr1KtLaO4/NP9tlla1YXcoWiHt6S/lyYGmlGgJ0Zr5eKBoywZKo2fFE09qqkxGgE+ +699HOE+XAFcrR4AIYIZbOQfv9aFekmPN6fd1G1UiOcjzq5yVvyjR/NzqaVfFFisI9d5 MOaOrIWVXy4K8JbLSQ5IRbi6sMMtMohkuOewhMFCG0uggzp1b38bUpp3QU6nRnv9g9cN CQf9B18zjiwgYCJ75IFtxXykGgPP9io4KwYSTZ7g2zRXI+IlUDXKMqZDAJEWA41VeXl9 UPdWjbh0T/ghx5K4u/p1SX2ZDllUuthzTnBO0A5+543kn1/J7ZhuiVyqHTME1tmkE06i CSHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Nj4Lfx4RBIkW7BkT83ganNaPcX2b1UPE5UuHOUQIP/k=; b=Knph8mW6+SGeIwtRNfm6n/RjtOT2dXyOuGn7AdZ7SbdpSUM1/h+Vs4cw7J6ixZoYoZ S94YjIBI/TqrLSZ+3M4UXjHgEvzkvqv9+mN82jlCYVkzamxYw5/63xLf2VXGBLIUHyG4 NZITTUa/jTW+LnctiY+zt8tK+T6U14e1snw4OKjcaS0UixuG9zADLlfPa4WwHNy2yoh5 ejDKuVfPnLrUwxChzV7OGky0fsTLJFY9Asly9ESiq39HQZUpS/kzOksRALlt5HQB4GvS 2x2vkbM0dz10g7jIqSWhWwgtALAVruachBEHnUppcqEUG45swaZt9rx5zIi9BfUlo8EF kBkQ== X-Gm-Message-State: AOAM533Xw+2e1fIZcyDBswkocpEg8ooJnydkvRXduC5rFqsURZdFkxDD kfHRtuk8padYmtKX7ICCz0Ut1q2hI6GL X-Google-Smtp-Source: ABdhPJwvpQIv+znxityhogRj6zoOeSFEDLfbBBRns10dBwlnGJToVJKnQCrYhTdzCU5GSNUlw+0Kk2km+ios X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a5d:43cc:: with SMTP id v12mr14983560wrr.287.1614697235745; Tue, 02 Mar 2021 07:00:35 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:43 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-14-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 13/32] KVM: arm64: Enable access to sanitized CPU features at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150206_530983_6A384C21 X-CRM114-Status: GOOD ( 19.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce the infrastructure in KVM enabling to copy CPU feature registers into EL2-owned data-structures, to allow reading sanitised values directly at EL2 in nVHE. Given that only a subset of these features are being read by the hypervisor, the ones that need to be copied are to be listed under together with the name of the nVHE variable that will hold the copy. While at it, introduce the first user of this infrastructure by implementing __flush_dcache_area at EL2, which needs arm64_ftr_reg_ctrel0. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/include/asm/kvm_cpufeature.h | 17 +++++++++++++++++ arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kernel/cpufeature.c | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/cache.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/cpufeature.c | 8 ++++++++ arch/arm64/kvm/sys_regs.c | 21 +++++++++++++++++++++ 8 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/kvm_cpufeature.h create mode 100644 arch/arm64/kvm/hyp/nvhe/cache.S create mode 100644 arch/arm64/kvm/hyp/nvhe/cpufeature.c diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 61177bac49fa..a85cea2cac57 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -607,6 +607,7 @@ void check_local_cpu_capabilities(void); u64 read_sanitised_ftr_reg(u32 id); u64 __read_sysreg_by_encoding(u32 sys_id); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst); static inline bool cpu_supports_mixed_endian_el0(void) { diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h new file mode 100644 index 000000000000..d34f85cba358 --- /dev/null +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include + +#ifndef KVM_HYP_CPU_FTR_REG +#if defined(__KVM_NVHE_HYPERVISOR__) +#define KVM_HYP_CPU_FTR_REG(id, name) extern struct arm64_ftr_reg name; +#else +#define KVM_HYP_CPU_FTR_REG(id, name) DECLARE_KVM_NVHE_SYM(name); +#endif +#endif + +KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 06ca4828005f..459ee557f87c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -751,9 +751,13 @@ void kvm_clr_pmu_events(u32 clr); void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu); void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); + +void setup_kvm_el2_caps(void); #else static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {} static inline void kvm_clr_pmu_events(u32 clr) {} + +static inline void setup_kvm_el2_caps(void) {} #endif void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 066030717a4c..f2d8b479ff74 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1154,6 +1154,18 @@ u64 read_sanitised_ftr_reg(u32 id) } EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); +int copy_ftr_reg(u32 id, struct arm64_ftr_reg *dst) +{ + struct arm64_ftr_reg *regp = get_arm64_ftr_reg(id); + + if (!regp) + return -EINVAL; + + memcpy(dst, regp, sizeof(*regp)); + + return 0; +} + #define read_sysreg_case(r) \ case r: val = read_sysreg_s(r); break; @@ -2773,6 +2785,7 @@ void __init setup_cpu_features(void) setup_system_capabilities(); setup_elf_hwcaps(arm64_elf_hwcaps); + setup_kvm_el2_caps(); if (system_supports_32bit_el0()) setup_elf_hwcaps(compat_elf_hwcaps); diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 6894a917f290..0033591553fc 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -13,7 +13,8 @@ lib-objs := clear_page.o copy_page.o memcpy.o memset.o lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ - hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o + hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ + cache.o cpufeature.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/cache.S b/arch/arm64/kvm/hyp/nvhe/cache.S new file mode 100644 index 000000000000..36cef6915428 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cache.S @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Code copied from arch/arm64/mm/cache.S. + */ + +#include +#include +#include + +SYM_FUNC_START_PI(__flush_dcache_area) + dcache_by_line_op civac, sy, x0, x1, x2, x3 + ret +SYM_FUNC_END_PI(__flush_dcache_area) diff --git a/arch/arm64/kvm/hyp/nvhe/cpufeature.c b/arch/arm64/kvm/hyp/nvhe/cpufeature.c new file mode 100644 index 000000000000..a887508f996f --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/cpufeature.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#define KVM_HYP_CPU_FTR_REG(id, name) struct arm64_ftr_reg name; +#include diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 4f2f1e3145de..84be93df52fa 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -2775,3 +2776,23 @@ void kvm_sys_reg_table_init(void) /* Clear all higher bits. */ cache_levels &= (1 << (i*3))-1; } + +#undef KVM_HYP_CPU_FTR_REG +#define KVM_HYP_CPU_FTR_REG(id, name) \ + { .sys_id = id, .dst = (struct arm64_ftr_reg *)&kvm_nvhe_sym(name) }, +struct __ftr_reg_copy_entry { + u32 sys_id; + struct arm64_ftr_reg *dst; +} hyp_ftr_regs[] __initdata = { + #include +}; + +void __init setup_kvm_el2_caps(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(hyp_ftr_regs); i++) { + WARN(copy_ftr_reg(hyp_ftr_regs[i].sys_id, hyp_ftr_regs[i].dst), + "%u feature register not found\n", hyp_ftr_regs[i].sys_id); + } +} From patchwork Tue Mar 2 14:59:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2519C433E0 for ; Wed, 3 Mar 2021 15:00:34 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E7ED60240 for ; Wed, 3 Mar 2021 15:00:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E7ED60240 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0a4WhGor+qH54PVEY+iMLI75WyS/nzsxK6cOcs4RVJM=; b=NKhlzhBvQqZKr+ ZPPvA7jw5aRdni464CSeAterVGrdc/kvm4WNjl/J509O+BT4rkVZwPJkrElLaepK0QkfcNMfg2YRS JDAAer3SLHbKRH772QHpnF2f8xap1ksIL3QpWoXQF3yduUq2Sz1MqenfyyN4RMw8pv6YVJ3Bk2LZG QkbCi2TWpFUIH1mEODyyrZIqz5eCXDJlD8klafXHsAnGohjwDDlnfFoKOawoQYAN7AWYyYERqP4sY Avk9ZHJ5P877gIlHCEi5vk1BOM/AYoCcFZILLkyyf0jDdgjhhH4VXWTErbIi++S9HBlcwF8gyxEMh Njh4MoAgQ0ye9WB8jGGQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHSwM-005Fmw-9c; Wed, 03 Mar 2021 14:57:32 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSBT-0054NU-PT for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:09:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=84bkslXA7Qi7zlX0xHNbjR4XGFTrfSQqHB0l+SkjRTk=; b=WCfbmWYNNqr9gexqlv5bbMOBn0 E+1j+1YGmLAFMTC8ENhn87vDv/JfN902cbtq0wwPFh7nQLfXNCUErPB17Yv0UHjVQO7soZWkjeLBt J40ZufL/qu/2qvh0p9x+Aaq3PXwe2/TuMcR7Y3yVPCr4sQWSXKpcP/zirKskDwySHU1WvcPGNcm9g +VtUqXSTT720T/XDCCuRgWO+q9nyYLxXYtWTE04a99jjZrpnJV5brah8TPtbR/ibpzybGy5vytW5p pwTdYr9QcBch4DIJdYUcJl9u90/3gqABRTjsy9SDh9IUc84iZzfWHMpCDUsrvwta2llVnjfiWt4FJ ljZ8Ps1w==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XE-00HJJG-OL for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:07 +0000 Received: by mail-wr1-x44a.google.com with SMTP id m9so5489908wrx.6 for ; Tue, 02 Mar 2021 07:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=84bkslXA7Qi7zlX0xHNbjR4XGFTrfSQqHB0l+SkjRTk=; b=aEyIjsz9Q39OmQlGKhPsdWMeCxwA+8gfzAIcUr3vtuH7vJkQ/TC8zCkBwOeeBT65tq UvSouJ8Birgm8kdd91FSgiyZCYROychF8+9Sg1dSYXCER9zOEopyAV4P/kgm4Ysxeidy 7ft7SR+/KlrHCJgsZMEZ51u1Cmk9vBYllJNwbmgZ1ompvmxQEqTIg6O0Ypw4YW+TiLNY Bdfy60xclmeqbqcp8GDWtDDVvsqYWew2CdzBsipN3fqfz7oI3broHa+hdFZwgg01rnPc IAtkChQhjHl8hf9R0x8GkU6smUb+gwabjgaL22vSTdg2PoB4sWRTIVOgQBrjdOfDE6Mh olZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=84bkslXA7Qi7zlX0xHNbjR4XGFTrfSQqHB0l+SkjRTk=; b=gA7NkhodVdWimsMrGCSCWhdVWQB5IJ5VeULgUkayIYRjaE4JuJVHWf7zzgDLwjvqid /ktn3nto6EjO5zQx/vAU5TMkguPuBTICFuOsTBQGTJqIBTxN1N3It+s5c/2O6Q9CpBk1 Ko+2rg39WJIB8lsY6YyfNRDQKd8cjkON4f1eUfZnheaKquQiCZbpHbEByp/8kzlAIpUE 4ymJXmIa+tmEQRVhioEmGVDI9MJaRcPC4MzHlTU3n/k/Gs4Tqfl8gDnXuRxL5BgWdnsI iplwxSr9qMnr9CfTE8yrZ6jqrob8I7yDS451lseJA1UA0pe7mEyeNwH9V0C++0m5QXJe 4AjA== X-Gm-Message-State: AOAM531qtf6CGEfwAMCkUnL4vWd56J9oZrbHj7ZKsCqVXG4bBIceLhIx mMJVp/aniV33DurbJIRUUhE6HfpzM/x9 X-Google-Smtp-Source: ABdhPJwFUzBVYVUNUz65kgYHvUCXtvq4NXrt8gCrlNsOFSMTng7gs+AjbNDT7VknGe5P7PaLfd9Y7549RF9U X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:2e90:: with SMTP id u138mr496527wmu.0.1614697237854; Tue, 02 Mar 2021 07:00:37 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:44 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-15-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 14/32] KVM: arm64: Factor out vector address calculation From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150207_913707_F283F2B2 X-CRM114-Status: GOOD ( 11.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-map the guest vectors at EL2 when pKVM is enabled, refactor __kvm_vector_slot2idx() and kvm_init_vector_slot() to move all the address calculation logic in a static inline function. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 8 ++++++++ arch/arm64/kvm/arm.c | 9 +-------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 90873851f677..5c42ec023cc7 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -168,6 +168,14 @@ phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); int kvm_mmu_init(void); +static inline void *__kvm_vector_slot2addr(void *base, + enum arm64_hyp_spectre_vector slot) +{ + int idx = slot - (slot != HYP_VECTOR_DIRECT); + + return base + (idx * SZ_2K); +} + struct kvm; #define kvm_flush_dcache_to_poc(a,l) __flush_dcache_area((a), (l)) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 3f8bcf8db036..26e573cdede3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1345,16 +1345,9 @@ static unsigned long nvhe_percpu_order(void) /* A lookup table holding the hypervisor VA for each vector slot */ static void *hyp_spectre_vector_selector[BP_HARDEN_EL2_SLOTS]; -static int __kvm_vector_slot2idx(enum arm64_hyp_spectre_vector slot) -{ - return slot - (slot != HYP_VECTOR_DIRECT); -} - static void kvm_init_vector_slot(void *base, enum arm64_hyp_spectre_vector slot) { - int idx = __kvm_vector_slot2idx(slot); - - hyp_spectre_vector_selector[slot] = base + (idx * SZ_2K); + hyp_spectre_vector_selector[slot] = __kvm_vector_slot2addr(base, slot); } static int kvm_init_vector_slots(void) From patchwork Tue Mar 2 14:59:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAA5BC433DB for ; Wed, 3 Mar 2021 19:24:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 45A1364EBB for ; Wed, 3 Mar 2021 19:24:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 45A1364EBB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ksNTg5iakQMTT1yX60kQ02curwSU7QXJEDgeJzE02ek=; b=j2I6VsQCPXDJ0W seDSBlH/wdYl4VSuyO/5YslwbbTwQUwCtyPEFEDcovr6+BWcu2EwjKJAxyRDY0J5z/TZ6gvvxKFOU T9G884aabXWEB+zVaS3dE02HHRmYukWC9wPXFwLOKscAl/I7wlKulpUksCfIaFztN1vZ1ekbQiLv9 zy4iRBqadueCXuN8TmUGE92/Y+yPeFFEzTobsMearOr26vzhaer9nomhgEdSGQjCIgvGyM6a9Ho/0 Jb06w392FRdr4rEFS1E9V87VsW0TcCusDn6wl0jZKk7kBOeAfSnOj66H5fDd5KywGq68SDoibiXh2 5wqJuRAz3lGT0cxDGJrA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHX4Y-006C3n-9O; Wed, 03 Mar 2021 19:22:14 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT39-005HxE-B6 for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:04:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=NEnt3wiXIT5QMfy/+XzfhNHFcefIECwKmv2frqwDPNM=; b=KLXlsu07Y2fCmJqQRJXQ9jXZQc H5t3r6qsjIl74P1X5FRir1AczGenkfVNZCRvXeAQ7xC5uRfTEb/HhOJQrDTxKXFeu9no7cF/+QU3v EVPpRkhYgokTk1dHos/UvTEkZUJyn85xN+3yMo42SoBhauRVBbxYRVlFx/40KhEHHqjWirEcBaBnQ sTVY38Wm4ZCKdbFSThyHLSM9H+UF2KbQDylqVBu6iw+gRQCt9C3YrFJpdnPqSwEigmy7AUhz3sIz/ SeVZkJtDZjOGGGCzanp4KJzGVd9tDEjMO+rCSz0wG/vAWkJjagDiFXaAJmLsajY9C1llUhL6iAxXt HqZ04XtA==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XJ-00HJJo-CO for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:12 +0000 Received: by mail-wr1-x449.google.com with SMTP id h30so11248465wrh.10 for ; Tue, 02 Mar 2021 07:01:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=NEnt3wiXIT5QMfy/+XzfhNHFcefIECwKmv2frqwDPNM=; b=JjXvVinN5rNedtUuBbq4vuYTCovP3dk6Zetgmj5UKnyvBZDfOjZN72np9YleG9xWeJ 3B8ieMBnTB+t4O2ebINTKGetWCLS9RPFigmwveHL7p5FVzv7QUl9MfkDzdOEOU9NoShY VjjtwZd8zUJ8AqjlIV84Dm5RaLS6Hdj0JUxIDdHmgGblh76c3A+9gp66xokYhQwjPl40 vF+e3fEHoElwIpcqzAG9Uv1hoxS6oQ8onoCsD7GPQe1s4m/AYEyFezvWZHIf42flZfbT DaBkEf0u7tSaYrUXBUOyMxhkDavxSU2ByzHejBqdwYBbMUezeSr3twJSFbDYjPy4gsVB ke3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NEnt3wiXIT5QMfy/+XzfhNHFcefIECwKmv2frqwDPNM=; b=JGtycInALP7t4YsWXwiSu7OcCnNsk/X2w1U9n1DgeZaG8+9UzatLOKaF9kcpWAIocg sra30ifTu3KqPTj+ZvRkfuMsD/lwTJZPh3Sy/ktBFoHG1ByZA+P2F0ZPMzTdTfm9WwLY 4QRrWr7j61kR9nplgHxKc7szflVBisYZ473/eN5hq1+O05kFvlh0IzHRVXBQvkPJRGSt upCtbP9gdsHT4nqwa+9weQahl+x06+HekNzE44R8YcQcYyB3dEqKEbbWyKQKX6zfVS4J 68NujDl/7xW8CVpQjCV41Y3iKK2OR9nCfvGCSZhAuBy5MsC8JSPjLVbaztk4AymTv8LK QpGA== X-Gm-Message-State: AOAM532JJ0ArmimT0tVw2HzNzQEHr3QFJUAfjhyO4X1IA+QemWoGbRWQ naGzHdt4Gp7a/plHBBL7uYbcJwueeiZn X-Google-Smtp-Source: ABdhPJzrz1NTB0MU1KqEhGkHw+tfkaLGzzObYKV9UbgPzXQSfTOZ772mB09MovnMa3P7LasNQSJOPkkMhJzM X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c3c1:: with SMTP id t1mr4405363wmj.47.1614697240220; Tue, 02 Mar 2021 07:00:40 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:45 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-16-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 15/32] KVM: arm64: Prepare the creation of s1 mappings at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150212_695055_C3AA2D32 X-CRM114-Status: GOOD ( 31.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When memory protection is enabled, the EL2 code needs the ability to create and manage its own page-table. To do so, introduce a new set of hypercalls to bootstrap a memory management system at EL2. This leads to the following boot flow in nVHE Protected mode: 1. the host allocates memory for the hypervisor very early on, using the memblock API; 2. the host creates a set of stage 1 page-table for EL2, installs the EL2 vectors, and issues the __pkvm_init hypercall; 3. during __pkvm_init, the hypervisor re-creates its stage 1 page-table and stores it in the memory pool provided by the host; 4. the hypervisor then extends its stage 1 mappings to include a vmemmap in the EL2 VA space, hence allowing to use the buddy allocator introduced in a previous patch; 5. the hypervisor jumps back in the idmap page, switches from the host-provided page-table to the new one, and wraps up its initialization by enabling the new allocator, before returning to the host. 6. the host can free the now unused page-table created for EL2, and will now need to issue hypercalls to make changes to the EL2 stage 1 mappings instead of modifying them directly. Note that for the sake of simplifying the review, this patch focuses on the hypervisor side of things. In other words, this only implements the new hypercalls, but does not make use of them from the host yet. The host-side changes will follow in a subsequent patch. Credits to Will for __pkvm_init_switch_pgd. Co-authored-by: Will Deacon Signed-off-by: Will Deacon Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_host.h | 7 + arch/arm64/include/asm/kvm_hyp.h | 8 ++ arch/arm64/include/asm/kvm_pgtable.h | 2 + arch/arm64/kernel/image-vars.h | 16 +++ arch/arm64/kvm/hyp/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/mm.h | 71 ++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 4 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 31 +++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 49 +++++++ arch/arm64/kvm/hyp/nvhe/mm.c | 173 ++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 195 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 2 - arch/arm64/kvm/hyp/reserved_mem.c | 92 +++++++++++++ arch/arm64/mm/init.c | 3 + 15 files changed, 654 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mm.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mm.c create mode 100644 arch/arm64/kvm/hyp/nvhe/setup.c create mode 100644 arch/arm64/kvm/hyp/reserved_mem.c diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 22d933e9b59e..db20a9477870 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -57,6 +57,10 @@ #define __KVM_HOST_SMCCC_FUNC___kvm_get_mdcr_el2 12 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs 13 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs 14 +#define __KVM_HOST_SMCCC_FUNC___pkvm_init 15 +#define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16 +#define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 +#define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 459ee557f87c..b9d45a1f8538 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -781,5 +781,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features)) int kvm_trng_call(struct kvm_vcpu *vcpu); +#ifdef CONFIG_KVM +extern phys_addr_t hyp_mem_base; +extern phys_addr_t hyp_mem_size; +void __init kvm_hyp_reserve(void); +#else +static inline void kvm_hyp_reserve(void) { } +#endif #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index c0450828378b..ae55351b99a4 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -100,4 +100,12 @@ void __noreturn hyp_panic(void); void __noreturn __hyp_do_panic(bool restore_host, u64 spsr, u64 elr, u64 par); #endif +#ifdef __KVM_NVHE_HYPERVISOR__ +void __pkvm_init_switch_pgd(phys_addr_t phys, unsigned long size, + phys_addr_t pgd, void *sp, void *cont_fn); +int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, + unsigned long *per_cpu_base, u32 hyp_va_bits); +void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); +#endif + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 3c306f90f7da..7898610fbeeb 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -11,6 +11,8 @@ #include #include +#define KVM_PGTABLE_MAX_LEVELS 4U + typedef u64 kvm_pte_t; /** diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 4eb7a15c8b60..940c378fa837 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -115,6 +115,22 @@ KVM_NVHE_ALIAS_HYP(__memcpy, __pi_memcpy); KVM_NVHE_ALIAS_HYP(__memset, __pi_memset); #endif +/* Kernel memory sections */ +KVM_NVHE_ALIAS(__start_rodata); +KVM_NVHE_ALIAS(__end_rodata); +KVM_NVHE_ALIAS(__bss_start); +KVM_NVHE_ALIAS(__bss_stop); + +/* Hyp memory sections */ +KVM_NVHE_ALIAS(__hyp_idmap_text_start); +KVM_NVHE_ALIAS(__hyp_idmap_text_end); +KVM_NVHE_ALIAS(__hyp_text_start); +KVM_NVHE_ALIAS(__hyp_text_end); +KVM_NVHE_ALIAS(__hyp_bss_start); +KVM_NVHE_ALIAS(__hyp_bss_end); +KVM_NVHE_ALIAS(__hyp_rodata_start); +KVM_NVHE_ALIAS(__hyp_rodata_end); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile index 687598e41b21..b726332eec49 100644 --- a/arch/arm64/kvm/hyp/Makefile +++ b/arch/arm64/kvm/hyp/Makefile @@ -10,4 +10,4 @@ subdir-ccflags-y := -I$(incdir) \ -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o +obj-$(CONFIG_KVM) += vhe/ nvhe/ pgtable.o reserved_mem.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h new file mode 100644 index 000000000000..ac0f7fcffd08 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_HYP_MM_H +#define __KVM_HYP_MM_H + +#include +#include +#include +#include + +#include +#include + +#define HYP_MEMBLOCK_REGIONS 128 +extern struct memblock_region kvm_nvhe_sym(hyp_memory)[]; +extern unsigned int kvm_nvhe_sym(hyp_memblock_nr); +extern struct kvm_pgtable pkvm_pgtable; +extern hyp_spinlock_t pkvm_pgd_lock; +extern struct hyp_pool hpool; +extern u64 __io_map_base; + +int hyp_create_idmap(u32 hyp_va_bits); +int hyp_map_vectors(void); +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); +int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); +int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +int __pkvm_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, enum kvm_pgtable_prot prot); +unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot); + +static inline void hyp_vmemmap_range(phys_addr_t phys, unsigned long size, + unsigned long *start, unsigned long *end) +{ + unsigned long nr_pages = size >> PAGE_SHIFT; + struct hyp_page *p = hyp_phys_to_page(phys); + + *start = (unsigned long)p; + *end = *start + nr_pages * sizeof(struct hyp_page); + *start = ALIGN_DOWN(*start, PAGE_SIZE); + *end = ALIGN(*end, PAGE_SIZE); +} + +static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) +{ + unsigned long total = 0, i; + + /* Provision the worst case scenario */ + for (i = 0; i < KVM_PGTABLE_MAX_LEVELS; i++) { + nr_pages = DIV_ROUND_UP(nr_pages, PTRS_PER_PTE); + total += nr_pages; + } + + return total; +} + +static inline unsigned long hyp_s1_pgtable_pages(void) +{ + unsigned long res = 0, i; + + /* Cover all of memory with page-granularity */ + for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) { + struct memblock_region *reg = &kvm_nvhe_sym(hyp_memory)[i]; + res += __hyp_pgtable_max_pages(reg->size >> PAGE_SHIFT); + } + + /* Allow 1 GiB for private mappings */ + res += __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); + + return res; +} +#endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 0033591553fc..e204ea77ab27 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -14,9 +14,9 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o + cache.o cpufeature.o setup.o mm.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ - ../fpsimd.o ../hyp-entry.o ../exception.o + ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) ## diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index c631e29fb001..bc56ea92b812 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -244,4 +244,35 @@ alternative_else_nop_endif SYM_CODE_END(__kvm_handle_stub_hvc) +SYM_FUNC_START(__pkvm_init_switch_pgd) + /* Turn the MMU off */ + pre_disable_mmu_workaround + mrs x2, sctlr_el2 + bic x3, x2, #SCTLR_ELx_M + msr sctlr_el2, x3 + isb + + tlbi alle2 + + /* Install the new pgtables */ + ldr x3, [x0, #NVHE_INIT_PGD_PA] + phys_to_ttbr x4, x3 +alternative_if ARM64_HAS_CNP + orr x4, x4, #TTBR_CNP_BIT +alternative_else_nop_endif + msr ttbr0_el2, x4 + + /* Set the new stack pointer */ + ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] + mov sp, x0 + + /* And turn the MMU back on! */ + dsb nsh + isb + msr sctlr_el2, x2 + ic iallu + isb + ret x1 +SYM_FUNC_END(__pkvm_init_switch_pgd) + .popsection diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f012f8665ecc..ae6503c9be15 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -6,12 +6,14 @@ #include +#include #include #include #include #include #include +#include #include DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); @@ -106,6 +108,49 @@ static void handle___vgic_v3_restore_aprs(struct kvm_cpu_context *host_ctxt) __vgic_v3_restore_aprs(kern_hyp_va(cpu_if)); } +static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, nr_cpus, host_ctxt, 3); + DECLARE_REG(unsigned long *, per_cpu_base, host_ctxt, 4); + DECLARE_REG(u32, hyp_va_bits, host_ctxt, 5); + + /* + * __pkvm_init() will return only if an error occurred, otherwise it + * will tail-call in __pkvm_init_finalise() which will have to deal + * with the host context directly. + */ + cpu_reg(host_ctxt, 1) = __pkvm_init(phys, size, nr_cpus, per_cpu_base, + hyp_va_bits); +} + +static void handle___pkvm_cpu_set_vector(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(enum arm64_hyp_spectre_vector, slot, host_ctxt, 1); + + cpu_reg(host_ctxt, 1) = pkvm_cpu_set_vector(slot); +} + +static void handle___pkvm_create_mappings(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(unsigned long, start, host_ctxt, 1); + DECLARE_REG(unsigned long, size, host_ctxt, 2); + DECLARE_REG(unsigned long, phys, host_ctxt, 3); + DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 4); + + cpu_reg(host_ctxt, 1) = __pkvm_create_mappings(start, size, phys, prot); +} + +static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, phys, host_ctxt, 1); + DECLARE_REG(size_t, size, host_ctxt, 2); + DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); + + cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); +} + typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -125,6 +170,10 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_get_mdcr_el2), HANDLE_FUNC(__vgic_v3_save_aprs), HANDLE_FUNC(__vgic_v3_restore_aprs), + HANDLE_FUNC(__pkvm_init), + HANDLE_FUNC(__pkvm_cpu_set_vector), + HANDLE_FUNC(__pkvm_create_mappings), + HANDLE_FUNC(__pkvm_create_private_mapping), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c new file mode 100644 index 000000000000..a8efdf0f9003 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct kvm_pgtable pkvm_pgtable; +hyp_spinlock_t pkvm_pgd_lock; +u64 __io_map_base; + +struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS]; +unsigned int hyp_memblock_nr; + +int __pkvm_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, enum kvm_pgtable_prot prot) +{ + int err; + + hyp_spin_lock(&pkvm_pgd_lock); + err = kvm_pgtable_hyp_map(&pkvm_pgtable, start, size, phys, prot); + hyp_spin_unlock(&pkvm_pgd_lock); + + return err; +} + +unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, + enum kvm_pgtable_prot prot) +{ + unsigned long addr; + int err; + + hyp_spin_lock(&pkvm_pgd_lock); + + size = PAGE_ALIGN(size + offset_in_page(phys)); + addr = __io_map_base; + __io_map_base += size; + + /* Are we overflowing on the vmemmap ? */ + if (__io_map_base > __hyp_vmemmap) { + __io_map_base -= size; + addr = (unsigned long)ERR_PTR(-ENOMEM); + goto out; + } + + err = kvm_pgtable_hyp_map(&pkvm_pgtable, addr, size, phys, prot); + if (err) { + addr = (unsigned long)ERR_PTR(err); + goto out; + } + + addr = addr + offset_in_page(phys); +out: + hyp_spin_unlock(&pkvm_pgd_lock); + + return addr; +} + +int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) +{ + unsigned long start = (unsigned long)from; + unsigned long end = (unsigned long)to; + unsigned long virt_addr; + phys_addr_t phys; + + start = start & PAGE_MASK; + end = PAGE_ALIGN(end); + + for (virt_addr = start; virt_addr < end; virt_addr += PAGE_SIZE) { + int err; + + phys = hyp_virt_to_phys((void *)virt_addr); + err = __pkvm_create_mappings(virt_addr, PAGE_SIZE, phys, prot); + if (err) + return err; + } + + return 0; +} + +int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back) +{ + unsigned long start, end; + + hyp_vmemmap_range(phys, size, &start, &end); + + return __pkvm_create_mappings(start, end - start, back, PAGE_HYP); +} + +static void *__hyp_bp_vect_base; +int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot) +{ + void *vector; + + switch (slot) { + case HYP_VECTOR_DIRECT: { + vector = __kvm_hyp_vector; + break; + } + case HYP_VECTOR_SPECTRE_DIRECT: { + vector = __bp_harden_hyp_vecs; + break; + } + case HYP_VECTOR_INDIRECT: + case HYP_VECTOR_SPECTRE_INDIRECT: { + vector = (void *)__hyp_bp_vect_base; + break; + } + default: + return -EINVAL; + } + + vector = __kvm_vector_slot2addr(vector, slot); + *this_cpu_ptr(&kvm_hyp_vector) = (unsigned long)vector; + + return 0; +} + +int hyp_map_vectors(void) +{ + phys_addr_t phys; + void *bp_base; + + if (!cpus_have_const_cap(ARM64_SPECTRE_V3A)) + return 0; + + phys = __hyp_pa(__bp_harden_hyp_vecs); + bp_base = (void *)__pkvm_create_private_mapping(phys, + __BP_HARDEN_HYP_VECS_SZ, + PAGE_HYP_EXEC); + if (IS_ERR_OR_NULL(bp_base)) + return PTR_ERR(bp_base); + + __hyp_bp_vect_base = bp_base; + + return 0; +} + +int hyp_create_idmap(u32 hyp_va_bits) +{ + unsigned long start, end; + + start = hyp_virt_to_phys((void *)__hyp_idmap_text_start); + start = ALIGN_DOWN(start, PAGE_SIZE); + + end = hyp_virt_to_phys((void *)__hyp_idmap_text_end); + end = ALIGN(end, PAGE_SIZE); + + /* + * One half of the VA space is reserved to linearly map portions of + * memory -- see va_layout.c for more details. The other half of the VA + * space contains the trampoline page, and needs some care. Split that + * second half in two and find the quarter of VA space not conflicting + * with the idmap to place the IOs and the vmemmap. IOs use the lower + * half of the quarter and the vmemmap the upper half. + */ + __io_map_base = start & BIT(hyp_va_bits - 2); + __io_map_base ^= BIT(hyp_va_bits - 2); + __hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3); + + return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c new file mode 100644 index 000000000000..178ec06f2b49 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -0,0 +1,195 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +struct hyp_pool hpool; +struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; +unsigned long hyp_nr_cpus; + +#define hyp_percpu_size ((unsigned long)__per_cpu_end - \ + (unsigned long)__per_cpu_start) + +static void *vmemmap_base; +static void *hyp_pgt_base; + +static int divide_memory_pool(void *virt, unsigned long size) +{ + unsigned long vstart, vend, nr_pages; + + hyp_early_alloc_init(virt, size); + + hyp_vmemmap_range(__hyp_pa(virt), size, &vstart, &vend); + nr_pages = (vend - vstart) >> PAGE_SHIFT; + vmemmap_base = hyp_early_alloc_contig(nr_pages); + if (!vmemmap_base) + return -ENOMEM; + + nr_pages = hyp_s1_pgtable_pages(); + hyp_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!hyp_pgt_base) + return -ENOMEM; + + return 0; +} + +static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, + unsigned long *per_cpu_base, + u32 hyp_va_bits) +{ + void *start, *end, *virt = hyp_phys_to_virt(phys); + unsigned long pgt_size = hyp_s1_pgtable_pages() << PAGE_SHIFT; + int ret, i; + + /* Recreate the hyp page-table using the early page allocator */ + hyp_early_alloc_init(hyp_pgt_base, pgt_size); + ret = kvm_pgtable_hyp_init(&pkvm_pgtable, hyp_va_bits, + &hyp_early_alloc_mm_ops); + if (ret) + return ret; + + ret = hyp_create_idmap(hyp_va_bits); + if (ret) + return ret; + + ret = hyp_map_vectors(); + if (ret) + return ret; + + ret = hyp_back_vmemmap(phys, size, hyp_virt_to_phys(vmemmap_base)); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_text_start, __hyp_text_end, PAGE_HYP_EXEC); + if (ret) + return ret; + + ret = pkvm_create_mappings(__start_rodata, __end_rodata, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_rodata_start, __hyp_rodata_end, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_bss_start, __hyp_bss_end, PAGE_HYP); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_bss_end, __bss_stop, PAGE_HYP_RO); + if (ret) + return ret; + + ret = pkvm_create_mappings(virt, virt + size, PAGE_HYP); + if (ret) + return ret; + + for (i = 0; i < hyp_nr_cpus; i++) { + start = (void *)kern_hyp_va(per_cpu_base[i]); + end = start + PAGE_ALIGN(hyp_percpu_size); + ret = pkvm_create_mappings(start, end, PAGE_HYP); + if (ret) + return ret; + + end = (void *)per_cpu_ptr(&kvm_init_params, i)->stack_hyp_va; + start = end - PAGE_SIZE; + ret = pkvm_create_mappings(start, end, PAGE_HYP); + if (ret) + return ret; + } + + return 0; +} + +static void update_nvhe_init_params(void) +{ + struct kvm_nvhe_init_params *params; + unsigned long i; + + for (i = 0; i < hyp_nr_cpus; i++) { + params = per_cpu_ptr(&kvm_init_params, i); + params->pgd_pa = __hyp_pa(pkvm_pgtable.pgd); + __flush_dcache_area(params, sizeof(*params)); + } +} + +static void *hyp_zalloc_hyp_page(void *arg) +{ + return hyp_alloc_pages(&hpool, 0); +} + +void __noreturn __pkvm_init_finalise(void) +{ + struct kvm_host_data *host_data = this_cpu_ptr(&kvm_host_data); + struct kvm_cpu_context *host_ctxt = &host_data->host_ctxt; + unsigned long nr_pages, reserved_pages, pfn; + int ret; + + /* Now that the vmemmap is backed, install the full-fledged allocator */ + pfn = hyp_virt_to_pfn(hyp_pgt_base); + nr_pages = hyp_s1_pgtable_pages(); + reserved_pages = hyp_early_alloc_nr_used_pages(); + ret = hyp_pool_init(&hpool, pfn, nr_pages, reserved_pages); + if (ret) + goto out; + + pkvm_pgtable_mm_ops.zalloc_page = hyp_zalloc_hyp_page; + pkvm_pgtable_mm_ops.phys_to_virt = hyp_phys_to_virt; + pkvm_pgtable_mm_ops.virt_to_phys = hyp_virt_to_phys; + pkvm_pgtable_mm_ops.get_page = hyp_get_page; + pkvm_pgtable_mm_ops.put_page = hyp_put_page; + pkvm_pgtable.mm_ops = &pkvm_pgtable_mm_ops; + +out: + /* + * We tail-called to here from handle___pkvm_init() and will not return, + * so make sure to propagate the return value to the host. + */ + cpu_reg(host_ctxt, 1) = ret; + + __host_enter(host_ctxt); +} + +int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, + unsigned long *per_cpu_base, u32 hyp_va_bits) +{ + struct kvm_nvhe_init_params *params; + void *virt = hyp_phys_to_virt(phys); + void (*fn)(phys_addr_t params_pa, void *finalize_fn_va); + int ret; + + if (phys % PAGE_SIZE || size % PAGE_SIZE) + return -EINVAL; + + hyp_spin_lock_init(&pkvm_pgd_lock); + hyp_nr_cpus = nr_cpus; + + ret = divide_memory_pool(virt, size); + if (ret) + return ret; + + ret = recreate_hyp_mappings(phys, size, per_cpu_base, hyp_va_bits); + if (ret) + return ret; + + update_nvhe_init_params(); + + /* Jump in the idmap page to switch to the new page-tables */ + params = this_cpu_ptr(&kvm_init_params); + fn = (typeof(fn))__hyp_pa(__pkvm_init_switch_pgd); + fn(__hyp_pa(params), __pkvm_init_finalise); + + unreachable(); +} diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b975a67d1f85..7ce0969203e8 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -10,8 +10,6 @@ #include #include -#define KVM_PGTABLE_MAX_LEVELS 4U - #define KVM_PTE_VALID BIT(0) #define KVM_PTE_TYPE BIT(1) diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c new file mode 100644 index 000000000000..9bc6a6d27904 --- /dev/null +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2020 - Google LLC + * Author: Quentin Perret + */ + +#include +#include + +#include + +#include +#include + +static struct memblock_region *hyp_memory = kvm_nvhe_sym(hyp_memory); +static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr); + +phys_addr_t hyp_mem_base; +phys_addr_t hyp_mem_size; + +static int __init register_memblock_regions(void) +{ + struct memblock_region *reg; + + for_each_mem_region(reg) { + if (*hyp_memblock_nr_ptr >= HYP_MEMBLOCK_REGIONS) + return -ENOMEM; + + hyp_memory[*hyp_memblock_nr_ptr] = *reg; + (*hyp_memblock_nr_ptr)++; + } + + return 0; +} + +void __init kvm_hyp_reserve(void) +{ + u64 nr_pages, prev, hyp_mem_pages = 0; + int ret; + + if (!is_hyp_mode_available() || is_kernel_in_hyp_mode()) + return; + + if (kvm_get_mode() != KVM_MODE_PROTECTED) + return; + + ret = register_memblock_regions(); + if (ret) { + *hyp_memblock_nr_ptr = 0; + kvm_err("Failed to register hyp memblocks: %d\n", ret); + return; + } + + hyp_mem_pages += hyp_s1_pgtable_pages(); + + /* + * The hyp_vmemmap needs to be backed by pages, but these pages + * themselves need to be present in the vmemmap, so compute the number + * of pages needed by looking for a fixed point. + */ + nr_pages = 0; + do { + prev = nr_pages; + nr_pages = hyp_mem_pages + prev; + nr_pages = DIV_ROUND_UP(nr_pages * sizeof(struct hyp_page), PAGE_SIZE); + nr_pages += __hyp_pgtable_max_pages(nr_pages); + } while (nr_pages != prev); + hyp_mem_pages += nr_pages; + + /* + * Try to allocate a PMD-aligned region to reduce TLB pressure once + * this is unmapped from the host stage-2, and fallback to PAGE_SIZE. + */ + hyp_mem_size = hyp_mem_pages << PAGE_SHIFT; + hyp_mem_base = memblock_find_in_range(0, memblock_end_of_DRAM(), + ALIGN(hyp_mem_size, PMD_SIZE), + PMD_SIZE); + if (!hyp_mem_base) + hyp_mem_base = memblock_find_in_range(0, memblock_end_of_DRAM(), + hyp_mem_size, PAGE_SIZE); + else + hyp_mem_size = ALIGN(hyp_mem_size, PMD_SIZE); + + if (!hyp_mem_base) { + kvm_err("Failed to reserve hyp memory\n"); + return; + } + memblock_reserve(hyp_mem_base, hyp_mem_size); + + kvm_info("Reserved %lld MiB at 0x%llx\n", hyp_mem_size >> 20, + hyp_mem_base); +} diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 0ace5e68efba..2b1b41b1ff9b 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -429,6 +430,8 @@ void __init bootmem_init(void) dma_pernuma_cma_reserve(); + kvm_hyp_reserve(); + /* * sparse_init() tries to allocate memory from memblock, so must be * done after the fixed reservations From patchwork Tue Mar 2 14:59:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25AB8C433DB for ; Wed, 3 Mar 2021 16:43:05 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AD39B614A7 for ; Wed, 3 Mar 2021 16:43:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD39B614A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mxC9/+qgZ5Q6xp7d2lSxEbMwsTlvCmntVs+kLfbVSJ0=; b=fqS6Pi/WVTj64D 2IYaFPr+Gxj5dCldjcMfD93xAiip1CIe8EftfutoyrRaaIZaQG5Pjaz+FTCUb7u7ZLE0UiMP7trPk yp0uraOOuFzbMQLnvrChZfoK7Br4c2/swk0Uz7pW4NjqudaP0o6J/YBvuh1i8+utFDX6KD+y5Hchj ErnKxEqlpbpfXS5nX7twlP+qy/PknqXPZ/pLrwRCvGElqa+8gtNgB4H9HMUkjZ13iXjj7kByXjR84 7EKUyi9Z+//K7EV4OPtc3vKEp7ZUpiFK6y2wGRVNlVIbkTiQ1Ts+jU/rYOMNG26tpNiSmCBoEQWJj ZfTOuH2qKBMGTGJjsnrw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHUYA-005dpf-T5; Wed, 03 Mar 2021 16:40:40 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSqY-005EGe-Ne for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:51:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=SyC2r4edwP6B0yWtbCjHDwPrvUJOrNqhRtLgavKZNac=; b=lqfgIzXyEM4lLEz1BBWctyQte6 Ueha7RDiCmq78eAZVja97MO1M6JTqq9pefdFVc1peoc9REUd372g08/asxHdSjjsUUee1OBOZ0sGd YPGLdkzxa7TgkwtKHvbE9JKzOuDq57vJO28LXXtsnZe3j8F4VPoDSWshTSE5dwxGzQeXLUvIDbiNc gRJlyEoqeBeHtywN8KiLLW0E+uvcm9mbTJDhJO3QblrbiGX1XzCz0qjioam0+V8hlGAEqBe2MQM8t y0qAs4w70riUe09Oe1qDa0XzhpyCauX3sa0/xXSVvZfROkrVhMNAnrrmnYjQph+ikqDOcHgOFDmEe 5iOkU3KA==; Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XH-00HJK4-SO for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:09 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id d15so15076852qvn.16 for ; Tue, 02 Mar 2021 07:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=SyC2r4edwP6B0yWtbCjHDwPrvUJOrNqhRtLgavKZNac=; b=Xqn+6oZNaHl/40KM2Wc7bBy6MtV4gYKmvgdRk9bfP4ZmCUU/VF963mPFlkvVo7js/2 WzTtK8XXLsS0Mh+FthXr84xC+sK+FgUR9E8U6H0sj3d+Yhx2MmfPoIrkiQnEwHKt+R0g BZ+hTNkISFQA8VAnEaUu7+ipW1hpl+g9zUpTDiAzQ7n5AeUOt+XoGmTmNXjSVC4i5tsv ZtFm2HZF7Omdgi2JPr1Ca8X55ToX+ug8EzyMPVPwQ18uocaj7msXq8q23ry1OBeiy2Ob fFc3r8b6O6F0Fw2srPJMBAWfzi8EyIR3Zt7fs++XsQGAzevRU5xo1Sf61aTKkN5cprwp XX0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SyC2r4edwP6B0yWtbCjHDwPrvUJOrNqhRtLgavKZNac=; b=YksfaBnrSRR2iQV8RM3KbLU5zBgDnDW811MAlWNC8tCG1WK7suaVI4oY1aVhxMGBaY nIiTKGA91cvWFt1X7acaImW2NN7j26Ukw+5ClPOp5JKxuJr8lN5qMzhcwRulJPg0NCsz W9UWtf+Avy4H/QDpyryDkVpWDaStjZV1yZ5WpbT88uyLtY1sAjqb3aih5EBeOAcWQGrW FqUmp9TdU7BfrWYTQSa5+TiBnr0M2cABWEraPr3QHp72CGge4johkOtseobEQ7Wa5r8U vTFpJ+Esk6ZEXIjNmV/RbUZRLTHHu4Aj3FXhGf+ttletUXNh6xebKNUKVgCc4u2huPNc xdcw== X-Gm-Message-State: AOAM533fBESAOpMVeJyf7e7aqywHjpcZpQDRI+xrVs8Z4Im3uWgQH7bC 6vPgUEXLqOcVoHDquhN2xl63pj90GXIY X-Google-Smtp-Source: ABdhPJw7QYqL2U7XfAj+Gjs6VA7D5Zu5g4S0ut2jMfRdsJ2Foi4hzz5d401xlIBhBsaM1dnO3TcsvNZU9z2v X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:ea29:: with SMTP id t9mr3984058qvp.52.1614697242537; Tue, 02 Mar 2021 07:00:42 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:46 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-17-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 16/32] KVM: arm64: Elevate hypervisor mappings creation at EL2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150209_916715_F046270F X-CRM114-Status: GOOD ( 22.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Previous commits have introduced infrastructure to enable the EL2 code to manage its own stage 1 mappings. However, this was preliminary work, and none of it is currently in use. Put all of this together by elevating the mapping creation at EL2 when memory protection is enabled. In this case, the host kernel running at EL1 still creates _temporary_ EL2 mappings, only used while initializing the hypervisor, but frees them right after. As such, all calls to create_hyp_mappings() after kvm init has finished turn into hypercalls, as the host now has no 'legal' way to modify the hypevisor page tables directly. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 2 +- arch/arm64/kvm/arm.c | 87 +++++++++++++++++++++++++++++--- arch/arm64/kvm/mmu.c | 43 ++++++++++++++-- 3 files changed, 120 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 5c42ec023cc7..ce02a4052dcf 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -166,7 +166,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); phys_addr_t kvm_mmu_get_httbr(void); phys_addr_t kvm_get_idmap_vector(void); -int kvm_mmu_init(void); +int kvm_mmu_init(u32 *hyp_va_bits); static inline void *__kvm_vector_slot2addr(void *base, enum arm64_hyp_spectre_vector slot) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 26e573cdede3..5a97abdc3ba6 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1421,7 +1421,7 @@ static void cpu_prepare_hyp_mode(int cpu) kvm_flush_dcache_to_poc(params, sizeof(*params)); } -static void cpu_init_hyp_mode(void) +static void hyp_install_host_vector(void) { struct kvm_nvhe_init_params *params; struct arm_smccc_res res; @@ -1439,6 +1439,11 @@ static void cpu_init_hyp_mode(void) params = this_cpu_ptr_nvhe_sym(kvm_init_params); arm_smccc_1_1_hvc(KVM_HOST_SMCCC_FUNC(__kvm_hyp_init), virt_to_phys(params), &res); WARN_ON(res.a0 != SMCCC_RET_SUCCESS); +} + +static void cpu_init_hyp_mode(void) +{ + hyp_install_host_vector(); /* * Disabling SSBD on a non-VHE system requires us to enable SSBS @@ -1481,7 +1486,10 @@ static void cpu_set_hyp_vector(void) struct bp_hardening_data *data = this_cpu_ptr(&bp_hardening_data); void *vector = hyp_spectre_vector_selector[data->slot]; - *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + if (!is_protected_kvm_enabled()) + *this_cpu_ptr_hyp_sym(kvm_hyp_vector) = (unsigned long)vector; + else + kvm_call_hyp_nvhe(__pkvm_cpu_set_vector, data->slot); } static void cpu_hyp_reinit(void) @@ -1489,13 +1497,14 @@ static void cpu_hyp_reinit(void) kvm_init_host_cpu_context(&this_cpu_ptr_hyp_sym(kvm_host_data)->host_ctxt); cpu_hyp_reset(); - cpu_set_hyp_vector(); if (is_kernel_in_hyp_mode()) kvm_timer_init_vhe(); else cpu_init_hyp_mode(); + cpu_set_hyp_vector(); + kvm_arm_init_debug(); if (vgic_present) @@ -1691,18 +1700,59 @@ static void teardown_hyp_mode(void) } } +static int do_pkvm_init(u32 hyp_va_bits) +{ + void *per_cpu_base = kvm_ksym_ref(kvm_arm_hyp_percpu_base); + int ret; + + preempt_disable(); + hyp_install_host_vector(); + ret = kvm_call_hyp_nvhe(__pkvm_init, hyp_mem_base, hyp_mem_size, + num_possible_cpus(), kern_hyp_va(per_cpu_base), + hyp_va_bits); + preempt_enable(); + + return ret; +} + +static int kvm_hyp_init_protection(u32 hyp_va_bits) +{ + void *addr = phys_to_virt(hyp_mem_base); + int ret; + + ret = create_hyp_mappings(addr, addr + hyp_mem_size, PAGE_HYP); + if (ret) + return ret; + + ret = do_pkvm_init(hyp_va_bits); + if (ret) + return ret; + + free_hyp_pgds(); + + return 0; +} + /** * Inits Hyp-mode on all online CPUs */ static int init_hyp_mode(void) { + u32 hyp_va_bits; int cpu; - int err = 0; + int err = -ENOMEM; + + /* + * The protected Hyp-mode cannot be initialized if the memory pool + * allocation has failed. + */ + if (is_protected_kvm_enabled() && !hyp_mem_base) + return err; /* * Allocate Hyp PGD and setup Hyp identity mapping */ - err = kvm_mmu_init(); + err = kvm_mmu_init(&hyp_va_bits); if (err) goto out_err; @@ -1818,6 +1868,14 @@ static int init_hyp_mode(void) goto out_err; } + if (is_protected_kvm_enabled()) { + err = kvm_hyp_init_protection(hyp_va_bits); + if (err) { + kvm_err("Failed to init hyp memory protection\n"); + goto out_err; + } + } + return 0; out_err: @@ -1826,6 +1884,16 @@ static int init_hyp_mode(void) return err; } +static int finalize_hyp_mode(void) +{ + if (!is_protected_kvm_enabled()) + return 0; + + static_branch_enable(&kvm_protected_mode_initialized); + + return 0; +} + static void check_kvm_target_cpu(void *ret) { *(int *)ret = kvm_target_cpu(); @@ -1942,8 +2010,15 @@ int kvm_arch_init(void *opaque) if (err) goto out_hyp; + if (!in_hyp_mode) { + err = finalize_hyp_mode(); + if (err) { + kvm_err("Failed to finalize Hyp protection\n"); + goto out_hyp; + } + } + if (is_protected_kvm_enabled()) { - static_branch_enable(&kvm_protected_mode_initialized); kvm_info("Protected nVHE mode initialized successfully\n"); } else if (in_hyp_mode) { kvm_info("VHE mode initialized successfully\n"); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 4d41d7838d53..9d331bf262d2 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -221,15 +221,39 @@ void free_hyp_pgds(void) if (hyp_pgtable) { kvm_pgtable_hyp_destroy(hyp_pgtable); kfree(hyp_pgtable); + hyp_pgtable = NULL; } mutex_unlock(&kvm_hyp_pgd_mutex); } +static bool kvm_host_owns_hyp_mappings(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + return false; + + /* + * This can happen at boot time when __create_hyp_mappings() is called + * after the hyp protection has been enabled, but the static key has + * not been flipped yet. + */ + if (!hyp_pgtable && is_protected_kvm_enabled()) + return false; + + WARN_ON(!hyp_pgtable); + + return true; +} + static int __create_hyp_mappings(unsigned long start, unsigned long size, unsigned long phys, enum kvm_pgtable_prot prot) { int err; + if (!kvm_host_owns_hyp_mappings()) { + return kvm_call_hyp_nvhe(__pkvm_create_mappings, + start, size, phys, prot); + } + mutex_lock(&kvm_hyp_pgd_mutex); err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot); mutex_unlock(&kvm_hyp_pgd_mutex); @@ -291,6 +315,16 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, unsigned long base; int ret = 0; + if (!kvm_host_owns_hyp_mappings()) { + base = kvm_call_hyp_nvhe(__pkvm_create_private_mapping, + phys_addr, size, prot); + if (IS_ERR_OR_NULL((void *)base)) + return PTR_ERR((void *)base); + *haddr = base; + + return 0; + } + mutex_lock(&kvm_hyp_pgd_mutex); /* @@ -1270,10 +1304,9 @@ static struct kvm_pgtable_mm_ops kvm_hyp_mm_ops = { .virt_to_phys = kvm_host_pa, }; -int kvm_mmu_init(void) +int kvm_mmu_init(u32 *hyp_va_bits) { int err; - u32 hyp_va_bits; hyp_idmap_start = __pa_symbol(__hyp_idmap_text_start); hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE); @@ -1287,8 +1320,8 @@ int kvm_mmu_init(void) */ BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); - hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); - kvm_debug("Using %u-bit virtual addresses at EL2\n", hyp_va_bits); + *hyp_va_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + kvm_debug("Using %u-bit virtual addresses at EL2\n", *hyp_va_bits); kvm_debug("IDMAP page: %lx\n", hyp_idmap_start); kvm_debug("HYP VA range: %lx:%lx\n", kern_hyp_va(PAGE_OFFSET), @@ -1313,7 +1346,7 @@ int kvm_mmu_init(void) goto out; } - err = kvm_pgtable_hyp_init(hyp_pgtable, hyp_va_bits, &kvm_hyp_mm_ops); + err = kvm_pgtable_hyp_init(hyp_pgtable, *hyp_va_bits, &kvm_hyp_mm_ops); if (err) goto out_free_pgtable; From patchwork Tue Mar 2 14:59:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26E5DC433DB for ; Wed, 3 Mar 2021 14:06:27 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D959B64E7C for ; Wed, 3 Mar 2021 14:06:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D959B64E7C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xf2C4vXfQRS+a3VyDL+G2Z1+BLgZq9P7Nkm9E60QK8o=; b=d4a95oaIY1J+wA VbeQsRMC7h1F5VnTVoVdOEFFKyLdReSjEbG19KOv8GlGjqzqATlqP3bNBfbwnNJMmmqB3JN3Q1Iig InYyDaDje4+Q/jQP5CT+QSgYz/tWr+gMOjbQ/DEVRL6UHCS8KtgnwFGTArTytytlO7GWXmFzSers1 amwYzg2gAsIj6z/i0UPiFUrPTN8Ej/R1XssCezfKgWuZeZ8fAMO+hZZLjEBbhzNo4cuwLJgDFnyyN N8YAlGqSg63P1t5+szGOxbxXon+YgxTyYGWliMK3pRaejcfgbgOGlzKNgcZN8zQl7zWgiP14t2XzT bZyTTKXgfAgn+6UrPk0g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHS5k-0052LQ-Nu; Wed, 03 Mar 2021 14:03:08 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHS3R-0051p6-Pc for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:00:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=G2AeCN7EWhkuYtFbkPhDKNEy1oVSwJAjORcxRM9VGLM=; b=VCMwmVpWmgTZ93ZYERr6qGqw2E 7fGUxhohhwhpK/HKMuarjrXsTt18jdg71+ASwE6AtlYf8n/62M42tqRJar0N7lLxYNhLWZ4JoJmsQ YK2qGB+gfNDIqUpXVGoKBFQb0uKmQ0jBWi45/x6hvtHuZfZoHl73q0V0T9+At/PfwpYOswEGZ0jHf awVJ9iMqULXUcFYm34OX6X7zP55OwloKav85eS61p1DmydAgn85R0QzJf4btrTQDgDii8B4OXAM69 EESbSSLukfTqyYfCL1TYFhaCRxRKqSKPXV6kSUSIsI2cyGLekyKcLI2U610KR0mZ2Mu6In0KG7CPj s+HC+WSQ==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XK-00HJKP-B2 for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:13 +0000 Received: by mail-wr1-x449.google.com with SMTP id f3so4384475wrt.14 for ; Tue, 02 Mar 2021 07:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=G2AeCN7EWhkuYtFbkPhDKNEy1oVSwJAjORcxRM9VGLM=; b=Dt0TxCXfs9c8jH2bVodVvIJerD5ufsP7ZFcR78FzQli/GcMwnUQb9f1H+QfiLX46BM 2Fb6lkOenRdyIPSYRLBFiuqS9jrfJxP3AY2xLec7qqciaO5yu+6Simfe8Mx9bfqQmGqz 6znfOSv6M9SiQhinpKMc1WKEI0hAgIcSppjPBroVtDfipQDXbG7fAZNBsRLkK7pzwkr6 C21ukzmfaq6LTMD814VkT42lxaZvxsN5wPoYRFXyus1F/Yi0LAthE+VMNbfcNW2LbVSb ms8jbMf49WD8yLYKISSizdbQgNBbKlpR2ugbE8R0HcQnaEk0BBl2KMImZCWPE3WpvvLw uUdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=G2AeCN7EWhkuYtFbkPhDKNEy1oVSwJAjORcxRM9VGLM=; b=mRM6mr6gcqA16tufUICKn/txFODciFQB53+dNN1OVH/CmXgCtb/2xrYapzSb8KsXFR JvMIZXixq2JZuqo+sY5oWe86njNuIkrn/jcT1rzFvbFXszSjTkDCx51m49mdDlZBOdmU SIJ62V8U0Xe0ZRZ+nh0myYZgGVntxZaE7FX0c7WFkusAfkPy/QMKcczdgIfeqPvSImpi K4fOVUowzNk6CWadlNmu8jfmv5jXA8CKXoNvNdPcKwjTy6AyDBEpXjCXK0HD2BL6v6wF baoYMh3jW2OwBW4mytHEv5wfwlgum3iMRK87X3Cw+nYQKSlEu0VngDHfzpWTy3QW2jH8 nLmg== X-Gm-Message-State: AOAM531ICWXuBhZjq9f4OqeL8HFOiDONd+kVwvURJrnHKuCsZNVC10sT 97/Y5awHajo10I99IUfYYgx+qTY4K8dt X-Google-Smtp-Source: ABdhPJwLj20ogsXve2nNaQuzCTevuLCR5AMDfnsGA+66q9RbQn2Lw/suoS7gkshS8KR9yQywFM1C2SmNTxIY X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:8041:: with SMTP id b62mr4675280wmd.0.1614697244902; Tue, 02 Mar 2021 07:00:44 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:47 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-18-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 17/32] KVM: arm64: Use kvm_arch for stage 2 pgtable From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150213_421002_439A1DA3 X-CRM114-Status: GOOD ( 12.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, use struct kvm_arch in lieu of struct kvm as the host will have the former but not the latter. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 5 +++-- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- arch/arm64/kvm/mmu.c | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 7898610fbeeb..a8255d55c168 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -162,12 +162,13 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, /** * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. - * @kvm: KVM structure representing the guest virtual machine. + * @arch: Arch-specific KVM structure representing the guest virtual + * machine. * @mm_ops: Memory management callbacks. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops); /** diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 7ce0969203e8..3d79c8094cdd 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -879,11 +879,11 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, +int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops) { size_t pgd_sz; - u64 vtcr = kvm->arch.vtcr; + u64 vtcr = arch->vtcr; u32 ia_bits = VTCR_EL2_IPA(vtcr); u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; @@ -896,7 +896,7 @@ int kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm *kvm, pgt->ia_bits = ia_bits; pgt->start_level = start_level; pgt->mm_ops = mm_ops; - pgt->mmu = &kvm->arch.mmu; + pgt->mmu = &arch->mmu; /* Ensure zeroed PGD pages are visible to the hardware walker */ dsb(ishst); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9d331bf262d2..41f9c03cbcc3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -457,7 +457,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, kvm, &kvm_s2_mm_ops); + err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; From patchwork Tue Mar 2 14:59:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A35DC433E0 for ; Wed, 3 Mar 2021 18:09:51 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B7F8B64EE4 for ; Wed, 3 Mar 2021 18:09:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7F8B64EE4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eOli8+ez0tBg8X4Ik3hJXecbQQgvaoFvL9EGRteHWoQ=; b=X0nt4Wbd/6QYIX 7zI9nGkvgz2map331SOVCxeb68QDOWjxcJacOQ7wSR2v9EZvcTZlQ4mcQPxp0xl+S+rsg1w+GSWVK f8ulAmw687P/6nKPc4PgN2vjGfclkCxDI6QvOni/q3gDYtIwdozGfeDDTemRQ/JBOboQJFEnhZ75j QRWt4dRkkiZSoK3hhYmzMZ7M7x5yu53I2YfftVV+5DbuutqFYtpGbaTNyzFIUAlF4HNC/n3V8isV2 gOt0OSKzis2Hgl61FpeJ3NX0nNpi9n2iDjMcA1AgN7Hr57GndYzDzAZwX5ZYqo2iUV9Hk4QuVa8A3 56N+xQRPuIloAePMwTpQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHVu6-005wqd-Ux; Wed, 03 Mar 2021 18:07:26 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSyb-005GRV-NA for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:59:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=bwje5/mTjAPXp3ucjvZl1d3xYnpG4DOJqI6qSewZTKg=; b=rLCArenzA89LGAJ46BTkQzROrk wwKoOuFjYtN7Lk4IPTVcCSd1g3AMKEfndlaXODfLLgYp21czsC6URJ3oaC8E2x/o0Kwou58bUZeIu ptukeqtv5vj33HynXL/g38L7oXuBS/RtkS4xr0GXRDmePvKVf4FZnLvJJdbsvYAwzhUBWj9+cWpWh pYkbq2AYoEWbM/gakmVIjInWm82IUBw1yg11oGexgaafSpE7Q8zg/gD5mZNSfhBV9j1EI9yi5sGB0 sT8e5qT9sNu80trTY21OZISEgQT13N9kjFH3iYv5FWNgdM5l93NmiFaHWB9sGwqOPwcPAwEKvIZJn E8Cx5VKA==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XG-00HJKQ-5D for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:09 +0000 Received: by mail-wm1-x34a.google.com with SMTP id c9so697356wme.5 for ; Tue, 02 Mar 2021 07:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=bwje5/mTjAPXp3ucjvZl1d3xYnpG4DOJqI6qSewZTKg=; b=tSH114d1S+uNQoX1kRD+PT///jcNFZH5jQl4y1I39FKxDMU/3Sk/nzK2nY32s3DZf9 mL7k3exN2AHB4ZPq66MGZyG7q1FEaXJALsmwnbUFPXQI/YMQ/GmienaMnA3fR32YF/0i XTK0G6ii03eAGLnaGVfNIse2zOnd+Zg7Xl7jhIdY9I8AE1QjfhEfugAQDu99o4kpmYVD 26I0Hhq2R71cD5A2582Cf2YgETUOfD4hGCDIviMAgjW9yes0SiyksL6bpH2ex5Iljrzf W8T6pvOEg+SLKYRCoHtIR/ruVvjA92TJcf94f9E66xMC2DC/9t07gQ3jGjztpQdJQR9e XK0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bwje5/mTjAPXp3ucjvZl1d3xYnpG4DOJqI6qSewZTKg=; b=nCYXmcd5GDqxnbS6P2mmDuVh9IPO5ELgq7t+zaP4DDHk6GFap3H3zAplRNUFMgXvR/ FoU8CcxozJwnWA8VAzIxJ4VtRe3vVFa6iVT+g6wpvycuboqIlNFDXQ+iWX2H6BEeWNYR gM+6KU81S1uWDH3C+AZ9WBPiz3Vu2pLHMG91IN6jypUpZ2yoouXK6FqCz02+1BhjcGd5 ZscwO7wR1Ja1Fw3UvtffS/TF5ZPsdoXU9RzD/CuiEpB6fNgrP8nlAcQPiG//DlgWFu2H aya+9PAJarY25WKW6lC3joALOkmiafaCrCLVqDsfnZk3sQL69s5yoCWEYa5/DjInwWE2 1Zvw== X-Gm-Message-State: AOAM530ar9JNBNYdGxWJbaNF+bp4jVQz9lc527XIQp44PhBWMtM8e3dL ejNp79mxxg4Nel44oJQ7eHCYGWqKt0+a X-Google-Smtp-Source: ABdhPJxElOYIMAIdgYSrQfXOebmZeuQAKDUTFyUl0M9C/M5W0sRKgrJje34shxxo7rpQfJ8MEv13l1k/xiXX X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:6000:24b:: with SMTP id m11mr14742698wrz.393.1614697247009; Tue, 02 Mar 2021 07:00:47 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:48 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-19-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 18/32] KVM: arm64: Use kvm_arch in kvm_s2_mmu From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150209_497190_C03032E3 X-CRM114-Status: GOOD ( 14.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to make use of the stage 2 pgtable code for the host stage 2, change kvm_s2_mmu to use a kvm_arch pointer in lieu of the kvm pointer, as the host will have the former but not the latter. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/include/asm/kvm_mmu.h | 6 +++++- arch/arm64/kvm/mmu.c | 8 ++++---- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index b9d45a1f8538..90565782ce3e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -94,7 +94,7 @@ struct kvm_s2_mmu { /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; - struct kvm *kvm; + struct kvm_arch *arch; }; struct kvm_arch_memory_slot { diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ce02a4052dcf..6f743e20cb06 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -272,7 +272,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) */ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) { - write_sysreg(kern_hyp_va(mmu->kvm)->arch.vtcr, vtcr_el2); + write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -283,5 +283,9 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) +{ + return container_of(mmu->arch, struct kvm, arch); +} #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 41f9c03cbcc3..3257cadfab24 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -165,7 +165,7 @@ static void *kvm_host_va(phys_addr_t phys) static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size, bool may_block) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); phys_addr_t end = start + size; assert_spin_locked(&kvm->mmu_lock); @@ -470,7 +470,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; - mmu->kvm = kvm; + mmu->arch = &kvm->arch; mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); mmu->vmid.vmid_gen = 0; @@ -552,7 +552,7 @@ void stage2_unmap_vm(struct kvm *kvm) void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); struct kvm_pgtable *pgt = NULL; spin_lock(&kvm->mmu_lock); @@ -621,7 +621,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, */ static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { - struct kvm *kvm = mmu->kvm; + struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect); } From patchwork Tue Mar 2 14:59:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB00FC433E6 for ; Wed, 3 Mar 2021 14:25:06 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 775F864EDF for ; Wed, 3 Mar 2021 14:25:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 775F864EDF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rhRJQOKIrdsADLlQDL0tOty7pkhJPj1h901U44lwpQ8=; b=pufl8Oe4ZbPNIW cLeYfaxtu70Nt4OPlwuw01/U1bulWWGkCErOa8LuyMIyBX7ajD8RE+g/RDfDCRgI3tXGaQfEf5oMR QqqeJEFSXk1fAvpooJNz2OsvySor7h9wH8wssdvsiD0LWIczNXlJKxj1HJwhk7ZIuhSYjvLzaF+Dm h0IO41DVcLmCQoPrse7GRdvEq8MtjNmxt++PUhhn2vNLqBvQ4FjVGyIOm1LfTbNAL7Z8yvPEK+MlD 8kGL87/1qJfz0atdpJJ2cbzc+8I1xhKTwQFz28kr4/zUhAJzxQ+KCFrjUwGyem19HfPZKC9IKWF3C 5J6G7QV/i+755flHSvLg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHSNW-0056kK-Pd; Wed, 03 Mar 2021 14:21:31 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHS7O-0052mS-SZ for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:04:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Mj/u+lI3QsG0AfvrUV/jHTPuQYjw+9cw0HbYqsdheKs=; b=m5bjWEYMK79N6PjvcPjQWT2sJn 8xlMfkeG62g6PEbAA7QivL8t4+iKpw8GbI6UQX0b3iRaSeNQX4NUdvdIcJU2GJTTLisb+um4gNc/q wV+hDH3gQQBWQXLqC9lGRgzNLBg+TU427nbcsH3jxNUh3FqBH/2uJ4rUbhK85nbjkCp6NQva57diO Eo1KmJOxCAdiwVF35CWvlXm4F1V4sNdwhNKhX5gGW3ujFFEYRX+3VhJ8aP/L3CtTEyhnrIa+O/raV 0yur2m6euQxViw7qwVhtPl92ojjY26nItZ0Th5VRbkttllbFoKXNZR1JgiJxEPEcVnfIRWJ/uip9d 6TDbQu2A==; Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XM-00HJKf-6P for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:14 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id jx11so8625307qvb.10 for ; Tue, 02 Mar 2021 07:01:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Mj/u+lI3QsG0AfvrUV/jHTPuQYjw+9cw0HbYqsdheKs=; b=swqPzUzuAnnTJJB2GB+K+3W4L+H7WEAr5B7fKqcoNpqxJCf6ac3/k3HwRvJxFQt76h g/Zk7MBBYHULmVjMDun6ZSftD/nPxKajm3eRzVHHZgg7VtMW8jffRUgqxgiuu0RV9dKz U5R0fiHsFtkp3OwYT3/hO7iTBdtx64ARXixWDEQO/DS9h42F8AwerJfgyBH0YDYUrHIA YsADDv1nhohi+P4YqppfSPPNiJjHxnY7dZt0jyfj8wjNzNROWh6Q6z75geCtykKd36am IBn2myIfZHkFpF1JXec2Lgt0mlPOKDhoSFIfuzR677xFTc9x5/SS+eK63zPcfRz2yKqZ w+hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Mj/u+lI3QsG0AfvrUV/jHTPuQYjw+9cw0HbYqsdheKs=; b=Wyvht5SGobQJJZ8bDd2EvjzIyXAgbO951SEPolCzPjwF1glxDTznN4mgm/h5auOaf0 XGBDWg3fCSaY5r8BWpjaGJf2CtNpt5xGWZNM9duCURJYM+FlcrmYZ0K2+5io3LZ3Px6l tvQiH76Bo+psghgn5EHiEKyePkhacAh/FlTFb9xVmWXRAZwGj9DI2jt8bX+J9nmxoyIZ E0JbWTkhvRmLYj4gXcc9dKebIADsK4wMFBjjD8b+ax1fRQDtfTcWnPMaogU1yLJZ8hyk NtsMbm2lCX6+gEBQDQgf8zfSiWJgfOYvz2/IDBY48bAxM717L+l+fOjdIlQ9p0OwF45b A6Ow== X-Gm-Message-State: AOAM532lOdgaxp+cHRoUNwUQ7ka0LVaz6iKC73oQDGe2RdLBxE2275H2 e73Sgq2cWmRjXGuAHxmOepAIeFhM9Fys X-Google-Smtp-Source: ABdhPJzgyoA/MqucA9uvkne0CTz79JaHIGOrbq0Qqc+8hD0/Fm3YGIiA29J7VVM7LEhkUxid/JWZUc7n8Z4m X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:ad4:4f28:: with SMTP id fc8mr17683863qvb.10.1614697249099; Tue, 02 Mar 2021 07:00:49 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:49 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-20-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 19/32] KVM: arm64: Set host stage 2 using kvm_nvhe_init_params From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150214_668397_C1F4DF67 X-CRM114-Status: GOOD ( 12.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move the registers relevant to host stage 2 enablement to kvm_nvhe_init_params to prepare the ground for enabling it in later patches. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 3 +++ arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kvm/arm.c | 5 +++++ arch/arm64/kvm/hyp/nvhe/hyp-init.S | 14 +++++++++----- arch/arm64/kvm/hyp/nvhe/switch.c | 5 +---- 5 files changed, 21 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index db20a9477870..6dce860f8bca 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -158,6 +158,9 @@ struct kvm_nvhe_init_params { unsigned long tpidr_el2; unsigned long stack_hyp_va; phys_addr_t pgd_pa; + unsigned long hcr_el2; + unsigned long vttbr; + unsigned long vtcr; }; /* Translate a kernel address @ptr into its equivalent linear mapping */ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index a36e2fc330d4..8930b42f6418 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -120,6 +120,9 @@ int main(void) DEFINE(NVHE_INIT_TPIDR_EL2, offsetof(struct kvm_nvhe_init_params, tpidr_el2)); DEFINE(NVHE_INIT_STACK_HYP_VA, offsetof(struct kvm_nvhe_init_params, stack_hyp_va)); DEFINE(NVHE_INIT_PGD_PA, offsetof(struct kvm_nvhe_init_params, pgd_pa)); + DEFINE(NVHE_INIT_HCR_EL2, offsetof(struct kvm_nvhe_init_params, hcr_el2)); + DEFINE(NVHE_INIT_VTTBR, offsetof(struct kvm_nvhe_init_params, vttbr)); + DEFINE(NVHE_INIT_VTCR, offsetof(struct kvm_nvhe_init_params, vtcr)); #endif #ifdef CONFIG_CPU_PM DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp)); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 5a97abdc3ba6..b6a818f88051 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1413,6 +1413,11 @@ static void cpu_prepare_hyp_mode(int cpu) params->stack_hyp_va = kern_hyp_va(per_cpu(kvm_arm_hyp_stack_page, cpu) + PAGE_SIZE); params->pgd_pa = kvm_mmu_get_httbr(); + if (is_protected_kvm_enabled()) + params->hcr_el2 = HCR_HOST_NVHE_PROTECTED_FLAGS; + else + params->hcr_el2 = HCR_HOST_NVHE_FLAGS; + params->vttbr = params->vtcr = 0; /* * Flush the init params from the data cache because the struct will diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index bc56ea92b812..f312672d895e 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -83,11 +83,6 @@ SYM_CODE_END(__kvm_hyp_init) * x0: struct kvm_nvhe_init_params PA */ SYM_CODE_START_LOCAL(___kvm_hyp_init) -alternative_if ARM64_KVM_PROTECTED_MODE - mov_q x1, HCR_HOST_NVHE_PROTECTED_FLAGS - msr hcr_el2, x1 -alternative_else_nop_endif - ldr x1, [x0, #NVHE_INIT_TPIDR_EL2] msr tpidr_el2, x1 @@ -97,6 +92,15 @@ alternative_else_nop_endif ldr x1, [x0, #NVHE_INIT_MAIR_EL2] msr mair_el2, x1 + ldr x1, [x0, #NVHE_INIT_HCR_EL2] + msr hcr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTTBR] + msr vttbr_el2, x1 + + ldr x1, [x0, #NVHE_INIT_VTCR] + msr vtcr_el2, x1 + ldr x1, [x0, #NVHE_INIT_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index f3d0e9eca56c..979a76cdf9fb 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -97,10 +97,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT; write_sysreg(mdcr_el2, mdcr_el2); - if (is_protected_kvm_enabled()) - write_sysreg(HCR_HOST_NVHE_PROTECTED_FLAGS, hcr_el2); - else - write_sysreg(HCR_HOST_NVHE_FLAGS, hcr_el2); + write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); write_sysreg(CPTR_EL2_DEFAULT, cptr_el2); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } From patchwork Tue Mar 2 14:59:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 798C9C433E0 for ; Wed, 3 Mar 2021 18:39:00 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 28E6864EE1 for ; Wed, 3 Mar 2021 18:38:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 28E6864EE1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SmtB+T8lhKO6yW9iJJAPJmeJS3H4aZ2x5sFMioMSxjM=; b=PEzJg5eiCZ1UGz mH720Gp5zw2/TtZ+UlcpZygks+TluMmGsgouPHsjnfOh34YwC15eYr3kyENPIKvJT3+9uxb3P5jXu YFb796+DGwKcg6cwFSNjIDiM5OOZIxGCDEfV0v0/SPpMzaSrtK4ovXDMfnGr/tjDMAvfuqmtbbSww SmJjXcn8wP/9V2sXqkH6mfjiERdbWnjs5c4Mg0hBMckXXozFu4sNHo+wMIyCtPw670nmpzAE758GJ CEs2vMM3xGOId9/FFUpKnZH/0rRRE/g/PwkbiJDxa77k7i+nqCp96hQvw6YLYA5Wrcc8AR2BKJIUa l0fEEeZiu5+tc6JAk84w==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHWM7-0062gT-OP; Wed, 03 Mar 2021 18:36:24 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT0u-005HA5-Iy for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:02:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ApvDaxD9yFwcIV9MjcPOXencqrrzEHGgE5T6Doqk/4k=; b=eqqm+9OK5+WwiAXq+1oGU4Ns+7 AJKBFf3CaefjgcsFDQLkTY02SabPeGzEwcaDUvi1j0DAyp+LVaJx0yJjN9peTSVpPM7JQCMA3qaF4 WCRvPrfyLw4nqV3P1Wy43ittAx+T7RuKPKwk2gsODMsJOb4OOQQ3ln1OZ3jvHQEAKSGxq9cYw3h/3 KGSmFnJgmUk2fKKlOjcPrwIkNTPFsforYbAmFC4QmFP6hCaNtB0T78s065Vd4p8j8OQVkC9b/EoCJ fbqq3B7uPHbeQZZMFHacryOhI6+gqgm2BjUMa+F4QQkyCRA0ZbpOsD8ppsfpc7y6uba1gPadUTU/V Cz6zM+6A==; Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XO-00HJKw-JR for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:18 +0000 Received: by mail-qv1-xf49.google.com with SMTP id s16so11377844qvw.3 for ; Tue, 02 Mar 2021 07:01:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ApvDaxD9yFwcIV9MjcPOXencqrrzEHGgE5T6Doqk/4k=; b=KCiWLWckSZhRksnbN+vPqoiKRuDDlpJlOZ2KnY4D1ROtlsSCjGjvEnWtF5KX3i5wB9 D7esZug4LltwGDArJAgy2/yrZU8133mHtcE3lKZJeHKIdqVXk7i+RWlgdOACK+Asazl+ Fe9tsBQPqxprXWKZiwE9J0q6VIAXPKhXDqCxIb1C9WiDrRwJWUQNLUeHdtXVHvSP3RNg ktgl1D8M2Wcp2D9QcID1r3N7NCM7NpWVKiIohxDMHW5DPnlz5BpwioehESslhVGAm8ko onl5BIbG4EQQI212mLhZMffJVmesufaK0Su51qyXqUE485Yq1+zOr6hMIxiLETxVjfBT y8Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ApvDaxD9yFwcIV9MjcPOXencqrrzEHGgE5T6Doqk/4k=; b=LT4lNJFC61jmzWMsDU+vbJbunbQWNCJZek1CZpS35uTtDGnxA1AFhyS5KASt9Y8iDk 321SVC2mb0JPrtaF773YbPq+XiAWi0GrnGGHQbQQSMYL8IXYDgqtRweiEIi+HOBkwfHH 0dToj2IKAnexTxx8aEVyoh1jBvcbkFPLzu2zNB5r7TBKIUvhAqrM303aYIf9HZTbRa6n j41PyuyYiNO775dmYGrm58qwVw83YrKYiwfGs7mLDsXwlzQHVWjyTwW9n/3I7bNcSQab AMMcMuwxIiU6NOl3U8E2qLcy5BfWDZJ0vf0uFo65QFYO33gmxqeaUJweRRv8r8YKlGYq fIHw== X-Gm-Message-State: AOAM533LtOjjdv5/7knmhgJmGHHFlQQmcIefYhFBVq0GX1JD8E+RFR5Q XVjHro4KPswBPZoGqyQHko5cMfk72CIt X-Google-Smtp-Source: ABdhPJwb78ZNM9QdASSl2yzPONyu+s67Xr0FkvQfsmdjTmj9GHmOdx/2Jrt01F3tfTfL7Ro1dIeICSZU6baw X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:f505:: with SMTP id j5mr4042437qvm.61.1614697251211; Tue, 02 Mar 2021 07:00:51 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:50 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-21-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 20/32] KVM: arm64: Refactor kvm_arm_setup_stage2() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150218_695857_75CCB144 X-CRM114-Status: GOOD ( 17.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to re-use some of the stage 2 setup code at EL2, factor parts of kvm_arm_setup_stage2() out into separate functions. No functional change intended. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_pgtable.h | 26 +++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 32 +++++++++++++++++++++ arch/arm64/kvm/reset.c | 42 +++------------------------- 3 files changed, 62 insertions(+), 38 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index a8255d55c168..21e0985d2e00 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -13,6 +13,16 @@ #define KVM_PGTABLE_MAX_LEVELS 4U +static inline u64 kvm_get_parange(u64 mmfr0) +{ + u64 parange = cpuid_feature_extract_unsigned_field(mmfr0, + ID_AA64MMFR0_PARANGE_SHIFT); + if (parange > ID_AA64MMFR0_PARANGE_MAX) + parange = ID_AA64MMFR0_PARANGE_MAX; + + return parange; +} + typedef u64 kvm_pte_t; /** @@ -159,6 +169,22 @@ void kvm_pgtable_hyp_destroy(struct kvm_pgtable *pgt); int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot); +/** + * kvm_get_vtcr() - Helper to construct VTCR_EL2 + * @mmfr0: Sanitized value of SYS_ID_AA64MMFR0_EL1 register. + * @mmfr1: Sanitized value of SYS_ID_AA64MMFR1_EL1 register. + * @phys_shfit: Value to set in VTCR_EL2.T0SZ. + * + * The VTCR value is common across all the physical CPUs on the system. + * We use system wide sanitised values to fill in different fields, + * except for Hardware Management of Access Flags. HA Flag is set + * unconditionally on all CPUs, as it is safe to run with or without + * the feature and the bit is RES0 on CPUs that don't support it. + * + * Return: VTCR_EL2 value + */ +u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift); + /** * kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3d79c8094cdd..296675e5600d 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -9,6 +9,7 @@ #include #include +#include #define KVM_PTE_VALID BIT(0) @@ -449,6 +450,37 @@ struct stage2_map_data { struct kvm_pgtable_mm_ops *mm_ops; }; +u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) +{ + u64 vtcr = VTCR_EL2_FLAGS; + u8 lvls; + + vtcr |= kvm_get_parange(mmfr0) << VTCR_EL2_PS_SHIFT; + vtcr |= VTCR_EL2_T0SZ(phys_shift); + /* + * Use a minimum 2 level page table to prevent splitting + * host PMD huge pages at stage2. + */ + lvls = stage2_pgtable_levels(phys_shift); + if (lvls < 2) + lvls = 2; + vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); + + /* + * Enable the Hardware Access Flag management, unconditionally + * on all CPUs. The features is RES0 on CPUs without the support + * and must be ignored by the CPUs. + */ + vtcr |= VTCR_EL2_HA; + + /* Set the vmid bits */ + vtcr |= (get_vmid_bits(mmfr1) == 16) ? + VTCR_EL2_VS_16BIT : + VTCR_EL2_VS_8BIT; + + return vtcr; +} + static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, struct stage2_map_data *data) { diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 47f3f035f3ea..6aae118c960a 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -332,19 +332,10 @@ int kvm_set_ipa_limit(void) return 0; } -/* - * Configure the VTCR_EL2 for this VM. The VTCR value is common - * across all the physical CPUs on the system. We use system wide - * sanitised values to fill in different fields, except for Hardware - * Management of Access Flags. HA Flag is set unconditionally on - * all CPUs, as it is safe to run with or without the feature and - * the bit is RES0 on CPUs that don't support it. - */ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) { - u64 vtcr = VTCR_EL2_FLAGS, mmfr0; - u32 parange, phys_shift; - u8 lvls; + u64 mmfr0, mmfr1; + u32 phys_shift; if (type & ~KVM_VM_TYPE_ARM_IPA_SIZE_MASK) return -EINVAL; @@ -359,33 +350,8 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) } mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - parange = cpuid_feature_extract_unsigned_field(mmfr0, - ID_AA64MMFR0_PARANGE_SHIFT); - if (parange > ID_AA64MMFR0_PARANGE_MAX) - parange = ID_AA64MMFR0_PARANGE_MAX; - vtcr |= parange << VTCR_EL2_PS_SHIFT; - - vtcr |= VTCR_EL2_T0SZ(phys_shift); - /* - * Use a minimum 2 level page table to prevent splitting - * host PMD huge pages at stage2. - */ - lvls = stage2_pgtable_levels(phys_shift); - if (lvls < 2) - lvls = 2; - vtcr |= VTCR_EL2_LVLS_TO_SL0(lvls); - - /* - * Enable the Hardware Access Flag management, unconditionally - * on all CPUs. The features is RES0 on CPUs without the support - * and must be ignored by the CPUs. - */ - vtcr |= VTCR_EL2_HA; + mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1); + kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); - /* Set the vmid bits */ - vtcr |= (kvm_get_vmid_bits() == 16) ? - VTCR_EL2_VS_16BIT : - VTCR_EL2_VS_8BIT; - kvm->arch.vtcr = vtcr; return 0; } From patchwork Tue Mar 2 14:59:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11BFBC433E0 for ; Wed, 3 Mar 2021 18:27:44 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13B0864EEF for ; Wed, 3 Mar 2021 18:27:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13B0864EEF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/jM+N60xCx+X5CQ8U0/QkoTPm76I6oa9sBdHtH4+4wk=; b=ftiDwuvV+zgxCH 2CoOEahsFNOS7ouly5kagUv7BK80GFn+K2MkftoS99bl5MGeHNzdQOc8fKw1sjpPBgUwoBKwtnkyP mA54NUzUBe3FEIIozUuq9tH0VZOdJqNO0Ky8A5yYxZrylH1+hgAMoKXfquuqdbipE6mQ/EoLrGmvp 6IqT+7g8vgNoNZtl7F86ADsPTFsIKluSRZBUQrsyQxdiK2WoHsEuYRRMv3u/0ypJ0n9Xrq3gvDzel 3gdwT7Iw7azYisLMwIkZDsls9nVZOFMo7u9CAVhwSTY6fQKZhgE1eG2h+tpnJ+BbY756O+n1VFw3v znPRPresxUe5Zkl+n0SA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHWBJ-0060Yr-TF; Wed, 03 Mar 2021 18:25:10 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT0K-005GyP-Hv for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:01:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=4oxp+YpplIoIB69u+tbsX2FMX4bca6h5fwiDtQ6/a1E=; b=d/RybOtIPy8Rwmy6OPt4X2RboO im3HqX5kSBlS72usv680/Ag/5ZBfVS/FAbQBacLO1fJkfw0m40bl/o9+gygnNe9Ymr3TkYRTwVPTD p86tUVkyvvhnxbzvVeM49aFC4i8BvlaCv52hBDdUBc6uK7jOKLwkNuI+VOTrziD8oHYDsJS68XQEs 1V93L6tHrzs5akd1C0nS2QAZuTQGa0Pe0315eAF+vRCikD7eSL4tWs+yy/Y1182IZvjvVywSTQXQH bJMJNKm714k+MxvGvAEzTi90N5bhxtF6PVkGxhlX8j5cPssJGSKLA+X1Z9j3Bl9Uz1Stkr8kN+uE7 VQW3Wi/w==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XN-00HJKv-AB for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:16 +0000 Received: by mail-wr1-x44a.google.com with SMTP id y5so776422wrp.2 for ; Tue, 02 Mar 2021 07:01:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=4oxp+YpplIoIB69u+tbsX2FMX4bca6h5fwiDtQ6/a1E=; b=DVwV4PEk1sNj/JtjY53/uNRLb44IFXbqnfCgK0gXSCwMPvjRNvm9Fcb1TcjqwrCfku euCCdmLYqJP39qBt+5Bew9C6jJ+Q/bGJwemVcfgXRh3nQXn8NkcDAuaezl/nhpgkNrDK f94J8Xjfqwl7DDHczcWMPaseBthcz2iTf0vYl5mo3WKmn0nIsdDOVisnf0UEOT7gqpCd Bx4inz0gG/xCu+KGf1nv2v5uR5Hk3FOUX9Hn8r4v/5ujdvnBCU7lrahHnPK/nXuRosk1 W9lQkbXQttUHawOFtS1GUul9l6p1xAhGprMNslTZTrXUK43WU14pnWI770t3Dfmxfred +t5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4oxp+YpplIoIB69u+tbsX2FMX4bca6h5fwiDtQ6/a1E=; b=AGoCCPdcePToGm1/VWgTYbfqFXprZm2KNsXJVdqhle4R7tQIw9uEOkxEaUAvArbV8y uEMd8fV07/nuPD+Q4oiNuBfdyV7yC6GZI0hNtyBsgSItzGo6WaE7LlAxQuZngWaMveB3 344gcKP9B8vOW9H7UNaVgb5L0OX2gbWK7RUrid2d1v+3clpL6ZMDbRFvmPckkMIiruJL 4QMONzCIdA9maBkFWjfrlEqOo/BiC9BN56MliM8AbkYUJrUFBA7RPEkvZoLbBT1Lkzrk OvG43gubjvdNPmMD+Fakw5HqPA5YiHKzzO+PBgyI1cvmxP1z+chMQe5zS7Wqgcpe3JIx gdWg== X-Gm-Message-State: AOAM532bX6sfuACTDLp7vk1mSEzbnMwG+4PG7A3tN0lBr4r5ux+WV7Iz i2HEBCn4nU4RCJXjDczwKWItmXURRom1 X-Google-Smtp-Source: ABdhPJwfPjt5pFKOfHuPj31XXVqR4/+bIz9+vX0UnJf2kL9ZXviOPHu8S3WCnbrQ6ZUfzWMvmubTRIE+eq90 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:b6d7:: with SMTP id g206mr4471661wmf.123.1614697253532; Tue, 02 Mar 2021 07:00:53 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:51 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-22-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 21/32] KVM: arm64: Refactor __load_guest_stage2() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150216_499141_8FE88704 X-CRM114-Status: UNSURE ( 9.84 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __load_guest_stage2() to introduce __load_stage2() which will be re-used when loading the host stage 2. Acked-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_mmu.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 6f743e20cb06..9d64fa73ee67 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -270,9 +270,9 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) * Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu, unsigned long vtcr) { - write_sysreg(kern_hyp_va(mmu->arch)->vtcr, vtcr_el2); + write_sysreg(vtcr, vtcr_el2); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* @@ -283,6 +283,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) +{ + __load_stage2(mmu, kern_hyp_va(mmu->arch)->vtcr); +} + static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) { return container_of(mmu->arch, struct kvm, arch); From patchwork Tue Mar 2 14:59:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B1C1C433DB for ; Wed, 3 Mar 2021 21:51:17 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 516C764E74 for ; Wed, 3 Mar 2021 21:51:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 516C764E74 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SF4fwtUftmLxKhRpbkBe9LgwCCfPOeZnge7iNk0kMJk=; b=ZXHQQyG+2pPGLj iHSNw0ZOOS2lXTrPGNSQjOgWWOr3h9iAXzzC/K/PbxkdZP1mr2ARQrGdj6Dw5gunwk17J800dqpi8 SZ/QuQ2TH1uCreVVHqYz2222O+DYBDdyeyu8gLGvCeX+lpRudUlQtdg4IQsF/cSp8gRrplpB+VmBm Wb5yYqofTH0GVINp8nJw7wkmFXu6Btm9jKQKuPyjejef8gRKWrbokw8hmStDlGFSwfp+BkgCTbQ7k WOsSzG8bjPk7nPE1KqYIjxQqGiIzcL0/Y7ohnXubPX8LZHZQbICbMZStV1tXLhM8V6dtrnil4fyH/ nAeIW2ylLedfSXd+Eu7Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHZLJ-006di4-Ry; Wed, 03 Mar 2021 21:47:43 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT9E-005K7Q-Vm for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:10:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VoWvRhVqzXHkQ0OsvxSGeS/VJPiNA3/g3FAmYYygOdw=; b=RHyIG7aCeC1/nkNRO05Ko9Hy0q kxDrB94oKUbRvvteGxySzDr3l18pP9r9rdlzIksXBGuvKjQuF0tVvvBAd9s0z4UsSswlO2+GdSmMg k3lw6J7jhiXOSNGjbEpZbTDaqGfIJk76LrDyA0/LIUccV/E4D3/SwpBVuE/9R/QJ2ygymw4asosiI yxolSNS5yxdFDlEOvyCUS3KKc6K1pB99sQFqbFm28XAAjRPaOXolSkRlAvg/QgYnO9grAQmZP1XXL kjMPlwMcWmSjF59Ro9wRV4rG72VVEZyUK0E+pOicxn7DeAwGfmoADBC0f8hpafGaRCK5xc6XbswMD OohbbVag==; Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XQ-00HJKz-Dx for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:19 +0000 Received: by mail-qt1-x84a.google.com with SMTP id b18so6551486qtt.6 for ; Tue, 02 Mar 2021 07:01:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=VoWvRhVqzXHkQ0OsvxSGeS/VJPiNA3/g3FAmYYygOdw=; b=AjUn7I37yjHmuqeaXLfgA7dtYSSQwwN/4mcAzk8HpWqk8XmE81BG8HC75qZhUGfls/ M0UchiZU+6VwuNndrGxdk2BjIPU+8enIfuinKdzPQa6PWDKoVD3qVJDHHGjdE6lUFUFm xROrWqmqeI6LH01tc3+3KqZwIQDZDeWu20eRjLkzQT7gxd/alLd29IAr46KfDHDLfQcB r0h8AcNcvuZw2QoF5e09oSwkTD8PSMKtQa5NUUlkyhGhG24SVC+dv1fuFYkLXn2SaIp8 jRsY6UXZUSmEu2kqopQE9vx8d5HAX6OtZIq9auAHd5qMkR6/zmg55RGtssFV8WLZh1QH eXcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=VoWvRhVqzXHkQ0OsvxSGeS/VJPiNA3/g3FAmYYygOdw=; b=Sb1fN2DPu6kSh7KrJ+ISYdt8vQJS3qY49cS6LKy/D2fqppjBog3UM0uqEh2MIc30HS e+ppzwn9RjBbSy9wKgYKZnczBwwAYwhjPXPl9G8BXBECmvzEqr1db2kIfPcGEvnOLKgm zPteTZ1K0CzWGqhopsiiqtlJZ8AiNHQXvp/0xyejdPc+miG54sffabOKN2vSSFpn1hNi tKphKOJP9NP5YH+rmdc+xKKWWr29xY1FtQh65LHkXUDm43A5q3Dy7D8XNI9FTwsFPXD+ vBwJFa7lyzDqaJCtaqWBRzEndi3jl9DbFHIOnpNCqRNsCYFxWT/MB0VXxhKh4XWvM/Sa hGwg== X-Gm-Message-State: AOAM533sp6KKETF4pn6T7tEDlhWptBUFmwuEbR5FBBJH0lzAlYiTHVrM ol0u/VpCrl+r4Q9rhgYXCHOii97miJIo X-Google-Smtp-Source: ABdhPJx8y0kZKpPj1sFhkhggFZKcZCCqwCconUOz/jXAArtDcupPU7deynqeh0MVM45/MhZGwK2ChR5h6DCc X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:ad4:5901:: with SMTP id ez1mr3999273qvb.38.1614697255713; Tue, 02 Mar 2021 07:00:55 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:52 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-23-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 22/32] KVM: arm64: Refactor __populate_fault_info() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150219_745131_7C1F17C0 X-CRM114-Status: GOOD ( 11.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor __populate_fault_info() to introduce __get_fault_info() which will be used once the host is wrapped in a stage 2. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/include/hyp/switch.h | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 6c1f51f25eb3..1f017c9851bb 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -160,19 +160,9 @@ static inline bool __translate_far_to_hpfar(u64 far, u64 *hpfar) return true; } -static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +static inline bool __get_fault_info(u64 esr, struct kvm_vcpu_fault_info *fault) { - u8 ec; - u64 esr; - u64 hpfar, far; - - esr = vcpu->arch.fault.esr_el2; - ec = ESR_ELx_EC(esr); - - if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) - return true; - - far = read_sysreg_el2(SYS_FAR); + fault->far_el2 = read_sysreg_el2(SYS_FAR); /* * The HPFAR can be invalid if the stage 2 fault did not @@ -188,14 +178,29 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) if (!(esr & ESR_ELx_S1PTW) && (cpus_have_final_cap(ARM64_WORKAROUND_834220) || (esr & ESR_ELx_FSC_TYPE) == FSC_PERM)) { - if (!__translate_far_to_hpfar(far, &hpfar)) + if (!__translate_far_to_hpfar(fault->far_el2, &fault->hpfar_el2)) return false; } else { - hpfar = read_sysreg(hpfar_el2); + fault->hpfar_el2 = read_sysreg(hpfar_el2); } - vcpu->arch.fault.far_el2 = far; - vcpu->arch.fault.hpfar_el2 = hpfar; + return true; +} + +static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) +{ + u8 ec; + u64 esr; + + esr = vcpu->arch.fault.esr_el2; + ec = ESR_ELx_EC(esr); + + if (ec != ESR_ELx_EC_DABT_LOW && ec != ESR_ELx_EC_IABT_LOW) + return true; + + if (!__get_fault_info(esr, &vcpu->arch.fault)) + return false; + return true; } From patchwork Tue Mar 2 14:59:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB62EC433E0 for ; Wed, 3 Mar 2021 21:25:45 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1E353600EF for ; Wed, 3 Mar 2021 21:25:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E353600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1mBehAhyCaM0t54IzebT5w8cTnVUvN3WeoI+W3D5Vao=; b=SYafuemuHk8wJJ oeM7YXWajL/KrAWXIk0VGyOr5c/VkIM/hSbQD64GWwXRxUsDzSHS+2l8CPwRPPuHLU7qAmyionkO2 RHIAduuDwxX/GVhQYCwJ1lXbKOitPeT8viK5uxQC52NFsPRgtc2N+ZptdrrhqfSl5YxKYO5PYZ4qY RydzPWUFHDvc/A5emBqNcUZqx5ERzhs01rGyaIIcW3k5tSjc4IdMiS42/o3o7bFltojFVNSWYZ4r5 AH4V7wnDtTESnQdLsBa4nn0O7S0F/shc7HwOuWo1IJaA9EMMdQNC/gf+rnv8zsYGQpRTC960sf04R D7RJERpuhP95bafPMP/A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYsm-006ZTK-AR; Wed, 03 Mar 2021 21:18:13 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT84-005JgI-Ol for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:09:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=eue23Q78bOa6+/qbv0+6w883I/AngR/tMntsoJXpnuQ=; b=tFTY45oq55DQ8U2Z1f1AQk/U5D C/d+0OZVfgSkqgq1UeHQ2Qg4rIRKPnz1La5ZBh+Shra9YiuVm5ZJ2g0QD1aNPvRwRYOa2+zsnZC5m WR8zuYZSNZKwu2T4q3vl8vAQ3Ta+BAi7C2TS7WZ1OXgas0wOCZxMtHgKwG3PjgjV25OQULEgUrsJ7 3S5ZnYBwpOimcgI65qhpUH8NEuYY/o5kBjCEIS3c6+8gy7kb8Zqm4VbG84aeG7+6rlENmZTEOpc6v fjGBwMpBE/KS8g6H1i39PLPiWEATlXO/DDQCtG1xgogobmZGS0fjUzWYTRAF5sLQKaGNbtadR8pJm xBCAsK1w==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XS-00HJLH-HS for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:21 +0000 Received: by mail-wr1-x449.google.com with SMTP id h30so11249190wrh.10 for ; Tue, 02 Mar 2021 07:02:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=eue23Q78bOa6+/qbv0+6w883I/AngR/tMntsoJXpnuQ=; b=nXDl1eBiQI2Aja03chUT9gTykyTfm89UeBlE5QXB1z3z62rESZY1NMmtIQrrBv8WOF Ass0zOOFiuM+77TewQc5zQ9f+75XlVxMT9kaHRSFP5pdy5PGn8GGHcVrFZ/Dn8X0pSDl zAxJ+7yMDdHK5Hubnm4RtSb+mytr0rfVYMT0DDBLLEV2kdB3bjFW48XozEWVs1ckcr3N dDr5smJaj/nTDAeNxI35YfwpklQwX+z7ZLIMRdSwDbzl0Vce1qAdeTeoJb2lQi68my3Y yyWI94ahVEOhCs2D1coNOmWYjpek8J80PGFS3oGu7r+aS3BF/QiUyFoyeeaOb2T+gs2+ HY0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eue23Q78bOa6+/qbv0+6w883I/AngR/tMntsoJXpnuQ=; b=U4WZsPRpy428nEThGl+Z+pgTEs5fOu2lGay+Ch+Cz9zWr/3K1RkUFmC2/9SSDoHrSf bbYWaTl/reSJ0kIQGT3LM0pQKJkIk277a10d+7skkcPjtFViRrSK1E4EQwTEqiXTGIYN ++DnXCrXq4aEWLZjfb+bx6BM0FpqvmqbuFgfea2aUltzNulooRdrxT0syUTXAZCBmRYk QLe3kb0n1Twe82Kp+oQNgPtMJ1QJSemK4YC9UyPRtH9DCZoP/+5HDXkQyV8AGB7iDXBv Q3FEX6Ej/3vaPwEy1zdrbvSbbuzpfguyu4ZH7juFo+nFZ/9z9Ez+CohO0uHd1m5vecZv 4c7g== X-Gm-Message-State: AOAM531DjZcVALxd8T++hpdtQtsGTACa7RX+kVdpk3Df42LYGcKqjhmv 1GE9p8ZzcVJsTAvgoB216zWM1CxMX4Pt X-Google-Smtp-Source: ABdhPJxEqIQZb2BYYx4oJbXqcI8pQfBowRyHhxZCpcsx84Q/DAuukPbVQiX0S9b4CaaJzjpiugoRR377rmIT X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:b6d7:: with SMTP id g206mr4472055wmf.123.1614697257976; Tue, 02 Mar 2021 07:00:57 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:53 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-24-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 23/32] KVM: arm64: Make memcache anonymous in pgtable allocator From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150221_065467_BE43A66D X-CRM114-Status: GOOD ( 13.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current stage2 page-table allocator uses a memcache to get pre-allocated pages when it needs any. To allow re-using this code at EL2 which uses a concept of memory pools, make the memcache argument of kvm_pgtable_stage2_map() anonymous, and let the mm_ops zalloc_page() callbacks use it the way they need to. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_pgtable.h | 6 +++--- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 21e0985d2e00..9935dbae2cc1 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -213,8 +213,8 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * @size: Size of the mapping. * @phys: Physical address of the memory to map. * @prot: Permissions and attributes for the mapping. - * @mc: Cache of pre-allocated GFP_PGTABLE_USER memory from which to - * allocate page-table pages. + * @mc: Cache of pre-allocated and zeroed memory from which to allocate + * page-table pages. * * The offset of @addr within a page is ignored, @size is rounded-up to * the next page boundary and @phys is rounded-down to the previous page @@ -236,7 +236,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); */ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc); + void *mc); /** * kvm_pgtable_stage2_unmap() - Remove a mapping from a guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 296675e5600d..bdd6e3d4eeb6 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -445,7 +445,7 @@ struct stage2_map_data { kvm_pte_t *anchor; struct kvm_s2_mmu *mmu; - struct kvm_mmu_memory_cache *memcache; + void *memcache; struct kvm_pgtable_mm_ops *mm_ops; }; @@ -669,7 +669,7 @@ static int stage2_map_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, enum kvm_pgtable_prot prot, - struct kvm_mmu_memory_cache *mc) + void *mc) { int ret; struct stage2_map_data map_data = { From patchwork Tue Mar 2 14:59:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1AF8C433DB for ; Wed, 3 Mar 2021 15:37:51 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 79A7464EBD for ; Wed, 3 Mar 2021 15:37:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 79A7464EBD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=89RjxyvnkF/UhMl19r1m3pQFlY4R8n+F+b3F6c1uDwI=; b=GkzxC4cGnmVp5Q rjUFSrDTRKiCFdNfVfhbMlbh284cP2tBuw6T0XbdR2isuqUCdJcwN/tpPBiiG0/qJiTqQDQtv4y0r zw/7WWZyNDyuDwxjvCfWfTWKT8EpSOR2p+zeSQeCzZfupN7uNKpbUTIowjYkYpGe/pFrsCrqrFu27 yHi8mo6PjhGBh2aw0S2si1KENn28L+NVS0famvlnFAbiHTftvcAU63YdkoaF8NBXh5iDJMf87JALr fBTklTUpj4d2RJ3os9lvZDAawAKPobjkffnZ5EoYroV+aOtFrfodslqH5xaSzYm1FWqUNbFLfJCeo 1Mhv27VxU2r00eRp/PDg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHTXA-005PUm-LS; Wed, 03 Mar 2021 15:35:32 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSUX-0058BQ-VJ for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:28:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ul+o/tKmUBDOvYf9j24b0RRB4JIOz9FDoTsxVSxSXxM=; b=T0r4g2U+5VlFj5oR8Fxn5K4+XY kWLKgxt/b9rubA0Cq9utFhfw7eodgKwDlxNQWDwMC1EPsXDKtoxEQOjODXWjC8uL8DBZyhZ+X//+2 qui1g0hkwO9jeCV0QpZQQSqCRK0Npxmlw/LmG6OE2BT0YYuxuDDtIKDOhmsLZdeQBtCOa768y7Pyl HUfNC8sAjOxbdZsRTIvSGuovHG7VhtNkbhNok9AWKOy8Gv2novP5dCTY/6y4nNKpBmIqi7UXLQPhy Tu5XhYASZclS7Cj23EXGMSgRRaSMP9mr7ylg495iSiqwlDAXWQ3Xlji+GeQmiQ0refQHlLkq2/Z6L YhJVzFiw==; Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XT-00HJLY-Fr for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:22 +0000 Received: by mail-qt1-x849.google.com with SMTP id h13so13305990qti.21 for ; Tue, 02 Mar 2021 07:02:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ul+o/tKmUBDOvYf9j24b0RRB4JIOz9FDoTsxVSxSXxM=; b=pW+u784h5cAco+WjZOeMEiOygQj1G34XbUjvs4ZXhx71GX9qwAHjt8XzF6oHZuDXwa wgH83z4PpOrQySx6tyjngu/mMG9opxnaGjMuW8fPgRw9ksCt32BlXarMT6Il7ZiaqZpn 7mUiXkToZK1hphH6grtrtlOBQCywTGirf+rTHNtbVoZP+um/QY9vdrNEQNwSj0MNJiS+ MIfwIQ4+QJGxoFsu7TAUOUcWNUeh+1aslU2OU1gnPefp+jW4YYjwrcvgJL0BBeFgLTJt L9uyxaiydQT4m4S48zXW72AmDfQx9xnJu2HB5JldL4/g248HlHOYz2ya1frny7PUEufs S6Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ul+o/tKmUBDOvYf9j24b0RRB4JIOz9FDoTsxVSxSXxM=; b=iTncnmOp/zGm8rakFUhmiEdoquBZvsgJKdSZUb8nB4kRf+zm0bGWIGE//f39DwqyQi r66/a6se3snRMEcSRzm2M9DuT4I9mPXu5XxEAXKyiREnuRlV21cJG/jN1s9K7Jm4bU1F Ex7l+NCKLeckO3zOncUYovJPSXA1s99riV61IbIjFfKpAZZBRtQoDg1SyJplhgPmkHI6 ftUsBC6tXersL2PFd9Sh676NCk2kjcCyTRj0bu64CUXUZ+Pwd7vOJKOxyuJn4zFnovEC KYNxTkSh9Ss0Et7Tg37SwudkFlTQPKrHUaFlpXC1yHnGR9kmGvg5l4sGl4xUKttkTgRK 4rgw== X-Gm-Message-State: AOAM5335Lb1bBkfZSsZhcI94y/Pei5H0kXfOl22L9963gmmc/oRUB4Tt aov1A+oBesbh6fKOM3O75f55w7jja4+u X-Google-Smtp-Source: ABdhPJximV251/TA30yUjyxYuWzjKdxc4ib1kbnL5JjOJcV3XkuAkOyM4ZeRll9baZr9V40/GNB29ZHFcFmT X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:b59f:: with SMTP id g31mr19967030qve.28.1614697260716; Tue, 02 Mar 2021 07:01:00 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:54 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-25-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 24/32] KVM: arm64: Reserve memory for host stage 2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150222_237369_0EF559C6 X-CRM114-Status: GOOD ( 14.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extend the memory pool allocated for the hypervisor to include enough pages to map all of memory at page granularity for the host stage 2. While at it, also reserve some memory for device mappings. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/include/nvhe/mm.h | 23 ++++++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 12 ++++++++++++ arch/arm64/kvm/hyp/reserved_mem.c | 2 ++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index ac0f7fcffd08..411a35db949c 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -53,7 +53,7 @@ static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages) return total; } -static inline unsigned long hyp_s1_pgtable_pages(void) +static inline unsigned long __hyp_pgtable_total_pages(void) { unsigned long res = 0, i; @@ -63,9 +63,30 @@ static inline unsigned long hyp_s1_pgtable_pages(void) res += __hyp_pgtable_max_pages(reg->size >> PAGE_SHIFT); } + return res; +} + +static inline unsigned long hyp_s1_pgtable_pages(void) +{ + unsigned long res; + + res = __hyp_pgtable_total_pages(); + /* Allow 1 GiB for private mappings */ res += __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); return res; } + +static inline unsigned long host_s2_mem_pgtable_pages(void) +{ + return __hyp_pgtable_total_pages() + 16; +} + +static inline unsigned long host_s2_dev_pgtable_pages(void) +{ + /* Allow 1 GiB for private mappings */ + return __hyp_pgtable_max_pages(SZ_1G >> PAGE_SHIFT); +} + #endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 178ec06f2b49..7e923b25271c 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -24,6 +24,8 @@ unsigned long hyp_nr_cpus; static void *vmemmap_base; static void *hyp_pgt_base; +static void *host_s2_mem_pgt_base; +static void *host_s2_dev_pgt_base; static int divide_memory_pool(void *virt, unsigned long size) { @@ -42,6 +44,16 @@ static int divide_memory_pool(void *virt, unsigned long size) if (!hyp_pgt_base) return -ENOMEM; + nr_pages = host_s2_mem_pgtable_pages(); + host_s2_mem_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_mem_pgt_base) + return -ENOMEM; + + nr_pages = host_s2_dev_pgtable_pages(); + host_s2_dev_pgt_base = hyp_early_alloc_contig(nr_pages); + if (!host_s2_dev_pgt_base) + return -ENOMEM; + return 0; } diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index 9bc6a6d27904..fd42705a3c26 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -52,6 +52,8 @@ void __init kvm_hyp_reserve(void) } hyp_mem_pages += hyp_s1_pgtable_pages(); + hyp_mem_pages += host_s2_mem_pgtable_pages(); + hyp_mem_pages += host_s2_dev_pgtable_pages(); /* * The hyp_vmemmap needs to be backed by pages, but these pages From patchwork Tue Mar 2 14:59:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81D71C433DB for ; Wed, 3 Mar 2021 19:52:35 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F2D5864EBA for ; Wed, 3 Mar 2021 19:52:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2D5864EBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HIsgB0QaNX35kdh9UYNCfS2ACfudaPbPyL5eQHEGyKU=; b=gf4bZnIVXdzp1d 3+oqAJaD18gOwJulR0alHWH0hXbIAO74p5oSb3ZlYvzSru48Xm7kAtDfu+YWPZyy3OKPO44j/5PRQ 4ldM6uF9HYyuGGG2fUMwULB2ejpYOX0jUY2F1TIXa9Zt+L/XzqZXw5S2oLzLnb+rr9fawxqcRapyK 2kAUWRPfyL4MJdfC+2+CYBPjWlaIKd2St6EhRbasx1ZphJBRaNhhCpl7AGJRVwH9FrIqhC/u4cT9m C9xFQ9QV7FMpGtlMnDIa7V7Fki9ttaUQdKAaTJAG0d5v6a8VTudLkkQpt5q4bY6qGZUciGTYkvlVq Sh7ITCbTslHpEWs3+Bzg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHXVM-006IDI-2i; Wed, 03 Mar 2021 19:49:56 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT4R-005INy-KT for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:05:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=0DGi8WZ41csXemjNsmACrndajQddZ4pCIIm4i2MnuBE=; b=i8Ib93cvwz/FA54q0mN+dP6GAk WS5mkB6GJ/WpKoPr68ePPDvJDi4NA80HNRgAe5Gm2sqGSQDIDkb+VtwoQhro7nE//RVrcEmbh9g6/ ae8puTlkXTKCJsBdJXQ/zJELP6ROWRv5M/M/YgzjBAkYOoMzYJpgc97RUsz3OsorqW5+tRIl4obam nP6l0h186WIvRSkQ6S6WAV7FXwt7PoN/ulMd1sbV/eX9Kwlxn7bgPcaTElUwgEwmvcwLiht08BkC6 sR50VbSqdPZwY6mYW8NgGaNHLzXoAXPmqfmS1391/gCb5J5DbhwgoJkbUm1/Kuc63QcHfVvoh+tCS X/WYP2Nw==; Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XU-00HJLZ-Uf for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:25 +0000 Received: by mail-qt1-x84a.google.com with SMTP id a41so13354306qtk.0 for ; Tue, 02 Mar 2021 07:02:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=0DGi8WZ41csXemjNsmACrndajQddZ4pCIIm4i2MnuBE=; b=NE6Q1GAgjfYI2WSnlZK6ccjnYEU6vpTfApOHpsJa8pc8twhDpFxuOBo32r89W8t2Kc YIB2XzSRTuyKBC2SBUZYIpJ7X4Lwzup0jJUpmkpiEcqkF3y7b0xBe/fYkbHGMCWJVaUl +pZoQqqiNkX+eDaJA/FSLnqVZb4bFe+3fBD3LjgG7dXQpUDq1BampNoAf4Yqpv6jDuu2 qsKdLOcxfPInMZ+6hngaTT3gdvCEg2NqfDhIB467NAn1rkMN9qG1UhjNDSuKBp35gTvX Z0FQb+JQ83j4h0wmWUQ5CY9BbZ2yBx+w4CwV0O68SlSrz6GTdXVcdwtswSt+dQHcseV+ WnsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0DGi8WZ41csXemjNsmACrndajQddZ4pCIIm4i2MnuBE=; b=iQ7kfO2G9xC7kugTrGtSSJOdYrlZrVJdkzEHS+zklbwcNNAdK1rqQBPnszGjcOkUHr 29SFq9FUiuNMmkVSLAwbOUuxs/XqZUj6OJbdTxqbZfUS6VRXcthPuCogmbtROYkA8sVX 34mFWeugFD0XpXseKZEzn00+nU3kKG28qlXX9+8wQfFtk21eBgQ5EGlSCyaV3KH+X/SW ILOXke5kyC4LvL3HK0lUhv5wggGd4/P0FaM0mpv3Jlwp3c6Nh+dZBvLjc4YbyyXtDUsW 2KaYBvI4pgCI2SfCtxsr2KO6+H5ySJV/QA7DXjIzB1gZZsjbno7hZ4pc3vbUEoAo7RtX yLuA== X-Gm-Message-State: AOAM53362B6buxoEsORPW4taqETeZTcWz9mcV5yVfbIoKg/AeTzThVDe iyhpRcYlPn7QbcCDZPNTmPmksgzu0Xce X-Google-Smtp-Source: ABdhPJxW29HrCwNVsnKe95k20fj29Sf36Q0W3cKnVBDqrtxU2VNcdgoeoSV78CPoA72FRV1PB5O9++BrTYke X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:ed45:: with SMTP id v5mr19863045qvq.13.1614697263353; Tue, 02 Mar 2021 07:01:03 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:55 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-26-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 25/32] KVM: arm64: Sort the hypervisor memblocks From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150225_013050_BF15FCD6 X-CRM114-Status: GOOD ( 11.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will soon need to check if a Physical Address belongs to a memblock at EL2, so make sure to sort them so this can be done efficiently. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/hyp/reserved_mem.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/kvm/hyp/reserved_mem.c b/arch/arm64/kvm/hyp/reserved_mem.c index fd42705a3c26..83ca23ac259b 100644 --- a/arch/arm64/kvm/hyp/reserved_mem.c +++ b/arch/arm64/kvm/hyp/reserved_mem.c @@ -6,6 +6,7 @@ #include #include +#include #include @@ -18,6 +19,23 @@ static unsigned int *hyp_memblock_nr_ptr = &kvm_nvhe_sym(hyp_memblock_nr); phys_addr_t hyp_mem_base; phys_addr_t hyp_mem_size; +static int cmp_hyp_memblock(const void *p1, const void *p2) +{ + const struct memblock_region *r1 = p1; + const struct memblock_region *r2 = p2; + + return r1->base < r2->base ? -1 : (r1->base > r2->base); +} + +static void __init sort_memblock_regions(void) +{ + sort(hyp_memory, + *hyp_memblock_nr_ptr, + sizeof(struct memblock_region), + cmp_hyp_memblock, + NULL); +} + static int __init register_memblock_regions(void) { struct memblock_region *reg; @@ -29,6 +47,7 @@ static int __init register_memblock_regions(void) hyp_memory[*hyp_memblock_nr_ptr] = *reg; (*hyp_memblock_nr_ptr)++; } + sort_memblock_regions(); return 0; } From patchwork Tue Mar 2 14:59:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4FBEC433DB for ; Wed, 3 Mar 2021 16:08:44 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DB0BA64EDC for ; Wed, 3 Mar 2021 16:08:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB0BA64EDC Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kWpPA8mNOQZhumyjDgQ3juUflnh779v7B8cYS0RJwk8=; b=bn7iYSRZJUd9u7 h9UFrcSfHYhvUhSE0GvDkIv99DvHdBDRw8Z81s+dJ4yi2SJtslk9oaudc2aZEluFKaR0zEJsxZdlH i4Ble/VlrcYy2kJmWRLAI3B27kf8nF5HRZJJoEpKfNv8Yqz9LnF2y4BcYQNbGuAriZ0v7k2VfTiks vu5QER8I828/exMDSls/O3J6OimUnwsVqz/Pj3GKFROyXmXoQsMcf9w4VT4JqhuQJvXU/AyCHcfL1 PC5AB3dnL331CT9MQ25om479ilZhnKWmgOBj3DP7LqU4SlDEAmUYYJOoFP1kb9jvfz2Av0mZS2OTM v0InRdkHUN1HUF28F6zw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHU1B-005Wsa-7L; Wed, 03 Mar 2021 16:06:33 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSoY-005Dd3-OH for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=fnu4kZLmantAPLgmfgZ7PZzVHqXbW8DjkSliDC/e8I8=; b=ah1HJcdACzBCBM2MffK3Nco3se uPqoM6vYhTz4DHaLCwzyii20db6Yj1wBVWVZLeEmd8dm3DedF8vTMnRfha6hZRVz2FzSSbPCXiyA/ fEYbeveBpFMUKonxAT+MWoDCpmjqTfvtguVF6Xig7Xh80mif53pbNnWWAPGsyAvHQGhYI5QkZiNai hh0sVhQtx3sNE0rptHeOZ3FCq42mIOgKI+5k6G3MWrl8i9s4ErZy3wY4DMrDnO1hx4jYGKx0278Jb zjq5eY5sEG3aDDmc97mq3OBrnR16OUcTwvTsA0J7qZGlCU/6+SMWH21uFIRiy8IKbrZhVBzzQgNMc ZiH/TyGw==; Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XX-00HJLu-Cb for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:26 +0000 Received: by mail-qk1-x749.google.com with SMTP id b78so13585499qkg.13 for ; Tue, 02 Mar 2021 07:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=fnu4kZLmantAPLgmfgZ7PZzVHqXbW8DjkSliDC/e8I8=; b=uUNW1RSwz/PPPl1Ua1IuQ64hOM0R+lTodrWU3UbAefyenAEai8RA0+im6R8w77uKKJ g4DVj/KqEe1TwMkcz/Ygt9jjIn5f38pgXuncyWnVA/+aa/N++uGeKO34r+o678oP9Mq8 67RAjq0NNwnf+jsn3QGIHhc250LRidVe1EfBP+5OIoO/7rtNLutaZrtEM1c24NPrdROO AFGrLf0EhJQzjScHxY2yPE6vMV/MjHb4X61lyzoWiVZ4RX2UG8RcNll2OE8OrY1N1eKg 6/SG5RQNSa0TOnAV1pRMMAOGJJqvxbtGjwYBUsH85FOGQKa/2mvXOYVgOo4Aa+FxfQOa yteA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fnu4kZLmantAPLgmfgZ7PZzVHqXbW8DjkSliDC/e8I8=; b=tvvUJjAwUc5ewED5sVBrLP7djOJZtfw+PDz7PYTVqojOIGmsIpqE9VVNdMAXPhsYQ8 mz1tbGleq90Kd7CFTGUsc2NtqHqLUgVt8ZdKpCzQTsfp/AaWhD1OFgb1RbDErAmdxlJA KiLEZrcerNmJ5KtX11o3L1mTwjrYx4ekaXf5MJjiLJnH5L9NGF8mY0lqhRWLSf4WUQ++ v9YzalxRz90Ti1KU7pDhFsgtqm3UyZcS7k/CocYqnoQa+4qa+yKjI16PWEHsMPg5e5Ck f68nAvCsQvCQPqbpgEUYKKk2ibXQmOR2b0tQMzaDLUPlQqfi3rAd7YisPdzBwlCX6FM2 6IyA== X-Gm-Message-State: AOAM533/DWTdwS8wdjUZ+cWolKS72k1nItpYklpSrZzP9wYWHfEbdqRO s2f/pf0MUOjoeEjwqXFHAyo4AWrFz6yT X-Google-Smtp-Source: ABdhPJw8Dq6/V08jN1owoo4yABfxU5wp1FwJVtfeIaUa56rS2EjFaWfxxNpGQxARvmsP7DWrgv0nk/HKhje6 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:c78c:: with SMTP id k12mr3964748qvj.47.1614697265404; Tue, 02 Mar 2021 07:01:05 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:56 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-27-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 26/32] KVM: arm64: Introduce PROT_NONE mappings for stage 2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150226_026567_7F10C96B X-CRM114-Status: GOOD ( 15.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Once we start unmapping portions of memory from the host stage 2 (such as e.g. the hypervisor memory sections, or pages that belong to protected guests), we will need a way to track page ownership. And given that all mappings in the host stage 2 will be identity-mapped, we can use the host stage 2 page-table itself as a simplistic rmap. As a first step towards this, introduce a new protection attribute in the stage 2 page table code, called KVM_PGTABLE_PROT_NONE, which allows to annotate portions of the IPA space as inaccessible. For simplicity, PROT_NONE mappings are created as invalid mappings with a software bit set. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/kvm/hyp/pgtable.c | 26 ++++++++++++++++++++++++-- 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 9935dbae2cc1..c9f6ed76e0ad 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -80,6 +80,7 @@ struct kvm_pgtable { * @KVM_PGTABLE_PROT_W: Write permission. * @KVM_PGTABLE_PROT_R: Read permission. * @KVM_PGTABLE_PROT_DEVICE: Device attributes. + * @KVM_PGTABLE_PROT_NONE: No permission. */ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_X = BIT(0), @@ -87,6 +88,7 @@ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_R = BIT(2), KVM_PGTABLE_PROT_DEVICE = BIT(3), + KVM_PGTABLE_PROT_NONE = BIT(4), }; #define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index bdd6e3d4eeb6..8e7059fcfd40 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -48,6 +48,8 @@ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) +#define KVM_PTE_LEAF_SW_BIT_PROT_NONE BIT(55) + struct kvm_pgtable_walk_data { struct kvm_pgtable *pgt; struct kvm_pgtable_walker *walker; @@ -120,6 +122,16 @@ static bool kvm_pte_valid(kvm_pte_t pte) return pte & KVM_PTE_VALID; } +static bool kvm_pte_prot_none(kvm_pte_t pte) +{ + return pte & KVM_PTE_LEAF_SW_BIT_PROT_NONE; +} + +static inline bool stage2_is_permanent_mapping(kvm_pte_t pte) +{ + return kvm_pte_prot_none(pte); +} + static bool kvm_pte_table(kvm_pte_t pte, u32 level) { if (level == KVM_PGTABLE_MAX_LEVELS - 1) @@ -182,7 +194,8 @@ static kvm_pte_t kvm_init_valid_leaf_pte(u64 pa, kvm_pte_t attr, u32 level) pte |= attr & (KVM_PTE_LEAF_ATTR_LO | KVM_PTE_LEAF_ATTR_HI); pte |= FIELD_PREP(KVM_PTE_TYPE, type); - pte |= KVM_PTE_VALID; + if (!kvm_pte_prot_none(pte)) + pte |= KVM_PTE_VALID; return pte; } @@ -317,7 +330,7 @@ static int hyp_map_set_prot_attr(enum kvm_pgtable_prot prot, u32 ap = (prot & KVM_PGTABLE_PROT_W) ? KVM_PTE_LEAF_ATTR_LO_S1_AP_RW : KVM_PTE_LEAF_ATTR_LO_S1_AP_RO; - if (!(prot & KVM_PGTABLE_PROT_R)) + if (!(prot & KVM_PGTABLE_PROT_R) || (prot & KVM_PGTABLE_PROT_NONE)) return -EINVAL; if (prot & KVM_PGTABLE_PROT_X) { @@ -489,6 +502,13 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, PAGE_S2_MEMATTR(NORMAL); u32 sh = KVM_PTE_LEAF_ATTR_LO_S2_SH_IS; + if (prot & KVM_PGTABLE_PROT_NONE) { + if (prot != KVM_PGTABLE_PROT_NONE) + return -EINVAL; + attr |= KVM_PTE_LEAF_SW_BIT_PROT_NONE; + goto out; + } + if (!(prot & KVM_PGTABLE_PROT_X)) attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN; else if (device) @@ -502,6 +522,8 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF; + +out: data->attr = attr; return 0; } From patchwork Tue Mar 2 14:59:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CAD3C433E0 for ; Wed, 3 Mar 2021 16:32:23 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9BDAA64ED0 for ; Wed, 3 Mar 2021 16:32:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BDAA64ED0 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=p7F2wdMGF53/6hN6nN311Ti6Ikjqtb8hTjV4TRQButU=; b=RT3pDd6ivPktm4 SAyNQbyjGftwP3JWdzq9UGBssnQp+TEfKuAOs1QCp85SfZJbSDTyd26ynNIcDErhbya643R4GOVnO iLOMdS7EUFbLHN36A57olAlw4jieEnId8BxDFXpyfi0cBt5TicXsw0VItiaOmCPlKOftIyb76YBJZ HQeNg0kxwWmWN4HLpMydsES3FwxV6HNot/0ppsljHU+pQZbln9sdY7jnTaFMg2pZf7w9OeKWya7x+ 2yOjTCvlEgeQjOzOM60+G8enx1FGCfLXeuJNWZBkIFk5lTAgAYqlp7V4bW05yTTVqiTX7pAfhIyN/ fCZNq4q2FvBlB9laQe4A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHUO0-005bpU-Jj; Wed, 03 Mar 2021 16:30:09 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSpm-005E2Y-SQ for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:50:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=MUiu5cL4XJy7bcbjb67w8AVccQ6swyc7cTqk7H/pito=; b=Sey22SUnQTuQjYACNqdlwG0guQ Hvh1+zq7C/4kbNCXUdf1nc52nimkk0MFFNkMtbpw8dkyKxDb+683YFQNeUvm/WFjn27PfoZheDaRN XPd6iU+aPeUFhm3gvRu1MWy6Uo50mCwzkrmPEoYVFydwqofiz/E7pEbyrvKrNoM6d7lLzZ3hyBFkT ORNjYti9lifMcegGhosO4/+T+EyV8AVSFtldlmu/vlZAc765iwPPjdviUNazyKJyZniTImYZscQar BKY7Ys6wavy2piYTeTf/H0MhfCHykkfcAHTbNSyLECjYiffQ68rhb3sHrpbHJM6C0hYdqyLlfzfRS jJ+kemXQ==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XZ-00HJNF-0I for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:28 +0000 Received: by mail-wm1-x349.google.com with SMTP id b62so1313427wmc.5 for ; Tue, 02 Mar 2021 07:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=MUiu5cL4XJy7bcbjb67w8AVccQ6swyc7cTqk7H/pito=; b=V2tQz8JIPctM0p8zJgSN56SE2f+sjG4GB6ZR55LMQiswnxYtR/ePVhbwO/o8W66G0b PbHVynwDMo4m5hQTkVbwHPUN6c0PkFKQE/sf5DH5nMyM0tggxmbD3FPQjbi6a0POeCz+ jh+2Nsxcsol/OZxoAVv6MqFZjCqNUXiv4O8MXjSVu9i+FDKGBgulOa9tkVjyrYK5Z9Bd Nq67f+OAletST31RgOjWXRR0vHMxg7CzWFcXeOoLWmi9Q4vWw9Jmfj0fTHu7FSTjC0o5 F2L8XjgbG6jShJL/W0Af7NMKKMKboklNsvT6OliD+fo6afqsvX1EexCiNHhJOW3b0KIF 4qmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MUiu5cL4XJy7bcbjb67w8AVccQ6swyc7cTqk7H/pito=; b=pER5O5UpDBWt2sDGgDe4+rtBqodbnynBhyDBgel6qXxtYIw/78kYWsofYz62RY4tri avXQGlrjPEqIE4eTDc4yIjGiRLf11Tr/3unJdlraeqZGmTHCqtk9gl3mpxwlnL9gALuZ kpi2DNqZva3Gy6HwtQK+KK71IbbFFmw2d6feyLiN3GjucEgTXyUpJ47BHoHEB3SSqoDS EnAV4Ij94AwA/6PDnarnJy4tzvYOQ9edVFbD+Hmn61Znjp3fAsIvf7+DhslUUWAao3+O NUMdiBBEl1H4YR6IaQD6DqSpuY2GATlFf7sQ9Rk+GmeKJFECz6lZO4+DXahfjgtfNfJZ MSYQ== X-Gm-Message-State: AOAM533cPd9pbrUY1OH396fy4fCJTD7fz3a3SBBPSo2sUkvsuZL8kYj1 4JuQ2BBWn5leuFYvMmz+CB08hbC+/LwX X-Google-Smtp-Source: ABdhPJz9F9KFBBfsTkBPKASwdRvs8WkwjMfKCJcm0vk7lvPEyGjjf4UWYBSGhgGbD6PbV21B/moA66tRIrG2 X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a7b:c18e:: with SMTP id y14mr495903wmi.1.1614697267665; Tue, 02 Mar 2021 07:01:07 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:57 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-28-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 27/32] KVM: arm64: Refactor stage2_map_set_prot_attr() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150228_265724_7E3FB892 X-CRM114-Status: GOOD ( 11.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to ease its re-use in other code paths, refactor stage2_map_set_prot_attr() to not depend on a stage2_map_data struct. No functional change intended. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 8e7059fcfd40..8aa01a9e2603 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -494,8 +494,7 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) return vtcr; } -static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, - struct stage2_map_data *data) +static kvm_pte_t stage2_get_prot_attr(enum kvm_pgtable_prot prot) { bool device = prot & KVM_PGTABLE_PROT_DEVICE; kvm_pte_t attr = device ? PAGE_S2_MEMATTR(DEVICE_nGnRE) : @@ -504,15 +503,15 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, if (prot & KVM_PGTABLE_PROT_NONE) { if (prot != KVM_PGTABLE_PROT_NONE) - return -EINVAL; + return 0; attr |= KVM_PTE_LEAF_SW_BIT_PROT_NONE; - goto out; + return attr; } if (!(prot & KVM_PGTABLE_PROT_X)) attr |= KVM_PTE_LEAF_ATTR_HI_S2_XN; else if (device) - return -EINVAL; + return 0; if (prot & KVM_PGTABLE_PROT_R) attr |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; @@ -523,9 +522,7 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF; -out: - data->attr = attr; - return 0; + return attr; } static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, @@ -708,9 +705,9 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .arg = &map_data, }; - ret = stage2_map_set_prot_attr(prot, &map_data); - if (ret) - return ret; + map_data.attr = stage2_get_prot_attr(prot); + if (!map_data.attr) + return -EINVAL; ret = kvm_pgtable_walk(pgt, addr, size, &walker); dsb(ishst); From patchwork Tue Mar 2 14:59:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 905BEC433DB for ; Wed, 3 Mar 2021 16:07:23 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBD3564ED0 for ; Wed, 3 Mar 2021 16:07:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBD3564ED0 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=p23b1L+wJ3fJWmdEb14vK+8yRst06AORJ/Vth8VJpDg=; b=iiwVRryp9xo13z oaLBnGN8eSM4bAW3DDQHJwv7gKeDqr466FvnkUGbaoYJLhO66voy5tl6jzvqcEkuxZHjWuyxvqt96 e/7tHEAvUNoNxRqzf3nm9V3IBY2cP1w3cV6Kks7ebX1oQZpmvPqjhK22uZIY/f4SoUisYaiNV0GjQ 0vXcieyjd03IkNnUyGLKHuHPJdasZeFvCFCTOpZCdXCpwU69W8kQKOt9rZ/u5T1R7o733JgrtYo96 oV6t7b8jJ/0LTA2hc2PAOzSCsE7+kFUDgagmW2ALcz/CZetLcOvpcx3vRw5UZsoQbv6N+vHaMn7qX TF4ONjskxVrIH4lD/xgQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHTzd-005WV8-SM; Wed, 03 Mar 2021 16:04:59 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSoP-005Da2-LS for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=5Zt+as2f0a/zAo23kFlziOtkEY/5uJ8Xq9YYontT/aE=; b=HZFuPJYgx+BLPQRYG/Dpu5yHLX qvQFQ5jEk30EVF67wmz+SPmjs9tRX2DJGxOBUmVzsm91U++92NS5bhvmHDJ7dOCxYrBwp3QDq3CzJ RYRLeK0z2S3nHeiv6Wdqa/XNgF0c9Xjzwqg9wpBeY5VOl4tHSdkDeZLcLsADr8wg2LvmdnLgRjhO4 Pd4/ReVOY9E5z8X2njC6vv59b6rIlzjaAaqGp2v8wWM0no2lys2pqo1DkfpHkKWQaMQzC5q6pwphh co8oWHtz0ZeqCniR2L9bqlToYFqZrPWX2mbD0s5WZAPWtt/XdbVhPt1XDg2GN4y0vNRCswpXDp8Yv 4LjwY6Eg==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6XZ-00HJNZ-Tw for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:30 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 73so1277365wma.3 for ; Tue, 02 Mar 2021 07:02:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5Zt+as2f0a/zAo23kFlziOtkEY/5uJ8Xq9YYontT/aE=; b=oy/UdeB5LbnLjtFmM/dbYim6mMzp2qHYBSC13VZs9FaJTFSyaT5CDRtQj/Eb4TG+BN 0qtiQe0mzOlHVdur176OUctyO9lmahpfO/cGM2Iz8/eLYjlye4xly371PfFUU6L1dmsW flOfBgxFPo02Ph1rABAUHRoOy8BisQRcd1qsoH8uwGogHwNT1F7aM+pl3qwYmMwE3rZ9 EOf4aVi4DTalvrWxvq7Nfmlls3aOTfLbQl8kPnYd+v0IKHNr1fcEHNU1ksLlXUbt8htj tPYk6Ic5sJ+UTglqHEL5gmqam/FGCqcrJCZzjWFrHaxx9v8IRAOvLk/SYa3HSSkqCdAe wljw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5Zt+as2f0a/zAo23kFlziOtkEY/5uJ8Xq9YYontT/aE=; b=Vg2ZNhAD+8G02UY32YUUHzzm5fnln9sG6LlGCchPxge5kn6AMncpXlssfcKJbJUNKD 4hVGDIw69iiwf7U/bLt0pK8kK93c7RpR2feQ1FT50NwiLU8z13uTmVUAOZLM4nryKE/n KCNBGm/Mb1O5goeH1EEIR4MKet5wE2dYRrbxuH4VK4cOWU8UqBxltKXL38qw0BG2noSZ MTca3qZJcRolUOz3TcQVmOPuhefPaW4jzRTHirQXOsaZbqZrjOzdWsqtyWhG7g3NP810 u4jc7c/lrlc6tzVBSeDSlyBK2PgkODZE2HX4cbZa7a8yER/4wtCYOV/WTisbd92N/RjQ 80GA== X-Gm-Message-State: AOAM533gjnG4vik5htAYLQJkrz92baWjieCMjvAkiN+hoGOgPabWLBup js8dS0BbQe51MMhyLnrI/i+Aw2Gr8DsX X-Google-Smtp-Source: ABdhPJxty67guDux2vyMCWfPogWra+Q8rpbOely2ebtEwkY4f4QkKw5qaFq5tmMfz2TLhcjiBewjHI5nTeut X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a05:600c:2254:: with SMTP id a20mr4602912wmm.115.1614697269979; Tue, 02 Mar 2021 07:01:09 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:58 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-29-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 28/32] KVM: arm64: Add kvm_pgtable_stage2_idmap_greedy() From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150230_803907_C0ED2C64 X-CRM114-Status: GOOD ( 20.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new map function to the KVM page-table library that allows to greedily create block identity-mappings. This will be useful to create lazily the host stage 2 page-table as it will own most of memory and will always be identity mapped. The new helper function creates the mapping in 2 steps: it first walks the page-table to compute the largest possible granule that can be used to idmap a given address without overriding existing incompatible mappings; and then creates a mapping accordingly. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 37 +++++++++ arch/arm64/kvm/hyp/pgtable.c | 119 +++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index c9f6ed76e0ad..e51dcce69a5e 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -96,6 +96,16 @@ enum kvm_pgtable_prot { #define PAGE_HYP_RO (KVM_PGTABLE_PROT_R) #define PAGE_HYP_DEVICE (PAGE_HYP | KVM_PGTABLE_PROT_DEVICE) +/** + * struct kvm_mem_range - Range of Intermediate Physical Addresses + * @start: Start of the range. + * @end: End of the range. + */ +struct kvm_mem_range { + u64 start; + u64 end; +}; + /** * enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk. * @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid @@ -379,4 +389,31 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size); int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, struct kvm_pgtable_walker *walker); +/** + * kvm_pgtable_stage2_idmap_greedy() - Identity-map an Intermediate Physical + * Address with a leaf entry at the highest + * possible level. + * @pgt: Page-table structure initialised by kvm_pgtable_*_init(). + * @addr: Input address to identity-map. + * @prot: Permissions and attributes for the mapping. + * @range: Boundaries of the maximum memory region to map. + * @mc: Cache of pre-allocated memory from which to allocate page-table + * pages. + * + * This function attempts to install high-level identity-mappings covering @addr + * without overriding existing mappings with incompatible permissions or + * attributes. An existing table entry may be coalesced into a block mapping + * if and only if it covers @addr and all its leafs are either invalid and/or + * have permissions and attributes strictly matching @prot. The mapping is + * guaranteed to be contained within the boundaries specified by @range at call + * time. If only a subset of the memory specified by @range is mapped (because + * of e.g. alignment issues or existing incompatible mappings), @range will be + * updated accordingly. + * + * Return: 0 on success, negative error code on failure. + */ +int kvm_pgtable_stage2_idmap_greedy(struct kvm_pgtable *pgt, u64 addr, + enum kvm_pgtable_prot prot, + struct kvm_mem_range *range, + void *mc); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 8aa01a9e2603..6897d771e2b2 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -987,3 +987,122 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgt->mm_ops->free_pages_exact(pgt->pgd, pgd_sz); pgt->pgd = NULL; } + +struct stage2_reduce_range_data { + kvm_pte_t attr; + u64 target_addr; + u32 start_level; + struct kvm_mem_range *range; +}; + +static int __stage2_reduce_range(struct stage2_reduce_range_data *data, u64 addr) +{ + u32 level = data->start_level; + + for (; level < KVM_PGTABLE_MAX_LEVELS; level++) { + u64 granule = kvm_granule_size(level); + u64 start = ALIGN_DOWN(data->target_addr, granule); + u64 end = start + granule; + + /* + * The pinned address is in the current range, try one level + * deeper. + */ + if (start == ALIGN_DOWN(addr, granule)) + continue; + + /* + * Make sure the current range is a reduction of the existing + * range before updating it. + */ + if (data->range->start <= start && end <= data->range->end) { + data->start_level = level; + data->range->start = start; + data->range->end = end; + return 0; + } + } + + return -EINVAL; +} + +#define KVM_PTE_LEAF_S2_COMPAT_MASK (KVM_PTE_LEAF_ATTR_S2_PERMS | \ + KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR | \ + KVM_PTE_LEAF_SW_BIT_PROT_NONE) + +static int stage2_reduce_range_walker(u64 addr, u64 end, u32 level, + kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + struct stage2_reduce_range_data *data = arg; + kvm_pte_t attr; + int ret; + + if (addr < data->range->start || addr >= data->range->end) + return 0; + + attr = *ptep & KVM_PTE_LEAF_S2_COMPAT_MASK; + if (!attr || attr == data->attr) + return 0; + + /* + * An existing mapping with incompatible protection attributes is + * 'pinned', so reduce the range if we hit one. + */ + ret = __stage2_reduce_range(data, addr); + if (ret) + return ret; + + return -EAGAIN; +} + +static int stage2_reduce_range(struct kvm_pgtable *pgt, u64 addr, + enum kvm_pgtable_prot prot, + struct kvm_mem_range *range) +{ + struct stage2_reduce_range_data data = { + .start_level = pgt->start_level, + .range = range, + .target_addr = addr, + }; + struct kvm_pgtable_walker walker = { + .cb = stage2_reduce_range_walker, + .flags = KVM_PGTABLE_WALK_LEAF, + .arg = &data, + }; + int ret; + + data.attr = stage2_get_prot_attr(prot) & KVM_PTE_LEAF_S2_COMPAT_MASK; + if (!data.attr) + return -EINVAL; + + /* Reduce the kvm_mem_range to a granule size */ + ret = __stage2_reduce_range(&data, range->end); + if (ret) + return ret; + + /* Walk the range to check permissions and reduce further if needed */ + do { + ret = kvm_pgtable_walk(pgt, range->start, range->end, &walker); + } while (ret == -EAGAIN); + + return ret; +} + +int kvm_pgtable_stage2_idmap_greedy(struct kvm_pgtable *pgt, u64 addr, + enum kvm_pgtable_prot prot, + struct kvm_mem_range *range, + void *mc) +{ + u64 size; + int ret; + + ret = stage2_reduce_range(pgt, addr, prot, range); + if (ret) + return ret; + + size = range->end - range->start; + return kvm_pgtable_stage2_map(pgt, range->start, size, range->start, + prot, mc); +} From patchwork Tue Mar 2 14:59:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E25CC433E0 for ; Wed, 3 Mar 2021 20:44:35 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D4DAC64EF8 for ; Wed, 3 Mar 2021 20:44:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D4DAC64EF8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iamtXNTPAYx8HRohJUOQyFodpvkInEBeqvHUvdxb7Vs=; b=EHXOa8T0ffCMi9 n2Bc35QL9aECocHLdeOmyoEPzzAlmIG+oXz8XhdKI4k0s1KhYVjSMVOTw8mTRwU0vj9LcBFPKAyhp VpBUPrkVnb+vGP/uauTTke3a1W/kbmyvmGYiWt+Z0R4TygKslINSUWkS7GFZKny5AVd0GgeZuGTtE I7EPDmgrImhaCvPUHXyKe2lWyCV1f8xQNHYBxmnitRig12ALoof4bD3e2lsWYVz9bE/FMXnpvC975 zy3ihYXz3xYnG+ztEuZMkFJ3jVsPDpLXEOgo79/x+9by0ljyYtobHleR+JyYTgPKJZds1gxyUmlFw kvDEHkReOlVNTf91fEtw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYHd-006SRv-61; Wed, 03 Mar 2021 20:39:49 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT6e-005J7T-9z for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:08:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Jg5go4Wd5FLiBP8kRkVAj+VGU2ss+/GL7/UumznY/w4=; b=Jl5UkoF0XfDe/0x4koWkgBSbMQ JVfuLc92tn4PAIm7YncKI+w2kiBofbN370ExkJ7YOtQUzFQB1eo5IHfUa/jvBa62kUDSWDxVEKM+b jprYjuhC9RrjgUTgUuGtw2K6YkMy7zo+qSMQObM/ZhhnmNXtT0fVklw/i3S4Pmo4g80uyJjiRCLQe F5gD+YR1IDYx0g8Z4EUrIqegKRFD0ZpDw7A6B4GJxCaHPAQcXZJDddgAXDUWShoBoTQJr02X2YXAZ ZnX+BqsyIJagrgcL+ZEp35F2/kxo5VyvJrBy6tdqXstAaUt/MvACqB94yuZt+EHWsXAq/9Y+otyqF 6+/V3Lmw==; Received: from mail-qk1-x749.google.com ([2607:f8b0:4864:20::749]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Xc-00HJNd-61 for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:31 +0000 Received: by mail-qk1-x749.google.com with SMTP id h21so17096182qkl.12 for ; Tue, 02 Mar 2021 07:02:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Jg5go4Wd5FLiBP8kRkVAj+VGU2ss+/GL7/UumznY/w4=; b=igasOdtuDiqpVcdxAPHRRksVB2sNzpiejUlUsNf4C0NgpeiqQekJZ58+YXImo1MIm2 5YYTVoFeW7pDPidv/lkWDaQcIXzUF81QTEMU6a/9G6agVkO5Cfp9t8ZUQ3iCmwtbbF8u AbzJLyM1T4hyEY24TIEuBOZEWJ2TzZUiKKz1AoRNbzOECeDEc8h/Zy9VQ/IxC6b2AH2D qhE2O02SLEEDvCKhMaFBIb3ATWJZqGsMkTDomc3/9IuCgUgB7c+cBK5Pc9UhpLX0+Zfr A7YX4LyFjSs3l9CLNrJPLho4gyziCvyf+YE1KvJPPWzhKyLcpc0BeT14U4/om7V63hXh bfUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Jg5go4Wd5FLiBP8kRkVAj+VGU2ss+/GL7/UumznY/w4=; b=CDKx/DpsFXNwHGwG9WlqKkijo2dScUY3EUB5AHSosEbGJhXLXOROrtWT8ganZ+oqV0 RPCcMqEf30n34gmzuExUFM04Xn+jNPVBKlJbU6xZrgin6orqN9aQlfmCR8oFT1rWdcw5 H7Lahb/Au87Fjfa0lhF6Q9TyYb4ooC8FUQd95uDag51NJ7cqPFV9lW/E6bvkIBs2n+fJ bakO16WKYUAHNPUGkQZl0pHNKcSm72A3WiGyrnQG4B1BeTmBlc3PUXNohgoiFjjOAeK7 nmWPQrc8yDO1jMQQEmMvmA3QPbhq8HTtnms4dveMVaG3QNt0xZIdUBfMPLko7tYw/UXt GF8w== X-Gm-Message-State: AOAM530ZNO2v75fLcmPYp+2Q+dfVxooVGFjblJqkLJ0YufZlwNjPEl8u 4AcFCckM0JCZSs1pw6qcZrQFq3ZTsKVv X-Google-Smtp-Source: ABdhPJz808GhNX5vbh3GFCLA3Ox+UkaMCaWk4Ecv3W+FM4bvmylVip7Cm+nrT6lSDce/iZ60Vy1/OZE5f16A X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a0c:fd47:: with SMTP id j7mr3869257qvs.22.1614697272185; Tue, 02 Mar 2021 07:01:12 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:59 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-30-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 29/32] KVM: arm64: Wrap the host with a stage 2 From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150231_540511_3A54229E X-CRM114-Status: GOOD ( 28.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM runs in protected nVHE mode, make use of a stage 2 page-table to give the hypervisor some control over the host memory accesses. The host stage 2 is created lazily using large block mappings if possible, and will default to page mappings in absence of a better solution. From this point on, memory accesses from the host to protected memory regions (e.g. marked PROT_NONE) are fatal and lead to hyp_panic(). Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/include/asm/kvm_cpufeature.h | 2 + arch/arm64/kernel/image-vars.h | 3 + arch/arm64/kvm/arm.c | 10 + arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 34 +++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 1 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 213 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 5 + arch/arm64/kvm/hyp/nvhe/switch.c | 7 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 4 +- 12 files changed, 286 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h create mode 100644 arch/arm64/kvm/hyp/nvhe/mem_protect.c diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 6dce860f8bca..b127af02bd45 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -61,6 +61,7 @@ #define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16 #define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 #define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 +#define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/include/asm/kvm_cpufeature.h b/arch/arm64/include/asm/kvm_cpufeature.h index d34f85cba358..74043a149322 100644 --- a/arch/arm64/include/asm/kvm_cpufeature.h +++ b/arch/arm64/include/asm/kvm_cpufeature.h @@ -15,3 +15,5 @@ #endif KVM_HYP_CPU_FTR_REG(SYS_CTR_EL0, arm64_ftr_reg_ctrel0) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR0_EL1, arm64_ftr_reg_id_aa64mmfr0_el1) +KVM_HYP_CPU_FTR_REG(SYS_ID_AA64MMFR1_EL1, arm64_ftr_reg_id_aa64mmfr1_el1) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 940c378fa837..d5dc2b792651 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -131,6 +131,9 @@ KVM_NVHE_ALIAS(__hyp_bss_end); KVM_NVHE_ALIAS(__hyp_rodata_start); KVM_NVHE_ALIAS(__hyp_rodata_end); +/* pKVM static key */ +KVM_NVHE_ALIAS(kvm_protected_mode_initialized); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index b6a818f88051..a31c56bc55b3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1889,12 +1889,22 @@ static int init_hyp_mode(void) return err; } +void _kvm_host_prot_finalize(void *discard) +{ + WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)); +} + static int finalize_hyp_mode(void) { if (!is_protected_kvm_enabled()) return 0; + /* + * Flip the static key upfront as that may no longer be possible + * once the host stage 2 is installed. + */ static_branch_enable(&kvm_protected_mode_initialized); + on_each_cpu(_kvm_host_prot_finalize, NULL, 1); return 0; } diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h new file mode 100644 index 000000000000..d293cb328cc4 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#ifndef __KVM_NVHE_MEM_PROTECT__ +#define __KVM_NVHE_MEM_PROTECT__ +#include +#include +#include +#include +#include + +struct host_kvm { + struct kvm_arch arch; + struct kvm_pgtable pgt; + struct kvm_pgtable_mm_ops mm_ops; + hyp_spinlock_t lock; +}; +extern struct host_kvm host_kvm; + +int __pkvm_prot_finalize(void); +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool); +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); + +static __always_inline void __load_host_stage2(void) +{ + if (static_branch_likely(&kvm_protected_mode_initialized)) + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + else + write_sysreg(0, vttbr_el2); +} +#endif /* __KVM_NVHE_MEM_PROTECT__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index e204ea77ab27..ce49795324a7 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -14,7 +14,7 @@ lib-objs := $(addprefix ../../../lib/, $(lib-objs)) obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o stub.o page_alloc.o \ - cache.o cpufeature.o setup.o mm.o + cache.o cpufeature.o setup.o mm.o mem_protect.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index f312672d895e..6fa01b04954f 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -119,6 +119,7 @@ alternative_else_nop_endif /* Invalidate the stale TLBs from Bootloader */ tlbi alle2 + tlbi vmalls12e1 dsb sy /* diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index ae6503c9be15..f47028d3fd0a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -13,6 +13,7 @@ #include #include +#include #include #include @@ -151,6 +152,10 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); } +static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) +{ + cpu_reg(host_ctxt, 1) = __pkvm_prot_finalize(); +} typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -174,6 +179,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_cpu_set_vector), HANDLE_FUNC(__pkvm_create_mappings), HANDLE_FUNC(__pkvm_create_private_mapping), + HANDLE_FUNC(__pkvm_prot_finalize), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) @@ -226,6 +232,11 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; + case ESR_ELx_EC_IABT_LOW: + fallthrough; + case ESR_ELx_EC_DABT_LOW: + handle_host_mem_abort(host_ctxt); + break; default: hyp_panic(); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c new file mode 100644 index 000000000000..2252ad1a8945 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2020 Google LLC + * Author: Quentin Perret + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include + +extern unsigned long hyp_nr_cpus; +struct host_kvm host_kvm; + +struct hyp_pool host_s2_mem; +struct hyp_pool host_s2_dev; + +static void *host_s2_zalloc_pages_exact(size_t size) +{ + return hyp_alloc_pages(&host_s2_mem, get_order(size)); +} + +static void *host_s2_zalloc_page(void *pool) +{ + return hyp_alloc_pages(pool, 0); +} + +static int prepare_s2_pools(void *mem_pgt_pool, void *dev_pgt_pool) +{ + unsigned long nr_pages, pfn; + int ret; + + pfn = hyp_virt_to_pfn(mem_pgt_pool); + nr_pages = host_s2_mem_pgtable_pages(); + ret = hyp_pool_init(&host_s2_mem, pfn, nr_pages, 0); + if (ret) + return ret; + + pfn = hyp_virt_to_pfn(dev_pgt_pool); + nr_pages = host_s2_dev_pgtable_pages(); + ret = hyp_pool_init(&host_s2_dev, pfn, nr_pages, 0); + if (ret) + return ret; + + host_kvm.mm_ops.zalloc_pages_exact = host_s2_zalloc_pages_exact; + host_kvm.mm_ops.zalloc_page = host_s2_zalloc_page; + host_kvm.mm_ops.phys_to_virt = hyp_phys_to_virt; + host_kvm.mm_ops.virt_to_phys = hyp_virt_to_phys; + host_kvm.mm_ops.page_count = hyp_page_count; + host_kvm.mm_ops.get_page = hyp_get_page; + host_kvm.mm_ops.put_page = hyp_put_page; + + return 0; +} + +static void prepare_host_vtcr(void) +{ + u32 parange, phys_shift; + u64 mmfr0, mmfr1; + + mmfr0 = arm64_ftr_reg_id_aa64mmfr0_el1.sys_val; + mmfr1 = arm64_ftr_reg_id_aa64mmfr1_el1.sys_val; + + /* The host stage 2 is id-mapped, so use parange for T0SZ */ + parange = kvm_get_parange(mmfr0); + phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange); + + host_kvm.arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift); +} + +int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool) +{ + struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; + int ret; + + prepare_host_vtcr(); + hyp_spin_lock_init(&host_kvm.lock); + + ret = prepare_s2_pools(mem_pgt_pool, dev_pgt_pool); + if (ret) + return ret; + + ret = kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch, + &host_kvm.mm_ops); + if (ret) + return ret; + + mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd); + mmu->arch = &host_kvm.arch; + mmu->pgt = &host_kvm.pgt; + mmu->vmid.vmid_gen = 0; + mmu->vmid.vmid = 0; + + return 0; +} + +int __pkvm_prot_finalize(void) +{ + struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; + struct kvm_nvhe_init_params *params = this_cpu_ptr(&kvm_init_params); + + params->vttbr = kvm_get_vttbr(mmu); + params->vtcr = host_kvm.arch.vtcr; + params->hcr_el2 |= HCR_VM; + if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) + params->hcr_el2 |= HCR_FWB; + kvm_flush_dcache_to_poc(params, sizeof(*params)); + + write_sysreg(params->hcr_el2, hcr_el2); + __load_stage2(&host_kvm.arch.mmu, host_kvm.arch.vtcr); + + __tlbi(vmalls12e1is); + dsb(ish); + isb(); + + return 0; +} + +static void host_stage2_unmap_dev_all(void) +{ + struct kvm_pgtable *pgt = &host_kvm.pgt; + struct memblock_region *reg; + u64 addr = 0; + int i; + + /* Unmap all non-memory regions to recycle the pages */ + for (i = 0; i < hyp_memblock_nr; i++, addr = reg->base + reg->size) { + reg = &hyp_memory[i]; + kvm_pgtable_stage2_unmap(pgt, addr, reg->base - addr); + } + kvm_pgtable_stage2_unmap(pgt, addr, ULONG_MAX); +} + +static bool find_mem_range(phys_addr_t addr, struct kvm_mem_range *range) +{ + int cur, left = 0, right = hyp_memblock_nr; + struct memblock_region *reg; + phys_addr_t end; + + range->start = 0; + range->end = ULONG_MAX; + + /* The list of memblock regions is sorted, binary search it */ + while (left < right) { + cur = (left + right) >> 1; + reg = &hyp_memory[cur]; + end = reg->base + reg->size; + if (addr < reg->base) { + right = cur; + range->end = reg->base; + } else if (addr >= end) { + left = cur + 1; + range->start = end; + } else { + range->start = reg->base; + range->end = end; + return true; + } + } + + return false; +} + +static int host_stage2_idmap(u64 addr) +{ + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W; + struct kvm_mem_range range; + bool is_memory = find_mem_range(addr, &range); + struct hyp_pool *pool = is_memory ? &host_s2_mem : &host_s2_dev; + int ret; + + if (is_memory) + prot |= KVM_PGTABLE_PROT_X; + + hyp_spin_lock(&host_kvm.lock); + ret = kvm_pgtable_stage2_idmap_greedy(&host_kvm.pgt, addr, prot, + &range, pool); + if (is_memory || ret != -ENOMEM) + goto unlock; + host_stage2_unmap_dev_all(); + ret = kvm_pgtable_stage2_idmap_greedy(&host_kvm.pgt, addr, prot, + &range, pool); +unlock: + hyp_spin_unlock(&host_kvm.lock); + + return ret; +} + +void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) +{ + struct kvm_vcpu_fault_info fault; + u64 esr, addr; + int ret = 0; + + esr = read_sysreg_el2(SYS_ESR); + if (!__get_fault_info(esr, &fault)) + hyp_panic(); + + addr = (fault.hpfar_el2 & HPFAR_MASK) << 8; + ret = host_stage2_idmap(addr); + if (ret && ret != -EAGAIN) + hyp_panic(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 7e923b25271c..94b9f14491f9 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -157,6 +158,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; + ret = kvm_host_prepare_stage2(host_s2_mem_pgt_base, host_s2_dev_pgt_base); + if (ret) + goto out; + pkvm_pgtable_mm_ops.zalloc_page = hyp_zalloc_hyp_page; pkvm_pgtable_mm_ops.phys_to_virt = hyp_phys_to_virt; pkvm_pgtable_mm_ops.virt_to_phys = hyp_virt_to_phys; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 979a76cdf9fb..31bc1a843bf8 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -28,6 +28,8 @@ #include #include +#include + /* Non-VHE specific context */ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); @@ -102,11 +104,6 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) write_sysreg(__kvm_hyp_host_vector, vbar_el2); } -static void __load_host_stage2(void) -{ - write_sysreg(0, vttbr_el2); -} - /* Save VGICv3 state on non-VHE systems */ static void __hyp_vgic_save_state(struct kvm_vcpu *vcpu) { diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index fbde89a2c6e8..255a23a1b2db 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -8,6 +8,8 @@ #include #include +#include + struct tlb_inv_context { u64 tcr; }; @@ -43,7 +45,7 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu, static void __tlb_switch_to_host(struct tlb_inv_context *cxt) { - write_sysreg(0, vttbr_el2); + __load_host_stage2(); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { /* Ensure write of the host VMID */ From patchwork Tue Mar 2 15:00:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 153B0C433E0 for ; Wed, 3 Mar 2021 17:42:31 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 823AF601FD for ; Wed, 3 Mar 2021 17:42:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 823AF601FD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=k2cxYPdL+vS5R7V3axK+FhXwyqGdH2MfU8zUQVzuB0k=; b=CblcxL27UBcnHX 8AHzkvD9wEN+0Lt3d/S6jtfgrfMeS73lf5W2yUPMownABZq6GeYT/pCxzk/V+1rjnSFQtIbI4IvAu XoEiBKCZiy8ESUNAJm2YNduEfPo6vhpdAOalwpaSM4qRNjjgknzO+tAbA2AJ5mt09yVVystBx3ikD V0x3nusYDSUhE62dATvb124i91IKLkB3gyTl5lpLhNm4wYBJAlC3hZxhZQbdBP5EDc5b6AxBAFVPl FxvZX/7lx2sV+k1arvRRGPptz/KAxnzRgrx6atERI6O2tPVYG//URn9/Kckz5jS9+mPtx9hZYRi6z 93NyGsS5sOujROBRoz7A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHVTP-005qM2-Px; Wed, 03 Mar 2021 17:39:48 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSvn-005FfJ-Ep for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:56:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=oOi4Pd0Cf29FRmhNkB2FdW4vfgElKPAhhlmUWrOeLOo=; b=Rtnx1zNif33hQQRIlr/mzic9rL N4Th+y9bEe4DP8uVmjK1ewOd37iIhuJXD0UYxHtxaPgxE+oC7UV0xhjfoL11vxLDp4HLhJKrT3rC2 gDUlptkWQ+GIxM5P+dWhYwNkX38sovWtJkx4fMfJkdYzVLrJZp6OWevpSNHEE9zm8bA8JZ0kM8wvT Gy97h5BsBDLfB1XyuYAM5OpsK9rXHSyljHMJnGdpV4eLdY+EBoE3qwvWbjnwcTR3OUKEPCX9sMh3b JN7wTjeqlaBYqK10SRwoezk8SS/Li9CUthRqTeFq6z70u+GA0qTh/wWXCRK92axH4XQ67zaBI6oaw mCAHlJaw==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Xh-00HJO5-36 for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:40 +0000 Received: by mail-wr1-x44a.google.com with SMTP id v1so715507wru.7 for ; Tue, 02 Mar 2021 07:02:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=oOi4Pd0Cf29FRmhNkB2FdW4vfgElKPAhhlmUWrOeLOo=; b=DAmMeWzNT5OYqZl5202wU65yxy+Ydp523Zp7hpAs3u6y50OwoTcx758BDuByCvqfpr lL66LK7SqeqjJ31D3FeWNNmuzeeAYFbG0jOXhwExojOAYwgKPFB/Upejy1HSGCOeocBz Q77qxjqgfA37YlEo0lvNhexw9R25bV/YneXYP1EAd0KGepP8GHBS0pghcx5PMdxRBZgc YnHcQtY1pOu1oGjAOse8QuPkG5uwHCk1qvJOhgP1bCeNHMroUAn9rsUMkI+Xf9c0o3tx ZwvKhjg5Mi3jbC3w5KrMFXC/hihqbKsMJfhqjnCBux8y/nIV1yUYUWlZYlo8euhBHzJV wVYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=oOi4Pd0Cf29FRmhNkB2FdW4vfgElKPAhhlmUWrOeLOo=; b=Pnhd8qGFFL0hbq0iBZhCeyyteSEpl5lmk2fqToht0dhFcyESgRykMxFLWDrJGJGdMT pXH6AivFv2sIG8pPJW0PVU6nCUPL4v//GPDL+MilKaYm04qQkvhhCXUOhbwJt/I71Xvr NY92ZUWqcir17kp0/sT4wo/AvAOpIVLw/442IPZH99YOg3j4gaQ01QC+KW4lMlCt67Cg dqIsL6pSlFdg/KQ2/McXgbnGnocz765mp9hDw3oSTEDB3RrUsxRC3QB6aOdtbfQaHjdB 4S6jw94Z1r4oNjDvHqt5OgpqFWwQlkYwdma72bTlFxgEWI7FXA+cwLru6GmVrZfV6UGB 06Aw== X-Gm-Message-State: AOAM531pafprW/Cg132KGMreBoCgcICCkSS+xOyqrtOgo8W80qu1Edpq qu4CcY/65HeYSYd+BRztYToXcJtQBky5 X-Google-Smtp-Source: ABdhPJy3ARsSSbx99em/qRXAe711Of+WD5kY+Gg+HksN47kkDAMNctzWR2nongDDAS+SviyOgYMF6zN0Mxfb X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:1dd5:: with SMTP id d204mr4338156wmd.127.1614697274353; Tue, 02 Mar 2021 07:01:14 -0800 (PST) Date: Tue, 2 Mar 2021 15:00:00 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-31-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 30/32] KVM: arm64: Page-align the .hyp sections From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150240_394619_84D5E409 X-CRM114-Status: GOOD ( 12.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will soon unmap the .hyp sections from the host stage 2 in Protected nVHE mode, which obvisouly works with at least page granularity, so make sure to align them correctly. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kernel/vmlinux.lds.S | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index e96173ce211b..709d2c433c5e 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -15,9 +15,11 @@ #define HYPERVISOR_DATA_SECTIONS \ HYP_SECTION_NAME(.rodata) : { \ + . = ALIGN(PAGE_SIZE); \ __hyp_rodata_start = .; \ *(HYP_SECTION_NAME(.data..ro_after_init)) \ *(HYP_SECTION_NAME(.rodata)) \ + . = ALIGN(PAGE_SIZE); \ __hyp_rodata_end = .; \ } @@ -72,21 +74,14 @@ ENTRY(_text) jiffies = jiffies_64; #define HYPERVISOR_TEXT \ - /* \ - * Align to 4 KB so that \ - * a) the HYP vector table is at its minimum \ - * alignment of 2048 bytes \ - * b) the HYP init code will not cross a page \ - * boundary if its size does not exceed \ - * 4 KB (see related ASSERT() below) \ - */ \ - . = ALIGN(SZ_4K); \ + . = ALIGN(PAGE_SIZE); \ __hyp_idmap_text_start = .; \ *(.hyp.idmap.text) \ __hyp_idmap_text_end = .; \ __hyp_text_start = .; \ *(.hyp.text) \ HYPERVISOR_EXTABLE \ + . = ALIGN(PAGE_SIZE); \ __hyp_text_end = .; #define IDMAP_TEXT \ @@ -322,11 +317,12 @@ SECTIONS #include "image-vars.h" /* - * The HYP init code and ID map text can't be longer than a page each, - * and should not cross a page boundary. + * The HYP init code and ID map text can't be longer than a page each. The + * former is page-aligned, but the latter may not be with 16K or 64K pages, so + * it should also not cross a page boundary. */ -ASSERT(__hyp_idmap_text_end - (__hyp_idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, - "HYP init code too big or misaligned") +ASSERT(__hyp_idmap_text_end - __hyp_idmap_text_start <= PAGE_SIZE, + "HYP init code too big") ASSERT(__idmap_text_end - (__idmap_text_start & ~(SZ_4K - 1)) <= SZ_4K, "ID map text too big or misaligned") #ifdef CONFIG_HIBERNATION From patchwork Tue Mar 2 15:00:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EB5FC433DB for ; Wed, 3 Mar 2021 20:54:37 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E8E2764EFD for ; Wed, 3 Mar 2021 20:54:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E8E2764EFD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=txHZI8f2WeuhiEANcIrvDJqgSmzAiKsX23BCc6FpRqk=; b=VwATmeB8Z+3lqn ZxTNtefXo5N4MdaJJo5TkEtHksThDPiFS0IF4fcgT5OiWXnk1c2BqYgjIvSoxnkg+oZq0aXi5xMJ5 vQkw0/kXWMcxLDeKLHdXeiD5I/t+x8jHYOgqt2xpaCCDo8SWJkDOAHUjRof7W0NCtNZsopUVlGfSF iM6bSz+TfssGunhI6Arrew7CsUW5flCIs8CmmGipqQvcZ/3A/UcpRdzSxIZryyrN6QpqPsVH1Kn+V HOW6z21oNaWE8rg75ffnqf8H1+P1GlLM6rEzELrGv2PrY+xcligoc8KOP3cghYyoCjq16E1wl5Lw0 sGfCdGDSSpddvkQSVPNA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHYRQ-006U2I-3u; Wed, 03 Mar 2021 20:49:56 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT75-005JIC-Dp for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:08:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=HnBOCbV9VZG795nVKhv9+ovMJqzwdkwkno12utvuuaM=; b=YhovO6ARlh3wd2UAmtwjrvSujW gQ5b0cCc6CjTjn4IxwSPZQ1Tju2mAPWvsGItik4RXGqf0qC6r/WOxL64AMR4FxWpZmfIUTzva1e/w 3vxelDtWF8j4vEWYE5tl0AvEMTqEQWDZvM9Q1dF6uqkFOykM1IjlkltWlgINfjcMllGxGpo1+Kcvo q5k35YURSpUeS7ORF2MslF75GlH26urYnkeGo50Al/4FrzMmzpJ8GV+vvak/D8HDPgml+Hk8S489Q lttaKJrkERf6LfH8bW3iXtu78p96AIAQKFHZ6PDZ4Ds49qtA3sd9sgowrNzp9c/sTGwzUedqZEj21 2JSz11jg==; Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Xf-00HJOZ-9H for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:37 +0000 Received: by mail-qk1-x74a.google.com with SMTP id v184so17007703qkd.22 for ; Tue, 02 Mar 2021 07:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=HnBOCbV9VZG795nVKhv9+ovMJqzwdkwkno12utvuuaM=; b=MKbgEjESvYdThRoEkGC8RJg+Cz2zkWhWnC7wgzpWZoobh8AkycND5mEZtaZxNGp2tA qMEecbBNJ3k3ERyG50+v/O8XaK00waLfRVvvbN9E3QmDrCvJ0/93A4GuOQMcQEo8MGCk KW1fqTwiueVCj6eHaOhm9O/4JCquufE/tW2kQHZIkPJNvVz4btOdP6ZSAwQe3mnRfPlP C8g40caoVrU0GmaEVlnlhp/nXz17sJ/jz1XYk3CXeUoxTeSCsR7/po0Rbj/Hjj2JDVka EUIXH/D9Ghscn3/N8tbMa84ZOQHuQsv2ycIjv+zXEQqXRP/n8+aHwHdkJu4dhtp5hMOJ onJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HnBOCbV9VZG795nVKhv9+ovMJqzwdkwkno12utvuuaM=; b=Px/lxIW4khYMwtx4rjAd7VQh/6UGOPFLBsz8cIrVJt+zxFueO9Nne2rCxrupUqcnyh W3XRNsFoDRNLQjghQ7vslK20OnWjcQJHnI3dpxsUKS3zkfERj/TiwsUDDKC94/5AbHfW hUK7PGRK5S9csVhs/9sxHenz5Bv/hHLZpeeJXM73lvYzN7H/F0S1IBkx5+ec+xqoPrir g+u21CaNgqWcqQ5gtyzt8tsOFEyXCalL7IcAZUek+qAarzC4sC2B5tv2yEBVlYaiGDfR nBMtj9IPaOvbpYxPxI5K22hj92G/njySyHwkosAfDxjHJkf/buMgIvlx4LcAckyj8k9+ SBBA== X-Gm-Message-State: AOAM530EhXr1I4qGmWJUy6hRCrXq+Ez+swZJ1wkMKbai1NbLc3xCIL2O pctUC020Ic1J+W0bvclW1g/ZiNL6ENI2 X-Google-Smtp-Source: ABdhPJxomthrvtGXmLscfNQ4jzi8tGZD1QtytMGc0+RJgY/tfETMl/DO5FUjo9TRqk9VS+/6RprJ6oK1LBhx X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:ad4:56e9:: with SMTP id cr9mr20353191qvb.4.1614697276369; Tue, 02 Mar 2021 07:01:16 -0800 (PST) Date: Tue, 2 Mar 2021 15:00:01 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-32-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 31/32] KVM: arm64: Disable PMU support in protected mode From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150237_585691_9CE60563 X-CRM114-Status: GOOD ( 12.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The host currently writes directly in EL2 per-CPU data sections from the PMU code when running in nVHE. In preparation for unmapping the EL2 sections from the host stage 2, disable PMU support in protected mode as we currently do not have a use-case for it. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/kvm/perf.c | 3 ++- arch/arm64/kvm/pmu.c | 8 ++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/perf.c b/arch/arm64/kvm/perf.c index 739164324afe..8f860ae56bb7 100644 --- a/arch/arm64/kvm/perf.c +++ b/arch/arm64/kvm/perf.c @@ -55,7 +55,8 @@ int kvm_perf_init(void) * hardware performance counters. This could ensure the presence of * a physical PMU and CONFIG_PERF_EVENT is selected. */ - if (IS_ENABLED(CONFIG_ARM_PMU) && perf_num_counters() > 0) + if (IS_ENABLED(CONFIG_ARM_PMU) && perf_num_counters() > 0 + && !is_protected_kvm_enabled()) static_branch_enable(&kvm_arm_pmu_available); return perf_register_guest_info_callbacks(&kvm_guest_cbs); diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c index faf32a44ba04..03a6c1f4a09a 100644 --- a/arch/arm64/kvm/pmu.c +++ b/arch/arm64/kvm/pmu.c @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp_sym(kvm_host_data); - if (!ctx || !kvm_pmu_switch_needed(attr)) + if (!kvm_arm_support_pmu_v3() || !ctx || !kvm_pmu_switch_needed(attr)) return; if (!attr->exclude_host) @@ -49,7 +49,7 @@ void kvm_clr_pmu_events(u32 clr) { struct kvm_host_data *ctx = this_cpu_ptr_hyp_sym(kvm_host_data); - if (!ctx) + if (!kvm_arm_support_pmu_v3() || !ctx) return; ctx->pmu_events.events_host &= ~clr; @@ -172,7 +172,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu) struct kvm_host_data *host; u32 events_guest, events_host; - if (!has_vhe()) + if (!kvm_arm_support_pmu_v3() || !has_vhe()) return; preempt_disable(); @@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu) struct kvm_host_data *host; u32 events_guest, events_host; - if (!has_vhe()) + if (!kvm_arm_support_pmu_v3() || !has_vhe()) return; host = this_cpu_ptr_hyp_sym(kvm_host_data); From patchwork Tue Mar 2 15:00:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12114143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F70AC433DB for ; Wed, 3 Mar 2021 19:25:59 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 83F39601FB for ; Wed, 3 Mar 2021 19:25:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 83F39601FB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/WuAPeWABPCjA7l73w23r/MPa3NX5l9V8KgFn+hJcbM=; b=Lz7QJw45NzVC1t kk6v4XtadFHXpI+WpAVi2ojtuT+dSm3MSsJiQ1/ZFvO2mbAB4T/571OXrmWRh/2RrkqDd3U60FMM5 NDkck6pTaFbgkXzTN6Xz5oAc39+9W1g8oVZQLXrKp0/P3UiAzXwqkINy7EoRZz81ZHuD6DKyxo4La Y6ZB2b2L0eDL2910gc5KyC8AF4RosUqxVkwbh8B/dC+IthLCm9DFSbdqCwz/vcRme5P7d79j5exUk vpfwbIhs0gQQFfBPMuWwtKW/5wr24ADb45WSG54NzZbYDfLMgVEUM1ecX1lmk1aJj0tco3UE48CfZ ovHEtmRwNfc4hsjX0MPw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHX5r-006CLN-Vt; Wed, 03 Mar 2021 19:23:36 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHT3C-005HyR-Mp for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 15:04:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=8yMjY9nGGdG3kCURKAswxRcBn78w62ceWoknmifn0Fc=; b=ttjQgMe+V8BpWPcduujoBtfTyj zGVySdA/J93ZHl8TRWzo5bdFvzn/B5wOTcSYb1ksh2bmNnW6zuDUAd42qw2nYJ2KsycAdoZiCZBKc jT4BKeaVnAxMsFoHOdMWp21w5r+Z3R95GL2jlw4vhaNGcP6dYM7H4KbEUTezJr94Vx+xBhkdMsR+a 4D0xAO/sxZnay9QwnXVYCwE2TF4JrZ+O4UIdcyPJGx+gqazhRcwBjI7SGJiSNsj8fRCsnPUy1b4YZ NcYDvjbjN5+22LKnu2LItZu6eaDJ+Fx9abdbZtt0tJyWyPYtonaD27BlHiduyN8Qbyb2Q+DEmgA9i 3wHK7/fg==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Xl-00HJOY-AH for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:02:41 +0000 Received: by mail-wr1-x449.google.com with SMTP id r12so7158186wro.15 for ; Tue, 02 Mar 2021 07:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=8yMjY9nGGdG3kCURKAswxRcBn78w62ceWoknmifn0Fc=; b=wSZRd5YyhLGwCBAFIt3Ga6LuNLgf8Fi8I6omo/pvR9foidKTkYk2IXlyNbnBoJFF34 KbnndPtsrm7ghrq1Wy+pnLcenm1IcfH2XxjeT4mApWic44zxOx6Uq8kuiHrlXdgJl4Ob 7kvIFN11fUx/ydfa1rpVa59aYAbnaYJGYgu8EHWbQE8o/aBq0LBE4sk9GsmrJKArwKTm B9v13xqdO1QuVwjYSH4t8Co9vDG9yCdw3TgpxYoQOjPkvVUJ6coXDKrHjKWL0v6UXTK4 qAr08axgDfhr9fJ3I0sIV4a/PFg0+4MF3lvC4K7GocSYm3sV7IWq6xwjI/diP+ESwcs+ 64IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8yMjY9nGGdG3kCURKAswxRcBn78w62ceWoknmifn0Fc=; b=uDuhdC4NeMY3rEXg/ISfzMwcHJ+kM1iRKQqGz4uvzh5UwUpqbeEgkZ2CUa6iTanKrm /pH2IyeIl2+xj7sFPnZM0Ikssl6MatiF50scH7pLHuZtK28UNVm30sspPvzDGr6rw5EY gDADy5xFZiq1bZiq3Qsa9SnteMIvlAfW/mOmoBrFg5ayKpeMNPK+i9p7CgOUA/sfiDYG OFSDrLqpx1fCSERDsz6z8FWkpjEsOqlYYUBfXuerbx5t+6OG8tKFQrpIK+fuHwEB3rN8 oS3fIUSo3U6XA834QtpsqCnxobVPZ3+EGyrgR35nrENsSH49wfl7G0n8FmqUHE18o3Qw E58g== X-Gm-Message-State: AOAM532/Ni4FFYAGPuq2iY6MDCt2Whwdrk96kLRCUf7USwvPWLPn9WcN dWQWRJ2tEQKDo4ag/qkQWCDs6bBk+dMm X-Google-Smtp-Source: ABdhPJxYQIjUkt2RSBck3HRxMcW3YX2laHG5V0FLpOXHWWygCA8hezD7vfJS55sNzIbZYYijKQfyTK90llcS X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:adf:f3cc:: with SMTP id g12mr13345460wrp.118.1614697278738; Tue, 02 Mar 2021 07:01:18 -0800 (PST) Date: Tue, 2 Mar 2021 15:00:02 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-33-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 32/32] KVM: arm64: Protect the .hyp sections from the host From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150241_817591_9FE7A88B X-CRM114-Status: GOOD ( 17.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When KVM runs in nVHE protected mode, use the host stage 2 to unmap the hypervisor sections. The long-term goal is to ensure the EL2 code can remain robust regardless of the host's state, so this starts by making sure the host cannot e.g. write to the .hyp sections directly. Signed-off-by: Quentin Perret Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/arm.c | 46 +++++++++++++++++++ arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 9 ++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 22 +++++++++ 5 files changed, 80 insertions(+) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index b127af02bd45..9accf5350858 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -62,6 +62,7 @@ #define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 #define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 #define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19 +#define __KVM_HOST_SMCCC_FUNC___pkvm_host_unmap 20 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index a31c56bc55b3..73c26d206542 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1894,11 +1894,57 @@ void _kvm_host_prot_finalize(void *discard) WARN_ON(kvm_call_hyp_nvhe(__pkvm_prot_finalize)); } +static inline int pkvm_host_unmap(phys_addr_t start, phys_addr_t end) +{ + return kvm_call_hyp_nvhe(__pkvm_host_unmap, start, end); +} + +#define pkvm_host_unmap_section(__section) \ + pkvm_host_unmap(__pa_symbol(__section##_start), \ + __pa_symbol(__section##_end)) + static int finalize_hyp_mode(void) { + int cpu, ret; + if (!is_protected_kvm_enabled()) return 0; + ret = pkvm_host_unmap_section(__hyp_idmap_text); + if (ret) + return ret; + + ret = pkvm_host_unmap_section(__hyp_text); + if (ret) + return ret; + + ret = pkvm_host_unmap_section(__hyp_rodata); + if (ret) + return ret; + + ret = pkvm_host_unmap_section(__hyp_bss); + if (ret) + return ret; + + ret = pkvm_host_unmap(hyp_mem_base, hyp_mem_base + hyp_mem_size); + if (ret) + return ret; + + for_each_possible_cpu(cpu) { + phys_addr_t start = virt_to_phys((void *)kvm_arm_hyp_percpu_base[cpu]); + phys_addr_t end = start + (PAGE_SIZE << nvhe_percpu_order()); + + ret = pkvm_host_unmap(start, end); + if (ret) + return ret; + + start = virt_to_phys((void *)per_cpu(kvm_arm_hyp_stack_page, cpu)); + end = start + PAGE_SIZE; + ret = pkvm_host_unmap(start, end); + if (ret) + return ret; + } + /* * Flip the static key upfront as that may no longer be possible * once the host stage 2 is installed. diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h index d293cb328cc4..39890d4f1dc8 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -21,6 +21,8 @@ struct host_kvm { extern struct host_kvm host_kvm; int __pkvm_prot_finalize(void); +int __pkvm_host_unmap(phys_addr_t start, phys_addr_t end); + int kvm_host_prepare_stage2(void *mem_pgt_pool, void *dev_pgt_pool); void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f47028d3fd0a..2069136fdaec 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -156,6 +156,14 @@ static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt) { cpu_reg(host_ctxt, 1) = __pkvm_prot_finalize(); } + +static void handle___pkvm_host_unmap(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, start, host_ctxt, 1); + DECLARE_REG(phys_addr_t, end, host_ctxt, 2); + + cpu_reg(host_ctxt, 1) = __pkvm_host_unmap(start, end); +} typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -180,6 +188,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_create_mappings), HANDLE_FUNC(__pkvm_create_private_mapping), HANDLE_FUNC(__pkvm_prot_finalize), + HANDLE_FUNC(__pkvm_host_unmap), }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 2252ad1a8945..ed480facdc88 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -196,6 +196,28 @@ static int host_stage2_idmap(u64 addr) return ret; } +int __pkvm_host_unmap(phys_addr_t start, phys_addr_t end) +{ + struct kvm_mem_range r1, r2; + int ret; + + /* + * host_stage2_unmap_dev_all() currently relies on MMIO mappings being + * non-persistent, so don't allow PROT_NONE in MMIO range. + */ + if (!find_mem_range(start, &r1) || !find_mem_range(end, &r2)) + return -EINVAL; + if (r1.start != r2.start) + return -EINVAL; + + hyp_spin_lock(&host_kvm.lock); + ret = kvm_pgtable_stage2_map(&host_kvm.pgt, start, end - start, start, + KVM_PGTABLE_PROT_NONE, &host_s2_mem); + hyp_spin_unlock(&host_kvm.lock); + + return ret; +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault;