From patchwork Mon Jul 19 10:47:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B22F8C07E9D for ; Mon, 19 Jul 2021 10:49:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D24C6101E for ; Mon, 19 Jul 2021 10:49:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7D24C6101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9+bCnENcg9F/B/eZpf0V9iGZtdw5ZKspAmy9Mzp7ThQ=; b=Fgez54TrXSnBAotW1UNLSVtW9L R2KiinZImB/Tk90mc98k4EpOgvJCav13cUFNDtSzgMDRs5n4mcuZmqj0TPWEC07yFweCV9rnC9T5o NYPBsENoHDLcNT9xKhX+jy/Wu+Zm31/iGPG8TK9fK9fDNd2X+MV5QtEd1wajU2jzUvlXeV2T4o/e8 ijQsFVZybwNKi3BgkgEC87Wrs7KCb5e3LUZLdo5MAumZ+Yu8cXV7uxVXnC+8OPRvKE+4oozm37Gpg Ad6yqzwoxCgcokpfGd+BNJV9KONAZ8Ukp4L8fRApi75mpCHPnGFHPtJKAs4zM036+VJiHcjy3LkKY wk+nLa7Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qop-009Kdu-QL; Mon, 19 Jul 2021 10:48:16 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoL-009KQj-7V for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:47 +0000 Received: by mail-qk1-x74a.google.com with SMTP id 81-20020a370b540000b02903b854c43335so14233687qkl.21 for ; Mon, 19 Jul 2021 03:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hd3eXFI12/cSmrxyxLEWCVWM4Lg6AoDwKdvReeRGEGk=; b=HGOnJdXy3Mnr4dlmP09v+4kaA2VwsExFHRqk7q43R4drwSHPnf1kYry4Wl5kKwf0Ee q/AgvmFXPCqnse7moGoX/peLgHs08Z53JM0cUkTPG1e+mtiHg1mJ2gYMZZUr8eQckACF wPuNxnbctRGG/p9PXW+RIQz859Ea6UKp0pNoSz1H4BUNYSpYklqGDGeZKmZm2FHM5kEr vGVayYqJLFtEQKuIrH7lDHglKIh66lQcveE6hfwNqQ1Tdyg10vEx/33yvUTrcxgKd8UJ LivnrEnDoNFECAWWZ+T5aV5YpgdGclh8bF5DU/PHmR1fk5vvK3XCxKeOwUm2PcqyFwgY tglQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hd3eXFI12/cSmrxyxLEWCVWM4Lg6AoDwKdvReeRGEGk=; b=bDubkLPu3BvwlMA6OORZsV6ZnB9frw2c+gm8yQFZxtOv8Q2PHD2smbTrImD/mtKtw8 GqD2JRqv0DYElJkrDQNGGSYzY8DxtQ3lj30xodmlYyag0RnJo560TicGe15BDxiEV5O6 uGfsKqnka2M89u6f977NsQuK4K593J4ZP/pla7+zyNNjbK71+91HjGKrLUBD1rZeeptJ 51YD3wq1q3OQ1gZR+aufjkzu5HiRiwevqHS1tPUil3OtuhCy4jqc9ef8LYCjVU/Cv4Yz hrtzV1va3PbD5T1He6cFkP+2JVqBH1JuTI0lqwoh2zHsmlr7LW+FQNRVVaZuQ4SyiIjy pteg== X-Gm-Message-State: AOAM532Z6Iki9ZpiZbTsCwt0HmelW+9E6wngpQnhdNjfVtGvcgWmLWQM sDqLPPcK561CNpcPB4YYFxbMdLKKgFQG X-Google-Smtp-Source: ABdhPJymJx2U/vPY7h8AjNJlQB0JopARk77JVivMgxwS6x59HR1TE3wCFjBaR7svOiHcbbVPqjLfxnY80gtQ X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a0c:c350:: with SMTP id j16mr24060301qvi.51.1626691663046; Mon, 19 Jul 2021 03:47:43 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:22 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-2-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 01/14] KVM: arm64: Provide the host_stage2_try() helper macro From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034745_341553_6731B94B X-CRM114-Status: GOOD ( 16.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We currently unmap all MMIO mappings from the host stage-2 to recycle the pages whenever we run out. In order to make this pattern easy to re-use from other places, factor the logic out into a dedicated macro. While at it, apply the macro for the kvm_pgtable_stage2_set_owner() calls. They're currently only called early on and are guaranteed to succeed, but making them robust to the -ENOMEM case doesn't hurt and will avoid painful debugging sessions later on. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 38 ++++++++++++++------------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index d938ce95d3bd..56f2117c877b 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -208,6 +208,23 @@ static inline int __host_stage2_idmap(u64 start, u64 end, prot, &host_s2_pool); } +/* + * The pool has been provided with enough pages to cover all of memory with + * page granularity, but it is difficult to know how much of the MMIO range + * we will need to cover upfront, so we may need to 'recycle' the pages if we + * run out. + */ +#define host_stage2_try(fn, ...) \ + ({ \ + int __ret = fn(__VA_ARGS__); \ + if (__ret == -ENOMEM) { \ + __ret = host_stage2_unmap_dev_all(); \ + if (!__ret) \ + __ret = fn(__VA_ARGS__); \ + } \ + __ret; \ + }) + static int host_stage2_idmap(u64 addr) { enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W; @@ -223,22 +240,7 @@ static int host_stage2_idmap(u64 addr) if (ret) goto unlock; - ret = __host_stage2_idmap(range.start, range.end, prot); - if (ret != -ENOMEM) - goto unlock; - - /* - * The pool has been provided with enough pages to cover all of memory - * with page granularity, but it is difficult to know how much of the - * MMIO range we will need to cover upfront, so we may need to 'recycle' - * the pages if we run out. - */ - ret = host_stage2_unmap_dev_all(); - if (ret) - goto unlock; - - ret = __host_stage2_idmap(range.start, range.end, prot); - + ret = host_stage2_try(__host_stage2_idmap, range.start, range.end, prot); unlock: hyp_spin_unlock(&host_kvm.lock); @@ -257,8 +259,8 @@ int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end) return -EINVAL; hyp_spin_lock(&host_kvm.lock); - ret = kvm_pgtable_stage2_set_owner(&host_kvm.pgt, start, end - start, - &host_s2_pool, pkvm_hyp_id); + ret = host_stage2_try(kvm_pgtable_stage2_set_owner, &host_kvm.pgt, + start, end - start, &host_s2_pool, pkvm_hyp_id); hyp_spin_unlock(&host_kvm.lock); return ret != -EAGAIN ? ret : 0; From patchwork Mon Jul 19 10:47:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A97C07E9B for ; Mon, 19 Jul 2021 10:50:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9F25C6101E for ; Mon, 19 Jul 2021 10:50:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F25C6101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EySfX9tgUIUcVFVNtNqQzGnQNxRalfHkG/TjQL779rk=; b=JvNgFXJXZqg0yD4NVPAb4btETU SvsFwy3oByG7Z7teu7G3/k9/tc+GLEoiD4X2mvvxIfvFWnLhCV5VESkj//y1fRUjW47j3MZ0QVOAd HFJxv3wC4HFqGUmpsykQyRle9lr0RlqV7nZdCu56Uxi2kRxWL5UPi9ZhhWvLzNqzSjm+jFRcYjUX8 i2Jt7aYBHr78cjA4HMXIB3S3c/54THqkDQtYsbR28qbYEVjH6SsKWhmXnYl3tnF34ayMM0bZ3vt5E iC+wnuMqbnzNjlDyATXUtEcc//iNZ8a+jnHKX4F/E/Sah7riHihCZC5yvKEXQMKsCMw/28IfBPez/ UVyPwxnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QpE-009KnG-UK; Mon, 19 Jul 2021 10:48:41 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoO-009KS9-83 for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:51 +0000 Received: by mail-wm1-x34a.google.com with SMTP id l6-20020a05600c1d06b0290225338d8f53so5282882wms.8 for ; Mon, 19 Jul 2021 03:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1v6USBe8mNAVx9OftewEBHt9DvMOxZznDU6MzJZEG7M=; b=tdSbuopKCmoJ0ZYg7u+hYeez6RMuzLcn3HSKhcMvEa15oU4q+BUnmsmih8waTH+10G c+KOEs+YuXqX5TSWUQQ0/9lQAsSPs3MxhyHKGv2WmtqS/tr6MuOkdtUuN1XRyalXyOko jLhvjnzUgRhWxsPGGCNHzueJi5oQGIiUNcv0paNQimuP3x4o57Mu4nIOjXPN+GhgHpjo 72M930OJqoABPALkQ0VW7pcKmOUAllWIf39LmYK+lIm7buRM7El/jepJCucRdx7w7O4U 1ZeyRt+T2RNNJpSSW3tFB63GYZ8RtnVDy4YXFMjfLoUKSRQbJq5GA9nOjY06ImjHKdAv XMog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1v6USBe8mNAVx9OftewEBHt9DvMOxZznDU6MzJZEG7M=; b=rhm4kfaN3Oxv5MAMZsoSjhTrAMIDmtvtEvipl29rQjJb9xtBFCPmpkqLG1iV2FfCOC D58jd6oxbaCp/i5KpJpZ+7WBEHHNUNaFTPfnJvvsHLC48BiY0MdfJFjE16SjaxOhjdgQ cpKp7KvS7ZqbmdG0MrduFTVfTmUdUONYBlAhnTR70hidJtXyUfLsYHcB2Hk7BlR7Hi8O hv29DpqrichPy26+H01WjIkizmVjvX0EQiDm4tqoEvxA4YtSocKhReYXcnpZl+wMwIUd Fv/KV1SMeFKiJJNO0z27WopjAifojuvIqyvMOYVeSQcBwjQ6MqKJ7Rw6p5VQvgBQ1NI5 kleQ== X-Gm-Message-State: AOAM530QUAypMaRtI3AgV/dvHOP7WZVJDTaDpmwnDASHe9DDZkSZotVR /4tFkG1EYV9+N9YcxkaXd/FTEOoH0uHX X-Google-Smtp-Source: ABdhPJyR/5wrZXgluPqs2G8EtvNO/lpHpYOHQfbllOEj1h1n4cc3TBv5wr06GFFQFtrZVrhIWfKadbzl5L0S X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a7b:c1c7:: with SMTP id a7mr1853502wmj.1.1626691665402; Mon, 19 Jul 2021 03:47:45 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:23 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-3-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 02/14] KVM: arm64: Optimize kvm_pgtable_stage2_find_range() From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034748_414887_061CF4C0 X-CRM114-Status: GOOD ( 25.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kvm_pgtable_stage2_find_range() function is used in the host memory abort path to try and look for the largest block mapping that can be used to map the faulting address. In order to do so, the function currently walks the stage-2 page-table and looks for existing incompatible mappings within the range of the largest possible block. If incompatible mappings are found, it tries the same procedure again, but using a smaller block range, and repeats until a matching range is found (potentially up to page granularity). While this approach has benefits (mostly in the fact that it proactively coalesces host stage-2 mappings), it can be slow if the ranges are fragmented, and it isn't optimized to deal with CPUs faulting on the same IPA as all of them will do all the work every time. To avoid these issues, rework kvm_pgtable_stage2_find_range() by walking the page-table only once to find the closest leaf to the input address, and return its corresponding range if it is invalid and not owned by another entity. If a valid leaf is found, return -EAGAIN similar to what is done in the kvm_pgtable_stage2_map() path to optimize concurrent faults. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 12 ++--- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 74 +++++++++++---------------- 3 files changed, 34 insertions(+), 54 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index f004c0115d89..d6649352c8b3 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -434,21 +434,17 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, /** * kvm_pgtable_stage2_find_range() - Find a range of Intermediate Physical - * Addresses with compatible permission - * attributes. + * Addresses with a given ownership. * @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*(). * @addr: Address that must be covered by the range. - * @prot: Protection attributes that the range must be compatible with. + * @owner: Expected owner of the pages of the range. * @range: Range structure used to limit the search space at call time and * that will hold the result. * - * The offset of @addr within a page is ignored. An IPA is compatible with @prot - * iff its corresponding stage-2 page-table entry has default ownership and, if - * valid, is mapped with protection attributes identical to @prot. + * The offset of @addr within a page is ignored. * * Return: 0 on success, negative error code on failure. */ int kvm_pgtable_stage2_find_range(struct kvm_pgtable *pgt, u64 addr, - enum kvm_pgtable_prot prot, - struct kvm_mem_range *range); + int owner, struct kvm_mem_range *range); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 56f2117c877b..58edc62be6f7 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -236,7 +236,7 @@ static int host_stage2_idmap(u64 addr) prot |= KVM_PGTABLE_PROT_X; hyp_spin_lock(&host_kvm.lock); - ret = kvm_pgtable_stage2_find_range(&host_kvm.pgt, addr, prot, &range); + ret = kvm_pgtable_stage2_find_range(&host_kvm.pgt, addr, 0, &range); if (ret) goto unlock; diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 05321f4165e3..978f341d02ca 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1103,76 +1103,60 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt) pgt->pgd = NULL; } -#define KVM_PTE_LEAF_S2_COMPAT_MASK (KVM_PTE_LEAF_ATTR_S2_PERMS | \ - KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR | \ - KVM_PTE_LEAF_ATTR_S2_IGNORED) +struct stage2_check_permission_data { + u32 level; + u8 owner; +}; static int stage2_check_permission_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag, void * const arg) { - kvm_pte_t old_attr, pte = *ptep, *new_attr = arg; + struct stage2_check_permission_data *data = arg; + kvm_pte_t pte = *ptep; - /* - * Compatible mappings are either invalid and owned by the page-table - * owner (whose id is 0), or valid with matching permission attributes. - */ - if (kvm_pte_valid(pte)) { - old_attr = pte & KVM_PTE_LEAF_S2_COMPAT_MASK; - if (old_attr != *new_attr) - return -EEXIST; - } else if (pte) { + if (kvm_pte_valid(pte)) + return -EAGAIN; + + if (FIELD_GET(KVM_INVALID_PTE_OWNER_MASK, pte) != data->owner) return -EEXIST; - } + + data->level = level; return 0; } int kvm_pgtable_stage2_find_range(struct kvm_pgtable *pgt, u64 addr, - enum kvm_pgtable_prot prot, - struct kvm_mem_range *range) + int owner, struct kvm_mem_range *range) { - kvm_pte_t attr; + struct stage2_check_permission_data data = { + .level = pgt->start_level, + .owner = owner, + }; struct kvm_pgtable_walker check_perm_walker = { .cb = stage2_check_permission_walker, .flags = KVM_PGTABLE_WALK_LEAF, - .arg = &attr, + .arg = &data, }; - u64 granule, start, end; - u32 level; + u64 granule, end, start = ALIGN_DOWN(addr, PAGE_SIZE); int ret; - ret = stage2_set_prot_attr(pgt, prot, &attr); + ret = kvm_pgtable_walk(pgt, start, PAGE_SIZE, &check_perm_walker); if (ret) return ret; - attr &= KVM_PTE_LEAF_S2_COMPAT_MASK; - for (level = pgt->start_level; level < KVM_PGTABLE_MAX_LEVELS; level++) { - granule = kvm_granule_size(level); + do { + granule = kvm_granule_size(data.level); start = ALIGN_DOWN(addr, granule); end = start + granule; + data.level++; + } while ((data.level < KVM_PGTABLE_MAX_LEVELS) && + (!kvm_level_supports_block_mapping(data.level) || + start < range->start || range->end < end)); - if (!kvm_level_supports_block_mapping(level)) - continue; - - if (start < range->start || range->end < end) - continue; + range->start = start; + range->end = end; - /* - * Check the presence of existing mappings with incompatible - * permissions within the current block range, and try one level - * deeper if one is found. - */ - ret = kvm_pgtable_walk(pgt, start, granule, &check_perm_walker); - if (ret != -EEXIST) - break; - } - - if (!ret) { - range->start = start; - range->end = end; - } - - return ret; + return 0; } From patchwork Mon Jul 19 10:47:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1EFDC07E9B for ; Mon, 19 Jul 2021 10:50:52 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72C856101E for ; Mon, 19 Jul 2021 10:50:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72C856101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9ei4mdxtIxJfCmuEiYKiGB7uz0Wjp/pA0MKsrPsf9RE=; b=q/saEMVrsloIoKbx8P7sjCp9yT fSv2SMWMz9AmmzYHfoOzbcT0vXAWWpzZCfKTpbNdL2BVtvl6K+SynNIxt67BhaHZpM6ZSFmsF1W4u V8JErOgGVgD1aLroSuLkhGdoonv+Gctl9oGdIg8C2z37v7Lq/qez3/A36ZoqAkAqOpG2SlLAgNvJd JYp05b75Vh1KU448V3NLis/NmK0vVyQnGwvoPSbHxllPNrLpGTNXTWM9ebO2gllDo1/1pMy0p2JBF ANOHu2ZdpepOeibzmEzNPS0IMCgO/qhcLCMiPvgD73tTJHFGuyDaczfcfTKurL5nCrpU/qkl532b2 zJe7dmsQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qpj-009Kys-C5; Mon, 19 Jul 2021 10:49:11 +0000 Received: from mail-qv1-xf4a.google.com ([2607:f8b0:4864:20::f4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoQ-009KTB-2P for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:52 +0000 Received: by mail-qv1-xf4a.google.com with SMTP id h13-20020a0cedad0000b0290310fecb5f78so3097597qvr.0 for ; Mon, 19 Jul 2021 03:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lb/gYwEJ52V8NJMW9ljOtVfH6PwbyXsZiRh6U1L061M=; b=dVcGRuTeuLeRZV2QwASmH66frNnilIZiSJsAaKb1s76JoXlM4AtD3npXiYQzxt28XK WRmijrISMsClSbfD/j1ZqOtq+NAIkXrS2vs0tgWk2FNH+4EGuiPN0pghsbdxN4MoQF5k dIL201YwNFekTdUovrgHjn/z4+PZxwtyG77qNXlJ+DbJY5BfxZyGA38C7WIW2DXBG+c7 cAVEg+Vw0+L5Ce2pVQUm3xmwl4XlB7do98+Wr4I207ZGws9WsRZZ3sWM5nrre3xUZJ+Q 0GH6Old6+be+vA+5Ga601BkY28wqGSPU4phFvCk5iEc0GkkR2DcpFMBULOEnVKIW5OcC zJew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lb/gYwEJ52V8NJMW9ljOtVfH6PwbyXsZiRh6U1L061M=; b=gSn42nkv4Gkd6yYTiIw3js/TOJf7dlx3dKdfBzo2Ye/7noJJgh3mhIPgy0Scsg2ZhZ TzZ+ZiUtCvbVpOf1DSYue99m/xeZezHp7j/Sr1Wkcc+2UneLS8o4JzpTrVG5Q30ddCH2 knL0Z+RWulDDOkDuPwpuS9vxtOhDLEu4uXNLuCiPb7jilCTdKHrQ+wLXQZh/ZGvOHO/W 85hcvLRX6hXsJ/VCndRSbp27JTqJz0DTt5CAZM7dODHibt0pg9fQBocV1kM62LV8L7D5 Ylv9U91jkFrY2d7DSRZI0PTdm6psP7JM7KkYk/HGbKQnYdXyX9Up8Oscu1Vs9D33IwGa 9jQQ== X-Gm-Message-State: AOAM533/ghhGOGIvKxQJe+ITd6E/ovflAWcJ3XF75VKaXyVJSvlTLrmZ xSUAXwzrCMs9GdCnHAuwsFJoCecJ5ocJ X-Google-Smtp-Source: ABdhPJytH3lHIJjCHy/P5Ydbwldz1i5qqlyWptr5/IHXQucIC5VXiU31XNnB+3cuKdHQC5Bm27x6uAI37Z5o X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a05:6214:7e9:: with SMTP id bp9mr24508149qvb.4.1626691668039; Mon, 19 Jul 2021 03:47:48 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:24 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-4-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 03/14] KVM: arm64: Continue stage-2 map when re-creating mappings From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret , Yanan Wang X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034750_204536_015D2246 X-CRM114-Status: GOOD ( 20.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The stage-2 map walkers currently return -EAGAIN when re-creating identical mappings or only changing access permissions. This allows to optimize mapping pages for concurrent (v)CPUs faulting on the same page. While this works as expected when touching one page-table leaf at a time, this can lead to difficult situations when mapping larger ranges. Indeed, a large map operation can fail in the middle if an existing mapping is found in the range, even if it has compatible attributes, hence leaving only half of the range mapped. To avoid having to deal with such failures in the caller, don't interrupt the map operation when hitting existing PTEs, but make sure to still return -EAGAIN so that user_mem_abort() can mark the page dirty when needed. Cc: Yanan Wang Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 2 +- arch/arm64/kvm/hyp/pgtable.c | 21 +++++++++++++++++---- 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index d6649352c8b3..af62203d2f7a 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -258,7 +258,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt); * If device attributes are not explicitly requested in @prot, then the * mapping will be normal, cacheable. * - * Note that the update of a valid leaf PTE in this function will be aborted, + * Note that the update of a valid leaf PTE in this function will be skipped, * if it's trying to recreate the exact same mapping or only change the access * permissions. Instead, the vCPU will exit one more time from guest if still * needed and then go through the path of relaxing permissions. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 978f341d02ca..bb73c5331b7c 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -475,6 +475,8 @@ struct stage2_map_data { void *memcache; struct kvm_pgtable_mm_ops *mm_ops; + + int ret; }; u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) @@ -612,8 +614,10 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, * the vCPU will exit one more time from guest if still needed * and then go through the path of relaxing permissions. */ - if (!stage2_pte_needs_update(old, new)) - return -EAGAIN; + if (!stage2_pte_needs_update(old, new)) { + data->ret = -EAGAIN; + goto out; + } stage2_put_pte(ptep, data->mmu, addr, level, mm_ops); } @@ -629,6 +633,7 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, smp_store_release(ptep, new); if (stage2_pte_is_counted(new)) mm_ops->get_page(ptep); +out: if (kvm_phys_is_valid(phys)) data->phys += granule; return 0; @@ -771,6 +776,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .mmu = pgt->mmu, .memcache = mc, .mm_ops = pgt->mm_ops, + .ret = 0, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -789,7 +795,10 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, ret = kvm_pgtable_walk(pgt, addr, size, &walker); dsb(ishst); - return ret; + if (ret) + return ret; + + return map_data.ret; } int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, @@ -802,6 +811,7 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, .memcache = mc, .mm_ops = pgt->mm_ops, .owner_id = owner_id, + .ret = 0, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -815,7 +825,10 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, return -EINVAL; ret = kvm_pgtable_walk(pgt, addr, size, &walker); - return ret; + if (ret) + return ret; + + return map_data.ret; } static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, From patchwork Mon Jul 19 10:47:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78898C07E9D for ; Mon, 19 Jul 2021 10:51:09 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 44F9B61006 for ; Mon, 19 Jul 2021 10:51:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44F9B61006 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tAklCkJZ3Q9M2AO9JgPryptKW1KAVZutsxAKKQWWv+0=; b=0DAk/mFCcxhKBlzdQmJYGQSgMG ftWa0Twh7BlBPvXVAMwn1vTtBgbRMPr7mW33YjCmmCpS83h+Jzz9ViRW9r1UpJ9noFHKcu1rEjeAL y2jEHUMHKq9+2KRhRc7Xi091GE8WyzdNUG/zR9ekj02YEffdkMhNqfBhrOUb/w2WwlDj+YVVgRP9G 2tJc0gNBO5Vve7blbokjlSlrSesKYSYDoe+/9/f/zltiJDHb5y0q0EFG51FkJpaCEDVd0qYPUk8nz tkAYgGm+jwNxyG4nwoWT7bkdrEnPx4SNu5d8u0jTSrc9BWvN6dBvSyzr7+ewVKJF9Z1UaLHpz0MGo Jp2YAQPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QqD-009LDT-V0; Mon, 19 Jul 2021 10:49:42 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoS-009KUK-OM for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:54 +0000 Received: by mail-qt1-x84a.google.com with SMTP id l19-20020a05622a1753b0290269bd8044e1so287521qtk.10 for ; Mon, 19 Jul 2021 03:47:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hF3abQWZBVX3OcEZC1+muydZc+hxmzmuOXsHVIQGXQY=; b=bKDVheVY1IoxMnY0wSKdcwzVYL3+9RUBExVTBuQLvOyjLLO3PPt0/L9UEBmEHL+1Dx 4GCQT5nMdj7k5BCms5Cp4cMXS2yqRf69jgSSFeaoSz1jZ8avNGOl6b181XVKQ4cCFaUh 2EXb6bfTh+tcgwI2a383qfuwgtr2+1sZO2XBFsp8DH6NTZhuD0ICB8n4FNcbW7Hlzrda YHTa8rkvihWr9onnhlMHiBLIYzudPGEcwL/Mb87H+EpZpOzDbgM0KKrePPGhBHLoHkMU i2BYhTw5ZbitjRvVXlW5OqXmbQ5tYM1XNWQVTNkVnIgSDNNNia7XPNcXQe/0SlM0ppTf GepA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hF3abQWZBVX3OcEZC1+muydZc+hxmzmuOXsHVIQGXQY=; b=fWftR2cm358tsUXKpqxZw7jc2z7DpX559B00140HX5YEYtytBOJZEqBXD8v/pHPi9s ASu0gSA5/QP80TOX9B4VOKKHPmIO5XAV6slnRRwXR9lDyKMdC5CD+5AJajlYnrNuDnRK 7mzr62D6kp2iMfLRvVKbRkq7Qlq5PnCAvCMlx4Tifi3cPFYApyRR4hV3XA3sZutMb28L t7akzJd7b9vrrpVDFCY8KrUxtWhFl2pr71W+x20us35a0iFJeYrlSoR8oJ0Dfk3dC3lQ 4MNl+Ede4f01F8Vr2+djnepFheDFB1LmMjPsOovS8PhW3SrT6XMK8K+W4aI5srEz5jEU SpJg== X-Gm-Message-State: AOAM533qlaznG95lGV8jKPggCpBWM5PhxxUYCv2oMLGmsl8IGakKPPvl 9RpHiDl0HLrfTjjeYXcw2vJqVjNpyEkV X-Google-Smtp-Source: ABdhPJywsBug5lNdL9XgQmCske7yV74xcWDxC3HLTB31TuI7A3Ec/12WOGGcTHohteYdwrcfEA0JudPZ10me X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a05:6214:1021:: with SMTP id k1mr24356756qvr.4.1626691670169; Mon, 19 Jul 2021 03:47:50 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:25 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-5-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 04/14] KVM: arm64: Rename KVM_PTE_LEAF_ATTR_S2_IGNORED From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034752_829162_E4897EB8 X-CRM114-Status: GOOD ( 11.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The ignored bits for both stage-1 and stage-2 page and block descriptors are in [55:58], so rename KVM_PTE_LEAF_ATTR_S2_IGNORED to make it applicable to both. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index bb73c5331b7c..a60653cbd8e5 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -40,6 +40,8 @@ #define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51) +#define KVM_PTE_LEAF_ATTR_IGNORED GENMASK(58, 55) + #define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54) #define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54) @@ -48,8 +50,6 @@ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) -#define KVM_PTE_LEAF_ATTR_S2_IGNORED GENMASK(58, 55) - #define KVM_INVALID_PTE_OWNER_MASK GENMASK(63, 56) #define KVM_MAX_OWNER_ID 1 From patchwork Mon Jul 19 10:47:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1639C07E9B for ; Mon, 19 Jul 2021 10:51:44 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BDC006101E for ; Mon, 19 Jul 2021 10:51:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BDC006101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+sj2VOvAIrlGctEwKva+nzb1htIH44/u/Jiw75E4AM4=; b=ZYFYHYbO916EZg1Lry4B8i6Bhq kGuk/4HTafC95GCRDW+3ed2MKWhk2zuo/h0/3DdN17qydvgne6uIAdgSV2iuxWv6rVKj+jw7WhhWZ r6knF1d7l9o5x+vdYkHmTvcxYjbrEJ0yuUtbVhC0G4UDjVN6xYggg/14fjrendm5lvsiOn/NfMjON WD0Q9+yTsghtEjPEfzayijyzbkE0Ebsp6Ke1j1Zu7WEkES/2oxtGF52qTv2O/lxjFlOT2i6A7wt3v d2UBQkwk+H3gSPWX46zjCJoNvUFiz55uGcc28IOJjwmiMW6/dRSV9GBjLq2cf11xcEsqOlr74zTI4 H7i7u+iQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qqf-009LQT-AV; Mon, 19 Jul 2021 10:50:09 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoU-009KVS-9L for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:55 +0000 Received: by mail-qt1-x84a.google.com with SMTP id j11-20020ac8664b0000b029026549e62339so1472578qtp.1 for ; Mon, 19 Jul 2021 03:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9b2DYjQT7/g/R4ArAtIC/jYlBeWl0FTuTlWq0insfwk=; b=PqGcmBO2wTNT/PQffInicuYzE+ddpQibVhLtljlpqJEXnf4+QFhSeP67SvzmIU5KQS 9Y8K7fKXU8Yc2JMPC9ZkvCioxx6Ssjg4gKGnar073enL+Qbu9mfXxey6Dr18AIdn2GyM Xcnr9YSul/mEPJCH8qfFwsaTQpnmJsqjqkfMVu5HSANq7OT62XP9gVzuUPOKP0g0bz56 VQvMF7nmoLyBRZ7zawRL2TtSoSiM5+3X3gxQFB8vZtyLtEjeCydgiiPljiynVaYzAwX/ lEJosv5Ud8X0u1ZfKviLaU84GaSDjbTS3GbiU1t4IlJ14sXx5GQjYGyJZQu2fpJGo0wu QkQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9b2DYjQT7/g/R4ArAtIC/jYlBeWl0FTuTlWq0insfwk=; b=nkh5WJmsMdSURcVbt2g8e7kiXdQ3HMlLq4MZA3WdJLehtqNiIUIsxvOtqkrZmQ/Y1n oqyPil9OGtuZYjgCakCsXykhP7UuebpMurxM4Nnq5VG9OFR0q8HSxaDCpLLdlBINALvx TRRsNS/Ln9FXAZHDo5ft4h2rV4smylqC4GCZLpMd2WSmyN9I2A/yko8RE9fOXm1r6nT1 4onvHaunW4UptA6hSumOlPZE6+VTJCpt2ETCtFcB6aToFSOWTRwiVhjkDLehxxI5iftp f+oD53jWfV7eey4+9JW+IOeHOPti36JTp+bbRjmW7m0b6sJkIz6db48fibVYF9CdCObD mbJQ== X-Gm-Message-State: AOAM5335qka5gj+VpdmHfcI+8zH8jlnN1AxNOeUm5q7S7TN46QDaFdfN eyXFhm50GAuMH0U83jJFYqm/sub7QbW3 X-Google-Smtp-Source: ABdhPJxrGQcU/bELmtAI4DiOZarAZdNIyxfly8cWQksXRDD7AQeTy6wEy0Y1zFBuD/hd8VGZA2WdmEFZdPU1 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a0c:ca0f:: with SMTP id c15mr24195317qvk.58.1626691672637; Mon, 19 Jul 2021 03:47:52 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:26 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-6-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 05/14] KVM: arm64: Don't overwrite ignored bits with owner id From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034754_404660_B3DAB4A2 X-CRM114-Status: GOOD ( 11.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The nVHE protected mode uses invalid mappings in the host stage-2 page-table to track the owner of each page in the system. In order to allow the usage of ignored bits (a.k.a. software bits) in these mappings, move the owner encoding away from the top bits. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index a60653cbd8e5..a0ac8c2bc174 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -50,7 +50,7 @@ KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ KVM_PTE_LEAF_ATTR_HI_S2_XN) -#define KVM_INVALID_PTE_OWNER_MASK GENMASK(63, 56) +#define KVM_INVALID_PTE_OWNER_MASK GENMASK(9, 2) #define KVM_MAX_OWNER_ID 1 struct kvm_pgtable_walk_data { From patchwork Mon Jul 19 10:47:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF0CC07E9D for ; Mon, 19 Jul 2021 10:52:27 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8B8CD61026 for ; Mon, 19 Jul 2021 10:52:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B8CD61026 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Lk8VraPvXsQbz2zGvWVPnJgU8K7j/vpbkoNQja8O9+w=; b=h6EKNUX705nRk0GehCAeVOa0Jd Ri7AjhC9Gtu8qm9gpRVhiCbBzOIw1Sb6ItF5aGWZn2xS+v7Dy2cymNqwyJfUIwEyIx6YvAOcKZdKp fxC9PPi4nPaA8D2Nz5sMoGZjcg6Da9Xvslz4tNMVsswaEPgLbnPM8ucMzU2O8IJ593qzOM+KrOKuR s2ThVKt6m6jQnOG5dZlKLi16DTr6mnpMH54Gx1JF9ZAeT7C2ic90x6Ih+O7NgaTJeugMXw/qJF7nN 92slUL2KlDT5KiVF3r1wPeAzhyPePN7QSmV0/VeYwwB9BtIgCiShFzGQcGzJAtQ2av8jlQvQ7oRbw sJcFc94Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QrC-009LgG-Sw; Mon, 19 Jul 2021 10:50:43 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoX-009KWR-Bb for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:47:58 +0000 Received: by mail-wr1-x449.google.com with SMTP id r18-20020adff1120000b029013e2b4ee624so8656567wro.1 for ; Mon, 19 Jul 2021 03:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4jouoD7WbUci7xn3YtWW0XHxA5H2d8wOqC0fOlAh9f8=; b=Uv/xPNZZd0zvHFvxq2GpDnhKRoZfKfkYeT7JrfEvu+elD14GHfRJV2UfPlcuLH0Nv9 xGP4lSDAuHT744H30Nr3FuQqtA7zcfsuBVg1gpqjrcW0+SAKgbz3XsLytXwgeRD38gsy 5U18heYUtdx1QOOER3ojuOyNMtIncmDBZRxyoXBEp12SJ5nq8mYIRIOoWUAxkLpxFWf/ 0MCRFOXhNv1Q4vAH+jiL5e8tyam8wKhtdPh0pvlTNUJuICxDUt4OlVPGpz2JloawDALr robsnCnc2EuFshx4qttMo42h1bgE0+RRFB1TJsGSde7AB8wG/OeSkfs/zeHYYE7cARWQ ybpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4jouoD7WbUci7xn3YtWW0XHxA5H2d8wOqC0fOlAh9f8=; b=n7zGe6Z3AGxS9Keye1BqTfzjV+EpGq0eTNSkMYsBOIpF0cDbvk/1kJ3Jr2uwPsxeSM H0X0jZh5TkeJ//W34eIz5btA3ztElfCqDDy+UGVvJlhTqAtz3jssooQJKM/g8TAtssKW zV28vewaGxt9fnsCuJxfv++mxvO6O3VkZn+htUARUNopIKrU7BioQT08xbRyrkbCNB8C OtNX7SdtKI/Po459vpLLAolAtOi649VvMcXZZWfsNVhtlCOjsHgI+sgokCdowzCFt1E5 eoqNLAY/Row5SBCo8o52CDwBnos0xlBiVq2spYHDyVI6rDpeUf0OkP6WNJi/ekiHnjpk 791g== X-Gm-Message-State: AOAM530upqWtiU5rjbDtib+MkJmh/0FsefEf8wgxCSovLnI23k6kBwQd QwhfKVS3GjUDR/xGZOcoXlWPJsj8bWy1 X-Google-Smtp-Source: ABdhPJy/15n8M0Ndba/dPTUyDBcKgkjR9P3nSTL+iqL9mRoICMhdiGix+x/jfDLh68tcSqBfBCw3QGLbUiX4 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a1c:1bd0:: with SMTP id b199mr25225892wmb.108.1626691674895; Mon, 19 Jul 2021 03:47:54 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:27 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-7-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 06/14] KVM: arm64: Tolerate re-creating hyp mappings to set ignored bits From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034757_465024_AD1847D8 X-CRM114-Status: GOOD ( 11.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current hypervisor stage-1 mapping code doesn't allow changing an existing valid mapping. Relax this condition by allowing changes that only target ignored bits, as that will soon be needed to annotate shared pages. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/pgtable.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index a0ac8c2bc174..34cf67997a82 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -362,6 +362,17 @@ static int hyp_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep) return 0; } +static bool hyp_pte_needs_update(kvm_pte_t old, kvm_pte_t new) +{ + if (old == new) + return false; + + if (!kvm_pte_valid(old)) + return true; + + return !WARN_ON((old ^ new) & ~KVM_PTE_LEAF_ATTR_IGNORED); +} + static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, struct hyp_map_data *data) { @@ -371,9 +382,12 @@ static bool hyp_map_walker_try_leaf(u64 addr, u64 end, u32 level, if (!kvm_block_mapping_supported(addr, end, phys, level)) return false; - /* Tolerate KVM recreating the exact same mapping */ + /* + * Tolerate KVM recreating the exact same mapping, or changing ignored + * bits. + */ new = kvm_init_valid_leaf_pte(phys, data->attr, level); - if (old != new && !WARN_ON(kvm_pte_valid(old))) + if (hyp_pte_needs_update(old, new)) smp_store_release(ptep, new); data->phys += granule; From patchwork Mon Jul 19 10:47:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 630C4C07E9B for ; Mon, 19 Jul 2021 10:53:10 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2D1EF6101E for ; Mon, 19 Jul 2021 10:53:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D1EF6101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1O19po99OfgOgElvNmx64k605AJYqZekw05ODsDd3D4=; b=C+lTu95t4tbYk9QKJAU7NxLgA8 8SvfYmO1OcjwUoFSW/TLoEUovFajQY8ltfFGx0/vZkoEuxbZ9qIHEyVtGiVTreKubPhsGzT2sSKcL /eDS8wtaiCHIDWUQ4MmLRumbm3m8i2KeE/uoJtirlrQ1cQ315n4yYa7sz1t8pY+PB7zfTMg/Pp6nd ZMzOGk/FBjqADweFdYP5/6cDrIcViM+NdlV1MT6cb7Br/pOGzwz0LnnxA4Z7JnLsx1R+2hRe/gfFi RxFZ9aS/pdyR7J9w2v3P9Xz4vjdR7PCcYp3dn0fkDftVsA4K+c5QmNFYTDTakg6MKz1cnJbAEQaBL yQ/W9ipA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qrs-009M0D-Cw; Mon, 19 Jul 2021 10:51:26 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QoY-009KXF-SP for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:00 +0000 Received: by mail-qv1-xf49.google.com with SMTP id d17-20020a0cfe910000b02902e01fc8ee77so14777278qvs.21 for ; Mon, 19 Jul 2021 03:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xQPZWURbGoPTKgOKUpwXSewirNhESBxNhrKKCWt7434=; b=L8RdXw4ttae5Vs2K7NINOKLZoBbdFGwfjXFWn8bkET0oJOKxBfwAPfw/a3CCFAChd3 rZ/mUyHzvmIR3hJe+r7LrLjJt56aB+o/eRqcS+1BSFUEs5hR1ChiyAxlcdn2Zz7cHnLt 5J4W9Oqi9c3E6GF4M6WDvGahS3AwZti3tVxlOF8inYB3t//iiG9q4OHpoPpJgF4pxAZ7 oGL8UyjLVBTJSTFqtMlARZktzU1Hwr6mrMMHZXqXSR75VYbVotw0Yfe5bjfFx4t8YYHa c19lmbnAmZIW+93r6iI3/FvOpyk5tTSdKVwTvq6eqA8GF1nxCYVLrPThZ2bTv9UbNbHu 7LPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xQPZWURbGoPTKgOKUpwXSewirNhESBxNhrKKCWt7434=; b=bEupp7UCTF+MO0rfTBn6rX5RrhY2mbRe9hE5SEthGTTcjy0o3h7p0wO11kEoV1s6W1 MD1+ABip9YX690Z+8bLGkqBU/47K+JD4tIjDmev1VZ5yFoEfdpl5cl4M8BQfKIX7mxsB 3EzzJ+VjboVLppjSI2BjvMP7cXCIBcrWUEu5IGz3VVqwpgP9TAWJOLvJfAKSgwQ4iBOI avc4YbQzp148z5KRGIab+ijoRadgo60/zMRNFZ2Szz0Uwu/6Mh4VOIU0Ac7okeF9RBvR PyNGhwz0mtIA19kJsRQkO2WNYoS02SK7Aeuuk5C/FyGl0JI8IjHDtwZSCd5g7sMUgQPb vwlA== X-Gm-Message-State: AOAM532FAYKHT6tqBM5MxnJaQLspb759m6keYc0B7jVaX5eUZcoMQBL1 NKtERDU3EIwS4B0gLVNjIPXFFatVXxBp X-Google-Smtp-Source: ABdhPJyrkesxT86s4Tj/djq2B3CEsycyHyzOhzkaxyjOrOLuPXgOsyycOyHK+NTGbCGeSB4g1fJRBUf3HDlq X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a0c:9ac7:: with SMTP id k7mr24138746qvf.49.1626691677038; Mon, 19 Jul 2021 03:47:57 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:28 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-8-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 07/14] KVM: arm64: Enable forcing page-level stage-2 mappings From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034759_061132_8C0ED76C X-CRM114-Status: GOOD ( 23.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Much of the stage-2 manipulation logic relies on being able to destroy block mappings if e.g. installing a smaller mapping in the range. The rationale for this behaviour is that stage-2 mappings can always be re-created lazily. However, this gets more complicated when the stage-2 page-table is used to store metadata about the underlying pages. In such a case, destroying a block mapping may lead to losing part of the state, and confuse the user of those metadata (such as the hypervisor in nVHE protected mode). To fix this, introduce a callback function in the pgtable struct which is called during all map operations to determine whether the mappings can us blocks, or should be forced to page-granularity level. This is used by the hypervisor when creating the host stage-2 to force page-level mappings when using non-default protection attributes. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 63 +++++++++++++++++---------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 16 +++++-- arch/arm64/kvm/hyp/pgtable.c | 20 +++++++-- 3 files changed, 69 insertions(+), 30 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index af62203d2f7a..dd72653314c7 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -75,25 +75,6 @@ enum kvm_pgtable_stage2_flags { KVM_PGTABLE_S2_IDMAP = BIT(1), }; -/** - * struct kvm_pgtable - KVM page-table. - * @ia_bits: Maximum input address size, in bits. - * @start_level: Level at which the page-table walk starts. - * @pgd: Pointer to the first top-level entry of the page-table. - * @mm_ops: Memory management callbacks. - * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. - */ -struct kvm_pgtable { - u32 ia_bits; - u32 start_level; - kvm_pte_t *pgd; - struct kvm_pgtable_mm_ops *mm_ops; - - /* Stage-2 only */ - struct kvm_s2_mmu *mmu; - enum kvm_pgtable_stage2_flags flags; -}; - /** * enum kvm_pgtable_prot - Page-table permissions and attributes. * @KVM_PGTABLE_PROT_X: Execute permission. @@ -109,11 +90,41 @@ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_DEVICE = BIT(3), }; -#define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W) +#define KVM_PGTABLE_PROT_RW (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W) +#define KVM_PGTABLE_PROT_RWX (KVM_PGTABLE_PROT_RW | KVM_PGTABLE_PROT_X) + +#define PAGE_HYP KVM_PGTABLE_PROT_RW #define PAGE_HYP_EXEC (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_X) #define PAGE_HYP_RO (KVM_PGTABLE_PROT_R) #define PAGE_HYP_DEVICE (PAGE_HYP | KVM_PGTABLE_PROT_DEVICE) +typedef bool (*kvm_pgtable_want_pte_cb_t)(u64 addr, u64 end, + enum kvm_pgtable_prot prot); + +/** + * struct kvm_pgtable - KVM page-table. + * @ia_bits: Maximum input address size, in bits. + * @start_level: Level at which the page-table walk starts. + * @pgd: Pointer to the first top-level entry of the page-table. + * @mm_ops: Memory management callbacks. + * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. + * @flags: Stage-2 page-table flags. + * @want_pte_cb: Callback function used during map operations to decide + * whether block mappings can be used to map the given IPA + * range. + */ +struct kvm_pgtable { + u32 ia_bits; + u32 start_level; + kvm_pte_t *pgd; + struct kvm_pgtable_mm_ops *mm_ops; + + /* Stage-2 only */ + struct kvm_s2_mmu *mmu; + enum kvm_pgtable_stage2_flags flags; + kvm_pgtable_want_pte_cb_t want_pte_cb; +}; + /** * struct kvm_mem_range - Range of Intermediate Physical Addresses * @start: Start of the range. @@ -216,21 +227,25 @@ int kvm_pgtable_hyp_map(struct kvm_pgtable *pgt, u64 addr, u64 size, u64 phys, u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift); /** - * kvm_pgtable_stage2_init_flags() - Initialise a guest stage-2 page-table. + * kvm_pgtable_stage2_init_full() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. * @arch: Arch-specific KVM structure representing the guest virtual * machine. * @mm_ops: Memory management callbacks. * @flags: Stage-2 configuration flags. + * @want_pte_cb: Callback function used during map operations to decide + * whether block mappings can be used to map the given IPA + * range. * * Return: 0 on success, negative error code on failure. */ -int kvm_pgtable_stage2_init_flags(struct kvm_pgtable *pgt, struct kvm_arch *arch, +int kvm_pgtable_stage2_init_full(struct kvm_pgtable *pgt, struct kvm_arch *arch, struct kvm_pgtable_mm_ops *mm_ops, - enum kvm_pgtable_stage2_flags flags); + enum kvm_pgtable_stage2_flags flags, + kvm_pgtable_want_pte_cb_t want_pte_cb); #define kvm_pgtable_stage2_init(pgt, arch, mm_ops) \ - kvm_pgtable_stage2_init_flags(pgt, arch, mm_ops, 0) + kvm_pgtable_stage2_init_full(pgt, arch, mm_ops, 0, NULL) /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 58edc62be6f7..cdace80d3e28 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -89,6 +89,7 @@ static void prepare_host_vtcr(void) id_aa64mmfr1_el1_sys_val, phys_shift); } +static bool host_stage2_want_pte_cb(u64 addr, u64 end, enum kvm_pgtable_prot prot); int kvm_host_prepare_stage2(void *pgt_pool_base) { struct kvm_s2_mmu *mmu = &host_kvm.arch.mmu; @@ -101,8 +102,9 @@ int kvm_host_prepare_stage2(void *pgt_pool_base) if (ret) return ret; - ret = kvm_pgtable_stage2_init_flags(&host_kvm.pgt, &host_kvm.arch, - &host_kvm.mm_ops, KVM_HOST_S2_FLAGS); + ret = kvm_pgtable_stage2_init_full(&host_kvm.pgt, &host_kvm.arch, + &host_kvm.mm_ops, KVM_HOST_S2_FLAGS, + host_stage2_want_pte_cb); if (ret) return ret; @@ -225,9 +227,17 @@ static inline int __host_stage2_idmap(u64 start, u64 end, __ret; \ }) +static bool host_stage2_want_pte_cb(u64 addr, u64 end, enum kvm_pgtable_prot prot) +{ + if (range_is_memory(addr, end)) + return prot != KVM_PGTABLE_PROT_RWX; + else + return prot != KVM_PGTABLE_PROT_RW; +} + static int host_stage2_idmap(u64 addr) { - enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W; + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_RW; struct kvm_mem_range range; bool is_memory = find_mem_range(addr, &range); int ret; diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 34cf67997a82..5bdbe7a31551 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -452,6 +452,8 @@ int kvm_pgtable_hyp_init(struct kvm_pgtable *pgt, u32 va_bits, pgt->start_level = KVM_PGTABLE_MAX_LEVELS - levels; pgt->mm_ops = mm_ops; pgt->mmu = NULL; + pgt->want_pte_cb = NULL; + return 0; } @@ -491,6 +493,7 @@ struct stage2_map_data { struct kvm_pgtable_mm_ops *mm_ops; int ret; + bool force_pte; }; u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift) @@ -613,6 +616,9 @@ static int stage2_map_walker_try_leaf(u64 addr, u64 end, u32 level, struct kvm_pgtable *pgt = data->mmu->pgt; struct kvm_pgtable_mm_ops *mm_ops = data->mm_ops; + if (data->force_pte && (level < (KVM_PGTABLE_MAX_LEVELS - 1))) + return -E2BIG; + if (!kvm_block_mapping_supported(addr, end, phys, level)) return -E2BIG; @@ -660,6 +666,9 @@ static int stage2_map_walk_table_pre(u64 addr, u64 end, u32 level, if (data->anchor) return 0; + if (data->force_pte && (level < (KVM_PGTABLE_MAX_LEVELS - 1))) + return 0; + if (!kvm_block_mapping_supported(addr, end, data->phys, level)) return 0; @@ -791,6 +800,7 @@ int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, .memcache = mc, .mm_ops = pgt->mm_ops, .ret = 0, + .force_pte = pgt->want_pte_cb && pgt->want_pte_cb(addr, addr + size, prot), }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -826,6 +836,7 @@ int kvm_pgtable_stage2_set_owner(struct kvm_pgtable *pgt, u64 addr, u64 size, .mm_ops = pgt->mm_ops, .owner_id = owner_id, .ret = 0, + .force_pte = true, }; struct kvm_pgtable_walker walker = { .cb = stage2_map_walker, @@ -1070,9 +1081,11 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) return kvm_pgtable_walk(pgt, addr, size, &walker); } -int kvm_pgtable_stage2_init_flags(struct kvm_pgtable *pgt, struct kvm_arch *arch, - struct kvm_pgtable_mm_ops *mm_ops, - enum kvm_pgtable_stage2_flags flags) + +int kvm_pgtable_stage2_init_full(struct kvm_pgtable *pgt, struct kvm_arch *arch, + struct kvm_pgtable_mm_ops *mm_ops, + enum kvm_pgtable_stage2_flags flags, + kvm_pgtable_want_pte_cb_t want_pte_cb) { size_t pgd_sz; u64 vtcr = arch->vtcr; @@ -1090,6 +1103,7 @@ int kvm_pgtable_stage2_init_flags(struct kvm_pgtable *pgt, struct kvm_arch *arch pgt->mm_ops = mm_ops; pgt->mmu = &arch->mmu; pgt->flags = flags; + pgt->want_pte_cb = want_pte_cb; /* Ensure zeroed PGD pages are visible to the hardware walker */ dsb(ishst); From patchwork Mon Jul 19 10:47:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4DC5C07E9B for ; Mon, 19 Jul 2021 10:54:50 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8221861006 for ; Mon, 19 Jul 2021 10:54:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8221861006 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EuYCn3WG8ZG+R33NA2iQswwqcCSyFOnoeBDG02tE3BM=; b=XGHlco0etMPL7f8LAd8K0SmY6x KVtX9aebfJc6fWBK8sjCtsHW7xg1qFXS9uP18nH6fzpP5fXd8QsEWQhqZXpvaWKYb2EczMQaaCzo0 pVF5ygIbcAuJGJrWwfc5kOHaPXbT27hymasAt25pDbSMvBH0Vb0eDtrfX1M3MCjBg6M97V1JAFe7M j+R2fy0KL+p+oXMTsl40te/w5WhsXaqpRIVn1jEIhLQQSzOzpdmymATbOA1DZ31JE56hZf5i6razL UrZSVf/hNmhLBcUa/60O9KKsTfmtvdjjoAPPYPCHOCL0LYMmuHUjeyB3py4OVoy6hFd1F73xfkNCY cr4en08Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qsg-009MLp-0S; Mon, 19 Jul 2021 10:52:14 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qob-009KYS-HT for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:03 +0000 Received: by mail-wm1-x349.google.com with SMTP id l18-20020a05600c1d12b02902278758ab90so3803388wms.9 for ; Mon, 19 Jul 2021 03:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jCVZUl7mrKXhOgc92CUwmjy/8GLL8qBZeqvKPdsULBo=; b=Iv6HLotZSYKnqU+lUooERsXuSEAWx3VwKrcAVoV7fvzbOe+NzdG1VEHTwtAWje0rIB FBjobDwaueZrOJExzeI7sdg7O2/F4Y5IOMh136apaQCyUSWAEPqaYj8KLp39ymbgQFzN AyCeCZTZKejh/1vxFoUB+cZ5GImkWfPknL6py9BFxSeKNLh1bEqTIwkkWL6/9QrqiZhx bufScWZMa+3N64Gm/bfzeL1huLY0dIepVZRrRZvKUeSf1CMleCvkwSbfE+lOV7YzkSqj LXRGJqpFmm/qRenArrM8tK3lx0xGKtC867Gs8dXTkq9S3SqJLJMrrTHmRgHEhXRK2f7q F7Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jCVZUl7mrKXhOgc92CUwmjy/8GLL8qBZeqvKPdsULBo=; b=GK9g3uqR+p1lQBTZJ5x4D/gc9CSAizcA02qkSRE1UGRh7ku7KvXlv9gzZAIr1UyJV0 eBrAusT6DCvb41GLtIeTV9cd4uK9Dxf4e5rrS4Z8FtSobe5xVqEc+MYN7GM3gr0zswK3 wpvzywjk8IVkdFa/dRAeP2VKJ0mKRQBA84AaMIzEEuX0FERosJ5JnSqLLCRyXVT6aIG3 5IkT28eyk9q0VYc6rohxKHQEvf9ENJkiHvSaR/6pXWCDSFN1islnld1HL31aKUEjoHBd qrzFPXEETB7kcxBUyS6GkUUaccWLfPDSroAORubP5r0qDps04ORJ82f44U8LJaf4wYpQ 7keg== X-Gm-Message-State: AOAM532c6kYDxs83Ib77kphsIxwPgxlbRuWikUx1DkOxBKel/xLfrVTl jVvs/uk8a6R51rv+V8L57Dv5WI45rA8J X-Google-Smtp-Source: ABdhPJzWhrN0PU3u2TAQJIPoHRtvJQ+RMyNfI8W8ip/YuHiS+H6N8RNCyCq0QLT9GDqXoBSdi1WyWB01xUsv X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a05:600c:287:: with SMTP id 7mr25592126wmk.1.1626691679361; Mon, 19 Jul 2021 03:47:59 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:29 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-9-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 08/14] KVM: arm64: Add support for tagging shared pages in page-table From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034801_672508_34579C0B X-CRM114-Status: GOOD ( 16.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hypervisor will soon be in charge of tracking ownership of all memory pages in the system. The current page-tracking infrastructure at EL2 only allows binary states: a page is either owned or not by an entity. But a number of use-cases will require more complex states for pages that are shared between two entities (host, hypervisor, or guests). In preparation for supporting these use-cases, introduce in the KVM page-table library some infrastructure allowing to tag shared pages using ignored bits (a.k.a. software bits) in PTEs. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 5 +++++ arch/arm64/kvm/hyp/pgtable.c | 25 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index dd72653314c7..f6d3d5c8910d 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -81,6 +81,8 @@ enum kvm_pgtable_stage2_flags { * @KVM_PGTABLE_PROT_W: Write permission. * @KVM_PGTABLE_PROT_R: Read permission. * @KVM_PGTABLE_PROT_DEVICE: Device attributes. + * @KVM_PGTABLE_STATE_SHARED: Page shared with another entity. + * @KVM_PGTABLE_STATE_BORROWED: Page borrowed from another entity. */ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_X = BIT(0), @@ -88,6 +90,9 @@ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_R = BIT(2), KVM_PGTABLE_PROT_DEVICE = BIT(3), + + KVM_PGTABLE_STATE_SHARED = BIT(4), + KVM_PGTABLE_STATE_BORROWED = BIT(5), }; #define KVM_PGTABLE_PROT_RW (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 5bdbe7a31551..51598b79dafc 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -211,6 +211,29 @@ static kvm_pte_t kvm_init_invalid_leaf_owner(u8 owner_id) return FIELD_PREP(KVM_INVALID_PTE_OWNER_MASK, owner_id); } +static kvm_pte_t pte_ignored_bit_prot(enum kvm_pgtable_prot prot) +{ + kvm_pte_t ignored_bits = 0; + + /* + * Ignored bits 0 and 1 are reserved to track the memory ownership + * state of each page: + * 00: The page is owned solely by the page-table owner. + * 01: The page is owned by the page-table owner, but is shared + * with another entity. + * 10: The page is shared with, but not owned by the page-table owner. + * 11: Reserved for future use (lending). + */ + if (prot & KVM_PGTABLE_STATE_SHARED) { + if (prot & KVM_PGTABLE_STATE_BORROWED) + ignored_bits |= BIT(1); + else + ignored_bits |= BIT(0); + } + + return FIELD_PREP(KVM_PTE_LEAF_ATTR_IGNORED, ignored_bits); +} + static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag) @@ -357,6 +380,7 @@ static int hyp_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep) attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_AP, ap); attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S1_SH, sh); attr |= KVM_PTE_LEAF_ATTR_LO_S1_AF; + attr |= pte_ignored_bit_prot(prot); *ptep = attr; return 0; @@ -558,6 +582,7 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p attr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_LO_S2_SH, sh); attr |= KVM_PTE_LEAF_ATTR_LO_S2_AF; + attr |= pte_ignored_bit_prot(prot); *ptep = attr; return 0; From patchwork Mon Jul 19 10:47:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51F98C07E9B for ; Mon, 19 Jul 2021 10:55:18 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1ED3E600EF for ; Mon, 19 Jul 2021 10:55:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1ED3E600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pxwo4Vhde3TpNaPPY0gTAcc0JOGpBMuPm7LXIvqrZUM=; b=FbQ3jir0vPvkcEi9UTdoDiHyuM pNpa41WXRljBWIbQDJA0tfzCqLjMXK8d2ZsuWa6zBpNJ4Xt7ihpRvBev08gRtHiAk2817qRFu1WNX vasg/w9OvnXeXE9dq/GPiJo0eINQeexEPgJX/DU0ObjTN9V+tZpPZ2nptp022xz+JrzKLOxt3kKHL ZGFiXN6bj14elBmnbEpGWLlglhdLafkaTN4gKcHsYt39LHVE+pF2jtJ1CUWbUaiEoe/0iwY9Wkd9t v3Q6vR0D7BbdLh+k71G+AqT+glGXw35mEzpEEpQyXJw6DVCefHhrylWqS/jD0Y/eInhYyOarZXpeQ /QJSb3Jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QtO-009MhI-RS; Mon, 19 Jul 2021 10:52:59 +0000 Received: from mail-ed1-x54a.google.com ([2a00:1450:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qod-009KZX-Vj for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:05 +0000 Received: by mail-ed1-x54a.google.com with SMTP id f11-20020a0564021e8bb02903b46e290a49so2342053edf.17 for ; Mon, 19 Jul 2021 03:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dNi42ehjO4NUi77Uj+cYvrOU4Ks5G49fj0fQYIq6rpI=; b=Zo9CxzDhUX2cZC2KYawro8tC3zo0gCGU9bsbwig5j0S3OAyeSDfj6KLCvhewOJXss3 ScGYWjjud+BfSjX1bw156hEet1J0T/Zqk+5fp1+CJaOunTaub7sgNqmoVkWv4/0g/rzU ZYfZpco3hJ67xUtZI9pTg9Vo5ayKvZcYbWYX32i7f9viXdeSPFpPtsmRiydpflNHKvaZ xJX3jC/HH6IpfzQrNolC63LcfYbtbE36mpB0gvl0+eobqmpRV69cepN1WIehO6m5AK3p /KS5RCENmYr5NttKLDRGsPdS7odJ5r7HBgi/UC8GW3gSGPXH/MBjxAGstC6Bk+7nxeXY xpPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dNi42ehjO4NUi77Uj+cYvrOU4Ks5G49fj0fQYIq6rpI=; b=Op3J0dwniqFz2DKZUvUWpFCezm8JVHQVul4q564jB/x7OkGJqH6eVzlJ5cObrSUxDJ PGRN47Ew+X59oqZ8CIGQCeDG6twzIvpFlxuiYA3i8iuxZor+znHGgkpdqIjukKW/xnpH 8ucFzc0k2bhrHjSYYpAd/d3lelI+/kVaa9X6WnUFvTOEsBxqM5z1joI962QXGZCOEq7L s01Qi3VgcqeCcAMVJ/E/gVzHHx2su5bGzsgxx2ROHBYtPk15Y1ZVxJ1qby8UGTXJfcNV 1wb1irj/elqAYffkd8xGa5eAgd8hSvalX40TVOaTybRplKm2TNmJMZ5lAN0wsbQRRDVm pWYg== X-Gm-Message-State: AOAM532P+EsBbHsQu81ufsIFpfIdAPmrSFNnDQ14GiUyEh042AIbow8l zca6qlF4L6LiUO+krzByZCIBBIKJHmKD X-Google-Smtp-Source: ABdhPJyAyrN45gMOAlaStcGcdibOpYIOibhoQ9RmoK/yB/n96peiRQ6P/R0p8ef0yM8amQWGs/batNr68Iti X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a17:906:d8da:: with SMTP id re26mr26655364ejb.205.1626691681533; Mon, 19 Jul 2021 03:48:01 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:30 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-10-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 09/14] KVM: arm64: Mark host bss and rodata section as shared From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034804_093002_3362D260 X-CRM114-Status: GOOD ( 20.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As the hypervisor maps the host's .bss and .rodata sections in its stage-1, make sure to tag them as shared in hyp and host page-tables. But since the hypervisor relies on the presence of these mappings, we cannot let the host in complete control of the memory regions -- it must not unshare or donate them to another entity for example. To prevent this, let's transfer the ownership of those ranges to the hypervisor itself, and share the page back with the host. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 7 ++- arch/arm64/kvm/hyp/nvhe/setup.c | 52 ++++++++++++++++--- 3 files changed, 51 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h index 9c227d87c36d..b39047463075 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -23,6 +23,7 @@ extern struct host_kvm host_kvm; int __pkvm_prot_finalize(void); int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end); +int host_stage2_idmap_locked(u64 start, u64 end, enum kvm_pgtable_prot prot); int kvm_host_prepare_stage2(void *pgt_pool_base); void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index cdace80d3e28..6f28edf58407 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -235,6 +235,11 @@ static bool host_stage2_want_pte_cb(u64 addr, u64 end, enum kvm_pgtable_prot pro return prot != KVM_PGTABLE_PROT_RW; } +int host_stage2_idmap_locked(u64 start, u64 end, enum kvm_pgtable_prot prot) +{ + return host_stage2_try(__host_stage2_idmap, start, end, prot); +} + static int host_stage2_idmap(u64 addr) { enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_RW; @@ -250,7 +255,7 @@ static int host_stage2_idmap(u64 addr) if (ret) goto unlock; - ret = host_stage2_try(__host_stage2_idmap, range.start, range.end, prot); + ret = host_stage2_idmap_locked(range.start, range.end, prot); unlock: hyp_spin_unlock(&host_kvm.lock); diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index 0b574d106519..74dce83a6fad 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -83,10 +83,6 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, if (ret) return ret; - ret = pkvm_create_mappings(__start_rodata, __end_rodata, PAGE_HYP_RO); - if (ret) - return ret; - ret = pkvm_create_mappings(__hyp_rodata_start, __hyp_rodata_end, PAGE_HYP_RO); if (ret) return ret; @@ -95,10 +91,6 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, if (ret) return ret; - ret = pkvm_create_mappings(__hyp_bss_end, __bss_stop, PAGE_HYP_RO); - if (ret) - return ret; - ret = pkvm_create_mappings(virt, virt + size, PAGE_HYP); if (ret) return ret; @@ -117,6 +109,25 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, return ret; } + /* + * Map the host's .bss and .rodata sections RO in the hypervisor, but + * transfer the ownerhsip from the host to the hypervisor itself to + * make sure it can't be donated or shared with another entity. + * + * The ownership transtion requires matching changes in the host + * stage-2. This will done later (see finalize_mappings()) once the + * hyp_vmemmap is addressable. + */ + ret = pkvm_create_mappings(__start_rodata, __end_rodata, + PAGE_HYP_RO | KVM_PGTABLE_STATE_SHARED); + if (ret) + return ret; + + ret = pkvm_create_mappings(__hyp_bss_end, __bss_stop, + PAGE_HYP_RO | KVM_PGTABLE_STATE_SHARED); + if (ret) + return ret; + return 0; } @@ -148,6 +159,27 @@ static void hpool_put_page(void *addr) hyp_put_page(&hpool, addr); } +static int finalize_mappings(void) +{ + enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_RWX; + int ret; + + /* + * The host's .bss and .rodata sections are now conceptually owned by + * the hypervisor, so mark them as 'borrowed' in the host stage-2. We + * can safely use host_stage2_idmap_locked() at this point since the + * host stage-2 has not been enabled yet. + */ + prot |= KVM_PGTABLE_STATE_SHARED | KVM_PGTABLE_STATE_BORROWED; + ret = host_stage2_idmap_locked(__hyp_pa(__start_rodata), + __hyp_pa(__end_rodata), prot); + if (ret) + return ret; + + return host_stage2_idmap_locked(__hyp_pa(__hyp_bss_end), + __hyp_pa(__bss_stop), prot); +} + void __noreturn __pkvm_init_finalise(void) { struct kvm_host_data *host_data = this_cpu_ptr(&kvm_host_data); @@ -167,6 +199,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; + ret = finalize_mappings(); + if (ret) + goto out; + pkvm_pgtable_mm_ops = (struct kvm_pgtable_mm_ops) { .zalloc_page = hyp_zalloc_hyp_page, .phys_to_virt = hyp_phys_to_virt, From patchwork Mon Jul 19 10:47:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3940CC07E9B for ; Mon, 19 Jul 2021 10:56:48 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E613F6101E for ; Mon, 19 Jul 2021 10:56:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E613F6101E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=RPi7AFx4ZTCB9g6F71rOPxZD56Er7y9qKcFvkcu/I+M=; b=pwfdu7dKjlh5yuNqKUh9eWkYwr saoUZGMABl8KLpQauWaTLkNp3alun7tCZTHvvIfsWfidWJiwiXcRh+TWdneIoW/G26WU73aQFVYdK 52UNfx9iPyA+wwZobtQPfqn9D53vtEq8KLdKLsxMcLj5UG21qeWZzFGIyZm3Jpff4r2mgfNIot6PW jjZWDAsUtxNxbBGSV+q5az/4o2xgntclFNKaP9EyIXin7QeGlle85qHW3reJ28gEu3VUMGth6GkFX zsHgATH3SelfJnKeK6DOwcpVe/XkQdpSYiC6nbVZu8y/DjVdl3Dgh/2Enbs7krn9DGhtA8EnykalB XXB9Yvig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5QuE-009N1w-JW; Mon, 19 Jul 2021 10:53:51 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qog-009Kaq-4s for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:07 +0000 Received: by mail-wm1-x34a.google.com with SMTP id l18-20020a05600c1d12b02902278758ab90so3803538wms.9 for ; Mon, 19 Jul 2021 03:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CdbaQ8ErF1YpwqlK/5v4bHdri/rt0mg1VAsqwWTr/Tw=; b=aS8jkLkK2wa4INhA28hZzPjEUZaTZKmwenqCsFfJQTdAz0UFabdJsQo6ri3VfTWEFw RGYXatDhYnjA27xu0fDx7wkypb2FYIlxvf85tFl8oYsJq2rHaR4IgkQpa+q4+FwBsWna y/5f6F8ckCQplGSacvD6+H8TPidKSuTLdk1aFl3aPp9BvtHNW1S7YV6XGANf94XI+K92 wG0tzBduwioFtmG8ODCJFZhpzoP2jF27+9hOe1r+E4fg0ujsayMVKVG7rMKfXMgeAsAF CGBfReNZE1y1Up6lw+wxGzVUgnz28Z7y6A7GjkAKvbDoo8/B1qcvnhtx5JDqVDbCH60+ RlAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CdbaQ8ErF1YpwqlK/5v4bHdri/rt0mg1VAsqwWTr/Tw=; b=ZtS12NuBKepvbdHQT/S/urJZAjO0DBNPZNFDlIed/opZHp3+RdN8zDKAR21Eb2mGFD ceJNZPCTuD8+O9b0qgoEt7fJMvgGSeBsKhRxb/+b4LCjT90EYnP7D2ScIUKPhmELMBH7 QYVg1PACMtL65/lZ/cDaddL/R0tKnxwspIzSBivKSjWbUpZMks6Vt6jKRidd1xFGrHde IpSpHSrzlVgmY3wLmt05Yo6j7TjEZ2T9s+u1d4U+DIr1O4iRirm6FKcDimsafINPhkJW eJSrI3nQnMLNOaSVXQXKI33vpG1g7vbm93wNcmpcVI11gZj6SVAzDek1nHWiDpHE8x3K gavg== X-Gm-Message-State: AOAM531ayPQuKQwTS7ELI9OJx7c4x17V/UIrAGMvMl8O43Ft1w1Cnbla EZsBguH3q82hxl/rHV8H7e4R/+dakaan X-Google-Smtp-Source: ABdhPJznScsyUNWBAsIAI2Jzd/ytZ2sHt95anHYybQcNiWFSnjuNiVNyg44CFTl1PSMEfVjtqavG8Cwof0vY X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a7b:ca50:: with SMTP id m16mr31242327wml.140.1626691683897; Mon, 19 Jul 2021 03:48:03 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:31 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-11-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 10/14] KVM: arm64: Enable retrieving protections attributes of PTEs From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034806_263554_595B6920 X-CRM114-Status: GOOD ( 12.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce helper functions in the KVM stage-2 and stage-1 page-table manipulation library allowing to retrieve the enum kvm_pgtable_prot of a PTE. This will be useful to implement custom walkers outside of pgtable.c. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 20 ++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 49 ++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index f6d3d5c8910d..1aa49d6aabb7 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -467,4 +467,24 @@ int kvm_pgtable_walk(struct kvm_pgtable *pgt, u64 addr, u64 size, */ int kvm_pgtable_stage2_find_range(struct kvm_pgtable *pgt, u64 addr, int owner, struct kvm_mem_range *range); + +/** + * kvm_pgtable_stage2_pte_prot() - Retrieve the protection attributes of a + * stage-2 Page-Table Entry. + * @pte: Page-table entry + * + * Return: protection attributes of the page-table entry in the enum + * kvm_pgtable_prot format. + */ +enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte); + +/** + * kvm_pgtable_hyp_pte_prot() - Retrieve the protection attributes of a stage-1 + * Page-Table Entry. + * @pte: Page-table entry + * + * Return: protection attributes of the page-table entry in the enum + * kvm_pgtable_prot format. + */ +enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 51598b79dafc..c7120797404a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -234,6 +234,20 @@ static kvm_pte_t pte_ignored_bit_prot(enum kvm_pgtable_prot prot) return FIELD_PREP(KVM_PTE_LEAF_ATTR_IGNORED, ignored_bits); } +static enum kvm_pgtable_prot pte_read_ignored_bits_prot(kvm_pte_t pte) +{ + enum kvm_pgtable_prot prot = 0; + kvm_pte_t ignored_bits = 0; + + ignored_bits = FIELD_GET(KVM_PTE_LEAF_ATTR_IGNORED, pte); + if (ignored_bits & BIT(1)) + prot |= KVM_PGTABLE_STATE_BORROWED | KVM_PGTABLE_STATE_SHARED; + if (ignored_bits & BIT(0)) + prot |= KVM_PGTABLE_STATE_SHARED; + + return prot; +} + static int kvm_pgtable_visitor_cb(struct kvm_pgtable_walk_data *data, u64 addr, u32 level, kvm_pte_t *ptep, enum kvm_pgtable_walk_flags flag) @@ -386,6 +400,25 @@ static int hyp_set_prot_attr(enum kvm_pgtable_prot prot, kvm_pte_t *ptep) return 0; } +enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte) +{ + enum kvm_pgtable_prot prot = 0; + u32 ap; + + if (!(pte & KVM_PTE_LEAF_ATTR_HI_S1_XN)) + prot |= KVM_PGTABLE_PROT_X; + + ap = FIELD_GET(KVM_PTE_LEAF_ATTR_LO_S1_AP, pte); + if (ap == KVM_PTE_LEAF_ATTR_LO_S1_AP_RO) + prot |= KVM_PGTABLE_PROT_R; + else if (ap == KVM_PTE_LEAF_ATTR_LO_S1_AP_RW) + prot |= KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W; + + prot |= pte_read_ignored_bits_prot(pte); + + return prot; +} + static bool hyp_pte_needs_update(kvm_pte_t old, kvm_pte_t new) { if (old == new) @@ -588,6 +621,22 @@ static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot p return 0; } +enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte) +{ + enum kvm_pgtable_prot prot = 0; + + if (pte & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R) + prot |= KVM_PGTABLE_PROT_R; + if (pte & KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W) + prot |= KVM_PGTABLE_PROT_W; + if (!(pte & KVM_PTE_LEAF_ATTR_HI_S2_XN)) + prot |= KVM_PGTABLE_PROT_X; + + prot |= pte_read_ignored_bits_prot(pte); + + return prot; +} + static bool stage2_pte_needs_update(kvm_pte_t old, kvm_pte_t new) { if (!kvm_pte_valid(old) || !kvm_pte_valid(new)) From patchwork Mon Jul 19 10:47:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04B0AC07E9D for ; Mon, 19 Jul 2021 10:56:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B4E87600EF for ; Mon, 19 Jul 2021 10:56:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4E87600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=LFLEf8v4X+trh+WkHldMrjJdr6AQFXdPgvNRZfCtQQA=; b=Oh9JCvOs8gyNnbamGyVN+MrIsd 1MjZvsMScs62U04W9BOh2b/A7pAXV3aPlcenPe+QmXW8itqvOVA+YZYdNwSpsVoeLbu7XWpGZEocv rcfpwWjxw/aoUQXBa8QZ+pGQUfWAP+pPKcibYaUA3FjcvpyNyUn2zYVnoT6vU2RKJarrETFhErK08 z6oH99OxSadEIOmB5uyxK8BLdGMW1kWdk3bKojRaV+EVAhq92XaqhsJ4BOVg1k0YAudm7o2xJMjMU XMdJScgk1LtRDljR2vJpNqD4ypoKV88vsFVpum+3/fziLjg+TpiVlkvOt4uVZG0jbtHhmqNGQmW1Q CdW4z/YA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Quw-009NJG-SQ; Mon, 19 Jul 2021 10:54:35 +0000 Received: from mail-qk1-x74a.google.com ([2607:f8b0:4864:20::74a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qoi-009Kbp-4R for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:09 +0000 Received: by mail-qk1-x74a.google.com with SMTP id l200-20020a37a2d10000b02903b9207abc7bso719974qke.4 for ; Mon, 19 Jul 2021 03:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+nFhosBzSZV7qNAcnaziksYJNq5Oy+qXrjjFGFJtf38=; b=NVj8E5o746nTD4LMKyEjLfcN0+iNKd4Q3+ARDYWcQq2Bc1tygxtGoghORVPHuz55fr 6XRVVKrCybVPnQoQ1dqIiRNFFsZO8Z9BJgYIHQDMxMPYPfrqI4GGFGxB1LTirahtqKgF xAKvcu5plneJQWJFaJ2c5xabR1g5t5l+kwW/aZ2D/8oPbSsqYoKBcR7AgbnOKB7FMw3S X7XC0MCHbF0+T/+/kcIrOaIFYgnutjaITulomfUfEJjKZS9PqaUiCF++55pWMzZV9q2E BJsPLP9JbGxG1U4PRz6OPrAsYUQlxu7PGOUlltDvQ/zsstSFZALQC40mm9bEPB4ctH8U aEWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+nFhosBzSZV7qNAcnaziksYJNq5Oy+qXrjjFGFJtf38=; b=J1Dln3osRT8hAmCLwE6qQWHORnWG/Vtn0k9uxvHPJ38YJaC2spAIMN5qO+dfvFYCD2 gyp+KMnbZtFQOU23pRyptjVDnS+f7RUzJQd/r51PlCavmiJWdEnoa5ZgpmMOJbAYaVlf V3fGptheH0NUyPw+mvF5LtUsC0yu+R7OpZ6MzrWSSzn16SgqJLxTLKgGbmvqCIclUXsF KoXfXQhOUJBg4KNciIB2PHbv4KJmf3aWV9gjp5Uk67qUpyJHEJyxltMIccAjwUzBtcPC 2lB4pUjB7TpVSAgw9swvdlceEX65I+flXJCT2jafV2g53yw8rDnXnG2Fewtpew9FahIJ r8aA== X-Gm-Message-State: AOAM5315DjBLLlCK6WI82eoMuLNPZUCYG6kYlfsGNabtZr2WjQ6rZufU w880pM/Ic5Pu0uD5AsXvdW20KBgq43yM X-Google-Smtp-Source: ABdhPJwNiWUH69H9j/rg7cEvViy/uJINohR7ix2A2F0PALIs2gR1NMlLOsKLwVie0SGTwrZNmWFRI0oB0D9I X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a0c:e313:: with SMTP id s19mr23719350qvl.26.1626691686389; Mon, 19 Jul 2021 03:48:06 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:32 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-12-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 11/14] KVM: arm64: Expose kvm_pte_valid() helper From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034808_256144_95BD964F X-CRM114-Status: GOOD ( 12.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The KVM pgtable API exposes the kvm_pgtable_walk() function to allow the definition of walkers outside of pgtable.c. However, it is not easy to implement any of those walkers without some of the low-level helpers, such as kvm_pte_valid(). Make it static inline, and move it to the header file to allow its re-use in other places. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_pgtable.h | 7 +++++++ arch/arm64/kvm/hyp/pgtable.c | 6 ------ 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 1aa49d6aabb7..8240c881ae1e 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -25,6 +25,13 @@ static inline u64 kvm_get_parange(u64 mmfr0) typedef u64 kvm_pte_t; +#define KVM_PTE_VALID BIT(0) + +static inline bool kvm_pte_valid(kvm_pte_t pte) +{ + return pte & KVM_PTE_VALID; +} + /** * struct kvm_pgtable_mm_ops - Memory management callbacks. * @zalloc_page: Allocate a single zeroed memory page. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c7120797404a..e0ae57dca827 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -11,7 +11,6 @@ #include #include -#define KVM_PTE_VALID BIT(0) #define KVM_PTE_TYPE BIT(1) #define KVM_PTE_TYPE_BLOCK 0 @@ -135,11 +134,6 @@ static u32 kvm_pgd_pages(u32 ia_bits, u32 start_level) return __kvm_pgd_page_idx(&pgt, -1ULL) + 1; } -static bool kvm_pte_valid(kvm_pte_t pte) -{ - return pte & KVM_PTE_VALID; -} - static bool kvm_pte_table(kvm_pte_t pte, u32 level) { if (level == KVM_PGTABLE_MAX_LEVELS - 1) From patchwork Mon Jul 19 10:47:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C56AC07E9D for ; Mon, 19 Jul 2021 10:58:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 287FA600EF for ; Mon, 19 Jul 2021 10:58:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 287FA600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nebOjIHI1g2BGbXCxIAJebNebj3RiGTOaoaCk2mM4pI=; b=cK9OUdWx7HuAaDSK8AyOw3RDIN up9aXwuGee9+lO4rnkVWL37qREqC1ydv2WDMTaUjibWLxVRYPNY/bwCxPb14XQcH//k7ZMhO9shbC OzBk9wEaqlx0V9zpmv+kHmq9ViWsAi7/0zRD+dZVr8XSsF1qKedAVDUGZTd4Dz/Y5X3M1NJojnEL0 l6B5azHqaHydphcYAQ5Ap505B5T5TaX6GdrzL9c87WVLEjqKtrupDEHH/LdH7HtndYdKd2rx0icJT 8H9qj+xn4xSAqNoLVgc/1aH1SnwhxJbQ+x6zZNwMuaSin+AMSEUliC0nAsuAAGsOp4cXN8Bj5i2Tb WVg67YLQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qw2-009Nkg-Nv; Mon, 19 Jul 2021 10:55:43 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qok-009Kcc-CX for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:11 +0000 Received: by mail-wm1-x34a.google.com with SMTP id g13-20020a05600c4ecdb0290242a8f4cf9cso1437527wmq.5 for ; Mon, 19 Jul 2021 03:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ToM5LGd9yFKNtYUInCRmxwlv14Vkpcedx2OC69eyExs=; b=jnbKDcOAOk2tJlEDs7bcxicOHljH0PrDLCq6CekugM0fVJCMwyzhRuLj4A8f+YzI4k IS7HwNRyj/txJNtmBi3fSo+YWQO3G/y4I3hjNMrK82HChzpG1HG0eZg0NWQmq4+LdBvK TQoiF+yO977gzIwlRBvykOORvCiNONiIq8EQHTaZxsykqzty35HsNCsVHtGzhM+agh6G uVUko0WhrSePAfAFzBuE3K0avxRFAUk2dDQemDZg6Ny+lm308lycNC3bbS1uNX5TrPAi 3C0ZsBtVJ57MYpkE8fJYxgc3GCMoYnLbLD7BAzbqf25uvdCwEEMDm60CeCvqf0KglcbB UxTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ToM5LGd9yFKNtYUInCRmxwlv14Vkpcedx2OC69eyExs=; b=uDeUsRtsLnq2lOP5eGLbzPCJYNQfwdaDZUA1mrKIRmzOgCV5eFLrfYyDiQnRbJgaKc 3o9tQ52iqYIYpsjAwmhmlGjiLavzeY6y9ljgN46zw/yguehKfofVLgSIUfvJVfu6OvLJ SJ7guWcpXY/2oAUKyIHBtjj+hestMA+VbDn2U/uB9hamudtfzdh6elaPpbdsv8ZHolAi FSHHZi6yAz2LI0RuG8Gm93dyNXRRI/gMs5gk/UEBIDSZTGDktEkJrbo57WZvX/dGsoDK OY3fjqvWhsGgElXHbLRbFPu25C41PKqWdVoODKI7XogUkaTczg8u0zmDycBPuUM64qS9 L57Q== X-Gm-Message-State: AOAM533jnpKt6pl8ICStrB0Tq967fI/uZUDVBHPe5PFaIglky0RyrUTS 9uSGOF4jUWWtmv6k+wzNzT+gnV9qkl3S X-Google-Smtp-Source: ABdhPJzgGJdAU0vtVnZruVhBnynMubFOxTgk0bZvp5vOKD+WWRlzrEeRgKopwaNKW+lR3zpb/2mloTXM4Szz X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a05:600c:214a:: with SMTP id v10mr25899621wml.67.1626691688581; Mon, 19 Jul 2021 03:48:08 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:33 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-13-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 12/14] KVM: arm64: Refactor pkvm_pgtable locking From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034810_493750_8B921846 X-CRM114-Status: GOOD ( 13.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor the hypervisor stage-1 locking in nVHE protected mode to expose a new pkvm_create_mappings_locked() function. This will be used in later patches to allow walking and changing the hypervisor stage-1 without releasing the lock. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/mm.h | 1 + arch/arm64/kvm/hyp/nvhe/mm.c | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index 8ec3a5a7744b..c76d7136ed9b 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -23,6 +23,7 @@ int hyp_map_vectors(void); int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); +int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot); int __pkvm_create_mappings(unsigned long start, unsigned long size, unsigned long phys, enum kvm_pgtable_prot prot); unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index a8efdf0f9003..dde22e2a322a 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -67,7 +67,7 @@ unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, return addr; } -int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) +int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot) { unsigned long start = (unsigned long)from; unsigned long end = (unsigned long)to; @@ -81,7 +81,8 @@ int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) int err; phys = hyp_virt_to_phys((void *)virt_addr); - err = __pkvm_create_mappings(virt_addr, PAGE_SIZE, phys, prot); + err = kvm_pgtable_hyp_map(&pkvm_pgtable, virt_addr, PAGE_SIZE, + phys, prot); if (err) return err; } @@ -89,6 +90,17 @@ int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) return 0; } +int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot) +{ + int ret; + + hyp_spin_lock(&pkvm_pgd_lock); + ret = pkvm_create_mappings_locked(from, to, prot); + hyp_spin_unlock(&pkvm_pgd_lock); + + return ret; +} + int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back) { unsigned long start, end; From patchwork Mon Jul 19 10:47:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1110EC07E9B for ; Mon, 19 Jul 2021 10:58:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D0459600EF for ; Mon, 19 Jul 2021 10:58:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D0459600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=LubBxarii2b9R7b81VJokDHtU9k/qoKjVPhGnOZ6kSk=; b=oGng225043/QtzJ2N7pZI2O3fT x7DFSbyo2JlntYHOZKbhZ+4Eyi/jpS0GL5fTopI7GjzlJFRWZme/paE60J7QaKJtYuqedkHqk4X2v VNNZ8NUlJoXme0HZkwyMQPSdV2pgqBdlc4VmErbn5gPZBJhJmG3QQcM3CMuv5CnveD4kYNuKRVguZ Lp+JZ+02qD4/nV8vckbKT0QFOK2n1Q88fnShw0DHklh/2EVVmLiYo52Yq9FZDefmgO/kJIs6bvdAK nxIBtqsAHmNCwgfkKPz9RkmZgJRrWoCjS5aUHRLJw3p1uPeemp5FqkaUXGWHle4f//oB9xp2g3DxV fNPh6owg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qwx-009O2j-0H; Mon, 19 Jul 2021 10:56:39 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qom-009KdQ-7K for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:14 +0000 Received: by mail-qt1-x84a.google.com with SMTP id d9-20020ac84e290000b0290256a44e9034so8974916qtw.22 for ; Mon, 19 Jul 2021 03:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=6HXCLD03nWm7m4mKg23888LnRiWjmXxsLxVSEAhtTQo=; b=htJ24n6fQww/7LSxCwUYTngGO3ZhD+fanDM43SvUdp9V+JprnNZTLSjlo1yits6NoI SJdeYKXm7pfhkh2wnd6zr/lQeRs20EpMXIyXPofHmuMRzkAcu9CCPSpm5O8qTC6TL96J LzXnuN7m2CB5hkX0u1goDNKlI7M9Y1oEGkrOoPx6u1Ae59CPJycL+ULMwtaopxlrxOjJ /23SaBVQESIYJom7yRf/Dm+L9f7zdQN1ULBBPVIkS8LmdbMDH9LOmKgpGgcdAfwM3hvs n8fWDDuSnpOCCheo6D5K4tnWQf4g27m4CYY1S9IIUyiw2899F+JSZLvv13qxosaGjtf/ S0eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6HXCLD03nWm7m4mKg23888LnRiWjmXxsLxVSEAhtTQo=; b=U429pNQDyl+VS8rrHot6jTCYYFVQdK7vRUC9NeXRkf04zUC/1deYsCDCttMu095wTW Rzd5ptcJ3l2pvtv4SUxLgtyySFZSomC5W/jCsvoWBaK4mZqjvoFrNvQCCg7KfGIKj3uF /iNq6a7i0fJH2ys0eodGRlokB2MJT8rUzIgBxf0LOyO22CiDzYT6de3nD1mx3/BJ5xmo NtIvk04NMHHZtIRcjIwIUlrdAshKs1ijHIVSqku5gfQrhdPzpsXZHoUSCTZzcTm3g7SX zGvPqAtSki9WvFQJQYzH//q4FIkr8elMP1hCrCmC43c3Ju4u42Vx46UmVar9/2xwUbtc vYRg== X-Gm-Message-State: AOAM532HKaz3pr8AhTSVXKlrkJ0CVhkFis8Ht1oysXh3+WhpObrmOHbF NubdQTf8E7zpRDiB5NY978QG8XjRSduK X-Google-Smtp-Source: ABdhPJzvF6UHk50ceyuXIIDIUrcFOqTtoUnYTL4ujE7EgY4G/ZJ5qKuw0wapxjd/5IkEEqk0W1jGDkSVeUrt X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a0c:e481:: with SMTP id n1mr24250352qvl.49.1626691690653; Mon, 19 Jul 2021 03:48:10 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:34 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-14-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 13/14] KVM: arm64: Restrict hyp stage-1 manipulation in protected mode From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034812_345706_019C9383 X-CRM114-Status: GOOD ( 22.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The host kernel is currently able to change EL2 stage-1 mappings without restrictions thanks to the __pkvm_create_mappings() hypercall. But in a world where the host is no longer part of the TCB, this clearly poses a problem. To fix this, introduce a new hypercall to allow the host to share a range of physical memory with the hypervisor, and remove the __pkvm_create_mappings() variant. The new hypercall implements ownership and permission checks before allowing the sharing operation, and it annotates the shared pages in the hypervisor stage-1 and host stage-2 page-tables. Signed-off-by: Quentin Perret --- arch/arm64/include/asm/kvm_asm.h | 2 +- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/include/nvhe/mm.h | 2 - arch/arm64/kvm/hyp/nvhe/hyp-main.c | 12 +- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 105 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/mm.c | 4 +- arch/arm64/kvm/mmu.c | 14 ++- 7 files changed, 124 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 9f0bf2109be7..78db818ae2c9 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -59,7 +59,7 @@ #define __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs 13 #define __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs 14 #define __KVM_HOST_SMCCC_FUNC___pkvm_init 15 -#define __KVM_HOST_SMCCC_FUNC___pkvm_create_mappings 16 +#define __KVM_HOST_SMCCC_FUNC___pkvm_host_share_hyp 16 #define __KVM_HOST_SMCCC_FUNC___pkvm_create_private_mapping 17 #define __KVM_HOST_SMCCC_FUNC___pkvm_cpu_set_vector 18 #define __KVM_HOST_SMCCC_FUNC___pkvm_prot_finalize 19 diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h index b39047463075..f37e4d3b831b 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -22,6 +22,7 @@ extern struct host_kvm host_kvm; int __pkvm_prot_finalize(void); int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end); +int __pkvm_host_share_hyp(phys_addr_t start, phys_addr_t end); int host_stage2_idmap_locked(u64 start, u64 end, enum kvm_pgtable_prot prot); int kvm_host_prepare_stage2(void *pgt_pool_base); diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index c76d7136ed9b..c9a8f535212e 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -24,8 +24,6 @@ int hyp_back_vmemmap(phys_addr_t phys, unsigned long size, phys_addr_t back); int pkvm_cpu_set_vector(enum arm64_hyp_spectre_vector slot); int pkvm_create_mappings(void *from, void *to, enum kvm_pgtable_prot prot); int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot); -int __pkvm_create_mappings(unsigned long start, unsigned long size, - unsigned long phys, enum kvm_pgtable_prot prot); unsigned long __pkvm_create_private_mapping(phys_addr_t phys, size_t size, enum kvm_pgtable_prot prot); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 1632f001f4ed..f05ecbd382d0 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -140,14 +140,12 @@ static void handle___pkvm_cpu_set_vector(struct kvm_cpu_context *host_ctxt) cpu_reg(host_ctxt, 1) = pkvm_cpu_set_vector(slot); } -static void handle___pkvm_create_mappings(struct kvm_cpu_context *host_ctxt) +static void handle___pkvm_host_share_hyp(struct kvm_cpu_context *host_ctxt) { - DECLARE_REG(unsigned long, start, host_ctxt, 1); - DECLARE_REG(unsigned long, size, host_ctxt, 2); - DECLARE_REG(unsigned long, phys, host_ctxt, 3); - DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 4); + DECLARE_REG(phys_addr_t, start, host_ctxt, 1); + DECLARE_REG(phys_addr_t, end, host_ctxt, 2); - cpu_reg(host_ctxt, 1) = __pkvm_create_mappings(start, size, phys, prot); + cpu_reg(host_ctxt, 1) = __pkvm_host_share_hyp(start, end); } static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ctxt) @@ -193,7 +191,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__vgic_v3_restore_aprs), HANDLE_FUNC(__pkvm_init), HANDLE_FUNC(__pkvm_cpu_set_vector), - HANDLE_FUNC(__pkvm_create_mappings), + HANDLE_FUNC(__pkvm_host_share_hyp), HANDLE_FUNC(__pkvm_create_private_mapping), HANDLE_FUNC(__pkvm_prot_finalize), HANDLE_FUNC(__pkvm_mark_hyp), diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 6f28edf58407..20b3cb3fdc67 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -262,6 +262,111 @@ static int host_stage2_idmap(u64 addr) return ret; } +static int hyp_range_is_shared_walker(u64 addr, u64 end, u32 level, + kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + enum kvm_pgtable_prot prot; + kvm_pte_t pte = *ptep; + + if (!kvm_pte_valid(pte)) + return -EPERM; + + prot = kvm_pgtable_hyp_pte_prot(pte); + if (!prot) + return -EPERM; + + /* Check that the page has been shared with the hypervisor before */ + if (prot != (PAGE_HYP | KVM_PGTABLE_STATE_SHARED | KVM_PGTABLE_STATE_BORROWED)) + return -EPERM; + + return 0; +} + +static int hyp_range_is_shared(phys_addr_t start, phys_addr_t end) +{ + struct kvm_pgtable_walker walker = { + .cb = hyp_range_is_shared_walker, + .flags = KVM_PGTABLE_WALK_LEAF, + }; + + return kvm_pgtable_walk(&pkvm_pgtable, (u64)__hyp_va(start), + end - start, &walker); +} + +static int check_host_share_hyp_walker(u64 addr, u64 end, u32 level, + kvm_pte_t *ptep, + enum kvm_pgtable_walk_flags flag, + void * const arg) +{ + enum kvm_pgtable_prot prot; + kvm_pte_t pte = *ptep; + + /* If invalid, only allow to share pristine pages */ + if (!kvm_pte_valid(pte)) + return pte ? -EPERM : 0; + + prot = kvm_pgtable_stage2_pte_prot(pte); + if (!prot) + return -EPERM; + + /* Cannot share a page that is not owned */ + if (prot & KVM_PGTABLE_STATE_BORROWED) + return -EPERM; + + /* Cannot share a page with restricted access */ + if ((prot & KVM_PGTABLE_PROT_RWX) ^ KVM_PGTABLE_PROT_RWX) + return -EPERM; + + /* Allow double-sharing (requires cross-checking the hyp stage-1) */ + if (prot & KVM_PGTABLE_STATE_SHARED) + return hyp_range_is_shared(addr, addr + 1); + + return 0; +} + +static int check_host_share_hyp(phys_addr_t start, phys_addr_t end) +{ + struct kvm_pgtable_walker walker = { + .cb = check_host_share_hyp_walker, + .flags = KVM_PGTABLE_WALK_LEAF, + }; + + return kvm_pgtable_walk(&host_kvm.pgt, start, end - start, &walker); +} + +int __pkvm_host_share_hyp(phys_addr_t start, phys_addr_t end) +{ + enum kvm_pgtable_prot prot; + int ret; + + if (!range_is_memory(start, end)) + return -EINVAL; + + hyp_spin_lock(&host_kvm.lock); + hyp_spin_lock(&pkvm_pgd_lock); + + ret = check_host_share_hyp(start, end); + if (ret) + goto unlock; + + prot = KVM_PGTABLE_PROT_RWX | KVM_PGTABLE_STATE_SHARED; + ret = host_stage2_idmap_locked(start, end, prot); + if (ret && ret != -EAGAIN) + goto unlock; + + prot = PAGE_HYP | KVM_PGTABLE_STATE_SHARED | KVM_PGTABLE_STATE_BORROWED; + ret = pkvm_create_mappings_locked(__hyp_va(start), __hyp_va(end), prot); + /* XXX - undo host stage-2 changes if ret != 0 */ + +unlock: + hyp_spin_unlock(&pkvm_pgd_lock); + hyp_spin_unlock(&host_kvm.lock); + + return ret; +} + int __pkvm_mark_hyp(phys_addr_t start, phys_addr_t end) { int ret; diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index dde22e2a322a..95f6c34a38ec 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -23,8 +23,8 @@ u64 __io_map_base; struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS]; unsigned int hyp_memblock_nr; -int __pkvm_create_mappings(unsigned long start, unsigned long size, - unsigned long phys, enum kvm_pgtable_prot prot) +static int __pkvm_create_mappings(unsigned long start, unsigned long size, + unsigned long phys, enum kvm_pgtable_prot prot) { int err; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 0625bf2353c2..2158d1e00acd 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -259,10 +259,8 @@ static int __create_hyp_mappings(unsigned long start, unsigned long size, { int err; - if (!kvm_host_owns_hyp_mappings()) { - return kvm_call_hyp_nvhe(__pkvm_create_mappings, - start, size, phys, prot); - } + if (WARN_ON(!kvm_host_owns_hyp_mappings())) + return -EINVAL; mutex_lock(&kvm_hyp_pgd_mutex); err = kvm_pgtable_hyp_map(hyp_pgtable, start, size, phys, prot); @@ -302,6 +300,14 @@ int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot) if (is_kernel_in_hyp_mode()) return 0; + if (!kvm_host_owns_hyp_mappings()) { + if (WARN_ON(prot != PAGE_HYP)) + return -EPERM; + return kvm_call_hyp_nvhe(__pkvm_host_share_hyp, + kvm_kaddr_to_phys(from), + kvm_kaddr_to_phys(to)); + } + start = start & PAGE_MASK; end = PAGE_ALIGN(end); From patchwork Mon Jul 19 10:47:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12385449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45FB4C07E9B for ; Mon, 19 Jul 2021 11:00:42 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1196C600EF for ; Mon, 19 Jul 2021 11:00:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1196C600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Xx9UIsbFUTITBWpVDGhqgwUYMQLAuDaswC/EJ3Az6po=; b=Oy+JlAgSaAB+GnW+eyoXtoHegq nT/yGC0IUw8sG9F3+QeQdRctyQfzh7QB+LHEski08Ddpu6o8zBZkHB5RjI5gXZTtzwct1mYOxjg70 uD5rXUsLKCfFAkS20YjeUYsfuAqlx0q7NJG23g+wQQmLgAD2r1pJkClSi2DRJT9fK63OCjD5ttLnT tF6nPfsa0lBaP3QqPh4llj9gLl+X1fa2GVPwUhvqPI/c+jBuk5rXg3TN3v+q7Tdq79vUfOoMRsBmT bEV+LekJ4H0PooDh01yXHjO4+0Zs9i2w3di/awgRHxztSy0x4VZv4nGYnrGdb4cmSJZvbXpl5hD4c INyv2B/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qxv-009OQg-Ti; Mon, 19 Jul 2021 10:57:40 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m5Qoo-009Ke8-TK for linux-arm-kernel@lists.infradead.org; Mon, 19 Jul 2021 10:48:16 +0000 Received: by mail-wr1-x449.google.com with SMTP id y15-20020a5d614f0000b029013cd60e9baaso8580607wrt.7 for ; Mon, 19 Jul 2021 03:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=xCrkHvgx5Vjp7sEk2L8Nl84wrwMoF/axMUKdY9pah24=; b=pDxR+oOIO361IE8JTpjXvK01glJPoXIl5CKZaNz+ZEfkmqJ2xvk3unbPd+7kb+OWda trEZrnZ3NzpP8uHCTZzYftalFdx/5wIdaoBQhrxNPVxgdiIghwpg+2wP8M3OPqfirjTd xnwKYQbjP84kmxrLM9ZxQ2L+sfZUek62Dy3owBDZGm0aQ2qax8dWHJG04y1UCCQkOeuM PcvsBY4gP2Eu0tIv5EGGfrTI+RlXVojwsHLkOMz6yutWmR06OxR8yq1klADXa4P0bQwV ool2uXbjJpW+F45cx8urCBsHAnKEob82EgvPFM7Eq5WwW85p/LaSm12Ccfkyebt5WyL7 ovvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xCrkHvgx5Vjp7sEk2L8Nl84wrwMoF/axMUKdY9pah24=; b=E38LnR47gG3sNS1JS+AzeHDUP5Cbj+k+cb7MzjGm5NTDQckFBaolMN4EhBl31dxWXZ YAsf+2kYEI/FIhCUDwnZyQ8JO6OHTQEPPPa07FObJtv8jKcE9r8olf8eyZEpwVzucdrq qKseq/QJdSsCB4P/uK3/qXw7bBUxP0/0MGGkUgSyPWnvaZnyi6YD0pj4/RgNupKNGlF3 a/zds61j73tjlG7WboTsZ1xSZrOqY/TBBZGrRGTfEH7wIvfKJ+6/98ofgZ6Az1GO+s4S yvbkYwioir5LdkEDOpgFVEKKaZzLo/JLKOEYxFjpuJUQ8XiLJK/Csx8PFbWw51AKa4WO +0rQ== X-Gm-Message-State: AOAM530fr85j8JL+gfOTctcznF7wbico1NYMmqJbbUquSayjBYLCKKT7 XSCMIlxUuY4GYMUws/JdalA0V8FNBNXY X-Google-Smtp-Source: ABdhPJyIruRnm597DeIoz+H4lQk2XaV0lp7fhRFMVmFaiZHZyDX1vs7MSEJSK9DYDl49AjGViAobxaMPEAN7 X-Received: from luke.lon.corp.google.com ([2a00:79e0:d:210:1a96:a43f:6c2e:bb5a]) (user=qperret job=sendgmr) by 2002:a1c:4c18:: with SMTP id z24mr32007850wmf.168.1626691692999; Mon, 19 Jul 2021 03:48:12 -0700 (PDT) Date: Mon, 19 Jul 2021 11:47:35 +0100 In-Reply-To: <20210719104735.3681732-1-qperret@google.com> Message-Id: <20210719104735.3681732-15-qperret@google.com> Mime-Version: 1.0 References: <20210719104735.3681732-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.402.g57bb445576-goog Subject: [PATCH 14/14] KVM: arm64: Prevent late calls to __pkvm_create_private_mapping() From: Quentin Perret To: maz@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, ardb@kernel.org, qwandor@google.com, tabba@google.com, dbrazdil@google.com, kernel-team@android.com, Quentin Perret X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210719_034815_047629_361491FC X-CRM114-Status: GOOD ( 14.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org __pkvm_create_private_mapping() allows the host kernel to create arbitrary mappings the hypervisor's "private" range. However, this is only needed early on, and there should be no good reason for the host to need this past the point where the pkvm static is set. Make sure to stub the hypercall past this point to ensure it can't be used by a malicious host. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index f05ecbd382d0..e1d12f8122a7 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -154,7 +154,10 @@ static void handle___pkvm_create_private_mapping(struct kvm_cpu_context *host_ct DECLARE_REG(size_t, size, host_ctxt, 2); DECLARE_REG(enum kvm_pgtable_prot, prot, host_ctxt, 3); - cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); + if (static_branch_unlikely(&kvm_protected_mode_initialized)) + cpu_reg(host_ctxt, 1) = -EPERM; + else + cpu_reg(host_ctxt, 1) = __pkvm_create_private_mapping(phys, size, prot); } static void handle___pkvm_prot_finalize(struct kvm_cpu_context *host_ctxt)