From patchwork Mon May 25 11:23:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhukeqian X-Patchwork-Id: 11568615 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D01151391 for ; Mon, 25 May 2020 11:26:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB04B20723 for ; Mon, 25 May 2020 11:26:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="MIh6MBbv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB04B20723 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=hj6IaJn6DUhpum+0XMkA9ad7NpVpKADEzX9Ta1ymY8g=; b=MIh6MBbvn5TsWv FdENs5O2m5OSl8mCWdCqhkgAUpWW1lN6BRJl2CD03haMtaqnu1qek0c8eWpTBK3dyUish0K/kkRkg Y9lkVZ2ESn8OAWIgDkoGfM1Wx4yZUlJLS6XpqTTVkaUCtJXKXAe/55wxzNmHfdvzhv8gX/4WLjkwG eOi2rS3mOZ4rhPaRp+9To0jh01YzmKNgZ5JQPos2cgy4wiCURrBrqjzpBQ3u+FHUMMS5r1a6KQ8DK 0FvLXkUv1DKls1uLjksLIGzvUspUTBZOyo6lHGhtViU91J5lidWDi3g6M5HOJzE9fPT7RNDUnj5mE gGDRsb2tjcCrcnkZtsrQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jdBFg-0007QR-V0; Mon, 25 May 2020 11:26:40 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jdBEN-0005gc-MS for linux-arm-kernel@lists.infradead.org; Mon, 25 May 2020 11:25:23 +0000 Received: from DGGEMS401-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 5A106A0EB3F34B6CB6FB; Mon, 25 May 2020 19:25:04 +0800 (CST) Received: from DESKTOP-5IS4806.china.huawei.com (10.173.221.230) by DGGEMS401-HUB.china.huawei.com (10.3.19.201) with Microsoft SMTP Server id 14.3.487.0; Mon, 25 May 2020 19:24:54 +0800 From: Keqian Zhu To: , , , Subject: [RFC PATCH 0/7] kvm: arm64: Support stage2 hardware DBM Date: Mon, 25 May 2020 19:23:59 +0800 Message-ID: <20200525112406.28224-1-zhukeqian1@huawei.com> X-Mailer: git-send-email 2.8.4.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.173.221.230] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200525_042519_937379_363CE83E X-CRM114-Status: GOOD ( 13.89 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.191 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.191 listed in wl.mailspike.net] 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Suzuki K Poulose , Catalin Marinas , Keqian Zhu , Sean Christopherson , Alexios Zavras , zhengxiang9@huawei.com, Mark Brown , James Morse , Marc Zyngier , wanghaibin.wang@huawei.com, Thomas Gleixner , Will Deacon , Andrew Morton , Julien Thierry Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This patch series add support for stage2 hardware DBM, and it is only used for dirty log for now. It works well under some migration test cases, including VM with 4K pages or 2M THP. I checked the SHA256 hash digest of all memory and they keep same for source VM and destination VM, which means no dirty pages is missed under hardware DBM. However, there are some known issues not solved. 1. Some mechanisms that rely on "write permission fault" become invalid, such as kvm_set_pfn_dirty and "mmap page sharing". kvm_set_pfn_dirty is called in user_mem_abort when guest issues write fault. This guarantees physical page will not be dropped directly when host kernel recycle memory. After using hardware dirty management, we have no chance to call kvm_set_pfn_dirty. For "mmap page sharing" mechanism, host kernel will allocate a new physical page when guest writes a page that is shared with other page table entries. After using hardware dirty management, we have no chance to do this too. I need to do some survey on how stage1 hardware DBM solve these problems. It helps if anyone can figure it out. 2. Page Table Modification Races: Though I have found and solved some data races when kernel changes page table entries, I still doubt that there are data races I am not aware of. It's great if anyone can figure them out. 3. Performance: Under Kunpeng 920 platform, for every 64GB memory, KVM consumes about 40ms to traverse all PTEs to collect dirty log. It will cause unbearable downtime for migration if memory size is too big. I will try to solve this problem in Patch v1. Keqian Zhu (7): KVM: arm64: Add some basic functions for hw DBM KVM: arm64: Set DBM bit of PTEs if hw DBM enabled KVM: arm64: Traverse page table entries when sync dirty log KVM: arm64: Steply write protect page table by mask bit kvm: arm64: Modify stage2 young mechanism to support hw DBM kvm: arm64: Save stage2 PTE dirty info if it is coverred KVM: arm64: Enable stage2 hardware DBM arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_mmu.h | 44 +++++- arch/arm64/include/asm/pgtable-prot.h | 1 + arch/arm64/include/asm/sysreg.h | 2 + arch/arm64/kvm/reset.c | 9 +- virt/kvm/arm/arm.c | 6 +- virt/kvm/arm/mmu.c | 202 ++++++++++++++++++++++++-- 7 files changed, 246 insertions(+), 19 deletions(-)