From patchwork Wed Nov 18 12:54:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Gruzdev X-Patchwork-Id: 11915063 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3497415E6 for ; Wed, 18 Nov 2020 13:03:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9D915221EB for ; Wed, 18 Nov 2020 13:03:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9D915221EB Authentication-Results: mail.kernel.org; dmarc=pass (p=none dis=none) header.from=nongnu.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35644 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kfN7U-00024b-Bi for patchwork-qemu-devel@patchwork.kernel.org; Wed, 18 Nov 2020 08:03:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:56828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kfMzK-0003Da-VB for qemu-devel@nongnu.org; Wed, 18 Nov 2020 07:55:07 -0500 Received: from relay.sw.ru ([185.231.240.75]:40568 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kfMz8-00072f-2u for qemu-devel@nongnu.org; Wed, 18 Nov 2020 07:55:06 -0500 Received: from [192.168.15.76] (helo=andrey-MS-7B54.sw.ru) by relay3.sw.ru with esmtp (Exim 4.94) (envelope-from ) id 1kfMyt-009AfQ-9w; Wed, 18 Nov 2020 15:54:39 +0300 To: qemu-devel@nongnu.org Cc: Den Lunev , Eric Blake , Paolo Bonzini , Juan Quintela , "Dr . David Alan Gilbert" , Markus Armbruster , Andrey Gruzdev Subject: [PATCH v1 0/7] UFFD write-tracking migration/snapshots Date: Wed, 18 Nov 2020 15:54:42 +0300 Message-Id: <20201118125449.311038-1-andrey.gruzdev@virtuozzo.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.gruzdev@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/18 06:22:34 X-ACL-Warn: Detected OS = Linux 3.11 and newer [fuzzy] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Reply-to: Andrey Gruzdev X-Patchwork-Original-From: Andrey Gruzdev via From: Andrey Gruzdev Currently the only way to make (external) live VM snapshot is using existing dirty page logging migration mechanism. The main problem is that it tends to produce a lot of page duplicates while running VM goes on updating already saved pages. That leads to the fact that vmstate image size is commonly several times bigger then non-zero part of virtual machine's RSS. Time required to converge RAM migration and the size of snapshot image severely depend on the guest memory write rate, sometimes resulting in unacceptably long snapshot creation time and huge image size. This series propose a way to solve the aforementioned problems. This is done by using different RAM migration mechanism based on UFFD write protection management introduced in v5.7 kernel. The migration strategy is to 'freeze' guest RAM content using write-protection and iteratively release protection for memory ranges that have already been saved to the migration stream. At the same time we read in pending UFFD write fault events and save those pages out-of-order with higher priority. How to use: 1. Enable write-tracking migration capability virsh qemu-monitor-command --hmp migrate_set_capability. track-writes-ram on 2. Start the external migration to a file virsh qemu-monitor-command --hmp migrate exec:'cat > ./vm_state' 3. Wait for the migration finish and check that the migration has completed. state. Andrey Gruzdev (7): Introduce 'track-writes-ram' migration capability. Introduced UFFD-WP low-level interface helpers. Implemented support for the whole RAM block memory protection/un-protection. Higher level ram_write_tracking_start() and ram_write_tracking_stop() to start/stop tracking memory writes on the whole VM memory. Support UFFD write fault processing in ram_save_iterate(). Implementation of write-tracking migration thread. Implementation of vm_start() BH. The rest of write tracking migration code. Introduced simple linear scan rate limiting mechanism for write tracking migration. include/exec/memory.h | 7 + migration/migration.c | 338 +++++++++++++++++++++++++++++++- migration/migration.h | 4 + migration/ram.c | 439 +++++++++++++++++++++++++++++++++++++++++- migration/ram.h | 4 + migration/savevm.c | 1 - migration/savevm.h | 2 + qapi/migration.json | 7 +- 8 files changed, 790 insertions(+), 12 deletions(-)