From patchwork Mon Sep 9 20:11:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6C4DECE587 for ; Mon, 9 Sep 2024 20:13:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkjw-0008Vg-2N; Mon, 09 Sep 2024 16:12:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkju-0008QW-Pp for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:11:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjs-0004zn-Hf for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:11:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fBQjFfFzHVzxG0I3N5y9yf1md/tdpjRtkmvTgUYzVGI=; b=H4g07S1D2CxCZT1A429vAGGrv+obXMV0xz05/Mf7I8HiVbVV8zOxrjf1g0BR5yjfsOPDLD 9EIf5TGueXXJo7a8OgLhE2GG/H0CFfauGYPBjFLkI0dHy/7hDrZd9fGqhVm1D8OIW6zob5 NmzJuqBKZFtrbiEm8yw/qW0AZ0w+uHo= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-59-IXDKOEhjOWG97pt3NoZ_6g-1; Mon, 09 Sep 2024 16:11:53 -0400 X-MC-Unique: IXDKOEhjOWG97pt3NoZ_6g-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-4582493b3f7so33478571cf.0 for ; Mon, 09 Sep 2024 13:11:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912712; x=1726517512; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fBQjFfFzHVzxG0I3N5y9yf1md/tdpjRtkmvTgUYzVGI=; b=a/H46KFvZgE8sIOeIoaQCFcV/UHQjCZh7yNba66lA6WVOw1oH/ptbc20rMj+b7Phwk G4qeeKYSML0Dw1gOy0m+DXC0tnDGz+bmyZ61fwNYt6ykwMUof3rzPGgabgNrI/P+iIH9 MVLAUaM8Mm2doeYjEFS42J53606Ru32dWI3HmVzQcjWa4OPS/iAb0SnRrZv0nZejO+OQ IaW840nSbEz4nlwkWlr5v4mkjGn9rhh501r6VKo1ZvzHN1agYo9295hQtjA7azRbDEHj xToatCyhhegbYdbQXIyqrr1sOsZ7VYtvpMZmwiZzUnJIoBV/P+7P7sZdSOEa4CguwAwe YWgQ== X-Gm-Message-State: AOJu0YzsCDhn4p4iCAxAgT691g56UuseQolllVKR3ggfMBhspkWkOfMV URpu5aYHsAsHraFzQVtmA2IwL/vdeS1j8e1COSlIZpWfNJ+Png1FSrqUmsvDiG0tCrO5/v8RJAc hR134WolhuSHXG4DJ7mMbGTs+FYzWlvG59heiytS8jp/gwdXfSC2Z6LJ+FaxPRblvnsgOGMVaBV HaZYu1qE88XLjSHs+LgLNWT0+mbAc4M1BTRg== X-Received: by 2002:a05:622a:198e:b0:458:2a05:6b71 with SMTP id d75a77b69052e-4582a056d83mr77020211cf.44.1725912711972; Mon, 09 Sep 2024 13:11:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEvJyf1WHaTw38FN7uUbFJR+Ockz72jyW878REbF7+6/sbDtbcD9uu/LfV2N0imB9FMGfr8PA== X-Received: by 2002:a05:622a:198e:b0:458:2a05:6b71 with SMTP id d75a77b69052e-4582a056d83mr77019761cf.44.1725912711415; Mon, 09 Sep 2024 13:11:51 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.11.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:11:50 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Mattias Nissler , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= Subject: [PULL 1/9] softmmu: Support concurrent bounce buffers Date: Mon, 9 Sep 2024 16:11:39 -0400 Message-ID: <20240909201147.3761639-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Mattias Nissler When DMA memory can't be directly accessed, as is the case when running the device model in a separate process without shareable DMA file descriptors, bounce buffering is used. It is not uncommon for device models to request mapping of several DMA regions at the same time. Examples include: * net devices, e.g. when transmitting a packet that is split across several TX descriptors (observed with igb) * USB host controllers, when handling a packet with multiple data TRBs (observed with xhci) Previously, qemu only provided a single bounce buffer per AddressSpace and would fail DMA map requests while the buffer was already in use. In turn, this would cause DMA failures that ultimately manifest as hardware errors from the guest perspective. This change allocates DMA bounce buffers dynamically instead of supporting only a single buffer. Thus, multiple DMA mappings work correctly also when RAM can't be mmap()-ed. The total bounce buffer allocation size is limited individually for each AddressSpace. The default limit is 4096 bytes, matching the previous maximum buffer size. A new x-max-bounce-buffer-size parameter is provided to configure the limit for PCI devices. Signed-off-by: Mattias Nissler Reviewed-by: Philippe Mathieu-Daudé Acked-by: Peter Xu Link: https://lore.kernel.org/r/20240819135455.2957406-1-mnissler@rivosinc.com Signed-off-by: Peter Xu --- include/exec/memory.h | 14 +++---- include/hw/pci/pci_device.h | 3 ++ hw/pci/pci.c | 8 ++++ system/memory.c | 5 ++- system/physmem.c | 82 ++++++++++++++++++++++++++----------- 5 files changed, 76 insertions(+), 36 deletions(-) diff --git a/include/exec/memory.h b/include/exec/memory.h index 296fd068c0..e5e865d1a9 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -1084,13 +1084,7 @@ typedef struct AddressSpaceMapClient { QLIST_ENTRY(AddressSpaceMapClient) link; } AddressSpaceMapClient; -typedef struct { - MemoryRegion *mr; - void *buffer; - hwaddr addr; - hwaddr len; - bool in_use; -} BounceBuffer; +#define DEFAULT_MAX_BOUNCE_BUFFER_SIZE (4096) /** * struct AddressSpace: describes a mapping of addresses to #MemoryRegion objects @@ -1110,8 +1104,10 @@ struct AddressSpace { QTAILQ_HEAD(, MemoryListener) listeners; QTAILQ_ENTRY(AddressSpace) address_spaces_link; - /* Bounce buffer to use for this address space. */ - BounceBuffer bounce; + /* Maximum DMA bounce buffer size used for indirect memory map requests */ + size_t max_bounce_buffer_size; + /* Total size of bounce buffers currently allocated, atomically accessed */ + size_t bounce_buffer_size; /* List of callbacks to invoke when buffers free up */ QemuMutex map_client_list_lock; QLIST_HEAD(, AddressSpaceMapClient) map_client_list; diff --git a/include/hw/pci/pci_device.h b/include/hw/pci/pci_device.h index 15694f2489..91df40f989 100644 --- a/include/hw/pci/pci_device.h +++ b/include/hw/pci/pci_device.h @@ -167,6 +167,9 @@ struct PCIDevice { /* ID of standby device in net_failover pair */ char *failover_pair_id; uint32_t acpi_index; + + /* Maximum DMA bounce buffer size used for indirect memory map requests */ + uint32_t max_bounce_buffer_size; }; static inline int pci_intx(PCIDevice *pci_dev) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index fab86d0567..d2caf3ee8b 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -85,6 +85,8 @@ static Property pci_props[] = { QEMU_PCIE_ERR_UNC_MASK_BITNR, true), DEFINE_PROP_BIT("x-pcie-ari-nextfn-1", PCIDevice, cap_present, QEMU_PCIE_ARI_NEXTFN_1_BITNR, false), + DEFINE_PROP_SIZE32("x-max-bounce-buffer-size", PCIDevice, + max_bounce_buffer_size, DEFAULT_MAX_BOUNCE_BUFFER_SIZE), DEFINE_PROP_END_OF_LIST() }; @@ -1204,6 +1206,8 @@ static PCIDevice *do_pci_register_device(PCIDevice *pci_dev, "bus master container", UINT64_MAX); address_space_init(&pci_dev->bus_master_as, &pci_dev->bus_master_container_region, pci_dev->name); + pci_dev->bus_master_as.max_bounce_buffer_size = + pci_dev->max_bounce_buffer_size; if (phase_check(PHASE_MACHINE_READY)) { pci_init_bus_master(pci_dev); @@ -2633,6 +2637,10 @@ static void pci_device_class_init(ObjectClass *klass, void *data) k->unrealize = pci_qdev_unrealize; k->bus_type = TYPE_PCI_BUS; device_class_set_props(k, pci_props); + object_class_property_set_description( + klass, "x-max-bounce-buffer-size", + "Maximum buffer size allocated for bounce buffers used for mapped " + "access to indirect DMA memory"); } static void pci_device_class_base_init(ObjectClass *klass, void *data) diff --git a/system/memory.c b/system/memory.c index 5e6eb459d5..f6f6fee6d8 100644 --- a/system/memory.c +++ b/system/memory.c @@ -3148,7 +3148,8 @@ void address_space_init(AddressSpace *as, MemoryRegion *root, const char *name) as->ioeventfds = NULL; QTAILQ_INIT(&as->listeners); QTAILQ_INSERT_TAIL(&address_spaces, as, address_spaces_link); - as->bounce.in_use = false; + as->max_bounce_buffer_size = DEFAULT_MAX_BOUNCE_BUFFER_SIZE; + as->bounce_buffer_size = 0; qemu_mutex_init(&as->map_client_list_lock); QLIST_INIT(&as->map_client_list); as->name = g_strdup(name ? name : "anonymous"); @@ -3158,7 +3159,7 @@ void address_space_init(AddressSpace *as, MemoryRegion *root, const char *name) static void do_address_space_destroy(AddressSpace *as) { - assert(!qatomic_read(&as->bounce.in_use)); + assert(qatomic_read(&as->bounce_buffer_size) == 0); assert(QLIST_EMPTY(&as->map_client_list)); qemu_mutex_destroy(&as->map_client_list_lock); diff --git a/system/physmem.c b/system/physmem.c index 94600a33ec..971bfa0855 100644 --- a/system/physmem.c +++ b/system/physmem.c @@ -3095,6 +3095,20 @@ void cpu_flush_icache_range(hwaddr start, hwaddr len) NULL, len, FLUSH_CACHE); } +/* + * A magic value stored in the first 8 bytes of the bounce buffer struct. Used + * to detect illegal pointers passed to address_space_unmap. + */ +#define BOUNCE_BUFFER_MAGIC 0xb4017ceb4ffe12ed + +typedef struct { + uint64_t magic; + MemoryRegion *mr; + hwaddr addr; + size_t len; + uint8_t buffer[]; +} BounceBuffer; + static void address_space_unregister_map_client_do(AddressSpaceMapClient *client) { @@ -3120,9 +3134,9 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh) QEMU_LOCK_GUARD(&as->map_client_list_lock); client->bh = bh; QLIST_INSERT_HEAD(&as->map_client_list, client, link); - /* Write map_client_list before reading in_use. */ + /* Write map_client_list before reading bounce_buffer_size. */ smp_mb(); - if (!qatomic_read(&as->bounce.in_use)) { + if (qatomic_read(&as->bounce_buffer_size) < as->max_bounce_buffer_size) { address_space_notify_map_clients_locked(as); } } @@ -3251,28 +3265,40 @@ void *address_space_map(AddressSpace *as, mr = flatview_translate(fv, addr, &xlat, &l, is_write, attrs); if (!memory_access_is_direct(mr, is_write)) { - if (qatomic_xchg(&as->bounce.in_use, true)) { + size_t used = qatomic_read(&as->bounce_buffer_size); + for (;;) { + hwaddr alloc = MIN(as->max_bounce_buffer_size - used, l); + size_t new_size = used + alloc; + size_t actual = + qatomic_cmpxchg(&as->bounce_buffer_size, used, new_size); + if (actual == used) { + l = alloc; + break; + } + used = actual; + } + + if (l == 0) { *plen = 0; return NULL; } - /* Avoid unbounded allocations */ - l = MIN(l, TARGET_PAGE_SIZE); - as->bounce.buffer = qemu_memalign(TARGET_PAGE_SIZE, l); - as->bounce.addr = addr; - as->bounce.len = l; + BounceBuffer *bounce = g_malloc0(l + sizeof(BounceBuffer)); + bounce->magic = BOUNCE_BUFFER_MAGIC; memory_region_ref(mr); - as->bounce.mr = mr; + bounce->mr = mr; + bounce->addr = addr; + bounce->len = l; + if (!is_write) { flatview_read(fv, addr, MEMTXATTRS_UNSPECIFIED, - as->bounce.buffer, l); + bounce->buffer, l); } *plen = l; - return as->bounce.buffer; + return bounce->buffer; } - memory_region_ref(mr); *plen = flatview_extend_translation(fv, addr, len, mr, xlat, l, is_write, attrs); @@ -3287,12 +3313,11 @@ void *address_space_map(AddressSpace *as, void address_space_unmap(AddressSpace *as, void *buffer, hwaddr len, bool is_write, hwaddr access_len) { - if (buffer != as->bounce.buffer) { - MemoryRegion *mr; - ram_addr_t addr1; + MemoryRegion *mr; + ram_addr_t addr1; - mr = memory_region_from_host(buffer, &addr1); - assert(mr != NULL); + mr = memory_region_from_host(buffer, &addr1); + if (mr != NULL) { if (is_write) { invalidate_and_set_dirty(mr, addr1, access_len); } @@ -3302,15 +3327,22 @@ void address_space_unmap(AddressSpace *as, void *buffer, hwaddr len, memory_region_unref(mr); return; } + + + BounceBuffer *bounce = container_of(buffer, BounceBuffer, buffer); + assert(bounce->magic == BOUNCE_BUFFER_MAGIC); + if (is_write) { - address_space_write(as, as->bounce.addr, MEMTXATTRS_UNSPECIFIED, - as->bounce.buffer, access_len); - } - qemu_vfree(as->bounce.buffer); - as->bounce.buffer = NULL; - memory_region_unref(as->bounce.mr); - /* Clear in_use before reading map_client_list. */ - qatomic_set_mb(&as->bounce.in_use, false); + address_space_write(as, bounce->addr, MEMTXATTRS_UNSPECIFIED, + bounce->buffer, access_len); + } + + qatomic_sub(&as->bounce_buffer_size, bounce->len); + bounce->magic = ~BOUNCE_BUFFER_MAGIC; + memory_region_unref(bounce->mr); + g_free(bounce); + /* Write bounce_buffer_size before reading map_client_list. */ + smp_mb(); address_space_notify_map_clients(as); } From patchwork Mon Sep 9 20:11:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9267BECE58B for ; Mon, 9 Sep 2024 20:13:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkjy-0000Fg-Nr; Mon, 09 Sep 2024 16:12:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjw-00004v-7B for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjt-00050S-LF for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:11:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912716; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8crV/ycQb1WW3+BtmyCL6RoebnxZjRylOFrGpe4gHdU=; b=ipT8s8jBABZGlQCva6UnvuFJ6DCwSBnf9lOjOCL3spZXEn7n6HzhB4Iybce+i4Zbe7Njw8 zl4V7ECOfa9wilE7JntW466PAuafz30TLtNl3oht2tupBAmbY9GYm2tPZYBnHM6OhkRZ3w ewY7XXEqCleEmEnTR9p5gvZgey8DWL0= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-653-mpDzQOiBOuWPyxC8_rq5xQ-1; Mon, 09 Sep 2024 16:11:54 -0400 X-MC-Unique: mpDzQOiBOuWPyxC8_rq5xQ-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-45798aa2bacso76109881cf.2 for ; Mon, 09 Sep 2024 13:11:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912714; x=1726517514; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8crV/ycQb1WW3+BtmyCL6RoebnxZjRylOFrGpe4gHdU=; b=SXyFYsJlQ9x7H3Qx2JMgDps0Bjse88mZhYJTlYz3aPrNVZHDRMr+caCP7HhhAWbdYa 3xFKer/MKJbnP3jnGo7dfzBTtNuH7pBn5ooNU1677MDupYA8Y6wg13jiCyX5Dgo5m/rK uVVsWznWpsxSpeD7ErJ2GI8VsKbYW01rs0UtiLt9aoqkQj6iFcrK6jfbAjECDJVPnhW9 w3eP1BZzuH5yVuZyRBufK02caSXSLKdUI39gfUoshXCP39F2XN5nbXmXAt0KMqvstqCX +tgtXUVC07ZFvFgWZknYMctBp/8IaR0Y97WKiS7aDGv/8GwmkkgmKrxX5yqpWmSI45/G F37Q== X-Gm-Message-State: AOJu0YwbYxTW1IcMNa96QvWTx5GHiCKnsxdMNQz2Az5adt0JrqcKdXTT wOHNuesyVPATwLxFS3Bg+OhR2kYT1xw5fx4EEneRemCaQa65XfFtOx+DIUnB4J9Ta7beY8TL3Kw XswjcYSG5Vi0Wxt76TD+JtO1SvESefVTpREf8u/xsSx2uf79aH0ZaCKCh807hENIPE31tJLpKMA EcPK2ngiFMSSIirDpJ+Ux2m4FaeY3DSL7bEg== X-Received: by 2002:a05:622a:64f:b0:458:2894:9870 with SMTP id d75a77b69052e-45828949c51mr83205741cf.10.1725912713701; Mon, 09 Sep 2024 13:11:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEcsa3tazYtKBEE6ZNofjJygtvH5SjHMnXo3i9kpBbCjJMj7Cub5GJrFW6Ef0H8m+mxVLZAew== X-Received: by 2002:a05:622a:64f:b0:458:2894:9870 with SMTP id d75a77b69052e-45828949c51mr83205091cf.10.1725912713171; Mon, 09 Sep 2024 13:11:53 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.11.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:11:52 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , David Hildenbrand , Stefan Hajnoczi , qemu-stable@nongnu.org, Paolo Bonzini , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= Subject: [PULL 2/9] softmmu/physmem: fix memory leak in dirty_memory_extend() Date: Mon, 9 Sep 2024 16:11:40 -0400 Message-ID: <20240909201147.3761639-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: David Hildenbrand As reported by Peter, we might be leaking memory when removing the highest RAMBlock (in the weird ram_addr_t space), and adding a new one. We will fail to realize that we already allocated bitmaps for more dirty memory blocks, and effectively discard the pointers to them. Fix it by getting rid of last_ram_page() and by remembering the number of dirty memory blocks that have been allocated already. While at it, let's use "unsigned int" for the number of blocks, which should be sufficient until we reach ~32 exabytes. Looks like this leak was introduced as we switched from using a single bitmap_zero_extend() to allocating multiple bitmaps: bitmap_zero_extend() relies on g_renew() which should have taken care of this. Resolves: https://lkml.kernel.org/r/CAFEAcA-k7a+VObGAfCFNygQNfCKL=AfX6A4kScq=VSSK0peqPg@mail.gmail.com Reported-by: Peter Maydell Fixes: 5b82b703b69a ("memory: RCU ram_list.dirty_memory[] for safe RAM hotplug") Reviewed-by: Stefan Hajnoczi Reviewed-by: Peter Xu Tested-by: Peter Maydell Cc: qemu-stable@nongnu.org Cc: Stefan Hajnoczi Cc: Paolo Bonzini Cc: Peter Xu Cc: Philippe Mathieu-Daudé Signed-off-by: David Hildenbrand Link: https://lore.kernel.org/r/20240828090743.128647-1-david@redhat.com Signed-off-by: Peter Xu --- include/exec/ramlist.h | 1 + system/physmem.c | 35 +++++++++-------------------------- 2 files changed, 10 insertions(+), 26 deletions(-) diff --git a/include/exec/ramlist.h b/include/exec/ramlist.h index 2ad2a81acc..d9cfe530be 100644 --- a/include/exec/ramlist.h +++ b/include/exec/ramlist.h @@ -50,6 +50,7 @@ typedef struct RAMList { /* RCU-enabled, writes protected by the ramlist lock. */ QLIST_HEAD(, RAMBlock) blocks; DirtyMemoryBlocks *dirty_memory[DIRTY_MEMORY_NUM]; + unsigned int num_dirty_blocks; uint32_t version; QLIST_HEAD(, RAMBlockNotifier) ramblock_notifiers; } RAMList; diff --git a/system/physmem.c b/system/physmem.c index 971bfa0855..d71a2b1bbd 100644 --- a/system/physmem.c +++ b/system/physmem.c @@ -1534,18 +1534,6 @@ static ram_addr_t find_ram_offset(ram_addr_t size) return offset; } -static unsigned long last_ram_page(void) -{ - RAMBlock *block; - ram_addr_t last = 0; - - RCU_READ_LOCK_GUARD(); - RAMBLOCK_FOREACH(block) { - last = MAX(last, block->offset + block->max_length); - } - return last >> TARGET_PAGE_BITS; -} - static void qemu_ram_setup_dump(void *addr, ram_addr_t size) { int ret; @@ -1799,13 +1787,11 @@ void qemu_ram_msync(RAMBlock *block, ram_addr_t start, ram_addr_t length) } /* Called with ram_list.mutex held */ -static void dirty_memory_extend(ram_addr_t old_ram_size, - ram_addr_t new_ram_size) +static void dirty_memory_extend(ram_addr_t new_ram_size) { - ram_addr_t old_num_blocks = DIV_ROUND_UP(old_ram_size, - DIRTY_MEMORY_BLOCK_SIZE); - ram_addr_t new_num_blocks = DIV_ROUND_UP(new_ram_size, - DIRTY_MEMORY_BLOCK_SIZE); + unsigned int old_num_blocks = ram_list.num_dirty_blocks; + unsigned int new_num_blocks = DIV_ROUND_UP(new_ram_size, + DIRTY_MEMORY_BLOCK_SIZE); int i; /* Only need to extend if block count increased */ @@ -1837,6 +1823,8 @@ static void dirty_memory_extend(ram_addr_t old_ram_size, g_free_rcu(old_blocks, rcu); } } + + ram_list.num_dirty_blocks = new_num_blocks; } static void ram_block_add(RAMBlock *new_block, Error **errp) @@ -1846,11 +1834,9 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) RAMBlock *block; RAMBlock *last_block = NULL; bool free_on_error = false; - ram_addr_t old_ram_size, new_ram_size; + ram_addr_t ram_size; Error *err = NULL; - old_ram_size = last_ram_page(); - qemu_mutex_lock_ramlist(); new_block->offset = find_ram_offset(new_block->max_length); @@ -1901,11 +1887,8 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) } } - new_ram_size = MAX(old_ram_size, - (new_block->offset + new_block->max_length) >> TARGET_PAGE_BITS); - if (new_ram_size > old_ram_size) { - dirty_memory_extend(old_ram_size, new_ram_size); - } + ram_size = (new_block->offset + new_block->max_length) >> TARGET_PAGE_BITS; + dirty_memory_extend(ram_size); /* Keep the list sorted from biggest to smallest block. Unlike QTAILQ, * QLIST (which has an RCU-friendly variant) does not have insertion at * tail, so save the last element in last_block. From patchwork Mon Sep 9 20:11:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABD7AECE58A for ; Mon, 9 Sep 2024 20:13:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkjx-0000Bx-S6; Mon, 09 Sep 2024 16:12:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjw-00005X-C1 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkju-00050p-Nv for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TNxl6tlilb/RVxkqHnM+uvvHD1t1Pzft8y1Y+cNR2E4=; b=K46yrog3J6fXAvWeWqkNWwlNfTlG+mzvwD8DR8O/zhcs0bt6rERaYOU4BHkqDqWcq/TTKN WfVRVNpWkcUGDFiYN4oLI65N/Aq25gwJjPWztojD54YpvaCckdPVa09QM4H09cxKCrZPsc VBH4qQIn+nJXit9JoEPpH9m0zZ9EA7s= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-664-j-WYBm8fMAC7qp5uGFnc3A-1; Mon, 09 Sep 2024 16:11:56 -0400 X-MC-Unique: j-WYBm8fMAC7qp5uGFnc3A-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4581b5172a6so65174981cf.3 for ; Mon, 09 Sep 2024 13:11:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912716; x=1726517516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TNxl6tlilb/RVxkqHnM+uvvHD1t1Pzft8y1Y+cNR2E4=; b=ctLV5xJXy7L4GNBd/PqPNHi/7Auf9CBY23ynufN/3+v8grU0Ie29awFI60xSJQjzV5 a92scquPO3Gqhm7AxBhWV04f+I3iCefm6J2RFTfeBrqOWjgp5SgxHi8Q1+GNpjMMWE4W SwRrLy1c2cGvFLVBS91e2pncjVevx0Fv43NkFM97AzSc9mi4+jwMe31/KTBeHLTWcckk 9MT9D4A+NHv0Tu2rx9aLGQmPLm+vljEtUjv3OT2w3YbjCPoXK0+2Xc0DsWlKw3NDtDIu qJeaGkeudsheEbbtgtkVPaMUlr3EB+jMJzUegHAxNYAVmJXbq/go/v+mcR5AkYSxoGe3 kJvg== X-Gm-Message-State: AOJu0Yymf3WmVs5JdfYBUY8BAxpl1GG0NqABNbo5vLigglTizNF2NU15 s7PG/W3I+qxfwL9PTd6kGrWd8VHFQKDJS7NoX19fuaW+8ZbVdMp+XMpgcdSWB3zXrGuoxGDLgMK XSttpndRojZlMz2KqJBE48gUoBdyVCSTRS8BmGauuQH3bK1bwDKsh1x7PBY9Mwndeqvod4gFdGy 5k1DVsbIkDmx5DLNuGBW9DNCL8/nIuJtUSTQ== X-Received: by 2002:a05:622a:487:b0:453:7533:a64f with SMTP id d75a77b69052e-45820102602mr124115761cf.21.1725912715825; Mon, 09 Sep 2024 13:11:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH4Qi28kgnyxXDsZksZp4q0S/oFQ7mypcDMp5xlC5zYzVjKj6fn319MCAJZLYEx6VbtupZkYw== X-Received: by 2002:a05:622a:487:b0:453:7533:a64f with SMTP id d75a77b69052e-45820102602mr124115391cf.21.1725912715353; Mon, 09 Sep 2024 13:11:55 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.11.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:11:53 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Thomas Huth Subject: [PULL 3/9] ci: migration: Don't run python tests in the compat job Date: Mon, 9 Sep 2024 16:11:41 -0400 Message-ID: <20240909201147.3761639-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Fabiano Rosas The vmstate-checker-script test has a bug that makes it flaky. It was also committed by mistake and will be removed. Since the migration-compat job takes the tests from the build-previous job instead of the current HEAD, neither a fix or a removal of the test will take effect for this release. Disable the faulty/undesirable test by taking advantage that it only runs if the PYTHON environment variable is set. This also disables the analyze-migration-script test, but this is fine because that test doesn't have migration compatibility implications. Signed-off-by: Fabiano Rosas Reviewed-by: Thomas Huth Link: https://lore.kernel.org/r/20240905185445.8179-1-farosas@suse.de [peterx: Added a TODO to remove the line after 9.2 release, per thuth] Signed-off-by: Peter Xu --- .gitlab-ci.d/buildtest.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index 1d2afae996..cfc51be08a 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -212,6 +212,14 @@ build-previous-qemu: # testing an old QEMU against new features/tests that it is not # compatible with. - cd build-previous + # Don't allow python-based tests to run. The + # vmstate-checker-script test has a race that causes it to fail + # sometimes. It cannot be fixed it because this job runs the test + # from the old QEMU version. The test will be removed on master, + # but this job will only see the change in the next release. + # + # TODO: remove this line after 9.2 release + - unset PYTHON # old to new - QTEST_QEMU_BINARY_SRC=./qemu-system-${TARGET} QTEST_QEMU_BINARY=../build/qemu-system-${TARGET} ./tests/qtest/migration-test From patchwork Mon Sep 9 20:11:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E765ECE587 for ; Mon, 9 Sep 2024 20:13:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkk1-0000Qu-9v; Mon, 09 Sep 2024 16:12:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjz-0000Jp-KW for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjx-00052K-J8 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HNelb35Lz3HQub41/Z2JM0wVsojlW9d18+WONVrmQ+4=; b=Z7MgcqV+oltulFj63pYECY8mYLxvKOiJ97bdObmBuvw2kiH1QBXrwj2Wq2LggJEX3F/mOS vX9TMNbINKon9dzL9z1OeFThVldZH3Yya+32REKepw7lZmjSfDIWmKKbnNDSOCF+YyMCHP f2coAnkmrjXvt/bnq4APDF8POnDMHrM= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-660-IgE0wJ-0M6SsM6MVj1vShg-1; Mon, 09 Sep 2024 16:11:59 -0400 X-MC-Unique: IgE0wJ-0M6SsM6MVj1vShg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-457cde3fd7bso60821cf.3 for ; Mon, 09 Sep 2024 13:11:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912718; x=1726517518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HNelb35Lz3HQub41/Z2JM0wVsojlW9d18+WONVrmQ+4=; b=U4FGHxlrfMJnaletVdr/qc0f1YSkkbdQmGAS24hqCL8PnNOsfytgFpSBQeoO010jUJ VXsnD/rWVun71H0wkaQPGzOpzS8SMVQchQXlYNo0En2MY/SZrGiYakp4eGcL2ko+LC/E AIcwLctDAka3wwvDzDyWNJMXgYXmJcLufysDqmzlEHlQo7iEA4IwyYkFginVexajvpbp uc+XHx0Vr9FMqOe0EMTgcWSxw7w151jLd1hl+wG3551/8otqfEg0AzcfffP0skuF5QhY 9JfRwICyKx9u8ohI9icz+XUfkcR07/UUOKORMoP+fBVi4c++CFhDlvbow+SKmjR3c3Aq ltDg== X-Gm-Message-State: AOJu0YxW7FsYJ2GZcNJkENOVn5oBuv6853RZv26yp25D6bi2imR0depp Wf1ZK2O3E9v0yqAieWUWt3KE1zCvgnXl1iIvjH9P/6xdDNMzyqwpzC4aVoeEBNLA+0W7XB6qjac fKnCn3NPZSuyymo5ay7WYxTu6ry3B1B+MAEAW/brHm7dqmc7pwWfQVO8qjoJcPDcsllValIoQLw 438cTt8iIBChMmTKG1CNELsC5/KKj0I93mIA== X-Received: by 2002:ac8:584e:0:b0:456:45c6:2c30 with SMTP id d75a77b69052e-4581f6f6a7bmr100772871cf.53.1725912718098; Mon, 09 Sep 2024 13:11:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEOrEkLSxXtaOjmBMWOBgp2bxdh3qkgdOu9VnHMg5IuBwaisJ/KzDkcXGGe+iHlMRl/IVwKUQ== X-Received: by 2002:ac8:584e:0:b0:456:45c6:2c30 with SMTP id d75a77b69052e-4581f6f6a7bmr100772261cf.53.1725912717448; Mon, 09 Sep 2024 13:11:57 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.11.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:11:56 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Yuan Liu , Nanhai Zou , Yichen Wang Subject: [PULL 4/9] docs/migration: add qatzip compression feature Date: Mon, 9 Sep 2024 16:11:42 -0400 Message-ID: <20240909201147.3761639-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Yuan Liu add Intel QATzip compression method introduction Reviewed-by: Nanhai Zou Reviewed-by: Peter Xu Reviewed-by: Fabiano Rosas Signed-off-by: Yuan Liu Signed-off-by: Yichen Wang Link: https://lore.kernel.org/r/20240830232722.58272-2-yichen.wang@bytedance.com Signed-off-by: Peter Xu --- docs/devel/migration/features.rst | 1 + docs/devel/migration/qatzip-compression.rst | 165 ++++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 docs/devel/migration/qatzip-compression.rst diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst index 58f8fd9e16..8f431d52f9 100644 --- a/docs/devel/migration/features.rst +++ b/docs/devel/migration/features.rst @@ -14,3 +14,4 @@ Migration has plenty of features to support different use cases. CPR qpl-compression uadk-compression + qatzip-compression diff --git a/docs/devel/migration/qatzip-compression.rst b/docs/devel/migration/qatzip-compression.rst new file mode 100644 index 0000000000..862b383164 --- /dev/null +++ b/docs/devel/migration/qatzip-compression.rst @@ -0,0 +1,165 @@ +================== +QATzip Compression +================== +In scenarios with limited network bandwidth, the ``QATzip`` solution can help +users save a lot of host CPU resources by accelerating compression and +decompression through the Intel QuickAssist Technology(``QAT``) hardware. + + +The following test was conducted using 8 multifd channels and 10Gbps network +bandwidth. The results show that, compared to zstd, ``QATzip`` significantly +saves CPU resources on the sender and reduces migration time. Compared to the +uncompressed solution, ``QATzip`` greatly improves the dirty page processing +capability, indicated by the Pages per Second metric, and also reduces the +total migration time. + +:: + + VM Configuration: 16 vCPU and 64G memory + VM Workload: all vCPUs are idle and 54G memory is filled with Silesia data. + QAT Devices: 4 + |-----------|--------|---------|----------|----------|------|------| + |8 Channels |Total |down |throughput|pages per | send | recv | + | |time(ms)|time(ms) |(mbps) |second | cpu %| cpu% | + |-----------|--------|---------|----------|----------|------|------| + |qatzip | 16630| 28| 10467| 2940235| 160| 360| + |-----------|--------|---------|----------|----------|------|------| + |zstd | 20165| 24| 8579| 2391465| 810| 340| + |-----------|--------|---------|----------|----------|------|------| + |none | 46063| 40| 10848| 330240| 45| 85| + |-----------|--------|---------|----------|----------|------|------| + + +QATzip Compression Framework +============================ + +``QATzip`` is a user space library which builds on top of the Intel QuickAssist +Technology to provide extended accelerated compression and decompression +services. + +For more ``QATzip`` introduction, please refer to `QATzip Introduction +`_ + +:: + + +----------------+ + | MultiFd Thread | + +-------+--------+ + | + | compress/decompress + +-------+--------+ + | QATzip library | + +-------+--------+ + | + +-------+--------+ + | QAT library | + +-------+--------+ + | user space + --------+--------------------- + | kernel space + +------+-------+ + | QAT Driver | + +------+-------+ + | + +------+-------+ + | QAT Devices | + +--------------+ + + +QATzip Installation +------------------- + +The ``QATzip`` installation package has been integrated into some Linux +distributions and can be installed directly. For example, the Ubuntu Server +24.04 LTS system can be installed using below command + +.. code-block:: shell + + #apt search qatzip + libqatzip-dev/noble 1.2.0-0ubuntu3 amd64 + Intel QuickAssist user space library development files + + libqatzip3/noble 1.2.0-0ubuntu3 amd64 + Intel QuickAssist user space library + + qatzip/noble,now 1.2.0-0ubuntu3 amd64 [installed] + Compression user-space tool for Intel QuickAssist Technology + + #sudo apt install libqatzip-dev libqatzip3 qatzip + +If your system does not support the ``QATzip`` installation package, you can +use the source code to build and install, please refer to `QATzip source code installation +`_ + +QAT Hardware Deployment +----------------------- + +``QAT`` supports physical functions(PFs) and virtual functions(VFs) for +deployment, and users can configure ``QAT`` resources for migration according +to actual needs. For more details about ``QAT`` deployment, please refer to +`Intel QuickAssist Technology Documentation +`_ + +For more ``QAT`` hardware introduction, please refer to `intel-quick-assist-technology-overview +`_ + +How To Use QATzip Compression +============================= + +1 - Install ``QATzip`` library + +2 - Build ``QEMU`` with ``--enable-qatzip`` parameter + + E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qatzip`` + +3 - Set ``migrate_set_parameter multifd-compression qatzip`` + +4 - Set ``migrate_set_parameter multifd-qatzip-level comp_level``, the default +comp_level value is 1, and it supports levels from 1 to 9 + +QAT Memory Requirements +======================= + +The user needs to reserve system memory for the QAT memory management to +allocate DMA memory. The size of the reserved system memory depends on the +number of devices used for migration and the number of multifd channels. + +Because memory usage depends on QAT configuration, please refer to `QAT Memory +Driver Queries +`_ +for memory usage calculation. + +.. list-table:: An example of a PF used for migration + :header-rows: 1 + + * - Number of channels + - Sender memory usage + - Receiver memory usage + * - 2 + - 10M + - 10M + * - 4 + - 12M + - 14M + * - 8 + - 16M + - 20M + +How To Choose Between QATzip and QPL +==================================== +Starting from 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids +processor(``SPR``), multiple built-in accelerators are supported including +``QAT`` and ``IAA``. The former can accelerate ``QATzip`` and the latter is +used to accelerate ``QPL``. + +Here are some suggestions: + +1 - If the live migration scenario is limited by network bandwidth and ``QAT`` +hardware resources exceed ``IAA``, use the ``QATzip`` method, which can save a +lot of host CPU resources for compression. + +2 - If the system cannot support shared virtual memory (SVM) technology, use +the ``QATzip`` method because ``QPL`` performance is not good without SVM +support. + +3 - For other scenarios, use the ``QPL`` method first. From patchwork Mon Sep 9 20:11:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86AC3ECE588 for ; Mon, 9 Sep 2024 20:13:15 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkk2-0000Tf-1x; Mon, 09 Sep 2024 16:12:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk0-0000O4-LH for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkjy-00052b-T8 for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8TZl7MGrUC/7HcoL7VayMG078Iin7hnDhhYjY6hvtU=; b=HuOVEU42klydjvMMLNZ4pRrgphmya4x9hAwrw8KKytdAFgBDrgJ0AznKa+VvvmzzB5OFte PmpHOfYPEKgXSUEXhFMakhLi344UexEklyGZRI2AnICT30DoktSJricbTvOMHKS5Tr/0ON 1rj1vNJ8ShXuXrjcpquKG/e40+XR8Xo= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-610-p0G91ztjMYuzKYMxx1i7wA-1; Mon, 09 Sep 2024 16:12:01 -0400 X-MC-Unique: p0G91ztjMYuzKYMxx1i7wA-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-454e94467ceso88982011cf.1 for ; Mon, 09 Sep 2024 13:12:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912720; x=1726517520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t8TZl7MGrUC/7HcoL7VayMG078Iin7hnDhhYjY6hvtU=; b=F4kYtl7K+/JyAiPxrbgk8X1xqG8OzrZTFPAttNmbWP3j/BQxjF1632WkVp5nrpFB5o nhS7dVAcCY49YdGjHlyxaOLhVXsjVrphGnlJdra+ChESqaScjX9TakEqBb63xMyH5S0G rjbmuEW8Lp8Mq8007IckXiL4PQSv3fJPwndZqFE6vYPP2OWhkpl37F5d6PmkZXhBAauW Oecm3oiu/MVoRgJOdV/pH91EKlu40BAQuM5foCV1iAqqkouDHYIBSsx9rQFRceJubONe Wx/a0t1AAPKxEfyxvnyzogGcrRnIqMimfT/wOiVUeDuo3eFkB1YvY9eUPuEeZ43s2qK6 N7OA== X-Gm-Message-State: AOJu0YxuYvMVzURfV9sMNqnAo/i6QbsUTHTum2ziUe7btbvuvzN4cjad TJzU2wpZTXPJD2nuoBLSOFUXNchxqiCIaHK58vqHpW+vsEvMFV9z7+ZA/eg2NsNZ9Mhthbq6nQ5 P9s1D+aVULTvEWVgPRmASGoRly+yYBPxwzS8Ed5On64BIgSO4gnULaV7a/owi0EmKT7MUGArPFh XYC3as3/z+/sZzm3RPQXzkCsy9gTmpItfmxQ== X-Received: by 2002:a05:622a:1304:b0:458:34be:88c2 with SMTP id d75a77b69052e-45834be8cafmr56965001cf.48.1725912720504; Mon, 09 Sep 2024 13:12:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG2tAB2X8NKL/8XWan95k8iIiFYTn5HAmT94iRzyWGhu3aPSA8adlc8PqgWtmf3plqOvLpPfA== X-Received: by 2002:a05:622a:1304:b0:458:34be:88c2 with SMTP id d75a77b69052e-45834be8cafmr56964621cf.48.1725912720027; Mon, 09 Sep 2024 13:12:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.11.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:11:58 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Bryan Zhang , Hao Xiang , Yichen Wang Subject: [PULL 5/9] meson: Introduce 'qatzip' feature to the build system Date: Mon, 9 Sep 2024 16:11:43 -0400 Message-ID: <20240909201147.3761639-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Bryan Zhang Add a 'qatzip' feature, which is automatically disabled, and which depends on the QATzip library if enabled. Reviewed-by: Fabiano Rosas Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang Signed-off-by: Yichen Wang Link: https://lore.kernel.org/r/20240830232722.58272-3-yichen.wang@bytedance.com Signed-off-by: Peter Xu --- meson.build | 10 ++++++++++ meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 15 insertions(+) diff --git a/meson.build b/meson.build index fbda17c987..b89b713e79 100644 --- a/meson.build +++ b/meson.build @@ -1262,6 +1262,14 @@ if not get_option('uadk').auto() or have_system uadk = declare_dependency(dependencies: [libwd, libwd_comp]) endif endif + +qatzip = not_found +if not get_option('qatzip').auto() or have_system + qatzip = dependency('qatzip', version: '>=1.1.2', + required: get_option('qatzip'), + method: 'pkg-config') +endif + virgl = not_found have_vhost_user_gpu = have_tools and host_os == 'linux' and pixman.found() @@ -2412,6 +2420,7 @@ config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id) config_host_data.set('CONFIG_ZSTD', zstd.found()) config_host_data.set('CONFIG_QPL', qpl.found()) config_host_data.set('CONFIG_UADK', uadk.found()) +config_host_data.set('CONFIG_QATZIP', qatzip.found()) config_host_data.set('CONFIG_FUSE', fuse.found()) config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found()) config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found()) @@ -4535,6 +4544,7 @@ summary_info += {'lzfse support': liblzfse} summary_info += {'zstd support': zstd} summary_info += {'Query Processing Library support': qpl} summary_info += {'UADK Library support': uadk} +summary_info += {'qatzip support': qatzip} summary_info += {'NUMA host support': numa} summary_info += {'capstone': capstone} summary_info += {'libpmem support': libpmem} diff --git a/meson_options.txt b/meson_options.txt index 0269fa0f16..f7b652b30d 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -261,6 +261,8 @@ option('qpl', type : 'feature', value : 'auto', description: 'Query Processing Library support') option('uadk', type : 'feature', value : 'auto', description: 'UADK Library support') +option('qatzip', type: 'feature', value: 'auto', + description: 'QATzip compression support') option('fuse', type: 'feature', value: 'auto', description: 'FUSE block device export') option('fuse_lseek', type : 'feature', value : 'auto', diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh index c97079a38c..5f377a6d81 100644 --- a/scripts/meson-buildoptions.sh +++ b/scripts/meson-buildoptions.sh @@ -163,6 +163,7 @@ meson_options_help() { printf "%s\n" ' pixman pixman support' printf "%s\n" ' plugins TCG plugins via shared library loading' printf "%s\n" ' png PNG support with libpng' + printf "%s\n" ' qatzip QATzip compression support' printf "%s\n" ' qcow1 qcow1 image format support' printf "%s\n" ' qed qed image format support' printf "%s\n" ' qga-vss build QGA VSS support (broken with MinGW)' @@ -427,6 +428,8 @@ _meson_option_parse() { --enable-png) printf "%s" -Dpng=enabled ;; --disable-png) printf "%s" -Dpng=disabled ;; --prefix=*) quote_sh "-Dprefix=$2" ;; + --enable-qatzip) printf "%s" -Dqatzip=enabled ;; + --disable-qatzip) printf "%s" -Dqatzip=disabled ;; --enable-qcow1) printf "%s" -Dqcow1=enabled ;; --disable-qcow1) printf "%s" -Dqcow1=disabled ;; --enable-qed) printf "%s" -Dqed=enabled ;; From patchwork Mon Sep 9 20:11:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9C140ECE588 for ; Mon, 9 Sep 2024 20:13:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkk6-0000ir-0B; Mon, 09 Sep 2024 16:12:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk4-0000dD-Du for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk2-00053O-EQ for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zyF+uB2XFDTWxO0PkzZtc9eGB9NRgyi0toAwC4na6TY=; b=ic7mX/jYSzQp6eClJGjpytw/F4/HMaAtz3WwSCWZshZDrCFHjTOMOgOJzca7SRvfla+BAY 8GNFvS4qYXE5jD3lJBJ6kcyMQoWTN4LQXDEeksEs+EkIg9R66m1mas8nR6ySivFcNTi2si OVGAI+7QWa03Rx66aR6FXXilrQ05O1I= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-446-gzn0EMHlOU-oQQAjoEQg7Q-1; Mon, 09 Sep 2024 16:12:04 -0400 X-MC-Unique: gzn0EMHlOU-oQQAjoEQg7Q-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3e04ec0155dso208797b6e.0 for ; Mon, 09 Sep 2024 13:12:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912723; x=1726517523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zyF+uB2XFDTWxO0PkzZtc9eGB9NRgyi0toAwC4na6TY=; b=BREVmDyVj+e0rRhANxHc1yD8OwKFzaomqyZ2AbRHzCCwseBJf5fWeSQUu2Z5uawWm3 iZI6WAQ1nd5LNdu5/P1bydf0xGQfbmyXeHdNbR93YH7o2Q8eLvexCxfmXFuO/xP5sxtn DmFnMYYbeBwjj7FQ2abFH4L8zZcnuB6m7Dzg3R9bQTY2mjaSIlVBrFuAhi87GGF7mPtB DxvO7nMxtWuh1S4zv6sjeVB5D5vRx99SmdxH8dvN7U+dKr7NqmQgkH21sJiQdJuXiYyS 1rInVf01FsWhVTI7SqaTjhB1Mqb0+p+jcjs0XzTgpffFg23MvpsFLpXGIDI+KEzr2F4x 7GpQ== X-Gm-Message-State: AOJu0Yw5V8t6ZwQZ6MGzOu1yQTjZEfkFnyuI5fH/chqna35QLMwb5b02 BRkKCTeZ18hAZ2xTG2pNybbxg5NKZQM8pvZway7oBI+HsEsX0j6WtsTtRSqrwNYkaZg+QKOKxJd H11dj6tZJn5sC2A7FVlhTM8j3FdDxrDt/sxaW50hyFyzLIiN9LhLPh57McIxCv51h/MbFzyh4kK 7/egQfRxSQSalD/ar4WrAN/FeP755XPDJMhA== X-Received: by 2002:a05:6808:1889:b0:3db:1516:2bcc with SMTP id 5614622812f47-3e037ae4badmr5722542b6e.31.1725912722985; Mon, 09 Sep 2024 13:12:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGIaqtkXI8xWWU00jT15xA39fI+oshHeb894QR6JyHUUl5dqe7a/eym2PXiK2Fe0YWpjikALQ== X-Received: by 2002:a05:6808:1889:b0:3db:1516:2bcc with SMTP id 5614622812f47-3e037ae4badmr5722501b6e.31.1725912722425; Mon, 09 Sep 2024 13:12:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.12.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:12:00 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Bryan Zhang , Markus Armbruster , Hao Xiang , Yichen Wang , Prasad Pandit Subject: [PULL 6/9] migration: Add migration parameters for QATzip Date: Mon, 9 Sep 2024 16:11:44 -0400 Message-ID: <20240909201147.3761639-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Bryan Zhang Adds support for migration parameters to control QATzip compression level. Acked-by: Markus Armbruster Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang Signed-off-by: Yichen Wang Reviewed-by: Fabiano Rosas Reviewed-by: Prasad Pandit Link: https://lore.kernel.org/r/20240830232722.58272-4-yichen.wang@bytedance.com Signed-off-by: Peter Xu --- qapi/migration.json | 18 ++++++++++++++++++ migration/options.h | 1 + migration/migration-hmp-cmds.c | 4 ++++ migration/options.c | 34 ++++++++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index 7324571e92..f4c27426c8 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -792,6 +792,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression @@ -852,6 +857,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', + 'multifd-qatzip-level', 'block-bitmap-mapping', { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', @@ -967,6 +973,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression @@ -1040,6 +1051,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', @@ -1171,6 +1183,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression @@ -1241,6 +1258,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', diff --git a/migration/options.h b/migration/options.h index a2397026db..a0bd6edc06 100644 --- a/migration/options.h +++ b/migration/options.h @@ -78,6 +78,7 @@ uint64_t migrate_max_postcopy_bandwidth(void); int migrate_multifd_channels(void); MultiFDCompression migrate_multifd_compression(void); int migrate_multifd_zlib_level(void); +int migrate_multifd_qatzip_level(void); int migrate_multifd_zstd_level(void); uint8_t migrate_throttle_trigger_threshold(void); const char *migrate_tls_authz(void); diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 7d608d26e1..28165cfc9e 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -576,6 +576,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zlib_level = true; visit_type_uint8(v, param, &p->multifd_zlib_level, &err); break; + case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL: + p->has_multifd_qatzip_level = true; + visit_type_uint8(v, param, &p->multifd_qatzip_level, &err); + break; case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL: p->has_multifd_zstd_level = true; visit_type_uint8(v, param, &p->multifd_zstd_level, &err); diff --git a/migration/options.c b/migration/options.c index 645f55003d..147cd2b8fd 100644 --- a/migration/options.c +++ b/migration/options.c @@ -55,6 +55,13 @@ #define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE /* 0: means nocompress, 1: best speed, ... 9: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1 +/* + * 1: best speed, ... 9: best compress ratio + * There is some nuance here. Refer to QATzip documentation to understand + * the mapping of QATzip levels to standard deflate levels. + */ +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1 + /* 0: means nocompress, 1: best speed, ... 20: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1 @@ -123,6 +130,9 @@ Property migration_properties[] = { DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState, parameters.multifd_zlib_level, DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL), + DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState, + parameters.multifd_qatzip_level, + DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL), DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL), @@ -787,6 +797,13 @@ int migrate_multifd_zlib_level(void) return s->parameters.multifd_zlib_level; } +int migrate_multifd_qatzip_level(void) +{ + MigrationState *s = migrate_get_current(); + + return s->parameters.multifd_qatzip_level; +} + int migrate_multifd_zstd_level(void) { MigrationState *s = migrate_get_current(); @@ -892,6 +909,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_compression = s->parameters.multifd_compression; params->has_multifd_zlib_level = true; params->multifd_zlib_level = s->parameters.multifd_zlib_level; + params->has_multifd_qatzip_level = true; + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; params->has_xbzrle_cache_size = true; @@ -946,6 +965,7 @@ void migrate_params_init(MigrationParameters *params) params->has_multifd_channels = true; params->has_multifd_compression = true; params->has_multifd_zlib_level = true; + params->has_multifd_qatzip_level = true; params->has_multifd_zstd_level = true; params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true; @@ -1038,6 +1058,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp) return false; } + if (params->has_multifd_qatzip_level && + ((params->multifd_qatzip_level > 9) || + (params->multifd_qatzip_level < 1))) { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_qatzip_level", + "a value between 1 and 9"); + return false; + } + if (params->has_multifd_zstd_level && (params->multifd_zstd_level > 20)) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level", @@ -1195,6 +1223,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params, if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } + if (params->has_multifd_qatzip_level) { + dest->multifd_qatzip_level = params->multifd_qatzip_level; + } if (params->has_multifd_zlib_level) { dest->multifd_zlib_level = params->multifd_zlib_level; } @@ -1315,6 +1346,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } + if (params->has_multifd_qatzip_level) { + s->parameters.multifd_qatzip_level = params->multifd_qatzip_level; + } if (params->has_multifd_zlib_level) { s->parameters.multifd_zlib_level = params->multifd_zlib_level; } From patchwork Mon Sep 9 20:11:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF9F3ECE588 for ; Mon, 9 Sep 2024 20:13:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkk8-0000tb-HO; Mon, 09 Sep 2024 16:12:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk7-0000nu-2c for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk4-00053V-JV for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912727; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gp9q23FGsWBNBV5XuIUFLE2Uwzbt24RL4MjU9ohmxj4=; b=Mg2FBGMPTmgTadPOXMRL1mdl8CYbELf1MTYeSjHedR4o4pYB9pLVII5MJJt12O6iHVieRT NjJNtkAn3z7IOH9uFlfjZ37DQTfVJ1xnhunSgyPwbTA9/Auj6QOfumyYPWa2z5h0as4lxY aHT2fCV1m8OosqckE/PaFQ8uVZUXYvY= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-481-kXbs6j1sNgq2FxoFYwu8SA-1; Mon, 09 Sep 2024 16:12:06 -0400 X-MC-Unique: kXbs6j1sNgq2FxoFYwu8SA-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4581253bd26so45577061cf.1 for ; Mon, 09 Sep 2024 13:12:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912725; x=1726517525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Gp9q23FGsWBNBV5XuIUFLE2Uwzbt24RL4MjU9ohmxj4=; b=CvecA09r0saZQ+ANeyk6TAP+XCs0mXAe7MSHGuOIlA956IaI4Q7tGuqxNtYP2UdYAc Yz4Dw/MvvNAqLpcxUjeugzeLp4QtDCLR6ka/JZINFLAM2G5VOuL0himWoQzTnR2YUf7X juIOyoZ8uxsPJdi0vNX2RyWBuaJ4LyhW/bm7Y15maPGVdrQFy7ZkwqV/jKqixQdZqTtj ckHfg3oSasx04GogS8n2ybsshQd7YykiAq6iN6lRVMyH83ErrC0pIMsghoNV2l+xAtXM SDXZU/ur05nS9D1PmwF6FOg8lYBRgx4aP0LNIdVXw8sFxgwmUZtSJl89U+FXR0X5d+jk bXiw== X-Gm-Message-State: AOJu0YwmQUXUq3AQy0VKOCt6H5lA/aRqMFOpJNlgo1DcASsv6sAaWJPq L8OE0rEYRUgNKnMOGQWoIU1UAuCP5PGbDUwNP1Hyt6M90PLxkGYkwmfZsbXO1XYUotFPmD0WRS4 Amk82b/89TW7vb+saFDWdNioWaw1jzCeCZhk80hOf8K/lrsOuYNFgXsD+9TiPeokW6T6tXCuOnb BHArchy0aEALBlmSp5iJXaMgvzOp/9WFC6uQ== X-Received: by 2002:ac8:7dc6:0:b0:458:3324:fff1 with SMTP id d75a77b69052e-458332502bcmr63956481cf.48.1725912725315; Mon, 09 Sep 2024 13:12:05 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEVrdtBovLGoHqDbqIKXvA5FQz+4LuoaZsSoQDE6npIXDr2JlwI1WIaxF3SG2JpihBeeU9UfA== X-Received: by 2002:ac8:7dc6:0:b0:458:3324:fff1 with SMTP id d75a77b69052e-458332502bcmr63955771cf.48.1725912724764; Mon, 09 Sep 2024 13:12:04 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:12:03 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Bryan Zhang , Markus Armbruster , Prasad Pandit , Hao Xiang , Yichen Wang Subject: [PULL 7/9] migration: Introduce 'qatzip' compression method Date: Mon, 9 Sep 2024 16:11:45 -0400 Message-ID: <20240909201147.3761639-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Bryan Zhang Adds support for 'qatzip' as an option for the multifd compression method parameter, and implements using QAT for 'qatzip' compression and decompression. Acked-by: Markus Armbruster Reviewed-by: Fabiano Rosas Reviewed-by: Prasad Pandit Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang Signed-off-by: Yichen Wang Link: https://lore.kernel.org/r/20240830232722.58272-5-yichen.wang@bytedance.com Signed-off-by: Peter Xu --- qapi/migration.json | 3 + migration/multifd.h | 5 +- hw/core/qdev-properties-system.c | 2 +- migration/multifd-qatzip.c | 394 +++++++++++++++++++++++++++++++ migration/meson.build | 1 + 5 files changed, 402 insertions(+), 3 deletions(-) create mode 100644 migration/multifd-qatzip.c diff --git a/qapi/migration.json b/qapi/migration.json index f4c27426c8..f1b7103dc8 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -561,6 +561,8 @@ # # @zstd: use zstd compression method. # +# @qatzip: use qatzip compression method. (Since 9.2) +# # @qpl: use qpl compression method. Query Processing Library(qpl) is # based on the deflate compression algorithm and use the Intel # In-Memory Analytics Accelerator(IAA) accelerated compression and @@ -573,6 +575,7 @@ { 'enum': 'MultiFDCompression', 'data': [ 'none', 'zlib', { 'name': 'zstd', 'if': 'CONFIG_ZSTD' }, + { 'name': 'qatzip', 'if': 'CONFIG_QATZIP'}, { 'name': 'qpl', 'if': 'CONFIG_QPL' }, { 'name': 'uadk', 'if': 'CONFIG_UADK' } ] } diff --git a/migration/multifd.h b/migration/multifd.h index 3bb96e9558..50d58c0c9c 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -36,14 +36,15 @@ MultiFDRecvData *multifd_get_recv_data(void); /* Multifd Compression flags */ #define MULTIFD_FLAG_SYNC (1 << 0) -/* We reserve 4 bits for compression methods */ -#define MULTIFD_FLAG_COMPRESSION_MASK (0xf << 1) +/* We reserve 5 bits for compression methods */ +#define MULTIFD_FLAG_COMPRESSION_MASK (0x1f << 1) /* we need to be compatible. Before compression value was 0 */ #define MULTIFD_FLAG_NOCOMP (0 << 1) #define MULTIFD_FLAG_ZLIB (1 << 1) #define MULTIFD_FLAG_ZSTD (2 << 1) #define MULTIFD_FLAG_QPL (4 << 1) #define MULTIFD_FLAG_UADK (8 << 1) +#define MULTIFD_FLAG_QATZIP (16 << 1) /* This value needs to be a multiple of qemu_target_page_size() */ #define MULTIFD_PACKET_SIZE (512 * 1024) diff --git a/hw/core/qdev-properties-system.c b/hw/core/qdev-properties-system.c index f13350b4fb..a56fbf728d 100644 --- a/hw/core/qdev-properties-system.c +++ b/hw/core/qdev-properties-system.c @@ -659,7 +659,7 @@ const PropertyInfo qdev_prop_fdc_drive_type = { const PropertyInfo qdev_prop_multifd_compression = { .name = "MultiFDCompression", .description = "multifd_compression values, " - "none/zlib/zstd/qpl/uadk", + "none/zlib/zstd/qpl/uadk/qatzip", .enum_table = &MultiFDCompression_lookup, .get = qdev_propinfo_get_enum, .set = qdev_propinfo_set_enum, diff --git a/migration/multifd-qatzip.c b/migration/multifd-qatzip.c new file mode 100644 index 0000000000..3c787ed879 --- /dev/null +++ b/migration/multifd-qatzip.c @@ -0,0 +1,394 @@ +/* + * Multifd QATzip compression implementation + * + * Copyright (c) Bytedance + * + * Authors: + * Bryan Zhang + * Hao Xiang + * Yichen Wang + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "exec/ramblock.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qapi/qapi-types-migration.h" +#include "options.h" +#include "multifd.h" +#include + +typedef struct { + /* + * Unique session for use with QATzip API + */ + QzSession_T sess; + + /* + * For compression: Buffer for pages to compress + * For decompression: Buffer for data to decompress + */ + uint8_t *in_buf; + uint32_t in_len; + + /* + * For compression: Output buffer of compressed data + * For decompression: Output buffer of decompressed data + */ + uint8_t *out_buf; + uint32_t out_len; +} QatzipData; + +/** + * qatzip_send_setup: Set up QATzip session and private buffers. + * + * @param p Multifd channel params + * @param errp Pointer to error, which will be set in case of error + * @return 0 on success, -1 on error (and *errp will be set) + */ +static int qatzip_send_setup(MultiFDSendParams *p, Error **errp) +{ + QatzipData *q; + QzSessionParamsDeflate_T params; + const char *err_msg; + int ret; + + q = g_new0(QatzipData, 1); + p->compress_data = q; + /* We need one extra place for the packet header */ + p->iov = g_new0(struct iovec, 2); + + /* + * Initialize QAT device with software fallback by default. This allows + * QATzip to use CPU path when QAT hardware reaches maximum throughput. + */ + ret = qzInit(&q->sess, true); + if (ret != QZ_OK && ret != QZ_DUPLICATE) { + err_msg = "qzInit failed"; + goto err; + } + + ret = qzGetDefaultsDeflate(¶ms); + if (ret != QZ_OK) { + err_msg = "qzGetDefaultsDeflate failed"; + goto err; + } + + /* Make sure to use configured QATzip compression level. */ + params.common_params.comp_lvl = migrate_multifd_qatzip_level(); + ret = qzSetupSessionDeflate(&q->sess, ¶ms); + if (ret != QZ_OK && ret != QZ_DUPLICATE) { + err_msg = "qzSetupSessionDeflate failed"; + goto err; + } + + if (MULTIFD_PACKET_SIZE > UINT32_MAX) { + err_msg = "packet size too large for QAT"; + goto err; + } + + q->in_len = MULTIFD_PACKET_SIZE; + /* + * PINNED_MEM is an enum from qatzip headers, which means to use + * kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device + * is not available or software fallback is used, the malloc flag needs to + * be set as COMMON_MEM. + */ + q->in_buf = qzMalloc(q->in_len, 0, PINNED_MEM); + if (!q->in_buf) { + q->in_buf = qzMalloc(q->in_len, 0, COMMON_MEM); + if (!q->in_buf) { + err_msg = "qzMalloc failed"; + goto err; + } + } + + q->out_len = qzMaxCompressedLength(MULTIFD_PACKET_SIZE, &q->sess); + q->out_buf = qzMalloc(q->out_len, 0, PINNED_MEM); + if (!q->out_buf) { + q->out_buf = qzMalloc(q->out_len, 0, COMMON_MEM); + if (!q->out_buf) { + err_msg = "qzMalloc failed"; + goto err; + } + } + + return 0; + +err: + error_setg(errp, "multifd %u: [sender] %s", p->id, err_msg); + return -1; +} + +/** + * qatzip_send_cleanup: Tear down QATzip session and release private buffers. + * + * @param p Multifd channel params + * @param errp Pointer to error, which will be set in case of error + * @return None + */ +static void qatzip_send_cleanup(MultiFDSendParams *p, Error **errp) +{ + QatzipData *q = p->compress_data; + + if (q) { + if (q->in_buf) { + qzFree(q->in_buf); + } + if (q->out_buf) { + qzFree(q->out_buf); + } + (void)qzTeardownSession(&q->sess); + (void)qzClose(&q->sess); + g_free(q); + } + + g_free(p->iov); + p->iov = NULL; + p->compress_data = NULL; +} + +/** + * qatzip_send_prepare: Compress pages and update IO channel info. + * + * @param p Multifd channel params + * @param errp Pointer to error, which will be set in case of error + * @return 0 on success, -1 on error (and *errp will be set) + */ +static int qatzip_send_prepare(MultiFDSendParams *p, Error **errp) +{ + MultiFDPages_t *pages = p->pages; + QatzipData *q = p->compress_data; + int ret; + unsigned int in_len, out_len; + + if (!multifd_send_prepare_common(p)) { + goto out; + } + + /* + * Unlike other multifd compression implementations, we use a non-streaming + * API and place all the data into one buffer, rather than sending each + * page to the compression API at a time. Based on initial benchmarks, the + * non-streaming API outperforms the streaming API. Plus, the logic in QEMU + * is friendly to using the non-streaming API anyway. If either of these + * statements becomes no longer true, we can revisit adding a streaming + * implementation. + */ + for (int i = 0; i < pages->normal_num; i++) { + memcpy(q->in_buf + (i * p->page_size), + pages->block->host + pages->offset[i], + p->page_size); + } + + in_len = pages->normal_num * p->page_size; + if (in_len > q->in_len) { + error_setg(errp, "multifd %u: unexpectedly large input", p->id); + return -1; + } + out_len = q->out_len; + + ret = qzCompress(&q->sess, q->in_buf, &in_len, q->out_buf, &out_len, 1); + if (ret != QZ_OK) { + error_setg(errp, "multifd %u: QATzip returned %d instead of QZ_OK", + p->id, ret); + return -1; + } + if (in_len != pages->normal_num * p->page_size) { + error_setg(errp, "multifd %u: QATzip failed to compress all input", + p->id); + return -1; + } + + p->iov[p->iovs_num].iov_base = q->out_buf; + p->iov[p->iovs_num].iov_len = out_len; + p->iovs_num++; + p->next_packet_size = out_len; + +out: + p->flags |= MULTIFD_FLAG_QATZIP; + multifd_send_fill_packet(p); + return 0; +} + +/** + * qatzip_recv_setup: Set up QATzip session and allocate private buffers. + * + * @param p Multifd channel params + * @param errp Pointer to error, which will be set in case of error + * @return 0 on success, -1 on error (and *errp will be set) + */ +static int qatzip_recv_setup(MultiFDRecvParams *p, Error **errp) +{ + QatzipData *q; + QzSessionParamsDeflate_T params; + const char *err_msg; + int ret; + + q = g_new0(QatzipData, 1); + p->compress_data = q; + + /* + * Initialize QAT device with software fallback by default. This allows + * QATzip to use CPU path when QAT hardware reaches maximum throughput. + */ + ret = qzInit(&q->sess, true); + if (ret != QZ_OK && ret != QZ_DUPLICATE) { + err_msg = "qzInit failed"; + goto err; + } + + ret = qzGetDefaultsDeflate(¶ms); + if (ret != QZ_OK) { + err_msg = "qzGetDefaultsDeflate failed"; + goto err; + } + + ret = qzSetupSessionDeflate(&q->sess, ¶ms); + if (ret != QZ_OK && ret != QZ_DUPLICATE) { + err_msg = "qzSetupSessionDeflate failed"; + goto err; + } + + /* + * Reserve extra spaces for the incoming packets. Current implementation + * doesn't send uncompressed pages in case the compression gets too big. + */ + q->in_len = MULTIFD_PACKET_SIZE * 2; + /* + * PINNED_MEM is an enum from qatzip headers, which means to use + * kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device + * is not available or software fallback is used, the malloc flag needs to + * be set as COMMON_MEM. + */ + q->in_buf = qzMalloc(q->in_len, 0, PINNED_MEM); + if (!q->in_buf) { + q->in_buf = qzMalloc(q->in_len, 0, COMMON_MEM); + if (!q->in_buf) { + err_msg = "qzMalloc failed"; + goto err; + } + } + + q->out_len = MULTIFD_PACKET_SIZE; + q->out_buf = qzMalloc(q->out_len, 0, PINNED_MEM); + if (!q->out_buf) { + q->out_buf = qzMalloc(q->out_len, 0, COMMON_MEM); + if (!q->out_buf) { + err_msg = "qzMalloc failed"; + goto err; + } + } + + return 0; + +err: + error_setg(errp, "multifd %u: [receiver] %s", p->id, err_msg); + return -1; +} + +/** + * qatzip_recv_cleanup: Tear down QATzip session and release private buffers. + * + * @param p Multifd channel params + * @return None + */ +static void qatzip_recv_cleanup(MultiFDRecvParams *p) +{ + QatzipData *q = p->compress_data; + + if (q) { + if (q->in_buf) { + qzFree(q->in_buf); + } + if (q->out_buf) { + qzFree(q->out_buf); + } + (void)qzTeardownSession(&q->sess); + (void)qzClose(&q->sess); + g_free(q); + } + p->compress_data = NULL; +} + + +/** + * qatzip_recv: Decompress pages and copy them to the appropriate + * locations. + * + * @param p Multifd channel params + * @param errp Pointer to error, which will be set in case of error + * @return 0 on success, -1 on error (and *errp will be set) + */ +static int qatzip_recv(MultiFDRecvParams *p, Error **errp) +{ + QatzipData *q = p->compress_data; + int ret; + unsigned int in_len, out_len; + uint32_t in_size = p->next_packet_size; + uint32_t expected_size = p->normal_num * p->page_size; + uint32_t flags = p->flags & MULTIFD_FLAG_COMPRESSION_MASK; + + if (in_size > q->in_len) { + error_setg(errp, "multifd %u: received unexpectedly large packet", + p->id); + return -1; + } + + if (flags != MULTIFD_FLAG_QATZIP) { + error_setg(errp, "multifd %u: flags received %x flags expected %x", + p->id, flags, MULTIFD_FLAG_QATZIP); + return -1; + } + + multifd_recv_zero_page_process(p); + if (!p->normal_num) { + assert(in_size == 0); + return 0; + } + + ret = qio_channel_read_all(p->c, (void *)q->in_buf, in_size, errp); + if (ret != 0) { + return ret; + } + + in_len = in_size; + out_len = q->out_len; + ret = qzDecompress(&q->sess, q->in_buf, &in_len, q->out_buf, &out_len); + if (ret != QZ_OK) { + error_setg(errp, "multifd %u: qzDecompress failed", p->id); + return -1; + } + if (out_len != expected_size) { + error_setg(errp, "multifd %u: packet size received %u size expected %u", + p->id, out_len, expected_size); + return -1; + } + + /* Copy each page to its appropriate location. */ + for (int i = 0; i < p->normal_num; i++) { + memcpy(p->host + p->normal[i], + q->out_buf + p->page_size * i, + p->page_size); + } + return 0; +} + +static MultiFDMethods multifd_qatzip_ops = { + .send_setup = qatzip_send_setup, + .send_cleanup = qatzip_send_cleanup, + .send_prepare = qatzip_send_prepare, + .recv_setup = qatzip_recv_setup, + .recv_cleanup = qatzip_recv_cleanup, + .recv = qatzip_recv +}; + +static void multifd_qatzip_register(void) +{ + multifd_register_ops(MULTIFD_COMPRESSION_QATZIP, &multifd_qatzip_ops); +} + +migration_init(multifd_qatzip_register); diff --git a/migration/meson.build b/migration/meson.build index 77f3abf08e..66d3de86f0 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -42,6 +42,7 @@ system_ss.add(when: rdma, if_true: files('rdma.c')) system_ss.add(when: zstd, if_true: files('multifd-zstd.c')) system_ss.add(when: qpl, if_true: files('multifd-qpl.c')) system_ss.add(when: uadk, if_true: files('multifd-uadk.c')) +system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c')) specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', if_true: files('ram.c', From patchwork Mon Sep 9 20:11:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82761ECE587 for ; Mon, 9 Sep 2024 20:13:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkk9-0000vm-37; Mon, 09 Sep 2024 16:12:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk7-0000qU-OQ for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk5-00053m-RI for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912729; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=22RXMR5DYkLZTisVq+tQn5/56hW2wP8HWKKqqEW05PY=; b=KyvoPDdA9byNFNd03uMEeaDKIqTwgdX0sVZHJQZKoOmEYqQyHnG0KdqbnB8pppspAuRHDs Wpuc+r10sreALWxg7Dtcitq8mkgVNhpin/7f5GTNVF8ZrsSG2LY065LjDn4obpiFWr15nB iDoE4hsXPTyh+Bc2CdR9LYHqQLfHOFM= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-356-sL8NK-VOMXagUV75VWrK8Q-1; Mon, 09 Sep 2024 16:12:08 -0400 X-MC-Unique: sL8NK-VOMXagUV75VWrK8Q-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4581efb73d3so45295861cf.1 for ; Mon, 09 Sep 2024 13:12:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912727; x=1726517527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=22RXMR5DYkLZTisVq+tQn5/56hW2wP8HWKKqqEW05PY=; b=tZcuGapDq5waxJjN9BYZpHB6tYUVr3mi99YCFxmFg4+UF6ZmAH/KbvU+MvkUq65igB KDUjg01beE4AnCjDH84tXy9GDetLZWvxVF93nXn7ks3MJ4MRYXNJi9cz8jhQsAEj4m17 y4f2ZKZSz9OLU/GevifMRW4m5RXvcZXLY8vOQjNexr2pMmTpWY57hpyB+F80KPnSdZkl J881que5P7Oq7O5IISYOBJWRfFAcqDr5ZezThTEB4djLPEs82XfmM2UDPUZYYaWwwXII GCpp9+X3JkY4aOPp9e6HnIsOizarf34E6g/xqRLBqWcBNMZ+5/jMMjEaV5zDbpiEVGP7 KVgQ== X-Gm-Message-State: AOJu0YySLctxZGuQGPkDLeYkjHFYbSwONeecWom/oMIqI5Ludt/fYuWO f2rMfcbs51WjD2dxZ7MnVoj4KIAbf414nY/ubUgfqfCSCnpuBewy9H9ot5dnu6XeprE+3cBSmkl hQSmghvoKhfRiFRfLIGvRHCZQIsXbN+7Ql7Azgjm2rK6cHLG3PJZuYrWmbvstuIvq2a2QuPRBwe w1igMQWbBk0oRdpmx5xXI7THL9MvFkdAr/vQ== X-Received: by 2002:a05:622a:11c5:b0:457:c7b7:a26a with SMTP id d75a77b69052e-4580c67e50dmr156436451cf.8.1725912727176; Mon, 09 Sep 2024 13:12:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEBNpc8PcW4Yu0Lh/xWDwHoJDT+rJRljxeQdI++FcsHcIXY5TJY3pvkGUpq48Lnpx+I4HXX/Q== X-Received: by 2002:a05:622a:11c5:b0:457:c7b7:a26a with SMTP id d75a77b69052e-4580c67e50dmr156436021cf.8.1725912726766; Mon, 09 Sep 2024 13:12:06 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.12.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:12:05 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , Bryan Zhang , Hao Xiang , Yichen Wang Subject: [PULL 8/9] tests/migration: Add integration test for 'qatzip' compression method Date: Mon, 9 Sep 2024 16:11:46 -0400 Message-ID: <20240909201147.3761639-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Bryan Zhang Adds an integration test for 'qatzip'. Reviewed-by: Fabiano Rosas Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang Signed-off-by: Yichen Wang Link: https://lore.kernel.org/r/20240830232722.58272-6-yichen.wang@bytedance.com Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 9d08101643..d6768d5d71 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2920,6 +2920,18 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestState *from, } #endif /* CONFIG_ZSTD */ +#ifdef CONFIG_QATZIP +static void * +test_migrate_precopy_tcp_multifd_qatzip_start(QTestState *from, + QTestState *to) +{ + migrate_set_parameter_int(from, "multifd-qatzip-level", 2); + migrate_set_parameter_int(to, "multifd-qatzip-level", 2); + + return test_migrate_precopy_tcp_multifd_start_common(from, to, "qatzip"); +} +#endif + #ifdef CONFIG_QPL static void * test_migrate_precopy_tcp_multifd_qpl_start(QTestState *from, @@ -3017,6 +3029,17 @@ static void test_multifd_tcp_zstd(void) } #endif +#ifdef CONFIG_QATZIP +static void test_multifd_tcp_qatzip(void) +{ + MigrateCommon args = { + .listen_uri = "defer", + .start_hook = test_migrate_precopy_tcp_multifd_qatzip_start, + }; + test_precopy_common(&args); +} +#endif + #ifdef CONFIG_QPL static void test_multifd_tcp_qpl(void) { @@ -3922,6 +3945,10 @@ int main(int argc, char **argv) migration_test_add("/migration/multifd/tcp/plain/zstd", test_multifd_tcp_zstd); #endif +#ifdef CONFIG_QATZIP + migration_test_add("/migration/multifd/tcp/plain/qatzip", + test_multifd_tcp_qatzip); +#endif #ifdef CONFIG_QPL migration_test_add("/migration/multifd/tcp/plain/qpl", test_multifd_tcp_qpl); From patchwork Mon Sep 9 20:11:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13797525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9039DECE58A for ; Mon, 9 Sep 2024 20:13:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1snkkC-00019V-LO; Mon, 09 Sep 2024 16:12:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkkB-000141-5e for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1snkk7-000557-RO for qemu-devel@nongnu.org; Mon, 09 Sep 2024 16:12:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725912731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0yCYyH1EuoqCtDP/Ff8kG1W6vFMEFbXZwtdKjoE01Z8=; b=ApZ5F63nEi/dFTGL9S9rm2npClBrqe5TfrnkzFStAwj1BbFwq+x/bgEf7c/Snf/OSoKHhF 0UmA7/harmPLQmdRGL9p6zQL72C+kuSXhrdXPwkw1uRIUp1XYCVE85eATK73q3LBEg+TVM cUOSGPhMAY2mLl7dmcTIHnOFHa2IjkQ= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-339-vMVvLR7oMFqcJjtHR__zGA-1; Mon, 09 Sep 2024 16:12:10 -0400 X-MC-Unique: vMVvLR7oMFqcJjtHR__zGA-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-458353d3635so25799041cf.0 for ; Mon, 09 Sep 2024 13:12:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725912729; x=1726517529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0yCYyH1EuoqCtDP/Ff8kG1W6vFMEFbXZwtdKjoE01Z8=; b=RQc63Lcut7QY7GAEpMoR6nSyKZ1pw+ue8gCUAzlS9j4cMtfVjhwzUMseWdfuMCdMa0 L5gUyMUfjEHKJhYGYnKOiiTQ/BwvMIRyk2cgKGZ52b0FI4HfTCsD24j5zweL6DK2jow4 xG30LOoqf6WL2n7/J6IlfPnbBe7dlZbSWShm2hncDgA/sLt2CxRiHcLeQErfZRgzQKVK KkyD/PcWkStI9HMKgYJCP1dO+mkLtgHc9wpgXqAgnoP0H1MbuBhPbBXxNNrsUGmRwz1X a3n5twHsrolzL6Xoait3EbVg3qtz2jNI8kCno8yv8EBgwQIaLH1SL5JL5p1kIyLwXgwi En7g== X-Gm-Message-State: AOJu0YzzO8iEOS7RTIUVBXVHMi/MFN3k8dssTWlcNQ+04UWmEqkUVNKL 1ImJJHh3TdUxCUNdMhGiDAIlbQhqVjnsDu8PD1swhZZoVTY+Vmo/O30uJ0g/Clq3t0ejrPE0/Ca MihX1zjLyU6GU41RMpVNdHo7suq5wMAVfnJncuoNBtKZscOSlhw9yXlO5gt0CglpsL5wvGX6YTy KUa65PWt4irT3E9E2mteB6GHRrLkzxHRdWwQ== X-Received: by 2002:a05:622a:1105:b0:458:3ab3:84f9 with SMTP id d75a77b69052e-4583c71e612mr18030971cf.9.1725912729365; Mon, 09 Sep 2024 13:12:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHiWn1w8zUW8ErmVutZLM011NNQdBen3vzF2zH0CPX5HEI06hiVq8oBjkWciZ17sNBr1r9s8g== X-Received: by 2002:a05:622a:1105:b0:458:3ab3:84f9 with SMTP id d75a77b69052e-4583c71e612mr18030401cf.9.1725912728895; Mon, 09 Sep 2024 13:12:08 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f93978sm22838561cf.83.2024.09.09.13.12.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 13:12:07 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Cc: Peter Maydell , Peter Xu , Fabiano Rosas , "Denis V. Lunev" , Paolo Bonzini Subject: [PULL 9/9] system: improve migration debug Date: Mon, 9 Sep 2024 16:11:47 -0400 Message-ID: <20240909201147.3761639-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240909201147.3761639-1-peterx@redhat.com> References: <20240909201147.3761639-1-peterx@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: "Denis V. Lunev" Right now migration_throttle() tracepoint lacks very important important information, i.e. no one could easily say how much the guest is throttled. This makes difficult to debug guest quality of service during migration. This patch adds one more tracepoint into cpu_throttle_set() which is actually doing this job. Signed-off-by: Denis V. Lunev CC: Peter Xu CC: Fabiano Rosas CC: Paolo Bonzini Link: https://lore.kernel.org/r/20240905191941.310592-1-den@openvz.org Signed-off-by: Peter Xu --- system/cpu-throttle.c | 3 +++ system/trace-events | 3 +++ 2 files changed, 6 insertions(+) diff --git a/system/cpu-throttle.c b/system/cpu-throttle.c index c951a6c65e..7632dc6143 100644 --- a/system/cpu-throttle.c +++ b/system/cpu-throttle.c @@ -28,6 +28,7 @@ #include "qemu/main-loop.h" #include "sysemu/cpus.h" #include "sysemu/cpu-throttle.h" +#include "trace.h" /* vcpu throttling controls */ static QEMUTimer *throttle_timer; @@ -95,6 +96,8 @@ void cpu_throttle_set(int new_throttle_pct) */ bool throttle_active = cpu_throttle_active(); + trace_cpu_throttle_set(new_throttle_pct); + /* Ensure throttle percentage is within valid range */ new_throttle_pct = MIN(new_throttle_pct, CPU_THROTTLE_PCT_MAX); new_throttle_pct = MAX(new_throttle_pct, CPU_THROTTLE_PCT_MIN); diff --git a/system/trace-events b/system/trace-events index 2ed1d59b1f..074d001e90 100644 --- a/system/trace-events +++ b/system/trace-events @@ -44,3 +44,6 @@ dirtylimit_state_finalize(void) dirtylimit_throttle_pct(int cpu_index, uint64_t pct, int64_t time_us) "CPU[%d] throttle percent: %" PRIu64 ", throttle adjust time %"PRIi64 " us" dirtylimit_set_vcpu(int cpu_index, uint64_t quota) "CPU[%d] set dirty page rate limit %"PRIu64 dirtylimit_vcpu_execute(int cpu_index, int64_t sleep_time_us) "CPU[%d] sleep %"PRIi64 " us" + +# cpu-throttle.c +cpu_throttle_set(int new_throttle_pct) "set guest CPU throttled by %d%%"