From patchwork Wed Oct 26 00:47:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13020012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99E97C38A2D for ; Wed, 26 Oct 2022 00:49:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229995AbiJZAtH (ORCPT ); Tue, 25 Oct 2022 20:49:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231381AbiJZAtF (ORCPT ); Tue, 25 Oct 2022 20:49:05 -0400 Received: from mail-qt1-x841.google.com (mail-qt1-x841.google.com [IPv6:2607:f8b0:4864:20::841]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB420DFB46 for ; Tue, 25 Oct 2022 17:49:04 -0700 (PDT) Received: by mail-qt1-x841.google.com with SMTP id h24so8900748qta.7 for ; Tue, 25 Oct 2022 17:49:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7/73U88aWKwNDfroPuUtEdVueHFIV0vL1R64OOolus=; b=FKmI7MFmg0vUcBItLfBUZyX5abdLhN+mCsU8k+WVfoqdr5hHERIc4Wvq3lZPrstRXg UU/idLJ5EniJ8I8PEU5jOTkhYDuLRapTw3SvwHkTc47NGprU07JIINHnCgL6OS9KkR44 vjFhw/GBNtDqLloY8Y9yKrAMJL+N00be2BQVNo1P7bPZGnw8xBlVJt8Pzyyg0/NoXIJH Z0aKwO26IYifO0lEu7wQCKDzIwVosxDU7NfS8uVYgVhO6YiRQ62OMeQNrv+hVNem51D5 VlMp54OW5qcLwmsyfB/Bz7PfKRfnLp72A6+FQUOCUXmIL+VJ8unKmOnEQPKTU/7vA68h kd4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d7/73U88aWKwNDfroPuUtEdVueHFIV0vL1R64OOolus=; b=txMlC0yhRMZU5+U4nmGsRsyUK+dkkyfXuGsSMIBGC5Kz+HMZtuevbRID0qw11z1Vmp YcDihPLZN0I5lIZZhqxJXvWC9BhATDnjzNAp8dfY6dllxy9PsOnCKNepen50hjjDKVvC vDKSajGiMw30+dbIGyI1e/8v6pzbIqgdTpg2MnVWs2ftWUls4WUJNti/K0rL9esMvury +UMfvqtQNZ0EvD9Wa0Sdl/fa7Yxu8p1mk1CDYGD8yP3AWUusCWi+zD2eSzn8MnaJU+FV qsqY1ECeNt29YdzVorCow50boTABTE6647CJz1NP5jgDgtH/SHAUi0cfRGGwg7Vh2p9K VyKg== X-Gm-Message-State: ACrzQf3pBmAFVQnM8xYrjEn7F1a58Rda4Ah8qWfbRPLdtbRdCMpZ+zLZ FQl31GJEozfsrTz070C2Ww== X-Google-Smtp-Source: AMsMyM6TE3OJhxhBREvSayYLxzwSr++OBb9psECz2cRe05REWMbChQNHncI+KtTroSDsIzP9zSKLZQ== X-Received: by 2002:ac8:7d0d:0:b0:39c:c5e0:37c0 with SMTP id g13-20020ac87d0d000000b0039cc5e037c0mr33442427qtb.537.1666745343935; Tue, 25 Oct 2022 17:49:03 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.48.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:48:43 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 1/4] hw/i386/pc.c: CXL Fixed Memory Window should not reserve e820 in bios Date: Tue, 25 Oct 2022 20:47:34 -0400 Message-Id: <20221026004737.3646-2-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Early-boot e820 records will be inserted by the bios/efi/early boot software and be reported to the kernel via insert_resource. Later, when CXL drivers iterate through the regions again, they will insert another resource and make the RESERVED memory area a child. This RESERVED memory area causes the memory region to become unusable, and as a result attempting to create memory regions with `cxl create-region ...` Will fail due to the RESERVED area intersecting with the CXL window. During boot the following traceback is observed: 0xffffffff81101650 in insert_resource_expand_to_fit () 0xffffffff83d964c5 in e820__reserve_resources_late () 0xffffffff83e03210 in pcibios_resource_survey () 0xffffffff83e04f4a in pcibios_init () Which produces a call to reserve the CFMWS area: (gdb) p *new $54 = {start = 0x290000000, end = 0x2cfffffff, name = "Reserved", flags = 0x200, desc = 0x7, parent = 0x0, sibling = 0x0, child = 0x0} Later the Kernel parses ACPI tables and reserves the exact same area as the CXL Fixed Memory Window: 0xffffffff811016a4 in insert_resource_conflict () insert_resource () 0xffffffff81a81389 in cxl_parse_cfmws () 0xffffffff818c4a81 in call_handler () acpi_parse_entries_array () (gdb) p/x *new $59 = {start = 0x290000000, end = 0x2cfffffff, name = "CXL Window 0", flags = 0x200, desc = 0x0, parent = 0x0, sibling = 0x0, child = 0x0} This produces the following output in /proc/iomem: 590000000-68fffffff : CXL Window 0 590000000-68fffffff : Reserved This reserved area causes `get_free_mem_region()` to fail due to a check against `__region_intersects()`. Due to this reserved area, the intersect check will only ever return REGION_INTERSECTS, which causes `cxl create-region` to always fail. Signed-off-by: Gregory Price --- hw/i386/pc.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/hw/i386/pc.c b/hw/i386/pc.c index 768982ae9a..203c90fedb 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -1062,7 +1062,6 @@ void pc_memory_init(PCMachineState *pcms, hwaddr cxl_size = MiB; cxl_base = pc_get_cxl_range_start(pcms); - e820_add_entry(cxl_base, cxl_size, E820_RESERVED); memory_region_init(mr, OBJECT(machine), "cxl_host_reg", cxl_size); memory_region_add_subregion(system_memory, cxl_base, mr); cxl_resv_end = cxl_base + cxl_size; @@ -1078,7 +1077,6 @@ void pc_memory_init(PCMachineState *pcms, memory_region_init_io(&fw->mr, OBJECT(machine), &cfmws_ops, fw, "cxl-fixed-memory-region", fw->size); memory_region_add_subregion(system_memory, fw->base, &fw->mr); - e820_add_entry(fw->base, fw->size, E820_RESERVED); cxl_fmw_base += fw->size; cxl_resv_end = cxl_fmw_base; } From patchwork Wed Oct 26 00:47:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13020013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47B9DC38A02 for ; Wed, 26 Oct 2022 00:49:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230134AbiJZAtS (ORCPT ); Tue, 25 Oct 2022 20:49:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230428AbiJZAtR (ORCPT ); Tue, 25 Oct 2022 20:49:17 -0400 Received: from mail-qv1-xf41.google.com (mail-qv1-xf41.google.com [IPv6:2607:f8b0:4864:20::f41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27F6EDFB52 for ; Tue, 25 Oct 2022 17:49:17 -0700 (PDT) Received: by mail-qv1-xf41.google.com with SMTP id i12so10251987qvs.2 for ; Tue, 25 Oct 2022 17:49:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/u9Srr6RXjECMW/z6fGCEplaACCtFUVAvHpD6VsFGYI=; b=lb2uPwdsVV8+Qr+6+gRs0GsA/2YzAY3YGpgCAt07Nih5ihzeki8P8UUaaBA6BrAHgB roB4jvw+wtgqFnpXt3hOdgnZePM+QxjVDjSPavlhDDHdvFQAeHMKOELFRI2RKdOBAw3K bF6sJEv31boKoe66WJpxmYhCE59Z6dE9Y7HxK+NGDqxpKCd2pzylEYJ2UF6456wUNUdj haYjnui/Kbv/N4m7ALzWSVCEnbLMET/M3a0GkJbp/P6YE2Tgcf9kEXPBQ+365T9a0CfK 1AbcQfoyNWwR7O+ltUkROuB2g3kDTIUwuYzV+rFqkSW+8ef5L3kks98s+uanuyB4VivB M47g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/u9Srr6RXjECMW/z6fGCEplaACCtFUVAvHpD6VsFGYI=; b=3HsbjOqChax40nq2xSPjBoTCMC1Vp0fN0LtsMPwVEu59D4yN8pdDMZDrW5wJ7+3fd2 9RRF9hjZZCL+ouQUlAP5nZxQ0EqpuIxPGhQ46OXudGxWRPuyWoCM9GnaPYq2/le5CG8q JsU2WYRZfIPH7XxKKFcDA7y/We4hAWKwAbpl2KBY5mho+yqgMT4Zaap9eDk+gRjWF5XV 92366jo1rR1PLmXNWlUzMCkzto5Ay/1AokdE3PwVco40Oj+ipX+gLpSMePUfjtbYjTBz Ig7FObp9qMmtAoVH6GNkzdwcq0dzkAyBWEIazBFwN15SVGGrCAA9pdIsx8UkRA1Nji5S I4lA== X-Gm-Message-State: ACrzQf2lsfO2IMWxeY5Gd6x69BeZeizDWdvdX9NllJYCF6C74R9xQA8r mo84HaeyJ2XDtW//P8/sTQ== X-Google-Smtp-Source: AMsMyM7kWNgW/Qc08xbI2ipjZ7uhFkt3FM7NaxfvqlUw8LQ6GxNqwnQ5fjbsSSstj5XOfsinCWFTng== X-Received: by 2002:ad4:5de8:0:b0:4b1:85db:3509 with SMTP id jn8-20020ad45de8000000b004b185db3509mr34393355qvb.121.1666745356253; Tue, 25 Oct 2022 17:49:16 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:11 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 2/4] hw/cxl: Add CXL_CAPACITY_MULTIPLIER definition Date: Tue, 25 Oct 2022 20:47:35 -0400 Message-Id: <20221026004737.3646-3-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Remove usage of magic numbers when accessing capacity fields and replace with CXL_CAPACITY_MULTIPLIER, matching the kernel definition. Signed-off-by: Gregory Price Reviewed-by: Davidlohr Bueso --- hw/cxl/cxl-mailbox-utils.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 3e23d29e2d..d7543fd5b4 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -15,6 +15,8 @@ #include "qemu/log.h" #include "qemu/uuid.h" +#define CXL_CAPACITY_MULTIPLIER 0x10000000 /* SZ_256M */ + /* * How to add a new command, example. The command set FOO, with cmd BAR. * 1. Add the command set and cmd to the enum. @@ -267,7 +269,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, } QEMU_PACKED *fw_info; QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); - if (cxl_dstate->pmem_size < (256 << 20)) { + if (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) { return CXL_MBOX_INTERNAL_ERROR; } @@ -412,7 +414,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); uint64_t size = cxl_dstate->pmem_size; - if (!QEMU_IS_ALIGNED(size, 256 << 20)) { + if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -422,8 +424,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, /* PMEM only */ snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); - id->total_capacity = size / (256 << 20); - id->persistent_capacity = size / (256 << 20); + id->total_capacity = size / CXL_CAPACITY_MULTIPLIER; + id->persistent_capacity = size / CXL_CAPACITY_MULTIPLIER; id->lsa_size = cvc->get_lsa_size(ct3d); id->poison_list_max_mer[1] = 0x1; /* 256 poison records */ @@ -444,14 +446,14 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); uint64_t size = cxl_dstate->pmem_size; - if (!QEMU_IS_ALIGNED(size, 256 << 20)) { + if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } /* PMEM only */ part_info->active_vmem = 0; part_info->next_vmem = 0; - part_info->active_pmem = size / (256 << 20); + part_info->active_pmem = size / CXL_CAPACITY_MULTIPLIER; part_info->next_pmem = 0; *len = sizeof(*part_info); From patchwork Wed Oct 26 00:47:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13020014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B979C38A2D for ; Wed, 26 Oct 2022 00:49:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230522AbiJZAtf (ORCPT ); Tue, 25 Oct 2022 20:49:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230428AbiJZAte (ORCPT ); Tue, 25 Oct 2022 20:49:34 -0400 Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5589A2663 for ; Tue, 25 Oct 2022 17:49:27 -0700 (PDT) Received: by mail-qt1-x843.google.com with SMTP id a24so8891392qto.10 for ; Tue, 25 Oct 2022 17:49:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3eCdkblp9N10CpvwBlCvOOWSX9hk0WPXXWNuoMHXung=; b=W+dtK43wYZKfPY1up96M7gKPiJYFd8iwcrFUuMybsCXNlHKyg/SUGOUZlXi5vB7JyI pXAWEvjiHlHIC4t8JxXZdncwYoHheg0ZIj0UOMpkHEAWrPAY+GXMwGTsfuH7gScKmDvH o0o+hzrtUTp3zm9VP+o1Pzqs94ljVpZSmb8Q2cKWBiB50nOpVID6Vs0/DbwJMOaUb6HT kDgn1VRH/b1foa7SG9dSmRjpiUNzTzHVSLVyYUVb6oFo/WaSRIcBYIfViPr9LsHZ1Tur RnyFA9wNVwBDAUZA6sNPyiFHMUGK2vx7dA6yvlg1uiYEiQ5ve/XI9L3BflXTxapr4bRV lmaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3eCdkblp9N10CpvwBlCvOOWSX9hk0WPXXWNuoMHXung=; b=PkOTMrtwzl+iE4bTBM2n0OdI4v22a/K2nsWrza7dedNXgJcBzgFScdQc/zMn1gIHEa 6m3hqjtNGVtZXMjK01o1+bWHWbBBCPsq/g2kSSfc0rI2++tXJYrWaB6vXTBYenIogpQY gC/ZjuBcq9JHgrHxB1D9Su999ViDzrHe/yu500co5BUp0L6N1hCZTZDGEAS+hx/qiFsm eS3NvnnDOJBOA8x6vA/U41jcLdekfJTZOBldm+BuehH4qX5nmA/EwOzIKChNT9efgaKt Akm5517zqMWmx2M1UKuS07tf6CFTDWTVyHZZrtBDezgXR/e2GKqan1MZxYtAkMxFl9qt /d2Q== X-Gm-Message-State: ACrzQf0+8737AMDb1oX8Qyw4z2AMIzP3XPKlKtYeCHRJ5d/G8dWkpLre pFUM6AkRqaE3z07BhiC5qg== X-Google-Smtp-Source: AMsMyM4KdvUGjoK1qeOBgzRxPNIzo2UXmA4fZ6uat3Ig2AevStpprICpMTj37vtwKCuwZNDk5PT3Qw== X-Received: by 2002:ac8:7f54:0:b0:39c:bb57:a1aa with SMTP id g20-20020ac87f54000000b0039cbb57a1aamr34896534qtk.382.1666745366008; Tue, 25 Oct 2022 17:49:26 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:19 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 3/4] hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent) Date: Tue, 25 Oct 2022 20:47:36 -0400 Message-Id: <20221026004737.3646-4-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org This commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region. Two new properties have been added to cxl-type3 device initialization: [volatile-memdev] and [persistent-memdev] The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property. Partitioning volatile memory from persistent memory is not yet supported. Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info. Signed-off-by: Gregory Price --- docs/system/devices/cxl.rst | 53 +++++-- hw/cxl/cxl-mailbox-utils.c | 21 ++- hw/mem/cxl_type3.c | 274 +++++++++++++++++++++++++++--------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/cxl-test.c | 111 +++++++++++---- 5 files changed, 348 insertions(+), 122 deletions(-) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index f25783a4ec..9e165064c8 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -300,15 +300,36 @@ Example topology involving a switch:: Example command lines --------------------- -A very simple setup with just one directly attached CXL Type 3 device:: +A very simple setup with just one directly attached CXL Type 3 Persistent Memory device:: qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ ... - -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \ - -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -object memory-backend-file,pmem=true,id=pmem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \ + -object memory-backend-file,pmem=true,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,persistent-memdev=pmem0,lsa=cxl-lsa1,id=cxl-pmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +A very simple setup with just one directly attached CXL Type 3 Volatile Memory device:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ + -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +The same volatile setup may optionally include an LSA region:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -object memory-backend-ram,id=vmem0,share=on,size=256M \ + -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ + -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ + -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ + -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way @@ -328,13 +349,13 @@ the CXL Type3 device directly attached (no switches).:: -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ - -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ + -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \ -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ - -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ + -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ - -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ + -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2 \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ - -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ + -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: @@ -354,15 +375,23 @@ An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ -device cxl-upstream,bus=root_port0,id=us0 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ - -device cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ + -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,size=256M \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ - -device cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ + -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,size=256M \ -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ - -device cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ + -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,size=256M \ -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ - -device cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ + -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,size=256M \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k +Deprecations +------------ + +The Type 3 device [memdev] attribute has been deprecated in favor +of the [persistent-memdev] and [volatile-memdev] attributes. [memdev] +will default to a persistent memory device for backward compatibility +and is incapable of being used in combination with [persistent-memdev]. + Kernel Configuration Options ---------------------------- diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index d7543fd5b4..c1183614b9 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -269,7 +269,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd, } QEMU_PACKED *fw_info; QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); - if (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) { + if (cxl_dstate->mem_size < CXL_CAPACITY_MULTIPLIER) { return CXL_MBOX_INTERNAL_ERROR; } @@ -412,20 +412,20 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd, CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); - uint64_t size = cxl_dstate->pmem_size; - if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { + if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } id = (void *)cmd->payload; memset(id, 0, sizeof(*id)); - /* PMEM only */ snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); - id->total_capacity = size / CXL_CAPACITY_MULTIPLIER; - id->persistent_capacity = size / CXL_CAPACITY_MULTIPLIER; + id->total_capacity = cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER; + id->persistent_capacity = cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER; + id->volatile_capacity = cxl_dstate->vmem_size / CXL_CAPACITY_MULTIPLIER; id->lsa_size = cvc->get_lsa_size(ct3d); id->poison_list_max_mer[1] = 0x1; /* 256 poison records */ @@ -444,16 +444,15 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)cmd->payload; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); - uint64_t size = cxl_dstate->pmem_size; - if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { + if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } - /* PMEM only */ - part_info->active_vmem = 0; + part_info->active_vmem = cxl_dstate->vmem_size / CXL_CAPACITY_MULTIPLIER; part_info->next_vmem = 0; - part_info->active_pmem = size / CXL_CAPACITY_MULTIPLIER; + part_info->active_pmem = cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER; part_info->next_pmem = 0; *len = sizeof(*part_info); diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 0317bd96a6..21e866dcaf 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -32,7 +32,8 @@ enum { }; static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr) + int dsmad_handle, MemoryRegion *mr, + bool is_pmem, uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; g_autofree CDATDslbis *dslbis0 = NULL; @@ -51,8 +52,8 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = CDAT_DSMAS_FLAG_NV, - .DPA_base = 0, + .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .DPA_base = dpa_base, .DPA_length = int128_get64(mr->size), }; @@ -151,33 +152,67 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) { g_autofree CDATSubHeader **table = NULL; - MemoryRegion *nonvolatile_mr; CXLType3Dev *ct3d = priv; + MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; int dsmad_handle = 0; + int cur_ent = 0; + int len = 0; int rc; - if (!ct3d->hostmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem) { return 0; } - nonvolatile_mr = host_memory_backend_get_memory(ct3d->hostmem); - if (!nonvolatile_mr) { - return -EINVAL; + if (ct3d->hostvmem) { + volatile_mr = host_memory_backend_get_memory(ct3d->hostvmem); + if (!volatile_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES; + } + + if (ct3d->hostpmem) { + nonvolatile_mr = host_memory_backend_get_memory(ct3d->hostpmem); + if (!nonvolatile_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES; } - table = g_malloc0(CT3_CDAT_NUM_ENTRIES * sizeof(*table)); + table = g_malloc0(len * sizeof(*table)); if (!table) { return -ENOMEM; } - rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, nonvolatile_mr); - if (rc < 0) { - return rc; + /* Now fill them in */ + if (volatile_mr) { + rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + true, 0); + if (rc < 0) { + return rc; + } + cur_ent = CT3_CDAT_NUM_ENTRIES; + } + + if (nonvolatile_mr) { + rc = ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, + nonvolatile_mr, true, (volatile_mr ? volatile_mr->size : 0)); + if (rc < 0) { + goto error_cleanup; + } + cur_ent += CT3_CDAT_NUM_ENTRIES; } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); - return CT3_CDAT_NUM_ENTRIES; + return len; +error_cleanup: + int i; + for (i = 0; i < cur_ent; i++) { + g_free(*cdat_table[i]); + } + return rc; } static void ct3_free_cdat_table(CDATSubHeader **cdat_table, int num, void *priv) @@ -378,16 +413,48 @@ static void build_dvsecs(CXLType3Dev *ct3d) CXLDVSECRegisterLocator *regloc_dvsec; uint8_t *dvsec; int i; + uint32_t range1_size_hi = 0, range1_size_lo = 0, + range1_base_hi = 0, range1_base_lo = 0, + range2_size_hi = 0, range2_size_lo = 0, + range2_base_hi = 0, range2_base_lo = 0; + + /* + * Volatile memory is mapped as (0x0) + * Persistent memory is mapped at (volatile->size) + */ + if (ct3d->hostvmem && ct3d->hostpmem) { + range1_size_hi = ct3d->hostvmem->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->hostvmem->size & 0xF0000000); + range1_base_hi = 0; + range1_base_lo = 0; + range2_size_hi = ct3d->hostpmem->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->hostpmem->size & 0xF0000000); + range2_base_hi = ct3d->hostvmem->size >> 32; + range2_base_lo = ct3d->hostvmem->size & 0xF0000000; + } else { + HostMemoryBackend *hmbe = ct3d->hostvmem ? + ct3d->hostvmem : ct3d->hostpmem; + range1_size_hi = hmbe->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (hmbe->size & 0xF0000000); + range1_base_hi = 0; + range1_base_lo = 0; + } dvsec = (uint8_t *)&(CXLDVSECDevice){ .cap = 0x1e, .ctrl = 0x2, .status2 = 0x2, - .range1_size_hi = ct3d->hostmem->size >> 32, - .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | - (ct3d->hostmem->size & 0xF0000000), - .range1_base_hi = 0, - .range1_base_lo = 0, + .range1_size_hi = range1_size_hi, + .range1_size_lo = range1_size_lo, + .range1_base_hi = range1_base_hi, + .range1_base_lo = range1_base_lo, + .range2_size_hi = range2_size_hi, + .range2_size_lo = range2_size_lo, + .range2_base_hi = range2_base_hi, + .range2_base_lo = range2_base_lo }; cxl_component_create_dvsec(cxl_cstate, CXL2_TYPE3_DEVICE, PCIE_CXL_DEVICE_DVSEC_LENGTH, @@ -475,33 +542,62 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) MemoryRegion *mr; char *name; - if (!ct3d->hostmem) { - error_setg(errp, "memdev property must be set"); + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + error_setg(errp, "at least one memdev property must be set"); return false; + } else if (ct3d->hostmem && ct3d->hostpmem) { + error_setg(errp, "[memdev] cannot be used with new " + "[persistent-memdev] property"); + return false; + } else if (ct3d->hostmem) { + /* Use of hostmem property implies pmem */ + ct3d->hostpmem = ct3d->hostmem; + ct3d->hostmem = NULL; } - mr = host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { - error_setg(errp, "memdev property must be set"); + if (ct3d->hostpmem && !ct3d->lsa) { + error_setg(errp, "lsa property must be set for persistent devices"); return false; } - memory_region_set_nonvolatile(mr, true); - memory_region_set_enabled(mr, true); - host_memory_backend_set_mapped(ct3d->hostmem, true); - if (ds->id) { - name = g_strdup_printf("cxl-type3-dpa-space:%s", ds->id); - } else { - name = g_strdup("cxl-type3-dpa-space"); + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + if (!mr) { + error_setg(errp, "volatile memdev must have backing device"); + return false; + } + memory_region_set_nonvolatile(mr, false); + memory_region_set_enabled(mr, true); + host_memory_backend_set_mapped(ct3d->hostvmem, true); + if (ds->id) { + name = g_strdup_printf("cxl-type3-dpa-vmem-space:%s", ds->id); + } else { + name = g_strdup("cxl-type3-dpa-vmem-space"); + } + address_space_init(&ct3d->hostvmem_as, mr, name); + ct3d->cxl_dstate.vmem_size = mr->size; + ct3d->cxl_dstate.mem_size += mr->size; + g_free(name); } - address_space_init(&ct3d->hostmem_as, mr, name); - g_free(name); - - ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size; - if (!ct3d->lsa) { - error_setg(errp, "lsa property must be set"); - return false; + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + if (!mr) { + error_setg(errp, "persistent memdev must have backing device"); + return false; + } + memory_region_set_nonvolatile(mr, true); + memory_region_set_enabled(mr, true); + host_memory_backend_set_mapped(ct3d->hostpmem, true); + if (ds->id) { + name = g_strdup_printf("cxl-type3-dpa-pmem-space:%s", ds->id); + } else { + name = g_strdup("cxl-type3-dpa-pmem-space"); + } + address_space_init(&ct3d->hostpmem_as, mr, name); + ct3d->cxl_dstate.pmem_size = mr->size; + ct3d->cxl_dstate.mem_size += mr->size; + g_free(name); } return true; @@ -609,7 +705,12 @@ err_free_spdm_socket: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: - address_space_destroy(&ct3d->hostmem_as); + if (ct3d->hostvmem) { + address_space_destroy(&ct3d->hostvmem_as); + } + if (ct3d->hostpmem) { + address_space_destroy(&ct3d->hostpmem_as); + } return; } @@ -623,7 +724,12 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); - address_space_destroy(&ct3d->hostmem_as); + if (ct3d->hostvmem) { + address_space_destroy(&ct3d->hostvmem_as); + } + if (ct3d->hostpmem) { + address_space_destroy(&ct3d->hostpmem_as); + } } /* TODO: Support multiple HDM decoders and DPA skip */ @@ -663,11 +769,17 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data, { CXLType3Dev *ct3d = CXL_TYPE3(d); uint64_t dpa_offset; - MemoryRegion *mr; + MemoryRegion *vmr = NULL, *pmr = NULL; + AddressSpace *as; + + if (ct3d->hostvmem) { + vmr = host_memory_backend_get_memory(ct3d->hostvmem); + } + if (ct3d->hostpmem) { + pmr = host_memory_backend_get_memory(ct3d->hostpmem); + } - /* TODO support volatile region */ - mr = host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { + if (!vmr && !pmr) { return MEMTX_ERROR; } @@ -675,11 +787,13 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data, return MEMTX_ERROR; } - if (dpa_offset > int128_get64(mr->size)) { + if (dpa_offset > int128_get64(ct3d->cxl_dstate.mem_size)) { return MEMTX_ERROR; } - return address_space_read(&ct3d->hostmem_as, dpa_offset, attrs, data, size); + as = (vmr && (dpa_offset <= int128_get64(vmr->size))) ? + &ct3d->hostvmem_as : &ct3d->hostpmem_as; + return address_space_read(as, dpa_offset, attrs, data, size); } MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data, @@ -687,10 +801,17 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data, { CXLType3Dev *ct3d = CXL_TYPE3(d); uint64_t dpa_offset; - MemoryRegion *mr; + MemoryRegion *vmr = NULL, *pmr = NULL; + AddressSpace *as; + + if (ct3d->hostvmem) { + vmr = host_memory_backend_get_memory(ct3d->hostvmem); + } + if (ct3d->hostpmem) { + pmr = host_memory_backend_get_memory(ct3d->hostpmem); + } - mr = host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { + if (!vmr && !pmr) { return MEMTX_OK; } @@ -698,11 +819,13 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data, return MEMTX_OK; } - if (dpa_offset > int128_get64(mr->size)) { + if (dpa_offset > int128_get64(ct3d->cxl_dstate.mem_size)) { return MEMTX_OK; } - return address_space_write(&ct3d->hostmem_as, dpa_offset, attrs, - &data, size); + + as = (vmr && (dpa_offset <= int128_get64(vmr->size))) ? + &ct3d->hostvmem_as : &ct3d->hostpmem_as; + return address_space_write(as, dpa_offset, attrs, &data, size); } static void ct3d_reset(DeviceState *dev) @@ -717,7 +840,11 @@ static void ct3d_reset(DeviceState *dev) static Property ct3_props[] = { DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND, - HostMemoryBackend *), + HostMemoryBackend *), /* for backward compatibility */ + DEFINE_PROP_LINK("persistent-memdev", CXLType3Dev, hostpmem, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), + DEFINE_PROP_LINK("volatile-memdev", CXLType3Dev, hostvmem, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), @@ -728,10 +855,12 @@ static Property ct3_props[] = { static uint64_t get_lsa_size(CXLType3Dev *ct3d) { - MemoryRegion *mr; - - mr = host_memory_backend_get_memory(ct3d->lsa); - return memory_region_size(mr); + MemoryRegion *mr = NULL; + if (ct3d->lsa) { + mr = host_memory_backend_get_memory(ct3d->lsa); + return memory_region_size(mr); + } + return 0; } static void validate_lsa_access(MemoryRegion *mr, uint64_t size, @@ -744,30 +873,35 @@ static void validate_lsa_access(MemoryRegion *mr, uint64_t size, static uint64_t get_lsa(CXLType3Dev *ct3d, void *buf, uint64_t size, uint64_t offset) { - MemoryRegion *mr; + MemoryRegion *mr = NULL; void *lsa; - mr = host_memory_backend_get_memory(ct3d->lsa); - validate_lsa_access(mr, size, offset); + if (ct3d->lsa) { + mr = host_memory_backend_get_memory(ct3d->lsa); + validate_lsa_access(mr, size, offset); - lsa = memory_region_get_ram_ptr(mr) + offset; - memcpy(buf, lsa, size); + lsa = memory_region_get_ram_ptr(mr) + offset; + memcpy(buf, lsa, size); + return size; + } - return size; + return 0; } static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, uint64_t offset) { - MemoryRegion *mr; - void *lsa; + MemoryRegion *mr = NULL; + void *lsa = NULL; - mr = host_memory_backend_get_memory(ct3d->lsa); - validate_lsa_access(mr, size, offset); + if (ct3d->lsa) { + mr = host_memory_backend_get_memory(ct3d->lsa); + validate_lsa_access(mr, size, offset); - lsa = memory_region_get_ram_ptr(mr) + offset; - memcpy(lsa, buf, size); - memory_region_set_dirty(mr, offset, size); + lsa = memory_region_get_ram_ptr(mr) + offset; + memcpy(lsa, buf, size); + memory_region_set_dirty(mr, offset, size); + } /* * Just like the PMEM, if the guest is not allowed to exit gracefully, label @@ -978,7 +1112,7 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->config_read = ct3d_config_read; set_bit(DEVICE_CATEGORY_STORAGE, dc->categories); - dc->desc = "CXL PMEM Device (Type 3)"; + dc->desc = "CXL Memory Device (Type 3)"; dc->reset = ct3d_reset; device_class_set_props(dc, ct3_props); diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 1cd71afcb6..1b366b739c 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -180,8 +180,10 @@ typedef struct cxl_device_state { uint64_t host_set; } timestamp; - /* memory region for persistent memory, HDM */ + /* memory region size, HDM */ + uint64_t mem_size; uint64_t pmem_size; + uint64_t vmem_size; struct cxl_cmd (*cxl_cmd_set)[256]; /* Move me later */ @@ -311,12 +313,15 @@ struct CXLType3Dev { PCIDevice parent_obj; /* Properties */ - HostMemoryBackend *hostmem; + HostMemoryBackend *hostmem; /* deprecated */ + HostMemoryBackend *hostvmem; + HostMemoryBackend *hostpmem; HostMemoryBackend *lsa; uint64_t sn; /* State */ - AddressSpace hostmem_as; + AddressSpace hostvmem_as; + AddressSpace hostpmem_as; CXLComponentState cxl_cstate; CXLDeviceState cxl_dstate; diff --git a/tests/qtest/cxl-test.c b/tests/qtest/cxl-test.c index 6baed747fa..a05ddc0c7b 100644 --- a/tests/qtest/cxl-test.c +++ b/tests/qtest/cxl-test.c @@ -34,29 +34,46 @@ "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \ "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 " -#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " - -#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " - -#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " \ - "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp1,memdev=cxl-mem1,lsa=lsa1,id=cxl-pmem1 " \ - "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp2,memdev=cxl-mem2,lsa=lsa2,id=cxl-pmem2 " \ - "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ - "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ - "-device cxl-type3,bus=rp3,memdev=cxl-mem3,lsa=lsa3,id=cxl-pmem3 " +#define QEMU_T3D_DEPRECATED \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 " + +#define QEMU_T3D_PMEM \ + "-object memory-backend-file,id=m0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,persistent-memdev=cxl-m0,lsa=lsa0,id=pmem0 " + +#define QEMU_T3D_VMEM \ + "-object memory-backend-ram,id=mem0,size=256M " \ + "-device cxl-type3,bus=rp0,volatile-memdev=mem0,id=mem0 " + +#define QEMU_T3D_VMEM_LSA \ + "-object memory-backend-ram,id=mem0,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,volatile-memdev=mem0,lsa=lsa0,id=mem0 " + +#define QEMU_2T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,persistent-memdev=cxl-mem0,lsa=lsa0,id=pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1,id=pmem1 " + +#define QEMU_4T3D \ + "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa0,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp0,persistent-memdev=cxl-mem0,lsa=lsa0,id=pmem0 " \ + "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa1,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp1,persistent-memdev=cxl-mem1,lsa=lsa1,id=pmem1 " \ + "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa2,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp2,persistent-memdev=cxl-mem2,lsa=lsa2,id=pmem2 " \ + "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \ + "-object memory-backend-file,id=lsa3,mem-path=%s,size=256M " \ + "-device cxl-type3,bus=rp3,persistent-memdev=cxl-mem3,lsa=lsa3,id=pmem3 " static void cxl_basic_hb(void) { @@ -95,14 +112,53 @@ static void cxl_2root_port(void) } #ifdef CONFIG_POSIX -static void cxl_t3d(void) +static void cxl_t3d_deprecated(void) { g_autoptr(GString) cmdline = g_string_new(NULL); g_autofree const char *tmpfs = NULL; tmpfs = g_dir_make_tmp("cxl-test-XXXXXX", NULL); - g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs, tmpfs); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_DEPRECATED, + tmpfs, tmpfs); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_persistent(void) +{ + g_autoptr(GString) cmdline = g_string_new(NULL); + g_autofree const char *tmpfs = NULL; + + tmpfs = g_dir_make_tmp("cxl-test-XXXXXX", NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_PMEM, + tmpfs, tmpfs); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_volatile(void) +{ + g_autoptr(GString) cmdline = g_string_new(NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_VMEM); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_volatile_lsa(void) +{ + g_autoptr(GString) cmdline = g_string_new(NULL); + g_autofree const char *tmpfs = NULL; + + tmpfs = g_dir_make_tmp("cxl-test-XXXXXX", NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_VMEM_LSA, + tmpfs); qtest_start(cmdline->str); qtest_end(); @@ -167,7 +223,10 @@ int main(int argc, char **argv) qtest_add_func("/pci/cxl/rp", cxl_root_port); qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port); #ifdef CONFIG_POSIX - qtest_add_func("/pci/cxl/type3_device", cxl_t3d); + qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated); + qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent); + qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile); + qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_lsa); qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d); qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d); From patchwork Wed Oct 26 00:47:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13020015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E597CC38A2D for ; Wed, 26 Oct 2022 00:49:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230428AbiJZAtp (ORCPT ); Tue, 25 Oct 2022 20:49:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230480AbiJZAtm (ORCPT ); Tue, 25 Oct 2022 20:49:42 -0400 Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D74515FD4 for ; Tue, 25 Oct 2022 17:49:39 -0700 (PDT) Received: by mail-qv1-xf2e.google.com with SMTP id w10so3956891qvr.3 for ; Tue, 25 Oct 2022 17:49:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=itdJjoWTzr0ylNc2qCCbqq49u8Zk8OI8xM4xXE0AET8=; b=DErcIejy+jTfZFrrHblUaN3ryaHEn6mgd2TCHyOHrRofiS/ZHNaoRKG8PfnL8yQP0X 2Yv6HFnPkkC+29wcpuzRi6ItVra8KrfgsNUwmlM8yephllXnY0cOTYRsLLbqGL2wMXdn cx7m7O0X4TxiYd+mLeC1m5Oq/wqLB0BA7Nmbjd3u92rDjmGYZf87nlJx4EfKhQTKv89g hcYuwn+YVnQikqdrgPe51EEYdRQv9FIrjG7HCieEIEyLLDaievMubaB+tMFBzj9y9fG2 bnIUG9Ym7bnLWSHGesGLhhYOt0F7Z7EKGPsp+u+L5kNcpexCXeDgW3ZBmu7ME3ZaQFt6 +NGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=itdJjoWTzr0ylNc2qCCbqq49u8Zk8OI8xM4xXE0AET8=; b=qvGoMNvJ3osOXP4tgOJycJzzS19bMBkIMNZEGJNt62RayOnSOV9FJbPVu/s5tkYM0+ pnDYYS+Pc8Pqe+F4CXLSQ7aOeeTmIB79eed9p6SSNLTctjfDdwsgVowPkmRChXVPvBm+ SOQqLZh3aUg/kbgtGL8yDwBGF3FrII2Q5jCCXBXYDt9K3HfNtMWU5t3HBgWBvf/JkmGw T5//co9i5ZWnFjnyXFxKUQdOS1awv/yTiazLSF+bRjqJCHpCK+SSBlM9+GRn9Y/mY9xk YSW2wJEhIDzu1HjpIBtFFF88w8J2ttp0sNCbypibvT5S9YGFA7fnLOZ5esXFwNib9aCi sfAQ== X-Gm-Message-State: ACrzQf3rH9L5beHeVUXns7lraRRQEe9Dv4V/0cDArjqtHje6x1WyDwLd RfvYhoAnT/B35IqNAG+ueu2mOCEogQ== X-Google-Smtp-Source: AMsMyM5JSL7W12XrKKbKyXJP9wWmNB6SvooPIuwe6FpViFuujBLsvC9GOqJBnqlfQPAVFkWcVoC2Ng== X-Received: by 2002:ad4:5fc8:0:b0:4b7:6ca0:adc6 with SMTP id jq8-20020ad45fc8000000b004b76ca0adc6mr27173507qvb.120.1666745378233; Tue, 25 Oct 2022 17:49:38 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:31 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 4/4] hw/acpi/cxl.c: Fill in SRAT for vmem/pmem if NUMA node is assigned Date: Tue, 25 Oct 2022 20:47:37 -0400 Message-Id: <20221026004737.3646-5-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org This patch enables the direct assignment of a NUMA node to a volatile or persistent memory region on a CXL type-3 device. This is useful for testing static mapping for type-3 device memory regions as memory and leveraging them directly via its NUMA node. Co-developed-By: Davidlohr Bueso Signed-off-by: Gregory Price --- docs/system/devices/cxl.rst | 21 ++++++++++++ hw/acpi/cxl.c | 67 +++++++++++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 4 +++ include/hw/acpi/cxl.h | 1 + 4 files changed, 93 insertions(+) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index 9e165064c8..32bf84a97c 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -332,6 +332,27 @@ The same volatile setup may optionally include an LSA region:: -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G + +Volatile and Persistent Memory regions may also be assigned an SRAT entry and statically +mapped into the system by manually assigning them a CPU-less NUMA node. This is an example +of a CXL Type 3 Volatile Memory device being assigned an SRAT entry via a NUMA node mapping:: + + qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \ + ... + -smp 4 \ + -enable-kvm \ + -nographic \ + -object memory-backend-ram,id=mem0,size=2G,share=on \ + -object memory-backend-ram,id=mem1,size=2G,share=on \ + -numa node,memdev=mem0,cpus=0-3,nodeid=0 \ + -numa node,memdev=mem1,nodeid=1, \ + -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \ + -device cxl-rp,port=0,id=rp0,bus=cxl.0,chassis=0,slot=0 \ + -device cxl-rp,port=1,id=rp1,bus=cxl.0,chassis=0,slot=1 \ + -device cxl-type3,bus=rp0,volatile-memdev=mem1,id=cxl-mem0 + + + A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way interleave across 2 CXL host bridges. Each host bridge has 2 CXL Root Ports, with the CXL Type3 device directly attached (no switches).:: diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c index 2bf8c07993..a8c6166b7f 100644 --- a/hw/acpi/cxl.c +++ b/hw/acpi/cxl.c @@ -254,3 +254,70 @@ void build_cxl_osc_method(Aml *dev) aml_append(dev, aml_name_decl("CTRC", aml_int(0))); aml_append(dev, __build_cxl_osc_method()); } + +static int cxl_device_list(Object *obj, void *opaque) +{ + GSList **list = opaque; + + if (object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + *list = g_slist_append(*list, DEVICE(obj)); + } + + object_child_foreach(obj, cxl_device_list, opaque); + return 0; +} + +static GSList *cxl_get_device_list(void) +{ + GSList *list = NULL; + + object_child_foreach(qdev_get_machine(), cxl_device_list, &list); + return list; +} + +void cxl_build_srat(GArray *table_data, NodeInfo* numa_info, int nb_numa_nodes) +{ + GSList *device_list, *list = cxl_get_device_list(); + int node = 0; + + for (device_list = list; device_list; device_list = device_list->next) { + DeviceState *dev = device_list->data; + CXLType3Dev *ct3d = CXL_TYPE3(dev); + MemoryRegion *mr = NULL; + + if (ct3d->hostvmem) { + /* Find the numa node associated with this memdev */ + for (node = 0; node < nb_numa_nodes; node++) { + if (numa_info[node].node_memdev == ct3d->hostvmem) { + break; + } + } + if (node != nb_numa_nodes) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + if (mr) { + build_srat_memory(table_data, mr->addr, mr->size, node, + (MEM_AFFINITY_ENABLED | MEM_AFFINITY_HOTPLUGGABLE)); + } + } + } + + if (ct3d->hostpmem) { + /* Find the numa node associated with this memdev */ + for (node = 0; node < nb_numa_nodes; node++) { + if (numa_info[node].node_memdev == ct3d->hostpmem) { + break; + } + } + if (node != nb_numa_nodes) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + if (mr) { + build_srat_memory(table_data, mr->addr, mr->size, node, + (MEM_AFFINITY_ENABLED | MEM_AFFINITY_HOTPLUGGABLE | + MEM_AFFINITY_NON_VOLATILE)); + } + } + } + } + + g_slist_free(list); +} diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index 4f54b61904..af62c888e5 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -2080,6 +2080,10 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine) * Memory devices may override proximity set by this entry, * providing _PXM method if necessary. */ + if (pcms->cxl_devices_state.is_enabled) { + cxl_build_srat(table_data, numa_info, nb_numa_nodes); + } + if (hotpluggable_address_space_size) { build_srat_memory(table_data, machine->device_memory->base, hotpluggable_address_space_size, nb_numa_nodes - 1, diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h index acf4418886..b4974297db 100644 --- a/include/hw/acpi/cxl.h +++ b/include/hw/acpi/cxl.h @@ -25,5 +25,6 @@ void cxl_build_cedt(GArray *table_offsets, GArray *table_data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id, CXLState *cxl_state); void build_cxl_osc_method(Aml *dev); +void cxl_build_srat(GArray *table_data, NodeInfo* numa_info, int nb_numa_nodes); #endif