From patchwork Fri Oct 14 15:10:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13007080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2DF3C4332F for ; Fri, 14 Oct 2022 15:11:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbiJNPLP (ORCPT ); Fri, 14 Oct 2022 11:11:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229867AbiJNPLP (ORCPT ); Fri, 14 Oct 2022 11:11:15 -0400 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BFC751A17 for ; Fri, 14 Oct 2022 08:11:13 -0700 (PDT) Received: from fraeml736-chm.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4MpqZN5ZWpz67P5L; Fri, 14 Oct 2022 23:10:20 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml736-chm.china.huawei.com (10.206.15.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 17:11:11 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 16:11:10 +0100 From: Jonathan Cameron To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy , "Gregory Price" , CC: , Jonathan Cameron Subject: [PATCH v9 1/5] hw/pci: PCIe Data Object Exchange emulation Date: Fri, 14 Oct 2022 16:10:41 +0100 Message-ID: <20221014151045.24781-2-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> References: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100003.china.huawei.com (7.191.160.210) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Huai-Cheng Kuo Emulation of PCIe Data Object Exchange (DOE) PCIE Base Specification r6.0 6.3 Data Object Exchange Supports multiple DOE PCIe Extended Capabilities for a single PCIe device. For each capability, a static array of DOEProtocol should be passed to pcie_doe_init(). The protocols in that array will be registered under the DOE capability structure. For each protocol, vendor ID, type, and corresponding callback function (handle_request()) should be implemented. This callback function represents how the DOE request for corresponding protocol will be handled. pcie_doe_{read/write}_config() must be appended to corresponding PCI device's config_read/write() handler to enable DOE access. In pcie_doe_read_config(), false will be returned if pci_config_read() offset is not within DOE capability range. In pcie_doe_write_config(), the function will have no affect if the address is not within the related DOE PCIE extended capability. Signed-off-by: Huai-Cheng Kuo Signed-off-by: Chris Browy Signed-off-by: Jonathan Cameron --- MAINTAINERS | 7 + hw/pci/meson.build | 1 + hw/pci/pcie_doe.c | 367 +++++++++++++++++++++++++++++++++++++ include/hw/pci/pci_ids.h | 3 + include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_doe.h | 123 +++++++++++++ include/hw/pci/pcie_regs.h | 4 + 7 files changed, 506 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 8ae2e43c83..562e1d02a0 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1832,6 +1832,13 @@ F: qapi/pci.json F: docs/pci* F: docs/specs/*pci* +PCIE DOE +M: Huai-Cheng Kuo +M: Chris Browy +S: Supported +F: include/hw/pci/pcie_doe.h +F: hw/pci/pcie_doe.c + ACPI/SMBIOS M: Michael S. Tsirkin M: Igor Mammedov diff --git a/hw/pci/meson.build b/hw/pci/meson.build index bcc9c75919..5aff7ed1c6 100644 --- a/hw/pci/meson.build +++ b/hw/pci/meson.build @@ -13,6 +13,7 @@ pci_ss.add(files( # allow plugging PCIe devices into PCI buses, include them even if # CONFIG_PCI_EXPRESS=n. pci_ss.add(files('pcie.c', 'pcie_aer.c')) +pci_ss.add(files('pcie_doe.c')) softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c', 'pcie_host.c')) softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss) diff --git a/hw/pci/pcie_doe.c b/hw/pci/pcie_doe.c new file mode 100644 index 0000000000..2210f86968 --- /dev/null +++ b/hw/pci/pcie_doe.c @@ -0,0 +1,367 @@ +/* + * PCIe Data Object Exchange + * + * Copyright (C) 2021 Avery Design Systems, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "qemu/log.h" +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "qemu/range.h" +#include "hw/pci/pci.h" +#include "hw/pci/pcie.h" +#include "hw/pci/pcie_doe.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" + +#define DWORD_BYTE 4 + +typedef struct DoeDiscoveryReq { + DOEHeader header; + uint8_t index; + uint8_t reserved[3]; +} QEMU_PACKED DoeDiscoveryReq; + +typedef struct DoeDiscoveryRsp { + DOEHeader header; + uint16_t vendor_id; + uint8_t data_obj_type; + uint8_t next_index; +} QEMU_PACKED DoeDiscoveryRsp; + +static bool pcie_doe_discovery(DOECap *doe_cap) +{ + DoeDiscoveryReq *req = pcie_doe_get_write_mbox_ptr(doe_cap); + DoeDiscoveryRsp rsp; + uint8_t index = req->index; + DOEProtocol *prot; + + /* Discard request if length does not match DoeDiscoveryReq */ + if (pcie_doe_get_obj_len(req) < + DIV_ROUND_UP(sizeof(DoeDiscoveryReq), DWORD_BYTE)) { + return false; + } + + rsp.header = (DOEHeader) { + .vendor_id = PCI_VENDOR_ID_PCI_SIG, + .data_obj_type = PCI_SIG_DOE_DISCOVERY, + .length = DIV_ROUND_UP(sizeof(DoeDiscoveryRsp), DWORD_BYTE), + }; + + /* Point to the requested protocol, index 0 must be Discovery */ + if (index == 0) { + rsp.vendor_id = PCI_VENDOR_ID_PCI_SIG; + rsp.data_obj_type = PCI_SIG_DOE_DISCOVERY; + } else { + if (index < doe_cap->protocol_num) { + prot = &doe_cap->protocols[index - 1]; + rsp.vendor_id = prot->vendor_id; + rsp.data_obj_type = prot->data_obj_type; + } else { + rsp.vendor_id = 0xFFFF; + rsp.data_obj_type = 0xFF; + } + } + + if (index + 1 == doe_cap->protocol_num) { + rsp.next_index = 0; + } else { + rsp.next_index = index + 1; + } + + pcie_doe_set_rsp(doe_cap, &rsp); + + return true; +} + +static void pcie_doe_reset_mbox(DOECap *st) +{ + st->read_mbox_idx = 0; + st->read_mbox_len = 0; + st->write_mbox_len = 0; + + memset(st->read_mbox, 0, PCI_DOE_DW_SIZE_MAX * DWORD_BYTE); + memset(st->write_mbox, 0, PCI_DOE_DW_SIZE_MAX * DWORD_BYTE); +} + +void pcie_doe_init(PCIDevice *dev, DOECap *doe_cap, uint16_t offset, + DOEProtocol *protocols, bool intr, uint16_t vec) +{ + pcie_add_capability(dev, PCI_EXT_CAP_ID_DOE, 0x1, offset, + PCI_DOE_SIZEOF); + + doe_cap->pdev = dev; + doe_cap->offset = offset; + + if (intr && (msi_present(dev) || msix_present(dev))) { + doe_cap->cap.intr = intr; + doe_cap->cap.vec = vec; + } + + doe_cap->write_mbox = g_malloc0(PCI_DOE_DW_SIZE_MAX * DWORD_BYTE); + doe_cap->read_mbox = g_malloc0(PCI_DOE_DW_SIZE_MAX * DWORD_BYTE); + + pcie_doe_reset_mbox(doe_cap); + + doe_cap->protocols = protocols; + for (; protocols->vendor_id; protocols++) { + doe_cap->protocol_num++; + } + assert(doe_cap->protocol_num < PCI_DOE_PROTOCOL_NUM_MAX); + + /* Increment to allow for the discovery protocol */ + doe_cap->protocol_num++; +} + +void pcie_doe_fini(DOECap *doe_cap) +{ + g_free(doe_cap->read_mbox); + g_free(doe_cap->write_mbox); + g_free(doe_cap); +} + +uint32_t pcie_doe_build_protocol(DOEProtocol *p) +{ + return DATA_OBJ_BUILD_HEADER1(p->vendor_id, p->data_obj_type); +} + +void *pcie_doe_get_write_mbox_ptr(DOECap *doe_cap) +{ + return doe_cap->write_mbox; +} + +/* + * Copy the response to read mailbox buffer + * This might be called in self-defined handle_request() if a DOE response is + * required in the corresponding protocol + */ +void pcie_doe_set_rsp(DOECap *doe_cap, void *rsp) +{ + uint32_t len = pcie_doe_get_obj_len(rsp); + + memcpy(doe_cap->read_mbox + doe_cap->read_mbox_len, rsp, len * DWORD_BYTE); + doe_cap->read_mbox_len += len; +} + +uint32_t pcie_doe_get_obj_len(void *obj) +{ + uint32_t len; + + if (!obj) { + return 0; + } + + /* Only lower 18 bits are valid */ + len = DATA_OBJ_LEN_MASK(((DOEHeader *)obj)->length); + + /* PCIe r6.0 Table 6.29: a value of 00000h indicates 2^18 DW */ + return (len) ? len : PCI_DOE_DW_SIZE_MAX; +} + +static void pcie_doe_irq_assert(DOECap *doe_cap) +{ + PCIDevice *dev = doe_cap->pdev; + + if (doe_cap->cap.intr && doe_cap->ctrl.intr) { + if (doe_cap->status.intr) { + return; + } + doe_cap->status.intr = 1; + + if (msix_enabled(dev)) { + msix_notify(dev, doe_cap->cap.vec); + } else if (msi_enabled(dev)) { + msi_notify(dev, doe_cap->cap.vec); + } + } +} + +static void pcie_doe_set_ready(DOECap *doe_cap, bool rdy) +{ + doe_cap->status.ready = rdy; + + if (rdy) { + pcie_doe_irq_assert(doe_cap); + } +} + +static void pcie_doe_set_error(DOECap *doe_cap, bool err) +{ + doe_cap->status.error = err; + + if (err) { + pcie_doe_irq_assert(doe_cap); + } +} + +/* + * Check incoming request in write_mbox for protocol format + */ +static void pcie_doe_prepare_rsp(DOECap *doe_cap) +{ + bool success = false; + int p; + bool (*handle_request)(DOECap *) = NULL; + + if (doe_cap->status.error) { + return; + } + + if (doe_cap->write_mbox[0] == + DATA_OBJ_BUILD_HEADER1(PCI_VENDOR_ID_PCI_SIG, PCI_SIG_DOE_DISCOVERY)) { + handle_request = pcie_doe_discovery; + } else { + for (p = 0; p < doe_cap->protocol_num - 1; p++) { + if (doe_cap->write_mbox[0] == + pcie_doe_build_protocol(&doe_cap->protocols[p])) { + handle_request = doe_cap->protocols[p].handle_request; + break; + } + } + } + + /* + * PCIe r6 DOE 6.30.1: + * If the number of DW transferred does not match the + * indicated Length for a data object, then the + * data object must be silently discarded. + */ + if (handle_request && (doe_cap->write_mbox_len == + pcie_doe_get_obj_len(pcie_doe_get_write_mbox_ptr(doe_cap)))) { + success = handle_request(doe_cap); + } + + if (success) { + pcie_doe_set_ready(doe_cap, 1); + } else { + pcie_doe_reset_mbox(doe_cap); + } +} + +/* + * Read from DOE config space. + * Return false if the address not within DOE_CAP range. + */ +bool pcie_doe_read_config(DOECap *doe_cap, uint32_t addr, int size, + uint32_t *buf) +{ + uint32_t shift; + uint16_t doe_offset = doe_cap->offset; + + if (!range_covers_byte(doe_offset + PCI_EXP_DOE_CAP, + PCI_DOE_SIZEOF - 4, addr)) { + return false; + } + + addr -= doe_offset; + *buf = 0; + + if (range_covers_byte(PCI_EXP_DOE_CAP, DWORD_BYTE, addr)) { + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_REG, INTR_SUPP, + doe_cap->cap.intr); + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_REG, DOE_INTR_MSG_NUM, + doe_cap->cap.vec); + } else if (range_covers_byte(PCI_EXP_DOE_CTRL, DWORD_BYTE, addr)) { + /* Must return ABORT=0 and GO=0 */ + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_CONTROL, DOE_INTR_EN, + doe_cap->ctrl.intr); + } else if (range_covers_byte(PCI_EXP_DOE_STATUS, DWORD_BYTE, addr)) { + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_STATUS, DOE_BUSY, + doe_cap->status.busy); + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_STATUS, DOE_INTR_STATUS, + doe_cap->status.intr); + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_STATUS, DOE_ERROR, + doe_cap->status.error); + *buf = FIELD_DP32(*buf, PCI_DOE_CAP_STATUS, DATA_OBJ_RDY, + doe_cap->status.ready); + /* Mailbox should be DW accessed */ + } else if (addr == PCI_EXP_DOE_RD_DATA_MBOX && size == DWORD_BYTE) { + if (doe_cap->status.ready && !doe_cap->status.error) { + *buf = doe_cap->read_mbox[doe_cap->read_mbox_idx]; + } + } + + /* Process Alignment */ + shift = addr % DWORD_BYTE; + *buf = extract32(*buf, shift * 8, size * 8); + + return true; +} + +/* + * Write to DOE config space. + * Return if the address not within DOE_CAP range or receives an abort + */ +void pcie_doe_write_config(DOECap *doe_cap, + uint32_t addr, uint32_t val, int size) +{ + uint16_t doe_offset = doe_cap->offset; + uint32_t shift; + + if (!range_covers_byte(doe_offset + PCI_EXP_DOE_CAP, + PCI_DOE_SIZEOF - 4, addr)) { + return; + } + + /* Process Alignment */ + shift = addr % DWORD_BYTE; + addr -= (doe_offset + shift); + val = deposit32(val, shift * 8, size * 8, val); + + switch (addr) { + case PCI_EXP_DOE_CTRL: + if (FIELD_EX32(val, PCI_DOE_CAP_CONTROL, DOE_ABORT)) { + pcie_doe_set_ready(doe_cap, 0); + pcie_doe_set_error(doe_cap, 0); + pcie_doe_reset_mbox(doe_cap); + return; + } + + if (FIELD_EX32(val, PCI_DOE_CAP_CONTROL, DOE_GO)) { + pcie_doe_prepare_rsp(doe_cap); + } + + if (FIELD_EX32(val, PCI_DOE_CAP_CONTROL, DOE_INTR_EN)) { + doe_cap->ctrl.intr = 1; + /* Clear interrupt bit located within the first byte */ + } else if (shift == 0) { + doe_cap->ctrl.intr = 0; + } + break; + case PCI_EXP_DOE_STATUS: + if (FIELD_EX32(val, PCI_DOE_CAP_STATUS, DOE_INTR_STATUS)) { + doe_cap->status.intr = 0; + } + break; + case PCI_EXP_DOE_RD_DATA_MBOX: + /* Mailbox should be DW accessed */ + if (size != DWORD_BYTE) { + return; + } + doe_cap->read_mbox_idx++; + if (doe_cap->read_mbox_idx == doe_cap->read_mbox_len) { + pcie_doe_reset_mbox(doe_cap); + pcie_doe_set_ready(doe_cap, 0); + } else if (doe_cap->read_mbox_idx > doe_cap->read_mbox_len) { + /* Underflow */ + pcie_doe_set_error(doe_cap, 1); + } + break; + case PCI_EXP_DOE_WR_DATA_MBOX: + /* Mailbox should be DW accessed */ + if (size != DWORD_BYTE) { + return; + } + doe_cap->write_mbox[doe_cap->write_mbox_len] = val; + doe_cap->write_mbox_len++; + break; + case PCI_EXP_DOE_CAP: + /* fallthrough */ + default: + break; + } +} diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h index d5ddea558b..bc9f834fd1 100644 --- a/include/hw/pci/pci_ids.h +++ b/include/hw/pci/pci_ids.h @@ -157,6 +157,9 @@ /* Vendors and devices. Sort key: vendor first, device next. */ +/* Ref: PCIe r6.0 Table 6-32 */ +#define PCI_VENDOR_ID_PCI_SIG 0x0001 + #define PCI_VENDOR_ID_LSI_LOGIC 0x1000 #define PCI_DEVICE_ID_LSI_53C810 0x0001 #define PCI_DEVICE_ID_LSI_53C895A 0x0012 diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h index 798a262a0a..698d3de851 100644 --- a/include/hw/pci/pcie.h +++ b/include/hw/pci/pcie.h @@ -26,6 +26,7 @@ #include "hw/pci/pcie_aer.h" #include "hw/pci/pcie_sriov.h" #include "hw/hotplug.h" +#include "hw/pci/pcie_doe.h" typedef enum { /* for attention and power indicator */ diff --git a/include/hw/pci/pcie_doe.h b/include/hw/pci/pcie_doe.h new file mode 100644 index 0000000000..ba4d8b03bd --- /dev/null +++ b/include/hw/pci/pcie_doe.h @@ -0,0 +1,123 @@ +/* + * PCIe Data Object Exchange + * + * Copyright (C) 2021 Avery Design Systems, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#ifndef PCIE_DOE_H +#define PCIE_DOE_H + +#include "qemu/range.h" +#include "qemu/typedefs.h" +#include "hw/register.h" + +/* + * Reference: + * PCIe r6.0 - 7.9.24 Data Object Exchange Extended Capability + */ +/* Capabilities Register - r6.0 7.9.24.2 */ +#define PCI_EXP_DOE_CAP 0x04 +REG32(PCI_DOE_CAP_REG, 0) + FIELD(PCI_DOE_CAP_REG, INTR_SUPP, 0, 1) + FIELD(PCI_DOE_CAP_REG, DOE_INTR_MSG_NUM, 1, 11) + +/* Control Register - r6.0 7.9.24.3 */ +#define PCI_EXP_DOE_CTRL 0x08 +REG32(PCI_DOE_CAP_CONTROL, 0) + FIELD(PCI_DOE_CAP_CONTROL, DOE_ABORT, 0, 1) + FIELD(PCI_DOE_CAP_CONTROL, DOE_INTR_EN, 1, 1) + FIELD(PCI_DOE_CAP_CONTROL, DOE_GO, 31, 1) + +/* Status Register - r6.0 7.9.24.4 */ +#define PCI_EXP_DOE_STATUS 0x0c +REG32(PCI_DOE_CAP_STATUS, 0) + FIELD(PCI_DOE_CAP_STATUS, DOE_BUSY, 0, 1) + FIELD(PCI_DOE_CAP_STATUS, DOE_INTR_STATUS, 1, 1) + FIELD(PCI_DOE_CAP_STATUS, DOE_ERROR, 2, 1) + FIELD(PCI_DOE_CAP_STATUS, DATA_OBJ_RDY, 31, 1) + +/* Write Data Mailbox Register - r6.0 7.9.24.5 */ +#define PCI_EXP_DOE_WR_DATA_MBOX 0x10 + +/* Read Data Mailbox Register - 7.9.xx.6 */ +#define PCI_EXP_DOE_RD_DATA_MBOX 0x14 + +/* PCI-SIG defined Data Object Types - r6.0 Table 6-32 */ +#define PCI_SIG_DOE_DISCOVERY 0x00 + +#define PCI_DOE_DW_SIZE_MAX (1 << 18) +#define PCI_DOE_PROTOCOL_NUM_MAX 256 + +#define DATA_OBJ_BUILD_HEADER1(v, p) (((p) << 16) | (v)) +#define DATA_OBJ_LEN_MASK(len) ((len) & (PCI_DOE_DW_SIZE_MAX - 1)) + +typedef struct DOEHeader DOEHeader; +typedef struct DOEProtocol DOEProtocol; +typedef struct DOECap DOECap; + +struct DOEHeader { + uint16_t vendor_id; + uint8_t data_obj_type; + uint8_t reserved; + uint32_t length; +} QEMU_PACKED; + +/* Protocol infos and rsp function callback */ +struct DOEProtocol { + uint16_t vendor_id; + uint8_t data_obj_type; + bool (*handle_request)(DOECap *); +}; + +struct DOECap { + /* Owner */ + PCIDevice *pdev; + + uint16_t offset; + + struct { + bool intr; + uint16_t vec; + } cap; + + struct { + bool abort; + bool intr; + bool go; + } ctrl; + + struct { + bool busy; + bool intr; + bool error; + bool ready; + } status; + + uint32_t *write_mbox; + uint32_t *read_mbox; + + /* Mailbox position indicator */ + uint32_t read_mbox_idx; + uint32_t read_mbox_len; + uint32_t write_mbox_len; + + /* Protocols and its callback response */ + DOEProtocol *protocols; + uint16_t protocol_num; +}; + +void pcie_doe_init(PCIDevice *pdev, DOECap *doe_cap, uint16_t offset, + DOEProtocol *protocols, bool intr, uint16_t vec); +void pcie_doe_fini(DOECap *doe_cap); +bool pcie_doe_read_config(DOECap *doe_cap, uint32_t addr, int size, + uint32_t *buf); +void pcie_doe_write_config(DOECap *doe_cap, uint32_t addr, + uint32_t val, int size); +uint32_t pcie_doe_build_protocol(DOEProtocol *p); +void *pcie_doe_get_write_mbox_ptr(DOECap *doe_cap); +void pcie_doe_set_rsp(DOECap *doe_cap, void *rsp); +uint32_t pcie_doe_get_obj_len(void *obj); +#endif /* PCIE_DOE_H */ diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h index 1db86b0ec4..963dc2e170 100644 --- a/include/hw/pci/pcie_regs.h +++ b/include/hw/pci/pcie_regs.h @@ -179,4 +179,8 @@ typedef enum PCIExpLinkWidth { #define PCI_ACS_VER 0x1 #define PCI_ACS_SIZEOF 8 +/* DOE Capability Register Fields */ +#define PCI_DOE_VER 0x1 +#define PCI_DOE_SIZEOF 24 + #endif /* QEMU_PCIE_REGS_H */ From patchwork Fri Oct 14 15:10:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13007081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E73CEC4167E for ; Fri, 14 Oct 2022 15:11:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229964AbiJNPLr (ORCPT ); Fri, 14 Oct 2022 11:11:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229908AbiJNPLp (ORCPT ); Fri, 14 Oct 2022 11:11:45 -0400 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E07231D0D7C for ; Fri, 14 Oct 2022 08:11:43 -0700 (PDT) Received: from fraeml735-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4MpqXY3J1Dz67Y4H; Fri, 14 Oct 2022 23:08:45 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml735-chm.china.huawei.com (10.206.15.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 17:11:41 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 16:11:41 +0100 From: Jonathan Cameron To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy , "Gregory Price" , CC: , Jonathan Cameron Subject: [PATCH v9 2/5] hw/mem/cxl-type3: Add MSIX support Date: Fri, 14 Oct 2022 16:10:42 +0100 Message-ID: <20221014151045.24781-3-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> References: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100003.china.huawei.com (7.191.160.210) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org This will be used by several upcoming patch sets so break it out such that it doesn't matter which one lands first. Signed-off-by: Jonathan Cameron --- hw/mem/cxl_type3.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index a71bf1afeb..568c9d62f5 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -13,6 +13,7 @@ #include "qemu/rcu.h" #include "sysemu/hostmem.h" #include "hw/cxl/cxl.h" +#include "hw/pci/msix.h" /* * Null value of all Fs suggested by IEEE RA guidelines for use of @@ -146,6 +147,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) ComponentRegisters *regs = &cxl_cstate->crb; MemoryRegion *mr = ®s->component_registers; uint8_t *pci_conf = pci_dev->config; + unsigned short msix_num = 1; + int i; if (!cxl_setup_memory(ct3d, errp)) { return; @@ -180,6 +183,12 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, &ct3d->cxl_dstate.device_registers); + + /* MSI(-X) Initailization */ + msix_init_exclusive_bar(pci_dev, msix_num, 4, NULL); + for (i = 0; i < msix_num; i++) { + msix_vector_use(pci_dev, i); + } } static void ct3_exit(PCIDevice *pci_dev) From patchwork Fri Oct 14 15:10:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13007082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC4CC43217 for ; Fri, 14 Oct 2022 15:12:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229928AbiJNPMR (ORCPT ); Fri, 14 Oct 2022 11:12:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229572AbiJNPMQ (ORCPT ); Fri, 14 Oct 2022 11:12:16 -0400 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4DE41D20D7 for ; Fri, 14 Oct 2022 08:12:14 -0700 (PDT) Received: from fraeml713-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4MpqZg0mLnz684JJ; Fri, 14 Oct 2022 23:10:35 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml713-chm.china.huawei.com (10.206.15.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 17:12:12 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 16:12:11 +0100 From: Jonathan Cameron To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy , "Gregory Price" , CC: , Jonathan Cameron Subject: [PATCH v9 3/5] hw/cxl/cdat: CXL CDAT Data Object Exchange implementation Date: Fri, 14 Oct 2022 16:10:43 +0100 Message-ID: <20221014151045.24781-4-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> References: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml100003.china.huawei.com (7.191.160.210) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Huai-Cheng Kuo The Data Object Exchange implementation of CXL Coherent Device Attribute Table (CDAT). This implementation is referring to "Coherent Device Attribute Table Specification, Rev. 1.03, July. 2022" and "Compute Express Link Specification, Rev. 3.0, July. 2022" This patch adds core support that will be shared by both end-points and switch port emulation. Signed-off-by: Huai-Cheng Kuo Signed-off-by: Chris Browy Signed-off-by: Jonathan Cameron --- hw/cxl/cxl-cdat.c | 224 +++++++++++++++++++++++++++++++++ hw/cxl/meson.build | 1 + include/hw/cxl/cxl_cdat.h | 165 ++++++++++++++++++++++++ include/hw/cxl/cxl_component.h | 7 ++ include/hw/cxl/cxl_device.h | 3 + include/hw/cxl/cxl_pci.h | 1 + 6 files changed, 401 insertions(+) diff --git a/hw/cxl/cxl-cdat.c b/hw/cxl/cxl-cdat.c new file mode 100644 index 0000000000..3653aa56f0 --- /dev/null +++ b/hw/cxl/cxl-cdat.c @@ -0,0 +1,224 @@ +/* + * CXL CDAT Structure + * + * Copyright (C) 2021 Avery Design Systems, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "hw/pci/pci.h" +#include "hw/cxl/cxl.h" +#include "qapi/error.h" +#include "qemu/error-report.h" + +static void cdat_len_check(CDATSubHeader *hdr, Error **errp) +{ + assert(hdr->length); + assert(hdr->reserved == 0); + + switch (hdr->type) { + case CDAT_TYPE_DSMAS: + assert(hdr->length == sizeof(CDATDsmas)); + break; + case CDAT_TYPE_DSLBIS: + assert(hdr->length == sizeof(CDATDslbis)); + break; + case CDAT_TYPE_DSMSCIS: + assert(hdr->length == sizeof(CDATDsmscis)); + break; + case CDAT_TYPE_DSIS: + assert(hdr->length == sizeof(CDATDsis)); + break; + case CDAT_TYPE_DSEMTS: + assert(hdr->length == sizeof(CDATDsemts)); + break; + case CDAT_TYPE_SSLBIS: + assert(hdr->length >= sizeof(CDATSslbisHeader)); + assert((hdr->length - sizeof(CDATSslbisHeader)) % + sizeof(CDATSslbe) == 0); + break; + default: + error_setg(errp, "Type %d is reserved", hdr->type); + } +} + +static void ct3_build_cdat(CDATObject *cdat, Error **errp) +{ + g_autofree CDATTableHeader *cdat_header = NULL; + g_autofree CDATEntry *cdat_st = NULL; + uint8_t sum = 0; + int ent, i; + + /* Use default table if fopen == NULL */ + assert(cdat->build_cdat_table); + + cdat_header = g_malloc0(sizeof(*cdat_header)); + if (!cdat_header) { + error_setg(errp, "Failed to allocate CDAT header"); + return; + } + + cdat->built_buf_len = cdat->build_cdat_table(&cdat->built_buf, cdat->private); + + if (!cdat->built_buf_len) { + /* Build later as not all data available yet */ + cdat->to_update = true; + return; + } + cdat->to_update = false; + + cdat_st = g_malloc0(sizeof(*cdat_st) * (cdat->built_buf_len + 1)); + if (!cdat_st) { + error_setg(errp, "Failed to allocate CDAT entry array"); + return; + } + + /* Entry 0 for CDAT header, starts with Entry 1 */ + for (ent = 1; ent < cdat->built_buf_len + 1; ent++) { + CDATSubHeader *hdr = cdat->built_buf[ent - 1]; + uint8_t *buf = (uint8_t *)cdat->built_buf[ent - 1]; + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + + cdat_header->length += hdr->length; + for (i = 0; i < hdr->length; i++) { + sum += buf[i]; + } + } + + /* CDAT header */ + cdat_header->revision = CXL_CDAT_REV; + /* For now, no runtime updates */ + cdat_header->sequence = 0; + cdat_header->length += sizeof(CDATTableHeader); + sum += cdat_header->revision + cdat_header->sequence + + cdat_header->length; + /* Sum of all bytes including checksum must be 0 */ + cdat_header->checksum = ~sum + 1; + + cdat_st[0].base = g_steal_pointer(&cdat_header); + cdat_st[0].length = sizeof(*cdat_header); + cdat->entry_len = 1 + cdat->built_buf_len; + cdat->entry = g_steal_pointer(&cdat_st); +} + +static void ct3_load_cdat(CDATObject *cdat, Error **errp) +{ + g_autofree CDATEntry *cdat_st = NULL; + uint8_t sum = 0; + int num_ent; + int i = 0, ent = 1, file_size = 0; + CDATSubHeader *hdr; + FILE *fp = NULL; + + /* Read CDAT file and create its cache */ + fp = fopen(cdat->filename, "r"); + if (!fp) { + error_setg(errp, "CDAT: Unable to open file"); + return; + } + + fseek(fp, 0, SEEK_END); + file_size = ftell(fp); + fseek(fp, 0, SEEK_SET); + cdat->buf = g_malloc0(file_size); + + if (fread(cdat->buf, file_size, 1, fp) == 0) { + error_setg(errp, "CDAT: File read failed"); + return; + } + + fclose(fp); + + if (file_size < sizeof(CDATTableHeader)) { + error_setg(errp, "CDAT: File too short"); + return; + } + i = sizeof(CDATTableHeader); + num_ent = 1; + while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); + cdat_len_check(hdr, errp); + i += hdr->length; + num_ent++; + } + if (i != file_size) { + error_setg(errp, "CDAT: File length missmatch"); + return; + } + + cdat_st = g_malloc0(sizeof(*cdat_st) * num_ent); + if (!cdat_st) { + error_setg(errp, "CDAT: Failed to allocate entry array"); + return; + } + + /* Set CDAT header, Entry = 0 */ + cdat_st[0].base = cdat->buf; + cdat_st[0].length = sizeof(CDATTableHeader); + i = 0; + + while (i < cdat_st[0].length) { + sum += cdat->buf[i++]; + } + + /* Read CDAT structures */ + while (i < file_size) { + hdr = (CDATSubHeader *)(cdat->buf + i); + cdat_len_check(hdr, errp); + + cdat_st[ent].base = hdr; + cdat_st[ent].length = hdr->length; + + while (cdat->buf + i < + (uint8_t *)cdat_st[ent].base + cdat_st[ent].length) { + assert(i < file_size); + sum += cdat->buf[i++]; + } + + ent++; + } + + if (sum != 0) { + warn_report("CDAT: Found checksum mismatch in %s", cdat->filename); + } + cdat->entry_len = num_ent; + cdat->entry = g_steal_pointer(&cdat_st); +} + +void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp) +{ + CDATObject *cdat = &cxl_cstate->cdat; + + if (cdat->filename) { + ct3_load_cdat(cdat, errp); + } else { + ct3_build_cdat(cdat, errp); + } +} + +void cxl_doe_cdat_update(CXLComponentState *cxl_cstate, Error **errp) +{ + CDATObject *cdat = &cxl_cstate->cdat; + + if (cdat->to_update) { + ct3_build_cdat(cdat, errp); + } +} + +void cxl_doe_cdat_release(CXLComponentState *cxl_cstate) +{ + CDATObject *cdat = &cxl_cstate->cdat; + + free(cdat->entry); + if (cdat->built_buf) { + cdat->free_cdat_table(cdat->built_buf, cdat->built_buf_len, + cdat->private); + } + if (cdat->buf) { + free(cdat->buf); + } +} diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build index f117b99949..cfa95ffd40 100644 --- a/hw/cxl/meson.build +++ b/hw/cxl/meson.build @@ -4,6 +4,7 @@ softmmu_ss.add(when: 'CONFIG_CXL', 'cxl-device-utils.c', 'cxl-mailbox-utils.c', 'cxl-host.c', + 'cxl-cdat.c', ), if_false: files( 'cxl-host-stubs.c', diff --git a/include/hw/cxl/cxl_cdat.h b/include/hw/cxl/cxl_cdat.h new file mode 100644 index 0000000000..52c232e912 --- /dev/null +++ b/include/hw/cxl/cxl_cdat.h @@ -0,0 +1,165 @@ +/* + * CXL CDAT Structure + * + * Copyright (C) 2021 Avery Design Systems, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#ifndef CXL_CDAT_H +#define CXL_CDAT_H + +#include "hw/cxl/cxl_pci.h" + +/* + * Reference: + * Coherent Device Attribute Table (CDAT) Specification, Rev. 1.03, July. 2022 + * Compute Express Link (CXL) Specification, Rev. 3.0, Aug. 2022 + */ +/* Table Access DOE - CXL r3.0 8.1.11 */ +#define CXL_DOE_TABLE_ACCESS 2 +#define CXL_DOE_PROTOCOL_CDAT ((CXL_DOE_TABLE_ACCESS << 16) | CXL_VENDOR_ID) + +/* Read Entry - CXL r3.0 8.1.11.1 */ +#define CXL_DOE_TAB_TYPE_CDAT 0 +#define CXL_DOE_TAB_ENT_MAX 0xFFFF + +/* Read Entry Request - CXL r3.0 8.1.11.1 Table 8-13 */ +#define CXL_DOE_TAB_REQ 0 +typedef struct CDATReq { + DOEHeader header; + uint8_t req_code; + uint8_t table_type; + uint16_t entry_handle; +} QEMU_PACKED CDATReq; + +/* Read Entry Response - CXL r3.0 8.1.11.1 Table 8-14 */ +#define CXL_DOE_TAB_RSP 0 +typedef struct CDATRsp { + DOEHeader header; + uint8_t rsp_code; + uint8_t table_type; + uint16_t entry_handle; +} QEMU_PACKED CDATRsp; + +/* CDAT Table Format - CDAT Table 1 */ +#define CXL_CDAT_REV 2 +typedef struct CDATTableHeader { + uint32_t length; + uint8_t revision; + uint8_t checksum; + uint8_t reserved[6]; + uint32_t sequence; +} QEMU_PACKED CDATTableHeader; + +/* CDAT Structure Types - CDAT Table 2 */ +typedef enum { + CDAT_TYPE_DSMAS = 0, + CDAT_TYPE_DSLBIS = 1, + CDAT_TYPE_DSMSCIS = 2, + CDAT_TYPE_DSIS = 3, + CDAT_TYPE_DSEMTS = 4, + CDAT_TYPE_SSLBIS = 5, +} CDATType; + +typedef struct CDATSubHeader { + uint8_t type; + uint8_t reserved; + uint16_t length; +} CDATSubHeader; + +/* Device Scoped Memory Affinity Structure - CDAT Table 3 */ +typedef struct CDATDsmas { + CDATSubHeader header; + uint8_t DSMADhandle; + uint8_t flags; +#define CDAT_DSMAS_FLAG_NV (1 << 2) +#define CDAT_DSMAS_FLAG_SHAREABLE (1 << 3) +#define CDAT_DSMAS_FLAG_HW_COHERENT (1 << 4) +#define CDAT_DSMAS_FLAG_DYNAMIC_CAP (1 << 5) + uint16_t reserved; + uint64_t DPA_base; + uint64_t DPA_length; +} QEMU_PACKED CDATDsmas; + +/* Device Scoped Latency and Bandwidth Information Structure - CDAT Table 5 */ +typedef struct CDATDslbis { + CDATSubHeader header; + uint8_t handle; + /* Definitions of these fields refer directly to HMAT fields */ + uint8_t flags; + uint8_t data_type; + uint8_t reserved; + uint64_t entry_base_unit; + uint16_t entry[3]; + uint16_t reserved2; +} QEMU_PACKED CDATDslbis; + +/* Device Scoped Memory Side Cache Information Structure - CDAT Table 6 */ +typedef struct CDATDsmscis { + CDATSubHeader header; + uint8_t DSMAS_handle; + uint8_t reserved[3]; + uint64_t memory_side_cache_size; + uint32_t cache_attributes; +} QEMU_PACKED CDATDsmscis; + +/* Device Scoped Initiator Structure - CDAT Table 7 */ +typedef struct CDATDsis { + CDATSubHeader header; + uint8_t flags; + uint8_t handle; + uint16_t reserved; +} QEMU_PACKED CDATDsis; + +/* Device Scoped EFI Memory Type Structure - CDAT Table 8 */ +typedef struct CDATDsemts { + CDATSubHeader header; + uint8_t DSMAS_handle; + uint8_t EFI_memory_type_attr; + uint16_t reserved; + uint64_t DPA_offset; + uint64_t DPA_length; +} QEMU_PACKED CDATDsemts; + +/* Switch Scoped Latency and Bandwidth Information Structure - CDAT Table 9 */ +typedef struct CDATSslbisHeader { + CDATSubHeader header; + uint8_t data_type; + uint8_t reserved[3]; + uint64_t entry_base_unit; +} QEMU_PACKED CDATSslbisHeader; + +/* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ +typedef struct CDATSslbe { + uint16_t port_x_id; + uint16_t port_y_id; + uint16_t latency_bandwidth; + uint16_t reserved; +} QEMU_PACKED CDATSslbe; + +typedef struct CDATSslbis { + CDATSslbisHeader sslbis_header; + CDATSslbe sslbe[]; +} QEMU_PACKED CDATSslbis; + +typedef struct CDATEntry { + void *base; + uint32_t length; +} CDATEntry; + +typedef struct CDATObject { + CDATEntry *entry; + int entry_len; + + int (*build_cdat_table)(CDATSubHeader ***cdat_table, void *priv); + void (*free_cdat_table)(CDATSubHeader **cdat_table, int num, void *priv); + bool to_update; + void *private; + char *filename; + uint8_t *buf; + struct CDATSubHeader **built_buf; + int built_buf_len; +} CDATObject; +#endif /* CXL_CDAT_H */ diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h index 94ec2f07d7..34075cfb72 100644 --- a/include/hw/cxl/cxl_component.h +++ b/include/hw/cxl/cxl_component.h @@ -19,6 +19,7 @@ #include "qemu/range.h" #include "qemu/typedefs.h" #include "hw/register.h" +#include "qapi/error.h" enum reg_type { CXL2_DEVICE, @@ -184,6 +185,8 @@ typedef struct cxl_component { struct PCIDevice *pdev; }; }; + + CDATObject cdat; } CXLComponentState; void cxl_component_register_block_init(Object *obj, @@ -220,4 +223,8 @@ static inline hwaddr cxl_decode_ig(int ig) CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb); +void cxl_doe_cdat_init(CXLComponentState *cxl_cstate, Error **errp); +void cxl_doe_cdat_release(CXLComponentState *cxl_cstate); +void cxl_doe_cdat_update(CXLComponentState *cxl_cstate, Error **errp); + #endif diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index e4d221cdb3..449b0edfe9 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -243,6 +243,9 @@ struct CXLType3Dev { AddressSpace hostmem_as; CXLComponentState cxl_cstate; CXLDeviceState cxl_dstate; + + /* DOE */ + DOECap doe_cdat; }; #define TYPE_CXL_TYPE3 "cxl-type3" diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h index 01cf002096..3cb79eca1e 100644 --- a/include/hw/cxl/cxl_pci.h +++ b/include/hw/cxl/cxl_pci.h @@ -13,6 +13,7 @@ #include "qemu/compiler.h" #include "hw/pci/pci.h" #include "hw/pci/pcie.h" +#include "hw/cxl/cxl_cdat.h" #define CXL_VENDOR_ID 0x1e98 From patchwork Fri Oct 14 15:10:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13007086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03F22C433FE for ; Fri, 14 Oct 2022 15:12:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229540AbiJNPMz (ORCPT ); Fri, 14 Oct 2022 11:12:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbiJNPMx (ORCPT ); Fri, 14 Oct 2022 11:12:53 -0400 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9363B1D4602 for ; Fri, 14 Oct 2022 08:12:45 -0700 (PDT) Received: from fraeml712-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4MpqYk4MHTz686PB; Fri, 14 Oct 2022 23:09:46 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml712-chm.china.huawei.com (10.206.15.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 17:12:43 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 16:12:42 +0100 From: Jonathan Cameron To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy , "Gregory Price" , CC: , Jonathan Cameron Subject: [PATCH v9 4/5] hw/mem/cxl-type3: Add CXL CDAT Data Object Exchange Date: Fri, 14 Oct 2022 16:10:44 +0100 Message-ID: <20221014151045.24781-5-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> References: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml500004.china.huawei.com (7.191.163.9) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Huai-Cheng Kuo The CDAT can be specified in two ways. One is to add ",cdat=" in "-device cxl-type3"'s command option. The file is required to provide the whole CDAT table in binary mode. The other is to use the default that provides some 'reasonable' numbers based on type of memory and size. The DOE capability supporting CDAT is added to hw/mem/cxl_type3.c with capability offset 0x190. The config read/write to this capability range can be generated in the OS to request the CDAT data. Signed-off-by: Huai-Cheng Kuo Signed-off-by: Chris Browy Signed-off-by: Jonathan Cameron Reported-by: Gregory Price Reported-by: Jonathan Cameron Signed-off-by: Ira Weiny --- Changes since v8: Thanks to Gregory Price and Michael Tsirkin. - Move enum out of function and prefix entries with CT3_CDAT Related refactoring now the size is visible. - Fix wrong naming of volatile which should be nonvolatile. --- hw/mem/cxl_type3.c | 255 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 255 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 568c9d62f5..255590201a 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -12,9 +12,246 @@ #include "qemu/range.h" #include "qemu/rcu.h" #include "sysemu/hostmem.h" +#include "sysemu/numa.h" #include "hw/cxl/cxl.h" #include "hw/pci/msix.h" +#define DWORD_BYTE 4 + +/* Default CDAT entries for a memory region */ +enum { + CT3_CDAT_DSMAS, + CT3_CDAT_DSLBIS0, + CT3_CDAT_DSLBIS1, + CT3_CDAT_DSLBIS2, + CT3_CDAT_DSLBIS3, + CT3_CDAT_DSEMTS, + CT3_CDAT_NUM_ENTRIES +}; + +static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, + int dsmad_handle, MemoryRegion *mr) +{ + g_autofree CDATDsmas *dsmas = NULL; + g_autofree CDATDslbis *dslbis0 = NULL; + g_autofree CDATDslbis *dslbis1 = NULL; + g_autofree CDATDslbis *dslbis2 = NULL; + g_autofree CDATDslbis *dslbis3 = NULL; + g_autofree CDATDsemts *dsemts = NULL; + + dsmas = g_malloc(sizeof(*dsmas)); + if (!dsmas) { + return -ENOMEM; + } + *dsmas = (CDATDsmas) { + .header = { + .type = CDAT_TYPE_DSMAS, + .length = sizeof(*dsmas), + }, + .DSMADhandle = dsmad_handle, + .flags = CDAT_DSMAS_FLAG_NV, + .DPA_base = 0, + .DPA_length = int128_get64(mr->size), + }; + + /* For now, no memory side cache, plausiblish numbers */ + dslbis0 = g_malloc(sizeof(*dslbis0)); + if (!dslbis0) { + return -ENOMEM; + } + *dslbis0 = (CDATDslbis) { + .header = { + .type = CDAT_TYPE_DSLBIS, + .length = sizeof(*dslbis0), + }, + .handle = dsmad_handle, + .flags = HMAT_LB_MEM_MEMORY, + .data_type = HMAT_LB_DATA_READ_LATENCY, + .entry_base_unit = 10000, /* 10ns base */ + .entry[0] = 15, /* 150ns */ + }; + + dslbis1 = g_malloc(sizeof(*dslbis1)); + if (!dslbis1) { + return -ENOMEM; + } + *dslbis1 = (CDATDslbis) { + .header = { + .type = CDAT_TYPE_DSLBIS, + .length = sizeof(*dslbis1), + }, + .handle = dsmad_handle, + .flags = HMAT_LB_MEM_MEMORY, + .data_type = HMAT_LB_DATA_WRITE_LATENCY, + .entry_base_unit = 10000, + .entry[0] = 25, /* 250ns */ + }; + + dslbis2 = g_malloc(sizeof(*dslbis2)); + if (!dslbis2) { + return -ENOMEM; + } + *dslbis2 = (CDATDslbis) { + .header = { + .type = CDAT_TYPE_DSLBIS, + .length = sizeof(*dslbis2), + }, + .handle = dsmad_handle, + .flags = HMAT_LB_MEM_MEMORY, + .data_type = HMAT_LB_DATA_READ_BANDWIDTH, + .entry_base_unit = 1000, /* GB/s */ + .entry[0] = 16, + }; + + dslbis3 = g_malloc(sizeof(*dslbis3)); + if (!dslbis3) { + return -ENOMEM; + } + *dslbis3 = (CDATDslbis) { + .header = { + .type = CDAT_TYPE_DSLBIS, + .length = sizeof(*dslbis3), + }, + .handle = dsmad_handle, + .flags = HMAT_LB_MEM_MEMORY, + .data_type = HMAT_LB_DATA_WRITE_BANDWIDTH, + .entry_base_unit = 1000, /* GB/s */ + .entry[0] = 16, + }; + + dsemts = g_malloc(sizeof(*dsemts)); + if (!dsemts) { + return -ENOMEM; + } + *dsemts = (CDATDsemts) { + .header = { + .type = CDAT_TYPE_DSEMTS, + .length = sizeof(*dsemts), + }, + .DSMAS_handle = dsmad_handle, + /* Reserved - the non volatile from DSMAS matters */ + .EFI_memory_type_attr = 2, + .DPA_offset = 0, + .DPA_length = int128_get64(mr->size), + }; + + /* Header always at start of structure */ + cdat_table[CT3_CDAT_DSMAS] = g_steal_pointer(&dsmas); + cdat_table[CT3_CDAT_DSLBIS0] = g_steal_pointer(&dslbis0); + cdat_table[CT3_CDAT_DSLBIS1] = g_steal_pointer(&dslbis1); + cdat_table[CT3_CDAT_DSLBIS2] = g_steal_pointer(&dslbis2); + cdat_table[CT3_CDAT_DSLBIS3] = g_steal_pointer(&dslbis3); + cdat_table[CT3_CDAT_DSEMTS] = g_steal_pointer(&dsemts); + + return 0; +} + +static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) +{ + g_autofree CDATSubHeader **table = NULL; + MemoryRegion *nonvolatile_mr; + CXLType3Dev *ct3d = priv; + int dsmad_handle = 0; + int rc; + + if (!ct3d->hostmem) { + return 0; + } + + nonvolatile_mr = host_memory_backend_get_memory(ct3d->hostmem); + if (!nonvolatile_mr) { + return -EINVAL; + } + + table = g_malloc0(CT3_CDAT_NUM_ENTRIES * sizeof(*table)); + if (!table) { + return -ENOMEM; + } + + rc = ct3_build_cdat_entries_for_mr(table, dsmad_handle++, nonvolatile_mr); + if (rc < 0) { + return rc; + } + + *cdat_table = g_steal_pointer(&table); + + return CT3_CDAT_NUM_ENTRIES; +} + +static void ct3_free_cdat_table(CDATSubHeader **cdat_table, int num, void *priv) +{ + int i; + + for (i = 0; i < num; i++) { + g_free(cdat_table[i]); + } + g_free(cdat_table); +} + +static bool cxl_doe_cdat_rsp(DOECap *doe_cap) +{ + CDATObject *cdat = &CXL_TYPE3(doe_cap->pdev)->cxl_cstate.cdat; + uint16_t ent; + void *base; + uint32_t len; + CDATReq *req = pcie_doe_get_write_mbox_ptr(doe_cap); + CDATRsp rsp; + + assert(cdat->entry_len); + + /* Discard if request length mismatched */ + if (pcie_doe_get_obj_len(req) < + DIV_ROUND_UP(sizeof(CDATReq), DWORD_BYTE)) { + return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp) { + .header = { + .vendor_id = CXL_VENDOR_ID, + .data_obj_type = CXL_DOE_TABLE_ACCESS, + .reserved = 0x0, + .length = DIV_ROUND_UP((sizeof(rsp) + len), DWORD_BYTE), + }, + .rsp_code = CXL_DOE_TAB_RSP, + .table_type = CXL_DOE_TAB_TYPE_CDAT, + .entry_handle = (ent < cdat->entry_len - 1) ? + ent + 1 : CXL_DOE_TAB_ENT_MAX, + }; + + memcpy(doe_cap->read_mbox, &rsp, sizeof(rsp)); + memcpy(doe_cap->read_mbox + DIV_ROUND_UP(sizeof(rsp), DWORD_BYTE), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + + return true; +} + +static uint32_t ct3d_config_read(PCIDevice *pci_dev, uint32_t addr, int size) +{ + CXLType3Dev *ct3d = CXL_TYPE3(pci_dev); + uint32_t val; + + if (pcie_doe_read_config(&ct3d->doe_cdat, addr, size, &val)) { + return val; + } + + return pci_default_read_config(pci_dev, addr, size); +} + +static void ct3d_config_write(PCIDevice *pci_dev, uint32_t addr, uint32_t val, + int size) +{ + CXLType3Dev *ct3d = CXL_TYPE3(pci_dev); + + pcie_doe_write_config(&ct3d->doe_cdat, addr, val, size); + pci_default_write_config(pci_dev, addr, val, size); +} + /* * Null value of all Fs suggested by IEEE RA guidelines for use of * EU, OUI and CID @@ -140,6 +377,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) return true; } +static DOEProtocol doe_cdat_prot[] = { + { CXL_VENDOR_ID, CXL_DOE_TABLE_ACCESS, cxl_doe_cdat_rsp }, + { } +}; + static void ct3_realize(PCIDevice *pci_dev, Error **errp) { CXLType3Dev *ct3d = CXL_TYPE3(pci_dev); @@ -189,6 +431,14 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp) for (i = 0; i < msix_num; i++) { msix_vector_use(pci_dev, i); } + + /* DOE Initailization */ + pcie_doe_init(pci_dev, &ct3d->doe_cdat, 0x190, doe_cdat_prot, true, 0); + + cxl_cstate->cdat.build_cdat_table = ct3_build_cdat_table; + cxl_cstate->cdat.free_cdat_table = ct3_free_cdat_table; + cxl_cstate->cdat.private = ct3d; + cxl_doe_cdat_init(cxl_cstate, errp); } static void ct3_exit(PCIDevice *pci_dev) @@ -197,6 +447,7 @@ static void ct3_exit(PCIDevice *pci_dev) CXLComponentState *cxl_cstate = &ct3d->cxl_cstate; ComponentRegisters *regs = &cxl_cstate->crb; + cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); address_space_destroy(&ct3d->hostmem_as); } @@ -296,6 +547,7 @@ static Property ct3_props[] = { DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), + DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_END_OF_LIST(), }; @@ -361,6 +613,9 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->device_id = 0xd93; /* LVF for now */ pc->revision = 1; + pc->config_write = ct3d_config_write; + pc->config_read = ct3d_config_read; + set_bit(DEVICE_CATEGORY_STORAGE, dc->categories); dc->desc = "CXL PMEM Device (Type 3)"; dc->reset = ct3d_reset; From patchwork Fri Oct 14 15:10:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13007087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A63A0C4332F for ; Fri, 14 Oct 2022 15:13:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229588AbiJNPNS (ORCPT ); Fri, 14 Oct 2022 11:13:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229997AbiJNPNR (ORCPT ); Fri, 14 Oct 2022 11:13:17 -0400 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B3771D3452 for ; Fri, 14 Oct 2022 08:13:15 -0700 (PDT) Received: from fraeml711-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Mpqcl2Knkz67dBN; Fri, 14 Oct 2022 23:12:23 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (7.191.163.240) by fraeml711-chm.china.huawei.com (10.206.15.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 17:13:13 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Fri, 14 Oct 2022 16:13:13 +0100 From: Jonathan Cameron To: , Michael Tsirkin , Ben Widawsky , , Huai-Cheng Kuo , Chris Browy , "Gregory Price" , CC: , Jonathan Cameron Subject: [PATCH v9 5/5] hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE Date: Fri, 14 Oct 2022 16:10:45 +0100 Message-ID: <20221014151045.24781-6-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> References: <20221014151045.24781-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhrpeml500004.china.huawei.com (7.191.163.9) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org This Data Object Exchange Mailbox allows software to query the latency and bandwidth between ports on the switch. For now only provide information on routes between the upstream port and each downstream port (not p2p). Signed-off-by: Jonathan Cameron --- Changes since v8: Mostly to match the type 3 equivalent - Move enum out of function and give it a more descriptive namespace. --- hw/pci-bridge/cxl_upstream.c | 195 ++++++++++++++++++++++++++++++++++- include/hw/cxl/cxl_cdat.h | 1 + 2 files changed, 195 insertions(+), 1 deletion(-) diff --git a/hw/pci-bridge/cxl_upstream.c b/hw/pci-bridge/cxl_upstream.c index a83a3e81e4..9b8b57df9d 100644 --- a/hw/pci-bridge/cxl_upstream.c +++ b/hw/pci-bridge/cxl_upstream.c @@ -10,11 +10,12 @@ #include "qemu/osdep.h" #include "qemu/log.h" +#include "hw/qdev-properties.h" #include "hw/pci/msi.h" #include "hw/pci/pcie.h" #include "hw/pci/pcie_port.h" -#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 1 +#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 2 #define CXL_UPSTREAM_PORT_MSI_OFFSET 0x70 #define CXL_UPSTREAM_PORT_PCIE_CAP_OFFSET 0x90 @@ -28,6 +29,7 @@ typedef struct CXLUpstreamPort { /*< public >*/ CXLComponentState cxl_cstate; + DOECap doe_cdat; } CXLUpstreamPort; CXLComponentState *cxl_usp_to_cstate(CXLUpstreamPort *usp) @@ -60,6 +62,9 @@ static void cxl_usp_dvsec_write_config(PCIDevice *dev, uint32_t addr, static void cxl_usp_write_config(PCIDevice *d, uint32_t address, uint32_t val, int len) { + CXLUpstreamPort *usp = CXL_USP(d); + + pcie_doe_write_config(&usp->doe_cdat, address, val, len); pci_bridge_write_config(d, address, val, len); pcie_cap_flr_write_config(d, address, val, len); pcie_aer_write_config(d, address, val, len); @@ -67,6 +72,18 @@ static void cxl_usp_write_config(PCIDevice *d, uint32_t address, cxl_usp_dvsec_write_config(d, address, val, len); } +static uint32_t cxl_usp_read_config(PCIDevice *d, uint32_t address, int len) +{ + CXLUpstreamPort *usp = CXL_USP(d); + uint32_t val; + + if (pcie_doe_read_config(&usp->doe_cdat, address, len, &val)) { + return val; + } + + return pci_default_read_config(d, address, len); +} + static void latch_registers(CXLUpstreamPort *usp) { uint32_t *reg_state = usp->cxl_cstate.crb.cache_mem_registers; @@ -119,6 +136,167 @@ static void build_dvsecs(CXLComponentState *cxl) REG_LOC_DVSEC_REVID, dvsec); } +static bool cxl_doe_cdat_rsp(DOECap *doe_cap) +{ + CDATObject *cdat = &CXL_USP(doe_cap->pdev)->cxl_cstate.cdat; + uint16_t ent; + void *base; + uint32_t len; + CDATReq *req = pcie_doe_get_write_mbox_ptr(doe_cap); + CDATRsp rsp; + + cxl_doe_cdat_update(&CXL_USP(doe_cap->pdev)->cxl_cstate, &error_fatal); + assert(cdat->entry_len); + + /* Discard if request length mismatched */ + if (pcie_doe_get_obj_len(req) < + DIV_ROUND_UP(sizeof(CDATReq), sizeof(uint32_t))) { + return false; + } + + ent = req->entry_handle; + base = cdat->entry[ent].base; + len = cdat->entry[ent].length; + + rsp = (CDATRsp) { + .header = { + .vendor_id = CXL_VENDOR_ID, + .data_obj_type = CXL_DOE_TABLE_ACCESS, + .reserved = 0x0, + .length = DIV_ROUND_UP((sizeof(rsp) + len), sizeof(uint32_t)), + }, + .rsp_code = CXL_DOE_TAB_RSP, + .table_type = CXL_DOE_TAB_TYPE_CDAT, + .entry_handle = (ent < cdat->entry_len - 1) ? + ent + 1 : CXL_DOE_TAB_ENT_MAX, + }; + + memcpy(doe_cap->read_mbox, &rsp, sizeof(rsp)); + memcpy(doe_cap->read_mbox + DIV_ROUND_UP(sizeof(rsp), sizeof(uint32_t)), + base, len); + + doe_cap->read_mbox_len += rsp.header.length; + + return true; +} + +static DOEProtocol doe_cdat_prot[] = { + { CXL_VENDOR_ID, CXL_DOE_TABLE_ACCESS, cxl_doe_cdat_rsp }, + { } +}; + +enum { + CXL_USP_CDAT_SSLBIS_LAT, + CXL_USP_CDAT_SSLBIS_BW, + CXL_USP_CDAT_NUM_ENTRIES +}; + +static int build_cdat_table(CDATSubHeader ***cdat_table, void *priv) +{ + g_autofree CDATSslbis *sslbis_latency = NULL; + g_autofree CDATSslbis *sslbis_bandwidth = NULL; + CXLUpstreamPort *us = CXL_USP(priv); + PCIBus *bus = &PCI_BRIDGE(us)->sec_bus; + int devfn, sslbis_size, i; + int count = 0; + uint16_t port_ids[256]; + + for (devfn = 0; devfn < ARRAY_SIZE(bus->devices); devfn++) { + PCIDevice *d = bus->devices[devfn]; + PCIEPort *port; + + if (!d || !pci_is_express(d) || !d->exp.exp_cap) { + continue; + } + + /* + * Whilst the PCI express spec doesn't allow anything other than + * downstream ports on this bus, let us be a little paranoid + */ + if (!object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) { + continue; + } + + port = PCIE_PORT(d); + port_ids[count] = port->port; + count++; + } + + /* May not yet have any ports - try again later */ + if (count == 0) { + return 0; + } + + sslbis_size = sizeof(CDATSslbis) + sizeof(*sslbis_latency->sslbe) * count; + sslbis_latency = g_malloc(sslbis_size); + if (!sslbis_latency) { + return -ENOMEM; + } + *sslbis_latency = (CDATSslbis) { + .sslbis_header = { + .header = { + .type = CDAT_TYPE_SSLBIS, + .length = sslbis_size, + }, + .data_type = HMATLB_DATA_TYPE_ACCESS_LATENCY, + .entry_base_unit = 10000, + }, + }; + + for (i = 0; i < count; i++) { + sslbis_latency->sslbe[i] = (CDATSslbe) { + .port_x_id = CDAT_PORT_ID_USP, + .port_y_id = port_ids[i], + .latency_bandwidth = 15, /* 150ns */ + }; + } + + sslbis_bandwidth = g_malloc(sslbis_size); + if (!sslbis_bandwidth) { + return 0; + } + *sslbis_bandwidth = (CDATSslbis) { + .sslbis_header = { + .header = { + .type = CDAT_TYPE_SSLBIS, + .length = sslbis_size, + }, + .data_type = HMATLB_DATA_TYPE_ACCESS_BANDWIDTH, + .entry_base_unit = 1000, + }, + }; + + for (i = 0; i < count; i++) { + sslbis_bandwidth->sslbe[i] = (CDATSslbe) { + .port_x_id = CDAT_PORT_ID_USP, + .port_y_id = port_ids[i], + .latency_bandwidth = 16, /* 16 GB/s */ + }; + } + + *cdat_table = g_malloc0(sizeof(*cdat_table) * CXL_USP_CDAT_NUM_ENTRIES); + if (!*cdat_table) { + return -ENOMEM; + } + + /* Header always at start of structure */ + (*cdat_table)[CXL_USP_CDAT_SSLBIS_LAT] = g_steal_pointer(&sslbis_latency); + (*cdat_table)[CXL_USP_CDAT_SSLBIS_BW] = g_steal_pointer(&sslbis_bandwidth); + + return CXL_USP_CDAT_NUM_ENTRIES; +} + +static void free_default_cdat_table(CDATSubHeader **cdat_table, int num, + void *priv) +{ + int i; + + for (i = 0; i < num; i++) { + g_free(cdat_table[i]); + } + g_free(cdat_table); +} + static void cxl_usp_realize(PCIDevice *d, Error **errp) { PCIEPort *p = PCIE_PORT(d); @@ -161,6 +339,14 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp) PCI_BASE_ADDRESS_MEM_TYPE_64, component_bar); + pcie_doe_init(d, &usp->doe_cdat, cxl_cstate->dvsec_offset, doe_cdat_prot, + true, 1); + + cxl_cstate->cdat.build_cdat_table = build_cdat_table; + cxl_cstate->cdat.free_cdat_table = free_default_cdat_table; + cxl_cstate->cdat.private = d; + cxl_doe_cdat_init(cxl_cstate, errp); + return; err_cap: @@ -179,6 +365,11 @@ static void cxl_usp_exitfn(PCIDevice *d) pci_bridge_exitfn(d); } +static Property cxl_upstream_props[] = { + DEFINE_PROP_STRING("cdat", CXLUpstreamPort, cxl_cstate.cdat.filename), + DEFINE_PROP_END_OF_LIST() +}; + static void cxl_upstream_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); @@ -186,6 +377,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data) k->is_bridge = true; k->config_write = cxl_usp_write_config; + k->config_read = cxl_usp_read_config; k->realize = cxl_usp_realize; k->exit = cxl_usp_exitfn; k->vendor_id = 0x19e5; /* Huawei */ @@ -194,6 +386,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, void *data) set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories); dc->desc = "CXL Switch Upstream Port"; dc->reset = cxl_usp_reset; + device_class_set_props(dc, cxl_upstream_props); } static const TypeInfo cxl_usp_info = { diff --git a/include/hw/cxl/cxl_cdat.h b/include/hw/cxl/cxl_cdat.h index 52c232e912..e9eda00142 100644 --- a/include/hw/cxl/cxl_cdat.h +++ b/include/hw/cxl/cxl_cdat.h @@ -131,6 +131,7 @@ typedef struct CDATSslbisHeader { uint64_t entry_base_unit; } QEMU_PACKED CDATSslbisHeader; +#define CDAT_PORT_ID_USP 0x100 /* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ typedef struct CDATSslbe { uint16_t port_x_id;