From patchwork Mon Mar 25 19:02:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602784 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9781F1C02 for ; Mon, 25 Mar 2024 19:04:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393463; cv=none; b=nAmHxKae7GowcCSwP7ZVwKrM5qlMuaq8DxB8mNwOcrBh7BYyZo4VLovlRT7qYacnygKMi3FrK4jFDz9r8axz/0ui4Sn+s7NICSMdam0ZpKzhANBGNxcDzumZlTUyi9JMICa4gYcOoGPkCRNO5Z583CDFIXazfE8GYbgvXA4sJb8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393463; c=relaxed/simple; bh=wy6mbTMBMsYfurZCiVtXv16F+h19N9tY2TYxi6DUmjc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sADvpDD3HKZsCJ9+58Ayn9IKejhcqJSU9Vj6VlTj1kzj9e+Uc727PRKYLaBo3Lvc3AKb+M9BsfgBU9Z/ZCaNOkLqfX7bxPGbvrs2YT7mL7Cj4CBTyBoY0xPt9r8838uh3hohJEYA827ZZsoH00TB1zrA4Gz+vION80FoVS7jKv4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ESJiMrVW; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ESJiMrVW" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1e04ac200a6so32868965ad.1 for ; Mon, 25 Mar 2024 12:04:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393461; x=1711998261; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xakP/WAiR69d8B5B18gJ85sFdqfiQ0c6HHa8yXwoSyo=; b=ESJiMrVWFVbFu1ktzZzbv0BwovYRSmnrivUlTuNPwwg17eyiNS/TT28TWANlaMYkQg o8xlcS6yPM9+akOFtGMqhf8jz73PM0TOlMKIgGxKmisONpNuzqR6Mnvvmj48wmTsdMrY bgqfMPDWlmFhCqsYJbffzVlux8AKfUFXhGBVuDCH3tzvII1GVQo7XJIEC7taS3epNoKD +vDXr2u42sQgr4Evzg0Z3I/N5VCjqrtdcallllVKcYe1kVOcIvr8EmmOR9GkwOlVeinO +xim99sLacix+02jILRlfq9qYaICiEPQ/SgRbJZqZ6EwSBKazlkz3QqZf9+ZeB+wCWIV sRww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393461; x=1711998261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xakP/WAiR69d8B5B18gJ85sFdqfiQ0c6HHa8yXwoSyo=; b=DPcMeI3Y3/Jbg14vJgr2H2nrxvwgTzltutw5VQoXLvvQtOn6wWci+OhSyBqERQlhPv Lufdh5ix83ZhMjGTo7vdj4S+DrAX7RxknbTotTwQ9vIiEzIKNkNk8DI5JTyZ1gTuODP0 mMhVCLxcYSo6hIIjiR/xoRWYO5ZZ8NqEDIoyd4oRbbuNTemhvkKXaFN2wteMvdPv5mll v9MPt6ng4BIy75ZhkxCfaGE03/5v2emDOcAy/mbMfVJKLK7I2FaeHm+L1pJJVQHRAf6V barsMf3fC7Z8APiK7GIzLd9hzUhCE6OcdWgGmvGDRptCe2ej3/4VDykVuhoobk9HUBBa mfmw== X-Forwarded-Encrypted: i=1; AJvYcCV1qCUD7eGoq1lAK+Yek3IRU9NkmkDMiKM7kLMZaf8qu20+gXT+BMOXKbQqEfhOvUzQFaM4ZercmHg0jx31yBvpyNeAXce4PrNe X-Gm-Message-State: AOJu0YyD7a/h0o3KCQr8+qBXcaaZtwZyjcf3ci64yG3GA+l1i9oKfTaN awpzLwBF8vNsO3cUc+adhthoycJ+FSxHvSSFrueYveIWtyCIMgN6 X-Google-Smtp-Source: AGHT+IHIlRLO0pl+CVLunHEPlD8oj7IDD/ym0H18wHpotJXetvT29Z/tMxVqVBK/ih6+S4JVauRffQ== X-Received: by 2002:a17:903:22c4:b0:1e0:c070:b082 with SMTP id y4-20020a17090322c400b001e0c070b082mr5039024plg.14.1711393460732; Mon, 25 Mar 2024 12:04:20 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:20 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 01/12] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Mon, 25 Mar 2024 12:02:19 -0700 Message-ID: <20240325190339.696686-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Based on CXL spec r3.1 Table 8-127 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 4bcd727f4c..ba1d9901df 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -780,8 +781,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -807,6 +809,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Mon Mar 25 19:02:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602785 Received: from mail-il1-f171.google.com (mail-il1-f171.google.com [209.85.166.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9F7A5C89 for ; Mon, 25 Mar 2024 19:04:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393466; cv=none; b=BjgW8h1e3iYkB773kZZnKeEl8DrnWDhzTtstMz29obsDz8yG9uYSsHCDZ1GPTpZI5LiiIiosCGNtL7n0/IYvo0vTVU6a2DdHuNz1GFl7+a7ljRXWHhKjjdLo/hhcOMTuk7vPtwAVCWOPZy9GYrrTzfJDGPEG76/R7QUbKL8QVng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393466; c=relaxed/simple; bh=0dq1W/Glim7fuQ3Y+NS9I1UvHbHTBpf01hrrAwybzj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uCijToXXQTv/KiaZel3+cAtWc7x6zugIRyzMmST6delNBKO3d/NLVjzmtUNDrInANVkAqb7/+NB47FpiY9HvCd8QRbRabem+qs8lC2P/o+SeSrCsUU+SI4fPlrKdnrbwRvvTHg0Th6D6OCy9/THNmMem2HoKnnSh6scb3r/Kjlw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AoeZBGpm; arc=none smtp.client-ip=209.85.166.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AoeZBGpm" Received: by mail-il1-f171.google.com with SMTP id e9e14a558f8ab-366a4bcb2a8so22350755ab.3 for ; Mon, 25 Mar 2024 12:04:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393464; x=1711998264; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bXGq4qKetnED0Sdh0jI6OjQ493ny9rPgw+BgPnzbFPY=; b=AoeZBGpm1yeyYKOP6DfYPgkjPgEQjNNI+4v7rLFRPLsiBpRudTOiZUdvS2ubReYRfe 5MCgTd39xfz6NZbmnECCD2Sh4ePfcCgIKjvyZywWe7nX1EZ7Dr4tzg2Rjfes8sm8slRQ tXsKUzq+hVCpRZ/sSFbakjpEQxd5YKRokBS/1Fmk3ALnv6ykedyo/wtOCjjylDU8oZJa fczbIKAQVk635bddFgGe3n40PN603n/nAdWxTOA2zulqBMGIrCC/mMpS3t3vwhm1MnfN gxr6RV29F0rgNTFCeEhxp4ApFWWKbKlCGu3VU7wi0KpK2TJAwunr7nAJDOqHyDKZ2Hq2 PE7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393464; x=1711998264; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bXGq4qKetnED0Sdh0jI6OjQ493ny9rPgw+BgPnzbFPY=; b=OcHRjhmI2x5sAl3mv3OX0bY1ntl8efBD/dpsP/BcN2Ir41Aba6lOlHnqk/nOl0Xk7G woC4zIhtgKeFJc5cQ7c4Q/9kefp454vrhf/O+/s2nrQnjPzCxSjzbZDjxcw+9Tog2S4m qYWq5Bdt/jxst6r4kol14Lm5g37/xJEy68dQnll/2PzDCefmdcKLnzJ3vbM6lW0JrWeq YeZdYZ+dHtZAj4nh3Xv8NMRebzThHjfYusaCac+N3y48Qz4qqXzs7hI9xcMjvnDwa6Ib 2dRIS9elwUE4oOOQ+YLkNoHVShZ0gls1L6ePLz0+Q742CsHl+/ToSC6X/IyvXbrPJVGE 6Kwg== X-Forwarded-Encrypted: i=1; AJvYcCWJo2e3DjYcV/+5v5PdMehjiRoZS3p+FRnDJsjDz/2x1ak0H7pznzADXYw3LWMwzkan3u9nDRZmGS78s0PGtSuIBdoKxuCxVqI4 X-Gm-Message-State: AOJu0YySoJKYpqN6AA8AHidgiotAMSpHI92SfmEFjijBQ+7Yj2ezEyzO TAB5xu+wmDFcLVRY1AjX81tsmrTZquQfHVN3ss9SzMvnJtg/PnJS X-Google-Smtp-Source: AGHT+IHZATtdkzshYDchLgt2ymdhu4ahwHp5LnKFx1fYpmUrGGBfyB69XYes6Gd/IMLECJQh81QZ3g== X-Received: by 2002:a92:2c02:0:b0:368:9b91:4d1 with SMTP id t2-20020a922c02000000b003689b9104d1mr916992ile.15.1711393463840; Mon, 25 Mar 2024 12:04:23 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:23 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 02/12] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Mon, 25 Mar 2024 12:02:20 -0700 Message-ID: <20240325190339.696686-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per cxl spec r3.1, add dynamic capacity region representative based on Table 8-165 and extend the cxl type3 device definition to include dc region information. Also, based on info in 8.2.9.9.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: we store region decode length as byte-wise length on the device, which should be divided by 256 * MiB before being returned to the host for "Get Dynamic Capacity Configuration" mailbox command per specification. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 96 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 +++++++ 2 files changed, 112 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ba1d9901df..49c7944d93 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -22,6 +22,8 @@ #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 +#define CXL_NUM_EXTENTS_SUPPORTED 512 +#define CXL_NUM_TAGS_SUPPORTED 0 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -80,6 +82,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1238,6 +1242,88 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint8_t region_cnt; + uint8_t start_rid; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint8_t num_regions; + uint8_t regions_returned; + uint8_t rsvd1[6]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED *out = (void *)payload_out; + struct { + uint32_t num_extents_supported; + uint32_t num_extents_available; + uint32_t num_tags_supported; + uint32_t num_tags_available; + } QEMU_PACKED *extra_out; + uint16_t record_count; + uint16_t i; + uint16_t out_pl_len; + uint8_t start_rid; + + start_rid = in->start_rid; + if (start_rid >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_rid, in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + extra_out = (void *)(payload_out + out_pl_len); + out_pl_len += sizeof(*extra_out); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + out->num_regions = ct3d->dc.num_regions; + out->regions_returned = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_rid + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_rid + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_rid + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_rid + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_rid + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_rid + i].flags; + } + /* + * TODO: Assign values once extents and tags are introduced + * to use. + */ + stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); + stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1282,6 +1368,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1487,7 +1578,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index a5f8e25020..e839370266 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -422,6 +422,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_NUM_REGION 8 + +typedef struct CXLDCRegion { + uint64_t base; /* aligned to 256*MiB */ + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -454,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCRegion regions[DCD_MAX_NUM_REGION]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Mon Mar 25 19:02:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602786 Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EC694C83 for ; Mon, 25 Mar 2024 19:04:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393470; cv=none; b=m7DjymuKUWq7lZ7Pa8bhtHWWLPT7G7SHfffh9O5yqw/MH3h1kGCszVQVf330PIJ7KnJIy8kVBPMyRr+hsXEixa9yftQqsD6rcmT0pdpb9tUr8Od6lvU+V8KfEer+5zseBC0Rsvc2S4WKXgQbP+u8JDKsFYlibjKXqYLNJm7Be4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393470; c=relaxed/simple; bh=FF+CLj7/4Ta1TG8dvkj7Od8+/8dXAIYu1QK2sou+H0A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YbVnM1NulqMQ2w6W4emeHYbv88yVw0zdZbSjlFwetENNrsoile1XispbgXZIrrmLDzAlsnMeds3DjV+tfXZd3OJISC5zEzhkDnXsMKLCX9hTQGmHWz7BSju/xXwNKde5XLQa7dpaG0yUX1tlj2fP8Ng9QD4NAqBIqoOgQnSgG4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=beXBXM10; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="beXBXM10" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-3689a0abfd2so2309955ab.2 for ; Mon, 25 Mar 2024 12:04:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393468; x=1711998268; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bsEOxMlU55nR+s6pTI/E0qn/RMJMO6SWzoYHx0Prrb0=; b=beXBXM10g2LZJ9l9CxLErSM7HB0AKkuShIw9i83GGe3bCqGx2gpcl7KVQTUlhNS2Uu UA6vG7gJrtF20aT/uE1TjsiyUqGkQ0N07cyirOqtbu5K0V39T9HpebnEAx1j1HuSEWzd BjYBEH3kT4YV5F2ZeGnBb81Q8a/cSIz1pIg2wiy7xLG6piSfpguxbhCeDKRALSq1CYXh wUNqWB0le5JXi2H7m13arC6wcUzH5gFN2WR/5X2LnpLLu6l1Inx+Xexwn+dF16iymwKB hCoC6a7t/cthgc18OztJ4Jsnh0EjBcn8d1ZMLhdYfcsUzeXlX86vcGkvgs2PUAJR2nBJ zaAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393468; x=1711998268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bsEOxMlU55nR+s6pTI/E0qn/RMJMO6SWzoYHx0Prrb0=; b=naRrNeNUgBWhYXWdwe2j2ntbg5XHY+Wll1dbvzwODGpRed5oy73R2MRGgtiUovhSEW +Hg2S4uVgnA+lt00FN/wp4ZA6riuUGjDfXPuAVL/Ee2g3kUNnrwU7v72BYPuw+1JAmG9 k2SRDFjxIbBnLBsJUV4kb5EC1OcSKqa9mOWHpHcb2o5Uoa9u3aFC628/ow9mdOuCQPKZ RFMhpE5bKTMYOkNsFiCBZffca5iPOncU5XlkDyXeWDD8Viri9TuY/HhBocGkOrKmLnpG ZUATDhI14CMLS70thYkueEeMFgcqQ9Gq78m03fVAgLEqYkdrl+PNzfZMOYvOnQn4IydN sPCw== X-Forwarded-Encrypted: i=1; AJvYcCWxqeXrlt4/bnAfGYAy5xpZVX16y3yfRhIOyB97Wi6hzmNLLpn8anhSlVFt6IscERpBA3ibxUniGTWxvCxVEFcuWx0A2v7PcYPq X-Gm-Message-State: AOJu0Yx26t8tVbqtU0xumwW7LeOBWxtwxfrWjOWTmCDbWsdaI6z3yNDU gyUglVJ5rrhG8ZG4iylaX0SGGIYdnj50T3tk+RRhfn0fJjd++J92 X-Google-Smtp-Source: AGHT+IGYRH/8bP9YC5VIIvc2uxzSieEgYRFC4mMjmWSJpGhRy1N8jHG8LqK2CC2nQfYyQSb+6MMuow== X-Received: by 2002:a92:903:0:b0:366:a93c:66c9 with SMTP id y3-20020a920903000000b00366a93c66c9mr596709ilg.4.1711393468227; Mon, 25 Mar 2024 12:04:28 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:27 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 03/12] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Mon, 25 Mar 2024 12:02:21 -0700 Message-ID: <20240325190339.696686-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 49c7944d93..0f2ad58a14 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -803,7 +803,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1179,7 +1179,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index b679dfae1c..0836e9135b 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -608,7 +608,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -631,7 +631,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -837,7 +837,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1010,7 +1010,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index e839370266..f7f56b44e3 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -234,7 +234,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; From patchwork Mon Mar 25 19:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602787 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33B131C02 for ; Mon, 25 Mar 2024 19:04:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393475; cv=none; b=G9PUgKFKALatAznwbq8bJDLoS93f3fI2hOT5hCwu9c5VRaSuF1uWe0iCaRKyrLbwmCG5336zWJlYBECINqW+x/OdhX2UQ9wLu8CHZoy+vdlwt0Ukpk3XwSQ20wAwFXNBsKI8W7uQMw20WHUDhGpLhucLZdZ6D0vohVZuI5KRaTE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393475; c=relaxed/simple; bh=ewkM/UJvE3/Pa35lnXVt+F2kW50k/hOnYaHAPl0Y2Tk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nK+50VkBKWnndTXpbazyGPHFUMaq/e55TIbmE7mn/JrEFY0OJpRHGZ0dqA3bEN4WYBeM7+0756AZWUTmmMc4i7OJlOLS4oOAIW2BAnQPx82bDdVDMgauTnYdYIPMRlHDmTOtNYFLYqYc0IgiWzGtAD+b9+ZCg/HJzO5IFDYbiX0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lYCvBsQk; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lYCvBsQk" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6e6ca2ac094so3883112b3a.0 for ; Mon, 25 Mar 2024 12:04:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393473; x=1711998273; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/SGocJmgk/sEnzEmc6Kt8P5rytGfU0O9oNlpZCxOq1A=; b=lYCvBsQkFHnJasovtW5wtcownys33xdl3DsaAHJKt4f08kKRXNznB9xgPEls8xpac1 mlcXzyODsW5Smqatc6H1x55aB0se26BPZP3yEYSEuxUTqTA5EnxTwTktxEcjmZevIVXx iVil17auo1mPLi0abOKP2qR5LzH8cSwwYsxkQsuWpCnSU1wKK7H/lfHeGHRkyKmfYRMO /FI1snw5iQu2BKQtnEn4y4lWmVQmXh4uM5SnIvM1fVoGfM7dvfzM8ktLS/hh4W7+bMjW h5tZJaIQYVYMvDKYKu0n0rsH2wwMCETYZrlE7v3zxIz5MO+HJo+lQNSQvd+VQkyJ0gLs F8uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393473; x=1711998273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/SGocJmgk/sEnzEmc6Kt8P5rytGfU0O9oNlpZCxOq1A=; b=BotutRBW8ZEdsVT+QovQrfVj95MIPkPVMH7BBTsUU3fhhdbA0t5q4wx9quP50IWwKT PWe8SQtQfMcHvxEaC35Cua1guzBRfnpkqJpay+J+67aGr1815CevqkumVFYju57KutC3 AFB9zxAzCwbP0W+ic1sa559dzHdq3nf+N19yd4wH4Hj8a1ovcprkPH8aHj0wQ6GLj4ol 9OE0T1uc/j823Wd8ch8FJ593GmMDt7sJ8u+AAJzCSPuvumj2M4DRR9Hkv/giOGEMpyCf NS/3qg5gv/oUgqLTnD/B3OK1auBO35vlnqA1toIF8QFjF9PxRM5FMrm57ZQsP63N1fjt m09g== X-Forwarded-Encrypted: i=1; AJvYcCXr5QM9yd/dducWjrfHnRMVbbl/oahpMMyS8tR2DTvnjiWSWKIPbll+tvauXNO7EhYSM5VgAr4Ev68hgCkYPV+qNP0JONJwmoDk X-Gm-Message-State: AOJu0YwRICs60VqKUJ9mLoQJSWakFThihfTwg7vg74/CGb3ADSiT5fy7 HfPZzfOcKtc5Ky/JsgHfiW15VwQhN4GXs6gYkHUTDX/d+BDgWCSZ X-Google-Smtp-Source: AGHT+IEet4CYlBzXytxndehASjVoSA2KP6NMROyP4aaNMcrhbw2pxprA5/Xl3Yf4Gtwld70yD4SjtA== X-Received: by 2002:a05:6a00:80d2:b0:6ea:ad01:3550 with SMTP id ei18-20020a056a0080d200b006eaad013550mr2813950pfb.18.1711393473264; Mon, 25 Mar 2024 12:04:33 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:32 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 04/12] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Mon, 25 Mar 2024 12:02:22 -0700 Message-ID: <20240325190339.696686-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support like below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 0836e9135b..c83d6f8004 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -30,6 +30,7 @@ #include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -567,6 +568,46 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: dc region configuration will be updated once host backend and address + * space support is added for DCD. + */ +static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 2 * GiB; + uint64_t blk_size = 2 * MiB; + CXLDCRegion *region; + MemoryRegion *mr; + + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + region_base += memory_region_size(mr); + } + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + region_base += memory_region_size(mr); + } + assert(region_base % CXL_CAPACITY_MULTIPLIER == 0); + + for (i = 0, region = &ct3d->dc.regions[0]; + i < ct3d->dc.num_regions; + i++, region++, region_base += region_len) { + *region = (CXLDCRegion) { + .base = region_base, + .decode_len = decode_len, + .len = region_len, + .block_size = blk_size, + /* dsmad_handle set when creating CDAT table entries */ + .flags = 0, + }; + } + + return true; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -635,6 +676,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } + return true; } @@ -930,6 +976,7 @@ static Property ct3_props[] = { HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Mon Mar 25 19:02:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602788 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F37DDF55 for ; Mon, 25 Mar 2024 19:04:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393479; cv=none; b=F66TGHYC03USpORD+oObh+Ho5wxxL3zf2oHE0Imdz90E+4QzfmcQovg/HydpdbG2F5auKw/AQLf8mlxXsapuEfExwFLIKdcb1D7uwj29UdEPohBoWSxYDu+v99kJfZFsNiVn4m/uCVkxUp/ML4gt6PMQpeaHAY5WV7LGN6PJI7Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393479; c=relaxed/simple; bh=mlfiT0e5J1VKDXSWHQx/SOiMufVP7Vqtegrajei8his=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YQ4P8bfWHzRARADHa5yyPuu4slpaXuhrXf6BJejo7CCnqVuqP7Zh5cT+c2lTw1dz1daWhXwnpsQyyDvHf30CQ8D1HeioZLjtg/wjl+L6h7twpSkJsgybnKwXRS47JtqLVx5Oi4tSTLkb/Z54MrvO+QIUsb17hL4Dm2Xz9tHuzVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RJq/1ZHk; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RJq/1ZHk" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e740fff1d8so3279873b3a.1 for ; Mon, 25 Mar 2024 12:04:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393477; x=1711998277; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eKnc16xzirIXrAZVe71LIxoKevlODpLkYCNSN1H5wx0=; b=RJq/1ZHkRM5ytUafJ3OpkMWHrJSBSyoc6g41myhie+hzSTSITVyop/gW52pEXRZmuB nOEHBcZ9BSWM6iD4PrifMvR36UFURIPUDpLbIiPNfKirbvege8UPdCD2J6e3ePZGQJ6s mmDOjS4/Abi5xKTLOorFes49U80Sps4bbbQlSKfRgqA3P0jzpcsATPVu+RH4xmkZvsvg kHPRve0+NzbYb1tqn4FurVcPXYm4r95MWySppsff5PDDJ2LslOpwuxV47sofIWSwi2vg +rDwARLK1hOJHG67yMedZYh1KQXgIZXWe9AUIS6sO8TSJg30C9M73FQ7gaj54UCcnr/r e8kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393477; x=1711998277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eKnc16xzirIXrAZVe71LIxoKevlODpLkYCNSN1H5wx0=; b=GQK4Yw2cJs/4j3LuEwB6VWXMLpar8eO/XU5dlW95kXCYloFqcglV3sBxLbOI3UcIAY d8/qjDsjgYtH4VsA+eJZ9Z1NZyEJ5WHrITuxZt5Cl8wZHCN0+3qIGuQfy9FSNAs30Buk X1TA/9s+jduIqn9OWU5DlXO+gUWkpiv+EOtUqJ3jvtcPsT54i/+CtLQIHbYQ/u5feeWl uWA82CkyCr49T6vakR3GJ20wCZIu2xXLuYxhIZs97bz/8lASLSQ9XXFSqhoIIrs+9wrT wb4y6oSxTFUvlTyXII3P3o8icJSWRnth5CMy6tBY7JUW7B7oi4Ua9ItM4cef5YZuYNA5 6mAQ== X-Forwarded-Encrypted: i=1; AJvYcCVaGZFr6S80299zNMR8hIZ0kBGdQg7ompBDvW2a2yBrh3ZTt4cGoL1V18sY40WzkLPKlB+sV1CrJ8lOox6UgI84F+n7QGvGBZYW X-Gm-Message-State: AOJu0YzonBhLOpf0ypAm77MnaBasXgZSc86Ze3i/Ugd/gbyLB/3Cx6ev ZCJsDqbG1q8W+IE8CUsvjeLb+/hT/CiiHHsVDrmhmUU9NV/7VbIL X-Google-Smtp-Source: AGHT+IGUNI8lrtdXnyQikcaBV7W7iMTI9ww/WxghqB3THSxV9ohDmqqaQaFdUnjSqcZVFPjmyDpBqQ== X-Received: by 2002:a05:6a20:158c:b0:1a3:68ff:5805 with SMTP id h12-20020a056a20158c00b001a368ff5805mr7672803pzj.44.1711393477474; Mon, 25 Mar 2024 12:04:37 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:37 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 05/12] hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr size instead of mr as argument Date: Mon, 25 Mar 2024 12:02:23 -0700 Message-ID: <20240325190339.696686-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni The function ct3_build_cdat_entries_for_mr only uses size of the passed memory region argument, refactor the function definition to make the passed arguments more specific. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c83d6f8004..a9e8bdc436 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -44,7 +44,7 @@ enum { }; static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, + int dsmad_handle, uint64_t size, bool is_pmem, uint64_t dpa_base) { CDATDsmas *dsmas; @@ -63,7 +63,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -132,7 +132,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -149,6 +149,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; @@ -163,6 +164,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -171,21 +173,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); } table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { - ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; + uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } assert(len == cur_ent); From patchwork Mon Mar 25 19:02:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602789 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA2D25C89 for ; Mon, 25 Mar 2024 19:04:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393484; cv=none; b=ChHmBZG9HrbL6QCEpi3F/XjhorZHKmdHCl1Mbo2S/VJjM6uLzs2QQblo9m4Xq1lS0NHHKHXQ4SAY+7o4INSusaZ2daYAv+En85C12Gz7Hl97YTlUvML75sxdBc9UheotmVpp8yjO10f5NsDl9Y/QEToyMOFS5tPzS8YfSEY5HVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393484; c=relaxed/simple; bh=q7/EpNk62kQJVptYmbLzwqp5L6mi027uNzO/izSWjDQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KetXcPdY5Gr7ie/TZ111wMY8tvi+sXpmLcnRoFo0kvkcYow3rDCeQv/nwL+IXF3ciJ2y6PhsQzhgVDanY7kwQ/t9MCTW5ffHugulnTrnz4q8O48TaAu0iPmENCMInM4l6C+IHHA079LA3CNCppmAKhAzggzbfBkceLL4RCrFTrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hdcJWP6K; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hdcJWP6K" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e73e8bdea2so3841517b3a.0 for ; Mon, 25 Mar 2024 12:04:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393481; x=1711998281; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=layG55zMLafcTRE8xI6SUYzjqXvFAco9vucbohotDjM=; b=hdcJWP6K/WsBsKyarDVffqrU9iU15nURRBemH01RjKOstA2j4lQEQYQUzLahF0v/bQ fI1hc2Mzwgs6WyYNDafJ9TlIuK1SGk8J/+65GrYAY7XeBD3xC1su7g8k9BuNRHMvIbsx a3rmjBbmVuME3dX9DPYgWHKKy7jJunMiDzATedJgXf/En/5nDhWRbvYad8JIuTu0vF5M XwMUVouxbFYZDMgnEv3goJy87qUB52XpCJy0BkfUYO+dUIQUCSGQhebvwf0bVPifC970 qkDqrJpC1qc2XbEFuMPfnxvpxHgvhs/PeJLw0t4RYUoFcLf4YMWqjKRGkI5mdsvm7WOF 7vKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393481; x=1711998281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=layG55zMLafcTRE8xI6SUYzjqXvFAco9vucbohotDjM=; b=WNn6wGcWCSeRzP7LjWG99U6w6+dmY/w4FlH9iikh3QT6n4NHzXhWLuNq2wno6XcvZH T6jKnMR59h4U3Sac/bTafbwEoXmAiqpAM4KIIPwl4K+OCbB9Vlr+m2DUnm0W5iGu1TSe 8SP76eejbv/KaYRBZ+ZdvZuPleqrPwLmH1001Z2FuHD0HWckHPDs2KmPKwAgnhBng5dy nwL9OfW5FDm8W92YwyEtyS7O/PSJI3etzc920rNSgu76tljKl/v38kuxLyT+kfJFQ5X2 sBVxj62HQ3v4gp+tqIp4tiaIWbkBZkJIDrXFQqMA/ZxTK5AV+eFiRIXWZNDayE47r1uw +5Ug== X-Forwarded-Encrypted: i=1; AJvYcCWkK4lpqQkEfFeJnzMz2RCuUXKUufr8+o+JTwPVwAatelTDX3c/fu7qL3xMGfxu9f60kUTYkrcJc2EEVbFKNwPIc09NoUc083bz X-Gm-Message-State: AOJu0YwvT+vLTbkEUDbIY1JfZU43iGPQ7q7LsWtbvvvMEmvChv1/cySZ IIrf+Jih1z2077drCVsGIlHzLeJ7Sxa8yj8R6RsHT8FKrII2b1or X-Google-Smtp-Source: AGHT+IHqI/jkK13CGXjf0792if5gMi5R2PUQeOfpcRtt/QLX60yi7AFNl7zJT1bkkf2Z+edMOihMNA== X-Received: by 2002:a05:6a20:c503:b0:1a3:72b5:528 with SMTP id gm3-20020a056a20c50300b001a372b50528mr6913095pzb.36.1711393481007; Mon, 25 Mar 2024 12:04:41 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:40 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 06/12] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Mon, 25 Mar 2024 12:02:24 -0700 Message-ID: <20240325190339.696686-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add (file/memory backed) host backend, all the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, following supports are added: 1. Add a new property to type3 device "volatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all dc regions share one host backend. 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region; Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 16 ++- hw/mem/cxl_type3.c | 187 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 8 ++ 3 files changed, 172 insertions(+), 39 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 0f2ad58a14..831cef0567 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -622,7 +622,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -636,7 +637,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -793,7 +795,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -835,9 +838,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1179,7 +1184,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index a9e8bdc436..75ea9b20e1 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -45,7 +45,8 @@ enum { static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, int dsmad_handle, uint64_t size, - bool is_pmem, uint64_t dpa_base) + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { CDATDsmas *dsmas; CDATDslbis *dslbis0; @@ -61,7 +62,8 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, .DPA_length = size, }; @@ -149,12 +151,13 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } @@ -176,21 +179,54 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) pmr_size = memory_region_size(nonvolatile_mr); } + if (ct3d->dc.num_regions) { + if (!ct3d->dc.host_dc) { + return -EINVAL; + } + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } + table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, - false, 0); + false, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - pmr_size, true, base); + pmr_size, true, false, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + int i; + uint64_t region_base = vmr_size + pmr_size; + + /* + * TODO: we assume the dynamic capacity to be volatile for now, + * non-volatile dynamic capacity will be added if needed in the + * future. + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + false, true, region_base); + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -300,11 +336,24 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_hi = ct3d->hostpmem->size >> 32; range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); + } + } else { + range1_size_hi = ct3d->dc.host_dc->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -579,11 +628,27 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) { int i; uint64_t region_base = 0; - uint64_t region_len = 2 * GiB; - uint64_t decode_len = 2 * GiB; + uint64_t region_len; + uint64_t decode_len; uint64_t blk_size = 2 * MiB; CXLDCRegion *region; MemoryRegion *mr; + uint64_t dc_size; + + mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(mr); + region_len = DIV_ROUND_UP(dc_size, ct3d->dc.num_regions); + + if (dc_size % (ct3d->dc.num_regions * CXL_CAPACITY_MULTIPLIER) != 0) { + error_setg(errp, "host backend size must be multiples of region len"); + return false; + } + if (region_len % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + decode_len = region_len; if (ct3d->hostvmem) { mr = host_memory_backend_get_memory(ct3d->hostvmem); @@ -606,6 +671,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) /* dsmad_handle set when creating CDAT table entries */ .flags = 0, }; + ct3d->dc.total_capacity += region->len; } return true; @@ -615,7 +681,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -679,9 +746,41 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } - if (!cxl_create_dc_regions(ct3d, errp)) { - error_setg(errp, "setup DC regions failed"); - return false; + ct3d->dc.total_capacity = 0; + if (ct3d->dc.num_regions) { + MemoryRegion *dc_mr; + char *dc_name; + + if (!ct3d->dc.host_dc) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + /* + * TODO: set dc as volatile for now, non-volatile support can be added + * in the future if needed. + */ + memory_region_set_nonvolatile(dc_mr, false); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } } return true; @@ -773,6 +872,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -791,6 +893,9 @@ static void ct3_exit(PCIDevice *pci_dev) pcie_aer_exit(pci_dev); cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -869,16 +974,23 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -886,19 +998,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -980,6 +1091,8 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1046,33 +1159,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index f7f56b44e3..c2c3df0d2a 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,14 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + /* + * total_capacity is equivalent to the dynamic capability + * memory region size. + */ + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; } dc; From patchwork Mon Mar 25 19:02:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602790 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3A1D5C89 for ; Mon, 25 Mar 2024 19:04:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393487; cv=none; b=LH/LokoY+6J6BvCwkl+U9cd/M+/+JPFQbhEhDOXgHmafJbzin2WIrE//JVuDXpnyDh0ubmOd0pKVudxiE2UlXT69TuF65+aL3l1ncsXk82Y4q9fr5Vp3kHI6u+O425iyiz3JZBDr5SIsi2YYdpXW63zb73FJMMWlREE5UjLqSjI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393487; c=relaxed/simple; bh=pwkmVjkEGHJOq8RLCJbrwJF+WiXXyw8DtEB/ws6pFiA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cQHjsbq6RLTY6Z143EaUOkBKHN/tYFbjyZvgwcao0rZD6yLW0sfZVPl1n2jLV9cTjv9h8lINZz5mmbmsP4zO5nppsb+stI5xEK2e0bbU9HGHhZtIL2AySHXPYYh96EnHqqoa9di8pbqTo+l4aAp21lJ1nn3dAuoFewt7Xu5g73k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HiUyHp/5; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HiUyHp/5" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5a53d5aaaa4so945694eaf.2 for ; Mon, 25 Mar 2024 12:04:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393485; x=1711998285; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZE+wSwqEFJL+GKQSjH/3heWejc1q4cGgoFbgOcUQ3o0=; b=HiUyHp/5qD4HcIs5QaoeeNWyJ0PRcOHzVaauOPG3aRxn7xRIuweaOhdjUfnSrY1G8C BGnG9fnLKTF0OoW9zuyHFpyaKoSHqzn+sUWC5gURMmm51ldh/XF2zfvU3+06l9usV8sV yukjOQTwplbaASBw9cIrkpesFj/N5YTVMI0uuB9YzC/RM1ZDr9KCnkpQTN5Hy6dZhwGs /RrtJASGdirwE8Xaab8D59tIrI/JJLXRepmgMqEjaVcBK0tx0+Ve/bH1xXEo/TNoBPkp juAKISrFy/Ry80LWk+rpqMOVDcSKdvbdEleOa2mvSdTY1sRSaxkAKUznDzpVxQhvuCYd EDZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393485; x=1711998285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZE+wSwqEFJL+GKQSjH/3heWejc1q4cGgoFbgOcUQ3o0=; b=wzHe3GUNy/8EnwsKOuXXXnILFEUL71qHS7/tOYFK7qFLvE6v4t2j5EXlliZBfp2ogM AZ3p98EFLoqBRWkmEmg6b8nqebbjjMlG09axVppOH+pCxGG1w6+Cw5vgk7Z8661eaLDZ XRJzOWZN+e3opiUT/Z211ET6LUf5wsFbHtj7ROrJiuvm/1/5N9jsWbzi8XV8q9C3+5r5 kWfPj0Gh2oui5FDTjXwQzpWQX5Joq9MqIG4FQ4wp/4FULOL3lsc9rAy3cAgjsq6NmbhX ymI4pgNDEn07QOqv0Xz3gJtJOwiluY53Xj4MpJl4b6BKhbjIF8ZICwAP5Oww2YwfbFD7 9KPg== X-Forwarded-Encrypted: i=1; AJvYcCUzcnfNllgCeR5jppwC8q0VmgtvG9chX8XOnfFIt6Un2GkUXS60i/C/suxTTRzM5/fzAq4m0WfpzqxvEO+1trtg9cWBjRXQcRpA X-Gm-Message-State: AOJu0YzQoEMsIXGP6QhHAsk76ahk17YzzJdr5LE3Whn2lHAQHMnCvT1Y 9zP9S6eqAfgiyjol1VpiZ+ydej5X9o46WWdi2SQtJYi+iuKdk6Qy X-Google-Smtp-Source: AGHT+IGh8nmchiuNN7ukVHIskLAnXpa5rDx0Y8mCAvpRUq1OfIHgTSakFVcZ+aTS1/yVwOJjvsfGkg== X-Received: by 2002:a05:6358:7695:b0:17c:1c76:84b1 with SMTP id e21-20020a056358769500b0017c1c7684b1mr10164285rwg.14.1711393484615; Mon, 25 Mar 2024 12:04:44 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:44 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 07/12] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Mon, 25 Mar 2024 12:02:25 -0700 Message-ID: <20240325190339.696686-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and implement get DC extent list mailbox command per CXL.spec.3.1:.8.2.9.9.9.2. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 75 ++++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 22 +++++++++++ 3 files changed, 97 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 831cef0567..30ef46a036 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1322,7 +1323,8 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, * to use. */ stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); - stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED - + ct3d->dc.total_extent_count); stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); @@ -1330,6 +1332,74 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.2: + * Get Dynamic Capacity Extent List (Opcode 4801h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED *out = (void *)payload_out; + uint32_t start_extent_id = in->start_extent_id; + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint16_t record_count = 0, i = 0, record_done = 0; + uint16_t out_pl_len, size; + CXLDCExtent *ent; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + size = CXL_MAILBOX_MAX_PAYLOAD_SIZE - sizeof(*out); + if (size / sizeof(out->records[0]) < record_count) { + record_count = size / sizeof(out->records[0]); + } + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + CXLDCExtentRaw *out_rec = &out->records[record_done]; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out_rec->start_dpa, ent->start_dpa); + stq_le_p(&out_rec->len, ent->len); + memcpy(&out_rec->tag, ent->tag, 0x10); + stw_le_p(&out_rec->shared_seq, ent->shared_seq); + + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1377,6 +1447,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 75ea9b20e1..5be3c904ba 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -673,6 +673,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) }; ct3d->dc.total_capacity += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return true; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c2c3df0d2a..6aec6ac983 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -424,6 +424,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_NUM_REGION 8 +typedef struct CXLDCExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCExtent) node; +} CXLDCExtent; +typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -474,6 +493,9 @@ struct CXLType3Dev { * memory region size. */ uint64_t total_capacity; /* 256M aligned */ + CXLDCExtentList extents; + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; From patchwork Mon Mar 25 19:02:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602791 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75EC45C96 for ; Mon, 25 Mar 2024 19:04:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393491; cv=none; b=GDNA8/i6YrWzUoiaazSbHbgmjr5/maUGr5gKTxwZi8vtul3kFhPUcmb6CpGwmgObJyGcwM6ASnOQH5/z8hIVeGCowPgcPJkjg88jcrT6RQqlCFqpF7pFJSZitxM2VU1PhztLigZg/L0vqJqXk4DLfO+YIBLSoh1L95Z9J2/gBOQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393491; c=relaxed/simple; bh=fe2WNdYbCGtRzqOjSkX6uORotWqrDdiwbcivX0a7Eb0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LRgGUeQfmYa3knKNWpMqAK4Eau7s2eihfWJgW1yBq7cHx0clYwP48gt3K1S+UANGhpGDaphlXnQdu+tuofHaSE5v/Fer7eFiS4Y2k+Ho6DUV/DMyoLENJKMU8673I82vGcO/AGBGopMndjwhgBv4iZooD+BiU8j5Y+An64/ZHtU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BojIzATy; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BojIzATy" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e6fb9a494aso3652118b3a.0 for ; Mon, 25 Mar 2024 12:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393489; x=1711998289; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KocXMXKZ/vSLPLCvDqw4K7nx5Zxbo+S+gqYrm3C3DUA=; b=BojIzATyOoeXpNWR0dhFE4XaxEh3s1IvpO60pDZJ2hDFCNtJpNgXOcowfNxP0vQ0ev iI7k6JUcM5qJLrhHJyzhFVsotR6Myap/kEYT+TdVmG0jSzqbF/tYzxEX48GUwoGnEEDq UOGer8jRsEQB1L4Ewawz7zDswmDvLGV0KgUN5UXyvOtHlmwHHvthRdpw/gwpAY99ELBs tukKHXCPF5p01+cVod4LTzjw0Fhq/APgpraYmAudFBw7y/wfkvhiRJjzoZQvQMnWiHTA eIGXS9s+GYqRko2wGOHGwImVygZjZmXU70e5zNGHGRjp4WojN5vzEZD7pEzZNox0zbp4 0V5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393489; x=1711998289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KocXMXKZ/vSLPLCvDqw4K7nx5Zxbo+S+gqYrm3C3DUA=; b=u0qNgHa5ORwIdKo41EiEjM5oGPPvhvTI8nBqcNkvfEDG75D3TBVt0ys5+tscVm/QHC JrzgaprlybrVpBikLD/y4eKpdKbNH+IbCIUNSgJYbhxBBO5v+UZFKyeS8+6erVoN/i14 rUCEQ74En++x4Tk7JR8dGgyV1Z3aZoKSy4d0PoPkzxxH00RvmxLG73DjJvkk8LikpVjE zOm/UIo776NAOeEbOUhsuDYB+Nq+al3+wz3a8sEORi8oHdB3duOqCEPuyFVu3zHmCk/e Pa6DBUAS3Br6cviqbrOTKs9D2MP+pppFJQ7sgoQOqQOhKiTtUDXDIvHft082cA45rOv0 JOLQ== X-Forwarded-Encrypted: i=1; AJvYcCWzqfWCDO3G4Vh1BVXsOcqgFnn3xBKPHfu6oAxjVAXlNYYVALd5AL8KDWw+QfFKMbkGrvbU7jyZSriOxRtDY0YCYoxSMmsZOBGz X-Gm-Message-State: AOJu0YyD6r1sYWXVkvmfg11Slcv4DClu2rfbk2ZMk2W5MSaTX/QCrAmS Tkb0Kax8trKeiCC0q+0ZOk/DPcWX6Cn3kGN4EmjbwN9mrNGzBnwn X-Google-Smtp-Source: AGHT+IHDIQp6JcLjApFBWqvNCDBiHPlSMj0CZaop9CJZf8Y06EBxx1V7DBDpnQxbpcbnNbbgU8V/5Q== X-Received: by 2002:a05:6a21:3a84:b0:1a3:bb3d:aa5d with SMTP id zv4-20020a056a213a8400b001a3bb3daa5dmr6261502pzb.59.1711393488729; Mon, 25 Mar 2024 12:04:48 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:48 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 08/12] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Mon, 25 Mar 2024 12:02:26 -0700 Message-ID: <20240325190339.696686-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per CXL spec 3.1, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.9.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.9.9.4. For the process of the above two commands, we use two-pass approach. Pass 1: Check whether the input payload is valid or not; if not, skip Pass 2 and return mailbox process error. Pass 2: Do the real work--add or release extents, respectively. Signed-off-by: Fan Ni Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 433 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 11 + include/hw/cxl/cxl_device.h | 4 + 3 files changed, 444 insertions(+), 4 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 30ef46a036..a9eca516c8 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "qemu/uuid.h" #include "sysemu/hostmem.h" +#include "qemu/range.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 @@ -85,6 +86,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1400,6 +1403,422 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + int i; + CXLDCRegion *region = &ct3d->dc.regions[0]; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.1 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + if (dpa + len > region->base + region->len) { + return NULL; + } + return region; + } + } + + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCExtent *extent; + + extent = g_new0(CXLDCExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent) +{ + QTAILQ_REMOVE(list, extent, node); + g_free(extent); +} + +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCRegion *region = &ct3d->dc.regions[0]; + CXLDCRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->base + lastregion->len - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint32_t i; + CXLDCExtent *ent; + uint64_t dpa, len; + Range range1, range2; + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + range_init_nofail(&range1, dpa, len); + + /* + * TODO: once the pending extent list is added, check against + * the list will be added here. + */ + + /* to-be-added range should not overlap with range already accepted */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return CXL_MBOX_INVALID_PA; + } + } + } + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.3: Add Dynamic Capacity Response (Opcode 4802h) + * An extent is added to the extent list and becomes usable only after the + * response is processed successfully + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_SUCCESS; + } + + /* Adding extents causes exceeding device's extent tracking ability. */ + if (in->num_entries_updated + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + return CXL_MBOX_RESOURCES_EXHAUSTED; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dcd_add_dyn_cap_rsp_dry_run(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + /* + * TODO: we will add a pending extent list based on event log record + * and process the list according here. + */ + } + + return CXL_MBOX_SUCCESS; +} + +/* + * Copy extent list from src to dst + * Return value: number of extents copied + */ +static uint32_t copy_extent_list(CXLDCExtentList *dst, + const CXLDCExtentList *src) +{ + uint32_t cnt = 0; + CXLDCExtent *ent; + + if (!dst || !src) { + return 0; + } + + QTAILQ_FOREACH(ent, src, node) { + cxl_insert_extent_to_extent_list(dst, ent->start_dpa, ent->len, + ent->tag, ent->shared_seq); + cnt++; + } + return cnt; +} + +static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + CXLDCExtent *ent, *ent_next; + uint64_t dpa, len; + uint32_t i; + int cnt_delta = 0; + CXLDCExtentList tmp_list; + CXLRetCode ret = CXL_MBOX_SUCCESS; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + QTAILQ_INIT(&tmp_list); + copy_extent_list(&tmp_list, &ct3d->dc.extents); + + for (i = 0; i < in->num_entries_updated; i++) { + Range range; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + while (len > 0) { + QTAILQ_FOREACH(ent, &tmp_list, node) { + range_init_nofail(&range, ent->start_dpa, ent->len); + + if (range_contains(&range, dpa)) { + uint64_t len1, len2, len_done = 0; + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + /* + * Found the exact extent or the subset of an existing + * extent. + */ + if (range_contains(&range, dpa + len - 1)) { + len1 = dpa - ent->start_dpa; + len2 = ent_start_dpa + ent_len - dpa - len; + len_done = ent_len - len1 - len2; + + cxl_remove_extent_from_extent_list(&tmp_list, ent); + cnt_delta--; + + if (len1) { + cxl_insert_extent_to_extent_list(&tmp_list, + ent_start_dpa, + len1, NULL, 0); + cnt_delta++; + } + if (len2) { + cxl_insert_extent_to_extent_list(&tmp_list, + dpa + len, + len2, NULL, 0); + cnt_delta++; + } + + if (cnt_delta + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + ret = CXL_MBOX_RESOURCES_EXHAUSTED; + goto free_and_exit; + } + } else { + /* + * TODO: we reject the attempt to remove an extent + * that overlaps with multiple extents in the device + * for now, we will allow it once superset release + * support is added. + */ + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + + len -= len_done; + /* len == 0 here until superset release is added */ + break; + } + } + if (len) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + } + } +free_and_exit: + QTAILQ_FOREACH_SAFE(ent, &tmp_list, node, ent_next) { + cxl_remove_extent_from_extent_list(&tmp_list, ent); + } + + return ret; +} + +/* + * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (Opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dc_extent_release_dry_run(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + /* From this point, all the extents to release are valid */ + for (i = 0; i < in->num_entries_updated; i++) { + Range range; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + while (len > 0) { + QTAILQ_FOREACH(ent, extent_list, node) { + range_init_nofail(&range, ent->start_dpa, ent->len); + + /* Found the extent overlapping with */ + if (range_contains(&range, dpa)) { + uint64_t len1, len2 = 0, len_done = 0; + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + + len1 = dpa - ent_start_dpa; + if (range_contains(&range, dpa + len - 1)) { + len2 = ent_start_dpa + ent_len - dpa - len; + } + len_done = ent_len - len1 - len2; + + cxl_remove_extent_from_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent_start_dpa, + len1, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + if (len2) { + cxl_insert_extent_to_extent_list(extent_list, + dpa + len, + len2, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + + len -= len_done; + /* + * len will always be 0 until superset release is add. + * TODO: superset release will be added. + */ + break; + } + } + } + } + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1413,15 +1832,15 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { [EVENTS][CLEAR_RECORDS] = { "EVENTS_CLEAR_RECORDS", cmd_events_clear_records, ~0, IMMEDIATE_LOG_CHANGE }, [EVENTS][GET_INTERRUPT_POLICY] = { "EVENTS_GET_INTERRUPT_POLICY", - cmd_events_get_interrupt_policy, 0, 0 }, + cmd_events_get_interrupt_policy, 0, 0 }, [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY", - cmd_events_set_interrupt_policy, - ~0, IMMEDIATE_CONFIG_CHANGE }, + cmd_events_set_interrupt_policy, + ~0, IMMEDIATE_CONFIG_CHANGE }, [FIRMWARE_UPDATE][GET_INFO] = { "FIRMWARE_UPDATE_GET_INFO", cmd_firmware_update_get_info, 0, 0 }, [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 }, [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, - 8, IMMEDIATE_POLICY_CHANGE }, + 8, IMMEDIATE_POLICY_CHANGE }, [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 }, [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 }, @@ -1450,6 +1869,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "DCD_ADD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "DCD_RELEASE_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 5be3c904ba..951bd79a82 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -678,6 +678,15 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) return true; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + CXLDCExtent *ent, *ent_next; + + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -874,6 +883,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -895,6 +905,7 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 6aec6ac983..df3511e91b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -551,4 +551,8 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent); #endif From patchwork Mon Mar 25 19:02:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602792 Received: from mail-oa1-f42.google.com (mail-oa1-f42.google.com [209.85.160.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98A3963A5 for ; Mon, 25 Mar 2024 19:04:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393496; cv=none; b=K1nLXg2U4P80yb/aknDXNFCRq0JPUuMGdDqV/5KWiIWyA2588vOMo2T02z+LvkV3H3f/XdbrlcZnVU6vGQnE/Vf0pFam4TLiXgZNN9NVwRPcHzxNCF1eZbXol0ygDPcwDdlzyqTfwdmcqQC0A0ZUfiyiadvdKoI1e9y9D7AxBgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393496; c=relaxed/simple; bh=47qI8XYnIAmvyXW628NTI+jlWRjPCLwQoZt/fkPKJ44=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z4dAuzN3mdoettZrUNoVNO0p0+1hmQH+FTT3z8zUO54KHzdiwy67MTby6IBGIywsRRyXfb37Vk5Zr/q+tD+9gQvzTZ8DWza1tyWw2c8DXD0rb0pG9RbUsbXpdH3UtL83LpusTtkFgh1b37fysYcY78GuSd1Qfa+jA85IkC6HpOA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VgZ82aWn; arc=none smtp.client-ip=209.85.160.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VgZ82aWn" Received: by mail-oa1-f42.google.com with SMTP id 586e51a60fabf-22a353217c3so736816fac.1 for ; Mon, 25 Mar 2024 12:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393493; x=1711998293; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k3vP3x9GTrRSsmfUjcE/YCCno0V4KVPXGg0cR6uT4I8=; b=VgZ82aWnOJE28MVeCZC4oK7EZkggJlMkZf73URtbH/CTZC0RKofNwiziEzCC40NO+i XBdyr0HheA9sRJkEoFznY/++ktcJ1NGOyyt341W8fLPh7MdFyIs09Q5RDh2vdnvyH7z8 b7Y+cxpTDAQ43QrjTnA6uNyv5tSjqrEeooYEnwmN9HnOXcJ0rhT+3kGp/JkOCY0ZLox7 9ATyTuRQYY/tgCf+jF2mwgu3vF/Hr4S8I3YDwgONJoxskaHRHtw9RdeDtiIRJ7aL5tFm QhtQ4PO/c7K6ZA1Fli0PojV5y19dJd5bec51LFb4sClkA9rTZ+QuC+e7nRsm7sCyiwcM npUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393493; x=1711998293; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k3vP3x9GTrRSsmfUjcE/YCCno0V4KVPXGg0cR6uT4I8=; b=wRXI0Qw5Ei86AlQJp5pIWFybCEjAAUNs6JR7nCZIdOJIkUmT05lLgAjAMtMZ+rc2ya MA+dn8Za8zMGjHYx+wfIt1zowDzJIYdulQshacHzPUEcOnnRQ0RF2RCJ7bxTlXEs0Lu+ oWKHsUPOGJzOqpgDZEnfn6VvX4OA63L6Uzvb89p+Dc4bYqXUolg6ALsgvfZK6Bd2Hc71 TeHSJnHOGyn2DYTQnGFe0pe6uazdsScJIsa2hGLmKaQbSzBBK1+Pclf+ZWhiGNvT0Jux RlsrVrNoBkhDAucQZ8X8/SlZpZQwgGaGKUjp6v7ZkNSIrcB20f3c/z74Ik9RvQWJ1lcP 3Kew== X-Forwarded-Encrypted: i=1; AJvYcCXx9TWlD9j0SnPgsrxiTSq/SrzT0ZL9y0YH7gYk8s6AyjbZXjNkwdYipiW4QAvEwcbWCkKgOUoOlHInO6M7Wo5vGWprtAdx9pwh X-Gm-Message-State: AOJu0Yynond8NVxAQP23ec99otLEwS1EnJ/2W+2nEBmm4ifuQn5y6Ia2 iG51ngPqUAARsN4WXQXRcUgIcOMGyqf+/ZiJFTkdA4r1xM7e2pHe X-Google-Smtp-Source: AGHT+IHHdO2UVy3f/phSyaNBb+ZONnyKcOG5/x+wtGWZrfvuPNU1iNTlmuDJ6E75KhK0r1fhZ58Rcw== X-Received: by 2002:a05:6871:3426:b0:22a:5423:45ff with SMTP id nh38-20020a056871342600b0022a542345ffmr1873641oac.40.1711393493524; Mon, 25 Mar 2024 12:04:53 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:53 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 09/12] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Mon, 25 Mar 2024 12:02:27 -0700 Message-ID: <20240325190339.696686-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni To simulate FM functionalities for initiating Dynamic Capacity Add (Opcode 5604h) and Dynamic Capacity Release (Opcode 5605h) as in CXL spec r3.1 7.6.7.6.5 and 7.6.7.6.6, we implemented two QMP interfaces to issue add/release dynamic capacity extents requests. With the change, we allow to release an extent only when its DPA range is contained by a single accepted extent in the device. That is to say, extent superset release is not supported yet. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "offset": 0, "len": 134217728 }, { "offset": 134217728, "len": 134217728 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) looks like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "offset": 134217728, "len": 134217728 } ] } } Signed-off-by: Fan Ni Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 26 ++-- hw/mem/cxl_type3.c | 252 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 14 ++ include/hw/cxl/cxl_device.h | 8 ++ include/hw/cxl/cxl_events.h | 18 +++ qapi/cxl.json | 61 ++++++++- 6 files changed, 367 insertions(+), 12 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index a9eca516c8..7094e007b9 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1407,7 +1407,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1446,7 +1446,7 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1552,10 +1552,11 @@ static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, range_init_nofail(&range1, dpa, len); - /* - * TODO: once the pending extent list is added, check against - * the list will be added here. - */ + /* host-accepted DPA range must be contained by pending extent */ + if (!cxl_extents_contains_dpa_range(&ct3d->dc.extents_pending, + dpa, len)) { + return CXL_MBOX_INVALID_PA; + } /* to-be-added range should not overlap with range already accepted */ QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { @@ -1585,9 +1586,13 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, CXLDCExtentList *extent_list = &ct3d->dc.extents; uint32_t i; uint64_t dpa, len; + CXLDCExtent *ent; CXLRetCode ret; if (in->num_entries_updated == 0) { + /* Always remove the first pending extent when response received. */ + ent = QTAILQ_FIRST(&ct3d->dc.extents_pending); + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, ent); return CXL_MBOX_SUCCESS; } @@ -1604,6 +1609,8 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, ret = cxl_dcd_add_dyn_cap_rsp_dry_run(ct3d, in); if (ret != CXL_MBOX_SUCCESS) { + ent = QTAILQ_FIRST(&ct3d->dc.extents_pending); + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, ent); return ret; } @@ -1613,10 +1620,9 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; - /* - * TODO: we will add a pending extent list based on event log record - * and process the list according here. - */ + + ent = QTAILQ_FIRST(&ct3d->dc.extents_pending); + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, ent); } return CXL_MBOX_SUCCESS; diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 951bd79a82..74cb64e843 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -674,6 +674,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) ct3d->dc.total_capacity += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending); return true; } @@ -685,6 +686,10 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents_pending, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, + ent); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -1449,7 +1454,8 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ + case CXL_EVENT_LOG_DYNCAP: + return CXL_EVENT_TYPE_DYNAMIC_CAP; default: return -EINVAL; } @@ -1700,6 +1706,250 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.1 Table 8-50: Dynamic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, +} CXLDCEventType; + +/* + * Check whether the range [dpa, dpa + len -1] has overlaps with extents in + * the list. + * Return value: return true if has overlaps; otherwise, return false + */ +static bool cxl_extents_overlaps_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return true; + } + } + return false; +} + +/* + * Check whether the range [dpa, dpa + len -1] is contained by extents in + * the list. + * Will check multiple extents containment once superset release is added. + * Return value: return true if range is contained; otherwise, return false + */ +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_contains_range(&range2, &range1)) { + return true; + } + } + return false; +} + +/* + * The main function to process dynamic capacity event. Currently DC extents + * add/release requests are processed. + */ +static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, + CXLDCEventType type, uint16_t hid, + uint8_t rid, + CXLDCExtentRecordList *records, + Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + g_autofree CXLDCExtentRaw *extents = NULL; + uint8_t enc_log; + uint64_t dpa, offset, len, block_size; + int i, rc; + g_autofree unsigned long *blk_bitmap = NULL; + + obj = object_resolve_path_type(path, TYPE_CXL_TYPE3, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve CXL type 3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + rc = ct3d_qmp_cxl_event_log_enc(log); + if (rc < 0) { + error_setg(errp, "Unhandled error log type"); + return; + } + enc_log = rc; + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = offset + dcd->dc.regions[rid].base; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + num_extents++; + if (type == DC_EVENT_RELEASE_CAPACITY) { + if (cxl_extents_overlaps_dpa_range(&dcd->dc.extents_pending, + dpa, len)) { + error_setg(errp, + "cannot release extent with pending DPA range"); + return; + } + if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, + dpa, len)) { + error_setg(errp, + "cannot release extent with non-existing DPA range"); + return; + } + } + list = list->next; + } + if (num_extents == 0) { + error_setg(errp, "no valid extents to send to process"); + return; + } + + /* Create extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = dcd->dc.regions[rid].base + offset; + + extents[i].start_dpa = dpa; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + list = list->next; + i++; + } + + /* + * CXL r3.1 section 8.2.9.2.1.6: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + /* FIXME: for now, validity flag is cleared */ + dCap.validity_flags = 0; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + /* + * FIXME: for now, the "More" flag is cleared as there is only one + * extent associating with each record and tag-based release is + * not supported. + */ + dCap.flags = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (type == DC_EVENT_ADD_CAPACITY) { + cxl_insert_extent_to_extent_list(&dcd->dc.extents_pending, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_ADD_CAPACITY, 0, + region_id, records, errp); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_RELEASE_CAPACITY, 0, + region_id, records, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..d913b11b4d 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,17 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index df3511e91b..b84063d9f4 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -494,6 +494,7 @@ struct CXLType3Dev { */ uint64_t total_capacity; /* 256M aligned */ CXLDCExtentList extents; + CXLDCExtentList extents_pending; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -555,4 +556,11 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, CXLDCExtent *extent); +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, + uint64_t len, uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size); +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index 5170b8dbf8..38cadaa0f3 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,22 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.1 section Table 8-50: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t validity_flags; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t flags; + uint8_t reserved2[2]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x18]; + uint32_t extents_avail; + uint32_t tags_avail; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 8cc4c72fa9..2645004666 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -19,13 +19,16 @@ # # @fatal: Fatal Event Log # +# @dyncap: Dynamic Capacity Event Log +# # Since: 8.1 ## { 'enum': 'CxlEventLog', 'data': ['informational', 'warning', 'failure', - 'fatal'] + 'fatal', + 'dyncap'] } ## @@ -361,3 +364,59 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset to the start of the region where the extent to be operated +# @len: length of the extent +# +# Since: 9.0 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The device will +# have to acknowledged the acceptance of the extents before they are usable. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add +# @extents: Extents to add +# +# Since : 9.0 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to release +# @extents: Extents to release +# +# Since : 9.0 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Mon Mar 25 19:02:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602793 Received: from mail-il1-f176.google.com (mail-il1-f176.google.com [209.85.166.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 943F44C83 for ; Mon, 25 Mar 2024 19:04:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393499; cv=none; b=t0OHj7i4RKR7CY8TibHt4am/GDa+SHKZssrIzDa7x+8l+UE6s9YqMngfVxZIo99A6r4PqULVbS7STtSN1r73VMiKsBivg3+QalZXq0KD3Pb+RlPhEK1Tjt24dZK231OCCR5/dFBaUOnXWCTODgjYZ2uebkOgqstjcZsW4xUW7Ic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393499; c=relaxed/simple; bh=+SYHBmlTlkSltya5tL8aSHbriUoG0N83e336Fnr8qME=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z0+p7/KYYBb+1v72KGPlR31LUnEvG5nVtQ1j9l9x5XxI2XXcBuE96jFRyrD3CBvWkPcJEBWMRDoy1J9DysJB3x9igmn1c/qCBAacBQl6dwb00MSH0VAj4LgmHZ+08lCEkpoyZ8VAxs7wfR3qC/0ffGnH4PY6Lir5ft+u25ccEDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Enn+CrYl; arc=none smtp.client-ip=209.85.166.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Enn+CrYl" Received: by mail-il1-f176.google.com with SMTP id e9e14a558f8ab-36897ed32f9so2750205ab.1 for ; Mon, 25 Mar 2024 12:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393497; x=1711998297; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9hl+9iwdY1IzIu9iDPkfmumLwvkEBhE4sR7cuyu/dHg=; b=Enn+CrYlRHYxLHkk61EevpjtAeDiWOEyLAvfdKSkvNn+gn9YlPjv83qCm53p1hC2Oq Sio6Ia4mINzckgbsBE3j2ldHbf2h77H5R9Lirk4YTIO8zuD4RodUzFg9GMvbaJVNF7rX w4rL4LkGIwiX1HXUtRlurte79vGZGW6KKWHgudMQc3y2l5kRo6b5kO+YRj2Cve5H4FEg SA6E0aE5lt/NgGD0KhFXM/5Dzbw9rrMWct1rEmYGMyOASOemZJ8/ldIv97j0Y3vZqQ5z six/vQEP6shKZG2kzYfKmMQdH5bQ/xmjakIbs1y0AScz5eQYdeJqaCTXNUds+gflzx0d l9BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393497; x=1711998297; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9hl+9iwdY1IzIu9iDPkfmumLwvkEBhE4sR7cuyu/dHg=; b=R2IuhT5XHgNM6YCKLv3NoMBid2zu8XYAcn7GO1z+EqemnDskohkvRSt/Lih1XYnViP lAbVzcP7QQ5hTgQi8i+mZA8GMOp3JG+ExxxrjqMuXuX4GjbxKlCLwETSPIEnRog9oo7t MpGPWnUMFMXTTYrjsJRGPm9tVpwkoyWhsgloLqgiD3WOGp+XG/Z0XFAVLOrp2WMY77SS MOOzSzF+T1LkVW8IDIMspjpT3CicQIgdlAQdTtLqGhzfdvNhcHMsMTYTVn93mYXjL1Ru WIHttYzlhimJl8UiPUTKIDqX54R5Egys+j0EPpeLiFZYQGwE8G7rT401RBpZozh0bJrC ak1A== X-Forwarded-Encrypted: i=1; AJvYcCXoKvSbqdNWSAb/raTbNHPNdJjmsWHZzWq1w+ZdY72hgHN1k0WwGe0IhF2ba0JqiHvGBS3ww7JzA2M7Mhk0NwP5YUYO7byCYnPo X-Gm-Message-State: AOJu0YzJlZl/cUgm/kfSXD9DWGsc2b6TGlfG2Lpo7/bKtqHc0nAl1moK 9+Wifu/V4ABrXaGm7KEbfjghNEES8sWQFioULL2VMqogPN3r5p2w X-Google-Smtp-Source: AGHT+IGc9FRWx0eL65nIDDAzsuPuf8IL9et/6B2jfyccxqQmw9OSUztCOPmfFxh3U4MmMzExS+vMCQ== X-Received: by 2002:a05:6e02:805:b0:366:89f6:82f8 with SMTP id u5-20020a056e02080500b0036689f682f8mr8256468ilm.1.1711393496642; Mon, 25 Mar 2024 12:04:56 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:04:55 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v6 10/12] hw/mem/cxl_type3: Add dpa range validation for accesses to DC regions Date: Mon, 25 Mar 2024 12:02:28 -0700 Message-ID: <20240325190339.696686-11-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni All dpa ranges in the DC regions are invalid to access until an extent covering the range has been added. Add a bitmap for each region to record whether a DC block in the region has been backed by DC extent. For the bitmap, a bit in the bitmap represents a DC block. When a DC extent is added, all the bits of the blocks in the extent will be set, which will be cleared when the extent is released. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 6 +++ hw/mem/cxl_type3.c | 76 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 7 ++++ 3 files changed, 89 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 7094e007b9..a0d2239176 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1620,6 +1620,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); ent = QTAILQ_FIRST(&ct3d->dc.extents_pending); cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, ent); @@ -1798,18 +1799,23 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, cxl_remove_extent_from_extent_list(extent_list, ent); ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, ent_start_dpa, + ent_len); if (len1) { cxl_insert_extent_to_extent_list(extent_list, ent_start_dpa, len1, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, ent_start_dpa, + len1); } if (len2) { cxl_insert_extent_to_extent_list(extent_list, dpa + len, len2, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa + len, len2); } len -= len_done; diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 74cb64e843..2628a6f50f 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -672,6 +672,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) .flags = 0, }; ct3d->dc.total_capacity += region->len; + region->blk_bitmap = bitmap_new(region->len / region->block_size); } QTAILQ_INIT(&ct3d->dc.extents); QTAILQ_INIT(&ct3d->dc.extents_pending); @@ -682,6 +683,8 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; + int i; + CXLDCRegion *region; QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); @@ -690,6 +693,11 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending, ent); } + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -921,6 +929,70 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len - 1] to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether the DPA range [dpa, dpa + len - 1] is backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + /* + * if bits between [dpa, dpa + len) are all 1s, meaning the DPA range is + * backed with DC extents, return true; else return false. + */ + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len -1] to be unbacked and inaccessible. + * This happens when a dc extent is released by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1025,6 +1097,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index b84063d9f4..bc90da2ca2 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -450,6 +450,7 @@ typedef struct CXLDCRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCRegion; struct CXLType3Dev { @@ -563,4 +564,10 @@ bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size); bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, uint64_t dpa, uint64_t len); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif From patchwork Mon Mar 25 19:02:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602794 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A115A748A for ; Mon, 25 Mar 2024 19:05:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393503; cv=none; b=UgpyO9vDYB6wvO76tiGaXPZ/+7v1aOtA1/lV5G6N8HfOfM3zu+kF2efujYcYRNzK4h4w4S5c5NK3Mrt6bwV0wDVsmEvbszZzJ0OYpYtFV5z3WA/99cjCfPd2/MhMR/skxHak8NFkxvxxKcfL3NMCZE2EB0Ejav/SJVHFxr6QLLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393503; c=relaxed/simple; bh=CZzdOAcOeX/C0dqOhd8+qNxV7JSySd44+FqRVJgf21w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WvQi7oyVcJUXjiToH5vkgSHN0z+HisvyEGMO31l1jWPx0QmQhAZ/cSPwbiSd1TX3BPMrqrDmB1ZAX1uMhOUVplO24UDGPOfM+TZD1fAyzl8Ob7D8+vaTabDzuiXIf7fvBLq+dib8jG0av7TriSWjCsX1AhTljv30hbuFhr9w1t0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dwX+0hFA; arc=none smtp.client-ip=209.85.166.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dwX+0hFA" Received: by mail-io1-f52.google.com with SMTP id ca18e2360f4ac-7c89cd8df36so96297839f.1 for ; Mon, 25 Mar 2024 12:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393501; x=1711998301; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Eyoni5qZtSKjt9z3QzKJDUPa9p/nHFzydRxiuFbxHok=; b=dwX+0hFA/5Q7mqBjP3PEomIgVWsKzKpuxnOMr349tqQZZBp5Mey4+cemu+zl+GDdBh KC6iRw/T5Lh+cJw7fbCTGkV2n0JGPP/YyxWx72L60/RchH7TuTcJY2U3pdiC7VE2WUY3 vaXftZd9A7zgdNdfZVKFy3Y51kYa0vOUopFiTAn4nlEVp7zejIUtmb/45nSZuxeFlbGN T/Z8jF8JlCNhD5qzVgLPQM2+cQ1zRCd/w/W1y5y/iH7O12Saawcgflok3POXsr+fXq2o LpbvusyiTGXFW43H3cSUAAK9uVahxo4k2Uk4r1drNg9hRgF39UovtIqVPrZZysNv4i2O M3mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393501; x=1711998301; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Eyoni5qZtSKjt9z3QzKJDUPa9p/nHFzydRxiuFbxHok=; b=etwT++AO0oQZT6Vq6TgmBFPMazCkLLvNW1ESkLGljl/+Y+fahX6AOLLA8408KWmDfZ rvg2cKy7hVBp5KPEJ6u7Tce2MbJbh6VMrgnXe8oVofnicgHE5KRK2KLpV5OeFvtJQzhG lNprAaYpIfJ9yoN0PbH6HgV+vFmT2cjpvIZP/AOQJ0dz/Smwnx9FX6v8vX8kew3zWUs1 3haU+Z43i70Q1vfh/JgyLlj86j9/8D3KKvIF8xgZgPRGnFPQwtGUXQK4AvcOaMNAlsDY ZI9nbkMUa1UA6VUbZ+GqUgqP0TlGtm5qeyUBVtMpnVoyIFcf13MIsUAvDYzzTCQhedh0 uwcA== X-Forwarded-Encrypted: i=1; AJvYcCXIChcXnqHFD8d8pnio/ZqPP5sXUQHp0Ivz9OTr3RI5hE9CrEqw1H0/sSDTn+rMsWIZfCG9y6GMB35U9FauZ5YQeL3U28dOicKg X-Gm-Message-State: AOJu0YzLgSO93ZiaUayVTiIqkX7o2iWb7Qc3E3ulwsdUtxMqmd8M07N2 NDbwRjJNya66ul39KKJlmAPd9hkRBH2LuhlMfV/pHZ+M0H4Yl5HfLg021SnI X-Google-Smtp-Source: AGHT+IFFOVVYKVLk+nUgND3cD8rgUQNgsrXraNM2gggO8CtW2mKT/2B7DOt2cMvN8p4yrw0yaEYmJg== X-Received: by 2002:a05:6602:c5:b0:7d0:3d39:5fbc with SMTP id z5-20020a05660200c500b007d03d395fbcmr7642622ioe.1.1711393500891; Mon, 25 Mar 2024 12:05:00 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:05:00 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 11/12] hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support Date: Mon, 25 Mar 2024 12:02:29 -0700 Message-ID: <20240325190339.696686-12-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, we extend the extent release mailbox command processing to allow more flexible release. As long as the DPA range of the extent to release is covered by accepted extent(s) in the device, the release can be performed. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 41 ++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index a0d2239176..3b7949c364 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1674,6 +1674,12 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, dpa = in->updated_entries[i].start_dpa; len = in->updated_entries[i].len; + /* Check if the DPA range is not fully backed with valid extents */ + if (!ct3_test_region_block_backed(ct3d, dpa, len)) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + /* After this point, extent overflow is the only error can happen */ while (len > 0) { QTAILQ_FOREACH(ent, &tmp_list, node) { range_init_nofail(&range, ent->start_dpa, ent->len); @@ -1713,25 +1719,27 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, goto free_and_exit; } } else { - /* - * TODO: we reject the attempt to remove an extent - * that overlaps with multiple extents in the device - * for now, we will allow it once superset release - * support is added. - */ - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; + len1 = dpa - ent_start_dpa; + len2 = 0; + len_done = ent_len - len1 - len2; + + cxl_remove_extent_from_extent_list(&tmp_list, ent); + cnt_delta--; + if (len1) { + cxl_insert_extent_to_extent_list(&tmp_list, + ent_start_dpa, + len1, NULL, 0); + cnt_delta++; + } } len -= len_done; - /* len == 0 here until superset release is added */ + if (len) { + dpa = ent_start_dpa + ent_len; + } break; } } - if (len) { - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; - } } } free_and_exit: @@ -1819,10 +1827,9 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, } len -= len_done; - /* - * len will always be 0 until superset release is add. - * TODO: superset release will be added. - */ + if (len > 0) { + dpa = ent_start_dpa + ent_len; + } break; } } From patchwork Mon Mar 25 19:02:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13602795 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B28BC63C8 for ; Mon, 25 Mar 2024 19:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393508; cv=none; b=HS23ZP1BeAaR/dXTj2iQfrG9ppc2XetEs9GUBUfdPSAIemhXPnNREcu3g9Ed3iO/rU6kzVxQYSFFWPXRdjs1ZwhteyJ+j8XqYh11SYQIYxZSvC1FTAZkQfM6gIHVQgyeOQG/G3LZmWz9xrm+UG9GdBNw8iAXmsQkoE+rLtmTREM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711393508; c=relaxed/simple; bh=bY9OUoQnlJcGkHF0LCPSTf31ilHkOgLIamkwuAvGnms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nPrHE+LTBFybiVBWVcRAkUnY3shVJ2g0E764ljitqmFwpLhCS8uRuc3w8lyUSLHY+tfamlh6jEdgU9TlxI+jiT03G3MNlOMTvq3awCDrpa+1LskndVUs+Clj914xnph7z5X2OB5YubMA5Vl4AtzkDPtMT3nAgOFL/n0aYyz3CIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fODg7s+/; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fODg7s+/" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6e6aaa2f02cso2888081b3a.2 for ; Mon, 25 Mar 2024 12:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711393506; x=1711998306; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j3xErw1D7pd0Oy+zBQHAytaYYoPYpk9Jd+HPJGbw/j8=; b=fODg7s+/lUaL/yWVsLD0ozWNg2eJ1qZKn+wG+89K11gXfuI0PfW6kMMH91n79euNlm xR+TIEAfYsvs2Xj9YdFpw5ZlMsVY133pn4oudapIqvoaK8wavUwehIO10js+8M5VWQv7 ArJJXRY/VfZ6brnqqCB6dRr/l0jXo0I9ZOaoKA0BpO3cspktSoADDxkG5TFZvfd9qKtv sxogURyUAbdJuooPjkuwCxP1NzkKQVe88ySzKPXCavuXQwr6M1KgPR5FT+W4VopBSJH3 bpMekX1p7FaqeYawroV8/io1eWnkObxVkt6HBCdZmig3L8iolQRP6RgWbUHmG1JjaF5r Wf2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711393506; x=1711998306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j3xErw1D7pd0Oy+zBQHAytaYYoPYpk9Jd+HPJGbw/j8=; b=UwOBKZ2MVQgnc7ygBLWtKQ24n1P9X5o/hb9HmbUG6xI26xiNSt33hcR5VfV5RZJlqI sERshfD//XQVv65WAVZ8CzA+VA+sxhsnlMCYS878/H4qa3etTvbsqksPCRBkpS3kfcN5 mmvbzvciCK8jqx9aSxXoZD+bwflBLClHJm7njnXDicUWlCM20246IFLOvFGLwAbvL6NS ZRSSfiDe6Kz9/S+RWt3kOPM8dBfuIt8xb9H+jMvlRr2GlIvbPW4rzecyaR5VwuvEmAI4 yugTDNyr93/eyq8CGiw8Kg6U/2WLprmbgDa7Fujlv59uhAFrWWECJYZEOjf+H/lgXprl KZYw== X-Forwarded-Encrypted: i=1; AJvYcCWqSCC78B5TLBIrHwWY9TtYSDmulcj4OfVP8Ir6PbBEDlS1hq6ZZKOlr8baQHOizHBBiY5/Bwa2qaCCPyEbTtvSnmyIWWFlePc3 X-Gm-Message-State: AOJu0YxccucepUIyvredJXmWaG2DHgvcKay8/TvNyqOBK00zvIKunxEG /AqoJq4bSQOyZL/k5kOqwj5dJILuv9UYO8yfzBT43rGzEp5ynEDWIQAU6Ieo X-Google-Smtp-Source: AGHT+IGAoTYpamTa5d5OnaOZK4Ce1M2BOIOF3MtObyKz38nf/HynrGLAJSbsZFdb3lLdMdbSuUH8OA== X-Received: by 2002:a05:6a20:12d5:b0:1a3:647b:b895 with SMTP id v21-20020a056a2012d500b001a3647bb895mr8351996pzg.47.1711393506060; Mon, 25 Mar 2024 12:05:06 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:7bed:2ef:bead:18b]) by smtp.gmail.com with ESMTPSA id z7-20020a634c07000000b005f05c9ee8d3sm1722663pga.93.2024.03.25.12.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 12:05:05 -0700 (PDT) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v6 12/12] hw/mem/cxl_type3: Allow to release extent superset in QMP interface Date: Mon, 25 Mar 2024 12:02:30 -0700 Message-ID: <20240325190339.696686-13-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325190339.696686-1-nifan.cxl@gmail.com> References: <20240325190339.696686-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Before the change, the QMP interface used for add/release DC extents only allows to release an extent whose DPA range is contained by a single accepted extent in the device. With the change, we relax the constraints. As long as the DPA range of the extent is covered by accepted extents, we allow the release. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/mem/cxl_type3.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 2628a6f50f..62c2022477 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1935,8 +1935,7 @@ static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, "cannot release extent with pending DPA range"); return; } - if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, - dpa, len)) { + if (!ct3_test_region_block_backed(dcd, dpa, len)) { error_setg(errp, "cannot release extent with non-existing DPA range"); return;