From patchwork Mon Mar 4 19:33:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581140 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74CBB7994A for ; Mon, 4 Mar 2024 19:44:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581449; cv=none; b=D64aKw94XP4VvYBBU34QJECNxJ3EfLbGxhX+qSkZlvMK2CUx1+yRMDx948yw3VczWYJUxc5E5kvgQ96MljK2hS7XA8yCBwJSVVR/E3VVo3AEiZSCxsRN84wJ5oXyYFn829A9ZAIbeAcEAZfNk1bjhY9kk9aiyIjFNQ/SY0QOitg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581449; c=relaxed/simple; bh=gkUBdmYA71Hv10Hl86RlZ3WRrrRHuAjCA/UFYnu1kv8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uDkS4dNegcS98xG5EAsa02V9sF9B4L3/PgpUA/y2aWRtPYDMYzdWcE37p2ufLS0RAXPoW05HdJjiJRQNuwbYFM9Q6UMLg08n++UZw/PV4REKj+bSiuL4KuTOE+DqTRE5TMwASCjoRIsC0Y8LASrclrIaiww4mOn06SBHjsfa0n8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N1jHr44S; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N1jHr44S" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-dbed0710c74so4372311276.1 for ; Mon, 04 Mar 2024 11:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581446; x=1710186246; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AkPmI8FuszQzNKc/W7C7bH++18bz1caF1vBxyxTADMI=; b=N1jHr44S33fMwvuJJjgQMLNpGwxr6DUzG6z/DeZ9NHINMtpCHvvqfCwF2wLUHmoeg6 /g5H9ZgyuymUE9JcO6yOUksK7JmeIZ3gnxqGsj6iN+IGRCN8yjZsf6afxGy/cNh30/Bs JNcLMhLZIXl6y4EaGNdBn6hvW5pA8ZyTvvyNE4ORpjJZkQ5DEPKdZl2HXnapyRbNyRik jA5HPGCzzzAf0E6NyCCuiTNJHuF8KzuGKdsjmhbkrYucqs6r/QjwfMVfeRTjLuEUxGIx 748A3x3HPjjmA5UDEzL5tux5vRDSufC/F5NWdFgNr5hZgQSWRH63+1V7a/jIsqJaL+DG w+xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581446; x=1710186246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AkPmI8FuszQzNKc/W7C7bH++18bz1caF1vBxyxTADMI=; b=nhtWIFHmj2rnWlScOQzKmf5y2AFo5fdWh7eNlXBPdX3w8TNsxU30/ycuRPKCP1CQ1q lAnKu5f3XTr2kAXztr9T+O7Rkv5hlfNiciUdSNSdH4iloi+mIhqrFlLI2PTPm5V30Vo7 ITeDUPbadQE7d9T/FdjC3VZn6dZZ1PaKpBfY02av1/9Tn3Xidxk2IkBvQ1V4Eq85YBbR zLHGo0N7eti/IMR8TgcdY+/PmkQa5b2O6cfGB+gg5KZ/06AQ+68UMxaxitQFL9LGhXyX NmVFZC3q009/2SwtrF20JuZVq4GW8rCewVfWNclmgf7xuMTM/prtEqugFGL9vsESeGtz 8dGA== X-Forwarded-Encrypted: i=1; AJvYcCWJ9ukyZJRVJM8aDfVVSe8K4nEWrqQApCnEbXRj4OU4TH93MPx7vKXabdlvl1HQiPoJx+kNuWCdDcA5yBYo0gUNhZ+kPLJfwKgb X-Gm-Message-State: AOJu0Yzr0u3rumSJT3aQs5tiKA6smGq8kfTZtjP6GiWjngceRRoLujPD 5hpENW6lWKND976NnO5eT/WDzJMJDnIRdRrXsGrLxEztXQFICWuN X-Google-Smtp-Source: AGHT+IF8Ycb0Ig4cvVD+KZC7NiE7FpVTQX5T1d6xPH2bLLUEk6Pr5KEZbkD4FKZ2xmlpNCQuOqLiug== X-Received: by 2002:a25:b109:0:b0:dcc:6112:f90d with SMTP id g9-20020a25b109000000b00dcc6112f90dmr7174783ybj.62.1709581446482; Mon, 04 Mar 2024 11:44:06 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:06 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 01/13] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Mon, 4 Mar 2024 11:33:56 -0800 Message-ID: <20240304194331.1586191-2-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Based on CXL spec r3.1 Table 8-127 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 4bcd727f4c..ba1d9901df 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -780,8 +781,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -807,6 +809,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Mon Mar 4 19:33:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581141 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF78F7994A for ; Mon, 4 Mar 2024 19:44:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581451; cv=none; b=ESOXhyWYlKwQBL/lm+wHeZZdV3Q6mwIxVeLOwRZtumn5EoBw00FmAfhJBSthyjOOpCcnd1mR9J1UzO0v1TiJGlnl4zixZL82Y+sPGCqp4xszEq1QwiJE93LXSa3zvu75YBwtWeCfRGnvvkrR0Dqk0EIBpq+fgE3J2ugcZwrLiSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581451; c=relaxed/simple; bh=VdbLca2kR5Vi3gPOYkHJguMJY8WrEBTxrSBNpkaV4ZU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UDhYEKunZN48/B8grL0U++YBTlmdl96f7JbpCZysrax0mivZuGzBc82rnxCpTc3zEWVYzl0g9bmVCVgkUmCdUPwhAn+6sYnOn35yqxbluDhFQnLk+yQT5CSa4ljr4e/5lI/+SbxusWI3T1FkbL3ez6nGYwrMIXkMu4nuzpf1hXM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LsPjTfXJ; arc=none smtp.client-ip=209.85.219.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LsPjTfXJ" Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-dd014003277so1793214276.2 for ; Mon, 04 Mar 2024 11:44:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581449; x=1710186249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9sd2BgVEGhVtZ8GHzjE9iedUGsXSGw1JofJzU7n5NJk=; b=LsPjTfXJsOqugi33neqhsN2VWaQfiiNFGVLuQXvlrMEVC44QxEBiX0M4xpht3LTArw mfUxru3CxOvZ+ZZHrP4uu9djuEslmSSx20RGPuozHeu/C1YOHTwB/jDAf9cjNHDe97Ot eQKjon8fJai11K64SzMVjIzpWSAHxAPkrF4aLGndHAbXL+KFv4T+aDrRMj0Y72ubKg9D c9l9yO59OhGnz7/7FX+u89fncXLGfVXa+QvhGd61p8GzGjTq70Rg38eexKDhoyyJu4xv qmyR7Dg/FGf/waSbHDH0aJygQHvTQn+wsTSmTqDHeKx76yIcq4xhzmYVTOq8TRLTnKIn zfXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581449; x=1710186249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9sd2BgVEGhVtZ8GHzjE9iedUGsXSGw1JofJzU7n5NJk=; b=t/rAwsxd5DoILtepk0x7uoE0w3ROgMktQVRDUStno6tML8clvz56YApookGAOJOxpO yVQmX4zH7lM5Zcipo34tedePlAbLbuXc4jSXfhw9/DzlQbSRV1vROhdKGJIuQ/05Cs4p F2mBIiItiNHIiu0qeb7mkKwT4qGHNya9R8nrSQjGj8Eo7NdL0Tic9RWd7wmWlQnotF7n dw4o+1E6CMKq9ly3eQs8/TbuYzeikyTlMno/139eQNftc5FIvlfsL8wrn3rzMr6HMdD3 30mIIfmCWNmVq6fCrgyDC1/Rsqyg1x/Ci4FyrHHM+6eUtTbI7E2CjlYgb3meT2DgktYR VYSg== X-Forwarded-Encrypted: i=1; AJvYcCWc1sf6jLsPjR0I9+JYtuT8WCFRXsCCBS6EQ0IoPLiZdzKC9XsYJJIoa0L5Dq9VcRvYEKbiwrdYXOxXRooPpMpg8HD6Fwge954I X-Gm-Message-State: AOJu0Yym89lMh1JzDi46zZNR0Y3uOj6wxDYjwHGi5BwIWExVdWnTsZ3x s1BsqPbwQSm6XnFFkw0hshbn5r2rczRv6vU7ihspyZ+656rmiexe X-Google-Smtp-Source: AGHT+IG39RPA5p0WekIO5yNVRaTUVzpbb/YPybrWX41+xzC4a8iu9naWr1DSiius45eB4Q8FHykk6A== X-Received: by 2002:a5b:f05:0:b0:dcc:8c5e:7c9b with SMTP id x5-20020a5b0f05000000b00dcc8c5e7c9bmr7046054ybr.57.1709581448659; Mon, 04 Mar 2024 11:44:08 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:08 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 02/13] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Mon, 4 Mar 2024 11:33:57 -0800 Message-ID: <20240304194331.1586191-3-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per cxl spec r3.1, add dynamic capacity region representative based on Table 8-165 and extend the cxl type3 device definition to include dc region information. Also, based on info in 8.2.9.9.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: we store region decode length as byte-wise length on the device, which should be divided by 256 * MiB before being returned to the host for "Get Dynamic Capacity Configuration" mailbox command per specification. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 99 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 ++++++ 2 files changed, 115 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ba1d9901df..5792010c12 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -22,6 +22,8 @@ #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 +#define CXL_NUM_EXTENTS_SUPPORTED 512 +#define CXL_NUM_TAGS_SUPPORTED 0 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -80,6 +82,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1238,6 +1242,91 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint8_t region_cnt; + uint8_t start_region_id; + } QEMU_PACKED *in; + struct { + uint8_t num_regions; + uint8_t regions_returned; + uint8_t rsvd1[6]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED *out; + struct { + uint32_t num_extents_supported; + uint32_t num_extents_available; + uint32_t num_tags_supported; + uint32_t num_tags_available; + } QEMU_PACKED *extra_out; + uint16_t record_count; + uint16_t i; + uint16_t out_pl_len; + uint8_t start_region_id; + + in = (void *)payload_in; + out = (void *)payload_out; + start_region_id = in->start_region_id; + if (start_region_id >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_region_id, + in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + extra_out = (void *)(payload_out + out_pl_len); + out_pl_len += sizeof(*extra_out); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + out->num_regions = ct3d->dc.num_regions; + out->regions_returned = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_region_id + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_region_id + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_region_id + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_region_id + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_region_id + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_region_id + i].flags; + } + /* + * TODO: will assign proper values when extents and tags are introduced + * to use. + */ + stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); + stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1282,6 +1371,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1487,7 +1581,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 3cf3077afa..93ce047b28 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -422,6 +422,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_NUM_REGION 8 + +typedef struct CXLDCRegion { + uint64_t base; /* aligned to 256*MiB */ + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -454,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCRegion regions[DCD_MAX_NUM_REGION]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Mon Mar 4 19:33:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581142 Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFA5E7994A for ; Mon, 4 Mar 2024 19:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581454; cv=none; b=A/t1wr1paxMayRZCEYAWb8a3She/W1caMfXRiayczVGK6xLconxACb/Q6ZSt6AFSJhSaptbs2hXULTIBqTBO3cXTEFSm1RHVT087ohfAY9CQPgKIu/nM48BUCTdkJZQZ3efo+O4g6l2llqcVu3u8yC6eM2opA+BryOlK+EUhDrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581454; c=relaxed/simple; bh=+RjIdzXfin30Va7NvOuSi1ob0+wL6GMLo+/DcgnOx1w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NNX6B8XF3x+vsthvZxetltfzRja+wSVqPj9c68yrR9jkhk3mg2qVSRrMgcYtCQL5o6GcqbEFg4UuzuUD/r+IZVS/4LJqJpxefmhkC5zdeNvaXhOqOCrEHqsxLkaV7j5ufYWFh9DSbIJnOxLPxfWusoIcuTCQe56o5lYtrULVwc8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DHGcagtG; arc=none smtp.client-ip=209.85.219.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DHGcagtG" Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-dcd7c526cc0so5184511276.1 for ; Mon, 04 Mar 2024 11:44:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581451; x=1710186251; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2InpeJRt+f0kCa8qliojbJhzR8/UO/mLPd8lEue23eI=; b=DHGcagtG287NJJguAKiW5KsUGB7HxGExXHHm+lhpAiLeUaBIQqtI/zFdiGyBN07eTE mKF4h3amrXkGVbVB3/NuJFfp8rc0NAVsDRNkedncPeBkekC2TrvuGrauwXE51oWTpYme fr7stmpMKPj/jCrxY1rK8QhRvGL9kpPvWb0fPhp4+utTG1dWqfbWPlX3sTWZB90Z/4cC 51LQzfJavm0ftfJirQyWoQZIG/IhUz/TBf4XwUMOHYYoB0H1OEyL7044aJVoUizZXV/3 gc488l0D7zHz3/OuGiu4jKYQ+VwtlPWEtxOnEhztDfjrcN9pVuuj29T8RFCI7PFoQ8k9 tXZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581451; x=1710186251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2InpeJRt+f0kCa8qliojbJhzR8/UO/mLPd8lEue23eI=; b=ubvpKtk+fAhD8jyfW625kFVVlec26z5hDPnG9Ws+UEpybWq53hJV6uMqCB7d1MW3JP C1Onthi480SY6UR3+weCp2j5tiDQk9ENB2zPc1LJZFSRQtZZE3sJhpQAWdaK9XIRxiSI gQ+ounWrYp9Dhzs2j1ySlxSZbXUktZdTpFR3+OI2mHiYlz9GJ3+UUToMLTfPKV0a63vJ FqRWTAZuyGFxXLveUCjLNuxLupPqtnnewgPHlcMf1wUsYiA8QsZATcI34/VoGNqSb2zr T+WiZ0Y5HTukY421rsmnW8O7Bs7A6zGAY9BPSeMOuoa93zxXPE+Jzhld/AfcNic646jK t9uw== X-Forwarded-Encrypted: i=1; AJvYcCVZzngzHC5OC2nLgNAm9zdxubgXvl/2FJMjerYgHTISqY2XIeW8vOf/tibgU9RC28UBoY1Kjy/MiiD6u78SZig9hQ/n3SN3qg11 X-Gm-Message-State: AOJu0YzlghZAx5GdHFSmZnRWgXkVukel6exFCJkSY97IVP3UAvrI7bRs Eu3qFx7RtIhI24diltWjRicEYapLNZ/5zkTBdfBW4k7W/xVk0X/5 X-Google-Smtp-Source: AGHT+IEklaIYAzLXknCQyGLMTYvknIhiLaansPxR45rNdr+9VPvP4NwG4q+IH2RxW6udFYT1qcQEPQ== X-Received: by 2002:a25:aa8b:0:b0:dc2:2b0d:613e with SMTP id t11-20020a25aa8b000000b00dc22b0d613emr7102455ybi.10.1709581450845; Mon, 04 Mar 2024 11:44:10 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:10 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 03/13] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Mon, 4 Mar 2024 11:33:58 -0800 Message-ID: <20240304194331.1586191-4-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 5792010c12..853dadba39 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -803,7 +803,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1179,7 +1179,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index e8801805b9..244d2b5fd5 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -608,7 +608,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -631,7 +631,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -837,7 +837,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1010,7 +1010,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 93ce047b28..f82d018422 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -234,7 +234,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; From patchwork Mon Mar 4 19:33:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581143 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD06F7995B for ; Mon, 4 Mar 2024 19:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581455; cv=none; b=BSL4szSruN+ZC70rPHHriSKukoiDjrqFF56rc+JcAOYhaqTKEqmEOjENXfriweur3RxmMtpIABZ+HQK796MAab7ENqozR/9AxDPQo9i7yZjQvu1knVT3C+ol/g2MYltAmE81/0zBRSzZEQMtGs46MJc+bAFjwk51VaBa0OfwBFU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581455; c=relaxed/simple; bh=KV5t5EDxM1Twm6IkWpgwsiNgnIc9JtxZq0y8ve0PU9E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cwir5YI+YLfTB4daFEO+Al7Rp1kzTecX6yzlrr5JpEVSy4myObMxOhgBXgeE7ZBBCj2+FRyhDd85YMx2OjRJJMW/ROwEjyvlnyERxm2aWhmHFzTqD0iIJICj+DQv98TLEBOs0DpTkCDKN0Gt5sXQRaOCUJpUE7qe5b6LQKAyt5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZC66557p; arc=none smtp.client-ip=209.85.128.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZC66557p" Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-6093e067220so44616507b3.0 for ; Mon, 04 Mar 2024 11:44:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581453; x=1710186253; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qbOcnbLy9moWsSBbE+Q42MVGWhpKJBtzuH2spFlbBIc=; b=ZC66557plN4KI5GYtb/H7cOuYa1qV3/K7V/NEHlSZo3lI0Fq88Cn6tnifBmZu6FKoa TYfjbKREyr5+Hfvs8mMcx/l5f7wNBRuoEPbhtpZxy2NheoCzzOZcdvbmLbnrUcRStIKS lFd6VFkjs3pZXodY1RT2VGEDjyZgFvuUBpzcu+Su7/tamnRmhJgIFZ9ER9eZvKLVtnOx r2all6cCEml/ZyJwu69zbXQLTxSIgLCMoudrgIi53eyp14qmnDXeqZxLz2/UESMfqn1f uFDpNj5xYlokDVhQgqqfJbT+j2Esecs2avP6yXgJSqs5dQhFi+4IpRgcVYz1EAt4TYCO 4knQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581453; x=1710186253; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qbOcnbLy9moWsSBbE+Q42MVGWhpKJBtzuH2spFlbBIc=; b=SD65PmLenlJ7WWc/ezXNkTSZRJigVOkdJJNs55H0kbkRPR9ro04Pg2H/gt+8adWYol vwIvHCgRZnKNMnyJmpB7Jm3J840AVHZzLzlXpEwKner+oiWvahLj//XmqVVxx9nJLTh+ GWj33IV/1wNvF9W8pTZqGXJ5+QLLuG6742KCxxTxDrsmeAY3KKnbGgVsu29XKPd6fcr7 EmFr6q0dEfRMy2QFpwGni4Wf57S0rbbMMFr9pphmfLbJLdH+wqXe/jpXXsZp6Uj94p1P 3BjEcDxywkP+9Pcgpg/DAzJWmMfkMUDI+BPegKI+Yv8I/QGwJvzMEVVBFrF1oTVIRh9A bmRA== X-Forwarded-Encrypted: i=1; AJvYcCU1tfMC2R1iPbjU2y4AWRCi7APeF/ifIIYeXmce/Mjl6sse9aVwHUZsVRo8bLVr156wvOtaJp3r3izGd7JQDivbKtpBsTL5Yr0O X-Gm-Message-State: AOJu0YypOkt5MsG5wqdlM7QQftBTAYgJigJJtszHbWdjUA68WINeKLR4 Q0Na3yVmuyx0NPwXOEvmHvlfXLvsruQ2SLuR9g4wgVXE4wA1QNLj X-Google-Smtp-Source: AGHT+IFj/6AKOb01CS8iZ0cx8I6hTpu+7AdNBUdow4ScUYPauwu6XEYgtcZPPXwXUBAP+6x/56QZMw== X-Received: by 2002:a25:9d88:0:b0:dcb:aa26:50fe with SMTP id v8-20020a259d88000000b00dcbaa2650femr6868080ybp.15.1709581452753; Mon, 04 Mar 2024 11:44:12 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:12 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 04/13] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Mon, 4 Mar 2024 11:33:59 -0800 Message-ID: <20240304194331.1586191-5-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support like below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/mem/cxl_type3.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 244d2b5fd5..a191211009 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -30,6 +30,7 @@ #include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -567,6 +568,45 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: dc region configuration will be updated once host backend and address + * space support is added for DCD. + */ +static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 2 * GiB; + uint64_t blk_size = 2 * MiB; + CXLDCRegion *region; + MemoryRegion *mr; + + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + region_base += memory_region_size(mr); + } + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + region_base += memory_region_size(mr); + } + assert(region_base % CXL_CAPACITY_MULTIPLIER == 0); + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + region->base = region_base; + region->decode_len = decode_len; + region->len = region_len; + region->block_size = blk_size; + /* dsmad_handle is set when creating cdat table entries */ + region->flags = 0; + + region_base += region->len; + } + + return true; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -635,6 +675,11 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } + return true; } @@ -930,6 +975,7 @@ static Property ct3_props[] = { HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Mon Mar 4 19:34:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581144 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4828D78664 for ; Mon, 4 Mar 2024 19:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581456; cv=none; b=CtAJMoKxHOFgTWXZVax8um3JL5odbRaqooeSyqO7NEAGU9nvcMpajB32W40XEDYsQjmzNjnuIpOd7WedOPZNjJh/gr5AckuIL6VqV5Jn/bflKZiSyqLYKVZb5ncWviXDcBlubtKcA1PnLdIGyByy9ADv3gjQvMu79OBZdKnQuQE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581456; c=relaxed/simple; bh=fSXXGXgXBdC7tFiMMXx33MQFn1N/OaZS7IeVp70U0gA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FpKo0slPpQc3qMAfdzj2wl0b7GUfnIt9wFVtODa/EJv0z1im/Dor70l6Jw9vYvUjHM0+BTRBkeDs+kQGHwOEtz5CaxpwioUOJqywRfLPIEgnas47ufaFPfOLE8UCOQjln2OKOmy9ZiB99zQruHzTPXLP6MJoQLUd2WpBS/K7bXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KOVLUA9A; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KOVLUA9A" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-dc74e33fe1bso4614019276.0 for ; Mon, 04 Mar 2024 11:44:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581454; x=1710186254; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XOJjP1Hc+ZaCeqP1kL3i8IBpTTnk/Z2dT1K8zGM/Ls8=; b=KOVLUA9ADB3bG0jCL5N7lLkNQl5jOS7kFJvT+nQBSFXnO8gpHwGbYFhAGpCKaowuQ6 4ufjrjBDfA93YgG1Bf4Hr64q/5LQS696Ny4834E36jyTKJ4U3h9r0yoBmRtTgKbn7K+f zoE4c6ft9L6Y4CjmqAXRstriHB1Enf+LHexPJTU0kdiWKurbX5cTnmGQd8IpvEkiewA5 MFY7rTGwgE0Ly42ry6ROTjLPgHriB6lfclomYc1i5tLg3cr1dOg2NdghmHDb28cu3W/o KMHs1kJ3u6oXxkNhEqcWvSY75M9X2pIG9FVO9YBwT2e3NB+MlbVJQSODDplK1BXM8Qmr yG1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581454; x=1710186254; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XOJjP1Hc+ZaCeqP1kL3i8IBpTTnk/Z2dT1K8zGM/Ls8=; b=jnHpehSGSTCJDmEsi9258iWFLh80oVuzF+yfPPh6ITOCkOqbVgzSkt+ucNK2zXbRuo X9d35oW6HP6v2eBjvrdPVTKMPREVxTGUhAG6cuN5bPs6cm3x76JhjhY0Sa3vT4GTBSaY 8uRh4bIS28pMaInEO9tmKiT0P7K68JWyI/q4H1N9otEYTRZT+C4sqQ7Hz/vy9raJSYFE wksR2Ix40BJ1RSSYV3SNPomuqzIVEFY3npkMeZlnrjTbz798NVrFwT3KZSBydHLyqBwa pGuptO1TLk2EmyTK5kgPVhrUWyN1K2lwpm69wDIdIdy4j+VcrVqXYtL8U0VFiHLmlue1 jk0w== X-Forwarded-Encrypted: i=1; AJvYcCVGxwig3JRDC51n8/FDeCUiw3QB6aPzEjqwIxYT2PtZZybZKbIDlXS0DSC2bZljMd7gGFYs2qObBBFDV8lEwGDiKZFI+6ttAZ+q X-Gm-Message-State: AOJu0Ywwb4Cz4CpVHgD1UtjoWWulzagrh0JD5AxbevLtUVeHzx+h/wOi TRXRlUB/XLXSafBxjE1MxUm9etuaib3J1VS81SMWqlJGTwvueS/h X-Google-Smtp-Source: AGHT+IFL7lKbuf/q1X2xmA0sVMudvhJVA07sf4CQSydEOjZZtUETHFFOZl201CiddOHVU9PW/PCFSQ== X-Received: by 2002:a05:6902:2306:b0:dc7:496e:42e1 with SMTP id do6-20020a056902230600b00dc7496e42e1mr7811375ybb.51.1709581454151; Mon, 04 Mar 2024 11:44:14 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:13 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 05/13] hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr size insead of mr as argument Date: Mon, 4 Mar 2024 11:34:00 -0800 Message-ID: <20240304194331.1586191-6-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni The function ct3_build_cdat_entries_for_mr only uses size of the passed memory region argument, refactor the function definition to make the passed arguments more specific. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index a191211009..c045fee32d 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -44,7 +44,7 @@ enum { }; static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, + int dsmad_handle, uint64_t size, bool is_pmem, uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; @@ -63,7 +63,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -132,7 +132,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -149,6 +149,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; @@ -163,6 +164,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -171,21 +173,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); } table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { - ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; + uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } assert(len == cur_ent); From patchwork Mon Mar 4 19:34:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581145 Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 250A379DD8 for ; Mon, 4 Mar 2024 19:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581459; cv=none; b=tAfhAPlt3w+110Tce9H0gWF//kmvU2earIUFVfm7KxjH6yMQc5yZy30JbqrdlVrruo8MPaskcihhE8hlWTO+GmbWgIsHOQugXwEp7tw11SpQhGdzSzzfDDp8ra3BSgRtXgzMHoR221E7IxXhEJY0PE/QXuHIm0YyF7FHGrTVhUQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581459; c=relaxed/simple; bh=0cSHm8N7/nK6BTyEID2HCCEAkKPBpvlYAwTyCsDJyCI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F5KxVWFbXQOj18v3YqiTO0z3P7o6O28PCnaBsWVhN3G6vI1KUmJ7tSXruxjhFu3gA12vK6BYyoeOlEp60ON+UztwqGQh3nhjcSRup+aK20hd/HbFrBxGw3elukE//Gt3//HF86w/e+FLyOjpF9PUgR91TKiVyJJGUmxtHWAe24o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X6FVKTzc; arc=none smtp.client-ip=209.85.219.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X6FVKTzc" Received: by mail-yb1-f173.google.com with SMTP id 3f1490d57ef6-dcc73148611so5118081276.3 for ; Mon, 04 Mar 2024 11:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581456; x=1710186256; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PfECBT/sLbntByDSR3pn5gaUEK0qP+L5lceYFnDxHOg=; b=X6FVKTzcmJSS1iKaiqEzu5de23bSY+frFggMMau7Xy5bkQcdbp46vonmcqKrcNbzuA r6ofjTTrHYCJPYP6SJkERJBep2u2SB2Q2Iy4I7+00InB9c5flb+PCEZH27hjmR+Tr4YN HHQ8Jzb0NWTb8pI7ClLTja4klIAnzkISjZbUFDsv89vXVz2wnc79FhsTJRjSyOJwFqSx jr7O4nh40Tt4lXJWjC9wU4ZGprrxYECrAjw0AIMDK78AjRAVFxzC1YfsATMu1V0Sm573 +5nzrOwfo7q7BChWuc1J3d34KUImFelI68J8WNP4vpp48Vuo/QQGxk2hnUCzjJBdjoXV Eekg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581456; x=1710186256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PfECBT/sLbntByDSR3pn5gaUEK0qP+L5lceYFnDxHOg=; b=NxIe06PXcp67BWzSsAOKzLUTfQIngO1koorC+iem0vOoCE3mKgCg8bOs9QZPU4UrSS IemL9fIV8BkHJF2L6/64Hyb7zmww0aBvjFNckcfSHJC3lYo9MhX/NhOBrGfUwd1+9OiB ViTKGMpEIBDt5aTUFTFESoyRO7sZEx9nPBZbtjiq90bnZpExTni4YIZ0oNDd+bSv6sH7 8S6kJQ7++Rd/oLbDpWUJVqINGgAWZn1qV72aNcgQwexpgmUY3AYvWiZqxZi3p1RfwMsj 838bRB8qN/WlL+XWg+dI+zJlM4RsNXs3iXWTmY/28iSCHZaPp66e0YyT2UUWJQX0Vfrq LjgQ== X-Forwarded-Encrypted: i=1; AJvYcCVFrwnsdTVyP6/vSjX+SaidKh0DtCsYzMqDyUAGqGJPnKX54rBeirEVKLVY0hciDg22oDYQq/EcSVzgRfe42ZsvsNz2h/eYFpnC X-Gm-Message-State: AOJu0YwM0ruhDoN2ha/397IEwO6QFDQAXbxp6z8MXkxO0HQDCo/R9f3r wjJVqG7B3DzJJf5DIs6zwyHZeJWgtwyZIdCadH/4Sc/TifRRH+D1 X-Google-Smtp-Source: AGHT+IH57Jlo7NmUoCdrFXG2iZYWUpRvZrlyGp9KTCdn+o1CPYk8N74rPmo/Nkf3DTxZgvVi/v4qvQ== X-Received: by 2002:a25:f44c:0:b0:dcd:4257:45e6 with SMTP id p12-20020a25f44c000000b00dcd425745e6mr6725453ybe.35.1709581456146; Mon, 04 Mar 2024 11:44:16 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:15 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 06/13] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Mon, 4 Mar 2024 11:34:01 -0800 Message-ID: <20240304194331.1586191-7-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add (file/memory backed) host backend, all the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, following supports are added: 1. Add a new property to type3 device "volatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all dc regions share one host backend. 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region; 4. Fix dvsec range registers to include DC regions. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 16 ++- hw/mem/cxl_type3.c | 189 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 4 + 3 files changed, 170 insertions(+), 39 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 853dadba39..8309f27a2b 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -622,7 +622,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -636,7 +637,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -793,7 +795,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -835,9 +838,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1179,7 +1184,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c045fee32d..2b380a260b 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -45,7 +45,8 @@ enum { static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, int dsmad_handle, uint64_t size, - bool is_pmem, uint64_t dpa_base) + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { g_autofree CDATDsmas *dsmas = NULL; g_autofree CDATDslbis *dslbis0 = NULL; @@ -61,7 +62,8 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, .DPA_length = size, }; @@ -149,12 +151,13 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } @@ -176,21 +179,55 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) pmr_size = memory_region_size(nonvolatile_mr); } + if (ct3d->dc.num_regions) { + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } else { + return -EINVAL; + } + } + table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, - false, 0); + false, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - pmr_size, true, base); + pmr_size, true, false, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + int i; + uint64_t region_base = vmr_size + pmr_size; + + /* + * TODO: we assume the dynamic capacity to be volatile for now, + * non-volatile dynamic capacity will be added if needed in the + * future. + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + false, true, region_base); + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -300,11 +337,24 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_hi = ct3d->hostpmem->size >> 32; range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + if (ct3d->dc.host_dc) { + range2_size_hi = ct3d->dc.host_dc->size >> 32; + range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); + } + } else { + range1_size_hi = ct3d->dc.host_dc->size >> 32; + range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | + (ct3d->dc.host_dc->size & 0xF0000000); } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -579,11 +629,27 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) { int i; uint64_t region_base = 0; - uint64_t region_len = 2 * GiB; - uint64_t decode_len = 2 * GiB; + uint64_t region_len; + uint64_t decode_len; uint64_t blk_size = 2 * MiB; CXLDCRegion *region; MemoryRegion *mr; + uint64_t dc_size; + + mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(mr); + region_len = DIV_ROUND_UP(dc_size, ct3d->dc.num_regions); + + if (region_len * ct3d->dc.num_regions > dc_size) { + error_setg(errp, "host backend size must be multiples of region len"); + return false; + } + if (region_len % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + decode_len = region_len; if (ct3d->hostvmem) { mr = host_memory_backend_get_memory(ct3d->hostvmem); @@ -605,6 +671,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region->flags = 0; region_base += region->len; + ct3d->dc.total_capacity += region->len; } return true; @@ -614,7 +681,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -678,9 +746,41 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } - if (!cxl_create_dc_regions(ct3d, errp)) { - error_setg(errp, "setup DC regions failed"); - return false; + ct3d->dc.total_capacity = 0; + if (ct3d->dc.num_regions) { + MemoryRegion *dc_mr; + char *dc_name; + + if (!ct3d->dc.host_dc) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + /* + * TODO: set dc as volatile for now, non-volatile support can be added + * in the future if needed. + */ + memory_region_set_nonvolatile(dc_mr, false); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } } return true; @@ -772,6 +872,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -790,6 +893,9 @@ static void ct3_exit(PCIDevice *pci_dev) pcie_aer_exit(pci_dev); cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -868,16 +974,24 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + /* Do we want dc_size to be dc_mr->size or not?? */ + dc_size = ct3d->dc.total_capacity; } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -885,19 +999,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -979,6 +1092,8 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1045,33 +1160,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = ct3d->dc.total_capacity; + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index f82d018422..265679302c 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,10 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; } dc; From patchwork Mon Mar 4 19:34:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581146 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4214379DD8 for ; Mon, 4 Mar 2024 19:44:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581461; cv=none; b=aPZZKTtEkTm14yqMW8+7+NU+yofzeM32J2yW2X6j7iJ6q1sVGffo/bNXs7izOtB7Z7uecf6W/gf0sUEOkJfk/G2Nz3rA2Cz0hCC4njYuG7X4X8XOyXFfw6PYtfw+OBwVBVWm2SSUVUgGAvvUXVn+oX/DEncLsL3pHEb7QgJ4hEc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581461; c=relaxed/simple; bh=5qNrTOKGQ+usnNUjS3Z59MU3P0kLLq50tv/qxYua/JM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=havWlmNZ6Dj+W1/m+gxUm6BNEGT5H5M1BotjT68M/tWTYqY5W6nlBlJIZr4DyJ7MqeUTr0zbkSd8oEopruzyGJXmhiWNIQ2RHsKhyKTv363X18zhGS+XqzH3yjX4i+SvqokKBDS6gZaeEFb0BgH6G/isYLO5bOBJQX13a7Foxxw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JUHnxfgw; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JUHnxfgw" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-60978e6f9a3so47680447b3.3 for ; Mon, 04 Mar 2024 11:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581459; x=1710186259; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g8lPD938tgQqBzbyLXCNS0GDAHx1dZoU8O4sZyz4+ao=; b=JUHnxfgwXblW98g2dR86brciHYVtUuNOitIgWyYV1pENXoNXzJa/nHt69e1Gs3RwK3 FtwzF2uAoGxBQn5jlpu4OtaVbiXjaH8q/QLCp0zE0FxHTN/H89a28N4uU+HR010TlcKP 4fUaFBRmd4gy2XGGQBDrYHXklBv+S4gAGi500Zy0awp1D1nJa5POTLqVOGVP2UetcP6u rXaO+4/U4cRA15m32refoUnGvIvltpDaxt2n6lYnVE0TcsRfUQwppoPn1n0ExPSRAHsD tDPyCHEfmmj5ntedIBT6Y1NkShTcsz3pIOIkDp6g5ZtdUOOAh3FyodTuItEwcgm9lu5/ 9tEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581459; x=1710186259; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g8lPD938tgQqBzbyLXCNS0GDAHx1dZoU8O4sZyz4+ao=; b=jWVe+LA5Dgz+ycbHSl6K3k8B4fwDaJxll63icNR9/DN67XfsbmqTZnr6B7gab4WsVp iYsJQifHswm34HtQaIWxPbNOyPqhxdKf5gRO0X1EZPu0id+zeMqQlkWCZ5FV2ovFUlKk iT/xK6Y3/RXxVYZBFQoafiErdWcdb90ew0jfzSd+31EXcJwbxphGL6VEbchDjjPs4VHO zr9JNLJRHSyHJqTxzmfOeOCysz14kCeDMKolyY+ZVFbDq4xDnHv/NPcJAiLIknikE5wo PmfR8znPKB0QsFF2neaUlC6qEq0RnJ3MJ3b7oogudjF/2obBSumvAxjHS3q5RSjRarwU E3XA== X-Forwarded-Encrypted: i=1; AJvYcCV4E2eXIE7MQR2Atz8SnBdGGTUszV9eFqHF1aoHpIJgQPkeNGH9+45wFJxnYWUUbQavqCit7xpSJMQPwnsd0lfs9GtJXNFcNvZF X-Gm-Message-State: AOJu0Yx34N0xEpptT5023ERcbfhnznOBp3VTdexursZdvXiwMu5cO5x0 KTvFC+fHLh84z6KOLKCirwYRh6aqDICey30GlWw5ptXdCxuobH0v X-Google-Smtp-Source: AGHT+IGMxwD7M3abEPAmA4B7Id72YrzI5ZSRaZbmHCQNrQ/PXng4IFuOLlXZLROHo70LffTiLhR0wg== X-Received: by 2002:a25:5f48:0:b0:dbf:6267:eba4 with SMTP id h8-20020a255f48000000b00dbf6267eba4mr7589407ybm.27.1709581458654; Mon, 04 Mar 2024 11:44:18 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:18 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 07/13] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Mon, 4 Mar 2024 11:34:02 -0800 Message-ID: <20240304194331.1586191-8-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and add get DC extent list mailbox command per CXL.spec.3.1:.8.2.9.9.9.2. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 71 ++++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 22 ++++++++++++ 3 files changed, 93 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8309f27a2b..425b378a2c 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1325,7 +1326,8 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, * to use. */ stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); - stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED - + ct3d->dc.total_extent_count); stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); @@ -1333,6 +1335,70 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.2: + * Get Dynamic Capacity Extent List (Opcode 4801h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED *out = (void *)payload_out; + uint16_t record_count = 0, i = 0, record_done = 0; + uint16_t out_pl_len; + uint32_t start_extent_id = in->start_extent_id; + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out->records[record_done].start_dpa, ent->start_dpa); + stq_le_p(&out->records[record_done].len, ent->len); + memcpy(&out->records[record_done].tag, ent->tag, 0x10); + stw_le_p(&out->records[record_done].shared_seq, ent->shared_seq); + + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1380,6 +1446,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 2b380a260b..102fa8151e 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -673,6 +673,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region_base += region->len; ct3d->dc.total_capacity += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return true; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 265679302c..8148bcc34b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -424,6 +424,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_NUM_REGION 8 +typedef struct CXLDCExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCExtent) node; +} CXLDCExtent; +typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -470,6 +489,9 @@ struct CXLType3Dev { HostMemoryBackend *host_dc; AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ + CXLDCExtentList extents; + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; From patchwork Mon Mar 4 19:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581147 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8636F79DA9 for ; Mon, 4 Mar 2024 19:44:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581464; cv=none; b=PJegKBTXc1nU6+ZgQ3iR4WVYqb97h3N9TBrQXJ9GtFeZTdMORVoSpq2jAfkDWCEJRtfxICqpJ6GUDZKKvP+TeoPIOLDwWVTEN/A6EyDcS8vOmrLc8cWI/iP7CAZqsbWyObHRKfXBYSQWHpRQ1wh7NWckP2ybnK8bB8NrTehxbjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581464; c=relaxed/simple; bh=mwKlr7IKx2OT1XRjWKPDlpfpPWXCkQAf0UoWAp1F91M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AOfdrNb7r93S5XbOBw5SY7RjS4mOxR8fk6lAxVwMbYV2Q3tBpLFCNS+SQx1xWjONezNBfpK+R42tsSDnw8L371TsQXZ/GknUkpp50P/GuzjZNiIi1tme48B+ndKf6IVY43DvFuBcJWNqJNqQAypIs/7vlDF9TQn7/EBazqvluo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jHMtkWeJ; arc=none smtp.client-ip=209.85.219.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jHMtkWeJ" Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-dccb1421bdeso5214117276.1 for ; Mon, 04 Mar 2024 11:44:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581461; x=1710186261; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nsJMpRBsiDoinCTIfmIF1KNqDp8uRG05Rbh+KLifL8w=; b=jHMtkWeJ7c8eIQ9/Wq+jw/pusTBHNoZiAshuWJdE7ljioFY/zZ4KHrfQ3gIBFodGZv D8j+sUkM5FF6OBcMEBXaEAjun+OALMZgsPAwP2z3n2ARAEegBGIJCiFocpoFEc+25UQ5 RiTWRLjaK7wUGQhOUP31nsFIP2UE6rPKaSHPVgDMIbqGMmTVFor47aYAXe5NZIrfXFmV 4nPNKV0H1+6salBMwnFdeOBnoaIoaeXWhjqcn3Fsxq8e3Flqfqz7Ddnz+blKbikaN37p rS8ExYjcuXnwBngMsG82CzPgdGh+032Yz5kZb9TW4rK1ZOlGbEsZdCF139qYqsNTD8ht GKNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581461; x=1710186261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nsJMpRBsiDoinCTIfmIF1KNqDp8uRG05Rbh+KLifL8w=; b=DkmNYMT/iYu/R/SL4EPOXeIxsOpuTzZzlbtNYdeMO8iZtBNVS1YTC/aCjlNB97sFiZ f+6Ys7T1Nt0tos5Sduf6FeDMRUAwvOUCEvN/6SjuwtQJ9gKfubc0OSGokC1gz6dEz4va zoPQwcexfJuXQdC+eWperkCBPKZBuK+4QLQsA839JreMVzI2BlUihRp1W7Pwns0RyqC4 zu4cOFQA9BwTQ65hARaYFiu7Dfo/sSkrTL+yeXcHA/hFjlOtCeDOI/b2imUZ910L5f11 TSfjmlXZo3kqrkfNIL8EkEAf5HsvYh4p2McHA3lFbQHCYiQNuGbxsz2VeYy5HXacKbpg SGpQ== X-Forwarded-Encrypted: i=1; AJvYcCWtRCMa5lxXnm96kgMRoCz+EKuzE+p/HePdERWAFyo+VnjGQpV0LuSsSWpXblhqh2dCVmpjqjc6DfHGBzevS0CCQQmqlvL8787f X-Gm-Message-State: AOJu0YxthnVtkF2liQCeQevDyOgB1xI9bSXye6uAAXr1caFoRYV4+Qay RnbZtm142ErzB9+SqM8e49yp34iFa/jROlpJgimTKZMgKMjZ4vVT X-Google-Smtp-Source: AGHT+IEywjb671TDLGTuxKZsMTEqBabdGbzhbtW53K0+2HFJPtOshNZi3a6jONCN9Wy4Ur+BDkvDlw== X-Received: by 2002:a05:6902:1890:b0:dcd:49d0:eece with SMTP id cj16-20020a056902189000b00dcd49d0eecemr9446480ybb.60.1709581461549; Mon, 04 Mar 2024 11:44:21 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:21 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 08/13] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Mon, 4 Mar 2024 11:34:03 -0800 Message-ID: <20240304194331.1586191-9-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per CXL spec 3.1, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.9.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.9.9.4. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 310 ++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 12 ++ include/hw/cxl/cxl_device.h | 4 + 3 files changed, 326 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 425b378a2c..8c59635a9f 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -85,6 +85,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1399,6 +1401,308 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + int i; + CXLDCRegion *region = &ct3d->dc.regions[0]; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.1 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + if (dpa + len > region->base + region->len) { + return NULL; + } + return region; + } + } + + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCExtent *extent; + + extent = g_new0(CXLDCExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent) +{ + QTAILQ_REMOVE(list, extent, node); + g_free(extent); +} + +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCRegion *region = &ct3d->dc.regions[0]; + CXLDCRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->base + lastregion->len - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.3: Add Dynamic Capacity Response (Opcode 4802h) + * An extent is added to the extent list and becomes usable only after the + * response is processed successfully + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_SUCCESS; + } + + /* Adding extents causes exceeding device's extent tracking ability. */ + if (in->num_entries_updated + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + return CXL_MBOX_RESOURCES_EXHAUSTED; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + /* + * Check if the DPA range of the to-be-added extent overlaps with + * existing extent list maintained by the device. + */ + QTAILQ_FOREACH(ent, extent_list, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + return CXL_MBOX_INVALID_PA; + /* Overlapping one end of the other */ + } else if ((dpa < ent->start_dpa + ent->len && + dpa + len > ent->start_dpa + ent->len) || + (dpa < ent->start_dpa && dpa + len > ent->start_dpa)) { + return CXL_MBOX_INVALID_PA; + } + } + + /* + * TODO: we will add a pending extent list based on event log record + * and verify the input response; also, the "More" flag is not + * considered at the moment. + */ + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (Opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + CXLDCExtent *ent; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + bool found = false; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + QTAILQ_FOREACH(ent, extent_list, node) { + /* Found the extent overlapping with */ + if (ent->start_dpa <= dpa && dpa < ent->start_dpa + ent->len) { + if (dpa + len <= ent->start_dpa + ent->len) { + /* + * The incoming extent covers a portion of an extent + * in the device extent list, remove only the overlapping + * portion, meaning + * 1. the portions that are not covered by the incoming + * extent at both end of the original extent will become + * new extents and inserted to the extent list; and + * 2. the original extent is removed from the extent list; + * 3. DC extent count is updated accordingly. + */ + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + uint64_t len1 = dpa - ent_start_dpa; + uint64_t len2 = ent_start_dpa + ent_len - dpa - len; + + /* + * TODO: checking for possible extent overflow, will be + * moved into a dedicated function of detecting extent + * overflow. + */ + if (len1 && len2 && ct3d->dc.total_extent_count == + CXL_NUM_EXTENTS_SUPPORTED) { + return CXL_MBOX_RESOURCES_EXHAUSTED; + } + + found = true; + cxl_remove_extent_from_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent_start_dpa, len1, + NULL, 0); + ct3d->dc.total_extent_count += 1; + } + if (len2) { + cxl_insert_extent_to_extent_list(extent_list, dpa + len, + len2, NULL, 0); + ct3d->dc.total_extent_count += 1; + } + break; + } else { + /* + * TODO: we reject the attempt to remove an extent that + * overlaps with multiple extents in the device for now, + * once the bitmap indicating whether a DPA range is + * covered by valid extents is introduced, will allow it. + */ + return CXL_MBOX_INVALID_PA; + } + } + } + + if (!found) { + /* Try to remove a non-existing extent. */ + return CXL_MBOX_INVALID_PA; + } + } + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1449,6 +1753,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "DCD_ADD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "DCD_RELEASE_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 102fa8151e..dccfaaad3a 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -678,6 +678,16 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) return true; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + CXLDCExtent *ent; + + while (!QTAILQ_EMPTY(&ct3d->dc.extents)) { + ent = QTAILQ_FIRST(&ct3d->dc.extents); + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -874,6 +884,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -895,6 +906,7 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 8148bcc34b..341260e6e4 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -547,4 +547,8 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent); #endif From patchwork Mon Mar 4 19:34:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581148 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAF9F4653A for ; Mon, 4 Mar 2024 19:44:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581468; cv=none; b=t6miCjOCGFXnRsdMQSJh6TrQnWndNPhwpNl+SMffyWNCfO7BM854Yc6wSKkVZsj5ff17mXnWLUdP+KMXHdbfdNceRAGC+EwTZ3CHWkfNKLZaCUqEK9yMuaJ19izQK6WeE5hMmQ025fBbHYgwX3DLM+VKnumMwjdcXWFDYFwTU+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581468; c=relaxed/simple; bh=LSo6oc4LwS+/6ubh53NECIeVJQF1M/NbBbQ/cWccBxA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SArzBVe/YZVKU1gX+wEGlvuLCXSmL2AIpacUiTGWgTU+hbFIhXzSmy3yAulUREF3p7wWAhbToQpzM8Q8JeDGJmNmclZqsEuUcqIZom90ZQnlASqbiOx4v0+5t3EOLGDYbO6jjTGvuu4fBwQ7iTWi9UVduQvJS+S2zhiXtLFoXfc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mWxWr9E4; arc=none smtp.client-ip=209.85.219.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mWxWr9E4" Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-dbed179f0faso4172136276.1 for ; Mon, 04 Mar 2024 11:44:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581466; x=1710186266; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9IBOSIGSvnMADzgZ+mW+5zQVhNDHXh+pawO9uLnLzyc=; b=mWxWr9E4y9ut5cRhkyFn1KcYsyuHK6Ld+w4DtuBMZpheO4AiYuXa9G36/oLZfS22kW H87ux+uc59SCBTAgJ+zyH9p78c2U0TebSG764rCzLX1oSDSc5lJdCecJ93Eforxeo3GC jhyK833VARfOrnjffiuYPJtIe+NMZq4+9BSiU04BYiA8J4lXla7IX97/fMPpLqNJlnxo MPk50iFpXv7sayFS3Nb1K3hyfjXcXGlqbohbSfnPjKEXuA76CBCT6SaraMLzKpGa8tiL xgcM6M6f796rRJl0fdcAODivJBPuPvNbW00xfkafiXpyFIIGsFkQ11XGH0mLxbLFmn4a tpvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581466; x=1710186266; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9IBOSIGSvnMADzgZ+mW+5zQVhNDHXh+pawO9uLnLzyc=; b=NKiyV/Z6CypxT5/kp1JUpYZ/qrKipE4VApPCwb7sJ0FmcRsTL/5gopgZXearaE9MbL PvNvhVoH9dbCZ9RL33QyDrhfs3gnFIX6fV90gRNn/yQJdA6s/GtB4JJOtkWWOgRy0EpU UvgmExeqOgcQqLpIheQhIUBmJYN3hINALFGPlHIMUH+Y3wgjGm9bLqwDE+4CWNtEGR5U vU1tCHymbbkaZjsFlL4FurOWIUnzPHaKBdpf/X+YapYcQy1KkYqfza4OuZIYGIW41PSb O1AGhLtkdLOMRCWkWiom9gTNKOfgJ+8jjokQJV8B+DnFgYLks92zx0cTxH/AtzWKocMn vX6g== X-Forwarded-Encrypted: i=1; AJvYcCUOVYrX1Oz8wAQjyWmLN9SjeUbIAV1mNWDeiMHxaqbSvpmxp0d3JZX1sGLRxf2cR1e/WggeEr0PN8muk3JEEigxkGsLcDUw5MZu X-Gm-Message-State: AOJu0YzXKCmEMDCCKLJX/R75thmZ2l49HMYELY/gTPh0hm7dsGk5ffog RaUt6NgmdIS1X1qgN+GOpEB96NZiuYlwHe5OuX/beKs56hPsnc1lwymSVSvG X-Google-Smtp-Source: AGHT+IHKrxBL4gj+jTPCrIcVjmwDt26gJvL7p07d8UAeOHaLX1Zkbnj3sPAl0YyHB5/As3vhFckawQ== X-Received: by 2002:a5b:10f:0:b0:dc7:4265:1e92 with SMTP id 15-20020a5b010f000000b00dc742651e92mr448450ybx.23.1709581465690; Mon, 04 Mar 2024 11:44:25 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:25 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 09/13] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Mon, 4 Mar 2024 11:34:04 -0800 Message-ID: <20240304194331.1586191-10-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Since fabric manager emulation is not supported yet, the change implements the functions to add/release dynamic capacity extents as QMP interfaces. Note: we skips any FM issued extent release request if the exact extent does not exist in the extent list of the device. We will loose the restriction later once we have partial release support in the kernel. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 0, "len": 134217728 }, { "dpa": 134217728, "len": 134217728 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) look like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "region-id": 0, "extents": [ { "dpa": 134217728, "len": 134217728 } ] } } Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 26 ++-- hw/mem/cxl_type3.c | 245 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 14 +++ include/hw/cxl/cxl_device.h | 6 + include/hw/cxl/cxl_events.h | 18 +++ qapi/cxl.json | 61 ++++++++- 6 files changed, 361 insertions(+), 9 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 8c59635a9f..53ebc526ae 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1405,7 +1405,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1444,7 +1444,7 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1591,16 +1591,28 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, } } - /* - * TODO: we will add a pending extent list based on event log record - * and verify the input response; also, the "More" flag is not - * considered at the moment. - */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents_pending_to_add, node) { + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + break; + } + } + if (!ent) { + return CXL_MBOX_INVALID_PA; + } + + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending_to_add, + ent); cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; } + /* + * TODO: extents_pending_to_add needs to be cleared so the extents not + * accepted can be reclaimed base on spec r3.1: 8.2.9.9.9.3 + */ + return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index dccfaaad3a..e9c8994cdb 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -674,6 +674,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) ct3d->dc.total_capacity += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); return true; } @@ -686,6 +687,12 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) ent = QTAILQ_FIRST(&ct3d->dc.extents); cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } + + while (!QTAILQ_EMPTY(&ct3d->dc.extents_pending_to_add)) { + ent = QTAILQ_FIRST(&ct3d->dc.extents_pending_to_add); + cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending_to_add, + ent); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -1451,7 +1458,8 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ + case CXL_EVENT_LOG_DYNCAP: + return CXL_EVENT_TYPE_DYNAMIC_CAP; default: return -EINVAL; } @@ -1702,6 +1710,241 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.1 Table 8-50: Dynamic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, +} CXLDCEventType; + +/* + * Check whether the exact extent exists in the list + * Return value: the extent pointer in the list; else null + */ +static CXLDCExtent *cxl_dc_extent_exists(CXLDCExtentList *list, + CXLDCExtentRaw *ext) +{ + CXLDCExtent *ent; + + if (!ext || !list) { + return NULL; + } + + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa != ext->start_dpa) { + continue; + } + + /* Found exact extent */ + return ent->len == ext->len ? ent : NULL; + } + + return NULL; +} + +/* + * The main function to process dynamic capacity event. Currently DC extents + * add/release requests are processed. + */ +static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, + CXLDCEventType type, uint16_t hid, + uint8_t rid, + CXLDCExtentRecordList *records, + Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + g_autofree CXLDCExtentRaw *extents = NULL; + uint8_t enc_log; + uint64_t offset, len, block_size; + int i; + int rc; + g_autofree unsigned long *blk_bitmap = NULL; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + rc = ct3d_qmp_cxl_event_log_enc(log); + if (rc < 0) { + error_setg(errp, "Unhandled error log type"); + return; + } + enc_log = rc; + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset; + len = list->value->len; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + num_extents++; + list = list->next; + } + if (num_extents == 0) { + error_setg(errp, "No extents found in the command"); + return; + } + + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Create Extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + CXLDCExtent *ent; + bool skip_extent = false; + + offset = list->value->offset; + len = list->value->len; + + extents[i].start_dpa = offset + dcd->dc.regions[rid].base; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + + if (type == DC_EVENT_RELEASE_CAPACITY || + type == DC_EVENT_FORCED_RELEASE_CAPACITY) { + /* + * if the extent is still pending to be added to the host, + * remove it from the pending extent list, so later when the add + * response for the extent arrives, the device can reject the + * extent as it is not in the pending list. + */ + ent = cxl_dc_extent_exists(&dcd->dc.extents_pending_to_add, + &extents[i]); + if (ent) { + QTAILQ_REMOVE(&dcd->dc.extents_pending_to_add, ent, node); + g_free(ent); + skip_extent = true; + } else if (!cxl_dc_extent_exists(&dcd->dc.extents, &extents[i])) { + /* If the exact extent is not in the accepted list, skip */ + skip_extent = true; + } + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + list = list->next; + if (!skip_extent) { + i++; + } + } + num_extents = i; + + /* + * CXL r3.1 section 8.2.9.2.1.6: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + /* FIXME: for now, validity flag is cleared */ + dCap.validity_flags = 0; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + /* + * FIXME: for now, the "More" flag is cleared as there is only one + * extent associating with each record and tag-based release is + * not supported. + */ + dCap.flags = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (type == DC_EVENT_ADD_CAPACITY) { + cxl_insert_extent_to_extent_list(&dcd->dc.extents_pending_to_add, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_ADD_CAPACITY, 0, + region_id, records, errp); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + qmp_cxl_process_dynamic_capacity(path, CXL_EVENT_LOG_DYNCAP, + DC_EVENT_RELEASE_CAPACITY, 0, + region_id, records, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..d913b11b4d 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,17 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 341260e6e4..b524c5e699 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -490,6 +490,7 @@ struct CXLType3Dev { AddressSpace host_dc_as; uint64_t total_capacity; /* 256M aligned */ CXLDCExtentList extents; + CXLDCExtentList extents_pending_to_add; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -551,4 +552,9 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, CXLDCExtent *extent); +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, + uint64_t len, uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index 5170b8dbf8..38cadaa0f3 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,22 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.1 section Table 8-50: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t validity_flags; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t flags; + uint8_t reserved2[2]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x18]; + uint32_t extents_avail; + uint32_t tags_avail; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 8cc4c72fa9..2645004666 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -19,13 +19,16 @@ # # @fatal: Fatal Event Log # +# @dyncap: Dynamic Capacity Event Log +# # Since: 8.1 ## { 'enum': 'CxlEventLog', 'data': ['informational', 'warning', 'failure', - 'fatal'] + 'fatal', + 'dyncap'] } ## @@ -361,3 +364,59 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset to the start of the region where the extent to be operated +# @len: length of the extent +# +# Since: 9.0 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The device will +# have to acknowledged the acceptance of the extents before they are usable. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to add +# @extents: Extents to add +# +# Since : 9.0 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @region-id: id of the region where the extent to release +# @extents: Extents to release +# +# Since : 9.0 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'region-id': 'uint8', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Mon Mar 4 19:34:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581149 Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2BBA79955 for ; Mon, 4 Mar 2024 19:44:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581470; cv=none; b=Ouqdjcb3pi8fHcANAtWoYPKQdVLe2SBfVFZMvp0xSpnpNR6Va3f4VVczQIL/3RJe5Kgpvip0AsodJzZNKrO+FsF8HXYzi1EO47h2j8U8gF0tlv2sjcOaFKi1KjkemjC8IWt+DSEdnHGNuIUVIJYLdOL8cQwxpfn3P6LFO8rraDI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581470; c=relaxed/simple; bh=mVm0UNgh3ZwpCrXeqT/IHKRH5OFdo9MzIgXnwdgWnak=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DjXuVc9WKnorrmwsnnBgN9f3rGNM/hs1rpV3sgr6sO9f0OUF0hJmaRe5qoBIKWHBphXRkORKGq78BFUCxRKZrgRIRv/ptwldHpnatnGiCErqQqTx/g1SebgsPwFdR+UQ/RPmy0BJhZvOgs1tYDh9xlGE8US0zSOtz/gLBw2iGK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MJL7pwBS; arc=none smtp.client-ip=209.85.219.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MJL7pwBS" Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-dbed0710c74so4372636276.1 for ; Mon, 04 Mar 2024 11:44:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581468; x=1710186268; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zJmbpLHmX91RSD+FS2bbI4ESdtGu0icFFQqbt0j7En8=; b=MJL7pwBSiubmwVnt+Zwshf/PX1EQN0SyMe6BzVesHffztvVIBxaR6gqWgR3KiQ9IQw 1gGgMw5riBalzYuQKKv7/3hXIL175yn2e/0X/4udhJ9pUGO4NJC3mf4L5JF5u1cxI13S 8ktdriTk10Q53pBsI7b+vA8uSdVdfRipmc/JYeB3xlfRnKfAYZuLeAIOFHgTT0fVYkce a0i7AIUXxjb+ffqmLIUy0UnQvELp3rK++GWBmWqAClf43DLoR+mKlOR41gtXjY6hI2OW Gp5EzXNM54FFU8cLdWZaHNKcBFKqojijTfYorX7Dkxv1LMyGzs9Dt1ISpnTBXIPX9Sua TxxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581468; x=1710186268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zJmbpLHmX91RSD+FS2bbI4ESdtGu0icFFQqbt0j7En8=; b=AFoHH0GLWIRw5Ncr/nzw7GibTS1oq93bbjIQ0oYfcnv93BicWoJm6uVVp7pNzr9T7B z+RSJBSP30b1ptScD+ykrTVNA61dJqKvrQO+W2ptCbutKL2oKzteRH2w+eWtRAVqIQ6+ FQIhJelB0B14S3iHveuMczbYl5+ivUacXc8WFdajZckkxKi/5K1fnhcjoZEj1XA6PKEj ZTm282uXvdS7fbT07ZIcWfTTr7q1XN8oWzZsDgVTGFms4NcXTOEIf/kREhRxInK9DLg7 QGdOccfVLfuiPuWaFpuhFWuk5KTCympyrK+PBmo5QAyp1moh9Iib96XF330DZCO8lRtm LjxA== X-Forwarded-Encrypted: i=1; AJvYcCWuogJky1uSwFOPBuzLiLkx3JwdZcWD6CTuBXKPUJMTUUyTM2MBLKGXcoYVv1ByhSjomMjbyFtb1Rnf6f1n+Ef8Pb9xC0DRXVGy X-Gm-Message-State: AOJu0Yxyd5juDajPZg+7nTZCJMNCzibN4T1VcBEnz5QCV2LryZqtT8GZ UcGY+3Y0DeSvUg905kHErhPvXPVTOJgMY3dXm8icX8N3cNmLm/JZ X-Google-Smtp-Source: AGHT+IFgVk4gFsTBw619+d2h77tRgbIEeyXBAg4E1CIdEk27elXaNIecJfX30qCONXyB3gh3/cBbGg== X-Received: by 2002:a5b:644:0:b0:dc2:41de:b744 with SMTP id o4-20020a5b0644000000b00dc241deb744mr7144765ybq.32.1709581468003; Mon, 04 Mar 2024 11:44:28 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:27 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 10/13] hw/mem/cxl_type3: Add dpa range validation for accesses to DC regions Date: Mon, 4 Mar 2024 11:34:05 -0800 Message-ID: <20240304194331.1586191-11-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Not all dpa range in the DC regions is valid to access until an extent covering the range has been added. Add a bitmap for each region to record whether a DC block in the region has been backed by DC extent. For the bitmap, a bit in the bitmap represents a DC block. When a DC extent is added, all the bits of the blocks in the extent will be set, which will be cleared when the extent is released. Signed-off-by: Fan Ni Reviewed-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 4 ++ hw/mem/cxl_type3.c | 76 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 7 ++++ 3 files changed, 87 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 53ebc526ae..b538297bb5 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1606,6 +1606,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } /* @@ -1681,17 +1682,20 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, found = true; cxl_remove_extent_from_extent_list(extent_list, ent); ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, ent_start_dpa, ent_len); if (len1) { cxl_insert_extent_to_extent_list(extent_list, ent_start_dpa, len1, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, ent_start_dpa, len1); } if (len2) { cxl_insert_extent_to_extent_list(extent_list, dpa + len, len2, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa + len, len2); } break; } else { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index e9c8994cdb..c164cf4580 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -672,6 +672,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) region_base += region->len; ct3d->dc.total_capacity += region->len; + region->blk_bitmap = bitmap_new(region->len / region->block_size); } QTAILQ_INIT(&ct3d->dc.extents); QTAILQ_INIT(&ct3d->dc.extents_pending_to_add); @@ -682,6 +683,8 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent; + int i; + CXLDCRegion *region; while (!QTAILQ_EMPTY(&ct3d->dc.extents)) { ent = QTAILQ_FIRST(&ct3d->dc.extents); @@ -693,6 +696,11 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) cxl_remove_extent_from_extent_list(&ct3d->dc.extents_pending_to_add, ent); } + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -924,6 +932,70 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len) to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether the DPA range [dpa, dpa + len) is backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + /* + * if bits between [dpa, dpa + len) are all 1s, meaning the DPA range is + * backed with DC extents, return true; else return false. + */ + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len) to be unbacked and inaccessible. This + * happens when a dc extent is released by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1029,6 +1101,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index b524c5e699..b213149de2 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -450,6 +450,7 @@ typedef struct CXLDCRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCRegion; struct CXLType3Dev { @@ -557,4 +558,10 @@ void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint16_t shared_seq); bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif From patchwork Mon Mar 4 19:34:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581150 Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4235F4653A for ; Mon, 4 Mar 2024 19:44:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581474; cv=none; b=t1gDj1t+6P7gSxTcoBxC1BZIYVhuvm0GR3ufyR9+a6boKHygjyr9KfWvahryJiF14mTUph/owWgbLmao/cmof2qqqu2jSB2Hqa9ZbLquE8A95jO4izpizbrM8joHOUFdKjSPNkvnRoOUb+S5JI4jHeWmniGCIG1qr/h5+vREzGQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581474; c=relaxed/simple; bh=I78tZvzuBy/60MDuBAkpolsxmDLlEQKWKSrv2goItbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X4457d8/Nz5zb8O1x78UEU88qlbu3S4hWvaRB7oNdFg0pqT3HTT3WlFEAe5FbXzCP5Sz/DyRS3AAnyS+MPBmOAS5bwr1ssiygTJpk1PelTE2mQw8g6xRKyMB+McE4vl+JSGlIlEECziqb6KIbuH2Q+j8ue/6QrT083FgTttcvs8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kGFppLTC; arc=none smtp.client-ip=209.85.219.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kGFppLTC" Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-dc238cb1b17so5221359276.0 for ; Mon, 04 Mar 2024 11:44:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581471; x=1710186271; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=283fafKoCAhZyon638Mj48RQ1VuKIm6CVjiYXSW1c5Y=; b=kGFppLTCwEPqMetbWEDacEFi79MX1lWnULV/Iud2om0Fb+MplKL9pLI2omc6A0hwBx HYhJ2URLZcmQj6LQBCQRzzl/Pae3vIz/ns8zcZYAF6YtJbJHIm4G0cGzkM/zZugq2KlI nIpx3WWncVqGUw+OaP+ZhRWb0hUJhR3S/NmPh/nCR6QM72gfu0//CQ1N0nWK4Ev6X5tw d6fDZyIRaoYqLQyASoxEQhWU/V9zmCWMU5OW4VLQ6nrxENGYm/1lvQGij+8NTfmGQ6Qp Horp8nrvWzhNiFKPqNbacrsNrotwYPWZQ/wfeLqV7bDv87PsuvyafipjZH/EEqLmVtsU OJJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581471; x=1710186271; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=283fafKoCAhZyon638Mj48RQ1VuKIm6CVjiYXSW1c5Y=; b=eCtTJzrOsEjjGT0V1MavKFNxcvh1mv5Uq3uv+Vicawh9aUYCRmOxGtl88mHZ20bkvt mgCU+w3wioQiRtrHPPsUhGroA2Fo6CYKI4pz7SM6iQ+eeDs2q38u2zO2I3YiN+A/3dSa LCuFQMhEld1iNkzECkJ3JxInJUwG+V2MwmqOaZTtr396rnRCi29SJfCW2FHDcZVSmRU2 vKN6VhgEQFTbHzkmwbKhs0KtXt7NxwPsJb0be7TLUrWfvHHIoZyyuMeVT4zeOa4Ng9VS PPjAQ3f+2DJL/sZHXL6bpmJPx0BgjHOmlQIq1W3bMWqXFUHCw8D9tosGYr7YnbZwem9y axPw== X-Forwarded-Encrypted: i=1; AJvYcCUkmU3AL/5pmYARagO2aWi2dC2ArbpVd6Z1BlYRZrnz9xs5BWv7cn9wThk+WqA+0nMJOY2NIjqCTsa/CSpyDVWVke1AeCJg62r9 X-Gm-Message-State: AOJu0YznkVV8Hwrl/fJNVaz/5UOejQxhHaJmts1U2NKS9n8AIKNa760r h3I09nuo43eGYAYosBhBUHWOTcMPXlibWfZpUJmI8DuOsYkN9I6t X-Google-Smtp-Source: AGHT+IFK+f2/iwwB2tpKeab2vQ3PeNmoLam8Uf4HcS3wFtyitfX3H5sfkoJ3VHSCHggO5FT0xZVaWw== X-Received: by 2002:a25:7412:0:b0:dcc:79ab:e51a with SMTP id p18-20020a257412000000b00dcc79abe51amr8056756ybc.57.1709581471312; Mon, 04 Mar 2024 11:44:31 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:31 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 11/13] hw/cxl/cxl-mailbox-utils: Add partial and superset extent release mailbox support Date: Mon, 4 Mar 2024 11:34:06 -0800 Message-ID: <20240304194331.1586191-12-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, we extend the extent release mailbox command processing to allow more flexible release. As long as the DPA range of the extent to release is covered by valid extent(s) in the device, the release can be performed. Signed-off-by: Fan Ni --- hw/cxl/cxl-mailbox-utils.c | 211 +++++++++++++++++++++++++++++++++---- 1 file changed, 188 insertions(+), 23 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index b538297bb5..eaff5c4c93 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1617,6 +1617,155 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Return value: the id of the DC region that covers the DPA range + * [dpa, dpa+len) The assumption is that the range is valid and within + * a DC region. + */ +static uint8_t cxl_find_dc_region_id(const CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + int i; + const CXLDCRegion *region; + + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + break; + } + } + return i; +} + +/* + * Copy extent list from src to dst + * Return value: number of extents copied + */ +static uint32_t copy_extent_list(CXLDCExtentList *dst, + const CXLDCExtentList *src) +{ + uint32_t cnt = 0; + CXLDCExtent *ent; + + if (!dst || !src) { + return 0; + } + + QTAILQ_FOREACH(ent, src, node) { + cxl_insert_extent_to_extent_list(dst, ent->start_dpa, ent->len, + ent->tag, ent->shared_seq); + cnt++; + } + return cnt; +} + +/* + * Detect potential extent overflow caused by extent split during processing + * extent release requests, also allow releasing superset of extents where the + * extent to release covers the range of multiple extents in the device. + * Note: + * 1.we will reject releasing an extent if some portion of its rang is + * not covered by valid extents. + * 2.This function is called after cxl_detect_malformed_extent_list so checks + * already performed there will be skipped. + */ +static CXLRetCode cxl_detect_extent_overflow(const CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t nbits, offset; + const CXLDCRegion *region; + unsigned long **bitmaps_copied; + uint64_t dpa, len; + int i, rid; + CXLRetCode ret = CXL_MBOX_SUCCESS; + long extent_cnt_delta = 0; + CXLDCExtentList tmp_list; + CXLDCExtent *ent; + + QTAILQ_INIT(&tmp_list); + copy_extent_list(&tmp_list, &ct3d->dc.extents); + + bitmaps_copied = g_new0(unsigned long *, ct3d->dc.num_regions); + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + nbits = region->len / region->block_size; + bitmaps_copied[i] = bitmap_new(nbits); + bitmap_copy(bitmaps_copied[i], region->blk_bitmap, nbits); + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + rid = cxl_find_dc_region_id(ct3d, dpa, len); + region = &ct3d->dc.regions[rid]; + offset = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + + /* Check whether range [dpa, dpa + len) is covered by valid range */ + if (find_next_zero_bit(bitmaps_copied[rid], offset + nbits, offset) < + offset + nbits) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + + QTAILQ_FOREACH(ent, &tmp_list, node) { + /* Only split within an extent can cause extent count increase */ + if (ent->start_dpa <= dpa && + dpa + len <= ent->start_dpa + ent->len) { + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + uint64_t len1 = dpa - ent_start_dpa; + uint64_t len2 = ent_start_dpa + ent_len - dpa - len; + + extent_cnt_delta += len1 && len2 ? 2 : (len1 || len2 ? 1 : 0); + extent_cnt_delta -= 1; + if (ct3d->dc.total_extent_count + extent_cnt_delta > + CXL_NUM_EXTENTS_SUPPORTED) { + ret = CXL_MBOX_RESOURCES_EXHAUSTED; + goto free_and_exit; + } + + offset = (ent->start_dpa - region->base) / region->block_size; + nbits = ent->len / region->block_size; + bitmap_clear(bitmaps_copied[rid], offset, nbits); + cxl_remove_extent_from_extent_list(&tmp_list, ent); + + if (len1) { + offset = (dpa - region->base) / region->block_size; + nbits = len1 / region->block_size; + bitmap_set(bitmaps_copied[rid], offset, nbits); + cxl_insert_extent_to_extent_list(&tmp_list, + ent_start_dpa, len1, + NULL, 0); + } + + if (len2) { + offset = (dpa + len - region->base) / region->block_size; + nbits = len2 / region->block_size; + bitmap_set(bitmaps_copied[rid], offset, nbits); + cxl_insert_extent_to_extent_list(&tmp_list, dpa + len, + len2, NULL, 0); + } + break; + } + } + } + +free_and_exit: + for (i = 0; i < ct3d->dc.num_regions; i++) { + g_free(bitmaps_copied[i]); + } + g_free(bitmaps_copied); + + while (!QTAILQ_EMPTY(&tmp_list)) { + ent = QTAILQ_FIRST(&tmp_list); + cxl_remove_extent_from_extent_list(&tmp_list, ent); + } + + return ret; +} + /* * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (Opcode 4803h) */ @@ -1644,15 +1793,28 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, return ret; } - for (i = 0; i < in->num_entries_updated; i++) { - bool found = false; + ret = cxl_detect_extent_overflow(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + /* + * After this point, it is guaranteed that the extents in the + * updated extent list to release is valid, that means: + * 1. All extents in the list have no overlaps; + * 2. Each extent belongs to a valid DC region; + * 3. The DPA range of each extent is covered by valid extent + * in the device. + */ + for (i = 0; i < in->num_entries_updated; i++) { dpa = in->updated_entries[i].start_dpa; len = in->updated_entries[i].len; +process_leftover: QTAILQ_FOREACH(ent, extent_list, node) { /* Found the extent overlapping with */ if (ent->start_dpa <= dpa && dpa < ent->start_dpa + ent->len) { + /* Case 1: The to-release extent is subset of ent */ if (dpa + len <= ent->start_dpa + ent->len) { /* * The incoming extent covers a portion of an extent @@ -1669,17 +1831,6 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, uint64_t len1 = dpa - ent_start_dpa; uint64_t len2 = ent_start_dpa + ent_len - dpa - len; - /* - * TODO: checking for possible extent overflow, will be - * moved into a dedicated function of detecting extent - * overflow. - */ - if (len1 && len2 && ct3d->dc.total_extent_count == - CXL_NUM_EXTENTS_SUPPORTED) { - return CXL_MBOX_RESOURCES_EXHAUSTED; - } - - found = true; cxl_remove_extent_from_extent_list(extent_list, ent); ct3d->dc.total_extent_count -= 1; ct3_clear_region_block_backed(ct3d, ent_start_dpa, ent_len); @@ -1700,20 +1851,34 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, break; } else { /* - * TODO: we reject the attempt to remove an extent that - * overlaps with multiple extents in the device for now, - * once the bitmap indicating whether a DPA range is - * covered by valid extents is introduced, will allow it. + * Case 2: the to-release extent overlaps with multiple + * extents, including the superset case */ - return CXL_MBOX_INVALID_PA; + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + uint64_t len1 = dpa - ent_start_dpa; + + cxl_remove_extent_from_extent_list(extent_list, ent); + ct3d->dc.total_extent_count -= 1; + ct3_clear_region_block_backed(ct3d, ent_start_dpa, ent_len); + + if (len1) { + cxl_insert_extent_to_extent_list(extent_list, + ent_start_dpa, len1, + NULL, 0); + ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, ent_start_dpa, len1); + } + /* + * processing the portion of the range following current + * extent + */ + len = dpa + len - ent_start_dpa - ent_len; + dpa = ent_start_dpa + ent_len; + goto process_leftover; } } } - - if (!found) { - /* Try to remove a non-existing extent. */ - return CXL_MBOX_INVALID_PA; - } } return CXL_MBOX_SUCCESS; From patchwork Mon Mar 4 19:34:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581151 Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EDA279DC1 for ; Mon, 4 Mar 2024 19:44:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581476; cv=none; b=ITla4errsU+J+Vv4KycCu7n3CgsqmvtWNcIi4Jzpcn3S3l/TKVwIXfx0wP+Y/2DazMke4Os7ro657KFazzE0pM847/2M2r5WuswLvkAcUNOPSLc4XdWch7gCVSipeJ3f2AbcilbQNFQrKI4J2jeKig12AeArgvu3PjeP1eTcHSQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581476; c=relaxed/simple; bh=yXmDFhk1WuzmTEaSI2rn7rAngNLiKIOtu3hJ1D5oISk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S0MdyUBOYbUohOWnGiHvVP6Xj3gFZ5UXjjtNFoQlMePuy7WYSOLteRVBOvdozcgrmr3sNYWnN0BNs28TQuFHTDHb1kLtzjqW+zdN26VuUOu1WO/196KVNntIp/SsDWgA+GKvVNKZSlYgosjHgNyey90d+aBDefj5gA84CX7h3RA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VqrtouZ1; arc=none smtp.client-ip=209.85.219.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VqrtouZ1" Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-db3a09e96daso4303950276.3 for ; Mon, 04 Mar 2024 11:44:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581473; x=1710186273; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AaOcSj55m2vuHo7NOezYbOQrG1G43UvBtyGzB+x7te4=; b=VqrtouZ19ZibmHZZPBBfNVwGbsURSs5KGNT0PInroYAzVB+BtW03LxMonMd3ym5yXy Zq76hBobgQtKJqTxz1LvOFj0BFQK7g6k0Pt0mR1OO7hWOIozdCAB5/kJBPYgLUSatElS vPJIXOiKXqzXXxvZcy7uhHXa6d72eHVuICm+oKRDz7VAfm0lze86ZPF8+Tp8G42YrDTr 5hHnQLdCz+cEQlTSWONCwefWZnihSPSJ8D5HowLA4zXYVXmu6kG1xg8rNSvdaUdqEQ16 zbAGDc+TWn7MMWuOfm9skjTw9Fsru/eDnPubmE5Wl5pEieHBO/1pVVWYDE1+y6v/zgc/ wZuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581473; x=1710186273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AaOcSj55m2vuHo7NOezYbOQrG1G43UvBtyGzB+x7te4=; b=UcsCUk1Y3f4ExoI6YJ5OEKZzOqWvzp9zoE5dfCXxz6QNQfW4zmbUjFX6e+4eLgPhq6 TiJharbktXAaIZTj2d3YFiQNFW1UxwtSNdvB6WdDckycvBaiQHFEDGngxHMTwPgJtEqX nRtvBLkCarWh6N4KTb0bOtxcA8H2OUbncWd7lQS+eH62z/7MHP2LYXr/6MpTMWes3GK6 33O/EQKtPYEYNtWyvbaAwPCA33p4orOID7fJu5kV3vAGfsmydyNo3j7Gs3y8TTIlVK5s i/N30kCM4svarAgktLFPZeIFgPjEViSgfW4M0J4xPm6oY9oKNwaCqzWku5J1dMUsUN3r 8PfQ== X-Forwarded-Encrypted: i=1; AJvYcCWzC7KgOUZPSYd2lmOr/eog8QRDvgN9K3WlZ6e5CbPAOiF+5lpkReqa6soDgHGrn8E8DJ3uGrDy//b8QRnHhkyoR978Hs9wFkU4 X-Gm-Message-State: AOJu0Yw50gVcwxCWjwwHHeJ0OO9eH9v+JSzVCJpDwx3Zqne200LmAV81 WD2EZ5wTvmNjg6+UISG12/35rqjR6wqVSgEiEYjQaLmhinQinP2Grjx+R9tL X-Google-Smtp-Source: AGHT+IGAUQ4G1g2oUYmUyuIpiZpE/G0LLDXICATQRHvrTwniiC32r7Y2rbFBwsEBLZHMsINmZ5Ysdg== X-Received: by 2002:a5b:cd1:0:b0:dc2:1f53:3a4f with SMTP id e17-20020a5b0cd1000000b00dc21f533a4fmr7160003ybr.5.1709581473236; Mon, 04 Mar 2024 11:44:33 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:33 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 12/13] hw/mem/cxl_type3: Allow to release partial extent and extent superset in QMP interface Date: Mon, 4 Mar 2024 11:34:07 -0800 Message-ID: <20240304194331.1586191-13-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Before the change, the QMP interface used for add/release DC extents only allows to release extents that exist in either pending-to-add list or accepted list in the device, which means the DPA range of the extent must match exactly that of an extent in either list. Otherwise, the release request will be ignored. With the change, we relax the constraints. As long as the DPA range of the extent to release is covered by extents in one of the two lists mentioned above, we allow the release. Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 110 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 89 insertions(+), 21 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c164cf4580..5bd64e604e 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1802,28 +1802,79 @@ typedef enum CXLDCEventType { } CXLDCEventType; /* - * Check whether the exact extent exists in the list - * Return value: the extent pointer in the list; else null + * Testing whether the DPA range [dpa, dpa + len) is covered by + * extents in the list. */ -static CXLDCExtent *cxl_dc_extent_exists(CXLDCExtentList *list, - CXLDCExtentRaw *ext) +static bool cxl_test_dpa_range_covered_by_extents(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) { CXLDCExtent *ent; - if (!ext || !list) { - return NULL; + if (!list) { + return false; } - QTAILQ_FOREACH(ent, list, node) { - if (ent->start_dpa != ext->start_dpa) { - continue; - } + while (len) { + bool has_leftover = false; - /* Found exact extent */ - return ent->len == ext->len ? ent : NULL; + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa <= dpa && dpa < ent->start_dpa + ent->len) { + if (dpa + len <= ent->start_dpa + ent->len) { + return true; + } else { + len = dpa + len - ent->start_dpa - ent->len; + dpa = ent->start_dpa + ent->len; + has_leftover = true; + break; + } + } + } + if (!has_leftover) { + break; + } } + return false; +} + +/* + * Remove all extents whose DPA range has overlaps with the DPA range + * [dpa, dpa + len) from the list, and delete the overlapped portion. + * Note: + * 1. If the removed extents is fully within the DPA range, delete the extent; + * 2. Otherwise, keep the portion that does not overlap, insert new extents to + * the list if needed for the un-coverlapped part. + */ +static void cxl_delist_extent_by_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; - return NULL; +process_leftover: + QTAILQ_FOREACH(ent, list, node) { + if (ent->start_dpa <= dpa && dpa < ent->start_dpa + ent->len) { + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + uint64_t len1 = dpa - ent_start_dpa; + + cxl_remove_extent_from_extent_list(list, ent); + if (len1) { + cxl_insert_extent_to_extent_list(list, ent_start_dpa, + len1, NULL, 0); + } + + if (dpa + len <= ent_start_dpa + ent_len) { + uint64_t len2 = ent_start_dpa + ent_len - dpa - len; + if (len2) { + cxl_insert_extent_to_extent_list(list, dpa + len, + len2, NULL, 0); + } + } else { + len = dpa + len - ent_start_dpa - ent_len; + dpa = ent_start_dpa + ent_len; + goto process_leftover; + } + } + } } /* @@ -1915,8 +1966,8 @@ static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, list = records; extents = g_new0(CXLDCExtentRaw, num_extents); while (list) { - CXLDCExtent *ent; bool skip_extent = false; + CXLDCExtentList *extent_list; offset = list->value->offset; len = list->value->len; @@ -1933,15 +1984,32 @@ static void qmp_cxl_process_dynamic_capacity(const char *path, CxlEventLog log, * remove it from the pending extent list, so later when the add * response for the extent arrives, the device can reject the * extent as it is not in the pending list. + * Now, we can handle the case where the extent covers the DPA + * range of multiple extents in the pending_to_add list. + * TODO: we do not allow the extent covers range of extents in + * pending_to_add list and accepted list at the same time for now. */ - ent = cxl_dc_extent_exists(&dcd->dc.extents_pending_to_add, - &extents[i]); - if (ent) { - QTAILQ_REMOVE(&dcd->dc.extents_pending_to_add, ent, node); - g_free(ent); + extent_list = &dcd->dc.extents_pending_to_add; + if (cxl_test_dpa_range_covered_by_extents(extent_list, + extents[i].start_dpa, + extents[i].len)) { + cxl_delist_extent_by_dpa_range(extent_list, + extents[i].start_dpa, + extents[i].len); + } else if (!ct3_test_region_block_backed(dcd, extents[i].start_dpa, + extents[i].len)) { + /* + * If the DPA range of the extent is not covered by extents + * in the accepted list, skip + */ skip_extent = true; - } else if (!cxl_dc_extent_exists(&dcd->dc.extents, &extents[i])) { - /* If the exact extent is not in the accepted list, skip */ + } + } else if (type == DC_EVENT_ADD_CAPACITY) { + extent_list = &dcd->dc.extents; + /* If the extent is ready pending to add, skip */ + if (cxl_test_dpa_range_covered_by_extents(extent_list, + extents[i].start_dpa, + extents[i].len)) { skip_extent = true; } } From patchwork Mon Mar 4 19:34:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13581152 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAF1579DA9 for ; Mon, 4 Mar 2024 19:44:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581479; cv=none; b=hNivzozoJNj62loP4H5e2iTFy7BX4TWr6yg+EdRKYFUlRQUFyL191iL2yjiqZ82hbi3Me4tp2kUgJnHmO2KngfpJqKGB36B1tgxbdsthZtVUrwoJ3zFZkDbwwgUJ6Xrl4rCzqy36rsFWpz6fIKjuh64mZau55yfIhETjZV2wzP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709581479; c=relaxed/simple; bh=sxiGml8sSyPhzFwRAZ2KX5EroGcFTa5vXUesyH6hOR4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GO4v2zUVpIV8U9gEp2FLwRPf+4kclNPjXVIA3ypBlFMPB3zlMGMAr7F2o6YosGoSpW5oTT1bWXcHFOaA5ShyJKLYsuveqZKp+tehiSoqen6glrjsWn/5L8zGby1CJzQD6T+SR8J6GsIBOxS10wnX3PXbh+FaEnZuo6LGPluL7NE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XB5WgSXY; arc=none smtp.client-ip=209.85.219.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XB5WgSXY" Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-dc742543119so4933308276.0 for ; Mon, 04 Mar 2024 11:44:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709581476; x=1710186276; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TKU7EMmrCEGIbEXLnUmzxmQlpe4p33ksgB+5YiuAYXs=; b=XB5WgSXYxtLITOXoGonqGBTz6Q8G+owJNeK7Xss1yQfPnnK/a2LafMTq3oa9ZyU4tO 9yY6m/wLbvGw0oj4HlsRCkXrHXKTut3XBiIw+p5FLIJtk2Pq4QMY/ATj+aSSj1rpP3em FD496hldtYH75s5x0abhA/n5zVNnVBjMuOLecKwiq3xgetaRlwChBm2+Gh3dZr2r7p+o xG4VxQKPA+SNMp2jKPQeCGQVWOr7XJC/D1jMuV6FPNbgY9W56L4ATzSX+f3WjTTkBvbk GOMAaC1bh5/nKFK7lt/ioxVXAKqi/utKHRYEUf32nkJvr58DMTzJyoGdkgLQy3u68qCb j7WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709581476; x=1710186276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TKU7EMmrCEGIbEXLnUmzxmQlpe4p33ksgB+5YiuAYXs=; b=cIGFCE0K1udoIRs4QH+rYzVVGN0zM1n6koUqw1O7vMMYY+Am+/vGDy6+e4VLIGtkt+ 0xLb0WFh/86eTs23CqO5gEKrYMnfDyC7iLtNXMFg5QxnNXcVscXWnUtn0QJeFPz1/OnN PduFVd+J9b12SqZkzTUZUHv/OCjbpaLFO2QHhMcXb48I4NVIdSqfHtT9ZAxfEsMp083M pbITKhmtJZfSgUefya3720KO1tUPD9XAtoOuwqzOsTiRl0XCRUv8kxDg9krE/ajZR26h 4l+gWqhr0Sm7ACaQQfXVHJI5RbnhxUBHsVLwAdqNDsfITNRvZBr1OVNJb59/DfajZLyo XgjA== X-Forwarded-Encrypted: i=1; AJvYcCWu3PZZCP/9Nuhct+qvkXnG2nS600pIaMsic0J12ODzfe0K1Z1uQsPAEjIKC7etEw0v1fb1Mj+BjyJvCcS3qxFbrjFCD12Gq0eY X-Gm-Message-State: AOJu0Yy8w4SQOSBLdZsW/P8/yHWOTPT4r8MhKVTLtOnzudD3z3Zit3NB EXHwQYfU4VHPH2WX0GBq+Dl8vjfmgAWWc0QpLjf1JAMbX6YhAWD6 X-Google-Smtp-Source: AGHT+IGJaenBrz1oSac79n8j93VzbYMt0qiN1tC/RpBnCVT3yoX1a0NYTi0pRDY7jv1mKlEMaKgrEw== X-Received: by 2002:a25:2690:0:b0:dc6:ff12:13d7 with SMTP id m138-20020a252690000000b00dc6ff1213d7mr5343408ybm.60.1709581476386; Mon, 04 Mar 2024 11:44:36 -0800 (PST) Received: from localhost.localdomain ([50.205.20.42]) by smtp.gmail.com with ESMTPSA id h1-20020a255f41000000b00dc62edd58dasm2282646ybm.40.2024.03.04.11.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 11:44:36 -0800 (PST) From: nifan.cxl@gmail.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v5 13/13] qapi/cxl.json: Add QMP interfaces to print out accepted and pending DC extents Date: Mon, 4 Mar 2024 11:34:08 -0800 Message-ID: <20240304194331.1586191-14-nifan.cxl@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240304194331.1586191-1-nifan.cxl@gmail.com> References: <20240304194331.1586191-1-nifan.cxl@gmail.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, we add the following two QMP interfaces to print out extents information in the device, 1. cxl-display-accepted-dc-extents: print out the accepted DC extents in the device; 2. cxl-display-pending-to-add-dc-extents: print out the pending-to-add DC extents in the device; The output is appended to a file passed to the command and by default it is /tmp/dc-extent.txt. Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 80 ++++++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3_stubs.c | 12 ++++++ qapi/cxl.json | 32 ++++++++++++++++ 3 files changed, 124 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 5bd64e604e..6a08e7ae40 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -2089,6 +2089,86 @@ void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, region_id, records, errp); } +static void cxl_dcd_display_extent_list(const CXLType3Dev *dcd, const char *f, + bool accepted_list, Error **errp) +{ + const CXLDCExtentList *list; + CXLDCExtent *ent; + FILE *fp = NULL; + int i = 0; + + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + if (!f) { + fp = fopen("/tmp/dc-extent.txt", "a+"); + } else { + fp = fopen(f, "a+"); + } + + if (!fp) { + error_setg(errp, "Open log file failed"); + return; + } + if (accepted_list) { + list = &dcd->dc.extents; + fprintf(fp, "Print accepted extent info:\n"); + } else { + list = &dcd->dc.extents_pending_to_add; + fprintf(fp, "Print pending-to-add extent info:\n"); + } + + QTAILQ_FOREACH(ent, list, node) { + fprintf(fp, "%d: [0x%lx - 0x%lx]\n", i++, ent->start_dpa, + ent->start_dpa + ent->len); + } + fprintf(fp, "In total, %d extents printed!\n", i); + fclose(fp); +} + +void qmp_cxl_display_accepted_dc_extents(const char *path, const char *f, + Error **errp) +{ + Object *obj; + CXLType3Dev *dcd; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + cxl_dcd_display_extent_list(dcd, f, true, errp); +} + +void qmp_cxl_display_pending_to_add_dc_extents(const char *path, const char *f, + Error **errp) +{ + Object *obj; + CXLType3Dev *dcd; + + obj = object_resolve_path(path, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve path"); + return; + } + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + error_setg(errp, "Path not point to a valid CXL type3 device"); + return; + } + + + dcd = CXL_TYPE3(obj); + cxl_dcd_display_extent_list(dcd, f, false, errp); +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index d913b11b4d..d896758301 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -81,3 +81,15 @@ void qmp_cxl_release_dynamic_capacity(const char *path, uint8_t region_id, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_display_accepted_dc_extents(const char *path, const char *f, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_display_pending_to_add_dc_extents(const char *path, const char *f, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/qapi/cxl.json b/qapi/cxl.json index 2645004666..6f10300ec6 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -420,3 +420,35 @@ 'extents': [ 'CXLDCExtentRecord' ] } } + +## +# @cxl-display-accepted-dc-extents: +# +# Command to print out all the accepted DC extents in the device +# +# @path: CXL DCD canonical QOM path +# @output: path of output file to dump the results to +# +# Since : 9.0 +## +{ 'command': 'cxl-display-accepted-dc-extents', + 'data': { 'path': 'str', + 'output': 'str' + } +} + +## +# @cxl-display-pending-to-add-dc-extents: +# +# Command to print out all the pending-to-add DC extents in the device +# +# @path: CXL DCD canonical QOM path +# @output: path of output file to dump the results to +# +# Since : 9.0 +## +{ 'command': 'cxl-display-pending-to-add-dc-extents', + 'data': { 'path': 'str', + 'output': 'str' + } +}