From patchwork Thu Apr 18 23:10:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635514 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8E7C2AF0F for ; Thu, 18 Apr 2024 23:30:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483005; cv=none; b=XYiIXWTius49AUFOuQjfDgzHcEzwkrxl7eZ9EcBPQyxih/WKVKT5FAnxv8BG/+qZdZU4Pn8/2DijoQ8+Z6o2L0fUnoTsYbvAhpnnl2eBvoP+TF98Ydi7T0drt2w9gJc95g2h9FPWK4mYaMRAiXgOt28jfTumLETvfwM/kS0Ez7s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483005; c=relaxed/simple; bh=wy6mbTMBMsYfurZCiVtXv16F+h19N9tY2TYxi6DUmjc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qS7S2zaTGcQgFuABenvfoboNJ3Xw2tFd0OG6kwKKgRPJL3eIgEYyMDgHgyxTKhFSeZ3rtGqouAlBfZAN2q9DjVk6xiBj3l3eMB7+1n/vuxNDOxvoWW4mWnRd27jXOb4njt+0mM43wQUza3MKecaRpFM8zGtW5VwsTWO8CfmOUb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m75j+bkA; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m75j+bkA" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6ed691fb83eso1299187b3a.1 for ; Thu, 18 Apr 2024 16:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483003; x=1714087803; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xakP/WAiR69d8B5B18gJ85sFdqfiQ0c6HHa8yXwoSyo=; b=m75j+bkAJnVAXiSudvMa1VXhH19h81difEXKoU+T3PTfYGjFH/Pq16BsGieTSVSI48 manV73aavhNqQr44sr4ezk8CHaHH4qHSsNZpE0QqLpQ2WVDy+D8FrO5qMYrPOeZ1UVw0 YRO5n5/QJrWGEXhwbf6J6GakylEiLC33l3CKi+vKimJ+q39gy8Oq5oo2fRVosx+5y8ke b2GHtVpBNKgQi8gUnRMSAxMccthGhQKIiv94/UihaLxr/cwtYm8+NLv5BsH+KVkqn7J+ HsK8GXJWEDQKPt+bqrG51km8mIZBNy2Tt9wMAv5bjdQ66hW/WvKJK7eh4+yD6Bo6bvJc pnuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483003; x=1714087803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xakP/WAiR69d8B5B18gJ85sFdqfiQ0c6HHa8yXwoSyo=; b=q2x6eLavZSWkN4PU7Zh3+dZDT1fgweXxz6rdAD26DWL4AxW/X4Zl1po3ZF4mtU1Bhh MrYdmGwZ9y185kHfhKz+PQbZ5NKs5dz/0uoh5dbq61O1R4y6KMgwgWOGyN12bZYoJukz ybYByCPszOaXLzC7Kq2vqoJRwz+kSip/sw6XLLJdDnLSRND8F/Jm76+ReJLHeECDk7M6 cbxuu79+6IZ8RqPCL9eRBWGje4OP/QeHKEjpf5bR14t/MIWFoT59auHInPdU9z43GEWu eU+ecJPosSloINgWnpkk0LvRN7atr7cGjxMuBFAKBjyFApxVZimKoQjW0DWDBjhIpqEE DjfA== X-Forwarded-Encrypted: i=1; AJvYcCXNbROYqdYC0YwQWrVag0RBFFze41s3mm5f+NX8c9tnt7jDAdpB5sHZwiezWd6ORpwDVwjMbIdLOgRf7Kt82RW3Eyy7wKTyHXJr X-Gm-Message-State: AOJu0YwZib0JYYtNrREZ4yS3zrDAs0UcTjINi5cTZLOkzTWmx3uOQFri YgmJOzYSXSKz5e4oJ0RKacPAdB7DDTHvMdFVEBM3ylhnSiLAPUPs X-Google-Smtp-Source: AGHT+IFTdEgcMmuT3whZaWUvTVxWchLBAzKu3B7PBQT8Sr326EKv/JjWOH3PFNWfM1ggd3BO+9oIvw== X-Received: by 2002:a05:6a00:8c9:b0:6ea:bdbc:6a4 with SMTP id s9-20020a056a0008c900b006eabdbc06a4mr770808pfu.13.1713483002795; Thu, 18 Apr 2024 16:30:02 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.29.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:02 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 01/12] hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output payload of identify memory device command Date: Thu, 18 Apr 2024 16:10:52 -0700 Message-ID: <20240418232902.583744-2-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Based on CXL spec r3.1 Table 8-127 (Identify Memory Device Output Payload), dynamic capacity event log size should be part of output of the Identify command. Add dc_event_log_size to the output payload for the host to get the info. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 4bcd727f4c..ba1d9901df 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -21,6 +21,7 @@ #include "sysemu/hostmem.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) +#define CXL_DC_EVENT_LOG_SIZE 8 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -780,8 +781,9 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, uint16_t inject_poison_limit; uint8_t poison_caps; uint8_t qos_telemetry_caps; + uint16_t dc_event_log_size; } QEMU_PACKED *id; - QEMU_BUILD_BUG_ON(sizeof(*id) != 0x43); + QEMU_BUILD_BUG_ON(sizeof(*id) != 0x45); CXLType3Dev *ct3d = CXL_TYPE3(cci->d); CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; @@ -807,6 +809,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, st24_le_p(id->poison_list_max_mer, 256); /* No limit - so limited by main poison record limit */ stw_le_p(&id->inject_poison_limit, 0); + stw_le_p(&id->dc_event_log_size, CXL_DC_EVENT_LOG_SIZE); *len_out = sizeof(*id); return CXL_MBOX_SUCCESS; From patchwork Thu Apr 18 23:10:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635515 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFB282AF0F for ; Thu, 18 Apr 2024 23:30:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483008; cv=none; b=ZmSbRDyrlE5CiYYHa7zPIxyMxGD0WykVu8hQNjVC8i6MNdApMugtf6Ig3gGBQczG2QADLldqikvvtCLYWkBXMa3GzEug/THBH02DpVWneKwPsA+L4sa9wr+yQN7AsKKBMA1iPg1s3/Tzr/kmkBQkmGDIYYTtqbapKIVjTOFZkXs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483008; c=relaxed/simple; bh=JsASg/fg6c9+nXNE2bbmEEfp6aRNEel3b+xUkUQt6uc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cgIimgKlRAGp89+ln2y+HLsVQtvk+RYXbdJlWJTIX+bJx6KO6FpfeGndYTis7k6V+5fFi/pGp197QkGdyqr0YCkawMaV5DmOSshdmaDJd6Ve7EkbM72threNtIIrsbwihArs0ueLCgAAfS+/dzdPzugNXSsx3AeekK0NVYJ7eeQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CwCK6HsA; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CwCK6HsA" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6ed04c91c46so1449958b3a.0 for ; Thu, 18 Apr 2024 16:30:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483006; x=1714087806; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wAys60q15bX2FDoPastpCKCHWQud2bcFghMDEoFsPeo=; b=CwCK6HsAFjomRXu3zeY9lvJO0sg+ABifMAycYQFG6gURYvNZq69l67NuV5NKCOLwOy ZqcAdWJTVs8Dmefp06uIK6zFUOHEi9DT1eLOl+g94wYDbM6XflhOc09Tb7wSeoMuW1vY vBY+s1SBb01IcJGo10fPman3TZczzlQr+foYUuqZsw2UN1HM4tb+gc4rnjrQn6ewONCJ eXHGnj3olsyfQRc0+duwzxE/QYPki/xvjQsOI3vIJ2qYp5r3Lud4qQcP7oSSCpF5MFho jgshyW53WlsQ/6FLd4WLCEPjcVYD13i1e3n5z9R5LYZGxQe/RN96uPIeB5NSsyHOCDk/ Efzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483006; x=1714087806; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wAys60q15bX2FDoPastpCKCHWQud2bcFghMDEoFsPeo=; b=Bprg9udya+DHd+1GBBow6RicsXx/s+1LXCO9AWhTb1oMkUNHt9o3YKeIOfg7DgyjoC OyaSXfZ9NY0VO0d5HEDg+atp7uotLerYRzLpusCNP3QYmXmmI1KdQkxCn9U93QWsalqs JIkB4+be86lnhCWCMwqedtAyUpeTAn6ANNB3zmCWnbpkpicp5YKDS3bWplL5T/8ETRgA E9aX3oWVu4ox0Bog3pkcMu935mqXaZcL6zxWDqpdRYjjucxkgtLCaTWtBoZrqV6kyWW3 WHCRyBYoE8MT5fAKVX64+mhHhxaXGgsiWJrlotvG/tyJ/ku8HsLo2qd8BFtuIDaYCKXm Apyg== X-Forwarded-Encrypted: i=1; AJvYcCVJVpUXOyrGiKANT5evdx1v5ZcpPKCYeAlnd9s0/r1hppIbCfZph6bg/SSZ7r/NQf8d6ygbULGNB86sTvqQMRp2nvj4W/6vF0ba X-Gm-Message-State: AOJu0YwNMH91+mPHYtEU/NahCMv8r4UC6M+OL8GSLin40UWUy/jZynGB KZFnRbPONUsd2T6+S0yy1UZH62F96o6I11y/cWPt8BHNnHXtiOXt/kkmWw== X-Google-Smtp-Source: AGHT+IH71h5HXU7UEOIDmxSsT9ALHXKk/ZgkOabosSJEp4T4jkfwimaHu3SvhuUftZxT7JeDT5WImg== X-Received: by 2002:a05:6a00:3d04:b0:6ea:c4e5:a252 with SMTP id lo4-20020a056a003d0400b006eac4e5a252mr691623pfb.5.1713483006087; Thu, 18 Apr 2024 16:30:06 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:05 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 02/12] hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative and mailbox command support Date: Thu, 18 Apr 2024 16:10:53 -0700 Message-ID: <20240418232902.583744-3-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per cxl spec r3.1, add dynamic capacity region representative based on Table 8-165 and extend the cxl type3 device definition to include DC region information. Also, based on info in 8.2.9.9.9.1, add 'Get Dynamic Capacity Configuration' mailbox support. Note: we store region decode length as byte-wise length on the device, which should be divided by 256 * MiB before being returned to the host for "Get Dynamic Capacity Configuration" mailbox command per specification. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 96 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 16 +++++++ 2 files changed, 112 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index ba1d9901df..49c7944d93 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -22,6 +22,8 @@ #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 +#define CXL_NUM_EXTENTS_SUPPORTED 512 +#define CXL_NUM_TAGS_SUPPORTED 0 /* * How to add a new command, example. The command set FOO, with cmd BAR. @@ -80,6 +82,8 @@ enum { #define GET_POISON_LIST 0x0 #define INJECT_POISON 0x1 #define CLEAR_POISON 0x2 + DCD_CONFIG = 0x48, + #define GET_DC_CONFIG 0x0 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1238,6 +1242,88 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.1: Get Dynamic Capacity Configuration + * (Opcode: 4800h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint8_t region_cnt; + uint8_t start_rid; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint8_t num_regions; + uint8_t regions_returned; + uint8_t rsvd1[6]; + struct { + uint64_t base; + uint64_t decode_len; + uint64_t region_len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; + uint8_t rsvd2[3]; + } QEMU_PACKED records[]; + } QEMU_PACKED *out = (void *)payload_out; + struct { + uint32_t num_extents_supported; + uint32_t num_extents_available; + uint32_t num_tags_supported; + uint32_t num_tags_available; + } QEMU_PACKED *extra_out; + uint16_t record_count; + uint16_t i; + uint16_t out_pl_len; + uint8_t start_rid; + + start_rid = in->start_rid; + if (start_rid >= ct3d->dc.num_regions) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(ct3d->dc.num_regions - in->start_rid, in->region_cnt); + + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + extra_out = (void *)(payload_out + out_pl_len); + out_pl_len += sizeof(*extra_out); + assert(out_pl_len <= CXL_MAILBOX_MAX_PAYLOAD_SIZE); + + out->num_regions = ct3d->dc.num_regions; + out->regions_returned = record_count; + for (i = 0; i < record_count; i++) { + stq_le_p(&out->records[i].base, + ct3d->dc.regions[start_rid + i].base); + stq_le_p(&out->records[i].decode_len, + ct3d->dc.regions[start_rid + i].decode_len / + CXL_CAPACITY_MULTIPLIER); + stq_le_p(&out->records[i].region_len, + ct3d->dc.regions[start_rid + i].len); + stq_le_p(&out->records[i].block_size, + ct3d->dc.regions[start_rid + i].block_size); + stl_le_p(&out->records[i].dsmadhandle, + ct3d->dc.regions[start_rid + i].dsmadhandle); + out->records[i].flags = ct3d->dc.regions[start_rid + i].flags; + } + /* + * TODO: Assign values once extents and tags are introduced + * to use. + */ + stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); + stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1282,6 +1368,11 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { cmd_media_clear_poison, 72, 0 }, }; +static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { + [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", + cmd_dcd_get_dyn_cap_config, 2, 0 }, +}; + static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 }, [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS", @@ -1487,7 +1578,12 @@ void cxl_initialize_mailbox_swcci(CXLCCI *cci, DeviceState *intf, void cxl_initialize_mailbox_t3(CXLCCI *cci, DeviceState *d, size_t payload_max) { + CXLType3Dev *ct3d = CXL_TYPE3(d); + cxl_copy_cci_commands(cci, cxl_cmd_set); + if (ct3d->dc.num_regions) { + cxl_copy_cci_commands(cci, cxl_cmd_set_dcd); + } cci->d = d; /* No separation for PCI MB as protocol handled in PCI device */ diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index a5f8e25020..e839370266 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -422,6 +422,17 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 +#define DCD_MAX_NUM_REGION 8 + +typedef struct CXLDCRegion { + uint64_t base; /* aligned to 256*MiB */ + uint64_t decode_len; /* aligned to 256*MiB */ + uint64_t len; + uint64_t block_size; + uint32_t dsmadhandle; + uint8_t flags; +} CXLDCRegion; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -454,6 +465,11 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + struct dynamic_capacity { + uint8_t num_regions; /* 0-8 regions */ + CXLDCRegion regions[DCD_MAX_NUM_REGION]; + } dc; }; #define TYPE_CXL_TYPE3 "cxl-type3" From patchwork Thu Apr 18 23:10:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635516 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2D2C2AF0F for ; Thu, 18 Apr 2024 23:30:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483010; cv=none; b=EJwkixr1Bbft0iAufZ8KN5UAYVCsEjxNGhvYu8fAssn0q0rNFpNPdtZmZ9MP41T1Is3rC82Dl0bkxbePX3r4xq0vQpqTXjJZZ4yyLGf3WBnr3m7nP+CH/2n7xVhRirwCY2RrTDTIT10Vyc9hCozN/I0NXW32y/iNfjjmXj25f70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483010; c=relaxed/simple; bh=pA0UZ5S09ZRfYx5zbrKTB2Eppma02dLoAJjpBiJ6/zc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SXtvNVrWXGUe7BUahqTzkIXu0Zip1VT63Xi+0LuyDsV6cAhEkRYv5b2tB58PbDn0EbpSDaOA2iGIilUAUVx3otRUoIO0ENq0AGCXVTb4TiJEZ4/PLRgCZ5LSdw9XFvxASUxsU6FJ9ur0wkyz2ZrmbqxRpBtZ+Ge4dqle7mbCG9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bo70e4Gq; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bo70e4Gq" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-5d8b519e438so1114920a12.1 for ; Thu, 18 Apr 2024 16:30:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483008; x=1714087808; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lcorWsBRN7cgnG7Jkx2nUqtNQI6bccmxCNztfvmiAis=; b=bo70e4Gq0gYxaRMmjUOx++Hvlni6hdYJbddHS0iXv5/1neT5jBAfJwDiONUgnl7keC IWnU3444zVmB7y58dai5NXou7dQ3llLv78HBLMihRqflpkzBMscj16tKenFcA6tqy686 kQvTh13P392MvN+TnL8w1gJGR2LbomcdNY5TL/EMwVrB5m5PLPjAApxAoGziNEZHx3Ie yVSmUjS0KSEmw4XXZmsoSkgoj7LHSQuJdTKFR/yTQqof9owWORgoIsovsIidc8r7yqzN moXUMWNAzxpLztwHd72bzR7y6La9dQutQ6KBZWsxyjOurFDfTA9QGaul5oIT58p7iqv7 iRSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483008; x=1714087808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lcorWsBRN7cgnG7Jkx2nUqtNQI6bccmxCNztfvmiAis=; b=nKusod5FapnRYmjr7o2+CLDUMMPFAZab+IfU/0xBZrJtkgVcF5Zb60d5BHYE7fj4Lv gtBHv98kEmpampcVypdJLcGpeZ4WaIgqDuy8AuzkcWJMk3IacRu+8wRwr24SWsBzYJ6R cixrvxIQNUKDNzZsLqTLUvJa+JFvth0b1jMzh6QlpFVl6dx8kkWEdamnPI1T/lP3O2HZ hnTbWA42YN4MA2FKgfaNHO7h1efrHQ7hOun0mLcgKbyfMPvhlGyZER6Y2AUFNa29xnGJ zJWh3/QnO2P7e50zKKaYMaDpwMjUId4L3WE98wiEUoPfJG72I//XSEv3pJRUGImJMluO 2e5Q== X-Forwarded-Encrypted: i=1; AJvYcCXBESF/isQeeNt/iSZAdNq3W4GN8toPwGdnCXWv9/Hu0dhOkZKB8mUr0gZKg7uEHwTLMe4WhDNZfHnRlBg3od3tFpoirJ27jQ8z X-Gm-Message-State: AOJu0Yxie7dkPEubH4U0dOU+A+pX9Dt98/F0+mLMcAJenaTl94ya8tl0 z6z2jwfh5noaP//EMZIl3nxJLRrSsElo3lh4aPPx62tN5XP6+r21apfvIg== X-Google-Smtp-Source: AGHT+IEe2oQ/rY1RuB2G5jd9JSOk0Pj/3msZn0RnPw2T1zzw3M+jGX51KaWwSuwKTz3hTUuooSlAUg== X-Received: by 2002:a05:6a21:3482:b0:1a7:a2f2:c71a with SMTP id yo2-20020a056a21348200b001a7a2f2c71amr963853pzb.0.1713483008094; Thu, 18 Apr 2024 16:30:08 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:07 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 03/12] include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for type3 memory devices Date: Thu, 18 Apr 2024 16:10:54 -0700 Message-ID: <20240418232902.583744-4-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Rename mem_size as static_mem_size for type3 memdev to cover static RAM and pmem capacity, preparing for the introduction of dynamic capacity to support dynamic capacity devices. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 4 ++-- hw/mem/cxl_type3.c | 8 ++++---- include/hw/cxl/cxl_device.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 49c7944d93..0f2ad58a14 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -803,7 +803,7 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); stq_le_p(&id->total_capacity, - cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER); + cxl_dstate->static_mem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->persistent_capacity, cxl_dstate->pmem_size / CXL_CAPACITY_MULTIPLIER); stq_le_p(&id->volatile_capacity, @@ -1179,7 +1179,7 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index b0a7e9f11b..5d6d3ab87d 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -608,7 +608,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostvmem_as, vmr, v_name); ct3d->cxl_dstate.vmem_size = memory_region_size(vmr); - ct3d->cxl_dstate.mem_size += memory_region_size(vmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(vmr); g_free(v_name); } @@ -631,7 +631,7 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) } address_space_init(&ct3d->hostpmem_as, pmr, p_name); ct3d->cxl_dstate.pmem_size = memory_region_size(pmr); - ct3d->cxl_dstate.mem_size += memory_region_size(pmr); + ct3d->cxl_dstate.static_mem_size += memory_region_size(pmr); g_free(p_name); } @@ -838,7 +838,7 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.mem_size) { + if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { return -EINVAL; } @@ -1011,7 +1011,7 @@ static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { return false; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index e839370266..f7f56b44e3 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -234,7 +234,7 @@ typedef struct cxl_device_state { } timestamp; /* memory region size, HDM */ - uint64_t mem_size; + uint64_t static_mem_size; uint64_t pmem_size; uint64_t vmem_size; From patchwork Thu Apr 18 23:10:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635517 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C2FC2AF0F for ; Thu, 18 Apr 2024 23:30:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483012; cv=none; b=eEvb7n1oFOBkAIjJut+o0OEuykQZNTusBy7qS8seMxar3tT8g0B/xyzUoKbcYHp1vmyV4jTbHZBx1jsf7BeHHDhj2jqF+h4//L8WvnEUUO1CQz6/AgWe4fzaCQBgH4GhKLeImYs/qwEGupsksSY9T8MkPvcS2+mi6sY5Wf2oAUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483012; c=relaxed/simple; bh=MbsEJu+P0LQqrL/fFPzA+QCKd0j/313kTt71hbsTn4o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q9+WXuY3K5pFnt8KCYr7xsv22hLpt9eGQ2xSrJg4xdRD4CrHqE20m+EfCcBZ17F7G/JTVBZ5Il+v/2a0rDOlxtMCLQXtD7JC2pPR7BpOAj5T/d1JTqsL9JlRK1iWtfoOS8PIF58Q5WUcfkofAZ6qKqwBaD4CKfSrn4AgLr+l6I4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WvcVyyaZ; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WvcVyyaZ" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6ed3cafd766so1345016b3a.0 for ; Thu, 18 Apr 2024 16:30:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483010; x=1714087810; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ow2honBLfLsYAJoSrWVd0jh7a+uzahn+825h2aAqTgw=; b=WvcVyyaZXnM7ZmDa5GKbK44M6JBu9r3a2+i87IhRhIOP+d/19qLVxKuoNhE/0OYz93 rUxsL6QNeOngKck0xZhlDFX97YDBl75qKPT9dK0GYeS8YewBCkTQOsDQDk042QbCNyTE JeBSU5jGnHEkZCpx6V46v4viAArMpZRkWDK5pvlaYhkcDmGdIyssfqD+zSFdOYD36JAd pkfsgcY1hcrdCNR+hN8Fb/kcbJw5g0ttFnBM7B3sHfKQFlda+JVMbDuCyHc2KFVyjPCG uQdwUYgHTKoVz0oULac1zo2oOoVOz1UoXGag6uhc6u0+rwIX+xMejQaiYAWqcxH4Eb7a tpxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483010; x=1714087810; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ow2honBLfLsYAJoSrWVd0jh7a+uzahn+825h2aAqTgw=; b=APoH6azm0WmQlQoJSO0Keyg5bwHNaBWptWT0x0a2C9J6lR2KvvnjWTbediiVB/SW3v /ysjDajG8uGTbmKfvDpVU/A8MQ3P4SZvH8IRQ8f4xeLbzGRJe35PZhxja2UmKd8SwJZ+ 9aa9ym2gbgA58xG/JASuCK8jeyLXINGZPJY9nkDVj9DyyvBPhGWby/xUYJYNCgV4fHdn CVieuUB/01yx8c0dJZht7v34AlL7C3Phx1oA2vA7/i8REIoO7s/KtSn+lgVRoOsqzLfJ lDjaSiRtnLNuKvVEoMI4UjAz+M3363TLusoSXGjsHDvGyP9t4v4OhQ0600yTxTaxqXaa xEsg== X-Forwarded-Encrypted: i=1; AJvYcCUQx/J58cBJ55k+z8i2xMVs7Z3gSiMf9F2mdeSn5N8oINB2/aXSuvXpw9VJzSPVTUF9GrxqOzaGKw5go+nSlkhi1J+c+HImgFoH X-Gm-Message-State: AOJu0Yyb+jOUep5TnljgN/9fltFV3m48fd1acVIg3og6VFISjFehEeaL jgxiQ0dDp++bPiAzFN8+0Ak/f6ancc5Xv2F+O0Rl4Vz1jJl7y7NE X-Google-Smtp-Source: AGHT+IE94ylxUx/UWiEiedIHul63+PwBYMukWUa1qdA/55cOY8EAPLn5Al2VYu6orH+j7LErQAQ6vg== X-Received: by 2002:a05:6a20:96ce:b0:1a7:61f0:9ca9 with SMTP id hq14-20020a056a2096ce00b001a761f09ca9mr647381pzc.58.1713483010243; Thu, 18 Apr 2024 16:30:10 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:09 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 04/12] hw/mem/cxl_type3: Add support to create DC regions to type3 memory devices Date: Thu, 18 Apr 2024 16:10:55 -0700 Message-ID: <20240418232902.583744-5-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, when setting up memory for type3 memory device, we can create DC regions. A property 'num-dc-regions' is added to ct3_props to allow users to pass the number of DC regions to create. To make it easier, other region parameters like region base, length, and block size are hard coded. If needed, these parameters can be added easily. With the change, we can create DC regions with proper kernel side support like below: region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity echo 1 > /sys/bus/cxl/devices/$region/interleave_ways echo "dc0" >/sys/bus/cxl/devices/decoder2.0/mode echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size echo 0x40000000 > /sys/bus/cxl/devices/$region/size echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0 echo 1 > /sys/bus/cxl/devices/$region/commit echo $region > /sys/bus/cxl/drivers/cxl_region/bind Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/mem/cxl_type3.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 5d6d3ab87d..5ceed0ab4c 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -30,6 +30,7 @@ #include "hw/pci/msix.h" #define DWORD_BYTE 4 +#define CXL_CAPACITY_MULTIPLIER (256 * MiB) /* Default CDAT entries for a memory region */ enum { @@ -567,6 +568,46 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value, } } +/* + * TODO: dc region configuration will be updated once host backend and address + * space support is added for DCD. + */ +static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) +{ + int i; + uint64_t region_base = 0; + uint64_t region_len = 2 * GiB; + uint64_t decode_len = 2 * GiB; + uint64_t blk_size = 2 * MiB; + CXLDCRegion *region; + MemoryRegion *mr; + + if (ct3d->hostvmem) { + mr = host_memory_backend_get_memory(ct3d->hostvmem); + region_base += memory_region_size(mr); + } + if (ct3d->hostpmem) { + mr = host_memory_backend_get_memory(ct3d->hostpmem); + region_base += memory_region_size(mr); + } + assert(region_base % CXL_CAPACITY_MULTIPLIER == 0); + + for (i = 0, region = &ct3d->dc.regions[0]; + i < ct3d->dc.num_regions; + i++, region++, region_base += region_len) { + *region = (CXLDCRegion) { + .base = region_base, + .decode_len = decode_len, + .len = region_len, + .block_size = blk_size, + /* dsmad_handle set when creating CDAT table entries */ + .flags = 0, + }; + } + + return true; +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -635,6 +676,13 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + if (ct3d->dc.num_regions > 0) { + if (!cxl_create_dc_regions(ct3d, errp)) { + error_setg(errp, "setup DC regions failed"); + return false; + } + } + return true; } @@ -931,6 +979,7 @@ static Property ct3_props[] = { HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), + DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), DEFINE_PROP_END_OF_LIST(), }; From patchwork Thu Apr 18 23:10:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635518 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF4B12AF0F for ; Thu, 18 Apr 2024 23:30:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483015; cv=none; b=HKcX7TNDWoUAsDEzq8h4nAGQHujjYe9armeUt7ukncORSZBSnmyGe8wdxWlaiLVBKRxuncrXZYVWnubx60mrd1N2S6pXr4vPmeQlFhr/aAP9O52181U166KVS8FgZtiuHYKs/g9IbTYi/JilIIpQeDqAstSKUQft82BryXBdzsw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483015; c=relaxed/simple; bh=ELAMW9qBwsov19JxnPsdJJPxtz0H+kXe0UPqPUDCjxU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sPDzKiWtz1g6C2tAL4PhVPSdRMA9igaTSZAWa7W3s3TZhQkiaCO25XCCQ1R2UPVGP1VwMXXnc1gCsv9kVn+7b56t1NrZgtv/6OJ4h6ZA9eDNdNIVFizSk+uyqE1KGqckJib29KboXIrM3vEG1qgCL557QqWESj01baD+/uF5Qxw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JYCIApV4; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JYCIApV4" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ed112c64beso1325136b3a.1 for ; Thu, 18 Apr 2024 16:30:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483013; x=1714087813; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ggrpoa55J+CuYa6WaJVJnd/Z1hpjsnV4FDGSfq86nOQ=; b=JYCIApV4zspdbip9o46hzxqpglLTL0AuaL7GZM4aGLHCkvfVRfobZO1QJQs71MBhL+ uz9Kqsh/AmKn1dNS/pLofG2Qc+2D/sCt9rtyTWDk+gpNXdiNJ+yVcptUP7b0ixEdDrzF bOIdHdtBQ08Usa3BHDoaSedNoZj+VZY0mN4i82b7TQGyl8BJApQhqaPIjVd0NR7ztClg ECNZxYk/J1lwuvM6pW7gv/dwQQGs6nj+iDFKdtop7TWjp5Bvfm7zArq8gAGWh8ZF5UDs PYy/EoUhP7Qxw0ssAqNEWOj7NwLom5CLoEf0miI6BPDMfnZMtkh4J39w2Jily7mal4qR gBYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483013; x=1714087813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ggrpoa55J+CuYa6WaJVJnd/Z1hpjsnV4FDGSfq86nOQ=; b=X5TMyDzOY5YA/MKatsHGVktKMi/XShELXrfz/953/N488IsTXid7HeuTEC+uU3Hpcy GRvDtRXO426nfvXU4MrXFO+yc26r8HjT6YgsDUIKKgWlAnHCoXRxFY1qgK/0SJuEzmgJ rBIzPArmN/2e2ECJE73f2pwgyRKsL59EWuNwuDvtBCC53aUY7DvY2qPYkj80ssq1iBDR PjSMuNqveTmqyJEEerUZ0GIFhRriKnEBgaUWHgBI1r1xB6Jzb9GcR90RMXlAbLqchWj4 0s4dMHbaRRV/oyYBmTqBS0CvhLayZvfNjOL0DNYShR8GfNNSy3ugRAMwwhZLaDFYtUI8 2U5Q== X-Forwarded-Encrypted: i=1; AJvYcCUKDqfdQHAoj4XdhzzwFuDKdeOazl+KzT8LNDRnXys2TX4U2xTXLVj+Ktogert/f5Q+ELN4ljBmjkxUCGIE0wiYIBUABwn8TN/K X-Gm-Message-State: AOJu0Yydc2hzmWCqM5JeNo57bpmfa1ObY0O5CqJkMGiK21S05Mlnwn2X IMO1aNeiu2IGVA52KBZkvuLA35X6PWGiO+cCFMDmoWGMK6VE387+ X-Google-Smtp-Source: AGHT+IFzgF73P2kI4foReYX74Mh4RfoG1oX0kH4u/p06dPT/Tbq0J7QDVqdIYlKQaXMV/Vaon1Tgtg== X-Received: by 2002:a05:6a00:27a0:b0:6ea:7981:d40b with SMTP id bd32-20020a056a0027a000b006ea7981d40bmr713005pfb.16.1713483012793; Thu, 18 Apr 2024 16:30:12 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:12 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 05/12] hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr size instead of mr as argument Date: Thu, 18 Apr 2024 16:10:56 -0700 Message-ID: <20240418232902.583744-6-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni The function ct3_build_cdat_entries_for_mr only uses size of the passed memory region argument, refactor the function definition to make the passed arguments more specific. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/mem/cxl_type3.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 5ceed0ab4c..a1fe268560 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -44,7 +44,7 @@ enum { }; static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *mr, + int dsmad_handle, uint64_t size, bool is_pmem, uint64_t dpa_base) { CDATDsmas *dsmas; @@ -63,7 +63,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .DSMADhandle = dsmad_handle, .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, .DPA_base = dpa_base, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* For now, no memory side cache, plausiblish numbers */ @@ -132,7 +132,7 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, */ .EFI_memory_type_attr = is_pmem ? 2 : 1, .DPA_offset = 0, - .DPA_length = memory_region_size(mr), + .DPA_length = size, }; /* Header always at start of structure */ @@ -149,6 +149,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; @@ -163,6 +164,7 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + vmr_size = memory_region_size(volatile_mr); } if (ct3d->hostpmem) { @@ -171,21 +173,22 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) return -EINVAL; } len += CT3_CDAT_NUM_ENTRIES; + pmr_size = memory_region_size(nonvolatile_mr); } table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { - ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volatile_mr, + ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { - uint64_t base = volatile_mr ? memory_region_size(volatile_mr) : 0; + uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - nonvolatile_mr, true, base); + pmr_size, true, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } assert(len == cur_ent); From patchwork Thu Apr 18 23:10:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635519 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 818AA2AF0F for ; Thu, 18 Apr 2024 23:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483018; cv=none; b=nMXf119NjjyujmvaJzuEOpour+n1vjfSWpUmBPWyAHkI197kR99eBowpwvd7JzBmX6PjAXd6phjXZTXKRYL5j9AL9FWixdjt+w3YvNpzUPNucp83BXrFUybSVhOQ35ElVFGxbpl7UoaNtkVX+w4ckTzajbT/w33dOgBiT/LXogw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483018; c=relaxed/simple; bh=pecmI63i8V+au0MCTYswh/QpGHcQy7+n+QqR0RNRWjM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PMAlgMa6ktuUnZSmE8aSiEilidOP0lxzAWBfkopZP2lg9AzjCxeDPf9qBFSfSuJoxdisEJaqKc1hrvGxENrx/EqMkbtsW3JXGSTJ0Abjrqkro45pWKykr8iYG+9Vbs36o34/rw2+Sacx6Hy6mXuwvFXkxI88yNhbjMgqcQos624= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dArkGlOr; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dArkGlOr" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-5f415fd71f8so1084110a12.3 for ; Thu, 18 Apr 2024 16:30:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483016; x=1714087816; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxule/gMtEI1MvLKkWxzoQAnJuD3vvD9B73sdceJ75M=; b=dArkGlOrcqlP8278zI0QebgKTiem5n+Fv3LERPbXhRRpwWXzbrf2/z5vdQloBu1U9+ DsClie3Dr4GqnvVFyLb7McK1zkm7V9h3q/5bhlinWZbJGcfjE8AlokVtDym14za6KBy0 X2UFlUM0fC/RqYf27vUGVvDQeHX363X8mlbuvh6rQrhzJwBWhvyeYExng117pen6XaPC LHpBlvIzWJIWnQtevAIdRIGBslq6TbZhMZqsgyZ1jDFSERyR4K/Cd/azXnC2IUy/vzPB oHc1yav3faj1Z5As219RdABGcccoS7KgX+s32/jF6YXAEWJZid64I1+MT2Me6Bxub9kE noRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483016; x=1714087816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxule/gMtEI1MvLKkWxzoQAnJuD3vvD9B73sdceJ75M=; b=tGBlRI8M8WFMyOi1jeUYKQyF38rMHf1vzjsK6SpZVkSNKDXDLYl1xyZDIX33WsiXFj Yd80fINe5PNgHVg1ModpxQmK88nV/g+t5MUslh8nNpwX0g9DtJSZ/CspkNHA/ZHKtai/ Uvn4SKd9ylvcNdSOuuNhSz5yNZ4YBoGd7K1s982T5c2zdev0i2t5PRytMchk05OZsJlA zXKviXVONHRy46Z+HA4sQNw+G/4EmnRvfI212kugSQfm08TfgkMjXqO7i0glk9cwBYIw CjwPbZTaEYrQhUW6i217Rp54ATywmYJBIFD10u3cGGhxZWc+fR4pbMITw+fzFa4azuZ/ auHA== X-Forwarded-Encrypted: i=1; AJvYcCUjExes0fbUAtCDSkhBbNknrD8dOrsm4YisJoxgBZWIPJd0c9hkL9abySwDR3iDDu0fWJuNhNIOh+FpW1DG0rf9HOzpgmcWQaym X-Gm-Message-State: AOJu0YwjjKgkCCbdXwcdWIj1a0mw6pi0ZYS7ApGwY7TlOsnD7JG+SmSL CvVHTuQA6EAfJ7Zim7YAfpaHySIYrmRLVFcPKCw3GnWG5lP7Hm/V X-Google-Smtp-Source: AGHT+IF7E3nwkc/Hek34L4khNmjp08wBGYvYa33glyGNnfXexfP/auxHVMBZVWbeo67G8IElhdxNvg== X-Received: by 2002:a05:6a20:3951:b0:1aa:a473:3ee0 with SMTP id r17-20020a056a20395100b001aaa4733ee0mr889618pzg.6.1713483015264; Thu, 18 Apr 2024 16:30:15 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:14 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v7 06/12] hw/mem/cxl_type3: Add host backend and address space handling for DC regions Date: Thu, 18 Apr 2024 16:10:57 -0700 Message-ID: <20240418232902.583744-7-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add (file/memory backed) host backend for DCD. All the dynamic capacity regions will share a single, large enough host backend. Set up address space for DC regions to support read/write operations to dynamic capacity for DCD. With the change, the following support is added: 1. Add a new property to type3 device "volatile-dc-memdev" to point to host memory backend for dynamic capacity. Currently, all DC regions share one host backend; 2. Add namespace for dynamic capacity for read/write support; 3. Create cdat entries for each dynamic capacity region. Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 16 ++-- hw/mem/cxl_type3.c | 172 +++++++++++++++++++++++++++++------- include/hw/cxl/cxl_device.h | 8 ++ 3 files changed, 160 insertions(+), 36 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 0f2ad58a14..831cef0567 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -622,7 +622,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, size_t *len_out, CXLCCI *cci) { - CXLDeviceState *cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; struct { uint8_t slots_supported; uint8_t slot_info; @@ -636,7 +637,8 @@ static CXLRetCode cmd_firmware_update_get_info(const struct cxl_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*fw_info) != 0x50); if ((cxl_dstate->vmem_size < CXL_CAPACITY_MULTIPLIER) || - (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER)) { + (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) || + (ct3d->dc.total_capacity < CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } @@ -793,7 +795,8 @@ static CXLRetCode cmd_identify_memory_device(const struct cxl_cmd *cmd, CXLDeviceState *cxl_dstate = &ct3d->cxl_dstate; if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -835,9 +838,11 @@ static CXLRetCode cmd_ccls_get_partition_info(const struct cxl_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info = (void *)payload_out; QEMU_BUILD_BUG_ON(sizeof(*part_info) != 0x20); + CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate); if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER)) || - (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))) { + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER)) || + (!QEMU_IS_ALIGNED(ct3d->dc.total_capacity, CXL_CAPACITY_MULTIPLIER))) { return CXL_MBOX_INTERNAL_ERROR; } @@ -1179,7 +1184,8 @@ static CXLRetCode cmd_media_clear_poison(const struct cxl_cmd *cmd, struct clear_poison_pl *in = (void *)payload_in; dpa = ldq_le_p(&in->dpa); - if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size) { + if (dpa + CXL_CACHE_LINE_SIZE > cxl_dstate->static_mem_size + + ct3d->dc.total_capacity) { return CXL_MBOX_INVALID_PA; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index a1fe268560..ac87398089 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -45,7 +45,8 @@ enum { static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, int dsmad_handle, uint64_t size, - bool is_pmem, uint64_t dpa_base) + bool is_pmem, bool is_dynamic, + uint64_t dpa_base) { CDATDsmas *dsmas; CDATDslbis *dslbis0; @@ -61,7 +62,8 @@ static void ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, .length = sizeof(*dsmas), }, .DSMADhandle = dsmad_handle, - .flags = is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .flags = (is_pmem ? CDAT_DSMAS_FLAG_NV : 0) | + (is_dynamic ? CDAT_DSMAS_FLAG_DYNAMIC_CAP : 0), .DPA_base = dpa_base, .DPA_length = size, }; @@ -149,12 +151,13 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) g_autofree CDATSubHeader **table = NULL; CXLType3Dev *ct3d = priv; MemoryRegion *volatile_mr = NULL, *nonvolatile_mr = NULL; + MemoryRegion *dc_mr = NULL; uint64_t vmr_size = 0, pmr_size = 0; int dsmad_handle = 0; int cur_ent = 0; int len = 0; - if (!ct3d->hostpmem && !ct3d->hostvmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem && !ct3d->dc.num_regions) { return 0; } @@ -176,21 +179,54 @@ static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) pmr_size = memory_region_size(nonvolatile_mr); } + if (ct3d->dc.num_regions) { + if (!ct3d->dc.host_dc) { + return -EINVAL; + } + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + return -EINVAL; + } + len += CT3_CDAT_NUM_ENTRIES * ct3d->dc.num_regions; + } + table = g_malloc0(len * sizeof(*table)); /* Now fill them in */ if (volatile_mr) { ct3_build_cdat_entries_for_mr(table, dsmad_handle++, vmr_size, - false, 0); + false, false, 0); cur_ent = CT3_CDAT_NUM_ENTRIES; } if (nonvolatile_mr) { uint64_t base = vmr_size; ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_handle++, - pmr_size, true, base); + pmr_size, true, false, base); cur_ent += CT3_CDAT_NUM_ENTRIES; } + + if (dc_mr) { + int i; + uint64_t region_base = vmr_size + pmr_size; + + /* + * TODO: we assume the dynamic capacity to be volatile for now. + * Non-volatile dynamic capacity will be added if needed in the + * future. + */ + for (i = 0; i < ct3d->dc.num_regions; i++) { + ct3_build_cdat_entries_for_mr(&(table[cur_ent]), + dsmad_handle++, + ct3d->dc.regions[i].len, + false, true, region_base); + ct3d->dc.regions[i].dsmadhandle = dsmad_handle - 1; + + cur_ent += CT3_CDAT_NUM_ENTRIES; + region_base += ct3d->dc.regions[i].len; + } + } + assert(len == cur_ent); *cdat_table = g_steal_pointer(&table); @@ -301,10 +337,16 @@ static void build_dvsecs(CXLType3Dev *ct3d) range2_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); } - } else { + } else if (ct3d->hostpmem) { range1_size_hi = ct3d->hostpmem->size >> 32; range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | (ct3d->hostpmem->size & 0xF0000000); + } else { + /* + * For DCD with no static memory, set memory active, memory class bits. + * No range is set. + */ + range1_size_lo = (2 << 5) | (2 << 2) | 0x3; } dvsec = (uint8_t *)&(CXLDVSECDevice){ @@ -579,11 +621,27 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) { int i; uint64_t region_base = 0; - uint64_t region_len = 2 * GiB; - uint64_t decode_len = 2 * GiB; + uint64_t region_len; + uint64_t decode_len; uint64_t blk_size = 2 * MiB; CXLDCRegion *region; MemoryRegion *mr; + uint64_t dc_size; + + mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(mr); + region_len = DIV_ROUND_UP(dc_size, ct3d->dc.num_regions); + + if (dc_size % (ct3d->dc.num_regions * CXL_CAPACITY_MULTIPLIER) != 0) { + error_setg(errp, "host backend size must be multiples of region len"); + return false; + } + if (region_len % CXL_CAPACITY_MULTIPLIER != 0) { + error_setg(errp, "DC region size is unaligned to %lx", + CXL_CAPACITY_MULTIPLIER); + return false; + } + decode_len = region_len; if (ct3d->hostvmem) { mr = host_memory_backend_get_memory(ct3d->hostvmem); @@ -606,6 +664,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) /* dsmad_handle set when creating CDAT table entries */ .flags = 0, }; + ct3d->dc.total_capacity += region->len; } return true; @@ -615,7 +674,8 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); - if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem + && !ct3d->dc.num_regions) { error_setg(errp, "at least one memdev property must be set"); return false; } else if (ct3d->hostmem && ct3d->hostpmem) { @@ -679,7 +739,37 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) g_free(p_name); } + ct3d->dc.total_capacity = 0; if (ct3d->dc.num_regions > 0) { + MemoryRegion *dc_mr; + char *dc_name; + + if (!ct3d->dc.host_dc) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + if (!dc_mr) { + error_setg(errp, "dynamic capacity must have a backing device"); + return false; + } + + /* + * TODO: set dc as volatile for now, non-volatile support can be added + * in the future if needed. + */ + memory_region_set_nonvolatile(dc_mr, false); + memory_region_set_enabled(dc_mr, true); + host_memory_backend_set_mapped(ct3d->dc.host_dc, true); + if (ds->id) { + dc_name = g_strdup_printf("cxl-dcd-dpa-dc-space:%s", ds->id); + } else { + dc_name = g_strdup("cxl-dcd-dpa-dc-space"); + } + address_space_init(&ct3d->dc.host_dc_as, dc_mr, dc_name); + g_free(dc_name); + if (!cxl_create_dc_regions(ct3d, errp)) { error_setg(errp, "setup DC regions failed"); return false; @@ -776,6 +866,9 @@ err_release_cdat: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -794,6 +887,9 @@ static void ct3_exit(PCIDevice *pci_dev) pcie_aer_exit(pci_dev); cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); + if (ct3d->dc.host_dc) { + address_space_destroy(&ct3d->dc.host_dc_as); + } if (ct3d->hostpmem) { address_space_destroy(&ct3d->hostpmem_as); } @@ -872,16 +968,23 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, AddressSpace **as, uint64_t *dpa_offset) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); + } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return -ENODEV; } @@ -889,19 +992,18 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, return -EINVAL; } - if (*dpa_offset > ct3d->cxl_dstate.static_mem_size) { + if (*dpa_offset >= vmr_size + pmr_size + dc_size) { return -EINVAL; } - if (vmr) { - if (*dpa_offset < memory_region_size(vmr)) { - *as = &ct3d->hostvmem_as; - } else { - *as = &ct3d->hostpmem_as; - *dpa_offset -= memory_region_size(vmr); - } - } else { + if (*dpa_offset < vmr_size) { + *as = &ct3d->hostvmem_as; + } else if (*dpa_offset < vmr_size + pmr_size) { *as = &ct3d->hostpmem_as; + *dpa_offset -= vmr_size; + } else { + *as = &ct3d->dc.host_dc_as; + *dpa_offset -= (vmr_size + pmr_size); } return 0; @@ -983,6 +1085,8 @@ static Property ct3_props[] = { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0), + DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_END_OF_LIST(), }; @@ -1049,33 +1153,39 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, static bool set_cacheline(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data) { - MemoryRegion *vmr = NULL, *pmr = NULL; + MemoryRegion *vmr = NULL, *pmr = NULL, *dc_mr = NULL; AddressSpace *as; + uint64_t vmr_size = 0, pmr_size = 0, dc_size = 0; if (ct3d->hostvmem) { vmr = host_memory_backend_get_memory(ct3d->hostvmem); + vmr_size = memory_region_size(vmr); } if (ct3d->hostpmem) { pmr = host_memory_backend_get_memory(ct3d->hostpmem); + pmr_size = memory_region_size(pmr); } + if (ct3d->dc.host_dc) { + dc_mr = host_memory_backend_get_memory(ct3d->dc.host_dc); + dc_size = memory_region_size(dc_mr); + } - if (!vmr && !pmr) { + if (!vmr && !pmr && !dc_mr) { return false; } - if (dpa_offset + CXL_CACHE_LINE_SIZE > ct3d->cxl_dstate.static_mem_size) { + if (dpa_offset + CXL_CACHE_LINE_SIZE > vmr_size + pmr_size + dc_size) { return false; } - if (vmr) { - if (dpa_offset < memory_region_size(vmr)) { - as = &ct3d->hostvmem_as; - } else { - as = &ct3d->hostpmem_as; - dpa_offset -= memory_region_size(vmr); - } - } else { + if (dpa_offset < vmr_size) { + as = &ct3d->hostvmem_as; + } else if (dpa_offset < vmr_size + pmr_size) { as = &ct3d->hostpmem_as; + dpa_offset -= vmr_size; + } else { + as = &ct3d->dc.host_dc_as; + dpa_offset -= (vmr_size + pmr_size); } address_space_write(as, dpa_offset, MEMTXATTRS_UNSPECIFIED, &data, diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index f7f56b44e3..c2c3df0d2a 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -467,6 +467,14 @@ struct CXLType3Dev { uint64_t poison_list_overflow_ts; struct dynamic_capacity { + HostMemoryBackend *host_dc; + AddressSpace host_dc_as; + /* + * total_capacity is equivalent to the dynamic capability + * memory region size. + */ + uint64_t total_capacity; /* 256M aligned */ + uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; } dc; From patchwork Thu Apr 18 23:10:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635520 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9C122AF0F for ; Thu, 18 Apr 2024 23:30:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483020; cv=none; b=II+MckohOdZaLcqpoXGA8N0SDJwkgzLGx45K/nee3tI9rLvcCmWDpcOCPA6lKBgDA4vyrq5J1xzJWju+BrxeiqhHwvvipj436X1w4DOpN0WfU1BuffNNU+GzZrksRgQQXQ8Jxj8AJlTbpoAxJ3rrRvlfAt1My+0mEYRPCOr6hjg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483020; c=relaxed/simple; bh=CItn3pJKl4ScZg3+BNRMMC8UjXssn/0Kjl34dNUphQo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kC5FHK+C1NdpRo5mcE/medBsxK20yBM3KUSqBCG3Zn2ddH1Ibe7mvVCiYPiEaEhbx6A4AcBMuf60GJ6xxv7U4Pte9XlxazaWdweQ8GT5TUrCmbTuvZ54kK9HBggTEe7hERq30ABehGFh14lRoJrkmG4ela2Qgk5wtD89ZRLm2hQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LES4qxYD; arc=none smtp.client-ip=209.85.160.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LES4qxYD" Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-2343ae31a9bso553848fac.1 for ; Thu, 18 Apr 2024 16:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483018; x=1714087818; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W7olyXsP/ZAkWB4fKTJScAZvlnaROR/es4Umj1YAQ2k=; b=LES4qxYDWfBOLewZOFP7dFx7DTmDK4nryfP2L1Q6c9liGUoHLmKLJReXGTyJnqQe1Q 4LzmaUA3NHTrI1y269ZuGq2sOc1v5TX/dl7SKf6/KQMz6R8ZuffovzfiELdZeOMPrmWY hgLe3HZjMk1XO3idY6ZPg5XYI19u0QYPRMUl35g2J2/4vXlDslniTf9HkYtFOoKXOnzK XbmPPxH/dlFCdxX7A8AmiT1JqFZSSbc8n6LpsJ5pr/3BNMpZLw0fkMj7DqwLqtU4o4km 1QKyEEuLuAVfjvMRMNWolAnwqeN0+juMtl+DghV7j4POg39o7XApdRUFZ+dv786KYJeV Sjnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483018; x=1714087818; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W7olyXsP/ZAkWB4fKTJScAZvlnaROR/es4Umj1YAQ2k=; b=aCW3uXUb0A8gKR+n5HyvBDhwV6Apyo1hvQJIKcbEFWwUmqbjyv051caOjXgX1btHBn ojJs0eIiymmfriml1+tHNB3pKESa+QEBjiQc1Wfx2kYK0iqPe9V6S1ipTidFIdrYKSNe XuLbpXR0/IteG4uIFsABzHMpsRzJojp6h8aUya6DlhCwMuV+tNz4ZstgivatPGgLTVKD 4OjrjeC4y4f6SX12I79dFkYv4YnoTSV94tEgI6Ia+B+7BZv7TwzCjMf4lbnCzATzwN/k yMBDF1eijqr1Q/3TDJnl7pdXPMKtVEpQDFY28XsINqFmqm3g5T+GMwJ6YLWkuOeLPny1 CUSw== X-Forwarded-Encrypted: i=1; AJvYcCW1ee1T91D9KWHUtuVulR/DfyG8dfKuZXy1YqdoOjHDvXhn9tJEszOoOJ93veYqqQyoRL/NnpJHbrpEZ2ZzcWCPsROc41ivlfGG X-Gm-Message-State: AOJu0YzbF7Oq5LFGdQ70ai2Sg0Vg0zlkIbW3ha2fy5A39/iR6i73gFb0 JP5HYJlDDNct27nCGvcH1RL2yc2j1lZhuz2J/vTqz0vG+b1Bwxnn X-Google-Smtp-Source: AGHT+IFV25Fem5Uis6ENXyOgVveGsLlr8kl7CIwk7wpZ3gbgR91RNkwk+Z7WA4kMX9bgdmOt2h59hw== X-Received: by 2002:a05:6870:2112:b0:22e:9c5a:233e with SMTP id f18-20020a056870211200b0022e9c5a233emr629483oae.43.1713483017919; Thu, 18 Apr 2024 16:30:17 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:17 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 07/12] hw/mem/cxl_type3: Add DC extent list representative and get DC extent list mailbox support Date: Thu, 18 Apr 2024 16:10:58 -0700 Message-ID: <20240418232902.583744-8-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Add dynamic capacity extent list representative to the definition of CXLType3Dev and implement get DC extent list mailbox command per CXL.spec.3.1:.8.2.9.9.9.2. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 73 ++++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 1 + include/hw/cxl/cxl_device.h | 22 +++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 831cef0567..1915959015 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,7 @@ enum { #define CLEAR_POISON 0x2 DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 + #define GET_DYN_CAP_EXT_LIST 0x1 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1322,7 +1323,8 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, * to use. */ stl_le_p(&extra_out->num_extents_supported, CXL_NUM_EXTENTS_SUPPORTED); - stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED); + stl_le_p(&extra_out->num_extents_available, CXL_NUM_EXTENTS_SUPPORTED - + ct3d->dc.total_extent_count); stl_le_p(&extra_out->num_tags_supported, CXL_NUM_TAGS_SUPPORTED); stl_le_p(&extra_out->num_tags_available, CXL_NUM_TAGS_SUPPORTED); @@ -1330,6 +1332,72 @@ static CXLRetCode cmd_dcd_get_dyn_cap_config(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * CXL r3.1 section 8.2.9.9.9.2: + * Get Dynamic Capacity Extent List (Opcode 4801h) + */ +static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + struct { + uint32_t extent_cnt; + uint32_t start_extent_id; + } QEMU_PACKED *in = (void *)payload_in; + struct { + uint32_t count; + uint32_t total_extents; + uint32_t generation_num; + uint8_t rsvd[4]; + CXLDCExtentRaw records[]; + } QEMU_PACKED *out = (void *)payload_out; + uint32_t start_extent_id = in->start_extent_id; + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint16_t record_count = 0, i = 0, record_done = 0; + uint16_t out_pl_len, size; + CXLDCExtent *ent; + + if (start_extent_id > ct3d->dc.total_extent_count) { + return CXL_MBOX_INVALID_INPUT; + } + + record_count = MIN(in->extent_cnt, + ct3d->dc.total_extent_count - start_extent_id); + size = CXL_MAILBOX_MAX_PAYLOAD_SIZE - sizeof(*out); + record_count = MIN(record_count, size / sizeof(out->records[0])); + out_pl_len = sizeof(*out) + record_count * sizeof(out->records[0]); + + stl_le_p(&out->count, record_count); + stl_le_p(&out->total_extents, ct3d->dc.total_extent_count); + stl_le_p(&out->generation_num, ct3d->dc.ext_list_gen_seq); + + if (record_count > 0) { + CXLDCExtentRaw *out_rec = &out->records[record_done]; + + QTAILQ_FOREACH(ent, extent_list, node) { + if (i++ < start_extent_id) { + continue; + } + stq_le_p(&out_rec->start_dpa, ent->start_dpa); + stq_le_p(&out_rec->len, ent->len); + memcpy(&out_rec->tag, ent->tag, 0x10); + stw_le_p(&out_rec->shared_seq, ent->shared_seq); + + record_done++; + if (record_done == record_count) { + break; + } + } + } + + *len_out = out_pl_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1377,6 +1445,9 @@ static const struct cxl_cmd cxl_cmd_set[256][256] = { static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DC_CONFIG] = { "DCD_GET_DC_CONFIG", cmd_dcd_get_dyn_cap_config, 2, 0 }, + [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { + "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, + 8, 0 }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index ac87398089..9fffeae613 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -666,6 +666,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) }; ct3d->dc.total_capacity += region->len; } + QTAILQ_INIT(&ct3d->dc.extents); return true; } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c2c3df0d2a..6aec6ac983 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -424,6 +424,25 @@ typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define DCD_MAX_NUM_REGION 8 +typedef struct CXLDCExtentRaw { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; +} QEMU_PACKED CXLDCExtentRaw; + +typedef struct CXLDCExtent { + uint64_t start_dpa; + uint64_t len; + uint8_t tag[0x10]; + uint16_t shared_seq; + uint8_t rsvd[0x6]; + + QTAILQ_ENTRY(CXLDCExtent) node; +} CXLDCExtent; +typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -474,6 +493,9 @@ struct CXLType3Dev { * memory region size. */ uint64_t total_capacity; /* 256M aligned */ + CXLDCExtentList extents; + uint32_t total_extent_count; + uint32_t ext_list_gen_seq; uint8_t num_regions; /* 0-8 regions */ CXLDCRegion regions[DCD_MAX_NUM_REGION]; From patchwork Thu Apr 18 23:10:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635521 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF9642AF0F for ; Thu, 18 Apr 2024 23:30:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483022; cv=none; b=TCWHRzT52DCd6LUWgBSmOyrM5nIb+6od3VAvTRZVq18+KI+XEu+vDuVZapaTL6OCvDv6NPGcQJPhe1MJ0vYiRYZOWR4BICL3S0ikf8SeR+Jjh4AYKsR8CYcsxeOF+9xyCvrl/xp2O84EHLVMGL+GGDB2Mcdu/EQLY9QIcgPslYU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483022; c=relaxed/simple; bh=H37QElZaYw8cc/bQLnTOkFgkBI1/83pqv8iSx94a1Nw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YYUHOycQSVG5N9lH7dru/DouWlJ4QP3dSSZHjDeLZ94LgI9hmGis9cHhaLrFyqGvd1y0RYFMdRNdJ0PYxTE29vqJIToIQs4yfErOicLnEnGGVygYGs4vtChRVKFLcZt///koiX/NaHPetuUl3UHsF6ee2525ns2NkE710EX7ouA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m4VfvAqp; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m4VfvAqp" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6ed3cafd766so1345110b3a.0 for ; Thu, 18 Apr 2024 16:30:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483020; x=1714087820; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NBTSn2xvmI/GgZYP55li9n8Yljqg5AbUJhV3uKwHfjY=; b=m4VfvAqpCWNdtklHGNKhGw0g8bGshdi7dNPkh4mKCyMUjLcghQNWy1DUQdkpykWVfF kbcF4/UaqizY+so3o9Wvkm+/JjOnAs0mxZaHHJSje+Ed0Rur7TbxByPBDozdFKu/7N82 AW7EQR3lAulBIRUkAOPYkJBRY/fkkFmwpj5HKTzp047wyYLyWQYyVCEhfBE7+k/gPHrg WmGIMHyItG++iAMX+IKOKwtgXpqGp4xM3kYs+uf161q0+wen/3hDuW4AkdE+fSl3vw52 jgTyOy2jUz5SEl9GC7J8mdhW20V4hGBEoMA+dLyXUvhgv/hTA/rxXAshgYs7j0b+w2c3 wy5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483020; x=1714087820; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NBTSn2xvmI/GgZYP55li9n8Yljqg5AbUJhV3uKwHfjY=; b=KQfXlwivAWDdn2G2ENbbBMVxrebYP3Z0k3vxbBE7ZVODFJYWLmBi4HUQLlH+A0lRlX SteA1gYZqt6/VubynvDpnDOicO2v/AN+HmIQv1b98jNeoMRA3TziQxK3o1ooBU5BMcBl 7RnZs+WH7pX2kGpZgPrDXQwq8hNp01GwhWKeHfgN35+98H03P11JbC7F+Te1V8qXg3vb UqaGH2bLXBEDFH6PncVpPfYyzSdpwFQPdmBfZgiNxM0LI7IQ0kbqhdmWg3LehwcpkZGE atlnDr5KxN942P/gXNHe3KuGsBlcZymRLlsDDAvj7a2ssfLMz9+7DRgc6G/RHhjOrijB sbrQ== X-Forwarded-Encrypted: i=1; AJvYcCVwhwbvi4TaQZ8pwc0MJIULpWwTemL3tZNaCcQAXVgfy9v5BcUqIGSQh1RnUN3L/r2cu7NUQ7YBkkCBdLvZPVeyWdj5yk2uYRHl X-Gm-Message-State: AOJu0YyH56YeqyxOskGqVWVftz0qokcZ9aV71EAHPzTL7jwmHDvlSR7H IuMt2VqMK9EQTnj1GbxZXfUiaMvhTvWhriRjmbIXPd6RJ1wl27hG X-Google-Smtp-Source: AGHT+IGXUXgQBD555Z5Uo9zBcoH9evvpZia10izHxrpIA5PHteZLB62WD7cfBkVrb4GRkBse3a7vrw== X-Received: by 2002:aa7:8890:0:b0:6ea:c634:ca0f with SMTP id z16-20020aa78890000000b006eac634ca0fmr701599pfe.21.1713483020058; Thu, 18 Apr 2024 16:30:20 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:19 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v7 08/12] hw/cxl/cxl-mailbox-utils: Add mailbox commands to support add/release dynamic capacity response Date: Thu, 18 Apr 2024 16:10:59 -0700 Message-ID: <20240418232902.583744-9-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Per CXL spec 3.1, two mailbox commands are implemented: Add Dynamic Capacity Response (Opcode 4802h) 8.2.9.9.9.3, and Release Dynamic Capacity (Opcode 4803h) 8.2.9.9.9.4. For the process of the above two commands, we use two-pass approach. Pass 1: Check whether the input payload is valid or not; if not, skip Pass 2 and return mailbox process error. Pass 2: Do the real work--add or release extents, respectively. Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 394 ++++++++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 11 + include/hw/cxl/cxl_device.h | 4 + 3 files changed, 409 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 1915959015..9d54e10cd4 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "qemu/uuid.h" #include "sysemu/hostmem.h" +#include "qemu/range.h" #define CXL_CAPACITY_MULTIPLIER (256 * MiB) #define CXL_DC_EVENT_LOG_SIZE 8 @@ -85,6 +86,8 @@ enum { DCD_CONFIG = 0x48, #define GET_DC_CONFIG 0x0 #define GET_DYN_CAP_EXT_LIST 0x1 + #define ADD_DYN_CAP_RSP 0x2 + #define RELEASE_DYN_CAP 0x3 PHYSICAL_SWITCH = 0x51, #define IDENTIFY_SWITCH_DEVICE 0x0 #define GET_PHYSICAL_PORT_STATE 0x1 @@ -1398,6 +1401,391 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } +/* + * Check whether any bit between addr[nr, nr+size) is set, + * return true if any bit is set, otherwise return false + */ +static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size) +{ + unsigned long res = find_next_bit(addr, size + nr, nr); + + return res < nr + size; +} + +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) +{ + int i; + CXLDCRegion *region = &ct3d->dc.regions[0]; + + if (dpa < region->base || + dpa >= region->base + ct3d->dc.total_capacity) { + return NULL; + } + + /* + * CXL r3.1 section 9.13.3: Dynamic Capacity Device (DCD) + * + * Regions are used in increasing-DPA order, with Region 0 being used for + * the lowest DPA of Dynamic Capacity and Region 7 for the highest DPA. + * So check from the last region to find where the dpa belongs. Extents that + * cross multiple regions are not allowed. + */ + for (i = ct3d->dc.num_regions - 1; i >= 0; i--) { + region = &ct3d->dc.regions[i]; + if (dpa >= region->base) { + if (dpa + len > region->base + region->len) { + return NULL; + } + return region; + } + } + + return NULL; +} + +static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + CXLDCExtent *extent; + + extent = g_new0(CXLDCExtent, 1); + extent->start_dpa = dpa; + extent->len = len; + if (tag) { + memcpy(extent->tag, tag, 0x10); + } + extent->shared_seq = shared_seq; + + QTAILQ_INSERT_TAIL(list, extent, node); +} + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent) +{ + QTAILQ_REMOVE(list, extent, node); + g_free(extent); +} + +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + +/* + * For the extents in the extent list to operate, check whether they are valid + * 1. The extent should be in the range of a valid DC region; + * 2. The extent should not cross multiple regions; + * 3. The start DPA and the length of the extent should align with the block + * size of the region; + * 4. The address range of multiple extents in the list should not overlap. + */ +static CXLRetCode cxl_detect_malformed_extent_list(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint64_t min_block_size = UINT64_MAX; + CXLDCRegion *region; + CXLDCRegion *lastregion = &ct3d->dc.regions[ct3d->dc.num_regions - 1]; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, len; + uint32_t i; + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + min_block_size = MIN(min_block_size, region->block_size); + } + + blk_bitmap = bitmap_new((lastregion->base + lastregion->len - + ct3d->dc.regions[0].base) / min_block_size); + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return CXL_MBOX_INVALID_PA; + } + + dpa -= ct3d->dc.regions[0].base; + if (dpa % region->block_size || len % region->block_size) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + /* the dpa range already covered by some other extents in the list */ + if (test_any_bits_set(blk_bitmap, dpa / min_block_size, + len / min_block_size)) { + return CXL_MBOX_INVALID_EXTENT_LIST; + } + bitmap_set(blk_bitmap, dpa / min_block_size, len / min_block_size); + } + + return CXL_MBOX_SUCCESS; +} + +static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in) +{ + uint32_t i; + CXLDCExtent *ent; + uint64_t dpa, len; + Range range1, range2; + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + range_init_nofail(&range1, dpa, len); + + /* + * TODO: once the pending extent list is added, check against + * the list will be added here. + */ + + /* to-be-added range should not overlap with range already accepted */ + QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return CXL_MBOX_INVALID_PA; + } + } + } + return CXL_MBOX_SUCCESS; +} + +/* + * CXL r3.1 section 8.2.9.9.9.3: Add Dynamic Capacity Response (Opcode 4802h) + * An extent is added to the extent list and becomes usable only after the + * response is processed successfully. + */ +static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList *extent_list = &ct3d->dc.extents; + uint32_t i; + uint64_t dpa, len; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + /* + * TODO: once the pending list is introduced, extents in the beginning + * will get wiped out. + */ + return CXL_MBOX_SUCCESS; + } + + /* Adding extents causes exceeding device's extent tracking ability. */ + if (in->num_entries_updated + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + return CXL_MBOX_RESOURCES_EXHAUSTED; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dcd_add_dyn_cap_rsp_dry_run(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + for (i = 0; i < in->num_entries_updated; i++) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); + ct3d->dc.total_extent_count += 1; + /* + * TODO: we will add a pending extent list based on event log record + * and process the list accordingly here. + */ + } + + return CXL_MBOX_SUCCESS; +} + +/* + * Copy extent list from src to dst + * Return value: number of extents copied + */ +static uint32_t copy_extent_list(CXLDCExtentList *dst, + const CXLDCExtentList *src) +{ + uint32_t cnt = 0; + CXLDCExtent *ent; + + if (!dst || !src) { + return 0; + } + + QTAILQ_FOREACH(ent, src, node) { + cxl_insert_extent_to_extent_list(dst, ent->start_dpa, ent->len, + ent->tag, ent->shared_seq); + cnt++; + } + return cnt; +} + +static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, + const CXLUpdateDCExtentListInPl *in, CXLDCExtentList *updated_list, + uint32_t *updated_list_size) +{ + CXLDCExtent *ent, *ent_next; + uint64_t dpa, len; + uint32_t i; + int cnt_delta = 0; + CXLRetCode ret = CXL_MBOX_SUCCESS; + + QTAILQ_INIT(updated_list); + copy_extent_list(updated_list, &ct3d->dc.extents); + + for (i = 0; i < in->num_entries_updated; i++) { + Range range; + + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + while (len > 0) { + QTAILQ_FOREACH(ent, updated_list, node) { + range_init_nofail(&range, ent->start_dpa, ent->len); + + if (range_contains(&range, dpa)) { + uint64_t len1, len2 = 0, len_done = 0; + uint64_t ent_start_dpa = ent->start_dpa; + uint64_t ent_len = ent->len; + + len1 = dpa - ent->start_dpa; + /* Found the extent or the subset of an existing extent */ + if (range_contains(&range, dpa + len - 1)) { + len2 = ent_start_dpa + ent_len - dpa - len; + } else { + /* + * TODO: we reject the attempt to remove an extent + * that overlaps with multiple extents in the device + * for now. We will allow it once superset release + * support is added. + */ + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + len_done = ent_len - len1 - len2; + + cxl_remove_extent_from_extent_list(updated_list, ent); + cnt_delta--; + + if (len1) { + cxl_insert_extent_to_extent_list(updated_list, + ent_start_dpa, + len1, NULL, 0); + cnt_delta++; + } + if (len2) { + cxl_insert_extent_to_extent_list(updated_list, + dpa + len, + len2, NULL, 0); + cnt_delta++; + } + + if (cnt_delta + ct3d->dc.total_extent_count > + CXL_NUM_EXTENTS_SUPPORTED) { + ret = CXL_MBOX_RESOURCES_EXHAUSTED; + goto free_and_exit; + } + + len -= len_done; + /* len == 0 here until superset release is added */ + break; + } + } + if (len) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + } + } +free_and_exit: + if (ret != CXL_MBOX_SUCCESS) { + QTAILQ_FOREACH_SAFE(ent, updated_list, node, ent_next) { + cxl_remove_extent_from_extent_list(updated_list, ent); + } + *updated_list_size = 0; + } else { + *updated_list_size = ct3d->dc.total_extent_count + cnt_delta; + } + + return ret; +} + +/* + * CXL r3.1 section 8.2.9.9.9.4: Release Dynamic Capacity (Opcode 4803h) + */ +static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLUpdateDCExtentListInPl *in = (void *)payload_in; + CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLDCExtentList updated_list; + CXLDCExtent *ent, *ent_next; + uint32_t updated_list_size; + CXLRetCode ret; + + if (in->num_entries_updated == 0) { + return CXL_MBOX_INVALID_INPUT; + } + + ret = cxl_detect_malformed_extent_list(ct3d, in); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + ret = cxl_dc_extent_release_dry_run(ct3d, in, &updated_list, + &updated_list_size); + if (ret != CXL_MBOX_SUCCESS) { + return ret; + } + + /* + * If the dry run release passes, the returned updated_list will + * be the updated extent list and we just need to clear the extents + * in the accepted list and copy extents in the updated_list to accepted + * list and update the extent count; + */ + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } + copy_extent_list(&ct3d->dc.extents, &updated_list); + QTAILQ_FOREACH_SAFE(ent, &updated_list, node, ent_next) { + cxl_remove_extent_from_extent_list(&updated_list, ent); + } + ct3d->dc.total_extent_count = updated_list_size; + + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1448,6 +1836,12 @@ static const struct cxl_cmd cxl_cmd_set_dcd[256][256] = { [DCD_CONFIG][GET_DYN_CAP_EXT_LIST] = { "DCD_GET_DYNAMIC_CAPACITY_EXTENT_LIST", cmd_dcd_get_dyn_cap_ext_list, 8, 0 }, + [DCD_CONFIG][ADD_DYN_CAP_RSP] = { + "DCD_ADD_DYNAMIC_CAPACITY_RESPONSE", cmd_dcd_add_dyn_cap_rsp, + ~0, IMMEDIATE_DATA_CHANGE }, + [DCD_CONFIG][RELEASE_DYN_CAP] = { + "DCD_RELEASE_DYNAMIC_CAPACITY", cmd_dcd_release_dyn_cap, + ~0, IMMEDIATE_DATA_CHANGE }, }; static const struct cxl_cmd cxl_cmd_set_sw[256][256] = { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 9fffeae613..c2cdd6d506 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -671,6 +671,15 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) return true; } +static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) +{ + CXLDCExtent *ent, *ent_next; + + QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); + } +} + static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) { DeviceState *ds = DEVICE(ct3d); @@ -868,6 +877,7 @@ err_free_special_ops: g_free(regs->special_ops); err_address_space_free: if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { @@ -889,6 +899,7 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); g_free(regs->special_ops); if (ct3d->dc.host_dc) { + cxl_destroy_dc_regions(ct3d); address_space_destroy(&ct3d->dc.host_dc_as); } if (ct3d->hostpmem) { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 6aec6ac983..df3511e91b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -551,4 +551,8 @@ void cxl_event_irq_assert(CXLType3Dev *ct3d); void cxl_set_poison_list_overflowed(CXLType3Dev *ct3d); +CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); + +void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, + CXLDCExtent *extent); #endif From patchwork Thu Apr 18 23:11:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635522 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73ADC2AF0F for ; Thu, 18 Apr 2024 23:30:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483028; cv=none; b=XXU9GSFWJLrtk8rktmPMcpgK7uUucJ/7ZUE6W6AzsSN4vlPflnrlVFnj2bhbbQfhi+Np51Tqo2Jbm6CuV0m3+MNPzHiEIw5rwattlMUSQQVurd9q9dbP9J14/ygQTqiiPHRbbXjnItJ/maBq35uo+eCsBEbpWhhm/g5vJWt9K4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483028; c=relaxed/simple; bh=wbCdyusvg1eYoQ6nw+6KgoqBv6EZ/Pfuvy3Q6MdjNDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=THZ5b+ss6Mb5r0ojhMHr5/zfPLfZJWX+T3ASweMk9TZ58UFeXAk/QrP5eOPAqMvNYvDjuAmYEde8Dl47LEbrsVAfci4xyp3lXH14LHi9V/j8M1nMXj+o10/UGn77cVe8Yj0KVY+JFiXTBqSyqEZZ4vylePQjaggqaw8mRgWWEvc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=F2Pdz/ko; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F2Pdz/ko" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6ed2170d89fso1720181b3a.1 for ; Thu, 18 Apr 2024 16:30:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483026; x=1714087826; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Iz8WyS3jOWqxr3x5tlzn+nUAA9YBx6cI9UN5ySmFlTI=; b=F2Pdz/koV2vvx1rJPUSwkH/CSKPRx0E1zgAUXXOdLUOb3MJithIPKAta3shB5TD2D5 quO5xJ9TTBESTtXcufwTH06aBxh/vsr5sPDCzQlhBEFITIc01glRa0+fzPT0FZSpGMjh knsc90D9Yz2+lQ1cSkiXGZ4k31qjqNLTDwEzs8q805eljam/iVf1DKKwJQHDV9q84XjS ilQa7+7IsNFBIMsX0kGwHDzcZYrVLIwezyCKaJtwZ7VUwj9UiazZmCee/l3hiWITpZbF 8fgzK3lHRrE61nRSbWntN1recUpFR61/fAt18THrJdluef/puD3rhx8jcki7OFk+6bR8 G58g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483026; x=1714087826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Iz8WyS3jOWqxr3x5tlzn+nUAA9YBx6cI9UN5ySmFlTI=; b=nNl6Eva+h7yJdDtNsxAzsATjxE5RW4yGjE2aFHS0/IyJotCQ1YXvX7DotT7w8ya11j wk76lJWBo3Z526Z8N9WgL9zJybgW9mG9JpxI7GQvvAaOsPVgHBjXEMBBoTI/Mb68yzFn BvUXkQ6uLfNcEzXEhi7uXTIJzaAwatTm7Odr1vYvy+aALeyJd+XkoPD5w7SKF+3NUgZL vhq+hIP1m6hKQC7ZLbppIOssVbSA6ebjlYRoSR1uKADMbgB2UQ3tf+wfIptm+hF0p7lm HsAdbL8+JmmUj/lygG8TmF3DMhhGhMTEraFWrx9KImfvJCJpo5CuPS5qPvaxmkj7KWWe gJcw== X-Forwarded-Encrypted: i=1; AJvYcCXbsopdktEQaoE9MmEFY2InvhL3rBSHAo6M5lKz7luLni1ZeeGMyn/I+tDUeEXHPKKhVgH/HZWQTM748qIrSSnL+o7/YuzzdYg5 X-Gm-Message-State: AOJu0YwffQosIYbbA9pHT7mzbYDO8BKmb5Z41uSgFr+0CiGqG8b8+wgI 4I5oIA2YaeDHmeHt2+hzsjXuvqpTANimp/yqMcjy4BxbDmAQAmfm X-Google-Smtp-Source: AGHT+IGuZOMidIvS1XKgBmbLmASwmgsktf1oY8FzS4YnTcwQxUm/aPxeFRXUScIR/WcRnT3kp04BiQ== X-Received: by 2002:a05:6a00:39a9:b0:6ed:4a97:10cc with SMTP id fi41-20020a056a0039a900b006ed4a9710ccmr5725099pfb.3.1713483025749; Thu, 18 Apr 2024 16:30:25 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:25 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v7 09/12] hw/cxl/events: Add qmp interfaces to add/release dynamic capacity extents Date: Thu, 18 Apr 2024 16:11:00 -0700 Message-ID: <20240418232902.583744-10-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni To simulate FM functionalities for initiating Dynamic Capacity Add (Opcode 5604h) and Dynamic Capacity Release (Opcode 5605h) as in CXL spec r3.1 7.6.7.6.5 and 7.6.7.6.6, we implemented two QMP interfaces to issue add/release dynamic capacity extents requests. With the change, we allow to release an extent only when its DPA range is contained by a single accepted extent in the device. That is to say, extent superset release is not supported yet. 1. Add dynamic capacity extents: For example, the command to add two continuous extents (each 128MiB long) to region 0 (starting at DPA offset 0) looks like below: { "execute": "qmp_capabilities" } { "execute": "cxl-add-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "hid": 0, "selection-policy": 2, "region-id": 0, "tag": "", "extents": [ { "offset": 0, "len": 134217728 }, { "offset": 134217728, "len": 134217728 } ] } } 2. Release dynamic capacity extents: For example, the command to release an extent of size 128MiB from region 0 (DPA offset 128MiB) looks like below: { "execute": "cxl-release-dynamic-capacity", "arguments": { "path": "/machine/peripheral/cxl-dcd0", "hid": 0, "flags": 1, "region-id": 0, "tag": "", "extents": [ { "offset": 134217728, "len": 134217728 } ] } } Signed-off-by: Fan Ni Reviewed-by: Gregory Price Signed-off-by: Fan Ni Tested-by: Svetly Todorov Reviewed-by: Gregory Price Signed-off-by: Fan Ni Signed-off-by: Jonathan Cameron --- hw/cxl/cxl-mailbox-utils.c | 62 +++++-- hw/mem/cxl_type3.c | 311 +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3_stubs.c | 20 +++ include/hw/cxl/cxl_device.h | 22 +++ include/hw/cxl/cxl_events.h | 18 +++ qapi/cxl.json | 69 ++++++++ 6 files changed, 489 insertions(+), 13 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 9d54e10cd4..3569902e9e 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1405,7 +1405,7 @@ static CXLRetCode cmd_dcd_get_dyn_cap_ext_list(const struct cxl_cmd *cmd, * Check whether any bit between addr[nr, nr+size) is set, * return true if any bit is set, otherwise return false */ -static bool test_any_bits_set(const unsigned long *addr, unsigned long nr, +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, unsigned long size) { unsigned long res = find_next_bit(addr, size + nr, nr); @@ -1444,7 +1444,7 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len) return NULL; } -static void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, uint64_t len, uint8_t *tag, @@ -1470,6 +1470,44 @@ void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, g_free(extent); } +/* + * Add a new extent to the extent "group" if group exists; + * otherwise, create a new group + * Return value: return the group where the extent is inserted. + */ +CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq) +{ + if (!group) { + group = g_new0(CXLDCExtentGroup, 1); + QTAILQ_INIT(&group->list); + } + cxl_insert_extent_to_extent_list(&group->list, dpa, len, + tag, shared_seq); + return group; +} + +void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, + CXLDCExtentGroup *group) +{ + QTAILQ_INSERT_TAIL(list, group, node); +} + +void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list) +{ + CXLDCExtent *ent, *ent_next; + CXLDCExtentGroup *group = QTAILQ_FIRST(list); + + QTAILQ_REMOVE(list, group, node); + QTAILQ_FOREACH_SAFE(ent, &group->list, node, ent_next) { + cxl_remove_extent_from_extent_list(&group->list, ent); + } + g_free(group); +} + /* * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload @@ -1541,6 +1579,7 @@ static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, { uint32_t i; CXLDCExtent *ent; + CXLDCExtentGroup *ext_group; uint64_t dpa, len; Range range1, range2; @@ -1551,9 +1590,13 @@ static CXLRetCode cxl_dcd_add_dyn_cap_rsp_dry_run(CXLType3Dev *ct3d, range_init_nofail(&range1, dpa, len); /* - * TODO: once the pending extent list is added, check against - * the list will be added here. + * The host-accepted DPA range must be contained by the first extent + * group in the pending list */ + ext_group = QTAILQ_FIRST(&ct3d->dc.extents_pending); + if (!cxl_extents_contains_dpa_range(&ext_group->list, dpa, len)) { + return CXL_MBOX_INVALID_PA; + } /* to-be-added range should not overlap with range already accepted */ QTAILQ_FOREACH(ent, &ct3d->dc.extents, node) { @@ -1586,10 +1629,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, CXLRetCode ret; if (in->num_entries_updated == 0) { - /* - * TODO: once the pending list is introduced, extents in the beginning - * will get wiped out. - */ + cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); return CXL_MBOX_SUCCESS; } @@ -1615,11 +1655,9 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; - /* - * TODO: we will add a pending extent list based on event log record - * and process the list accordingly here. - */ } + /* Remove the first extent group in the pending list*/ + cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); return CXL_MBOX_SUCCESS; } diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index c2cdd6d506..e892b3de7b 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -667,6 +667,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) ct3d->dc.total_capacity += region->len; } QTAILQ_INIT(&ct3d->dc.extents); + QTAILQ_INIT(&ct3d->dc.extents_pending); return true; } @@ -674,10 +675,19 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; + CXLDCExtentGroup *group, *group_next; QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } + + QTAILQ_FOREACH_SAFE(group, &ct3d->dc.extents_pending, node, group_next) { + QTAILQ_REMOVE(&ct3d->dc.extents_pending, group, node); + QTAILQ_FOREACH_SAFE(ent, &group->list, node, ent_next) { + cxl_remove_extent_from_extent_list(&group->list, ent); + } + g_free(group); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -1443,7 +1453,6 @@ static int ct3d_qmp_cxl_event_log_enc(CxlEventLog log) return CXL_EVENT_TYPE_FAIL; case CXL_EVENT_LOG_FATAL: return CXL_EVENT_TYPE_FATAL; -/* DCD not yet supported */ default: return -EINVAL; } @@ -1694,6 +1703,306 @@ void qmp_cxl_inject_memory_module_event(const char *path, CxlEventLog log, } } +/* CXL r3.1 Table 8-50: Dynamic Capacity Event Record */ +static const QemuUUID dynamic_capacity_uuid = { + .data = UUID(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, + 0x95, 0x26, 0x8e, 0x10, 0x1a, 0x2a), +}; + +typedef enum CXLDCEventType { + DC_EVENT_ADD_CAPACITY = 0x0, + DC_EVENT_RELEASE_CAPACITY = 0x1, + DC_EVENT_FORCED_RELEASE_CAPACITY = 0x2, + DC_EVENT_REGION_CONFIG_UPDATED = 0x3, + DC_EVENT_ADD_CAPACITY_RSP = 0x4, + DC_EVENT_CAPACITY_RELEASED = 0x5, +} CXLDCEventType; + +/* + * Check whether the range [dpa, dpa + len - 1] has overlaps with extents in + * the list. + * Return value: return true if has overlaps; otherwise, return false + */ +static bool cxl_extents_overlaps_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_overlaps_range(&range1, &range2)) { + return true; + } + } + return false; +} + +/* + * Check whether the range [dpa, dpa + len - 1] is contained by extents in + * the list. + * Will check multiple extents containment once superset release is added. + * Return value: return true if range is contained; otherwise, return false + */ +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtent *ent; + Range range1, range2; + + if (!list) { + return false; + } + + range_init_nofail(&range1, dpa, len); + QTAILQ_FOREACH(ent, list, node) { + range_init_nofail(&range2, ent->start_dpa, ent->len); + if (range_contains_range(&range2, &range1)) { + return true; + } + } + return false; +} + +static bool cxl_extent_groups_overlaps_dpa_range(CXLDCExtentGroupList *list, + uint64_t dpa, uint64_t len) +{ + CXLDCExtentGroup *group; + + if (!list) { + return false; + } + + QTAILQ_FOREACH(group, list, node) { + if (cxl_extents_overlaps_dpa_range(&group->list, dpa, len)) { + return true; + } + } + return false; +} + +/* + * The main function to process dynamic capacity event with extent list. + * Currently DC extents add/release requests are processed. + */ +static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, + uint16_t hid, CXLDCEventType type, uint8_t rid, + CXLDCExtentRecordList *records, Error **errp) +{ + Object *obj; + CXLEventDynamicCapacity dCap = {}; + CXLEventRecordHdr *hdr = &dCap.hdr; + CXLType3Dev *dcd; + uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; + uint32_t num_extents = 0; + CXLDCExtentRecordList *list; + CXLDCExtentGroup *group = NULL; + g_autofree CXLDCExtentRaw *extents = NULL; + uint8_t enc_log = CXL_EVENT_TYPE_DYNAMIC_CAP; + uint64_t dpa, offset, len, block_size; + g_autofree unsigned long *blk_bitmap = NULL; + int i; + + obj = object_resolve_path_type(path, TYPE_CXL_TYPE3, NULL); + if (!obj) { + error_setg(errp, "Unable to resolve CXL type 3 device"); + return; + } + + dcd = CXL_TYPE3(obj); + if (!dcd->dc.num_regions) { + error_setg(errp, "No dynamic capacity support from the device"); + return; + } + + + if (rid >= dcd->dc.num_regions) { + error_setg(errp, "region id is too large"); + return; + } + block_size = dcd->dc.regions[rid].block_size; + blk_bitmap = bitmap_new(dcd->dc.regions[rid].len / block_size); + + /* Sanity check and count the extents */ + list = records; + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = offset + dcd->dc.regions[rid].base; + + if (len == 0) { + error_setg(errp, "extent with 0 length is not allowed"); + return; + } + + if (offset % block_size || len % block_size) { + error_setg(errp, "dpa or len is not aligned to region block size"); + return; + } + + if (offset + len > dcd->dc.regions[rid].len) { + error_setg(errp, "extent range is beyond the region end"); + return; + } + + /* No duplicate or overlapped extents are allowed */ + if (test_any_bits_set(blk_bitmap, offset / block_size, + len / block_size)) { + error_setg(errp, "duplicate or overlapped extents are detected"); + return; + } + bitmap_set(blk_bitmap, offset / block_size, len / block_size); + + if (type == DC_EVENT_RELEASE_CAPACITY) { + if (cxl_extent_groups_overlaps_dpa_range(&dcd->dc.extents_pending, + dpa, len)) { + error_setg(errp, + "cannot release extent with pending DPA range"); + return; + } + if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, dpa, len)) { + error_setg(errp, + "cannot release extent with non-existing DPA range"); + return; + } + } else if (type == DC_EVENT_ADD_CAPACITY) { + if (cxl_extents_overlaps_dpa_range(&dcd->dc.extents, dpa, len)) { + error_setg(errp, + "cannot add DPA already accessible to the same LD"); + return; + } + if (cxl_extent_groups_overlaps_dpa_range(&dcd->dc.extents_pending, + dpa, len)) { + error_setg(errp, + "cannot add DPA again while still pending"); + return; + } + } + list = list->next; + num_extents++; + } + + /* Create extent list for event being passed to host */ + i = 0; + list = records; + extents = g_new0(CXLDCExtentRaw, num_extents); + while (list) { + offset = list->value->offset; + len = list->value->len; + dpa = dcd->dc.regions[rid].base + offset; + + extents[i].start_dpa = dpa; + extents[i].len = len; + memset(extents[i].tag, 0, 0x10); + extents[i].shared_seq = 0; + if (type == DC_EVENT_ADD_CAPACITY) { + group = cxl_insert_extent_to_extent_group(group, + extents[i].start_dpa, + extents[i].len, + extents[i].tag, + extents[i].shared_seq); + } + + list = list->next; + i++; + } + if (group) { + cxl_extent_group_list_insert_tail(&dcd->dc.extents_pending, group); + } + + /* + * CXL r3.1 section 8.2.9.2.1.6: Dynamic Capacity Event Record + * + * All Dynamic Capacity event records shall set the Event Record Severity + * field in the Common Event Record Format to Informational Event. All + * Dynamic Capacity related events shall be logged in the Dynamic Capacity + * Event Log. + */ + cxl_assign_event_header(hdr, &dynamic_capacity_uuid, flags, sizeof(dCap), + cxl_device_get_timestamp(&dcd->cxl_dstate)); + + dCap.type = type; + /* FIXME: for now, validity flag is cleared */ + dCap.validity_flags = 0; + stw_le_p(&dCap.host_id, hid); + /* only valid for DC_REGION_CONFIG_UPDATED event */ + dCap.updated_region_id = 0; + dCap.flags = 0; + for (i = 0; i < num_extents; i++) { + memcpy(&dCap.dynamic_capacity_extent, &extents[i], + sizeof(CXLDCExtentRaw)); + + if (i < num_extents - 1) { + /* Set "More" flag */ + dCap.flags |= BIT(0); + } + + if (cxl_event_insert(&dcd->cxl_dstate, enc_log, + (CXLEventRecordRaw *)&dCap)) { + cxl_event_irq_assert(dcd); + } + } +} + +void qmp_cxl_add_dynamic_capacity(const char *path, uint16_t hid, + uint8_t sel_policy, uint8_t region_id, + const char *tag, + CXLDCExtentRecordList *records, + Error **errp) +{ + enum { + CXL_SEL_POLICY_FREE, + CXL_SEL_POLICY_CONTIGUOUS, + CXL_SEL_POLICY_PRESCRIPTIVE, + CXL_SEL_POLICY_ENABLESHAREDACCESS, + }; + switch (sel_policy) { + case CXL_SEL_POLICY_PRESCRIPTIVE: + qmp_cxl_process_dynamic_capacity_prescriptive(path, hid, + DC_EVENT_ADD_CAPACITY, + region_id, records, errp); + return; + default: + error_setg(errp, "Selection policy not supported"); + return; + } +} + +#define REMOVAL_POLICY_MASK 0xf +#define REMOVAL_POLICY_PRESCRIPTIVE 1 +#define FORCED_REMOVAL_BIT BIT(4) + +void qmp_cxl_release_dynamic_capacity(const char *path, uint16_t hid, + uint8_t flags, uint8_t region_id, + const char *tag, + CXLDCExtentRecordList *records, + Error **errp) +{ + CXLDCEventType type = DC_EVENT_RELEASE_CAPACITY; + + if (flags & FORCED_REMOVAL_BIT) { + /* TODO: enable forced removal in the future */ + type = DC_EVENT_FORCED_RELEASE_CAPACITY; + error_setg(errp, "Forced removal not supported yet"); + return; + } + + switch (flags & REMOVAL_POLICY_MASK) { + case REMOVAL_POLICY_PRESCRIPTIVE: + qmp_cxl_process_dynamic_capacity_prescriptive(path, hid, type, + region_id, records, errp); + return; + default: + error_setg(errp, "Removal policy not supported"); + return; + } +} + static void ct3_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c index 3e1851e32b..810685e0d5 100644 --- a/hw/mem/cxl_type3_stubs.c +++ b/hw/mem/cxl_type3_stubs.c @@ -67,3 +67,23 @@ void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type, { error_setg(errp, "CXL Type 3 support is not compiled in"); } + +void qmp_cxl_add_dynamic_capacity(const char *path, + uint16_t hid, + uint8_t sel_policy, + uint8_t region_id, + const char *tag, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} + +void qmp_cxl_release_dynamic_capacity(const char *path, uint16_t hid, + uint8_t flags, uint8_t region_id, + const char *tag, + CXLDCExtentRecordList *records, + Error **errp) +{ + error_setg(errp, "CXL Type 3 support is not compiled in"); +} diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index df3511e91b..c69ff6b5de 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -443,6 +443,12 @@ typedef struct CXLDCExtent { } CXLDCExtent; typedef QTAILQ_HEAD(, CXLDCExtent) CXLDCExtentList; +typedef struct CXLDCExtentGroup { + CXLDCExtentList list; + QTAILQ_ENTRY(CXLDCExtentGroup) node; +} CXLDCExtentGroup; +typedef QTAILQ_HEAD(, CXLDCExtentGroup) CXLDCExtentGroupList; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ @@ -494,6 +500,7 @@ struct CXLType3Dev { */ uint64_t total_capacity; /* 256M aligned */ CXLDCExtentList extents; + CXLDCExtentGroupList extents_pending; uint32_t total_extent_count; uint32_t ext_list_gen_seq; @@ -555,4 +562,19 @@ CXLDCRegion *cxl_find_dc_region(CXLType3Dev *ct3d, uint64_t dpa, uint64_t len); void cxl_remove_extent_from_extent_list(CXLDCExtentList *list, CXLDCExtent *extent); +void cxl_insert_extent_to_extent_list(CXLDCExtentList *list, uint64_t dpa, + uint64_t len, uint8_t *tag, + uint16_t shared_seq); +bool test_any_bits_set(const unsigned long *addr, unsigned long nr, + unsigned long size); +bool cxl_extents_contains_dpa_range(CXLDCExtentList *list, + uint64_t dpa, uint64_t len); +CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, + uint64_t dpa, + uint64_t len, + uint8_t *tag, + uint16_t shared_seq); +void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, + CXLDCExtentGroup *group); +void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list); #endif diff --git a/include/hw/cxl/cxl_events.h b/include/hw/cxl/cxl_events.h index 5170b8dbf8..38cadaa0f3 100644 --- a/include/hw/cxl/cxl_events.h +++ b/include/hw/cxl/cxl_events.h @@ -166,4 +166,22 @@ typedef struct CXLEventMemoryModule { uint8_t reserved[0x3d]; } QEMU_PACKED CXLEventMemoryModule; +/* + * CXL r3.1 section Table 8-50: Dynamic Capacity Event Record + * All fields little endian. + */ +typedef struct CXLEventDynamicCapacity { + CXLEventRecordHdr hdr; + uint8_t type; + uint8_t validity_flags; + uint16_t host_id; + uint8_t updated_region_id; + uint8_t flags; + uint8_t reserved2[2]; + uint8_t dynamic_capacity_extent[0x28]; /* defined in cxl_device.h */ + uint8_t reserved[0x18]; + uint32_t extents_avail; + uint32_t tags_avail; +} QEMU_PACKED CXLEventDynamicCapacity; + #endif /* CXL_EVENTS_H */ diff --git a/qapi/cxl.json b/qapi/cxl.json index 4281726dec..2dcf03d973 100644 --- a/qapi/cxl.json +++ b/qapi/cxl.json @@ -361,3 +361,72 @@ ## {'command': 'cxl-inject-correctable-error', 'data': {'path': 'str', 'type': 'CxlCorErrorType'}} + +## +# @CXLDCExtentRecord: +# +# Record of a single extent to add/release +# +# @offset: offset to the start of the region where the extent to be operated +# @len: length of the extent +# +# Since: 9.1 +## +{ 'struct': 'CXLDCExtentRecord', + 'data': { + 'offset':'uint64', + 'len': 'uint64' + } +} + +## +# @cxl-add-dynamic-capacity: +# +# Command to start add dynamic capacity extents flow. The device will +# have to acknowledged the acceptance of the extents before they are usable. +# +# @path: CXL DCD canonical QOM path +# @hid: host id +# @selection-policy: policy to use for selecting extents for adding capacity +# @region-id: id of the region where the extent to add +# @tag: Context field +# @extents: Extents to add +# +# Since : 9.1 +## +{ 'command': 'cxl-add-dynamic-capacity', + 'data': { 'path': 'str', + 'hid': 'uint16', + 'selection-policy': 'uint8', + 'region-id': 'uint8', + 'tag': 'str', + 'extents': [ 'CXLDCExtentRecord' ] + } +} + +## +# @cxl-release-dynamic-capacity: +# +# Command to start release dynamic capacity extents flow. The host will +# need to respond to indicate that it has released the capacity before it +# is made unavailable for read and write and can be re-added. +# +# @path: CXL DCD canonical QOM path +# @hid: host id +# @flags: bit[3:0] for removal policy, bit[4] for forced removal, bit[5] for +# sanitize on release, bit[7:6] reserved +# @region-id: id of the region where the extent to release +# @tag: Context field +# @extents: Extents to release +# +# Since : 9.1 +## +{ 'command': 'cxl-release-dynamic-capacity', + 'data': { 'path': 'str', + 'hid': 'uint16', + 'flags': 'uint8', + 'region-id': 'uint8', + 'tag': 'str', + 'extents': [ 'CXLDCExtentRecord' ] + } +} From patchwork Thu Apr 18 23:11:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635523 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 130672AF0F for ; Thu, 18 Apr 2024 23:30:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483033; cv=none; b=rf6yatWbKEW2avtBfIJ1JL+y3hLvbj4J41mpQakfm5h6bdZKSB0Ld3Na7NVW83YN2xFZjw1/Pk398rEdV1M8bpajT7DG3ZxuG1zDubJfodzRHSH6K6Nls7qR+4tyM6thIBLkoBm56gy6PvKv2m/4PFNWtLOOoYMyPP7/VNcHrR0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483033; c=relaxed/simple; bh=nX6NN1JMMIifZVe1ytCAUoFcY+OVO0vFG4XhmY+0E9Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BvxrNGpYL4NANcldZyJD0wL6+iXtuzWb4FNrSFMZIdZR91Buc12HDsp9sTp5hfRpN59ssF6Lz8w33lzFMhQ7paHbKBCxDASpxPAeEGYtzXCWAuRodCCqm28zterxJT79Lw8wbhLRG4nPXAStiXNilH1e6FFvRgtg+OfPVVBCI1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Zzup0c1r; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Zzup0c1r" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6ed112c64beso1325276b3a.1 for ; Thu, 18 Apr 2024 16:30:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483031; x=1714087831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bRPAB1YOpQKw+pS55H92+TPaTEHmv3/u2yl2k1MvmB8=; b=Zzup0c1rRaWy1rRsfZQqSlQ5+qQTM64UliX4D5+l2RG1Koy4yTWRAF5OMFardjMQWW eX5yKMI7WsypegNEJ4Z0nFA7EQKZISGDVgSXuCvzfes5UOC9lMVkWokLZ3RQjdPuFuDQ uyR8/OOZkAtGR1Mnxc1MUVOXD2fT5TpCLIAUoex127mHdfr5kYe3G/gnXRD99cicv6Je Vgd4/MFopaG8dyEE+Y4JpT7ae77z3m/6J7yBkA7Kz5YYyqyOSbxvL8jLyANBm/XRUwL0 223QuN5rytx7cF7XNqTc0TEXfNZp6BGMXaaEFuIvwlm03cwssg86yYIOf5rmlRBlUM8A CaMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483031; x=1714087831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bRPAB1YOpQKw+pS55H92+TPaTEHmv3/u2yl2k1MvmB8=; b=cV9Twq4sRHHVNC/kYwRFd3IguKfFw+XbiqmK5vd80BnpLI/Chq+2MvgNoZnzEhzt62 f5UBUYlM9ycY3aiFtGQ20c8L4tLbGXeK4w7oC/BUPBZbvZJ1YX9yeaK7hKPqpCl6uDZA fcGp3GgxdhCq6buS+D/k2KOicZCijAHO0BjLk83ZSihD4HraQts9rP6nhyFTVDtUsNm7 3W82yhPwwmn4gcnaJHX29l61jZwO4/F2wxaCHICA1lxrzthcO4sIKQiySiwb0KMtNMdD Pj/wI5ICz1loAe1XQBsRlEzKK6/EnX5+uU3GjVGBgyL+ipt/9Crqh5iKGkcTEXhIWjUo PxBQ== X-Forwarded-Encrypted: i=1; AJvYcCVr57zTk10fZNoMNrvFxza5uptnojHG1asCUNCIM+nPIz9LWbS9SlZT2pyUd/bUzbMoZtuqT71P7TRQn8Aw7+R1NuPwJCn4fRZ6 X-Gm-Message-State: AOJu0YxemHePKYpHlD1ScuOK5lPmcccjloA9ZDYYHwlg9L22tubUPV4r 2dhP1pHiDNfIAVr733QS7lRxAwYH81AlhGuWDUXMqyvuu7YEGQVc X-Google-Smtp-Source: AGHT+IF0Bt8e8yYUh0X/UQv06dFjIPUQb5a3GzdZXIDmarEQcfM3YSVqkNKOCkTKcchLIAupP2339Q== X-Received: by 2002:a05:6a20:7b26:b0:1a7:8127:c919 with SMTP id s38-20020a056a207b2600b001a78127c919mr621804pzh.43.1713483031305; Thu, 18 Apr 2024 16:30:31 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:31 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 10/12] hw/mem/cxl_type3: Add DPA range validation for accesses to DC regions Date: Thu, 18 Apr 2024 16:11:01 -0700 Message-ID: <20240418232902.583744-11-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni All DPA ranges in the DC regions are invalid to access until an extent covering the range has been successfully accepted by the host. A bitmap is added to each region to record whether a DC block in the region has been backed by a DC extent. Each bit in the bitmap represents a DC block. When a DC extent is accepted, all the bits representing the blocks in the extent are set, which will be cleared when the extent is released. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 3 ++ hw/mem/cxl_type3.c | 76 +++++++++++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 7 ++++ 3 files changed, 86 insertions(+) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 3569902e9e..57f1ce9cce 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1655,6 +1655,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, cxl_insert_extent_to_extent_list(extent_list, dpa, len, NULL, 0); ct3d->dc.total_extent_count += 1; + ct3_set_region_block_backed(ct3d, dpa, len); } /* Remove the first extent group in the pending list*/ cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); @@ -1813,10 +1814,12 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, * list and update the extent count; */ QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { + ct3_clear_region_block_backed(ct3d, ent->start_dpa, ent->len); cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); } copy_extent_list(&ct3d->dc.extents, &updated_list); QTAILQ_FOREACH_SAFE(ent, &updated_list, node, ent_next) { + ct3_set_region_block_backed(ct3d, ent->start_dpa, ent->len); cxl_remove_extent_from_extent_list(&updated_list, ent); } ct3d->dc.total_extent_count = updated_list_size; diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index e892b3de7b..a3e1a5de25 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -665,6 +665,7 @@ static bool cxl_create_dc_regions(CXLType3Dev *ct3d, Error **errp) .flags = 0, }; ct3d->dc.total_capacity += region->len; + region->blk_bitmap = bitmap_new(region->len / region->block_size); } QTAILQ_INIT(&ct3d->dc.extents); QTAILQ_INIT(&ct3d->dc.extents_pending); @@ -676,6 +677,8 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; CXLDCExtentGroup *group, *group_next; + int i; + CXLDCRegion *region; QTAILQ_FOREACH_SAFE(ent, &ct3d->dc.extents, node, ent_next) { cxl_remove_extent_from_extent_list(&ct3d->dc.extents, ent); @@ -688,6 +691,11 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) } g_free(group); } + + for (i = 0; i < ct3d->dc.num_regions; i++) { + region = &ct3d->dc.regions[i]; + g_free(region->blk_bitmap); + } } static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) @@ -920,6 +928,70 @@ static void ct3_exit(PCIDevice *pci_dev) } } +/* + * Mark the DPA range [dpa, dap + len - 1] to be backed and accessible. This + * happens when a DC extent is added and accepted by the host. + */ +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + bitmap_set(region->blk_bitmap, (dpa - region->base) / region->block_size, + len / region->block_size); +} + +/* + * Check whether the DPA range [dpa, dpa + len - 1] is backed with DC extents. + * Used when validating read/write to dc regions + */ +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return false; + } + + nr = (dpa - region->base) / region->block_size; + nbits = DIV_ROUND_UP(len, region->block_size); + /* + * if bits between [dpa, dpa + len) are all 1s, meaning the DPA range is + * backed with DC extents, return true; else return false. + */ + return find_next_zero_bit(region->blk_bitmap, nr + nbits, nr) == nr + nbits; +} + +/* + * Mark the DPA range [dpa, dap + len - 1] to be unbacked and inaccessible. + * This happens when a dc extent is released by the host. + */ +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len) +{ + CXLDCRegion *region; + uint64_t nbits; + long nr; + + region = cxl_find_dc_region(ct3d, dpa, len); + if (!region) { + return; + } + + nr = (dpa - region->base) / region->block_size; + nbits = len / region->block_size; + bitmap_clear(region->blk_bitmap, nr, nbits); +} + static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa) { int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO; @@ -1024,6 +1096,10 @@ static int cxl_type3_hpa_to_as_and_dpa(CXLType3Dev *ct3d, *as = &ct3d->hostpmem_as; *dpa_offset -= vmr_size; } else { + if (!ct3_test_region_block_backed(ct3d, *dpa_offset, size)) { + return -ENODEV; + } + *as = &ct3d->dc.host_dc_as; *dpa_offset -= (vmr_size + pmr_size); } diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c69ff6b5de..0a4fcb2800 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -456,6 +456,7 @@ typedef struct CXLDCRegion { uint64_t block_size; uint32_t dsmadhandle; uint8_t flags; + unsigned long *blk_bitmap; } CXLDCRegion; struct CXLType3Dev { @@ -577,4 +578,10 @@ CXLDCExtentGroup *cxl_insert_extent_to_extent_group(CXLDCExtentGroup *group, void cxl_extent_group_list_insert_tail(CXLDCExtentGroupList *list, CXLDCExtentGroup *group); void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list); +void ct3_set_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); +bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa, + uint64_t len); #endif From patchwork Thu Apr 18 23:11:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635524 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE9E371B3D for ; Thu, 18 Apr 2024 23:30:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483040; cv=none; b=qFKxhQDEV/6Sw3uURgG/eflr3s9A9nbG/b1gMUOvwNqIulMx3HMKpkKryu/GDi5L8S8pnw7WJ16SbjFJmbK81onq27YA6htwG8DzP6kaXCNRZgOHgP9fRYQ/9xFIAqkjG9xV1ZN+yWe2DPN7rtL9K/H5zBLyb4aLqC7kC/B7YvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483040; c=relaxed/simple; bh=fKQfvxmJJEaAzzudRynJlBCu52ZiuXIxL0lJkEGYdj0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qu9XEWrfQ9ZNkuW0loEqUw5O3tuC1xeQTYosxM+ISHKucXk3utPB6j/bDJJbatmDHa4aXKv5+EPZVFQKGVY1LhFg7H9DI9vBvwgEdw9vqHrbr5ovCUXYeQ/sfXKkICDUjVH/9t6mK9kxzP55Qyj45dheOrcIvnNYmnEokN/sLIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kWo/YWrz; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kWo/YWrz" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6f074520c8cso1461324b3a.0 for ; Thu, 18 Apr 2024 16:30:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483038; x=1714087838; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w9/QnnmOfKtdXC4mQDt19ExscjIokR+8yGloQuw3o6U=; b=kWo/YWrzMGZFIicv/TT5YuxHN4s15phHlW9IQuFMu23Bm+hz2ytchE7f+/cAbktncL A9FsLseNwawnNZuL0o+9lpgJZHVq/RYTqgFP7pAVcLPLBxAqR/L0REKzwZMyztwTc0H3 zdj5YAMOCNfBpq3U82QZym8rMf46mLDHn4HebYLEsfP7m6wBJR++/S9ai99nK34W0dma fv1jrlkbwEOzEyjTKj/rYpv/GLpWLquln/nt9sinCxMKCmOPQDpYC13YcpCaBIuRLHF4 dI9oGYVQXGnhWBgSrl0dYVPSlF5zGwfZG8yf02kyJa92UfSHcuceqoVLL2YaFQltfH3d 3MxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483038; x=1714087838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w9/QnnmOfKtdXC4mQDt19ExscjIokR+8yGloQuw3o6U=; b=jHpB2MpsHOoB0CU/WWA7yz4seXATFOun1q5vmoF7mICLPBIkSlx6otgRNJS/7WuXR5 LF0Ih8b+0vcBYHVCohubQwwp6B8yX6hT2Whg/o75/Co0VEwQCNrLzW87L5FVucI+JQZp D2Ko7yRZC4f8q5ABvKQLLEfuykagmnsdv4/8Q20tgrHhuAOXRiRzmLnpLAuC1xW3MkUw vE+m2mgbInSb8HkKr/BPGqqDxOZ7o5hZuI0mkz/sRY4NRQu8sPYH1AZwZOrNnFCD0Tvf goQjzhUvhjVdHrR2xiQZFxMGhSXUBBfZrgthU6eV53P3Dwf2b+H5flxdvW4xJnmckf3j MWTA== X-Forwarded-Encrypted: i=1; AJvYcCVd97VyTxipqrFcu3/ODtFLY6hasOblAkQUtG3ly2Ein5tsM8t2U4HwXYVAv/H/F35jv9UK0J8mNoeuszQ+oH+NregE0wQhD4Wa X-Gm-Message-State: AOJu0YxhZTgGbLxfPYzp1/Eb9Dv6dGNHPhPDsPANHYsWW4oR2S//uG/K bcmu/wfIctxm+8zoIH9bst1QrXwqeqtxht2fNg/7sA1svx7621RA X-Google-Smtp-Source: AGHT+IFxmyePkARZkckg2WodGL2yH1e7nz5yCu71kjTaoavsu6MpQP1H/JH9iZsF6BUah9nvlvDQTQ== X-Received: by 2002:a05:6a00:1a8e:b0:6ee:1d68:d33 with SMTP id e14-20020a056a001a8e00b006ee1d680d33mr644097pfv.15.1713483038197; Thu, 18 Apr 2024 16:30:38 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:37 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni Subject: [PATCH v7 11/12] hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox support Date: Thu, 18 Apr 2024 16:11:02 -0700 Message-ID: <20240418232902.583744-12-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni With the change, we extend the extent release mailbox command processing to allow more flexible release. As long as the DPA range of the extent to release is covered by accepted extent(s) in the device, the release can be performed. Signed-off-by: Fan Ni Reviewed-by: Gregory Price --- hw/cxl/cxl-mailbox-utils.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 57f1ce9cce..89f0ab8116 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -1704,6 +1704,13 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, dpa = in->updated_entries[i].start_dpa; len = in->updated_entries[i].len; + /* Check if the DPA range is not fully backed with valid extents */ + if (!ct3_test_region_block_backed(ct3d, dpa, len)) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + + /* After this point, extent overflow is the only error can happen */ while (len > 0) { QTAILQ_FOREACH(ent, updated_list, node) { range_init_nofail(&range, ent->start_dpa, ent->len); @@ -1718,14 +1725,7 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, if (range_contains(&range, dpa + len - 1)) { len2 = ent_start_dpa + ent_len - dpa - len; } else { - /* - * TODO: we reject the attempt to remove an extent - * that overlaps with multiple extents in the device - * for now. We will allow it once superset release - * support is added. - */ - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; + dpa = ent_start_dpa + ent_len; } len_done = ent_len - len1 - len2; @@ -1752,14 +1752,9 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, } len -= len_done; - /* len == 0 here until superset release is added */ break; } } - if (len) { - ret = CXL_MBOX_INVALID_PA; - goto free_and_exit; - } } } free_and_exit: From patchwork Thu Apr 18 23:11:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Ni X-Patchwork-Id: 13635525 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7724458119 for ; Thu, 18 Apr 2024 23:30:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483043; cv=none; b=dw5JnZCx8ebiaVBzwUR5n/EWUjPtJiGwGrMPbBEkNsz8kWvymenttOosl4Tj+EOJUp1kRDXncSidgikD4JObKtTlZ/6omgrj24R2h34gJZJ/Y6p/rz4jXUVW+lp31UqT+FnqLaf7itLBTh5zTE5T5txPPf3s21XsleFPlCz8rhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713483043; c=relaxed/simple; bh=4/sZTtNVoOLJvZFPx0uxL1WcMwdkqpnwinTIQ7phZZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eNxmnxqaQd9bfMHSLndAUy6k5UItWevKA+xwH7Z9zyTktvW4IiMmfVkS6but15BtOE7w/ugT2R7l31bGpxf+4K4yc8lZMwbtidoil6jPwnc0Qr9VyrWhrDkcriHFt6LU9oo0XMkI1Yt23126Nh6/W93NDYUqJi4cYOaMEuy91rg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hvcjjcVs; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hvcjjcVs" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6ed691fb83eso1299522b3a.1 for ; Thu, 18 Apr 2024 16:30:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713483042; x=1714087842; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KdtJQ2//BvaL+ED8X5UWARoz6dYMy43pJ3SbcFIRv9I=; b=hvcjjcVs2YTt54+OCs6Uxwri44NHTleh/Q1kN5CoiB+5xfeqb1uA+XWogHK0YxUszi MUp8CZAN07KE5PNgSzNcvMXsHpIt4NtFbhyRjXK5P2Xg1/En7SdX2YAdcGVPDzC1eQ2p jg7vv8POG0kvyR//asQ+qdBv5PB1p8nwWOJ8BJlCZErN729lJSHdikjTCukHgvVx2dOQ LHt89b9F/i5HDmZgJXGWUGdG5WPjV2ptEv4QOhwZMa6zrdZ9TjCQouI9ojnAzj07eem5 0KdG6SHVHaA3iJUW4jDPHpZVNoP4QYb51iJTkifNFG7G4SYh1F6OKRYs43tRHcqWlJ0v u4pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713483042; x=1714087842; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KdtJQ2//BvaL+ED8X5UWARoz6dYMy43pJ3SbcFIRv9I=; b=O14JzshS229a33ajFMHyybCVMsNwc4TIJxigTyBNuH2i14cstWxqKfHmwNX2HA3Z3B RXlWgq6rGIJLradiPTCyUUYOM6wkSrL5UpD6o0L0E1k1Nf2EHZII9+AwLuitvDHajbzX pjQWIGYfknLKdQRfJkgghBldLK7mR13rqysxxs5arQ0Vw0Pp9uSh8/vl2erzikVZLjJ3 ekryqPyCiQxpDm+xNZo/Ly9zc7wnPlFRP0ZY/roL30jhIlraznuB4kaGIDmjtD9o5u4E 6+ckO8sx49bQh2DcRpWPC9URB2pWzjBK52wrwZF9QcKgdpE+0Ts8u/6+4Oa1sQIp0lah iJ6A== X-Forwarded-Encrypted: i=1; AJvYcCVlrmRnTXXUcuR8IQiOdLc6hfezsV/nEm6IfOvxTvU+1N1+BMbj4pLny1PpoyMno0klZjyu8D7iQU2yqvD8Ax2KiFeT5HLGHdso X-Gm-Message-State: AOJu0YwnIySCL7oKS9x2W731g2JvZqEOaRSKWTbxGrAKAZvuUI096v4a e+f3ZMiC6/9zar3t4K4P3Cl0TKkmpuW8iVZcx9dbpwllVrtlQfVM6YkcCg== X-Google-Smtp-Source: AGHT+IHGuUAdd1A/hFHKB74AL5T6aGZ9tFOxeEB/HZhO5eI3UqC99l2TIAslMMUB9EHdcx6813y6bg== X-Received: by 2002:a62:81c6:0:b0:6f0:c78f:75e1 with SMTP id t189-20020a6281c6000000b006f0c78f75e1mr498855pfd.0.1713483041813; Thu, 18 Apr 2024 16:30:41 -0700 (PDT) Received: from localhost.localdomain ([2601:641:300:14de:ed8b:f40f:7543:e9ea]) by smtp.gmail.com with ESMTPSA id h3-20020a056a00230300b006e6be006637sm2040783pfh.135.2024.04.18.16.30.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 16:30:41 -0700 (PDT) From: nifan.cxl@gmail.com X-Google-Original-From: fan.ni@samsung.com To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, gregory.price@memverge.com, ira.weiny@intel.com, dan.j.williams@intel.com, a.manzanares@samsung.com, dave@stgolabs.net, nmtadam.samsung@gmail.com, nifan.cxl@gmail.com, jim.harris@samsung.com, Jorgen.Hansen@wdc.com, wj28.lee@gmail.com, Fan Ni , Jonathan Cameron Subject: [PATCH v7 12/12] hw/mem/cxl_type3: Allow to release extent superset in QMP interface Date: Thu, 18 Apr 2024 16:11:03 -0700 Message-ID: <20240418232902.583744-13-fan.ni@samsung.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240418232902.583744-1-fan.ni@samsung.com> References: <20240418232902.583744-1-fan.ni@samsung.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Fan Ni Before the change, the QMP interface used for add/release DC extents only allows to release an extent whose DPA range is contained by a single accepted extent in the device. With the change, we relax the constraints. As long as the DPA range of the extent is covered by accepted extents, we allow the release. Reviewed-by: Jonathan Cameron Signed-off-by: Fan Ni --- hw/mem/cxl_type3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index a3e1a5de25..9e725647f1 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -1941,7 +1941,7 @@ static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, "cannot release extent with pending DPA range"); return; } - if (!cxl_extents_contains_dpa_range(&dcd->dc.extents, dpa, len)) { + if (!ct3_test_region_block_backed(dcd, dpa, len)) { error_setg(errp, "cannot release extent with non-existing DPA range"); return;