From patchwork Wed Apr 24 20:04:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Svetly Todorov X-Patchwork-Id: 13642455 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2128.outbound.protection.outlook.com [40.107.223.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5978913BC3A for ; Wed, 24 Apr 2024 20:04:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.128 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989070; cv=fail; b=M/tO+XSKHRuejQdtNFwfSqC8xhHS1Q3blfwC/nyIKqkxJRwmxe2LKT5soApQ8ich73+A7sm+UGbbBk1Rj9h6pYCWxK8K7cpVdoddFXjJRdH01nGrtGWRzrh+ovrzHXDbtjI+0w9BpOaO7ka2qRD1aGGcKPZEgShuYZ5HgPnK+Lk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989070; c=relaxed/simple; bh=n6uLLVpW754pheV8P0lm/gJ67umDXl4IO0f47kciLqQ=; h=From:Date:Subject:Content-Type:Message-Id:References:In-Reply-To: To:Cc:MIME-Version; b=syiXJ2JXz4gCdvues2N2k3Y3boYbSoV94i9wZGuE2TF+SkU8ONzfAhD45Oc1eI5kpWhBUm5b8te8GKRKQwnXx++DEMkNhTNNq3qj4mzqaGQTYWNff/F8HXndl4rAsR66vDdfUGvnzvMMDt6Otbjx5SdTNjfTecWmPDdcTv/MBvI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com; spf=pass smtp.mailfrom=memverge.com; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b=s/q0O0x0; arc=fail smtp.client-ip=40.107.223.128 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=memverge.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b="s/q0O0x0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xp0ICyZ1sfA4Sw2tJX0l9DT63LdbIPrxb/5SOZkurb29mCUgqvu7PheikZ9pBDGqAAyIKFt2Kjv4CpwuNTHy12bMMd5dTDqIRhwqCtkdpzDaU/BU+tncA/sTUmnIGIZQN70eQmmKwaosCPmE2S04xfS0psw5gWLf/DqAxZFJKQDw7FJgMFiw3XL80M1Labf3oSduRuUg0SlX4xjTOkm9PqcCj2ibFHvePe5+wSZwjOmpIKy+IbXopVDwQ9sy6mpIqlnVS5jjobzLuhHpMjFAQNhQxnLPJ5eyfTmlQmt0Gx355DD4VXjMbtCeGQPU0yO8kShljKxh5lw/vF3smJCXhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qnLdblsMRDSOzforEnHVVgC29eihoHuXi927zOmPpKo=; b=BM06Pbyx4NTojkaqYAkeEv83uTYt56x+UvtQua4Tc6uDquBR+j9GtUxG7z6UfCjDvh15QE8cLY8VWrzE+GcTrN5/cxcRQIlEWjt1yl/15YF71/nNYymGBtqMpRwq6Jw6nemTcJb0X965s/3REC/+PPT6WiPciZm4ECh8e1QnLkx+bD/OgllvSlRNZ62x1eSMI0oVmYPL0UggE+9hqkwi7kU9DvixVC+azq3+1QantIOQU3kv00NNP1EZj8D+jmUcuqamTHW8twKgtGHIYXhbKUSyI4TSUDS9gbtKVCMAltcJR7eZbOq6Q/bcTSWb5w1Q2xAfXsKWHWW5iqCwm+tV8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qnLdblsMRDSOzforEnHVVgC29eihoHuXi927zOmPpKo=; b=s/q0O0x0Bm7qb7c1mcqNlafTGInXxHY7yCtCNONhC/uNgJLbjSzjSMp2v8Ug2taff8VmVOJ06Qy49IHPOyl9NnCFhMOxEaUT4KphMZF/sKgeRhGaEW2ObITT+leUbh8/5tzGpdY2bXrncAtWV7JqH30qdyeZmmmVUofQKCv6bhU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=memverge.com; Received: from IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) by SN4PR17MB5846.namprd17.prod.outlook.com (2603:10b6:806:215::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Wed, 24 Apr 2024 20:04:24 +0000 Received: from IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a]) by IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a%6]) with mapi id 15.20.7519.021; Wed, 24 Apr 2024 20:04:24 +0000 From: Svetly Todorov Date: Wed, 24 Apr 2024 13:04:06 -0700 Subject: [PATCH RFC v2 1/3] cxl-mailbox-utils: move CXLUpdateDCExtentListInPl into header Message-Id: <20240424-cxl-2024-04-22-pullahead-v2-1-a87fc4597795@memverge.com> References: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> In-Reply-To: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> To: qemu-devel@nongnu.org Cc: Svetly Todorov , Gregory Price , jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, nifan.cxl@gmail.com X-Mailer: b4 0.14-dev X-Developer-Signature: v=1; a=ed25519-sha256; t=1713989063; l=2156; i=svetly.todorov@memverge.com; s=20240312; h=from:subject:message-id; bh=n6uLLVpW754pheV8P0lm/gJ67umDXl4IO0f47kciLqQ=; b=NF0R9bPEmDxA0o18wloZn9HlVXqr89v6zaKjXN7GhBjqsO7E1hEStX3kiv6ycuffKFw07/ycf FtNZ8rFXREKC0bvxfubVbR2fZK8gD3vKaLOeS710Y2e4oYIkmJrEOhb X-Developer-Key: i=svetly.todorov@memverge.com; a=ed25519; pk=bo0spdkY5tAEf+QP9ZH+jA9biE/razmOR7VcBXnymUE= X-ClientProxiedBy: BY3PR03CA0005.namprd03.prod.outlook.com (2603:10b6:a03:39a::10) To IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR17MB6347:EE_|SN4PR17MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: 776b3926-f936-4a35-912d-08dc6499bd36 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: =?utf-8?q?YWSjU841vbR5LBIC/4oJP1veOxwEWRu?= =?utf-8?q?pfdeTRyKR1H1DGh/K8W9JAoOwOFk+kKYibc4agguxF6B8kjBqzGWOgsuHbuk4orAc?= =?utf-8?q?rpO8+kYCKgKbeqBrawUZHoged/V/VFzNuHRYda+nMXRpnWTj2v5RJ6NKzpecbN1jR?= =?utf-8?q?3QgBbwtojFzO1GFm5gJuQHApUX6xsDh9+ya9g8zRtb6CRwQkBKnrLcys8FRva+RFj?= =?utf-8?q?FFsw4R9BC/ehbwXXh8cVBy6yGrLpu5IDGJqUll/1Uzs6TrMwBorW0wwf04zwsorZh?= =?utf-8?q?DN4DmGXnIbJxt/T48LXPVCRawcMiKE8Z2KzIKTxSNQ2uzY0biZCDW3iA6cpIZx5Op?= =?utf-8?q?2K00ydk2QkcJoxqbnvDWFMuPaOTv1CHUh38ZQSop0q+UdavOCImpSJh31XQDvZfxw?= =?utf-8?q?k2K08U3E3xYLZKA3naYbYFEr8FXXvkALX6lct5HMMvrTwdT72XLRZeQoMcuSp3M6b?= =?utf-8?q?rP2y7kwOtbwHMV3lpz1noJjR5/GgpkMraeqZkIoVSKurztVGGcwpyG25R+lVwmvim?= =?utf-8?q?KxOX9A+gEyOzzHV1VR8c/j+pdTlj3Y49S3+9zsIYIKCih3lWRjVu98lprMINKL9t6?= =?utf-8?q?3hLe2uPOKhjOVL0uXi5GJBIFwJUeGqaPGJ0qRgNYUr2erWiGJmtZtbEorPsthqcU8?= =?utf-8?q?7+3FCTCKZpQfL1yjjb+x8EzjZWZyQhd2v2HvHvFWXel8z8IqpFy12Bp0EL4GJ7l1x?= =?utf-8?q?zh/84Suwog4lROkxOb1iS+KIlXiC9h5ZWD7WM5VJYt/BuEUurJExW8OcLRs2KN6fh?= =?utf-8?q?6ZqdLqWjkLAX7XExA5cZ0wV5RhixsNkEbFPYOS5xzd0qwOH1Qd+Av1t8gMh3C4B7g?= =?utf-8?q?6RC638sBhUg2j0+MAdA2Yt/M6jaN107cz1BNM7PkrBwr/OPut6sx6p4vBJOdZo6fJ?= =?utf-8?q?v80Wme8L+15x4xS8yzIHM4gMj7kLVAdGhj4MI6g98HZ8C6VM418XLwhxkSMNyMdD0?= =?utf-8?q?CbquGXzmY/m5Dj3ErOxfIl3lHXHiXnyzZz6vqrJg7QCH5JVdxSlprkZX1QwsucmSK?= =?utf-8?q?xhvQ/EXoSH/H5a/5IFQqh5f933owzwKhdIpKipjNXUjKNXGJ4kF4TbKFJaDagkA43?= =?utf-8?q?sIn3YyP460bRYp7XHt/dC310hhSLFta4RWkTYVls1D0soVVvx3rf3O30N+Q5YlZPs?= =?utf-8?q?RUWYYuigz4/L//8ticU25ljFDlSN2b/sbVu8gdIHmNhn9/iY2cPDnfFRCF6i/ARRF?= =?utf-8?q?oLPMUPTeJRkLpYFaDJ5RuHujLP+d/fJ1FTCCnT64esO/R/etDnhL7HRU1LT9o+SUy?= =?utf-8?q?rjCYRzB9qullyMziYQx9vS01JYKKZ2VOm0Q=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA0PR17MB6347.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(376005)(52116005)(1800799015)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?BYqYNNgMcGCiFhskTeXBVdKlLChl?= =?utf-8?q?brRDxtTvW48rCBayqVjY9pojuB1WPMNEst6ws5wxprOCnrca/hkXqi68mgKaYKZwU?= =?utf-8?q?rHqpT+Rb7s2PjKSCC1/j9HXAV3NPj5jbyfdEyKoqrx9S3yutV68twYbe7wPJhJn4r?= =?utf-8?q?UvzNFHFGMjG5/q8BPLZSA264Zqd6IXn8vtsZbNWP0u5q+pd3EGHdA7kY17NmxleJk?= =?utf-8?q?Hx/il/5bUdt3oPsRzsBlj1/3ev42Y+5KF09HTJDya5legO8QzWqgZW3zfNNv+tvCi?= =?utf-8?q?n4+7846xA6ha7GMKilPelMrMPb6lcbsjrGXd0GReBxZP0OXbxX/2LUn7wo86pLvx9?= =?utf-8?q?izJ8r/yxwBaXsHyNX+i15ClTWv/2gigG9sQ9EMlyuld5XNsCt4tA8R/Lr6zs0ZARi?= =?utf-8?q?htBTwqgMmq9/ZdqA7PTELIKWpL5R+hh59FU9PYXGwytYSQzWo4SAq4rz+M6QvsrTb?= =?utf-8?q?dBz6bFCqkprQrcYQ/PfbW6Vqw7/zDQdezbabk2eRS/gHuKMwSPDMu10YijdQknuWA?= =?utf-8?q?+In/E3qzfZNUF1dgkbFg/fOdKxAbTY73ajfh39Ov6ufT7pI2Z4Uv9dG0OZWqtAgtW?= =?utf-8?q?hGAT+n+2CVGsiM/qBgQRyQIxx+CZJ8f7N35YltBKZNYX8HUFZAs+DFyVRpLOsnr1B?= =?utf-8?q?lsNkCExI6qFSzQWuCrNYBR/nzJpDXh1RjCnIBbwhMx7i98mkJhHCPdLPwcK0iN2NW?= =?utf-8?q?4gCf0/dX9lVEYfVeoW5TlO38+fZ8GWcJ4P8eEtdCdlVXe9VW0MS+lElv380ydeffi?= =?utf-8?q?TWfqqIHSiR9jK3hhHdF26QUNBSTdJLOI3JhGiKT+y5SP+r5e6VVrm0vJD2x607G1n?= =?utf-8?q?4L+UjkravXjFkeKFFH33vzfemeqtNgl5CcgaUAdtVrkudMQx3oCBcfT9VptVBi3tQ?= =?utf-8?q?WRWbt3inqIw7sUXN5n7nf9RWexnYQ/pnzL9DzIPS3BFdoOsTVfRHgQZGb/5NrxCMe?= =?utf-8?q?G8Iuy40i+zZ35XrutvB4N9G3DZmZggasa88fbqx0nxDYbLCKezWNp17RLIa6y/KR1?= =?utf-8?q?gelmo2USA0XF07lm/XF/yb7Qb1hV88Z4MW9NBUeBVUQljmV1zo3qMTXVLhMFpibej?= =?utf-8?q?txf4zF1Pxmk3rjzAr8ZsYp5B0FuD35xcypCzmfhaWcIYiGYzbcZ/MtVqTYCtxPbrI?= =?utf-8?q?tSmfpyhlGlkWR0g8z/9JFjPKh6zUc7aJaiYW1f3csxNti2SdbnKVOeWQ6jqpx3vRv?= =?utf-8?q?SOlF449441pUhrPEmOyZeJYXAyBwotc1igo7aUmDrPtZ/fpAnASqBIkzh2skF+ZYU?= =?utf-8?q?vyybUZfkqmuurRdO9dTlVMYqqfWqqSksHa5/qGejvWh/13FqkflSe7l8+lZncqh2o?= =?utf-8?q?xwg9bhrzlwqFWUKboikgSyTW4TWoHNMcjEjumKwSHt7HuprzLbz5Yhar8jXz6YnWe?= =?utf-8?q?1l+H5fR7GscFeYoiQzIDjreWkjdJxGW9foIsLHEsBFb1fNBXvuYSz35RECApZNHLX?= =?utf-8?q?+2+1wmLnGoMWrDjxvnJOSFhJ/bEtaucIv3xlNTikVYcSOvrPZ9/8p0CY1tNLy7XvU?= =?utf-8?q?M2QpWeguyaYopYg33Tz/+qQ+U8iSTvrvUonS5CspxOlrEcDG0x9FAzk=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 776b3926-f936-4a35-912d-08dc6499bd36 X-MS-Exchange-CrossTenant-AuthSource: IA0PR17MB6347.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 20:04:24.7841 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zJr6xDH85fuZO9Csgwwxpqkyml6SjK0V58/Jg2X2ZA6PpIegYMzFnZIXFVlmkSLYyn8HVukkf4O2Vr840K7t51cDLOYIOJuVtXA+tE+kz4g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR17MB5846 Allows other CXL devices to access host DCD-add-response payload. --- hw/cxl/cxl-mailbox-utils.c | 16 ---------------- include/hw/cxl/cxl_device.h | 16 ++++++++++++++++ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 413673da59..dc916f5bb3 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -2184,22 +2184,6 @@ void cxl_extent_group_list_delete_front(CXLDCExtentGroupList *list) g_free(group); } -/* - * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload - * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload - */ -typedef struct CXLUpdateDCExtentListInPl { - uint32_t num_entries_updated; - uint8_t flags; - uint8_t rsvd[3]; - /* CXL r3.1 Table 8-169: Updated Extent */ - struct { - uint64_t start_dpa; - uint64_t len; - uint8_t rsvd[8]; - } QEMU_PACKED updated_entries[]; -} QEMU_PACKED CXLUpdateDCExtentListInPl; - /* * For the extents in the extent list to operate, check whether they are valid * 1. The extent should be in the range of a valid DC region; diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index cb95ccbd61..5c60342556 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -525,6 +525,22 @@ typedef struct CXLDCExtentGroup { } CXLDCExtentGroup; typedef QTAILQ_HEAD(, CXLDCExtentGroup) CXLDCExtentGroupList; +/* + * CXL r3.1 Table 8-168: Add Dynamic Capacity Response Input Payload + * CXL r3.1 Table 8-170: Release Dynamic Capacity Input Payload + */ +typedef struct CXLUpdateDCExtentListInPl { + uint32_t num_entries_updated; + uint8_t flags; + uint8_t rsvd[3]; + /* CXL r3.1 Table 8-169: Updated Extent */ + struct { + uint64_t start_dpa; + uint64_t len; + uint8_t rsvd[8]; + } QEMU_PACKED updated_entries[]; +} QEMU_PACKED CXLUpdateDCExtentListInPl; + typedef struct CXLDCRegion { uint64_t base; /* aligned to 256*MiB */ uint64_t decode_len; /* aligned to 256*MiB */ From patchwork Wed Apr 24 20:04:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Svetly Todorov X-Patchwork-Id: 13642456 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2128.outbound.protection.outlook.com [40.107.223.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D7C913C9C8 for ; Wed, 24 Apr 2024 20:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.128 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989072; cv=fail; b=RerJd6Gkw25XLWj5tfL478xYYZlpvX3lHiR8FCn4jCFQV+UmJzPgqwCxghkL70IW8YUpJXDKqyzQDd/bvouf0mleCoOoAytrhojv5OAm2i2m+Q5eskQflpmySdRVRo0bG9A89PS0teSeEn6xj9kgD2GMlBl4uwza6cEeoBgIAjY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989072; c=relaxed/simple; bh=a3chGYz40NH+1tOj9J+3vAQ33JJx6r5pMZAwyiD1jvw=; h=From:Date:Subject:Content-Type:Message-Id:References:In-Reply-To: To:Cc:MIME-Version; b=AQKOyNUWQXTE95m2stjHzijtrYZ27AJr34P/rBQi1rDfj7b9YXCLv71EA8nm66qRKSniq3uW8OyXEkK13GOgcqrSlWCg2HVaMYXH2DwwQglm03hh6qFcsItlRX9Q9d9EGtaiX4Q8+B+AQx1kDoLHpCx6hYche9RQOgkq7IqHXZ0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com; spf=pass smtp.mailfrom=memverge.com; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b=oFkMnGja; arc=fail smtp.client-ip=40.107.223.128 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=memverge.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b="oFkMnGja" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g55Kk5uHkPEg9DU3RuzM0fiMyRbsuRWMwmddOqEUHYt2/ExqQoqNhtVVvzlqAFuf8nWDBzMK7zJCiPgSwraMrQAwnfOoE1ZR+g/v4pp4t+Nmm7TZcQj07sbrxdybeYOJswGdTYrVdn8R94tzr1W7YXzqQ7EePoYI+6v4YOR8f8aMjvHKNt6aosUj4hNklL12qNtNSzU2UxmgVQ7dt+D6pUMV0G/gu9zOV3zOt/BRWsF26ZZRVSacxX8FKUju0lrqTeK5XxwUpFZVl4EpUCxd+oazw1dYqDyeK6s+qgaB4UbBvSKUuohrRn5vS0QVYlZvmYGWrEyOjq3m5Jkm4Py6gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=T5M1rRrfg/O/7uLkDuQ5P9LfaGKsdcr+uqSTJGUCooQ=; b=k6vimMy+JjBXNWZv6URmHqMt3UaGbxLwSHwXNZUzMlxlgW+KZcOdCW7flwAGFYlD1/2BwGOj3lq+J1sVnXn/FVHylt/Arw+d2EtcoVYyP/IyQoEf9e1CGfukCpkezaufanW8b+lcuBzu7/jX+ESkC6jp4HZdEXcBHBohGRr0FUnFnvceRUeyGX0ZXOSb5nTukLhE8Hk64pIBnS3R5sW/QGlM0ty016ZcXuOmujOEOxUJhaMvnSq5LPEJr4rPBC2/21wztfmKbIIbMd7eLwMKDMput0+MFr4bccFwOycHO8JNtFF8jve7ZZlKEmZx/w2R26F2dJt7mU5n44+i3LuxJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T5M1rRrfg/O/7uLkDuQ5P9LfaGKsdcr+uqSTJGUCooQ=; b=oFkMnGja1Qit5IPpbn5fSztELnfFyvTC+fpDQty/V3tTWrHWtbkeOFFYua3aEGqZgU4mJSype77lpPxE5DIvvZ4NjEmJP7ePqmqrBJsJUR4n/fsVUlvNIwYp35OjiBvomH8bB+GWHK0daJVsAAY3Rjbzyxnb85hiKorJmMnhAYg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=memverge.com; Received: from IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) by SN4PR17MB5846.namprd17.prod.outlook.com (2603:10b6:806:215::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Wed, 24 Apr 2024 20:04:26 +0000 Received: from IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a]) by IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a%6]) with mapi id 15.20.7519.021; Wed, 24 Apr 2024 20:04:26 +0000 From: Svetly Todorov Date: Wed, 24 Apr 2024 13:04:07 -0700 Subject: [PATCH RFC v2 2/3] cxl_type3: add MHD callbacks Message-Id: <20240424-cxl-2024-04-22-pullahead-v2-2-a87fc4597795@memverge.com> References: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> In-Reply-To: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> To: qemu-devel@nongnu.org Cc: Svetly Todorov , Gregory Price , jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, nifan.cxl@gmail.com X-Mailer: b4 0.14-dev X-Developer-Signature: v=1; a=ed25519-sha256; t=1713989063; l=7417; i=svetly.todorov@memverge.com; s=20240312; h=from:subject:message-id; bh=a3chGYz40NH+1tOj9J+3vAQ33JJx6r5pMZAwyiD1jvw=; b=VpwbNRGlx9XMBlytQJuynsx3S9ft/AxXCRbQnomkFCz1/57Xjli6amTmUr5nbsFya0f7TpTL0 aJftVMpL9wnAjmHyu6jsbWTqTE03Spk66dEWfiujGFuR+5YIlkjLUrb X-Developer-Key: i=svetly.todorov@memverge.com; a=ed25519; pk=bo0spdkY5tAEf+QP9ZH+jA9biE/razmOR7VcBXnymUE= X-ClientProxiedBy: BY3PR03CA0005.namprd03.prod.outlook.com (2603:10b6:a03:39a::10) To IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR17MB6347:EE_|SN4PR17MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: dc54affa-a2bf-4ed4-3a65-08dc6499bdaf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: =?utf-8?q?4jrx2lI2TQ8n8cpXbqrgNpoQrpshmy6?= =?utf-8?q?H3n91sPJDlj2V0dvXt+ah3dvyM/sMW4IMaMls9opX7pL5fSLC/uWPWKyGu893Fo+s?= =?utf-8?q?YDdkV1QV3WwuYFMEsRWSZx3RleJdI3nSJhiBDiOJu8Aj5STqZyL4ChVMKkZgLJl3F?= =?utf-8?q?TWh29P1KVObFbGs2to4cClDqwMu0EKaMEovxM52krtd5QOr5pGxMAlARGe205IhLS?= =?utf-8?q?0Y3erGzZNjSGQ/obKl0VJSRhWdiyfXgbJNiljjDCMzedxgHJfEHPYlWg409v0Gk6w?= =?utf-8?q?vnEL16Nfvn/dCl13kb8v5ZFAUL9aFD6ez3VGro3v+90sMg+MqBFcPNZB1fe3X0lJN?= =?utf-8?q?c0nhNCSWzrpWz6Zv/x1G0v1PT2osirRUIl5ktXZAa9Nugm/mMkNrTHyhUbQxGmHYG?= =?utf-8?q?GXPavqqgBuBh6JWaSWU2kFdkm6TrLJEygJkzqRcjFBpRkXFer+nymqgQ8W6RczL0r?= =?utf-8?q?mILICpSKVY5qY+w9XUr5j1eq1iZlPJlOTwLY7x3m9Qya+lYnOwEfgAREXyAIHoLwT?= =?utf-8?q?LfoImagI5UPwqP/Z2YWvEij7ecKmGTIHYFJLIGNVin+4Ccoj3CZ5dyr/WDOo/H0e6?= =?utf-8?q?nT6m9RKawrM3xzSWdoKwHzqE5zY9i2i8tfnXRx17iyFdhSvHba1rCLE26qx3hf9j2?= =?utf-8?q?iW46nYB2pzlA0sxIFD3dQN/z7C5mw89vfT4E+x8KuzA+s4uw2rr8N8PCzwHMFoXIP?= =?utf-8?q?2kPLfhd6f6yUnVLLGzcmNc5/ZVMVHlG2L1iexlOoM7MM2Q42fnGsuPtz2yeyQbBxN?= =?utf-8?q?WR2kkqA3ALiy6etV8nd4e4NvSKKbkRHOYlT0iiFr7z1/0+45dEcltep7lj8/mz//7?= =?utf-8?q?Q1hjgZMBZV2kQYPTV0fsRNP8gXn5cY2SycyCdmRvFypB4vxPQOxNqVQ1KDd7wK4Z1?= =?utf-8?q?wxwkqxOpQoUhViRWOAviiXoPWSWamtNgd/eTpxjQcPM5/dl0KuI+cvJ0cbteamlV1?= =?utf-8?q?r98XxS3OgHOxKEGHKp6Jc61cqZlpHgZYUQPdz+uwj7JFuAYCFH2Z3BL6hnmjyONqG?= =?utf-8?q?RMPUth33LNIvUMt8Cemj4YBKO5ofT7quOr1zstZOWKy4WqiBJioAt5Q/4v+A75jA9?= =?utf-8?q?UgTUQqzKrm04cFnSzMJFomuoo5aKmXwBdzSJPh/5nAhkI1ixu+kCJrpXwSUwEAfuc?= =?utf-8?q?7g2HGM2dQzkBaxqfPHrtm/av5cfzrfRt60BPlh4bLkKXdJRsuhq5X0Wj6L7IhdcIS?= =?utf-8?q?Dn617IZpASw2gylAue/mbg1NjSa7oAkRKuMtMbAMaYv7KjCc3BnPGZG554m1ipLs4?= =?utf-8?q?nMVYs5QrqY7A87ADnJyWe7LNVKHD8+pphGg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA0PR17MB6347.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(376005)(52116005)(1800799015)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?AYCQA4bovmisnVKonpFh4wSbQFgj?= =?utf-8?q?idVVu7vUPfuQTLyrZiAI6OAB/xasDEYZhHLYoRqLkYmKG2cEzmj5UkAIZQeLAdv4t?= =?utf-8?q?tLHyzVi7BjQKZezDIms2KGC7oi7fzqE2eIYvEFEhb9rc095JyQs9CP9gWW3rF30Ys?= =?utf-8?q?hK6THgPdTY2BQ6Lq6GXutyrvsUzhapn4Dh/6a3mNxZ/O+UTQzklLazqAUEBNklcTQ?= =?utf-8?q?J42Gshn7Q+WpdUinSCUW/hRUznLyTArWzKbokEdinq7cDwGgXmNdz8Dfx8A/daK0e?= =?utf-8?q?uHKW9cceK3V0W433qyIcdUTwEVDfd7boC/Nrt4H/feLJTRqt6zO7Q3OUatTLp1O9G?= =?utf-8?q?zvZ+3TkotF4/eOhVXTtJiIAQOnCRCPfotKBjm1OqOU/v4A995U3UfuB/IVP6RrX1N?= =?utf-8?q?cjcwZYGY82DH8NsuicO5O2tVBS6GDHk2jDgwLQMHqYPBfVQ4XRvBTyf76pbSVnYIf?= =?utf-8?q?NLhv+ujrMg9GaMCUsoN+wcgG9rBvl2Q8wam6OtFf7YtsgdbMr2qLMhWZ6RFgieNHL?= =?utf-8?q?4dGqQ2LjEmT/eyFCZ7ffAy1COLdx4yJTtWozxBvjEoueb+InPUw/TVmrTCoOS5VJf?= =?utf-8?q?Jmk8j5VnyxPRW3isxh9CU+ihZN1A9Vc/XuY6gCH9T9eCVlDHtREHAdeZfpdzIXQF/?= =?utf-8?q?dgkTo/svyvmSx/8JN4+y7Is0ptIdauTlt90JYV02geDLn0tis1WfjG0uY4hEjhdSu?= =?utf-8?q?O81/lL5LsSymI3Yh0vLvHbnQt4uvCKio1MakF9gv7CDhmUDf5OadJDQkqLayfkbuF?= =?utf-8?q?e69+kAkT609ge1GrtYY81uvZAp12/XzpL0T8Wj+Iq4e7rPQHmvk2tqK7o9F9DRJJl?= =?utf-8?q?RFmXOkXWABDKdYAWQTK/2z02/3HVlOyhN5AjDEzZnnLiik7sb9MDGfJFDbk/RZYxc?= =?utf-8?q?14hYg5c0fe+ezAWhMoPILHvnsYHctEe7NSBSlju7QnSENpVOYpzmx0a+cVLlxB9Es?= =?utf-8?q?SxRDCbIHmA/BcbNPBhuVHYdWHd7Cjw6xeOMx70OnNaEADA9aijILtRY7KuRXQrWS1?= =?utf-8?q?JRdiAkU7AM00bJilu4Aodxz7/dE8Jt6ijqNhXy2/g+cRc5TiR+oGvMpDX0UZJ5irz?= =?utf-8?q?S83wfN+yJNwv/pCN9X06gMGZSevzAjZIVvL1XIhFB9hkBRx0qtXvxjKT+b1kKfED3?= =?utf-8?q?I+Fd19HCFAQQgFW08sdIg7uNqgO16yHscMq2xEKmQQ4u6oVign9aFl6QS2ErEbGH1?= =?utf-8?q?M1CI6Y0F5SquXfZYwityqkYLKEgln2C5SQXwaYis/NZohCUZ/CexhHNqKz8YS2T85?= =?utf-8?q?/aNrKtJOyv3xTBdsYm6sbS8swWpvAMcaSzQtVFLSoiubpS335KzJ7qhKLuTxa2woz?= =?utf-8?q?L6kqb6qe0AKqEjG+z5BWzwbe36IqQq5R0MPCTFtyQLW0pIQ5xCUzolaf/YYrVDSjU?= =?utf-8?q?1I2VWsmmR65JJoxmxZmtRqUTVB5oTloB5vW8Vm43g4n3K098Cs30+kjVxqyaaZ4EJ?= =?utf-8?q?/QbPB7W8Oz1bYoqZIy3gDMhqEhX0VaIm0MpXu5tezNd8+owN1mtHxBzJ6ucnpU08Z?= =?utf-8?q?1nWzOlcgx5YZ3Zw2T7j01KtzOvLM+2bz8yp5vTG7SW+3bI5dSpnrBXc=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: dc54affa-a2bf-4ed4-3a65-08dc6499bdaf X-MS-Exchange-CrossTenant-AuthSource: IA0PR17MB6347.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 20:04:25.5750 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a3svedPsAYpIG2kKJDVLU9/qVtOEz7VlfvSMoKVVwaEOCCYL7aQHE/CSvPQyQn02VI/lH3IR0mLJ68TB903YAFPVTy6ep9oOm1rQ3hdbdQs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR17MB5846 Introduce an API for validating DC adds, removes, and responses against a multi-headed device. mhd_reserve_extents() is called during a DC add request. This allows a multi-headed device to check whether the requested extents belong to another host. If not, then this function can claim those extents in the MHD state and allow the cxl_type3 code to follow suit in the host-local blk_bitmap. mhd_reclaim_extents() is called during the DC add response. It allows the MHD to reclaim extents that were preallocated to a host during the request but rejected in the response. mhd_release_extent() is called during the DC release response. It can be invoked after a host frees an extent in its local bitmap, allowing the MHD handler to release that same extent in the multi-host state. --- hw/cxl/cxl-mailbox-utils.c | 28 +++++++++++++++++++++++++++- hw/mem/cxl_type3.c | 17 +++++++++++++++++ include/hw/cxl/cxl_device.h | 8 ++++++++ 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index dc916f5bb3..0429bb4816 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -2283,6 +2283,7 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, { CXLUpdateDCExtentListInPl *in = (void *)payload_in; CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDCExtentList *extent_list = &ct3d->dc.extents; uint32_t i; uint64_t dpa, len; @@ -2317,6 +2318,11 @@ static CXLRetCode cmd_dcd_add_dyn_cap_rsp(const struct cxl_cmd *cmd, ct3d->dc.total_extent_count += 1; ct3_set_region_block_backed(ct3d, dpa, len); } + + if (cvc->mhd_reclaim_extents) + cvc->mhd_reclaim_extents(&ct3d->parent_obj, &ct3d->dc.extents_pending, + in); + /* Remove the first extent group in the pending list*/ cxl_extent_group_list_delete_front(&ct3d->dc.extents_pending); @@ -2350,6 +2356,7 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, uint32_t *updated_list_size) { CXLDCExtent *ent, *ent_next; + CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); uint64_t dpa, len; uint32_t i; int cnt_delta = 0; @@ -2370,6 +2377,13 @@ static CXLRetCode cxl_dc_extent_release_dry_run(CXLType3Dev *ct3d, goto free_and_exit; } + /* In an MHD, check that this DPA range belongs to this host */ + if (cvc->mhd_access_valid && + !cvc->mhd_access_valid(&ct3d->parent_obj, dpa, len)) { + ret = CXL_MBOX_INVALID_PA; + goto free_and_exit; + } + /* After this point, extent overflow is the only error can happen */ while (len > 0) { QTAILQ_FOREACH(ent, updated_list, node) { @@ -2442,9 +2456,11 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, { CXLUpdateDCExtentListInPl *in = (void *)payload_in; CXLType3Dev *ct3d = CXL_TYPE3(cci->d); + CXLType3Class *cvc = CXL_TYPE3_GET_CLASS(ct3d); CXLDCExtentList updated_list; CXLDCExtent *ent, *ent_next; - uint32_t updated_list_size; + uint32_t updated_list_size, i; + uint64_t dpa, len; CXLRetCode ret; if (in->num_entries_updated == 0) { @@ -2462,6 +2478,16 @@ static CXLRetCode cmd_dcd_release_dyn_cap(const struct cxl_cmd *cmd, return ret; } + /* Updated_entries contains the released extents. Free those in the MHD */ + for (i = 0; cvc->mhd_release_extent && i < in->num_entries_updated; ++i) { + dpa = in->updated_entries[i].start_dpa; + len = in->updated_entries[i].len; + + if (cvc->mhd_release_extent) { + cvc->mhd_release_extent(&ct3d->parent_obj, dpa, len); + } + } + /* * If the dry run release passes, the returned updated_list will * be the updated extent list and we just need to clear the extents diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 88facd49ed..e69cb6f7b7 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -799,6 +799,7 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) { CXLDCExtent *ent, *ent_next; CXLDCExtentGroup *group, *group_next; + CXLType3Class *cvc = CXL_TYPE3_CLASS(ct3d); int i; CXLDCRegion *region; @@ -817,6 +818,10 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d) for (i = 0; i < ct3d->dc.num_regions; i++) { region = &ct3d->dc.regions[i]; g_free(region->blk_bitmap); + if (cvc->mhd_release_extent) { + cvc->mhd_release_extent(&ct3d->parent_obj, region->base, + region->len); + } } } @@ -2104,6 +2109,7 @@ static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, CXLEventDynamicCapacity dCap = {}; CXLEventRecordHdr *hdr = &dCap.hdr; CXLType3Dev *dcd; + CXLType3Class *cvc; uint8_t flags = 1 << CXL_EVENT_TYPE_INFO; uint32_t num_extents = 0; CXLDCExtentRecordList *list; @@ -2121,6 +2127,7 @@ static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, } dcd = CXL_TYPE3(obj); + cvc = CXL_TYPE3_GET_CLASS(dcd); if (!dcd->dc.num_regions) { error_setg(errp, "No dynamic capacity support from the device"); return; @@ -2193,6 +2200,13 @@ static void qmp_cxl_process_dynamic_capacity_prescriptive(const char *path, num_extents++; } + /* If this is an MHD, attempt to reserve the extents */ + if (type == DC_EVENT_ADD_CAPACITY && cvc->mhd_reserve_extents && + !cvc->mhd_reserve_extents(&dcd->parent_obj, records, rid)) { + error_setg(errp, "mhsld is enabled and extent reservation failed"); + return; + } + /* Create extent list for event being passed to host */ i = 0; list = records; @@ -2336,6 +2350,9 @@ static void ct3_class_init(ObjectClass *oc, void *data) cvc->set_cacheline = set_cacheline; cvc->mhd_get_info = NULL; cvc->mhd_access_valid = NULL; + cvc->mhd_reserve_extents = NULL; + cvc->mhd_reclaim_extents = NULL; + cvc->mhd_release_extent = NULL; } static const TypeInfo ct3d_info = { diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 5c60342556..f7ac0c34d2 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -14,6 +14,7 @@ #include "hw/pci/pci_device.h" #include "hw/register.h" #include "hw/cxl/cxl_events.h" +#include "qapi/qapi-commands-cxl.h" #include "hw/cxl/cxl_cpmu.h" /* @@ -651,6 +652,13 @@ struct CXLType3Class { size_t *len_out, CXLCCI *cci); bool (*mhd_access_valid)(PCIDevice *d, uint64_t addr, unsigned int size); + bool (*mhd_reserve_extents)(PCIDevice *d, + CXLDCExtentRecordList *records, + uint8_t rid); + bool (*mhd_reclaim_extents)(PCIDevice *d, + CXLDCExtentGroupList *groups, + CXLUpdateDCExtentListInPl *in); + bool (*mhd_release_extent)(PCIDevice *d, uint64_t dpa, uint64_t len); }; struct CSWMBCCIDev { From patchwork Wed Apr 24 20:04:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Svetly Todorov X-Patchwork-Id: 13642457 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2128.outbound.protection.outlook.com [40.107.223.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFAB013D25E for ; Wed, 24 Apr 2024 20:04:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.128 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989074; cv=fail; b=j3gRj2kE9ztpUwSuAB/wvJOPmUqHPPVTCyDBcqULfPUQ62hjgKZGvzA+xrGD15hGXDzsdUZkynTSeLD1fCwZPx91WndQhjTLVe+jKXi0jngA3pkgkmNIVBV4wQREKHFfbdFYRVeItifulOXOYWX+WAjPyTmuuL+ClXidk8F8oyM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989074; c=relaxed/simple; bh=Mmf0qdxK449WcZGHC/bZlcgno9+Nn5fb0FgJIQKg1r8=; h=From:Date:Subject:Content-Type:Message-Id:References:In-Reply-To: To:Cc:MIME-Version; b=J+PyZMd8H2MUV9W4P8gsUaFqd/ovO1RTFc+XZXAk+GrZ+iDQlRsKDsREjqx1s+Jv5W5l3qklGAcemkbCq9mA9SyxZ3BuiuqtDfX60vxMNaUgZzmNfYFwAQlwBTj4wknZDhuYiV+ch3PcvxfQgj3wMTNHhFNTIdwUDf1RURrVca8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com; spf=pass smtp.mailfrom=memverge.com; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b=ZjgdkOD0; arc=fail smtp.client-ip=40.107.223.128 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=memverge.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b="ZjgdkOD0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dsaZ+UD86ViIakYkvvO6M8TU7A+XDpWqCpmhphphwaNXpRyTuhCdGCqiv2SRZSZajAxXodvXiF/TWtNwft6hNV0+uYTv8AvIvKnW8aUdO0ves8qATkO9I9sHNYHxw9FtCMXU51V3lh57kB0zHJg0nYWbCMmTqWWvYcOCMglyJkzn2HvUy2UDgSbK2535aV0Hxr63gyKRtQ8t3MooPLLkWvkPcgQ9xZtIbGIdPdhzF24NK4CRxWOLmV4hc53752Hi2LyX7TvWswUwmuDXd5M6Zzyn2JqQnZgUdU79uGlFxmXJRdwcXxBrVm8QdQZodJm0QCT47phA1kw5TXZ7L8vl6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HCgoHLPds5uQj0aDjHRdIkKXNsoAoSa9JC+RkTC23FQ=; b=ZlT1IlnRXnvzRjj7cUQa/DKU+ndjgcjR8py074jZ8mnWvFkYlksEyxeyWdQsm7ZRTH6vyRwFQ1JNXNSOLTLEl++SQV2e9DtqCWY8dJRlhLynk/layKw9/VF9rrRppBn8BFlSh93Qgq1u3Yg3cA8k2tUQzmglMuMYBsvr5wM61XFgtJgYzJsbmFEtZzyMC/ssKJjAkLQsEVDggsGRpz8CIIarhHVvogkS3j0nC6mG+XNoG7jJeTfqu0icBdqnngzhNdW/3UTMasrTk0ydEEYcdU7rNCU8Oi17iSIelh0hcodA4p7R1ATV6jGkUE7Y0uCjE6RDfDmf33KVRzhKLb4WxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HCgoHLPds5uQj0aDjHRdIkKXNsoAoSa9JC+RkTC23FQ=; b=ZjgdkOD0/IaPyy4o620dNohZXu5QO72Bnk3r7dJAjC7dNhikqa0A32IIikNrUs6l0WzsSvsMokls/aQDr5mFDJwFHJSZxgrm/2gpNRIx2QbUJLEVSwkp1ebFxPaxfnNiUBqMy1xdDYVYISHA6fAEfSCb4IhkG2ifZm1JN3rgas8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=memverge.com; Received: from IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) by SN4PR17MB5846.namprd17.prod.outlook.com (2603:10b6:806:215::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Wed, 24 Apr 2024 20:04:26 +0000 Received: from IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a]) by IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a%6]) with mapi id 15.20.7519.021; Wed, 24 Apr 2024 20:04:26 +0000 From: Svetly Todorov Date: Wed, 24 Apr 2024 13:04:08 -0700 Subject: [PATCH RFC v2 3/3] mhsld: implement MHSLD device Message-Id: <20240424-cxl-2024-04-22-pullahead-v2-3-a87fc4597795@memverge.com> References: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> In-Reply-To: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> To: qemu-devel@nongnu.org Cc: Svetly Todorov , Gregory Price , jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, nifan.cxl@gmail.com X-Mailer: b4 0.14-dev X-Developer-Signature: v=1; a=ed25519-sha256; t=1713989063; l=18941; i=svetly.todorov@memverge.com; s=20240312; h=from:subject:message-id; bh=Mmf0qdxK449WcZGHC/bZlcgno9+Nn5fb0FgJIQKg1r8=; b=pIssofizIzxeAbX4wsWnWB7QhAzcEGb2Q8d0ggfSU0xx3yDjK4utqNDINWCE+NAIu3fMw8EOz O8Mq+0wA5PnA+SME9PwBQxxGrQi31kaqYsUNH8oeAENgtgAzKSmS07O X-Developer-Key: i=svetly.todorov@memverge.com; a=ed25519; pk=bo0spdkY5tAEf+QP9ZH+jA9biE/razmOR7VcBXnymUE= X-ClientProxiedBy: BY3PR03CA0005.namprd03.prod.outlook.com (2603:10b6:a03:39a::10) To IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR17MB6347:EE_|SN4PR17MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: 8e777ca6-ec33-4ae8-721f-08dc6499be29 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: =?utf-8?q?e3qpMEafqzSKX3I+/rkEjseDCuJm2KL?= =?utf-8?q?E/4V2GMOkIYKy+m9K0Pl1glCWStg9p6mFnJyv2LTdNfGGIWcbyevBdnwVc0FSfgF2?= =?utf-8?q?G/UQro0jNaLAovAPFLf7V2clDIp6ZEr5Oh4Ljs6jPnZV8B8+PB9L9Z1BQS7vEWcAU?= =?utf-8?q?GSVX/jPublRKTdJhBu8zm7d/xY/SV5zTOF4W/3XNKKT1+eS2VvYaWiT8n3OMl2BSY?= =?utf-8?q?XtfYM6Q05eNw17mh8Xe/VxjdhTGlV67xbYgnRY+9UHGw1BR+rLNoqNrwqkBX+Gw5A?= =?utf-8?q?vLl/ct01YFdiqIBM9QPvMl3I1YzxcYK8M40mUYei/j69njqyv5RAAiY3zGIxQf0Kz?= =?utf-8?q?n9BpMz6oBq0bCnOgnK2jUS0JyxyM0Gx8KUYYbAxeDS0uSlEAWNBtO3JmB+jhnFHI5?= =?utf-8?q?BnLvrEo8UmUZUOXDNuyEx3RovqIdp/9P/3ax663oPDpvXv90lo2pkpW/OX2Pn700k?= =?utf-8?q?aq31uxPK8fOxda+6DAbE0fufNB+gVuf8opjjcpDHqTSoTnUnU+OhziIFvleAEk1iq?= =?utf-8?q?in2pXvmL9WHmK7IP39+lGQjegqxsQUASqsLspcyLl5S06DJLWzfw2GFOPHWHB7Ocn?= =?utf-8?q?PaVc4jVx8z0bK3YUB3ObOpaJ1lx7ZEech7E7ClVqA5FRhEnd7hz2MEECjq/imsFib?= =?utf-8?q?AHo6VfS0UJSZxWmNUswDy1hrV5CTHWojNdUaowBOuq7jTTVvOFCEYp/8kc1rxqRG9?= =?utf-8?q?YbQngGyDfJUq3xKncqk+egqm+tI/jGDF1GXMuby26142cDyCPpHHbAJ+uXnnpz35g?= =?utf-8?q?MUqapGBnGCImovh5Oo1/6k/gLU1tthmpI/PvPRiHNJDfTmXmwzOlQlUmCgDPxxUQU?= =?utf-8?q?FbuCQIKoKruH6KVAZr9/NunFtHusXmJot4BuTvfh06vvxd+QucSbQOSuQTIqmVXJR?= =?utf-8?q?qORR4RC86Oq6eoc+8WN0rfKk2OSsgUQOY17ZgHHocoEXKlaqqgkEmE43D4FpyYcIe?= =?utf-8?q?vvGDiFDKstsTBXX/1yzTXEx3E6Joy14WJPugfb2ZkBwHg62hN+Tk8P3CPHvTlulZb?= =?utf-8?q?NDjTiYfc1lkJCOQWKBjTynCRet/QUwo8iNnfx3UmciO0feU3vaMSG39GExl5plCP7?= =?utf-8?q?JkeOvG8bFGVIhUZJv+z415qQFWFss1Cch0oPtMDirC5+aJhkwsl4PRZv5nuH3EXjD?= =?utf-8?q?r6FeL9t3B4BIctPFZvjRaPymi4KO/DGTx3ZGsqgGgIMJKBBZ4PQg5isg114kijO8K?= =?utf-8?q?iSjylh2QBTqI+l/mA1Eg8E+KUpmArOdXpP2xQ4sqD2enQ15ZJzawX2TOYkGyg6+IO?= =?utf-8?q?DQ8dMQaM4U8pl0EgckVrjqiZK+7pte7HCXg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA0PR17MB6347.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(376005)(52116005)(1800799015)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?cAatTLvLeYK9KMyvTAs4354hzKpz?= =?utf-8?q?sp2sNDIVTjYs16bBgmbJ5hfEoiMd1X/p/M6tnyuAMBZlHtK830g7292mI/cugZe+h?= =?utf-8?q?wk9yxna4cCsKEV1bOowFu6xzWNMsD7pqEBHa/qqVmhJAJVYKmgZ1wyPpJk2MAIea9?= =?utf-8?q?M6ZVg8cMlVcg/7n0+ipByXQHtpG5LWDytw5FctSj8YaPPpMe6CsXv0x8ZPPNrbDJD?= =?utf-8?q?m95worreEY3IlZLKykg88Y6JqdQG3S0VFxmjqncGhoe2gvo3XNLzLwzQGEiDx6Jgn?= =?utf-8?q?Fm+rwzgZy1i2+P/oL7g38nXa0frcCitgLhO2hD6hNKMxwuHJo9FV3rKQmriEq35p6?= =?utf-8?q?JVCoNZxUzYJfWvtMAiNHrFgYkRetgyUTjGAwrHDT6ABl63LQe4MZiLMlw+wwSKc2Y?= =?utf-8?q?KyHo//VVtwuHnl7sFl1awHyoLMEx58S4HAaCUShjpi6gfyTFsSntwHSlqWpQs0mxR?= =?utf-8?q?kuBSHOcjNm30V/N2RK7CcI1dT4pWtDUwA3z20qMaJkrC9enR0KAzuXiDVjkL1cleG?= =?utf-8?q?C7KtnzIcpndALfYCd78NCbPEA3pCBW1wWpSxrNaZ0YPpO7FZcxVVCQItFC1Wk36uV?= =?utf-8?q?aUdY9quB6hvxk4GHEiCCMAHm0yDiv8LbM0U79qIksQ12ncemjUNlAYsq1V24PIAGc?= =?utf-8?q?BfzRjJNYXTFx/xHVpD/u2vtfSmIzIikRfi1TrDIcqcTzfHxTGxG2yQ4c6ytzrWtRI?= =?utf-8?q?q8v2LZSVgTJq9iJI+tABvhsojToz2pO87T2F/xZCg0Ht1rrTqzD3jDt1RhXAMSZ5W?= =?utf-8?q?0s/UAlO72mE56uZlikjsoFzY9BdqQYpa7bx2oXKdtGQGiA4AfzXPJDodV2FbQtSKs?= =?utf-8?q?SHGm86GOPV5T5u3xG77HjL0myfzlUtZJZU7gw73Rt39JAPQpjCUyCut+AZ1zJ0Jwm?= =?utf-8?q?9hF1LY6tvONgAd82u6SMfOh3zLwa4gSTav/N6DgedDFNdyXJwSQ8RYzvK76spWsw9?= =?utf-8?q?dQGSOQfnPmsYVfgJSNXSBogRdIBppPqR93sg97hzakFBHJjtVtV43ks8w5DyVRJ5K?= =?utf-8?q?UNq+klyaFjnWm5c3kb9ZJIrbGl3OpeyRTVzOFL29t0qJ0Z8WUEDONdQ1b4FJUVxmw?= =?utf-8?q?aU+XJ1o+F0NX1HEwCVXb8F0srCc3arr3qC0QbBvfP0SnNxiFoyMkT0reMAgmmbHbb?= =?utf-8?q?dy1kIKX4YeiqwukiiXEHnonmINU3jT5EgomKSYc0YL0LO+FLNS937Kkpl7SsHfk7x?= =?utf-8?q?cWQOpJN9ISvL4mCaC1E2/S9qlhQpZBgkmLs2Q14XM/yGTjX719XaQI9aTR3uubtBe?= =?utf-8?q?F2xQLFCjo0+jUlNsLlP1ZzQh11v0RZRxQC26HrTRHfvzSdcRV4avj3o+lkdsIVENn?= =?utf-8?q?UmgVPC5351l99oO7J48r8/pRvvE3HYF70ZKJg3+yTblUmGrkEkTWlgnb85pqeVlVG?= =?utf-8?q?hE3V36j5r8bBKzc5fKg/eK/mOcAtuhBJBkoiyxR6F5Tol35VkEC6wCG7LtdEe9X+j?= =?utf-8?q?l8KKIB30JKHKucxIiuV+nUtLqUI7Q6iFFkJdVbE2JmKg761tv5EkY+kbZNDFW8UqL?= =?utf-8?q?TS1sZ034mbkA+49tS2eyQcbICebEqD+kXyOKoSM4Pe142n8eloXWZ6o=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8e777ca6-ec33-4ae8-721f-08dc6499be29 X-MS-Exchange-CrossTenant-AuthSource: IA0PR17MB6347.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 20:04:26.3890 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FhY6yvbUR0byMcyc6wkR9pPKMtrPL7oO9XdY6GxJ7QmkMHdG0n7i8P82U40zAhVsjPayrkNZSHeimyQfByL756YS4e9zaImShupP5Eos0OU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR17MB5846 Using a shared-memory bytemap, validates that DC adds, releases, and reclamations happen on extents belonging to the appropriate host. The MHSLD device inherits from the CXL_TYPE3 class and adds the following configuration options: --mhd-head= --mhd-state_file= --mhd-init= --mhd-head specifies the head ID of the host on the given device. --mhd-state_file is the name of the shared-memory-backed file used to store the MHD state. --mhd-init indicates whether this QEMU instance should initialize the state_file; if so, the instance will create the file if it does not exist, ftruncate it to the appropriate size, and initialize its header. It is assumed that the --mhd-init instance is run and allowed to completely finish configuration before any other guests access the shared state. The shared state file only needs to be intialized once. Even if a guest dies without clearing the ownership bits associated with its head-ID, future guests with that ID will clear those bits in cxl_mhsld_realize(), regardless of whether mhd_init is true or false. The following command line options create an MHSLD with 4GB of backing memory, whose state is tracked in /dev/shm/mhd_metadata. --mhd-init=true tells this instance to initialize the state as described above. ./qemu-system_x86-64 \ [... other options ...] \ -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \ -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \ -object memory-backend-ram,id=mem0,size=4G \ -device cxl-mhsld,bus=rp0,num-dc-regions=1,volatile-dc-memdev=mem0,id=cxl-mem0,sn=66667,mhd-head=0,mhd-state_file=mhd_metadata,mhd-init=true \ -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G \ -qmp unix:/tmp/qmp-sock-1,server,nowait Once this guest completes setup, other guests looking to access the device can be booted with the same configuration options, but with --mhd-head != 0, --mhd-init=false, and a different QMP socket. --- hw/cxl/Kconfig | 1 + hw/cxl/meson.build | 1 + hw/cxl/mhsld/Kconfig | 4 + hw/cxl/mhsld/meson.build | 3 + hw/cxl/mhsld/mhsld.c | 456 +++++++++++++++++++++++++++++++++++++++++++++++ hw/cxl/mhsld/mhsld.h | 75 ++++++++ 6 files changed, 540 insertions(+) diff --git a/hw/cxl/Kconfig b/hw/cxl/Kconfig index e603839a62..919e59b598 100644 --- a/hw/cxl/Kconfig +++ b/hw/cxl/Kconfig @@ -1,3 +1,4 @@ +source mhsld/Kconfig source vendor/Kconfig config CXL diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build index e8c8c1355a..394750dd19 100644 --- a/hw/cxl/meson.build +++ b/hw/cxl/meson.build @@ -16,4 +16,5 @@ system_ss.add(when: 'CONFIG_I2C_MCTP_CXL', if_true: files('i2c_mctp_cxl.c')) system_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c')) +subdir('mhsld') subdir('vendor') diff --git a/hw/cxl/mhsld/Kconfig b/hw/cxl/mhsld/Kconfig new file mode 100644 index 0000000000..dc2be15140 --- /dev/null +++ b/hw/cxl/mhsld/Kconfig @@ -0,0 +1,4 @@ +config CXL_MHSLD + bool + depends on CXL_MEM_DEVICE + default y diff --git a/hw/cxl/mhsld/meson.build b/hw/cxl/mhsld/meson.build new file mode 100644 index 0000000000..c595558f8a --- /dev/null +++ b/hw/cxl/mhsld/meson.build @@ -0,0 +1,3 @@ +if host_os == 'linux' + system_ss.add(when: 'CONFIG_CXL_MHSLD', if_true: files('mhsld.c',)) +endif diff --git a/hw/cxl/mhsld/mhsld.c b/hw/cxl/mhsld/mhsld.c new file mode 100644 index 0000000000..14f4b6f6c9 --- /dev/null +++ b/hw/cxl/mhsld/mhsld.c @@ -0,0 +1,456 @@ +/* + * SPDX-License-Identifier: GPL-2.0-or-later + * + * Copyright (c) 2024 MemVerge Inc. + * + */ + +#include +#include "qemu/osdep.h" +#include "qemu/bitmap.h" +#include "hw/irq.h" +#include "migration/vmstate.h" +#include "qapi/error.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_mailbox.h" +#include "hw/cxl/cxl_device.h" +#include "hw/pci/pcie.h" +#include "hw/pci/pcie_port.h" +#include "hw/qdev-properties.h" +#include "sysemu/hostmem.h" +#include "mhsld.h" + +#define TYPE_CXL_MHSLD "cxl-mhsld" +OBJECT_DECLARE_TYPE(CXLMHSLDState, CXLMHSLDClass, CXL_MHSLD) + +/* + * CXL r3.0 section 7.6.7.5.1 - Get Multi-Headed Info (Opcode 5500h) + * + * This command retrieves the number of heads, number of supported LDs, + * and Head-to-LD mapping of a Multi-Headed device. + */ +static CXLRetCode cmd_mhd_get_info(const struct cxl_cmd *cmd, + uint8_t *payload_in, size_t len_in, + uint8_t *payload_out, size_t *len_out, + CXLCCI * cci) +{ + CXLMHSLDState *s = CXL_MHSLD(cci->d); + MHDGetInfoInput *input = (void *)payload_in; + MHDGetInfoOutput *output = (void *)payload_out; + + uint8_t start_ld = input->start_ld; + uint8_t ldmap_len = input->ldmap_len; + uint8_t i; + + if (start_ld >= s->mhd_state->nr_lds) { + return CXL_MBOX_INVALID_INPUT; + } + + output->nr_lds = s->mhd_state->nr_lds; + output->nr_heads = s->mhd_state->nr_heads; + output->resv1 = 0; + output->start_ld = start_ld; + output->resv2 = 0; + + for (i = 0; i < ldmap_len && (start_ld + i) < output->nr_lds; i++) { + output->ldmap[i] = s->mhd_state->ldmap[start_ld + i]; + } + output->ldmap_len = i; + + *len_out = sizeof(*output) + output->ldmap_len; + return CXL_MBOX_SUCCESS; +} + +static const struct cxl_cmd cxl_cmd_set_mhsld[256][256] = { + [MHSLD_MHD][GET_MHD_INFO] = {"GET_MULTI_HEADED_INFO", + cmd_mhd_get_info, 2, 0}, +}; + +static Property cxl_mhsld_props[] = { + DEFINE_PROP_UINT32("mhd-head", CXLMHSLDState, mhd_head, ~(0)), + DEFINE_PROP_STRING("mhd-state_file", CXLMHSLDState, mhd_state_file), + DEFINE_PROP_BOOL("mhd-init", CXLMHSLDState, mhd_init, false), + DEFINE_PROP_END_OF_LIST(), +}; + +static int cxl_mhsld_state_open(const char *filename, int flags) +{ + char name[128]; + snprintf(name, sizeof(name), "/%s", filename); + return shm_open(name, flags, 0666); +} + +static int cxl_mhsld_state_unlink(const char *filename) +{ + char name[128]; + snprintf(name, sizeof(name), "/%s", filename); + return shm_unlink(name); +} + +static int cxl_mhsld_state_create(const char *filename, size_t size) +{ + int fd, rc; + + fd = cxl_mhsld_state_open(filename, O_RDWR | O_CREAT); + if (fd == -1) { + return -1; + } + + rc = ftruncate(fd, size); + + if (rc) { + close(fd); + return -1; + } + + return fd; +} + +static bool cxl_mhsld_state_set(CXLMHSLDState *s, size_t block_start, + size_t block_count) +{ + uint8_t prev, val, *block; + size_t i; + + val = (1 << s->mhd_head); + + /* + * Try to claim all extents from start -> start + count; + * break early if a claimed extent is encountered + */ + for (i = 0; i < block_count; ++i) { + block = &s->mhd_state->blocks[block_start + i]; + prev = __sync_val_compare_and_swap(block, 0, val); + if (prev != 0) { + break; + } + } + + if (prev == 0) { + return true; + } + + /* Roll back incomplete claims */ + for (;; --i) { + block = &s->mhd_state->blocks[block_start + i]; + __sync_fetch_and_and(block, ~(1u << s->mhd_head)); + if (i == 0) { + break; + } + } + + return false; +} + +static void cxl_mhsld_state_clear(CXLMHSLDState *s, size_t block_start, + size_t block_count) +{ + size_t i; + uint8_t *block; + + for (i = 0; i < block_count; ++i) { + block = &s->mhd_state->blocks[block_start + i]; + __sync_fetch_and_and(block, ~(1u << s->mhd_head)); + } +} + +static void cxl_mhsld_state_initialize(CXLMHSLDState *s, size_t dc_size) +{ + if (!s->mhd_init) { + cxl_mhsld_state_clear(s, 0, dc_size / MHSLD_BLOCK_SZ); + return; + } + + memset(s->mhd_state, 0, s->mhd_state_size); + s->mhd_state->nr_heads = MHSLD_HEADS; + s->mhd_state->nr_lds = MHSLD_HEADS; + s->mhd_state->nr_blocks = dc_size / MHSLD_BLOCK_SZ; +} + +/* Returns starting index of region in MHD map. */ +static inline size_t cxl_mhsld_find_dc_region_start(PCIDevice *d, + CXLDCRegion *r) +{ + CXLType3Dev *dcd = CXL_TYPE3(d); + size_t start = 0; + uint8_t rid; + + for (rid = 0; rid < dcd->dc.num_regions; ++rid) { + if (&dcd->dc.regions[rid] == r) { + break; + } + start += dcd->dc.regions[rid].len / dcd->dc.regions[rid].block_size; + } + + return start; +} + +static MHSLDSharedState *cxl_mhsld_state_map(CXLMHSLDState *s) +{ + void *map; + size_t size = s->mhd_state_size; + int fd = s->mhd_state_fd; + + if (fd < 0) { + return NULL; + } + + map = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (map == MAP_FAILED) { + return NULL; + } + + return (MHSLDSharedState *)map; +} + +/* + * Triggered during an add_capacity command to a CXL device: + * takes a list of extent records and preallocates them, + * in anticipation of a "dcd accept" response from the host. + * + * Extents that are not accepted by the host will be rolled + * back later. + */ +static bool cxl_mhsld_reserve_extents(PCIDevice *d, + CXLDCExtentRecordList *records, + uint8_t rid) +{ + uint64_t len, dpa; + bool rc; + + CXLMHSLDState *s = CXL_MHSLD(d); + CXLDCExtentRecordList *list = records, *rollback = NULL; + + CXLType3Dev *ct3d = CXL_TYPE3(d); + CXLDCRegion *region = &ct3d->dc.regions[rid]; + + for (; list; list = list->next) { + len = list->value->len / MHSLD_BLOCK_SZ; + dpa = (list->value->offset + region->base) / MHSLD_BLOCK_SZ; + + rc = cxl_mhsld_state_set(s, dpa, len); + + if (!rc) { + rollback = records; + break; + } + } + + /* Setting the mhd state failed. Roll back the extents that were added */ + for (; rollback; rollback = rollback->next) { + len = rollback->value->len / MHSLD_BLOCK_SZ; + dpa = (list->value->offset + region->base) / MHSLD_BLOCK_SZ; + + cxl_mhsld_state_clear(s, dpa, len); + + if (rollback == list) { + return false; + } + } + + return true; +} + +static bool cxl_mhsld_reclaim_extents(PCIDevice *d, + CXLDCExtentGroupList *ext_groups, + CXLUpdateDCExtentListInPl *in) +{ + CXLMHSLDState *s = CXL_MHSLD(d); + CXLType3Dev *ct3d = CXL_TYPE3(d); + CXLDCExtentGroup *ext_group = QTAILQ_FIRST(ext_groups); + CXLDCExtent *ent; + CXLDCRegion *region; + g_autofree unsigned long *blk_bitmap = NULL; + uint64_t dpa, off, len, size, i; + + /* Get the DCD region via the first requested extent */ + ent = QTAILQ_FIRST(&ext_group->list); + dpa = ent->start_dpa; + len = ent->len; + region = cxl_find_dc_region(ct3d, dpa, len); + size = region->len / MHSLD_BLOCK_SZ; + blk_bitmap = bitmap_new(size); + + /* Set all requested extents to 1 in a bitmap */ + QTAILQ_FOREACH(ent, &ext_group->list, node) { + off = ent->start_dpa - region->base; + len = ent->len; + bitmap_set(blk_bitmap, off / MHSLD_BLOCK_SZ, len / MHSLD_BLOCK_SZ); + } + + /* Clear bits associated with accepted extents */ + for (i = 0; i < in->num_entries_updated; i++) { + off = in->updated_entries[i].start_dpa - region->base; + len = in->updated_entries[i].len; + bitmap_clear(blk_bitmap, off / MHSLD_BLOCK_SZ, len / MHSLD_BLOCK_SZ); + } + + /* + * Reclaim only the extents that belong to unaccepted extents, + * i.e. those whose bits are still raised in blk_bitmap + */ + for (off = find_first_bit(blk_bitmap, size); off < size;) { + len = find_next_zero_bit(blk_bitmap, size, off) - off; + cxl_mhsld_state_clear(s, off, len); + off = find_next_bit(blk_bitmap, size, off + len); + } + + return true; +} + +static bool cxl_mhsld_release_extent(PCIDevice *d, uint64_t dpa, uint64_t len) +{ + cxl_mhsld_state_clear(CXL_MHSLD(d), dpa / MHSLD_BLOCK_SZ, + len / MHSLD_BLOCK_SZ); + return true; +} + +static bool cxl_mhsld_access_valid(PCIDevice *d, uint64_t addr, + unsigned int size) +{ + CXLType3Dev *ct3d = CXL_TYPE3(d); + CXLMHSLDState *s = CXL_MHSLD(d); + CXLDCRegion *r = cxl_find_dc_region(ct3d, addr, size); + size_t i; + + addr = addr / r->block_size; + size = size / r->block_size; + + for (i = 0; i < size; ++i) { + if (s->mhd_state->blocks[addr + i] != (1 << s->mhd_head)) { + return false; + } + } + + return true; +} + +static void cxl_mhsld_realize(PCIDevice *pci_dev, Error **errp) +{ + CXLMHSLDState *s = CXL_MHSLD(pci_dev); + MemoryRegion *mr; + int fd = -1; + size_t dc_size; + + ct3_realize(pci_dev, errp); + + /* Get number of blocks from dcd size */ + mr = host_memory_backend_get_memory(s->ct3d.dc.host_dc); + if (!mr) { + return; + } + dc_size = memory_region_size(mr); + if (!dc_size) { + error_setg(errp, "MHSLD does not have dynamic capacity to manage"); + return; + } + + s->mhd_state_size = (dc_size / MHSLD_BLOCK_SZ) + sizeof(MHSLDSharedState); + + /* Sanity check the head idx */ + if (s->mhd_head >= MHSLD_HEADS) { + error_setg(errp, "MHD Head ID must be between 0-7"); + return; + } + + /* Create the state file if this is the 'mhd_init' instance */ + if (s->mhd_init) { + fd = cxl_mhsld_state_create(s->mhd_state_file, s->mhd_state_size); + } else { + fd = cxl_mhsld_state_open(s->mhd_state_file, O_RDWR); + } + + if (fd < 0) { + error_setg(errp, "failed to open mhsld state errno %d", errno); + return; + } + + s->mhd_state_fd = fd; + + /* Map the state and initialize it as needed */ + s->mhd_state = cxl_mhsld_state_map(s); + if (!s->mhd_state) { + error_setg(errp, "Failed to mmap mhd state file"); + close(fd); + cxl_mhsld_state_unlink(s->mhd_state_file); + return; + } + + cxl_mhsld_state_initialize(s, dc_size); + + /* Set the LD ownership for this head to this system */ + s->mhd_state->ldmap[s->mhd_head] = s->mhd_head; + return; +} + + +static void cxl_mhsld_exit(PCIDevice *pci_dev) +{ + CXLMHSLDState *s = CXL_MHSLD(pci_dev); + + ct3_exit(pci_dev); + + if (s->mhd_state_fd) { + munmap(s->mhd_state, s->mhd_state_size); + close(s->mhd_state_fd); + cxl_mhsld_state_unlink(s->mhd_state_file); + s->mhd_state = NULL; + } +} + +static void cxl_mhsld_reset(DeviceState *d) +{ + CXLMHSLDState *s = CXL_MHSLD(d); + + ct3d_reset(d); + cxl_add_cci_commands(&s->ct3d.cci, cxl_cmd_set_mhsld, 512); + + cxl_mhsld_state_clear(s, 0, s->mhd_state->nr_blocks); +} + +/* + * Example: DCD-add events need to validate that the requested extent + * does not already have a mapping (or, if it does, it is + * a shared extent with the right tagging). + * + * Since this operates on the shared state, we will need to serialize + * these callbacks across QEMU instances via a mutex in shared state. + */ + +static void cxl_mhsld_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pc = PCI_DEVICE_CLASS(klass); + + pc->realize = cxl_mhsld_realize; + pc->exit = cxl_mhsld_exit; + dc->reset = cxl_mhsld_reset; + device_class_set_props(dc, cxl_mhsld_props); + + CXLType3Class *cvc = CXL_TYPE3_CLASS(klass); + cvc->mhd_get_info = cmd_mhd_get_info; + cvc->mhd_access_valid = cxl_mhsld_access_valid; + cvc->mhd_reserve_extents = cxl_mhsld_reserve_extents; + cvc->mhd_reclaim_extents = cxl_mhsld_reclaim_extents; + cvc->mhd_release_extent = cxl_mhsld_release_extent; +} + +static const TypeInfo cxl_mhsld_info = { + .name = TYPE_CXL_MHSLD, + .parent = TYPE_CXL_TYPE3, + .class_size = sizeof(struct CXLMHSLDClass), + .class_init = cxl_mhsld_class_init, + .instance_size = sizeof(CXLMHSLDState), + .interfaces = (InterfaceInfo[]) { + { INTERFACE_CXL_DEVICE }, + { INTERFACE_PCIE_DEVICE }, + {} + }, +}; + +static void cxl_mhsld_register_types(void) +{ + type_register_static(&cxl_mhsld_info); +} + +type_init(cxl_mhsld_register_types) diff --git a/hw/cxl/mhsld/mhsld.h b/hw/cxl/mhsld/mhsld.h new file mode 100644 index 0000000000..e7ead1f0d2 --- /dev/null +++ b/hw/cxl/mhsld/mhsld.h @@ -0,0 +1,75 @@ +/* + * SPDX-License-Identifier: GPL-2.0-or-later + * + * Copyright (c) 2024 MemVerge Inc. + * + */ + +#ifndef CXL_MHSLD_H +#define CXL_MHSLD_H +#include +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_mailbox.h" +#include "hw/cxl/cxl_device.h" +#include "qemu/units.h" + +#define MHSLD_BLOCK_SZ (2 * MiB) + +/* + * We limit the number of heads to prevent the shared state + * region from becoming a major memory hog. We need 512MB of + * memory space to track 8-host ownership of 4GB of memory in + * blocks of 2MB. This can change if the block size is increased. + */ +#define MHSLD_HEADS (8) + +/* + * The shared state cannot have 2 variable sized regions + * so we have to max out the ldmap. + */ +typedef struct MHSLDSharedState { + uint8_t nr_heads; + uint8_t nr_lds; + uint8_t ldmap[MHSLD_HEADS]; + uint64_t nr_blocks; + uint8_t blocks[]; +} MHSLDSharedState; + +struct CXLMHSLDState { + CXLType3Dev ct3d; + bool mhd_init; + char *mhd_state_file; + int mhd_state_fd; + size_t mhd_state_size; + uint32_t mhd_head; + MHSLDSharedState *mhd_state; +}; + +struct CXLMHSLDClass { + CXLType3Class parent_class; +}; + +enum { + MHSLD_MHD = 0x55, + #define GET_MHD_INFO 0x0 +}; + +/* + * MHD Get Info Command + * Returns information the LD's associated with this head + */ +typedef struct MHDGetInfoInput { + uint8_t start_ld; + uint8_t ldmap_len; +} QEMU_PACKED MHDGetInfoInput; + +typedef struct MHDGetInfoOutput { + uint8_t nr_lds; + uint8_t nr_heads; + uint16_t resv1; + uint8_t start_ld; + uint8_t ldmap_len; + uint16_t resv2; + uint8_t ldmap[]; +} QEMU_PACKED MHDGetInfoOutput; +#endif