From patchwork Wed Apr 24 20:04:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Svetly Todorov X-Patchwork-Id: 13642454 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2128.outbound.protection.outlook.com [40.107.223.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75CAD1474B3 for ; Wed, 24 Apr 2024 20:04:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.128 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989069; cv=fail; b=KvJWMa+yNuNJlbxj/jIBEURR6r0IcLw40AcF1Ep7xqa2enD9BY5wa0/ako7EsUK1kxPMto0eLaUvtdy77wn38TkNTWbSwVRiwprZOiBgcLBKLwJTHiCfBBAElaC92g5tcO2Bo2SdNBnVSnLcxavdwotTffU6NW46sQYyjofrpJg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713989069; c=relaxed/simple; bh=BSjVtaO9NBh+Y7TbtFAKyzXM/xn9MGFLTeiMzP4i7A0=; h=From:Subject:Date:Message-Id:Content-Type:To:Cc:MIME-Version; b=h53HO+0P7XwYr0lA90sJ32DE6JmqxDl7lFk4HKRmf3FYC5LDjF/alHOymLJZPx1RZgZCctmvHUiA1mhHdcUZjiAXPjQC6xGMg3+iuft0I0rfoGgoQPcMtKZ+R8X5ICmnaT7CVQTYch5dhfE9NhdnlTyHoOo9u/Ieh1GM0dpIujs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com; spf=pass smtp.mailfrom=memverge.com; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b=C15kOMU8; arc=fail smtp.client-ip=40.107.223.128 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=memverge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=memverge.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=memverge.com header.i=@memverge.com header.b="C15kOMU8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gzaKrVShoWwRk/piSjIDYiyBdycKrUlG0Qx8b9I6CKBB0XW+s75ss6v1FcMCFd5RGG0jCMw8Sy8ugqXsLtz2RrWeYYBJCn/uIybmdLjbtDoAFK/QaOxKCa8iqPx7bJMAoxC820aHRlUTmQyZ/B2H25qAOljvSaH9MJYq+0CMs3OwE1+sOwmGMQ6N/JH0myU1irczdOI3mVErCJPK9N6CoRWLEjruseFBSofGpx2v2MTfH/qGX186UxkYHqSqsmfdOG2QQrVRidmDPQxG6TyNFhy6aMy6ZIzgk/leB9VIB0W3ZqWVRY+vZacGt8mMpL6dAisXwsDBYNrtZZfVoxzaxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Wx7TUQpgXn2svQwcZic5gTntpFzkqSKnIrikoW7iQf0=; b=TQFqYZxdD1/m3GisOqfdRsWO/C6AkgUWPevVWK4F5VD3yedS/PmtUKAy4iB8yO5wtxAXtcwo2ErV4lwSD4OiyOmI3B/WXHsxuf4VlbtBImyjWyxp5azgyFYcxfXWu9ulbSKfKpYL3WJv273ic7f+3dB4Dn97kHAmlXZkEA2TRppt2GfPVqBJwfS7Da12ONnXBZF69+W/PvmHyzAdO5zKk76Y/ol18L56NwHpDyT6MJVR9jMPMRWp5QnaabZEMJAR2+XEeEUmOd4M1soxphxG2woxfTHW7LyHer3HfvJlW48F2YLWCSnKgD+cc/kZf+8Dfr+W1w4OkoTlP92HJCfgcQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Wx7TUQpgXn2svQwcZic5gTntpFzkqSKnIrikoW7iQf0=; b=C15kOMU864/ugcsT6bTaLll4b2OdivzcZKb3gEgJSwhWzzA9uMUv9xIH8im2m3VrMGV2b0aPzzZl+dUjxnv/dbTfDf1MYvaGd2md3CyQA+mRN878tZjZI5xs7kxznszuB2UyVOCIe5/kD+lVBd24IytyfkqbYaRpRnkKN7AcX/k= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=memverge.com; Received: from IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) by SN4PR17MB5846.namprd17.prod.outlook.com (2603:10b6:806:215::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Wed, 24 Apr 2024 20:04:24 +0000 Received: from IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a]) by IA0PR17MB6347.namprd17.prod.outlook.com ([fe80::200e:df84:29a9:8c9a%6]) with mapi id 15.20.7519.021; Wed, 24 Apr 2024 20:04:24 +0000 From: Svetly Todorov Subject: [PATCH RFC v2 0/3] cxl: Multi-headed Single Logical Device (MHSLD) Date: Wed, 24 Apr 2024 13:04:05 -0700 Message-Id: <20240424-cxl-2024-04-22-pullahead-v2-0-a87fc4597795@memverge.com> X-B4-Tracking: v=1; b=H4sIALVlKWYC/x3MMQqAMAxA0atIZgM1FEGvIg7FpBooKi1KoXh3q +Mb/i+QJKokGJsCUW5NeuwV1DawbG5fBZWrgQxZY4lwyQE/oLFYeV4huE0co+fe8yA9d0JQ8zO K1/yvp/l5XgKMHEZqAAAA To: qemu-devel@nongnu.org Cc: Svetly Todorov , Gregory Price , jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, nifan.cxl@gmail.com X-Mailer: b4 0.14-dev X-Developer-Signature: v=1; a=ed25519-sha256; t=1713989063; l=4382; i=svetly.todorov@memverge.com; s=20240312; h=from:subject:message-id; bh=BSjVtaO9NBh+Y7TbtFAKyzXM/xn9MGFLTeiMzP4i7A0=; b=98qdpKg/Y71Z5Mi+h5s5MoT8o0e1niqL247WOx8dKwp7E9jphWkEwrdMMnYgi/F0fsjdZa3b+ iBrxmo5rAhrBpniE42bt1J7SxFhiiQmj3OndxdbBRMcWf5J7iSSaTbo X-Developer-Key: i=svetly.todorov@memverge.com; a=ed25519; pk=bo0spdkY5tAEf+QP9ZH+jA9biE/razmOR7VcBXnymUE= X-ClientProxiedBy: BY3PR03CA0005.namprd03.prod.outlook.com (2603:10b6:a03:39a::10) To IA0PR17MB6347.namprd17.prod.outlook.com (2603:10b6:208:435::22) Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR17MB6347:EE_|SN4PR17MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: 3a0fb8be-ae44-4a30-b752-08dc6499bcb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: =?utf-8?q?hG0clmGGtHL2AH/00OBTIJaZ41CvP7y?= =?utf-8?q?+gwrBdXkUAHG7HhqfIFsAcKiTkYtjVJKVAxcyFMdMyOqC82uCXRgRC7G34+EDEb/n?= =?utf-8?q?NvdcAYKZq34czSOzlzFCi2oeAa0CTFRsQFBAKU34Cej7zh8cFqETNLIJTtMb9ls7z?= =?utf-8?q?D8LqqSOYhFuHk/cHQa9MXEGibRSPteQyEQQ1vUFUDCkqYqa8rTBGYP/pnubV4G0bH?= =?utf-8?q?UUcqCxR9RHDM8wHqhM0FhgMcEhEiAb+ZDJkulRP+veMf0Rf/LjLWyk1kCurdO7RiO?= =?utf-8?q?m4rk42ey7uTdAqYluUrEQUR+i/gTFBGA8lEW/iF4ZAEI8kB6sZMekcHclXb82v9PY?= =?utf-8?q?vyuADEudo489jCUKhvYHfwyWGPoNChdlngGk9dQVSaO8aEVNIfWQ+YMXUIGI2j0G/?= =?utf-8?q?/zRJynzVOsdomOtIpetR6Hs3vY3EdV3cy4Ss5/Xoii7iX8HXGrNZC+4i1a3wvCNz5?= =?utf-8?q?N9y/PBeVjr+qHOQ5p0v60L8nm+oo9Vf1zirL0Hzmu3c+pmx7moR9VvJR2nDuZpd0r?= =?utf-8?q?DM1RVzNogRiHVL6e+rgEyDbAU6p5RjJwYXmBLv6TfqWT4ao1njG6zkdcxnplA1NPo?= =?utf-8?q?LtnPMNfM/QSGqqKNu0bcIYr4YjH5Y6tyhlaHKrUywuQHuh4Uw4w/Rrhnch3omgP0T?= =?utf-8?q?pwkLNbDPU94chulC/CXp4m8y/qt3tAXIGV8va+PQIpg5wtaZSkP1RnqZyk5s0vV3t?= =?utf-8?q?MHlkbqQdyRlD0pjfj8R3cjs79Jpvc4Xz949e6TstFH9HLdXZDw7iEocpu/lxOaWRN?= =?utf-8?q?Nqj9jGt33Jco4OoutHnGM0T/YRTPnG+nba8Y0R/I/2HnZ2SgzLftuoELQD3ffCim1?= =?utf-8?q?mM/ZYnlrZiL6kXg4l8Gbe+oaqKQ+Z89O/k8XVcNeTikC5YbN3bZhMBSWVXF3SIOqn?= =?utf-8?q?xf1+5WzijufLqxf9pilksxYAZnbscwRxVUziVaA+WrpllJpC9Dw5YT7ISeykmyMSn?= =?utf-8?q?WfCpNoAis3WhnJBnpROExJTal2gygyHBPIQyDqRbpXm6icCI2KBT5xWasR+rq5PR1?= =?utf-8?q?+pb9QVJnrJAwuP2dpTcZYnUMrQU3InrPVpR8PYSqJ+lkS89DXwgRE22ODJ9bzOuqF?= =?utf-8?q?NSBknEwhzRPvEpkLnFpW7HLK1YpUbVB09lmwNbNKWAONKfIHlRI29y+OheHURO2Eb?= =?utf-8?q?K0Gzgl5EQxyVw/wwNOVX7hKD3jlOr7/YY218YZ5HCxGQPqrVPav+HtTywuFPF6lZ/?= =?utf-8?q?H6bMgc1WcuLQY+v59bSC+7AdmROUcC4YZ1bfHLGKLF/TedXErZ27db8aZ+qUXLkNX?= =?utf-8?q?6yxQqWbsXDTETvslBXLzXOX3H1AfLM3Uzgg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA0PR17MB6347.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(376005)(52116005)(1800799015)(38350700005);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?ea0bfPCbj3Aeo2YOJqGUgEb3gbsI?= =?utf-8?q?aL7gno/ioLFcn48C4VnsskaiOe+1rDmyDjUqLuYgfM0nZIRQnTAHx0hkzTT9QGx1m?= =?utf-8?q?VaxCCbFEhYBIkQwvjbh61tIqX40MKy6VxLujky5Tib+lpDHZ+kaknZY7XcjqezZaP?= =?utf-8?q?/Gkp1KD1pgpOLvsUXN6Ee8X5AtzgrvtL4YFrhCn5+zZavusiQPukYZ9ZN5Ay2PHiQ?= =?utf-8?q?Yb3IH/1yfIFsJZIrv/al72FBYt5QCYV+7BEOhAzANzXZSUDZprokBkT4HRaTXWFKj?= =?utf-8?q?c2IDtG5ZaAwqoj3bYPbFzckbyl1S9JOhbAMfWx9OLQY7PCrYSKynzIwUS44z2InVD?= =?utf-8?q?062j3EHg7vgvtogAYQASRpkaypezu9Uycs6RGgW4pYkhfb5RtjUWPInXusawKb7nX?= =?utf-8?q?0+ylzeRt2lLXeqjAmAS1bWRgOoPRC22cjjzU9Bb+VZz2O52RD/Nr9j0YUsGnTl/ah?= =?utf-8?q?49T49p69AG/0dkpPNnFuwwzgLGClg1Xy3AYGdjUorZ1/QKRFwUU3G5uzS6gMrr5Q+?= =?utf-8?q?DJ2NgMY4EIzwMg4Jpt6F+chB64/XvhtHY0WUNv0jjeUf9o1Uqofn/8TT/E+VtNuzW?= =?utf-8?q?8sGWwz4M6ROxK9trMUb10tuWR9/P6UoON+InRBNJh0O4Ctafi8P/COPM62jq/2I3U?= =?utf-8?q?8qbhKpp4F1MPv3RIAzQ4tdidNmQZ+qJ7nvIeeUigU1Nihlk30sqWrDmxrCjEtQVQE?= =?utf-8?q?Z3gX0QzNer0AIVrLI29sEDejwRZFIKcDZUylhSnq6bydjAG7RGuj9qZHeeXrUQr3i?= =?utf-8?q?vFXlOSb+dHkJ9RnUqnmvtwa4y0Z/Xm5YPHF1WDvRnJbXfu34hjShq3g0kSCX8ZuYC?= =?utf-8?q?SSkgZ1cR8LeyqN7oOKIALDvEa/DXJHB3E8mmcjDudwRJiZ92B9Kf3NYf3RlFi4EiR?= =?utf-8?q?rw+e/jE3KcGXagF6LZbCWxDbdKmgeNlEj/OH4thf74WBU/3soBUZiyrUpxNxX3skq?= =?utf-8?q?fUde+h+AJJJvwj61AHh6zmrm2z4QCqFUOrfeb8CfPHPXCbMwPjmXbV12zJGBsnJW7?= =?utf-8?q?dB0EoSWiwkYwXZ51ediVh6JsBEmoc0KBHUJ5PnbEgbflfobqiJVN79iG3MBNUg8Aw?= =?utf-8?q?D/fQt5oUSAt5K3yYjQ1Y6+uP5CvnRWLS7eB9BaCsnsx0Xg4WSMdfbL/ssk1FEeWvl?= =?utf-8?q?kHj+iUvmV1ybH92EKmxbPqzan3ZsrSd9Tc3Oyfdk2WgLLxYAu3F7bMgpbpKBJRtor?= =?utf-8?q?pE+MrbZPgWrFS9xAH2V886h3JS9uejv5NcIW0hP1/Njr08+aVnF00bQCUaDICxhuu?= =?utf-8?q?KV4rYyaqPMzuj/t3pPqVelaK4o82+X+yclV87ZnZYCHEa1bDfo+Akd/8ihm7cnL6h?= =?utf-8?q?wZ33zTcnd/rBUsIE6ATNb3xNLLQsMXJt0UpK5WGr5aobxmNZP2UL3b13+aJvS9flS?= =?utf-8?q?MyaKC/t4wQb6+ZMY7junxTIGkdatyZ/yxz+s+mkWp641sGPb0Q8f6x9054wxWC3Pj?= =?utf-8?q?EgZFAlcur03HFs6gtJX2rAJalvhkSOHcmNU/VCATJqoe0TyWdQkTuZcoD2cW3iGzz?= =?utf-8?q?cnby236ks4UczG90KSrHm6o8yZEkdlD9qQ=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3a0fb8be-ae44-4a30-b752-08dc6499bcb7 X-MS-Exchange-CrossTenant-AuthSource: IA0PR17MB6347.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 20:04:23.9839 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NSKfE3EHSU148TPgvtxUYlbY1/zlQgunJsqmke0u9xomW2JPWXYCw92DsiL1dqm07fJ/4BTUUcDQ2/2vVFy1NbEHPGVZFH5bw6VtlD0nzlk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR17MB5846 MHSLDs allow multiple hosts to access dynamic capacity on a single backing device. This complicates DC management because adds, removals, and accesses need to be vetted such that hosts don't stomp on each other's data. This patchset proposes a set of hooks to be called in cxl_type3.c when each of the above events happens. The results of the hooks can be used to prevent illegal DC operations in the corresponding cxl_* functions. It also proposes an implementation for MHSLD emulation. The device's state is stored in a shared memory-backed file (on Linux, under /dev/shm). Extent ownership is tracked in a bytemap, wherein every byte acts as a set of flags showing which host ID can access the extent at that index. Blocks in an extent are claimed via compare-and-swap. Failure to claim the entirety of an extent is treated as a failure; the host will parse back over any blocks claimed so far and mask out its corresponding ownership bit. Operations on block states are done via atomics. This allows claims on unrelated extents to be serviced concurrently. Block state is relegated to a u8 to conserve space. Therefore only 8 heads are currently supported. The MHSLD device inherits from the CXL_TYPE3 class and adds the following configuration options: --mhd-head= --mhd-state_file= --mhd-init= --mhd-head specifies the head ID of the host on the given device. --mhd-state_file is the name of the shared-memory-backed file used to store the MHD state. --mhd-init indicates whether this QEMU instance should initialize the state_file; if so, the instance will create the file if it does not exist, ftruncate it to the appropriate size, and initialize its header. It is assumed that the --mhd-init instance is run and allowed to completely finish configuration before any other guests access the shared state. The shared state file only needs to be intialized once. Even if a guest dies without clearing the ownership bits associated with its head-ID, future guests with that ID will clear those bits in cxl_mhsld_realize(), regardless of whether mhd_init is true or false. The following command line options create an MHSLD with 4GB of backing memory, whose state is tracked in /dev/shm/mhd_metadata. --mhd-init=true tells this instance to initialize the file as described above. ./qemu-system_x86-64 \ [... other options ...] \ -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \ -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \ -object memory-backend-ram,id=mem0,size=4G \ -device cxl-mhsld,bus=rp0,num-dc-regions=1,volatile-dc-memdev=mem0,id=cxl-mem0,sn=66667,mhd-head=0,mhd-state_file=mhd_metadata,mhd-init=true \ -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G \ -qmp unix:/tmp/qmp-sock-1,server,nowait Once this guest completes setup, other guests looking to access the device can be booted with the same configuration options, but with --mhd-head != 0, --mhd-init=false, and a different QMP socket. The first patch in the series moves the CXLUpdateDCExtentListInPl struct to a shared header. This allows DC-management code outside of cxl_type3.c (like mhsld.c) to see which extents have been accepted by a host in its DC-add response. Signed-off-by: Svetly Todorov Signed-off-by: Gregory Price