From patchwork Tue Dec 10 09:48:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bharata B Rao X-Patchwork-Id: 13901172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E609E7717F for ; Tue, 10 Dec 2024 09:49:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD2166B0160; Tue, 10 Dec 2024 04:49:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A82006B0162; Tue, 10 Dec 2024 04:49:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D4E76B0163; Tue, 10 Dec 2024 04:49:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 68D2D6B0160 for ; Tue, 10 Dec 2024 04:49:11 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1C2071A0686 for ; Tue, 10 Dec 2024 09:49:11 +0000 (UTC) X-FDA: 82878575682.16.1B41B29 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2067.outbound.protection.outlook.com [40.107.237.67]) by imf02.hostedemail.com (Postfix) with ESMTP id B451F80007 for ; Tue, 10 Dec 2024 09:48:26 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=etz1KsKZ; spf=pass (imf02.hostedemail.com: domain of bharata@amd.com designates 40.107.237.67 as permitted sender) smtp.mailfrom=bharata@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733824125; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XaNMzxoG2eScPPvYOEWiCLCN/uTciIDbMbfl7KLstPQ=; b=eHhe52O1JB05Uqyf505uFQoZ1AOye619/dLHKGVcvRDel60utKy8J5yfqnQ0furP5j6IRf Shv7Eda5QxG412JIs59rQH63rHXAWaiM1+asG6z5njMzx5h/cBYrJO1Kw70BjMAZGVAP2U bgCpR7VKQQIbdGvwY4kJDnv0P1MEdC8= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=etz1KsKZ; spf=pass (imf02.hostedemail.com: domain of bharata@amd.com designates 40.107.237.67 as permitted sender) smtp.mailfrom=bharata@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1733824125; a=rsa-sha256; cv=pass; b=X6E2hCtTpYzX0vaET2U4wkSMQxFRrQJxTLEzAvsiKPZjaSJnOUs6v2INzASGiPNsUUD6cU BStMmGrAnhRazF4A8xUBtUWk9SQmLhKg32NHwiCR+Du46vLmDBPu4LO0zm5rvIeEKfCUnq 61goyEhyJ6NNE6ygKa3XZjWGuPng0Mo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FPBHn5E5NLEDx2M5mVhmeFCgEP8qW2i//elzXjTCBw0gVZPGz1EdpK367IvPpeT4iesCW6cBclqreZQbxUPvabn+uYTHVd+KpUm2P8BJ47VBuircoubpUoZXWrj3tB/IMG5ycnfgBGj4hCiHO55rLgdl7725oy+GbRb3YYz579rAkPR2xc7Lb/9uOaYFiaeYZ3H0rI2qtgn7/iMj7LgCJ4jpqbkZUJoUXr/WDxyn59O8xYnhUwGT9hoCq3YszIgu42Wii6AG2FQeXFktmCxoO7cIOMhNUAzW/ehGb3A0CsDwpTF3Hc531RFMMnSWoSERzJFslOGRAn8OZsTtEiOPqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XaNMzxoG2eScPPvYOEWiCLCN/uTciIDbMbfl7KLstPQ=; b=ln0+1EdOrbhrtVpUTR39OHzF0FddqiF9liGI6ZtqxLhpp1i5+zy/8BNsvgqC0vWaRr926yz0klhguTOwtMI1oGxsFeLworDpqx+M4V3NO0LbEmH4aM919RTj2UPLaIoJpxoPfeuNZqwh2lVID8g+S3xnViVq+klwpS7d+lLUChQzDTx76/o85ljkwtFl1nvYCUcE/0ASRCNlhlxaHE1PFu7JUsPspvXb610Xn+8edyrAuMfU8p1Xb17CUWZRUu3Fyv4+WjprNAgrj8vzHMGK8lMPCdjyY0AG3hHmFs1+ry0M0SSfQAlfwschjNQ2YOeuI0eXLokiESY9C5knmZF68g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kernel.dk smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XaNMzxoG2eScPPvYOEWiCLCN/uTciIDbMbfl7KLstPQ=; b=etz1KsKZ4C/8YXM94B0RAg78iPizwPZEMZRvjEqRHj70F95ZYjHY+TKqlMALYGwBPj4AoMymrNDvChaaOvk+cgU0ulkIirPp3MCyvN/JivalShwLyqRzprd08bYVdS+FBOBtaYivxqH/oEbP2cHBAHOTGRWjqHf5lZ4Pa5cWYXE= Received: from BL1PR13CA0259.namprd13.prod.outlook.com (2603:10b6:208:2ba::24) by SA1PR12MB9492.namprd12.prod.outlook.com (2603:10b6:806:459::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.18; Tue, 10 Dec 2024 09:49:02 +0000 Received: from BL6PEPF00020E64.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::40) by BL1PR13CA0259.outlook.office365.com (2603:10b6:208:2ba::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8251.12 via Frontend Transport; Tue, 10 Dec 2024 09:49:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF00020E64.mail.protection.outlook.com (10.167.249.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Tue, 10 Dec 2024 09:49:01 +0000 Received: from BLR-L-BHARARAO.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 10 Dec 2024 03:48:58 -0600 From: Bharata B Rao To: CC: , , , , , , , Subject: [PATCHSET v6 0/12] Uncached buffered IO Date: Tue, 10 Dec 2024 15:18:42 +0530 Message-ID: <20241210094842.204504-1-bharata@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E64:EE_|SA1PR12MB9492:EE_ X-MS-Office365-Filtering-Correlation-Id: 64000f0b-e162-4ebb-f0d0-08dd18ffe05b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?q?nwI+k1Ww+eDWNbXZ4IbVF/e8msM3ZCR?= =?utf-8?q?nfYx7VBpGNPtQeAGiOdEWdiqxgwaNcU5kyllzfjchuDmJq2LNAhh0aV2D2FIGGTMC?= =?utf-8?q?ODXlLB6OMCky659al5eRBOEjSTw0eiEJzRFD7MfFjSpgYorggQz7JUHtTiMYUoYL1?= =?utf-8?q?GCs5q3NTJCMbw466O+wIsNjkbYdJBeFqrmMW4kCAE+lEuJA+lzFlMAQwCkTdhIwex?= =?utf-8?q?F8hZB3Xme7YXn2RL/3nuivJcBWjn3eZgYyTmE6JxAYOeDLArlnjhF0MXmNZtyAU5S?= =?utf-8?q?XQ2Bo803RgI4vxNvE6NadLVBpE6TPKMyHO0MjDSGWDkBSO8CJ5xTNrgQu7qmLEZaF?= =?utf-8?q?ryoVGcPB2UtGRyKSF4dFlrSLwu9ZsgGbFT0JaQVwfUrSX7IJmtPtJqZb3VxLwjhoM?= =?utf-8?q?Kh193fDznSlN/52IoXH46PY85DkjQwpEBMOrXh78YW8ctLbJvthCn8fVlHH6D4WYf?= =?utf-8?q?aW9DsGwis5up6AnBqRRVfdIkWM6PoA9BQRDoG/QVBMvf6hV4MnNkN8S1JNiM1sGC4?= =?utf-8?q?SP7NvtmJXhn3+jXv+u/d7GsZq0Y6hzgvK9aCCZ0NjPmKBTbTuM35FrQJCah1s2l1m?= =?utf-8?q?T0ciiylElxIfygi/nvD46or7ILHE7ZvI80ZUjs9wiUszVkxSHmhj7H8CTXdREmb8H?= =?utf-8?q?4EeG9rtEN9TyYOAUCxqL2Tmy3MymVc3WLWqhr9WMtiDobNTf3Qxs69pFYUltW7ze1?= =?utf-8?q?VJI9tfUnKryfLUl6CmrnzNIGJ1pbCiPbGjeXdHosNS0CoSkL9JrC9JNDunA6Wot5g?= =?utf-8?q?ETVPjCJzksk0AmTzbP+2oEG+XK0tUyjZgdMATWGduG7n+4Lo3HrpnKgg16lg2rvaK?= =?utf-8?q?reZJJ2J/N8nvogJ48UeZtknURxGrQ37nHrOYhZs+v8IMj3aQhp+asypg3X/oOmd9Q?= =?utf-8?q?RAo+KiiUHD5p0E0WkE3oHwNtRYQWoFkXtSUOAukRVmt+x2OEgU03gO+J7k4Qs3k1J?= =?utf-8?q?clP+VoBHVOQlPWxVGUF3fnaTkp/+He7I3zd/lnHGpprAi6THRraoGCH5E8QB7RCOX?= =?utf-8?q?tDFLDOCexY73XFts/+7CyVFLTGi1wIsOa6CHKHco1ZDOp94Bb3u67WJPtGnEEoY2U?= =?utf-8?q?8dYakibxU8TVlFrUcOtMF7uZB+8r8eaiYgyTJvztrs1Dd82ZgzHLLUsci6anzPPaT?= =?utf-8?q?nIGDR+KmEJvAvXUpJxjHpg4x0iW3rj9GBnGfq5r5kRwmvX0Gs7yrYd1eAS6UtrNfI?= =?utf-8?q?h/BbiYNPThxVcuDZNNZ4+DvPUQNt61jBC1AdlJrVhNJcCT1qsunM+4EaXqQAlNJ5w?= =?utf-8?q?vQSSOaBJJL85ZHplazIulaYR0JTAGbeC9uGUP7uDYCcK1f7Yu6Xb0fB3nq6Z3AVsF?= =?utf-8?q?Xw4I6ktR0RGSgRvlB/vun5CFYaKMLxgzCQ=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Dec 2024 09:49:01.5702 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64000f0b-e162-4ebb-f0d0-08dd18ffe05b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E64.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB9492 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B451F80007 X-Rspam-User: X-Stat-Signature: xwpmt4extpz15papcxjcgujemointsck X-HE-Tag: 1733824106-899302 X-HE-Meta: U2FsdGVkX1+1YfNiubdazx7Nojjc9u+QtKwcIgbwiwco+BaF7gaORbB/4e8dYTkW9JqiZ+cIah85SDBHItsKkT0p1hTU2hLM4zjgQHp6xVoT3a3H/YXGl3ME9CmUYGk736/4bcXCS4UehhQjrSdmFusK9AHRXpM2D8t4ufF2gIPUbP7m6ej6PAkkzHoYF73z0B/vhVIcjj6PY+A094bHX9x62J449yXXZhewSC1Yh5HRyJzw1KsSlrILQmtOhsW8L1Rucyv6Ot0gZmBJMnn59Gi4wWdjwy+jb19pouSpfcwI8H+F0YC7anOuPLR1REtG5NFEYQv1XvrDmJMOV64vdqZtONalY4GcNMLF87l/a8Zo/iGhKAJqG4d2OUZyEkc61C30Co6NsE4vkyIh7DggrM3lQngDgnnL9EudnC/bFwaz+kLp01RGg499dJlI/9AaVQzImmDwG3bmOJVNfm3glG479Met3Vh7vs1/9zGu+ExTwKs/agDlgZBqTln1DAM6yC47mNI0w4kj2hzC0Rb2Zsuv1BvK1TfeQGznrF3212bF2wbihDCRjWKe6zXTRgBHZLbn7pfyhabUurx7UjS0aZ1INOqfwoqO6ka9LSvsXZI4JAE7d0QmESeBGPTLhsWX625Uoi3aN3CebmU0YiSDIifQFDiL6qMAr9+L97g7n7Q7i5ExMDH01jBOHEyeg+u6sSN1pTGHgJ6azAv6VOb2CQYEKL588Ee9XKxHqZ1x7LEfxEfxSzvTM9mRJcAv8qg/J2lfjVpa9TvWF6bcgub/xp+dpLvFN/J0Ey7HfpsIdLGzAZwWuW1P1UKHksT3jVvVsT/ALFW/24PB9vX93qVZ4s/7pU0H6cA7QdPQ/WZvEkdmfXGKV+yHUsj3i2FRf6SzZdLOqlDEYolywByEZrt0VMQtoLVwhuvOm6T86iWD8QD4/A/sghOgnifxr+sfTWZABmI2JwnKZrWbLeYQG4x vaIXuq6R jKiUDjCwNh8H68n+uS1PM1cBnJnpqhN4ZC/03/UUeM5iIFBKJDNkGdArs4LTfJpHqBzYeQVX4RbfWxGCX9FFxmCdHVN8Y13STAPHhN080IzyqQuhYJKMMxyqBAu13IK58QRVA+M3KgCkC0Gbvtu294UeEF2rB6FYIF6kurRrzNLilbXsyuGh9CFBJi3jDz+Jn5dKxaRbm4KJm1Otzn9p+YdWnkMoB5L2+jkXQgB2PFTLv0b1mycLUqqaSXZGePxL0l0FNhm+E0zkxAxbXWp/nbXc3Cgk+29F9f8Ngrxdkstz7J5sqAa58PS3JJ0+TdgmDduy5jLJYdnfy5WjZBcN/DWfvEemnFDWfPPcSTLEmUdrMQESO6kQ6JIYk0juu5CagYxeC++i4nka8dB4iwOKo1X3ARK+kltRNMqshjhvihVbB6T9gnXYhYMWAxEyiA9ebvOl8iI2DGWmdyO/zcAy0DrRY2ZrrxmwpIMoD0hcQ/J+f4J51+rEnotVthgHnzTZ5MYz1aQkyS8SnvqaCy//dpDGMwbX1hSiAsLon5wG6WNWqsxqv+pQRZxYPlWJxFut9vgWLqd693CaKQ/rLrGZtw11UaORVfAQHmARzP2/VzIN2oBo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Jens, I ran a couple of variants of FIO to check how this patchset affects the FIO numbers. My other motivation with this patchset is to check if it improves the scalability issues that were discussed in [1] and [2]. But for now, here are some initial numbers. To enabled uncached buffered IO, I have modified FIO pvsync2 engine to issue preadv2()/pwritev2() calls with RWF_UNCACHED set. The FIO change looks like below and I assume this is good enough to correctly use this patchset. Also note that I am using your buffered-uncached.8 branch from https://git.kernel.dk/cgit/linux/log/?h=buffered-uncached.8 that has changes to enable uncached buffered IO for EXT4 and block devices. In the below reported numbers, 'base' means kernel from buffered-uncached.8 branch and 'patched' means kernel from buffered-uncached.8 branch + above shown FIO change FIO on EXT4 partitions ====================== nvme1n1 259:12 0 3.5T 0 disk ├─nvme1n1p1 259:13 0 894.3G 0 part /mnt1 ├─nvme1n1p2 259:14 0 894.3G 0 part /mnt2 ├─nvme1n1p3 259:15 0 894.3G 0 part /mnt3 └─nvme1n1p4 259:16 0 894.1G 0 part /mnt4 fio -directory=/mnt4/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt3/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt1/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -directory=/mnt2/ -direct=0 -thread -size=3G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest Four NVME devices are formatted with EXT4 and four parallel FIO instances are run on them with the options as shown above. FIO output looks like this: base: READ: bw=1233MiB/s (1293MB/s), 1233MiB/s-1233MiB/s (1293MB/s-1293MB/s), io=4335GiB (4654GB), run=3600097-3600097msec WRITE: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=1858GiB (1995GB), run=3600097-3600097msec READ: bw=1248MiB/s (1308MB/s), 1248MiB/s-1248MiB/s (1308MB/s-1308MB/s), io=4387GiB (4710GB), run=3600091-3600091msec WRITE: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=1880GiB (2019GB), run=3600091-3600091msec READ: bw=1235MiB/s (1294MB/s), 1235MiB/s-1235MiB/s (1294MB/s-1294MB/s), io=4340GiB (4660GB), run=3600094-3600094msec WRITE: bw=529MiB/s (555MB/s), 529MiB/s-529MiB/s (555MB/s-555MB/s), io=1860GiB (1997GB), run=3600094-3600094msec READ: bw=1234MiB/s (1294MB/s), 1234MiB/s-1234MiB/s (1294MB/s-1294MB/s), io=4337GiB (4657GB), run=3600093-3600093msec WRITE: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=1859GiB (1996GB), run=3600093-3600093msec patched: READ: bw=1400MiB/s (1469MB/s), 1400MiB/s-1400MiB/s (1469MB/s-1469MB/s), io=4924GiB (5287GB), run=3600100-3600100msec WRITE: bw=600MiB/s (629MB/s), 600MiB/s-600MiB/s (629MB/s-629MB/s), io=2110GiB (2266GB), run=3600100-3600100msec READ: bw=1395MiB/s (1463MB/s), 1395MiB/s-1395MiB/s (1463MB/s-1463MB/s), io=4904GiB (5266GB), run=3600148-3600148msec WRITE: bw=598MiB/s (627MB/s), 598MiB/s-598MiB/s (627MB/s-627MB/s), io=2102GiB (2257GB), run=3600148-3600148msec READ: bw=1385MiB/s (1452MB/s), 1385MiB/s-1385MiB/s (1452MB/s-1452MB/s), io=4868GiB (5227GB), run=3600136-3600136msec WRITE: bw=594MiB/s (622MB/s), 594MiB/s-594MiB/s (622MB/s-622MB/s), io=2087GiB (2241GB), run=3600136-3600136msec READ: bw=1376MiB/s (1443MB/s), 1376MiB/s-1376MiB/s (1443MB/s-1443MB/s), io=4837GiB (5194GB), run=3600145-3600145msec WRITE: bw=590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io=2073GiB (2226GB), run=3600145-3600145msec FIO on block devices ==================== nvme1n1 259:12 0 3.5T 0 disk ├─nvme1n1p1 259:13 0 894.3G 0 part ├─nvme1n1p2 259:14 0 894.3G 0 part ├─nvme1n1p3 259:15 0 894.3G 0 part └─nvme1n1p4 259:16 0 894.1G 0 part fio -filename=/dev/nvme1n1p4 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p1 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest fio -filename=/dev/nvme1n1p3 -direct=0 -thread -size=800G -rw=rw -rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=pvsync2 -bs=64k -numjobs=252 -runtime=3600 --time_based -group_reporting -name=mytest Four instances of FIO are run on four different NVME block devices with the options as shown above. base: READ: bw=8712MiB/s (9135MB/s), 8712MiB/s-8712MiB/s (9135MB/s-9135MB/s), io=29.9TiB (32.9TB), run=3600011-3600011msec WRITE: bw=3734MiB/s (3915MB/s), 3734MiB/s-3734MiB/s (3915MB/s-3915MB/s), io=12.8TiB (14.1TB), run=3600011-3600011msec READ: bw=8727MiB/s (9151MB/s), 8727MiB/s-8727MiB/s (9151MB/s-9151MB/s), io=30.0TiB (32.9TB), run=3600005-3600005msec WRITE: bw=3740MiB/s (3922MB/s), 3740MiB/s-3740MiB/s (3922MB/s-3922MB/s), io=12.8TiB (14.1TB), run=3600005-3600005msec READ: bw=8701MiB/s (9123MB/s), 8701MiB/s-8701MiB/s (9123MB/s-9123MB/s), io=29.9TiB (32.8TB), run=3600004-3600004msec WRITE: bw=3729MiB/s (3910MB/s), 3729MiB/s-3729MiB/s (3910MB/s-3910MB/s), io=12.8TiB (14.1TB), run=3600004-3600004msec READ: bw=8706MiB/s (9128MB/s), 8706MiB/s-8706MiB/s (9128MB/s-9128MB/s), io=29.9TiB (32.9TB), run=3600005-3600005msec WRITE: bw=3731MiB/s (3913MB/s), 3731MiB/s-3731MiB/s (3913MB/s-3913MB/s), io=12.8TiB (14.1TB), run=3600005-3600005msec patched: READ: bw=1844MiB/s (1933MB/s), 1844MiB/s-1844MiB/s (1933MB/s-1933MB/s), io=6500GiB (6980GB), run=3610641-3610641msec WRITE: bw=790MiB/s (828MB/s), 790MiB/s-790MiB/s (828MB/s-828MB/s), io=2786GiB (2991GB), run=3610642-3610642msec READ: bw=1753MiB/s (1838MB/s), 1753MiB/s-1753MiB/s (1838MB/s-1838MB/s), io=6235GiB (6695GB), run=3641973-3641973msec WRITE: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=2672GiB (2869GB), run=3641969-3641969msec READ: bw=1078MiB/s (1130MB/s), 1078MiB/s-1078MiB/s (1130MB/s-1130MB/s), io=3788GiB (4068GB), run=3600007-3600007msec WRITE: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=1624GiB (1743GB), run=3600007-3600007msec READ: bw=1752MiB/s (1838MB/s), 1752MiB/s-1752MiB/s (1838MB/s-1838MB/s), io=6234GiB (6694GB), run=3642657-3642657msec WRITE: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=2672GiB (2869GB), run=3642622-3642622msec While FIO on FS shows improvement, FIO on block shows numbers going down. Is this expected or am I missing enabling anything else for the block option? Regards, Bharata. [1] https://lore.kernel.org/linux-mm/d2841226-e27b-4d3d-a578-63587a3aa4f3@amd.com/ [2] https://lore.kernel.org/linux-fsdevel/20241127054737.33351-1-bharata@amd.com/ diff --git a/engines/sync.c b/engines/sync.c index b8be4eb3..44e9da3d 100644 --- a/engines/sync.c +++ b/engines/sync.c @@ -170,6 +170,8 @@ static enum fio_q_status fio_pvsyncio2_queue(struct thread_data *td, if (o->nowait) flags |= RWF_NOWAIT; + flags |= RWF_UNCACHED; + iov->iov_base = io_u->xfer_buf; iov->iov_len = io_u->xfer_buflen;