From patchwork Thu Jul 18 08:45:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ofir Gal X-Patchwork-Id: 13736252 Received: from DU2PR03CU002.outbound.protection.outlook.com (mail-northeuropeazon11022116.outbound.protection.outlook.com [52.101.66.116]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7ABB12DD90; Thu, 18 Jul 2024 08:45:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.66.116 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721292334; cv=fail; b=qtcDp3mxnc0XGVKcGBJOh/pax6guPLa4x4zu2/j2TCI7FXFvt2AKuD/AJD9yeHoQqR7d8R1gw9WAZK+id8hyD+LB1pVonHYfgf3TLmEHbTBROCJH5/8/ic6o7uUiuDSPskzDF6TAQzgTe3sPl9EVgh3VLIrSSbTavosbSFxp8PQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721292334; c=relaxed/simple; bh=d8/7VPD9Uje8UO+P8LZRe+KqKIkoEZ6YXRkBUcLWC6U=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=tQe3m7BO2lw0rkYecwQ82r+ssnVRlkTewtqVL0Z+3GlzNimEu81EBZR8DlKmAiiMc30Bom2o7E0LJDLOkKYntPI2SDVlote1v3phJbq4eBACQwvDWeZXt2BODMg7gQmsPvCTVQAQOF8BBgZbXES9jKNDLj5BYEog2NZcZipCCNg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=volumez.com; spf=pass smtp.mailfrom=volumez.com; dkim=pass (2048-bit key) header.d=storingio.onmicrosoft.com header.i=@storingio.onmicrosoft.com header.b=fJvfwWUa; arc=fail smtp.client-ip=52.101.66.116 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=volumez.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=volumez.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=storingio.onmicrosoft.com header.i=@storingio.onmicrosoft.com header.b="fJvfwWUa" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uXduwTDUfCvg6RKbgz4DS+fJ6X4mLYwIsqiKM7fBdJwOaw2tx239HtfW3z/Kz/WWCMDemyXDs1cMnkYkC3O0xOgzZj3YPz8D6wt3gZcxdmhT8a9Wc9ZbxB2tf20dfOapIjcxCYhon9f4V/rq18/blGWMZ8Xu4+HEWqiWZOBmsvWrmgE6/38d79+5iBRB5QZMj0aHUnT0m5ZP4tlKspgJzK5kvTCFIlclS6EvRC96qEo67m2LnE0EquhuD12cW9aV5JjlLEmbErvKCRmKBcRAgvV1KVs5/03yYyM2Qq7l5ZJAUj1yK6P1pdv9FYndVNkMq0PMqlzzTVQbGZYgjaoI3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=thX/hIPBbq9Dop7ooUlbEo+vRH353mCOFs7ehP9bt4A=; b=ig9F2UL0uCOeSWgBSvfgLC1RssRBDNlhKP9nB+ZxpOs9QQ6hyQ0835kuKt4Jk6kSfwZ8aPq+nS91Wl3KSBC/gDsfBYh22pnF8x9/KTsbWQWD8YZKJj1+bje7Svr4hhZamamGNDakGExtxMOs8kCBLxs8c0Zc85qzMvHDfTlFfRxtdARBIat4A4J8OEkBU+SlFkDMVAqEktG/fz0uwroVs/BTkkv4XIM4y3ziWSbL+U2oug3ZqZ5pj7aUl59ZoW6VRg1hSANAq2H/dxLRFSr33hdfaqEsJeUOKlnPJcE9JHHogKe5yAgBQNBQ1lQWVAUhKFpoOcfEuAzFkQo9KgRO4w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=volumez.com; dmarc=pass action=none header.from=volumez.com; dkim=pass header.d=volumez.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=storingio.onmicrosoft.com; s=selector1-storingio-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=thX/hIPBbq9Dop7ooUlbEo+vRH353mCOFs7ehP9bt4A=; b=fJvfwWUaY5APp8LKufPTB6cCaUmtpRFJXqZPMeqnKGvJSsS+Z3+qXEc4nhJETRlYizZOCGf+fmE1d82JnQCR+FglxUkqHa6tSsikwjLjdcdm0K2hS1RHgPQH4W4KIwtdSRx2cB3LLzSOBU6L9lBR/EHjlekY+ZKdU4hDwotI+ZoBT8EMAmDaRYsNgiB7NY+1tn/wEixfp9wwdbQPqalBgqmxj2d+vyBB7XHXqrJJWniMiUL6TpOvTPWBPK4NiagNXobRMwHMoO+Jci16DcApxQ4+7ZmMySIWj4FVmOlMEaQEvExmuoogHSF9uSeSidjSR+E+2V9x90Vj5bvFmYOx/Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=volumez.com; Received: from AS8PR04MB8344.eurprd04.prod.outlook.com (2603:10a6:20b:3b3::20) by PAXPR04MB9005.eurprd04.prod.outlook.com (2603:10a6:102:210::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.19; Thu, 18 Jul 2024 08:45:26 +0000 Received: from AS8PR04MB8344.eurprd04.prod.outlook.com ([fe80::d3e7:36d9:18b3:3bc7]) by AS8PR04MB8344.eurprd04.prod.outlook.com ([fe80::d3e7:36d9:18b3:3bc7%5]) with mapi id 15.20.7784.016; Thu, 18 Jul 2024 08:45:25 +0000 From: Ofir Gal To: davem@davemloft.net, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, netdev@vger.kernel.org, ceph-devel@vger.kernel.org Cc: dhowells@redhat.com, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kbusch@kernel.org, axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, christoph.boehmwalder@linbit.com, idryomov@gmail.com, xiubli@redhat.com Subject: [PATCH v5 0/3] bugfix: Introduce sendpages_ok() to check sendpage_ok() on contiguous pages Date: Thu, 18 Jul 2024 11:45:11 +0300 Message-ID: <20240718084515.3833733-1-ofir.gal@volumez.com> X-Mailer: git-send-email 2.45.1 X-ClientProxiedBy: TL2P290CA0030.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:3::16) To AS8PR04MB8344.eurprd04.prod.outlook.com (2603:10a6:20b:3b3::20) Precedence: bulk X-Mailing-List: ceph-devel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR04MB8344:EE_|PAXPR04MB9005:EE_ X-MS-Office365-Filtering-Correlation-Id: ca030147-1061-4271-6d8b-08dca705f7e9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?gVAFvmRCnMJriBy3KAMXPTOn+s34mOy?= =?utf-8?q?6iyMJGi0QUkg0ORHQk2KaLrlfTUaJUv7opQGxZ8JmIRHdRgQsQgv4Pxt8/AQw1feO?= =?utf-8?q?a9k0RDqHNrZ8GGmWdFdnt62J0cxIrfEFH4LB+QdThUcY+X2QGC/jLsmYSLKPGFr2q?= =?utf-8?q?xA1TFrUWCQepEfjqdr0ougFrk214mC8mLmMRQmeu3aXFxLHEBOz9/1eYNI0bY9v1/?= =?utf-8?q?sVe7qqZNzDRnBRVAHXrl3Vntavo8joTEk1wV1p80w+gj+nJJDMWtVD7SnkTblmpMx?= =?utf-8?q?KuONRSFWpDfjiz2SgsvsYRmOfePZHEQRgLgewxYDoRtzbxbhOm6oo/S4aOW1zbFuF?= =?utf-8?q?rc3wn208pCOysx2CLojBXs+8CrSMD8r9WQsaL8dpMyLxIyjtUR/HaGsBBPsV+dlI3?= =?utf-8?q?+Ju00BpDYLjjGEj01QeGbVjIZ+U0ah6G3utJLIVYZfCpTcFqdF7gZP+efCa7uE/gX?= =?utf-8?q?ybpFGOqsxbLX9p2jHKmklgVt29AW06e9ID2K1HybtSuO3jxqBUCwoARtFbFBlrjoh?= =?utf-8?q?boX96cH5yt69BKgTkyqfIA1JkA/htZHlUHmzOsUWHf7VgeJqA24qP+Ib8RT/p9wZw?= =?utf-8?q?cvbNg/MAUl/3GOZB6u9kIhaE4QuvQJwVUPEVQVxE55ziDaD2iJx0xHv0zNgUuxd2T?= =?utf-8?q?jfVmeaGEPKxm+f4+wQtn8O04/1sGiciAW4RPLxVh5iEymh0IduGgnwNZl88vy8Z9L?= =?utf-8?q?GIlCiyBS6MzUszyLqM/JEz5Tn1oiK1oOChMdwDLgYIKTeri2ikjNheFp1a0kIA4tY?= =?utf-8?q?GcnhzHovKB3v3mOro/skLQqQuslMp6wG7v7jdVzqYGNqO7UfP3Ey2Ta0WHCxA10jL?= =?utf-8?q?c154Jp04pyqQrre0hmjcKMVd6R1SnWKpFiUCs5Qvdo5E367v5d1cW+MpZbc8KK95+?= =?utf-8?q?iBw4a+M4XteXhWp/6cEJZ7/lUXhLFs8UGsXkLEivkZQxJKFFhGeoI/Pvo3CTOFzOh?= =?utf-8?q?Niw6Upoj8jPSy2LwOJZd5q5HybWnrw2q+OGER1yNj8022oqXbZb7GjuFMsiVngsSe?= =?utf-8?q?u82Qg70Nt+A2oS1krn6GC5VnqdANSnS41sNEHIsGwPxwNhZaQwUyScm0Kw62Pbfks?= =?utf-8?q?YGZrD+9XbDglaNZ6Lrd2Q2imB/5nNDZ/Cx03W0JRxP0ilaTykrQE7AyxMl5IZQy32?= =?utf-8?q?1CpDm6N4n+eicCHLiTdr6LbtFgQKKgZBI0uDqCYM4zIa19hYE3xMgleVa3O3ZJAxx?= =?utf-8?q?Yx4EhxnDZxirZxja0NQj5Y/V4xgyVrJvDr5rSxWpn3uif6mDeFcPTLHsZIal37mZA?= =?utf-8?q?QdHlWSjloeCKRNwoYaoyAptJZlCWz/pvLBljRTjbjN3hlS0CIpTBZ2hW8432wTJl4?= =?utf-8?q?jXVSbRS73iy+CFwyKkf1uIYXnx07avFnSA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR04MB8344.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?fTpQmr5PM9vXgZZoJOA3fOI7iiWB?= =?utf-8?q?F8/guGH6J+zRqBC1l28fdZw/LVm7kYlRVXV0qaEEK5aHuPwXoljKsy8XbE6eGsp7W?= =?utf-8?q?NLfRKR0xJCiINhHe4IxqpwZo2KgnjkprsOBWnXpyCwlmruT4AkC2l2sdNpi7D6sZv?= =?utf-8?q?YhGdN1Hbk87gvMhoB4poDYsCs8kScGtS690Sqyv/32syMoCg8a/zuXdFakWXe2o3N?= =?utf-8?q?bXfA8CtR55l3dBDQ4MNJ3KqIsWADbQFOlJocCq7J31Mlwj+zVBOOAnsanmDPG7foi?= =?utf-8?q?fzU30J7s2VhUkZ2mrdMXU25qX6DRzqOb9Cfvb8K/H2IUb+rznWI+ZFbYUJ51COitQ?= =?utf-8?q?uskrKD1xfY/weGReMFXG6W0ILEGBbrtJ7hl03E+E44BbFSF/gE4pWCXA/HH0tWvwE?= =?utf-8?q?TouzIj+eU5K6M9wfcJh0tEgjoGUqjsQbHCPcn2nvOhaX0kERTPnk0WDezzu9p3ZwY?= =?utf-8?q?ICJB5G14Nv0PQjyTcrKtqWyAVpbJLieuurDVInzum5fwacFuLiez95kpIoTeFnegT?= =?utf-8?q?frdYco5llbVx118J/iz6RgRTZC5DTK6fyOxscMeAvTsAbWlMShd5ZFlHV1jPkNupC?= =?utf-8?q?QarI4aKkEs02FhQYntiYWwtH+5T3dqvrg/qDjC+R5noOA7IdDAZJ9NltdiEt5mBhc?= =?utf-8?q?9N3zzwQoYeyr2boKK4No0ijELHXPfrkgo9/K7NlVvKm/jOwl0DWxxYzoS2wH1nTLE?= =?utf-8?q?daFrKetltIDgodrcAyxLJAj9Fa9oTB+NwrhNQ2ntQmzHLPVIuhSS1M/4DQJTGpgDa?= =?utf-8?q?nDL8shJMzR4OHrJv2nbqcyAv2Vq1l9NLwh3BEwoT80ke9SjSooaCIfNrG9JMx7Bbw?= =?utf-8?q?GSwxcKDQ1g9RH9lLUntf3WenVWiPseYmEcMCAN2lZowBctKDcnAkU8n2B3i5TTwIS?= =?utf-8?q?AvyL9YYUmS3Agi3L3xONIjP8dSnpAKW9uMYjuqwpjLUlMvxjO5wfNCg9XfxGUX4gf?= =?utf-8?q?vYp8qxlZR1g8HM42AltYlvsElBBRppGKkL2P12uUxMHF5TwvTsOMb0nt+jC4322Ej?= =?utf-8?q?Oy+b4MjFdAjKmiZi6QHmF2UC9usL63BM4IVAr6Q4yroBsp6PqJYLc8wVfPvCeZciT?= =?utf-8?q?WZC+JWKi0PuEgwos0gWygHuNa2fBFKLc1N3LV9j7jEPtZ8LDiwdiITcaZrdox6yJN?= =?utf-8?q?wZPnKKXkybodA5Q9ggWzK8FspUJ5i2h/Y7dYSr2TJERwYJ9jk1q8z4pTbCx+2+q5T?= =?utf-8?q?5b452mG4mEWYvTQvpA44mJxK8TbuX3Ha0u8mrYJ57kBTB/M/d2Zq0YHGKjv4VkdnL?= =?utf-8?q?apLWNa3+jVRAZOlqPLNS0WHj+Z/pUU0I/lHi3vFh2/DkkxPtN2MTPN25NyuMoxCZ7?= =?utf-8?q?SKSScOb00icecuDVEeXpwn9NMBb1fJY53pNN4fiuHU8VY2Zn6bAAxkx5DIKKnjs2f?= =?utf-8?q?MXX/tn6oINSCOvQt2io21SZ2C4XeooOUMkDS5+H/rIqxNX+kVx8nNnhE/+N4V83IQ?= =?utf-8?q?92RJ5ZEHTKN5Rp3pzBCAx8+SdUNTUMod/T7x+h4b3tNS0DWrg3OyOzvfAdMyR8ubv?= =?utf-8?q?VCWJHrFvNnk3?= X-OriginatorOrg: volumez.com X-MS-Exchange-CrossTenant-Network-Message-Id: ca030147-1061-4271-6d8b-08dca705f7e9 X-MS-Exchange-CrossTenant-AuthSource: AS8PR04MB8344.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2024 08:45:25.8323 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: b1841924-914b-4377-bb23-9f1fac784a1d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6G2dfvnpms6F9Mfyvlr4bAcLT3HMuUfEqYX/hVw9Pxq3DZeYrBkJhMGz7fS3f1WmKi+Byrfi7nFL/fw5bHXwiA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR04MB9005 skb_splice_from_iter() warns on !sendpage_ok() which results in nvme-tcp data transfer failure. This warning leads to hanging IO. nvme-tcp using sendpage_ok() to check the first page of an iterator in order to disable MSG_SPLICE_PAGES. The iterator can represent a list of contiguous pages. When MSG_SPLICE_PAGES is enabled skb_splice_from_iter() is being used, it requires all pages in the iterator to be sendable. skb_splice_from_iter() checks each page with sendpage_ok(). nvme_tcp_try_send_data() might allow MSG_SPLICE_PAGES when the first page is sendable, but the next one are not. skb_splice_from_iter() will attempt to send all the pages in the iterator. When reaching an unsendable page the IO will hang. The patch introduces a helper sendpages_ok(), it returns true if all the continuous pages are sendable. Drivers who want to send contiguous pages with MSG_SPLICE_PAGES may use this helper to check whether the page list is OK. If the helper does not return true, the driver should remove MSG_SPLICE_PAGES flag. The root cause of the bug is a bug in md-bitmap, it sends a pages that wasn't allocated for the bitmap. This cause the IO to be a mixture of slab and non slab pages. As Christoph Hellwig said in the v2, the issue can occur in similar cases due to IO merges. The bug is reproducible, in order to reproduce we need nvme-over-tcp controllers with optimal IO size bigger than PAGE_SIZE. Creating a raid with bitmap over those devices reproduces the bug. In order to simulate large optimal IO size you can use dm-stripe with a single device. Test to reproduce the issue on top of brd devices using dm-stripe is being added to blktests ("md: add regression test for "md/md-bitmap: fix writing non bitmap pages"). The bug won't reproduce once "md/md-bitmap: fix writing non bitmap pages" is merged, becuase it solve the root cause issue. A different test can be done to reproduce the bug. I have added 3 prints to test my theory. One in nvme_tcp_try_send_data() and two others in skb_splice_from_iter() the first before sendpage_ok() and the second on !sendpage_ok(), after the warning. ... nvme_tcp: sendpage_ok, page: 0x654eccd7 (pfn: 120755), len: 262144, offset: 0 skbuff: before sendpage_ok - i: 0. page: 0x654eccd7 (pfn: 120755) skbuff: before sendpage_ok - i: 1. page: 0x1666a4da (pfn: 120756) skbuff: before sendpage_ok - i: 2. page: 0x54f9f140 (pfn: 120757) WARNING: at net/core/skbuff.c:6848 skb_splice_from_iter+0x142/0x450 skbuff: !sendpage_ok - page: 0x54f9f140 (pfn: 120757). is_slab: 1, page_count: 1 ... stack trace: ... WARNING: at net/core/skbuff.c:6848 skb_splice_from_iter+0x141/0x450 Workqueue: nvme_tcp_wq nvme_tcp_io_work Call Trace: ? show_regs+0x6a/0x80 ? skb_splice_from_iter+0x141/0x450 ? __warn+0x8d/0x130 ? skb_splice_from_iter+0x141/0x450 ? report_bug+0x18c/0x1a0 ? handle_bug+0x40/0x70 ? exc_invalid_op+0x19/0x70 ? asm_exc_invalid_op+0x1b/0x20 ? skb_splice_from_iter+0x141/0x450 tcp_sendmsg_locked+0x39e/0xee0 ? _prb_read_valid+0x216/0x290 tcp_sendmsg+0x2d/0x50 inet_sendmsg+0x43/0x80 sock_sendmsg+0x102/0x130 ? vprintk_default+0x1d/0x30 ? vprintk+0x3c/0x70 ? _printk+0x58/0x80 nvme_tcp_try_send_data+0x17d/0x530 nvme_tcp_try_send+0x1b7/0x300 nvme_tcp_io_work+0x3c/0xc0 process_one_work+0x22e/0x420 worker_thread+0x50/0x3f0 ? __pfx_worker_thread+0x10/0x10 kthread+0xd6/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x3c/0x60 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 ... --- Changelog: v5, removed libceph patch as it not necessary v4, move assigment to declaration at sendpages_ok(), add review tags from Sagi Grimberg v3, removed the ROUND_DIV_UP as sagi suggested. add reviewed tags from Christoph Hellwig, Hannes Reinecke and Christoph Böhmwalder. Add explanation to the root cause issue in the cover letter. v2, fix typo in patch subject Ofir Gal (3): net: introduce helper sendpages_ok() nvme-tcp: use sendpages_ok() instead of sendpage_ok() drbd: use sendpages_ok() instead of sendpage_ok() drivers/block/drbd/drbd_main.c | 2 +- drivers/nvme/host/tcp.c | 2 +- include/linux/net.h | 19 +++++++++++++++++++ 3 files changed, 21 insertions(+), 2 deletions(-)