From patchwork Wed Jan 24 03:25:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?RnJhbmsgSHNpYW8g6JWt5rOV5a6j?= X-Patchwork-Id: 13528483 Received: from APC01-PSA-obe.outbound.protection.outlook.com (mail-psaapc01on2112.outbound.protection.outlook.com [40.107.255.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45AA88475 for ; Wed, 24 Jan 2024 03:25:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.255.112 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706066725; cv=fail; b=RyyKYNPLeKoGVYFWTK9AECYSKRdOP4GIcinT4iid6LsR/bniUku9aSC8pvu11YAnM+hPjpn2zRnz6NzOy6n20S8uLdWFh5DaXHO+0pjj9Q+D+RWEH5hHhOamPwxx8I2+ThjlKqBwVH51OW4YOx0cMsmDr/3MAi0KzofwTaRiHdc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706066725; c=relaxed/simple; bh=rlGkVuEJ5HUvWUDvRWRVJ6j253bzsMN1YM04gnzp8X0=; h=From:To:Subject:Date:Message-ID:Content-Type:MIME-Version; b=ejC7RaMTlAPkOptoZkpm3cbtVXj0zU+PWZ6ymOaOT1ike6321OwXoaVaMQAHDUJV/37LEaUIevQJga5Mw0B9eEiAyibppI8iW9FdTd/kHWoVPagZCyxomkJCbibThs6J2nNiwSfWKxy1VuZlI+sQEvNIT0YKZjbB9dMP7fcNoDo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=qnap.com; spf=pass smtp.mailfrom=qnap.com; dkim=pass (2048-bit key) header.d=qnap.com header.i=@qnap.com header.b=UjD/+uzw; arc=fail smtp.client-ip=40.107.255.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=qnap.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=qnap.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qnap.com header.i=@qnap.com header.b="UjD/+uzw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PGxZNhEjfPfryBwvJfxpb8jmWry4Z/b6dpCRZUynqekUVEuaqoxqnyqA4ez5aGxCQlQE9fhchKk5qXnIOHf4irVVQEp397VVStRseMpsi6qOo9TacKz/wg3LG87mQHkCPnCP2p66TCKgitKRHpIwvBQHwujiF5+caUuzMEFxglohUAuP5FT4Bvf9PdkzSj/Lotvii+HbpBo76T+NS6CJgTFQyUYiVHCCOwaQivnKb3qGbXE1pZtpJrgVStg0wRP2hkjfrgbV6zcSHgT34axaxsPuStdMSuhXkntZDhBHWhM2AsZ8B1t4exVhITWXDbeo+bNElgKut12zGQYtEYU9dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rlGkVuEJ5HUvWUDvRWRVJ6j253bzsMN1YM04gnzp8X0=; b=EVXeOchpoaL6Qxqw4q3MdV9+NiTVaBXATWYH9tSjndrMqgUFimQb4FQ0CNyM261+gWLrGucZ807BPbDyogVFz0AdyepZRdOl0kgAx4M6poWClmE+2LJp5r/xviGkTEGCOdW3WUyjwJpY+kbCnf2Rtj2hxx1hb2GVk8sZFe+grEr4G86I6YDDt7WWpOqCBdt2tpI+CDO5fquAknKMrBH533HOLfLinjnKesSXzAejEP3Ou6J4FQ7dnKu8sVHcWgszk1q/jqvKlE08UIILuuGzaOjzQB6LEok9PdcfWVYZ3Mg+nr5/mjnHF+pzMh11IOa9cNBREaph7GVr+BvtcXyqqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=qnap.com; dmarc=pass action=none header.from=qnap.com; dkim=pass header.d=qnap.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qnap.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rlGkVuEJ5HUvWUDvRWRVJ6j253bzsMN1YM04gnzp8X0=; b=UjD/+uzwaDX+LjZz3irNKYAkWpAZc4DuKNxPYWogF18mypvNbejEjlFBzILtGjSOlWoLj/Kc2w8n7kJIeSklYm3AVW042NMMIaqLp4MJ6P+mHoTy2sqRF6l9LFqqbko3OqA7tynja4nPIUYBvCfd0FscOaytUaGro06cJ22MYYzDVZwFwP38WkHJ0t1l1D8TeHr+x2AZccMg/oYm9hxhzAc36rETaqvqidp3EfJ5HOJgOCGJNhOlte1HYgtwqJMgZTddf2mu2oYcGdetYZXUEmrmCzbfZuGjXHqo8AakuK5Yvq5/PEct1gmjB5X9zjB+WX3K9gEUhEg5mbmIdBG1pw== Received: from SEZPR04MB6972.apcprd04.prod.outlook.com (2603:1096:101:ef::7) by OSQPR04MB7730.apcprd04.prod.outlook.com (2603:1096:604:27e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22; Wed, 24 Jan 2024 03:25:18 +0000 Received: from SEZPR04MB6972.apcprd04.prod.outlook.com ([fe80::43a3:875d:d7bd:409b]) by SEZPR04MB6972.apcprd04.prod.outlook.com ([fe80::43a3:875d:d7bd:409b%4]) with mapi id 15.20.7202.035; Wed, 24 Jan 2024 03:25:17 +0000 From: =?utf-8?b?RnJhbmsgSHNpYW8g6JWt5rOV5a6j?= To: "ceph-devel@vger.kernel.org" Subject: Read operation gets EOF return when there is multi-client read/write after linux 5.16-rc1 Thread-Topic: Read operation gets EOF return when there is multi-client read/write after linux 5.16-rc1 Thread-Index: AQHaTnBMaFmF63blQUOSXPXYY4zuRA== Date: Wed, 24 Jan 2024 03:25:17 +0000 Message-ID: Accept-Language: zh-TW, en-US Content-Language: zh-TW X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=qnap.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SEZPR04MB6972:EE_|OSQPR04MB7730:EE_ x-ms-office365-filtering-correlation-id: ec09b565-0882-4932-0670-08dc1c8c1651 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cx8IlC/bN8+bCe+aJPiAEvLiLjKOcIOOKaDRxshO3bDWR2xy3exDJVlXCIO/NcpjzY35/12yF8CnGZW1lKd+Sq8unjagQPsMmw/wEUE10f2WbV3gUjKjlTFuJB7SVbgps1L3XkHSr653HLz13rnxYua2iSswcKGpa6zHJU6ZbDT+YdeLcBEhoKG/NuWagti5CoK1gS7ySILN8sgUe8w7E+ZVH6FpPD0QirkzYG25BS5hbymKY822d9CyFIPIN0p809z+8FlxRi9FK360EJjv5ASixjtp1kATqPXvNiiFzFXivTrmBzDdjryVRFGK/pDwESQ0EmBnZrEgu5DTAUdWkjszzbA3ybgDzsGqf8D0ozxhfyhVI/2Q8+LnJqmH6IhAfiV/tKpacTS+9eI7W1//8x5mKlJqymhkoIpy48i6vSOG6IKEbPOseV1jVAIXJpuB0/3HvxKk26tRk59b/6wWxRzE56yOXBMqHaqKNs87O/Zu/u6uPUW8IZpTxH/lIUlqYIuhzu2f4PQIZ8dBGh26eQJx9VGJ/AsL/LTFlJVhYI9DSWFz9UOUnv1cwNzC0kiDC2PWi7XW1kwYhovEPgG4SVYFqenJ4j6CGEpofEibAP5525qL2qbv9FybR3IYxxAS x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SEZPR04MB6972.apcprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(136003)(346002)(39860400002)(396003)(366004)(230922051799003)(451199024)(1800799012)(64100799003)(186009)(8676002)(8936002)(9686003)(52536014)(33656002)(26005)(83380400001)(2906002)(38070700009)(5660300002)(41300700001)(478600001)(64756008)(316002)(6916009)(6506007)(7696005)(66446008)(122000001)(38100700002)(91956017)(71200400001)(76116006)(66946007)(66476007)(66556008)(86362001)(55016003)(55236004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?j8PQKCDuA/T1x7iUIVuBso0bxBi3?= =?utf-8?q?aFMql8v5YVsjo+X1syh+adG1YdlYY3oS2zW3h+NZEnYdyOsDqZjh+Ujh/aZMWub0E?= =?utf-8?q?E3Un2ZCQ6tI/nn3swNaLeuh+AqHGyFBUE2f0ZGXG53WMYdito2APkBeoiAEJLld9B?= =?utf-8?q?jC+PPcGSphSD2TUR7YzzpkQ1jkIQfdF0XxwyZ9ATJxRFKB62aG4WnK7m7iam2P/Fq?= =?utf-8?q?1A8Px1PP4EOcjh7TUTlSN2UUkbRh4uKc1PiEhk6/TvJWYCoARsYDkS1dSpvrjwGfj?= =?utf-8?q?YJNWe6dE0trIXLiZXWI0oRySflNrLL7nswHj7aEej8zjrw5BsryLI3aLrzApQg0vO?= =?utf-8?q?RyyfTblal70bJi/x2TfujpbUcaPCblqTT/S2R1EeDVl1WEdbBXH2IDudJI+bDrPMD?= =?utf-8?q?dZ0QxbsQIqvEnCQFZK14RWe/WlMbYeiRvAGFpOfOl9U58zhsprietDMUPCUBuXKpn?= =?utf-8?q?yNqMwHuEGsC7bNFdiNREq7W6NrpKhlJMip/ydRIeSHXEc3kN1v15zgDP9IRUqtcsg?= =?utf-8?q?K6t4L6Q+buW64TfhrfEEwB1xaNRi3/C9BB34y5TEpIc6VzyIeFY6ie2/PH7cyzcfE?= =?utf-8?q?gtcPCuONNfAov850hkogANPeMv+BjrCgFdSarAl7fCTdu5ymyn2R9OulP9Wu/bn1y?= =?utf-8?q?3+CP/1hPtHISF7CKBfv4whGM0o2RHRyNh8PZRpNS68zobZbami3Hb7nDqlrEryHf0?= =?utf-8?q?3HlbAkm9W/oXQKMU6TIPIS302FfF8hSaaZKTw8yyMq0kmQH6sVGkxXiGSeyx7lbxi?= =?utf-8?q?P7/0r0YqIoDc+970ZoszEEyy0gAeoT60MMWnBAwPY5hEDujibZnu0+XGeymvVGyrV?= =?utf-8?q?t+nWX81vpO5TUL25vB2x395uELXlutS5lBuCGn7lR9La9+ZutKJKMw7fMk0U/ZVhX?= =?utf-8?q?yLnpDhys9L/WS7ei1EaFlc2jVLlMIidtpO+NHbNU+GZyEeUPrdPJjylM1QHPv7bN7?= =?utf-8?q?dJW5HLiOBmG848L5woZ2BzzhJ0MZ7E1Sr+r/qvwVudUoHloViX7KO5qBiYUM6lhvn?= =?utf-8?q?6LpahJJv/8RW4NZ4E0M+XpbyPxyUU2bq9BQZwdHhfL05swwmb6bpYwkvpIZjk5Jqs?= =?utf-8?q?Rhk52Hklh2e7DSpqHi/83FT76H7NSD5Z3rQWHlGd9ENcT+KGWwDJ+Soh1PJGj2IsR?= =?utf-8?q?bU37/z54Rk8OarNm9vz71mMVpOF76EJbdIR7+IpOcY9I5s93Bz62Ij5si6lmUgwdR?= =?utf-8?q?+YGseidwYnGsYwlVO69+8slY2+VPEzR7LptOO7a243bn6o6kVLHoUx1M9Ec4cTC3m?= =?utf-8?q?TJQk1+TW1IXyfyGJ0yzUySEJAHdGiWikLB3pFOaYjKhCuuhRzBbBZCQT71eDo8zWG?= =?utf-8?q?vkxlB//tjG5QVIeobXeTVcThH6rnefsR7vyCVkheLummk9mJ5pq6yMlrdGVDhFlBk?= =?utf-8?q?pVwhob3B1FE6tI5DRtRqJVd8dOPGzGSyy30Fh2645I0ysW70S+BxppUHY+Ieh1Ato?= =?utf-8?q?YDhR9jQY3DWb6ychdubf9Y9jsTUGUkemuoo04Z4m54fjQlemMRfvl6IU=3D?= Precedence: bulk X-Mailing-List: ceph-devel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-OriginatorOrg: qnap.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SEZPR04MB6972.apcprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ec09b565-0882-4932-0670-08dc1c8c1651 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jan 2024 03:25:17.4670 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 6eba8807-6ef0-4e31-890c-a6ecfbb98568 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ++nQ3V4v8UHMziGdeDjOWugq7Zi3vSmkoQ84XRkY+JZWk/djIRcHSNaKBpVSMqYIngGAho9EI/iHl+W86TMd8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: OSQPR04MB7730 When multiple ceph kernel clients perform read/write on the same file, the read operation(ceph_sync_read) returns EOF(ret = 0) even though the file has been written by another client. My envs use Ceph quincy(v17.2.6) and mount cephfs by ceph kernel client. For the client side, I use Samba(v4.18.8) to export the folder as smb share and test it with smbtorture. The test case is smb2.rw.rw1 with the following failure message: test: samba4.smb2.rw.rw1 Checking data integrity over 10 ops read failed(NT_STATUS_END_OF_FILE) failure: samba4.smb2.rw.rw1 [ Exception: read 0, expected 440 ] After some testing, I figured out that the failure only happens when I have linux kernel version>=5.16-rc1, specifically after commit c3d8e0b5de487a7c462781745bc17694a4266696. Kernel logs as below(on 5.16-rc1): [Wed Jan 10 09:44:56 2024] [153221] ceph_read_iter:1559: ceph:  aio_sync_read 00000000789dccee 100000010ef.fffffffffffffffe 0~440 got cap refs on Fr [Wed Jan 10 09:44:56 2024] [153221] ceph_sync_read:852: ceph:  sync_read on file 00000000d9e861fb 0~440 [Wed Jan 10 09:44:56 2024] [153221] ceph_sync_read:913: ceph:  sync_read 0~440 got 440 i_size 0 [Wed Jan 10 09:44:56 2024] [153221] ceph_sync_read:966: ceph:  sync_read result 0 retry_op 2 ... [Wed Jan 10 09:44:57 2024] [153221] ceph_read_iter:1559: ceph:  aio_sync_read 00000000789dccee 100000010ef.fffffffffffffffe 0~440 got cap refs on Fr [Wed Jan 10 09:44:57 2024] [153221] ceph_sync_read:852: ceph:  sync_read on file 00000000d9e861fb 0~0 The logs indicate that: 1. ceph_sync_read may read data but i_size is obsolete in simultaneous rw situation 2. The commit in 5.16-rc1 cap ret to i_size and set retry_op = CHECK_EOF 3. When retrying, ceph_sync_read gets len=0 since iov count has modified in copy_page_to_iter 4. ceph_read_iter return 0 I'm not sure if my understanding is correct. As a reference, here is my simple patch and I need more comments. The purpose of the patch is to prevent sync read handler from doing copy page when ret > i_size. Thanks. diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 220a41831b46..5897f52ee998 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -926,6 +926,9 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,                 idx = 0;                 left = ret > 0 ? ret : 0; +               if (left > i_size) { +                       left = i_size; +               }                 while (left > 0) {                         size_t len, copied;                         page_off = off & ~PAGE_MASK; @@ -952,7 +955,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,                         break;         } -       if (off > iocb->ki_pos) { +       if (off > iocb->ki_pos || i_size == 0) {                 if (off >= i_size) {                         *retry_op = CHECK_EOF;                         ret = i_size - iocb->ki_pos;