From patchwork Tue May 10 23:22:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junxiao Bi X-Patchwork-Id: 12845598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aib29ajc251.phx1.oracleemaildelivery.com (aib29ajc251.phx1.oracleemaildelivery.com [192.29.103.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06736C433F5 for ; Tue, 10 May 2022 23:22:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=oss-phx-1109; d=oss.oracle.com; h=Date:To:From:Subject:Message-Id:MIME-Version:Sender; bh=oot5o0qIZmbDAGZWqKds9BQfufCXugpakMlBP0h8hjQ=; b=LKBxYm8tlQty/8KNS8wt7ErOGrCq7woe/eIBdcNXbjzyVhn2C0u7+Bo61CckSVdGoTn6e2w2Txd1 ebeL3ftyLIEd/j0gGt90b56f9YWTsa6Ewrw96TFf+9DnxNStSxY0d26C/NfOTbQVFwJ2iyTKWmNZ G5fAjvewHYSryqTW+dlrpk5ELwjttaaz1TZ8KIVi8f0lJ4sxto6IJEjoHtXJyjuPZZ8NQaYKdlw3 LF7CM3HeK4Ej5cPZm2u66ZncpRtOAnACqw+3YtppH6HYfkZX6y0C/G1O3szKi3qksgzZYv2fZR0i 38MP9Zd9VQfBR3EwH/FbW7eDF+N55kWE8/iz9A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=prod-phx-20191217; d=phx1.rp.oracleemaildelivery.com; h=Date:To:From:Subject:Message-Id:MIME-Version:Sender; bh=oot5o0qIZmbDAGZWqKds9BQfufCXugpakMlBP0h8hjQ=; b=J21JW3Y3c4gPXMO4PMGTQw2V00LGYdxhqQeBVdru+27NMN3iRmPNceJ+A6gZAMhTyGheddCOWB39 ZzROcBw5IpQlm9UvPXKUjjXSyUVXls9L+zrZrVTlWQRZFzsJQJ3aeIbev5/LPynckufe04iOvTUg /b71wD9KUJS6/HISIrIChGA9ltwzGbYQKU8zsesIDkoGsQdugZU+aVkRSH16Be8Ql7ya/Rnqmcq2 SLhpkGDwYd8LvPcl1aWOyw0XxvhMHSRoTJMyBP0v1bkYP6mtg0kqdzOS+Pr3yEbd/fA9fJmcIML7 U664QUW+/XkYBEBw1/dAWiQR9EG83CBRXCZQew== Received: by omta-ad2-fd3-202-us-phoenix-1.omtaad2.vcndpphx.oraclevcn.com (Oracle Communications Messaging Server 8.1.0.1.20220413 64bit (built Apr 13 2022)) with ESMTPS id <0RBO00FZUWXXLTA0@omta-ad2-fd3-202-us-phoenix-1.omtaad2.vcndpphx.oraclevcn.com> for ocfs2-devel@archiver.kernel.org; Tue, 10 May 2022 23:22:45 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=SfUYpXTh3sOhXD4xLozBhzbO6i94j01xvPlCV8ipnhA=; b=cwGQg9yBqZ6IahIyi+x/4I0HxDn47YcFrXiqAeEc24tSTMCl3fCvI42OGNmtzyCJ7/1S oiOT0PIEJhQ5ZPz8MeJktdSXj2grRfCKUSdbKdN0Ba38vhp8ZdzaOAypjdR+jfpYVjuD a9WGurjWADAIWX7/f4q8Fn7vsFaZMSkfKL+gLhhs7xlkUecm7WuO8aJ6QRTmVXFHHwZ0 tFze+vmrIX1X+1cJqObbYdTV9VDqm7dimyQa0bPIO7HQAnmSFwVxTTnnfVYUBA0J027g cWYFiV3ouUwdgV3yVsa6gGy64Q0CGimjnnpqN8OuxQ9WqzSGvfzdOxeSh0C6//qYnwmY sw== ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UrgPj5t6XrIUylXcgBaoD2k4bgv8gsHsmFeKn3gXVLC7b798CPD3/OrHEYVAqq1It0wcv07Au8ic1svlm1mf7un/bVrRrYyFdpGVktcCXdH3h6O6QzuPOFxnzb9zxYv23MUCLUFwsAtORBxsEeL8r29ztNcccxcNzZ3HQmUDCeCslzDtfVcxGomnkxEP8ltYciZ6+8h0b0eX7O+AEdRJ/ErNtk+7+jMV/84UhLDxB0Tjj171dXVtp5XaqULwoOT+YMli68ufy/11IGSYIITmbTKstn8OttZHetPpfuE4p60grrj1di0u2ZLlSdW2HYThhuP034UxCC4lfDG+li48xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SfUYpXTh3sOhXD4xLozBhzbO6i94j01xvPlCV8ipnhA=; b=EwG3CZE6bWpVT0KtWVrpMs7+yqM+H9qFn3uZpxRBogaMazPBPewCZNYfFXiaUuBq8aMk50fPdqCRKDbL+RG80kLB6SNH1NE3F4AZ11jcqC3syw3OHZHpUE+JikJFCyWTqomCIWIUrQg8w4RNMz/V4YghDD+Twly5B3WWUNFmg76HO+Og396ft4HfPb/+SJCF41l17V349+TU8o8gzYyz4XKtQ0ceUhPgO/cLKxB4mM7au7VBcq/wmYbu/UmOn40w5LEtWHACZ8lAFS5Fu5/E0eMy6VjTaKgtrmLcBN/Mhtfou/j7hC+bm9vIMYH8oLEhZiCNZY57ZDEJCabCInhrFA== ARC-Authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SfUYpXTh3sOhXD4xLozBhzbO6i94j01xvPlCV8ipnhA=; b=s5i9Dwwo7oJ9EJRCS6vFF8SGHgVdoPdmq6YSwkXFDnbQbV1KGHB0fi336xN64ueEcNZUW2yE4ov8VwIBRn6v3G3m6o3kly1z2nhqLuxVLbJfam2EX6+vENA+j221CH6VaWuKKeO1Wf008f4Cs3sh+/IOlHhPIOGOc1WKxS44u6g= To: ocfs2-devel@oss.oracle.com Date: Tue, 10 May 2022 16:22:12 -0700 Message-id: <20220510232213.23435-2-junxiao.bi@oracle.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-reply-to: <20220510232213.23435-1-junxiao.bi@oracle.com> References: <20220510232213.23435-1-junxiao.bi@oracle.com> MIME-version: 1.0 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:-1; SRV:; IPV:NLI; SFV:SKI; H:SJ0PR10MB4752.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:; DIR:INB; X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2022 23:22:33.1737 (UTC) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-10_07:2022-05-09, 2022-05-10 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205100097 Subject: [Ocfs2-devel] [PATCH 2/2] ocfs2: dlmfs: fix error handling of user_dlm_destroy_lock X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Junxiao Bi via Ocfs2-devel Reply-to: Junxiao Bi Content-type: text/plain; charset="us-ascii" Content-transfer-encoding: 7bit Errors-to: ocfs2-devel-bounces@oss.oracle.com X-ClientProxiedBy: BYAPR07CA0053.namprd07.prod.outlook.com (2603:10b6:a03:60::30) To SJ0PR10MB4752.namprd10.prod.outlook.com (2603:10b6:a03:2d7::19) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8ebc8b82-3e3e-47b2-9426-08da32dbf5df X-MS-TrafficTypeDiagnostic: MWHPR10MB1840:EE_ X-Oracle-Tenancy: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aUYaqFQvEKswLVqhEvOCqfMlKKElTtKz4DiB7laHaORPTq4k1eYGPznYWTpS4QhFFq3NLvMDpJA8SpWamkCPT4hADMHiugpqkyBeOwgkbTDr46aJMIdzirhrWHri/Tl7xtBYl66pYdVVflV7fjuAWVo6wRXv0YVmFwhak2CcZ1xF/KdZfvbzYPGGpvBSJy5oZwiwj37nKElV5EGJlFKjW6zWymaL2SP/zK70Jd0zWZcy1C+n3q9CR8wy53UbQfHZfCzHOYjv7kXMGG9ARrsQHNw+YhiJfHZ970uoYkEPxVSnzgk/P3fG4CAq/dc5JHAtbWyknxwT5EW+p/22wQbwtHlZ3N+PH0CtizZBDnDP5JG9m3vD8309IYGoxTeE2KLNd2PxnHuLXLZOpcZWVALurntrmshBrYDNRZdJLNwRmjb8ORetp+Dn9Ta5KABCWXXzeRnmTU5OKvq1wNhbnGlXvfrF09HBoErwLwO+XrgKpDO8SF/FxcMAfKzBOZzqGFPyyDtyACdF4kAzXx8sYsgNZdrz/owojdpVOuI1T1GSB4cCujewW+BR2fkuBApb/DSyJnTFdhTRg67urW2gqvRsflQ8VxpMrPMFmA6bl9FYAAo= X-MS-Exchange-CrossTenant-Network-Message-Id: 8ebc8b82-3e3e-47b2-9426-08da32dbf5df X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4752.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: USCbOuaV/KXkCHD3B/m05jgjthJO439Pw5ip+x90ee2CqO6oajl16/o8MkWL2HEsFwDX4J5+6ntAOjR7XFsjWA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1840 X-Proofpoint-GUID: w0axRpGTJsunaDH0bGZRny6q9dvw1kwB X-Proofpoint-ORIG-GUID: w0axRpGTJsunaDH0bGZRny6q9dvw1kwB Reporting-Meta: AAEel7/S7llhyei3oZi65/BgO8vrvDTu+AjahXqQOVX6O5qZar9Uri0ufkI/y8Jw tO1x9rANAmZ7G31rYIyGcEydWGIBq9HAhHr0EC0bPFdH4CqbVdgE3TXeuTvWNZnb EYPZ7VI6q9BtLcIFpgY/rD4jGZ2MAH5bOOa1hKmzgEkPhUnbP64jV8STdwaLFP76 SoCmwmvyZN0PGywfCwcD+6Ziek9poVMlX6Ze/PbSxOHyN+DY4K73fbyeA4JXhyaP ep58IC+LC++IxN6sPRyXTcNG9K65zNLC14/Qv0axpgzy65SfWQG0esdsW3pPAY8V UklZvj09gB+JLOCu/1EAWKRtFGWXC0q+Xv/L4I4ECXsFjY9DrMOImxeXx603/YYb Cb1NnJjWYElwP8cwmP9MeTFFTANLBbbT0yeIGvpGwSwFtn4GiwgLtCbASpQuhIYz UODI8Ylz7p20iI2MvMVL2O+t8nGuqGlSMAarsP4z2puWO6ZW+CpO8zPgqypT/+Q9 r24LbNU5ldI6lU7IY6mBioArZlT6mDEjKRKZAqs1Pi0= When user_dlm_destroy_lock failed, it didn't clean up the flags it set before exit. For USER_LOCK_IN_TEARDOWN, if this function fails because of lock is still in used, next time when unlink invokes this function, it will return succeed, and then unlink will remove inode and dentry if lock is not in used(file closed), but the dlm lock is still linked in dlm lock resource, then when bast come in, it will trigger a panic due to user-after-free. See the following panic call trace. To fix this, USER_LOCK_IN_TEARDOWN should be reverted if fail. And also error should be returned if USER_LOCK_IN_TEARDOWN is set to let user know that unlink fail. For the case of ocfs2_dlm_unlock failure, besides USER_LOCK_IN_TEARDOWN, USER_LOCK_BUSY is also required to be cleared. Even though spin lock is released in between, but USER_LOCK_IN_TEARDOWN is still set, for USER_LOCK_BUSY, if before every place that waits on this flag, USER_LOCK_IN_TEARDOWN is checked to bail out, that will make sure no flow waits on the busy flag set by user_dlm_destroy_lock(), then we can simplely revert USER_LOCK_BUSY when ocfs2_dlm_unlock fails. Fix user_dlm_cluster_lock() which is the only function not following this. [ 941.336392] (python,26174,16):dlmfs_unlink:562 ERROR: unlink 004fb0000060000b5a90b8c847b72e1, error -16 from destroy [ 989.757536] ------------[ cut here ]------------ [ 989.757709] kernel BUG at fs/ocfs2/dlmfs/userdlm.c:173! [ 989.757876] invalid opcode: 0000 [#1] SMP [ 989.758027] Modules linked in: ksplice_2zhuk2jr_ib_ipoib_new(O) ksplice_2zhuk2jr(O) mptctl mptbase xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn cdc_ether usbnet mii ocfs2 jbd2 rpcsec_gss_krb5 auth_rpcgss nfsv4 nfsv3 nfs_acl nfs fscache lockd grace ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bnx2fc fcoe libfcoe libfc scsi_transport_fc sunrpc ipmi_devintf bridge stp llc rds_rdma rds bonding ib_sdp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) mlx4_vnic falcon_kal(E) falcon_lsm_pinned_13402(E) mlx4_ib ib_sa ib_mad ib_core ib_addr xenfs xen_privcmd dm_multipath iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core i2c_i801 lpc_ich mfd_core ipmi_ssif i2c_core ipmi_si ipmi_msghandler [ 989.760686] ioatdma sg ext3 jbd mbcache sd_mod ahci libahci ixgbe dca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel megaraid_sas mlx4_core crc32c_intel be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi ipv6 cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ksplice_2zhuk2jr_ib_ipoib_old] [ 989.761987] CPU: 10 PID: 19102 Comm: dlm_thread Tainted: P OE 4.1.12-124.57.1.el6uek.x86_64 #2 [ 989.762290] Hardware name: Oracle Corporation ORACLE SERVER X5-2/ASM,MOTHERBOARD,1U, BIOS 30350100 06/17/2021 [ 989.762599] task: ffff880178af6200 ti: ffff88017f7c8000 task.ti: ffff88017f7c8000 [ 989.762848] RIP: e030:[] [] __user_dlm_queue_lockres.part.4+0x76/0x80 [ocfs2_dlmfs] [ 989.763185] RSP: e02b:ffff88017f7cbcb8 EFLAGS: 00010246 [ 989.763353] RAX: 0000000000000000 RBX: ffff880174d48008 RCX: 0000000000000003 [ 989.763565] RDX: 0000000000120012 RSI: 0000000000000003 RDI: ffff880174d48170 [ 989.763778] RBP: ffff88017f7cbcc8 R08: ffff88021f4293b0 R09: 0000000000000000 [ 989.763991] R10: ffff880179c8c000 R11: 0000000000000003 R12: ffff880174d48008 [ 989.764204] R13: 0000000000000003 R14: ffff880179c8c000 R15: ffff88021db7a000 [ 989.764422] FS: 0000000000000000(0000) GS:ffff880247480000(0000) knlGS:ffff880247480000 [ 989.764685] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 989.764865] CR2: ffff8000007f6800 CR3: 0000000001ae0000 CR4: 0000000000042660 [ 989.765081] Stack: [ 989.765167] 0000000000000003 ffff880174d48040 ffff88017f7cbd18 ffffffffc07d455f [ 989.765442] ffff88017f7cbd88 ffffffff816fb639 ffff88017f7cbd38 ffff8800361b5600 [ 989.765717] ffff88021db7a000 ffff88021f429380 0000000000000003 ffffffffc0453020 [ 989.765991] Call Trace: [ 989.766093] [] user_bast+0x5f/0xf0 [ocfs2_dlmfs] [ 989.766287] [] ? schedule_timeout+0x169/0x2d0 [ 989.766475] [] ? o2dlm_lock_ast_wrapper+0x20/0x20 [ocfs2_stack_o2cb] [ 989.766738] [] o2dlm_blocking_ast_wrapper+0x1a/0x20 [ocfs2_stack_o2cb] [ 989.767010] [] dlm_do_local_bast+0x46/0xe0 [ocfs2_dlm] [ 989.767217] [] ? dlm_lockres_calc_usage+0x4c/0x60 [ocfs2_dlm] [ 989.767466] [] dlm_thread+0xa31/0x1140 [ocfs2_dlm] [ 989.767662] [] ? __schedule+0x24a/0x810 [ 989.767834] [] ? __schedule+0x23e/0x810 [ 989.768006] [] ? __schedule+0x24a/0x810 [ 989.768178] [] ? __schedule+0x23e/0x810 [ 989.768349] [] ? __schedule+0x24a/0x810 [ 989.768521] [] ? __schedule+0x23e/0x810 [ 989.768693] [] ? __schedule+0x24a/0x810 [ 989.768893] [] ? __schedule+0x23e/0x810 [ 989.769067] [] ? __schedule+0x24a/0x810 [ 989.769241] [] ? wait_woken+0x90/0x90 [ 989.769411] [] ? dlm_kick_thread+0x80/0x80 [ocfs2_dlm] [ 989.769617] [] kthread+0xcb/0xf0 [ 989.769774] [] ? __schedule+0x24a/0x810 [ 989.769945] [] ? __schedule+0x24a/0x810 [ 989.770117] [] ? kthread_create_on_node+0x180/0x180 [ 989.770321] [] ret_from_fork+0x61/0x90 [ 989.770492] [] ? kthread_create_on_node+0x180/0x180 [ 989.770689] Code: d0 00 00 00 f0 45 7d c0 bf 00 20 00 00 48 89 83 c0 00 00 00 48 89 83 c8 00 00 00 e8 55 c1 8c c0 83 4b 04 10 48 83 c4 08 5b 5d c3 <0f> 0b 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 41 54 53 48 83 [ 989.771892] RIP [] __user_dlm_queue_lockres.part.4+0x76/0x80 [ocfs2_dlmfs] [ 989.772174] RSP [ 989.772704] ---[ end trace ebd1e38cebcc93a8 ]--- [ 989.772907] Kernel panic - not syncing: Fatal exception [ 989.773173] Kernel Offset: disabled Cc: Signed-off-by: Junxiao Bi --- fs/ocfs2/dlmfs/userdlm.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/fs/ocfs2/dlmfs/userdlm.c b/fs/ocfs2/dlmfs/userdlm.c index af0be612589c..617c92e7b925 100644 --- a/fs/ocfs2/dlmfs/userdlm.c +++ b/fs/ocfs2/dlmfs/userdlm.c @@ -433,6 +433,11 @@ int user_dlm_cluster_lock(struct user_lock_res *lockres, } spin_lock(&lockres->l_lock); + if (lockres->l_flags & USER_LOCK_IN_TEARDOWN) { + spin_unlock(&lockres->l_lock); + status = -EAGAIN; + goto bail; + } /* We only compare against the currently granted level * here. If the lock is blocked waiting on a downconvert, @@ -595,7 +600,7 @@ int user_dlm_destroy_lock(struct user_lock_res *lockres) spin_lock(&lockres->l_lock); if (lockres->l_flags & USER_LOCK_IN_TEARDOWN) { spin_unlock(&lockres->l_lock); - return 0; + goto bail; } lockres->l_flags |= USER_LOCK_IN_TEARDOWN; @@ -609,12 +614,17 @@ int user_dlm_destroy_lock(struct user_lock_res *lockres) } if (lockres->l_ro_holders || lockres->l_ex_holders) { + lockres->l_flags &= ~USER_LOCK_IN_TEARDOWN; spin_unlock(&lockres->l_lock); goto bail; } status = 0; if (!(lockres->l_flags & USER_LOCK_ATTACHED)) { + /* + * lock is never requested, leave USER_LOCK_IN_TEARDOWN set + * to avoid new lock request coming in. + */ spin_unlock(&lockres->l_lock); goto bail; } @@ -624,6 +634,10 @@ int user_dlm_destroy_lock(struct user_lock_res *lockres) status = ocfs2_dlm_unlock(conn, &lockres->l_lksb, DLM_LKF_VALBLK); if (status) { + spin_lock(&lockres->l_lock); + lockres->l_flags &= ~USER_LOCK_IN_TEARDOWN; + lockres->l_flags &= ~USER_LOCK_BUSY; + spin_unlock(&lockres->l_lock); user_log_dlm_error("ocfs2_dlm_unlock", status, lockres); goto bail; }