From patchwork Tue Apr 15 12:35:33 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guozhonghua X-Patchwork-Id: 3992561 Return-Path: X-Original-To: patchwork-ocfs2-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 8C7F29F2CC for ; Tue, 15 Apr 2014 12:37:35 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C49B920219 for ; Tue, 15 Apr 2014 12:37:31 +0000 (UTC) Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E35B520155 for ; Tue, 15 Apr 2014 12:37:26 +0000 (UTC) Received: from acsinet21.oracle.com (acsinet21.oracle.com [141.146.126.237]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s3FCaJfE003898 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 15 Apr 2014 12:36:20 GMT Received: from oss.oracle.com (oss-external.oracle.com [137.254.96.51]) by acsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s3FCaCTN024581 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 15 Apr 2014 12:36:13 GMT Received: from localhost ([127.0.0.1] helo=oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1Wa2ay-0005He-G0; Tue, 15 Apr 2014 05:36:12 -0700 Received: from ucsinet22.oracle.com ([156.151.31.94]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1Wa2aX-0005H9-J2; Tue, 15 Apr 2014 05:35:45 -0700 Received: from aserp1020.oracle.com (aserp1020.oracle.com [141.146.126.67]) by ucsinet22.oracle.com (8.14.5+Sun/8.14.5) with ESMTP id s3FCZin3021400 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 15 Apr 2014 12:35:45 GMT Received: from h3cedge03-ex.h3c.com (smtp.h3c.com [221.12.31.13]) by aserp1020.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s3FCZcOa003442; Tue, 15 Apr 2014 12:35:39 GMT Received: from H3CHUB01-EX.srv.huawei-3com.com (10.63.16.180) by h3cedge03-ex.h3c.com (172.25.12.83) with Microsoft SMTP Server (TLS) id 14.1.355.2; Tue, 15 Apr 2014 20:35:33 +0800 Received: from H3CHUB03-EX.srv.huawei-3com.com (10.63.20.169) by H3CHUB01-EX.srv.huawei-3com.com (10.63.16.181) with Microsoft SMTP Server (TLS) id 14.1.355.2; Tue, 15 Apr 2014 20:35:35 +0800 Received: from H3CMLB02-EX.srv.huawei-3com.com ([fe80::399d:90f6:c5bb:9cc0]) by H3CHUB03-EX.srv.huawei-3com.com ([fe80::649b:7e79:362f:5d39%15]) with mapi id 14.01.0355.002; Tue, 15 Apr 2014 20:35:34 +0800 From: Guozhonghua To: "ocfs2-devel@oss.oracle.com" , "ocfs2-users@oss.oracle.com" Thread-Topic: OCFS2 issue patch requesting reviews, is there anyone with some good advices, thanks. Thread-Index: Ac9YpzkwDtmGWoEyQ5eSOsTFtasHMQ== Date: Tue, 15 Apr 2014 12:35:33 +0000 Message-ID: <71604351584F6A4EBAE558C676F37CA42EC3FBD5@H3CMLB02-EX.srv.huawei-3com.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.153.28.122] MIME-Version: 1.0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1404150209 X-Proofpoint-ZeroHour: threat=unknown, refid=1B2M2Y8AsgTpgAmY7PhCfg X-Proofpoint-Reputation: rule=pass Subject: [Ocfs2-devel] OCFS2 issue patch requesting reviews, is there anyone with some good advices, thanks. X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: acsinet21.oracle.com [141.146.126.237] X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,HTML_MESSAGE, MIME_HTML_MOSTLY, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi, everyone: As the disk is umounted, the host is panic or bloked. The host must be repowered. The test scenario is with Linux kernel 3.13.6. I reviewed the code and cannot find where is the lock's lvb changed which make the lvb difference. So many code with lvb which is changed. I changed some code to avoid BUG() which can cause the host panic or blocked. The patch and log is as below, and I would like to receive some good ideas about it. I think it may be a bug of ocfs2 kernel code, so there is another good way to fix it. Thanks a lot. root@gzh-139:/vms/linux_kernel# diff -u -p linux-3.13.6/fs/ocfs2/dlm/dlmrecovery.c linux-3.13.6.changed/ocfs2-ko-3.13/dlm/dlmrecovery.c /* returns 1 if this lock fills the network structure, @@ -1215,6 +1217,13 @@ static int dlm_add_lock_to_array(struct struct dlm_migratable_lock *ml; int lock_num = mres->num_locks; + if (lock->lksb) { + /* if failed, return 1 and send the lock message immeditely */ + if (dlm_prepare_lvb_for_migration(lock, mres, queue)) { + return 1; + } + } + ml = &(mres->ml[lock_num]); ml->cookie = lock->ml.cookie; ml->type = lock->ml.type; @@ -1223,7 +1232,6 @@ static int dlm_add_lock_to_array(struct ml->list = queue; if (lock->lksb) { ml->flags = lock->lksb->flags; - dlm_prepare_lvb_for_migration(lock, mres, queue); } ml->node = lock->ml.node; mres->num_locks++; Apr 12 20:55:01 ZJ-HZDX-0321-D20-CVK-03 kernel: [870221.355731] sd 7:0:0:0: [sdl] Very big device. Trying to use READ CAPACITY(16). Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782364] (umount,44814,19):dlm_prepare_lvb_for_migration:1205 ERROR: Mismatched lvb in lock cookie=2:519367, name=M00000000000000000002094cc0d288, node=2 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782383] lockres: M00000000000000000002094cc0d288, owner=3, state=32 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782386] last used: 0, refcnt: 4, on purge list: no Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782389] on dirty list: no, on reco list: no, migrating pending: no Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782392] inflight locks: 0, asts reserved: 0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782394] refmap nodes: [ 1 2 ], inflight=0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782400] granted queue: Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782405] type=3, conv=-1, node=1, cookie=1:7, ref=2, ast=(empty=y,pend=n), bast=(empty=y,pend=n), pending=(conv=n,lock=n,cancel=n,unlock=n) Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782410] type=3, conv=-1, node=2, cookie=2:519367, ref=2, ast=(empty=y,pend=n), bast=(empty=y,pend=n), pending=(conv=n,lock=n,cancel=n,unlock=n) Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782412] converting queue: Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782414] blocked queue: Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782457] ------------[ cut here ]------------ Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782462] Kernel BUG at ffffffffa02f8d4f [verbose debug info unavailable] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782469] invalid opcode: 0000 [#1] SMP Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782476] Modules linked in: ext2(F) ocfs2(OF) quota_tree(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) 8021q(F) mrp(F) garp(F) stp(F) llc(F) vhost_net(F) macvtap(F) macvlan(F) vhost(F) kvm_intel(F) kvm(F) ib_iser(F) rdma_cm(F) ib_cm(F) iw_cm(F) ib_sa(F) ib_mad(F) ib_core(F) ib_addr(F) iscsi_tcp(F) libiscsi_tcp(F) libiscsi(F) scsi_transport_iscsi(F) ocfs2_dlmfs(OF) ocfs2_stack_o2cb(OF) ocfs2_dlm(OF) ocfs2_nodemanager(OF) ocfs2_stackglue(OF) configfs(F) openvswitch(OF) gre(F) nfsd(F) nfs_acl(F) auth_rpcgss(F) nfs(F) fscache(F) lockd(F) sunrpc(F) psmouse(F) dm_multipath(F) sb_edac(F) ipmi_si(F) edac_core(F) serio_raw(F) ioatdma(F) hpilo(F) gpio_ich(F) scsi_dh(F) hpwdt(F) mac_hid(F) dca(F) acpi_power_meter(F) lpc_ich(F) lp(F) parport(F) tg3(F) ptp(F) hpsa(F) pps_core(F) bnx2x(F) libcrc32c(F) mdio(F) nbd(F) Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782597] CPU: 19 PID: 44814 Comm: umount Tainted: GF O 3.13.6 #1 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782603] Hardware name: H3C FlexServer R390, BIOS P70 09/18/2013 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782609] task: ffff881772fae000 ti: ffff881385d8a000 task.ti: ffff881385d8a000 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782616] RIP: 0010:[] [] dlm_add_lock_to_array+0x1cf/0x1e0 [ocfs2_dlm] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782637] RSP: 0018:ffff881385d8b9d8 EFLAGS: 00010246 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782643] RAX: 0000000000000000 RBX: ffff880049d33600 RCX: 0000000000000006 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782650] RDX: 0000000000000007 RSI: 0000000002680266 RDI: ffff8817fbf57170 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782656] RBP: ffff881385d8ba28 R08: 000000000000000a R09: 0000000000000000 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782662] R10: 0000000000047a48 R11: 0000000000047a47 R12: ffff8811b3d5b000 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782669] R13: ffff8811b3d5b080 R14: ffff8817fbf570e8 R15: 0000000000000000 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782676] FS: 00007fbdb8d5e800(0000) GS:ffff88183f8e0000(0000) knlGS:0000000000000000 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782689] CR2: 00007fbdb8378120 CR3: 00000015de25f000 CR4: 00000000000407e0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782695] Stack: Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782699] ffff881300000002 000000000007ecc7 ffff88170000001f ffff8817faf6a9e0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782712] 0000000000000002 0000000000000002 0000000000000000 ffff880049d33600 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782723] 0000000000000002 ffff8811b3d5b000 ffff881385d8bae8 ffffffffa02fd5eb Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782734] Call Trace: Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782750] [] dlm_send_one_lockres+0x19b/0x4f0 [ocfs2_dlm] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782765] [] ? flush_workqueue+0x1c9/0x610 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782780] [] dlm_empty_lockres+0x4cb/0x1140 [ocfs2_dlm] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782795] [] ? autoremove_wake_function+0x16/0x40 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782804] [] ? __wake_up_common+0x58/0x90 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782817] [] dlm_unregister_domain+0x270/0x890 [ocfs2_dlm] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782829] [] ? check_preempt_curr+0x75/0xa0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782840] [] ? o2cb_cluster_disconnect+0x3c/0x60 [ocfs2_stack_o2cb] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782855] [] ? kfree+0x134/0x170 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782864] [] o2cb_cluster_disconnect+0x44/0x60 [ocfs2_stack_o2cb] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782878] [] ocfs2_cluster_disconnect+0x2e/0x68 [ocfs2_stackglue] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782916] [] ocfs2_dlm_shutdown+0xb7/0x100 [ocfs2] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782952] [] ocfs2_dismount_volume+0x202/0x3f0 [ocfs2] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782965] [] ? filemap_fdatawait+0x2b/0x30 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.782974] [] ? filemap_write_and_wait+0x34/0x60 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783004] [] ocfs2_put_super+0x37/0x90 [ocfs2] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783017] [] generic_shutdown_super+0x7e/0x110 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783025] [] kill_block_super+0x30/0x80 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783053] [] ocfs2_kill_sb+0x83/0xa0 [ocfs2] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783062] [] deactivate_locked_super+0x4d/0x80 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783070] [] deactivate_super+0x4e/0x70 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783082] [] mntput_no_expire+0xc8/0x150 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783092] [] SyS_umount+0xaf/0x3b0 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783106] [] tracesys+0xe1/0xe6 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783111] Code: 48 81 c6 c0 04 00 00 41 b9 b5 04 00 00 49 c7 c0 20 51 31 a0 48 c7 c7 60 7c 31 a0 31 c0 e8 c0 2a 45 e1 48 8b 7b 40 e8 71 d5 ff ff <0f> 0b 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783179] RIP [] dlm_add_lock_to_array+0x1cf/0x1e0 [ocfs2_dlm] Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.783192] RSP Apr 12 20:55:08 ZJ-HZDX-0321-D20-CVK-03 kernel: [870227.844940] ---[ end trace ccf348a85391d27e ]--- Apr 12 20:55:28 ZJ-HZDX-0321-D20-CVK-03 kernel: [870247.737249] o2dlm: Leaving domain 1220B17D51D141C784B30E8FE4C7E19C Apr 12 20:55:30 ZJ-HZDX-0321-D20-CVK-03 kernel: [870249.949859] ocfs2: Unmounting device (8,176) on (node 3) Apr 12 20:55:30 ZJ-HZDX-0321-D20-CVK-03 multipathd: sdl: remove path (uevent) ------------------------------------------------------------------------------------------------------------------------------------- ???????????????????????????????????????? ???????????????????????????????????????? ???????????????????????????????????????? ??? This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! --- linux-3.13.6/fs/ocfs2/dlm/dlmrecovery.c 2014-03-07 14:07:02.000000000 +0800 +++ linux-3.13.6.changed/ocfs2-ko-3.13/dlm/dlmrecovery.c 2014-04-15 20:10:34.024541267 +0800 @@ -1173,29 +1173,29 @@ static void dlm_init_migratable_lockres( mres->master = master; } -static void dlm_prepare_lvb_for_migration(struct dlm_lock *lock, +static int dlm_prepare_lvb_for_migration(struct dlm_lock *lock, struct dlm_migratable_lockres *mres, int queue) { if (!lock->lksb) - return; + return 0; /* Ignore lvb in all locks in the blocked list */ if (queue == DLM_BLOCKED_LIST) - return; + return 0; /* Only consider lvbs in locks with granted EX or PR lock levels */ if (lock->ml.type != LKM_EXMODE && lock->ml.type != LKM_PRMODE) - return; + return 0; if (dlm_lvb_is_empty(mres->lvb)) { memcpy(mres->lvb, lock->lksb->lvb, DLM_LVB_LEN); - return; + return 0; } /* Ensure the lvb copied for migration matches in other valid locks */ if (!memcmp(mres->lvb, lock->lksb->lvb, DLM_LVB_LEN)) - return; + return 0; mlog(ML_ERROR, "Mismatched lvb in lock cookie=%u:%llu, name=%.*s, " "node=%u\n", @@ -1204,7 +1204,9 @@ static void dlm_prepare_lvb_for_migratio lock->lockres->lockname.len, lock->lockres->lockname.name, lock->ml.node); dlm_print_one_lock_resource(lock->lockres); - BUG(); + /* BUG();*/ + + return 1; }