From patchwork Sun Aug 11 09:48:49 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Toralf_F=C3=B6rster?= X-Patchwork-Id: 2842697 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 652669F294 for ; Sun, 11 Aug 2013 09:49:39 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5DCB020211 for ; Sun, 11 Aug 2013 09:49:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 27A552020E for ; Sun, 11 Aug 2013 09:49:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752801Ab3HKJsz (ORCPT ); Sun, 11 Aug 2013 05:48:55 -0400 Received: from mout-xforward.gmx.net ([82.165.159.42]:55506 "EHLO mout-xforward.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752788Ab3HKJsx (ORCPT ); Sun, 11 Aug 2013 05:48:53 -0400 Received: from [92.224.126.156] ([92.224.126.156]) by mail.gmx.com (mrgmx001) with ESMTPSA (Nemesis) id 0M3iU5-1Vz2zD3RE9-00rHOc for ; Sun, 11 Aug 2013 11:48:51 +0200 Message-ID: <52075E01.7030506@gmx.de> Date: Sun, 11 Aug 2013 11:48:49 +0200 From: =?UTF-8?B?VG9yYWxmIEbDtnJzdGVy?= User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130807 Thunderbird/17.0.8 MIME-Version: 1.0 To: Linux NFS mailing list CC: "user-mode-linux-devel@lists.sourceforge.net" , linux-ext4@vger.kernel.org, Linux Kernel Subject: Issues with a rather unusual configured NFS server X-Enigmail-Version: 1.5.2 X-Provags-ID: V03:K0:Yp/PR2biHuwsQGvzLkNFHjNPxDgYclzGjVGOGHhUCFwNWsZoOA0 bj5outGnKjJlBtyvet3spKi0PbAFa2YB2fE3t+YciSqA7kIRc1bXgmeYsvUx9tTGL1Y0F4I tlmPFF1io4EYjUAxRwtobJKRlDA4LXpRE8cjFqJ8nr5P0kvdFIpv+ocmbC7+/jDkvJ0uFnq aG19S4EVMTaGlZLXXlO3Q== Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP so that the server either crashes (if it is a user mode linux image) or at least its reboot functionality got broken - if the NFS server is hammered with scary NFS calls using a fuzzy tool running at a remote NFS client under a non-privileged user id. It can re reproduced, if - the NFS share is an EXT3 or EXT4 directory - and it is created at file located at tempfs and mounted via loop device - and the NFS server is forced to umount the NFS share - and the server forced to restart the NSF service afterwards - and trinity is used I could find a scenario for an automated bisect. 2 times it brought this commit commit 68a3396178e6688ad7367202cdf0af8ed03c8727 Author: J. Bruce Fields Date: Thu Mar 21 11:21:50 2013 -0400 nfsd4: shut down more of delegation earlier to be the one after which the user mode linux server crashes with a back trace like this: $ cat /mnt/ramdisk/bt.v3.11-rc4-172-g8ae3f1d [New LWP 14025] Core was generated by `/home/tfoerste/devel/linux/linux earlyprintk ubda=/home/tfoerste/virtual/uml/tr'. Program terminated with signal 6, Aborted. #0 0xb77ef424 in __kernel_vsyscall () #0 0xb77ef424 in __kernel_vsyscall () #1 0x083a33c5 in kill () #2 0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93 #3 0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138 #4 0x080613a7 in panic_exit (self=0x85a1518 , unused1=0, unused2=0x85d6ce0 ) at arch/um/kernel/um_arch.c:240 #5 0x0809a3b8 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 , nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93 #6 0x0809a503 in __atomic_notifier_call_chain (nr_calls=, nr_to_call=, v=, val=, nh=) at kernel/notifier.c:182 #7 atomic_notifier_call_chain (nh=0x85d6cc4 , val=0, v=0x85d6ce0 ) at kernel/notifier.c:191 #8 0x08400ba8 in panic (fmt=0x0) at kernel/panic.c:128 #9 0x0818edf4 in ext4_put_super (sb=0x4a042690) at fs/ext4/super.c:818 #10 0x081010d2 in generic_shutdown_super (sb=0x4a042690) at fs/super.c:418 #11 0x0810209a in kill_block_super (sb=0x0) at fs/super.c:1028 #12 0x08100f6a in deactivate_locked_super (s=0x4a042690) at fs/super.c:299 #13 0x08101001 in deactivate_super (s=0x4a042690) at fs/super.c:324 #14 0x08118e0c in mntfree (mnt=) at fs/namespace.c:891 #15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929 #16 0x0811a2f5 in SYSC_umount (flags=, name=) at fs/namespace.c:1335 #17 SyS_umount (name=134541632, flags=0) at fs/namespace.c:1305 #18 0x0811a369 in SYSC_oldumount (name=) at fs/namespace.c:1347 #19 SyS_oldumount (name=134541632) at fs/namespace.c:1345 #20 0x080618e2 in handle_syscall (r=0x49e919d4) at arch/um/kernel/skas/syscall.c:35 #21 0x08073c0d in handle_trap (local_using_sysemu=, regs=, pid=) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x49e919d4) at arch/um/os-Linux/skas/process.c:431 #23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160 #24 0x00000000 in ?? () A real system however would not crash bug would give a kernel BUG as reported here: http://article.gmane.org/gmane.comp.file-systems.ext4/38915 Furthermore the server won't be able any longer to reboot - it would hang infinitely in the reboot phase. Just the magic sysrq keys still works then. Steps to reproduce at two 32 bit Gentoo Linux user mode linux images: 1. prepare the server : mkdir /mnt/ramdisk/victims dd if=/dev/zero of=/mnt/ramdisk/disk1 bs=1M count=257 2>/dev/null yes | mkfs.ext4 -q /mnt/ramdisk/disk1 1>/dev/null mount -o loop /mnt/ramdisk/disk1 /mnt/ramdisk/victims chmod 777 /mnt/ramdisk/victims /etc/init.d/nfs restart 2. prepare the client mount the NFS share onto the local mount point /mnt/ramdisk/victims/ with NFSv4 3. run the fuzzy tool trinity at the client: while [[ : ]]; do <(re-)create and fill /mnt/ramdisk/victims/v1/v2 with 100 empty files and 100 empty directories> trinity -V /mnt/ramdisk/victims/v1/v2 -C 1 -N 10000 -q sleep 3 done 4. after 15 min kill the user mode linux client with -9 5. now run at the server umount /mnt/ramdisk/victims || /etc/init.d/nfs restart && umount /mnt/ramdisk/victims && echo ' no issue so far' You might need this patch too from Oleg Nesterov (not in mainline currently) . --- x/kernel/exit.c +++ x/kernel/exit.c @@ -783,8 +783,8 @@ void do_exit(long code) exit_shm(tsk); exit_files(tsk); exit_fs(tsk); - exit_task_namespaces(tsk); exit_task_work(tsk); + exit_task_namespaces(tsk); check_stack_usage(); exit_thread();