From patchwork Mon Apr 6 21:53:58 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hugo Vincent X-Patchwork-Id: 16679 Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by demeter.kernel.org (8.14.2/8.14.2) with ESMTP id n36LvN1e015089 for ; Mon, 6 Apr 2009 21:57:23 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760334AbZDFVyK (ORCPT ); Mon, 6 Apr 2009 17:54:10 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1760345AbZDFVyJ (ORCPT ); Mon, 6 Apr 2009 17:54:09 -0400 Received: from mail-gx0-f160.google.com ([209.85.217.160]:40234 "EHLO mail-gx0-f160.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1760319AbZDFVyF (ORCPT ); Mon, 6 Apr 2009 17:54:05 -0400 Received: by gxk4 with SMTP id 4so5167886gxk.13 for ; Mon, 06 Apr 2009 14:54:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=HR2sc2YP4IvYlyRy2fje+vBNryryXZsAv/CwSlNyatA=; b=N0m3xFCBhpvGpauh+ohPxtjKAOofKzUvXVcoGxyQos7+X+ffCjNGmOJU3yXgBXoEaj ZqUrM5e6v/NO/s/FcZreek5zyNX3/u9vwnHDTMV6STb2fJDYzC54DeYjeAth/f1KUtjK ABlcwqWhG62zXBbxTXXxiMtQh+8vwN5wSh+dM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=plDdXa2x6n/VM0DWupMCj5VkWj8vwQNddigOzc68jTqBO/d1QkS4aXFGnDj08khg4r IHSeWut75ldpsDVElYBTL8RgW+eN2DounP9Fzmkfr21l2Sa+DScL5QTnS/GO+K4daeAW kV0RFaZgh7mWb1+rzOnJfU0KssLJiX7Otq0kE= MIME-Version: 1.0 Received: by 10.231.12.138 with SMTP id x10mr1047970ibx.16.1239054838586; Mon, 06 Apr 2009 14:53:58 -0700 (PDT) In-Reply-To: <200904060325.45758.david-b@pacbell.net> References: <5a7b8b7b0904052025g74325484h66335606bebea5a3@mail.gmail.com> <200904060325.45758.david-b@pacbell.net> Date: Tue, 7 Apr 2009 09:53:58 +1200 Message-ID: <5a7b8b7b0904061453m725eee67m4b74c73bbffeb994@mail.gmail.com> Subject: Re: Preempt-RT on OMAP3? From: Hugo Vincent To: David Brownell Cc: linux-omap@vger.kernel.org Sender: linux-omap-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-omap@vger.kernel.org Here are some of the crashes I've seen. Warning - I'm new to -rt and to the linux-omap tree, so I'll apologize in advance if these are just a result of me missing something obvious. Context: using USB-gadget ethernet (g_ether) over musb (configured as just peripheral, not OTG). Most of the time it works, but after a while, or when it encounters a large packet (e.g fping -b30000) it crashes like so (apparently something to do with receive DMA usage). Although the fping -b case is kindof pathological, I've seen the same crash when ssh/scping files across to the board, so it does occur in real usage. Unable to handle kernel NULL pointer dereference at virtual address 00000000 pgd = c0004000 [00000000] *pgd=00000000 Internal error: Oops: 817 [#1] PREEMPT Modules linked in: g_ether ftdi_sio usbserial ipv6 CPU: 0 Not tainted (2.6.29-rt1-omap1-g7648048 #1) PC is at dma_channel_program+0x90/0x108 LR is at rxstate+0xc8/0x1b4 pc : [] lr : [] psr: 60000013 sp : cf93be10 ip : cf93be58 fp : cf93be54 r10: 00000000 r9 : 00000000 r8 : 8fa1d002 r7 : 00000200 r6 : cf98b668 r5 : cf98b668 r4 : cf98b600 r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : cf98b668 Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel Control: 10c5387d Table: 8fa90019 DAC: 00000017 Process IRQ-93 (pid: 191, stack limit = 0xcf93a2e8) Stack: (0xcf93be10 to 0xcf93c000) be00: bf062078 c024b8f4 00000000 cf05f3e0 be20: cf85f8e0 cf85f8e0 00000000 cf98b600 cf85f5e0 cf83b000 00000000 00002003 be40: d80ab110 00000001 cf93be94 cf93be58 c024b760 c024c73c 00000000 c02f8d18 be60: cf83b000 60000113 cf93a000 00002003 cf85f5e0 cf83b000 00000003 cf98b668 be80: d80ab110 00000001 cf93bedc cf93be98 c024bfc0 c024b6a4 00000003 00000002 bea0: cf93a000 cf94537c cf83b19c cf83b1d8 c004d80c cf98b650 00000002 00000001 bec0: 8fa1dc02 d80ab000 cf83b000 cf98b600 cf93beec cf93bee0 c02489e4 c024bc9c bee0: cf93bf3c cf93bef0 c024c988 c024899c 00000002 cf820000 ffffffff c040caa4 bf00: af2001b4 cf94537c d80ab060 00000004 c006b43c c03ebbc8 cf93a000 cf8c09e0 bf20: 0000005d 00000000 0000005d 00000000 cf93bf74 cf93bf40 c007cef0 c024c844 bf40: cfa29e40 00000000 cf93bf84 c03ebbc8 cf93a000 0000005d 0000005d cf8c09e0 bf60: c03ebc20 c04186a4 cf93bf9c cf93bf78 c007d33c c007ce30 c03ebbc8 cf93a000 bf80: 0000005d 00000000 60000113 c03ebc08 cf93bfd4 cf93bfa0 c007d45c c007d2dc bfa0: 00000000 00000032 00000000 cf93a000 c03ebbc8 c007d390 00000000 00000000 bfc0: 00000000 00000000 cf93bff4 cf93bfd8 c0065b54 c007d39c 00000000 00000000 bfe0: 00000000 00000000 00000000 cf93bff8 c0053b28 c0065b04 017bee08 08f7ef39 Backtrace: [] (dma_channel_program+0x0/0x108) from [] (rxstate+0xc8/0x1b4) [] (rxstate+0x0/0x1b4) from [] (musb_g_rx+0x330/0x3ac) [] (musb_g_rx+0x0/0x3ac) from [] (musb_dma_completion+0x54/0x58) [] (musb_dma_completion+0x0/0x58) from [] (dma_controller_irq+0x150/0x18c) [] (dma_controller_irq+0x0/0x18c) from [] (handle_IRQ_event+0xcc/0x1d8) [] (handle_IRQ_event+0x0/0x1d8) from [] (thread_simple_irq+0x6c/0xc0) [] (thread_simple_irq+0x0/0xc0) from [] (do_irqd+0xcc/0x334) [] (do_irqd+0x0/0x334) from [] (kthread+0x5c/0x94) [] (kthread+0x0/0x94) from [] (do_exit+0x0/0x73c) r6:00000000 r5:00000000 r4:00000000 Code: 13a03000 03a03001 1a000002 e3a03000 (e5833000) ---[ end trace af756bf803843539 ]--- After the above crash has been seen, reboot/shutdown or rmmod do this: Unable to handle kernel NULL pointer dereference at virtual address 00000000 pgd = cf054000 [00000000] *pgd=8f322031, *pte=00000000, *ppte=00000000 Internal error: Oops: 17 [#2] PREEMPT Modules linked in: ftdi_sio usbserial ipv6 CPU: 0 Tainted: G D (2.6.29-rt1-omap1-g8dde6bd-dirty #1) PC is at plist_add+0x5c/0xb0 LR is at task_blocks_on_rt_mutex+0x144/0x204 pc : [] lr : [] psr: 60000093 sp : cfb6dd58 ip : cfb6dd70 fp : cfb6dd6c r10: cfb6ddbc r9 : 60000013 r8 : cfb6ddb0 r7 : cf1d0d60 r6 : cfb6ddb0 r5 : cf1d111c r4 : cfb6ddc4 r3 : fffffffc r2 : cfb6ddd0 r1 : cf1d111c r0 : fffffffc Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user Control: 10c5387d Table: 8f054019 DAC: 00000015 Process reboot (pid: 1335, stack limit = 0xcfb6c2e8) Stack: (0xcfb6dd58 to 0xcfb6e000) dd40: cfb6c000 cf1d111c dd60: cfb6ddac cfb6dd70 c0074690 c01bd048 00000001 c040e320 00000000 cf83b0d0 dd80: cfa144e0 cf83b0d0 60000013 cfb6c000 0000006c 00000000 00000000 00000000 dda0: cfb6de0c cfb6ddb0 c02f55ac c0074558 00000078 cf83b0d0 cf83b0d0 cf83b0d8 ddc0: cf83b0d8 00000078 cfb6ddc8 cfb6ddc8 cfb6ddd0 cfb6ddd0 cfaba620 cf83b0d0 dde0: 0000006c cf83b0d0 00000000 fee1dead 0000006c c002d0c4 cfb6c000 00000000 de00: cfb6de1c cfb6de10 c02f60f4 c02f54d4 cfb6de34 cfb6de20 c0241964 c02f60c0 de20: c03e1148 c041eaa8 cfb6de44 cfb6de38 c01ee7a0 c0241950 cfb6de5c cfb6de48 de40: c01e9f0c c01ee78c 00000000 28121969 cfb6de6c cfb6de60 c00604dc c01e9eb8 de60: cfb6de84 cfb6de70 c0060530 c00604bc 01234567 01234567 cfb6dfa4 cfb6de88 de80: c00606ec c0060524 cfb6de84 00000000 c00bfa7c cf525e30 cf525e30 c03f0c2c dea0: 00200200 00100100 cfb6decc cfb6deb8 c00bbb7c c00a7dc4 cfb6dee4 cf525e30 dec0: cfb6dee4 cfb6ded0 c00bbbcc c00bbb30 cf525e30 cf401548 cfb6defc cfb6dee8 dee0: c00bbc3c cf80e688 cf80e620 cf4078a0 cfb6df24 cfb6df00 c00c2f48 c0074a04 df00: c007ff60 cf1aea40 00000008 cf4078a0 cf525e30 cf80e620 cfb6df54 cfb6df28 df20: c00ad63c c00c2f28 00000000 c02f5420 cfb6df5c cf1aea40 00000000 cf881b00 df40: cf1aea40 c002d0c4 cfb6df64 cfb6df58 c00ad67c c00ad480 cfb6df84 cfb6df68 df60: c00aa328 c00ad658 cfb6df84 00000003 cf881b00 cf881b24 cfb6dfa4 cfb6df88 df80: c00aa3dc c00aa2b8 00000000 00000001 00000004 00000058 00000000 cfb6dfa8 dfa0: c002cf40 c0060588 00000000 00000001 fee1dead 28121969 01234567 0000006c dfc0: 00000000 00000001 00000004 00000058 00000001 00000001 00000000 00000001 dfe0: 400d8620 befd2cc0 00009210 400d8638 60000010 fee1dead cfb6dff4 00000000 Backtrace: [] (plist_add+0x0/0xb0) from [] (task_blocks_on_rt_mutex+0x144/0x204) r5:cf1d111c r4:cfb6c000 [] (task_blocks_on_rt_mutex+0x0/0x204) from [] (rt_spin_lock_slowlock+0xe4/0x27c) [] (rt_spin_lock_slowlock+0x0/0x27c) from [] (rt_spin_lock+0x40/0x44) [] (rt_spin_lock+0x0/0x44) from [] (musb_shutdown+0x20/0x68) [] (musb_shutdown+0x0/0x68) from [] (platform_drv_shutdown+0x20/0x24) r5:c041eaa8 r4:c03e1148 [] (platform_drv_shutdown+0x0/0x24) from [] (device_shutdown+0x60/0xb8) [] (device_shutdown+0x0/0xb8) from [] (kernel_restart_prepare+0x2c/0x3c) r5:28121969 r4:00000000 [] (kernel_restart_prepare+0x0/0x3c) from [] (kernel_restart+0x18/0x4c) [] (kernel_restart+0x0/0x4c) from [] (sys_reboot+0x170/0x1dc) r4:01234567 [] (sys_reboot+0x0/0x1dc) from [] (ret_fast_syscall+0x0/0x2c) r7:00000058 r6:00000004 r5:00000001 r4:00000000 Code: ba000007 e1a00001 0a00000c e1a03000 (e5b32004) ---[ end trace d7d34d233af69b06 ]--- note: reboot[1335] exited with preempt_count 2 Segmentation fault Finally, there is this crash (when musb is configured as OTG). When you try to write to several of the sysfs entries, e.g. /sys/devices/platform/musb_hdrc/mode, the driver crashes. Note that this one doesn't appear to be limited to just -rt. root@overo-grc:/sys/devices/platform/musb_hdrc# echo host > mode Unable to handle kernel NULL pointer dereference at virtual address 00000000 pgd = cfbe4000 [00000000] *pgd=8fbaf031, *pte=00000000, *ppte=00000000 Internal error: Oops: 0 [#1] PREEMPT Modules linked in: ftdi_sio usbserial ipv6 CPU: 0 Not tainted (2.6.29-rt1-omap1-g8dde6bd-dirty #1) PC is at 0x0 LR is at musb_platform_set_mode+0x50/0x70 pc : [<00000000>] lr : [] psr: 60000013 sp : cf0e3ed8 ip : cf83b0d0 fp : cf0e3ee4 r10: cf0e3f70 r9 : cfbf9238 r8 : c03ff1e8 r7 : c03e15ac r6 : 00000005 r5 : cf83b0d0 r4 : cfaa7000 r3 : 00000081 r2 : d80ab000 r1 : cf83b000 r0 : cf83b1c4 Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user Control: 10c5387d Table: 8fbe4019 DAC: 00000015 Process sh (pid: 1233, stack limit = 0xcf0e22e8) Stack: (0xcf0e3ed8 to 0xcf0e4000) 3ec0: cf0e3f04 cf0e3ee8 3ee0: c0241ce0 c0243020 cf07bc00 00000005 cf92ef08 cfbf9220 cf0e3f14 cf0e3f08 3f00: c01e9dfc c0241c68 cf0e3f44 cf0e3f18 c00ef7bc c01e9de4 00000000 cf07bc00 3f20: 4001e000 cf0e3f70 00000005 00000005 cf0e2000 00000001 cf0e3f6c cf0e3f48 3f40: c00aca90 c00ef6b8 cf07bc00 00000000 00000000 00000000 cf07bc00 4001e000 3f60: cf0e3fa4 cf0e3f70 c00acbe4 c00ac9e8 00000000 00000000 cf0e3fa4 00000000 3f80: c00b8fe8 00000005 4001e000 401ac600 00000004 c002d0c4 00000000 cf0e3fa8 3fa0: c002cf40 c00acbac 00000005 4001e000 00000001 4001e000 00000005 00000000 3fc0: 00000005 4001e000 401ac600 00000004 00000005 000a93f0 00000001 000a9008 3fe0: 00000000 bea345e8 400f578c 4014299c 60000010 00000001 00000000 00000000 Backtrace: [] (musb_platform_set_mode+0x0/0x70) from [] (musb_mode_store+0x84/0xac) [] (musb_mode_store+0x0/0xac) from [] (dev_attr_store+0x24/0x28) r6:cfbf9220 r5:cf92ef08 r4:00000005 [] (dev_attr_store+0x0/0x28) from [] (sysfs_write_file+0x110/0x144) [] (sysfs_write_file+0x0/0x144) from [] (vfs_write+0xb4/0x144) [] (vfs_write+0x0/0x144) from [] (sys_write+0x44/0x70) r7:4001e000 r6:cf07bc00 r5:00000000 r4:00000000 [] (sys_write+0x0/0x70) from [] (ret_fast_syscall+0x0/0x2c) r8:c002d0c4 r7:00000004 r6:401ac600 r5:4001e000 r4:00000005 Code: bad PC value. ---[ end trace d7d34d233af69b05 ]--- I'm building git 90e758af52ba803cba233fabee81176d99589f09 (2.6.29 final) using openembedded linux-omap recipe. My config is attached. I have applied the -rt3 (or -rt2 or -rt1 - same errors) patch from http://www.kernel.org/pub/linux/kernel/projects/rt/. I can't get the 2.6.29.1-rtX patch to apply cleanly because I can't figure out which git revision corresponds to 2.6.29.1.... There is one other oops I see during 8250/16550 serial port initialization with -rt, but that was easily fixed (see attached patch). As I said, I'm new to RT so I've got no idea if the way I did it in the patch is valid, but it seems to work, and seeing as it's only during initialization, it seems fairly safe. Serial: 8250/16550 driver, 8 ports, IRQ sharing enabled irq 72: nobody cared (try booting with the "irqpoll" option) [] (dump_stack+0x0/0x14) from [] (__report_bad_irq+0x3c/0x98) [] (__report_bad_irq+0x0/0x98) from [] (note_interrupt+0x150/0x1d8) r4:c03eb2ec [] (note_interrupt+0x0/0x1d8) from [] (thread_simple_irq+0xa0/0xc0) [] (thread_simple_irq+0x0/0xc0) from [] (do_irqd+0xcc/0x334) [] (do_irqd+0x0/0x334) from [] (kthread+0x5c/0x94) [] (kthread+0x0/0x94) from [] (do_exit+0x0/0x73c) r6:00000000 r5:00000000 r4:00000000 handlers: [] (omap_uart_interrupt+0x0/0x1c) Disabling IRQ #72 serial8250.0: ttyS0 at MMIO 0x4806a000 (irq = 72) is a ST16654 irq 73: nobody cared (try booting with the "irqpoll" option) [] (dump_stack+0x0/0x14) from [] (__report_bad_irq+0x3c/0x98) [] (__report_bad_irq+0x0/0x98) from [] (note_interrupt+0x150/0x1d8) r4:c03eb358 [] (note_interrupt+0x0/0x1d8) from [] (thread_simple_irq+0xa0/0xc0) [] (thread_simple_irq+0x0/0xc0) from [] (do_irqd+0xcc/0x334) [] (do_irqd+0x0/0x334) from [] (kthread+0x5c/0x94) [] (kthread+0x0/0x94) from [] (do_exit+0x0/0x73c) r6:00000000 r5:00000000 r4:00000000 handlers: [] (omap_uart_interrupt+0x0/0x1c) Disabling IRQ #73 serial8250.0: ttyS1 at MMIO 0x4806c000 (irq = 73) is a ST16654 serial8250.0: ttyS2 at MMIO 0x49020000 (irq = 74) is a ST16654 console [ttyS2] enabled Thanks in advance, Hugo Vincent On Mon, Apr 6, 2009 at 10:25 PM, David Brownell wrote: > On Sunday 05 April 2009, Hugo Vincent wrote: >> I'm trying to get the realtime patch set to work on OMAP3 (Gumstix >> Overo). Linux-omap 2.6.29 with the RT patches -rt1 or -rt2 work mostly >> as is (MUSB and the usb-gadget layer have quite a few problems, and > > Like what?  Way back in 2.6.10 MUSB behaved relatively sanely. > But that's a while back.  ;) > > >> the realtime self-test stuff picks up a few problems to do with >> spinlock usage etc, but most things seem to mostly work correctly for >> my needs). Running cyclictest with various types of loads gives an >> average latency of ~30-50 usec and worst case up around 300 usec. > > > diff --git a/drivers/serial/8250.c b/drivers/serial/8250.c index 0d4f4c6..122b694 100644 --- a/drivers/serial/8250.c +++ b/drivers/serial/8250.c @@ -1073,6 +1073,7 @@ static void autoconfig(struct uart_8250_port *up, unsigned int probeflags) unsigned char status1, scratch, scratch2, scratch3; unsigned char save_lcr, save_mcr; unsigned long flags; + DEFINE_RAW_SPINLOCK(raw_lock); if (!up->port.iobase && !up->port.mapbase && !up->port.membase) return; @@ -1085,6 +1086,7 @@ static void autoconfig(struct uart_8250_port *up, unsigned int probeflags) * be frobbing the chips IRQ enable register to see if it exists. */ spin_lock_irqsave(&up->port.lock, flags); + spin_lock_irqsave(&raw_lock, flags); up->capabilities = 0; up->bugs = 0; @@ -1240,6 +1242,7 @@ static void autoconfig(struct uart_8250_port *up, unsigned int probeflags) serial_outp(up, UART_IER, 0); out: + spin_unlock_irqrestore(&raw_lock, flags); spin_unlock_irqrestore(&up->port.lock, flags); DEBUG_AUTOCONF("type=%s\n", uart_config[up->port.type].name); }