| Message ID | 1358829606.3464.3151.camel@edumazet-glaptop (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 01/21/2013 08:40 PM, Eric Dumazet wrote: > On Mon, 2013-01-21 at 16:32 -0800, Ben Greear wrote: >> On 01/21/2013 01:07 PM, Ben Greear wrote: >>> I posted about this a few days ago, but this time the patches applied >>> are minimal and there are no out-of-tree kernel modules loaded. >> >> Here's another crash, this time with SLUB memory debugging turned on. >> >> Seems much harder to hit this way...I've only managed this one. I >> believe the RCX register might be interesting...that 6b is probably >> freed memory poisioning. Maybe skb or skb_dest() is already freed? >> >> I have added a 'verify_mem_not_deleted(skb)' before the >> dst_input line in ip_rcv_finish..but so far, it hasn't >> hit the problem... >> > > There is no way skb is freed here. > > I would say macvlan is at fault here. > > It probably lacks a proper (dev->flags & IFF_UP) test. Thanks for the patch. It's running OK so far, but I'll need to do a bunch more testing tomorrow to make sure I'm not just getting lucky! I'll let you know how it goes tomorrow. Thanks, Ben
On 01/21/2013 09:57 PM, Ben Greear wrote: > On 01/21/2013 08:40 PM, Eric Dumazet wrote: >> On Mon, 2013-01-21 at 16:32 -0800, Ben Greear wrote: >>> On 01/21/2013 01:07 PM, Ben Greear wrote: >>>> I posted about this a few days ago, but this time the patches applied >>>> are minimal and there are no out-of-tree kernel modules loaded. >>> >>> Here's another crash, this time with SLUB memory debugging turned on. >>> >>> Seems much harder to hit this way...I've only managed this one. I >>> believe the RCX register might be interesting...that 6b is probably >>> freed memory poisioning. Maybe skb or skb_dest() is already freed? >>> >>> I have added a 'verify_mem_not_deleted(skb)' before the >>> dst_input line in ip_rcv_finish..but so far, it hasn't >>> hit the problem... >>> >> >> There is no way skb is freed here. >> >> I would say macvlan is at fault here. >> >> It probably lacks a proper (dev->flags & IFF_UP) test. > > Thanks for the patch. It's running OK so far, but I'll need to > do a bunch more testing tomorrow to make sure I'm not just getting > lucky! > > I'll let you know how it goes tomorrow. Unfortunately, I hit it again this morning after the first restart of my application (which bounces all 3000 interfaces). Memory poisoning was disabled. (gdb) l *(ip_rcv_finish+0x2b9) 0xffffffff814a8ab3 is in ip_rcv_finish (/home/greearb/git/linux-3.7.dev.y/net/ipv4/ip_input.c:373). 368 skb->len); 369 } else if (rt->rt_type == RTN_BROADCAST) 370 IP_UPD_PO_STATS_BH(dev_net(rt->dst.dev), IPSTATS_MIB_INBCAST, 371 skb->len); 372 373 return dst_input(skb); 374 375 drop: 376 kfree_skb(skb); 377 return NET_RX_DROP; (gdb) (gdb) l *(skb_dst+0x5a) 0xffffffff814a87fa is in ip_rcv_finish (/home/greearb/git/linux-3.7.dev.y/net/ipv4/ip_input.c:320). 315 316 int sysctl_ip_early_demux __read_mostly = 1; 317 EXPORT_SYMBOL(sysctl_ip_early_demux); 318 319 static int ip_rcv_finish(struct sk_buff *skb) 320 { 321 const struct iphdr *iph = ip_hdr(skb); 322 struct rtable *rt; 323 324 if (sysctl_ip_early_demux && !skb_dst(skb)) { nfs: server 10.1.0.1 not responding, timed out nfs: server 10.1.0.1 not responding, timed out BUG: unable to handle kernel NULL pointer dereference at (null) IP: [< (null)>] (null) PGD 0 Oops: 0010 [#1] PREEMPT SMP Modules linked in: nf_nat_ipv4 nf_nat nfsv4 auth_rpcgss nfs fscache 8021q garp stp llc lockd sunrpc macvlan pktgen uinput coretemp hwmon kvm_intel kvm iTCO_wdt iTCO_vendor_support gpio_ich microcode pcspkr i2c_i801 lpc_ich e1000e i7core_edac ioatdma edac_core igb ptp pps_core dca ipv6 mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core [last unloaded: iptable_nat] CPU 0 Pid: 9, comm: rcuc/0 Tainted: G C O 3.7.3+ #43 Iron Systems Inc. EE2610R/X8ST3 RIP: 0010:[<0000000000000000>] [< (null)>] (null) RSP: 0018:ffff88041fc03da0 EFLAGS: 00010286 RAX: ffff88023045f5c0 RBX: ffff88034491ab00 RCX: 00000000ffff8800 RDX: ffff88025920d4fc RSI: ffffffff81a2a500 RDI: ffff88034491ab00 RBP: ffff88041fc03dc8 R08: ffffffff814a87fa R09: ffff88041fc03d90 R10: ffff88025920d4fc R11: ffff88041fc03e28 R12: ffff88025920d4fc R13: ffff88041fc18b60 R14: ffff88040d3f8000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000001a0b000 CR4: 00000000000007f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process rcuc/0 (pid: 9, threadinfo ffff88040d4e2000, task ffff88040d4e8000) Stack: ffffffff814a8ab3 ffff88034491ab00 ffffffff814a87fa ffff88034491ab00 ffff88040d3f8000 ffff88041fc03df8 ffffffff814a8e66 0000000080000000 ffffffff81472e61 ffff88034491ab00 ffff88040d3f8000 ffff88041fc03e28 Call Trace: <IRQ> [<ffffffff814a8ab3>] ? ip_rcv_finish+0x2b9/0x2d1 [<ffffffff814a87fa>] ? skb_dst+0x5a/0x5a [<ffffffff814a8e66>] NF_HOOK.clone.1+0x4c/0x54 [<ffffffff81472e61>] ? dev_seq_stop+0xb/0xb [<ffffffff814a90f3>] ip_rcv+0x237/0x268 [<ffffffff81473def>] __netif_receive_skb+0x487/0x530 [<ffffffff81473f91>] process_backlog+0xf9/0x1da nfs: server 10.1.8.1 not responding, timed out [<ffffffff8147639a>] net_rx_action+0xad/0x218 [<ffffffff8108d50a>] __do_softirq+0x9c/0x161 [<ffffffff81538ddc>] call_softirq+0x1c/0x30 <EOI> [<ffffffff8100bd21>] do_softirq+0x41/0x7e [<ffffffff8108d68b>] _local_bh_enable_ip+0x7a/0x9f [<ffffffff8108d6c8>] local_bh_enable+0xd/0x11 [<ffffffff810f3661>] rcu_cpu_kthread+0xe6/0x11f [<ffffffff810a7ebe>] smpboot_thread_fn+0x253/0x259 [<ffffffff810a7c6b>] ? test_ti_thread_flag.clone.0+0x11/0x11 [<ffffffff810a0a6d>] kthread+0xc2/0xca [<ffffffff810a09ab>] ? __init_kthread_worker+0x56/0x56 [<ffffffff81537afc>] ret_from_fork+0x7c/0xb0 [<ffffffff810a09ab>] ? __init_kthread_worker+0x56/0x56 Code: Bad RIP value.nfs: server 10.1.0.1 not responding, timed out nfs: server 10.1.0.1 not responding, timed out > > Thanks, > Ben > > >
On Tue, 2013-01-22 at 09:08 -0800, Ben Greear wrote: > Unfortunately, I hit it again this morning after the first restart of > my application (which bounces all 3000 interfaces). Memory poisoning > was disabled. Is your NFS traffic using TCP or UDP ? -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 01/22/2013 09:17 AM, Eric Dumazet wrote: > On Tue, 2013-01-22 at 09:08 -0800, Ben Greear wrote: > >> Unfortunately, I hit it again this morning after the first restart of >> my application (which bounces all 3000 interfaces). Memory poisoning >> was disabled. > > Is your NFS traffic using TCP or UDP ? It's using TCP. Thanks, Ben >
diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c index d3fb97d..c07bdef 100644 --- a/drivers/net/macvlan.c +++ b/drivers/net/macvlan.c @@ -111,9 +111,15 @@ static int macvlan_broadcast_one(struct sk_buff *skb, const struct ethhdr *eth, bool local) { struct net_device *dev = vlan->dev; + if (!skb) return NET_RX_DROP; + if (!(dev->flags & IFF_UP)) { + kfree_skb(skb); + return NET_RX_DROP; + } + if (local) return vlan->forward(dev, skb);