| Message ID | 1307505541.3102.12.camel@edumazet-laptop (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 08/06/11 11:59, Eric Dumazet wrote: > Well, a bisection definitely should help, but needs a lot of time in > your case. Yes. compile, test, crash, walk out to the other building to press reset, lather, rinse, repeat. I need a reset button on the end of a 50M wire, or a hardware watchdog! Actually it's not so bad. If I turn off slub debugging the kernel panics and reboots itself. This.. : [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1 [ 2.913066] netconsole: device eth0 not up yet, forcing it [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz. [ 3.660118] Switching to clocksource tsc [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch rtl_nic/rtl8168e-1.fw (-2) [ 63.223513] r8169 0000:03:00.0: eth0: link down [ 63.223556] r8169 0000:03:00.0: eth0: link down ..is slowing down reboots considerably. 3.0-rc does _not_ like some timing hardware in my machine. Having said that, at least it does not randomly panic on SCSI like 2.6.39 does. Ok, I've ruled out TCPMSS. Found out where it was being set and neutered it. I've replicated it with only the single DNAT rule. > Could you try following patch, because this is the 'usual suspect' I had > yesterday : > > diff --git a/net/core/skbuff.c b/net/core/skbuff.c > index 46cbd28..9f548f9 100644 > --- a/net/core/skbuff.c > +++ b/net/core/skbuff.c > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta; > } > > +#if 0 > if (fastpath&& > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) { > memmove(skb->head + size, skb_shinfo(skb), > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, > off = nhead; > goto adjust_others; > } > - > +#endif > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); > if (!data) > goto nodata; > > > Nope.. that's not it. <sigh> That might have changed the characteristic of the fault slightly, but unfortunately I got caught with a couple of fsck's, so I only got to test it 3 times tonight. It's unfortunate that this is a production system, so I can only take it down between about 9pm and 1am. That would normally be pretty productive, except that an fsck of a 14TB ext4 can take 30 minutes if it panics at the wrong time. I'm out of time tonight, but I'll have a crack at some bisection tomorrow night. Now I just have to go back far enough that it works, and be near enough not to have to futz around with /proc /sys or drivers. I really, really, really appreciate you guys helping me with this. It has been driving me absolutely bonkers. If I'm ever in the same town as any of you, dinner and drinks are on me. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit : > On 08/06/11 11:59, Eric Dumazet wrote: > > > Well, a bisection definitely should help, but needs a lot of time in > > your case. > > Yes. compile, test, crash, walk out to the other building to press > reset, lather, rinse, repeat. > > I need a reset button on the end of a 50M wire, or a hardware watchdog! > > Actually it's not so bad. If I turn off slub debugging the kernel panics > and reboots itself. > > This.. : > [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1 > [ 2.913066] netconsole: device eth0 not up yet, forcing it > [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz. > [ 3.660118] Switching to clocksource tsc > [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch > rtl_nic/rtl8168e-1.fw (-2) > [ 63.223513] r8169 0000:03:00.0: eth0: link down > [ 63.223556] r8169 0000:03:00.0: eth0: link down > > ..is slowing down reboots considerably. 3.0-rc does _not_ like some > timing hardware in my machine. Having said that, at least it does not > randomly panic on SCSI like 2.6.39 does. > > Ok, I've ruled out TCPMSS. Found out where it was being set and neutered > it. I've replicated it with only the single DNAT rule. > > > > Could you try following patch, because this is the 'usual suspect' I had > > yesterday : > > > > diff --git a/net/core/skbuff.c b/net/core/skbuff.c > > index 46cbd28..9f548f9 100644 > > --- a/net/core/skbuff.c > > +++ b/net/core/skbuff.c > > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, > > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta; > > } > > > > +#if 0 > > if (fastpath&& > > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) { > > memmove(skb->head + size, skb_shinfo(skb), > > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, > > off = nhead; > > goto adjust_others; > > } > > - > > +#endif > > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); > > if (!data) > > goto nodata; > > > > > > > > Nope.. that's not it. <sigh> That might have changed the characteristic > of the fault slightly, but unfortunately I got caught with a couple of > fsck's, so I only got to test it 3 times tonight. > > It's unfortunate that this is a production system, so I can only take it > down between about 9pm and 1am. That would normally be pretty > productive, except that an fsck of a 14TB ext4 can take 30 minutes if it > panics at the wrong time. > > I'm out of time tonight, but I'll have a crack at some bisection > tomorrow night. Now I just have to go back far enough that it works, and > be near enough not to have to futz around with /proc /sys or drivers. > > I really, really, really appreciate you guys helping me with this. It > has been driving me absolutely bonkers. If I'm ever in the same town as > any of you, dinner and drinks are on me. Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not appropriate for production :( -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote: > On 08/06/11 11:59, Eric Dumazet wrote: > > >Well, a bisection definitely should help, but needs a lot of time in > >your case. > > Yes. compile, test, crash, walk out to the other building to press > reset, lather, rinse, repeat. > > I need a reset button on the end of a 50M wire, or a hardware watchdog! Not strictly on-topic, but in situations where I have machines that either don't have lights-out facilities or have broken ones I find that network controlled power switches to be very useful. At one point I would have need an 8000km long wire to the reset switch :-) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 11-06-09 10:52 PM, Simon Horman wrote: > On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote: >> On 08/06/11 11:59, Eric Dumazet wrote: >> >>> Well, a bisection definitely should help, but needs a lot of time in >>> your case. >> >> Yes. compile, test, crash, walk out to the other building to press >> reset, lather, rinse, repeat. >> >> I need a reset button on the end of a 50M wire, or a hardware watchdog! Something many of us don't realize is that nearly all Intel chipsets have a built-in hardware watchdog timer. This includes chipset for consumer desktop boards as well as the big iron server stuff. It's the "i8xx_tco" driver in the kernel enables use of them: modprobe i8xx_tco Cheers -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 10 Jun 2011, Mark Lord wrote: > Something many of us don't realize is that nearly all Intel chipsets > have a built-in hardware watchdog timer. This includes chipset for > consumer desktop boards as well as the big iron server stuff. > > It's the "i8xx_tco" driver in the kernel enables use of them: That's the old module name, but yes, it is very useful in desktops and laptops (when it works). Server-class hardware will have a baseboard management unit that can really power-cycle the system instead of just rebooting. And test it first before you depend on it triggering at a remote location, as the firmware might cause the Intel chipset watchdog to actually hang the box instead of causing a proper reboot (happens on the IBM thinkpad T43, for example).
On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)
Even more off-topic, there has been a case when a 200,000,000 km long
wire to the reset button was needed. IIRC they got away with a watchdog.
diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 46cbd28..9f548f9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta; } +#if 0 if (fastpath && size + sizeof(struct skb_shared_info) <= ksize(skb->head)) { memmove(skb->head + size, skb_shinfo(skb), @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, off = nhead; goto adjust_others; } - +#endif data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask); if (!data) goto nodata;