| Message ID | 20150316000438.GD8656@n2100.arm.linux.org.uk (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Mar 16, 2015 at 12:04:38AM +0000, Russell King - ARM Linux wrote: > On Sun, Mar 15, 2015 at 09:33:30PM +0000, Russell King - ARM Linux wrote: > > I'm going to try a few other kernels to try and track down what's going > > on - whether something from arm-soc or my tree is responsible for this > > really weird behaviour. > > Okay, this is weird - it seems that it's caused by the FIQ oops > dumping code/FIQ changes which I've carried for many months > unchanged in my tree. More weirdness. Progressing forwards through my development code showed that when I merged the patch I mentioned in the previous mail, things started to fail. As I also mentioned, I'd drop that branch (two patches, one adding the IPI backtrace stuff and the second one updating the GIC to allow it to raise FIQs on suitably equipped platforms.) I would have expected that to have worked, but it just failed after four boot iterations. So either it's not the FIQ, or it is the FIQ code _and_ also something else. Or it has something to do with the placement of functions in the kernel. I'll try more stuff tomorrow, working from where I presently am (which is basically last night's code minus the FIQ changes) by removing other changes to see what brings us back to a working system. As I've already said - this is really weird because all of these changes were also tested against -rc1... those which weren't are: mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE mm: split ET_DYN ASLR from mmap ASLR mm: move randomize_et_dyn into ELF_ET_DYN_BASE mm: expose arch_mmap_rnd when available arm: factor out mmap ASLR into mmap_rnd and a number of clkdev rework patches (to make it use clk_hw internally.) Neither of these should be affecting it, but that's something I will be testing tomorrow.
On Mon, Mar 16, 2015 at 12:42:39AM +0000, Russell King - ARM Linux wrote: > On Mon, Mar 16, 2015 at 12:04:38AM +0000, Russell King - ARM Linux wrote: > > On Sun, Mar 15, 2015 at 09:33:30PM +0000, Russell King - ARM Linux wrote: > > > I'm going to try a few other kernels to try and track down what's going > > > on - whether something from arm-soc or my tree is responsible for this > > > really weird behaviour. > > > > Okay, this is weird - it seems that it's caused by the FIQ oops > > dumping code/FIQ changes which I've carried for many months > > unchanged in my tree. > > More weirdness. Progressing forwards through my development code > showed that when I merged the patch I mentioned in the previous mail, > things started to fail. > > As I also mentioned, I'd drop that branch (two patches, one adding > the IPI backtrace stuff and the second one updating the GIC to allow > it to raise FIQs on suitably equipped platforms.) I would have > expected that to have worked, but it just failed after four boot > iterations. So either it's not the FIQ, or it is the FIQ code _and_ > also something else. Or it has something to do with the placement > of functions in the kernel. > > I'll try more stuff tomorrow, working from where I presently am > (which is basically last night's code minus the FIQ changes) by > removing other changes to see what brings us back to a working > system. > > As I've already said - this is really weird because all of these > changes were also tested against -rc1... those which weren't are: > > mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE > mm: split ET_DYN ASLR from mmap ASLR > mm: move randomize_et_dyn into ELF_ET_DYN_BASE > mm: expose arch_mmap_rnd when available > arm: factor out mmap ASLR into mmap_rnd > > and a number of clkdev rework patches (to make it use clk_hw > internally.) Neither of these should be affecting it, but that's > something I will be testing tomorrow. Okay, reverting the ASLR changes and the clkdev changes annoyingly still results in random failure.
On Mon, Mar 16, 2015 at 09:35:53AM +0000, Russell King - ARM Linux wrote: > On Mon, Mar 16, 2015 at 12:42:39AM +0000, Russell King - ARM Linux wrote: > > On Mon, Mar 16, 2015 at 12:04:38AM +0000, Russell King - ARM Linux wrote: > > > On Sun, Mar 15, 2015 at 09:33:30PM +0000, Russell King - ARM Linux wrote: > > > > I'm going to try a few other kernels to try and track down what's going > > > > on - whether something from arm-soc or my tree is responsible for this > > > > really weird behaviour. > > > > > > Okay, this is weird - it seems that it's caused by the FIQ oops > > > dumping code/FIQ changes which I've carried for many months > > > unchanged in my tree. > > > > More weirdness. Progressing forwards through my development code > > showed that when I merged the patch I mentioned in the previous mail, > > things started to fail. > > > > As I also mentioned, I'd drop that branch (two patches, one adding > > the IPI backtrace stuff and the second one updating the GIC to allow > > it to raise FIQs on suitably equipped platforms.) I would have > > expected that to have worked, but it just failed after four boot > > iterations. So either it's not the FIQ, or it is the FIQ code _and_ > > also something else. Or it has something to do with the placement > > of functions in the kernel. > > > > I'll try more stuff tomorrow, working from where I presently am > > (which is basically last night's code minus the FIQ changes) by > > removing other changes to see what brings us back to a working > > system. > > > > As I've already said - this is really weird because all of these > > changes were also tested against -rc1... those which weren't are: > > > > mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE > > mm: split ET_DYN ASLR from mmap ASLR > > mm: move randomize_et_dyn into ELF_ET_DYN_BASE > > mm: expose arch_mmap_rnd when available > > arm: factor out mmap ASLR into mmap_rnd > > > > and a number of clkdev rework patches (to make it use clk_hw > > internally.) Neither of these should be affecting it, but that's > > something I will be testing tomorrow. > > Okay, reverting the ASLR changes and the clkdev changes annoyingly still > results in random failure. After ruling out ASLR and clkdev, I started progressively reverting other stuff in the build tree. Eventually, I got down to reverting the L2C change I've been carrying since the L2C cleanups. With that lot reverted, which is slightly more than the previously known good test, it booted five times without issue. So, I thought I'd add that L2C change to the list of bad commits, and try omitting _just_ the L2C and FIQ changes... and it still fails - on the first test boot iteration. I think I'm going to declare that this is all down to some obscure hardware problem with Versatile Express, which is tickled by the layout of the kernel against the cache, and take it out of the nightly system (it's pointless having unstable hardware being tested; random failures are completely meaningless.)
Hi Russell, On 16/03/15 13:04, Russell King - ARM Linux wrote: > On Mon, Mar 16, 2015 at 09:35:53AM +0000, Russell King - ARM Linux wrote: >> On Mon, Mar 16, 2015 at 12:42:39AM +0000, Russell King - ARM Linux wrote: >>> On Mon, Mar 16, 2015 at 12:04:38AM +0000, Russell King - ARM Linux wrote: >>>> On Sun, Mar 15, 2015 at 09:33:30PM +0000, Russell King - ARM Linux wrote: >>>>> I'm going to try a few other kernels to try and track down what's going >>>>> on - whether something from arm-soc or my tree is responsible for this >>>>> really weird behaviour. >>>> >>>> Okay, this is weird - it seems that it's caused by the FIQ oops >>>> dumping code/FIQ changes which I've carried for many months >>>> unchanged in my tree. >>> >>> More weirdness. Progressing forwards through my development code >>> showed that when I merged the patch I mentioned in the previous mail, >>> things started to fail. >>> >>> As I also mentioned, I'd drop that branch (two patches, one adding >>> the IPI backtrace stuff and the second one updating the GIC to allow >>> it to raise FIQs on suitably equipped platforms.) I would have >>> expected that to have worked, but it just failed after four boot >>> iterations. So either it's not the FIQ, or it is the FIQ code _and_ >>> also something else. Or it has something to do with the placement >>> of functions in the kernel. >>> >>> I'll try more stuff tomorrow, working from where I presently am >>> (which is basically last night's code minus the FIQ changes) by >>> removing other changes to see what brings us back to a working >>> system. >>> >>> As I've already said - this is really weird because all of these >>> changes were also tested against -rc1... those which weren't are: >>> >>> mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE >>> mm: split ET_DYN ASLR from mmap ASLR >>> mm: move randomize_et_dyn into ELF_ET_DYN_BASE >>> mm: expose arch_mmap_rnd when available >>> arm: factor out mmap ASLR into mmap_rnd >>> >>> and a number of clkdev rework patches (to make it use clk_hw >>> internally.) Neither of these should be affecting it, but that's >>> something I will be testing tomorrow. >> >> Okay, reverting the ASLR changes and the clkdev changes annoyingly still >> results in random failure. > > After ruling out ASLR and clkdev, I started progressively reverting other > stuff in the build tree. Eventually, I got down to reverting the L2C > change I've been carrying since the L2C cleanups. > > With that lot reverted, which is slightly more than the previously known > good test, it booted five times without issue. > > So, I thought I'd add that L2C change to the list of bad commits, and try > omitting _just_ the L2C and FIQ changes... and it still fails - on the > first test boot iteration. > > I think I'm going to declare that this is all down to some obscure > hardware problem with Versatile Express, which is tickled by the layout > of the kernel against the cache, and take it out of the nightly system > (it's pointless having unstable hardware being tested; random failures > are completely meaningless.) > I was able to see exact behaviour on my VExpress setup with CA9X4 core-tile. Few observations from my side: 1. This issue can be reproduced even on v3.19 2. As you suspected L2C, I tried disabling L2C and it seems to solve the issue 3. Since it's very random and enabling LL_DEBUG made it difficult to reproduce the issue, I tried to dump the stack using DS5 debugger 4. The stack is exactly same always both on v4.0-rc* and v3.19 kernel and on multiple runs 5. Connecting to h/w debugger, stopping and re-starting the CPUs, solves the issue. It's helping CPUs to get out of __radix_tree_lookup somehow Stacktrace ========== (sorry it's looks different from std. Linux backtrace as this one id dump from DS5) CPU 0 ---- #0 __radix_tree_lookup( root = <Value currently has no location>, index = 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at radix-tree.c:517 #1 generic_handle_irq( irq = 16 ) at irqdesc.c:349 #2 __handle_domain_irq( domain = (struct irq_domain*) 0xBF004400, hwirq = 16, lookup = <Value currently has no location>, regs = <Value currently has no location> ) at irqdesc.c:391 #3 __raw_readl( addr = <Value optimised away by compiler> ) at io.h:121 #4 gic_handle_irq( regs = (struct pt_regs*) 0x805F1F40 ) at irq-gic.c:277 #5 [__irq_svc+0x40] CPU1 ---- #0 __radix_tree_lookup( root = <Value currently has no location>, index = 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at radix-tree.c:517 #1 __irq_get_desc_lock( irq = <Value currently has no location>, flags = (long unsigned int*) 0xBF08BF94, bus = false, check = 3 ) at irqdesc.c:544 #2 enable_percpu_irq( irq = 16, type = 0 ) at manage.c:1583 #3 twd_timer_cpu_notify( self = <Value not available : Undefined value in stack frame for register R0>, action = <Value currently has no location>, hcpu = <Value not available : Undefined value in stack frame for register R2> ) at smp_twd.c:322 #4 notifier_call_chain( nl = <Value currently has no location>, val = <Value not available : Undefined value in stack frame for register R1>, v = <Value not available : Undefined value in stack frame for register R2>, nr_to_call = <Value not available : Undefined value in stack frame for register R3>, nr_calls = (int*) 0x0 ) at notifier.c:95 #5 notifier_to_errno( ret = <Value currently has no location> ) at notifier.h:179 #6 cpu_notify( val = <Value currently has no location>, v = <Value currently has no location> ) at cpu.c:234 #7 secondary_start_kernel() at smp.c:367 CPU2 & CPU3 ----------- Not booted yet, still waiting in bootloader
On Mon, Mar 16, 2015 at 05:47:46PM +0000, Sudeep Holla wrote: > Hi Russell, > > I was able to see exact behaviour on my VExpress setup with CA9X4 core-tile. > Few observations from my side: > > 1. This issue can be reproduced even on v3.19 > 2. As you suspected L2C, I tried disabling L2C and it seems to solve > the issue My L2C says it's cache ID is 0x410000c3 - which is indeed a L2C-310, but with an undocumented revision ID of 3, which as far as we can make out, it's a R1Px where x > 0. > 3. Since it's very random and enabling LL_DEBUG made it difficult to > reproduce the issue, I tried to dump the stack using DS5 debugger > 4. The stack is exactly same always both on v4.0-rc* and v3.19 kernel > and on multiple runs Hmm, I haven't seen them before I moved to 4.0-rc3 - before then my nightly boot tests (which run two boots on the platform each night) always seemed to succeed. > 5. Connecting to h/w debugger, stopping and re-starting the CPUs, > solves the issue. It's helping CPUs to get out of __radix_tree_lookup > somehow Interesting. Are the traces below from 4.0-rc3 or an older kernel? > Stacktrace > ========== > (sorry it's looks different from std. Linux backtrace as this one id dump > from DS5) > > CPU 0 > ---- > #0 __radix_tree_lookup( root = <Value currently has no location>, index = > 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at > radix-tree.c:517 Can you dump the disassembly around this location for both CPU0 and CPU1 and the register values please? I think it would be interesting to see if they're both stuck on exactly the same address access. I've currently narrowed down my latest potential culpret to something in my Cubox-i code... specifically something in my "cubox-i-sdio" or "imx-drm^" branches. The cubox-i-sdio branch contains Olof's modifications to MMC to support resets and regulators associated with wifi cards, which would be built, but we would not have executed any of the MMC code at the point where we'd be bringing the secondary CPUs up. The imx-drm^ changes don't touch any file which is built into my Versatile Express kernel, so it's unlikely to affect anything (though, I'm build-boot-testing with imx-drm^ but cubox-i-sdio dropped just to make absolutely sure.) One thing I've tried is turning off are the Cortex-A9 features - early BRESP and full line of zeros. That seems to make no apparent difference, though it's hard to tell when #if 0'ing out the code, because that changes the code placement and seems to stop the problem triggering. I did have a case where disabling FLZ (via #if 0'ing it out) seemed to solve it with errata 588369 enabled, but changing the code to clear the FLZ bit instead (which should have had the same effect) resulted in the problem re-appearing. I'm beginning to believe at this point that there /is/ a bug in the L2C on the test chip, and that we're probably better off changing the Versatile Express DT files to disable the L2C cache controller... what are your thoughts on that? I'm currently doing up to 8 boot tests - if I can do 8 consecutive boot tests which all succeed, I'm declaring it a pass, otherwise it's a fail. Generally, I've found that it will fail very early (like the first) but sometimes up to the 4th. I guess one thing we need to confirm is whether we have exactly the same hardware and firmware versions. Here's my board's early boot messages: ARM V2M Boot loader v1.1.1 HBI0190 build 2313 ARM V2M Firmware v3.1.2 Build Date: Apr 16 2013 Date: Mon 30 Mar 2009 Time: 16:59:14 Cmd> reboot Powering up system... Daughterboard fitted to site 1. Switching on ATXPSU... ATX3V3: ON VIOset: 1.8V MBtemp: 27 degC Configuring motherboard (rev D, var A)... IOFPGA config: PASSED MUXFPGA config: PASSED OSC CLK config: PASSED Testing SMC devices (FPGA build 8)... SRAM 32MB test: PASSED VRAM 8MB test: PASSED LAN9118 test: PASSED USB & OTG test: PASSED KMI1/KMI2 test: PASSED MMC & SD test: PASSED DVI image test: PASSED AACI AC97 test: PASSED CF card test: PASSED UART port test: PASSED MAC addrs test: PASSED Reading Site 1 Board File \SITE1\HBI0191B\board.txt DB1 JTAG configuration complete. Setting DB1 OSCCLKS... DB1.0 DCC 0 SPI configuration complete. Writing SCC 0x40610000 with 0xBB8A802A Writing SCC 0x40610001 with 0x00001F09 Writing SCC 0x40610002 with 0x00000000 DB1.0 DCC 0 SCC configuration complete. DB SMB clock enabled. Waiting for SITE1 CB_READY... Testing SMB clock... Configuring MUXFPGA for MB. Setting DVI mode for VGA. Releasing Daughterboard resets. Switching MCC log to UART1. Warning: Card Format not recognised, please check card. ARM Versatile Express Boot Monitor Version: V5.2.1 Build Date: Apr 4 2013 Daughterboard Site 1: V2P-CA9 Cortex A9 Daughterboard Site 2: Not Used Running boot script from flash - BOOTSCRIPT U-Boot 2013.01.-rc1-00003-g43ee87aabf17-dirty (Jan 07 2014 - 00:00:38) ...
On 16/03/15 18:16, Russell King - ARM Linux wrote: > On Mon, Mar 16, 2015 at 05:47:46PM +0000, Sudeep Holla wrote: >> Hi Russell, >> >> I was able to see exact behaviour on my VExpress setup with CA9X4 core-tile. >> Few observations from my side: >> >> 1. This issue can be reproduced even on v3.19 >> 2. As you suspected L2C, I tried disabling L2C and it seems to solve >> the issue > > My L2C says it's cache ID is 0x410000c3 - which is indeed a L2C-310, but > with an undocumented revision ID of 3, which as far as we can make out, > it's a R1Px where x > 0. > >> 3. Since it's very random and enabling LL_DEBUG made it difficult to >> reproduce the issue, I tried to dump the stack using DS5 debugger >> 4. The stack is exactly same always both on v4.0-rc* and v3.19 kernel >> and on multiple runs > > Hmm, I haven't seen them before I moved to 4.0-rc3 - before then my > nightly boot tests (which run two boots on the platform each night) > always seemed to succeed. > >> 5. Connecting to h/w debugger, stopping and re-starting the CPUs, >> solves the issue. It's helping CPUs to get out of __radix_tree_lookup >> somehow > > Interesting. Are the traces below from 4.0-rc3 or an older kernel? > This one is with v3.19 but I get exact same trace with v4.0-rc* kernel. >> Stacktrace >> ========== >> (sorry it's looks different from std. Linux backtrace as this one id dump >> from DS5) >> >> CPU 0 >> ---- >> #0 __radix_tree_lookup( root = <Value currently has no location>, index = >> 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at >> radix-tree.c:517 > > Can you dump the disassembly around this location for both CPU0 and CPU1 > and the register values please? I think it would be interesting to see > if they're both stuck on exactly the same address access. > (with v4.0-rc4 this time) CPU#0 ===== #0 __radix_tree_lookup( root = <Value currently has no location>, index = 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at radix-tree.c:517 node = (struct radix_tree_node*) 0xBEC00001 parent = <Value optimised away by compiler> height = 1 shift = 0 slot = <Value currently has no location> #1 generic_handle_irq( irq = 16 ) at irqdesc.c:349 desc = <Value optimised away by compiler> #2 __handle_domain_irq( domain = (struct irq_domain*) 0xBF004400, hwirq = 16, lookup = <Value currently has no location>, regs = <Value currently has no location> ) at irqdesc.c:391 old_regs = (struct pt_regs*) 0x0 irq = <Value optimised away by compiler> ret = 0 #3 __raw_readl( addr = <Value optimised away by compiler> ) at io.h:121 #4 gic_handle_irq( regs = (struct pt_regs*) 0x805F1F40 ) at irq-gic.c:277 irqstat = 2147518036 irqnr = <Value currently has no location> gic = <Value optimised away by compiler> cpu_base = (void*) 0xC0802100 #5 [__irq_svc+0x40] S:0x8021F80C : LSL lr,r4,#3 S:0x8021F810 : SUB lr,lr,r4,LSL #1 S:0x8021F814 : SUB lr,lr,#6 S:0x8021F818 : B {pc}+8 ; 0x8021f820 S:0x8021F81C : MOV r5,r0 S:0x8021F820 : LSR r12,r1,lr S:0x8021F824 : SUB lr,lr,#6 S:0x8021F828 : AND r12,r12,#0x3f S:0x8021F82C : ADD r12,r12,#6 S:0x8021F830 : LDR r0,[r5,r12,LSL #2] Core registers: R0 0x0000003F R1 0x00000010 R2 0x00000000 R3 0x00000000 R4 0x00000001 R5 0xBEC00000 R6 0x00000000 R7 0x00000000 R8 0xBF004400 R9 0x805F1F90 R10 0x00000001 R11 0x805EEB08 R12 0xBEC00001 SP 0x805F1EFC LR 0x00000000 PC 0x8021F820 CPSR 0x80000193 CPU#1 ===== #0 __radix_tree_lookup( root = <Value currently has no location>, index = 16, nodep = (struct radix_tree_node**) 0x0, slotp = (void***) 0x0 ) at radix-tree.c:517 node = (struct radix_tree_node*) 0xBEC00001 parent = <Value optimised away by compiler> height = 1 shift = 0 slot = <Value currently has no location> #1 __irq_get_desc_lock( irq = <Value currently has no location>, flags = (long unsigned int*) 0xBF08BF94, bus = false, check = 3 ) at irqdesc.c:544 desc = <Value optimised away by compiler> #2 enable_percpu_irq( irq = 16, type = 0 ) at manage.c:1583 cpu = 1 flags = <Value currently has no location> desc = <Value optimised away by compiler> #3 twd_timer_cpu_notify( self = <Value not available : Undefined value in stack frame for register R0>, action = <Value currently has no location>, hcpu = <Value not available : Undefined value in stack frame for register R2> ) at smp_twd.c:322 #4 notifier_call_chain( nl = <Value currently has no location>, val = <Value not available : Undefined value in stack frame for register R1>, v = <Value not available : Undefined value in stack frame for register R2>, nr_to_call = <Value not available : Undefined value in stack frame for register R3>, nr_calls = (int*) 0x0 ) at notifier.c:95 ret = <Value currently has no location> nb = <Value optimised away by compiler> next_nb = <Value optimised away by compiler> #5 notifier_to_errno( ret = <Value currently has no location> ) at notifier.h:179 #6 cpu_notify( val = <Value currently has no location>, v = <Value currently has no location> ) at cpu.c:234 #7 secondary_start_kernel() at smp.c:367 mm = <Value optimised away by compiler> cpu = 1 #8 [S:0x60008724] Disassembly: S:0x8021F80C : LSL lr,r4,#3 S:0x8021F810 : SUB lr,lr,r4,LSL #1 S:0x8021F814 : SUB lr,lr,#6 S:0x8021F818 : B {pc}+8 ; 0x8021f820 S:0x8021F81C : MOV r5,r0 S:0x8021F820 : LSR r12,r1,lr S:0x8021F824 : SUB lr,lr,#6 S:0x8021F828 : AND r12,r12,#0x3f S:0x8021F82C : ADD r12,r12,#6 S:0x8021F830 : LDR r0,[r5,r12,LSL #2] Core registers: R0 0x0000003F R1 0x00000010 R2 0x00000000 R3 0x00000000 R4 0x00000001 R5 0xBEC00000 R6 0xBF08BF94 R7 0x00000000 R8 0x805F92A0 R9 0x00000000 R10 0x00000000 R11 0x00000000 R12 0xBEC00001 SP 0xBF08BF6C LR 0x00000000 PC 0x8021F820 CPSR 0x800001D3 Nzcvq_ge3ge2ge1ge0_inactive_eAIFtj_SVC [...] > I'm beginning to believe at this point that there /is/ a bug in the L2C on > the test chip, and that we're probably better off changing the Versatile > Express DT files to disable the L2C cache controller... what are your > thoughts on that? > I was thinking of taking the dump of L2C register settings and comparing them. But currently I am facing issues booting even v3.18 on my setup, it seem to fails somewhere else which I need to look at. > I'm currently doing up to 8 boot tests - if I can do 8 consecutive boot > tests which all succeed, I'm declaring it a pass, otherwise it's a fail. > Generally, I've found that it will fail very early (like the first) but > sometimes up to the 4th. > > I guess one thing we need to confirm is whether we have exactly the same > hardware and firmware versions. Here's my board's early boot messages: > ARM V2M Boot loader v1.1.2 HBI0190 build 2313 ARM V2M Firmware v3.1.2 Build Date: Apr 16 2013 Date: Mon 16 Mar 2015 Time: 18:57:21 Powering up system... Daughterboard fitted to site 1. Switching on ATXPSU... ATX3V3: ON VIOset: 1.8V MBtemp: 26 degC Configuring motherboard (rev D, var A)... IOFPGA config: PASSED MUXFPGA config: PASSED OSC CLK config: PASSED Testing SMC devices (FPGA build 8)... SRAM 32MB test: PASSED VRAM 8MB test: PASSED LAN9118 test: PASSED USB & OTG test: PASSED KMI1/KMI2 test: PASSED MMC & SD test: PASSED DVI image test: PASSED AACI AC97 test: PASSED CF card test: PASSED UART port test: PASSED MAC addrs test: PASSED Reading Site 1 Board File \SITE1\HBI0191B\board.txt DB1 JTAG configuration complete. Setting DB1 OSCCLKS... DB1.0 DCC 0 SPI configuration complete. Writing SCC 0x40610000 with 0xBB8A802A Writing SCC 0x40610001 with 0x00001F09 Writing SCC 0x40610002 with 0x00000000 DB1.0 DCC 0 SCC configuration complete. DB SMB clock enabled. Waiting for SITE1 CB_READY... Testing SMB clock... Configuring MUXFPGA for MB. Setting DVI mode for VGA. Releasing Daughterboard resets. Switching MCC log to UART1. %BootMonitor-Warning, Unable to open SYSTEM.DAT ARM Versatile Express Boot Monitor Version: V5.2.1 Build Date: Apr 4 2013 Daughterboard Site 1: V2P-CA9 Cortex A9 Daughterboard Site 2: Not Used Running boot script from flash - BOOTSCRIPT
On Mon, Mar 16, 2015 at 07:16:05PM +0000, Sudeep Holla wrote: > On 16/03/15 18:16, Russell King - ARM Linux wrote: > >Can you dump the disassembly around this location for both CPU0 and CPU1 > >and the register values please? I think it would be interesting to see > >if they're both stuck on exactly the same address access. > > (with v4.0-rc4 this time) Thanks. > CPU#0 > ===== ... > S:0x8021F80C : LSL lr,r4,#3 > S:0x8021F810 : SUB lr,lr,r4,LSL #1 > S:0x8021F814 : SUB lr,lr,#6 > S:0x8021F818 : B {pc}+8 ; 0x8021f820 > S:0x8021F81C : MOV r5,r0 > S:0x8021F820 : LSR r12,r1,lr > S:0x8021F824 : SUB lr,lr,#6 > S:0x8021F828 : AND r12,r12,#0x3f > S:0x8021F82C : ADD r12,r12,#6 > S:0x8021F830 : LDR r0,[r5,r12,LSL #2] > > Core registers: > R0 0x0000003F > R1 0x00000010 > R2 0x00000000 > R3 0x00000000 > R4 0x00000001 > R5 0xBEC00000 > R6 0x00000000 > R7 0x00000000 > R8 0xBF004400 > R9 0x805F1F90 > R10 0x00000001 > R11 0x805EEB08 > R12 0xBEC00001 > SP 0x805F1EFC > LR 0x00000000 > PC 0x8021F820 > CPSR 0x80000193 > > CPU#1 > ===== ... > S:0x8021F80C : LSL lr,r4,#3 > S:0x8021F810 : SUB lr,lr,r4,LSL #1 > S:0x8021F814 : SUB lr,lr,#6 > S:0x8021F818 : B {pc}+8 ; 0x8021f820 > S:0x8021F81C : MOV r5,r0 > S:0x8021F820 : LSR r12,r1,lr > S:0x8021F824 : SUB lr,lr,#6 > S:0x8021F828 : AND r12,r12,#0x3f > S:0x8021F82C : ADD r12,r12,#6 > S:0x8021F830 : LDR r0,[r5,r12,LSL #2] > > Core registers: > R0 0x0000003F > R1 0x00000010 > R2 0x00000000 > R3 0x00000000 > R4 0x00000001 > R5 0xBEC00000 > R6 0xBF08BF94 > R7 0x00000000 > R8 0x805F92A0 > R9 0x00000000 > R10 0x00000000 > R11 0x00000000 > R12 0xBEC00001 > SP 0xBF08BF6C > LR 0x00000000 > PC 0x8021F820 > CPSR 0x800001D3 Nzcvq_ge3ge2ge1ge0_inactive_eAIFtj_SVC And we find that both CPUs have stopped at exactly the same place, which is an arithmetic instruction. If I had to guess, I'd say the reason it's stopped there (exactly on a cache line boundary) is because both CPUs are waiting for an instruction fetch to complete into its L1 I-cache, and for some reason, the L2 cache is not satisfying the request from either CPU. The question of course is... why not. > >I guess one thing we need to confirm is whether we have exactly the same > >hardware and firmware versions. Here's my board's early boot messages: Looks like we're broadly the same, apart from the boot loader version. You have 1.1.2, whereas I have 1.1.1. Co-incidentally, I just looked at the disassembly of my __radix_tree_lookup: c0199750: e0050495 mul r5, r5, r4 c0199754: e2455006 sub r5, r5, #6 c0199758: ea000000 b c0199760 <__radix_tree_lookup+0x70> c019975c: e1a0c000 mov ip, r0 c0199760: e1a06531 lsr r6, r1, r5 c0199764: e206603f and r6, r6, #63 ; 0x3f c0199768: e2866006 add r6, r6, #6 c019976c: e79c0106 ldr r0, [ip, r6, lsl #2] The code is slightly different, but notice that the alignment of the LSR instruction is the same as yours - at first I wondered whether that's coincidence or not. However, taking Olof's MMC changes back out of my tree (which results in a booting kernel) makes no difference to the placement of this code. The start of the read-only data section doesn't change between the working and non-working kernels, but the location of the spinlock and some scheduler code does change (along with all the networking code.) There's changes in the read-only data section, there's also changes to a set of "descriptor.NNNNN" symbols towards the end of the data section, which goes on to change the placement of the bss section. The diff between the System.map is unpostable - it's about 1.3MB. :(
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 17e54f1df258..e0ba62117c5a 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -543,7 +543,7 @@ static void ipi_cpu_stop(unsigned int cpu) cpu_relax(); } -static void ipi_cpu_backtrace(struct pt_regs *regs) +void ipi_cpu_backtrace(struct pt_regs *regs) { int cpu = smp_processor_id(); diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 439138d3437e..f88af68f9345 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -60,6 +60,7 @@ static int __init user_debug_setup(char *str) __setup("user_debug=", user_debug_setup); #endif +extern void ipi_cpu_backtrace(struct pt_regs *regs); static void dump_mem(const char *, const char *, unsigned long, unsigned long); void dump_backtrace_entry(unsigned long where, unsigned long from, unsigned long frame) @@ -480,6 +481,9 @@ asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs) nmi_enter(); /* nop. FIQ handlers for special arch/arm features can be added here. */ +#ifdef CONFIG_SMP + ipi_cpu_backtrace(regs); +#endif nmi_exit(); diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c index dda6dbc23565..786a662f9842 100644 --- a/drivers/irqchip/irq-gic.c +++ b/drivers/irqchip/irq-gic.c @@ -48,6 +48,8 @@ #include "irq-gic-common.h" #include "irqchip.h" +#define GICD_ENABLE_GRP1 0x2 + union gic_base { void __iomem *common_base; void __percpu * __iomem *percpu_base; @@ -102,7 +104,7 @@ static struct gic_chip_data gic_data[MAX_GIC_NR] __read_mostly; #ifdef CONFIG_GIC_NON_BANKED static void __iomem *gic_get_percpu_base(union gic_base *base) { - return *__this_cpu_ptr(base->percpu_base); + return raw_cpu_read(*base->percpu_base); } static void __iomem *gic_get_common_base(union gic_base *base) @@ -270,8 +272,7 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs) irqnr = irqstat & GICC_IAR_INT_ID_MASK; if (likely(irqnr > 15 && irqnr < 1021)) { - irqnr = irq_find_mapping(gic->domain, irqnr); - handle_IRQ(irqnr, regs); + handle_domain_irq(gic->domain, irqnr, regs); continue; } if (irqnr < 16) { @@ -298,8 +299,8 @@ static void gic_handle_cascade_irq(unsigned int irq, struct irq_desc *desc) status = readl_relaxed(gic_data_cpu_base(chip_data) + GIC_CPU_INTACK); raw_spin_unlock(&irq_controller_lock); - gic_irq = (status & 0x3ff); - if (gic_irq == 1023) + gic_irq = (status & GICC_IAR_INT_ID_MASK); + if (gic_irq == GICC_INT_SPURIOUS) goto out; cascade_irq = irq_find_mapping(chip_data->domain, gic_irq); @@ -353,6 +354,25 @@ static u8 gic_get_cpumask(struct gic_chip_data *gic) return mask; } +static void gic_cpu_if_up(void) +{ + void __iomem *cpu_base = gic_data_cpu_base(&gic_data[0]); + void __iomem *dist_base = gic_data_dist_base(&gic_data[0]); + u32 bypass = 0; + + /* + * Preserve bypass disable bits to be written back later + */ + bypass = readl(cpu_base + GIC_CPU_CTRL); + bypass &= GICC_DIS_BYPASS_MASK; + + if (readl_relaxed(dist_base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) + bypass |= 0x1e; + + writel_relaxed(bypass | GICC_ENABLE, cpu_base + GIC_CPU_CTRL); +} + + static void __init gic_dist_init(struct gic_chip_data *gic) { unsigned int i; @@ -360,7 +380,7 @@ static void __init gic_dist_init(struct gic_chip_data *gic) unsigned int gic_irqs = gic->gic_irqs; void __iomem *base = gic_data_dist_base(gic); - writel_relaxed(0, base + GIC_DIST_CTRL); + writel_relaxed(GICD_DISABLE, base + GIC_DIST_CTRL); /* * Set all global interrupts to this CPU only. @@ -371,9 +391,19 @@ static void __init gic_dist_init(struct gic_chip_data *gic) for (i = 32; i < gic_irqs; i += 4) writel_relaxed(cpumask, base + GIC_DIST_TARGET + i * 4 / 4); + writel_relaxed(GICD_ENABLE_GRP1, base + GIC_DIST_CTRL); + + /* + * Optionally set all global interrupts to be group 1. + */ + if (readl_relaxed(base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) { + for (i = 32; i < gic_irqs; i += 32) + writel_relaxed(0xffffffff, base + GIC_DIST_IGROUP + i * 4 / 32); + } + gic_dist_config(base, gic_irqs, NULL); - writel_relaxed(1, base + GIC_DIST_CTRL); + writel_relaxed(GICD_ENABLE | GICD_ENABLE_GRP1, base + GIC_DIST_CTRL); } static void gic_cpu_init(struct gic_chip_data *gic) @@ -400,14 +430,29 @@ static void gic_cpu_init(struct gic_chip_data *gic) gic_cpu_config(dist_base, NULL); - writel_relaxed(0xf0, base + GIC_CPU_PRIMASK); - writel_relaxed(1, base + GIC_CPU_CTRL); + /* + * Set all PPI and SGI interrupts to be group 1. + * + * If grouping is not available (not implemented or prohibited by + * security mode) these registers are read-as-zero/write-ignored. + */ + if (readl_relaxed(dist_base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) { + writel_relaxed(0xfffffeff, dist_base + GIC_DIST_IGROUP + 0); + writel_relaxed(0xa0a0a000, dist_base + GIC_DIST_PRI + 8); + } + + writel_relaxed(GICC_INT_PRI_THRESHOLD, base + GIC_CPU_PRIMASK); + gic_cpu_if_up(); } void gic_cpu_if_down(void) { void __iomem *cpu_base = gic_data_cpu_base(&gic_data[0]); - writel_relaxed(0, cpu_base + GIC_CPU_CTRL); + u32 val = 0; + + val = readl(cpu_base + GIC_CPU_CTRL); + val &= ~GICC_ENABLE; + writel_relaxed(val, cpu_base + GIC_CPU_CTRL); } #ifdef CONFIG_CPU_PM @@ -467,14 +512,14 @@ static void gic_dist_restore(unsigned int gic_nr) if (!dist_base) return; - writel_relaxed(0, dist_base + GIC_DIST_CTRL); + writel_relaxed(GICD_DISABLE, dist_base + GIC_DIST_CTRL); for (i = 0; i < DIV_ROUND_UP(gic_irqs, 16); i++) writel_relaxed(gic_data[gic_nr].saved_spi_conf[i], dist_base + GIC_DIST_CONFIG + i * 4); for (i = 0; i < DIV_ROUND_UP(gic_irqs, 4); i++) - writel_relaxed(0xa0a0a0a0, + writel_relaxed(GICD_INT_DEF_PRI_X4, dist_base + GIC_DIST_PRI + i * 4); for (i = 0; i < DIV_ROUND_UP(gic_irqs, 4); i++) @@ -485,7 +530,7 @@ static void gic_dist_restore(unsigned int gic_nr) writel_relaxed(gic_data[gic_nr].saved_spi_enable[i], dist_base + GIC_DIST_ENABLE_SET + i * 4); - writel_relaxed(1, dist_base + GIC_DIST_CTRL); + writel_relaxed(GICD_ENABLE | 2, dist_base + GIC_DIST_CTRL); } static void gic_cpu_save(unsigned int gic_nr) @@ -504,11 +549,11 @@ static void gic_cpu_save(unsigned int gic_nr) if (!dist_base || !cpu_base) return; - ptr = __this_cpu_ptr(gic_data[gic_nr].saved_ppi_enable); + ptr = raw_cpu_ptr(gic_data[gic_nr].saved_ppi_enable); for (i = 0; i < DIV_ROUND_UP(32, 32); i++) ptr[i] = readl_relaxed(dist_base + GIC_DIST_ENABLE_SET + i * 4); - ptr = __this_cpu_ptr(gic_data[gic_nr].saved_ppi_conf); + ptr = raw_cpu_ptr(gic_data[gic_nr].saved_ppi_conf); for (i = 0; i < DIV_ROUND_UP(32, 16); i++) ptr[i] = readl_relaxed(dist_base + GIC_DIST_CONFIG + i * 4); @@ -530,19 +575,31 @@ static void gic_cpu_restore(unsigned int gic_nr) if (!dist_base || !cpu_base) return; - ptr = __this_cpu_ptr(gic_data[gic_nr].saved_ppi_enable); + ptr = raw_cpu_ptr(gic_data[gic_nr].saved_ppi_enable); for (i = 0; i < DIV_ROUND_UP(32, 32); i++) writel_relaxed(ptr[i], dist_base + GIC_DIST_ENABLE_SET + i * 4); - ptr = __this_cpu_ptr(gic_data[gic_nr].saved_ppi_conf); + ptr = raw_cpu_ptr(gic_data[gic_nr].saved_ppi_conf); for (i = 0; i < DIV_ROUND_UP(32, 16); i++) writel_relaxed(ptr[i], dist_base + GIC_DIST_CONFIG + i * 4); for (i = 0; i < DIV_ROUND_UP(32, 4); i++) - writel_relaxed(0xa0a0a0a0, dist_base + GIC_DIST_PRI + i * 4); + writel_relaxed(GICD_INT_DEF_PRI_X4, + dist_base + GIC_DIST_PRI + i * 4); - writel_relaxed(0xf0, cpu_base + GIC_CPU_PRIMASK); - writel_relaxed(1, cpu_base + GIC_CPU_CTRL); + /* + * Set all PPI and SGI interrupts to be group 1. + * + * If grouping is not available (not implemented or prohibited by + * security mode) these registers are read-as-zero/write-ignored. + */ + if (readl_relaxed(dist_base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1) { + writel_relaxed(0xfffffeff, dist_base + GIC_DIST_IGROUP + 0); + writel_relaxed(0xa0a0a000, dist_base + GIC_DIST_PRI + 8); + } + + writel_relaxed(GICC_INT_PRI_THRESHOLD, cpu_base + GIC_CPU_PRIMASK); + gic_cpu_if_up(); } static int gic_notifier(struct notifier_block *self, unsigned long cmd, void *v) @@ -600,10 +657,19 @@ static void __init gic_pm_init(struct gic_chip_data *gic) #endif #ifdef CONFIG_SMP +static bool sgi_is_nonsecure(int irq, struct gic_chip_data *gic) +{ + void __iomem *dist_base = gic_data_dist_base(gic); + /* FIXME: this should be done in a more generic way */ + return irq != 8 && readl_relaxed(dist_base + GIC_DIST_CTRL) & GICD_ENABLE_GRP1; +} + static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq) { - int cpu; + struct gic_chip_data *gic = &gic_data[0]; unsigned long flags, map = 0; + unsigned int softirq; + int cpu; raw_spin_lock_irqsave(&irq_controller_lock, flags); @@ -617,8 +683,14 @@ static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq) */ dmb(ishst); + softirq = map << 16 | irq; + + /* SATT only has effect if we are running in the secure world */ + if (sgi_is_nonsecure(irq, gic)) + softirq |= 0x8000; + /* this always happens on GIC0 */ - writel_relaxed(map << 16 | irq, gic_data_dist_base(&gic_data[0]) + GIC_DIST_SOFTINT); + writel_relaxed(softirq, gic_data_dist_base(gic) + GIC_DIST_SOFTINT); raw_spin_unlock_irqrestore(&irq_controller_lock, flags); }