diff mbox

[V2] PCI/portdrv: Fix MSI/MSI-X bug for PCIe port service drivers

Message ID 55c0f3e3-879e-054a-6c09-ddf75f0dee8b@huawei.com (mailing list archive)
State New, archived
Delegated to: Bjorn Helgaas
Headers show

Commit Message

Dongdong Liu Sept. 6, 2017, 3:36 a.m. UTC 
Hi Bjorn
Many thanks for your review.
在 2017/9/6 8:04, Bjorn Helgaas 写道:
> On Wed, Aug 30, 2017 at 07:09:35PM +0800, Dongdong Liu wrote:
>> Current code is broken as calling pci_free_irq_vectors()
>> invalidates the IRQ numbers returned before by pci_irq_vectors();
>> so we need to move all the assignment of the Linux IRQ numbers at
>> the bottom of the function.
>>
>> After removing and adding back the PCI root port device,
>> we see the PCIe port service drivers request irq failed.
>
> What exactly is the connection between the root port removal/addition
> and the request IRQ failure?  The current code does:
>
>   pci_alloc_irq_vectors(dev, ...)
>   irqs[x] = pci_irq_vector(dev, entry)
>   pci_free_irq_vectors(dev)
>   pci_alloc_irq_vectors(dev, ...)
>
> So I guess the problem is that the second pci_alloc_irq_vectors() call
> gets different vectors than the first one?

Yes, you are right.

[  477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[  477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI
[  477.149352] pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
[  477.193697] pcieport 0000:00:00.0: request AER IRQ 34 failed
[  477.211684] aer: probe of 0000:00:00.0:pcie002 failed with error -22

> How is this related to removal/addition?

-[0000:00]-+-00.0-[01-02]--+-00.0  Device 8086:10fb
            |               \-00.1  Device 8086:10fb

After we boot the os.
ixgbe 0000:01:00.0: irq 23-24.
ixgbe 0000:01:00.1: irq 25-26.
pcieport 0000:00:00.0:  irq [27-29].

After we remove the rp(echo 1 > /sys/devices/pci0000:00/0000:00:00.0/remove).
irq 23-29 are free, but the number of irqs does not have 32 (23-29 ) .
After we rescan the rp (echo 1 > /sys/bus/pci/rescan)
[  477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[  477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI

I debug this, the detail information, please see the attchment boot.log.
I have not found this.

>
>> pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
>> aer: probe of 0000:00:00.0:pcie002 failed with error -22
>> pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd-
>> PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
>> pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller
>> pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)
>> dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22
>> dpc: probe of 0000:00:00.0:pcie010 failed with error -22
>>
>> Cc: <stable@vger.kernel.org>
>> Fixes: 3674cc4 ("PCI/portdrv: Use pci_irq_alloc_vectors()")
>
> Please use a 12-character SHA1.
Ok, I will change it as below.
Fixes: 3674cc49da9a ("PCI/portdrv: Use pci_irq_alloc_vectors()")
>
>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
>> Signed-off-by: Gabriele Paoloni <gabriele.paoloni@huawei.com>
>> ---
>> v1->v2:
>> - Fix comments on PATCH v1.
>> - Simplify implementation.
>> ---
>>  drivers/pci/pcie/portdrv_core.c | 17 ++++++++++++-----
>>  1 file changed, 12 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
>> index 313a21d..89f4cf5 100644
>> --- a/drivers/pci/pcie/portdrv_core.c
>> +++ b/drivers/pci/pcie/portdrv_core.c
>> @@ -55,7 +55,8 @@ static void release_pcie_device(struct device *dev)
>>  static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)
>>  {
>>  	int nr_entries, entry, nvec = 0;
>> -
>> +	int i;
>> +	int idx[PCIE_PORT_DEVICE_MAXSERVICES];
>>  	/*
>>  	 * Allocate as many entries as the port wants, so that we can check
>>  	 * which of them will be useful.  Moreover, if nr_entries is correctly
>
> This is not so much a question about your patch, but about the whole
> interrupt vector allocation strategy in pcie_port_enable_irq_vec().
>
> The current code basically does this:
>
>   - allocate 32 vectors
>   - figure out vector used by PME and hotplug
>   - figure out vector used by AER
>   - figure out vector used by DPC
>   - free the 32 vectors we allocated
>   - allocate only as many vectors as we need
Correct, that is the pcie_port_enable_irq_vec() do now.
>
> What's the point of allocating those 32 vectors up front?  Why can't
> we first look up the interrupt message numbers used by each service
> (PME, hotplut, AER, DPC), figure out how many vectors we need,
> allocate them, and then fill in the slots in the irqs[] table?
Agree.
>
> For example, something like this:
>
>   if (mask & (PCIE_PORT_SERVICE_PME | PCIE_PORT_SERVICE_HP)) {
>     pcie_capability_read_word(dev, PCI_EXP_FLAGS, &reg16);
>     pme_msg = (reg16 & PCI_EXP_FLAGS_IRQ) >> 9;
>   }
>
>   if (mask & PCIE_PORT_SERVICE_AER) {
>     pci_read_config_dword(dev, dev->aer_cap + PCI_ERR_ROOT_STATUS, &reg32);
>     aer_msg = reg32 >> 27;
>   }
>
>   if (mask & PCIE_PORT_SERVICE_DPC) {
>     pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_DPC);
>     pci_read_config_word(dev, pos + PCI_EXP_DPC_CAP, &reg16);
>     dpc_msg = reg16 & 0x1f;
>   }
>
>   nvec = max(pme_msg, max(aer_msg, dpc_msg)) + 1;
>   pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI);
>
>   if (mask & PCIE_PORT_SERVICE_PME)
>     irqs[PCIE_PORT_SERVICE_PME_SHIFT] = pci_irq_vector(dev, pme_msg);
>   if (mask & PCIE_PORT_SERVICE_HP)
>     irqs[PCIE_PORT_SERVICE_HP_SHIFT] = pci_irq_vector(dev, pme_msg);
>   if (mask & PCIE_PORT_SERVICE_AER)
>     irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer_msg);
>   if (mask & PCIE_PORT_SERVICE_DPC)
>     irqs[PCIE_PORT_SERVICE_DPC_SHIFT] = pci_irq_vector(dev, dpc_msg);
>
> We currently only look at each service once, and what I'm proposing
> would require looking at each service twice (once to read the message
> number from the hardware and again (after allocating the vectors) to
> fill in the irqs[] table).  But I think it would be simpler overall.

It looks good to me.

Thanks,
Dongdong
[    0.000000] Booting Linux on physical CPU 0x30000
[    0.000000] Linux version 4.13.0-rc7-dirty (l00290354@linux-ioko) (gcc version 4.9.3 20150211 (prerelease) (20150316)) #49 SMP PREEMPT Wed Sep 6 10:29:35 CST 2017
[    0.000000] Boot CPU: AArch64 Processor [480fd010]
[    0.000000] earlycon: pl11 at MMIO32 0x0000000094080000 (options '')
[    0.000000] bootconsole [pl11] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by EDK II
[    0.000000] efi:  ACPI 2.0=0x39ce0000  MEMATTR=0x3e8c4018 
[    0.000000] cma: Reserved 16 MiB at 0x000000003b400000
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x0000000039CE0000 000024 (v02 HISI  )
[    0.000000] ACPI: XSDT 0x0000000039CD0000 00007C (v01 HISI   HIP08    00000000      01000013)
[    0.000000] ACPI: FACP 0x0000000039C10000 000114 (v06 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: DSDT 0x0000000039BB0000 000971 (v02 HISI   HIP08    00000000 INTL 20170119)
[    0.000000] ACPI: GTDT 0x0000000039C00000 000098 (v02 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: DBG2 0x0000000039BF0000 00005A (v00 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: MCFG 0x0000000039BE0000 00003C (v01 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: SPCR 0x0000000039BD0000 000050 (v02 HISI   HIP07    00000000 INTL 20151124)
[    0.000000] ACPI: APIC 0x0000000039BC0000 0000C0 (v04 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: IORT 0x0000000039BA0000 000120 (v00 HISI   D08      00000000 INTL 20170119)
[    0.000000] ACPI: HEST 0x0000000039B80000 00013C (v01 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: EINJ 0x0000000039B60000 000170 (v01 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: ERST 0x0000000031A90000 000230 (v01 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: BERT 0x0000000031A80000 008030 (v01 HISI   HIP08    00000000 INTL 20151124)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0x94080000,115200
[    0.000000] ACPI: NUMA: Failed to initialise from firmware
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000003fbfffff]
[    0.000000] NUMA: NODE_DATA [mem 0x3fbee500-0x3fbeffff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fbfffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000031a5ffff]
[    0.000000]   node   0: [mem 0x0000000031a60000-0x0000000031a9ffff]
[    0.000000]   node   0: [mem 0x0000000031aa0000-0x0000000039b5ffff]
[    0.000000]   node   0: [mem 0x0000000039b60000-0x0000000039b6ffff]
[    0.000000]   node   0: [mem 0x0000000039b70000-0x0000000039b79fff]
[    0.000000]   node   0: [mem 0x0000000039b7a000-0x0000000039b8ffff]
[    0.000000]   node   0: [mem 0x0000000039b90000-0x0000000039b91fff]
[    0.000000]   node   0: [mem 0x0000000039b92000-0x000000003a0cffff]
[    0.000000]   node   0: [mem 0x000000003a0d0000-0x000000003a0d1fff]
[    0.000000]   node   0: [mem 0x000000003a0d2000-0x000000003a0dbfff]
[    0.000000]   node   0: [mem 0x000000003a0dc000-0x000000003f0bffff]
[    0.000000]   node   0: [mem 0x000000003f0c0000-0x000000003f0effff]
[    0.000000]   node   0: [mem 0x000000003f0f0000-0x000000003fbfffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fbfffff]
[    0.000000] On node 0 totalpages: 261120
[    0.000000]   DMA zone: 4080 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261120 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from ACPI.
NOTICE:  [psci_smc_handler]:[397L] PSCI_VERSION CALL
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 24 pages/cpu @ffff80003fbd3000 s59264 r8192 d30848 u98304
[    0.000000] pcpu-alloc: s59264 r8192 d30848 u98304 alloc=24*4096
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 257040
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: earlycon=pl011,mmio32,0x94080000 initrd=filesystem.cpio.gz acpi=force loglevel=8
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory: 959336K/1044480K available (9148K kernel code, 1108K rwdata, 4136K rodata, 1088K init, 398K bss, 68760K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008970000   (  9152 KB)
[    0.000000]     .rodata : 0xffff000008970000 - 0xffff000008d80000   (  4160 KB)
[    0.000000]       .init : 0xffff000008d80000 - 0xffff000008e90000   (  1088 KB)
[    0.000000]       .data : 0xffff000008e90000 - 0xffff000008fa5200   (  1109 KB)
[    0.000000]        .bss : 0xffff000008fa5200 - 0xffff000009008c70   (   399 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e0000ff0000   (    15 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff80003fc00000   (  1020 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] ACPI: SRAT not present
[    0.000000] ITS [mem 0xa8100000-0xa811ffff]
[    0.000000] ITS@0x00000000a8100000: allocated 8192 Devices @3a100000 (indirect, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000000a8100000: allocated 1024 Virtual CPUs @3a0e4000 (flat, esz 16, psz 4K, shr 1)
[    0.000000] ITS@0x00000000a8100000: allocated 256 Interrupt Collections @3a0e8000 (flat, esz 16, psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x000000003a110000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 30000 region 0:0x00000000aa100000
[    0.000000] CPU0: using LPI pending table @0x000000003a120000
[    0.000000] arch_timer: cp15 timer(s) running at 6.25MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e6b, max_idle_ns: 440795202301 ns
[    0.000329] sched_clock: 56 bits at 6MHz, resolution 160ns, wraps every 4398046511040ns
[    0.022603] Console: colour dummy device 80x25
[    0.034500] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.50 BogoMIPS (lpj=25000)
[    0.053277] pid_max: default: 32768 minimum: 301
[    0.062980] ACPI: Core revision 20170531
[    0.119428] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    0.136157] Security Framework initialized
[    0.190025] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.225984] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.240352] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.253407] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.362723] ASID allocator initialised with 65536 entries
[    0.379364] Hierarchical SRCU implementation.
[    0.409156] PCI/MSI: ITS@0xa8100000 domain created
[    0.420450] Platform MSI: ITS@0xa8100000 domain created
[    0.437286] Remapping and enabling EFI services.
[    0.448258]   EFI remap 0x0000000031a60000 => 0000000020000000
[    0.459979]   EFI remap 0x0000000039c30000 => 0000000020020000
[    0.471578]   EFI remap 0x0000000039c80000 => 0000000020070000
[    0.483099]   EFI remap 0x0000000039cf0000 => 00000000200c0000
[    0.494579]   EFI remap 0x0000000039d40000 => 0000000020110000
[    0.506102]   EFI remap 0x0000000039d90000 => 0000000020160000
[    0.517611]   EFI remap 0x0000000039de0000 => 00000000201b0000
[    0.529248]   EFI remap 0x0000000039e30000 => 0000000020200000
[    0.540737]   EFI remap 0x0000000039e80000 => 0000000020250000
[    0.552216]   EFI remap 0x0000000039ed0000 => 00000000202a0000
[    0.563700]   EFI remap 0x0000000039f20000 => 00000000202f0000
[    0.575540]   EFI remap 0x0000000039f70000 => 0000000020340000
[    0.587015]   EFI remap 0x0000000039fe0000 => 00000000203b0000
[    0.598629]   EFI remap 0x000000003a030000 => 0000000020400000
[    0.610118]   EFI remap 0x000000003a080000 => 0000000020450000
[    0.621288]   EFI remap 0x000000003f0c0000 => 00000000204a0000
[    0.650692] smp: Bringing up secondary CPUs ...
[    0.659943] smp: Brought up 1 node, 1 CPU
[    0.667468] SMP: Total of 1 processors activated.
[    0.676346] CPU features: detected feature: GIC system register CPU interface
[    0.689110] CPU features: detected feature: Privileged Access Never
[    0.700384] CPU features: detected feature: User Access Override
[    0.711175] CPU features: detected feature: Virtualization Host Extensions
[    0.731364] CPU: All CPU(s) started at EL2
[    0.739396] alternatives: patching kernel code
[    0.920360] devtmpfs: initialized
[    0.990978] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.031879] futex hash table entries: 256 (order: 3, 32768 bytes)
[    1.100210] pinctrl core: initialized pinctrl subsystem
[    1.231764] random: fast init done
[    1.441138] DMI not present or invalid.
[    1.531527] NET: Registered protocol family 16
[    1.954694] cpuidle: using governor menu
[    1.997861] vdso: 2 pages (1 code @ ffff000008976000, 1 data @ ffff000008e95000)
[    2.014553] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    2.086616] DMA: preallocated 256 KiB pool for atomic allocations
[    2.169828] ACPI: bus type PCI registered
[    2.269948] Serial: AMBA PL011 UART driver
[    6.266490] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    6.442895] ACPI: Added _OSI(Module Device)
[    6.452727] ACPI: Added _OSI(Processor Device)
[    6.461662] ACPI: Added _OSI(3.0 _SCP Extensions)
[    6.470751] ACPI: Added _OSI(Processor Aggregator Device)
[    6.567598] ACPI: Interpreter enabled
[    6.575160] ACPI: Using GIC for interrupt routing
[    6.595078] ACPI: MCFG table detected, 1 entries
[    7.427562] ARMH0011:00: ttyAMA0 at MMIO 0x94080000 (irq = 5, base_baud = 0) is a SBSA
[    7.446220] console [ttyAMA0] enabled
[    7.446220] console [ttyAMA0] enabled
[    7.458472] bootconsole [pl11] disabled
[    7.458472] bootconsole [pl11] disabled
[    7.536276] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3f])
[    7.544988] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    7.584851] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability]
[    7.617867] acpi PNP0A08:00: [Firmware Bug]: ECAM area [mem 0xd0000000-0xd3ffffff] not reserved in ACPI namespace
[    7.630773] acpi PNP0A08:00: ECAM at [mem 0xd0000000-0xd3ffffff] for [bus 00-3f]
[    7.645534] Remapped I/O 0x00000000e0000000 to [io  0x0000-0xffff window]
[    7.669434] PCI host bridge to bus 0000:00
[    7.682031] pci_bus 0000:00: root bus resource [mem 0x80000000000-0x83fffffffff pref window]
[    7.700166] pci_bus 0000:00: root bus resource [mem 0xe0010000-0xe7ffffff window]
[    7.709762] pci_bus 0000:00: root bus resource [io  0x0000-0xffff window]
[    7.718134] pci_bus 0000:00: root bus resource [bus 00-3f]
[    7.725526] pci_bus 0000:00: scanning bus
[    7.732374] pci 0000:00:00.0: [19e5:a120] type 01 class 0x060400
[    7.748480] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    7.756070] pci 0000:00:00.0: PME# disabled
[    7.781356] pci 0000:00:08.0: [19e5:a120] type 01 class 0x060400
[    7.809322] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[    7.817775] pci 0000:00:08.0: PME# disabled
[    7.842221] pci_bus 0000:00: fixups for bus
[    7.857795] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 0
[    7.881935] pci_bus 0000:01: scanning bus
[    7.894711] pci 0000:01:00.0: [8086:10fb] type 00 class 0x020000
[    7.905129] pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x84
[    7.914365] pci 0000:01:00.0: reg 0x10: [mem 0x80000c00000-0x80000ffffff 64bit pref]
[    7.923845] pci 0000:01:00.0: reg 0x18: [io  0x0000-0x001f]
[    7.931977] pci 0000:01:00.0: reg 0x20: [mem 0x80001404000-0x80001407fff 64bit pref]
[    7.941331] pci 0000:01:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    7.954316] pci 0000:01:00.0: PME# supported from D0 D3hot
[    7.960786] pci 0000:01:00.0: PME# disabled
[    7.967206] pci 0000:01:00.0: reg 0x184: [mem 0x80001304000-0x80001307fff 64bit pref]
[    7.975904] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x80001304000-0x80001403fff 64bit pref] (contains BAR0 for 64 VFs)
[    7.988004] pci 0000:01:00.0: reg 0x190: [mem 0x80001204000-0x80001207fff 64bit pref]
[    7.996493] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x80001204000-0x80001303fff 64bit pref] (contains BAR3 for 64 VFs)
[    8.036499] pci 0000:01:00.1: [8086:10fb] type 00 class 0x020000
[    8.055046] pci 0000:01:00.1: calling quirk_f0_vpd_link+0x0/0x84
[    8.064331] pci 0000:01:00.1: reg 0x10: [mem 0x80000800000-0x80000bfffff 64bit pref]
[    8.074492] pci 0000:01:00.1: reg 0x18: [io  0x0000-0x001f]
[    8.082047] pci 0000:01:00.1: reg 0x20: [mem 0x80001200000-0x80001203fff 64bit pref]
[    8.091130] pci 0000:01:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    8.103548] pci 0000:01:00.1: PME# supported from D0 D3hot
[    8.109878] pci 0000:01:00.1: PME# disabled
[    8.115950] pci 0000:01:00.1: reg 0x184: [mem 0x80001100000-0x80001103fff 64bit pref]
[    8.124516] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x80001100000-0x800011fffff 64bit pref] (contains BAR0 for 64 VFs)
[    8.136598] pci 0000:01:00.1: reg 0x190: [mem 0x80001000000-0x80001003fff 64bit pref]
[    8.145071] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x80001000000-0x800010fffff 64bit pref] (contains BAR3 for 64 VFs)
[    8.186151] pci_bus 0000:01: fixups for bus
[    8.200134] pci_bus 0000:01: bus scan returning with max=01
[    8.209477] pci 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 0
[    8.231444] pci_bus 0000:03: scanning bus
[    8.244266] pci 0000:03:00.0: [8086:1521] type 00 class 0x020000
[    8.255004] pci 0000:03:00.0: calling quirk_f0_vpd_link+0x0/0x84
[    8.264702] pci 0000:03:00.0: reg 0x10: [mem 0xe0300000-0xe03fffff]
[    8.273645] pci 0000:03:00.0: reg 0x1c: [mem 0xe040c000-0xe040ffff]
[    8.282075] pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    8.295226] pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
[    8.302214] pci 0000:03:00.0: PME# disabled
[    8.308536] pci 0000:03:00.0: reg 0x184: [mem 0x800015e0000-0x800015e3fff 64bit pref]
[    8.317179] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0x800015e0000-0x800015fffff 64bit pref] (contains BAR0 for 8 VFs)
[    8.329199] pci 0000:03:00.0: reg 0x190: [mem 0x800015c0000-0x800015c3fff 64bit pref]
[    8.337636] pci 0000:03:00.0: VF(n) BAR3 space: [mem 0x800015c0000-0x800015dffff 64bit pref] (contains BAR3 for 8 VFs)
[    8.368526] pci 0000:03:00.1: [8086:1521] type 00 class 0x020000
[    8.386885] pci 0000:03:00.1: calling quirk_f0_vpd_link+0x0/0x84
[    8.395857] pci 0000:03:00.1: reg 0x10: [mem 0xe0200000-0xe02fffff]
[    8.404993] pci 0000:03:00.1: reg 0x1c: [mem 0xe0408000-0xe040bfff]
[    8.413251] pci 0000:03:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    8.426293] pci 0000:03:00.1: PME# supported from D0 D3hot D3cold
[    8.433273] pci 0000:03:00.1: PME# disabled
[    8.439321] pci 0000:03:00.1: reg 0x184: [mem 0x800015a0000-0x800015a3fff 64bit pref]
[    8.447914] pci 0000:03:00.1: VF(n) BAR0 space: [mem 0x800015a0000-0x800015bffff 64bit pref] (contains BAR0 for 8 VFs)
[    8.459963] pci 0000:03:00.1: reg 0x190: [mem 0x80001580000-0x80001583fff 64bit pref]
[    8.468410] pci 0000:03:00.1: VF(n) BAR3 space: [mem 0x80001580000-0x8000159ffff 64bit pref] (contains BAR3 for 8 VFs)
[    8.499856] pci 0000:03:00.2: [8086:1521] type 00 class 0x020000
[    8.518734] pci 0000:03:00.2: calling quirk_f0_vpd_link+0x0/0x84
[    8.527539] pci 0000:03:00.2: reg 0x10: [mem 0xe0100000-0xe01fffff]
[    8.536650] pci 0000:03:00.2: reg 0x1c: [mem 0xe0404000-0xe0407fff]
[    8.550703] pci 0000:03:00.2: PME# supported from D0 D3hot D3cold
[    8.557839] pci 0000:03:00.2: PME# disabled
[    8.563852] pci 0000:03:00.2: reg 0x184: [mem 0x80001560000-0x80001563fff 64bit pref]
[    8.572450] pci 0000:03:00.2: VF(n) BAR0 space: [mem 0x80001560000-0x8000157ffff 64bit pref] (contains BAR0 for 8 VFs)
[    8.584466] pci 0000:03:00.2: reg 0x190: [mem 0x80001540000-0x80001543fff 64bit pref]
[    8.592948] pci 0000:03:00.2: VF(n) BAR3 space: [mem 0x80001540000-0x8000155ffff 64bit pref] (contains BAR3 for 8 VFs)
[    8.623880] pci 0000:03:00.3: [8086:1521] type 00 class 0x020000
[    8.643212] pci 0000:03:00.3: calling quirk_f0_vpd_link+0x0/0x84
[    8.652565] pci 0000:03:00.3: reg 0x10: [mem 0xe0000000-0xe00fffff]
[    8.661184] pci 0000:03:00.3: reg 0x1c: [mem 0xe0400000-0xe0403fff]
[    8.675795] pci 0000:03:00.3: PME# supported from D0 D3hot D3cold
[    8.683456] pci 0000:03:00.3: PME# disabled
[    8.689539] pci 0000:03:00.3: reg 0x184: [mem 0x80001520000-0x80001523fff 64bit pref]
[    8.698171] pci 0000:03:00.3: VF(n) BAR0 space: [mem 0x80001520000-0x8000153ffff 64bit pref] (contains BAR0 for 8 VFs)
[    8.710194] pci 0000:03:00.3: reg 0x190: [mem 0x80001500000-0x80001503fff 64bit pref]
[    8.718661] pci 0000:03:00.3: VF(n) BAR3 space: [mem 0x80001500000-0x8000151ffff 64bit pref] (contains BAR3 for 8 VFs)
[    8.751770] pci_bus 0000:03: fixups for bus
[    8.767326] pci_bus 0000:03: bus scan returning with max=03
[    8.775771] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 1
[    8.784088] pci 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 1
[    8.792775] pci_bus 0000:00: bus scan returning with max=04
[    8.802531] pci 0000:00:00.0: BAR 14: assigned [mem 0xe0400000-0xe0bfffff]
[    8.811578] pci 0000:00:00.0: BAR 15: assigned [mem 0x80000000000-0x80000dfffff 64bit pref]
[    8.820739] pci 0000:00:08.0: BAR 14: assigned [mem 0xe0c00000-0xe12fffff]
[    8.828338] pci 0000:00:08.0: BAR 15: assigned [mem 0x80000e00000-0x80000ffffff 64bit pref]
[    8.837354] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    8.844176] pci 0000:00:08.0: BAR 13: assigned [io  0x2000-0x2fff]
[    8.851420] pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000000-0x800003fffff 64bit pref]
[    8.860790] pci 0000:01:00.0: BAR 6: assigned [mem 0xe0400000-0xe07fffff pref]
[    8.868711] pci 0000:01:00.1: BAR 0: assigned [mem 0x80000400000-0x800007fffff 64bit pref]
[    8.878024] pci 0000:01:00.1: BAR 6: assigned [mem 0xe0800000-0xe0bfffff pref]
[    8.885861] pci 0000:01:00.0: BAR 4: assigned [mem 0x80000800000-0x80000803fff 64bit pref]
[    8.895108] pci 0000:01:00.0: BAR 7: assigned [mem 0x80000804000-0x80000903fff 64bit pref]
[    8.904170] pci 0000:01:00.0: BAR 10: assigned [mem 0x80000904000-0x80000a03fff 64bit pref]
[    8.913312] pci 0000:01:00.1: BAR 4: assigned [mem 0x80000a04000-0x80000a07fff 64bit pref]
[    8.922554] pci 0000:01:00.1: BAR 7: assigned [mem 0x80000a08000-0x80000b07fff 64bit pref]
[    8.931594] pci 0000:01:00.1: BAR 10: assigned [mem 0x80000b08000-0x80000c07fff 64bit pref]
[    8.940710] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    8.947512] pci 0000:01:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    8.956549] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[    8.962641] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    8.969541] pci 0000:00:00.0:   bridge window [mem 0xe0400000-0xe0bfffff]
[    8.977069] pci 0000:00:00.0:   bridge window [mem 0x80000000000-0x80000dfffff 64bit pref]
[    8.986830] pci 0000:03:00.0: BAR 0: assigned [mem 0xe0c00000-0xe0cfffff]
[    8.994486] pci 0000:03:00.0: BAR 6: assigned [mem 0xe0d00000-0xe0dfffff pref]
[    9.002391] pci 0000:03:00.1: BAR 0: assigned [mem 0xe0e00000-0xe0efffff]
[    9.009901] pci 0000:03:00.1: BAR 6: assigned [mem 0xe0f00000-0xe0ffffff pref]
[    9.017770] pci 0000:03:00.2: BAR 0: assigned [mem 0xe1000000-0xe10fffff]
[    9.025268] pci 0000:03:00.3: BAR 0: assigned [mem 0xe1100000-0xe11fffff]
[    9.032763] pci 0000:03:00.0: BAR 3: assigned [mem 0xe1200000-0xe1203fff]
[    9.040281] pci 0000:03:00.0: BAR 7: assigned [mem 0x80000e00000-0x80000e1ffff 64bit pref]
[    9.049319] pci 0000:03:00.0: BAR 10: assigned [mem 0x80000e20000-0x80000e3ffff 64bit pref]
[    9.058434] pci 0000:03:00.1: BAR 3: assigned [mem 0xe1204000-0xe1207fff]
[    9.065936] pci 0000:03:00.1: BAR 7: assigned [mem 0x80000e40000-0x80000e5ffff 64bit pref]
[    9.074953] pci 0000:03:00.1: BAR 10: assigned [mem 0x80000e60000-0x80000e7ffff 64bit pref]
[    9.084074] pci 0000:03:00.2: BAR 3: assigned [mem 0xe1208000-0xe120bfff]
[    9.091562] pci 0000:03:00.2: BAR 7: assigned [mem 0x80000e80000-0x80000e9ffff 64bit pref]
[    9.100587] pci 0000:03:00.2: BAR 10: assigned [mem 0x80000ea0000-0x80000ebffff 64bit pref]
[    9.109690] pci 0000:03:00.3: BAR 3: assigned [mem 0xe120c000-0xe120ffff]
[    9.117187] pci 0000:03:00.3: BAR 7: assigned [mem 0x80000ec0000-0x80000edffff 64bit pref]
[    9.126233] pci 0000:03:00.3: BAR 10: assigned [mem 0x80000ee0000-0x80000efffff 64bit pref]
[    9.135511] pci 0000:00:08.0: PCI bridge to [bus 03-04]
[    9.141357] pci 0000:00:08.0:   bridge window [io  0x2000-0x2fff]
[    9.148217] pci 0000:00:08.0:   bridge window [mem 0xe0c00000-0xe12fffff]
[    9.155707] pci 0000:00:08.0:   bridge window [mem 0x80000e00000-0x80000ffffff 64bit pref]
[    9.560015] vgaarb: loaded
[    9.640424] SCSI subsystem initialized
[    9.698093] libata version 3.00 loaded.
[    9.780441] ACPI: bus type USB registered
[    9.826557] usbcore: registered new interface driver usbfs
[    9.854891] usbcore: registered new interface driver hub
[    9.878112] usbcore: registered new device driver usb
[   10.024058] pps_core: LinuxPPS API ver. 1 registered
[   10.040988] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[   10.058492] PTP clock support registered
[   10.136679] dmi: Firmware registration failed.
[   10.150629] Registered efivars operations
[   10.218533] Advanced Linux Sound Architecture Driver Initialized.
[   10.372769] clocksource: Switched to clocksource arch_sys_counter
[   10.396969] VFS: Disk quotas dquot_6.6.0
[   10.406240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[   10.454282] pnp: PnP ACPI init
[   10.496852] pnp: PnP ACPI: found 0 devices
[   12.594530] NET: Registered protocol family 2
[   12.641412] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[   12.653888] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[   12.671318] TCP: Hash tables configured (established 8192 bind 8192)
[   12.687407] UDP hash table entries: 512 (order: 2, 16384 bytes)
[   12.696177] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[   12.713559] NET: Registered protocol family 1
[   12.751919] RPC: Registered named UNIX socket transport module.
[   12.760732] RPC: Registered udp transport module.
[   12.766108] RPC: Registered tcp transport module.
[   12.772643] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   12.783117] pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[   12.791253] pci 0000:01:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[   12.798827] pci 0000:03:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[   12.805985] PCI: CLS mismatch (64 != 128), using 128 bytes
[   12.812477] pci 0000:03:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[   12.819852] pci 0000:03:00.2: calling quirk_e100_interrupt+0x0/0x1c0
[   12.827500] pci 0000:03:00.3: calling quirk_e100_interrupt+0x0/0x1c0
[   12.853503] Unpacking initramfs...
[   93.115023] Initramfs unpacking failed: junk in compressed archive
[   94.520143] Freeing initrd memory: 28172K
[   94.541265] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 13 counters available
[   94.551107] kvm [1]: 16-bit VMID
[   94.555180] kvm [1]: VHE mode initialized successfully
[   94.561474] kvm [1]: GICv3: no GICV resource entry
[   94.566985] kvm [1]: disabling GICv2 emulation
[   94.573285] kvm [1]: GIC system register CPU interface enabled
[   94.583554] kvm [1]: vgic interrupt IRQ1
[   94.589094] kvm [1]: virtual timer IRQ3
[   94.980154] audit: initializing netlink subsys (disabled)
[   95.009820] audit: type=2000 audit(92.583:1): state=initialized audit_enabled=0 res=1
[   95.127062] workingset: timestamp_bits=44 max_order=18 bucket_order=0
[   96.798116] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[   96.969554] NFS: Registering the id_resolver key type
[   96.979188] Key type id_resolver registered
[   96.985193] Key type id_legacy registered
[   96.991028] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[   97.018200] 9p: Installing v9fs 9p2000 file system support
[   97.330580] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[   97.351086] io scheduler noop registered
[   97.403202] io scheduler cfq registered (default)
[   97.418177] io scheduler mq-deadline registered
[   97.426968] io scheduler kyber registered
[   98.764128] acpi-ged ACPI0013:00: GED listening GSI 129 @ IRQ 6
[  100.898280] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[  101.310451] SuperH (H)SCI(F) driver initialized
[  101.381953] msm_serial: driver initialized
[  101.533722] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0
[  101.545745] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit (features 0x00000fef)
[  101.578260] arm-smmu-v3 arm-smmu-v3.0.auto: failed to allocate MSIs
[  101.622191] cacheinfo: Unable to detect cache hierarchy for CPU 0
[  102.507569] loop: module loaded
[  102.734046] hisi_sas: driver version v1.6
[  103.361973] libphy: Fixed MDIO Bus: probed
[  103.582183] tun: Universal TUN/TAP device driver, 1.6
[  103.811192] libphy: Hisilicon MII Bus: probed
[  103.886133] hns-nic HISI00C2:00: not find ae-handle
[  103.902399] hns-nic: probe of HISI00C2:00 failed with error -22
[  103.920648] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  103.937988] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  103.961915] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[  103.981971] igb: Copyright (c) 2007-2014 Intel Corporation.
[  104.026744] iommu: Adding device 0000:03:00.0 to group 0
[  104.039941] igb 0000:03:00.0: runtime IRQ mapping not provided by arch
[  104.060977] igb 0000:03:00.0: enabling device (0000 -> 0002)
[  104.069357] igb 0000:03:00.0: enabling bus mastering
[  104.107285] igb 0000:03:00.0: irq 9 for MSI
[  104.137704] igb 0000:03:00.0: irq 9 for MSI
[  104.143757] igb 0000:03:00.0: irq 10 for MSI
[  104.149258] igb 0000:03:00.0: irq 11 for MSI
[  104.762718] igb 0000:03:00.0: added PHC on eth0
[  104.778186] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[  104.790609] igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:61
[  104.800924] igb 0000:03:00.0: eth0: PBA No: 106300-000
[  104.807883] igb 0000:03:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[  104.839294] iommu: Adding device 0000:03:00.1 to group 1
[  104.853148] igb 0000:03:00.1: runtime IRQ mapping not provided by arch
[  104.873162] igb 0000:03:00.1: enabling device (0000 -> 0002)
[  104.881690] igb 0000:03:00.1: enabling bus mastering
[  104.919958] igb 0000:03:00.1: irq 13 for MSI
[  104.947522] igb 0000:03:00.1: irq 13 for MSI
[  104.954060] igb 0000:03:00.1: irq 14 for MSI
[  104.959320] igb 0000:03:00.1: irq 15 for MSI
[  105.493974] igb 0000:03:00.1: added PHC on eth1
[  105.507684] igb 0000:03:00.1: Intel(R) Gigabit Ethernet Network Connection
[  105.520897] igb 0000:03:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:62
[  105.530370] igb 0000:03:00.1: eth1: PBA No: 106300-000
[  105.536968] igb 0000:03:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[  105.567646] iommu: Adding device 0000:03:00.2 to group 2
[  105.581190] igb 0000:03:00.2: runtime IRQ mapping not provided by arch
[  105.601115] igb 0000:03:00.2: enabling device (0000 -> 0002)
[  105.609543] igb 0000:03:00.2: enabling bus mastering
[  105.646074] igb 0000:03:00.2: irq 17 for MSI
[  105.674381] igb 0000:03:00.2: irq 17 for MSI
[  105.683278] igb 0000:03:00.2: irq 18 for MSI
[  105.688778] igb 0000:03:00.2: irq 19 for MSI
[  106.217634] igb 0000:03:00.2: added PHC on eth2
[  106.233860] igb 0000:03:00.2: Intel(R) Gigabit Ethernet Network Connection
[  106.243657] igb 0000:03:00.2: eth2: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:63
[  106.254372] igb 0000:03:00.2: eth2: PBA No: 106300-000
[  106.263316] igb 0000:03:00.2: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[  106.295470] iommu: Adding device 0000:03:00.3 to group 3
[  106.308924] igb 0000:03:00.3: runtime IRQ mapping not provided by arch
[  106.321877] pci 0000:00:08.0: can't derive routing for PCI INT D
[  106.328793] igb 0000:03:00.3: PCI INT D: no GSI
[  106.334925] igb 0000:03:00.3: enabling device (0000 -> 0002)
[  106.342987] igb 0000:03:00.3: enabling bus mastering
[  106.381164] igb 0000:03:00.3: irq 20 for MSI
[  106.395108] pci 0000:00:08.0: can't derive routing for PCI INT D
[  106.402247] igb 0000:03:00.3: PCI INT D: no GSI
[  106.423980] igb 0000:03:00.3: irq 20 for MSI
[  106.430404] igb 0000:03:00.3: irq 21 for MSI
[  106.435604] igb 0000:03:00.3: irq 22 for MSI
[  106.969195] igb 0000:03:00.3: added PHC on eth3
[  106.986599] igb 0000:03:00.3: Intel(R) Gigabit Ethernet Network Connection
[  106.997192] igb 0000:03:00.3: eth3: (PCIe:2.5Gb/s:Width x4) 60:08:10:b9:d8:64
[  107.006052] igb 0000:03:00.3: eth3: PBA No: 106300-000
[  107.014340] igb 0000:03:00.3: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[  107.034774] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[  107.051759] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[  107.073731] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[  107.093439] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[  107.147616] iommu: Adding device 0000:01:00.0 to group 4
[  107.161275] ixgbe 0000:01:00.0: runtime IRQ mapping not provided by arch
[  107.176976] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[  107.185395] ixgbe 0000:01:00.0: enabling bus mastering
[  108.634988] ixgbe 0000:01:00.0: irq 23 for MSI
[  108.643436] ixgbe 0000:01:00.0: irq 24 for MSI
[  108.651323] ixgbe 0000:01:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  108.663349] ixgbe 0000:01:00.0: PCI Express bandwidth of 8GT/s available
[  108.670828] ixgbe 0000:01:00.0: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[  108.678675] ixgbe 0000:01:00.0: This is not sufficient for optimal performance of this card.
[  108.687764] ixgbe 0000:01:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[  108.697626] ixgbe 0000:01:00.0: A slot with more lanes and/or higher speed is suggested.
[  108.706705] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[  108.713609] ixgbe 0000:01:00.0: 9c:37:f4:90:cd:23
[  110.602795] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[  110.637473] iommu: Adding device 0000:01:00.1 to group 5
[  110.650676] ixgbe 0000:01:00.1: runtime IRQ mapping not provided by arch
[  110.665725] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[  110.674161] ixgbe 0000:01:00.1: enabling bus mastering
[  111.150789] ixgbe 0000:01:00.1: irq 25 for MSI
[  111.159276] ixgbe 0000:01:00.1: irq 26 for MSI
[  111.167407] ixgbe 0000:01:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  111.179704] ixgbe 0000:01:00.1: PCI Express bandwidth of 8GT/s available
[  111.187549] ixgbe 0000:01:00.1: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[  111.195360] ixgbe 0000:01:00.1: This is not sufficient for optimal performance of this card.
[  111.204463] ixgbe 0000:01:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[  111.214037] ixgbe 0000:01:00.1: A slot with more lanes and/or higher speed is suggested.
[  111.223129] ixgbe 0000:01:00.1: MAC: 2, PHY: 17, SFP+: 10, PBA No: FFFFFF-0FF
[  111.230947] ixgbe 0000:01:00.1: 9c:37:f4:90:cd:24
[  113.207022] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[  113.322096] sky2: driver version 1.30
[  113.462071] VFIO - User Level meta-driver version: 0.3
[  113.771122] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[  113.787380] ehci-pci: EHCI PCI platform driver
[  113.806857] ehci-platform: EHCI generic platform driver
[  113.839730] ehci-orion: EHCI orion driver
[  113.870892] ehci-exynos: EHCI EXYNOS driver
[  113.901359] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[  113.935225] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[  113.954224] ohci-pci: OHCI PCI platform driver
[  113.975549] ohci-platform: OHCI generic platform driver
[  114.010294] ohci-exynos: OHCI EXYNOS driver
[  114.141756] usbcore: registered new interface driver usb-storage
[  114.773837] i2c /dev entries driver
[  115.641863] sdhci: Secure Digital Host Controller Interface driver
[  115.663068] sdhci: Copyright(c) Pierre Ossman
[  115.729786] Synopsys Designware Multimedia Card Interface Driver
[  115.918274] sdhci-pltfm: SDHCI platform and OF driver helper
[  116.189809] ledtrig-cpu: registered to indicate activity on CPUs
[  116.554506] usbcore: registered new interface driver usbhid
[  116.569432] usbhid: USB HID core driver
[  116.995307] NET: Registered protocol family 17
[  117.008752] 9pnet: Installing 9P2000 support
[  117.025356] Key type dns_resolver registered
[  117.170735] registered taskstats version 1
[  117.231555] iommu: Adding device 0000:00:00.0 to group 6
[  117.245350] pcieport 0000:00:00.0: runtime IRQ mapping not provided by arch
[  117.297232] pcieport 0000:00:00.0: irq [27-58] for MSI
[  117.361464] pcieport 0000:00:00.0: irq [27-29] for MSI
[  117.389455] pcieport 0000:00:00.0: Signaling PME with IRQ 27
[  117.446120] pcieport 0000:00:00.0: AER enabled with IRQ 28
[  117.475969] pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[  117.506102] pci_bus 0000:01: dev 00, created physical slot 0
[  117.592065] dpc 0000:00:00.0:pcie010: DPC error containment capabilities: Int Msg #2, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
[  117.632624] iommu: Adding device 0000:00:08.0 to group 7
[  117.645307] pcieport 0000:00:08.0: runtime IRQ mapping not provided by arch
[  117.693898] pcieport 0000:00:08.0: irq [30-61] for MSI
[  117.757333] pcieport 0000:00:08.0: irq [30-32] for MSI
[  117.785514] pcieport 0000:00:08.0: Signaling PME with IRQ 30
[  117.833459] pcieport 0000:00:08.0: AER enabled with IRQ 31
[  117.870840] pciehp 0000:00:08.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[  117.898814] pci_bus 0000:03: dev 00, created physical slot 0-1
[  117.966720] dpc 0000:00:08.0:pcie010: DPC error containment capabilities: Int Msg #2, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
[  118.017679] hctosys: unable to open rtc device (rtc0)
[  118.609748] ALSA device list:
[  118.614432]   No soundcards found.
[  118.691983] Freeing unused kernel memory: 1088K
root@(none)$ 
root@(none)$ cd /home/
root@(none)$ ./lspci -tv
-[0000:00]-+-00.0-[01-02]--+-00.0  Device 8086:10fb
           |               \-00.1  Device 8086:10fb
           \-08.0-[03-04]--+-00.0  Device 8086:1521
                           +-00.1  Device 8086:1521
                           +-00.2  Device 8086:1521
                           \-00.3  Device 8086:1521
root@(none)$ echo1  1 > /sys/devices/pci0000:00/0000:00:00.0/remove  
[  447.630066] ixgbe 0000:01:00.1: PME# disabled
[  449.503490] ixgbe 0000:01:00.1: complete
[  449.515380] ixgbe 0000:01:00.0: PME# disabled
[  451.323235] ixgbe 0000:01:00.0: complete
[  451.335069] pcieport 0000:00:00.0: PME# disabled
[  451.398016] pci_bus 0000:01: dev 00, dec refcount to 0
[  451.414247] pci_bus 0000:01: dev 00, released physical slot 0
[  451.523677] iommu: Removing device 0000:01:00.0 from group 4
[  451.563037] iommu: Removing device 0000:01:00.1 from group 5
[  451.596134] pci_bus 0000:01: busn_res: [bus 01-02] is released
[  451.643249] iommu: Removing device 0000:00:00.0 from group 6
root@(none)$ echo 1 > /sys/bus/pci/rescan 
[  476.261084] pci_bus 0000:00: scanning bus
[  476.268738] pci 0000:00:00.0: [19e5:a120] type 01 class 0x060400
[  476.285212] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  476.292934] pci 0000:00:00.0: PME# disabled
[  476.323146] pcieport 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 0
[  476.342380] pci_bus 0000:03: scanning bus
[  476.355514] pci_bus 0000:03: bus scan returning with max=03
[  476.363649] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 0
[  476.389654] pci_bus 0000:01: scanning bus
[  476.403577] pci 0000:01:00.0: [8086:10fb] type 00 class 0x020000
[  476.418389] pci 0000:01:00.0: calling quirk_f0_vpd_link+0x0/0x84
[  476.427450] pci 0000:01:00.0: reg 0x10: [mem 0x80000000000-0x800003fffff 64bit pref]
[  476.438299] pci 0000:01:00.0: reg 0x18: [io  0x1000-0x101f]
[  476.446411] pci 0000:01:00.0: reg 0x20: [mem 0x80000800000-0x80000803fff 64bit pref]
[  476.456342] pci 0000:01:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[  476.469753] pci 0000:01:00.0: PME# supported from D0 D3hot
[  476.476429] pci 0000:01:00.0: PME# disabled
[  476.482955] pci 0000:01:00.0: reg 0x184: [mem 0x80000804000-0x80000807fff 64bit pref]
[  476.491789] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x80000804000-0x80000903fff 64bit pref] (contains BAR0 for 64 VFs)
[  476.504795] pci 0000:01:00.0: reg 0x190: [mem 0x80000904000-0x80000907fff 64bit pref]
[  476.513379] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0x80000904000-0x80000a03fff 64bit pref] (contains BAR3 for 64 VFs)
[  476.555511] pci 0000:01:00.1: [8086:10fb] type 00 class 0x020000
[  476.578766] pci 0000:01:00.1: calling quirk_f0_vpd_link+0x0/0x84
[  476.589784] pci 0000:01:00.1: reg 0x10: [mem 0x80000400000-0x800007fffff 64bit pref]
[  476.599717] pci 0000:01:00.1: reg 0x18: [io  0x1020-0x103f]
[  476.608373] pci 0000:01:00.1: reg 0x20: [mem 0x80000a04000-0x80000a07fff 64bit pref]
[  476.617225] pci 0000:01:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[  476.629753] pci 0000:01:00.1: PME# supported from D0 D3hot
[  476.636368] pci 0000:01:00.1: PME# disabled
[  476.642766] pci 0000:01:00.1: reg 0x184: [mem 0x80000a08000-0x80000a0bfff 64bit pref]
[  476.651533] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0x80000a08000-0x80000b07fff 64bit pref] (contains BAR0 for 64 VFs)
[  476.664255] pci 0000:01:00.1: reg 0x190: [mem 0x80000b08000-0x80000b0bfff 64bit pref]
[  476.672824] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0x80000b08000-0x80000c07fff 64bit pref] (contains BAR3 for 64 VFs)
[  476.717233] pci_bus 0000:01: fixups for bus
[  476.734843] pci_bus 0000:01: bus scan returning with max=01
[  476.743552] pcieport 0000:00:08.0: scanning [bus 03-04] behind bridge, pass 1
[  476.753981] pci 0000:00:00.0: scanning [bus 01-02] behind bridge, pass 1
[  476.762484] pci_bus 0000:00: bus scan returning with max=04
[  476.771676] pci 0000:00:00.0: bridge window [mem 0x00200000-0x00bfffff 64bit pref] to [bus 01-02] add_size 400000 add_align 200000
[  476.786395] pci 0000:00:00.0: bridge window [mem 0x00400000-0x00bfffff] to [bus 01-02] add_size 400000 add_align 400000
[  476.798969] pci 0000:00:00.0: BAR 14: assigned [mem 0xe1400000-0xe1ffffff]
[  476.806837] pci 0000:00:00.0: BAR 15: assigned [mem 0x80000000000-0x80000dfffff 64bit pref]
[  476.816253] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[  476.825090] pci 0000:01:00.0: BAR 0: assigned [mem 0x80000000000-0x800003fffff 64bit pref]
[  476.834626] pci 0000:01:00.0: BAR 6: assigned [mem 0xe1400000-0xe17fffff pref]
[  476.842698] pci 0000:01:00.1: BAR 0: assigned [mem 0x80000400000-0x800007fffff 64bit pref]
[  476.852353] pci 0000:01:00.1: BAR 6: assigned [mem 0xe1800000-0xe1bfffff pref]
[  476.860311] pci 0000:01:00.0: BAR 4: assigned [mem 0x80000800000-0x80000803fff 64bit pref]
[  476.869637] pci 0000:01:00.0: BAR 7: assigned [mem 0x80000804000-0x80000903fff 64bit pref]
[  476.878800] pci 0000:01:00.0: BAR 10: assigned [mem 0x80000904000-0x80000a03fff 64bit pref]
[  476.888354] pci 0000:01:00.1: BAR 4: assigned [mem 0x80000a04000-0x80000a07fff 64bit pref]
[  476.897688] pci 0000:01:00.1: BAR 7: assigned [mem 0x80000a08000-0x80000b07fff 64bit pref]
[  476.906856] pci 0000:01:00.1: BAR 10: assigned [mem 0x80000b08000-0x80000c07fff 64bit pref]
[  476.916342] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[  476.923231] pci 0000:01:00.1: BAR 2: assigned [io  0x1020-0x103f]
[  476.930426] pci 0000:00:00.0: PCI bridge to [bus 01-02]
[  476.936429] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[  476.943383] pci 0000:00:00.0:   bridge window [mem 0xe1400000-0xe1ffffff]
[  476.950994] pci 0000:00:00.0:   bridge window [mem 0x80000000000-0x80000dfffff 64bit pref]
[  476.989468] iommu: Adding device 0000:00:00.0 to group 4
[  477.001974] pcieport 0000:00:00.0: runtime IRQ mapping not provided by arch
[  477.055527] pcieport 0000:00:00.0: irq [33-64] for MSI
[  477.122631] pcieport 0000:00:00.0: irq [23-25] for MSI
[  477.149352] pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22
[  477.193697] pcieport 0000:00:00.0: request AER IRQ 34 failed
[  477.211684] aer: probe of 0000:00:00.0:pcie002 failed with error -22
[  477.237964] pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+
[  477.267030] pci_bus 0000:01: dev 00, created physical slot 0
[  477.286902] pciehp 0000:00:00.0:pcie004: Cannot get irq 33 for the hotplug controller
[  477.305707] pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)
[  477.317644] pci_bus 0000:01: dev 00, dec refcount to 0
[  477.332681] pci_bus 0000:01: dev 00, released physical slot 0
[  477.392148] dpc 0000:00:00.0:pcie010: request IRQ35 failed: -22
[  477.410632] dpc: probe of 0000:00:00.0:pcie010 failed with error -22
[  477.430064] pci 0000:01:00.0: calling quirk_e100_interrupt+0x0/0x1c0
[  477.464772] iommu: Adding device 0000:01:00.0 to group 5
[  477.477650] ixgbe 0000:01:00.0: runtime IRQ mapping not provided by arch
[  477.494175] ixgbe 0000:01:00.0: enabling bus mastering
[  478.942614] ixgbe 0000:01:00.0: irq 26 for MSI
[  478.952806] ixgbe 0000:01:00.0: irq 27 for MSI
[  478.961336] ixgbe 0000:01:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  478.973341] ixgbe 0000:01:00.0: PCI Express bandwidth of 8GT/s available
[  478.980921] ixgbe 0000:01:00.0: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[  478.988765] ixgbe 0000:01:00.0: This is not sufficient for optimal performance of this card.
[  478.997873] ixgbe 0000:01:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[  479.007462] ixgbe 0000:01:00.0: A slot with more lanes and/or higher speed is suggested.
[  479.016667] ixgbe 0000:01:00.0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[  479.023628] ixgbe 0000:01:00.0: 9c:37:f4:90:cd:23
[  481.014736] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[  481.030916] pci 0000:01:00.1: calling quirk_e100_interrupt+0x0/0x1c0
[  481.066757] iommu: Adding device 0000:01:00.1 to group 6
[  481.079297] ixgbe 0000:01:00.1: runtime IRQ mapping not provided by arch
[  481.095780] ixgbe 0000:01:00.1: enabling bus mastering
[  481.574664] ixgbe 0000:01:00.1: irq 28 for MSI
[  481.584598] ixgbe 0000:01:00.1: irq 29 for MSI
[  481.593121] ixgbe 0000:01:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[  481.605182] ixgbe 0000:01:00.1: PCI Express bandwidth of 8GT/s available
[  481.612810] ixgbe 0000:01:00.1: (Speed:2.5GT/s, Width: x4, Encoding Loss:20%)
[  481.620683] ixgbe 0000:01:00.1: This is not sufficient for optimal performance of this card.
[  481.629823] ixgbe 0000:01:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[  481.639444] ixgbe 0000:01:00.1: A slot with more lanes and/or higher speed is suggested.
[  481.648550] ixgbe 0000:01:00.1: MAC: 2, PHY: 17, SFP+: 10, PBA No: FFFFFF-0FF
[  481.656423] ixgbe 0000:01:00.1: 9c:37:f4:90:cd:24
[  483.786457] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
diff mbox

Patch

diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 48eadf4..888230d 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -382,9 +382,9 @@  int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
         for_each_msi_entry(desc, dev) {
                 virq = desc->irq;
                 if (desc->nvec_used == 1)
-                   dev_dbg(dev, "irq %d for MSI\n", virq);
+                 dev_info(dev, "irq %d for MSI\n", virq);
                 else
-                   dev_dbg(dev, "irq [%d-%d] for MSI\n",
+                 dev_info(dev, "irq [%d-%d] for MSI\n",
                                 virq, virq + desc->nvec_used - 1);

 > Is there some cleanup we're missing during the removal?