| Message ID | 20210729132132.19691-1-vbabka@suse.cz (mailing list archive) |
|---|---|
| Headers | show |
| Series | SLUB: reduce irq disabled scope and make it RT compatible | expand |
On 2021-07-29 15:20:57 [+0200], Vlastimil Babka wrote:
> Changes since v2 [5]:
With PARTIAL enabled on top of -rc3:
| root@debpg:~# grep ^kmalloc-512 /proc/slabinfo
| kmalloc-512 3552 3552 512 32 4 : tunables 0 0 0 : slabdata 111 111 0
| root@debpg:~# hackbench -g80
| Running in process mode with 80 groups using 40 file descriptors each (== 3200 tasks)
| Each sender will pass 100 messages of 100 bytes
| Time: 0.643
| root@debpg:~# grep ^kmalloc-512 /proc/slabinfo
| kmalloc-512 954080 954080 512 32 4 : tunables 0 0 0 : slabdata 29815 29815 0
| root@debpg:~# hackbench -g80
| Running in process mode with 80 groups using 40 file descriptors each (== 3200 tasks)
| Each sender will pass 100 messages of 100 bytes
| Time: 0.604
| root@debpg:~# grep ^kmalloc-512 /proc/slabinfo
| kmalloc-512 1647904 1647904 512 32 4 : tunables 0 0 0 : slabdata 51497 51497 0
| root@debpg:~# echo 1 > /sys/kernel/slab/kmalloc-512/shrink
| root@debpg:~# grep ^kmalloc-512 /proc/slabinfo
| kmalloc-512 640 1120 512 32 4 : tunables 0 0 0 : slabdata 35 35 0
otherwise a few more hackbench invocations without manual shirnk lead to
OOM-killer:
| oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,task=systemd-logind,pid=1713,uid=0
| Out of memory: Killed process 1713 (systemd-logind) total-vm:15720kB, anon-rss:956kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
| Mem-Info:
| active_anon:56 inactive_anon:24782 isolated_anon:0
| active_file:13 inactive_file:45 isolated_file:0
| unevictable:0 dirty:0 writeback:0
| slab_reclaimable:8749 slab_unreclaimable:894017
| mapped:68 shmem:118 pagetables:28612 bounce:0
| free:8407 free_pcp:36 free_cma:0
| Node 0 active_anon:224kB inactive_anon:99128kB active_file:260kB inactive_file:712kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:764kB dirty:0kB writebaco
| Node 0 DMA free:15360kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writependiB
| lowmem_reserve[]: 0 1939 3915 3915
| Node 0 DMA32 free:11696kB min:3960kB low:5944kB high:7928kB reserved_highatomic:0KB active_anon:0kB inactive_anon:40740kB active_file:0kB inactive_file:4kB unevictable:0kB
| lowmem_reserve[]: 0 0 1975 1975
| Node 0 Normal free:5692kB min:4032kB low:6052kB high:8072kB reserved_highatomic:0KB active_anon:224kB inactive_anon:58440kB active_file:440kB inactive_file:100kB unevictaB
| lowmem_reserve[]: 0 0 0 0
| Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
| Node 0 DMA32: 11*4kB (UM) 15*8kB (M) 20*16kB (UME) 12*32kB (UME) 7*64kB (ME) 5*128kB (UME) 4*256kB (UM) 6*512kB (ME) 4*1024kB (M) 1*2048kB (M) 0*4096kB = 12196kB
| Node 0 Normal: 324*4kB (UME) 221*8kB (UME) 60*16kB (UM) 24*32kB (UME) 5*64kB (UM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5368kB
| Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
| 189 total pagecache pages
| 0 pages in swap cache
| Swap cache stats: add 0, delete 0, find 0/0
| Free swap = 0kB
| Total swap = 0kB
| 1048432 pages RAM
| 0 pages HighMem/MovableOnly
| 41108 pages reserved
| Unreclaimable slab info:
…
| kmalloc-512 2144352KB 2144352KB
This does not happen if I disable SLUB_CPU_PARTIAL.
Sebastian
On 7/29/21 5:24 PM, Sebastian Andrzej Siewior wrote: > On 2021-07-29 15:20:57 [+0200], Vlastimil Babka wrote: >> Changes since v2 [5]: > > With PARTIAL enabled on top of -rc3: Is that also PREEMPT_RT? Interesting... > | root@debpg:~# grep ^kmalloc-512 /proc/slabinfo > | kmalloc-512 3552 3552 512 32 4 : tunables 0 0 0 : slabdata 111 111 0 > | root@debpg:~# hackbench -g80 > | Running in process mode with 80 groups using 40 file descriptors each (== 3200 tasks) > | Each sender will pass 100 messages of 100 bytes > | Time: 0.643 > | root@debpg:~# grep ^kmalloc-512 /proc/slabinfo > | kmalloc-512 954080 954080 512 32 4 : tunables 0 0 0 : slabdata 29815 29815 0 > | root@debpg:~# hackbench -g80 > | Running in process mode with 80 groups using 40 file descriptors each (== 3200 tasks) > | Each sender will pass 100 messages of 100 bytes > | Time: 0.604 > | root@debpg:~# grep ^kmalloc-512 /proc/slabinfo > | kmalloc-512 1647904 1647904 512 32 4 : tunables 0 0 0 : slabdata 51497 51497 0 > | root@debpg:~# echo 1 > /sys/kernel/slab/kmalloc-512/shrink > | root@debpg:~# grep ^kmalloc-512 /proc/slabinfo > | kmalloc-512 640 1120 512 32 4 : tunables 0 0 0 : slabdata 35 35 0 > > otherwise a few more hackbench invocations without manual shirnk lead to > OOM-killer: > | oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,task=systemd-logind,pid=1713,uid=0 > | Out of memory: Killed process 1713 (systemd-logind) total-vm:15720kB, anon-rss:956kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0 > | Mem-Info: > | active_anon:56 inactive_anon:24782 isolated_anon:0 > | active_file:13 inactive_file:45 isolated_file:0 > | unevictable:0 dirty:0 writeback:0 > | slab_reclaimable:8749 slab_unreclaimable:894017 > | mapped:68 shmem:118 pagetables:28612 bounce:0 > | free:8407 free_pcp:36 free_cma:0 > | Node 0 active_anon:224kB inactive_anon:99128kB active_file:260kB inactive_file:712kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:764kB dirty:0kB writebaco > | Node 0 DMA free:15360kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writependiB > | lowmem_reserve[]: 0 1939 3915 3915 > | Node 0 DMA32 free:11696kB min:3960kB low:5944kB high:7928kB reserved_highatomic:0KB active_anon:0kB inactive_anon:40740kB active_file:0kB inactive_file:4kB unevictable:0kB > | lowmem_reserve[]: 0 0 1975 1975 > | Node 0 Normal free:5692kB min:4032kB low:6052kB high:8072kB reserved_highatomic:0KB active_anon:224kB inactive_anon:58440kB active_file:440kB inactive_file:100kB unevictaB > | lowmem_reserve[]: 0 0 0 0 > | Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB > | Node 0 DMA32: 11*4kB (UM) 15*8kB (M) 20*16kB (UME) 12*32kB (UME) 7*64kB (ME) 5*128kB (UME) 4*256kB (UM) 6*512kB (ME) 4*1024kB (M) 1*2048kB (M) 0*4096kB = 12196kB > | Node 0 Normal: 324*4kB (UME) 221*8kB (UME) 60*16kB (UM) 24*32kB (UME) 5*64kB (UM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5368kB > | Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > | 189 total pagecache pages > | 0 pages in swap cache > | Swap cache stats: add 0, delete 0, find 0/0 > | Free swap = 0kB > | Total swap = 0kB > | 1048432 pages RAM > | 0 pages HighMem/MovableOnly > | 41108 pages reserved > | Unreclaimable slab info: > … > | kmalloc-512 2144352KB 2144352KB > > This does not happen if I disable SLUB_CPU_PARTIAL. > > Sebastian >
On 2021-07-29 17:27:18 [+0200], Vlastimil Babka wrote: > On 7/29/21 5:24 PM, Sebastian Andrzej Siewior wrote: > > On 2021-07-29 15:20:57 [+0200], Vlastimil Babka wrote: > >> Changes since v2 [5]: > > > > With PARTIAL enabled on top of -rc3: > > Is that also PREEMPT_RT? Interesting... No, plain -rc3. Sebastian
On 2021-07-29 17:29:02 [+0200], To Vlastimil Babka wrote: > On 2021-07-29 17:27:18 [+0200], Vlastimil Babka wrote: > > On 7/29/21 5:24 PM, Sebastian Andrzej Siewior wrote: > > > On 2021-07-29 15:20:57 [+0200], Vlastimil Babka wrote: > > >> Changes since v2 [5]: > > > > > > With PARTIAL enabled on top of -rc3: > > > > Is that also PREEMPT_RT? Interesting... > > No, plain -rc3. but it also happens with PREEMPT_RT. Just wanted to make sure that it happens without RT before I report :) Sebastian
On 7/29/21 5:29 PM, Sebastian Andrzej Siewior wrote: > On 2021-07-29 17:27:18 [+0200], Vlastimil Babka wrote: >> On 7/29/21 5:24 PM, Sebastian Andrzej Siewior wrote: >> > On 2021-07-29 15:20:57 [+0200], Vlastimil Babka wrote: >> >> Changes since v2 [5]: >> > >> > With PARTIAL enabled on top of -rc3: >> >> Is that also PREEMPT_RT? Interesting... > > No, plain -rc3. Thanks, probably screwed up put_cpu_partial() with my cleanups, will check. > Sebastian >
On 7/29/21 3:20 PM, Vlastimil Babka wrote: > Changes since v2 [5]: > * Rebase to 5.14-rc3 > * A number of fixes to the RT parts, big thanks to Mike Galbraith for testing > and debugging! > * The largest fix is to protect kmem_cache_cpu->partial by local_lock instead > of cmpxchg tricks, which are insufficient on RT. To avoid divergence > between RT and !RT, just do it everywhere. Affected mainly patch 25 and a > new patch 33. This also addresses a theoretical race raised earlier by Jann > Horn. > * Smaller fixes reported by Sebastian Andrzej Siewior and Cyrill Gorcunov > > Changes since RFC v1 [1]: > * Addressed feedback from Christoph and Mel, added their acks. > * Finished RT conversion, adopting 2 patches from the RT tree. > * The local_lock conversion has to sacrifice lockless fathpaths on PREEMPT_RT > * Added some more cleanup patches to the front. > > This series was initially inspired by Mel's pcplist local_lock rewrite, and > also interest to better understand SLUB's locking and the new primitives and RT > variants and implications. It should make SLUB more preemption-friendly, > especially for RT, hopefully without noticeable regressions, as the fast paths > are not affected. > > Series is based on 5.14-rc3 and also available as a git branch: > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r1 Branch with fixed memory leak in patch 33: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r2
On 2021-07-29 17:47:20 [+0200], Vlastimil Babka wrote: > Branch with fixed memory leak in patch 33: > > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r2 This looks stable, nothing happen during the night. Sebastian
On Thu, Jul 29, 2021 at 03:20:57PM +0200, Vlastimil Babka wrote: > Changes since v2 [5]: > * Rebase to 5.14-rc3 > * A number of fixes to the RT parts, big thanks to Mike Galbraith for testing > and debugging! > * The largest fix is to protect kmem_cache_cpu->partial by local_lock instead > of cmpxchg tricks, which are insufficient on RT. To avoid divergence > between RT and !RT, just do it everywhere. Affected mainly patch 25 and a > new patch 33. This also addresses a theoretical race raised earlier by Jann > Horn. > * Smaller fixes reported by Sebastian Andrzej Siewior and Cyrill Gorcunov > > Changes since RFC v1 [1]: > * Addressed feedback from Christoph and Mel, added their acks. > * Finished RT conversion, adopting 2 patches from the RT tree. > * The local_lock conversion has to sacrifice lockless fathpaths on PREEMPT_RT > * Added some more cleanup patches to the front. > > This series was initially inspired by Mel's pcplist local_lock rewrite, and > also interest to better understand SLUB's locking and the new primitives and RT > variants and implications. It should make SLUB more preemption-friendly, > especially for RT, hopefully without noticeable regressions, as the fast paths > are not affected. > > Series is based on 5.14-rc3 and also available as a git branch: > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r1 > FWIW, I ran a corrected version of this series through a few tests. Some small gains, no major regressions in terms of performance on a !PREEMPT_RT configuration across 6 different machines.
On 8/4/21 2:05 PM, Mel Gorman wrote: > On Thu, Jul 29, 2021 at 03:20:57PM +0200, Vlastimil Babka wrote: >> Series is based on 5.14-rc3 and also available as a git branch: >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r1 >> > > FWIW, I ran a corrected version of this series through a few tests. Some > small gains, no major regressions in terms of performance on a !PREEMPT_RT > configuration across 6 different machines. Thanks a lot, Mel!
Changes since v2 [5]: * Rebase to 5.14-rc3 * A number of fixes to the RT parts, big thanks to Mike Galbraith for testing and debugging! * The largest fix is to protect kmem_cache_cpu->partial by local_lock instead of cmpxchg tricks, which are insufficient on RT. To avoid divergence between RT and !RT, just do it everywhere. Affected mainly patch 25 and a new patch 33. This also addresses a theoretical race raised earlier by Jann Horn. * Smaller fixes reported by Sebastian Andrzej Siewior and Cyrill Gorcunov Changes since RFC v1 [1]: * Addressed feedback from Christoph and Mel, added their acks. * Finished RT conversion, adopting 2 patches from the RT tree. * The local_lock conversion has to sacrifice lockless fathpaths on PREEMPT_RT * Added some more cleanup patches to the front. This series was initially inspired by Mel's pcplist local_lock rewrite, and also interest to better understand SLUB's locking and the new primitives and RT variants and implications. It should make SLUB more preemption-friendly, especially for RT, hopefully without noticeable regressions, as the fast paths are not affected. Series is based on 5.14-rc3 and also available as a git branch: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-local-lock-v3r1 The series should now be sufficiently tested in both RT and !RT configs, mainly thanks to Mike. The RFC/v1 version also got basic performance screening by Mel that didn't show major regressions. Mike's testing with hackbench of v2 on !RT reported negligible differences [6]: virgin(ish) tip 5.13.0.g60ab3ed-tip 7,320.67 msec task-clock # 7.792 CPUs utilized ( +- 0.31% ) 221,215 context-switches # 0.030 M/sec ( +- 3.97% ) 16,234 cpu-migrations # 0.002 M/sec ( +- 4.07% ) 13,233 page-faults # 0.002 M/sec ( +- 0.91% ) 27,592,205,252 cycles # 3.769 GHz ( +- 0.32% ) 8,309,495,040 instructions # 0.30 insn per cycle ( +- 0.37% ) 1,555,210,607 branches # 212.441 M/sec ( +- 0.42% ) 5,484,209 branch-misses # 0.35% of all branches ( +- 2.13% ) 0.93949 +- 0.00423 seconds time elapsed ( +- 0.45% ) 0.94608 +- 0.00384 seconds time elapsed ( +- 0.41% ) (repeat) 0.94422 +- 0.00410 seconds time elapsed ( +- 0.43% ) 5.13.0.g60ab3ed-tip +slub-local-lock-v2r3 7,343.57 msec task-clock # 7.776 CPUs utilized ( +- 0.44% ) 223,044 context-switches # 0.030 M/sec ( +- 3.02% ) 16,057 cpu-migrations # 0.002 M/sec ( +- 4.03% ) 13,164 page-faults # 0.002 M/sec ( +- 0.97% ) 27,684,906,017 cycles # 3.770 GHz ( +- 0.45% ) 8,323,273,871 instructions # 0.30 insn per cycle ( +- 0.28% ) 1,556,106,680 branches # 211.901 M/sec ( +- 0.31% ) 5,463,468 branch-misses # 0.35% of all branches ( +- 1.33% ) 0.94440 +- 0.00352 seconds time elapsed ( +- 0.37% ) 0.94830 +- 0.00228 seconds time elapsed ( +- 0.24% ) (repeat) 0.93813 +- 0.00440 seconds time elapsed ( +- 0.47% ) (repeat) RT configs showed some throughput regressions, but that's expected tradeoff for the preemption improvements through the RT mutex. It didn't prevent the v2 to be incorporated to the 5.13 RT tree [7], leading to testing exposure and bugfixes. Before the series, SLUB is lockless in both allocation and free fast paths, but elsewhere, it's disabling irqs for considerable periods of time - especially in allocation slowpath and the bulk allocation, where IRQs are re-enabled only when a new page from the page allocator is needed, and the context allows blocking. The irq disabled sections can then include deactivate_slab() which walks a full freelist and frees the slab back to page allocator or unfreeze_partials() going through a list of percpu partial slabs. The RT tree currently has some patches mitigating these, but we can do much better in mainline too. Patches 1-6 are straightforward improvements or cleanups that could exist outside of this series too, but are prerequsities. Patches 7-10 are also preparatory code changes without functional changes, but not so useful without the rest of the series. Patch 11 simplifies the fast paths on systems with preemption, based on (hopefully correct) observation that the current loops to verify tid are unnecessary. Patches 12-21 focus on reducing irq disabled scope in the allocation slowpath. Patch 12 moves disabling of irqs into ___slab_alloc() from its callers, which are the allocation slowpath, and bulk allocation. Instead these callers only disable preemption to stabilize the cpu. The following patches then gradually reduce the scope of disabled irqs in ___slab_alloc() and the functions called from there. As of patch 15, the re-enabling of irqs based on gfp flags before calling the page allocator is removed from allocate_slab(). As of patch 18, it's possible to reach the page allocator (in case of existing slabs depleted) without disabling and re-enabling irqs a single time. Pathces 22-27 reduce the scope of disabled irqs in functions related to unfreezing percpu partial slab. Patch 28 is preparatory. Patch 29 is adopted from the RT tree and converts the flushing of percpu slabs on all cpus from using IPI to workqueue, so that the processing isn't happening with irqs disabled in the IPI handler. The flushing is not performance critical so it should be acceptable. Patch 30 also comes from RT tree and makes object_map_lock RT compatible. Patches 31-32 make slab_lock irq-safe on RT where we cannot rely on having irq disabled from the list_lock spin lock usage. Patch 33 changes kmem_cache_cpu->partial handling in put_cpu_partial() from cmpxchg loop to a short irq disabled section, which is used by all other code modifying the field. This addresses a theoretical race scenario pointed out by Jann, and makes the critical section safe wrt with RT local_lock semantics after the conversion in patch 35. Patch 34 changes preempt disable to migrate disable, so that the nested list_lock spinlock is safe to take on RT. Because migrate_disable() is a function call even on !RT, a small set of private wrappers is introduced to keep using the cheaper preempt_disable() on !PREEMPT_RT configurations. As of this patch, SLUB should be compatible with RT's lock semantics, to the best of my knowledge. Finally, patch 35 changes irq disabled sections that protect kmem_cache_cpu fields in the slow paths, with a local lock. However on PREEMPT_RT it means the lockless fast paths can now preempt slow paths which don't expect that, so the local lock has to be taken also in the fast paths and they are no longer lockless. It's up to RT folks to decide if this is a good tradeoff. The patch also updates the locking documentation in the file's comment. The main results of this series: * irq disabling is only done for minimum amount of time needed to protect the kmem_cache_cpu data and as part of spin lock, local lock and bit lock operations to make them irq-safe * SLUB should be fully PREEMPT_RT compatible This should have obvious implications for better preemptibility, especially on RT. Some details are different than how the previous SLUB RT tree patches were implemented: mm: sl[au]b: Change list_lock to raw_spinlock_t [2] - the SLAB part can be dropped as a different patch restricts RT to SLUB anyway. And after this series the list_lock in SLUB is never used with irqs disabled before taking the lock so it doesn't have to be converted to raw_spinlock_t. mm: slub: Move discard_slab() invocations out of IRQ-off sections [3] should be unnecessary as this series does move these invocations outside irq disabled sections in a different way. The remaining patches to upstream from the RT tree are small ones related to KConfig. The patch that restricts PREEMPT_RT to SLUB (not SLAB or SLOB) makes sense. The patch that disables CONFIG_SLUB_CPU_PARTIAL with PREEMPT_RT could perhaps be re-evaluated as the series addresses some latency issues with it. [1] [RFC 00/26] SLUB: use local_lock for kmem_cache_cpu protection and reduce disabling irqs https://lore.kernel.org/lkml/20210524233946.20352-1-vbabka@suse.cz/ [2] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/tree/patches/0001-mm-sl-au-b-Change-list_lock-to-raw_spinlock_t.patch?h=linux-5.12.y-rt-patches [3] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/tree/patches/0004-mm-slub-Move-discard_slab-invocations-out-of-IRQ-off.patch?h=linux-5.12.y-rt-patches [4] https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/tree/patches/0005-mm-slub-Move-flush_cpu_slab-invocations-__free_slab-.patch?h=linux-5.12.y-rt-patches [5] https://lore.kernel.org/lkml/20210609113903.1421-1-vbabka@suse.cz/ [6] https://lore.kernel.org/lkml/891dc24e38106f8542f4c72831d52dc1a1863ae8.camel@gmx.de [7] https://lore.kernel.org/linux-rt-users/87tul5p2fa.ffs@nanos.tec.linutronix.de/ Sebastian Andrzej Siewior (2): mm: slub: Move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context mm: slub: Make object_map_lock a raw_spinlock_t Vlastimil Babka (33): mm, slub: don't call flush_all() from slab_debug_trace_open() mm, slub: allocate private object map for debugfs listings mm, slub: allocate private object map for validate_slab_cache() mm, slub: don't disable irq for debug_check_no_locks_freed() mm, slub: remove redundant unfreeze_partials() from put_cpu_partial() mm, slub: unify cmpxchg_double_slab() and __cmpxchg_double_slab() mm, slub: extract get_partial() from new_slab_objects() mm, slub: dissolve new_slab_objects() into ___slab_alloc() mm, slub: return slab page from get_partial() and set c->page afterwards mm, slub: restructure new page checks in ___slab_alloc() mm, slub: simplify kmem_cache_cpu and tid setup mm, slub: move disabling/enabling irqs to ___slab_alloc() mm, slub: do initial checks in ___slab_alloc() with irqs enabled mm, slub: move disabling irqs closer to get_partial() in ___slab_alloc() mm, slub: restore irqs around calling new_slab() mm, slub: validate slab from partial list or page allocator before making it cpu slab mm, slub: check new pages with restored irqs mm, slub: stop disabling irqs around get_partial() mm, slub: move reset of c->page and freelist out of deactivate_slab() mm, slub: make locking in deactivate_slab() irq-safe mm, slub: call deactivate_slab() without disabling irqs mm, slub: move irq control into unfreeze_partials() mm, slub: discard slabs in unfreeze_partials() without irqs disabled mm, slub: detach whole partial list at once in unfreeze_partials() mm, slub: separate detaching of partial list in unfreeze_partials() from unfreezing mm, slub: only disable irq with spin_lock in __unfreeze_partials() mm, slub: don't disable irqs in slub_cpu_dead() mm, slab: make flush_slab() possible to call with irqs enabled mm, slub: optionally save/restore irqs in slab_[un]lock()/ mm, slub: make slab_lock() disable irqs with PREEMPT_RT mm, slub: protect put_cpu_partial() with disabled irqs instead of cmpxchg mm, slub: use migrate_disable() on PREEMPT_RT mm, slub: convert kmem_cpu_slab protection to local_lock include/linux/slub_def.h | 2 + mm/slub.c | 808 +++++++++++++++++++++++++-------------- 2 files changed, 530 insertions(+), 280 deletions(-)