[v2,0/4] sched: Move task_mm_cid_work to mm delayed work

Message

Gabriele Monaco Dec. 13, 2024, 9:54 a.m. UTC

This patchset moves the task_mm_cid_work to a preemptible and migratable
context. This reduces the impact of this task to the scheduling latency
of real time tasks.
The change makes the recurrence of the task a bit more predictable.
We also add optimisation and fixes to make sure the task_mm_cid_work
works as intended.

Patch 1 contains the main changes, removing the task_work on the
scheduler tick and using a delayed_work instead.

Patch 2 adds some optimisations to the approach, since we rely
on a delayed_work, it is no longer required to check that the minimum
interval passed since execution, we however terminate the call
immediately if we see that no mm_cid is actually active, which could
happen on processes sleeping for long time or which exited but whose mm
has not been freed yet.

Patch 3 allows the mm_cids to be actually compacted when a process
reduces its number of threads, which was not the case since the same
mm_cids were reused to improve cache locality, more details in [1].

Patch 4 adds a selftest to validate the functionality of the
task_mm_cid_work (i.e. to compact the mm_cids), this test requires patch
3 to be applied.

Changes since V1 [1]:
* Re-arm the delayed_work at each invocation
* Cancel the work synchronously at mmdrop
* Remove next scan fields and completely rely on the delayed_work
* Shrink mm_cid allocation with nr thread/affinity (Mathieu Desnoyers)
* Add self test

OVERHEAD COMPARISON

In this section, I'm going to refer to head as the current state
upstream without my patch applied, patch is the same head with these
patches applied. Likewise, I'm going to refer to task_mm_cid_work as
either the task or the function. The experiments are run on an aarch64
machine with 128 cores. The kernel has a bare configuration with
PREEMPT_RT enabled.

- Memory

The patch introduces some memory overhead:
* head uses a callback_head per thread (16 bytes)
* patch relies on a delayed work per mm but drops a long (80 bytes net)

Tasks with 5 threads or less have lower memory footprint with the
current approach.
Considering a task_struct can be 7-13 kB and an mm_struct is about 1.4
kB, the overhead should be acceptable.

- Boot time

I tested the patch booting a virtual machine with vng[2], both head and
patch get similar boot times (around 8s).

- Runtime

I run some rather demanding tests to show what could possibly be a worst
case in the approach introduced by this patch. The following tests are
running again in vng to have a plain system, running mostly the
stressors (if there). Unless differently specified, time is in us. All
tests run for 30s.
The stress-ng tests were run with 128 stressors, I will omit from the
table for clarity.

No load                       head           patch
running processes(threads):   12(12)         12(12)
duration(avg,max,sum):        75,426,987     2,42,45ms
invocations:                  13             20k

stress-ng --cpu-load 80       head           patch
running processes(threads):   129(129)       129(129)
duration(avg,max,sum):        20,2ms,740ms   7,774,280ms
invocations:                  36k            39k

stress-ng --fork              head           patch
running processes(threads):   3.6k(3.6k)     4k(4k)
duration(avg,max,sum):        34,41,720      19,457,880ms
invocations:                  21             46k

stress-ng --pthread-max 4     head           patch
running processes(threads):   129(4k)        129(4k)
duration(avg,max,sum):        31,195,41ms    21,1ms,830ms
invocations:                  1290           38k

It is important to note that some of those stressors run for a very
short period of time to just fork/create a thread, this heavily favours
head since the task won't simply run as often.
Moreover, the duration time needs to be read carefully, since the task
can now be preempted by threads, I tried to exclude that from the
computation, but to keep the probes simple, I didn't exclude
interference caused by interrupts.
On the same system while isolated, the task runs in about 30-35ms, it is
hence highly likely that much larger values are only due to
interruptions, rather than the function actually running that long.

I will post another email with the scripts used to retrieve the data and
more details about the runtime distribution.

[1] - https://lore.kernel.org/linux-kernel/[e-mail]/
[2] - https://github.com/arighi/virtme-ng

Gabriele Monaco (3):
  sched: Move task_mm_cid_work to mm delayed work
  sched: Remove mm_cid_next_scan as obsolete
  rseq/selftests: Add test for mm_cid compaction

Mathieu Desnoyers (1):
  sched: Compact RSEQ concurrency IDs with reduced threads and affinity

 include/linux/mm_types.h                      |  23 ++-
 include/linux/sched.h                         |   1 -
 kernel/sched/core.c                           |  66 +-------
 kernel/sched/sched.h                          |  32 ++--
 tools/testing/selftests/rseq/.gitignore       |   1 +
 tools/testing/selftests/rseq/Makefile         |   2 +-
 .../selftests/rseq/mm_cid_compaction_test.c   | 157 ++++++++++++++++++
 7 files changed, 203 insertions(+), 79 deletions(-)
 create mode 100644 tools/testing/selftests/rseq/mm_cid_compaction_test.c


base-commit: 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b

Comments

Gabriele Monaco Dec. 13, 2024, 11:31 a.m. UTC | #1

On Fri, 2024-12-13 at 10:54 +0100, Gabriele Monaco wrote:
> OVERHEAD COMPARISON
>
> [..]
>
> I will post another email with the scripts used to retrieve the data
> and
> more details about the runtime distribution.

This message contains the performance results produced by my scripts, which are attached.
The tracing is done via bpftrace while a simple bash script is spawning and killing the loads.

From the histograms it's easier to see the distribution of the durations and if there are clear outliers.

TEST RESULTS ON HEAD

Running without loads on virtme-ng

@duration_max: 426
@duration_total: count 13, average 75, total 987

@durations:
[25, 30)               1 |@@@@@@@@@@@@@@@@@                                   |
[30, 35)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[35, 40)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[40, 45)               0 |                                                    |
[45, 50)               3 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[50, 55)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[55, 60)               0 |                                                    |
[60, 65)               1 |@@@@@@@@@@@@@@@@@                                   |
[65, 70)               0 |                                                    |
[70, 75)               0 |                                                    |
[75, 80)               0 |                                                    |
[80, 85)               0 |                                                    |
[85, 90)               0 |                                                    |
[90, 95)               1 |@@@@@@@@@@@@@@@@@                                   |
[95, 100)              0 |                                                    |
[100, ...)             1 |@@@@@@@@@@@@@@@@@                                   |

@processes: 12
@threads: 12

Running with cpu loads on virtme-ng

@duration_max: 2508
@duration_total: count 35948, average 20, total 742603

@durations:
[10, 15)            1889 |@@@@@                                               |
[15, 20)           17278 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[20, 25)           10742 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[25, 30)            3327 |@@@@@@@@@@                                          |
[30, 35)            2350 |@@@@@@@                                             |
[35, 40)             326 |                                                    |
[40, 45)               5 |                                                    |
[45, 50)               1 |                                                    |
[50, 55)               2 |                                                    |
[55, 60)               1 |                                                    |
[60, 65)               2 |                                                    |
[65, 70)               2 |                                                    |
[70, 75)               0 |                                                    |
[75, 80)               0 |                                                    |
[80, 85)               1 |                                                    |
[85, 90)               0 |                                                    |
[90, 95)               1 |                                                    |
[95, 100)              1 |                                                    |
[100, ...)            20 |                                                    |

@processes: 129
@threads: 129

Running with fork loads on virtme-ng

@duration_max: 41
@duration_total: count 21, average 34, total 720

@durations:
[30, 35)              12 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[35, 40)               8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[40, 45)               1 |@@@@                                                |

@processes: 3592
@threads: 3592

Running with thread loads on virtme-ng

@duration_max: 195
@duration_total: count 1286, average 31, total 41082

@durations:
(..., 10)            326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        |
[10, 15)              10 |@                                                   |
[15, 20)               0 |                                                    |
[20, 25)               1 |                                                    |
[25, 30)              61 |@@@@@@@@                                            |
[30, 35)             377 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[35, 40)             264 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[40, 45)              65 |@@@@@@@@                                            |
[45, 50)              32 |@@@@                                                |
[50, 55)              12 |@                                                   |
[55, 60)              13 |@                                                   |
[60, 65)               7 |                                                    |
[65, 70)              10 |@                                                   |
[70, 75)              10 |@                                                   |
[75, 80)              33 |@@@@                                                |
[80, 85)              26 |@@@                                                 |
[85, 90)              13 |@                                                   |
[90, 95)               6 |                                                    |
[95, 100)              2 |                                                    |
[100, ...)            18 |@@                                                  |

@processes: 129
@threads: 4096

TEST RESULTS ON PATCH

Running without loads on virtme-ng

@duration_max: 42
@duration_total: count 20601, average 2, total 45496

@durations:
(..., 10)          20304 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[10, 15)               1 |                                                    |
[15, 20)               4 |                                                    |
[20, 25)              29 |                                                    |
[25, 30)              33 |                                                    |
[30, 35)              11 |                                                    |
[35, 40)             156 |                                                    |
[40, 45)              63 |                                                    |

@processes: 12
@threads: 12

Running with cpu loads on virtme-ng

@duration_max: 774
@duration_total: count 38612, average 7, total 281558

@durations:
(..., 10)          34607 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[10, 15)            2558 |@@@                                                 |
[15, 20)             735 |@                                                   |
[20, 25)             454 |                                                    |
[25, 30)             225 |                                                    |
[30, 35)              17 |                                                    |
[35, 40)               8 |                                                    |
[40, 45)               2 |                                                    |
[45, 50)               4 |                                                    |
[50, 55)               0 |                                                    |
[55, 60)               0 |                                                    |
[60, 65)               0 |                                                    |
[65, 70)               0 |                                                    |
[70, 75)               0 |                                                    |
[75, 80)               0 |                                                    |
[80, 85)               0 |                                                    |
[85, 90)               0 |                                                    |
[90, 95)               0 |                                                    |
[95, 100)              0 |                                                    |
[100, ...)             2 |                                                    |

@processes: 129
@threads: 129

Running with fork loads on virtme-ng

@duration_max: 457
@duration_total: count 45683, average 19, total 878511

@durations:
(..., 10)           8452 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[10, 15)            7287 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                       |
[15, 20)           12727 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[20, 25)            2942 |@@@@@@@@@@@@                                        |
[25, 30)            2975 |@@@@@@@@@@@@                                        |
[30, 35)            7305 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                       |
[35, 40)            2994 |@@@@@@@@@@@@                                        |
[40, 45)             676 |@@                                                  |
[45, 50)             180 |                                                    |
[50, 55)              57 |                                                    |
[55, 60)              19 |                                                    |
[60, 65)               6 |                                                    |
[65, 70)               4 |                                                    |
[70, 75)               2 |                                                    |
[75, 80)               5 |                                                    |
[80, 85)               6 |                                                    |
[85, 90)               4 |                                                    |
[90, 95)               5 |                                                    |
[95, 100)              2 |                                                    |
[100, ...)            34 |                                                    |

@processes: 3982
@threads: 3982

Running with thread loads on virtme-ng

@duration_max: 1046
@duration_total: count 38643, average 21, total 833034

@durations:
(..., 10)           1631 |@@@@@                                               |
[10, 15)           11027 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@              |
[15, 20)           14832 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[20, 25)            1338 |@@@@                                                |
[25, 30)            1112 |@@@                                                 |
[30, 35)            3781 |@@@@@@@@@@@@@                                       |
[35, 40)            1994 |@@@@@@                                              |
[40, 45)             464 |@                                                   |
[45, 50)             262 |                                                    |
[50, 55)             200 |                                                    |
[55, 60)             294 |@                                                   |
[60, 65)             620 |@@                                                  |
[65, 70)             256 |                                                    |
[70, 75)             119 |                                                    |
[75, 80)             232 |                                                    |
[80, 85)             220 |                                                    |
[85, 90)              55 |                                                    |
[90, 95)              30 |                                                    |
[95, 100)             19 |                                                    |
[100, ...)           157 |                                                    |

@processes: 129
@threads: 4096