| Message ID | 20230801135632.1768830-1-hannes@cmpxchg.org (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | fac2650276eced3c94bcdbc21d0e5be637c1e582 |
| Headers | show |
| Series | selftests: cgroup: fix test_kmem_basic false positives | expand |
On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote: > This test fails routinely in our prod testing environment, and I can > reproduce it locally as well. > > The test allocates dcache inside a cgroup, then drops the memory limit > and checks that usage drops correspondingly. The reason it fails is > because dentries are freed with an RCU delay - a debugging sleep shows > that usage drops as expected shortly after. > > Insert a 1s sleep after dropping the limit. This should be good > enough, assuming that machines running those tests are otherwise not > very busy. > > Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> I am putting together something more formal, but this will certainly improve things, as Johannes says, assuming the system goes mostly idle during that one-second wait. So: Acked-by: Paul E. McKenney <paulmck@kernel.org> Yes, there are corner cases, such as the system having millions of RCU callbacks queued and being unable to invoke them all during that one-second interval. But that is a corner case, and that is exactly why I will be putting together something more formal. ;-) Thanx, Paul > --- > tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c > index 258ddc565deb..1b2cec9d18a4 100644 > --- a/tools/testing/selftests/cgroup/test_kmem.c > +++ b/tools/testing/selftests/cgroup/test_kmem.c > @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) > goto cleanup; > > cg_write(cg, "memory.high", "1M"); > + > + /* wait for RCU freeing */ > + sleep(1); > + > slab1 = cg_read_key_long(cg, "memory.stat", "slab "); > if (slab1 <= 0) > goto cleanup; > -- > 2.41.0 >
On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote: > On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote: > > This test fails routinely in our prod testing environment, and I can > > reproduce it locally as well. > > > > The test allocates dcache inside a cgroup, then drops the memory limit > > and checks that usage drops correspondingly. The reason it fails is > > because dentries are freed with an RCU delay - a debugging sleep shows > > that usage drops as expected shortly after. > > > > Insert a 1s sleep after dropping the limit. This should be good > > enough, assuming that machines running those tests are otherwise not > > very busy. > > > > Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> > > I am putting together something more formal, but this will certainly > improve things, as Johannes says, assuming the system goes mostly > idle during that one-second wait. So: > > Acked-by: Paul E. McKenney <paulmck@kernel.org> > > Yes, there are corner cases, such as the system having millions of > RCU callbacks queued and being unable to invoke them all during that > one-second interval. But that is a corner case, and that is exactly > why I will be putting together something more formal. ;-) > > Thanx, Paul > > > --- > > tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ > > 1 file changed, 4 insertions(+) > > > > diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c > > index 258ddc565deb..1b2cec9d18a4 100644 > > --- a/tools/testing/selftests/cgroup/test_kmem.c > > +++ b/tools/testing/selftests/cgroup/test_kmem.c > > @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) > > goto cleanup; > > > > cg_write(cg, "memory.high", "1M"); > > + > > + /* wait for RCU freeing */ > > + sleep(1); > > + > > slab1 = cg_read_key_long(cg, "memory.stat", "slab "); > > if (slab1 <= 0) > > goto cleanup; > > -- > > 2.41.0 > > The same issue exists in the test case test_kmem_memcg_deletion. I wouldn't mind posting the patch, but it seems you want to propose something more formal. Let me know your opinion. Thanks, Lucas
On Thu, Aug 03, 2023 at 12:13:26PM -0400, Lucas Karpinski wrote: > On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote: > > On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote: > > > This test fails routinely in our prod testing environment, and I can > > > reproduce it locally as well. > > > > > > The test allocates dcache inside a cgroup, then drops the memory limit > > > and checks that usage drops correspondingly. The reason it fails is > > > because dentries are freed with an RCU delay - a debugging sleep shows > > > that usage drops as expected shortly after. > > > > > > Insert a 1s sleep after dropping the limit. This should be good > > > enough, assuming that machines running those tests are otherwise not > > > very busy. > > > > > > Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> > > > > I am putting together something more formal, but this will certainly > > improve things, as Johannes says, assuming the system goes mostly > > idle during that one-second wait. So: > > > > Acked-by: Paul E. McKenney <paulmck@kernel.org> > > > > Yes, there are corner cases, such as the system having millions of > > RCU callbacks queued and being unable to invoke them all during that > > one-second interval. But that is a corner case, and that is exactly > > why I will be putting together something more formal. ;-) > > > > Thanx, Paul > > > > > --- > > > tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ > > > 1 file changed, 4 insertions(+) > > > > > > diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c > > > index 258ddc565deb..1b2cec9d18a4 100644 > > > --- a/tools/testing/selftests/cgroup/test_kmem.c > > > +++ b/tools/testing/selftests/cgroup/test_kmem.c > > > @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) > > > goto cleanup; > > > > > > cg_write(cg, "memory.high", "1M"); > > > + > > > + /* wait for RCU freeing */ > > > + sleep(1); > > > + > > > slab1 = cg_read_key_long(cg, "memory.stat", "slab "); > > > if (slab1 <= 0) > > > goto cleanup; > > > -- > > > 2.41.0 > > > > > The same issue exists in the test case test_kmem_memcg_deletion. I > wouldn't mind posting the patch, but it seems you want to propose > something more formal. Let me know your opinion. I am proposing a /sys/module/rcutree/parameters/do_rcu_barrier file. Writing a "1" into this file results in an rcu_barrier() in the kernel, but set up so that there is no more than a single rcu_barrier() call per second. So you could do the following: run-a-test echo 1 > /sys/module/rcutree/parameters/do_rcu_barrier # As root # All RCU callbacks from run-a-test have now been invoked run-another-test Please note that this handles only RCU, as in call_rcu(), and not SRCU, Tasks RCU, and so on. Thanx, Paul
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c index 258ddc565deb..1b2cec9d18a4 100644 --- a/tools/testing/selftests/cgroup/test_kmem.c +++ b/tools/testing/selftests/cgroup/test_kmem.c @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) goto cleanup; cg_write(cg, "memory.high", "1M"); + + /* wait for RCU freeing */ + sleep(1); + slab1 = cg_read_key_long(cg, "memory.stat", "slab "); if (slab1 <= 0) goto cleanup;
This test fails routinely in our prod testing environment, and I can reproduce it locally as well. The test allocates dcache inside a cgroup, then drops the memory limit and checks that usage drops correspondingly. The reason it fails is because dentries are freed with an RCU delay - a debugging sleep shows that usage drops as expected shortly after. Insert a 1s sleep after dropping the limit. This should be good enough, assuming that machines running those tests are otherwise not very busy. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> --- tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ 1 file changed, 4 insertions(+)