Message ID | 20220403135717.8294-1-minhquangbui99@gmail.com (mailing list archive) |
---|---|
State | Superseded |
Delegated to: | Netdev Maintainers |
Headers | show |
Series | cgroup: Kill the parent controller when its last child is killed | expand |
Context | Check | Description |
---|---|---|
netdev/tree_selection | success | Not a local patch |
Hi Bui, Thank you for the patch! Yet something to improve: [auto build test ERROR on tj-cgroup/for-next] [also build test ERROR on v5.17 next-20220401] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/intel-lab-lkp/linux/commits/Bui-Quang-Minh/cgroup-Kill-the-parent-controller-when-its-last-child-is-killed/20220403-215911 base: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next config: i386-randconfig-m021 (https://download.01.org/0day-ci/archive/20220403/202204032330.l2wsF3mf-lkp@intel.com/config) compiler: gcc-11 (Debian 11.2.0-19) 11.2.0 reproduce (this is a W=1 build): # https://github.com/intel-lab-lkp/linux/commit/2bc22feae8a913c7f371bc79ef9967122d8d326c git remote add linux-review https://github.com/intel-lab-lkp/linux git fetch --no-tags linux-review Bui-Quang-Minh/cgroup-Kill-the-parent-controller-when-its-last-child-is-killed/20220403-215911 git checkout 2bc22feae8a913c7f371bc79ef9967122d8d326c # save the config file to linux build tree mkdir build_dir make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash kernel/ If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All errors (new ones prefixed by >>): kernel/cgroup/cgroup.c: In function 'css_release_work_fn': >> kernel/cgroup/cgroup.c:5169:52: error: 'struct cgroup_bpf' has no member named 'refcnt' 5169 | if (!percpu_ref_is_dying(&cgrp->bpf.refcnt)) | ^ vim +5169 kernel/cgroup/cgroup.c 5148 5149 static void css_release_work_fn(struct work_struct *work) 5150 { 5151 struct cgroup_subsys_state *css = 5152 container_of(work, struct cgroup_subsys_state, destroy_work); 5153 struct cgroup_subsys *ss = css->ss; 5154 struct cgroup *cgrp = css->cgroup; 5155 struct cgroup *parent = cgroup_parent(cgrp); 5156 5157 mutex_lock(&cgroup_mutex); 5158 5159 css->flags |= CSS_RELEASED; 5160 list_del_rcu(&css->sibling); 5161 5162 /* 5163 * If parent doesn't have any children, start killing it. 5164 * And don't kill the default root. 5165 */ 5166 if (parent && list_empty(&parent->self.children) && 5167 parent != &cgrp_dfl_root.cgrp && 5168 !percpu_ref_is_dying(&parent->self.refcnt)) { > 5169 if (!percpu_ref_is_dying(&cgrp->bpf.refcnt)) 5170 cgroup_bpf_offline(parent); 5171 percpu_ref_kill(&parent->self.refcnt); 5172 } 5173 5174 if (ss) { 5175 /* css release path */ 5176 if (!list_empty(&css->rstat_css_node)) { 5177 cgroup_rstat_flush(cgrp); 5178 list_del_rcu(&css->rstat_css_node); 5179 } 5180 5181 cgroup_idr_replace(&ss->css_idr, NULL, css->id); 5182 if (ss->css_released) 5183 ss->css_released(css); 5184 } else { 5185 struct cgroup *tcgrp; 5186 5187 /* cgroup release path */ 5188 TRACE_CGROUP_PATH(release, cgrp); 5189 5190 cgroup_rstat_flush(cgrp); 5191 5192 spin_lock_irq(&css_set_lock); 5193 for (tcgrp = cgroup_parent(cgrp); tcgrp; 5194 tcgrp = cgroup_parent(tcgrp)) 5195 tcgrp->nr_dying_descendants--; 5196 spin_unlock_irq(&css_set_lock); 5197 5198 /* 5199 * There are two control paths which try to determine 5200 * cgroup from dentry without going through kernfs - 5201 * cgroupstats_build() and css_tryget_online_from_dir(). 5202 * Those are supported by RCU protecting clearing of 5203 * cgrp->kn->priv backpointer. 5204 */ 5205 if (cgrp->kn) 5206 RCU_INIT_POINTER(*(void __rcu __force **)&cgrp->kn->priv, 5207 NULL); 5208 } 5209 5210 mutex_unlock(&cgroup_mutex); 5211 5212 INIT_RCU_WORK(&css->destroy_rwork, css_free_rwork_fn); 5213 queue_rcu_work(cgroup_destroy_wq, &css->destroy_rwork); 5214 } 5215
Hi Bui, Thank you for the patch! Yet something to improve: [auto build test ERROR on tj-cgroup/for-next] [also build test ERROR on v5.17 next-20220401] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/intel-lab-lkp/linux/commits/Bui-Quang-Minh/cgroup-Kill-the-parent-controller-when-its-last-child-is-killed/20220403-215911 base: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next config: arm-mxs_defconfig (https://download.01.org/0day-ci/archive/20220404/202204040103.yTRTggqu-lkp@intel.com/config) compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project c4a1b07d0979e7ff20d7d541af666d822d66b566) reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # install arm cross compiling tool for clang build # apt-get install binutils-arm-linux-gnueabi # https://github.com/intel-lab-lkp/linux/commit/2bc22feae8a913c7f371bc79ef9967122d8d326c git remote add linux-review https://github.com/intel-lab-lkp/linux git fetch --no-tags linux-review Bui-Quang-Minh/cgroup-Kill-the-parent-controller-when-its-last-child-is-killed/20220403-215911 git checkout 2bc22feae8a913c7f371bc79ef9967122d8d326c # save the config file to linux build tree mkdir build_dir COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All errors (new ones prefixed by >>): >> kernel/cgroup/cgroup.c:5169:39: error: no member named 'refcnt' in 'struct cgroup_bpf' if (!percpu_ref_is_dying(&cgrp->bpf.refcnt)) ~~~~~~~~~ ^ 1 error generated. vim +5169 kernel/cgroup/cgroup.c 5148 5149 static void css_release_work_fn(struct work_struct *work) 5150 { 5151 struct cgroup_subsys_state *css = 5152 container_of(work, struct cgroup_subsys_state, destroy_work); 5153 struct cgroup_subsys *ss = css->ss; 5154 struct cgroup *cgrp = css->cgroup; 5155 struct cgroup *parent = cgroup_parent(cgrp); 5156 5157 mutex_lock(&cgroup_mutex); 5158 5159 css->flags |= CSS_RELEASED; 5160 list_del_rcu(&css->sibling); 5161 5162 /* 5163 * If parent doesn't have any children, start killing it. 5164 * And don't kill the default root. 5165 */ 5166 if (parent && list_empty(&parent->self.children) && 5167 parent != &cgrp_dfl_root.cgrp && 5168 !percpu_ref_is_dying(&parent->self.refcnt)) { > 5169 if (!percpu_ref_is_dying(&cgrp->bpf.refcnt)) 5170 cgroup_bpf_offline(parent); 5171 percpu_ref_kill(&parent->self.refcnt); 5172 } 5173 5174 if (ss) { 5175 /* css release path */ 5176 if (!list_empty(&css->rstat_css_node)) { 5177 cgroup_rstat_flush(cgrp); 5178 list_del_rcu(&css->rstat_css_node); 5179 } 5180 5181 cgroup_idr_replace(&ss->css_idr, NULL, css->id); 5182 if (ss->css_released) 5183 ss->css_released(css); 5184 } else { 5185 struct cgroup *tcgrp; 5186 5187 /* cgroup release path */ 5188 TRACE_CGROUP_PATH(release, cgrp); 5189 5190 cgroup_rstat_flush(cgrp); 5191 5192 spin_lock_irq(&css_set_lock); 5193 for (tcgrp = cgroup_parent(cgrp); tcgrp; 5194 tcgrp = cgroup_parent(tcgrp)) 5195 tcgrp->nr_dying_descendants--; 5196 spin_unlock_irq(&css_set_lock); 5197 5198 /* 5199 * There are two control paths which try to determine 5200 * cgroup from dentry without going through kernfs - 5201 * cgroupstats_build() and css_tryget_online_from_dir(). 5202 * Those are supported by RCU protecting clearing of 5203 * cgrp->kn->priv backpointer. 5204 */ 5205 if (cgrp->kn) 5206 RCU_INIT_POINTER(*(void __rcu __force **)&cgrp->kn->priv, 5207 NULL); 5208 } 5209 5210 mutex_unlock(&cgroup_mutex); 5211 5212 INIT_RCU_WORK(&css->destroy_rwork, css_free_rwork_fn); 5213 queue_rcu_work(cgroup_destroy_wq, &css->destroy_rwork); 5214 } 5215
Greeting, FYI, we noticed the following commit (built with gcc-11): commit: 2bc22feae8a913c7f371bc79ef9967122d8d326c ("[PATCH] cgroup: Kill the parent controller when its last child is killed") url: https://github.com/intel-lab-lkp/linux/commits/Bui-Quang-Minh/cgroup-Kill-the-parent-controller-when-its-last-child-is-killed/20220403-215911 base: https://git.kernel.org/cgit/linux/kernel/git/tj/cgroup.git for-next patch link: https://lore.kernel.org/lkml/20220403135717.8294-1-minhquangbui99@gmail.com in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <oliver.sang@intel.com> [ 6.033070][ T1] WARNING: CPU: 0 PID: 1 at lib/percpu-refcount.c:388 percpu_ref_kill_and_confirm (??:?) [ 6.040408][ T1] Modules linked in: [ 6.041099][ T1] CPU: 0 PID: 1 Comm: systemd Not tainted 5.16.0-06542-g2bc22feae8a9 #1 [ 6.042403][ T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 6.043846][ T1] RIP: 0010:percpu_ref_kill_and_confirm (??:?) [ 6.044927][ T1] Code: 3d 69 00 70 01 00 75 24 48 8b 45 08 48 c7 c6 a0 c7 09 82 48 c7 c7 26 a5 32 82 c6 05 4e 00 70 01 01 48 8b 50 08 e8 24 18 61 00 <0f> 0b 48 83 4d 00 02 4c 89 ee 48 89 ef e8 f0 fc ff ff 48 89 ef e8 All code ======== 0: 3d 69 00 70 01 cmp $0x1700069,%eax 5: 00 75 24 add %dh,0x24(%rbp) 8: 48 8b 45 08 mov 0x8(%rbp),%rax c: 48 c7 c6 a0 c7 09 82 mov $0xffffffff8209c7a0,%rsi 13: 48 c7 c7 26 a5 32 82 mov $0xffffffff8232a526,%rdi 1a: c6 05 4e 00 70 01 01 movb $0x1,0x170004e(%rip) # 0x170006f 21: 48 8b 50 08 mov 0x8(%rax),%rdx 25: e8 24 18 61 00 callq 0x61184e 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 83 4d 00 02 orq $0x2,0x0(%rbp) 31: 4c 89 ee mov %r13,%rsi 34: 48 89 ef mov %rbp,%rdi 37: e8 f0 fc ff ff callq 0xfffffffffffffd2c 3c: 48 89 ef mov %rbp,%rdi 3f: e8 .byte 0xe8 Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 83 4d 00 02 orq $0x2,0x0(%rbp) 7: 4c 89 ee mov %r13,%rsi a: 48 89 ef mov %rbp,%rdi d: e8 f0 fc ff ff callq 0xfffffffffffffd02 12: 48 89 ef mov %rbp,%rdi 15: e8 .byte 0xe8 [ 6.051482][ T1] RSP: 0000:ffffc90000013e28 EFLAGS: 00010082 [ 6.052450][ T1] RAX: 0000000000000000 RBX: ffff88812b68b000 RCX: 0000000000000027 [ 6.053757][ T1] RDX: 0000000000000003 RSI: 0000000000000001 RDI: ffff88842fc1b780 [ 6.055070][ T1] RBP: ffff88812b68b010 R08: 0000000000000000 R09: 0000000000000000 [ 6.056322][ T1] R10: 000000000000000b R11: ffffffff82204f78 R12: 0000000000000246 [ 6.058751][ T1] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88812b1097a0 [ 6.063819][ T1] FS: 0000000000000000(0000) GS:ffff88842fc00000(0063) knlGS:00000000f77e31c0 [ 6.065244][ T1] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 6.066251][ T1] CR2: 00000000566f5618 CR3: 000000012b4a4000 CR4: 00000000000406f0 [ 6.067524][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 6.068975][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 6.073751][ T1] Call Trace: [ 6.074375][ T1] <TASK> [ 6.074915][ T1] cgroup_destroy_locked (cgroup.c:?) [ 6.075771][ T1] cgroup_rmdir (??:?) [ 6.076502][ T1] kernfs_iop_rmdir (dir.c:?) [ 6.077415][ T1] vfs_rmdir (??:?) [ 6.078170][ T1] do_rmdir (??:?) [ 6.079048][ T1] __do_sys_rmdir (namei.c:?) [ 6.079829][ T1] do_int80_syscall_32 (??:?) [ 6.080698][ T1] entry_INT80_compat (??:?) [ 6.083561][ T1] RIP: 0023:0xf7f4ba02 [ 6.084304][ T1] Code: 95 01 00 05 25 36 02 00 83 ec 14 8d 80 e8 99 ff ff 50 6a 02 e8 1f ff 00 00 c7 04 24 7f 00 00 00 e8 7e 87 01 00 66 90 90 cd 80 <c3> 8d b6 00 00 00 00 8d bc 27 00 00 00 00 8b 1c 24 c3 8d b6 00 00 All code ======== 0: 95 xchg %eax,%ebp 1: 01 00 add %eax,(%rax) 3: 05 25 36 02 00 add $0x23625,%eax 8: 83 ec 14 sub $0x14,%esp b: 8d 80 e8 99 ff ff lea -0x6618(%rax),%eax 11: 50 push %rax 12: 6a 02 pushq $0x2 14: e8 1f ff 00 00 callq 0xff38 19: c7 04 24 7f 00 00 00 movl $0x7f,(%rsp) 20: e8 7e 87 01 00 callq 0x187a3 25: 66 90 xchg %ax,%ax 27: 90 nop 28: cd 80 int $0x80 2a:* c3 retq <-- trapping instruction 2b: 8d b6 00 00 00 00 lea 0x0(%rsi),%esi 31: 8d bc 27 00 00 00 00 lea 0x0(%rdi,%riz,1),%edi 38: 8b 1c 24 mov (%rsp),%ebx 3b: c3 retq 3c: 8d .byte 0x8d 3d: b6 00 mov $0x0,%dh ... Code starting with the faulting instruction =========================================== 0: c3 retq 1: 8d b6 00 00 00 00 lea 0x0(%rsi),%esi 7: 8d bc 27 00 00 00 00 lea 0x0(%rdi,%riz,1),%edi e: 8b 1c 24 mov (%rsp),%ebx 11: c3 retq 12: 8d .byte 0x8d 13: b6 00 mov $0x0,%dh ... [ 6.087389][ T1] RSP: 002b:00000000ffc47418 EFLAGS: 00000296 ORIG_RAX: 0000000000000028 [ 6.088890][ T1] RAX: ffffffffffffffda RBX: 00000000567287d0 RCX: 00000000f7b0928c [ 6.090266][ T1] RDX: 00000000f7f46440 RSI: 0000000000000000 RDI: 0000000000000001 [ 6.091612][ T1] RBP: 00000000f77e307c R08: 0000000000000000 R09: 0000000000000000 [ 6.092923][ T1] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 6.094354][ T1] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 6.095618][ T1] </TASK> [ 6.096174][ T1] ---[ end trace bc2fc7fcbaaa4bea ]--- [ 6.157221][ T2070] scsi 1:0:0:0: Attached scsi generic sg0 type 5 [ 6.181679][ T2070] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray [ 6.182949][ T2070] cdrom: Uniform CD-ROM driver Revision: 3.20 [ OK ] Started Raise network interfaces. [ OK ] Started udev Coldplug all Devices. [ 6.215433][ T2070] sr 1:0:0:0: Attached scsi CD-ROM sr0 [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting LSB: Execute the kexec -e command to reboot system... [ OK ] Started Regular background program processing daemon. [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. Starting System Logging Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Network. Starting OpenBSD Secure Shell server... Starting LKP bootstrap... Starting Permit User Sessions... [ OK ] Reached target Network is Online. Starting /etc/rc.local Compatibility... Starting LSB: Start and stop bmc-watchdog... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started LSB: Execute the kexec -e command to reboot system. [ 6.234372] rc.local[2437]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/lkp/lkp/src/bin [ OK ] Started /etc/rc.local Compatibility. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. Starting LSB: Load kernel image with kexec... [ OK ] Started LSB: Start and stop bmc-watchdog. [ 6.271554] rc.local[2437]: /lkp/lkp/src/bin/lkp-setup-rootfs: 78: /lkp/lkp/src/bin/lkp-setup-rootfs: cannot create /sys/devices/system/cpu/microcode/reload: Directory nonexistent [ OK ] Started OpenBSD Secure Shell server. [ OK ] Started LSB: Load kernel image with kexec. LKP: ttyS0: 2469: Kernel tests: Boot OK! LKP: ttyS0: 2469: HOSTNAME vm-snb-49, MAC d6:90:49:e5:94:ed, kernel 5.16.0-06542-g2bc22feae8a9 1 LKP: ttyS0: 2469: /lkp/lkp/src/bin/run-lkp /lkp/jobs/scheduled/vm-snb-49/boot-1-debian-i386-20191205.cgz-2bc22feae8a913c7f371bc79ef9967122d8d326c-20220404-23352-ifucqo-4.yaml [ 16.268522][ T2492] LKP: stdout: 2469: Kernel tests: Boot OK! [ 16.268534][ T2492] [ 20.649263][ T2492] LKP: stdout: 2469: HOSTNAME vm-snb-49, MAC d6:90:49:e5:94:ed, kernel 5.16.0-06542-g2bc22feae8a9 1 [ 20.649275][ T2492] LKP: ttyS0: 2469: LKP: rebooting forcely [ 21.111051][ T2492] install debs round one: dpkg -i --force-confdef --force-depends /opt/deb/gawk_1%3a4.1.4+dfsg-1_i386.deb [ 21.111062][ T2492] [ 21.115095][ T2492] Selecting previously unselected package gawk. [ 21.115102][ T2492] [ 21.119713][ T2492] (Reading database ... 16210 files and directories currently installed.) [ 21.119721][ T2492] [ 21.124086][ T2492] Preparing to unpack .../gawk_1%3a4.1.4+dfsg-1_i386.deb ... [ 21.124094][ T2492] To reproduce: # build kernel cd linux cp config-5.16.0-06542-g2bc22feae8a9 .config make HOSTCC=gcc-11 CC=gcc-11 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules make HOSTCC=gcc-11 CC=gcc-11 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install cd <mod-install-dir> find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email # if come across any failure that blocks the test, # please remove ~/.lkp and /lkp dir to run from a clean state.
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index a557eea7166f..220eb1742961 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -5157,12 +5157,25 @@ static void css_release_work_fn(struct work_struct *work) container_of(work, struct cgroup_subsys_state, destroy_work); struct cgroup_subsys *ss = css->ss; struct cgroup *cgrp = css->cgroup; + struct cgroup *parent = cgroup_parent(cgrp); mutex_lock(&cgroup_mutex); css->flags |= CSS_RELEASED; list_del_rcu(&css->sibling); + /* + * If parent doesn't have any children, start killing it. + * And don't kill the default root. + */ + if (parent && list_empty(&parent->self.children) && + parent != &cgrp_dfl_root.cgrp && + !percpu_ref_is_dying(&parent->self.refcnt)) { + if (!percpu_ref_is_dying(&cgrp->bpf.refcnt)) + cgroup_bpf_offline(parent); + percpu_ref_kill(&parent->self.refcnt); + } + if (ss) { /* css release path */ if (!list_empty(&css->rstat_css_node)) {
When umounting a cgroup controller, in case the controller has no children, the initial ref will be dropped in cgroup_kill_sb. In cgroup_rmdir path, the controller is deleted from the parent's children list in css_release_work_fn, which is run on a kernel worker. With this simple script #!/bin/sh mount -t cgroup -o none,name=test test ./tmp mkdir -p ./tmp/abc rmdir ./tmp/abc umount ./tmp sleep 5 cat /proc/self/cgroup The rmdir will remove the last child and umount is expected to kill the parent controller. However, when running the above script, we may get 1:name=test:/ This shows that the parent controller has not been killed. The reason is after rmdir is completed, it is not guaranteed that the parent's children list is empty as css_release_work_fn is deferred to run on a worker. In case cgroup_kill_sb is run before that work, it does not drop the initial ref. Later in the worker, it just removes the child from the list without checking the list is empty to kill the parent controller. As a result, the parent controller still has the initial ref but without any logical refs (children ref, mount ref). This commit adds a free parent controller path into the worker function to free up the parent controller when the last child is killed. Signed-off-by: Bui Quang Minh <minhquangbui99@gmail.com> --- kernel/cgroup/cgroup.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)