| Message ID | 20241203170933.2449307-1-edumazet@google.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 750e51603395e755537da08f745864c93e3ce741 |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net] net: avoid potential UAF in default_operstate() | expand |
Hi Eric, On Tue, Dec 03, 2024 at 05:09:33PM +0000, Eric Dumazet wrote: > syzbot reported an UAF in default_operstate() [1] > > Issue is a race between device and netns dismantles. > > After calling __rtnl_unlock() from netdev_run_todo(), > we can not assume the netns of each device is still alive. > > Make sure the device is not in NETREG_UNREGISTERED state, > and add an ASSERT_RTNL() before the call to > __dev_get_by_index(). > > We might move this ASSERT_RTNL() in __dev_get_by_index() > in the future. > > [1] > > BUG: KASAN: slab-use-after-free in __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > Read of size 8 at addr ffff888043eba1b0 by task syz.0.0/5339 > > CPU: 0 UID: 0 PID: 5339 Comm: syz.0.0 Not tainted 6.12.0-syzkaller-10296-gaaf20f870da0 #0 > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > Call Trace: > <TASK> > __dump_stack lib/dump_stack.c:94 [inline] > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120 > print_address_description mm/kasan/report.c:378 [inline] > print_report+0x169/0x550 mm/kasan/report.c:489 > kasan_report+0x143/0x180 mm/kasan/report.c:602 > __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > default_operstate net/core/link_watch.c:51 [inline] > rfc2863_policy+0x224/0x300 net/core/link_watch.c:67 > linkwatch_do_dev+0x3e/0x170 net/core/link_watch.c:170 > netdev_run_todo+0x461/0x1000 net/core/dev.c:10894 > rtnl_unlock net/core/rtnetlink.c:152 [inline] > rtnl_net_unlock include/linux/rtnetlink.h:133 [inline] > rtnl_dellink+0x760/0x8d0 net/core/rtnetlink.c:3520 > rtnetlink_rcv_msg+0x791/0xcf0 net/core/rtnetlink.c:6911 > netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2541 > netlink_unicast_kernel net/netlink/af_netlink.c:1321 [inline] > netlink_unicast+0x7f6/0x990 net/netlink/af_netlink.c:1347 > netlink_sendmsg+0x8e4/0xcb0 net/netlink/af_netlink.c:1891 > sock_sendmsg_nosec net/socket.c:711 [inline] > __sock_sendmsg+0x221/0x270 net/socket.c:726 > ____sys_sendmsg+0x52a/0x7e0 net/socket.c:2583 > ___sys_sendmsg net/socket.c:2637 [inline] > __sys_sendmsg+0x269/0x350 net/socket.c:2669 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > RIP: 0033:0x7f2a3cb80809 > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007f2a3d9cd058 EFLAGS: 00000246 ORIG_RAX: 000000000000002e > RAX: ffffffffffffffda RBX: 00007f2a3cd45fa0 RCX: 00007f2a3cb80809 > RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000008 > RBP: 00007f2a3cbf393e R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 > R13: 0000000000000000 R14: 00007f2a3cd45fa0 R15: 00007ffd03bc65c8 > </TASK> In the future could you please trim irrelevant stuff from dumps like this? > > Allocated by task 5339: > kasan_save_stack mm/kasan/common.c:47 [inline] > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > poison_kmalloc_redzone mm/kasan/common.c:377 [inline] > __kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:394 > kasan_kmalloc include/linux/kasan.h:260 [inline] > __kmalloc_cache_noprof+0x243/0x390 mm/slub.c:4314 > kmalloc_noprof include/linux/slab.h:901 [inline] > kmalloc_array_noprof include/linux/slab.h:945 [inline] > netdev_create_hash net/core/dev.c:11870 [inline] > netdev_init+0x10c/0x250 net/core/dev.c:11890 > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > copy_net_ns+0x33f/0x570 net/core/net_namespace.c:500 > create_new_namespaces+0x425/0x7b0 kernel/nsproxy.c:110 > unshare_nsproxy_namespaces+0x124/0x180 kernel/nsproxy.c:228 > ksys_unshare+0x57d/0xa70 kernel/fork.c:3314 > __do_sys_unshare kernel/fork.c:3385 [inline] > __se_sys_unshare kernel/fork.c:3383 [inline] > __x64_sys_unshare+0x38/0x40 kernel/fork.c:3383 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > Freed by task 12: > kasan_save_stack mm/kasan/common.c:47 [inline] > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:582 > poison_slab_object mm/kasan/common.c:247 [inline] > __kasan_slab_free+0x59/0x70 mm/kasan/common.c:264 > kasan_slab_free include/linux/kasan.h:233 [inline] > slab_free_hook mm/slub.c:2338 [inline] > slab_free mm/slub.c:4598 [inline] > kfree+0x196/0x420 mm/slub.c:4746 > netdev_exit+0x65/0xd0 net/core/dev.c:11992 > ops_exit_list net/core/net_namespace.c:172 [inline] > cleanup_net+0x802/0xcc0 net/core/net_namespace.c:632 > process_one_work kernel/workqueue.c:3229 [inline] > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > worker_thread+0x870/0xd30 kernel/workqueue.c:3391 > kthread+0x2f0/0x390 kernel/kthread.c:389 > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > The buggy address belongs to the object at ffff888043eba000 > which belongs to the cache kmalloc-2k of size 2048 > The buggy address is located 432 bytes inside of > freed 2048-byte region [ffff888043eba000, ffff888043eba800) > > The buggy address belongs to the physical page: > page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x43eb8 > head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0 > flags: 0x4fff00000000040(head|node=1|zone=1|lastcpupid=0x7ff) > page_type: f5(slab) > raw: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > raw: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > head: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > head: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > head: 04fff00000000003 ffffea00010fae01 ffffffffffffffff 0000000000000000 > head: 0000000000000008 0000000000000000 00000000ffffffff 0000000000000000 > page dumped because: kasan: bad access detected > page_owner tracks the page as allocated > page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 5339, tgid 5338 (syz.0.0), ts 69674195892, free_ts 69663220888 > set_page_owner include/linux/page_owner.h:32 [inline] > post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1556 > prep_new_page mm/page_alloc.c:1564 [inline] > get_page_from_freelist+0x3649/0x3790 mm/page_alloc.c:3474 > __alloc_pages_noprof+0x292/0x710 mm/page_alloc.c:4751 > alloc_pages_mpol_noprof+0x3e8/0x680 mm/mempolicy.c:2265 > alloc_slab_page+0x6a/0x140 mm/slub.c:2408 > allocate_slab+0x5a/0x2f0 mm/slub.c:2574 > new_slab mm/slub.c:2627 [inline] > ___slab_alloc+0xcd1/0x14b0 mm/slub.c:3815 > __slab_alloc+0x58/0xa0 mm/slub.c:3905 > __slab_alloc_node mm/slub.c:3980 [inline] > slab_alloc_node mm/slub.c:4141 [inline] > __do_kmalloc_node mm/slub.c:4282 [inline] > __kmalloc_noprof+0x2e6/0x4c0 mm/slub.c:4295 > kmalloc_noprof include/linux/slab.h:905 [inline] > sk_prot_alloc+0xe0/0x210 net/core/sock.c:2165 > sk_alloc+0x38/0x370 net/core/sock.c:2218 > __netlink_create+0x65/0x260 net/netlink/af_netlink.c:629 > __netlink_kernel_create+0x174/0x6f0 net/netlink/af_netlink.c:2015 > netlink_kernel_create include/linux/netlink.h:62 [inline] > uevent_net_init+0xed/0x2d0 lib/kobject_uevent.c:783 > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > page last free pid 1032 tgid 1032 stack trace: > reset_page_owner include/linux/page_owner.h:25 [inline] > free_pages_prepare mm/page_alloc.c:1127 [inline] > free_unref_page+0xdf9/0x1140 mm/page_alloc.c:2657 > __slab_free+0x31b/0x3d0 mm/slub.c:4509 > qlink_free mm/kasan/quarantine.c:163 [inline] > qlist_free_all+0x9a/0x140 mm/kasan/quarantine.c:179 > kasan_quarantine_reduce+0x14f/0x170 mm/kasan/quarantine.c:286 > __kasan_slab_alloc+0x23/0x80 mm/kasan/common.c:329 > kasan_slab_alloc include/linux/kasan.h:250 [inline] > slab_post_alloc_hook mm/slub.c:4104 [inline] > slab_alloc_node mm/slub.c:4153 [inline] > kmem_cache_alloc_node_noprof+0x1d9/0x380 mm/slub.c:4205 > __alloc_skb+0x1c3/0x440 net/core/skbuff.c:668 > alloc_skb include/linux/skbuff.h:1323 [inline] > alloc_skb_with_frags+0xc3/0x820 net/core/skbuff.c:6612 > sock_alloc_send_pskb+0x91a/0xa60 net/core/sock.c:2881 > sock_alloc_send_skb include/net/sock.h:1797 [inline] > mld_newpack+0x1c3/0xaf0 net/ipv6/mcast.c:1747 > add_grhead net/ipv6/mcast.c:1850 [inline] > add_grec+0x1492/0x19a0 net/ipv6/mcast.c:1988 > mld_send_initial_cr+0x228/0x4b0 net/ipv6/mcast.c:2234 > ipv6_mc_dad_complete+0x88/0x490 net/ipv6/mcast.c:2245 > addrconf_dad_completed+0x712/0xcd0 net/ipv6/addrconf.c:4342 > addrconf_dad_work+0xdc2/0x16f0 > process_one_work kernel/workqueue.c:3229 [inline] > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > > Memory state around the buggy address: > ffff888043eba080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ffff888043eba100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >ffff888043eba180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ^ > ffff888043eba200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ffff888043eba280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > Fixes: 8c55facecd7a ("net: linkwatch: only report IF_OPER_LOWERLAYERDOWN if iflink is actually down") > Reported-by: syzbot+1939f24bdb783e9e43d9@syzkaller.appspotmail.com > Closes: https://lore.kernel.org/netdev/674f3a18.050a0220.48a03.0041.GAE@google.com/T/#u > Signed-off-by: Eric Dumazet <edumazet@google.com> > --- > Cc: Vladimir Oltean <vladimir.oltean@nxp.com> > --- > net/core/link_watch.c | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/net/core/link_watch.c b/net/core/link_watch.c > index ab150641142aa1545c71fc5d3b11db33c70cf437..1b4d39e38084272269a51503c217fc1e5a1326eb 100644 > --- a/net/core/link_watch.c > +++ b/net/core/link_watch.c > @@ -45,9 +45,14 @@ static unsigned int default_operstate(const struct net_device *dev) > int iflink = dev_get_iflink(dev); > struct net_device *peer; > > - if (iflink == dev->ifindex) > + /* If called from netdev_run_todo()/linkwatch_sync_dev(), > + * dev_net(dev) can be already freed, and RTNL is not held. > + */ > + if (dev->reg_state == NETREG_UNREGISTERED || > + iflink == dev->ifindex) > return IF_OPER_DOWN; > > + ASSERT_RTNL(); > peer = __dev_get_by_index(dev_net(dev), iflink); > if (!peer) > return IF_OPER_DOWN; > -- > 2.47.0.338.g60cca15819-goog > Thanks for submitting a patch, the issue makes sense. Question: is the rtnl_mutex actually held in the problematic case though? The netdev_run_todo() call path is: __rtnl_unlock(); <- unlocks /* Wait for rcu callbacks to finish before next phase */ if (!list_empty(&list)) rcu_barrier(); list_for_each_entry_safe(dev, tmp, &list, todo_list) { if (unlikely(dev->reg_state != NETREG_UNREGISTERING)) { netdev_WARN(dev, "run_todo but not unregistering\n"); list_del(&dev->todo_list); continue; } WRITE_ONCE(dev->reg_state, NETREG_UNREGISTERED); linkwatch_sync_dev(dev); <- asserts } And on the same note: does linkwatch not have a chance to run also, concurrently with us, in this timeframe? Could we not catch the dev->reg_state in NETREG_UNREGISTERING?
On Tue, Dec 3, 2024 at 6:37 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > Hi Eric, > > On Tue, Dec 03, 2024 at 05:09:33PM +0000, Eric Dumazet wrote: > > syzbot reported an UAF in default_operstate() [1] > > > > Issue is a race between device and netns dismantles. > > > > After calling __rtnl_unlock() from netdev_run_todo(), > > we can not assume the netns of each device is still alive. > > > > Make sure the device is not in NETREG_UNREGISTERED state, > > and add an ASSERT_RTNL() before the call to > > __dev_get_by_index(). > > > > We might move this ASSERT_RTNL() in __dev_get_by_index() > > in the future. > > > > [1] > > > > BUG: KASAN: slab-use-after-free in __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > > Read of size 8 at addr ffff888043eba1b0 by task syz.0.0/5339 > > > > CPU: 0 UID: 0 PID: 5339 Comm: syz.0.0 Not tainted 6.12.0-syzkaller-10296-gaaf20f870da0 #0 > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > Call Trace: > > <TASK> > > __dump_stack lib/dump_stack.c:94 [inline] > > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120 > > print_address_description mm/kasan/report.c:378 [inline] > > print_report+0x169/0x550 mm/kasan/report.c:489 > > kasan_report+0x143/0x180 mm/kasan/report.c:602 > > __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > > default_operstate net/core/link_watch.c:51 [inline] > > rfc2863_policy+0x224/0x300 net/core/link_watch.c:67 > > linkwatch_do_dev+0x3e/0x170 net/core/link_watch.c:170 > > netdev_run_todo+0x461/0x1000 net/core/dev.c:10894 > > rtnl_unlock net/core/rtnetlink.c:152 [inline] > > rtnl_net_unlock include/linux/rtnetlink.h:133 [inline] > > rtnl_dellink+0x760/0x8d0 net/core/rtnetlink.c:3520 > > rtnetlink_rcv_msg+0x791/0xcf0 net/core/rtnetlink.c:6911 > > netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2541 > > netlink_unicast_kernel net/netlink/af_netlink.c:1321 [inline] > > netlink_unicast+0x7f6/0x990 net/netlink/af_netlink.c:1347 > > netlink_sendmsg+0x8e4/0xcb0 net/netlink/af_netlink.c:1891 > > sock_sendmsg_nosec net/socket.c:711 [inline] > > __sock_sendmsg+0x221/0x270 net/socket.c:726 > > ____sys_sendmsg+0x52a/0x7e0 net/socket.c:2583 > > ___sys_sendmsg net/socket.c:2637 [inline] > > __sys_sendmsg+0x269/0x350 net/socket.c:2669 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > RIP: 0033:0x7f2a3cb80809 > > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 > > RSP: 002b:00007f2a3d9cd058 EFLAGS: 00000246 ORIG_RAX: 000000000000002e > > RAX: ffffffffffffffda RBX: 00007f2a3cd45fa0 RCX: 00007f2a3cb80809 > > RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000008 > > RBP: 00007f2a3cbf393e R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 > > R13: 0000000000000000 R14: 00007f2a3cd45fa0 R15: 00007ffd03bc65c8 > > </TASK> > > In the future could you please trim irrelevant stuff from dumps like this? I prefer the full output, it can be very useful. It is relevant to me at least. > > > > > Allocated by task 5339: > > kasan_save_stack mm/kasan/common.c:47 [inline] > > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > > poison_kmalloc_redzone mm/kasan/common.c:377 [inline] > > __kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:394 > > kasan_kmalloc include/linux/kasan.h:260 [inline] > > __kmalloc_cache_noprof+0x243/0x390 mm/slub.c:4314 > > kmalloc_noprof include/linux/slab.h:901 [inline] > > kmalloc_array_noprof include/linux/slab.h:945 [inline] > > netdev_create_hash net/core/dev.c:11870 [inline] > > netdev_init+0x10c/0x250 net/core/dev.c:11890 > > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > > copy_net_ns+0x33f/0x570 net/core/net_namespace.c:500 > > create_new_namespaces+0x425/0x7b0 kernel/nsproxy.c:110 > > unshare_nsproxy_namespaces+0x124/0x180 kernel/nsproxy.c:228 > > ksys_unshare+0x57d/0xa70 kernel/fork.c:3314 > > __do_sys_unshare kernel/fork.c:3385 [inline] > > __se_sys_unshare kernel/fork.c:3383 [inline] > > __x64_sys_unshare+0x38/0x40 kernel/fork.c:3383 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > Freed by task 12: > > kasan_save_stack mm/kasan/common.c:47 [inline] > > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > > kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:582 > > poison_slab_object mm/kasan/common.c:247 [inline] > > __kasan_slab_free+0x59/0x70 mm/kasan/common.c:264 > > kasan_slab_free include/linux/kasan.h:233 [inline] > > slab_free_hook mm/slub.c:2338 [inline] > > slab_free mm/slub.c:4598 [inline] > > kfree+0x196/0x420 mm/slub.c:4746 > > netdev_exit+0x65/0xd0 net/core/dev.c:11992 > > ops_exit_list net/core/net_namespace.c:172 [inline] > > cleanup_net+0x802/0xcc0 net/core/net_namespace.c:632 > > process_one_work kernel/workqueue.c:3229 [inline] > > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > > worker_thread+0x870/0xd30 kernel/workqueue.c:3391 > > kthread+0x2f0/0x390 kernel/kthread.c:389 > > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > > > The buggy address belongs to the object at ffff888043eba000 > > which belongs to the cache kmalloc-2k of size 2048 > > The buggy address is located 432 bytes inside of > > freed 2048-byte region [ffff888043eba000, ffff888043eba800) > > > > The buggy address belongs to the physical page: > > page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x43eb8 > > head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0 > > flags: 0x4fff00000000040(head|node=1|zone=1|lastcpupid=0x7ff) > > page_type: f5(slab) > > raw: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > > raw: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > > head: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > > head: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > > head: 04fff00000000003 ffffea00010fae01 ffffffffffffffff 0000000000000000 > > head: 0000000000000008 0000000000000000 00000000ffffffff 0000000000000000 > > page dumped because: kasan: bad access detected > > page_owner tracks the page as allocated > > page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 5339, tgid 5338 (syz.0.0), ts 69674195892, free_ts 69663220888 > > set_page_owner include/linux/page_owner.h:32 [inline] > > post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1556 > > prep_new_page mm/page_alloc.c:1564 [inline] > > get_page_from_freelist+0x3649/0x3790 mm/page_alloc.c:3474 > > __alloc_pages_noprof+0x292/0x710 mm/page_alloc.c:4751 > > alloc_pages_mpol_noprof+0x3e8/0x680 mm/mempolicy.c:2265 > > alloc_slab_page+0x6a/0x140 mm/slub.c:2408 > > allocate_slab+0x5a/0x2f0 mm/slub.c:2574 > > new_slab mm/slub.c:2627 [inline] > > ___slab_alloc+0xcd1/0x14b0 mm/slub.c:3815 > > __slab_alloc+0x58/0xa0 mm/slub.c:3905 > > __slab_alloc_node mm/slub.c:3980 [inline] > > slab_alloc_node mm/slub.c:4141 [inline] > > __do_kmalloc_node mm/slub.c:4282 [inline] > > __kmalloc_noprof+0x2e6/0x4c0 mm/slub.c:4295 > > kmalloc_noprof include/linux/slab.h:905 [inline] > > sk_prot_alloc+0xe0/0x210 net/core/sock.c:2165 > > sk_alloc+0x38/0x370 net/core/sock.c:2218 > > __netlink_create+0x65/0x260 net/netlink/af_netlink.c:629 > > __netlink_kernel_create+0x174/0x6f0 net/netlink/af_netlink.c:2015 > > netlink_kernel_create include/linux/netlink.h:62 [inline] > > uevent_net_init+0xed/0x2d0 lib/kobject_uevent.c:783 > > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > > page last free pid 1032 tgid 1032 stack trace: > > reset_page_owner include/linux/page_owner.h:25 [inline] > > free_pages_prepare mm/page_alloc.c:1127 [inline] > > free_unref_page+0xdf9/0x1140 mm/page_alloc.c:2657 > > __slab_free+0x31b/0x3d0 mm/slub.c:4509 > > qlink_free mm/kasan/quarantine.c:163 [inline] > > qlist_free_all+0x9a/0x140 mm/kasan/quarantine.c:179 > > kasan_quarantine_reduce+0x14f/0x170 mm/kasan/quarantine.c:286 > > __kasan_slab_alloc+0x23/0x80 mm/kasan/common.c:329 > > kasan_slab_alloc include/linux/kasan.h:250 [inline] > > slab_post_alloc_hook mm/slub.c:4104 [inline] > > slab_alloc_node mm/slub.c:4153 [inline] > > kmem_cache_alloc_node_noprof+0x1d9/0x380 mm/slub.c:4205 > > __alloc_skb+0x1c3/0x440 net/core/skbuff.c:668 > > alloc_skb include/linux/skbuff.h:1323 [inline] > > alloc_skb_with_frags+0xc3/0x820 net/core/skbuff.c:6612 > > sock_alloc_send_pskb+0x91a/0xa60 net/core/sock.c:2881 > > sock_alloc_send_skb include/net/sock.h:1797 [inline] > > mld_newpack+0x1c3/0xaf0 net/ipv6/mcast.c:1747 > > add_grhead net/ipv6/mcast.c:1850 [inline] > > add_grec+0x1492/0x19a0 net/ipv6/mcast.c:1988 > > mld_send_initial_cr+0x228/0x4b0 net/ipv6/mcast.c:2234 > > ipv6_mc_dad_complete+0x88/0x490 net/ipv6/mcast.c:2245 > > addrconf_dad_completed+0x712/0xcd0 net/ipv6/addrconf.c:4342 > > addrconf_dad_work+0xdc2/0x16f0 > > process_one_work kernel/workqueue.c:3229 [inline] > > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > > > > Memory state around the buggy address: > > ffff888043eba080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > ffff888043eba100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > >ffff888043eba180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > ^ > > ffff888043eba200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > ffff888043eba280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > > Fixes: 8c55facecd7a ("net: linkwatch: only report IF_OPER_LOWERLAYERDOWN if iflink is actually down") > > Reported-by: syzbot+1939f24bdb783e9e43d9@syzkaller.appspotmail.com > > Closes: https://lore.kernel.org/netdev/674f3a18.050a0220.48a03.0041.GAE@google.com/T/#u > > Signed-off-by: Eric Dumazet <edumazet@google.com> > > --- > > Cc: Vladimir Oltean <vladimir.oltean@nxp.com> > > --- > > net/core/link_watch.c | 7 ++++++- > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > diff --git a/net/core/link_watch.c b/net/core/link_watch.c > > index ab150641142aa1545c71fc5d3b11db33c70cf437..1b4d39e38084272269a51503c217fc1e5a1326eb 100644 > > --- a/net/core/link_watch.c > > +++ b/net/core/link_watch.c > > @@ -45,9 +45,14 @@ static unsigned int default_operstate(const struct net_device *dev) > > int iflink = dev_get_iflink(dev); > > struct net_device *peer; > > > > - if (iflink == dev->ifindex) > > + /* If called from netdev_run_todo()/linkwatch_sync_dev(), > > + * dev_net(dev) can be already freed, and RTNL is not held. > > + */ > > + if (dev->reg_state == NETREG_UNREGISTERED || > > + iflink == dev->ifindex) > > return IF_OPER_DOWN; > > > > + ASSERT_RTNL(); > > peer = __dev_get_by_index(dev_net(dev), iflink); > > if (!peer) > > return IF_OPER_DOWN; > > -- > > 2.47.0.338.g60cca15819-goog > > > > Thanks for submitting a patch, the issue makes sense. > > Question: is the rtnl_mutex actually held in the problematic case though? > The netdev_run_todo() call path is: As explained in the comment, RTNL is not held in this case : /* If called from netdev_run_todo()/linkwatch_sync_dev(), * dev_net(dev) can be already freed, and RTNL is not held. */ In the future, we might change default_operstate() to use dev_get_by_index_rcu() and not rely on RTNL anymore, but after this patch, the ASSERT_RTNL() is fine. > > __rtnl_unlock(); <- unlocks > > /* Wait for rcu callbacks to finish before next phase */ > if (!list_empty(&list)) > rcu_barrier(); > > list_for_each_entry_safe(dev, tmp, &list, todo_list) { > if (unlikely(dev->reg_state != NETREG_UNREGISTERING)) { > netdev_WARN(dev, "run_todo but not unregistering\n"); > list_del(&dev->todo_list); > continue; > } > > WRITE_ONCE(dev->reg_state, NETREG_UNREGISTERED); // reg_state is set to NETREG_UNREGISTERING > linkwatch_sync_dev(dev); <- asserts > } > > And on the same note: does linkwatch not have a chance to run also, > concurrently with us, in this timeframe? Could we not catch the > dev->reg_state in NETREG_UNREGISTERING? I guess we can add a READ_ONCE() on many dev->reg_state reads. The race should not matter for linkwatch, if the device is going away.
On Tue, Dec 03, 2024 at 06:56:09PM +0100, Eric Dumazet wrote: > On Tue, Dec 3, 2024 at 6:37 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > > > Hi Eric, > > > > On Tue, Dec 03, 2024 at 05:09:33PM +0000, Eric Dumazet wrote: > > > syzbot reported an UAF in default_operstate() [1] > > > > > > Issue is a race between device and netns dismantles. > > > > > > After calling __rtnl_unlock() from netdev_run_todo(), > > > we can not assume the netns of each device is still alive. > > > > > > Make sure the device is not in NETREG_UNREGISTERED state, > > > and add an ASSERT_RTNL() before the call to > > > __dev_get_by_index(). > > > > > > We might move this ASSERT_RTNL() in __dev_get_by_index() > > > in the future. > > > > > > [1] > > > > > > BUG: KASAN: slab-use-after-free in __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > > > Read of size 8 at addr ffff888043eba1b0 by task syz.0.0/5339 > > > > > > CPU: 0 UID: 0 PID: 5339 Comm: syz.0.0 Not tainted 6.12.0-syzkaller-10296-gaaf20f870da0 #0 > > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > > Call Trace: > > > <TASK> > > > __dump_stack lib/dump_stack.c:94 [inline] > > > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120 > > > print_address_description mm/kasan/report.c:378 [inline] > > > print_report+0x169/0x550 mm/kasan/report.c:489 > > > kasan_report+0x143/0x180 mm/kasan/report.c:602 > > > __dev_get_by_index+0x5d/0x110 net/core/dev.c:852 > > > default_operstate net/core/link_watch.c:51 [inline] > > > rfc2863_policy+0x224/0x300 net/core/link_watch.c:67 > > > linkwatch_do_dev+0x3e/0x170 net/core/link_watch.c:170 > > > netdev_run_todo+0x461/0x1000 net/core/dev.c:10894 > > > rtnl_unlock net/core/rtnetlink.c:152 [inline] > > > rtnl_net_unlock include/linux/rtnetlink.h:133 [inline] > > > rtnl_dellink+0x760/0x8d0 net/core/rtnetlink.c:3520 > > > rtnetlink_rcv_msg+0x791/0xcf0 net/core/rtnetlink.c:6911 > > > netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2541 > > > netlink_unicast_kernel net/netlink/af_netlink.c:1321 [inline] > > > netlink_unicast+0x7f6/0x990 net/netlink/af_netlink.c:1347 > > > netlink_sendmsg+0x8e4/0xcb0 net/netlink/af_netlink.c:1891 > > > sock_sendmsg_nosec net/socket.c:711 [inline] > > > __sock_sendmsg+0x221/0x270 net/socket.c:726 > > > ____sys_sendmsg+0x52a/0x7e0 net/socket.c:2583 > > > ___sys_sendmsg net/socket.c:2637 [inline] > > > __sys_sendmsg+0x269/0x350 net/socket.c:2669 > > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > RIP: 0033:0x7f2a3cb80809 > > > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 > > > RSP: 002b:00007f2a3d9cd058 EFLAGS: 00000246 ORIG_RAX: 000000000000002e > > > RAX: ffffffffffffffda RBX: 00007f2a3cd45fa0 RCX: 00007f2a3cb80809 > > > RDX: 0000000000000000 RSI: 0000000020000000 RDI: 0000000000000008 > > > RBP: 00007f2a3cbf393e R08: 0000000000000000 R09: 0000000000000000 > > > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 > > > R13: 0000000000000000 R14: 00007f2a3cd45fa0 R15: 00007ffd03bc65c8 > > > </TASK> > > > > In the future could you please trim irrelevant stuff from dumps like this? > > I prefer the full output, it can be very useful. It is relevant to me at least. I mean the kasan, dump_stack and mm portion from the stack traces, as well as the register dump, are pretty much irrelevant. They make navigating the useful portion of the kasan splat more difficult. > > > Allocated by task 5339: > > > kasan_save_stack mm/kasan/common.c:47 [inline] > > > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > > > poison_kmalloc_redzone mm/kasan/common.c:377 [inline] > > > __kasan_kmalloc+0x98/0xb0 mm/kasan/common.c:394 > > > kasan_kmalloc include/linux/kasan.h:260 [inline] > > > __kmalloc_cache_noprof+0x243/0x390 mm/slub.c:4314 > > > kmalloc_noprof include/linux/slab.h:901 [inline] > > > kmalloc_array_noprof include/linux/slab.h:945 [inline] > > > netdev_create_hash net/core/dev.c:11870 [inline] > > > netdev_init+0x10c/0x250 net/core/dev.c:11890 > > > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > > > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > > > copy_net_ns+0x33f/0x570 net/core/net_namespace.c:500 > > > create_new_namespaces+0x425/0x7b0 kernel/nsproxy.c:110 > > > unshare_nsproxy_namespaces+0x124/0x180 kernel/nsproxy.c:228 > > > ksys_unshare+0x57d/0xa70 kernel/fork.c:3314 > > > __do_sys_unshare kernel/fork.c:3385 [inline] > > > __se_sys_unshare kernel/fork.c:3383 [inline] > > > __x64_sys_unshare+0x38/0x40 kernel/fork.c:3383 > > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > > > Freed by task 12: > > > kasan_save_stack mm/kasan/common.c:47 [inline] > > > kasan_save_track+0x3f/0x80 mm/kasan/common.c:68 > > > kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:582 > > > poison_slab_object mm/kasan/common.c:247 [inline] > > > __kasan_slab_free+0x59/0x70 mm/kasan/common.c:264 > > > kasan_slab_free include/linux/kasan.h:233 [inline] > > > slab_free_hook mm/slub.c:2338 [inline] > > > slab_free mm/slub.c:4598 [inline] > > > kfree+0x196/0x420 mm/slub.c:4746 > > > netdev_exit+0x65/0xd0 net/core/dev.c:11992 > > > ops_exit_list net/core/net_namespace.c:172 [inline] > > > cleanup_net+0x802/0xcc0 net/core/net_namespace.c:632 Where is __put_net() called from? The timeline is not clear to me. > > > process_one_work kernel/workqueue.c:3229 [inline] > > > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > > > worker_thread+0x870/0xd30 kernel/workqueue.c:3391 > > > kthread+0x2f0/0x390 kernel/kthread.c:389 > > > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 > > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > > > > > > The buggy address belongs to the object at ffff888043eba000 > > > which belongs to the cache kmalloc-2k of size 2048 > > > The buggy address is located 432 bytes inside of > > > freed 2048-byte region [ffff888043eba000, ffff888043eba800) > > > > > > The buggy address belongs to the physical page: > > > page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x43eb8 > > > head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0 > > > flags: 0x4fff00000000040(head|node=1|zone=1|lastcpupid=0x7ff) > > > page_type: f5(slab) > > > raw: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > > > raw: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > > > head: 04fff00000000040 ffff88801ac42000 dead000000000122 0000000000000000 > > > head: 0000000000000000 0000000000080008 00000001f5000000 0000000000000000 > > > head: 04fff00000000003 ffffea00010fae01 ffffffffffffffff 0000000000000000 > > > head: 0000000000000008 0000000000000000 00000000ffffffff 0000000000000000 > > > page dumped because: kasan: bad access detected > > > page_owner tracks the page as allocated > > > page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 5339, tgid 5338 (syz.0.0), ts 69674195892, free_ts 69663220888 > > > set_page_owner include/linux/page_owner.h:32 [inline] > > > post_alloc_hook+0x1f3/0x230 mm/page_alloc.c:1556 > > > prep_new_page mm/page_alloc.c:1564 [inline] > > > get_page_from_freelist+0x3649/0x3790 mm/page_alloc.c:3474 > > > __alloc_pages_noprof+0x292/0x710 mm/page_alloc.c:4751 > > > alloc_pages_mpol_noprof+0x3e8/0x680 mm/mempolicy.c:2265 > > > alloc_slab_page+0x6a/0x140 mm/slub.c:2408 > > > allocate_slab+0x5a/0x2f0 mm/slub.c:2574 > > > new_slab mm/slub.c:2627 [inline] > > > ___slab_alloc+0xcd1/0x14b0 mm/slub.c:3815 > > > __slab_alloc+0x58/0xa0 mm/slub.c:3905 > > > __slab_alloc_node mm/slub.c:3980 [inline] > > > slab_alloc_node mm/slub.c:4141 [inline] > > > __do_kmalloc_node mm/slub.c:4282 [inline] > > > __kmalloc_noprof+0x2e6/0x4c0 mm/slub.c:4295 > > > kmalloc_noprof include/linux/slab.h:905 [inline] > > > sk_prot_alloc+0xe0/0x210 net/core/sock.c:2165 > > > sk_alloc+0x38/0x370 net/core/sock.c:2218 > > > __netlink_create+0x65/0x260 net/netlink/af_netlink.c:629 > > > __netlink_kernel_create+0x174/0x6f0 net/netlink/af_netlink.c:2015 > > > netlink_kernel_create include/linux/netlink.h:62 [inline] > > > uevent_net_init+0xed/0x2d0 lib/kobject_uevent.c:783 > > > ops_init+0x31e/0x590 net/core/net_namespace.c:138 > > > setup_net+0x287/0x9e0 net/core/net_namespace.c:362 > > > page last free pid 1032 tgid 1032 stack trace: > > > reset_page_owner include/linux/page_owner.h:25 [inline] > > > free_pages_prepare mm/page_alloc.c:1127 [inline] > > > free_unref_page+0xdf9/0x1140 mm/page_alloc.c:2657 > > > __slab_free+0x31b/0x3d0 mm/slub.c:4509 > > > qlink_free mm/kasan/quarantine.c:163 [inline] > > > qlist_free_all+0x9a/0x140 mm/kasan/quarantine.c:179 > > > kasan_quarantine_reduce+0x14f/0x170 mm/kasan/quarantine.c:286 > > > __kasan_slab_alloc+0x23/0x80 mm/kasan/common.c:329 > > > kasan_slab_alloc include/linux/kasan.h:250 [inline] > > > slab_post_alloc_hook mm/slub.c:4104 [inline] > > > slab_alloc_node mm/slub.c:4153 [inline] > > > kmem_cache_alloc_node_noprof+0x1d9/0x380 mm/slub.c:4205 > > > __alloc_skb+0x1c3/0x440 net/core/skbuff.c:668 > > > alloc_skb include/linux/skbuff.h:1323 [inline] > > > alloc_skb_with_frags+0xc3/0x820 net/core/skbuff.c:6612 > > > sock_alloc_send_pskb+0x91a/0xa60 net/core/sock.c:2881 > > > sock_alloc_send_skb include/net/sock.h:1797 [inline] > > > mld_newpack+0x1c3/0xaf0 net/ipv6/mcast.c:1747 > > > add_grhead net/ipv6/mcast.c:1850 [inline] > > > add_grec+0x1492/0x19a0 net/ipv6/mcast.c:1988 > > > mld_send_initial_cr+0x228/0x4b0 net/ipv6/mcast.c:2234 > > > ipv6_mc_dad_complete+0x88/0x490 net/ipv6/mcast.c:2245 > > > addrconf_dad_completed+0x712/0xcd0 net/ipv6/addrconf.c:4342 > > > addrconf_dad_work+0xdc2/0x16f0 > > > process_one_work kernel/workqueue.c:3229 [inline] > > > process_scheduled_works+0xa63/0x1850 kernel/workqueue.c:3310 > > > > > > Memory state around the buggy address: > > > ffff888043eba080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > ffff888043eba100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > >ffff888043eba180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > ^ > > > ffff888043eba200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > ffff888043eba280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > > > > > > Fixes: 8c55facecd7a ("net: linkwatch: only report IF_OPER_LOWERLAYERDOWN if iflink is actually down") > > > Reported-by: syzbot+1939f24bdb783e9e43d9@syzkaller.appspotmail.com > > > Closes: https://lore.kernel.org/netdev/674f3a18.050a0220.48a03.0041.GAE@google.com/T/#u > > > Signed-off-by: Eric Dumazet <edumazet@google.com> > > > --- > > > Cc: Vladimir Oltean <vladimir.oltean@nxp.com> > > > --- > > > net/core/link_watch.c | 7 ++++++- > > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > diff --git a/net/core/link_watch.c b/net/core/link_watch.c > > > index ab150641142aa1545c71fc5d3b11db33c70cf437..1b4d39e38084272269a51503c217fc1e5a1326eb 100644 > > > --- a/net/core/link_watch.c > > > +++ b/net/core/link_watch.c > > > @@ -45,9 +45,14 @@ static unsigned int default_operstate(const struct net_device *dev) > > > int iflink = dev_get_iflink(dev); > > > struct net_device *peer; > > > > > > - if (iflink == dev->ifindex) > > > + /* If called from netdev_run_todo()/linkwatch_sync_dev(), > > > + * dev_net(dev) can be already freed, and RTNL is not held. > > > + */ > > > + if (dev->reg_state == NETREG_UNREGISTERED || > > > + iflink == dev->ifindex) > > > return IF_OPER_DOWN; > > > > > > + ASSERT_RTNL(); > > > peer = __dev_get_by_index(dev_net(dev), iflink); > > > if (!peer) > > > return IF_OPER_DOWN; > > > -- > > > 2.47.0.338.g60cca15819-goog > > > > > > > Thanks for submitting a patch, the issue makes sense. > > > > Question: is the rtnl_mutex actually held in the problematic case though? > > The netdev_run_todo() call path is: > > As explained in the comment, RTNL is not held in this case : > > /* If called from netdev_run_todo()/linkwatch_sync_dev(), > * dev_net(dev) can be already freed, and RTNL is not held. > */ > > In the future, we might change default_operstate() to use dev_get_by_index_rcu() > and not rely on RTNL anymore, but after this patch, the ASSERT_RTNL() is fine. Ah, ASSERT_RTNL() is only done if the reg_state is not NETREG_UNREGISTERED. I misinterpreted the order of operations and thought it is asserted unconditionally. My bad. > > > > __rtnl_unlock(); <- unlocks > > > > /* Wait for rcu callbacks to finish before next phase */ > > if (!list_empty(&list)) > > rcu_barrier(); > > > > list_for_each_entry_safe(dev, tmp, &list, todo_list) { > > if (unlikely(dev->reg_state != NETREG_UNREGISTERING)) { > > netdev_WARN(dev, "run_todo but not unregistering\n"); > > list_del(&dev->todo_list); > > continue; > > } > > > > WRITE_ONCE(dev->reg_state, NETREG_UNREGISTERED); > > // reg_state is set to NETREG_UNREGISTERING > > > linkwatch_sync_dev(dev); <- asserts > > } > > > > And on the same note: does linkwatch not have a chance to run also, > > concurrently with us, in this timeframe? Could we not catch the > > dev->reg_state in NETREG_UNREGISTERING? > > I guess we can add a READ_ONCE() on many dev->reg_state reads. > > The race should not matter for linkwatch, if the device is going away. I meant: linkwatch runs periodically, via linkwatch_event(). Isn't there a chance that linkwatch_event() can run once, immediately after __rtnl_unlock() in netdev_run_todo(), while the netdev is in the NETREG_UNREGISTERING state? Won't that create problems for __dev_get_by_index() too? I guess it depends on when the netns is torn down, which I couldn't find.
On Wed, Dec 4, 2024 at 12:41 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > I meant: linkwatch runs periodically, via linkwatch_event(). Isn't there > a chance that linkwatch_event() can run once, immediately after > __rtnl_unlock() in netdev_run_todo(), while the netdev is in the > NETREG_UNREGISTERING state? Won't that create problems for __dev_get_by_index() > too? I guess it depends on when the netns is torn down, which I couldn't find. I think lweventlist_lock and dev->link_watch_list are supposed to synchronize things. linkwatch_sync_dev() only calls linkwatch_do_dev() if the device was atomically unlinked from lweventlist
On Wed, Dec 04, 2024 at 12:46:11PM +0100, Eric Dumazet wrote: > On Wed, Dec 4, 2024 at 12:41 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > > > I meant: linkwatch runs periodically, via linkwatch_event(). Isn't there > > a chance that linkwatch_event() can run once, immediately after > > __rtnl_unlock() in netdev_run_todo(), while the netdev is in the > > NETREG_UNREGISTERING state? Won't that create problems for __dev_get_by_index() > > too? I guess it depends on when the netns is torn down, which I couldn't find. > > I think lweventlist_lock and dev->link_watch_list are supposed to > synchronize things. > > linkwatch_sync_dev() only calls linkwatch_do_dev() if the device was > atomically unlinked from lweventlist No, I don't mean calls from linkwatch_sync_dev(). I mean other call paths towards linkwatch_do_dev(), like for example linkwatch_fire_event() - carrier down, whatever. Can't these be pending on an unregistering net_device at the time we run __rtnl_unlock() in netdev_run_todo? Otherwise, why would netdev_wait_allrefs_any() have a linkwatch_run_queue() call just later?
On Wed, Dec 4, 2024 at 1:57 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > On Wed, Dec 04, 2024 at 12:46:11PM +0100, Eric Dumazet wrote: > > On Wed, Dec 4, 2024 at 12:41 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > > > > > I meant: linkwatch runs periodically, via linkwatch_event(). Isn't there > > > a chance that linkwatch_event() can run once, immediately after > > > __rtnl_unlock() in netdev_run_todo(), while the netdev is in the > > > NETREG_UNREGISTERING state? Won't that create problems for __dev_get_by_index() > > > too? I guess it depends on when the netns is torn down, which I couldn't find. > > > > I think lweventlist_lock and dev->link_watch_list are supposed to > > synchronize things. > > > > linkwatch_sync_dev() only calls linkwatch_do_dev() if the device was > > atomically unlinked from lweventlist > > No, I don't mean calls from linkwatch_sync_dev(). I mean other call > paths towards linkwatch_do_dev(), like for example linkwatch_fire_event() - > carrier down, whatever. Can't these be pending on an unregistering > net_device at the time we run __rtnl_unlock() in netdev_run_todo? > Otherwise, why would netdev_wait_allrefs_any() have a linkwatch_run_queue() > call just later? I do not know, this predates git history. All these questions seem orthogonal. My patch fixes an issue added recently. not something added 10 years ago. I suggest we fix proven issues first, step by step. If you want to take over and send a series, just say so. Thank you.
On Wed, Dec 04, 2024 at 02:38:14PM +0100, Eric Dumazet wrote: > On Wed, Dec 4, 2024 at 1:57 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > > > On Wed, Dec 04, 2024 at 12:46:11PM +0100, Eric Dumazet wrote: > > > On Wed, Dec 4, 2024 at 12:41 PM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > > > > > > > I meant: linkwatch runs periodically, via linkwatch_event(). Isn't there > > > > a chance that linkwatch_event() can run once, immediately after > > > > __rtnl_unlock() in netdev_run_todo(), while the netdev is in the > > > > NETREG_UNREGISTERING state? Won't that create problems for __dev_get_by_index() > > > > too? I guess it depends on when the netns is torn down, which I couldn't find. > > > > > > I think lweventlist_lock and dev->link_watch_list are supposed to > > > synchronize things. > > > > > > linkwatch_sync_dev() only calls linkwatch_do_dev() if the device was > > > atomically unlinked from lweventlist > > > > No, I don't mean calls from linkwatch_sync_dev(). I mean other call > > paths towards linkwatch_do_dev(), like for example linkwatch_fire_event() - > > carrier down, whatever. Can't these be pending on an unregistering > > net_device at the time we run __rtnl_unlock() in netdev_run_todo? > > Otherwise, why would netdev_wait_allrefs_any() have a linkwatch_run_queue() > > call just later? > > I do not know, this predates git history. > > All these questions seem orthogonal. > My patch fixes an issue added recently. not something added 10 years ago. > I suggest we fix proven issues first, step by step. > If you want to take over and send a series, just say so. > > Thank you. My understanding is certainly fuzzy, but I am not talking about some behavior from 10 years ago. If I made default_operstate() require rtnl_mutex last year, I did so for all call paths, not just for the direct linkwatch_sync_dev() call that you point out. I agree we can take them step by step if the UNREGISTERING state also proves problematic (I don't have enough data now), but I disagree that the problem is orthogonal.
On Tue, Dec 03, 2024 at 05:09:33PM +0000, Eric Dumazet wrote: > syzbot reported an UAF in default_operstate() [1] > > Issue is a race between device and netns dismantles. > > After calling __rtnl_unlock() from netdev_run_todo(), > we can not assume the netns of each device is still alive. > > Make sure the device is not in NETREG_UNREGISTERED state, > and add an ASSERT_RTNL() before the call to > __dev_get_by_index(). > > We might move this ASSERT_RTNL() in __dev_get_by_index() > in the future. > > Fixes: 8c55facecd7a ("net: linkwatch: only report IF_OPER_LOWERLAYERDOWN if iflink is actually down") > Reported-by: syzbot+1939f24bdb783e9e43d9@syzkaller.appspotmail.com > Closes: https://lore.kernel.org/netdev/674f3a18.050a0220.48a03.0041.GAE@google.com/T/#u > Signed-off-by: Eric Dumazet <edumazet@google.com> > --- > Cc: Vladimir Oltean <vladimir.oltean@nxp.com> > --- For now: Reviewed-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Hello: This patch was applied to netdev/net.git (main) by Paolo Abeni <pabeni@redhat.com>: On Tue, 3 Dec 2024 17:09:33 +0000 you wrote: > syzbot reported an UAF in default_operstate() [1] > > Issue is a race between device and netns dismantles. > > After calling __rtnl_unlock() from netdev_run_todo(), > we can not assume the netns of each device is still alive. > > [...] Here is the summary with links: - [net] net: avoid potential UAF in default_operstate() https://git.kernel.org/netdev/net/c/750e51603395 You are awesome, thank you!
diff --git a/net/core/link_watch.c b/net/core/link_watch.c index ab150641142aa1545c71fc5d3b11db33c70cf437..1b4d39e38084272269a51503c217fc1e5a1326eb 100644 --- a/net/core/link_watch.c +++ b/net/core/link_watch.c @@ -45,9 +45,14 @@ static unsigned int default_operstate(const struct net_device *dev) int iflink = dev_get_iflink(dev); struct net_device *peer; - if (iflink == dev->ifindex) + /* If called from netdev_run_todo()/linkwatch_sync_dev(), + * dev_net(dev) can be already freed, and RTNL is not held. + */ + if (dev->reg_state == NETREG_UNREGISTERED || + iflink == dev->ifindex) return IF_OPER_DOWN; + ASSERT_RTNL(); peer = __dev_get_by_index(dev_net(dev), iflink); if (!peer) return IF_OPER_DOWN;