[net] rxrpc: fix a race in rxrpc_exit_net()

Commit Message

Eric Dumazet April 4, 2022, 6:34 p.m. UTC

From: Eric Dumazet <[e-mail]>

Current code can lead to the following race:

CPU0                                                 CPU1

rxrpc_exit_net()
                                                     rxrpc_peer_keepalive_worker()
                                                       if (rxnet->live)

  rxnet->live = false;
  del_timer_sync(&rxnet->peer_keepalive_timer);

                                                             timer_reduce(&rxnet->peer_keepalive_timer, jiffies + delay);

  cancel_work_sync(&rxnet->peer_keepalive_work);

rxrpc_exit_net() exits while peer_keepalive_timer is still armed,
leading to use-after-free.

syzbot report was:

ODEBUG: free active (active state 0) object type: timer_list hint: rxrpc_peer_keepalive_timeout+0x0/0xb0
WARNING: CPU: 0 PID: 3660 at lib/debugobjects.c:505 debug_print_object+0x16e/0x250 lib/debugobjects.c:505
Modules linked in:
CPU: 0 PID: 3660 Comm: kworker/u4:6 Not tainted 5.17.0-syzkaller-13993-g88e6c0207623 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
RIP: 0010:debug_print_object+0x16e/0x250 lib/debugobjects.c:505
Code: ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 af 00 00 00 48 8b 14 dd 00 1c 26 8a 4c 89 ee 48 c7 c7 00 10 26 8a e8 b1 e7 28 05 <0f> 0b 83 05 15 eb c5 09 01 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3
RSP: 0018:ffffc9000353fb00 EFLAGS: 00010082
RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
RDX: ffff888029196140 RSI: ffffffff815efad8 RDI: fffff520006a7f52
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff815ea4ae R11: 0000000000000000 R12: ffffffff89ce23e0
R13: ffffffff8a2614e0 R14: ffffffff816628c0 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe1f2908924 CR3: 0000000043720000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 __debug_check_no_obj_freed lib/debugobjects.c:992 [inline]
 debug_check_no_obj_freed+0x301/0x420 lib/debugobjects.c:1023
 kfree+0xd6/0x310 mm/slab.c:3809
 ops_free_list.part.0+0x119/0x370 net/core/net_namespace.c:176
 ops_free_list net/core/net_namespace.c:174 [inline]
 cleanup_net+0x591/0xb00 net/core/net_namespace.c:598
 process_one_work+0x996/0x1610 kernel/workqueue.c:2289
 worker_thread+0x665/0x1080 kernel/workqueue.c:2436
 kthread+0x2e9/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:298
 </TASK>

Fixes: ace45bec6d77 ("rxrpc: Fix firewall route keepalive")
Signed-off-by: Eric Dumazet <[e-mail]>
Cc: David Howells <[e-mail]>
Cc: Marc Dionne <[e-mail]>
Cc: [e-mail]
Reported-by: syzbot <[e-mail]>
---
 net/rxrpc/net_ns.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

[e-mail] April 6, 2022, 1:10 p.m. UTC | #1

Hello:

This patch was applied to netdev/net.git (master)
by David S. Miller <[e-mail]>:

On Mon,  4 Apr 2022 11:34:39 -0700 you wrote:
> From: Eric Dumazet <[e-mail]>
> 
> Current code can lead to the following race:
> 
> CPU0                                                 CPU1
> 
> rxrpc_exit_net()
>                                                      rxrpc_peer_keepalive_worker()
>                                                        if (rxnet->live)
> 
> [...]

Here is the summary with links:
  - [net] rxrpc: fix a race in rxrpc_exit_net()
    https://git.kernel.org/netdev/net/c/1946014ca3b1

You are awesome, thank you!

David Howells April 8, 2022, 6:41 a.m. UTC | #2

Hi Eric,

[Note that your patch is appl/octet-stream for some reason.]

> 	rxnet->live = false;
> -	del_timer_sync(&rxnet->peer_keepalive_timer);
>  	cancel_work_sync(&rxnet->peer_keepalive_work);
> +	del_timer_sync(&rxnet->peer_keepalive_timer);

That fixes that problem, but introduces another.  The timer could be in the
throes of queueing the worker:

	CPU 1			CPU 2
	====================	=====================
				if (rxnet->live)
				<INTERRUPT>
	rxnet->live = false;
 	cancel_work_sync(&rxnet->peer_keepalive_work);
				rxrpc_queue_work(&rxnet->peer_keepalive_work);
	del_timer_sync(&rxnet->peer_keepalive_timer);

I think keeping the first del_timer_sync() that you removed and the one after
the cancel would be sufficient.

David

David Howells April 12, 2022, 8:06 a.m. UTC | #3

Do you have a syzbot ref for this?

David

Eric Dumazet April 12, 2022, 4:33 p.m. UTC | #4

On 4/12/22 01:06, David Howells wrote:
> Do you have a syzbot ref for this?

Try: https://syzkaller.appspot.com/bug?extid=724378c4bb58f703b09a


> David
>

Patch

diff --git a/net/rxrpc/net_ns.c b/net/rxrpc/net_ns.c
index 25bbc4cc8b1359f7b895f181dad227de088ed31d..f15d6942da45306e4fa15399473044281dcbfed9 100644
--- a/net/rxrpc/net_ns.c
+++ b/net/rxrpc/net_ns.c
@@ -113,8 +113,8 @@  static __net_exit void rxrpc_exit_net(struct net *net)
 	struct rxrpc_net *rxnet = rxrpc_net(net);
 
 	rxnet->live = false;
-	del_timer_sync(&rxnet->peer_keepalive_timer);
 	cancel_work_sync(&rxnet->peer_keepalive_work);
+	del_timer_sync(&rxnet->peer_keepalive_timer);
 	rxrpc_destroy_all_calls(rxnet);
 	rxrpc_destroy_all_connections(rxnet);
 	rxrpc_destroy_all_peers(rxnet);