[UPSTREAM,1/1] rds: Fix possible deadlock in rds_message_put

Commit Message

Allison Henderson Jan. 26, 2024, 5:26 p.m. UTC

From: Allison Henderson <[e-mail]>

Functions rds_still_queued and rds_clear_recv_queue lock a given socket
in order to safely iterate over the incoming rds messages. However
calling rds_inc_put while under this lock creates a potential deadlock.
rds_inc_put may eventually call rds_message_purge, which will lock
m_rs_lock. This is the incorrect locking order since m_rs_lock is
meant to be locked before the socket. To fix this, we move the message
item to a local list or variable that wont need rs_recv_lock protection.
Then we can safely call rds_inc_put on any item stored locally after
rs_recv_lock is released.

Fixes: possible deadlock in rds_message_put
Reported-by: [e-mail]

Signed-off-by: Allison Henderson <[e-mail]>
---
 net/rds/recv.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

Comments

Allison Henderson Jan. 26, 2024, 5:33 p.m. UTC | #1

On Fri, 2024-01-26 at 10:26 -0700, Allison Henderson via rds-devel
wrote:
> [rds-devel] [PATCH UPSTREAM 
Opps, this should just be "[PATCH] net/rds." Apologies for confusion

Allison

Jakub Kicinski Jan. 27, 2024, 1:28 a.m. UTC | #2

On Fri, 26 Jan 2024 10:26:52 -0700 [e-mail] wrote:
> +	if (to_drop) {
> +		rds_inc_put(to_drop);
> +
>  	rdsdebug("inc %p rs %p still %d dropped %d\n", inc, rs, ret, drop);
>  	return ret;
>  }

This does not build..

Allison Henderson Jan. 29, 2024, 9:03 p.m. UTC | #3

On Fri, 2024-01-26 at 17:28 -0800, Jakub Kicinski wrote:
> On Fri, 26 Jan 2024 10:26:52 -0700
> [e-mail] wrote:
> > +       if (to_drop) {
> > +               rds_inc_put(to_drop);
> > +
> >         rdsdebug("inc %p rs %p still %d dropped %d\n", inc, rs,
> > ret, drop);
> >         return ret;
> >  }
> 
> This does not build..
Opps, apologies, I will re-check my build configs.  Thank you!

Allison

Patch

diff --git a/net/rds/recv.c b/net/rds/recv.c
index c71b923764fd..d5fb0a22a9a2 100644
--- a/net/rds/recv.c
+++ b/net/rds/recv.c
@@ -425,6 +425,7 @@  static int rds_still_queued(struct rds_sock *rs, struct rds_incoming *inc,
 	struct sock *sk = rds_rs_to_sk(rs);
 	int ret = 0;
 	unsigned long flags;
+	struct rds_incoming *to_drop = NULL;
 
 	write_lock_irqsave(&rs->rs_recv_lock, flags);
 	if (!list_empty(&inc->i_item)) {
@@ -435,11 +436,14 @@  static int rds_still_queued(struct rds_sock *rs, struct rds_incoming *inc,
 					      -be32_to_cpu(inc->i_hdr.h_len),
 					      inc->i_hdr.h_dport);
 			list_del_init(&inc->i_item);
-			rds_inc_put(inc);
+			to_drop = inc;
 		}
 	}
 	write_unlock_irqrestore(&rs->rs_recv_lock, flags);
 
+	if (to_drop) {
+		rds_inc_put(to_drop);
+
 	rdsdebug("inc %p rs %p still %d dropped %d\n", inc, rs, ret, drop);
 	return ret;
 }
@@ -758,16 +762,21 @@  void rds_clear_recv_queue(struct rds_sock *rs)
 	struct sock *sk = rds_rs_to_sk(rs);
 	struct rds_incoming *inc, *tmp;
 	unsigned long flags;
+	LIST_HEAD(to_drop);
 
 	write_lock_irqsave(&rs->rs_recv_lock, flags);
 	list_for_each_entry_safe(inc, tmp, &rs->rs_recv_queue, i_item) {
 		rds_recv_rcvbuf_delta(rs, sk, inc->i_conn->c_lcong,
 				      -be32_to_cpu(inc->i_hdr.h_len),
 				      inc->i_hdr.h_dport);
+		list_move(&inc->i_item, &to_drop);
+	}
+	write_unlock_irqrestore(&rs->rs_recv_lock, flags);
+
+	list_for_each_entry_safe(inc, tmp, &to_drop, i_item) {
 		list_del_init(&inc->i_item);
 		rds_inc_put(inc);
 	}
-	write_unlock_irqrestore(&rs->rs_recv_lock, flags);
 }
 
 /*