diff mbox series

[for-next,3/3] RDMA/rtrs: Avoid use-after-free in rtrs_clt_rdma_cm_handler

Message ID 1681108984-2-4-git-send-email-lizhijian@fujitsu.com (mailing list archive)
State Changes Requested
Headers show
Series rtrs bugfix and cleanups | expand

Commit Message

Li Zhijian April 10, 2023, 6:43 a.m. UTC 
Currently, con will be destroyed when wait_event_interruptible_timeout()
returns ERESTARTSYS. But the in-flight event handler
rtrs_clt_rdma_cm_handler() could be rescheduled/wakeup which
may cause a use-after-free.

    WARNING: CPU: 0 PID: 14766 at drivers/infiniband/ulp/rtrs/rtrs-clt.c:1687 rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
     Modules linked in: rnbd_client rtrs_client rtrs_core rdma_cm iw_cm ib_cm rdma_rxe ib_uverbs ib_core libiscsi scsi_transport_iscsi crc32_generic udp_tunnel dax_pmem nd_pmem nd_btt virtiofs crc32c_intel nvme fuse nvme_core nfit
libnvdimm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
     CPU: 0 PID: 14766 Comm: kworker/u2:3 Kdump: loaded Tainted: G        W          6.2.0-rc6-roce-flush+ #56
     Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
     Workqueue: ib_addr process_one_req [ib_core]
     RIP: 0010:rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
     Code: 00 0f 85 5f fd ff ff 4c 8b 23 41 bd f4 ff ff ff e9 95 fb ff ff 0f 0b 4c 89 f7 41 bd ea ff ff ff e8 75 c8 92 ec e9 4b ff ff ff <0f> 0b 4c 89 f7 41 bd ea ff ff ff e8 60 c8 92 ec e9 36 ff ff ff e8
     RSP: 0018:ffffa4ef41cdbc60 EFLAGS: 00010246
     RAX: 0000000000000000 RBX: ffff9372c394e600 RCX: 0000000000000001
     RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffad634277
     RBP: ffffa4ef41cdbd00 R08: 0000000000000000 R09: 0000000000000001
     R10: 0000000000003ff3 R11: 0000000000000000 R12: ffff9372c3164800
     R13: ffff9372c3164800 R14: ffff9372c394e640 R15: ffff9372c5219020
     FS:  0000000000000000(0000) GS:ffff9372fbc00000(0000) knlGS:0000000000000000
     CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     CR2: 00007f35bb7d5de0 CR3: 0000000020c2a006 CR4: 00000000001706f0
     DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
     Call Trace:
      <TASK>
      ? mark_held_locks+0x49/0x80
      ? lock_is_held_type+0xd7/0x130
      ? cma_cm_event_handler+0x49/0x200 [rdma_cm]
      cma_cm_event_handler+0x49/0x200 [rdma_cm]
      addr_handler+0xf1/0x1e0 [rdma_cm]
      ? lock_acquire+0xca/0x2f0
      ? lock_acquire+0xda/0x2f0
      process_one_req+0x43/0x170 [ib_core]
      process_one_work+0x274/0x590
      worker_thread+0x4f/0x3d0
      ? __pfx_worker_thread+0x10/0x10
      kthread+0xe7/0x110
      ? __pfx_kthread+0x10/0x10
      ret_from_fork+0x2c/0x50
      </TASK>
     irq event stamp: 1432669
     hardirqs last  enabled at (1432683): [<ffffffffac508eb2>] __up_console_sem+0x52/0x60
     hardirqs last disabled at (1432698): [<ffffffffac508e97>] __up_console_sem+0x37/0x60
     softirqs last  enabled at (1432518): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
     softirqs last disabled at (1432509): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
     ---[ end trace 0000000000000000 ]---

Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
---
 drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

Comments

Leon Romanovsky April 10, 2023, 12:10 p.m. UTC | #1 
On Mon, Apr 10, 2023 at 06:43:04AM +0000, Li Zhijian wrote:
> Currently, con will be destroyed when wait_event_interruptible_timeout()
> returns ERESTARTSYS. But the in-flight event handler
> rtrs_clt_rdma_cm_handler() could be rescheduled/wakeup which
> may cause a use-after-free.
> 
>     WARNING: CPU: 0 PID: 14766 at drivers/infiniband/ulp/rtrs/rtrs-clt.c:1687 rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>      Modules linked in: rnbd_client rtrs_client rtrs_core rdma_cm iw_cm ib_cm rdma_rxe ib_uverbs ib_core libiscsi scsi_transport_iscsi crc32_generic udp_tunnel dax_pmem nd_pmem nd_btt virtiofs crc32c_intel nvme fuse nvme_core nfit
> libnvdimm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
>      CPU: 0 PID: 14766 Comm: kworker/u2:3 Kdump: loaded Tainted: G        W          6.2.0-rc6-roce-flush+ #56
>      Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
>      Workqueue: ib_addr process_one_req [ib_core]
>      RIP: 0010:rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>      Code: 00 0f 85 5f fd ff ff 4c 8b 23 41 bd f4 ff ff ff e9 95 fb ff ff 0f 0b 4c 89 f7 41 bd ea ff ff ff e8 75 c8 92 ec e9 4b ff ff ff <0f> 0b 4c 89 f7 41 bd ea ff ff ff e8 60 c8 92 ec e9 36 ff ff ff e8
>      RSP: 0018:ffffa4ef41cdbc60 EFLAGS: 00010246
>      RAX: 0000000000000000 RBX: ffff9372c394e600 RCX: 0000000000000001
>      RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffad634277
>      RBP: ffffa4ef41cdbd00 R08: 0000000000000000 R09: 0000000000000001
>      R10: 0000000000003ff3 R11: 0000000000000000 R12: ffff9372c3164800
>      R13: ffff9372c3164800 R14: ffff9372c394e640 R15: ffff9372c5219020
>      FS:  0000000000000000(0000) GS:ffff9372fbc00000(0000) knlGS:0000000000000000
>      CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>      CR2: 00007f35bb7d5de0 CR3: 0000000020c2a006 CR4: 00000000001706f0
>      DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>      DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>      Call Trace:
>       <TASK>
>       ? mark_held_locks+0x49/0x80
>       ? lock_is_held_type+0xd7/0x130
>       ? cma_cm_event_handler+0x49/0x200 [rdma_cm]
>       cma_cm_event_handler+0x49/0x200 [rdma_cm]
>       addr_handler+0xf1/0x1e0 [rdma_cm]
>       ? lock_acquire+0xca/0x2f0
>       ? lock_acquire+0xda/0x2f0
>       process_one_req+0x43/0x170 [ib_core]
>       process_one_work+0x274/0x590
>       worker_thread+0x4f/0x3d0
>       ? __pfx_worker_thread+0x10/0x10
>       kthread+0xe7/0x110
>       ? __pfx_kthread+0x10/0x10
>       ret_from_fork+0x2c/0x50
>       </TASK>
>      irq event stamp: 1432669
>      hardirqs last  enabled at (1432683): [<ffffffffac508eb2>] __up_console_sem+0x52/0x60
>      hardirqs last disabled at (1432698): [<ffffffffac508e97>] __up_console_sem+0x37/0x60
>      softirqs last  enabled at (1432518): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>      softirqs last disabled at (1432509): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>      ---[ end trace 0000000000000000 ]---
> 
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
> ---
>  drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> index 4c8f42e46e2f..760a7eb51297 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> @@ -2074,6 +2074,7 @@ static int create_cm(struct rtrs_clt_con *con)
>  		rtrs_err(s, "Failed to resolve address, err: %d\n", err);
>  		goto destroy_cm;
>  	}
> +again:
>  	/*
>  	 * Combine connection status and session events. This is needed
>  	 * for waiting two possible cases: cm_err has something meaningful
> @@ -2083,10 +2084,15 @@ static int create_cm(struct rtrs_clt_con *con)
>  			clt_path->state_wq,
>  			con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
>  			msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
> -	if (err == 0 || err == -ERESTARTSYS) {
> -		if (err == 0)
> -			err = -ETIMEDOUT;
> -		/* Timedout or interrupted */
> +	if (err == -ERESTARTSYS) {
> +		/* interrupted,
> +		 * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
> +		 * getting a use-after-free
> +		 */
> +		goto again;
> +	} else if (err == 0) {
> +		err = -ETIMEDOUT;
> +		/* Timedout */

Timedout -> Timeout

>  		goto errr;

errrr, sigh.

>  	}
>  	if (con->cm_err < 0) {
> -- 
> 2.29.2
>
Guoqing Jiang April 10, 2023, 1:13 p.m. UTC | #2 
On 4/10/23 14:43, Li Zhijian wrote:
> Currently, con will be destroyed when wait_event_interruptible_timeout()
> returns ERESTARTSYS. But the in-flight event handler
> rtrs_clt_rdma_cm_handler() could be rescheduled/wakeup which
> may cause a use-after-free.
>
>      WARNING: CPU: 0 PID: 14766 at drivers/infiniband/ulp/rtrs/rtrs-clt.c:1687 rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>       Modules linked in: rnbd_client rtrs_client rtrs_core rdma_cm iw_cm ib_cm rdma_rxe ib_uverbs ib_core libiscsi scsi_transport_iscsi crc32_generic udp_tunnel dax_pmem nd_pmem nd_btt virtiofs crc32c_intel nvme fuse nvme_core nfit
> libnvdimm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
>       CPU: 0 PID: 14766 Comm: kworker/u2:3 Kdump: loaded Tainted: G        W          6.2.0-rc6-roce-flush+ #56
>       Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
>       Workqueue: ib_addr process_one_req [ib_core]
>       RIP: 0010:rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>       Code: 00 0f 85 5f fd ff ff 4c 8b 23 41 bd f4 ff ff ff e9 95 fb ff ff 0f 0b 4c 89 f7 41 bd ea ff ff ff e8 75 c8 92 ec e9 4b ff ff ff <0f> 0b 4c 89 f7 41 bd ea ff ff ff e8 60 c8 92 ec e9 36 ff ff ff e8
>       RSP: 0018:ffffa4ef41cdbc60 EFLAGS: 00010246
>       RAX: 0000000000000000 RBX: ffff9372c394e600 RCX: 0000000000000001
>       RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffad634277
>       RBP: ffffa4ef41cdbd00 R08: 0000000000000000 R09: 0000000000000001
>       R10: 0000000000003ff3 R11: 0000000000000000 R12: ffff9372c3164800
>       R13: ffff9372c3164800 R14: ffff9372c394e640 R15: ffff9372c5219020
>       FS:  0000000000000000(0000) GS:ffff9372fbc00000(0000) knlGS:0000000000000000
>       CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>       CR2: 00007f35bb7d5de0 CR3: 0000000020c2a006 CR4: 00000000001706f0
>       DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>       DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>       Call Trace:
>        <TASK>
>        ? mark_held_locks+0x49/0x80
>        ? lock_is_held_type+0xd7/0x130
>        ? cma_cm_event_handler+0x49/0x200 [rdma_cm]
>        cma_cm_event_handler+0x49/0x200 [rdma_cm]
>        addr_handler+0xf1/0x1e0 [rdma_cm]
>        ? lock_acquire+0xca/0x2f0
>        ? lock_acquire+0xda/0x2f0
>        process_one_req+0x43/0x170 [ib_core]
>        process_one_work+0x274/0x590
>        worker_thread+0x4f/0x3d0
>        ? __pfx_worker_thread+0x10/0x10
>        kthread+0xe7/0x110
>        ? __pfx_kthread+0x10/0x10
>        ret_from_fork+0x2c/0x50
>        </TASK>
>       irq event stamp: 1432669
>       hardirqs last  enabled at (1432683): [<ffffffffac508eb2>] __up_console_sem+0x52/0x60
>       hardirqs last disabled at (1432698): [<ffffffffac508e97>] __up_console_sem+0x37/0x60
>       softirqs last  enabled at (1432518): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>       softirqs last disabled at (1432509): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>       ---[ end trace 0000000000000000 ]---
>
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
> ---
>   drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++++++----
>   1 file changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> index 4c8f42e46e2f..760a7eb51297 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> @@ -2074,6 +2074,7 @@ static int create_cm(struct rtrs_clt_con *con)
>   		rtrs_err(s, "Failed to resolve address, err: %d\n", err);
>   		goto destroy_cm;
>   	}
> +again:
>   	/*
>   	 * Combine connection status and session events. This is needed
>   	 * for waiting two possible cases: cm_err has something meaningful
> @@ -2083,10 +2084,15 @@ static int create_cm(struct rtrs_clt_con *con)
>   			clt_path->state_wq,
>   			con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
>   			msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
> -	if (err == 0 || err == -ERESTARTSYS) {
> -		if (err == 0)
> -			err = -ETIMEDOUT;
> -		/* Timedout or interrupted */
> +	if (err == -ERESTARTSYS) {
> +		/* interrupted,
> +		 * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
> +		 * getting a use-after-free
> +		 */
> +		goto again;
> +	} else if (err == 0) {
> +		err = -ETIMEDOUT;
> +		/* Timedout */
>   		goto errr;
>   	}

Can event handler still be triggered in case of timeout?

And I guess either stop_cm -> rdma_disconnect or destroy_cm -> 
rdma_destroy_id
should prevent this kind of racy issue.

Thanks,
Guoqing
Li Zhijian April 11, 2023, 1:33 a.m. UTC | #3 
On 10/04/2023 21:13, Guoqing Jiang wrote:
> 
> 
> On 4/10/23 14:43, Li Zhijian wrote:
>> Currently, con will be destroyed when wait_event_interruptible_timeout()
>> returns ERESTARTSYS. But the in-flight event handler
>> rtrs_clt_rdma_cm_handler() could be rescheduled/wakeup which
>> may cause a use-after-free.
>>
>>      WARNING: CPU: 0 PID: 14766 at drivers/infiniband/ulp/rtrs/rtrs-clt.c:1687 rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>>       Modules linked in: rnbd_client rtrs_client rtrs_core rdma_cm iw_cm ib_cm rdma_rxe ib_uverbs ib_core libiscsi scsi_transport_iscsi crc32_generic udp_tunnel dax_pmem nd_pmem nd_btt virtiofs crc32c_intel nvme fuse nvme_core nfit
>> libnvdimm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
>>       CPU: 0 PID: 14766 Comm: kworker/u2:3 Kdump: loaded Tainted: G        W          6.2.0-rc6-roce-flush+ #56
>>       Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
>>       Workqueue: ib_addr process_one_req [ib_core]
>>       RIP: 0010:rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>>       Code: 00 0f 85 5f fd ff ff 4c 8b 23 41 bd f4 ff ff ff e9 95 fb ff ff 0f 0b 4c 89 f7 41 bd ea ff ff ff e8 75 c8 92 ec e9 4b ff ff ff <0f> 0b 4c 89 f7 41 bd ea ff ff ff e8 60 c8 92 ec e9 36 ff ff ff e8
>>       RSP: 0018:ffffa4ef41cdbc60 EFLAGS: 00010246
>>       RAX: 0000000000000000 RBX: ffff9372c394e600 RCX: 0000000000000001
>>       RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffad634277
>>       RBP: ffffa4ef41cdbd00 R08: 0000000000000000 R09: 0000000000000001
>>       R10: 0000000000003ff3 R11: 0000000000000000 R12: ffff9372c3164800
>>       R13: ffff9372c3164800 R14: ffff9372c394e640 R15: ffff9372c5219020
>>       FS:  0000000000000000(0000) GS:ffff9372fbc00000(0000) knlGS:0000000000000000
>>       CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>       CR2: 00007f35bb7d5de0 CR3: 0000000020c2a006 CR4: 00000000001706f0
>>       DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>       DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>       Call Trace:
>>        <TASK>
>>        ? mark_held_locks+0x49/0x80
>>        ? lock_is_held_type+0xd7/0x130
>>        ? cma_cm_event_handler+0x49/0x200 [rdma_cm]
>>        cma_cm_event_handler+0x49/0x200 [rdma_cm]
>>        addr_handler+0xf1/0x1e0 [rdma_cm]
>>        ? lock_acquire+0xca/0x2f0
>>        ? lock_acquire+0xda/0x2f0
>>        process_one_req+0x43/0x170 [ib_core]
>>        process_one_work+0x274/0x590
>>        worker_thread+0x4f/0x3d0
>>        ? __pfx_worker_thread+0x10/0x10
>>        kthread+0xe7/0x110
>>        ? __pfx_kthread+0x10/0x10
>>        ret_from_fork+0x2c/0x50
>>        </TASK>
>>       irq event stamp: 1432669
>>       hardirqs last  enabled at (1432683): [<ffffffffac508eb2>] __up_console_sem+0x52/0x60
>>       hardirqs last disabled at (1432698): [<ffffffffac508e97>] __up_console_sem+0x37/0x60
>>       softirqs last  enabled at (1432518): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>>       softirqs last disabled at (1432509): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>>       ---[ end trace 0000000000000000 ]---
>>
>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
>> ---
>>   drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++++++----
>>   1 file changed, 10 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> index 4c8f42e46e2f..760a7eb51297 100644
>> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> @@ -2074,6 +2074,7 @@ static int create_cm(struct rtrs_clt_con *con)
>>           rtrs_err(s, "Failed to resolve address, err: %d\n", err);
>>           goto destroy_cm;
>>       }
>> +again:
>>       /*
>>        * Combine connection status and session events. This is needed
>>        * for waiting two possible cases: cm_err has something meaningful
>> @@ -2083,10 +2084,15 @@ static int create_cm(struct rtrs_clt_con *con)
>>               clt_path->state_wq,
>>               con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
>>               msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
>> -    if (err == 0 || err == -ERESTARTSYS) {
>> -        if (err == 0)
>> -            err = -ETIMEDOUT;
>> -        /* Timedout or interrupted */
>> +    if (err == -ERESTARTSYS) {
>> +        /* interrupted,
>> +         * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
>> +         * getting a use-after-free
>> +         */
>> +        goto again;
>> +    } else if (err == 0) {
>> +        err = -ETIMEDOUT;
>> +        /* Timedout */
>>           goto errr;
>>       }
> 
> Can event handler still be triggered in case of timeout?

I have never hit such race.

> 
> And I guess either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id
> should prevent this kind of racy issue.

In practice, they are possible that rtrs_clt_rdma_cm_handler() is in-flight during
'either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id'. rtrs_clt_rdma_cm_handler() and
cm's cleanup path need to hold mutex_lock(&con->con_mutex), once cm's cleanup path get this lock first
rtrs_clt_rdma_cm_handler has to sleep, when rtrs_clt_rdma_cm_handler is wakeup again, some resources has been
freed by cm's cleanup path.

Thanks

> 
> Thanks,
> Guoqing
Guoqing Jiang April 12, 2023, 1:15 a.m. UTC | #4 
On 4/11/23 09:33, Zhijian Li (Fujitsu) wrote:
>
>>> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>>> index 4c8f42e46e2f..760a7eb51297 100644
>>> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>>> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>>> @@ -2074,6 +2074,7 @@ static int create_cm(struct rtrs_clt_con *con)
>>>            rtrs_err(s, "Failed to resolve address, err: %d\n", err);
>>>            goto destroy_cm;
>>>        }
>>> +again:
>>>        /*
>>>         * Combine connection status and session events. This is needed
>>>         * for waiting two possible cases: cm_err has something meaningful
>>> @@ -2083,10 +2084,15 @@ static int create_cm(struct rtrs_clt_con *con)
>>>                clt_path->state_wq,
>>>                con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
>>>                msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
>>> -    if (err == 0 || err == -ERESTARTSYS) {
>>> -        if (err == 0)
>>> -            err = -ETIMEDOUT;
>>> -        /* Timedout or interrupted */
>>> +    if (err == -ERESTARTSYS) {
>>> +        /* interrupted,
>>> +         * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
>>> +         * getting a use-after-free
>>> +         */
>>> +        goto again;
>>> +    } else if (err == 0) {
>>> +        err = -ETIMEDOUT;
>>> +        /* Timedout */
>>>            goto errr;
>>>        }
>> Can event handler still be triggered in case of timeout?
> I have never hit such race.

But it is still possible with the theory.

>> And I guess either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id
>> should prevent this kind of racy issue.
> In practice, they are possible that rtrs_clt_rdma_cm_handler() is in-flight during
> 'either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id'. rtrs_clt_rdma_cm_handler() and
> cm's cleanup path need to hold mutex_lock(&con->con_mutex), once cm's cleanup path get this lock first
> rtrs_clt_rdma_cm_handler has to sleep, when rtrs_clt_rdma_cm_handler is wakeup again, some resources has been
> freed by cm's cleanup path.

First, stop_cm doesn't need to hold &con->con_mutex. But 
rtrs_clt_rdma_cm_handler
is called from rdma core layer which need  id_priv->handler_mutexinstead 
of con_mutex
I thinnk. Also RTRS has similar behavior as nvme host rdma, ib_srp and 
iser_verb.

@Jinpu/Haris, can we move destroy_cm right after stop_cm?*

*Thanks,
Guoqing
diff mbox series

Patch

diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 4c8f42e46e2f..760a7eb51297 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -2074,6 +2074,7 @@  static int create_cm(struct rtrs_clt_con *con)
 		rtrs_err(s, "Failed to resolve address, err: %d\n", err);
 		goto destroy_cm;
 	}
+again:
 	/*
 	 * Combine connection status and session events. This is needed
 	 * for waiting two possible cases: cm_err has something meaningful
@@ -2083,10 +2084,15 @@  static int create_cm(struct rtrs_clt_con *con)
 			clt_path->state_wq,
 			con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
 			msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
-	if (err == 0 || err == -ERESTARTSYS) {
-		if (err == 0)
-			err = -ETIMEDOUT;
-		/* Timedout or interrupted */
+	if (err == -ERESTARTSYS) {
+		/* interrupted,
+		 * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
+		 * getting a use-after-free
+		 */
+		goto again;
+	} else if (err == 0) {
+		err = -ETIMEDOUT;
+		/* Timedout */
 		goto errr;
 	}
 	if (con->cm_err < 0) {