From patchwork Thu Jun 1 09:42:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saravanan Vajravel X-Patchwork-Id: 13263239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26650C77B7E for ; Thu, 1 Jun 2023 09:42:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229834AbjFAJmw (ORCPT ); Thu, 1 Jun 2023 05:42:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232834AbjFAJmt (ORCPT ); Thu, 1 Jun 2023 05:42:49 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31DB812F for ; Thu, 1 Jun 2023 02:42:39 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-53fdae76f3aso597938a12.0 for ; Thu, 01 Jun 2023 02:42:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1685612558; x=1688204558; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=F81y+PuQpgox/76IQYtFXpCDM7Hna6HG9+ZmfSqXBYE=; b=Y2xtKh8NBldD2mpzfCytPN6TVeRGkWyMlxM8aV4m0f8/SGAFrR7p1KxWSNVlakovXG u4lWOkJEwIIxzpojw7jHAy89LCuGxn5eiPG17ds6hkhon1mQ8u2ztLnWMQqUBbG+8l2u qpQsPHMOI742tlqSmeO90wZNENjZy8VXpyUTc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685612558; x=1688204558; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F81y+PuQpgox/76IQYtFXpCDM7Hna6HG9+ZmfSqXBYE=; b=IMXbPcJdq6BpotGsSE3CQwni7nkw0ENNM9kRtcvPRuSGAN5R8s2d8g5f4JkC2zAiUG ILnaby0te1GnGJkHql13dk13JGRyskt6Lp8fWm+dkYDDTm3L0UeytEzInazpx0qNyqFw MGJlK+0jZM86acxUHfqPl6apLI2KoQp0cttnkWbCeeMj013xtGFYJAlfd1R0Mj69mMVP o4d1JY1ZZwi4Ud6iy969U/s4WwGmkH8umGjMqn0k6lJ8CIqxpsjnIWn1hkhaCY9tzb/4 wV6kDlC3MCy1clsJLN8wUXJ6upvTqzV+vNyvxys5BxwHIfRyzvzZ4p19GsUVuaw41Jnf kLDg== X-Gm-Message-State: AC+VfDwWbkoQimbIdatlQsZ1dR+ynn3sv/cz1lk1bUV6XJr/bzRw47mu 9A6vtECf2yV9FZyHXk6I56T0wg== X-Google-Smtp-Source: ACHHUZ5YKEZkRkU70gkyit31XqeefGUbxQRjbDqwDCS9iZBnfLfsaskcuQHja2CP49wqckOMizqoFA== X-Received: by 2002:a05:6a20:e486:b0:10b:cdb1:3563 with SMTP id ni6-20020a056a20e48600b0010bcdb13563mr7925268pzb.46.1685612558386; Thu, 01 Jun 2023 02:42:38 -0700 (PDT) Received: from localhost.localdomain ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id e12-20020a63ee0c000000b00502e7115cbdsm2744960pgi.51.2023.06.01.02.42.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jun 2023 02:42:37 -0700 (PDT) From: Saravanan Vajravel To: selvin.xavier@broadcom.com, jgg@ziepe.ca, leon@kernel.org, sagi@grimberg.me Cc: linux-rdma@vger.kernel.org, Saravanan Vajravel Subject: [PATCH for-rc 1/3] IB/isert: Fix dead lock in ib_isert Date: Thu, 1 Jun 2023 02:42:18 -0700 Message-Id: <20230601094220.64810-2-saravanan.vajravel@broadcom.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> References: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org - When a iSER session is released, ib_isert module is taking a mutex lock and releasing all pending connections. As part of this, ib_isert is destroying rdma cm_id. To destroy cm_id, rdma_cm module is sending CM events to CMA handler of ib_isert. This handler is taking same mutex lock. Hence it leads to deadlock between ib_isert & rdma_cm modules. - For fix, created local list of pending connections and release the connection outside of mutex lock. Calltrace: --------- [ 1229.791410] INFO: task kworker/10:1:642 blocked for more than 120 seconds. [ 1229.791416] Tainted: G OE --------- - - 4.18.0-372.9.1.el8.x86_64 #1 [ 1229.791418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1229.791419] task:kworker/10:1 state:D stack: 0 pid: 642 ppid: 2 flags:0x80004000 [ 1229.791424] Workqueue: ib_cm cm_work_handler [ib_cm] [ 1229.791436] Call Trace: [ 1229.791438] __schedule+0x2d1/0x830 [ 1229.791445] ? select_idle_sibling+0x23/0x6f0 [ 1229.791449] schedule+0x35/0xa0 [ 1229.791451] schedule_preempt_disabled+0xa/0x10 [ 1229.791453] __mutex_lock.isra.7+0x310/0x420 [ 1229.791456] ? select_task_rq_fair+0x351/0x990 [ 1229.791459] isert_cma_handler+0x224/0x330 [ib_isert] [ 1229.791463] ? ttwu_queue_wakelist+0x159/0x170 [ 1229.791466] cma_cm_event_handler+0x25/0xd0 [rdma_cm] [ 1229.791474] cma_ib_handler+0xa7/0x2e0 [rdma_cm] [ 1229.791478] cm_process_work+0x22/0xf0 [ib_cm] [ 1229.791483] cm_work_handler+0xf4/0xf30 [ib_cm] [ 1229.791487] ? move_linked_works+0x6e/0xa0 [ 1229.791490] process_one_work+0x1a7/0x360 [ 1229.791491] ? create_worker+0x1a0/0x1a0 [ 1229.791493] worker_thread+0x30/0x390 [ 1229.791494] ? create_worker+0x1a0/0x1a0 [ 1229.791495] kthread+0x10a/0x120 [ 1229.791497] ? set_kthread_struct+0x40/0x40 [ 1229.791499] ret_from_fork+0x1f/0x40 [ 1229.791739] INFO: task targetcli:28666 blocked for more than 120 seconds. [ 1229.791740] Tainted: G OE --------- - - 4.18.0-372.9.1.el8.x86_64 #1 [ 1229.791741] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1229.791742] task:targetcli state:D stack: 0 pid:28666 ppid: 5510 flags:0x00004080 [ 1229.791743] Call Trace: [ 1229.791744] __schedule+0x2d1/0x830 [ 1229.791746] schedule+0x35/0xa0 [ 1229.791748] schedule_preempt_disabled+0xa/0x10 [ 1229.791749] __mutex_lock.isra.7+0x310/0x420 [ 1229.791751] rdma_destroy_id+0x15/0x20 [rdma_cm] [ 1229.791755] isert_connect_release+0x115/0x130 [ib_isert] [ 1229.791757] isert_free_np+0x87/0x140 [ib_isert] [ 1229.791761] iscsit_del_np+0x74/0x120 [iscsi_target_mod] [ 1229.791776] lio_target_np_driver_store+0xe9/0x140 [iscsi_target_mod] [ 1229.791784] configfs_write_file+0xb2/0x110 [ 1229.791788] vfs_write+0xa5/0x1a0 [ 1229.791792] ksys_write+0x4f/0xb0 [ 1229.791794] do_syscall_64+0x5b/0x1a0 [ 1229.791798] entry_SYSCALL_64_after_hwframe+0x65/0xca Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier Reviewed-by: Sagi Grimberg --- drivers/infiniband/ulp/isert/ib_isert.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index f290cd49698e..b3471ac82c1a 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2431,6 +2431,7 @@ isert_free_np(struct iscsi_np *np) { struct isert_np *isert_np = np->np_context; struct isert_conn *isert_conn, *n; + LIST_HEAD(drop_conn_list); if (isert_np->cm_id) rdma_destroy_id(isert_np->cm_id); @@ -2450,7 +2451,7 @@ isert_free_np(struct iscsi_np *np) node) { isert_info("cleaning isert_conn %p state (%d)\n", isert_conn, isert_conn->state); - isert_connect_release(isert_conn); + list_move_tail(&isert_conn->node, &drop_conn_list) } } @@ -2461,11 +2462,16 @@ isert_free_np(struct iscsi_np *np) node) { isert_info("cleaning isert_conn %p state (%d)\n", isert_conn, isert_conn->state); - isert_connect_release(isert_conn); + list_move_tail(&isert_conn->node, &drop_conn_list); } } mutex_unlock(&isert_np->mutex); + list_for_each_entry_safe(isert_conn, n, &drop_conn_list, node) { + list_del_init(&isert_conn->node); + isert_connect_release(isert_conn); + } + np->np_context = NULL; kfree(isert_np); } From patchwork Thu Jun 1 09:42:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saravanan Vajravel X-Patchwork-Id: 13263240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EB46C77B7E for ; Thu, 1 Jun 2023 09:42:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232834AbjFAJm4 (ORCPT ); Thu, 1 Jun 2023 05:42:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232713AbjFAJmz (ORCPT ); Thu, 1 Jun 2023 05:42:55 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6253B1B0 for ; Thu, 1 Jun 2023 02:42:43 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-25690e009c8so134324a91.0 for ; Thu, 01 Jun 2023 02:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1685612563; x=1688204563; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=UdbkOY64oCOpk6c4lcew+CZzCgf7GSFBj5w5kk41xZI=; b=fV7V2vba++Ph0rbPeQPR4eOn8RiK1C+TDKV4NnW2t6BggGPsCFRdG88zrzKqf0AdGO fBMmAD7e5ZgAgdN8cY86nXYBNNsWONApKXWBzQhDeVJApR1WfX2tDTlid37Q3pampNL4 jzwOlcZTDLYBP4sgtXgYGd1LmscBcHbDDmcfM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685612563; x=1688204563; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UdbkOY64oCOpk6c4lcew+CZzCgf7GSFBj5w5kk41xZI=; b=U8B+jy8IAJASPeHxV8eWfAJcDvnP+kCybi2iYVfZahm4JIy75dyx34Qrpm7SgIdEuE iId7A9Yo/PZGg3XyM4vD1Y7nJAODGQk/z2ydo1ahqymMVnPPe9KUmMyJcnoWN6f0qItt phgEPWcFO4AieLAlIKfkhnu2/wv15S8KbZxnMANAPBiKjkrPHtheCqLpZf5T8qtYfY3P tY0GtWkFadhYr6C8thyypIHS77Q1xO6n8XvvMzhmMwLNPLRrDnQhfIdWPLDLHVB9PvCw FjPLRz2A7fIQSPOq0fHQ6JOsgwuHlupo4le65ejKRzmq/2JgOaAQ3CTW8IuqxhmUbsMt qfww== X-Gm-Message-State: AC+VfDwmjcHj3JdFjdtLFvJnb0djGFDpWNySFEtukt27E2N2ZdCtnZ/z ivlb1k8H6Ko8PJ6YuBraqPfBmQ== X-Google-Smtp-Source: ACHHUZ6tww5MKcHls6RbhjWNubL7NLASMtkQ0pFpMFD0GJnqyw0GYpX5Zp+b1zhJ5Vl6bC8Y4PzkWQ== X-Received: by 2002:a17:90a:8a98:b0:256:31f3:1f03 with SMTP id x24-20020a17090a8a9800b0025631f31f03mr6029085pjn.21.1685612562534; Thu, 01 Jun 2023 02:42:42 -0700 (PDT) Received: from localhost.localdomain ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id e12-20020a63ee0c000000b00502e7115cbdsm2744960pgi.51.2023.06.01.02.42.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jun 2023 02:42:42 -0700 (PDT) From: Saravanan Vajravel To: selvin.xavier@broadcom.com, jgg@ziepe.ca, leon@kernel.org, sagi@grimberg.me Cc: linux-rdma@vger.kernel.org, Saravanan Vajravel Subject: [PATCH for-rc 2/3] IB/isert: Fix possible list corruption in CMA handler Date: Thu, 1 Jun 2023 02:42:19 -0700 Message-Id: <20230601094220.64810-3-saravanan.vajravel@broadcom.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> References: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org When ib_isert module receives connection error event, it is releasing the isert session and removes corresponding list node but it doesn't take appropriate mutex lock to remove the list node. This can lead to linked list corruption Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier --- drivers/infiniband/ulp/isert/ib_isert.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index b3471ac82c1a..64af8d966adf 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -657,11 +657,15 @@ static int isert_connect_error(struct rdma_cm_id *cma_id) { struct isert_conn *isert_conn = cma_id->qp->qp_context; + struct isert_np *isert_np = cma_id->context; ib_drain_qp(isert_conn->qp); + + mutex_lock(&isert_np->mutex); list_del_init(&isert_conn->node); isert_conn->cm_id = NULL; isert_put_conn(isert_conn); + mutex_unlock(&isert_np->mutex); return -1; } From patchwork Thu Jun 1 09:42:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saravanan Vajravel X-Patchwork-Id: 13263241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38C4BC77B7E for ; Thu, 1 Jun 2023 09:43:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231690AbjFAJnG (ORCPT ); Thu, 1 Jun 2023 05:43:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232297AbjFAJnE (ORCPT ); Thu, 1 Jun 2023 05:43:04 -0400 Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EDB1E62 for ; Thu, 1 Jun 2023 02:42:47 -0700 (PDT) Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-5555765c6d3so518280eaf.1 for ; Thu, 01 Jun 2023 02:42:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1685612566; x=1688204566; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=mJ5zBVvByf2yO78KESoPlYAeCNrqepIQbuqjnGer02c=; b=Dx6DRjFWcTAXM2OiWu3rS4bBStC/fiFvsxYdz0jhB1u6xQsCTWxrItifs1HhqcSGFo 5oP5GgNyiRM3x2eYwulBjo48/G6fBEtF3ieu/cgS1CAdCtlui88bOHuNrIKKEdoQNa0Y f3uEgSotMw7kmaxY803fErkpewISfDQTUh8zc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685612566; x=1688204566; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mJ5zBVvByf2yO78KESoPlYAeCNrqepIQbuqjnGer02c=; b=Rm3SLGFBkePU5hsGXK5XOynUg2nJ4NCoRAlLgGNwbHGwmaXtztKyKqasIbFPMGdK6b Kj8eHtsl47s/gLAfBbDZWvGBUfYLEZ/waoIdw0PD2U3EfI46zAZwgHamZoe7oFojVFhZ McZ4R6Yxeiax7D6kcrXrsKY9Myi0jwKQb92TueL6m/zxvVqmurezvTGwwkHBQFZBYhBG i0etvPI1KYYiaHtK/naemcivSQscC84GLH85sz7aGeSj3vdXdH6PFmeR9tom4naI53Xt QHd03y43w3Ky/2LOM3hgmma+/ORvHNSgPR7FkdfN5hp8sEp3CFPA2cxha4hNf/38ahnc c/Zg== X-Gm-Message-State: AC+VfDzBp2jjBcqHej9bnoK/KefVxQla9X18PbDaFqlvAJSws6WJ+E/N YHfGlRCR6X8lmt88U0eB1ji+Cw== X-Google-Smtp-Source: ACHHUZ5xeXOjpJsPf3JyNp/fePU3MWxW2mkmgK3q7CzdoNH2fj3Vcga2dcfdiONqhd0GpYGP0EgDFQ== X-Received: by 2002:a05:6358:9218:b0:125:80a4:4733 with SMTP id d24-20020a056358921800b0012580a44733mr6110251rwb.10.1685612566302; Thu, 01 Jun 2023 02:42:46 -0700 (PDT) Received: from localhost.localdomain ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id e12-20020a63ee0c000000b00502e7115cbdsm2744960pgi.51.2023.06.01.02.42.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jun 2023 02:42:46 -0700 (PDT) From: Saravanan Vajravel To: selvin.xavier@broadcom.com, jgg@ziepe.ca, leon@kernel.org, sagi@grimberg.me Cc: linux-rdma@vger.kernel.org, Saravanan Vajravel Subject: [PATCH for-rc 3/3] IB/isert: Fix incorrect release of isert connextion Date: Thu, 1 Jun 2023 02:42:20 -0700 Message-Id: <20230601094220.64810-4-saravanan.vajravel@broadcom.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> References: <20230601094220.64810-1-saravanan.vajravel@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The ib_isert module is releasing the isert connection both in isert_wait_conn() handler as well as isert_free_conn() handler. In isert_wait_conn() handler, it is expected to wait for iSCSI session logout operation to complete. It should free the isert connection only in isert_free_conn() handler. When a bunch of iSER target is cleared, this issue can lead to use-after-free memory issue as isert conn is twice released Signed-off-by: Saravanan Vajravel Signed-off-by: Selvin Xavier Reviewed-by: Sagi Grimberg --- drivers/infiniband/ulp/isert/ib_isert.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 64af8d966adf..873c8cbaaa5f 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2570,8 +2570,6 @@ static void isert_wait_conn(struct iscsit_conn *conn) isert_put_unsol_pending_cmds(conn); isert_wait4cmds(conn); isert_wait4logout(isert_conn); - - queue_work(isert_release_wq, &isert_conn->release_work); } static void isert_free_conn(struct iscsit_conn *conn)