From patchwork Tue Mar 17 12:04:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 11442633 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98DB11668 for ; Tue, 17 Mar 2020 12:04:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 79D7920724 for ; Tue, 17 Mar 2020 12:04:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gq9QEJ9S" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726733AbgCQMEz (ORCPT ); Tue, 17 Mar 2020 08:04:55 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:54799 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725868AbgCQMEz (ORCPT ); Tue, 17 Mar 2020 08:04:55 -0400 Received: by mail-wm1-f65.google.com with SMTP id n8so21141297wmc.4 for ; Tue, 17 Mar 2020 05:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=COkwzP8/QBM86a4+9gtNcPvvXdjnZzsPYEyWNAYkIZ0=; b=gq9QEJ9S/wyWVpx2FefEKGaBOUqGOzpWwM9RFtoLItm0FEQm2hAUhvT3weMZj63qG6 UcZckOL0sXib/7ssQX3vd4MK1T8j/1GOQKxvUVquQuONQq97JUyjLuGYGsgkzEAaP/HN gE2MLM1OUgZBP7l8MErsJlj0811oFBruRpHp2xUPYO/YuYjanHrsVhdZOf7dZTwVLnF9 GF4xU/BIc3PDU052W3qdQUdT/d3vhn4WnaHUNne+UecZ6dtsknVR8NCQO4uGB/ihUPSD UWhkK6IN4TFVTIN2SBDR4u1CDKSU0f9Rv3+OWHmbAojWSoMYigbhB5opCoDzqUWrpQnG EnrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=COkwzP8/QBM86a4+9gtNcPvvXdjnZzsPYEyWNAYkIZ0=; b=jAiuc5aXglMG3d3B9Uec07yVBLled4ADu8EaoCTv4GIS6kaqUOrYKidzYSVszFjiJv 2obliGHY2Vrl3nsvpjxMm+1++KM/3AO7gMH65OH+RICp4F9wtAf8IhQAyjGhoHiGp8/o Gt04RUB0N/4Tc17sAclwx0b1vu36r7Pd9duBDWmy0O6fnt9F8JRWayVQLk8aI8vnwun+ 4vGbND2nfqM3CkvxSQ5VViQO5ycpFXEORc+zxhpw7CWUKWX2P6ZSfNAa0BcLNfa7CoyG 5MrSoEVBC+B5xxadGYJKZkskc3RXbLXWeY/Ms6IegDRctGEIPgour3d8f3MezeTJy/gC tnKg== X-Gm-Message-State: ANhLgQ20jOwLfk8MV+DGbU1LFbzDIjkKVTdRVlbA19aMefHMVI6CtD8p X7TIuNvNqnoOX2RPLzJXxfeXJosWWSc= X-Google-Smtp-Source: ADFU+vvnyvFqeD8p1T4dwIKBkT+dYE1/varWt1bSpUJOtoPQtHUnPkHuSgl3SUdyIEjcjGapaC9jQA== X-Received: by 2002:a7b:cdfa:: with SMTP id p26mr5194130wmj.39.1584446691973; Tue, 17 Mar 2020 05:04:51 -0700 (PDT) Received: from kwango.local (ip-94-112-129-237.net.upcbroadband.cz. [94.112.129.237]) by smtp.gmail.com with ESMTPSA id p8sm4416706wrw.19.2020.03.17.05.04.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 05:04:49 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Subject: [PATCH 1/3] rbd: avoid a deadlock on header_rwsem when flushing notifies Date: Tue, 17 Mar 2020 13:04:20 +0100 Message-Id: <20200317120422.3406-2-idryomov@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20200317120422.3406-1-idryomov@gmail.com> References: <20200317120422.3406-1-idryomov@gmail.com> MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org rbd_unregister_watch() flushes notifies and therefore cannot be called under header_rwsem because a header update notify takes header_rwsem to synchronize with "rbd map". If mapping an image fails after the watch is established and a header update notify sneaks in, we deadlock when erroring out from rbd_dev_image_probe(). Move watch registration and unregistration out of the critical section. The only reason they were put there was to make header_rwsem management slightly more obvious. Fixes: 811c66887746 ("rbd: fix rbd map vs notify races") Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index a4e7b494344c..f0ce30a6fc69 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -4527,6 +4527,10 @@ static void cancel_tasks_sync(struct rbd_device *rbd_dev) cancel_work_sync(&rbd_dev->unlock_work); } +/* + * header_rwsem must not be held to avoid a deadlock with + * rbd_dev_refresh() when flushing notifies. + */ static void rbd_unregister_watch(struct rbd_device *rbd_dev) { cancel_tasks_sync(rbd_dev); @@ -6907,6 +6911,8 @@ static void rbd_dev_image_release(struct rbd_device *rbd_dev) * device. If this image is the one being mapped (i.e., not a * parent), initiate a watch on its header object before using that * object to get detailed information about the rbd image. + * + * On success, returns with header_rwsem held for write. */ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth) { @@ -6936,6 +6942,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth) } } + down_write(&rbd_dev->header_rwsem); ret = rbd_dev_header_info(rbd_dev); if (ret) { if (ret == -ENOENT && !need_watch) @@ -6987,6 +6994,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth) err_out_probe: rbd_dev_unprobe(rbd_dev); err_out_watch: + up_write(&rbd_dev->header_rwsem); if (need_watch) rbd_unregister_watch(rbd_dev); err_out_format: @@ -7050,12 +7058,9 @@ static ssize_t do_rbd_add(struct bus_type *bus, goto err_out_rbd_dev; } - down_write(&rbd_dev->header_rwsem); rc = rbd_dev_image_probe(rbd_dev, 0); - if (rc < 0) { - up_write(&rbd_dev->header_rwsem); + if (rc < 0) goto err_out_rbd_dev; - } if (rbd_dev->opts->alloc_size > rbd_dev->layout.object_size) { rbd_warn(rbd_dev, "alloc_size adjusted to %u", From patchwork Tue Mar 17 12:04:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 11442635 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DBDB92A for ; Tue, 17 Mar 2020 12:05:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F219C20724 for ; Tue, 17 Mar 2020 12:04:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mOXHvMKb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726736AbgCQME7 (ORCPT ); Tue, 17 Mar 2020 08:04:59 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:39197 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725868AbgCQME7 (ORCPT ); Tue, 17 Mar 2020 08:04:59 -0400 Received: by mail-wr1-f66.google.com with SMTP id h6so4962501wrs.6 for ; Tue, 17 Mar 2020 05:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=JqNH+AB/pWHQv45BKXgKcEwpSBlHFvw8DFsiMZU8sAo=; b=mOXHvMKbzKUEd4hQL36LInWipxxOiPg3Re6TleV0V2ogztqtI1zVSbXovEKYdUQLn7 doGak2APFZhw2GMUUNhpCkw/i4WZui1kqB6bszrULx6wqulcR4vuvuJQ7Ba1RK1gDZ8e 5MrVcZVwJOsvpTJHejBrWZmliA9lvrfaDbm7HQvlBlNcFleMc987xxBdjgOymsEQWs3W I9Q/IRqpOFCc92yQGND0MoOlWThlBex8m6tzrL2w5fTp0JCuuTlv+vzvSZdUpz+udZ7v KxWNOSRfZWId1UzSHiy+UkQKdIZj0pRRgVODpjQL+0OVsBO7UOFHS3QqFLzHAiDXfXRz Baqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JqNH+AB/pWHQv45BKXgKcEwpSBlHFvw8DFsiMZU8sAo=; b=IVnyGxdt6F2IX8lLp4z46SissM/l6qs0M9U5pqecKindgRuB88QCf4FkF65Sq22VKP bYBiNJmBDbS9Q1Zav4+Yug3vY9hYbzLq4YwvogT+aBeEJo2CXbcnCfGY9htf6nC/L/l5 l5qv52Qj6vG9O9KsLQz5dyl9zfJcO7KFCSvdtwrRXNJ3VyxAQw1iADrV4Gyj2xDtCLlP sSZaGGfkb2Z+lQ8YDv2a8HueguKd8cnDl0lM1DZdfWRk4uKYcGbEqkgtklyXvgASW1wi T3Be06EV/cBxml2slY1Mv+llJ3BmSalO/r3skYoeTbsX/E2ifYoGf4QTVMi9pplpfJI0 2hCg== X-Gm-Message-State: ANhLgQ1yaaICDlEgU8eEgNF8QEzpCe0VsS1ddMqjoG6Xx0p6q2Ls3Gkh UaYuVslERYscP9F6aO/Tv/xMlg/MxOI= X-Google-Smtp-Source: ADFU+vsEy2aIAvcQ8aqns01G7FtGe6htFH3X2O2TAzt/M2qcGlr0IGd6eW3gGpwoTm2CEBXES226cA== X-Received: by 2002:a05:6000:41:: with SMTP id k1mr6040978wrx.53.1584446696781; Tue, 17 Mar 2020 05:04:56 -0700 (PDT) Received: from kwango.local (ip-94-112-129-237.net.upcbroadband.cz. [94.112.129.237]) by smtp.gmail.com with ESMTPSA id p8sm4416706wrw.19.2020.03.17.05.04.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 05:04:52 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Subject: [PATCH 2/3] rbd: call rbd_dev_unprobe() after unwatching and flushing notifies Date: Tue, 17 Mar 2020 13:04:21 +0100 Message-Id: <20200317120422.3406-3-idryomov@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20200317120422.3406-1-idryomov@gmail.com> References: <20200317120422.3406-1-idryomov@gmail.com> MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org rbd_dev_unprobe() is supposed to undo most of rbd_dev_image_probe(), including rbd_dev_header_info(), which means that rbd_dev_header_info() isn't supposed to be called after rbd_dev_unprobe(). However, rbd_dev_image_release() calls rbd_dev_unprobe() before rbd_unregister_watch(). This is racy because a header update notify can sneak in: "rbd unmap" thread ceph-watch-notify worker rbd_dev_image_release() rbd_dev_unprobe() free and zero out header rbd_watch_cb() rbd_dev_refresh() rbd_dev_header_info() read in header The same goes for "rbd map" because rbd_dev_image_probe() calls rbd_dev_unprobe() on errors. In both cases this results in a memory leak. Fixes: fd22aef8b47c ("rbd: move rbd_unregister_watch() call into rbd_dev_image_release()") Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index f0ce30a6fc69..e590dc484c18 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -6898,9 +6898,10 @@ static void rbd_print_dne(struct rbd_device *rbd_dev, bool is_snap) static void rbd_dev_image_release(struct rbd_device *rbd_dev) { - rbd_dev_unprobe(rbd_dev); if (rbd_dev->opts) rbd_unregister_watch(rbd_dev); + + rbd_dev_unprobe(rbd_dev); rbd_dev->image_format = 0; kfree(rbd_dev->spec->image_id); rbd_dev->spec->image_id = NULL; @@ -6947,7 +6948,7 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth) if (ret) { if (ret == -ENOENT && !need_watch) rbd_print_dne(rbd_dev, false); - goto err_out_watch; + goto err_out_probe; } /* @@ -6992,11 +6993,10 @@ static int rbd_dev_image_probe(struct rbd_device *rbd_dev, int depth) return 0; err_out_probe: - rbd_dev_unprobe(rbd_dev); -err_out_watch: up_write(&rbd_dev->header_rwsem); if (need_watch) rbd_unregister_watch(rbd_dev); + rbd_dev_unprobe(rbd_dev); err_out_format: rbd_dev->image_format = 0; kfree(rbd_dev->spec->image_id); From patchwork Tue Mar 17 12:04:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 11442637 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBC9192A for ; Tue, 17 Mar 2020 12:05:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A96C920724 for ; Tue, 17 Mar 2020 12:05:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="THmO/1y2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726741AbgCQMFD (ORCPT ); Tue, 17 Mar 2020 08:05:03 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43041 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725868AbgCQMFC (ORCPT ); Tue, 17 Mar 2020 08:05:02 -0400 Received: by mail-wr1-f68.google.com with SMTP id b2so19062412wrj.10 for ; Tue, 17 Mar 2020 05:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cBUpox85RxX9oUWWDs/rPz95nR2HzjMae7+1O3FTjKI=; b=THmO/1y2uMXi2jLZ5V0LjMy+8whCSgSy6HTRC0U4KBe8+fKhLg+Sx+XGWW+7FH0pj9 u9mqBBR1g99Xou8D1CGuPW3rJsClDy4rV0/RqlWlFjfglCLQNIk6Mqva/L2CA0SshLh0 QmSoTyJYQXXQ9y9n3tIiQpz20076EILiX0LMEzIQpKnoO5VQfN2E2E7Fpp3f7HD+zwRJ iQTXQCYK/C9LCiEaz3iuD3ZWx6lqUOVu2c0g9xcado2cP84HglF9bDJMysrVg1xk4T8b sMfUCR4YWmJGKTvUgGHaLV2tamrnShW+9Pe9YTz8ki+NuobgPSfpM/kR18e+Aqf7Ocn2 aidA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cBUpox85RxX9oUWWDs/rPz95nR2HzjMae7+1O3FTjKI=; b=OIwt4/r5ld8Mah29TGmeegY9par51qvfnFIfb/1OSPNkoNvdUBgkBFbKcYL1uu6pJ3 doeVwLbEVrCWptpGeqiU7iaKJc1BM02oDizhep/mn5/peGxLcRDiyyPFMAGjnxRZCXoY pTV5oCVG4ZghkQ8r9kbEw4fLRaLwwxlKd83FkruNcntpVU+DxIXdmMLmIXmiZ+Ghz+2i yQVWVk7+V0gHkAiCkNfcI0RmaW+bPN+c1QCLez/B78CAx5+V4wgF1jhAolV2wZlZP6D3 PsGKoZOxgoXceACr+IF3kWPufYEsaDfLqisW/ymncuYt18hFWPtsFz5kil6Bxo833yqk dCYA== X-Gm-Message-State: ANhLgQ38kpwJQMrYDb8DrxqUr4aK3krXEFjdEVgxSI5cuJ0vE44Shhka TNvWlJuBF/howqE+4zM99OG5Kwx59fA= X-Google-Smtp-Source: ADFU+vvJxxQyT8YNcdTwIzGkkySkr+gtGUL3G2Vt/BBCllxLxM8Izr0BCFoHR1mwQDGVJMpbs6Kd6g== X-Received: by 2002:a5d:6591:: with SMTP id q17mr5546268wru.22.1584446700460; Tue, 17 Mar 2020 05:05:00 -0700 (PDT) Received: from kwango.local (ip-94-112-129-237.net.upcbroadband.cz. [94.112.129.237]) by smtp.gmail.com with ESMTPSA id p8sm4416706wrw.19.2020.03.17.05.04.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 05:04:57 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Subject: [PATCH 3/3] rbd: don't test rbd_dev->opts in rbd_dev_image_release() Date: Tue, 17 Mar 2020 13:04:22 +0100 Message-Id: <20200317120422.3406-4-idryomov@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20200317120422.3406-1-idryomov@gmail.com> References: <20200317120422.3406-1-idryomov@gmail.com> MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org rbd_dev->opts is used to distinguish between the image that is being mapped and a parent. However, because we no longer establish watch for read-only mappings, this test is imprecise and results in unnecessary rbd_unregister_watch() calls. Make it consistent with need_watch in rbd_dev_image_probe(). Fixes: b9ef2b8858a0 ("rbd: don't establish watch for read-only mappings") Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index e590dc484c18..f44ce9ccadd6 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -6898,7 +6898,7 @@ static void rbd_print_dne(struct rbd_device *rbd_dev, bool is_snap) static void rbd_dev_image_release(struct rbd_device *rbd_dev) { - if (rbd_dev->opts) + if (!rbd_is_ro(rbd_dev)) rbd_unregister_watch(rbd_dev); rbd_dev_unprobe(rbd_dev);