From patchwork Tue Jul 25 21:28:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 13327056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF70AC04A6A for ; Tue, 25 Jul 2023 21:29:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229798AbjGYV3S (ORCPT ); Tue, 25 Jul 2023 17:29:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229626AbjGYV3Q (ORCPT ); Tue, 25 Jul 2023 17:29:16 -0400 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FCA01FE6 for ; Tue, 25 Jul 2023 14:29:13 -0700 (PDT) Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-4fb7dc16ff0so9241224e87.2 for ; Tue, 25 Jul 2023 14:29:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690320551; x=1690925351; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eL1WFQdHL939xNzGULY0ZSwlbWxB4jBimZZ1LkCYXgA=; b=Gkbcg116v093mc0+b/2znL2A+jlUpoOvTHgz3s2yJwyPFpkvjL0K6mtcjR0CO8HDTt hWG0IOrZF2dE6M3Ejw0vu4yI0q1embyYInpum0OuFPfTdzUb2NLVsLvEigEUP6H6Us9V nZHL9L/WDQixIiKB5hz3buXbh/x+ap1sQJjJw09JYzG066AOrKwcvpsuPWbVdo3c3vSA wOrpEbHN/lauhviOXXHFq+fSrPO3ezS+0YahtHO+ki+gH1iQ2YE/xr/gj9lDA7ybpg49 zb2JixAbjEHiHVdwu5Vt9Ie1YzRCImWdP/cFAmkQ7ZxWEF2K/cEbbRPP7mMo7+U6sXJJ FDAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320551; x=1690925351; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eL1WFQdHL939xNzGULY0ZSwlbWxB4jBimZZ1LkCYXgA=; b=Rfa5WQxAYlbF/yXgWI+VEQvhImiVBrzMYU0dbVxvw/B+RFCwJAOqcoZY5T17M8ru05 B1yJedsNop77/gl5dPaCSA7PLHQs+XJ0ohLkpgc0SNja8QV6aoFz2883MHdpVSNb3Sca si+eEOUNizpLCW8ZVe6WfNrtPzsnG43FUnIkKty+zNYNp9ednVTan3z9/Nai+KAfjoHv L0mXpO/EDiBfcU2BJhvNHzQ+MhEGIkzIYQTQ8svByFsIyEvO50TCG0fJqC06K3ZDAbdw 5X0MJcuUkG7zyN5ftuj8qOh0ptr2c5xVR+wxQe1akM4ZWwgafUPv+8kaGTWSKMM6WTmf oBUw== X-Gm-Message-State: ABy/qLboNHFhfqm+gFSeEMzOccHb6DM6OIyeGQ3HMzA41nPtj+POLx1F TuTCcKD/FGk11Caqs4tVNn1vzH574Q4= X-Google-Smtp-Source: APBJJlGf7fvtX7RWGrBsUBiCpwS6oWKqTekmS8LZv5gX3FLj0fvw6gUg2WmO7csx/ZZUI80DrnU06Q== X-Received: by 2002:a05:6512:2521:b0:4fa:ad2d:6c58 with SMTP id be33-20020a056512252100b004faad2d6c58mr74841lfb.61.1690320550600; Tue, 25 Jul 2023 14:29:10 -0700 (PDT) Received: from localhost.localdomain (ip-94-112-167-15.bb.vodafone.cz. [94.112.167.15]) by smtp.gmail.com with ESMTPSA id f23-20020a05640214d700b005224ec27dd7sm1200778edx.66.2023.07.25.14.29.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 14:29:10 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Cc: Dongsheng Yang Subject: [PATCH v2 1/3] rbd: make get_lock_owner_info() return a single locker or NULL Date: Tue, 25 Jul 2023 23:28:44 +0200 Message-ID: <20230725212847.137672-2-idryomov@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230725212847.137672-1-idryomov@gmail.com> References: <20230725212847.137672-1-idryomov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org Make the "num_lockers can be only 0 or 1" assumption explicit and simplify the API by getting rid of output parameters in preparation for calling get_lock_owner_info() twice before blocklisting. Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 84 +++++++++++++++++++++++++++------------------ 1 file changed, 51 insertions(+), 33 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index bd0e075a5d89..dca6c1e5f6bc 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3849,10 +3849,17 @@ static void wake_lock_waiters(struct rbd_device *rbd_dev, int result) list_splice_tail_init(&rbd_dev->acquiring_list, &rbd_dev->running_list); } -static int get_lock_owner_info(struct rbd_device *rbd_dev, - struct ceph_locker **lockers, u32 *num_lockers) +static void free_locker(struct ceph_locker *locker) +{ + if (locker) + ceph_free_lockers(locker, 1); +} + +static struct ceph_locker *get_lock_owner_info(struct rbd_device *rbd_dev) { struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc; + struct ceph_locker *lockers; + u32 num_lockers; u8 lock_type; char *lock_tag; int ret; @@ -3861,39 +3868,45 @@ static int get_lock_owner_info(struct rbd_device *rbd_dev, ret = ceph_cls_lock_info(osdc, &rbd_dev->header_oid, &rbd_dev->header_oloc, RBD_LOCK_NAME, - &lock_type, &lock_tag, lockers, num_lockers); - if (ret) - return ret; + &lock_type, &lock_tag, &lockers, &num_lockers); + if (ret) { + rbd_warn(rbd_dev, "failed to retrieve lockers: %d", ret); + return ERR_PTR(ret); + } - if (*num_lockers == 0) { + if (num_lockers == 0) { dout("%s rbd_dev %p no lockers detected\n", __func__, rbd_dev); + lockers = NULL; goto out; } if (strcmp(lock_tag, RBD_LOCK_TAG)) { rbd_warn(rbd_dev, "locked by external mechanism, tag %s", lock_tag); - ret = -EBUSY; - goto out; + goto err_busy; } if (lock_type == CEPH_CLS_LOCK_SHARED) { rbd_warn(rbd_dev, "shared lock type detected"); - ret = -EBUSY; - goto out; + goto err_busy; } - if (strncmp((*lockers)[0].id.cookie, RBD_LOCK_COOKIE_PREFIX, + WARN_ON(num_lockers != 1); + if (strncmp(lockers[0].id.cookie, RBD_LOCK_COOKIE_PREFIX, strlen(RBD_LOCK_COOKIE_PREFIX))) { rbd_warn(rbd_dev, "locked by external mechanism, cookie %s", - (*lockers)[0].id.cookie); - ret = -EBUSY; - goto out; + lockers[0].id.cookie); + goto err_busy; } out: kfree(lock_tag); - return ret; + return lockers; + +err_busy: + kfree(lock_tag); + ceph_free_lockers(lockers, num_lockers); + return ERR_PTR(-EBUSY); } static int find_watcher(struct rbd_device *rbd_dev, @@ -3947,51 +3960,56 @@ static int find_watcher(struct rbd_device *rbd_dev, static int rbd_try_lock(struct rbd_device *rbd_dev) { struct ceph_client *client = rbd_dev->rbd_client->client; - struct ceph_locker *lockers; - u32 num_lockers; + struct ceph_locker *locker; int ret; for (;;) { + locker = NULL; + ret = rbd_lock(rbd_dev); if (ret != -EBUSY) - return ret; + goto out; /* determine if the current lock holder is still alive */ - ret = get_lock_owner_info(rbd_dev, &lockers, &num_lockers); - if (ret) - return ret; - - if (num_lockers == 0) + locker = get_lock_owner_info(rbd_dev); + if (IS_ERR(locker)) { + ret = PTR_ERR(locker); + locker = NULL; + goto out; + } + if (!locker) goto again; - ret = find_watcher(rbd_dev, lockers); + ret = find_watcher(rbd_dev, locker); if (ret) goto out; /* request lock or error */ rbd_warn(rbd_dev, "breaking header lock owned by %s%llu", - ENTITY_NAME(lockers[0].id.name)); + ENTITY_NAME(locker->id.name)); ret = ceph_monc_blocklist_add(&client->monc, - &lockers[0].info.addr); + &locker->info.addr); if (ret) { - rbd_warn(rbd_dev, "blocklist of %s%llu failed: %d", - ENTITY_NAME(lockers[0].id.name), ret); + rbd_warn(rbd_dev, "failed to blocklist %s%llu: %d", + ENTITY_NAME(locker->id.name), ret); goto out; } ret = ceph_cls_break_lock(&client->osdc, &rbd_dev->header_oid, &rbd_dev->header_oloc, RBD_LOCK_NAME, - lockers[0].id.cookie, - &lockers[0].id.name); - if (ret && ret != -ENOENT) + locker->id.cookie, &locker->id.name); + if (ret && ret != -ENOENT) { + rbd_warn(rbd_dev, "failed to break header lock: %d", + ret); goto out; + } again: - ceph_free_lockers(lockers, num_lockers); + free_locker(locker); } out: - ceph_free_lockers(lockers, num_lockers); + free_locker(locker); return ret; } From patchwork Tue Jul 25 21:28:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 13327057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07C07C001DF for ; Tue, 25 Jul 2023 21:29:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229746AbjGYV3R (ORCPT ); Tue, 25 Jul 2023 17:29:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229611AbjGYV3Q (ORCPT ); Tue, 25 Jul 2023 17:29:16 -0400 Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D4BF1FEB for ; Tue, 25 Jul 2023 14:29:13 -0700 (PDT) Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-4fbb281eec6so9502850e87.1 for ; Tue, 25 Jul 2023 14:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690320552; x=1690925352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Si8u31JjykIlBGeMY3l1rmiBprxRVKXt5XCzLIWo/Bw=; b=gleUB2OaQ+8fsAkALfjqHESQQulHZmPU7nHDZw2R2IrzZyyyDzDBvOZEkC/T6wii+X Z95BKpKN1n156XC49BCZnksAoLiU+B7xSrfIvLZ0KElIJZWU/THmPTB2RvLOFMquqCeS YacsjY/sf24DYIflmAt3zsUnCde2+MH7zbo4wr+flG+gglSZzS9SMfkpQK8UuDImoVwq 13oehnMPMmBnR1kNCvDoSg4Rtu/Ny/U3iFeY/0DsXRg07NWHNbWYHBs25D4aGOy5dSqK zBdhTbIJXWqf+SuZPJezqw6OkFzSy9QPH+XuUMgfX/UM4V07Xz8ABHKtqi3fVNxUK1Kf TSBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320552; x=1690925352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Si8u31JjykIlBGeMY3l1rmiBprxRVKXt5XCzLIWo/Bw=; b=QDcdv7MyxoNiriEZp7nQDStRfQNw/FHlyFmna8TwV1bMZ/e+2Kw74OEvMfL9bLWzGS eTABqzZxoiWHbcNzIpAZLkzl8KXEOiHKDk/AyoY3fZ8qUPYM7xXU84YU0ulv9SNXEu4o pBy0bJlwVKKq4Dp6/l9OL6gfqXp/fLJMP48aV6I07+Vt5Cjl++A0Hzi8OP8XcBQvdO7Z BZkPGnDX5bh4kdeDty5yuEBzc98v1yWHu3BKkMH8CBGFqoGhdOnkK1tKlS10+jMWuAPJ jT87vvlAGwkgSpEUt/mnq6dyhCQELPHtzlFcZwEa2qxvO6EONEIwgYM3FQNDbZmdGaiZ eaIQ== X-Gm-Message-State: ABy/qLZRoA8NwhaRcnMXcKH+mbwOonGbTcwUVKZxterk+i78amfeX9rc 9arrJ5MzukCelLuWS3aXDNw2pIdIwxc= X-Google-Smtp-Source: APBJJlHcxE50i4gFVW58Vdd5aMm9nijBlF5DTyeQw2r+4DRuqcZnO+bh1mkBhWIpYJ+JfC+krSSqjA== X-Received: by 2002:a05:6512:202c:b0:4f8:7781:9870 with SMTP id s12-20020a056512202c00b004f877819870mr55716lfs.60.1690320551509; Tue, 25 Jul 2023 14:29:11 -0700 (PDT) Received: from localhost.localdomain (ip-94-112-167-15.bb.vodafone.cz. [94.112.167.15]) by smtp.gmail.com with ESMTPSA id f23-20020a05640214d700b005224ec27dd7sm1200778edx.66.2023.07.25.14.29.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 14:29:11 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Cc: Dongsheng Yang Subject: [PATCH v2 2/3] rbd: harden get_lock_owner_info() a bit Date: Tue, 25 Jul 2023 23:28:45 +0200 Message-ID: <20230725212847.137672-3-idryomov@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230725212847.137672-1-idryomov@gmail.com> References: <20230725212847.137672-1-idryomov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org - we want the exclusive lock type, so test for it directly - use sscanf() to actually parse the lock cookie and avoid admitting invalid handles - bail if locker has a blank address Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 21 +++++++++++++++------ net/ceph/messenger.c | 1 + 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index dca6c1e5f6bc..94629e826369 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3862,10 +3862,9 @@ static struct ceph_locker *get_lock_owner_info(struct rbd_device *rbd_dev) u32 num_lockers; u8 lock_type; char *lock_tag; + u64 handle; int ret; - dout("%s rbd_dev %p\n", __func__, rbd_dev); - ret = ceph_cls_lock_info(osdc, &rbd_dev->header_oid, &rbd_dev->header_oloc, RBD_LOCK_NAME, &lock_type, &lock_tag, &lockers, &num_lockers); @@ -3886,18 +3885,28 @@ static struct ceph_locker *get_lock_owner_info(struct rbd_device *rbd_dev) goto err_busy; } - if (lock_type == CEPH_CLS_LOCK_SHARED) { - rbd_warn(rbd_dev, "shared lock type detected"); + if (lock_type != CEPH_CLS_LOCK_EXCLUSIVE) { + rbd_warn(rbd_dev, "incompatible lock type detected"); goto err_busy; } WARN_ON(num_lockers != 1); - if (strncmp(lockers[0].id.cookie, RBD_LOCK_COOKIE_PREFIX, - strlen(RBD_LOCK_COOKIE_PREFIX))) { + ret = sscanf(lockers[0].id.cookie, RBD_LOCK_COOKIE_PREFIX " %llu", + &handle); + if (ret != 1) { rbd_warn(rbd_dev, "locked by external mechanism, cookie %s", lockers[0].id.cookie); goto err_busy; } + if (ceph_addr_is_blank(&lockers[0].info.addr)) { + rbd_warn(rbd_dev, "locker has a blank address"); + goto err_busy; + } + + dout("%s rbd_dev %p got locker %s%llu@%pISpc/%u handle %llu\n", + __func__, rbd_dev, ENTITY_NAME(lockers[0].id.name), + &lockers[0].info.addr.in_addr, + le32_to_cpu(lockers[0].info.addr.nonce), handle); out: kfree(lock_tag); diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c index cd7b0bf5369e..5eb4898cccd4 100644 --- a/net/ceph/messenger.c +++ b/net/ceph/messenger.c @@ -1123,6 +1123,7 @@ bool ceph_addr_is_blank(const struct ceph_entity_addr *addr) return true; } } +EXPORT_SYMBOL(ceph_addr_is_blank); int ceph_addr_port(const struct ceph_entity_addr *addr) { From patchwork Tue Jul 25 21:28:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 13327058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1766EB64DD for ; Tue, 25 Jul 2023 21:29:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229835AbjGYV3T (ORCPT ); Tue, 25 Jul 2023 17:29:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229678AbjGYV3Q (ORCPT ); Tue, 25 Jul 2023 17:29:16 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FA441FEC for ; Tue, 25 Jul 2023 14:29:14 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id 4fb4d7f45d1cf-51e619bcbf9so7955220a12.3 for ; Tue, 25 Jul 2023 14:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690320552; x=1690925352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sKxfLmNS2Yxgb7gGh2D3p1/z7+JKv9s8t2rHAKNWc8Q=; b=bhwGvz1WLMG5sohk0I+SkHhlJ4vWvDb6I83cbGQV0V+XOu0yb0c6ChDyDxBo1jwjuj rsNX6F1WXTnl24Oa0mOtcl0mIcChQ3bl+RqroyxF4k5RLxWwpL/H9mYtUI2MRVDUGJOu 89QlkW5NMPertqiZIZG2PPO5ESe+yD+A7+uyHeIaRpzvic6cj2PtVj+ArOriU2w/9bqo Dx3U5kFRezX/uFAScLjBStvi484BRRKR6cIdVScjLZwQYaSJaRCzplIvziWaLNymz7E2 1eOD8H+TRzm8I0NHNO4CB5rg4dQRWgHRMRgONyqVwIUVB95vjFFUuo/M42cq2aJrXLj2 MO+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320552; x=1690925352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sKxfLmNS2Yxgb7gGh2D3p1/z7+JKv9s8t2rHAKNWc8Q=; b=bbC3DFolxPSd5S4XwvTeRf6VM9Fte5dnRmr7C9DpW+861M3/eAgwQZASXHM8g/CqFl 5pt+JwoQh91SnCcju2WGBFIImmEQQLdlAHlqnVUwp50HPYUrQ1E5BAdRThZXXKMK3fff WBDg5v9ZP0aZnNhr8iypa9Rxd01y5MpjPAhGLmfAoVVmAetsSq29420YID6tSMBTXvuO Gf2dRfqjKzeQJq0EeuH72IPbLmszmPZQ9GKckcMdPXzFWKkD1LB0Ds+3hU9xDXw9BOnI uD1wc6yEjXPgb3BPv9s5DD6fQgjwFFXqgnK0IElyyG5VfyTN5H84RwnoroDClKl1qKTq s30Q== X-Gm-Message-State: ABy/qLZORFYnACPmLsH48dXZQ0eTZl+8EyEEyxBSObQgT9f4McWZjVAL FSNMrFOUiYSWL5Lc9f00Z2nAkdWc2Io= X-Google-Smtp-Source: APBJJlETMdX3tzwtZbawnDwcihX3tr09CwKKOnyIGFprax86Y41qz6nnKtwYF8lDZnfIqX4xcXtgBw== X-Received: by 2002:aa7:d055:0:b0:51a:2c81:72ee with SMTP id n21-20020aa7d055000000b0051a2c8172eemr102666edo.20.1690320552537; Tue, 25 Jul 2023 14:29:12 -0700 (PDT) Received: from localhost.localdomain (ip-94-112-167-15.bb.vodafone.cz. [94.112.167.15]) by smtp.gmail.com with ESMTPSA id f23-20020a05640214d700b005224ec27dd7sm1200778edx.66.2023.07.25.14.29.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 14:29:12 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Cc: Dongsheng Yang Subject: [PATCH v2 3/3] rbd: retrieve and check lock owner twice before blocklisting Date: Tue, 25 Jul 2023 23:28:46 +0200 Message-ID: <20230725212847.137672-4-idryomov@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230725212847.137672-1-idryomov@gmail.com> References: <20230725212847.137672-1-idryomov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org An attempt to acquire exclusive lock can race with the current lock owner closing the image: 1. lock is held by client123, rbd_lock() returns -EBUSY 2. get_lock_owner_info() returns client123 instance details 3. client123 closes the image, lock is released 4. find_watcher() returns 0 as there is no matching watcher anymore 5. client123 instance gets erroneously blocklisted Particularly impacted is mirror snapshot scheduler in snapshot-based mirroring since it happens to open and close images a lot (images are opened only for as long as it takes to take the next mirror snapshot, the same client instance is used for all images). To reduce the potential for erroneous blocklisting, retrieve the lock owner again after find_watcher() returns 0. If it's still there, make sure it matches the previously detected lock owner. Cc: stable@vger.kernel.org # ba6b7b6db4df: rbd: make get_lock_owner_info() return a single locker or NULL Cc: stable@vger.kernel.org # c476a060136a: rbd: harden get_lock_owner_info() a bit Cc: stable@vger.kernel.org Signed-off-by: Ilya Dryomov --- drivers/block/rbd.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 94629e826369..24afcc93ac01 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3849,6 +3849,15 @@ static void wake_lock_waiters(struct rbd_device *rbd_dev, int result) list_splice_tail_init(&rbd_dev->acquiring_list, &rbd_dev->running_list); } +static bool locker_equal(const struct ceph_locker *lhs, + const struct ceph_locker *rhs) +{ + return lhs->id.name.type == rhs->id.name.type && + lhs->id.name.num == rhs->id.name.num && + !strcmp(lhs->id.cookie, rhs->id.cookie) && + ceph_addr_equal_no_type(&lhs->info.addr, &rhs->info.addr); +} + static void free_locker(struct ceph_locker *locker) { if (locker) @@ -3969,11 +3978,11 @@ static int find_watcher(struct rbd_device *rbd_dev, static int rbd_try_lock(struct rbd_device *rbd_dev) { struct ceph_client *client = rbd_dev->rbd_client->client; - struct ceph_locker *locker; + struct ceph_locker *locker, *refreshed_locker; int ret; for (;;) { - locker = NULL; + locker = refreshed_locker = NULL; ret = rbd_lock(rbd_dev); if (ret != -EBUSY) @@ -3993,6 +4002,16 @@ static int rbd_try_lock(struct rbd_device *rbd_dev) if (ret) goto out; /* request lock or error */ + refreshed_locker = get_lock_owner_info(rbd_dev); + if (IS_ERR(refreshed_locker)) { + ret = PTR_ERR(refreshed_locker); + refreshed_locker = NULL; + goto out; + } + if (!refreshed_locker || + !locker_equal(locker, refreshed_locker)) + goto again; + rbd_warn(rbd_dev, "breaking header lock owned by %s%llu", ENTITY_NAME(locker->id.name)); @@ -4014,10 +4033,12 @@ static int rbd_try_lock(struct rbd_device *rbd_dev) } again: + free_locker(refreshed_locker); free_locker(locker); } out: + free_locker(refreshed_locker); free_locker(locker); return ret; }