From patchwork Mon Mar 18 09:15:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dongsheng Yang X-Patchwork-Id: 10857023 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 10EF6139A for ; Mon, 18 Mar 2019 09:24:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC16428F49 for ; Mon, 18 Mar 2019 09:24:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E062228FD9; Mon, 18 Mar 2019 09:24:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C09228F49 for ; Mon, 18 Mar 2019 09:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726935AbfCRJYU (ORCPT ); Mon, 18 Mar 2019 05:24:20 -0400 Received: from m97134.mail.qiye.163.com ([220.181.97.134]:53189 "EHLO m97134.mail.qiye.163.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726795AbfCRJYT (ORCPT ); Mon, 18 Mar 2019 05:24:19 -0400 Received: from atest-guest.localdomain (unknown [218.94.118.90]) by smtp5 (Coremail) with SMTP id huCowABHr4m4YY9cO8zEAg--.331S13; Mon, 18 Mar 2019 17:15:42 +0800 (CST) From: Dongsheng Yang To: idryomov@gmail.com, jdillama@redhat.com, sage@redhat.com, elder@kernel.org Cc: ceph-devel@vger.kernel.org, Dongsheng Yang Subject: [PATCH v2 11/16] rbd: wait image request all complete in lock releasing Date: Mon, 18 Mar 2019 05:15:29 -0400 Message-Id: <1552900534-29026-12-git-send-email-dongsheng.yang@easystack.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1552900534-29026-1-git-send-email-dongsheng.yang@easystack.cn> References: <1552900534-29026-1-git-send-email-dongsheng.yang@easystack.cn> X-CM-TRANSID: huCowABHr4m4YY9cO8zEAg--.331S13 X-Coremail-Antispam: 1Uf129KBjvJXoWxZry5trykJF4Uur4kAFy5Arb_yoW5ZFyfp3 y3JasIkrWUWwn7Ww1fJayrZr15Wa18t347WryIkw17CFn3JrWvvr1IkFyUZFW7Ar93Ar4x GF45tFs5CF4jgrDanT9S1TB71UUUUUJqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0JbGT5JUUUUU= X-Originating-IP: [218.94.118.90] X-CM-SenderInfo: 5grqw2pkhqwhp1dqwq5hdv52pwdfyhdfq/1tbiHx57eluyGkr-BwAAsB Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP currently, we only sync all osdc request in lock releasing, but if we are going to support journaling, we need to wait all img_request complete, not only the low-level in osd_client. Signed-off-by: Dongsheng Yang --- drivers/block/rbd.c | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 4c5f36e..a583c2e 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -386,6 +386,9 @@ struct rbd_device { struct list_head node; + atomic_t inflight_ios; + struct completion inflight_wait; + /* sysfs related */ struct device dev; unsigned long open_count; /* protected by lock */ @@ -1654,6 +1657,7 @@ static struct rbd_img_request *rbd_img_request_create( spin_lock_init(&img_request->completion_lock); INIT_LIST_HEAD(&img_request->object_extents); kref_init(&img_request->kref); + atomic_inc(&rbd_dev->inflight_ios); dout("%s: rbd_dev %p %s -> img %p\n", __func__, rbd_dev, obj_op_name(op_type), img_request); @@ -1670,6 +1674,8 @@ static void rbd_img_request_destroy(struct kref *kref) dout("%s: img %p\n", __func__, img_request); + atomic_dec(&img_request->rbd_dev->inflight_ios); + complete_all(&img_request->rbd_dev->inflight_wait); for_each_obj_request_safe(img_request, obj_request, next_obj_request) rbd_img_obj_request_del(img_request, obj_request); rbd_assert(img_request->obj_request_count == 0); @@ -3075,26 +3081,39 @@ static void rbd_acquire_lock(struct work_struct *work) } } +static int rbd_inflight_wait(struct rbd_device *rbd_dev) +{ + int ret = 0; + + while (atomic_read(&rbd_dev->inflight_ios)) { + ret = wait_for_completion_interruptible(&rbd_dev->inflight_wait); + if (ret) + break; + } + + return ret; +} + /* * lock_rwsem must be held for write */ static bool rbd_release_lock(struct rbd_device *rbd_dev) { + int ret = 0; + dout("%s rbd_dev %p read lock_state %d\n", __func__, rbd_dev, rbd_dev->lock_state); if (rbd_dev->lock_state != RBD_LOCK_STATE_LOCKED) return false; rbd_dev->lock_state = RBD_LOCK_STATE_RELEASING; - downgrade_write(&rbd_dev->lock_rwsem); - /* - * Ensure that all in-flight IO is flushed. - * - * FIXME: ceph_osdc_sync() flushes the entire OSD client, which - * may be shared with other devices. - */ - ceph_osdc_sync(&rbd_dev->rbd_client->client->osdc); - up_read(&rbd_dev->lock_rwsem); + up_write(&rbd_dev->lock_rwsem); + + ret = rbd_inflight_wait(rbd_dev); + if (ret) { + down_write(&rbd_dev->lock_rwsem); + return false; + } down_write(&rbd_dev->lock_rwsem); dout("%s rbd_dev %p write lock_state %d\n", __func__, rbd_dev, @@ -4392,6 +4411,8 @@ static struct rbd_device *__rbd_dev_create(struct rbd_client *rbdc, INIT_LIST_HEAD(&rbd_dev->node); init_rwsem(&rbd_dev->header_rwsem); + atomic_set(&rbd_dev->inflight_ios, 0); + init_completion(&rbd_dev->inflight_wait); rbd_dev->header.data_pool_id = CEPH_NOPOOL; ceph_oid_init(&rbd_dev->header_oid); rbd_dev->header_oloc.pool = spec->pool_id;