From patchwork Mon Dec 3 12:50:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dongsheng Yang X-Patchwork-Id: 10709497 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E476613BF for ; Mon, 3 Dec 2018 12:56:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5B032B408 for ; Mon, 3 Dec 2018 12:56:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C95722B4AF; Mon, 3 Dec 2018 12:56:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69AE22B408 for ; Mon, 3 Dec 2018 12:56:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726557AbeLCM5F (ORCPT ); Mon, 3 Dec 2018 07:57:05 -0500 Received: from m50211.mail.qiye.163.com ([123.125.50.211]:22080 "EHLO m50211.mail.qiye.163.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726553AbeLCM5F (ORCPT ); Mon, 3 Dec 2018 07:57:05 -0500 Received: from atest-guest.localdomain (unknown [218.94.118.90]) by smtp3 (Coremail) with SMTP id PdOowEDpa0CfJgVcOwAAAA--.14S10; Mon, 03 Dec 2018 20:50:42 +0800 (CST) From: Dongsheng Yang To: idryomov@gmail.com, sage@redhat.com, elder@kernel.org, jdillama@redhat.com Cc: dongsheng.yang@easystack.cn, ceph-devel@vger.kernel.org Subject: [PATCH 08/11] rbd: wait image request all complete in lock releasing Date: Mon, 3 Dec 2018 07:50:32 -0500 Message-Id: <1543841435-13652-9-git-send-email-dongsheng.yang@easystack.cn> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1543841435-13652-1-git-send-email-dongsheng.yang@easystack.cn> References: <1543841435-13652-1-git-send-email-dongsheng.yang@easystack.cn> X-CM-TRANSID: PdOowEDpa0CfJgVcOwAAAA--.14S10 X-Coremail-Antispam: 1Uf129KBjvJXoWxZry5trykJF4Uur4kAFy5Arb_yoW5ZFyfp3 y3JasIkrWUWwn7Ww1fJayrZr1Yga18t347WryIkw1xCFn3JrWvvF1IkFyUZFW7Ar93Ar4x GF45tFs5CF42grDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0JbZKZAUUUUU= X-Originating-IP: [218.94.118.90] X-CM-SenderInfo: 5grqw2pkhqwhp1dqwq5hdv52pwdfyhdfq/1tbiiwISeltVe-R37QAAs6 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP currently, we only sync all osdc request in lock releasing, but if we are going to support journaling, we need to wait all img_request complete, not only the low-level in osd_client. Signed-off-by: Dongsheng Yang --- drivers/block/rbd.c | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 8e5140b..44d1285 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -386,6 +386,9 @@ struct rbd_device { struct list_head node; + atomic_t inflight_ios; + struct completion inflight_wait; + /* sysfs related */ struct device dev; unsigned long open_count; /* protected by lock */ @@ -1654,6 +1657,7 @@ static struct rbd_img_request *rbd_img_request_create( spin_lock_init(&img_request->completion_lock); INIT_LIST_HEAD(&img_request->object_extents); kref_init(&img_request->kref); + atomic_inc(&rbd_dev->inflight_ios); dout("%s: rbd_dev %p %s -> img %p\n", __func__, rbd_dev, obj_op_name(op_type), img_request); @@ -1670,6 +1674,8 @@ static void rbd_img_request_destroy(struct kref *kref) dout("%s: img %p\n", __func__, img_request); + atomic_dec(&img_request->rbd_dev->inflight_ios); + complete_all(&img_request->rbd_dev->inflight_wait); for_each_obj_request_safe(img_request, obj_request, next_obj_request) rbd_img_obj_request_del(img_request, obj_request); rbd_assert(img_request->obj_request_count == 0); @@ -3070,26 +3076,39 @@ static void rbd_acquire_lock(struct work_struct *work) } } +static int rbd_inflight_wait(struct rbd_device *rbd_dev) +{ + int ret = 0; + + while (atomic_read(&rbd_dev->inflight_ios)) { + ret = wait_for_completion_interruptible(&rbd_dev->inflight_wait); + if (ret) + break; + } + + return ret; +} + /* * lock_rwsem must be held for write */ static bool rbd_release_lock(struct rbd_device *rbd_dev) { + int ret = 0; + dout("%s rbd_dev %p read lock_state %d\n", __func__, rbd_dev, rbd_dev->lock_state); if (rbd_dev->lock_state != RBD_LOCK_STATE_LOCKED) return false; rbd_dev->lock_state = RBD_LOCK_STATE_RELEASING; - downgrade_write(&rbd_dev->lock_rwsem); - /* - * Ensure that all in-flight IO is flushed. - * - * FIXME: ceph_osdc_sync() flushes the entire OSD client, which - * may be shared with other devices. - */ - ceph_osdc_sync(&rbd_dev->rbd_client->client->osdc); - up_read(&rbd_dev->lock_rwsem); + up_write(&rbd_dev->lock_rwsem); + + ret = rbd_inflight_wait(rbd_dev); + if (ret) { + down_write(&rbd_dev->lock_rwsem); + return false; + } down_write(&rbd_dev->lock_rwsem); dout("%s rbd_dev %p write lock_state %d\n", __func__, rbd_dev, @@ -4387,6 +4406,8 @@ static struct rbd_device *__rbd_dev_create(struct rbd_client *rbdc, INIT_LIST_HEAD(&rbd_dev->node); init_rwsem(&rbd_dev->header_rwsem); + atomic_set(&rbd_dev->inflight_ios, 0); + init_completion(&rbd_dev->inflight_wait); rbd_dev->header.data_pool_id = CEPH_NOPOOL; ceph_oid_init(&rbd_dev->header_oid); rbd_dev->header_oloc.pool = spec->pool_id;