From patchwork Wed Aug 16 20:40:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 9904591 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8294960244 for ; Wed, 16 Aug 2017 20:40:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7530B287F2 for ; Wed, 16 Aug 2017 20:40:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6996C28984; Wed, 16 Aug 2017 20:40:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03A86287F2 for ; Wed, 16 Aug 2017 20:40:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752138AbdHPUky (ORCPT ); Wed, 16 Aug 2017 16:40:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58778 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752105AbdHPUkx (ORCPT ); Wed, 16 Aug 2017 16:40:53 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 48CC67E441; Wed, 16 Aug 2017 20:40:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 48CC67E441 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=longman@redhat.com Received: from llong.com (dhcp-17-6.bos.redhat.com [10.18.17.6]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5081362499; Wed, 16 Aug 2017 20:40:50 +0000 (UTC) From: Waiman Long To: Jens Axboe , Jeff Layton , "J. Bruce Fields" , Steven Rostedt , Ingo Molnar Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, Waiman Long Subject: [PATCH v2] blktrace: Fix potentail deadlock between delete & sysfs ops Date: Wed, 16 Aug 2017 16:40:40 -0400 Message-Id: <1502916040-18067-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 16 Aug 2017 20:40:53 +0000 (UTC) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The lockdep code had reported the following unsafe locking scenario: CPU0 CPU1 ---- ---- lock(s_active#228); lock(&bdev->bd_mutex/1); lock(s_active#228); lock(&bdev->bd_mutex); *** DEADLOCK *** The deadlock may happen when one task (CPU1) is trying to delete a partition in a block device and another task (CPU0) is accessing tracing sysfs file in that partition. To avoid that, accessing tracing sysfs file will now use a mutex trylock loop and the operation will fail if a delete operation is in progress. Signed-off-by: Waiman Long --- v2: - Use READ_ONCE() and smp_store_mb() to read and write bd_deleting. - Check for signal in the mutex_trylock loops. - Use usleep() instead of schedule() for RT tasks. block/ioctl.c | 3 +++ include/linux/fs.h | 1 + kernel/trace/blktrace.c | 39 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 41 insertions(+), 2 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index 0de02ee..b920329 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -86,12 +86,15 @@ static int blkpg_ioctl(struct block_device *bdev, struct blkpg_ioctl_arg __user return -EBUSY; } /* all seems OK */ + smp_store_mb(bdev->bd_deleting, 1); fsync_bdev(bdevp); invalidate_bdev(bdevp); mutex_lock_nested(&bdev->bd_mutex, 1); delete_partition(disk, partno); mutex_unlock(&bdev->bd_mutex); + smp_store_mb(bdev->bd_deleting, 0); + mutex_unlock(&bdevp->bd_mutex); bdput(bdevp); diff --git a/include/linux/fs.h b/include/linux/fs.h index 6e1fd5d..c2ba35e 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -427,6 +427,7 @@ struct block_device { #endif struct block_device * bd_contains; unsigned bd_block_size; + int bd_deleting; struct hd_struct * bd_part; /* number of times partitions within this device have been opened. */ unsigned bd_part_count; diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index bc364f8..b2dffa9 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -27,6 +27,8 @@ #include #include #include +#include +#include #include "../../block/blk.h" @@ -1605,6 +1607,23 @@ static struct request_queue *blk_trace_get_queue(struct block_device *bdev) return bdev_get_queue(bdev); } +/* + * Read/write to the tracing sysfs file requires taking references to the + * sysfs file and then acquiring the bd_mutex. Deleting a block device + * requires acquiring the bd_mutex and then waiting for all the sysfs + * references to be gone. This can lead to deadlock if both operations + * happen simultaneously. To avoid this problem, read/write to the + * the tracing sysfs files can now fail if the bd_mutex cannot be + * acquired while a deletion operation is in progress. + * + * A mutex trylock loop is used assuming that tracing sysfs operations + * aren't frequently enough to cause any contention problem. + * + * For RT tasks, a running high priority task will prevent any lower + * priority RT tasks from being run. So they do need to actually sleep + * when the trylock fails to allow lower priority tasks to make forward + * progress. + */ static ssize_t sysfs_blk_trace_attr_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -1622,7 +1641,15 @@ static ssize_t sysfs_blk_trace_attr_show(struct device *dev, if (q == NULL) goto out_bdput; - mutex_lock(&bdev->bd_mutex); + while (!mutex_trylock(&bdev->bd_mutex)) { + if (READ_ONCE(bdev->bd_deleting)) + goto out_bdput; + if (signal_pending(current)) { + ret = -EINTR; + goto out_bdput; + } + rt_task(current) ? usleep_range(10, 10) : schedule(); + } if (attr == &dev_attr_enable) { ret = sprintf(buf, "%u\n", !!q->blk_trace); @@ -1683,7 +1710,15 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev, if (q == NULL) goto out_bdput; - mutex_lock(&bdev->bd_mutex); + while (!mutex_trylock(&bdev->bd_mutex)) { + if (READ_ONCE(bdev->bd_deleting)) + goto out_bdput; + if (signal_pending(current)) { + ret = -EINTR; + goto out_bdput; + } + rt_task(current) ? usleep_range(10, 10) : schedule(); + } if (attr == &dev_attr_enable) { if (value)