From patchwork Wed Mar 19 01:45:57 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaohua Li X-Patchwork-Id: 3850101 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Original-To: patchwork-dm-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 547E29F334 for ; Wed, 19 Mar 2014 01:52:07 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 412412037F for ; Wed, 19 Mar 2014 01:52:06 +0000 (UTC) Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) by mail.kernel.org (Postfix) with ESMTP id 0879A2037D for ; Wed, 19 Mar 2014 01:52:04 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s2J1kLMK015293; Tue, 18 Mar 2014 21:46:22 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s2J1kKXo010047 for ; Tue, 18 Mar 2014 21:46:20 -0400 Received: from mx1.redhat.com (ext-mx11.extmail.prod.ext.phx2.redhat.com [10.5.110.16]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s2J1kJv0005806; Tue, 18 Mar 2014 21:46:20 -0400 Received: from mail-pb0-f52.google.com (mail-pb0-f52.google.com [209.85.160.52]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s2J1kId0011667; Tue, 18 Mar 2014 21:46:18 -0400 Received: by mail-pb0-f52.google.com with SMTP id rr13so8186439pbb.11 for ; Tue, 18 Mar 2014 18:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=kZQFBO6amIINnCN5GeANx6p3E520XS1yK7JBsDBoN4g=; b=MewA61CCKxE/ASroucrYfnauf5M76KismC84osVl+vWo3R4iUGxgg9v6JSc6LmBfYd nIi17zHukUwQ+EWCwqGrhE0ENugGdm45YB3Nf6mYA+/SVlSPX4IHD1PO4nHdk3uPqO1l 0a6kqVhTNPFQg9ZpwFWYQxrLU7d1rURJdBlcuHQznqjaNbqxR70hpY6wy3xY8dIi/Vgt vnXwcDaJ7bzgIO0aWKsJFb6aD2WCIm2xdnUL2OTP4qdPNXGWH6p3geOqK2PHFtSGdvBP Qc7pJxWzQ6tq1FLcLyq/5Fu5syij8rASH0K0j/k5Zqpf30sZge8S+cCGNirQyyQS59Ts QIBw== X-Received: by 10.68.162.1 with SMTP id xw1mr19302653pbb.128.1395193577909; Tue, 18 Mar 2014 18:46:17 -0700 (PDT) Received: from shli-virt ([58.34.35.1]) by mx.google.com with ESMTPSA id vd8sm2163175pac.12.2014.03.18.18.46.11 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 18 Mar 2014 18:46:16 -0700 (PDT) Date: Wed, 19 Mar 2014 09:45:57 +0800 From: Shaohua Li To: Mike Snitzer Message-ID: <20140319014557.GA1631@kernel.org> References: <20140218101304.GA24889@kernel.org> <20140307075733.GB21790@kernel.org> <20140310135256.GA28665@redhat.com> <20140314094008.GA2386@kernel.org> <20140314224444.GA18027@redhat.com> <20140317095653.GA1014@kernel.org> <20140317200039.GC3269@redhat.com> <20140318074145.GA14397@kernel.org> <20140318212842.GA10850@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20140318212842.GA10850@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-RedHat-Spam-Score: -3 (BAYES_00, DCC_REPUT_00_12, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, SPF_PASS) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Scanned-By: MIMEDefang 2.68 on 10.5.110.16 X-loop: dm-devel@redhat.com Cc: axboe@kernel.dk, dm-devel@redhat.com, linux-kernel@vger.kernel.org, agk@redhat.com Subject: Re: [dm-devel] [patch v3]DM: dm-insitu-comp: a compressed DM target for SSD X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID,T_RP_MATCHES_RCVD,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Tue, Mar 18, 2014 at 05:28:43PM -0400, Mike Snitzer wrote: > On Tue, Mar 18 2014 at 3:41am -0400, > Shaohua Li wrote: > > > On Mon, Mar 17, 2014 at 04:00:40PM -0400, Mike Snitzer wrote: > > > > > > I folded your changes in, and then committed a patch ontop that cleans > > > some code up. But added 2 FIXMEs that still speak to pretty fundamental > > > problems with the architecture of the dm-insitu-comp target, see: > > > https://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/commit/?h=for-3.15-insitu-comp&id=8565ab6b04837591d03c94851c2f9f9162ce12f4 > > > > > > Unfortunately the single insitu_comp_wq workqueue that all insitu-comp > > > targets are to share isn't a workable solution. Each target needs to > > > have resource isolation from other targets (imagine insitu-comp used for > > > multiple SSDs). This is important for suspend too because you'll need > > > to flush/stop the workqueue. > > > > Is this just because of suspend? I didn't see fundamental reason why the > > workqueue can't be shared even for several targets. > > I'm not seeing how you are guaranteeing that all queued work is > completed during suspend. insitu_comp_queue_req() just calls > queue_work_on(). > > BTW, queue_work_on()'s comment above its implementation says: > "We queue the work to a specific CPU, the caller must ensure it can't go > away." -- you're not able to insure a cpu isn't hotplugged so... but I > also see you've used it in your raid5 perf improvement changes so you > obviously have experience with using this interface. Good point, I did miss this. the raid5 case hold a lock, while this case doesn't. A fix is attached below. > > > You introduced a state machine for tracking suspending, suspended, > > > resumed. This really isn't necessary. During suspend you need to > > > flush_workqueue(). On resume you shouldn't need to do anything special. > > > > > > As I noted in the commit, the thin and cache targets can serve as > > > references for how you can manage the workqueue across suspend/resume > > > and the lifetime of these workqueues relative to .ctr and .dtr. > > > > As far as I checking the code, .postsuspend is called after all requests are > > finished. This already guarantees no pending requests running in insitu-comp > > workqueue. > > I could easily be missing something obvious, but I don't see where that > guarantee is implemented. Alright, so this is the divergence. dm_suspend() calls dm_wait_for_completion() and then dm_table_postsuspend_targets(). As far as I understand, dm_wait_for_completion() will wait all pending requests to finish. The comments declaim this too. Am I missing anything? Basically the two kinds of IO. IO requests from upper layer, which dm_wait_for_completion() will guarantee they are finished. Metadata IO requests, which .postsuspend should make sure they are finished. > > Doing a workqueue flush isn't required. The writeback thread is > > running in background and waiting for requests completion can't guarantee the > > thread isn't running, so we must make sure it is safely parked. > > Sure, but you don't need a state machine to do that. The DM core takes > care of calling these hooks, so you just need to stop the writeback > thread during suspend and (re)start/kick it on resume (preresume). Yep, I need wait the writeback thread finish all pending metadata IO, the state machine works well here. --- drivers/md/dm-insitu-comp.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux/drivers/md/dm-insitu-comp.c =================================================================== --- linux.orig/drivers/md/dm-insitu-comp.c 2014-03-17 17:40:01.106660303 +0800 +++ linux/drivers/md/dm-insitu-comp.c 2014-03-19 08:49:12.314627050 +0800 @@ -704,14 +704,19 @@ static void insitu_comp_queue_req(struct struct insitu_comp_req *req) { unsigned long flags; - struct insitu_comp_io_worker *worker = - &insitu_comp_io_workers[req->cpu]; + struct insitu_comp_io_worker *worker; + + preempt_disable(); + if (!cpu_online(req->cpu)) + req->cpu = cpumask_any(cpu_online_mask); + worker = &insitu_comp_io_workers[req->cpu]; spin_lock_irqsave(&worker->lock, flags); list_add_tail(&req->sibling, &worker->pending); spin_unlock_irqrestore(&worker->lock, flags); queue_work_on(req->cpu, insitu_comp_wq, &worker->work); + preempt_enable(); } static void insitu_comp_queue_req_list(struct insitu_comp_info *info,