From patchwork Thu Dec 28 13:19:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hans Holmberg X-Patchwork-Id: 10134649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D32AD60388 for ; Thu, 28 Dec 2017 13:21:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C151B2D6CC for ; Thu, 28 Dec 2017 13:21:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B39A92D6B7; Thu, 28 Dec 2017 13:21:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A4342D6B7 for ; Thu, 28 Dec 2017 13:21:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753468AbdL1NVl (ORCPT ); Thu, 28 Dec 2017 08:21:41 -0500 Received: from mail-lf0-f68.google.com ([209.85.215.68]:35350 "EHLO mail-lf0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753362AbdL1NVk (ORCPT ); Thu, 28 Dec 2017 08:21:40 -0500 Received: by mail-lf0-f68.google.com with SMTP id h5so1628440lfj.2 for ; Thu, 28 Dec 2017 05:21:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=owltronix-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bgWr9TkIq9PJtfEo4XLQ+eK2UlDI91T9Lnlc3vPswHo=; b=WvzhM5ELTIEo6PbvW1QjzOP/JSmn+vqb3PBWegdmgo4yNKLw8iLqGN4/yiosDrX+N0 ceTUISkwU2jmfB/7OOb1Pg7oqhXtwr1N+LPMFdID47PEjNeEYWEtvVtscnVzuLM57oeK TIMHnNp5v39bruwmMBw4dbvkVdyXVGX1a8IA5Dek/EDmer1SfID/kKnxOcaTMh3ky+5N cdr3jZMqzNC8AKyEmRpaXJGu7QNTG2Y+mOIkTSqAvIGCtD7eLntajTnkKPEYar4C5RlF ozIqsfZ0PBkQET2KkZdLnkzEwkalX1rhD8mnfsk3XrvO4CCaZf83C138GRaeWa9hOXOC UxQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bgWr9TkIq9PJtfEo4XLQ+eK2UlDI91T9Lnlc3vPswHo=; b=fcZnvkfODdEhbLRYVWgCXIzC5+czigmsYlLe5d6zVRu/iox94N0hloVQlOdnqatf96 JQp/2neTSJ3TR5JzpwH4YRnfosg3bas1S4nDD0wnHM1x05P5nQ7Dtx9sa5re75NDX8kT aiZ+0f0P0pP/3ec8cWQPa4AsXE4keNjnsPlwhjK34fVgnCgzmGsu3k+eFcKk7J0m770P OvD8+k78aK7hRCXF3iS9wwKrFEqpNSVuzEC5/xV6pYbvcChzTtSsFg0/ouDMSvhmzf/w OWEeGvdVmKf7BNN29rXnzzbybW92ERgUi3IuNAauj/fRza1ug0PvL06/g27wM4Ahj5xj Xw4Q== X-Gm-Message-State: AKGB3mIwyxA/6hMRUmWXz9n1AX9nj91pklzG+nQ4VimhffU9zDhR1Oas srx2PXRRIWx9lxgm14bBHSlMsRSj+KY= X-Google-Smtp-Source: ACJfBosvubgUuHTjIQtAQVS/zyPSo8i2TyAxkkkpCH7RtSXfB+AqOPpmEUjJqeC4J/oJfTv0l/z23g== X-Received: by 10.25.152.77 with SMTP id a74mr18981847lfe.104.1514467298494; Thu, 28 Dec 2017 05:21:38 -0800 (PST) Received: from titan.bredbandsbolaget.se (c-e1f0e055.03-91-6d6c6d4.cust.bredbandsbolaget.se. [85.224.240.225]) by smtp.gmail.com with ESMTPSA id d27sm7318866lfb.93.2017.12.28.05.21.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 28 Dec 2017 05:21:37 -0800 (PST) From: Hans Holmberg X-Google-Original-From: Hans Holmberg To: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Matias=20Bj=C3=B8rling?= Cc: Hans Holmberg , =?UTF-8?q?Javier=20Gonz=C3=A1lez?= Subject: [PATCH v2] lightnvm: pblk: clear flush point on completed writes Date: Thu, 28 Dec 2017 14:19:52 +0100 Message-Id: <1514467192-5694-1-git-send-email-hans.holmberg@cnexlabs.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <20171220172205.26464-15-m@bjorling.me> References: <20171220172205.26464-15-m@bjorling.me> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move completion of syncs and clearing of flush points to the write completion path - this ensures that the data has been comitted to the media before completing bios containing syncs. Signed-off-by: Hans Holmberg Signed-off-by: Javier González --- Changes since v1: * Fixed an off-by one issue when clearing flush points drivers/lightnvm/pblk-rb.c | 58 +++++++++++++++++++++---------------------- drivers/lightnvm/pblk-write.c | 17 ++++++++++++- 2 files changed, 44 insertions(+), 31 deletions(-) diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c index 2c61d74..fe5465d 100644 --- a/drivers/lightnvm/pblk-rb.c +++ b/drivers/lightnvm/pblk-rb.c @@ -353,17 +353,17 @@ static int pblk_rb_flush_point_set(struct pblk_rb *rb, struct bio *bio, unsigned int pos) { struct pblk_rb_entry *entry; - unsigned int subm, flush_point; + unsigned int sync, flush_point; - subm = READ_ONCE(rb->subm); + sync = READ_ONCE(rb->sync); + + if (pos == sync) + return 0; #ifdef CONFIG_NVM_DEBUG atomic_inc(&rb->inflight_flush_point); #endif - if (pos == subm) - return 0; - flush_point = (pos == 0) ? (rb->nr_entries - 1) : (pos - 1); entry = &rb->entries[flush_point]; @@ -604,22 +604,6 @@ unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd, return NVM_IO_ERR; } - if (flags & PBLK_FLUSH_ENTRY) { - unsigned int flush_point; - - flush_point = READ_ONCE(rb->flush_point); - if (flush_point == pos) { - /* Protect flush points */ - smp_store_release(&rb->flush_point, - EMPTY_ENTRY); - } - - flags &= ~PBLK_FLUSH_ENTRY; -#ifdef CONFIG_NVM_DEBUG - atomic_dec(&rb->inflight_flush_point); -#endif - } - flags &= ~PBLK_WRITTEN_DATA; flags |= PBLK_SUBMITTED_ENTRY; @@ -735,15 +719,24 @@ void pblk_rb_sync_end(struct pblk_rb *rb, unsigned long *flags) unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries) { - unsigned int sync; - unsigned int i; - + unsigned int sync, flush_point; lockdep_assert_held(&rb->s_lock); sync = READ_ONCE(rb->sync); + flush_point = READ_ONCE(rb->flush_point); - for (i = 0; i < nr_entries; i++) - sync = (sync + 1) & (rb->nr_entries - 1); + if (flush_point != EMPTY_ENTRY) { + unsigned int secs_to_flush; + + secs_to_flush = pblk_rb_ring_count(flush_point, sync, + rb->nr_entries); + if (secs_to_flush < nr_entries) { + /* Protect flush points */ + smp_store_release(&rb->flush_point, EMPTY_ENTRY); + } + } + + sync = (sync + nr_entries) & (rb->nr_entries - 1); /* Protect from counts */ smp_store_release(&rb->sync, sync); @@ -751,22 +744,27 @@ unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries) return sync; } +/* Calculate how many sectors to submit up to the current flush point. */ unsigned int pblk_rb_flush_point_count(struct pblk_rb *rb) { - unsigned int subm, flush_point; - unsigned int count; + unsigned int subm, sync, flush_point; + unsigned int submitted, to_flush; /* Protect flush points */ flush_point = smp_load_acquire(&rb->flush_point); if (flush_point == EMPTY_ENTRY) return 0; + /* Protect syncs */ + sync = smp_load_acquire(&rb->sync); + subm = READ_ONCE(rb->subm); + submitted = pblk_rb_ring_count(subm, sync, rb->nr_entries); /* The sync point itself counts as a sector to sync */ - count = pblk_rb_ring_count(flush_point, subm, rb->nr_entries) + 1; + to_flush = pblk_rb_ring_count(flush_point, sync, rb->nr_entries) + 1; - return count; + return (submitted < to_flush) ? (to_flush - submitted) : 0; } /* diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c index 018af87..aae86ed 100644 --- a/drivers/lightnvm/pblk-write.c +++ b/drivers/lightnvm/pblk-write.c @@ -21,13 +21,28 @@ static unsigned long pblk_end_w_bio(struct pblk *pblk, struct nvm_rq *rqd, struct pblk_c_ctx *c_ctx) { struct bio *original_bio; + struct pblk_rb *rwb = &pblk->rwb; unsigned long ret; int i; for (i = 0; i < c_ctx->nr_valid; i++) { struct pblk_w_ctx *w_ctx; + int pos = c_ctx->sentry + i; + int flags; + + w_ctx = pblk_rb_w_ctx(rwb, pos); + flags = READ_ONCE(w_ctx->flags); + + if (flags & PBLK_FLUSH_ENTRY) { + flags &= ~PBLK_FLUSH_ENTRY; + /* Release flags on context. Protect from writes */ + smp_store_release(&w_ctx->flags, flags); + +#ifdef CONFIG_NVM_DEBUG + atomic_dec(&rwb->inflight_flush_point); +#endif + } - w_ctx = pblk_rb_w_ctx(&pblk->rwb, c_ctx->sentry + i); while ((original_bio = bio_list_pop(&w_ctx->bios))) bio_endio(original_bio); }