From patchwork Tue Jun 30 22:04:56 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Hansen X-Patchwork-Id: 6698661 Return-Path: X-Original-To: patchwork-linux-fsdevel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 1D96A9F3A0 for ; Tue, 30 Jun 2015 22:05:22 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 35A1C20549 for ; Tue, 30 Jun 2015 22:05:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1A905203B4 for ; Tue, 30 Jun 2015 22:05:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753218AbbF3WFM (ORCPT ); Tue, 30 Jun 2015 18:05:12 -0400 Received: from mga03.intel.com ([134.134.136.65]:56749 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752416AbbF3WE5 (ORCPT ); Tue, 30 Jun 2015 18:04:57 -0400 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP; 30 Jun 2015 15:04:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.15,380,1432623600"; d="scan'208";a="753579035" Received: from viggo.jf.intel.com (HELO localhost.localdomain) ([10.54.39.19]) by fmsmga002.fm.intel.com with ESMTP; 30 Jun 2015 15:04:56 -0700 Subject: [PATCH 3/3] fs: replace memory barrier in __sb_end_write() with RCU To: dave@sr71.net Cc: dave.hansen@linux.intel.com, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, paulmck@linux.vnet.ibm.com, ak@linux.intel.com From: Dave Hansen Date: Tue, 30 Jun 2015 15:04:56 -0700 References: <20150630220455.7DE1D236@viggo.jf.intel.com> In-Reply-To: <20150630220455.7DE1D236@viggo.jf.intel.com> Message-Id: <20150630220456.285461B7@viggo.jf.intel.com> Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Dave Hansen If I sit in a loop and do write()s to small tmpfs files, __sb_end_write() is third-hottest kernel function due to its smp_mb(). __sb_end_write() uses the barrier to avoid races with freeze_super() and its calls to sb_wait_write(). But, now that freeze_super() is calling synchronize_rcu() before each sb_wait_write() call, we can use that to our advantage. The synchronize_rcu() ensures that all __sb_end_write() will see freeze_super()'s updates to s_writers.counter. That, in turn, guarantees that __sb_end_write() will try to wake up any subsequent call by freeze_super() to sb_wait_write(). This improves the number of writes/second I can do by 6.1% on top of the previous patch. The total improvement is 27.1% over a completely unpatched kernel. Signed-off-by: Dave Hansen Reviewed-by: Jan Kara Cc: Alexander Viro Cc: linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: Paul E. McKenney Cc: Andi Kleen --- b/fs/super.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff -puN fs/super.c~selectively-do-barriers-in-__sb_end_write fs/super.c --- a/fs/super.c~selectively-do-barriers-in-__sb_end_write 2015-06-30 15:03:57.565433061 -0700 +++ b/fs/super.c 2015-06-30 15:03:57.568433196 -0700 @@ -1146,14 +1146,23 @@ out: */ void __sb_end_write(struct super_block *sb, int level) { + rcu_read_lock(); percpu_counter_dec(&sb->s_writers.counter[level-1]); /* - * Make sure s_writers are updated before we wake up waiters in - * freeze_super(). + * We are racing here with freeze_super()'s calls to + * sb_wait_write(). We want to ensure that we call + * wake_up() whenever one of those calls _might_ be + * in sb_wait_write(). + * + * Since freeze_super() does a synchronize_rcu() before + * each of its sb_wait_write() calls, it can guarantee + * that it sees our update to s_writers.counter as well + * as that we see its update to s_writers.frozen. */ - smp_mb(); - if (waitqueue_active(&sb->s_writers.wait)) + if (unlikely(sb->s_writers.frozen >= level)) wake_up(&sb->s_writers.wait); + rcu_read_unlock(); + rwsem_release(&sb->s_writers.lock_map[level-1], 1, _RET_IP_); } EXPORT_SYMBOL(__sb_end_write);