From patchwork Thu Feb 7 19:07:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10802063 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B89361669 for ; Thu, 7 Feb 2019 19:46:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A70942DCA6 for ; Thu, 7 Feb 2019 19:46:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 97C4C2DCB5; Thu, 7 Feb 2019 19:46:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0E2AC2DCA6 for ; Thu, 7 Feb 2019 19:46:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=a5OoQROw6byaJrY6ACvNMcSWno4xxMsWk6yQdSXy41k=; b=XbMpBrxmCn2xbTYW0u8BD2fOZs A4tHPYLDdsgdkzyyhu2iGMQcGpA7rD/ukyrfh+J4k+VBQMZUGJ3TIX2XCSt/M96No9IZq8AS7ptAE T6kawVzcxvFHTQ2nAvFOCyqBuTTgaP4f4J10hd5z8kuQsCxMuRxe0c+jTz2GLTSnuzhxlHqxy1MGV ZJylwHPabq5oTgJTit+fW6X7E/vuMCigCYoRMC99EkVmh5KbMk/fxITIh3sQMiebbiLmgT5daBfTo y4fxyN81MPaWrhmhy1q4EksuigPzm/5YpN+ev3iDtqPw3+nXVKa46VdYBwBLi5M54oMDlVr3pKyvi QPAK3b4w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1grpdC-0003tq-MN; Thu, 07 Feb 2019 19:46:42 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1grpcU-00039e-Em for linux-arm-kernel@bombadil.infradead.org; Thu, 07 Feb 2019 19:45:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=S6BozX0ZflDFh62oo9WzNNY5SpL5t7zul0vmeRYpZwo=; b=YZff5F/Wyu+mCh34ZWAn6bJKc 11QtRvNUt2HVyggQR7NSlnhMCgu22Stfo+JJ+hh+wLaSnPEtSeSQTRhyOHaQOhgaMeRIxJVp389qw 0B2OYp73MBuUxFI3fpOAe5zrKdgXwTVV5kPpHA3AxiPGHJ9cvp7EIAwGYgiKdbB9oARF+FH70R/Ys xRhWUd8owPup9dAW/UFHYbWXCfeTf897M0up0CusZ7rHQ6B25Sdc4ZALEY/jDd6hbzDhUQrle4lmI +aM3d61fICB5y5qKV+aq6d0lDpSiJnuQofUF3QxZsoYG0EQqcDJY8W84j3PPu4BA+U0MviMQaOUfO IUr50sPBw==; Received: from mx1.redhat.com ([209.132.183.28]) by merlin.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1grp3c-0006jf-UY for linux-arm-kernel@lists.infradead.org; Thu, 07 Feb 2019 19:09:57 +0000 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id ED98F7AE81; Thu, 7 Feb 2019 19:09:55 +0000 (UTC) Received: from llong.com (dhcp-17-35.bos.redhat.com [10.18.17.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id BF6A560FDF; Thu, 7 Feb 2019 19:09:53 +0000 (UTC) From: Waiman Long To: Peter Zijlstra , Ingo Molnar , Will Deacon , Thomas Gleixner Subject: [PATCH-tip 16/22] locking/rwsem: Remove redundant computation of writer lock word Date: Thu, 7 Feb 2019 14:07:20 -0500 Message-Id: <1549566446-27967-17-git-send-email-longman@redhat.com> In-Reply-To: <1549566446-27967-1-git-send-email-longman@redhat.com> References: <1549566446-27967-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Thu, 07 Feb 2019 19:09:56 +0000 (UTC) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190207_140957_148267_8E34AD92 X-CRM114-Status: GOOD ( 19.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linux-xtensa@linux-xtensa.org, Davidlohr Bueso , linux-ia64@vger.kernel.org, Tim Chen , Arnd Bergmann , linux-sh@vger.kernel.org, linux-hexagon@vger.kernel.org, x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org, Linus Torvalds , Borislav Petkov , linux-alpha@vger.kernel.org, sparclinux@vger.kernel.org, Waiman Long , Andrew Morton , linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On 64-bit architectures, each rwsem writer will have its unique lock word for acquiring the lock. Right now, the writer code recomputes the lock word every time it tries to acquire the lock. This is a waste of time. The lock word is now cached and reused when it is needed. On 32-bit architectures, the extra constant argument to rwsem_try_write_lock() and rwsem_try_write_lock_unqueued() should be optimized out by the compiler. Signed-off-by: Waiman Long --- kernel/locking/rwsem-xadd.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 0869fbf..16dc7a1 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -216,8 +216,8 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, * race conditions between checking the rwsem wait list and setting the * sem->count accordingly. */ -static inline bool -rwsem_try_write_lock(long count, struct rw_semaphore *sem, bool first) +static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem, + const long wlock, bool first) { long new; @@ -227,7 +227,7 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, if (!first && RWSEM_COUNT_HANDOFF(count)) return false; - new = (count & ~RWSEM_FLAG_HANDOFF) + RWSEM_WRITER_LOCKED - + new = (count & ~RWSEM_FLAG_HANDOFF) + wlock - (list_is_singular(&sem->wait_list) ? RWSEM_FLAG_WAITERS : 0); if (atomic_long_cmpxchg_acquire(&sem->count, count, new) == count) { @@ -242,7 +242,8 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem, /* * Try to acquire write lock before the writer has been put on wait queue. */ -static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) +static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem, + const long wlock) { long old, count = atomic_long_read(&sem->count); @@ -251,7 +252,7 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) return false; old = atomic_long_cmpxchg_acquire(&sem->count, count, - count + RWSEM_WRITER_LOCKED); + count + wlock); if (old == count) { rwsem_set_owner(sem); lockevent_inc(rwsem_opt_wlock); @@ -338,7 +339,7 @@ static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem) return is_rwsem_owner_spinnable(rwsem_get_owner(sem)); } -static bool rwsem_optimistic_spin(struct rw_semaphore *sem) +static bool rwsem_optimistic_spin(struct rw_semaphore *sem, const long wlock) { bool taken = false; @@ -362,7 +363,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem) /* * Try to acquire the lock */ - if (rwsem_try_write_lock_unqueued(sem)) { + if (rwsem_try_write_lock_unqueued(sem, wlock)) { taken = true; break; } @@ -392,7 +393,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem) return taken; } #else -static bool rwsem_optimistic_spin(struct rw_semaphore *sem) +static bool rwsem_optimistic_spin(struct rw_semaphore *sem, const long wlock) { return false; } @@ -514,9 +515,10 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem) struct rwsem_waiter waiter; struct rw_semaphore *ret = sem; DEFINE_WAKE_Q(wake_q); + const long wlock = RWSEM_WRITER_LOCKED; /* do optimistic spinning and steal lock if possible */ - if (rwsem_optimistic_spin(sem)) + if (rwsem_optimistic_spin(sem, wlock)) return sem; /* @@ -569,7 +571,7 @@ static bool rwsem_optimistic_spin(struct rw_semaphore *sem) /* wait until we successfully acquire the lock */ set_current_state(state); while (true) { - if (rwsem_try_write_lock(count, sem, first)) + if (rwsem_try_write_lock(count, sem, wlock, first)) break; raw_spin_unlock_irq(&sem->wait_lock);