From patchwork Wed Dec 21 18:46:38 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Nicolai_H=C3=A4hnle?= X-Patchwork-Id: 9483637 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 67EA6601D3 for ; Wed, 21 Dec 2016 18:47:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C15B28419 for ; Wed, 21 Dec 2016 18:47:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50F9F28458; Wed, 21 Dec 2016 18:47:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E828E28419 for ; Wed, 21 Dec 2016 18:47:18 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9FA196F0D2; Wed, 21 Dec 2016 18:47:03 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7C3786E0F0 for ; Wed, 21 Dec 2016 18:47:00 +0000 (UTC) Received: by mail-wm0-x243.google.com with SMTP id u144so32073672wmu.0 for ; Wed, 21 Dec 2016 10:47:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ywCr6L8S3en/6Px/J8pSmxyyo6Y9gJog06kzCIoyYAI=; b=GOiT4bxL1r0Le8jDnUhO3q2n85SNwozvT6qLVaGKsaCODMpccymN+jjOumU/kD51Aa FFJNvPZzCqkmNYKqVWZi8M73QwTEEhtm57dvmLW7pqValztvMDFI6RM1+aEdmlJo85Ib TGDF0Vg/gdyRUnnVJJe25/3Lwi6DuSML7KhanHzRKn2SqmiYM0Zua30PbmzB9VHk8u8o JmawsDjhnQ0IkCsFzLeHZuA3R2wFDhHEJEo8sUVINHLD93Qt7Dj5OHbOSksEG7yORjdi LlDtCUCaYJfBLJTVqZSBjSSli3OnZ/cv5eZpvYYTLkWTOkVZhccFJFQ2sPNahnimKPgw Qj+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ywCr6L8S3en/6Px/J8pSmxyyo6Y9gJog06kzCIoyYAI=; b=X30/8kC0L8VefBgvFCk1k20hTxWZ7RLUB3jAERpoV240dgopa5gU4fZ8bkbnX1Gwen YsQr98cVZbHn1QoGoKehQG60Ormv3C7PhP+2cHlQx9yG3Q78ISV8KMyN/dBwmtipppVe W1Bt5U7qhrxual04F2YN5jsND0GfYlyCwylyaKyU6+oQ90NK3jcEdkDOXEaYwl9SskoQ gqL5D9xPxKB1a3SlfKkwUpn1j4/Z+ZkbMdydcop9d6k9u4IbK2DlJZHAcHdePfmCZF6E FCDDtoMB0W9fVBie4ujhLnXo2e0dlkj+z3qbBB3ENoQETyIbK7X8NEwtkJJArutuyWso Vbbw== X-Gm-Message-State: AIkVDXJt2oGEL35qGjvlVWWgeGjL2oMGnPBXEu4nbUGeur6LGi/AKrKzlFrwLf1+VUGxgw== X-Received: by 10.28.111.138 with SMTP id c10mr8035241wmi.135.1482346018713; Wed, 21 Dec 2016 10:46:58 -0800 (PST) Received: from cassiopeia.fritz.box ([2001:a61:110a:1701:2511:6687:1ba5:b45b]) by smtp.gmail.com with ESMTPSA id ke6sm31840526wjb.21.2016.12.21.10.46.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 21 Dec 2016 10:46:58 -0800 (PST) From: =?UTF-8?q?Nicolai=20H=C3=A4hnle?= To: linux-kernel@vger.kernel.org Subject: [PATCH v3 10/12] locking/ww_mutex: Yield to other waiters from optimistic spin Date: Wed, 21 Dec 2016 19:46:38 +0100 Message-Id: <1482346000-9927-11-git-send-email-nhaehnle@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1482346000-9927-1-git-send-email-nhaehnle@gmail.com> References: <1482346000-9927-1-git-send-email-nhaehnle@gmail.com> MIME-Version: 1.0 Cc: Maarten Lankhorst , =?UTF-8?q?Nicolai=20H=C3=A4hnle?= , Peter Zijlstra , dri-devel@lists.freedesktop.org, Ingo Molnar X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Nicolai Hähnle Lock stealing is less beneficial for w/w mutexes since we may just end up backing off if we stole from a thread with an earlier acquire stamp that already holds another w/w mutex that we also need. So don't spin optimistically unless we are sure that there is no other waiter that might cause us to back off. Median timings taken of a contention-heavy GPU workload: Before: real 0m52.946s user 0m7.272s sys 1m55.964s After: real 0m53.086s user 0m7.360s sys 1m46.204s This particular workload still spends 20%-25% of CPU in mutex_spin_on_owner according to perf, but my attempts to further reduce this spinning based on various heuristics all lead to an increase in measured wall time despite the decrease in sys time. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Maarten Lankhorst Cc: Daniel Vetter Cc: Chris Wilson Cc: dri-devel@lists.freedesktop.org Signed-off-by: Nicolai Hähnle --- kernel/locking/mutex.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 6f62695..0bafb37 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -374,7 +374,8 @@ ww_mutex_set_context_slowpath(struct ww_mutex *lock, */ static noinline bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, - bool use_ww_ctx, struct ww_acquire_ctx *ww_ctx) + bool use_ww_ctx, struct ww_acquire_ctx *ww_ctx, + struct mutex_waiter *waiter) { bool ret = true; @@ -397,7 +398,7 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, break; } - if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) { + if (use_ww_ctx && ww_ctx) { struct ww_mutex *ww; ww = container_of(lock, struct ww_mutex, base); @@ -413,7 +414,30 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, * Check this in every inner iteration because we may * be racing against another thread's ww_mutex_lock. */ - if (READ_ONCE(ww->ctx)) { + if (ww_ctx->acquired > 0 && READ_ONCE(ww->ctx)) { + ret = false; + break; + } + + /* + * If we aren't on the wait list yet, cancel the spin + * if there are waiters. We want to avoid stealing the + * lock from a waiter with an earlier stamp, since the + * other thread may already own a lock that we also + * need. + */ + if (!waiter && + (atomic_long_read(&lock->owner) & + MUTEX_FLAG_WAITERS)) { + ret = false; + break; + } + + /* + * Similarly, stop spinning if we are no longer the + * first waiter. + */ + if (waiter && !__mutex_waiter_is_first(lock, waiter)) { ret = false; break; } @@ -479,7 +503,8 @@ static inline int mutex_can_spin_on_owner(struct mutex *lock) */ static bool mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx, - const bool use_ww_ctx, const bool waiter) + const bool use_ww_ctx, + struct mutex_waiter *waiter) { struct task_struct *task = current; @@ -518,12 +543,12 @@ static bool mutex_optimistic_spin(struct mutex *lock, } if (!mutex_spin_on_owner(lock, owner, use_ww_ctx, - ww_ctx)) + ww_ctx, waiter)) goto fail_unlock; } /* Try to acquire the mutex if it is unlocked. */ - if (__mutex_trylock(lock, waiter)) + if (__mutex_trylock(lock, waiter != NULL)) break; /* @@ -565,7 +590,8 @@ static bool mutex_optimistic_spin(struct mutex *lock, #else static bool mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx, - const bool use_ww_ctx, const bool waiter) + const bool use_ww_ctx, + struct mutex_waiter *waiter) { return false; } @@ -737,7 +763,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip); if (__mutex_trylock(lock, false) || - mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx, false)) { + mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx, NULL)) { /* got the lock, yay! */ lock_acquired(&lock->dep_map, ip); if (use_ww_ctx && ww_ctx) @@ -843,7 +869,8 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, } if ((first && - mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx, true)) || + mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx, + &waiter)) || __mutex_trylock(lock, handoff)) break;