From patchwork Tue Jan 22 07:34:23 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Srivatsa S. Bhat" X-Patchwork-Id: 2015901 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by patchwork1.kernel.org (Postfix) with ESMTP id A760A3FCDE for ; Tue, 22 Jan 2013 07:39:56 +0000 (UTC) Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.76 #1 (Red Hat Linux)) id 1TxYQ8-0001wm-3V; Tue, 22 Jan 2013 07:37:24 +0000 Received: from e23smtp07.au.ibm.com ([202.81.31.140]) by merlin.infradead.org with esmtps (Exim 4.76 #1 (Red Hat Linux)) id 1TxYPK-0001SO-Ht for linux-arm-kernel@lists.infradead.org; Tue, 22 Jan 2013 07:36:36 +0000 Received: from /spool/local by e23smtp07.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 22 Jan 2013 17:30:09 +1000 Received: from d23dlp03.au.ibm.com (202.81.31.214) by e23smtp07.au.ibm.com (202.81.31.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 22 Jan 2013 17:30:08 +1000 Received: from d23relay05.au.ibm.com (d23relay05.au.ibm.com [9.190.235.152]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id 1A5483578051 for ; Tue, 22 Jan 2013 18:36:27 +1100 (EST) Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay05.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r0M7OWTp4850012 for ; Tue, 22 Jan 2013 18:24:32 +1100 Received: from d23av04.au.ibm.com (loopback [127.0.0.1]) by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r0M7aPBj004708 for ; Tue, 22 Jan 2013 18:36:26 +1100 Received: from srivatsabhat.in.ibm.com (srivatsabhat.in.ibm.com [9.124.35.112]) by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r0M7a8kK004305; Tue, 22 Jan 2013 18:36:09 +1100 From: "Srivatsa S. Bhat" Subject: [PATCH v5 06/45] percpu_rwlock: Allow writers to be readers, and add lockdep annotations To: tglx@linutronix.de, peterz@infradead.org, tj@kernel.org, oleg@redhat.com, paulmck@linux.vnet.ibm.com, rusty@rustcorp.com.au, mingo@kernel.org, akpm@linux-foundation.org, namhyung@kernel.org Date: Tue, 22 Jan 2013 13:04:23 +0530 Message-ID: <20130122073416.13822.96504.stgit@srivatsabhat.in.ibm.com> In-Reply-To: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> User-Agent: StGIT/0.14.3 MIME-Version: 1.0 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13012207-0260-0000-0000-0000026FFD51 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20130122_023635_114395_05F71A06 X-CRM114-Status: GOOD ( 18.89 ) X-Spam-Score: -1.9 (-) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-1.9 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, medium trust [202.81.31.140 listed in list.dnswl.org] 3.0 KHOP_BIG_TO_CC Sent to 10+ recipients instaed of Bcc or a list -0.7 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: linux-arch@vger.kernel.org, linux@arm.linux.org.uk, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, fweisbec@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, rostedt@goodmis.org, xiaoguangrong@linux.vnet.ibm.com, rjw@sisk.pl, sbw@mit.edu, wangyun@linux.vnet.ibm.com, srivatsa.bhat@linux.vnet.ibm.com, netdev@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-arm-kernel-bounces@lists.infradead.org Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org CPU hotplug (which will be the first user of per-CPU rwlocks) has a special requirement with respect to locking: the writer, after acquiring the per-CPU rwlock for write, must be allowed to take the same lock for read, without deadlocking and without getting complaints from lockdep. In comparison, this is similar to what get_online_cpus()/put_online_cpus() does today: it allows a hotplug writer (who holds the cpu_hotplug.lock mutex) to invoke it without locking issues, because it silently returns if the caller is the hotplug writer itself. This can be easily achieved with per-CPU rwlocks as well (even without a "is this a writer?" check) by incrementing the per-CPU refcount of the writer immediately after taking the global rwlock for write, and then decrementing the per-CPU refcount before releasing the global rwlock. This ensures that any reader that comes along on that CPU while the writer is active (on that same CPU), notices the non-zero value of the nested counter and assumes that it is a nested read-side critical section and proceeds by just incrementing the refcount. Thus we prevent the reader from taking the global rwlock for read, which prevents the writer from deadlocking itself. Add that support and teach lockdep about this special locking scheme so that it knows that this sort of usage is valid. Also add the required lockdep annotations to enable it to detect common locking problems with per-CPU rwlocks. Cc: David Howells Signed-off-by: Srivatsa S. Bhat --- lib/percpu-rwlock.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/lib/percpu-rwlock.c b/lib/percpu-rwlock.c index a8d177a..054a50a 100644 --- a/lib/percpu-rwlock.c +++ b/lib/percpu-rwlock.c @@ -84,6 +84,10 @@ void percpu_read_lock_irqsafe(struct percpu_rwlock *pcpu_rwlock) if (likely(!writer_active(pcpu_rwlock))) { this_cpu_inc(*pcpu_rwlock->reader_refcnt); + + /* Pretend that we take global_rwlock for lockdep */ + rwlock_acquire_read(&pcpu_rwlock->global_rwlock.dep_map, + 0, 0, _RET_IP_); } else { /* Writer is active, so switch to global rwlock. */ @@ -108,6 +112,12 @@ void percpu_read_lock_irqsafe(struct percpu_rwlock *pcpu_rwlock) if (!writer_active(pcpu_rwlock)) { this_cpu_inc(*pcpu_rwlock->reader_refcnt); read_unlock(&pcpu_rwlock->global_rwlock); + + /* + * Pretend that we take global_rwlock for lockdep + */ + rwlock_acquire_read(&pcpu_rwlock->global_rwlock.dep_map, + 0, 0, _RET_IP_); } } } @@ -128,6 +138,14 @@ void percpu_read_unlock_irqsafe(struct percpu_rwlock *pcpu_rwlock) if (reader_nested_percpu(pcpu_rwlock)) { this_cpu_dec(*pcpu_rwlock->reader_refcnt); smp_wmb(); /* Paired with smp_rmb() in sync_reader() */ + + /* + * If this is the last decrement, then it is time to pretend + * to lockdep that we are releasing the read lock. + */ + if (!reader_nested_percpu(pcpu_rwlock)) + rwlock_release(&pcpu_rwlock->global_rwlock.dep_map, + 1, _RET_IP_); } else { read_unlock(&pcpu_rwlock->global_rwlock); } @@ -205,11 +223,14 @@ void percpu_write_lock_irqsave(struct percpu_rwlock *pcpu_rwlock, announce_writer_active(pcpu_rwlock); sync_all_readers(pcpu_rwlock); write_lock_irqsave(&pcpu_rwlock->global_rwlock, *flags); + this_cpu_inc(*pcpu_rwlock->reader_refcnt); } void percpu_write_unlock_irqrestore(struct percpu_rwlock *pcpu_rwlock, unsigned long *flags) { + this_cpu_dec(*pcpu_rwlock->reader_refcnt); + /* * Inform all readers that we are done, so that they can switch back * to their per-cpu refcounts. (We don't need to wait for them to