From patchwork Mon Jul 3 09:47:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93750C001B3 for ; Mon, 3 Jul 2023 09:49:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230444AbjGCJtr (ORCPT ); Mon, 3 Jul 2023 05:49:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbjGCJtq (ORCPT ); Mon, 3 Jul 2023 05:49:46 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 649B518D; Mon, 3 Jul 2023 02:49:40 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-e5-64a299b173bd From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 01/25] llist: Move llist_{head,node} definition to types.h Date: Mon, 3 Jul 2023 18:47:28 +0900 Message-Id: <20230703094752.79269-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiGec/He04rnSediWdinKvBLZqhLLo9Mc6YYeLrjzkN0R+QqJ09 kUZatCgCkwRWREVAIYEKIpSKtWlR8KCbAyGIEUFiqdowZkoHCJsoX0OLdiBKNf65cyX381y/ bp5WN7CLeL3xkGQyahM1WMkoR8OrvpZLbbrVLycUUJi3GgIvTzBQXluDwXPFhaDmWhYFw3c2 w59TIwim73fRYCn2IKjq76XhWpsfQZPjVwyPBj8Bb2AcQ0fxKQzmC7UYHjyfocBXUkSBS/4R Os/YKGgJ/suAZRjDOYuZmounFATtTg7smZEw4CjjYKY/Gjr83Sw0PV4JpRU+DDebOhhouzFA waOGcgz+mrcsdLa1M+ApzGfh8pgNw/MpOw32wDgHD1usFNRlz4lyXsyycDe/hYKc6qsUeP9q RNB8oo8CuaYbw+3ACAX1cjEN/1+6g2CgYJSDY3lBDs5lFSA4dayEga43d1nI9q2F6dfleOM6 cntknCbZ9UdI05SVIfdsIvmjrJcj2c2POWKVD5N6xwpy4eYwRaomAyyRnScxkSeLOJI76qXI mNvNkfaz0wwZ9FqobRFxyvU6KVGfIplWbdijTMh7+IQ9cF6ZOnQFMlEzn4sUvCisEZ91zTIf 2f7Ez4UYC1+KPT1BOsQLhKViff4/bC5S8rRwfJ7omLiPQ8Wnwi6x2uljQ8wIkeJIZ857Vglr xcpgO/4g/Vx01bW8FymEb8Wh1wUoxOq5G1+pH4ekomBWiI5XFejDw2fiLUcPcwaprCjMidR6 Y4pBq09cE5WQZtSnRu1NMshoblH2jJn4G2jSE9uKBB5pwlU9v1Tp1Kw2JTnN0IpEntYsUJn7 K3VqlU6bli6ZknabDidKya0ogmc0C1XfTB3RqYV92kPSfkk6IJk+thSvWJSJzFuXfBHWvefV 06wle13blymtqVtjazW5TsvgaXtDmTu6z9U7+3PrdW9dxOjBq0xh386hhTsevIk1zM/4ziDE xE9mbbx+sYCtbkzS/u3+KRD3n21T5PrFF6c3J7nTPVu+Ot4Y9cNveTJZrjoalhG+rNu9IYb7 /ft049Cm0t3Pdsbsz2/QMMkJ2ugVtClZ+w4jw7xDTQMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTYRzGe99zznuOxxaHJXSooFhEYGRWGv8wJOhDx6KIIroQ6WqHNnIW m5p2AWtm3slKTV3lJaaopW0F3RRRvCxpmYppraWWlXkLdebSLq7oy8MPnoffp4ejlGZmMaeL ipYNUepIFeFpfleIac39vGJNYMnnAMhKDwT3ZDIN5qpKAm33KhBUPriAYbBxG7yeGkYw8+Il BbnZbQiK+t5R8KDJhaCm7CKBjo8LoNM9RsCenUbAVFJF4NXQLAZnzlUMFdad0HqlGEOd5zMN uYMECnJNeC6+YPBYylmwJKyE/rJ8Fmb71oHd1cVAw007AzVvVkPeLSeBZzV2Gpoe9WPoeGIm 4Kr8zUBrUwsNbVkZDNwdLSYwNGWhwOIeY6G9rhBDdeKcLWniFwPNGXUYku7cx9DZ8xRBbXIv BmtlF4EG9zAGmzWbgh+ljQj6M0dYuJTuYaHgQiaCtEs5NLz82cxAojMYZqbNZEuI1DA8RkmJ ttNSzVQhLT0vFqXH+e9YKbH2DSsVWmMkW5m/VPJsEEtF425GspanEMk6fpWVUkc6sTTqcLBS y40ZWvrYmYt3Lz3Eb9bIkbpY2bA2NILXprd/YE7d5OMG7kECquVSkQ8nCkGi5YOL9TIRVond 3R7Ky37CctGW8YlJRTxHCZd9xbJvL4i3WCgcEe+UOxkv08JKcbg16S8rhGDxtqeF/JMuEyuq 6/6KfISN4sB0JvKycm7jzHORK4gvRPPKkZ8uKlav1kUGBxhPaOOjdHEBx07qrWjuM5bzs1mP 0GTHtnokcEg1X9F9tkijZNSxxnh9PRI5SuWnMPXd1igVGnX8GdlwMtwQEykb69ESjlYtUmzf L0cohePqaPmELJ+SDf9bzPksTkB90Qv4tyNhSdrmLJIibz207HBBqWLTufGlVct7Nuzssc+M Ps8fWu/4NbkiLZRvfMj76bcc2JEW/mUo8NNEjr9HuB42ookBfFQbZH6/uSks4sd3R0pQmH7S kWEP+v1tT7rTpj1t0p7dXr3vScDANd/A61MTmw6GJn/tXejJV+81hqhoo1a9zp8yGNV/AArV ij0vAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org llist_head and llist_node can be used by very primitives. For example, Dept for tracking dependency uses llist things in its header. To avoid header dependency, move those to types.h. Signed-off-by: Byungchul Park --- include/linux/llist.h | 8 -------- include/linux/types.h | 8 ++++++++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/llist.h b/include/linux/llist.h index 85bda2d02d65..99cc3c30f79c 100644 --- a/include/linux/llist.h +++ b/include/linux/llist.h @@ -53,14 +53,6 @@ #include #include -struct llist_head { - struct llist_node *first; -}; - -struct llist_node { - struct llist_node *next; -}; - #define LLIST_HEAD_INIT(name) { NULL } #define LLIST_HEAD(name) struct llist_head name = LLIST_HEAD_INIT(name) diff --git a/include/linux/types.h b/include/linux/types.h index 688fb943556a..0ddb0d722b3d 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -193,6 +193,14 @@ struct hlist_node { struct hlist_node *next, **pprev; }; +struct llist_head { + struct llist_node *first; +}; + +struct llist_node { + struct llist_node *next; +}; + struct ustat { __kernel_daddr_t f_tfree; #ifdef CONFIG_ARCH_32BIT_USTAT_F_TINODE From patchwork Mon Jul 3 09:47:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B45A4EB64DC for ; Mon, 3 Jul 2023 09:50:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231461AbjGCJt7 (ORCPT ); Mon, 3 Jul 2023 05:49:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231266AbjGCJtv (ORCPT ); Mon, 3 Jul 2023 05:49:51 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 3CC7CDD; Mon, 3 Jul 2023 02:49:40 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-f5-64a299b224b4 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 02/25] dept: Implement Dept(Dependency Tracker) Date: Mon, 3 Jul 2023 18:47:29 +0900 Message-Id: <20230703094752.79269-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSWUwTURSGvXdm7pRKcaxExiVBmxAVI6ABcmLcX5wHTYwG1xht7EQa2SyK FjRBqaggRFGsQNWCWBuoisUHF0pqDQUkCkpFQEApKBJZFC1awQUwvJx8+f9zvqcjoeQVzGyJ Ou6QqIlTxiiIlJb2+xYuseYVqcIs3kC4cC4MPN/P0GC4ayHQcKcUgeX+CQy9VevhzXAfgpHn 9RTocxsQFHa2U3Df2YHAZj5JoLHbD1yeQQK1uZkE0m7cJfDy8yiGtss5GEqtG6HufBEGu7eH Bn0vgQJ9Gh4bnzB4TSUsmFKDwG3OZ2G0cynUdjQxYGtdDHnX2ghU2GppcD5wY2h8ZCDQYfnL QJ2zhoaGC1kM3B4oIvB52ESByTPIwiu7EUOZbkyU/u0PA9VZdgzpxfcwuFoeI6g88x6D1dJE 4KmnD0O5NZeCX7eqELiz+1k4dc7LQsGJbASZpy7TUP+7mgFdWwSM/DSQNcuFp32DlKArPyLY ho208KyIFx7mt7OCrrKVFYzWw0K5OVi4UdGLhcIhDyNYS84SwTqUwwoZ/S4sDLx4wQo1V0Zo odulx5vm7JSuUIkx6iRRE7pqrzTantGCEp79ZY6+bV+Zii666AzkI+G5cL61OJ2a5EZD90RO uAV8c7N3Ivfn5vHlWR+ZDCSVUNzpqbz5y3MyXszgtvC/DD+Ycaa5ID7d+WGCZVwE/1avY/9L A/nSMvuEyIeL5D/8zEbjLB/bacvrIONSnsv04Q1f7Pj/wSz+ibmZPo9kRjSlBMnVcUmxSnVM eEi0Nk59NGRffKwVjf2U6fjorgdoqGGLA3ESpPCVNacUquSMMilRG+tAvIRS+MvSOq+r5DKV UpssauL3aA7HiIkONEdCKwJky4aPqOTcfuUh8YAoJoiayRZLfGanotgZqVs9H22B60yNUTcT Bk1S2ZMfkdVlEfPZPD+LuymNTFl0KZz1DbiqpMIccwP3d+11ztKt3r5vZo22a9vuO6Hukfy1 yV+NOd8GFla+27B70TQttWPhsbLXJt/psvDfa1s7azYerNVpDz7uTdFMD47S9VStvnfN0b64 pX5zS0F9SK6CToxWLg2mNInKf+HhpnNPAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTcRyG+5/rXC1OU/BgQbWIyMgyMn6lXSjKk5EFiVJfarRDjqatLS2L QHOZzQtq2JwumxZzeCnbIrqoDM3lksraKjWVtHWRvJQ1bW5lavTl5YH35fn0CnBxORkikCef 4lXJUoWEEhLC2MjM1Xf0lbK1XQWRUJi7Fjw/swkw3K6loONWDYLauxkYDLZGw9vxIQS+Zy9w 0BV3IKjo78Xhrr0PQaP5AgVO93xweUYpcBTnUJB54zYFL7/6Mei5WoRBjWUvtBdUYmDzfiZA N0hBmS4Tm44vGHhN1TSY0pfDgLmUBn9/ODj63pDQcs1BQmP3KtCX91DQ0OggwH5/AAPnQwMF fbVTJLTb2wjoKMwjoW6kkoKv4yYcTJ5RGl7ZjBjUa6ZtWT/+kPAkz4ZB1s07GLi6HiFoyn6P gaX2DQUtniEMrJZiHCarWhEM5A/TcDHXS0NZRj6CnItXCXjx+wkJmp4I8P0yUNsiuZahUZzT WE9zjeNGgntayXIPSntpTtPUTXNGSwpnNYdyNxoGMa5izENylurLFGcZK6I57bAL40aeP6e5 thIfwbldOmz/okPCKBmvkKfyqjVbjggTbdoupHw6RZ5517s5HV1xEVoUIGCZ9azT4J5lilnB dnZ68RkOYpaw1rxPpBYJBThzaS5r/vaMmikCmQPspGGCnGGCWc5m2T/OsoiJYN/pNPQ/6WK2 pt42KwpgNrAff+WjGRZPb3r0fVQBEhrRnGoUJE9OTZLKFRFh6uOJacnyM2FHTyRZ0PRrTOf9 hffRT2d0M2IESDJP1HmuQiYmpanqtKRmxApwSZAos/+6TCySSdPO8qoTh1UpCl7djBYKCEmw KCaBPyJmjklP8cd5Xsmr/reYICAkHW31rVLujC/1fOq1F2jcD1uFRetWbp8/6V5gVPiC7crX tpiEuth7SdlVB93Z9Wu+aOIOhHZbdj/eODJhTilbHbej5LsyJHzpnvbTzpxNWf6hsDr9B/+1 sL3eP/u+b5iIjh8LzLi18GRguW9R1IpdO62vJ40rJ5YRhcHrKtrq9HkHHwRJCHWiNDwUV6ml fwExfndoMQMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org CURRENT STATUS -------------- Lockdep tracks acquisition order of locks in order to detect deadlock, and IRQ and IRQ enable/disable state as well to take accident acquisitions into account. Lockdep should be turned off once it detects and reports a deadlock since the data structure and algorithm are not reusable after detection because of the complex design. PROBLEM ------- *Waits* and their *events* that never reach eventually cause deadlock. However, Lockdep is only interested in lock acquisition order, forcing to emulate lock acqusition even for just waits and events that have nothing to do with real lock. Even worse, no one likes Lockdep's false positive detection because that prevents further one that might be more valuable. That's why all the kernel developers are sensitive to Lockdep's false positive. Besides those, by tracking acquisition order, it cannot correctly deal with read lock and cross-event e.g. wait_for_completion()/complete() for deadlock detection. Lockdep is no longer a good tool for that purpose. SOLUTION -------- Again, *waits* and their *events* that never reach eventually cause deadlock. The new solution, Dept(DEPendency Tracker), focuses on waits and events themselves. Dept tracks waits and events and report it if any event would be never reachable. Dept does: . Works with read lock in the right way. . Works with any wait and event e.i. cross-event. . Continue to work even after reporting multiple times. . Provides simple and intuitive APIs. . Does exactly what dependency checker should do. Q & A ----- Q. Is this the first try ever to address the problem? A. No. Cross-release feature (b09be676e0ff2 locking/lockdep: Implement the 'crossrelease' feature) addressed it 2 years ago that was a Lockdep extension and merged but reverted shortly because: Cross-release started to report valuable hidden problems but started to give report false positive reports as well. For sure, no one likes Lockdep's false positive reports since it makes Lockdep stop, preventing reporting further real problems. Q. Why not Dept was developed as an extension of Lockdep? A. Lockdep definitely includes all the efforts great developers have made for a long time so as to be quite stable enough. But I had to design and implement newly because of the following: 1) Lockdep was designed to track lock acquisition order. The APIs and implementation do not fit on wait-event model. 2) Lockdep is turned off on detection including false positive. Which is terrible and prevents developing any extension for stronger detection. Q. Do you intend to totally replace Lockdep? A. No. Lockdep also checks if lock usage is correct. Of course, the dependency check routine should be replaced but the other functions should be still there. Q. Do you mean the dependency check routine should be replaced right away? A. No. I admit Lockdep is stable enough thanks to great efforts kernel developers have made. Lockdep and Dept, both should be in the kernel until Dept gets considered stable. Q. Stronger detection capability would give more false positive report. Which was a big problem when cross-release was introduced. Is it ok with Dept? A. It's ok. Dept allows multiple reporting thanks to simple and quite generalized design. Of course, false positive reports should be fixed anyway but it's no longer as a critical problem as it was. Signed-off-by: Byungchul Park --- include/linux/dept.h | 577 ++++++ include/linux/hardirq.h | 3 + include/linux/sched.h | 3 + init/init_task.c | 2 + init/main.c | 2 + kernel/Makefile | 1 + kernel/dependency/Makefile | 3 + kernel/dependency/dept.c | 3009 +++++++++++++++++++++++++++++++ kernel/dependency/dept_hash.h | 10 + kernel/dependency/dept_object.h | 13 + kernel/exit.c | 1 + kernel/fork.c | 2 + kernel/module/main.c | 4 + kernel/sched/core.c | 9 + lib/Kconfig.debug | 27 + lib/locking-selftest.c | 2 + 16 files changed, 3668 insertions(+) create mode 100644 include/linux/dept.h create mode 100644 kernel/dependency/Makefile create mode 100644 kernel/dependency/dept.c create mode 100644 kernel/dependency/dept_hash.h create mode 100644 kernel/dependency/dept_object.h diff --git a/include/linux/dept.h b/include/linux/dept.h new file mode 100644 index 000000000000..b6d45b4b1fd6 --- /dev/null +++ b/include/linux/dept.h @@ -0,0 +1,577 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * DEPT(DEPendency Tracker) - runtime dependency tracker + * + * Started by Byungchul Park : + * + * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park + */ + +#ifndef __LINUX_DEPT_H +#define __LINUX_DEPT_H + +#ifdef CONFIG_DEPT + +#include + +struct task_struct; + +#define DEPT_MAX_STACK_ENTRY 16 +#define DEPT_MAX_WAIT_HIST 64 +#define DEPT_MAX_ECXT_HELD 48 + +#define DEPT_MAX_SUBCLASSES 16 +#define DEPT_MAX_SUBCLASSES_EVT 2 +#define DEPT_MAX_SUBCLASSES_USR (DEPT_MAX_SUBCLASSES / DEPT_MAX_SUBCLASSES_EVT) +#define DEPT_MAX_SUBCLASSES_CACHE 2 + +#define DEPT_SIRQ 0 +#define DEPT_HIRQ 1 +#define DEPT_IRQS_NR 2 +#define DEPT_SIRQF (1UL << DEPT_SIRQ) +#define DEPT_HIRQF (1UL << DEPT_HIRQ) + +struct dept_ecxt; +struct dept_iecxt { + struct dept_ecxt *ecxt; + int enirq; + /* + * for preventing to add a new ecxt + */ + bool staled; +}; + +struct dept_wait; +struct dept_iwait { + struct dept_wait *wait; + int irq; + /* + * for preventing to add a new wait + */ + bool staled; + bool touched; +}; + +struct dept_class { + union { + struct llist_node pool_node; + struct { + /* + * reference counter for object management + */ + atomic_t ref; + + /* + * unique information about the class + */ + const char *name; + unsigned long key; + int sub_id; + + /* + * for BFS + */ + unsigned int bfs_gen; + int bfs_dist; + struct dept_class *bfs_parent; + + /* + * for hashing this object + */ + struct hlist_node hash_node; + + /* + * for linking all classes + */ + struct list_head all_node; + + /* + * for associating its dependencies + */ + struct list_head dep_head; + struct list_head dep_rev_head; + + /* + * for tracking IRQ dependencies + */ + struct dept_iecxt iecxt[DEPT_IRQS_NR]; + struct dept_iwait iwait[DEPT_IRQS_NR]; + + /* + * classified by a map embedded in task_struct, + * not an explicit map + */ + bool sched_map; + }; + }; +}; + +struct dept_key { + union { + /* + * Each byte-wise address will be used as its key. + */ + char base[DEPT_MAX_SUBCLASSES]; + + /* + * for caching the main class pointer + */ + struct dept_class *classes[DEPT_MAX_SUBCLASSES_CACHE]; + }; +}; + +struct dept_map { + const char *name; + struct dept_key *keys; + + /* + * subclass that can be set from user + */ + int sub_u; + + /* + * It's local copy for fast access to the associated classes. + * Also used for dept_key for static maps. + */ + struct dept_key map_key; + + /* + * wait timestamp associated to this map + */ + unsigned int wgen; + + /* + * whether this map should be going to be checked or not + */ + bool nocheck; +}; + +#define DEPT_MAP_INITIALIZER(n, k) \ +{ \ + .name = #n, \ + .keys = (struct dept_key *)(k), \ + .sub_u = 0, \ + .map_key = { .classes = { NULL, } }, \ + .wgen = 0U, \ + .nocheck = false, \ +} + +struct dept_stack { + union { + struct llist_node pool_node; + struct { + /* + * reference counter for object management + */ + atomic_t ref; + + /* + * backtrace entries + */ + unsigned long raw[DEPT_MAX_STACK_ENTRY]; + int nr; + }; + }; +}; + +struct dept_ecxt { + union { + struct llist_node pool_node; + struct { + /* + * reference counter for object management + */ + atomic_t ref; + + /* + * function that entered to this ecxt + */ + const char *ecxt_fn; + + /* + * event function + */ + const char *event_fn; + + /* + * associated class + */ + struct dept_class *class; + + /* + * flag indicating which IRQ has been + * enabled within the event context + */ + unsigned long enirqf; + + /* + * where the IRQ-enabled happened + */ + unsigned long enirq_ip[DEPT_IRQS_NR]; + struct dept_stack *enirq_stack[DEPT_IRQS_NR]; + + /* + * where the event context started + */ + unsigned long ecxt_ip; + struct dept_stack *ecxt_stack; + + /* + * where the event triggered + */ + unsigned long event_ip; + struct dept_stack *event_stack; + }; + }; +}; + +struct dept_wait { + union { + struct llist_node pool_node; + struct { + /* + * reference counter for object management + */ + atomic_t ref; + + /* + * function causing this wait + */ + const char *wait_fn; + + /* + * the associated class + */ + struct dept_class *class; + + /* + * which IRQ the wait was placed in + */ + unsigned long irqf; + + /* + * where the IRQ wait happened + */ + unsigned long irq_ip[DEPT_IRQS_NR]; + struct dept_stack *irq_stack[DEPT_IRQS_NR]; + + /* + * where the wait happened + */ + unsigned long wait_ip; + struct dept_stack *wait_stack; + + /* + * whether this wait is for commit in scheduler + */ + bool sched_sleep; + }; + }; +}; + +struct dept_dep { + union { + struct llist_node pool_node; + struct { + /* + * reference counter for object management + */ + atomic_t ref; + + /* + * key data of dependency + */ + struct dept_ecxt *ecxt; + struct dept_wait *wait; + + /* + * This object can be referred without dept_lock + * held but with IRQ disabled, e.g. for hash + * lookup. So deferred deletion is needed. + */ + struct rcu_head rh; + + /* + * for BFS + */ + struct list_head bfs_node; + + /* + * for hashing this object + */ + struct hlist_node hash_node; + + /* + * for linking to a class object + */ + struct list_head dep_node; + struct list_head dep_rev_node; + }; + }; +}; + +struct dept_hash { + /* + * hash table + */ + struct hlist_head *table; + + /* + * size of the table e.i. 2^bits + */ + int bits; +}; + +struct dept_pool { + const char *name; + + /* + * object size + */ + size_t obj_sz; + + /* + * the number of the static array + */ + atomic_t obj_nr; + + /* + * offset of ->pool_node + */ + size_t node_off; + + /* + * pointer to the pool + */ + void *spool; + struct llist_head boot_pool; + struct llist_head __percpu *lpool; +}; + +struct dept_ecxt_held { + /* + * associated event context + */ + struct dept_ecxt *ecxt; + + /* + * unique key for this dept_ecxt_held + */ + struct dept_map *map; + + /* + * class of the ecxt of this dept_ecxt_held + */ + struct dept_class *class; + + /* + * the wgen when the event context started + */ + unsigned int wgen; + + /* + * subclass that only works in the local context + */ + int sub_l; +}; + +struct dept_wait_hist { + /* + * associated wait + */ + struct dept_wait *wait; + + /* + * unique id of all waits system-wise until wrapped + */ + unsigned int wgen; + + /* + * local context id to identify IRQ context + */ + unsigned int ctxt_id; +}; + +struct dept_task { + /* + * all event contexts that have entered and before exiting + */ + struct dept_ecxt_held ecxt_held[DEPT_MAX_ECXT_HELD]; + int ecxt_held_pos; + + /* + * ring buffer holding all waits that have happened + */ + struct dept_wait_hist wait_hist[DEPT_MAX_WAIT_HIST]; + int wait_hist_pos; + + /* + * sequential id to identify each IRQ context + */ + unsigned int irq_id[DEPT_IRQS_NR]; + + /* + * for tracking IRQ-enabled points with cross-event + */ + unsigned int wgen_enirq[DEPT_IRQS_NR]; + + /* + * for keeping up-to-date IRQ-enabled points + */ + unsigned long enirq_ip[DEPT_IRQS_NR]; + + /* + * current effective IRQ-enabled flag + */ + unsigned long eff_enirqf; + + /* + * for reserving a current stack instance at each operation + */ + struct dept_stack *stack; + + /* + * for preventing recursive call into DEPT engine + */ + int recursive; + + /* + * for staging data to commit a wait + */ + struct dept_map stage_m; + bool stage_sched_map; + const char *stage_w_fn; + unsigned long stage_ip; + + /* + * the number of missing ecxts + */ + int missing_ecxt; + + /* + * for tracking IRQ-enable state + */ + bool hardirqs_enabled; + bool softirqs_enabled; + + /* + * whether the current is on do_exit() + */ + bool task_exit; + + /* + * whether the current is running __schedule() + */ + bool in_sched; +}; + +#define DEPT_TASK_INITIALIZER(t) \ +{ \ + .wait_hist = { { .wait = NULL, } }, \ + .ecxt_held_pos = 0, \ + .wait_hist_pos = 0, \ + .irq_id = { 0U }, \ + .wgen_enirq = { 0U }, \ + .enirq_ip = { 0UL }, \ + .eff_enirqf = 0UL, \ + .stack = NULL, \ + .recursive = 0, \ + .stage_m = DEPT_MAP_INITIALIZER((t)->stage_m, NULL), \ + .stage_sched_map = false, \ + .stage_w_fn = NULL, \ + .stage_ip = 0UL, \ + .missing_ecxt = 0, \ + .hardirqs_enabled = false, \ + .softirqs_enabled = false, \ + .task_exit = false, \ + .in_sched = false, \ +} + +extern void dept_on(void); +extern void dept_off(void); +extern void dept_init(void); +extern void dept_task_init(struct task_struct *t); +extern void dept_task_exit(struct task_struct *t); +extern void dept_free_range(void *start, unsigned int sz); +extern void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u, const char *n); +extern void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, const char *n); +extern void dept_map_copy(struct dept_map *to, struct dept_map *from); + +extern void dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l); +extern void dept_stage_wait(struct dept_map *m, struct dept_key *k, unsigned long ip, const char *w_fn); +extern void dept_request_event_wait_commit(void); +extern void dept_clean_stage(void); +extern void dept_stage_event(struct task_struct *t, unsigned long ip); +extern void dept_ecxt_enter(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *c_fn, const char *e_fn, int sub_l); +extern bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f); +extern void dept_request_event(struct dept_map *m); +extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *e_fn); +extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long ip); +extern void dept_sched_enter(void); +extern void dept_sched_exit(void); + +static inline void dept_ecxt_enter_nokeep(struct dept_map *m) +{ + dept_ecxt_enter(m, 0UL, 0UL, NULL, NULL, 0); +} + +/* + * for users who want to manage external keys + */ +extern void dept_key_init(struct dept_key *k); +extern void dept_key_destroy(struct dept_key *k); +extern void dept_map_ecxt_modify(struct dept_map *m, unsigned long e_f, struct dept_key *new_k, unsigned long new_e_f, unsigned long new_ip, const char *new_c_fn, const char *new_e_fn, int new_sub_l); + +extern void dept_softirq_enter(void); +extern void dept_hardirq_enter(void); +extern void dept_softirqs_on_ip(unsigned long ip); +extern void dept_hardirqs_on(void); +extern void dept_hardirqs_on_ip(unsigned long ip); +extern void dept_softirqs_off_ip(unsigned long ip); +extern void dept_hardirqs_off(void); +extern void dept_hardirqs_off_ip(unsigned long ip); +#else /* !CONFIG_DEPT */ +struct dept_key { }; +struct dept_map { }; +struct dept_task { }; + +#define DEPT_MAP_INITIALIZER(n, k) { } +#define DEPT_TASK_INITIALIZER(t) { } + +#define dept_on() do { } while (0) +#define dept_off() do { } while (0) +#define dept_init() do { } while (0) +#define dept_task_init(t) do { } while (0) +#define dept_task_exit(t) do { } while (0) +#define dept_free_range(s, sz) do { } while (0) +#define dept_map_init(m, k, su, n) do { (void)(n); (void)(k); } while (0) +#define dept_map_reinit(m, k, su, n) do { (void)(n); (void)(k); } while (0) +#define dept_map_copy(t, f) do { } while (0) + +#define dept_wait(m, w_f, ip, w_fn, sl) do { (void)(w_fn); } while (0) +#define dept_stage_wait(m, k, ip, w_fn) do { (void)(k); (void)(w_fn); } while (0) +#define dept_request_event_wait_commit() do { } while (0) +#define dept_clean_stage() do { } while (0) +#define dept_stage_event(t, ip) do { } while (0) +#define dept_ecxt_enter(m, e_f, ip, c_fn, e_fn, sl) do { (void)(c_fn); (void)(e_fn); } while (0) +#define dept_ecxt_holding(m, e_f) false +#define dept_request_event(m) do { } while (0) +#define dept_event(m, e_f, ip, e_fn) do { (void)(e_fn); } while (0) +#define dept_ecxt_exit(m, e_f, ip) do { } while (0) +#define dept_sched_enter() do { } while (0) +#define dept_sched_exit() do { } while (0) +#define dept_ecxt_enter_nokeep(m) do { } while (0) +#define dept_key_init(k) do { (void)(k); } while (0) +#define dept_key_destroy(k) do { (void)(k); } while (0) +#define dept_map_ecxt_modify(m, e_f, n_k, n_e_f, n_ip, n_c_fn, n_e_fn, n_sl) do { (void)(n_k); (void)(n_c_fn); (void)(n_e_fn); } while (0) + +#define dept_softirq_enter() do { } while (0) +#define dept_hardirq_enter() do { } while (0) +#define dept_softirqs_on_ip(ip) do { } while (0) +#define dept_hardirqs_on() do { } while (0) +#define dept_hardirqs_on_ip(ip) do { } while (0) +#define dept_softirqs_off_ip(ip) do { } while (0) +#define dept_hardirqs_off() do { } while (0) +#define dept_hardirqs_off_ip(ip) do { } while (0) +#endif +#endif /* __LINUX_DEPT_H */ diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index d57cab4d4c06..bb279dbbe748 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -106,6 +107,7 @@ void irq_exit_rcu(void); */ #define __nmi_enter() \ do { \ + dept_off(); \ lockdep_off(); \ arch_nmi_enter(); \ BUG_ON(in_nmi() == NMI_MASK); \ @@ -128,6 +130,7 @@ void irq_exit_rcu(void); __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ arch_nmi_exit(); \ lockdep_on(); \ + dept_on(); \ } while (0) #define nmi_exit() \ diff --git a/include/linux/sched.h b/include/linux/sched.h index eed5d65b8d1f..bb8f8e00b9ed 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -38,6 +38,7 @@ #include #include #include +#include /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context; @@ -1170,6 +1171,8 @@ struct task_struct { struct held_lock held_locks[MAX_LOCK_DEPTH]; #endif + struct dept_task dept_task; + #if defined(CONFIG_UBSAN) && !defined(CONFIG_UBSAN_TRAP) unsigned int in_ubsan; #endif diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..eb36ad68c912 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -12,6 +12,7 @@ #include #include #include +#include #include @@ -194,6 +195,7 @@ struct task_struct init_task .curr_chain_key = INITIAL_CHAIN_KEY, .lockdep_recursion = 0, #endif + .dept_task = DEPT_TASK_INITIALIZER(init_task), #ifdef CONFIG_FUNCTION_GRAPH_TRACER .ret_stack = NULL, .tracing_graph_pause = ATOMIC_INIT(0), diff --git a/init/main.c b/init/main.c index af50044deed5..107e83a77cf4 100644 --- a/init/main.c +++ b/init/main.c @@ -65,6 +65,7 @@ #include #include #include +#include #include #include #include @@ -1017,6 +1018,7 @@ asmlinkage __visible void __init __no_sanitize_address __noreturn start_kernel(v panic_param); lockdep_init(); + dept_init(); /* * Need to run this when irqs are enabled, because it wants diff --git a/kernel/Makefile b/kernel/Makefile index b69c95315480..871f3c618492 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -51,6 +51,7 @@ obj-y += livepatch/ obj-y += dma/ obj-y += entry/ obj-$(CONFIG_MODULES) += module/ +obj-y += dependency/ obj-$(CONFIG_KCMP) += kcmp.o obj-$(CONFIG_FREEZER) += freezer.o diff --git a/kernel/dependency/Makefile b/kernel/dependency/Makefile new file mode 100644 index 000000000000..b5cfb8a03c0c --- /dev/null +++ b/kernel/dependency/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_DEPT) += dept.o diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c new file mode 100644 index 000000000000..8ec638254e5f --- /dev/null +++ b/kernel/dependency/dept.c @@ -0,0 +1,3009 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * DEPT(DEPendency Tracker) - Runtime dependency tracker + * + * Started by Byungchul Park : + * + * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park + * + * DEPT provides a general way to detect deadlock possibility in runtime + * and the interest is not limited to typical lock but to every + * syncronization primitives. + * + * The following ideas were borrowed from LOCKDEP: + * + * 1) Use a graph to track relationship between classes. + * 2) Prevent performance regression using hash. + * + * The following items were enhanced from LOCKDEP: + * + * 1) Cover more deadlock cases. + * 2) Allow muliple reports. + * + * TODO: Both LOCKDEP and DEPT should co-exist until DEPT is considered + * stable. Then the dependency check routine should be replaced with + * DEPT after. It should finally look like: + * + * + * + * As is: + * + * LOCKDEP + * +-----------------------------------------+ + * | Lock usage correctness check | <-> locks + * | | + * | | + * | +-------------------------------------+ | + * | | Dependency check | | + * | | (by tracking lock acquisition order)| | + * | +-------------------------------------+ | + * | | + * +-----------------------------------------+ + * + * DEPT + * +-----------------------------------------+ + * | Dependency check | <-> waits/events + * | (by tracking wait and event context) | + * +-----------------------------------------+ + * + * + * + * To be: + * + * LOCKDEP + * +-----------------------------------------+ + * | Lock usage correctness check | <-> locks + * | | + * | | + * | (Request dependency check) | + * | T | + * +--------------------|--------------------+ + * | + * DEPT V + * +-----------------------------------------+ + * | Dependency check | <-> waits/events + * | (by tracking wait and event context) | + * +-----------------------------------------+ + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +static int dept_stop; +static int dept_per_cpu_ready; + +#define DEPT_READY_WARN (!oops_in_progress) + +/* + * Make all operations using DEPT_WARN_ON() fail on oops_in_progress and + * prevent warning message. + */ +#define DEPT_WARN_ON_ONCE(c) \ + ({ \ + int __ret = 0; \ + \ + if (likely(DEPT_READY_WARN)) \ + __ret = WARN_ONCE(c, "DEPT_WARN_ON_ONCE: " #c); \ + __ret; \ + }) + +#define DEPT_WARN_ONCE(s...) \ + ({ \ + if (likely(DEPT_READY_WARN)) \ + WARN_ONCE(1, "DEPT_WARN_ONCE: " s); \ + }) + +#define DEPT_WARN_ON(c) \ + ({ \ + int __ret = 0; \ + \ + if (likely(DEPT_READY_WARN)) \ + __ret = WARN(c, "DEPT_WARN_ON: " #c); \ + __ret; \ + }) + +#define DEPT_WARN(s...) \ + ({ \ + if (likely(DEPT_READY_WARN)) \ + WARN(1, "DEPT_WARN: " s); \ + }) + +#define DEPT_STOP(s...) \ + ({ \ + WRITE_ONCE(dept_stop, 1); \ + if (likely(DEPT_READY_WARN)) \ + WARN(1, "DEPT_STOP: " s); \ + }) + +#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) + +static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; +static arch_spinlock_t stage_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; + +/* + * DEPT internal engine should be careful in using outside functions + * e.g. printk at reporting since that kind of usage might cause + * untrackable deadlock. + */ +static atomic_t dept_outworld = ATOMIC_INIT(0); + +static inline void dept_outworld_enter(void) +{ + atomic_inc(&dept_outworld); +} + +static inline void dept_outworld_exit(void) +{ + atomic_dec(&dept_outworld); +} + +static inline bool dept_outworld_entered(void) +{ + return atomic_read(&dept_outworld); +} + +static inline bool dept_lock(void) +{ + while (!arch_spin_trylock(&dept_spin)) + if (unlikely(dept_outworld_entered())) + return false; + return true; +} + +static inline void dept_unlock(void) +{ + arch_spin_unlock(&dept_spin); +} + +/* + * whether to stack-trace on every wait or every ecxt + */ +static bool rich_stack = true; + +enum bfs_ret { + BFS_CONTINUE, + BFS_CONTINUE_REV, + BFS_DONE, + BFS_SKIP, +}; + +static inline bool after(unsigned int a, unsigned int b) +{ + return (int)(b - a) < 0; +} + +static inline bool before(unsigned int a, unsigned int b) +{ + return (int)(a - b) < 0; +} + +static inline bool valid_stack(struct dept_stack *s) +{ + return s && s->nr > 0; +} + +static inline bool valid_class(struct dept_class *c) +{ + return c->key; +} + +static inline void invalidate_class(struct dept_class *c) +{ + c->key = 0UL; +} + +static inline struct dept_ecxt *dep_e(struct dept_dep *d) +{ + return d->ecxt; +} + +static inline struct dept_wait *dep_w(struct dept_dep *d) +{ + return d->wait; +} + +static inline struct dept_class *dep_fc(struct dept_dep *d) +{ + return dep_e(d)->class; +} + +static inline struct dept_class *dep_tc(struct dept_dep *d) +{ + return dep_w(d)->class; +} + +static inline const char *irq_str(int irq) +{ + if (irq == DEPT_SIRQ) + return "softirq"; + if (irq == DEPT_HIRQ) + return "hardirq"; + return "(unknown)"; +} + +static inline struct dept_task *dept_task(void) +{ + return ¤t->dept_task; +} + +/* + * Dept doesn't work either when it's stopped by DEPT_STOP() or in a nmi + * context. + */ +static inline bool dept_working(void) +{ + return !READ_ONCE(dept_stop) && !in_nmi(); +} + +/* + * Even k == NULL is considered as a valid key because it would use + * &->map_key as the key in that case. + */ +struct dept_key __dept_no_validate__; +static inline bool valid_key(struct dept_key *k) +{ + return &__dept_no_validate__ != k; +} + +/* + * Pool + * ===================================================================== + * DEPT maintains pools to provide objects in a safe way. + * + * 1) Static pool is used at the beginning of booting time. + * 2) Local pool is tried first before the static pool. Objects that + * have been freed will be placed. + */ + +enum object_t { +#define OBJECT(id, nr) OBJECT_##id, + #include "dept_object.h" +#undef OBJECT + OBJECT_NR, +}; + +#define OBJECT(id, nr) \ +static struct dept_##id spool_##id[nr]; \ +static DEFINE_PER_CPU(struct llist_head, lpool_##id); + #include "dept_object.h" +#undef OBJECT + +static struct dept_pool pool[OBJECT_NR] = { +#define OBJECT(id, nr) { \ + .name = #id, \ + .obj_sz = sizeof(struct dept_##id), \ + .obj_nr = ATOMIC_INIT(nr), \ + .node_off = offsetof(struct dept_##id, pool_node), \ + .spool = spool_##id, \ + .lpool = &lpool_##id, }, + #include "dept_object.h" +#undef OBJECT +}; + +/* + * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is + * enabled or not because NMI and other contexts in the same CPU never + * run inside of DEPT concurrently by preventing reentrance. + */ +static void *from_pool(enum object_t t) +{ + struct dept_pool *p; + struct llist_head *h; + struct llist_node *n; + + /* + * llist_del_first() doesn't allow concurrent access e.g. + * between process and IRQ context. + */ + if (DEPT_WARN_ON(!irqs_disabled())) + return NULL; + + p = &pool[t]; + + /* + * Try local pool first. + */ + if (likely(dept_per_cpu_ready)) + h = this_cpu_ptr(p->lpool); + else + h = &p->boot_pool; + + n = llist_del_first(h); + if (n) + return (void *)n - p->node_off; + + /* + * Try static pool. + */ + if (atomic_read(&p->obj_nr) > 0) { + int idx = atomic_dec_return(&p->obj_nr); + + if (idx >= 0) + return p->spool + (idx * p->obj_sz); + } + + DEPT_INFO_ONCE("---------------------------------------------\n" + " Some of Dept internal resources are run out.\n" + " Dept might still work if the resources get freed.\n" + " However, the chances are Dept will suffer from\n" + " the lack from now. Needs to extend the internal\n" + " resource pools. Ask max.byungchul.park@gmail.com\n"); + return NULL; +} + +static void to_pool(void *o, enum object_t t) +{ + struct dept_pool *p = &pool[t]; + struct llist_head *h; + + preempt_disable(); + if (likely(dept_per_cpu_ready)) + h = this_cpu_ptr(p->lpool); + else + h = &p->boot_pool; + + llist_add(o + p->node_off, h); + preempt_enable(); +} + +#define OBJECT(id, nr) \ +static void (*ctor_##id)(struct dept_##id *a); \ +static void (*dtor_##id)(struct dept_##id *a); \ +static inline struct dept_##id *new_##id(void) \ +{ \ + struct dept_##id *a; \ + \ + a = (struct dept_##id *)from_pool(OBJECT_##id); \ + if (unlikely(!a)) \ + return NULL; \ + \ + atomic_set(&a->ref, 1); \ + \ + if (ctor_##id) \ + ctor_##id(a); \ + \ + return a; \ +} \ + \ +static inline struct dept_##id *get_##id(struct dept_##id *a) \ +{ \ + atomic_inc(&a->ref); \ + return a; \ +} \ + \ +static inline void put_##id(struct dept_##id *a) \ +{ \ + if (!atomic_dec_return(&a->ref)) { \ + if (dtor_##id) \ + dtor_##id(a); \ + to_pool(a, OBJECT_##id); \ + } \ +} \ + \ +static inline void del_##id(struct dept_##id *a) \ +{ \ + put_##id(a); \ +} \ + \ +static inline bool id##_consumed(struct dept_##id *a) \ +{ \ + return a && atomic_read(&a->ref) > 1; \ +} +#include "dept_object.h" +#undef OBJECT + +#define SET_CONSTRUCTOR(id, f) \ +static void (*ctor_##id)(struct dept_##id *a) = f + +static void initialize_dep(struct dept_dep *d) +{ + INIT_LIST_HEAD(&d->bfs_node); + INIT_LIST_HEAD(&d->dep_node); + INIT_LIST_HEAD(&d->dep_rev_node); +} +SET_CONSTRUCTOR(dep, initialize_dep); + +static void initialize_class(struct dept_class *c) +{ + int i; + + for (i = 0; i < DEPT_IRQS_NR; i++) { + struct dept_iecxt *ie = &c->iecxt[i]; + struct dept_iwait *iw = &c->iwait[i]; + + ie->ecxt = NULL; + ie->enirq = i; + ie->staled = false; + + iw->wait = NULL; + iw->irq = i; + iw->staled = false; + iw->touched = false; + } + c->bfs_gen = 0U; + + INIT_LIST_HEAD(&c->all_node); + INIT_LIST_HEAD(&c->dep_head); + INIT_LIST_HEAD(&c->dep_rev_head); +} +SET_CONSTRUCTOR(class, initialize_class); + +static void initialize_ecxt(struct dept_ecxt *e) +{ + int i; + + for (i = 0; i < DEPT_IRQS_NR; i++) { + e->enirq_stack[i] = NULL; + e->enirq_ip[i] = 0UL; + } + e->ecxt_ip = 0UL; + e->ecxt_stack = NULL; + e->enirqf = 0UL; + e->event_ip = 0UL; + e->event_stack = NULL; +} +SET_CONSTRUCTOR(ecxt, initialize_ecxt); + +static void initialize_wait(struct dept_wait *w) +{ + int i; + + for (i = 0; i < DEPT_IRQS_NR; i++) { + w->irq_stack[i] = NULL; + w->irq_ip[i] = 0UL; + } + w->wait_ip = 0UL; + w->wait_stack = NULL; + w->irqf = 0UL; +} +SET_CONSTRUCTOR(wait, initialize_wait); + +static void initialize_stack(struct dept_stack *s) +{ + s->nr = 0; +} +SET_CONSTRUCTOR(stack, initialize_stack); + +#define OBJECT(id, nr) \ +static void (*ctor_##id)(struct dept_##id *a); + #include "dept_object.h" +#undef OBJECT + +#undef SET_CONSTRUCTOR + +#define SET_DESTRUCTOR(id, f) \ +static void (*dtor_##id)(struct dept_##id *a) = f + +static void destroy_dep(struct dept_dep *d) +{ + if (dep_e(d)) + put_ecxt(dep_e(d)); + if (dep_w(d)) + put_wait(dep_w(d)); +} +SET_DESTRUCTOR(dep, destroy_dep); + +static void destroy_ecxt(struct dept_ecxt *e) +{ + int i; + + for (i = 0; i < DEPT_IRQS_NR; i++) + if (e->enirq_stack[i]) + put_stack(e->enirq_stack[i]); + if (e->class) + put_class(e->class); + if (e->ecxt_stack) + put_stack(e->ecxt_stack); + if (e->event_stack) + put_stack(e->event_stack); +} +SET_DESTRUCTOR(ecxt, destroy_ecxt); + +static void destroy_wait(struct dept_wait *w) +{ + int i; + + for (i = 0; i < DEPT_IRQS_NR; i++) + if (w->irq_stack[i]) + put_stack(w->irq_stack[i]); + if (w->class) + put_class(w->class); + if (w->wait_stack) + put_stack(w->wait_stack); +} +SET_DESTRUCTOR(wait, destroy_wait); + +#define OBJECT(id, nr) \ +static void (*dtor_##id)(struct dept_##id *a); + #include "dept_object.h" +#undef OBJECT + +#undef SET_DESTRUCTOR + +/* + * Caching and hashing + * ===================================================================== + * DEPT makes use of caching and hashing to improve performance. Each + * object can be obtained in O(1) with its key. + * + * NOTE: Currently we assume all the objects in the hashs will never be + * removed. Implement it when needed. + */ + +/* + * Some information might be lost but it's only for hashing key. + */ +static inline unsigned long mix(unsigned long a, unsigned long b) +{ + int halfbits = sizeof(unsigned long) * 8 / 2; + unsigned long halfmask = (1UL << halfbits) - 1UL; + + return (a << halfbits) | (b & halfmask); +} + +static bool cmp_dep(struct dept_dep *d1, struct dept_dep *d2) +{ + return dep_fc(d1)->key == dep_fc(d2)->key && + dep_tc(d1)->key == dep_tc(d2)->key; +} + +static unsigned long key_dep(struct dept_dep *d) +{ + return mix(dep_fc(d)->key, dep_tc(d)->key); +} + +static bool cmp_class(struct dept_class *c1, struct dept_class *c2) +{ + return c1->key == c2->key; +} + +static unsigned long key_class(struct dept_class *c) +{ + return c->key; +} + +#define HASH(id, bits) \ +static struct hlist_head table_##id[1 << (bits)]; \ + \ +static inline struct hlist_head *head_##id(struct dept_##id *a) \ +{ \ + return table_##id + hash_long(key_##id(a), bits); \ +} \ + \ +static inline struct dept_##id *hash_lookup_##id(struct dept_##id *a) \ +{ \ + struct dept_##id *b; \ + \ + hlist_for_each_entry_rcu(b, head_##id(a), hash_node) \ + if (cmp_##id(a, b)) \ + return b; \ + return NULL; \ +} \ + \ +static inline void hash_add_##id(struct dept_##id *a) \ +{ \ + get_##id(a); \ + hlist_add_head_rcu(&a->hash_node, head_##id(a)); \ +} \ + \ +static inline void hash_del_##id(struct dept_##id *a) \ +{ \ + hlist_del_rcu(&a->hash_node); \ + put_##id(a); \ +} +#include "dept_hash.h" +#undef HASH + +static inline struct dept_dep *lookup_dep(struct dept_class *fc, + struct dept_class *tc) +{ + struct dept_ecxt onetime_e = { .class = fc }; + struct dept_wait onetime_w = { .class = tc }; + struct dept_dep onetime_d = { .ecxt = &onetime_e, + .wait = &onetime_w }; + return hash_lookup_dep(&onetime_d); +} + +static inline struct dept_class *lookup_class(unsigned long key) +{ + struct dept_class onetime_c = { .key = key }; + + return hash_lookup_class(&onetime_c); +} + +/* + * Report + * ===================================================================== + * DEPT prints useful information to help debuging on detection of + * problematic dependency. + */ + +static inline void print_ip_stack(unsigned long ip, struct dept_stack *s) +{ + if (ip) + print_ip_sym(KERN_WARNING, ip); + + if (valid_stack(s)) { + pr_warn("stacktrace:\n"); + stack_trace_print(s->raw, s->nr, 5); + } + + if (!ip && !valid_stack(s)) + pr_warn("(N/A)\n"); +} + +#define print_spc(spc, fmt, ...) \ + pr_warn("%*c" fmt, (spc) * 4, ' ', ##__VA_ARGS__) + +static void print_diagram(struct dept_dep *d) +{ + struct dept_ecxt *e = dep_e(d); + struct dept_wait *w = dep_w(d); + struct dept_class *fc = dep_fc(d); + struct dept_class *tc = dep_tc(d); + unsigned long irqf; + int irq; + bool firstline = true; + int spc = 1; + const char *w_fn = w->wait_fn ?: "(unknown)"; + const char *e_fn = e->event_fn ?: "(unknown)"; + const char *c_fn = e->ecxt_fn ?: "(unknown)"; + const char *fc_n = fc->sched_map ? "" : (fc->name ?: "(unknown)"); + const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); + + irqf = e->enirqf & w->irqf; + for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + if (!firstline) + pr_warn("\nor\n\n"); + firstline = false; + + print_spc(spc, "[S] %s(%s:%d)\n", c_fn, fc_n, fc->sub_id); + print_spc(spc, " <%s interrupt>\n", irq_str(irq)); + print_spc(spc + 1, "[W] %s(%s:%d)\n", w_fn, tc_n, tc->sub_id); + print_spc(spc, "[E] %s(%s:%d)\n", e_fn, fc_n, fc->sub_id); + } + + if (!irqf) { + print_spc(spc, "[S] %s(%s:%d)\n", c_fn, fc_n, fc->sub_id); + print_spc(spc, "[W] %s(%s:%d)\n", w_fn, tc_n, tc->sub_id); + print_spc(spc, "[E] %s(%s:%d)\n", e_fn, fc_n, fc->sub_id); + } +} + +static void print_dep(struct dept_dep *d) +{ + struct dept_ecxt *e = dep_e(d); + struct dept_wait *w = dep_w(d); + struct dept_class *fc = dep_fc(d); + struct dept_class *tc = dep_tc(d); + unsigned long irqf; + int irq; + const char *w_fn = w->wait_fn ?: "(unknown)"; + const char *e_fn = e->event_fn ?: "(unknown)"; + const char *c_fn = e->ecxt_fn ?: "(unknown)"; + const char *fc_n = fc->sched_map ? "" : (fc->name ?: "(unknown)"); + const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); + + irqf = e->enirqf & w->irqf; + for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + pr_warn("%s has been enabled:\n", irq_str(irq)); + print_ip_stack(e->enirq_ip[irq], e->enirq_stack[irq]); + pr_warn("\n"); + + pr_warn("[S] %s(%s:%d):\n", c_fn, fc_n, fc->sub_id); + print_ip_stack(e->ecxt_ip, e->ecxt_stack); + pr_warn("\n"); + + pr_warn("[W] %s(%s:%d) in %s context:\n", + w_fn, tc_n, tc->sub_id, irq_str(irq)); + print_ip_stack(w->irq_ip[irq], w->irq_stack[irq]); + pr_warn("\n"); + + pr_warn("[E] %s(%s:%d):\n", e_fn, fc_n, fc->sub_id); + print_ip_stack(e->event_ip, e->event_stack); + } + + if (!irqf) { + pr_warn("[S] %s(%s:%d):\n", c_fn, fc_n, fc->sub_id); + print_ip_stack(e->ecxt_ip, e->ecxt_stack); + pr_warn("\n"); + + pr_warn("[W] %s(%s:%d):\n", w_fn, tc_n, tc->sub_id); + print_ip_stack(w->wait_ip, w->wait_stack); + pr_warn("\n"); + + pr_warn("[E] %s(%s:%d):\n", e_fn, fc_n, fc->sub_id); + print_ip_stack(e->event_ip, e->event_stack); + } +} + +static void save_current_stack(int skip); + +/* + * Print all classes in a circle. + */ +static void print_circle(struct dept_class *c) +{ + struct dept_class *fc = c->bfs_parent; + struct dept_class *tc = c; + int i; + + dept_outworld_enter(); + save_current_stack(6); + + pr_warn("===================================================\n"); + pr_warn("DEPT: Circular dependency has been detected.\n"); + pr_warn("%s %.*s %s\n", init_utsname()->release, + (int)strcspn(init_utsname()->version, " "), + init_utsname()->version, + print_tainted()); + pr_warn("---------------------------------------------------\n"); + pr_warn("summary\n"); + pr_warn("---------------------------------------------------\n"); + + if (fc == tc) + pr_warn("*** AA DEADLOCK ***\n\n"); + else + pr_warn("*** DEADLOCK ***\n\n"); + + i = 0; + do { + struct dept_dep *d = lookup_dep(fc, tc); + + pr_warn("context %c\n", 'A' + (i++)); + print_diagram(d); + if (fc != c) + pr_warn("\n"); + + tc = fc; + fc = fc->bfs_parent; + } while (tc != c); + + pr_warn("\n"); + pr_warn("[S]: start of the event context\n"); + pr_warn("[W]: the wait blocked\n"); + pr_warn("[E]: the event not reachable\n"); + + i = 0; + do { + struct dept_dep *d = lookup_dep(fc, tc); + + pr_warn("---------------------------------------------------\n"); + pr_warn("context %c's detail\n", 'A' + i); + pr_warn("---------------------------------------------------\n"); + pr_warn("context %c\n", 'A' + (i++)); + print_diagram(d); + pr_warn("\n"); + print_dep(d); + + tc = fc; + fc = fc->bfs_parent; + } while (tc != c); + + pr_warn("---------------------------------------------------\n"); + pr_warn("information that might be helpful\n"); + pr_warn("---------------------------------------------------\n"); + dump_stack(); + + dept_outworld_exit(); +} + +/* + * BFS(Breadth First Search) + * ===================================================================== + * Whenever a new dependency is added into the graph, search the graph + * for a new circular dependency. + */ + +static inline void enqueue(struct list_head *h, struct dept_dep *d) +{ + list_add_tail(&d->bfs_node, h); +} + +static inline struct dept_dep *dequeue(struct list_head *h) +{ + struct dept_dep *d; + + d = list_first_entry(h, struct dept_dep, bfs_node); + list_del(&d->bfs_node); + return d; +} + +static inline bool empty(struct list_head *h) +{ + return list_empty(h); +} + +static void extend_queue(struct list_head *h, struct dept_class *cur) +{ + struct dept_dep *d; + + list_for_each_entry(d, &cur->dep_head, dep_node) { + struct dept_class *next = dep_tc(d); + + if (cur->bfs_gen == next->bfs_gen) + continue; + next->bfs_gen = cur->bfs_gen; + next->bfs_dist = cur->bfs_dist + 1; + next->bfs_parent = cur; + enqueue(h, d); + } +} + +static void extend_queue_rev(struct list_head *h, struct dept_class *cur) +{ + struct dept_dep *d; + + list_for_each_entry(d, &cur->dep_rev_head, dep_rev_node) { + struct dept_class *next = dep_fc(d); + + if (cur->bfs_gen == next->bfs_gen) + continue; + next->bfs_gen = cur->bfs_gen; + next->bfs_dist = cur->bfs_dist + 1; + next->bfs_parent = cur; + enqueue(h, d); + } +} + +typedef enum bfs_ret bfs_f(struct dept_dep *d, void *in, void **out); +static unsigned int bfs_gen; + +/* + * NOTE: Must be called with dept_lock held. + */ +static void bfs(struct dept_class *c, bfs_f *cb, void *in, void **out) +{ + LIST_HEAD(q); + enum bfs_ret ret; + + if (DEPT_WARN_ON(!cb)) + return; + + /* + * Avoid zero bfs_gen. + */ + bfs_gen = bfs_gen + 1 ?: 1; + + c->bfs_gen = bfs_gen; + c->bfs_dist = 0; + c->bfs_parent = c; + + ret = cb(NULL, in, out); + if (ret == BFS_DONE) + return; + if (ret == BFS_SKIP) + return; + if (ret == BFS_CONTINUE) + extend_queue(&q, c); + if (ret == BFS_CONTINUE_REV) + extend_queue_rev(&q, c); + + while (!empty(&q)) { + struct dept_dep *d = dequeue(&q); + + ret = cb(d, in, out); + if (ret == BFS_DONE) + break; + if (ret == BFS_SKIP) + continue; + if (ret == BFS_CONTINUE) + extend_queue(&q, dep_tc(d)); + if (ret == BFS_CONTINUE_REV) + extend_queue_rev(&q, dep_fc(d)); + } + + while (!empty(&q)) + dequeue(&q); +} + +/* + * Main operations + * ===================================================================== + * Add dependencies - Each new dependency is added into the graph and + * checked if it forms a circular dependency. + * + * Track waits - Waits are queued into the ring buffer for later use to + * generate appropriate dependencies with cross-event. + * + * Track event contexts(ecxt) - Event contexts are pushed into local + * stack for later use to generate appropriate dependencies with waits. + */ + +static inline unsigned long cur_enirqf(void); +static inline int cur_irq(void); +static inline unsigned int cur_ctxt_id(void); + +static inline struct dept_iecxt *iecxt(struct dept_class *c, int irq) +{ + return &c->iecxt[irq]; +} + +static inline struct dept_iwait *iwait(struct dept_class *c, int irq) +{ + return &c->iwait[irq]; +} + +static inline void stale_iecxt(struct dept_iecxt *ie) +{ + if (ie->ecxt) + put_ecxt(ie->ecxt); + + WRITE_ONCE(ie->ecxt, NULL); + WRITE_ONCE(ie->staled, true); +} + +static inline void set_iecxt(struct dept_iecxt *ie, struct dept_ecxt *e) +{ + /* + * ->ecxt will never be updated once getting set until the class + * gets removed. + */ + if (ie->ecxt) + DEPT_WARN_ON(1); + else + WRITE_ONCE(ie->ecxt, get_ecxt(e)); +} + +static inline void stale_iwait(struct dept_iwait *iw) +{ + if (iw->wait) + put_wait(iw->wait); + + WRITE_ONCE(iw->wait, NULL); + WRITE_ONCE(iw->staled, true); +} + +static inline void set_iwait(struct dept_iwait *iw, struct dept_wait *w) +{ + /* + * ->wait will never be updated once getting set until the class + * gets removed. + */ + if (iw->wait) + DEPT_WARN_ON(1); + else + WRITE_ONCE(iw->wait, get_wait(w)); + + iw->touched = true; +} + +static inline void touch_iwait(struct dept_iwait *iw) +{ + iw->touched = true; +} + +static inline void untouch_iwait(struct dept_iwait *iw) +{ + iw->touched = false; +} + +static inline struct dept_stack *get_current_stack(void) +{ + struct dept_stack *s = dept_task()->stack; + + return s ? get_stack(s) : NULL; +} + +static inline void prepare_current_stack(void) +{ + struct dept_stack *s = dept_task()->stack; + + /* + * The dept_stack is already ready. + */ + if (s && !stack_consumed(s)) { + s->nr = 0; + return; + } + + if (s) + put_stack(s); + + s = dept_task()->stack = new_stack(); + if (!s) + return; + + get_stack(s); + del_stack(s); +} + +static void save_current_stack(int skip) +{ + struct dept_stack *s = dept_task()->stack; + + if (!s) + return; + if (valid_stack(s)) + return; + + s->nr = stack_trace_save(s->raw, DEPT_MAX_STACK_ENTRY, skip); +} + +static void finish_current_stack(void) +{ + struct dept_stack *s = dept_task()->stack; + + if (stack_consumed(s)) + save_current_stack(2); +} + +/* + * FIXME: For now, disable LOCKDEP while DEPT is working. + * + * Both LOCKDEP and DEPT report it on a deadlock detection using + * printk taking the risk of another deadlock that might be caused by + * locks of console or printk between inside and outside of them. + * + * For DEPT, it's no problem since multiple reports are allowed. But it + * would be a bad idea for LOCKDEP since it will stop even on a singe + * report. So we need to prevent LOCKDEP from its reporting the risk + * DEPT would take when reporting something. + */ +#include + +void dept_off(void) +{ + dept_task()->recursive++; + lockdep_off(); +} + +void dept_on(void) +{ + dept_task()->recursive--; + lockdep_on(); +} + +static inline unsigned long dept_enter(void) +{ + unsigned long flags; + + flags = arch_local_irq_save(); + dept_off(); + prepare_current_stack(); + return flags; +} + +static inline void dept_exit(unsigned long flags) +{ + finish_current_stack(); + dept_on(); + arch_local_irq_restore(flags); +} + +static inline unsigned long dept_enter_recursive(void) +{ + unsigned long flags; + + flags = arch_local_irq_save(); + return flags; +} + +static inline void dept_exit_recursive(unsigned long flags) +{ + arch_local_irq_restore(flags); +} + +/* + * NOTE: Must be called with dept_lock held. + */ +static struct dept_dep *__add_dep(struct dept_ecxt *e, + struct dept_wait *w) +{ + struct dept_dep *d; + + if (DEPT_WARN_ON(!valid_class(e->class))) + return NULL; + + if (DEPT_WARN_ON(!valid_class(w->class))) + return NULL; + + if (lookup_dep(e->class, w->class)) + return NULL; + + d = new_dep(); + if (unlikely(!d)) + return NULL; + + d->ecxt = get_ecxt(e); + d->wait = get_wait(w); + + /* + * Add the dependency into hash and graph. + */ + hash_add_dep(d); + list_add(&d->dep_node, &dep_fc(d)->dep_head); + list_add(&d->dep_rev_node, &dep_tc(d)->dep_rev_head); + return d; +} + +static enum bfs_ret cb_check_dl(struct dept_dep *d, + void *in, void **out) +{ + struct dept_dep *new = (struct dept_dep *)in; + + /* + * initial condition for this BFS search + */ + if (!d) { + dep_tc(new)->bfs_parent = dep_fc(new); + + if (dep_tc(new) != dep_fc(new)) + return BFS_CONTINUE; + + /* + * AA circle does not make additional deadlock. We don't + * have to continue this BFS search. + */ + print_circle(dep_tc(new)); + return BFS_DONE; + } + + /* + * Allow multiple reports. + */ + if (dep_tc(d) == dep_fc(new)) + print_circle(dep_tc(new)); + + return BFS_CONTINUE; +} + +/* + * This function is actually in charge of reporting. + */ +static inline void check_dl_bfs(struct dept_dep *d) +{ + bfs(dep_tc(d), cb_check_dl, (void *)d, NULL); +} + +static enum bfs_ret cb_find_iw(struct dept_dep *d, void *in, void **out) +{ + int irq = *(int *)in; + struct dept_class *fc; + struct dept_iwait *iw; + + if (DEPT_WARN_ON(!out)) + return BFS_DONE; + + /* + * initial condition for this BFS search + */ + if (!d) + return BFS_CONTINUE_REV; + + fc = dep_fc(d); + iw = iwait(fc, irq); + + /* + * If any parent's ->wait was set, then the children would've + * been touched. + */ + if (!iw->touched) + return BFS_SKIP; + + if (!iw->wait) + return BFS_CONTINUE_REV; + + *out = iw; + return BFS_DONE; +} + +static struct dept_iwait *find_iw_bfs(struct dept_class *c, int irq) +{ + struct dept_iwait *iw = iwait(c, irq); + struct dept_iwait *found = NULL; + + if (iw->wait) + return iw; + + /* + * '->touched == false' guarantees there's no parent that has + * been set ->wait. + */ + if (!iw->touched) + return NULL; + + bfs(c, cb_find_iw, (void *)&irq, (void **)&found); + + if (found) + return found; + + untouch_iwait(iw); + return NULL; +} + +static enum bfs_ret cb_touch_iw_find_ie(struct dept_dep *d, void *in, + void **out) +{ + int irq = *(int *)in; + struct dept_class *tc; + struct dept_iecxt *ie; + struct dept_iwait *iw; + + if (DEPT_WARN_ON(!out)) + return BFS_DONE; + + /* + * initial condition for this BFS search + */ + if (!d) + return BFS_CONTINUE; + + tc = dep_tc(d); + ie = iecxt(tc, irq); + iw = iwait(tc, irq); + + touch_iwait(iw); + + if (!ie->ecxt) + return BFS_CONTINUE; + + if (!*out) + *out = ie; + + return BFS_CONTINUE; +} + +static struct dept_iecxt *touch_iw_find_ie_bfs(struct dept_class *c, + int irq) +{ + struct dept_iecxt *ie = iecxt(c, irq); + struct dept_iwait *iw = iwait(c, irq); + struct dept_iecxt *found = ie->ecxt ? ie : NULL; + + touch_iwait(iw); + bfs(c, cb_touch_iw_find_ie, (void *)&irq, (void **)&found); + return found; +} + +/* + * Should be called with dept_lock held. + */ +static void __add_idep(struct dept_iecxt *ie, struct dept_iwait *iw) +{ + struct dept_dep *new; + + /* + * There's nothing to do. + */ + if (!ie || !iw || !ie->ecxt || !iw->wait) + return; + + new = __add_dep(ie->ecxt, iw->wait); + + /* + * Deadlock detected. Let check_dl_bfs() report it. + */ + if (new) { + check_dl_bfs(new); + stale_iecxt(ie); + stale_iwait(iw); + } + + /* + * If !new, it would be the case of lack of object resource. + * Just let it go and get checked by other chances. Retrying is + * meaningless in that case. + */ +} + +static void set_check_iecxt(struct dept_class *c, int irq, + struct dept_ecxt *e) +{ + struct dept_iecxt *ie = iecxt(c, irq); + + set_iecxt(ie, e); + __add_idep(ie, find_iw_bfs(c, irq)); +} + +static void set_check_iwait(struct dept_class *c, int irq, + struct dept_wait *w) +{ + struct dept_iwait *iw = iwait(c, irq); + + set_iwait(iw, w); + __add_idep(touch_iw_find_ie_bfs(c, irq), iw); +} + +static void add_iecxt(struct dept_class *c, int irq, struct dept_ecxt *e, + bool stack) +{ + /* + * This access is safe since we ensure e->class has set locally. + */ + struct dept_task *dt = dept_task(); + struct dept_iecxt *ie = iecxt(c, irq); + + if (DEPT_WARN_ON(!valid_class(c))) + return; + + if (unlikely(READ_ONCE(ie->staled))) + return; + + /* + * Skip add_iecxt() if ie->ecxt has ever been set at least once. + * Which means it has a valid ->ecxt or been staled. + */ + if (READ_ONCE(ie->ecxt)) + return; + + if (unlikely(!dept_lock())) + return; + + if (unlikely(ie->staled)) + goto unlock; + if (ie->ecxt) + goto unlock; + + e->enirqf |= (1UL << irq); + + /* + * Should be NULL since it's the first time that these + * enirq_{ip,stack}[irq] have ever set. + */ + DEPT_WARN_ON(e->enirq_ip[irq]); + DEPT_WARN_ON(e->enirq_stack[irq]); + + e->enirq_ip[irq] = dt->enirq_ip[irq]; + e->enirq_stack[irq] = stack ? get_current_stack() : NULL; + + set_check_iecxt(c, irq, e); +unlock: + dept_unlock(); +} + +static void add_iwait(struct dept_class *c, int irq, struct dept_wait *w) +{ + struct dept_iwait *iw = iwait(c, irq); + + if (DEPT_WARN_ON(!valid_class(c))) + return; + + if (unlikely(READ_ONCE(iw->staled))) + return; + + /* + * Skip add_iwait() if iw->wait has ever been set at least once. + * Which means it has a valid ->wait or been staled. + */ + if (READ_ONCE(iw->wait)) + return; + + if (unlikely(!dept_lock())) + return; + + if (unlikely(iw->staled)) + goto unlock; + if (iw->wait) + goto unlock; + + w->irqf |= (1UL << irq); + + /* + * Should be NULL since it's the first time that these + * irq_{ip,stack}[irq] have ever set. + */ + DEPT_WARN_ON(w->irq_ip[irq]); + DEPT_WARN_ON(w->irq_stack[irq]); + + w->irq_ip[irq] = w->wait_ip; + w->irq_stack[irq] = get_current_stack(); + + set_check_iwait(c, irq, w); +unlock: + dept_unlock(); +} + +static inline struct dept_wait_hist *hist(int pos) +{ + struct dept_task *dt = dept_task(); + + return dt->wait_hist + (pos % DEPT_MAX_WAIT_HIST); +} + +static inline int hist_pos_next(void) +{ + struct dept_task *dt = dept_task(); + + return dt->wait_hist_pos % DEPT_MAX_WAIT_HIST; +} + +static inline void hist_advance(void) +{ + struct dept_task *dt = dept_task(); + + dt->wait_hist_pos++; + dt->wait_hist_pos %= DEPT_MAX_WAIT_HIST; +} + +static inline struct dept_wait_hist *new_hist(void) +{ + struct dept_wait_hist *wh = hist(hist_pos_next()); + + hist_advance(); + return wh; +} + +static void add_hist(struct dept_wait *w, unsigned int wg, unsigned int ctxt_id) +{ + struct dept_wait_hist *wh = new_hist(); + + if (likely(wh->wait)) + put_wait(wh->wait); + + wh->wait = get_wait(w); + wh->wgen = wg; + wh->ctxt_id = ctxt_id; +} + +/* + * Should be called after setting up e's iecxt and w's iwait. + */ +static void add_dep(struct dept_ecxt *e, struct dept_wait *w) +{ + struct dept_class *fc = e->class; + struct dept_class *tc = w->class; + struct dept_dep *d; + int i; + + if (lookup_dep(fc, tc)) + return; + + if (unlikely(!dept_lock())) + return; + + /* + * __add_dep() will lookup_dep() again with lock held. + */ + d = __add_dep(e, w); + if (d) { + check_dl_bfs(d); + + for (i = 0; i < DEPT_IRQS_NR; i++) { + struct dept_iwait *fiw = iwait(fc, i); + struct dept_iecxt *found_ie; + struct dept_iwait *found_iw; + + /* + * '->touched == false' guarantees there's no + * parent that has been set ->wait. + */ + if (!fiw->touched) + continue; + + /* + * find_iw_bfs() will untouch the iwait if + * not found. + */ + found_iw = find_iw_bfs(fc, i); + + if (!found_iw) + continue; + + found_ie = touch_iw_find_ie_bfs(tc, i); + __add_idep(found_ie, found_iw); + } + } + dept_unlock(); +} + +static atomic_t wgen = ATOMIC_INIT(1); + +static void add_wait(struct dept_class *c, unsigned long ip, + const char *w_fn, int sub_l, bool sched_sleep) +{ + struct dept_task *dt = dept_task(); + struct dept_wait *w; + unsigned int wg = 0U; + int irq; + int i; + + if (DEPT_WARN_ON(!valid_class(c))) + return; + + w = new_wait(); + if (unlikely(!w)) + return; + + WRITE_ONCE(w->class, get_class(c)); + w->wait_ip = ip; + w->wait_fn = w_fn; + w->wait_stack = get_current_stack(); + w->sched_sleep = sched_sleep; + + irq = cur_irq(); + if (irq < DEPT_IRQS_NR) + add_iwait(c, irq, w); + + /* + * Avoid adding dependency between user aware nested ecxt and + * wait. + */ + for (i = dt->ecxt_held_pos - 1; i >= 0; i--) { + struct dept_ecxt_held *eh; + + eh = dt->ecxt_held + i; + + /* + * the case of invalid key'ed one + */ + if (!eh->ecxt) + continue; + + if (eh->ecxt->class != c || eh->sub_l == sub_l) + add_dep(eh->ecxt, w); + } + + if (!wait_consumed(w) && !rich_stack) { + if (w->wait_stack) + put_stack(w->wait_stack); + w->wait_stack = NULL; + } + + /* + * Avoid zero wgen. + */ + wg = atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen); + add_hist(w, wg, cur_ctxt_id()); + + del_wait(w); +} + +static bool add_ecxt(struct dept_map *m, struct dept_class *c, + unsigned long ip, const char *c_fn, + const char *e_fn, int sub_l) +{ + struct dept_task *dt = dept_task(); + struct dept_ecxt_held *eh; + struct dept_ecxt *e; + unsigned long irqf; + int irq; + + if (DEPT_WARN_ON(!valid_class(c))) + return false; + + if (DEPT_WARN_ON_ONCE(dt->ecxt_held_pos >= DEPT_MAX_ECXT_HELD)) + return false; + + if (m->nocheck) { + eh = dt->ecxt_held + (dt->ecxt_held_pos++); + eh->ecxt = NULL; + eh->map = m; + eh->class = get_class(c); + eh->wgen = atomic_read(&wgen); + eh->sub_l = sub_l; + + return true; + } + + e = new_ecxt(); + if (unlikely(!e)) + return false; + + e->class = get_class(c); + e->ecxt_ip = ip; + e->ecxt_stack = ip && rich_stack ? get_current_stack() : NULL; + e->event_fn = e_fn; + e->ecxt_fn = c_fn; + + eh = dt->ecxt_held + (dt->ecxt_held_pos++); + eh->ecxt = get_ecxt(e); + eh->map = m; + eh->class = get_class(c); + eh->wgen = atomic_read(&wgen); + eh->sub_l = sub_l; + + irqf = cur_enirqf(); + for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) + add_iecxt(c, irq, e, false); + + del_ecxt(e); + return true; +} + +static int find_ecxt_pos(struct dept_map *m, struct dept_class *c, + bool newfirst) +{ + struct dept_task *dt = dept_task(); + int i; + + if (newfirst) { + for (i = dt->ecxt_held_pos - 1; i >= 0; i--) { + struct dept_ecxt_held *eh; + + eh = dt->ecxt_held + i; + if (eh->map == m && eh->class == c) + return i; + } + } else { + for (i = 0; i < dt->ecxt_held_pos; i++) { + struct dept_ecxt_held *eh; + + eh = dt->ecxt_held + i; + if (eh->map == m && eh->class == c) + return i; + } + } + return -1; +} + +static bool pop_ecxt(struct dept_map *m, struct dept_class *c) +{ + struct dept_task *dt = dept_task(); + int pos; + int i; + + pos = find_ecxt_pos(m, c, true); + if (pos == -1) + return false; + + if (dt->ecxt_held[pos].class) + put_class(dt->ecxt_held[pos].class); + + if (dt->ecxt_held[pos].ecxt) + put_ecxt(dt->ecxt_held[pos].ecxt); + + dt->ecxt_held_pos--; + + for (i = pos; i < dt->ecxt_held_pos; i++) + dt->ecxt_held[i] = dt->ecxt_held[i + 1]; + return true; +} + +static inline bool good_hist(struct dept_wait_hist *wh, unsigned int wg) +{ + return wh->wait != NULL && before(wg, wh->wgen); +} + +/* + * Binary-search the ring buffer for the earliest valid wait. + */ +static int find_hist_pos(unsigned int wg) +{ + int oldest; + int l; + int r; + int pos; + + oldest = hist_pos_next(); + if (unlikely(good_hist(hist(oldest), wg))) { + DEPT_INFO_ONCE("Need to expand the ring buffer.\n"); + return oldest; + } + + l = oldest + 1; + r = oldest + DEPT_MAX_WAIT_HIST - 1; + for (pos = (l + r) / 2; l <= r; pos = (l + r) / 2) { + struct dept_wait_hist *p = hist(pos - 1); + struct dept_wait_hist *wh = hist(pos); + + if (!good_hist(p, wg) && good_hist(wh, wg)) + return pos % DEPT_MAX_WAIT_HIST; + if (good_hist(wh, wg)) + r = pos - 1; + else + l = pos + 1; + } + return -1; +} + +static void do_event(struct dept_map *m, struct dept_class *c, + unsigned int wg, unsigned long ip) +{ + struct dept_task *dt = dept_task(); + struct dept_wait_hist *wh; + struct dept_ecxt_held *eh; + unsigned int ctxt_id; + int end; + int pos; + int i; + + if (DEPT_WARN_ON(!valid_class(c))) + return; + + if (m->nocheck) + return; + + /* + * The event was triggered before wait. + */ + if (!wg) + return; + + pos = find_ecxt_pos(m, c, false); + if (pos == -1) + return; + + eh = dt->ecxt_held + pos; + + if (DEPT_WARN_ON(!eh->ecxt)) + return; + + eh->ecxt->event_ip = ip; + eh->ecxt->event_stack = get_current_stack(); + + /* + * The ecxt already has done what it needs. + */ + if (!before(wg, eh->wgen)) + return; + + pos = find_hist_pos(wg); + if (pos == -1) + return; + + ctxt_id = cur_ctxt_id(); + end = hist_pos_next(); + end = end > pos ? end : end + DEPT_MAX_WAIT_HIST; + for (wh = hist(pos); pos < end; wh = hist(++pos)) { + if (after(wh->wgen, eh->wgen)) + break; + + if (dt->in_sched && wh->wait->sched_sleep) + continue; + + if (wh->ctxt_id == ctxt_id) + add_dep(eh->ecxt, wh->wait); + } + + for (i = 0; i < DEPT_IRQS_NR; i++) { + struct dept_ecxt *e; + + if (before(dt->wgen_enirq[i], wg)) + continue; + + e = eh->ecxt; + add_iecxt(e->class, i, e, false); + } +} + +static void del_dep_rcu(struct rcu_head *rh) +{ + struct dept_dep *d = container_of(rh, struct dept_dep, rh); + + preempt_disable(); + del_dep(d); + preempt_enable(); +} + +/* + * NOTE: Must be called with dept_lock held. + */ +static void disconnect_class(struct dept_class *c) +{ + struct dept_dep *d, *n; + int i; + + list_for_each_entry_safe(d, n, &c->dep_head, dep_node) { + list_del_rcu(&d->dep_node); + list_del_rcu(&d->dep_rev_node); + hash_del_dep(d); + call_rcu(&d->rh, del_dep_rcu); + } + + list_for_each_entry_safe(d, n, &c->dep_rev_head, dep_rev_node) { + list_del_rcu(&d->dep_node); + list_del_rcu(&d->dep_rev_node); + hash_del_dep(d); + call_rcu(&d->rh, del_dep_rcu); + } + + for (i = 0; i < DEPT_IRQS_NR; i++) { + stale_iecxt(iecxt(c, i)); + stale_iwait(iwait(c, i)); + } +} + +/* + * Context control + * ===================================================================== + * Whether a wait is in {hard,soft}-IRQ context or whether + * {hard,soft}-IRQ has been enabled on the way to an event is very + * important to check dependency. All those things should be tracked. + */ + +static inline unsigned long cur_enirqf(void) +{ + struct dept_task *dt = dept_task(); + int he = dt->hardirqs_enabled; + int se = dt->softirqs_enabled; + + if (he) + return DEPT_HIRQF | (se ? DEPT_SIRQF : 0UL); + return 0UL; +} + +static inline int cur_irq(void) +{ + if (lockdep_softirq_context(current)) + return DEPT_SIRQ; + if (lockdep_hardirq_context()) + return DEPT_HIRQ; + return DEPT_IRQS_NR; +} + +static inline unsigned int cur_ctxt_id(void) +{ + struct dept_task *dt = dept_task(); + int irq = cur_irq(); + + /* + * Normal process context + */ + if (irq == DEPT_IRQS_NR) + return 0U; + + return dt->irq_id[irq] | (1UL << irq); +} + +static void enirq_transition(int irq) +{ + struct dept_task *dt = dept_task(); + int i; + + /* + * READ wgen >= wgen of an event with IRQ enabled has been + * observed on the way to the event means, the IRQ can cut in + * within the ecxt. Used for cross-event detection. + * + * wait context event context(ecxt) + * ------------ ------------------- + * wait event + * WRITE wgen + * observe IRQ enabled + * READ wgen + * keep the wgen locally + * + * on the event + * check the local wgen + */ + dt->wgen_enirq[irq] = atomic_read(&wgen); + + for (i = dt->ecxt_held_pos - 1; i >= 0; i--) { + struct dept_ecxt_held *eh; + struct dept_ecxt *e; + + eh = dt->ecxt_held + i; + e = eh->ecxt; + if (e) + add_iecxt(e->class, irq, e, true); + } +} + +static void enirq_update(unsigned long ip) +{ + struct dept_task *dt = dept_task(); + unsigned long irqf; + unsigned long prev; + int irq; + + prev = dt->eff_enirqf; + irqf = cur_enirqf(); + dt->eff_enirqf = irqf; + + /* + * Do enirq_transition() only on an OFF -> ON transition. + */ + for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + if (prev & (1UL << irq)) + continue; + + dt->enirq_ip[irq] = ip; + enirq_transition(irq); + } +} + +/* + * Ensure it has been called on ON/OFF transition. + */ +static void dept_enirq_transition(unsigned long ip) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + /* + * IRQ ON/OFF transition might happen while Dept is working. + * We cannot handle recursive entrance. Just ingnore it. + * Only transitions outside of Dept will be considered. + */ + if (dt->recursive) + return; + + flags = dept_enter(); + + enirq_update(ip); + + dept_exit(flags); +} + +void dept_softirqs_on_ip(unsigned long ip) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->softirqs_enabled = true; + dept_enirq_transition(ip); +} + +void dept_hardirqs_on(void) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->hardirqs_enabled = true; + dept_enirq_transition(_RET_IP_); +} +EXPORT_SYMBOL_GPL(dept_hardirqs_on); + +void dept_hardirqs_on_ip(unsigned long ip) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->hardirqs_enabled = true; + dept_enirq_transition(ip); +} +EXPORT_SYMBOL_GPL(dept_hardirqs_on_ip); + +void dept_softirqs_off_ip(unsigned long ip) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->softirqs_enabled = false; + dept_enirq_transition(ip); +} + +void dept_hardirqs_off(void) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->hardirqs_enabled = false; + dept_enirq_transition(_RET_IP_); +} +EXPORT_SYMBOL_GPL(dept_hardirqs_off); + +void dept_hardirqs_off_ip(unsigned long ip) +{ + /* + * Assumes that it's called with IRQ disabled so that accessing + * current's fields is not racy. + */ + dept_task()->hardirqs_enabled = false; + dept_enirq_transition(ip); +} +EXPORT_SYMBOL_GPL(dept_hardirqs_off_ip); + +/* + * Ensure it's the outmost softirq context. + */ +void dept_softirq_enter(void) +{ + struct dept_task *dt = dept_task(); + + dt->irq_id[DEPT_SIRQ] += 1UL << DEPT_IRQS_NR; +} + +/* + * Ensure it's the outmost hardirq context. + */ +void dept_hardirq_enter(void) +{ + struct dept_task *dt = dept_task(); + + dt->irq_id[DEPT_HIRQ] += 1UL << DEPT_IRQS_NR; +} + +void dept_sched_enter(void) +{ + dept_task()->in_sched = true; +} + +void dept_sched_exit(void) +{ + dept_task()->in_sched = false; +} + +/* + * Exposed APIs + * ===================================================================== + */ + +static inline void clean_classes_cache(struct dept_key *k) +{ + int i; + + for (i = 0; i < DEPT_MAX_SUBCLASSES_CACHE; i++) { + if (!READ_ONCE(k->classes[i])) + continue; + + WRITE_ONCE(k->classes[i], NULL); + } +} + +void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u, + const char *n) +{ + unsigned long flags; + + if (unlikely(!dept_working())) { + m->nocheck = true; + return; + } + + if (DEPT_WARN_ON(sub_u < 0)) { + m->nocheck = true; + return; + } + + if (DEPT_WARN_ON(sub_u >= DEPT_MAX_SUBCLASSES_USR)) { + m->nocheck = true; + return; + } + + /* + * Allow recursive entrance. + */ + flags = dept_enter_recursive(); + + clean_classes_cache(&m->map_key); + + m->keys = k; + m->sub_u = sub_u; + m->name = n; + m->wgen = 0U; + m->nocheck = !valid_key(k); + + dept_exit_recursive(flags); +} +EXPORT_SYMBOL_GPL(dept_map_init); + +void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, + const char *n) +{ + unsigned long flags; + + if (unlikely(!dept_working())) { + m->nocheck = true; + return; + } + + /* + * Allow recursive entrance. + */ + flags = dept_enter_recursive(); + + if (k) { + clean_classes_cache(&m->map_key); + m->keys = k; + m->nocheck = !valid_key(k); + } + + if (sub_u >= 0 && sub_u < DEPT_MAX_SUBCLASSES_USR) + m->sub_u = sub_u; + + if (n) + m->name = n; + + m->wgen = 0U; + + dept_exit_recursive(flags); +} +EXPORT_SYMBOL_GPL(dept_map_reinit); + +void dept_map_copy(struct dept_map *to, struct dept_map *from) +{ + if (unlikely(!dept_working())) { + to->nocheck = true; + return; + } + + *to = *from; + + /* + * XXX: 'to' might be in a stack or something. Using the address + * in a stack segment as a key is meaningless. Just ignore the + * case for now. + */ + if (!to->keys) { + to->nocheck = true; + return; + } + + /* + * Since the class cache can be modified concurrently we could + * observe half pointers (64bit arch using 32bit copy insns). + * Therefore clear the caches and take the performance hit. + * + * XXX: Doesn't work well with lockdep_set_class_and_subclass() + * since that relies on cache abuse. + */ + clean_classes_cache(&to->map_key); +} + +static LIST_HEAD(classes); + +static inline bool within(const void *addr, void *start, unsigned long size) +{ + return addr >= start && addr < start + size; +} + +void dept_free_range(void *start, unsigned int sz) +{ + struct dept_task *dt = dept_task(); + struct dept_class *c, *n; + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) { + DEPT_STOP("Failed to successfully free Dept objects.\n"); + return; + } + + flags = dept_enter(); + + /* + * dept_free_range() should not fail. + * + * FIXME: Should be fixed if dept_free_range() causes deadlock + * with dept_lock(). + */ + while (unlikely(!dept_lock())) + cpu_relax(); + + list_for_each_entry_safe(c, n, &classes, all_node) { + if (!within((void *)c->key, start, sz) && + !within(c->name, start, sz)) + continue; + + hash_del_class(c); + disconnect_class(c); + list_del(&c->all_node); + invalidate_class(c); + + /* + * Actual deletion will happen on the rcu callback + * that has been added in disconnect_class(). + */ + del_class(c); + } + dept_unlock(); + dept_exit(flags); + + /* + * Wait until even lockless hash_lookup_class() for the class + * returns NULL. + */ + might_sleep(); + synchronize_rcu(); +} + +static inline int sub_id(struct dept_map *m, int e) +{ + return (m ? m->sub_u : 0) + e * DEPT_MAX_SUBCLASSES_USR; +} + +static struct dept_class *check_new_class(struct dept_key *local, + struct dept_key *k, int sub_id, + const char *n, bool sched_map) +{ + struct dept_class *c = NULL; + + if (DEPT_WARN_ON(sub_id >= DEPT_MAX_SUBCLASSES)) + return NULL; + + if (DEPT_WARN_ON(!k)) + return NULL; + + /* + * XXX: Assume that users prevent the map from using if any of + * the cached keys has been invalidated. If not, the cache, + * local->classes should not be used because it would be racy + * with class deletion. + */ + if (local && sub_id < DEPT_MAX_SUBCLASSES_CACHE) + c = READ_ONCE(local->classes[sub_id]); + + if (c) + return c; + + c = lookup_class((unsigned long)k->base + sub_id); + if (c) + goto caching; + + if (unlikely(!dept_lock())) + return NULL; + + c = lookup_class((unsigned long)k->base + sub_id); + if (unlikely(c)) + goto unlock; + + c = new_class(); + if (unlikely(!c)) + goto unlock; + + c->name = n; + c->sched_map = sched_map; + c->sub_id = sub_id; + c->key = (unsigned long)(k->base + sub_id); + hash_add_class(c); + list_add(&c->all_node, &classes); +unlock: + dept_unlock(); +caching: + if (local && sub_id < DEPT_MAX_SUBCLASSES_CACHE) + WRITE_ONCE(local->classes[sub_id], c); + + return c; +} + +/* + * Called between dept_enter() and dept_exit(). + */ +static void __dept_wait(struct dept_map *m, unsigned long w_f, + unsigned long ip, const char *w_fn, int sub_l, + bool sched_sleep, bool sched_map) +{ + int e; + + /* + * Be as conservative as possible. In case of mulitple waits for + * a single dept_map, we are going to keep only the last wait's + * wgen for simplicity - keeping all wgens seems overengineering. + * + * Of course, it might cause missing some dependencies that + * would rarely, probabily never, happen but it helps avoid + * false positive report. + */ + for_each_set_bit(e, &w_f, DEPT_MAX_SUBCLASSES_EVT) { + struct dept_class *c; + struct dept_key *k; + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, + sub_id(m, e), m->name, sched_map); + if (!c) + continue; + + add_wait(c, ip, w_fn, sub_l, sched_sleep); + } +} + +/* + * Called between dept_enter() and dept_exit(). + */ +static void __dept_event(struct dept_map *m, unsigned long e_f, + unsigned long ip, const char *e_fn, + bool sched_map) +{ + struct dept_class *c; + struct dept_key *k; + int e; + + e = find_first_bit(&e_f, DEPT_MAX_SUBCLASSES_EVT); + + if (DEPT_WARN_ON(e >= DEPT_MAX_SUBCLASSES_EVT)) + goto exit; + + /* + * An event is an event. If the caller passed more than single + * event, then warn it and handle the event corresponding to + * the first bit anyway. + */ + DEPT_WARN_ON(1UL << e != e_f); + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, sub_id(m, e), m->name, sched_map); + + if (c && add_ecxt(m, c, 0UL, NULL, e_fn, 0)) { + do_event(m, c, READ_ONCE(m->wgen), ip); + pop_ecxt(m, c); + } +exit: + /* + * Keep the map diabled until the next sleep. + */ + WRITE_ONCE(m->wgen, 0U); +} + +void dept_wait(struct dept_map *m, unsigned long w_f, + unsigned long ip, const char *w_fn, int sub_l) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) + return; + + if (m->nocheck) + return; + + flags = dept_enter(); + + __dept_wait(m, w_f, ip, w_fn, sub_l, false, false); + + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_wait); + +void dept_stage_wait(struct dept_map *m, struct dept_key *k, + unsigned long ip, const char *w_fn) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + if (m && m->nocheck) + return; + + /* + * Either m or k should be passed. Which means Dept relies on + * either its own map or the caller's position in the code when + * determining its class. + */ + if (DEPT_WARN_ON(!m && !k)) + return; + + /* + * Allow recursive entrance. + */ + flags = dept_enter_recursive(); + + arch_spin_lock(&stage_spin); + + /* + * Ensure the outmost dept_stage_wait() works. + */ + if (dt->stage_m.keys) + goto unlock; + + if (m) { + dt->stage_m = *m; + + /* + * Ensure dt->stage_m.keys != NULL and it works with the + * map's map_key, not stage_m's one when ->keys == NULL. + */ + if (!m->keys) + dt->stage_m.keys = &m->map_key; + } else { + dt->stage_m.name = w_fn; + dt->stage_sched_map = true; + } + + /* + * dept_map_reinit() includes WRITE_ONCE(->wgen, 0U) that + * effectively disables the map just in case real sleep won't + * happen. dept_request_event_wait_commit() will enable it. + */ + dept_map_reinit(&dt->stage_m, k, -1, NULL); + + dt->stage_w_fn = w_fn; + dt->stage_ip = ip; +unlock: + arch_spin_unlock(&stage_spin); + + dept_exit_recursive(flags); +} +EXPORT_SYMBOL_GPL(dept_stage_wait); + +void dept_clean_stage(void) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + /* + * Allow recursive entrance. + */ + flags = dept_enter_recursive(); + + arch_spin_lock(&stage_spin); + memset(&dt->stage_m, 0x0, sizeof(struct dept_map)); + dt->stage_sched_map = false; + dt->stage_w_fn = NULL; + dt->stage_ip = 0UL; + arch_spin_unlock(&stage_spin); + + dept_exit_recursive(flags); +} +EXPORT_SYMBOL_GPL(dept_clean_stage); + +/* + * Always called from __schedule(). + */ +void dept_request_event_wait_commit(void) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + unsigned int wg; + unsigned long ip; + const char *w_fn; + bool sched_map; + + if (unlikely(!dept_working())) + return; + + /* + * It's impossible that __schedule() is called while Dept is + * working that already disabled IRQ at the entrance. + */ + if (DEPT_WARN_ON(dt->recursive)) + return; + + flags = dept_enter(); + + /* + * Checks if current has staged a wait. + */ + if (!dt->stage_m.keys) + goto exit; + + w_fn = dt->stage_w_fn; + ip = dt->stage_ip; + sched_map = dt->stage_sched_map; + + /* + * Avoid zero wgen. + */ + wg = atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen); + WRITE_ONCE(dt->stage_m.wgen, wg); + + __dept_wait(&dt->stage_m, 1UL, ip, w_fn, 0, true, sched_map); +exit: + dept_exit(flags); +} + +/* + * Always called from try_to_wake_up(). + */ +void dept_stage_event(struct task_struct *t, unsigned long ip) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + struct dept_map m; + bool sched_map; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) + return; + + flags = dept_enter(); + + arch_spin_lock(&stage_spin); + m = t->dept_task.stage_m; + sched_map = t->dept_task.stage_sched_map; + arch_spin_unlock(&stage_spin); + + /* + * ->stage_m.keys should not be NULL if it's in use. Should + * make sure that it's not NULL when staging a valid map. + */ + if (!m.keys) + goto exit; + + __dept_event(&m, 1UL, ip, "try_to_wake_up", sched_map); +exit: + dept_exit(flags); +} + +/* + * Modifies the latest ecxt corresponding to m and e_f. + */ +void dept_map_ecxt_modify(struct dept_map *m, unsigned long e_f, + struct dept_key *new_k, unsigned long new_e_f, + unsigned long new_ip, const char *new_c_fn, + const char *new_e_fn, int new_sub_l) +{ + struct dept_task *dt = dept_task(); + struct dept_ecxt_held *eh; + struct dept_class *c; + struct dept_key *k; + unsigned long flags; + int pos = -1; + int new_e; + int e; + + if (unlikely(!dept_working())) + return; + + /* + * XXX: Couldn't handle re-enterance cases. Ingore it for now. + */ + if (dt->recursive) + return; + + /* + * Should go ahead no matter whether ->nocheck == true or not + * because ->nocheck value can be changed within the ecxt area + * delimitated by dept_ecxt_enter() and dept_ecxt_exit(). + */ + + flags = dept_enter(); + + for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) { + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, + sub_id(m, e), m->name, false); + if (!c) + continue; + + /* + * When it found an ecxt for any event in e_f, done. + */ + pos = find_ecxt_pos(m, c, true); + if (pos != -1) + break; + } + + if (unlikely(pos == -1)) + goto exit; + + eh = dt->ecxt_held + pos; + new_sub_l = new_sub_l >= 0 ? new_sub_l : eh->sub_l; + + new_e = find_first_bit(&new_e_f, DEPT_MAX_SUBCLASSES_EVT); + + if (new_e < DEPT_MAX_SUBCLASSES_EVT) + /* + * Let it work with the first bit anyway. + */ + DEPT_WARN_ON(1UL << new_e != new_e_f); + else + new_e = e; + + pop_ecxt(m, c); + + /* + * Apply the key to the map. + */ + if (new_k) + dept_map_reinit(m, new_k, -1, NULL); + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, sub_id(m, new_e), m->name, false); + + if (c && add_ecxt(m, c, new_ip, new_c_fn, new_e_fn, new_sub_l)) + goto exit; + + /* + * Successfully pop_ecxt()ed but failed to add_ecxt(). + */ + dt->missing_ecxt++; +exit: + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_map_ecxt_modify); + +void dept_ecxt_enter(struct dept_map *m, unsigned long e_f, unsigned long ip, + const char *c_fn, const char *e_fn, int sub_l) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + struct dept_class *c; + struct dept_key *k; + int e; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) { + dt->missing_ecxt++; + return; + } + + /* + * Should go ahead no matter whether ->nocheck == true or not + * because ->nocheck value can be changed within the ecxt area + * delimitated by dept_ecxt_enter() and dept_ecxt_exit(). + */ + + flags = dept_enter(); + + e = find_first_bit(&e_f, DEPT_MAX_SUBCLASSES_EVT); + + if (e >= DEPT_MAX_SUBCLASSES_EVT) + goto missing_ecxt; + + /* + * An event is an event. If the caller passed more than single + * event, then warn it and handle the event corresponding to + * the first bit anyway. + */ + DEPT_WARN_ON(1UL << e != e_f); + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, sub_id(m, e), m->name, false); + + if (c && add_ecxt(m, c, ip, c_fn, e_fn, sub_l)) + goto exit; +missing_ecxt: + dt->missing_ecxt++; +exit: + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_ecxt_enter); + +bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + bool ret = false; + int e; + + if (unlikely(!dept_working())) + return false; + + if (dt->recursive) + return false; + + flags = dept_enter(); + + for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) { + struct dept_class *c; + struct dept_key *k; + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, + sub_id(m, e), m->name, false); + if (!c) + continue; + + if (find_ecxt_pos(m, c, true) != -1) { + ret = true; + break; + } + } + + dept_exit(flags); + + return ret; +} +EXPORT_SYMBOL_GPL(dept_ecxt_holding); + +void dept_request_event(struct dept_map *m) +{ + unsigned long flags; + unsigned int wg; + + if (unlikely(!dept_working())) + return; + + if (m->nocheck) + return; + + /* + * Allow recursive entrance. + */ + flags = dept_enter_recursive(); + + /* + * Avoid zero wgen. + */ + wg = atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen); + WRITE_ONCE(m->wgen, wg); + + dept_exit_recursive(flags); +} +EXPORT_SYMBOL_GPL(dept_request_event); + +void dept_event(struct dept_map *m, unsigned long e_f, + unsigned long ip, const char *e_fn) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) { + /* + * Dept won't work with this even though an event + * context has been asked. Don't make it confused at + * handling the event. Disable it until the next. + */ + WRITE_ONCE(m->wgen, 0U); + return; + } + + if (m->nocheck) + return; + + flags = dept_enter(); + + __dept_event(m, e_f, ip, e_fn, false); + + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_event); + +void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, + unsigned long ip) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + int e; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) { + dt->missing_ecxt--; + return; + } + + /* + * Should go ahead no matter whether ->nocheck == true or not + * because ->nocheck value can be changed within the ecxt area + * delimitated by dept_ecxt_enter() and dept_ecxt_exit(). + */ + + flags = dept_enter(); + + for_each_set_bit(e, &e_f, DEPT_MAX_SUBCLASSES_EVT) { + struct dept_class *c; + struct dept_key *k; + + k = m->keys ?: &m->map_key; + c = check_new_class(&m->map_key, k, + sub_id(m, e), m->name, false); + if (!c) + continue; + + /* + * When it found an ecxt for any event in e_f, done. + */ + if (pop_ecxt(m, c)) + goto exit; + } + + dt->missing_ecxt--; +exit: + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_ecxt_exit); + +void dept_task_exit(struct task_struct *t) +{ + struct dept_task *dt = &t->dept_task; + int i; + + if (unlikely(!dept_working())) + return; + + raw_local_irq_disable(); + + if (dt->stack) + put_stack(dt->stack); + + for (i = 0; i < dt->ecxt_held_pos; i++) { + if (dt->ecxt_held[i].class) + put_class(dt->ecxt_held[i].class); + if (dt->ecxt_held[i].ecxt) + put_ecxt(dt->ecxt_held[i].ecxt); + } + + for (i = 0; i < DEPT_MAX_WAIT_HIST; i++) + if (dt->wait_hist[i].wait) + put_wait(dt->wait_hist[i].wait); + + dt->task_exit = true; + dept_off(); + + raw_local_irq_enable(); +} + +void dept_task_init(struct task_struct *t) +{ + memset(&t->dept_task, 0x0, sizeof(struct dept_task)); +} + +void dept_key_init(struct dept_key *k) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + int sub_id; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive) { + DEPT_STOP("Key initialization fails.\n"); + return; + } + + flags = dept_enter(); + + clean_classes_cache(k); + + /* + * dept_key_init() should not fail. + * + * FIXME: Should be fixed if dept_key_init() causes deadlock + * with dept_lock(). + */ + while (unlikely(!dept_lock())) + cpu_relax(); + + for (sub_id = 0; sub_id < DEPT_MAX_SUBCLASSES; sub_id++) { + struct dept_class *c; + + c = lookup_class((unsigned long)k->base + sub_id); + if (!c) + continue; + + DEPT_STOP("The class(%s/%d) has not been removed.\n", + c->name, sub_id); + break; + } + + dept_unlock(); + dept_exit(flags); +} +EXPORT_SYMBOL_GPL(dept_key_init); + +void dept_key_destroy(struct dept_key *k) +{ + struct dept_task *dt = dept_task(); + unsigned long flags; + int sub_id; + + if (unlikely(!dept_working())) + return; + + if (dt->recursive == 1 && dt->task_exit) { + /* + * Need to allow to go ahead in this case where + * ->recursive has been set to 1 by dept_off() in + * dept_task_exit() and ->task_exit has been set to + * true in dept_task_exit(). + */ + } else if (dt->recursive) { + DEPT_STOP("Key destroying fails.\n"); + return; + } + + flags = dept_enter(); + + /* + * dept_key_destroy() should not fail. + * + * FIXME: Should be fixed if dept_key_destroy() causes deadlock + * with dept_lock(). + */ + while (unlikely(!dept_lock())) + cpu_relax(); + + for (sub_id = 0; sub_id < DEPT_MAX_SUBCLASSES; sub_id++) { + struct dept_class *c; + + c = lookup_class((unsigned long)k->base + sub_id); + if (!c) + continue; + + hash_del_class(c); + disconnect_class(c); + list_del(&c->all_node); + invalidate_class(c); + + /* + * Actual deletion will happen on the rcu callback + * that has been added in disconnect_class(). + */ + del_class(c); + } + + dept_unlock(); + dept_exit(flags); + + /* + * Wait until even lockless hash_lookup_class() for the class + * returns NULL. + */ + might_sleep(); + synchronize_rcu(); +} +EXPORT_SYMBOL_GPL(dept_key_destroy); + +static void move_llist(struct llist_head *to, struct llist_head *from) +{ + struct llist_node *first = llist_del_all(from); + struct llist_node *last; + + if (!first) + return; + + for (last = first; last->next; last = last->next); + llist_add_batch(first, last, to); +} + +static void migrate_per_cpu_pool(void) +{ + const int boot_cpu = 0; + int i; + + /* + * The boot CPU has been using the temperal local pool so far. + * From now on that per_cpu areas have been ready, use the + * per_cpu local pool instead. + */ + DEPT_WARN_ON(smp_processor_id() != boot_cpu); + for (i = 0; i < OBJECT_NR; i++) { + struct llist_head *from; + struct llist_head *to; + + from = &pool[i].boot_pool; + to = per_cpu_ptr(pool[i].lpool, boot_cpu); + move_llist(to, from); + } +} + +#define B2KB(B) ((B) / 1024) + +/* + * Should be called after setup_per_cpu_areas() and before no non-boot + * CPUs have been on. + */ +void __init dept_init(void) +{ + size_t mem_total = 0; + + local_irq_disable(); + dept_per_cpu_ready = 1; + migrate_per_cpu_pool(); + local_irq_enable(); + +#define HASH(id, bits) BUILD_BUG_ON(1 << (bits) <= 0); + #include "dept_hash.h" +#undef HASH +#define OBJECT(id, nr) mem_total += sizeof(struct dept_##id) * nr; + #include "dept_object.h" +#undef OBJECT +#define HASH(id, bits) mem_total += sizeof(struct hlist_head) * (1 << (bits)); + #include "dept_hash.h" +#undef HASH + + pr_info("DEPendency Tracker: Copyright (c) 2020 LG Electronics, Inc., Byungchul Park\n"); + pr_info("... DEPT_MAX_STACK_ENTRY: %d\n", DEPT_MAX_STACK_ENTRY); + pr_info("... DEPT_MAX_WAIT_HIST : %d\n", DEPT_MAX_WAIT_HIST); + pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD); + pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES); +#define OBJECT(id, nr) \ + pr_info("... memory used by %s: %zu KB\n", \ + #id, B2KB(sizeof(struct dept_##id) * nr)); + #include "dept_object.h" +#undef OBJECT +#define HASH(id, bits) \ + pr_info("... hash list head used by %s: %zu KB\n", \ + #id, B2KB(sizeof(struct hlist_head) * (1 << (bits)))); + #include "dept_hash.h" +#undef HASH + pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total)); + pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task)); +} diff --git a/kernel/dependency/dept_hash.h b/kernel/dependency/dept_hash.h new file mode 100644 index 000000000000..fd85aab1fdfb --- /dev/null +++ b/kernel/dependency/dept_hash.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * HASH(id, bits) + * + * id : Id for the object of struct dept_##id. + * bits: 1UL << bits is the hash table size. + */ + +HASH(dep, 12) +HASH(class, 12) diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h new file mode 100644 index 000000000000..0b7eb16fe9fb --- /dev/null +++ b/kernel/dependency/dept_object.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * OBJECT(id, nr) + * + * id: Id for the object of struct dept_##id. + * nr: # of the object that should be kept in the pool. + */ + +OBJECT(dep, 1024 * 8) +OBJECT(class, 1024 * 8) +OBJECT(stack, 1024 * 32) +OBJECT(ecxt, 1024 * 16) +OBJECT(wait, 1024 * 32) diff --git a/kernel/exit.c b/kernel/exit.c index edb50b4c9972..8d3850eded25 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -923,6 +923,7 @@ void __noreturn do_exit(long code) exit_tasks_rcu_finish(); lockdep_free_task(tsk); + dept_task_exit(tsk); do_task_dead(); } diff --git a/kernel/fork.c b/kernel/fork.c index 41c964104b58..20fa77c47db2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -99,6 +99,7 @@ #include #include #include +#include #include #include @@ -2460,6 +2461,7 @@ __latent_entropy struct task_struct *copy_process( #ifdef CONFIG_LOCKDEP lockdep_init_task(p); #endif + dept_task_init(p); #ifdef CONFIG_DEBUG_MUTEXES p->blocked_on = NULL; /* not blocked yet */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 4e2cf784cf8c..aa4bd4dcc9aa 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1235,12 +1235,14 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); + dept_free_range(mod_mem->base, mod_mem->size); if (mod_mem->size) module_memory_free(mod_mem->base, type); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); + dept_free_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); } @@ -3019,6 +3021,8 @@ static int load_module(struct load_info *info, const char __user *uargs, for_class_mod_mem_type(type, core_data) { lockdep_free_key_range(mod->mem[type].base, mod->mem[type].size); + dept_free_range(mod->mem[type].base, + mod->mem[type].size); } module_deallocate(mod, info); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a68d1276bab0..243f3de42721 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -64,6 +64,7 @@ #include #include #include +#include #ifdef CONFIG_PREEMPT_DYNAMIC # ifdef CONFIG_GENERIC_ENTRY @@ -4162,6 +4163,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) int cpu, success = 0; preempt_disable(); + dept_stage_event(p, _RET_IP_); if (p == current) { /* * We're waking current, this means 'p->on_rq' and 'task_cpu(p) @@ -6560,6 +6562,12 @@ static void __sched notrace __schedule(unsigned int sched_mode) rq = cpu_rq(cpu); prev = rq->curr; + prev_state = READ_ONCE(prev->__state); + if (sched_mode != SM_PREEMPT && prev_state & TASK_NORMAL) + dept_request_event_wait_commit(); + + dept_sched_enter(); + schedule_debug(prev, !!sched_mode); if (sched_feat(HRTICK) || sched_feat(HRTICK_DL)) @@ -6674,6 +6682,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) __balance_callbacks(rq); raw_spin_rq_unlock_irq(rq); } + dept_sched_exit(); } void __noreturn do_task_dead(void) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803..aa62caa4dc14 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1207,6 +1207,33 @@ config DEBUG_PREEMPT menu "Lock Debugging (spinlocks, mutexes, etc...)" +config DEPT + bool "Dependency tracking (EXPERIMENTAL)" + depends on DEBUG_KERNEL && LOCK_DEBUGGING_SUPPORT + select DEBUG_SPINLOCK + select DEBUG_MUTEXES + select DEBUG_RT_MUTEXES if RT_MUTEXES + select DEBUG_RWSEMS + select DEBUG_WW_MUTEX_SLOWPATH + select DEBUG_LOCK_ALLOC + select TRACE_IRQFLAGS + select STACKTRACE + select FRAME_POINTER if !MIPS && !PPC && !ARM && !S390 && !MICROBLAZE && !ARC && !X86 + select KALLSYMS + select KALLSYMS_ALL + select PROVE_LOCKING + default n + help + Check dependencies between wait and event and report it if + deadlock possibility has been detected. Multiple reports are + allowed if there are more than a single problem. + + This feature is considered EXPERIMENTAL that might produce + false positive reports because new dependencies start to be + tracked, that have never been tracked before. It's worth + noting, to mitigate the impact by the false positives, multi + reporting has been supported. + config LOCK_DEBUGGING_SUPPORT bool depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index 8d24279fad05..cd89138d62ba 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -1398,6 +1398,8 @@ static void reset_locks(void) local_irq_disable(); lockdep_free_key_range(&ww_lockdep.acquire_key, 1); lockdep_free_key_range(&ww_lockdep.mutex_key, 1); + dept_free_range(&ww_lockdep.acquire_key, 1); + dept_free_range(&ww_lockdep.mutex_key, 1); I1(A); I1(B); I1(C); I1(D); I1(X1); I1(X2); I1(Y1); I1(Y2); I1(Z1); I1(Z2); From patchwork Mon Jul 3 09:47:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 667C0EB64DD for ; Mon, 3 Jul 2023 09:50:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231277AbjGCJtv (ORCPT ); Mon, 3 Jul 2023 05:49:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbjGCJtr (ORCPT ); Mon, 3 Jul 2023 05:49:47 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 650991A1; Mon, 3 Jul 2023 02:49:40 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-08-64a299b2432f From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 03/25] dept: Add single event dependency tracker APIs Date: Mon, 3 Jul 2023 18:47:30 +0900 Message-Id: <20230703094752.79269-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTZxjG/c7lO22146Qz4YhRSBMlYfECXvLGLMs0Ub7EkZCZLJvLsjX2 KM3KxYLlFgMOUFYBUYOlQEYBUxoolx38Q0Wwwwmi4SJWKYpVCF64T7Z2VNhci/OfN788z/s8 fz0yWnWdDZPpktJEQ5JGr8YKRjG7pmaLZKnVbu9+uQHOF20H71+FDFS1ODAMNjcicFw5RcHk 7VgY9s0gWOoboMFcNoigZuwpDVe6PQg67D9heDDxEbi88xh6y85iyKtrwXB/epmC0UsXKGiU 4uBeaS0FTv8rBsyTGCrNeVTgvKbAb2vgwJa7CcbtFRwsj0VDr+cRCx2PPwHLL6MYbnT0MtB9 dZyCB9erMHgc71i4132HgcHzxSw0zdVimPbZaLB55zkYclopaM0PFJ3+818WeoqdFJy+/CsF rpF2BJ2FzymQHI8w3PLOUNAmldHwtv42gvGSWQ4KivwcVJ4qQXC24BIDA//0sJA/uguWFqvw 53vIrZl5muS3pZMOn5Uhd2sFcq3iKUfyOx9zxCqdIG32KFJ3Y5IiNQtelkgNP2MiLVzgiGnW RZG5/n6O3ClfYsiEy0zFrz+s+FQr6nVG0bDtsx8UCV2lFWzKcGiGzxyei1wqE5LLBH6nUF3Y zn3gvrE3bJAxHym43X46yGv5CKGt+GVAV8ho/sxqwf5HHw4aH/PfCFP111YCDL9JaHcsrASU /C7B6jSz70vDhcZW54ou53cLLxZLUJBVgZ9RiwcHSwX+jFx4eG7k/8A64Te7mylFSita1YBU uiRjokan37k1ITNJl7H1SHKihAKLsp1c/vYqWhg81IV4GVKvUbqza7QqVmNMzUzsQoKMVq9V 5o1Va1VKrSYzSzQkf284oRdTu9B6GaMOVcb40rUq/pgmTfxRFFNEwweXksnDctHe/dnGAxmR BZb4Z++ytdVf4HMxI6uPD9v64vdugO/SNn9l3JHeXLR/Y5h0sTLZtDElZ0qIywrJuhneGeI3 HV11bKilOK5naqRu6g35OqfOMz30ImLPycXIg6asOfc+9ORh7N+08mJU7Mxi+cRyyPywvrM/ 5+aXEQd/l1sObW4qf4XUTGqCJjqKNqRq/gOTtbUrTQMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTcRjG+5/L/xxXi9OSPFhRjCIoMoOMtwsRRXToRhAURWWjHXSl07Yy Z2nmrTInGtjSls1pa+rMOvahm7Im3grNckyLJSl2sSzL2mppl2n05eXH87zP8+lhSYWZDmc1 2qOiTquKU2IZJdu2KnOxVGxVR5orKCjMiwTft7MUmGsdGDpvVCNw3D5NwGDTRuj2DyEYbX9C gqmoE0FZ30sSbjf3Iqi3Z2DoGpgKbt8whrai8xgyy2sxPP0wRoD34gUCqqWt8LjASoAz8JYC 0yCGy6ZMInjeERCwVTFgS58P/fYSBsb6lkJbr4eGxittNNS/WATFpV4MD+rbKGi+009A1z0z hl7HHxoeN7dS0FlopKHmkxXDB7+NBJtvmIFnTgsBN7OCbTlff9PQYnQSkFNxiwD38/sIGs6+ IkByeDA0+oYIqJOKSPh5vQlBf/5HBrLzAgxcPp2P4Hz2RQqe/GqhIcsbBaM/zHjtKqFxaJgU suqOC/V+CyU8svLC3ZKXjJDV8IIRLNIxoc6+UCh/MEgIZSM+WpCqzmFBGrnACLkf3YTwqaOD EVovjVLCgNtEbJ+1R7ZaLcZpkkTdkjUHZLGughI6sTss2W+ak47cilwUwvLcMr697ws9zphb wPf0BMhxDuXm8nXGN0FdxpLcmcm8/XM7Hjemc7v599fvTgQobj5/3zEyEZBzUbzFaaL/lc7h q286J/QQbjn/+kc+GmdF8Mdb3IsLkMyCJlWhUI02KV6liYuK0B+ONWg1yREHE+IlFNyMLXWs 8A761rXRhTgWKafIe06UqRW0KklviHchniWVofLMvqtqhVytMqSIuoRo3bE4Ue9CM1lKGSbf tEs8oOBiVEfFw6KYKOr+uwQbEp6OtOs2pc6UYvY3dpyy7FAYajwb9m0Ob3EebD40Y7pkr658 uPxadPLULWuM642O2qaMPMM0c+S2xOzPO+8Fdt+w4oQGz8qw3JbsJWlYP5AW873ii2v9m5Pl EQ6Pl927qDhxdcHP1IeV6zaXprBpfqXlROvegdkrlRk5866tGExJr+w+oqT0saqlC0mdXvUX eQqn0i8DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Wrapped the base APIs for easier annotation on wait and event. Start with supporting waiters on each single event. More general support for multiple events is a future work. Do more when the need arises. How to annotate (the simplest way): 1. Initaialize a map for the interesting wait. /* * Recommand to place along with the wait instance. */ struct dept_map my_wait; /* * Recommand to place in the initialization code. */ sdt_map_init(&my_wait); 2. Place the following at the wait code. sdt_wait(&my_wait); 3. Place the following at the event code. sdt_event(&my_wait); That's it! Signed-off-by: Byungchul Park --- include/linux/dept_sdt.h | 62 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 include/linux/dept_sdt.h diff --git a/include/linux/dept_sdt.h b/include/linux/dept_sdt.h new file mode 100644 index 000000000000..12a793b90c7e --- /dev/null +++ b/include/linux/dept_sdt.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Single-event Dependency Tracker + * + * Started by Byungchul Park : + * + * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park + */ + +#ifndef __LINUX_DEPT_SDT_H +#define __LINUX_DEPT_SDT_H + +#include +#include + +#ifdef CONFIG_DEPT +#define sdt_map_init(m) \ + do { \ + static struct dept_key __key; \ + dept_map_init(m, &__key, 0, #m); \ + } while (0) + +#define sdt_map_init_key(m, k) dept_map_init(m, k, 0, #m) + +#define sdt_wait(m) \ + do { \ + dept_request_event(m); \ + dept_wait(m, 1UL, _THIS_IP_, __func__, 0); \ + } while (0) + +/* + * sdt_might_sleep() and its family will be committed in __schedule() + * when it actually gets to __schedule(). Both dept_request_event() and + * dept_wait() will be performed on the commit. + */ + +/* + * Use the code location as the class key if an explicit map is not used. + */ +#define sdt_might_sleep_start(m) \ + do { \ + struct dept_map *__m = m; \ + static struct dept_key __key; \ + dept_stage_wait(__m, __m ? NULL : &__key, _THIS_IP_, __func__);\ + } while (0) + +#define sdt_might_sleep_end() dept_clean_stage() + +#define sdt_ecxt_enter(m) dept_ecxt_enter(m, 1UL, _THIS_IP_, "start", "event", 0) +#define sdt_event(m) dept_event(m, 1UL, _THIS_IP_, __func__) +#define sdt_ecxt_exit(m) dept_ecxt_exit(m, 1UL, _THIS_IP_) +#else /* !CONFIG_DEPT */ +#define sdt_map_init(m) do { } while (0) +#define sdt_map_init_key(m, k) do { (void)(k); } while (0) +#define sdt_wait(m) do { } while (0) +#define sdt_might_sleep_start(m) do { } while (0) +#define sdt_might_sleep_end() do { } while (0) +#define sdt_ecxt_enter(m) do { } while (0) +#define sdt_event(m) do { } while (0) +#define sdt_ecxt_exit(m) do { } while (0) +#endif +#endif /* __LINUX_DEPT_SDT_H */ From patchwork Mon Jul 3 09:47:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C61AEC04FE0 for ; Mon, 3 Jul 2023 09:49:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231244AbjGCJtt (ORCPT ); Mon, 3 Jul 2023 05:49:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229834AbjGCJtq (ORCPT ); Mon, 3 Jul 2023 05:49:46 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7E1E9BE; Mon, 3 Jul 2023 02:49:40 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-16-64a299b26989 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 04/25] dept: Add lock dependency tracker APIs Date: Mon, 3 Jul 2023 18:47:31 +0900 Message-Id: <20230703094752.79269-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTZxjG933n2krNsQM502SaxlswU9jQvC7TkJi5E2cTdYlGiXHVcwKN UEihINvIKlTFC0xNALlEobraAAqcYsYmdRUmiAToRqNgapWLG8RyCa4VBN0KxH/e/PI8T568 fzwsoW6klrF6Q7pkNOiSNLSSVI6GWT+RS6xitKV2FVw8Hw2Bf/NIKK+tocF9qxpBTcMJDCP3 v4LHQT+Cmc5uAooL3Qgq+58S0NDqQ+C059DQM7QYPIFxGtoLz9GQe62Whj9fzmLwFl3CUC1r oeOCFYNr+h8SikdoKCvOxaEzjGHaVsWAzbwaBuylDMz2x0C77xEFzifroeSKl4YmZzsJrY0D GHp+K6fBV/MfBR2tD0hwX8yn4OaYlYaXQRsBtsA4A3+5KjDUWUJFp169o6At34Xh1PV6DJ6+ Owju5j3HINc8oqEl4MfgkAsJeHPjPoKBglEGTp6fZqDsRAGCcyeLSOh+20aBxbsJZqbK6bjP hRb/OCFYHJmCM1hBCg+tvPBr6VNGsNx9wggVsklw2KOEa00jWKicDFCCXHWGFuTJS4xwdtSD hbGuLkZ4cHmGFIY8xXj38oPKL0QpSZ8hGTdu+1aZaAl9lzocefxnlxmZUbX6LFKwPBfL//TW R7znZ01F1BzT3Fq+t3d6Xg/nVvKO/L9DupIluNOLePtEJz1nfMjt4fN+nwuxLMmt5m3meVRx m3jZfGChcgVfXeear1Fwm/kXUwVojtWhiLfERy9kchV8wZtVC/wRf8/eS15Aqgr0QRVS6w0Z yTp9UuyGxCyD/viGoynJMgrtyZY9G9+IJt3fNCOORZowVe/3laKa0mWkZSU3I54lNOGq3P6r olol6rK+k4wph42mJCmtGS1nSU2k6tNgpqjmEnTp0jFJSpWM713MKpaZ0a4flzZ5Ww51ur/m I7SvV7b9MRvRc2Qo1bEtezBhvym5fnAsbnH4lntT2eSZy9E31/wQPNgX9MTvrduuZZfsnJBH sOn6fltsX9zwwBbtuqgjNtOOEnGreGVXype/1KbHrJ/Y2hD5WY7ztH93pD+svW9H+DEkLtln uK3d/nFOW5kiIV9DpiXqYqIIY5ruf0PRll1LAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0hTcRTH+93H715Xi9sSvFRQLCywMgcaByyLgroEvSiKpKjlLjmbFlua 9oDVlunSkcKcr8pHraFmdTWycjWUzBWa5nImZik9lCx7TTLtsRX9c/hwvl8+nD8OSypK6Fms NuWwqE9R65RYRsk2xpqWSEUVmqjM1mmQlxMF/m9ZFJReq8HQUVuNoKb+JAHDD9aBb2wEwUTb ExLstg4E5QMvSKhv6Ufgcp7C0PV6Onj9oxg8trMYTJXXMHS+nySgryCfgGppAzw+V0GAe/wd BfZhDCV2ExEYQwSMO6oYcBjDYdBZzMDkgAo8/d00NJ/30ODqXQRFF/owNLo8FLQ0DBLQdacU Q3/Nbxoet7RS0JGXS8PVjxUY3o85SHD4Rxl46i4j4Lo5YMv8+ouGh7luAjIv3SDA+/wugntZ rwiQaroxNPtHCKiTbCT8uPIAwaD1AwOnc8YZKDlpRXD2dAEFT34+pMHcFwMT30vxqliheWSU FMx1RwTXWBklPKrghdvFLxjBfK+XEcqkVKHOGSFUNg4TQvkXPy1IVdlYkL7kM4Llg5cQPra3 M0Jr4QQlvPbaic1z4mXLNaJOmybql8btlSWaA9cdGgpLv+w2IiOqVlhQCMtz0fzLxgI6yJhb yPf0jJNBDuXm8XW5bwN7GUtyZ6byzk9tOBjM5LbwWfeDJZaluHDeYfyLci6Gl4w7/ynn8tXX 3X81Idwy/s13KwqyIlDpK+rH55CsDE2pQqHalLRktVYXE2k4kJiRok2PTDiYLKHAyzhOTOY1 oG9d65oQxyLlNHnPsXKNglanGTKSmxDPkspQuWngokYh16gzjor6g3v0qTrR0IRms5QyTL5+ h7hXwe1XHxYPiOIhUf8/JdiQWUa01RW1ulS1afriwvqG2043O0f27Piz2p/ZldHeW/mOnCvS Mut8q27S2Jl+bP7Xys9Y77HdNaqidtGLfNsTTL7VhTtGh1QXLb64BFuBJWnrhuTupKTd7e1h kWERSeH7btp7J351rqmKt0xdMTaDu+/L7lmbsK8ornhlbXyDJnPbgi1KypCoVkWQeoP6D0Di ZREuAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Wrapped the base APIs for easier annotation on typical lock. Signed-off-by: Byungchul Park --- include/linux/dept_ldt.h | 77 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 include/linux/dept_ldt.h diff --git a/include/linux/dept_ldt.h b/include/linux/dept_ldt.h new file mode 100644 index 000000000000..062613e89fc3 --- /dev/null +++ b/include/linux/dept_ldt.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Lock Dependency Tracker + * + * Started by Byungchul Park : + * + * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park + */ + +#ifndef __LINUX_DEPT_LDT_H +#define __LINUX_DEPT_LDT_H + +#include + +#ifdef CONFIG_DEPT +#define LDT_EVT_L 1UL +#define LDT_EVT_R 2UL +#define LDT_EVT_W 1UL +#define LDT_EVT_RW (LDT_EVT_R | LDT_EVT_W) +#define LDT_EVT_ALL (LDT_EVT_L | LDT_EVT_RW) + +#define ldt_init(m, k, su, n) dept_map_init(m, k, su, n) +#define ldt_lock(m, sl, t, n, i) \ + do { \ + if (n) \ + dept_ecxt_enter_nokeep(m); \ + else if (t) \ + dept_ecxt_enter(m, LDT_EVT_L, i, "trylock", "unlock", sl);\ + else { \ + dept_wait(m, LDT_EVT_L, i, "lock", sl); \ + dept_ecxt_enter(m, LDT_EVT_L, i, "lock", "unlock", sl);\ + } \ + } while (0) + +#define ldt_rlock(m, sl, t, n, i, q) \ + do { \ + if (n) \ + dept_ecxt_enter_nokeep(m); \ + else if (t) \ + dept_ecxt_enter(m, LDT_EVT_R, i, "read_trylock", "read_unlock", sl);\ + else { \ + dept_wait(m, q ? LDT_EVT_RW : LDT_EVT_W, i, "read_lock", sl);\ + dept_ecxt_enter(m, LDT_EVT_R, i, "read_lock", "read_unlock", sl);\ + } \ + } while (0) + +#define ldt_wlock(m, sl, t, n, i) \ + do { \ + if (n) \ + dept_ecxt_enter_nokeep(m); \ + else if (t) \ + dept_ecxt_enter(m, LDT_EVT_W, i, "write_trylock", "write_unlock", sl);\ + else { \ + dept_wait(m, LDT_EVT_RW, i, "write_lock", sl); \ + dept_ecxt_enter(m, LDT_EVT_W, i, "write_lock", "write_unlock", sl);\ + } \ + } while (0) + +#define ldt_unlock(m, i) dept_ecxt_exit(m, LDT_EVT_ALL, i) + +#define ldt_downgrade(m, i) \ + do { \ + if (dept_ecxt_holding(m, LDT_EVT_W)) \ + dept_map_ecxt_modify(m, LDT_EVT_W, NULL, LDT_EVT_R, i, "downgrade", "read_unlock", -1);\ + } while (0) + +#define ldt_set_class(m, n, k, sl, i) dept_map_ecxt_modify(m, LDT_EVT_ALL, k, 0UL, i, "lock_set_class", "(any)unlock", sl) +#else /* !CONFIG_DEPT */ +#define ldt_init(m, k, su, n) do { (void)(k); } while (0) +#define ldt_lock(m, sl, t, n, i) do { } while (0) +#define ldt_rlock(m, sl, t, n, i, q) do { } while (0) +#define ldt_wlock(m, sl, t, n, i) do { } while (0) +#define ldt_unlock(m, i) do { } while (0) +#define ldt_downgrade(m, i) do { } while (0) +#define ldt_set_class(m, n, k, sl, i) do { } while (0) +#endif +#endif /* __LINUX_DEPT_LDT_H */ From patchwork Mon Jul 3 09:47:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86DC5EB64DD for ; Mon, 3 Jul 2023 09:49:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231228AbjGCJts (ORCPT ); Mon, 3 Jul 2023 05:49:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229816AbjGCJtq (ORCPT ); Mon, 3 Jul 2023 05:49:46 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 647B712C; Mon, 3 Jul 2023 02:49:40 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-26-64a299b2da09 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 05/25] dept: Tie to Lockdep and IRQ tracing Date: Mon, 3 Jul 2023 18:47:32 +0900 Message-Id: <20230703094752.79269-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0xTZxjHfd9zznsOlZpDNeEgWdQmXqbRgVHzZDHoJ3eMMermh8XLXGNP pI6LawUFo3Kpggj1FqwIzlK0VuhQTzUiF4MQUMBLHZ1DAlUICAxKBS2x4zIB45cnv+T///8+ PRylqmTmcrq4g5I+ThOjJgpa4Q22LpfzrNoIuTcCzmVHgP9jJg0FtxwEXKUlCBx3UzH01f0A /4wMIBh99oICc64LQWFHOwV36z0IquxpBJq7ZoHb7yPQkHuaQHrRLQIv+8cwtF08j6FE3gxN Z60YqgM9NJj7COSb0/Hk6cUQsBWzYEtZCJ32yyyMdURCg+cVA1WtyyDvjzYClVUNNNSXdWJo Li8g4HH8z0BT/RMaXOdyGPhz0Eqgf8RGgc3vY+GvaguG28ZJ0ckPEww8zqnGcPLaHQzu1xUI Hma+xSA7XhGo9Q9gcMq5FPx3ow5Bp8nLwonsAAv5qSYEp09cpOHF+GMGjG2rYfRTAVn/vVg7 4KNEo/OQWDViocVGqyA+uNzOisaHraxokRNEp32pWFTZh8XCYT8jysWniCgPn2fFLK8bi4PP n7Pik0ujtNjlNuOt4TsUa7VSjC5R0n8X9asi2jTezRx4nYYO15eOUynItS8LBXECv0oYLGpE XzmtoB9PMeEXCy0tAWqK5/DzBWfOOyYLKTiKz5gp2N8/I1PBbH6LkHG/d3pA8wuFl+YH0yIl v1qoyKjFX6TzhJLb1dOiIH6N0P3JNN1RTXba8jxkSirwZ4KEct8b5ssgTHhkb6HPIqUFzShG Kl1cYqxGF7NqRXRSnO7wir3xsTKa/Cnb0bGdZWjY9VMN4jmkDla2HCnUqhhNoiEptgYJHKWe o0zvuKpVKbWapGRJH79HnxAjGWpQOEerQ5UrRw5pVfw+zUHpN0k6IOm/ppgLmpuCFlnUi4xr n265+eOmM0ONzcasxkt6U6inU5e8f+W6iaPK7XnHF+x1DoUs2VgX4k1NWXOvp2e+o/bf8HSm 6yobOhAw7VZ7oxJ2/c0N/bzhmz2zfaVh7fkXKjpUblczad32Ljis8lhI+VtPctixivB43bdP EyZ+v15muBKZnfnLve4mX5SaNkRrIpdSeoPmM54e2dFPAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hUWxiGW2vvvfY4NbGZ49DuRjUVB+zUqcj4yoqgqEXQlaIQDsfB2ScH r8yoZRGYjk2ZigY6WlZeYhQ10z2CetIwxVGzbLxUFjqkXWW8RDmSeUuN/nw88L7v8+tTMOo7 3DKFISxSMobpQrREySqP+MVvlLPy9JvvFmggLWkzeEavspD9oISAs7QYQUnFZQwDjQfh1dgg golnzxmwpjsR5Pb1MlDhcCGoLYwj0Pl+MXR5Rgi0pF8nEJ//gEC7exJDT8YNDMXyYWhNzcNQ N/6JBesAgVvWeDx7PmMYtxXxYItdD/2FN3mY7NsCLa6XHDTcbuGg9s0GyLrTQ6CmtoUFR1U/ hs7/swm4SmY4aHU0s+BMS+bg/nAeAfeYjQGbZ4SHjrocDGXmWduVb9McNCXXYbhyrxxD1+uH CB5dfYtBLnlJoMEziMEupzPwo6ARQX/KEA8JSeM83LqcguB6QgYLz6eaODD3+MLE92yy1482 DI4w1Gw/R2vHclj6JE+k1Td7eWp+9IanOXIUtRf60PyaAUxzv3o4KhddI1T+eoOniUNdmA63 tfG0OXOCpe+7rPjYCn/lLr0UYoiWjH/vCVAGpUx94CJex6HzjtIpJhY5zyYiL4UobBPjst14 jonwp9jdPc7MsbewWrQnf+QSkVLBCJaFYuGXZ2Qu+EM4KloqP88PWGG92G6tRnOsEnzFh5YG /Eu6Siwuq5sXeQnbxQ/fU+Y76tlOT5aLpCJlDlpQhLwNYdGhOkOI7yZTcFBMmOH8psDwUBnN fo3t0mRaFRrtPFiPBAXSLlJ1X8zVqzldtCkmtB6JCkbrrYrvu6tXq/S6mAuSMfxfY1SIZKpH yxWsdonq0GkpQC2c1UVKwZIUIRl/p1jhtSwWVZ6UtueXSzMFTg1Z2lFdsKaj1e/Jl9whTVCk KrDScnxHTcU+98oZc+BOzY7MrY0vVp9ICsjyujC81z+qMjryTDtNDc84cDg84XRMksY1Pap+ XHYquOifddNTk0vNK/13lja7/6rKt/Q27f6PkzNLHW0vqu2GExtCI/YvUdd7v1v7VMuagnRb fBijSfcTi55bujEDAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Yes. How to place Dept in here looks so ugly. But it's inevitable as long as relying on Lockdep. The way should be enhanced gradually. 1. Basically relies on Lockdep to track typical locks and IRQ things. 2. Dept fails to recognize IRQ situation so it generates false alarms when raw_local_irq_*() APIs are used. So made it track those too. 3. Lockdep doesn't track the outmost {hard,soft}irq entracnes but Dept makes use of it. So made it track those too. Signed-off-by: Byungchul Park --- include/linux/irqflags.h | 22 +++++- include/linux/local_lock_internal.h | 1 + include/linux/lockdep.h | 102 ++++++++++++++++++++++------ include/linux/lockdep_types.h | 3 + include/linux/mutex.h | 1 + include/linux/percpu-rwsem.h | 2 +- include/linux/rtmutex.h | 1 + include/linux/rwlock_types.h | 1 + include/linux/rwsem.h | 1 + include/linux/seqlock.h | 2 +- include/linux/spinlock_types_raw.h | 3 + include/linux/srcu.h | 2 +- kernel/dependency/dept.c | 4 +- kernel/locking/lockdep.c | 23 +++++++ 14 files changed, 139 insertions(+), 29 deletions(-) diff --git a/include/linux/irqflags.h b/include/linux/irqflags.h index 5ec0fa71399e..0ebc5ec2dbd4 100644 --- a/include/linux/irqflags.h +++ b/include/linux/irqflags.h @@ -13,6 +13,7 @@ #define _LINUX_TRACE_IRQFLAGS_H #include +#include #include #include @@ -60,8 +61,10 @@ extern void trace_hardirqs_off(void); # define lockdep_softirqs_enabled(p) ((p)->softirqs_enabled) # define lockdep_hardirq_enter() \ do { \ - if (__this_cpu_inc_return(hardirq_context) == 1)\ + if (__this_cpu_inc_return(hardirq_context) == 1) { \ current->hardirq_threaded = 0; \ + dept_hardirq_enter(); \ + } \ } while (0) # define lockdep_hardirq_threaded() \ do { \ @@ -136,6 +139,8 @@ do { \ # define lockdep_softirq_enter() \ do { \ current->softirq_context++; \ + if (current->softirq_context == 1) \ + dept_softirq_enter(); \ } while (0) # define lockdep_softirq_exit() \ do { \ @@ -170,17 +175,28 @@ extern void warn_bogus_irq_restore(void); /* * Wrap the arch provided IRQ routines to provide appropriate checks. */ -#define raw_local_irq_disable() arch_local_irq_disable() -#define raw_local_irq_enable() arch_local_irq_enable() +#define raw_local_irq_disable() \ + do { \ + arch_local_irq_disable(); \ + dept_hardirqs_off(); \ + } while (0) +#define raw_local_irq_enable() \ + do { \ + dept_hardirqs_on(); \ + arch_local_irq_enable(); \ + } while (0) #define raw_local_irq_save(flags) \ do { \ typecheck(unsigned long, flags); \ flags = arch_local_irq_save(); \ + dept_hardirqs_off(); \ } while (0) #define raw_local_irq_restore(flags) \ do { \ typecheck(unsigned long, flags); \ raw_check_bogus_irq_restore(); \ + if (!arch_irqs_disabled_flags(flags)) \ + dept_hardirqs_on(); \ arch_local_irq_restore(flags); \ } while (0) #define raw_local_save_flags(flags) \ diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 975e33b793a7..39f67788fd95 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -21,6 +21,7 @@ typedef struct { .name = #lockname, \ .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ }, \ .owner = NULL, diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 74bd269a80a2..f6bf7567b8df 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -12,6 +12,7 @@ #include #include +#include #include struct task_struct; @@ -39,6 +40,8 @@ static inline void lockdep_copy_map(struct lockdep_map *to, */ for (i = 0; i < NR_LOCKDEP_CACHING_CLASSES; i++) to->class_cache[i] = NULL; + + dept_map_copy(&to->dmap, &from->dmap); } /* @@ -458,7 +461,8 @@ enum xhlock_context_t { * Note that _name must not be NULL. */ #define STATIC_LOCKDEP_MAP_INIT(_name, _key) \ - { .name = (_name), .key = (void *)(_key), } + { .name = (_name), .key = (void *)(_key), \ + .dmap = DEPT_MAP_INITIALIZER(_name, _key) } static inline void lockdep_invariant_state(bool force) {} static inline void lockdep_free_task(struct task_struct *task) {} @@ -540,33 +544,89 @@ extern bool read_lock_is_recursive(void); #define lock_acquire_shared(l, s, t, n, i) lock_acquire(l, s, t, 1, 1, n, i) #define lock_acquire_shared_recursive(l, s, t, n, i) lock_acquire(l, s, t, 2, 1, n, i) -#define spin_acquire(l, s, t, i) lock_acquire_exclusive(l, s, t, NULL, i) -#define spin_acquire_nest(l, s, t, n, i) lock_acquire_exclusive(l, s, t, n, i) -#define spin_release(l, i) lock_release(l, i) - -#define rwlock_acquire(l, s, t, i) lock_acquire_exclusive(l, s, t, NULL, i) +#define spin_acquire(l, s, t, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_exclusive(l, s, t, NULL, i); \ +} while (0) +#define spin_acquire_nest(l, s, t, n, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, n, i); \ + lock_acquire_exclusive(l, s, t, n, i); \ +} while (0) +#define spin_release(l, i) \ +do { \ + ldt_unlock(&(l)->dmap, i); \ + lock_release(l, i); \ +} while (0) +#define rwlock_acquire(l, s, t, i) \ +do { \ + ldt_wlock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_exclusive(l, s, t, NULL, i); \ +} while (0) #define rwlock_acquire_read(l, s, t, i) \ do { \ + ldt_rlock(&(l)->dmap, s, t, NULL, i, !read_lock_is_recursive());\ if (read_lock_is_recursive()) \ lock_acquire_shared_recursive(l, s, t, NULL, i); \ else \ lock_acquire_shared(l, s, t, NULL, i); \ } while (0) - -#define rwlock_release(l, i) lock_release(l, i) - -#define seqcount_acquire(l, s, t, i) lock_acquire_exclusive(l, s, t, NULL, i) -#define seqcount_acquire_read(l, s, t, i) lock_acquire_shared_recursive(l, s, t, NULL, i) -#define seqcount_release(l, i) lock_release(l, i) - -#define mutex_acquire(l, s, t, i) lock_acquire_exclusive(l, s, t, NULL, i) -#define mutex_acquire_nest(l, s, t, n, i) lock_acquire_exclusive(l, s, t, n, i) -#define mutex_release(l, i) lock_release(l, i) - -#define rwsem_acquire(l, s, t, i) lock_acquire_exclusive(l, s, t, NULL, i) -#define rwsem_acquire_nest(l, s, t, n, i) lock_acquire_exclusive(l, s, t, n, i) -#define rwsem_acquire_read(l, s, t, i) lock_acquire_shared(l, s, t, NULL, i) -#define rwsem_release(l, i) lock_release(l, i) +#define rwlock_release(l, i) \ +do { \ + ldt_unlock(&(l)->dmap, i); \ + lock_release(l, i); \ +} while (0) +#define seqcount_acquire(l, s, t, i) \ +do { \ + ldt_wlock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_exclusive(l, s, t, NULL, i); \ +} while (0) +#define seqcount_acquire_read(l, s, t, i) \ +do { \ + ldt_rlock(&(l)->dmap, s, t, NULL, i, false); \ + lock_acquire_shared_recursive(l, s, t, NULL, i); \ +} while (0) +#define seqcount_release(l, i) \ +do { \ + ldt_unlock(&(l)->dmap, i); \ + lock_release(l, i); \ +} while (0) +#define mutex_acquire(l, s, t, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_exclusive(l, s, t, NULL, i); \ +} while (0) +#define mutex_acquire_nest(l, s, t, n, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, n, i); \ + lock_acquire_exclusive(l, s, t, n, i); \ +} while (0) +#define mutex_release(l, i) \ +do { \ + ldt_unlock(&(l)->dmap, i); \ + lock_release(l, i); \ +} while (0) +#define rwsem_acquire(l, s, t, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_exclusive(l, s, t, NULL, i); \ +} while (0) +#define rwsem_acquire_nest(l, s, t, n, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, n, i); \ + lock_acquire_exclusive(l, s, t, n, i); \ +} while (0) +#define rwsem_acquire_read(l, s, t, i) \ +do { \ + ldt_lock(&(l)->dmap, s, t, NULL, i); \ + lock_acquire_shared(l, s, t, NULL, i); \ +} while (0) +#define rwsem_release(l, i) \ +do { \ + ldt_unlock(&(l)->dmap, i); \ + lock_release(l, i); \ +} while (0) #define lock_map_acquire(l) lock_acquire_exclusive(l, 0, 0, NULL, _THIS_IP_) #define lock_map_acquire_try(l) lock_acquire_exclusive(l, 0, 1, NULL, _THIS_IP_) diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index 59f4fb1626ea..fc3e0c136b86 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -11,6 +11,7 @@ #define __LINUX_LOCKDEP_TYPES_H #include +#include #define MAX_LOCKDEP_SUBCLASSES 8UL @@ -77,6 +78,7 @@ struct lock_class_key { struct hlist_node hash_entry; struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES]; }; + struct dept_key dkey; }; extern struct lock_class_key __lockdep_no_validate__; @@ -186,6 +188,7 @@ struct lockdep_map { int cpu; unsigned long ip; #endif + struct dept_map dmap; }; struct pin_cookie { unsigned int val; }; diff --git a/include/linux/mutex.h b/include/linux/mutex.h index 8f226d460f51..58bf314eddeb 100644 --- a/include/linux/mutex.h +++ b/include/linux/mutex.h @@ -25,6 +25,7 @@ , .dep_map = { \ .name = #lockname, \ .wait_type_inner = LD_WAIT_SLEEP, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ } #else # define __DEP_MAP_MUTEX_INITIALIZER(lockname) diff --git a/include/linux/percpu-rwsem.h b/include/linux/percpu-rwsem.h index 36b942b67b7d..e871aca04645 100644 --- a/include/linux/percpu-rwsem.h +++ b/include/linux/percpu-rwsem.h @@ -21,7 +21,7 @@ struct percpu_rw_semaphore { }; #ifdef CONFIG_DEBUG_LOCK_ALLOC -#define __PERCPU_RWSEM_DEP_MAP_INIT(lockname) .dep_map = { .name = #lockname }, +#define __PERCPU_RWSEM_DEP_MAP_INIT(lockname) .dep_map = { .name = #lockname, .dmap = DEPT_MAP_INITIALIZER(lockname, NULL) }, #else #define __PERCPU_RWSEM_DEP_MAP_INIT(lockname) #endif diff --git a/include/linux/rtmutex.h b/include/linux/rtmutex.h index 7d049883a08a..35889ac5eeae 100644 --- a/include/linux/rtmutex.h +++ b/include/linux/rtmutex.h @@ -81,6 +81,7 @@ do { \ .dep_map = { \ .name = #mutexname, \ .wait_type_inner = LD_WAIT_SLEEP, \ + .dmap = DEPT_MAP_INITIALIZER(mutexname, NULL),\ } #else #define __DEP_MAP_RT_MUTEX_INITIALIZER(mutexname) diff --git a/include/linux/rwlock_types.h b/include/linux/rwlock_types.h index 1948442e7750..6e58dfc84997 100644 --- a/include/linux/rwlock_types.h +++ b/include/linux/rwlock_types.h @@ -10,6 +10,7 @@ .dep_map = { \ .name = #lockname, \ .wait_type_inner = LD_WAIT_CONFIG, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL), \ } #else # define RW_DEP_MAP_INIT(lockname) diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h index efa5c324369a..4f856e745dce 100644 --- a/include/linux/rwsem.h +++ b/include/linux/rwsem.h @@ -21,6 +21,7 @@ .dep_map = { \ .name = #lockname, \ .wait_type_inner = LD_WAIT_SLEEP, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ }, #else # define __RWSEM_DEP_MAP_INIT(lockname) diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index 3926e9027947..6ba00bcbc11a 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -81,7 +81,7 @@ static inline void __seqcount_init(seqcount_t *s, const char *name, #ifdef CONFIG_DEBUG_LOCK_ALLOC # define SEQCOUNT_DEP_MAP_INIT(lockname) \ - .dep_map = { .name = #lockname } + .dep_map = { .name = #lockname, .dmap = DEPT_MAP_INITIALIZER(lockname, NULL) } /** * seqcount_init() - runtime initializer for seqcount_t diff --git a/include/linux/spinlock_types_raw.h b/include/linux/spinlock_types_raw.h index 91cb36b65a17..3dcc551ded25 100644 --- a/include/linux/spinlock_types_raw.h +++ b/include/linux/spinlock_types_raw.h @@ -31,11 +31,13 @@ typedef struct raw_spinlock { .dep_map = { \ .name = #lockname, \ .wait_type_inner = LD_WAIT_SPIN, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ } # define SPIN_DEP_MAP_INIT(lockname) \ .dep_map = { \ .name = #lockname, \ .wait_type_inner = LD_WAIT_CONFIG, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ } # define LOCAL_SPIN_DEP_MAP_INIT(lockname) \ @@ -43,6 +45,7 @@ typedef struct raw_spinlock { .name = #lockname, \ .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ + .dmap = DEPT_MAP_INITIALIZER(lockname, NULL),\ } #else # define RAW_SPIN_DEP_MAP_INIT(lockname) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41c4b26fb1c1..49efe1f427fa 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -35,7 +35,7 @@ int __init_srcu_struct(struct srcu_struct *ssp, const char *name, __init_srcu_struct((ssp), #ssp, &__srcu_key); \ }) -#define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map = { .name = #srcu_name }, +#define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map = { .name = #srcu_name, .dmap = DEPT_MAP_INITIALIZER(srcu_name, NULL) }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ int init_srcu_struct(struct srcu_struct *ssp); diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 8ec638254e5f..d3b6d2f4cd7b 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -245,10 +245,10 @@ static inline bool dept_working(void) * Even k == NULL is considered as a valid key because it would use * &->map_key as the key in that case. */ -struct dept_key __dept_no_validate__; +extern struct lock_class_key __lockdep_no_validate__; static inline bool valid_key(struct dept_key *k) { - return &__dept_no_validate__ != k; + return &__lockdep_no_validate__.dkey != k; } /* diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 4dfd2f3e09b2..97eaf13cddd8 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -1221,6 +1221,8 @@ void lockdep_register_key(struct lock_class_key *key) struct lock_class_key *k; unsigned long flags; + dept_key_init(&key->dkey); + if (WARN_ON_ONCE(static_obj(key))) return; hash_head = keyhashentry(key); @@ -4343,6 +4345,8 @@ void noinstr lockdep_hardirqs_on(unsigned long ip) { struct irqtrace_events *trace = ¤t->irqtrace; + dept_hardirqs_on_ip(ip); + if (unlikely(!debug_locks)) return; @@ -4408,6 +4412,8 @@ EXPORT_SYMBOL_GPL(lockdep_hardirqs_on); */ void noinstr lockdep_hardirqs_off(unsigned long ip) { + dept_hardirqs_off_ip(ip); + if (unlikely(!debug_locks)) return; @@ -4452,6 +4458,8 @@ void lockdep_softirqs_on(unsigned long ip) { struct irqtrace_events *trace = ¤t->irqtrace; + dept_softirqs_on_ip(ip); + if (unlikely(!lockdep_enabled())) return; @@ -4490,6 +4498,9 @@ void lockdep_softirqs_on(unsigned long ip) */ void lockdep_softirqs_off(unsigned long ip) { + + dept_softirqs_off_ip(ip); + if (unlikely(!lockdep_enabled())) return; @@ -4837,6 +4848,8 @@ void lockdep_init_map_type(struct lockdep_map *lock, const char *name, { int i; + ldt_init(&lock->dmap, &key->dkey, subclass, name); + for (i = 0; i < NR_LOCKDEP_CACHING_CLASSES; i++) lock->class_cache[i] = NULL; @@ -5581,6 +5594,12 @@ void lock_set_class(struct lockdep_map *lock, const char *name, { unsigned long flags; + /* + * dept_map_(re)init() might be called twice redundantly. But + * there's no choice as long as Dept relies on Lockdep. + */ + ldt_set_class(&lock->dmap, name, &key->dkey, subclass, ip); + if (unlikely(!lockdep_enabled())) return; @@ -5598,6 +5617,8 @@ void lock_downgrade(struct lockdep_map *lock, unsigned long ip) { unsigned long flags; + ldt_downgrade(&lock->dmap, ip); + if (unlikely(!lockdep_enabled())) return; @@ -6398,6 +6419,8 @@ void lockdep_unregister_key(struct lock_class_key *key) unsigned long flags; bool found = false; + dept_key_destroy(&key->dkey); + might_sleep(); if (WARN_ON_ONCE(static_obj(key))) From patchwork Mon Jul 3 09:47:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8010FC001DD for ; Mon, 3 Jul 2023 09:49:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231336AbjGCJty (ORCPT ); Mon, 3 Jul 2023 05:49:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231219AbjGCJts (ORCPT ); Mon, 3 Jul 2023 05:49:48 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 832CF1B5; Mon, 3 Jul 2023 02:49:46 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-38-64a299b2593d From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 06/25] dept: Add proc knobs to show stats and dependency graph Date: Mon, 3 Jul 2023 18:47:33 +0900 Message-Id: <20230703094752.79269-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSb0xTVxjGPefee+6ls+am03iVZZomZsYxFYPkFXVu84MnWzQ6xZjtw6zr je0ohbSI4LIFBzKHUpEEa6FRWk1toAK2ZCuTMgTknxFxdloVqyDqCEUWtAwE3ADjlzdP8vye 36dXYFR13GJBb8yQTUaNQU0UrGJoruMjr82pXV3vT4STx1dD9OVRFuzVHgLdVZUIPLWHMQxc 3QJ3RiMIJq7fYMBa0o3A0fuAgdrWMIKA+ycCt/rnQTA6TKCj5BiB3HPVBG4OTmLoOVWModK7 Fa4VOTE0jj9jwTpAoMyai6fP3xjGXRU8uHKWQZ+7lIfJ3njoCN/mIHDvQ7Cd6SFQH+hgodXf h+HW73YCYc9/HFxrbWeh+2QhBxefOwkMjroYcEWHefizsRxDTd60KP/Faw7aChsx5J+/hCF4 9zKChqOPMHg9twk0RyMYfN4SBl5duIqgzzLEw5Hj4zyUHbYgOHbkFAs3pto4yOtZCxNjdvJJ Em2ODDM0z3eQBkbLWdrplGhd6QOe5jXc42m59wD1uVfQc/UDmDpGohz1VvxCqHekmKcFQ0FM n3d18bT99ARL+4NWvD32K8UGrWzQZ8qmVR/vVej8LXaSHkrMOjsYm4N+iytAMYIkJkgPHW3k bS6NTKGZTMQPpFBonJnJ88Wlkq/wKVeAFAIj/vyO5P7n+uzgXXGf9PJS+yzEisuk4pHzuAAJ glJcK1VXrHrjXCJV1jTOIjFiovRkzDLrV00jPbYwmXFKojVGutnpwm8Gi6Qr7hBbhJTlaE4F UumNmakavSFhpS7bqM9a+W1aqhdNP5Trh8mv/Wike2cTEgWknqsMfe/QqjhNpjk7tQlJAqOe r8ztPatVKbWa7EOyKe0b0wGDbG5CsQKrXqhcM3pQqxL3azLkFFlOl01vWyzELM5ByZtsXZvM Ge+/uP9qQXKEaV6iK7Mesl127jKT/S1ZX27cbCuuTd6wMz/935rSrRtb7WO613uD6+P8cY+v +IZUnDHpx5YiSxXzh5/u2PaZJ6Vz3RfPUqYefvdrznvJlhNJmfENw03L2TTDnv5w4MlfVbtr P63LTcgu4i31nydtX6Det17NmnWa+BWMyaz5H+NaW0dMAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0xTZxzGfd9zznsOnXVnHYkn4uKsccbLBBZx/wRidnHxdcbbEjXxixzt iXRCNa0y2TIDA52rQCwJVko3SjWFlSp6SjIUcQhSYARl0LhqoBsEtxFuyiihK7oBy748eZLn l9+nR2B05dwywWg6qZhNcqaeaFjNrtT8t9UytyHpetUbYCtMgsjUeRactT4C3ddrEPjq8jAM t26DX6ZHEcS6HjJgL+1GUDnQz0BdIIygsforAr1DSyAYmSDQUXqBQP6VWgI/j8xi6LtUgqFG 3QmdF90YmqJ/sGAfJlBuz8dz8SeGqMfLgyd3NQxWO3iYHUiGjvAjDlq+7eCg8cl6KPuuj8Cd xg4WAvWDGHpvOwmEff9w0BloZ6HbVsTBtXE3gZFpDwOeyAQPPU0uDDcK5mzn/nrJQVtRE4Zz V29iCD5uQHD3/G8YVN8jAi2RUQx+tZSBv6taEQwWj/FwtjDKQ3leMYILZy+x8PBFGwcFfSkQ m3GS91Jpy+gEQwv8n9HGaRdLf3JL9Jajn6cFd5/w1KWeov7qdfTKnWFMKycjHFW93xCqTpbw 1DoWxHT8wQOetl+OsXQoaMd7lh/UpBmUTGO2Yk7ckq7JqL/vJCdCm09XjCTkoh82WFGcIImb JMfoCzTfibhGCoWizHyPF9+U/EW/c1akERjx61ek6mddZH54XTwsTd1sX4BYcbVUMnkVW5Eg aMUUqdab+J9zhVRzo2kBiRM3S09nihf8ujmkryxMLiKNCy3yonijKTtLNmambLQcy8gxGU9v PHI8S0Vzl/F8OWurR1O925qRKCD9Ym3oi0qDjpOzLTlZzUgSGH28Nn+gwqDTGuSczxXz8UPm U5mKpRklCKx+qfbjA0q6Tjwqn1SOKcoJxfz/ioW4ZbkoaLOnfXggcOad/v1717e1Hip81/V+ TaKDbt0k3/vUp4/NTHqzMqye50nfjxnG3fvrbM6EhseB8Edr6zopm5a0I/2D7dGXS0t/7HHv S+t59fC+md3LB+tTn5rkhu7FUus90+U9q966L6Y+W5kdS/4kLzS0t0rHqK+ZjnrcXb/6xxxb 9KwlQ05ex5gt8r+Be7T2LgMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org It'd be useful to show Dept internal stats and dependency graph on runtime via proc for better information. Introduced the knobs. Signed-off-by: Byungchul Park --- kernel/dependency/Makefile | 1 + kernel/dependency/dept.c | 24 +++----- kernel/dependency/dept_internal.h | 26 +++++++++ kernel/dependency/dept_proc.c | 95 +++++++++++++++++++++++++++++++ 4 files changed, 131 insertions(+), 15 deletions(-) create mode 100644 kernel/dependency/dept_internal.h create mode 100644 kernel/dependency/dept_proc.c diff --git a/kernel/dependency/Makefile b/kernel/dependency/Makefile index b5cfb8a03c0c..92f165400187 100644 --- a/kernel/dependency/Makefile +++ b/kernel/dependency/Makefile @@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DEPT) += dept.o +obj-$(CONFIG_DEPT) += dept_proc.o diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index d3b6d2f4cd7b..c5e23e9184b8 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -74,6 +74,7 @@ #include #include #include +#include "dept_internal.h" static int dept_stop; static int dept_per_cpu_ready; @@ -261,20 +262,13 @@ static inline bool valid_key(struct dept_key *k) * have been freed will be placed. */ -enum object_t { -#define OBJECT(id, nr) OBJECT_##id, - #include "dept_object.h" -#undef OBJECT - OBJECT_NR, -}; - #define OBJECT(id, nr) \ static struct dept_##id spool_##id[nr]; \ static DEFINE_PER_CPU(struct llist_head, lpool_##id); #include "dept_object.h" #undef OBJECT -static struct dept_pool pool[OBJECT_NR] = { +struct dept_pool dept_pool[OBJECT_NR] = { #define OBJECT(id, nr) { \ .name = #id, \ .obj_sz = sizeof(struct dept_##id), \ @@ -304,7 +298,7 @@ static void *from_pool(enum object_t t) if (DEPT_WARN_ON(!irqs_disabled())) return NULL; - p = &pool[t]; + p = &dept_pool[t]; /* * Try local pool first. @@ -339,7 +333,7 @@ static void *from_pool(enum object_t t) static void to_pool(void *o, enum object_t t) { - struct dept_pool *p = &pool[t]; + struct dept_pool *p = &dept_pool[t]; struct llist_head *h; preempt_disable(); @@ -2136,7 +2130,7 @@ void dept_map_copy(struct dept_map *to, struct dept_map *from) clean_classes_cache(&to->map_key); } -static LIST_HEAD(classes); +LIST_HEAD(dept_classes); static inline bool within(const void *addr, void *start, unsigned long size) { @@ -2168,7 +2162,7 @@ void dept_free_range(void *start, unsigned int sz) while (unlikely(!dept_lock())) cpu_relax(); - list_for_each_entry_safe(c, n, &classes, all_node) { + list_for_each_entry_safe(c, n, &dept_classes, all_node) { if (!within((void *)c->key, start, sz) && !within(c->name, start, sz)) continue; @@ -2244,7 +2238,7 @@ static struct dept_class *check_new_class(struct dept_key *local, c->sub_id = sub_id; c->key = (unsigned long)(k->base + sub_id); hash_add_class(c); - list_add(&c->all_node, &classes); + list_add(&c->all_node, &dept_classes); unlock: dept_unlock(); caching: @@ -2958,8 +2952,8 @@ static void migrate_per_cpu_pool(void) struct llist_head *from; struct llist_head *to; - from = &pool[i].boot_pool; - to = per_cpu_ptr(pool[i].lpool, boot_cpu); + from = &dept_pool[i].boot_pool; + to = per_cpu_ptr(dept_pool[i].lpool, boot_cpu); move_llist(to, from); } } diff --git a/kernel/dependency/dept_internal.h b/kernel/dependency/dept_internal.h new file mode 100644 index 000000000000..007c1eec6bab --- /dev/null +++ b/kernel/dependency/dept_internal.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Dept(DEPendency Tracker) - runtime dependency tracker internal header + * + * Started by Byungchul Park : + * + * Copyright (c) 2020 LG Electronics, Inc., Byungchul Park + */ + +#ifndef __DEPT_INTERNAL_H +#define __DEPT_INTERNAL_H + +#ifdef CONFIG_DEPT + +enum object_t { +#define OBJECT(id, nr) OBJECT_##id, + #include "dept_object.h" +#undef OBJECT + OBJECT_NR, +}; + +extern struct list_head dept_classes; +extern struct dept_pool dept_pool[]; + +#endif +#endif /* __DEPT_INTERNAL_H */ diff --git a/kernel/dependency/dept_proc.c b/kernel/dependency/dept_proc.c new file mode 100644 index 000000000000..7d61dfbc5865 --- /dev/null +++ b/kernel/dependency/dept_proc.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Procfs knobs for Dept(DEPendency Tracker) + * + * Started by Byungchul Park : + * + * Copyright (C) 2021 LG Electronics, Inc. , Byungchul Park + */ +#include +#include +#include +#include "dept_internal.h" + +static void *l_next(struct seq_file *m, void *v, loff_t *pos) +{ + /* + * XXX: Serialize list traversal if needed. The following might + * give a wrong information on contention. + */ + return seq_list_next(v, &dept_classes, pos); +} + +static void *l_start(struct seq_file *m, loff_t *pos) +{ + /* + * XXX: Serialize list traversal if needed. The following might + * give a wrong information on contention. + */ + return seq_list_start_head(&dept_classes, *pos); +} + +static void l_stop(struct seq_file *m, void *v) +{ +} + +static int l_show(struct seq_file *m, void *v) +{ + struct dept_class *fc = list_entry(v, struct dept_class, all_node); + struct dept_dep *d; + const char *prefix; + + if (v == &dept_classes) { + seq_puts(m, "All classes:\n\n"); + return 0; + } + + prefix = fc->sched_map ? " " : ""; + seq_printf(m, "[%p] %s%s\n", (void *)fc->key, prefix, fc->name); + + /* + * XXX: Serialize list traversal if needed. The following might + * give a wrong information on contention. + */ + list_for_each_entry(d, &fc->dep_head, dep_node) { + struct dept_class *tc = d->wait->class; + + prefix = tc->sched_map ? " " : ""; + seq_printf(m, " -> [%p] %s%s\n", (void *)tc->key, prefix, tc->name); + } + seq_puts(m, "\n"); + + return 0; +} + +static const struct seq_operations dept_deps_ops = { + .start = l_start, + .next = l_next, + .stop = l_stop, + .show = l_show, +}; + +static int dept_stats_show(struct seq_file *m, void *v) +{ + int r; + + seq_puts(m, "Availability in the static pools:\n\n"); +#define OBJECT(id, nr) \ + r = atomic_read(&dept_pool[OBJECT_##id].obj_nr); \ + if (r < 0) \ + r = 0; \ + seq_printf(m, "%s\t%d/%d(%d%%)\n", #id, r, nr, (r * 100) / (nr)); + #include "dept_object.h" +#undef OBJECT + + return 0; +} + +static int __init dept_proc_init(void) +{ + proc_create_seq("dept_deps", S_IRUSR, NULL, &dept_deps_ops); + proc_create_single("dept_stats", S_IRUSR, NULL, dept_stats_show); + return 0; +} + +__initcall(dept_proc_init); From patchwork Mon Jul 3 09:47:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 607E4C001DF for ; Mon, 3 Jul 2023 09:50:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231852AbjGCJu4 (ORCPT ); Mon, 3 Jul 2023 05:50:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231217AbjGCJts (ORCPT ); Mon, 3 Jul 2023 05:49:48 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 8D0A01B6; Mon, 3 Jul 2023 02:49:46 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-46-64a299b2c9a4 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 07/25] dept: Apply sdt_might_sleep_{start,end}() to wait_for_completion()/complete() Date: Mon, 3 Jul 2023 18:47:34 +0900 Message-Id: <20230703094752.79269-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTH9zz33ufedlavlekzNdM1cRqM77CcLWqMX7zZZuJiSLaZTRt7 I90KmoJINSZVwBcEIkyovCktpuugm3KLEV/KOlQUVKjYICNALBEVoeLQory42WL8cvLL+Z/z +/QXGO1lbrZgTE6Vzcl6k46oWXVoin2JUuwwLG+wrob8nOUQfnmEhbKzbgL+P6sRuGsPYOi/ vgHujwwiGL/TyoCt0I/AHuxmoLaxB4HXdZDAvYdTIRAeItBUeIxARuVZAncHJjB0FRVgqFY2 wq3jDgy+0ccs2PoJlNoycGQ8wTDqrOLBaV0Ava4SHiaCK6Cpp50Db+diKD7VReCKt4mFxrpe DPculRHocf/Pwa3Gmyz483M5+OOZg8DAiJMBZ3iIhzZfBYZzmRHRoRf/cXAj14fh0JkaDIF/ LiOoP/IAg+JuJ3A1PIjBoxQyMPbbdQS9eSEesnJGeSg9kIfgWFYRC61vbnCQ2RUP46/LyLov pauDQ4yU6dkjeUcqWKnZQaWLJd28lFnfyUsVym7J44qVKq/0Y8k+HOYkpeookZThAl7KDgWw 9KylhZdunhxnpYcBG9405wf1aoNsMqbJ5mVrt6kTB7whftdf09PL/dd4K6qdmo1UAhXjaN/9 Mfyea2s6uSgTcSHt6Bhlohwjzqee3EeRvVpgxMMfUtfzOyQazBDT6YtX9ZPMigtoccHQpEgj xtOJdjf7TjqPVp/zTYpU4ue073UeirI2ctNV3EOiUir+qqI5Sjl59/Ax/dvVwR5Hmgr0QRXS GpPTkvRGU9zSREuyMX3p9p1JCopUyrl/YksdGvZvbkCigHRTNB377AYtp09LsSQ1ICowuhhN RvC0Qasx6C17ZfPOrebdJjmlAc0RWN0szcqRPQatuEOfKv8iy7tk8/sUC6rZVmR9Vcl+P/fn 57etv4ebz3xVnv5079MQu/LapzNbVPlt875wtl2ip/9NzRaWzbKM5cZ6dxTkL150wlVzovvr LZ8ctGc13/3pRxRIWHP+5bSEb4LTP9qM7Kbz4meOygv2vpIinOCrW9/bGrf9u2kxahpvWXuy dOuqi7ctG8psm771dAcLdTo2JVG/IpYxp+jfAl/A82VOAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0yTZxiG977fkW7Vz1rjt26JpBGNmgkmgk/mMpfsh++WbPGPJ/whjf20 DQdJqwjqErTonAoTNmRAUUApDVSoX2uCaJHAqBwiVGkQGkAhqGsAWYCCHHZoXfbnyZX7vnP9 enhKVcZoeGPaCcmUpkvRsgpa8f1Oy2dycaU+ru7XGMi/Ggeh2Us0WOsdLPjqahE43OcwBNt2 w/O5CQRLT3ooKCr0IagYGaLA7R1G4LGfZ6F3bAX4Q1MsdBReYcFyq56Fp+PLGAavF2Colb+D rmuVGJoX3tBQFGShtMiCw+cPDAu2Gg5s2TEwai/hYHlkG3QM9zHQWtbBgCewBYpvDLLw0NNB g7dhFENvo5WFYcc/DHR522nw5ecycOdtJQvjczYKbKEpDp41l2Nw5oRtF2f+ZuBxbjOGi7fv YvAPPEDQdOklBtnRx0JraAKDSy6kYLG6DcFo3iQHF64ucFB6Lg/BlQvXaej56zEDOYPxsPTO yn61k7ROTFEkx3WKeObKadJZKZL7JUMcyWkKcKRcPklc9s3k1sMgJhXTIYbINT+xRJ4u4Mjl ST8mb7u7OdL+2xJNxvxFeM+niYov9FKKMUMyxX6ZpDCMeya59EerMst8v3PZyL3iMoriRWG7 6L4bYCLMChvF/v4FKsJqIVp05b4O5wqeEn78ULT/+YSNFKuFTHFmvuk900KMWFwwhSOsFOLF 5T4H/Z90nVjrbH4vihISxFfv8lCEVeHNYPEwew0pytEHNUhtTMtI1RlT4reakw1ZacbMrUeO p8oo/DS2H5bzG9Bs7+4WJPBI+5Gy/0yFXsXoMsxZqS1I5CmtWmkZualXKfW6rNOS6fhh08kU ydyCPuFp7Vrlt/ulJJVwTHdCSpakdMn0f4v5KE02cq7ZV/v1lslOxdL8ITFTLD1brQxqDq0M zFbvjf4laZ2BCezN9388dFDj6bSquxMbY7vXnj86Lfvt3wxs2vXadiNhTzTxFgg7fJqzzjV9 L1421NVbFxuPVZVsiB1wz/fvMqyaWd36uanw3vY2NZ8tl/zsTV6//kB6leW22RVcTHJqabNB t20zZTLr/gX+XJbcMAMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dependencies by wait_for_completion()/complete(). Signed-off-by: Byungchul Park --- include/linux/completion.h | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/completion.h b/include/linux/completion.h index 62b32b19e0a8..32d535abebf3 100644 --- a/include/linux/completion.h +++ b/include/linux/completion.h @@ -10,6 +10,7 @@ */ #include +#include /* * struct completion - structure used to maintain state for a "completion" @@ -26,14 +27,33 @@ struct completion { unsigned int done; struct swait_queue_head wait; + struct dept_map dmap; }; +#define init_completion(x) \ +do { \ + sdt_map_init(&(x)->dmap); \ + __init_completion(x); \ +} while (0) + +/* + * XXX: No use cases for now. Fill the body when needed. + */ #define init_completion_map(x, m) init_completion(x) -static inline void complete_acquire(struct completion *x) {} -static inline void complete_release(struct completion *x) {} + +static inline void complete_acquire(struct completion *x) +{ + sdt_might_sleep_start(&x->dmap); +} + +static inline void complete_release(struct completion *x) +{ + sdt_might_sleep_end(); +} #define COMPLETION_INITIALIZER(work) \ - { 0, __SWAIT_QUEUE_HEAD_INITIALIZER((work).wait) } + { 0, __SWAIT_QUEUE_HEAD_INITIALIZER((work).wait), \ + .dmap = DEPT_MAP_INITIALIZER(work, NULL), } #define COMPLETION_INITIALIZER_ONSTACK_MAP(work, map) \ (*({ init_completion_map(&(work), &(map)); &(work); })) @@ -75,13 +95,13 @@ static inline void complete_release(struct completion *x) {} #endif /** - * init_completion - Initialize a dynamically allocated completion + * __init_completion - Initialize a dynamically allocated completion * @x: pointer to completion structure that is to be initialized * * This inline function will initialize a dynamically created completion * structure. */ -static inline void init_completion(struct completion *x) +static inline void __init_completion(struct completion *x) { x->done = 0; init_swait_queue_head(&x->wait); From patchwork Mon Jul 3 09:47:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA76AC04A6A for ; Mon, 3 Jul 2023 09:49:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231382AbjGCJt4 (ORCPT ); Mon, 3 Jul 2023 05:49:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231231AbjGCJtt (ORCPT ); Mon, 3 Jul 2023 05:49:49 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1FA76BE; Mon, 3 Jul 2023 02:49:46 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-56-64a299b25f73 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 08/25] dept: Apply sdt_might_sleep_{start,end}() to PG_{locked,writeback} wait Date: Mon, 3 Jul 2023 18:47:35 +0900 Message-Id: <20230703094752.79269-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0yTZxTHfZ732mr1TSXbKySoXcgSzVAZmBO8xJgsPl5jtsQPmmzr1nej GVTSAgJqBCmgyEUxWJHiSjG1gw5ZSxYRqR1G5BJqHQ1DU8koTMcEUbQotzFg8uXkl/M/5/fp z1PKW0w4r9WlSHqdOlHFymn5yDLrJ65yq2ZjxwAPFwo3QujNGRrMNxws+OpqETgasjEM3dsF f4wPI5jqekCBqcyHoKr/CQUNrX0Imu2nWegeXA7+0CgL7WXnWMipvsHCw+fTGAKXSjHUOvdD 53krBs/EMxpMQyxUmHLw3Pgbw4SthgNbVhQE7Vc4mO7fBO19PQw0P14P5VcDLNxubqeh9WYQ Q/ctMwt9jlkGOlvbaPBdKGLg5xdWFp6P2yiwhUY5+N1jwVBvnBPlvf6XgftFHgx5137B4H/U hMB95k8MTkcPC3dDwxhczjIKJq/fQxAsHuEgt3CCg4rsYgTnci/R8GDmPgPGQBxMvTOzO+LJ 3eFRihhdx0jzuIUmHVaRNF55whGj+zFHLM5U4rKvI9W3hzCpGgsxxFlzliXOsVKOFIz4MXnh 9XKk7fIUTQb9Jnww4rB8q0ZK1KZJ+g3bv5YnVMz6qeQZRXplz0dZKLC0AMl4UYgVvdketMiz pzupeWaFj8Xe3okFDhPWiK6ip0wBkvOUkL9UtL/sYueDlYJeLKmrX2BaiBJNz/zMPCuEOLHQ PfVeulqsrfcsiGTCZvGvd8ULe+XcTaC8j52XikK+TDzb2Ij/f1gl/mbvpc8jhQUtqUFKrS4t Sa1NjI1OyNBp06O/PZrkRHONsp2cPnITjfm+aEECj1TLFL3HqzRKRp1myEhqQSJPqcIUOf0/ apQKjTojU9If/UqfmigZWlAET6s+VMSMH9Mohe/VKdIPkpQs6RdTzMvCs9AH0QWf79u8OzhJ 7SzOjPyma8/Ffe2p3+V43Y0HVjAPt7w6NdnzKsborvOWtJ5qiKzc0l048HZ6pcaQ989gaf7O t/tnkz+ToirS9h6JRNvkKTHk1xMHrpW+3GZuk+5sCBPuxFLr1wbzTuw2h4fKZNyXOn/T8g6d 5dBPb+I/bYp7lNvlU9GGBPWmdZTeoP4PmcZHNU0DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTYRjHe99zznvmbHFYVgctqoUEXUwh7aEb9clDVCQEXSBytVMub7Wp aTe0qZm3VLBlzlKrKWqlsw928ZKiaZJarmUyR1ppomWZW81LpkZfHn78/39+nx4JJTcw7hJ1 WISoCVOGKIiUlu7doltvyilUecf9WAeZqd5gH0uiwfCwjEDHg1IEZY/iMAw2+sM7xzCCiVft FOizOxAU9PZQ8KjJhqC6+DKBzk8LwGwfIdCSnUJAd+chgddDkxis17MwlJr2QGtGIYY65wAN +kECuXodnjlfMDiNJSwYYz2hr/gmC5O9PtBiszDQkNfCQHX3Wsi5ZSXwrLqFhqaqPgydTwwE bGXTDLQ2NdPQkZnGwP1vhQSGHEYKjPYRFt7U5WMoj5+xJf78w8CLtDoMiXcrMJjfP0VQk/QB g6nMQqDBPoyh0pRNwXhRI4K+9K8sJKQ6WciNS0eQknCdhvapFwzEW31h4reB7NgiNAyPUEJ8 5Vmh2pFPCy8LeeHxzR5WiK/pZoV8U6RQWbxGuPNsEAsFo3ZGMJVcJYJpNIsVkr+asfCtrY0V mm9M0MInsx7vW3pYulUlhqijRM2G7YHSoNxpM3V6ShadZ1kVi6yuychFwnMb+enLrdQsE241 39XlnGM3bgVfmdbPJCOphOKuuPLF31+R2WIhp+GvPSifY5rz5PUDZmaWZZwvn1ozgf5Jl/Ol 5XVzIhfOj//8O30ul89srDk2koGk+WheCXJTh0WFKtUhvl7a4KCYMHW01/HwUBOa+RnjxcnM KjTW6V+POAlSzJd1nS9QyRlllDYmtB7xEkrhJtP13lbJZSplzDlRE35UExkiauuRh4RWLJHt OiAGyrmTyggxWBRPi5r/LZa4uMciv/TG/QG3kpOOuW4iHvd0207qd7/c6TEe0JyZMXXkuOWg 1DIaWes6vvbQqCEj/PHCgrbn9x0/lGeKzEubPT9uDi7dTVXUXlokum93rDrhIldPRjvsFyo2 Bp/I/Whpt833SVwmH+pPcFRtyMk65VjsjZ3h2oGpiF9vx1IO9p/rCfBfqaC1QUqfNZRGq/wL uvVjgC8DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dependencies by PG_{locked,writeback} waits. Signed-off-by: Byungchul Park --- mm/filemap.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 83dda76d1fc3..eed64dc88e43 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1219,6 +1220,9 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; +static struct dept_map __maybe_unused PG_locked_map = DEPT_MAP_INITIALIZER(PG_locked_map, NULL); +static struct dept_map __maybe_unused PG_writeback_map = DEPT_MAP_INITIALIZER(PG_writeback_map, NULL); + static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { @@ -1230,6 +1234,11 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, unsigned long pflags; bool in_thrashing; + if (bit_nr == PG_locked) + sdt_might_sleep_start(&PG_locked_map); + else if (bit_nr == PG_writeback) + sdt_might_sleep_start(&PG_writeback_map); + if (bit_nr == PG_locked && !folio_test_uptodate(folio) && folio_test_workingset(folio)) { delayacct_thrashing_start(&in_thrashing); @@ -1331,6 +1340,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, */ finish_wait(q, wait); + sdt_might_sleep_end(); + if (thrashing) { delayacct_thrashing_end(&in_thrashing); psi_memstall_leave(&pflags); From patchwork Mon Jul 3 09:47:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A74C7C19F4F for ; Mon, 3 Jul 2023 09:50:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231428AbjGCJt5 (ORCPT ); Mon, 3 Jul 2023 05:49:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231234AbjGCJtt (ORCPT ); Mon, 3 Jul 2023 05:49:49 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 4A4E918D; Mon, 3 Jul 2023 02:49:47 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-66-64a299b232c5 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 09/25] dept: Apply sdt_might_sleep_{start,end}() to swait Date: Mon, 3 Jul 2023 18:47:36 +0900 Message-Id: <20230703094752.79269-10-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTH9zz33udeOrvdFBMfX5KxRrNMFFHRnZhlMWbGuyxmxA0/aJat 896MZryYVqp1L0FeHCsvqTrsUKK0klqhTLg1EV/qGMwqI0DRqkiwCFGxArKwtRNhutbNLye/ /M85v5wPR2B057kFgjFvl2LKM+ToiYbVTMxxLvfVuOT09q7X4UBFOkT/KmOh9rSXQPDnRgTe M/swRC5vgluxcQQz3b0MOKqDCJzDdxg4Ewgj8HuKCFy/9xqEopMEOqvLCRSfOE2gb2wWw+Dh gxga1c3QZXdhaJseZcERIXDUUYzj5SGGaXcDD+7CJTDiOcLD7PBK6Azf5MA/kAo1xwYJXPR3 shBoHcFw/XwtgbD3OQddgassBA9UctD02EVgLOZmwB2d5OFaWx2G5pK4aP+fzzi4UtmGYX99 C4bQ7QsILpXdxaB6bxLoiI5j8KnVDDw9eRnBSNUED6UV0zwc3VeFoLz0MAu9/1zhoGRwDcw8 qSXr10kd45OMVOLbLfljdaz0u4tK547c4aWSSwO8VKcWSD7PUunExQiWnFNRTlIbfiCSOnWQ l2wTISw97unhpas/zbDSvZADZy7cpnlXVnKMFsW04r3PNdk9FUF+Z0DYc9Y/wBWiXmJDSQIV M6i91cu85AZbB5tgIr5F+/unX+RzxRTqq3zA2ZBGYMTvX6WeP7rjy4KQLH5KH45sTcyw4hL6 KHwXJ1grrqUD9nr2P+cbtLG57YUnKZ7ff1KFEqwT19DBmjBJOKl4KIkW20P/HzSf/urpZ+1I W4deaUA6Y54l12DMyUjLtuYZ96TtyM9VUfyj3N/Obm9FU8GP25EoIP0cbf/XTlnHGSxma247 ogKjn6stHj4u67SywbpXMeV/ZirIUcztaKHA6udpV8V2yzrxS8Mu5StF2amYXnaxkLSgEH3U /M7JrExt5EP/aDTVsb1oRYu/QM5IWVSkdm+0Wk6lfVeWu3bLN11ZT1c3oRa6LnmZ5f3I0NbM t/veTP0ReT5omb1hTNm8+G9YNFTfTYOPbJs+canPanqUZcSeXx74pQ/ONqY3LY7Jkd/GXFmZ G7Kc1lHiNJd+sX4o+TbodbKeNWcbVi5lTGbDv3+aBwVNAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTHfZ5773Mv1bqbjsiNGueaGRPcnCSwnKgxxky5LG76wYExMdqs N9AJhbTIRMMCFhny0hQT7JCqpWW1od3Q1mXqrCBMtBgpGy9Cg400voCALGiJCFpbl305+eV/ zv/36XCUwsIs5zTaQkmnVeUqiYyWfbPJ8JmnwabecD+QCnU1GyDyspIGS6ubQO9vLgTuy2UY xm+lw/3ZSQTz9wIUmOt7ETSNPqDgclcIgc95nEDfo6XQH5km4K+vJmCwtxL4e2IBw8jpUxhc nq/hrsmGoX3uKQ3mcQKNZgOOjTEMc44WFhylayDsPMPCwmgK+EODDHSe9TPgC66DhnMjBK77 /DR0XQlj6LtmIRByRxm423WHht66WgZ+fW4jMDHroMARmWbhn3YrhovlMVvFi7cM3K5tx1DR fAlD//CfCG5UPsTgcQ8S6IxMYvB66il4feEWgrBxioUTNXMsNJYZEVSfOE1D4M1tBspH0mD+ lYVs3SR2Tk5TYrn3B9E3a6XFbpsgXj3zgBXLbwRZ0eo5LHqdyaL9+jgWm2YijOhpOUlEz8wp Vqya6sfi854eVrzz8zwtPuo3490r98k2q6VcTZGk+3zLQVlOT00vW9DFHfnDF2RKUYBUoQRO 4FOFlqpOOs6EXysMDc1RcU7kVwve2idMFZJxFP/TYsH5771YgeM+5PcLY+HM+A3NrxGehR7i OMv5L4SgqZn+z/mR4LrY/t6TEMsfvzKiOCv4NGGkIURMSGZFi1pQokZblKfS5Kat1x/KKdZq jqz/Lj/Pg2I/4yhZqLuCXvaldyCeQ8ol8qFjTWoFoyrSF+d1IIGjlIlyw+h5tUKuVhUflXT5 B3SHcyV9B1rB0cok+VdZ0kEFn60qlA5JUoGk+3+LuYTlpQimo5+69mW9iJa0qV8vk4dMO/dy F56SHSklA9lfRjO+7zak+5K6Vw/8ldmYOqXcvuPmMu8H1ZWM10DP7zk7GG0umPlk47Nw6Nsh i32sOMuabyr7fSK4WftLZtJOPzTBHlcgI4M3Ok2uHz9uk++yG23D9ifGmwMVeqt5caV71TYl rc9RpSRTOr3qHWmXBMkvAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dependencies by swaits. Signed-off-by: Byungchul Park --- include/linux/swait.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/swait.h b/include/linux/swait.h index 6a8c22b8c2a5..02848211cef5 100644 --- a/include/linux/swait.h +++ b/include/linux/swait.h @@ -6,6 +6,7 @@ #include #include #include +#include #include /* @@ -161,6 +162,7 @@ extern void finish_swait(struct swait_queue_head *q, struct swait_queue *wait); struct swait_queue __wait; \ long __ret = ret; \ \ + sdt_might_sleep_start(NULL); \ INIT_LIST_HEAD(&__wait.task_list); \ for (;;) { \ long __int = prepare_to_swait_event(&wq, &__wait, state);\ @@ -176,6 +178,7 @@ extern void finish_swait(struct swait_queue_head *q, struct swait_queue *wait); cmd; \ } \ finish_swait(&wq, &__wait); \ + sdt_might_sleep_end(); \ __out: __ret; \ }) From patchwork Mon Jul 3 09:47:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88E6EEB64DD for ; Mon, 3 Jul 2023 09:50:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231843AbjGCJuw (ORCPT ); Mon, 3 Jul 2023 05:50:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231255AbjGCJtu (ORCPT ); Mon, 3 Jul 2023 05:49:50 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 369E8196; Mon, 3 Jul 2023 02:49:47 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-76-64a299b33678 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 10/25] dept: Apply sdt_might_sleep_{start,end}() to waitqueue wait Date: Mon, 3 Jul 2023 18:47:37 +0900 Message-Id: <20230703094752.79269-11-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSXUxTZxjH977nnPecdnY5qWyc6cVME6NBUXS4PNnMsgsS35slRrzSRNfQ M2kslLRSRMYE+RD5UkgAUVTaulKhDG3dRBCCEEDEDxCiXYOdZTrHpGJw7ewAJ0W9efLL/5// 7+oRGHUHt0LQpx+QTelag4YoWWVwmTXeU2/TJfz+UyxUlSdA6J8SFhraXARGfm5B4Lqcj2Gq fxs8CE8jmLt9l4G6mhEE1sBDBi4P+BF0OY8QGHv8EYyHZggM1ZQRKLC3ERh9No9horYaQ4v7 Wxg+YcPQE3nKQt0UgdN1BXjx/IUh4mjmwZG3Giadp3iYD2yCIf99Drp866D+7ASBa11DLAy0 T2IY62gg4Hf9z8HwwA0WRqoqOGh9biPwLOxgwBGa4eFeTyOGi4WLouKXrzkYrOjBUHz+Eobx 3zoRdJc8wuB23SfQF5rG4HHXMPBfUz+CycogD0XlER5O51ciKCuqZeHuwiAHhRNbYO5VA/nm S9o3PcPQQk8W7Qo3svSmTaJXTz3kaWG3j6eN7kzqccZR+7UpTK2zIY66m48R6p6t5mlpcBzT 53fu8PTGyTmWPh6vw9tX7lJu1ckGvUU2bfz6O2XqUGdOxgXFwe7jPpSHmvhSpBAkMVFq7e7D 73msLZ+JMhHXSF5vZIljxFWSp+JPrhQpBUY8+qHkfHGbRIvl4vfSrTNVbJRZcbV0vjq8JFKJ X0hP/QPvpJ9JLRd7lkSKxfzJq0oUZbW4RZqo95OoVBKPK6RQ5A/u7eBT6brTy55Aqkb0QTNS 69MtaVq9IXFDana6/uCGFGOaGy1+lCN3fnc7mh1J7kWigDTLVN4cq07NaS3m7LReJAmMJkZV EDinU6t02uxDssm415RpkM29aKXAamJVm8NZOrW4T3tA3i/LGbLpfYsFxYo8lNT/xMgdK0ke /NHSkfDyX5+948Jw3tEaY9aqQVvkRdlXvvwjsVealr8uyw3uWxvgF3J2B6c/2frxA/fhzrmx uKQfdmby69fabc2Rz63lM0KuYY9jNDkladS7Lm2H8aZFDtBfqxZ8ife8e//e8cvGItcjza5D s0kKC7EX18YfNpl9Gtacqt0Ux5jM2jepiRhWTQMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0xTdxyG/f/PlUqXk4rxBMlkVeKim0IC+lPI1H3hhERnTIyZX6RZj9Jw 0bSCojGCrUyhgCViBUELmNJBBXYwjItljIZivRZoFElphKhIQFCkCINdqMu+vHnyvsnz6WUJ RQUVzmoyToraDFWakpaRsn3x+m+lsmp19IuCBDAZoyEwe4mEikY7DZ6GegT2u7kYxnsS4fnc JILFx08JMJd6EFSNDBNw1+VH4LBdoGHg1RfgDUzT4C4toEFf00hD38QSBt+1Egz10l54eKUa Q9fCGAnmcRpumPV4Od5iWLDWMWDNiYJRWzkDSyMx4PY/o8BZ6abAMbQZym76aLjncJPgah3F MNBeQYPf/g8FD133SfCYCim4M1VNw8SclQBrYJqB/i4LhibDsi3v498U9BZ2Yci7/SsG74sO BJ2XXmKQ7M9ocAYmMTRLpQT8WduDYLToHQMXjQsM3MgtQlBw8RoJT//qpcDgi4PF+Qp6d7zg nJwmBEPzKcExZyGFB9W80FY+zAiGziFGsEiZQrNtk1BzbxwLVTMBSpDqLtOCNFPCCPnvvFiY evKEEe5fXySFV14z3h9xWJagFtM0WaJ263fJshR3x9kTv4Sc7iweQjmolslHISzPxfIDjblE kGluIz84uPCZw7hIvrnwDZWPZCzB/bySt71/TAeHVdxR/lGliQwyyUXxt0vmcJDl3DZ+zO/C /0nX8fVNXZ9FIcv96/kiFGQFF8f7yvz0FSSzoBV1KEyTkZWu0qTFbdGlpmRnaE5v+el4uoSW P2M9t2RqRbMDid2IY5EyVD54tkqtoFRZuuz0bsSzhDJMrh+5pVbI1arsM6L2+BFtZpqo60Zr WVK5Rp50SExWcMdUJ8VUUTwhav9fMRsSnoN+m2lIighb9fv2LPbB16Er+nr2loeu7/9yj/3H pu+Hd5ijfbolV3FPa/KsZyo+taMtPOHCxC5T0vnR+V0tBrmldip0LHVnlTHR9j5pfW9TrDfy g6P4mx/ObV5TVGCMdFb2n1rdrgmXth5wug9uUJbl61s+5Vzefd2YfpVoOXhV8dUfHiWpS1HF bCK0OtW/ZFm/Ti8DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dependencies by waitqueue waits. Signed-off-by: Byungchul Park --- include/linux/wait.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/wait.h b/include/linux/wait.h index a0307b516b09..ff349e609da7 100644 --- a/include/linux/wait.h +++ b/include/linux/wait.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -303,6 +304,7 @@ extern void init_wait_entry(struct wait_queue_entry *wq_entry, int flags); struct wait_queue_entry __wq_entry; \ long __ret = ret; /* explicit shadow */ \ \ + sdt_might_sleep_start(NULL); \ init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \ for (;;) { \ long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\ @@ -318,6 +320,7 @@ extern void init_wait_entry(struct wait_queue_entry *wq_entry, int flags); cmd; \ } \ finish_wait(&wq_head, &__wq_entry); \ + sdt_might_sleep_end(); \ __out: __ret; \ }) From patchwork Mon Jul 3 09:47:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9450C10F00 for ; Mon, 3 Jul 2023 09:50:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231496AbjGCJuA (ORCPT ); Mon, 3 Jul 2023 05:50:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231281AbjGCJtv (ORCPT ); Mon, 3 Jul 2023 05:49:51 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id BD15212C; Mon, 3 Jul 2023 02:49:48 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-86-64a299b34ffc From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 11/25] dept: Apply sdt_might_sleep_{start,end}() to hashed-waitqueue wait Date: Mon, 3 Jul 2023 18:47:38 +0900 Message-Id: <20230703094752.79269-12-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSfUzMcRzHfb+/x05nvx2bH23YbcbymInPDMsf1s9Dxox5+IOb+9FxXbn0 aKaUpEe15fSA6+LcuvN0h4mylFKUosahjlooelrczbmKK/XPZ6+9P+/Pe+8/Piwhe0TNZlWa E6JWo1DLaQkp6fM1LLHlG5TLe/PmQU7GcnD+SiWh6LaFhuZbZgSWe4kYemqC4Z2rF4GnsYkA XV4zguKOdgLu1ToQVJjO0NDSNQ1anQM01Oel05BUcpuG1z+GMbRdzMVgtobAywsGDJXubyTo emgo1CVh7+jG4DaWMmBMmA+dpgIGhjsCoN7xloKKD4sg/0obDeUV9STUPuzE0PKoiAaH5S8F L2vrSGjOyaTgZr+Bhh8uIwFG5wADbyr1GO4ke4NSfo5S8DyzEkPKtbsYWt8/RvAk9TMGq+Ut DdXOXgw2ax4Bf27UIOjM6mPgbIabgcLELATpZy+S0DTynILktkDw/C6ig9YI1b0DhJBsixEq XHpSeGHghbKCdkZIfvKBEfTWKMFm8hdKynuwUDzkpARr6XlasA7lMkJaXysW+l+9YoS6Sx5S 6GrV4e1++yRrlaJaFS1ql60/KAn91p5DRdjZ2JFnZUwCstNpyIfluZW8W5/NTHLT9z/UGNPc At5udxNjPIObx9syv3p1CUtw56bypsHG8ePpXBiflpgybiK5+fz5S/3jupRbxY86cyZC5/Lm O5XjHh+v/uV3FhpjGRfIt+U7Jkpk+/Ddgzv+8yz+qclOXkBSPZpSimQqTXSYQqVeuTQ0TqOK XXooPMyKvB9lPDW8/yEaat5ZhTgWyX2l9pPFShmliI6MC6tCPEvIZ0iTOq4qZVKlIi5e1IYf 0Eapxcgq5MeS8pnSFa4YpYw7ojghHhPFCFE7ucWsz+wEtO1TbUBaQqAuvaFOdi3mQFD1rg2f UNnezY3HozUbcL7ar7jGvDveUdBV+HFOKd1R1bRVe3/x4Z0W39XhR9ZkHOxueeB556HmhC8M jj0TvC5VIupCzLK+FVsbWIcyZeP16ljHEld50abLp3F2w6zACKUiaosuPmJ7SQY6+npB157L cjIyVBHgT2gjFf8AEZjQbE0DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTH99yX516qXW4qZndoMtdpTDQqhqFnQsw+4TO3OT+YzOzDpLE3 UoFaW4uiYfIuoBBLggUFhbrUBvCFls3XmgoIFgIUS6Qa2ghBHMqLAYpAUVdY9uXkl9/5n/+n w9OKKjaK12iPSXqtKkWJZYxsT1zOJkeFRR39/hEG07loCE4XMFB5sx6D50YdgvrGLApGHu+C vplRBKHObhrMZR4ENQN+GhpbAwictmwM3qHPoTc4gcFddhZDztWbGHreLlDQf6GUgjr7z9Bx 3kKBa+41A+YRDJfMOVR4/EPBnLWWA2vmOhi0XeRgYWAruAPPWGiucrPgfLERKi73Y3jgdDPQ emeQAu+9SgyB+k8sdLQ+YcBjKmbh+rgFw9sZKw3W4AQHT13VFNzKDbflT31koa3YRUH+nw0U 9D6/j+BhwUsK7PXPMDQHRylw2MtomL/2GMFgyRgHeefmOLiUVYLgbN4FBro/tLGQ2x8LodlK /H0caR6doEmu4zhxzlQzpN0ikrsX/RzJffiCI9V2I3HYNpCrD0YoUjMZZIm9thAT+2QpR4rG eiky3tXFkSflIYYM9Zqpvat/k8WrpRRNmqTfsjNRlvTab2J1Pv7Eh5a7XCby4SIUwYvCt2L3 m3l2kbGwXvT55uhFjhTWiI7i4bCX8bRwZploe9e5dLBCSBWLsvKXQoywTiwsH1/ycmGb+DFo 4v4r/Uqsu+VaykSE/avZErTICiFW7K8I4PNIVo0+q0WRGm1aqkqTErvZkJyUrtWc2HzwSKod hZ/GmrFguoOmvbuakMAj5XK571SNWsGq0gzpqU1I5GllpDxn4IpaIVer0k9K+iMH9MYUydCE VvGM8gv57l+lRIVwSHVMSpYknaT/f0vxEVGZqCUUlefVbNSFGnTlv3zXZVtLjiZ4HGXb9s3/ 8fRQQkys5Sd/jNEvvHL9/U3L6ss/Ru6HtTHtiTr51+4+o9BTWJo1tcO5s3Py9309a4ZGdw+b Gz3t3tuWwS+r1KGMlXkDZriS3TDb98MN7fbh+GijdnnGHr4jeSLjr5OnA21Few/LlIwhSbV1 A603qP4FQqFgYTADAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dependencies by hashed-waitqueue waits. Signed-off-by: Byungchul Park --- include/linux/wait_bit.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/wait_bit.h b/include/linux/wait_bit.h index 7725b7579b78..fe89282c3e96 100644 --- a/include/linux/wait_bit.h +++ b/include/linux/wait_bit.h @@ -6,6 +6,7 @@ * Linux wait-bit related types and methods: */ #include +#include struct wait_bit_key { void *flags; @@ -246,6 +247,7 @@ extern wait_queue_head_t *__var_waitqueue(void *p); struct wait_bit_queue_entry __wbq_entry; \ long __ret = ret; /* explicit shadow */ \ \ + sdt_might_sleep_start(NULL); \ init_wait_var_entry(&__wbq_entry, var, \ exclusive ? WQ_FLAG_EXCLUSIVE : 0); \ for (;;) { \ @@ -263,6 +265,7 @@ extern wait_queue_head_t *__var_waitqueue(void *p); cmd; \ } \ finish_wait(__wq_head, &__wbq_entry.wq_entry); \ + sdt_might_sleep_end(); \ __out: __ret; \ }) From patchwork Mon Jul 3 09:47:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9924AC001B3 for ; Mon, 3 Jul 2023 09:50:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231670AbjGCJur (ORCPT ); Mon, 3 Jul 2023 05:50:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231308AbjGCJtw (ORCPT ); Mon, 3 Jul 2023 05:49:52 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0944A1A1; Mon, 3 Jul 2023 02:49:48 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-96-64a299b31a91 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 12/25] dept: Distinguish each syscall context from another Date: Mon, 3 Jul 2023 18:47:39 +0900 Message-Id: <20230703094752.79269-13-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSfUzMcRzHfb+/xzvOfs7Tz7MdzcbkYbLPMOMfvhs2G5N57OZ+dNPFrkTM 1imhq1ZRSY0rdk5dyu88RMrJxImK0pPTlKeaksXdqstDV/PPe6+935/3+68PT6nvM1N5fXik ZAzXhmlYJa3sHpO70JGVp1t8r28epCYuBs+vszTkFNlZqL1ZgMB+24Sh8+l6aPR2IfC9qqEg M70WQW7bewpuV7YiKLOdYqHu01io9/Sw4Eo3sxB7tYiF198GMbgz0jAUyJugKiUPg7P/Kw2Z nSxkZ8biIenA0G/N58AaEwDttkscDLYtAVdrAwNlLQsg67KbhYdlLhoqS9ox1D3IYaHV/peB qsrnNNSmJjFQ+D2PhW9eKwVWTw8Hb5wWDMVxQ0PxP/8w8CzJiSH+2i0M9c2lCMrPfsAg2xtY eOLpwuCQ0ykYuP4UQXtyNwenE/s5yDYlIzCfzqCh5vczBuLcQeDry2HXrCBPunooEuc4Ssq8 Fpq8yBPJ/UvvORJX3sIRi3yEOGzzydWHnZjk9noYIuefY4ncm8aRhO56TL5XV3Pk+UUfTT7V Z+LN03YoV+mkMH2UZFy0OkQZ+rPURx2+EHLsc5oZx6CLGxOQgheFZWJpcwlKQPwwF8mBfpsV 5olNTf2UnycIs0VH0hcmASl5SjgzWrT9eMX6g/HCHjHD1kj7mRYCxHcNicMFlbBcdMS3UCP7 s8SCYucwK4b8z33JyM9qIUh0Z7WyIzfnFWK2ec0ITxEf25roFKSyoFH5SK0PjzJo9WHLAkOj w/XHAvcdMsho6KGsJwd3lqDe2i0VSOCRZoyq6USuTs1ooyKiDRVI5CnNBFVs2xWdWqXTRh+X jIf2Go+ESREVaBpPayarlnqP6tTCAW2kdFCSDkvG/ynmFVNjUHDwx1/Tc2qkdSmh6s2yYeDO ia1z7roe71bHbl9rkSrs3Iwp9/pM+SFnGrZ2LrR3GO5e7922/9w4xYaJFtXBVTNvzI4exW90 BdU1V5+a1P4O2yxOn3fA6za8Xb8ruPDySuXxO+RrWvJoU5Y9teXRrrkdkcWVVQ8+CC/LTcFR 5sn6Rg0dEapdMp8yRmj/ARbQfIZMAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzXSa0yTZxQHcJ/nvVKseVOJe6ckLl1wESJoMsiJ6Gb2hScGFvdhaoiZNPZV Gi5CCx3sYrhfCwMiN0WlxdQGuoFvjTco6yAgjHCZEGSsECGwgSIsYAkMkAGJX05+Of/8z6fD U6oaZj+vi0+S9PGaWDWroBVfhmYecVRbtEcfLu2BUtNR8LzNo6Gm0c7CwC8NCOwP0jHMdoTB i+U5BGu9/RRUlg8gME+MUfCgcxyB05bBwuDUHhjyLLDQXV7IQmZdIwt/vF7H4K4ow9AgR0BP iQWDa/UfGipnWbhZmYm3xgyGVWs9B9Y0P5i03eBgfeIYdI8PM9B+q5sB52gAVN92s9Di7Kah 8/EkhsGnNSyM2zcZ6OnsomGgtIiBn+ctLLxetlJg9Sxw8NxVi6Epa+taztI7Bp4VuTDk3L2P YejPZgSteS8xyPZhFto9cxgccjkF/93rQDBZ/IaDbNMqBzfTixEUZlfQ0L/xjIEsdzCsrdSw p0JJ+9wCRbIc3xLnci1NfreI5MmNMY5ktY5ypFZOJg6bP6lrmcXEvOhhiFyfzxJ5sYwjBW+G MJnv6+NIV9UaTaaGKvEZ30jFCa0UqzNK+qDPohTRS81rVML1qJTpskKchqrCCxDPi8KnYqMc WIC8eFb4RBwZWaW27SN8JDqK/mYKkIKnhFxv0fZvL7sd7BW+EStsL+ht04Kf+NewaaegFEJE R87ojkXhoNjQ5Nqx19Z+eqUYbVslBIvu6nG2BClq0a565KOLN8ZpdLHBgYaY6NR4XUrgpatx Mtr6GeuP66WP0dvBsDYk8Ei9WznyvVmrYjRGQ2pcGxJ5Su2jzJy4o1UptZrU7yT91Yv65FjJ 0IYO8LT6A+Xpc1KUSriiSZJiJClB0r9PMe+1Pw0dCJUtLe25kSnpi4kRzRkb1PlU6WldQOJ8 zG+mkWuvuBPMy7MXeuZDwnoPNbUadjdUHf/hVkxa9uEgV+T1IL9wY+LH5V9/cXGfU9W1uc/k fe+ybD9zusO9YSH5M4+Y5x/m+ee26nwP14ebFz//Kenhr+aZMeOrrwKSFMexu63/5OAhNW2I 1hzzp/QGzf9Tn8A7LwMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org It enters kernel mode on each syscall and each syscall handling should be considered independently from the point of view of Dept. Otherwise, Dept may wrongly track dependencies across different syscalls. That might be a real dependency from user mode. However, now that Dept just started to work, conservatively let Dept not track dependencies across different syscalls. Signed-off-by: Byungchul Park --- arch/arm64/kernel/syscall.c | 2 ++ arch/x86/entry/common.c | 4 +++ include/linux/dept.h | 39 ++++++++++++--------- kernel/dependency/dept.c | 67 +++++++++++++++++++------------------ 4 files changed, 63 insertions(+), 49 deletions(-) diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index da84cf855c44..d5a43e721173 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -105,6 +106,7 @@ static void el0_svc_common(struct pt_regs *regs, int scno, int sc_nr, */ local_daif_restore(DAIF_PROCCTX); + dept_kernel_enter(); if (flags & _TIF_MTE_ASYNC_FAULT) { /* diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 6c2826417b33..7cdd27abe529 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -19,6 +19,7 @@ #include #include #include +#include #ifdef CONFIG_XEN_PV #include @@ -72,6 +73,7 @@ static __always_inline bool do_syscall_x32(struct pt_regs *regs, int nr) __visible noinstr void do_syscall_64(struct pt_regs *regs, int nr) { + dept_kernel_enter(); add_random_kstack_offset(); nr = syscall_enter_from_user_mode(regs, nr); @@ -120,6 +122,7 @@ __visible noinstr void do_int80_syscall_32(struct pt_regs *regs) { int nr = syscall_32_enter(regs); + dept_kernel_enter(); add_random_kstack_offset(); /* * Subtlety here: if ptrace pokes something larger than 2^31-1 into @@ -140,6 +143,7 @@ static noinstr bool __do_fast_syscall_32(struct pt_regs *regs) int nr = syscall_32_enter(regs); int res; + dept_kernel_enter(); add_random_kstack_offset(); /* * This cannot use syscall_enter_from_user_mode() as it has to diff --git a/include/linux/dept.h b/include/linux/dept.h index b6d45b4b1fd6..f62c7b6f42c6 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -25,11 +25,16 @@ struct task_struct; #define DEPT_MAX_SUBCLASSES_USR (DEPT_MAX_SUBCLASSES / DEPT_MAX_SUBCLASSES_EVT) #define DEPT_MAX_SUBCLASSES_CACHE 2 -#define DEPT_SIRQ 0 -#define DEPT_HIRQ 1 -#define DEPT_IRQS_NR 2 -#define DEPT_SIRQF (1UL << DEPT_SIRQ) -#define DEPT_HIRQF (1UL << DEPT_HIRQ) +enum { + DEPT_CXT_SIRQ = 0, + DEPT_CXT_HIRQ, + DEPT_CXT_IRQS_NR, + DEPT_CXT_PROCESS = DEPT_CXT_IRQS_NR, + DEPT_CXTS_NR +}; + +#define DEPT_SIRQF (1UL << DEPT_CXT_SIRQ) +#define DEPT_HIRQF (1UL << DEPT_CXT_HIRQ) struct dept_ecxt; struct dept_iecxt { @@ -94,8 +99,8 @@ struct dept_class { /* * for tracking IRQ dependencies */ - struct dept_iecxt iecxt[DEPT_IRQS_NR]; - struct dept_iwait iwait[DEPT_IRQS_NR]; + struct dept_iecxt iecxt[DEPT_CXT_IRQS_NR]; + struct dept_iwait iwait[DEPT_CXT_IRQS_NR]; /* * classified by a map embedded in task_struct, @@ -207,8 +212,8 @@ struct dept_ecxt { /* * where the IRQ-enabled happened */ - unsigned long enirq_ip[DEPT_IRQS_NR]; - struct dept_stack *enirq_stack[DEPT_IRQS_NR]; + unsigned long enirq_ip[DEPT_CXT_IRQS_NR]; + struct dept_stack *enirq_stack[DEPT_CXT_IRQS_NR]; /* * where the event context started @@ -252,8 +257,8 @@ struct dept_wait { /* * where the IRQ wait happened */ - unsigned long irq_ip[DEPT_IRQS_NR]; - struct dept_stack *irq_stack[DEPT_IRQS_NR]; + unsigned long irq_ip[DEPT_CXT_IRQS_NR]; + struct dept_stack *irq_stack[DEPT_CXT_IRQS_NR]; /* * where the wait happened @@ -406,19 +411,19 @@ struct dept_task { int wait_hist_pos; /* - * sequential id to identify each IRQ context + * sequential id to identify each context */ - unsigned int irq_id[DEPT_IRQS_NR]; + unsigned int cxt_id[DEPT_CXTS_NR]; /* * for tracking IRQ-enabled points with cross-event */ - unsigned int wgen_enirq[DEPT_IRQS_NR]; + unsigned int wgen_enirq[DEPT_CXT_IRQS_NR]; /* * for keeping up-to-date IRQ-enabled points */ - unsigned long enirq_ip[DEPT_IRQS_NR]; + unsigned long enirq_ip[DEPT_CXT_IRQS_NR]; /* * current effective IRQ-enabled flag @@ -470,7 +475,7 @@ struct dept_task { .wait_hist = { { .wait = NULL, } }, \ .ecxt_held_pos = 0, \ .wait_hist_pos = 0, \ - .irq_id = { 0U }, \ + .cxt_id = { 0U }, \ .wgen_enirq = { 0U }, \ .enirq_ip = { 0UL }, \ .eff_enirqf = 0UL, \ @@ -509,6 +514,7 @@ extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long ip); extern void dept_sched_enter(void); extern void dept_sched_exit(void); +extern void dept_kernel_enter(void); static inline void dept_ecxt_enter_nokeep(struct dept_map *m) { @@ -560,6 +566,7 @@ struct dept_task { }; #define dept_ecxt_exit(m, e_f, ip) do { } while (0) #define dept_sched_enter() do { } while (0) #define dept_sched_exit() do { } while (0) +#define dept_kernel_enter() do { } while (0) #define dept_ecxt_enter_nokeep(m) do { } while (0) #define dept_key_init(k) do { (void)(k); } while (0) #define dept_key_destroy(k) do { (void)(k); } while (0) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index c5e23e9184b8..4165cacf4ebb 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -221,9 +221,9 @@ static inline struct dept_class *dep_tc(struct dept_dep *d) static inline const char *irq_str(int irq) { - if (irq == DEPT_SIRQ) + if (irq == DEPT_CXT_SIRQ) return "softirq"; - if (irq == DEPT_HIRQ) + if (irq == DEPT_CXT_HIRQ) return "hardirq"; return "(unknown)"; } @@ -407,7 +407,7 @@ static void initialize_class(struct dept_class *c) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_iecxt *ie = &c->iecxt[i]; struct dept_iwait *iw = &c->iwait[i]; @@ -432,7 +432,7 @@ static void initialize_ecxt(struct dept_ecxt *e) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { e->enirq_stack[i] = NULL; e->enirq_ip[i] = 0UL; } @@ -448,7 +448,7 @@ static void initialize_wait(struct dept_wait *w) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { w->irq_stack[i] = NULL; w->irq_ip[i] = 0UL; } @@ -487,7 +487,7 @@ static void destroy_ecxt(struct dept_ecxt *e) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) if (e->enirq_stack[i]) put_stack(e->enirq_stack[i]); if (e->class) @@ -503,7 +503,7 @@ static void destroy_wait(struct dept_wait *w) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) if (w->irq_stack[i]) put_stack(w->irq_stack[i]); if (w->class) @@ -652,7 +652,7 @@ static void print_diagram(struct dept_dep *d) const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); irqf = e->enirqf & w->irqf; - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { if (!firstline) pr_warn("\nor\n\n"); firstline = false; @@ -685,7 +685,7 @@ static void print_dep(struct dept_dep *d) const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); irqf = e->enirqf & w->irqf; - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { pr_warn("%s has been enabled:\n", irq_str(irq)); print_ip_stack(e->enirq_ip[irq], e->enirq_stack[irq]); pr_warn("\n"); @@ -911,7 +911,7 @@ static void bfs(struct dept_class *c, bfs_f *cb, void *in, void **out) */ static inline unsigned long cur_enirqf(void); -static inline int cur_irq(void); +static inline int cur_cxt(void); static inline unsigned int cur_ctxt_id(void); static inline struct dept_iecxt *iecxt(struct dept_class *c, int irq) @@ -1459,7 +1459,7 @@ static void add_dep(struct dept_ecxt *e, struct dept_wait *w) if (d) { check_dl_bfs(d); - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_iwait *fiw = iwait(fc, i); struct dept_iecxt *found_ie; struct dept_iwait *found_iw; @@ -1495,7 +1495,7 @@ static void add_wait(struct dept_class *c, unsigned long ip, struct dept_task *dt = dept_task(); struct dept_wait *w; unsigned int wg = 0U; - int irq; + int cxt; int i; if (DEPT_WARN_ON(!valid_class(c))) @@ -1511,9 +1511,9 @@ static void add_wait(struct dept_class *c, unsigned long ip, w->wait_stack = get_current_stack(); w->sched_sleep = sched_sleep; - irq = cur_irq(); - if (irq < DEPT_IRQS_NR) - add_iwait(c, irq, w); + cxt = cur_cxt(); + if (cxt == DEPT_CXT_HIRQ || cxt == DEPT_CXT_SIRQ) + add_iwait(c, cxt, w); /* * Avoid adding dependency between user aware nested ecxt and @@ -1594,7 +1594,7 @@ static bool add_ecxt(struct dept_map *m, struct dept_class *c, eh->sub_l = sub_l; irqf = cur_enirqf(); - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) add_iecxt(c, irq, e, false); del_ecxt(e); @@ -1746,7 +1746,7 @@ static void do_event(struct dept_map *m, struct dept_class *c, add_dep(eh->ecxt, wh->wait); } - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_ecxt *e; if (before(dt->wgen_enirq[i], wg)) @@ -1788,7 +1788,7 @@ static void disconnect_class(struct dept_class *c) call_rcu(&d->rh, del_dep_rcu); } - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { stale_iecxt(iecxt(c, i)); stale_iwait(iwait(c, i)); } @@ -1813,27 +1813,21 @@ static inline unsigned long cur_enirqf(void) return 0UL; } -static inline int cur_irq(void) +static inline int cur_cxt(void) { if (lockdep_softirq_context(current)) - return DEPT_SIRQ; + return DEPT_CXT_SIRQ; if (lockdep_hardirq_context()) - return DEPT_HIRQ; - return DEPT_IRQS_NR; + return DEPT_CXT_HIRQ; + return DEPT_CXT_PROCESS; } static inline unsigned int cur_ctxt_id(void) { struct dept_task *dt = dept_task(); - int irq = cur_irq(); + int cxt = cur_cxt(); - /* - * Normal process context - */ - if (irq == DEPT_IRQS_NR) - return 0U; - - return dt->irq_id[irq] | (1UL << irq); + return dt->cxt_id[cxt] | (1UL << cxt); } static void enirq_transition(int irq) @@ -1884,7 +1878,7 @@ static void enirq_update(unsigned long ip) /* * Do enirq_transition() only on an OFF -> ON transition. */ - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { if (prev & (1UL << irq)) continue; @@ -1983,6 +1977,13 @@ void dept_hardirqs_off_ip(unsigned long ip) } EXPORT_SYMBOL_GPL(dept_hardirqs_off_ip); +void dept_kernel_enter(void) +{ + struct dept_task *dt = dept_task(); + + dt->cxt_id[DEPT_CXT_PROCESS] += 1UL << DEPT_CXTS_NR; +} + /* * Ensure it's the outmost softirq context. */ @@ -1990,7 +1991,7 @@ void dept_softirq_enter(void) { struct dept_task *dt = dept_task(); - dt->irq_id[DEPT_SIRQ] += 1UL << DEPT_IRQS_NR; + dt->cxt_id[DEPT_CXT_SIRQ] += 1UL << DEPT_CXTS_NR; } /* @@ -2000,7 +2001,7 @@ void dept_hardirq_enter(void) { struct dept_task *dt = dept_task(); - dt->irq_id[DEPT_HIRQ] += 1UL << DEPT_IRQS_NR; + dt->cxt_id[DEPT_CXT_HIRQ] += 1UL << DEPT_CXTS_NR; } void dept_sched_enter(void) From patchwork Mon Jul 3 09:47:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEF49C001B3 for ; Mon, 3 Jul 2023 09:50:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231524AbjGCJuA (ORCPT ); Mon, 3 Jul 2023 05:50:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231310AbjGCJtx (ORCPT ); Mon, 3 Jul 2023 05:49:53 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C0B00BE; Mon, 3 Jul 2023 02:49:49 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-a6-64a299b3d23d From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 13/25] dept: Distinguish each work from another Date: Mon, 3 Jul 2023 18:47:40 +0900 Message-Id: <20230703094752.79269-14-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzXSbUxTZxQHcJ/n3vvcS6XmpmK8ajKXJsbMF9T5dlzmy/yyu8QtTdgHM0xm Y6/S2CIpiKCS4aji0EvQiIiWScF0DXSAt8QgUkUYICJYtUoltZGKYkMRxyyBwWTUzS8nv5xz 8s/5cDhKc51ZyBlTMyRLqt6kJSpaNRxvX+kurTCsDjckwpnTqyH67iQNtloXAW9NNQJX/TEM 4bavoXcsgmCy+z4FJcVeBPb+ZxTUtwcReJw/E3g0MAd80RECncWnCORV1hJ4MDSFIXD+LIZq 5VvoKqrA0DwxSENJmMClkjw8U15jmHBUseDIXQIh50UWpvrXQGfwCQOevuVQ+muAQJOnk4b2 hhCGR402AkHXNANd7Xdo8J6RGfj9TQWBoTEHBY7oCAsPm8sx1Flngk789Z6BDrkZw4krVzH4 nt5AcPPkcwyK6wmB1mgEg1sppuDv39oQhAqHWTh+eoKFS8cKEZw6fp6G+/90MGANrIfJcRvZ 9oXYGhmhRKv7kOgZK6fFuxWCeP3iM1a03uxjxXLloOh2LhMrm8JYtI9GGVGp+oWIyuhZViwY 9mHxTU8PK965MEmLA74SrFv0g+pLg2QyZkqWVVt2q1JqXT4q7e3crN76eblI4QtQHCfw64Se 1mn80da+Iipmwi8V/P6JD07gPxXc8iumAKk4is+fLTjfdpMCxHFz+SQhMKqJ7dD8EuFesI7E rOY3CPfkMuq/zMVCdV3zB8fN9F+OF6KYNfx6IVAaJLFMgc+PEy44nP8fsUC47fTTRUhdjmZV IY0xNdOsN5rWJaZkpxqzEvccMCto5qEcOVPJDWjUm9SCeA5p49X+I3aDhtFnpmebW5DAUdoE dV7/ZYNGbdBnH5YsB360HDRJ6S1oEUdr56s/Hztk0PD79BnSfklKkywfp5iLW5iLyjZcOzKU ti3UkPTCP7W0K3/+4/jBb7YavVfN/ue2to7tyY2Rz74LfZWhQ59Y92bvf51XI3fvXiFbdf3T 8h6ZTTbpQ13D4caE0pxNkd4/R+TNT8eP+qzmnGjNw6adAybdOe+tn8q+r1e2eDYOrvCudQvv dPY/zIPxu+DcjrtZth1aOj1Fv2YZZUnX/wuCRdYFTAMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzXSf0yMcRwHcN/v8zzf5+k4e3bCM0x2m5ksMuKzMTP+8Gh++4PxB8/cMx2V dlcpP7bokHStIukH+rXT6ijPNb9Lrl11hOiW5NyU/LgpJS7l8qOYfz577fP+7P3Xh6M0+cw0 Th8dKxuipUgtUdGqDcuSQ2y5xbpQx/MpkJkWCr5vKTQUVFoJtFyrQGCtPobB61gDLwZ7EPgf P6UgJ7sFQVHnawqqGzwIasqOE2jtngguXx8BZ/YZAskllQSefRrB4D6fhaFCWQ+PMoox1A1/ oCHHSyA/JxmPjo8Yhi3lLFiSZkNXWR4LI50LwelpY6D+opOBmo55kHvJTeBejZOGhltdGFrv FBDwWH8z8KihiYaWTDMDVz8XE/g0aKHA4utj4XldIYYq02jbya+/GGg012E4WXodg+vlXQS1 KW8wKNY2AvW+Hgw2JZuCH1ccCLrSe1k4kTbMQv6xdARnTpyn4enPRgZM7jDwDxWQlcvE+p4+ SjTZDoo1g4W0+LBYEG/nvWZFU20HKxYqcaKtLFgsuefFYtGAjxGV8tNEVAayWDG114XFz0+e sGLTBT8tdrty8KYZO1TLdXKkPl42LFixWxVRaXVRMf2TEl5UT05CCp+KAjiBXyyYOjKoMRN+ jtDePvzXgfwswWZ+z6QiFUfxp8YLZf2PSSriuEn8VsE9oBm7ofnZQrOnioxZzS8Rms0XqX+d QUJFVd1fB4zu3w2lozFr+DDBneshGUhViMaVo0B9dHyUpI8Mm2/cH5EYrU+Yv+dAlIJGX8Zy dCTzFvrWusaOeA5pJ6jbDxfpNIwUb0yMsiOBo7SB6uTOyzqNWiclHpINB3YZ4iJlox1N52jt VHX4Nnm3ht8rxcr7ZTlGNvxPMRcwLQnZpoecqxKCG2/gbrI2LVBf1Ptl/SKFHr/y1PagnQ57 f0rczO+umJBmc2f5jY1LH6wuCfLffNvkP12aN9MZvsoy93tFn96/etHygq/VFx5cjr17fZCT t3gd0pDgv9QWOnTEd1Z6F+q1t9xvvLl5X0NWtxcn2Kh4T23GOms4++pqhJY2RkgLgymDUfoD Zj/+/y4DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Workqueue already provides concurrency control. By that, any wait in a work doesn't prevents events in other works with the control enabled. Thus, each work would better be considered a different context. So let Dept assign a different context id to each work. Signed-off-by: Byungchul Park --- include/linux/dept.h | 2 ++ kernel/dependency/dept.c | 10 ++++++++++ kernel/workqueue.c | 3 +++ 3 files changed, 15 insertions(+) diff --git a/include/linux/dept.h b/include/linux/dept.h index f62c7b6f42c6..d9ca9dd50219 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -515,6 +515,7 @@ extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long extern void dept_sched_enter(void); extern void dept_sched_exit(void); extern void dept_kernel_enter(void); +extern void dept_work_enter(void); static inline void dept_ecxt_enter_nokeep(struct dept_map *m) { @@ -567,6 +568,7 @@ struct dept_task { }; #define dept_sched_enter() do { } while (0) #define dept_sched_exit() do { } while (0) #define dept_kernel_enter() do { } while (0) +#define dept_work_enter() do { } while (0) #define dept_ecxt_enter_nokeep(m) do { } while (0) #define dept_key_init(k) do { (void)(k); } while (0) #define dept_key_destroy(k) do { (void)(k); } while (0) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 4165cacf4ebb..6cf17f206b78 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -1977,6 +1977,16 @@ void dept_hardirqs_off_ip(unsigned long ip) } EXPORT_SYMBOL_GPL(dept_hardirqs_off_ip); +/* + * Assign a different context id to each work. + */ +void dept_work_enter(void) +{ + struct dept_task *dt = dept_task(); + + dt->cxt_id[DEPT_CXT_PROCESS] += 1UL << DEPT_CXTS_NR; +} + void dept_kernel_enter(void) { struct dept_task *dt = dept_task(); diff --git a/kernel/workqueue.c b/kernel/workqueue.c index c913e333cce8..fa23d876a8b5 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -52,6 +52,7 @@ #include #include #include +#include #include "workqueue_internal.h" @@ -2318,6 +2319,8 @@ __acquires(&pool->lock) lockdep_copy_map(&lockdep_map, &work->lockdep_map); #endif + dept_work_enter(); + /* ensure we're on the correct CPU */ WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu); From patchwork Mon Jul 3 09:47:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 257AAC10F00 for ; Mon, 3 Jul 2023 09:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231289AbjGCJuE (ORCPT ); Mon, 3 Jul 2023 05:50:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231316AbjGCJtx (ORCPT ); Mon, 3 Jul 2023 05:49:53 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D9E2D18D; Mon, 3 Jul 2023 02:49:49 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-b7-64a299b31b80 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 14/25] dept: Add a mechanism to refill the internal memory pools on running out Date: Mon, 3 Jul 2023 18:47:41 +0900 Message-Id: <20230703094752.79269-15-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSf0yMcRzHfb/PzzuOxzEetWE31iZSlH2YYfOHZ8zGzObXxnHPdFOX3SWX ZiuVKDVMutPRdezcrijPmYWyk4m0ODp0dt0qhdYvOy6dCv2Yfz577f1+7/XXhyWUD6kIVqtL FfU6dZKKlpPy/hm2FS6zTRNr7QW4dCEWQj/PkWCpqqTBc7cCQeX9LAw9z7fAx6E+BCPNbwgo KfYgKO9oI+B+QwBBneMMDS1dM8EbGqShsbiAhuybVTS87R3F4L96GUOFtB2aLtowuMNfSSjp oaG0JBuPn28YwnYnA/bMpdDpuMbAaEccNAY+UFD3KRrMN/w01NY1ktBQ04mh5ZGFhkDlXwqa Gl6S4LlUSMGdARsNvUN2AuyhQQbeua0YqnPGRWd//KHgRaEbw9lb9zB4fY8RPDnXjkGq/EDD s1AfBpdUTMDv288RdBb1M5B7IcxAaVYRgoLcqyS8GXtBQY4/AUaGLfSmdcKzvkFCyHGdFOqG rKTwysYLD6+1MULOk0+MYJVOCC7HMuFmbQ8WyoMhSpCc52lBCl5mhPx+LxYGXr9mhJemEVLo 8pbgHZH75Os1YpI2TdSv3HBInmjJfYWPOzcZP/vcVCYqWJ2PWJbn4nmfIzUfyabQb2YmmOai +NbWMDHBc7nFvKvwC5WP5CzB5U3nHd+b6YliDmfgzbXdk0xyS3npvQVNsIJbw5uandSUdBFf Ue2eFMnG8+7hosmNkkvg/eYAPbXJk/HdWSumeAH/1NFKXkQKK5rmREqtLi1ZrU2Kj0lM12mN MUdSkiU0/lD206P7a1DQs6secSxSzVC0ZpRrlJQ6zZCeXI94llDNVWR3lGmUCo06/ZSoTzmo P5EkGupRJEuq5itWDZ3UKLmj6lTxmCgeF/X/W8zKIjIRu8FwID6AnIotNYq2DOzLKPM3zTuy M+7zFY81MdUkRZt2Ny1M+WXnI9r7Rz4OZy4aa+nee/C6YzPeeiX51J68sihje2x4Z3qMk1+w cdU209Mu456xB9P2LxngDp+XzfYv/zZryQ1N6ba1QXvFwgSz8YHX/jduZuhWdHVkcAB8OhVp SFTHLSP0BvU/01gjckwDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSW0iTcRjG+/+/o6vV1xL6qKhYWamUSRkvdL4oPyIjgjC6qEb7cKtpsall UWmbHTyhgdp0moeYY1PLzYsOGkvJWqFZmk2ZKy0rc2YHJ5lmqdHNy4/nefhdvSwhK6IWsOq4 eFEbp9DIaQkp2bNRv9phLFOunfAuhpyMteAfvkKC6VYlDa3VNgSVtSkY+h9FwusRH4Kx5ucE 5Oe2Iijt6SagtsmLoN5ykYa297Oh3T9Egys3nQZ9+S0aXgyMY/DkXcNgs0fBs+wyDM7RjyTk 99NQmK/Hk+cThlGzlQFzchD0WgoYGO8JB5e3g4LGIhcF9V2hYCz20FBX7yKh6U4vhrZ7Jhq8 lX8oeNb0hITWnEwKqr6U0TAwYibA7B9i4KWzBMNtw6Tt0o8JCh5nOjFculmDob3zPoIHV95i sFd20NDo92Fw2HMJ+FXxCEFv1iADqRmjDBSmZCFIT80j4fnvxxQYPBEw9tNEb9soNPqGCMHg OCXUj5SQwtMyXrhb0M0IhgddjFBiTxAclhChvK4fC6Xf/ZRgt16lBfv3a4yQNtiOhS8tLYzw 5PoYKbxvz8d7Fx2UbFKKGnWiqA3bckSiMqU+xSet206/63RSySh9XRoKYHluPd/pMTJTTHMr ebd7lJjiQG4p78j8QKUhCUtwl2fylq/N9FQxj9Pxxrq+aSa5IN7+yoSmWMpt4K83W6l/0iW8 7bZzWhQwmff9zJreyLgI3mP00tlIUoJmWFGgOi4xVqHWRKzRHVclxalPrzl6ItaOJp/GfG48 5w4abotsQByL5LOk7rOlShmlSNQlxTYgniXkgVJ9zw2lTKpUJJ0RtScOaxM0oq4BLWRJ+Xzp rmjxiIyLUcSLx0XxpKj932I2YEEyCqrBUed3Hlwabutj5x7rLw+Zs/mQu7H0g63bs6kDr19W PTGDSd2qifn0Jnr5u5dOV7Fq+92qX6uy3bvPLs+qqGhVmLJr5+sLCF/G+WWva+99toUlDP8h V6gv7JjzbdaFQVWYkyncZxmo9sXLWx4eCN5Q4Uzh9s+rCZ6ZEhrsUQ13yUmdShEeQmh1ir8q 62/RMAMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Dept engine works in a constrained environment. For example, Dept cannot make use of dynamic allocation e.g. kmalloc(). So Dept has been using static pools to keep memory chunks Dept uses. However, Dept would barely work once any of the pools gets run out. So implemented a mechanism for the refill on the lack by any chance, using irq work and workqueue that fits on the contrained environment. Signed-off-by: Byungchul Park --- include/linux/dept.h | 19 ++++-- kernel/dependency/dept.c | 104 +++++++++++++++++++++++++++----- kernel/dependency/dept_object.h | 10 +-- kernel/dependency/dept_proc.c | 8 +-- 4 files changed, 112 insertions(+), 29 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index d9ca9dd50219..583e8fe2dd7b 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -336,9 +336,19 @@ struct dept_pool { size_t obj_sz; /* - * the number of the static array + * the remaining number of the object in spool */ - atomic_t obj_nr; + int obj_nr; + + /* + * the number of the object in spool + */ + int tot_nr; + + /* + * accumulated amount of memory used by the object in byte + */ + atomic_t acc_sz; /* * offset of ->pool_node @@ -348,9 +358,10 @@ struct dept_pool { /* * pointer to the pool */ - void *spool; + void *spool; /* static pool */ + void *rpool; /* reserved pool */ struct llist_head boot_pool; - struct llist_head __percpu *lpool; + struct llist_head __percpu *lpool; /* local pool */ }; struct dept_ecxt_held { diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 6cf17f206b78..8454f0a14d67 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -74,6 +74,9 @@ #include #include #include +#include +#include +#include #include "dept_internal.h" static int dept_stop; @@ -122,10 +125,12 @@ static int dept_per_cpu_ready; WARN(1, "DEPT_STOP: " s); \ }) -#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO_ONCE(s...) pr_warn_once("DEPT_INFO_ONCE: " s) +#define DEPT_INFO(s...) pr_warn("DEPT_INFO: " s) static arch_spinlock_t dept_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; static arch_spinlock_t stage_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; +static arch_spinlock_t dept_pool_spin = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; /* * DEPT internal engine should be careful in using outside functions @@ -264,6 +269,7 @@ static inline bool valid_key(struct dept_key *k) #define OBJECT(id, nr) \ static struct dept_##id spool_##id[nr]; \ +static struct dept_##id rpool_##id[nr]; \ static DEFINE_PER_CPU(struct llist_head, lpool_##id); #include "dept_object.h" #undef OBJECT @@ -272,14 +278,70 @@ struct dept_pool dept_pool[OBJECT_NR] = { #define OBJECT(id, nr) { \ .name = #id, \ .obj_sz = sizeof(struct dept_##id), \ - .obj_nr = ATOMIC_INIT(nr), \ + .obj_nr = nr, \ + .tot_nr = nr, \ + .acc_sz = ATOMIC_INIT(sizeof(spool_##id) + sizeof(rpool_##id)), \ .node_off = offsetof(struct dept_##id, pool_node), \ .spool = spool_##id, \ + .rpool = rpool_##id, \ .lpool = &lpool_##id, }, #include "dept_object.h" #undef OBJECT }; +static void dept_wq_work_fn(struct work_struct *work) +{ + int i; + + for (i = 0; i < OBJECT_NR; i++) { + struct dept_pool *p = dept_pool + i; + int sz = p->tot_nr * p->obj_sz; + void *rpool; + bool need; + + arch_spin_lock(&dept_pool_spin); + need = !p->rpool; + arch_spin_unlock(&dept_pool_spin); + + if (!need) + continue; + + rpool = vmalloc(sz); + + if (!rpool) { + DEPT_STOP("Failed to extend internal resources.\n"); + break; + } + + arch_spin_lock(&dept_pool_spin); + if (!p->rpool) { + p->rpool = rpool; + rpool = NULL; + atomic_add(sz, &p->acc_sz); + } + arch_spin_unlock(&dept_pool_spin); + + if (rpool) + vfree(rpool); + else + DEPT_INFO("Dept object(%s) just got refilled successfully.\n", p->name); + } +} + +static DECLARE_WORK(dept_wq_work, dept_wq_work_fn); + +static void dept_irq_work_fn(struct irq_work *w) +{ + schedule_work(&dept_wq_work); +} + +static DEFINE_IRQ_WORK(dept_irq_work, dept_irq_work_fn); + +static void request_rpool_refill(void) +{ + irq_work_queue(&dept_irq_work); +} + /* * Can use llist no matter whether CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG is * enabled or not because NMI and other contexts in the same CPU never @@ -315,19 +377,31 @@ static void *from_pool(enum object_t t) /* * Try static pool. */ - if (atomic_read(&p->obj_nr) > 0) { - int idx = atomic_dec_return(&p->obj_nr); + arch_spin_lock(&dept_pool_spin); + + if (!p->obj_nr) { + p->spool = p->rpool; + p->obj_nr = p->rpool ? p->tot_nr : 0; + p->rpool = NULL; + request_rpool_refill(); + } + + if (p->obj_nr) { + void *ret; + + p->obj_nr--; + ret = p->spool + (p->obj_nr * p->obj_sz); + arch_spin_unlock(&dept_pool_spin); - if (idx >= 0) - return p->spool + (idx * p->obj_sz); + return ret; } + arch_spin_unlock(&dept_pool_spin); - DEPT_INFO_ONCE("---------------------------------------------\n" - " Some of Dept internal resources are run out.\n" - " Dept might still work if the resources get freed.\n" - " However, the chances are Dept will suffer from\n" - " the lack from now. Needs to extend the internal\n" - " resource pools. Ask max.byungchul.park@gmail.com\n"); + DEPT_INFO("------------------------------------------\n" + " Dept object(%s) is run out.\n" + " Dept is trying to refill the object.\n" + " Nevertheless, if it fails, Dept will stop.\n", + p->name); return NULL; } @@ -3000,8 +3074,8 @@ void __init dept_init(void) pr_info("... DEPT_MAX_ECXT_HELD : %d\n", DEPT_MAX_ECXT_HELD); pr_info("... DEPT_MAX_SUBCLASSES : %d\n", DEPT_MAX_SUBCLASSES); #define OBJECT(id, nr) \ - pr_info("... memory used by %s: %zu KB\n", \ - #id, B2KB(sizeof(struct dept_##id) * nr)); + pr_info("... memory initially used by %s: %zu KB\n", \ + #id, B2KB(sizeof(spool_##id) + sizeof(rpool_##id))); #include "dept_object.h" #undef OBJECT #define HASH(id, bits) \ @@ -3009,6 +3083,6 @@ void __init dept_init(void) #id, B2KB(sizeof(struct hlist_head) * (1 << (bits)))); #include "dept_hash.h" #undef HASH - pr_info("... total memory used by objects and hashs: %zu KB\n", B2KB(mem_total)); + pr_info("... total memory initially used by objects and hashs: %zu KB\n", B2KB(mem_total)); pr_info("... per task memory footprint: %zu bytes\n", sizeof(struct dept_task)); } diff --git a/kernel/dependency/dept_object.h b/kernel/dependency/dept_object.h index 0b7eb16fe9fb..4f936adfa8ee 100644 --- a/kernel/dependency/dept_object.h +++ b/kernel/dependency/dept_object.h @@ -6,8 +6,8 @@ * nr: # of the object that should be kept in the pool. */ -OBJECT(dep, 1024 * 8) -OBJECT(class, 1024 * 8) -OBJECT(stack, 1024 * 32) -OBJECT(ecxt, 1024 * 16) -OBJECT(wait, 1024 * 32) +OBJECT(dep, 1024 * 4 * 2) +OBJECT(class, 1024 * 4) +OBJECT(stack, 1024 * 4 * 8) +OBJECT(ecxt, 1024 * 4 * 2) +OBJECT(wait, 1024 * 4 * 4) diff --git a/kernel/dependency/dept_proc.c b/kernel/dependency/dept_proc.c index 7d61dfbc5865..f07a512b203f 100644 --- a/kernel/dependency/dept_proc.c +++ b/kernel/dependency/dept_proc.c @@ -73,12 +73,10 @@ static int dept_stats_show(struct seq_file *m, void *v) { int r; - seq_puts(m, "Availability in the static pools:\n\n"); + seq_puts(m, "Accumulated amount of memory used by pools:\n\n"); #define OBJECT(id, nr) \ - r = atomic_read(&dept_pool[OBJECT_##id].obj_nr); \ - if (r < 0) \ - r = 0; \ - seq_printf(m, "%s\t%d/%d(%d%%)\n", #id, r, nr, (r * 100) / (nr)); + r = atomic_read(&dept_pool[OBJECT_##id].acc_sz); \ + seq_printf(m, "%s\t%d KB\n", #id, r / 1024); #include "dept_object.h" #undef OBJECT From patchwork Mon Jul 3 09:47:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92527C10F1A for ; Mon, 3 Jul 2023 09:50:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231558AbjGCJuC (ORCPT ); Mon, 3 Jul 2023 05:50:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231313AbjGCJtx (ORCPT ); Mon, 3 Jul 2023 05:49:53 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 450B11B4; Mon, 3 Jul 2023 02:49:50 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-c7-64a299b31a63 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 15/25] locking/lockdep, cpu/hotplus: Use a weaker annotation in AP thread Date: Mon, 3 Jul 2023 18:47:42 +0900 Message-Id: <20230703094752.79269-16-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiG977nnPecVstOKtuO8GOmC1ExIi66PDFmIdmSnS0xMfGPw2Ta rCejSkGLFtlHhit+lUKACB20uFKW0rRVpMVEhpAKEUSDVCRYK54Ic04CCGEWrYDaYvbnyZX7 yX39ujlK3cGkcfqCo5KxQJuvIUpaObO6aXOw3qXLDg3uhGprNsSen6HB0eonEL7oQ+BvP4Fh 8vpXcG9hGsHi4BAFttowgqbxhxS098kIujy/Ebj7OAVGYrMEBmrLCZibWwncmVrCMFZXg8EX 2AW3qlwYQvF/abBNErDbzDhxnmKIu70suEszYMLTwMLS+FYYkEcZ6IpugvrzYwSudg3Q0Hdl AsPdvxwEZP8bBm713aAhXF3BwIVnLgJTC24K3LFZFoZDTgyXyhKiU/+9ZqC/IoTh1J9tGEbu dyLoPvMIQ8A/SqA3No0hGKil4FXLdQQTlTMsnLTGWbCfqERQfrKOhqHlfgbKxrbD4ksHydkh 9k7PUmJZsFjsWnDS4k2XIHY0PGTFsu4oKzoDx8SgJ1NsvjqJxab5GCMGvGeJGJivYUXLzAgW n92+zYo3fl+kxccjNrw7PVe5Uyfl602SccvnB5R55+4v48Mt7PG5v+1sKaogFqTgBH6bMHth nLUgboV9tRnJmPDrhUgkTiU5lV8nBCueMBak5Cj+9CrBMze40l3DG4TuHplNMs1nCJGhy3TS o+I/E2Rr8Tv9x4LvUmjFo0jE/7ysRElW89uFsXqZJJ0Cb1MInc9b6HeFtcI1T4SuQiones+L 1PoCk0Grz9+WlVdSoD+e9X2hIYASg3L/srTvCpoP7+lBPIc0q1WRn5p0akZrKiox9CCBozSp KvP4Hzq1Sqct+VEyFu43HsuXinpQOkdrPlJ9ulCsU/M/aI9KhyTpsGT8/4s5RVopMqTlmuJr yZeJfSJriRMrvrP79fIna6yNc42p/aacnO4tv0r727znpuOtma/C5lyU5Z4x762L2pt7L0fT czsf1ES+FvZ0TA0PV73/hepnOdr25MNIzr3O0UJux67MNy5HytN9siPF98GEu/Hshm+WU1+0 H2l4UV3uvWjZmH3wWw1dlKfdmkkZi7RvAbLwAENMAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTVxzGd86999xLpdu1ot6oi1sTMuMrGDH/6TTGLx4xM/PDMsdisNqb 0QDFtFBlZglIdVopAxQRClpgqw2wqbea4EYJQijiW5kQrKY2wpxIADHONnS8OGrilye/PE/y +/QIjMbBLREMxlzZZNRlaYmKVe3eXLTGU1WvT6qNJEJZcRKE35xkoeZyM4He35sQNF8rxDDS tQMeRsYQTN3zM1BZ0YugbvAJA9d8IQRe9zECfc8+hP7wBIGeitMEihouE/hrdBpD8Fw5hibl S7hTWo+hPTrMQuUIAUdlEZ6LFxiirkYeXAWJMOSu5mF6MBl6QgMcdNb2cOB9vAqqLgQJtHp7 WPC1DGHo+6OGQKj5LQd3fLdY6C2zc/Dby3oCoxEXA67wBA8P2p0YrljnbCf+neWg296O4cQv VzH0P/oTQdvJpxiU5gECneExDB6lgoH/LnUhGCoZ5+F4cZQHR2EJgtPHz7Hgn+nmwBpMganJ GrJtM+0cm2Co1XOYeiNOlt6ul+iN6ic8tbY95qlTyaMe90ra0DqCad3rMEeVxlOEKq/LeWob 78f05f37PL11foqlz/or8VfL0lRf6OUsg0U2rdu6X5Vx9tEMPnSJP/LqbwdfgOzEhgRBEjdI TRWJNhQnEPEzKRCIMjFOED+RPPbnnA2pBEb8aZ7kfnWPxIYFYrbU1hHiY8yKiVLAf52NedTi RilUfDhWS+JyqelK+ztP3Fz9z2QJirFGTJGCVSFSilRO9EEjSjAYLdk6Q1bKWnNmRr7RcGTt wZxsBc1dxvXjdFkLetO3owOJAtLGqwNH6/QaTmcx52d3IElgtAnqosGLeo1ar8v/QTblpJvy smRzB1oqsNrF6tRv5P0a8Xtdrpwpy4dk0/sVC3FLChD1tu1ar3y6rHyvQXc0krZgZsO6uwdW vZ1dfSHV0lKV617jvxmfvzC9c6ez++riYJT2LWrYRSw+33rH7i3eyfmLbGc2/nw7fWCFdbvS OuEfNQ+b6/bZk6ozC7+bOrb329T4j2a6lNk91w8Y0j42GEu/9teO5wU+T/71zKac4YKbs3d9 WtacoUteyZjMuv8Bvvgvmy4DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org cb92173d1f0 ("locking/lockdep, cpu/hotplug: Annotate AP thread") was introduced to make lockdep_assert_cpus_held() work in AP thread. However, the annotation is too strong for that purpose. We don't have to use more than try lock annotation for that. Furthermore, now that Dept was introduced, false positive alarms was reported by that. Replaced it with try lock annotation. Signed-off-by: Byungchul Park --- kernel/cpu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cpu.c b/kernel/cpu.c index f4a2c5845bcb..19076f798b34 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -356,7 +356,7 @@ int lockdep_is_cpus_held(void) static void lockdep_acquire_cpus_lock(void) { - rwsem_acquire(&cpu_hotplug_lock.dep_map, 0, 0, _THIS_IP_); + rwsem_acquire(&cpu_hotplug_lock.dep_map, 0, 1, _THIS_IP_); } static void lockdep_release_cpus_lock(void) From patchwork Mon Jul 3 09:47:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2AF3C04FDF for ; Mon, 3 Jul 2023 09:50:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231593AbjGCJuD (ORCPT ); Mon, 3 Jul 2023 05:50:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231318AbjGCJtx (ORCPT ); Mon, 3 Jul 2023 05:49:53 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 921E6DD; Mon, 3 Jul 2023 02:49:51 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-d8-64a299b373f9 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 16/25] dept: Apply sdt_might_sleep_{start,end}() to dma fence wait Date: Mon, 3 Jul 2023 18:47:43 +0900 Message-Id: <20230703094752.79269-17-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0xTZxiA/c7lO6fF4rFj8agRtQkhYQx1EfOG6eKPRY8/nCYaY1QinT0b Hde1grLNDCyIk4uIA+QSbSuptRQvbVVEaxiEe4BuNAy1koHoRrhtaDsK6NZi/PPmyfPmfX69 LCl/QK9i1anHRU2qMlmBpZR0cqnhY0elUbVx6g0DFwo3gvf1WQpqbloxuG7UIbA6cggYa90J v/smEMz39JFQUeZCYBh+RoKjbQiB03waQ/9oKLi90xg6ywow6K7exPDr+AIBnvJSAupsu6G7 xEhAk/9PCirGMFRX6IjA+IsAv8nCgCk7AkbMVQwsDG+CzqEBGpxPPoLKyx4MD52dFLQ1jBDQ 31iDYcj6Hw3dbR0UuC4U0VA/ZcQw7jORYPJOM/Bbk56AW7mB0JlXb2loL2oi4EztbQLcjx8g eHT2DwJs1gEMLd4JAuy2MhLmrrUiGCmeZCCv0M9AdU4xgoK8cgr63rTTkOuJhfnZGrw9TmiZ mCaFXPsJwenTU0KXkRfuVz1jhNxHTxhBb8sQ7OYo4erDMUIwzHhpwWb5CQu2mVJGODfpJoSp 3l5G6Lg0Twmj7gpi7+pD0q0qMVmdKWo2fJYgTbzo78XpObKTenstk41eSM8hCctzm3nd09Pk ey7/10gHGXOR/OCgf9GHcet4e9HLgJeyJJcfwpv/7sHBxQfcV7y3twsFmeIieEfDDBNkGbeF 1+sG0LvoWr7uVtNiSBLwL2aLF72ci+U9lUM4GOW58xI+P3scvztYyf9iHqRKkEyPlliQXJ2a maJUJ2+OScxKVZ+MOZaWYkOBlzKdWjjcgGZc+5oRxyLFUtng9waVnFZmarNSmhHPkoowmW74 ikouUymzvhM1aUc1GcmithmtZinFCtknvhMqOfe18riYJIrpoub9lmAlq7JRfNXnU4U6fCDB clkVWlKgPPLPhth78ykHo5e1pW2fK+/pn92fEREZF8nGa/uS7nx599tpSyPVGa0NOSWJ3/Gy fu1ceFfo8lKHZK+zbNl59GO1Yc22n58794S76VHaQlvrLr7u/vCbcCouP/6HhI6IT9eHvK3N W+e7vsulv/RFjGfNcwWlTVRuiiI1WuX/9KM2Pk4DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0xTZxjH977nnPccqjUnFeMJuqDN1MSFWyL6bKjB7IMni87LF40ukWoP 0liqaSmXMRJYq3PcIizAuGkppDbQCWshQaENQkBBRByMMVORNjhGRFmYZVaQrXXZlye//P7J 79PDUYp6JorT6DIkvU6lVRIZLfsiyRTjqraq44uaYqCsOB4Cr6/SUNfqIDB6qwWBo70Aw1z/ Qfh1aR7B8sNHFFRVjCJo8D2loH1gCoHb/g2BsZl1MB5YIDBYUUTA1NhK4PGLFQzeynIMLc7D 8OCaFUNPcJaGqjkCtVUmHDp/YAjamlmw5W8Dv72GhRVfAgxOTTDQVz/IgPvJx1B93Uug2z1I w0CnH8PYnToCU45/GHgwcJ+G0bISBn58ZSXwYslGgS2wwMLPPRYMbeZQ7cpfqwzcK+nBcKXp Jwzjv3Uh8FydxuB0TBDoC8xjcDkrKHh7sx+Bv/QlC5eLgyzUFpQiKLpcScOjd/cYMHsTYflN HUlOEvvmFyjR7MoS3UsWWhyyCuLtmqesaPY8YUWL0yi67DvFxu45LDYsBhjR2fwdEZ2L5axY +HIci69GRljx/g/LtDgzXoWPbj4l26uWtJpMSR+3P0WW9n1whFwqkGdbXE1sPnouK0QRnMDv Eir/tjJhJvwOYXIySIU5kt8iuEp+D3kZR/HfrhHsfz4k4WE9nyoERoZQmGl+m9DeuciGWc7v FiymCfRfNFpoaet5H4oI+edvSt97BZ8oeKunyDUks6APmlGkRpeZrtJoE2MNF9JydJrs2HMX 050o9DS2vJWyTvR67GAv4jmkXCufzG1QKxhVpiEnvRcJHKWMlJt8N9QKuVqV85Wkv3hGb9RK hl60iaOVG+Wfn5BSFPx5VYZ0QZIuSfr/V8xFROUjpWfN9gPDQnRGF702d3R34UqW23ynaN/M jo67v3gOGdcplnxnU0H+2RZNOXNow9c+fzD32JcdW5OFuGfyJHvjsN8xFj1bU2Jp+TRqOm/2 wEd743GdV5F6OsL4iXG140NOtmmotWt/sX86RkeOr1fmbVe7N54sPlJL1fv2eLTJq0rakKZK 2EnpDap/ATXR0DQwAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Makes Dept able to track dma fence waits. Signed-off-by: Byungchul Park --- drivers/dma-buf/dma-fence.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index f177c56269bb..ad2d7a94c868 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -16,6 +16,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -782,6 +783,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) cb.task = current; list_add(&cb.base.node, &fence->cb_list); + sdt_might_sleep_start(NULL); while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) { if (intr) __set_current_state(TASK_INTERRUPTIBLE); @@ -795,6 +797,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) if (ret > 0 && intr && signal_pending(current)) ret = -ERESTARTSYS; } + sdt_might_sleep_end(); if (!list_empty(&cb.base.node)) list_del(&cb.base.node); @@ -884,6 +887,7 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count, } } + sdt_might_sleep_start(NULL); while (ret > 0) { if (intr) set_current_state(TASK_INTERRUPTIBLE); @@ -898,6 +902,7 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count, if (ret > 0 && intr && signal_pending(current)) ret = -ERESTARTSYS; } + sdt_might_sleep_end(); __set_current_state(TASK_RUNNING); From patchwork Mon Jul 3 09:47:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D1B8C25B4F for ; Mon, 3 Jul 2023 09:50:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231294AbjGCJuG (ORCPT ); Mon, 3 Jul 2023 05:50:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231351AbjGCJtz (ORCPT ); Mon, 3 Jul 2023 05:49:55 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E881B12C; Mon, 3 Jul 2023 02:49:51 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-e9-64a299b3ea2e From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 17/25] dept: Track timeout waits separately with a new Kconfig Date: Mon, 3 Jul 2023 18:47:44 +0900 Message-Id: <20230703094752.79269-18-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSW0yTdxjG9/9/x1a7fOkwfmqMrgtZhplnlzdGF8Mu9l24hMSLLfMCO/pZ qqWQIig6I9jioQoqWhAh0lKtlcPE0gs2qaFVTlahKiKaUtfGqYRCXbc2YrsDZePmzS/Pk+d3 9bKE/Ba1lNXo9ol6nVKroKWkdGqh9XNXfbNqbWCYg3On10L8zxMkNN5oo8H/UyuCNlcFhone r+FpIoIg+WCYgDqzH4E1NE6Aqy+IwO04SsPjlx/CSDxKw6D5FA0G2w0aHk6mMARqazC0Or8B 39lmDD0zr0mom6Choc6AZ88bDDP2Fgbs5ZkQdlxiIBVaB4PBUQrcz1dB/eUADd3uQRL6usIY Hv/SSEOw7R8KfH0DJPjPVVHQPt1Mw2TCToA9HmXgUY8FQ4dxVnTsj78p6K/qwXDsyk0MI89u Ibh94lcMzrZRGu7EIxg6nWYC3l/rRRCunmKg8vQMAw0V1QhOVdaSMPxXPwXGwCZIvmukt20W 7kSihGDs3C+4ExZSuNfMCz9fGmcE4+3njGBxlgidjizB1j2BBWssTgnOlpO04IzVMIJpagQL 00NDjDBwMUkKL0fqcM6y76VbVKJWUyrq13y5S5rfPm4kimK7Dnh7r+Jy5NluQhKW5zbyqUkr M8+2yvdkmmnuU35sbIZIcwa3ku+sekWZkJQluOMLeMfbB3S6+Ij7gW8d7Z4bkFwm31FrptIs 477gfYYa6j/pCr61o2dOJJnNf3tXjdIs5zbxgfognZby3HkJb0vG/x8s4T2OMfIsklnQBy1I rtGVFig12o2r88t0mgOr8woLnGj2peyHUzu7UMy/w4s4FikWysYOWVVySllaXFbgRTxLKDJk hlCTSi5TKcsOivrCXH2JViz2omUsqVgsW5/Yr5JzauU+ca8oFon6+RazkqXl6Kur30XyXJ4n o0xTqKhxc9Fyj7rw+CKf2/Tkk+i39/y7g9lo/Pd+t7r0vuXuZ1vJrS4De+ZVuMswVBCLXtgb fJE3dcZz8nqFy6dNGvtqSw5fyx2I/FhydMFlU8A2dJ05sjy8bcOeVXrJtD+lVmdLDppX6jJN GYmQt4H6OHt9ziMFWZyvXJdF6IuV/wJlNwa9TgMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTZxjG933nnO8c6rqcVDJPnMtMF2OikUti8Y0as5CpXxYxms2waKKt 9kQ6oGirDDQuYAs6bgESqChMKNo1gAqtGi+gDOTSEQSFAJLSDHRT7ku1xALTUZf98+aX58nz ++sVGFUFt1IwGE/IJqMuSU0UrGL3FssGd5ldH9Xh00BRXhQE3pxnofxGHYHe67UI6m5mYhhv 2wmDc1MIFrp7GLCV9CKoGh1h4Ga7D0GT8yyBvhefQH9gloCnJJeApfoGgSeTixi8pcUYal1x 0FVox9AcfMmCbZzAJZsFL51XGIKOGh4cGWtgzHmRh8XRaPD4BjhorfBw0DS8Hsp+8RJobPKw 0H5nDEPfvXICvrr3HHS1d7LQW5TPwbUZO4HJOQcDjsAsD0+bKzHUW5ds2a/fcdCR34wh+0oD hv5n9xE8OP8HBlfdAIHWwBQGt6uEgflf2xCMFUzzkJUX5OFSZgGC3KxSFnr+6eDA6tXAwtty 8tUW2jo1y1Cr+0faNFfJ0t/tEr17cYSn1gfDPK10naRu5zpa3TiOaZU/wFFXzc+EuvzFPM2Z 7sd05vFjnnZeWGDpi34b3rNqv2KrXk4ypMqmyG1aRcK1EStzzK9Na2m7ijPQb7tyUJggiRul 6qx5NsREXCsNDQWZEIeLqyV3/l9cDlIIjHhumeT8u5uEiuXiYal2oPHDgBXXSPWlJVyIlWKM 1GUp5v6TfiHV1jd/EIUt5X++LUAhVokayVvmI4VIUYk+qkHhBmNqss6QpIkwJyakGw1pEUdS kl1o6WkcZxaL7qA3fTtbkCgg9cfKodNVehWnSzWnJ7cgSWDU4UrL6GW9SqnXpZ+STSmHTCeT ZHML+kxg1SuU38TLWpV4VHdCTpTlY7Lp/xYLYSszUGzULU3etpw0c8Vt/TO72uccSTyifd3z ufUHf/yjr89m5jbk71qMHds3eXTe/2SPZyFih/bL+DgyLNimJ+jE8TNR6slVE4WJl7O3f7oJ DX7bp4iEzZGtBwV+uyOWPvcyir3fLRu6ih5+H3elIfzchL77/YWY3bbggbWpm4wzP+0fVLPm BF30OsZk1v0LT6YcmDADAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Waits with valid timeouts don't actually cause deadlocks. However, Dept has been reporting the cases as well because it's worth informing the circular dependency for some cases where, for example, timeout is used to avoid a deadlock but not meant to be expired. However, yes, there are also a lot of, even more, cases where timeout is used for its clear purpose and meant to be expired. Let Dept report these as an information rather than shouting DEADLOCK. Plus, introduced CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT Kconfig to make it optional so that any reports involving waits with timeouts can be turned on/off depending on the purpose. Signed-off-by: Byungchul Park --- include/linux/dept.h | 15 ++++++--- include/linux/dept_ldt.h | 6 ++-- include/linux/dept_sdt.h | 12 +++++--- kernel/dependency/dept.c | 66 ++++++++++++++++++++++++++++++++++------ lib/Kconfig.debug | 10 ++++++ 5 files changed, 89 insertions(+), 20 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index 583e8fe2dd7b..0aa8d90558a9 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -270,6 +270,11 @@ struct dept_wait { * whether this wait is for commit in scheduler */ bool sched_sleep; + + /* + * whether a timeout is set + */ + bool timeout; }; }; }; @@ -458,6 +463,7 @@ struct dept_task { bool stage_sched_map; const char *stage_w_fn; unsigned long stage_ip; + bool stage_timeout; /* * the number of missing ecxts @@ -496,6 +502,7 @@ struct dept_task { .stage_sched_map = false, \ .stage_w_fn = NULL, \ .stage_ip = 0UL, \ + .stage_timeout = false, \ .missing_ecxt = 0, \ .hardirqs_enabled = false, \ .softirqs_enabled = false, \ @@ -513,8 +520,8 @@ extern void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u, con extern void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, const char *n); extern void dept_map_copy(struct dept_map *to, struct dept_map *from); -extern void dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l); -extern void dept_stage_wait(struct dept_map *m, struct dept_key *k, unsigned long ip, const char *w_fn); +extern void dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l, long timeout); +extern void dept_stage_wait(struct dept_map *m, struct dept_key *k, unsigned long ip, const char *w_fn, long timeout); extern void dept_request_event_wait_commit(void); extern void dept_clean_stage(void); extern void dept_stage_event(struct task_struct *t, unsigned long ip); @@ -566,8 +573,8 @@ struct dept_task { }; #define dept_map_reinit(m, k, su, n) do { (void)(n); (void)(k); } while (0) #define dept_map_copy(t, f) do { } while (0) -#define dept_wait(m, w_f, ip, w_fn, sl) do { (void)(w_fn); } while (0) -#define dept_stage_wait(m, k, ip, w_fn) do { (void)(k); (void)(w_fn); } while (0) +#define dept_wait(m, w_f, ip, w_fn, sl, t) do { (void)(w_fn); } while (0) +#define dept_stage_wait(m, k, ip, w_fn, t) do { (void)(k); (void)(w_fn); } while (0) #define dept_request_event_wait_commit() do { } while (0) #define dept_clean_stage() do { } while (0) #define dept_stage_event(t, ip) do { } while (0) diff --git a/include/linux/dept_ldt.h b/include/linux/dept_ldt.h index 062613e89fc3..8adf298dfcb8 100644 --- a/include/linux/dept_ldt.h +++ b/include/linux/dept_ldt.h @@ -27,7 +27,7 @@ else if (t) \ dept_ecxt_enter(m, LDT_EVT_L, i, "trylock", "unlock", sl);\ else { \ - dept_wait(m, LDT_EVT_L, i, "lock", sl); \ + dept_wait(m, LDT_EVT_L, i, "lock", sl, false); \ dept_ecxt_enter(m, LDT_EVT_L, i, "lock", "unlock", sl);\ } \ } while (0) @@ -39,7 +39,7 @@ else if (t) \ dept_ecxt_enter(m, LDT_EVT_R, i, "read_trylock", "read_unlock", sl);\ else { \ - dept_wait(m, q ? LDT_EVT_RW : LDT_EVT_W, i, "read_lock", sl);\ + dept_wait(m, q ? LDT_EVT_RW : LDT_EVT_W, i, "read_lock", sl, false);\ dept_ecxt_enter(m, LDT_EVT_R, i, "read_lock", "read_unlock", sl);\ } \ } while (0) @@ -51,7 +51,7 @@ else if (t) \ dept_ecxt_enter(m, LDT_EVT_W, i, "write_trylock", "write_unlock", sl);\ else { \ - dept_wait(m, LDT_EVT_RW, i, "write_lock", sl); \ + dept_wait(m, LDT_EVT_RW, i, "write_lock", sl, false);\ dept_ecxt_enter(m, LDT_EVT_W, i, "write_lock", "write_unlock", sl);\ } \ } while (0) diff --git a/include/linux/dept_sdt.h b/include/linux/dept_sdt.h index 12a793b90c7e..21fce525f031 100644 --- a/include/linux/dept_sdt.h +++ b/include/linux/dept_sdt.h @@ -22,11 +22,12 @@ #define sdt_map_init_key(m, k) dept_map_init(m, k, 0, #m) -#define sdt_wait(m) \ +#define sdt_wait_timeout(m, t) \ do { \ dept_request_event(m); \ - dept_wait(m, 1UL, _THIS_IP_, __func__, 0); \ + dept_wait(m, 1UL, _THIS_IP_, __func__, 0, t); \ } while (0) +#define sdt_wait(m) sdt_wait_timeout(m, -1L) /* * sdt_might_sleep() and its family will be committed in __schedule() @@ -37,12 +38,13 @@ /* * Use the code location as the class key if an explicit map is not used. */ -#define sdt_might_sleep_start(m) \ +#define sdt_might_sleep_start_timeout(m, t) \ do { \ struct dept_map *__m = m; \ static struct dept_key __key; \ - dept_stage_wait(__m, __m ? NULL : &__key, _THIS_IP_, __func__);\ + dept_stage_wait(__m, __m ? NULL : &__key, _THIS_IP_, __func__, t);\ } while (0) +#define sdt_might_sleep_start(m) sdt_might_sleep_start_timeout(m, -1L) #define sdt_might_sleep_end() dept_clean_stage() @@ -52,7 +54,9 @@ #else /* !CONFIG_DEPT */ #define sdt_map_init(m) do { } while (0) #define sdt_map_init_key(m, k) do { (void)(k); } while (0) +#define sdt_wait_timeout(m, t) do { } while (0) #define sdt_wait(m) do { } while (0) +#define sdt_might_sleep_start_timeout(m, t) do { } while (0) #define sdt_might_sleep_start(m) do { } while (0) #define sdt_might_sleep_end() do { } while (0) #define sdt_ecxt_enter(m) do { } while (0) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 8454f0a14d67..52537c099b68 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -740,6 +740,8 @@ static void print_diagram(struct dept_dep *d) if (!irqf) { print_spc(spc, "[S] %s(%s:%d)\n", c_fn, fc_n, fc->sub_id); print_spc(spc, "[W] %s(%s:%d)\n", w_fn, tc_n, tc->sub_id); + if (w->timeout) + print_spc(spc, "--------------- >8 timeout ---------------\n"); print_spc(spc, "[E] %s(%s:%d)\n", e_fn, fc_n, fc->sub_id); } } @@ -793,6 +795,24 @@ static void print_dep(struct dept_dep *d) static void save_current_stack(int skip); +static bool is_timeout_wait_circle(struct dept_class *c) +{ + struct dept_class *fc = c->bfs_parent; + struct dept_class *tc = c; + + do { + struct dept_dep *d = lookup_dep(fc, tc); + + if (d->wait->timeout) + return true; + + tc = fc; + fc = fc->bfs_parent; + } while (tc != c); + + return false; +} + /* * Print all classes in a circle. */ @@ -815,10 +835,14 @@ static void print_circle(struct dept_class *c) pr_warn("summary\n"); pr_warn("---------------------------------------------------\n"); - if (fc == tc) + if (is_timeout_wait_circle(c)) { + pr_warn("NOT A DEADLOCK BUT A CIRCULAR DEPENDENCY\n"); + pr_warn("CHECK IF THE TIMEOUT IS INTENDED\n\n"); + } else if (fc == tc) { pr_warn("*** AA DEADLOCK ***\n\n"); - else + } else { pr_warn("*** DEADLOCK ***\n\n"); + } i = 0; do { @@ -1564,7 +1588,8 @@ static void add_dep(struct dept_ecxt *e, struct dept_wait *w) static atomic_t wgen = ATOMIC_INIT(1); static void add_wait(struct dept_class *c, unsigned long ip, - const char *w_fn, int sub_l, bool sched_sleep) + const char *w_fn, int sub_l, bool sched_sleep, + bool timeout) { struct dept_task *dt = dept_task(); struct dept_wait *w; @@ -1584,6 +1609,7 @@ static void add_wait(struct dept_class *c, unsigned long ip, w->wait_fn = w_fn; w->wait_stack = get_current_stack(); w->sched_sleep = sched_sleep; + w->timeout = timeout; cxt = cur_cxt(); if (cxt == DEPT_CXT_HIRQ || cxt == DEPT_CXT_SIRQ) @@ -2338,7 +2364,7 @@ static struct dept_class *check_new_class(struct dept_key *local, */ static void __dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l, - bool sched_sleep, bool sched_map) + bool sched_sleep, bool sched_map, bool timeout) { int e; @@ -2361,7 +2387,7 @@ static void __dept_wait(struct dept_map *m, unsigned long w_f, if (!c) continue; - add_wait(c, ip, w_fn, sub_l, sched_sleep); + add_wait(c, ip, w_fn, sub_l, sched_sleep, timeout); } } @@ -2403,14 +2429,23 @@ static void __dept_event(struct dept_map *m, unsigned long e_f, } void dept_wait(struct dept_map *m, unsigned long w_f, - unsigned long ip, const char *w_fn, int sub_l) + unsigned long ip, const char *w_fn, int sub_l, + long timeoutval) { struct dept_task *dt = dept_task(); unsigned long flags; + bool timeout; if (unlikely(!dept_working())) return; + timeout = timeoutval > 0 && timeoutval < MAX_SCHEDULE_TIMEOUT; + +#if !defined(CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT) + if (timeout) + return; +#endif + if (dt->recursive) return; @@ -2419,21 +2454,30 @@ void dept_wait(struct dept_map *m, unsigned long w_f, flags = dept_enter(); - __dept_wait(m, w_f, ip, w_fn, sub_l, false, false); + __dept_wait(m, w_f, ip, w_fn, sub_l, false, false, timeout); dept_exit(flags); } EXPORT_SYMBOL_GPL(dept_wait); void dept_stage_wait(struct dept_map *m, struct dept_key *k, - unsigned long ip, const char *w_fn) + unsigned long ip, const char *w_fn, + long timeoutval) { struct dept_task *dt = dept_task(); unsigned long flags; + bool timeout; if (unlikely(!dept_working())) return; + timeout = timeoutval > 0 && timeoutval < MAX_SCHEDULE_TIMEOUT; + +#if !defined(CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT) + if (timeout) + return; +#endif + if (m && m->nocheck) return; @@ -2481,6 +2525,7 @@ void dept_stage_wait(struct dept_map *m, struct dept_key *k, dt->stage_w_fn = w_fn; dt->stage_ip = ip; + dt->stage_timeout = timeout; unlock: arch_spin_unlock(&stage_spin); @@ -2506,6 +2551,7 @@ void dept_clean_stage(void) dt->stage_sched_map = false; dt->stage_w_fn = NULL; dt->stage_ip = 0UL; + dt->stage_timeout = false; arch_spin_unlock(&stage_spin); dept_exit_recursive(flags); @@ -2523,6 +2569,7 @@ void dept_request_event_wait_commit(void) unsigned long ip; const char *w_fn; bool sched_map; + bool timeout; if (unlikely(!dept_working())) return; @@ -2545,6 +2592,7 @@ void dept_request_event_wait_commit(void) w_fn = dt->stage_w_fn; ip = dt->stage_ip; sched_map = dt->stage_sched_map; + timeout = dt->stage_timeout; /* * Avoid zero wgen. @@ -2552,7 +2600,7 @@ void dept_request_event_wait_commit(void) wg = atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen); WRITE_ONCE(dt->stage_m.wgen, wg); - __dept_wait(&dt->stage_m, 1UL, ip, w_fn, 0, true, sched_map); + __dept_wait(&dt->stage_m, 1UL, ip, w_fn, 0, true, sched_map, timeout); exit: dept_exit(flags); } diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index aa62caa4dc14..f78b3d721a2b 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1234,6 +1234,16 @@ config DEPT noting, to mitigate the impact by the false positives, multi reporting has been supported. +config DEPT_AGGRESSIVE_TIMEOUT_WAIT + bool "Aggressively track even timeout waits" + depends on DEPT + default n + help + Timeout wait doesn't contribute to a deadlock. However, + informing a circular dependency might be helpful for cases + that timeout is used to avoid a deadlock. Say N if you'd like + to avoid verbose reports. + config LOCK_DEBUGGING_SUPPORT bool depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT From patchwork Mon Jul 3 09:47:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEBCFC04FE2 for ; Mon, 3 Jul 2023 09:50:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231391AbjGCJuJ (ORCPT ); Mon, 3 Jul 2023 05:50:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231359AbjGCJtz (ORCPT ); Mon, 3 Jul 2023 05:49:55 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id ECF491B2; Mon, 3 Jul 2023 02:49:52 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-fa-64a299b4f969 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 18/25] dept: Apply timeout consideration to wait_for_completion()/complete() Date: Mon, 3 Jul 2023 18:47:45 +0900 Message-Id: <20230703094752.79269-19-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTHfZ5773NvO+tuqomPkOyl0ZCwqWhgOVsWY5ZsPh9mMFkMifuw dfZmVKE0RRA2t/AmKgoIETqQOaimNsBAbtlSpsUKWmAEZKNxaGgHzM2x8qK4Mhkooyx+Ofnn nN//9+lInN4jxEhmyxHFZjGmGYiW106vdWxtr3GYEh4FNkLFmQSI/H2Sh7rWZgJDLU0Imtvz MUze2gO/zE8hWBy4zYG9aghBw3iQg3Z/CIHXVUBg+P46CERmCfRVnSZQeLGVwE/hJQyj1ZUY mtS90H/WgcG38IAH+ySB8/ZCvDL+xLDgbBTBmbcFJly1IiyN74C+0B0BvPdeg5oLowSueft4 8HsmMAz/UEcg1LwsQL+/l4ehilIBvp1xEAjPOzlwRmZF+NlXj+FK0Yqo+PEzAXpKfRiKL7Vh CNy9iqDz5BgGtfkOge7IFAa3WsXBv5dvIZgomxbh+JkFEc7nlyE4fbyah9tPewQoGk2CxSd1 ZPdbrHtqlmNF7qPMO1/Psx8dlHXUBkVW1HlPZPVqFnO74tnFa5OYNcxFBKY2niJMnasUWcl0 ALOZwUGR9X61yLP7ATveF3tA+7ZJSTNnK7btuz7WptYu20XrBW3O3fw2nIfGpBIkSVROpOVd h55H9TtUgjQSkePoyMgCF80b5Feou/QPoQRpJU4+8QJ1PRwgUX69bKWt4ZQow8tbaMflxtWu Tn6Dnupuw9FM5Zdp0xXfqkezsv/9Sdkqo5eT6GhNiESdVD6hoTf6/hH+L2yiN1wj/Fmkq0dr GpHebMlON5rTErel5lrMOdsOZqSraOWfnF8sfehBc0MfdCFZQoa1upHPG0x6wZidmZvehajE GTboCse/Mel1JmPuZ4ot4yNbVpqS2YViJd6wUbdz/qhJL39qPKIcVhSrYnt+xZImJg8Vv1sz bKndfqg17E/Ud25e12HuyRnTlccNfJl8yTrTsrdg99empEItKzjnjD/2iclCY/e8+vqbNzfF uZU1D67/umxtkVMq1IODnt8ItZY/2p+iO+d/+P2LRl9/wtPkZ1PrvdO9u95Ldr8T3h8az9K8 tDO4Naby/X2ev6qDkHHscXCzgc9MNe6I52yZxv8AioNHJUsDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSf0yMcRzHfb/Pz46zxwnP/IqbYklqlA+Zsdk8bMz8oWmTbu6ZburKXSJm SolSTSynOlTsOnVUz4VDWevWj2P9cpWyNDVJ65LFNSk/qs0/7733/rw+77/eLKHIp5ayGm2c qNOqopS0jJQdCEneYM0tUgcMS9shOyMA3D+ukmAss9DQ+rgUgaUyCcNQ3R54N+5CMNnUQoAh pxVBYd8HAirrexFUmy/R4Pw0H9rdozQ4cq7RkHy/jIa24SkMPbduYCiV9sOb60UYaiYGSTAM 0ZBvSMbT8gXDhKmEAVOiN/Sb8xiY6gsER28nBfY7Dgqq36+H3Ls9NFRVO0iot/VjcL4w0tBr +UvBm/pGElqzMyl49LWIhuFxEwEm9ygDb2sKMJSnTLelfv9DQUNmDYbUBxUY2rtfInh19SMG ydJJg93twmCVcgj4VVyHoD9rhIHLGRMM5CdlIbh2+RYJLb8bKEjpCYLJn0Z6Z4hgd40SQor1 jFA9XkAKr4t44XneB0ZIefWeEQqk04LV7CvcrxrCQuGYmxKkkjRakMZuMEL6SDsWvjY3M0Lj 7UlS+NRuwAeXh8m2q8UoTbyo27gjQhaZ99fAxN6Vne1OqsCJ6CObjliW5zbz0hOUjjxYmlvL d3VNEDPek1vFWzM/U+lIxhLclbm8+VsTPcMv5GL5suHQGYbkvPnnxSWzv3IumE+zV+AZz3Ne fGl5zWyPx3Q+8DNrllFwQXxPbi99HckK0JwS5KnRxkerNFFB/vqTkQlazVn/4zHREpqejOnC VLYN/XDuqUUci5Tz5F3nC9UKShWvT4iuRTxLKD3lyX331Aq5WpVwTtTFHNOdjhL1tWgZSyqX yPeFihEK7oQqTjwpirGi7v8Vsx5LExHuDh4wrPxm3Glef6xyQXbrlcUuv7DlY50cLtZ1XLKF 707du2XQkmZLOHwo4Gl4sXZ0nbdP7ODR1cbg2z73Bo50OA7abTE31zTvYqrmerlW1J4L8Up6 Vqfd5u/niYz6L6ozfMTWlRfSNqx9h5ybFr0+1dDW9/C4x0VnZ8aA79jkeaWS1EeqAn0JnV71 D1FprvouAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now that CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT was introduced, apply the consideration to wait_for_completion()/complete(). Signed-off-by: Byungchul Park --- include/linux/completion.h | 4 ++-- kernel/sched/completion.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/completion.h b/include/linux/completion.h index 32d535abebf3..15eede01a451 100644 --- a/include/linux/completion.h +++ b/include/linux/completion.h @@ -41,9 +41,9 @@ do { \ */ #define init_completion_map(x, m) init_completion(x) -static inline void complete_acquire(struct completion *x) +static inline void complete_acquire(struct completion *x, long timeout) { - sdt_might_sleep_start(&x->dmap); + sdt_might_sleep_start_timeout(&x->dmap, timeout); } static inline void complete_release(struct completion *x) diff --git a/kernel/sched/completion.c b/kernel/sched/completion.c index d57a5c1c1cd9..261807fa7118 100644 --- a/kernel/sched/completion.c +++ b/kernel/sched/completion.c @@ -100,7 +100,7 @@ __wait_for_common(struct completion *x, { might_sleep(); - complete_acquire(x); + complete_acquire(x, timeout); raw_spin_lock_irq(&x->wait.lock); timeout = do_wait_for_common(x, action, timeout, state); From patchwork Mon Jul 3 09:47:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B33A4C04A6A for ; Mon, 3 Jul 2023 09:50:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231464AbjGCJuM (ORCPT ); Mon, 3 Jul 2023 05:50:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231395AbjGCJt5 (ORCPT ); Mon, 3 Jul 2023 05:49:57 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 402041B6; Mon, 3 Jul 2023 02:49:53 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-0b-64a299b42e6c From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 19/25] dept: Apply timeout consideration to swait Date: Mon, 3 Jul 2023 18:47:46 +0900 Message-Id: <20230703094752.79269-20-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTcRjG+5/L/8zl6rQunooujKIosowubxIVfajzoUKICKyo0Q65nFO2 XJlFmlt201Iwr9S2ai1dWZuEXRRTNHV5KUeusKXSzbzFbKM5u0yjLy8Pv+d5n0+PiJQ+oeeI lOpjgkYtV8mwmBIPhptWVBSaFas6G+dDzuVV4PtxnoKSchuG9vtlCGwV6QT01W+HTv8AgmBL Gwn5ee0ITD3vSaho8CCosp7F0PFxCrh8wxia8i5hyLhZjuFV/xgBXddyCSiz7wTnVTMBNYEv FOT3YSjOzyBC5ysBAUspA5a0xdBrLWJgrCcKmjxvaKh6txwKr3dheFbVREFDZS8BHU9KMHhs f2hwNjRS0J6TRcO9ITOGfr+FBItvmIHXNUYCHuhDRedGftPwIquGgHO3HhLgevsUQfX5bgLs tjcY6nwDBDjseSSM3qlH0Js9yIDhcoCB4vRsBJcM1yho+/WCBn3XWgj+LMFbovm6gWGS1zuO 81V+I8U3mzn+cdF7htdXv2N4oz2Zd1iX8Tef9RG8yeujeXvpBczbvbkMf3HQRfBDra0M31gQ pPiPrnwiZm6seKNCUCl1gmblpkPiuK/dO5NG8In06u9UGmqmL6IwEceu4Rwlt8n/2tk/MsEx u4RzuwMTfAa7kHNkfQ5xsYhkMydz1u8teNyYzu7hOoOZEyGKXcyNGIuZcS1h13G+Nj31r3QB V/agZiITFuKffmajcS1l13JdhR48XsqxV8K4zG/dzL+H2dxzq5u6iiRGNKkUSZVqXYJcqVoT GZeiVp6IPJyYYEehRVlOj+2rRN723bWIFSFZuMSdalJIablOm5JQizgRKZshyei5oZBKFPKU k4Im8aAmWSVoa9FcESWLkKz2H1dI2SPyY0K8ICQJmv8uIQqbk4aS/NHpfY3r21arClKHyNSg OifStXU09/UHw5mB5s2q7gj3/tScu/Uxs3SSe/GBDebwmK0mr011Zku2yTndM5o875f4pXKW NiHx1I7YukXRR7dVTlp6INNbuGHRh2JdvDZt+/qKiJiZBUX7DI+iY/frfVF7DYFpU10dd3fl NpvLFzpllDZOHrWM1GjlfwG8LgDSTQMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiGed9zzntKtebYsXimJmoT48fiB3G4J9EYoz98o4H4xxH3Rxt7 HJ2laCsM1EWwFUWEiAYrH85STUdKVTwlmXOWMIhIqUKVCswgAvGLgaKUVrH4UTD+eXLlvu9c vx4Fo67kZiv0xv2Syag1aIiSVaassSyrK3PoVgZGCJScXAnhseMsVF51EwhcqUHgrsvDMHhr E3RFhhFE77YzYCsNIKjqf8RAXXMvAm/1EQIdT2ZAMBxT+EoLCVguXiVwb2gCQ8/Z0xhq5GTw n3JgaBh/zoJtkECFzYJj5wWGcaeLB2fuQhioLudhoj8RfL2dHDSd93Hgffg9lP3RQ+Cm18dC 8/UBDB03Kgn0uj9x4G9uYSFQUsTB5VcOAkMRJwPO8AgP9xvsGGqtMVt+6CMHt4saMORfuoYh +N8/COqP92GQ3Z0EmsLDGDxyKQPv/7yFYKD4JQ9HT47zUJFXjKDw6FkW2j/c5sDakwTRd5Vk /RraNDzCUKvnN+qN2Fna6hDp3+WPeGqtf8hTu5xJPdVL6cWbg5hWjYY5KrsKCJVHT/P0xMsg pq/a2njaci7K0idBG94692flWp1k0GdJphXrdirTXvQl7w2R7Lz612wuauVOoHiFKPwg+odC U0yERWJ39zgzyQnCfNFT9CyWKxWMcGyaWP36LpksvhG2iV3RY1MjVlgohuwV/CSrhNViuN3K fpHOE2tqG6Y28bH86btiNMlqIUnsKeslp5DSjuJcKEFvzErX6g1Jy8170nKM+uzluzLSZRT7 GefvEyXX0VjHpkYkKJBmuqr7YJVOzWmzzDnpjUhUMJoElaX/gk6t0mlzDkimjB2mTINkbkRz FKxmlmpzqrRTLfyi3S/tkaS9kulrixXxs3PR7ju1HxKWuf6yrQvc6zx86H3J+u1zpxe6C8rO 7LtiWZUv290RHKno8jQt2P3WZ1hM/PRB0v+LzwxuPKL/7ttZvz5bknkw23u+ZXvKqDG1IIUU /dh3//CiaNyWunzHjA3FGQUzbcn/hqIGZ7zc6nrTtqH8bdzET8FyP3shPLzixtjjRA1rTtMm LmVMZu1n8BLdTi8DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now that CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT was introduced, apply the consideration to swait, assuming an input 'ret' in ___swait_event() macro is used as a timeout value. Signed-off-by: Byungchul Park --- include/linux/swait.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/swait.h b/include/linux/swait.h index 02848211cef5..def1e47bb678 100644 --- a/include/linux/swait.h +++ b/include/linux/swait.h @@ -162,7 +162,7 @@ extern void finish_swait(struct swait_queue_head *q, struct swait_queue *wait); struct swait_queue __wait; \ long __ret = ret; \ \ - sdt_might_sleep_start(NULL); \ + sdt_might_sleep_start_timeout(NULL, __ret); \ INIT_LIST_HEAD(&__wait.task_list); \ for (;;) { \ long __int = prepare_to_swait_event(&wq, &__wait, state);\ From patchwork Mon Jul 3 09:47:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD9C5C05051 for ; Mon, 3 Jul 2023 09:50:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231737AbjGCJuk (ORCPT ); Mon, 3 Jul 2023 05:50:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231397AbjGCJt5 (ORCPT ); Mon, 3 Jul 2023 05:49:57 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E1FE4BE; Mon, 3 Jul 2023 02:49:53 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-1b-64a299b4ca43 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 20/25] dept: Apply timeout consideration to waitqueue wait Date: Mon, 3 Jul 2023 18:47:47 +0900 Message-Id: <20230703094752.79269-21-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0hTcRjG+//P1eXksIxOCRUjK4wuVtpL9yjofAm6fBAyqtEOOZpm07x0 Ic1puVKyWOal2qbMpZY1DTJTluWti1mNXGaSYhfLtbK2nNPKGX15+fE8PL9PL0vIaqkZrCou UdTEKdRyWkJKnIHGhTUFJuWSz0WhkHd2Cbh/niahuKqSho4bFQgqa9IxDDRthk7PIALf02cE 5Os7EBh73xJQ09yDoN5ykoaX/UFgd7toaNOfoSGjpIqG519GMXRfPI+hwroFHp8zYbB5P5KQ P0BDUX4GHj+fMHjN5QyY00Khz1LIwGhvOLT1vKKgvmsBFFzppuFefRsJzXf6MLy8W0xDT+Uf Ch43t5LQkZdDwfWvJhq+eMwEmN0uBl7YDBhuasdFWT9+U9CSY8OQVXoLg/11HYKG0+8wWCtf 0fDAPYih2qonYKSsCUFfrpOBzLNeBorScxGcybxIwrOxFgq03RHgGy6m168UHgy6CEFbnSzU ewyk8MjEC7WFbxlB29DFCAbrYaHaEiaU3BvAgnHITQnW8mxasA6dZwSd046Fr+3tjNB6yUcK /fZ8vDVkp2S1UlSrkkTN4rV7JTGG7y4q/jaTcnlAnYa0tA4FsDy3nM9y6RgdYidYrw33xzQ3 j3c4vISfg7nZfHXOB0qHJCzBnZrMW749ndhO4Xbz+mwL6WeSC+Vr655TfpZykXxhlxb/88/i K27aJkQB4/n74VzkZxkXwXcX9NB+Kc9dCODLHOnMv8F0/r7FQZ5DUgOaVI5kqrikWIVKvXxR TGqcKmXRvoOxVjT+UObjo9F30FDHjkbEsUgeKHUcNSpllCIpITW2EfEsIQ+WZvReVcqkSkXq EVFzcI/msFpMaEQhLCmfJl3qSVbKuP2KRPGAKMaLmv8tZgNmpKGZ3zuL44Pmt09NnJUH22SR I56N+1d37S4X2+csDFtz+8nm+7vG1hH2DSuOrfQtrlMf1w/WnfBlJoJv+x+j7RDv1D+sckp8 1rFNpSdq31y/JmkxSdelHWUvVPRnZ0+ZHeVtjZirCJMkhxs/yH8t+2Tu3NCmCYoqiWSaWne+ S1kV7ZGTCTGK8DBCk6D4CxOfu69MAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzXSbUxTZxgG4L3vOec9h47iSUe2EzHTdFGjiw4WWZ4FZ/jnmxnFHxq/EqXa E6mWj7TSibqsjoqKgEBSkIJa0NQK+HVA5wcQUkK1GhRGwxgpjZBu2oBWHSV04CZdsj9Prtx3 cv96BEZTzy0UDHmHZFOezqglKla1KaN4VXtdkz614noSVJWlQnTqFAsNN1oJ9F9vQdDafhxD uHc9/DY9iWC27xkDtfZ+BI1jowy0e4MIOt0/ExgMJYE/GiHgs58hUHzpBoGBiTkMgZpqDC3K RnhS2YShO/aChdowgfraYjx/XmKIuZp5cFmXwrjbwcPcWBr4gkMc9Jz3cdA58iXUXQgQ6Oj0 seC9O45h8H4DgWDrvxw88T5iob+qnINrr5sITEy7GHBFIzz82u3EcNM2v1by1z8cPCzvxlBy +RYG/+8PEHSdeo5BaR0i0BOdxNCm2Bn4+0ovgvGKVzycKIvxUH+8AsGZEzUsPHv/kANbIB1m ZxpIZgbtmYww1Nb2A+2cdrL0cZNE7zlGeWrrGuGpUymkbe6V9FJHGNPGd1GOKs2nCVXeVfO0 9JUf09dPn/L00blZlob8tXjzop2qtXrZaLDIpq/WZatynG8jXMFt/vD5sNGKbKQUCYIkrpHs trRSlCAQcbk0PBxj4k4Wl0ht5X9ypUglMOLJjyX3mz4SLz4Rd0v20242blZcKt17MMDFrRa/ kRwjNhy3JC6WWm52/zeUMJ//MVOB4taI6VKgLkgqkcqJPmpGyYY8S67OYExfbT6YU5RnOLx6 X36uguZfxvXjXNVdNDW43oNEAWkT1cNHG/UaTmcxF+V6kCQw2mR18dhFvUat1xUdkU35e0yF RtnsQSkCq/1M/f02OVsj7tcdkg/KcoFs+r/FQsJCK+rYzmd/+8XU2O7exmOh54o1K3ONEFo2 6svKVD5dsS2xK+WnBZUHet9zBrpKP/vmTp+y6BerumrL3kDGnQmLY+ZxamK1ZkflUMmGwmDA U7AYq7d+Hrq6I99bMuA4W7bL8XX6dE312mX6cU34O2nnvpSI721Z0hFv1rqY09Lv9yw5oGXN Obq0lYzJrPsA5RcDsy4DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now that CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT was introduced, apply the consideration to waitqueue wait, assuming an input 'ret' in ___wait_event() macro is used as a timeout value. Signed-off-by: Byungchul Park --- include/linux/wait.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/wait.h b/include/linux/wait.h index ff349e609da7..aa1bd964be1e 100644 --- a/include/linux/wait.h +++ b/include/linux/wait.h @@ -304,7 +304,7 @@ extern void init_wait_entry(struct wait_queue_entry *wq_entry, int flags); struct wait_queue_entry __wq_entry; \ long __ret = ret; /* explicit shadow */ \ \ - sdt_might_sleep_start(NULL); \ + sdt_might_sleep_start_timeout(NULL, __ret); \ init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0); \ for (;;) { \ long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\ From patchwork Mon Jul 3 09:47:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4A4FC0015E for ; Mon, 3 Jul 2023 09:50:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231523AbjGCJud (ORCPT ); Mon, 3 Jul 2023 05:50:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231401AbjGCJt5 (ORCPT ); Mon, 3 Jul 2023 05:49:57 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E2A2618D; Mon, 3 Jul 2023 02:49:53 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-2b-64a299b4f274 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 21/25] dept: Apply timeout consideration to hashed-waitqueue wait Date: Mon, 3 Jul 2023 18:47:48 +0900 Message-Id: <20230703094752.79269-22-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAz2SbWyLaxjH3ffzulp5UhMPEi9N5MQwnHByEcQH4UYkEt+OBE/0OdacbpPO ZiOS0WFmq5l0tTdbi55mqzOn25wdNme2dMyMOqsp68q6HSz2IqOjNqMlfLnyy/9/Xb9PF0+p rjOzeG3iAVmfKOnUrIJWDEZbl9QUWjXLJlyL4WzOMgi+y6KhpMrBgvvPSgSOmqMY+l2b4PHo AIKx9gcUmE1uBJaebgpqWvwIGuzHWOjomwKe4DALrabTLBguVrHw8PU4Bl9BPoZK5zZoy7Ni aAy9pMHcz0Kx2YDD4xWGkK2CA1vGAgjYizgY71kOrf5OBhqeLoLCCz4W6htaaWipC2DouF7C gt/xmYG2ljs0uM/mMnBlyMrC61EbBbbgMAf/NZZjuJoZFp14O8HA7dxGDCcu/YXB8+QGgptZ zzE4HZ0sNAcHMFQ7TRR8/MOFIGAc5OB4ToiD4qNGBKePF9Dw4NNtBjJ9K2HsQwm7fjVpHhim SGb1QdIwWk6Tu1aR/FPUzZHMm085Uu5MIdX2WHKxvh8Ty0iQIc6KUyxxjuRzJHvQg8nQ/fsc uXN+jCZ9HjPePvtXxRqNrNOmyvql6/Yo4u8ZqtB+I5f2xpeLM1APk42ieFFYIXocl39wW30f jjAr/CR6vSEqwjHCPLE690V4R8FTwsnJov1NOxsppgmyGGjupyNMCwtEk98SznleKfwieo/F fHPOFSuvNn71RIXj/z8YUYRVwkrRV+hnI05ROBMlZl/7m/t2MFO8ZffSeUhZjiZVIJU2MTVB 0upWxMWnJ2rT4vYmJThR+KNsR8Z31qER944mJPBIHa30HrZoVIyUmpye0IREnlLHKA09ZRqV UiOlH5L1Sbv1KTo5uQnN5mn1DOXPowc1KmGfdED+XZb3y/rvLeajZmUgab6l2P52beneOUUp pXU1TXuS1jIPddcWZmjsl+sWdtg2XihJ6lzlcLl630/O8ew0u4XmR5sLdi3qGpsUv2Vrml+M U+m7/p0I3ru0r7coYUqWqda+4WN0+yMpUFaYnjf3XGto46GyZ1pjrbd0yDK967eplZs2kytt G/KDq6fyt84Z1HRyvLQ8ltInS18ABizWgE0DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSX0xTZxjG+b5zzncOddWTUrcTxEyboIlGhCjmTZRpdqEnS1h2YaLxYqOx J9IABVtEmZoArX+xDZAAQxgrldQKbEDBhM2ihCpaUSlSEbF2gyDaiGCUklZQ187s5s0vz/Pk d/VylKKBSeS0ukJJr1PnqoiMln2/3bipu86mSa3xroTKC6kQmj9LQ0N7GwHvH60I2rpLMQRv 7YHHCzMIFu8PUVBb7UXQNPGMgu6BAIJeRxmBkanl4AvNEfBUlxMwXmonMPxqCYO/pgpDqzMT BitsGPoiL2ioDRKorzXi6HmJIWJvYcFekgyTjossLE2kgScwyoD7Vw8DveMboa7RT8DV66Fh oGcSw8hfDQQCbZ8YGBy4Q4O30szA77M2Aq8W7BTYQ3MsPOyzYugwRW2n331k4La5D8Pp5k4M vifXEFw/+w8GZ9soAXdoBkOXs5qC95dvIZi0vGbh1IUIC/WlFgTlp2poGPpwmwGTPx0Www1k 13bRPTNHiaauo2LvgpUW79oE8c+Lz1jRdH2cFa3OI2KXY4N4yRXEYtPbECM6W84R0fm2ihXP v/ZhcfbBA1a888siLU75avEPSQdkOzRSrrZI0m/+JkuWfc/Yjgos7LE3fjMuQRPMeRTPCfxW YdA1hWNM+PXC2FiEirGSXyN0maejGxlH8WeWCY4390msSOAlYdIdpGNM88lCdaApmnOcnN8m jJUpPzu/Flo7+v7zxEfj52ELirGCTxf8dQFSgWRWFNeClFpdUZ5am5ueYsjJLtZpj6UczM9z oujP2E8uVfag+ZE9/YjnkOoL+djxJo2CURcZivP6kcBRKqXcOPGbRiHXqIt/lvT5P+mP5EqG frSKo1Vfyb/bJ2Up+EPqQilHkgok/f8t5uITS9CgOeHb3esG+CthOqxu3PLji/dlB+JqtP22 HRnj7lCpJ+Nd5Zf6ly3L/z7DVeXPFu6+TDL8hx+dWPLtX1Fmmd572GY7lzK+4pq9NOnKqqGd /pHUlfFh02rv2uEbQVf3FrbZmjR8VfO0omB1TkJmeUdkaufNE4mu1NFpHbNV71C6O1W0IVud toHSG9T/AuhAtaAvAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now that CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT was introduced, apply the consideration to hashed-waitqueue wait, assuming an input 'ret' in ___wait_var_event() macro is used as a timeout value. Signed-off-by: Byungchul Park --- include/linux/wait_bit.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/wait_bit.h b/include/linux/wait_bit.h index fe89282c3e96..3ef450d9a7c5 100644 --- a/include/linux/wait_bit.h +++ b/include/linux/wait_bit.h @@ -247,7 +247,7 @@ extern wait_queue_head_t *__var_waitqueue(void *p); struct wait_bit_queue_entry __wbq_entry; \ long __ret = ret; /* explicit shadow */ \ \ - sdt_might_sleep_start(NULL); \ + sdt_might_sleep_start_timeout(NULL, __ret); \ init_wait_var_entry(&__wbq_entry, var, \ exclusive ? WQ_FLAG_EXCLUSIVE : 0); \ for (;;) { \ From patchwork Mon Jul 3 09:47:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 693C3C001DD for ; Mon, 3 Jul 2023 09:50:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231727AbjGCJui (ORCPT ); Mon, 3 Jul 2023 05:50:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231420AbjGCJt5 (ORCPT ); Mon, 3 Jul 2023 05:49:57 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E2943DD; Mon, 3 Jul 2023 02:49:53 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-3b-64a299b4365d From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 22/25] dept: Apply timeout consideration to dma fence wait Date: Mon, 3 Jul 2023 18:47:49 +0900 Message-Id: <20230703094752.79269-23-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAz2SbUxTZxiGfd9zzntOCzVnlbij/tA0Mzo/8GOgT5xxJvvhq5lGs+mykaiN PZNqqaYoirgMpSqCBZFAAesGZZQGEPWAEac1DCdfRulGw6pDNvBjEIsoUmIFphQz/zy5ct+5 r1+PwGivcdMFo3mfbDHrTTqiZtX9kSULawudhsVNb2Ih59RiCA6ls+C4UEXAW12JoKr2CIa+ W2vgz+EAgpE7bQzY87wISrofMFDb2IXA4z5KoP3RZPAFBwi05GUSSCu9QOD3p6MYOvPPYKhU 1sPt004M9aF/WbD3EThrT8PjpxdDyFXBgyt1NvS4i3gY7V4CLV0dHHjuz4fCHzsJXPe0sNBY 14Oh/RcHga6qNxzcbmxmwZtj4+D8MyeBp8MuBlzBAR7+qC/GcNE6Ljr+8j8Ommz1GI7/fAmD 7941BDfS/8GgVHUQuBkMYKhR8hh4XX4LQU9WPw/HToV4OHskC0HmsXwW2saaOLB2xsLIKwdZ vYLeDAww1FpzgHqGi1na6pTo1aIHPLXeuM/TYmU/rXHPo6XX+zAtGQxyVKk4SagyeIanGf0+ TJ/dvcvT5oIRlj7y2fHGGd+qVxpkkzFJtixatV0dP+j9De1tFw76R3rZVFTJZyCVIIkx0glv 1nu2/d2Gw0zEOZLfH2LCHCXOkmpsT7gMpBYY8USE5H5+h4SLKeJWyfO4dWLAirOlsivVEwON uEzKz84l76QzpcqL9RO5ajx//CoLhVkrxkqdhV0kLJXEXJV05fIQfjeYJv3q9rOnkaYYTapA WqM5KUFvNMVExyebjQejd+xJUND4S7m+H42rQ4PeLxuQKCBdpMafUmLQcvqkxOSEBiQJjC5K k9b9k0GrMeiTD8mWPdss+01yYgOaIbC6DzVLhw8YtOJO/T55tyzvlS3/t1hQTU9Fjo8DZVEJ YvYL7YKIT23KR9khK6xasaw1d/65sWmbA5qCLzzfbF5XMeoq2/QXcpz7Sh2y52zL5CI9RamF zT/M7f1EVT6Hy384K8K5PFqOCq6dKsREm00dH/iU18+/i/vs8+qkr3eLxrFdMw8zJ9PWHX4J NCV9aO2Lpqkb7sVtqUtR69jEeP2SeYwlUf8WgJPvAE4DAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTdxzF/f3uvb97qVbvOpLdoAbThCyiU0nG8k18zMyoPxV1/5n4Go29 G42lYItV3FxQqgIK8gggUrZStRLoRFuysc2yhpcUQ0FpsCPIgPgiIihaYgc+AOM/J5+ck3P+ OgKjquCiBJ0hTTYaNHo1UbCKHaszv6grs2tX/R5aBAXnVkHoVRYL1longa5rNQicdScwDLds hnsTIwgmOzoZKC3uQlA5eJ+ButZ+BJ6qkwS6H8yHQGiMgK/4LIHMS7UE7jydwtBXUoihxrUd bufbMXjDj1koHSZQXpqJp+UJhrCjmgdHRgwMVV3kYWowDnz9PRw0Vfg48PQug7Jf+gjc9PhY aK0fwtD9l5VAv/MdB7db21joKsjl4LdRO4GnEw4GHKExHu56bRiuW6bXTr98y8GtXC+G05dv YAj8+zeChqwBDC5nD4Gm0AgGt6uYgf+vtiAYynvGw6lzYR7KT+QhOHuqhIXON7c4sPTFw+Rr K1m/mjaNjDHU4j5CPRM2lrbbJfrnxfs8tTT08tTmOkzdVbH00s1hTCvHQxx1VWcT6hov5GnO swCmo34/T9suTLL0QaAUf7tot2KNVtbrzLJx5bpERdJ4VzNK7RaOBiefsBmohs9BEYIkfinl /teJZ5iIn0vBYJiZ4UhxieTOfcTlIIXAiGfmSlXPO8hM8Km4X/I8bJ8tsGKMdOWPa7MFpfiV VHK+iHwYjZZqrntn/Yhp/+HrPDTDKjFe6ivrJ/lIYUNzqlGkzmBO1uj08StMB5PSDbqjKw6k JLvQ9Gkcx6cK6tGr7s2NSBSQep4y+GOlVsVpzKb05EYkCYw6Upk5+KtWpdRq0o/JxpTvjIf1 sqkRLRRY9WfKrbvkRJX4gyZNPijLqbLxY4qFiKgMZE00Lx0oth/CC2M2tHib1i2OXmBLCAcM Zqd+SnzHD9Q2b/vHr3dvuZfzUwubEJ26MexYVpTv3x/8ein9hrjbbmQlxG7qjfP/3MxbGnlt 4MXK55e3+BqWPz6e31N4Nduasq/ekbR9bP6e9dnBI8G7FWnS2k8Gdrfv7RjdWW7/XlukZk1J mrhYxmjSvAe7+XJlMAMAAA== X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Now that CONFIG_DEPT_AGGRESSIVE_TIMEOUT_WAIT was introduced, apply the consideration to dma fence wait. Signed-off-by: Byungchul Park --- drivers/dma-buf/dma-fence.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index ad2d7a94c868..ab10b228a147 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -783,7 +783,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) cb.task = current; list_add(&cb.base.node, &fence->cb_list); - sdt_might_sleep_start(NULL); + sdt_might_sleep_start_timeout(NULL, timeout); while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) { if (intr) __set_current_state(TASK_INTERRUPTIBLE); @@ -887,7 +887,7 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count, } } - sdt_might_sleep_start(NULL); + sdt_might_sleep_start_timeout(NULL, timeout); while (ret > 0) { if (intr) set_current_state(TASK_INTERRUPTIBLE); From patchwork Mon Jul 3 09:47:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56F61C05051 for ; Mon, 3 Jul 2023 09:50:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231622AbjGCJuQ (ORCPT ); Mon, 3 Jul 2023 05:50:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbjGCJt6 (ORCPT ); Mon, 3 Jul 2023 05:49:58 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 89B26E58; Mon, 3 Jul 2023 02:49:55 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-4b-64a299b4b568 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 23/25] dept: Record the latest one out of consecutive waits of the same class Date: Mon, 3 Jul 2023 18:47:50 +0900 Message-Id: <20230703094752.79269-24-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTdxTH+d3H77aVLjeVZFc0sjQxJm46WdCc4EJAk3ElWXwsMb6SebVX aFaqK4KgWcKjMnkqbFBQslEwtaNFasEMUUhFBZEoVSoi1k4IyLC8RFul4CbF+M/JJ+d8z+ev r4RUXKPDJWrtMVGnFTRKLKNkE6HGtU2VNar1vf9sgpLC9eB7c5qCqgYrBuclCwJrUxYBY7fj 4bF/HMHcvR4SDGVOBMbBZyQ0dXgQtJqzMfQOfwYu3xSGrrICDDm1DRgeeOcJcJeXEmCxfw/d Z2sIcMyOUmAYw3DekEMsjH8JmDXVMWDKXAVD5nMMzA9GQpenj4bWgS+h8g83huutXRR0NA8R 0NtShcFj/Z+G7o47FDhLimion6zB4PWbSDD5phh46KgmwKZfEOW+/o+GziIHAbkXLhPgenIN Qdvp5wTYrX0YbvrGCWi0l5EQuHgbwVDxBAOnCmcZOJ9VjKDgVDkFPe87adC7N8DcuyocG83f HJ8ieX3jcb7VX03xd2s4/uq5Zwyvbxtg+Gp7Kt9oXsPXXh8jeOOMj+btdXmYt8+UMnz+hIvg J+/fZ/g7FXMUP+wyENuX75V9qxI16jRR93XMAVnSi6oK+ujbJelFD9qoTNQrzUdSCcdGcS3N ncwnHv7rbxxkzK7m+vtnySCHsV9wjUUv6Hwkk5Dsr0s48/S9xdBS9mfueXeACjLFruLyvOUo yHJ2I9fS4aA/SiM4i82xKJIu7EfeFS9mFOwGzl3pwUEpx56RcvorbvLjwzLuhrmfOovk1Sik DinU2rRkQa2JWpeUoVWnrzt0JNmOFipl+mV+XzOacf7QjlgJUobK+08aVQpaSEvJSG5HnIRU hslzBv9UKeQqIeOEqDvyoy5VI6a0o+USSvm5/Bv/cZWCTRSOiT+J4lFR9+lKSKThmWhj9IrI 0fhpzAYeekfEkaSelY67jnDNeOyeuLDVoY+uNkTsCFg216f/VuD86jubt/biU2tYoWDofto5 2XPrcJ8rM+DZlphtPzgadTBRKD3kt+2Pdp0pjvl995udV2Qxr6Jstw7U+xJePuZeJRjjjXup 2JFL87lx2q2H91t27QvdEqKkUpKEyDWkLkX4AG/pntpOAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiG977nnPe0dV2OlcwTNJnpJBjmVJLVPAnGEBP1zYxsIVGT/cHG noxOQGkVZcQIa0UFS8CFTyuDoh1CBW2JHyCkKREthA8tcQyPBEhlNCBszqJ8KALL/jy5ct93 rl+PgtFc5SIVxrQTkilNn6IlKlaVEGf5uqncYdh2x7cRii5tg/CbCyzYG10E+hrqEbiacjCE Hu6FP2YmEcx39zJQWtyHoHrkBQNNHUMIWmt/IRAIfgb94WkC/uJ8ApaaRgJPJhYwyCWXMdS7 90NXoQODd/YvFkpDBK6UWvDSGccw66zjwZkdBaO1FTwsjMSCf+gZB+1X/Ry0Dn4F5ZUygQet fhY67o1iCDTbCQy5Fjno6njMQl+RjYObUw4CEzNOBpzhaR6eeqsw3LIu2XL//cDBI5sXQ+61 2xj6/2xB0HZhGIPb9YxAe3gSg8ddzMDc7w8RjBa84uHcpVkeruQUIMg/V8JC7/tHHFhlHcy/ s5P4ONo+Oc1Qq+cUbZ2pYmmnQ6T3K17w1No2yNMq90nqqY2hNQ9CmFa/DnPUXXeRUPfryzzN e9WP6VRPD08fl82zNNhfir9f/4Nqh0FKMWZIpq07D6uSx+xl3PG3q07bnrSx2SigzENKhSh8 IwZv3CXLTIRocWBgllnmCGGD6LGNcXlIpWCE86vE2r+7V0ZrhHRxuGuOXWZWiBIvTpSgZVYL 28XmDi/3n/QLsf6Wd0WkXMpfvitY2WgEnSiXD5FCpKpCn9ShCGNaRqremKLbYj6anJlmPL3l yLFUN1p6GueZhaJ76E1grw8JCqT9VD2QVW3QcPoMc2aqD4kKRhuhtoz8ZtCoDfrMnyXTsSTT yRTJ7EPrFKx2rfrbQ9JhjfCj/oR0VJKOS6b/W6xQRmYj+55Nm6NaxtWxNQk5eHW3vDMrWohE Olf8KYdvPmYx8Z/hRtvLs8rvps6SXxsS5c0Lu1s6Nx0IBUP7749VymavRkfXr/kp3vJ53K6K wYaQ5+5iel5qcKThy2ZlTmFFtMzJuc+vJ+VHdM7J7Nr39aJ/pld/MC4wtU/qDbt9k1Yta07W x8YwJrP+I9840PEwAwAA X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The current code records all the waits for later use to track relation between waits and events in each context. However, since the same class is handled the same way, it'd be okay to record only one on behalf of the others if they all have the same class. Even though it's the ideal to search the whole history buffer for that, since it'd cost too high, alternatively, let's keep the latest one at least when the same class'ed waits consecutively appear. Signed-off-by: Byungchul Park --- kernel/dependency/dept.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 52537c099b68..cdfda4acff58 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -1522,9 +1522,28 @@ static inline struct dept_wait_hist *new_hist(void) return wh; } +static inline struct dept_wait_hist *last_hist(void) +{ + int pos_n = hist_pos_next(); + struct dept_wait_hist *wh_n = hist(pos_n); + + /* + * This is the first try. + */ + if (!pos_n && !wh_n->wait) + return NULL; + + return hist(pos_n + DEPT_MAX_WAIT_HIST - 1); +} + static void add_hist(struct dept_wait *w, unsigned int wg, unsigned int ctxt_id) { - struct dept_wait_hist *wh = new_hist(); + struct dept_wait_hist *wh; + + wh = last_hist(); + + if (!wh || wh->wait->class != w->class || wh->ctxt_id != ctxt_id) + wh = new_hist(); if (likely(wh->wait)) put_wait(wh->wait); From patchwork Mon Jul 3 09:47:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1412C25B73 for ; Mon, 3 Jul 2023 09:50:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231695AbjGCJuR (ORCPT ); Mon, 3 Jul 2023 05:50:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231485AbjGCJt7 (ORCPT ); Mon, 3 Jul 2023 05:49:59 -0400 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0BB7712C; Mon, 3 Jul 2023 02:49:55 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-5b-64a299b4dcef From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 24/25] dept: Make Dept able to work with an external wgen Date: Mon, 3 Jul 2023 18:47:51 +0900 Message-Id: <20230703094752.79269-25-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0yTZxiG977fkWrNl2LCJ5hoaswSDYIO5xMgHn75umHC4kyWGTMb+7lW AVmrCJpFkIOMU4ANKkiUFlIrdCCFMXWcxBU5RC3a1CqHAOnmkNPCbAOCuALxz5Mr933n+vXw lOIeE8xrE85JugRVnJKV0bKptabQ38pM6nDjm0AoygsH79tsGirqrSw46moRWJvSMIzbD8IL 3ySChcdPKTCUOBAYR4coaOoaRtBqucLCc886cHpnWOgpyWUhvaqehf6JRQyDpcUYam2Hoa/Q hKFj/jUNhnEWrhvSsf/8g2HeXMOBOXUrjFnKOVgc3Qk9wy4GWl9th7Ibgyy0tPbQ0HV3DMPz +xUsDFs/MNDX1U2DoyifgV+nTSxM+MwUmL0zHDzrqMRwJ8MvyvpviYFH+R0YsqobMDhf/oGg LXsEg83qYuGhdxJDo62Egne37AjGCqY4yMyb5+B6WgGC3MxSGp6+f8RAxuBuWJirYPdHkoeT MxTJaLxAWn2VNOk1ieRe+RBHMtpecaTSdp40WraRqpZxTIyzXobYan5iiW22mCM5U05Mpp88 4Uj3tQWaeJwGHBvyrSxaLcVpkyRd2N4TMo21qo9JbNmXXDpZh1PRtc9yUAAvChGi6+pb7iOn NRSjZWaFT0W3e55a5vXCZrEx/28mB8l4Sri6RrT8+5hdLgKF4+Kfde4VpoWtoqva7h/xvFz4 XFzqJavOTWLtnY4VT4A//muuYMWvEHaLg2XD7Orm5wCx6UbMKm8QH1jcdCGSV6JPapBCm5AU r9LGRezQpCRok3ecPBtvQ/6HMv+4eOwumnUc6UQCj5Rr5e5LRrWCUSXpU+I7kchTyvXy9NGb aoVcrUq5KOnOfqc7HyfpO1EITyuD5Lt8F9QK4XvVOemMJCVKuo8t5gOCU9Gl3B88A8bEU8x9 e1D51IBzJKZ9y5tf+HWXG5p7L7f/7tFEBY60h0b2nJpz2W93bcgMdbVlxRx+3V+XOpCWG06n R0f2f6g+UPhF7MzXyZ0b9UNf1R+NfnH7m3j9SFjeri1nynv3Ns/ysWGG0xpxz3vXuNnh2xg1 dsQ18WVR801PxKGjSlqvUe3cRun0qv8BsWmlvEwDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiGfd9zznsOnSVnlcQTtkzTaEyYX0QxT+LiXELiOxN18YcmRjMr PZFqKaZVkC1sIBUdAgESWqodFmq6Bjo+Wpb4QU3Dp5UoKA0gqWRlOsf4WhglFpiusOzPkyv3 fef69QiMys4lCzrDRdlo0OjVRMEqDu8t2tZmq9funLFuhcrSnRCdv86CvdlDYKCpEYGnrRDD RPcBGF6YQrD0pJ8Ba/UAgrrISwbaesYQ+N1XCAy+SoRQdJZAsPoGgSJnM4Fnk8sYwpYqDI3e Q9BXUY8hEHvDgnWCwC1rEY6fPzDEXA08uAo2w7j7Jg/LkVQIjg1x0PljkAP/6Kdgqw0TaPcH Wei5O45h8L6dwJjnPQd9PY9YGKgs4+DnmXoCkwsuBlzRWR6eBxwYWsxxW/Hf7zjoLQtgKL7T iiH04gGCh9d/xeD1DBHojE5h8HmrGVj8qRvBePk0D1dLYzzcKixHcOOqhYX+f3o5MIfTYOmt nezfSzunZhlq9uVS/4KDpY/rJXrv5kuemh+O8tThvUR97hTqbJ/AtG4uylFvww+EeueqeFoy HcJ05ulTnj6qWWLpq5AVf/XxCcVnWlmvy5GNO/adVmR6nH3chfbPL1ummnABqtlVghIESdwt FbZWoRUm4hZpZCTGrHCSuFHylf3OlSCFwIjXPpDcfz0hK8U68ZTU1TSyyqy4WRq60x0fCYJS 3CO9e0z/c26QGlsCq56EePz6bfmqXyWmSWHbGKlACgda04CSdIacLI1On7bddD4zz6C7vD0j O8uL4j/jyl+uvIvmBw90IFFA6rXKkW/rtCpOk2PKy+pAksCok5RFkdtalVKryftGNmZ/bbyk l00d6COBVa9XHjwun1aJZzUX5fOyfEE2/t9iISG5ANV25W8IDpdUpR6udXy/vMln6Ukszq7Y dyzbYN6t53OZ3NLp2TRDytFIToZr4Vxy9+v+rkWrsemQZDzZPHQk8Od3c/M2PqN3S8T527E+ EhpcPzn9S92O/MUvT6194x+NtWoULenh4wdzP7lnCQ1Hvrhv338i/YozPfHMh4pyW9hdo2ZN mZrUFMZo0vwLACZWUy8DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org There is a case where total maps for its wait/event is so large in size. For instance, struct page for PG_locked and PG_writeback is the case. The additional memory size for the maps would be 'the # of pages * sizeof(struct dept_map)' if each struct page keeps its map all the way, which might be too big to accept. It'd be better to keep the minimum data in the case, which is timestamp called 'wgen' that Dept makes use of. So made Dept able to work with an external wgen when needed. Signed-off-by: Byungchul Park --- include/linux/dept.h | 18 ++++++++++++++---- include/linux/dept_sdt.h | 4 ++-- kernel/dependency/dept.c | 30 +++++++++++++++++++++--------- 3 files changed, 37 insertions(+), 15 deletions(-) diff --git a/include/linux/dept.h b/include/linux/dept.h index 0aa8d90558a9..ad32ea7b57bb 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -487,6 +487,13 @@ struct dept_task { bool in_sched; }; +/* + * for subsystems that requires compact use of memory e.g. struct page + */ +struct dept_ext_wgen{ + unsigned int wgen; +}; + #define DEPT_TASK_INITIALIZER(t) \ { \ .wait_hist = { { .wait = NULL, } }, \ @@ -518,6 +525,7 @@ extern void dept_task_exit(struct task_struct *t); extern void dept_free_range(void *start, unsigned int sz); extern void dept_map_init(struct dept_map *m, struct dept_key *k, int sub_u, const char *n); extern void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, const char *n); +extern void dept_ext_wgen_init(struct dept_ext_wgen *ewg); extern void dept_map_copy(struct dept_map *to, struct dept_map *from); extern void dept_wait(struct dept_map *m, unsigned long w_f, unsigned long ip, const char *w_fn, int sub_l, long timeout); @@ -527,8 +535,8 @@ extern void dept_clean_stage(void); extern void dept_stage_event(struct task_struct *t, unsigned long ip); extern void dept_ecxt_enter(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *c_fn, const char *e_fn, int sub_l); extern bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f); -extern void dept_request_event(struct dept_map *m); -extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *e_fn); +extern void dept_request_event(struct dept_map *m, struct dept_ext_wgen *ewg); +extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *e_fn, struct dept_ext_wgen *ewg); extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long ip); extern void dept_sched_enter(void); extern void dept_sched_exit(void); @@ -559,6 +567,7 @@ extern void dept_hardirqs_off_ip(unsigned long ip); struct dept_key { }; struct dept_map { }; struct dept_task { }; +struct dept_ext_wgen { }; #define DEPT_MAP_INITIALIZER(n, k) { } #define DEPT_TASK_INITIALIZER(t) { } @@ -571,6 +580,7 @@ struct dept_task { }; #define dept_free_range(s, sz) do { } while (0) #define dept_map_init(m, k, su, n) do { (void)(n); (void)(k); } while (0) #define dept_map_reinit(m, k, su, n) do { (void)(n); (void)(k); } while (0) +#define dept_ext_wgen_init(wg) do { } while (0) #define dept_map_copy(t, f) do { } while (0) #define dept_wait(m, w_f, ip, w_fn, sl, t) do { (void)(w_fn); } while (0) @@ -580,8 +590,8 @@ struct dept_task { }; #define dept_stage_event(t, ip) do { } while (0) #define dept_ecxt_enter(m, e_f, ip, c_fn, e_fn, sl) do { (void)(c_fn); (void)(e_fn); } while (0) #define dept_ecxt_holding(m, e_f) false -#define dept_request_event(m) do { } while (0) -#define dept_event(m, e_f, ip, e_fn) do { (void)(e_fn); } while (0) +#define dept_request_event(m, wg) do { } while (0) +#define dept_event(m, e_f, ip, e_fn, wg) do { (void)(e_fn); } while (0) #define dept_ecxt_exit(m, e_f, ip) do { } while (0) #define dept_sched_enter() do { } while (0) #define dept_sched_exit() do { } while (0) diff --git a/include/linux/dept_sdt.h b/include/linux/dept_sdt.h index 21fce525f031..8cdac7982036 100644 --- a/include/linux/dept_sdt.h +++ b/include/linux/dept_sdt.h @@ -24,7 +24,7 @@ #define sdt_wait_timeout(m, t) \ do { \ - dept_request_event(m); \ + dept_request_event(m, NULL); \ dept_wait(m, 1UL, _THIS_IP_, __func__, 0, t); \ } while (0) #define sdt_wait(m) sdt_wait_timeout(m, -1L) @@ -49,7 +49,7 @@ #define sdt_might_sleep_end() dept_clean_stage() #define sdt_ecxt_enter(m) dept_ecxt_enter(m, 1UL, _THIS_IP_, "start", "event", 0) -#define sdt_event(m) dept_event(m, 1UL, _THIS_IP_, __func__) +#define sdt_event(m) dept_event(m, 1UL, _THIS_IP_, __func__, NULL) #define sdt_ecxt_exit(m) dept_ecxt_exit(m, 1UL, _THIS_IP_) #else /* !CONFIG_DEPT */ #define sdt_map_init(m) do { } while (0) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index cdfda4acff58..335e5f67bf55 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -2230,6 +2230,11 @@ void dept_map_reinit(struct dept_map *m, struct dept_key *k, int sub_u, } EXPORT_SYMBOL_GPL(dept_map_reinit); +void dept_ext_wgen_init(struct dept_ext_wgen *ewg) +{ + WRITE_ONCE(ewg->wgen, 0U); +} + void dept_map_copy(struct dept_map *to, struct dept_map *from) { if (unlikely(!dept_working())) { @@ -2415,7 +2420,7 @@ static void __dept_wait(struct dept_map *m, unsigned long w_f, */ static void __dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, const char *e_fn, - bool sched_map) + bool sched_map, unsigned int *wgp) { struct dept_class *c; struct dept_key *k; @@ -2437,14 +2442,14 @@ static void __dept_event(struct dept_map *m, unsigned long e_f, c = check_new_class(&m->map_key, k, sub_id(m, e), m->name, sched_map); if (c && add_ecxt(m, c, 0UL, NULL, e_fn, 0)) { - do_event(m, c, READ_ONCE(m->wgen), ip); + do_event(m, c, READ_ONCE(*wgp), ip); pop_ecxt(m, c); } exit: /* * Keep the map diabled until the next sleep. */ - WRITE_ONCE(m->wgen, 0U); + WRITE_ONCE(*wgp, 0U); } void dept_wait(struct dept_map *m, unsigned long w_f, @@ -2654,7 +2659,7 @@ void dept_stage_event(struct task_struct *t, unsigned long ip) if (!m.keys) goto exit; - __dept_event(&m, 1UL, ip, "try_to_wake_up", sched_map); + __dept_event(&m, 1UL, ip, "try_to_wake_up", sched_map, &m.wgen); exit: dept_exit(flags); } @@ -2833,10 +2838,11 @@ bool dept_ecxt_holding(struct dept_map *m, unsigned long e_f) } EXPORT_SYMBOL_GPL(dept_ecxt_holding); -void dept_request_event(struct dept_map *m) +void dept_request_event(struct dept_map *m, struct dept_ext_wgen *ewg) { unsigned long flags; unsigned int wg; + unsigned int *wgp; if (unlikely(!dept_working())) return; @@ -2849,32 +2855,38 @@ void dept_request_event(struct dept_map *m) */ flags = dept_enter_recursive(); + wgp = ewg ? &ewg->wgen : &m->wgen; + /* * Avoid zero wgen. */ wg = atomic_inc_return(&wgen) ?: atomic_inc_return(&wgen); - WRITE_ONCE(m->wgen, wg); + WRITE_ONCE(*wgp, wg); dept_exit_recursive(flags); } EXPORT_SYMBOL_GPL(dept_request_event); void dept_event(struct dept_map *m, unsigned long e_f, - unsigned long ip, const char *e_fn) + unsigned long ip, const char *e_fn, + struct dept_ext_wgen *ewg) { struct dept_task *dt = dept_task(); unsigned long flags; + unsigned int *wgp; if (unlikely(!dept_working())) return; + wgp = ewg ? &ewg->wgen : &m->wgen; + if (dt->recursive) { /* * Dept won't work with this even though an event * context has been asked. Don't make it confused at * handling the event. Disable it until the next. */ - WRITE_ONCE(m->wgen, 0U); + WRITE_ONCE(*wgp, 0U); return; } @@ -2883,7 +2895,7 @@ void dept_event(struct dept_map *m, unsigned long e_f, flags = dept_enter(); - __dept_event(m, e_f, ip, e_fn, false); + __dept_event(m, e_f, ip, e_fn, false, wgp); dept_exit(flags); } From patchwork Mon Jul 3 09:47:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13299805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D873C25B77 for ; Mon, 3 Jul 2023 09:50:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231646AbjGCJuS (ORCPT ); Mon, 3 Jul 2023 05:50:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231540AbjGCJuB (ORCPT ); Mon, 3 Jul 2023 05:50:01 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9FC04BE; Mon, 3 Jul 2023 02:49:57 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-6b-64a299b4c184 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v10 rebased on v6.4 25/25] dept: Track the potential waits of PG_{locked,writeback} Date: Mon, 3 Jul 2023 18:47:52 +0900 Message-Id: <20230703094752.79269-26-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230703094752.79269-1-byungchul@sk.com> References: <20230703094752.79269-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxiGfc/Hew7FmrNq9Kg/po0MUzIFI/rEODXR6JsYE42JJjqj1Z7Y ZoDaIoofsVoUrYJg5EsQKc6uaTvBgqZTwA4URaJUQS0EiBAEG760rISvTSlmf55cue/c16+H pxVudh6vS0iU9AnqOCWWMbL+6Zaf7+cVa6J9xoWQeSUagv9cZKCgxInBe9eBwFl+lgL/003w frgPwfjLBhpysrwILB1tNJTXtiOotJ3D0Ng1A5qCgxjqsi5jMN0uwfC6d4KC1uxrFDhcW6A+ o5gCz2gPAzl+DPk5JmryfKJg1GrnwGqMgE7bDQ4mOmKgrv0dC5UtUZBX2IqhorKOgVp3JwWN DwswtDu/slBf+5wBb2YaC38OFGPoHbbSYA0OcvDGU0RBacqk6MLQfyw8S/NQcOH3exQ0NT9C UHXxAwUu5zsMNcE+CspcWTSM/fEUQWd6Pwfnr4xykH82HcHl89kMNPz7jIWU1lgYHynA61aR mr5BmqSUHSOVw0UMeVEskr9utHEkpaqFI0Wuo6TMpiK3K/wUsQSCLHHZL2HiClzjiLm/iSID r15x5HnuOEO6mnKorfN3yVZrpDhdkqRfumafTPvevONw287jQ4UrjCiXmBHPi8JyscF/xozC prA5cJULMRYiRZ9vlA7xLGGBWJbWzZqRjKeF1HDR9vklDhUzhQOixTMwNWCECLHq0ecplgsr xIeNQ+x36Y+io9QzJQqbzD+OpKMQK4RYsTWvHYekomAKE8f68unvg7ni3zYfk4HkRWiaHSl0 CUnxal3c8iXa5ATd8SUHDsW70OQ/WU9P7HajgHd7NRJ4pJwu9520aBSsOsmQHF+NRJ5WzpKb Om5pFHKNOvmEpD+0V380TjJUo/k8o5wjXzZ8TKMQDqoTpd8k6bCk/7+l+LB5RrTYsWFmT8XI Ro1w5+vsJ0NH9rcw3frrO2r2TWxS1RqdJr+9xdmgLWm+1+OO/rjdcVKMcC/y1melRvpK7apf E3N/0fZmZyoN/Xnxp2PXM4tqvoR3wVj3qYyYhaqVdunLDzOchbbNFWtL3Tgy6W1U4HH4T4XJ 7AO7a2/qNsm552a4kjFo1TEqWm9QfwNgMMjtSwMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAAzWSa0iTYRiGe9/vNFeLryX0UVEx6GxmmPaAYRGVL4HVDymKoEb7yOGcspVp UVhT85CHWWqmmdOYonaa0mlpQ1Ez8VAzM1PJdVI8lTpxaQc1+vNwcd/c169HQslzmKUStfa0 qNMqNQpWSkv3+xk2VWQXqLxSx73BeNULnOPxNOTeL2Oh9V4pgrKKSxj6awPg3cQggqmmFgqy MloRmHq7Kaio60FQWXyZBfvnhdDmHGGhISOJBUPhfRZeD0xj6MpMx1BqCYTGtAIMNtc3GrL6 WcjJMuCZ04fBZS7hwBy9GhzFNzmY7t0CDT3tDNTcamCgsnMjZOd1sfC8soGGuicODPZnuSz0 lP1hoLHuJQ2txmQG7g4XsDAwYabA7Bzh4I0tH8ODmBlb3NhvBuqTbRji7jzE0PbeiqAq/iMG S1k7CzXOQQzllgwKfhbVInCkDHEQe9XFQc6lFARJsZk0tPyqZyCmywemJnPZnX6kZnCEIjHl Z0nlRD5NXhUI5OnNbo7EVHVyJN9yhpQXbyCFz/sxMY06GWIpSWCJZTSdI4lDbZgMNzdz5OWN KZp8bsvCB5cflW5XiRp1hKjb7H9CGvwu8VB49+HIsTzfaHSDJCI3icBvFd6PpnKzzPJrhY4O FzXL7vwqoTz5K5OIpBKKvzJfKP7exM4Wi/mTgsk2PDeg+dVClfX7HMt4X+GZfYz5J10plD6w zYncZvIvkyloluW8j9CV3cOmIWk+mleC3NXaiFClWuPjqQ8JjtKqIz1PhoVa0MzLmC9MG5+g cXtANeIlSLFA1nHepJIzygh9VGg1EiSUwl1m6L2tkstUyqhzoi7suO6MRtRXo2USWrFEtu+w eELOn1KeFkNEMVzU/W+xxG1pNDricXH3nseu2rg1CZ3dfS/q431/HDfXmUiRdVeNMPRoYJu3 wvg2Muit3ZFRmrAoemIFSa0/1Woscnj7EQ+PuLPXA7E6hB9qWrj+wLrhyL5fj0ZeHcu+vVG7 ffneTzs0tH+sq/FDs7/sTmbarjef1gZ6BJgHPYMMk4esP661i4vNVgWtD1Zu2UDp9Mq/vizu 8S4DAAA= X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently, Dept only tracks the real waits of PG_{locked,writeback} that actually happened having gone through __schedule() to avoid false positives. However, it ends in limited capacity for deadlock detection, because anyway there might be still way more potential dependencies by the waits that have yet to happen but may happen in the future so as to cause a deadlock. So let Dept assume that when PG_{locked,writeback} bit gets cleared, there might be waits on the bit to be woken up. Even though false positives may increase with the aggressive tracking, it's worth doing it because it's going to be useful in practice. See the following link for instance: https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/ Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 3 + include/linux/page-flags.h | 112 +++++++++++++++++++++++++++++++++---- include/linux/pagemap.h | 7 ++- mm/filemap.c | 11 +++- mm/mm_init.c | 3 + 5 files changed, 121 insertions(+), 15 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..ac5048b66e5c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,6 +19,7 @@ #include #include #include +#include #include @@ -228,6 +229,8 @@ struct page { #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif + struct dept_ext_wgen PG_locked_wgen; + struct dept_ext_wgen PG_writeback_wgen; } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 92a2063a0a23..d91e67ed194c 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -196,6 +196,50 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H +#ifdef CONFIG_DEPT +#include +#include + +extern struct dept_map PG_locked_map; +extern struct dept_map PG_writeback_map; + +/* + * Place the following annotations in its suitable point in code: + * + * Annotate dept_page_set_bit() around firstly set_bit*() + * Annotate dept_page_clear_bit() around clear_bit*() + * Annotate dept_page_wait_on_bit() around wait_on_bit*() + */ + +static inline void dept_page_set_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_request_event(&PG_locked_map, &p->PG_locked_wgen); + else if (bit_nr == PG_writeback) + dept_request_event(&PG_writeback_map, &p->PG_writeback_wgen); +} + +static inline void dept_page_clear_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_event(&PG_locked_map, 1UL, _RET_IP_, __func__, &p->PG_locked_wgen); + else if (bit_nr == PG_writeback) + dept_event(&PG_writeback_map, 1UL, _RET_IP_, __func__, &p->PG_writeback_wgen); +} + +static inline void dept_page_wait_on_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_wait(&PG_locked_map, 1UL, _RET_IP_, __func__, 0, -1L); + else if (bit_nr == PG_writeback) + dept_wait(&PG_writeback_map, 1UL, _RET_IP_, __func__, 0, -1L); +} +#else +#define dept_page_set_bit(p, bit_nr) do { } while (0) +#define dept_page_clear_bit(p, bit_nr) do { } while (0) +#define dept_page_wait_on_bit(p, bit_nr) do { } while (0) +#endif + #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); @@ -377,44 +421,88 @@ static __always_inline int Page##uname(struct page *page) \ #define SETPAGEFLAG(uname, lname, policy) \ static __always_inline \ void folio_set_##lname(struct folio *folio) \ -{ set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_set_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void SetPage##uname(struct page *page) \ -{ set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define CLEARPAGEFLAG(uname, lname, policy) \ static __always_inline \ void folio_clear_##lname(struct folio *folio) \ -{ clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_clear_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void ClearPage##uname(struct page *page) \ -{ clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define __SETPAGEFLAG(uname, lname, policy) \ static __always_inline \ void __folio_set_##lname(struct folio *folio) \ -{ __set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + __set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_set_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void __SetPage##uname(struct page *page) \ -{ __set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define __CLEARPAGEFLAG(uname, lname, policy) \ static __always_inline \ void __folio_clear_##lname(struct folio *folio) \ -{ __clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + __clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); \ + dept_page_clear_bit(&folio->page, PG_##lname); \ +} \ static __always_inline void __ClearPage##uname(struct page *page) \ -{ __clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define TESTSETFLAG(uname, lname, policy) \ static __always_inline \ bool folio_test_set_##lname(struct folio *folio) \ -{ return test_and_set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + bool ret = test_and_set_bit(PG_##lname, folio_flags(folio, FOLIO_##policy));\ + if (!ret) \ + dept_page_set_bit(&folio->page, PG_##lname); \ + return ret; \ +} \ static __always_inline int TestSetPage##uname(struct page *page) \ -{ return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_set_bit(PG_##lname, &policy(page, 1)->flags);\ + if (!ret) \ + dept_page_set_bit(page, PG_##lname); \ + return ret; \ +} #define TESTCLEARFLAG(uname, lname, policy) \ static __always_inline \ bool folio_test_clear_##lname(struct folio *folio) \ -{ return test_and_clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy)); } \ +{ \ + bool ret = test_and_clear_bit(PG_##lname, folio_flags(folio, FOLIO_##policy));\ + if (ret) \ + dept_page_clear_bit(&folio->page, PG_##lname); \ + return ret; \ +} \ static __always_inline int TestClearPage##uname(struct page *page) \ -{ return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_clear_bit(PG_##lname, &policy(page, 1)->flags);\ + if (ret) \ + dept_page_clear_bit(page, PG_##lname); \ + return ret; \ +} #define PAGEFLAG(uname, lname, policy) \ TESTPAGEFLAG(uname, lname, policy) \ diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index a56308a9d1a4..a88e2430f415 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -915,7 +915,12 @@ void folio_unlock(struct folio *folio); */ static inline bool folio_trylock(struct folio *folio) { - return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0))); + bool ret = !test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)); + + if (ret) + dept_page_set_bit(&folio->page, PG_locked); + + return likely(ret); } /* diff --git a/mm/filemap.c b/mm/filemap.c index eed64dc88e43..f05208bb50dc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1101,6 +1101,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, if (flags & WQ_FLAG_CUSTOM) { if (test_and_set_bit(key->bit_nr, &key->folio->flags)) return -1; + dept_page_set_bit(&key->folio->page, key->bit_nr); flags |= WQ_FLAG_DONE; } } @@ -1210,6 +1211,7 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, if (wait->flags & WQ_FLAG_EXCLUSIVE) { if (test_and_set_bit(bit_nr, &folio->flags)) return false; + dept_page_set_bit(&folio->page, bit_nr); } else if (test_bit(bit_nr, &folio->flags)) return false; @@ -1220,8 +1222,10 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; -static struct dept_map __maybe_unused PG_locked_map = DEPT_MAP_INITIALIZER(PG_locked_map, NULL); -static struct dept_map __maybe_unused PG_writeback_map = DEPT_MAP_INITIALIZER(PG_writeback_map, NULL); +struct dept_map __maybe_unused PG_locked_map = DEPT_MAP_INITIALIZER(PG_locked_map, NULL); +struct dept_map __maybe_unused PG_writeback_map = DEPT_MAP_INITIALIZER(PG_writeback_map, NULL); +EXPORT_SYMBOL(PG_locked_map); +EXPORT_SYMBOL(PG_writeback_map); static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) @@ -1234,6 +1238,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, unsigned long pflags; bool in_thrashing; + dept_page_wait_on_bit(&folio->page, bit_nr); if (bit_nr == PG_locked) sdt_might_sleep_start(&PG_locked_map); else if (bit_nr == PG_writeback) @@ -1331,6 +1336,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, wait->flags |= WQ_FLAG_DONE; break; } + dept_page_set_bit(&folio->page, bit_nr); /* * If a signal happened, this 'finish_wait()' may remove the last @@ -1538,6 +1544,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + dept_page_clear_bit(&folio->page, PG_locked); if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) folio_wake_bit(folio, PG_locked); } diff --git a/mm/mm_init.c b/mm/mm_init.c index 7f7f9c677854..a339f0cbe1b2 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -558,6 +559,8 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, page_mapcount_reset(page); page_cpupid_reset_last(page); page_kasan_tag_reset(page); + dept_ext_wgen_init(&page->PG_locked_wgen); + dept_ext_wgen_init(&page->PG_writeback_wgen); INIT_LIST_HEAD(&page->lru); #ifdef WANT_PAGE_VIRTUAL