From patchwork Wed May 8 09:47:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13658343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 668A6C19F4F for ; Wed, 8 May 2024 09:48:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 922406B00FD; Wed, 8 May 2024 05:48:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D30A6B00FF; Wed, 8 May 2024 05:48:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B1CC6B0100; Wed, 8 May 2024 05:48:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2F4E66B00FD for ; Wed, 8 May 2024 05:48:00 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DD017140C79 for ; Wed, 8 May 2024 09:47:59 +0000 (UTC) X-FDA: 82094752278.02.496B1BE Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf19.hostedemail.com (Postfix) with ESMTP id C1A531A0016 for ; Wed, 8 May 2024 09:47:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715161678; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=FcuGbxG7SMpK3jYpDaZii9W3jhM+xViMxAg/jvlCqh0=; b=XXLJ72qWsKMy0JfoAfjba+Y9M7uLN3+HD008YO3c8joa+ZWyd6zuKqP6fwZQ++AdctbJZE dP/OtemreyLbLZDUT71mVVeTfUbbmJXmV4TbrKYNaqIA3q2Jp/15RiIcgCBQ7RaCzNOre9 sihXIJt741VlX4rb0uRgDsWbMtnl7gU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715161678; a=rsa-sha256; cv=none; b=vd9DewSyQlmXd29vEnY1DdXe4cZdrgMyDlfX1cZinFemcN649OM5mhYQ2HAWLkn47XdQuL 0QOKvfBr9xbgg5mTcax+/i2NcKfDA6OpVwIb8F1fnxt0he3k2j7pwrYrBqsvTLIwNg9lvL TozrTz+vOKweM5oUMUEG38eI9eINTFI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d85ff70000001748-89-663b4a3b41e5 From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v14 23/28] dept: Track PG_locked with dept Date: Wed, 8 May 2024 18:47:20 +0900 Message-Id: <20240508094726.35754-24-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240508094726.35754-1-byungchul@sk.com> References: <20240508094726.35754-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0xTZxjG/b5zek5prDmrZp6hidpIjFfU6PaK8/KP8omamJglZpdsjRxG t1LM4VaWmIBUQCoIKFaBaLmkEKiIrRqxVisEBjNgkQZQLgMkaEeBjNluHfVS2PznzS/P+7xP nj9eKaW4IwmXqrXJgqhVaZSMjJZNLa7YvDtmd9zWvs71UHR+K/je5NJQftPCgKuhHoHldiYG T2s09Pm9COY6n1JgLHEhqBgdouB22zACR+0ZBnrGl4DbN8NAR4mBgayqmwx0TwYxDF4uxlBv PQpPCisxOAOvaDB6GCgzZuHQeI0hYK5jwZwRAWO1pSwER7dBx3CvBBwvNsLVa4MMPHB00NB2 bwxDz/1yBoYt7yXwpK2dBldRvgRuTFcyMOk3U2D2zbDwzGnC0KgPBWX/9U4Cv+Y7MWRX38Lg fm5H8DB3BIPV0stAi8+LwWYtoeDfmlYEYwVTLJw9H2ChLLMAgeHsZRr0gzth7p9yZv8u0uKd oYjelkYcfhNNfqvkSVPpEEv0D1+wxGRNIbbaDaTqgQeTilmfhFjrzjHEOlvMkrwpNybTXV0s ab8yR5NxtxEfC/9a9mWsoFGnCmLk3h9k8cNlNvrUwLe6i+VuNgP5D+ehMCnP7eCL35ahj3xj IojnmeHW8f39AWqel3GreVv+hGSeKc4r46u7Ds7zUi6KP5c7tXBLcxG8y2xf8Mu5z3lv3wj+ L3MVX9/oXNDDQvrzV9MLfgW3k7dnlbJ5SBbyvJHyVx4F/y/xGf+4tp8uRHITWlSHFGptaoJK rdmxJT5dq9ZtOZmYYEWhZzKfDn5zD826jjcjToqUi+XO5VFxCokqNSk9oRnxUkq5TN6a80Wc Qh6rSv9FEBO/F1M0QlIzWiGllcvl2/1psQruR1Wy8LMgnBLEj1ssDQvPQBlR9n04ct2RPTJN 9MregcyYyUummgN3cpoMYvNXZ747LJZsPI2ekqzkeFU3W/ip82DDmkd/nrh1SD87eqzwQLuj QRe7R7vI0GTZVdrY7tl0cejvPzIvOGXGTh5VBQ3dayhvhO5x9svfc/bf9ay9HiNcsouT2k88 NQU/BQyKFbpqJZ0Ur9q2gRKTVB8AooWPyUgDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe0hTcRzF+/3u09XqtlZd9I9iZKmRKWR9yQiDHreiCAqMEHLUNYePyaYr g8DaMt9koSu1UItlU1M3C8ssUdTM0pXz2dQS0cSp9JhkiuWM/jl8OOdw/josITNSnqwqNl7U xCqjFbSElBwP1m8LPhIcEdBnWAPZGQHg+plCQkFFGQ22J6UIyqqvYhhvOgQ9M04Ec+87CDDm 2BAUfRkgoLp5EEFdyTUaOkdWgt01TUNrTjoN+gcVNHyYmMfgyL2FodRyDNpuFmOonx0jwThO Q75RjxflK4ZZk5kBU5I3DJfkMTD/JRBaB7spaLzXSkFd/1a4e99Bw8u6VhKaa4YxdL4ooGGw 7A8Fbc1vSLBlZ1JQPlVMw8SMiQCTa5qBj/WFGCoNi2vJPxYoaMmsx5D8sAqDva8WwauUzxgs Zd00NLqcGKyWHAJ+P2pCMJw1ycD1jFkG8q9mIUi/nkuCwREEc78K6JDdQqNzmhAM1otC3Uwh Kbwt5oXneQOMYHjVzwiFlgTBWuInPHg5joWi7y5KsJhTacHy/RYjpE3asTDV3s4Ib+7MkcKI 3YhPeJ2R7DkvRqt0omb73nBJ5GC+lYz7FHbpdoGdSUIzR9OQB8tzO/jy0XnsZprbwvf2zhJu lnMbeWvmKOVmgnNK+IftB928htvNp6ZMIjeTnDdvM9Uu9aXcTt7Z8xn/29zAl1bWL/kei37f 2NRSX8YF8bX6POYmkhSiZWYkV8XqYpSq6CB/bVRkYqzqkv85dYwFLd7FdGU+uwb97DzUgDgW KVZIbXRwhIxS6rSJMQ2IZwmFXNp0Y1eETHpemXhZ1KjPahKiRW0D8mJJxXrpkVAxXMZdUMaL UaIYJ2r+p5j18ExC+4fOHG7bY/C69ti4r2vtQqj9tDrktY/8cpPv6mfyLSzrKBo65zA9VRxH 6UMnOzbk6T0N5sMj5paxzTS0nKre9Dw3njWsOkDohjuX5U/cqXhRrsX9Jf7VzT9CBy5kNHjq 1F3rvNP831lWdWfd/vY1TqbxCfKJKl1eGVY1OuCrTlCQ2khloB+h0Sr/AvoTEUUqAwAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: C1A531A0016 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: pn8od3htafkqci9h6sago6a7gk6gmt71 X-HE-Tag: 1715161677-76901 X-HE-Meta: U2FsdGVkX18l1pwXog5yGtoGtr/1KrWPBmfMn6GHO0HQmEQlC6G2DDtrtZ/dX2r2SB15ZVLxJvSqB0RS0z6QJwmtP3quXDXioHHz5VHOzm9FhLVGka7VrAF98Sq6Daw2X2Ff+5yT/ivBKAC/mdaqjyGlkGABgAiiV5vbW8ibDShu4lMCnbk+ReCxuzmcPiXjRv1fn6C8Gt+XnR2+xetaWk8PKwAnwb2up0oqgoAxTamib1IYr4iDmtTU2+X98iYgHDaRncLvQBzg4QLpopSrcdaBRT2Sxm5xBjFV5pcLXrYb2b1D1Vg7LgpRO5XAF1JOjGqIM9+7ZRaONZvVQBZtI9k+W3QlhxR3Giw4qNh/hjZ+WmcSmPj/vQgKLePLD8ZgmTOj7qN0AfjdQNuYO7f+AXY19m6vGTP7zRMOEsmJkTMuZuoHBVyIO66XCTUIppjYvKnO+4nPwND6uXmgU9pEUrqtSpLGY+TITuaqwoNqW3T7brCmSbSa3QJDKtAlD5Cnphop0lvfCOj3pxCbWFLu34FqQ1iOysxNnsi1LOO50xkGcFKLGgqQNqsFDa7jcavOxXx45gWIkwEtJ5r5GDD/tY9L1TwUkaifHeH0JAv0cFP/yU8V18RwKvjYKK3eAlML8fZD1bfovEBiW/qHP4g0e4QHSqMNyabk7+pR7Jimi6qWgtjyPxq0jXDFL4q3LeKqCU5j7oNb7CmEZNjVoD3IyeSoMf/uOOl20naDPFs3DW0i40Mholo91mOoiwa8dTxE0aqiVrSzMRMoUqW7POJJNF+Ynl+HTNkSp+hjt5ym4fwDUTgu7QJzMbDrjAjuJQWkk1s1lqndxvILp3dCFyHwUTcX2RDOG1aRUOVPBsA3Y7ZB4Ld/igrDz6+uwyBaBRAcWDruK9KlR5wQWrFHvkcs65PyXIGaoFI7VjWYpUCnDx37b0Owfytw2E6qLCQVuUD1a59FSwFCnpZ1BiqqwOr w8nz1mes bAx//Gavq4CGcGCdj8Dy4mGSekk1OtrZGovDTPNlOk/KdV7Jy7EXdOabRyVGEXHby3X30y6d4yK6XBPIz5Esf0ynYbd6TRvknup2EFdy6ITrNBWBKgvwncFrrAe2gniIl3dF3hahVpPJHbBRf8H4PFgxrQ8OgDaeDpmYuuG9qaWK7XYsxI/68QjyuHvl41jzgIV5v8tUfefdhLsrqv+xLVlTtoYqpIQDBeMnv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Makes Dept able to track PG_locked waits and events. It's going to be useful in practice. See the following link that shows dept worked with PG_locked and can detect real issues: https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/ Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 2 + include/linux/page-flags.h | 125 +++++++++++++++++++++++++++++++++---- include/linux/pagemap.h | 7 ++- mm/filemap.c | 26 ++++++++ mm/mm_init.c | 2 + 5 files changed, 149 insertions(+), 13 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca33..d21b2e298cdd 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,6 +19,7 @@ #include #include #include +#include #include @@ -203,6 +204,7 @@ struct page { struct page *kmsan_shadow; struct page *kmsan_origin; #endif + struct dept_ext_wgen PG_locked_wgen; } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 4bf1c25fd1dc..74cbbf694c18 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -197,6 +197,61 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H +#ifdef CONFIG_DEPT +#include +#include + +extern struct dept_map PG_locked_map; + +/* + * Place the following annotations in its suitable point in code: + * + * Annotate dept_page_set_bit() around firstly set_bit*() + * Annotate dept_page_clear_bit() around clear_bit*() + * Annotate dept_page_wait_on_bit() around wait_on_bit*() + */ + +static inline void dept_page_set_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_request_event(&PG_locked_map, &p->PG_locked_wgen); +} + +static inline void dept_page_clear_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_event(&PG_locked_map, 1UL, _RET_IP_, __func__, &p->PG_locked_wgen); +} + +static inline void dept_page_wait_on_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_wait(&PG_locked_map, 1UL, _RET_IP_, __func__, 0, -1L); +} + +static inline void dept_folio_set_bit(struct folio *f, int bit_nr) +{ + dept_page_set_bit(&f->page, bit_nr); +} + +static inline void dept_folio_clear_bit(struct folio *f, int bit_nr) +{ + dept_page_clear_bit(&f->page, bit_nr); +} + +static inline void dept_folio_wait_on_bit(struct folio *f, int bit_nr) +{ + dept_page_wait_on_bit(&f->page, bit_nr); +} +#else +#define dept_page_set_bit(p, bit_nr) do { } while (0) +#define dept_page_clear_bit(p, bit_nr) do { } while (0) +#define dept_page_wait_on_bit(p, bit_nr) do { } while (0) +#define dept_folio_set_bit(f, bit_nr) do { } while (0) +#define dept_folio_clear_bit(f, bit_nr) do { } while (0) +#define dept_folio_wait_on_bit(f, bit_nr) do { } while (0) +#endif + #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); @@ -381,27 +436,51 @@ static __always_inline bool folio_test_##name(const struct folio *folio) \ #define FOLIO_SET_FLAG(name, page) \ static __always_inline void folio_set_##name(struct folio *folio) \ -{ set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + set_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_set_bit(folio, PG_##name); \ +} #define FOLIO_CLEAR_FLAG(name, page) \ static __always_inline void folio_clear_##name(struct folio *folio) \ -{ clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + clear_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_clear_bit(folio, PG_##name); \ +} #define __FOLIO_SET_FLAG(name, page) \ static __always_inline void __folio_set_##name(struct folio *folio) \ -{ __set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + __set_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_set_bit(folio, PG_##name); \ +} #define __FOLIO_CLEAR_FLAG(name, page) \ static __always_inline void __folio_clear_##name(struct folio *folio) \ -{ __clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + __clear_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_clear_bit(folio, PG_##name); \ +} #define FOLIO_TEST_SET_FLAG(name, page) \ static __always_inline bool folio_test_set_##name(struct folio *folio) \ -{ return test_and_set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + bool __ret = test_and_set_bit(PG_##name, folio_flags(folio, page)); \ + \ + if (!__ret) \ + dept_folio_set_bit(folio, PG_##name); \ + return __ret; \ +} #define FOLIO_TEST_CLEAR_FLAG(name, page) \ static __always_inline bool folio_test_clear_##name(struct folio *folio) \ -{ return test_and_clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + bool __ret = test_and_clear_bit(PG_##name, folio_flags(folio, page)); \ + \ + if (__ret) \ + dept_folio_clear_bit(folio, PG_##name); \ + return __ret; \ +} #define FOLIO_FLAG(name, page) \ FOLIO_TEST_FLAG(name, page) \ @@ -416,32 +495,54 @@ static __always_inline int Page##uname(const struct page *page) \ #define SETPAGEFLAG(uname, lname, policy) \ FOLIO_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline void SetPage##uname(struct page *page) \ -{ set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define CLEARPAGEFLAG(uname, lname, policy) \ FOLIO_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline void ClearPage##uname(struct page *page) \ -{ clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define __SETPAGEFLAG(uname, lname, policy) \ __FOLIO_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline void __SetPage##uname(struct page *page) \ -{ __set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __set_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_set_bit(page, PG_##lname); \ +} #define __CLEARPAGEFLAG(uname, lname, policy) \ __FOLIO_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline void __ClearPage##uname(struct page *page) \ -{ __clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + __clear_bit(PG_##lname, &policy(page, 1)->flags); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define TESTSETFLAG(uname, lname, policy) \ FOLIO_TEST_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline int TestSetPage##uname(struct page *page) \ -{ return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_set_bit(PG_##lname, &policy(page, 1)->flags);\ + if (!ret) \ + dept_page_set_bit(page, PG_##lname); \ + return ret; \ +} #define TESTCLEARFLAG(uname, lname, policy) \ FOLIO_TEST_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline int TestClearPage##uname(struct page *page) \ -{ return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } +{ \ + bool ret = test_and_clear_bit(PG_##lname, &policy(page, 1)->flags);\ + if (ret) \ + dept_page_clear_bit(page, PG_##lname); \ + return ret; \ +} #define PAGEFLAG(uname, lname, policy) \ TESTPAGEFLAG(uname, lname, policy) \ diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..a438d8f038de 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1008,7 +1008,12 @@ void folio_unlock(struct folio *folio); */ static inline bool folio_trylock(struct folio *folio) { - return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0))); + bool ret = !test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)); + + if (ret) + dept_page_set_bit(&folio->page, PG_locked); + + return likely(ret); } /* diff --git a/mm/filemap.c b/mm/filemap.c index 30de18c4fd28..ceb24a7ee0b1 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1108,6 +1109,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, if (flags & WQ_FLAG_CUSTOM) { if (test_and_set_bit(key->bit_nr, &key->folio->flags)) return -1; + dept_page_set_bit(&key->folio->page, key->bit_nr); flags |= WQ_FLAG_DONE; } } @@ -1191,6 +1193,7 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, if (wait->flags & WQ_FLAG_EXCLUSIVE) { if (test_and_set_bit(bit_nr, &folio->flags)) return false; + dept_page_set_bit(&folio->page, bit_nr); } else if (test_bit(bit_nr, &folio->flags)) return false; @@ -1201,6 +1204,9 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; +struct dept_map __maybe_unused PG_locked_map = DEPT_MAP_INITIALIZER(PG_locked_map, NULL); +EXPORT_SYMBOL(PG_locked_map); + static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { @@ -1212,6 +1218,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, unsigned long pflags; bool in_thrashing; + dept_page_wait_on_bit(&folio->page, bit_nr); + if (bit_nr == PG_locked && !folio_test_uptodate(folio) && folio_test_workingset(folio)) { delayacct_thrashing_start(&in_thrashing); @@ -1305,6 +1313,23 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, break; } + /* + * dept_page_set_bit() might have been called already in + * folio_trylock_flag(), wake_page_function() or somewhere. + * However, call it again to reset the wgen of dept to ensure + * dept_page_wait_on_bit() is called prior to + * dept_page_set_bit(). + * + * Remind dept considers all the waits between + * dept_page_set_bit() and dept_page_clear_bit() as potential + * event disturbers. Ensure the correct sequence so that dept + * can make correct decisions: + * + * wait -> acquire(set bit) -> release(clear bit) + */ + if (wait->flags & WQ_FLAG_DONE) + dept_page_set_bit(&folio->page, bit_nr); + /* * If a signal happened, this 'finish_wait()' may remove the last * waiter from the wait-queues, but the folio waiters bit will remain @@ -1481,6 +1506,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + dept_page_clear_bit(&folio->page, PG_locked); if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) folio_wake_bit(folio, PG_locked); } diff --git a/mm/mm_init.c b/mm/mm_init.c index 549e76af8f82..a0c9069d3740 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -570,6 +571,7 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, page_mapcount_reset(page); page_cpupid_reset_last(page); page_kasan_tag_reset(page); + dept_ext_wgen_init(&page->PG_locked_wgen); INIT_LIST_HEAD(&page->lru); #ifdef WANT_PAGE_VIRTUAL