From patchwork Mon Dec 16 15:04:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13909880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE8B8E77183 for ; Mon, 16 Dec 2024 15:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79BC46B0095; Mon, 16 Dec 2024 10:05:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7253A6B0096; Mon, 16 Dec 2024 10:05:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 550566B0098; Mon, 16 Dec 2024 10:05:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2F4A26B0095 for ; Mon, 16 Dec 2024 10:05:18 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D9AA84532B for ; Mon, 16 Dec 2024 15:05:17 +0000 (UTC) X-FDA: 82901144634.22.4BA4EA4 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf18.hostedemail.com (Postfix) with ESMTP id 82A4B1C001F for ; Mon, 16 Dec 2024 15:05:02 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eiPPgVL7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734361502; a=rsa-sha256; cv=none; b=WOhG+127DacXMObA6SOoKh5nxh233TWJVoo9s0SKxd+UXQXuJkoP07zkqeKy/FcyLUzDhe ua/CXuVIXh//M9Jt+6O3jRvJsXP157086/0hWOLDLucYyBNmdz/JwhRHTK+p7XQOZpOWzI I6rxvsCcskQPzJFF1vs/V74lQiT0STA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eiPPgVL7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734361502; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BCMEvH8Krrbs2dY96UiQdD2oaiZyPKRNYnh7gL+Mupw=; b=QnPwmRLWbKhzqx1OpyCDqnyiB4BXpbGdCtACiLoRFrlJaw03KMaIR1YqdoTk3RxruTgd3D t6aNckt4Z1PNlp9xQvEe0M8eUXRvJ6Dw0kHA7UdMbKnYtXQnCS7bQDPT7DUOVzIakBp4vH d2c7t81T45LW+mqEtk96vK+ysm7aYRA= Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-725f4623df7so3617661b3a.2 for ; Mon, 16 Dec 2024 07:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734361514; x=1734966314; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BCMEvH8Krrbs2dY96UiQdD2oaiZyPKRNYnh7gL+Mupw=; b=eiPPgVL7pAgynN5E9viQJTnq6aPE6Jln6g1RM9Cw9WYKNyGS+QI6mWVzMyjVzBc0YX 12nebhdo4CNlEKmVRwQZu4tPLkh5B4lNKMuc5vyRKQyyeLdRNTo3I5Hq+KIFrjzMU/Qj 3o27mQRQzfViskJ7lNaeg0JaQZKi1ogJXJUg8nfr4eaIbKCkdkhhKcflCUNOVCr+FmFC Pb4nxFgrnhK23grIhZaKrJbFlU/ZsqP2Dwpw6YcS+MH3mw3wSDNpEqmSIvdUlHr4Av1A Ox7C5OOZtQrPAWtmcSzplD2blEa3eY5bX6mErgZhAfenLnoGQbscaTA95I8zsMkz5+kb SIjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734361514; x=1734966314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BCMEvH8Krrbs2dY96UiQdD2oaiZyPKRNYnh7gL+Mupw=; b=gldqpx8zeIPcYCHlKr7l5SwzSBPtS84a3TfAnLutYG+kezPgbkzgJQwIXIojF8M3Z1 1rdyo6IfJAHeAQAKNhAbINj54sowBJudi+eFOhi+ype8ma5UwMZ6kraxQqa9LDxZN2dp pWm8sB8C0aiOG6C90R+hZ2gPuBMJu5fE3SPHyZsi6SHlwYj888ppyBS0LotU41YjhjHU 0oTfFmx2Dx39tpBvo/wzzr43Bu+WTfQpp00B4PxAf48Qo4AcySKokZ9yDOY5oOAyheSD fJoISqiIqMJpSOGuB3bMyTzBZNBGFq66Eg08Wb7g5VIMFokzDY1pUPqgaMwVaFMCqKSO kxEw== X-Forwarded-Encrypted: i=1; AJvYcCXuzcNW3q948l8BrRqF0o0jxbm+TCp2SQep5mAr8FLLHYLDpj/MFyHa4PEYhc/79Jns1+LEfmywrw==@kvack.org X-Gm-Message-State: AOJu0YzHg8Tguvq25Ab+uISkMixOB3S3l28V83CVz9J0eOIL1t8I0Ugn ui9FY2sDIJ6L0dWRtBrsYvuPAH9Z1YqUx8O4oeTyNgeMysHcqP70 X-Gm-Gg: ASbGncsLHwnfeFCQnC32qL0YKVpUx1yoCXdltlrN6oFImM03tUUpyNPeKYoOdxJucrG qUbrCVYyecN0kCcAVI0+D4Q3DZYAhdmfzY8fCMrF41Aw4PQsnrH77NSEmOtkQ0VQvQasK7nMfC5 DDZSBpI14hRORh099JpPcldrS2QkJMcrBIpRDY046Gz38nrIIRELWn9JKaKN80EDMwnOyudieqF 6DzOBegrzK2r8vQggUdfOIrezVvxsVi56Xgutfi4fjGU5RJHmrvjUqbOuVuVh7+OWI= X-Google-Smtp-Source: AGHT+IHenymbq1naD5Q9pzchgJ5RTMcWmEHccxP5+wPQ45I2sXUsZ/pUpfYR4mUKVV0wf0JBoyugPQ== X-Received: by 2002:a05:6a00:929b:b0:724:e75b:22d1 with SMTP id d2e1a72fcca58-7290c25d977mr17443660b3a.16.1734361514385; Mon, 16 Dec 2024 07:05:14 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72918ad5ccdsm4845518b3a.74.2024.12.16.07.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2024 07:05:13 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , Vishal Moola , Alex Shi , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alex Shi Subject: [PATCH v9 mm-unstable 02/18] mm/zsmalloc: use zpdesc in trylock_zspage()/lock_zspage() Date: Tue, 17 Dec 2024 00:04:33 +0900 Message-ID: <20241216150450.1228021-3-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241216150450.1228021-1-42.hyeyoo@gmail.com> References: <20241216150450.1228021-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 82A4B1C001F X-Stat-Signature: bmohq9w4e8897ei4wxh3uxrpmx7azx8m X-HE-Tag: 1734361502-127897 X-HE-Meta: U2FsdGVkX191hZ4uaFuBQXM6rxTejxwi0wc9qVCPBG8K+SsgqN8ozeZy5/HVw1R1V0gFiJR4ajH6NmEV35QMbmCvIen1OxpO8NX9G17IKURrdpeW+6/clksQqbTHEBq7m0Hmc4IZoskzMdD9tu59QnYLw+3q9EGf3AQpKpar4tgrDmH4m9odkaym76UHCSfQ+CXkldUCT+7BNNl5onWb+druSa6zEHm/HWyTn/iDcOk+FbYIC/WOJ28u54BAhLT/IE9nRTIOmgV6HJwht5L29gagptb18CyIQtCXavKkS7IA5CgL1nDeKZhyMbSovP9fsNyRttLHd7CMqQvpdVssSEBlbthDs0e7Rl/gOrKzUxo3wQtvYe4aLbCVBbls3T/zAL5AmdUwqOWnw7LYjmavJD26DrniAgydEt92CHskVWaR0jnJwpQ/Vs58ll2rz6dL9/4je/22fosrRbmjqNaq1/e/I8nw90laTnXu5ABqGvO4SCFtfh5YH/BOAeZehUvNbZL44vMDgW0Iq5AE29Shu+q44MlXmDsikImhy1JQWvpHXdiOwsdtVvS65b+pSbr+IZTOzngF+iedgImyfSWlsnZh3+mKNjyomZGHKyDcitnNnaZkErMk+gDLcyzqd3W1zT7WmH7awvPk1B1IalfF7GJPt1qSRVry2/hmosHF+DDzUAbCsG0KoAhwR8gX/uAyhlREyc0fyBSwPMjj6VA8khbF/Bx67L7qbqTECyxcEtUTWKdkQaHz0N5dqT3k+aCizMnO+YcGG+K3oFCdiYCOdpveP7MEnuenPq15uNDgE3S3uJUXzNKNpqfzCVE3FNRNsbeVwcZ6zxnLehMmNpRc8YhTNdAi6nibkc6Hga3KLGdT6caWT+BnHd5v+B8SDCpts8PP6PjOd/+R7+pRzlsdl6B7VRNmXzS2zF1h5P0PoVmHq78ae16zTcZqBGhfga/1lSMrrbTQYLlhs32zwHQ LgDtt7L+ DfDn29htytsXmdVOMHBuv+twhs5zXLyC6s8kFdn1xqTiGn/WNOrjA9juSkpPMJqaNqhH/9AZQj/o2kKuQ4UFXA/zjGgaDIxUKH+1K7dkKhCKpCLUZvtcmRja5rSM2/0hI1bzuBakGxTlwJXeplMjFxF+SA85GxtvDHlv02qUgUUAe/2dKDbbRFyKGr15uf9ly9leu5ibKcemFRYgvOY4RbNTlydSubsrzeZr5mvTPo2OlOccpd9Ar8VCfV7aKZ4J21QLViRR77KtysLLi6DJqvUu+jf1dN005rWmQFG7+xMG3yeyrIg25ukrXs/C3VNSpz/pRM6sqcPFSSJq4UApSuBUfiKOLuPvE3IR551E8KqdRRzbyo0Ht8sbZZcksMzEsNuGW6/QNNP5E/q25N5r9ZHLajPQuAPacO6lv/AcI2TxvWp45eB66eJqHqvJfXBlbUOGQaFpKahcsfBtZu/YyoO2cn1WHGRxqWdyQ015Dw8Kg4FQOSKdArSXZzOT6g5xsRIKdoyY/ig7p+buyyENOIA2x17ZwQQo7TBuoVwYJji+ygAs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.018469, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alex Shi Convert trylock_zspage() and lock_zspage() to use zpdesc. To achieve that, introduce a couple of helper functions: - zpdesc_lock() - zpdesc_unlock() - zpdesc_trylock() - zpdesc_wait_locked() - zpdesc_get() - zpdesc_put() Here we use the folio version of functions for 2 reasons. First, zswap.zpool currently only uses order-0 pages and using folio could save some compound_head checks. Second, folio_put could bypass devmap checking that we don't need. BTW, thanks Intel LKP found a build warning on the patch. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/zpdesc.h | 30 ++++++++++++++++++++++++ mm/zsmalloc.c | 64 ++++++++++++++++++++++++++++++++++----------------- 2 files changed, 73 insertions(+), 21 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index e0852498aecf..c866758feec3 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -104,4 +104,34 @@ static_assert(sizeof(struct zpdesc) <= sizeof(struct page)); const struct page *: (const struct zpdesc *)(p), \ struct page *: (struct zpdesc *)(p))) +static inline void zpdesc_lock(struct zpdesc *zpdesc) +{ + folio_lock(zpdesc_folio(zpdesc)); +} + +static inline bool zpdesc_trylock(struct zpdesc *zpdesc) +{ + return folio_trylock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_unlock(struct zpdesc *zpdesc) +{ + folio_unlock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_wait_locked(struct zpdesc *zpdesc) +{ + folio_wait_locked(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_get(struct zpdesc *zpdesc) +{ + folio_get(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_put(struct zpdesc *zpdesc) +{ + folio_put(zpdesc_folio(zpdesc)); +} + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 00d111f011be..51f4a9b78023 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -428,13 +428,17 @@ static __maybe_unused int is_first_page(struct page *page) return PagePrivate(page); } +static inline bool is_first_zpdesc(struct zpdesc *zpdesc) +{ + return PagePrivate(zpdesc_page(zpdesc)); +} + /* Protected by class->lock */ static inline int get_zspage_inuse(struct zspage *zspage) { return zspage->inuse; } - static inline void mod_zspage_inuse(struct zspage *zspage, int val) { zspage->inuse += val; @@ -448,6 +452,14 @@ static inline struct page *get_first_page(struct zspage *zspage) return first_page; } +static struct zpdesc *get_first_zpdesc(struct zspage *zspage) +{ + struct zpdesc *first_zpdesc = zspage->first_zpdesc; + + VM_BUG_ON_PAGE(!is_first_zpdesc(first_zpdesc), zpdesc_page(first_zpdesc)); + return first_zpdesc; +} + #define FIRST_OBJ_PAGE_TYPE_MASK 0xffffff static inline unsigned int get_first_obj_offset(struct page *page) @@ -734,6 +746,16 @@ static struct page *get_next_page(struct page *page) return (struct page *)page->index; } +static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) +{ + struct zspage *zspage = get_zspage(zpdesc_page(zpdesc)); + + if (unlikely(ZsHugePage(zspage))) + return NULL; + + return zpdesc->next; +} + /** * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value @@ -803,11 +825,11 @@ static void reset_page(struct page *page) static int trylock_zspage(struct zspage *zspage) { - struct page *cursor, *fail; + struct zpdesc *cursor, *fail; - for (cursor = get_first_page(zspage); cursor != NULL; cursor = - get_next_page(cursor)) { - if (!trylock_page(cursor)) { + for (cursor = get_first_zpdesc(zspage); cursor != NULL; cursor = + get_next_zpdesc(cursor)) { + if (!zpdesc_trylock(cursor)) { fail = cursor; goto unlock; } @@ -815,9 +837,9 @@ static int trylock_zspage(struct zspage *zspage) return 1; unlock: - for (cursor = get_first_page(zspage); cursor != fail; cursor = - get_next_page(cursor)) - unlock_page(cursor); + for (cursor = get_first_zpdesc(zspage); cursor != fail; cursor = + get_next_zpdesc(cursor)) + zpdesc_unlock(cursor); return 0; } @@ -1635,7 +1657,7 @@ static int putback_zspage(struct size_class *class, struct zspage *zspage) */ static void lock_zspage(struct zspage *zspage) { - struct page *curr_page, *page; + struct zpdesc *curr_zpdesc, *zpdesc; /* * Pages we haven't locked yet can be migrated off the list while we're @@ -1647,24 +1669,24 @@ static void lock_zspage(struct zspage *zspage) */ while (1) { migrate_read_lock(zspage); - page = get_first_page(zspage); - if (trylock_page(page)) + zpdesc = get_first_zpdesc(zspage); + if (zpdesc_trylock(zpdesc)) break; - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); } - curr_page = page; - while ((page = get_next_page(curr_page))) { - if (trylock_page(page)) { - curr_page = page; + curr_zpdesc = zpdesc; + while ((zpdesc = get_next_zpdesc(curr_zpdesc))) { + if (zpdesc_trylock(zpdesc)) { + curr_zpdesc = zpdesc; } else { - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); migrate_read_lock(zspage); } }