From patchwork Mon Dec 16 15:04:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13909879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6253E7717F for ; Mon, 16 Dec 2024 15:05:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5EEB26B008C; Mon, 16 Dec 2024 10:05:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 576306B0093; Mon, 16 Dec 2024 10:05:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EFEE6B0095; Mon, 16 Dec 2024 10:05:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1FDB96B008C for ; Mon, 16 Dec 2024 10:05:16 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9EC2614110C for ; Mon, 16 Dec 2024 15:05:15 +0000 (UTC) X-FDA: 82901144466.10.ED0536E Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf20.hostedemail.com (Postfix) with ESMTP id BDCCA1C0026 for ; Mon, 16 Dec 2024 15:04:42 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MblSN8TZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734361499; a=rsa-sha256; cv=none; b=iYxfIEpj44vvbgARsS/LGvZMNbRji7qAKV35sDfWEQdSJj2yxYY8WSWKrEf7pIocGpRdKf aVn5QCBRpHFmPLuDMwsrVBocwJdnk/TgDiyg4HdUP3CZRtFczCvuH1Z+gYwugrqLFg4pg5 f9SCEeVwCSNOwRUr4FaNj9jkG4IEqJk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MblSN8TZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734361499; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KJPcQRcezA/6SvJATreo3LdPfQhC3fUseHgH36/RUEg=; b=yF/64dhRCW7MrwgyqabR7UKp1L7LJ9nP02SjEVumkG1lGpU6HO1KQOFaSyG8N4CUQHQJuD HSA4PkVHWLyriz/IFFLGlsh1im/ugy6qKo40TbbXu143J5B4nJOxu2M66ZWlZBylDMXjAy hGWL36f0j2udOusP4/L7Kd4yoYc7T3A= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-728e3826211so3311612b3a.0 for ; Mon, 16 Dec 2024 07:05:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734361512; x=1734966312; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KJPcQRcezA/6SvJATreo3LdPfQhC3fUseHgH36/RUEg=; b=MblSN8TZtWYrt1M521b0zz+XhEcrBy0Rhd9oAiMzrwiTutht2Jhwmc1Lj1m6Bn65Eb a3wvhPXXy2nJsf4ZCjJYYCV2SkDJBsz7fz/UbZTpaFhwx/cWl22cw54vzBIP1evaIb6p aTnEEIpzQTHPCVER5kYtJEirTtI2tLvBDePIWWkX67s0N0SbtXLsZ0Xdj77TElmLwyIB ib4Sqlh677/Co4a9WYHBClx2pTy5tTP16JhFZibcwjLWmyOQwpiDrsyMs9A49caKglh7 7FFtTEM6IpMHC0HD0oUqjKJk4DrV1Akh5pXEBZjLQmXiGGVxI/CT20NL02+j2RtbsK7A v4ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734361512; x=1734966312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KJPcQRcezA/6SvJATreo3LdPfQhC3fUseHgH36/RUEg=; b=DrXGe/o7WFbYrN9vWy0/Ydc9zUpYFTkEt4UhAFgE0awmYSicZJzW2xy9HnqoTc+s/7 QAsKEZKbapTXu2r3IMAZxYh9geE0k7TopgH9YYndBJsJ97sCKBej+cMtULDeJcJyDLrD +Zg1TXqAJFhAuiS1UEFcstQd+GZ4CwH/2R4wjN+VrfsVa9IjzVx9WB3vQzJ308PDHJkb fZm4flGQIUvoWefS4jHyEpA1sWuvyLkHR9DI8MEUmnYbYUzXNgWzFkUbjxtuSz8zdwV/ 0irZJkChLR5FtfwSqL5nJHBTFXpSJljmxANaiNlocyhfBJp0qptMKGt34GDfwTWAQ2CH SsnQ== X-Forwarded-Encrypted: i=1; AJvYcCXLp6Hsdc/9GtWt4ZsZeeqoT/1rG5h8lLcplpvQBzREs922/JuYLK0bGTZFP3SaUEN1Nwasnwg9qQ==@kvack.org X-Gm-Message-State: AOJu0YxTxI/EYpVcSxg01co7yzSghYI/W4LfDdWUflMSYwIBms4dqw6H jZteHnvaCf5IZF+1L4FwjAz1lDcLNosJT1jQOLhYXEsbOuZz2lI8 X-Gm-Gg: ASbGncuMMXW+8PTWr1jCWEqX90hECQETFl0KxlX0oFg/M+TaypSgIte/YZiyBQE127f ZNHbdsmzknP9PoTlJUAhZkXZI+oAj6SFSHS21zan81D5HU2ZXWyGDXn1qBENoVKZbokwOFcqGBF +7KXWHawKh3K6IbPpfc5k3TCdPL6XrA9krjIu8TfyOM6YZiCl4K6GXyYxCRnmWdCIptjjj9xgoT mp94eK5fBnOVudHiSga+I3BXLBqqYXuXknn6ozSntH6MHVOfaT8JMKBaZBwpY3b5Xo= X-Google-Smtp-Source: AGHT+IEATRy0BZlRxn8Dcu/3lz3Ry4v8RprlwhYAwyyxFpZKyNVsuPuD/mPrsn3MyrD2Fn/ugMKziA== X-Received: by 2002:a05:6a00:338e:b0:725:eb85:f802 with SMTP id d2e1a72fcca58-7290c11587cmr18819402b3a.2.1734361511273; Mon, 16 Dec 2024 07:05:11 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72918ad5ccdsm4845518b3a.74.2024.12.16.07.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2024 07:05:10 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , Vishal Moola , Alex Shi , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alex Shi Subject: [PATCH v9 mm-unstable 01/18] mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool Date: Tue, 17 Dec 2024 00:04:32 +0900 Message-ID: <20241216150450.1228021-2-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241216150450.1228021-1-42.hyeyoo@gmail.com> References: <20241216150450.1228021-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BDCCA1C0026 X-Stat-Signature: hid7qf49mjpjaxynfngnmetobgrabo45 X-HE-Tag: 1734361482-190866 X-HE-Meta: U2FsdGVkX1/Nj3uIPDe5g7xkKIwWyNrV8h2dmk1IAoR/R/1tDHc1xbmb1MSJ8uSoQ1zqHeUUCYRcWbUyzJIRS0qZgPNEsFe/pIURhW6Xv/mjDEbZbUpn+QEZIdsLGshjF7aSwIg2h6ACcMX9ZqjzMX8O2fP2YzLs0eNZX9J80pnFk++TqmcAKjfnYuXonm3BL+kEjMVQNTP2VG7Btoml9RdWsuRneeSWypbGjyvmSpInTDCmBbADYBzo6FrC+Eeh5YSk5pg1qKNA5qEZsnTqfNhVrKfcy3luWwvOqLn9FN10a6sCZ/jXrzIq96oFpa5y7ZDXnQVUSn1eqBhAnVMn07QqrL05mQhDzqNfv0w38sFOIeRMggp9NAG8uS5+KTz9fEUxj5aAeh1kcYiWqD1Sj6ZyrbZPQJYYaXVckBtlS4rR9g92kgykoxBuAgQuc8NLajFcl+n6/zsTl0LAfYmnZRc9P99B9e927ikoKEGE8FU6X7mlg9TjbDK6bNvWFG4b/j9kwXJlKapVNslKjLXyUPAqj0za6QO6iC//up3sXz+jXzaqqp4iS77cLr9eAmduSfnITRnK505OrtJucbim0Y6af/UnDDnKrY6MPRTSz716aE3f3Ejmjswto1OpW1QHlVmTs+/cdDXAVyUXKdonvUGelJntNVrbKSSUCOFwy1cNnhE9wU0ZhWaUzZXXdWMxHoSu490DWSDS2qhLqEcFOkX/4QzxXrvzUq2Ya4/XRcIs2j834vu6kMBrPfRr8hmh/IGN0YTTP52+n7rSQKq5MnzZQ/A0fYwjValpfVoEUc/sUNnjrnp9hvH97IvTX36dZp90QRLSqBaJkT14TO/WBn5KhrmcOode5+huhz8vjiAX23fv+iSbBIpe9cHgnSIlS6L4VkhDSql/53cd1oq495ZU6Te/3DonpckwbIeNBl+na8WXZ5W58y/TL5Tq//YyHoBKqORFCUXkArYfaME IwjmlllA uA5xlLZ9hcg/uL5BPm1HIaC389tGpt2/+T843Br/BrETF04lJ3kb/FPEw3HeTdzPdqsCLReW7J9kgJzK23lcNfkEaKDuhFtryQWsCao0fXh+WN4iDh5k4eLaudn09alhILOdd8fR7i70kN1B7ZSBIhRNVH3qNWxOB+mhzcR+R9I1QnGoVC0CRdwoKoUWbEhnBBuvo8OEwdDV7VjIeiwuB4aPQKkEL7i8BxwREuLcZYJb2FJg148n/2izoYmA8bBj1p99j4sBUh9V9tCWjGS9suf2XwgWyse7nGm2oPpepqHLVBQXUkYEfA/TDYkHxD9WHTO8KItVbxjNEPFWV8njqWp8k/qxgQ/5gB5SnAxPdD14Uepp6U/tmuTldtMOsSZiPYwhr6X+paAiOxJrBFRPbxveMugWRpdpbfzpjHhEE5RzIC9rfLKhL+1DpeC4CapQgqHvDf1cOv75a7l/9tpJJ/7SATWGMRXoa1pZjo4IqFWMTytRpY5yQkukAmZ3UXylzCtpw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000126, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alex Shi The 1st patch introduces new memory descriptor zpdesc and renames zspage.first_page to zspage.first_zpdesc, with no functional change. We removed the comment about PG_owner_priv_1 since it is no longer used after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage"). [42.hyeyoo: rework comments a little bit] Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/zpdesc.h | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/zsmalloc.c | 28 +++---------- 2 files changed, 112 insertions(+), 23 deletions(-) create mode 100644 mm/zpdesc.h diff --git a/mm/zpdesc.h b/mm/zpdesc.h new file mode 100644 index 000000000000..e0852498aecf --- /dev/null +++ b/mm/zpdesc.h @@ -0,0 +1,107 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* zpdesc.h: zswap.zpool memory descriptor + * + * Written by Alex Shi + * Hyeonggon Yoo <42.hyeyoo@gmail.com> + */ +#ifndef __MM_ZPDESC_H__ +#define __MM_ZPDESC_H__ + +/* + * struct zpdesc - Memory descriptor for zpool memory. + * @flags: Page flags, mostly unused by zsmalloc. + * @lru: Indirectly used by page migration. + * @movable_ops: Used by page migration. + * @next: Next zpdesc in a zspage in zsmalloc zpool. + * @handle: For huge zspage in zsmalloc zpool. + * @zspage: Points to the zspage this zpdesc is a part of. + * @first_obj_offset: First object offset in zsmalloc zpool. + * @_refcount: The number of references to this zpdesc. + * + * This struct overlays struct page for now. Do not modify without a good + * understanding of the issues. In particular, do not expand into the overlap + * with memcg_data. + * + * Page flags used: + * * PG_private identifies the first component page. + * * PG_locked is used by page migration code. + */ +struct zpdesc { + unsigned long flags; + struct list_head lru; + unsigned long movable_ops; + union { + struct zpdesc *next; + unsigned long handle; + }; + struct zspage *zspage; + /* + * Only the lower 24 bits are available for offset, limiting a page + * to 16 MiB. The upper 8 bits are reserved for PGTY_zsmalloc. + * + * Do not access this field directly. + * Instead, use {get,set}_first_obj_offset() helpers. + */ + unsigned int first_obj_offset; + atomic_t _refcount; +}; +#define ZPDESC_MATCH(pg, zp) \ + static_assert(offsetof(struct page, pg) == offsetof(struct zpdesc, zp)) + +ZPDESC_MATCH(flags, flags); +ZPDESC_MATCH(lru, lru); +ZPDESC_MATCH(mapping, movable_ops); +ZPDESC_MATCH(index, next); +ZPDESC_MATCH(index, handle); +ZPDESC_MATCH(private, zspage); +ZPDESC_MATCH(page_type, first_obj_offset); +ZPDESC_MATCH(_refcount, _refcount); +#undef ZPDESC_MATCH +static_assert(sizeof(struct zpdesc) <= sizeof(struct page)); + +/* + * zpdesc_page - The first struct page allocated for a zpdesc + * @zp: The zpdesc. + * + * A convenience wrapper for converting zpdesc to the first struct page of the + * underlying folio, to communicate with code not yet converted to folio or + * struct zpdesc. + * + */ +#define zpdesc_page(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct page *)(zp), \ + struct zpdesc *: (struct page *)(zp))) + +/** + * zpdesc_folio - The folio allocated for a zpdesc + * @zpdesc: The zpdesc. + * + * Zpdescs are descriptors for zpool memory. The zpool memory itself is + * allocated as folios that contain the zpool objects, and zpdesc uses specific + * fields in the first struct page of the folio - those fields are now accessed + * by struct zpdesc. + * + * It is occasionally necessary convert to back to a folio in order to + * communicate with the rest of the mm. Please use this helper function + * instead of casting yourself, as the implementation may change in the future. + */ +#define zpdesc_folio(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct folio *)(zp), \ + struct zpdesc *: (struct folio *)(zp))) +/** + * page_zpdesc - Converts from first struct page to zpdesc. + * @p: The first (either head of compound or single) page of zpdesc. + * + * A temporary wrapper to convert struct page to struct zpdesc in situations + * where we know the page is the compound head, or single order-0 page. + * + * Long-term ideally everything would work with struct zpdesc directly or go + * through folio to struct zpdesc. + * + * Return: The zpdesc which contains this page + */ +#define page_zpdesc(p) (_Generic((p), \ + const struct page *: (const struct zpdesc *)(p), \ + struct page *: (struct zpdesc *)(p))) + +#endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 64b66a4d3e6e..00d111f011be 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -13,24 +13,6 @@ * Released under the terms of GNU General Public License Version 2.0 */ -/* - * Following is how we use various fields and flags of underlying - * struct page(s) to form a zspage. - * - * Usage of struct page fields: - * page->private: points to zspage - * page->index: links together all component pages of a zspage - * For the huge page, this is always 0, so we use this field - * to store handle. - * page->page_type: PGTY_zsmalloc, lower 24 bits locate the first object - * offset in a subpage of a zspage - * - * Usage of struct page flags: - * PG_private: identifies the first component page - * PG_owner_priv_1: identifies the huge component page - * - */ - #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt /* @@ -67,6 +49,7 @@ #include #include #include +#include "zpdesc.h" #define ZSPAGE_MAGIC 0x58 @@ -254,7 +237,7 @@ struct zspage { }; unsigned int inuse; unsigned int freeobj; - struct page *first_page; + struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; rwlock_t lock; @@ -459,7 +442,7 @@ static inline void mod_zspage_inuse(struct zspage *zspage, int val) static inline struct page *get_first_page(struct zspage *zspage) { - struct page *first_page = zspage->first_page; + struct page *first_page = zpdesc_page(zspage->first_zpdesc); VM_BUG_ON_PAGE(!is_first_page(first_page), first_page); return first_page; @@ -952,7 +935,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage, set_page_private(page, (unsigned long)zspage); page->index = 0; if (i == 0) { - zspage->first_page = page; + zspage->first_zpdesc = page_zpdesc(page); SetPagePrivate(page); if (unlikely(class->objs_per_zspage == 1 && class->pages_per_zspage == 1)) @@ -1317,8 +1300,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, /* record handle in the header of allocated chunk */ link->handle = handle | OBJ_ALLOCATED_TAG; else - /* record handle to page->index */ - zspage->first_page->index = handle | OBJ_ALLOCATED_TAG; + zspage->first_zpdesc->handle = handle | OBJ_ALLOCATED_TAG; kunmap_local(vaddr); mod_zspage_inuse(zspage, 1);