From patchwork Mon Dec 16 15:04:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13909882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 525DBE77180 for ; Mon, 16 Dec 2024 15:05:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E03D66B0099; Mon, 16 Dec 2024 10:05:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D662B6B009A; Mon, 16 Dec 2024 10:05:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBA316B009B; Mon, 16 Dec 2024 10:05:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 92C476B0099 for ; Mon, 16 Dec 2024 10:05:24 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 478C6453D2 for ; Mon, 16 Dec 2024 15:05:24 +0000 (UTC) X-FDA: 82901143668.17.EADA68C Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf05.hostedemail.com (Postfix) with ESMTP id 026A4100024 for ; Mon, 16 Dec 2024 15:04:25 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WJ+M4YXE; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734361494; a=rsa-sha256; cv=none; b=md4NdRKw/h4dzZDjRdUTTbc9K24ezFXJHYDbUNKW4x2wE60p/qXs3rbn4E5sDD2vORBWVt SBZiaGvjYd3D5wS7q1u80mNvjtwQPHACpVJSRPtkyxeBKr1tI/dl8hkvwK7ORpIR21uV7D Z3e4dCqS0bGRw1oYGWpFVlbrLZfXpIc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WJ+M4YXE; spf=pass (imf05.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734361494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rflbkDAp6RcNtVda+UG2slAkUpkyNSVFqWrX1eClUo4=; b=4yZYrEkrhBLvIA5KfKlJneuMZV+nJIeMF/ekO0fODtNs/IJ0MBLBWJEa/QbEOwkIcXhx20 ZUUHy6SpTqdMxVNLHlONA4aF9RiNEC0ZLOO4ebW4SxlJqFZB1p+ozAPs45cn3UduLDEvJg Ozdvr13SHyYC6pPbAP4Xy+rOqaVaRs0= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-728e81257bfso3284929b3a.2 for ; Mon, 16 Dec 2024 07:05:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734361521; x=1734966321; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rflbkDAp6RcNtVda+UG2slAkUpkyNSVFqWrX1eClUo4=; b=WJ+M4YXEnr5GApkOUu2yqq4mIp+Cicx9//r5YA5J9gmNQgEZpGPdGmQaxC6fMDBTDA aRhBdanKTV5TsTTJuIfYmtQeLIqhhtnVZs9jNzrQWna/KXfOVqtk9fXxsMZvPwKjOLnQ o36kdvckk5X6rUUZLLKBog/i++Nuh09dIXirJdx+oPumEBkuZkZwymWG9uAXYQNY6e7o 2PRLM0xC7ti8BFMo7HyWM7XM+wUTopMGz9U7oYFbpOjMWoYXo1xunqUTKJZ3g7R5zWh/ VW1oY0SWbnYOaSLFmt/QjsAhhv2INrIpFE3c080BAyqvErPp48sTbDEUl59RhBCJQYVl AXuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734361521; x=1734966321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rflbkDAp6RcNtVda+UG2slAkUpkyNSVFqWrX1eClUo4=; b=WyAjXaENzNkHq1ymCLG5+TxJuNC/O4YAY5gw0jkPM3eFG59bQ7ZkMq06iIMnUCCJvn r3Dgs3p9ZpNUgEXmFaxnWEY9GKUNN0bKP8jGOvqTm7nOy78MUTjycJE8mz8FuZf3EdFK LOISgkikMcfKs5LFA9yTEOhwLf9kR9gT+Ln+fBemZBGenyQd93E4yPWz/DhPj9W7zu+T BP0ysLoGe4kBryoQJKQMHRy4iPGaj3moys1xFJieVobZDwwlKLWuPNH5ZQfkKm5V2CCi 0IZM8/SribzDQJQ03PJhxXnZhpRWmTsLbF5f8JOtmHSmB9ch5WygGdnKcEC7PnAylwoN XJ1g== X-Forwarded-Encrypted: i=1; AJvYcCXKEvEpykzRzU8K5Vppo9YwEmEWGbfbbZG8NekkNFRQ6GIMXlEokxRjJZwre7EUaM/LuiSyPrivow==@kvack.org X-Gm-Message-State: AOJu0YzxSVYkcq3sAwi6AEQLjHVPb76qgK+2mS83eMTcJLqtbxsDilW+ 9zrc2jaGFFEwOYSKTgxi+aEXuGLonjrbCItDrOa9/+b47Po/XuEZ X-Gm-Gg: ASbGncsobMMV5jNEoDH3c2IEzHL3/fh5WBEF+z5sFdhBCu6eFl17mDhrSNR9ytqZnp+ 580z8UMh9Y3E4WEBozk891tbUQLgYJHKQn5TD2SKCILGW3vN/2y6LXr0XFk4OvUn5jGOd0HplvJ v0563OFpYbXjh2yPCKckImlTlM5XZpmT7ULSeuqJRNsVtmw900pUOvATov67jyBxJ0EuRYKq86+ hkhhNNhS48bApJ1uk/woBqnjdrDIW3Mcu21X0XsWD7v3ff4nCCpqTpCA85EQ2GLges= X-Google-Smtp-Source: AGHT+IHTXzJ38/pN9SU9hmW2MVA6I4WUxZ9xDsFXfQGWf70PGg/75+A1Wb+l05U0+9QH4TJPtl8H3w== X-Received: by 2002:a05:6a00:13aa:b0:725:c8ea:b320 with SMTP id d2e1a72fcca58-7290c1eb757mr18879829b3a.14.1734361520606; Mon, 16 Dec 2024 07:05:20 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72918ad5ccdsm4845518b3a.74.2024.12.16.07.05.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2024 07:05:19 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , Vishal Moola , Alex Shi , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alex Shi Subject: [PATCH v9 mm-unstable 04/18] mm/zsmalloc: add and use pfn/zpdesc seeking funcs Date: Tue, 17 Dec 2024 00:04:35 +0900 Message-ID: <20241216150450.1228021-5-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241216150450.1228021-1-42.hyeyoo@gmail.com> References: <20241216150450.1228021-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 026A4100024 X-Stat-Signature: uipzdetxzqymsumjz6ibtxruhmuh1nrx X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734361465-855721 X-HE-Meta: U2FsdGVkX1+dCNzB3UzIOBW7hptuj+h5WLYkaAIq/3T4itmmE4CSkLntXw63UFzYiLHoWvXE14V3t/nKbPYYPitEkbKUFUSyG+2yCO7XkXbIyYG7ATqL4Y9RA7YKW7gG8JO8NW5rbkqhwPGcU/2H6326vSqWZD2BSX8f0h0bhPoP95xAOKKdIHp5GpXgSxDmOS2BgDz4jCKXaH4GVcVArfiA+enMKXjLmGHj0zNuI3jBUVKLzilOluaWRBz1+Vauj1Vv8GK6ITBf9shPJ6CUM7YzTjm1kXEUCR4/oHY86wbxTpxFfX4sBBJsaviFCt4RTjjYRt9oaUVMmRhmwDgxOYzn692eCgW+8N/Ts49FneQwflZCJRUyXbWVwS0Bq4qxieQn9cMQLb4lao+7xKFibDv/9yie8qxfdfy1bU9g2QLRrbqsrbs4uxDEsNplZ4FF5CMPikUCPJjuHkywwM6gaMWvIcPk8godCy6Hv7WtaiUyHisibhMt+q2re4v7s+zcBtXMnTDccBhxlA4a7FMwCwYdSz4NeD768iOjxAhTkYBQLQnpHdWcw6ciFcxxojRQhS4YFKG8Cnfid07/bObLzZzxUOcHB6pAmoTyafnJjyGqTL2ey8KjAEQ2CM9GaLtmCRWGA2IsGghKlQjvJr5MVPjGz1iImSakZ0tgttsU6Qx5RaQkkRouzJ2z36/whi5sCxpjjQwH7t/AGPUesWNbZk3FC95njPLJeRMuu6ZMFzyzw2SxYpHdKybhxzPNb6ubGbOzsNLS1vpB8IOzocNJisiKAQxsFCgCJOPpKOaqiJ+r3EtpTTq0CgYCncPU6Tyhdhp1xer4ei6D17Yi9ROcUGcKcGs53IywLuQQGACaMu5Pa25CG9SVMdHm3WNCegRVADAruH9fk2+yzVRumllMk43iBwmY1bd8PjOiymolZVYRdBRSsonHngvzpIfARme0MvItS/EheAZ7550VfBC /43np3ol 97JqMIIS6WA8fori3Rh9c3z4U6PrIc6/1+/BuuCNJFlcLLKrHi7zaDTEr7kgRLL4XZQ1+E5gd8rK+qRlLl013QfNpjyy64b5Rxn/Ilb38WGA4XZH8FpUdPjMdBNquFUioN9owuvLPY/Rz+az7hiEdqs0BNpNS6tjf2wR1KKOpfSEOJwxQQK9h4ppCJ+S3AZvBDbGIV/TsjVce+emPK9HP28IgTQ2kok7SqiPOEdlvq24dCbmwkmZ8DkHuGVw/6/pY1b/IrAwiCNolspcUK8Z0461FgJ0ioB5R3u1XEsE97bd2ytO6S9634C9Abe8A48ZeuFOKqFOrSUQfmC9seweo8j1J8Dkpa+EMyB0oVDpUSoaWPnBpVogV7s8q2++bvwVd6j1+W8RYFiuNz7lN7HodX/hehJdr8ubJsXpDsYl2I+1xCzNKBpAV7oOZrpqCYJ1yHSlrCEIqYKMsYTe32yF1TVipXZoUeB2K5WEer5I+E0pu4UBYNzYp2Bt287MTHW+U/Sb5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.009500, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add pfn_zpdesc(), pfn_zpdesc() and kmap_local_zpdesc(). Convert obj_to_location() to take zpdesc and also convert its users to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 14 ++++++++++ mm/zsmalloc.c | 75 ++++++++++++++++++++++++++------------------------- 2 files changed, 52 insertions(+), 37 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index c866758feec3..223d0381a444 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -134,4 +134,18 @@ static inline void zpdesc_put(struct zpdesc *zpdesc) folio_put(zpdesc_folio(zpdesc)); } +static inline void *kmap_local_zpdesc(struct zpdesc *zpdesc) +{ + return kmap_local_page(zpdesc_page(zpdesc)); +} + +static inline unsigned long zpdesc_pfn(struct zpdesc *zpdesc) +{ + return page_to_pfn(zpdesc_page(zpdesc)); +} + +static inline struct zpdesc *pfn_zpdesc(unsigned long pfn) +{ + return page_zpdesc(pfn_to_page(pfn)); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c038caaef3a8..e71da84ad73a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -757,15 +757,15 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) } /** - * obj_to_location - get (, ) from encoded object value + * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value - * @page: page object resides in zspage + * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index */ -static void obj_to_location(unsigned long obj, struct page **page, +static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc, unsigned int *obj_idx) { - *page = pfn_to_page(obj >> OBJ_INDEX_BITS); + *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS); *obj_idx = (obj & OBJ_INDEX_MASK); } @@ -1181,13 +1181,13 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, enum zs_mapmode mm) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; struct size_class *class; struct mapping_area *area; - struct page *pages[2]; + struct zpdesc *zpdescs[2]; void *ret; /* @@ -1200,8 +1200,8 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, /* It guarantees it can get zspage from handle safely */ read_lock(&pool->migrate_lock); obj = handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage = get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc_page(zpdesc)); /* * migration cannot move any zpages in this zspage. Here, class->lock @@ -1220,17 +1220,17 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, area->vm_mm = mm; if (off + class->size <= PAGE_SIZE) { /* this object is contained entirely within a page */ - area->vm_addr = kmap_local_page(page); + area->vm_addr = kmap_local_zpdesc(zpdesc); ret = area->vm_addr + off; goto out; } /* this object spans two pages */ - pages[0] = page; - pages[1] = get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] = zpdesc; + zpdescs[1] = get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); - ret = __zs_map_object(area, (struct zpdesc **)pages, off, class->size); + ret = __zs_map_object(area, zpdescs, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret += ZS_HANDLE_SIZE; @@ -1242,7 +1242,7 @@ EXPORT_SYMBOL_GPL(zs_map_object); void zs_unmap_object(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; @@ -1250,8 +1250,8 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) struct mapping_area *area; obj = handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage = get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc_page(zpdesc)); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1259,13 +1259,13 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) if (off + class->size <= PAGE_SIZE) kunmap_local(area->vm_addr); else { - struct page *pages[2]; + struct zpdesc *zpdescs[2]; - pages[0] = page; - pages[1] = get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] = zpdesc; + zpdescs[1] = get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); - __zs_unmap_object(area, (struct zpdesc **)pages, off, class->size); + __zs_unmap_object(area, zpdescs, off, class->size); } local_unlock(&zs_map_area.lock); @@ -1406,23 +1406,24 @@ static void obj_free(int class_size, unsigned long obj) { struct link_free *link; struct zspage *zspage; - struct page *f_page; + struct zpdesc *f_zpdesc; unsigned long f_offset; unsigned int f_objidx; void *vaddr; - obj_to_location(obj, &f_page, &f_objidx); + + obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset = offset_in_page(class_size * f_objidx); - zspage = get_zspage(f_page); + zspage = get_zspage(zpdesc_page(f_zpdesc)); - vaddr = kmap_local_page(f_page); + vaddr = kmap_local_zpdesc(f_zpdesc); link = (struct link_free *)(vaddr + f_offset); /* Insert this object in containing zspage's freelist */ if (likely(!ZsHugePage(zspage))) link->next = get_freeobj(zspage) << OBJ_TAG_BITS; else - f_page->index = 0; + f_zpdesc->handle = 0; set_freeobj(zspage, f_objidx); kunmap_local(vaddr); @@ -1467,7 +1468,7 @@ EXPORT_SYMBOL_GPL(zs_free); static void zs_object_copy(struct size_class *class, unsigned long dst, unsigned long src) { - struct page *s_page, *d_page; + struct zpdesc *s_zpdesc, *d_zpdesc; unsigned int s_objidx, d_objidx; unsigned long s_off, d_off; void *s_addr, *d_addr; @@ -1476,8 +1477,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, s_size = d_size = class->size; - obj_to_location(src, &s_page, &s_objidx); - obj_to_location(dst, &d_page, &d_objidx); + obj_to_location(src, &s_zpdesc, &s_objidx); + obj_to_location(dst, &d_zpdesc, &d_objidx); s_off = offset_in_page(class->size * s_objidx); d_off = offset_in_page(class->size * d_objidx); @@ -1488,8 +1489,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, if (d_off + class->size > PAGE_SIZE) d_size = PAGE_SIZE - d_off; - s_addr = kmap_local_page(s_page); - d_addr = kmap_local_page(d_page); + s_addr = kmap_local_zpdesc(s_zpdesc); + d_addr = kmap_local_zpdesc(d_zpdesc); while (1) { size = min(s_size, d_size); @@ -1514,17 +1515,17 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, if (s_off >= PAGE_SIZE) { kunmap_local(d_addr); kunmap_local(s_addr); - s_page = get_next_page(s_page); - s_addr = kmap_local_page(s_page); - d_addr = kmap_local_page(d_page); + s_zpdesc = get_next_zpdesc(s_zpdesc); + s_addr = kmap_local_zpdesc(s_zpdesc); + d_addr = kmap_local_zpdesc(d_zpdesc); s_size = class->size - written; s_off = 0; } if (d_off >= PAGE_SIZE) { kunmap_local(d_addr); - d_page = get_next_page(d_page); - d_addr = kmap_local_page(d_page); + d_zpdesc = get_next_zpdesc(d_zpdesc); + d_addr = kmap_local_zpdesc(d_zpdesc); d_size = class->size - written; d_off = 0; } @@ -1763,7 +1764,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, struct zs_pool *pool; struct size_class *class; struct zspage *zspage; - struct page *dummy; + struct zpdesc *dummy; void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle;