From patchwork Thu Jul 13 04:20:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13311278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32501C001DF for ; Thu, 13 Jul 2023 04:21:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C80E08E0009; Thu, 13 Jul 2023 00:21:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C09708E0001; Thu, 13 Jul 2023 00:21:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5CD28E0009; Thu, 13 Jul 2023 00:21:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 92F938E0001 for ; Thu, 13 Jul 2023 00:21:37 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 617EAA00FB for ; Thu, 13 Jul 2023 04:21:37 +0000 (UTC) X-FDA: 81005289834.22.7CEE320 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 8F064180005 for ; Thu, 13 Jul 2023 04:21:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ZwLsHKCM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689222095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/by1wk+lV1+/9owJgKTgxMHTJboWG5CTUy71o+eRxGw=; b=8UBByQn+4otEmc7wGmy26o1YWog32/04E3p5j7EtdNrz1wzWHZPjnJWQ2XyqGoSNYWgSU6 n6weT57sJ4/oaI5e/cZX0U5UlfTKyRlAhKkDhEV6XglNbMbfime5vsawCL3mCaVUC+tPSf 4gdj5+iek3DLfEJW3O6unS3fFXsm8lc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ZwLsHKCM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689222095; a=rsa-sha256; cv=none; b=y4hcES1x2Dn0yUU+MK3BAgSvSuN3KmDmlFlIzpP+dXV2wSdLtaVXDRDUm/jFIifvQsqWsn BrsdvhTPXTjdOFZ/xKVQXpTpOUdHl9ZOnZmhJRXoGeVCs0am7pgdbHck0rGmfpAB9WNZLQ ++mFwyQEcEm71ji+0CuSEQGhGYWpxek= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-666ecb21f86so296580b3a.3 for ; Wed, 12 Jul 2023 21:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689222094; x=1691814094; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/by1wk+lV1+/9owJgKTgxMHTJboWG5CTUy71o+eRxGw=; b=ZwLsHKCMGtSfeOwsjPa0Ht517nH/Y0IoWXfzkMMAR8zYkK/vu4HdAF46+mxiWe/roe 8EelYNkbLmFMwUTu8yqR6eqjQQEZELmkL0j5hPFs2j/dvv04mhUiCuF4GqCCxNQOvHJM 15lB1o6EhTt8eXMOEcjpqczfk/daeZabPbwWkIh/3bBXdpM3MSLYmmp/G+F4XIechk3c igz9mrzUUga5BdUJq8NrakVnxKm5sOPFoBF1tD+fv/XrIHaelaFD8r2jMn2tpTxX8icj QP5KDdfRxq+vtvLqAVuOREzSHDU9V/n8VLkX1DfcInOK7adg45jBcWmgh22ac8mHhHZI qZtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689222094; x=1691814094; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/by1wk+lV1+/9owJgKTgxMHTJboWG5CTUy71o+eRxGw=; b=kOB844ca1IWjKmVKi0/fvrco8eDvlOHm1DsIu9HLtl6k2QRvKZ1ySMDQDUkx7VA5Qu lDpjW+9H1ueizWohAGLkuGj4JFsAMXYZpYYNTQNppp7/2JBLiiJc3hiwjb65wFBPBlCX Qo+hxUB0GhJvgm1vTRc6CAVmNdiPd+a/lCdFc74Xeh8WqGsGeIABtzbQ+xNXkZKYaeLO PzlS2kXnCONkREq4o9Fkws8IEtHBN7qGUvNWSaTnqpZn0mksVqR2R/9ZDjO2C9jOe06Y 6M63e4NSaBnaU7yoqrtz1u/JkXgsh7sDtL2i3FTlhOrYbIDKjQu7ASt9t0iezW7XYwXV +/iA== X-Gm-Message-State: ABy/qLaqdpWusaF67fwvQEqejrnnPNWQa/Fn3AqzTohJLbgcrHTP8jUU M8fHr1YhL8gc3BRPILcZkSU= X-Google-Smtp-Source: APBJJlHQXaD+zIgZjEwNmV8ncpgx7dFNmq4wW4mHfF3i/lYKQg9jn3S5Evn9lGjxsR9uHDQqJZv4fA== X-Received: by 2002:a05:6a00:14c5:b0:668:83b6:bfe8 with SMTP id w5-20020a056a0014c500b0066883b6bfe8mr621472pfu.9.1689222094431; Wed, 12 Jul 2023 21:21:34 -0700 (PDT) Received: from fedora.. ([1.245.179.104]) by smtp.gmail.com with ESMTPSA id u18-20020aa78392000000b006827c26f147sm4346601pfm.138.2023.07.12.21.21.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 21:21:33 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Matthew Wilcox , Mike Rapoport , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [RFC PATCH v2 14/21] mm/zsmalloc: convert zs_page_{isolate,migrate,putback} to use zsdesc Date: Thu, 13 Jul 2023 13:20:29 +0900 Message-ID: <20230713042037.980211-15-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230713042037.980211-1-42.hyeyoo@gmail.com> References: <20230713042037.980211-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: nn6g4uqwnyuzbcrsnhsmr5hn3qfbfqz1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8F064180005 X-HE-Tag: 1689222095-581601 X-HE-Meta: U2FsdGVkX1/GotvccbmPqHxCTCylFPNlviOMExBiI6txfHNocT0kuSOW1Y7Z9R98gbJz6P4DhwDl+JEQHDt2WrUsQY15U6MaEcGlnNLtN5b3KQjExBSpaD1S+mBs4DYw7nrActPR02hnIWc7cbkE7VmNQn5R2wRmVDk0RZfUyv0UANGRS7mD2Jz4vgCc7umM7lkrJ7n7HX8/p6bMgiQrJomKSpuAQWXY0N7kiH3riT4ahWsHeZLoard9ySFu0xfzZC56tvzPE00pjDxUevwTPivVqS7j91sVl6+fJzfWI8vRbXhZp6b8tlvWa5XJoX8rS+ptCZnudMsmhTmPNXDc39/Z6a2TxcjdPld3IzLC5aCG4pr0fIe1wnBFgk7gcncFwZrlYagUJ0pSUn2yBkv38XxkmPW3wbjwpjrBIiXUfSNJDuMclUn1Fe8shWnxbHsBEGtcr5NTx1eS+aqE9/eLitd2TiqCVOIqcmu49Kq8pF87v4E1qayJDqJBgNjkE0k/DwfQNKkahB5BUhoYI0DyUO3SKivIGbcdv4/OJfBLwnxjk0FOzd74jwapFctD1SIOzuyMCUf2QGx8y/TFo/BHupHoxf2f0oJNF5W+7CkYygE4Q9Gl3zKHjTyK74gRvO500AoZxrvtONP+wlcXJGPd9RwAeZsnNhua2IE0hv/R92JxmpYYq6hgwr7Q2uMyMBcCvYhFKlgdYp1vhA26lqp3n49dqb9IMbiKYdgBDluqpDxWC+0/14t0cRWP+wKL60o5o+7rCaShSWWAK6xZlUkXPPo+3LTiFQ6JWBcQJXBlYlaB/HJEfOZTTcmugukOvneoU5UcqfD/nYS5Mt7zTvQ5lJqnaPG1CiAM0pIcPmmFO0zLyEGe6+EdGQtR0HQzi0xYh7BaVkgUQ4wqe6gutbfw9BQuOHzb/RLyCT4cE9+FTL3ybyyO2A66kpAD9+o29nNUS0idsT1vp4lnOPOYW/3 ME+XTOml yu6EimpEPQDu7hBHIfI7Sp6amyvFtrJhXbCSQk2zXOS29D7Mm25IlTZSPXJZX38irCq92/3w6yQ4/E8u+seIxO++TsI4yURFrbHmFz5lyPojDjtVTQt4ySc6ThsJZvSZgFFy1faBmnz9XvKJxOBsIisRqTTu5jtFjw8TQzOAm9b3NDu5ygGSyNTMFwIGbmm3907yevc6IB1yT3ndF80WsTPdrU366mzwy0e309QjhVCeheNuUtXpW1UaZB8WHDjQPuVrLb3evkxCAc9U9n5uxk8IyWsad2iu2A0FajotmjTFynK3uVhN6jx0OdmJMfyPJgqbn9zsU8WgKPdqwzB6W5B2Kb5prFkPAHUmuXkMo+k0p2+qNOsk88qCgZMMJiqUze6HsEavWqiCwDAGOzMINTcjBlx+Ey9NLybgNo3U/vxjanY/nS5e8SToER3XbNuUvUqWOyfVJT505nYprT0SYPHxMlkawCMXA7OMIuGOaBIM7yd1YU5ARkgbTdMwGGati8uo0thI84QKJQQ5BIHKsk1pobdDMLfJ1M0ELRYjTD7IiuXtZle606rQLGxR0+FJiKP4OJVfP72xEYmGpmuMT2/ZL8A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the functions for movable operations of zsmalloc to use zsdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/zsmalloc.c | 50 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 085f5c791a03..9e4ced14e1eb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -399,6 +399,16 @@ static void reset_zsdesc(struct zsdesc *zsdesc) page->index = 0; } +static inline bool zsdesc_is_isolated(struct zsdesc *zsdesc) +{ + return PageIsolated(zsdesc_page(zsdesc)); +} + +struct zone *zsdesc_zone(struct zsdesc *zsdesc) +{ + return page_zone(zsdesc_page(zsdesc)); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -1928,14 +1938,15 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage, static bool zs_page_isolate(struct page *page, isolate_mode_t mode) { struct zspage *zspage; + struct zsdesc *zsdesc = page_zsdesc(page); /* * Page is locked so zspage couldn't be destroyed. For detail, look at * lock_zspage in free_zspage. */ - VM_BUG_ON_PAGE(PageIsolated(page), page); + VM_BUG_ON_PAGE(zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); migrate_write_lock(zspage); inc_zspage_isolation(zspage); migrate_write_unlock(zspage); @@ -1950,6 +1961,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, struct size_class *class; struct zspage *zspage; struct zsdesc *dummy; + struct zsdesc *new_zsdesc = page_zsdesc(newpage); + struct zsdesc *zsdesc = page_zsdesc(page); void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; @@ -1964,10 +1977,10 @@ static int zs_page_migrate(struct page *newpage, struct page *page, if (mode == MIGRATE_SYNC_NO_COPY) return -EINVAL; - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); /* The page is locked, so this pointer must remain valid */ - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); pool = zspage->pool; /* @@ -1980,30 +1993,30 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); - offset = get_first_obj_offset(page); - s_addr = kmap_atomic(page); + offset = get_first_obj_offset(zsdesc_page(zsdesc)); + s_addr = zsdesc_kmap_atomic(zsdesc); /* * Here, any user cannot access all objects in the zspage so let's move. */ - d_addr = kmap_atomic(newpage); + d_addr = zsdesc_kmap_atomic(new_zsdesc); memcpy(d_addr, s_addr, PAGE_SIZE); kunmap_atomic(d_addr); for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE; addr += class->size) { - if (obj_allocated(page_zsdesc(page), addr, &handle)) { + if (obj_allocated(zsdesc, addr, &handle)) { old_obj = handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj = (unsigned long)location_to_obj(newpage, + new_obj = (unsigned long)location_to_obj(zsdesc_page(new_zsdesc), obj_idx); record_obj(handle, new_obj); } } kunmap_atomic(s_addr); - replace_sub_page(class, zspage, page_zsdesc(newpage), page_zsdesc(page)); + replace_sub_page(class, zspage, new_zsdesc, zsdesc); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release the pool's lock. @@ -2012,14 +2025,14 @@ static int zs_page_migrate(struct page *newpage, struct page *page, dec_zspage_isolation(zspage); migrate_write_unlock(zspage); - get_page(newpage); - if (page_zone(newpage) != page_zone(page)) { - dec_zone_page_state(page, NR_ZSPAGES); - inc_zone_page_state(newpage, NR_ZSPAGES); + zsdesc_get(new_zsdesc); + if (zsdesc_zone(new_zsdesc) != zsdesc_zone(zsdesc)) { + zsdesc_dec_zone_page_state(zsdesc); + zsdesc_inc_zone_page_state(new_zsdesc); } - reset_zsdesc(page_zsdesc(page)); - put_page(page); + reset_zsdesc(zsdesc); + zsdesc_put(zsdesc); return MIGRATEPAGE_SUCCESS; } @@ -2027,10 +2040,11 @@ static int zs_page_migrate(struct page *newpage, struct page *page, static void zs_page_putback(struct page *page) { struct zspage *zspage; + struct zsdesc *zsdesc = page_zsdesc(page); - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); migrate_write_lock(zspage); dec_zspage_isolation(zspage); migrate_write_unlock(zspage);