From patchwork Thu Nov 30 10:12:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13474211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FFC1C4167B for ; Thu, 30 Nov 2023 10:13:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A51658D0035; Thu, 30 Nov 2023 05:13:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D8DC8D0001; Thu, 30 Nov 2023 05:13:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82B1B8D0035; Thu, 30 Nov 2023 05:13:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 673D38D0001 for ; Thu, 30 Nov 2023 05:13:43 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 47B8114014A for ; Thu, 30 Nov 2023 10:13:43 +0000 (UTC) X-FDA: 81514209126.15.B16FEE0 Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) by imf11.hostedemail.com (Postfix) with ESMTP id 7252240011 for ; Thu, 30 Nov 2023 10:13:41 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=W6SMhXpF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701339221; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/e5l2dXqHayaHruWo9RQrUpw6Q1W4wtar29KNaAiiUA=; b=zWMJWkSnGnAcCW2yoXKYdyPabKMO0AgUi2Ioh3+1Ao8n2m4pQcGjQq6opGSonFNcNmckfY kaqkXB+vtrSFq4/SMiOLUXmEufD1R7c5pRA90WsA5MEcEPnMfURU1+R+EQ+I5d3FINYgUf KGlM17j9X3G6JrrKNy9nru1izXPxIII= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=W6SMhXpF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701339221; a=rsa-sha256; cv=none; b=lmGY4JtoWBCP6l68lAiXuGZ8LEqT+v2WeYVt+Ins0JZ6jY8yI8xNM4t5aGnT728srXMznE 8CrlU8EqMuiebwxclePcJVlTkDktWSv/Xc79GC1k4uV6JlCDod77Bo6lhE9qT+gkw78L+G XqVdGJm6Usmx3R5k/H3q/uGeqvZf1UE= Received: by mail-ot1-f43.google.com with SMTP id 46e09a7af769-6d80245b6easo447530a34.0 for ; Thu, 30 Nov 2023 02:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701339220; x=1701944020; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/e5l2dXqHayaHruWo9RQrUpw6Q1W4wtar29KNaAiiUA=; b=W6SMhXpFtPGhwoSMNu0uFc5UlHPeqtSIcr76by8qCPJw+bfp2PgxIP5+rarmVw81AS Me/7DVh0FbzonCbxxHNO6tP1ppYuY0lzyLfp8ow+fzsahQUF/eL/u38pXiiP/nwbBh51 ejRlMGKE4uwcjqt4qECgNPIQcrKOcVQqHO84eo/apSsg8oAfISta6/5B3L1uj46GV/rQ UfZwTWvuwYk9c+S1aLrEwJcJIpfsyoNXbmQuFczHpcVe6z/JO5PmVHSmaJsp8JRpzL3f 1KnWwIC+TS0DkdM2Z2qQd8Y35M2//0/v5Ghbc3/0/xcYfPc/t9jhAv+wmeP43+m1GBZQ V8XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701339220; x=1701944020; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/e5l2dXqHayaHruWo9RQrUpw6Q1W4wtar29KNaAiiUA=; b=YVc1pU0VSQy5Frj1Q2SFmm/1ZwIRfLI/YxtKPsF5GrA8ZQdmmPrT3vzX4Zf751kQRu dfJrZY78/IfPHFPpQnA/CT2XcViUpSfWQFuJYrrlPUKTHrEOCde5Ob1GHdKTYn7kinvP sXlARrD0Lyj05e7fPjqWj4TWwK5IkiVJvco1FgXRArMshnBb2Yq2qoYOvoggL7QyBvLo BgTocAUkRjaFf4eaDOSkgJVV0dzIRekcYuqLq766kkRm2/plpLeeqrR3GbChFMbLgUWo 4pD5SvvhQDMi4KGVi+MdrYfHDWTjI+AiBSvmrEDbKPitjYHMICuVs6Tfpz97JTVoTOiE fEqQ== X-Gm-Message-State: AOJu0YwK7/MWtQUhw5mCLkN+b9Vr1Yi98XCSNkPoYHRlD/k1d+tzNI2R oaZ/9R8BYXmYj9bueiwRE+Q= X-Google-Smtp-Source: AGHT+IETR1NlufdrUSgFJMvtG+dAI87A/6d57BBFerzgb4wtwONV1yX1F5zTgjOb5u/gK+LOby/JvQ== X-Received: by 2002:a05:6870:6245:b0:1e9:96c6:e04c with SMTP id r5-20020a056870624500b001e996c6e04cmr23707871oak.32.1701339220482; Thu, 30 Nov 2023 02:13:40 -0800 (PST) Received: from localhost.localdomain ([1.245.180.67]) by smtp.gmail.com with ESMTPSA id r27-20020aa78b9b000000b006900cb919b8sm841402pfd.53.2023.11.30.02.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Nov 2023 02:13:39 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Minchan Kim , Sergey Senozhatsky Cc: Matthew Wilcox , Mike Rapoport , Yosry Ahmed , linux-mm@kvack.org, Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [RFC PATCH v3 14/21] mm/zsmalloc: convert zs_page_{isolate,migrate,putback} to use zsdesc Date: Thu, 30 Nov 2023 19:12:35 +0900 Message-Id: <20231130101242.2590384-15-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231130101242.2590384-1-42.hyeyoo@gmail.com> References: <20231130101242.2590384-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7252240011 X-Stat-Signature: znt6qfnhw8czmmrc9nyatgmjxgdec7z3 X-HE-Tag: 1701339221-793348 X-HE-Meta: U2FsdGVkX18UCxScXOU2+pupBEDJhpX6mkEzgdRHGPmHSxyNPCdMqgVDvm4Kwa+dEUpJqODqoFEAO8pTHCNbdDpIX5BWRDWKuIvwwzfGu6zrzyuKjUuXLuR70ZmNC4pIZn+CE3pyca7rajkiMFYRZ9jR+8DioHm3kKFv0EnHDBsVJh60Sg9tbMDhlQfHqPw7CM1ohKsYyiaNOwgpRUIQJCiUPuiYEYy1FenEQAMTBwFpvy2oMHg+tboQqkDm1+mZvBhicdhf7WJCIweem6EbN3JpHdq6c1zNeqOo+f4tj1XSQoaE7qvx15+C6wOH7GRvLDyFaD6rcbh8/ZnhrEwskrSPS/As9VNGdLiW3fjrN47REOLuP2g1oQZHkrTASiYxISKK08ZDG0DT06PMquUR/EfqWE5ZivUw/XPn33e5/ciXsKNttrDKQ40lrT1TgFHsZuIgKLJRTExv5cgqMowgFmI14Zar1MnL4FOcLhd6zSoCI2T9dglFFaHjXnvnwLnlj5nul6Dbn5jcOHcdD6Yrb23dOAUfmQKB7qQmm6mvKXa9Egfjh0gAXdtt69hDgJJRLo22rGPSeTK/KCBU+T8P97nEBJ4FtoNZ8EfS5RR5aIXD4fi7nS1/ADuBF+BLVf4mo1QJn30TMdF+1rP+LtlBOzG1RYkh9g3MtOsRB2cMp0kzC/1sMaXUgP7+Fub92Mlq4kbyROH2tFu0oq6/+Hw3S+2FNzfpIX8zzjN+BfnkMAHyukWa9BcXB4IKcRpHtNbwyC6C7pFAs77pi23fAH1vIV4lWO1CWXiJu34KnnT6vcIooe8I4rV5weApv1RVzbM49vDL8KXw3aPprw34aHC26U1LRauWtgur82G1KVa6pR7naxtlvFA1gBuAwOPhtEENtRFjQRR3+MDCV8d3YT5PNUX+yFIpigfFbcKIhM3p40MzIa+4DA8CthvJAoyBvvr7VSMYjDrVGCJPGAuJNxm lXXLZDZU nHpyJrfPYpH3JGN8ZhVWoK2GMTLH2VPc1FdDs6WHKAJKTGH6QgJblA9My+BQC9ejLfF1/LyxIptCpTgE/dkS15SMZUJdtX5yRymQNbASfoCdGp/kx3qrwC9fEGbV7bhCWrCXQOl+grxKtWCIdRCYAZxK7mMf20Hx7HoG3MhY3+fpuEehBg2StJZI0LzGxjjG7Zfg7vhfx+5TN94TyzCvLoMAEZk+HGE4u4BocfRO1f8+TKgalJe3A+OAheDwt5D+06W9T9dJZWVND4aAidSoEYJHbX0oP0r87rgiIwevYH6R0DQgGh13/1Q4Y2oZ9do6zBvSfKo3+6uKAD2yqTShjGMEckeCCk6Wm52PKME+t5/UHPNplRiUzKPoA+Xuby0OJqlCqD4MhL/Cgg9EpyUoBLVky6bqLOPRdAEJDsl8FkXqKn3Dd+SMcrtzQMo9hxAHEwnAhhftRPBJN+FV2pXE9jA669Jt5RnJKNmSfOjUlLD/K0KIaAMyX3tdSe9KXuDJOzuoH1NrQgouQ+h++U9UPKQ+Y+ZaNg8L7ceCIvePdGNPCtPOktoKiXCltd1JkzLUwPey4R5GjU7y2Ek+PiwTO51/TQQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert the functions for movable operations of zsmalloc to use zsdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/zsmalloc.c | 50 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1252120c28bc..92641a3b2d98 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -380,6 +380,16 @@ static void reset_zsdesc(struct zsdesc *zsdesc) page->index = 0; } +static inline bool zsdesc_is_isolated(struct zsdesc *zsdesc) +{ + return PageIsolated(zsdesc_page(zsdesc)); +} + +struct zone *zsdesc_zone(struct zsdesc *zsdesc) +{ + return page_zone(zsdesc_page(zsdesc)); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -1933,14 +1943,15 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode) { struct zs_pool *pool; struct zspage *zspage; + struct zsdesc *zsdesc = page_zsdesc(page); /* * Page is locked so zspage couldn't be destroyed. For detail, look at * lock_zspage in free_zspage. */ - VM_BUG_ON_PAGE(PageIsolated(page), page); + VM_BUG_ON_PAGE(zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); pool = zspage->pool; spin_lock(&pool->lock); inc_zspage_isolation(zspage); @@ -1956,6 +1967,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, struct size_class *class; struct zspage *zspage; struct zsdesc *dummy; + struct zsdesc *new_zsdesc = page_zsdesc(newpage); + struct zsdesc *zsdesc = page_zsdesc(page); void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; @@ -1970,10 +1983,10 @@ static int zs_page_migrate(struct page *newpage, struct page *page, if (mode == MIGRATE_SYNC_NO_COPY) return -EINVAL; - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); /* The page is locked, so this pointer must remain valid */ - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); pool = zspage->pool; /* @@ -1986,30 +1999,30 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); - offset = get_first_obj_offset(page); - s_addr = kmap_atomic(page); + offset = get_first_obj_offset(zsdesc_page(zsdesc)); + s_addr = zsdesc_kmap_atomic(zsdesc); /* * Here, any user cannot access all objects in the zspage so let's move. */ - d_addr = kmap_atomic(newpage); + d_addr = zsdesc_kmap_atomic(new_zsdesc); copy_page(d_addr, s_addr); kunmap_atomic(d_addr); for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE; addr += class->size) { - if (obj_allocated(page_zsdesc(page), addr, &handle)) { + if (obj_allocated(zsdesc, addr, &handle)) { old_obj = handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj = (unsigned long)location_to_obj(newpage, + new_obj = (unsigned long)location_to_obj(zsdesc_page(new_zsdesc), obj_idx); record_obj(handle, new_obj); } } kunmap_atomic(s_addr); - replace_sub_page(class, zspage, page_zsdesc(newpage), page_zsdesc(page)); + replace_sub_page(class, zspage, new_zsdesc, zsdesc); dec_zspage_isolation(zspage); /* * Since we complete the data copy and set up new zspage structure, @@ -2018,14 +2031,14 @@ static int zs_page_migrate(struct page *newpage, struct page *page, spin_unlock(&pool->lock); migrate_write_unlock(zspage); - get_page(newpage); - if (page_zone(newpage) != page_zone(page)) { - dec_zone_page_state(page, NR_ZSPAGES); - inc_zone_page_state(newpage, NR_ZSPAGES); + zsdesc_get(new_zsdesc); + if (zsdesc_zone(new_zsdesc) != zsdesc_zone(zsdesc)) { + zsdesc_dec_zone_page_state(zsdesc); + zsdesc_inc_zone_page_state(new_zsdesc); } - reset_zsdesc(page_zsdesc(page)); - put_page(page); + reset_zsdesc(zsdesc); + zsdesc_put(zsdesc); return MIGRATEPAGE_SUCCESS; } @@ -2034,10 +2047,11 @@ static void zs_page_putback(struct page *page) { struct zs_pool *pool; struct zspage *zspage; + struct zsdesc *zsdesc = page_zsdesc(page); - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zsdesc_is_isolated(zsdesc), zsdesc_page(zsdesc)); - zspage = get_zspage(page); + zspage = get_zspage(zsdesc_page(zsdesc)); pool = zspage->pool; spin_lock(&pool->lock); dec_zspage_isolation(zspage);