From patchwork Sat Jun 24 05:12:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13291598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52A67EB64DD for ; Sat, 24 Jun 2023 05:31:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C71D8D0003; Sat, 24 Jun 2023 01:31:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 778358D0001; Sat, 24 Jun 2023 01:31:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 616708D0003; Sat, 24 Jun 2023 01:31:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5242C8D0001 for ; Sat, 24 Jun 2023 01:31:32 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 12AD51C932A for ; Sat, 24 Jun 2023 05:31:32 +0000 (UTC) X-FDA: 80936518824.16.45435E2 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf16.hostedemail.com (Postfix) with ESMTP id 3884A180009 for ; Sat, 24 Jun 2023 05:31:30 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LANh1sDQ; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.215.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687584690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A39gdokmxlm4U38P/Zql/B/dp1bRrgyGGMBd2ncXVBo=; b=l6XCiYGnapvVv+++ObG9elI4kphyBDRho+4xEwZuT6AoBM1fiWuAfQa8D/NUtmIpI4p4dX HicSH21Yov3vme2wrvajAJ62hvnO4RtzIMIWTK2fC46AznhOzyjyt3kT7SaL0t0jXgb4EH bv1U2ctQ99jDcxJBosiUPkDhBVwkMHs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LANh1sDQ; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.215.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687584690; a=rsa-sha256; cv=none; b=WBDUMDGOk+ICLZMKYJ6f3Nkf0U/e/+SKUciDq6FZNqjxcWEMFtPeAOTIwN0Ljv9+OJTs4x +OHhJbKLXT9bZeHKjVwXWboU2w2I4OU2jaIfLgQWYklVtMxnwLzhaVXGaPeQhs3rDnGdgx tFPBiPazeiZ8g9KEHc44QKRD6r+0FO4= Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-54f85f8b961so1108563a12.3 for ; Fri, 23 Jun 2023 22:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1687584689; x=1690176689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A39gdokmxlm4U38P/Zql/B/dp1bRrgyGGMBd2ncXVBo=; b=LANh1sDQ0Cqew/nk2fWgtBz4HwaykMfnm7u6tguZ9kC8E+QeXfexEIVbVk2ZI1cxKA iQQ1M/FSjdcMGc/TD9zjcogPnBiT5Dm3/lo/jXUz2HCzhl0UochpEpS3Afhg5MXLPkzb UbOHYYKYV/58KxdqAgSLYHZjfLyRODgzI9/7s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687584689; x=1690176689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A39gdokmxlm4U38P/Zql/B/dp1bRrgyGGMBd2ncXVBo=; b=jh3v5O4OtzOzcKemxOD5LWvf6EvZXnuh/wRxM+mgyj0JkY+Vjt8xuCV7IuJt0Ta13T tfalQ3cPZ983SqKHy5g3VDTYZU9nbBmuu86yqpAoo/i5m2qzY/lzn6Oh07WpMBYDGRFq ynKPaHWiMKlf3RHJ/vV9tSDFChLFfjM9b31/dNB3gUiWgUbzJHSH9xp5xcwawN/qXJMi fURF8973JnhuShnNFLG89/uMmgmSJsRCRFIA6g1eBBhQh3LEGs1ToT1drf1lsjKuQESb 37f5MdbjZpsOHVmE7hWjSyXjDsrqkZRhStNz/HPlUnGcT/JgEJAJD5yU/u1B3yBrAaIq b+SA== X-Gm-Message-State: AC+VfDxCNYK1DyME6L25BfOn4GIifGJZmy7hFgqE1PduwS74Unp8vfTD UcHK88OrXyW8Ux8zyWMLwobcUg== X-Google-Smtp-Source: ACHHUZ52NiJUM2vh+TDHLpF+yhX0gxBj2JOBI8yeSeM/qrXW8datRm1xeFixMs2KJoNyN7fYpdbnyg== X-Received: by 2002:a17:90b:147:b0:25c:571:8670 with SMTP id em7-20020a17090b014700b0025c05718670mr22087582pjb.46.1687584688858; Fri, 23 Jun 2023 22:31:28 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:d7fc:8a97:4a18:6a18]) by smtp.gmail.com with ESMTPSA id 16-20020a17090a0c1000b002563e8a0225sm567069pjs.48.2023.06.23.22.31.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 22:31:28 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Alexey Romanov Subject: [PATCHv2 1/3] zsmalloc: do not scan for allocated objects in empty zspage Date: Sat, 24 Jun 2023 14:12:14 +0900 Message-ID: <20230624053120.643409-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog In-Reply-To: <20230624053120.643409-1-senozhatsky@chromium.org> References: <20230624053120.643409-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3884A180009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: yihs9j4ujkgx6xrdcdyf41i9gw94yy8w X-HE-Tag: 1687584690-566819 X-HE-Meta: U2FsdGVkX1+fs6LAM4ln0A7Tvqb9cmK2/Ql4XS7f61XiiEhKtBmG9j46n0bj5iIH6COQnvQqmQepPOlM7+wJWawTxEvaKGjHnIy2v69PsV9SHmXWinEvUefcJHjRNXrmeZrc7Js55HcBzfixgTrEF2wtoGC2ZfBqWsMpER59CeBR+4Aot1OcYVdknoCXwocagm6sf8BS1vOmoTevaRCgEI2rUor3R0PrVQ3fBKOxI+AKtHXbQt8BxyN4E3vEVXdZ6TAU3Z4eOYiNLOx444wB1jC+4gksH81UyO5XrUDbFXWPf3NWvrjFWRoaa8ijZ+Kvn9TMYtPfBCUFFaGoeS8ZX+5dlLDKEpT3UnT1MC0gWi2+nC0meg/T+HN9Ou7Zkk2Dx6btIbW7EVOjEafzi82iyJIFC/kjVlgvI5nQfHu6eHFEcZZ8H5/LQ4ww17QV8MVeWkgFlCWxAw2LeEim2b94w7mww8cpfYfCYICAbRo1LiCdjxuh3pOxXOmtyDoiN7jvLVHph5M3UqVGEfk5UETvZqZVEs7Gyu0g3hrl9jxzYQmBOzWW4NELJAJstXkPmnPM7MQlz3EwFbji5a3hO9tfv6R4DH1sJsRPBzmG/tky50rNEDH33IMcVossE/6WCWE+cmoyFyn++5sOFtSNMnN8uyUwvxI15EkHvz1O7ZCI479qoppkMPeWI7+spNTnUb6GOanhGM2uz42CMZIBOBpJO6mKcXA+W7eKsBl1A571A9lKRyyiu5M8EdrBZcrl44J0RZiql/Aic6dnUKlpfoMAx31muIoGwLwPkwvMlEs7V7LanSamOO8pHF/hbN8E+lMoxRo5MkujgccHCFfAz6ua2xGOCO34bYUifycjSKhPnoDA0xohy7ubYIpOE6GH/kvz6W+WhH/OQy9MynHGf03daXjC/425OLLRAZhn2zmbCN5KtbiczL0Nf87cHIofe0GwKYCLO1GXqRwBfu1VT9q JbXFET2C IO9Cg9uHv+McsVmPhYTlcVf83sEsgCuXtOVW8M1+OiKCyogzDrNe3Wq6ohHv4BLAY/GVNSW//hQ2+srq5lY9wDa5i0zXG72cF4V6UZia0bbwEn8ZtyLqjZt346N5E2RFTp+uv9inBlBYdVjxc2RAROZkh+E5oTKim5Y3KNcWiqLIvW8ky/fJbUEH05kKy6SjuWs7R1T10i+tI919ls8gM9deJP8oe8IAo1XmIEPttI0pYbpfM1dzrv9F/HMpXXoR9/2ZWcR1yza7b0INEdJGsoeIKI4OCeuvPHanbJG7Cm5DeE4DA8DCjRJkgDiSfmgvNEgKPY15a3uBaOwXILHlely2xU35gQ8Qb191h8IwbK3DSWRuTMi6LRgRWMWZYVVuNYAUfCIGaBKnjMBGZOTTPB2FrwrzdtPNbJsyiNSjGXCaDW1nP3fsLBocfLBHmKfG2r37y1qSVHTiTRbYcAKdXlHoQFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: zspage migration can terminate as soon as it moves the last allocated object from the source zspage. Add a simple helper zspage_empty() that tests zspage ->inuse on each migration iteration. Suggested-by: Alexey Romanov Signed-off-by: Sergey Senozhatsky Reviewed-by: Alexey Romanov Acked-by: Minchan Kim --- mm/zsmalloc.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3f057970504e..5d60eaedc3b7 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1147,6 +1147,11 @@ static bool zspage_full(struct size_class *class, struct zspage *zspage) return get_zspage_inuse(zspage) == class->objs_per_zspage; } +static bool zspage_empty(struct zspage *zspage) +{ + return get_zspage_inuse(zspage) == 0; +} + /** * zs_lookup_class_index() - Returns index of the zsmalloc &size_class * that hold objects of the provided size. @@ -1625,6 +1630,10 @@ static void migrate_zspage(struct zs_pool *pool, struct size_class *class, obj_idx++; record_obj(handle, free_obj); obj_free(class->size, used_obj); + + /* Stop if there are no more objects to migrate */ + if (zspage_empty(get_zspage(s_page))) + break; } /* Remember last position in this iteration */ From patchwork Sat Jun 24 05:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13291600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97F46EB64DC for ; Sat, 24 Jun 2023 05:31:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECC3B8D0005; Sat, 24 Jun 2023 01:31:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7BA68D0001; Sat, 24 Jun 2023 01:31:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1D548D0005; Sat, 24 Jun 2023 01:31:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C3AAE8D0001 for ; Sat, 24 Jun 2023 01:31:33 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9E9C8140F11 for ; Sat, 24 Jun 2023 05:31:33 +0000 (UTC) X-FDA: 80936518866.06.626D79F Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) by imf25.hostedemail.com (Postfix) with ESMTP id 8D0FAA0009 for ; Sat, 24 Jun 2023 05:31:31 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hEVV1PKn; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf25.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.167.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687584691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JXozlQ7AGAAc+Nx/NUdVZ9+4KpO/wevqL//8Wl3bnA8=; b=xgKSJcPWWgwsyhGhbiLGB3nMAOyVRCAwDEtSpdTWlLnUc5rj315hbXzZMh/mmITmmRHuzH BQAjLoYxTxpYyeevnU4rGiYYEtRaNUaCND6AHCcNCfGqACBCJiXBhuosKpFHKTgGwE8q4S 2A976VA/4pwJBwNducAzUd32qIJa1OU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hEVV1PKn; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf25.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.167.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687584691; a=rsa-sha256; cv=none; b=yT/Ai3G6b+Vg0djtLVu0pt30CpVXPBua4ykEzziyQkBsJ5hLMyLiaLnR48eZMtXy/rb7Pz PSk47MxvRCxqqe//kEan9yipxaszp5gH7o3rLjaE7Z89vBzjeTCbcuv2VN3Kz2gCn27cKj Qkf0Fp4GbwwPpU/RnMM5jgXWJ5C9eDQ= Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3a0457d6b77so1129088b6e.0 for ; Fri, 23 Jun 2023 22:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1687584690; x=1690176690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JXozlQ7AGAAc+Nx/NUdVZ9+4KpO/wevqL//8Wl3bnA8=; b=hEVV1PKnoqRDlkdskPail3UwTi3zNM443LUxpwYsTN8oCAj3XIMmoMXFJnsnxbV+DF wzOj3dtWSlkKaMPk+9t+QowWS3A47K6hiYh9dpNM6u2Wfey5nHSVhk3Ym5ox5sYyZzes K0dyi1ixw9OexMvrCK8P+ImImK/L+xRxsbVYI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687584690; x=1690176690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JXozlQ7AGAAc+Nx/NUdVZ9+4KpO/wevqL//8Wl3bnA8=; b=P1Jz4vANHqvdj86CEEXbRdsQv0W59eDRxnUY1WLWFWxOpidssVjVYWPg4hEOHFZ0KJ 8LW0kBT1MRX7NIPnSS2/TavRzUR0Ux82YbtpJ/ksj7l63AIESl9hSzR0rpenP97RJNt5 Th09DoCOTa/2VHp0wnrBYFKlh51FUOGPljGXYvCR0fno+cZZTmHo7Urtisq+bOhCuKeK en1lttAiIMQz6eU38fgqEgptdI+XTRZuCaLuMj2kwrojtbZZGPCV5VjJb5HHzJXrL6nI WCb1j8DjiN3MMEUOeoEo3RmSKRBpCJ742OBz3ezIJOgzzNIUf8uldokSFCb0i9z0QKtJ 7gCw== X-Gm-Message-State: AC+VfDzkFABE+eYiXOKn4ZaKqDXW6HS24TzpOcl5W7lnn72roFFd1o/C AHrbR+5elxXckamVLLc7Lk1ZMQ== X-Google-Smtp-Source: ACHHUZ5mbSYPloXGVoYuwedg5AUrAtc+B9JDYGgNuwbVgnUzI6YASvLhcPdKURnsv6EOXnYZm8NkVA== X-Received: by 2002:a05:6808:1524:b0:3a1:c0bd:c51f with SMTP id u36-20020a056808152400b003a1c0bdc51fmr270508oiw.28.1687584690736; Fri, 23 Jun 2023 22:31:30 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:d7fc:8a97:4a18:6a18]) by smtp.gmail.com with ESMTPSA id 16-20020a17090a0c1000b002563e8a0225sm567069pjs.48.2023.06.23.22.31.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 22:31:30 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 2/3] zsmalloc: move migration destination zspage inuse check Date: Sat, 24 Jun 2023 14:12:15 +0900 Message-ID: <20230624053120.643409-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog In-Reply-To: <20230624053120.643409-1-senozhatsky@chromium.org> References: <20230624053120.643409-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8D0FAA0009 X-Stat-Signature: d8a5onm5jaq6r4dupfgsgzfb7mroacn4 X-Rspam-User: X-HE-Tag: 1687584691-174677 X-HE-Meta: U2FsdGVkX1+1T1UMpeZpPVKzGJ4kb6sI5ERqHsMeK0FPgPulQ9lvFaoRaJ4e0UgKUEEvftc4Hs2+aq1eLt3uzRONOHnhD79w86xS88kYovpa2EErOh0YhRMJUvggFruQEibXzmPXH9GHkVhFp2dM0WvDzS/UEEkN8f17SA5ys6lJhfaUJkUYhrikBpTuAFEBYbMo8FW5M7Qrdb8OP89tLEj1a++4dK2Tqgbbr5JQ45uge1e56NhGJ3OgeSJJ4HU28DtYjaoJWTos6RxIEUoe1Ou13sDVQAiF1+dq92ueUN12FFn/uBgXBoCjc9NeuSgnVIX8wqkVEutuW9rcdyJ+BcfCndxTM4XK2HIgAs0l5sFDPZQM2YKN2lM1AidmY46xQc64Ycz+7JutvFpTZ+QTzhEeRgyZ+UPveqULnOfGFZiF0+fNy7bWkSKdTWyzvLSEXzTP37BQCEIOMkrjKbfkPrqLU4YZwMPm5RrOb5xbWjNkTemlhVgDBYD2DFtzNYyXW0N89SJLg6YIf/szJVnMVrZ7XDB+sI6oPelzBbc2MSsR4iuAUq/yqQjQkTM/od5wvm//SRn2F6N25SEXbnA/C6XYwCyTT8j1QlV0xL/228Qn+/541UwR21DkiMdfzWBD85IeCcx9oc2JrtkkhE7FUroxlEsx5iIv0j0APEvJ1d0ymVvjczlDSMyMOr+IhnpHhhqDe34yOJ3FJhki82j5dxotznCWkoW3dPYntFGR0Hs1oDhjRAnLbmEWekWZzfvrpGic1dYvQTcX8jzo5UFxYJQsqHlwLcbCHx3Tv1U0u5V+vot0xXx/UwZ4QzE3NM9RjVWnONBojPYTMcjWLoTABPsVywInInRuaXBONJXet3bBhO9noN1/03Z4ZB4QFXtVLUGpkHAGngYisMcaE7aUOBUrTXLuk5hlAOGoJa1gwmXJk39nGNikii6w08QxNPOZi1UCX7c3/IrnI96CRa2 5bQVZ74f eCLb6l0Vuld3hdO/0p5Y7U395gFcOWJS5ZsJPseymW/0fdIDwWDmzKWGinyAe+PjcycssAmmn1lGu2vhyd1DI1Bs5Y7GUDw2yqq5AcONR3j4sea8ZwqA+2qnyiIv7ZZDa7kb/4ZHtaKTtwU+srFBm8rDXF2y3k2RK0b7/2LiSsiAVzOoiBFK5qq8q6D0afoa009Wd1XsLYU7kUQN3UXGgxDZEO8/BYB2rQwPP/sLzg32GtuONQbSwESvbNLEVtn49CTU6tqzz2TCaZmB1OMhYgEepjN6LV47YJtubbsQRq6Hut1jO6axP4xlhM+MRTnJ3vqTwNwYn9QP+ZA0FRhaUgWpAKQ0maKldJ602MGwOMhMK2CpDrrkx47bGtBquQuMjdWP2NDbs5sWrS7j9MngNg+WUfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Destination zspage fullness check need to be done after zs_object_copy() because that's where source and destination zspages fullness change. Checking destination zspage fullness before zs_object_copy() may cause migration to loop through source zspage sub-pages scanning for allocate objects just to find out at the end that the destination zspage is full. Signed-off-by: Sergey Senozhatsky Acked-by: Minchan Kim --- mm/zsmalloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 5d60eaedc3b7..4a84f7877669 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1620,10 +1620,6 @@ static void migrate_zspage(struct zs_pool *pool, struct size_class *class, continue; } - /* Stop if there is no more space */ - if (zspage_full(class, get_zspage(d_page))) - break; - used_obj = handle_to_obj(handle); free_obj = obj_malloc(pool, get_zspage(d_page), handle); zs_object_copy(class, free_obj, used_obj); @@ -1631,6 +1627,10 @@ static void migrate_zspage(struct zs_pool *pool, struct size_class *class, record_obj(handle, free_obj); obj_free(class->size, used_obj); + /* Stop if there is no more space */ + if (zspage_full(class, get_zspage(d_page))) + break; + /* Stop if there are no more objects to migrate */ if (zspage_empty(get_zspage(s_page))) break; From patchwork Sat Jun 24 05:12:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13291599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34CA5C001B3 for ; Sat, 24 Jun 2023 05:31:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94D358D0006; Sat, 24 Jun 2023 01:31:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FDF08D0001; Sat, 24 Jun 2023 01:31:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79DCA8D0006; Sat, 24 Jun 2023 01:31:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6BAFE8D0001 for ; Sat, 24 Jun 2023 01:31:36 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4C8761C932A for ; Sat, 24 Jun 2023 05:31:36 +0000 (UTC) X-FDA: 80936518992.23.E135569 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf08.hostedemail.com (Postfix) with ESMTP id 5A11B16001E for ; Sat, 24 Jun 2023 05:31:33 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="Tkj6/aDL"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.222.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687584693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DWxKMGlhVIOcwxYZSkni1ma+NCE3p+LmC8CwV158uoQ=; b=bsssgJ9Xh/RCPZfr9JpkViSC5HnNwT4u/+CqYLwSXzU6yIX613d5GZokjBvrMK+kZgWtHf qDKr/asHIaVZmBcDNGIIX6y/fT/i8Y/5zaR3THulfHX4aeyTNc7wopaaXkPFbIEc9WpPti 2fxZzTMcgzIRKJ6d7IEofIU3mJAZdWg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="Tkj6/aDL"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.222.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687584693; a=rsa-sha256; cv=none; b=A6f4s4+wTcgH7iAGPdVidUupAppCeGJbm5EtGDWqqvbKVP7b4cwAt8C4GotOJFtiJyj4Hc o5PB1W4dIQcD0kc/4vPNV4GjL+pYD0gdAx6nRNw2ANK1Y3Qtte777D9RUyZse8pm+BvUl9 CG1hfEB67bHgSe9gW2b+SxorlvGVhzc= Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7624af57b21so124706485a.1 for ; Fri, 23 Jun 2023 22:31:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1687584692; x=1690176692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DWxKMGlhVIOcwxYZSkni1ma+NCE3p+LmC8CwV158uoQ=; b=Tkj6/aDLrxhyMZp9C0Bu7YwtXXituRwejp04Q9FvaCfS63ZOBmOGbRv0zf+BfnHFn3 Dm3Y+hAKYvuBGfNyxiguJGE5Drz60p9JekBkkqhLQ2R6o3wYXZ5geKHjrtuZ9xwR6qDB 5A84oWan6OPS9WS8PXeYJtB6IXqRpOLT3EcjY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687584692; x=1690176692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DWxKMGlhVIOcwxYZSkni1ma+NCE3p+LmC8CwV158uoQ=; b=doTQfEBsF84Q4qfQT2v8zS59VS2MLM/8+cxlYbxgS3SX1Jc++ziygJkQDWf1Qe7Awy 5cI/z7b2q4u60SE798lABVu6tePGyyJhLDqBPSV6bAPdykyDEBiis6SYrNvjuDWAea/o x2cdGkSk0M7lpUvgymeY3vg06k4xmOkHci2DWKO0n51YnszTNXb7ah9/B/6hOdT0IMR+ 5oPYb7bMZh0NpeEDEkU/iKsYbD8n1WJdvjJ/S/HTfGHdGYZGAddpz3nxG5FsxHEEPPKh 4g5hN2c+43zDn5B/wAPEzeqwRm5eRKZWCkufknvTBnUhm9aJaqalyqqa1dDeblfDtBY7 yseA== X-Gm-Message-State: AC+VfDxSmLhAxowYpzMRyd2GmoMZ+iNfK+AdWFLpIc74JleDv2l29z+D kBWZNDDipvhuBtRna6R05t2K4695o7OUn7UUWjc= X-Google-Smtp-Source: ACHHUZ6cpRSLnEvIdQL6C5yzq2Jz418DRJ8c8EEo6m/ckQabR2EwOmEVP4W9pYHLeacaP+WSoxKr7w== X-Received: by 2002:a05:620a:4451:b0:763:dfe8:7be6 with SMTP id w17-20020a05620a445100b00763dfe87be6mr12050561qkp.20.1687584692322; Fri, 23 Jun 2023 22:31:32 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:d7fc:8a97:4a18:6a18]) by smtp.gmail.com with ESMTPSA id 16-20020a17090a0c1000b002563e8a0225sm567069pjs.48.2023.06.23.22.31.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 22:31:32 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCHv2 3/3] zsmalloc: remove zs_compact_control Date: Sat, 24 Jun 2023 14:12:16 +0900 Message-ID: <20230624053120.643409-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog In-Reply-To: <20230624053120.643409-1-senozhatsky@chromium.org> References: <20230624053120.643409-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A11B16001E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 658r5c7fm94omarpxq4cc99nnzsep5xw X-HE-Tag: 1687584693-266168 X-HE-Meta: U2FsdGVkX18O+XOxqoe6R3wFn9ZVbwP790mX1Bk7KuwkxD+zbfT0EU2vfrrnC5+1KGcmWwB5jp0RVIKJqak7tu5amx0qxGHkoE3ehPMLuvr3PCsDNgegW1ubKxVIO7wbaA0MjLUkDLq1EmWtde2DszyAqsZ0FKYIK/z1Ctf0c+DVhUT5EMhNnAZr+sPyJlYw33Xkgh9cQhF0QONsThsR2V47wq0n3jtlUGD/3ZjKA170lm6l2HjMqWj7XkJ1dzWFEHsO4RTHin0hTflPBIbF66qfUQ22Wu6CW2ZR7IYOG2sqrRMyATm5DlwQexgXvgUW2YZuV1db2gO5OfKTTcgO+Bk5Znn3T98S0BEz/zrsY+DUGa2oFggHUELAqfoWTVXO972r2pRSdBYGKSFxmy8OztMNCVYsxPVEzq6iklLDaIh/w9SplCfyqd7mS6Ho+0AZ0yhdk7MUt7Zsf9/wpxM9d6PyaLyrErSn+rp3e2d4jJaHlMRErLsteaqdecuvr+a6/pYl3qJgF47dT/nr65XchAlx6FmhWqIKJyzDx2I+/WaCygTDmg70tbnaQfU6cnzXyX8lqRCZ89AdVjYCZrEFfMs3i2A0Q8arv9i6wuLRPPK2ZDGkLb1ZYcIJzRJbqrUlNTQSRlCBKcOENd0ZjRwEiJxR1mI6tE6hQkOf05SwRG4x2MWM7aq2CTGRTdGPzoEo1R1ylGhdyPjvHO4uOYmRAraYaJgb2uSJ6bBqhjeNsBi4s8cU56gBnd+VE19DPbH7JoRde6v9Y57MGNELDvgu5OImHROE4nefkg2YZaFOD6Pth9oAmhAszpSS73o680Obpph7hBmvPjHC0bapDjmYRZo6stXopTYMDWmdqPqpONFVs4VMtCKsHY5N4jycyzZLtG1tOqq/skuguvvAzhBTiOrRZ6U1nQjqeukh1M9G35LaSTc5isvmTvpgalIgyfzkyO1jNuyZo71VWw5r6Qc +SMXx3Mn MBEc7v5F2m5PcXu5PhWP5IJGTQokJOzZ1EzDTj1B2ruzFZyTBUI0mWX44z7ahwSdXIyQQGgGehEdBFq4VJN3qfn1OVoRV4LPavXZVTe2NtPaPrvPzW4oSjAWp4JB/hw21OcY4W7BrkW6vJE3XHsNwVLzxHZrs79bxS+9R4l/GIPB550arli2vuuHXkp+uLsvxy9RAAFE8csCvaxi1GXE6fLntu3fIegG2rGhZZ+z65qwaqPVu18bL9SJhDqM4up3qlTveqbkPhizeTxOcSZunVrigGRIgT6rcrwMZENWcyJfTaUKihECuWWs25uy95zkzZYQZ5/VZ30I/ir8LP/haFogM8kGQJVP9MenfwSLEWabcWXcyECynWhkna4pTnWlz/Vxb4K7zAMjskQ15Ax803LWgsUQrbREC3Cf1aGqG4C3LOXCiOrwjj0u7omka0u8l3xs1t3CxOHsZvAQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Minchan Kim __zs_compact always putback src_zspage into class list after migrate_zspage. Thus, we don't need to keep last position of src_zspage any more. Let's remove it. Signed-off-by: Minchan Kim Acked-by: Sergey Senozhatsky --- mm/zsmalloc.c | 37 +++++++++---------------------------- 1 file changed, 9 insertions(+), 28 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 4a84f7877669..84beadc088b8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1590,25 +1590,14 @@ static unsigned long find_alloced_obj(struct size_class *class, return find_tagged_obj(class, page, obj_idx, OBJ_ALLOCATED_TAG); } -struct zs_compact_control { - /* Source spage for migration which could be a subpage of zspage */ - struct page *s_page; - /* Destination page for migration which should be a first page - * of zspage. */ - struct page *d_page; - /* Starting object index within @s_page which used for live object - * in the subpage. */ - int obj_idx; -}; - -static void migrate_zspage(struct zs_pool *pool, struct size_class *class, - struct zs_compact_control *cc) +static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage, + struct zspage *dst_zspage) { unsigned long used_obj, free_obj; unsigned long handle; - struct page *s_page = cc->s_page; - struct page *d_page = cc->d_page; - int obj_idx = cc->obj_idx; + int obj_idx = 0; + struct page *s_page = get_first_page(src_zspage); + struct size_class *class = pool->size_class[src_zspage->class]; while (1) { handle = find_alloced_obj(class, s_page, &obj_idx); @@ -1621,24 +1610,20 @@ static void migrate_zspage(struct zs_pool *pool, struct size_class *class, } used_obj = handle_to_obj(handle); - free_obj = obj_malloc(pool, get_zspage(d_page), handle); + free_obj = obj_malloc(pool, dst_zspage, handle); zs_object_copy(class, free_obj, used_obj); obj_idx++; record_obj(handle, free_obj); obj_free(class->size, used_obj); /* Stop if there is no more space */ - if (zspage_full(class, get_zspage(d_page))) + if (zspage_full(class, dst_zspage)) break; /* Stop if there are no more objects to migrate */ - if (zspage_empty(get_zspage(s_page))) + if (zspage_empty(src_zspage)) break; } - - /* Remember last position in this iteration */ - cc->s_page = s_page; - cc->obj_idx = obj_idx; } static struct zspage *isolate_src_zspage(struct size_class *class) @@ -2013,7 +1998,6 @@ static unsigned long zs_can_compact(struct size_class *class) static unsigned long __zs_compact(struct zs_pool *pool, struct size_class *class) { - struct zs_compact_control cc; struct zspage *src_zspage = NULL; struct zspage *dst_zspage = NULL; unsigned long pages_freed = 0; @@ -2031,7 +2015,6 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!dst_zspage) break; migrate_write_lock(dst_zspage); - cc.d_page = get_first_page(dst_zspage); } src_zspage = isolate_src_zspage(class); @@ -2040,9 +2023,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, migrate_write_lock_nested(src_zspage); - cc.obj_idx = 0; - cc.s_page = get_first_page(src_zspage); - migrate_zspage(pool, class, &cc); + migrate_zspage(pool, src_zspage, dst_zspage); fg = putback_zspage(class, src_zspage); migrate_write_unlock(src_zspage);