From patchwork Sat Mar 4 03:48:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13159779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30A7BC678D5 for ; Sat, 4 Mar 2023 03:48:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADE786B0074; Fri, 3 Mar 2023 22:48:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A672F6B0075; Fri, 3 Mar 2023 22:48:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B9CC6B0078; Fri, 3 Mar 2023 22:48:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 73B6F6B0074 for ; Fri, 3 Mar 2023 22:48:47 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3F819140321 for ; Sat, 4 Mar 2023 03:48:47 +0000 (UTC) X-FDA: 80529834294.11.94DB917 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf07.hostedemail.com (Postfix) with ESMTP id 7219E4000C for ; Sat, 4 Mar 2023 03:48:44 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OSN6KWn7; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677901724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SEq3MDoFcR7yz1bLKCGz9x04eW60/3Ets++p8vd9Ctw=; b=3pV2snq9/+EcSuKngxvcq8bKFQo6cUodMnyXsfO8OFApkjFoBk2kPPvUkK0xRMuucA2a+F I85WvvH7Ro6c0qjjKRtsP/s5/D6wG09dJWcqO00k5IUrZ6HGG9WEhAP82vjw5aQKxUNshP n2UmRqcOp0kzFU7wqcvEtGJGfRQX+b4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OSN6KWn7; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677901724; a=rsa-sha256; cv=none; b=Hnjne3L7/9Uz+/EQm8H1pzXkqEEs0srkYecSPDzbIfqWB57BZwgLAULut+rJ67I3OpnBKX fViGFvyOVPQznU25hm8xoue0COBkgjMnmYiT1u5Sf1BIEjC+304SlEMP9/P/TBSGNIXMJ9 g+FWySOz7Dy82R5F4rq37CJehVl2FSw= Received: by mail-pl1-f169.google.com with SMTP id p20so4742368plw.13 for ; Fri, 03 Mar 2023 19:48:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEq3MDoFcR7yz1bLKCGz9x04eW60/3Ets++p8vd9Ctw=; b=OSN6KWn7hEXJmiSwcCwf+j4CxdT8wPCagEal0qiSFk+/V8atkJMsWd9LYZUtjmsPXr 9CRmtA+NB7lVYdNh98oMEGUcsk0IzMtp3hVLlNUeYtTmnR1IdOQSDJgHlTb/6bPe258k cYr52SWQApBs9ogj8FNmQ3LcCt0GmMZsOzeLY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEq3MDoFcR7yz1bLKCGz9x04eW60/3Ets++p8vd9Ctw=; b=awtnDHsUeedglbREMpQHta3St0WVggTfpGE/PGG4TvYdnaKDk8DTxDBD1txKJJicub o88zmDnq/Mm6GB20TQD47UQjetZkh8PLSbLl5kpw5RmiCBFKn7MJbuGlfYkhfKk1dvor BuPinWCm7hClWtwXqFNDlgsFpKT2AtwOwY6cCU33zCKcrommJb2zrTIvt+Mptgj+CmA2 Z4ZC0MNoT2QsGeuj50RIfkU67aqySJtvehXGNFC0Y/9VEsTm/bDwOx0uxJ1qPlb7r1qP Vw8rjA+I81Io9eSPK3ndMjc1QGH4u69bbQ9qcaEQTaWra1jUG8yBH6NAxTOJv8FyFDSh IcDg== X-Gm-Message-State: AO0yUKVVganu43viYZox5dbLRjG46cz0IlZO41pv1Hw65vEAKuPMzt5v EfDcbSzEZY9I6xnXc1npYoT5VA== X-Google-Smtp-Source: AK7set/tt+odDA27bkzjmHOQ+zF7n6+n5T9hL7wxUbL0XcZs6V/tJCFYEzPfag5hO0XniqIk8YzmhQ== X-Received: by 2002:a17:902:ce92:b0:19e:27a1:dd94 with SMTP id f18-20020a170902ce9200b0019e27a1dd94mr5079134plg.35.1677901723382; Fri, 03 Mar 2023 19:48:43 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6ac2:6eee:5465:7ee6]) by smtp.gmail.com with ESMTPSA id d6-20020a170902c18600b00199025284b3sm2249204pld.151.2023.03.03.19.48.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 19:48:43 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv4 1/4] zsmalloc: remove insert_zspage() ->inuse optimization Date: Sat, 4 Mar 2023 12:48:32 +0900 Message-Id: <20230304034835.2082479-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230304034835.2082479-1-senozhatsky@chromium.org> References: <20230304034835.2082479-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Stat-Signature: h8r7o8gz4nkqa9sgc388rz1bn3unyhh9 X-Rspam-User: X-Rspamd-Queue-Id: 7219E4000C X-Rspamd-Server: rspam06 X-HE-Tag: 1677901724-982186 X-HE-Meta: U2FsdGVkX1/bZHugJb62ZpIsFR650NCcNtVsoqCxeIiK7I8agz6FGtujcr1nlYEiFeg2a+Lw+1fDc3rZEJLTSLMUHWxg9/Xm8AYvn8vMXuB+MyaQiXe1/PfM0zXpRLSXjxw78hFkA5eTOfVn62Zdx4UbchyUdDujZ7ByW704c1qEx35RCT7sPWwEcAdUW8UnA/JEklgYvDNHQ/SFKOG/t9t2ov6rXxc8dGc+qEE7iS0VVKvBdTHOZEM01b5/ciAX4WwZdSqi8vNrf8a64HRQJkRTUgscpNtFABMyWZ+Ls0NKrvA/k1WrpIHkGGiFhKaCYPOl9H7peM99ksqxxI/iK8vwDpvEgcxO7O3kZ3uJxSHR9XYGftzwvK8WEV5m0vjis/EUJS+5eVERMhIxnkThCxaQ0oHYBL7VmXInGBdWgm8Y5abixgMCaZDnxrAiKbGJCPhFWKfoJnQ3Gn9fRYLjSifZg/DvAgytEceKRMDjAeA9tqvQx5Qv7vr9h2jHphT0nIHaQZvdZc3cKZDvH3DiYFv3P3zEzKkXor2DP/RgmvnUN0AtoRUXptkRj+VDhRrpTYPzRVClY7GUwjCOb+NJPLZupg3rj8CZgDq6KDEsj6uJ0jXRIAbP+vUrdZelZtmuwxLfuStR2g1K6ZkFebX6V2NnS08Zlxj55gMzRtJ4PVnv4FMrMwXKpyweHtKSAbtJdtAyQHzaSe0FFUq8AjHcEwblya374cyzs5umlH4Kzn4nnPW/leR1Adn5rpgAyOhcwqlBZ31bwZc+SjrD6z8ay4jTSrHnmyOvHYsi+0DfS9igoLc3VgpXa2ujgWyHBVy630pS7wFJBE4ubXrWDgFU8SCXfhr6iDvDMmDroyysxicGJravJCDB7799K4Y5Mmpm9AQNyurVModU7UsJRzNzyJs2FMzgdDSTHe6fkMGo3C4/Yz8T4if8d1h92G2S2SnuGFD+CeJpi8UFQz9G2FK 0CankMqM fiXtDGDmAjkbb1DICijqEC0JYPVA7GeyQN3JlqOSKckm2HqeUY8tK/nkGCb5W+jOoQNLU7OcYQyoBd+xv0KMGc5Cnw7sPiI3my2MVRCNEoAHSvYePKt7EXipyFrCPc2o90kt3U5pvOOdC5iKiKslSMTcv0BpNbx2HzwK/R/Mp33NXyh1ZY3nIuoYDoG3c7G+VBlog+U6TqmkOKJrXlFLRaqwlP9pVvXfVFd17eVj/byRUvercQNkelmBCSl6gv5ktZLUXfYPATQaaIfAdnQjitoTxBBB56euH9xaNt27j3RYeBTHEEx4ghyayYW5P+rRFh02dOKRFwD6407vKvfXGQgAPiJ9d2aAbzlCE8vj0wPA9QPrZUQCeQA3SQQwy15QUKgBl2ceMXrgiNGxRMJpUxf2dHjfFdrUvlEH1nD3EWFQ0qS3xRkwylme7jMHyTbQGVBIYhnpmHqPy9NxhOLjcVUYc7SQoXcwvj0bu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000076, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This optimization has no effect. It only ensures that when a zspage was added to its corresponding fullness list, its "inuse" counter was higher or lower than the "inuse" counter of the zspage at the head of the list. The intention was to keep busy zspages at the head, so they could be filled up and moved to the ZS_FULL fullness group more quickly. However, this doesn't work as the "inuse" counter of a zspage can be modified by obj_free() but the zspage may still belong to the same fullness list. So, fix_fullness_group() won't change the zspage's position in relation to the head's "inuse" counter, leading to a largely random order of zspages within the fullness list. For instance, consider a printout of the "inuse" counters of the first 10 zspages in a class that holds 93 objects per zspage: ZS_ALMOST_EMPTY: 36 67 68 64 35 54 63 52 As we can see the zspage with the lowest "inuse" counter is actually the head of the fullness list. Remove this pointless "optimisation". Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3aed46ab7e6c..abe0c4d7942d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -762,19 +762,8 @@ static void insert_zspage(struct size_class *class, struct zspage *zspage, enum fullness_group fullness) { - struct zspage *head; - class_stat_inc(class, fullness, 1); - head = list_first_entry_or_null(&class->fullness_list[fullness], - struct zspage, list); - /* - * We want to see more ZS_FULL pages and less almost empty/full. - * Put pages with higher ->inuse first. - */ - if (head && get_zspage_inuse(zspage) < get_zspage_inuse(head)) - list_add(&zspage->list, &head->list); - else - list_add(&zspage->list, &class->fullness_list[fullness]); + list_add(&zspage->list, &class->fullness_list[fullness]); } /* From patchwork Sat Mar 4 03:48:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13159780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C71BC6FA8E for ; Sat, 4 Mar 2023 03:48:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B52E86B0075; Fri, 3 Mar 2023 22:48:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8E306B0078; Fri, 3 Mar 2023 22:48:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E03A6B007B; Fri, 3 Mar 2023 22:48:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 777E16B0075 for ; Fri, 3 Mar 2023 22:48:50 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 45E1380576 for ; Sat, 4 Mar 2023 03:48:50 +0000 (UTC) X-FDA: 80529834420.28.71E04EA Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf11.hostedemail.com (Postfix) with ESMTP id 6498940004 for ; Sat, 4 Mar 2023 03:48:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SoaMlTEv; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.51 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677901727; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=INMm7/hLJTx2Fj9d5WNhewPWXG18XbmQ5Z9CNzzHrFk=; b=T3i8fcJxPNSpBwPSs5dDjtic7UXuUONMnCgFlVqtoL7WWP2Lb489ZUFM65qxj8/EjOVlka 2Vwa94oCc7ZdeBnQjg6VnVFgYNLeivtlsjFyws4JG+mvTWV1HDbn1fmJrxTVxcuoRTqqCo 2hkWcJdlPtn25QMlsx45sZYqgOUHeP4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SoaMlTEv; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.51 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677901727; a=rsa-sha256; cv=none; b=WE359N73s6SHt9e98wZd3PYZ7LsKxmIWK26Tg9sCgKBuU80c4tcZzlfsdF1eXO2NuGr2pq 0ovwLQCB9jQ4tC54RAy8jXEmDMcdoK7TOiaOuu0bOnOMPKztGKB6xMBxXuw+gko4k0exyS 8d0j28uwS8gqqEvjif/d8rcv4CD6GoU= Received: by mail-pj1-f51.google.com with SMTP id bo22so4590116pjb.4 for ; Fri, 03 Mar 2023 19:48:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=INMm7/hLJTx2Fj9d5WNhewPWXG18XbmQ5Z9CNzzHrFk=; b=SoaMlTEvZDHc49/2GSVpvjEkRvnBLIEerQ8iyW9U4gt9XZ8+LycTaBewMczNArmI2Z ucPl0zr5zq4PJC0/sUazh1P/TutyXZLRJpmo8tHfZ9u4uTUqNWy/MPKlEgoBLZ7z+55A Ek9w6F8N27QCaKKE8CoQP149nM6gQOhqHz99w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=INMm7/hLJTx2Fj9d5WNhewPWXG18XbmQ5Z9CNzzHrFk=; b=4mO/FhittW9wBnb1L11Lup/BtU/c/IfmB/dXdmgYFoSX2FOq4Cfhb7HHP4d5ob/T0F 93m11eTVER2VztHiUWIum4JAt4XRzYbrjWfAkU4vdhKZoe+uljok1ScWXKLFOUVCuvgP KHitN1Jq5/I/zWHxopCqwWZ2JysOx59K/zb+/+KUlaXqVwQESGrdJ1rfZlMQ8KBEO55U tCsPvCz99OegEqDdhXYqmygARqaKTtYmOcKVnpHyLTi/Uvdd8oZWlBmV8oVq061b/ZeL Ez4wxZo1SOHkdd1ZMDX+8lDC96YF0cUnVN765mUUHm9VdPhK/KHeAdDJypKz3I016UVQ xmrw== X-Gm-Message-State: AO0yUKWKlXTXKbEOuFf203EsEAFfBHSftwjnMcc4d5n1T3c4IHapVCHL Y4L4jECnWTOAFBM5H7TCLRK1/Q== X-Google-Smtp-Source: AK7set9RMrUQhTc4WlbS1mnljKuOCdXpvvqCSzlH5usKmgHObVWE0odhxRuUQVvdCgcSXfFaVWNrVA== X-Received: by 2002:a17:902:f54a:b0:19a:98c9:8cea with SMTP id h10-20020a170902f54a00b0019a98c98ceamr5024162plf.39.1677901726091; Fri, 03 Mar 2023 19:48:46 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6ac2:6eee:5465:7ee6]) by smtp.gmail.com with ESMTPSA id d6-20020a170902c18600b00199025284b3sm2249204pld.151.2023.03.03.19.48.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 19:48:45 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv4 2/4] zsmalloc: fine-grained inuse ratio based fullness grouping Date: Sat, 4 Mar 2023 12:48:33 +0900 Message-Id: <20230304034835.2082479-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230304034835.2082479-1-senozhatsky@chromium.org> References: <20230304034835.2082479-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Stat-Signature: 978bo5zyr9ho756tyn8hybnwyy51ouqt X-Rspam-User: X-Rspamd-Queue-Id: 6498940004 X-Rspamd-Server: rspam06 X-HE-Tag: 1677901727-63539 X-HE-Meta: U2FsdGVkX1/YqeMjQZBSQZ5MEPLxCnOmZakGQTccpE5D4oYHcNWHLILKmMn8iYVLTLGhL3b9xIZxhL6x8xsxL1Djuz2VJ3qlsIb+ZBe7ZzsB8/TWwqOn9oLw2chkgwspA+dWaD5WTCuMCPaegndF1fEDVbAscQZvPH26JBKJstZIqklDh9XQUAhE4CHKEvko1kuUuN6LQwLTQykaXwpcFvcYs+GN2nVJW+tWKJuTHmiXTidz7AB8b75Efxgbd0PDbh855E+9iKb37s2AvMKCS4Mhp9fJDD6VrF16nTs/aRy4S9UO428LXjV6W2D/I7l9P/lsLmZlyUOGcd6Y1mkxzkugDWg94B7Hd90ATMkNjzVTgP/iqnDpvc6gCft+1T0tcxYfYADJysHH/NWnC3rY4Apf7ZeEu0GUWVEOlmcv8c00X/XsH70xRU4AqF/r+U0VTa3RvNHCHeiRfBRiEKwvO6kmj+t/T21hNwUe2moxNhRfe4Kb1cdx/VCaNOBhTjQQ2jXVlthdXGhrbvkRw8O3V8TBGhBEyFBfpIXU87ZjesupTQOBTJX//21Zp/FaWRSBeX7IZzSmW1mHVsZM+M8M+LuAuA9Gh0IYwdAUBtm+6DZPhlqv6EZ7qdAo/WOf4l1njyGjFkATDTBeuqT9ZpyIpvU9/7kFDaqC2DRZEIKJ6r73X0bsf8Xm6uk6p9r6dxnXILBOpCM9gW1RjbjAgwKOGEwtb7DgCCXFISvk406xeZ4PoNEHJZzEHcYOwFbH0+go8Xdo7hlIRf3vNX4/PRB7wqJ2IRHMociJXdazqbJjIXoaaGM2+A/z4mIv/JTvNY2GfFEfAHugiaomFezlTndaUqLDpsBNgol815sOTM/aI2Sk+VDmXCG38czuaNEvF3+43vlcxR8V72AshByUzXMLeVlCN4p9HS+1fTbUlKIANFGEPIiYFzXClU7YebXVDGRJ/03BjxXqki5FDoVhUmJ m5ER6eji 0DarhnCvQqfqwU38T8AaBal0byvLq3qVsgedmcBPsA4uNRQDSxdu58ssXnQ5zUzKzGg44QCqT0IKG63rfxepn6du1PXNEaUGMt2b31H2NJwuxE0SrqydeqC7OwossHPq3YVue+4VYs9UQkTndlZaAQDhR8cKNmhi8M20f3+fjXlJAgUcH0+GTys9DaHkxCAnYYffk0Xa+Kj+vai5c65Bmpd9mQOO/Qw18bHHDFtxIxEimZvNlqEJs4KcHJNDiKKm7fW+czFSEoCCZnqEozM35wwbjcvrA20cnts2A6mwNi/P9NHChAs6rTk9xXwtiXodaBr55HTtZhyG4jgPBvcelAl5ZOHovYwGvZ8czCdl6OhNKgpx3JLnVgoZPc8oxCjU53wjX6T9PzcW3roeAAkbP8wFblnkuqiY5VIHIh3IbA9iE6yuN7MOOVP8c5FNwSumlY6WK9gDSruyfqU//bXU8AzCh8cxiD8CMB/Oc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Each zspage maintains ->inuse counter which keeps track of the number of objects stored in the zspage. The ->inuse counter also determines the zspage's "fullness group" which is calculated as the ratio of the "inuse" objects to the total number of objects the zspage can hold (objs_per_zspage). The closer the ->inuse counter is to objs_per_zspage, the better. Each size class maintains several fullness lists, that keep track of zspages of particular "fullness". Pages within each fullness list are stored in random order with regard to the ->inuse counter. This is because sorting the zspages by ->inuse counter each time obj_malloc() or obj_free() is called would be too expensive. However, the ->inuse counter is still a crucial factor in many situations. For the two major zsmalloc operations, zs_malloc() and zs_compact(), we typically select the head zspage from the corresponding fullness list as the best candidate zspage. However, this assumption is not always accurate. For the zs_malloc() operation, the optimal candidate zspage should have the highest ->inuse counter. This is because the goal is to maximize the number of ZS_FULL zspages and make full use of all allocated memory. For the zs_compact() operation, the optimal source zspage should have the lowest ->inuse counter. This is because compaction needs to move objects in use to another page before it can release the zspage and return its physical pages to the buddy allocator. The fewer objects in use, the quicker compaction can release the zspage. Additionally, compaction is measured by the number of pages it releases. This patch reworks the fullness grouping mechanism. Instead of having two groups - ZS_ALMOST_EMPTY (usage ratio below 3/4) and ZS_ALMOST_FULL (usage ration above 3/4) - that result in too many zspages being included in the ALMOST_EMPTY group for specific classes, size classes maintain a larger number of fullness lists that give strict guarantees on the minimum and maximum ->inuse values within each group. Each group represents a 10% change in the ->inuse ratio compared to neighboring groups. In essence, there are groups for zspages with 0%, 10%, 20% usage ratios, and so on, up to 100%. This enhances the selection of candidate zspages for both zs_malloc() and zs_compact(). A printout of the ->inuse counters of the first 7 zspages per (random) class fullness group: class-768 objs_per_zspage 16: fullness 100%: empty fullness 99%: empty fullness 90%: empty fullness 80%: empty fullness 70%: empty fullness 60%: 8 8 9 9 8 8 8 fullness 50%: empty fullness 40%: 5 5 6 5 5 5 5 fullness 30%: 4 4 4 4 4 4 4 fullness 20%: 2 3 2 3 3 2 2 fullness 10%: 1 1 1 1 1 1 1 fullness 0%: empty The zs_malloc() function searches through the groups of pages starting with the one having the highest usage ratio. This means that it always selects a zspage from the group with the least internal fragmentation (highest usage ratio) and makes it even less fragmented by increasing its usage ratio. The zs_compact() function, on the other hand, begins by scanning the group with the highest fragmentation (lowest usage ratio) to locate the source page. The first available zspage is selected, and then the function moves downward to find a destination zspage in the group with the lowest internal fragmentation (highest usage ratio). Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 248 ++++++++++++++++++++++++++------------------------ 1 file changed, 130 insertions(+), 118 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index abe0c4d7942d..cc59336a966a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -127,7 +127,7 @@ #define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1) #define HUGE_BITS 1 -#define FULLNESS_BITS 2 +#define FULLNESS_BITS 4 #define CLASS_BITS 8 #define ISOLATED_BITS 5 #define MAGIC_VAL_BITS 8 @@ -159,51 +159,46 @@ #define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ ZS_SIZE_CLASS_DELTA) + 1) +/* + * Pages are distinguished by the ratio of used memory (that is the ratio + * of ->inuse objects to all objects that page can store). For example, + * INUSE_RATIO_10 means that the ratio of used objects is > 0% and <= 10%. + * + * The number of fullness groups is not random. It allows us to keep + * difference between the least busy page in the group (minimum permitted + * number of ->inuse objects) and the most busy page (maximum permitted + * number of ->inuse objects) at a reasonable value. + */ enum fullness_group { - ZS_EMPTY, - ZS_ALMOST_EMPTY, - ZS_ALMOST_FULL, - ZS_FULL, - NR_ZS_FULLNESS, + ZS_INUSE_RATIO_0, + ZS_INUSE_RATIO_10, + /* NOTE: 5 more fullness groups here */ + ZS_INUSE_RATIO_70 = 7, + /* NOTE: 2 more fullness groups here */ + ZS_INUSE_RATIO_99 = 10, + ZS_INUSE_RATIO_100, + NR_FULLNESS_GROUPS, }; enum class_stat_type { - CLASS_EMPTY, - CLASS_ALMOST_EMPTY, - CLASS_ALMOST_FULL, - CLASS_FULL, - OBJ_ALLOCATED, - OBJ_USED, - NR_ZS_STAT_TYPE, + /* NOTE: stats for 12 fullness groups here: from inuse 0 to 100 */ + ZS_OBJS_ALLOCATED = NR_FULLNESS_GROUPS, + ZS_OBJS_INUSE, + NR_CLASS_STAT_TYPES, }; struct zs_size_stat { - unsigned long objs[NR_ZS_STAT_TYPE]; + unsigned long objs[NR_CLASS_STAT_TYPES]; }; #ifdef CONFIG_ZSMALLOC_STAT static struct dentry *zs_stat_root; #endif -/* - * We assign a page to ZS_ALMOST_EMPTY fullness group when: - * n <= N / f, where - * n = number of allocated objects - * N = total number of objects zspage can store - * f = fullness_threshold_frac - * - * Similarly, we assign zspage to: - * ZS_ALMOST_FULL when n > N / f - * ZS_EMPTY when n == 0 - * ZS_FULL when n == N - * - * (see: fix_fullness_group()) - */ -static const int fullness_threshold_frac = 4; static size_t huge_class_size; struct size_class { - struct list_head fullness_list[NR_ZS_FULLNESS]; + struct list_head fullness_list[NR_FULLNESS_GROUPS]; /* * Size of objects stored in this class. Must be multiple * of ZS_ALIGN. @@ -547,8 +542,8 @@ static inline void set_freeobj(struct zspage *zspage, unsigned int obj) } static void get_zspage_mapping(struct zspage *zspage, - unsigned int *class_idx, - enum fullness_group *fullness) + unsigned int *class_idx, + int *fullness) { BUG_ON(zspage->magic != ZSPAGE_MAGIC); @@ -557,14 +552,14 @@ static void get_zspage_mapping(struct zspage *zspage, } static struct size_class *zspage_class(struct zs_pool *pool, - struct zspage *zspage) + struct zspage *zspage) { return pool->size_class[zspage->class]; } static void set_zspage_mapping(struct zspage *zspage, - unsigned int class_idx, - enum fullness_group fullness) + unsigned int class_idx, + int fullness) { zspage->class = class_idx; zspage->fullness = fullness; @@ -588,23 +583,19 @@ static int get_size_class_index(int size) return min_t(int, ZS_SIZE_CLASSES - 1, idx); } -/* type can be of enum type class_stat_type or fullness_group */ static inline void class_stat_inc(struct size_class *class, int type, unsigned long cnt) { class->stats.objs[type] += cnt; } -/* type can be of enum type class_stat_type or fullness_group */ static inline void class_stat_dec(struct size_class *class, int type, unsigned long cnt) { class->stats.objs[type] -= cnt; } -/* type can be of enum type class_stat_type or fullness_group */ -static inline unsigned long zs_stat_get(struct size_class *class, - int type) +static inline unsigned long zs_stat_get(struct size_class *class, int type) { return class->stats.objs[type]; } @@ -646,16 +637,27 @@ static int zs_stats_size_show(struct seq_file *s, void *v) "pages_per_zspage", "freeable"); for (i = 0; i < ZS_SIZE_CLASSES; i++) { + int fg; + class = pool->size_class[i]; if (class->index != i) continue; spin_lock(&pool->lock); - class_almost_full = zs_stat_get(class, CLASS_ALMOST_FULL); - class_almost_empty = zs_stat_get(class, CLASS_ALMOST_EMPTY); - obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); - obj_used = zs_stat_get(class, OBJ_USED); + class_almost_full = 0; + class_almost_empty = 0; + /* + * Replicate old behaviour for almost_full and almost_empty + * stats. + */ + for (fg = ZS_INUSE_RATIO_70; fg <= ZS_INUSE_RATIO_99; fg++) + class_almost_full += zs_stat_get(class, fg); + for (fg = ZS_INUSE_RATIO_10; fg < ZS_INUSE_RATIO_70; fg++) + class_almost_empty += zs_stat_get(class, fg); + + obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); + obj_used = zs_stat_get(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); spin_unlock(&pool->lock); @@ -726,30 +728,28 @@ static inline void zs_pool_stat_destroy(struct zs_pool *pool) /* * For each size class, zspages are divided into different groups - * depending on how "full" they are. This was done so that we could - * easily find empty or nearly empty zspages when we try to shrink - * the pool (not yet implemented). This function returns fullness + * depending on their usage ratio. This function returns fullness * status of the given page. */ -static enum fullness_group get_fullness_group(struct size_class *class, - struct zspage *zspage) +static int get_fullness_group(struct size_class *class, struct zspage *zspage) { - int inuse, objs_per_zspage; - enum fullness_group fg; + int inuse, objs_per_zspage, ratio; inuse = get_zspage_inuse(zspage); objs_per_zspage = class->objs_per_zspage; if (inuse == 0) - fg = ZS_EMPTY; - else if (inuse == objs_per_zspage) - fg = ZS_FULL; - else if (inuse <= 3 * objs_per_zspage / fullness_threshold_frac) - fg = ZS_ALMOST_EMPTY; - else - fg = ZS_ALMOST_FULL; + return ZS_INUSE_RATIO_0; + if (inuse == objs_per_zspage) + return ZS_INUSE_RATIO_100; - return fg; + ratio = 100 * inuse / objs_per_zspage; + /* + * Take integer division into consideration: a page with one inuse + * object out of 127 possible, will end up having 0 usage ratio, + * which is wrong as it belongs in ZS_INUSE_RATIO_10 fullness group. + */ + return ratio / 10 + 1; } /* @@ -760,7 +760,7 @@ static enum fullness_group get_fullness_group(struct size_class *class, */ static void insert_zspage(struct size_class *class, struct zspage *zspage, - enum fullness_group fullness) + int fullness) { class_stat_inc(class, fullness, 1); list_add(&zspage->list, &class->fullness_list[fullness]); @@ -772,7 +772,7 @@ static void insert_zspage(struct size_class *class, */ static void remove_zspage(struct size_class *class, struct zspage *zspage, - enum fullness_group fullness) + int fullness) { VM_BUG_ON(list_empty(&class->fullness_list[fullness])); @@ -783,17 +783,16 @@ static void remove_zspage(struct size_class *class, /* * Each size class maintains zspages in different fullness groups depending * on the number of live objects they contain. When allocating or freeing - * objects, the fullness status of the page can change, say, from ALMOST_FULL - * to ALMOST_EMPTY when freeing an object. This function checks if such - * a status change has occurred for the given page and accordingly moves the - * page from the freelist of the old fullness group to that of the new - * fullness group. + * objects, the fullness status of the page can change, for instance, from + * INUSE_RATIO_80 to INUSE_RATIO_70 when freeing an object. This function + * checks if such a status change has occurred for the given page and + * accordingly moves the page from the list of the old fullness group to that + * of the new fullness group. */ -static enum fullness_group fix_fullness_group(struct size_class *class, - struct zspage *zspage) +static int fix_fullness_group(struct size_class *class, struct zspage *zspage) { int class_idx; - enum fullness_group currfg, newfg; + int currfg, newfg; get_zspage_mapping(zspage, &class_idx, &currfg); newfg = get_fullness_group(class, zspage); @@ -966,7 +965,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { struct page *page, *next; - enum fullness_group fg; + int fg; unsigned int class_idx; get_zspage_mapping(zspage, &class_idx, &fg); @@ -974,7 +973,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, assert_spin_locked(&pool->lock); VM_BUG_ON(get_zspage_inuse(zspage)); - VM_BUG_ON(fg != ZS_EMPTY); + VM_BUG_ON(fg != ZS_INUSE_RATIO_0); /* Free all deferred handles from zs_free */ free_handles(pool, class, zspage); @@ -992,9 +991,8 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, cache_free_zspage(pool, zspage); - class_stat_dec(class, OBJ_ALLOCATED, class->objs_per_zspage); - atomic_long_sub(class->pages_per_zspage, - &pool->pages_allocated); + class_stat_dec(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); + atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); } static void free_zspage(struct zs_pool *pool, struct size_class *class, @@ -1013,7 +1011,7 @@ static void free_zspage(struct zs_pool *pool, struct size_class *class, return; } - remove_zspage(class, zspage, ZS_EMPTY); + remove_zspage(class, zspage, ZS_INUSE_RATIO_0); #ifdef CONFIG_ZPOOL list_del(&zspage->lru); #endif @@ -1149,9 +1147,9 @@ static struct zspage *find_get_zspage(struct size_class *class) int i; struct zspage *zspage; - for (i = ZS_ALMOST_FULL; i >= ZS_EMPTY; i--) { + for (i = ZS_INUSE_RATIO_99; i >= ZS_INUSE_RATIO_0; i--) { zspage = list_first_entry_or_null(&class->fullness_list[i], - struct zspage, list); + struct zspage, list); if (zspage) break; } @@ -1510,7 +1508,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) { unsigned long handle, obj; struct size_class *class; - enum fullness_group newfg; + int newfg; struct zspage *zspage; if (unlikely(!size || size > ZS_MAX_ALLOC_SIZE)) @@ -1532,7 +1530,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); record_obj(handle, obj); - class_stat_inc(class, OBJ_USED, 1); + class_stat_inc(class, ZS_OBJS_INUSE, 1); spin_unlock(&pool->lock); return handle; @@ -1552,10 +1550,9 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) insert_zspage(class, zspage, newfg); set_zspage_mapping(zspage, class->index, newfg); record_obj(handle, obj); - atomic_long_add(class->pages_per_zspage, - &pool->pages_allocated); - class_stat_inc(class, OBJ_ALLOCATED, class->objs_per_zspage); - class_stat_inc(class, OBJ_USED, 1); + atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); + class_stat_inc(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); + class_stat_inc(class, ZS_OBJS_INUSE, 1); /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); @@ -1611,7 +1608,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) struct page *f_page; unsigned long obj; struct size_class *class; - enum fullness_group fullness; + int fullness; if (IS_ERR_OR_NULL((void *)handle)) return; @@ -1626,7 +1623,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) zspage = get_zspage(f_page); class = zspage_class(pool, zspage); - class_stat_dec(class, OBJ_USED, 1); + class_stat_dec(class, ZS_OBJS_INUSE, 1); #ifdef CONFIG_ZPOOL if (zspage->under_reclaim) { @@ -1644,7 +1641,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj, NULL); fullness = fix_fullness_group(class, zspage); - if (fullness == ZS_EMPTY) + if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); spin_unlock(&pool->lock); @@ -1826,22 +1823,33 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, return ret; } -static struct zspage *isolate_zspage(struct size_class *class, bool source) +static struct zspage *isolate_src_zspage(struct size_class *class) { - int i; struct zspage *zspage; - enum fullness_group fg[2] = {ZS_ALMOST_EMPTY, ZS_ALMOST_FULL}; + int fg; - if (!source) { - fg[0] = ZS_ALMOST_FULL; - fg[1] = ZS_ALMOST_EMPTY; + for (fg = ZS_INUSE_RATIO_10; fg <= ZS_INUSE_RATIO_99; fg++) { + zspage = list_first_entry_or_null(&class->fullness_list[fg], + struct zspage, list); + if (zspage) { + remove_zspage(class, zspage, fg); + return zspage; + } } - for (i = 0; i < 2; i++) { - zspage = list_first_entry_or_null(&class->fullness_list[fg[i]], - struct zspage, list); + return zspage; +} + +static struct zspage *isolate_dst_zspage(struct size_class *class) +{ + struct zspage *zspage; + int fg; + + for (fg = ZS_INUSE_RATIO_99; fg >= ZS_INUSE_RATIO_10; fg--) { + zspage = list_first_entry_or_null(&class->fullness_list[fg], + struct zspage, list); if (zspage) { - remove_zspage(class, zspage, fg[i]); + remove_zspage(class, zspage, fg); return zspage; } } @@ -1854,12 +1862,11 @@ static struct zspage *isolate_zspage(struct size_class *class, bool source) * @class: destination class * @zspage: target page * - * Return @zspage's fullness_group + * Return @zspage's fullness status */ -static enum fullness_group putback_zspage(struct size_class *class, - struct zspage *zspage) +static int putback_zspage(struct size_class *class, struct zspage *zspage) { - enum fullness_group fullness; + int fullness; fullness = get_fullness_group(class, zspage); insert_zspage(class, zspage, fullness); @@ -2123,7 +2130,7 @@ static void async_free_zspage(struct work_struct *work) int i; struct size_class *class; unsigned int class_idx; - enum fullness_group fullness; + int fullness; struct zspage *zspage, *tmp; LIST_HEAD(free_pages); struct zs_pool *pool = container_of(work, struct zs_pool, @@ -2135,7 +2142,8 @@ static void async_free_zspage(struct work_struct *work) continue; spin_lock(&pool->lock); - list_splice_init(&class->fullness_list[ZS_EMPTY], &free_pages); + list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], + &free_pages); spin_unlock(&pool->lock); } @@ -2144,7 +2152,7 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); get_zspage_mapping(zspage, &class_idx, &fullness); - VM_BUG_ON(fullness != ZS_EMPTY); + VM_BUG_ON(fullness != ZS_INUSE_RATIO_0); class = pool->size_class[class_idx]; spin_lock(&pool->lock); #ifdef CONFIG_ZPOOL @@ -2192,8 +2200,8 @@ static inline void zs_flush_migration(struct zs_pool *pool) { } static unsigned long zs_can_compact(struct size_class *class) { unsigned long obj_wasted; - unsigned long obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); - unsigned long obj_used = zs_stat_get(class, OBJ_USED); + unsigned long obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); + unsigned long obj_used = zs_stat_get(class, ZS_OBJS_INUSE); if (obj_allocated <= obj_used) return 0; @@ -2217,7 +2225,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ spin_lock(&pool->lock); - while ((src_zspage = isolate_zspage(class, true))) { + while ((src_zspage = isolate_src_zspage(class))) { /* protect someone accessing the zspage(i.e., zs_map_object) */ migrate_write_lock(src_zspage); @@ -2227,7 +2235,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, cc.obj_idx = 0; cc.s_page = get_first_page(src_zspage); - while ((dst_zspage = isolate_zspage(class, false))) { + while ((dst_zspage = isolate_dst_zspage(class))) { migrate_write_lock_nested(dst_zspage); cc.d_page = get_first_page(dst_zspage); @@ -2252,7 +2260,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); - if (putback_zspage(class, src_zspage) == ZS_EMPTY) { + if (putback_zspage(class, src_zspage) == ZS_INUSE_RATIO_0) { migrate_write_unlock(src_zspage); free_zspage(pool, class, src_zspage); pages_freed += class->pages_per_zspage; @@ -2410,7 +2418,7 @@ struct zs_pool *zs_create_pool(const char *name) int pages_per_zspage; int objs_per_zspage; struct size_class *class; - int fullness = 0; + int fullness; size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; if (size > ZS_MAX_ALLOC_SIZE) @@ -2464,9 +2472,12 @@ struct zs_pool *zs_create_pool(const char *name) class->pages_per_zspage = pages_per_zspage; class->objs_per_zspage = objs_per_zspage; pool->size_class[i] = class; - for (fullness = ZS_EMPTY; fullness < NR_ZS_FULLNESS; - fullness++) + + fullness = ZS_INUSE_RATIO_0; + while (fullness < NR_FULLNESS_GROUPS) { INIT_LIST_HEAD(&class->fullness_list[fullness]); + fullness++; + } prev_class = class; } @@ -2512,11 +2523,12 @@ void zs_destroy_pool(struct zs_pool *pool) if (class->index != i) continue; - for (fg = ZS_EMPTY; fg < NR_ZS_FULLNESS; fg++) { - if (!list_empty(&class->fullness_list[fg])) { - pr_info("Freeing non-empty class with size %db, fullness group %d\n", - class->size, fg); - } + for (fg = ZS_INUSE_RATIO_0; fg < NR_FULLNESS_GROUPS; fg++) { + if (list_empty(&class->fullness_list[fg])) + continue; + + pr_err("Class-%d fullness group %d is not empty\n", + class->size, fg); } kfree(class); } @@ -2618,7 +2630,7 @@ static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) unsigned long handle; struct zspage *zspage; struct page *page; - enum fullness_group fullness; + int fullness; /* Lock LRU and fullness list */ spin_lock(&pool->lock); @@ -2688,7 +2700,7 @@ static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) * while the page is removed from the pool. Fix it * up for the check in __free_zspage(). */ - zspage->fullness = ZS_EMPTY; + zspage->fullness = ZS_INUSE_RATIO_0; __free_zspage(pool, class, zspage); spin_unlock(&pool->lock); From patchwork Sat Mar 4 03:48:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13159781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3042DC678D5 for ; Sat, 4 Mar 2023 03:48:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C16C96B007B; Fri, 3 Mar 2023 22:48:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B7A796B007D; Fri, 3 Mar 2023 22:48:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9CBFF6B007E; Fri, 3 Mar 2023 22:48:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8334A6B007B for ; Fri, 3 Mar 2023 22:48:52 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4F5C8AAEE3 for ; Sat, 4 Mar 2023 03:48:52 +0000 (UTC) X-FDA: 80529834504.18.5673D47 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf24.hostedemail.com (Postfix) with ESMTP id 79D0D180003 for ; Sat, 4 Mar 2023 03:48:49 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=PoJckSaX; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677901729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mxRoi4TukkX6UlbjK9WtsqOLfwrBv6GFUJd0cC4XWDw=; b=g6zzZfMKm7eQAiZ69pXUJjdwUBXt6kYtrWxjBPA7lzqeFMPcFTSVUzb2FCNYEDLgDRMlmz SQ5ZKyZSfSI9e0Lpk/C4FwimX6yHoUg52JS5mI5KHAiPHzQWH4VAx+MH5dKHeowwfyrHDE S2Fnn9qIazZGR+HFTowxeEXkBBFrSLs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=PoJckSaX; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677901729; a=rsa-sha256; cv=none; b=8CaQCOBafQB+YOUwTHWxmMD9rsMDs00omusrMbCduh0G0CtoIpytNYkbtghFFOGsiw+4I1 dR3a1hHDUh/HvIzPGgGuOF9IBaH41UBOncZvRVj4Rybc/ER5nBVr+y2bS5D2jr1DXe0oUb oPTWto4vCj89wM1HNgLjxeLtT31KrHE= Received: by mail-pl1-f178.google.com with SMTP id y11so4817354plg.1 for ; Fri, 03 Mar 2023 19:48:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mxRoi4TukkX6UlbjK9WtsqOLfwrBv6GFUJd0cC4XWDw=; b=PoJckSaXrWf6myxlSmql82ndp3fdHyLe8QuReRZtA53h8cgepOrcFc3wAxf5T+880d JxwL2bFvEx57bJWamf060g4cq+HktGLGh3pibGPW+OlzO+ceBhngA6O7ZtD+EcQopQ6T MaCJNdUlbXoZGJC2ah7Uxd+e2hfQW5hlYfvYw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mxRoi4TukkX6UlbjK9WtsqOLfwrBv6GFUJd0cC4XWDw=; b=ckK62XuV5FKAOgw2GxOxSVrfuXBR30yU4Pk+LyFGR7KWZbCTcT8g/JrTklmvMU2OeD nEMbbyMruVNe085TdKRx9T4JwDrdtx0W/OlLmsZEr7xgXTAIDOvuA9wA+tTU2ll/ycyY LpCE1vnHDkzq5fGvLfcj2FF2dHMbRql618pt52GI9RPbB+bfBbaiar8+PiCf6PF2joiq 8lKjw+zRyJIal1O/ZGg7WtIuPrKKTaZJYo0L1GvCp87//V+DXsZSBHu1SHshePNiA2Qk NF15xOk4jJKDOGmxV5pXQM+mVAl1MSmsDtkvAvVJOqHnMr52STXAMuJ8pvgGaRdQMzGE Oukw== X-Gm-Message-State: AO0yUKVPFeanZQu6saF16ehbWmpNZYEPTjXoFvGC3lnTXtN+4Vp9pAsE VNRuRGUs1sfHqyIJwWFbGLlLDw== X-Google-Smtp-Source: AK7set8zpcSUINaMZnuyOqbfw1GR5LVivHiboO47W0G55Vv1CAgAtziIWwGghZEnWnNpAu3eWy170Q== X-Received: by 2002:a17:903:1cb:b0:19c:eaf0:9859 with SMTP id e11-20020a17090301cb00b0019ceaf09859mr5610861plh.38.1677901728469; Fri, 03 Mar 2023 19:48:48 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6ac2:6eee:5465:7ee6]) by smtp.gmail.com with ESMTPSA id d6-20020a170902c18600b00199025284b3sm2249204pld.151.2023.03.03.19.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 19:48:48 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv4 3/4] zsmalloc: rework compaction algorithm Date: Sat, 4 Mar 2023 12:48:34 +0900 Message-Id: <20230304034835.2082479-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230304034835.2082479-1-senozhatsky@chromium.org> References: <20230304034835.2082479-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 79D0D180003 X-Stat-Signature: qrap9h6f8jyp9uebh71p9p8xskczpx8z X-HE-Tag: 1677901729-360327 X-HE-Meta: U2FsdGVkX1/djC3/mCCwr/ft1+KSCB0Ljublu6yy38bqcbiPrrMyBFUe9DHSCq5b8fDlhMcKblk0pmV8i5QIOpTYJqMzAbdcm0DP6xzAybbZ1d3iRzKKuR3PAieWiEgLheq8DxP9tMuHiTSMGy4pCb1fLm0+jcyEHjKOgkFUAig2QYps8m0Oc40kGjyiSbUBRc/kW0TVsEXCG0/Oy/af8aHIFJbDc+76OIU+Z6gCGRmD77eBLbVO/HpXUlSLjI4mkIMdiRXcLjanZE3/y3h3lgH6pNvov9KN8hQ0TlKsoRfxOsWWEmSboOHuEoD+A9aKPxoo8K9eghX9NiK9PXY0KMZbR68QUt5WEAj8cqzXJ3iu4Wta6B5jAb67FDNXzrq8ODvlOmQ6Yw+E7OWc6SND8f/Yqri0y13x+59tzCawbd0KCz05nPld4g2i/lZhzIeart5Mj33fj3KxVrUm1oUpWV/3TWogFk/c3rjx2DK9+/6PFfIBdBLE6UfjAKq0d0oDQOoZZW3EmWLZuWqyi+AiqEY66agA9qx/Ydz6x23I7ef+Cqr0l6OqYtukd1ZaspYfkAOOJKuWNA140XPhxr9QnOIIydNxoTbj5BdFpUPtV7mWwY3D2du6shMdO842vYpLTIfH+ymp321W5/ItpTD6b9f7ovwVYt3AxU2rk0dDCSl2Ke5Sg3iNnCJD8X1I1Tp9SCHMNaV8kbVbJ11gs7jVc4H97bjhXPGVkjy+YtSoY1Cj5+aiYlnk+UQhUIcIxuZvNdDj80bd3sZ/iGDfvH8RgMcGAyjfEJNHAyT6bSHx0RM+qTqbT5FNYDb+P7WLO+LD91n7dL64NWplbZxT3TbNUFXrG+ulSxEdnrF7HZyyxlYOmmg3yitD3djYxn8A9CY4iuN8M+uODxLtQJH+plqEJ4SfaxV4JLk79lebIjIVeq7ds6F02zBZs8SZFyOnd32HhoqT2O8Z7MFdF6AGcNH ERq3ZtvH ttuO1hmScG+AuXqJY4K9REQTfAYm3Qw6/awUPddLmu6kmaW+43adu78C6EeJw6kha4FpqmUI06qnL92tWHivoE4O+DbTd7H5pOYPU3pTXJ8UzB/Zth+unPVF0uPveh6c3/gsNEcLwB0L5OrQ0ci7k0JJvXVaQscM4bsn+oZNrqrWmFWdY8rB4CIgKkNsznhTg30c3d3DA44ZbIEGgTGb8hBbAuCidBDg5GMEvduvQMRLyYUzXJdDEvsdGx78ObBFR+4Pa2LtXyAqXf1FyqgBkzc2uX1g47BkIdgthkeJeV0nUXeSUzshA0vvFrA4M0C5TXuA7iPYIQ3zrZIt/215Ov2ooEwVxRwFs6o5i1JGKZv+543rs5kqZRIOpFmnralB2UxEZT23YhxNls3j44px1J9PZGD+rrUFzbl7ygpovM4IguGg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.040623, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The zsmalloc compaction algorithm has the potential to waste some CPU cycles, particularly when compacting pages within the same fullness group. This is due to the way it selects the head page of the fullness list for source and destination pages, and how it reinserts those pages during each iteration. The algorithm may first use a page as a migration destination and then as a migration source, leading to an unnecessary back-and-forth movement of objects. Consider the following fullness list: PageA PageB PageC PageD PageE During the first iteration, the compaction algorithm will select PageA as the source and PageB as the destination. All of PageA's objects will be moved to PageB, and then PageA will be released while PageB is reinserted into the fullness list. PageB PageC PageD PageE During the next iteration, the compaction algorithm will again select the head of the list as the source and destination, meaning that PageB will now serve as the source and PageC as the destination. This will result in the objects being moved away from PageB, the same objects that were just moved to PageB in the previous iteration. To prevent this avalanche effect, the compaction algorithm should not reinsert the destination page between iterations. By doing so, the most optimal page will continue to be used and its usage ratio will increase, reducing internal fragmentation. The destination page should only be reinserted into the fullness list if: - It becomes full - No source page is available. TEST ==== It's very challenging to reliably test this series. I ended up developing my own synthetic test that has 100% reproducibility. The test generates significan fragmentation (for each size class) and then performs compaction for each class individually and tracks the number of memcpy() in zs_object_copy(), so that we can compare the amount work compaction does on per-class basis. Total amount of work (zram mm_stat objs_moved) ---------------------------------------------- Old fullness grouping, old compaction algorithm: 323977 memcpy() in zs_object_copy(). Old fullness grouping, new compaction algorithm: 262944 memcpy() in zs_object_copy(). New fullness grouping, new compaction algorithm: 213978 memcpy() in zs_object_copy(). Per-class compaction memcpy() comparison (T-test) ------------------------------------------------- x Old fullness grouping, old compaction algorithm + Old fullness grouping, new compaction algorithm N Min Max Median Avg Stddev x 140 349 3513 2461 2314.1214 806.03271 + 140 289 2778 2006 1878.1714 641.02073 Difference at 95.0% confidence -435.95 +/- 170.595 -18.8387% +/- 7.37193% (Student's t, pooled s = 728.216) x Old fullness grouping, old compaction algorithm + New fullness grouping, new compaction algorithm N Min Max Median Avg Stddev x 140 349 3513 2461 2314.1214 806.03271 + 140 226 2279 1644 1528.4143 524.85268 Difference at 95.0% confidence -785.707 +/- 159.331 -33.9527% +/- 6.88516% (Student's t, pooled s = 680.132) Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 78 ++++++++++++++++++++++++--------------------------- 1 file changed, 36 insertions(+), 42 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index cc59336a966a..a61540afbb28 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1782,15 +1782,14 @@ struct zs_compact_control { int obj_idx; }; -static int migrate_zspage(struct zs_pool *pool, struct size_class *class, - struct zs_compact_control *cc) +static void migrate_zspage(struct zs_pool *pool, struct size_class *class, + struct zs_compact_control *cc) { unsigned long used_obj, free_obj; unsigned long handle; struct page *s_page = cc->s_page; struct page *d_page = cc->d_page; int obj_idx = cc->obj_idx; - int ret = 0; while (1) { handle = find_alloced_obj(class, s_page, &obj_idx); @@ -1803,10 +1802,8 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, } /* Stop if there is no more space */ - if (zspage_full(class, get_zspage(d_page))) { - ret = -ENOMEM; + if (zspage_full(class, get_zspage(d_page))) break; - } used_obj = handle_to_obj(handle); free_obj = obj_malloc(pool, get_zspage(d_page), handle); @@ -1819,8 +1816,6 @@ static int migrate_zspage(struct zs_pool *pool, struct size_class *class, /* Remember last position in this iteration */ cc->s_page = s_page; cc->obj_idx = obj_idx; - - return ret; } static struct zspage *isolate_src_zspage(struct size_class *class) @@ -2216,7 +2211,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, struct size_class *class) { struct zs_compact_control cc; - struct zspage *src_zspage; + struct zspage *src_zspage = NULL; struct zspage *dst_zspage = NULL; unsigned long pages_freed = 0; @@ -2225,50 +2220,45 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ spin_lock(&pool->lock); - while ((src_zspage = isolate_src_zspage(class))) { - /* protect someone accessing the zspage(i.e., zs_map_object) */ - migrate_write_lock(src_zspage); + while (zs_can_compact(class)) { + int fg; - if (!zs_can_compact(class)) + if (!dst_zspage) { + dst_zspage = isolate_dst_zspage(class); + if (!dst_zspage) + break; + migrate_write_lock(dst_zspage); + cc.d_page = get_first_page(dst_zspage); + } + + src_zspage = isolate_src_zspage(class); + if (!src_zspage) break; + migrate_write_lock_nested(src_zspage); + cc.obj_idx = 0; cc.s_page = get_first_page(src_zspage); + migrate_zspage(pool, class, &cc); + fg = putback_zspage(class, src_zspage); + migrate_write_unlock(src_zspage); - while ((dst_zspage = isolate_dst_zspage(class))) { - migrate_write_lock_nested(dst_zspage); - - cc.d_page = get_first_page(dst_zspage); - /* - * If there is no more space in dst_page, resched - * and see if anyone had allocated another zspage. - */ - if (!migrate_zspage(pool, class, &cc)) - break; + if (fg == ZS_INUSE_RATIO_0) { + free_zspage(pool, class, src_zspage); + pages_freed += class->pages_per_zspage; + src_zspage = NULL; + } + if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 + || spin_is_contended(&pool->lock)) { putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); dst_zspage = NULL; - if (spin_is_contended(&pool->lock)) - break; - } - /* Stop if we couldn't find slot */ - if (dst_zspage == NULL) - break; - - putback_zspage(class, dst_zspage); - migrate_write_unlock(dst_zspage); - - if (putback_zspage(class, src_zspage) == ZS_INUSE_RATIO_0) { - migrate_write_unlock(src_zspage); - free_zspage(pool, class, src_zspage); - pages_freed += class->pages_per_zspage; - } else - migrate_write_unlock(src_zspage); - spin_unlock(&pool->lock); - cond_resched(); - spin_lock(&pool->lock); + spin_unlock(&pool->lock); + cond_resched(); + spin_lock(&pool->lock); + } } if (src_zspage) { @@ -2276,6 +2266,10 @@ static unsigned long __zs_compact(struct zs_pool *pool, migrate_write_unlock(src_zspage); } + if (dst_zspage) { + putback_zspage(class, dst_zspage); + migrate_write_unlock(dst_zspage); + } spin_unlock(&pool->lock); return pages_freed; From patchwork Sat Mar 4 03:48:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13159782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277D3C6FD18 for ; Sat, 4 Mar 2023 03:48:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B55C76B007D; Fri, 3 Mar 2023 22:48:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB7BB6B007E; Fri, 3 Mar 2023 22:48:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92ED56B0080; Fri, 3 Mar 2023 22:48:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 844D56B007D for ; Fri, 3 Mar 2023 22:48:54 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 55CD1A0243 for ; Sat, 4 Mar 2023 03:48:54 +0000 (UTC) X-FDA: 80529834588.01.31C377B Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf04.hostedemail.com (Postfix) with ESMTP id 8A72640012 for ; Sat, 4 Mar 2023 03:48:52 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ma0hJFSM; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677901732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VDa9Rt5gN+wqseziIq8dAlRWmwkEfGNI/W086ilXD04=; b=rdVU9tsVa0hQnOZasgDQJnGhgyNwLwYWcuLcTt0RYKar4HuANEquo6t+blDnexdtCcUrE4 Z6kYNWPqvjusEcGbxqaxglL0mNpiP230qLVB9xrD15dM7jEnA4dI3YX41prPNZ5H1GiQ/x ndl16WGxWPxR93CcsxaR0+8k+tvC7LA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ma0hJFSM; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677901732; a=rsa-sha256; cv=none; b=G7JsNAsaNP/3UivlDnXmjbweo2oRMY5asGXG8GF0QPrmGDkGgtL+6n7q57MjtagAJcnE+z QjICZteLoAzb9wRDTicQd3He4Qdq2X1Riomezt/50wQs7Pn5mAL74lH/QRvWtXQ6j3mKXT DGxkG7VK5PnkUTI50V4jU4byhr/n5lo= Received: by mail-pl1-f182.google.com with SMTP id v11so4771673plz.8 for ; Fri, 03 Mar 2023 19:48:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VDa9Rt5gN+wqseziIq8dAlRWmwkEfGNI/W086ilXD04=; b=ma0hJFSMBbjPeOiDUGSPZH6UdO2Bu0ejhFXINdaxj+UMAC+rJPSDWse/UcI3XjdJB7 XawTUDuCcYalqmSOEk5p0AaYQkVlIA52BdOk2CEjoo8NtEnudjtGwVlkDwdoKSwqAHFz nVpsbZEnKloellNxEGTwamXMjFp6+noHmgOSg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VDa9Rt5gN+wqseziIq8dAlRWmwkEfGNI/W086ilXD04=; b=VkNB51DDi8ajWS/Ghg2scdTt0HSasdd69ObMzSeuEI64tT3M3CfS3ibizc+taLrNX4 2FZ4iBvH5QGoao1iU+DSk9mbteAniSyeSBnJcFmabF1EJhYDwmCpCtBnk11eNSpAGG/W QeTRJooUMPmZcXIuA9u5COdkYw2zFTL9fImljIxjPqtp80rmYCxwubg1s4UhhhjJ46LT dCcWlXafZvwZTbXqMM4F7717iejn89GjPJaFpFoD5v0NNHhX9EKWlhdZNMVzp3+yMJnG bdk6/eO/eLF2+34220KpjXopec6Q4wOzqlB8SsxVxeBL1n/5hZnLFDhE37QBMsnrU6eM g9ug== X-Gm-Message-State: AO0yUKVdM8xi6h2Mtc2XSWrdTFyMZ3Ig8uy9mIka1nlRzZ6LzWNDyJag 7xVLVQHzqEPtnUntFYcIXZGzFw== X-Google-Smtp-Source: AK7set+fQINFAZ8pGr8zRqFr/P11wMfjLik9Ns7+gC6nbPtWNZv37U4HFsrWhKuOWZaethOn90CMKw== X-Received: by 2002:a17:903:1249:b0:19a:ad90:4223 with SMTP id u9-20020a170903124900b0019aad904223mr4358621plh.48.1677901731582; Fri, 03 Mar 2023 19:48:51 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:6ac2:6eee:5465:7ee6]) by smtp.gmail.com with ESMTPSA id d6-20020a170902c18600b00199025284b3sm2249204pld.151.2023.03.03.19.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 19:48:51 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv4 4/4] zsmalloc: show per fullness group class stats Date: Sat, 4 Mar 2023 12:48:35 +0900 Message-Id: <20230304034835.2082479-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230304034835.2082479-1-senozhatsky@chromium.org> References: <20230304034835.2082479-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8A72640012 X-Rspam-User: X-Stat-Signature: hi4dzpeq1h9rggr78j9ssmk99ba9i53u X-HE-Tag: 1677901732-443148 X-HE-Meta: U2FsdGVkX19jNpFrx2p3S3+mwP96FRrlaKVg3QSX/aaGGxxE+v7smv77pGYZUdFFJ1cP8qxKlUhD1bt/l3L1QknaYUjmHGcs7s6hYs+bEFSlFCBg2zzhYHjrKhLJDDexW6OqZbY5/a80pI/p/wDG+1hFChRUVkgAmJEhxzvcOLNXRLq8yYs8Ke4Qp/4ZTMcs9z2ajPdc93WrwosE9yYhLo2ntRmqJ3v9XzjnLw9TAvbFKRBXTPLmVt2zkz3bqARwLYWO0P9QnjFt/QlLCGiNp2F5+UWQlWlE9mvyFTHFRDCA0lMpW7/5laQnjQUMY5Mhb+KjvMEvru7gzuq0Jjwn8Gnn6djgwyt2s6oW4kzJ1PY+VxsPJJZ1yyF52a27/3TvypD3NhlyeVvyJ8QnL+LC6HFeyg7zyBIBpZREduu0ETyu5tIUf4+xicKMILUHRSzSuYx9HtwsvsOoupSzXH8hOvk8MEi99zw6bwC3vdvewh8UXY4cOhZP5GOYISwOpkojclUqXlmFjbS1ij5gkHewGIwWgOTKL8ga+zs2P0mq0ibr3hT5zMscD1J23xLwiDGFeEB8fgHEBELJRLl+RBcTd8+kuD2godUZGZ3jLDNye9rsVCPM4l0HLqVBn/jCHje9s1XowmyAlLympZ6ivMyeMRNC8Rxx2o4HqprAW5yMC2uCUPUoYl9Ivf9NgGYlGPlIvDEdy9T3rTn37gnOXBuYUHeNDBErhlFxoILYZAwNKyUeTRYFcz0lLpSpveczNFFE886EUKd1Y54eJ2+ZriSUwAJG1WaazJSKFtlfxEMhHBksM7kbFQyfv8Fe924Yo+CZXDXGF5lFuP7CA/G2IGfnM6u8kh/MN1UHwamMw+UeddZJXaSk3uvXBXydm/fnT+DSVj8iNmH94Yw9MSY9MzEcxVGrWAnrJCpYAwovBScRJvO4jBDMAlYtlC1L/7P1hQDy1Wq8sTEgdv11Cvz3XiN OoAMUDCI PPBxIsn1PXF9mInQigyK/fAnyAyT1pht52dy6YORxpTJGqV2YKVfp8oyFNL5gJkh0Wg/a6Om+cO5Ya7cjWpoG0+Yvg3Uia5JICYIpX+7D1DdXkDNtX4yGVmD7YTJWz9pcB6wdqTMdyvAIZq0k54W0YXaJoDUOOyr458NrmCdZvvcOsNxlM6e/1A9ZKkoyl3WWb5rQsvjz17uJf39iuOC0DBmdhVvf+0PKoJqGowuKUczkBwMDwejFzzUAvoO9Aa+TQEizSI9rcTxsJdjaBikoeVmWkAErnRFiapAIzycJb7LURHkqq2NJrQxfr3HA7P8k+fsdrP/cZ92g6Jd8wyber7+g7NwhReMvbCDQKZEAHSPu82+HP3+89Xuy61I71wJ/XQEF9e7XSdZSUffmKkdjyAGD4ALTQa2A8sIuFX55EH4fvl4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We keep the old fullness (3/4 threshold) reporting in zs_stats_size_show(). Switch from allmost full/empty stats to fine-grained per inuse ratio (fullness group) reporting, which gives signicantly more data on classes fragmentation. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 53 ++++++++++++++++++++++----------------------------- 1 file changed, 23 insertions(+), 30 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a61540afbb28..aea50e2aa350 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -172,9 +172,7 @@ enum fullness_group { ZS_INUSE_RATIO_0, ZS_INUSE_RATIO_10, - /* NOTE: 5 more fullness groups here */ - ZS_INUSE_RATIO_70 = 7, - /* NOTE: 2 more fullness groups here */ + /* NOTE: 8 more fullness groups here */ ZS_INUSE_RATIO_99 = 10, ZS_INUSE_RATIO_100, NR_FULLNESS_GROUPS, @@ -621,23 +619,22 @@ static unsigned long zs_can_compact(struct size_class *class); static int zs_stats_size_show(struct seq_file *s, void *v) { - int i; + int i, fg; struct zs_pool *pool = s->private; struct size_class *class; int objs_per_zspage; - unsigned long class_almost_full, class_almost_empty; unsigned long obj_allocated, obj_used, pages_used, freeable; - unsigned long total_class_almost_full = 0, total_class_almost_empty = 0; unsigned long total_objs = 0, total_used_objs = 0, total_pages = 0; unsigned long total_freeable = 0; + unsigned long inuse_totals[NR_FULLNESS_GROUPS] = {0, }; - seq_printf(s, " %5s %5s %11s %12s %13s %10s %10s %16s %8s\n", - "class", "size", "almost_full", "almost_empty", + seq_printf(s, " %5s %5s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %13s %10s %10s %16s %8s\n", + "class", "size", "10%", "20%", "30%", "40%", + "50%", "60%", "70%", "80%", "90%", "99%", "100%", "obj_allocated", "obj_used", "pages_used", "pages_per_zspage", "freeable"); for (i = 0; i < ZS_SIZE_CLASSES; i++) { - int fg; class = pool->size_class[i]; @@ -645,16 +642,12 @@ static int zs_stats_size_show(struct seq_file *s, void *v) continue; spin_lock(&pool->lock); - class_almost_full = 0; - class_almost_empty = 0; - /* - * Replicate old behaviour for almost_full and almost_empty - * stats. - */ - for (fg = ZS_INUSE_RATIO_70; fg <= ZS_INUSE_RATIO_99; fg++) - class_almost_full += zs_stat_get(class, fg); - for (fg = ZS_INUSE_RATIO_10; fg < ZS_INUSE_RATIO_70; fg++) - class_almost_empty += zs_stat_get(class, fg); + + seq_printf(s, " %5u %5u ", i, class->size); + for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { + inuse_totals[fg] += zs_stat_get(class, fg); + seq_printf(s, "%9lu ", zs_stat_get(class, fg)); + } obj_allocated = zs_stat_get(class, ZS_OBJS_ALLOCATED); obj_used = zs_stat_get(class, ZS_OBJS_INUSE); @@ -665,14 +658,10 @@ static int zs_stats_size_show(struct seq_file *s, void *v) pages_used = obj_allocated / objs_per_zspage * class->pages_per_zspage; - seq_printf(s, " %5u %5u %11lu %12lu %13lu" - " %10lu %10lu %16d %8lu\n", - i, class->size, class_almost_full, class_almost_empty, - obj_allocated, obj_used, pages_used, - class->pages_per_zspage, freeable); + seq_printf(s, "%13lu %10lu %10lu %16d %8lu\n", + obj_allocated, obj_used, pages_used, + class->pages_per_zspage, freeable); - total_class_almost_full += class_almost_full; - total_class_almost_empty += class_almost_empty; total_objs += obj_allocated; total_used_objs += obj_used; total_pages += pages_used; @@ -680,10 +669,14 @@ static int zs_stats_size_show(struct seq_file *s, void *v) } seq_puts(s, "\n"); - seq_printf(s, " %5s %5s %11lu %12lu %13lu %10lu %10lu %16s %8lu\n", - "Total", "", total_class_almost_full, - total_class_almost_empty, total_objs, - total_used_objs, total_pages, "", total_freeable); + seq_printf(s, " %5s %5s ", "Total", ""); + + for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) + seq_printf(s, "%9lu ", inuse_totals[fg]); + + seq_printf(s, "%13lu %10lu %10lu %16s %8lu\n", + total_objs, total_used_objs, total_pages, "", + total_freeable); return 0; }