From patchwork Mon Jun 12 09:38:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Domenico Cerasuolo X-Patchwork-Id: 13275952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F4F8C7EE2E for ; Mon, 12 Jun 2023 09:38:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9FEA8E0002; Mon, 12 Jun 2023 05:38:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4C9F6B0075; Mon, 12 Jun 2023 05:38:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B5758E0002; Mon, 12 Jun 2023 05:38:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6194A6B0074 for ; Mon, 12 Jun 2023 05:38:25 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2A7361C75A9 for ; Mon, 12 Jun 2023 09:38:25 +0000 (UTC) X-FDA: 80893595370.27.31D4917 Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by imf21.hostedemail.com (Postfix) with ESMTP id 2BC761C000C for ; Mon, 12 Jun 2023 09:38:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=KZfdk105; spf=pass (imf21.hostedemail.com: domain of cerasuolodomenico@gmail.com designates 209.85.218.53 as permitted sender) smtp.mailfrom=cerasuolodomenico@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686562703; a=rsa-sha256; cv=none; b=nLdrTCwaGlW0Msdw1IaX/XMCpwCYyuIXzpRxfSFu/CYQYnlmP54+p+zehQOoXVSsq+GJhI ptGINnkEyVnFUG/NfUj0y5l7xi2qqJvzzs77a4I6G1mjJjrpvo0xrft4WkxNujPxSvaIJl mj84nvItdomUohipFGh4aEPyVZ56bGs= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=KZfdk105; spf=pass (imf21.hostedemail.com: domain of cerasuolodomenico@gmail.com designates 209.85.218.53 as permitted sender) smtp.mailfrom=cerasuolodomenico@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686562703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ovwqjP88DLJU174wX93bEruUbTnFh4DciYUv65znDHc=; b=UW3t6cay1Uk4MX+HG7pi2f/EJ0x+i9uZZLLyRwqng5jgKec3wkAK9LUNzybH0W2nu++B8a /y+CfhoGuaIYKsRDrS0xv/HHFYEknE/sAFPvFrpxEHoYelooGIQJnPWOENdloqjrfapmbv mZl1ED0uHOxlfqqlknoATf2SDulVpqE= Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-97458c97333so695630366b.2 for ; Mon, 12 Jun 2023 02:38:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686562702; x=1689154702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ovwqjP88DLJU174wX93bEruUbTnFh4DciYUv65znDHc=; b=KZfdk105SVW1ZlUxEwaUHuC2cQAmWKzBmQ2NNpCN4F3/Qm43WAlHWWV3ZtTSbNAihm p/oy3xv28rDYE9OelY/atO/IhnMW0LnAxswzBt3BBluRaJPJhflX1fAuPHKNl4QabPnL /zX/+xDo7pqMY9INVkzGlS3GF/+a6aJ9/dcCU/2uKOsDojvqrzLi9W8lMq8ROjICCmAU IPQPF5kpHn+H1Jvnlh0lJK0QMPzPcLeSllzhYMeaSY5YYMp46GgDI65x5/ra8wpgkTNx J05FuZ2c3f4xqHcZrD/Xq1x9+4K5ss3hewqNuytZuxN8/2L2aXrnj6RQhSVphksTW9xp B1lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686562702; x=1689154702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ovwqjP88DLJU174wX93bEruUbTnFh4DciYUv65znDHc=; b=idZaPqwN9EKBy9BXe0QOkbCUJPKLDG6A2o3CpPTczLqBQr35q4j3iBLi2lWaUH9hzF i5+gIpU/BDioSdgNNWNKFvcarCaA/RtoQE4AOqcndVnET4cLqUb0l454d9I5Kl/qOshj 1LEHdokrWuN4g4M2QC0wfxzWpcA/AcfP/XQRLHDLgAhJ+J1KsXj3RZLNIKQmIi99Fxjz 9eJVqefFi4+I3JSY+GTsYl0Ktd1KDPQdp361EiCmf+fxFhuE+jTxm9j5GbJ2ClrN1Ftq 6NwiP+I4Scz2G1Y+dFzJg6ePeVedhswo+So05kUtmGYhg/SYT1+QHOvFVMrgH+649v45 m8ZA== X-Gm-Message-State: AC+VfDxAUj6/lpIjCYpPfjw0gDKNhN/qn6tGSuW/McunwVXWQhiUMkgu JwQmDmHpEUqvx9rFR2PLiHY= X-Google-Smtp-Source: ACHHUZ6od08fUS5Y5YBcxZ9WTfK/OI6dT7FI8Sb1oVaF31Hk1VrgKGdgSM0hpkvAGBFP0SFa0Z/byw== X-Received: by 2002:a17:907:1c93:b0:97d:f8aa:6d37 with SMTP id nb19-20020a1709071c9300b0097df8aa6d37mr8045773ejc.61.1686562701654; Mon, 12 Jun 2023 02:38:21 -0700 (PDT) Received: from lelloman-5950.homenet.telecomitalia.it (host-212-171-43-8.retail.telecomitalia.it. [212.171.43.8]) by smtp.gmail.com with ESMTPSA id ce23-20020a170906b25700b0097887b68c17sm4951358ejb.98.2023.06.12.02.38.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jun 2023 02:38:21 -0700 (PDT) From: Domenico Cerasuolo To: vitaly.wool@konsulko.com, minchan@kernel.org, senozhatsky@chromium.org, yosryahmed@google.com, linux-mm@kvack.org Cc: ddstreet@ieee.org, sjenning@redhat.com, nphamcs@gmail.com, hannes@cmpxchg.org, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Domenico Cerasuolo Subject: [PATCH v3 1/7] mm: zswap: add pool shrinking mechanism Date: Mon, 12 Jun 2023 11:38:09 +0200 Message-Id: <20230612093815.133504-2-cerasuolodomenico@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230612093815.133504-1-cerasuolodomenico@gmail.com> References: <20230612093815.133504-1-cerasuolodomenico@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2BC761C000C X-Stat-Signature: 5b11ofewubuf7b87edpmw3onkcqu6r1b X-Rspam-User: X-HE-Tag: 1686562702-647358 X-HE-Meta: U2FsdGVkX194OXPmsZKmCxNWmnWhPXB6NrEjFLveLe5sSiPgnxR4BvFkwdf+3PjpONeB2ibT06n7WdLf/EAk7IBh3DUxB/gvhs22VO0j4xIEZ8oa8/e5osg1PtDfqFj5KprPt9nmAKYcrrkv8OTuxoiKHqsLWYSO4LQyNlhf/8WGJxKOms0kbBUvG+3uzwP08ct5IgqF4qMRbQL6AZtIoTgcKQxEus9tf9FZv73Wri9I3beKNvCO+SqxxvUvdFmhUuXVj+sJAC78jV5WqdnI72J1AkQ+mz+15dkM58MBb1DPge45w+tKkjPazj5R6/D++uzqU04VMcmHYQ8mc3zAOofgbOENLRWaP2OM1aAEJ32KroVriYGrKrN1Pw3749yJPPvW8pdWgQZqqxZEvfAXKWmAmZ8/BJmjKiIBhjpbOOy6IT35XQJyzGQglETBrqQ5J/8pe2HCCtM21sRhpB2WEjNlSc0kHraVCAnwE9kmXGPwWAeAItY2jjr5aauxRbsgCtiP/zShgdq+jcLrWeTw3k2q/lEgmKo8enZFb0TtHRloxjo4JIBA29n/U+BOp3uzpZ55NCSY6Mrk1K8gEM/qLbGNi9iNABB0MLr7B9Nb13jzWMC9u1l7WLfF8fRDg7jcg6hAOO2fKEmyzmUHT47miSIw3rX5g/2s4SJqf/jERbKRpp8fWQgvsrjBba78oh4y47Awov76FFDkZ98hTd5r62JzqBlpabb1ndtUlGcKgOK+qFcFDE3sXEYPrN9HixR6fdACHBAnVTUSm38quE7264cLJIICd+e8cKuzoebAiy8YKYwTHRCBH6sYz4+g+q5yAanUfBYuSdFM7GlJtcf60MhyFIivuFb+GyfyiKInyq6ppzb2FYSDrID3BdfB6TwgXL0MfpQpQuZUY9/r2nfFS50rvAoLDhLcw6tpl3rJsUJEU1IfBhVhdHxus7tilO6sIw4tauXfIS91Mt2TK71 DH4dBqoj 1aDclD/UqKx6+GliGHSB26/D/mdA2YMMvkrBgSch56MxV9V88C0X5L50Qe3XMeKVg4wkhoEvFV8SqYf25tpiy2XNyI/Ldtl5kqfSqtwKbKAq/tfIb3UKk5XY+s/weiOuU+AGT9HXKLrtm4EQB1+mc0QquInax6xmytVv6tH39xx9qJHLSVl35KIK9NqWvLgv/ba2DIl6QhT6PbZ0B+kdumK2YgCWZ48jCtXJvVOiTlCJI2wIoAdlHWErKaHZLbhnLQ2jFQ7CHiLrqEnyYoRsDMRrvtby3UqyIUhBZg0P3papauGQEuAh/Zrr2e5Iy/OEkMrmJRYQsHZ6kgrq0FP2A3Q8gERjvVXmJO/9TXHILrewxk/J9vM6KAfDJXjVroeV7WtRiK+4v93Fmz4Rm2xMi6yKDO0Ap+Q46wRfa+C7y1bnBfa9Xlv5ctlrK+AhSqHIJswvCbI78SmS5mGHvGM1/dHPjPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Each zpool driver (zbud, z3fold and zsmalloc) implements its own shrink function, which is called from zpool_shrink. However, with this commit, a unified shrink function is added to zswap. The ultimate goal is to eliminate the need for zpool_shrink once all zpool implementations have dropped their shrink code. To ensure the functionality of each commit, this change focuses solely on adding the mechanism itself. No modifications are made to the backends, meaning that functionally, there are no immediate changes. The zswap mechanism will only come into effect once the backends have removed their shrink code. The subsequent commits will address the modifications needed in the backends. Acked-by: Nhat Pham Tested-by: Yosry Ahmed Signed-off-by: Domenico Cerasuolo Acked-by: Johannes Weiner Reviewed-by: Yosry Ahmed --- mm/zswap.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 92 insertions(+), 5 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 9fa86265f6dd..0024ec5ed574 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -154,6 +154,12 @@ struct crypto_acomp_ctx { struct mutex *mutex; }; +/* + * The lock ordering is zswap_tree.lock -> zswap_pool.lru_lock. + * The only case where lru_lock is not acquired while holding tree.lock is + * when a zswap_entry is taken off the lru for writeback, in that case it + * needs to be verified that it's still valid in the tree. + */ struct zswap_pool { struct zpool *zpool; struct crypto_acomp_ctx __percpu *acomp_ctx; @@ -163,6 +169,8 @@ struct zswap_pool { struct work_struct shrink_work; struct hlist_node node; char tfm_name[CRYPTO_MAX_ALG_NAME]; + struct list_head lru; + spinlock_t lru_lock; }; /* @@ -180,10 +188,12 @@ struct zswap_pool { * be held while changing the refcount. Since the lock must * be held, there is no reason to also make refcount atomic. * length - the length in bytes of the compressed page data. Needed during - * decompression. For a same value filled page length is 0. + * decompression. For a same value filled page length is 0, and both + * pool and lru are invalid and must be ignored. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data * value - value of the same-value filled pages which have same content + * lru - handle to the pool's lru used to evict pages. */ struct zswap_entry { struct rb_node rbnode; @@ -196,6 +206,7 @@ struct zswap_entry { unsigned long value; }; struct obj_cgroup *objcg; + struct list_head lru; }; struct zswap_header { @@ -368,6 +379,12 @@ static void zswap_free_entry(struct zswap_entry *entry) if (!entry->length) atomic_dec(&zswap_same_filled_pages); else { + /* zpool_evictable will be removed once all 3 backends have migrated */ + if (!zpool_evictable(entry->pool->zpool)) { + spin_lock(&entry->pool->lru_lock); + list_del(&entry->lru); + spin_unlock(&entry->pool->lru_lock); + } zpool_free(entry->pool->zpool, entry->handle); zswap_pool_put(entry->pool); } @@ -588,14 +605,72 @@ static struct zswap_pool *zswap_pool_find_get(char *type, char *compressor) return NULL; } +static int zswap_reclaim_entry(struct zswap_pool *pool) +{ + struct zswap_header *zhdr; + struct zswap_entry *entry; + struct zswap_tree *tree; + pgoff_t swpoffset; + int ret; + + /* Get an entry off the LRU */ + spin_lock(&pool->lru_lock); + if (list_empty(&pool->lru)) { + spin_unlock(&pool->lru_lock); + return -EINVAL; + } + entry = list_last_entry(&pool->lru, struct zswap_entry, lru); + list_del_init(&entry->lru); + zhdr = zpool_map_handle(pool->zpool, entry->handle, ZPOOL_MM_RO); + tree = zswap_trees[swp_type(zhdr->swpentry)]; + zpool_unmap_handle(pool->zpool, entry->handle); + /* + * Once the lru lock is dropped, the entry might get freed. The + * swpoffset is copied to the stack, and entry isn't deref'd again + * until the entry is verified to still be alive in the tree. + */ + swpoffset = swp_offset(zhdr->swpentry); + spin_unlock(&pool->lru_lock); + + /* Check for invalidate() race */ + spin_lock(&tree->lock); + if (entry != zswap_rb_search(&tree->rbroot, swpoffset)) { + ret = -EAGAIN; + goto unlock; + } + /* Hold a reference to prevent a free during writeback */ + zswap_entry_get(entry); + spin_unlock(&tree->lock); + + ret = zswap_writeback_entry(pool->zpool, entry->handle); + + spin_lock(&tree->lock); + if (ret) { + /* Writeback failed, put entry back on LRU */ + spin_lock(&pool->lru_lock); + list_move(&entry->lru, &pool->lru); + spin_unlock(&pool->lru_lock); + } + + /* Drop local reference */ + zswap_entry_put(tree, entry); +unlock: + spin_unlock(&tree->lock); + return ret ? -EAGAIN : 0; +} + static void shrink_worker(struct work_struct *w) { struct zswap_pool *pool = container_of(w, typeof(*pool), shrink_work); int ret, failures = 0; + /* zpool_evictable will be removed once all 3 backends have migrated */ do { - ret = zpool_shrink(pool->zpool, 1, NULL); + if (zpool_evictable(pool->zpool)) + ret = zpool_shrink(pool->zpool, 1, NULL); + else + ret = zswap_reclaim_entry(pool); if (ret) { zswap_reject_reclaim_fail++; if (ret != -EAGAIN) @@ -659,6 +734,8 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) */ kref_init(&pool->kref); INIT_LIST_HEAD(&pool->list); + INIT_LIST_HEAD(&pool->lru); + spin_lock_init(&pool->lru_lock); INIT_WORK(&pool->shrink_work, shrink_worker); zswap_pool_debug("created", pool); @@ -1274,7 +1351,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, } /* store */ - hlen = zpool_evictable(entry->pool->zpool) ? sizeof(zhdr) : 0; + hlen = sizeof(zhdr); gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; if (zpool_malloc_support_movable(entry->pool->zpool)) gfp |= __GFP_HIGHMEM | __GFP_MOVABLE; @@ -1317,6 +1394,12 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, zswap_entry_put(tree, dupentry); } } while (ret == -EEXIST); + /* zpool_evictable will be removed once all 3 backends have migrated */ + if (entry->length && !zpool_evictable(entry->pool->zpool)) { + spin_lock(&entry->pool->lru_lock); + list_add(&entry->lru, &entry->pool->lru); + spin_unlock(&entry->pool->lru_lock); + } spin_unlock(&tree->lock); /* update stats */ @@ -1398,8 +1481,7 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, /* decompress */ dlen = PAGE_SIZE; src = zpool_map_handle(entry->pool->zpool, entry->handle, ZPOOL_MM_RO); - if (zpool_evictable(entry->pool->zpool)) - src += sizeof(struct zswap_header); + src += sizeof(struct zswap_header); if (!zpool_can_sleep_mapped(entry->pool->zpool)) { memcpy(tmp, src, entry->length); @@ -1432,6 +1514,11 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, if (!ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); *exclusive = true; + } else if (entry->length && !zpool_evictable(entry->pool->zpool)) { + /* zpool_evictable will be removed once all 3 backends have migrated */ + spin_lock(&entry->pool->lru_lock); + list_move(&entry->lru, &entry->pool->lru); + spin_unlock(&entry->pool->lru_lock); } spin_unlock(&tree->lock);