From patchwork Tue Oct 24 20:32:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13435242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61E76C25B6C for ; Tue, 24 Oct 2023 20:33:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD1E36B02E9; Tue, 24 Oct 2023 16:33:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D816C6B02EA; Tue, 24 Oct 2023 16:33:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5FEA6B02EB; Tue, 24 Oct 2023 16:33:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A49B86B02E9 for ; Tue, 24 Oct 2023 16:33:08 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7AF6A120B07 for ; Tue, 24 Oct 2023 20:33:08 +0000 (UTC) X-FDA: 81381504456.02.651270E Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf26.hostedemail.com (Postfix) with ESMTP id 66CBF140016 for ; Tue, 24 Oct 2023 20:33:06 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="c/y2qnKD"; spf=pass (imf26.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698179586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5pFcPVumquKH2Gwm0GjVQh+sz7Zkp1krGgtTHD7g2j8=; b=y8D0yo7ew4346IA8d+jKmsdEF/lfSN7UnSdFSl1NKFbabYSEZ9A8eu4zy9Is+PW18Huewt 8yPg5ojgv8FgscIzeD/TNMRqtfvzdsygezDMBAw7drOIPgP1poBHYTLCSxmNmEa5CrxXtt Z3pRjP8rLLEvAQmBwF3PpJfwhe+Mn0k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698179586; a=rsa-sha256; cv=none; b=i9VGyjbY9AYrq//u1n9mJKah4p1JIeKq4ERWf1or0ZbVYt4Jufy8FM97gPe0z4j5qa9tmE uK8htk98uwviMLgTfWFNQ5hdpDxHymR5gWVXxg1+XCrbUhyzUsRi7/1O5f438Oi0hz5RZn xpwKP0uioeT3XnlpjgWX/ibuVwzMpp0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="c/y2qnKD"; spf=pass (imf26.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1bf55a81eeaso34516095ad.0 for ; Tue, 24 Oct 2023 13:33:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698179585; x=1698784385; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5pFcPVumquKH2Gwm0GjVQh+sz7Zkp1krGgtTHD7g2j8=; b=c/y2qnKDTfM8ybFVsiiA/kW4wpnNBkpdGmWpgjVK2Mb5WZQ92tvy5GttftHVkcAqyE A5Noq9W0XQSZMM15cUN5O4O9ve/oy/HcMr+3+thcvOchip0Jp+EB7pm6v6PeQVsfXf6h VrUhCz+5+T7GfmTJ5wNzM4XBLdDEv4vpxi6yZvekJZxjxNxRVPJ5iy2e7q/3Vc7jm9Q8 ymYjAS3+XDbIX1z76u9/k4mq8uLuB7ejpLvZAoa0jvxEqGK/NdYCvtx8qCZYzXwbre48 +nW1k4cJQMYDQRPUKupPeYtkwF+mOO2q0nMXnp/3I05nQzpRZDEWiZvfNjBBgGVw1AdX Uw1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698179585; x=1698784385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5pFcPVumquKH2Gwm0GjVQh+sz7Zkp1krGgtTHD7g2j8=; b=ogxMUwUdvYiQ0dBlfUL/Bq2wFGSa5rR8sTcTRFwxJnNi2yy9oZnOeWqOXoX8cNaOg5 LjvEQ+eKOIvaDE98QONmXzR8fSz63AKqJQd1qHKH1EdFFGynHJSXYmzyuIKBuIvg9eXg 6B3XrwpEqaZKAY1eyY8G5+045tIP0D47Im9au1PORucAnyKKSosByKuVitPNjQYkIh70 XX/gSa2YR5NEy0lL+6jEg/9TOPmnPNXbdgSf8+dHvVPgzKAn/36o/Pe9a9+bvmKhVuG7 h8XN4gdd/ldFqOn0y7I1zEQ24UJf5hcnrjvV9cKOCKWlxdCsZBSZejhEJRhicH0E0v+P PPbA== X-Gm-Message-State: AOJu0YzdiNaTN2Xik9LqYnsKiO6r5uLMRnny4hnaSnUt02L2o81FaFdj X2EbmzwjaL8fMIRpbVBny6U= X-Google-Smtp-Source: AGHT+IGtiSfm4fnGqjTyjPWhsh4m1mtNrKKsbC5+bKMi27RKG/Yi1VW+KSPVGmr49Nj1j4wano8fFQ== X-Received: by 2002:a17:902:ea09:b0:1c5:de06:9e5a with SMTP id s9-20020a170902ea0900b001c5de069e5amr10697237plg.21.1698179585101; Tue, 24 Oct 2023 13:33:05 -0700 (PDT) Received: from localhost (fwdproxy-prn-005.fbsv.net. [2a03:2880:ff:5::face:b00c]) by smtp.gmail.com with ESMTPSA id q13-20020a17090311cd00b001c9b2e66795sm7834751plh.85.2023.10.24.13.33.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 13:33:04 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, shuah@kernel.org Subject: [PATCH v4 1/5] list_lru: allows explicit memcg and NUMA node selection Date: Tue, 24 Oct 2023 13:32:58 -0700 Message-Id: <20231024203302.1920362-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231024203302.1920362-1-nphamcs@gmail.com> References: <20231024203302.1920362-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 66CBF140016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: aed1rkp3mpqt1mndambzc1xtdon5wstk X-HE-Tag: 1698179586-479361 X-HE-Meta: U2FsdGVkX19D7PrtCUtdlMOYxW1UmAi8QinoRSKAhhRo/sfgMPFBUTzPynkynNKTe7DGpdsv3QrGO8XctYqYoCgj4+4hd3d2ZEmsS8PtUbNrMAJaLM+TIkT30ILwc9bnCmoHZlrJw6SsQs8cRx7/xetPhDkU3ITDDCTYIaxg8/kg+H/PhgVZUSMyv29U1YuChVYH3MvYa+cF1UHCrhudYFQTOym7NpDKevrMT2nITKT4WNWNvpmVK08Yv1OEurR6dvv+R884PehhHYl4DDYRTdO1JlT8c3wTW+L1/COUoOU3OD7zDZV+OTredd6zRsF+qsWW0t7TJ6r/qr7V2YnYzeDJSi0AJS77DB6awupoE4VmxCpq3yqGfv04GYLFgad+Fek9eRqXR5WabwEfJHUpPS7Wz4KTcpLhik2gE4XFtJtULqctiY7dz0is7MmEVmy6cjz8wCoue+fMp7V2oxq56/bg5G5+xYmzmLGUZPlal618ACC9hRAqxl/56HATHgYdFdbJJpV7r/haFRo6V816wLfdHf3cqEO/Rk+7QpytSCUPOTcZ1SltdhLnxKUvx7l4ntAec9UmCh+WngYrxv/IkHL5ddR12Awf5IwOJg+e7v7BD2k+2hmboqb69NBvuWQeAN1Wlr03YjdJGMvA7AXETnR9DUO8ah5Ajm389nUFdgn77oqExpYiDPDVfB7e2+zVWQtNfnPu5duIQZU67iEVdsMF7F15gF2VHCSE3MKu1QoE4T3uEXj2ZMc/UOi/Bu/aGmbsNJ4u1GoDdZIC8lTCK/dd2CYMMxyp+JdgIrF46ucfzHIQRfzQYuMhOT4bUGsS4I7PNPUdmi+EMATis/A5u1uT7noFwRtKl7zf3knmIEvsMTPZ0dLrJjN8n+VcD+8CQvQMowF7X/24kZ0sSkV962mDMwOc1MLocZlvvxeIOtW3hpviJVsq+ZezoKNmpSH7e7KE5qD58nQ1JMXAd4K 8tdlxps1 e6STx2JP7VtWscPOtr+1CoY7U4/YgBUhwhuMjTuFf1IkG8aQsMiSKZsrnhnq0B2vhO4nRbQuNErP6MkFRalrr5aCuuxpvKgMRwMZJHgws+xulu4YRdOYPBzDfMrGPD4A7Zs/qMJiZQYq9hQMWolZmhTBNDag4Q9PlJDv7GGAQnQd6DbH6JZtvv7BFKSF4Z5MqPhkv86EWGPYYycC2Ck4pn8qAujZ265JyFXLucEAFBItn92Ohlo0SU6BZgJEfqBMmZlqtTBHVsqsuMMhkUOm+BLia0WPn7rDdYj+wkqKAkPwfGadpToIr8JYlzgJ+5AmavBCjivLMXLJBIVp5GY/4vwELVn+6Uhyd/SlFrvpBCd1ihPVDQko8NKYCmxcaW7REqTSwjnaXfdUZVPrpFPBcCdd+dz6jMeZgcEkoEnZsPZfbYXoc7zXcwwV7q/KiW0o0pYnEA0HL/Z7dIi6OqT77WGvm9Y5KCPfc4BV9Jro9w1pt0+6A/YVVnFuG0ey3oX7UftyUlLuEn8oeXyb+oynbhtyKlMg+hzVdRoLg7juzB/Lly4gaJboJcXYxv9TyM50pvWLagpTAYmEJy2M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The interface of list_lru is based on the assumption that the list node and the data it represents belong to the same allocated on the correct node/memcg. While this assumption is valid for existing slab objects LRU such as dentries and inodes, it is undocumented, and rather inflexible for certain potential list_lru users (such as the upcoming zswap shrinker and the THP shrinker). It has caused us a lot of issues during our development. This patch changes list_lru interface so that the caller must explicitly specify numa node and memcg when adding and removing objects. The old list_lru_add() and list_lru_del() are renamed to list_lru_add_obj() and list_lru_del_obj(), respectively. It also extends the list_lru API with a new function, list_lru_putback, which undoes a previous list_lru_isolate call. Unlike list_lru_add, it does not increment the LRU node count (as list_lru_isolate does not decrement the node count). list_lru_putback also allows for explicit memcg and NUMA node selection. Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham --- drivers/android/binder_alloc.c | 5 ++-- fs/dcache.c | 8 +++--- fs/gfs2/quota.c | 6 ++--- fs/inode.c | 4 +-- fs/nfs/nfs42xattr.c | 8 +++--- fs/nfsd/filecache.c | 4 +-- fs/xfs/xfs_buf.c | 6 ++--- fs/xfs/xfs_dquot.c | 2 +- fs/xfs/xfs_qm.c | 2 +- include/linux/list_lru.h | 46 +++++++++++++++++++++++++++++--- mm/list_lru.c | 48 ++++++++++++++++++++++++++++------ mm/workingset.c | 4 +-- 12 files changed, 108 insertions(+), 35 deletions(-) diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 138f6d43d13b..e80669d4e037 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -285,7 +285,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, trace_binder_free_lru_start(alloc, index); - ret = list_lru_add(&binder_alloc_lru, &page->lru); + ret = list_lru_add_obj(&binder_alloc_lru, &page->lru); WARN_ON(!ret); trace_binder_free_lru_end(alloc, index); @@ -848,7 +848,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) if (!alloc->pages[i].page_ptr) continue; - on_lru = list_lru_del(&binder_alloc_lru, + on_lru = list_lru_del_obj(&binder_alloc_lru, &alloc->pages[i].lru); page_addr = alloc->buffer + i * PAGE_SIZE; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, @@ -1287,4 +1287,3 @@ int binder_alloc_copy_from_buffer(struct binder_alloc *alloc, return binder_alloc_do_buffer_copy(alloc, false, buffer, buffer_offset, dest, bytes); } - diff --git a/fs/dcache.c b/fs/dcache.c index 25ac74d30bff..482d1b34d88d 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -428,7 +428,8 @@ static void d_lru_add(struct dentry *dentry) this_cpu_inc(nr_dentry_unused); if (d_is_negative(dentry)) this_cpu_inc(nr_dentry_negative); - WARN_ON_ONCE(!list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); + WARN_ON_ONCE(!list_lru_add_obj( + &dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } static void d_lru_del(struct dentry *dentry) @@ -438,7 +439,8 @@ static void d_lru_del(struct dentry *dentry) this_cpu_dec(nr_dentry_unused); if (d_is_negative(dentry)) this_cpu_dec(nr_dentry_negative); - WARN_ON_ONCE(!list_lru_del(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); + WARN_ON_ONCE(!list_lru_del_obj( + &dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } static void d_shrink_del(struct dentry *dentry) @@ -1240,7 +1242,7 @@ static enum lru_status dentry_lru_isolate(struct list_head *item, * * This is guaranteed by the fact that all LRU management * functions are intermediated by the LRU API calls like - * list_lru_add and list_lru_del. List movement in this file + * list_lru_add_obj and list_lru_del_obj. List movement in this file * only ever occur through this functions or through callbacks * like this one, that are called from the LRU API. * diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index 2f1328af34f4..72015594bc83 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -271,7 +271,7 @@ static struct gfs2_quota_data *gfs2_qd_search_bucket(unsigned int hash, if (qd->qd_sbd != sdp) continue; if (lockref_get_not_dead(&qd->qd_lockref)) { - list_lru_del(&gfs2_qd_lru, &qd->qd_lru); + list_lru_del_obj(&gfs2_qd_lru, &qd->qd_lru); return qd; } } @@ -344,7 +344,7 @@ static void qd_put(struct gfs2_quota_data *qd) } qd->qd_lockref.count = 0; - list_lru_add(&gfs2_qd_lru, &qd->qd_lru); + list_lru_add_obj(&gfs2_qd_lru, &qd->qd_lru); spin_unlock(&qd->qd_lockref.lock); } @@ -1508,7 +1508,7 @@ void gfs2_quota_cleanup(struct gfs2_sbd *sdp) lockref_mark_dead(&qd->qd_lockref); spin_unlock(&qd->qd_lockref.lock); - list_lru_del(&gfs2_qd_lru, &qd->qd_lru); + list_lru_del_obj(&gfs2_qd_lru, &qd->qd_lru); list_add(&qd->qd_lru, &dispose); } spin_unlock(&qd_lock); diff --git a/fs/inode.c b/fs/inode.c index 84bc3c76e5cc..f889ba8dccd9 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -462,7 +462,7 @@ static void __inode_add_lru(struct inode *inode, bool rotate) if (!mapping_shrinkable(&inode->i_data)) return; - if (list_lru_add(&inode->i_sb->s_inode_lru, &inode->i_lru)) + if (list_lru_add_obj(&inode->i_sb->s_inode_lru, &inode->i_lru)) this_cpu_inc(nr_unused); else if (rotate) inode->i_state |= I_REFERENCED; @@ -480,7 +480,7 @@ void inode_add_lru(struct inode *inode) static void inode_lru_list_del(struct inode *inode) { - if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru)) + if (list_lru_del_obj(&inode->i_sb->s_inode_lru, &inode->i_lru)) this_cpu_dec(nr_unused); } diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index 2ad66a8922f4..49aaf28a6950 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -132,7 +132,7 @@ nfs4_xattr_entry_lru_add(struct nfs4_xattr_entry *entry) lru = (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) ? &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; - return list_lru_add(lru, &entry->lru); + return list_lru_add_obj(lru, &entry->lru); } static bool @@ -143,7 +143,7 @@ nfs4_xattr_entry_lru_del(struct nfs4_xattr_entry *entry) lru = (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) ? &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; - return list_lru_del(lru, &entry->lru); + return list_lru_del_obj(lru, &entry->lru); } /* @@ -349,7 +349,7 @@ nfs4_xattr_cache_unlink(struct inode *inode) oldcache = nfsi->xattr_cache; if (oldcache != NULL) { - list_lru_del(&nfs4_xattr_cache_lru, &oldcache->lru); + list_lru_del_obj(&nfs4_xattr_cache_lru, &oldcache->lru); oldcache->inode = NULL; } nfsi->xattr_cache = NULL; @@ -474,7 +474,7 @@ nfs4_xattr_get_cache(struct inode *inode, int add) kref_get(&cache->ref); nfsi->xattr_cache = cache; cache->inode = inode; - list_lru_add(&nfs4_xattr_cache_lru, &cache->lru); + list_lru_add_obj(&nfs4_xattr_cache_lru, &cache->lru); } spin_unlock(&inode->i_lock); diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 9c62b4502539..82352c100b49 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -322,7 +322,7 @@ nfsd_file_check_writeback(struct nfsd_file *nf) static bool nfsd_file_lru_add(struct nfsd_file *nf) { set_bit(NFSD_FILE_REFERENCED, &nf->nf_flags); - if (list_lru_add(&nfsd_file_lru, &nf->nf_lru)) { + if (list_lru_add_obj(&nfsd_file_lru, &nf->nf_lru)) { trace_nfsd_file_lru_add(nf); return true; } @@ -331,7 +331,7 @@ static bool nfsd_file_lru_add(struct nfsd_file *nf) static bool nfsd_file_lru_remove(struct nfsd_file *nf) { - if (list_lru_del(&nfsd_file_lru, &nf->nf_lru)) { + if (list_lru_del_obj(&nfsd_file_lru, &nf->nf_lru)) { trace_nfsd_file_lru_del(nf); return true; } diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 9e7ba04572db..9c2654a8d24b 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -169,7 +169,7 @@ xfs_buf_stale( atomic_set(&bp->b_lru_ref, 0); if (!(bp->b_state & XFS_BSTATE_DISPOSE) && - (list_lru_del(&bp->b_target->bt_lru, &bp->b_lru))) + (list_lru_del_obj(&bp->b_target->bt_lru, &bp->b_lru))) atomic_dec(&bp->b_hold); ASSERT(atomic_read(&bp->b_hold) >= 1); @@ -1047,7 +1047,7 @@ xfs_buf_rele( * buffer for the LRU and clear the (now stale) dispose list * state flag */ - if (list_lru_add(&bp->b_target->bt_lru, &bp->b_lru)) { + if (list_lru_add_obj(&bp->b_target->bt_lru, &bp->b_lru)) { bp->b_state &= ~XFS_BSTATE_DISPOSE; atomic_inc(&bp->b_hold); } @@ -1060,7 +1060,7 @@ xfs_buf_rele( * was on was the disposal list */ if (!(bp->b_state & XFS_BSTATE_DISPOSE)) { - list_lru_del(&bp->b_target->bt_lru, &bp->b_lru); + list_lru_del_obj(&bp->b_target->bt_lru, &bp->b_lru); } else { ASSERT(list_empty(&bp->b_lru)); } diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index ac6ba646624d..49f619f5aa96 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -1064,7 +1064,7 @@ xfs_qm_dqput( struct xfs_quotainfo *qi = dqp->q_mount->m_quotainfo; trace_xfs_dqput_free(dqp); - if (list_lru_add(&qi->qi_lru, &dqp->q_lru)) + if (list_lru_add_obj(&qi->qi_lru, &dqp->q_lru)) XFS_STATS_INC(dqp->q_mount, xs_qm_dquot_unused); } xfs_dqunlock(dqp); diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 94a7932ac570..67d0a8564ff3 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -171,7 +171,7 @@ xfs_qm_dqpurge( * hits zero, so it really should be on the freelist here. */ ASSERT(!list_empty(&dqp->q_lru)); - list_lru_del(&qi->qi_lru, &dqp->q_lru); + list_lru_del_obj(&qi->qi_lru, &dqp->q_lru); XFS_STATS_DEC(dqp->q_mount, xs_qm_dquot_unused); xfs_qm_dqdestroy(dqp); diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index b35968ee9fb5..5ef217443299 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -75,6 +75,8 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren * list_lru_add: add an element to the lru list's tail * @list_lru: the lru pointer * @item: the item to be added. + * @memcg: the cgroup of the sublist to add the item to. + * @nid: the node id of the sublist to add the item to. * * If the element is already part of a list, this function returns doing * nothing. Therefore the caller does not need to keep state about whether or @@ -87,12 +89,28 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren * * Return value: true if the list was updated, false otherwise */ -bool list_lru_add(struct list_lru *lru, struct list_head *item); +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); /** - * list_lru_del: delete an element to the lru list + * list_lru_add_obj: add an element to the lru list's tail + * @list_lru: the lru pointer + * @item: the item to be added. + * + * This function is similar to list_lru_add(), but the NUMA node and the + * memcg of the sublist is determined by @item list_head. This assumption is + * valid for slab objects LRU such as dentries, inodes, etc. + * + * Return value: true if the list was updated, false otherwise + */ +bool list_lru_add_obj(struct list_lru *lru, struct list_head *item); + +/** + * list_lru_del: delete an element from the lru list * @list_lru: the lru pointer * @item: the item to be deleted. + * @memcg: the cgroup of the sublist to delete the item from. + * @nid: the node id of the sublist to delete the item from. * * This function works analogously as list_lru_add in terms of list * manipulation. The comments about an element already pertaining to @@ -100,7 +118,21 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item); * * Return value: true if the list was updated, false otherwise */ -bool list_lru_del(struct list_lru *lru, struct list_head *item); +bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_del_obj: delete an element from the lru list + * @list_lru: the lru pointer + * @item: the item to be deleted. + * + * This function is similar to list_lru_del(), but the NUMA node and the + * memcg of the sublist is determined by @item list_head. This assumption is + * valid for slab objects LRU such as dentries, inodes, etc. + * + * Return value: true if the list was updated, false otherwise. + */ +bool list_lru_del_obj(struct list_lru *lru, struct list_head *item); /** * list_lru_count_one: return the number of objects currently held by @lru @@ -136,6 +168,14 @@ static inline unsigned long list_lru_count(struct list_lru *lru) void list_lru_isolate(struct list_lru_one *list, struct list_head *item); void list_lru_isolate_move(struct list_lru_one *list, struct list_head *item, struct list_head *head); +/* + * list_lru_putback: undo list_lru_isolate. + * + * Since we might have dropped the LRU lock in between, recompute list_lru_one + * from the node's id and memcg. + */ +void list_lru_putback(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); typedef enum lru_status (*list_lru_walk_cb)(struct list_head *item, struct list_lru_one *list, spinlock_t *lock, void *cb_arg); diff --git a/mm/list_lru.c b/mm/list_lru.c index a05e5bef3b40..fcca67ac26ec 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -116,21 +116,19 @@ list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, } #endif /* CONFIG_MEMCG_KMEM */ -bool list_lru_add(struct list_lru *lru, struct list_head *item) +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) { - int nid = page_to_nid(virt_to_page(item)); struct list_lru_node *nlru = &lru->node[nid]; - struct mem_cgroup *memcg; struct list_lru_one *l; spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_kmem(lru, nid, item, &memcg); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -140,15 +138,25 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) } EXPORT_SYMBOL_GPL(list_lru_add); -bool list_lru_del(struct list_lru *lru, struct list_head *item) +bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { int nid = page_to_nid(virt_to_page(item)); + struct mem_cgroup *memcg = list_lru_memcg_aware(lru) ? + mem_cgroup_from_slab_obj(item) : NULL; + + return list_lru_add(lru, item, nid, memcg); +} +EXPORT_SYMBOL_GPL(list_lru_add_obj); + +bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_kmem(lru, nid, item, NULL); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -160,6 +168,16 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) } EXPORT_SYMBOL_GPL(list_lru_del); +bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) +{ + int nid = page_to_nid(virt_to_page(item)); + struct mem_cgroup *memcg = list_lru_memcg_aware(lru) ? + mem_cgroup_from_slab_obj(item) : NULL; + + return list_lru_del(lru, item, nid, memcg); +} +EXPORT_SYMBOL_GPL(list_lru_del_obj); + void list_lru_isolate(struct list_lru_one *list, struct list_head *item) { list_del_init(item); @@ -175,6 +193,20 @@ void list_lru_isolate_move(struct list_lru_one *list, struct list_head *item, } EXPORT_SYMBOL_GPL(list_lru_isolate_move); +void list_lru_putback(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ + struct list_lru_one *list = + list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + + if (list_empty(item)) { + list_add_tail(item, &list->list); + if (!list->nr_items++) + set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); + } +} +EXPORT_SYMBOL_GPL(list_lru_putback); + unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { diff --git a/mm/workingset.c b/mm/workingset.c index 11045febc383..7d3dacab8451 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -631,12 +631,12 @@ void workingset_update_node(struct xa_node *node) if (node->count && node->count == node->nr_values) { if (list_empty(&node->private_list)) { - list_lru_add(&shadow_nodes, &node->private_list); + list_lru_add_obj(&shadow_nodes, &node->private_list); __inc_lruvec_kmem_state(node, WORKINGSET_NODES); } } else { if (!list_empty(&node->private_list)) { - list_lru_del(&shadow_nodes, &node->private_list); + list_lru_del_obj(&shadow_nodes, &node->private_list); __dec_lruvec_kmem_state(node, WORKINGSET_NODES); } } From patchwork Tue Oct 24 20:32:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13435243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 875A8C25B6F for ; Tue, 24 Oct 2023 20:33:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3726F6B02EB; Tue, 24 Oct 2023 16:33:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FA596B02ED; Tue, 24 Oct 2023 16:33:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D72E6B02EC; Tue, 24 Oct 2023 16:33:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F20336B02EA for ; Tue, 24 Oct 2023 16:33:09 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C32E9C09E9 for ; Tue, 24 Oct 2023 20:33:09 +0000 (UTC) X-FDA: 81381504498.19.016364F Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf30.hostedemail.com (Postfix) with ESMTP id DCF9480005 for ; Tue, 24 Oct 2023 20:33:07 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e4g9PPJi; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698179588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EjKUlFpL39TSfsl04qi0nTIpn8KUue1UgxxO5OJDRwY=; b=reL3OSMYP7cjx5YQvz13AIaULYioav1uKVi23BKAK2K+o+pORitRsU2q0c/ykS1kt/ejU8 J6TMYAzM3VVfCaYAKlCtZ4vWVlxg7AOHjvJSDW1cDCnhf9A3/dSokRQk7oVzA48LTDgVuy Qyc1oq+Iy1tngzm9srLerXTpGzkjRT8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698179588; a=rsa-sha256; cv=none; b=VOKja7zfw116Jc7VcHz0PNd9wkUmA/n8GFP6kReh+C0IXljxM94i9TBQ2apLIbpumsOdKF GrfzmKi0cSw7wjQ/ztvkK0ZUJU9R5vgjaSLHR8/p9vZyV3GpzVy3UGjKk4NpmsudJfBBLC iFDzAGsoZ5D2tpXBy5fQzxL/VaBjzGQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e4g9PPJi; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1c8a1541232so42235145ad.0 for ; Tue, 24 Oct 2023 13:33:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698179586; x=1698784386; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EjKUlFpL39TSfsl04qi0nTIpn8KUue1UgxxO5OJDRwY=; b=e4g9PPJieFU5HORlfQAsTjNhyHgOzqhIxfxazKlxZ2vs8wbcfpR/PFkfXmp3AiAVJx Uoe1I/8d1Gk9NyRy3/hB4A8zu/XoB2RadYGFbM7W4furiNsqdpFY0iDhcd8ykdcupkZp I4E7DlUQrrYdAk98NX8WYdD1fG4hKab4QGNgfxbvf6OoARB0+tyX4+X6y5RvJA9tNqKj EMy5c/uVLJzefCNpH+/a52ud/JYHYooV/mpil4q1BjBQ/07IMqQMu8rpamqlWazJgHvb 6lEyyZosPwOydYsIBZxZfPvDvEzWFTsuYXN0lMtDYRhl2AWab3tMRjKSPagoZfub8V9t AqoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698179586; x=1698784386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EjKUlFpL39TSfsl04qi0nTIpn8KUue1UgxxO5OJDRwY=; b=PdzM5Wonf5bXB5p7Jj10MDQuBlypywzasbaR9pUyuNj1If2PfuLkBSu13zdy/tIhX+ XR32C2ZdC2cixMWnuXgEz7p44tXN/sCtfz0FJO47rMspra1n7eIzOSg8BdR4U7ts07BH 0N/CtrGYdtjq0oF13rKtGYgs+41BQchV1ae5Cdvnme7wBQAdI39l7eJK3ICU+JQfvl1H +87JmAzByyW3DaExI5dgp+Gokc1NyS110qX657Lhn/YUOv3lzFa3PjcI+F2kDK5T/anE TXuex4sj/GDmyW47JNLBavk/ZF7BwmpATxOeeXZDQbNcDacpZMN0K3jD97P545Mf4vZ4 PPHA== X-Gm-Message-State: AOJu0YyC9BXFi4H6rEf9+EPn7BqcXaH74FC4362k4686Sh0hFhciFWwE /d4f/PM04YmF8pVn1sucDcg= X-Google-Smtp-Source: AGHT+IHRRe+HuzcQ5jAQnfmTr+/4TdrSZsTeScvqfN8rvNtcoNMqYlI9CIXuSaCE2SnrDuBk1pVaag== X-Received: by 2002:a17:902:c404:b0:1c3:ed30:ce0a with SMTP id k4-20020a170902c40400b001c3ed30ce0amr17552867plk.19.1698179586180; Tue, 24 Oct 2023 13:33:06 -0700 (PDT) Received: from localhost (fwdproxy-prn-007.fbsv.net. [2a03:2880:ff:7::face:b00c]) by smtp.gmail.com with ESMTPSA id 4-20020a170902ee4400b001c877f27d1fsm7837632plo.11.2023.10.24.13.33.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 13:33:05 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, shuah@kernel.org Subject: [PATCH v4 2/5] zswap: make shrinking memcg-aware Date: Tue, 24 Oct 2023 13:32:59 -0700 Message-Id: <20231024203302.1920362-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231024203302.1920362-1-nphamcs@gmail.com> References: <20231024203302.1920362-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 5semhz89ece63fosqsazpaocrqegnayg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DCF9480005 X-Rspam-User: X-HE-Tag: 1698179587-428149 X-HE-Meta: U2FsdGVkX1+t77HeoLk5MNV8H5VeCEqVVWRYXA3O1K/rGwGaEjolw12aph9QHw6N4VZG1jMiTivON3HH7UUQjMUwi6zZzjGNGOeJOJVTsdpYkeX8pbdjqwWPYoDZIAxkoIdCCD9I2x/ap0Fb6wf6R2q931+/hmlG2+vdaalw5s+OLuMBI3EddXnokH4JWZn2tfxsilNd+GGxD17XVADr+CnfK994bDLHuw7qnAgUi8ZNas48XRIUOmR1gCtI+HZmMjBojyA+Iq3JHEF5WXVEH4QY00/4hXoS0I33SZDSNB3nnFIkmS19/Zy0ycXcji7S/16He8BYnmtAwu1bRovNPhuXets20FB6MXC2OcLZY1V8tNDuLCTwJJiFsURVJ9yTocClFAfMF91XrntdNAYCSVAkp5e6hcs9pv8J3b+GC+0T+gb1tE2tMgwPOR4DFxWETUveJ5vR7LvyileMixfOgohb083pLBbtW7hLCcHBPKv43YFOK/j3jfWEmBbipDCpKY5+agCPJk/5tBnXEGEMnkjHMqb+RO/9CHLDevET/xAJURDrqc3V7l1l11Imyq9DhlfPOrMCVRyGm3vcqpqvfn3CLr0Y8e1KVKXXiqv3kLrt+BvCM+Ph5ylj0gYU48PCxmtAbdyzkRD79r+AQqc1XA+7Wn4d6tulWoRJhfKcuSz9YwRoVQDn5oUE2hzVyZXkXkOn5O5b+EoonTmqRPxr3POqVR7Pe8BHecvNEI7QWgyYO873v2EaGy8rkkI1h8mkpnGOKUFVc4z1RQU/YmuQwjblF8/QaQeh5Kq7p/+5L6l7TWSijPqcwcDRZrX0fU/ZK4PnQ1ccK6R3vSA+69H1MSvGKd5FLLzMiodV50ZqTLYNKfkpgL7dA7liPf6wvdARCK4PGkANmGB5dkvlIy8N6IUcyO5FBccvmKK0Q07gMZrJKz3wLdiAERp71iaqF+3pRemEDHfGsLZ6UGTEcst 0S4RmsIw BJezNacMKP1RCRdsqOCioDb3LAMFFtxicB+7LD6X1+lPFy278ArNWOMR1I8QuHlb3qJSE9U6blf80uMukH13xbJkMSHhFN232DXP+/3jLNN9fEK3gV7BN3344sM0J7BJxu0zQLq6aaolzoY532BJ0gVcv2uBoVQJ+s2arVgcQhAW49mZhx4TxUcYQDEItl2XgiEt/nzHdyaxOLjNkmKhjVVwpKFP8V64IYvMGAs0ZhlerYluePM3l3ZahZrNjOFRrcV29cKZYpqGN6UVfyecWXDz8PPSzOIk8Sg/kMCfXb7iHpzr3eGtH1vqXYxA9f/y9WIPFF/mrTPmxjhAfwCh0kNq8CiCC8VY7dxMPoe86/jyqkHwKOxx/Xjs/hcmttdDAHvV9TdGBU9zxnV1BmbE8s9sN3kaW9V9Rl8hh25bawFMygRggXDnFotcRN52u6Kl14CLM6jfPWvaLoFgkZ/4TqXMoByu5IXKHUuoVyT8Imk/so/Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Domenico Cerasuolo Currently, we only have a single global LRU for zswap. This makes it impossible to perform worload-specific shrinking - an memcg cannot determine which pages in the pool it owns, and often ends up writing pages from other memcgs. This issue has been previously observed in practice and mitigated by simply disabling memcg-initiated shrinking: https://lore.kernel.org/all/20230530232435.3097106-1-nphamcs@gmail.com/T/#u This patch fully resolves the issue by replacing the global zswap LRU with memcg- and NUMA-specific LRUs, and modify the reclaim logic: a) When a store attempt hits an memcg limit, it now triggers a synchronous reclaim attempt that, if successful, allows the new hotter page to be accepted by zswap. b) If the store attempt instead hits the global zswap limit, it will trigger an asynchronous reclaim attempt, in which an memcg is selected for reclaim in a round-robin-like fashion. Signed-off-by: Domenico Cerasuolo Co-developed-by: Nhat Pham Signed-off-by: Nhat Pham --- include/linux/memcontrol.h | 5 + mm/swap.h | 3 +- mm/swap_state.c | 23 +++-- mm/zswap.c | 188 ++++++++++++++++++++++++++----------- 4 files changed, 156 insertions(+), 63 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6edd3ec4d8d5..c1846e57011b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1187,6 +1187,11 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page) return NULL; } +static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *objcg) +{ + return NULL; +} + static inline bool folio_memcg_kmem(struct folio *folio) { return false; diff --git a/mm/swap.h b/mm/swap.h index 73c332ee4d91..c0dc73e10e91 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -51,7 +51,8 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct swap_iocb **plug); struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx, - bool *new_page_allocated); + bool *new_page_allocated, + bool skip_if_exists); struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, diff --git a/mm/swap_state.c b/mm/swap_state.c index 85d9e5806a6a..040639e1c77e 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -412,7 +412,8 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx, - bool *new_page_allocated) + bool *new_page_allocated, + bool skip_if_exists) { struct swap_info_struct *si; struct folio *folio; @@ -470,6 +471,16 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, if (err != -EEXIST) goto fail_put_swap; + /* Protect against a recursive call to __read_swap_cache_async() + * on the same entry waiting forever here because SWAP_HAS_CACHE + * is set but the folio is not the swap cache yet. This can + * happen today if mem_cgroup_swapin_charge_folio() below + * triggers reclaim through zswap, which may call + * __read_swap_cache_async() in the writeback path. + */ + if (skip_if_exists) + goto fail_put_swap; + /* * We might race against __delete_from_swap_cache(), and * stumble across a swap_map entry whose SWAP_HAS_CACHE @@ -537,7 +548,7 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, mpol = get_vma_policy(vma, addr, 0, &ilx); page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, - &page_allocated); + &page_allocated, false); mpol_cond_put(mpol); if (page_allocated) @@ -654,7 +665,7 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, /* Ok, do the async read-ahead now */ page = __read_swap_cache_async( swp_entry(swp_type(entry), offset), - gfp_mask, mpol, ilx, &page_allocated); + gfp_mask, mpol, ilx, &page_allocated, false); if (!page) continue; if (page_allocated) { @@ -672,7 +683,7 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, skip: /* The page was likely read above, so no need for plugging here */ page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, - &page_allocated); + &page_allocated, false); if (unlikely(page_allocated)) swap_readpage(page, false, NULL); return page; @@ -827,7 +838,7 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, pte_unmap(pte); pte = NULL; page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, - &page_allocated); + &page_allocated, false); if (!page) continue; if (page_allocated) { @@ -847,7 +858,7 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, skip: /* The page was likely read above, so no need for plugging here */ page = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, - &page_allocated); + &page_allocated, false); if (unlikely(page_allocated)) swap_readpage(page, false, NULL); return page; diff --git a/mm/zswap.c b/mm/zswap.c index 2e691cd1a466..ee8e227e7b0b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -35,6 +35,7 @@ #include #include #include +#include #include "swap.h" #include "internal.h" @@ -172,8 +173,8 @@ struct zswap_pool { struct work_struct shrink_work; struct hlist_node node; char tfm_name[CRYPTO_MAX_ALG_NAME]; - struct list_head lru; - spinlock_t lru_lock; + struct list_lru list_lru; + struct mem_cgroup *next_shrink; }; /* @@ -289,15 +290,25 @@ static void zswap_update_total_size(void) zswap_pool_total_size = total; } +static inline struct mem_cgroup *get_mem_cgroup_from_entry(struct zswap_entry *entry) +{ + return entry->objcg ? get_mem_cgroup_from_objcg(entry->objcg) : NULL; +} + +static inline int entry_to_nid(struct zswap_entry *entry) +{ + return page_to_nid(virt_to_page(entry)); +} + /********************************* * zswap entry functions **********************************/ static struct kmem_cache *zswap_entry_cache; -static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp) +static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) { struct zswap_entry *entry; - entry = kmem_cache_alloc(zswap_entry_cache, gfp); + entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); if (!entry) return NULL; entry->refcount = 1; @@ -310,6 +321,29 @@ static void zswap_entry_cache_free(struct zswap_entry *entry) kmem_cache_free(zswap_entry_cache, entry); } +/********************************* +* lru functions +**********************************/ +static bool zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) +{ + struct mem_cgroup *memcg = get_mem_cgroup_from_entry(entry); + int nid = entry_to_nid(entry); + bool added = list_lru_add(list_lru, &entry->lru, nid, memcg); + + mem_cgroup_put(memcg); + return added; +} + +static bool zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *entry) +{ + struct mem_cgroup *memcg = get_mem_cgroup_from_entry(entry); + int nid = entry_to_nid(entry); + bool removed = list_lru_del(list_lru, &entry->lru, nid, memcg); + + mem_cgroup_put(memcg); + return removed; +} + /********************************* * rbtree functions **********************************/ @@ -394,9 +428,7 @@ static void zswap_free_entry(struct zswap_entry *entry) if (!entry->length) atomic_dec(&zswap_same_filled_pages); else { - spin_lock(&entry->pool->lru_lock); - list_del(&entry->lru); - spin_unlock(&entry->pool->lru_lock); + zswap_lru_del(&entry->pool->list_lru, entry); zpool_free(zswap_find_zpool(entry), entry->handle); zswap_pool_put(entry->pool); } @@ -630,21 +662,16 @@ static void zswap_invalidate_entry(struct zswap_tree *tree, zswap_entry_put(tree, entry); } -static int zswap_reclaim_entry(struct zswap_pool *pool) +static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_one *l, + spinlock_t *lock, void *arg) { - struct zswap_entry *entry; + struct zswap_entry *entry = container_of(item, struct zswap_entry, lru); + struct mem_cgroup *memcg; struct zswap_tree *tree; pgoff_t swpoffset; - int ret; + enum lru_status ret = LRU_REMOVED_RETRY; + int writeback_result; - /* Get an entry off the LRU */ - spin_lock(&pool->lru_lock); - if (list_empty(&pool->lru)) { - spin_unlock(&pool->lru_lock); - return -EINVAL; - } - entry = list_last_entry(&pool->lru, struct zswap_entry, lru); - list_del_init(&entry->lru); /* * Once the lru lock is dropped, the entry might get freed. The * swpoffset is copied to the stack, and entry isn't deref'd again @@ -652,28 +679,37 @@ static int zswap_reclaim_entry(struct zswap_pool *pool) */ swpoffset = swp_offset(entry->swpentry); tree = zswap_trees[swp_type(entry->swpentry)]; - spin_unlock(&pool->lru_lock); + list_lru_isolate(l, item); + /* + * It's safe to drop the lock here because we return either + * LRU_REMOVED_RETRY or LRU_RETRY. + */ + spin_unlock(lock); /* Check for invalidate() race */ spin_lock(&tree->lock); - if (entry != zswap_rb_search(&tree->rbroot, swpoffset)) { - ret = -EAGAIN; + if (entry != zswap_rb_search(&tree->rbroot, swpoffset)) goto unlock; - } + /* Hold a reference to prevent a free during writeback */ zswap_entry_get(entry); spin_unlock(&tree->lock); - ret = zswap_writeback_entry(entry, tree); + writeback_result = zswap_writeback_entry(entry, tree); spin_lock(&tree->lock); - if (ret) { - /* Writeback failed, put entry back on LRU */ - spin_lock(&pool->lru_lock); - list_move(&entry->lru, &pool->lru); - spin_unlock(&pool->lru_lock); + if (writeback_result) { + zswap_reject_reclaim_fail++; + memcg = get_mem_cgroup_from_entry(entry); + spin_lock(lock); + /* we cannot use zswap_lru_add here, because it increments node's lru count */ + list_lru_putback(&entry->pool->list_lru, item, entry_to_nid(entry), memcg); + spin_unlock(lock); + mem_cgroup_put(memcg); + ret = LRU_RETRY; goto put_unlock; } + zswap_written_back_pages++; /* * Writeback started successfully, the page now belongs to the @@ -687,7 +723,34 @@ static int zswap_reclaim_entry(struct zswap_pool *pool) zswap_entry_put(tree, entry); unlock: spin_unlock(&tree->lock); - return ret ? -EAGAIN : 0; + spin_lock(lock); + return ret; +} + +static int shrink_memcg(struct mem_cgroup *memcg) +{ + struct zswap_pool *pool; + int nid, shrunk = 0; + + /* + * Skip zombies because their LRUs are reparented and we would be + * reclaiming from the parent instead of the dead memcg. + */ + if (memcg && !mem_cgroup_online(memcg)) + return -ENOENT; + + pool = zswap_pool_current_get(); + if (!pool) + return -EINVAL; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + unsigned long nr_to_walk = 1; + + shrunk += list_lru_walk_one(&pool->list_lru, nid, memcg, + &shrink_memcg_cb, NULL, &nr_to_walk); + } + zswap_pool_put(pool); + return shrunk ? 0 : -EAGAIN; } static void shrink_worker(struct work_struct *w) @@ -696,15 +759,17 @@ static void shrink_worker(struct work_struct *w) shrink_work); int ret, failures = 0; + /* global reclaim will select cgroup in a round-robin fashion. */ do { - ret = zswap_reclaim_entry(pool); - if (ret) { - zswap_reject_reclaim_fail++; - if (ret != -EAGAIN) - break; - if (++failures == MAX_RECLAIM_RETRIES) - break; - } + pool->next_shrink = mem_cgroup_iter(NULL, pool->next_shrink, NULL); + + ret = shrink_memcg(pool->next_shrink); + + if (ret == -EINVAL) + break; + if (ret && ++failures == MAX_RECLAIM_RETRIES) + break; + cond_resched(); } while (!zswap_can_accept()); zswap_pool_put(pool); @@ -765,8 +830,7 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) */ kref_init(&pool->kref); INIT_LIST_HEAD(&pool->list); - INIT_LIST_HEAD(&pool->lru); - spin_lock_init(&pool->lru_lock); + list_lru_init_memcg(&pool->list_lru, NULL); INIT_WORK(&pool->shrink_work, shrink_worker); zswap_pool_debug("created", pool); @@ -832,6 +896,9 @@ static void zswap_pool_destroy(struct zswap_pool *pool) cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); free_percpu(pool->acomp_ctx); + list_lru_destroy(&pool->list_lru); + if (pool->next_shrink) + mem_cgroup_put(pool->next_shrink); for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) zpool_destroy_pool(pool->zpools[i]); kfree(pool); @@ -1079,7 +1146,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* try to allocate swap cache page */ mpol = get_task_policy(current); page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, - NO_INTERLEAVE_INDEX, &page_was_allocated); + NO_INTERLEAVE_INDEX, &page_was_allocated, true); if (!page) { ret = -ENOMEM; goto fail; @@ -1145,7 +1212,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* start writeback */ __swap_writepage(page, &wbc); put_page(page); - zswap_written_back_pages++; return ret; @@ -1202,8 +1268,10 @@ bool zswap_store(struct folio *folio) struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; struct obj_cgroup *objcg = NULL; + struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; struct zpool *zpool; + int lru_alloc_ret; unsigned int dlen = PAGE_SIZE; unsigned long handle, value; char *buf; @@ -1233,15 +1301,15 @@ bool zswap_store(struct folio *folio) zswap_invalidate_entry(tree, dupentry); } spin_unlock(&tree->lock); - - /* - * XXX: zswap reclaim does not work with cgroups yet. Without a - * cgroup-aware entry LRU, we will push out entries system-wide based on - * local cgroup limits. - */ objcg = get_obj_cgroup_from_folio(folio); - if (objcg && !obj_cgroup_may_zswap(objcg)) - goto reject; + if (objcg && !obj_cgroup_may_zswap(objcg)) { + memcg = get_mem_cgroup_from_objcg(objcg); + if (shrink_memcg(memcg)) { + mem_cgroup_put(memcg); + goto reject; + } + mem_cgroup_put(memcg); + } /* reclaim space if needed */ if (zswap_is_full()) { @@ -1258,7 +1326,7 @@ bool zswap_store(struct folio *folio) } /* allocate entry */ - entry = zswap_entry_cache_alloc(GFP_KERNEL); + entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page)); if (!entry) { zswap_reject_kmemcache_fail++; goto reject; @@ -1285,6 +1353,15 @@ bool zswap_store(struct folio *folio) if (!entry->pool) goto freepage; + if (objcg) { + memcg = get_mem_cgroup_from_objcg(objcg); + lru_alloc_ret = memcg_list_lru_alloc(memcg, &entry->pool->list_lru, GFP_KERNEL); + mem_cgroup_put(memcg); + + if (lru_alloc_ret) + goto put_pool; + } + /* compress */ acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); @@ -1361,9 +1438,8 @@ bool zswap_store(struct folio *folio) zswap_invalidate_entry(tree, dupentry); } if (entry->length) { - spin_lock(&entry->pool->lru_lock); - list_add(&entry->lru, &entry->pool->lru); - spin_unlock(&entry->pool->lru_lock); + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&entry->pool->list_lru, entry); } spin_unlock(&tree->lock); @@ -1376,6 +1452,7 @@ bool zswap_store(struct folio *folio) put_dstmem: mutex_unlock(acomp_ctx->mutex); +put_pool: zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); @@ -1470,9 +1547,8 @@ bool zswap_load(struct folio *folio) zswap_invalidate_entry(tree, entry); folio_mark_dirty(folio); } else if (entry->length) { - spin_lock(&entry->pool->lru_lock); - list_move(&entry->lru, &entry->pool->lru); - spin_unlock(&entry->pool->lru_lock); + zswap_lru_del(&entry->pool->list_lru, entry); + zswap_lru_add(&entry->pool->list_lru, entry); } zswap_entry_put(tree, entry); spin_unlock(&tree->lock); From patchwork Tue Oct 24 20:33:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13435244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24144C25B6C for ; Tue, 24 Oct 2023 20:33:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14EFB6B02EC; Tue, 24 Oct 2023 16:33:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B0F26B02ED; Tue, 24 Oct 2023 16:33:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6D966B02EE; Tue, 24 Oct 2023 16:33:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D3D586B02EC for ; Tue, 24 Oct 2023 16:33:10 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ABC11408CE for ; Tue, 24 Oct 2023 20:33:10 +0000 (UTC) X-FDA: 81381504540.09.5F922F0 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf25.hostedemail.com (Postfix) with ESMTP id DB475A0008 for ; Tue, 24 Oct 2023 20:33:08 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SJHkdn8d; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698179588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ct7ILXQfwShOguQLCzN2H+ZHxo/ASPBQwkijnwC3QMY=; b=j1Nztu1D1d/6JxA8M0hBDXHt1Jfvc0qURgx30ND70DXZaNtgqhjyJoKTL9x960kOeA1E/Z L3Rftnq37VYfHj0HiCecREtlKgqa/SuBZs+BaE2v6fyE6UzNN1f+1TM9RrtlgrPs/piCGj j6z3OdPOxglWDI/7RjKta33e1CjG8+I= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SJHkdn8d; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698179588; a=rsa-sha256; cv=none; b=rEfzuO7kQwOhSVVHoTXBCTDLxcG3tW/u0z4rCmRR2uihNCwQWxwp5V3AtRqWADigKD8xIg Oc0rF7jojayHDG3uoW0kM0X3F8GRvB1CmpCb/C15mpfaHdHb1wFG8psvqWOaSVZUxhCiX8 L5upGfxaciXu4aqfhrrmeflZwVeQCvc= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6b6f4c118b7so4052019b3a.0 for ; Tue, 24 Oct 2023 13:33:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698179587; x=1698784387; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ct7ILXQfwShOguQLCzN2H+ZHxo/ASPBQwkijnwC3QMY=; b=SJHkdn8dTIikrOwAhRa3dmdcyxZng/vqpSeG9P4ss2eazIZzUNXd2uuDAzreaBEnGM xK7ArRBWnCFxy1OVUK0qAmHFnwFywMUFALsZCCMPc+OiaiVhBEADoAcNedrJdt7Y9QoI aKOedbNsrws/2CISLhLxVPvKDpqOZINJF9MTL/Zp2yj1Q0xABUVhWqnJLp8He/qfHKxI otadnWDNgmc07YuiLCRrzFGgOtVoXIGMgO7iVDimSYLMTsOXJF9nZ8Q20v2LZ7UOXWuq 6U4VBh+owVaSKGteR6usay5ywlXgZhW05oLlelxLnecZde7PbMgMPPCkJ03mpc7KTfKO 9tNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698179587; x=1698784387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ct7ILXQfwShOguQLCzN2H+ZHxo/ASPBQwkijnwC3QMY=; b=m6AthU6PhNlRFwj7wrUkN4g0RA14No8S2S2UVgJnBpXRCk64KpuefAOuWEiF8ACPCg Yp4hgESp1mguxKjNey4ZIdP6RkpQKdUnSmoG8Ff25W/QWb1WbJJ85AKYV3ftjfcroNyq cN7UCymZLF+pySdEoZ0DyzCr7OCB8USCUaPjEFI5XYxuMUEKI6coKkjld4SQ6Mz5dU04 yyZoHkpg9aCZXhwngCxtOsHkrU9MeAV94Zvv0toqkjYoJWv8qMTtyoyvqknCqBlgG5dK lLc4D25cr9hXkQoS/GAhp0DHsAK6qTR/+2M9+4D7P04krTJQAhvNpy/KGcgjPRno7BXe C8KA== X-Gm-Message-State: AOJu0Yw/b2RspUy8Lvk09uTPZ0KUI6ngPMprVXFXOTZZPLPwMu6WQREz 3mVMt+ZGHdhPuUmBOI9t+QY= X-Google-Smtp-Source: AGHT+IEqQ/yO6Q0UWV3ZQ4PNu7f2MkIruUL16KnuV71cH8iufU01U3EFGNtF5ep2fwen3nE8TNzFNg== X-Received: by 2002:a05:6a00:1142:b0:6be:23dd:d612 with SMTP id b2-20020a056a00114200b006be23ddd612mr12348489pfm.16.1698179587511; Tue, 24 Oct 2023 13:33:07 -0700 (PDT) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id z2-20020aa79f82000000b00692cac7a065sm7990229pfr.151.2023.10.24.13.33.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 13:33:06 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, shuah@kernel.org Subject: [PATCH v4 3/5] mm: memcg: add per-memcg zswap writeback stat Date: Tue, 24 Oct 2023 13:33:00 -0700 Message-Id: <20231024203302.1920362-4-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231024203302.1920362-1-nphamcs@gmail.com> References: <20231024203302.1920362-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 34fw7xy697w75i7z41kc3s1txpyz7r8f X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DB475A0008 X-HE-Tag: 1698179588-608699 X-HE-Meta: U2FsdGVkX18AVsO6drcsh/FueRV1LdB1qEReClDDGTUT+sMyTfS/gQ/udxsKkltnZJxMSDs2UKa2eFqfhNLvq1K9aj5vSO8smTzAlbrpGS4ryEsDZM0mBtc4+NlWt1kI1OHYB2qQwGD880yswc+p6MCipxmvoNIrRZaCPIhZbTZXKzUFTC21cbwwexzCqfdr9jucxw0PDHqYCCVuGEgjhXu5tFIWxR0jEJmN4qauMlITZWJ4X+M4VOOX8isHfiq4fnMKq7QBNECuKGcvmOp+6Phg3pjGMHZtxJD5syNbIeC7R1ZleGdJ2vdHv77WJZKbDbPPDnL360f9VBkK5axlPfgUX0NSqx/LL03zvOjck3xFOavrhi0JyDpj/39SP9K/+AGLBR03bhpFyjbFvf1a2f7yaj+26x/2lPAUZW3rUrkWFNOli9BCDrPapuvGVEOx84juh6pDk3jO+oxrzzc2ehMx1JPjM0x+zKwJhyEoHOIPUoXrKDKc4tqL/+CL3DekHX7Q40SzdVjjIe6/jT5lhw/cPKPiteI0HU1s6ENLsOBbNuLBtYOQ/ThFAieHeRW2Xo1Ylkh+4ZrlZnPBAI3qa3+3WbVUSWDexDk0MyoEgVqdpCGObg5H+NywavSlK4yJ6QJEEUbzTECxql8npm4zA1KtjkQJipXiir9WdeCj+tCS9BvUT+Ni1oadTyCtt0tYf5I22dYCVgigL9wZIUoNxda3L8zmDgef+6/dSPmrDNZzhBW8kH1tq7U1qJHFeHs2l9dC9+PFMnzkhxseE4Ua7uK4AMyn2trPAG/utgeeC22SKKLo6p8dep1xpfbmi53CbP+GZ02CNNfjDaKQ+yv5xqAjASgoPE4Uosh5oQb7WK+fOLyURS7Pu+Rtj6360HSKVNelOiHri8MZtDR7Pg6G49t0NtqT0VXDnVImQQ7AFpufrjlQCVRHcvPd5Qb6cGxTqbMq1RyIGD9qjW85lg7 AEjIXf5E K+xPj0w5kCSOZJRwh0QZssKIJWLbS4CU8dZLj+3RJkWT6p8UjgRfRNs6sm1NOd4AxfD4jRVS5zR2e0IrQTQU1LZTa0CjnGaXtW8MatsvCpatZ11ofdGT0PuaNdrkoUH7zJBzRBZh7mxiB5AGTIe2fHP1MnjMqEvzpBQLx6Z1nCMJO31qE+J3UunxVBdQkWCY1IF/6zkNFY8u/3c2cwJxqQSVK51I3aaKZQTfbvPCemqwd0aeXjS+HuCHR9Feg1Ta3rT0xFEwuhfxUrkOkOGF6vGTqPr/1WcfYX/qmKejtcyR+V3h+3a6gcHn+BFIcdqNDfErJAaBJmpAKsKR/WtFZ3kE0haSsQ3aiEgQhxPs/EoKKd8JxrzF1aL5ACWCPyAmaumB103PvFoP+uz+NgIdb8r4HpN7R4aZTLBjWzrmOgn/8TsEW6KgiXkln/CMv8WsdP9XJrLmb7kdWzFTiKXea9SGhHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Domenico Cerasuolo Since zswap now writes back pages from memcg-specific LRUs, we now need a new stat to show writebacks count for each memcg. Suggested-by: Nhat Pham Signed-off-by: Domenico Cerasuolo Signed-off-by: Nhat Pham --- include/linux/vm_event_item.h | 1 + mm/memcontrol.c | 1 + mm/vmstat.c | 1 + mm/zswap.c | 3 +++ 4 files changed, 6 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 8abfa1240040..3153359c3841 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -145,6 +145,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_ZSWAP ZSWPIN, ZSWPOUT, + ZSWP_WB, #endif #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 61c0c46c2d62..568d9d037a59 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -593,6 +593,7 @@ static const unsigned int memcg_vm_event_stat[] = { #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) ZSWPIN, ZSWPOUT, + ZSWP_WB, #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE THP_FAULT_ALLOC, diff --git a/mm/vmstat.c b/mm/vmstat.c index 359460deb377..5e5572f3b456 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1401,6 +1401,7 @@ const char * const vmstat_text[] = { #ifdef CONFIG_ZSWAP "zswpin", "zswpout", + "zswp_wb", #endif #ifdef CONFIG_X86 "direct_map_level2_splits", diff --git a/mm/zswap.c b/mm/zswap.c index ee8e227e7b0b..b87311e48de9 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -711,6 +711,9 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o } zswap_written_back_pages++; + if (entry->objcg) + count_objcg_event(entry->objcg, ZSWP_WB); + /* * Writeback started successfully, the page now belongs to the * swapcache. Drop the entry from zswap - unless invalidate already From patchwork Tue Oct 24 20:33:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13435245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A58AC07545 for ; Tue, 24 Oct 2023 20:33:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC5356B02ED; Tue, 24 Oct 2023 16:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A74E86B02EE; Tue, 24 Oct 2023 16:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89FC26B02EF; Tue, 24 Oct 2023 16:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7665F6B02ED for ; Tue, 24 Oct 2023 16:33:13 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 50F13B5DC1 for ; Tue, 24 Oct 2023 20:33:13 +0000 (UTC) X-FDA: 81381504666.14.C3323B6 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 6473110003F for ; Tue, 24 Oct 2023 20:33:11 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O5TWvnnT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698179591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9qKeVPLjWvPnKBs3WedCuvvPJRyMgM4Wh5BWftdbSso=; b=hk1d2k12YlG50zxVz1E4PAaHyvZ5B1mFhzh8/6a2Yz2rAKOU2DLyEuB6u2r3c+vlnkNH7b R4u+Ixx/5Gppm56sKSjmEtIhxh99N5J8FFv0505HKrMi8u6aGALQmyoerczGHq+YPrhwM5 wkMZOvjNreOHf36KUqNR0zcwUqFmCX4= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O5TWvnnT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698179591; a=rsa-sha256; cv=none; b=C2LM9Y083r2SNGPYGxfnFDpGxfGiDRq6qmxZvlRrqdgdCiGSVA71al1Lb0t/8Ed5kjJ0bF KFJsj7nT4z7CZ72K2+LXnquK7GHVygHXjrkef4UVvlgRSCGZ0Bi8Sr+/OWWVWe/oXO9O59 ckPXBbStyxpYKaVuIbA2j8FcxtDiAVE= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1c9e072472bso34355635ad.2 for ; Tue, 24 Oct 2023 13:33:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698179590; x=1698784390; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9qKeVPLjWvPnKBs3WedCuvvPJRyMgM4Wh5BWftdbSso=; b=O5TWvnnTb2VUen+GtBkvvBxv+6kidt9wgedw+/H4EA0AfAg3VKldJLRzwEYlQ17ewg PVJda4/0kzIkbjHwcRXL4D20uFI1AoY3BSj2HRJrYUZQ14fnxT38GYIpmEw4UmMlI7iF Yz+I2TFf51Ht1YK2ycMvj2HbMbpcQ7wQoWuXR5NyKuEQf7NsSI8Cwx08c+xp+MtQH60u DxQhPo7TkE4tUTs4Nr5nQ+/NHfmur9HCauAvxzCrwRqtJ95HIRpGMzJfvPl3u7+VR0cp 0MjFnn+RTFM4Mh073Wa4/b3pGCuD53gly9KkwaZLlGdv/m5qY6RjYo3zuGeDYWZSRc3F jXFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698179590; x=1698784390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9qKeVPLjWvPnKBs3WedCuvvPJRyMgM4Wh5BWftdbSso=; b=Fas01DTJ9HJXQFLQ1rJ5/JU5+OKe4Tc+odggO0bSHTfcIzavddqgz9AdJQGRuqbryK XNIY+LoSSvEVL6dDg6rnnNYpJlO0LnjnxVe6o+oYuxytWM+Z96eLq3BtyBKnq2zhXWnC zV8xq7PaVmiyjJ/MUeMs6kgSQvjRJ4WXFqocVhVSB8dtQyQYZVcyNnw9KyJL82vH8RQJ 5knCo5I8TfHl4iawTYbhv9Bh19AxRFbBKJyYaZiZamMXLe/HN5EasyRrvCtnxrZ/U6Qz uVZp889x0kvDhyFyLmkxoF2Y7DHNMcENESoTo4BE7ECyYTq8tcuBNHKPOPa2SEENmfdq CYzA== X-Gm-Message-State: AOJu0Yw6AOvGrNwi8f763XzzKIniYf5vDguMtcjUx7+YAmBeAcvQ5A/n BYXBbJ4C6FflfhJKWZNYvNk= X-Google-Smtp-Source: AGHT+IERx5hzi6G0let1CM31sv7+Us629KVVsLgKquopZJuJHwt3LRp3wcNM3+CzEUJJDXQYdvKu3g== X-Received: by 2002:a17:902:ce81:b0:1c6:2acc:62ea with SMTP id f1-20020a170902ce8100b001c62acc62eamr12845742plg.57.1698179590064; Tue, 24 Oct 2023 13:33:10 -0700 (PDT) Received: from localhost (fwdproxy-prn-001.fbsv.net. [2a03:2880:ff:1::face:b00c]) by smtp.gmail.com with ESMTPSA id p7-20020a170902e74700b001c7453fae33sm7731487plf.280.2023.10.24.13.33.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 13:33:09 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, shuah@kernel.org Subject: [PATCH v4 5/5] zswap: shrinks zswap pool based on memory pressure Date: Tue, 24 Oct 2023 13:33:02 -0700 Message-Id: <20231024203302.1920362-6-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231024203302.1920362-1-nphamcs@gmail.com> References: <20231024203302.1920362-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6473110003F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nyzt36fxc3i9cyhgpdednpbo7ocjhsfn X-HE-Tag: 1698179591-421441 X-HE-Meta: U2FsdGVkX19Gpkw/FJWL3X+lv+M0ltMVSauKdxx/MYzr8XT08KfVWY14ByixlyNEWSzm64aTVoQId0KE/KjIOR/Xl84JPNc3SoCqqMqoP1b6vGFLx8513MXa9ZEwFLtEA5j3RspaRtcsNh97AaCMlTMR/2PgJSjn7Q9yAOebjSiByCKyJzp4Un0c7yfcjxAy+Jpy1D/fCtu2WrtNDuYD0k3HBJkOZg8Owlk70PA9lGSX6D2M17SYDw8HVawFhpYhpf6R4TR9YLmVyVbH+err0/SFpFSnbeAXB0gg2B4yFqfa+24lHGrNiJLECMjqz0Zhjvy3ycT9GRasjKa9bAhHEiNPXgqsk0kzfOPYLurBhd8gkMWAk57pahnsL7wZxpP+AA3fS9264TOprAmHDM/WF8I6vEm5P0QA8Mb/xQtUnyBy8gcCTlz91pPAKBW9chXupYbWvNry9n+F0PCtFQOdH5qj1UQN+tjdBIng7sG86cpyoBLcDobCNupVpXD3zIpTY3Y29kq6un6q6nwMG0vJEwuwhi5d6iWsFTdlqGuWrL37Y4/SK9tir6Q9DccDdu49+3S8zSg83tkyasNnCyVmoH7nTT22WfyOoFC75R1lB8IcwdcBvI35QGHx2Cl5No94PLbVpgcpgMFfs/L7NLKH2+ZhH6iggIaVB4SYQDxRMOJdoYmZEjvl+60EemW86rI+USUrnmldgbiIneGsATu3BPMAigcSHrNwvdpqf73ZbVZYOZBfvVq2mP/Zek7HoKxvJMhFb8A6I2pY7FuZ0FYahtMcxfsy6sOa+UKehsixFy2Bjuk7tvn9k+mWY/PzGCtPX54XdqlaQmsnsn5p22pDjjmKtKRrsg4vZfS9/DxP5bAsEQ0zrjzS8GzTe63jLdqmeKGxIXPnfUm7E0Zptz4j3f4YfTKx/Xdsi9Nx4XzJNUFB03iXie8sc/yK0wb+qxwcmKf2n5Hm+vL2rTqgVJk OG2Fe3Lp foIzKjZqAOC6G+Okv9AyAcMJJ6iubpGb/CGd3s8+hFwegn8HxrFDK/CnBxsgLeCBEN9G0NO6tP8Y0DiTYZ27lTVh2U9nhK2f025RmTsLU3J1QsdnLaYmkOIXrjTYEshcLq3UCL+eIWkLW4TPVsZ9pK+i47p/yDnV03ct4nyvQH6CckiS5lwpAFUTYNym4lJaBOyVSl0Eo6EsgE3L15QVduCxgYgp7S7ie3cNWg6CcgtVuwLIXRe6DwRnF6AJb3Gp/f6IHUmHgB9UoGlH0iA57Q8zwrzfzyRFpi93AluoH2ACRdR/jLnhlYpFvqbmC/jseS+m98GDjkXI+FBB9ff+BMw695jWgD93jX5ZfXGRgDpC4yQY+hTpRJy+AUrX1CWkuqVGmi86+wxt7eJPEfZNwzI13hswsfNnwnjkI+1myYE8x5WVagpmPBpSk2Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, we only shrink the zswap pool when the user-defined limit is hit. This means that if we set the limit too high, cold data that are unlikely to be used again will reside in the pool, wasting precious memory. It is hard to predict how much zswap space will be needed ahead of time, as this depends on the workload (specifically, on factors such as memory access patterns and compressibility of the memory pages). This patch implements a memcg- and NUMA-aware shrinker for zswap, that is initiated when there is memory pressure. The shrinker does not have any parameter that must be tuned by the user, and can be opted in or out on a per-memcg basis. Furthermore, to make it more robust for many workloads and prevent overshrinking (i.e evicting warm pages that might be refaulted into memory), we build in the following heuristics: * Estimate the number of warm pages residing in zswap, and attempt to protect this region of the zswap LRU. * Scale the number of freeable objects by an estimate of the memory saving factor. The better zswap compresses the data, the fewer pages we will evict to swap (as we will otherwise incur IO for relatively small memory saving). * During reclaim, if the shrinker encounters a page that is also being brought into memory, the shrinker will cautiously terminate its shrinking action, as this is a sign that it is touching the warmer region of the zswap LRU. As a proof of concept, we ran the following synthetic benchmark: build the linux kernel in a memory-limited cgroup, and allocate some cold data in tmpfs to see if the shrinker could write them out and improved the overall performance. Depending on the amount of cold data generated, we observe from 14% to 35% reduction in kernel CPU time used in the kernel builds. Signed-off-by: Nhat Pham --- Documentation/admin-guide/mm/zswap.rst | 7 + include/linux/mmzone.h | 2 + include/linux/zswap.h | 25 +++- mm/mmzone.c | 1 + mm/swap_state.c | 2 + mm/zswap.c | 178 ++++++++++++++++++++++++- 6 files changed, 208 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/mm/zswap.rst b/Documentation/admin-guide/mm/zswap.rst index 45b98390e938..522ae22ccb84 100644 --- a/Documentation/admin-guide/mm/zswap.rst +++ b/Documentation/admin-guide/mm/zswap.rst @@ -153,6 +153,13 @@ attribute, e. g.:: Setting this parameter to 100 will disable the hysteresis. +When there is a sizable amount of cold memory residing in the zswap pool, it +can be advantageous to proactively write these cold pages to swap and reclaim +the memory for other use cases. By default, the zswap shrinker is disabled. +User can enable it as follows: + + echo Y > /sys/module/zswap/parameters/shrinker_enabled + A debugfs interface is provided for various statistic about pool size, number of pages stored, same-value filled pages and various counters for the reasons pages are rejected. diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 12f31633be05..633afdb96c40 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -22,6 +22,7 @@ #include #include #include +#include #include /* Free memory management - zoned buddy allocator. */ @@ -637,6 +638,7 @@ struct lruvec { #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif + struct zswap_lruvec_state zswap_lruvec_state; }; /* Isolate for asynchronous migration */ diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 2a60ce39cfde..64d0c3644185 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -5,19 +5,39 @@ #include #include +struct lruvec; + extern u64 zswap_pool_total_size; extern atomic_t zswap_stored_pages; #ifdef CONFIG_ZSWAP +struct zswap_lruvec_state { + /* + * Number of pages in zswap that should be protected from the shrinker. + * This number is an estimate of the following counts: + * + * a) Recent page faults. + * b) Recent insertion to the zswap LRU. This includes new zswap stores, + * as well as recent zswap LRU rotations. + * + * These pages are likely to be warm, and might incur IO if the are written + * to swap. + */ + atomic_long_t nr_zswap_protected; +}; + bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); void zswap_invalidate(int type, pgoff_t offset); void zswap_swapon(int type); void zswap_swapoff(int type); - +void zswap_lruvec_state_init(struct lruvec *lruvec); +void zswap_lruvec_swapin(struct page *page); #else +struct zswap_lruvec_state {}; + static inline bool zswap_store(struct folio *folio) { return false; @@ -31,7 +51,8 @@ static inline bool zswap_load(struct folio *folio) static inline void zswap_invalidate(int type, pgoff_t offset) {} static inline void zswap_swapon(int type) {} static inline void zswap_swapoff(int type) {} - +static inline void zswap_lruvec_init(struct lruvec *lruvec) {} +static inline void zswap_lruvec_swapin(struct page *page) {} #endif #endif /* _LINUX_ZSWAP_H */ diff --git a/mm/mmzone.c b/mm/mmzone.c index b594d3f268fe..c01896eca736 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -78,6 +78,7 @@ void lruvec_init(struct lruvec *lruvec) memset(lruvec, 0, sizeof(struct lruvec)); spin_lock_init(&lruvec->lru_lock); + zswap_lruvec_state_init(lruvec); for_each_lru(lru) INIT_LIST_HEAD(&lruvec->lists[lru]); diff --git a/mm/swap_state.c b/mm/swap_state.c index 040639e1c77e..38ce9981b0be 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -686,6 +686,7 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, &page_allocated, false); if (unlikely(page_allocated)) swap_readpage(page, false, NULL); + zswap_lruvec_swapin(page); return page; } @@ -861,6 +862,7 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, &page_allocated, false); if (unlikely(page_allocated)) swap_readpage(page, false, NULL); + zswap_lruvec_swapin(page); return page; } diff --git a/mm/zswap.c b/mm/zswap.c index b87311e48de9..c40697f07ba3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -146,6 +146,10 @@ module_param_named(exclusive_loads, zswap_exclusive_loads_enabled, bool, 0644); /* Number of zpools in zswap_pool (empirically determined for scalability) */ #define ZSWAP_NR_ZPOOLS 32 +/* Enable/disable memory pressure-based shrinker. */ +static bool zswap_shrinker_enabled; +module_param_named(shrinker_enabled, zswap_shrinker_enabled, bool, 0644); + /********************************* * data structures **********************************/ @@ -175,6 +179,8 @@ struct zswap_pool { char tfm_name[CRYPTO_MAX_ALG_NAME]; struct list_lru list_lru; struct mem_cgroup *next_shrink; + struct shrinker *shrinker; + atomic_t nr_stored; }; /* @@ -273,17 +279,26 @@ static bool zswap_can_accept(void) DIV_ROUND_UP(zswap_pool_total_size, PAGE_SIZE); } +static u64 get_zswap_pool_size(struct zswap_pool *pool) +{ + u64 pool_size = 0; + int i; + + for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) + pool_size += zpool_get_total_size(pool->zpools[i]); + + return pool_size; +} + static void zswap_update_total_size(void) { struct zswap_pool *pool; u64 total = 0; - int i; rcu_read_lock(); list_for_each_entry_rcu(pool, &zswap_pools, list) - for (i = 0; i < ZSWAP_NR_ZPOOLS; i++) - total += zpool_get_total_size(pool->zpools[i]); + total += get_zswap_pool_size(pool); rcu_read_unlock(); @@ -321,6 +336,24 @@ static void zswap_entry_cache_free(struct zswap_entry *entry) kmem_cache_free(zswap_entry_cache, entry); } +/********************************* +* zswap lruvec functions +**********************************/ +void zswap_lruvec_state_init(struct lruvec *lruvec) +{ + atomic_long_set(&lruvec->zswap_lruvec_state.nr_zswap_protected, 0); +} + +void zswap_lruvec_swapin(struct page *page) +{ + struct lruvec *lruvec; + + if (page) { + lruvec = folio_lruvec(page_folio(page)); + atomic_long_inc(&lruvec->zswap_lruvec_state.nr_zswap_protected); + } +} + /********************************* * lru functions **********************************/ @@ -328,8 +361,24 @@ static bool zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) { struct mem_cgroup *memcg = get_mem_cgroup_from_entry(entry); int nid = entry_to_nid(entry); + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); bool added = list_lru_add(list_lru, &entry->lru, nid, memcg); + atomic_long_t *nr_zswap_protected = + &lruvec->zswap_lruvec_state.nr_zswap_protected; + unsigned long lru_size, old, new; + if (added) { + lru_size = list_lru_count_one(list_lru, nid, memcg); + old = atomic_long_inc_return(nr_zswap_protected); + + /* + * Decay to avoid overflow and adapt to changing workloads. + * This is based on LRU reclaim cost decaying heuristics. + */ + do { + new = old > lru_size / 4 ? old / 2 : old; + } while (!atomic_long_try_cmpxchg(nr_zswap_protected, &old, new)); + } mem_cgroup_put(memcg); return added; } @@ -430,6 +479,7 @@ static void zswap_free_entry(struct zswap_entry *entry) else { zswap_lru_del(&entry->pool->list_lru, entry); zpool_free(zswap_find_zpool(entry), entry->handle); + atomic_dec(&entry->pool->nr_stored); zswap_pool_put(entry->pool); } zswap_entry_cache_free(entry); @@ -471,6 +521,95 @@ static struct zswap_entry *zswap_entry_find_get(struct rb_root *root, return entry; } +/********************************* +* shrinker functions +**********************************/ +static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_one *l, + spinlock_t *lock, void *arg); + +static unsigned long zswap_shrinker_scan(struct shrinker *shrinker, + struct shrink_control *sc) +{ + struct lruvec *lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); + unsigned long shrink_ret, nr_protected, lru_size; + struct zswap_pool *pool = shrinker->private_data; + bool encountered_page_in_swapcache = false; + + nr_protected = + atomic_long_read(&lruvec->zswap_lruvec_state.nr_zswap_protected); + lru_size = list_lru_shrink_count(&pool->list_lru, sc); + + /* + * Abort if the shrinker is disabled or if we are shrinking into the + * protected region. + */ + if (!zswap_shrinker_enabled || nr_protected >= lru_size - sc->nr_to_scan) { + sc->nr_scanned = 0; + return SHRINK_STOP; + } + + shrink_ret = list_lru_shrink_walk(&pool->list_lru, sc, &shrink_memcg_cb, + &encountered_page_in_swapcache); + + if (encountered_page_in_swapcache) + return SHRINK_STOP; + + return shrink_ret ? shrink_ret : SHRINK_STOP; +} + +static unsigned long zswap_shrinker_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + struct zswap_pool *pool = shrinker->private_data; + struct mem_cgroup *memcg = sc->memcg; + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(sc->nid)); + unsigned long nr_backing, nr_stored, nr_freeable, nr_protected; + +#ifdef CONFIG_MEMCG_KMEM + cgroup_rstat_flush(memcg->css.cgroup); + nr_backing = memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT; + nr_stored = memcg_page_state(memcg, MEMCG_ZSWAPPED); +#else + /* use pool stats instead of memcg stats */ + nr_backing = get_zswap_pool_size(pool) >> PAGE_SHIFT; + nr_stored = atomic_read(&pool->nr_stored); +#endif + + if (!zswap_shrinker_enabled || !nr_stored) + return 0; + + nr_protected = + atomic_long_read(&lruvec->zswap_lruvec_state.nr_zswap_protected); + nr_freeable = list_lru_shrink_count(&pool->list_lru, sc); + /* + * Subtract the lru size by an estimate of the number of pages + * that should be protected. + */ + nr_freeable = nr_freeable > nr_protected ? nr_freeable - nr_protected : 0; + + /* + * Scale the number of freeable pages by the memory saving factor. + * This ensures that the better zswap compresses memory, the fewer + * pages we will evict to swap (as it will otherwise incur IO for + * relatively small memory saving). + */ + return mult_frac(nr_freeable, nr_backing, nr_stored); +} + +static void zswap_alloc_shrinker(struct zswap_pool *pool) +{ + pool->shrinker = + shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE, "mm-zswap"); + if (!pool->shrinker) + return; + + pool->shrinker->private_data = pool; + pool->shrinker->scan_objects = zswap_shrinker_scan; + pool->shrinker->count_objects = zswap_shrinker_count; + pool->shrinker->batch = 0; + pool->shrinker->seeks = DEFAULT_SEEKS; +} + /********************************* * per-cpu code **********************************/ @@ -666,8 +805,10 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o spinlock_t *lock, void *arg) { struct zswap_entry *entry = container_of(item, struct zswap_entry, lru); + bool *encountered_page_in_swapcache = (bool *)arg; struct mem_cgroup *memcg; struct zswap_tree *tree; + struct lruvec *lruvec; pgoff_t swpoffset; enum lru_status ret = LRU_REMOVED_RETRY; int writeback_result; @@ -705,8 +846,22 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o /* we cannot use zswap_lru_add here, because it increments node's lru count */ list_lru_putback(&entry->pool->list_lru, item, entry_to_nid(entry), memcg); spin_unlock(lock); - mem_cgroup_put(memcg); ret = LRU_RETRY; + + /* + * Encountering a page already in swap cache is a sign that we are shrinking + * into the warmer region. We should terminate shrinking (if we're in the dynamic + * shrinker context). + */ + if (writeback_result == -EEXIST && encountered_page_in_swapcache) { + ret = LRU_SKIP; + *encountered_page_in_swapcache = true; + } + lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(entry_to_nid(entry))); + /* Increment the protection area to account for the LRU rotation. */ + atomic_long_inc(&lruvec->zswap_lruvec_state.nr_zswap_protected); + + mem_cgroup_put(memcg); goto put_unlock; } zswap_written_back_pages++; @@ -826,6 +981,11 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) &pool->node); if (ret) goto error; + + zswap_alloc_shrinker(pool); + if (!pool->shrinker) + goto error; + pr_debug("using %s compressor\n", pool->tfm_name); /* being the current pool takes 1 ref; this func expects the @@ -833,13 +993,19 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) */ kref_init(&pool->kref); INIT_LIST_HEAD(&pool->list); - list_lru_init_memcg(&pool->list_lru, NULL); + if (list_lru_init_memcg(&pool->list_lru, pool->shrinker)) + goto lru_fail; + shrinker_register(pool->shrinker); INIT_WORK(&pool->shrink_work, shrink_worker); + atomic_set(&pool->nr_stored, 0); zswap_pool_debug("created", pool); return pool; +lru_fail: + list_lru_destroy(&pool->list_lru); + shrinker_free(pool->shrinker); error: if (pool->acomp_ctx) free_percpu(pool->acomp_ctx); @@ -897,6 +1063,7 @@ static void zswap_pool_destroy(struct zswap_pool *pool) zswap_pool_debug("destroying", pool); + shrinker_free(pool->shrinker); cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); free_percpu(pool->acomp_ctx); list_lru_destroy(&pool->list_lru); @@ -1443,6 +1610,7 @@ bool zswap_store(struct folio *folio) if (entry->length) { INIT_LIST_HEAD(&entry->lru); zswap_lru_add(&entry->pool->list_lru, entry); + atomic_inc(&entry->pool->nr_stored); } spin_unlock(&tree->lock);