From patchwork Wed Jun 19 09:20:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chris Li X-Patchwork-Id: 13703552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BAE4C2BA1A for ; Wed, 19 Jun 2024 09:20:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 918BE6B02DD; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A06A6B02E0; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F1106B02DF; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2994C8D004F for ; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9EE1F40FA7 for ; Wed, 19 Jun 2024 09:20:42 +0000 (UTC) X-FDA: 82247093124.06.DA4C826 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf09.hostedemail.com (Postfix) with ESMTP id 4EF4A14000D for ; Wed, 19 Jun 2024 09:20:39 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="aqt/s8BR"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718788834; a=rsa-sha256; cv=none; b=6yIzdYEBK12yZQj63sepSj6suFkLEoPfvSXvl+8WeC8ksy/DptYhn21mtNQclOkmQrVN6u QkaVCJ6J3RQ1+QfPfpWEd3WX3JVohP9/pT49ZVCByFNWTefebY7ufntEdR8y6/gIkQMxNB +HH3MkQ7mzbz7HIdlXDRIa3md90IPew= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="aqt/s8BR"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718788834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xq2n0tw+klVSEtQIdrBTohuWHoCN5V/F69rNHfL+Me4=; b=CRHmaXD4NnV5jJE17q8hpRaR7v8+qjRwpJF7Fsy4pVMS+76qR0vX6L02oESEts2V5mYcxz KdXaTggqQQh07j5Pp1tL1PxVCUW5DzLv/zwKpsIWkF+KcsIJvA0e8nustE82Mgo097a05T pvLgYk7whHUXI1qZyXg+JZ6HtyziGRo= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 6BD20CE1E37; Wed, 19 Jun 2024 09:20:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6AFE4C4AF49; Wed, 19 Jun 2024 09:20:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718788836; bh=FKmZEOOj+8xe3zO8a1V3U71PZl7D+ukIzZ+enpM2r0A=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=aqt/s8BRPJSS1M01V9JHeMII64IVR1p02doAtxRFDlR/L+DPjSim/Pqt13AvzEHD9 dh3CcggMBedVDbdEC0Agp3EnRQ0dlwlCZQXKrGJul8lPzNpfEml1ggoyc2REh08bDh fpeupgIz/feW1X98iKCWGIteeJVzmdCNfFePwBtSX16RUkBgq/tCo7be9T5hdJJiDx KI1cQi2WgByDXQMUDYVnwpw/5EzJOnvEjngE+fSozFZc+RAiIdBjNCWchUHYDCUW5C CNH7JgKsN3F3cCd3XTw3C9nm0YRJyLOpXKMN+tlrET2ToGJx3a0Ryk10wXY8qAKEfq waMXB9N+j//OQ== From: Chris Li Date: Wed, 19 Jun 2024 02:20:30 -0700 Subject: [PATCH v3 2/2] mm: swap: mTHP allocate swap entries from nonfull list MIME-Version: 1.0 Message-Id: <20240619-swap-allocator-v3-2-e973a3102444@kernel.org> References: <20240619-swap-allocator-v3-0-e973a3102444@kernel.org> In-Reply-To: <20240619-swap-allocator-v3-0-e973a3102444@kernel.org> To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , Kairui Song , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Rspamd-Queue-Id: 4EF4A14000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3ynmypxdm4fknhs67ga8hfyjgnokzxgi X-HE-Tag: 1718788839-371016 X-HE-Meta: U2FsdGVkX199UcgsAao6LrQZaZ+ich5uYUuRfNNLjp3aaEfmPPyOeEkg4SFI6LrK7vqKhE7Vy1HKVh3yRqqaXtVLfBFES4eWs0r1BUakmik8LsuC1UtsBPCxV9R9mzROp5iSaYI1xVq1w8xA7PgP1RbjAZu9qKWUo1ruClIPRmyjDyCDwcWghdzKl3RtSoKcQzCRP8RRMwtKM5zd5XwQ02G9rPpeMIJOSnmOe5yS3bwe9HzHqEXVb94z8DvsN7B8vtOdqh7qw7tcf+A1PrXNDQfh7FlbmrDSFyouFq5XDL8vBc9XrnD2ltWDDXiU9Mruqls/fO4IEDSbGwoXerLHIcPLjWGFrYVu8cpbUGCylFs8IZgKkP6RuEGvhM/2DGZTUoZrYBXrhyy2IuNzy0uFdkmS3p+KaQj/9tcgwRmcl/9F7VAqGOMnAtawka022r9486uV4YEMc9cX6YMLimDnV21pbKLCDvw155/i1j7YWKyHcT5ZWQSr8RQDv4gUaOGS1iP1Q9lD6AWjt6o4MK2oTAPNgAcu3dL1nvvf9dypLlSoFLM1zKdH9HlZw4Xo2KbLYnNCDxKh6gk71fj7I7CW5A6n8QSICl3Nxu3/pyPXdCsNJc4W+5UiHiIKAYtl5eD9uPp/Iv8FRnL1AvC5M78Nh6bCl32AYVCMI3aTCTyLLGjcg75jIB12Cob27nuXKfRr3WjpBNyDRc/Am9DVrQja0qDr6LAUmWelAGGMUMzen6WGeqhBONsZ5xNGMCVPX6P7uKd0JYC18PrIy5tyarIfF2UuE9DKUWujxCobIIzh6EORl+8fY3IknuH0P5asZYJD5GV9WcXgxXk10xW+wxkVDvZzUUbuF0dFZh+E9nvIRMOKhreJ0al1wN5SD7CMlB4+xPmvsowvyItu0MXhlI4pZibz7QoeEWMu7MzSSu1zX8SnPlgiQKDc3FhvIWxWNvepe9j1OScHZh1VhWKgU3V E2OXV+wN S6tWbukAq8XOMSRS5rwdXtH1OsrZBH0dmvqs33k0KMGEqzIPPA3cEZMHoPAz8fu5QTUvrafrCQRdxNbZUxxc/sRYChqrhhDKyAM+DBhuAdZQvwl4/FJlQ8UVz4zuRrqvFozLvLsdUb7rtJg2BoWEbfT095QC4fiwyhCDjC4PJrBoQYGEyL6ATztvjZHCqCXnyHGDNYL98Zj59EbSPPAApQPwSRP90hNKMVsUIkQxXJLRBIS/kjXFR5D/WKZdcNsSiASU1BnZ8JgJ6dze216q7A7vRmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list.  When the free cluster list is empty, also allocate from the nonempty list of that order. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. Signed-off-by: Chris Li --- include/linux/swap.h | 4 ++++ mm/swapfile.c | 27 ++++++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 690a04f06674..92613bb4a87b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -254,9 +254,11 @@ struct swap_cluster_info { */ u16 count; u8 flags; + u8 order; struct list_head list; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ /* @@ -295,6 +297,8 @@ struct swap_info_struct { unsigned char *swap_map; /* vmalloc'ed array of usage counts */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 0b11c437f9cc..ba6676a4a8ef 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -361,8 +361,11 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->list, &si->free_clusters); + else + list_add_tail(&ci->list, &si->free_clusters); ci->flags = CLUSTER_FLAG_FREE; - list_add_tail(&ci->list, &si->free_clusters); } /* @@ -485,7 +488,12 @@ static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluste ci->count--; if (!ci->count) - free_cluster(p, ci); + return free_cluster(p, ci); + + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); + ci->flags |= CLUSTER_FLAG_NONFULL; + } } /* @@ -542,10 +550,18 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, cluster = this_cpu_ptr(si->percpu_cluster); tmp = cluster->next[order]; if (tmp == SWAP_NEXT_INVALID) { - if (!list_empty(&si->free_clusters)) { + if (!list_empty(&si->nonfull_clusters[order])) { + ci = list_first_entry(&si->nonfull_clusters[order], struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->flags = 0; + spin_unlock(&ci->lock); + tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->free_clusters)) { ci = list_first_entry(&si->free_clusters, struct swap_cluster_info, list); list_del(&ci->list); spin_lock(&ci->lock); + ci->order = order; ci->flags = 0; spin_unlock(&ci->lock); tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; @@ -576,6 +592,7 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, break; tmp += nr_pages; } + WARN_ONCE(ci->order != order, "expecting order %d got %d", order, ci->order); unlock_cluster(ci); } if (tmp >= max) { @@ -954,6 +971,7 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) ci = lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); ci->count = 0; + ci->order = 0; ci->flags = 0; free_cluster(si, ci); unlock_cluster(ci); @@ -2898,6 +2916,9 @@ static int setup_swap_map_and_extents(struct swap_info_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); + for (i = 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i = 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr = swap_header->info.badpages[i]; if (page_nr == 0 || page_nr > swap_header->info.last_page)