From patchwork Wed Jun 19 09:20:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Li X-Patchwork-Id: 13703551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8301C27C53 for ; Wed, 19 Jun 2024 09:20:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D5956B02DB; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25CD16B02DD; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DA688D004F; Wed, 19 Jun 2024 05:20:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DBC156B02DB for ; Wed, 19 Jun 2024 05:20:42 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8B3AD16101E for ; Wed, 19 Jun 2024 09:20:42 +0000 (UTC) X-FDA: 82247093124.13.DB9BAD6 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf07.hostedemail.com (Postfix) with ESMTP id 37DF740014 for ; Wed, 19 Jun 2024 09:20:39 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BkJEyVUb; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718788834; a=rsa-sha256; cv=none; b=VnQgjc0XB9PkpbscHnu5isiGCQa8VhgR1wBcC9sDFzVMAR5EY5MA7PU6C9q9pUVO1jD7nk YfTRSqC1Y0EZkXZRi8cafEucmRy9+1wUGGMJKw5ARhYtTXEsVuV9xRdufKne9mIsbgKywn XGDeOsMCgNLcMEhhygA3koaMFf3Ej1E= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BkJEyVUb; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718788834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=kiWRlK+henIPEWV5yefSrp709H3TrT3lZbJJescu+7w=; b=8kTvBqjDxh0+XEmSYvvriXlIIBrPyecVfX4CoA5QsyGebuguBjMgiP9bxcUQnXSbPeVtmd pk9wu8qS0liy2YKIN4hl4lf7huAYBfZoUv0ZONh/tjoaakY68prGUDb35L8lxjIklMGf8V YPI97OE79oABnJmvdaVozoNL88BFSdI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 8044BCE1E18; Wed, 19 Jun 2024 09:20:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74094C2BBFC; Wed, 19 Jun 2024 09:20:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718788835; bh=COkjaeKz8fplJ2Dg3wgX4Wj7712CRaMzdyemxxKURNA=; h=From:Subject:Date:To:Cc:From; b=BkJEyVUbJYJggEbKPHubYTt8uEq+AnhSxxHyyE9szcog0Mhldx0glFjUUTioi0a3/ VF3YC7aAEOrhQZMfnftIR7hY6RO42Gl9txADrlIDHAtGeIRdjzcMbiUBKhqqO9PWpE tnqmpl3l+rT4CRUCh3AKGAiy4mMiJEngj3iQEzJQtnx6ZqkCO3tuUW5b5DMRDVk1u/ VssPNiWuuCEmSCV0uGORAVu5WbA+vN8aGrT8aB+NKXT7Lx4HTMajtu3R/yqE4YfuuP eLxyOVI8UbllFLuzSFUsgVpzgiKLbn8zR27YTyHH+BQG4bVYHLrHyxukQbbXPXzWeB vQ98F8NlsMIeg== From: Chris Li Subject: [PATCH v3 0/2] mm: swap: mTHP swap allocator base on swap cluster order Date: Wed, 19 Jun 2024 02:20:28 -0700 Message-Id: <20240619-swap-allocator-v3-0-e973a3102444@kernel.org> MIME-Version: 1.0 X-B4-Tracking: v=1; b=H4sIANyicmYC/2WMwQ6CMBAFf4X0bE273QLx5H8YD6Uu0EgoaU3VE P7dQmKMepyXNzOzSMFRZIdiZoGSi86PGdSuYLY3Y0fcXTIzEIBCg+LxbiZuhsFbc/OBS63QYi3 IErIsTYFa99iCp3Pm3sV8e279JNf1ncLfVJJccKzqUjYIqoHyeKUw0rD3oWNrK8HHL+W/D9kHo 6Vq0FQttF/+siwvADQUQ+0AAAA= To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , Kairui Song , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 37DF740014 X-Stat-Signature: ktt4zxex5zpnxabqp9u6t4f84yj1z7nd X-Rspam-User: X-HE-Tag: 1718788839-60315 X-HE-Meta: U2FsdGVkX18i5x+ZaBqoOX6WsNrydnaSdpHL/Dp2lLFQ8gZ1Sg+CAxDI5B1RorGlvSW9BX/cWvjrvvLBEIoN6puVwUaxK6biJBMWI9t0zsy8B+/2DAGQO1PKNlZEOxnWOzInYvQgSxyQdtImD0jaqnB54nPi1CXG3WOjT2kpUapiKT5e6vdkC0diVDvPMrN3YiWuwNgHXjjjrrOrSGffYHsHR4+jY5vW850mSTy1HGAesdl6b8CUTCvWa5rGntCkpdZQun2aapes9UrSv+QVM8VEU1No56UmhcgmTGvm4bu4+5eBr3Y97JDgg+QJe+4Tplg1XOE0/qMhL7FBMEGkdcsjfnJtjFtiCFkUKMQKdrIwJ/4DJ6XgqEpA+K2Xn7uUAsvb85fxz9aTW1tW7JOKEf/6i+NSnwM2eRdKerL3MmywxvhWSgTytmEOozX/WHv+DMZ6bHL0J5ypEosea505qayPkBnXzHfNHHDswg8piYFXAgwbVmPGhdMVbwEMbRjHqZj19nuKZFwQbJW4u0zyZx3aBeRe0P4+McO9nrG8ykgdueJEj//X1VCNREsIkya1atvcgDY0OvCHZ/FeGlUWhO1i8GzGCugC/y7iCBc8rXB0Ua8mKd5xAqLTtRfDLg9Doqu1DmT6cEB3BHrUZocLinAfJLMeWZIb/Btees6ROTCDRq5tct4eg+bLWOE2umSxibYSPIh5o61PC21R7d2h+Hwxfc0H1txbx+vYpxia3xxMJmEajN+pvsGzr/IBHH7GA4l1D2Z1fRucfwESQ6HFUnRzCQ17AFHNbLwBvcZ7XMVlij9hEiEKtoUei0ox9WS2R1jnGeahKa4DrmVVf5leQvT2+S02M1nJQgsQpukIllmc8GQdsUtAbLcCWYBde9vAzceEj4dzfB7xa2SIfYLKh3ewgkBOrRFHt+oF6L3qH1Zjb3EzJY3n6MIHkt2Gu5p9YNlsMiTPxTph+n+kKom fggJB6gE L4+MZs9IB3ZffRFLmSJfoHAqT1iZ23xW/N0bvkIQt9zby/okQCwHq2iVcTEAMOC96dG8YI9njTP8ZABbJL+iOhcCK8vS6RcUYzLTavHonVZhqJu9q/8RfdBIwH+CwWco8GVbHSs/FBV5j9j1oQxWylBpn4Xa3dVAacBZcooDNcCm/AEvMD+Lp7IJ8321Q6L1LLKyQZrzu/PmBKqtPbGqMKNvxgD3uaKKCg4lvR7g7uS3xUOjSgln1w0t5/l+tjrPADx9RseFQPLA+8DU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is the short term solutiolns "swap cluster order" listed in my "Swap Abstraction" discussion slice 8 in the recent LSF/MM conference. When commit 845982eb264bc "mm: swap: allow storage of all mTHP orders" is introduced, it only allocates the mTHP swap entries from new empty cluster list. It has a fragmentation issue reported by Barry. https://lore.kernel.org/all/CAGsJ_4zAcJkuW016Cfi6wicRr8N9X+GJJhgMQdSMp+Ah+NSgNQ@mail.gmail.com/ The reason is that all the empty cluster has been exhausted while there are planty of free swap entries to in the cluster that is not 100% free. Remember the swap allocation order in the cluster. Keep track of the per order non full cluster list for later allocation. User impact: For users that allocate and free mix order mTHP swapping, It greatly improves the success rate of the mTHP swap allocation after the initial phase. Barry provides a test program to show the effect: https://lore.kernel.org/linux-mm/20240615084714.37499-1-21cnbao@gmail.com/ Without: $ mthp-swapout Iteration 1: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 110, swpout fallback inc: 117, Fallback percentage: 51.54% Iteration 6: swpout inc: 0, swpout fallback inc: 230, Fallback percentage: 100.00% Iteration 7: swpout inc: 0, swpout fallback inc: 229, Fallback percentage: 100.00% Iteration 8: swpout inc: 0, swpout fallback inc: 223, Fallback percentage: 100.00% Iteration 9: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00% Iteration 10: swpout inc: 0, swpout fallback inc: 216, Fallback percentage: 100.00% Iteration 11: swpout inc: 0, swpout fallback inc: 212, Fallback percentage: 100.00% Iteration 12: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00% Iteration 13: swpout inc: 0, swpout fallback inc: 214, Fallback percentage: 100.00% $ mthp-swapout -s Iteration 1: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 33, swpout fallback inc: 197, Fallback percentage: 85.65% Iteration 6: swpout inc: 0, swpout fallback inc: 229, Fallback percentage: 100.00% Iteration 7: swpout inc: 0, swpout fallback inc: 223, Fallback percentage: 100.00% Iteration 8: swpout inc: 0, swpout fallback inc: 219, Fallback percentage: 100.00% Iteration 9: swpout inc: 0, swpout fallback inc: 212, Fallback percentage: 100.00% With: $ mthp-swapout Iteration 1: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 6: swpout inc: 230, swpout fallback inc: 0, Fallback percentage: 0.00% ... Iteration 94: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 95: swpout inc: 221, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 96: swpout inc: 229, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 97: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 98: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 99: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 100: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00% $ mthp-swapout -s Iteration 1: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 2: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 3: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 4: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 5: swpout inc: 230, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 6: swpout inc: 229, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 7: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 8: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00% ... Iteration 94: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 95: swpout inc: 212, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 96: swpout inc: 220, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 97: swpout inc: 220, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 98: swpout inc: 216, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 99: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00% Iteration 100: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00% Reported-by: Barry Song <21cnbao@gmail.com> Signed-off-by: Chris Li --- Changes in v3: - Using V1 as base. - Rename "next" to "list" for the list field, suggested by Ying. - Update comment for the locking rules for cluster fields and list, suggested by Ying. - Allocate from the nonfull list before attempting free list, suggested by Kairui. - Link to v2: https://lore.kernel.org/r/20240614-swap-allocator-v2-0-2a513b4a7f2f@kernel.org Changes in v2: - Abandoned. - Link to v1: https://lore.kernel.org/r/20240524-swap-allocator-v1-0-47861b423b26@kernel.org --- Chris Li (2): mm: swap: swap cluster switch to double link list mm: swap: mTHP allocate swap entries from nonfull list include/linux/swap.h | 30 +++---- mm/swapfile.c | 248 +++++++++++++++++---------------------------------- 2 files changed, 95 insertions(+), 183 deletions(-) --- base-commit: 19b8422c5bd56fb5e7085995801c6543a98bda1f change-id: 20240523-swap-allocator-1534c480ece4 Best regards,