From patchwork Thu Jul 11 02:13:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13729932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11D45C3DA41 for ; Thu, 11 Jul 2024 02:13:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59B966B0096; Wed, 10 Jul 2024 22:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FD936B0098; Wed, 10 Jul 2024 22:13:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8766B0099; Wed, 10 Jul 2024 22:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0DDBB6B0096 for ; Wed, 10 Jul 2024 22:13:33 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BD91C1202A6 for ; Thu, 11 Jul 2024 02:13:32 +0000 (UTC) X-FDA: 82325850264.13.B57CD00 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 09FF0C000C for ; Thu, 11 Jul 2024 02:13:30 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Wb/uPt/U"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3yj-PZgYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yj-PZgYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720663984; a=rsa-sha256; cv=none; b=Wk/skJ33Pa1svY3yaI1ROxGHrDNqFm+8G+9Gi3JI/odUVXcxuUvHEuFb7p9BTPMgTOkU/Q fNgi/huYKiIfOWUk9yfYmLcdVLjQZb6PGJ2JdfSlSpuajmkvFYYbd8hT7T1kkzC30Bhv7p x6Mnlr6wpCZhew2LtVLVjbKbUyp6g8A= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Wb/uPt/U"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3yj-PZgYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yj-PZgYKCME516ohvnvvnsl.jvtspu14-ttr2hjr.vyn@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720663984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WhNUH7SAP/MNFXjYcDX49dOz4gKNgHUZEXTQwt2QQyA=; b=eoc0mhOPKSKk2yaDLZB5EZ/JL8EQZ+L+hl9VkjKMlryL7LOD+J3njQEEBMxFSokbIS7/uy kg/aZH94mos3b4oYnjewAQAWSvFNUNI6Oo0XfBQH9DOWPyUGpIUG8SJXWmv/H3EtVaTNTI eaX3wewQi7wTBgQ9sJXfSg07tqg83F4= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e035f7b5976so889977276.0 for ; Wed, 10 Jul 2024 19:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720664010; x=1721268810; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WhNUH7SAP/MNFXjYcDX49dOz4gKNgHUZEXTQwt2QQyA=; b=Wb/uPt/UA2vib45JkW404UaHaR+kxMwYFXdpJ8EqSNGs2ApWoM4lDUYLzxsFWyuhFS drMUoM4+eleHOtlIyEE9+vwg0+sJ1QyKTf1zRv0RlWw86NzYpABGvGqyldm0Z9MZjT6r M/sJV0uhKcxr0kKMy+ak5of1yEQDaBTSjJ20JNOs0Mh12aB5fDcDLyRGU8b1l4znsdbi 7p+5CThjfIjDRE9uuEMtGquoXBf+AMMnwjWMHVxshyFKH2HlOAmU/1VSmISa5jNR6F7K SDgjHRLE/fpPwxnCeyZ2QTQMPi2u+lFXnSRczvSkGieCbLX7PmwnpgQqMYCdoDDmFOE3 nwYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720664010; x=1721268810; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WhNUH7SAP/MNFXjYcDX49dOz4gKNgHUZEXTQwt2QQyA=; b=E3sETzz1GYJM7cl9fLC78l95rvFx2LF8Gpw2RdonLaqwlOLzg0FgHkEdu7eNnoGkxu SKj6uCZN6ubvIQu96FaaMC0fkEjbXJYkCRC+jg6r96kcAFkkH7kEHFrEvRcLnSYYYOFj KCkf70gNQsvGD1HKqk0dcO7MxhPChZNbEoLJBy7IxZg7BiyGUNrtScakPDSIg+oodOVf /YE69kJZJ4nhUiAmnBEH3mAHeY6s0Il0wyjNz/SMe+VeTliMRb+zxRt7wHBrjPLMtx88 AlJjjaYJtBeFn9maLeovNPUj5IzuJJGs7UwRDBMuprOKVuCbsmuuh+OcZLcHmd5ms8xZ Fvbg== X-Gm-Message-State: AOJu0YwuCDi91XqsSGEQAafFwuewjd0jliaTsRBWYKyf7n6ypr817ArR b5C0LB1yem+fPfezSwBA1GgCA+ABzyPaVaooa9kxsnJyhtdmdF/5+6pBtxapn5QRYO+PJhjXWqG MDw== X-Google-Smtp-Source: AGHT+IGkIwS4bVyfM5ByoYpmZQq2iqjOHx1IN9uZ+1eS+2/SoQ8AfYNeitOzBddCLqNu0G2MAu021nNjM8U= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:9b06:2f28:6675:a98b]) (user=yuzhao job=sendgmr) by 2002:a05:6902:32a6:b0:e03:61bb:6032 with SMTP id 3f1490d57ef6-e0577fa164amr20670276.1.1720664010014; Wed, 10 Jul 2024 19:13:30 -0700 (PDT) Date: Wed, 10 Jul 2024 20:13:15 -0600 In-Reply-To: <20240711021317.596178-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240711021317.596178-1-yuzhao@google.com> X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240711021317.596178-4-yuzhao@google.com> Subject: [PATCH mm-unstable v1 3/5] mm/swap: fold lru_rotate into cpu_fbatches From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 09FF0C000C X-Stat-Signature: qs8kkpsud1181k98u47n88r4ymt6fosr X-Rspam-User: X-HE-Tag: 1720664010-539159 X-HE-Meta: U2FsdGVkX1+u7LSGO9lxcUi6u+hdBqxEtG7i55Gn1k6y3VKEYbg6l6Snht+QOLtKipMlfZnYeg0cRmLKEB66vvRrJ6JSYSHVafnuQThDDjuHL13VDwan2uDKrn595DbIoFl5tpMKN84XeKE1j6W2BTEOgPfkt3koSb0zcEE0WFoew/QNSTAdpE38etdMtLMuU+MtDwcWSCOsjeHYmsKvoLcYI0VIGj7kBFCiDvTu8XUJ3D2VcrjMfDMEHbxPtbpdI+2T4Ge/yS4QpXizDKc20+sFtLATf9BziT82Cq6yeBIqoVB0MkR+LnEJcASL8juEfOi01QUpxf7RrYSWmWXMK2xhTCpomHHZWx1seJMoaNmBnzS6IjKRTUEPrGIYUUomAWKV//lrlFrFxAMCPhS/OLIMePaMw4lMyb6hB1nAfwDa3YeCxjKMLE1xHdfDesAX9U48qWkNX2cOZhBT5etXI7eGeMj0uOTZeyB3RsU5KgcbbpmfHFF1hTMZ+9zDXr9oTyB9XuxcU795903dCeiMySuTnjREZ5pP3rW5xyqjuy4+eoA/LK0EHggrUWBkIUMNRl9lSGKTnVdZz0h+tmn/hqzJygtskjw1LYfBss/ybmgE3fM9+0Rsrg9wew9V+G57Z3iwpmCeMsmixXv23qFmy+W81UciMHmR5n9HNbp8RGrYPcf2HB3qt97lK234gAXLrMEkQvxv9m2rdH9Vp7dGvWXp8p+IBVbk5E506RzN1Rs/tYxkDqaUCnz2AATwuFhxAN1UDqThtVe4hOS8OoBpFmcHflXLqaIj/gqltN3ge+M+GS+96RarGGyerP+4DbpgaungGcNL9W3NmBZ+giEM9efIIM3Fmj8mZMFEvkb/Q9IVNM2AdaZVG+GiUxS4GYMZhUSNI8FRBG/8j2oElICXn4mQZQqow6yjdesnNnwl3v7EPQfUNYe2jdiKWQsVY4YqVCaJvPZx+nUAFA9sKk6 UFY7N+CA ddBov9XYqvr93sPrVgK/W/BaUi0WeCDPhWsUPk6ub8jq0qjj1RNLYJpatuAM47aOtLIvxeYzMWdUoH3pY5jOniSzD/fhF/PJdeyrXF7W5R8vZEpCXFR/9dTw8/8DX1nY13Svt++LqcU2ODO8AHCPZ/+0NgzEUpCa6b4LSde6pz8H6HvckumetbJQOFSU67g6Nv4Ug2bAGwhNZKnbdeb0hAnRo/L8WA8cbgSDsWIMFHd1e3XoWBhSP8OZUm2L/VqMkqvSZdJ8CbVEZZX3vOSp+9DRcvoNLIqk0kbQz1QQFqzYm7FMSMwloiBRicqbhZKjsvmcuTiWMw9VINHuYnMxU9ucaThE3PXq56teQ1DBcXzRbw3DSr0IB0BoUgdvg8nGZ79MkM90WwLLiWHGpl39gdZudlyK0dyXJGEavCKZtBHazcEoDsWkeCV4ZRnKPG8f9QYUoYATAVQLsQAo8I0XyY9SrIrnrkw4BU4ESDElpxS4fe0y/ownw8BvvLTkOVD3DDB8GSlgo/r/qFyas1pQgrTM/mHDYW/21ZEPAskIm+IrH53yLGWJDHRAKnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Fold lru_rotate into cpu_fbatches, and rename the folio_batch and the lock protecting it to lru_move_tail and lock_irq respectively so that all the boilerplate can be removed at the end of this series. Also remove data_race() around folio_batch_count(), which is out of place: all folio_batch_count() calls on remote cpu_fbatches are subject to data_race(), and therefore data_race() should be inside folio_batch_count(). Signed-off-by: Yu Zhao --- mm/swap.c | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index e4745b88a964..774ae9eab1e6 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -47,20 +47,11 @@ int page_cluster; const int page_cluster_max = 31; -/* Protecting only lru_rotate.fbatch which requires disabling interrupts */ -struct lru_rotate { - local_lock_t lock; - struct folio_batch fbatch; -}; -static DEFINE_PER_CPU(struct lru_rotate, lru_rotate) = { - .lock = INIT_LOCAL_LOCK(lock), -}; - -/* - * The following folio batches are grouped together because they are protected - * by disabling preemption (and interrupts remain enabled). - */ struct cpu_fbatches { + /* + * The following folio batches are grouped together because they are protected + * by disabling preemption (and interrupts remain enabled). + */ local_lock_t lock; struct folio_batch lru_add; struct folio_batch lru_deactivate_file; @@ -69,9 +60,14 @@ struct cpu_fbatches { #ifdef CONFIG_SMP struct folio_batch lru_activate; #endif + /* Protecting the following batches which require disabling interrupts */ + local_lock_t lock_irq; + struct folio_batch lru_move_tail; }; + static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), + .lock_irq = INIT_LOCAL_LOCK(lock_irq), }; static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, @@ -267,10 +263,10 @@ void folio_rotate_reclaimable(struct folio *folio) return; } - local_lock_irqsave(&lru_rotate.lock, flags); - fbatch = this_cpu_ptr(&lru_rotate.fbatch); + local_lock_irqsave(&cpu_fbatches.lock_irq, flags); + fbatch = this_cpu_ptr(&cpu_fbatches.lru_move_tail); folio_batch_add_and_move(fbatch, folio, lru_move_tail_fn); - local_unlock_irqrestore(&lru_rotate.lock, flags); + local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags); } void lru_note_cost(struct lruvec *lruvec, bool file, @@ -668,15 +664,15 @@ void lru_add_drain_cpu(int cpu) if (folio_batch_count(fbatch)) folio_batch_move_lru(fbatch, lru_add_fn); - fbatch = &per_cpu(lru_rotate.fbatch, cpu); + fbatch = &fbatches->lru_move_tail; /* Disabling interrupts below acts as a compiler barrier. */ if (data_race(folio_batch_count(fbatch))) { unsigned long flags; /* No harm done if a racing interrupt already did this */ - local_lock_irqsave(&lru_rotate.lock, flags); + local_lock_irqsave(&cpu_fbatches.lock_irq, flags); folio_batch_move_lru(fbatch, lru_move_tail_fn); - local_unlock_irqrestore(&lru_rotate.lock, flags); + local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags); } fbatch = &fbatches->lru_deactivate_file; @@ -825,7 +821,7 @@ static bool cpu_needs_drain(unsigned int cpu) /* Check these in order of likelihood that they're not zero */ return folio_batch_count(&fbatches->lru_add) || - data_race(folio_batch_count(&per_cpu(lru_rotate.fbatch, cpu))) || + folio_batch_count(&fbatches->lru_move_tail) || folio_batch_count(&fbatches->lru_deactivate_file) || folio_batch_count(&fbatches->lru_deactivate) || folio_batch_count(&fbatches->lru_lazyfree) ||