From patchwork Mon Jun 3 21:07:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E3D714E5 for ; Mon, 3 Jun 2019 21:08:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F4EC2807B for ; Mon, 3 Jun 2019 21:08:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 738882867C; Mon, 3 Jun 2019 21:08:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F40612807B for ; Mon, 3 Jun 2019 21:08:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B9696B0270; Mon, 3 Jun 2019 17:08:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 142BC6B0271; Mon, 3 Jun 2019 17:08:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02F156B0272; Mon, 3 Jun 2019 17:08:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id BECD26B0270 for ; Mon, 3 Jun 2019 17:08:26 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id 145so1262886pfv.18 for ; Mon, 03 Jun 2019 14:08:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=0EoeyvI+6nSZVdPsXoo6A3bLjv7jCmnakg13/Vo0S7g=; b=eeEw2RcJHS2e+djt/AFU9JQw3EnH2GUSXUIFCys6HgEOJ8INAOK++77IdgIYBk5MJz R7sF+HuBTXVyQSLGStrFG0JlGdWIzKFnzArsJNXKH3onBdTzj5jxUOwX9Sj9DaaLKkVD o5eUUSrnJ3nL/T99unU7WiY2M2WgMNHSLPxyNaC7FZACt0xgWD4wPH4FscFxsPPHXOad XPHKOHLGwilTMMGggIodQHxcTL6Jz41zGWFKWpzuwsLg1JzkP2D4leBgmm+sMVvUbQjH W/uo7L+qCTukSCPiJ837Z567m6DzJS3o/vp3nbOwXo4q+kEdowOLkD01u+plV/hb+1x/ z7TQ== X-Gm-Message-State: APjAAAVrp2sr2GLhqhghsxA2Byt27N+x8pT2ewCpXhf2Lgxmn4qaMqRk zPFfKH2DpBEoJ/p6kzUveDzkmClVA7rusUB0X/NP6YxTpmmstP1wMjV2L3cXpicO7pc2j1GUmoK xpURj1KWOI6/tRUtU012owtjylY2xI7q7liD66JwIK/WA/BSDAhttcKQX7kydzAsVZw== X-Received: by 2002:a17:90a:ac11:: with SMTP id o17mr24365621pjq.134.1559596106298; Mon, 03 Jun 2019 14:08:26 -0700 (PDT) X-Received: by 2002:a17:90a:ac11:: with SMTP id o17mr24365464pjq.134.1559596104842; Mon, 03 Jun 2019 14:08:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596104; cv=none; d=google.com; s=arc-20160816; b=xwM12bB9+Fxmx9N1tZdclczVxm0nJ/J7OZjS7lmtfKzMNF6vllVUL9uHCVx/MIsTE5 zaWD3aZN1tqunlQsQR9NTPDFxjUmngmVmuxzmFBx1VSQ7tzrf7Hj9VhPkP3POROftfmq 6Z4nipdfvvsBITNyURHC3OvupccE6e1Dts5Stb+lGpLUWemxN25ZRaU9wQa5st1ByiKc kgAARsq9NcLVOJgQWyMYy6j1RIaIGdE8FlxI6yHYqqHQbAQIQ2uznLImb0iwrXkOG7iL GklktQ9NpQvkaDKmhd5HYfrmHC20FR0z2Rql8B2jFQaXiktELPUEMDEjBXsE185f3fdp /2FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=0EoeyvI+6nSZVdPsXoo6A3bLjv7jCmnakg13/Vo0S7g=; b=T0MFE6QsVHxNyHAN03j+LNnQ4ifNRCeFZdJi+DQcqDxmbe8CZpFDGD0WLAw6nGwA0X Ixblukq6y7aZDqAnsInn6uuOZ78YnJpz2XRosyMWcLuKYzUDS08zmnynz6GPddfunpBB gr83/UHKDEcJuvi5ahWx8ynfMMdmALeDstpstUtgILgBjhyvifQVeQmAAHM5PFKA76G6 ZShgMTzORzF8Ie8Xfg/1kT35XLoUT0RYVQtyL1wk1/rfgASFWQHSfwkUtrnv5TZCJc9q 6RDUEWgbE28DKVfBLC3K/qNTjOwpyHpX9QKLME3r/X37Uhy/9AnOtX04rvjWPsMhTVDY XLEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=jQQKpymP; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v13sor16041698pgs.55.2019.06.03.14.08.24 for (Google Transport Security); Mon, 03 Jun 2019 14:08:24 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=jQQKpymP; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0EoeyvI+6nSZVdPsXoo6A3bLjv7jCmnakg13/Vo0S7g=; b=jQQKpymP0x+HXM6AidEDpl+oryXhrFlsULaSJxrk+bRuLIK2zph32NK7fQ9YAJaCBe N7+enaR2Vc1oYLo8aptLEfjl+dIsZ+yBuVWMXGIzNgNQI9SdCmD/jLhITwBQ9ehLSkb+ XEVP/h3PMZVC5vzDLRMA2ctacgD5ioiLhXBcIfjUB5IUWXVcT3hHScDrk9eh/9mNvu5o PWXDbM+Um+WQ9hcOjSXq+jo3FDIH6cYOI+MipuzXRWJvoaNI+qxII2acSgoctNl9nHHf 375AuTgxJjHK4jIuhHyjGLr4uaIo2/oQ034Ox+mp98rzfJnHVOkExpOIKkw9vVPM1s+r yyag== X-Google-Smtp-Source: APXvYqxa0AysWp131gGXPRfGsUoOrmmAU0oaOx10qyqfliSX/gy0fcowTdd1qaAaihU44ugQyNlEYQ== X-Received: by 2002:a63:a34c:: with SMTP id v12mr30337699pgn.198.1559596104078; Mon, 03 Jun 2019 14:08:24 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id n32sm7753279pji.29.2019.06.03.14.08.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:23 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 01/11] mm: vmscan: move inactive_list_is_low() swap check to the caller Date: Mon, 3 Jun 2019 17:07:36 -0400 Message-Id: <20190603210746.15800-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP inactive_list_is_low() should be about one thing: checking the ratio between inactive and active list. Kitchensink checks like the one for swap space makes the function hard to use and modify its callsites. Luckly, most callers already have an understanding of the swap situation, so it's easy to clean up. get_scan_count() has its own, memcg-aware swap check, and doesn't even get to the inactive_list_is_low() check on the anon list when there is no swap space available. shrink_list() is called on the results of get_scan_count(), so that check is redundant too. age_active_anon() has its own totalswap_pages check right before it checks the list proportions. The shrink_node_memcg() site is the only one that doesn't do its own swap check. Add it there. Then delete the swap check from inactive_list_is_low(). Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 84dcb651d05c..f396424850aa 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2165,13 +2165,6 @@ static bool inactive_list_is_low(struct lruvec *lruvec, bool file, unsigned long refaults; unsigned long gb; - /* - * If we don't have swap space, anonymous page deactivation - * is pointless. - */ - if (!file && !total_swap_pages) - return false; - inactive = lruvec_lru_size(lruvec, inactive_lru, sc->reclaim_idx); active = lruvec_lru_size(lruvec, active_lru, sc->reclaim_idx); @@ -2592,7 +2585,7 @@ static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memc * Even if we did not try to evict anon pages at all, we want to * rebalance the anon lru active/inactive ratio. */ - if (inactive_list_is_low(lruvec, false, sc, true)) + if (total_swap_pages && inactive_list_is_low(lruvec, false, sc, true)) shrink_active_list(SWAP_CLUSTER_MAX, lruvec, sc, LRU_ACTIVE_ANON); } From patchwork Mon Jun 3 21:07:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE80D14E5 for ; Mon, 3 Jun 2019 21:08:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD9F52807B for ; Mon, 3 Jun 2019 21:08:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A0F3A2867C; Mon, 3 Jun 2019 21:08:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C31702807B for ; Mon, 3 Jun 2019 21:08:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A6106B0269; Mon, 3 Jun 2019 17:08:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 22EC46B0271; Mon, 3 Jun 2019 17:08:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A5686B0272; Mon, 3 Jun 2019 17:08:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id BBCA66B0269 for ; Mon, 3 Jun 2019 17:08:29 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id o184so3890483pfg.1 for ; Mon, 03 Jun 2019 14:08:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=z23PZhEFw+QhEt/mQOsZQQYtYHRN5d+bP7DeTd5XrzA=; b=MBj/xrLvM0pny7ZhDbkqAH8NnUkxFi3UOIuWFOTuhPeQN9SEkgxLJMKwitGQafPUBt hmzFNQZMjS7RU6GtC2hQBS5hVZ0XlRr9/9YqVMRWUyf0PhOISr+xKNFclrp0yMxg5k1V 4goPfoyTYdFMGMUfbgaL79zetdsnNGJABvxYDTa+XPp3T/Et5E34qmFiYiX3GuvluD1n GxgxFGG+pQ6aC/Pw2w5C4pZ8RaiQXMgL1QBQ3oOROAIxoeRqp2pfprPhX/TgMZnSFK39 LTI4fK8V1NmrYvJcSGVIThgA1bYvRoOWnEvGsl0e3tR/5O/s5gITtnwEEtT/GBEAUwvl AczQ== X-Gm-Message-State: APjAAAUBHL6rUR2V7K8Sq8OTD0v81fDeVJg+jEGK9++qDvK2nR1HdcIs ZZsDddS8q9wpADPjzpPlUITyFwDDjCiedywNAg02OuawKH8pxobmF1W4DkofPLuy+Guu/csxa6F zkzghIsws1Kr3ccWndBnXMwoTNLXpcpObdVXbTmIuCODtZf0ytayOvRLLrRh5M2mKgQ== X-Received: by 2002:a63:4147:: with SMTP id o68mr31264211pga.76.1559596109150; Mon, 03 Jun 2019 14:08:29 -0700 (PDT) X-Received: by 2002:a63:4147:: with SMTP id o68mr31264050pga.76.1559596107640; Mon, 03 Jun 2019 14:08:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596107; cv=none; d=google.com; s=arc-20160816; b=0h/XknehBVEGkgE1xBDlkZETzZeolmaGXN9is17mWCj1QoZJ/Yk3EPGj2025CQnvG/ UAmajrcymPg/bo4LW+pIO2HXhQQV0GcSNUVM0rbLT6xbvDD2/eX6Py+MRH5GMbDG0wAJ erYsjQiarXcnpwKX+f7fyChJEV8LgX67+lZrxQ1mfN//vbln1b8MkTt5Lg7/5KigYpGt W//eTSSQTuWIELHayZ/1Evw1jxkOR8ZnVh5RMxswGcx6MMqRfsGKA1/dXWFhpm51v+EE uY3TS4yWTE+evengZY2/4HvAC+L2AUZYCeczWXkjIvhUYFSIKR1sU2mKeKnEPqVY7RAy TpVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=z23PZhEFw+QhEt/mQOsZQQYtYHRN5d+bP7DeTd5XrzA=; b=kf9wYsyuh9IqgmG8nz2ngJ2uS1D+7M9Q1jqC+HfrxjLc85kppFxNQApf24w47HEGhF wETPEGbKNBnLfLYjvvpSsPO252KLAUAX94kLYYdU2Oh1kBSRmyBKd3c0uMiatal2IfAq JC6KmVFmy2jqIwHGbPfcME7plNlZG0GwaIDNFbr+2l7DAYr6MjNQRfBLtPcnRSu13FXx U1XMwpnTurPR0Jmr0wFsWfS/ccHEi5blWh+CfvtJ2mOYvLXsALzcJHjNGe8JYpfV8N7T w4IPB2mRkONLTXJw0viPY9gS1WkPkDGG6H4AJTNHYacK8xQ6sUj0ebdzf0S949+E3xaI PAiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=DVH2GCKN; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h8sor3564675pgv.31.2019.06.03.14.08.27 for (Google Transport Security); Mon, 03 Jun 2019 14:08:27 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=DVH2GCKN; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z23PZhEFw+QhEt/mQOsZQQYtYHRN5d+bP7DeTd5XrzA=; b=DVH2GCKNrNgADjceKBBm6o8NfcLS2rdp3xbn242c7DdynX2bTZq4lklw8oXnSFQnsP NGyQM9FAa2/b8444R5NSB4X4p380FKi//ZWqYSF8qzZaECNMNRpbAybJZ2BM1iWaV+s3 3eFr0kg1fA8ronXzk9Pr181m/Ny9gvc2kpgw9erAMN5iYNk2qxPccfp3hm+epjm3HBXt 1bbGpvoMnBQUqho/8UxBBQGfussO1/i8lX0lnXrCrpNWBTzZQjUao2PJ+zbEwfsPvsbW URn6vyjMORUJY2XLzMBMrP5keu32Tll/EFKeFSibqGMVbJAeC06fgohHiwrA/2dgpz5v 4obA== X-Google-Smtp-Source: APXvYqwW2xQ5NzacN/EZXV09Xd6hmdDoAcI7Jqj6YbOTZQDIG92ILIAFvZ/468Og785TyGk0vKofnA== X-Received: by 2002:a63:1a5e:: with SMTP id a30mr27923212pgm.433.1559596106992; Mon, 03 Jun 2019 14:08:26 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id j15sm18799263pfn.187.2019.06.03.14.08.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:25 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 02/11] mm: clean up and clarify lruvec lookup procedure Date: Mon, 3 Jun 2019 17:07:37 -0400 Message-Id: <20190603210746.15800-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There is a per-memcg lruvec and a NUMA node lruvec. Which one is being used is somewhat confusing right now, and it's easy to make mistakes - especially when it comes to global reclaim. How it works: when memory cgroups are enabled, we always use the root_mem_cgroup's per-node lruvecs. When memory cgroups are not compiled in or disabled at runtime, we use pgdat->lruvec. Document that in a comment. Due to the way the reclaim code is generalized, all lookups use the mem_cgroup_lruvec() helper function, and nobody should have to find the right lruvec manually right now. But to avoid future mistakes, rename the pgdat->lruvec member to pgdat->__lruvec and delete the convenience wrapper that suggests it's a commonly accessed member. While in this area, swap the mem_cgroup_lruvec() argument order. The name suggests a memcg operation, yet it takes a pgdat first and a memcg second. I have to double take every time I call this. Fix that. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 26 +++++++++++++------------- include/linux/mmzone.h | 15 ++++++++------- mm/memcontrol.c | 6 +++--- mm/page_alloc.c | 2 +- mm/vmscan.c | 6 +++--- mm/workingset.c | 8 ++++---- 6 files changed, 32 insertions(+), 31 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index fa1e8cb1b3e2..fc32cfaebf32 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -382,22 +382,22 @@ mem_cgroup_nodeinfo(struct mem_cgroup *memcg, int nid) } /** - * mem_cgroup_lruvec - get the lru list vector for a node or a memcg zone - * @node: node of the wanted lruvec + * mem_cgroup_lruvec - get the lru list vector for a memcg & node * @memcg: memcg of the wanted lruvec + * @node: node of the wanted lruvec * - * Returns the lru list vector holding pages for a given @node or a given - * @memcg and @zone. This can be the node lruvec, if the memory controller - * is disabled. + * Returns the lru list vector holding pages for a given @memcg & + * @node combination. This can be the node lruvec, if the memory + * controller is disabled. */ -static inline struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat, - struct mem_cgroup *memcg) +static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, + struct pglist_data *pgdat) { struct mem_cgroup_per_node *mz; struct lruvec *lruvec; if (mem_cgroup_disabled()) { - lruvec = node_lruvec(pgdat); + lruvec = &pgdat->__lruvec; goto out; } @@ -716,7 +716,7 @@ static inline void __mod_lruvec_page_state(struct page *page, return; } - lruvec = mem_cgroup_lruvec(pgdat, page->mem_cgroup); + lruvec = mem_cgroup_lruvec(page->mem_cgroup, pgdat); __mod_lruvec_state(lruvec, idx, val); } @@ -887,16 +887,16 @@ static inline void mem_cgroup_migrate(struct page *old, struct page *new) { } -static inline struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat, - struct mem_cgroup *memcg) +static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, + struct pglist_data *pgdat) { - return node_lruvec(pgdat); + return &pgdat->__lruvec; } static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgdat) { - return &pgdat->lruvec; + return &pgdat->__lruvec; } static inline bool mm_match_cgroup(struct mm_struct *mm, diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 427b79c39b3c..95d63a395f40 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -761,7 +761,13 @@ typedef struct pglist_data { #endif /* Fields commonly accessed by the page reclaim scanner */ - struct lruvec lruvec; + + /* + * NOTE: THIS IS UNUSED IF MEMCG IS ENABLED. + * + * Use mem_cgroup_lruvec() to look up lruvecs. + */ + struct lruvec __lruvec; unsigned long flags; @@ -784,11 +790,6 @@ typedef struct pglist_data { #define node_start_pfn(nid) (NODE_DATA(nid)->node_start_pfn) #define node_end_pfn(nid) pgdat_end_pfn(NODE_DATA(nid)) -static inline struct lruvec *node_lruvec(struct pglist_data *pgdat) -{ - return &pgdat->lruvec; -} - static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat) { return pgdat->node_start_pfn + pgdat->node_spanned_pages; @@ -826,7 +827,7 @@ static inline struct pglist_data *lruvec_pgdat(struct lruvec *lruvec) #ifdef CONFIG_MEMCG return lruvec->pgdat; #else - return container_of(lruvec, struct pglist_data, lruvec); + return container_of(lruvec, struct pglist_data, __lruvec); #endif } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c193aef3ba9e..6de8ca735ee2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1200,7 +1200,7 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd struct lruvec *lruvec; if (mem_cgroup_disabled()) { - lruvec = &pgdat->lruvec; + lruvec = &pgdat->__lruvec; goto out; } @@ -1518,7 +1518,7 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, static bool test_mem_cgroup_node_reclaimable(struct mem_cgroup *memcg, int nid, bool noswap) { - struct lruvec *lruvec = mem_cgroup_lruvec(NODE_DATA(nid), memcg); + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); if (lruvec_page_state(lruvec, NR_INACTIVE_FILE) || lruvec_page_state(lruvec, NR_ACTIVE_FILE)) @@ -3406,7 +3406,7 @@ static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, static unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, int nid, unsigned int lru_mask) { - struct lruvec *lruvec = mem_cgroup_lruvec(NODE_DATA(nid), memcg); + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); unsigned long nr = 0; enum lru_list lru; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a345418b548e..cd8e64e536f7 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6619,7 +6619,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) pgdat_page_ext_init(pgdat); spin_lock_init(&pgdat->lru_lock); - lruvec_init(node_lruvec(pgdat)); + lruvec_init(&pgdat->__lruvec); } static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid, diff --git a/mm/vmscan.c b/mm/vmscan.c index f396424850aa..853be16ee5e2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2477,7 +2477,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memcg, struct scan_control *sc) { - struct lruvec *lruvec = mem_cgroup_lruvec(pgdat, memcg); + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long nr[NR_LRU_LISTS]; unsigned long targets[NR_LRU_LISTS]; unsigned long nr_to_scan; @@ -2988,7 +2988,7 @@ static void snapshot_refaults(struct mem_cgroup *root_memcg, pg_data_t *pgdat) unsigned long refaults; struct lruvec *lruvec; - lruvec = mem_cgroup_lruvec(pgdat, memcg); + lruvec = mem_cgroup_lruvec(memcg, pgdat); refaults = lruvec_page_state_local(lruvec, WORKINGSET_ACTIVATE); lruvec->refaults = refaults; } while ((memcg = mem_cgroup_iter(root_memcg, memcg, NULL))); @@ -3351,7 +3351,7 @@ static void age_active_anon(struct pglist_data *pgdat, memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - struct lruvec *lruvec = mem_cgroup_lruvec(pgdat, memcg); + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); if (inactive_list_is_low(lruvec, false, sc, true)) shrink_active_list(SWAP_CLUSTER_MAX, lruvec, diff --git a/mm/workingset.c b/mm/workingset.c index e0b4edcb88c8..2aaa70bea99c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -233,7 +233,7 @@ void *workingset_eviction(struct page *page) VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(!PageLocked(page), page); - lruvec = mem_cgroup_lruvec(pgdat, memcg); + lruvec = mem_cgroup_lruvec(memcg, pgdat); eviction = atomic_long_inc_return(&lruvec->inactive_age); return pack_shadow(memcgid, pgdat, eviction, PageWorkingset(page)); } @@ -280,7 +280,7 @@ void workingset_refault(struct page *page, void *shadow) memcg = mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !memcg) goto out; - lruvec = mem_cgroup_lruvec(pgdat, memcg); + lruvec = mem_cgroup_lruvec(memcg, pgdat); refault = atomic_long_read(&lruvec->inactive_age); active_file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE, MAX_NR_ZONES); @@ -345,7 +345,7 @@ void workingset_activation(struct page *page) memcg = page_memcg_rcu(page); if (!mem_cgroup_disabled() && !memcg) goto out; - lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page)); atomic_long_inc(&lruvec->inactive_age); out: rcu_read_unlock(); @@ -428,7 +428,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, struct lruvec *lruvec; int i; - lruvec = mem_cgroup_lruvec(NODE_DATA(sc->nid), sc->memcg); + lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, NR_LRU_BASE + i); From patchwork Mon Jun 3 21:07:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973983 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C087E933 for ; Mon, 3 Jun 2019 21:08:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B18A82850F for ; Mon, 3 Jun 2019 21:08:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A527B2807B; Mon, 3 Jun 2019 21:08:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41B982807B for ; Mon, 3 Jun 2019 21:08:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C2D46B0271; Mon, 3 Jun 2019 17:08:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5FD5E6B0272; Mon, 3 Jun 2019 17:08:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 476956B0273; Mon, 3 Jun 2019 17:08:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 111436B0271 for ; Mon, 3 Jun 2019 17:08:31 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id o184so3890516pfg.1 for ; Mon, 03 Jun 2019 14:08:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hW9XEaleHf0fxFzrzaLu3/5nxBFHbI/f8wU4NVTruL4=; b=Ze2H44b4Pyg0CWjLtwsEl6XxSi0E2HchtsSihVBqgIF9Qwabg3ERbZQrdzZmye9Zg4 Ouvl4eT8LW9IiYmhl7QnIRXmaI44S964IeYPKz+tHeS40WA3fwSYe+bvSABFdDkvFcUk gdl78lnVrY4ekH3fEgWlOn2MBIbynX77f9YrWqH3LgFqtv8+klgQ86KQZbDF93eM7YBi 5/c+8/H4IGoGbDh9qMPFu9yAdDv9ITRMRE/huQqGrohHQ9M5a59m5ZycTKweNkPwNfVB iA7ra+z9YfYCj53peaooBkObuj1pfxsyLnT3ePzQhw1PMI4nbpEfE7S3ixTC6fiBPRLn UyRQ== X-Gm-Message-State: APjAAAUrsYDVhSBBw1IlF7XgCcyoLiMqOcAUmVVOn7DY5c2xFIm5gQH1 40UYiKnIr+fmB2z/ZnFW2nfJBRn2uVOVFx6Ljqc6Xx1TFszdBNQG6OM5L5Q097oM7HU8JlZl4fo +3xgl0MtV27fvzDzGQPQbVwlmyyHiVKBb5P8NKRy+6R/s4oqnIS6qpfmMbwCYSLapEg== X-Received: by 2002:a63:e408:: with SMTP id a8mr31653746pgi.146.1559596110485; Mon, 03 Jun 2019 14:08:30 -0700 (PDT) X-Received: by 2002:a63:e408:: with SMTP id a8mr31653624pgi.146.1559596109456; Mon, 03 Jun 2019 14:08:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596109; cv=none; d=google.com; s=arc-20160816; b=yWIzWiRoswgZB38DqPePLEWEjZp4GMC/xRMduHJNxD0eijk26zfVoyc0ksiZlFonGh rZ+7pfE/5gtkDko0Xv62kxtTBne7mq1wchH8kEPckNnnkrIT7ywdL7Ggus72MsxtPaEW gsOaPGUinHdD5NSzIX2HkatB6LxoiPggznj6v3sSubtvE5VCrrQ9jj2DYMSC6NUEl5qb FwUTjNkI6awqZq2j23y8rCTq1NWLybTGVPI4WSluWDqdS5Dj8LDeqvqJG/JJ08ZFou3A TdI/We8XLPah5e9GxfyhoUaVU43P76sSxNA2ZAKYjHYbVliquGkTn0GR6I/RAntB0V0X K7Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=hW9XEaleHf0fxFzrzaLu3/5nxBFHbI/f8wU4NVTruL4=; b=nR2Q6NZWElt1ZG56AOmsddEID7obZn8kZJvV3pJrR+Uxm8jV4u6MCHBc2vYBJ9evZG ajhUvTtiWs0Jn5pPqoSLsW/u7UPeK6nAedHw7+TlOs+faAfepSZVCU/c/wrVOEKlZIcv jyYf+rN6ktou6gfQm5rsOASY6G4KxWlD46jTv6BnMgocMMTXW4DvjeJmy81AwXrxGvc0 gUPQGVIxq0M/8sBxe2qCI7m7VRqFC0X7f4o2q2OIzi93ssqo7TdrdwbpV6W1SzXGlLD8 9UTKUFFCyKNpcuWz0zlqM6s1woe+tOFMwYinLNAnzwI16OPGgDuLvECcZCfkGOq+f0jt MSVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b="GiZl3I0/"; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id bx5sor18153568pjb.22.2019.06.03.14.08.29 for (Google Transport Security); Mon, 03 Jun 2019 14:08:29 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b="GiZl3I0/"; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hW9XEaleHf0fxFzrzaLu3/5nxBFHbI/f8wU4NVTruL4=; b=GiZl3I0/1nrhXFLsEYVx8i8I4at3od658KebcT8wvAtJDzTnhyqemuN29q2/fcr3KW tDsgy9TzZFg8DTfH/bgCfQ9DMl4vc7J4plUg5mA9cty9aXI8rxvDidvnbUu121RWDy3r e1RxDwqqABcmlJXwZaGPTzP0keY1G2zSdow7H3XEkR3F49wcmjR9HuHqyQb5DFvxXdrE L+ebT5LxPIX7aGWwOApYSGaixnBATTA3AvWr+WjSjHJsz0lxIbyDxEPuHK3xJkUAPctk yP5msqE7q+xLnKa1SZBpeRAnKA5wg+rgZtDOy4sVffJyMyZx1w3RUzvp12AR2PMSm3mf jkdA== X-Google-Smtp-Source: APXvYqxRKIe7ZlIxfiZR5KZpvX0iyzwEdC5f5c2JCVHHT7fr7Ha6rZsuX4a/y9QMM1RAzJK5pY+vwA== X-Received: by 2002:a17:90a:30a1:: with SMTP id h30mr33354770pjb.14.1559596109171; Mon, 03 Jun 2019 14:08:29 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id p18sm3454267pff.93.2019.06.03.14.08.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:28 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 03/11] mm: vmscan: simplify lruvec_lru_size() Date: Mon, 3 Jun 2019 17:07:38 -0400 Message-Id: <20190603210746.15800-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This function currently takes the node or lruvec size and subtracts the zones that are excluded by the classzone index of the allocation. It uses four different types of counters to do this. Just add up the eligible zones. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 853be16ee5e2..69c4c82a9b5a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -342,30 +342,21 @@ unsigned long zone_reclaimable_pages(struct zone *zone) */ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx) { - unsigned long lru_size; + unsigned long size = 0; int zid; - if (!mem_cgroup_disabled()) - lru_size = lruvec_page_state_local(lruvec, NR_LRU_BASE + lru); - else - lru_size = node_page_state(lruvec_pgdat(lruvec), NR_LRU_BASE + lru); - - for (zid = zone_idx + 1; zid < MAX_NR_ZONES; zid++) { + for (zid = 0; zid <= zone_idx; zid++) { struct zone *zone = &lruvec_pgdat(lruvec)->node_zones[zid]; - unsigned long size; if (!managed_zone(zone)) continue; if (!mem_cgroup_disabled()) - size = mem_cgroup_get_zone_lru_size(lruvec, lru, zid); + size += mem_cgroup_get_zone_lru_size(lruvec, lru, zid); else - size = zone_page_state(&lruvec_pgdat(lruvec)->node_zones[zid], - NR_ZONE_LRU_BASE + lru); - lru_size -= min(size, lru_size); + size += zone_page_state(zone, NR_ZONE_LRU_BASE + lru); } - - return lru_size; + return size; } From patchwork Mon Jun 3 21:07:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973985 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8340914E5 for ; Mon, 3 Jun 2019 21:08:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 734812807B for ; Mon, 3 Jun 2019 21:08:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 677E12870C; Mon, 3 Jun 2019 21:08:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB5DB2807B for ; Mon, 3 Jun 2019 21:08:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AAFE6B0272; Mon, 3 Jun 2019 17:08:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ED94A6B0273; Mon, 3 Jun 2019 17:08:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCABA6B0274; Mon, 3 Jun 2019 17:08:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 83B376B0272 for ; Mon, 3 Jun 2019 17:08:33 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id c4so10757830pgm.21 for ; Mon, 03 Jun 2019 14:08:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=/ECwKX3tAPVrwDHDKYOADHOjDCqITvrKBeA6eRqjWZs=; b=OeolZ0PFZIGfzdhA04/6V1hZBTKstaOBtd3jPBofhMJhW31q2NUMKqU9jfdFhEvgeZ EoxxV795m0cXC0e32xUOpDdLxVE0xK2/DNpGZrAG/Ln9x3Hlka1XKnMnGoenus0F5LJn 7C7A5QcWuCcy/WaHmtAZVZIoArCY4SM3YipDJZtY47MAnx18xbcmiQhRiwTXR7mlhxVm yS4/daEnFHQa0uDxlsRlPfgejElBOjPkFWee/ClG+4XZ11UMbFUA58wpGwkelPQB1vvV 05FSjI0qJCi0esuH7UoQWxyRT6IXMfGvDXa+uZXdWU9Rbxcw9+VTadbZjEB2E3u7NYdW WOFg== X-Gm-Message-State: APjAAAXvHwCFa1fUrBq4z1LgESW+BQDC9qapXf04sqkne+fA3V5tdMBJ FpO4vAP5SWog+6racyq9ca5C7y/NP+vO9QvKs/UayjvjAAhytEaZo8dm2+KSYtd+iS3ulDQdUtF b0hiriy/sopygvnapuyJENCeYMMciDuBqId/zO9O/aIyVG3JkfrhS06f02KtsreBXtg== X-Received: by 2002:aa7:8ece:: with SMTP id b14mr24584821pfr.244.1559596113048; Mon, 03 Jun 2019 14:08:33 -0700 (PDT) X-Received: by 2002:aa7:8ece:: with SMTP id b14mr24584693pfr.244.1559596112005; Mon, 03 Jun 2019 14:08:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596112; cv=none; d=google.com; s=arc-20160816; b=HXEWjXbgzRaw+/QKMqT33YFnNJ1mdXL01bU/p9zuxDxJpi/pIinItkDK2qsZBSO3Jv A3a6ci12tPx07q9dkirH1TRDYCyab1lXFUN6YmgQ/HPnEnmZiueXTVYMTRyg+n3JZkei Yv3hr5SL603F92xl5aktzEMfHQO0JhDy9ef5XRNlwbEOpHsb8G0t3gJE+waYSjG1cfpK YRP43CtvhpexeqsNFE2x5EIOQsoM8yLhiM2Q9f8Zxj2hUqfZsE4m/6vfFEK2t7g2WYZh 6voqGmVM/wLCdamAVqNSbkkGp5H/0F8n0LiZIy4z/uYxSS9czdzL9RXecLBLgE6WzAoj pa1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=/ECwKX3tAPVrwDHDKYOADHOjDCqITvrKBeA6eRqjWZs=; b=vpfZtuytBQr3MLaNIv6ejWUSZlTHrLhxGn4g1uWZcLvslUcr8JzNh0jANQkhMT/6tE +8LLySwg83hz3kLgrC4yMsDGqNisuD9Oy+40LUj5ynd0JFkk4VLdR2V5c6w7mMZkLkdv Lw3ETU4FeiiQIq0btLJ+h5Bqy3ZqVZi7bKL9O+3h87E+0kEkueamKkblloquYa6Ei+Nq oxpRRSXfqzDvTaHuTiT03Wriq+TwVVPZH23K4EECDJqT5lR6bIoNncl6HryMdRB5AWi/ lUmDIDIIwyD9G/cLCrgrphA05JaGWwnOWjkTeK+XfvSFaGJVXzIpqRMVLYtWPrNfjWfM 4BEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=wc1IAI5z; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m65sor18304424pfm.66.2019.06.03.14.08.31 for (Google Transport Security); Mon, 03 Jun 2019 14:08:31 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=wc1IAI5z; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/ECwKX3tAPVrwDHDKYOADHOjDCqITvrKBeA6eRqjWZs=; b=wc1IAI5zkjEeNFD/HUHIIB2H4ZEiiwVM44Ba+r/EDaRF3+H7i/JdKOmLOc7GkpUaXZ nsmUhjtAzV6Xt8bZLVEcSDL1/ZUOBooQnG9O1/17mij+ggkqxQYuN5S6vLcYdIDVLQFd HheOIOvrDxa75BK5RdfurEfDBmnxdJSk7pB/Sr4G/PRfMm3jCCWh7pWJW+C3c8Ue7voh Bk09b4PFXAcp/azMkFm0cUu0GP9FagGP0M0WlvdSyz5/KYs0QJo2xMGEH9BHt+MEwAqK fFXTKl+Ep/ZRQl2YI8RuqcPatNZBQ540LUZXo2NCHuCOvic+kaKnJ8EiOvdAO80Fzg8A C+wg== X-Google-Smtp-Source: APXvYqx/5Lrk+LdVqxZQme1ApdkTNrnq9skskkPna5IZ39YLgvkLp8SXhS67uQwguS5V8QjCLUPvmA== X-Received: by 2002:aa7:8b49:: with SMTP id i9mr6357080pfd.74.1559596111356; Mon, 03 Jun 2019 14:08:31 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id m8sm23997383pff.137.2019.06.03.14.08.30 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:30 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 04/11] mm: vmscan: naming fixes: cgroup_reclaim() and writeback_working() Date: Mon, 3 Jun 2019 17:07:39 -0400 Message-Id: <20190603210746.15800-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Seven years after introducing the global_reclaim() function, I still have to double take when reading a callsite. I don't know how others do it. This is a terrible name. Invert the meaning and rename it to cgroup_reclaim(). [ After all, "global reclaim" is just regular reclaim invoked from the page allocator. It's reclaim on behalf of a cgroup limit that is a special case of reclaim, and should be explicit - not the reverse. ] sane_reclaim() isn't very descriptive either: it tests whether we can use the regular writeback throttling - available during regular page reclaim or cgroup2 limit reclaim - or need to use the broken wait_on_page_writeback() method. Rename it to writeback_working(). Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 69c4c82a9b5a..afd5e2432a8e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -239,13 +239,13 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) #endif /* CONFIG_MEMCG_KMEM */ #ifdef CONFIG_MEMCG -static bool global_reclaim(struct scan_control *sc) +static bool cgroup_reclaim(struct scan_control *sc) { - return !sc->target_mem_cgroup; + return sc->target_mem_cgroup; } /** - * sane_reclaim - is the usual dirty throttling mechanism operational? + * writeback_working - is the usual dirty throttling mechanism unavailable? * @sc: scan_control in question * * The normal page dirty throttling mechanism in balance_dirty_pages() is @@ -257,11 +257,9 @@ static bool global_reclaim(struct scan_control *sc) * This function tests whether the vmscan currently in progress can assume * that the normal dirty throttling mechanism is operational. */ -static bool sane_reclaim(struct scan_control *sc) +static bool writeback_working(struct scan_control *sc) { - struct mem_cgroup *memcg = sc->target_mem_cgroup; - - if (!memcg) + if (!cgroup_reclaim(sc)) return true; #ifdef CONFIG_CGROUP_WRITEBACK if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) @@ -293,12 +291,12 @@ static bool memcg_congested(pg_data_t *pgdat, } #else -static bool global_reclaim(struct scan_control *sc) +static bool cgroup_reclaim(struct scan_control *sc) { - return true; + return false; } -static bool sane_reclaim(struct scan_control *sc) +static bool writeback_working(struct scan_control *sc) { return true; } @@ -1211,7 +1209,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, goto activate_locked; /* Case 2 above */ - } else if (sane_reclaim(sc) || + } else if (writeback_working(sc) || !PageReclaim(page) || !may_enter_fs) { /* * This is slightly racy - end_page_writeback() @@ -1806,7 +1804,7 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, if (current_is_kswapd()) return 0; - if (!sane_reclaim(sc)) + if (!writeback_working(sc)) return 0; if (file) { @@ -1957,7 +1955,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, reclaim_stat->recent_scanned[file] += nr_taken; item = current_is_kswapd() ? PGSCAN_KSWAPD : PGSCAN_DIRECT; - if (global_reclaim(sc)) + if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned); spin_unlock_irq(&pgdat->lru_lock); @@ -1971,7 +1969,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_lock_irq(&pgdat->lru_lock); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; - if (global_reclaim(sc)) + if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); reclaim_stat->recent_rotated[0] += stat.nr_activate[0]; @@ -2239,7 +2237,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, * using the memory controller's swap limit feature would be * too expensive. */ - if (!global_reclaim(sc) && !swappiness) { + if (cgroup_reclaim(sc) && !swappiness) { scan_balance = SCAN_FILE; goto out; } @@ -2263,7 +2261,7 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, * thrashing file LRU becomes infinitely more attractive than * anon pages. Try to detect this based on file LRU size. */ - if (global_reclaim(sc)) { + if (!cgroup_reclaim(sc)) { unsigned long pgdatfile; unsigned long pgdatfree; int z; @@ -2494,7 +2492,7 @@ static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memc * abort proportional reclaim if either the file or anon lru has already * dropped to zero at the first pass. */ - scan_adjusted = (global_reclaim(sc) && !current_is_kswapd() && + scan_adjusted = (!cgroup_reclaim(sc) && !current_is_kswapd() && sc->priority == DEF_PRIORITY); blk_start_plug(&plug); @@ -2816,7 +2814,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) * Legacy memcg will stall in page writeback so avoid forcibly * stalling in wait_iff_congested(). */ - if (!global_reclaim(sc) && sane_reclaim(sc) && + if (cgroup_reclaim(sc) && writeback_working(sc) && sc->nr.dirty && sc->nr.dirty == sc->nr.congested) set_memcg_congestion(pgdat, root, true); @@ -2911,7 +2909,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) * Take care memory controller reclaiming has small influence * to global LRU. */ - if (global_reclaim(sc)) { + if (!cgroup_reclaim(sc)) { if (!cpuset_zone_allowed(zone, GFP_KERNEL | __GFP_HARDWALL)) continue; @@ -3011,7 +3009,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist, retry: delayacct_freepages_start(); - if (global_reclaim(sc)) + if (!cgroup_reclaim(sc)) __count_zid_vm_events(ALLOCSTALL, sc->reclaim_idx, 1); do { From patchwork Mon Jun 3 21:07:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973987 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6984714E5 for ; Mon, 3 Jun 2019 21:08:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 596632807B for ; Mon, 3 Jun 2019 21:08:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4DA912867C; Mon, 3 Jun 2019 21:08:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C9612850F for ; Mon, 3 Jun 2019 21:08:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 931816B0273; Mon, 3 Jun 2019 17:08:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8949F6B0274; Mon, 3 Jun 2019 17:08:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EA9B6B0276; Mon, 3 Jun 2019 17:08:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 21E946B0273 for ; Mon, 3 Jun 2019 17:08:36 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id a13so5375050pgw.19 for ; Mon, 03 Jun 2019 14:08:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xjW+KHnHD//MqYRxY0A82usCY+qgrBu78W4ErXk80GU=; b=QlIUEm8I8cLfh6w8Xr1nNnW2FAziXTLpznapT8OFCjr1Ri9gJZubY90FZG3KCqJyQA eqvg6sISpWJ1zGSRU4ihGw/y2ep766RS2WueDgaMNJO3tXicf5W1PgJdTglEhSzQ/9Xf jhqMogywRSM73hp7zx8FZBLyM8pDDsZsJIFYPf4QKOoi1ygLcgLHHDIU8jVWH+H3gHqC czkxQDbq/nzOfc7xdmIrqJa3c+u9TYR0WL/tNVadxFDDqv25hvVleDNa72nXshkPAG7X RXSaXx5UoDnWj935GH1SdosGwduQEt+jPXxNjjN7vl/D9lvGXnmDGFd6ea6IgNiABws6 uwbQ== X-Gm-Message-State: APjAAAWJ9cHBdTNeIYAICSZoBrB5V6FDsh5WG5Ut8vGOuiWd1B35BH6w qEI4mWSIA19qboHttPyY6afGsf5qDAso9yMheBar9EZNZ4DZBsx0d3+y47Fc+BH0avTlepTUIDR z/GCNK4BYKsBTM81RMALjd4JSHEYCZ/hAGnHK8aMT/5RbMECAClHmXvgFQVzG2k7uDQ== X-Received: by 2002:a63:18e:: with SMTP id 136mr31019773pgb.277.1559596115631; Mon, 03 Jun 2019 14:08:35 -0700 (PDT) X-Received: by 2002:a63:18e:: with SMTP id 136mr31019594pgb.277.1559596113904; Mon, 03 Jun 2019 14:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596113; cv=none; d=google.com; s=arc-20160816; b=jbV8hONA3Od0R1bM+AXx5QtB2YOwxDV7OmCXkzfqSFLDvwXRqbAbLtvc0EUASNPiRD Dtw9r3bdYO95n+5iaEgA2UOlLIcD3YUNplC205AvIMh6wK/P1JfdrkBCGmWs06pyIiHr PMUPSTJSceoqil0SdF9zMVVNrAimvvBEs78Hlm5pjH6GmXYAIhpmv0HALS0Q9+aoeDJj 5kzrfQZC/wPsebWOAmiSTtRggTyjroXQB0+7ce6YisEYC1DYhP0HVw+4077Ct+aZMpoi VrcJzPw3t1D+1xt6v98aeOYJf5NOQrwaY6+QKj7MPSQAIhQmp7v+HHT9Qsbe9rraXyEF GwuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=xjW+KHnHD//MqYRxY0A82usCY+qgrBu78W4ErXk80GU=; b=Zq2p4Tnh2UNWkNLe/eocvtDh/45lPAZdvEvqiOmFnJK2lOH2mJnJs5uX8jmRC+S9Xb cTg0HFP3zSAcEjH10+ucozjY5TLjzHvLOu/7hFDqBhKVfJPaQzHFOtLEy60ghExL492h gZrCPUAsOPEbh9oFEJwWQQ6BwSjQQtB+MWMIctABxY2phTTzGuDv8YwgJ9BNS483XwgM rbbLQl0yCe1XtK6D8ZybrEcGKzKRmXHsljVKMjeZLVhXV2RXyOjsNXkPAkbIuQUxSTZk GRAG5Y+2FXjntGWwQi47OLesyb3YHqgll8FItdXXrrATuSiUraUrZjj25W2zmtgH4nLc XYmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=bc7TrWgT; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m13sor17890811pff.45.2019.06.03.14.08.33 for (Google Transport Security); Mon, 03 Jun 2019 14:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=bc7TrWgT; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xjW+KHnHD//MqYRxY0A82usCY+qgrBu78W4ErXk80GU=; b=bc7TrWgTSGZgsPvNADkSHVc76wW+ddItC6br4BoP2u6J2Pd8x17XCsWC3tGSkAaHij oDcMSFJAURwkI1EsRGyUV04wA/UtJNTcgot7wyikPKZfLvqoi6mEr5mnVocvC0sTBFhE g0b3BThoDChYC55UwB0tCBNdqej7Q0Yz1Gli7Xn8kG9PLT/OKEki2zZyQkfL6rGKUnYs IhBE+6SRBybCA4Z4ALlC/2E8iG4/Lj7Ku4+nALz8YwsvMit9wJIScjmbYbcNW8nNdZNK TPuwsI7mqRB+pFgEWLd8ALXu3vRcJfmSXTvT9yk7gHuUaX/0bTi7PuDMoT/E+qB+7sUQ gyaw== X-Google-Smtp-Source: APXvYqze2WOGrqc7zcRp11zL0Su9NKxuQeLHHGNTzfXyE/gd72jk+MrGfhvjph0MXtK8eEj/GjOljw== X-Received: by 2002:aa7:8292:: with SMTP id s18mr2491755pfm.111.1559596113541; Mon, 03 Jun 2019 14:08:33 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id w62sm2586095pfw.132.2019.06.03.14.08.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:32 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 05/11] mm: vmscan: replace shrink_node() loop with a retry jump Date: Mon, 3 Jun 2019 17:07:40 -0400 Message-Id: <20190603210746.15800-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Most of the function body is inside a loop, which imposes an additional indentation and scoping level that makes the code a bit hard to follow and modify. The looping only happens in case of reclaim-compaction, which isn't the common case. So rather than adding yet another function level to the reclaim path and have every reclaim invocation go through a level that only exists for one specific cornercase, use a retry goto. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 266 ++++++++++++++++++++++++++-------------------------- 1 file changed, 133 insertions(+), 133 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index afd5e2432a8e..304974481146 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2672,164 +2672,164 @@ static bool pgdat_memcg_congested(pg_data_t *pgdat, struct mem_cgroup *memcg) static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) { struct reclaim_state *reclaim_state = current->reclaim_state; + struct mem_cgroup *root = sc->target_mem_cgroup; + struct mem_cgroup_reclaim_cookie reclaim = { + .pgdat = pgdat, + .priority = sc->priority, + }; unsigned long nr_reclaimed, nr_scanned; bool reclaimable = false; + struct mem_cgroup *memcg; - do { - struct mem_cgroup *root = sc->target_mem_cgroup; - struct mem_cgroup_reclaim_cookie reclaim = { - .pgdat = pgdat, - .priority = sc->priority, - }; - struct mem_cgroup *memcg; - - memset(&sc->nr, 0, sizeof(sc->nr)); +again: + memset(&sc->nr, 0, sizeof(sc->nr)); - nr_reclaimed = sc->nr_reclaimed; - nr_scanned = sc->nr_scanned; + nr_reclaimed = sc->nr_reclaimed; + nr_scanned = sc->nr_scanned; - memcg = mem_cgroup_iter(root, NULL, &reclaim); - do { - unsigned long reclaimed; - unsigned long scanned; + memcg = mem_cgroup_iter(root, NULL, &reclaim); + do { + unsigned long reclaimed; + unsigned long scanned; - switch (mem_cgroup_protected(root, memcg)) { - case MEMCG_PROT_MIN: - /* - * Hard protection. - * If there is no reclaimable memory, OOM. - */ + switch (mem_cgroup_protected(root, memcg)) { + case MEMCG_PROT_MIN: + /* + * Hard protection. + * If there is no reclaimable memory, OOM. + */ + continue; + case MEMCG_PROT_LOW: + /* + * Soft protection. + * Respect the protection only as long as + * there is an unprotected supply + * of reclaimable memory from other cgroups. + */ + if (!sc->memcg_low_reclaim) { + sc->memcg_low_skipped = 1; continue; - case MEMCG_PROT_LOW: - /* - * Soft protection. - * Respect the protection only as long as - * there is an unprotected supply - * of reclaimable memory from other cgroups. - */ - if (!sc->memcg_low_reclaim) { - sc->memcg_low_skipped = 1; - continue; - } - memcg_memory_event(memcg, MEMCG_LOW); - break; - case MEMCG_PROT_NONE: - /* - * All protection thresholds breached. We may - * still choose to vary the scan pressure - * applied based on by how much the cgroup in - * question has exceeded its protection - * thresholds (see get_scan_count). - */ - break; } - - reclaimed = sc->nr_reclaimed; - scanned = sc->nr_scanned; - shrink_node_memcg(pgdat, memcg, sc); - - if (sc->may_shrinkslab) { - shrink_slab(sc->gfp_mask, pgdat->node_id, - memcg, sc->priority); - } - - /* Record the group's reclaim efficiency */ - vmpressure(sc->gfp_mask, memcg, false, - sc->nr_scanned - scanned, - sc->nr_reclaimed - reclaimed); - + memcg_memory_event(memcg, MEMCG_LOW); + break; + case MEMCG_PROT_NONE: /* - * Kswapd have to scan all memory cgroups to fulfill - * the overall scan target for the node. - * - * Limit reclaim, on the other hand, only cares about - * nr_to_reclaim pages to be reclaimed and it will - * retry with decreasing priority if one round over the - * whole hierarchy is not sufficient. + * All protection thresholds breached. We may + * still choose to vary the scan pressure + * applied based on by how much the cgroup in + * question has exceeded its protection + * thresholds (see get_scan_count). */ - if (!current_is_kswapd() && - sc->nr_reclaimed >= sc->nr_to_reclaim) { - mem_cgroup_iter_break(root, memcg); - break; - } - } while ((memcg = mem_cgroup_iter(root, memcg, &reclaim))); + break; + } + + reclaimed = sc->nr_reclaimed; + scanned = sc->nr_scanned; + shrink_node_memcg(pgdat, memcg, sc); - if (reclaim_state) { - sc->nr_reclaimed += reclaim_state->reclaimed_slab; - reclaim_state->reclaimed_slab = 0; + if (sc->may_shrinkslab) { + shrink_slab(sc->gfp_mask, pgdat->node_id, + memcg, sc->priority); } - /* Record the subtree's reclaim efficiency */ - vmpressure(sc->gfp_mask, sc->target_mem_cgroup, true, - sc->nr_scanned - nr_scanned, - sc->nr_reclaimed - nr_reclaimed); + /* Record the group's reclaim efficiency */ + vmpressure(sc->gfp_mask, memcg, false, + sc->nr_scanned - scanned, + sc->nr_reclaimed - reclaimed); - if (sc->nr_reclaimed - nr_reclaimed) - reclaimable = true; + /* + * Kswapd have to scan all memory cgroups to fulfill + * the overall scan target for the node. + * + * Limit reclaim, on the other hand, only cares about + * nr_to_reclaim pages to be reclaimed and it will + * retry with decreasing priority if one round over the + * whole hierarchy is not sufficient. + */ + if (!current_is_kswapd() && + sc->nr_reclaimed >= sc->nr_to_reclaim) { + mem_cgroup_iter_break(root, memcg); + break; + } + } while ((memcg = mem_cgroup_iter(root, memcg, &reclaim))); - if (current_is_kswapd()) { - /* - * If reclaim is isolating dirty pages under writeback, - * it implies that the long-lived page allocation rate - * is exceeding the page laundering rate. Either the - * global limits are not being effective at throttling - * processes due to the page distribution throughout - * zones or there is heavy usage of a slow backing - * device. The only option is to throttle from reclaim - * context which is not ideal as there is no guarantee - * the dirtying process is throttled in the same way - * balance_dirty_pages() manages. - * - * Once a node is flagged PGDAT_WRITEBACK, kswapd will - * count the number of pages under pages flagged for - * immediate reclaim and stall if any are encountered - * in the nr_immediate check below. - */ - if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) - set_bit(PGDAT_WRITEBACK, &pgdat->flags); + if (reclaim_state) { + sc->nr_reclaimed += reclaim_state->reclaimed_slab; + reclaim_state->reclaimed_slab = 0; + } - /* - * Tag a node as congested if all the dirty pages - * scanned were backed by a congested BDI and - * wait_iff_congested will stall. - */ - if (sc->nr.dirty && sc->nr.dirty == sc->nr.congested) - set_bit(PGDAT_CONGESTED, &pgdat->flags); + /* Record the subtree's reclaim efficiency */ + vmpressure(sc->gfp_mask, sc->target_mem_cgroup, true, + sc->nr_scanned - nr_scanned, + sc->nr_reclaimed - nr_reclaimed); - /* Allow kswapd to start writing pages during reclaim.*/ - if (sc->nr.unqueued_dirty == sc->nr.file_taken) - set_bit(PGDAT_DIRTY, &pgdat->flags); + if (sc->nr_reclaimed - nr_reclaimed) + reclaimable = true; - /* - * If kswapd scans pages marked marked for immediate - * reclaim and under writeback (nr_immediate), it - * implies that pages are cycling through the LRU - * faster than they are written so also forcibly stall. - */ - if (sc->nr.immediate) - congestion_wait(BLK_RW_ASYNC, HZ/10); - } + if (current_is_kswapd()) { + /* + * If reclaim is isolating dirty pages under writeback, + * it implies that the long-lived page allocation rate + * is exceeding the page laundering rate. Either the + * global limits are not being effective at throttling + * processes due to the page distribution throughout + * zones or there is heavy usage of a slow backing + * device. The only option is to throttle from reclaim + * context which is not ideal as there is no guarantee + * the dirtying process is throttled in the same way + * balance_dirty_pages() manages. + * + * Once a node is flagged PGDAT_WRITEBACK, kswapd will + * count the number of pages under pages flagged for + * immediate reclaim and stall if any are encountered + * in the nr_immediate check below. + */ + if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) + set_bit(PGDAT_WRITEBACK, &pgdat->flags); /* - * Legacy memcg will stall in page writeback so avoid forcibly - * stalling in wait_iff_congested(). + * Tag a node as congested if all the dirty pages + * scanned were backed by a congested BDI and + * wait_iff_congested will stall. */ - if (cgroup_reclaim(sc) && writeback_working(sc) && - sc->nr.dirty && sc->nr.dirty == sc->nr.congested) - set_memcg_congestion(pgdat, root, true); + if (sc->nr.dirty && sc->nr.dirty == sc->nr.congested) + set_bit(PGDAT_CONGESTED, &pgdat->flags); + + /* Allow kswapd to start writing pages during reclaim.*/ + if (sc->nr.unqueued_dirty == sc->nr.file_taken) + set_bit(PGDAT_DIRTY, &pgdat->flags); /* - * Stall direct reclaim for IO completions if underlying BDIs - * and node is congested. Allow kswapd to continue until it - * starts encountering unqueued dirty pages or cycling through - * the LRU too quickly. + * If kswapd scans pages marked marked for immediate + * reclaim and under writeback (nr_immediate), it + * implies that pages are cycling through the LRU + * faster than they are written so also forcibly stall. */ - if (!sc->hibernation_mode && !current_is_kswapd() && - current_may_throttle() && pgdat_memcg_congested(pgdat, root)) - wait_iff_congested(BLK_RW_ASYNC, HZ/10); + if (sc->nr.immediate) + congestion_wait(BLK_RW_ASYNC, HZ/10); + } + + /* + * Legacy memcg will stall in page writeback so avoid forcibly + * stalling in wait_iff_congested(). + */ + if (cgroup_reclaim(sc) && writeback_working(sc) && + sc->nr.dirty && sc->nr.dirty == sc->nr.congested) + set_memcg_congestion(pgdat, root, true); + + /* + * Stall direct reclaim for IO completions if underlying BDIs + * and node is congested. Allow kswapd to continue until it + * starts encountering unqueued dirty pages or cycling through + * the LRU too quickly. + */ + if (!sc->hibernation_mode && !current_is_kswapd() && + current_may_throttle() && pgdat_memcg_congested(pgdat, root)) + wait_iff_congested(BLK_RW_ASYNC, HZ/10); - } while (should_continue_reclaim(pgdat, sc->nr_reclaimed - nr_reclaimed, - sc->nr_scanned - nr_scanned, sc)); + if (should_continue_reclaim(pgdat, sc->nr_reclaimed - nr_reclaimed, + sc->nr_scanned - nr_scanned, sc)) + goto again; /* * Kswapd gives up on balancing particular nodes after too From patchwork Mon Jun 3 21:07:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973989 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C487D933 for ; Mon, 3 Jun 2019 21:08:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B5E5F2807B for ; Mon, 3 Jun 2019 21:08:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA2CF2867C; Mon, 3 Jun 2019 21:08:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21B632850F for ; Mon, 3 Jun 2019 21:08:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 672A86B0274; Mon, 3 Jun 2019 17:08:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 622646B0276; Mon, 3 Jun 2019 17:08:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44DA06B0277; Mon, 3 Jun 2019 17:08:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 0F2996B0274 for ; Mon, 3 Jun 2019 17:08:38 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id x18so5132445pfj.4 for ; Mon, 03 Jun 2019 14:08:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=TZYaz3eXPzGxVmhv3HMCDwJQRLheeIorOMAlH7HJ9WE=; b=K/Q3gG/NEef+FUiYlnXQALxfxqvsaYFL89MnfHIiDFFUOXCsAoYTu/MlmPSTEmOOeU 3MpeHZGe4YlnvAZutoMvsDqz4Rj4P6fJH09etk1jRPqlb0HeZI3HBezSW47r0+BIX0Er UFQqcrGr1z3l7D/B834G6GTMwwuvNxpDXwwfqJNnpX5xK8NKPxr8SxatIj2JTrbyNErP H3GDUzIGVfjdw234pkH9//Qc3xbMH/xlAkAcP1ILJlZNJgEVJKbd1+p6AzFyGavWr0az 54vuPLcCpu1dyI8UasImC9xAnmClzV+rORsy333X2PnEL2q30XunmwXVFNAjCyF1mr24 lnQg== X-Gm-Message-State: APjAAAUeToX3O9xH48aoRfye9jueGEVSIFS3Eheef6qPaQgXZ1Vw27z2 NOGFnlB9dVmMdxG26Kv++kRkqOPcJGohjHAjf0NX+I2JV2i3XB+P2ugQu14q0S7pKu1K7ewU7Hs jOMxDNM4QiT4XpFBSyZ8WWBsstWBU9GOoqoWFMyAe+5s7Uu8RFk1WWPNebZjWAEuWbA== X-Received: by 2002:a62:1c82:: with SMTP id c124mr20987564pfc.39.1559596117574; Mon, 03 Jun 2019 14:08:37 -0700 (PDT) X-Received: by 2002:a62:1c82:: with SMTP id c124mr20987420pfc.39.1559596116225; Mon, 03 Jun 2019 14:08:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596116; cv=none; d=google.com; s=arc-20160816; b=I8xRRlnKIK6LhxMR53Jy0MXy5jD7X8DkuMAx+/UUTDoPzpyAH0FZmMHzZZbmge93ht sbX5lN/GCc/wBLWg6KmjpSo1sYooxc68gRb+TSWydWhTRTV5UvVmLGt3XY3WeMckZk1o BCLe+4r86le5b+U6vvKejRzlCoMU6d0PEtiTT2EixdB3H9wrJfEcpgvmcpgZ4RJiEFzR Rf0z1LwKxrN5lgIwUDzZ5gXhbfBUoxH1SxaMk5JLwEHySyJjym7UICM4T4VKCeXguKBw YgJ+rv6X2gCrtUFh4eunjAVlaIu/df+bDvYqXR2syFRDcHV0J72vyLr5EyEAx1JliXdP WD0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TZYaz3eXPzGxVmhv3HMCDwJQRLheeIorOMAlH7HJ9WE=; b=d9PQBMmAQrSPE4kDJL6DYolvxVzQc1g+O5hjcEhCvlPMcdrLIQeKSd8yKTDjsCrVJY 2DN9rMG2SUrNcEpnLCLwdC2g55zS41srY15yOz+zBvnRc0oZBcDUleCQ1xibhxJhn8dB 2Yst49/bLAJ8UQo8pUypOtdgtYjoRuUjQ9rXGjWbD8AcTh8jwTnDhJsfwyROSVtT/I2i aN2y/ubbYYRccr/UMNU+J5O+efwCABTFz05CewiDY13xtGT7jJnxgoM+S2HXhSXyPSsT gG18J98zXYBlwxCGoxO+T/1Vnby0HmWxwY0kmOtdNnjRby3iYR2CTRWRUA2lLEs9Hhk0 DnmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=WQm7ItMw; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z16sor16037864pgl.47.2019.06.03.14.08.36 for (Google Transport Security); Mon, 03 Jun 2019 14:08:36 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=WQm7ItMw; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TZYaz3eXPzGxVmhv3HMCDwJQRLheeIorOMAlH7HJ9WE=; b=WQm7ItMwHktG69xBNCOEpRK0igijLT4KjtCpBChKUtJC8aBS9I40wEtrTmWdpuarBT EckEfBxI/uccO9dAsIBtIyhtxBfgqHV0832w0TwgkUwpZ7L5A7gL+qjqAq5+n37IDIcL QOnCWjesoJnT4Ik5GCuPeO/EyCm9kSCpcQ26IFQLfuLJY5XXNcXM+pfT84cIDT5o+y6o +MhC8h281MnBe0uI2y+qvuR2asCG9iNhPDsNn9dwW9QRXVOtO8309ZMSiP9qFEiRH84p fQJBUtGGkOa0hmoDQAgB3Q9oqONyoSWSGmot04Ty0joMo26ffwXfKHDAmgMunBMEa7eZ vQvw== X-Google-Smtp-Source: APXvYqyUI6JZB6XXqVKgsi+gR5x7wSmX1jg48nP5ImJujjIGhoL+bC3ILJjIPqmTRQB1s7taN2/0gw== X-Received: by 2002:a65:494a:: with SMTP id q10mr30911726pgs.201.1559596115744; Mon, 03 Jun 2019 14:08:35 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id g8sm7238285pgd.29.2019.06.03.14.08.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:35 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 06/11] mm: vmscan: turn shrink_node_memcg() into shrink_lruvec() Date: Mon, 3 Jun 2019 17:07:41 -0400 Message-Id: <20190603210746.15800-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A lruvec holds LRU pages owned by a certain NUMA node and cgroup. Instead of awkwardly passing around a combination of a pgdat and a memcg pointer, pass down the lruvec as soon as we can look it up. Nested callers that need to access node or cgroup properties can look them them up if necessary, but there are only a few cases. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 304974481146..b85111474ee2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2210,9 +2210,10 @@ enum scan_balance { * nr[0] = anon inactive pages to scan; nr[1] = anon active pages to scan * nr[2] = file inactive pages to scan; nr[3] = file active pages to scan */ -static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, - struct scan_control *sc, unsigned long *nr) +static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, + unsigned long *nr) { + struct mem_cgroup *memcg = lruvec_memcg(lruvec); int swappiness = mem_cgroup_swappiness(memcg); struct zone_reclaim_stat *reclaim_stat = &lruvec->reclaim_stat; u64 fraction[2]; @@ -2460,13 +2461,8 @@ static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg, } } -/* - * This is a basic per-node page freer. Used by both kswapd and direct reclaim. - */ -static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memcg, - struct scan_control *sc) +static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { - struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long nr[NR_LRU_LISTS]; unsigned long targets[NR_LRU_LISTS]; unsigned long nr_to_scan; @@ -2476,7 +2472,7 @@ static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memc struct blk_plug plug; bool scan_adjusted; - get_scan_count(lruvec, memcg, sc, nr); + get_scan_count(lruvec, sc, nr); /* Record the original scan target for proportional adjustments later */ memcpy(targets, nr, sizeof(nr)); @@ -2689,6 +2685,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) memcg = mem_cgroup_iter(root, NULL, &reclaim); do { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); unsigned long reclaimed; unsigned long scanned; @@ -2725,7 +2722,8 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) reclaimed = sc->nr_reclaimed; scanned = sc->nr_scanned; - shrink_node_memcg(pgdat, memcg, sc); + + shrink_lruvec(lruvec, sc); if (sc->may_shrinkslab) { shrink_slab(sc->gfp_mask, pgdat->node_id, @@ -3243,6 +3241,7 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, pg_data_t *pgdat, unsigned long *nr_scanned) { + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); struct scan_control sc = { .nr_to_reclaim = SWAP_CLUSTER_MAX, .target_mem_cgroup = memcg, @@ -3268,7 +3267,7 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, * will pick up pages from other mem cgroup's as well. We hack * the priority and make it zero. */ - shrink_node_memcg(pgdat, memcg, &sc); + shrink_lruvec(lruvec, &sc); trace_mm_vmscan_memcg_softlimit_reclaim_end( cgroup_ino(memcg->css.cgroup), From patchwork Mon Jun 3 21:07:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973991 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B20F933 for ; Mon, 3 Jun 2019 21:08:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BD2B2807B for ; Mon, 3 Jun 2019 21:08:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3055F2867C; Mon, 3 Jun 2019 21:08:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F4602807B for ; Mon, 3 Jun 2019 21:08:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C68BF6B0276; Mon, 3 Jun 2019 17:08:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BC93B6B0277; Mon, 3 Jun 2019 17:08:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A91FE6B0278; Mon, 3 Jun 2019 17:08:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 722AB6B0276 for ; Mon, 3 Jun 2019 17:08:40 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id x18so5132534pfj.4 for ; Mon, 03 Jun 2019 14:08:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=TqYkozLWwlwjh5tlY84d20KyoxNGzeaiPZCNbBAwhx4=; b=VyOlXNC6buTiunA49cG2F/afluNhQbVD4s0Krj/22tXKNurG4/ruLlCy7GsPpWd9aJ Fkfpnp4IFmLATXOJYYcgoUjVDglRfU0WmfMjDj2IzV3yEMzzdcAn48rt3hKZTk7Zm3eT HY1vblLH00zre5wS75hwmB3YGS7mKz1w3r6iNgr4JRNAziJsutTsG1xpgxwmatImZ4L8 sFAa/kZogUTINftVY0bbpEaOyMe0c/Cbgv89aRq0IFrQBwiAk3ZPgVBWQsLXdQMgujnS +McqcbfA1+4fNb08ThZ9XbOA9GOlaK2dIbJ7CDLkjCtOokAtDfEmTbCJLmRzsRvp+N7x U6+w== X-Gm-Message-State: APjAAAWI3ykByN9KjBF/wpkq1rmupFvecxuSowV5++d6GRy8jOEpRhQI aMBMXtQ8YffBOaXyv3ZhZxQMktDhRXvwuLGkJ1EOHTE+ztMyqk5b6NeV19N/rsY0Cd2ALCugDos 6ScYoKS5HCPShPm2Lc/Uh2gVet4tWfuOk+/ByvCwJA/YyX5iFtu3JTSp9bbwkAaV14Q== X-Received: by 2002:a63:1516:: with SMTP id v22mr9295230pgl.204.1559596119979; Mon, 03 Jun 2019 14:08:39 -0700 (PDT) X-Received: by 2002:a63:1516:: with SMTP id v22mr9295052pgl.204.1559596118301; Mon, 03 Jun 2019 14:08:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596118; cv=none; d=google.com; s=arc-20160816; b=Wy6UPI9/ybRIYtpQ//84OsBib+TVcMHVX5fRSeXumGAQ3vCb276T6fBCTQDr5QBuyu TMXjXJYthewpxYhUPDc4TfYG/hE3PAVxCF11Y+av6uAql/139OMt1OIebHYM7Obg+rAa tRlkE/Z+lzmsjAkGgiVO+3ofo+lMEH5DPWZX62HNLKrRCnSXdrLTL1VIxr2xOvjO9RBf GHCldM+EFmpSY00/mG7KMNSyWXEDguCMbPiSljd3LoHM7QQ+2g8/Al38mRDCoeLZW/XO N8gwI4WmJ4fEO41nmR4BuX/RFydoTVrYLZSEs1Y7asWHSY41v6JfZIb9oq5FGXCI9eXX vzDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TqYkozLWwlwjh5tlY84d20KyoxNGzeaiPZCNbBAwhx4=; b=wGgjun0+vEkCD4HyLzwHxcw2KQIxvg559cx9KR2UPiSByG5mnEPtxdWjK7LddhKaOh 8wBlkdZn3iHUXH118+WanAxiJblPVj5kcHOTLMRLdZ+BnC59qIDi8NBaWObf5YH0hVik ivUm2sNqdzW6voNzfDxQP04Hr1ge6gHlGbv2CLlwVXfjMAJQ0x0NFi6jVC1ycOqk9Vw2 DIJjjT7B5Vf4LWJci5MA+hBP4Ee5vplw7lz9JbGr58JyXYDBuWdndhgvbXIPavZ8Pt3H rH1au65r2hs73GAtTvHYV4/UWQqO4S9P9uAwyHL7ULTyqXLWXR+MMQmApsQ3kMCC2LG1 cllA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=CfO5Ti2i; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c9sor10307737pjr.0.2019.06.03.14.08.38 for (Google Transport Security); Mon, 03 Jun 2019 14:08:38 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=CfO5Ti2i; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TqYkozLWwlwjh5tlY84d20KyoxNGzeaiPZCNbBAwhx4=; b=CfO5Ti2iu2mKkyrYwQLDB8QQe+3afMqMfaNa/wJWa+mXj+X31SmaTgT+uw/gY/rdTf jyGegdzwHQpljpHWA2wVsG+b+pBVziQt1WX1xTIGjq8MuGjOuPpGwAopHMkHZqKp3+Ui tIjfvPdEXyrjLHGUp3YnhPfJubq1kFYnfoDUBGlOpXwawLoXGpyTrhQT4Z9tXoIbW1ou FA5RvsNPdN/E2REMNh3zWT5ebiR0nab16veulDQZkupuNnzCdYJtcdh5eyxzTO6LIVCK 7VRPrTbECquvW9jgMxA0+O34IrqMJjOL2ObDNuSMq/SrUHZlwsk4bT5BfEeNMBlTPcMO wkEw== X-Google-Smtp-Source: APXvYqzH0EJ92OjOv5FwdHpAISabNaN9/8D6YdAXlhN10L2LyvWxDpYUP+V6sT0plAQOTb0Hmr9Pgg== X-Received: by 2002:a17:90a:192:: with SMTP id 18mr33065596pjc.107.1559596117989; Mon, 03 Jun 2019 14:08:37 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id i25sm16343173pfr.73.2019.06.03.14.08.36 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:37 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 07/11] mm: vmscan: split shrink_node() into node part and memcgs part Date: Mon, 3 Jun 2019 17:07:42 -0400 Message-Id: <20190603210746.15800-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This function is getting long and unwieldy. The new shrink_node() handles the generic (node) reclaim aspects: - global vmpressure notifications - writeback and congestion throttling - reclaim/compaction management - kswapd giving up on unreclaimable nodes It then calls shrink_node_memcgs() which handles cgroup specifics: - the cgroup tree traversal - memory.low considerations - per-cgroup slab shrinking callbacks - per-cgroup vmpressure notifications Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b85111474ee2..ee79b39d0538 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2665,24 +2665,15 @@ static bool pgdat_memcg_congested(pg_data_t *pgdat, struct mem_cgroup *memcg) (memcg && memcg_congested(pgdat, memcg)); } -static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) +static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) { - struct reclaim_state *reclaim_state = current->reclaim_state; struct mem_cgroup *root = sc->target_mem_cgroup; struct mem_cgroup_reclaim_cookie reclaim = { .pgdat = pgdat, .priority = sc->priority, }; - unsigned long nr_reclaimed, nr_scanned; - bool reclaimable = false; struct mem_cgroup *memcg; -again: - memset(&sc->nr, 0, sizeof(sc->nr)); - - nr_reclaimed = sc->nr_reclaimed; - nr_scanned = sc->nr_scanned; - memcg = mem_cgroup_iter(root, NULL, &reclaim); do { struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); @@ -2750,6 +2741,22 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) break; } } while ((memcg = mem_cgroup_iter(root, memcg, &reclaim))); +} + +static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) +{ + struct reclaim_state *reclaim_state = current->reclaim_state; + struct mem_cgroup *root = sc->target_mem_cgroup; + unsigned long nr_reclaimed, nr_scanned; + bool reclaimable = false; + +again: + memset(&sc->nr, 0, sizeof(sc->nr)); + + nr_reclaimed = sc->nr_reclaimed; + nr_scanned = sc->nr_scanned; + + shrink_node_memcgs(pgdat, sc); if (reclaim_state) { sc->nr_reclaimed += reclaim_state->reclaimed_slab; @@ -2757,7 +2764,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) } /* Record the subtree's reclaim efficiency */ - vmpressure(sc->gfp_mask, sc->target_mem_cgroup, true, + vmpressure(sc->gfp_mask, root, true, sc->nr_scanned - nr_scanned, sc->nr_reclaimed - nr_reclaimed); From patchwork Mon Jun 3 21:07:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5606814E5 for ; Mon, 3 Jun 2019 21:08:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46D062807B for ; Mon, 3 Jun 2019 21:08:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3AE502867C; Mon, 3 Jun 2019 21:08:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 702E02807B for ; Mon, 3 Jun 2019 21:08:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22D7A6B0277; Mon, 3 Jun 2019 17:08:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1DDF96B0278; Mon, 3 Jun 2019 17:08:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CCEA6B0279; Mon, 3 Jun 2019 17:08:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BE6056B0277 for ; Mon, 3 Jun 2019 17:08:42 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id 30so1059887pgk.16 for ; Mon, 03 Jun 2019 14:08:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=B6ockcEeDUtUJIvlx/DoCdCq85by+mHeIaqq/AmMtrQ=; b=R3jEy7UKHvc5MRnGJ95GCSd5YwjtSY/423jONGH3SV0/JQSgQF0oBo4naHEZGLz02u 0E2FqSEOQlCw3s3yfhLkjL9gvaqPzHhTODDJjS3urNwc08+oZKSTprCrSj54uOb54iUR hmzi07jNoKPR1qLrG9PyMu5rhJ1x5yw4VuTXL1N/7AoDPlYMHkUDDIB1+OQAZJ+4FcKd NmsGM7SxVS+R+l/oxd6R34hoPB3+7/6RTM3ezPdDPwpqMqiSmZ/yJFyggWZnv6NhFH0f ibNCFIHODX3pn6F0MAPcqaoGVVZyskPvn7nZbczJqrpEvpU6IcoJ4FuRUQWgTRVqWCqr xf6g== X-Gm-Message-State: APjAAAWrid/NOq10ihmb7Cwd/+Y0cKlR4SJ5nanFsrlYA+r6CjJWlPkc hQ5bNb7isFjiOnwm4mw+8MH2Qbb9hH34MtWkNt+d0ALhk7+t71swSae2V2OzkVUrd4JXa7KGAL9 GVL6dHRKxDneVKhSks3vZlPOkkMEcKqkmraxZqX7wAMUsVaGHP0D5pBykBtwCjWeAhw== X-Received: by 2002:a17:90a:19d:: with SMTP id 29mr4927801pjc.71.1559596122291; Mon, 03 Jun 2019 14:08:42 -0700 (PDT) X-Received: by 2002:a17:90a:19d:: with SMTP id 29mr4927611pjc.71.1559596120593; Mon, 03 Jun 2019 14:08:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596120; cv=none; d=google.com; s=arc-20160816; b=nSPpRY4x1WaTTJoMW0+guW8CmRWehQQh0co8U26bfxWLL19yl6ka4SvXx+p1YxxTtM fuhPIsYn29tVELWOs9hVGXZn1uUwPfT6102Jcirdt9t8Y14hM64vcWpfBlcSxPHEVNui /Zf+egqEcCydNUn89fi4SS6fnuQQgAeSymjhLxKbAbwxE6EQU6iyYubqjznI3isj8c1i Fxz5/P5q64PMLMfh5EWy5qPDmjHo8MGZcLxibMXD793CPlrrNY6dtl4Obb8ztRdUy4yq 9kRFay9+UiBIzsFsMjeZ/YPO55DS7RKWweR6BpCbKb4voMLslJArYsECfX2RcTaDfqFj gF8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=B6ockcEeDUtUJIvlx/DoCdCq85by+mHeIaqq/AmMtrQ=; b=TnywK8Ch/ggfjW4ww3SrMqdyvFsc14IG7teGt0JNyvfBo+eY7EkZEbi8hJ0pbchKm9 M7xVHZ7jvpdSUCy5EWRMPlcyni/nTaLadUG9jW58zDbros39LpjPCwThr6uUVwgqmn2t +RGn7bawRARUOWRJY76JFlipeSFY7DuN1yPilsQgAkT8Or9pEsOe+MHv2kMxiqNjA6Yu ybLofCduYXxSNf5awRxMFBUw6rt1s8AByvPDX7WwMNis1L/GFzJsvwqEueb2J6lY0VL0 rOa1CqB0+rblDIlxx+r4PZhs5zaFjlLQZqIRpCB6rI1DryN64keCtQ1g3PKjyPrZM97V z0XQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=yTHNUkzu; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k13sor17612577pfi.31.2019.06.03.14.08.40 for (Google Transport Security); Mon, 03 Jun 2019 14:08:40 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=yTHNUkzu; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B6ockcEeDUtUJIvlx/DoCdCq85by+mHeIaqq/AmMtrQ=; b=yTHNUkzufxtfKAvZ8UcncxQaDV9qHqMwPaRk5AmPUtvARbAXYSqoPt3ofMv01d0IVK iq5U7nvDJ8vBmko1GaK2TR709/MyNG3qm2uSTW7dRg9eE71uW5vLMYGsmYte6+wfLRaE 5GKSSrFSsRhMFmNR16jq/rSVOoTVNyroChHLeKRWVj/c2bh0mvOgipIaBpWZcXeGN/D9 SvQM0GjhTA0oTOIG+m//ObL5L/IiQ6efS1GDg/J2rtN05JnTIU1VTH/rZWVL9SG0APMg /Yqy/0xBtIr31FfzTp67H6lg0r6x+o2rQFjdwXB4VjuV6R0OSBzszTpbsOt3VRhQlWSx f5kw== X-Google-Smtp-Source: APXvYqzuuFNtUjyKK1QDwjQku2LKUXQVeGjjjtfFcm0GcLid0BXMv6q6FaLzEqUddR/mQYmveXzu6A== X-Received: by 2002:a62:3741:: with SMTP id e62mr34248051pfa.213.1559596120271; Mon, 03 Jun 2019 14:08:40 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id l20sm15695900pff.102.2019.06.03.14.08.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:39 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 08/11] mm: vmscan: harmonize writeback congestion tracking for nodes & memcgs Date: Mon, 3 Jun 2019 17:07:43 -0400 Message-Id: <20190603210746.15800-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The current writeback congestion tracking has separate flags for kswapd reclaim (node level) and cgroup limit reclaim (memcg-node level). This is unnecessarily complicated: the lruvec is an existing abstraction layer for that node-memcg intersection. Introduce lruvec->flags and LRUVEC_CONGESTED. Then track that at the reclaim root level, which is either the NUMA node for global reclaim, or the cgroup-node intersection for cgroup reclaim. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 6 +-- include/linux/mmzone.h | 11 ++++-- mm/vmscan.c | 80 ++++++++++++-------------------------- 3 files changed, 36 insertions(+), 61 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index fc32cfaebf32..d33e09c51acc 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -144,9 +144,6 @@ struct mem_cgroup_per_node { unsigned long usage_in_excess;/* Set to the value by which */ /* the soft limit is exceeded*/ bool on_tree; - bool congested; /* memcg has many dirty pages */ - /* backed by a congested BDI */ - struct mem_cgroup *memcg; /* Back pointer, we cannot */ /* use container_of */ }; @@ -401,6 +398,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg, goto out; } + if (!memcg) + memcg = root_mem_cgroup; + mz = mem_cgroup_nodeinfo(memcg, pgdat->node_id); lruvec = &mz->lruvec; out: diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 95d63a395f40..b3ab64cf5619 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -293,6 +293,12 @@ struct zone_reclaim_stat { unsigned long recent_scanned[2]; }; +enum lruvec_flags { + LRUVEC_CONGESTED, /* lruvec has many dirty pages + * backed by a congested BDI + */ +}; + struct lruvec { struct list_head lists[NR_LRU_LISTS]; struct zone_reclaim_stat reclaim_stat; @@ -300,6 +306,8 @@ struct lruvec { atomic_long_t inactive_age; /* Refaults at the time of last reclaim cycle */ unsigned long refaults; + /* Various lruvec state flags (enum lruvec_flags) */ + unsigned long flags; #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif @@ -562,9 +570,6 @@ struct zone { } ____cacheline_internodealigned_in_smp; enum pgdat_flags { - PGDAT_CONGESTED, /* pgdat has many dirty pages backed by - * a congested BDI - */ PGDAT_DIRTY, /* reclaim scanning has recently found * many dirty file pages at the tail * of the LRU. diff --git a/mm/vmscan.c b/mm/vmscan.c index ee79b39d0538..eb535c572733 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -267,29 +267,6 @@ static bool writeback_working(struct scan_control *sc) #endif return false; } - -static void set_memcg_congestion(pg_data_t *pgdat, - struct mem_cgroup *memcg, - bool congested) -{ - struct mem_cgroup_per_node *mn; - - if (!memcg) - return; - - mn = mem_cgroup_nodeinfo(memcg, pgdat->node_id); - WRITE_ONCE(mn->congested, congested); -} - -static bool memcg_congested(pg_data_t *pgdat, - struct mem_cgroup *memcg) -{ - struct mem_cgroup_per_node *mn; - - mn = mem_cgroup_nodeinfo(memcg, pgdat->node_id); - return READ_ONCE(mn->congested); - -} #else static bool cgroup_reclaim(struct scan_control *sc) { @@ -300,18 +277,6 @@ static bool writeback_working(struct scan_control *sc) { return true; } - -static inline void set_memcg_congestion(struct pglist_data *pgdat, - struct mem_cgroup *memcg, bool congested) -{ -} - -static inline bool memcg_congested(struct pglist_data *pgdat, - struct mem_cgroup *memcg) -{ - return false; - -} #endif /* @@ -2659,12 +2624,6 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat, return true; } -static bool pgdat_memcg_congested(pg_data_t *pgdat, struct mem_cgroup *memcg) -{ - return test_bit(PGDAT_CONGESTED, &pgdat->flags) || - (memcg && memcg_congested(pgdat, memcg)); -} - static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) { struct mem_cgroup *root = sc->target_mem_cgroup; @@ -2748,8 +2707,11 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) struct reclaim_state *reclaim_state = current->reclaim_state; struct mem_cgroup *root = sc->target_mem_cgroup; unsigned long nr_reclaimed, nr_scanned; + struct lruvec *target_lruvec; bool reclaimable = false; + target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); + again: memset(&sc->nr, 0, sizeof(sc->nr)); @@ -2792,14 +2754,6 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) set_bit(PGDAT_WRITEBACK, &pgdat->flags); - /* - * Tag a node as congested if all the dirty pages - * scanned were backed by a congested BDI and - * wait_iff_congested will stall. - */ - if (sc->nr.dirty && sc->nr.dirty == sc->nr.congested) - set_bit(PGDAT_CONGESTED, &pgdat->flags); - /* Allow kswapd to start writing pages during reclaim.*/ if (sc->nr.unqueued_dirty == sc->nr.file_taken) set_bit(PGDAT_DIRTY, &pgdat->flags); @@ -2815,12 +2769,17 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) } /* + * Tag a node/memcg as congested if all the dirty pages + * scanned were backed by a congested BDI and + * wait_iff_congested will stall. + * * Legacy memcg will stall in page writeback so avoid forcibly * stalling in wait_iff_congested(). */ - if (cgroup_reclaim(sc) && writeback_working(sc) && + if ((current_is_kswapd() || + (cgroup_reclaim(sc) && writeback_working(sc))) && sc->nr.dirty && sc->nr.dirty == sc->nr.congested) - set_memcg_congestion(pgdat, root, true); + set_bit(LRUVEC_CONGESTED, &target_lruvec->flags); /* * Stall direct reclaim for IO completions if underlying BDIs @@ -2828,8 +2787,9 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) * starts encountering unqueued dirty pages or cycling through * the LRU too quickly. */ - if (!sc->hibernation_mode && !current_is_kswapd() && - current_may_throttle() && pgdat_memcg_congested(pgdat, root)) + if (!current_is_kswapd() && current_may_throttle() && + !sc->hibernation_mode && + test_bit(LRUVEC_CONGESTED, &target_lruvec->flags)) wait_iff_congested(BLK_RW_ASYNC, HZ/10); if (should_continue_reclaim(pgdat, sc->nr_reclaimed - nr_reclaimed, @@ -3043,8 +3003,16 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist, if (zone->zone_pgdat == last_pgdat) continue; last_pgdat = zone->zone_pgdat; + snapshot_refaults(sc->target_mem_cgroup, zone->zone_pgdat); - set_memcg_congestion(last_pgdat, sc->target_mem_cgroup, false); + + if (cgroup_reclaim(sc)) { + struct lruvec *lruvec; + + lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, + zone->zone_pgdat); + clear_bit(LRUVEC_CONGESTED, &lruvec->flags); + } } delayacct_freepages_end(); @@ -3419,7 +3387,9 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int classzone_idx) /* Clear pgdat state for congested, dirty or under writeback. */ static void clear_pgdat_congested(pg_data_t *pgdat) { - clear_bit(PGDAT_CONGESTED, &pgdat->flags); + struct lruvec *lruvec = mem_cgroup_lruvec(NULL, pgdat); + + clear_bit(LRUVEC_CONGESTED, &lruvec->flags); clear_bit(PGDAT_DIRTY, &pgdat->flags); clear_bit(PGDAT_WRITEBACK, &pgdat->flags); } From patchwork Mon Jun 3 21:07:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AB9C14E5 for ; Mon, 3 Jun 2019 21:08:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B3CB2867C for ; Mon, 3 Jun 2019 21:08:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3A4328724; Mon, 3 Jun 2019 21:08:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6869B2867C for ; Mon, 3 Jun 2019 21:08:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B09826B0278; Mon, 3 Jun 2019 17:08:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A422B6B0279; Mon, 3 Jun 2019 17:08:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BDA36B027A; Mon, 3 Jun 2019 17:08:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 512F36B0278 for ; Mon, 3 Jun 2019 17:08:44 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id s4so13405280pfh.14 for ; Mon, 03 Jun 2019 14:08:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uM4S4y/Y5SrPdENg0QBPLDuc00s8nib+/ajfOkcZMik=; b=ieCPocYJ+uTSsQOHzqZ52kIKbuz+6nFnrUs28VD2AHEadJOjR/BBu//RQdFAlmbdDV ez6MXCdfwdfosMAXV/iBrkeTfwdFzeAZ2l+ls67mddW993didFcUdAaOMr2pbf2EP/ct d7JB8NkaUOTjo6jb8OFVCGFdPSeB6O5OkrsZKgyU7UOIdk0BGZquY1euc3UR7LtqcCQv WVZyfUsxvwM49Jiwnnm9jrz61gBJOfhWf96jJAA57XdQp4lf47zOnEh2/kgi8BEnh68B SjqsiKSwK+XHPM+dDUf1i3Bh/SN/iztQJWwlzNaoAtF1u55TWcHMLyKW0R0uEvBpOsO2 03ew== X-Gm-Message-State: APjAAAUrNxBAXLLnZhRxAck2KYNQq8719QkgOhzzNppELQUrwSf3HaCg iSSw44ajOCO9H6evZ7XhIfYveuwthmXLhmKs341xgErH9y6y91d1fVxJhegwRkb2zpW9brVAVIr A8mawZ6bIcuJBZoWHq5ibRS6dceaxT0RB8fkFGHowpK0fE86TSiHSKaJbNAHQjkG2MQ== X-Received: by 2002:aa7:8c0f:: with SMTP id c15mr7679189pfd.113.1559596123848; Mon, 03 Jun 2019 14:08:43 -0700 (PDT) X-Received: by 2002:aa7:8c0f:: with SMTP id c15mr7679073pfd.113.1559596122780; Mon, 03 Jun 2019 14:08:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596122; cv=none; d=google.com; s=arc-20160816; b=BJEmDh5s4lLxUMlZOW9qRf0hAkQoyud0R86gXDegMylxialJS+rvJVZFkKJ5wYd/SX CL/G6GOnO74/Yzls7TMxW4OV8s5YA5dPQulwpjJDPjRcegzd5MJ6PTKcEpjC/7t1x/gU +aJdsPVK4PHp4xtCi7yXIgNj4AUz2v4AsBUUyS1hh4TUsbXqBahwOgBcX7HbJiIUthEm dma8AASGgeeQSULKptKvcni4sTCcuPXdUVKmEQHU3mE6qRuzGa8T0XTPSnmSfwCVTkL/ kR4+gZ0pfTcpxSHqaCnH7DRjQvL5M6y1RwD4HCm9pgfxfLG7P+6fMcQ+r66KIDI6Ly9f hSlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uM4S4y/Y5SrPdENg0QBPLDuc00s8nib+/ajfOkcZMik=; b=ouRdDk6B3h245/IZXW5FpS4c6HG7aHIvTBO4fqoJk8/aN0x8NP3OJL37EpBpaaHbrO 9TgKv13ZSHpwZTYon+DlFbjtUUpzV2+H1ZczdUxWPFw1L4ygypgxqFHWIt5VfKnqs1io jtS2HlYboen2+Dmw7fjRQ6I9UJN/8gkxnC1mPD9pFctebVgLRiShbZWr0Mjinycya8Ie /1fXLNm58HtIlAaD8Rc7dzqWkRHuGvZ0y1hkVb3clTiWrT7bmRNpc3F410oKzT+TPDxp Uqy4aeSjog2LcU4cJNXawQBXtsfDWz4ljU4GBPgZrKazsUKDrmvQTA9XquOYwVu1GH6a 8bpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=E5wKX75L; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v22sor15969333pfm.18.2019.06.03.14.08.42 for (Google Transport Security); Mon, 03 Jun 2019 14:08:42 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=E5wKX75L; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uM4S4y/Y5SrPdENg0QBPLDuc00s8nib+/ajfOkcZMik=; b=E5wKX75LYj21K7r3b5uPvJHFB3a+wonjs7ZsPnT7rqWS2Jj94E45SuqIpvC70AjZgc JIPqfhBiJe5qxRDHnW0a11OgkMQCZGxN/JgMiMb2uO+dp8dsS4hv+h8pIvllW6xzlqex 2ajtI1YvJTzHyOEvFyG0WZ424MseQjErWrB1aHCv6+yd8mrPV/JSYvxp5kyasLqQo2Pf RCFBx4MsTkG80s0S4efJn9Fsp/kMF8ETw52Oxn7dHom+bNsgFQe2Z7qz52Aj2oZ8A3IN cQzPnohrFrwqFS/0osPnM8G9h7NO7drtGI+CLw2kKPFUoAAEjC2xRAV9cMROAgznBSCR 4Dhg== X-Google-Smtp-Source: APXvYqwbMD8TVz0HqZSC+oHhO3LjocjnT1Uw0hT+H5KZ3//8wtU6n995qzN2B8+Dl5CLBrRD1SHU2A== X-Received: by 2002:a63:c106:: with SMTP id w6mr21948229pgf.422.1559596122497; Mon, 03 Jun 2019 14:08:42 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id t25sm11786407pgv.30.2019.06.03.14.08.41 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:41 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 09/11] mm: vmscan: move file exhaustion detection to the node level Date: Mon, 3 Jun 2019 17:07:44 -0400 Message-Id: <20190603210746.15800-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When file pages are lower than the watermark on a node, we try to force scan anonymous pages to counter-act the balancing algorithms preference for new file pages when they are likely thrashing. This is node-level decision, but it's currently made each time we look at an lruvec. This is unnecessarily expensive and also a layering violation that makes the code harder to understand. Clean this up by making the check once per node and setting a flag in the scan_control. Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt --- mm/vmscan.c | 80 ++++++++++++++++++++++++++++------------------------- 1 file changed, 42 insertions(+), 38 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index eb535c572733..cabf94dfa92d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -104,6 +104,9 @@ struct scan_control { /* One of the zones is ready for compaction */ unsigned int compaction_ready:1; + /* The file pages on the current node are dangerously low */ + unsigned int file_is_tiny:1; + /* Allocation order */ s8 order; @@ -2219,45 +2222,16 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, } /* - * Prevent the reclaimer from falling into the cache trap: as - * cache pages start out inactive, every cache fault will tip - * the scan balance towards the file LRU. And as the file LRU - * shrinks, so does the window for rotation from references. - * This means we have a runaway feedback loop where a tiny - * thrashing file LRU becomes infinitely more attractive than - * anon pages. Try to detect this based on file LRU size. + * If the system is almost out of file pages, force-scan anon. + * But only if there are enough inactive anonymous pages on + * the LRU. Otherwise, the small LRU gets thrashed. */ - if (!cgroup_reclaim(sc)) { - unsigned long pgdatfile; - unsigned long pgdatfree; - int z; - unsigned long total_high_wmark = 0; - - pgdatfree = sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); - pgdatfile = node_page_state(pgdat, NR_ACTIVE_FILE) + - node_page_state(pgdat, NR_INACTIVE_FILE); - - for (z = 0; z < MAX_NR_ZONES; z++) { - struct zone *zone = &pgdat->node_zones[z]; - if (!managed_zone(zone)) - continue; - - total_high_wmark += high_wmark_pages(zone); - } - - if (unlikely(pgdatfile + pgdatfree <= total_high_wmark)) { - /* - * Force SCAN_ANON if there are enough inactive - * anonymous pages on the LRU in eligible zones. - * Otherwise, the small LRU gets thrashed. - */ - if (!inactive_list_is_low(lruvec, false, sc, false) && - lruvec_lru_size(lruvec, LRU_INACTIVE_ANON, sc->reclaim_idx) - >> sc->priority) { - scan_balance = SCAN_ANON; - goto out; - } - } + if (sc->file_is_tiny && + !inactive_list_is_low(lruvec, false, sc, false) && + lruvec_lru_size(lruvec, LRU_INACTIVE_ANON, + sc->reclaim_idx) >> sc->priority) { + scan_balance = SCAN_ANON; + goto out; } /* @@ -2718,6 +2692,36 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; + /* + * Prevent the reclaimer from falling into the cache trap: as + * cache pages start out inactive, every cache fault will tip + * the scan balance towards the file LRU. And as the file LRU + * shrinks, so does the window for rotation from references. + * This means we have a runaway feedback loop where a tiny + * thrashing file LRU becomes infinitely more attractive than + * anon pages. Try to detect this based on file LRU size. + */ + if (!cgroup_reclaim(sc)) { + unsigned long file; + unsigned long free; + int z; + unsigned long total_high_wmark = 0; + + free = sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); + file = node_page_state(pgdat, NR_ACTIVE_FILE) + + node_page_state(pgdat, NR_INACTIVE_FILE); + + for (z = 0; z < MAX_NR_ZONES; z++) { + struct zone *zone = &pgdat->node_zones[z]; + if (!managed_zone(zone)) + continue; + + total_high_wmark += high_wmark_pages(zone); + } + + sc->file_is_tiny = file + free <= total_high_wmark; + } + shrink_node_memcgs(pgdat, sc); if (reclaim_state) { From patchwork Mon Jun 3 21:07:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973997 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 621AD933 for ; Mon, 3 Jun 2019 21:08:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 511232850F for ; Mon, 3 Jun 2019 21:08:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 451E02870C; Mon, 3 Jun 2019 21:08:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43E0D2850F for ; Mon, 3 Jun 2019 21:08:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C16C76B0279; Mon, 3 Jun 2019 17:08:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BC90C6B027A; Mon, 3 Jun 2019 17:08:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1A506B027B; Mon, 3 Jun 2019 17:08:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 641926B0279 for ; Mon, 3 Jun 2019 17:08:47 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id x18so5132765pfj.4 for ; Mon, 03 Jun 2019 14:08:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=RkU+GNI+12gM0/3ezDUIhdwRxbBKO5hwfFKf3/tY+1A=; b=HkuR/0Urrw90ofOTXIPY0F9R/H12rr/ZoUVPsa7HdMBVWsXBUJ1sFpzz7uO9IpFEuW LCCCwowxl7hJIQ+DkrNo5Ds16hLia1jWV93tIPWg0NNZD06CNVKj9Fw6PKqiy/Vyjnhq q8I90NEgXyteGsJ260Q8/rd9Ys5Oh3wzz2pRA3GdU4mHwsPZ0gNx5lkhxyYrGnyljgdW UA2a9Kg+jZ7rz7LA7Q+g8SGv5ht+m6D2b2tOv5jet1LRyerUQevXrDPs8Wj6g4WtLXOX H+fvudExch2kjQq9ZcAqZPUj+sm7VaWgCtW7Z18dJNluI4dq0DirM0AutrtH0R5JXk1p cXiA== X-Gm-Message-State: APjAAAW2sTbDNZKUq89mHxlUYJg989XozYr1VQNLjJSZ+ZI/R21PhTZW e6SH+AiiIuQ58z9j5q2g3aMPBwVamCCCbN2WFofDGLXxN7/xgpX/9mdc7sDOT/op0r4KMSD0BhA HC6Vdu1mWn0L5SFeivXu0q22Ujxd2kT1CVonpppOcgFonOqAq+VoZcrhxktoaejmDZQ== X-Received: by 2002:a17:90a:aa0d:: with SMTP id k13mr31394745pjq.53.1559596126874; Mon, 03 Jun 2019 14:08:46 -0700 (PDT) X-Received: by 2002:a17:90a:aa0d:: with SMTP id k13mr31394550pjq.53.1559596125120; Mon, 03 Jun 2019 14:08:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596125; cv=none; d=google.com; s=arc-20160816; b=kZaYS1opI6tj9uqSOn66x9/rbeQTLgcTgAqDj4ATIGUzbIflLjU+70Kn5s4jOfbnV+ 3FgCEOkbIcHISrpWinrb/8hdOT8AWBd5Si5pCceL5guFcKFsZa7X/S3/i5WFiK2rzdiC dVu8Uq4Y/EHu6QKIezTGmwvv7dsy7631nsrII+FxepIKVxSucWk7qOL0VzcX6T/kbNhR MKbStGUPStlS5/UQv6nuNLhKgdOEO/yb/X7771EYA9jvO+sbmoh90N8yJy0x0Z1JfK7g D+fUJm1ukNrWC89dB6YY7l9YVMQojlvW3TlMyIHvr6G59prGPv05Xi3AVByXxjq36Kdj 2NXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=RkU+GNI+12gM0/3ezDUIhdwRxbBKO5hwfFKf3/tY+1A=; b=CH/wzv/FwESJL+Hg70WLOXewcgvyAeFhW0kDJk7c/7TzdghUCR5DD9aR0AhKkiMZc8 Z/9eJp57knjMWQKby+57HgPIpyNFF8a1axRXdk49sgTGF5HrP6WaSzSuuPPxUnOs9Uwq cmF7+bMtjMwh63FgNIYfjQnuAa+Bl8CnpgxmlTZXLLVMeZJZg5NL5BiIfnLZ1x1bRPKx d8JdqI4iqM3CmOpRjTDEKS/GptYPoe979IgUf95NsYnwqNjH+EhNq+FBHZblC9+54ErL NJMjNuPp3aNW9JOEr1ep5JNE+UmTaDWuPsqkImxhMi/gw6lCXibxULo/fpUrWiwhKsld Udig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b="oy/P/HEL"; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id l10sor16259951pgg.80.2019.06.03.14.08.45 for (Google Transport Security); Mon, 03 Jun 2019 14:08:45 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b="oy/P/HEL"; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RkU+GNI+12gM0/3ezDUIhdwRxbBKO5hwfFKf3/tY+1A=; b=oy/P/HELxxb/EJXE/apahDsNV+eoErpsgMRA3rDtqEtlDDlbZphuEOvhKuoRrjMqEx 46qUU/kTsBQsZvcKbHxIKwzTwvp9WV/Rlw4iq/wVvMh29zU7r3Bzk+PY5TaRifY2Kdid PzfFZwnKEwIFkeknv/LoIp5EIwI5/1kQmv83ryrRw5vYrNlWlEFArwKYxpko2I9RGQcV qGj9X132iKsMr59hbcUjFBmjb3YkYvtJh/b6wDhM3kov0KQPoQgnPNBmZ1zSoYVKC6wV FahT0+Qi4yXU5iUfE9lKgEBCCf322eaHmQWH1qEq/uJTCM1UkuLONmaoc7RN55tbh+Wt vLNA== X-Google-Smtp-Source: APXvYqzvNF8OO/3aIw1yj8y8UKddwiouLy94HRA+PfYIFGFrI5SBPLc37aAR3Vqxgd2Ut2v61xQdSg== X-Received: by 2002:a63:1919:: with SMTP id z25mr31688677pgl.440.1559596124724; Mon, 03 Jun 2019 14:08:44 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id e4sm15405027pgi.80.2019.06.03.14.08.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:44 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 10/11] mm: vmscan: detect file thrashing at the reclaim root Date: Mon, 3 Jun 2019 17:07:45 -0400 Message-Id: <20190603210746.15800-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We use refault information to determine whether the cache workingset is stable or transitioning, and dynamically adjust the inactive:active file LRU ratio so as to maximize protection from one-off cache during stable periods, and minimize IO during transitions. With cgroups and their nested LRU lists, we currently don't do this correctly. While recursive cgroup reclaim establishes a relative LRU order among the pages of all involved cgroups, refaults only affect the local LRU order in the cgroup in which they are occuring. As a result, cache transitions can take longer in a cgrouped system as the active pages of sibling cgroups aren't challenged when they should be. [ Right now, this is somewhat theoretical, because the siblings, under continued regular reclaim pressure, should eventually run out of inactive pages - and since inactive:active *size* balancing is also done on a cgroup-local level, we will challenge the active pages eventually in most cases. But the next patch will move that relative size enforcement to the reclaim root as well, and then this patch here will be necessary to propagate refault pressure to siblings. ] This patch moves refault detection to the root of reclaim. Instead of remembering the cgroup owner of an evicted page, remember the cgroup that caused the reclaim to happen. When refaults later occur, they'll correctly influence the cross-cgroup LRU order that reclaim follows. I.e. if global reclaim kicked out pages in some subgroup A/B/C, the refault of those pages will challenge the global LRU order, and not just the local order down inside C. Signed-off-by: Johannes Weiner --- include/linux/memcontrol.h | 5 +++ include/linux/swap.h | 2 +- mm/vmscan.c | 32 ++++++++--------- mm/workingset.c | 72 +++++++++++++++++++++++++++++--------- 4 files changed, 77 insertions(+), 34 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d33e09c51acc..70dcb6ab4e68 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -899,6 +899,11 @@ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page, return &pgdat->__lruvec; } +static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg) +{ + return NULL; +} + static inline bool mm_match_cgroup(struct mm_struct *mm, struct mem_cgroup *memcg) { diff --git a/include/linux/swap.h b/include/linux/swap.h index de2c67a33b7e..c65c11158a6c 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -307,7 +307,7 @@ struct vma_swap_readahead { }; /* linux/mm/workingset.c */ -void *workingset_eviction(struct page *page); +void *workingset_eviction(struct mem_cgroup *target_memcg, struct page *page); void workingset_refault(struct page *page, void *shadow); void workingset_activation(struct page *page); diff --git a/mm/vmscan.c b/mm/vmscan.c index cabf94dfa92d..6b7bd5708c73 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -834,7 +834,7 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, * gets returned with a refcount of 0. */ static int __remove_mapping(struct address_space *mapping, struct page *page, - bool reclaimed) + bool reclaimed, struct mem_cgroup *target_memcg) { unsigned long flags; int refcount; @@ -909,7 +909,7 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, */ if (reclaimed && page_is_file_cache(page) && !mapping_exiting(mapping) && !dax_mapping(mapping)) - shadow = workingset_eviction(page); + shadow = workingset_eviction(target_memcg, page); __delete_from_page_cache(page, shadow); xa_unlock_irqrestore(&mapping->i_pages, flags); @@ -932,7 +932,7 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, */ int remove_mapping(struct address_space *mapping, struct page *page) { - if (__remove_mapping(mapping, page, false)) { + if (__remove_mapping(mapping, page, false, NULL)) { /* * Unfreezing the refcount with 1 rather than 2 effectively * drops the pagecache ref for us without requiring another @@ -1410,7 +1410,8 @@ static unsigned long shrink_page_list(struct list_head *page_list, count_vm_event(PGLAZYFREED); count_memcg_page_event(page, PGLAZYFREED); - } else if (!mapping || !__remove_mapping(mapping, page, true)) + } else if (!mapping || !__remove_mapping(mapping, page, true, + sc->target_mem_cgroup)) goto keep_locked; unlock_page(page); @@ -2119,6 +2120,7 @@ static bool inactive_list_is_low(struct lruvec *lruvec, bool file, enum lru_list inactive_lru = file * LRU_FILE; unsigned long inactive, active; unsigned long inactive_ratio; + struct lruvec *target_lruvec; unsigned long refaults; unsigned long gb; @@ -2130,8 +2132,9 @@ static bool inactive_list_is_low(struct lruvec *lruvec, bool file, * is being established. Disable active list protection to get * rid of the stale workingset quickly. */ - refaults = lruvec_page_state_local(lruvec, WORKINGSET_ACTIVATE); - if (file && actual_reclaim && lruvec->refaults != refaults) { + target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); + refaults = lruvec_page_state(target_lruvec, WORKINGSET_ACTIVATE); + if (file && actual_reclaim && target_lruvec->refaults != refaults) { inactive_ratio = 0; } else { gb = (inactive + active) >> (30 - PAGE_SHIFT); @@ -2937,19 +2940,14 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) sc->gfp_mask = orig_mask; } -static void snapshot_refaults(struct mem_cgroup *root_memcg, pg_data_t *pgdat) +static void snapshot_refaults(struct mem_cgroup *target_memcg, pg_data_t *pgdat) { - struct mem_cgroup *memcg; - - memcg = mem_cgroup_iter(root_memcg, NULL, NULL); - do { - unsigned long refaults; - struct lruvec *lruvec; + struct lruvec *target_lruvec; + unsigned long refaults; - lruvec = mem_cgroup_lruvec(memcg, pgdat); - refaults = lruvec_page_state_local(lruvec, WORKINGSET_ACTIVATE); - lruvec->refaults = refaults; - } while ((memcg = mem_cgroup_iter(root_memcg, memcg, NULL))); + target_lruvec = mem_cgroup_lruvec(target_memcg, pgdat); + refaults = lruvec_page_state(target_lruvec, WORKINGSET_ACTIVATE); + target_lruvec->refaults = refaults; } /* diff --git a/mm/workingset.c b/mm/workingset.c index 2aaa70bea99c..2ae67a6d770c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -213,28 +213,53 @@ static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat, *workingsetp = workingset; } +static void advance_inactive_age(struct mem_cgroup *memcg, pg_data_t *pgdat) +{ + /* + * Reclaiming a cgroup means reclaiming all its children in a + * round-robin fashion. That means that each cgroup has an LRU + * order that is composed of the LRU orders of its child + * cgroups; and every page has an LRU position not just in the + * cgroup that owns it, but in all of that group's ancestors. + * + * So when the physical inactive list of a leaf cgroup ages, + * the virtual inactive lists of all its parents, including + * the root cgroup's, age as well. + */ + do { + struct lruvec *lruvec; + + lruvec = mem_cgroup_lruvec(memcg, pgdat); + atomic_long_inc(&lruvec->inactive_age); + } while (memcg && (memcg = parent_mem_cgroup(memcg))); +} + /** * workingset_eviction - note the eviction of a page from memory + * @target_memcg: the cgroup that is causing the reclaim * @page: the page being evicted * * Returns a shadow entry to be stored in @page->mapping->i_pages in place * of the evicted @page so that a later refault can be detected. */ -void *workingset_eviction(struct page *page) +void *workingset_eviction(struct mem_cgroup *target_memcg, struct page *page) { struct pglist_data *pgdat = page_pgdat(page); - struct mem_cgroup *memcg = page_memcg(page); - int memcgid = mem_cgroup_id(memcg); unsigned long eviction; struct lruvec *lruvec; + int memcgid; /* Page is fully exclusive and pins page->mem_cgroup */ VM_BUG_ON_PAGE(PageLRU(page), page); VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(!PageLocked(page), page); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - eviction = atomic_long_inc_return(&lruvec->inactive_age); + advance_inactive_age(page_memcg(page), pgdat); + + /* XXX: target_memcg can be NULL, go through lruvec */ + lruvec = mem_cgroup_lruvec(target_memcg, pgdat); + memcgid = mem_cgroup_id(lruvec_memcg(lruvec)); + eviction = atomic_long_read(&lruvec->inactive_age); return pack_shadow(memcgid, pgdat, eviction, PageWorkingset(page)); } @@ -244,10 +269,13 @@ void *workingset_eviction(struct page *page) * @shadow: shadow entry of the evicted page * * Calculates and evaluates the refault distance of the previously - * evicted page in the context of the node it was allocated in. + * evicted page in the context of the node and the memcg whose memory + * pressure caused the eviction. */ void workingset_refault(struct page *page, void *shadow) { + struct mem_cgroup *eviction_memcg; + struct lruvec *eviction_lruvec; unsigned long refault_distance; struct pglist_data *pgdat; unsigned long active_file; @@ -277,12 +305,12 @@ void workingset_refault(struct page *page, void *shadow) * would be better if the root_mem_cgroup existed in all * configurations instead. */ - memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !memcg) + eviction_memcg = mem_cgroup_from_id(memcgid); + if (!mem_cgroup_disabled() && !eviction_memcg) goto out; - lruvec = mem_cgroup_lruvec(memcg, pgdat); - refault = atomic_long_read(&lruvec->inactive_age); - active_file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE, MAX_NR_ZONES); + eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); + refault = atomic_long_read(&eviction_lruvec->inactive_age); + active_file = lruvec_page_state(eviction_lruvec, NR_ACTIVE_FILE); /* * Calculate the refault distance @@ -302,6 +330,17 @@ void workingset_refault(struct page *page, void *shadow) */ refault_distance = (refault - eviction) & EVICTION_MASK; + /* + * The activation decision for this page is made at the level + * where the eviction occurred, as that is where the LRU order + * during page reclaim is being determined. + * + * However, the cgroup that will own the page is the one that + * is actually experiencing the refault event. + */ + memcg = get_mem_cgroup_from_mm(current->mm); + lruvec = mem_cgroup_lruvec(memcg, pgdat); + inc_lruvec_state(lruvec, WORKINGSET_REFAULT); /* @@ -310,10 +349,10 @@ void workingset_refault(struct page *page, void *shadow) * the memory was available to the page cache. */ if (refault_distance > active_file) - goto out; + goto out_memcg; SetPageActive(page); - atomic_long_inc(&lruvec->inactive_age); + advance_inactive_age(memcg, pgdat); inc_lruvec_state(lruvec, WORKINGSET_ACTIVATE); /* Page was active prior to eviction */ @@ -321,6 +360,9 @@ void workingset_refault(struct page *page, void *shadow) SetPageWorkingset(page); inc_lruvec_state(lruvec, WORKINGSET_RESTORE); } + +out_memcg: + mem_cgroup_put(memcg); out: rcu_read_unlock(); } @@ -332,7 +374,6 @@ void workingset_refault(struct page *page, void *shadow) void workingset_activation(struct page *page) { struct mem_cgroup *memcg; - struct lruvec *lruvec; rcu_read_lock(); /* @@ -345,8 +386,7 @@ void workingset_activation(struct page *page) memcg = page_memcg_rcu(page); if (!mem_cgroup_disabled() && !memcg) goto out; - lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page)); - atomic_long_inc(&lruvec->inactive_age); + advance_inactive_age(memcg, page_pgdat(page)); out: rcu_read_unlock(); } From patchwork Mon Jun 3 21:07:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 10973999 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE83B14E5 for ; Mon, 3 Jun 2019 21:08:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC2292807B for ; Mon, 3 Jun 2019 21:08:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A00D82867C; Mon, 3 Jun 2019 21:08:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 93A572850F for ; Mon, 3 Jun 2019 21:08:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F7836B027A; Mon, 3 Jun 2019 17:08:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 058526B027B; Mon, 3 Jun 2019 17:08:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D51B46B027C; Mon, 3 Jun 2019 17:08:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 9A0136B027A for ; Mon, 3 Jun 2019 17:08:49 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id c17so2029130pfb.21 for ; Mon, 03 Jun 2019 14:08:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=KSFw6UqIUAUxKrMDMZ5bH9ogt1Vx0fiDNTjI7FIJO1I=; b=ej8kgniT5ndTT6mc298nW99iSsUpJa5CbFYYaRjH6OtBlH5d4R0xjNdFMjEBp+nWD4 jYkQZI7xRIQmwS2fmv2gDQMn9WsIAng0WC1iDVMdS/vT/Vev3aCmfJchAlEUW0vsri1K 6sGZ00HfE/Ce2bOdLmIjfOnGvUYU7GTlsdHbm/0myOK4Vkn6nAjSMcbZ+QFf4lGCLJQU q76Ex2DrJum0J3BNhtxAoZCuGudQb20ajAxjKtAseukQxrQ/eEpif3qZ032WW3vZK8iF WzX3Kq2U9PKBN5h6ZFQvSBoDJVmyjmhEnBApfdJ1utaccM73TDRS31muIXWQPRub5A2t 0JQg== X-Gm-Message-State: APjAAAXfUqO9cBLzvzif9HCKrpobG6vnT4Pz9RzgGgqyEP7agIzXOTZT NxyNjyeSZpZHEMZ9r0aOxPxffbP91OrXFDwBPABL4MqUM1U582jvbT/mDscSESLNVBaRi6t/ykJ iacI3DHePl253Ar9RSFRm7yriIJuD2P2pUefgic6b0HMcJWYHKUpp1i5N3qDbV4/p5w== X-Received: by 2002:aa7:9203:: with SMTP id 3mr34617227pfo.123.1559596129079; Mon, 03 Jun 2019 14:08:49 -0700 (PDT) X-Received: by 2002:aa7:9203:: with SMTP id 3mr34617061pfo.123.1559596127640; Mon, 03 Jun 2019 14:08:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559596127; cv=none; d=google.com; s=arc-20160816; b=txZD844hHmyw3q/JK3wfXayWZtUCNOR4A89C6yKGR2qSRHIHJ2lfw3Rk8C5xqPq9Bx SWAdER5yA3kp4naHfcEqHud4Uu1cQtVNz3S/oJph41NxV9rHv9jPTjcJq7WWVPBlK04f FN1eQzRjduidxkGuAClRp8XbICFL6EsdFAFbO5pRYcT/D4mcZXHjtgJsIuTBrownFIY5 s4S/lrvB4Ir9Z5i6TnYwAwhZI+ItyJlIrGuNN9/Y+meDJ2OTLb2ExVs5Vmr8qMY1X9nd N1N1aH9mm8EooqTyhTnhaRDg6DLcrheFRZqyFM4FRMeKO2/hNM5QbNZ+k7ByeotjLT2E +oJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=KSFw6UqIUAUxKrMDMZ5bH9ogt1Vx0fiDNTjI7FIJO1I=; b=AgdO72EZIcd6/EgN928CVM719VLLD4YuTA+IIuyDGdcRkba7ay5gKkHR8ITUOV6HBB zjTBZExizJkdn7GLO4Scs0TIretngJJQYlT7qQ9hN7b25lb256uO0llwuEuMWEH5ld19 iF+xi5uPu74mFYx7eDM+RgQmWW4v/Fp1GQFhQYEAIvZJtKSLzu5YjvaiRfKdzY8USOt+ nopFlZqJxWKF4ItMpJedV1ZunV/War12RVycmaKLVISp2GvGB8x7+ix9KvpfOmxFCG7p on8N+FA3hZGrksx2a23syFNIf+B1dDdPRm7u+NHAoEVyBCz8G+0jKddpJbhFm7SYEFQB cbtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=JQ5yhRf4; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id ce5sor18229682plb.17.2019.06.03.14.08.47 for (Google Transport Security); Mon, 03 Jun 2019 14:08:47 -0700 (PDT) Received-SPF: pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=JQ5yhRf4; spf=pass (google.com: domain of hannes@cmpxchg.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cmpxchg.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KSFw6UqIUAUxKrMDMZ5bH9ogt1Vx0fiDNTjI7FIJO1I=; b=JQ5yhRf4xOamEI7RraqoEXJphDySauJdGOeiTVixaZFJSg7e45kJelp1MVAg2pWzl8 2pSdMhIaHfjVUfb58hAQ0h42WNUytSIxCl+jHgwplFWjdF/ert2FjaLjMk4qr65RnfF9 PeopQtt/DlL/qVNZgoRHtZDKi/tvZwRrTIDkoDol1umDJ08XDwiKMRSDJTEzKW1w3Y5k qmYr7UwqwPrP19G46F9cHLRDTOR1dB9+K9Nk7WIj0cG2UKQJ8ae4S7qwuLyVKDKNQ5Ez To8a9pSigqlGJfBnTnLbEuEU0vg1Zp6uscBSqJlVLsJH2aJKq2meu/IHN3qP9OJ27ksZ XUeQ== X-Google-Smtp-Source: APXvYqxhekrAeS4MNWMZ11e88dncPTFMLk2ht4wkuPxZo4rVekMrpq/gu1c9FfM4OkjwntgkJLKc4g== X-Received: by 2002:a17:902:7c03:: with SMTP id x3mr7501223pll.242.1559596126911; Mon, 03 Jun 2019 14:08:46 -0700 (PDT) Received: from localhost ([2620:10d:c091:500::1:9fa4]) by smtp.gmail.com with ESMTPSA id j23sm4643592pgb.63.2019.06.03.14.08.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jun 2019 14:08:46 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Andrey Ryabinin , Suren Baghdasaryan , Michal Hocko , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 11/11] mm: vmscan: enforce inactive:active ratio at the reclaim root Date: Mon, 3 Jun 2019 17:07:46 -0400 Message-Id: <20190603210746.15800-12-hannes@cmpxchg.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603210746.15800-1-hannes@cmpxchg.org> References: <20190603210746.15800-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We split the LRU lists into inactive and an active parts to maximize workingset protection while allowing just enough inactive cache space to faciltate readahead and writeback for one-off file accesses (e.g. a linear scan through a file, or logging); or just enough inactive anon to maintain recent reference information when reclaim needs to swap. With cgroups and their nested LRU lists, we currently don't do this correctly. While recursive cgroup reclaim establishes a relative LRU order among the pages of all involved cgroups, inactive:active size decisions are done on a per-cgroup level. As a result, we'll reclaim a cgroup's workingset when it doesn't have cold pages, even when one of its siblings has plenty of it that should be reclaimed first. For example: workload A has 50M worth of hot cache but doesn't do any one-off file accesses; meanwhile, parallel workload B scans files and rarely accesses the same page twice. If these workloads were to run in an uncgrouped system, A would be protected from the high rate of cache faults from B. But if they were put in parallel cgroups for memory accounting purposes, B's fast cache fault rate would push out the hot cache pages of A. This is unexpected and undesirable - the "scan resistance" of the page cache is broken. This patch moves inactive:active size balancing decisions to the root of reclaim - the same level where the LRU order is established. It does this by looking at the recursize size of the inactive and the active file sets of the cgroup subtree at the beginning of the reclaim cycle, and then making a decision - scan or skip active pages - that applies throughout the entire run and to every cgroup involved. With that in place, in the test above, the VM will recognize that there are plenty of inactive pages in the combined cache set of workloads A and B and prefer the one-off cache in B over the hot pages in A. The scan resistance of the cache is restored. Signed-off-by: Johannes Weiner --- include/linux/mmzone.h | 4 +- mm/vmscan.c | 183 +++++++++++++++++++++++++---------------- 2 files changed, 116 insertions(+), 71 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b3ab64cf5619..8d100905c2ec 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -270,12 +270,12 @@ enum lru_list { #define for_each_evictable_lru(lru) for (lru = 0; lru <= LRU_ACTIVE_FILE; lru++) -static inline int is_file_lru(enum lru_list lru) +static inline bool is_file_lru(enum lru_list lru) { return (lru == LRU_INACTIVE_FILE || lru == LRU_ACTIVE_FILE); } -static inline int is_active_lru(enum lru_list lru) +static inline bool is_active_lru(enum lru_list lru) { return (lru == LRU_ACTIVE_ANON || lru == LRU_ACTIVE_FILE); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 6b7bd5708c73..6af35bb02da0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -79,6 +79,11 @@ struct scan_control { */ struct mem_cgroup *target_mem_cgroup; + /* Can active pages be deactivated as part of reclaim? anon=0 file=1 */ + unsigned int may_deactivate:2; + unsigned int force_deactivate:1; + unsigned int skipped_deactivate:1; + /* Writepage batching in laptop mode; RECLAIM_WRITE */ unsigned int may_writepage:1; @@ -104,6 +109,9 @@ struct scan_control { /* One of the zones is ready for compaction */ unsigned int compaction_ready:1; + /* There is easily reclaimable cold cache in the current node */ + unsigned int cache_trim_mode:1; + /* The file pages on the current node are dangerously low */ unsigned int file_is_tiny:1; @@ -2084,6 +2092,20 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_deactivate, nr_rotated, sc->priority, file); } +static unsigned long shrink_list(enum lru_list lru, unsigned long nr_to_scan, + struct lruvec *lruvec, struct scan_control *sc) +{ + if (is_active_lru(lru)) { + if (sc->may_deactivate & (1 << is_file_lru(lru))) + shrink_active_list(nr_to_scan, lruvec, sc, lru); + else + sc->skipped_deactivate = 1; + return 0; + } + + return shrink_inactive_list(nr_to_scan, lruvec, sc, lru); +} + /* * The inactive anon list should be small enough that the VM never has * to do too much work. @@ -2112,59 +2134,25 @@ static void shrink_active_list(unsigned long nr_to_scan, * 1TB 101 10GB * 10TB 320 32GB */ -static bool inactive_list_is_low(struct lruvec *lruvec, bool file, - struct scan_control *sc, bool actual_reclaim) +static bool inactive_is_low(struct lruvec *lruvec, enum lru_list inactive_lru) { - enum lru_list active_lru = file * LRU_FILE + LRU_ACTIVE; - struct pglist_data *pgdat = lruvec_pgdat(lruvec); - enum lru_list inactive_lru = file * LRU_FILE; + enum lru_list active_lru = inactive_lru + LRU_ACTIVE; unsigned long inactive, active; unsigned long inactive_ratio; - struct lruvec *target_lruvec; - unsigned long refaults; unsigned long gb; - inactive = lruvec_lru_size(lruvec, inactive_lru, sc->reclaim_idx); - active = lruvec_lru_size(lruvec, active_lru, sc->reclaim_idx); + inactive = lruvec_page_state(lruvec, inactive_lru); + active = lruvec_page_state(lruvec, active_lru); - /* - * When refaults are being observed, it means a new workingset - * is being established. Disable active list protection to get - * rid of the stale workingset quickly. - */ - target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); - refaults = lruvec_page_state(target_lruvec, WORKINGSET_ACTIVATE); - if (file && actual_reclaim && target_lruvec->refaults != refaults) { - inactive_ratio = 0; - } else { - gb = (inactive + active) >> (30 - PAGE_SHIFT); - if (gb) - inactive_ratio = int_sqrt(10 * gb); - else - inactive_ratio = 1; - } - - if (actual_reclaim) - trace_mm_vmscan_inactive_list_is_low(pgdat->node_id, sc->reclaim_idx, - lruvec_lru_size(lruvec, inactive_lru, MAX_NR_ZONES), inactive, - lruvec_lru_size(lruvec, active_lru, MAX_NR_ZONES), active, - inactive_ratio, file); + gb = (inactive + active) >> (30 - PAGE_SHIFT); + if (gb) + inactive_ratio = int_sqrt(10 * gb); + else + inactive_ratio = 1; return inactive * inactive_ratio < active; } -static unsigned long shrink_list(enum lru_list lru, unsigned long nr_to_scan, - struct lruvec *lruvec, struct scan_control *sc) -{ - if (is_active_lru(lru)) { - if (inactive_list_is_low(lruvec, is_file_lru(lru), sc, true)) - shrink_active_list(nr_to_scan, lruvec, sc, lru); - return 0; - } - - return shrink_inactive_list(nr_to_scan, lruvec, sc, lru); -} - enum scan_balance { SCAN_EQUAL, SCAN_FRACT, @@ -2226,28 +2214,17 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, /* * If the system is almost out of file pages, force-scan anon. - * But only if there are enough inactive anonymous pages on - * the LRU. Otherwise, the small LRU gets thrashed. */ - if (sc->file_is_tiny && - !inactive_list_is_low(lruvec, false, sc, false) && - lruvec_lru_size(lruvec, LRU_INACTIVE_ANON, - sc->reclaim_idx) >> sc->priority) { + if (sc->file_is_tiny) { scan_balance = SCAN_ANON; goto out; } /* - * If there is enough inactive page cache, i.e. if the size of the - * inactive list is greater than that of the active list *and* the - * inactive list actually has some pages to scan on this priority, we - * do not reclaim anything from the anonymous working set right now. - * Without the second condition we could end up never scanning an - * lruvec even if it has plenty of old anonymous pages unless the - * system is under heavy pressure. + * If there is enough inactive page cache, we do not reclaim + * anything from the anonymous working right now. */ - if (!inactive_list_is_low(lruvec, true, sc, false) && - lruvec_lru_size(lruvec, LRU_INACTIVE_FILE, sc->reclaim_idx) >> sc->priority) { + if (sc->cache_trim_mode) { scan_balance = SCAN_FILE; goto out; } @@ -2512,7 +2489,7 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) * Even if we did not try to evict anon pages at all, we want to * rebalance the anon lru active/inactive ratio. */ - if (total_swap_pages && inactive_list_is_low(lruvec, false, sc, true)) + if (total_swap_pages && inactive_is_low(lruvec, LRU_INACTIVE_ANON)) shrink_active_list(SWAP_CLUSTER_MAX, lruvec, sc, LRU_ACTIVE_ANON); } @@ -2686,6 +2663,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) unsigned long nr_reclaimed, nr_scanned; struct lruvec *target_lruvec; bool reclaimable = false; + unsigned long file; target_lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); @@ -2695,6 +2673,44 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) nr_reclaimed = sc->nr_reclaimed; nr_scanned = sc->nr_scanned; + /* + * Target desirable inactive:active list ratios for the anon + * and file LRU lists. + */ + if (!sc->force_deactivate) { + unsigned long refaults; + + if (inactive_is_low(target_lruvec, LRU_INACTIVE_ANON)) + sc->may_deactivate |= 1; + else + sc->may_deactivate &= ~1; + + /* + * When refaults are being observed, it means a new + * workingset is being established. Deactivate to get + * rid of any stale active pages quickly. + */ + refaults = lruvec_page_state(target_lruvec, + WORKINGSET_ACTIVATE); + if (refaults != target_lruvec->refaults || + inactive_is_low(target_lruvec, LRU_INACTIVE_FILE)) + sc->may_deactivate |= 2; + else + sc->may_deactivate &= ~2; + } else + sc->may_deactivate = 3; + + /* + * If we have plenty of inactive file pages that aren't + * thrashing, try to reclaim those first before touching + * anonymous pages. + */ + file = lruvec_page_state(target_lruvec, LRU_INACTIVE_FILE); + if (file >> sc->priority && !(sc->may_deactivate & 2)) + sc->cache_trim_mode = 1; + else + sc->cache_trim_mode = 0; + /* * Prevent the reclaimer from falling into the cache trap: as * cache pages start out inactive, every cache fault will tip @@ -2705,10 +2721,9 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) * anon pages. Try to detect this based on file LRU size. */ if (!cgroup_reclaim(sc)) { - unsigned long file; - unsigned long free; - int z; unsigned long total_high_wmark = 0; + unsigned long free, anon; + int z; free = sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); file = node_page_state(pgdat, NR_ACTIVE_FILE) + @@ -2722,7 +2737,17 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) total_high_wmark += high_wmark_pages(zone); } - sc->file_is_tiny = file + free <= total_high_wmark; + /* + * If anon is low too, this isn't a runaway file + * reclaim problem, but rather just extreme memory + * pressure. Reclaim as per usual in that case. + */ + anon = node_page_state(pgdat, NR_INACTIVE_ANON); + + sc->file_is_tiny = + file + free <= total_high_wmark && + !(sc->may_deactivate & 1) && + anon >> sc->priority; } shrink_node_memcgs(pgdat, sc); @@ -3026,9 +3051,27 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist, if (sc->compaction_ready) return 1; + /* + * We make inactive:active ratio decisions based on the node's + * composition of memory, but a restrictive reclaim_idx or a + * memory.low cgroup setting can exempt large amounts of + * memory from reclaim. Neither of which are very common, so + * instead of doing costly eligibility calculations of the + * entire cgroup subtree up front, we assume the estimates are + * good, and retry with forcible deactivation if that fails. + */ + if (sc->skipped_deactivate) { + sc->priority = initial_priority; + sc->force_deactivate = 1; + sc->skipped_deactivate = 0; + goto retry; + } + /* Untapped cgroup reserves? Don't OOM, retry. */ if (sc->memcg_low_skipped) { sc->priority = initial_priority; + sc->force_deactivate = 0; + sc->skipped_deactivate = 0; sc->memcg_low_reclaim = 1; sc->memcg_low_skipped = 0; goto retry; @@ -3310,18 +3353,20 @@ static void age_active_anon(struct pglist_data *pgdat, struct scan_control *sc) { struct mem_cgroup *memcg; + struct lruvec *lruvec; if (!total_swap_pages) return; + lruvec = mem_cgroup_lruvec(NULL, pgdat); + if (!inactive_is_low(lruvec, LRU_INACTIVE_ANON)) + return; + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat); - - if (inactive_list_is_low(lruvec, false, sc, true)) - shrink_active_list(SWAP_CLUSTER_MAX, lruvec, - sc, LRU_ACTIVE_ANON); - + lruvec = mem_cgroup_lruvec(memcg, pgdat); + shrink_active_list(SWAP_CLUSTER_MAX, lruvec, + sc, LRU_ACTIVE_ANON); memcg = mem_cgroup_iter(NULL, memcg, NULL); } while (memcg); }