From patchwork Thu Mar 3 01:46:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12766951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC9E2C433F5 for ; Thu, 3 Mar 2022 01:46:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19C058D0002; Wed, 2 Mar 2022 20:46:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 14CA48D0001; Wed, 2 Mar 2022 20:46:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 013898D0002; Wed, 2 Mar 2022 20:46:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0035.hostedemail.com [216.40.44.35]) by kanga.kvack.org (Postfix) with ESMTP id E8BAF8D0001 for ; Wed, 2 Mar 2022 20:46:39 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 940F682D86 for ; Thu, 3 Mar 2022 01:46:39 +0000 (UTC) X-FDA: 79201385718.24.8EA922D Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf31.hostedemail.com (Postfix) with ESMTP id 25BC920009 for ; Thu, 3 Mar 2022 01:46:38 +0000 (UTC) Received: by mail-qk1-f178.google.com with SMTP id c7so2861120qka.7 for ; Wed, 02 Mar 2022 17:46:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:mime-version; bh=J7xXuqfykwCtxVDAzhMeaJ1p1sPJc+/o/m7algVP+XU=; b=LxbbblGvWdIwXNSBF/l440APj7rIdhHp3HbEjtRK6L+Kc3gy9EQf03FaC5R/XK3LGt 34+6YQzy7MKI31d2l4ru7+5wvpa769J89UtoUHyCqA5Ie7oOOiCDKaclgTDzvcrffmtY c9UFxxTh+n/E/3DBq+q/EkPPLLANyCX6Ia3dQWXwEEoyVZ3JhI3Q7+EUbKmZB1yob3Df rj0gVKG98Nkf10bdoTf43cIfwwYrZYe2ad6ExVNheohwS0HpAcOhkrg8ZTcIysb6Mt/T knGYfkxF547Jz/pqxpdhtzUHNHoXfblXEn+cJXM7zNjNPsNg/dmBmdaQRLKAg3AtQSNu FD9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version; bh=J7xXuqfykwCtxVDAzhMeaJ1p1sPJc+/o/m7algVP+XU=; b=lrEQRG9bSWxxHvjieRZEA+a2Q7Jc1mWWLrSvZLImH3hSdbjdoHzk7NRx/sw6DDq7zl 6EWob2+jrHWJL3YZCYO+qAqK0LTKq6UpCq5EPpnmVeiTp1mp7ql6GpxPqL+b1wg61cl6 k/PtZdTW7cTHijYbXLFMFLQUf3uGhr2aqahD8oeLQcnlDSS7WuIdz8VIP7Vo9u/KvAkW e9thjB4Nhk7K6NL8qSlum46FC0yGF3Nd2nR4u/xbOt9R+9LtBH0yCv+ApL84G7Nh+TOf B8EiekPicJAHaOiMtBKvVVdEf7/31LMqrZ6K2GRDlYKL7k83cuh+NIfxUUeMQ+4qQ30M u1wA== X-Gm-Message-State: AOAM533EJivoh9mN7TexOe6W7PtcP14+aFFluzcA+lphhQKkcxwMF8wB gIFbgtX3LK3zTapW/sKdsaUXYw== X-Google-Smtp-Source: ABdhPJzSdm9kJNA7iq0MRCiU5ChVHuEyiGnzJ9xrtAreIygjRATU8KqqwoV0BFz8ybLGwsbBA4wiMw== X-Received: by 2002:a37:5d4:0:b0:508:18c2:30c1 with SMTP id 203-20020a3705d4000000b0050818c230c1mr17681554qkf.376.1646271998161; Wed, 02 Mar 2022 17:46:38 -0800 (PST) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t66-20020ae9df45000000b0064915aff85fsm411000qkf.45.2022.03.02.17.46.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Mar 2022 17:46:37 -0800 (PST) Date: Wed, 2 Mar 2022 17:46:35 -0800 (PST) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: "Kirill A. Shutemov" , Yang Shi , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH mmotm] mm/thp: fix NR_FILE_MAPPED accounting in page_*_file_rmap() Message-ID: MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 25BC920009 X-Rspam-User: Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LxbbblGv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf31.hostedemail.com: domain of hughd@google.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=hughd@google.com X-Stat-Signature: gxowco3ptwod4gef7pdhjss4sgtr48pp X-HE-Tag: 1646271998-824985 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: NR_FILE_MAPPED accounting in mm/rmap.c (for /proc/meminfo "Mapped" and /proc/vmstat "nr_mapped" and the memcg's memory.stat "mapped_file") is slightly flawed for file or shmem huge pages. It is well thought out, and looks convincing, but there's a racy case when the careful counting in page_remove_file_rmap() (without page lock) gets discarded. So that in a workload like two "make -j20" kernel builds under memory pressure, with cc1 on hugepage text, "Mapped" can easily grow by a spurious 5MB or more on each iteration, ending up implausibly bigger than most other numbers in /proc/meminfo. And, hypothetically, might grow to the point of seriously interfering in mm/vmscan.c's heuristics, which do take NR_FILE_MAPPED into some consideration. Fixed by moving the __mod_lruvec_page_state() down to where it will not be missed before return (and I've grown a bit tired of that oft-repeated but-not-everywhere comment on the __ness: it gets lost in the move here). Does page_add_file_rmap() need the same change? I suspect not, because page lock is held in all relevant cases, and its skipping case looks safe; but it's much easier to be sure, if we do make the same change. Fixes: dd78fedde4b9 ("rmap: support file thp") Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- If this were thought serious enough to backport (I don't feel strongly, but it is something I keep in my own trees), it needs a little more care near "out", because the mm/munlock series has removed some action there. mm/rmap.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1238,14 +1238,14 @@ void page_add_new_anon_rmap(struct page *page, void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { - int i, nr = 1; + int i, nr = 0; VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page); lock_page_memcg(page); if (compound && PageTransHuge(page)) { int nr_pages = thp_nr_pages(page); - for (i = 0, nr = 0; i < nr_pages; i++) { + for (i = 0; i < nr_pages; i++) { if (atomic_inc_and_test(&page[i]._mapcount)) nr++; } @@ -1262,11 +1262,12 @@ void page_add_file_rmap(struct page *page, VM_WARN_ON_ONCE(!PageLocked(page)); SetPageDoubleMap(compound_head(page)); } - if (!atomic_inc_and_test(&page->_mapcount)) - goto out; + if (atomic_inc_and_test(&page->_mapcount)) + nr++; } - __mod_lruvec_page_state(page, NR_FILE_MAPPED, nr); out: + if (nr) + __mod_lruvec_page_state(page, NR_FILE_MAPPED, nr); unlock_page_memcg(page); mlock_vma_page(page, vma, compound); @@ -1274,7 +1275,7 @@ void page_add_file_rmap(struct page *page, static void page_remove_file_rmap(struct page *page, bool compound) { - int i, nr = 1; + int i, nr = 0; VM_BUG_ON_PAGE(compound && !PageHead(page), page); @@ -1289,12 +1290,12 @@ static void page_remove_file_rmap(struct page *page, bool compound) if (compound && PageTransHuge(page)) { int nr_pages = thp_nr_pages(page); - for (i = 0, nr = 0; i < nr_pages; i++) { + for (i = 0; i < nr_pages; i++) { if (atomic_add_negative(-1, &page[i]._mapcount)) nr++; } if (!atomic_add_negative(-1, compound_mapcount_ptr(page))) - return; + goto out; if (PageSwapBacked(page)) __mod_lruvec_page_state(page, NR_SHMEM_PMDMAPPED, -nr_pages); @@ -1302,16 +1303,12 @@ static void page_remove_file_rmap(struct page *page, bool compound) __mod_lruvec_page_state(page, NR_FILE_PMDMAPPED, -nr_pages); } else { - if (!atomic_add_negative(-1, &page->_mapcount)) - return; + if (atomic_add_negative(-1, &page->_mapcount)) + nr++; } - - /* - * We use the irq-unsafe __{inc|mod}_lruvec_page_state because - * these counters are not modified in interrupt context, and - * pte lock(a spinlock) is held, which implies preemption disabled. - */ - __mod_lruvec_page_state(page, NR_FILE_MAPPED, -nr); +out: + if (nr) + __mod_lruvec_page_state(page, NR_FILE_MAPPED, -nr); } static void page_remove_anon_compound_rmap(struct page *page)