From patchwork Mon Feb 14 07:09:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12745059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38148C433F5 for ; Mon, 14 Feb 2022 07:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FB0A6B0072; Mon, 14 Feb 2022 02:09:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 983786B0073; Mon, 14 Feb 2022 02:09:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 823B96B0078; Mon, 14 Feb 2022 02:09:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0004.hostedemail.com [216.40.44.4]) by kanga.kvack.org (Postfix) with ESMTP id 6F3656B0072 for ; Mon, 14 Feb 2022 02:09:34 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2015F181AC9C6 for ; Mon, 14 Feb 2022 07:09:34 +0000 (UTC) X-FDA: 79140509868.19.2C93D1A Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) by imf19.hostedemail.com (Postfix) with ESMTP id 9BFF51A0005 for ; Mon, 14 Feb 2022 07:09:33 +0000 (UTC) Received: by mail-qv1-f41.google.com with SMTP id a28so14045778qvb.10 for ; Sun, 13 Feb 2022 23:09:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=SEHRUOH5N9XtoyGwgSgWRzInurRzgz1DBRcTDOVUssY=; b=fYylqBx1VbPSDvQ1vsXVYifG3U4E5UUlQfhYmCvW9f8OQg5HqmrmXpmU1ZZw/wPM54 eeIHBwFNApBDtRdRkud+nFh7AVqz0BVnnTXxAn+aSO8v3/HJP4lWL02MVthAOwFkayOZ 9zXNwElwbO2hgsOfJMFGa3z1t2QyJCpVAPMINKwDTDSfgEnmY0jy3HgKReSyZSQtuC0C 73mapZRUrwuz1l20M2ad9dsk+VeKYR3XPY+EKIitF/kWptZfgSP+JBU6x8/8mN+4o/ef WvyMK3lDAhLboZ1QsByFFBep96wbOCCCUPWNxnntm5T9a4bupBXDiSNGbiMt8qj62C0L Pj9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=SEHRUOH5N9XtoyGwgSgWRzInurRzgz1DBRcTDOVUssY=; b=xyE8cbLrDaGNFZAkx9kVdcAbSMxBCpDvJoptDTg+Cr+u2oCVRTZSlNaoTmx0HLzgF1 X52hxSYAJWU4EDXbIs9XRwM5PNSf7DfBJyogPjWqmn9G7fYUwv3QZDElakjIt3AYvGjB LBA7luZ02KosRget1t5MLaxgjDEAcdy1QhUgdgZRnXxwjRKXhvz7ivxUT99cHLA4gFNe 5vCzhupyTUlMYtDjfwHauuua1tZ45qhmfriDEFdN8aK8MJkozU6jjnuD4cpvaDIiWzKL XZQEtxEt43TyWYB6ocwmfSbLcmvlsy/Rks0OHsn2k81aIkT7AShU9+VIrRxk/NUGbXp5 b1bA== X-Gm-Message-State: AOAM533YqoOMtKItyJ7u8ICZ1I0wX/ql22FzQ6UHFNLV3lC9Jamwrczv dOuqq5t9hJEzwnf4tGBaclPaQw== X-Google-Smtp-Source: ABdhPJwVQQ6IpmghCfwP/T33P9ZmisG3yFHJDZD1+EC+aKxPwikF8R+8CSu1N3G7kYDsBCZy5hFivQ== X-Received: by 2002:a05:6214:21a9:: with SMTP id t9mr333224qvc.129.1644822572786; Sun, 13 Feb 2022 23:09:32 -0800 (PST) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id g21sm17529350qtb.70.2022.02.13.23.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Feb 2022 23:09:31 -0800 (PST) Date: Sun, 13 Feb 2022 23:09:29 -0800 (PST) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , Vlastimil Babka , Michal Hocko , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Alistair Popple , Johannes Weiner , Rik van Riel , Suren Baghdasaryan , Yu Zhao , Greg Thelen , Shakeel Butt , Hillf Danton , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 07/13] mm/munlock: mlock_pte_range() when mlocking or munlocking In-Reply-To: Message-ID: <4b27d14-31e5-81e2-3732-b6ea1ff36b6@google.com> References: <8e4356d-9622-a7f0-b2c-f116b5f2efea@google.com> <8bc3ee8c-7f1-d812-7f22-4f9f6d436bc@google.com> <2ec49f65-fe4e-26a0-4059-c18e6dab0af4@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9BFF51A0005 X-Rspam-User: Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fYylqBx1; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.219.41 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: zdoz3na5oq9x34bxqt53odogmnsm73fu X-HE-Tag: 1644822573-822312 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fill in missing pieces: reimplementation of munlock_vma_pages_range(), required to lower the mlock_counts when munlocking without munmapping; and its complement, implementation of mlock_vma_pages_range(), required to raise the mlock_counts on pages already there when a range is mlocked. Combine them into just the one function mlock_vma_pages_range(), using walk_page_range() to run mlock_pte_range(). This approach fixes the "Very slow unlockall()" of unpopulated PROT_NONE areas, reported in https://lore.kernel.org/linux-mm/70885d37-62b7-748b-29df-9e94f3291736@gmail.com/ Munlock clears VM_LOCKED at the start, under exclusive mmap_lock; but if a racing truncate or holepunch (depending on i_mmap_rwsem) gets to the pte first, it will not try to munlock the page: leaving release_pages() to correct it when the last reference to the page is gone - that's okay, a page is not evictable anyway while it is held by an extra reference. Mlock sets VM_LOCKED at the start, under exclusive mmap_lock; but if a racing remove_migration_pte() or try_to_unmap_one() (depending on i_mmap_rwsem) gets to the pte first, it will try to mlock the page, then mlock_pte_range() mlock it a second time. This is harder to reproduce, but a more serious race because it could leave the page unevictable indefinitely though the area is munlocked afterwards. Guard against it by setting the (inappropriate) VM_IO flag, and modifying mlock_vma_page() to decline such vmas. Signed-off-by: Hugh Dickins Acked-by: Vlastimil Babka --- v2: Delete reintroduced VM_LOCKED_CLEAR_MASKing here, per Vlastimil. Use oldflags instead of vma->vm_flags in mlock_fixup(), per Vlastimil. Make clearer comment on mmap_lock and WRITE_ONCE, per Hillf. mm/internal.h | 3 +- mm/mlock.c | 111 ++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 91 insertions(+), 23 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index a43d79335c16..b3f0dd3ffba2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -412,7 +412,8 @@ void mlock_page(struct page *page); static inline void mlock_vma_page(struct page *page, struct vm_area_struct *vma, bool compound) { - if (unlikely(vma->vm_flags & VM_LOCKED) && + /* VM_IO check prevents migration from double-counting during mlock */ + if (unlikely((vma->vm_flags & (VM_LOCKED|VM_IO)) == VM_LOCKED) && (compound || !PageTransCompound(page))) mlock_page(page); } diff --git a/mm/mlock.c b/mm/mlock.c index f8a3a54687dd..581ea8bf1b83 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -127,25 +128,91 @@ void munlock_page(struct page *page) unlock_page_memcg(page); } +static int mlock_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) + +{ + struct vm_area_struct *vma = walk->vma; + spinlock_t *ptl; + pte_t *start_pte, *pte; + struct page *page; + + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + if (!pmd_present(*pmd)) + goto out; + if (is_huge_zero_pmd(*pmd)) + goto out; + page = pmd_page(*pmd); + if (vma->vm_flags & VM_LOCKED) + mlock_page(page); + else + munlock_page(page); + goto out; + } + + start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { + if (!pte_present(*pte)) + continue; + page = vm_normal_page(vma, addr, *pte); + if (!page) + continue; + if (PageTransCompound(page)) + continue; + if (vma->vm_flags & VM_LOCKED) + mlock_page(page); + else + munlock_page(page); + } + pte_unmap(start_pte); +out: + spin_unlock(ptl); + cond_resched(); + return 0; +} + /* - * munlock_vma_pages_range() - munlock all pages in the vma range.' - * @vma - vma containing range to be munlock()ed. + * mlock_vma_pages_range() - mlock any pages already in the range, + * or munlock all pages in the range. + * @vma - vma containing range to be mlock()ed or munlock()ed * @start - start address in @vma of the range - * @end - end of range in @vma. - * - * For mremap(), munmap() and exit(). + * @end - end of range in @vma + * @newflags - the new set of flags for @vma. * - * Called with @vma VM_LOCKED. - * - * Returns with VM_LOCKED cleared. Callers must be prepared to - * deal with this. + * Called for mlock(), mlock2() and mlockall(), to set @vma VM_LOCKED; + * called for munlock() and munlockall(), to clear VM_LOCKED from @vma. */ -static void munlock_vma_pages_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) +static void mlock_vma_pages_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, vm_flags_t newflags) { - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + static const struct mm_walk_ops mlock_walk_ops = { + .pmd_entry = mlock_pte_range, + }; - /* Reimplementation to follow in later commit */ + /* + * There is a slight chance that concurrent page migration, + * or page reclaim finding a page of this now-VM_LOCKED vma, + * will call mlock_vma_page() and raise page's mlock_count: + * double counting, leaving the page unevictable indefinitely. + * Communicate this danger to mlock_vma_page() with VM_IO, + * which is a VM_SPECIAL flag not allowed on VM_LOCKED vmas. + * mmap_lock is held in write mode here, so this weird + * combination should not be visible to other mmap_lock users; + * but WRITE_ONCE so rmap walkers must see VM_IO if VM_LOCKED. + */ + if (newflags & VM_LOCKED) + newflags |= VM_IO; + WRITE_ONCE(vma->vm_flags, newflags); + + lru_add_drain(); + walk_page_range(vma->vm_mm, start, end, &mlock_walk_ops, NULL); + lru_add_drain(); + + if (newflags & VM_IO) { + newflags &= ~VM_IO; + WRITE_ONCE(vma->vm_flags, newflags); + } } /* @@ -164,10 +231,9 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, pgoff_t pgoff; int nr_pages; int ret = 0; - int lock = !!(newflags & VM_LOCKED); - vm_flags_t old_flags = vma->vm_flags; + vm_flags_t oldflags = vma->vm_flags; - if (newflags == vma->vm_flags || (vma->vm_flags & VM_SPECIAL) || + if (newflags == oldflags || (oldflags & VM_SPECIAL) || is_vm_hugetlb_page(vma) || vma == get_gate_vma(current->mm) || vma_is_dax(vma) || vma_is_secretmem(vma)) /* don't set VM_LOCKED or VM_LOCKONFAULT and don't count */ @@ -199,9 +265,9 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, * Keep track of amount of locked VM. */ nr_pages = (end - start) >> PAGE_SHIFT; - if (!lock) + if (!(newflags & VM_LOCKED)) nr_pages = -nr_pages; - else if (old_flags & VM_LOCKED) + else if (oldflags & VM_LOCKED) nr_pages = 0; mm->locked_vm += nr_pages; @@ -211,11 +277,12 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, * set VM_LOCKED, populate_vma_page_range will bring it back. */ - if (lock) + if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { + /* No work to do, and mlocking twice would be wrong */ vma->vm_flags = newflags; - else - munlock_vma_pages_range(vma, start, end); - + } else { + mlock_vma_pages_range(vma, start, end, newflags); + } out: *prev = vma; return ret;