From patchwork Tue Feb 15 02:38:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12746455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2445BC433F5 for ; Tue, 15 Feb 2022 02:38:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2A436B0078; Mon, 14 Feb 2022 21:38:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DB576B007B; Mon, 14 Feb 2022 21:38:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 854596B007D; Mon, 14 Feb 2022 21:38:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 767626B0078 for ; Mon, 14 Feb 2022 21:38:52 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 34380181AC9C6 for ; Tue, 15 Feb 2022 02:38:52 +0000 (UTC) X-FDA: 79143456504.28.1FAC599 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf28.hostedemail.com (Postfix) with ESMTP id AD1BCC0003 for ; Tue, 15 Feb 2022 02:38:51 +0000 (UTC) Received: by mail-qt1-f169.google.com with SMTP id s1so17257603qtw.9 for ; Mon, 14 Feb 2022 18:38:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=7RY/qhKlJPQeH1IwAyGGlnES1Mny5ocT5QtyB/fayZE=; b=CssZuZOe3DBbjf8JJaBjw58IQNx1PjFb/exoxSarxXEXb0Oku+9ZRuCUj5Z7Ybceav LxQzjdOLq1kQgG8GGiKW3Qcn+MN7WOn76nP1logW5v3dLX60adeSoNzxXJZkNF2yUrwx XIP/VjCPuz4BreDgp6azxwICU4A7RuibWm8BmOIru5OKZPerzNFnTXIE3RgfavBFXe4E L7GwFOHtBTXGz0268x1zY/1+ABnLkf1a3F32q/cqc8MXzk5h1SULSzWDWIgkwDyxJKDQ RsslYR7OvL/db5PKlOCiW1UED3kwElfK3o6xq0CcIoCuGhhXPzPh8MgCllnZA31k3/Uj c6MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=7RY/qhKlJPQeH1IwAyGGlnES1Mny5ocT5QtyB/fayZE=; b=bP7i80+XD/bjZmQ9jU0N8TGO9+OkY2MXP3d4Eg2RjLswFyO5g0SUeRjRgsoo+5j18D C2cnoc3F94YntYKrZH4wUdWCUpfSz4D3FDUp/xTaDrjvqB6UuIhkx6pFOTe6Uf4yxhcc oW3bpgMaj9lKYKUVhCSv5Ac2nDv4qBShQZdEDi44qhh0KHtC7hsaTdhu4UpF5uh5Qijm mmeTYCbW0XlOgg4YqMgDb6Ik1lPO1IYZYhtNlNyvVO5xqElL/lxN6WBe7PSQGzSy6MYn CedGCUMRn8ORO0fuALZWZJuuVUvOPxxNGTZhY2BCtGUX/HgDmksljUuiEsrYzTX+ouKk bB/A== X-Gm-Message-State: AOAM533X2SpsdN2Qci1hc5J3hQHcjHLqjlpa+LL+LjqPQ2T9kDiNVsDX YvkWPwIPAcDXNJk3DD+uCH3BQw== X-Google-Smtp-Source: ABdhPJxVwCU2R6EW1pfoD9ezIoDE5R0PkX8gv3sUUegKvg3dhIR6WvgfD5QXF7JGwdRw6BNI9vYXVA== X-Received: by 2002:a05:622a:1ce:: with SMTP id t14mr1338296qtw.25.1644892730863; Mon, 14 Feb 2022 18:38:50 -0800 (PST) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r13sm10993253qkp.129.2022.02.14.18.38.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Feb 2022 18:38:50 -0800 (PST) Date: Mon, 14 Feb 2022 18:38:47 -0800 (PST) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Michal Hocko , Vlastimil Babka , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Alistair Popple , Johannes Weiner , Rik van Riel , Suren Baghdasaryan , Yu Zhao , Greg Thelen , Shakeel Butt , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 11/13] mm/munlock: page migration needs mlock pagevec drained In-Reply-To: <55a49083-37f9-3766-1de9-9feea7428ac@google.com> Message-ID: <9e2ed861-951a-6e86-e298-a09d2d8e9b9f@google.com> References: <55a49083-37f9-3766-1de9-9feea7428ac@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AD1BCC0003 X-Stat-Signature: ib94wdx35xj7mkxcqjek8cfwiwz6uxc5 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CssZuZOe; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1644892731-999795 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page migration of a VM_LOCKED page tends to fail, because when the old page is unmapped, it is put on the mlock pagevec with raised refcount, which then fails the freeze. At first I thought this would be fixed by a local mlock_page_drain() at the upper rmap_walk() level - which would have nicely batched all the munlocks of that page; but tests show that the task can too easily move to another cpu, leaving pagevec residue behind which fails the migration. So try_to_migrate_one() drain the local pagevec after page_remove_rmap() from a VM_LOCKED vma; and do the same in try_to_unmap_one(), whose TTU_IGNORE_MLOCK users would want the same treatment; and do the same in remove_migration_pte() - not important when successfully inserting a new page, but necessary when hoping to retry after failure. Any new pagevec runs the risk of adding a new way of stranding, and we might discover other corners where mlock_page_drain() or lru_add_drain() would now help. Signed-off-by: Hugh Dickins Acked-by: Vlastimil Babka --- v2: Delete suggestion of sysctl from changelog, per Vlastimil. mm/migrate.c | 2 ++ mm/rmap.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/mm/migrate.c b/mm/migrate.c index f4bcf1541b62..e7d0b68d5dcb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -251,6 +251,8 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, page_add_file_rmap(new, vma, false); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } + if (vma->vm_flags & VM_LOCKED) + mlock_page_drain(smp_processor_id()); /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, pvmw.address, pvmw.pte); diff --git a/mm/rmap.c b/mm/rmap.c index 5442a5c97a85..714bfdc72c7b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1656,6 +1656,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * See Documentation/vm/mmu_notifier.rst */ page_remove_rmap(subpage, vma, PageHuge(page)); + if (vma->vm_flags & VM_LOCKED) + mlock_page_drain(smp_processor_id()); put_page(page); } @@ -1930,6 +1932,8 @@ static bool try_to_migrate_one(struct page *page, struct vm_area_struct *vma, * See Documentation/vm/mmu_notifier.rst */ page_remove_rmap(subpage, vma, PageHuge(page)); + if (vma->vm_flags & VM_LOCKED) + mlock_page_drain(smp_processor_id()); put_page(page); }