From patchwork Sat Jun 22 06:48:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ge Yang X-Patchwork-Id: 13708248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EDBAC27C53 for ; Sat, 22 Jun 2024 06:48:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C41A8E0002; Sat, 22 Jun 2024 02:48:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6747B8D01A5; Sat, 22 Jun 2024 02:48:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53BB18E0002; Sat, 22 Jun 2024 02:48:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 35C3B8D01A5 for ; Sat, 22 Jun 2024 02:48:23 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7F28C121433 for ; Sat, 22 Jun 2024 06:48:22 +0000 (UTC) X-FDA: 82257595644.23.CCAB82C Received: from m16.mail.126.com (m16.mail.126.com [220.197.31.9]) by imf26.hostedemail.com (Postfix) with ESMTP id 7BBE0140010 for ; Sat, 22 Jun 2024 06:48:19 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=126.com header.s=s110527 header.b="D/eTzu26"; spf=pass (imf26.hostedemail.com: domain of yangge1116@126.com designates 220.197.31.9 as permitted sender) smtp.mailfrom=yangge1116@126.com; dmarc=pass (policy=none) header.from=126.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719038890; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:dkim-signature; bh=4PQv0PvTVRL/ItON0fL1teaBFOQB7W1EZ8kO5egpB0c=; b=gpuY/xAozVF0slEG5weoB7WJV0Aqyn0fClHZadLP/vaEAF3LToEWM4sWizic+ZSkzly6TN 6KuGHmPs7j+n3RdIlgVyyXF0oCOzU9vJKgQ/LVcAvVqpQ6TnLfYvNIkLy+rFunZ9/Lmw8Y i0WpqF00QjuqzKU67H3tIIesi84qir0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719038890; a=rsa-sha256; cv=none; b=Jx37XxJkrhmpYiz5Y16dWqGHBTQocG/ya3rZgXiwfF51p2IQmR3jY5zMli+TuchGpfBuFB pdfh9D9r0vpFpK5FHJgw2Pkzz1jljK8GtAsMveaWrWt9N8YLtJAzKZQNS2Ui4L3Ju9SCGz kEbpazc0C5AEPQfJ8GNAlLqJ7ckr7KA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=126.com header.s=s110527 header.b="D/eTzu26"; spf=pass (imf26.hostedemail.com: domain of yangge1116@126.com designates 220.197.31.9 as permitted sender) smtp.mailfrom=yangge1116@126.com; dmarc=pass (policy=none) header.from=126.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=From:Subject:Date:Message-Id; bh=4PQv0PvTVRL/ItON0f L1teaBFOQB7W1EZ8kO5egpB0c=; b=D/eTzu26yO2HFv+BJzfEzXGSpFAl8iajwD GAbph4rFmCFJAHeu3KgkSkYg8qnIu8Bd3rmez+K5SenSs77ctEsDdN1Xly4ksoze eCe830XisU6FxLBg65QOCmWfZogVmQ1RL4CgVJCjOJoWTERFAYYX/8LwGV+uVKHu Ekcfl5ijQ= Received: from hg-OptiPlex-7040.hygon.cn (unknown [118.242.3.34]) by gzga-smtp-mta-g1-0 (Coremail) with SMTP id _____wDH7WCmc3ZmqHYEAA--.12787S2; Sat, 22 Jun 2024 14:48:07 +0800 (CST) From: yangge1116@126.com To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, 21cnbao@gmail.com, david@redhat.com, baolin.wang@linux.alibaba.com, liuzixing@hygon.cn, yangge Subject: [PATCH V2] mm/gup: Clear the LRU flag of a page before adding to LRU batch Date: Sat, 22 Jun 2024 14:48:04 +0800 Message-Id: <1719038884-1903-1-git-send-email-yangge1116@126.com> X-Mailer: git-send-email 2.7.4 X-CM-TRANSID: _____wDH7WCmc3ZmqHYEAA--.12787S2 X-Coremail-Antispam: 1Uf129KBjvJXoW3WF4rXF4xZrWUGFWUJFyUtrb_yoW7XF1xpF W7Gr9IqF4DGFnrWr47Xw15Jr1Yk393Xa1UJFWxGry7AF15Xw1qkF1xtw1UJa9xJryruFn3 Z3W8JF1vgF1UAF7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jjLvNUUUUU= X-Originating-IP: [118.242.3.34] X-CM-SenderInfo: 51dqwwjhrrila6rslhhfrp/1tbiOggGG2VEw83b2AAAsD X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7BBE0140010 X-Stat-Signature: yg4wrnsium4t1qrkd35setchkxgxhggx X-HE-Tag: 1719038899-786667 X-HE-Meta: U2FsdGVkX1+Nut66SuZhBQfzrKzFSwHgk51eOjSdQ8GCH8ZRmAWO4LvsmA6PaI16BxNvEjH73DJDAqlp6k1YjQpQRsMehVKK7G/4ZNTfl5RmOxsebcyhNwxeCnDe+OOOMi5wh9AoX1tweQpNAX+FsGhNLNXZ+Oq431v3FMNdglwnZObzgJW27lFeuPBJ/EBq0gWWuQBnPkOPDFQpVVAklXDAolSL722FHas4nDNr8MgbqylDrCFTwWJHlMLAQ+197sX8LpIWOUP7SKTl/XuQ/nyti8wnp2K/MxcZfKvMXQb0eoJRo58cT36MvUyafsrUvmQlFCz/P6qpBujvLr1YuMF3hizOkw4WBWmBe0QYijKQF800q8ZQg99bp90v12YRahGWXDzdFkjTxaIj8jWwwipU8OcYGglwEvLmPaOBbMv86+tmwcTHgFF5B2OC8cKrqnrfCPDpz/CuHwIjuqSxRE1AChCSJMh/Z/7cSsPbQMn4uULjsGpamybezXSPWJkok9xIHbaEwYR5hnwsZtWUTWPIVK1L7k5PJsuaazDqS+MBJjJyvM0DqXR1RV3c/I1TeArP8pcarZVGZ+D7D3yo+vi8gOiAUw97f7UGqvkjx33roZhV4A2/mhHocoQTa9aCQB+Ph9kzZPFMyf8Vc3MKgRSMd7sJksRab3HTAL9O9tP7AebVI8yWM9v2lonIJlpW7zQLNM5lGZSoM6DKAftZmkl5n/sc/bGmBs9XxFdfFDoEsYGCW+U26YQ7KVULiH4UTkcT50E/I+uSZUDbDugtOs81YwZrZVL9UYzhmLiwgWXRMhUccApEjKNp4SyZSqBhPzqwz9QXc3S/IUaFCeFPUyQDKRyRmmbhnR4vsS0gOKLjUlCDTt0s0EjFD2vHvv25okgNnpFHAoajP9ds/pItV3L20sIcXTBNCEQyXB61vjnWVAU3cDumBqJv/P7T8Pa1RwPwwyxWRNzBj8lpp2k CiewraIm pvAJaHXlslr9G3wKNq7nMsE5thwGhfgrpSp0TFRuDBeG71RaF32eRBc+mO71lJ0vtBBr8TNWK81pvPa70PPNmA/pnkDav/drpgB45HWni6mDZ4CNug+K+YK+x/Z1r0QvEfzE5dA2j0bpt7CuCEGX9fBku22XINbex5PXjG504RBxhB4EOEDQ+OMzN1zInfYhywhgNKvjEli3ZRctBEgBeNKMIlfRwXzkGmPxOIK7YBmk7aWGRj7XozMqI23KK/1zALZ5QdeVvgji9xgeGuuIVAr2+bAxGQdUrQbwG702s5eVZNMAUZQdFUnSaPUFAyyzeR9/kU7ZeqsDj6tAu+bliEPFWzg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: yangge If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But the current code will cause the migration failure due to unexpected page refcounts, and eventually cause the virtual machine fail to start. If a page is added in LRU batch, its refcount increases one, remove the page from LRU batch decreases one. Page migration requires the page is not referenced by others except page mapping. Before migrating a page, we should try to drain the page from LRU batch in case the page is in it, however, folio_test_lru() is not sufficient to tell whether the page is in LRU batch or not, if the page is in LRU batch, the migration will fail. To solve the problem above, we modify the logic of adding to LRU batch. Before adding a page to LRU batch, we clear the LRU flag of the page so that we can check whether the page is in LRU batch by folio_test_lru(page). Seems making the LRU flag of the page invisible a long time is no problem, because a new page is allocated from buddy and added to the lru batch, its LRU flag is also not visible for a long time. Cc: Signed-off-by: yangge Tested-by: Chris Li --- mm/swap.c | 43 +++++++++++++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 12 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index dc205bd..9caf6b0 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -211,10 +211,6 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) for (i = 0; i < folio_batch_count(fbatch); i++) { struct folio *folio = fbatch->folios[i]; - /* block memcg migration while the folio moves between lru */ - if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) - continue; - folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); @@ -255,11 +251,16 @@ static void lru_move_tail_fn(struct lruvec *lruvec, struct folio *folio) void folio_rotate_reclaimable(struct folio *folio) { if (!folio_test_locked(folio) && !folio_test_dirty(folio) && - !folio_test_unevictable(folio) && folio_test_lru(folio)) { + !folio_test_unevictable(folio)) { struct folio_batch *fbatch; unsigned long flags; folio_get(folio); + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + local_lock_irqsave(&lru_rotate.lock, flags); fbatch = this_cpu_ptr(&lru_rotate.fbatch); folio_batch_add_and_move(fbatch, folio, lru_move_tail_fn); @@ -352,11 +353,15 @@ static void folio_activate_drain(int cpu) void folio_activate(struct folio *folio) { - if (folio_test_lru(folio) && !folio_test_active(folio) && - !folio_test_unevictable(folio)) { + if (!folio_test_active(folio) && !folio_test_unevictable(folio)) { struct folio_batch *fbatch; folio_get(folio); + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + local_lock(&cpu_fbatches.lock); fbatch = this_cpu_ptr(&cpu_fbatches.activate); folio_batch_add_and_move(fbatch, folio, folio_activate_fn); @@ -700,6 +705,11 @@ void deactivate_file_folio(struct folio *folio) return; folio_get(folio); + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + local_lock(&cpu_fbatches.lock); fbatch = this_cpu_ptr(&cpu_fbatches.lru_deactivate_file); folio_batch_add_and_move(fbatch, folio, lru_deactivate_file_fn); @@ -716,11 +726,16 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_lru(folio) && !folio_test_unevictable(folio) && - (folio_test_active(folio) || lru_gen_enabled())) { + if (!folio_test_unevictable(folio) && (folio_test_active(folio) || + lru_gen_enabled())) { struct folio_batch *fbatch; folio_get(folio); + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + local_lock(&cpu_fbatches.lock); fbatch = this_cpu_ptr(&cpu_fbatches.lru_deactivate); folio_batch_add_and_move(fbatch, folio, lru_deactivate_fn); @@ -737,12 +752,16 @@ void folio_deactivate(struct folio *folio) */ void folio_mark_lazyfree(struct folio *folio) { - if (folio_test_lru(folio) && folio_test_anon(folio) && - folio_test_swapbacked(folio) && !folio_test_swapcache(folio) && - !folio_test_unevictable(folio)) { + if (folio_test_anon(folio) && folio_test_swapbacked(folio) && + !folio_test_swapcache(folio) && !folio_test_unevictable(folio)) { struct folio_batch *fbatch; folio_get(folio); + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + local_lock(&cpu_fbatches.lock); fbatch = this_cpu_ptr(&cpu_fbatches.lru_lazyfree); folio_batch_add_and_move(fbatch, folio, lru_lazyfree_fn);