From patchwork Tue Feb 2 06:55:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Goldsworthy X-Patchwork-Id: 12060815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 252E8C433DB for ; Tue, 2 Feb 2021 06:56:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B28964EDF for ; Tue, 2 Feb 2021 06:56:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B28964EDF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 331A26B0096; Tue, 2 Feb 2021 01:56:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E2596B0098; Tue, 2 Feb 2021 01:56:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F7F16B0099; Tue, 2 Feb 2021 01:56:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0038.hostedemail.com [216.40.44.38]) by kanga.kvack.org (Postfix) with ESMTP id F35786B0096 for ; Tue, 2 Feb 2021 01:56:05 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C2C8B8249980 for ; Tue, 2 Feb 2021 06:56:05 +0000 (UTC) X-FDA: 77772418290.26.debt36_2f10607275c8 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 9969D1804B656 for ; Tue, 2 Feb 2021 06:56:05 +0000 (UTC) X-HE-Tag: debt36_2f10607275c8 X-Filterd-Recvd-Size: 8032 Received: from so15.mailgun.net (so15.mailgun.net [198.61.254.15]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Tue, 2 Feb 2021 06:56:03 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1612248965; h=References: In-Reply-To: References: In-Reply-To: Message-Id: Date: Subject: Cc: To: From: Sender; bh=0GJGgAjAx2/rdTTd5QE4pvrmTTzFGOM9WmXIW06Tfs4=; b=sQJ2EI/0NtN5SNeFtrYuW/KjgBSmH88opE9OzhWbMME6d65XdhWKuVJYzYb99pAs6UqULHSq 0S73fCMFp56Gf2o5vAjEOpEr8kIHvSn7N+vy+84lrjgoEcohVQaDngc2cYr1TZ+q+9HhxJtd oyGB1eF3qtzNlez9K2tj0WdS1BE= X-Mailgun-Sending-Ip: 198.61.254.15 X-Mailgun-Sid: WyIwY2Q3OCIsICJsaW51eC1tbUBrdmFjay5vcmciLCAiYmU5ZTRhIl0= Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n05.prod.us-east-1.postgun.com with SMTP id 6018f77f6dd11a022158ba75 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Tue, 02 Feb 2021 06:55:59 GMT Received: by smtp.codeaurora.org (Postfix, from userid 1001) id C7613C43462; Tue, 2 Feb 2021 06:55:58 +0000 (UTC) Received: from cgoldswo-linux.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cgoldswo) by smtp.codeaurora.org (Postfix) with ESMTPSA id 00D56C433C6; Tue, 2 Feb 2021 06:55:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 00D56C433C6 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=fail smtp.mailfrom=cgoldswo@codeaurora.org From: Chris Goldsworthy To: Andrew Morton , Alexander Viro Cc: Minchan Kim , Matthew Wilcox , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chris Goldsworthy Subject: [PATCH] [RFC] mm: fs: Invalidate BH LRU during page migration Date: Mon, 1 Feb 2021 22:55:47 -0800 Message-Id: <695193a165bf538f35de84334b4da2cc3544abe0.1612248395.git.cgoldswo@codeaurora.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: In-Reply-To: References: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pages containing buffer_heads that are in the buffer_head LRU cache will be pinned and thus cannot be migrated. Correspondingly, invalidate the BH LRU before a migration starts and stop any buffer_head from being cached in the LRU, until migration has finished. Signed-off-by: Chris Goldsworthy Cc: Minchan Kim Cc: Matthew Wilcox --- fs/buffer.c | 6 ++++++ include/linux/buffer_head.h | 3 +++ include/linux/migrate.h | 2 ++ mm/migrate.c | 18 ++++++++++++++++++ mm/page_alloc.c | 3 +++ mm/swap.c | 3 +++ 6 files changed, 35 insertions(+) diff --git a/fs/buffer.c b/fs/buffer.c index 96c7604..39ec4ec 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1289,6 +1289,8 @@ static inline void check_irqs_on(void) #endif } +bool bh_migration_done = true; + /* * Install a buffer_head into this cpu's LRU. If not already in the LRU, it is * inserted at the front, and the buffer_head at the back if any is evicted. @@ -1303,6 +1305,9 @@ static void bh_lru_install(struct buffer_head *bh) check_irqs_on(); bh_lru_lock(); + if (!bh_migration_done) + goto out; + b = this_cpu_ptr(&bh_lrus); for (i = 0; i < BH_LRU_SIZE; i++) { swap(evictee, b->bhs[i]); @@ -1313,6 +1318,7 @@ static void bh_lru_install(struct buffer_head *bh) } get_bh(bh); +out: bh_lru_unlock(); brelse(evictee); } diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 6b47f94..ae4eb6d 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -193,6 +193,9 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, gfp_t gfp); struct buffer_head *__bread_gfp(struct block_device *, sector_t block, unsigned size, gfp_t gfp); + +extern bool bh_migration_done; + void invalidate_bh_lrus(void); struct buffer_head *alloc_buffer_head(gfp_t gfp_flags); void free_buffer_head(struct buffer_head * bh); diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3a38963..9e4a2dc 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -46,6 +46,7 @@ extern int isolate_movable_page(struct page *page, isolate_mode_t mode); extern void putback_movable_page(struct page *page); extern void migrate_prep(void); +extern void migrate_finish(void); extern void migrate_prep_local(void); extern void migrate_page_states(struct page *newpage, struct page *page); extern void migrate_page_copy(struct page *newpage, struct page *page); @@ -67,6 +68,7 @@ static inline int isolate_movable_page(struct page *page, isolate_mode_t mode) { return -EBUSY; } static inline int migrate_prep(void) { return -ENOSYS; } +static inline int migrate_finish(void) { return -ENOSYS; } static inline int migrate_prep_local(void) { return -ENOSYS; } static inline void migrate_page_states(struct page *newpage, struct page *page) diff --git a/mm/migrate.c b/mm/migrate.c index a69da8a..08c981d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -64,6 +64,19 @@ */ void migrate_prep(void) { + bh_migration_done = false; + + /* + * This barrier ensures that callers of bh_lru_install() between + * the barrier and the call to invalidate_bh_lrus() read + * bh_migration_done() as false. + */ + /* + * TODO: Remove me? lru_add_drain_all() already has an smp_mb(), + * but it would be good to ensure that the barrier isn't forgotten. + */ + smp_mb(); + /* * Clear the LRU lists so pages can be isolated. * Note that pages may be moved off the LRU after we have @@ -73,6 +86,11 @@ void migrate_prep(void) lru_add_drain_all(); } +void migrate_finish(void) +{ + bh_migration_done = true; +} + /* Do the necessary work of migrate_prep but not if it involves other CPUs */ void migrate_prep_local(void) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6446778..e4cb959 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8493,6 +8493,9 @@ static int __alloc_contig_migrate_range(struct compact_control *cc, ret = migrate_pages(&cc->migratepages, alloc_migration_target, NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE); } + + migrate_finish(); + if (ret < 0) { putback_movable_pages(&cc->migratepages); return ret; diff --git a/mm/swap.c b/mm/swap.c index 31b844d..97efc49 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "internal.h" @@ -759,6 +760,8 @@ void lru_add_drain_all(void) if (WARN_ON(!mm_percpu_wq)) return; + invalidate_bh_lrus(); + /* * Guarantee pagevec counter stores visible by this CPU are visible to * other CPUs before loading the current drain generation.