From patchwork Tue Jan 10 07:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D51C54EBE for ; Tue, 10 Jan 2023 07:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3D628E000A; Tue, 10 Jan 2023 02:53:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC6938E0001; Tue, 10 Jan 2023 02:53:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C40168E000A; Tue, 10 Jan 2023 02:53:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AE2978E0001 for ; Tue, 10 Jan 2023 02:53:59 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 750E8803A9 for ; Tue, 10 Jan 2023 07:53:59 +0000 (UTC) X-FDA: 80338125798.05.23434B2 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 5F141C0004 for ; Tue, 10 Jan 2023 07:53:57 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="IL/CmecD"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nvlOY5thobMMfBBwSIL7UZ3rHYZV4yteEfs4Uw0/IUk=; b=5drwduAFbCOJFO+pvtdzYLLU/JV1TpZjRqVKQmNiSH18Zws897AH3tO/oqMboivvyimjm9 Em847t2BscMKtUrhUsssJdqrSkdp5InNT+7lMzfX/DfFMYGCFnUuR+HUB7PebcUVkaEKXP 7nlT4pA+LNT+wtJsC1miloAASFkQOo8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="IL/CmecD"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337237; a=rsa-sha256; cv=none; b=xoST9sTmaDmS0Dq5ln0kwiClBE+LfFj4yj/1APxCJJh7xw7/hqkNSTsOPc3LUtFEqCJ+Fk Au5klUNWrHz9l8R6hXeKfn4LXWePCS/iTcVnNPkTKTvzMwX2t+ut4bTy7xJWtYLE87DwRP 9RcE0F7OpkF93pbPZLWnvkdpIB0i2Q4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337237; x=1704873237; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Jj7YUOuBj4VP8ir6ICoR9mdplgusD9YiP4zbuT3Oc1o=; b=IL/CmecDNoCcF1oE+B0g7/3xxPVCJHhBUGFPMqzLE/87yAfMYdmbB9ms IY+fcKwAzc+Kh9CZSH8F70wZjltFuJKHRrhlus+8qcuBe60jV30rdCrH+ IN5YbMz6xigfWrgM+pSDyVvdXa5YfRQ8hMdwyjtmdqeeozmt1Wf41TIjM F+ZGflIbYdCpOaAgH7tl6vb2UuTwasMK4LcLQQZ654YvLW+NX+qtrwXZ0 2fGzEzMw65yG74ExdStiqTO+kGPXmxHwEykYbXLj4bpBdUIwuJivXAd26 qhqFlH6xXTU/9YaoxIIh46Gp9vQhh+lT/yzCeFOvXuiFsPakke4TQcjWL Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449364" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449364" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287176" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287176" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:53 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 7/9] migrate_pages: share more code between _unmap and _move Date: Tue, 10 Jan 2023 15:53:25 +0800 Message-Id: <20230110075327.590514-8-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5F141C0004 X-Stat-Signature: bz6idtnuqk1o6z68pfcfiod5ztfckgta X-HE-Tag: 1673337237-322933 X-HE-Meta: U2FsdGVkX19gmdhGtk+z/EzAgWEQA8H36Zv2YO067CmRG6qGhEVNqiNwHSy3UdBF6SQhECx7+NweEfOz+xFaItdn4TpZuSJyJqzbK6A5ZmSq17t4SqMGCog6WZ8azCOrDSt3GozuXpa3iEcunVjPuPfcW86pIYgkhXL+4sTcLfkDI4jF8G2vs78wdvlBkUpcVEUAC3FyqaulqSD/ydIOUVERFl5iyCDWPKhuXrmEHLMvJ9p8IdAjJldSQ6TQHpu1iG6tqwdEpnTeotlAr3AYpPPcyS5KNY9OxpSVtxrjDSvd8bpkML4pgnOCC3hNJZG/Yj4BBiEq++i3g68QSDYSnlrmOdDMmEulGEuzUT2fGxqHSBvEI3m1BM5/gGb+0Hpzl05g0I3rPBw+Pq2EsA0TfWJ2Y4MzEyPE2ajE63UNhUBHP+7peZOBfMPwUopkcaQiftdmKqJwmBDzIOPiersya4p57nU3wQ6DKvzOEeshHs+60TTo9vg0amdcz7aL1eSofbZPZeNCFPYDRy+ANxYT1e7Bh3qg/iPJYLHuD54rKrS/4UQGSZ4lD570zYp2OFElJR3R+LAYCg8ML3MzBRJtrNpSZ8Lex0CcJb8LLqofNJpszzS09H15jmaI4uh9Z77AqA5GA+/oFwCclvnCL+lkHyDoE9+1AYYjGGKmbsLkcccL4852zeJn+eyKYVKf/pLoWenK4m0wlrsIcifQhyz5kdyh0ad7SN0kY8bQkN8E26iAEc0Ff84pjLfS9Z9BGtQvGiljc4IWbUuJxj8q9txhR4ZoRF47+xIHQrVXoLzGmBnt0UTfYUk/OMtPR3ykzoSl9YxtdB7YrmHldKaED1vf/M/cb0HT8BR9zRbMhmjwGYhdJu8D6qpulOx1/qKdWEOXjY3h7QZFtHSwCbwnatxYMO7DIANx9tX3mOb+J2uIUXG90qdB0BzjYDZO+dDxxTzeEJezMipJWwE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 207 ++++++++++++++++++++------------------------------- 1 file changed, 82 insertions(+), 125 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 08ce68ba5085..6c721b897efd 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1037,6 +1037,7 @@ static void __migrate_folio_extract(struct folio *dst, static void migrate_folio_undo_src(struct folio *src, int page_was_mapped, struct anon_vma *anon_vma, + bool locked, struct list_head *ret) { if (page_was_mapped) @@ -1044,16 +1045,20 @@ static void migrate_folio_undo_src(struct folio *src, /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - folio_unlock(src); - list_move_tail(&src->lru, ret); + if (locked) + folio_unlock(src); + if (ret) + list_move_tail(&src->lru, ret); } /* Restore the destination folio to the original state upon failure */ static void migrate_folio_undo_dst(struct folio *dst, + bool locked, free_page_t put_new_page, unsigned long private) { - folio_unlock(dst); + if (locked) + folio_unlock(dst); if (put_new_page) put_new_page(&dst->page, private); else @@ -1078,13 +1083,42 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, bool force_lock, enum migrate_mode mode) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { + struct folio *dst; int rc = -EAGAIN; + struct page *newpage = NULL; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + bool locked = false; + bool dst_locked = false; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) @@ -1119,6 +1153,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, folio_lock(src); } + locked = true; if (folio_test_writeback(src)) { /* @@ -1133,10 +1168,10 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, break; default: rc = -EBUSY; - goto out_unlock; + goto out; } if (!force) - goto out_unlock; + goto out; folio_wait_writeback(src); } @@ -1166,7 +1201,8 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!folio_trylock(dst))) - goto out_unlock; + goto out; + dst_locked = true; if (unlikely(!is_lru)) { __migrate_folio_record(dst, page_was_mapped, anon_vma); @@ -1188,7 +1224,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (!src->mapping) { if (folio_test_private(src)) { try_to_free_buffers(src); - goto out_unlock_both; + goto out; } } else if (folio_mapped(src)) { /* Establish migration ptes */ @@ -1203,75 +1239,26 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return MIGRATEPAGE_UNMAP; } - - if (page_was_mapped) - remove_migration_ptes(src, src, false); - -out_unlock_both: - folio_unlock(dst); -out_unlock: - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); out: - - return rc; -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - /* * A page that has not been migrated will have kept its * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); + if (rc == -EAGAIN || rc == -EDEADLOCK) + ret = NULL; - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); + migrate_folio_undo_dst(dst, dst_locked, put_new_page, private); return rc; } -static int __migrate_folio_move(struct folio *src, struct folio *dst, - enum migrate_mode mode) +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { int rc; int page_was_mapped = 0; @@ -1280,9 +1267,10 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; - if (rc != -EAGAIN) - list_del(&dst->lru); + list_del(&dst->lru); /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1292,74 +1280,43 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ - if (rc == MIGRATEPAGE_SUCCESS) { - folio_add_lru(dst); - if (page_was_mapped) - lru_add_drain(); - } - - if (rc == -EAGAIN) { - __migrate_folio_record(dst, page_was_mapped, anon_vma); - return rc; - } - + folio_add_lru(dst); if (page_was_mapped) - remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + lru_add_drain(); + if (page_was_mapped) + remove_migration_ptes(src, dst, false); folio_unlock(dst); - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); + set_page_owner_migrate_reason(&dst->page, reason); /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ - if (rc == MIGRATEPAGE_SUCCESS) - folio_put(dst); - - return rc; -} - -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_page_t put_new_page, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - int rc; - - rc = __migrate_folio_move(src, dst, mode); - if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(&dst->page, reason); - - if (rc != -EAGAIN) { - /* - * A folio that has been migrated has all references - * removed and will be freed. A folio that has not been - * migrated will have kept its references and be restored. - */ - list_del(&src->lru); - } + folio_put(dst); /* - * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the folio to right list unless - * we want to retry. + * A page that has been migrated has all references removed + * and will be freed. */ - if (rc == MIGRATEPAGE_SUCCESS) { - migrate_folio_done(src, reason); - } else if (rc != -EAGAIN) { - list_add_tail(&src->lru, ret); + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + migrate_folio_done(src, reason); - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + return rc; +out: + if (rc == -EAGAIN) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; } + migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_page, private); + return rc; } @@ -1891,9 +1848,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); migrate_folio_undo_src(folio, page_was_mapped, anon_vma, - ret_folios); + true, ret_folios); list_del(&dst->lru); - migrate_folio_undo_dst(dst, put_new_page, private); + migrate_folio_undo_dst(dst, true, put_new_page, private); dst = dst2; dst2 = list_next_entry(dst, lru); }