From patchwork Thu Jul 14 14:47:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charan Teja Kalla X-Patchwork-Id: 12918069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9292C433EF for ; Thu, 14 Jul 2022 14:48:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC3259401AC; Thu, 14 Jul 2022 10:48:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C73269401A5; Thu, 14 Jul 2022 10:48:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B620C9401AC; Thu, 14 Jul 2022 10:48:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A7C6E9401A5 for ; Thu, 14 Jul 2022 10:48:16 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 73403356F6 for ; Thu, 14 Jul 2022 14:48:16 +0000 (UTC) X-FDA: 79685985792.15.E8AE5E7 Received: from alexa-out-sd-02.qualcomm.com (alexa-out-sd-02.qualcomm.com [199.106.114.39]) by imf20.hostedemail.com (Postfix) with ESMTP id 85A571C007A for ; Thu, 14 Jul 2022 14:48:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; i=@quicinc.com; q=dns/txt; s=qcdkim; t=1657810095; x=1689346095; h=from:to:cc:subject:date:message-id:mime-version; bh=aEQ9TQKBEZAJvnZaCHYvMD/w0XIhUbWB8CftNPiZ0Ng=; b=uJPggLOfx0v/0z602R1MHe2qJELZtA/3AeX6jQK+18PR8KyuWfHXVbYL RTbVSnjt5lIhNvj6Nd+N7YJuwU/F7OSKOrc1OB1it3OROQKIs0s9U68zV PVLXFQ5iyhajX2QjBRERlUCiEEyE00fsyZQuHr60y9vre/leJ+5uk8qkf k=; Received: from unknown (HELO ironmsg03-sd.qualcomm.com) ([10.53.140.143]) by alexa-out-sd-02.qualcomm.com with ESMTP; 14 Jul 2022 07:48:14 -0700 X-QCInternal: smtphost Received: from nasanex01c.na.qualcomm.com ([10.47.97.222]) by ironmsg03-sd.qualcomm.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2022 07:48:13 -0700 Received: from nalasex01a.na.qualcomm.com (10.47.209.196) by nasanex01c.na.qualcomm.com (10.47.97.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Thu, 14 Jul 2022 07:48:13 -0700 Received: from hu-charante-hyd.qualcomm.com (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Thu, 14 Jul 2022 07:48:08 -0700 From: Charan Teja Kalla To: , , , , , , , , , , CC: , , Charan Teja Kalla Subject: [PATCH] mm: fix use-after free of page_ext after race with memory-offline Date: Thu, 14 Jul 2022 20:17:43 +0530 Message-ID: <1657810063-28938-1-git-send-email-quic_charante@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01a.na.qualcomm.com (10.52.223.231) To nalasex01a.na.qualcomm.com (10.47.209.196) ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcdkim header.b=uJPggLOf; dmarc=pass (policy=none) header.from=quicinc.com; spf=pass (imf20.hostedemail.com: domain of quic_charante@quicinc.com designates 199.106.114.39 as permitted sender) smtp.mailfrom=quic_charante@quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657810095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=aEQ9TQKBEZAJvnZaCHYvMD/w0XIhUbWB8CftNPiZ0Ng=; b=izhPqwbmElXHp4Wf0ap3SEPXHj1D6XqRdkOeqITySPGV19zZsZYK2QLEtREOaX/xL587Nq tBi2Dy83XqVam5BojSWLxxiyutLWocW4pCdtIi9YCoYcl77zQUwg+umjRu4e6iv2Du4nl8 zbFRHUL/BIbO8e8F5sIuQD+fdxIQQvc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657810095; a=rsa-sha256; cv=none; b=kvFDqM1oWY93rmOM1TBgouIgHKA9pNzsrkJXGDeMbnN7GtfWuGDf5q9n003OD+vaZqP/Ns XWN0Vs5+BlplEqLzK8zWyGi42NsOZ0zALKExB1Ow3scNEKZDxvsdtTdksYEKhqh6SFdKon oxV+H9hgrAsbbRPRvL5Cp7EPdiXvSZY= X-Rspam-User: X-Rspamd-Queue-Id: 85A571C007A X-Rspamd-Server: rspam02 X-Stat-Signature: p7d1pmjgxqeptf4qh5cdx13s14paoejc Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcdkim header.b=uJPggLOf; dmarc=pass (policy=none) header.from=quicinc.com; spf=pass (imf20.hostedemail.com: domain of quic_charante@quicinc.com designates 199.106.114.39 as permitted sender) smtp.mailfrom=quic_charante@quicinc.com X-HE-Tag: 1657810095-343410 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The below is one path where race between page_ext and offline of the respective memory blocks will cause use-after-free on the access of page_ext structure. process1 process2 --------- --------- a)doing /proc/page_owner doing memory offline through offline_pages. b)PageBuddy check is failed thus proceed to get the page_owner information through page_ext access. page_ext = lookup_page_ext(page); migrate_pages(); ................. Since all pages are successfully migrated as part of the offline operation,send MEM_OFFLINE notification where for page_ext it calls: offline_page_ext()--> __free_page_ext()--> free_page_ext()--> vfree(ms->page_ext) mem_section->page_ext = NULL c) Check for the PAGE_EXT flags in the page_ext->flags access results into the use-after-free(leading to the translation faults). As mentioned above, there is really no synchronization between page_ext access and its freeing in the memory_offline. The memory offline steps(roughly) on a memory block is as below: 1) Isolate all the pages 2) while(1) try free the pages to buddy.(->free_list[MIGRATE_ISOLATE]) 3) delete the pages from this buddy list. 4) Then free page_ext.(Note: The struct page is still alive as it is freed only during hot remove of the memory which frees the memmap, which steps the user might not perform). This design leads to the state where struct page is alive but the struct page_ext is freed, where the later is ideally part of the former which just representing the page_flags. This seems to be a wrong design where 'struct page' as a whole is not accessible(Thanks to Minchan for pointing this out). The above mentioned race is just one example __but the problem persists in the other paths too involving page_ext->flags access(eg: page_is_idle())__. Since offline waits till the last reference on the page goes down i.e. any path that took the refcount on the page can make the memory offline operation to wait. Eg: In the migrate_pages() operation, we do take the extra refcount on the pages that are under migration and then we do copy page_owner by accessing page_ext. For Fix those paths where offline races with page_ext access by maintaining synchronization with rcu lock. Thanks to David Hildenbrand for his views/suggestions on the initial discussion[1]. [1] https://lore.kernel.org/linux-mm/59edde13-4167-8550-86f0-11fc67882107@quicinc.com/ Signed-off-by: Charan Teja Kalla --- include/linux/page_ext.h | 19 +++++++++++++++++++ include/linux/page_idle.h | 40 ++++++++++++++++++++++++++++++---------- mm/page_ext.c | 3 ++- mm/page_owner.c | 18 +++++++++++++++--- mm/page_table_check.c | 10 +++++++--- 5 files changed, 73 insertions(+), 17 deletions(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index fabb2e1..df5d353 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -64,6 +64,25 @@ static inline struct page_ext *page_ext_next(struct page_ext *curr) return next; } +static inline struct page_ext *get_page_ext(struct page *page) +{ + struct page_ext *page_ext; + + rcu_read_lock(); + page_ext = lookup_page_ext(page); + if (!page_ext) { + rcu_read_unlock(); + return NULL; + } + + return page_ext; +} + +static inline void put_page_ext(void) +{ + rcu_read_unlock(); +} + #else /* !CONFIG_PAGE_EXTENSION */ struct page_ext; diff --git a/include/linux/page_idle.h b/include/linux/page_idle.h index 4663dfe..0d80f78 100644 --- a/include/linux/page_idle.h +++ b/include/linux/page_idle.h @@ -13,65 +13,85 @@ * If there is not enough space to store Idle and Young bits in page flags, use * page ext flags instead. */ - static inline bool folio_test_young(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + bool page_young; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return false; - return test_bit(PAGE_EXT_YOUNG, &page_ext->flags); + page_young = test_bit(PAGE_EXT_YOUNG, &page_ext->flags); + put_page_ext(); + + return page_young; } static inline void folio_set_young(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return; set_bit(PAGE_EXT_YOUNG, &page_ext->flags); + put_page_ext(); } static inline bool folio_test_clear_young(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + bool page_young; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return false; - return test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags); + page_young = test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags); + put_page_ext(); + + return page_young; } static inline bool folio_test_idle(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + bool page_idle; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return false; - return test_bit(PAGE_EXT_IDLE, &page_ext->flags); + page_idle = test_bit(PAGE_EXT_IDLE, &page_ext->flags); + put_page_ext(); + + return page_idle; } static inline void folio_set_idle(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return; set_bit(PAGE_EXT_IDLE, &page_ext->flags); + put_page_ext(); } static inline void folio_clear_idle(struct folio *folio) { - struct page_ext *page_ext = lookup_page_ext(&folio->page); + struct page_ext *page_ext; + page_ext = get_page_ext(&folio->page); if (unlikely(!page_ext)) return; clear_bit(PAGE_EXT_IDLE, &page_ext->flags); + put_page_ext(); } #endif /* !CONFIG_64BIT */ diff --git a/mm/page_ext.c b/mm/page_ext.c index 3dc715d..5ccd3ee 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -299,8 +299,9 @@ static void __free_page_ext(unsigned long pfn) if (!ms || !ms->page_ext) return; base = get_entry(ms->page_ext, pfn); - free_page_ext(base); ms->page_ext = NULL; + synchronize_rcu(); + free_page_ext(base); } static int __meminit online_page_ext(unsigned long start_pfn, diff --git a/mm/page_owner.c b/mm/page_owner.c index e4c6f3f..d4b9417 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -195,14 +195,16 @@ noinline void __set_page_owner(struct page *page, unsigned short order, void __set_page_owner_migrate_reason(struct page *page, int reason) { - struct page_ext *page_ext = lookup_page_ext(page); + struct page_ext *page_ext; struct page_owner *page_owner; + page_ext = get_page_ext(page); if (unlikely(!page_ext)) return; page_owner = get_page_owner(page_ext); page_owner->last_migrate_reason = reason; + put_page_ext(); } void __split_page_owner(struct page *page, unsigned int nr) @@ -288,7 +290,10 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m, pageblock_mt = get_pageblock_migratetype(page); + rcu_read_lock(); for (; pfn < block_end_pfn; pfn++) { + if (need_resched()) + cond_resched_rcu(); /* The pageblock is online, no need to recheck. */ page = pfn_to_page(pfn); @@ -327,6 +332,7 @@ void pagetypeinfo_showmixedcount_print(struct seq_file *m, } pfn += (1UL << page_owner->order) - 1; } + rcu_read_unlock(); } /* Print counts */ @@ -492,6 +498,7 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) struct page_ext *page_ext; struct page_owner *page_owner; depot_stack_handle_t handle; + ssize_t ret = 0; if (!static_branch_unlikely(&page_owner_inited)) return -EINVAL; @@ -506,7 +513,10 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) drain_all_pages(NULL); /* Find an allocated page */ + rcu_read_lock(); for (; pfn < max_pfn; pfn++) { + if (need_resched()) + cond_resched_rcu(); /* * If the new page is in a new MAX_ORDER_NR_PAGES area, * validate the area as existing, skip it if not @@ -563,11 +573,13 @@ read_page_owner(struct file *file, char __user *buf, size_t count, loff_t *ppos) /* Record the next PFN to read in the file offset */ *ppos = (pfn - min_low_pfn) + 1; - return print_page_owner(buf, count, pfn, page, + ret = print_page_owner(buf, count, pfn, page, page_owner, handle); + break; } + rcu_read_unlock(); - return 0; + return ret; } static void init_pages_in_zone(pg_data_t *pgdat, struct zone *zone) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index e206274..66fd49a 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -68,7 +68,7 @@ static void page_table_check_clear(struct mm_struct *mm, unsigned long addr, return; page = pfn_to_page(pfn); - page_ext = lookup_page_ext(page); + page_ext = get_page_ext(page); anon = PageAnon(page); for (i = 0; i < pgcnt; i++) { @@ -83,6 +83,7 @@ static void page_table_check_clear(struct mm_struct *mm, unsigned long addr, } page_ext = page_ext_next(page_ext); } + put_page_ext(); } /* @@ -103,7 +104,7 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, return; page = pfn_to_page(pfn); - page_ext = lookup_page_ext(page); + page_ext = get_page_ext(page); anon = PageAnon(page); for (i = 0; i < pgcnt; i++) { @@ -118,6 +119,7 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, } page_ext = page_ext_next(page_ext); } + put_page_ext(); } /* @@ -126,9 +128,10 @@ static void page_table_check_set(struct mm_struct *mm, unsigned long addr, */ void __page_table_check_zero(struct page *page, unsigned int order) { - struct page_ext *page_ext = lookup_page_ext(page); + struct page_ext *page_ext; unsigned long i; + page_ext = get_page_ext(page); BUG_ON(!page_ext); for (i = 0; i < (1ul << order); i++) { struct page_table_check *ptc = get_page_table_check(page_ext); @@ -137,6 +140,7 @@ void __page_table_check_zero(struct page *page, unsigned int order) BUG_ON(atomic_read(&ptc->file_map_count)); page_ext = page_ext_next(page_ext); } + put_page_ext(); } void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr,