From patchwork Thu Jan 18 11:10:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13522714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A026C4707B for ; Thu, 18 Jan 2024 11:12:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 304886B0093; Thu, 18 Jan 2024 06:12:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4356B0095; Thu, 18 Jan 2024 06:12:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 156516B0096; Thu, 18 Jan 2024 06:12:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 021976B0093 for ; Thu, 18 Jan 2024 06:12:48 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C85A2A096A for ; Thu, 18 Jan 2024 11:12:47 +0000 (UTC) X-FDA: 81692169174.17.07734B3 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 003E610001D for ; Thu, 18 Jan 2024 11:12:45 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dlnTSPVM; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705576366; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; b=OfPRC16YL9nEShP2tZSn9FmIR5VP5Gjaiw8LCAVKGgjJWHKBZbWd9KtLj2iPz4TBgheFEI n4cepubEqZp1bizw8ZJiAB+sDTnZ9v3paCPrPM1LODBuu7E9TjWULXDT5bTrFrnKfCRaxn Zmfb5G2A8sHQyhRCFyKpSElxQuFJ8Wk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705576366; a=rsa-sha256; cv=none; b=Zk+PHU4/VBXbZeJdBMr3pC6JgI1NF9zAsdlAUvV2UiWG+o1RHX263uU9S4kOsqi/XZUISZ 0fU3apEz+EARWgIOH2cyxGXxIcvZ2Ziba8ZqklKuC2FIEGGZXutoTStDOflhRHjrem3EWX kpgu2DjgIt/7yGOkhh4+gPWjKbEI6ds= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dlnTSPVM; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6d9b37f4804so393868b3a.1 for ; Thu, 18 Jan 2024 03:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576365; x=1706181165; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; b=dlnTSPVMKHCZCbWoKgVAhaWXCGW/ZzIlLBQ+f6BJqm5fC6zk2s2rO/WabEs8FTvD9O W6bznuLLZ0f/gRMlvNmZq9uUxL4V4X8Fsqy3pyt43bh40SLSEsuaKiMg+p+R8En+98cC QU/HVwwyt1D5VULepZNq/TiEgrsZdw4APyzQ64MzraM8VArfEFIO4BSt3Xyf14PRURGk 2w4Hf/VtGrdT6khUDW91ZCVsscc6WFMpceXMdOeSYkexXKQT4t4sdR2YFjrHiTNv/bGr 7eaMyu3H9bpPyjwTQJ+L2D33ERTGmhA11rLV+aeURUfNPDYq8ZxcVtymMNwIHd2c5H3S njkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576365; x=1706181165; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; b=rRZldn4PFo91q9x1A6pgwiIbcImuFbqXGsFQA+AQWKnbaAAu+A7if8NwFB2KDpGef1 MSwHUBtJSlE+/CQzEL3WSBe6unDFQDnci/kqcIa/u/4DbzJDupNsa7uQ8KY55THEkfEr hzsNdK9xzzpnbJAEVtLTeJR2cW7y14EKbJvBZMhqhekTC7/vYrkhbT6b0pS22ol1QSuY tS+pzNqiEazfR3+At3eznpDg+7t5wdSVG6qk/BqCp0h4ijpH/QxlP2NVspdtdzSkzF9b bRv0T/HOuo6uSmCTGT8MzcQcCtrxLxZ8pqHaADQu2PdTlRltGBv6h2W4J1ZYSyz+jhmb VOVw== X-Gm-Message-State: AOJu0Yzq5lOOUDLFWpP6ullLJdB1O6a9GfDmOd2WFSE7+FsM5T6gpHLm K3Us672o+nIYt0Lk0HQ+6ERKnjw4qc3bzQQWls3gcx31ED+ko4C3iFzAdVsr X-Google-Smtp-Source: AGHT+IHPb2V5o+Kp/DcFse2JrVkm3MDRb3SQdSn4vIY2cU1g3ZhVG+tEV+84Z29sP1LjBn1UyzwA9g== X-Received: by 2002:a05:6a20:548b:b0:19a:fbfa:b16a with SMTP id i11-20020a056a20548b00b0019afbfab16amr1121941pzk.30.1705576364885; Thu, 18 Jan 2024 03:12:44 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.12.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:12:44 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Chuanhua Han , Barry Song Subject: [PATCH RFC 6/6] mm: madvise: don't split mTHP for MADV_PAGEOUT Date: Fri, 19 Jan 2024 00:10:36 +1300 Message-Id: <20240118111036.72641-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 003E610001D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: g1ecx1enpir9o7b9ajokk73k57mi9bzq X-HE-Tag: 1705576365-608926 X-HE-Meta: U2FsdGVkX18IoUpWhE6k6TJI8KYKcNNGX1gmlgIErglUDpaQRysrZ1Mk9gef6tfWpY2pMZsW7Ms0DM44wKNYCkRVK4zZX5TvvJgp+7cnR4biXBEJ9ssHznUfs4I1QvKeQ0H5BR0k69Ld1khQK9oKGBcA9LJs4P5lmp7PzwyQSQVuDsnXlcHIghzzBM1aVUYE72Mxft4OGasWCCMwhC5slQU3CPyjPauMEmhYPZ4bICCa89cwyW3ELWO0Rkleav0AuLLdd+gcX8JYlZJ35BNebAAHQG+UJR/c+qazXmUVR7DkiTYG7siAXqHTbDDL5Q/Dnl85BJnxWv+xXSOJYWs1jo2c1FOMW9xFR7iAZyohjTQe+bGlemb9/6xESMvNg6jj0M3a3KuAMEzw79/wC3RhOdl025CbQvyt3gKeUIYzQyMUTJrPM6qLjo6v0xszAG8N/tPOwUYOijU5xsAzsGDlPKEVfHhVjfkDuRSIGB7Wj4nkPqagzUO9wrUwu3KOK8gL9QedaIiGQNpaPNgQKOdRQRf7Qw54vtT0zObXaLmdhAzbTsdrsJEZi4sJfV+JkcPeRst6AQNMr5cNJZ75y1BgaQSu+at88yr7g89hMmJEsSqel4ayQs/0iR9+4pzWwPkzA2yVnUsRAzB5chDVWxUEXxNjd+nbxjq8dpFPmSzACKCzJsdXBcJqFGv/2LRET3KhgZPo5Q831/GUuTaPwGt9gakBuG20uSpy/dI4MysjYM1vkod4dL7j543qb0ucCWWvwN58Dwo9AWliYq1P17Opl3jhjdFCPIOdstax0eUzsyrdtqD5HOelKp+YcJMGJz2QeGvFHl05GApCHERRNG+lsUge352MUYar2aqWo4pkac1YK+FVwyv84Yl4qydhW/rnM01KDK3WOYWErNY6feAJgZ/0BmP8XmFkU25k93aZntHiCv8bsOasWWcOUHU58H8m9BHZbbyexZjvqKb3MU0 tfecW3D/ xR1gRMc/BR+8AJNiPcntcECpcXoVO0inixyC0EV7Huf7al6GK18d8vSJxMx5j7YWcxKzkjsz+fKYjNKOMS/r11qkU6HsTjDog4C4NjVLfSo4OqkgmOCWm1GkBv5ZsEsgKw5X6ajXOeZoJ/sIcbbd19PdrBAW2OPqAeUsbA62Z9zcSCLCct9jD/FrnXuMaNooBl3HUEDz0c8Sb58hMB0WoKfeu/TIvQCbiHq3LnYxw/XQ19donpNTTwdJvwVM0v76Ta8o9Kj4ZUgL6M0zeoOxyulvRHRk+z+I8/kfcCT7ldh9rRYaPl3Nq7YFiRB9VADd0uXn9dTgQQPVIrywFk7GRHFgIg6af3tIgwdNV9j8p4vDipqILuazUnIWbdVQ3Zt0wTuuGouIYCUhTREu+lY7srUKzPphqLxBjC8gybu8pUfqAUF7QWYByxUphkYGzMKNf07LBsehbn0rFl40FnxqCz1brcPpnRewB9qEjIL4vcwfSxXU2zgtuKFdj3GQkl9etJj9VL7gtpRgU1v8D8hWoLzsZ5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han MADV_PAGEOUT and MADV_FREE are common cases in Android. Ryan's patchset has supported swapping large folios out as a whole for vmscan case. This patch extends the feature to madvise. If madvised range covers the whole large folio, we don't split it. Otherwise, we still need to split it. This patch doesn't depend on ARM64's CONT-PTE, alternatively, it defines one helper named pte_range_cont_mapped() to check if all PTEs are contiguously mapped to a large folio. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Signed-off-by: Barry Song --- include/asm-generic/tlb.h | 10 +++++++ include/linux/pgtable.h | 60 +++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 48 +++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..f894e22da5d6 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -608,6 +608,16 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) +#define tlb_remove_nr_tlb_entry(tlb, ptep, address, nr) \ + do { \ + int i; \ + tlb_flush_pte_range(tlb, address, \ + PAGE_SIZE * nr); \ + for (i = 0; i < nr; i++) \ + __tlb_remove_tlb_entry(tlb, ptep + i, \ + address + i * PAGE_SIZE); \ + } while (0) + #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ unsigned long _sz = huge_page_size(h); \ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 37fe83b0c358..da0c1cf447e3 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -320,6 +320,42 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) } #endif +#ifndef pte_range_cont_mapped +static inline bool pte_range_cont_mapped(unsigned long start_pfn, + pte_t *start_pte, + unsigned long start_addr, + int nr) +{ + int i; + pte_t pte_val; + + for (i = 0; i < nr; i++) { + pte_val = ptep_get(start_pte + i); + + if (pte_none(pte_val)) + return false; + + if (pte_pfn(pte_val) != (start_pfn + i)) + return false; + } + + return true; +} +#endif + +#ifndef pte_range_young +static inline bool pte_range_young(pte_t *start_pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) + if (pte_young(ptep_get(start_pte + i))) + return true; + + return false; +} +#endif + #ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, @@ -580,6 +616,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, } #endif +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_RANGE_FULL +static inline pte_t ptep_get_and_clear_range_full(struct mm_struct *mm, + unsigned long start_addr, + pte_t *start_pte, + int nr, int full) +{ + int i; + pte_t pte; + + pte = ptep_get_and_clear_full(mm, start_addr, start_pte, full); + + for (i = 1; i < nr; i++) + ptep_get_and_clear_full(mm, start_addr + i * PAGE_SIZE, + start_pte + i, full); + + return pte; +} /* * If two threads concurrently fault at the same page, the thread that @@ -995,6 +1048,13 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) }) #endif +#ifndef pte_nr_addr_end +#define pte_nr_addr_end(addr, size, end) \ +({ unsigned long __boundary = ((addr) + size) & (~(size - 1)); \ + (__boundary - 1 < (end) - 1)? __boundary: (end); \ +}) +#endif + /* * When walking page tables, we usually want to skip any p?d_none entries; * and any p?d_bad entries - reporting the error before resetting to none. diff --git a/mm/madvise.c b/mm/madvise.c index 912155a94ed5..262460ac4b2e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -452,6 +452,54 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (folio_test_large(folio)) { int err; + if (!folio_test_pmd_mappable(folio)) { + int nr_pages = folio_nr_pages(folio); + unsigned long folio_size = PAGE_SIZE * nr_pages; + unsigned long start_addr = ALIGN_DOWN(addr, nr_pages * PAGE_SIZE);; + unsigned long start_pfn = page_to_pfn(folio_page(folio, 0)); + pte_t *start_pte = pte - (addr - start_addr) / PAGE_SIZE; + unsigned long next = pte_nr_addr_end(addr, folio_size, end); + + if (!pte_range_cont_mapped(start_pfn, start_pte, start_addr, nr_pages)) + goto split; + + if (next - addr != folio_size) { + goto split; + } else { + /* Do not interfere with other mappings of this page */ + if (folio_estimated_sharers(folio) != 1) + goto skip; + + VM_BUG_ON(addr != start_addr || pte != start_pte); + + if (pte_range_young(start_pte, nr_pages)) { + ptent = ptep_get_and_clear_range_full(mm, start_addr, start_pte, + nr_pages, tlb->fullmm); + ptent = pte_mkold(ptent); + + set_ptes(mm, start_addr, start_pte, ptent, nr_pages); + tlb_remove_nr_tlb_entry(tlb, start_pte, start_addr, nr_pages); + } + + folio_clear_referenced(folio); + folio_test_clear_young(folio); + if (pageout) { + if (folio_isolate_lru(folio)) { + if (folio_test_unevictable(folio)) + folio_putback_lru(folio); + else + list_add(&folio->lru, &folio_list); + } + } else + folio_deactivate(folio); + } +skip: + pte += (next - PAGE_SIZE - (addr & PAGE_MASK))/PAGE_SIZE; + addr = next - PAGE_SIZE; + continue; + + } +split: if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio))