From patchwork Mon Oct 16 20:05:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13424040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75504CDB474 for ; Mon, 16 Oct 2023 20:07:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D45DD80027; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD11C80008; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B478880027; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9D79080008 for ; Mon, 16 Oct 2023 16:07:03 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6CE6F1CB3A6 for ; Mon, 16 Oct 2023 20:07:03 +0000 (UTC) X-FDA: 81352408326.07.317DC36 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 9708540004 for ; Mon, 16 Oct 2023 20:07:01 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B3a2rpc9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697486821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=vCGQZOaijy6IwvG6JWEIu/pagfoyMSKpk26+Yv+3chcrBYIRthrIEK2zmW3hTnOLaMsJgc HxPacgzQ+Mexbjz4TXJKM2fExNtxTQ7JPb2WV7o48zpQCc3GAGYTK7GzDnjugg10NmSrSX 5WbR0gpPIKngOqtkTWgArL3S0jt/tis= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B3a2rpc9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697486821; a=rsa-sha256; cv=none; b=DpnRlAmfqMLPKrYiN0kZkZL+UfUvCC717I+aPNyIN1oI7FSGpbCplJAi30GH/mIeRP3e0r cVLKcuxti29QY3wALDNYAMVGyG1woWnERbvlnvvlt4mJK7id3egHsm3l4Ae3txbm6ShKP0 dd4CtoqKC+dxLUnj/ZGfgr4JiL6Km68= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-53fbf2c42bfso3665655a12.3 for ; Mon, 16 Oct 2023 13:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697486820; x=1698091620; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=B3a2rpc9f2yt8X4FzcDPWsSY6/+mFaJTKki6vukwAkUq0z0gNksP1SAMmEuF4cQXCK JDyKfEQhajlR+tvrd5sHmx+5fCUTlJTTi2EQha/kt7VIxaDycUNbBVbpmctS7csxYXr3 wsbJy4z0aF1uC62XRN0usW+g1UAm3/3FdJzEMJ5EpOdJ3d3+1VMdKmWOf0mZf0Z3dGxN m2uizjRPCOQXhi94dN9ekJNpYcvsPqGlFBCmePtgdFjfnv2TvRWYV/eZonA+0SJgRKXZ Yg8nIWiy3sw1msiSC1909bNNr4nTCF9rUUVNo58ZnxKRhw+Vl55yd0jdZ92gBYqSXPKe 6ZPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697486820; x=1698091620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x1jVF5fDDtB5E1xXw4u2h0Qq1/g1/HjZ8iamB/5OFoA=; b=QRxw4l8WQd4/AK1YQABGoM5tSNievREIkf+j+TBHC8GbCmpUqYAQlIRsaEyA/Y3jRW b99N1uEeM4rcllgeZkEuR1EcKjkcWoUlyFkDHP/weswBzRBxn4eaiwJMjIZaTXwvL9e+ 0cUOe6y8Xoe4sa97PJ/lZZtyoYUMNEBsIn5fhE+4DAYx/v5lF65HkYBxwH/8T50dJe4t XSypEau8eRK0e2ZTbeY58sW/xq1fBEPbpNuC8V++LP4JUGSpA9dwLqneOsd8rxCz8t/7 CeK+ucaNT5NAONK9yiQx2VKwobZY0vO2JwfYY0Nkduz4YbcYc150Nu4EvEPjDLay8syK CNsQ== X-Gm-Message-State: AOJu0Yyz4FVMoThn+FRsgeZUSULjDf2Tz3GGizBrIDSzcqcUS0fftd10 r6BHnGV7db7uP8fsYvbVwgVGOhO0Px4= X-Google-Smtp-Source: AGHT+IFPrGp2LxyePxJ2SGP9Uv5jwNSK/jlVnwKjorWrK64yF1KjrffZTKbz4NdtbcNvcYG72+5GCQ== X-Received: by 2002:a17:90b:368d:b0:27d:f85:9505 with SMTP id mj13-20020a17090b368d00b0027d0f859505mr159453pjb.24.1697486820185; Mon, 16 Oct 2023 13:07:00 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id t16-20020a17090b019000b002772faee740sm5162367pjs.5.2023.10.16.13.06.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 13:06:59 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Mon, 16 Oct 2023 13:05:10 -0700 Message-Id: <20231016200510.7387-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231016200510.7387-1-vishal.moola@gmail.com> References: <20231016200510.7387-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9708540004 X-Stat-Signature: wehsqmdt13mu3eszgfpo6oig74rcyo5w X-Rspam-User: X-HE-Tag: 1697486821-377195 X-HE-Meta: U2FsdGVkX18RtrQCCDE0izASUEq9GhRdEF31rle25oxE9sJtmgHNtdde0MHgXVeRMmbE/+MSq4ASKQalk5lWTDgGW98BLUeuiNLEXppoYPksk7oQGhmNDzggVrNvPTDgID5KkqQzIfhNT6SZe7xet7Mo5P9sfkJ/iSC0n2k6MDLdMYfop89YalcShTUukFZdRTCnn2AyzlRsyv0BwBtmJRP6C/YnBcYno8EXIpg6ZTkj2MqrCG10ONrVh+7h7v9Pb36LwLYcnh5AZUZaJufIMlahpBPlAUNOh2Om6YLkUhzNF9PapNWyOzFgLJcb8uGeJWGqARWWdwBVkzCu5PLt4ctynvhf8pZZRRqHvXXcjU5qbFyds0BSQ4THlidc26Jhj4u7gZF6QvduRjuox/LhxRO49HeNmIYKIZHehin44EISyl62g1UClI7+Hl2CB/PLmo18cedBQbu3vOyI5f12V4jiLwmtalySGOD+fUW6pG+OxYF52mH47VJ6DdMjyXCj0fLZB8M4t41Wjlvnt6OrQ7zbYFmAJa4tt0JUmnSN/+tHzAaLfzmrYtHFi4YWMMnSuw5Iq398B5q9goqWuu8yGQfEPxfE6pPyMkzhvWad01h05Cql5SncsECXPCDivR3g3kTpUocAc12OIPkDcmIPvEzUmq8j5p3eZYaFCkIJGGlHW52EPcJzpiMiQ+kCgpzryP4M0krZXXna5Qpq1fPcaV3bBBJH0TW/WEma83yNo+E5w2p7Vd6F6Sy9YNoF27R1xkKf0RuNANkpJ2JOzKWmxZx0PsvF2NRUsmfGMZ6dI3wCuZsZRfHtJ/1QD3b8bfolPzCzZl4hVSyzmnR4F5ABzAv2U0qi0hsR30A+srnWM+zbeo5N2tBKLehASJ9U57t+Uxg5+adOPRp9ensr91NJUHWH84yP6qGAxrdFNeaXOfnSZ7471SkqQPAf94GKqMIG7hoTE9QjMrMXPTolc2u JJIh7hjm Ktg2pEoCaGnYJkNBfyEVx/rUoQJOpi50R+GUbcmJXo8KlS8enwxmUElVI5aBD0N3aB0L40vupIyCwhvZ859WuBbD2VVPH3hBoc9n5uGi1dlwqO2KN9/T1d7MnXJwkHcVdyAx4zsXKnAOfeUNK40jhZ2uoUy1HEHfDjLSSvZcWL+6xUcf7De9xp8KpkgPgIgUPqaaCXboHqpWLxgwWmUpkLZStTNt/QzYbdEGStRMQhAfkhJYS5iXwG32RC+cpA1XSWmFxjN+JnAXkXf4oUDQE/QPV8tnHy6/FFsSYbLM1ZCvIbHoJO3gYqaLwq87RyI9F6tjxA8Ty9Y711gAKrbEf4waJIfDVtZoRsDU7oi1rHZawiMTo4t0/rVfJg5bxiQdrr8Y41gh8QoY7SoBJSZveQirKw0oO3aDjGYxgcfd/WMD/jILMLTNIyS8dbQEgpPMU/whb/d5aCUy9pD6+HYBVHi7XPzpDFxKQbwIJi0JGtHLhsKlbNgN3MCSsfm0R8tzd58LxbufPkwhttNEoIo4M3Dlp4g7b1a3+n6MWW15LCnUf1z3ZQ39z+m+aRhG0j+UJ7nuV6v4cJNSBtqAhowVQfxHl+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we can get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 70bba8ddea13..99c437979848 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1471,7 +1471,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1504,19 +1504,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (!folio) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1530,13 +1525,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1561,7 +1556,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1576,7 +1571,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1600,7 +1595,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1609,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1628,8 +1623,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1653,14 +1648,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1668,9 +1663,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }