From patchwork Wed Oct 18 20:32:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13427844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 149C9CDB482 for ; Wed, 18 Oct 2023 20:33:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B921B80020; Wed, 18 Oct 2023 16:33:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF1528D0016; Wed, 18 Oct 2023 16:33:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 942EE80020; Wed, 18 Oct 2023 16:33:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7F00F8D0016 for ; Wed, 18 Oct 2023 16:33:14 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 59EA4A0D92 for ; Wed, 18 Oct 2023 20:33:14 +0000 (UTC) X-FDA: 81359731908.23.95CD5D2 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf01.hostedemail.com (Postfix) with ESMTP id 79FB540024 for ; Wed, 18 Oct 2023 20:33:12 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KUqsdhvw; spf=pass (imf01.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697661192; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AcNlnHmZuXPr4rVPOwy7uUYyAYPeSLuYyJXFxeMfebs=; b=g1Dd3eGBi/A82pFyTMQJ7v6Jpwbs4XavVaVmAKvBLZR0bmHHg5Dj7OyOPt+H8GtQk1o7Dy Xx6gr4T2sVq7mbVL+IiFeJqKv6vt+f+X7vieeLYYEAOuFcTcjNeiLI9oV413U62wEMVmbt lbOftACnTH5ebozSIL2LHZuT8/QpoMo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KUqsdhvw; spf=pass (imf01.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697661192; a=rsa-sha256; cv=none; b=asloCwf4xB/+z5I85aNnCbvPT1UNtCZNQB5ap2Ds/D6UceJ1OaVO+yB5QWxOzbEmf69hz0 wrVTrGmVNARaKM3Fn833y40IyTLJ05IK4U13bC8vN0MGHns7xmEPIMQ8nKHJ+G/UsTuLEt /COhXZue/2VKKS2MCYMEHLUMMAS+QEs= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1c9e95aa02dso776355ad.0 for ; Wed, 18 Oct 2023 13:33:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697661191; x=1698265991; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AcNlnHmZuXPr4rVPOwy7uUYyAYPeSLuYyJXFxeMfebs=; b=KUqsdhvwlbZNq4yI1oN1btXYWwSgVSA2HdtpmPYsTqtMP8NclIpzB4OEQiMPE8deNf HEiTRCwgTYGdcjTU0rJESrYQPYAeKFkZlYSxO/Gb8r300HPHAeTt2t/ouWA9o9v3b/Cu 2PrkGA6c9Z9hyFZeAIRRXnPToFeqgSpBZz0/rukjnBIcRH36uIxBOyD80IpdXHalIEhJ ZGzQ/rpt6d/gnp3mHqMSZOduOpXSow8/bVAvEEOrmLvQ2BCQnzNyN8hsdox3Zj4DoPpN HXIsrwQrj6mCJ4LFETkDDRX/Z3rri9S7VEwPc5LttmC8GWaIWBPDbNQVZg676eSyY/ku 8nXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697661191; x=1698265991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AcNlnHmZuXPr4rVPOwy7uUYyAYPeSLuYyJXFxeMfebs=; b=n9nwCAOcarHNtk3PuO3mJjftA8l8cuVtJxBwYOp6R4xU75XweDLRKczy4uhuj8JP6P dn6d113GhgP/FeOlBEQ22wCS8Ln8Qgm2bwiEopYoaHmo16gJ0sd57b6Nv9G20G5U1Bf2 7KnfPk4HI9cYaF8X8jCPbSP/9SKb9jv0/2ImmZ/zlBwll4iWSwgv0qI04U5tuics7JvE /+nVlNXZC2h9yYa8ZKvLlFWC5IAg52fZMp3P/BdBGBX2V2sG2loDNqsN/JiigIWiMvOT B1DPYsaw7yNUXtwuhjOWATUAVaByELZqdGLks0UlhGn3neYlIPCJIEAyoAZEqj7G3yh0 8MlA== X-Gm-Message-State: AOJu0YzWY1ZmfCgg+faUKjmDMmALgRY51Hh17J5iZzAFtCGa03suVDtq Zp5ENGC8+gfp6r7x0fwj7tN+qTOUpJ0= X-Google-Smtp-Source: AGHT+IEf7Gnz9O9WMz3CQOd+A4MjLM6TTZK4jiZl9DPWEVFFAHxMsHTxdrbBpGMTCyZdudHzKKxBQw== X-Received: by 2002:a17:902:e5cb:b0:1c6:e1d:8be0 with SMTP id u11-20020a170902e5cb00b001c60e1d8be0mr677167plf.2.1697661191282; Wed, 18 Oct 2023 13:33:11 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id w12-20020a170902d3cc00b001c61e628e9dsm354412plb.77.2023.10.18.13.33.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 13:33:10 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, shy828301@gmail.com, "Vishal Moola (Oracle)" Subject: [PATCH v2 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Wed, 18 Oct 2023 13:32:13 -0700 Message-Id: <20231018203213.50224-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231018203213.50224-1-vishal.moola@gmail.com> References: <20231018203213.50224-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 79FB540024 X-Rspam-User: X-Stat-Signature: a3duebz4rfgfeb9zm13fwo8qpt8q181h X-Rspamd-Server: rspam01 X-HE-Tag: 1697661192-363886 X-HE-Meta: U2FsdGVkX1/dnQX7kl9NwNCvX45Ho+vtCI8vzEXGkk4uBE/KXV32wGnMgsCv0rFo0EtxDYlswziN0rTJIm8YCgW68WOTPB6fwcqT1w78DAck4YmeQwEgjkY4+91uqTEP61uLBq4n/Uw2sLRKupfA1xwlTVKi2xke6mm13ZLeVtBLb7EZZf6SfXfytj4fSmU+7bd37QfV2KIgsQCevtEwomYEVrh80oCjnlPHkoB5OGsIUzgtPhtS4qbLxRHwU9YVU8FrVamtkWXQZnzT/sAUF7ceASXtbz7oK5Z0EWxjhbHt3eH0tLQcovNcg8Yi0wFlO+J8c1r81VaipZ+1tlNQAPJ226cq5EXq20PaWDgm8S/DI6mW4usy9pso92hDFmbc40RfCIlQnuI5PJFVvsf35ey5ocumHo7ITYZ2W0CY21vUvAJasbH//E0Q5dkI5+8NZGXzkVsaPqxsmwtZo0tKw5+VOJl7XkqU1vLCeqVEs+SGgByH5joTlRl+nNucmJDTZ/X5qmW/GTIiCPbaeHokKAvvq0J8cRSXP5XI+59dy9qaW3FZMkLCApP1PoCD1PCxYv3IeCaeVVl68l8Ejnz72uXEd/sHM70KjtZDZ/a1ct3Fz4GfvYpRp3VXFjBqxthV5k4/dBzUxJ6D138j82GiuiNuQB33IyTdtZ2BIRKs51RFfdgaiCMo2Y8Nonp5enRBDy4Z9QjnAqM830FGJxxqo7H0JFfaz/iGDKyi77jrJOMFzlqT6ZCz+LO6kg9uK6oDUZE6q25bS+6pMCaVJzqUngTEUWTw35n1VCDzCA2Ms04oXZSf+iFpWbyij+Z8CQL98Z/TUEEm3zEUsloQPXi357S0SorBoN+1Qvu9j4tPgRLuV0Cxu0bRtfiQrw6ZH+ZBIltTfPvqBbz8W0zAMP6wj9UOgTs7UaG59C7s6kz95dpruZpJCXilaTlu+peAyLGf7RN/Dvv7JstA3kJxOhJ m7QTpeTu RB3R0EkIqtQIPme3GE5QiTWh6b751ymPC3DHqAcw/gUWAMR359/Ewus3g8g4/iRb0jbkfvycrvudAP/7Zhfi/dPTda6d4YkG4BUwJD5xH2Eje3GHRgyg5U1kY8+gYxDzm5OAsYRKijIS3939zA8Av+PM5B5eUf5eqSr8xiE863u0lFu/op8gM/hQQY6iIJhilomyccrfwQmUyoVs90LnqrHL0+nfb5Geozoezc2NveUvSzLBzbc925l+/MI1CTj0jRNZrPgFnnz3Vnk7Wxz1UgWpQs9Pry5VQDBz7sRjr2gFyBpJFOaGJIufndln2q5tQOynXOWeXXyHohlOnJnEZhWrqSxyD22zDl3MtFGbDAiZE2b+pDhhr51ov+3F+Dxzd/HNC7G0dQjqmqIC+MGm+FCPAVCRAySgHj7K7ZvGlBUZFQB6JIfVlNrqVmkzNU1dOS9iqY0mQGeolvxvRTfmunawKT6jT3sIadDB5OW5WV0MD19grbxPLMlIbYy/uX5V+eLeYAGE+TYDBmQpJJdMYYtq50ygyMoSUMAJRsLF1QMWJH6wA5VNgJng2inaaE3brX8wpPjwjaXlN5+kq1Cvjilt2gA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we can get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) Signed-off-by: Vishal Moola (Oracle) --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d49aa22d99c9..94c1dd09a8a6 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1475,7 +1475,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1508,19 +1508,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (!folio) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1534,13 +1529,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1565,7 +1560,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1580,7 +1575,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1604,7 +1599,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1613,7 +1608,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1632,8 +1627,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1657,14 +1652,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1672,9 +1667,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }