From patchwork Mon Mar 25 22:33:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13603118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 779C4CD11DC for ; Mon, 25 Mar 2024 22:33:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEF7E6B0082; Mon, 25 Mar 2024 18:33:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9D716B0083; Mon, 25 Mar 2024 18:33:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C63636B0085; Mon, 25 Mar 2024 18:33:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B79496B0082 for ; Mon, 25 Mar 2024 18:33:46 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 863E81C0B16 for ; Mon, 25 Mar 2024 22:33:46 +0000 (UTC) X-FDA: 81937014852.28.6236315 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf16.hostedemail.com (Postfix) with ESMTP id B22C7180003 for ; Mon, 25 Mar 2024 22:33:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ARjN5ctS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711406024; a=rsa-sha256; cv=none; b=FPGgkmOMQveDjtPqtyAJoLmG0F2D6TGUywTKpGLFCT3OMqqBxGIHIo9NNcCepAN8Ef/t3E KZRDD4dv7mTsm5+uWLTuABsawD3912tj/tgVmrspRatk6CHXARcT2G7tiH2z6dFzgAXD1F UeITNgV3Po+14nZKUY3UPi/EGqvjU6M= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ARjN5ctS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711406024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7wuUkLEO8wS3NnK8pO29sbwl4okjZZVm7bwiudBCZKQ=; b=i+hzY0NEGFJgIOUFosrpuAEGK46u3rutcvkY24fuKiUJMPM7uRi+yg1ajVJ/S2GdXOFTjq QKhhGanntTYNjDmo8n1w/OkaXc62i6NCkZ8Xgs3HVGwuc/yU5AkVM7TUl6OdIbtxuc2Ppi +PsN+esqMXCj9+qkRVSj0SqMSPXEXO4= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6e6f4ad4c57so3669526b3a.2 for ; Mon, 25 Mar 2024 15:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406023; x=1712010823; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7wuUkLEO8wS3NnK8pO29sbwl4okjZZVm7bwiudBCZKQ=; b=ARjN5ctSp95n0RRJ2l1dEhCHo8RqqFxaei78f21s2rkpAemHRwpfa+o7PDM9qQzhRo 9pNqT9xRCe0g4V2iwvXfR3W/ZAeXt3ar2n86FCXab2jQ946Z+XaauV15XJ2F+jtXB2Uw XfNdNc+hh+wgfPLDfJrwNekf39aMQJuFVo3zkU+lNn0WlqIw2j+fbS0INmM7Y2WYu+M2 zB/q3HoqeUyt/NuoIX4leM0ELpIbvZZxb2rMA6KR+jhc83RfKGvExw/wtNNI10H6rNLr eHyzgUSPqcBKd8S6wEcP6jfllH37GDWcCtdidgrP4eNJRsIZ6YqlfqPjaNaz2nr7bJiK ZLmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406023; x=1712010823; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wuUkLEO8wS3NnK8pO29sbwl4okjZZVm7bwiudBCZKQ=; b=tFBe7axyQ3IfSK6YCuO9br0XXfRIzUYsXrFkRmQooChW5YGV76Q06x657RrtOALx2D 9gjJSqWyo/7dP1ElOp6HWghzm2cyLzPeqAe3Mh2eKpQ81mNWMsueG+JCISy9auDntkVy HDSR6CA5DDD0B6JXZ2b3gKv35bNaFtpEw/ekQuLfwmedQQoO/pDdeWmaZXSuFSxvuAui KojLuSplp1ymsSgT5029DDaiqsSNTY98w3OITl2yDiBcMkgWL7M89lmGoRcM2VVH+Avo aUxyAHfUiUnpsS/4kWaAYAmaoQuoKN+nKfAlm0Uzz8NiEE7JquknXYSMrLoxr9Vioxhq QgPA== X-Gm-Message-State: AOJu0Yyxf6vqBqaXanSIUpyPu4+9I4vX4U6CoEQysrp3a8Y3lyZpK6n9 2WWWKHhDf7WKczwYJ/oo7dz5I3b4CGSVjcAE9O0AlEcxiyDIG2QhBDqzyaiBlG8= X-Google-Smtp-Source: AGHT+IFl6KFavtRUNTSrxnwpqS8RSMDEAwpK70R/1RBLnR40J5C5dnwH9mZD8fjRJHapYDc5t8I8Zw== X-Received: by 2002:a05:6a21:3942:b0:1a3:cd06:20cf with SMTP id ac2-20020a056a21394200b001a3cd0620cfmr4148137pzc.16.1711406023376; Mon, 25 Mar 2024 15:33:43 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 1/5] hugetlb: Convert hugetlb_fault() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:35 -0700 Message-ID: <20240325223339.169350-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B22C7180003 X-Stat-Signature: 9qe3icsnh7j9t31xn1yyuhurymq7xnye X-HE-Tag: 1711406024-183630 X-HE-Meta: U2FsdGVkX18YdG7zbDxHXCHHaXL9EfmLmbFNteRGmKH0tpRqzVIl+Jut/XtzkO8sFhbdmbuFIQMKW+CYHaWPubEGxw/C+Ibk9cuH4iMYiZlqGifwan9pK8Iv1Mek1bgRMfIGQbzmzgW7+ZiDdyczQeAIlsnHJ1v1xIGCkp5G/Fh//YKqfX/9KQMYX/g0wC7w5SmegXwwGfcGIwea4nri1PZKz5ySnGs8kjtuan44Q2Jm8GOW1W+JSOpdo4ypuvWC9NyX+syV3X7/IiQiL1CuJZ1SLcZp9SDQWB6f83crhxlJw9kcxhd87fG3A81l2yfmP8zidTe6kuJf0zjBI3qZKgL+muALGWT8vxRlVpRXlLMmH2Sq7evnzhc3i1G1HTKnFa6YxbpZtW/l5LfyglDpz9xzwHIiK1ZeU3AM58s39uCOLWyIwHRLxqPzgtl8nHsDjec9fILFBYHBYOiXWvRhIGSJJNsj+JDD4i1dpBZmqAZq8ulYJK4ZZZhMKaaDpW2OxN8k/eDijCQFSyREfcfRYWu85IfQXdAnirevSWY7dS7jJgKPJbtzHgVcIVnoSF451TxqFv36tq4zf+SdMRtOVGeEdcVmJ9eT5ad/KpoMyyStSBfTCu1cLaU3vfeYQdMimAsXlQjUltViohGIzhTvpGfcPmGd2SHb2wpc3rFwdCCcYBujYh7X1wPgx4arQxLIc0ThmQqA7jPBzmwzRnjosoJjeoUtg2h0KMtfhRq/xkyOIsDsP6QJrPdOdrWLyrHq5bu0Ony0QAB6IVztT1OjX2KY9VvDzEZTJRKhbBe5r1NpteIhKkNv2rhKEbjeFOaHqfga0Xs8I33E3TmyqnulukyoxviJUqse5Ck+moYX+Upih7tPIqewkfz9utFpqDGADORA63sehM5koIJq7bG+aleVDJdrE3plnfzp9B656wglrM80BUU/6zbokikKQgUs4utCF9QvFsEshE1ZDu5 awl9YtgE fsHlFvDtsVdkPbvLfUJGMJQMhhmfbbMEczQ/wivVq9jaKFn3ojd9GgG28lXsY/lIFons3sEnZBuZgCEsCfWxODZtefRsehlkmmT3egXSqlWR7/E1lAt7QfnUN4e0gRoi6YjCUj2YJIiCRgYGTX0GLPSk3W4v2rjH54pjLk4yP6MxK1WOHJsuEIifebEvIj0ulG3z24cGbr1hWm5LtgcHECaA+2nGsrJXQkH+iMugYVCOMTAcDJ3Fz9z6gEslcaecqUOk0evtvcrPy/stnwFVu4+CHJrmk5gytX9oUqu7lqWVvfba/3mYaahb5i6pI1x3vU7qYof2EHjAClvBRkybOVjNmcq39YhClJR2J5Fh2x5xSKPbL4O+vQcratawhxUl2y7SpIZs+nS5S5Fw9cA9Doo4/PYWDHKM1WNvVHFYGmy5K/YChna+U3GX94kuSmib02pzgJvp0sJPLMTRjJFMxd+KUqOQ8k5Ip4dcLtocH4pyotMTj2pwBzT2THXVRO/2U/dTS0Bs2gxiQjakijStZbBWgXNu44+tT1wp2j55goFcQJPbJTvgSRQOeT0aU+UnHJIbNonC8I8ehmkZR/SPA3TC4/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that hugetlb_fault() has a vm_fault available for fault tracking, use it throughout. This cleans up the code by removing 2 variables, and prepares hugetlb_fault() to take in a struct vm_fault argument. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 84 +++++++++++++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 43 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 688017ca0cc2..81e8ade53b64 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6330,8 +6330,6 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx) vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags) { - pte_t *ptep, entry; - spinlock_t *ptl; vm_fault_t ret; u32 hash; struct folio *folio = NULL; @@ -6339,13 +6337,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, struct hstate *h = hstate_vma(vma); struct address_space *mapping; int need_wait_lock = 0; - unsigned long haddr = address & huge_page_mask(h); struct vm_fault vmf = { .vma = vma, - .address = haddr, + .address = address & huge_page_mask(h), .real_address = address, .flags = flags, - .pgoff = vma_hugecache_offset(h, vma, haddr), + .pgoff = vma_hugecache_offset(h, vma, + address & huge_page_mask(h)), /* TODO: Track hugetlb faults using vm_fault */ /* @@ -6365,22 +6363,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire vma lock before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with vmf.pte. This prevents huge_pmd_unshare from + * being called elsewhere and making the vmf.pte no longer valid. */ hugetlb_vma_lock_read(vma); - ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h)); - if (!ptep) { + vmf.pte = huge_pte_alloc(mm, vma, vmf.address, huge_page_size(h)); + if (!vmf.pte) { hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return VM_FAULT_OOM; } - entry = huge_ptep_get(ptep); - if (huge_pte_none_mostly(entry)) { - if (is_pte_marker(entry)) { + vmf.orig_pte = huge_ptep_get(vmf.pte); + if (huge_pte_none_mostly(vmf.orig_pte)) { + if (is_pte_marker(vmf.orig_pte)) { pte_marker marker = - pte_marker_get(pte_to_swp_entry(entry)); + pte_marker_get(pte_to_swp_entry(vmf.orig_pte)); if (marker & PTE_MARKER_POISONED) { ret = VM_FAULT_HWPOISON_LARGE; @@ -6395,20 +6393,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * mutex internally, which make us return immediately. */ return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - ptep, entry, flags, &vmf); + vmf.pte, vmf.orig_pte, flags, &vmf); } ret = 0; /* - * entry could be a migration/hwpoison entry at this point, so this - * check prevents the kernel from going below assuming that we have - * an active hugepage in pagecache. This goto expects the 2nd page - * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will - * properly handle it. + * vmf.orig_pte could be a migration/hwpoison vmf.orig_pte at this + * point, so this check prevents the kernel from going below assuming + * that we have an active hugepage in pagecache. This goto expects + * the 2nd page fault, and is_hugetlb_entry_(migration|hwpoisoned) + * check will properly handle it. */ - if (!pte_present(entry)) { - if (unlikely(is_hugetlb_entry_migration(entry))) { + if (!pte_present(vmf.orig_pte)) { + if (unlikely(is_hugetlb_entry_migration(vmf.orig_pte))) { /* * Release the hugetlb fault lock now, but retain * the vma lock, because it is needed to guard the @@ -6417,9 +6415,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, ptep); + migration_entry_wait_huge(vma, vmf.pte); return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) ret = VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; @@ -6433,13 +6431,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * determine if a reservation has been consumed. */ if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(entry)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf.orig_pte)) { + if (vma_needs_reservation(h, vma, vmf.address) < 0) { ret = VM_FAULT_OOM; goto out_mutex; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf.address); pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, vmf.pgoff); @@ -6447,17 +6445,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, pagecache_folio = NULL; } - ptl = huge_pte_lock(h, mm, ptep); + vmf.ptl = huge_pte_lock(h, mm, vmf.pte); /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) + if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && - (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && + (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { if (!userfaultfd_wp_async(vma)) { - spin_unlock(ptl); + spin_unlock(vmf.ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); folio_put(pagecache_folio); @@ -6467,18 +6465,18 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, return handle_userfault(&vmf, VM_UFFD_WP); } - entry = huge_pte_clear_uffd_wp(entry); - set_huge_pte_at(mm, haddr, ptep, entry, + vmf.orig_pte = huge_pte_clear_uffd_wp(vmf.orig_pte); + set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, huge_page_size(hstate_vma(vma))); /* Fallthrough to CoW */ } /* - * hugetlb_wp() requires page locks of pte_page(entry) and + * hugetlb_wp() requires page locks of pte_page(vmf.orig_pte) and * pagecache_folio, so here we need take the former one * when folio != pagecache_folio or !pagecache_folio. */ - folio = page_folio(pte_page(entry)); + folio = page_folio(pte_page(vmf.orig_pte)); if (folio != pagecache_folio) if (!folio_trylock(folio)) { need_wait_lock = 1; @@ -6488,24 +6486,24 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, folio_get(folio); if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { - if (!huge_pte_write(entry)) { - ret = hugetlb_wp(mm, vma, address, ptep, flags, - pagecache_folio, ptl, &vmf); + if (!huge_pte_write(vmf.orig_pte)) { + ret = hugetlb_wp(mm, vma, address, vmf.pte, flags, + pagecache_folio, vmf.ptl, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { - entry = huge_pte_mkdirty(entry); + vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte); } } - entry = pte_mkyoung(entry); - if (huge_ptep_set_access_flags(vma, haddr, ptep, entry, + vmf.orig_pte = pte_mkyoung(vmf.orig_pte); + if (huge_ptep_set_access_flags(vma, vmf.address, vmf.pte, vmf.orig_pte, flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, haddr, ptep); + update_mmu_cache(vma, vmf.address, vmf.pte); out_put_page: if (folio != pagecache_folio) folio_unlock(folio); folio_put(folio); out_ptl: - spin_unlock(ptl); + spin_unlock(vmf.ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); From patchwork Mon Mar 25 22:33:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13603119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1776C54E64 for ; Mon, 25 Mar 2024 22:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A91996B0083; Mon, 25 Mar 2024 18:33:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1A316B0085; Mon, 25 Mar 2024 18:33:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 848576B0087; Mon, 25 Mar 2024 18:33:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 72AC76B0083 for ; Mon, 25 Mar 2024 18:33:47 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 52EFDA086B for ; Mon, 25 Mar 2024 22:33:47 +0000 (UTC) X-FDA: 81937014894.23.43171FE Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by imf22.hostedemail.com (Postfix) with ESMTP id AAD94C0009 for ; Mon, 25 Mar 2024 22:33:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=D5mFHqEh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711406025; a=rsa-sha256; cv=none; b=JHXF8vDmXVmCrKZBsL6WVR8DEvRniWRN5B+ybI8IlcjwG2sj50SMqCvdkVmhBkdHK5yiZk EEs9sMqTCWvZ71m3557aj2SzBexVcS7IY4OvXoff+zWlhFUbHegRTSG0UWPp0JFJi/fGvN BEFB9HTUl26SI5wgmpZ3+4XPLbdblqA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=D5mFHqEh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.215.172 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711406025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sP/HrRtOaF5YRIfX9u0RpV4cBbAlDIrGYsjsP3+O3X8=; b=MF67zR9kWt8n1m9IukJH6a1rTqVcWBIQBQDfyiK8jOj4bjULikv1jXxJTUwOyxPEHORv0X i6blqHkwQ+3OL6sSnJh9n/2m13S5w1am3lS/hH0bFCAyO3cVA1e6dQUxlqqWSmrEGJcoKO OHD5Y7wCLOso11g4zoBh366AoxDJ738= Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-5d8ddbac4fbso3427619a12.0 for ; Mon, 25 Mar 2024 15:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406024; x=1712010824; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sP/HrRtOaF5YRIfX9u0RpV4cBbAlDIrGYsjsP3+O3X8=; b=D5mFHqEh7hunY5rNB51rqkjO0sbgp7WoLH+OaBUC1aE2cYUlErQ56B+IxOZnpzksG3 Ce8dFsNubFi7SAHCJcn3Q1wRGjOwHXDKwx0zJ/9I56PVfMpRuB4QGQxSD3vUUPWym5c5 mOjM81IakItPb13U8/wPYbFIZx3rN+egpU8gLHzNwCGCl28oKbpecg+ihkGaNyuLjBAe kEf13KNrjNpv6JvnBLRcoYYCdsPimre2t+ZI3fmc8EcDQp60lp26wd7FAIJ4uYBY53je AQAxzUCyy8/Flq2Au9GoiA9/7/yr6jrYQpgLm8nlxIAGKNJtr2V/izbIiKxaLRmXrxa8 fA+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406024; x=1712010824; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sP/HrRtOaF5YRIfX9u0RpV4cBbAlDIrGYsjsP3+O3X8=; b=CY6WHYPaK9pdTcywRG1f1m+w2WWjHG8hZBzWRHDxE12eJUu2dxRuRihPwYC2vKchch 2WqJZ+ec68HcZhJka1mreELB6HAA3QK1rqNUL5+24Pu9YjdIwSK4T4NN0m9EzuRgHkT7 TEfMdvFJLO5ykIjCjITEVvS4Lv7t9rMprVe1aqNzo82pIllKe7M3lF/1gd/ypUQkZ9f0 rZ+bycCz6+Xe/RMPfsM+vaFTgSw2Vvdx5a0EvKLYMxWIvVtfwXAFCpV4uJZFAq4IWMTt X8LTibNXD5Xxm3lyrwScrf40Ntp8g14TFiixG0nMqbqiH5DxmC5HRrTltcLaY7kmw3jW 7D4g== X-Gm-Message-State: AOJu0Yz04Ifoe0oG3DXSHaUEZgxWcoWXhbGsF8SYkL/IotLL/aroaf72 v/mUvGji3cGbEfqXOR3HpvnsvA5Q+ovF8Ngf4y4qwz5Avx0/YMuX4tNfs+Z3KBY= X-Google-Smtp-Source: AGHT+IGyWxFaoFa1+IvC2ORjow3Na1wp3zhzntyvupejForDNfjnFm53ZyuyQXm9jJ+7DPPZog8puQ== X-Received: by 2002:a17:90a:2f26:b0:29c:7769:419b with SMTP id s35-20020a17090a2f2600b0029c7769419bmr8194248pjd.9.1711406024293; Mon, 25 Mar 2024 15:33:44 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:43 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 2/5] hugetlb: Convert hugetlb_no_page() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:36 -0700 Message-ID: <20240325223339.169350-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AAD94C0009 X-Stat-Signature: uz77zhsik9gt46mgqrmsxruiiq9m3ghg X-HE-Tag: 1711406025-325698 X-HE-Meta: U2FsdGVkX18A5ZLoZ7gXQxXiOitR/RkvEt63PlfA3HovGyOwytpp/p82HeLjrIhXksbbtSWT5agIiVJ+RvoS7BwcHQZ24YHsaaBVSa6MnFvMapElLuSUyr3ztDVqXGESFv6jUpAxZ1yuyQ3/v4/HesAOmJeCfU8OQrne+qtS8Lz5tkbBim8jooNWXJNpDMUqfWohgaq3sUqGEPxGO2ifoD46RgyqU5Yh6myWg7h+W+RPLgx0loqQs+D965LWDlAqKk2Gt1Ddp9n1YZ/8paPCns9B/slzUzfKNl6vuQlM2ZM1VmWIYSBhtFTtcRFX7/EAj22bI3fOPELz7EfTmrXRLZ2UBZlt+HlS6dPnuM6aMa2nT/cLBPNQO+vrJrWwrl6uCq5CE1HoWECwqLDN5OFgek81Ug02L5nPj7Y9VfrGccgSmYWbAQovFPa33VfeJziChjIvKS4JLAchERQhMFAssdF+f10yjt86PMRD2uJtIlf/nmvIkVZwJx6ohWgz+4OQeId/kt/jAxFtZYcyu43/rgoIG72bcpfG3GMlYBUI9Kf/Lg9c3Ufc4e3Glf7zPNOd0QqUYqvXh6ugHGBMmHbkN11AQ4DyJopuycuAv5u03lW7M09rbtmv4Eatjg9jM4Lu20Rdrp13h2MiSIM//Wdn9Dktf5GbmRu3flRlDQBtP+250atAr3OhskIe+FVxYRUmi7iRsdb8fLHVLHvSbAHlMgq46MIzFf3fgo4oLZxvP3TfGAUTRVmdzcvGbi8fVN3hhsPH2BVEtWWL0F9aziTbL6OAxZEFSiMsHbYiK7v7Voq/apm/3ooe9nSuaLq34uYuQYhnah2jpTlmZKDvl8ROpmQIqbck6dMXoiO7uXG+Z/mu69C026d+lvvwHW0CmYbgnwxJGU5vSOVRhXF0p7NZP6RWeoQeLdjvPXX3PbXwbW/t1Lrz3n8mn7uRCd/GpMR237EGxGLCo8MR/YiRvD7 WqrSABYf Io+n0KociGXJz8y9fOTroGIkgKi2fY8N1oSN76EsCfZ/HM1ZzPlHOBa73oXCtLNZahao4pQf2KpoDkEq5bXvEvyeP/Ef9YzWm7XygXnW0LsfbGwum8VeDjqPaMnAWgdQA3peT6wCbJCQpEPaQ7IJ3ybv/oMS/ws5ri5ASWGX7v6Zn/l7Qx+l7wHkvXjVs//GXv0epQwNBn+jIc6k3XRs51fYHFUtV//s5U6+LQ4kymInAuJZGer7Ci51/8qjKTe/6fy8wV/50vwGYNak= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_no_page() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 7 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 59 ++++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 81e8ade53b64..819a6d067985 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6096,9 +6096,7 @@ static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, - pte_t old_pte, unsigned int flags, + struct address_space *mapping, struct vm_fault *vmf) { struct hstate *h = hstate_vma(vma); @@ -6107,10 +6105,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, unsigned long size; struct folio *folio; pte_t new_pte; - spinlock_t *ptl; - unsigned long haddr = address & huge_page_mask(h); bool new_folio, new_pagecache_folio = false; - u32 hash = hugetlb_fault_mutex_hash(mapping, idx); + u32 hash = hugetlb_fault_mutex_hash(mapping, vmf->pgoff); /* * Currently, we are forced to kill the process in the event the @@ -6129,10 +6125,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * before we get page_table_lock. */ new_folio = false; - folio = filemap_lock_hugetlb_folio(h, mapping, idx); + folio = filemap_lock_hugetlb_folio(h, mapping, vmf->pgoff); if (IS_ERR(folio)) { size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) + if (vmf->pgoff >= size) goto out; /* Check for page in userfault range */ if (userfaultfd_missing(vma)) { @@ -6153,7 +6149,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * never happen on the page after UFFDIO_COPY has * correctly installed the page and returned. */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6162,7 +6158,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, VM_UFFD_MISSING); } - folio = alloc_hugetlb_folio(vma, haddr, 0); + folio = alloc_hugetlb_folio(vma, vmf->address, 0); if (IS_ERR(folio)) { /* * Returning error will result in faulting task being @@ -6176,18 +6172,20 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * here. Before returning error, get ptl and make * sure there really is no pte entry. */ - if (hugetlb_pte_stable(h, mm, ptep, old_pte)) + if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) ret = vmf_error(PTR_ERR(folio)); else ret = 0; goto out; } - clear_huge_page(&folio->page, address, pages_per_huge_page(h)); + clear_huge_page(&folio->page, vmf->real_address, + pages_per_huge_page(h)); __folio_mark_uptodate(folio); new_folio = true; if (vma->vm_flags & VM_MAYSHARE) { - int err = hugetlb_add_to_page_cache(folio, mapping, idx); + int err = hugetlb_add_to_page_cache(folio, mapping, + vmf->pgoff); if (err) { /* * err can't be -EEXIST which implies someone @@ -6196,7 +6194,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * to the page cache. So it's safe to call * restore_reserve_on_error() here. */ - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, + folio); folio_put(folio); goto out; } @@ -6226,7 +6225,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, folio_unlock(folio); folio_put(folio); /* See comment in userfaultfd_missing() block above */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6241,23 +6240,23 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * any allocations necessary to record that reservation occur outside * the spinlock. */ - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if (vma_needs_reservation(h, vma, vmf->address) < 0) { ret = VM_FAULT_OOM; goto backout_unlocked; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf->address); } - ptl = huge_pte_lock(h, mm, ptep); + vmf->ptl = huge_pte_lock(h, mm, vmf->pte); ret = 0; /* If pte changed from under us, retry */ - if (!pte_same(huge_ptep_get(ptep), old_pte)) + if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte)) goto backout; if (anon_rmap) - hugetlb_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, vmf->address); else hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6266,17 +6265,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * If this pte was previously wr-protected, keep it wr-protected even * if populated. */ - if (unlikely(pte_marker_uffd_wp(old_pte))) + if (unlikely(pte_marker_uffd_wp(vmf->orig_pte))) new_pte = huge_pte_mkuffd_wp(new_pte); - set_huge_pte_at(mm, haddr, ptep, new_pte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h)); hugetlb_count_add(pages_per_huge_page(h), mm); - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, address, ptep, flags, folio, ptl, vmf); + ret = hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, + vmf->flags, folio, vmf->ptl, vmf); } - spin_unlock(ptl); + spin_unlock(vmf->ptl); /* * Only set hugetlb_migratable in newly allocated pages. Existing pages @@ -6293,10 +6293,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, return ret; backout: - spin_unlock(ptl); + spin_unlock(vmf->ptl); backout_unlocked: if (new_folio && !new_pagecache_folio) - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, folio); folio_unlock(folio); folio_put(folio); @@ -6392,8 +6392,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ - return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - vmf.pte, vmf.orig_pte, flags, &vmf); + return hugetlb_no_page(mm, vma, mapping, &vmf); } ret = 0; From patchwork Mon Mar 25 22:33:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13603120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88294C54E64 for ; Mon, 25 Mar 2024 22:33:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 230746B0087; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B7EB6B0088; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F276B6B0089; Mon, 25 Mar 2024 18:33:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CDB036B0087 for ; Mon, 25 Mar 2024 18:33:48 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 95B90C0122 for ; Mon, 25 Mar 2024 22:33:48 +0000 (UTC) X-FDA: 81937014936.14.E4F40BB Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf01.hostedemail.com (Postfix) with ESMTP id BCFA540014 for ; Mon, 25 Mar 2024 22:33:46 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NzA5hU7H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711406026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mVOzucr5mvu7KOVzAqxUx6FA7sdm3qrl6lnpegwv/As=; b=CPHhDEXBjcBgNLYkLdSOJUiDeyJKIc3qRJPjlKzt+GO5V0X9p8aE9epSx2bWZegdW4RZYz irsTb108fsLdAJrVcfht8nW7FfU9C/cZpx+EXkIm3qUKXQkk5BF9pg7kmL5cVnE7aSQ6FO 03sZ2qssXv07/8mm83jQXcz0mU84lMA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NzA5hU7H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711406026; a=rsa-sha256; cv=none; b=hB8I229YMoH7eG960/gzV7Wp83BvVU6k1QrU//YrTiZOFYh3YQx1kecE3TQF1tL+8/rX1K IS8mVYNr+pIO8dIXFUFYD8qflr1XBrgmbmPD5sLathaULfcZiPdNX8oyfQ9Y4DDJttalDU XvH3tW1nb3OdIst3Dl2m0GdXz6dd+SI= Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-29ddfada0d0so2569310a91.3 for ; Mon, 25 Mar 2024 15:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406025; x=1712010825; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mVOzucr5mvu7KOVzAqxUx6FA7sdm3qrl6lnpegwv/As=; b=NzA5hU7HikQM9BPyXb5DNEsrFG6jf3rJDKCs83qJFu5nCAW5wnmeAOmwNkAtcVOFj7 pgCJKdITON5H2RGXV3PHkYxlRYk+4GgcYObpEAJW0OmeqahpWgGD8ZGcxStpZVlWmN/E s6Z7y/q4t9w7n1KEh8IhoKWQnBJazEUSMlqxql52fEC6vkTDnwszuj4Cg2+DCS/QStxW FPgLa+H7WNZhErcGrpnid7Ugajsf8H7C6w8pfXS8CVYs/a07zGxr5Avft4o+VEwSgUQ+ LBOOiGtosJKzDvneoJNGsu2SmZCcG1lZ+LdeKUObktBnHLR3taXtEq2ocXQS93AEqQmg ahTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406025; x=1712010825; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mVOzucr5mvu7KOVzAqxUx6FA7sdm3qrl6lnpegwv/As=; b=A9sXa/cE6V/jYqrRSA1/0Dar9fDQv6tngKkpmzPzQoaQf/V99+M7Yo3HDnenZko6Hm F7Y89N1C6j9tNNWPi2upwFXTuQi+d9T+lOAttZpu0Imot0u6/VI69NM8iNsrg+at0Qin 41NQOmMpE1hqeDvzqnrB3FzBkl9SWhZAwBpNwjPB8g1Iz6e4GA3u9dznrlpMGYgQsVHU RUYl3/6wmJTSXowFbSvIi+wCIGTs0G7P2cffUeaoCDz2AT0wWe+/+LG1ciuoepIxv/Db RmHr19iFsN3KdCGRyyOub8+AfPQfP8w8IpJmjp85XTsJndFIFiiKaTm5HjTj7eQbjcwo 2ucA== X-Gm-Message-State: AOJu0Yw33K/VQnndsv7TXXwXUtvXV20DBhua1knB4Ibl9M8suZ/oY7Y6 Cg+GZk26YAHyuXQTCfcWJtixrwuN7c4B6OqB9XnJekgA8tmRHkVEeWSBprAOax0= X-Google-Smtp-Source: AGHT+IFDk0utxLL00lJGKaaeGgNK3Hrl4HLtx0LXzLecUbp3ONOONhamzCvdDjqJbMq5Hf8iz/uE5Q== X-Received: by 2002:a17:90a:8a03:b0:29c:6000:a12b with SMTP id w3-20020a17090a8a0300b0029c6000a12bmr1164130pjn.38.1711406025338; Mon, 25 Mar 2024 15:33:45 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 3/5] hugetlb: Convert hugetlb_wp() to use struct vm_fault Date: Mon, 25 Mar 2024 15:33:37 -0700 Message-ID: <20240325223339.169350-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BCFA540014 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: t567hnia4b8sug7n9zzd3ba78ua6mgcf X-HE-Tag: 1711406026-962908 X-HE-Meta: U2FsdGVkX1/0SaSgwx5u4TADlib8YDXRBnpmayRqZxdrPyHLmWtbEsLB2zfoghJ8pFLJ13TScB5OwWfSryKk9W53V1w60nveFcQmZ0f/OTKSBMhGOZUu3u9RqEWUWU/MYCsKs+kqCu6EqypwFWEMKm1HkiccAui+gyFlAaWV8BnSTM8OAfPxgmd+8ZhrGYEzZEZCgVXBDsrmZ/ElnaIWXSR4lSd1mY/zQDdTNS+9uJhLdZISMJhv/UwCu/OAPTpAREh5unhYKbWhhLohWzHbmjQ/8MfXr+pPBMMgrPUOJMlPs8T2fASzxnpRZkeaGu5yBmGQ1fipeO/PAuSmBtrTCIwjMLsQrLkcWaQXQ8T42zHlQReIfH5Iur8MPlH3qDCodqOO0eM8IlKgDqkwmgfR++ZLL3ISSqzasrxPK/XR0trvP5fma52my0TOfgyafW8Dsgu3vXwqTRnbX8WK3DFWqGeES00vpxkwvuwzNFsKDZt2SVzKPcAq5/hIR8uRAXRieEhwnF3NU7r/1HpdpqhP/0yeW+E4VId4kdphfeJgGNBcgCzB10ym5q4cAH1GPlq1JAr4MaBUUWWGhHWGmTagZkzJjStvjfH3j9Vkm/ZoFACL9zJ1ZG4ftLsgeRATnl9VCwKP2Z8vNLu6t088UjwIDFvl+dvL1uxrVsqjHyd5kn45T8U0DYhkTSJ4s3rDEj6HBizMzNHGCdt8yF1PmpI7pojIDaj1JvSUUM+GCsuqS+pN4il7uHGwPl/V6XPBV3B9ln2QOjadUTgVre5J2mgBKmG6LFWyBXToDS28knOSsSVdTg8VwlVxScO8snSYtd5mnO72QGGJV1v/loltxCQfJlEGM5jMVrN9Y8+IUf6ynacuBQdJ5xOVDu5/lg7sk5oPXpsomZidpPHXNWojROhuke8Gf9QHlRHdV8OFmD5+NBZHJnsjrkD0nAMih8ELkYe1DaGMhnFM53cwXIz0ueC cv9ldWkk L5IligGqTsLX/ROLSdZio+8ScXCmey9GqKlQJtH3b8sUQSQBg2LXdUAZBdbc5GuVIYU5TjyHrxv0wqr8fztGQsjuZaoJ1iP9+dxlF7g7zqmfGLa3Ybc6RK3zQJe7l9UV3Tw26ljxoof3C3g4rbYz8OIswMicC95WpYYYuji2BvOLeAGJu/rKN/b8i3Bb+mHqgpaJRBEdst04e0zjEx03Vzkln7Wg/+JJw30N0kyaQXtz8goK8a/5O8pZ+2YgibXnuZRZm1mWZVWrhvJiMv/0csLmedmMCv/t1ThhrKpu/ZFeOIuYL7M1QJVdtS5wA8/VnxolJUtlXUPk6/FvFteBz4WpZoiYjk5cFfEJodUHbDTFEX+F6beUhaJ3OwIIJA+We6US7JNj6M6Cl/tSuiF6EUEI4aT2pbocLSi4kMEa1DQj/99hEJO+lpyml03O16D5jBQiiZr2XiE42nga70d2CcuQFKhJEQ0xoLNSaYw4cAR+Qb4Vi7n5O8R1pLanMz6AuS+oyIvt9fUihJBTVBRSvBHBjZFNxPmCY+h7vCXASjHms557wtO8I1Oy2UXljlkX61/kt4tr+/Quk6hFbo+FbL9r+fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_wp() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 5 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 61 ++++++++++++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 819a6d067985..107b47329b9f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5825,18 +5825,16 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, * Keep the pte_same checks anyway to make transition from the mutex easier. */ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, unsigned int flags, - struct folio *pagecache_folio, spinlock_t *ptl, + struct folio *pagecache_folio, struct vm_fault *vmf) { - const bool unshare = flags & FAULT_FLAG_UNSHARE; - pte_t pte = huge_ptep_get(ptep); + const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; + pte_t pte = huge_ptep_get(vmf->pte); struct hstate *h = hstate_vma(vma); struct folio *old_folio; struct folio *new_folio; int outside_reserve = 0; vm_fault_t ret = 0; - unsigned long haddr = address & huge_page_mask(h); struct mmu_notifier_range range; /* @@ -5859,7 +5857,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Let's take out MAP_SHARED mappings first. */ if (vma->vm_flags & VM_MAYSHARE) { - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); return 0; } @@ -5878,7 +5876,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, SetPageAnonExclusive(&old_folio->page); } if (likely(!unshare)) - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); delayacct_wpcopy_end(); return 0; @@ -5905,8 +5903,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * Drop page table lock as buddy allocator may be called. It will * be acquired again before returning to the caller, as expected. */ - spin_unlock(ptl); - new_folio = alloc_hugetlb_folio(vma, haddr, outside_reserve); + spin_unlock(vmf->ptl); + new_folio = alloc_hugetlb_folio(vma, vmf->address, outside_reserve); if (IS_ERR(new_folio)) { /* @@ -5931,19 +5929,21 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * * Reacquire both after unmap operation. */ - idx = vma_hugecache_offset(h, vma, haddr); + idx = vma_hugecache_offset(h, vma, vmf->address); hash = hugetlb_fault_mutex_hash(mapping, idx); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - unmap_ref_private(mm, vma, &old_folio->page, haddr); + unmap_ref_private(mm, vma, &old_folio->page, + vmf->address); mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && - pte_same(huge_ptep_get(ptep), pte))) + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, + huge_page_size(h)); + if (likely(vmf->pte && + pte_same(huge_ptep_get(vmf->pte), pte))) goto retry_avoidcopy; /* * race occurs while re-acquiring page table @@ -5965,37 +5965,38 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(ret)) goto out_release_all; - if (copy_user_large_folio(new_folio, old_folio, address, vma)) { + if (copy_user_large_folio(new_folio, old_folio, vmf->real_address, vma)) { ret = VM_FAULT_HWPOISON_LARGE; goto out_release_all; } __folio_mark_uptodate(new_folio); - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, - haddr + huge_page_size(h)); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, vmf->address, + vmf->address + huge_page_size(h)); mmu_notifier_invalidate_range_start(&range); /* * Retake the page table lock to check for racing updates * before the page tables are altered */ - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); + if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare); /* Break COW or unshare */ - huge_ptep_clear_flush(vma, haddr, ptep); + huge_ptep_clear_flush(vma, vmf->address, vmf->pte); hugetlb_remove_rmap(old_folio); - hugetlb_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, vmf->address); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); - set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, newpte, + huge_page_size(h)); folio_set_hugetlb_migratable(new_folio); /* Make the old page be freed below */ new_folio = old_folio; } - spin_unlock(ptl); + spin_unlock(vmf->ptl); mmu_notifier_invalidate_range_end(&range); out_release_all: /* @@ -6003,12 +6004,12 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * unshare) */ if (new_folio != old_folio) - restore_reserve_on_error(h, vma, haddr, new_folio); + restore_reserve_on_error(h, vma, vmf->address, new_folio); folio_put(new_folio); out_release_old: folio_put(old_folio); - spin_lock(ptl); /* Caller expects lock to be held */ + spin_lock(vmf->ptl); /* Caller expects lock to be held */ delayacct_wpcopy_end(); return ret; @@ -6272,8 +6273,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, hugetlb_count_add(pages_per_huge_page(h), mm); if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, - vmf->flags, folio, vmf->ptl, vmf); + ret = hugetlb_wp(mm, vma, folio, vmf); } spin_unlock(vmf->ptl); @@ -6486,8 +6486,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { if (!huge_pte_write(vmf.orig_pte)) { - ret = hugetlb_wp(mm, vma, address, vmf.pte, flags, - pagecache_folio, vmf.ptl, &vmf); + ret = hugetlb_wp(mm, vma, pagecache_folio, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte); From patchwork Mon Mar 25 22:33:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13603121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28824C54E58 for ; Mon, 25 Mar 2024 22:33:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA9596B0088; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B059E6B0089; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 957F16B008A; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 80D7A6B0088 for ; Mon, 25 Mar 2024 18:33:49 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4A31580842 for ; Mon, 25 Mar 2024 22:33:49 +0000 (UTC) X-FDA: 81937014978.02.B9E99C8 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf15.hostedemail.com (Postfix) with ESMTP id 74EB7A0013 for ; Mon, 25 Mar 2024 22:33:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MYRAprQA; spf=pass (imf15.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711406027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BlCoIx7jLqN5Is/HEKy53SFm13f31lm8FF5rHEKGzHA=; b=sPPLP3zuvap2bL9sjbTEArdwSr3kmUd1aC/4ubxWbuTcPn/M0lskF8T7TOCRDt3WK6VKCs oolrTUMjx1EL61WQxOJAsQGZVSVqPaVl2vUFBtxIUbJrILZolkEUV0X5JdkG1fKOKOg/kC jgz/oE/x9TJGswqmzU75klP1AU2CSY0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MYRAprQA; spf=pass (imf15.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711406027; a=rsa-sha256; cv=none; b=o0WXd1XgRGqCqH8arpbRKkDwQrUS3uaNXk1YMCJqUFjGJdVUOx97VAtcpeYAnoXNVVvaMZ JaWxrinBZjtmh+hpk3GYyYLDejjAEHiQeSx3PeqHAi3jz2Rkhb7NAINQm3Uh2pes4CXo7m Euyt6D07FBkVwBO2OsYdbHGnPTfsO5U= Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-29de4e12d12so3408308a91.3 for ; Mon, 25 Mar 2024 15:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406026; x=1712010826; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BlCoIx7jLqN5Is/HEKy53SFm13f31lm8FF5rHEKGzHA=; b=MYRAprQAtL/RQfaKTB+aCxEO8OAKD07VKFm8g1wOXV/fPAcQBw8azuJvqAh7pYkaUU pBJ6oxtaxQjShCPMJF7S1k9QTRSt3xPzC7LrTmB5pWyT9IHvLRVzSsAO36DhS+1JBwQs Jus/aGIS8BhUXvX1XmMLBPM0yznPCQ+xSPd9S0GOgFFKL5xq1Ka1KAUgCAKHscjSfx4F nRt98+ryTzrvvHXcJ+kK2N5uXajIGWBdl3eCsDqdFJ8L0UjajUAtlco3Jw5gJRtsIcRJ ifB+guCPy+6t68pDKQlFihc1xfPG6ChUByQsZt2tMr3YEeis+w+9ei1zHvfy985NFyGd 8Gig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406026; x=1712010826; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BlCoIx7jLqN5Is/HEKy53SFm13f31lm8FF5rHEKGzHA=; b=vVbU6vYtCsSmlRvGmJPnSXgu/pAmG4BPveqf+pDJ2qHzSajJlF9aeaCf9CR62jpqry 7zt/nh9dhLOK1avMjeE772xVSeoWZtcvH7/BJI98CRIH2g7WvIpHaW6zSLh8RDQpC2KD o+WGZm7GnyIDxPI/OynVTA6e6fx42ezAqGvUQcSHu5b/dpSMia67xPDeV9gD3NAK+rUL GAh4UFG9WjAAnL9Wzbs31T+VcwGv7gC8lfvCfbnGZaaeH24loQAg4KDWQCZFalCik4Zy OHeNlps/yrOjjnUgNciZxodxusv/ruspFJK3+LyQew3GXXW8ijNGMIAdKv59TdAgf2ah Q9dw== X-Gm-Message-State: AOJu0YxvOOIw1COjvlS0CzKFHePMuVXE6ll8s/8OL0ElQoek77pRvaZL U5XyN9ZUVLQ/spNOqEacedI9xAOXw2TNLF1fCw9HC/AEBaJLGkMuBd8q4BOj+eo= X-Google-Smtp-Source: AGHT+IHqkDjxiCBXirJRGAONJPRKYdtlto3CBeWqAPDu5xBIWL2dfiUZrx6HsrjVwyLsjkfUNcQ0aQ== X-Received: by 2002:a17:90b:f08:b0:29b:c2b3:2712 with SMTP id br8-20020a17090b0f0800b0029bc2b32712mr6441165pjb.26.1711406026161; Mon, 25 Mar 2024 15:33:46 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:45 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 4/5] mm: Make pgoff non-const in struct vm_fault Date: Mon, 25 Mar 2024 15:33:38 -0700 Message-ID: <20240325223339.169350-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 74EB7A0013 X-Rspam-User: X-Stat-Signature: 8yw397opzru5bkq46i7qadx3tj7h6wza X-Rspamd-Server: rspam01 X-HE-Tag: 1711406027-129043 X-HE-Meta: U2FsdGVkX1/aEQasnDP1uYWf1yw+LBQALtN/Ax9YDKt+57MMttx1AHvj75JRjZTA7vpawaGI1OepKBj5OEJGXgCstCUruwAC/Iq4FF7oyO0Oebf09qf3DF9rzSSS8ADQzM3Qw0bCRMYin1ggK264QS2GltCAM4iOgSD5njMloAYy3sN04gQ7fpSo5mMQJ1dNiOCWtWrsvTG4VA/bmQAxx4p89OErgj/Cffjqb78o8464row9vwzZI/s/+mGKmyWB13SBvOf7l9SUOiZy6hSmOV1NI8GRkJ9gx6K2HH1W+qOrYbfhjAtNH6R535MSW09gH2/j5FEDU8RHfTpgOLd+YwSjArXedW4DrHmFdqxKHSQqRg9IDzdlnZ9LEQmkBBGK9cm9+N7KdSjVzTlnut9e/pP7taub0X91tlni/xbAs3YCkkQqlPpj5rivbojUxM9/G3edpKRmqPgoYyxQ1R0n1Mhk6eYjUMHL54XP40TYxZ+Ut6iInXPyjR1lnKugFyNiiKqkfWL8YDh/LeS17s5EduloCvGVvIFoo98+MJSbYJXU88hpUeVVIrgQpmRoP4QXEO0Dr2oQh8vM4AFhR4tILgrQTvCLH/G26ESfLtG8yT9iZ0wSuzGZOhTObTZpG9kEe5q2lTYMaeuyqmcoABzYzP2EwrDoP/mRRw85pw6veeWYW2m+/rZcu2ys04nVbxZpijCAAJ/cyC9eE0pJOEu0eNVOyylPVQv/weC6lSaQIUMPVHbQEa7oEZnyx2TquxH+iDOWDSnhJlehFL+/g07pXyKq5n9qhKOC2mIDyJ0pS9PBCi1u5F3AHKwz8R1hiLM6Vxzdph0eoGD4VFGFxF0/Hg8NLp/sSzoShAZt7/fwCMUY+kUQ/BWS2ubz0NDLCZz8i1T1dks7EMeFwXTpNqeZXNkPVsaa+omCJJFOmj3tIAYtq74apcy8VmMsVlFv4Gz/wHar4JaGi+rVFr/lep/ M170mRYf yrY5tvCEVceG+9oW+KXcv6osSAzrW8L7C17COwXqqQYnnrXka/BZ9oY6HY+1VBHssIqmyg5tZcLUmgMOqr+y/LlXp8+xEn6jmSHOCxNl9lG8Qk8EJ4htAph8rr5wedARJy5U8yDMgu1GSD11zCCCBk7+azfd+Iq2lQPvxjXE1DECI9n59WmD7NrcP/PcbywtRsvEa2FoXygDUqj7sx8fGVKe0sRxkEVWv6nt+gTxvJ6JxEVDndw47XMg1434wRs2dBlU+ri5VKF6qe6OVvr8wtZHaZYTAWuGTF37BPZufMkFvU4Bw5wITGWfhXix2pHZAd6qUDWrHmIV9RGgmqdTGz6Pqv0AT/BoJDnKmQhu5umFyyWL50wFEv7MY/YrGDRdMxLBpIUVLFmnYeSuiG7VdAFo6OEoLLmzu1sraFFK3dK/xCAp6PDDUQZvjIBi5IJKNa1E7rd/z18/3jm1Du7e3D2SQ3g9WvAcafwCwwH/xxGsB7jkwgwQaqIVC7PqOa+p5AP4zQ5Aa94HGpalIVPgIoCyvEC5LNv36UNhXs4lWdQg9HbvuV7LGFhIk/+MB/Luiu7l1yDp1os0m5Lq3EC18qoBJtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugetlb calculates addresses and page offsets differently from the rest of mm. In order to pass struct vm_fault through the fault pathway we will let hugetlb_fault() and __handle_mm_fault() set those variables themselves instead. Signed-off-by: Vishal Moola (Oracle) --- include/linux/mm.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f5a97dec5169..c6874aa7b7f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -507,10 +507,11 @@ struct vm_fault { const struct { struct vm_area_struct *vma; /* Target VMA */ gfp_t gfp_mask; /* gfp mask to be used for allocations */ - pgoff_t pgoff; /* Logical page offset based on vma */ - unsigned long address; /* Faulting virtual address - masked */ unsigned long real_address; /* Faulting virtual address - unmasked */ }; + unsigned long address; /* Faulting virtual address - masked */ + pgoff_t pgoff; /* Logical page offset based on vma */ + enum fault_flag flags; /* FAULT_FLAG_xxx flags * XXX: should really be 'const' */ pmd_t *pmd; /* Pointer to pmd entry matching From patchwork Mon Mar 25 22:33:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13603122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41106C54E64 for ; Mon, 25 Mar 2024 22:34:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F59F6B0089; Mon, 25 Mar 2024 18:33:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44EBA6B008A; Mon, 25 Mar 2024 18:33:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27A706B0092; Mon, 25 Mar 2024 18:33:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0C0406B0089 for ; Mon, 25 Mar 2024 18:33:51 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B53F3C0405 for ; Mon, 25 Mar 2024 22:33:50 +0000 (UTC) X-FDA: 81937015020.23.0596A4D Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf14.hostedemail.com (Postfix) with ESMTP id 0B84410000F for ; Mon, 25 Mar 2024 22:33:48 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kIpGyQXV; spf=pass (imf14.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711406029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xa0qOW5rWdjbo+w+QnIgdTmY7x8+eguqE2o56xDAiCo=; b=3fqVBlfEgXTgXdIre4BVqhROGfKo/WQSyxH/IiJKW1wgfwcf7LgQ97BxEgO12z/M8EYBi7 rxxhB5q5vlBjFqvoNkwLOZ65MNZqJkvuQLR+du3qjhfbtsPJGJGqdIuwo2EVy+srXmlQsY AHojlIRSLUjochFC7Rpnkqa/kh+hIXk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kIpGyQXV; spf=pass (imf14.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711406029; a=rsa-sha256; cv=none; b=p0r4AqiqE4YaCMQ8WGU9UQcCZV6CWYWY/NLlfGlEggkOxrhIx1ZfGaIl/JJERoXcRe+geh Ms+Idpv/gw+8JlzNUX3Fkn1a4UuvWmP6oTRK2aFQbNGF+UNYXhYAOYUVn9wUI7wpem69kC /kLrxpy+w/ranioHmPfvqvih3nq26fQ= Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2a072747fc6so797351a91.2 for ; Mon, 25 Mar 2024 15:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711406028; x=1712010828; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xa0qOW5rWdjbo+w+QnIgdTmY7x8+eguqE2o56xDAiCo=; b=kIpGyQXVNfYo87VqM8XLcyz6qKIzy4Gnp+OFbuw6X0tsh6mv7MQuW4mlUwSmiZOz4t 3BPiOCou5FGfVNtM0EmyQ2tyPAD/6Mxu9dZZrjx8j4rHP1wiO+ioSSHQFXD3rpNyoivG Eqbrw6lN+XxQK33jiIiC2RthRiAvj3wlq5w6Gin8MgtWrDOTIdKg8FL+7ZIOBxHo8mYg hS7fR99Cv/Kmzy0EAvyISDDM/tQhWeHgh39mGgWREPYS+ngp4b8PhWKwlOreSB/jS+mm 1zofp+4xyNkgqrvZIRtUhlAf2FrMTSXD7qyvtFHTnyqFE4rsLel/bz3zdWR5XFRiEFql lePw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711406028; x=1712010828; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xa0qOW5rWdjbo+w+QnIgdTmY7x8+eguqE2o56xDAiCo=; b=s8WBzP6+wsmYXVKgrAY4uilODr7AIzcQ7qC9LSTjsNHZN7yiKPkZVdXwRANwalesFJ d0ZXwyFHQoqCalBVoNLz1bg0nL2PEAMqAh5oiYFo12vpPiIWHhlLj5HfLOYq8heyGYC+ mzbNFlhFb1VQbp2yJRBxS/zteS1PwhdD/PHi90D7rMtGfv3p/l04tPmes2W1kNfz3fre iZ78F6fHFCJFBMXTUFoDS9hAGhn4XcadY4TaMh/nRN1cXgtNZ+e3XEOllXF2ZNXYEC66 EM4kG+1aEaH8xLpK/yFKm12H4L7tQJsyjWXH3UC1n0+0yn3ElaCQjhEW3kjAxPEA2btf w3nQ== X-Gm-Message-State: AOJu0YxKtw3vHx+lA8HxevKAjSqFTCxWBj76ia+8XfXK/SF3Ebpuerbu wiaFFchZ81Z9Id/5sGqZiB/UDBEa+Nd0PXk3AC9b+Bk1ZctaUc/+ilZFN+Pbwfs= X-Google-Smtp-Source: AGHT+IFRm0SsVYiVz6wu55o5NNu1OirYSh8Wnv/Y+CNLuAFTeLpCWCaL50rUVsBgY/cv4dzw/fjRUw== X-Received: by 2002:a17:90a:16c2:b0:29a:3c70:1525 with SMTP id y2-20020a17090a16c200b0029a3c701525mr5308309pje.45.1711406027625; Mon, 25 Mar 2024 15:33:47 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id sx16-20020a17090b2cd000b002a053cdd4e5sm4356173pjb.9.2024.03.25.15.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Mar 2024 15:33:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, "Vishal Moola (Oracle)" Subject: [PATCH 5/5] memory: Define struct vm_fault in handle_mm_fault() Date: Mon, 25 Mar 2024 15:33:39 -0700 Message-ID: <20240325223339.169350-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325223339.169350-1-vishal.moola@gmail.com> References: <20240325223339.169350-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B84410000F X-Rspam-User: X-Stat-Signature: koj6kkirhfrqek3kaj8sczji7ck34zno X-Rspamd-Server: rspam01 X-HE-Tag: 1711406028-423719 X-HE-Meta: U2FsdGVkX18IsIr1UitiMpsZFQFSdOcPl6z0KFFNLvr9utbM20Xib6CdqwAZJdPPi+qwkr0wJHf+oZX8bl6QNZAHTu/xLuDeXLojwr1tj7Il/jQJx605cv/v82rV8ZSoGYkaKCqeq4/PMvfRWSkXV095R8cb6Db0Xa1+1kFZUcA5alHsR/JOcgQV55kBAuXagyfE1NVHFCM6cEg6X1NLKUdfVKRPQs6Lp/NafmqOoD5p9y4FGnPT/rp9nyB8u987B+T00lpYksAn9WiPSRsCelm/Jw1Gq0pkiBBnGgwBXMWmcPMUCNeIbooB7mRu4B+jtjZspWAARpe8noEAsfo5VgVeyLzFKsr37W9jW8HNjO3CDyDmmLD9ctLv4XRkq6zVvvAYMa6ruVqxlA4A0tFlPIzRJNuK72YKZSc4mAvmvCx8FgTlPuqud+WZukLMK2ujjsO9g+2GfBJCtbBHwOmoKfFxuHclmvgnGoq55e1QrGkcOOLYwJ7HFhDyPx2Ni5pvZU9YWWHenhO5P3i2oDGtyse3iJ+YpdgCCUDyIxJg4IFZrE5RAKcUTK13qXUo9sQcpT5lda2xNmwOgxWeLr1XmqpCN6h0GjQC5EakL0jqQCKAsJ+dd8ioMNgLBFovcjtVwtSxjP+c4cpAvgCDKNawRcqH/rvzcD+CTJsTWw9ToGXy1mXf28dt9ZQ+COjSoINTBlHsNHR1v63RXY/z5pXizKUSW1UctoFpRutOzK9jUT+RRxcaGjd5g/pvXb5F/W7oPLR6Zio9aYSldjkn1IFdOPpqGRQVQ8+XzbTrEsYPy97CqmO9nu9WTuMZ3y+Ru/w3rRiVYAUosOid7e3/MonBfEyh5N0WRLCkoQhq6bqnNzFX+0+P22VzBws/I6zFsTv2WcISL/At6jMYSfgxRi0wVGDUYc7zhZFDyN67xmT9Lb+Krb+rBhYMxCM6K+5DVepIPl06N8wg5clSIJYaOmp NE0QZ0v1 vpDwnsGTKJCnyf40kccjBBBKMMsI7fU40xMNFR6TvC8nYIFWx6Sy8jLlf2spktFGTKncEs3cXMhoBaIy3C9aCnX2gEewfpy/jT3DHCWm0dUuIbUOjB6u2SN/Ys9YZ4pX1yItO7GrlkAIhxLgdL8C9ABeMzvSfUASnp4Gtgs85QzleMperk1MbLyKlfmufmE7tqd3+B/rof51lRnSc5BQZGw5HYTOLJIw12FRq9irfVms0N4dbI4yR7kBmwzN6ax4vUXdcIHhNlOguQHI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Define struct vm_fault in handle_mm_fault() to be passed throughout the rest of the fault pathway. Pass it through to hugetlb_fault() and __handle_mm_fault(), making any necessary trivial changes. Signed-off-by: Vishal Moola (Oracle) --- include/linux/hugetlb.h | 7 +-- mm/hugetlb.c | 106 +++++++++++++++++++--------------------- mm/memory.c | 87 +++++++++++++++++---------------- 3 files changed, 98 insertions(+), 102 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c1ee640d87b1..0e0a93b4d9fc 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -147,8 +147,7 @@ void hugetlb_report_meminfo(struct seq_file *); int hugetlb_report_node_meminfo(char *buf, int len, int nid); void hugetlb_show_meminfo_node(int nid); unsigned long hugetlb_total_pages(void); -vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, unsigned int flags); +vm_fault_t hugetlb_fault(struct vm_fault *vmf); #ifdef CONFIG_USERFAULTFD int hugetlb_mfill_atomic_pte(pte_t *dst_pte, struct vm_area_struct *dst_vma, @@ -482,9 +481,7 @@ static inline void __unmap_hugepage_range(struct mmu_gather *tlb, BUG(); } -static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long address, - unsigned int flags) +static inline vm_fault_t hugetlb_fault(struct vm_fault *vmf) { BUG(); return 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 107b47329b9f..7ecc680f4681 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6327,30 +6327,24 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx) } #endif -vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +vm_fault_t hugetlb_fault(struct vm_fault *vmf) { vm_fault_t ret; u32 hash; + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; struct folio *folio = NULL; struct folio *pagecache_folio = NULL; struct hstate *h = hstate_vma(vma); struct address_space *mapping; int need_wait_lock = 0; - struct vm_fault vmf = { - .vma = vma, - .address = address & huge_page_mask(h), - .real_address = address, - .flags = flags, - .pgoff = vma_hugecache_offset(h, vma, - address & huge_page_mask(h)), - /* TODO: Track hugetlb faults using vm_fault */ - - /* - * Some fields may not be initialized, be careful as it may - * be hard to debug if called functions make assumptions - */ - }; + /* + * Some fields of vmf may not be initialized, be careful as it may + * be hard to debug if called functions make assumptions + */ + vmf->address = vmf->real_address & huge_page_mask(h); + vmf->pgoff = vma_hugecache_offset(h, vma, + vmf->address & huge_page_mask(h)); /* * Serialize hugepage allocation and instantiation, so that we don't @@ -6358,27 +6352,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * the same page in the page cache. */ mapping = vma->vm_file->f_mapping; - hash = hugetlb_fault_mutex_hash(mapping, vmf.pgoff); + hash = hugetlb_fault_mutex_hash(mapping, vmf->pgoff); mutex_lock(&hugetlb_fault_mutex_table[hash]); /* * Acquire vma lock before calling huge_pte_alloc and hold - * until finished with vmf.pte. This prevents huge_pmd_unshare from - * being called elsewhere and making the vmf.pte no longer valid. + * until finished with vmf->pte. This prevents huge_pmd_unshare from + * being called elsewhere and making the vmf->pte no longer valid. */ hugetlb_vma_lock_read(vma); - vmf.pte = huge_pte_alloc(mm, vma, vmf.address, huge_page_size(h)); - if (!vmf.pte) { + vmf->pte = huge_pte_alloc(mm, vma, vmf->address, huge_page_size(h)); + if (!vmf->pte) { hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return VM_FAULT_OOM; } - vmf.orig_pte = huge_ptep_get(vmf.pte); - if (huge_pte_none_mostly(vmf.orig_pte)) { - if (is_pte_marker(vmf.orig_pte)) { + vmf->orig_pte = huge_ptep_get(vmf->pte); + if (huge_pte_none_mostly(vmf->orig_pte)) { + if (is_pte_marker(vmf->orig_pte)) { pte_marker marker = - pte_marker_get(pte_to_swp_entry(vmf.orig_pte)); + pte_marker_get(pte_to_swp_entry(vmf->orig_pte)); if (marker & PTE_MARKER_POISONED) { ret = VM_FAULT_HWPOISON_LARGE; @@ -6392,20 +6386,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ - return hugetlb_no_page(mm, vma, mapping, &vmf); + return hugetlb_no_page(mm, vma, mapping, vmf); } ret = 0; /* - * vmf.orig_pte could be a migration/hwpoison vmf.orig_pte at this + * vmf->orig_pte could be a migration/hwpoison vmf->orig_pte at this * point, so this check prevents the kernel from going below assuming * that we have an active hugepage in pagecache. This goto expects * the 2nd page fault, and is_hugetlb_entry_(migration|hwpoisoned) * check will properly handle it. */ - if (!pte_present(vmf.orig_pte)) { - if (unlikely(is_hugetlb_entry_migration(vmf.orig_pte))) { + if (!pte_present(vmf->orig_pte)) { + if (unlikely(is_hugetlb_entry_migration(vmf->orig_pte))) { /* * Release the hugetlb fault lock now, but retain * the vma lock, because it is needed to guard the @@ -6414,9 +6408,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, vmf.pte); + migration_entry_wait_huge(vma, vmf->pte); return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) + } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf->orig_pte))) ret = VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; @@ -6429,53 +6423,53 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * spinlock. Also lookup the pagecache page now as it is used to * determine if a reservation has been consumed. */ - if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf.orig_pte)) { - if (vma_needs_reservation(h, vma, vmf.address) < 0) { + if ((vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && + !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf->orig_pte)) { + if (vma_needs_reservation(h, vma, vmf->address) < 0) { ret = VM_FAULT_OOM; goto out_mutex; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, vmf.address); + vma_end_reservation(h, vma, vmf->address); pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, - vmf.pgoff); + vmf->pgoff); if (IS_ERR(pagecache_folio)) pagecache_folio = NULL; } - vmf.ptl = huge_pte_lock(h, mm, vmf.pte); + vmf->ptl = huge_pte_lock(h, mm, vmf->pte); /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) + if (unlikely(!pte_same(vmf->orig_pte, huge_ptep_get(vmf->pte)))) goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && - (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf->pte)) && + (vmf->flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf->orig_pte)) { if (!userfaultfd_wp_async(vma)) { - spin_unlock(vmf.ptl); + spin_unlock(vmf->ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); folio_put(pagecache_folio); } hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - return handle_userfault(&vmf, VM_UFFD_WP); + return handle_userfault(vmf, VM_UFFD_WP); } - vmf.orig_pte = huge_pte_clear_uffd_wp(vmf.orig_pte); - set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, + vmf->orig_pte = huge_pte_clear_uffd_wp(vmf->orig_pte); + set_huge_pte_at(mm, vmf->address, vmf->pte, vmf->orig_pte, huge_page_size(hstate_vma(vma))); /* Fallthrough to CoW */ } /* - * hugetlb_wp() requires page locks of pte_page(vmf.orig_pte) and + * hugetlb_wp() requires page locks of pte_page(vmf->orig_pte) and * pagecache_folio, so here we need take the former one * when folio != pagecache_folio or !pagecache_folio. */ - folio = page_folio(pte_page(vmf.orig_pte)); + folio = page_folio(pte_page(vmf->orig_pte)); if (folio != pagecache_folio) if (!folio_trylock(folio)) { need_wait_lock = 1; @@ -6484,24 +6478,24 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, folio_get(folio); - if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { - if (!huge_pte_write(vmf.orig_pte)) { - ret = hugetlb_wp(mm, vma, pagecache_folio, &vmf); + if (vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { + if (!huge_pte_write(vmf->orig_pte)) { + ret = hugetlb_wp(mm, vma, pagecache_folio, vmf); goto out_put_page; - } else if (likely(flags & FAULT_FLAG_WRITE)) { - vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte); + } else if (likely(vmf->flags & FAULT_FLAG_WRITE)) { + vmf->orig_pte = huge_pte_mkdirty(vmf->orig_pte); } } - vmf.orig_pte = pte_mkyoung(vmf.orig_pte); - if (huge_ptep_set_access_flags(vma, vmf.address, vmf.pte, vmf.orig_pte, - flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, vmf.address, vmf.pte); + vmf->orig_pte = pte_mkyoung(vmf->orig_pte); + if (huge_ptep_set_access_flags(vma, vmf->address, vmf->pte, + vmf->orig_pte, vmf->flags & FAULT_FLAG_WRITE)) + update_mmu_cache(vma, vmf->address, vmf->pte); out_put_page: if (folio != pagecache_folio) folio_unlock(folio); folio_put(folio); out_ptl: - spin_unlock(vmf.ptl); + spin_unlock(vmf->ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); diff --git a/mm/memory.c b/mm/memory.c index c93b058adfb2..a2fcb0322b11 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5191,39 +5191,35 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * the result, the mmap_lock is not held on exit. See filemap_fault() * and __folio_lock_or_retry(). */ -static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static vm_fault_t __handle_mm_fault(struct vm_fault *vmf) { - struct vm_fault vmf = { - .vma = vma, - .address = address & PAGE_MASK, - .real_address = address, - .flags = flags, - .pgoff = linear_page_index(vma, address), - .gfp_mask = __get_fault_gfp_mask(vma), - }; + struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; unsigned long vm_flags = vma->vm_flags; + const unsigned long address = vmf->real_address; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; + vmf->address = address & PAGE_MASK; + vmf->pgoff = linear_page_index(vma, address); pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) return VM_FAULT_OOM; - vmf.pud = pud_alloc(mm, p4d, address); - if (!vmf.pud) + vmf->pud = pud_alloc(mm, p4d, address); + if (!vmf->pud) return VM_FAULT_OOM; retry_pud: - if (pud_none(*vmf.pud) && - thp_vma_allowable_order(vma, vm_flags, false, true, true, PUD_ORDER)) { - ret = create_huge_pud(&vmf); + if (pud_none(*vmf->pud) && + thp_vma_allowable_order(vma, vm_flags, false, true, + true, PUD_ORDER)) { + ret = create_huge_pud(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pud_t orig_pud = *vmf.pud; + pud_t orig_pud = *vmf->pud; barrier(); if (pud_trans_huge(orig_pud) || pud_devmap(orig_pud)) { @@ -5232,57 +5228,60 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, * TODO once we support anonymous PUDs: NUMA case and * FAULT_FLAG_UNSHARE handling. */ - if ((flags & FAULT_FLAG_WRITE) && !pud_write(orig_pud)) { - ret = wp_huge_pud(&vmf, orig_pud); + if ((vmf->flags & FAULT_FLAG_WRITE) && + !pud_write(orig_pud)) { + ret = wp_huge_pud(vmf, orig_pud); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pud_set_accessed(&vmf, orig_pud); + huge_pud_set_accessed(vmf, orig_pud); return 0; } } } - vmf.pmd = pmd_alloc(mm, vmf.pud, address); - if (!vmf.pmd) + vmf->pmd = pmd_alloc(mm, vmf->pud, address); + if (!vmf->pmd) return VM_FAULT_OOM; /* Huge pud page fault raced with pmd_alloc? */ - if (pud_trans_unstable(vmf.pud)) + if (pud_trans_unstable(vmf->pud)) goto retry_pud; - if (pmd_none(*vmf.pmd) && - thp_vma_allowable_order(vma, vm_flags, false, true, true, PMD_ORDER)) { - ret = create_huge_pmd(&vmf); + if (pmd_none(*vmf->pmd) && + thp_vma_allowable_order(vma, vm_flags, false, true, + true, PMD_ORDER)) { + ret = create_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - vmf.orig_pmd = pmdp_get_lockless(vmf.pmd); + vmf->orig_pmd = pmdp_get_lockless(vmf->pmd); - if (unlikely(is_swap_pmd(vmf.orig_pmd))) { + if (unlikely(is_swap_pmd(vmf->orig_pmd))) { VM_BUG_ON(thp_migration_supported() && - !is_pmd_migration_entry(vmf.orig_pmd)); - if (is_pmd_migration_entry(vmf.orig_pmd)) - pmd_migration_entry_wait(mm, vmf.pmd); + !is_pmd_migration_entry(vmf->orig_pmd)); + if (is_pmd_migration_entry(vmf->orig_pmd)) + pmd_migration_entry_wait(mm, vmf->pmd); return 0; } - if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { - if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) - return do_huge_pmd_numa_page(&vmf); - - if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !pmd_write(vmf.orig_pmd)) { - ret = wp_huge_pmd(&vmf); + if (pmd_trans_huge(vmf->orig_pmd) || + pmd_devmap(vmf->orig_pmd)) { + if (pmd_protnone(vmf->orig_pmd) && vma_is_accessible(vma)) + return do_huge_pmd_numa_page(vmf); + + if ((vmf->flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) + && !pmd_write(vmf->orig_pmd)) { + ret = wp_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pmd_set_accessed(&vmf); + huge_pmd_set_accessed(vmf); return 0; } } } - return handle_pte_fault(&vmf); + return handle_pte_fault(vmf); } /** @@ -5421,6 +5420,12 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, /* If the fault handler drops the mmap_lock, vma may be freed */ struct mm_struct *mm = vma->vm_mm; vm_fault_t ret; + struct vm_fault vmf = { + .vma = vma, + .real_address = address, + .flags = flags, + .gfp_mask = __get_fault_gfp_mask(vma), + }; __set_current_state(TASK_RUNNING); @@ -5445,9 +5450,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, lru_gen_enter_fault(vma); if (unlikely(is_vm_hugetlb_page(vma))) - ret = hugetlb_fault(vma->vm_mm, vma, address, flags); + ret = hugetlb_fault(&vmf); else - ret = __handle_mm_fault(vma, address, flags); + ret = __handle_mm_fault(&vmf); lru_gen_exit_fault();