From patchwork Mon Apr 1 20:26:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13613000 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C358DCD1288 for ; Mon, 1 Apr 2024 20:27:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADC8A6B0088; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EEEE6B008A; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8668A6B008C; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6B00B6B0088 for ; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3893B1A01F4 for ; Mon, 1 Apr 2024 20:27:01 +0000 (UTC) X-FDA: 81962097042.16.665C1DD Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by imf30.hostedemail.com (Postfix) with ESMTP id 4C1F18001C for ; Mon, 1 Apr 2024 20:26:59 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aOFVR9oL; spf=pass (imf30.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712003219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLJjpIJjwOYqxNhCjRA5yTdkpLaMKAJot06YpeH+/j4=; b=G6D00P4GqATH6VjX9UKkyE9+T/buIKercqVfsDU8JuMjw+nuDSW8MYt/f+4q+8P1hGdKQO nyEqDIjUjVD/C4e7ueMudB7EBk6zpB0KKUOAlkAVHxIohow9E/voeCYRD6xrA9GGeIS08O 6onWsGRSi1Ukki7ratrrbP83M9JdR4o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712003219; a=rsa-sha256; cv=none; b=BoNB+rRuEN2vdBLWVGUNdz0hTz8doILMfAztjYwFUk1gPxVNUkpY4XswifQ8SRd6NBrGBI ABEoyy1XKGOOytvaUfIjiwlmg+eKqYThcQEipko8jkr1Vs1OVERz/r2Cq0ovxnGLP9nU4A TeFK4ngvb6/BN4dy/fxZQtvB/Ce1mXA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aOFVR9oL; spf=pass (imf30.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-6151d2489b4so816367b3.0 for ; Mon, 01 Apr 2024 13:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712003218; x=1712608018; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OLJjpIJjwOYqxNhCjRA5yTdkpLaMKAJot06YpeH+/j4=; b=aOFVR9oL1UGkoQUuDnuuwan4eVTOC+r5fzDhhgAi65jnPUISBbbejkjfcQ2pDpYRfI YpNrYjkbLzNl0rGPw7KVVRzpRZIWxYPVR8rzXTRyRkKCS/gAzYGkGKW/y3O+VBK5mw0X sMEVVY607r0wMF/SoCltwO39aV6cKvLBeRcVfrrZEFUpxWoziawWPLafGKCn9RZUcgxq 4swmZPUFfOs0o5Duk8DarUcKXBzfv0tar3mdKAFTkVNoWV0Hq6MGVQJ5u5e8s5jlQL3c YoLcvSmrA88pdyOtIN4k/10otLf9k3kMdXaRVnPA1aDq/9Cos6QxeU7ai1f+iN4EWDAl VRyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712003218; x=1712608018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OLJjpIJjwOYqxNhCjRA5yTdkpLaMKAJot06YpeH+/j4=; b=QfzVEg2clOhyYkuLSrs16r7yHBYAryAdBuRGlrLkWkQxM4SBALzCINldx8MWZH7Pkk YjCG8qvmgfP1/kD9N33ZV0eG5hLG204jFJQTwDyTTbQpvFS4PipOMnf7Jix8L+Kw4OQW NHYHFVlgWNdPky3KAZlzVykcXCOM0qy3MoBWq3lT70vlIp99e0NIdgcyVCcZobuXlV+W Ac8UIGc+zDmI2zXYbdVb4YuRd5HYMoRGSRIZV2jMWK5tvv5r7rZwVmzJ3kE3nAzdx9x1 z0ViHBsgAxZYybpMdWNfigKz2xNG6DIEURCtfvcvB1IawPmOapbIk2X/r7xT9ObjJC3j VILw== X-Gm-Message-State: AOJu0YzbccteaHkFK/EeuXJt3XpDDSpd9MAs9jwClVZnQkJqu2CFcFJx E5vrCyMD+SyzbRP5rMK/OJMECt3MD9QDMGzX2OpX9fBS8qcSlEOWYKaFqO0X X-Google-Smtp-Source: AGHT+IFFZ8j7M33dVd9jUeXLmj/uGlOxSzCXNksanWE/EfYlyKZd130C69mAXSmbpVhDWvYUfcCEdg== X-Received: by 2002:a0d:e211:0:b0:60a:243:547c with SMTP id l17-20020a0de211000000b0060a0243547cmr9827366ywe.44.1712003216926; Mon, 01 Apr 2024 13:26:56 -0700 (PDT) Received: from fedora.attlocal.net ([2600:1700:2f7d:1800::23]) by smtp.googlemail.com with ESMTPSA id y72-20020a81a14b000000b006142210a31esm1171181ywg.23.2024.04.01.13.26.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Apr 2024 13:26:56 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, willy@infradead.org, "Vishal Moola (Oracle)" Subject: [PATCH v2 1/3] hugetlb: Convert hugetlb_fault() to use struct vm_fault Date: Mon, 1 Apr 2024 13:26:49 -0700 Message-ID: <20240401202651.31440-2-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240401202651.31440-1-vishal.moola@gmail.com> References: <20240401202651.31440-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4C1F18001C X-Rspam-User: X-Stat-Signature: nkeanskbzumx33ogd1pw86t8qf8bcxph X-Rspamd-Server: rspam03 X-HE-Tag: 1712003219-963926 X-HE-Meta: U2FsdGVkX19NyEJUefSyoehiNXAc8fcXZTFz0pLvGI29oO8yz2LMC3itAkqlOI4i7H2I6WRLWDelciCjDrl/XNwpa9zaHWeqEBF5SCqkS1TQw1fMm9ue+knvC6lnsjCU2DdrUc0zLju+VZRtV+I/W6Kd1afX1SNaBjv57lZJ73LtZZql3GF1cCy+N3vIu05MEmXJsjgloMc0S5M/47oAXoeQydH7DXoDWvqhnNWFN8zWCrnKbPpwz6LZFChJNnVzDpmawJYmhbFdnfAT+uL3BxWystY8P19XmayeUSG/+skroc1fzkfIEQIGS3E0tjjPDY9ss8oIQRord6KYwHvbZ4qExblomspBXTAN18NL0Y2QOkqulwuX6zaf90tC9ypdl9lfEkoqxNmRERrPekJwcEwEtHHgIebDychomROSZeiM3t8uMLSPBhdceHe4xLksqEHffcQvCTC7N53plnAQOfPmhe3sDodJ9lqBlDULZj7Pn4UvicjyWaHh2kZA9eTe7VZ4wqHUJgdoYj6FhBz7dui4soB9IGBdhWKLt/wqWItmWVTiGvpbYyOG686o0u54y/9tZUGNKusrpHsJsVkaU2t7kxvcNNTrAMIqgleST3hqprkamBm13FQyBMqqrQlfixuANheybQagznC9QskhwfG5wkRhkiDSf1M2ERK1d6CQmms+0+LoT7NmXTs9Jm77u8PZYAhEa6/a3hWnK8sif/limZcHSkXbe1IMh0LapwaWqc6cy8H+ErCvhx4ypIcJq1VUpx+Et0HgZHqVjdhbx8nxYzH6heZqkHm0nC/oID7plUkixkU3ITgEQZYibrvyWVU6WW76w1YL+tpiMpA5YcoA+elEnshaatoDRo5j2dthjmEUoU2xNEA+b/HHIARgtpFMAfzpWZqRGPApkHf7xR2fbOo+BcSjUbQR6ayC8lUf6i6wkahtuFICmCncXNqwyYGseZ50G+r4ztWJnhQ 2nwDs7Hw RilUU2vKYUlpDCfKVKAN0YwAQWImEJjbDaF4A8Ot0Hle/I6SHfRXdzr5zWzJV3wDnuANp5NqAjohD0gFp08SpS4akMcah7j78gO6kOumv/yqGMVZbr3F8Tud2XfwKFqeCQ2zzSGyR520Arq8KU376kqLtXpDhg/vBVIYieFeJPwC5kSrjU2dbli+gwn0l8OgsmDhtpIT19vIPTQ+uyMsh+68lq0caa98raaOmaYw/q6C1pX5cUpOhDa+l8Khy3tyOKPu/gINS9TfUzMhpSnfSl86MsVvoGGn5vyyzV1raLRgr7qKHPlpt89+Z6SdX8sleXnRvMCCA40gsXrU7sFjVqsYVuRWUiRBe494qmucmVMG/+3d3PyUHE7X3CLHaD3BRbk2QrUwUx1j7oeuQ4PU3+bzWEGncOUVRDLlXBM8He48QB9ZbURnBlCGD9s2VGSEru6riUDdGvj5YtUoBKPO84mMAo3npQyk6x08GckeaSteFXMRlEZoj+ZIBk7NyfDzZ9EMjhMDhGLRkRw+8dxobllYX2LsMMnTLb922vaSVHyo1T2RKTKpxybqw6ac/oi1yNeLT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that hugetlb_fault() has a vm_fault available for fault tracking, use it throughout. This cleans up the code by removing 2 variables, and prepares hugetlb_fault() to take in a struct vm_fault argument. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song --- mm/hugetlb.c | 84 +++++++++++++++++++++++++--------------------------- 1 file changed, 41 insertions(+), 43 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8267e221ca5d..360b82374a89 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6423,8 +6423,6 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx) vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags) { - pte_t *ptep, entry; - spinlock_t *ptl; vm_fault_t ret; u32 hash; struct folio *folio = NULL; @@ -6432,13 +6430,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, struct hstate *h = hstate_vma(vma); struct address_space *mapping; int need_wait_lock = 0; - unsigned long haddr = address & huge_page_mask(h); struct vm_fault vmf = { .vma = vma, - .address = haddr, + .address = address & huge_page_mask(h), .real_address = address, .flags = flags, - .pgoff = vma_hugecache_offset(h, vma, haddr), + .pgoff = vma_hugecache_offset(h, vma, + address & huge_page_mask(h)), /* TODO: Track hugetlb faults using vm_fault */ /* @@ -6458,22 +6456,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire vma lock before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with vmf.pte. This prevents huge_pmd_unshare from + * being called elsewhere and making the vmf.pte no longer valid. */ hugetlb_vma_lock_read(vma); - ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h)); - if (!ptep) { + vmf.pte = huge_pte_alloc(mm, vma, vmf.address, huge_page_size(h)); + if (!vmf.pte) { hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); return VM_FAULT_OOM; } - entry = huge_ptep_get(ptep); - if (huge_pte_none_mostly(entry)) { - if (is_pte_marker(entry)) { + vmf.orig_pte = huge_ptep_get(vmf.pte); + if (huge_pte_none_mostly(vmf.orig_pte)) { + if (is_pte_marker(vmf.orig_pte)) { pte_marker marker = - pte_marker_get(pte_to_swp_entry(entry)); + pte_marker_get(pte_to_swp_entry(vmf.orig_pte)); if (marker & PTE_MARKER_POISONED) { ret = VM_FAULT_HWPOISON_LARGE; @@ -6488,20 +6486,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * mutex internally, which make us return immediately. */ return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - ptep, entry, flags, &vmf); + vmf.pte, vmf.orig_pte, flags, &vmf); } ret = 0; /* - * entry could be a migration/hwpoison entry at this point, so this - * check prevents the kernel from going below assuming that we have - * an active hugepage in pagecache. This goto expects the 2nd page - * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will - * properly handle it. + * vmf.orig_pte could be a migration/hwpoison vmf.orig_pte at this + * point, so this check prevents the kernel from going below assuming + * that we have an active hugepage in pagecache. This goto expects + * the 2nd page fault, and is_hugetlb_entry_(migration|hwpoisoned) + * check will properly handle it. */ - if (!pte_present(entry)) { - if (unlikely(is_hugetlb_entry_migration(entry))) { + if (!pte_present(vmf.orig_pte)) { + if (unlikely(is_hugetlb_entry_migration(vmf.orig_pte))) { /* * Release the hugetlb fault lock now, but retain * the vma lock, because it is needed to guard the @@ -6510,9 +6508,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, ptep); + migration_entry_wait_huge(vma, vmf.pte); return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) ret = VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; @@ -6526,13 +6524,13 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * determine if a reservation has been consumed. */ if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) && - !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(entry)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(vmf.orig_pte)) { + if (vma_needs_reservation(h, vma, vmf.address) < 0) { ret = VM_FAULT_OOM; goto out_mutex; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf.address); pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, vmf.pgoff); @@ -6540,17 +6538,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, pagecache_folio = NULL; } - ptl = huge_pte_lock(h, mm, ptep); + vmf.ptl = huge_pte_lock(h, mm, vmf.pte); /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) + if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && - (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && + (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { if (!userfaultfd_wp_async(vma)) { - spin_unlock(ptl); + spin_unlock(vmf.ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); folio_put(pagecache_folio); @@ -6560,18 +6558,18 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, return handle_userfault(&vmf, VM_UFFD_WP); } - entry = huge_pte_clear_uffd_wp(entry); - set_huge_pte_at(mm, haddr, ptep, entry, + vmf.orig_pte = huge_pte_clear_uffd_wp(vmf.orig_pte); + set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte, huge_page_size(hstate_vma(vma))); /* Fallthrough to CoW */ } /* - * hugetlb_wp() requires page locks of pte_page(entry) and + * hugetlb_wp() requires page locks of pte_page(vmf.orig_pte) and * pagecache_folio, so here we need take the former one * when folio != pagecache_folio or !pagecache_folio. */ - folio = page_folio(pte_page(entry)); + folio = page_folio(pte_page(vmf.orig_pte)); if (folio != pagecache_folio) if (!folio_trylock(folio)) { need_wait_lock = 1; @@ -6581,24 +6579,24 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, folio_get(folio); if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { - if (!huge_pte_write(entry)) { - ret = hugetlb_wp(mm, vma, address, ptep, flags, - pagecache_folio, ptl, &vmf); + if (!huge_pte_write(vmf.orig_pte)) { + ret = hugetlb_wp(mm, vma, address, vmf.pte, flags, + pagecache_folio, vmf.ptl, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { - entry = huge_pte_mkdirty(entry); + vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte); } } - entry = pte_mkyoung(entry); - if (huge_ptep_set_access_flags(vma, haddr, ptep, entry, + vmf.orig_pte = pte_mkyoung(vmf.orig_pte); + if (huge_ptep_set_access_flags(vma, vmf.address, vmf.pte, vmf.orig_pte, flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, haddr, ptep); + update_mmu_cache(vma, vmf.address, vmf.pte); out_put_page: if (folio != pagecache_folio) folio_unlock(folio); folio_put(folio); out_ptl: - spin_unlock(ptl); + spin_unlock(vmf.ptl); if (pagecache_folio) { folio_unlock(pagecache_folio); From patchwork Mon Apr 1 20:26:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13612999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D55ACD128A for ; Mon, 1 Apr 2024 20:27:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EB056B0085; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24CB46B0088; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C7B46B008A; Mon, 1 Apr 2024 16:27:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E6BBC6B0085 for ; Mon, 1 Apr 2024 16:27:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A31CA1C0867 for ; Mon, 1 Apr 2024 20:27:00 +0000 (UTC) X-FDA: 81962097000.06.9B2DD23 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by imf25.hostedemail.com (Postfix) with ESMTP id DAB5BA000B for ; Mon, 1 Apr 2024 20:26:58 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nduwGQq4; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712003218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WwRQK61zxAOtZTuzQkMVrkDP5lES+e4npSppnKc4AGI=; b=f3cB3gFyUj6Qz3numeck+aBaFG6V5qKztEtvnPFZ8B6dfbZXh9b/5AOXLCtUMb2Rm6C6Vc r/l3gSDNjZ1bFOdvgIf/ClITYtN8cp05cv9VbFkKDWn3j8cBTB1nyPtmcKK6uxJjXzLp/2 N3Wm6s8GeaESE77YoUSSY9+3cKhuuD4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712003218; a=rsa-sha256; cv=none; b=dMwpat6gGRX55sNJvBwocfP1w6M+rrB2amXcTgljwlXzvanCh98TR6EzmlNcM3ul7Glj1H 5H4BAfqRh7GPuf0QqkCCdDxHyWeZtew+qJvDEr+5UGES7qLdGPQJG3wlzKcxXSsjepuLCC ZBqBfoMRHMCwkCdT8W4ZrWqtS19JB6U= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nduwGQq4; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-60a3c48e70fso43810747b3.1 for ; Mon, 01 Apr 2024 13:26:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712003218; x=1712608018; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WwRQK61zxAOtZTuzQkMVrkDP5lES+e4npSppnKc4AGI=; b=nduwGQq4riahKCuvefKEowOr37Ie5H8KFnzLX2mfx4ZZb/Fooo/NqiE5cqXztSOJNf 49+frWqlTrEjbCo256qUiri2QMAD4GwMRrTv5J0ac0lZI4S4M5/CfUBxduF/hkgzAkkO NnXKGyF1u44T1Ie0Hu/hQDfq3Hhcri5SSaN4ZaCQvFM20AEfqf4IpgO6TTnN8IGyLRP6 K4o0lxdk1E+vff+rrhkYC+iSiGWgQ8e0vLzSEgCKNuZoEjRSZlqO/vWOxe3hJOQo9Vko D9bN9D9HbQwbwcsclSbsxIt7CKZibAtTYyw1C+s9Xevms2MViVYDzLUQ/rNW6JSlsqoy zRwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712003218; x=1712608018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WwRQK61zxAOtZTuzQkMVrkDP5lES+e4npSppnKc4AGI=; b=VRze8sGe/PVWkE6U3TYzETDSHCmAs4u3N8EJUTrjmaEh9lNHqa3pE6/TCc2rhZvLO/ vrovZSqKT49XLhvHJVGhy/NR7S4AOxqBmH2xyB+5taKA2Q8/N9GUjtdCuah+xHuztNnl J14CuCKGsxhlqzVHr8ZS6+9T6FVqlQTmPpuCK9QLcfgcLAfD9op7JWeP0WhTxUTU2Cth 0gF8Gmc6w+8UQ0KfIYw84HYV590VZLiS2/VIuE/88pLkv7SeCs4liuFH8ub7wSSqJVM6 JPkDrdE7qwRmWk0xnC5M5dUC1+HL8Zn2bDozcYvjWPVm4Wxx0aa8NwCcP0RB8VJsQ9ji jBKA== X-Gm-Message-State: AOJu0Yx5qYG6GCPmn875Uq4bum9qzXyTKUjYU1x5FHCwtapzjnKjN+/R OABsGbstyD6jZdtX8P2IIndHricLx6QJ/64hInPecSpI7o6f2M1rdiLlBmXh X-Google-Smtp-Source: AGHT+IFisZq4EuxDLU4uS1EpjFvcKMqEqYfO+yamypsLgW77pMy3m4jr7Iv/oNYRYg9OIrA5cUAWAg== X-Received: by 2002:a0d:e7c6:0:b0:611:30a2:1758 with SMTP id q189-20020a0de7c6000000b0061130a21758mr9909937ywe.37.1712003217812; Mon, 01 Apr 2024 13:26:57 -0700 (PDT) Received: from fedora.attlocal.net ([2600:1700:2f7d:1800::23]) by smtp.googlemail.com with ESMTPSA id y72-20020a81a14b000000b006142210a31esm1171181ywg.23.2024.04.01.13.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Apr 2024 13:26:57 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, willy@infradead.org, "Vishal Moola (Oracle)" Subject: [PATCH v2 2/3] hugetlb: Convert hugetlb_no_page() to use struct vm_fault Date: Mon, 1 Apr 2024 13:26:50 -0700 Message-ID: <20240401202651.31440-3-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240401202651.31440-1-vishal.moola@gmail.com> References: <20240401202651.31440-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DAB5BA000B X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: n75qm8yxxypih5dsoxzwn4955bt7bjn5 X-HE-Tag: 1712003218-975339 X-HE-Meta: U2FsdGVkX19GldGfPBtK/MzClPmdmFRCje0hufoyN55YTIfzaN4KrXUYyrsP5XTcvLOV45jpE1gdUA+DGdsPTj2Toq5LV0+nEW/xWleUiKPZq2Sg6wNzuqftlkAB/qtlZoU56SO1gBP2+P9sp+Uqgi29gZW6QiPeNGWiWi/tZI8UlU0nr4m5wU7t/gqlWcmTeaUcJkAaIZsv3WagcWpLb2ayl1mMGO9et8UoklnwKYc+fGWanW2aXTKkjnW0KVSVLjr7l+h41Gi4nUQttoGxIAf/3P5GflWzCJ2n9FmQfWTns8qZIrMZ9YsmF4wJ8idhoAE2rsD6E32p/e+043Y8xBkiF8N4C+SsOlHBr9CbTBk22i3+H79iDa8qSaYRoEOcz2HWkFDu00d1P5YKxvKjAzuB7P/tVXJfzSgMcxvVS6nZOQfqhBQt/T4DIdAPX2sIQpLi8VKou9gX23OZ6g+VzoCEpnEJyEBTLMRKJ9a11L+CMZbLLhiba7OW9TdWA00aNfhREYlHm3V147UeEPlYtT/xsa+T4AMNC3eVe5TlGttKAVRRWcAvHmu1zKFcyaDgUU0aNoWGBTwgG0d2pGsbmCKqXN5I6WOeX5GmxIEu3+vFttiiOlktoqrVHgoPsEWWVhwa6EnXvU+YEdxYYqLURUDumAC11/QexsjGMCB2q5VVRNBDATPP+/0ULkkZGG36+bnXxOAkK0sjt6aMQY2xSpZgvkx31RCm4u/EBicWm7DOel6Amb4JogY7VW3CJgmtG3nQ6J5VEpdp2k6pDoWjcc1GT3yuJtZkqTmAyDrFUEmbzwWz3i1Jto5pazJzvKcCNnMF4MIOJj3fn3lsoY2eitCktvEsojwuO19URLDDh4REdQFsaFkyY7IrAjGbcTleCSraxti1paAaxH3S5OsrXyTg0Y1qw1E8n2bXURZpl2ikYBy0k5f9X5vby/s7lozJ/vCuy8jCEZw6JPsL4pu Kft3nb0n 64RsynEi0PQ3mlf+0YeSPMchy/zXBpQqp6qhTvJIXxEh2dtMTESSaLnhu/ldQ11f5EnnyKSvccBc4LpjsYlTkUMbNF+gZ4eFN/BUzzAmYCtciRJQm8sZb5NWTYcBteN3oX4nROlpolcNdVOM8LyNRm1Mkr+9WF1Nhxl8NKWZW5/C3rNRJnjRNSGPuFhDfC+GncbKQOOrRXh8h+h64qRjn+ztIcCVGSc5GuO00sgjrVwxJRAE22psMabmu31aD3ZY3cvr7FWx40liDLEHnPH7VsAiDqGC0C3eHSTGaOeU5PUnJnTVZnl8OuA3Dqsw+wIDfjA6DYwVvftQW3VKXNi4BN+hbgHaKgRsj0KIlmzP2CIeY6qAzVERSBr+OEllMSGkbCmX2V+O9KwFuRoQI6tZEA+/m45Ow2Kg5OsZeWPkHTxI5FQmi19VwANBwa1iHGTORF2nMMmONkMNSZX05O4/jGLH2zgHHdozGIsxbWSe1LDnHjDm56FOPefrq+rHz3V+0ecgluzLvrq9kNTK3Y7TD5b1T3VtHFQwWLqgaBQZAPcgXxrTZsJQX2ZgEA9JsetTGAc9C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_no_page() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 7 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Oscar Salvador Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 59 ++++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 360b82374a89..aca2f11b4138 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6189,9 +6189,7 @@ static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, - pte_t old_pte, unsigned int flags, + struct address_space *mapping, struct vm_fault *vmf) { struct hstate *h = hstate_vma(vma); @@ -6200,10 +6198,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, unsigned long size; struct folio *folio; pte_t new_pte; - spinlock_t *ptl; - unsigned long haddr = address & huge_page_mask(h); bool new_folio, new_pagecache_folio = false; - u32 hash = hugetlb_fault_mutex_hash(mapping, idx); + u32 hash = hugetlb_fault_mutex_hash(mapping, vmf->pgoff); /* * Currently, we are forced to kill the process in the event the @@ -6222,10 +6218,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * before we get page_table_lock. */ new_folio = false; - folio = filemap_lock_hugetlb_folio(h, mapping, idx); + folio = filemap_lock_hugetlb_folio(h, mapping, vmf->pgoff); if (IS_ERR(folio)) { size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) + if (vmf->pgoff >= size) goto out; /* Check for page in userfault range */ if (userfaultfd_missing(vma)) { @@ -6246,7 +6242,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * never happen on the page after UFFDIO_COPY has * correctly installed the page and returned. */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6255,7 +6251,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, VM_UFFD_MISSING); } - folio = alloc_hugetlb_folio(vma, haddr, 0); + folio = alloc_hugetlb_folio(vma, vmf->address, 0); if (IS_ERR(folio)) { /* * Returning error will result in faulting task being @@ -6269,18 +6265,20 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * here. Before returning error, get ptl and make * sure there really is no pte entry. */ - if (hugetlb_pte_stable(h, mm, ptep, old_pte)) + if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) ret = vmf_error(PTR_ERR(folio)); else ret = 0; goto out; } - clear_huge_page(&folio->page, address, pages_per_huge_page(h)); + clear_huge_page(&folio->page, vmf->real_address, + pages_per_huge_page(h)); __folio_mark_uptodate(folio); new_folio = true; if (vma->vm_flags & VM_MAYSHARE) { - int err = hugetlb_add_to_page_cache(folio, mapping, idx); + int err = hugetlb_add_to_page_cache(folio, mapping, + vmf->pgoff); if (err) { /* * err can't be -EEXIST which implies someone @@ -6289,7 +6287,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * to the page cache. So it's safe to call * restore_reserve_on_error() here. */ - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, + folio); folio_put(folio); goto out; } @@ -6319,7 +6318,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, folio_unlock(folio); folio_put(folio); /* See comment in userfaultfd_missing() block above */ - if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6334,23 +6333,23 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * any allocations necessary to record that reservation occur outside * the spinlock. */ - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { - if (vma_needs_reservation(h, vma, haddr) < 0) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if (vma_needs_reservation(h, vma, vmf->address) < 0) { ret = VM_FAULT_OOM; goto backout_unlocked; } /* Just decrements count, does not deallocate */ - vma_end_reservation(h, vma, haddr); + vma_end_reservation(h, vma, vmf->address); } - ptl = huge_pte_lock(h, mm, ptep); + vmf->ptl = huge_pte_lock(h, mm, vmf->pte); ret = 0; /* If pte changed from under us, retry */ - if (!pte_same(huge_ptep_get(ptep), old_pte)) + if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte)) goto backout; if (anon_rmap) - hugetlb_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, vmf->address); else hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6359,17 +6358,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * If this pte was previously wr-protected, keep it wr-protected even * if populated. */ - if (unlikely(pte_marker_uffd_wp(old_pte))) + if (unlikely(pte_marker_uffd_wp(vmf->orig_pte))) new_pte = huge_pte_mkuffd_wp(new_pte); - set_huge_pte_at(mm, haddr, ptep, new_pte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h)); hugetlb_count_add(pages_per_huge_page(h), mm); - if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, address, ptep, flags, folio, ptl, vmf); + ret = hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, + vmf->flags, folio, vmf->ptl, vmf); } - spin_unlock(ptl); + spin_unlock(vmf->ptl); /* * Only set hugetlb_migratable in newly allocated pages. Existing pages @@ -6386,10 +6386,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, return ret; backout: - spin_unlock(ptl); + spin_unlock(vmf->ptl); backout_unlocked: if (new_folio && !new_pagecache_folio) - restore_reserve_on_error(h, vma, haddr, folio); + restore_reserve_on_error(h, vma, vmf->address, folio); folio_unlock(folio); folio_put(folio); @@ -6485,8 +6485,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * hugetlb_no_page will drop vma lock and hugetlb fault * mutex internally, which make us return immediately. */ - return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address, - vmf.pte, vmf.orig_pte, flags, &vmf); + return hugetlb_no_page(mm, vma, mapping, &vmf); } ret = 0; From patchwork Mon Apr 1 20:26:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13613001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41B86CD128A for ; Mon, 1 Apr 2024 20:27:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F79E6B008A; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CF306B0092; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4232C6B0093; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 134406B008A for ; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D501DA052D for ; Mon, 1 Apr 2024 20:27:01 +0000 (UTC) X-FDA: 81962097042.03.F018A22 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) by imf25.hostedemail.com (Postfix) with ESMTP id 13A49A0023 for ; Mon, 1 Apr 2024 20:26:59 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GihV+NOn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712003220; a=rsa-sha256; cv=none; b=eA83LXyhIy6IwNmcqYhbBV8tpuD2Yowj3Wrb132MbkUP5KqTz03RQEzScBNvzE8uSTfdOF OBNqCOEja6E7IJMPDK5+wFpolf3Vg1dxwlz5x0xHdRycgfOq07XWWvFSU7BWilFVC6ZJGJ dLnSol7fbZMgLOcg06I406HFrTp8VcE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GihV+NOn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712003220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=xSgBYWX26v+6sh+G9yThzwuJx0aWHhwzQjPYNr25MahVpKJG0FM91tGMgHVNlYOLncab6L Tk/yH2GJ5hDZtEGrntJHwMkGSr5vbqTiWBlB7nDMfuiZLwwhMWbhHZ3XGVubofk/QYUJhZ ZDB6OJvltOrJNZyNFdXpIyCagWBMujA= Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-6151d2489b4so816407b3.0 for ; Mon, 01 Apr 2024 13:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712003219; x=1712608019; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=GihV+NOnIEC35JFcYp9it42+9nm5y+d4TN9LsCFVmdm8kyeXqvUk7ttj2IfP+FXtGO mXf1eJmLBWe9+Cp8+uHu3Csf/3BJQnpvJb3on05OQeZG/9pYWwEZzDWrke3vtYv/WmTP HXLdUF64FUOOHbog89eygFE7ROyrT39HMZrwX7bh0m1DCb/dy5pksjTniVEkOsTp3XH7 3GWHiI0ntuHETjVkyF1h1xv8MuzgYRvjNasmXu4B3CJE+Q0ICxdN7A21m83GmapGAOGJ u/IKj06x/RQjo7J6WPL7YqxyfSDJADJ2VxEZRBDh2quCLHgFUWSeqHr2hDL/+fKO5q5X Vhmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712003219; x=1712608019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=Xi+QW8LG6XHZt2XN1nvcomHs31HC2K3SSHSS9W8AaOnZAo+/fX/z9IebG6hrNbolcr VYuyqlz/T2z/m7RV3WbqP5AwJkDNqdUPXynupRzoNTvtMTJLTfQHFKw8xeicoOzDFfKL cO7UubZEL7L3pAgXKIuxG51DG84D3tcnkWZ/ZMt1eFNF4+YJSXholv6+aXZJ5eUkzYew /nZKAyPijJ8DbLfqmeabiPce6UH2ChgKwq2tIj5txqBwHXSjXI5g0a//JdiWeCdpgS1A Z+DpEOPAXrAYLd+rQXukyBhN/f9KD6DCgX0t72YZ/nWGCT0uH2gf5Fm6ak4V97yw9idk afwg== X-Gm-Message-State: AOJu0Yw67VTnRPE9pWfmycLTrzMDwHxDO93YwpGoTnQaTAjMwNwUzZ5P CY5VH7GpIXuUrPRrzHQucCz+5M3MI3bMFOKL9M/gbM/2kEa2hYI6v/TAz2Xb X-Google-Smtp-Source: AGHT+IFZ180aOPTb1CRRZ2Bu4ACYmdeIb3qE3iLlY2duuKwa7eQd71eLTyZ9BqrB+5kaF1LaPNv20Q== X-Received: by 2002:a81:6f03:0:b0:610:1a19:14fa with SMTP id k3-20020a816f03000000b006101a1914famr9131808ywc.50.1712003218915; Mon, 01 Apr 2024 13:26:58 -0700 (PDT) Received: from fedora.attlocal.net ([2600:1700:2f7d:1800::23]) by smtp.googlemail.com with ESMTPSA id y72-20020a81a14b000000b006142210a31esm1171181ywg.23.2024.04.01.13.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Apr 2024 13:26:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, willy@infradead.org, "Vishal Moola (Oracle)" Subject: [PATCH v2 3/3] hugetlb: Convert hugetlb_wp() to use struct vm_fault Date: Mon, 1 Apr 2024 13:26:51 -0700 Message-ID: <20240401202651.31440-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240401202651.31440-1-vishal.moola@gmail.com> References: <20240401202651.31440-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 13A49A0023 X-Stat-Signature: ygyejmd7w6n4mqekniifz366rmmqrmhg X-HE-Tag: 1712003219-268790 X-HE-Meta: U2FsdGVkX19IK50A1x1KCWdmmHE1CJafFOy66/SFI+r6FUcO5pHRNM+crF3n0JwjR3RCMeokQyHPCdbXt78PsvUmRFoOuAMmSEPgzRrsSjPsXJW00Z35n7dfKiBc7xaR1O35Y1BCq7csToiaVdBVUZYJu/7hbGQ3u1Z2aU1LovG6mOYq9fhXldFbOVIGc2SvOJmiBbjSnfcsyX+cqvDOQjLBw0hH4H+F/CXIbnoEiULjZByf7OmmaGGBRnr0wjSVoRfDa5nwmeWmp+1WRMZNCgTB7Cm1BqJyKqia/TcgWz6YwBePIKAXoSiICLAdg23CmWrk/owxA1avoT0TZ2xfj9dWlOnfDMIjZllangGHsyOtQ7YU5cTBMcYukCbddKOY1ePDCv61RQnAAofO+suZ/14ie09plgWX2JhBPq1LrlYxMpZMumQ8VdwBbzDFJ4kH6hA7RWcJNuuIyav+4T1Mb71HiDok2FPKiQSUd1iYp7zIqOg9jRcspiCJZnpm6xSA05LjTSaCvahbx1rRvZ4WV+sOAMUpuz9hynMiy1ExKmTcDWuUarDk3UY+J7Tjp+Iqd97RpZ9uxmkwCbzoDTw6cnZkTB35MH9RiVv3Jk71WOB658EiuFuPLLWJRn90oqQIuKMHB9pDrB2T8aEJ02SNv5gx1tAFzcnWI4yccjGSxed8gbCg4Ga9NW9w7KbGIFc8iZtiFIxsdAn7fxxKzVNwMKYHXo39ez52v5TZrl8b/AfH9RdAEbDMd/+Dt0Hgs1wH6DdRPiAkMisLDdemAqSsgobFEo8VG5d1+g1/dt8BBySkYoXsOhBGUh2Etzix+Vgs08R8msGU2VCNk2/17BGn3HY8BrKCFT83uTo9CMwvcMU/XHE1NJELLp6TKbb/3cR1ZrbWpZewt8qLfNnlIN+4xP7IFN2K4hWPo4j0TP4xwh4TPG9ySW4LO3DHretXz8Uc5/EmyhPJlCHEJ5/rHIW 87mtlV2l lx8lOSBZ4P0ZekKyzuwICz5XvdPXCgzlVnG76veoqKIRzBdTjDTVWst+/ZuzcJ+vMYZOfnKfuQXD4j+OdcbrHy2nAUZdE7qiuIN64EFVQ0M+DBs0RNDsbvjin3aNmeUTWIpe5ny+U6nj0JryxPP3ICxwjuXqKfQgQsSIFYSZ8S34nYl9a6CEeyGaKEbnyT2hAnF6xC/YJK59pZXeHCE5KajOdmcEmqmMKE1AqDXFiSw42e07Q9+OaUo/uYOXOeIFFLrpByNO3rCX+3+65Irs5fsv6TBWq7O6G90Syoo9J/LDlqFuI7q5oxH7P80OUh+1w64+wrgtn5V1+T5uvXJiF7swCVS2IcR7kgWsPxA7J5R0qx+aiDITHNKjDkh2ecs4L6s7Kfq8CKwayJMLD25z72IKbwWsePgDmLKHSmUgnazVmxDTuD3k+DrcKs2z/ZnBMOiOxV5aCifDIOmm2CHYdvzwFf8WKpkhce3FbUIEwPZcmLfPy5hk/YBwAqJCrm3CgHHiHH5AWGJLZKYE9Xtavr4eykSGfr4utsZ+DycYQ+98l3x26CHi6WUt9aav2/ZNIY4KS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_wp() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 5 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Oscar Salvador Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 61 ++++++++++++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index aca2f11b4138..d4f26947173e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5918,18 +5918,16 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, * Keep the pte_same checks anyway to make transition from the mutex easier. */ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, unsigned int flags, - struct folio *pagecache_folio, spinlock_t *ptl, + struct folio *pagecache_folio, struct vm_fault *vmf) { - const bool unshare = flags & FAULT_FLAG_UNSHARE; - pte_t pte = huge_ptep_get(ptep); + const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; + pte_t pte = huge_ptep_get(vmf->pte); struct hstate *h = hstate_vma(vma); struct folio *old_folio; struct folio *new_folio; int outside_reserve = 0; vm_fault_t ret = 0; - unsigned long haddr = address & huge_page_mask(h); struct mmu_notifier_range range; /* @@ -5952,7 +5950,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Let's take out MAP_SHARED mappings first. */ if (vma->vm_flags & VM_MAYSHARE) { - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); return 0; } @@ -5971,7 +5969,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, SetPageAnonExclusive(&old_folio->page); } if (likely(!unshare)) - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); delayacct_wpcopy_end(); return 0; @@ -5998,8 +5996,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * Drop page table lock as buddy allocator may be called. It will * be acquired again before returning to the caller, as expected. */ - spin_unlock(ptl); - new_folio = alloc_hugetlb_folio(vma, haddr, outside_reserve); + spin_unlock(vmf->ptl); + new_folio = alloc_hugetlb_folio(vma, vmf->address, outside_reserve); if (IS_ERR(new_folio)) { /* @@ -6024,19 +6022,21 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * * Reacquire both after unmap operation. */ - idx = vma_hugecache_offset(h, vma, haddr); + idx = vma_hugecache_offset(h, vma, vmf->address); hash = hugetlb_fault_mutex_hash(mapping, idx); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - unmap_ref_private(mm, vma, &old_folio->page, haddr); + unmap_ref_private(mm, vma, &old_folio->page, + vmf->address); mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && - pte_same(huge_ptep_get(ptep), pte))) + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, + huge_page_size(h)); + if (likely(vmf->pte && + pte_same(huge_ptep_get(vmf->pte), pte))) goto retry_avoidcopy; /* * race occurs while re-acquiring page table @@ -6058,37 +6058,38 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(ret)) goto out_release_all; - if (copy_user_large_folio(new_folio, old_folio, address, vma)) { + if (copy_user_large_folio(new_folio, old_folio, vmf->real_address, vma)) { ret = VM_FAULT_HWPOISON_LARGE; goto out_release_all; } __folio_mark_uptodate(new_folio); - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, - haddr + huge_page_size(h)); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, vmf->address, + vmf->address + huge_page_size(h)); mmu_notifier_invalidate_range_start(&range); /* * Retake the page table lock to check for racing updates * before the page tables are altered */ - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); + if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare); /* Break COW or unshare */ - huge_ptep_clear_flush(vma, haddr, ptep); + huge_ptep_clear_flush(vma, vmf->address, vmf->pte); hugetlb_remove_rmap(old_folio); - hugetlb_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, vmf->address); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); - set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, newpte, + huge_page_size(h)); folio_set_hugetlb_migratable(new_folio); /* Make the old page be freed below */ new_folio = old_folio; } - spin_unlock(ptl); + spin_unlock(vmf->ptl); mmu_notifier_invalidate_range_end(&range); out_release_all: /* @@ -6096,12 +6097,12 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * unshare) */ if (new_folio != old_folio) - restore_reserve_on_error(h, vma, haddr, new_folio); + restore_reserve_on_error(h, vma, vmf->address, new_folio); folio_put(new_folio); out_release_old: folio_put(old_folio); - spin_lock(ptl); /* Caller expects lock to be held */ + spin_lock(vmf->ptl); /* Caller expects lock to be held */ delayacct_wpcopy_end(); return ret; @@ -6365,8 +6366,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, hugetlb_count_add(pages_per_huge_page(h), mm); if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, - vmf->flags, folio, vmf->ptl, vmf); + ret = hugetlb_wp(mm, vma, folio, vmf); } spin_unlock(vmf->ptl); @@ -6579,8 +6579,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { if (!huge_pte_write(vmf.orig_pte)) { - ret = hugetlb_wp(mm, vma, address, vmf.pte, flags, - pagecache_folio, vmf.ptl, &vmf); + ret = hugetlb_wp(mm, vma, pagecache_folio, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte);