From patchwork Mon Apr 1 20:26:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13613001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41B86CD128A for ; Mon, 1 Apr 2024 20:27:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F79E6B008A; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CF306B0092; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4232C6B0093; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 134406B008A for ; Mon, 1 Apr 2024 16:27:02 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D501DA052D for ; Mon, 1 Apr 2024 20:27:01 +0000 (UTC) X-FDA: 81962097042.03.F018A22 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) by imf25.hostedemail.com (Postfix) with ESMTP id 13A49A0023 for ; Mon, 1 Apr 2024 20:26:59 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GihV+NOn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712003220; a=rsa-sha256; cv=none; b=eA83LXyhIy6IwNmcqYhbBV8tpuD2Yowj3Wrb132MbkUP5KqTz03RQEzScBNvzE8uSTfdOF OBNqCOEja6E7IJMPDK5+wFpolf3Vg1dxwlz5x0xHdRycgfOq07XWWvFSU7BWilFVC6ZJGJ dLnSol7fbZMgLOcg06I406HFrTp8VcE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GihV+NOn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712003220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=xSgBYWX26v+6sh+G9yThzwuJx0aWHhwzQjPYNr25MahVpKJG0FM91tGMgHVNlYOLncab6L Tk/yH2GJ5hDZtEGrntJHwMkGSr5vbqTiWBlB7nDMfuiZLwwhMWbhHZ3XGVubofk/QYUJhZ ZDB6OJvltOrJNZyNFdXpIyCagWBMujA= Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-6151d2489b4so816407b3.0 for ; Mon, 01 Apr 2024 13:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712003219; x=1712608019; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=GihV+NOnIEC35JFcYp9it42+9nm5y+d4TN9LsCFVmdm8kyeXqvUk7ttj2IfP+FXtGO mXf1eJmLBWe9+Cp8+uHu3Csf/3BJQnpvJb3on05OQeZG/9pYWwEZzDWrke3vtYv/WmTP HXLdUF64FUOOHbog89eygFE7ROyrT39HMZrwX7bh0m1DCb/dy5pksjTniVEkOsTp3XH7 3GWHiI0ntuHETjVkyF1h1xv8MuzgYRvjNasmXu4B3CJE+Q0ICxdN7A21m83GmapGAOGJ u/IKj06x/RQjo7J6WPL7YqxyfSDJADJ2VxEZRBDh2quCLHgFUWSeqHr2hDL/+fKO5q5X Vhmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712003219; x=1712608019; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=09S6vJM8RrkdrC/rkJ0OHkr1C8kqrWar9y0fEbWEbHY=; b=Xi+QW8LG6XHZt2XN1nvcomHs31HC2K3SSHSS9W8AaOnZAo+/fX/z9IebG6hrNbolcr VYuyqlz/T2z/m7RV3WbqP5AwJkDNqdUPXynupRzoNTvtMTJLTfQHFKw8xeicoOzDFfKL cO7UubZEL7L3pAgXKIuxG51DG84D3tcnkWZ/ZMt1eFNF4+YJSXholv6+aXZJ5eUkzYew /nZKAyPijJ8DbLfqmeabiPce6UH2ChgKwq2tIj5txqBwHXSjXI5g0a//JdiWeCdpgS1A Z+DpEOPAXrAYLd+rQXukyBhN/f9KD6DCgX0t72YZ/nWGCT0uH2gf5Fm6ak4V97yw9idk afwg== X-Gm-Message-State: AOJu0Yw67VTnRPE9pWfmycLTrzMDwHxDO93YwpGoTnQaTAjMwNwUzZ5P CY5VH7GpIXuUrPRrzHQucCz+5M3MI3bMFOKL9M/gbM/2kEa2hYI6v/TAz2Xb X-Google-Smtp-Source: AGHT+IFZ180aOPTb1CRRZ2Bu4ACYmdeIb3qE3iLlY2duuKwa7eQd71eLTyZ9BqrB+5kaF1LaPNv20Q== X-Received: by 2002:a81:6f03:0:b0:610:1a19:14fa with SMTP id k3-20020a816f03000000b006101a1914famr9131808ywc.50.1712003218915; Mon, 01 Apr 2024 13:26:58 -0700 (PDT) Received: from fedora.attlocal.net ([2600:1700:2f7d:1800::23]) by smtp.googlemail.com with ESMTPSA id y72-20020a81a14b000000b006142210a31esm1171181ywg.23.2024.04.01.13.26.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Apr 2024 13:26:58 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, willy@infradead.org, "Vishal Moola (Oracle)" Subject: [PATCH v2 3/3] hugetlb: Convert hugetlb_wp() to use struct vm_fault Date: Mon, 1 Apr 2024 13:26:51 -0700 Message-ID: <20240401202651.31440-4-vishal.moola@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240401202651.31440-1-vishal.moola@gmail.com> References: <20240401202651.31440-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 13A49A0023 X-Stat-Signature: ygyejmd7w6n4mqekniifz366rmmqrmhg X-HE-Tag: 1712003219-268790 X-HE-Meta: U2FsdGVkX19IK50A1x1KCWdmmHE1CJafFOy66/SFI+r6FUcO5pHRNM+crF3n0JwjR3RCMeokQyHPCdbXt78PsvUmRFoOuAMmSEPgzRrsSjPsXJW00Z35n7dfKiBc7xaR1O35Y1BCq7csToiaVdBVUZYJu/7hbGQ3u1Z2aU1LovG6mOYq9fhXldFbOVIGc2SvOJmiBbjSnfcsyX+cqvDOQjLBw0hH4H+F/CXIbnoEiULjZByf7OmmaGGBRnr0wjSVoRfDa5nwmeWmp+1WRMZNCgTB7Cm1BqJyKqia/TcgWz6YwBePIKAXoSiICLAdg23CmWrk/owxA1avoT0TZ2xfj9dWlOnfDMIjZllangGHsyOtQ7YU5cTBMcYukCbddKOY1ePDCv61RQnAAofO+suZ/14ie09plgWX2JhBPq1LrlYxMpZMumQ8VdwBbzDFJ4kH6hA7RWcJNuuIyav+4T1Mb71HiDok2FPKiQSUd1iYp7zIqOg9jRcspiCJZnpm6xSA05LjTSaCvahbx1rRvZ4WV+sOAMUpuz9hynMiy1ExKmTcDWuUarDk3UY+J7Tjp+Iqd97RpZ9uxmkwCbzoDTw6cnZkTB35MH9RiVv3Jk71WOB658EiuFuPLLWJRn90oqQIuKMHB9pDrB2T8aEJ02SNv5gx1tAFzcnWI4yccjGSxed8gbCg4Ga9NW9w7KbGIFc8iZtiFIxsdAn7fxxKzVNwMKYHXo39ez52v5TZrl8b/AfH9RdAEbDMd/+Dt0Hgs1wH6DdRPiAkMisLDdemAqSsgobFEo8VG5d1+g1/dt8BBySkYoXsOhBGUh2Etzix+Vgs08R8msGU2VCNk2/17BGn3HY8BrKCFT83uTo9CMwvcMU/XHE1NJELLp6TKbb/3cR1ZrbWpZewt8qLfNnlIN+4xP7IFN2K4hWPo4j0TP4xwh4TPG9ySW4LO3DHretXz8Uc5/EmyhPJlCHEJ5/rHIW 87mtlV2l lx8lOSBZ4P0ZekKyzuwICz5XvdPXCgzlVnG76veoqKIRzBdTjDTVWst+/ZuzcJ+vMYZOfnKfuQXD4j+OdcbrHy2nAUZdE7qiuIN64EFVQ0M+DBs0RNDsbvjin3aNmeUTWIpe5ny+U6nj0JryxPP3ICxwjuXqKfQgQsSIFYSZ8S34nYl9a6CEeyGaKEbnyT2hAnF6xC/YJK59pZXeHCE5KajOdmcEmqmMKE1AqDXFiSw42e07Q9+OaUo/uYOXOeIFFLrpByNO3rCX+3+65Irs5fsv6TBWq7O6G90Syoo9J/LDlqFuI7q5oxH7P80OUh+1w64+wrgtn5V1+T5uvXJiF7swCVS2IcR7kgWsPxA7J5R0qx+aiDITHNKjDkh2ecs4L6s7Kfq8CKwayJMLD25z72IKbwWsePgDmLKHSmUgnazVmxDTuD3k+DrcKs2z/ZnBMOiOxV5aCifDIOmm2CHYdvzwFf8WKpkhce3FbUIEwPZcmLfPy5hk/YBwAqJCrm3CgHHiHH5AWGJLZKYE9Xtavr4eykSGfr4utsZ+DycYQ+98l3x26CHi6WUt9aav2/ZNIY4KS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_wp() can use the struct vm_fault passed in from hugetlb_fault(). This alleviates the stack by consolidating 5 variables into a single struct. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Oscar Salvador Signed-off-by: Vishal Moola (Oracle) --- mm/hugetlb.c | 61 ++++++++++++++++++++++++++-------------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index aca2f11b4138..d4f26947173e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5918,18 +5918,16 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, * Keep the pte_same checks anyway to make transition from the mutex easier. */ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, unsigned int flags, - struct folio *pagecache_folio, spinlock_t *ptl, + struct folio *pagecache_folio, struct vm_fault *vmf) { - const bool unshare = flags & FAULT_FLAG_UNSHARE; - pte_t pte = huge_ptep_get(ptep); + const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; + pte_t pte = huge_ptep_get(vmf->pte); struct hstate *h = hstate_vma(vma); struct folio *old_folio; struct folio *new_folio; int outside_reserve = 0; vm_fault_t ret = 0; - unsigned long haddr = address & huge_page_mask(h); struct mmu_notifier_range range; /* @@ -5952,7 +5950,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Let's take out MAP_SHARED mappings first. */ if (vma->vm_flags & VM_MAYSHARE) { - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); return 0; } @@ -5971,7 +5969,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, SetPageAnonExclusive(&old_folio->page); } if (likely(!unshare)) - set_huge_ptep_writable(vma, haddr, ptep); + set_huge_ptep_writable(vma, vmf->address, vmf->pte); delayacct_wpcopy_end(); return 0; @@ -5998,8 +5996,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * Drop page table lock as buddy allocator may be called. It will * be acquired again before returning to the caller, as expected. */ - spin_unlock(ptl); - new_folio = alloc_hugetlb_folio(vma, haddr, outside_reserve); + spin_unlock(vmf->ptl); + new_folio = alloc_hugetlb_folio(vma, vmf->address, outside_reserve); if (IS_ERR(new_folio)) { /* @@ -6024,19 +6022,21 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * * Reacquire both after unmap operation. */ - idx = vma_hugecache_offset(h, vma, haddr); + idx = vma_hugecache_offset(h, vma, vmf->address); hash = hugetlb_fault_mutex_hash(mapping, idx); hugetlb_vma_unlock_read(vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - unmap_ref_private(mm, vma, &old_folio->page, haddr); + unmap_ref_private(mm, vma, &old_folio->page, + vmf->address); mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && - pte_same(huge_ptep_get(ptep), pte))) + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, + huge_page_size(h)); + if (likely(vmf->pte && + pte_same(huge_ptep_get(vmf->pte), pte))) goto retry_avoidcopy; /* * race occurs while re-acquiring page table @@ -6058,37 +6058,38 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, if (unlikely(ret)) goto out_release_all; - if (copy_user_large_folio(new_folio, old_folio, address, vma)) { + if (copy_user_large_folio(new_folio, old_folio, vmf->real_address, vma)) { ret = VM_FAULT_HWPOISON_LARGE; goto out_release_all; } __folio_mark_uptodate(new_folio); - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, - haddr + huge_page_size(h)); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, vmf->address, + vmf->address + huge_page_size(h)); mmu_notifier_invalidate_range_start(&range); /* * Retake the page table lock to check for racing updates * before the page tables are altered */ - spin_lock(ptl); - ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { + spin_lock(vmf->ptl); + vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); + if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare); /* Break COW or unshare */ - huge_ptep_clear_flush(vma, haddr, ptep); + huge_ptep_clear_flush(vma, vmf->address, vmf->pte); hugetlb_remove_rmap(old_folio); - hugetlb_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, vmf->address); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); - set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); + set_huge_pte_at(mm, vmf->address, vmf->pte, newpte, + huge_page_size(h)); folio_set_hugetlb_migratable(new_folio); /* Make the old page be freed below */ new_folio = old_folio; } - spin_unlock(ptl); + spin_unlock(vmf->ptl); mmu_notifier_invalidate_range_end(&range); out_release_all: /* @@ -6096,12 +6097,12 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * unshare) */ if (new_folio != old_folio) - restore_reserve_on_error(h, vma, haddr, new_folio); + restore_reserve_on_error(h, vma, vmf->address, new_folio); folio_put(new_folio); out_release_old: folio_put(old_folio); - spin_lock(ptl); /* Caller expects lock to be held */ + spin_lock(vmf->ptl); /* Caller expects lock to be held */ delayacct_wpcopy_end(); return ret; @@ -6365,8 +6366,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, hugetlb_count_add(pages_per_huge_page(h), mm); if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, vmf->real_address, vmf->pte, - vmf->flags, folio, vmf->ptl, vmf); + ret = hugetlb_wp(mm, vma, folio, vmf); } spin_unlock(vmf->ptl); @@ -6579,8 +6579,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { if (!huge_pte_write(vmf.orig_pte)) { - ret = hugetlb_wp(mm, vma, address, vmf.pte, flags, - pagecache_folio, vmf.ptl, &vmf); + ret = hugetlb_wp(mm, vma, pagecache_folio, &vmf); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte);