From patchwork Thu Jun 29 20:50:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 13297312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA4A9C0015E for ; Thu, 29 Jun 2023 20:50:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 384FD8D0003; Thu, 29 Jun 2023 16:50:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 33B728D0001; Thu, 29 Jun 2023 16:50:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2060C8D0001; Thu, 29 Jun 2023 16:50:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0DC648D0001 for ; Thu, 29 Jun 2023 16:50:53 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D6EEAB017B for ; Thu, 29 Jun 2023 20:50:52 +0000 (UTC) X-FDA: 80956979544.09.494F755 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id E811F100016 for ; Thu, 29 Jun 2023 20:50:50 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=a8O0wagy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3qu6dZA0KCCsHeLSYHZTbZZLUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--axelrasmussen.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3qu6dZA0KCCsHeLSYHZTbZZLUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--axelrasmussen.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688071851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R8jkAmCGbYNygAZgSop/zKP0BAnGYhZIuFEhOa1in5M=; b=SRAzgUmKk3WHDYFlZmDH2lNMmcCgpot7Eb1Itkd+sOzALCg16eMn4NNynJkmpT0jY64P08 8OnrOcl1TZC+SRiA8xAlhLbvYdGMUnjl7a3o1wsodmv6hEX5fSS4WFcLvSdlYlQKuHES33 kTaHpt270Cf0+QRs6LKw0wsUjGLiYrs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=a8O0wagy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3qu6dZA0KCCsHeLSYHZTbZZLUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--axelrasmussen.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3qu6dZA0KCCsHeLSYHZTbZZLUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--axelrasmussen.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688071851; a=rsa-sha256; cv=none; b=X5Bk4EDSgtfEGFUhdZalQBaEXLJxNGPAq91abJsafuV71LPcW335ZD7rpHpooWkcKOEHao 6wDak/Gp5nvS3237HZ74pLdGH+ni+KthZ/NxAv6cZHidYG1knhFYQ2oYo2toIqn9v7baqr fJmeoiwxZZS2Pm+Co5b6CttMK2rnhnI= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-bfee66a6398so919142276.3 for ; Thu, 29 Jun 2023 13:50:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688071850; x=1690663850; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=R8jkAmCGbYNygAZgSop/zKP0BAnGYhZIuFEhOa1in5M=; b=a8O0wagyUtUHav4dJ+0FTU4BfULv1hqXducKGWGQFuo+G+tIPjHy7DS1ADLv5ngmlB RSngMc0CnR4cfT4Dl1/3ekbmm7f4jkeC6RGHf37xUtQNDImjBtBI9DhIP8Z9AdXeugxu EEIqVXFJE09pcCYmkIu4KbepOYvezPBB0mUQ8g4biI54D+MfCLg+3QhsHFv6yxYBACnS hW5iDv/qv08V0YHMgwph0e9b+5pGUlXVr84spJ6yslQMHWase0Q3RfrmBefxTbxnzdJF i1oPLS3gMTFjIwMQZ2twKVJdYWAw+m3Lbi649+iUQ0hPFKbJ4IuR/hqznd3O1UEd0bUk 66bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688071850; x=1690663850; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R8jkAmCGbYNygAZgSop/zKP0BAnGYhZIuFEhOa1in5M=; b=jhD4TAbBAdaBOW/tZMSAHVIEz3KnNLxys7QENaAI3bqGSylAPQMpHrZrRzxUQqpWVI UAv89XVwdiGO8HoIK9d/bac04QPguJq5lgbbn8YCow+WzsiIsCCfMQtiehJMG+P7ay2m kdH1fDafmNPXN81Y/OrFOmpmMOycw2hg58mO+qwxFs+HaWVK4YJ/WvSeix3vGye65tQj 2I+WXrTJ5LkA8oW067UyV5lyk+H/Xw5BbY/5bZk/0Krm1vjwHhUG0MxoHUCs2Mjd97+1 RG7AMRh+f1SEg24ir5srKHSQY/q/c1Icpsgn/BJglpDYnWGzqWfjuIfkT76kE1sGkLbv 0Ffw== X-Gm-Message-State: ABy/qLbsyxQvfL84wZDcvwxpgE42X1UUn2yWzV62Tm551sOvFMaKeE5Z 1jthNW9TTUqOQEyK+HYbSWH1EHzoHcVqVvnrDMHu X-Google-Smtp-Source: APBJJlEqqtwXeQ2F+CB5P4eAYTWY2GLrYOnygDGt3hdnYMvy8/GPfEVbxlgghXg+IFwNtZpThqff4X0A++LKH4Bo56yy X-Received: from axel.svl.corp.google.com ([2620:15c:2a3:200:e20f:5917:3efa:d4bb]) (user=axelrasmussen job=sendgmr) by 2002:a25:738e:0:b0:bcc:285c:66ea with SMTP id o136-20020a25738e000000b00bcc285c66eamr5716ybc.3.1688071850044; Thu, 29 Jun 2023 13:50:50 -0700 (PDT) Date: Thu, 29 Jun 2023 13:50:36 -0700 In-Reply-To: <20230629205040.665834-1-axelrasmussen@google.com> Mime-Version: 1.0 References: <20230629205040.665834-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230629205040.665834-2-axelrasmussen@google.com> Subject: [PATCH v2 2/6] mm: userfaultfd: refactor hugetlb folio allocation / lookup code From: Axel Rasmussen To: Alexander Viro , Andrew Morton , Christian Brauner , David Hildenbrand , Huang Ying , Hugh Dickins , James Houghton , Jiaqi Yan , Jonathan Corbet , "Liam R. Howlett" , Miaohe Lin , Mike Kravetz , "Mike Rapoport (IBM)" , Muchun Song , Nadav Amit , Naoya Horiguchi , Peter Xu , Shuah Khan , ZhangPeng Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Axel Rasmussen X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E811F100016 X-Stat-Signature: go581oy5p5esosn4xyfaydh53dou5jqq X-HE-Tag: 1688071850-777160 X-HE-Meta: U2FsdGVkX18p/SNLOG4HVjFG3bvKCcJ+wYkACkZlkKhCoCvRgdm+U7Tvks2YLOvMx/6igyZHID64xPoSLC/az1hx8/TKvDPTK62Jc+IbnnvGrov37jw26G81TO4ihslpZZcpTvizvmh8yf6jBcDlceI2jbNXl9OelAMIGzppPTKtVDMUXyGbCL3mKA4oTjxQyt0MaCj9HydJBVzLKltKa2bWyNf482dq1fr+D5FUQo7PKeepCj3uWVuiYk3Ffow+76Lc0gBpoSVPLhNXJNT0OagTW6uvyfCKaMCZAX3CGbKaNUVUhVc4aOLNNmElCcFLPhukhb2UoFj2r2sWnZ5SVqMZmEe5FYv0Rg94d+bF/q87i1mBV5/rDWFRA+C3bF3dIzDu8IbviLSqvw/OFwAd6+ObfCJS0BE5SJZ8IEihM+Zkpq9yk7vOk5cMb5Ix4GCipcjRgl6ocUiMDyEwjOsUrSqM6TNBIucskgGhBgVejvouO+5oRICYAQAb7sDpaO0gZThVt/tQ+LJ9yK7X/efrhVStNO3Aj1hZxHWNb/aMnEYbeN0lpu5iecXGGjojudVjisEGbhfQWd1UpgXee9PFTrPJ71qXXSPhsvWD4F2vA0tvUrg1QLvXSDWwou0LxLQCyE2Md3zQyVwDcHqTbjNZzWql9GWtQlWtyiB7lM0nZ60jRGQpa1re856GvfBGwHy5kIP3A+gncri5k7wxeeW4BsPl7tt17AlEcWRoVpoHnpghkc2Rio9PIZtFoBjXJ1Pz3pAqh7xp9sa8tvkA7o6j1e3cExvS+v2ycPDdeq01WRHyuihTm8ZOQ24yaySOAMt+7u8odWzGj1UGaL58Q17/um2IhP/H/hTADxZpy/nLqr5x1CvxnTwDAXsj/3kl/Nee8/GRSkk5Z4rYRY+Ua6emRYZjVlI3B3k6JWQioa6irrhDQc7r00/gKgPqPd/jHqe2GgiCYA1IVGX1LeBcAXU 9NuagtUR RUDNn1nriOgIEPv8Lu7RnGzYsWVleXhO7Y8hZAzSD8bLaEGVVFv+3SIyvdAhK1gIarJ940jkS7VrNd6xGS6dIZ4a+5w6l0UvWfwuqSHFUvOTaZFjJFsDOTI4smLlDLOC2QYhAQtX9jVsdVoIgiVQzqm3o7OsKh1AE75D3e4NDrbYb7ttfMD1GHJsgy3/TUM2r6T/Fva4sw+Hup+ZPoSTYMcRWsFreRrOYCoJy6y9Uhe5BS7xRpS64Xy80IjN4wUlscs5S2gmdGgjQM1DFjW6mjYF8/ynIFI0optcLZFhZZ9xkaQfZFYa+xPsDmpme/ktrmeTjyFpwvnJPinrl9U2Yz9HrhNMFTl/NoPV1WL6INk1bIULFVW60ytTuOe9bjXIx85LVbupkrfBsPYO+Zv8feeRuU3XLkgk0exCe6S9Onzz3hU26VRB5Hb/KV10ONGCIgPFd+6EbpmDjQ1JKxJXYD5oX98EuaPCns7mfFVQHZngQb54IyATblwJYzNIhz8neluH6ZE/QHNwI9kJ2SR5wr460/WE+6eHpfVQnKfHzOePduwCd9IorrG1/AklVnU2fYUqymIEyC+cn/T42I0dfO4MwpR0dThXGR7NSAxZOVQoSrwjrCA408+hPHTNaBGWnelVuMevWiOBCuQz112uLeSSsmIp5DKtm6rpW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: At the top of `hugetlb_mfill_atomic_pte`, we need to get the folio we're going to be mapping. There are three basic cases we're dealing with here: 1. We're doing a UFFDIO_CONTINUE, in which case we lookup an existing folio in the pagecache, instead of allocating a new one. 2. We need to allocate a new folio. 3. We previously failed while populating our new folio, so we "returned" a temporary folio using `foliop` and had our caller retry. In a future commit I'm going to add a fourth case for UFFDIO_POISON, where we aren't going to map a folio at all (newly allocated or otherwise). This end state will be simpler, and we can re-use a bit more code, if we stop using `if (...)` to distinguish the cases. So, refactor the cases so they share most of the same code, and instead switch to `goto` to skip some parts depending on the case at hand. Signed-off-by: Axel Rasmussen --- mm/hugetlb.c | 53 +++++++++++++++++++++++++--------------------------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bce28cca73a1..38711d49e4db 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6259,22 +6259,32 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (IS_ERR(folio)) goto out; folio_in_pagecache = true; - } else if (!*foliop) { - /* If a folio already exists, then it's UFFDIO_COPY for - * a non-missing case. Return -EEXIST. - */ - if (vm_shared && - hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { - ret = -EEXIST; - goto out; + goto ready; + } + + /* If a folio already exists, then it's UFFDIO_COPY for + * a non-missing case. Return -EEXIST. + */ + if (vm_shared && hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { + ret = -EEXIST; + if (*foliop) { + folio_put(*foliop); + *foliop = NULL; } + goto out; + } - folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0); - if (IS_ERR(folio)) { - ret = -ENOMEM; - goto out; + folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0); + if (IS_ERR(folio)) { + ret = -ENOMEM; + if (*foliop) { + folio_put(*foliop); + *foliop = NULL; } + goto out; + } + if (!*foliop) { ret = copy_folio_from_user(folio, (const void __user *) src_addr, false); @@ -6302,22 +6312,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, */ goto out; } - } else { - if (vm_shared && - hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { - folio_put(*foliop); - ret = -EEXIST; - *foliop = NULL; - goto out; - } - - folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0); - if (IS_ERR(folio)) { - folio_put(*foliop); - ret = -ENOMEM; - *foliop = NULL; - goto out; - } + } else { /* Caller retried because we set *foliop previously */ ret = copy_user_large_folio(folio, *foliop, dst_addr, dst_vma); folio_put(*foliop); *foliop = NULL; @@ -6327,6 +6322,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } } +ready: /* `folio` ready to map (non-NULL, populated) */ + /* * The memory barrier inside __folio_mark_uptodate makes sure that * preceding stores to the page contents become visible before