From patchwork Wed Jun 14 02:13:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13279461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67879EB64DA for ; Wed, 14 Jun 2023 02:14:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BCF88E0006; Tue, 13 Jun 2023 22:14:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 647B48E0002; Tue, 13 Jun 2023 22:14:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AE738E0006; Tue, 13 Jun 2023 22:14:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 29CCB8E0002 for ; Tue, 13 Jun 2023 22:14:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0742D1606B1 for ; Wed, 14 Jun 2023 02:14:30 +0000 (UTC) X-FDA: 80899734300.03.E48C937 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 453CF100011 for ; Wed, 14 Jun 2023 02:14:28 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=YVGVxBZ2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686708868; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Aw61vajIkA6oxKB73VTHpA/VflnxZ90Ui6bDTaJJ3WI=; b=X6F/tNvYgXymCsKVT5+Ge+8U4xUuQxaAEsuLP9nVlCp1zGGkvYShgBjt1p9XcUvXzM9uqD T32Bagjjhv8Euq8SSgdwirElowrOJmw8HZiaT4BPYsNwmLFhPIM9THfILg+8A/douOcApe vfd7oGQOUeimyntvbSFf/uuYvaz299k= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=YVGVxBZ2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686708868; a=rsa-sha256; cv=none; b=SXPbWMU16rV738ZZimL12atzNzOiYobpbdpUrvuwqAt9+D2c9lYhBFXFLg+eWhZ3zPIOtr GUusPNp5CThRx98pKt3fiLJCEJ/vdb6xjzGTc7fcMSL8cIvstSeT7YEPLAnJHPc4B045RA 5bhT9MUaPxKbh5YXJgYQwRpeV+XAD4Q= Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-56cf6f4211aso2221717b3.1 for ; Tue, 13 Jun 2023 19:14:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686708867; x=1689300867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Aw61vajIkA6oxKB73VTHpA/VflnxZ90Ui6bDTaJJ3WI=; b=YVGVxBZ24BVkgipFXR1ege6SG0fquUM1JH7q42WM3bdfobtzIc/5hyKqeObocHTx9s YtuexTTE4n9ZN6g09ubR4ZWkQxBVPZrZ8m1eDguoe2G4wXW96CQ6FI8nblgkM5bY4jU+ gUFK4MUuHGorqR4D0Vp98p+Kscxi4CjzlTH61wP3tNZNnMnXRyz9SHgK4NDbi2jc0QhP ASVH5Zm5MkUGhggDoxVovPSkseAh9bRkRrXskHJeRr2xlK3DyGjUP3H9bWMNiAXnEgNF ZDU9v507kHNPrF/05L1OOJKm9zQGf/tzfJKh2qSoF+lgxnCuOtSSp7Iq6vL0G8ISpxMP vRQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686708867; x=1689300867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Aw61vajIkA6oxKB73VTHpA/VflnxZ90Ui6bDTaJJ3WI=; b=hcq42zYZ0t0/Nq08LFze8FerhvuNk35EgDCLEHfW7Xt9dayAw094trmRIFFZ9z3N9h rgybwuxOswNHMX5OfvTq8SBsnGEUpvG1khWjZPaktOHBBYRM+N0RCx0QSaumA+T5XtZE KSaVhDmI5njee0JBp1Fy9IZgAaDqub2IInRpFJNHI0Y+FadxwlKiFBphsSkiYrmkblWi ZBtA/h4QCm7tXm3PdCB6d/bVfei+tghPZX1ckU1Sr76igGEpea/GrmBESIkxswliCOcI e6n1EaZMf8HPiYXlpmfGBYP/pBtco9N1AmJTOgHBtMjY57w/lwDEy3hC6EJzyad8kODC 1xmA== X-Gm-Message-State: AC+VfDypoT4FbUlVYMcvgvRvip/StvTGaVIfC84lVFoh2XpYx+bpce2h N9gn+jnpOHdyuux4yb7haqo= X-Google-Smtp-Source: ACHHUZ44hRkvmSZRyOdyCv9N6hHgq9+NVCEiHaCWIGQfc+HCHZ7D1N44hd00/7Ix8TjS3ef0oKpmIw== X-Received: by 2002:a81:84d3:0:b0:56d:5a2:89ff with SMTP id u202-20020a8184d3000000b0056d05a289ffmr488167ywf.33.1686708867379; Tue, 13 Jun 2023 19:14:27 -0700 (PDT) Received: from unknowna0e70b2ca394.attlocal.net ([2600:1700:2f7d:1800::e]) by smtp.googlemail.com with ESMTPSA id p3-20020a817403000000b0056d30f1b508sm1463931ywc.107.2023.06.13.19.14.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 19:14:27 -0700 (PDT) From: "Vishal Moola (Oracle)" To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Vishal Moola (Oracle)" Subject: [PATCH v2 4/5] mm/gup.c: Reorganize try_get_folio() Date: Tue, 13 Jun 2023 19:13:11 -0700 Message-Id: <20230614021312.34085-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230614021312.34085-1-vishal.moola@gmail.com> References: <20230614021312.34085-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 453CF100011 X-Stat-Signature: sgm7xcmonbt9ychsmizdtdpc6adaugjg X-HE-Tag: 1686708868-570309 X-HE-Meta: U2FsdGVkX19ZIGG6ih48Czsbuv4jKXvXYG7DE5VHBAR/9TS2GaNBlWjVpTZTV+jtymvRh7dOLS+vHKJMd5inxuyLbBnICWejvLqPXTSX1UpmKbISFZlGjACktpjfmBVfTQtgd7MBNpbfdDlb8C8OMyMCCA0CWvOxYsWR7g3cKW9ec/+Ll2RVIYaxfb37FMg4lw/L9PN/1jIXmLKla6V7afK846B7nzwnBPnHkJzD4gLuVNvGHM5VE/kXRpkzhEbvvt2MaFbiFfCSrgO5YuCDW3YRhFl0DrZwPSb5mxohWoynv6jB31ldPQrFArZGHj7vV2bSGlDOLjdr+sjsEiREN/4uw9AaNcKHAPTfFVArVIJrJDMYksrwNU0+0F+49+Sw8h+Mf9bx053kfb+w9nAOtlrL/6UO9eR+KErOyBNUhdDAEWwXEmvsdw8Ld47yDyyYxQ680My6K8akF8nLofEaQzcRzAAlDd/MsmWaogkPRN1mvDl0mWXUOvQ9GDQfx9myphAz+wiUjviUlNICoI9WYFfMOfAp9o4XZ6onrmPHMSMDOcx+PbBEi4OaX5ufvPAyAEgAm6E+leKLhIG6Ez7vLo9b6C5Oc1F2P7DRzoC+HcZDZogeBkAlVSVco3MibTPUbotlnm1hrGSHUcjHu8vCKYbQowH65uLmJCOwV9Bp+s4Ktt2/IlcbzNQ6WIw6LcNAZajTQ60ri0cu474HO5CynEREMFkI1sSmHlI6rkQ9TORWlH+m5cVRpNMkM+NRe1cLUCGt1nb9j7qEkkdUMAOLMT6IOsjrwAE+dPFPdVt3hHtEvxxX0QhPPT3/HmN8e2+9dN5XsJgS8OZqa0TLi3W3M+VCwwcbpEveNCl2wRjSPJqocKDU6SvaoT58JUJmhBrYb1hQ7ArCKlJ2aovl2RkTzR+ECa2xSoMrI+fZAZ1VxB8mybKw+BGvA5Ci07E8Vg4dgu4tpZ8FmfdeeMUjYHl Nu84K9I8 1LWWJGrlECEhy8BaOw4pEct2DXmOTF4+nBtPImdpRoynaJVAq+prWdjiuWxuEfu8SAmPUdScakFFUnX1J4g78rvKr1NlqztOysKLfvP5T6sC0a0TzAHOU6Ws+N+uUksYfqq20opSIdjjdW3fx7eTnrlCbTu606npIpjfxkP0LUAAnVZ8ZkCbjUtVIleJj7A1iD+eH9IZ5NRR54i1DQ9Tz9DWOPvkOLYYTlWRfyOCd2vYFI6xXDSW2SG4Hs24PdAaqCKfTJ4pqn/dXuLwfxzKVhTSN7JgwLCBd9ZFppjtFxXYM4brYfWVdXtgM5NiYJ9ekyri1vSm9dMlMMr/6mRZnrV/2mEk2VvTAPMAnA4OVzLJC/fQF4bcWNXR7CEO4GmrNoVzbim+BmEqxmmtKHQA6P2jF1AMIDScJ1mu3Z1edMY211nkmNdQnJ5K09rD7vhAD9DkzDyraxetEcmIIFLI8SYlwdXvpH0SphN3cl1+Mzcy37i0QEijeLnNAMure+68+F/fAZvbwPSUVjfZ6Fz3ACfZ++aa3Pag1YpaywI/gky+IrnFlooGRY3ortM7gW2OG+J7FcCRkbwiwvMc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: try_get_folio() takes in a page, then chooses to do some folio operations based on the flags (either FOLL_GET or FOLL_PIN). We can rewrite this function to be more purpose oriented. After calling try_get_folio(), if neither FOLL_GET nor FOLL_PIN are set, warn and fail. If FOLL_GET is set we can return the result. If FOLL_GET is not set then FOLL_PIN is set, so we pin the folio. This change assists with folio conversions, and makes the function more readable. Signed-off-by: Vishal Moola (Oracle) --- mm/gup.c | 86 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 43 insertions(+), 43 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..6ec5c9629dc9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -123,58 +123,58 @@ static inline struct folio *try_get_folio(struct page *page, int refs) */ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) { + struct folio *folio; + + if (WARN_ON_ONCE((flags & (FOLL_GET | FOLL_PIN)) == 0)) + return NULL; + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page))) return NULL; - if (flags & FOLL_GET) - return try_get_folio(page, refs); - else if (flags & FOLL_PIN) { - struct folio *folio; + folio = try_get_folio(page, refs); - /* - * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a - * right zone, so fail and let the caller fall back to the slow - * path. - */ - if (unlikely((flags & FOLL_LONGTERM) && - !is_longterm_pinnable_page(page))) - return NULL; + if (flags & FOLL_GET) + return folio; - /* - * CAUTION: Don't use compound_head() on the page before this - * point, the result won't be stable. - */ - folio = try_get_folio(page, refs); - if (!folio) - return NULL; + /* FOLL_PIN is set */ + if (!folio) + return NULL; - /* - * When pinning a large folio, use an exact count to track it. - * - * However, be sure to *also* increment the normal folio - * refcount field at least once, so that the folio really - * is pinned. That's why the refcount from the earlier - * try_get_folio() is left intact. - */ - if (folio_test_large(folio)) - atomic_add(refs, &folio->_pincount); - else - folio_ref_add(folio, - refs * (GUP_PIN_COUNTING_BIAS - 1)); - /* - * Adjust the pincount before re-checking the PTE for changes. - * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). - */ - smp_mb__after_atomic(); + /* + * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a + * right zone, so fail and let the caller fall back to the slow + * path. + */ + if (unlikely((flags & FOLL_LONGTERM) && + !folio_is_longterm_pinnable(folio))) { + if (!put_devmap_managed_page_refs(&folio->page, refs)) + folio_put_refs(folio, refs); + return NULL; + } - node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); + /* + * When pinning a large folio, use an exact count to track it. + * + * However, be sure to *also* increment the normal folio + * refcount field at least once, so that the folio really + * is pinned. That's why the refcount from the earlier + * try_get_folio() is left intact. + */ + if (folio_test_large(folio)) + atomic_add(refs, &folio->_pincount); + else + folio_ref_add(folio, + refs * (GUP_PIN_COUNTING_BIAS - 1)); + /* + * Adjust the pincount before re-checking the PTE for changes. + * This is essentially a smp_mb() and is paired with a memory + * barrier in page_try_share_anon_rmap(). + */ + smp_mb__after_atomic(); - return folio; - } + node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); - WARN_ON_ONCE(1); - return NULL; + return folio; } static void gup_put_folio(struct folio *folio, int refs, unsigned int flags)