From patchwork Tue Jun 13 20:18:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vishal Moola X-Patchwork-Id: 13279175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33649EB64DA for ; Tue, 13 Jun 2023 20:18:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7437F6B0080; Tue, 13 Jun 2023 16:18:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CC556B0081; Tue, 13 Jun 2023 16:18:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51F0B6B0082; Tue, 13 Jun 2023 16:18:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 44CAC6B0080 for ; Tue, 13 Jun 2023 16:18:39 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 12D3AA05E2 for ; Tue, 13 Jun 2023 20:18:39 +0000 (UTC) X-FDA: 80898837558.28.7CD6E58 Received: from mail-ua1-f41.google.com (mail-ua1-f41.google.com [209.85.222.41]) by imf15.hostedemail.com (Postfix) with ESMTP id 16906A0017 for ; Tue, 13 Jun 2023 20:18:36 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=MAhhKx2u; spf=pass (imf15.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.222.41 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686687517; a=rsa-sha256; cv=none; b=XLQ7UtT4M4IZLmmwABt6XUZLBTad0m4Tkknn5Dixrm7YyDiaPmE8+pf+VRYzX716k2wSrk yjzqpqxESqbrDMdpbJDX/IaH0Ai0REej7ddXoT/Xw5+1d6M58KiD5hYjTuomUAG0J1UfDt 3krbAe5gGgYJ7nVq697vwL0DkD4AYiU= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=MAhhKx2u; spf=pass (imf15.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.222.41 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686687517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=amkbPFI+P7Pfwa/Tib2X8h1et/cH9zCscfc2pych/Qg=; b=kjOYnc5NqqqAPwdx8NzCJ2tW1nC0QbTjt527u5vTAzat5GrHazWPSlHAaFkPlQLeqb9L9n 2EwxcXVqdsWcH2EWp8mjb7Es0N9YjNKrZQ7pZN2LSCu6CeTiRed2qhJncLCroB4Q3Vuewa aVLxNfWpVp8ep6PBaw7wHYpTt1E2YeA= Received: by mail-ua1-f41.google.com with SMTP id a1e0cc1a2514c-783f88ce557so566880241.3 for ; Tue, 13 Jun 2023 13:18:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686687516; x=1689279516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=amkbPFI+P7Pfwa/Tib2X8h1et/cH9zCscfc2pych/Qg=; b=MAhhKx2uS7CtxHxYidopGlbP50mkEqVOHFIPJKoj5ZOU1awqmCAZySq2DuJ4PVyvwg NDzdpU61axPHkbgwy/Y2ElyR0ktqDf33BW1AKFl/xU7NB5LA46T0GJmHmkvZXBDKcPcl kYmptMQ0XgUVBW119yk0+pHJJeUdN2rQvdUbCPLskahLtJ1NDrGQk09ZYYQROPT6i5kH XjdwUoPGnkS5/pQJyWEzn6WBdetGabgHMrCzlwO0Ziqalg0ukpQbM8MFXCzLE5iPi596 3rcpmemEQcFBBqrfIWtEwnuM0nbjB8OD5Xlal27Ey2acFTT8jaHFTkIo/BYOwcK8NDny wVjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686687516; x=1689279516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=amkbPFI+P7Pfwa/Tib2X8h1et/cH9zCscfc2pych/Qg=; b=R8GZLeExTkhW1b7307r4axKNZZ5enK6F0LvbCq412gnehg+frqnXDUD4B5f9bVEGwW 4ixsQFP6f6A1XoqlKI4ewqn5uUSaya0meOQnC76eW9qI5JzEXa2QLDEL/bNBwIpGA0FQ CgJ8D5qAL2js9uy4x6Lg3yEdA1AtF6LpG+oW0AvVjaL07VeXD71YFMAaBvaATmA2qn9p fziGKZhmglWzoCkruk9sjqGlYtzOpQwTO31Q+A5lI2l/RHwaxAa2mO6GNxxa+mHpHyHE nJ3V8RPju3x75fp655ZM/OiUGV4MS9ayFNx05uprJcVoopBR54f1Tn2tMvN0+pd/v5Wy XmVw== X-Gm-Message-State: AC+VfDyDpiIuPJ7ZS3rK1U6l7I3lmo3l2wimyEf3SPGFkbsfdB8Myz0J faf8IC8q522bBVrrWQbbjBbl8TaArJkphg== X-Google-Smtp-Source: ACHHUZ4LLhgbvxObZNDOLYg9OIjMPobYKsI88dGjSOVbRgCeIyTAYxBHlyg7Q/xF8YsEkkdXEeS+eA== X-Received: by 2002:a1f:5f4d:0:b0:46e:70f8:c8ef with SMTP id t74-20020a1f5f4d000000b0046e70f8c8efmr1653501vkb.13.1686687516019; Tue, 13 Jun 2023 13:18:36 -0700 (PDT) Received: from unknowna0e70b2ca394.attlocal.net ([2600:1700:2f7d:1800::e]) by smtp.googlemail.com with ESMTPSA id b72-20020a0dd94b000000b0056cea0480c6sm2509985ywe.2.2023.06.13.13.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 13:18:35 -0700 (PDT) From: "Vishal Moola (Oracle)" To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Vishal Moola (Oracle)" Subject: [PATCH 4/5] mm/gup.c: Reorganize try_get_folio() Date: Tue, 13 Jun 2023 13:18:26 -0700 Message-Id: <20230613201827.9441-5-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613201827.9441-1-vishal.moola@gmail.com> References: <20230613201827.9441-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 16906A0017 X-Stat-Signature: xsz4da965fpkxzt3ub3x91hppdw35gbe X-Rspam-User: X-HE-Tag: 1686687516-912911 X-HE-Meta: U2FsdGVkX18n+YRvIGjkD17heGVSE04N60/BJL+Fm2MHp5gbK0WrjRcyKdQRzisXrQYWsoFkAcjzKvSdoSxS4TvNx0xKlgvrHiCYiaZFD/KD0O4Xx2C7XsibJpdaUuAf3oM6bvIst0r0o27svXMqtQgFM7aM7mh8fQ3e2jMXpvtrFnTQF9LjirHCaO9oWFWy3Og4oBdb3wW6QHxbTlykddWdfGzGYurIClvbmFPLsOc9EmHQPlITpehJSD+b82qCGl4fUn2LeX+ld+T7r/i4aYv0O4rcUnIwqMpprkkKuwPD9h1U+J4QSAnxnHv9i3zJ6hL+rXUlpilm3FkgY//6M3NvDblyuNKfhM6LyoHyCs/kdzMSNgbQ9iLsKVsvKiFcgpE7UkNgxKUpGU6v3DITEDxDfM7/FRKAr1t6/GBgo2Q/xTOojOPuB/R2VwuV7iCuT2AjEepO0PUd/RTrN70LoldnugmXNlwcmaeMRHgBFrzmsvhdApgpzGtMagE7vv9j4I5QTTKVBXlbOHwsidZCDBrysUKC7KjM5VWz2qfC4B8+qkEEAf7wTJ98RLF5dulMUr1mLYK+dxVrjutIyOPtEkwAfxlvtwdv4Ryc6etfLW1oppMcayGnkEZ2go+Wdcrg9bmnWi7ZMbAlINwqwIcm3iKUmsp3Kh9jOD9qaT2o656NXZNp+ql5kzeCc9awqyeKVletsxM57BOtXFnMeVP5ta7/lkPeBjtRA4syHnNji1TeKu+GtCSmiop22c+ZbWjtGvXerVgEOoFNyZ92eVX3pG5Hc50lBLDVf181eDmo85ILuCklJjXbdGEVoj8WtJWUhBvakZfrxnd38BkZSAziDMg+odEKfzjj7y7fSGbxHjSvj57tm7Qxox1DZNpI4lQKMjbMUcuPIvJLFlf17xM4kjMDX2M+DQNPs8hAjFpjBSZMtqcGXLSedI53LCgZIij2Xv/WxCGXlG0ZZy3Mb1H B/jNZjh6 QgN4b79va9qAU+D0tWQvVX6pmV6mdihM9Iq4PPdYCfRL4iBBmpP3b37wSVqMV5aGAirO1lYKLGql2ZrbH7LF1RojZ4Rgq57OMwrhVmJPJohFycoKxYfYC3fzdzkv5+HgxJB6jnA6WxcO2FmQTsMafUpQLy7y0BsgZll04KWibnGLSl/LNdrvy4lbWrtcqLozr5myBxVXktHMIgT9lEvlJNyrB3bs9r4PObv0e5s+5JOWtRBj5RKKCiH8Rzgp2rKzBjFUCfXgoscy7u2ORJExc+MOvDlZjtPHJPXSg/rquL+P6yYVvqs3qeebi72piINdP3Nvi4bOxNXUPmrEuApI4vAtC3JKCxbv4dOmvNieNtil6ZJZrBUVcSjU5nOS7PQ41QOMWhSyKO7CnhaEuSiiQNmdWFivuPtF4zHCjg3KgwYb4kvwGszxOCPtqbYCd/iC7+Kg12IWakEc20szib4sfRJr/RKqTklpSUDGaoQ6+GdHuM+0nUZSwF63CjfUGNYcmdhwsZsXA1UFZiJJa+hyA/kfOhgXy3rTMH+iQ75SHLqJl2kn/YNSFJti9GMLdjMWy1bU3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: try_get_folio() takes in a page, then chooses to do some folio operations based on the flags (either FOLL_GET or FOLL_PIN). We can rewrite this function to be more purpose oriented. After calling try_get_folio(), if FOLL_GET is set we can return the result and end the function. If FOLL_GET is not set and FOLL_PIN is not set then it's a bug so we warn and fail. Otherwise we simply proceed to pin the folio and return that as well. This change assists with folio conversions, and makes the function more readable. Signed-off-by: Vishal Moola (Oracle) --- mm/gup.c | 88 +++++++++++++++++++++++++++++--------------------------- 1 file changed, 46 insertions(+), 42 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..adbd81f888f5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -123,58 +123,62 @@ static inline struct folio *try_get_folio(struct page *page, int refs) */ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) { + struct folio *folio; if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page))) return NULL; + folio = try_get_folio(page, refs); + if (flags & FOLL_GET) - return try_get_folio(page, refs); - else if (flags & FOLL_PIN) { - struct folio *folio; + return folio; - /* - * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a - * right zone, so fail and let the caller fall back to the slow - * path. - */ - if (unlikely((flags & FOLL_LONGTERM) && - !is_longterm_pinnable_page(page))) - return NULL; + if (unlikely(!(flags & FOLL_PIN))) { + WARN_ON_ONCE(1); + return NULL; + } - /* - * CAUTION: Don't use compound_head() on the page before this - * point, the result won't be stable. - */ - folio = try_get_folio(page, refs); - if (!folio) - return NULL; + /* + * CAUTION: Don't use compound_head() on the page before this + * point, the result won't be stable. + */ + if (!folio) + return NULL; - /* - * When pinning a large folio, use an exact count to track it. - * - * However, be sure to *also* increment the normal folio - * refcount field at least once, so that the folio really - * is pinned. That's why the refcount from the earlier - * try_get_folio() is left intact. - */ - if (folio_test_large(folio)) - atomic_add(refs, &folio->_pincount); - else - folio_ref_add(folio, - refs * (GUP_PIN_COUNTING_BIAS - 1)); - /* - * Adjust the pincount before re-checking the PTE for changes. - * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). - */ - smp_mb__after_atomic(); + /* + * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a + * right zone, so fail and let the caller fall back to the slow + * path. + */ + if (unlikely((flags & FOLL_LONGTERM) && + !folio_is_longterm_pinnable(folio))) { + if (!put_devmap_managed_page_refs(&folio->page, refs)) + folio_put_refs(folio, refs); + return NULL; + } - node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); + /* + * When pinning a large folio, use an exact count to track it. + * + * However, be sure to *also* increment the normal folio + * refcount field at least once, so that the folio really + * is pinned. That's why the refcount from the earlier + * try_get_folio() is left intact. + */ + if (folio_test_large(folio)) + atomic_add(refs, &folio->_pincount); + else + folio_ref_add(folio, + refs * (GUP_PIN_COUNTING_BIAS - 1)); + /* + * Adjust the pincount before re-checking the PTE for changes. + * This is essentially a smp_mb() and is paired with a memory + * barrier in page_try_share_anon_rmap(). + */ + smp_mb__after_atomic(); - return folio; - } + node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, refs); - WARN_ON_ONCE(1); - return NULL; + return folio; } static void gup_put_folio(struct folio *folio, int refs, unsigned int flags)