From patchwork Tue Mar 26 14:32:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13604285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 332AEC6FD1F for ; Tue, 26 Mar 2024 14:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7637E6B0092; Tue, 26 Mar 2024 10:32:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EF446B0093; Tue, 26 Mar 2024 10:32:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F02D6B0095; Tue, 26 Mar 2024 10:32:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 376D96B0092 for ; Tue, 26 Mar 2024 10:32:50 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F0469C0C2F for ; Tue, 26 Mar 2024 14:32:49 +0000 (UTC) X-FDA: 81939431658.11.A6356EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id CE5921C0015 for ; Tue, 26 Mar 2024 14:32:47 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LYNcYh7U; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711463567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cZdlw1b/kcLUrflUxwTe3odXoIWBotMqjI0wjKr5kQg=; b=IoEL/aXgBMSqIRu1NeZ/2eM8Da2MLMHR0LA7KkMF/KCElBFdR2R8BwQPbL5HDu5Gc3zL2z SNpmdgJuEzvO7T5d4Fw0Z0fJeX/cgsh0VAbbn1JzhHE8rd7g1hiFtfArUpWbDXagNHRrAa R28IrkYW+r0OExPWIBknJDH9E23RzIs= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LYNcYh7U; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711463567; a=rsa-sha256; cv=none; b=CrXraIa1AuTV1rzA37mc+uKpvqSax8NzSM/TCaOZ4V96RA9QAS27MrTfp1j0NE2uzSUcyi ggGWC1sLj29eBE7nMBQS4LE5qZmDOxSaQrmm7oFQ0j9wFol1tcqHoc0cHKg2Gm/+nqf0TF AU0SXviQATz9ETj18sOHWcRonOctm+Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711463567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cZdlw1b/kcLUrflUxwTe3odXoIWBotMqjI0wjKr5kQg=; b=LYNcYh7UQ5Q+o2W1IZqd3DEaU4NUl3RjfrC1E7yI9EN5jB7J3X+8ukRpMzIP0Jy/rWP/Tz 8Rga+tkjrcan2EyFJomiJzxUrttEc8+NJwYFEW32qcgOuENOAgvMfwgp5CMaN4+XWIIFoe wbvVmBJL4uIpPB0riCTBBwEy1B8BmAQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-468-AuwEDpdsMcGdxkD-3r8M6g-1; Tue, 26 Mar 2024 10:32:43 -0400 X-MC-Unique: AuwEDpdsMcGdxkD-3r8M6g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B404D800266; Tue, 26 Mar 2024 14:32:42 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.164]) by smtp.corp.redhat.com (Postfix) with ESMTP id 43F7540D2982; Tue, 26 Mar 2024 14:32:40 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Mike Rapoport , Miklos Szeredi , Lorenzo Stoakes , xingwei lee , yue sun Subject: [PATCH v2 3/3] mm: merge folio_is_secretmem() and folio_fast_pin_allowed() into gup_fast_folio_allowed() Date: Tue, 26 Mar 2024 15:32:10 +0100 Message-ID: <20240326143210.291116-4-david@redhat.com> In-Reply-To: <20240326143210.291116-1-david@redhat.com> References: <20240326143210.291116-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspam-User: X-Stat-Signature: etdet1b8ib1b1hoef6wracud4q6ob7ua X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CE5921C0015 X-HE-Tag: 1711463567-539294 X-HE-Meta: U2FsdGVkX1+YHg8ZVRj6AjX6ajuAiZvjtjQcBqePSObs9qLBbl9RVzNqjoZj6TjsVpc7bzOfQlnS6ZMND8VPLuYf8WFLNiX9ZwljzGiiFrL6njE+GhTFygI8TO6gYoP1QDeNzwZ45p9b4F7ZmNctqfW4BnvkNqM0kwNkQ1rI2fJiqdvhB3q3N2f3iESsUkDUfbAacgPdhpJd3DoBuj8bQE6XzKQnJUI/HHLxd5ouJf6k7P81bF/UHWlg7RzijviNU7AkPZLAGpjnJHd5YzvKK9EUACCnaRU0K5HVv9/dXagaSI45/VHQDLb9pu+VuxYYerfmKyko07S0ccGHODri6nDzrlO3wD7eI0O0F5PhJLbZlSua6kwGuCWiOtLQTfL280xglIvN8JxVz/KKsjXk6HfhlZ6EMC52RY9E9xDXWhoOfVwbDMNPdA1Ca8Jcn8554M/NnYR21fa/r9mB5lYoWgM72X14MqLtNBCc4sqgmrP/jPM1Yw32MkORTBpacDrZelEmFNaPN0+6uQALcm5p+ZmW8yO4c9r0VLUPKWdJ97z3adTuVCHI4xHCsPdWZGgseDaFZtIBxhhnY3mwJxh8g0C/fs3CX7U7NA/EqDWMVt+DpVrfWrMrhmkiGyNnoKVJFFsnDBWS7S/2AG5D5YPc+TFpKPaWRst18sr/VVyz2fCiCil+hgEZzlG/OA6laEeb68YdJRL/oJ4owhOovoIR8ZA1wecw0gv9+s1TgsHHJEcXchFDsIGEwDrUdTX8x39v1qJFm2/JMlOZUOmvgRS97u7fbmwe/zcmW0dZ49yUADhnr9DsRX8CzsGKXLhvPtoPzmCxA2xZ+mw5rJ5pi6LOB0CYytkiFufWVi5D/0KfJZJQR2KsxZ6ZnmPD2cLDZFo+R2MDZibiAxJCGIieIoOYKNRcnvwwfq89NxX3s3xovwfY8lN9F4p4hebo1iIE3xUQiRWXu+MTx+A1W6NeKKO 4tjY4Cv6 Vu9Ld+cjVXdhoRDL/S0oZJ1Jqt/Gm8tl5Uq3OtxVm2YBDpg6qHNqlSflVfAgDIWJnTAOdrKdnUvr3s4OFYLqwJeJm3Lv7+7Tv9N4VgJz3uE/v9IN4wpAcR0TVS9umtJGe0Ps+0hRhREN6nD4pK6OL2OCcnG4UgltGt0O9jyfj52lcAzNNs2rQVzCkI/t0SrSuMhoHLaJbwUjzEa+6kir3u33/XuGzd63pUexpBx1wAsvr6cbkLqS3FfbalfqR5CU9DFfFbaBwdTNtNhYW4OTx9jj8mg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_is_secretmem() is currently only used during GUP-fast. Nowadays, folio_fast_pin_allowed() performs similar checks during GUP-fast and contains a lot of careful handling -- READ_ONCE() -- , sanity checks -- lockdep_assert_irqs_disabled() -- and helpful comments on how this handling is safe and correct. So let's merge folio_is_secretmem() into folio_fast_pin_allowed(). Rename folio_fast_pin_allowed() to gup_fast_folio_allowed(), to better match the new semantics. Reviewed-by: Mike Rapoport (IBM) Signed-off-by: David Hildenbrand --- include/linux/secretmem.h | 21 ++--------------- mm/gup.c | 48 +++++++++++++++++++++++---------------- 2 files changed, 30 insertions(+), 39 deletions(-) diff --git a/include/linux/secretmem.h b/include/linux/secretmem.h index acf7e1a3f3de..e918f96881f5 100644 --- a/include/linux/secretmem.h +++ b/include/linux/secretmem.h @@ -6,25 +6,8 @@ extern const struct address_space_operations secretmem_aops; -static inline bool folio_is_secretmem(struct folio *folio) +static inline bool secretmem_mapping(struct address_space *mapping) { - struct address_space *mapping; - - /* - * Using folio_mapping() is quite slow because of the actual call - * instruction. - * We know that secretmem pages are not compound, so we can - * save a couple of cycles here. - */ - if (folio_test_large(folio)) - return false; - - mapping = (struct address_space *) - ((unsigned long)folio->mapping & ~PAGE_MAPPING_FLAGS); - - if (!mapping || mapping != folio->mapping) - return false; - return mapping->a_ops == &secretmem_aops; } @@ -38,7 +21,7 @@ static inline bool vma_is_secretmem(struct vm_area_struct *vma) return false; } -static inline bool folio_is_secretmem(struct folio *folio) +static inline bool secretmem_mapping(struct address_space *mapping) { return false; } diff --git a/mm/gup.c b/mm/gup.c index e7510b6ce765..03b74b148e30 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2466,12 +2466,14 @@ EXPORT_SYMBOL(get_user_pages_unlocked); #ifdef CONFIG_HAVE_FAST_GUP /* - * Used in the GUP-fast path to determine whether a pin is permitted for a - * specific folio. + * Used in the GUP-fast path to determine whether GUP is permitted to work on + * a specific folio. * * This call assumes the caller has pinned the folio, that the lowest page table * level still points to this folio, and that interrupts have been disabled. * + * GUP-fast must reject all secretmem folios. + * * Writing to pinned file-backed dirty tracked folios is inherently problematic * (see comment describing the writable_file_mapping_allowed() function). We * therefore try to avoid the most egregious case of a long-term mapping doing @@ -2481,25 +2483,34 @@ EXPORT_SYMBOL(get_user_pages_unlocked); * in the fast path, so instead we whitelist known good cases and if in doubt, * fall back to the slow path. */ -static bool folio_fast_pin_allowed(struct folio *folio, unsigned int flags) +static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags) { + bool reject_file_backed = false; struct address_space *mapping; + bool check_secretmem = false; unsigned long mapping_flags; /* * If we aren't pinning then no problematic write can occur. A long term * pin is the most egregious case so this is the one we disallow. */ - if ((flags & (FOLL_PIN | FOLL_LONGTERM | FOLL_WRITE)) != + if ((flags & (FOLL_PIN | FOLL_LONGTERM | FOLL_WRITE)) == (FOLL_PIN | FOLL_LONGTERM | FOLL_WRITE)) - return true; + reject_file_backed = true; + + /* We hold a folio reference, so we can safely access folio fields. */ - /* The folio is pinned, so we can safely access folio fields. */ + /* secretmem folios are always order-0 folios. */ + if (IS_ENABLED(CONFIG_SECRETMEM) && !folio_test_large(folio)) + check_secretmem = true; + + if (!reject_file_backed && !check_secretmem) + return true; if (WARN_ON_ONCE(folio_test_slab(folio))) return false; - /* hugetlb mappings do not require dirty-tracking. */ + /* hugetlb neither requires dirty-tracking nor can be secretmem. */ if (folio_test_hugetlb(folio)) return true; @@ -2535,10 +2546,12 @@ static bool folio_fast_pin_allowed(struct folio *folio, unsigned int flags) /* * At this point, we know the mapping is non-null and points to an - * address_space object. The only remaining whitelisted file system is - * shmem. + * address_space object. */ - return shmem_mapping(mapping); + if (check_secretmem && secretmem_mapping(mapping)) + return false; + /* The only remaining allowed file system is shmem. */ + return !reject_file_backed || shmem_mapping(mapping); } static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, @@ -2624,18 +2637,13 @@ static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, if (!folio) goto pte_unmap; - if (unlikely(folio_is_secretmem(folio))) { - gup_put_folio(folio, 1, flags); - goto pte_unmap; - } - if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } - if (!folio_fast_pin_allowed(folio, flags)) { + if (!gup_fast_folio_allowed(folio, flags)) { gup_put_folio(folio, 1, flags); goto pte_unmap; } @@ -2832,7 +2840,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { + if (!gup_fast_folio_allowed(folio, flags)) { gup_put_folio(folio, refs, flags); return 0; } @@ -2903,7 +2911,7 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { + if (!gup_fast_folio_allowed(folio, flags)) { gup_put_folio(folio, refs, flags); return 0; } @@ -2947,7 +2955,7 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { + if (!gup_fast_folio_allowed(folio, flags)) { gup_put_folio(folio, refs, flags); return 0; } @@ -2992,7 +3000,7 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { + if (!gup_fast_folio_allowed(folio, flags)) { gup_put_folio(folio, refs, flags); return 0; }