From patchwork Wed Jan 4 22:52:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13089174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFF40C46467 for ; Wed, 4 Jan 2023 22:52:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C0A68E0003; Wed, 4 Jan 2023 17:52:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0726B8E0001; Wed, 4 Jan 2023 17:52:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7AE88E0003; Wed, 4 Jan 2023 17:52:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DB7668E0001 for ; Wed, 4 Jan 2023 17:52:17 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B34BDA01AF for ; Wed, 4 Jan 2023 22:52:17 +0000 (UTC) X-FDA: 80318616714.23.CAA79C3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id D1EBEC000F for ; Wed, 4 Jan 2023 22:52:15 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hjZ2GtNo; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672872735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iUQecF0p0KDaGb3ic+3dlRjRyqZfOtSt4XLNwuIXR0Y=; b=OqQAzYBBnvAagkTdkbjRD0uuNe405PklQ/8+3pTdlv71ckRcWCeMcOK7lyUJf/TPP3SiLl kEe1t9ID91SzTnLceTTTZRZDFrqT8ddWgOEjgijzPD4y+y4mYtjyZ2jbgeIWM0bIzndWNI UjYWOQfLzXnEekt4UXhf+iYLxrRAVf0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hjZ2GtNo; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672872735; a=rsa-sha256; cv=none; b=LwXmHCb2CphdEgUWY/IYmmVLex33UohBOu88iEtNjDo4TlcZdwouEPy/TmZdGG3saUjv4Z F5MWqCKUnM0hVusyHgRkWZV0hcGSLb4YLxAkUHX4aFEGz5ptLlh/NtImBsCcQnN5Jy8Uso 9TKCA3cEBMnruyrsyrTJIX8pvnxG3X4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872735; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iUQecF0p0KDaGb3ic+3dlRjRyqZfOtSt4XLNwuIXR0Y=; b=hjZ2GtNoQ6UeWrzJFIc2rNsYL00+stijIi8F5XWxNdYvVWYTF28FGP1vDoTQAQYVHZ5xXw 8ri9c9KUmKV6uTuXrhtYlUjlVAR0L8+q/DglzRuNHog2mToFJCOx7rrIcoJI0igmNPGxmS E+R2yn+dHbULLv5GidC/NKPvAYq+oaM= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-41-q7-ZY4wWNPWuScvdBAviXA-1; Wed, 04 Jan 2023 17:52:14 -0500 X-MC-Unique: q7-ZY4wWNPWuScvdBAviXA-1 Received: by mail-qv1-f69.google.com with SMTP id ob12-20020a0562142f8c00b004c6c72bf1d0so18224471qvb.9 for ; Wed, 04 Jan 2023 14:52:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iUQecF0p0KDaGb3ic+3dlRjRyqZfOtSt4XLNwuIXR0Y=; b=rD/oetmLbYn8k2gNIl1I8Xbvz+nam3xqx4L7u6nS2jmEfQzcT8xvJOVPfLyZya7ukq 3/48NR1TRAEr85PlCtNNuD1KtIEDvc7HEhDB/XAMdjp+p/tLhR8RSt93GqiLCHrWAnfw tEKFjL6tA5ZO1fLWn8ytFzjQOKHx+hrfLemTeVUXX6Kwxk9oGeB5lUyOVy+cMb6fnh5i QAvYtsxDnmm17K8UkpX/c/7rijgFdYVbvAhI2pVvw/8JWPrjBj+wHUsexBf732pZU71G zME6eJu1KvkjqUB2MDJCblqoPpiHBjnhl/gJpj3k1iWufvnU37eNmw3TO+lyfKqYKp6B m3Gg== X-Gm-Message-State: AFqh2krwcax90f4F7Yeg/H/1BdW3nHVnlP8JM4tGjEwzZKeH0Hd5TJz0 fB1i3IsV1pA8w2Gm+20qBHStkcbK4IsnpYf2eYMeD2FZo0bDFL60Mtf4FZV+TO0tjjPPi3nLxOM C0agJkrtAAtU4nYJ+Pyz4hpYL9q1CdGduUjOKfBe8qpqZphIdmtz2XlfWwB/u X-Received: by 2002:a0c:f3ce:0:b0:531:7b1e:cdaa with SMTP id f14-20020a0cf3ce000000b005317b1ecdaamr56474342qvm.44.1672872732944; Wed, 04 Jan 2023 14:52:12 -0800 (PST) X-Google-Smtp-Source: AMrXdXvpKKemP1V8IrIbogqNKolIRqZbJHS4f3LCuRtRb0RPSmf1nkmMiOVux8MqIUMl3pEtxplS2A== X-Received: by 2002:a0c:f3ce:0:b0:531:7b1e:cdaa with SMTP id f14-20020a0cf3ce000000b005317b1ecdaamr56474310qvm.44.1672872732634; Wed, 04 Jan 2023 14:52:12 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:11 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton , linux-stable Subject: [PATCH 1/3] mm/hugetlb: Pre-allocate pgtable pages for uffd wr-protects Date: Wed, 4 Jan 2023 17:52:05 -0500 Message-Id: <20230104225207.1066932-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: zszw9nd47wjtpkcubtipren8j9495fsb X-Rspam-User: X-Rspamd-Queue-Id: D1EBEC000F X-Rspamd-Server: rspam06 X-HE-Tag: 1672872735-447869 X-HE-Meta: U2FsdGVkX19OTmfWybToCulhrqy/JxmGkRsgNfo85bqlQDNMvCZN+L7kmNnNssBJkaVRsZgPYDSL02/AugYv6K5YrF5D/d5fR02qSkwZfD5wzjTEHxbP1YtGYO+EFUL/5Wq/JShCcoVKwxrSBjlRJRmR7geVT0Oo2fERyiwFqW7AsK0LBvoj6nBDI7KsNxySk1C4qM579UVx90UltD8nCTJw0tUt3x6QMwrFKAYKP0FwryG8+7IjLTmxRHnnv7Gpd3IQuPDh2tpLw7EVmg+3BMY9o0ezVR9KClxgdNk8LYsLGX5uvhNLooXsJRBCphXEr62snXf+wXguRlyQiPetccsXG0MaFho0+55v3q7cimT3FZ93coHHRcPsam4DP6mAWErmdkp0cELGfMGdSMvPzeEKUPQp7EjLGyXeqM0YyaupQVrEfEMTcXtlmS5C2gscyT9OEldgfLhAGt6LUhhK7sGsSNCpiFt1cugZA+Ke7Qa1cUfDiDP1GtDpZFjIyFGMxzoZyeoF3gIzdYQSxXdnsSBi7zV7YDSkmhZLlj3UjcUKS1qc4dkqnBX3dpejfrdPf5yRYF6xA0LdqqiRi5iPHvYErK4c17eJszTOtNF/P/i0kH443EmLCJIWzWluBel9UdGjcdnLFYCDHiGziah+q7sj4x6dH5/PZdB8JXgX8RYrsfmvPrs0GtJgxBQ1fBAYhzgdx6r15RpHzkQqEHdGQpfj6zXALgKY2gvdyoa/8E2Z7GAWNpzCQCl49G0bgMGFWFex3hJgQIA35EFj1VXgvel5QWkBxWwoIuQTOosZXgj6IKkMW0lb3cqPE0s/xrRuCQV0IgDxgAy4XSzcF8vqdKH7V3vCbh5AdXYwEHFPNp8nnZD/Pp0rzrE2SoSNA6z05KoxW4TFN56twJM5JDE5jIxUbIKzU9U4l4gOr4hxuvMxQNttquf4QOlG9stvRX5X3W7NmBqRQUgbvvnUnbg 5ADxVtdX sf0OhoOI4xWL36S89oLYpICZgQRd5ZFAx2QjGN9H8+PJi5aPOQYYrK4v+SPtKc1C8WrFC6CcVG1bMnllae36ZvC146OHIKEbFAN9NJ0VX/csF7e1XZ3NOHpcO5ZvcM6chr+qzGvDfuK17UAhAqDDMwGwDkzSZJpb1E0XpkvrvHgonxqHTz7ieCePUSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Userfaultfd-wp uses pte markers to mark wr-protected pages for both shmem and hugetlb. Shmem has pre-allocation ready for markers, but hugetlb path was overlooked. Doing so by calling huge_pte_alloc() if the initial pgtable walk fails to find the huge ptep. It's possible that huge_pte_alloc() can fail with high memory pressure, in that case stop the loop immediately and fail silently. This is not the most ideal solution but it matches with what we do with shmem meanwhile it avoids the splat in dmesg. Cc: linux-stable # 5.19+ Fixes: 60dfaad65aa9 ("mm/hugetlb: allow uffd wr-protect none ptes") Reported-by: James Houghton Signed-off-by: Peter Xu Acked-by: James Houghton Acked-by: David Hildenbrand Reviewed-by: Mike Kravetz --- mm/hugetlb.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf7a1f628357..017d9159cddf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6649,8 +6649,17 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, spinlock_t *ptl; ptep = hugetlb_walk(vma, address, psize); if (!ptep) { - address |= last_addr_mask; - continue; + if (!uffd_wp) { + address |= last_addr_mask; + continue; + } + /* + * Userfaultfd wr-protect requires pgtable + * pre-allocations to install pte markers. + */ + ptep = huge_pte_alloc(mm, vma, address, psize); + if (!ptep) + break; } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, address, ptep)) { From patchwork Wed Jan 4 22:52:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13089175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F2ECC53210 for ; Wed, 4 Jan 2023 22:52:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5CCD8E0001; Wed, 4 Jan 2023 17:52:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0D2D900002; Wed, 4 Jan 2023 17:52:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AE2A8E0005; Wed, 4 Jan 2023 17:52:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7E4E08E0001 for ; Wed, 4 Jan 2023 17:52:20 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 43BF4C01BF for ; Wed, 4 Jan 2023 22:52:20 +0000 (UTC) X-FDA: 80318616840.06.3EF2EF5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 5C18AC0012 for ; Wed, 4 Jan 2023 22:52:18 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iT9TS1Ct; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672872738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YDqbXjeAi3fC7Raz+Oh8InMZYaeFp0RXTarC/jVXe5Y=; b=XjFeIfenZrwMks1KfcOJ3EELwPXXTsvE5ehywvLInZKMdFSiIj0q2S3Wx6SKbP6CoVdoMo zI5l47qEsjUdu0o47Z+1SfTopZ/tm0IccsE8paArglM/qYBX5g7Iy0YXqrJPk3X5ETnUe4 GODcFLAuzVvsfvdnEU98DyYdscY821c= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iT9TS1Ct; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672872738; a=rsa-sha256; cv=none; b=3/gjE0PdtEM3QQX1+hX1Ere/xQG63g3xJnVymjCjO2HCN7up7lvh5L5+4t3do8kn2fhJcq Qq17hgCrJh09m7+EhWW7dGW7d2pGvCxJGK6W33MTlYRQgWJ32LXmVKl/9J908fVJr2FtLC 0lstzudQ5z8FW/tOYWr1htFBR/SUuig= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872737; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YDqbXjeAi3fC7Raz+Oh8InMZYaeFp0RXTarC/jVXe5Y=; b=iT9TS1CtmvkCTD4kLRY+4veHtK4Rf+pVsviVX32iMx/fRAg1qjVUxlwFlfR5N0RmUmf77R FUOi2iRy81vmWdrgc7Zk1tASpdTmQ6QgkcmPhhpbkqOZHJ1Ig74C5o7nyTAnrHeG72osDP WnfqjbkeDfpqkMzVfjwGZk/0FC6B0Cs= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-265-yEi_AqwNPkCQGg5UEOJ_Rw-1; Wed, 04 Jan 2023 17:52:16 -0500 X-MC-Unique: yEi_AqwNPkCQGg5UEOJ_Rw-1 Received: by mail-qk1-f197.google.com with SMTP id o13-20020a05620a2a0d00b006cf9085682dso24028115qkp.7 for ; Wed, 04 Jan 2023 14:52:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDqbXjeAi3fC7Raz+Oh8InMZYaeFp0RXTarC/jVXe5Y=; b=4e19s3DN+vuwaymYFmZnSltWmqD2JiGQtvD/Iz//uGTyiULCWKpcJPfU+u6uRhfzdz FSXiZMN8oRETXl7ndSIMN5MUt3/WaanB5l42Jk98dCM5WwisQFq7ETvYVXo8a1k59niJ oqKrwVDntiWRsTUMlFvYE54UBcOZ7hTaVSQAX7fiRmTf2OJ2UiymKrXbOqpjmHBgw7zz vaeigcqvB5kIwc3dEmNNMyi3oKkkR1+uPopKyjHF3PW0z6+JZF6CJCq0P0hgiHHSCsGt Rv1JSUFPsBFiEr+rT4BujE4YgljW8aNan6pzNQy7+XLute0kac4wV1HAz53PEnI/VyGw sTLQ== X-Gm-Message-State: AFqh2krcB506grbmZKwgAEksjnxCHREvlxOdG9pBQ5SNxlZe5bfuUIa6 TGN63TEfiTDaFXfJvQYWDpEGetHXOYHLG30XG2bMCeGn1VV1BjeeWEZ0UvHtR2D0qSdQbK71WrH IpxQcPcV5vR45B8YcdZJUMwTh5jbFHSesNTqsa1WhI0eL1M2t/bjh9ZQMRvK7 X-Received: by 2002:a05:622a:5a98:b0:3ab:8c3f:328b with SMTP id fz24-20020a05622a5a9800b003ab8c3f328bmr54270804qtb.4.1672872735142; Wed, 04 Jan 2023 14:52:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXs1Sxv8n1AMZb/GH6LHO27LqRI5J2ATLlS4R+n+Mc6li+QeToMA1bMR//t6nBuvUswKomKeqQ== X-Received: by 2002:a05:622a:5a98:b0:3ab:8c3f:328b with SMTP id fz24-20020a05622a5a9800b003ab8c3f328bmr54270772qtb.4.1672872734836; Wed, 04 Jan 2023 14:52:14 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:14 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton Subject: [PATCH 2/3] mm/mprotect: Use long for page accountings and retval Date: Wed, 4 Jan 2023 17:52:06 -0500 Message-Id: <20230104225207.1066932-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 6p7ixuzyd3ig1xk6pgnx1dpzsigf6uip X-Rspam-User: X-Rspamd-Queue-Id: 5C18AC0012 X-Rspamd-Server: rspam06 X-HE-Tag: 1672872738-227305 X-HE-Meta: U2FsdGVkX1/frYoORzRR0yYIVUdo6Uzg2+W0e0Aymxq3LYbcmb+YYGwG+gwtQKUMi4W28l63QGK9vsarpcLA27OGfHfGcD7S3FY5XczPADfyMGfUmPdAjrVmkTvHKqZxYbC3RotJ5JNAzXwl2flBqHsLT5kLY3QC5khlNy/UEkiadf29WdOln9N0g0wOjyI4hJweqbeOK4/IBjAbwTxh0uV4CKYUVgG2Nk7p38mUdH3MX6+LavMN+42yQbGJBYrRCrhddXUsURMKAkX6YETKch+s0jnZCkTLSHiDhvTlca0/Q6vK4ImAFTxdkDL/PomhtBIPHMd9DqyZHJpTCkqSHEcW6ko7S0wHOOxS2sSMte7AsxSz1tnNymLRMna79WSZ6xnTrj2Kx9tj3/DwEbK/4jqsdDcxFJFA2rqIhgKqLHa0pobRTcyqrvF9yqCCU4fthTm5Fc7zTtcCzp7T1IkSYEBcfqJRyNqfU+7CsBK1nc0aO54KueE/01lE0Qfmj76+W7scILBOo6jQEvS2lidkkO2Sz89xab1wHtaCix+t6BgQHDd0DM3DNKZZcD/Z+Kx99AERYYvySs5AMyiYksY2DUkfnzzBoEeYwyNnzCx0GzPskiTYkGWr7S7pTX0n6GMDB6WNEyqIpZMiwCWKHm4xoG1GdbkrG9tNrvvDJiy/yESVleWnqjb4iU2F33LgCNKjFtSeKZwsJIInntjvduI6kHiPp/Y+AuKFYRBctsfWlBqn4mjUSnHs1/8KClxN2Yszu/ux7UoTLkDdtl0G5Ye+eE5UJyCNl8XDggYH/LGL9Ld5iDr0rwRaQtxNZJgjwemam+kc8s1uDMg+odNbCCf0jAiTN/HMYq5jaLZdVMb1T8TQ9++0VUayepgYfKOUU8upzXdd3Q89HXNy/QaqvoRouP9Ubql9jC3fz+WqW6Vv82SQ2C2uPbNeRiWN844sSdAvVT+jmU12WtmaYHKNTIq FfBUsnUG Lo7y4hJIGtbI0LbR8BzDDc2/i5F3m0sXctAE2FXSvt5oRowYm4EzjypN+fH9zK7+ys7FKxAGchdwC38/zyw8lkSXcHqVToPTDIaZmrBkpJtNMYSCP8jEdftF5BT0d2jUdr0AlFdBAIqlDy18mLhVMkCprBvFg5XHprQWCtC9vLt/V++TcYdD/2exmyw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Switch to use type "long" for page accountings and retval across the whole procedure of change_protection(). The change should have shrinked the possible maximum page number to be half comparing to previous (ULONG_MAX / 2), but it shouldn't overflow on any system either because the maximum possible pages touched by change protection should be ULONG_MAX / PAGE_SIZE. Two reasons to switch from "unsigned long" to "long": 1. It suites better on count_vm_numa_events(), whose 2nd parameter takes a long type. 2. It paves way for returning negative (error) values in the future. Currently the only caller that consumes this retval is change_prot_numa(), where the unsigned long was converted to an int. Since at it, touching up the numa code to also take a long, so it'll avoid any possible overflow too during the int-size convertion. Signed-off-by: Peter Xu Acked-by: James Houghton Acked-by: Mike Kravetz --- include/linux/hugetlb.h | 4 ++-- include/linux/mm.h | 2 +- mm/hugetlb.c | 4 ++-- mm/mempolicy.c | 2 +- mm/mprotect.c | 26 +++++++++++++------------- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b6b10101bea7..e3aa336df900 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -248,7 +248,7 @@ void hugetlb_vma_lock_release(struct kref *kref); int pmd_huge(pmd_t pmd); int pud_huge(pud_t pud); -unsigned long hugetlb_change_protection(struct vm_area_struct *vma, +long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags); @@ -437,7 +437,7 @@ static inline void move_hugetlb_state(struct folio *old_folio, { } -static inline unsigned long hugetlb_change_protection( +static inline long hugetlb_change_protection( struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) diff --git a/include/linux/mm.h b/include/linux/mm.h index c37f9330f14e..86fe17e6ded7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2132,7 +2132,7 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma } bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, pte_t pte); -extern unsigned long change_protection(struct mmu_gather *tlb, +extern long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags); extern int mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 017d9159cddf..84bc665c7c86 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6613,7 +6613,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, return i ? i : err; } -unsigned long hugetlb_change_protection(struct vm_area_struct *vma, +long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { @@ -6622,7 +6622,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pte_t *ptep; pte_t pte; struct hstate *h = hstate_vma(vma); - unsigned long pages = 0, psize = huge_page_size(h); + long pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; unsigned long last_addr_mask; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d3558248a0f0..a86b8f15e2f0 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -631,7 +631,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long addr, unsigned long end) { struct mmu_gather tlb; - int nr_updated; + long nr_updated; tlb_gather_mmu(&tlb, vma->vm_mm); diff --git a/mm/mprotect.c b/mm/mprotect.c index 71358e45a742..0af22ab59ea8 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -80,13 +80,13 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, return pte_dirty(pte); } -static unsigned long change_pte_range(struct mmu_gather *tlb, +static long change_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pte_t *pte, oldpte; spinlock_t *ptl; - unsigned long pages = 0; + long pages = 0; int target_node = NUMA_NO_NODE; bool prot_numa = cp_flags & MM_CP_PROT_NUMA; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; @@ -353,13 +353,13 @@ uffd_wp_protect_file(struct vm_area_struct *vma, unsigned long cp_flags) } \ } while (0) -static inline unsigned long change_pmd_range(struct mmu_gather *tlb, +static inline long change_pmd_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pmd_t *pmd; unsigned long next; - unsigned long pages = 0; + long pages = 0; unsigned long nr_huge_updates = 0; struct mmu_notifier_range range; @@ -367,7 +367,7 @@ static inline unsigned long change_pmd_range(struct mmu_gather *tlb, pmd = pmd_offset(pud, addr); do { - unsigned long this_pages; + long this_pages; next = pmd_addr_end(addr, end); @@ -437,13 +437,13 @@ static inline unsigned long change_pmd_range(struct mmu_gather *tlb, return pages; } -static inline unsigned long change_pud_range(struct mmu_gather *tlb, +static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pud_t *pud; unsigned long next; - unsigned long pages = 0; + long pages = 0; pud = pud_offset(p4d, addr); do { @@ -458,13 +458,13 @@ static inline unsigned long change_pud_range(struct mmu_gather *tlb, return pages; } -static inline unsigned long change_p4d_range(struct mmu_gather *tlb, +static inline long change_p4d_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { p4d_t *p4d; unsigned long next; - unsigned long pages = 0; + long pages = 0; p4d = p4d_offset(pgd, addr); do { @@ -479,14 +479,14 @@ static inline unsigned long change_p4d_range(struct mmu_gather *tlb, return pages; } -static unsigned long change_protection_range(struct mmu_gather *tlb, +static long change_protection_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mm_struct *mm = vma->vm_mm; pgd_t *pgd; unsigned long next; - unsigned long pages = 0; + long pages = 0; BUG_ON(addr >= end); pgd = pgd_offset(mm, addr); @@ -505,12 +505,12 @@ static unsigned long change_protection_range(struct mmu_gather *tlb, return pages; } -unsigned long change_protection(struct mmu_gather *tlb, +long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags) { pgprot_t newprot = vma->vm_page_prot; - unsigned long pages; + long pages; BUG_ON((cp_flags & MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL); From patchwork Wed Jan 4 22:52:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13089176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37BBCC54E76 for ; Wed, 4 Jan 2023 22:52:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C58FD900003; Wed, 4 Jan 2023 17:52:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C08C5900002; Wed, 4 Jan 2023 17:52:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8362900003; Wed, 4 Jan 2023 17:52:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 978CF900002 for ; Wed, 4 Jan 2023 17:52:22 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 609EF1A047F for ; Wed, 4 Jan 2023 22:52:22 +0000 (UTC) X-FDA: 80318616924.26.47A687A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 6C46EC000C for ; Wed, 4 Jan 2023 22:52:20 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HItE3CXJ; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672872740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w8KxT0fw2Kti4PKRWkIZPhnz1STfAelBM/L1tGxVI0U=; b=dxO+GdRSXbpdwXLVZtuQxEI+Oi9G+VImJv4sTX0jqrzelGQu3TT9zb4yvlLkY62YsTPT2J LlAysHrbqWjTqu9/eI17L5QUALPzlPbWbA7I+31kxgHCgdiBrFGSYEHunb3Tq60Kljjndr J+DcpWHUvd/gCq0wnbz/5EMSE4gQtSQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HItE3CXJ; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672872740; a=rsa-sha256; cv=none; b=r2K7rwyvx60hBTcOh8e0wunBYtvmnkcpFEEv4ro9vBxVLVsez30PYCGtVW7674APueg06C 86FKMD2VX4PCnzc+ihXH1P8BmjZ0njFccNLhRmRGU4DmDiChwo73oCZCYJYdGS+CCELfLE Ek7ULgjp9l4Qe3qlr0kLHjotL+ynHTI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872739; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w8KxT0fw2Kti4PKRWkIZPhnz1STfAelBM/L1tGxVI0U=; b=HItE3CXJfVT+f2C9YaUkDt8RACzJJPtsZt8NpwP0RvSno5xBAKBZ0N4s0t/rKBhWNWWSY4 3Zp4EtpQAfjngq/uXVc978QbeSFCFfgIioLKD69ickNAKfgM/KI1n6Yu+JIJcWRd8NmJ8B w7qx+n8nUDyGQOFUfSla0x7zL7uSkOU= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-66-3LvA_dJXNbKrgK8SQX4aJA-1; Wed, 04 Jan 2023 17:52:18 -0500 X-MC-Unique: 3LvA_dJXNbKrgK8SQX4aJA-1 Received: by mail-qv1-f69.google.com with SMTP id f11-20020a0cf7cb000000b005319ce47af9so9889845qvo.15 for ; Wed, 04 Jan 2023 14:52:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w8KxT0fw2Kti4PKRWkIZPhnz1STfAelBM/L1tGxVI0U=; b=plcEWROqauETfPQRZZxhiP0oIZ8nzxnG1KGOBk81lRm34TDdF1/Va//JPVdwRnPYhW vlywxHZUOawyd9XWWiqibe5qhoyqJALTItfXWwzUu7wpxCTlFWSUNMzMlsAaZ4R6JvAv cRk/8nASS55oOL7esoDfs03EpPMdbwyQ358uTyZVxiuJgd0oMavgHhydI31+RbOX87IY 1GgWKij4RJHbJAXGaeZVauFDQQ0+prj7isNlz+4+DqJdPkGzqQbY5y17hYA6cA55Sjed LToCB1ErOBRr6iyzBj+TNSUjtBXYet+H41og6ydZS3/Cm/MSCOz6sA4Ic6qv0y/yF8qn wo9Q== X-Gm-Message-State: AFqh2ko0hyuIP2kZU3l3HJbGKsSIzTmfbRvP4oASwT69WComr3zGXd3S tzWFRakaT0V+JrNlLzNFkyZBTpdxvUUuJRoFpSauJT0vguBO92SiGz/YtbAXE9B6SVM6sgZBRX3 uxg9bN3HMtDGzIqswbTZv7+GagsAPtAc+2wTwdCoxWQfgUT7pEfZInrp6jLj4 X-Received: by 2002:a05:6214:8e3:b0:521:ae4d:ea6a with SMTP id dr3-20020a05621408e300b00521ae4dea6amr65901342qvb.20.1672872737638; Wed, 04 Jan 2023 14:52:17 -0800 (PST) X-Google-Smtp-Source: AMrXdXu3dqYKFJ4Eo3TV1tw30PJqBr+Re6V+2aX8VJI591b6bZwEcKu5VZE1Ef0m6xJeQKmWoMv59Q== X-Received: by 2002:a05:6214:8e3:b0:521:ae4d:ea6a with SMTP id dr3-20020a05621408e300b00521ae4dea6amr65901311qvb.20.1672872737186; Wed, 04 Jan 2023 14:52:17 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:16 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton Subject: [PATCH 3/3] mm/uffd: Detect pgtable allocation failures Date: Wed, 4 Jan 2023 17:52:07 -0500 Message-Id: <20230104225207.1066932-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 7kg4jzw39pacnnhyxgsiuxr1wgeqoiei X-Rspam-User: X-Rspamd-Queue-Id: 6C46EC000C X-Rspamd-Server: rspam06 X-HE-Tag: 1672872740-635340 X-HE-Meta: U2FsdGVkX19Mex3y4jZWqTFH4ohile10wV/0zLfU6RDXhLaD80TtUogcVEgcP1mUXa/dWgmXtEgbGK7rPF2ZVZakaUf8O5zjoZCHW7ctP7K+WxtXzXbL/n1+0HoBBUyPa3DnvhQ4k6f9kqOR0UbPy0zlsz45qnokgYwJjP7yLFudXu50h55OTXNtDXaEgb1zqlNTBKFFZgkHVVIBBgkBgYE2gw/1rXT+aLO1r8PiR4kYxpfIwrr61/vruKh0YKB8LXsLxhc9nI2JcpJvYJHp3bG0uHprICcpSfMHsrKOq+oTehQDX1vEQb1BznPySf9QqRpy2yBDgXM1vyGygrVsbTygEmij5+UuLAbnX3xBQlHYdPq57GzjEytqNV4GwF3kcIGcrqQ3ZaiW3JGK+Upm6b5kcxxawsHRjQrqZ4bTJDAT7y/8xK9pB0IYaoVvKcXfDr+v9pMv1Z4ApWaHN8Uyh2/DQoPsI7nescYy9ZETx7+Efl7UPOSwGeFqc2HXtGPaKuEIe7fiYbN9545h9dNm7vnGOxdcXaSe2ZzuA5eYQsLg/s2bPARUNGLnAZUE1LhA9wKMZ2kHrzwAemgEjp5nehII2oZaWD2pmtKoC6gX45Dl/neHIDOx+Bu0epc2nOdKSUSxz5qIyhzGtolnZJGudbxeISUOtxRbLoeCEjAJixFj3Uj/ke0raYejcxKybU5/0/fz0DFjd2+bZQe9fasNZt7kGjNk1ttZn6CJDAHCSVURorhaRiXPuadBnTQDILRKTmpatdLLJundktw5kU2K0cf6zF5GMCS5HH1ptBU+GbqzEI5xzoyPMjdkX21MNvkK+OXPQQDich1OGnzBgSpoxs8WuAS8MfosrQwuRLO1tt0QE3Snq7DmFV7NBaFRZFdnKIKTou41XBTd+r2scsF7N/vtumaa2iH0vI4HKx0G/P9WLwuC02SUw4LJGeW6kwHJlHLwygqqtuE5EHTnuAk oDp9NJbE vVqBMy2wvCGECAIwZAdfT9AOMkGr1GV5LFMFG5jfd9HCJFbgRMgxoD9VOcFPGBpSHbzA69hs0L7a/yd6cX0qIpHjQVZv91zuPtUTfXZueX0/yig/at2i+FfEIqkm/nfG1mTgIY5UmLHF0IEL6OJcLa1BGmWSUFBLKNxH0W7Cwxy3NzBMdFX4q/OqTqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before this patch, when there's any pgtable allocation issues happened during change_protection(), the error will be ignored from the syscall. For shmem, there will be an error dumped into the host dmesg. Two issues with that: (1) Doing a trace dump when allocation fails is not anything close to grace.. (2) The user should be notified with any kind of such error, so the user can trap it and decide what to do next, either by retrying, or stop the process properly, or anything else. For userfault users, this will change the API of UFFDIO_WRITEPROTECT when pgtable allocation failure happened. It should not normally break anyone, though. If it breaks, then in good ways. One man-page update will be on the way to introduce the new -ENOMEM for UFFDIO_WRITEPROTECT. Not marking stable so we keep the old behavior on the 5.19-till-now kernels. Reported-by: James Houghton Signed-off-by: Peter Xu Acked-by: James Houghton --- include/linux/userfaultfd_k.h | 2 +- mm/hugetlb.c | 6 ++- mm/mempolicy.c | 2 +- mm/mprotect.c | 69 +++++++++++++++++++++++------------ mm/userfaultfd.c | 16 +++++--- 5 files changed, 62 insertions(+), 33 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 9df0b9a762cc..3767f18114ef 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -73,7 +73,7 @@ extern ssize_t mcopy_continue(struct mm_struct *dst_mm, unsigned long dst_start, extern int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, unsigned long len, bool enable_wp, atomic_t *mmap_changing); -extern void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *vma, +extern long uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *vma, unsigned long start, unsigned long len, bool enable_wp); /* mm helpers */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 84bc665c7c86..d82d97e03eae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6658,8 +6658,10 @@ long hugetlb_change_protection(struct vm_area_struct *vma, * pre-allocations to install pte markers. */ ptep = huge_pte_alloc(mm, vma, address, psize); - if (!ptep) + if (!ptep) { + pages = -ENOMEM; break; + } } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, address, ptep)) { @@ -6749,7 +6751,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); mmu_notifier_invalidate_range_end(&range); - return pages << h->order; + return pages > 0 ? (pages << h->order) : pages; } /* Return true if reservation was successful, false otherwise. */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index a86b8f15e2f0..85a34f1f3ab8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -636,7 +636,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, tlb_gather_mmu(&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, MM_CP_PROT_NUMA); - if (nr_updated) + if (nr_updated > 0) count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated); tlb_finish_mmu(&tlb); diff --git a/mm/mprotect.c b/mm/mprotect.c index 0af22ab59ea8..ade0d5f85a36 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -330,28 +330,34 @@ uffd_wp_protect_file(struct vm_area_struct *vma, unsigned long cp_flags) /* * If wr-protecting the range for file-backed, populate pgtable for the case * when pgtable is empty but page cache exists. When {pte|pmd|...}_alloc() - * failed it means no memory, we don't have a better option but stop. + * failed we treat it the same way as pgtable allocation failures during + * page faults by kicking OOM and returning error. */ #define change_pmd_prepare(vma, pmd, cp_flags) \ - do { \ + ({ \ + long err = 0; \ if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ - if (WARN_ON_ONCE(pte_alloc(vma->vm_mm, pmd))) \ - break; \ + if (pte_alloc(vma->vm_mm, pmd)) \ + err = -ENOMEM; \ } \ - } while (0) + err; \ + }) + /* * This is the general pud/p4d/pgd version of change_pmd_prepare(). We need to * have separate change_pmd_prepare() because pte_alloc() returns 0 on success, * while {pmd|pud|p4d}_alloc() returns the valid pointer on success. */ #define change_prepare(vma, high, low, addr, cp_flags) \ - do { \ - if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ - low##_t *p = low##_alloc(vma->vm_mm, high, addr); \ - if (WARN_ON_ONCE(p == NULL)) \ - break; \ - } \ - } while (0) + ({ \ + long err = 0; \ + if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ + low##_t *p = low##_alloc(vma->vm_mm, high, addr); \ + if (p == NULL) \ + err = -ENOMEM; \ + } \ + err; \ + }) static inline long change_pmd_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr, @@ -367,11 +373,15 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd = pmd_offset(pud, addr); do { - long this_pages; + long ret; next = pmd_addr_end(addr, end); - change_pmd_prepare(vma, pmd, cp_flags); + ret = change_pmd_prepare(vma, pmd, cp_flags); + if (ret) { + pages = ret; + break; + } /* * Automatic NUMA balancing walks the tables with mmap_lock * held for read. It's possible a parallel update to occur @@ -401,7 +411,11 @@ static inline long change_pmd_range(struct mmu_gather *tlb, * cleared; make sure pmd populated if * necessary, then fall-through to pte level. */ - change_pmd_prepare(vma, pmd, cp_flags); + ret = change_pmd_prepare(vma, pmd, cp_flags); + if (ret) { + pages = ret; + break; + } } else { /* * change_huge_pmd() does not defer TLB flushes, @@ -422,9 +436,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, } /* fall through, the trans huge pmd just split */ } - this_pages = change_pte_range(tlb, vma, pmd, addr, next, - newprot, cp_flags); - pages += this_pages; + pages += change_pte_range(tlb, vma, pmd, addr, next, + newprot, cp_flags); next: cond_resched(); } while (pmd++, addr = next, addr != end); @@ -443,12 +456,14 @@ static inline long change_pud_range(struct mmu_gather *tlb, { pud_t *pud; unsigned long next; - long pages = 0; + long pages = 0, ret; pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); - change_prepare(vma, pud, pmd, addr, cp_flags); + ret = change_prepare(vma, pud, pmd, addr, cp_flags); + if (ret) + return ret; if (pud_none_or_clear_bad(pud)) continue; pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, @@ -464,12 +479,14 @@ static inline long change_p4d_range(struct mmu_gather *tlb, { p4d_t *p4d; unsigned long next; - long pages = 0; + long pages = 0, ret; p4d = p4d_offset(pgd, addr); do { next = p4d_addr_end(addr, end); - change_prepare(vma, p4d, pud, addr, cp_flags); + ret = change_prepare(vma, p4d, pud, addr, cp_flags); + if (ret) + return ret; if (p4d_none_or_clear_bad(p4d)) continue; pages += change_pud_range(tlb, vma, p4d, addr, next, newprot, @@ -486,14 +503,18 @@ static long change_protection_range(struct mmu_gather *tlb, struct mm_struct *mm = vma->vm_mm; pgd_t *pgd; unsigned long next; - long pages = 0; + long pages = 0, ret; BUG_ON(addr >= end); pgd = pgd_offset(mm, addr); tlb_start_vma(tlb, vma); do { next = pgd_addr_end(addr, end); - change_prepare(vma, pgd, p4d, addr, cp_flags); + ret = change_prepare(vma, pgd, p4d, addr, cp_flags); + if (ret) { + pages = ret; + break; + } if (pgd_none_or_clear_bad(pgd)) continue; pages += change_p4d_range(tlb, vma, pgd, addr, next, newprot, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 65ad172add27..53c3d916ff66 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -710,11 +710,12 @@ ssize_t mcopy_continue(struct mm_struct *dst_mm, unsigned long start, mmap_changing, 0); } -void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma, +long uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma, unsigned long start, unsigned long len, bool enable_wp) { unsigned int mm_cp_flags; struct mmu_gather tlb; + long ret; if (enable_wp) mm_cp_flags = MM_CP_UFFD_WP; @@ -730,8 +731,10 @@ void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vma, if (!enable_wp && vma_wants_manual_pte_write_upgrade(dst_vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; tlb_gather_mmu(&tlb, dst_mm); - change_protection(&tlb, dst_vma, start, start + len, mm_cp_flags); + ret = change_protection(&tlb, dst_vma, start, start + len, mm_cp_flags); tlb_finish_mmu(&tlb); + + return ret; } int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, @@ -740,7 +743,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, { struct vm_area_struct *dst_vma; unsigned long page_mask; - int err; + long err; /* * Sanitize the command parameters: @@ -779,9 +782,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, goto out_unlock; } - uffd_wp_range(dst_mm, dst_vma, start, len, enable_wp); + err = uffd_wp_range(dst_mm, dst_vma, start, len, enable_wp); + + /* Return 0 on success, <0 on failures */ + if (err > 0) + err = 0; - err = 0; out_unlock: mmap_read_unlock(dst_mm); return err;