From patchwork Thu Jul 6 22:50:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 13304274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9AF9EB64DC for ; Thu, 6 Jul 2023 22:51:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 474538D0006; Thu, 6 Jul 2023 18:51:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4266E8D0002; Thu, 6 Jul 2023 18:51:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DB5A8D0006; Thu, 6 Jul 2023 18:51:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0BE558D0002 for ; Thu, 6 Jul 2023 18:51:01 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C6226A0CBC for ; Thu, 6 Jul 2023 22:51:00 +0000 (UTC) X-FDA: 80982683880.11.1C191FD Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id EE49F14000F for ; Thu, 6 Jul 2023 22:50:58 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=D2k9xCsf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3UUWnZA0KCKQEbIPVEWQYWWIRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--axelrasmussen.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UUWnZA0KCKQEbIPVEWQYWWIRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--axelrasmussen.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688683859; a=rsa-sha256; cv=none; b=oCVJtcjsTsaT4JHHr2ygHcCJ2+FaygbUNTMUXfKj4/7cYALn/nN5MLngS6o4u4uSa71os5 QeAYI5sMtRZwYCAq05CnHM7+10OPrySbSsmrgctstyi4/F5qfKwyuzJlQbC9aBMW/ypyUh hLMIoljJGymHB/EIRieWsAD17Y4zlgY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=D2k9xCsf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3UUWnZA0KCKQEbIPVEWQYWWIRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--axelrasmussen.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UUWnZA0KCKQEbIPVEWQYWWIRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--axelrasmussen.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688683859; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PJqf0m2qw+Xjs8Quv7ufLANzkWFzCscIfWrhHyj74r4=; b=6N7ftu79KN9Q4VwdhbT4nihlAW/5kXNTI7q/LlpT9hzrNhT1WEq57aNAyQLy2GtRcEcJhf 2gAIcH1TC45hstJsz/qZJSLuvmF+TL/GZt09emCRZ2HYFuHAsIrr7kpP9qoTGXDTVS/WFx Jn4mwtVbfdRqJWFKS49E3qGvTU45uqk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-57003dac4a8so34056827b3.1 for ; Thu, 06 Jul 2023 15:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688683858; x=1691275858; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PJqf0m2qw+Xjs8Quv7ufLANzkWFzCscIfWrhHyj74r4=; b=D2k9xCsfeg61xPSIpw+bsBr3jPwLNm9nVEHyGhqfkG8lUTP35cd6IFL212tdgreqNz 0YlkyIjNjF30q+hjO/oW6G1cNPIRff4f/oHqF7Wy+jVNAq1/WuXr6jz15FTr0ttknNyr 0KjASHAlCDktsRxguFNoBBTItvv/YFICr/K4mQRPcS2hhFcyPon/giYdNfLAM48U0Jx0 eXvwik0EB73EKMqASVZMZZsmmHmToMBPVsvzhfYgIrpjZiLkZdUoFdyj4hnakuxSym+M ulEcJTqFS3AoXIeAsnUQhvNIC+QtVFc1duLHDu/kIEV70/F62oZksyCAFiDKBKPrHFko dgvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688683858; x=1691275858; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PJqf0m2qw+Xjs8Quv7ufLANzkWFzCscIfWrhHyj74r4=; b=cBd77nACOsyjC5H7lUKB9GKx/eRbvN86xD7ykpvTW7wbh1tJz7Zq9omZjVRhIQLEqE ZmBj4z5hrWcCJPengIiQKdFRK0oWmANlf8eQ8Asrpiv8KDZzFq7tY8un3JZ2lgBd3HSj R40fHvOZJsU6x/l9n52l0Jz2YfuhD7BkZci4msvWcPNKU05lJMqT99ikpB3FjYr9PiGW 8TuwXWzV6LTpE+akeyT4V5m1JU0sAKmqeAQnZyp04piv7ACubvuZf27vw371BDYQzQKt 2aXx5uGUlTTtB5pPAuxeWHJkIihf4bhNbYWypfK/W9zl4bMvjxlOxEJPLsRTuWlNJV61 u6AQ== X-Gm-Message-State: ABy/qLY3FRSuohMbUfkQeNlB5tsJStJSR8lORVzlCvUfxX5fX2nFIRdy dskkDoO3SRJURmzWkXTjn+wQOv+dzwPcbH+oHdzx X-Google-Smtp-Source: APBJJlE+eG+2pcJk+y2idAnX2/Q8CxWYDm8cs4gjQZ7N5JmAkaEanN8FC+O33rAco+pnBzwZaUMX3uj6yPdrSXWcsU/9 X-Received: from axel.svl.corp.google.com ([2620:15c:2a3:200:bec3:2b1c:87a:fca2]) (user=axelrasmussen job=sendgmr) by 2002:a25:d17:0:b0:bd5:dc2d:9d7f with SMTP id 23-20020a250d17000000b00bd5dc2d9d7fmr78836ybn.4.1688683857962; Thu, 06 Jul 2023 15:50:57 -0700 (PDT) Date: Thu, 6 Jul 2023 15:50:32 -0700 In-Reply-To: <20230706225037.1164380-1-axelrasmussen@google.com> Mime-Version: 1.0 References: <20230706225037.1164380-1-axelrasmussen@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230706225037.1164380-5-axelrasmussen@google.com> Subject: [PATCH v3 4/8] mm: userfaultfd: add new UFFDIO_POISON ioctl From: Axel Rasmussen To: Alexander Viro , Andrew Morton , Brian Geffon , Christian Brauner , David Hildenbrand , Gaosheng Cui , Huang Ying , Hugh Dickins , James Houghton , "Jan Alexander Steffens (heftig)" , Jiaqi Yan , Jonathan Corbet , Kefeng Wang , "Liam R. Howlett" , Miaohe Lin , Mike Kravetz , "Mike Rapoport (IBM)" , Muchun Song , Nadav Amit , Naoya Horiguchi , Peter Xu , Ryan Roberts , Shuah Khan , Suleiman Souhlal , Suren Baghdasaryan , "T.J. Alumbaugh" , Yu Zhao , ZhangPeng Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Axel Rasmussen X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EE49F14000F X-Stat-Signature: eee9uyp95httfknkg8f453fs1bkrhpmn X-HE-Tag: 1688683858-769062 X-HE-Meta: U2FsdGVkX18uSNSrvXy6kyGUYcovbXgu5p66M8vkPyFD9yWpUzO09UZpKyyTbMJo2xkKnyES3BsUqeHcnfqpeWdGB8woWqvbns0rmUlLRMJp9ixzwEQJiHIMMjTJIyiXvRNn2oUjHLqbtIA0vH2k2jUfjbKlYRR0V7Pzlei/CrOSzyquC6wYe5IB8pmE+6TwjlyulYIkgPs8MWUFj3GDszEsNQaNa9cD8+M3pEB+pFORGSx3NnjXQzDw3/mp1OGioJPKX1lP65g4YC71CPrl1R3Ay7qOpcp3rTLetchS3DJNyTU7dAn/2Z3GM/R02VLgQWRbaiMW1OhMBYyZZD0s6hTYObt2Sq88jBmmxQoG6pWcVQIjvAuSXu1hsOL6Lc2FgeoIZRZnbl74MrmjxFY4s/WrYgyCTMEuWW/o4PjgvaklJba68bdDifU1dzhbc8aIlQ8NLYX/h50cA0tKvMp5pIQg7RistfrrLHR6XFXZkmd58QJbhxISP+80eN7gbS+KeH3N21G76MM8dMRUVXEfsCLMAFGeBc89ha8YvTE4CoaXxAIPaXToC64rKMgHcEgJaNlAt0lW8JjrLsKoeEamgmZuXjgg8dyPghOFUa3yKxQdII6buWRx6IcdUMz3BspiiGWkCytlfAWwKctB9Cw+sx63bxL77lwfD4RJ0sreSKlkE54QgBKtIcvjxo5yI3q7W1ZwbwoL05hhr/nm0eG7wkI+NCFbb+K1K5HhqeNPu89MoMxlMESCXTK8vlrv21Z0mK15BPYDAeWZL+xvXg/9lnp0iW+6QSWipE8Hf7fir6rwD0ko1Xm66Hm5q3m/yeitt70kKiCBx8R2p7y8CKjavAR/+USYp01PnUguQ+CQK9Av/ChRRpXyfe5oZ8R3yBBOgsmgTE+3abYdXi9xfw16WtiMXi+r9jjviAV1slDflIAZ3u+22eweCKsppxIJ1K+JxY2KONf+qbgHAcCZqga fKM07qFL LHA8cvidWKCR/O8x4TvqwHe/pg7lYcCZd053oby7ANhu3KARV4SMDJYK4e6wStZmXPMP4+VMBJ5en7lSuRInoprYRMXAR+aClppXs+96RnMfFTDB4YagXfKUd2FwwmtfBDibLdHGnWx7MQwxSJSzVAmuXVzsNKCiA/VfhH9y1/x0XVMR1/COVaEWnlABAXtwztfPskevbc1Nd3te65HvGGmrQLHk77z8HpXGjz8L1zDqfe61oMs067XRtayTKeOtRrJ/MQgQvxgBkO32SwBMYJ+BOS4QgWe+VKuHjnqErQ04VArBhbFFLJmVWNneHeWj8/r899OXsB17UjQUt0HEXpJ9Gsg+CPrDN2xmvZomrfU5wPUWHmhFgovRe1z50EW5flFe3IuL09b+qI4ZlUh3NjGVOJGNyz+FumcA0w+4ouAk5/9bzSZ6feukPR4rKZISA+nF9gYaACVjOCKfx8fijL7hEnrAtEZl59/oVDhsW3lPrkstb326FScIrxTY43MrqOlsq5P12KArNenD27UgASJqzDyKCJJFJyB1KyCPaebhuX59sXj9Ay/6HuQ59EcFra9zbzJm3t0E9o2SahLIIH/EqIhaCW+gQUjAr631hu4ITZs1VpHmC1EPs2k/CrHxyosaHtU7kaGc9w2dDtLzGqUSLYWzBa/VYPZgJ8UT3V6uxZRKGGriPIJMfbQW5ahJYi1L7xU7zoGjDBSMNIAeAsJDwbw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The basic idea here is to "simulate" memory poisoning for VMs. A VM running on some host might encounter a memory error, after which some page(s) are poisoned (i.e., future accesses SIGBUS). They expect that once poisoned, pages can never become "un-poisoned". So, when we live migrate the VM, we need to preserve the poisoned status of these pages. When live migrating, we try to get the guest running on its new host as quickly as possible. So, we start it running before all memory has been copied, and before we're certain which pages should be poisoned or not. So the basic way to use this new feature is: - On the new host, the guest's memory is registered with userfaultfd, in either MISSING or MINOR mode (doesn't really matter for this purpose). - On any first access, we get a userfaultfd event. At this point we can communicate with the old host to find out if the page was poisoned. - If so, we can respond with a UFFDIO_POISON - this places a swap marker so any future accesses will SIGBUS. Because the pte is now "present", future accesses won't generate more userfaultfd events, they'll just SIGBUS directly. UFFDIO_POISON does not handle unmapping previously-present PTEs. This isn't needed, because during live migration we want to intercept all accesses with userfaultfd (not just writes, so WP mode isn't useful for this). So whether minor or missing mode is being used (or both), the PTE won't be present in any case, so handling that case isn't needed. Similarly, UFFDIO_POISON won't replace existing PTE markers. This might be okay to do, but it seems to be safer to just refuse to overwrite any existing entry (like a UFFD_WP PTE marker). Signed-off-by: Axel Rasmussen Acked-by: Peter Xu --- fs/userfaultfd.c | 58 ++++++++++++++++++++++++++++++++ include/linux/userfaultfd_k.h | 4 +++ include/uapi/linux/userfaultfd.h | 16 +++++++++ mm/userfaultfd.c | 48 +++++++++++++++++++++++++- 4 files changed, 125 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 2e84684c46f0..53a7220c4679 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1956,6 +1956,61 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg) return ret; } +static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long arg) +{ + __s64 ret; + struct uffdio_poison uffdio_poison; + struct uffdio_poison __user *user_uffdio_poison; + struct userfaultfd_wake_range range; + + user_uffdio_poison = (struct uffdio_poison __user *)arg; + + ret = -EAGAIN; + if (atomic_read(&ctx->mmap_changing)) + goto out; + + ret = -EFAULT; + if (copy_from_user(&uffdio_poison, user_uffdio_poison, + /* don't copy the output fields */ + sizeof(uffdio_poison) - (sizeof(__s64)))) + goto out; + + ret = validate_range(ctx->mm, uffdio_poison.range.start, + uffdio_poison.range.len); + if (ret) + goto out; + + ret = -EINVAL; + if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE) + goto out; + + if (mmget_not_zero(ctx->mm)) { + ret = mfill_atomic_poison(ctx->mm, uffdio_poison.range.start, + uffdio_poison.range.len, + &ctx->mmap_changing, 0); + mmput(ctx->mm); + } else { + return -ESRCH; + } + + if (unlikely(put_user(ret, &user_uffdio_poison->updated))) + return -EFAULT; + if (ret < 0) + goto out; + + /* len == 0 would wake all */ + BUG_ON(!ret); + range.len = ret; + if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) { + range.start = uffdio_poison.range.start; + wake_userfault(ctx, &range); + } + ret = range.len == uffdio_poison.range.len ? 0 : -EAGAIN; + +out: + return ret; +} + static inline unsigned int uffd_ctx_features(__u64 user_features) { /* @@ -2057,6 +2112,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, case UFFDIO_CONTINUE: ret = userfaultfd_continue(ctx, arg); break; + case UFFDIO_POISON: + ret = userfaultfd_poison(ctx, arg); + break; } return ret; } diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index ac7b0c96d351..ac8c6854097c 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -46,6 +46,7 @@ enum mfill_atomic_mode { MFILL_ATOMIC_COPY, MFILL_ATOMIC_ZEROPAGE, MFILL_ATOMIC_CONTINUE, + MFILL_ATOMIC_POISON, NR_MFILL_ATOMIC_MODES, }; @@ -83,6 +84,9 @@ extern ssize_t mfill_atomic_zeropage(struct mm_struct *dst_mm, extern ssize_t mfill_atomic_continue(struct mm_struct *dst_mm, unsigned long dst_start, unsigned long len, atomic_t *mmap_changing, uffd_flags_t flags); +extern ssize_t mfill_atomic_poison(struct mm_struct *dst_mm, unsigned long start, + unsigned long len, atomic_t *mmap_changing, + uffd_flags_t flags); extern int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, unsigned long len, bool enable_wp, atomic_t *mmap_changing); diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 66dd4cd277bd..b5f07eacc697 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -71,6 +71,7 @@ #define _UFFDIO_ZEROPAGE (0x04) #define _UFFDIO_WRITEPROTECT (0x06) #define _UFFDIO_CONTINUE (0x07) +#define _UFFDIO_POISON (0x08) #define _UFFDIO_API (0x3F) /* userfaultfd ioctl ids */ @@ -91,6 +92,8 @@ struct uffdio_writeprotect) #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ struct uffdio_continue) +#define UFFDIO_POISON _IOWR(UFFDIO, _UFFDIO_POISON, \ + struct uffdio_poison) /* read() structure */ struct uffd_msg { @@ -225,6 +228,7 @@ struct uffdio_api { #define UFFD_FEATURE_EXACT_ADDRESS (1<<11) #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) +#define UFFD_FEATURE_POISON (1<<14) __u64 features; __u64 ioctls; @@ -321,6 +325,18 @@ struct uffdio_continue { __s64 mapped; }; +struct uffdio_poison { + struct uffdio_range range; +#define UFFDIO_POISON_MODE_DONTWAKE ((__u64)1<<0) + __u64 mode; + + /* + * Fields below here are written by the ioctl and must be at the end: + * the copy_from_user will not read past here. + */ + __s64 updated; +}; + /* * Flags for the userfaultfd(2) system call itself. */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 4244ca7ee903..899aa621d7c1 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -288,6 +288,40 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, goto out; } +/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ +static int mfill_atomic_pte_poison(pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, + uffd_flags_t flags) +{ + int ret; + struct mm_struct *dst_mm = dst_vma->vm_mm; + pte_t _dst_pte, *dst_pte; + spinlock_t *ptl; + + _dst_pte = make_pte_marker(PTE_MARKER_ERROR); + dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); + + if (mfill_file_over_size(dst_vma, dst_addr)) { + ret = -EFAULT; + goto out_unlock; + } + + ret = -EEXIST; + /* Refuse to overwrite any PTE, even a PTE marker (e.g. UFFD WP). */ + if (!pte_none(*dst_pte)) + goto out_unlock; + + set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(dst_vma, dst_addr, dst_pte); + ret = 0; +out_unlock: + pte_unmap_unlock(dst_pte, ptl); + return ret; +} + static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address) { pgd_t *pgd; @@ -339,7 +373,8 @@ static __always_inline ssize_t mfill_atomic_hugetlb( * by THP. Since we can not reliably insert a zero page, this * feature is not supported. */ - if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) { + if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE) || + uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) { mmap_read_unlock(dst_mm); return -EINVAL; } @@ -483,6 +518,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) { return mfill_atomic_pte_continue(dst_pmd, dst_vma, dst_addr, flags); + } else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) { + return mfill_atomic_pte_poison(dst_pmd, dst_vma, + dst_addr, flags); } /* @@ -704,6 +742,14 @@ ssize_t mfill_atomic_continue(struct mm_struct *dst_mm, unsigned long start, uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE)); } +ssize_t mfill_atomic_poison(struct mm_struct *dst_mm, unsigned long start, + unsigned long len, atomic_t *mmap_changing, + uffd_flags_t flags) +{ + return mfill_atomic(dst_mm, start, 0, len, mmap_changing, + uffd_flags_set_mode(flags, MFILL_ATOMIC_POISON)); +} + long uffd_wp_range(struct vm_area_struct *dst_vma, unsigned long start, unsigned long len, bool enable_wp) {