From patchwork Tue Nov 21 17:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13463397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E090C61D85 for ; Tue, 21 Nov 2023 17:17:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D23B66B04CD; Tue, 21 Nov 2023 12:17:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CB7966B04CE; Tue, 21 Nov 2023 12:17:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB1C86B04CF; Tue, 21 Nov 2023 12:17:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 956EF6B04CD for ; Tue, 21 Nov 2023 12:17:01 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6BA1CC0A28 for ; Tue, 21 Nov 2023 17:17:01 +0000 (UTC) X-FDA: 81482616642.09.5B69E65 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 7F6B11C001C for ; Tue, 21 Nov 2023 17:16:59 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nZ7U3VsJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3CuZcZQYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CuZcZQYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700587019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p+YU4xakrSZAEeiCshOnJPmfZHIUA9DyOQyL2tdclCw=; b=o4D3yGF2LiZBI5cDvV2YpLNCUuBS1XW4l+jNA2qZL1p7WVB1T/VelWMZWRUObLb4j1KTFN IzJZV3PPLGhsaclcLRkdCouR7+l/wRtx8F5zDUsMXHtPlz3aWV6k0VC71nnm3RIoeBiX/7 JwvA5IJKg9/lLtOF0O5xdm8wy4AeVls= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nZ7U3VsJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3CuZcZQYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CuZcZQYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700587019; a=rsa-sha256; cv=none; b=A2vR1acqkY5LhPM4yUe88r/3m0rZzPzqFnol0yY09mESqhwXpwAGwUBDXMd2I4KLLvWWx2 Mhz5Z9SmSoTSzMLR/PvJ42Kkd17UMRbI9OwJy8vPhsG+M0HYUkpqzQ449udTqNLQ5JKLb2 fb+J8vA+TT2OSJXMJAWRdpsFpZB0Igg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-da3dd6a72a7so7370947276.0 for ; Tue, 21 Nov 2023 09:16:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700587018; x=1701191818; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p+YU4xakrSZAEeiCshOnJPmfZHIUA9DyOQyL2tdclCw=; b=nZ7U3VsJCXJVEod0WsQANYbyCpXud+Kdo3BEbPD8k7ldA1S/O3CqIaxjB0LWHKS/p6 wWi7wPUdhgU/I47MKw8qlxOV3LBcKBLdibiqMdN7Ewo9HLO+dYKHzQspTTowPVLYPqal 98hb2PYzJN4tzMIY2ukRLr6PCi0fFQHQgBGIjh+CxQy9qLfHZ1zE5VvLEcrX74XHqD3F /Ke7nPWTLZSX6hpoNJyiaGKmX+UUIM/xKFZwNPlaQ6C7gyMhtaZmYo0OmnjOi/xzPC9p D5kIVMDDYlJY0pkJsxI+OF+cI9qmJ2w0xQjVq89D5xmla/E7AnS8yY2T1pp2PPKd4qy+ 1jVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700587018; x=1701191818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p+YU4xakrSZAEeiCshOnJPmfZHIUA9DyOQyL2tdclCw=; b=Cgy7BaPI6SSX2SK/DAS7DgL7nzsujrjv8eTpAvEUKHT8rxgs8laRwtQaoVf978kJ5D ztvCEHF18hh2AtLh3+K9hcq0LbYg6KZQ9/qYfPCCshub+uxMfoxPUqUkVN2xkaHJXNbl wt0Qnnx+fhPEQUrqRRp//vw4Xuxb6NB9GnZXb2IZy40Ja7cMWpyHd8+L3fdnDJsxRlm2 Phj7xNAig6WCIb1ijTGszolTs1WSrhIzLOzISW1WQWBecu2xtr3xGNbKlphDTaneuEjZ 33R1EbhHDL4qtwmzec8UilFW4+LHUvVixjV6aOdw2nS06IBs81sgi75bzl0V+TYBjbQh ufaw== X-Gm-Message-State: AOJu0YwB94YNNnR4s3nnE0MUa/g8jfgTdg+bmKK3GtQHw6rVM0nf8fcD GUmK+KtSPekJUEEXrn5RuOpbCV/yTGs= X-Google-Smtp-Source: AGHT+IGnjdfTVee3DSZlNOy3KhbC3j8nIypXTk1N0NzS6E/A59+6kLC/Rjntyn5b7LkHCGkdnVF0AOMslLI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:2045:f6d2:f01d:3fff]) (user=surenb job=sendgmr) by 2002:a25:b84:0:b0:d9a:c27e:5f37 with SMTP id 126-20020a250b84000000b00d9ac27e5f37mr249877ybl.3.1700587018586; Tue, 21 Nov 2023 09:16:58 -0800 (PST) Date: Tue, 21 Nov 2023 09:16:38 -0800 In-Reply-To: <20231121171643.3719880-1-surenb@google.com> Mime-Version: 1.0 References: <20231121171643.3719880-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231121171643.3719880-6-surenb@google.com> Subject: [PATCH v5 5/5] selftests/mm: add UFFDIO_MOVE ioctl test From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, shuah@kernel.org, aarcange@redhat.com, lokeshgidra@google.com, peterx@redhat.com, david@redhat.com, hughd@google.com, mhocko@suse.com, axelrasmussen@google.com, rppt@kernel.org, willy@infradead.org, Liam.Howlett@oracle.com, jannh@google.com, zhangpeng362@huawei.com, bgeffon@google.com, kaleshsingh@google.com, ngeoffray@google.com, jdduke@google.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@android.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7F6B11C001C X-Stat-Signature: yb9ru3p8pfdmimrtrmuuxtjcfxxyb7rt X-HE-Tag: 1700587019-216480 X-HE-Meta: U2FsdGVkX1+JCqZ1Tu3wGEBRx42Z9XJ8R9lfC79RK5PD9NSEdZgmYrg8czJD7l79ChhAzMCKSwFpTDfU8wo+P8IgM4MthftD9MySv2THqZZ35kCRpnBO9GDNbdB7FSsLgtXzU8fINcq9ZNHeUezbjCwq6k/dya3iBVZrwa9kEmDr9eVar+vddG2Qu/JabpfRTgFG/RJQE2pWQDBqY15bhoVj0wID04OI3IFBNjT8OTYfiry/iegxOaTfVhvKlBLyCCr/ruCo/LX29NZP0d9AI7CWLfycdtoinAchneJZHCC5chz81FQUGLh0MGQorPbg4kyMDWCZOLNsdvCvI+KlA2+qV//tA5zO6Ik9STily1vPv36OmyYtWSxtWPXrc2D/N6PydHL0ChgNIynyXExruomy6iaOVzfrZ/hgvAoyvV0bRuGk6t18UhJgUkF2r4lDZ0IAF0pw0K0BsssLcolw16vRfTUR2waHZh8snOzlx0XbFHPS3XaVLvHrEEFHDiRF55nbO0oQT2zhFGfUDJ9stOoQ+V/zdw4AX4jwAF8QdnJhz+u5uc2he224uc2Y78mIb92NqF79Eo+h6zF488tqWCbJY9iha/C2dZwEhllIM9pyexNR3JSqAxBhfLtSVIqGYICpKGKFUcVaaq3R7csC9s6fHt3SCQk4MAUiYIBqs+B7Z5/zANXZtAfdNpDWo236CIiJW4YuN1eYgDCbNlC+4AJRn8DtjoUydFknDTBJjFVZgDWA0L9a9Guoqzjq2Wq3gn0U13ktzVI7KhfYBzNy4vduT53yX5exI8Dac/l1kboUxFNUhrwA1po9MquvNAPZbekpnxUflyZQE6TuVv7qrS0RjI0ZhgVw0TGNftienLIuiTqEwt6d2vt1q64c7akgO5i5nmGvWm/HYX/AEsxYdFjEX0uXLJT9io5g2ekg/3Dw3Sc3/c5x4ZhRV4kbGlSL/5Flctrby/EBM5WaVvr j/Bz7Ok9 wf/EYMpreDK5xZG4oMxGMXxgn5r+CD3CccoKya9/H4IdllvJa55v4J6IQFPTH1TOGj5rw+Q5/U7CvZVz85LufaNRKvLq341g/A1vwK3dupx8iDGASecI5XbDszYRxVnfDTEKo/PqQZZiqxkmwt3fl4pQq5TIlP3oPw+glB85c0ETEknCZLGazynod+n4Eo7JyBt+03WiBxC+jqk3/gHf8Jz113RlqRq77vblVy+9z2v+KB09GFw7SLG7gMVswPyAytUW5hgdAQEAX/rlGesjwBwy9RPw399Za1K35MTdHD5Y04JRQ2OIfX0wjj4Bh2TssItCODvTKeFoQqVD6GsLK7Ve8ywK3Y9IihRm6oELcDWsQ6F64DkpWXWk9PXXS3Jp36s7MzAM+RMinyr2OH8C+7pFqReGft0UvggPLWrF5HGlazEk8M0YQZEda2atMB5sGGtMUip1yE6q2eg2yh53b3hIQr5cmgrn2RVCrzpqRiuF06qBCNHE2hY+bXJambXTw8uJgoEOkQk5wpYPYSFTTF/5xm2fZNTGMfxBwQekMgP57qAEViAfsI3x5pWMhUD0L4OjBBTHVYHA9a7JzRYSgLBt2rYnagb1hbFZXzav5sTZCJtxbZ+suKSu+k3ddaoc0++C/z1OhFBJr61blEz7dGaRBDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add tests for new UFFDIO_MOVE ioctl which uses uffd to move source into destination buffer while checking the contents of both after the move. After the operation the content of the destination buffer should match the original source buffer's content while the source buffer should be zeroed. Separate tests are designed for PMD aligned and unaligned cases because they utilize different code paths in the kernel. Signed-off-by: Suren Baghdasaryan --- tools/testing/selftests/mm/uffd-common.c | 24 +++ tools/testing/selftests/mm/uffd-common.h | 1 + tools/testing/selftests/mm/uffd-unit-tests.c | 189 +++++++++++++++++++ 3 files changed, 214 insertions(+) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index fb3bbc77fd00..b0ac0ec2356d 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -631,6 +631,30 @@ int copy_page(int ufd, unsigned long offset, bool wp) return __copy_page(ufd, offset, false, wp); } +int move_page(int ufd, unsigned long offset, unsigned long len) +{ + struct uffdio_move uffdio_move; + + if (offset + len > nr_pages * page_size) + err("unexpected offset %lu and length %lu\n", offset, len); + uffdio_move.dst = (unsigned long) area_dst + offset; + uffdio_move.src = (unsigned long) area_src + offset; + uffdio_move.len = len; + uffdio_move.mode = UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; + uffdio_move.move = 0; + if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + /* real retval in uffdio_move.move */ + if (uffdio_move.move != -EEXIST) + err("UFFDIO_MOVE error: %"PRId64, + (int64_t)uffdio_move.move); + wake_range(ufd, uffdio_move.dst, len); + } else if (uffdio_move.move != len) { + err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); + } else + return 1; + return 0; +} + int uffd_open_dev(unsigned int flags) { int fd, uffd; diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selftests/mm/uffd-common.h index 774595ee629e..cb055282c89c 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -119,6 +119,7 @@ void wp_range(int ufd, __u64 start, __u64 len, bool wp); void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); int copy_page(int ufd, unsigned long offset, bool wp); +int move_page(int ufd, unsigned long offset, unsigned long len); void *uffd_poll_thread(void *arg); int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c index debc423bdbf4..e4e271511db9 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -23,6 +23,9 @@ #define MEM_ALL (MEM_ANON | MEM_SHMEM | MEM_SHMEM_PRIVATE | \ MEM_HUGETLB | MEM_HUGETLB_PRIVATE) +#define ALIGN_UP(x, align_to) \ + ((__typeof__(x))((((unsigned long)(x)) + ((align_to)-1)) & ~((align_to)-1))) + struct mem_type { const char *name; unsigned int mem_flag; @@ -1064,6 +1067,178 @@ static void uffd_poison_test(uffd_test_args_t *targs) uffd_test_pass(); } +static void +uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, + unsigned long len) +{ + unsigned long offset; + + if (msg->event != UFFD_EVENT_PAGEFAULT) + err("unexpected msg event %u", msg->event); + + if (msg->arg.pagefault.flags & + (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_FLAG_WRITE)) + err("unexpected fault type %llu", msg->arg.pagefault.flags); + + offset = (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset &= ~(len-1); + + if (move_page(uffd, offset, len)) + args->missing_faults++; +} + +static void uffd_move_handle_fault(struct uffd_msg *msg, + struct uffd_args *args) +{ + uffd_move_handle_fault_common(msg, args, page_size); +} + +static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, + struct uffd_args *args) +{ + uffd_move_handle_fault_common(msg, args, default_huge_page_size()); +} + +static void +uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, + void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +{ + unsigned long nr; + pthread_t uffd_mon; + char c; + unsigned long long count; + struct uffd_args args = { 0 }; + char *orig_area_src, *orig_area_dst; + unsigned long step_size, step_count; + unsigned long src_offs = 0; + unsigned long dst_offs = 0; + + /* Prevent source pages from being mapped more than once */ + if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + err("madvise(MADV_DONTFORK) failure"); + + if (uffd_register(uffd, area_dst, nr_pages * page_size, + true, false, false)) + err("register failure"); + + args.handle_fault = handle_fault; + if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) + err("uffd_poll_thread create"); + + step_size = chunk_size / page_size; + step_count = nr_pages / step_size; + + if (step_size > page_size) { + char *aligned_src = ALIGN_UP(area_src, chunk_size); + char *aligned_dst = ALIGN_UP(area_dst, chunk_size); + + if (aligned_src != area_src || aligned_dst != area_dst) { + src_offs = (aligned_src - area_src) / page_size; + dst_offs = (aligned_dst - area_dst) / page_size; + step_count--; + } + orig_area_src = area_src; + orig_area_dst = area_dst; + area_src = aligned_src; + area_dst = aligned_dst; + } + + /* + * Read each of the pages back using the UFFD-registered mapping. We + * expect that the first time we touch a page, it will result in a missing + * fault. uffd_poll_thread will resolve the fault by moving source + * page to destination. + */ + for (nr = 0; nr < step_count * step_size; nr += step_size) { + unsigned long i; + + /* Check area_src content */ + for (i = 0; i < step_size; i++) { + count = *area_count(area_src, nr + i); + if (count != count_verify[src_offs + nr + i]) + err("nr %lu source memory invalid %llu %llu\n", + nr + i, count, count_verify[src_offs + nr + i]); + } + + /* Faulting into area_dst should move the page or the huge page */ + for (i = 0; i < step_size; i++) { + count = *area_count(area_dst, nr + i); + if (count != count_verify[dst_offs + nr + i]) + err("nr %lu memory corruption %llu %llu\n", + nr, count, count_verify[dst_offs + nr + i]); + } + + /* Re-check area_src content which should be empty */ + for (i = 0; i < step_size; i++) { + count = *area_count(area_src, nr + i); + if (count != 0) + err("nr %lu move failed %llu %llu\n", + nr, count, count_verify[src_offs + nr + i]); + } + } + if (step_size > page_size) { + area_src = orig_area_src; + area_dst = orig_area_dst; + } + + if (write(pipefd[1], &c, sizeof(c)) != sizeof(c)) + err("pipe write"); + if (pthread_join(uffd_mon, NULL)) + err("join() failed"); + + if (args.missing_faults != step_count || args.minor_faults != 0) + uffd_test_fail("stats check error"); + else + uffd_test_pass(); +} + +static void uffd_move_test(uffd_test_args_t *targs) +{ + uffd_move_test_common(targs, page_size, uffd_move_handle_fault); +} + +static void uffd_move_pmd_test(uffd_test_args_t *targs) +{ + uffd_move_test_common(targs, default_huge_page_size(), + uffd_move_pmd_handle_fault); +} + +static int prevent_hugepages(const char **errmsg) +{ + /* This should be done before source area is populated */ + if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=n */ + if (errno != EINVAL) { + if (errmsg) + *errmsg = "madvise(MADV_NOHUGEPAGE) failed"; + return -errno; + } + } + return 0; +} + +static int request_hugepages(const char **errmsg) +{ + /* This should be done before source area is populated */ + if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (errmsg) { + *errmsg = (errno == EINVAL) ? + "CONFIG_TRANSPARENT_HUGEPAGE is not set" : + "madvise(MADV_HUGEPAGE) failed"; + } + return -errno; + } + return 0; +} + +struct uffd_test_case_ops uffd_move_test_case_ops = { + .post_alloc = prevent_hugepages, +}; + +struct uffd_test_case_ops uffd_move_test_pmd_case_ops = { + .post_alloc = request_hugepages, +}; + /* * Test the returned uffdio_register.ioctls with different register modes. * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. @@ -1141,6 +1316,20 @@ uffd_test_case_t uffd_tests[] = { .mem_targets = MEM_ALL, .uffd_feature_required = 0, }, + { + .name = "move", + .uffd_fn = uffd_move_test, + .mem_targets = MEM_ANON, + .uffd_feature_required = UFFD_FEATURE_MOVE, + .test_case_ops = &uffd_move_test_case_ops, + }, + { + .name = "move-pmd", + .uffd_fn = uffd_move_pmd_test, + .mem_targets = MEM_ANON, + .uffd_feature_required = UFFD_FEATURE_MOVE, + .test_case_ops = &uffd_move_test_pmd_case_ops, + }, { .name = "wp-fork", .uffd_fn = uffd_wp_fork_test,