From patchwork Wed Sep 30 22:21:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 11810405 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CD1D6CA for ; Wed, 30 Sep 2020 22:22:06 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 130D220719 for ; Wed, 30 Sep 2020 22:22:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="EZCDNSH7"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="P+s7piSR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 130D220719 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=IcA9F6T4Y7T2Tu5sE3w5ctSlsFaJNhAnTaFt+82NoRs=; b=EZCDNSH78mTZSoaFOtfoTCkEK KpTwT/9r4vL8CB/VTmllkjS92ZZKmH3Plyk1+WbLLnmq11O/YySDbeB1oW8mperDeDgnVmA50ldGt B/6u5ReLtu7gKM88WIewJ784vxKRag68s5W5MR5ZQsSZVIOVysaPGF2iEenInQzRBpeF/eGb553TF 4YRBLOQtXeb9kgzTyJ0DBSb7g7hC2e1k5UApV57NZ1ToLHDnf10DbtaFxtRBMlgwsdO6vJsMT2lpG 78L+eYWZqx35dAkP0NVRuRjUbBwuNcoVJCO44d0pBn8C2G4I7ScAtAiwQCk8RZ5C/0BIF7uEXkTsd rh+mqG/bg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkTx-00069N-19; Wed, 30 Sep 2020 22:21:53 +0000 Received: from mail-qt1-x849.google.com ([2607:f8b0:4864:20::849]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkTt-00067M-GH for linux-arm-kernel@lists.infradead.org; Wed, 30 Sep 2020 22:21:50 +0000 Received: by mail-qt1-x849.google.com with SMTP id o13so2248320qtl.6 for ; Wed, 30 Sep 2020 15:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:cc; bh=XbvI12WaHJN29uXJhtZ+EOB9wipW/gZZ4hVhoy3A3pk=; b=P+s7piSRTOYlMasrhJ2oHEQPHoelb8iyusd1TAd6av+DBZjV3mCsezsGb0aM8Cbq8F wcWniTz7nt0cgEMZzRfuoQ9pfRQazas3/PsqYjDs/NBqewHVQbn3A9FR3YF5x8RLQ+yX jcg16G459PC61aytMKkZj8myAxNaZIkOeqF5C6gBGdEB4NIfzn9r9+I4XIjXfWBQTRL9 nS21h4UueeuQ2VbqCnuA3kMG+4TjoDJl/gd2FxnZTiVHWLNyDBQ+8gxZZrGH2WQCMMeD XR49bF8yCpSjBarMyiS0KGX73IcaryYShv5dSNVrdWlHauwdqr5/O7le3VC/vIKhaCKe Hxzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=XbvI12WaHJN29uXJhtZ+EOB9wipW/gZZ4hVhoy3A3pk=; b=OVJBbUG66pXhPkt/tm6Q80YC05FGMLXQH41T0WunvhyOJQr2z0FCgQNZ4BuYj52cOx PUfh1sGFDd1bYbX5bHLiF4goPDUbnoEre+cRRQIspnRa9uKPZwepQwtt2Ar6swao5XTK YgZ8VkC1uWxORWKN9+hB7WJv4k2GhoEmNptld+SPrvJ+KinEgnaOqCoIxOvv55++BB8g 64j9s6wpf2z1n3Jwf7GeWNb+wtzRR+Na/b1I6Q7MjXUeBvdnoPixD5C7FYFJU4FAsYy8 jY7jakPoDozKci+ACozs5JLqQNSL/K/v165x75O/GkZNY0XQIJxtxbSL/Jcj1qVhmYQX 7l7A== X-Gm-Message-State: AOAM531CeAPj8C0hPBKzrHH7gHd9HMk2GSNzO9xe+CN5Xz0n1XfAL5tA lPYiNYgJ7gIUWiSzjqzN5c2BfJNoi+rn9HI9Qw== X-Google-Smtp-Source: ABdhPJwwSEexZldCHo9VW8lyUDwdXeRxfa1mYE1k7wqr1xRB7FpnAJwFMYg556it3BiSNrrerzkgakSQGznEqtJ0mQ== X-Received: from kaleshsingh.c.googlers.com ([fda3:e722:ac3:10:14:4d90:c0a8:2145]) (user=kaleshsingh job=sendgmr) by 2002:a0c:9d03:: with SMTP id m3mr4759136qvf.54.1601504506102; Wed, 30 Sep 2020 15:21:46 -0700 (PDT) Date: Wed, 30 Sep 2020 22:21:18 +0000 In-Reply-To: <20200930222130.4175584-1-kaleshsingh@google.com> Message-Id: <20200930222130.4175584-2-kaleshsingh@google.com> Mime-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH 1/5] kselftests: vm: Add mremap tests From: Kalesh Singh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200930_182149_557921_9C9697A7 X-CRM114-Status: GOOD ( 30.11 ) X-Spam-Score: -7.0 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:849 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 1.2 MISSING_HEADERS Missing To: header 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: joelaf@google.com, Mark Rutland , Gavin Shan , Brian Geffon , Peter Zijlstra , Catalin Marinas , kaleshsingh@google.com, Ram Pai , Dave Hansen , Will Deacon , lokeshgidra@google.com, linux-kselftest@vger.kernel.org, "H. Peter Anvin" , Christian Brauner , Shuah Khan , Mina Almasry , Jia He , Arnd Bergmann , "Aneesh Kumar K.V" , Masahiro Yamada , x86@kernel.org, Krzysztof Kozlowski , Jason Gunthorpe , Ingo Molnar , Sami Tolvanen , Ira Weiny , kernel-team@android.com, Dave Martin , Hassan Naveed , Masami Hiramatsu , Ralph Campbell , Kees Cook , minchan@google.com, Zhenyu Ye , John Hubbard , Frederic Weisbecker , Borislav Petkov , Thomas Gleixner , Zi Yan , surenb@google.com, linux-arm-kernel@lists.infradead.org, SeongJae Park , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Andrew Morton , Mike Rapoport , Sandipan Das Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Test mremap on regions of various sizes and alignments and validate data after remapping. Also provide total time for remapping the region which is useful for performance comparison of the mremap optimizations that move pages at the PMD/PUD levels if HAVE_MOVE_PMD and/or HAVE_MOVE_PUD are enabled. Signed-off-by: Kalesh Singh --- tools/testing/selftests/vm/.gitignore | 1 + tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/mremap_test.c | 243 +++++++++++++++++++++++ tools/testing/selftests/vm/run_vmtests | 11 + 4 files changed, 256 insertions(+) create mode 100644 tools/testing/selftests/vm/mremap_test.c diff --git a/tools/testing/selftests/vm/.gitignore b/tools/testing/selftests/vm/.gitignore index 849e8226395a..b3a183c36cb5 100644 --- a/tools/testing/selftests/vm/.gitignore +++ b/tools/testing/selftests/vm/.gitignore @@ -8,6 +8,7 @@ thuge-gen compaction_test mlock2-tests mremap_dontunmap +mremap_test on-fault-limit transhuge-stress protection_keys diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index a9026706d597..f044808b45fa 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -16,6 +16,7 @@ TEST_GEN_FILES += map_populate TEST_GEN_FILES += mlock-random-test TEST_GEN_FILES += mlock2-tests TEST_GEN_FILES += mremap_dontunmap +TEST_GEN_FILES += mremap_test TEST_GEN_FILES += on-fault-limit TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress diff --git a/tools/testing/selftests/vm/mremap_test.c b/tools/testing/selftests/vm/mremap_test.c new file mode 100644 index 000000000000..09dc9a1ef81f --- /dev/null +++ b/tools/testing/selftests/vm/mremap_test.c @@ -0,0 +1,243 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Google LLC + */ +#define _GNU_SOURCE + +#include +#include +#include +#include + +#include "../kselftest.h" + +#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) + +struct config { + unsigned long long src_alignment; + unsigned long long dest_alignment; + unsigned long long region_size; + int overlapping; +}; + +struct test { + const char *name; + struct config config; + int expect_failure; +}; + +enum { + _1KB = 1ULL << 10, /* 1KB -> not page aligned */ + _4KB = 4ULL << 10, + _8KB = 8ULL << 10, + _1MB = 1ULL << 20, + _2MB = 2ULL << 20, + _4MB = 4ULL << 20, + _1GB = 1ULL << 30, + _2GB = 2ULL << 30, + PTE = _4KB, + PMD = _2MB, + PUD = _1GB, +}; + +#define MAKE_TEST(source_align, destination_align, size, \ + overlaps, should_fail, test_name) \ +{ \ + .name = test_name, \ + .config = { \ + .src_alignment = source_align, \ + .dest_alignment = destination_align, \ + .region_size = size, \ + .overlapping = overlaps, \ + }, \ + .expect_failure = should_fail \ +} + +#define MAKE_SIMPLE_TEST(source_align, destination_align, size) \ + MAKE_TEST(source_align, destination_align, size, 0, 0, \ + #size " mremap - Source " #source_align \ + " aligned, Destination " #destination_align \ + " aligned") + +/* + * Returns the start address of the mapping on success, else returns + * NULL on failure. + */ +static void *get_source_mapping(struct config c) +{ + unsigned long long addr = 0ULL; + void *src_addr = NULL; +retry: + addr += c.src_alignment; + src_addr = mmap((void *) addr, c.region_size, PROT_READ | PROT_WRITE, + MAP_FIXED | MAP_ANONYMOUS | MAP_SHARED, -1, 0); + if (src_addr == MAP_FAILED) { + if (errno == EPERM) + goto retry; + goto error; + } + /* + * Check that the address is aligned to the specified alignment. Addresses + * which have alignments that are multiples of that specified are not considered + * valid. For instance, 1GB address is 2MB-aligned, however it will not be + * considered valid for a requested alignment of 2MB. This is done to + * reduce coincidental alignment in the tests. + */ + if (((unsigned long long) src_addr & (c.src_alignment - 1)) || + !((unsigned long long) src_addr & c.src_alignment)) + goto retry; + + if (!src_addr) + goto error; + + return src_addr; +error: + ksft_print_msg("Failed to map source region: %s\n", + strerror(errno)); + return NULL; +} + +/* Returns the time taken for the remap on success else returns -1. */ +static long long remap_region(struct config c) +{ + void *addr, *src_addr, *dest_addr; + int i, j; + struct timespec t_start = {0, 0}, t_end = {0, 0}; + long long start_ns, end_ns, align_mask, ret, offset; + char pattern[] = {0xa8, 0xcd, 0xfe}; + int pattern_size = ARRAY_SIZE(pattern); + + src_addr = get_source_mapping(c); + if (!src_addr) { + ret = -1; + goto out; + } + + /* Set byte pattern */ + for (i = 0; i < c.region_size; i++) { + for (j = 0; i+j < c.region_size && j < pattern_size; j++) + memset((char *) src_addr + i+j, pattern[j], 1); + i += pattern_size-1; + } + + align_mask = ~(c.dest_alignment - 1); + offset = (c.overlapping) ? -c.dest_alignment : c.dest_alignment; + addr = (void *) (((unsigned long long) src_addr + c.region_size + offset) + & align_mask); + + /* See comment in get_source_mapping() */ + if (!((unsigned long long) addr & c.dest_alignment)) + addr = (void *) ((unsigned long long) addr | c.dest_alignment); + + clock_gettime(CLOCK_MONOTONIC, &t_start); + dest_addr = mremap(src_addr, c.region_size, c.region_size, + MREMAP_MAYMOVE|MREMAP_FIXED, (char *) addr); + clock_gettime(CLOCK_MONOTONIC, &t_end); + + if (dest_addr == MAP_FAILED) { + ksft_print_msg("mremap failed: %s\n", strerror(errno)); + ret = -1; + goto clean_up_src; + } + + /* Verify byte pattern after remapping */ + for (i = 0; i < c.region_size; i++) { + for (j = 0; i+j < c.region_size && j < pattern_size; j++) { + if (((char *) dest_addr)[i+j] != (char) pattern[j]) { + ksft_print_msg("Data after remap doesn't match at offset %d\n", + i+j); + ksft_print_msg("Expected: %#x\t Got: %#x\n", pattern[j] & 0xff, + ((char *) dest_addr)[i+j] & 0xff); + ret = -1; + goto clean_up_dest; + } + } + i += pattern_size-1; + } + + start_ns = t_start.tv_sec * 1000000000ULL + t_start.tv_nsec; + end_ns = t_end.tv_sec * 1000000000ULL + t_end.tv_nsec; + ret = end_ns - start_ns; + +/* + * Since the destination address is specified using MREMAP_FIXED, subsequent mremap will unmap any + * previous mapping at the address range specified by dest_addr and region_size. This significantly + * affects the remap time of subsequent tests. So we clean up mappings after each test. + */ +clean_up_dest: + munmap(dest_addr, c.region_size); +clean_up_src: + munmap(src_addr, c.region_size); +out: + return ret; +} + +static void run_mremap_test_case(struct test test_case, int *failures) +{ + long long remap_time = remap_region(test_case.config); + + if (remap_time < 0) { + if (test_case.expect_failure) + ksft_test_result_pass("%s\n\tExpected mremap failure\n", test_case.name); + else { + ksft_test_result_fail("%s\n", test_case.name); + *failures += 1; + } + } else + ksft_test_result_pass("%s\n\tmremap time: %12lldns\n", test_case.name, remap_time); +} + +int main(int argc, char *argv[]) +{ + int failures = 0; + int i; + + struct test test_cases[] = { + /* Expected mremap failures */ + MAKE_TEST(_4KB, _4KB, _4KB, 1 /* overlaps */, 1 /* fails */, + "mremap - Source and Destination Regions Overlapping"), + MAKE_TEST(_4KB, _1KB, _4KB, 0 /* overlaps */, 1 /* fails */, + "mremap - Destination Address Misaligned (1KB-aligned)"), + MAKE_TEST(_1KB, _4KB, _4KB, 0 /* overlaps */, 1 /* fails */, + "mremap - Source Address Misaligned (1KB-aligned)"), + + /* Src addr PTE aligned */ + MAKE_SIMPLE_TEST(PTE, PTE, _8KB), + + /* Src addr 1MB aligned */ + MAKE_SIMPLE_TEST(_1MB, PTE, _2MB), + MAKE_SIMPLE_TEST(_1MB, _1MB, _2MB), + + /* Src addr PMD aligned */ + MAKE_SIMPLE_TEST(PMD, PTE, _4MB), + MAKE_SIMPLE_TEST(PMD, _1MB, _4MB), + MAKE_SIMPLE_TEST(PMD, PMD, _4MB), + + /* Src addr PUD aligned */ + MAKE_SIMPLE_TEST(PUD, PTE, _2GB), + MAKE_SIMPLE_TEST(PUD, _1MB, _2GB), + MAKE_SIMPLE_TEST(PUD, PMD, _2GB), + MAKE_SIMPLE_TEST(PUD, PUD, _2GB), + }; + + struct test perf_test_cases[] = { + /* mremap 1GB region - Page table level aligned time comparison */ + MAKE_SIMPLE_TEST(PTE, PTE, _1GB), + MAKE_SIMPLE_TEST(PMD, PMD, _1GB), + MAKE_SIMPLE_TEST(PUD, PUD, _1GB), + }; + + ksft_set_plan(ARRAY_SIZE(test_cases) + ARRAY_SIZE(perf_test_cases)); + + for (i = 0; i < ARRAY_SIZE(test_cases); i++) + run_mremap_test_case(test_cases[i], &failures); + + ksft_print_msg("\nmremap HAVE_MOVE_PMD/PUD optimization time comparison for 1GB region:\n"); + for (i = 0; i < ARRAY_SIZE(perf_test_cases); i++) + run_mremap_test_case(perf_test_cases[i], &failures); + + if (failures > 0) + ksft_exit_fail(); + else + ksft_exit_pass(); +} diff --git a/tools/testing/selftests/vm/run_vmtests b/tools/testing/selftests/vm/run_vmtests index a3f4f30f0a2e..d578ad831813 100755 --- a/tools/testing/selftests/vm/run_vmtests +++ b/tools/testing/selftests/vm/run_vmtests @@ -241,6 +241,17 @@ else echo "[PASS]" fi +echo "-------------------" +echo "running mremap_test" +echo "-------------------" +./mremap_test +if [ $? -ne 0 ]; then + echo "[FAIL]" + exitcode=1 +else + echo "[PASS]" +fi + echo "-----------------" echo "running thuge-gen" echo "-----------------" From patchwork Wed Sep 30 22:21:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 11810411 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 536B4112C for ; Wed, 30 Sep 2020 22:22:15 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 16AF42075F for ; Wed, 30 Sep 2020 22:22:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="orFWCMT9"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="boDpHG4Y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16AF42075F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=d2hEfCZg5hD5TB2/kKWMybFCCWqvJ2ake0IL3FKW3O4=; b=orFWCMT9c4qfzeu6GbkkBPHXO n0KgubKojtWtJkkT8TAOsEMxZ/QxeWZUe0Pewgd+oEBTaed2cQ/ZWAyPorw8bxOiiF9kQCCs/vpW2 58ISkL+nHwTNH228GDibZlM7OMU6Wug6aE7Vx8D6D9h0dUGx40mv50BEgMq4g/45X/E/Y1S2/gEIC CmXHcLOqDI8K29jDEC5HUaDs1cia42yIj279hbs9QeGTyDBx9imtUo3ofo7oVsSpgY5iQnaGpGaji 77hhFnelWaAK0vvtdo6tFy7jw8XcJDiF4IYBaTPXdpuL9rWgjS1BcVY0mlq4yQVJfYchCnss57cNK /Cw8HdkLQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkU8-0006DQ-6s; Wed, 30 Sep 2020 22:22:04 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkU5-0006AW-NY for linux-arm-kernel@lists.infradead.org; Wed, 30 Sep 2020 22:22:02 +0000 Received: by mail-yb1-xb49.google.com with SMTP id b8so3212638yba.10 for ; Wed, 30 Sep 2020 15:21:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:cc; bh=M0Jnd5wDs989jegufD80A647gOObA9k7fcoXOfGIBiI=; b=boDpHG4YKr/2sBYi+LXa7BYu2zahi8BWNiXF87MohJyyIman1YZgh2emp0eq+ALxVF N8Rorega2dCLqmupg5pggd4PEd0bioiDmLQNT/uayIfQYI/mK9sOXXXaXnkfAPLGDmJx /Co7dHpFXUZPb6JhgzEuOs4vsL7nB/ouZzo9v6V4300dN5vLptrbHO4vapViZXwBudAh Ek2t7vH58qT4ABFOsGeFw8dE9HSppt8Cv93fwPd7G3MmAMn3Em8oG/bsv6UWDPJ3jGbg m/Zl/8FLGMadTgwwz/0FCcE3Bg5lmumAkFDQr0+kGeF1/sPrtYOu+Lj4tH/9bYQCLPbI YrzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=M0Jnd5wDs989jegufD80A647gOObA9k7fcoXOfGIBiI=; b=mmv7I4hc0MZjvkrbutONRU8yN5XoFhj9jajYW1BYUxHrukelYjgpV/SyMlIXgTZpbL E80UWjCTI+SdIMAyKN+VdF8pTgdnhyaTW6pBQrYGOr8SpZis4XaXscicwd27ialLz4WG qZRBFlcjUj3XP7GQBS2u/3bXTtQQ+iKtFa0gEG6BOP3zITF18WbaQ13OZucVjbZshOR+ diGdAmZo/XeIHynWLIN1AQrcxQsKi1iR3iJuUQN5+nwm4uhx7sCKxRYM2igMMqY2o3Pn zgfPU8BVR01H/gJNx8TrE08g94XOSCO6JzWtfId1T4nbJrucMhMRXm2XJ2Vx3O4eTgTy kIhA== X-Gm-Message-State: AOAM530z2snOJeAWL/U+OUGfy8q5WSOCcFbHI63cnEHQ9mHY5+PnK7M4 GSWVfk5bN8GyTJKZhjGNPWSfXf2NgyKbZbF8TA== X-Google-Smtp-Source: ABdhPJzpG054ym1h9s82GZx0gEkFQHO8lOnhV52qK7oqfRRBlQ4XabfPxVWgnQguS08M76s/iC40L0YtMGRjB+Vw+Q== X-Received: from kaleshsingh.c.googlers.com ([fda3:e722:ac3:10:14:4d90:c0a8:2145]) (user=kaleshsingh job=sendgmr) by 2002:a25:ae9f:: with SMTP id b31mr6084827ybj.437.1601504516920; Wed, 30 Sep 2020 15:21:56 -0700 (PDT) Date: Wed, 30 Sep 2020 22:21:19 +0000 In-Reply-To: <20200930222130.4175584-1-kaleshsingh@google.com> Message-Id: <20200930222130.4175584-3-kaleshsingh@google.com> Mime-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH 2/5] arm64: mremap speedup - Enable HAVE_MOVE_PMD From: Kalesh Singh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200930_182201_842580_8790E116 X-CRM114-Status: GOOD ( 18.00 ) X-Spam-Score: -7.0 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:b49 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 1.2 MISSING_HEADERS Missing To: header 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: joelaf@google.com, Mark Rutland , Gavin Shan , Brian Geffon , Peter Zijlstra , Catalin Marinas , kaleshsingh@google.com, Ram Pai , Dave Hansen , Will Deacon , lokeshgidra@google.com, linux-kselftest@vger.kernel.org, "H. Peter Anvin" , Christian Brauner , Shuah Khan , Mina Almasry , Jia He , Arnd Bergmann , "Aneesh Kumar K.V" , Masahiro Yamada , x86@kernel.org, Krzysztof Kozlowski , Ingo Molnar , Sami Tolvanen , kernel-team@android.com, Hassan Naveed , Masami Hiramatsu , Ralph Campbell , Kees Cook , minchan@google.com, Zhenyu Ye , John Hubbard , Frederic Weisbecker , Mark Brown , Borislav Petkov , Thomas Gleixner , surenb@google.com, linux-arm-kernel@lists.infradead.org, SeongJae Park , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Colin Ian King , Andrew Morton , Mike Rapoport , Sandipan Das Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org HAVE_MOVE_PMD enables remapping pages at the PMD level if both the source and destination addresses are PMD-aligned. HAVE_MOVE_PMD is already enabled on x86. The original patch [1] that introduced this config did not enable it on arm64 at the time because of performance issues with flushing the TLB on every PMD move. These issues have since been addressed in more recent releases with improvements to the arm64 TLB invalidation and core mmu_gather code as Will Deacon mentioned in [2]. From the data below, it can be inferred that there is approximately 8x improvement in performance when HAVE_MOVE_PMD is enabled on arm64. --------- Test Results ---------- The following results were obtained on an arm64 device running a 5.4 kernel, by remapping a PMD-aligned, 1GB sized region to a PMD-aligned destination. The results from 10 iterations of the test are given below. All times are in nanoseconds. Control HAVE_MOVE_PMD 9220833 1247761 9002552 1219896 9254115 1094792 8725885 1227760 9308646 1043698 9001667 1101771 8793385 1159896 8774636 1143594 9553125 1025833 9374010 1078125 9100885.4 1134312.6 <-- Mean Time in nanoseconds Total mremap time for a 1GB sized PMD-aligned region drops from ~9.1 milliseconds to ~1.1 milliseconds. (~8x speedup). [1] https://lore.kernel.org/r/20181108181201.88826-3-joelaf@google.com [2] https://www.mail-archive.com/linuxppc-dev@lists.ozlabs.org/msg140837.html Signed-off-by: Kalesh Singh --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6d232837cbee..844d089668e3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -121,6 +121,7 @@ config ARM64 select GENERIC_VDSO_TIME_NS select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND + select HAVE_MOVE_PMD select HAVE_PCI select HAVE_ACPI_APEI if (ACPI && EFI) select HAVE_ALIGNED_STRUCT_PAGE if SLUB From patchwork Wed Sep 30 22:21:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 11810421 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3BBD112C for ; Wed, 30 Sep 2020 22:22:33 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A8D1520719 for ; Wed, 30 Sep 2020 22:22:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="AIPrEynw"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="uc1lC+BR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A8D1520719 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3oL1gX81aggSS9ez3uSNSXNxR9nsv9zVuDA12PI0iko=; b=AIPrEynwmNplz1zvp5bfT71LO AT0XQ9PJUUC5TpkcVAJkg10aA/MTgXonH3SEpGyf86skL5QSE8ymZnsOVC3UBawDZNOSu/FGQ1Cd5 Cy4U6t0Spsw/RbDbm7T4pH4vNVsjb1CeiAPsTLkEc4mIjDDghkPglOPQr3isv56Z/4e3OVrlFJA5s CwNBIky5++/5vB8nlQgYiLlg7wfNaq++3Eja53Bkak18Fe9ZVhOzgS5F7i/3az9BGcVq8520L5P30 21EccdS1uhMWMIbxmfrkFlZu8YMRNKoiEawo/DkgWNTSSjR+DBLVU6nVPBgDR1AP23p7jqh3zU6RJ IHo6JT8lw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUM-0006Ir-CN; Wed, 30 Sep 2020 22:22:18 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUI-0006EZ-OQ for linux-arm-kernel@lists.infradead.org; Wed, 30 Sep 2020 22:22:16 +0000 Received: by mail-yb1-xb49.google.com with SMTP id r9so3166762ybd.20 for ; Wed, 30 Sep 2020 15:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:cc:content-transfer-encoding; bh=5bHwRAkIFgRUKgngW4gcP068wEdPRj/cA85ER5ge5+w=; b=uc1lC+BRciNCCenfHEg9UpdZN5PsHA6dts/eMegcTHFRbNmxAt+14yp2sd85qBnhWS /iQ6LFsh4BgYcnuYARjFzrcEz/ZOnAaRHA8vceh6iMmgL2CcCA1ShQXN6A0EkeRzto87 XP2ahfz3G1rWtHeMk2Iie02Bh5Bh0tv3f3+acB7crNXPg47WZEE6HxKCeN+Ddv1yICAG sReM4qnVO2nhs3CV0JpJOlQduy3bOPn+tHSdXaPjkP+C6Geq901I1kmdC3Y2kNiZU8q4 1q6YqqR+HGbk++rIzTfi4ZkgncraaFN0QqjeC/1v+W4BQT1+gF6vukHux4c1ngvQ68DW nFBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:cc:content-transfer-encoding; bh=5bHwRAkIFgRUKgngW4gcP068wEdPRj/cA85ER5ge5+w=; b=AsRC+KYCVL6e5BaKr0xbeipxAssVS1P+S7zaB/zNLMxX6t4j1qOFhTH96eokP+lpQc XmDdyOkDOBEyHXx5atGHQ1S9sw8IxZwi8eBq69yGetl2qs7WoYggipeKcja4zccrS8HW ngGKB/BS+mSmlmC2sqzGGurK47PIXgjBhZHPH5w82cuaS3jE6secz+2DpCZr6xMB4h+V UNteT1xow50HIged9DSorn7+ZorL43dxXMRAOQkSMg0iz2OliUmHHI+kWyG/R10HNSjC iREPsxavaogLvfhKoKuHyILPdx8bQ2S58BZTZhkfy2fbk1NiNgvaTVMizD5+pvRdRKff kzFw== X-Gm-Message-State: AOAM532YIO1vzOQ1lh+9NcN3cmUU6ccdcN8TVngdEm593MZ0OfHBZkQi SD+1JZ6+ctw3e3cmxyfcWGOxhl0HV7e0ykztrw== X-Google-Smtp-Source: ABdhPJwBYpAmlRALKHYyir3OblZbIEhsgUn0acpHu8aRfQEb5kag4nDCQqATpwKZEagaltStuNxSB3SptnIGiDEZdw== X-Received: from kaleshsingh.c.googlers.com ([fda3:e722:ac3:10:14:4d90:c0a8:2145]) (user=kaleshsingh job=sendgmr) by 2002:a25:5d8:: with SMTP id 207mr6822598ybf.444.1601504527236; Wed, 30 Sep 2020 15:22:07 -0700 (PDT) Date: Wed, 30 Sep 2020 22:21:20 +0000 In-Reply-To: <20200930222130.4175584-1-kaleshsingh@google.com> Message-Id: <20200930222130.4175584-4-kaleshsingh@google.com> Mime-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH 3/5] mm: Speedup mremap on 1GB or larger regions From: Kalesh Singh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200930_182214_824600_5558674A X-CRM114-Status: GOOD ( 27.11 ) X-Spam-Score: -7.0 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:b49 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 1.2 MISSING_HEADERS Missing To: header 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: joelaf@google.com, Mark Rutland , Gavin Shan , Brian Geffon , Peter Zijlstra , Catalin Marinas , kaleshsingh@google.com, Ram Pai , linux-mm@kvack.org, Dave Hansen , Will Deacon , lokeshgidra@google.com, linux-kselftest@vger.kernel.org, "H. Peter Anvin" , Christian Brauner , Shuah Khan , Mina Almasry , Jia He , Arnd Bergmann , "Aneesh Kumar K.V" , Masahiro Yamada , x86@kernel.org, Krzysztof Kozlowski , Ingo Molnar , Sami Tolvanen , kernel-team@android.com, Hassan Naveed , Masami Hiramatsu , Ralph Campbell , Kees Cook , minchan@google.com, Zhenyu Ye , John Hubbard , Frederic Weisbecker , Mark Brown , Borislav Petkov , Thomas Gleixner , surenb@google.com, linux-arm-kernel@lists.infradead.org, Chris von Recklinghausen , William Kucharski , Stephen Boyd , SeongJae Park , linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Andrew Morton , Mike Rapoport , Sandipan Das Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Android needs to move large memory regions for garbage collection. Optimize mremap for >= 1GB-sized regions by moving at the PUD/PGD level if the source and destination addresses are PUD-aligned. For CONFIG_PGTABLE_LEVELS == 3, moving at the PUD level in effect moves PGD entries, since the PUD entry is “folded back” onto the PGD entry. Add HAVE_MOVE_PUD so that architectures where moving at the PUD level isn't supported/tested can turn this off by not selecting the config. Signed-off-by: Kalesh Singh --- arch/Kconfig | 7 + arch/arm64/include/asm/pgtable.h | 1 + mm/mremap.c | 211 ++++++++++++++++++++++++++----- 3 files changed, 189 insertions(+), 30 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..5eabaa00bf9b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -602,6 +602,13 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PUD + bool + help + Architectures that select this are able to move page tables at the + PUD level. If there are only 3 page table levels, the move effectively + happens at the PGD level. + config HAVE_MOVE_PMD bool help diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index d5d3fbe73953..8848125e3024 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -415,6 +415,7 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd)) +#define set_pud_at(mm, addr, pudp, pud) set_pte_at(mm, addr, (pte_t *)pudp, pud_pte(pud)) #define __p4d_to_phys(p4d) __pte_to_phys(p4d_pte(p4d)) #define __phys_to_p4d_val(phys) __phys_to_pte_val(phys) diff --git a/mm/mremap.c b/mm/mremap.c index 138abbae4f75..a5a1440bd366 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -249,14 +249,167 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, return true; } +#else +static inline bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) +{ + return false; +} #endif +#ifdef CONFIG_HAVE_MOVE_PUD +static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + + pgd = pgd_offset(mm, addr); + if (pgd_none_or_clear_bad(pgd)) + return NULL; + + p4d = p4d_offset(pgd, addr); + if (p4d_none_or_clear_bad(p4d)) + return NULL; + + pud = pud_offset(p4d, addr); + if (pud_none_or_clear_bad(pud)) + return NULL; + + return pud; +} + +static pud_t *alloc_new_pud(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + + pgd = pgd_offset(mm, addr); + p4d = p4d_alloc(mm, pgd, addr); + if (!p4d) + return NULL; + pud = pud_alloc(mm, p4d, addr); + if (!pud) + return NULL; + + return pud; +} + +static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) +{ + spinlock_t *old_ptl, *new_ptl; + struct mm_struct *mm = vma->vm_mm; + pud_t pud; + + /* + * The destination pud shouldn't be established, free_pgtables() + * should have released it. + */ + if (WARN_ON_ONCE(!pud_none(*new_pud))) + return false; + + /* + * We don't have to worry about the ordering of src and dst + * ptlocks because exclusive mmap_lock prevents deadlock. + */ + old_ptl = pud_lock(vma->vm_mm, old_pud); + new_ptl = pud_lockptr(mm, new_pud); + if (new_ptl != old_ptl) + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + /* Clear the pud */ + pud = *old_pud; + pud_clear(old_pud); + + VM_BUG_ON(!pud_none(*new_pud)); + + /* Set the new pud */ + set_pud_at(mm, new_addr, new_pud, pud); + flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE); + if (new_ptl != old_ptl) + spin_unlock(new_ptl); + spin_unlock(old_ptl); + + return true; +} +#else +static inline bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) +{ + return false; +} +#endif + +enum pgt_entry { + NORMAL_PMD, + HPAGE_PMD, + NORMAL_PUD, +}; + +/* + * Returns an extent of the corresponding size for the pgt_entry specified if valid. + * Else returns a smaller extent bounded by the end of the source and destination + * pgt_entry. Returns 0 if an invalid pgt_entry is specified. + */ +static unsigned long get_extent(enum pgt_entry entry, unsigned long old_addr, + unsigned long old_end, unsigned long new_addr) +{ + unsigned long next, extent, mask, size; + + if (entry == NORMAL_PMD || entry == HPAGE_PMD) { + mask = PMD_MASK; + size = PMD_SIZE; + } else if (entry == NORMAL_PUD) { + mask = PUD_MASK; + size = PUD_SIZE; + } else + return 0; + + next = (old_addr + size) & mask; + /* even if next overflowed, extent below will be ok */ + extent = (next > old_end) ? old_end - old_addr : next - old_addr; + next = (new_addr + size) & mask; + if (extent > next - new_addr) + extent = next - new_addr; + return extent; +} + +/* + * Attempts to speedup the move by moving entry at the level corresponding to + * pgt_entry. Returns true if the move was successful, else false. + */ +static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_struct *vma, + unsigned long old_addr, unsigned long new_addr, void *old_entry, + void *new_entry, bool need_rmap_locks) +{ + bool moved = false; + + /* See comment in move_ptes() */ + if (need_rmap_locks) + take_rmap_locks(vma); + if (entry == NORMAL_PMD) + moved = move_normal_pmd(vma, old_addr, new_addr, old_entry, new_entry); + else if (entry == NORMAL_PUD) + moved = move_normal_pud(vma, old_addr, new_addr, old_entry, new_entry); + else if (entry == HPAGE_PMD) + moved = move_huge_pmd(vma, old_addr, new_addr, old_entry, new_entry); + else + WARN_ON_ONCE(1); + if (need_rmap_locks) + drop_rmap_locks(vma); + + return moved; +} + unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, bool need_rmap_locks) { - unsigned long extent, next, old_end; + unsigned long extent, old_end; struct mmu_notifier_range range; pmd_t *old_pmd, *new_pmd; @@ -269,14 +422,27 @@ unsigned long move_page_tables(struct vm_area_struct *vma, for (; old_addr < old_end; old_addr += extent, new_addr += extent) { cond_resched(); - next = (old_addr + PMD_SIZE) & PMD_MASK; - /* even if next overflowed, extent below will be ok */ - extent = next - old_addr; - if (extent > old_end - old_addr) - extent = old_end - old_addr; - next = (new_addr + PMD_SIZE) & PMD_MASK; - if (extent > next - new_addr) - extent = next - new_addr; +#ifdef CONFIG_HAVE_MOVE_PUD + /* + * If extent is PUD-sized try to speed up the move by moving at the + * PUD level if possible. + */ + extent = get_extent(NORMAL_PUD, old_addr, old_end, new_addr); + if (extent == PUD_SIZE) { + pud_t *old_pud, *new_pud; + + old_pud = get_old_pud(vma->vm_mm, old_addr); + if (!old_pud) + continue; + new_pud = alloc_new_pud(vma->vm_mm, vma, new_addr); + if (!new_pud) + break; + if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new_addr, + old_pud, new_pud, need_rmap_locks)) + continue; + } +#endif + extent = get_extent(NORMAL_PMD, old_addr, old_end, new_addr); old_pmd = get_old_pmd(vma->vm_mm, old_addr); if (!old_pmd) continue; @@ -284,18 +450,10 @@ unsigned long move_page_tables(struct vm_area_struct *vma, if (!new_pmd) break; if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) || pmd_devmap(*old_pmd)) { - if (extent == HPAGE_PMD_SIZE) { - bool moved; - /* See comment in move_ptes() */ - if (need_rmap_locks) - take_rmap_locks(vma); - moved = move_huge_pmd(vma, old_addr, new_addr, - old_pmd, new_pmd); - if (need_rmap_locks) - drop_rmap_locks(vma); - if (moved) - continue; - } + if (extent == HPAGE_PMD_SIZE && + move_pgt_entry(HPAGE_PMD, vma, old_addr, new_addr, old_pmd, + new_pmd, need_rmap_locks)) + continue; split_huge_pmd(vma, old_pmd, old_addr); if (pmd_trans_unstable(old_pmd)) continue; @@ -305,15 +463,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma, * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible. */ - bool moved; - - if (need_rmap_locks) - take_rmap_locks(vma); - moved = move_normal_pmd(vma, old_addr, new_addr, - old_pmd, new_pmd); - if (need_rmap_locks) - drop_rmap_locks(vma); - if (moved) + if (move_pgt_entry(NORMAL_PMD, vma, old_addr, new_addr, old_pmd, + new_pmd, need_rmap_locks)) continue; #endif } From patchwork Wed Sep 30 22:21:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 11810425 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CCCE112C for ; Wed, 30 Sep 2020 22:22:45 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E955B20719 for ; Wed, 30 Sep 2020 22:22:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="SC/wAjVy"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="W1iWTcIg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E955B20719 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ptK3dNZe87NWQrHKOoIgWMaPofwnJUVGQLwXMXyiZng=; b=SC/wAjVyPP2wkZoQVpBGqdsmL MK5TmmNXcQnh6Yhikwollux15LGn9TqusYphbrQZAZWz5P0HRzMU7NRC7oaVAlcQ8WZfRlJoQUClj KEToqckk/n6EYcdTzI3NaPDb9zUFNmsSzFf7cmxMxHcpd++TY2Ed7ri6LfeX8w2qNhjaM8xVcggGz bygxr0Wp0rD6Z/Ot6mbk+wJx2zun5fqInvKt1bd4Y/L38v+3bjaLVl16Fek/iU+q9cxN7gT3gnoXX Z4NbVvqlX6TuRE1/Qop0b6ENT5FrqCHkP0xuHQvPUtmpDOQwMJPD6xbd0JnC3IQJd35YggD2R5aB8 2pMB/obpg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUX-0006Np-To; Wed, 30 Sep 2020 22:22:30 +0000 Received: from mail-qv1-xf49.google.com ([2607:f8b0:4864:20::f49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUP-0006J7-Hg for linux-arm-kernel@lists.infradead.org; Wed, 30 Sep 2020 22:22:22 +0000 Received: by mail-qv1-xf49.google.com with SMTP id 99so1899826qva.1 for ; Wed, 30 Sep 2020 15:22:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:cc; bh=r9i4rFwlO0Ri1Z8TnSbsRvxZfWKWakVdyJrZ8aN7lSY=; b=W1iWTcIgxIUy2UVtGwKg/FWOAbFwxj1yZWmUPz5Pi/125KT5znQ+tf41yAd1LXtU3d Uib3iflcCUIsFkmFwF65SPdIo0tOd05bt6iEjAmwvgkuxVugygP1L5ds1Ja8WufaiuMV xz81lcY5xrAL+nyfM3Z0hdPj/tJ43t16HfcFkrJ+MxmieS3E1BV/xbXh8PvXr1ROpI8b rCKYtMByUOGystQtiQJjncYyQgIpF3rGdUXr5N8/ZwBTqs43v+RaP/0IRmqIv5X0KeCw wjabq5p+HsWNPVNiSZT1vRrkK3OoQqkBQQ+qiqurkNU1dwg9e3kAqbptm+mdcIc9xW87 dADg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=r9i4rFwlO0Ri1Z8TnSbsRvxZfWKWakVdyJrZ8aN7lSY=; b=iqrKQSsoP7R2UCby7/fygevsOeJQhadzUmihE/pSM0XQQA8nydTTIgqEQGyhPBTkbn ApSV2z4Lp1lGlUdnr+9LHHUcQbsJ5QZYVV7LBPwoCDxaKto4daXBOCSSAspChNSujHbY iT5ZUTt3OoBuAASSiILmacD0sh2AUaey4EWBrygM8jqGkd7SuWfnxfHopLYzWc1X20YI H5qry4YSXUoHLb/sQotQBHfZQWDJtKMAchjYaxHdIo4ubvPdeZfrq3NPYGwwLI80HaAA z0KP826sMDLDjfgAfFA5Bdj0f0FngUEzV080k9DELUe1iC7mVnCLFc9d81Ek0tMHZcKP D2oA== X-Gm-Message-State: AOAM531N2WsqVnni6BdjvUc9t8PlDhdHziZxAyejizrf/6JX82BJJmT2 rYzcZ1v+LwVIS7tm9h2D0TMOg9yPOgRYqIX1nA== X-Google-Smtp-Source: ABdhPJw/sR64l5kK5cZN6wHaMabXV7sFr12f7agBjeiB0rYDcGu7AUGvx40hqtjxrcgjvKmLH0L0Diy+P9Slii6SiQ== X-Received: from kaleshsingh.c.googlers.com ([fda3:e722:ac3:10:14:4d90:c0a8:2145]) (user=kaleshsingh job=sendgmr) by 2002:ad4:58e3:: with SMTP id di3mr4707465qvb.54.1601504537414; Wed, 30 Sep 2020 15:22:17 -0700 (PDT) Date: Wed, 30 Sep 2020 22:21:21 +0000 In-Reply-To: <20200930222130.4175584-1-kaleshsingh@google.com> Message-Id: <20200930222130.4175584-5-kaleshsingh@google.com> Mime-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH 4/5] arm64: mremap speedup - Enable HAVE_MOVE_PUD From: Kalesh Singh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200930_182221_634726_118C5864 X-CRM114-Status: GOOD ( 15.33 ) X-Spam-Score: -7.0 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:f49 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 1.2 MISSING_HEADERS Missing To: header 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: joelaf@google.com, Mark Rutland , Gavin Shan , Brian Geffon , Peter Zijlstra , Catalin Marinas , kaleshsingh@google.com, Ram Pai , Kamalesh Babulal , Dave Hansen , Will Deacon , lokeshgidra@google.com, linux-kselftest@vger.kernel.org, "H. Peter Anvin" , Christian Brauner , Shuah Khan , Mina Almasry , Jia He , Arnd Bergmann , "Aneesh Kumar K.V" , Masahiro Yamada , x86@kernel.org, Krzysztof Kozlowski , Jason Gunthorpe , Ingo Molnar , Sami Tolvanen , kernel-team@android.com, Dave Martin , Hassan Naveed , Masami Hiramatsu , Ralph Campbell , Kees Cook , minchan@google.com, Zhenyu Ye , John Hubbard , Frederic Weisbecker , Mark Brown , Borislav Petkov , Thomas Gleixner , Zi Yan , surenb@google.com, linux-arm-kernel@lists.infradead.org, SeongJae Park , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Andrew Morton , Mike Rapoport , Sandipan Das Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org HAVE_MOVE_PUD enables remapping pages at the PUD level if both the source and destination addresses are PUD-aligned. With HAVE_MOVE_PUD enabled it can be inferred that there is approximately a 19x improvement in performance on arm64. (See data below). ------- Test Results --------- The following results were obtained using a 5.4 kernel, by remapping a PUD-aligned, 1GB sized region to a PUD-aligned destination. The results from 10 iterations of the test are given below: Total mremap times for 1GB data on arm64. All times are in nanoseconds. Control HAVE_MOVE_PUD 1247761 74271 1219896 46771 1094792 59687 1227760 48385 1043698 76666 1101771 50365 1159896 52500 1143594 75261 1025833 61354 1078125 48697 1134312.6 59395.7 <-- Mean time in nanoseconds A 1GB mremap completion time drops from ~1.1 milliseconds to ~59 microseconds on arm64. (~19x speed up). Signed-off-by: Kalesh Singh --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 844d089668e3..4d521f0a5863 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -122,6 +122,7 @@ config ARM64 select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_MOVE_PMD + select HAVE_MOVE_PUD select HAVE_PCI select HAVE_ACPI_APEI if (ACPI && EFI) select HAVE_ALIGNED_STRUCT_PAGE if SLUB From patchwork Wed Sep 30 22:21:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh Singh X-Patchwork-Id: 11810427 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E0466CA for ; Wed, 30 Sep 2020 22:23:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4DC2A20719 for ; Wed, 30 Sep 2020 22:23:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="LDYGfmST"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="jDIKOMNU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DC2A20719 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=qMOvjJB8vl4fYeGRuH+sYCGMUJO4paYy/xSNQcl/VzA=; b=LDYGfmSTFXE5acXSXyu6R/rZC TRr3ZO5jnkfZ+sfFyI++RJtkbM/IOs10mxW4KrHL9cviWaRoOD/jbq7xrwVLF7iw+OROd+rgDtFXn 42+W17CK7pu5JrLhAQdKoTTDwxoM71NXCxfO/c+FA+VrdiKTKStm0pS3jcEKsfLnhEEMD8RWLD1/8 278uySEUzN2z0obqBGodQn/orjoWT0HUljE+bRuvu4dyCqNKMtUHjhVqpWrar9nOCcpo6WYWR8+9L e5dJ/vfqBclFPI8JrpDPJAbsMtAKRD4h9ihdbQSX0Tq86v7tv45cDENrVtfRStt7hk1Y9XzjqFxjZ vwFCTydxw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUp-0006UL-Rh; Wed, 30 Sep 2020 22:22:47 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kNkUY-0006NY-UT for linux-arm-kernel@lists.infradead.org; Wed, 30 Sep 2020 22:22:34 +0000 Received: by mail-qt1-x84a.google.com with SMTP id y53so2269954qth.2 for ; Wed, 30 Sep 2020 15:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:cc; bh=Mra73tAqzotShWNgSXv1vhLyePg3f0HjrcFvwMDeq9o=; b=jDIKOMNUrsBzgzsboYR6brg4WOYp4kCRZqrCWAvAZbpkekxYpp5uwWu1VMa3jBM1D4 qePDAkhwPaS7ozmJhvpiPn8rqjAEru5rJyPx6RWU/8Ay/7inrR7CHVJ77p9zaJZ1IwI9 Dy5ixhsQoV7e7t6GjskB9KRoGi6vhUEOJIOmUECHIHMNA/6dx0Mfp5sk/qPevZgAIWJD dSDgUoz8vPO96wn0mRJrwb6JU+kn6sz1n4gmMp04q1wpnavCaWjLkLkQER3ZYnT7jxHz JbrMiGc+vRmaiWx7InwR9v7+c9MnvzgX9b4pL1qHoSDq+DWQYdeR0dkqLDkbjTEQELIk LT7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:cc; bh=Mra73tAqzotShWNgSXv1vhLyePg3f0HjrcFvwMDeq9o=; b=LH57SlC5jLW2XPP9Hj/1j1L5OGYEvAPebdFlutdIxlP1X7fcxsn+VND97gslKF3UxU UTsZWISth3Rr4ExS+WHivHHV0o+WX/pvAcvKOjZxQ2UC62v4q+tUGP8nHz18oZ/Vgd7Q 0K0rFP03T8ogOUvUJDBX588HSM5r7IMh+4MPKCf3zyE1C0z/HpgP5mCr9NUnBNLhyekp QLLoz4bqqat1Y7xRu5QH8PtslNY/l+72f5NDoANM7fApslDsprJSa6R9sW9LupeeGkE+ cgnFM63ZBcgqIfVA8yXbI/kEGGh8QrpSg9r3BseeqkByOk4aosi08ZpbHY7mW37hlW5x Unuw== X-Gm-Message-State: AOAM531EU8pZJVaaFmOPXKzW1BHrdd8DaxEzH/BvkAFWlFqm3PujtTjp ijXLa+oY6hEMwlKkP548cLiG+e1cxNcrpYywuQ== X-Google-Smtp-Source: ABdhPJyLR6MYlSTS4XQqoJzvYxjrHwSWIVNtfhqTdtx45lsRl3oJwBhgd7X3m6yDztZJ8ANUp7tpdCXWxqNC36OyAw== X-Received: from kaleshsingh.c.googlers.com ([fda3:e722:ac3:10:14:4d90:c0a8:2145]) (user=kaleshsingh job=sendgmr) by 2002:a0c:b39a:: with SMTP id t26mr4865208qve.19.1601504547782; Wed, 30 Sep 2020 15:22:27 -0700 (PDT) Date: Wed, 30 Sep 2020 22:21:22 +0000 In-Reply-To: <20200930222130.4175584-1-kaleshsingh@google.com> Message-Id: <20200930222130.4175584-6-kaleshsingh@google.com> Mime-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH 5/5] x86: mremap speedup - Enable HAVE_MOVE_PUD From: Kalesh Singh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200930_182231_035444_FD240E49 X-CRM114-Status: GOOD ( 15.24 ) X-Spam-Score: -7.0 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:84a listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 1.2 MISSING_HEADERS Missing To: header 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.5 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: joelaf@google.com, Mark Rutland , Gavin Shan , Brian Geffon , Peter Zijlstra , Catalin Marinas , kaleshsingh@google.com, Ram Pai , Dave Hansen , Will Deacon , lokeshgidra@google.com, linux-kselftest@vger.kernel.org, "H. Peter Anvin" , Christian Brauner , Shuah Khan , Mina Almasry , Jia He , Arnd Bergmann , "Aneesh Kumar K.V" , Masahiro Yamada , x86@kernel.org, Krzysztof Kozlowski , Steven Price , Ingo Molnar , Sami Tolvanen , kernel-team@android.com, Hassan Naveed , Masami Hiramatsu , Ralph Campbell , Kees Cook , minchan@google.com, Zhenyu Ye , John Hubbard , Frederic Weisbecker , Mark Brown , Borislav Petkov , Thomas Gleixner , surenb@google.com, linux-arm-kernel@lists.infradead.org, SeongJae Park , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Andrew Morton , Mike Rapoport , Sandipan Das Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org HAVE_MOVE_PUD enables remapping pages at the PUD level if both the source and destination addresses are PUD-aligned. With HAVE_MOVE_PUD enabled it can be inferred that there is approximately a 13x improvement in performance on x86. (See data below). ------- Test Results --------- The following results were obtained using a 5.4 kernel, by remapping a PUD-aligned, 1GB sized region to a PUD-aligned destination. The results from 10 iterations of the test are given below: Total mremap times for 1GB data on x86. All times are in nanoseconds. Control HAVE_MOVE_PUD 180394 15089 235728 14056 238931 25741 187330 13838 241742 14187 177925 14778 182758 14728 160872 14418 205813 15107 245722 13998 205721.5 15594 <-- Mean time in nanoseconds A 1GB mremap completion time drops from ~205 microseconds to ~15 microseconds on x86. (~13x speed up). Signed-off-by: Kalesh Singh --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7101ac64bb20..ff6e2755cab8 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -198,6 +198,7 @@ config X86 select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC select HAVE_MOVE_PMD + select HAVE_MOVE_PUD select HAVE_NMI select HAVE_OPROFILE select HAVE_OPTPROBES