From patchwork Thu May 6 23:25:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12243603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADEBBC433B4 for ; Thu, 6 May 2021 23:25:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 47C8661289 for ; Thu, 6 May 2021 23:25:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 47C8661289 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D9E166B0071; Thu, 6 May 2021 19:25:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D745E6B0072; Thu, 6 May 2021 19:25:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC8B06B0073; Thu, 6 May 2021 19:25:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 9ECE96B0071 for ; Thu, 6 May 2021 19:25:45 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 51692181AEF2A for ; Thu, 6 May 2021 23:25:45 +0000 (UTC) X-FDA: 78112390650.02.61E085A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf25.hostedemail.com (Postfix) with ESMTP id DA1976000104 for ; Thu, 6 May 2021 23:25:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620343544; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T2/Kp/6DMcvb66SGLUrQ0litwe4gG/CMDxNGXzk5jHI=; b=agWYeuKAVb2tjpUbbK8YNJ+PPo84OIW0ts8sEui9CGy1wrv4kWEb3kUwQQaHI49rq0hTw2 chhTfyPKSQIwu4CT0yDUAHtHgEYWX/09KpqtXeip1/XtSRkzK452Vu/2tPLhnmcXUTcexK j95Vht4vYYoRwkFFTCbwTPL+JPX/LZE= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-21-R9EJSJlEMPida8MZHQG0cw-1; Thu, 06 May 2021 19:25:42 -0400 X-MC-Unique: R9EJSJlEMPida8MZHQG0cw-1 Received: by mail-qt1-f200.google.com with SMTP id h4-20020ac858440000b029019d657b9f21so4575176qth.9 for ; Thu, 06 May 2021 16:25:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T2/Kp/6DMcvb66SGLUrQ0litwe4gG/CMDxNGXzk5jHI=; b=cbrbrd+ynkn9yXYbBWe+uZ1mJi0Jsl7qr5KNvPxLYtJha9r7SnjPVt5UvtamLvwefG y0opVkQ8G9p7O3LeIrUQohTcLs4hV2JLysMzNzGM/FffsP6gXk9nZSmh68Hq90as4Ps4 6XdrIYgSJfr41jY5eKsxV5GHJHOt/LplmS+zkAxz1HBq9cXX7mIlfN+CTa8lSg0Tlops lUCdDVm/DL2d2l0vHA1031W6nIooJPzp9aYzREcPmJPiPOVp9JMMCxwxHGLAAHHKBLyP 5OC8ykxXRIYaE0DW4TDgDFQBgFNXkU+RLspOmAy89YhY9mb2NOGZ5+AjYjouL3f3j93l /kow== X-Gm-Message-State: AOAM530gyPqLXg0jIW8Xo+j7sT989YaXK4XSoP2B4A/UgjhQbiCbgpkZ rtiC2KwueCC9XrB4LemLhg47SBCfhbCDKtYrEz//lLzYVHKnC3rAZ95uQnMtI3gGnc/K8vMBOxR I9ShQtimQKJay8rg+bgkzP4wDCIXweNH51ykYGbQ7AjAiz2g4TTVOtOFkNUrQ X-Received: by 2002:ac8:5fd4:: with SMTP id k20mr6521819qta.239.1620343541695; Thu, 06 May 2021 16:25:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwiu2OYXRhtEUQX7rvGo4YfZRmLbpxFTcAg2tSwrt6j14FGtqU73BnoVhIbOEEpZWoT1F1fFQ== X-Received: by 2002:ac8:5fd4:: with SMTP id k20mr6521788qta.239.1620343541343; Thu, 06 May 2021 16:25:41 -0700 (PDT) Received: from t490s.redhat.com (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id q13sm1605026qkn.10.2021.05.06.16.25.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 May 2021 16:25:40 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , peterx@redhat.com, John Hubbard , Jan Kara , Kirill Shutemov , Jason Gunthorpe , Andrew Morton , Kirill Tkhai , Michal Hocko , Oleg Nesterov , Jann Horn , Linus Torvalds , Matthew Wilcox , Andrea Arcangeli Subject: [PATCH 1/3] mm/gup_benchmark: Support threading Date: Thu, 6 May 2021 19:25:35 -0400 Message-Id: <20210506232537.165788-2-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210506232537.165788-1-peterx@redhat.com> References: <20210506232537.165788-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=agWYeuKA; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf25.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: jc5ajdjrk9wh1gh3iu5g8n1gunparzpi X-Rspamd-Queue-Id: DA1976000104 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620343537-401620 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new parameter "-j N" to support concurrent gup test. Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- tools/testing/selftests/vm/gup_test.c | 94 ++++++++++++++++++--------- 1 file changed, 63 insertions(+), 31 deletions(-) diff --git a/tools/testing/selftests/vm/gup_test.c b/tools/testing/selftests/vm/gup_test.c index 1e662d59c502a..3f0fcc697c3fc 100644 --- a/tools/testing/selftests/vm/gup_test.c +++ b/tools/testing/selftests/vm/gup_test.c @@ -6,6 +6,7 @@ #include #include #include +#include #include "../../../../mm/gup_test.h" #define MB (1UL << 20) @@ -15,6 +16,12 @@ #define FOLL_WRITE 0x01 /* check pte is writable */ #define FOLL_TOUCH 0x02 /* mark page accessed */ +static unsigned long cmd = GUP_FAST_BENCHMARK; +static int gup_fd, repeats = 1; +static unsigned long size = 128 * MB; +/* Serialize prints */ +static pthread_mutex_t print_mutex = PTHREAD_MUTEX_INITIALIZER; + static char *cmd_to_str(unsigned long cmd) { switch (cmd) { @@ -34,17 +41,55 @@ static char *cmd_to_str(unsigned long cmd) return "Unknown command"; } +void *gup_thread(void *data) +{ + struct gup_test gup = *(struct gup_test *)data; + int i; + + /* Only report timing information on the *_BENCHMARK commands: */ + if ((cmd == PIN_FAST_BENCHMARK) || (cmd == GUP_FAST_BENCHMARK) || + (cmd == PIN_LONGTERM_BENCHMARK)) { + for (i = 0; i < repeats; i++) { + gup.size = size; + if (ioctl(gup_fd, cmd, &gup)) + perror("ioctl"), exit(1); + + pthread_mutex_lock(&print_mutex); + printf("%s: Time: get:%lld put:%lld us", + cmd_to_str(cmd), gup.get_delta_usec, + gup.put_delta_usec); + if (gup.size != size) + printf(", truncated (size: %lld)", gup.size); + printf("\n"); + pthread_mutex_unlock(&print_mutex); + } + } else { + gup.size = size; + if (ioctl(gup_fd, cmd, &gup)) { + perror("ioctl"); + exit(1); + } + + pthread_mutex_lock(&print_mutex); + printf("%s: done\n", cmd_to_str(cmd)); + if (gup.size != size) + printf("Truncated (size: %lld)\n", gup.size); + pthread_mutex_unlock(&print_mutex); + } + + return NULL; +} + int main(int argc, char **argv) { struct gup_test gup = { 0 }; - unsigned long size = 128 * MB; - int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 1; - unsigned long cmd = GUP_FAST_BENCHMARK; + int filed, i, opt, nr_pages = 1, thp = -1, write = 1, threads = 1; int flags = MAP_PRIVATE, touch = 0; char *file = "/dev/zero"; + pthread_t *tid; char *p; - while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHpz")) != -1) { + while ((opt = getopt(argc, argv, "m:r:n:F:f:abcj:tTLUuwWSHpz")) != -1) { switch (opt) { case 'a': cmd = PIN_FAST_BENCHMARK; @@ -74,6 +119,9 @@ int main(int argc, char **argv) /* strtol, so you can pass flags in hex form */ gup.gup_flags = strtol(optarg, 0, 0); break; + case 'j': + threads = atoi(optarg); + break; case 'm': size = atoi(optarg) * MB; break; @@ -154,8 +202,8 @@ int main(int argc, char **argv) if (write) gup.gup_flags |= FOLL_WRITE; - fd = open("/sys/kernel/debug/gup_test", O_RDWR); - if (fd == -1) { + gup_fd = open("/sys/kernel/debug/gup_test", O_RDWR); + if (gup_fd == -1) { perror("open"); exit(1); } @@ -185,32 +233,16 @@ int main(int argc, char **argv) p[0] = 0; } - /* Only report timing information on the *_BENCHMARK commands: */ - if ((cmd == PIN_FAST_BENCHMARK) || (cmd == GUP_FAST_BENCHMARK) || - (cmd == PIN_LONGTERM_BENCHMARK)) { - for (i = 0; i < repeats; i++) { - gup.size = size; - if (ioctl(fd, cmd, &gup)) - perror("ioctl"), exit(1); - - printf("%s: Time: get:%lld put:%lld us", - cmd_to_str(cmd), gup.get_delta_usec, - gup.put_delta_usec); - if (gup.size != size) - printf(", truncated (size: %lld)", gup.size); - printf("\n"); - } - } else { - gup.size = size; - if (ioctl(fd, cmd, &gup)) { - perror("ioctl"); - exit(1); - } - - printf("%s: done\n", cmd_to_str(cmd)); - if (gup.size != size) - printf("Truncated (size: %lld)\n", gup.size); + tid = malloc(sizeof(pthread_t) * threads); + if (!tid) { + perror("malloc()"); + exit(1); } + for (i = 0; i < threads; i++) + pthread_create(&tid[i], NULL, gup_thread, &gup); + for (i = 0; i < threads; i++) + pthread_join(tid[i], NULL); + free(tid); return 0; } From patchwork Thu May 6 23:25:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12243605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 776E6C43460 for ; Thu, 6 May 2021 23:25:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E89ED61289 for ; Thu, 6 May 2021 23:25:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E89ED61289 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 735B46B0072; Thu, 6 May 2021 19:25:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C0336B0073; Thu, 6 May 2021 19:25:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 512066B0074; Thu, 6 May 2021 19:25:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id 2F4436B0072 for ; Thu, 6 May 2021 19:25:47 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EAFDC99BE for ; Thu, 6 May 2021 23:25:46 +0000 (UTC) X-FDA: 78112390692.26.E645373 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 313A4135 for ; Thu, 6 May 2021 23:25:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620343546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wdtiudLSXqX9mG3wVVY1C6HQ27m2h7nYkQwJQU/EgKg=; b=NEDAjdfLyxYbZtXGj8TFuYZe7iaBCgjCS2lfn1Ue8HFU2PC2ZZ8P1HdvR0NEhXFLI8RGgQ 9+fJYS8o10d5D+mvQMFexJ4IjjxseqkbypMchrBp8SSuIfDMEKAAGiwrn5TvDGAOyI6b8c WnTp1SD1Xo9cSWwqdyI2aJvteSeFWAE= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-585-xjVkqgRUMI-NNmOv5CA6KA-1; Thu, 06 May 2021 19:25:44 -0400 X-MC-Unique: xjVkqgRUMI-NNmOv5CA6KA-1 Received: by mail-qt1-f199.google.com with SMTP id b19-20020ac84f130000b02901d543c52248so1391577qte.1 for ; Thu, 06 May 2021 16:25:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wdtiudLSXqX9mG3wVVY1C6HQ27m2h7nYkQwJQU/EgKg=; b=GlSUDMULHjNp0SCn6CVhF/cq0itX2AV/0eAdP3MUq3CAXRLYJ7m4ZwuEWnBFHRzdi+ Daz73RV/0ZvvXnisU/tJvoM2IF++cxpHukPjmDFk+01+dfDHwlXZXJdhHroOE5/D5VJ6 GHqXwTYkws0D0JVN+Qkat6KlCXcKfQeiJS/z43hEq4dE0MNEs1ppNrzsHq+DCHlSPg3K gAHMfZw5zyRD3Xyw0NONfvnUKlao67kzLMCG9mRLjEEQmMVoah3dHXO1lKQXoDfdbNwg jnixYPk0n8IEtzA/f1a+jVNzeWgUHv07AW/VtjQdCQlWZyTyKH93UwUFdYPkaawRfyXM mwXQ== X-Gm-Message-State: AOAM531fsWWWWzNjHj9KXOPaC6RzRZTx5vmJAkSMkKXyJqc2GMMlgHgg Zd7QUet1z32gmkViNE+kaLJX+937/sZOatDpOfUfPEPDb2pCNZA3r2H2urh7iv9fVNZZnj3vxA2 uN7FesIXz0IN4qfz2npHNmlGRROKbIUVkeNhS1Kz9DVzIJDxIJkxEK6mECFYS X-Received: by 2002:ac8:5fd5:: with SMTP id k21mr3071570qta.231.1620343543637; Thu, 06 May 2021 16:25:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz3dk/t7SX3sfbhwUbA67QTp2M/7n3z1E6C0HGX6OS4HRvcdqLfWnmNr2vPMSeg4IVAFExKaA== X-Received: by 2002:ac8:5fd5:: with SMTP id k21mr3071528qta.231.1620343543208; Thu, 06 May 2021 16:25:43 -0700 (PDT) Received: from t490s.redhat.com (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id q13sm1605026qkn.10.2021.05.06.16.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 May 2021 16:25:42 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , peterx@redhat.com, John Hubbard , Jan Kara , Kirill Shutemov , Jason Gunthorpe , Andrew Morton , Kirill Tkhai , Michal Hocko , Oleg Nesterov , Jann Horn , Linus Torvalds , Matthew Wilcox , Andrea Arcangeli Subject: [PATCH 2/3] mm: gup: allow FOLL_PIN to scale in SMP Date: Thu, 6 May 2021 19:25:36 -0400 Message-Id: <20210506232537.165788-3-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210506232537.165788-1-peterx@redhat.com> References: <20210506232537.165788-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NEDAjdfL; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam03 X-Stat-Signature: 8qwj6ypij4cd7bx65zz78uw91udzd5ut X-Rspamd-Queue-Id: 313A4135 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620343539-987139 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrea Arcangeli has_pinned cannot be written by each pin-fast or it won't scale in SMP. This isn't "false sharing" strictly speaking (it's more like "true non-sharing"), but it creates the same SMP scalability bottleneck of "false sharing". To verify the improvement a new "pin_fast.c" program was added to the will-it-scale benchmark. == pin_fast.c - start == /* SPDX-License-Identifier: GPL-2.0-or-later */ #define _GNU_SOURCE #include #include #include #include #include #include #include #include /* exercises pin_user_pages_fast, requires a kernel with CONFIG_GUP_TEST=y */ char *testcase_description = "pin_user_pages_fast SMP scalability benchmark"; static int gup_fd; #define NR_PAGES 1024 #define BUFLEN (getpagesize() * NR_PAGES) #define GUP_TEST_MAX_PAGES_TO_DUMP 8 #define PIN_FAST_BENCHMARK _IOWR('g', 2, struct gup_test) struct gup_test { __u64 get_delta_usec; __u64 put_delta_usec; __u64 addr; __u64 size; __u32 nr_pages_per_call; __u32 flags; /* * Each non-zero entry is the number of the page (1-based: first page is * page 1, so that zero entries mean "do nothing") from the .addr base. */ __u32 which_pages[GUP_TEST_MAX_PAGES_TO_DUMP]; }; void testcase_prepare(unsigned long nr_tasks) { gup_fd = open("/sys/kernel/debug/gup_test", O_RDWR); assert(gup_fd >= 0); } void testcase(unsigned long long *iterations, unsigned long nr) { char *p = aligned_alloc(getpagesize()*512, BUFLEN); assert(p); assert(!madvise(p, BUFLEN, MADV_HUGEPAGE)); for (int i = 0; i < NR_PAGES; i++) p[getpagesize()*i] = 0; struct gup_test gup = { .size = BUFLEN, .addr = (unsigned long)p, .nr_pages_per_call = 1, }; while (1) { assert(!ioctl(gup_fd, PIN_FAST_BENCHMARK, &gup)); (*iterations)++; } free(p); } void testcase_cleanup(void) { assert(!close(gup_fd)); } == pin_fast.c - end == The pin_fast will-it-scale benchmark was run with 1 thread per-CPU on this 2 NUMA nodes system: available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 node 0 size: 128792 MB node 0 free: 126741 MB node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 1 size: 128944 MB node 1 free: 127330 MB node distances: node 0 1 0: 10 32 1: 32 10 Before this commit (average 25617 +- 0.16%): tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,25641,0.17,0 tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,25652,0.16,0 tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,25559,0.16,0 After this commit (average 1194790 +- 0.11%): tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,1196513,0.19,0 tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,1194664,0.19,0 tasks,processes,processes_idle,threads,threads_idle,linear 0,0,100,0,100,0 256,0,0.00,1193194,0.19,0 This commits increases the SMP scalability of pin_user_pages_fast() executed by different threads of the same process by more than 4000%. Signed-off-by: Andrea Arcangeli Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 63a079e361a3d..8b513e1723b45 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1292,7 +1292,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, BUG_ON(*locked != 1); } - if (flags & FOLL_PIN) + if (flags & FOLL_PIN && !atomic_read(&mm->has_pinned)) atomic_set(&mm->has_pinned, 1); /* @@ -2617,7 +2617,7 @@ static int internal_get_user_pages_fast(unsigned long start, FOLL_FAST_ONLY))) return -EINVAL; - if (gup_flags & FOLL_PIN) + if (gup_flags & FOLL_PIN && !atomic_read(¤t->mm->has_pinned)) atomic_set(¤t->mm->has_pinned, 1); if (!(gup_flags & FOLL_FAST_ONLY)) From patchwork Thu May 6 23:25:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12243607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5835C433B4 for ; Thu, 6 May 2021 23:25:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 79EF3613B5 for ; Thu, 6 May 2021 23:25:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 79EF3613B5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1B67B6B0073; Thu, 6 May 2021 19:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 18DC56B0074; Thu, 6 May 2021 19:25:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02F646B0075; Thu, 6 May 2021 19:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id DBC526B0073 for ; Thu, 6 May 2021 19:25:51 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 929166C11 for ; Thu, 6 May 2021 23:25:51 +0000 (UTC) X-FDA: 78112390902.36.B4E1B26 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 9D102E000105 for ; Thu, 6 May 2021 23:25:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620343550; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OvDl4+FUwbO0MY99vMQj8Ya/mqNaGR1M/DGULDYnMW4=; b=dxs5njPcawi5ps/8DEsTkP+mBcj9TiYvxmlpuMOG5gCWLQf18FNKv8AKSRqi3eRA0irxFX o/qv2GJC+VVxGmCqB1IxUjypIcGjL2CRp5VE9xJkPa5Nrm1AECxIb0av2+CxSi3dxGgjcu ijPAoZXfUhglc5lDRBJBT2AB3+kh1RY= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-447-s32_z9VKPcKDB6jubt0jmw-1; Thu, 06 May 2021 19:25:49 -0400 X-MC-Unique: s32_z9VKPcKDB6jubt0jmw-1 Received: by mail-qt1-f200.google.com with SMTP id 1-20020aed31010000b029019d1c685840so4609288qtg.3 for ; Thu, 06 May 2021 16:25:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OvDl4+FUwbO0MY99vMQj8Ya/mqNaGR1M/DGULDYnMW4=; b=WkzZexu/c7SG+7erAgskELbESt9tVAS3Fz/wCXkDsCDGWgzES7wPmZ5FwO3HianRmi l/OKLSnEKXEgdHVc4/0Mw59i2ciqEv3AimltnGZ1kGOUzy67XnXNNmNQVJy0BmizwRD/ 1wzNAjQxHbXYoSXU3uq8gAxQy6syhUA7ulLLsf91pkuXLDxC5F8ppRkhqlI5C6ZlcBO+ GhgofDtUlfIj+p3OourT8jcr6zGcI4v0GzACKAv6MyZGmzJc/PUU791cOnPdjbHw+NoM n+ANLi400WjNiySL+ru6JN3R3Q3R1VQATdWtS27xmnz4MbB/XOdNQMxOpD8NK16OuJik o1BQ== X-Gm-Message-State: AOAM530XcMy8TSp9KogGgewZKJLEb6lJhD2lPXYKR5YRmwckE0ZQ5xG/ gUFq6m1/EUWNCUfMf/KyLj2CDrw5EY/2qjTliQqu5i1fzVPC6/PTY7xTLk73UBDpClna+s15WVY FOrFDisrtGDurGyEBB7OwJ+5mMTaqfsGJ2jIUGNonCLhQV7IkwNaGKRS3w/yV X-Received: by 2002:a0c:b38b:: with SMTP id t11mr7421862qve.25.1620343545576; Thu, 06 May 2021 16:25:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwxZpHe561yR92rMxp1tyWoGD0aKXhful40fuYgTCuul6pIfMwnyAkv8pzsKfXqXsBDgp200g== X-Received: by 2002:a0c:b38b:: with SMTP id t11mr7421822qve.25.1620343545141; Thu, 06 May 2021 16:25:45 -0700 (PDT) Received: from t490s.redhat.com (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id q13sm1605026qkn.10.2021.05.06.16.25.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 May 2021 16:25:44 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , peterx@redhat.com, John Hubbard , Jan Kara , Kirill Shutemov , Jason Gunthorpe , Andrew Morton , Kirill Tkhai , Michal Hocko , Oleg Nesterov , Jann Horn , Linus Torvalds , Matthew Wilcox , Andrea Arcangeli Subject: [PATCH 3/3] mm: gup: pack has_pinned in MMF_HAS_PINNED Date: Thu, 6 May 2021 19:25:37 -0400 Message-Id: <20210506232537.165788-4-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210506232537.165788-1-peterx@redhat.com> References: <20210506232537.165788-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dxs5njPc; spf=none (imf21.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9D102E000105 X-Stat-Signature: jrp9xbafq1b566oi8gfq3ooqxgdp8re4 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620343546-605775 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrea Arcangeli has_pinned 32bit can be packed in the MMF_HAS_PINNED bit as a noop cleanup. Any atomic_inc/dec to the mm cacheline shared by all threads in pin-fast would reintroduce a loss of SMP scalability to pin-fast, so there's no future potential usefulness to keep an atomic in the mm for this. set_bit(MMF_HAS_PINNED) will be theoretically a bit slower than WRITE_ONCE (atomic_set is equivalent to WRITE_ONCE), but the set_bit (just like atomic_set after this commit) has to be still issued only once per "mm", so the difference between the two will be lost in the noise. will-it-scale "mmap2" shows no change in performance with enterprise config as expected. will-it-scale "pin_fast" retains the > 4000% SMP scalability performance improvement against upstream as expected. This is a noop as far as overall performance and SMP scalability are concerned. Signed-off-by: Andrea Arcangeli [peterx: Fix build for task_mmu.c] Signed-off-by: Peter Xu --- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 2 +- include/linux/mm_types.h | 10 ---------- include/linux/sched/coredump.h | 1 + kernel/fork.c | 1 - mm/gup.c | 9 +++++---- 6 files changed, 8 insertions(+), 17 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4c95cc57a66a8..6144571942db9 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1049,7 +1049,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, return false; if (!is_cow_mapping(vma->vm_flags)) return false; - if (likely(!atomic_read(&vma->vm_mm->has_pinned))) + if (likely(!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags))) return false; page = vm_normal_page(vma, addr, pte); if (!page) diff --git a/include/linux/mm.h b/include/linux/mm.h index d6790ab0cf575..94dc84f6d8658 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1331,7 +1331,7 @@ static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, if (!is_cow_mapping(vma->vm_flags)) return false; - if (!atomic_read(&vma->vm_mm->has_pinned)) + if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; return page_maybe_dma_pinned(page); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6613b26a88946..15d79858fadbd 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,16 +435,6 @@ struct mm_struct { */ atomic_t mm_count; - /** - * @has_pinned: Whether this mm has pinned any pages. This can - * be either replaced in the future by @pinned_vm when it - * becomes stable, or grow into a counter on its own. We're - * aggresive on this bit now - even if the pinned pages were - * unpinned later on, we'll still keep this bit set for the - * lifecycle of this mm just for simplicity. - */ - atomic_t has_pinned; - /** * @write_protect_seq: Locked when any thread is write * protecting pages mapped by this mm to enforce a later COW, diff --git a/include/linux/sched/coredump.h b/include/linux/sched/coredump.h index dfd82eab29025..bf45badd63e6d 100644 --- a/include/linux/sched/coredump.h +++ b/include/linux/sched/coredump.h @@ -73,6 +73,7 @@ static inline int get_dumpable(struct mm_struct *mm) #define MMF_OOM_VICTIM 25 /* mm is the oom victim */ #define MMF_OOM_REAP_QUEUED 26 /* mm was queued for oom_reaper */ #define MMF_MULTIPROCESS 27 /* mm is shared between processes */ +#define MMF_HAS_PINNED 28 /* FOLL_PIN has run, never cleared */ #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ diff --git a/kernel/fork.c b/kernel/fork.c index 502dc046fbc62..a71e73707ef59 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1026,7 +1026,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; - atomic_set(&mm->has_pinned, 0); atomic64_set(&mm->pinned_vm, 0); memset(&mm->rss_stat, 0, sizeof(mm->rss_stat)); spin_lock_init(&mm->page_table_lock); diff --git a/mm/gup.c b/mm/gup.c index 8b513e1723b45..78416b0909873 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1292,8 +1292,8 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, BUG_ON(*locked != 1); } - if (flags & FOLL_PIN && !atomic_read(&mm->has_pinned)) - atomic_set(&mm->has_pinned, 1); + if (flags & FOLL_PIN && !test_bit(MMF_HAS_PINNED, &mm->flags)) + set_bit(MMF_HAS_PINNED, &mm->flags); /* * FOLL_PIN and FOLL_GET are mutually exclusive. Traditional behavior @@ -2617,8 +2617,9 @@ static int internal_get_user_pages_fast(unsigned long start, FOLL_FAST_ONLY))) return -EINVAL; - if (gup_flags & FOLL_PIN && !atomic_read(¤t->mm->has_pinned)) - atomic_set(¤t->mm->has_pinned, 1); + if (gup_flags & FOLL_PIN && + !test_bit(MMF_HAS_PINNED, ¤t->mm->flags)) + set_bit(MMF_HAS_PINNED, ¤t->mm->flags); if (!(gup_flags & FOLL_FAST_ONLY)) might_lock_read(¤t->mm->mmap_lock);