From patchwork Wed Jun 6 14:02:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10450291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DC14660146 for ; Wed, 6 Jun 2018 14:05:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B658729538 for ; Wed, 6 Jun 2018 14:05:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D48D29587; Wed, 6 Jun 2018 14:05:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3903329594 for ; Wed, 6 Jun 2018 14:03:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4464B6B0005; Wed, 6 Jun 2018 10:02:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3FB6A6B0006; Wed, 6 Jun 2018 10:02:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E7FE6B000D; Wed, 6 Jun 2018 10:02:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr0-f200.google.com (mail-wr0-f200.google.com [209.85.128.200]) by kanga.kvack.org (Postfix) with ESMTP id C36C06B0005 for ; Wed, 6 Jun 2018 10:02:58 -0400 (EDT) Received: by mail-wr0-f200.google.com with SMTP id b12-v6so3659444wrs.10 for ; Wed, 06 Jun 2018 07:02:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:date:from:to :cc:subject:message-id:mime-version:content-disposition:user-agent; bh=EwwIS+ktha9qhWt2vYuRPgc4uO8MCgXeUOKXGQnsuxI=; b=LEGJLBZF798nLvOL2LiWOM/7mbfGBIlXvEWc79akUcs8X7GDHUKxuJSeootST34DBj uohB213D13QS6+PQ4kqHlgQZm7SWy7aobewAUgHR1habw7FrxGVLFd/1tS9IxCMfquTD cjx0QwYydEjQ0fQA8ZOCq2FRs89b0KWPn2nTJGATJ6NhQcgdQ+9ofjEkvjnngAUlNOk3 Awcut4Qeht/RjkHEollZMD8dVeFrnblZcbKbc++uvRmviWysjIjF9E2cQJKTOQrkE5+c QH0656b29yrsdPF/V38Z9VsULpO0yLBvHBlCRuDv0k8h8zdKcqKDucREhH0j6UPtp69S jAqg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: APt69E3RlPc76kj7tUXgbmShc8iOwlvuZLRUetTzGsCtYLzCQy/yFyvw e/8xPhg590qS9EXAHRuA6ihhcDEpSYWz9N6QHWvyeEfXsTKvubqU27u7t1AMUYpRh2QplP/44Ms DrZ7bdWxINmaJgMbQA45TWg+bHmNRgeNBesYMdyMcmzM8rfwFfNkR77mzhTd16SRJbQ== X-Received: by 2002:a50:8b03:: with SMTP id l3-v6mr3731186edl.265.1528293778360; Wed, 06 Jun 2018 07:02:58 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIFuvNhaP9l2ZSazMmei4GN9O9c23K/YWpQY2h5J5Y4A+jQTxcprp7e9blDSt9OWtkQPunE X-Received: by 2002:a50:8b03:: with SMTP id l3-v6mr3731078edl.265.1528293777365; Wed, 06 Jun 2018 07:02:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528293777; cv=none; d=google.com; s=arc-20160816; b=lsn5iwLfRHuaTtnVoMCK5YyrVf0FozR3hi4i2YZnfaSyO/RTr7FAnjxR3kOcLOKx/f hE2jm6xrrtYRFqKqo9ua1G46xPw3VM3jc/3+diKyLVLE1r6Qh64qzcIDp8Q45Z8EZd1E LDLt7FLRyFQgIpc30nrA2QM+RYJgruplk4wRnE5Lh8g0qWS4mnnj1oX7UE7mTjJnsMoz gjj/7LaMHNCSqWBrnF7Ke3CpZKvyEHZ2fQ2+sphhHH/iFYYgXoFYkwOeHtKGyAIWGwMn nqvbjbHTD29eVkFk2FBw8SqPB34EBcHhEM5WL7uJN6R+FYvUa8j5VwZeWo6Tw7o0Ibxu 8HhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:content-disposition:mime-version:message-id:subject:cc :to:from:date:arc-authentication-results; bh=EwwIS+ktha9qhWt2vYuRPgc4uO8MCgXeUOKXGQnsuxI=; b=HvVrOa27ZkNkJsW+FHxWeENTT5c2uGilmYqzE/PLRZgJ40+d1XkaUAo2H8NitiTEAe suoIgYzf+5bVX1cy5Lid8A+GCRNfdfM6ljUkOMyUWttVmv2HAKLP9BDU7zx1oLVxLitM sCrLqwlpOkuk0ct9N0AYTWcdVA04H8Y0VSLR7AMrj5PEEb7EFpDfksxNzBS/YzaW6PIG UFBDtG84FTSjk6RpwxG7RV9I1SRzyade2/8eXkRt5kGbBGNpqzoGn1XKqcHuB2B+sWhW HwXyJCYNuOfxEJZcg841Pw1mrpfT23p+7YZ5DDoC3ehzbmYFLXFxGXoXAklrRO6suLPV REHw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id d6-v6si715978edk.225.2018.06.06.07.02.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Jun 2018 07:02:57 -0700 (PDT) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id E27F61C2731 for ; Wed, 6 Jun 2018 15:02:56 +0100 (IST) Received: (qmail 7866 invoked from network); 6 Jun 2018 14:02:56 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.237.73]) by 81.17.254.9 with ESMTPSA (DHE-RSA-AES256-SHA encrypted, authenticated); 6 Jun 2018 14:02:56 -0000 Date: Wed, 6 Jun 2018 15:02:55 +0100 From: Mel Gorman To: Andrew Morton Cc: Dave Hansen , mhocko@kernel.org, vbabka@suse.cz, Aaron Lu , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH] mremap: Increase LATENCY_LIMIT of mremap to reduce the number of TLB shootdowns Message-ID: <20180606140255.br5ztpeqdmwfto47@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline User-Agent: NeoMutt/20170912 (1.9.0) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Commit 5d1904204c99 ("mremap: fix race between mremap() and page cleanning") fixed races between mremap and other operations for both file-backed and anonymous mappings. The file-backed was the most critical as it allowed the possibility that data could be changed on a physical page after page_mkclean returned which could trigger data loss or data integrity issues. A customer reported that the cost of the TLBs for anonymous regressions was excessive and resulting in a 30-50% drop in performance overall since this commit on a microbenchmark. Unfortunately I neither have access to the test-case nor can I describe what it does other than saying that mremap operations dominate heavily. This patch increases the LATENCY_LIMIT to handle TLB flushes on a PMD boundary instead of every 64 pages. This reduces the number of TLB shootdowns by a factor of 8 which is not reported to completely restore performance but gets it within an acceptable percentage. The given metric here is simply described as "higher is better". Baseline that was known good 002: Metric: 91.05 004: Metric: 109.45 008: Metric: 73.08 016: Metric: 58.14 032: Metric: 61.09 064: Metric: 57.76 128: Metric: 55.43 Current 001: Metric: 54.98 002: Metric: 56.56 004: Metric: 41.22 008: Metric: 35.96 016: Metric: 36.45 032: Metric: 35.71 064: Metric: 35.73 128: Metric: 34.96 With patch 001: Metric: 61.43 002: Metric: 81.64 004: Metric: 67.92 008: Metric: 51.67 016: Metric: 50.47 032: Metric: 52.29 064: Metric: 50.01 128: Metric: 49.04 So for low threads, it's not restored but for larger number of threads, it's closer to the "known good" baseline. The downside is that PTL lock hold times will be slightly higher but it's unlikely that an mremap and another operation will contend on the same PMD. This is the first time I encountered a realistic workload that was mremap intensive (thousands of calls per second with small ranges dominating). Using a different mremap-intensive workload that is not representative of the real workload there is little difference observed outside of noise in the headline metrics However, the TLB shootdowns are reduced by 11% on average and at the peak, TLB shootdowns were reduced by 21%. Interrupts were sampled every second while the workload ran to get those figures. It's known that the figures will vary as the non-representative load is non-deterministic. An alternative patch was posted that should have significantly reduced the TLB flushes but unfortunately it does not perform as well as this version on the customer test case. If revisited, the two patches can stack on top of each other. Signed-off-by: Mel Gorman --- mm/mremap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 049470aa1e3e..b5017cb2e1e9 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -191,7 +191,7 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, drop_rmap_locks(vma); } -#define LATENCY_LIMIT (64 * PAGE_SIZE) +#define LATENCY_LIMIT (PMD_SIZE) unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma,