From patchwork Fri Apr 30 19:52:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37E21C433B4 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9C5B2613F8 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C5B2613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 140376B0072; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EF056B0074; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED0DD6B0073; Fri, 30 Apr 2021 15:52:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0137.hostedemail.com [216.40.44.137]) by kanga.kvack.org (Postfix) with ESMTP id C4CF36B0070 for ; Fri, 30 Apr 2021 15:52:35 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 820E5180CCDF9 for ; Fri, 30 Apr 2021 19:52:35 +0000 (UTC) X-FDA: 78090080670.07.CD92C06 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf07.hostedemail.com (Postfix) with ESMTP id 07AD7A0003AF for ; Fri, 30 Apr 2021 19:52:33 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=dcNLsbG8ESP+Ilb2xQiBJzAJk+o/uP7LiOGbYa0CH8k=; b=iJHs7S2FhfzxsM5ZRN/CH8+fjzCuZRuo/hgPczw1mcsHrnYWGrcytzQk4NwGC92fRXUCU uiSidepjrq/aPuGBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=dcNLsbG8ESP+Ilb2xQiBJzAJk+o/uP7LiOGbYa0CH8k=; b=sqVkVXJ54YWrcpEgKimcRpZ3iFRXUGqdiCVvBF7rKz+gE60KC7qjif9ujbBFpfoVFiQD9 nkrCNg1nPyBt6kLu5GjQhXzrZYswefKB5kwcfLnqoKD6JeXsBBhz4m++FtCPxxbXt2MkB0B H4vAE8H4X/yUazmvbq5X5t1k6OSMDnyzb+Qn0GTQqojPRFSwAOfsi7ZRbNQwbQ1rAURWhT2 Wt37f3z59KF3ATgtogcYZt6dQ9JZbsxo8rikFyYLuDzhTSDiAr0jOu1WhRdu6LPgSDLcKqo 3KEYNVWT76EXbKMvU6p1Qalm4p/j7DCJh1E+8KrDqNbbC/+f2cVLunOLzknA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 1F95B16030B; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0036A19F520; Fri, 30 Apr 2021 12:52:32 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 01/29] mm: export dump_mm Date: Fri, 30 Apr 2021 12:52:02 -0700 Message-Id: <20210430195232.30491-2-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=iJHs7S2F; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=sqVkVXJ5; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf07.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: 69maa3ek9rksmodriaos7nnwoekajzf6 X-Rspamd-Queue-Id: 07AD7A0003AF Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812353-35459 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is necessary in order to allow VM_BUG_ON_MM to be used in modules (I encountered the issue when adding VM_BUG_ON_MM in mmap locking functions). Signed-off-by: Michel Lespinasse --- mm/debug.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/debug.c b/mm/debug.c index 0bdda8407f71..979d505e2c6d 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -275,6 +275,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL(dump_mm); static bool page_init_poisoning __read_mostly = true; From patchwork Fri Apr 30 19:52:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44B43C43460 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4067613F8 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4067613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 475E46B0036; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 421E76B006E; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13CE36B0036; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id BE9D26B0036 for ; Fri, 30 Apr 2021 15:52:35 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 705943CF8 for ; Fri, 30 Apr 2021 19:52:35 +0000 (UTC) X-FDA: 78090080670.08.3118601 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf18.hostedemail.com (Postfix) with ESMTP id D8111200027D for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=NVL7JPD4xKaWrJCrCCQAhxOaIfpgRthzGvlvMAcHMhY=; b=seFAG+swPapZKqpejXszQzpaIdmpkFiiKJBqE4tgh7BHr7TiZ7gnyCpxzN8EIlmwm8g58 W9e7ERKDPuPRXxvCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=NVL7JPD4xKaWrJCrCCQAhxOaIfpgRthzGvlvMAcHMhY=; b=Hr12W9SJ/k10T+AhevxFw7Xi/5TWRi39IsNggQfIKgJvy8FTTJ/1TIWDD4/EM8/GYwZpv wOk+SqRuC5x5GH5yfrje+m/9tY/aG8W7ipIjF2hwVdNO3FcbbB82d+GTS7m9JoiNr0ywZrn mElAhRQk77OBX5vcMLgCS9rQAva0NBgBVAFWJzGc81wJJOeImQFD3aKpNbJ+r+0VMABNEyL 50XdZ6f4s2Y8lv/vqQRJllplvm4dz8f0IcOlja45ytEz53pLQ755IUI7QSltT6/N+Zwz8TK /sCXe+fw3IZD5K1NRhozh6yhxnVV821EBybHtmFvC/e3DjInR9QutLwILhGQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 1698016030A; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0638519F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 02/29] mmap locking API: mmap_lock_is_contended returns a bool Date: Fri, 30 Apr 2021 12:52:03 -0700 Message-Id: <20210430195232.30491-3-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8111200027D Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=seFAG+sw; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=Hr12W9SJ; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf18.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: pwt7sn3ejw3yx5pr3ewi5k9b6twen1ow Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812356-311821 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change mmap_lock_is_contended to return a bool value, rather than an int which the callers are then supposed to interpret as a bool. This is to ensure consistency with other mmap lock API functions (such as the trylock functions). Signed-off-by: Michel Lespinasse --- include/linux/mmap_lock.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 0540f0156f58..4e27f755766b 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -171,9 +171,9 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } -static inline int mmap_lock_is_contended(struct mm_struct *mm) +static inline bool mmap_lock_is_contended(struct mm_struct *mm) { - return rwsem_is_contended(&mm->mmap_lock); + return rwsem_is_contended(&mm->mmap_lock) != 0; } #endif /* _LINUX_MMAP_LOCK_H */ From patchwork Fri Apr 30 19:52:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56F24C433ED for ; Fri, 30 Apr 2021 19:52:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C900E613F8 for ; Fri, 30 Apr 2021 19:52:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C900E613F8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 707D56B006E; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66E076B0074; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 276576B0070; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id C114E6B006E for ; Fri, 30 Apr 2021 15:52:35 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 78EBE82499B9 for ; Fri, 30 Apr 2021 19:52:35 +0000 (UTC) X-FDA: 78090080670.05.21DB7A7 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf14.hostedemail.com (Postfix) with ESMTP id B72BAC0007E3 for ; Fri, 30 Apr 2021 19:52:17 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=sRgr3eTHMkYG0a033c5wSjP6+Yvrcvk4eJe6GnX/cXk=; b=pu4gEEk8gaS/ZROdf2FE6TRCwKA3WG48lPgW+lbTvRhfaZiXKDPKC9Ab/r9nTYZxiRuSc oCeVJTcBB8K0IfVAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=sRgr3eTHMkYG0a033c5wSjP6+Yvrcvk4eJe6GnX/cXk=; b=OJ/qR0VZF0/gCuHmcQvd9wL3u8ggpTpELa5K5Ixo9ImrXlgyebYkv+gh7VbA4Oqt247Jh 8Dbgd9DQXJjIO6xeGUV6CvAdM8lohz6e/MyQl9EnmjOSIGedHtdE5y21J/h2RzDYBxt/9O4 2BgjQihL+CXBHVw4WBP8WKugN0Zmgd7TthmDaEWPZJPdtSNW5Y4lVplSE8/LT2d7KprQmBb EvaxTSSuJXKuLgr8BEvcU8nFcvs51kLDtBc0yi7/v6HVis2w5RGQH3MrvycJhtJ/lyzvTFt wbL+fbDoTZeH4ZTYaqxXhiw7/u9ZMFL++Qe2UuCz6X7igqi/n60uPIaF4qrQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 244BA16030D; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0D7FE19F524; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 03/29] mmap locking API: name the return values Date: Fri, 30 Apr 2021 12:52:04 -0700 Message-Id: <20210430195232.30491-4-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=pu4gEEk8; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b="OJ/qR0VZ"; spf=pass (imf14.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B72BAC0007E3 X-Stat-Signature: argrg6177w9bs5edzkhy984d74y98tz3 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812337-465822 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the mmap locking API, the *_killable() functions return an error (or 0 on success), and the *_trylock() functions return a boolean (true on success). Rename the return values "int error" and "bool ok", respectively, rather than using "ret" for both cases which I find less readable. Signed-off-by: Michel Lespinasse --- include/linux/mmap_lock.h | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 4e27f755766b..8ff276a7560e 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -81,22 +81,22 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) static inline int mmap_write_lock_killable(struct mm_struct *mm) { - int ret; + int error; __mmap_lock_trace_start_locking(mm, true); - ret = down_write_killable(&mm->mmap_lock); - __mmap_lock_trace_acquire_returned(mm, true, ret == 0); - return ret; + error = down_write_killable(&mm->mmap_lock); + __mmap_lock_trace_acquire_returned(mm, true, !error); + return error; } static inline bool mmap_write_trylock(struct mm_struct *mm) { - bool ret; + bool ok; __mmap_lock_trace_start_locking(mm, true); - ret = down_write_trylock(&mm->mmap_lock) != 0; - __mmap_lock_trace_acquire_returned(mm, true, ret); - return ret; + ok = down_write_trylock(&mm->mmap_lock) != 0; + __mmap_lock_trace_acquire_returned(mm, true, ok); + return ok; } static inline void mmap_write_unlock(struct mm_struct *mm) @@ -120,22 +120,22 @@ static inline void mmap_read_lock(struct mm_struct *mm) static inline int mmap_read_lock_killable(struct mm_struct *mm) { - int ret; + int error; __mmap_lock_trace_start_locking(mm, false); - ret = down_read_killable(&mm->mmap_lock); - __mmap_lock_trace_acquire_returned(mm, false, ret == 0); - return ret; + error = down_read_killable(&mm->mmap_lock); + __mmap_lock_trace_acquire_returned(mm, false, !error); + return error; } static inline bool mmap_read_trylock(struct mm_struct *mm) { - bool ret; + bool ok; __mmap_lock_trace_start_locking(mm, false); - ret = down_read_trylock(&mm->mmap_lock) != 0; - __mmap_lock_trace_acquire_returned(mm, false, ret); - return ret; + ok = down_read_trylock(&mm->mmap_lock) != 0; + __mmap_lock_trace_acquire_returned(mm, false, ok); + return ok; } static inline void mmap_read_unlock(struct mm_struct *mm) From patchwork Fri Apr 30 19:52:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16ABEC433B4 for ; Fri, 30 Apr 2021 19:52:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4C6F613F8 for ; Fri, 30 Apr 2021 19:52:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4C6F613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 93E566B0070; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B7EF6B0071; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AD696B0073; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id D7CFE6B0071 for ; Fri, 30 Apr 2021 15:52:35 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7FCB7181AF5F4 for ; Fri, 30 Apr 2021 19:52:35 +0000 (UTC) X-FDA: 78090080670.39.43B353A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id 8F7436000113 for ; Fri, 30 Apr 2021 19:52:29 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XrNJKiD+gmmouaQBEDV3mKD7UNcz8msknbq0XbuTisY=; b=c/51/1unLYjIqVdN/mCnwDipupc8y9BaUfUWoaPw+8lcfZzM8MTrGl0GR8iCl86G/el+9 VkBqOtDJ9Dy7dbLAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XrNJKiD+gmmouaQBEDV3mKD7UNcz8msknbq0XbuTisY=; b=WpKVbH3zCSFSsOUWUEnw+ZlensRRNG8hYH+1jHHA3Hyq1YoY2rSXA/wnfwQ2Jc2ePKIeO eWfcLF3MFa7LhjKg4RzrswI+/6Lydp+qTR0Tt98CAhdDUki3z08/wYDH6Ulm41YdbbRgA50 r6AMvdvhQN2OO1GXwCcjLxj8tDxcX7HgfD8vGOL60hYuAJkE7QYQO8KSbZFJBEfIcrjeQf8 lSVKjvYwHUvz0RnDbdJwU+f3+Kr4q/GMwN6YXyeVLiU4MZQo9tXfg28en2dVA1y+GeSKHnJ W1Vv+qI7lse73tZxG0LvC5oySnCnQqHqywx2us2XmTIDnarGZZgsuI9PWLrA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 2635516030E; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 144E919F525; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 04/29] do_anonymous_page: use update_mmu_tlb() Date: Fri, 30 Apr 2021 12:52:05 -0700 Message-Id: <20210430195232.30491-5-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b="c/51/1un"; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=WpKVbH3z; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf25.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: 7emabjy3zyt76xwn9gqz34peqeemxtmb X-Rspamd-Queue-Id: 8F7436000113 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812349-940072 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: update_mmu_tlb() can be used instead of update_mmu_cache() when the page fault handler detects that it lost the race to another page fault. (TODO double check with Bibo Mao ) Signed-off-by: Michel Lespinasse --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 550405fc3b5e..59ff65cb3ab4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3567,7 +3567,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (!pte_none(*vmf->pte)) { - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; } From patchwork Fri Apr 30 19:52:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09B60C43460 for ; Fri, 30 Apr 2021 19:52:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7108613F8 for ; Fri, 30 Apr 2021 19:52:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7108613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C70B6B0071; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C2AB6B0085; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E4BB6B0082; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id DA13C6B0073 for ; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A4E974DDF for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.29.F6122A4 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf08.hostedemail.com (Postfix) with ESMTP id 209ED80192E2 for ; Fri, 30 Apr 2021 19:52:12 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=L3Bw9EVL1MYkwLbI9Xaunh3Hgdst8TtQR/SFegAk96c=; b=dgCd4PjRE2gRf4Fvqw/3C0vbv1RI67Lc5tiLnsYvcOefclwXVvfmLlYs5DRsYQv8k1q/r 50pZgxYv532WNKDBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=L3Bw9EVL1MYkwLbI9Xaunh3Hgdst8TtQR/SFegAk96c=; b=DKKWNug2D0nEHRfHWZkXtQrfqBuob06IxxmDvfQFn2hznrY9WylKR9Ccof2HLGZsjtp7S 2FmaRVMf0xslg7o2vheP+jGauUS20xpkBolDtINNlgJnv1lbHLVzpVcveZhBUq7ZUiNm4MC Hj460hi2GoUX1GpqD7eEp5M4AXVEteG8pY0Q9RUexOzp58SghweGhBz2z3WQLrrI6fq7XCW BC8Lui4j9v304CWh/t6Xmfx2SrkjOSi/SQRHWbWynMW6MDfvY1icVxfeilv7M+BdlSy9abF 0MY9BL6CoNWEykmhw2nR/38Pb5RNWT6+NrPyjAmvzV+SBy8kdnlfzSdzhN0w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 2A14E16030F; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 1A3DA19F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 05/29] do_anonymous_page: reduce code duplication Date: Fri, 30 Apr 2021 12:52:06 -0700 Message-Id: <20210430195232.30491-6-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 209ED80192E2 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=dgCd4PjR; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=DKKWNug2; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf08.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: i3a4rha74qm9mg1rehy4xye718mmbpkn Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812332-210429 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In do_anonymous_page(), we have separate cases for the zero page vs allocating new anonymous pages. However, once the pte entry has been computed, the rest of the handling (mapping and locking the page table, checking that we didn't lose a race with another page fault handler, etc) is identical between the two cases. This change reduces the code duplication between the two cases. Signed-off-by: Michel Lespinasse --- mm/memory.c | 85 +++++++++++++++++++++++------------------------------ 1 file changed, 37 insertions(+), 48 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 59ff65cb3ab4..217c31c616f4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3495,7 +3495,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) static vm_fault_t do_anonymous_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct page *page; + struct page *page = NULL; vm_fault_t ret = 0; pte_t entry; @@ -3525,77 +3525,66 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) !mm_forbids_zeropage(vma->vm_mm)) { entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address), vma->vm_page_prot)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (!pte_none(*vmf->pte)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); - goto unlock; - } - ret = check_stable_address_space(vma->vm_mm); - if (ret) - goto unlock; - /* Deliver the page fault to userland, check inside PT lock */ - if (userfaultfd_missing(vma)) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - return handle_userfault(vmf, VM_UFFD_MISSING); - } - goto setpte; + } else { + /* Allocate our own private page. */ + if (unlikely(anon_vma_prepare(vma))) + goto oom; + page = alloc_zeroed_user_highpage_movable(vma, vmf->address); + if (!page) + goto oom; + + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + goto oom_free_page; + cgroup_throttle_swaprate(page, GFP_KERNEL); + + /* + * The memory barrier inside __SetPageUptodate makes sure that + * preceding stores to the page contents become visible before + * the set_pte_at() write. + */ + __SetPageUptodate(page); + + entry = mk_pte(page, vma->vm_page_prot); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry)); } - /* Allocate our own private page. */ - if (unlikely(anon_vma_prepare(vma))) - goto oom; - page = alloc_zeroed_user_highpage_movable(vma, vmf->address); - if (!page) - goto oom; - - if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) - goto oom_free_page; - cgroup_throttle_swaprate(page, GFP_KERNEL); - - /* - * The memory barrier inside __SetPageUptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. - */ - __SetPageUptodate(page); - - entry = mk_pte(page, vma->vm_page_prot); - if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (!pte_none(*vmf->pte)) { update_mmu_tlb(vma, vmf->address, vmf->pte); - goto release; + goto unlock; } ret = check_stable_address_space(vma->vm_mm); if (ret) - goto release; + goto unlock; /* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); - put_page(page); + if (page) + put_page(page); return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, vma, vmf->address, false); - lru_cache_add_inactive_or_unevictable(page, vma); -setpte: + if (page) { + inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + page_add_new_anon_rmap(page, vma, vmf->address, false); + lru_cache_add_inactive_or_unevictable(page, vma); + } + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); + if (page) + put_page(page); return ret; -release: - put_page(page); - goto unlock; oom_free_page: put_page(page); oom: From patchwork Fri Apr 30 19:52:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 195F6C433ED for ; Fri, 30 Apr 2021 19:53:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BB06861476 for ; Fri, 30 Apr 2021 19:53:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BB06861476 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 95CD46B0081; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 33D466B0074; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9077D6B009C; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id 415CA6B0080 for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 030314FFA for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.38.38E36A2 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf27.hostedemail.com (Postfix) with ESMTP id 8BBFC80192ED for ; Fri, 30 Apr 2021 19:52:11 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=45X0hICb7YyLurpFubxOZ+gBL34g/MPyB7oW/aDWSiM=; b=5gnptII9qFi2vooFljqITSE4K6KmNZmsZmUPuqA2aMK3ARQMebMw8B1ZPqFQY3ds9B1t2 IhTIiOxATX/AphyDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=45X0hICb7YyLurpFubxOZ+gBL34g/MPyB7oW/aDWSiM=; b=D3rIm98jspzMJBMzKScap9IiDti8fm4LvQR4SgfW7kvabLrgbhZrxbLJrAf5SMZcI3Sj5 JNgjQhZfRHtA0ssRxK4rqcWQqNQYno/lv3EYRtp4IjbDAWCX6wkDXnYC+bQjKEzp4gTRRYn gfD/+gjMyeFu0OVJDY869fxTD7GebVa0YSAMoNtkKvifQ3Ero2ezc1mpuaBFMtJFzOYphYB G55SHqK9BMFKChPa/vhHCUs4RDHIs8uYd5tb4C88OxVErkXSgXU/aaSCE164Ky+YjPxltnc ey6wXUJ0vLUoy/ord59B2CZdz19w9q7g5Dci3Vyz1SaF8tlLzZ/hBvu3xwrg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 2F30F160310; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 2107F19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 06/29] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Date: Fri, 30 Apr 2021 12:52:07 -0700 Message-Id: <20210430195232.30491-7-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: h5af6si4k9gsyiiafjrce3f91werag75 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8BBFC80192ED Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=5gnptII9; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=D3rIm98j; spf=pass (imf27.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812331-453097 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the code needed to handle speculative page fault. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the speculative page fault handler is called from the architecture's page faulting code, and some code has to be added there to try speculative fault handling first. Signed-off-by: Michel Lespinasse --- mm/Kconfig | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 24c045b24b95..322bda319dea 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -872,4 +872,26 @@ config MAPPING_DIRTY_HELPERS config KMAP_LOCAL bool +config ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT + def_bool n + +config SPECULATIVE_PAGE_FAULT + bool "Speculative page faults" + default y + depends on ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT && MMU && SMP + help + Try to handle user space page faults without holding the mmap lock. + + Instead of blocking writers through the use of mmap lock, + the page fault handler merely verifies, at the end of the page + fault, that no writers have been running concurrently with it. + + In high concurrency situations, the speculative fault handler + gains a throughput advantage by avoiding having to update the + mmap lock reader count. + + If the check fails due to a concurrent writer, or due to hitting + an unsupported case, the fault handler falls back to classical + processing using the mmap read lock. + endmenu From patchwork Fri Apr 30 19:52:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5712C43460 for ; Fri, 30 Apr 2021 19:52:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 78DF9613F8 for ; Fri, 30 Apr 2021 19:52:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 78DF9613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 516A46B0099; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9A106B0085; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BB7F6B0075; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id D40656B0071 for ; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 96993181AF5D0 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.11.B8CFB24 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf04.hostedemail.com (Postfix) with ESMTP id 114903C5 for ; Fri, 30 Apr 2021 19:52:31 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=9K/3R1jhcIp0kkAhPFf8lQIqNdjGzS/Dm2et+ThCVUU=; b=t9iVfFmnuULsrM9oO60Hza/+gie/Dmpqd4k2nIs6SwThp5++nNppPCoAFZbqrB3NfW9JY dfvfPJa4Uo5WFApAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=9K/3R1jhcIp0kkAhPFf8lQIqNdjGzS/Dm2et+ThCVUU=; b=uEw7dwjyMOux+wwSljJXnIIqA/dK09jeoMJkyTCBCKK5n/WQFc2Jh1sBuHb7wdGSlG+vA szGj7oU2M48eP7P4pRSbOyMeEYx4k3EqrWwtUBkB1aW/51TMwsThVt5E3GH232n+7KvpnLu tLLAk7PW2CmZIXaRUyEgVLyjFQEmpyMQ0qQwayxn7wQTIvagBEQkLbYehTiXIWsm2UMNCQ+ e/RvX5+VotMZSpZWu7qd3QFEvGdFf8211H/4/EPu31iLqZ0UOtMlw0Ys0+BBakQNxvith3F GJJ8mE6+rBT78syxuJYCgCDidAoII+PzyuxW66nEDKW+9RUoK1PaljZpGcTw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 39D0D160311; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 2811719F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 07/29] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Fri, 30 Apr 2021 12:52:08 -0700 Message-Id: <20210430195232.30491-8-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=t9iVfFmn; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=uEw7dwjy; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf04.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: 4f3b3zxspgex6nmrqmx7jrbt9s6qr7dg X-Rspamd-Queue-Id: 114903C5 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812351-151239 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT so that the speculative fault handling code can be compiled on this architecture. Signed-off-by: Michel Lespinasse --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879d398e..a93e4ed7040e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -33,6 +33,7 @@ config X86_64 select NEED_DMA_MAP_STATE select SWIOTLB select ARCH_HAS_ELFCORE_COMPAT + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT config FORCE_DYNAMIC_FTRACE def_bool y From patchwork Fri Apr 30 19:52:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6647FC433ED for ; Fri, 30 Apr 2021 19:52:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 15481613F8 for ; Fri, 30 Apr 2021 19:52:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 15481613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C3DA26B008C; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A913E6B0087; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9EE36B007D; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id 0B4FD6B007B for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B152A45B3 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.09.2A3C03D Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id DA3F260006C0 for ; Fri, 30 Apr 2021 19:52:30 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=52xfj//MRWSrbPuyYyafld4PYpnTwPxqkOlVrkxjHfQ=; b=TpC4loRIPK/cBkY+s+mjCdbTT494398fFNDLLdCWbY8EjH0I7Q6xyMRMBhAMkCrGIlfJb WLj8GxiP95c6LU6Cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=52xfj//MRWSrbPuyYyafld4PYpnTwPxqkOlVrkxjHfQ=; b=Rcaa6wo5In854wslLI8fHcjWWfxBAKiir/rzHwUBRFIEnuVBPxCeFwAG02rsrjXM7Bq0Q VbrQJ58H7fljGUwNJiymWqrH99zWjGRobxOBG9yeQ19O42RornJsrL5dYksqq7Nnv5+v/l4 R4Z6gk0nTJMTPzhQ0jKX1jf/6k91/HazkacrdXkZupbgQ2NikLzFcxlAEMVMbKoc1y0ZTfh H2E6I24Yo4YWFnBohNLUIfbP5zt1E/C4reBuh9X3iD3aMEP901bBgjHrLdjRxSohz1fPXIM 1wXb4ixDcUSAyq6ywFADgJIDN0/q+GIkFpghPd0QHf4p2YkLcNXNEObrTppg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 3DAB9160312; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 2F05F19F524; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 08/29] mm: add FAULT_FLAG_SPECULATIVE flag Date: Fri, 30 Apr 2021 12:52:09 -0700 Message-Id: <20210430195232.30491-9-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=TpC4loRI; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=Rcaa6wo5; spf=pass (imf25.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: DA3F260006C0 X-Stat-Signature: rprwg1xgn6g7zj75asqykrtdghiit887 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812350-607233 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define the new FAULT_FLAG_SPECULATIVE flag, which indicates when we are attempting speculative fault handling (without holding the mmap lock). Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8ba434287387..021fdab5b721 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -434,6 +434,7 @@ extern pgprot_t protection_map[16]; * @FAULT_FLAG_REMOTE: The fault is not for current task/mm. * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch. * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals. + * @FAULT_FLAG_SPECULATIVE: The fault is handled without holding the mmap_sem. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two @@ -464,6 +465,7 @@ extern pgprot_t protection_map[16]; #define FAULT_FLAG_REMOTE 0x80 #define FAULT_FLAG_INSTRUCTION 0x100 #define FAULT_FLAG_INTERRUPTIBLE 0x200 +#define FAULT_FLAG_SPECULATIVE 0x400 /* * The default fault flags that should be used by most of the @@ -501,7 +503,8 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_SPECULATIVE, "SPECULATIVE" } /* * vm_fault is filled by the pagefault handler and passed to the vma's From patchwork Fri Apr 30 19:52:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F17AC433ED for ; Fri, 30 Apr 2021 19:52:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CFEE8613F8 for ; Fri, 30 Apr 2021 19:52:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFEE8613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BA3716B007B; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A166D6B0083; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BD4E6B0082; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0218.hostedemail.com [216.40.44.218]) by kanga.kvack.org (Postfix) with ESMTP id E11256B0074 for ; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9B91B4DDD for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.13.9E52A52 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf14.hostedemail.com (Postfix) with ESMTP id F30F5C0007E4 for ; Fri, 30 Apr 2021 19:52:18 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=BLfb3AqVSS1iZWKnBAI32as/v1HrLIIvwM3hMYePNUI=; b=8cMpbIOWRRdLq4qrH07ZoApzOUUqxKhtXC5TX3uGlt6u+RUpiVpcE9PrmNQePGgU2/K/e a5JOxs2y8H+oTEoCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=BLfb3AqVSS1iZWKnBAI32as/v1HrLIIvwM3hMYePNUI=; b=DpIS0cFIofpJRFqMCJ61gjVijTff1dHnb1HPNRT1ZgY0tp/JXtLD9wd4oL+SrkGgcVvBO KMzFQepiwY1MBrqX3MbFVG0nrARR9h8xY42hFlnylvzNSekRPkJlpHirQ5xtxFdzcAU3cB9 1gj9/pjpbfCUZguQONX4fogFhe0N7MEDP1Nk9+74LwpTp5KxOfkBaDq09oPjlQqpEMdGeWr LANhkPfVDjI7lUJ+HQNB5DgVXhPDmlON4GsxCTQDgKHIJg2HVSe2lchFlRpo3M5eUnVe6yG qK6biGFrXSuBT8XVyl8swV17MF/DsmFVvc5054DirfuOL+J7p3G4+H+kBQUw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 437BF16031E; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 3666A19F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 09/29] mm: add do_handle_mm_fault() Date: Fri, 30 Apr 2021 12:52:10 -0700 Message-Id: <20210430195232.30491-10-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=8cMpbIOW; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=DpIS0cFI; spf=pass (imf14.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: F30F5C0007E4 X-Stat-Signature: hfro3roneytyehk9w4wg54b57ixbesdj Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812338-686238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new do_handle_mm_fault function, which extends the existing handle_mm_fault() API by adding an mmap sequence count, to be used in the FAULT_FLAG_SPECULATIVE case. In the initial implementation, FAULT_FLAG_SPECULATIVE always fails (by returning VM_FAULT_RETRY). The existing handle_mm_fault() API is kept as a wrapper around do_handle_mm_fault() so that we do not have to immediately update every handle_mm_fault() call site. Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 12 +++++++++--- mm/memory.c | 10 +++++++--- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 021fdab5b721..d5988e78e6ab 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1724,9 +1724,15 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page); int invalidate_inode_page(struct page *page); #ifdef CONFIG_MMU -extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - struct pt_regs *regs); +extern vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + unsigned long seq, struct pt_regs *regs); +static inline vm_fault_t handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + struct pt_regs *regs) +{ + return do_handle_mm_fault(vma, address, flags, 0, regs); +} extern int fixup_user_fault(struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); diff --git a/mm/memory.c b/mm/memory.c index 217c31c616f4..8258ff93a055 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4509,11 +4509,15 @@ static inline void mm_account_fault(struct pt_regs *regs, * The mmap_lock may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, - unsigned int flags, struct pt_regs *regs) +vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + unsigned long seq, struct pt_regs *regs) { vm_fault_t ret; + if (flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; + __set_current_state(TASK_RUNNING); count_vm_event(PGFAULT); @@ -4555,7 +4559,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, return ret; } -EXPORT_SYMBOL_GPL(handle_mm_fault); +EXPORT_SYMBOL_GPL(do_handle_mm_fault); #ifndef __PAGETABLE_P4D_FOLDED /* From patchwork Fri Apr 30 19:52:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FA8FC43603 for ; Fri, 30 Apr 2021 19:52:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF2446145A for ; Fri, 30 Apr 2021 19:52:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF2446145A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 346606B0089; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D1DAE6B0082; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 513506B0087; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2E4106B0081 for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D82EC4FE6 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.08.3B2C6EB Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf22.hostedemail.com (Postfix) with ESMTP id 9EFAEC0007E5 for ; Fri, 30 Apr 2021 19:52:29 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Lp7zWyr2ROUjDnOgQqp9EncQocBAA2b6/i5LywQL7GQ=; b=LjusKaSBGYg8Bx3zS9rqTLDOjgNifpgywKlBuYYjCZ3gila8jYw1rVUqK+41wEFKbCucK R/wftX/VwnQp3lICg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Lp7zWyr2ROUjDnOgQqp9EncQocBAA2b6/i5LywQL7GQ=; b=JXvW+3Af/q77l6tKiG+MNtKTXIEkcUnjCsKluWpK4GaXu7jaRtcn8qmgMfjPMLgiGmz3N I26sPxQDnr/g+hd5JmENpCxgDTON4Ed1ERjv9/tzjYBWUr8n+mSvLpje0W47poDBiqcF4Fd +oHdyHHGpLh6UIP3XHnHx+GquBrLfZjyyvf3IZc2+mITquMHqa4ZFqmFEStEru5g2g621q8 nRGgWvQaKgM20f/t365uawgiPWw+eZxABFDa3rMTTKkTuLyYJ2h0Fy1iJiSC6GUQfIdOU4K 8RFXi6PN9XUQup5h8+tgSy1VG37RdPbgbujzNIG8pMwoF7chkXxPZRBy+w3Q== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 4D3D416031F; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 3D67319F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 10/29] mm: add per-mm mmap sequence counter for speculative page fault handling. Date: Fri, 30 Apr 2021 12:52:11 -0700 Message-Id: <20210430195232.30491-11-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9EFAEC0007E5 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=LjusKaSB; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=JXvW+3Af; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf22.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: ocagj16rrmt87s45h55tmrk8o6ugr458 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812349-915086 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The counter's write side is hooked into the existing mmap locking API: mmap_write_lock() increments the counter to the next (odd) value, and mmap_write_unlock() increments it again to the next (even) value. The counter's speculative read side is supposed to be used as follows: seq = mmap_seq_read_start(mm); if (seq & 1) goto fail; .... speculative handling here .... if (!mmap_seq_read_check(mm, seq) goto fail; This API guarantees that, if none of the "fail" tests abort speculative execution, the speculative code section did not run concurrently with any mmap writer. This is very similar to a seqlock, but both the writer and speculative readers are allowed to block. In the fail case, the speculative reader does not spin on the sequence counter; instead it should fall back to a different mechanism such as grabbing the mmap lock read side. Signed-off-by: Michel Lespinasse --- include/linux/mm_types.h | 4 +++ include/linux/mmap_lock.h | 58 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6613b26a8894..70882e628908 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -461,6 +461,10 @@ struct mm_struct { * counters */ struct rw_semaphore mmap_lock; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + unsigned long mmap_seq; +#endif + struct list_head mmlist; /* List of maybe swapped mm's. These * are globally strung together off diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8ff276a7560e..8f4eca2d0f43 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -8,8 +8,16 @@ #include #include -#define MMAP_LOCK_INITIALIZER(name) \ - .mmap_lock = __RWSEM_INITIALIZER((name).mmap_lock), +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +#define MMAP_LOCK_SEQ_INITIALIZER(name) \ + .mmap_seq = 0, +#else +#define MMAP_LOCK_SEQ_INITIALIZER(name) +#endif + +#define MMAP_LOCK_INITIALIZER(name) \ + .mmap_lock = __RWSEM_INITIALIZER((name).mmap_lock), \ + MMAP_LOCK_SEQ_INITIALIZER(name) DECLARE_TRACEPOINT(mmap_lock_start_locking); DECLARE_TRACEPOINT(mmap_lock_acquire_returned); @@ -63,13 +71,52 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + mm->mmap_seq = 0; +#endif } +static inline void __mmap_seq_write_lock(struct mm_struct *mm) +{ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + VM_BUG_ON_MM(mm->mmap_seq & 1, mm); + mm->mmap_seq++; + smp_wmb(); +#endif +} + +static inline void __mmap_seq_write_unlock(struct mm_struct *mm) +{ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + smp_wmb(); + mm->mmap_seq++; + VM_BUG_ON_MM(mm->mmap_seq & 1, mm); +#endif +} + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static inline unsigned long mmap_seq_read_start(struct mm_struct *mm) +{ + unsigned long seq; + + seq = READ_ONCE(mm->mmap_seq); + smp_rmb(); + return seq; +} + +static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq) +{ + smp_rmb(); + return seq == READ_ONCE(mm->mmap_seq); +} +#endif + static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, true, true); + __mmap_seq_write_lock(mm); } static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) @@ -77,6 +124,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); __mmap_lock_trace_acquire_returned(mm, true, true); + __mmap_seq_write_lock(mm); } static inline int mmap_write_lock_killable(struct mm_struct *mm) @@ -86,6 +134,8 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); error = down_write_killable(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, true, !error); + if (likely(!error)) + __mmap_seq_write_lock(mm); return error; } @@ -96,17 +146,21 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ok = down_write_trylock(&mm->mmap_lock) != 0; __mmap_lock_trace_acquire_returned(mm, true, ok); + if (likely(ok)) + __mmap_seq_write_lock(mm); return ok; } static inline void mmap_write_unlock(struct mm_struct *mm) { + __mmap_seq_write_unlock(mm); up_write(&mm->mmap_lock); __mmap_lock_trace_released(mm, true); } static inline void mmap_write_downgrade(struct mm_struct *mm) { + __mmap_seq_write_unlock(mm); downgrade_write(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, false, true); } From patchwork Fri Apr 30 19:52:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4B43C43460 for ; Fri, 30 Apr 2021 19:52:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 649406146D for ; Fri, 30 Apr 2021 19:52:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 649406146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 911E86B0073; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70EBD6B0078; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0F066B007E; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id 124656B007E for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C804182499B9 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.22.B27CA41 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf10.hostedemail.com (Postfix) with ESMTP id E6D4D40002E1 for ; Fri, 30 Apr 2021 19:52:24 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=0J/hy/JQlPleTiIXPLrlBogxDyG+sG63ZteLm7sGfKo=; b=wyAG608rKGunu7qqRpNcV3RhywqilJmsoDkuatV7pMb2CuBB2Z5fN14HlhxI88n50qQXn JOext8U+9dt1WmYDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=0J/hy/JQlPleTiIXPLrlBogxDyG+sG63ZteLm7sGfKo=; b=I7GqfkMzcJrmxSVWKzaFi3Mbfg0My9nLk6La+4wSu0ynTAHGbVd/YLF+eo94NlSin0y2x 5UJRM0G2jchXTm699A4f2yZErshAfuGTcwTU1+KKv7fPd1guFKelgOEdbBOltZG1xjZF6Hr cu7453EsgVC4XX/ZmrD7UtuP5ToQJjNcX3bWRZE95xJsL4I79FyjIFIr80NBa5p4b08KDHM XKTi1v+b5tKZc28JAcJGdSPPF4ww/d2hsg//WZqoozCHPYBVygjUUATnUIQ147goNNaLGCk M9dUpDnqaZNX8DgV5FOjh+BWX1KVhPjjD8OvB/QV47TAysg3vXstHO+Jy0Lw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 531B2160320; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 44ACA19F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 11/29] mm: rcu safe vma freeing Date: Fri, 30 Apr 2021 12:52:12 -0700 Message-Id: <20210430195232.30491-12-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E6D4D40002E1 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=wyAG608r; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=I7GqfkMz; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf10.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: 4c67bejbpsq9mkwazeusfbraqp4j1xt1 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812344-421709 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This prepares for speculative page faults looking up and copying vmas under protection of an rcu read lock, instead of the usual mmap read lock. Note - it might also be feasible to just use SLAB_TYPESAFE_BY_RCU when creating the vm_area_cachep, but that's probably too subtle to consider here. Signed-off-by: Michel Lespinasse --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 70882e628908..024970635921 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -304,12 +304,18 @@ struct vm_userfaultfd_ctx {}; struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 426cd0c51f9e..7c22bf2b1f9d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -369,9 +369,22 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Fri Apr 30 19:52:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C922BC433B4 for ; Fri, 30 Apr 2021 19:52:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6E188613F8 for ; Fri, 30 Apr 2021 19:52:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E188613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 71F076B007E; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CD876B0073; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97EE26B0074; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id 0DA1B6B007D for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B4A4A82499A8 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.31.A390D3A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf29.hostedemail.com (Postfix) with ESMTP id E31ACF2 for ; Fri, 30 Apr 2021 19:52:31 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=mkNeF95yymhY25nTphKSx+4hdLGj+j0544x2VC2sxV8=; b=Vzuvm5PMGxgNrmbQBB0aCENsJ84r2DFUqmlU9p6r1v+c+rf0FUVVAq3Ad/RKxVHckmML+ OgEgx80PSi7aAVFBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=mkNeF95yymhY25nTphKSx+4hdLGj+j0544x2VC2sxV8=; b=StQ30JKSY5VmWjfOF5DeEw34dLbaZjHYA1mDovwHFgabfvjbIbUeyEtBCodyalX1aE3dv WZV4Iy3RwL1bLk6Ku5564VxRlUAH4UkbhxPCt8Hel32RFwrHiRojFQqXWjNCE5Tr4qix53W ugwiblrYAzCpSFCYR1+/6qDGLXceGJZR/RmH2TpIds26S1XZFUHXjecDbt5WecbdD6mHtgs D+fkUERUM7dItifrCwSxK1QMaHvo+lpSoRMP1t3UE9tKkNJXFaEG2hZipSA/jY9XOvYt7Oj p6XkBOfn+YpGHDBwnthF7z4BbqfQBc3xT9WIps8gX7fyItZt7BLmw9ffQiiA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 5AEB3160321; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 4B8E819F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 12/29] x86/mm: attempt speculative mm faults first Date: Fri, 30 Apr 2021 12:52:13 -0700 Message-Id: <20210430195232.30491-13-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=Vzuvm5PM; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=StQ30JKS; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf29.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: ypcgwhyfmyxszr9yq9enzhyer5kjbfr1 X-Rspamd-Queue-Id: E31ACF2 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812351-28263 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. The speculative handling closely mirrors the non-speculative logic. This includes some x86 specific bits such as the access_error() call. This is why we chose to implement the speculative handling in arch/x86 rather than in common code. The vma is first looked up and copied, under protection of the rcu read lock. The mmap lock sequence count is used to verify the integrity of the copied vma, and passed to do_handle_mm_fault() to allow checking against races with mmap writers when finalizing the fault. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 42 +++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 5 +++++ include/linux/vm_event_item.h | 4 ++++ mm/vmstat.c | 4 ++++ 4 files changed, 55 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a73347e2cdfc..0e8abe43d032 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1219,6 +1219,10 @@ void do_user_addr_fault(struct pt_regs *regs, struct mm_struct *mm; vm_fault_t fault; unsigned int flags = FAULT_FLAG_DEFAULT; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct vm_area_struct pvma; + unsigned long seq; +#endif tsk = current; mm = tsk->mm; @@ -1316,6 +1320,41 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) + goto spf_abort; + rcu_read_lock(); + vma = find_vma(mm, address); + if (!vma || vma->vm_start > address) { + rcu_read_unlock(); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq)) + goto spf_abort; + vma = &pvma; + if (unlikely(access_error(error_code, vma))) + goto spf_abort; + fault = do_handle_mm_fault(vma, address, + flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR); + return; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +#endif + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1412,6 +1451,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 024970635921..d2bfffcbe364 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -300,6 +300,11 @@ struct vm_userfaultfd_ctx {}; * per VM-area/task. A VM area is any part of the process virtual memory * space that has a special rule for the page-fault handlers (ie a shared * library, the executable area etc). + * + * Note that speculative page faults make an on-stack copy of the VMA, + * so the structure size matters. + * (TODO - it would be preferable to copy only the required vma attributes + * rather than the entire vma). */ struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 18e75974d4e3..cc4f8d14e43f 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -120,6 +120,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SWAP SWAP_RA, SWAP_RA_HIT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + SPF_ATTEMPT, + SPF_ABORT, #endif NR_VM_EVENT_ITEMS }; diff --git a/mm/vmstat.c b/mm/vmstat.c index 74b2c374b86c..9ae1c27a549e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1365,6 +1365,10 @@ const char * const vmstat_text[] = { "swap_ra", "swap_ra_hit", #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + "spf_attempt", + "spf_abort", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Fri Apr 30 19:52:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E74DC433ED for ; Fri, 30 Apr 2021 19:52:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E86356145A for ; Fri, 30 Apr 2021 19:52:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E86356145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 012726B0087; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C3A7C6B009B; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E8D06B0082; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0218.hostedemail.com [216.40.44.218]) by kanga.kvack.org (Postfix) with ESMTP id 09A2A6B0078 for ; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B52BA181AF5F4 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.24.AC343B4 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf09.hostedemail.com (Postfix) with ESMTP id 9D2D260006C1 for ; Fri, 30 Apr 2021 19:52:28 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=AZMDU3YPUBP+A8E2sGu7mqnPCKXc6C4guT20cQnVLhI=; b=/pCAvRaOx2Si6FofNn8xF8A4WTxfnjNRKNUDSB8g0cwI/beqZTwY7cyDrJpAZu0D/iqo3 LQUWoCnxovEtWzLBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=AZMDU3YPUBP+A8E2sGu7mqnPCKXc6C4guT20cQnVLhI=; b=ZkO9R1pq60ZomZNC7gBkDtC53nRGzuc/y5a8x0+AHLNqjySZhHkBI3MVnw/GNC4XOaKpz AuHxRAomu7xORLhD1hDjxg1DPb9z84v9e8VwekRDtkFt/w/d/cvL9Ga/JKkiGTXTtB4k3Xu nSS/6Q9uOT3hkIFe8dy2U/5rg4Wx02H2Oo+kbx3vaYoVMYzMuqZKTuR5GHKJsrGlZxq1C7N hDXG6idcwcAw43SyrQKhk7slDaGxm5XGk92fRHU8dyIrcsMSw4/mRw/h35DQjLrZKPL8+Fa DHmlnOgonJqUpHWSCLosdIeh3ActgiuBby9lV33SymfaV0UyGKoWtugQjxMg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 61BFE160324; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 5277D19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 13/29] mm: add speculative_page_walk_begin() and speculative_page_walk_end() Date: Fri, 30 Apr 2021 12:52:14 -0700 Message-Id: <20210430195232.30491-14-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b="/pCAvRaO"; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=ZkO9R1pq; spf=pass (imf09.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9D2D260006C1 X-Stat-Signature: nd3yjbuj7uqruzzxofsrsejj8upej6bc Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812348-275426 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Speculative page faults will use these to protect against races with page table reclamation. This could always be handled by disabling local IRQs as the fast GUP code does; however speculative page faults do not need to protect against races with THP page splitting, so a weaker rcu read lock is sufficient in the MMU_GATHER_RCU_TABLE_FREE case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 8258ff93a055..b28047765de7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2566,6 +2566,28 @@ int apply_to_existing_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_existing_page_range); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + +/* + * speculative_page_walk_begin() ... speculative_page_walk_end() protects + * against races with page table reclamation. + * + * This is similar to what fast GUP does, but fast GUP also needs to + * protect against races with THP page splitting, so it always needs + * to disable interrupts. + * Speculative page faults only need to protect against page table reclamation, + * so rcu_read_lock() is sufficient in the MMU_GATHER_RCU_TABLE_FREE case. + */ +#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE +#define speculative_page_walk_begin() rcu_read_lock() +#define speculative_page_walk_end() rcu_read_unlock() +#else +#define speculative_page_walk_begin() local_irq_disable() +#define speculative_page_walk_end() local_irq_enable() +#endif + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* * handle_pte_fault chooses page fault handler according to an entry which was * read non-atomically. Before making any commitment, on those architectures From patchwork Fri Apr 30 19:52:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F19C2C433B4 for ; Fri, 30 Apr 2021 19:52:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9CF566146D for ; Fri, 30 Apr 2021 19:52:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CF566146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E16DE6B0075; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0CC16B0082; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CF5E6B0073; Fri, 30 Apr 2021 15:52:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0037.hostedemail.com [216.40.44.37]) by kanga.kvack.org (Postfix) with ESMTP id EDD3E6B0075 for ; Fri, 30 Apr 2021 15:52:36 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AE4EE180945C5 for ; Fri, 30 Apr 2021 19:52:36 +0000 (UTC) X-FDA: 78090080712.27.9396D4E Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf26.hostedemail.com (Postfix) with ESMTP id A6DD740002E2 for ; Fri, 30 Apr 2021 19:52:26 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=/loUZ02DI4vLNq42pqfAgn2FTqDP65md0qhN70FxHpk=; b=IwSGLPkn4zooFRT7/gWLCebPara7aRpy5Qj+LPZ0tukfbpfTRenz2PxZwOHaT0wEqGD31 h5DpMgPRFwLsUemAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=/loUZ02DI4vLNq42pqfAgn2FTqDP65md0qhN70FxHpk=; b=QCnRPKzkUxRk2aTYrFT8g5Z9E+pght0bvblf0FiErmKwNeEu/SgIE+yk0D47V3Sml/KhL ozkbM6lBlWu8LbZm7ex+QeqNn3HC68pgPjG/DJaqj/7bS9xI9cuUZVqUGBX9T5SkO6cUko/ /UW9i09OLWfQVhfmU7z7s9HfDlTQwokpKDbzuvWEacFEH3m7ga9Nzzahwi6mbNsGAqIjGIV HYj0ov47CZyiJcfF6gb98pOvWiAmK52W2C0l3KeS0Rr0Na4LS82iTJEwr0G+jMSxCfKQ6xW g8HfGfrU63q1nT0cE//UZWITnSuZc2MWRjuQ+HcfxBv+APJFbEAZvx+7kqIw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 68992160325; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 59C6019F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 14/29] mm: refactor __handle_mm_fault() / handle_pte_fault() Date: Fri, 30 Apr 2021 12:52:15 -0700 Message-Id: <20210430195232.30491-15-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=IwSGLPkn; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=QCnRPKzk; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf26.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: 7unnn9gb94gwcgda4rsy7a513wqgyc8p X-Rspamd-Queue-Id: A6DD740002E2 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812346-364927 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the code that initializes vmf->pte and vmf->orig_pte from handle_pte_fault() to its single call site in __handle_mm_fault(). This ensures vmf->pte is now initialized together with the higher levels of the page table hierarchy. This also prepares for speculative page fault handling, where the entire page table walk (higher levels down to ptes) needs special care in the speculative case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 98 ++++++++++++++++++++++++++--------------------------- 1 file changed, 49 insertions(+), 49 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index b28047765de7..45696166b10f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3538,7 +3538,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; - /* See comment in handle_pte_fault() */ + /* See comment in __handle_mm_fault() */ if (unlikely(pmd_trans_unstable(vmf->pmd))) return 0; @@ -3819,7 +3819,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } - /* See comment in handle_pte_fault() */ + /* See comment in __handle_mm_fault() */ if (pmd_devmap_trans_unstable(vmf->pmd)) return 0; @@ -4275,53 +4275,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; - if (unlikely(pmd_none(*vmf->pmd))) { - /* - * Leave __pte_alloc() until later: because vm_ops->fault may - * want to allocate huge page, and if we expose page table - * for an instant, it will be difficult to retract from - * concurrent faults and from rmap lookups. - */ - vmf->pte = NULL; - } else { - /* - * If a huge pmd materialized under us just retry later. Use - * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead - * of pmd_trans_huge() to ensure the pmd didn't become - * pmd_trans_huge under us and then back to pmd_none, as a - * result of MADV_DONTNEED running immediately after a huge pmd - * fault in a different thread of this mm, in turn leading to a - * misleading pmd_trans_huge() retval. All we have to ensure is - * that it is a regular pmd that we can walk with - * pte_offset_map() and we can do that through an atomic read - * in C, which is what pmd_trans_unstable() provides. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return 0; - /* - * A regular pmd is established and it can't morph into a huge - * pmd from under us anymore at this point because we hold the - * mmap_lock read mode and khugepaged takes it in write mode. - * So now it's safe to run pte_offset_map(). - */ - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - vmf->orig_pte = *vmf->pte; - - /* - * some architectures can have larger ptes than wordsize, - * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and - * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic - * accesses. The code below just needs a consistent view - * for the ifs and we later double check anyway with the - * ptl lock held. So here a barrier will do. - */ - barrier(); - if (pte_none(vmf->orig_pte)) { - pte_unmap(vmf->pte); - vmf->pte = NULL; - } - } - if (!vmf->pte) { if (vma_is_anonymous(vmf->vma)) return do_anonymous_page(vmf); @@ -4461,6 +4414,53 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, } } + if (unlikely(pmd_none(*vmf.pmd))) { + /* + * Leave __pte_alloc() until later: because vm_ops->fault may + * want to allocate huge page, and if we expose page table + * for an instant, it will be difficult to retract from + * concurrent faults and from rmap lookups. + */ + vmf.pte = NULL; + } else { + /* + * If a huge pmd materialized under us just retry later. Use + * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead + * of pmd_trans_huge() to ensure the pmd didn't become + * pmd_trans_huge under us and then back to pmd_none, as a + * result of MADV_DONTNEED running immediately after a huge pmd + * fault in a different thread of this mm, in turn leading to a + * misleading pmd_trans_huge() retval. All we have to ensure is + * that it is a regular pmd that we can walk with + * pte_offset_map() and we can do that through an atomic read + * in C, which is what pmd_trans_unstable() provides. + */ + if (pmd_devmap_trans_unstable(vmf.pmd)) + return 0; + /* + * A regular pmd is established and it can't morph into a huge + * pmd from under us anymore at this point because we hold the + * mmap_lock read mode and khugepaged takes it in write mode. + * So now it's safe to run pte_offset_map(). + */ + vmf.pte = pte_offset_map(vmf.pmd, vmf.address); + vmf.orig_pte = *vmf.pte; + + /* + * some architectures can have larger ptes than wordsize, + * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and + * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic + * accesses. The code below just needs a consistent view + * for the ifs and we later double check anyway with the + * ptl lock held. So here a barrier will do. + */ + barrier(); + if (pte_none(vmf.orig_pte)) { + pte_unmap(vmf.pte); + vmf.pte = NULL; + } + } + return handle_pte_fault(&vmf); } From patchwork Fri Apr 30 19:52:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4673FC433B4 for ; Fri, 30 Apr 2021 19:53:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E14816145A for ; Fri, 30 Apr 2021 19:53:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E14816145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0CD4A6B0098; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F9D96B0083; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81BF36B009E; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 335E06B0095 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E2097181AF5F4 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.28.2C953A1 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf29.hostedemail.com (Postfix) with ESMTP id 06510F6 for ; Fri, 30 Apr 2021 19:52:32 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=rYXn/93pk6t1tplCIUNdeG9K949Nx8Nhz+Jc2wP0R1Q=; b=newF3cfvGTS9NZHu8Z9WaTA6N891W7+LcXoBNpRst00ZibYoGbtRvwmRC4rI9PeS1dVL4 ywG6C8+DYN7OLn5Ag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=rYXn/93pk6t1tplCIUNdeG9K949Nx8Nhz+Jc2wP0R1Q=; b=lq1GjaSqce8boFrQcjPEUKQ/j7fShwyrCrWm0+mf8OdhWocXr/ZEjuG1q+Eb/JSPdxf0v MF4X4ZL/cUAg/T8w08i5+YfyObI/RWtNRFEj6Hoj1RofQGK6HXxcyyBVGnkZB73xigLFRg1 /nt84j2+hYZpwF5R1SDZ0ddaIBMppSckoH+lpr18YgkWxSxK4bf1XpcRgR0/BBVWGlyytcT YdZLpB5op6orZRAsS9mb+EZFHVjyicfFkAcIKsWPbiEVzx63ALx8bNrVSu5Zp+VhIE/spFt OAIq04qEeqX2U1tCOto3PsN2VdkhoMmXwhclJ77F1/sDDkeYRTf+DCc/5etQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 70663160327; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 60D9C19F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 15/29] mm: implement speculative handling in __handle_mm_fault(). Date: Fri, 30 Apr 2021 12:52:16 -0700 Message-Id: <20210430195232.30491-16-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=newF3cfv; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=lq1GjaSq; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf29.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: zp9hngybrowhnj4zu6urrr9kig4piki6 X-Rspamd-Queue-Id: 06510F6 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812352-988375 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The speculative path calls speculative_page_walk_begin() before walking the page table tree to prevent page table reclamation. The logic is otherwise similar to the non-speculative path, but with additional restrictions: in the speculative path, we do not handle huge pages or wiring new pages tables. Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 4 +++ mm/memory.c | 77 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d5988e78e6ab..dee8a4833779 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -525,6 +525,10 @@ struct vm_fault { }; unsigned int flags; /* FAULT_FLAG_xxx flags * XXX: should really be 'const' */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + unsigned long seq; + pmd_t orig_pmd; +#endif pmd_t *pmd; /* Pointer to pmd entry matching * the 'address' */ pud_t *pud; /* Pointer to pud entry matching diff --git a/mm/memory.c b/mm/memory.c index 45696166b10f..3f5c3d6c0197 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4329,7 +4329,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * return value. See filemap_fault() and __lock_page_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, unsigned long seq) { struct vm_fault vmf = { .vma = vma, @@ -4344,6 +4344,79 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + if (flags & FAULT_FLAG_SPECULATIVE) { + pgd_t pgdval; + p4d_t p4dval; + pud_t pudval; + + vmf.seq = seq; + + speculative_page_walk_begin(); + pgd = pgd_offset(mm, address); + pgdval = READ_ONCE(*pgd); + if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) + goto spf_fail; + + p4d = p4d_offset(pgd, address); + p4dval = READ_ONCE(*p4d); + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) + goto spf_fail; + + vmf.pud = pud_offset(p4d, address); + pudval = READ_ONCE(*vmf.pud); + if (pud_none(pudval) || unlikely(pud_bad(pudval)) || + unlikely(pud_trans_huge(pudval)) || + unlikely(pud_devmap(pudval))) + goto spf_fail; + + vmf.pmd = pmd_offset(vmf.pud, address); + vmf.orig_pmd = READ_ONCE(*vmf.pmd); + + /* + * pmd_none could mean that a hugepage collapse is in + * progress in our back as collapse_huge_page() mark + * it before invalidating the pte (which is done once + * the IPI is catched by all CPU and we have interrupt + * disabled). For this reason we cannot handle THP in + * a speculative way since we can't safely identify an + * in progress collapse operation done in our back on + * that PMD. + */ + if (unlikely(pmd_none(vmf.orig_pmd) || + is_swap_pmd(vmf.orig_pmd) || + pmd_trans_huge(vmf.orig_pmd) || + pmd_devmap(vmf.orig_pmd))) + goto spf_fail; + + /* + * The above does not allocate/instantiate page-tables because + * doing so would lead to the possibility of instantiating + * page-tables after free_pgtables() -- and consequently + * leaking them. + * + * The result is that we take at least one non-speculative + * fault per PMD in order to instantiate it. + */ + + vmf.pte = pte_offset_map(vmf.pmd, address); + vmf.orig_pte = READ_ONCE(*vmf.pte); + barrier(); + if (pte_none(vmf.orig_pte)) { + pte_unmap(vmf.pte); + vmf.pte = NULL; + } + + speculative_page_walk_end(); + + return handle_pte_fault(&vmf); + + spf_fail: + speculative_page_walk_end(); + return VM_FAULT_RETRY; + } +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -4563,7 +4636,7 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, if (unlikely(is_vm_hugetlb_page(vma))) ret = hugetlb_fault(vma->vm_mm, vma, address, flags); else - ret = __handle_mm_fault(vma, address, flags); + ret = __handle_mm_fault(vma, address, flags, seq); if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); From patchwork Fri Apr 30 19:52:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0FFBC43611 for ; Fri, 30 Apr 2021 19:53:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 95D9C61483 for ; Fri, 30 Apr 2021 19:53:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 95D9C61483 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6280C6B0088; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 209196B008A; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B039D6B0089; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 112E86B0081 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CF69D4FEC for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.29.25E5F66 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf20.hostedemail.com (Postfix) with ESMTP id 18894F4 for ; Fri, 30 Apr 2021 19:52:28 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JLbOD6CQvK5+Vbi6lHoPmCez/F/O/tswSP9m74uqZKA=; b=ZHIRuBYI42TPV0EKvuDcckJ/sm5j5f0A5ghiyVmRets6zFEr5EvGBrfObyqWSuNH0ixzZ miFe3Szsn+X1IdJCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JLbOD6CQvK5+Vbi6lHoPmCez/F/O/tswSP9m74uqZKA=; b=IEDqf76yGYIduS5KKKlHNO9UJwI6AkwAdi+FRipbomT86Si8cIyNoBlVo6LnSP2V9dZiB PsYJDHd97hdwOEUEvnaCnBUwcGglz4ElbbVSrNX4ZTBSEu2cT9bGtWbTs23fFB5ZzTIM/GN /TtD6alSyHZ/E45XIpaHdYoDLFV3Qs3cNEljS4ZmsalC/c3+2OVgKtAVV+yreeuIfq7JdqA 5xDkj/SM8+Hmegt72zXCejv9qyp9f7w9dy2gBe23gaGmz32wlP6BQDp0O8y/83K0xdZWlUW bzoH4Ih6nFnIPoUHZ+DVDwakUnpXPjUDXm8IoLfrbS+H6ZH4XbNe08xo1YHw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 773F4160328; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 682A219F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 16/29] mm: add pte_map_lock() and pte_spinlock() Date: Fri, 30 Apr 2021 12:52:17 -0700 Message-Id: <20210430195232.30491-17-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 18894F4 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=ZHIRuBYI; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=IEDqf76y; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf20.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: ffg5d4o3z1ji9k57ib1s98sh8iuokuxe Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812348-838560 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_map_lock() and pte_spinlock() are used by fault handlers to ensure the pte is mapped and locked before they commit the faulted page to the mm's address space at the end of the fault. The functions differ in their preconditions; pte_map_lock() expects the pte to be unmapped prior to the call, while pte_spinlock() expects it to be already mapped. In the speculative fault case, the functions verify, after locking the pte, that the mmap sequence count has not changed since the start of the fault, and thus that no mmap lock writers have been running concurrently with the fault. After that point the page table lock serializes any further races with concurrent mmap lock writers. If the mmap sequence count check fails, both functions will return false with the pte being left unmapped and unlocked. Signed-off-by: Michel Lespinasse Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/mm.h | 36 +++++++++++++++++++++++++ mm/memory.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dee8a4833779..8124cd53ce15 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3183,5 +3183,41 @@ extern int sysctl_nr_trim_pages; void mem_dump_obj(void *object); +#ifdef CONFIG_MMU +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + +bool __pte_map_lock(struct vm_fault *vmf); + +static inline bool pte_map_lock(struct vm_fault *vmf) +{ + VM_BUG_ON(vmf->pte); + return __pte_map_lock(vmf); +} + +static inline bool pte_spinlock(struct vm_fault *vmf) +{ + VM_BUG_ON(!vmf->pte); + return __pte_map_lock(vmf); +} + +#else /* !CONFIG_SPECULATIVE_PAGE_FAULT */ + +static inline bool pte_map_lock(struct vm_fault *vmf) +{ + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); + return true; +} + +static inline bool pte_spinlock(struct vm_fault *vmf) +{ + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + return true; +} + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ +#endif /* CONFIG_MMU */ + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index 3f5c3d6c0197..e2f9e4c096dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2586,6 +2586,72 @@ EXPORT_SYMBOL_GPL(apply_to_existing_page_range); #define speculative_page_walk_end() local_irq_enable() #endif +bool __pte_map_lock(struct vm_fault *vmf) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + pmd_t pmdval; +#endif + pte_t *pte = vmf->pte; + spinlock_t *ptl; + + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + if (!pte) + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + spin_lock(vmf->ptl); + return true; + } + + speculative_page_walk_begin(); + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + goto fail; + /* + * The mmap sequence count check guarantees that the page + * tables are still valid at that point, and + * speculative_page_walk_begin() ensures that they stay around. + */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * We check if the pmd value is still the same to ensure that there + * is not a huge collapse operation in progress in our back. + */ + pmdval = READ_ONCE(*vmf->pmd); + if (!pmd_same(pmdval, vmf->orig_pmd)) + goto fail; +#endif + ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + if (!pte) + pte = pte_offset_map(vmf->pmd, vmf->address); + /* + * Try locking the page table. + * + * Note that we might race against zap_pte_range() which + * invalidates TLBs while holding the page table lock. + * We are still under the speculative_page_walk_begin() section, + * and zap_pte_range() could thus deadlock with us if we tried + * using spin_lock() here. + * + * We also don't want to retry until spin_trylock() succeeds, + * because of the starvation potential against a stream of lockers. + */ + if (unlikely(!spin_trylock(ptl))) + goto fail; + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + goto unlock_fail; + speculative_page_walk_end(); + vmf->pte = pte; + vmf->ptl = ptl; + return true; + +unlock_fail: + spin_unlock(ptl); +fail: + if (pte) + pte_unmap(pte); + speculative_page_walk_end(); + return false; +} + #endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ /* From patchwork Fri Apr 30 19:52:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CBE6C43461 for ; Fri, 30 Apr 2021 19:53:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F15116146D for ; Fri, 30 Apr 2021 19:53:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F15116146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 600196B0092; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B78D56B009B; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 250F26B009C; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id 10A126B0074 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D29361809B362 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.40.5DD8327 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf28.hostedemail.com (Postfix) with ESMTP id 4E976200027A for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=3M8/87TWLrXUkV85tkfyDqeQTIij8rhB4JQYDhsc088=; b=IZjIOz6DZDI+XBf2HUWjGljUENDtk9JZDSEfDvngxRCnnk8zbadfNKvDaZKujidNQsLue opWTz5oBYI1wlRSCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=3M8/87TWLrXUkV85tkfyDqeQTIij8rhB4JQYDhsc088=; b=t7R2CPkdcuf0Gtgd++7LItWfcxuIgKHJzCb7pW1eIcDGF66HMlZ+vKl1dmodbHzkmn4XY uX99g0LXMoO9BY+cYVTdhCe9dOIoWjm4OGn76klITLZI9OtrTr1DJoh6yK+j7cnmDPs/vvy 6OKuko1PQapizZu+rs42+H1clr6su6rn0M5EzzUzjdUobY6IJjfrB1fAKROAhpKnpPq1bpx M2YUFn1HRw+hrmrfnCXrbuAqU8qOxZX1Aw37EwxS2pZmyFalbAe0npuqcutQrKKDWM+a6pJ PEhvvDkTl8iSpJKt6TuzvJhxnrZfVOQCo3/r4QjS7G+AjVIoAnj2vUiMTmEg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 7F1CD16032A; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 6F2ED19F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 17/29] mm: implement speculative handling in do_anonymous_page() Date: Fri, 30 Apr 2021 12:52:18 -0700 Message-Id: <20210430195232.30491-18-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=IZjIOz6D; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=t7R2CPkd; spf=pass (imf28.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4E976200027A X-Stat-Signature: qptook6ktef4k67s4qousdcs7tbt1hz4 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812358-854626 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change do_anonymous_page() to handle the speculative case. This involves aborting speculative faults if they have to allocate a new anon_vma, and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Signed-off-by: Michel Lespinasse --- mm/memory.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e2f9e4c096dd..d95826c48f1d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3615,8 +3615,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vma->vm_page_prot)); } else { /* Allocate our own private page. */ - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; + if (__anon_vma_prepare(vma)) + goto oom; + } page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!page) goto oom; @@ -3637,8 +3641,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = pte_mkwrite(pte_mkdirty(entry)); } - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto release; + } if (!pte_none(*vmf->pte)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto unlock; @@ -3653,6 +3659,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); if (page) put_page(page); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; return handle_userfault(vmf, VM_UFFD_MISSING); } @@ -3670,6 +3678,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return 0; unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); +release: if (page) put_page(page); return ret; From patchwork Fri Apr 30 19:52:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F566C433ED for ; Fri, 30 Apr 2021 19:53:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D5FC46146D for ; Fri, 30 Apr 2021 19:53:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5FC46146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C4CC26B0074; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E0496B0098; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61FE16B0082; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 077506B0088 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CA08382499A8 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.24.C2C9676 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf24.hostedemail.com (Postfix) with ESMTP id 43175A0003A2 for ; Fri, 30 Apr 2021 19:52:26 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=M/Ev9iks2mpuHbjoGtjSBN2w1TDQW6+EFN9w5Lcsf9s=; b=qorJ25mBWnaXh0kvHjZNSgrUysLUviTXD8V7S8+PwBGxSEadwdDSK9yKOVct7cU71wJ+c lJxszXP8H/LLEEjBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=M/Ev9iks2mpuHbjoGtjSBN2w1TDQW6+EFN9w5Lcsf9s=; b=MN4K5XVli33rheSlcHq6WD+tJr0i33+3ECUXM250WDhfV/hi/b9nxA2FOVZ2OJ4c29FMR PyUziXaigVfVLZ1msDdWuOEdkbN3if8wd+k7eOw4ytg1Hxosv0a4tz95lFMYMjH2RiIZHgX /A+eAocU1836zWv3dQ21RQfQGT5kn7sVQa9NF8CJ+0ZfoRMvHeEO0fZjcI4/tALfGyNfpiK LLM0GR1pOO3Bk/qWTd68h7gbggzMpcm0odD7zAxxEftdcjUxSyIh1TK5iMS0d2YrvWzYXfA FeEzFo+fcTRfEqx93bgoQS6GbIIwnE2+6cPHHMwNSOpAI7T5V02hJNd1PxDw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 85DF616032B; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 765F619F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 18/29] mm: enable speculative fault handling through do_anonymous_page() Date: Fri, 30 Apr 2021 12:52:19 -0700 Message-Id: <20210430195232.30491-19-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=qorJ25mB; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=MN4K5XVl; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf24.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: gpwsu63ggs7ed19pxb3yhe79hyzjq5od X-Rspamd-Queue-Id: 43175A0003A2 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812346-201776 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: in x86 fault handler, only attempt spf if the vma is anonymous. In do_handle_mm_fault(), let speculative page faults proceed as long as they fall into anonymous vmas. This enables the speculative handling code in __handle_mm_fault() and do_anonymous_page(). In handle_pte_fault(), if vmf->pte is set (the original pte was not pte_none), catch speculative faults and return VM_FAULT_RETRY as those cases are not implemented yet. Also assert that do_fault() is not reached in the speculative case. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 2 +- mm/memory.c | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 0e8abe43d032..463061186827 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1327,7 +1327,7 @@ void do_user_addr_fault(struct pt_regs *regs, goto spf_abort; rcu_read_lock(); vma = find_vma(mm, address); - if (!vma || vma->vm_start > address) { + if (!vma || vma->vm_start > address || !vma_is_anonymous(vma)) { rcu_read_unlock(); goto spf_abort; } diff --git a/mm/memory.c b/mm/memory.c index d95826c48f1d..eceb1b6e904c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4120,6 +4120,8 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ @@ -4357,6 +4359,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) return do_fault(vmf); } + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); @@ -4685,8 +4692,7 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, { vm_fault_t ret; - if (flags & FAULT_FLAG_SPECULATIVE) - return VM_FAULT_RETRY; + VM_BUG_ON((flags & FAULT_FLAG_SPECULATIVE) && !vma_is_anonymous(vma)); __set_current_state(TASK_RUNNING); @@ -4708,10 +4714,12 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, if (flags & FAULT_FLAG_USER) mem_cgroup_enter_user_fault(); - if (unlikely(is_vm_hugetlb_page(vma))) + if (unlikely(is_vm_hugetlb_page(vma))) { + VM_BUG_ON(flags & FAULT_FLAG_SPECULATIVE); ret = hugetlb_fault(vma->vm_mm, vma, address, flags); - else + } else { ret = __handle_mm_fault(vma, address, flags, seq); + } if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); From patchwork Fri Apr 30 19:52:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 328A7C43461 for ; Fri, 30 Apr 2021 19:53:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D5C836145A for ; Fri, 30 Apr 2021 19:53:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5C836145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 33CF46B0080; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A5B16B0093; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C89DD6B0095; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0104.hostedemail.com [216.40.44.104]) by kanga.kvack.org (Postfix) with ESMTP id 3E1EE6B007D for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F301E18069D46 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.21.6BBD405 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id 1762160006C0 for ; Fri, 30 Apr 2021 19:52:31 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=xyC4YWMYQXg5okBOT+I25AzHoSz3IJwuaB8/iA0p/AY=; b=4alexkTsyC3uZni2zW955kLIH7/cxJEXaY5kAd+KYDy1zPDGn17cBxujc6Cmd2xX8mlsi 9BjuR3Z6KRJsIohCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=xyC4YWMYQXg5okBOT+I25AzHoSz3IJwuaB8/iA0p/AY=; b=F9lBNAo30q0ql99dAFCdVKf27nzoKGBO6/YkgLB360VcrFXl9oOJn/NiXASR/OpX6yu5q AYM1MAxJgm+qsk5q1FYDKUbOplpLlwf8c6J1NbjZN/N6L+5dTdI3pxu2ZjmTK+6IVX+nuH5 LMrFAIzKe0xIJwC3UqPVQlQBLow7LKLo6lS4ULhcubgcjVWe7ZzrvfbqAW6+/ukLUWOGY/b Enp2whRD+IPVFO3yWGpb5vwaAVi84bDRLUYwoIVDyFE0wwYC//bioKbWK0vm8u+PPo6Z4uA SITn326/KMoS5eM2sILhnDcUVF5qUpYVtm71XlOCgBfBLHuQ55YAnrE5CuOA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 8ACF116033B; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 7D23219F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 19/29] mm: implement speculative handling in do_numa_page() Date: Fri, 30 Apr 2021 12:52:20 -0700 Message-Id: <20210430195232.30491-20-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: 45y6akr8jnm1mpqdyhmre4x78cxmq443 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1762160006C0 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=4alexkTs; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=F9lBNAo3; spf=pass (imf25.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812351-971314 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: change do_numa_page() to use pte_spinlock() when locking the page table, so that the mmap sequence counter will be validated in the speculative case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eceb1b6e904c..a2230269e034 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4198,8 +4198,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * validation through pte_unmap_same(). It's of NUMA type but * the pfn may be screwed if the read is non atomic. */ - vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; From patchwork Fri Apr 30 19:52:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3717AC433ED for ; Fri, 30 Apr 2021 19:53:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D17526146D for ; Fri, 30 Apr 2021 19:53:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D17526146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9AAAF6B0095; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D03236B008A; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 538476B009F; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id 28EDD6B0092 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin35.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DA923180AF920 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.35.C865AA5 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf30.hostedemail.com (Postfix) with ESMTP id F37CAE000123 for ; Fri, 30 Apr 2021 19:52:15 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1L50YJ40ZUGhKi7U56okv5p4ypQrfJ9Qy1e+oZQLfEs=; b=Brwwj7gJyZRfaNkZXJQVGbs8sZpxuIJtgdHMAi1QT3YHt8Bee/ZoafHWKKtqdak76+hm7 Zkxt0jIKt4dvdwkAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1L50YJ40ZUGhKi7U56okv5p4ypQrfJ9Qy1e+oZQLfEs=; b=ouNOvDWkDxT2pSbRkmI32QMVAK9o/l9HE++tUyWfAi5lB/O0izySgxB6nzXJ+UKfpnHmx +n7kKQ2SWEHvsaTGG90ts4eyff88aM3tiLqQOmAEnz9AXk5kHBJ2L+aANfN7P/HfDqQCXL5 4hmfCw4JsNphr/TmFIYIJrnBWT5/2zMSnD84vK5AqH8mZ9S/h6EEtmt4R8x9RXkcdi2NJXY QCmVi9n1n0vvwDg6lR7ogUk7bR4uz7bSPyUCsNZz1m4dCBHBkNqRdGgmWdzK4QSNzjI/qlz XCvGtDkIE1sX7/T4etK/S4N4HWmfKfIJIocGbiT19PAprQylJxc6pSF04W7Q== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 90A4D16033C; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 8343919F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 20/29] mm: enable speculative fault handling in do_numa_page() Date: Fri, 30 Apr 2021 12:52:21 -0700 Message-Id: <20210430195232.30491-21-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=Brwwj7gJ; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=ouNOvDWk; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf30.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: 516x5j6365yqx41umu9uo7nchkc3uzj6 X-Rspamd-Queue-Id: F37CAE000123 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812335-19745 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change handle_pte_fault() to allow speculative fault execution to proceed through do_numa_page(). do_swap_page() does not implement speculative execution yet, so it needs to abort with VM_FAULT_RETRY in that case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a2230269e034..286776b7795b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3363,6 +3363,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) goto out; @@ -4359,17 +4364,17 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) return do_fault(vmf); } - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); - return VM_FAULT_RETRY; - } - if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); entry = vmf->orig_pte; From patchwork Fri Apr 30 19:52:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A015FC433ED for ; Fri, 30 Apr 2021 19:53:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 47D746145A for ; Fri, 30 Apr 2021 19:53:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 47D746145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C94A26B0078; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A14B6B0085; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53A856B0080; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 333D26B0083 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E886F82499B9 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.38.05E951B Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf16.hostedemail.com (Postfix) with ESMTP id F0CA080192F0 for ; Fri, 30 Apr 2021 19:52:31 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=3PnBKJvO5QY1tr24tZ6y3ew9eVWtL5kDFtGxo4ArR+E=; b=lXBA0+F6lchhVkLLvb5J6eYEHqc6P+VKSYX9K1A7aPH85rzdTMVLqatH87PnLBZegRP2q TKXtRyQ757vAs9hDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=3PnBKJvO5QY1tr24tZ6y3ew9eVWtL5kDFtGxo4ArR+E=; b=dzV0TyOIMhHowJCNEtGaYZ4ibmh4ZJg9dIbE7IsVg2FpdmcPq1bssfGyF7kQVPJbkmNvx Kwmh7ga4jyZjaIrleHYGhE9SMzXra/cTW9R3dwAPEXGjmhpHT/eXRkPg1Nnry2lXDsNUtpX Q1tj9sbYOJfdDX2vfKtOJWW76kguZwY67cwBe0lotTABt7pky5vRkvnSW9lK5xV/zaLk5Gj CnerETNmvyR/VGKkoRoM0uLBnldZvqYRqXnVPjbR7aVBlShhTqQB1QZHHYYT0vkuauozP3Z VQevU/v/tMkbT3rUnZW6GW59pNgyZ/TP7CaAWcoj/DU2GAB5Uq4fLz+3dcNQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 9870A16033F; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 88DF119F524; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 21/29] mm: implement speculative handling in wp_page_copy() Date: Fri, 30 Apr 2021 12:52:22 -0700 Message-Id: <20210430195232.30491-22-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F0CA080192F0 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=lXBA0+F6; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=dzV0TyOI; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf16.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: huexyae7gj1nrkpno7i1x3mjdkof8en4 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812351-821622 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change wp_page_copy() to handle the speculative case. This involves aborting speculative faults if they have to allocate an anon_vma, and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Also change call sites to clear vmf->pte after unmapping the page table, in order to satisfy pte_map_lock()'s preconditions. Signed-off-by: Michel Lespinasse --- mm/memory.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 286776b7795b..cb66585f5145 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2928,20 +2928,27 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; + vm_fault_t ret = VM_FAULT_OOM; - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret = VM_FAULT_RETRY; + goto out; + } + if (__anon_vma_prepare(vma)) + goto out; + } if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!new_page) - goto oom; + goto out; } else { new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!new_page) - goto oom; + goto out; if (!cow_user_page(new_page, old_page, vmf)) { /* @@ -2958,7 +2965,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } if (mem_cgroup_charge(new_page, mm, GFP_KERNEL)) - goto oom_free_new; + goto out_free_new; cgroup_throttle_swaprate(new_page, GFP_KERNEL); __SetPageUptodate(new_page); @@ -2971,7 +2978,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* * Re-check the pte - we dropped the lock */ - vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + /* put_page() will uncharge the page */ + goto out_free_new; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { if (old_page) { if (!PageAnon(old_page)) { @@ -3059,12 +3070,12 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) put_page(old_page); } return page_copied ? VM_FAULT_WRITE : 0; -oom_free_new: +out_free_new: put_page(new_page); -oom: +out: if (old_page) put_page(old_page); - return VM_FAULT_OOM; + return ret; } /** @@ -3207,6 +3218,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return wp_pfn_shared(vmf); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); } @@ -3245,6 +3257,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); } From patchwork Fri Apr 30 19:52:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05932C433ED for ; Fri, 30 Apr 2021 19:53:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A13FC6146D for ; Fri, 30 Apr 2021 19:53:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A13FC6146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E4F346B0085; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49B1E6B009C; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F61F6B0096; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 531D06B009A for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 187BF52B0 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.36.F1F9A28 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf21.hostedemail.com (Postfix) with ESMTP id 10205E000125 for ; Fri, 30 Apr 2021 19:52:33 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6izgSjuxJC/sQuVYlPY55kmOkeAgaUYZ6vcRO71jz20=; b=6MkGLKvM/4VtjyiymcFk2dABknx20mqKn0lGnp7hMFyHV9yFVB5srF0QXxkBHbz+uWWsP ZNa4wxs6B3JQuVwDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6izgSjuxJC/sQuVYlPY55kmOkeAgaUYZ6vcRO71jz20=; b=A01bdPoVLNR+cYeuMSIDL5MvLvHSWzUFdNnBnPCOojMiIGawDh+laCnZlI26QdeJgQ2d7 HtJ8ez8EsInqYn47P5AqrtvW44W2A4iaJ+DXOm4cyjpK1gHE1ei9ThzshowQotc/PYU71GR NWtJNkcpJEYKx0QA9/204+L3r8e07/58DiEs56YhlJrr6RGoN6/EGStz6dOvMYGS1SaTjGj JkP/bRYRI6D/AyQCI+ZvWaOz/Snj3vfs9GJgyUEh3uA/9KfMI0l7ulaFCwoqbHCnxX4TJtK 4LOh2CSRTHTUxH1RfGFRbGOVN0+ndlqMVNKzGdVxrUVbX4nlsb6k6JOl3l0g== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 9E4B616035C; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 8ED4019F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 22/29] mm: implement and enable speculative fault handling in handle_pte_fault() Date: Fri, 30 Apr 2021 12:52:23 -0700 Message-Id: <20210430195232.30491-23-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 10205E000125 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=6MkGLKvM; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=A01bdPoV; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf21.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: ct8gomcggqg3tj3yp1jczyw19izhfqbn Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812353-831662 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In handle_pte_fault(), allow speculative execution to proceed. Use pte_spinlock() to validate the mmap sequence count when locking the page table. If speculative execution proceeds through do_wp_page(), ensure that we end up in the wp_page_reuse() or wp_page_copy() paths, rather than wp_pfn_shared() or wp_page_shared() (both unreachable as we only handle anon vmas so far) or handle_userfault() (needs an explicit abort to handle non-speculatively). Signed-off-by: Michel Lespinasse --- mm/memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index cb66585f5145..c3cd29d3acc6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3120,6 +3120,7 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) { vm_fault_t ret; @@ -3140,6 +3141,8 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = VM_FAULT_WRITE; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + get_page(vmf->page); if (vma->vm_ops && vma->vm_ops->page_mkwrite) { @@ -3193,6 +3196,8 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; return handle_userfault(vmf, VM_UFFD_WP); } @@ -4383,13 +4388,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); + if (!pte_spinlock(vmf)) return VM_FAULT_RETRY; - } - - vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) { update_mmu_tlb(vmf->vma, vmf->address, vmf->pte); From patchwork Fri Apr 30 19:52:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ADF1C433B4 for ; Fri, 30 Apr 2021 19:53:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BD8BF615FF for ; Fri, 30 Apr 2021 19:53:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD8BF615FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5225A6B0093; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9FFC6B009E; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D997C6B0074; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0108.hostedemail.com [216.40.44.108]) by kanga.kvack.org (Postfix) with ESMTP id 4A87D6B0098 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 000FE180A7770 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080796.03.D691D45 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf04.hostedemail.com (Postfix) with ESMTP id 63A9C12E for ; Fri, 30 Apr 2021 19:52:33 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=0Db61unRWfcOHO6d229/jewgQ0C/qszooUu2NMYgmHo=; b=wECn2pFk6yfclXbzl4xeFMOTpkfBKYUUEUA9L2CmfdZCgVHwuNdtOLagIcDKQWaDDliB3 adzhucR//qTZdnACA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=0Db61unRWfcOHO6d229/jewgQ0C/qszooUu2NMYgmHo=; b=p3VnJ3pqGKo0IRSSphAfvBns7M9kpQHPho8pBepwe2qRrsusie9ZtBcrtOJo3aZtUirya zWaUa1TwgHgqn3lvgFfpbynhYbmhBWuCRKftrOzNT34vRXgBKDvaU0f0WOVbEmu1TqB1sUT KhEd1OiSOVM5eTGcmumsE1RgAp5Afwty8iskFi0dCnmBD6oDlUDd1Kipa0HNgaNvtNkXQlo PW2EY2Pb2tVn7QYrOwamlD6R2BmF3zK9bbjpNIMoAaIRZKDDlqAjeBCUAhqUKVwhKfB6rv8 d7mHt6xB/Sc5RRvTztw8bY2eU5ysjleebk93JIvJV1pjCv3cXpO4DE1Dgc4w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id A332216035D; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 9594C19F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 23/29] mm: implement speculative handling in do_swap_page() Date: Fri, 30 Apr 2021 12:52:24 -0700 Message-Id: <20210430195232.30491-24-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=wECn2pFk; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=p3VnJ3pq; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf04.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: ectycm63fg8bcfxwsmqb1kbc9pyodn4k X-Rspamd-Queue-Id: 63A9C12E Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812353-768884 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the pte is larger than long, use pte_spinlock() to lock the page table when verifying the pte - pte_spinlock() is necessary to ensure the page table is still valid when we are locking it. Abort speculative faults if the pte is not a swap entry, or if the desired page is not found in swap cache, to keep things as simple as possible. Only use trylock when locking the swapped page - again to keep things simple, and also the usual lock_page_or_retry would otherwise try to release the mmap lock which is not held in the speculative case. Use pte_map_lock() to ensure proper synchronization when finally committing the faulted page to the mm address space. Signed-off-by: Michel Lespinasse --- mm/memory.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c3cd29d3acc6..a3708b4a616c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2654,30 +2654,6 @@ bool __pte_map_lock(struct vm_fault *vmf) #endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ -/* - * handle_pte_fault chooses page fault handler according to an entry which was - * read non-atomically. Before making any commitment, on those architectures - * or configurations (e.g. i386 with PAE) which might give a mix of unmatched - * parts, do_swap_page must check under lock before unmapping the pte and - * proceeding (but do_wp_page is only called after already making such a check; - * and do_anonymous_page can safely check later on). - */ -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd, - pte_t *page_table, pte_t orig_pte) -{ - int same = 1; -#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) - if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(mm, pmd); - spin_lock(ptl); - same = pte_same(*page_table, orig_pte); - spin_unlock(ptl); - } -#endif - pte_unmap(page_table); - return same; -} - static inline bool cow_user_page(struct page *dst, struct page *src, struct vm_fault *vmf) { @@ -3386,12 +3362,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return VM_FAULT_RETRY; } - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) - goto out; +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) + if (sizeof(pte_t) > sizeof(unsigned long)) { + /* + * vmf->orig_pte was read non-atomically. Before making + * any commitment, on those architectures or configurations + * (e.g. i386 with PAE) which might give a mix of + * unmatched parts, we must check under lock before + * unmapping the pte and proceeding. + * + * (but do_wp_page is only called after already making + * such a check; and do_anonymous_page can safely + * check later on). + */ + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; + if (!pte_same(*vmf->pte, vmf->orig_pte)) + goto unlock; + spin_unlock(vmf->ptl); + } +#endif + pte_unmap(vmf->pte); + vmf->pte = NULL; entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { - if (is_migration_entry(entry)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret = VM_FAULT_RETRY; + } else if (is_migration_entry(entry)) { migration_entry_wait(vma->vm_mm, vmf->pmd, vmf->address); } else if (is_device_private_entry(entry)) { @@ -3412,8 +3410,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = page; if (!page) { - struct swap_info_struct *si = swp_swap_info(entry); + struct swap_info_struct *si; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + return VM_FAULT_RETRY; + } + + si = swp_swap_info(entry); if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ @@ -3476,7 +3480,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_release; } - locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + locked = trylock_page(page); + else + locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); delayacct_clear_flag(DELAYACCT_PF_SWAPIN); if (!locked) { @@ -3504,10 +3511,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) cgroup_throttle_swaprate(page, GFP_KERNEL); /* - * Back out if somebody else already faulted in this pte. + * Back out if the VMA has changed in our back during a speculative + * page fault or if somebody else already faulted in this pte. */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto out_page; + } if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; From patchwork Fri Apr 30 19:52:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F390C433B4 for ; Fri, 30 Apr 2021 19:53:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 020526145A for ; Fri, 30 Apr 2021 19:53:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 020526145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9F08E6B008A; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 030246B009D; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CCEE6B009F; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id 38FEE6B0096 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0191F5003 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.34.BE3C390 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf26.hostedemail.com (Postfix) with ESMTP id DAAB740002C1 for ; Fri, 30 Apr 2021 19:52:27 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=V+gT9krnriafAz+sMsWczyeeM3oMiD/qdk5LAIhe9Yg=; b=6H5+DJWUUSgWfXs0KVFOLhcS3YlB+lkUS1LWjaqexBE/zZ535kKaA1aq7BdXF2oe1B1z/ Roh6vuKhxFHU5ZECA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=V+gT9krnriafAz+sMsWczyeeM3oMiD/qdk5LAIhe9Yg=; b=HZvNX4/POpcRD77pCbupyXrI0ZmJA5Ak9aJV9RKOn3IjsghQfdP4EY8R9oWq8ApOp3Q5h bnXXSYPi7IGlYLGHvGFOBsIlqvIE02S7g1eRgIsTJ4PR+60JlTtpiq1Rs8WECoEq8WWYhGD MVFB1wl4X4qDgBNt4ez4i5eb+sibOS6zzu64RdvYKa/nMa4uJXf7MEsXQf2L2b2KV+pVXlV rUpSob+T26o99noRz5hgTLUi//lhC1ySxclol2A4whI6ZNXh9B9AuafHlB27hv93+jOKr54 FI6DV6eaeyyN03FicM8YcMjSmRe9Wb8uKohTjSYTfEdYxw0urz/cvOE9h7MA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id A9075160360; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 9B55D19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 24/29] mm: enable speculative fault handling through do_swap_page() Date: Fri, 30 Apr 2021 12:52:25 -0700 Message-Id: <20210430195232.30491-25-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DAAB740002C1 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=6H5+DJWU; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b="HZvNX4/P"; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf26.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: infjkswrbnp1sskbkc5qwb77tmpro1m4 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812347-643622 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change do_swap_page() to allow speculative fault execution to proceed. Signed-off-by: Michel Lespinasse --- mm/memory.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a3708b4a616c..cf1a1c0196f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3357,11 +3357,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); - return VM_FAULT_RETRY; - } - #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { /* From patchwork Fri Apr 30 19:52:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E12E7C433B4 for ; Fri, 30 Apr 2021 19:53:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8F4466145A for ; Fri, 30 Apr 2021 19:53:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F4466145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D9F0B6B007D; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5ACC16B0093; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB2A46B009E; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 2D93A6B0093 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DCFF74FFA for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.09.21F44AF Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf18.hostedemail.com (Postfix) with ESMTP id 4B91A200027C for ; Fri, 30 Apr 2021 19:52:39 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8JAcgap2Y6VBqeBYCFSL5shic9T9aRbFwbzzfMi53Zk=; b=cHUjMGrj2X1IDinPMY0+dKYe5adB499TJ7H+bfr7ETPsZilLoPfMVym38TQeMy1ziuWpJ F9gf1FUC9B6sDyRAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8JAcgap2Y6VBqeBYCFSL5shic9T9aRbFwbzzfMi53Zk=; b=UOOGCy41woDuBjIDj9AxgKSCdxZe8jlgAyT+CGVtOzwTmo1GlRVsIqe2Zdz1vQCn5TA1M /JofEHwPBChM0jwFBGv0l1piRk7vawrDfwuf8LY+3xed2fTKScXpJPoombyg5uNGM7eV5ys xx3jPzEFN1dABh3CoMDkDS3TWTTz69MLk5zsxFvPx3xklT+UzFtYV9KZhNn5QP/dj3HJLuE oWD2jRpym5v+B+ektmeW8VK4vXJsA6jn/3xVM28w/qteRSICYJ9RWdZkWdfSooPqFR0JBEo CbTC/gyL58DqgVVo/N8uTUPO51ELndImk4v7llW3NZgVtzXawHqurgd+x5uA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id B03AB160361; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id A135219F524; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 25/29] mm: disable speculative faults for single threaded user space Date: Fri, 30 Apr 2021 12:52:26 -0700 Message-Id: <20210430195232.30491-26-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=cHUjMGrj; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=UOOGCy41; spf=pass (imf18.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4B91A200027C X-Stat-Signature: dnmse59a9zfdxstpc37fgsq3164n6ygt Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812359-394053 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Performance tuning: single threaded userspace does not benefit from speculative page faults, so we turn them off to avoid any related (small) extra overheads. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 463061186827..b5c21585e35f 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1321,6 +1321,14 @@ void do_user_addr_fault(struct pt_regs *regs, #endif #ifdef CONFIG_SPECULATIVE_PAGE_FAULT + + /* + * No need to try speculative faults for kernel or + * single threaded user space. + */ + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + count_vm_event(SPF_ATTEMPT); seq = mmap_seq_read_start(mm); if (seq & 1) @@ -1353,7 +1361,9 @@ void do_user_addr_fault(struct pt_regs *regs, spf_abort: count_vm_event(SPF_ABORT); -#endif +no_spf: + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ /* * Kernel-mode access to the user address space should only occur From patchwork Fri Apr 30 19:52:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BECDC433ED for ; Fri, 30 Apr 2021 19:53:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4DB276145A for ; Fri, 30 Apr 2021 19:53:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DB276145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 281F06B0082; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 893BA6B0092; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0081C6B0081; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 24A746B008C for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DA8F0181AF5D0 for ; Fri, 30 Apr 2021 19:52:37 +0000 (UTC) X-FDA: 78090080754.08.5230EBE Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf08.hostedemail.com (Postfix) with ESMTP id 5B3DD80192E9 for ; Fri, 30 Apr 2021 19:52:14 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=GfsT7km1e4u9essBB1Wu9kC04UOBGN5ahWAVC6IN8es=; b=QJjr+jhIyojTjZZUIgmmlJdhwxp1SplGNE0HRwgP8Nt0MWqhEvdhVMBV6dKGh9nbTPbnl EwzltSshob+oxI6BQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=GfsT7km1e4u9essBB1Wu9kC04UOBGN5ahWAVC6IN8es=; b=wKDBDf36V1KWAErgdFzL/otW+avow+vT/EaKwdglJLtyRP95dA3S7xlKS+lK3mwNFj0lj MHPXq5SZRut4O8MZby6+wWCIuBmJtC8/Zex9JuElWKPSuaeBrbklTpODIHHKojHY3Qp7Sd2 aw4XbgGFObcnn57pMDO9JTHRFsG3cDTzhVCXEamUpBn5vRExQ/8E4+1vI8x6HIx2ZEDzkwR WGynlLA+SouY0L+mVA+lVh/dbllPZsBTwFI86XwEbVz/jaQWEcVzBLsbhScf1TY/3XqI8cF maV2tfOCc2KsvtVGDc/TEZFftizW8kBY1t6UJYzRq1dQRS+JV+xLHrEtv2EQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id B6CC8160365; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id A802819F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 26/29] mm: disable rcu safe vma freeing for single threaded user space Date: Fri, 30 Apr 2021 12:52:27 -0700 Message-Id: <20210430195232.30491-27-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=QJjr+jhI; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=wKDBDf36; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf08.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: kmnisxhurqoyuc13m1xp3ckx3ak6z5h1 X-Rspamd-Queue-Id: 5B3DD80192E9 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812334-323111 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Performance tuning: as single threaded userspace does not use speculative page faults, it does not require rcu safe vma freeing. Turn this off to avoid the related (small) extra overheads. For multi threaded userspace, we often see a performance benefit from the rcu safe vma freeing - even in tests that do not have any frequent concurrent page faults ! This is because rcu safe vma freeing prevents recently released vmas from being immediately reused in a new thread. Signed-off-by: Michel Lespinasse --- kernel/fork.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 7c22bf2b1f9d..18659d802d24 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -381,10 +381,12 @@ static void __vm_area_free(struct rcu_head *head) void vm_area_free(struct vm_area_struct *vma) { #ifdef CONFIG_SPECULATIVE_PAGE_FAULT - call_rcu(&vma->vm_rcu, __vm_area_free); -#else - kmem_cache_free(vm_area_cachep, vma); + if (atomic_read(&vma->vm_mm->mm_users) > 1) { + call_rcu(&vma->vm_rcu, __vm_area_free); + return; + } #endif + kmem_cache_free(vm_area_cachep, vma); } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Fri Apr 30 19:52:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40264C433ED for ; Fri, 30 Apr 2021 20:02:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CA4C3613F8 for ; Fri, 30 Apr 2021 20:02:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA4C3613F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F33D6B006E; Fri, 30 Apr 2021 16:02:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A39E6B0070; Fri, 30 Apr 2021 16:02:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 243498D0002; Fri, 30 Apr 2021 16:02:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 064356B006E for ; Fri, 30 Apr 2021 16:02:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B389C82499B9 for ; Fri, 30 Apr 2021 20:02:06 +0000 (UTC) X-FDA: 78090104652.29.3CFB8D5 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf30.hostedemail.com (Postfix) with ESMTP id 382A2E005F2D for ; Fri, 30 Apr 2021 20:00:44 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=in6C9xm2aQi+RZ2gD0g9zMXOcx4SOVEAt3QJRKwzXzw=; b=y1VcywCk3wkhDGxVwvBnn9Gel88/ciqw6Jv2ouuGUtJZiG0dNktLTHO6MYFT0EDEIfh0f jqVrg+kyNtI8l5FDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=in6C9xm2aQi+RZ2gD0g9zMXOcx4SOVEAt3QJRKwzXzw=; b=gPEA2N1bdoR3t7KK+ez4hQ2VWS85UMrFRpj5G+wbozJK/cWXoO6q7L+GhlHKlQOZqwbq3 wk120gGfQIwQ4QEjVOFcQE9vs9Ak7bjPzpGqiBFNZf3oUns3CO/+gPJv0ew25LmtN07OLeZ 0URGLJlIjfHTn5aR4dCCxWDfxpBYvBH1F+FPV8oTwplwSvI49SV/j6UUfbS8hkVFg5sDA7E wmSt/TumK+CMwLWyX7IyT/cNi1K5lKre7ngOJ4nhbSYSufKVTLayvoTQCRcarGMe3tnXJwK /leE2vHFgpGBbIHtnpuu8OTb5Nr7fpRV+nIoUpd7Mi/nzbdbwraRJDpLjLOg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id BF80E160366; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id AFBBA19F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 27/29] mm: anon spf statistics Date: Fri, 30 Apr 2021 12:52:28 -0700 Message-Id: <20210430195232.30491-28-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=y1VcywCk; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=gPEA2N1b; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf30.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: 8zqieye3ywi3qrq5bhtcio68yqazde97 X-Rspamd-Queue-Id: 382A2E005F2D X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812844-327782 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_SPECULATIVE_PAGE_FAULT_STATS config option, and dump extra statistics about executed spf cases and abort reasons when the option is set. Signed-off-by: Michel Lespinasse Reported-by: kernel test robot --- arch/x86/mm/fault.c | 18 ++++++++--- include/linux/mmap_lock.h | 19 +++++++++-- include/linux/vm_event_item.h | 23 ++++++++++++++ include/linux/vmstat.h | 6 ++++ mm/Kconfig.debug | 7 ++++ mm/memory.c | 60 ++++++++++++++++++++++++++++------- mm/vmstat.c | 23 ++++++++++++++ 7 files changed, 139 insertions(+), 17 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index b5c21585e35f..7d8c99023a82 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1331,21 +1331,31 @@ void do_user_addr_fault(struct pt_regs *regs, count_vm_event(SPF_ATTEMPT); seq = mmap_seq_read_start(mm); - if (seq & 1) + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); goto spf_abort; + } rcu_read_lock(); vma = find_vma(mm, address); - if (!vma || vma->vm_start > address || !vma_is_anonymous(vma)) { + if (!vma || vma->vm_start > address) { rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_is_anonymous(vma)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); goto spf_abort; } pvma = *vma; rcu_read_unlock(); - if (!mmap_seq_read_check(mm, seq)) + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) goto spf_abort; vma = &pvma; - if (unlikely(access_error(error_code, vma))) + if (unlikely(access_error(error_code, vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); goto spf_abort; + } fault = do_handle_mm_fault(vma, address, flags | FAULT_FLAG_SPECULATIVE, seq, regs); diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8f4eca2d0f43..98f24a9910a9 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -7,6 +7,7 @@ #include #include #include +#include #ifdef CONFIG_SPECULATIVE_PAGE_FAULT #define MMAP_LOCK_SEQ_INITIALIZER(name) \ @@ -104,12 +105,26 @@ static inline unsigned long mmap_seq_read_start(struct mm_struct *mm) return seq; } -static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq) +static inline bool __mmap_seq_read_check(struct mm_struct *mm, + unsigned long seq) { smp_rmb(); return seq == READ_ONCE(mm->mmap_seq); } -#endif + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS +static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq, + enum vm_event_item fail_event) +{ + if (__mmap_seq_read_check(mm, seq)) + return true; + count_vm_event(fail_event); + return false; +} +#else +#define mmap_seq_read_check(mm, seq, fail) __mmap_seq_read_check(mm, seq) +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT_STATS */ +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ static inline void mmap_write_lock(struct mm_struct *mm) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index cc4f8d14e43f..42e57db1623b 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -124,6 +124,29 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SPECULATIVE_PAGE_FAULT SPF_ATTEMPT, SPF_ABORT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS + SPF_ABORT_ODD, + SPF_ABORT_UNMAPPED, + SPF_ABORT_NO_SPECULATE, + SPF_ABORT_VMA_COPY, + SPF_ABORT_ACCESS_ERROR, + SPF_ABORT_PUD, + SPF_ABORT_PMD, + SPF_ABORT_ANON_VMA, + SPF_ABORT_PTE_MAP_LOCK_SEQ1, + SPF_ABORT_PTE_MAP_LOCK_PMD, + SPF_ABORT_PTE_MAP_LOCK_PTL, + SPF_ABORT_PTE_MAP_LOCK_SEQ2, + SPF_ABORT_USERFAULTFD, + SPF_ABORT_FAULT, + SPF_ABORT_NON_SWAP_ENTRY, + SPF_ABORT_SWAP_NOPAGE, + SPF_ATTEMPT_ANON, + SPF_ATTEMPT_SWAP, + SPF_ATTEMPT_NUMA, + SPF_ATTEMPT_PTE, + SPF_ATTEMPT_WP, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 506d625163a1..34e05604a93f 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -109,6 +109,12 @@ static inline void vm_events_fold_cpu(int cpu) #endif /* CONFIG_VM_EVENT_COUNTERS */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS +#define count_vm_spf_event(x) count_vm_event(x) +#else +#define count_vm_spf_event(x) do {} while (0) +#endif + #ifdef CONFIG_NUMA_BALANCING #define count_vm_numa_event(x) count_vm_event(x) #define count_vm_numa_events(x, y) count_vm_events(x, y) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 1e73717802f8..6be8ca7950ee 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -150,3 +150,10 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + +config SPECULATIVE_PAGE_FAULT_STATS + bool "Additional statistics for speculative page faults" + depends on SPECULATIVE_PAGE_FAULT + help + Additional statistics for speculative page faults. + If in doubt, say N. diff --git a/mm/memory.c b/mm/memory.c index cf1a1c0196f0..838482b7ffc5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2603,7 +2603,8 @@ bool __pte_map_lock(struct vm_fault *vmf) } speculative_page_walk_begin(); - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, + SPF_ABORT_PTE_MAP_LOCK_SEQ1)) goto fail; /* * The mmap sequence count check guarantees that the page @@ -2616,8 +2617,10 @@ bool __pte_map_lock(struct vm_fault *vmf) * is not a huge collapse operation in progress in our back. */ pmdval = READ_ONCE(*vmf->pmd); - if (!pmd_same(pmdval, vmf->orig_pmd)) + if (!pmd_same(pmdval, vmf->orig_pmd)) { + count_vm_spf_event(SPF_ABORT_PTE_MAP_LOCK_PMD); goto fail; + } #endif ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); if (!pte) @@ -2634,9 +2637,12 @@ bool __pte_map_lock(struct vm_fault *vmf) * We also don't want to retry until spin_trylock() succeeds, * because of the starvation potential against a stream of lockers. */ - if (unlikely(!spin_trylock(ptl))) + if (unlikely(!spin_trylock(ptl))) { + count_vm_spf_event(SPF_ABORT_PTE_MAP_LOCK_PTL); goto fail; - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + } + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, + SPF_ABORT_PTE_MAP_LOCK_SEQ2)) goto unlock_fail; speculative_page_walk_end(); vmf->pte = pte; @@ -2908,6 +2914,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) if (unlikely(!vma->anon_vma)) { if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_ANON_VMA); ret = VM_FAULT_RETRY; goto out; } @@ -3170,10 +3177,15 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_WP); + if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_USERFAULTFD); return VM_FAULT_RETRY; + } return handle_userfault(vmf, VM_UFFD_WP); } @@ -3357,6 +3369,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_SWAP); + #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { /* @@ -3383,6 +3398,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_NON_SWAP_ENTRY); ret = VM_FAULT_RETRY; } else if (is_migration_entry(entry)) { migration_entry_wait(vma->vm_mm, vmf->pmd, @@ -3409,6 +3425,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (vmf->flags & FAULT_FLAG_SPECULATIVE) { delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + count_vm_spf_event(SPF_ABORT_SWAP_NOPAGE); return VM_FAULT_RETRY; } @@ -3615,6 +3632,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vm_fault_t ret = 0; pte_t entry; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_ANON); + /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) return VM_FAULT_SIGBUS; @@ -3644,8 +3664,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) } else { /* Allocate our own private page. */ if (unlikely(!vma->anon_vma)) { - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_ANON_VMA); return VM_FAULT_RETRY; + } if (__anon_vma_prepare(vma)) goto oom; } @@ -3687,8 +3709,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); if (page) put_page(page); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_USERFAULTFD); return VM_FAULT_RETRY; + } return handle_userfault(vmf, VM_UFFD_MISSING); } @@ -4221,6 +4245,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) bool was_writable = pte_savedwrite(vmf->orig_pte); int flags = 0; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_NUMA); + /* * The "pte" at this point cannot be used safely without * validation through pte_unmap_same(). It's of NUMA type but @@ -4393,6 +4420,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_PTE); + if (!pte_spinlock(vmf)) return VM_FAULT_RETRY; entry = vmf->orig_pte; @@ -4460,20 +4490,26 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, speculative_page_walk_begin(); pgd = pgd_offset(mm, address); pgdval = READ_ONCE(*pgd); - if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) + if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } p4d = p4d_offset(pgd, address); p4dval = READ_ONCE(*p4d); - if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } vmf.pud = pud_offset(p4d, address); pudval = READ_ONCE(*vmf.pud); if (pud_none(pudval) || unlikely(pud_bad(pudval)) || unlikely(pud_trans_huge(pudval)) || - unlikely(pud_devmap(pudval))) + unlikely(pud_devmap(pudval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } vmf.pmd = pmd_offset(vmf.pud, address); vmf.orig_pmd = READ_ONCE(*vmf.pmd); @@ -4491,8 +4527,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (unlikely(pmd_none(vmf.orig_pmd) || is_swap_pmd(vmf.orig_pmd) || pmd_trans_huge(vmf.orig_pmd) || - pmd_devmap(vmf.orig_pmd))) + pmd_devmap(vmf.orig_pmd))) { + count_vm_spf_event(SPF_ABORT_PMD); goto spf_fail; + } /* * The above does not allocate/instantiate page-tables because diff --git a/mm/vmstat.c b/mm/vmstat.c index 9ae1c27a549e..dbaefae62da3 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1369,6 +1369,29 @@ const char * const vmstat_text[] = { "spf_attempt", "spf_abort", #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS + "SPF_ABORT_ODD", + "SPF_ABORT_UNMAPPED", + "SPF_ABORT_NO_SPECULATE", + "SPF_ABORT_VMA_COPY", + "SPF_ABORT_ACCESS_ERROR", + "SPF_ABORT_PUD", + "SPF_ABORT_PMD", + "SPF_ABORT_ANON_VMA", + "SPF_ABORT_PTE_MAP_LOCK_SEQ1", + "SPF_ABORT_PTE_MAP_LOCK_PMD", + "SPF_ABORT_PTE_MAP_LOCK_PTL", + "SPF_ABORT_PTE_MAP_LOCK_SEQ2", + "SPF_ABORT_USERFAULTFD", + "SPF_ABORT_FAULT", + "SPF_ABORT_NON_SWAP_ENTRY", + "SPF_ABORT_SWAP_NOPAGE", + "SPF_ATTEMPT_ANON", + "SPF_ATTEMPT_SWAP", + "SPF_ATTEMPT_NUMA", + "SPF_ATTEMPT_PTE", + "SPF_ATTEMPT_WP", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Fri Apr 30 19:52:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C6A1C433B4 for ; Fri, 30 Apr 2021 19:53:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 08EB861474 for ; Fri, 30 Apr 2021 19:53:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 08EB861474 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 20AB56B009A; Fri, 30 Apr 2021 15:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CF276B0083; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C40A56B009B; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0158.hostedemail.com [216.40.44.158]) by kanga.kvack.org (Postfix) with ESMTP id 6D9AC6B009D for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 22D7E3CF8 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.20.271972B Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf27.hostedemail.com (Postfix) with ESMTP id 9EAAD801A814 for ; Fri, 30 Apr 2021 19:52:12 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=yz4li7EYAF4Bbcpj+eNLJ7tuvrtZJHEzYg77XHgRWJ4=; b=MKdiv+PQUj9ZqU64bNBi4AWTpDktH7ZMN/8ag8ekw2nIACtjMx6TefVYVA+TS1fIW+PIz 6N3e3/lGMrma3NeDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=yz4li7EYAF4Bbcpj+eNLJ7tuvrtZJHEzYg77XHgRWJ4=; b=G+BJB9MUJJ2/tltjsww/hWrhr9xEhno6hVjbw7OujVBEqr4i7u3YSYzYmxyXr7vGud1Cv sUzd7k2sHIyAATrkfMtxynWWAvZpNLDteDaJswKJUviJlczPJDnMOe4uVHWIi0V61iZ10rH HsSqiPW9xLPe3FWeh45L5uLWIkUDdR3QIdiI5UtGy5SSrCo4DPzhzxi+ilWQKRRs9HVPCoL PJONhmb75V6wEbXYN8W52RbHJKl08b7L8QLitjDnqSIZrN6Zvlt7Wkk1q64LvhiST6dM/ay OI9SNyE9qf30/mXzg/jhjYrpriQBCOnxTAXitEqZ4jAo8CXs4MJ1hOX3a9lA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id C4688160367; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id B6D3B19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 28/29] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Fri, 30 Apr 2021 12:52:29 -0700 Message-Id: <20210430195232.30491-29-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=MKdiv+PQ; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=G+BJB9MU; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf27.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: w3ej1bqoihjufc355bx13rh8ctyfgygi X-Rspamd-Queue-Id: 9EAAD801A814 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812332-813408 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT so that the speculative fault handling code can be compiled on this architecture. Signed-off-by: Michel Lespinasse --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index dfdc3e0af5e1..a344f2d5f146 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -208,6 +208,7 @@ config ARM64 select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT help ARM 64-bit (AArch64) Linux support. From patchwork Fri Apr 30 19:52:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97DFFC433B4 for ; Fri, 30 Apr 2021 19:53:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4CDA26146D for ; Fri, 30 Apr 2021 19:53:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4CDA26146D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 488756B0083; Fri, 30 Apr 2021 15:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 954E56B009E; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEB076B009A; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 7D6C86B0085 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3917D824934B for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.21.F741FA1 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf14.hostedemail.com (Postfix) with ESMTP id 8CBBDC0007E0 for ; Fri, 30 Apr 2021 19:52:20 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=FTmd8q5bXeYtX6zqFZAxa8+xy7tYwTDwfSzy7iAnfZo=; b=ZulnIIq8a4SfDWkrfX/OUtIvvvF7dZx7gaKOvE4F0it1bDwwukBrcIykrRJ86Xw5/d0Ri g3rZSQUQjZOfd6JBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=FTmd8q5bXeYtX6zqFZAxa8+xy7tYwTDwfSzy7iAnfZo=; b=PmpwOhrUj9PXxXes471bRWBh2GoLrbZ85V8g5GC4KXE0WEvlpmV0L03hppw1kXO7SmZ6a g0LcSRCyYWeWKva4E7tFkRYs/PQGhfa3Pseay7k6iGYcQitfgZVH2HddXgr5oQLiqnvtm5W blzgBE54N/Y1xL4UstFKJRsEKXrxr1oMCJvBVPVpoQa1VlnrpucugfXBZnAqLrnHs7ZUO42 B8DnYDMypFIvB0IskZIHlD1AXOGeJnEmbDUIekM0mtMERqMY4tdE6Vu6kOQef2NkmV3a5IN /12olEZIe2g2Cu36KvTlwMYeEjMQ4tC7WO4m/beAYKqcUWqCFrAKQDDEb+bQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id CE7B3160368; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id BE0D219F521; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 29/29] arm64/mm: attempt speculative mm faults first Date: Fri, 30 Apr 2021 12:52:30 -0700 Message-Id: <20210430195232.30491-30-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=ZulnIIq8; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=PmpwOhrU; spf=pass (imf14.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org; dmarc=pass (policy=none) header.from=lespinasse.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8CBBDC0007E0 X-Stat-Signature: jnpspoxpnq1ufzwp74qww9o7cyuojja3 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812340-687500 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. This follows the lines of the x86 speculative fault handling code, but with some minor arch differences such as the way that the VM_FAULT_BADACCESS case is handled. Signed-off-by: Michel Lespinasse --- arch/arm64/mm/fault.c | 63 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f37d4e3830b7..f2d09f8e4bc2 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -530,6 +531,11 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, unsigned long vm_flags = VM_ACCESS_FLAGS; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct vm_area_struct *vma; + struct vm_area_struct pvma; + unsigned long seq; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -564,6 +570,60 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + + /* + * No need to try speculative faults for kernel or + * single threaded user space. + */ + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); + goto spf_abort; + } + rcu_read_lock(); + vma = find_vma(mm, addr); + if (!vma || vma->vm_start > addr) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_is_anonymous(vma)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) + goto spf_abort; + vma = &pvma; + if (!(vma->vm_flags & vm_flags)) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } + fault = do_handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +no_spf: + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -604,6 +664,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, } } mmap_read_unlock(mm); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +done: +#endif /* * Handle the "normal" (no error) case first. From patchwork Fri Apr 30 19:52:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 107D4C433ED for ; Fri, 30 Apr 2021 19:53:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B7B5E6145A for ; Fri, 30 Apr 2021 19:53:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7B5E6145A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 760C06B009B; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EFD456B008A; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E13FA6B009C; Fri, 30 Apr 2021 15:52:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id 49F296B0080 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0A0E052A8 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.37.104E7E1 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf02.hostedemail.com (Postfix) with ESMTP id C732A40002E6 for ; Fri, 30 Apr 2021 19:52:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=K9qzS95Du968LjRCrHXRx96HCrrSPyD4wWr6Q/DHLHc=; b=d9q9qz/3wSk/IOC/TamTz1qgltxdIp2Qq78zyFlGxp4JYo7jL1bBSwHJUYU+SBDuwlSaV unqZ0vRzJrfWxAvAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=K9qzS95Du968LjRCrHXRx96HCrrSPyD4wWr6Q/DHLHc=; b=PWxVdNNRAVDxeO5TuEYDUiSGfnUyPZFBwdsH2TCZ3EsH8LSl5lPHlNIaFp82R297cFJcH BxQK4ZVpAzx+jxrmylIMlDHyHf2Af7xvGmU+fGnpAtEoidaf4fAh9cuNIxBhJqEZGkCHBV/ wV5YiBO6Nrc4B7hu24/9SyBZ6ytUfKt7zhpzjbTsqLbt1LATodqp5WXU0EaMpC2FGdxPzjo UOUqugLHxsU/C9P6h89JfxiMoj9RfjCSt2epqZsMFTJcK4f87q4kYuVzz3hA/kCfYuqcBFa llmosuq7uq1cROmS6TSDVHI2lF5QwaGOWnQlr1UqkDjgbKVU7LU877x0F8dw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id D33E016036E; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id C52BC19F522; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 30/31] powerpc/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Fri, 30 Apr 2021 12:52:31 -0700 Message-Id: <20210430195232.30491-31-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b="d9q9qz/3"; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=PWxVdNNR; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf02.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspamd-Server: rspam03 X-Stat-Signature: 98rgui1ib696zgioiydgexbd1qeph5fc X-Rspamd-Queue-Id: C732A40002E6 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812326-173271 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT so that the speculative fault handling code can be compiled on this architecture. Signed-off-by: Michel Lespinasse --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 386ae12d8523..e08af3e5424b 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -257,6 +257,7 @@ config PPC select PPC_DAWR if PPC64 select RTC_LIB select SPARSE_IRQ + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT if PPC_BOOK3S_64 select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK select VIRT_TO_BUS if !PPC64 From patchwork Fri Apr 30 19:52:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 790EBC433B4 for ; Fri, 30 Apr 2021 19:54:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E61F61476 for ; Fri, 30 Apr 2021 19:54:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E61F61476 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 760A56B009C; Fri, 30 Apr 2021 15:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B7BC56B0096; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CA226B0078; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0181.hostedemail.com [216.40.44.181]) by kanga.kvack.org (Postfix) with ESMTP id BAA516B0078 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6C8D145B3 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.23.0EDECAE Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf19.hostedemail.com (Postfix) with ESMTP id 59BD190009F6 for ; Fri, 30 Apr 2021 19:52:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=eULoJwQoISRJDyWh6sQIkPbJRau/grHqDfGiWr2ndBc=; b=hNOAqV+Bi3m/G9I1+p/zbTgoypBTOIz9B9w7EuLXDESQcB7YB+wNpudjM89Jlv4pGyn1d eppGrtsnZHr/cNRBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=eULoJwQoISRJDyWh6sQIkPbJRau/grHqDfGiWr2ndBc=; b=LLcNJN4Q0cNreUi2QNRCmvi9zlH+OSkbXB/q3IC4eYa3yeOiFOzGf7On1WAFSRGdYS4Hy jDWbNZpT8l72PTG6DCd9W22CDTlebRhwdA69DEVWWQB8UZGfktRR0w/+VDx3geYu4Qt37My vBe5z2c36M/iQlLy1RWER9bUFrR63ZNDfJnizhjBtrQsR5QMmlaz2b/zziuf959oknLxTF6 XHkWLqzy4v3htaBTYFcJVZqEnFtjBbFHdLnnsvn1gxQj7eIjoYN3F62t5hvwB9Ky2p/2+Gh S6Lvbcn9dW4G2XDxg5zPVBk62nFgrXWu/01L/iNX///ZO6Nc3ebsipAsTkeQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id DBD20160372; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id CCB6D19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 31/31] powerpc/mm: attempt speculative mm faults first Date: Fri, 30 Apr 2021 12:52:32 -0700 Message-Id: <20210430195232.30491-32-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=hNOAqV+B; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=LLcNJN4Q; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf19.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: pg7ue6qece5es3qxgyxnid9ctfnrgg85 X-Rspamd-Queue-Id: 59BD190009F6 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812327-296063 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. This follows the lines of the x86 speculative fault handling code, but with some minor arch differences such as the way that the access_pkey_error case is handled Signed-off-by: Michel Lespinasse --- arch/powerpc/mm/fault.c | 65 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index bb368257b55c..d7c820751a58 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -398,6 +398,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, int is_write = page_fault_is_write(error_code); vm_fault_t fault, major = 0; bool kprobe_fault = kprobe_page_fault(regs, 11); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct vm_area_struct pvma; + unsigned long seq; +#endif if (unlikely(debugger_fault_handler(regs) || kprobe_fault)) return 0; @@ -450,6 +454,64 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + + /* + * No need to try speculative faults for kernel or + * single threaded user space. + */ + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); + goto spf_abort; + } + rcu_read_lock(); + vma = find_vma(mm, address); + if (!vma || vma->vm_start > address) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_is_anonymous(vma)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) + goto spf_abort; + vma = &pvma; +#ifdef CONFIG_PPC_MEM_KEYS + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } +#endif /* CONFIG_PPC_MEM_KEYS */ + if (unlikely(access_error(is_write, is_exec, vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } + fault = do_handle_mm_fault(vma, address, + flags | FAULT_FLAG_SPECULATIVE, seq, regs); + major |= fault & VM_FAULT_MAJOR; + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +no_spf: + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -525,6 +587,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, } mmap_read_unlock(current->mm); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault);