From patchwork Wed Apr 7 01:44:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 659B4C433ED for ; Wed, 7 Apr 2021 01:45:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D07DA610CE for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D07DA610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 800636B0081; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7352F6B0082; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C95B6B0085; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EE4466B007E for ; Tue, 6 Apr 2021 21:45:05 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A926C8249980 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) X-FDA: 78003877770.12.A7E6192 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf12.hostedemail.com (Postfix) with ESMTP id 58A9D12E for ; Wed, 7 Apr 2021 01:45:02 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=NVL7JPD4xKaWrJCrCCQAhxOaIfpgRthzGvlvMAcHMhY=; b=OESJ03wes2NwJiVPB7yoVDujCoaMAblbZeph6lEd4X/+SlFZQRQ4XSY8m+asM02bxuSyh JWXeIWR0c/wMGsPBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=NVL7JPD4xKaWrJCrCCQAhxOaIfpgRthzGvlvMAcHMhY=; b=NNc+BYnodpTFTj+++F+FgTCacE2ASOXUqbP91WbtiNnglS6UaYxtuEGt+wSqU2Blxqvdl vlnWo0J0DnkKTLm9oabmwHdK/K8/u/6GOD3CN9KYd6gcN983vIxvIcUWdB0+umwetM+RNiq Q4y/Yah0dxXG6hwrUQpaG8j0Mdsh9eBGK3GUFwrsFLgtiyxoartjV2cLKdWQibkgRzkzF6f tbbCTYe1r/njg5mU7JFS/OZ7Z38g/uRJzwbh4mIxv/M1GNvyUS86YzzPLbjVGRzzXG+IncQ EN/IYPAxtrN4Wr4ECCqxohOG16G6xNiN8qL8r5T8FqrcOgwBH8BorhL2AFAA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 6B2A61601D1; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 5C25B19F30B; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 01/37] mmap locking API: mmap_lock_is_contended returns a bool Date: Tue, 6 Apr 2021 18:44:26 -0700 Message-Id: <20210407014502.24091-2-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 58A9D12E X-Stat-Signature: geung4wok48ema4euy8yt8wc4yozmqpd Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759902-543785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change mmap_lock_is_contended to return a bool value, rather than an int which the callers are then supposed to interpret as a bool. This is to ensure consistency with other mmap lock API functions (such as the trylock functions). Signed-off-by: Michel Lespinasse --- include/linux/mmap_lock.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 0540f0156f58..4e27f755766b 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -171,9 +171,9 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } -static inline int mmap_lock_is_contended(struct mm_struct *mm) +static inline bool mmap_lock_is_contended(struct mm_struct *mm) { - return rwsem_is_contended(&mm->mmap_lock); + return rwsem_is_contended(&mm->mmap_lock) != 0; } #endif /* _LINUX_MMAP_LOCK_H */ From patchwork Wed Apr 7 01:44:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68570C433ED for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CD8F6610CE for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD8F6610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 319EB6B007E; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F33A6B007D; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 145576B0083; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0214.hostedemail.com [216.40.44.214]) by kanga.kvack.org (Postfix) with ESMTP id DDA756B007D for ; Tue, 6 Apr 2021 21:45:05 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id AA98181CF for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) X-FDA: 78003877770.13.4AA56AE Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf08.hostedemail.com (Postfix) with ESMTP id 5E1AD80192D4 for ; Wed, 7 Apr 2021 01:44:58 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=sRgr3eTHMkYG0a033c5wSjP6+Yvrcvk4eJe6GnX/cXk=; b=5EhncmT4Gv438y6/5NG8ncKXAEzImmCZNTdJ3BxGL42bGoEFICVQKsc5B2hxAx8fr7H4e xJ/4tdl7nubFCxJCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=sRgr3eTHMkYG0a033c5wSjP6+Yvrcvk4eJe6GnX/cXk=; b=l9qp+/x/nFDxeFdkHEoF5i+V2o64nYkMphTKPV+I1CsWjxM3TxYvrT3odZdhMkHUWNk7P mJP519TzYNxVsZRJ5bWfMQ4XA/VwL5/m+55+duBR7nt6WMjCdzRqzAIT9uNSt+hForKTjF1 vYafd0yyNFnhcNrFcuASUOHtJ4PL9q9UdQ2S9QSDDo6ObI65oDsQE7NLqJvFpg/nCTfZkXQ 6RjIKhSCrcNJS43RcFFrQIOYKiT4ISh2jSANhns1iZaXnAGFiXYs4iv3jNfy2jsXh1x6Vaf LBpGuioVpY/32Plv1fYa63ymJhJ9ve3uUAyf8tv7eYHnQoypIIbXGU9nNNGQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 757ED160224; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 6242919F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 02/37] mmap locking API: name the return values Date: Tue, 6 Apr 2021 18:44:27 -0700 Message-Id: <20210407014502.24091-3-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: pqfbttoeeb83jeh7koszo56a4o9xfue3 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5E1AD80192D4 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759898-505562 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the mmap locking API, the *_killable() functions return an error (or 0 on success), and the *_trylock() functions return a boolean (true on success). Rename the return values "int error" and "bool ok", respectively, rather than using "ret" for both cases which I find less readable. Signed-off-by: Michel Lespinasse --- include/linux/mmap_lock.h | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 4e27f755766b..8ff276a7560e 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -81,22 +81,22 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) static inline int mmap_write_lock_killable(struct mm_struct *mm) { - int ret; + int error; __mmap_lock_trace_start_locking(mm, true); - ret = down_write_killable(&mm->mmap_lock); - __mmap_lock_trace_acquire_returned(mm, true, ret == 0); - return ret; + error = down_write_killable(&mm->mmap_lock); + __mmap_lock_trace_acquire_returned(mm, true, !error); + return error; } static inline bool mmap_write_trylock(struct mm_struct *mm) { - bool ret; + bool ok; __mmap_lock_trace_start_locking(mm, true); - ret = down_write_trylock(&mm->mmap_lock) != 0; - __mmap_lock_trace_acquire_returned(mm, true, ret); - return ret; + ok = down_write_trylock(&mm->mmap_lock) != 0; + __mmap_lock_trace_acquire_returned(mm, true, ok); + return ok; } static inline void mmap_write_unlock(struct mm_struct *mm) @@ -120,22 +120,22 @@ static inline void mmap_read_lock(struct mm_struct *mm) static inline int mmap_read_lock_killable(struct mm_struct *mm) { - int ret; + int error; __mmap_lock_trace_start_locking(mm, false); - ret = down_read_killable(&mm->mmap_lock); - __mmap_lock_trace_acquire_returned(mm, false, ret == 0); - return ret; + error = down_read_killable(&mm->mmap_lock); + __mmap_lock_trace_acquire_returned(mm, false, !error); + return error; } static inline bool mmap_read_trylock(struct mm_struct *mm) { - bool ret; + bool ok; __mmap_lock_trace_start_locking(mm, false); - ret = down_read_trylock(&mm->mmap_lock) != 0; - __mmap_lock_trace_acquire_returned(mm, false, ret); - return ret; + ok = down_read_trylock(&mm->mmap_lock) != 0; + __mmap_lock_trace_acquire_returned(mm, false, ok); + return ok; } static inline void mmap_read_unlock(struct mm_struct *mm) From patchwork Wed Apr 7 01:44:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E07AC433ED for ; Wed, 7 Apr 2021 01:45:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D79D661165 for ; Wed, 7 Apr 2021 01:45:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D79D661165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9F4966B0080; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 881FF6B0085; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539546B0080; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0184.hostedemail.com [216.40.44.184]) by kanga.kvack.org (Postfix) with ESMTP id F31E46B0080 for ; Tue, 6 Apr 2021 21:45:05 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A57E16C2F for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) X-FDA: 78003877770.16.47F6C1B Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf17.hostedemail.com (Postfix) with ESMTP id 8C3AB40002D2 for ; Wed, 7 Apr 2021 01:45:03 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Tl7c+fKXT/KEyeD0xB/+YiIdgIzXbLApcUUyx6tYhAc=; b=FhQJE30mUdou3MxgfjANMalYk0yghDWwQJi43mxwrXTvsJtXHEht3xYoY0uKAJ6JB1ifc I0uHh5I2VEUVnMwAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Tl7c+fKXT/KEyeD0xB/+YiIdgIzXbLApcUUyx6tYhAc=; b=alvJkzgZCHNhI/J/Oxso+FnyCCyAhh/3VhhOpJiJCr7s9TEQrVPgikpbTw51VxBgDCCQI w7FkTc0lTMUagVapscii/LJG9Zm08jY9YDNgDXUE0z4lW5kR1W+S9Bm7eahNxbSt7i1DF/K 4An6MUsN2GjPOpl1dUda0RSDWa3cLilqgXOLffoFdN0/sfyYZzX5wCQWbvcMLlU4vZczU++ RshCSwV2fhPQrh/qq33PSljbSoowyZ9Hha9n3Tc6S2Z3iQmF0nZ/D/jZQ3zPPrxmYhm2yiV f5fZLHOHzsrbKR0OXKHH9hRON9E8GJ3/o+TyF0pKb2X/3KjsPh7xbhDfDKOg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 7C44916022D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 68FC519F320; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 03/37] do_anonymous_page: use update_mmu_tlb() Date: Tue, 6 Apr 2021 18:44:28 -0700 Message-Id: <20210407014502.24091-4-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8C3AB40002D2 X-Stat-Signature: kew1o3ae5gjtfyoy3jbo8ejqijqfotga X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759903-640981 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: update_mmu_tlb() can be used instead of update_mmu_cache() when the page fault handler detects that it lost the race to another page fault. Signed-off-by: Michel Lespinasse --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 5efa07fb6cdc..8ee4bd239303 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3567,7 +3567,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (!pte_none(*vmf->pte)) { - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; } From patchwork Wed Apr 7 01:44:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B9FDC433B4 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CB3FC610CE for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB3FC610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5B7716B007D; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DB566B0087; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22D1E6B0082; Tue, 6 Apr 2021 21:45:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 002766B0081 for ; Tue, 6 Apr 2021 21:45:05 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id ADE6F81D1 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) X-FDA: 78003877770.13.6373A7F Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf09.hostedemail.com (Postfix) with ESMTP id 6D4096000106 for ; Wed, 7 Apr 2021 01:45:03 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=qFFkvrhk4zVFpz6vGg/7+He914LTpz3CtnbnmIYjjhw=; b=QiPmsuFxjp6vtGaBEF+syYCj20gQ4wqnl9mlmMdhGAQ74iz4ri7g5S0O/C9jaB5SvkIMD Mk8++zIFSTpWU1eBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=qFFkvrhk4zVFpz6vGg/7+He914LTpz3CtnbnmIYjjhw=; b=bYxYuhgP8J1pVW2XF9UT2txE3O37aOVg3E95O1q4kbVTuMfS4TSwZL9PB9koMcsp7qgu5 6+zPaNIuKQ7cVsabNyU6BRNDQ0JGlWmyMqTmT3ifOxmubQD/9Hx77kVZRc/TMTuyf6xjPZB /8Li4+djjpxq7BIUDXBP/Pdk5/UgmmlZluecW96DBqBFSmQ4xBtQnMYxc80s/jIDWGjSZ5/ VTMX7zDyyXpJtZE85hOnaaoxCxowhNeuO8BiaCPtGeG93XCHsWyCrAfHowl6z4ZepkJBWjK JKNRwXrDPWyaPK3MpS+vZJVN7z2zIIaj1UNrTvTFQTLOY1l/KdOcjRR8apEg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 80273160233; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 700D219F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 04/37] do_anonymous_page: reduce code duplication Date: Tue, 6 Apr 2021 18:44:29 -0700 Message-Id: <20210407014502.24091-5-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6D4096000106 X-Stat-Signature: q1fascexaw74wpiqpapcxorx6wdyajsw Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759903-240814 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In do_anonymous_page(), we have separate cases for the zero page vs allocating new anonymous pages. However, once the pte entry has been computed, the rest of the handling (mapping and locking the page table, checking that we didn't lose a race with another page fault handler, etc) is identical between the two cases. This change reduces the code duplication between the two cases. Signed-off-by: Michel Lespinasse --- mm/memory.c | 85 +++++++++++++++++++++++------------------------------ 1 file changed, 37 insertions(+), 48 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8ee4bd239303..477c98bfdd9d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3495,7 +3495,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) static vm_fault_t do_anonymous_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct page *page; + struct page *page = NULL; vm_fault_t ret = 0; pte_t entry; @@ -3525,77 +3525,66 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) !mm_forbids_zeropage(vma->vm_mm)) { entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address), vma->vm_page_prot)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (!pte_none(*vmf->pte)) { - update_mmu_tlb(vma, vmf->address, vmf->pte); - goto unlock; - } - ret = check_stable_address_space(vma->vm_mm); - if (ret) - goto unlock; - /* Deliver the page fault to userland, check inside PT lock */ - if (userfaultfd_missing(vma)) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - return handle_userfault(vmf, VM_UFFD_MISSING); - } - goto setpte; + } else { + /* Allocate our own private page. */ + if (unlikely(anon_vma_prepare(vma))) + goto oom; + page = alloc_zeroed_user_highpage_movable(vma, vmf->address); + if (!page) + goto oom; + + if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) + goto oom_free_page; + cgroup_throttle_swaprate(page, GFP_KERNEL); + + /* + * The memory barrier inside __SetPageUptodate makes sure that + * preceding stores to the page contents become visible before + * the set_pte_at() write. + */ + __SetPageUptodate(page); + + entry = mk_pte(page, vma->vm_page_prot); + if (vma->vm_flags & VM_WRITE) + entry = pte_mkwrite(pte_mkdirty(entry)); } - /* Allocate our own private page. */ - if (unlikely(anon_vma_prepare(vma))) - goto oom; - page = alloc_zeroed_user_highpage_movable(vma, vmf->address); - if (!page) - goto oom; - - if (mem_cgroup_charge(page, vma->vm_mm, GFP_KERNEL)) - goto oom_free_page; - cgroup_throttle_swaprate(page, GFP_KERNEL); - - /* - * The memory barrier inside __SetPageUptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. - */ - __SetPageUptodate(page); - - entry = mk_pte(page, vma->vm_page_prot); - if (vma->vm_flags & VM_WRITE) - entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (!pte_none(*vmf->pte)) { update_mmu_tlb(vma, vmf->address, vmf->pte); - goto release; + goto unlock; } ret = check_stable_address_space(vma->vm_mm); if (ret) - goto release; + goto unlock; /* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); - put_page(page); + if (page) + put_page(page); return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, vma, vmf->address, false); - lru_cache_add_inactive_or_unevictable(page, vma); -setpte: + if (page) { + inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); + page_add_new_anon_rmap(page, vma, vmf->address, false); + lru_cache_add_inactive_or_unevictable(page, vma); + } + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); + if (page) + put_page(page); return ret; -release: - put_page(page); - goto unlock; oom_free_page: put_page(page); oom: From patchwork Wed Apr 7 01:44:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5210BC433B4 for ; Wed, 7 Apr 2021 01:45:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E9CC6610CE for ; Wed, 7 Apr 2021 01:45:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9CC6610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A4A458D0002; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 983DF6B0089; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53E826B0089; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 226F26B0082 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D96518249980 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) X-FDA: 78003877812.24.6AF6E80 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf01.hostedemail.com (Postfix) with ESMTP id F33295001528 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=45X0hICb7YyLurpFubxOZ+gBL34g/MPyB7oW/aDWSiM=; b=Ezxxe9KDQsCFhUHAu9I8tXq3tzT+ETWkukiiTwYTUJy0+mEiULNUYHD1E3OGsCJ2WVgK+ 6sVgt4/DR/scN5rDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=45X0hICb7YyLurpFubxOZ+gBL34g/MPyB7oW/aDWSiM=; b=OIBfjJ1bYm37SwxCxTZxWBm6/JHot+77xHwF5V/8sYbQYkkLwxhE3KbqcmCsOgfilCuOe kPpyZrn7Wf5hzU3ilIC1gXGABhNrY4ctFg9vLpbXQhB1MQ3fIfDvSVlFjbQDpnshhEhCE2W jSgP0RqvYexT8jbDex/isE6RL/ElD3hYo1qqrkUe4YxlrI17K1OC3DDgjSOe4QxyDePMiNS ByZf0vq9GK578fKtRJ3Q9uSgRs1RMCV7tyWz+2ukebX+vT6OkKeM3iUxIQtur5QmxepoA+U J97XdXAFSjABk9diVgwmNmv0b2H4KQq+SRvewPy4ziSva9rDbfiYKXbB6KLw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 850A616023A; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 773DB19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 05/37] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Date: Tue, 6 Apr 2021 18:44:30 -0700 Message-Id: <20210407014502.24091-6-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: F33295001528 X-Stat-Signature: mbzqehcah57wgnom5yf8tmshcmq7c9qc Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-480571 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the code needed to handle speculative page fault. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the speculative page fault handler is called from the architecture's page faulting code, and some code has to be added there to try speculative fault handling first. Signed-off-by: Michel Lespinasse --- mm/Kconfig | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 24c045b24b95..322bda319dea 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -872,4 +872,26 @@ config MAPPING_DIRTY_HELPERS config KMAP_LOCAL bool +config ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT + def_bool n + +config SPECULATIVE_PAGE_FAULT + bool "Speculative page faults" + default y + depends on ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT && MMU && SMP + help + Try to handle user space page faults without holding the mmap lock. + + Instead of blocking writers through the use of mmap lock, + the page fault handler merely verifies, at the end of the page + fault, that no writers have been running concurrently with it. + + In high concurrency situations, the speculative fault handler + gains a throughput advantage by avoiding having to update the + mmap lock reader count. + + If the check fails due to a concurrent writer, or due to hitting + an unsupported case, the fault handler falls back to classical + processing using the mmap read lock. + endmenu From patchwork Wed Apr 7 01:44:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0687C433ED for ; Wed, 7 Apr 2021 01:45:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5CEEA61165 for ; Wed, 7 Apr 2021 01:45:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5CEEA61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 185538D0001; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D78756B0082; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 873166B008A; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0239.hostedemail.com [216.40.44.239]) by kanga.kvack.org (Postfix) with ESMTP id 477C66B0085 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 057A0180ACF8F for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.27.4305DB0 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf02.hostedemail.com (Postfix) with ESMTP id A24DB40002C6 for ; Wed, 7 Apr 2021 01:44:58 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=9K/3R1jhcIp0kkAhPFf8lQIqNdjGzS/Dm2et+ThCVUU=; b=HZg/9fcGH82GArMwzKRhgd90k9JTjmE0UcCqxUku3rDl/DNQoea5vEv+M1GVlvq2t5dFM o2dx1RgPy8IXCEPBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=9K/3R1jhcIp0kkAhPFf8lQIqNdjGzS/Dm2et+ThCVUU=; b=alyEbiamP4Wh9jF+Cx2z90l+ZloP7zp+sCdFqGMTrBl+ngldVXVxZFPfgoVFLtRW+aOvr w6NEEz/icCe8JtewfJfXSqdFKXwsuLj8L4yIU57lYCc0csIkmQXnIbNRH+P4/8dtGGhXJej ajivCfQ7cuXMtobqJclVzCilRXtrRCGS74qXlHBbu4Wa1RBeKgaSEI9lACDFGUNHyepK3j1 9Y82GPymzoZOKcFghOl4lzJvjkX5TsBoqSFpjOnPEcu9ciMWyzuzpXJeIL4iVk6NNR4KoyL zUrtAX7SGcN3sJaa4P1JJSwXkv6gN5VzeapLU0UDYhiiEmf30KHXCCp5NiQA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 8BF7D160244; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 7E43919F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 06/37] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Tue, 6 Apr 2021 18:44:31 -0700 Message-Id: <20210407014502.24091-7-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A24DB40002C6 X-Stat-Signature: 3dd84t1jjxyn8ri1kmm4ac7wp599orsp X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759898-98310 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT so that the speculative fault handling code can be compiled on this architecture. Signed-off-by: Michel Lespinasse --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879d398e..a93e4ed7040e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -33,6 +33,7 @@ config X86_64 select NEED_DMA_MAP_STATE select SWIOTLB select ARCH_HAS_ELFCORE_COMPAT + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT config FORCE_DYNAMIC_FTRACE def_bool y From patchwork Wed Apr 7 01:44:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3786C433B4 for ; Wed, 7 Apr 2021 01:45:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D410610CE for ; Wed, 7 Apr 2021 01:45:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D410610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 25C6C6B0089; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B6816B0095; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E79668E0001; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 5E5B46B008C for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 17949180ACEE4 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.18.5BD943C Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf09.hostedemail.com (Postfix) with ESMTP id ED0AD6000106 for ; Wed, 7 Apr 2021 01:45:04 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=52xfj//MRWSrbPuyYyafld4PYpnTwPxqkOlVrkxjHfQ=; b=GvnGI3H89uTnazRHTDjnxLW+ccgB0I6UmagHwVWO7+HvJmu/Xz4zrBwhYVsgpQqJep04E /mSNP7hSXpYMwncBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=52xfj//MRWSrbPuyYyafld4PYpnTwPxqkOlVrkxjHfQ=; b=OPSlT2/JohSmnlkUlugEyrQWJGbWdvh3hCrR2zpCuBAkb0PqOs9L/8a1xMIkjUTPb6y/Y cqCd+Rc9KW52GcFFxiN8SD6vfmn3I1JFhCo5KiccQgfvcM6CJc3W8zaXuT5ThUVmeQhKxwF Vv3Tln3K/YyIUTnOCtq8l27Jcl2tKkKlNdHXJkUu60voSM8GWMYr/XoFYw5Fn5K+GWMGnRZ bjeI1gOPLQVsR1Dt87WfKPyDXGDlQTD2GsUHD0sXH2S2FGmhMwJAXew2nTK7G34kOVswtcU MBvREvEpTSbD6cIqxKwy6GYP58Sd0uTstf6uUtlS27n+sX9m3uunc/9bmcmA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 93B5816024D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 8550819F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 07/37] mm: add FAULT_FLAG_SPECULATIVE flag Date: Tue, 6 Apr 2021 18:44:32 -0700 Message-Id: <20210407014502.24091-8-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ED0AD6000106 X-Stat-Signature: nz3yrhtrnrueabbsgz4p9fiznxw9spmw Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759904-418229 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define the new FAULT_FLAG_SPECULATIVE flag, which indicates when we are attempting speculative fault handling (without holding the mmap lock). Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8ba434287387..021fdab5b721 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -434,6 +434,7 @@ extern pgprot_t protection_map[16]; * @FAULT_FLAG_REMOTE: The fault is not for current task/mm. * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch. * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals. + * @FAULT_FLAG_SPECULATIVE: The fault is handled without holding the mmap_sem. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two @@ -464,6 +465,7 @@ extern pgprot_t protection_map[16]; #define FAULT_FLAG_REMOTE 0x80 #define FAULT_FLAG_INSTRUCTION 0x100 #define FAULT_FLAG_INTERRUPTIBLE 0x200 +#define FAULT_FLAG_SPECULATIVE 0x400 /* * The default fault flags that should be used by most of the @@ -501,7 +503,8 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_SPECULATIVE, "SPECULATIVE" } /* * vm_fault is filled by the pagefault handler and passed to the vma's From patchwork Wed Apr 7 01:44:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76CB7C43460 for ; Wed, 7 Apr 2021 01:45:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F099C613A7 for ; Wed, 7 Apr 2021 01:45:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F099C613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7D066B0088; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6D066B0087; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C4B36B0088; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 3315E6B0083 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id ED1EE6C2F for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) X-FDA: 78003877812.34.E328646 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf11.hostedemail.com (Postfix) with ESMTP id E718C200025C for ; Wed, 7 Apr 2021 01:45:01 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JSd8USyA44o7aisWRjJJmP/cvdrCX+QQ/xlTQKWni9A=; b=Mcq+tvl8z5tdgleIqie0lPGvo52Y/rVkX9Q70gSnW7/1nyO4h6x8+Y2tLz3+q6ZOfws7j 20Cpmhpc2n1VjYPDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JSd8USyA44o7aisWRjJJmP/cvdrCX+QQ/xlTQKWni9A=; b=ZSb7AHc/7/Pz09SX05JdQA0njLz+syzdV2IXZAgfAHmhRiolcNlHFH/3CNdEGBhnxENFM B4iV1cVjTFVKtRgcF4s2AFnptqp+XUyF7ZraP83Sk13Iq+aIpCYR4kv2jqtjwSs7I42jsH/ l/ipYRoZmqWwLoG3sounZvnGVXm4LAHBE+W00nAPf4xNvab25u0YkZT2c/RNx1xuPlofCU3 +i0wp85qLr/QEz1QOtdIas/1QNehEw9qRfnBzJBkX2sN6eRc9crG95VDt8WMl9ODNYEWAbK T9yhTEI5D4yBMKN8q4v7gViGKwCe8eL6g0oyVdtx6idZFEqT8ARf7Cd0ZItg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 9B7ED160253; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 8C3BA19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 08/37] mm: add do_handle_mm_fault() Date: Tue, 6 Apr 2021 18:44:33 -0700 Message-Id: <20210407014502.24091-9-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E718C200025C X-Stat-Signature: u6h9mx95xzhc3ec354p5aroyjxtqpq1r Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759901-749081 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new do_handle_mm_fault function, which extends the existing handle_mm_fault() API by adding an mmap sequence count, to be used in the FAULT_FLAG_SPECULATIVE case. In the initial implementation, FAULT_FLAG_SPECULATIVE always fails (by returning VM_FAULT_RETRY). The existing handle_mm_fault() API is kept as a wrapper around do_handle_mm_fault() so that we do not have to immediately update every handle_mm_fault() call site. Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 12 +++++++++--- mm/memory.c | 10 +++++++--- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 021fdab5b721..d5988e78e6ab 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1724,9 +1724,15 @@ int generic_error_remove_page(struct address_space *mapping, struct page *page); int invalidate_inode_page(struct page *page); #ifdef CONFIG_MMU -extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - struct pt_regs *regs); +extern vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + unsigned long seq, struct pt_regs *regs); +static inline vm_fault_t handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + struct pt_regs *regs) +{ + return do_handle_mm_fault(vma, address, flags, 0, regs); +} extern int fixup_user_fault(struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); diff --git a/mm/memory.c b/mm/memory.c index 477c98bfdd9d..3691be1f1319 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4509,11 +4509,15 @@ static inline void mm_account_fault(struct pt_regs *regs, * The mmap_lock may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, - unsigned int flags, struct pt_regs *regs) +vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, + unsigned long address, unsigned int flags, + unsigned long seq, struct pt_regs *regs) { vm_fault_t ret; + if (flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; + __set_current_state(TASK_RUNNING); count_vm_event(PGFAULT); @@ -4555,7 +4559,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, return ret; } -EXPORT_SYMBOL_GPL(handle_mm_fault); +EXPORT_SYMBOL_GPL(do_handle_mm_fault); #ifndef __PAGETABLE_P4D_FOLDED /* From patchwork Wed Apr 7 01:44:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BE18C433ED for ; Wed, 7 Apr 2021 01:45:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DC553610CE for ; Wed, 7 Apr 2021 01:45:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC553610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 75E9A8D000D; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16BC56B0083; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 519826B008A; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id 8F1206B0085 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 56F446C3B for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.21.628B30D Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf16.hostedemail.com (Postfix) with ESMTP id 5D18E80192C0 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Lp7zWyr2ROUjDnOgQqp9EncQocBAA2b6/i5LywQL7GQ=; b=JEBujiJmSTWdYEc7UtZmGCFy+tbS5UD1rkml1lTz27un8tBkop3pmKK9qOmdQFN78PJm5 QdI3ifaVwMwmbIBBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Lp7zWyr2ROUjDnOgQqp9EncQocBAA2b6/i5LywQL7GQ=; b=ZBLnIfbrq/6RNMCWxODmT5W8qYFafannsw9Gj6pTdWHn03bFfT4iWUSrv0lC7A0AQ/YGN ghwMAxsaqOHn3bzTP+TeGkMgPiNurvKBaSJWkZUPdNUd5DU679gip6NmxJP//7ocCqB3n2T qDXHA5hqETKAqd3GSWwUdbANZFOeO7r7rd4ZC17L2RSPsRUJ0+2eYItyF3iCxGpAW4RmQ+j j8aC8Bc2eC/1G6/AwJVuXMomylHnTN1HBMXE5G4Bdxy6Ajp9djsG2v9C7HYmOBUJeMBKvY1 0q33N01ai7e355VCLJXh4q5GcgSJoawSxl2est8H6nVsuHPQRc79PqaAzQ7w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id A353C160283; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 951AA19F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 09/37] mm: add per-mm mmap sequence counter for speculative page fault handling. Date: Tue, 6 Apr 2021 18:44:34 -0700 Message-Id: <20210407014502.24091-10-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5D18E80192C0 X-Stat-Signature: t9dyks9go5ty13hx17bdn6jb1kdno6j5 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-530990 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The counter's write side is hooked into the existing mmap locking API: mmap_write_lock() increments the counter to the next (odd) value, and mmap_write_unlock() increments it again to the next (even) value. The counter's speculative read side is supposed to be used as follows: seq = mmap_seq_read_start(mm); if (seq & 1) goto fail; .... speculative handling here .... if (!mmap_seq_read_check(mm, seq) goto fail; This API guarantees that, if none of the "fail" tests abort speculative execution, the speculative code section did not run concurrently with any mmap writer. This is very similar to a seqlock, but both the writer and speculative readers are allowed to block. In the fail case, the speculative reader does not spin on the sequence counter; instead it should fall back to a different mechanism such as grabbing the mmap lock read side. Signed-off-by: Michel Lespinasse --- include/linux/mm_types.h | 4 +++ include/linux/mmap_lock.h | 58 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6613b26a8894..70882e628908 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -461,6 +461,10 @@ struct mm_struct { * counters */ struct rw_semaphore mmap_lock; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + unsigned long mmap_seq; +#endif + struct list_head mmlist; /* List of maybe swapped mm's. These * are globally strung together off diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8ff276a7560e..8f4eca2d0f43 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -8,8 +8,16 @@ #include #include -#define MMAP_LOCK_INITIALIZER(name) \ - .mmap_lock = __RWSEM_INITIALIZER((name).mmap_lock), +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +#define MMAP_LOCK_SEQ_INITIALIZER(name) \ + .mmap_seq = 0, +#else +#define MMAP_LOCK_SEQ_INITIALIZER(name) +#endif + +#define MMAP_LOCK_INITIALIZER(name) \ + .mmap_lock = __RWSEM_INITIALIZER((name).mmap_lock), \ + MMAP_LOCK_SEQ_INITIALIZER(name) DECLARE_TRACEPOINT(mmap_lock_start_locking); DECLARE_TRACEPOINT(mmap_lock_acquire_returned); @@ -63,13 +71,52 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + mm->mmap_seq = 0; +#endif } +static inline void __mmap_seq_write_lock(struct mm_struct *mm) +{ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + VM_BUG_ON_MM(mm->mmap_seq & 1, mm); + mm->mmap_seq++; + smp_wmb(); +#endif +} + +static inline void __mmap_seq_write_unlock(struct mm_struct *mm) +{ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + smp_wmb(); + mm->mmap_seq++; + VM_BUG_ON_MM(mm->mmap_seq & 1, mm); +#endif +} + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static inline unsigned long mmap_seq_read_start(struct mm_struct *mm) +{ + unsigned long seq; + + seq = READ_ONCE(mm->mmap_seq); + smp_rmb(); + return seq; +} + +static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq) +{ + smp_rmb(); + return seq == READ_ONCE(mm->mmap_seq); +} +#endif + static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, true, true); + __mmap_seq_write_lock(mm); } static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) @@ -77,6 +124,7 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); __mmap_lock_trace_acquire_returned(mm, true, true); + __mmap_seq_write_lock(mm); } static inline int mmap_write_lock_killable(struct mm_struct *mm) @@ -86,6 +134,8 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); error = down_write_killable(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, true, !error); + if (likely(!error)) + __mmap_seq_write_lock(mm); return error; } @@ -96,17 +146,21 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ok = down_write_trylock(&mm->mmap_lock) != 0; __mmap_lock_trace_acquire_returned(mm, true, ok); + if (likely(ok)) + __mmap_seq_write_lock(mm); return ok; } static inline void mmap_write_unlock(struct mm_struct *mm) { + __mmap_seq_write_unlock(mm); up_write(&mm->mmap_lock); __mmap_lock_trace_released(mm, true); } static inline void mmap_write_downgrade(struct mm_struct *mm) { + __mmap_seq_write_unlock(mm); downgrade_write(&mm->mmap_lock); __mmap_lock_trace_acquire_returned(mm, false, true); } From patchwork Wed Apr 7 01:44:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11A43C433ED for ; Wed, 7 Apr 2021 01:45:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A556B61165 for ; Wed, 7 Apr 2021 01:45:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A556B61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5998E6B0082; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F5768E0003; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A465B8D0001; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id 484756B0087 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 090CE824999B for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.34.3BDAC23 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf21.hostedemail.com (Postfix) with ESMTP id F2B68E000107 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6AcV49LGNOIZDA+U0gUwsONW88L3iArajwLBo0wi7SM=; b=F/JpeEdbZWigeVCVPTTSdqZJ3RmaCAHjRMzqvo/hTl26Yo6cwAedoClDxm9iWkPyKyi+O VKNDqSGQWJJeSSUAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6AcV49LGNOIZDA+U0gUwsONW88L3iArajwLBo0wi7SM=; b=lOJnzzUAMXbVIAnQXUc17f7ODQpWkTLCGso6d9sNX2p5sFxA+RybJ/ZRVZnHdHU7FaNJg IfHJX/dxb0wlOIz+fQZs0lG8Q9BG/eQdeBCAUqG6kQFusXFdR0h6ztBi4yUyVGGFAS2m9+K xtdu6TgWjUuI5s4nYBdpYef3SiHwztecJhR9uDtW/78irlsStfiyMLvBZvGtXNjQMxIbSDt yOGmSHGUhrGfKyAsFTWwFk5iUlb6h3VqlVEtQIshhVrGvkzY2JJSac4ItU9CtuSUucZJQBB 7A+n6k5oyhzd11dcLHzh+Q/ozbJrXboZhwyKnbNg4U+8azA7seHZbHbxF9nQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id AA2801602A5; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 9A4F419F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 10/37] mm: rcu safe vma freeing Date: Tue, 6 Apr 2021 18:44:35 -0700 Message-Id: <20210407014502.24091-11-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: y4kgijnwqg7onsxid8n8rkxyysw3mbjk X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F2B68E000107 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-372120 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This prepares for speculative page faults looking up and copying vmas under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 11 ++++++++++- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 70882e628908..024970635921 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -304,12 +304,18 @@ struct vm_userfaultfd_ctx {}; struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 426cd0c51f9e..b6078e546114 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -369,11 +369,20 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static void __vm_area_free(struct rcu_head *head) { + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); kmem_cache_free(vm_area_cachep, vma); } +void vm_area_free(struct vm_area_struct *vma) +{ + call_rcu(&vma->vm_rcu, __vm_area_free); +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + static void account_kernel_stack(struct task_struct *tsk, int account) { void *stack = task_stack_page(tsk); From patchwork Wed Apr 7 01:44:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AD47C433B4 for ; Wed, 7 Apr 2021 01:45:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7D04613A7 for ; Wed, 7 Apr 2021 01:45:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7D04613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 94BB28D0003; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36C7E8E0005; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB69A6B0096; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 58CCB6B0082 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1FCA4181AEF3E for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.01.96B7E22 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf28.hostedemail.com (Postfix) with ESMTP id 0A9F7200024C for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=wmzTNOpmvJgU2+ktObZqYl7ztD+8ceDz1BXrR5WVsHA=; b=WqfE2X+chCev1QmePRjeuCrLGUj6TXALSmqGZlONI43Pl3PjzGxGJqBzjzNYC5wCMyWzX 7kAAXcM9WXxDLhPDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=wmzTNOpmvJgU2+ktObZqYl7ztD+8ceDz1BXrR5WVsHA=; b=j2O1Dn7hwEqXxPaFt1j92dvD5LnaAj1gNu182V8uNbDW77KXiHD/IvchklgWFg+3HdPiu 0Wc9n3eoMVW6ufA4LGoVG1h8aMAZnDdsDM87uQC2QGKU45z5x8VGSn2NolG0WP5mUmLyC5w wANilgJ5mW8LgdMoel/qRH9I9DLbNlbun1WZwh52o/0QvbkjyX47f9yNvp5K8r6/VRFO8X+ oTiFyJ4wp1U6mYbAjJaydTepcGkgyyY80luQ3L+1EJIBqxNkK8dts6OyilOpk2JiMaoB/j5 pIlBWfU3k+iopAQl7hfoymM6wZ6vImnYBVfbuLM7xeJbWFHLEyBN0wo7t5cw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id AF0A61602CB; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id A162919F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 11/37] x86/mm: attempt speculative mm faults first Date: Tue, 6 Apr 2021 18:44:36 -0700 Message-Id: <20210407014502.24091-12-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0A9F7200024C X-Stat-Signature: dkxgkfrttcdqnkkx6f9pk9khdeih3emz Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-603995 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. The speculative handling closely mirrors the non-speculative logic. This includes some x86 specific bits such as the access_error() call. This is why we chose to implement the speculative handling in arch/x86 rather than in common code. The vma is first looked up and copied, under protection of the rcu read lock. The mmap lock sequence count is used to verify the integrity of the copied vma, and passed to do_handle_mm_fault() to allow checking against races with mmap writers when finalizing the fault. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 36 +++++++++++++++++++++++++++++++++++ include/linux/vm_event_item.h | 4 ++++ mm/vmstat.c | 4 ++++ 3 files changed, 44 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a73347e2cdfc..f8c8e325af77 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1219,6 +1219,8 @@ void do_user_addr_fault(struct pt_regs *regs, struct mm_struct *mm; vm_fault_t fault; unsigned int flags = FAULT_FLAG_DEFAULT; + struct vm_area_struct pvma; + unsigned long seq; tsk = current; mm = tsk->mm; @@ -1316,6 +1318,39 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) + goto spf_abort; + rcu_read_lock(); + vma = find_vma(mm, address); + if (!vma || vma->vm_start > address) { + rcu_read_unlock(); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq)) + goto spf_abort; + vma = &pvma; + if (unlikely(access_error(error_code, vma))) + goto spf_abort; + fault = do_handle_mm_fault(vma, address, + flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR); + return; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1412,6 +1447,7 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +done: if (likely(!(fault & VM_FAULT_ERROR))) return; diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 18e75974d4e3..cc4f8d14e43f 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -120,6 +120,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SWAP SWAP_RA, SWAP_RA_HIT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + SPF_ATTEMPT, + SPF_ABORT, #endif NR_VM_EVENT_ITEMS }; diff --git a/mm/vmstat.c b/mm/vmstat.c index 74b2c374b86c..9ae1c27a549e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1365,6 +1365,10 @@ const char * const vmstat_text[] = { "swap_ra", "swap_ra_hit", #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + "spf_attempt", + "spf_abort", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Wed Apr 7 01:44:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E0A0C433B4 for ; Wed, 7 Apr 2021 01:45:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3CB56610CE for ; Wed, 7 Apr 2021 01:45:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3CB56610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A5DA8E000B; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A83E8E0009; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 945D98E0006; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id 99D306B0092 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5AC448249980 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.29.60C1785 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id 5EE26600010B for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=mO5MU45MyVZcFHxHRLPz46S2R58Y3f1o/RvXopsTgug=; b=x8Ntsiwc/FMdqgxnWlPWGRMTMXKGXUd7FH7ae8ZNK07QdvjB9kdBhnKqEpDWUGep8WKgH agBG8aS8vepwuR9DQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=mO5MU45MyVZcFHxHRLPz46S2R58Y3f1o/RvXopsTgug=; b=psR8q2EllG8INTzPGm5GoRHikfJCeKeFiVpoGicOrWGGGHq0JVzKS7chnTxPadxN82J4S jgkUavAqZfvS7qtYoZ30w4IeLh1awXgYDxl7BwEvUJ4oQE6+AhCFkcE0Qzh08C4HXiovxGS hLlPXxtWwLKFOgCkA3eP9WXrmsitGCWenDzpIgjLC7olWkErJSelAlrPp2mvIB7lLX50swf YYOHYT/YnPjrziUErSUmlNX1HI0Z670r9mkloDUGgn4ugJbzNQcZloAxwZcMhUFKxiONRWf tBhnOM8mm/8aax6ivgVkb1HvaqDBIQwG/lC6CHAPXCUknPqfVjHYSjx3gBhg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id B8D011602D1; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id A8AAC19F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 12/37] mm: refactor __handle_mm_fault() / handle_pte_fault() Date: Tue, 6 Apr 2021 18:44:37 -0700 Message-Id: <20210407014502.24091-13-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: u1jcx86p5rg9r1acjogb4wfm8uafktig X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5EE26600010B Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-77036 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the code that initializes vmf->pte and vmf->orig_pte from handle_pte_fault() to its single call site in __handle_mm_fault(). This ensures vmf->pte is now initialized together with the higher levels of the page table hierarchy. This also prepares for speculative page fault handling, where the entire page table walk (higher levels down to ptes) needs special care in the speculative case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 98 ++++++++++++++++++++++++++--------------------------- 1 file changed, 49 insertions(+), 49 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3691be1f1319..66e7a4554c54 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3516,7 +3516,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; - /* See comment in handle_pte_fault() */ + /* See comment in __handle_mm_fault() */ if (unlikely(pmd_trans_unstable(vmf->pmd))) return 0; @@ -3797,7 +3797,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } - /* See comment in handle_pte_fault() */ + /* See comment in __handle_mm_fault() */ if (pmd_devmap_trans_unstable(vmf->pmd)) return 0; @@ -4253,53 +4253,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; - if (unlikely(pmd_none(*vmf->pmd))) { - /* - * Leave __pte_alloc() until later: because vm_ops->fault may - * want to allocate huge page, and if we expose page table - * for an instant, it will be difficult to retract from - * concurrent faults and from rmap lookups. - */ - vmf->pte = NULL; - } else { - /* - * If a huge pmd materialized under us just retry later. Use - * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead - * of pmd_trans_huge() to ensure the pmd didn't become - * pmd_trans_huge under us and then back to pmd_none, as a - * result of MADV_DONTNEED running immediately after a huge pmd - * fault in a different thread of this mm, in turn leading to a - * misleading pmd_trans_huge() retval. All we have to ensure is - * that it is a regular pmd that we can walk with - * pte_offset_map() and we can do that through an atomic read - * in C, which is what pmd_trans_unstable() provides. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return 0; - /* - * A regular pmd is established and it can't morph into a huge - * pmd from under us anymore at this point because we hold the - * mmap_lock read mode and khugepaged takes it in write mode. - * So now it's safe to run pte_offset_map(). - */ - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - vmf->orig_pte = *vmf->pte; - - /* - * some architectures can have larger ptes than wordsize, - * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and - * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic - * accesses. The code below just needs a consistent view - * for the ifs and we later double check anyway with the - * ptl lock held. So here a barrier will do. - */ - barrier(); - if (pte_none(vmf->orig_pte)) { - pte_unmap(vmf->pte); - vmf->pte = NULL; - } - } - if (!vmf->pte) { if (vma_is_anonymous(vmf->vma)) return do_anonymous_page(vmf); @@ -4439,6 +4392,53 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, } } + if (unlikely(pmd_none(*vmf.pmd))) { + /* + * Leave __pte_alloc() until later: because vm_ops->fault may + * want to allocate huge page, and if we expose page table + * for an instant, it will be difficult to retract from + * concurrent faults and from rmap lookups. + */ + vmf.pte = NULL; + } else { + /* + * If a huge pmd materialized under us just retry later. Use + * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead + * of pmd_trans_huge() to ensure the pmd didn't become + * pmd_trans_huge under us and then back to pmd_none, as a + * result of MADV_DONTNEED running immediately after a huge pmd + * fault in a different thread of this mm, in turn leading to a + * misleading pmd_trans_huge() retval. All we have to ensure is + * that it is a regular pmd that we can walk with + * pte_offset_map() and we can do that through an atomic read + * in C, which is what pmd_trans_unstable() provides. + */ + if (pmd_devmap_trans_unstable(vmf.pmd)) + return 0; + /* + * A regular pmd is established and it can't morph into a huge + * pmd from under us anymore at this point because we hold the + * mmap_lock read mode and khugepaged takes it in write mode. + * So now it's safe to run pte_offset_map(). + */ + vmf.pte = pte_offset_map(vmf.pmd, vmf.address); + vmf.orig_pte = *vmf.pte; + + /* + * some architectures can have larger ptes than wordsize, + * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and + * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic + * accesses. The code below just needs a consistent view + * for the ifs and we later double check anyway with the + * ptl lock held. So here a barrier will do. + */ + barrier(); + if (pte_none(vmf.orig_pte)) { + pte_unmap(vmf.pte); + vmf.pte = NULL; + } + } + return handle_pte_fault(&vmf); } From patchwork Wed Apr 7 01:44:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D89C7C433B4 for ; Wed, 7 Apr 2021 01:45:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 85D50610CE for ; Wed, 7 Apr 2021 01:45:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 85D50610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 54C5D6B008C; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E06598E0005; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 259278D0003; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 791D26B0083 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2B06E181AF5D7 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.04.0C8ACDD Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf18.hostedemail.com (Postfix) with ESMTP id AA210200025D for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=gxiEjs3UamgICAjcExK6Xv0vUYOisQw5tbFkDNnL1Yg=; b=M+Vbi9XcUFyJS7ZDkkqa+H8KEL7SSWkzVMvlTUJSLA+6CaPV7D2MyEmCXhfHkUJcpheXp tUCcSGJqFdDrivMBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=gxiEjs3UamgICAjcExK6Xv0vUYOisQw5tbFkDNnL1Yg=; b=EYPg/iPPjrjCT9TaAf8GFH8k/qPsEVBVUWi0ZnzpFFwnKitFbiU3tDpvErtzM9mk+iW8t FGX2YbY+0JnVfjP7tL5JGK0QozItPLpqtNpHsKXlvlnxJJRZfkkjFh+SLNculSEXMk82CW8 syVpBS3X22Y/T5/XeOKsHc/rf/POShxJ70Nx5legJvCtgJhSjgxY9NSigMbuvpG26KiNe39 m7QDX/BDQSDL66UnVqLABDoNr5fG8XEmM7agCZ61RXsUyVmSGBgAUkLAQHTq0Uf1hgBz8+c XYgU/XndwJq3g3MAD4WzFtmb1Ax4rvEe1ax1dqZ5IcnBAGoxbSayx/NcnQPw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id BEA7C1602D2; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id AFD7B19F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 13/37] mm: implement speculative handling in __handle_mm_fault(). Date: Tue, 6 Apr 2021 18:44:38 -0700 Message-Id: <20210407014502.24091-14-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: p5e3ffj8j3ujjct77djr9wjax9dr49o1 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AA210200025D Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-172394 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page table tree is walked with local irqs disabled, which prevents page table reclamation (similarly to what fast GUP does). The logic is otherwise similar to the non-speculative path, but with additional restrictions: in the speculative path, we do not handle huge pages or wiring new pages tables. Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 4 +++ mm/memory.c | 77 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d5988e78e6ab..dee8a4833779 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -525,6 +525,10 @@ struct vm_fault { }; unsigned int flags; /* FAULT_FLAG_xxx flags * XXX: should really be 'const' */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + unsigned long seq; + pmd_t orig_pmd; +#endif pmd_t *pmd; /* Pointer to pmd entry matching * the 'address' */ pud_t *pud; /* Pointer to pud entry matching diff --git a/mm/memory.c b/mm/memory.c index 66e7a4554c54..a17704aac019 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4307,7 +4307,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * return value. See filemap_fault() and __lock_page_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, unsigned long seq) { struct vm_fault vmf = { .vma = vma, @@ -4322,6 +4322,79 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + if (flags & FAULT_FLAG_SPECULATIVE) { + pgd_t pgdval; + p4d_t p4dval; + pud_t pudval; + + vmf.seq = seq; + + local_irq_disable(); + pgd = pgd_offset(mm, address); + pgdval = READ_ONCE(*pgd); + if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) + goto spf_fail; + + p4d = p4d_offset(pgd, address); + p4dval = READ_ONCE(*p4d); + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) + goto spf_fail; + + vmf.pud = pud_offset(p4d, address); + pudval = READ_ONCE(*vmf.pud); + if (pud_none(pudval) || unlikely(pud_bad(pudval)) || + unlikely(pud_trans_huge(pudval)) || + unlikely(pud_devmap(pudval))) + goto spf_fail; + + vmf.pmd = pmd_offset(vmf.pud, address); + vmf.orig_pmd = READ_ONCE(*vmf.pmd); + + /* + * pmd_none could mean that a hugepage collapse is in + * progress in our back as collapse_huge_page() mark + * it before invalidating the pte (which is done once + * the IPI is catched by all CPU and we have interrupt + * disabled). For this reason we cannot handle THP in + * a speculative way since we can't safely identify an + * in progress collapse operation done in our back on + * that PMD. + */ + if (unlikely(pmd_none(vmf.orig_pmd) || + is_swap_pmd(vmf.orig_pmd) || + pmd_trans_huge(vmf.orig_pmd) || + pmd_devmap(vmf.orig_pmd))) + goto spf_fail; + + /* + * The above does not allocate/instantiate page-tables because + * doing so would lead to the possibility of instantiating + * page-tables after free_pgtables() -- and consequently + * leaking them. + * + * The result is that we take at least one non-speculative + * fault per PMD in order to instantiate it. + */ + + vmf.pte = pte_offset_map(vmf.pmd, address); + vmf.orig_pte = READ_ONCE(*vmf.pte); + barrier(); + if (pte_none(vmf.orig_pte)) { + pte_unmap(vmf.pte); + vmf.pte = NULL; + } + + local_irq_enable(); + + return handle_pte_fault(&vmf); + +spf_fail: + local_irq_enable(); + return VM_FAULT_RETRY; + } +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -4541,7 +4614,7 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, if (unlikely(is_vm_hugetlb_page(vma))) ret = hugetlb_fault(vma->vm_mm, vma, address, flags); else - ret = __handle_mm_fault(vma, address, flags); + ret = __handle_mm_fault(vma, address, flags, seq); if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); From patchwork Wed Apr 7 01:44:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6117EC43462 for ; Wed, 7 Apr 2021 01:45:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 08CA061165 for ; Wed, 7 Apr 2021 01:45:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 08CA061165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0FFA8E0003; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71ED56B0098; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D4178E0002; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id 7E7FD6B0093 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 37CD14408 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.04.42BB85A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf26.hostedemail.com (Postfix) with ESMTP id 6262F40002C0 for ; Wed, 7 Apr 2021 01:45:04 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6Xk9Sm2CAPu5UeFzJfU4JDeDX0RiT2wns7kOozzzECg=; b=7grdFgIOFoL+sFz8eTYiEWlsZlFNu3QaQ7BjZW/uEilCXudw/Peh55wHDyXVBajW+uJn7 vbXGmZj0pr/8Rg2Bw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=6Xk9Sm2CAPu5UeFzJfU4JDeDX0RiT2wns7kOozzzECg=; b=NZYHSkYE1Fgtl4w74m11l7+TLPmFTlO0VLSwgHrpfs1dB9loZbm9gBfrfN2dVOk+8QSWu 7Yk9CsMDybHvAqobUgqKdZ1kfMA1Srp57g90H2PyY0iA9rLLEndU14xKiYnRLFE2Q2qZkpk Lx8pDdT7mA1ZFwW6dIOR2B9czle+y2AytZoCaryNtROKJQk+V3/Y7NENQYEkyllygAlMYLi aCzPxKEzK7QNNU4Ekddh3dMo7ok73SsSo7H6FfTAeIraY9bzDETnW6bMQN1yqKw6HTrxDUL bEeN0e1eqM1fwFsH4x05mEuArrTM2yPRH9xpvykz1hyHe++qFBlSgIQOjhVg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id C9B431602D3; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id B755B19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 14/37] mm: add pte_map_lock() and pte_spinlock() Date: Tue, 6 Apr 2021 18:44:39 -0700 Message-Id: <20210407014502.24091-15-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6262F40002C0 X-Stat-Signature: 15k3dpkartzbyxcgxyegg7grt5e8n139 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759904-268503 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_map_lock() and pte_spinlock() are used by fault handlers to ensure the pte is mapped and locked before they commit the faulted page to the mm's address space at the end of the fault. The functions differ in their preconditions; pte_map_lock() expects the pte to be unmapped prior to the call, while pte_spinlock() expects it to be already mapped. In the speculative fault case, the functions verify, after locking the pte, that the mmap sequence count has not changed since the start of the fault, and thus that no mmap lock writers have been running concurrently with the fault. After that point the page table lock serializes any further races with concurrent mmap lock writers. If the mmap sequence count check fails, both functions will return false with the pte being left unmapped and unlocked. Signed-off-by: Michel Lespinasse --- include/linux/mm.h | 34 ++++++++++++++++++++++ mm/memory.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dee8a4833779..f26490aff514 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3183,5 +3183,39 @@ extern int sysctl_nr_trim_pages; void mem_dump_obj(void *object); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + +bool __pte_map_lock(struct vm_fault *vmf); + +static inline bool pte_map_lock(struct vm_fault *vmf) +{ + VM_BUG_ON(vmf->pte); + return __pte_map_lock(vmf); +} + +static inline bool pte_spinlock(struct vm_fault *vmf) +{ + VM_BUG_ON(!vmf->pte); + return __pte_map_lock(vmf); +} + +#else /* !CONFIG_SPECULATIVE_PAGE_FAULT */ + +static inline bool pte_map_lock(struct vm_fault *vmf) +{ + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); + return true; +} + +static inline bool pte_spinlock(struct vm_fault *vmf) +{ + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + return true; +} + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + #endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index a17704aac019..3e192d5f89a6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2566,6 +2566,77 @@ int apply_to_existing_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_existing_page_range); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + +bool __pte_map_lock(struct vm_fault *vmf) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + pmd_t pmdval; +#endif + pte_t *pte = vmf->pte; + spinlock_t *ptl; + + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + if (!pte) + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + spin_lock(vmf->ptl); + return true; + } + + local_irq_disable(); + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + goto fail; + /* + * The mmap sequence count check guarantees that the page + * tables are still valid at that point, and having IRQs + * disabled ensures that they stay around (see Fast GUP + * comment in mm/gup.c). + */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * We check if the pmd value is still the same to ensure that there + * is not a huge collapse operation in progress in our back. + */ + pmdval = READ_ONCE(*vmf->pmd); + if (!pmd_same(pmdval, vmf->orig_pmd)) + goto fail; +#endif + ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + if (!pte) + pte = pte_offset_map(vmf->pmd, vmf->address); + /* + * Try locking the page table. + * + * Note that we might race against zap_pte_range() which + * invalidates TLBs while holding the page table lock. + * We still have local IRQs disabled here to prevent the + * page table from being reclaimed, and zap_pte_range() could + * thus deadlock with us if we tried using spin_lock() here. + * + * We also don't want to retry until spin_trylock() succeeds, + * because of the starvation potential against a stream of lockers. + */ + if (unlikely(!spin_trylock(ptl))) + goto fail; + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + goto unlock_fail; + local_irq_enable(); + vmf->pte = pte; + vmf->ptl = ptl; + return true; + +unlock_fail: + spin_unlock(ptl); +fail: + if (pte) + pte_unmap(pte); + local_irq_enable(); + return false; +} + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* * handle_pte_fault chooses page fault handler according to an entry which was * read non-atomically. Before making any commitment, on those architectures From patchwork Wed Apr 7 01:44:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE84DC433ED for ; Wed, 7 Apr 2021 01:45:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 89A51613A7 for ; Wed, 7 Apr 2021 01:45:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 89A51613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C75DB8E0002; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64FFA6B0098; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D08668D000D; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 3D0506B0083 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 075B562E8 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.11.618B51B Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf01.hostedemail.com (Postfix) with ESMTP id 155AD5001528 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Iko8U1DIjbSXvB60SArqfpM0eO/JE4XY/9ww0rF5/tY=; b=JVh6JjMxzNn89HcbkhtljBCgR0nVPb//5K8shZAbIxj2Lntf8NNBDz3YEx3UFhO1kxgKJ 5YqE9FEx0+LD369DQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Iko8U1DIjbSXvB60SArqfpM0eO/JE4XY/9ww0rF5/tY=; b=dQPZHYw1jXJV689ACtM7eUIebt/kBXCd1dvdwBsYusq7L0GLbnqv2wFVikzZALG4eQHo0 hoYZ698ZhW9JVnYeTvVtYzKqNbyqh3sPMi8zirwDBwYhqXnd3a1YM2zYR+HuhFYu7FhwnJk VxWIMqJfqNWERV00aB2/UL2n2xZ2W6Rk2nQIPywaAzkmH+w2K6j4UepxDlR4xyiX9KSqoF0 lAwEqlEnrdlUjWpgBD+rlPCDDg75oOc+D9ZigT09EvxvepHfigE3X9S5ubzIc+GPWGt8qAy nyTOnIaOpDbdhTh2qphkLIKriI8niYvWYpvqDBXm8jrwEDynAzF4+3+P9+LQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id CE9BA1602D7; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id BE65319F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 15/37] mm: implement speculative handling in do_anonymous_page() Date: Tue, 6 Apr 2021 18:44:40 -0700 Message-Id: <20210407014502.24091-16-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 155AD5001528 X-Stat-Signature: cpdq5aa4ykf7apsc5f6pa3ryq9rj15xd Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-795460 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change do_anonymous_page() to handle the speculative case. This involves aborting speculative faults if they have to allocate a new anon_vma, and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Signed-off-by: Michel Lespinasse --- mm/memory.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3e192d5f89a6..fd84576f9c01 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3598,8 +3598,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vma->vm_page_prot)); } else { /* Allocate our own private page. */ - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; + if (__anon_vma_prepare(vma)) + goto oom; + } page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!page) goto oom; @@ -3620,8 +3624,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = pte_mkwrite(pte_mkdirty(entry)); } - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto release; + } if (!pte_none(*vmf->pte)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto unlock; @@ -3636,6 +3642,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); if (page) put_page(page); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; return handle_userfault(vmf, VM_UFFD_MISSING); } @@ -3653,6 +3661,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return 0; unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); +release: if (page) put_page(page); return ret; From patchwork Wed Apr 7 01:44:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93090C433ED for ; Wed, 7 Apr 2021 01:45:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3EBDC610CE for ; Wed, 7 Apr 2021 01:45:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3EBDC610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2D7088E0011; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD2AD6B0085; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14B658E0007; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0125.hostedemail.com [216.40.44.125]) by kanga.kvack.org (Postfix) with ESMTP id 586636B008C for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 155926C2F for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.27.50DDF7A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf29.hostedemail.com (Postfix) with ESMTP id 2B7B6D6 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Mm4sRKZ9oPWWjYrlQVTDWSlP7K2SJOXE1Iezs0Haf/4=; b=hJKxBzc+YjcRxjNqDuMMXauTqpgX6EMXmblbyYlLywC9uaLxI4Z+Bio3AkTWuAVx/1JC4 4hIOBBUYzycNFX2Aw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=Mm4sRKZ9oPWWjYrlQVTDWSlP7K2SJOXE1Iezs0Haf/4=; b=oNyM6JjSd6DoO5pFFb7RFw2adq0RuCvDIcOprNuCa7GNE9NteCWGBTYF1RjLLOYs5q6NG hyyyt+UrHy8Svrf6p35yGy6pUQg6E26BAYUi1wuMWsq24vIL4XqSvQhJ449QW2yaNFTEHDX eX4Bs6oSx9b4ZiV2n/gVaK7o28yIR31cY2TMUdmFE2EBUx1WOe0xWBqcPcCZYRdp0ZSJLhb Tv1wO7MMA3GpFI6tr4H67khd0Kdb5EKmCcjdB4GXl5KjEwBkoI/dNUMuMVffJsBZYLvPWfy vrB3LR7z41KD9Pn3d3tMfA76/tU1tf90D+gAKXg8hwdc3LA9T7VTmlGsZh+A== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id D62311602F1; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id C57F019F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 16/37] mm: enable speculative fault handling through do_anonymous_page() Date: Tue, 6 Apr 2021 18:44:41 -0700 Message-Id: <20210407014502.24091-17-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2B7B6D6 X-Stat-Signature: mci7zqjito6c9fdc7p4gumqjwjteckn1 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-111043 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: in x86 fault handler, only attempt spf if the vma is anonymous. In do_handle_mm_fault(), let speculative page faults proceed as long as they fall into anonymous vmas. This enables the speculative handling code in __handle_mm_fault() and do_anonymous_page(). In handle_pte_fault(), if vmf->pte is set (the original pte was not pte_none), catch speculative faults and return VM_FAULT_RETRY as those cases are not implemented yet. Also assert that do_fault() is not reached in the speculative case. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 2 +- mm/memory.c | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index f8c8e325af77..fbf265f56a06 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1324,7 +1324,7 @@ void do_user_addr_fault(struct pt_regs *regs, goto spf_abort; rcu_read_lock(); vma = find_vma(mm, address); - if (!vma || vma->vm_start > address) { + if (!vma || vma->vm_start > address || !vma_is_anonymous(vma)) { rcu_read_unlock(); goto spf_abort; } diff --git a/mm/memory.c b/mm/memory.c index fd84576f9c01..a2c5bf29f989 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4103,6 +4103,8 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ @@ -4340,6 +4342,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) return do_fault(vmf); } + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); @@ -4668,8 +4675,7 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, { vm_fault_t ret; - if (flags & FAULT_FLAG_SPECULATIVE) - return VM_FAULT_RETRY; + VM_BUG_ON((flags & FAULT_FLAG_SPECULATIVE) && !vma_is_anonymous(vma)); __set_current_state(TASK_RUNNING); @@ -4691,10 +4697,12 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, if (flags & FAULT_FLAG_USER) mem_cgroup_enter_user_fault(); - if (unlikely(is_vm_hugetlb_page(vma))) + if (unlikely(is_vm_hugetlb_page(vma))) { + VM_BUG_ON(flags & FAULT_FLAG_SPECULATIVE); ret = hugetlb_fault(vma->vm_mm, vma, address, flags); - else + } else { ret = __handle_mm_fault(vma, address, flags, seq); + } if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); From patchwork Wed Apr 7 01:44:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31491C43600 for ; Wed, 7 Apr 2021 01:45:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D2B0B613A7 for ; Wed, 7 Apr 2021 01:45:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D2B0B613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EDC358E000D; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 914248E0005; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1924F6B0085; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 4D40A6B0087 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1F969180ACEE4 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.09.6AFA25B Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf06.hostedemail.com (Postfix) with ESMTP id C519DC0007C9 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=oDsaekR07u2ZtR1enj5FUb1cWHHQoW5twL22pxVnaDw=; b=aVLbPOwuUGPHPf1Ujhsv2FvPJz0fLCHY7Qk5nH/FdfC6t9RP45dBC+UVkh7OEJwRA5NGO gINHLP2nA2rEEImAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=oDsaekR07u2ZtR1enj5FUb1cWHHQoW5twL22pxVnaDw=; b=IPnOCuRQxi3KY+ZyKablrM9wkR6ZD4fXn+28c+kkjDkkYrkusvUcQX/piob3616D3gf62 867C15f7ac0Z7Yfe2+ANNkmYV/Cdn2iHdUIvLzrvtXTwyzroXU/F5+tdBV5gCgTsZ3DRPD+ OFCC83+W+Btw7i/N6KVbq6llm0/a8KLfAe8uAGtIzc7jsTC3lMCAcfScT/Pn6mbPtVzzzBp DIryKpr1+tOH7RwTYt56bIOomT/HwAqSgI7HR+WPaGM4BMfuzgA2XHI1YwuXQfFKmdmvBqU q1Cav343GpyKux7Qqo3LPUSsXtGPMH/GpzC+UyOiansbtZ54CG0zffUbwEcA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id DDE761602FF; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id CC30019F320; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 17/37] mm: implement speculative handling in do_numa_page() Date: Tue, 6 Apr 2021 18:44:42 -0700 Message-Id: <20210407014502.24091-18-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C519DC0007C9 X-Stat-Signature: kbd7mw8nmdjbnmdk147urycwqtjcsek8 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759908-367115 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: change do_numa_page() to use pte_spinlock() when locking the page table, so that the mmap sequence counter will be validated in the speculative case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a2c5bf29f989..78cc36749754 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4181,8 +4181,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * validation through pte_unmap_same(). It's of NUMA type but * the pfn may be screwed if the read is non atomic. */ - vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; From patchwork Wed Apr 7 01:44:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8309C433B4 for ; Wed, 7 Apr 2021 01:45:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6673861165 for ; Wed, 7 Apr 2021 01:45:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6673861165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 90EA16B0083; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F005E8E000E; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C98B8E000A; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 6B25A6B0093 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1C47F180ACF8F for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.28.F0456AF Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf21.hostedemail.com (Postfix) with ESMTP id 15769E00010E for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=7VQ5ACimMolgPW4s1A3qwSpAn4d6FTK3GCINFHzHKDI=; b=eoRVwI4VRjX6AijehpC7K3tU1RbT1JP295NDDl/XBGqvnKyJaXaUPR05Zgk1H6zM6EGZ1 u9NTXZPxmckNnhYCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=7VQ5ACimMolgPW4s1A3qwSpAn4d6FTK3GCINFHzHKDI=; b=B99SXq/PA3x+rlEbA0HjxJukiKRbVLDPhETEfSwSgzxEn+W9W06Uf2s5k9EroMwNl4xav 7NZeXigGVzV5aRvtNC5NWBkCKJtYRKHbtiXpjNex0oEdrMrneYB7rYvSW0O2603yNKvxSRR q80ALnrADxKmUhjyzYZOpjQ8i1FyAKRlRJj1LusRsAWcyJ/YNLqE8dPdmY+f141Q57CSUX/ DsPjOqypmX+BuOBIPyXCowz87r1zD8tlu3XmF6BLwrlhnnLfDy3RSdwVl4eCnRwAEuzDFPY 0YWa70k+FHAYSd6yR44f4WEHzjhB6k+E1R6+PETGVFqYf0mGkGvOQYorrk+Q== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id E2DAD160302; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id D29FC19F321; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 18/37] mm: enable speculative fault handling in do_numa_page() Date: Tue, 6 Apr 2021 18:44:43 -0700 Message-Id: <20210407014502.24091-19-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: jcbaa6uris5ywtbgxrn6t47ynp7j8fa8 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 15769E00010E Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-993884 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change handle_pte_fault() to allow speculative fault execution to proceed through do_numa_page(). do_swap_page() does not implement speculative execution yet, so it needs to abort with VM_FAULT_RETRY in that case. Signed-off-by: Michel Lespinasse --- mm/memory.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 78cc36749754..eea72bd78d06 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3346,6 +3346,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) goto out; @@ -4342,17 +4347,17 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) return do_fault(vmf); } - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); - return VM_FAULT_RETRY; - } - if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + pte_unmap(vmf->pte); + return VM_FAULT_RETRY; + } + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); entry = vmf->orig_pte; From patchwork Wed Apr 7 01:44:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A942AC43461 for ; Wed, 7 Apr 2021 01:45:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4D03C610CE for ; Wed, 7 Apr 2021 01:45:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D03C610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 667EB8E0012; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE7AD6B008A; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD43F8E0007; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0073.hostedemail.com [216.40.44.73]) by kanga.kvack.org (Postfix) with ESMTP id 976558D0005 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 52FAC180ACEE6 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.19.6EBBE39 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf26.hostedemail.com (Postfix) with ESMTP id 8CDCC40002C1 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8COoG1kyMAyzmcjmtCVhUuZfAMmUT556bb2lAEkl2Jk=; b=PHxlIUq8Gez4GGg1T1vZB3f/iGsF8+rBmKPA5lGRVNAhdo4q0AcUCTl4jG0ud1ymS98BV pXuyifekidg4jc1BQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=8COoG1kyMAyzmcjmtCVhUuZfAMmUT556bb2lAEkl2Jk=; b=Xj0qGgONLfNk5zEaM5IEBqutABXQbilGjAJuQCGy/NJElYzNQNWVz/qj9s1C6jR2zMKsp CjgSDeSQCsFCM+kAwSicmrOf/jMLccvhMVOhjghOl1FsBmbeoi8ks/+cPqSCRaAA0S5gXBo wH6j2k0bx/8iypailFquYrsYVPFz097+Hi4V3IjA75FYXoRM5H3/nt2VmtdgV28sESHbThT nPem9Matb4e4goClyC4hD+hIaDWZ5mKOjFOWWWxDQ0KasRhTnPowHxV9A95siPdM+tgbJcM rKG/7yOOiDxRufdYxGz6E0iQRg9rMSq43GJ4K/9MnRhNIYT5b2usXjV3CPcA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id E7B4016031A; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id D9CBE19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 19/37] mm: implement speculative handling in wp_page_copy() Date: Tue, 6 Apr 2021 18:44:44 -0700 Message-Id: <20210407014502.24091-20-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8CDCC40002C1 X-Stat-Signature: 7x6fys8xdjyrt4xmxh9ia8fc5zq4foda X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-508584 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change wp_page_copy() to handle the speculative case. This involves aborting speculative faults if they have to allocate an anon_vma, and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Also change call sites to clear vmf->pte after unmapping the page table, in order to satisfy pte_map_lock()'s preconditions. Signed-off-by: Michel Lespinasse --- mm/memory.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eea72bd78d06..547d9d0ee962 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2911,20 +2911,27 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; + vm_fault_t ret = VM_FAULT_OOM; - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret = VM_FAULT_RETRY; + goto out; + } + if (__anon_vma_prepare(vma)) + goto out; + } if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!new_page) - goto oom; + goto out; } else { new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!new_page) - goto oom; + goto out; if (!cow_user_page(new_page, old_page, vmf)) { /* @@ -2941,7 +2948,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } if (mem_cgroup_charge(new_page, mm, GFP_KERNEL)) - goto oom_free_new; + goto out_free_new; cgroup_throttle_swaprate(new_page, GFP_KERNEL); __SetPageUptodate(new_page); @@ -2954,7 +2961,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* * Re-check the pte - we dropped the lock */ - vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + /* put_page() will uncharge the page */ + goto out_free_new; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { if (old_page) { if (!PageAnon(old_page)) { @@ -3042,12 +3053,12 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) put_page(old_page); } return page_copied ? VM_FAULT_WRITE : 0; -oom_free_new: +out_free_new: put_page(new_page); -oom: +out: if (old_page) put_page(old_page); - return VM_FAULT_OOM; + return ret; } /** @@ -3190,6 +3201,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return wp_pfn_shared(vmf); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); } @@ -3228,6 +3240,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); } From patchwork Wed Apr 7 01:44:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F38B3C433B4 for ; Wed, 7 Apr 2021 01:45:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4FA361165 for ; Wed, 7 Apr 2021 01:45:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4FA361165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 633F58E0015; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E141C8E0008; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36B948E0008; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 647E86B0092 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2B3AB81CF for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.34.8E25F41 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf23.hostedemail.com (Postfix) with ESMTP id 9B25EA000395 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=amUPmRB3lhaJkohzuLZ4RtQ8DU3T54MYn5RloG5fdOk=; b=z2Fs4Sr8tgnyNRxzBI/Qpcy1P6DLuAkRsEWkyC+RTtb0wEJIEd05hnB2MY0kNtj3w4TPx izNmbCPXX4Oli6tCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=amUPmRB3lhaJkohzuLZ4RtQ8DU3T54MYn5RloG5fdOk=; b=STR1621+RtKTEUrfNGZuKFxncsNgsYjBbFmP0TY1i7nINnGvGd8RC7HxYZylM4TwF9ipc 2lJKhgHBNzrI/mmtaYhir2hDOmgFM0uMh2+9qOId0KEXhtaP/BYpzV/1c0420if3gvpykh2 q9FoAdafWcOOGwVE6Llm1KbgxG63CwFM3Dzf4b3fB8u3iUZNPLvGVlYBu29QFKF+igMX3bv Lcnb2SWEBQKu+0iLdI9bRaoQ7UYa4nEnI6AiUFRWpcLtXmiHhcmXP4x+Bk+ezGFAp4Q11Zt ehgmwHil39XlGZvKvXYkBiSkRNfP8THZyYuG9Y/SJYaePQGz3RwdeWVRGYlw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id EF7E4160330; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id E0F2F19F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 20/37] mm: implement and enable speculative fault handling in handle_pte_fault() Date: Tue, 6 Apr 2021 18:44:45 -0700 Message-Id: <20210407014502.24091-21-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9B25EA000395 X-Stat-Signature: py4gt3gzbqdqfewtjcjjazfht6mqsyer X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-13611 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In handle_pte_fault(), allow speculative execution to proceed. Use pte_spinlock() to validate the mmap sequence count when locking the page table. If speculative execution proceeds through do_wp_page(), ensure that we end up in the wp_page_reuse() or wp_page_copy() paths, rather than wp_pfn_shared() or wp_page_shared() (both unreachable as we only handle anon vmas so far) or handle_userfault() (needs an explicit abort to handle non-speculatively). Signed-off-by: Michel Lespinasse --- mm/memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 547d9d0ee962..fc555fae0844 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3103,6 +3103,7 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) { vm_fault_t ret; @@ -3123,6 +3124,8 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret = VM_FAULT_WRITE; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + get_page(vmf->page); if (vma->vm_ops && vma->vm_ops->page_mkwrite) { @@ -3176,6 +3179,8 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; return handle_userfault(vmf, VM_UFFD_WP); } @@ -4366,13 +4371,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); + if (!pte_spinlock(vmf)) return VM_FAULT_RETRY; - } - - vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) { update_mmu_tlb(vmf->vma, vmf->address, vmf->pte); From patchwork Wed Apr 7 01:44:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3651C43461 for ; Wed, 7 Apr 2021 01:45:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8FB5D61165 for ; Wed, 7 Apr 2021 01:45:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8FB5D61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1CB3F8E000E; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B5F28E0017; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1BA86B0096; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 8CA368E0002 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4C4C58249980 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.17.D9A78C6 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf07.hostedemail.com (Postfix) with ESMTP id 9C974A000396 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=CjG9cXBfCGz37cJgHr3H1dKyoE0oEoz4J1dkmh5NNCg=; b=yM6xNR3TMq76ZYckpLMGjkzx0S/w7f/gz1/7CSWCS9RX/0FOrYG+SfxcWVhU1zxZJBmOD /HlPaPF2xGrhS5hBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=CjG9cXBfCGz37cJgHr3H1dKyoE0oEoz4J1dkmh5NNCg=; b=q0/QAhX2NFAO+hY4Y//y0LFODZO05QGiHTCbTcypAuapYNan4SDX7zfsyzfEt1XdUkjr7 2KVgBZm7Sxj37H/8sMXwgaOt4pNU6ISdqlxBHJ0oACTMN+bFbaj1KbpnlXvuk+HEMHnlV9k BM80kjgIYIBVMbDF14NTSi51Vnfz3GkYiggzBWUTur9OWzbefrZ6DUXChgzsLZxoBGGrSh0 lpCPsKML05GVhgOWfIPLGOOOQ1jTOSeYb8oWE8z55oIj99zsZ9ILV2I34xYo68E7Nw4uv3p DmODchp4sYxTrYsb8lFUKDXU2IgXjF4tIyCCmzCzh47MYSEEJ0e6/veFr7VA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 023FA160350; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id E81F619F31F; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 21/37] mm: implement speculative handling in do_swap_page() Date: Tue, 6 Apr 2021 18:44:46 -0700 Message-Id: <20210407014502.24091-22-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9C974A000396 X-Stat-Signature: 8zihohywctqyir7arayjgzfhistmq8wx Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-936046 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the pte is larger than long, use pte_spinlock() to lock the page table when verifying the pte - pte_spinlock() is necessary to ensure the page table is still valid when we are locking it. Abort speculative faults if the pte is not a swap entry, or if the desired page is not found in swap cache, to keep things as simple as possible. Only use trylock when locking the swapped page - again to keep things simple, and also the usual lock_page_or_retry would otherwise try to release the mmap lock which is not held in the speculative case. Use pte_map_lock() to ensure proper synchronization when finally committing the faulted page to the mm address space. Signed-off-by: Michel Lespinasse --- mm/memory.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fc555fae0844..ab3160719bf3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2637,30 +2637,6 @@ bool __pte_map_lock(struct vm_fault *vmf) #endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ -/* - * handle_pte_fault chooses page fault handler according to an entry which was - * read non-atomically. Before making any commitment, on those architectures - * or configurations (e.g. i386 with PAE) which might give a mix of unmatched - * parts, do_swap_page must check under lock before unmapping the pte and - * proceeding (but do_wp_page is only called after already making such a check; - * and do_anonymous_page can safely check later on). - */ -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd, - pte_t *page_table, pte_t orig_pte) -{ - int same = 1; -#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) - if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(mm, pmd); - spin_lock(ptl); - same = pte_same(*page_table, orig_pte); - spin_unlock(ptl); - } -#endif - pte_unmap(page_table); - return same; -} - static inline bool cow_user_page(struct page *dst, struct page *src, struct vm_fault *vmf) { @@ -3369,12 +3345,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return VM_FAULT_RETRY; } - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) - goto out; +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) + if (sizeof(pte_t) > sizeof(unsigned long)) { + /* + * vmf->orig_pte was read non-atomically. Before making + * any commitment, on those architectures or configurations + * (e.g. i386 with PAE) which might give a mix of + * unmatched parts, we must check under lock before + * unmapping the pte and proceeding. + * + * (but do_wp_page is only called after already making + * such a check; and do_anonymous_page can safely + * check later on). + */ + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; + if (!pte_same(*vmf->pte, vmf->orig_pte)) + goto unlock; + spin_unlock(vmf->ptl); + } +#endif + pte_unmap(vmf->pte); + vmf->pte = NULL; entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { - if (is_migration_entry(entry)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret = VM_FAULT_RETRY; + } else if (is_migration_entry(entry)) { migration_entry_wait(vma->vm_mm, vmf->pmd, vmf->address); } else if (is_device_private_entry(entry)) { @@ -3395,8 +3393,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = page; if (!page) { - struct swap_info_struct *si = swp_swap_info(entry); + struct swap_info_struct *si; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + return VM_FAULT_RETRY; + } + + si = swp_swap_info(entry); if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ @@ -3459,7 +3463,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_release; } - locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + locked = trylock_page(page); + else + locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags); delayacct_clear_flag(DELAYACCT_PF_SWAPIN); if (!locked) { @@ -3487,10 +3494,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) cgroup_throttle_swaprate(page, GFP_KERNEL); /* - * Back out if somebody else already faulted in this pte. + * Back out if the VMA has changed in our back during a speculative + * page fault or if somebody else already faulted in this pte. */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto out_page; + } if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; From patchwork Wed Apr 7 01:44:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CCA7C433ED for ; Wed, 7 Apr 2021 01:45:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0922461165 for ; Wed, 7 Apr 2021 01:45:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0922461165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BA8EE8E0006; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E30D8E0005; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E7DA6B0095; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0248.hostedemail.com [216.40.44.248]) by kanga.kvack.org (Postfix) with ESMTP id 71E336B0096 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3200881D6 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.12.2AD47C1 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf05.hostedemail.com (Postfix) with ESMTP id 54217E000110 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=VxqJ3bt6CreFlT+E4DjXvWhsZS5gJ+ukh7zd7QuHvYk=; b=X3Xxdqgv/cgatkx6bKCq22bsDyuGIIFuutQU1wa2874aUSk8GjJZaziUZfVG9XXLdrI4V 2OUJDOleRQVwLd7Cw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=VxqJ3bt6CreFlT+E4DjXvWhsZS5gJ+ukh7zd7QuHvYk=; b=bt136Q8H0jbNSpwOh6Dm3mmUCo51a8AE75Ga5NiYl+NqjmDdqfmbcMcqVRCabeL568R29 QOV1rl19/dycIxYj6qjh5xGGTb3JZcZseB4ugflSTEx91ZKfCqN1yNS+gNUcjEe70Tjov8O jSOu/5mupmTj/x2dv6Jj/fvMkZ7DljWhsoae5tgXf2v17pkoqhqMGUHetO/9qIvSHqf0IB3 43aCaoZX9oubJLJZkd+NJnnie6w1ZqcrAHGbxuTWPlHsB7G4O2657zl+ev+3Pdv9MKz8N8l L39OQy2kIRL9HG0UzDQQJhoLw44rV7Cq+5tzxdGWfWNoaM4jIuwHLRFc02YQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 09034160356; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id EF43A19F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 22/37] mm: enable speculative fault handling through do_swap_page() Date: Tue, 6 Apr 2021 18:44:47 -0700 Message-Id: <20210407014502.24091-23-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: enyraddzj66yusyukg4nj66wwbky8gmd X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 54217E000110 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-970127 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change do_swap_page() to allow speculative fault execution to proceed. Signed-off-by: Michel Lespinasse --- mm/memory.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ab3160719bf3..6eddd7b4e89c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3340,11 +3340,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; - if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - pte_unmap(vmf->pte); - return VM_FAULT_RETRY; - } - #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { /* From patchwork Wed Apr 7 01:44:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C343C43460 for ; Wed, 7 Apr 2021 01:46:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB80C61165 for ; Wed, 7 Apr 2021 01:46:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB80C61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD7F18E000C; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14ACB8E000A; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16EE38E000F; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id B61DF8D0009 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6F06081D1 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.06.962C992 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf22.hostedemail.com (Postfix) with ESMTP id 36958C0007CA for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=4ziyv/pIXZ+c0Q5G5PWzYQBRlES+IZwjStJKbTMM+os=; b=1zfU2U/hZRjQOfQbf+ngGzUxUD0CAmdlCfE75iKwCJji3T80V31VCIjGrlAz4Vg2/qnUR cLMt72B2etEV5ErCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=4ziyv/pIXZ+c0Q5G5PWzYQBRlES+IZwjStJKbTMM+os=; b=Zk4enQyEEfaPQoXqLFfW8qu8XPvJ29ctA8eQmIOu5f7cy23VXpXwFyszzj7f/B4A5c+I/ T7XEBxP5iCS4irHJX9swHIlVK9DWYUe+gnQn6KljNjZOzg5yZ16CPVI3wqpxLm4Ugn/w4VE vw38o6Ovajp+16OwAsnx9pl7eoq3wSkeokZSFYFNjcOStaJd005IjYgCf5yzpOcVgY6+kPn qzJKLnN2RLPyB7Q2I2eroqhgkEI0TZpRrB+Y74rRmtcb/E6ldd2NZIY6004/l9BS8lp5Spb vUVer8Bak0nHBymTqRtUxDOEqcnxKHoYW0EZOExcseVtnKJW9tXHTh6KJywQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 10E05160369; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0272C19F31E; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 23/37] mm: rcu safe vma->vm_file freeing Date: Tue, 6 Apr 2021 18:44:48 -0700 Message-Id: <20210407014502.24091-24-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 36958C0007CA X-Stat-Signature: xqhqq4p9dur8su9gqx99exd65hosdfgo X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-461697 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Defer freeing of vma->vm_file when freeing vmas. This is to allow speculative page faults in the mapped file case. Signed-off-by: Michel Lespinasse Reported-by: kernel test robot --- fs/exec.c | 1 + kernel/fork.c | 17 +++++++++++++++-- mm/mmap.c | 11 +++++++---- mm/nommu.c | 6 ++---- 4 files changed, 25 insertions(+), 10 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index 18594f11c31f..c9da73eb0f53 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -286,6 +286,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) mmap_write_unlock(mm); err_free: bprm->vma = NULL; + VM_BUG_ON(vma->vm_file); vm_area_free(vma); return err; } diff --git a/kernel/fork.c b/kernel/fork.c index b6078e546114..2f20a5c5fed8 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -369,19 +369,31 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +static inline void ____vm_area_free(struct vm_area_struct *vma) +{ + if (vma->vm_file) + fput(vma->vm_file); + kmem_cache_free(vm_area_cachep, vma); +} + #ifdef CONFIG_SPECULATIVE_PAGE_FAULT static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); - kmem_cache_free(vm_area_cachep, vma); + ____vm_area_free(vma); } +#endif + void vm_area_free(struct vm_area_struct *vma) { +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT call_rcu(&vma->vm_rcu, __vm_area_free); +#else + ____vm_area_free(vma); +#endif } -#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ static void account_kernel_stack(struct task_struct *tsk, int account) { @@ -621,6 +633,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, fail_nomem_anon_vma_fork: mpol_put(vma_policy(tmp)); fail_nomem_policy: + tmp->vm_file = NULL; /* prevents fput within vm_area_free() */ vm_area_free(tmp); fail_nomem: retval = -ENOMEM; diff --git a/mm/mmap.c b/mm/mmap.c index 3f287599a7a3..cc2323e243bb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -178,9 +178,8 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) might_sleep(); if (vma->vm_ops && vma->vm_ops->close) vma->vm_ops->close(vma); - if (vma->vm_file) - fput(vma->vm_file); mpol_put(vma_policy(vma)); + /* fput(vma->vm_file) happens in vm_area_free after an RCU delay. */ vm_area_free(vma); return next; } @@ -949,7 +948,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (remove_next) { if (file) { uprobe_munmap(next, next->vm_start, next->vm_end); - fput(file); + /* fput(file) happens whthin vm_area_free(next) */ + VM_BUG_ON(file != next->vm_file); } if (next->anon_vma) anon_vma_merge(vma, next); @@ -1828,7 +1828,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * fput the vma->vm_file here or we would add an extra fput for file * and cause general protection fault ultimately. */ - fput(vma->vm_file); + /* fput happens within vm_area_free */ vm_area_free(vma); vma = merge; /* Update vm_flags to pick up the change. */ @@ -1907,6 +1907,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if (vm_flags & VM_DENYWRITE) allow_write_access(file); free_vma: + VM_BUG_ON(vma->vm_file); vm_area_free(vma); unacct_error: if (charged) @@ -2779,6 +2780,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, out_free_mpol: mpol_put(vma_policy(new)); out_free_vma: + new->vm_file = NULL; /* prevents fput within vm_area_free() */ vm_area_free(new); return err; } @@ -3343,6 +3345,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, out_free_mempol: mpol_put(vma_policy(new_vma)); out_free_vma: + new_vma->vm_file = NULL; /* Prevent fput within vm_area_free */ vm_area_free(new_vma); out: return NULL; diff --git a/mm/nommu.c b/mm/nommu.c index 5c9ab799c0e6..06a0dc0b913b 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -664,9 +664,8 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma) { if (vma->vm_ops && vma->vm_ops->close) vma->vm_ops->close(vma); - if (vma->vm_file) - fput(vma->vm_file); put_nommu_region(vma->vm_region); + /* fput(vma->vm_file) happens within vm_area_free() */ vm_area_free(vma); } @@ -1267,8 +1266,7 @@ unsigned long do_mmap(struct file *file, if (region->vm_file) fput(region->vm_file); kmem_cache_free(vm_region_jar, region); - if (vma->vm_file) - fput(vma->vm_file); + /* fput(vma->vm_file) happens within vm_area_free() */ vm_area_free(vma); return ret; From patchwork Wed Apr 7 01:44:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9837CC433ED for ; Wed, 7 Apr 2021 01:46:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4486661396 for ; Wed, 7 Apr 2021 01:46:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4486661396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9368F8E0007; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E66D28E0014; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F7EB6B0093; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 559006B0092 for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1DDAE8249980 for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) X-FDA: 78003877938.39.96E891D Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf08.hostedemail.com (Postfix) with ESMTP id C0B5F80192D4 for ; Wed, 7 Apr 2021 01:45:01 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=KSwVFXcXC1LvgNAhlUzawOb6cyImWm4fVIR5ZQJd5xc=; b=gRHSZhDTNNGCbnIvlycMkhUVlPwyxniNsdMuORhL95atZDObl7ItbtuphSDdUjX23gpa2 5g/6FTl+SQbi/A/Ag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=KSwVFXcXC1LvgNAhlUzawOb6cyImWm4fVIR5ZQJd5xc=; b=MKIjsemPluEEIll0eWbtQ+LZ5lhWdvrPCIPD45WpylLuLds09nW2fmnBC0BmLlFlKs2TE z+Lj8BxURaoMDZEGU6ShDzgdUM73C8UPBdI2ut40XJBRoFJdPKDR5fIZoALC1Wc7q2vJmq/ rkoreBCPgtSezyWIDekGZoGnWX/wmypzyEXeFSNVR5xeRO5qyroG1rwM8043lZjBRC1FaWB tlBOQw9nwVBE4W9R8+hOsrIf6MwXYLl1ZGQNV8/QsjZ1T4uTfgweNCuxKJlNtxHTrRzAKK4 aF/5rzRtLwNtEAKGDik+k8MEqgZ7ImHLBgSe8amwi/4oPZclODAzWRvqHH+w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 18CB516036D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 09A2819F31F; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 24/37] mm: implement speculative handling in __do_fault() Date: Tue, 6 Apr 2021 18:44:49 -0700 Message-Id: <20210407014502.24091-25-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C0B5F80192D4 X-Stat-Signature: 1cpzma4f8yu3yqssd4mtdk5catex7sj8 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759901-44549 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the speculative case, call the vm_ops->fault() method from within an rcu read locked section, and verify the mmap sequence lock at the start of the section. A match guarantees that the original vma is still valid at that time, and that the associated vma->vm_file stays valid while the vm_ops->fault() method is running. Note that this implies that speculative faults can not sleep within the vm_ops->fault method. We will only attempt to fetch existing pages from the page cache during speculative faults; any miss (or prefetch) will be handled by falling back to non-speculative fault handling. The speculative handling case also does not preallocate page tables, as it is always called with a pre-existing page table. Signed-off-by: Michel Lespinasse --- mm/memory.c | 63 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6eddd7b4e89c..7139004c624d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3709,29 +3709,50 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; - /* - * Preallocate pte before we take page_lock because this might lead to - * deadlocks for memcg reclaim which waits for pages under writeback: - * lock_page(A) - * SetPageWriteback(A) - * unlock_page(A) - * lock_page(B) - * lock_page(B) - * pte_alloc_one - * shrink_page_list - * wait_on_page_writeback(A) - * SetPageWriteback(B) - * unlock_page(B) - * # flush A, B to clear the writeback - */ - if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) { - vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); - if (!vmf->prealloc_pte) - return VM_FAULT_OOM; - smp_wmb(); /* See comment in __pte_alloc() */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + rcu_read_lock(); + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) { + ret = VM_FAULT_RETRY; + } else { + /* + * The mmap sequence count check guarantees that the + * vma we fetched at the start of the fault was still + * current at that point in time. The rcu read lock + * ensures vmf->vma->vm_file stays valid. + */ + ret = vma->vm_ops->fault(vmf); + } + rcu_read_unlock(); + } else +#endif + { + /* + * Preallocate pte before we take page_lock because + * this might lead to deadlocks for memcg reclaim + * which waits for pages under writeback: + * lock_page(A) + * SetPageWriteback(A) + * unlock_page(A) + * lock_page(B) + * lock_page(B) + * pte_alloc_one + * shrink_page_list + * wait_on_page_writeback(A) + * SetPageWriteback(B) + * unlock_page(B) + * # flush A, B to clear writeback + */ + if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) { + vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); + if (!vmf->prealloc_pte) + return VM_FAULT_OOM; + smp_wmb(); /* See comment in __pte_alloc() */ + } + + ret = vma->vm_ops->fault(vmf); } - ret = vma->vm_ops->fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY | VM_FAULT_DONE_COW))) return ret; From patchwork Wed Apr 7 01:44:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A53B1C433B4 for ; Wed, 7 Apr 2021 01:46:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 531C8610CE for ; Wed, 7 Apr 2021 01:46:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 531C8610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 83D706B0085; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D30928E0007; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E62A38E0009; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0156.hostedemail.com [216.40.44.156]) by kanga.kvack.org (Postfix) with ESMTP id 9B6658D0006 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 56F7F181AF5D7 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.33.A7CC970 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf12.hostedemail.com (Postfix) with ESMTP id 168FAE6 for ; Wed, 7 Apr 2021 01:45:04 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=hNw2E9n5catZ5ChciAAiyAEd51ldZQS2D3FL9TJJQ3Y=; b=LdeanG/cHGgwnR5X1yW6N3lnK/QBKoWHRD02KGL1fyNhPOTgq2puWO0/ibCPXZbYgb1lA acWdyb1vjbMAdEdAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=hNw2E9n5catZ5ChciAAiyAEd51ldZQS2D3FL9TJJQ3Y=; b=aaxklxxmwoZqKEa5hd21kEyumQSxeowwY9l01vrrM5iLdUOjf+YPBOX1buQKr4Tpo6iXn PjWEp27BcS9ggHt/rGwAdb43w/7zCOqGOt8OY2xxeogzRt0gQkbxjFyb1BtIwAd7Ymmf4L8 mR++NEo/XdU2b87VmLebX8/hkpDDaF/HgQOJlJGLPMP2mZB7xSPyMlbIEUetHzIizKXYYrw PfYM3zytRhEF2Vv+5tom1RbamQBi6ZH93HkYCrMNDa6FpyhASkN56Hqn8tuAz6cDL0esVO6 piKK+VlFmJ3tMvrOyIDDQ5K1q3Q5Y1sP6Mtk8rcehQ7J8KevBg7XCKXtHslQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 1F86516038C; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 10D2D19F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 25/37] mm: implement speculative handling in filemap_fault() Date: Tue, 6 Apr 2021 18:44:50 -0700 Message-Id: <20210407014502.24091-26-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 168FAE6 X-Stat-Signature: 9s8qbie3cnb9entnk3319s3ntiy9z5gy X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759904-968356 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Extend filemap_fault() to handle speculative faults. In the speculative case, we will only be fishing existing pages out of the page cache. The logic we use mirrors what is done in the non-speculative case, assuming that pages are found in the page cache, are up to date and not already locked, and that readahead is not necessary at this time. In all other cases, the fault is aborted to be handled non-speculatively. Signed-off-by: Michel Lespinasse --- mm/filemap.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 43700480d897..6e8505fe5df9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2851,7 +2851,9 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, * it in the page cache, and handles the special cases reasonably without * having a lot of duplicated code. * - * vma->vm_mm->mmap_lock must be held on entry. + * If FAULT_FLAG_SPECULATIVE is set, this function runs within an rcu + * read locked section and with mmap lock not held. + * Otherwise, vma->vm_mm->mmap_lock must be held on entry. * * If our return value has VM_FAULT_RETRY set, it's because the mmap_lock * may be dropped before doing I/O or by lock_page_maybe_drop_mmap(). @@ -2876,6 +2878,47 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) struct page *page; vm_fault_t ret = 0; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + page = find_get_page(mapping, offset); + if (unlikely(!page) || unlikely(PageReadahead(page))) + return VM_FAULT_RETRY; + + if (!trylock_page(page)) + return VM_FAULT_RETRY; + + if (unlikely(compound_head(page)->mapping != mapping)) + goto page_unlock; + VM_BUG_ON_PAGE(page_to_pgoff(page) != offset, page); + if (unlikely(!PageUptodate(page))) + goto page_unlock; + + max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); + if (unlikely(offset >= max_off)) + goto page_unlock; + + /* + * Update readahead mmap_miss statistic. + * + * Note that we are not sure if finish_fault() will + * manage to complete the transaction. If it fails, + * we'll come back to filemap_fault() non-speculative + * case which will update mmap_miss a second time. + * This is not ideal, we would prefer to guarantee the + * update will happen exactly once. + */ + if (!(vmf->vma->vm_flags & VM_RAND_READ) && ra->ra_pages) { + unsigned int mmap_miss = READ_ONCE(ra->mmap_miss); + if (mmap_miss) + WRITE_ONCE(ra->mmap_miss, --mmap_miss); + } + + vmf->page = page; + return VM_FAULT_LOCKED; +page_unlock: + unlock_page(page); + return VM_FAULT_RETRY; + } + max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(offset >= max_off)) return VM_FAULT_SIGBUS; From patchwork Wed Apr 7 01:44:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00597C433B4 for ; Wed, 7 Apr 2021 01:46:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A796361165 for ; Wed, 7 Apr 2021 01:46:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A796361165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F076D8E000A; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38FF68E000F; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D05F8E0010; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id C81D78D000B for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 844BE180ACEE4 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.38.4CE0377 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf18.hostedemail.com (Postfix) with ESMTP id 045312000254 for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JYD3Wgnyn5x/hzU8kfQiO6ZPV5GuIxiA+PXDzohUvGA=; b=zcbm6TYGSbg1fogm0a+YNIILcHGu1mT6vUKLDLx8xjYgGqbgnupaWpyFUw2L92FSmnjah aEpUwZWtViFUqIICw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=JYD3Wgnyn5x/hzU8kfQiO6ZPV5GuIxiA+PXDzohUvGA=; b=lMWJGAHOqeFns527xMlZ3Fyq6hnMk8R4Zi2s4J9BM4hIMv3x4lvNwIXSLvh4Re3SmRDIp QGdMUOLpTfyjweRiRr7ZfG6L/r96l8Q2zb9f40lLLvOEJjDy1dbzpNDwgBwHsux6qykTCmt XjTi6WRidOzS4QLMOOBd6GxKcUgtXKRBWI37pGuwdjraLXfHcYfFMJa4SA38fN+jOoYr5GQ klq7PozMEUef5/a5ixDM0l4FrhbuhaVk3X4jtOHqYrc8HtYOqWZJCPiDRK9waQs5f1YlVvU hH4vEg45qTroyA2tacBcSHTZIlQxGzBz9XwxiZT/OV1aW6mLHzBgv+owcBbQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 274A0160390; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 17FCA19F31E; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 26/37] mm: implement speculative fault handling in finish_fault() Date: Tue, 6 Apr 2021 18:44:51 -0700 Message-Id: <20210407014502.24091-27-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: az6ic6jxobapr7bs844ch3wazniarach X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 045312000254 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-6921 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the speculative case, we want to avoid direct pmd checks (which would require some extra synchronization to be safe), and rely on pte_map_lock which will both lock the page table and verify that the pmd has not changed from its initial value. Signed-off-by: Michel Lespinasse --- mm/memory.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7139004c624d..13e2aaf900e5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3915,23 +3915,25 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return ret; } - if (pmd_none(*vmf->pmd)) { - if (PageTransCompound(page)) { - ret = do_set_pmd(vmf, page); - if (ret != VM_FAULT_FALLBACK) - return ret; + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { + if (pmd_none(*vmf->pmd)) { + if (PageTransCompound(page)) { + ret = do_set_pmd(vmf, page); + if (ret != VM_FAULT_FALLBACK) + return ret; + } + + if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) + return VM_FAULT_OOM; } - if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) - return VM_FAULT_OOM; + /* See comment in __handle_mm_fault() */ + if (pmd_devmap_trans_unstable(vmf->pmd)) + return 0; } - /* See comment in __handle_mm_fault() */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return 0; - - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) + return VM_FAULT_RETRY; ret = 0; /* Re-check under ptl */ if (likely(pte_none(*vmf->pte))) From patchwork Wed Apr 7 01:44:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E458C433ED for ; Wed, 7 Apr 2021 01:45:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2F5C961396 for ; Wed, 7 Apr 2021 01:45:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F5C961396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 425DB8E0010; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 899276B0085; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7EA76B0083; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id B088C8D0008 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 7194A82499A8 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.17.C8C6250 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf04.hostedemail.com (Postfix) with ESMTP id D6399132 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=AZPl31a+Qwjg2yKK4/58wswJf9/r+P5+G7HgfFwgwPw=; b=myO1JlTLVwjj24PaAXDauPLhrNYZf5SriBcRFoxRq6uYNze2pldTo0iqgg/1h9Tk22dkA lmYyTY+Bn1u8AvOCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=AZPl31a+Qwjg2yKK4/58wswJf9/r+P5+G7HgfFwgwPw=; b=LpXTrEgOQVv1SNRQC0QiFxqtt5r3kW1xS0lKV129jQ3pGExTQLhVAfhaY4ophF2qd9GPM uGdaUjRHHgg/cP2jS10ipaOcP2chwNgqYP7lIJCCkO1TYdBA2Uj42Nzr3HGPQ98ywqLCNZU wzKEZVBj0iYDunZ850zmNvpx0p69hMJ3GQMHiSQzgUYmUNKFhKheqm4Cc2AIAHsKRBoODwU PpvRQPAzz4QgLLQvr5iXXjiYa4nZOwdnPUo77vKtDVwue3bzaanPHOHhoPE9n9Zmktm4Iez 1rE0/jqbi44fDIfdauryJOpQyygXVN/sHP49P9/8OIvq80NkuOX3/i4FFtlA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 301ED16046E; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 1F68A19F320; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 27/37] mm: implement speculative handling in do_fault_around() Date: Tue, 6 Apr 2021 18:44:52 -0700 Message-Id: <20210407014502.24091-28-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: cuc33mzb8ei68n9qbiohbxu4pt63b8ye X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D6399132 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-333205 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call the vm_ops->map_pages method within an rcu read locked section. In the speculative case, verify the mmap sequence lock at the start of the section. A match guarantees that the original vma is still valid at that time, and that the associated vma->vm_file stays valid while the vm_ops->map_pages() method is running. Do not test vmf->pmd in the speculative case - we only speculate when a page table already exists, and and this saves us from having to handle synchronization around the vmf->pmd read. Change xfs_filemap_map_pages() account for the fact that it can not block anymore, as it is now running within an rcu read lock. Signed-off-by: Michel Lespinasse --- fs/xfs/xfs_file.c | 3 +++ mm/memory.c | 22 ++++++++++++++++++++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index a007ca0711d9..b360732b20ae 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1387,6 +1387,9 @@ xfs_filemap_map_pages( struct inode *inode = file_inode(vmf->vma->vm_file); vm_fault_t ret; + if (!xfs_ilock_nowait(XFS_I(inode), XFS_MMAPLOCK_SHARED)) + return (vmf->flags & FAULT_FLAG_SPECULATIVE) ? + VM_FAULT_RETRY : 0; xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED); ret = filemap_map_pages(vmf, start_pgoff, end_pgoff); xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED); diff --git a/mm/memory.c b/mm/memory.c index 13e2aaf900e5..a20e13d84145 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4012,6 +4012,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) pgoff_t start_pgoff = vmf->pgoff; pgoff_t end_pgoff; int off; + vm_fault_t ret; nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK; @@ -4030,14 +4031,31 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) end_pgoff = min3(end_pgoff, vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1, start_pgoff + nr_pages - 1); - if (pmd_none(*vmf->pmd)) { + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && + pmd_none(*vmf->pmd)) { vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; smp_wmb(); /* See comment in __pte_alloc() */ } - return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); + rcu_read_lock(); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) { + rcu_read_unlock(); + return VM_FAULT_RETRY; + } + /* + * the mmap sequence check verified that vmf->vma was still + * current at that point in time. + * The rcu read lock ensures vmf->vma->vm_file stays valid. + */ + } +#endif + ret = vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); + rcu_read_unlock(); + return ret; } static vm_fault_t do_read_fault(struct vm_fault *vmf) From patchwork Wed Apr 7 01:44:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E96EC433ED for ; Wed, 7 Apr 2021 01:45:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF42A61165 for ; Wed, 7 Apr 2021 01:45:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF42A61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EB8158E0005; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58E418E0013; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 711056B0093; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id 8B8AC8E0001 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4E592181AEF3E for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.29.8C87004 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf02.hostedemail.com (Postfix) with ESMTP id F2A9840002C3 for ; Wed, 7 Apr 2021 01:44:59 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=T+2gjG7c+P1JGIgyykqnKaQiD2Wtt2L+tO+pRnvCt2JO1FKjXP26ZCM6hHIjsZg0F91Pv IUbkovCOPxRyi8dAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=XkD4f0ULJymIOIeAqVxcBhuCK1FNqNRxb6Ygkc1ApCs=; b=RTBwrQVEuGgTfXMpMfbrh/D5mOUGsMVm80V0u7rXaSdXDkPKYPpLKgLTiCRWVRop96qn6 0H69mSmJu7OkmjqPwNX7g+7lzVmBTBOnkqtihAaT+U/O2gxSb293SkAW7EIfEHbmGHMYZWk pOscXZjW7Z0CSgRkvL7L15AFZtvdGsJOlu2QZxqeLxSHccd/OsdnNeVdL56bmw0Rd1+uEX3 VukmTQUvytixFC1AIzrmU1IDhnsHFdOkwUHs5+5/3GbtMI8p+2GamYUAQkZO2nK072EtqZp iyjY45FT2zBP25QHQ+U8WOHV240vWPDjZQGCB3bXmbBMsDK2fiPRHyEjp86w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 35EA7160531; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 26A2B19F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 28/37] mm: implement speculative handling in filemap_map_pages() Date: Tue, 6 Apr 2021 18:44:53 -0700 Message-Id: <20210407014502.24091-29-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: F2A9840002C3 X-Stat-Signature: 5bbku3x36usu9kx8gjwksuyxbexpcgzo Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759899-375029 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the speculative case, we know the page table already exists, and it must be locked with pte_map_lock(). In the case where no page is found for the given address, return VM_FAULT_RETRY which will abort the fault before we get into the vm_ops->fault() callback. This is fine because if filemap_map_pages does not find the page in page cache, vm_ops->fault() will not either. Initialize addr and last_pgoff to correspond to the pte at the original fault address (which was mapped with pte_map_lock()), rather than the pte at start_pgoff. The choice of initial values doesn't matter as they will all be adjusted together before use, so they just need to be consistent with each other, and using the original fault address and pte allows us to reuse pte_map_lock() without any changes to it. Signed-off-by: Michel Lespinasse --- mm/filemap.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6e8505fe5df9..d496771749e6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3136,25 +3136,31 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, struct vm_area_struct *vma = vmf->vma; struct file *file = vma->vm_file; struct address_space *mapping = file->f_mapping; - pgoff_t last_pgoff = start_pgoff; + pgoff_t last_pgoff; unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); struct page *head, *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - vm_fault_t ret = 0; + vm_fault_t ret = (vmf->flags & FAULT_FLAG_SPECULATIVE) ? + VM_FAULT_RETRY : 0; - rcu_read_lock(); + /* filemap_map_pages() is called within an rcu read lock already. */ head = first_map_page(mapping, &xas, end_pgoff); if (!head) - goto out; + return ret; - if (filemap_map_pmd(vmf, head)) { - ret = VM_FAULT_NOPAGE; - goto out; + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && + filemap_map_pmd(vmf, head)) + return VM_FAULT_NOPAGE; + + if (!pte_map_lock(vmf)) { + unlock_page(head); + put_page(head); + return VM_FAULT_RETRY; } + addr = vmf->address; + last_pgoff = vmf->pgoff; - addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { page = find_subpage(head, xas.xa_index); if (PageHWPoison(page)) @@ -3184,8 +3190,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, put_page(head); } while ((head = next_map_page(mapping, &xas, end_pgoff)) != NULL); pte_unmap_unlock(vmf->pte, vmf->ptl); -out: - rcu_read_unlock(); + vmf->pte = NULL; WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret; } From patchwork Wed Apr 7 01:44:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF90EC433ED for ; Wed, 7 Apr 2021 01:46:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED72C613A7 for ; Wed, 7 Apr 2021 01:46:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED72C613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 403B18E0009; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B64B8E0017; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62F198E0014; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0079.hostedemail.com [216.40.44.79]) by kanga.kvack.org (Postfix) with ESMTP id BD93D8D000A for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 798DD81D8 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.19.B36291A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf30.hostedemail.com (Postfix) with ESMTP id 9EE7BE000113 for ; Wed, 7 Apr 2021 01:45:02 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=VhmmiWl9tNNsyaupBkysTD3lPGnrlR57ucyCgOq4BOY=; b=R8Xr3EA9fRzTQ5E7xJ3qTkjA4Fwi0FWi+flPvc3/ZT6JmH9ecOXHL6K3KPSeyNY0xgklD XN0oTjWBXkv5ndiDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=VhmmiWl9tNNsyaupBkysTD3lPGnrlR57ucyCgOq4BOY=; b=ehDI2OUmefmW+R59iI9a+gz+Lhtzi+v/Tw6YpiAyesjZ+F8w8BC+0MdbGiUlJxQE3JVB2 S/bICvdAoX/UzTViHaMGpYaHFbslTfPYlB/eUpSOGs3WUstZ9I/SO8ktT5Q9qKrgLXkzaWs NE0zT6JBjV/EmRjAXMMj2r0152f4F9ruuwOwGMUqBPwtohhiKGb2NRJIMlIA50ZKDw/yjYj tiC23Y4IEei9FBG+ZSz2PG4SY9SQdXAmzE9IuiQsciMJRCnxVO3k/TfawOvAtzCzHJVfYsP k7mCPPMT2uCiGlkhSASBWUk7K3HNi9R/opVjU+DbRN3FOdDL6eNhg6VddV8Q== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 3BCCA160555; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 2DD8119F31E; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 29/37] fs: list file types that support speculative faults. Date: Tue, 6 Apr 2021 18:44:54 -0700 Message-Id: <20210407014502.24091-30-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9EE7BE000113 X-Stat-Signature: o474hhkp35mat5xgp9u8o4ac8zm7nbf5 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759902-668950 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a speculative field to the vm_operations_struct, which indicates if the associated file type supports speculative faults. Initially this is set for files that implement fault() with filemap_fault(). Signed-off-by: Michel Lespinasse --- fs/btrfs/file.c | 1 + fs/cifs/file.c | 1 + fs/fuse/file.c | 1 + fs/nfs/file.c | 1 + fs/ubifs/file.c | 1 + fs/vboxsf/file.c | 1 + include/linux/mm.h | 7 +++++++ mm/filemap.c | 1 + 8 files changed, 14 insertions(+) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 0e155f013839..b31851271e51 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -2293,6 +2293,7 @@ static const struct vm_operations_struct btrfs_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = btrfs_page_mkwrite, + .speculative = true, }; static int btrfs_file_mmap(struct file *filp, struct vm_area_struct *vma) diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 042e24aad410..a0d5fbb25c62 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -4116,6 +4116,7 @@ static const struct vm_operations_struct cifs_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = cifs_page_mkwrite, + .speculative = true, }; int cifs_file_strict_mmap(struct file *file, struct vm_area_struct *vma) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 8cccecb55fb8..c4874240d157 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -2354,6 +2354,7 @@ static const struct vm_operations_struct fuse_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = fuse_page_mkwrite, + .speculative = true, }; static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma) diff --git a/fs/nfs/file.c b/fs/nfs/file.c index 16ad5050e046..e653c6bc23ca 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -588,6 +588,7 @@ static const struct vm_operations_struct nfs_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = nfs_vm_page_mkwrite, + .speculative = true, }; static int nfs_need_check_write(struct file *filp, struct inode *inode, diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c index 0e4b4be3aa26..3d97f1c3e9c7 100644 --- a/fs/ubifs/file.c +++ b/fs/ubifs/file.c @@ -1598,6 +1598,7 @@ static const struct vm_operations_struct ubifs_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = ubifs_vm_page_mkwrite, + .speculative = true, }; static int ubifs_file_mmap(struct file *file, struct vm_area_struct *vma) diff --git a/fs/vboxsf/file.c b/fs/vboxsf/file.c index c4ab5996d97a..e0a6a3af9cb9 100644 --- a/fs/vboxsf/file.c +++ b/fs/vboxsf/file.c @@ -146,6 +146,7 @@ static const struct vm_operations_struct vboxsf_file_vm_ops = { .close = vboxsf_vma_close, .fault = filemap_fault, .map_pages = filemap_map_pages, + .speculative = true, }; static int vboxsf_file_mmap(struct file *file, struct vm_area_struct *vma) diff --git a/include/linux/mm.h b/include/linux/mm.h index f26490aff514..b4c0c10e434e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -641,6 +641,13 @@ struct vm_operations_struct { */ struct page *(*find_special_page)(struct vm_area_struct *vma, unsigned long addr); + /* + * speculative indicates that the vm_operations support + * speculative page faults. This allows ->fault and ->map_pages + * to be called with FAULT_FLAG_SPECULATIVE set; such calls will + * run within an rcu read locked section and with mmap lock not held. + */ + bool speculative; }; static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) diff --git a/mm/filemap.c b/mm/filemap.c index d496771749e6..b83040c672d3 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3226,6 +3226,7 @@ const struct vm_operations_struct generic_file_vm_ops = { .fault = filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = filemap_page_mkwrite, + .speculative = true, }; /* This is used for a general mmap of a disk file */ From patchwork Wed Apr 7 01:44:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2256C433B4 for ; Wed, 7 Apr 2021 01:46:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8B1FA61396 for ; Wed, 7 Apr 2021 01:46:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B1FA61396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AAA1C6B008A; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E15458E000C; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05A168E000A; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id AA5418D0007 for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6B4ED62E8 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.29.E083B63 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf03.hostedemail.com (Postfix) with ESMTP id D8E4DC0007CB for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=TsUjPBR98OR+HycMa67lX3AwybmOXiXbr14JHEP/HPI=; b=NoQsbiuzomfmwqnnFDorqLDd23clwVI/I9qt180/mo9M7RjoPMhjRhfpsYGAlpOxxDY55 QM3jB6ilptjwQwUAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=TsUjPBR98OR+HycMa67lX3AwybmOXiXbr14JHEP/HPI=; b=LCojAYHaLAo69slIhG0Zb2BOh6dMlEDVhhPB5aAHC2mq/JDWdr89S0Iru7tPJUmymI0ag h+RSJdTiN2E0VTSuQl97AgZdurjW6CCUju1Ei52zI+Qp/QLBkOKNJ/46scl+D5SDytGyBYl JSFUmit6FYMMvWBl7amZIS9ZFJLo3Yq6r2QBg4gEkurs3Zx3mpA4ZAUe4xXkuW62SkQRTUt doMkKI3WAAlOQIOnCf9qKEBTEKjfmXa+3kBm2EBnVccrdNYvQXdOo4pOvN6tK3dTTLNzxN7 N+LgdPNmKnA7zdFsNgB0keq59CzhVyg0nm5VtBtSJsPguhY1mrda+pumYopg== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 449EB160564; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 34C4C19F31F; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 30/37] mm: enable speculative fault handling for supported file types. Date: Tue, 6 Apr 2021 18:44:55 -0700 Message-Id: <20210407014502.24091-31-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Stat-Signature: 1ptbumw7uoanhh6srwpc81hpfjirq648 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8E4DC0007CB Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-876356 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce vma_can_speculate(), which allows speculative handling for VMAs mapping supported file types. From do_handle_mm_fault(), speculative handling will follow through __handle_mm_fault(), handle_pte_fault() and do_fault(). At this point, we expect speculative faults to continue through one of: - do_read_fault(), fully implemented; - do_cow_fault(), which might abort if missing anon vmas, - do_shared_fault(), not implemented yet (would require ->page_mkwrite() changes). vma_can_speculate() provides an early abort for the do_shared_fault() case, limiting the time spent on trying that unimplemented case. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 3 ++- include/linux/mm.h | 14 ++++++++++++++ mm/memory.c | 17 ++++++++++++----- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fbf265f56a06..48b86911a6df 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1324,7 +1324,8 @@ void do_user_addr_fault(struct pt_regs *regs, goto spf_abort; rcu_read_lock(); vma = find_vma(mm, address); - if (!vma || vma->vm_start > address || !vma_is_anonymous(vma)) { + if (!vma || vma->vm_start > address || + !vma_can_speculate(vma, flags)) { rcu_read_unlock(); goto spf_abort; } diff --git a/include/linux/mm.h b/include/linux/mm.h index b4c0c10e434e..edb809e9036b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -700,6 +700,20 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) return vma->vm_flags & VM_ACCESS_FLAGS; } +static inline bool vma_can_speculate(struct vm_area_struct *vma, + unsigned int flags) +{ + if (vma_is_anonymous(vma)) + return true; + if (!vma->vm_ops->speculative) + return false; + if (!(flags & FAULT_FLAG_WRITE)) + return true; + if (!(vma->vm_flags & VM_SHARED)) + return true; + return false; +} + #ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow diff --git a/mm/memory.c b/mm/memory.c index a20e13d84145..074945faf1ab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4090,8 +4090,12 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; - if (unlikely(anon_vma_prepare(vma))) - return VM_FAULT_OOM; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + } vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!vmf->cow_page) @@ -4128,6 +4132,8 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret, tmp; + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; @@ -4172,12 +4178,12 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; - VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); - /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ if (!vma->vm_ops->fault) { + VM_BUG_ON(vmf->flags & FAULT_FLAG_SPECULATIVE); + /* * If we find a migration pmd entry or a none pmd entry, which * should never happen, return SIGBUS @@ -4739,7 +4745,8 @@ vm_fault_t do_handle_mm_fault(struct vm_area_struct *vma, { vm_fault_t ret; - VM_BUG_ON((flags & FAULT_FLAG_SPECULATIVE) && !vma_is_anonymous(vma)); + VM_BUG_ON((flags & FAULT_FLAG_SPECULATIVE) && + !vma_can_speculate(vma, flags)); __set_current_state(TASK_RUNNING); From patchwork Wed Apr 7 01:44:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1978C43460 for ; Wed, 7 Apr 2021 01:46:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7EF5F61165 for ; Wed, 7 Apr 2021 01:46:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7EF5F61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2450F8E0008; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5641B8E0013; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42D988E0012; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id C9FAE8D000C for ; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8C2C86C2F for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.20.BF8C7A0 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf17.hostedemail.com (Postfix) with ESMTP id 824A840002CF for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=2b40fHZOymmhGJxI0G3GGByTYESRQj8MqQXnbVkgf4k=; b=rXM4NzWSzWcgaoFSX84ZUtXtVSmb0ie4Dv/LeHHqXZgS+oSmJL5ZAZWqJxRFBNKpvDTpg /KFSj6xdLkApTLYDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=2b40fHZOymmhGJxI0G3GGByTYESRQj8MqQXnbVkgf4k=; b=E6lEMweRaJ8ikAuN3ggqOm/+deuwAvAe4/+tRAmlHrlqKXySUlRlCu4mnDsXMira74kqQ 8QZkmrF5S2O6hHxn043dWyoXl1FkgpudzojALP7Fr/J+FQsR/a1Bq9ZafuU6i9afr9EhOXC Q/BIDlLVeje/Sd6UVGnP17Brbqh1c6PHPBbTxiWFAVICNivtw7Nj1qfKu4aEtzkKi3wWNX3 eyfrL1t+Ac82SzSaY0g93MQ43iLQLa2ruFPUKGXDb1s81im1za7WVMUbqGQ4JJp2HjRGcJD xdYooB7fsmS6ZvdnMWfRTCRi226A6A8mLMHOLViq2SWrn32gTzX5x84wB82Q== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 4B721160565; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 3B94A19F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 31/37] ext4: implement speculative fault handling Date: Tue, 6 Apr 2021 18:44:56 -0700 Message-Id: <20210407014502.24091-32-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 824A840002CF X-Stat-Signature: eesmsg1mconkr616i4g3c9erg44i7b96 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-156649 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We just need to make sure ext4_filemap_fault() doesn't block in the speculative case as it is called with an rcu read lock held. Signed-off-by: Michel Lespinasse --- fs/ext4/file.c | 1 + fs/ext4/inode.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 194f5d00fa32..546b9d4aa9d7 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -742,6 +742,7 @@ static const struct vm_operations_struct ext4_file_vm_ops = { .fault = ext4_filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = ext4_page_mkwrite, + .speculative = true, }; static int ext4_file_mmap(struct file *file, struct vm_area_struct *vma) diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 0948a43f1b3d..370484403c71 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -6192,7 +6192,12 @@ vm_fault_t ext4_filemap_fault(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); vm_fault_t ret; - down_read(&EXT4_I(inode)->i_mmap_sem); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + if (!down_read_trylock(&EXT4_I(inode)->i_mmap_sem)) + return VM_FAULT_RETRY; + } else { + down_read(&EXT4_I(inode)->i_mmap_sem); + } ret = filemap_fault(vmf); up_read(&EXT4_I(inode)->i_mmap_sem); From patchwork Wed Apr 7 01:44:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77F7CC433B4 for ; Wed, 7 Apr 2021 01:46:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 25A20610CE for ; Wed, 7 Apr 2021 01:46:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 25A20610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7B9E28E0017; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C48238E0007; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 822696B0087; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id 2A9746B008A for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E8C8B62E8 for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.34.3BB6C31 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf15.hostedemail.com (Postfix) with ESMTP id AAFACA00038D for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=ju1TjADkBaEpCEmd9Uqz5VcWNno1/6nZtl53IX9hdVk=; b=/HgvOzN+4gZlXg2dAh/dfK0OJzmp3nnbAfpOyKVZoxCdy/rO8VsbHIDa7Q1uVsUm7LtGD bvH0KxRDERX19z8BQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=ju1TjADkBaEpCEmd9Uqz5VcWNno1/6nZtl53IX9hdVk=; b=Rcx14jUFCUpobMRsi8OfvXrqjWZOos2yAkHdBv79ZVxOdRacMFpS8AsU3XbW3uHcixZB0 /TCi7hSxf0Z6Nynm7jE5MjAn5NeNb1Gn/QZTRkp0ETZkiK1cXV4QS0m13vSMNRHM11eblxb fSh42ACDbmP4crGwYQmmZGl0TyO87HVPSbgMucCnANko0TsXp7XbIRZ3sNlx2q7DSpjk7It 9dUw6OVWwxnTC5OfRNzODM+M75isQYJRrL/a1oFedkln0KPu80bmhZro68EOOo0NyCF3V+t PBGZA0rAdIUXBGvMRZAqO246iJzXzfXJhxoSZCEnQRoaI+mhig14LMymHUZQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 5048116064D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 42E6D19F31E; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 32/37] f2fs: implement speculative fault handling Date: Tue, 6 Apr 2021 18:44:57 -0700 Message-Id: <20210407014502.24091-33-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: AAFACA00038D X-Stat-Signature: itofyocypxi9fe3i6ofpoo7q4niduw3t X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-434049 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We just need to make sure f2fs_filemap_fault() doesn't block in the speculative case as it is called with an rcu read lock held. Signed-off-by: Michel Lespinasse --- fs/f2fs/file.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index d26ff2ae3f5e..c1cfdc3ec98e 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -37,7 +37,12 @@ static vm_fault_t f2fs_filemap_fault(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); vm_fault_t ret; - down_read(&F2FS_I(inode)->i_mmap_sem); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + if (!down_read_trylock(&F2FS_I(inode)->i_mmap_sem)) + return VM_FAULT_RETRY; + } else { + down_read(&F2FS_I(inode)->i_mmap_sem); + } ret = filemap_fault(vmf); up_read(&F2FS_I(inode)->i_mmap_sem); @@ -171,6 +176,7 @@ static const struct vm_operations_struct f2fs_file_vm_ops = { .fault = f2fs_filemap_fault, .map_pages = filemap_map_pages, .page_mkwrite = f2fs_vm_page_mkwrite, + .speculative = true, }; static int get_parent_ino(struct inode *inode, nid_t *pino) From patchwork Wed Apr 7 01:44:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72973C433B4 for ; Wed, 7 Apr 2021 01:46:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1AC62613A7 for ; Wed, 7 Apr 2021 01:46:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AC62613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 60FDB8E0001; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A25CA6B0092; Tue, 6 Apr 2021 21:45:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7145A8E0016; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 1DA816B0087 for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D1013181AF5FD for ; Wed, 7 Apr 2021 01:45:08 +0000 (UTC) X-FDA: 78003877896.33.D3EC076 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf10.hostedemail.com (Postfix) with ESMTP id 856C240002C0 for ; Wed, 7 Apr 2021 01:45:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=dxj+e2mCTN/mq8/QKyjl2SahvK7RHudQGhkISRnBVig=; b=+Q3LDhDV38Tl/5VVpqEc6k0vUGZdg8aC5vuZTmv9XW2HYt5SSVn8xVrXOLUhh39OTr4Yk BKC/hKD7DF5025fBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=dxj+e2mCTN/mq8/QKyjl2SahvK7RHudQGhkISRnBVig=; b=JZCFDV30/yXf6H5J+f3SdyNh3hYLqYl3vEW/Ni6crQkjRh6o64d2cKCZQ12mIdV86AgjG 0xb5dvncR9Q3KIcUxaTww6pmfH1dLtaCq1+ywsEV/VkJRKDmMqq2UPyWsGYMvmOB2h81Mps ZWcf0+Bkuo1o4Aul81AclAy77gUiuOhdyIRW9wyIF6E+BzIZARbKoxx0xNxmxbu11pF0uPI HpgNlVPPyxwfH5rbkK60cLgd+29R2XTs+4kyo2nRLVJ4eIM7XcMOQenuTzpR9GikW/t7IEy h18AfoPFjs2trfBmdlGX++FxPVo1dYiS2yW0X60gIkpTzML88X/k546hS3/A== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 59267160650; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 4A03A19F31F; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 33/37] mm: enable speculative fault handling only for multithreaded user space Date: Tue, 6 Apr 2021 18:44:58 -0700 Message-Id: <20210407014502.24091-34-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 856C240002C0 X-Stat-Signature: ftm63kxezwhmx1sjd43osq1oeey65d68 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759905-732151 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Performance tuning: single threaded userspace does not benefit from speculative page faults, so we turn them off to avoid any related (small) extra overheads. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 48b86911a6df..b1a07ca82d59 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1318,6 +1318,10 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif + /* Only try spf for multithreaded user space faults. */ + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + count_vm_event(SPF_ATTEMPT); seq = mmap_seq_read_start(mm); if (seq & 1) @@ -1351,6 +1355,7 @@ void do_user_addr_fault(struct pt_regs *regs, spf_abort: count_vm_event(SPF_ABORT); +no_spf: /* * Kernel-mode access to the user address space should only occur From patchwork Wed Apr 7 01:44:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90069C433ED for ; Wed, 7 Apr 2021 01:51:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1D793613E1 for ; Wed, 7 Apr 2021 01:51:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1D793613E1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F3B46B0088; Tue, 6 Apr 2021 21:51:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A3CF6B0089; Tue, 6 Apr 2021 21:51:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 644266B0095; Tue, 6 Apr 2021 21:51:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id 48C3C6B0088 for ; Tue, 6 Apr 2021 21:51:42 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 01459758E for ; Wed, 7 Apr 2021 01:51:41 +0000 (UTC) X-FDA: 78003894444.16.58DE5F5 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf03.hostedemail.com (Postfix) with ESMTP id 2BFB9C0001F7 for ; Wed, 7 Apr 2021 01:51:39 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=lXqSL1NodLGVhPgvf3TxvfQ3Pft8FvuZOwvv5Jh0BNQ=; b=FlfakSxMf6XS9pz7xRsk0OujrO1Y78RHmmo/v92A3WTWxqHlRuojP7dpuZjf1jcdjqeSc qyqJph9vO91y4NKAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=lXqSL1NodLGVhPgvf3TxvfQ3Pft8FvuZOwvv5Jh0BNQ=; b=SyzMVd/oPjXcTxIr1A09NN6Sy6JN5EtqFXUb6m3buvczgQ6z+XTzaVJKtKnllmdM9j5M1 aPxK0kiyf3ZhqUYkO11qmeybD7YLnpvV96GI0jWIp+EACru/Mg/MeH1B9aVU7sSnNDnBc+k KoSW+sD8JIMrPNMvgDvYJdhyF9IGT+Q3LUMAur5iLxaufNy68Vfa4Fdz60mILR1CZPiQCx8 xz+MpCbQG9r9eqzGhlfg4dDWaS6vAl+sR4VEDtucW39saGCi7PdpfSX1/j6bqPaeIlzSpGE FUNB94XMo5K84qpqdA35DgxBaqyZrZAXjtA5jgO+3sQSjH61yDozMsxOp32w== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 604D316068A; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 513F419F31D; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 34/37] mm: rcu safe vma freeing only for multithreaded user space Date: Tue, 6 Apr 2021 18:44:59 -0700 Message-Id: <20210407014502.24091-35-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2BFB9C0001F7 X-Stat-Signature: zw7tx7c3t5tcajynp7y6hsrkf8571scb Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617760299-877276 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Performance tuning: as single threaded userspace does not use speculative page faults, it does not require rcu safe vma freeing. Turn this off to avoid the related (small) extra overheads. For multi threaded userspace, we often see a performance benefit from the rcu safe vma freeing - even in tests that do not have any frequent concurrent page faults ! This is because rcu safe vma freeing prevents recently released vmas from being immediately reused in a new thread. Signed-off-by: Michel Lespinasse --- kernel/fork.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 2f20a5c5fed8..623875e8e742 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -389,10 +389,12 @@ static void __vm_area_free(struct rcu_head *head) void vm_area_free(struct vm_area_struct *vma) { #ifdef CONFIG_SPECULATIVE_PAGE_FAULT - call_rcu(&vma->vm_rcu, __vm_area_free); -#else - ____vm_area_free(vma); + if (atomic_read(&vma->vm_mm->mm_users) > 1) { + call_rcu(&vma->vm_rcu, __vm_area_free); + return; + } #endif + ____vm_area_free(vma); } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Wed Apr 7 01:45:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D441DC433ED for ; Wed, 7 Apr 2021 01:46:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6E8FB610CE for ; Wed, 7 Apr 2021 01:46:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E8FB610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B33C78E0014; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19B408E0016; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B07AB8E0008; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0214.hostedemail.com [216.40.44.214]) by kanga.kvack.org (Postfix) with ESMTP id 924B48E0006 for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 61432180ACF8F for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) X-FDA: 78003877938.28.B5CDBA2 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf13.hostedemail.com (Postfix) with ESMTP id D7243E00010B for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=iBh5A+5wTMstGnPqgpP3ljuSrzHKXXv6cU8okVcPLeE=; b=XsGhiULxRA9nCIbrMh7+FdFhtJ/TpkWSJjnJ2m5rjzIiXhSUWPXJDpg+7W2RDy7HlKcpX dSpTVJSFqHW5l9kCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=iBh5A+5wTMstGnPqgpP3ljuSrzHKXXv6cU8okVcPLeE=; b=o55KlhOAIFr9rk/nOBWWIlXE7tCEMPE/X3Y1YYo/ocQwXKYNzIVBPbOREyxDJzHGy4pEX 6/vMoRjvWqD4SD7ls+IAFbm7ffBwNmB90h4kQOezDNFoU602wt41yIcjnjGVm/KhqPmvZy7 L7f8WXO+Z4pBJQ+o3Uqo9M9MCw+xdh1a2rCt8cNx/acAACjhob3xWJ3Cx6cHkl2C8Dbni0F HVYOioO3Auv2k5W1tUqmRIQW0OZd+cnj1JXmRDNZmSeirKSTEwFJyGH3bBQ4KPAyObPwAMd dgTMvQcLEoyK7RsnAgmkgiqqafsPta6wkwadZhZZZor4JyGkHDIpK1KVZ6vA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 66D09160698; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 58AE819F31E; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 35/37] mm: spf statistics Date: Tue, 6 Apr 2021 18:45:00 -0700 Message-Id: <20210407014502.24091-36-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D7243E00010B X-Stat-Signature: 96i3373mmcodhe1rqnkbnztqs96kyu5y Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-243732 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_SPECULATIVE_PAGE_FAULT_STATS config option, and dump extra statistics about executed spf cases and abort reasons when the option is set. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 19 +++++++--- include/linux/mmap_lock.h | 19 +++++++++- include/linux/vm_event_item.h | 24 ++++++++++++ include/linux/vmstat.h | 6 +++ mm/Kconfig.debug | 7 ++++ mm/memory.c | 71 ++++++++++++++++++++++++++++------- mm/vmstat.c | 24 ++++++++++++ 7 files changed, 149 insertions(+), 21 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index b1a07ca82d59..e210bbcb8bc5 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1324,22 +1324,31 @@ void do_user_addr_fault(struct pt_regs *regs, count_vm_event(SPF_ATTEMPT); seq = mmap_seq_read_start(mm); - if (seq & 1) + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); goto spf_abort; + } rcu_read_lock(); vma = find_vma(mm, address); - if (!vma || vma->vm_start > address || - !vma_can_speculate(vma, flags)) { + if (!vma || vma->vm_start > address) { rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_can_speculate(vma, flags)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); goto spf_abort; } pvma = *vma; rcu_read_unlock(); - if (!mmap_seq_read_check(mm, seq)) + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) goto spf_abort; vma = &pvma; - if (unlikely(access_error(error_code, vma))) + if (unlikely(access_error(error_code, vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); goto spf_abort; + } fault = do_handle_mm_fault(vma, address, flags | FAULT_FLAG_SPECULATIVE, seq, regs); diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8f4eca2d0f43..98f24a9910a9 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -7,6 +7,7 @@ #include #include #include +#include #ifdef CONFIG_SPECULATIVE_PAGE_FAULT #define MMAP_LOCK_SEQ_INITIALIZER(name) \ @@ -104,12 +105,26 @@ static inline unsigned long mmap_seq_read_start(struct mm_struct *mm) return seq; } -static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq) +static inline bool __mmap_seq_read_check(struct mm_struct *mm, + unsigned long seq) { smp_rmb(); return seq == READ_ONCE(mm->mmap_seq); } -#endif + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS +static inline bool mmap_seq_read_check(struct mm_struct *mm, unsigned long seq, + enum vm_event_item fail_event) +{ + if (__mmap_seq_read_check(mm, seq)) + return true; + count_vm_event(fail_event); + return false; +} +#else +#define mmap_seq_read_check(mm, seq, fail) __mmap_seq_read_check(mm, seq) +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT_STATS */ +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ static inline void mmap_write_lock(struct mm_struct *mm) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index cc4f8d14e43f..6d25fd9ce4d1 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -124,6 +124,30 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SPECULATIVE_PAGE_FAULT SPF_ATTEMPT, SPF_ABORT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS + SPF_ABORT_ODD, + SPF_ABORT_UNMAPPED, + SPF_ABORT_NO_SPECULATE, + SPF_ABORT_VMA_COPY, + SPF_ABORT_ACCESS_ERROR, + SPF_ABORT_PUD, + SPF_ABORT_PMD, + SPF_ABORT_ANON_VMA, + SPF_ABORT_PTE_MAP_LOCK_SEQ1, + SPF_ABORT_PTE_MAP_LOCK_PMD, + SPF_ABORT_PTE_MAP_LOCK_PTL, + SPF_ABORT_PTE_MAP_LOCK_SEQ2, + SPF_ABORT_USERFAULTFD, + SPF_ABORT_FAULT, + SPF_ABORT_NON_SWAP_ENTRY, + SPF_ABORT_SWAP_NOPAGE, + SPF_ATTEMPT_ANON, + SPF_ATTEMPT_FILE, + SPF_ATTEMPT_SWAP, + SPF_ATTEMPT_NUMA, + SPF_ATTEMPT_PTE, + SPF_ATTEMPT_WP, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 506d625163a1..34e05604a93f 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -109,6 +109,12 @@ static inline void vm_events_fold_cpu(int cpu) #endif /* CONFIG_VM_EVENT_COUNTERS */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS +#define count_vm_spf_event(x) count_vm_event(x) +#else +#define count_vm_spf_event(x) do {} while (0) +#endif + #ifdef CONFIG_NUMA_BALANCING #define count_vm_numa_event(x) count_vm_event(x) #define count_vm_numa_events(x, y) count_vm_events(x, y) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 1e73717802f8..6be8ca7950ee 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -150,3 +150,10 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + +config SPECULATIVE_PAGE_FAULT_STATS + bool "Additional statistics for speculative page faults" + depends on SPECULATIVE_PAGE_FAULT + help + Additional statistics for speculative page faults. + If in doubt, say N. diff --git a/mm/memory.c b/mm/memory.c index 074945faf1ab..6165d340e134 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2585,7 +2585,7 @@ bool __pte_map_lock(struct vm_fault *vmf) } local_irq_disable(); - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, SPF_ABORT_PTE_MAP_LOCK_SEQ1)) goto fail; /* * The mmap sequence count check guarantees that the page @@ -2599,8 +2599,10 @@ bool __pte_map_lock(struct vm_fault *vmf) * is not a huge collapse operation in progress in our back. */ pmdval = READ_ONCE(*vmf->pmd); - if (!pmd_same(pmdval, vmf->orig_pmd)) + if (!pmd_same(pmdval, vmf->orig_pmd)) { + count_vm_spf_event(SPF_ABORT_PTE_MAP_LOCK_PMD); goto fail; + } #endif ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); if (!pte) @@ -2617,9 +2619,11 @@ bool __pte_map_lock(struct vm_fault *vmf) * We also don't want to retry until spin_trylock() succeeds, * because of the starvation potential against a stream of lockers. */ - if (unlikely(!spin_trylock(ptl))) + if (unlikely(!spin_trylock(ptl))) { + count_vm_spf_event(SPF_ABORT_PTE_MAP_LOCK_PTL); goto fail; - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) + } + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, SPF_ABORT_PTE_MAP_LOCK_SEQ2)) goto unlock_fail; local_irq_enable(); vmf->pte = pte; @@ -2891,6 +2895,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) if (unlikely(!vma->anon_vma)) { if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_ANON_VMA); ret = VM_FAULT_RETRY; goto out; } @@ -3153,10 +3158,15 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_WP); + if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_USERFAULTFD); return VM_FAULT_RETRY; + } return handle_userfault(vmf, VM_UFFD_WP); } @@ -3340,6 +3350,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_SWAP); + #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { /* @@ -3366,6 +3379,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_NON_SWAP_ENTRY); ret = VM_FAULT_RETRY; } else if (is_migration_entry(entry)) { migration_entry_wait(vma->vm_mm, vmf->pmd, @@ -3392,6 +3406,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (vmf->flags & FAULT_FLAG_SPECULATIVE) { delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + count_vm_spf_event(SPF_ABORT_SWAP_NOPAGE); return VM_FAULT_RETRY; } @@ -3598,6 +3613,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vm_fault_t ret = 0; pte_t entry; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_ANON); + /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) return VM_FAULT_SIGBUS; @@ -3627,8 +3645,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) } else { /* Allocate our own private page. */ if (unlikely(!vma->anon_vma)) { - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_ANON_VMA); return VM_FAULT_RETRY; + } if (__anon_vma_prepare(vma)) goto oom; } @@ -3670,8 +3690,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); if (page) put_page(page); - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_USERFAULTFD); return VM_FAULT_RETRY; + } return handle_userfault(vmf, VM_UFFD_MISSING); } @@ -3712,7 +3734,8 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) #ifdef CONFIG_SPECULATIVE_PAGE_FAULT if (vmf->flags & FAULT_FLAG_SPECULATIVE) { rcu_read_lock(); - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) { + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, + SPF_ABORT_FAULT)) { ret = VM_FAULT_RETRY; } else { /* @@ -4042,7 +4065,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) rcu_read_lock(); #ifdef CONFIG_SPECULATIVE_PAGE_FAULT if (vmf->flags & FAULT_FLAG_SPECULATIVE) { - if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq)) { + if (!mmap_seq_read_check(vmf->vma->vm_mm, vmf->seq, + SPF_ABORT_FAULT)) { rcu_read_unlock(); return VM_FAULT_RETRY; } @@ -4091,8 +4115,10 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) vm_fault_t ret; if (unlikely(!vma->anon_vma)) { - if (vmf->flags & FAULT_FLAG_SPECULATIVE) + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + count_vm_spf_event(SPF_ABORT_ANON_VMA); return VM_FAULT_RETRY; + } if (__anon_vma_prepare(vma)) return VM_FAULT_OOM; } @@ -4178,6 +4204,9 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_FILE); + /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ @@ -4251,6 +4280,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) bool was_writable = pte_savedwrite(vmf->orig_pte); int flags = 0; + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_NUMA); + /* * The "pte" at this point cannot be used safely without * validation through pte_unmap_same(). It's of NUMA type but @@ -4423,6 +4455,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + count_vm_spf_event(SPF_ATTEMPT_PTE); + if (!pte_spinlock(vmf)) return VM_FAULT_RETRY; entry = vmf->orig_pte; @@ -4490,20 +4525,26 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, local_irq_disable(); pgd = pgd_offset(mm, address); pgdval = READ_ONCE(*pgd); - if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) + if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } p4d = p4d_offset(pgd, address); p4dval = READ_ONCE(*p4d); - if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } vmf.pud = pud_offset(p4d, address); pudval = READ_ONCE(*vmf.pud); if (pud_none(pudval) || unlikely(pud_bad(pudval)) || unlikely(pud_trans_huge(pudval)) || - unlikely(pud_devmap(pudval))) + unlikely(pud_devmap(pudval))) { + count_vm_spf_event(SPF_ABORT_PUD); goto spf_fail; + } vmf.pmd = pmd_offset(vmf.pud, address); vmf.orig_pmd = READ_ONCE(*vmf.pmd); @@ -4521,8 +4562,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (unlikely(pmd_none(vmf.orig_pmd) || is_swap_pmd(vmf.orig_pmd) || pmd_trans_huge(vmf.orig_pmd) || - pmd_devmap(vmf.orig_pmd))) + pmd_devmap(vmf.orig_pmd))) { + count_vm_spf_event(SPF_ABORT_PMD); goto spf_fail; + } /* * The above does not allocate/instantiate page-tables because diff --git a/mm/vmstat.c b/mm/vmstat.c index 9ae1c27a549e..ac4ff4343a49 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1369,6 +1369,30 @@ const char * const vmstat_text[] = { "spf_attempt", "spf_abort", #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT_STATS + "SPF_ABORT_ODD", + "SPF_ABORT_UNMAPPED", + "SPF_ABORT_NO_SPECULATE", + "SPF_ABORT_VMA_COPY", + "SPF_ABORT_ACCESS_ERROR", + "SPF_ABORT_PUD", + "SPF_ABORT_PMD", + "SPF_ABORT_ANON_VMA", + "SPF_ABORT_PTE_MAP_LOCK_SEQ1", + "SPF_ABORT_PTE_MAP_LOCK_PMD", + "SPF_ABORT_PTE_MAP_LOCK_PTL", + "SPF_ABORT_PTE_MAP_LOCK_SEQ2", + "SPF_ABORT_USERFAULTFD", + "SPF_ABORT_FAULT", + "SPF_ABORT_NON_SWAP_ENTRY", + "SPF_ABORT_SWAP_NOPAGE", + "SPF_ATTEMPT_ANON", + "SPF_ATTEMPT_FILE", + "SPF_ATTEMPT_SWAP", + "SPF_ATTEMPT_NUMA", + "SPF_ATTEMPT_PTE", + "SPF_ATTEMPT_WP", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Wed Apr 7 01:45:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63B49C433B4 for ; Wed, 7 Apr 2021 01:46:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0DACE61165 for ; Wed, 7 Apr 2021 01:46:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DACE61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F2B6F8E000F; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DBC38E0013; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB36C8E0001; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 8D50E8E0001 for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 511FE82499A8 for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) X-FDA: 78003877938.03.615AA26 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf20.hostedemail.com (Postfix) with ESMTP id 86AFAD6 for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=i4NbI3unpGfKmh/q3AwoGzRSmZqSP94rN8RwRpY2aUw=; b=ogyMJqgSLmExCrvp3/r1Eg9S61h33kAnPRVDh2Te/XblEDKT6dg06kOiBaxK2gikeeF52 N4XMq6tUs84jd0lAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=i4NbI3unpGfKmh/q3AwoGzRSmZqSP94rN8RwRpY2aUw=; b=jtbAFBUFlPRu6wvR0Ewud3YJqowLRXFk0qeM/CUxucqhxAT2GZCHQcPFJ4JSBbET3P/eB bQ95KETLzufZHm9mva1B1rEMc/8fxlllZeEp5NoY10XqNIH067D7NdQzPPj8G2tqSd61UoF dYw5UPH8/M2tbkXH8aoeX60sT2OuFImrT2/xmDo3h1dvqmCqXvARJg6ucmU79zLUwhqmrsp f1sT5VlqvUq71oCCVfooJ6kEpqH/FMxYTdpGIZtzdPShke2xpBOqTXZoWDkPSqhbykxMTuv GQrAg3VK7g8e2N4vMVzh03cWBHaLqpqJeUIDnUlY583azbD0peIQLRu1UGgQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 6E930160699; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 6021519F31F; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 36/37] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Tue, 6 Apr 2021 18:45:01 -0700 Message-Id: <20210407014502.24091-37-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 86AFAD6 X-Stat-Signature: bo41e7jdwpuoazgorxdp891wacggybp9 X-Rspamd-Server: rspam02 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-438131 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT so that the speculative fault handling code can be compiled on this architecture. Signed-off-by: Michel Lespinasse --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e4e1b6550115..6c2bbfe3fd38 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -208,6 +208,7 @@ config ARM64 select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT help ARM 64-bit (AArch64) Linux support. From patchwork Wed Apr 7 01:45:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AC84C433B4 for ; Wed, 7 Apr 2021 01:46:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE6D361165 for ; Wed, 7 Apr 2021 01:46:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE6D361165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D31FD8E0016; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BCE08E000F; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCD368E0009; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id A19BD8E000C for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6436B4408 for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) X-FDA: 78003877938.36.5439008 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id 7BAAA600010B for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1pcgL6msPKrfTRZ1fn8ZYvCmKEa5WSF8jkX3M1HqC8Q=; b=hQvw4qVAk9RjFRT0YurEoXBUfSyibVWfDTUoCsMeXQbkWLv85cnNBgYIduwwAJSYkk9nX dBueIiIj6nFh/usBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1pcgL6msPKrfTRZ1fn8ZYvCmKEa5WSF8jkX3M1HqC8Q=; b=iNI/FbYx7Xyn86mRVSOET3lACzR7zhpyLEsLKAwjwO+52vhn4w0U0kT1L0VFRgTFDJbI2 H+KlbaIoIomCY6TbjUUQPaYfgiJ6rZVCYGClP+LomXZzhbY1WTNzfv3hZEwwzs92mNG7C+v GP9g4TVH4xBdqqSwehvTgygusMqE2oUqC2cGKIhgfVDEQJW/t4ekzDi26VXIW9QP8njhll6 W0l7WVqhhxQ27zpWV/j/6rOgMkyYz/GQG816a5JEhhLkMGlGgeM+ljwkBxELBaJJrztnNGD dspGKP7paDbt7nX+v18rbCFEypfvuS9gOBP0r6AEOcN+wfVRX8sB8ybKE+HA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 7880A1606A3; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 673BE19F320; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 37/37] arm64/mm: attempt speculative mm faults first Date: Tue, 6 Apr 2021 18:45:02 -0700 Message-Id: <20210407014502.24091-38-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7BAAA600010B X-Stat-Signature: k55dodh4an7yp46z9bxhckqifxbneuwy Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-280984 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. This follows the lines of the x86 speculative fault handling code, but with some minor arch differences such as the way that the VM_FAULT_BADACCESS case is handled. Signed-off-by: Michel Lespinasse --- arch/arm64/mm/fault.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f37d4e3830b7..3757bfbb457a 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -530,6 +531,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, unsigned long vm_flags = VM_ACCESS_FLAGS; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); + struct vm_area_struct *vma; + struct vm_area_struct pvma; + unsigned long seq; if (kprobe_page_fault(regs, esr)) return 0; @@ -564,6 +568,53 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + /* Only try spf for multithreaded user space faults. */ + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); + goto spf_abort; + } + rcu_read_lock(); + vma = find_vma(mm, addr); + if (!vma || vma->vm_start > addr) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_can_speculate(vma, mm_flags)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) + goto spf_abort; + vma = &pvma; + if (!(vma->vm_flags & vm_flags)) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } + fault = do_handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +no_spf: + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -604,6 +655,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, } } mmap_read_unlock(mm); +done: /* * Handle the "normal" (no error) case first.