From patchwork Wed Apr 7 01:44:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AD47C433B4 for ; Wed, 7 Apr 2021 01:45:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7D04613A7 for ; Wed, 7 Apr 2021 01:45:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7D04613A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 94BB28D0003; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36C7E8E0005; Tue, 6 Apr 2021 21:45:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB69A6B0096; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 58CCB6B0082 for ; Tue, 6 Apr 2021 21:45:07 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1FCA4181AEF3E for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) X-FDA: 78003877854.01.96B7E22 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf28.hostedemail.com (Postfix) with ESMTP id 0A9F7200024C for ; Wed, 7 Apr 2021 01:45:06 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=wmzTNOpmvJgU2+ktObZqYl7ztD+8ceDz1BXrR5WVsHA=; b=WqfE2X+chCev1QmePRjeuCrLGUj6TXALSmqGZlONI43Pl3PjzGxGJqBzjzNYC5wCMyWzX 7kAAXcM9WXxDLhPDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759902; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=wmzTNOpmvJgU2+ktObZqYl7ztD+8ceDz1BXrR5WVsHA=; b=j2O1Dn7hwEqXxPaFt1j92dvD5LnaAj1gNu182V8uNbDW77KXiHD/IvchklgWFg+3HdPiu 0Wc9n3eoMVW6ufA4LGoVG1h8aMAZnDdsDM87uQC2QGKU45z5x8VGSn2NolG0WP5mUmLyC5w wANilgJ5mW8LgdMoel/qRH9I9DLbNlbun1WZwh52o/0QvbkjyX47f9yNvp5K8r6/VRFO8X+ oTiFyJ4wp1U6mYbAjJaydTepcGkgyyY80luQ3L+1EJIBqxNkK8dts6OyilOpk2JiMaoB/j5 pIlBWfU3k+iopAQl7hfoymM6wZ6vImnYBVfbuLM7xeJbWFHLEyBN0wo7t5cw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id AF0A61602CB; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id A162919F31D; Tue, 6 Apr 2021 18:45:02 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 11/37] x86/mm: attempt speculative mm faults first Date: Tue, 6 Apr 2021 18:44:36 -0700 Message-Id: <20210407014502.24091-12-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0A9F7200024C X-Stat-Signature: dkxgkfrttcdqnkkx6f9pk9khdeih3emz Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759906-603995 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. The speculative handling closely mirrors the non-speculative logic. This includes some x86 specific bits such as the access_error() call. This is why we chose to implement the speculative handling in arch/x86 rather than in common code. The vma is first looked up and copied, under protection of the rcu read lock. The mmap lock sequence count is used to verify the integrity of the copied vma, and passed to do_handle_mm_fault() to allow checking against races with mmap writers when finalizing the fault. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 36 +++++++++++++++++++++++++++++++++++ include/linux/vm_event_item.h | 4 ++++ mm/vmstat.c | 4 ++++ 3 files changed, 44 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a73347e2cdfc..f8c8e325af77 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1219,6 +1219,8 @@ void do_user_addr_fault(struct pt_regs *regs, struct mm_struct *mm; vm_fault_t fault; unsigned int flags = FAULT_FLAG_DEFAULT; + struct vm_area_struct pvma; + unsigned long seq; tsk = current; mm = tsk->mm; @@ -1316,6 +1318,39 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) + goto spf_abort; + rcu_read_lock(); + vma = find_vma(mm, address); + if (!vma || vma->vm_start > address) { + rcu_read_unlock(); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq)) + goto spf_abort; + vma = &pvma; + if (unlikely(access_error(error_code, vma))) + goto spf_abort; + fault = do_handle_mm_fault(vma, address, + flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR); + return; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1412,6 +1447,7 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +done: if (likely(!(fault & VM_FAULT_ERROR))) return; diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 18e75974d4e3..cc4f8d14e43f 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -120,6 +120,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SWAP SWAP_RA, SWAP_RA_HIT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + SPF_ATTEMPT, + SPF_ABORT, #endif NR_VM_EVENT_ITEMS }; diff --git a/mm/vmstat.c b/mm/vmstat.c index 74b2c374b86c..9ae1c27a549e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1365,6 +1365,10 @@ const char * const vmstat_text[] = { "swap_ra", "swap_ra_hit", #endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + "spf_attempt", + "spf_abort", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */