From patchwork Thu Jul 13 09:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13311659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41205EB64DD for ; Thu, 13 Jul 2023 09:40:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E7488D0003; Thu, 13 Jul 2023 05:40:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 997578D0001; Thu, 13 Jul 2023 05:40:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EB0C8D0003; Thu, 13 Jul 2023 05:40:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 643FB8D0001 for ; Thu, 13 Jul 2023 05:40:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3C08414015A for ; Thu, 13 Jul 2023 09:40:31 +0000 (UTC) X-FDA: 81006093462.06.003AA74 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf11.hostedemail.com (Postfix) with ESMTP id 9A8504000F for ; Thu, 13 Jul 2023 09:40:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689241229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=apbewmKGP8blQ1CiiHysDpzU5DYCinKGt5o+N+K+bcg=; b=IsTZollCZDtmqF6/96HkGod55MgPaFjgqWN42OtYjdEDV0tUJiHqrU8TkHe1LYuhMFlp76 4sKfe/FUP5GSlK8/FMllCDhS7yO3NIb/DeIs1/KUQJrmX1oCyTG3pJ6bzcdx8zIwaiytha /dbbS4q3AcDl+vjQJTN88ei3akBb7dQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689241229; a=rsa-sha256; cv=none; b=nld+LiT+IVHGelapUQCRrw1G0Q0z+21430QKa0DDS127P8WPOYc2w++CeWo+LcRjX5+ISn y6hpX5+ITc3sq4vBZ5W3w2r/nLxHHyvzG94aF0GLXOEx4geJxsuP8YUnX6V57H3K0Ehmbl 5x6lYeJFL3e8x5Jl6daUKZQ95ws8BSc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of wangkefeng.wang@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=wangkefeng.wang@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from dggpemm500001.china.huawei.com (unknown [172.30.72.57]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4R1qMR24J5zrRlb; Thu, 13 Jul 2023 17:39:47 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm500001.china.huawei.com (7.185.36.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 13 Jul 2023 17:40:22 +0800 From: Kefeng Wang To: , Andrew Morton , CC: Russell King , Catalin Marinas , Will Deacon , Huacai Chen , WANG Xuerui , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , , , , , , , , Kefeng Wang Subject: [PATCH rfc -next 01/10] mm: add a generic VMA lock-based page fault handler Date: Thu, 13 Jul 2023 17:53:29 +0800 Message-ID: <20230713095339.189715-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230713095339.189715-1-wangkefeng.wang@huawei.com> References: <20230713095339.189715-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm500001.china.huawei.com (7.185.36.107) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 9A8504000F X-Rspam-User: X-Stat-Signature: awiq693g3hbykgafaotq98ohot31hn43 X-Rspamd-Server: rspam03 X-HE-Tag: 1689241228-884174 X-HE-Meta: U2FsdGVkX19814nneLRSo0b+jPhA/b1sJ0BMerEJ73vKoaKSWWJ5Wjavr0417hgirbTXaHcIMCE9QXJ1S+6gRmEoQLIWOzJwNwiRGSyox13raJKMcagQ3EufOgKlYZQa7YI4FRA6QqNMy38eW9y7t2Yj33VG/fMuDkCM62iRPr0Z/Kun303CcjgzklaeMu5jYulXnfeIK1yOBu8U7WLQb/4UUTvMmD90tzjl0p/3Qw8NoV9njCO05+THqd1vo6evIpe4Eb7viUAxsnaF/yQ7Y1XR/udfVBLoBn9DswfM96onua+OfcOVkZryD9/qRUvqDi0iNhnWCWx6XqaaTrgYNY9rqF4ZgA/u/f+8k5nugizmZ2SRcoVMh6YD1j7KaJraA2GVH52/Kwd6jWhrKFKnJDg28OokcMLlkof8lYjA4vuBNo/TJSgWcEcA6HylVVe6qmE8AFRa5gH0A1fhqaO7YvaBNGoiIvY8X/CpGvgW6k/LGt5KQIqY2XDa7ka0fKCyP9M9oW/JR1zfjK++zAKkTFT/usgLfEOIPVYdVaOX8dHQQVArPfzUp1XNSpNYTzONklbZGBkwdVVIC35MkpMuVmitURhlbC5qxUlMk5D3bLqLnKU8aWTqp4a6DERs5nhWRiZwjm+lFbGSSZ40kraizraEdF6Babh4bqGcdUyq6Ee27xuCElv8B/8vQ68/IE7ZXI2fE6ywxLXpOk5va7j2UF/Rx8/ZvYFx1hgKqvN/fRjKZyhW2Y0G9YTvSQ+KZW2DcXD+jk8Kgme2Oj8bZLMAgbph8SHc0l9aaClcuUarPNJGUZwMa0ROCMkqq7L8itE9skuZxR7vYPfqKgvUCbtzAFmpb8mcYrde4/+/0TuVbic+t8F4bA+OK7kbKdqbkRlERl9GyXNY/qkhSucMkkv+6Xfb7EDXWep3Uvutb8IUynAePe6JGKXrBIyB6UkgSWP6eUbqLsB096RwxHfevcz 7NF/oUix +ZX9J5LIsu0tI/MWchqBgtCdJIpP/dkuk06UcuV7S94S5gCGu0WunfTWZDBJK+lsM7goky9FDcLVgq2y63Ov1iAoKEqLQPw375CrYNxM9uO57mixRE8LpNwVpAdEiH5RItoFn4HGI6QJuV7FLCUNiXQ8BHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are more and more architectures enabled ARCH_SUPPORTS_PER_VMA_LOCK, eg, x86, arm64, powerpc and s390, and riscv, those implementation are very similar which results in some duplicated codes, let's add a generic VMA lock-based page fault handler to eliminate them, and which also make it easy to support this feature on new architectures. Signed-off-by: Kefeng Wang --- include/linux/mm.h | 28 ++++++++++++++++++++++++++++ mm/memory.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index c7886784832b..cba1b7b19c9d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -633,6 +633,15 @@ static inline void vma_numab_state_init(struct vm_area_struct *vma) {} static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ +struct vm_locked_fault { + struct mm_struct *mm; + unsigned long address; + unsigned int fault_flags; + unsigned long vm_flags; + struct pt_regs *regs; + unsigned long fault_code; +}; + #ifdef CONFIG_PER_VMA_LOCK /* * Try to read-lock a vma. The function is allowed to occasionally yield false @@ -733,6 +742,19 @@ static inline void assert_fault_locked(struct vm_fault *vmf) struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address); +#define VM_LOCKED_FAULT_INIT(_name, _mm, _address, _fault_flags, _vm_flags, _regs, _fault_code) \ + _name.mm = _mm; \ + _name.address = _address; \ + _name.fault_flags = _fault_flags; \ + _name.vm_flags = _vm_flags; \ + _name.regs = _regs; \ + _name.fault_code = _fault_code + +int __weak arch_vma_check_access(struct vm_area_struct *vma, + struct vm_locked_fault *vmlf); + +int try_vma_locked_page_fault(struct vm_locked_fault *vmlf, vm_fault_t *ret); + #else /* CONFIG_PER_VMA_LOCK */ static inline bool vma_start_read(struct vm_area_struct *vma) @@ -742,6 +764,12 @@ static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} +#define VM_LOCKED_FAULT_INIT(_name, _mm, _address, _fault_flags, _vm_flags, _regs, _fault_code) +static inline int try_vma_locked_page_fault(struct vm_locked_fault *vmlf, + vm_fault_t *ret) +{ + return -EINVAL; +} static inline void release_fault_lock(struct vm_fault *vmf) { diff --git a/mm/memory.c b/mm/memory.c index ad790394963a..d3f5d1270e7a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5449,6 +5449,48 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, count_vm_vma_lock_event(VMA_LOCK_ABORT); return NULL; } + +int __weak arch_vma_check_access(struct vm_area_struct *vma, + struct vm_locked_fault *vmlf) +{ + if (!(vma->vm_flags & vmlf->vm_flags)) + return -EINVAL; + return 0; +} + +int try_vma_locked_page_fault(struct vm_locked_fault *vmlf, vm_fault_t *ret) +{ + struct vm_area_struct *vma; + vm_fault_t fault; + + if (!(vmlf->fault_flags & FAULT_FLAG_USER)) + return -EINVAL; + + vma = lock_vma_under_rcu(vmlf->mm, vmlf->address); + if (!vma) + return -EINVAL; + + if (arch_vma_check_access(vma, vmlf)) { + vma_end_read(vma); + return -EINVAL; + } + + fault = handle_mm_fault(vma, vmlf->address, + vmlf->fault_flags | FAULT_FLAG_VMA_LOCK, + vmlf->regs); + *ret = fault; + + if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) + vma_end_read(vma); + + if ((fault & VM_FAULT_RETRY)) + count_vm_vma_lock_event(VMA_LOCK_RETRY); + else + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + + return 0; +} + #endif /* CONFIG_PER_VMA_LOCK */ #ifndef __PAGETABLE_P4D_FOLDED