From patchwork Mon Jan 9 20:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6C44C5479D for ; Mon, 9 Jan 2023 20:57:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=J+MTkBXUxZhrcQZji8s9CA7vykfgbBMD4GSGMVLi6wA=; b=I2O9kpYtCOdX8CN9MdhXcAuZsN v5RaAbTiPspmV3lvjgwp8GMIJ5D349l11P0TnBSUGDwVgJprDFyC82Oydo5Nf3k5Y13iixHxte3sV BnKDcTivofS46T0VUsriniA+GrLnhkCdtacdVtL3xwK+jb1zhyNuqE+FaPH0Ohu1yU/etivNtZtrk QL9nWj9OhzSBlLPRVv0n+FRZ0+1yuhDJPE8p6sB8qUKmXUVafXoVXBMFnvB4f7DSrFM3LC8UNh7Lx iK6lnTnX3/peC+fPr2Do6UAF49i+zXDM6DOiVuw3k2l12qM6OwGQiYFWa3QhmLy8wraad/ZRIYqlt lNYjJcOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzBW-00412R-TX; Mon, 09 Jan 2023 20:56:00 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEzAk-0040b4-C4 for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 20:55:12 +0000 Received: by mail-yb1-xb49.google.com with SMTP id w9-20020a05690210c900b007b20e8d0c99so10271682ybu.0 for ; Mon, 09 Jan 2023 12:55:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=AELPgr/6hvWzT+1V7lD4mX315E9t1xeapwUzcEqCdKIYPiCzKk8dc+T92lAP4lgkah 7yRjN7psrT8Aldeuq4taS71dz7N2REjLgY5gl/EmOw6Bjz3k1TLxoTZM4lDuQnOU9Jya +UewXZCmP8ol69UqHnariEjrKTpgLVgEy+/Bgrxiodf6VRTx+vPl+chOix68VXTazEey Yz24CBUMjgzvQiO/GtdPs+W2QXpN/PA/WPYudNeoUrMmtRKn07dYT94LClSxQvVwRpgt N9P9H+WlrUeFi7Q8PmpAS+IgzxksFHRinwxPnTAAPwB27jZBUc2AgIvIZ8+W58HDzxG+ RgCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=hbxLHQSgg/wssSia/bAEPFNphlBoYy9qt+abS5pLvEY3NwxriW4AHMfyo84usQXhOJ XXqPLxpyZjAYgmkTKsw3AtbJ1Ok0a8sV9ekWWKMpjlBs3292+qUJal87pmX/EA6mWI8q KWy3Lfts5XiUbFzw4wzntTD1FKVGyjVMRKpViDKW9VtDbccpJ7U2Wr0xPwvpXkY8gtTO ihKFwPxiGGz2dK2jc2GYpw/4JRvBrZBEfZys3ANgql6Ejra9/ppaii8WNIk9/mlq/qQe qbMntZq0zPgFb1l0pPsEfST3Z42RcG6BG9llXAQ7iB0bvksMJ85WDcVD0+uFpuSkJEGo 7C8g== X-Gm-Message-State: AFqh2kq4v98YiXM9h3F7Jxgr0+aFZUce2qSWZsuw85iKktfqby1j4zT+ ch6BWOFWxxPTrCIExTpGip554Kug6iY= X-Google-Smtp-Source: AMrXdXu86J9QPZ8qYzzFeKpvTLXFdkNuTVtCgOtoNpSPm2yn8vVjTEAXybleIDx2Sx/3r3Ulgd8R54EZk2M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:b78c:0:b0:769:74cd:9c63 with SMTP id n12-20020a25b78c000000b0076974cd9c63mr6699949ybh.257.1673297708715; Mon, 09 Jan 2023 12:55:08 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:30 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-36-surenb@google.com> Subject: [PATCH 35/41] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_125510_448699_9E461A23 X-CRM114-Status: GOOD ( 14.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 03934808b2ed..829fa6d14a36 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 596f46dabe4e..833fa8bab291 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */