From patchwork Wed Sep 27 05:24:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 141FCE80A88 for ; Wed, 27 Sep 2023 05:25:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1A128D007E; Wed, 27 Sep 2023 01:25:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CA698D0002; Wed, 27 Sep 2023 01:25:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BA5B8D007E; Wed, 27 Sep 2023 01:25:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 75A918D0002 for ; Wed, 27 Sep 2023 01:25:25 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4546A1CACB3 for ; Wed, 27 Sep 2023 05:25:25 +0000 (UTC) X-FDA: 81281239410.05.ED8B8B5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 9F31714002B for ; Wed, 27 Sep 2023 05:25:23 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nx5z+q+Y; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792323; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NX7G4kfRjH/776I2tCnKq/UfkE/87x2ty/BZzBX3VcM=; b=0vVBMEmGnUyhfJIqnLPFQxqvSDkL6+2dQO7CCvwpcHW0uBx/jRrbcV4ECgPG4N0T29A8bH 80Y+TyBtKPzYpbqjdIufQ3vyWmG7L5HXsyCCn4E57dQcQ/OKtrQjHr1Ulxx/yZWhyfvxNu gYvXk8qTVIrTmvxniq/BYJUyS7c1FwQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nx5z+q+Y; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792323; a=rsa-sha256; cv=none; b=RQycAaEqGBwACT7j2Z8w4A4wE3qKPpDKyny7HSGRTBbl4wiz2Y35iJaa1hVi1aeYeNUp1g 06JmJUNx9cjvB6aJtH8XwAp/eNUFxb5ibuU1iC2y1aUyvTV/ZxDszEGGwlMclS5QdCF0R9 Bfb/c4zKXSAsL+Uq6nDxqyTP9ZhGc1I= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NX7G4kfRjH/776I2tCnKq/UfkE/87x2ty/BZzBX3VcM=; b=nx5z+q+Y5mfZUz97Yc0HLl2p+M GAmlRaw+SiyWfHH7+r2ydgo7PU4geTcapQ9XeQGVbyRUAv6Ok3MlipWmbne4y4ylD7JphFzLo3jze /r/xSTeRRPHvhmuvUN0AOfayn8ioX/VLzlwgCtcqCgiKuXT/g119JqyoE2dqaCshzkiwxPlnB4vxm CoOscPBWBDX3YL6OogbmBDOrA3eZ/tYhKdQPTMOU1ELQsqn2M4JR3hMk7K2cvH4qvmgBn+o8nZ48d 7I9pZnL/IrL8ysWWofIOQQwLoo2OJPRPm3wYNAq2JekXzGvhe4Uny5dtyTlJFsVhm/0AsmBVdP1UT nTKAjZNw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Bywo-GQ; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 1/6] mm: Make lock_folio_maybe_drop_mmap() VMA lock aware Date: Wed, 27 Sep 2023 06:24:59 +0100 Message-Id: <20230927052505.2855872-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9F31714002B X-Stat-Signature: ez6d55mzfhoi4ash9cb9zacdning5kfu X-HE-Tag: 1695792323-876536 X-HE-Meta: U2FsdGVkX18Lu1KIB/fuqyj4iKHFmUTuVa4aUTDShC8eycAjBrP+n9FYb6Tlf0XqjcjYZPdxlZ6cq6HT99K2TEz9RNlkw8CPn8lJSiLdf9EuWKkM5Pqhpr/mMBxe1bKKPknFGj3VyAMaIFdL6bMwAZGMijK9ihvi/1weRr1XGvmbJmxATNKsmxVigqhGyPqOjM0Eurk9up13wXMS1FoBW++GkT8NPOEy+OMW80C3+mPRmfQ9foAPyswY/gTvFnpsnwPLL3M9b/Gl6PqlnFNSwe74QQbtval3HjplLbq8Q38iOnc3AZQvjfZwsdB2/3NAtNPTK70wQQvDm61jj1fJdK0ifUqlOkbyaYCDYJvh6QvERxkfrJwKiKhykKdEMcRtVgp2vY4rXvLprJrRfpZxtZkxd+udGDjoBg7DdnLnTSZQLKuYpKtOsoQQN/IZHS31wV7vVI7NeIHk2HJui9H1vs1B/OwcuzDXA0Vp8Bib3083zEAfvbvae+BmvJDdsEZF3SMhpesdW0u0xiBqMtGK8EaR7g1fWWGXtjb3V/y/GV3BPejmNb+rpLxNolDR4sf/pXuIm7l+N4Ycdt9GXCjvWGEg7dr4jcNPaoSHKGgIWMCQs43OLRuitgG1e89R8nnQ2oCTH2mZzHHn/3eAWPqY8zeaps8j/ZtUoKG3oIsIT9rhkpmHV28bxefhiGLHVPWZsM6xPFNBW/knTYZzEiX5snFK8H22g6fSw6B1rwpP+shwwRBKcL99zCAPLbncZHdixph/cC/2128bm2PLiPXW1NF3v84gJ6oWaoNbCYjbCqMfnHziHQzBujMBYTnFOd6TYcfldqE6bOzz8upFQYqpN+5GLUmSu2u/whMf+M7CTwHvOmKgfoeJiYGZCVYcluiyl1eAB0bxfwW6IhW1+wQiFTvYIY6dbSeJrere7jHntHM4kZLtWSyBjtwc30rpZW4aSOF+mKydCWdOvecKWoz KP+WmvId 1bImJzm98gd17229ogjjB5kFi8XEzQZA5nNoCJh5+KZ7W+f24h9gaV4L/OpWel5OeKqiP2ctIZIABX54P6sJLZIJGN+moAQ+Xzdnv4yvR9GyJ+VsE5zAbaRjM3LmiFg8Z1SDieoxbZAafQ+A87d/SuitZwgyLrH+CqXvIneSI4/RNQKbVKtIVn35spgV1Zmzf3MRAWQ058IYPF7CzpdgilvXqSjB/2A48UPZa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop the VMA lock instead of the mmap_lock if that's the one which is held. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 9481ffaf24e6..a598872d62cc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3104,7 +3104,7 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, /* * NOTE! This will make us return with VM_FAULT_RETRY, but with - * the mmap_lock still held. That's how FAULT_FLAG_RETRY_NOWAIT + * the fault lock still held. That's how FAULT_FLAG_RETRY_NOWAIT * is supposed to work. We have way too many special cases.. */ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) @@ -3114,13 +3114,14 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, if (vmf->flags & FAULT_FLAG_KILLABLE) { if (__folio_lock_killable(folio)) { /* - * We didn't have the right flags to drop the mmap_lock, - * but all fault_handlers only check for fatal signals - * if we return VM_FAULT_RETRY, so we need to drop the - * mmap_lock here and return 0 if we don't have a fpin. + * We didn't have the right flags to drop the + * fault lock, but all fault_handlers only check + * for fatal signals if we return VM_FAULT_RETRY, + * so we need to drop the fault lock here and + * return 0 if we don't have a fpin. */ if (*fpin == NULL) - mmap_read_unlock(vmf->vma->vm_mm); + release_fault_lock(vmf); return 0; } } else From patchwork Wed Sep 27 05:25:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 803BFE80A88 for ; Wed, 27 Sep 2023 05:25:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E85F68D0016; Wed, 27 Sep 2023 01:25:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E10A18D0002; Wed, 27 Sep 2023 01:25:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD88B8D0016; Wed, 27 Sep 2023 01:25:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BA8468D0002 for ; Wed, 27 Sep 2023 01:25:11 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 84DF140987 for ; Wed, 27 Sep 2023 05:25:11 +0000 (UTC) X-FDA: 81281238822.07.3C7C3DA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 3E0BC80006 for ; Wed, 27 Sep 2023 05:25:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mHmn8Ro7; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=IGu8Ba/4mZRAK+1j42cC/MSwoS7mvDnVdMdFTPDxyav0YazLXh/xDhVFRgLOThGz00TkM8 m7DIxDI6J5CwR3GPBpa0kAs8JZ9BhlNIC+jwvRy6CDJwk903xVcvp5ebSKrwREBH4gYH7f +4ZSxkIfdQSgP3yYVqXT6mKpKq8q9gk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mHmn8Ro7; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792310; a=rsa-sha256; cv=none; b=fW2MXj45W5IE0y0hELKB6GS9zrHYcfRGRwO2/6hWlMHvrF9UQhMYIvorcFOwMl+VJ4bU9s 8HQWM/BTa/uQUOlWd8nkrWslUgbYHKtJb99Ny+8P6qBDZNQzuwlX+lYgxYuXX9Zw1WPyXH 43PTqkN6Hix51eUj2UI8m3BtAouNiho= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=mHmn8Ro7tjrGPcAYhpGJcpU4ap jTzmUsOkJXJkLUe5Zripo0Q+kcFaAGeS8dRlLY7urN5xbSVedow39IvYW4cWKRlRiXfs6BX8vUqh+ VMtpyHT8T29PUjlhx70sRgL/6lJuNg8ouo/fTBTdCeSqCXQaxWf1UPXi8utcBRk+bSYvkcLhrdFFG 2cTF9D7aCnHTcmD+7PAUIydsu/LLZqjZXRXNUBtyDbwswJVlqzrr7zOVaEwWyKRPfYyKPk7iU/p+A YsWUph/7hVSy7xm/jD9zUeHLXm6D1gJdQ1kgleXNj1Izkyx+8CNMFbeCQDd6LdYn+2oSTOdzzXoKM AFJ0iI3w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Bywq-J7; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 2/6] mm: Call wp_page_copy() under the VMA lock Date: Wed, 27 Sep 2023 06:25:00 +0100 Message-Id: <20230927052505.2855872-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 3E0BC80006 X-Stat-Signature: heu4i87sfnhznm6mj36yc1e9w48qpi1t X-HE-Tag: 1695792308-410330 X-HE-Meta: U2FsdGVkX185rSuEzmkLpoKtys4nlxOL6DvWw3Ewf7vhjbRlfQuAA8apsJsFOATcrHczSMa/ZEfZz4EFyjV23zzgiHlczDITnocGxcqwDWNMVBu28rYY5S6ILQnC7EuYGUMBqBah4j+tICiOPGRVgCleXmE5emlrqnWTA0DGHBpsSlJjCRLV/64WAV+oas09j1sIvcs6eq+ylWt/Ad4ixB8nFpd5NDvwRs4oTxS6AyYRn9pUoHUywbznRcB9gL4O6J891xqTL+hEBUsPsHwaHCnLHVaQkqKHSYUXFiQGVEpJ+wZBn1dYSLyyDVB4BAbeZ6zul21aKGOcn8b3W7nvPIOJ7svWs9llr2S+AEZtUu+F1cls2RvV5afWCdpvQth30/57xndUHAk8aoz6pFv+JrHNLfd69AJrSCEe6HJSRslHragIUijJp8GA1qTp80KceL/Xl+3o7ZLlE0E15Q+nYgf4HGvJ5QZET4YMLfDA/F6V9p0XA6ppEQg4C2lHjs8yYxSoYXtpJPEeUzZiWVXCzVt3ME/ArILqL9ELhVfxCGTxpnYJMhbwzmQx7MHureKQmAmhP2sr49tvgAh6zaimb8ZGT+DM05DnRk3iPTDfBHwHA8Q/8I7SDmadCGepr4cPPpqFJVfzYI/JIEj9m/6WKZZ6N/uIGnTVjvA2TEcSKcDWVAKA60CDl5Tw84Q3M8mcdpULE7oQmLJ43OnIYy4skalZYF6SPsHPI1VJYhMIGT8nu5AH8VUCINkbPq2upbi/Fc2MfJCtv8hQ7RovrK0pMyIIBNimnUTb4OIsO9PVI6v6ndP+fCgI3Y7SZycSOMjTGENUxVadL1Fyo8+hBWJ/Q9af0VLTEUK2Vm4W9amxAY8k1XGGAAbeC1giAYHpA/OnbBuH5Wt+P5HVTPQkcRpt0/plSpj3Y1nWLGVD0Xo1WIEDBz4oyL3Rtpk8XUZHyy60ASOv+TSxRDwWbE6IwJq 7Pl8j7FK h8cxyFSFLeSKPVUriAqkRIt7pBjEUzO3UXLMvBkJAWsxee7fSSugUvvG1zPhzdYp6kLBYFZ6uHNJH0Wa5fP9F95xKAJ2OZXDzyr5PDRnYyWfEfR1axXQgAeUxVs7qLcLdlpf+yYMcZu7U3y2IuvO1vxkEcGnHotMyB9YzUnpGgxIbsKRcWXWo0vtTkKXF8hMmI+SNf4OFg8hYDGsm5XFpHtPWsy7K7nZxNnr8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is usually safe to call wp_page_copy() under the VMA lock. The only unsafe situation is when no anon_vma has been allocated for this VMA, and we have to look at adjacent VMAs to determine if their anon_vma can be shared. Since this happens only for the first COW of a page in this VMA, the majority of calls to wp_page_copy() do not need to fall back to the mmap_sem. Add vmf_anon_prepare() as an alternative to anon_vma_prepare() which will return RETRY if we currently hold the VMA lock and need to allocate an anon_vma. This lets us drop the check in do_wp_page(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 97f860d6cd2a..cff78c496728 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (likely(vma->anon_vma)) + return 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + return 0; +} + /* * Handle the case of a page which we actually need to copy to a new page, * either due to COW or unsharing. @@ -3069,27 +3084,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; - int ret; + vm_fault_t ret; delayacct_wpcopy_start(); if (vmf->page) old_folio = page_folio(vmf->page); - if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); if (!new_folio) goto oom; } else { + int err; new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (!new_folio) goto oom; - ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); - if (ret) { + err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); + if (err) { /* * COW failed, if the fault was solved by other, * it's fine. If not, userspace would re-fault on @@ -3102,7 +3119,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(old_folio); delayacct_wpcopy_end(); - return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; + return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } kmsan_copy_page_meta(&new_folio->page, vmf->page); } @@ -3212,11 +3229,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) oom_free_new: folio_put(new_folio); oom: + ret = VM_FAULT_OOM; +out: if (old_folio) folio_put(old_folio); delayacct_wpcopy_end(); - return VM_FAULT_OOM; + return ret; } /** @@ -3458,12 +3477,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return 0; } copy: - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Ok, we need to copy. Oh, well.. */ From patchwork Wed Sep 27 05:25:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E984AE80A89 for ; Wed, 27 Sep 2023 05:25:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BA658D007C; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 869E88D0002; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 759978D007C; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 60A5D8D0002 for ; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 35764A0F54 for ; Wed, 27 Sep 2023 05:25:19 +0000 (UTC) X-FDA: 81281239158.28.12FF488 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 9AB6640021 for ; Wed, 27 Sep 2023 05:25:17 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BSQTXxPD; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792317; a=rsa-sha256; cv=none; b=YHW4REZpMBNB/FENtLXGvSvXDcTCbS852Sswz5XRhMx2pyejyvy4jozkwTq390fMA1haph YeZxN07RimpruUq11PgRwWcQZ/n2/sisY18hyv084widFO9WwsYaj95rkn2OxL+NLwVebN Yo/yMOM6eQTeHnZqhu/h5xomvqJ8nwU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BSQTXxPD; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792317; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vACvD6A8iEuqKKhqCoG2NbL9weZtzxwL09B4YNYsPv4=; b=U742L2YsxgXUjMk1Z7w9NctxqKXIcvMKmTidWK3E5GL26WFqDcIWLuXUtLKn67K9zrPRp1 cNidF042/ywgkF1MjMRlT/JaQbCX+X9J8kmZVqVmuXNc6NR/aNuVEYTe9+qZUAEGgtA3Hr ezKKCN/RCBXKzekd9Heugvr4Cflxeog= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vACvD6A8iEuqKKhqCoG2NbL9weZtzxwL09B4YNYsPv4=; b=BSQTXxPDX+1Kl9mwmPhVKnB0Qo 8edzTyOe58NeuMrcsFv4tBjacPPzioLUtdkCLEhAOlwV73RJshNAK/RKmdA78JgFk3duPJ+Am4P+E GPnfH8IpPeAo/KHiK+cWKYnNCGqTip+ASc0apsJkJjWBebwT4gXrD/QjRFD9Ce/r3tCSJlJA/yevq xKKaMrDUz2BTQnTOM7IE04xi8kgWfTO9CErNXvnoFEl7Z32S950cf4jAD/sLAbKp3fOVkXPMqbPFL Pf51d24HsbQRKAISvJ/vZ7JG/5ieLLzrGMwI3lYdr/xs3C4cLSMkM9TSYKMw10+6qTRjaTi6XVH1u tj3B++YQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Byws-Ls; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 3/6] mm: Handle shared faults under the VMA lock Date: Wed, 27 Sep 2023 06:25:01 +0100 Message-Id: <20230927052505.2855872-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9AB6640021 X-Stat-Signature: 38x7ao55cet146f7jrb6k4k83ns9ospi X-HE-Tag: 1695792317-986811 X-HE-Meta: U2FsdGVkX18nKZP61u822h4jY71AhMNbcGmU3X3+Pw6u4XnoVJcQa+tXlqWWkR0S92gWGg2axoIv+sJapBFzRbgduVWjXbeuGzub8AFzhW5bQTSSeoLN37TDe2kWJubJ/Ky3ZiB++HVugI1xIc9zLb5fRn4n7sT4p1ve/beSuTt9HJW8L2h0liy8/CrcTBRe8Jn6gV5ndGkDPX3eSGGi8V/ZQ56KhmcfVw7V/CL7L2V5zGs7W3LPcn/C8ZFVpme88EqwFNWk+0MRhMhPAx9bOiH3zhS8O9P+dmS+9yRccqg29mihazxeIG8GYeMFf+0P8y+52BHU/iQ9AbUjs7CZ2s7APxn+yw2PY2jtDq5n8tdrhrMsMw0p26kTEUjk67Gvq9GtPEVkCJLqH/QETmrdNXElWfwb9S0zlBOoa7jvbsi3aYfK28OIqgLAQn3GwTKHouSsz1poAftos0rEgVrS6156b/6cKKRtYPZuI5HAI+q+KHfXwQlxAIl2yxwp1CG4J9Ub4+SEytfjrV+VMqpQ9IaM392SMHMJgpXaVY1amPCvNadGwkN9RLIgFClci67iN6YsM9W9ZOjnLbyXtzmQV1rKsKkzWGgrWgVRCy6LJpzgiSjn3koYWk5+VedvBuxMAPJZl3ReP0dv3uLOgQW7mZySCakmT1ATqwoYBINILCit6IiAkM+BTBnFZH2hYbypXANAbHKO/yrZlvXa/f+Ki8rA7e/CQePfIWpGMOwlqIP7m89zOgMRf8o2mVasL+StA1LJLtRN5Mc3y2y6oWylFPN2lHlQ6Q4NXcH9iARHapo2yZuhIT/66ltOz+tbCg6cX51O5vSf1MZbYRtqS4vRKf3mxuDGewRg4mOnGd6ps9De6H5oeeG5Q6winWsU0eeOJQhTgmur2XdL4ZlMDS4EVWq+GW8uDN27pFcQMlo4Lrtx9XrH03grkevrNzkhZRFE0E0Yl71w0GItp43llax 2H+EzLBD 9Iujh3aeDG84KOYwKjzwJ2g7E1yAhuXShiGV4CFpq2CaHGBRU5CkR3rR4QZ9e725z+ipfVo8QgJhE7Nvbik0D3Y6x84CN2a3Q6cTnh3gPeh+4Qj8mIX0+C06acyLn46khYp/MbTAyBzG02tj5NS0gps4Xqd8VAppY6gMEa2inHUr1i/TvyJlqRyrtDQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are many implementations of ->fault and some of them depend on mmap_lock being held. All vm_ops that implement ->map_pages() end up calling filemap_fault(), which I have audited to be sure it does not rely on mmap_lock. So (for now) key off ->map_pages existing as being a flag to indicate that it's safe to call ->fault while only holding the vma lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index cff78c496728..0f3da4889230 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +/* + * We could add a bitflag somewhere, but for now, we know that all + * vm_ops that have a ->map_pages have been audited and don't need + * the mmap_lock to be held. + */ +static inline vm_fault_t vmf_maybe_unlock_vma(const struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK)) + return 0; + vma_end_read(vma); + return VM_FAULT_RETRY; +} + static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -4669,10 +4684,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) vm_fault_t ret, tmp; struct folio *folio; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } + ret = vmf_maybe_unlock_vma(vmf); + if (ret) + return ret; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) From patchwork Wed Sep 27 05:25:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BAF2E80A89 for ; Wed, 27 Sep 2023 05:25:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAE808D007A; Wed, 27 Sep 2023 01:25:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E38D88D0002; Wed, 27 Sep 2023 01:25:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C62148D007A; Wed, 27 Sep 2023 01:25:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B57CD8D0002 for ; Wed, 27 Sep 2023 01:25:13 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8901DB41EA for ; Wed, 27 Sep 2023 05:25:13 +0000 (UTC) X-FDA: 81281238906.27.7F6DA0A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id EF198120013 for ; Wed, 27 Sep 2023 05:25:11 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PxqTP9iT; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Yk0vVaBtqAZEgIGaPP794XvY+bzQS0FZH+cnfPYchQ=; b=X+3Yamht8qdqIm72nDvCfUnucr4Wheos2DJNw4Q+4NqtYj+rTz1ePtywZCae8O69w33BZu KYHvs6frM40DA8ssEfOqon8Hx9QlnclX4sYPsckQUWi+49sjrJwsJpYZR5n1f9cKEAw0tg +bONlnypgMJ4kUpyAExSlSdfiWTgpdI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PxqTP9iT; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792312; a=rsa-sha256; cv=none; b=VNnsqziw1lvXyfKNO/5WKsX2Aol0dqMxQxSHjqykW9EDfL0GkhibXGpwXvf3o+Iwuq490F AJyYFnnCRRbsxEirWVR78b1ontn1SpGxR/Xjmh6UqbqyyFeBshAal+X1xJ3mVH0omZCtA/ NMVCQSIpXfeFjIeFSXnez2JNJ5bpvnU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3Yk0vVaBtqAZEgIGaPP794XvY+bzQS0FZH+cnfPYchQ=; b=PxqTP9iTybWuc2gw9i0H0Bfmmz Q06CrSXhJl0/Zky87Af85WtzKt/i6dHt1bGBE+62LKdJeQO+PQar+XNkLsGXTIegJgJpgxWuIQjlp fewN8OSDOICNeOiH9ks5kc0COJJh5D36kkLBdacWDH4UaVYXsdi4vcmRXvu4p/3wY+9vJp+v6If6b yV44LiSuJLMFwaJ3EE9Ex61/TJsDj98uOBz5jFVR8ffeeWs0u0h9er+XuZBAoBR+sy2Shavcuh46l lYf97/cXvlahmp3A+CFexkiEbRb3JSuth36xGuxbSbZ08IMD86unZv6Ry35LOhky6TSjL1qNyXpBa dS2fteDw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Bywu-QP; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 4/6] mm: Handle COW faults under the VMA lock Date: Wed, 27 Sep 2023 06:25:02 +0100 Message-Id: <20230927052505.2855872-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EF198120013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hk6bmkqdtetbe8yint5ob5h56f88ouiw X-HE-Tag: 1695792311-904856 X-HE-Meta: U2FsdGVkX18kgA7AAVu6rfcxGLrTCZwqp7dUrjCsy6ACJHY3WeWVi+cNLd/WuqsiDHQR/FLyteGd6LTLHdF/wLt5ahiwBua5I/RMWh6qGXxuZnphG5PrLQtbEvDgxjT0PgJhznJpNZJIR8MT7Jsa0W7vcWlRsWD2s9cWKnUJv/XusI1D8hl4DQB415CaKcBC3S0c5jtcVkU1+S0Z4cOsX0Vh8+RNCfAC6RjNTnu/01bBBT2jH1WiyeVVGZreoQ8DxzXYZNcLi6U2SsDMIVthzdPmagcQflQodYcdKdRGxZNuzKqc52I3IDC7EzHN2i4r7XxqX70hxkcKW5ZxSVdkWkdGMVOI692PTMnnAvzNELjw7b2+vqjwOYozuFhPhO2uTlkibjGNqcu0e+iyjaKhmAge4W+LMrrmxjLOhGZuMdp8aOBplnlho4F8ZJchU4fF3QZbxEnZl7WVEMT8pOqM6BMQ6IiOyJkZ0bFOMWOBmyA9Sc5qvqrE47fDqlVLq85HVYW7V2WHpm7H8yzRLSSjAVFA2X6vjNkV99cMUvKQYSIzYORuJcHxSpi3cTnz/EzCIXsTd+WpZuI2bPIEtPj7/Wn9M6gA2v7oPQh8b6BCLDKnjm9kOqQwynwBKwsfdZS7zWBiA+FWZEjOO4S8ctQDX5qeydN7YYA3Zw6gFDqtMobodIin6tReUGcqXn8u/6+s+jR9MtDyRT74QSv+Kw2ZsoFlGxBu66cHzpcv0hhS5CJbSDCKN+k9CcAnSJCHSR02PQXHFukQ10bkeWSQ27ipF32RpL0U5rggTt0ohuHTtxOUoeFqb4/JAc0hgcUW8s8h35Lmh0mrQMgSeVBlvcWoqA4FyFlstbygjUtEpbed2tvxLMurgvVLt7sW58CmAAYdJ057oXMBNTgJxzvukOtLJUnmg93597A6w2HBeafHB3lM4njDWV2trFdUSnU+4lEM8bfsnL4xqZCEE4hwiQ5 1WH8F78t 8MDMTTn2fjXvjIr76Wvc37MH/0jPohHAZptJ08VoasWBR13W8FrDWBoNkR8ugkh+bLKZRZehWwjTB0G+XnDnpjw4sQ6CsfoJO3ITUfN81LEo9ZkTlWYblCyyss5y5DmGMX+fn3Rjm5369zLjD8fSd4RFswJLCkOJLXSyjrlNWBECQlR/0zcHxI7rVTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the page is not currently present in the page tables, we need to call the page fault handler to find out which page we're supposed to COW, so we need to both check that there is already an anon_vma and that the fault handler doesn't need the mmap_lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0f3da4889230..02231a9394ed 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4639,13 +4639,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - - if (unlikely(anon_vma_prepare(vma))) - return VM_FAULT_OOM; + ret = vmf_maybe_unlock_vma(vmf); + if (!ret) + ret = vmf_anon_prepare(vmf); + if (ret) + return ret; vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!vmf->cow_page) From patchwork Wed Sep 27 05:25:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57876E80A89 for ; Wed, 27 Sep 2023 05:25:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E385C8D007D; Wed, 27 Sep 2023 01:25:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE8168D0002; Wed, 27 Sep 2023 01:25:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD72B8D007D; Wed, 27 Sep 2023 01:25:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B8E508D0002 for ; Wed, 27 Sep 2023 01:25:22 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 80FD8120F01 for ; Wed, 27 Sep 2023 05:25:22 +0000 (UTC) X-FDA: 81281239284.17.C0B8641 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id DEEF3C0002 for ; Wed, 27 Sep 2023 05:25:20 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ercYuRq6; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bcm+M3iLEHwImzaKDo5AQyWqQd+Kb1x46vwh8DrPUKQ=; b=utgNH3Inn9NUJU1MbbUbN3ee6Rz/lMfyk66XbbvQHezdsTosXJ1dUhGNNfooReOnUw6gsv OP+O9jbkR2FJEcJ0uY1KH00QMH+Y4NFFxRj36DcdJINQVz33AgdZrQ6YnRog4gAIVmbwUQ f2Ks9qwW97vrvaiKID2z/cEAn5zPOg0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792321; a=rsa-sha256; cv=none; b=gfFODweJmRQZ1Wh+8yiiLoFN2OaGSNwWsUew/GzZN1aufQP1H+U8EJJSLpTH+ng/Ar/Nd+ z+fPRncGKV+7Iw7zQUnd5ZlUg1FFu3MhDhPdxSFFmxMlssQZ41MRip6vckzHAH+Mrz7bdY SKTmN30xi6zKfsSsG0upjBd1JQ9lwvE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ercYuRq6; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bcm+M3iLEHwImzaKDo5AQyWqQd+Kb1x46vwh8DrPUKQ=; b=ercYuRq6ui0ImEvYQJHJdJYNg5 tYohqyb352TcyFFAPYkQdsGmXSNyDvbWGnuJt0UTrPhJ57vva7JWwteAyrmbvVcUldNz/tcafzA1I EEIWwOtNGt7LvjZXcQUeEu74JlE5+ge933oT5Zvw2kiNfWMKTU7qGoiY6x1qxtZ0tEtgvHkTTW2ep pj5kQdZgw3rMAogSpMgvi5v09gP+E2lrZzKIHFlnaWf1Y5vunJmrJtKBlCsV3ZLZzwAY9WpU3q8av R7KtCBWs0Bl1Iw/TtGnOZCTY/5s8yjnWzJdyOx6YIE9xZ5khuH293xAmGzkRjDbayCxZ0MtrmEzLi oaFwSNqg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Bywx-Sc; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 5/6] mm: Handle read faults under the VMA lock Date: Wed, 27 Sep 2023 06:25:03 +0100 Message-Id: <20230927052505.2855872-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: DEEF3C0002 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: b1jdzdi59so5951nhw9t5r86fg7s9kcu X-HE-Tag: 1695792320-210053 X-HE-Meta: U2FsdGVkX1/HNFcjFUj8DsXNGqKcwu9B/IaMq4Y2QfhODcQtB/WmQHk9Oiq/axVv2Ejg4lCARF8K4+x4maE7/J1VIpCLrLhBWXBgx9LgtrFcWuhbbcvHh/HobOmHsKI0BcjN5X7d1iGoE6yWZBbgvELHfIDh+d8AyPCqDkKGRfJybFcx1f9JzzHlkpIqkkAzjz7kQbAwnZVeaqwelbDvCJvj/VfvtsxD5ydvkTpkR714d3s2hVWokhfWrGvuvYYv6L/s9ep4+5/Hu+k2Llr3ajWk9gpOrDU/fG4QzGPG0fBcC9hegNQ0ySr0b3uyWH5oyaQgl9SF93cdntljt3S0yKd3EuzntfVjKoXJq6tlk2voGqPlb+0f7qB8A2BadxkrikgGPAcWn8i+aZbJFgXFKfbq/w6qV0cMQQ7MbzELfCLQRueGezpTuV87Wj0NFHKF3gdtTzAJfu4owWxvgmcA1LGfWKGC/gFaGKpELptr0A71oAElKTfM0PcB4lwgWRL+Y33H8OP/JVw0PtjrTlWFcdkXouD6Ho6AjmV+4lqem8h7u3fvc8rJfMbCqWzlYMk6aZHC2jppGAJ3Itk1EmEHKu7uVzxE57xyG78cQ2dzf0ozp+6h/+29WEwtFg0k4WQOWnCGJsYRJA+9edbZ9djhtsnMkQ8RfadZ32e/FgD2hdeC/kELRJBi9wsN1q/iCBYT//m/hdobTI6j5yBEx/FgKClEEB/KcyeTgscZeAjNUZC9qL9cgWmH2U4C1uUm800K5k4WIBZWa869kc83kv1TEIyYm8oHA3GT/bi/+NscIAgteHTes3frQMiJ7UE0aTH+W1opyw5HwnFIaRvToeKaxZagzhX/6OES96O/2dG8g29dSThxngeyBritXaHLXdB7KfmDgMI5An7Ra82Mua7MpzPYuvylrL6bOqxprilgr/7Wkmap/ASc1HJUygkohAba2ezImTuCLt4vHJA/QvC eHEmAwu6 3keOiPm0/mg8BOk4cthZDBWhreAoGYaviAiAazW7j+AsmKOLzXwfrCei2e1FkxK+K+KuTGMPy1Yjoygxk0PXoIKCPO6t9NHoy/C6rLvGLVuHFaVu81JTBnSgGKRPQBW7huQbDSzMs1nZDrTHIDDWW/UXo2PWbPB4zLSUsx/1PetIJEGjlEVrZ1sb4xQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most file-backed faults are already handled through ->map_pages(), but if we need to do I/O we'll come this way. Since filemap_fault() is now safe to be called under the VMA lock, we can handle these faults under the VMA lock now. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 02231a9394ed..5f92126dc527 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4617,10 +4617,9 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) return ret; } - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } + ret = vmf_maybe_unlock_vma(vmf); + if (ret) + return ret; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) From patchwork Wed Sep 27 05:25:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6C6CE80A89 for ; Wed, 27 Sep 2023 05:25:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 844E28D007F; Wed, 27 Sep 2023 01:25:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F4B08D0002; Wed, 27 Sep 2023 01:25:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6701F8D007F; Wed, 27 Sep 2023 01:25:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4A1B28D0002 for ; Wed, 27 Sep 2023 01:25:28 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1E3061A0FD1 for ; Wed, 27 Sep 2023 05:25:28 +0000 (UTC) X-FDA: 81281239536.24.208F604 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 7702CC0018 for ; Wed, 27 Sep 2023 05:25:26 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C555FVAk; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792326; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=THwc4fnOhp9odGE13p2PG8xoZR7MkxHpU6yd8j7wazU=; b=LFEsgkY3WikamGDqNGs8pbx8NYAcBRBT1ti5mXCyo+7le+QuyGIOFkaz323lSXyauO/36k A6aC2ZFLqQNT0oOSHxS1OS9M9ewVwdhai30T76x6uKEDqeuZjPKmXTnDXRYb3OTzys90+O u6gQa2GnplSANkL7RiZGQMwYTT7WZZ0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792326; a=rsa-sha256; cv=none; b=7QPMgUbxOa4fE7boZkrXdv2E7W9B/jbvaVjslg60sA34XaNgenFUPKAHQsxTdkE5t5bODX yhwRDYIWS3uwGRjG49sydHuHQddeMx6wa3dajUT09M9TXUPHX8ndv1Hta8IYsE/pnSBcHr ByLXprgOwdD0RmHdBTkmYzBXmxaKu3Y= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C555FVAk; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=THwc4fnOhp9odGE13p2PG8xoZR7MkxHpU6yd8j7wazU=; b=C555FVAkORYddRC9Hc7wFKafyK o8Vtg/i1YL3DlOC99+hT1ecHohXzAkHV/WOHgk7/aRLWCaTRadWJ9rrHwT7XrxHAcLrLszEwCk088 soRFEeMiMtXt54wu6QwzHDW/8s1DhVKYBqDnTYysJo1G5ypoPK7c7mHKTEdXS+kuhfyZb5Oj+71/U YIwhg1V2zYT04GwRwGZ/lGPJJphhfedt3ai0qx0se7N6kjVjqz+Qvdrj57q8kAXZZ392RYnxYaua9 Pk+wOZe+i4k8RQCIHkV8H9aekSkeSKHZ2v3WTdlvTwfKk3VRyQgmfGDyRwrhd2/Pc2rJ1KQK4ZfO2 l29bpzfQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Byx0-Uk; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 6/6] mm: Handle write faults to RO pages under the VMA lock Date: Wed, 27 Sep 2023 06:25:04 +0100 Message-Id: <20230927052505.2855872-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7702CC0018 X-Rspam-User: X-Stat-Signature: booud15w49ccqwquik7jz5ntzjjkgt6o X-Rspamd-Server: rspam03 X-HE-Tag: 1695792326-492187 X-HE-Meta: U2FsdGVkX186HfcKJyEDKeSSqCIVRRXam0YO9Mg+l3L0FwcoqZ5dHvI/050dvcgCcHl5585mYVAHq0J9RMXlAmoxoLoNSEDFJu4mwMXazZ1e6ynSOPZsQAeCfIOjEY/KVWdqvGDAGhpUzrZtsAb7cPQzy6brRJjMv1coAEbpSjetkAnUbmdqijPkGqci1Z94ZoiIfZIjNcEDnb2eawAVCxSwJu2o+vH2gwFivJvEube5Q7XFNCeJIj08Dsco47Mewe2Xz6zJjseEy4cJHdcM2R9mzhLIeDX7FH+Y1vY7g0Xj5BvZDGYN/IphahEnApYuEevXxsBWn5C3Pvd87t090o7KOjeUl6Bg2rQRZn07PUoAcZaNTLG5kOKgBIl2XyOYsu/cp5YFSNrWG45W81dFV41WFZfi3sj5/rsVzhXlXinDrQdyUh7PRgiW6OcyA94huC+trpgneIWRHVrro3xHyJWgVXB05uLlwM18N9iw15pRrPB7eiDvXpYL/WThx+5LwFVVzCwuOcjbPGme9nPpFIQkUvuN837txfr66uUWPA6UZ52KgId5gWsdF6a8Ce+fLtOl5Le/N43c91IWAIEPyJISLNK8dGABTu3TCdkkkDfWN2TABJuVg9W2JFQTdVKaDM7aiE9M5vNpOWlhGag+2W0jVhsWPti4M5VrIgugi8jDX4jM5NU3gmCFMBKoXlJueAAGrLg3rl+HC1i+qbo1spMrpIW13V2HibjblV1lBUuCzQEFMGDTsfKIUSl950xxfnspoOmc1cS0mvaGQMErZjTM2bh3xR3wbB82cJOq0Q2MpTZTA1vY7vA8REiWbNKbIXE5flSPR46JR14ftkajNwDdyL2Hvb1SIZWgo9pQZBR7EBVdOmWgPQ07fnh5gyuR8fGZFW1lcpBMGAK9fipT21Y3mdIy05LeDbW364PrtfFpJYSbDJTb5NoFfQqNUX4JxROV9gErxa1kHxVef+2 1nOWbg7M twRwvdAbPKrg5HEuDHgFqgQtt460ucBIn2sCtAGHBIAJPdPQix5gp8y5HfAPd/Dz4qILOANUMluOXiuhwoHU9Jjx5qL70PzQX6naMuCi/XcOnOnpU/CC9n1md5wKIrH4SWoiTTwZfwI+vwM84HqWiZiuGEHJvY9DRQQTO2huRU/q77vxIReyuiYuSNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I think this is a pretty rare occurrence, but for consistency handle faults with the VMA lock held the same way that we handle other faults with the VMA lock held. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5f92126dc527..45ffa0a527ca 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3301,10 +3301,9 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) vm_fault_t ret; pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } + ret = vmf_maybe_unlock_vma(vmf); + if (ret) + return ret; vmf->flags |= FAULT_FLAG_MKWRITE; ret = vma->vm_ops->pfn_mkwrite(vmf); @@ -3328,10 +3327,10 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) vm_fault_t tmp; pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + tmp = vmf_maybe_unlock_vma(vmf); + if (tmp) { folio_put(folio); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; + return tmp; } tmp = do_page_mkwrite(vmf, folio);