From patchwork Fri Oct 6 19:53:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 623AAE94104 for ; Fri, 6 Oct 2023 19:53:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9543780010; Fri, 6 Oct 2023 15:53:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DE3780008; Fri, 6 Oct 2023 15:53:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A52480010; Fri, 6 Oct 2023 15:53:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6FE1680008 for ; Fri, 6 Oct 2023 15:53:28 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 41F0716023E for ; Fri, 6 Oct 2023 19:53:28 +0000 (UTC) X-FDA: 81316086096.02.4897938 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 9BF542002C for ; Fri, 6 Oct 2023 19:53:26 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=G0jTXjL1; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622006; a=rsa-sha256; cv=none; b=KX3Ad38W+bfOs0vRmryXV7VHTgkjDXlFl35HcJtozY4Mo5vkiKLwGF0QZ/uUxLAMfNwZ0N XTPG/kPXZqj1WSgOE/Q0uJQFGYXggu3CmSunafEdY97JvcFY721DcmacII9fSktyd7mMkI GFKqDyJ0rO9WvyJeSIwzpDcFdjBXuH0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=G0jTXjL1; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NX7G4kfRjH/776I2tCnKq/UfkE/87x2ty/BZzBX3VcM=; b=UpRDXMSzxyp/GNQDY2qwHZRYaAmdpdmZ06xcONGxKuGGKNReJVngUPqKb9c8JGmdAnzeF5 z4BnUzlHSmSwK10kki1B8H9KIQn4ioPWIzPM7sT5Qe8j9fIT7X9jw5fqXYiQ/9KFmvVU86 NYYv5Q4IlGT7HI+NvQYKQY6wCrdNpi4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NX7G4kfRjH/776I2tCnKq/UfkE/87x2ty/BZzBX3VcM=; b=G0jTXjL1c6LMjaqYnahWKeF0jZ dS4UZQjFIGJaobWM6x2L5lg6IU7Q8rbAZAmE6pj6oe+UlcFNjEaM5+rAoACm9N4j8UFoIAA5K0UJ4 GzRFfR7lczEorH3MJI/Uk54ux9XRSVpBS7Ny1RaNA82OTpR7hbhCeFURiJ7Wb7EhHb0jAqUJqUwnQ SzA+GG7dMCTyaAeJR+GbkL/6uD9We3EfChbKldlYtCTO+wH5Xa9UfPfO++WH2roaZZFoRB/9SfiWQ ttqDJU13zRXCzmOGBgOvgTsR5lGf07q2zSdKGVT3yEmqpnWKkSjv4sG9zvk/vta2qiuzwHzpjONTe rID602TQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9GS-H3; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 1/6] mm: Make lock_folio_maybe_drop_mmap() VMA lock aware Date: Fri, 6 Oct 2023 20:53:13 +0100 Message-Id: <20231006195318.4087158-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9BF542002C X-Stat-Signature: 68z53qpemekwy68w7e4m8m4nsp7un1nb X-Rspam-User: X-HE-Tag: 1696622006-725452 X-HE-Meta: U2FsdGVkX18ezSBgFaBeVxgZn02auT5dRkJGFFCjJN2FPsy+poy48A3G+PHwpyoElOUIleWmEd+tI5lhnJwR0MVTipf2qKfN+A854lwkDrb/2zOS7p8HVQiFGUnw/e4DvT1tpqQFzFUhe8noY0BzJR7ruKiInywu4uTpFdXezf3BDSKb6R75w86QMjiigITz1VGXt5bueMCstsje0VX0gKYKvpBVXiNIP3aC+wrmXZ6uzcxloNjfptZK3fz3uHIqOOHbvpyo7279NWNIIilLehJeYxmU6IphGO4OGurlRkT7raV2AjRmo3afobkJ8B7gaXDWmXlhBHcARtfYzQiJMGoqkmdilS2S380jB9TOEiaijn1IVpFv/B886bGukH9NYpXlEDcZ9hyaw943wXa+8MmTSAl4BOQHvbUKkDmER9G7L8/cwCw8iUhMB/kUiT0Bxuz+qZsQ7Juo/1zzjp0bdb6ucg+rPNSA7LzV08nySW9umJqBXA1Rh0q6J4gctidOXKXEDeT6iydrkjYK8WxI788WtY5JunmgsL5TJVTo8DPo2O5fq7SQOfBEXw2ypLw4YrQT7IRz0jORparjZw7U7LbgFyHIJnW+6MX0kVo3oP3BH9GFyN5sX+7vmoOI9hIemATkOFppBqzw5DShbHPZvCKi7PyS4VIyRb2scj958eFrYgRIa8dWU0gRonQ69DyNEEza5Fb3otkWdBnkQkQpt2XvK2egcU6d0WqxDNVTk+MHr8wVchJB+r2NFvkNkl6ocVQv2SD+4G79nb34JnMsvQZcF4kS5QquOP3E8183lWwE5LvaN68FcknmHNsa9Z2hEYerlXZOdHXK9txEXINI2VkzAqlDeyFo3TgKitofgZUGtd0PqAfnibUG3Fy8XsW2LIYQcjsKGi8W1dVmv0C6q0Q4nqfVdaxIllMeeM0X1iBp4h/rEJv+cMm44yZ3otSksfZMjx2oI9HOX+Qp2j8 ubcctchF oiv71wZqyJZ/oqWJSAm0FSRqOLlLM6RkHoLAGvJzS1DDV8pjP65SFNy7KzM8Oqloe3tCdG8R5lMKELMFUFRyTLrWv4cgMgrMv8+FjNh7DgTNzVNFWGoiZCahW1pS8C7Z40/+Yq7xVCOpFzF5mNKVV3tZEF5gTEeDNKEtTewFGm7zNFWpO1HMH1GcQvJ20N6pkmNODtVGlWdyi1/oaKYaWcjGwSODQFRVeiDIH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Drop the VMA lock instead of the mmap_lock if that's the one which is held. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/filemap.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 9481ffaf24e6..a598872d62cc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3104,7 +3104,7 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, /* * NOTE! This will make us return with VM_FAULT_RETRY, but with - * the mmap_lock still held. That's how FAULT_FLAG_RETRY_NOWAIT + * the fault lock still held. That's how FAULT_FLAG_RETRY_NOWAIT * is supposed to work. We have way too many special cases.. */ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) @@ -3114,13 +3114,14 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, if (vmf->flags & FAULT_FLAG_KILLABLE) { if (__folio_lock_killable(folio)) { /* - * We didn't have the right flags to drop the mmap_lock, - * but all fault_handlers only check for fatal signals - * if we return VM_FAULT_RETRY, so we need to drop the - * mmap_lock here and return 0 if we don't have a fpin. + * We didn't have the right flags to drop the + * fault lock, but all fault_handlers only check + * for fatal signals if we return VM_FAULT_RETRY, + * so we need to drop the fault lock here and + * return 0 if we don't have a fpin. */ if (*fpin == NULL) - mmap_read_unlock(vmf->vma->vm_mm); + release_fault_lock(vmf); return 0; } } else From patchwork Fri Oct 6 19:53:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8256E94108 for ; Fri, 6 Oct 2023 19:53:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A22E80014; Fri, 6 Oct 2023 15:53:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02AA180008; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBFE880014; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CFEA880008 for ; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A708D1CA565 for ; Fri, 6 Oct 2023 19:53:36 +0000 (UTC) X-FDA: 81316086432.05.A63B616 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 11C1510000C for ; Fri, 6 Oct 2023 19:53:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WPOQ3j8I; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=nv6NCqFy8QgnoeH1EF/zjdFU4qb08q0NwkCOE07Cc7eZ6FfkoovgdBlSApgb8zH10N6G5V LKhA0q6uD3/5Bau1si2tj+dvwhWsw5MvPDgeNH6uYLWx8UOKrQaNFYuqLHQLKVFz1Bh8Ib DwEOvfNYwlRkZdriQxIX5JJOaDAW05k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622015; a=rsa-sha256; cv=none; b=hvinOGCMxXTLYK29YaePHbzZWNLT40HWkQI4Cz20/iV2kt82HqaYEHsvjE94fEYn3mQCaP MFY/q+xGwiwbXfeE+EybFnOBZF1jXeLnPO4U2lKMKSr34WzCt9u+Y5PZWsKVkk97DFMX4Q CGuYHCpDCTO4g3bGbAXkhxN+gfa34hc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WPOQ3j8I; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=WPOQ3j8I8UN/sxknZsBmPVtDO5 IqHWplA1iyqL2NR3MriNYZfRR9AujPlQQklM1j9pLdVTha+sM9wvd/FIH3kbf3uxsxKAJUtTilBDX MPzpKfgfEWrJIQeogojlkHmYPvJw+QDS45N5K0mhvoWLPgY/P0HVwZMoaXVt7ocIQVC4CjD+GZXGH z2qUmJITqTRgno+Ls54OudvQCYf3kNuaDOu+yODUiq+CLt7VE7ZLOW0a2poIrIb19gWEoSBbKzcbH XyrLj0nYgP+dv9M3Pco1wL4EQuJJ2ORGUt7VRVq4rctpnilvWFvFJ64xER+UaIR59QlK28IJEZNu2 WjVwNQwQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9GU-Jj; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 2/6] mm: Call wp_page_copy() under the VMA lock Date: Fri, 6 Oct 2023 20:53:14 +0100 Message-Id: <20231006195318.4087158-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: y5gdj3hywzutfkoix78dnyesiqywmme5 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 11C1510000C X-Rspam-User: X-HE-Tag: 1696622014-479231 X-HE-Meta: U2FsdGVkX1+EPBy7+comaT1r8/qqphTy47ne/WepgpQZC3dZBDZURGd7POIpMejhCZUDVecW0UWSMTo80kCF39ILo6TgTmVy1ffr8DPA3hmP0fFTYl/zropqDtAfG3Tw8UcfPAJOIZ/COWPQ/io1Qe+2xe6wwR+uNwpDFJj+cM6PhvydaIMAFqPaMFXbXJGpVEyNP1TwzTDqjxoaOwD2u2lJ0gT5T4+2CHfuO/yjDZ33oYsIDY5n6OmoOS1kHT0+LQJbqAEzDfehAIg8dfz3sk6KXUDTwjWEkkjvSpyaIQ3zFDCjMouwmMECgr/uqeL9Gju8wRYops3kbYqY4aSaHAGhoE8NjVBDwmnwHZVdakiPQ6l3+ck6/jxzXGl733aaehMVwqmYrLmPHyKVNgFvVu6mTohPJLyAsLGVwmmuk29rLy1KP+7So8NHswZ+0Am7Ee+fgKnxeubj2nFn8StbAj08nb6emMt9UoefRLFVFuHBeQAk6HPlA/sPGK3BQREZBQz2BWkwMiAY7LvPctTz1ye+igX0rFKqAzWDX/xPbELKfNHkDePVZZLqyKYZQse6503fmgdraSD8onT4mMmk+5k0txEeFJ/0VgwSE/TLzf8ZFAUlMG/P92beb4mAXZShqRXGKxabZ5vle1rwTbVBs8uOUWnX83mDQ27TbIEpnV81wYJjkC8t93fsKJXnHC8ZcFvDxTEh2GRop0slSH6ylS5oKdMJbxnOhLe6MsfVaw1CiaqD7oNYtA5uilqK26pxoFAjNxcS6wLjcciRKpkuMVb1ldQZHrHawmrlLTOh1ThGbZUFBgRdm96PWx8SpWZkSgHy1N/Lc/V3aVE13iTKnRKeI28j1LwsPom5i+5S5dJQFgNQ5+zwo6auFxAIxHYoXF6O05dy0jerRUccdFT4vN7G1tfTciu6XD209wN90CcmssGV/YXUQgXUyPY8Rp777X+gMmSheSoSmTLCzI+ j3qZzor+ HTzmMEGUBZm2DNun1gPsAccg/HtmkemSfrWbCVHuvWboHlGHVwK3Rg+uYa0OsQq/YrxC9CA/sFPEkHAVkx0y0At2gjnyTi0xV8py6LBxYhHP3PadfYC05QYCBpaUzoALgxoRd1tXJIxfh2vljyo2gu/fh7DbNCk8NRDc19h7qAPFLBmcfEBDBRdCTAaX4HlkvJvpv4S6PH9UV2Aw8e74V4/7wY01VegxPj12w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is usually safe to call wp_page_copy() under the VMA lock. The only unsafe situation is when no anon_vma has been allocated for this VMA, and we have to look at adjacent VMAs to determine if their anon_vma can be shared. Since this happens only for the first COW of a page in this VMA, the majority of calls to wp_page_copy() do not need to fall back to the mmap_sem. Add vmf_anon_prepare() as an alternative to anon_vma_prepare() which will return RETRY if we currently hold the VMA lock and need to allocate an anon_vma. This lets us drop the check in do_wp_page(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 97f860d6cd2a..cff78c496728 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (likely(vma->anon_vma)) + return 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + return 0; +} + /* * Handle the case of a page which we actually need to copy to a new page, * either due to COW or unsharing. @@ -3069,27 +3084,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; - int ret; + vm_fault_t ret; delayacct_wpcopy_start(); if (vmf->page) old_folio = page_folio(vmf->page); - if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); if (!new_folio) goto oom; } else { + int err; new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (!new_folio) goto oom; - ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); - if (ret) { + err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); + if (err) { /* * COW failed, if the fault was solved by other, * it's fine. If not, userspace would re-fault on @@ -3102,7 +3119,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(old_folio); delayacct_wpcopy_end(); - return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; + return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } kmsan_copy_page_meta(&new_folio->page, vmf->page); } @@ -3212,11 +3229,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) oom_free_new: folio_put(new_folio); oom: + ret = VM_FAULT_OOM; +out: if (old_folio) folio_put(old_folio); delayacct_wpcopy_end(); - return VM_FAULT_OOM; + return ret; } /** @@ -3458,12 +3477,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return 0; } copy: - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Ok, we need to copy. Oh, well.. */ From patchwork Fri Oct 6 19:53:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 969EFE94104 for ; Fri, 6 Oct 2023 19:53:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EC3280016; Fri, 6 Oct 2023 15:53:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 274E580008; Fri, 6 Oct 2023 15:53:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13C1A80016; Fri, 6 Oct 2023 15:53:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0332C80008 for ; Fri, 6 Oct 2023 15:53:43 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D16141602A4 for ; Fri, 6 Oct 2023 19:53:42 +0000 (UTC) X-FDA: 81316086684.13.815994E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 3916BC0011 for ; Fri, 6 Oct 2023 19:53:40 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=up0u2uMR; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ydoOqfQy5A7MAdCeQYAn0mIGPLOFD23QKtcFfuFHsNs=; b=ZqUu7/wH2Oo/Zm5PYUHWX7HMHVpAVv87LZKZjAJqRbrnMk9O35Nk83b4417zU/M1n30qeB U41kHnW+m2DuDJ2g4L4kD+9kJu836YpXINi38xypW/oD9gQLxzVShGlDxPKiR9C4FyPhCo W8yRUjAHj9SmGYzZWq6Qmrm4jE5FUow= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=up0u2uMR; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622021; a=rsa-sha256; cv=none; b=4nfxpt2rQnR7q9Pbil7ZOEn9ugn2UnywB82hBK326Q0BZkOQ+K0W9PoJVrRItJPGQTUiPo 9AcceoJ8YeBR35qVXnmtf02Gz7B6OBC2FQ/1X45ttV2NnPFZz5rCM65eggX9CRSWeRlOFF ieBC3t7P0fGUpOoAp8L8EbbtHKx3b2U= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ydoOqfQy5A7MAdCeQYAn0mIGPLOFD23QKtcFfuFHsNs=; b=up0u2uMR3as+tpovtVU2cjcNoI mcKSM3xaxoUzO6/k+YPPB+JEKIKGX/r0Laevwp8oOslgXCOByoOluj0Ihg3B/O2A+QuAiNq8S6Qm7 8nBk4i1sWoRIptn6J15/aNCpIMM4sADUvaZ5vNQExzroC6RpzHZ2Rk6UCsY6JYwPm4yKKFSqRLa1w IlLTfHmxFwUF2Ps1suUQ+tLZ/2dcAUe/1DOThSJE8Pe00lsLfYzvOR4FAC4MbmrnTzzrYYkaEpWxT T09LJEH7CgnjhYX4hLj5cBufAwAsch3s7HSSDBXkAzRPQ9dAIgkfT12ezxl1Mc+N13IxpMhj/m4qX 3AW0QbPg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9GX-MT; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 3/6] mm: Handle shared faults under the VMA lock Date: Fri, 6 Oct 2023 20:53:15 +0100 Message-Id: <20231006195318.4087158-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3916BC0011 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: x5tkguh1sphqqirzixgidiiod3str51i X-HE-Tag: 1696622020-634666 X-HE-Meta: U2FsdGVkX19G8scMF4uNiKk9LNxhtvxZu7cNIREdcwNzQkB5QcFiqmIP/QYOYHKkqLXWD+WjhlTDn60DKs/HDgRJ+rywko3OiyrWmlnnE5T0vUIsJZG+RealfLZkj/YTse2fgFKHm5dWJ6uOZp874vI4/PdcNoeUFHHA5W+WmHzwmky33japu3tKShAWyiGsuUve/++ukzUMq2+cIroJjB0daqf2eDS6+nR9MkVs3t0x8WFINLEbrSXJQOTuGG1jgWa4O4huvW+In2eR5NOaXFx7PhX7JIgzwDTNXQmZ+nk4oR9bL1tr7JpdZ0EvVZKBbReqknXphQjV/mAsENcPDL8WRXA1BZ8wvruSkwK3qqkwuCjqBKy4tQPSHsPzoR9FK2B5CzN5ODnEac7tdNX9sO7IpvFcFVOY6ak9zwP3KVY+FIP6nzq8XYo4bN8Zip3D2pTFp34z3IfFmVRzanPaY5oCEvpKqN6raUcDTmTA43ctcOvhaOEjvE7boCRNpfHoBQSWYYLf2smgk5jC1EVYk1+RES/ZysnCXUeSpIx6fhyEOcgs/RIwUS2KF5OIgL/UEyv0sWzLhjX0EacJP0QnlfCaPc6dQqcOX+hwWnKU1FD2SBcDWjIBBYBy7CulfPTJsmXllTSV/4zL4rW9ehYcuz67R7aRIL1/kVGqAK01+MBoa4CrtIQs0T7LVO8TygjIboO2dtQbszmGydw4jwpHkAw5d14Gs9buXs3YpWQixVo5jCcR2iZgMg5MZAnILUoTlZOZF1QyNRwf85ThcjKdPoZXOY+hG8xG/Yd32UkejGJrz958f0lpMaINvGSBHi+jGYNPuoxb3Y1Jwb/I6z2LX9g466WrLVCN6KZLMDykdkU4sEjhOuo1vgW13VonrxBFm6cQiHU5h7UnI0WhxUDopjJxq0djX5pwUayjl+NJHtqiP08i/XizmyAqyYfKOR5G0ZYoAWiKNgrOgo+/TGm IyAJgW8f jBhItDaZxokVVfMl5wO5LsTjKntERPIy8/o1XUx8oK7pqHEHV5fE2xNp2LAiKqFTTBx2E8uoX0gYbK8MGjKNsXpe9LwUSiTWDdbvRuv3UhyItk3md4W0sc6WNT0SrnThsG9hpjeUhbLVvPY2+el3sGb/4tkyJgOcP6VC9F6ttcB/Q9kei5KNehm1g6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are many implementations of ->fault and some of them depend on mmap_lock being held. All vm_ops that implement ->map_pages() end up calling filemap_fault(), which I have audited to be sure it does not rely on mmap_lock. So (for now) key off ->map_pages existing as a flag to indicate that it's safe to call ->fault while only holding the vma lock. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index cff78c496728..a9b0c135209a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +/* + * We could add a bitflag somewhere, but for now, we know that all + * vm_ops that have a ->map_pages have been audited and don't need + * the mmap_lock to be held. + */ +static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK)) + return 0; + vma_end_read(vma); + return VM_FAULT_RETRY; +} + static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -4669,10 +4684,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) vm_fault_t ret, tmp; struct folio *folio; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } + ret = vmf_can_call_fault(vmf); + if (ret) + return ret; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) From patchwork Fri Oct 6 19:53:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 119BFE94107 for ; Fri, 6 Oct 2023 19:53:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D15680012; Fri, 6 Oct 2023 15:53:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5849580008; Fri, 6 Oct 2023 15:53:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FB6A80012; Fri, 6 Oct 2023 15:53:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 326B680008 for ; Fri, 6 Oct 2023 15:53:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 03C51120453 for ; Fri, 6 Oct 2023 19:53:33 +0000 (UTC) X-FDA: 81316086348.06.628EB4F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 5C610C0008 for ; Fri, 6 Oct 2023 19:53:32 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JQS2xDtf; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622012; a=rsa-sha256; cv=none; b=u1n2pcGvJTILbURKZ3Q5SDGoUeMZ6Wuf+YKJgo/dpjAJMgH22S4zkMtcGRMi8sY0Phu4NU rBXphRcRbhrzIC0pMmhsIbZ1v3lJBn8LavWdyxQ+sp2WNKYLzAtdLL7WQZoh4ak2tL0oTl ZSXPnf6u1JcD0ur+WnYw4ALxNLE31OU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JQS2xDtf; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pCVcNzH6myRez0yNN5euVSe8WBMi6PtlgZsJ2oeW4+8=; b=t4GkfAHvd6+wAAan7l9XwKS1h3Y5rZEhiWhYmk8M3VOzCbu7040tPYWr5simNIrwKDL1w4 jBJ7p0w70OzajZz9zaCycpKDOL7OujCtJ2xWSrwFpkcC+Bld/L3/+3dumOy3xDUO5w0JSP F8NabMGbgc4xhYHjR6+RBHpWNwb5EK8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pCVcNzH6myRez0yNN5euVSe8WBMi6PtlgZsJ2oeW4+8=; b=JQS2xDtfi0C/mqngYZah8k/Ev6 GoyRy9VcRubAdE7iggIgTkOrt0SUy4GRGLOq8+ut+y47vyTh8iT+tDhI8T+8ZifCEMjPr1SI8tWSm K2lFSs+EfSz0huOMW1SSO65kAnT9LqQ3X1l9RGtno54d9pPt7n2zxTCuUblUPUwCf2tdxbJzvazwp ffWpbWkCiUrCXoEBjR9nC4ZM/RUpU9TNcJo9QXtMQgZjUu1FCx7G0R5ejO3vL0yS3JYpsknxFV8tu Zz4NFPfmd4UpT574IirCHIfZQXeE1vq9m2m8f619w30oMHtvl3R5OaR7U5lrT6nbzBqLYenit0tCv c8Uw+DAg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9Gf-Ph; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 4/6] mm: Handle COW faults under the VMA lock Date: Fri, 6 Oct 2023 20:53:16 +0100 Message-Id: <20231006195318.4087158-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5C610C0008 X-Stat-Signature: m4qhj654kdb5x5oagtcme1n86dan4ma3 X-HE-Tag: 1696622012-240891 X-HE-Meta: U2FsdGVkX19HdRtElymOTLqUwoYyPSp/CrE73jbUPnglwmXptjgKhxuNg8+GCVPneJjW4iaH/1cqqnGHE1ligeiGXTmcwLcwI0Mk7MsPd+cjY+9XmVvjR48wQV+Mz5o/v+29cnGorwIDi4i1lmt2DSvRJu8RJpp16v+AJVEoIuV+ID+/cqitzfwek6F5sIeO+5AvfE8UsdFVkChR+nPfQUMlmItoFfS7YhgObbtTtNCFHHHYcxIs9Ar6TavTUFuXblNtygCFFG45KfyTBWV2esc87/dmUT/5COYf79JTwjc6iBC/ZG6Y4T7ETRLrEpnvbLqJdhL2eDpSckXJgTCaa3JYEU9yay4bSxzWbwX+Y9s6PJSdyYuX9n6Hk7Qx7obQHt/+awXEBA0NOcDkPwIyOBpnR+MS/MOaSD3FMVd3g+q3UpfqcXY3wqWg+vX597VXdlIkx+92tPCO7r5iO1zRtlOd5uhFZ9Hh8sWvikIiOvbcLplcZMiRqIeuyxhwshIFEZvcyUPSfMKpcmLtrAcVM0+kLVRfWBvZPhN1i92x2lAEeihS57xO8BaDDgZYMkh5pIH6svIGDlwVM+rJ3gUn7lMEmv7/kiYoaqrX4e0UOwm6wq0/m5pWnP6OpQfpHRU05vSptaL86gUHvYdjbaLJ6zjbC9Bc7/+fuIGNhlfMXyK09vacRLOVMnSCoM6UJI5RsxJVMIdWTvokboAnR7vjsQLz7Jfu7EYmNmmsiisACoyprgDua1593o1EHKMcWLGO1RGtJyaZoX+zgBaajHRlIYYqOqFSxc2yN33wqMu4gvVuETiTNY5ebO2XtibUSFTujHB1fBKIaiqvFs30Ax0SNO9nS2R0CGWD9XbC5Cp9K/ulTTrKY0eB8sM2f75G5/5SV69BLtOHgJZSf8uTUw/3M9doVth5pH/XFof72zdZ420ZU9XcZAah0keSuKiUiR3VlKFq7h7xzxwrLwRmW60 yYrQUQsz 1f1hGvcxIe5Su0ykSMsUiN4KlG5XbYIAFl3SIdpGh5Ccu1ry74JgPqIaP50apd2pNJ5axTkwiVvB3F+CH8gGsRcXxlrbihth3vEJtfGvAtBAQEc8V7zLRW8NcUGR7WZLrxSi+uUrql9G7h0oyKkhyXJJNRK9jDy62zNwn4cBfdmZzFZL2MgL0PaVUWA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the page is not currently present in the page tables, we need to call the page fault handler to find out which page we're supposed to COW, so we need to both check that there is already an anon_vma and that the fault handler doesn't need the mmap_lock. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a9b0c135209a..938f481df0ab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4639,13 +4639,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - - if (unlikely(anon_vma_prepare(vma))) - return VM_FAULT_OOM; + ret = vmf_can_call_fault(vmf); + if (!ret) + ret = vmf_anon_prepare(vmf); + if (ret) + return ret; vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!vmf->cow_page) From patchwork Fri Oct 6 19:53:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56FB6E94107 for ; Fri, 6 Oct 2023 19:53:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E2648000F; Fri, 6 Oct 2023 15:53:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 692FE80008; Fri, 6 Oct 2023 15:53:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 581218000F; Fri, 6 Oct 2023 15:53:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 48E8080008 for ; Fri, 6 Oct 2023 15:53:26 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1CFF216023E for ; Fri, 6 Oct 2023 19:53:26 +0000 (UTC) X-FDA: 81316086012.28.C5765B7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id C581FC0029 for ; Fri, 6 Oct 2023 19:53:23 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=saME0q+j; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622004; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KGw4Os6nnxvFPBRCSbndQ6uNf8PmBQJxLw366sMUW34=; b=MNM/uaiErutAPcaCUupbOFelmXEj2J7Ds3COnMfKMnnMggOuz17vX6CkvSHiA5SF19ripU B2EXTIDxoBl/jQwajnRkQVsqwPoS/scVOyx7Wgc2FKqiGTSMlSu4FDFPt2OCZQ5WJby6+w 3MatLhKUGzN3GQdlRlpMjr+xSwQBkoI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622004; a=rsa-sha256; cv=none; b=eisREseUGIH9m+Sf1rRSbXMXQlNj9KdVfKCAzLOpoKmQwhsEy60Ol9pV7dpiqQEp9yNTAw JQsBbkrLY5++fJ0LbM93X2Cgk08RRLVovfpRGJN9daWzJf2bZ/WSenXBuyRy/R/SSCCQau gieUw9Z2PKy4DjjtJ+jQcRFrn+vZLnc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=saME0q+j; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=KGw4Os6nnxvFPBRCSbndQ6uNf8PmBQJxLw366sMUW34=; b=saME0q+jjAQaNw8vi6kTdgrnxx vKQOnRzPU0ZyodLmCTmaLVdUOfCPWoDnM0PuVWgJUPh6yt960B+8HjRrR0Mg7MAjVGCPw7AbzUNVD HIjihQx7Ct+JYW5UDw4v8JAWjC26pAVRhnT3pL1oPapO8XhDxEqB+lA/k4RYS+iflD7zxnupV60g0 X4N8sV0RIo5KWgSDqOCHotgH6IqEn9a9hWlA/rX+mR9wQhAB5zJUogek7N8n2OSBjxnInY2zsNs7o XyYNOPc2XC8W80G7VCHb3fVAJK2FRSPTZ4L7UvzizkhQPbqVznwfFoiozPS13186xBsCWUIay0aam rJLZ3Mvg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9Gj-SA; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 5/6] mm: Handle read faults under the VMA lock Date: Fri, 6 Oct 2023 20:53:17 +0100 Message-Id: <20231006195318.4087158-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C581FC0029 X-Rspam-User: X-Stat-Signature: 5s4yj4z8xibppjfetc8rruc7q1kw1ipm X-Rspamd-Server: rspam03 X-HE-Tag: 1696622003-411671 X-HE-Meta: U2FsdGVkX18XZVhicxh9lP/hd0O1v5efm7KjQ5kfl2zFtb10sDFtT+x5yi7CbWG59uyJ8mDGBfrORcxKEejvm7HlBlaFYBqm5arhZjYV1AyvauemqCGw0XUdW0GNdIe7i7varnok5n7HUUyHSKwb9n961u6wtIhbQBQXGtPaxFYaDRIxw9nUm6bTICjYe5O8sR7abK6g+xlblO3EkCYKZXfpqSG8xF240zh7Ph1E7MpV+D7hXPKCaSM02K8KXh2TJ9d2JJ2hLoB6Jn4bpb0J5AjH5xuRzxJJb4+UoO7sM/kNiwjIjswrhZs3mQy6zKJPEa6g4h2ZMYmSNIVYt1bExp5ooSdcvOgn25Jc4u94r1MtXha5+dlm7tOWJM7f0RCysJlM2fkvfiOiekzmHkuYfIf53c9tpW90eI2WBQOHpe1tidgakGZEAfALorHHTplTkFNVFZ3oZqJGt0aVM7qugtnyziKaMIHJJPKOjc1cbe63ovCAaVqnV8cyO5e0Q5MUKhY46beM1xivzUAXudsI6csP8BdKe4wyA2BCdnqSIC3Y22DPFgQRXVSJf+QCzhP9FmwUdeHF/J8SlK0ARHVtXX/hKUpSPvfZE+gdB3pj8BsFw9xejEeJyRu5+SOprafEA7nbJYa9wcXW+M0ZK4XL9KJdmEpzxr/DM9eOsv7lIYb+HR/mdxGwJDfnsUzHqtnpLbT6rqSeQsUn9AyMoR3q0eFqVl6gyXfsBB7ZPdGrnCyH2fPclgxhpgy64ziSVnHxiOpRmEnQxC4amJZJiNHP25NfykZAaOVvM1dL91ytn4fEgpiAggldwIzgeh/VwRy89Uv+gfovKbByx15uclzBffJUOSOmqLiD4ZWAfkPHkO/AeqnWwxcDqypdzg9PrFOk+K3kG7MMD7KhfulTDkx5NHe5VD7sNRuMWigTOys0VOb+UWesSm+MWxgBQf+SoI8XfOdZzn6FJixUb2DnQzZ TTjyWIhW fZrhPlDg8DOgC50e5yJGqVec3oZh2r/l15Qw1ohoaZHeQlgb/z7nH9xkAAFsWwU1IC1mliyO6knwZDosdw8HU7Mrm0hVuTdakB5ED6gwdeWV+4ZXYjL0aUAq+Hfdrexkui2LmO03DI7/D5V+5okm6DRqgX/lpDXMvJhk76wPo1EFTHUazjDm1lyl/xQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most file-backed faults are already handled through ->map_pages(), but if we need to do I/O we'll come this way. Since filemap_fault() is now safe to be called under the VMA lock, we can handle these faults under the VMA lock now. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 938f481df0ab..e615afd28db2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4617,10 +4617,9 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) return ret; } - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } + ret = vmf_can_call_fault(vmf); + if (ret) + return ret; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) From patchwork Fri Oct 6 19:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E7CCE94107 for ; Fri, 6 Oct 2023 19:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07BD680011; Fri, 6 Oct 2023 15:53:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02B6880008; Fri, 6 Oct 2023 15:53:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5BC980011; Fri, 6 Oct 2023 15:53:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D96E980008 for ; Fri, 6 Oct 2023 15:53:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B25631CA4D9 for ; Fri, 6 Oct 2023 19:53:30 +0000 (UTC) X-FDA: 81316086180.03.A3D0786 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 1F256A001F for ; Fri, 6 Oct 2023 19:53:28 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QVcBvqic; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7oiMnjIM4hSkmqR8ZCJCtaNuaV9dY5uCykUhLQHJ5E0=; b=VYLJQkAObk35EnRHP/tz1gGamwuccyIFVfpZnF0dHfLvF70X4RWxMgP1ywRnRj3y2WrUkJ n/t0S548dKRtKqNBKiibkCl6TERVq/y2axV1VhxFM/0MEiJHMOuhb6E4XgWq8U9Y/uT6uk 4JyWkhhIJdZnlhoeGGjgZNkd4G7ew1M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622009; a=rsa-sha256; cv=none; b=B25agC9mj1ZEp3BQdzMys969pd6AaHlaN5j4sEddaAkRpHE7rXHRwj+WWdfIppPqb/1l0h qw0q1GfuRv4wfwQTkQQXG4OGbLeO18SzSqfKOWY4g4tIlm1jspsv82//7s+jqp2U1+OLHU 7GSY9OmzpsoBpRZBlLekL2FjXg9NA90= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QVcBvqic; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7oiMnjIM4hSkmqR8ZCJCtaNuaV9dY5uCykUhLQHJ5E0=; b=QVcBvqicFZyjYIraMPx3PklnEs dhlC6U2v9hVGbzTBNfv3ToBcyaQEaPBbJDz2y5wWxjAcGWdxDjODyn9qFTwPtFXEkBkMuxPqs1cQ+ i3I34tIp/qFDGliRowh2H5uZx8JLHfG6g3s4OHxVXPF4+nhPerhazD3LAROmh6JXKTwH/K9fDyZ00 HgVz+l7bvQgVFk8na2jjWsflWNKoR3BfUSn/eD2iQeTC70Uyq4WomaWVFvQ2N3n4tO6rRiTg8EtrC 3WRRXNlR3YHQaXzIISoAFftQ8BJ/ooDxmnJieU095v3BQMBk2+HdI3T54tjGUncozkclAFKt5yC9Q V+z7xxvw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9Gq-VH; Fri, 06 Oct 2023 19:53:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 6/6] mm: Handle write faults to RO pages under the VMA lock Date: Fri, 6 Oct 2023 20:53:18 +0100 Message-Id: <20231006195318.4087158-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 7xiw9gx3odyhy4hgtcjz5d8k7y3q5px3 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1F256A001F X-Rspam-User: X-HE-Tag: 1696622008-48554 X-HE-Meta: U2FsdGVkX1/+dWC9vc8h6E6XAslT9mnGrbf3ZS1KgBpnudjlvt6SMa1i6RW/i4V6kvXsD/CKzz3bY5p8ZDmDjCeZhrDPqIQNpRAisPzWa2lCQ4ETlLQVlBQxhpBtF84ecHUO8b8Lh3okLPYpuJmr12hsqTEjMbabWt+OpbQ7m0L5wgufgIUYp4Pvhk+gvDfuKXNIXRYJv+GhfQKq+96xyGY/BXqJNUCn+MGQWttev8a9doseKJUhz1C0Nf00ER+t/L3GeC1ljd0tYpPRpxOjsfKgibbwdhdtW+PrIYpPYUf3UbwSAOWOg39/tD5gkcqjZeSwCo/1z9W9R39xESXopTubZmd2EcQ5wm5jdMMrmqwh5Mng//L65oxToOqI7PoZ4AF66Ost/3mZMk5kpZ0syTuujVUmkz/AidY0KJQo5Pa8HYSK8H7yqiU411iZ61RxWZR5reVE4lGMnjLSW/i53jyGP3Jz6HROpl15kFiUkwRsGCt1uQcnlvxeSLf/iHRnmCrjBwU795YLEmv/BWtxMPkasdDpCnm1McAKNxSzwLWS0fPlX7HGasV1WFNx+6rBrzP1yJCf0hLxLoIbh13CPbgejyvUIndtGhQkDs1lZm9bISvMrLZCPUyK4qWzA7pN19xtUWG2GFJnpZPbND2m7mOYE49ANjfH6bn/MuiXFUFldOmwzq34ctsHr18WCVpQYaFm5gvCCw5Sk+Rfu85p3Ub7JanbttPxADrPDW6E4bmki/Xw1U7QBI6SkvRDiTBHldQ2z1rqeL9IrD4Fxo6z7y9xz6eGggyoOfyUKUk1U8WoM94wkQyBUicBBwkxs+1UO5cHzoe1Wez0qwBrCuBKDJP9pgXxfYxoOXA2eZfKnKzvr/qHh2licqILIOwKpzeCGXAheJexYvBZFpMdglYjZQmj77e8cmOzITaX5hQpRhdzPq3TpNKodSZPgoq7Pc5AmgF6FThrIPxfJEomEMK +xQkSBsm spoBdMuBCt6Dxc1pwroYGC6nYw121xy6rw3nbEAPg90mAWE89HRyfDHZKGecT8gZROYJTxJObgGvXP31UjTBgesCK4AtwssXnBBxMWo3VW/urj7Bsec3BJRL0pUhIyL00ll6ZGunfkfzOfrEh7ki40YUDwgzEDKj9mHTKew/tf8VGLE4Jd4hef1emMw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I think this is a pretty rare occurrence, but for consistency handle faults with the VMA lock held the same way that we handle other faults with the VMA lock held. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e615afd28db2..3d1bc622e344 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3301,10 +3301,9 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) vm_fault_t ret; pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } + ret = vmf_can_call_fault(vmf); + if (ret) + return ret; vmf->flags |= FAULT_FLAG_MKWRITE; ret = vma->vm_ops->pfn_mkwrite(vmf); @@ -3328,10 +3327,10 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) vm_fault_t tmp; pte_unmap_unlock(vmf->pte, vmf->ptl); - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + tmp = vmf_can_call_fault(vmf); + if (tmp) { folio_put(folio); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; + return tmp; } tmp = do_page_mkwrite(vmf, folio);