From patchwork Tue Jul 11 20:20:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B8B8C001B0 for ; Tue, 11 Jul 2023 20:21:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BFC68D0002; Tue, 11 Jul 2023 16:21:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 170168E0001; Tue, 11 Jul 2023 16:21:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 039078D0008; Tue, 11 Jul 2023 16:21:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E8A9D8D0002 for ; Tue, 11 Jul 2023 16:21:24 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BD1511A03E1 for ; Tue, 11 Jul 2023 20:21:24 +0000 (UTC) X-FDA: 81000450888.14.251F9EC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 06CFB18001E for ; Tue, 11 Jul 2023 20:21:22 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BzG7fNes; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106883; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5FN0HiT2a6i6oQBEDqgGmkCDTbMkD96qK8TmIcdDOvU=; b=PD6SVRDCuXntQj1Y5dguULSRfs6IT7bdiBeoWtd413aVcm9Cllyx5+ETrliD0lb+ypdAsH 9HZOKgVcxGFfoPHEw6PQRKYVCtqIogRwjnjrLsisEQZ7bci76ZbuncF08qVWpMmFMgRCwl F5IEnyTttf0Qwbf5uld1w2f79U5TXSM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BzG7fNes; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106883; a=rsa-sha256; cv=none; b=KlsU5fY5vdJIzgIkx+DuTJy9OkkTa8V3Ez5hm68eiNgyW1+L+dWrkiDL1BUmbuHgeyrewk 3JSUKU68BxXY0guZD6eRXCCgk524ZoQ8rPW+RqW888X/qiQq/k4Q/zSUlX4agHIfjOnOJV djw/khEcrM6IMx55A6nuOLGa5w9xrqc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5FN0HiT2a6i6oQBEDqgGmkCDTbMkD96qK8TmIcdDOvU=; b=BzG7fNesDSv9f5c/jHckIpGYxr 1AjjHU4Ah6jva8EK1Xcg1/biocRXvpql2nETap2r1zt63XApMDK8MNuoMrkEge7p433afITqveoC3 Lh9PyA4r1OYCcI8kixo3vPG870zrKJVBOEc45zm+IEQoWThCsGK0JCfwO5zglXVRuurCxIV49tMb6 tlkYbLY36M0L1CpBIWxZD89hARza9OQ7yNWvq3+9zcyasAmecs0SbqtEGsAY8bVkyE7Y/8n3U9YG8 hOj3abiKh2aB7siI996/oscxpg5i29HTOtetWXWuMHJK8XmiAJaslK8w9KiTK8fuihRlHUcdFaRBh GJRAiejQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqr-00G1Q6-EC; Tue, 11 Jul 2023 20:20:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 1/9] Revert "tcp: Use per-vma locking for receive zerocopy" Date: Tue, 11 Jul 2023 21:20:39 +0100 Message-Id: <20230711202047.3818697-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 8zyxmbrp1rzmbepjghnpasusz7unqgeq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 06CFB18001E X-HE-Tag: 1689106882-523944 X-HE-Meta: U2FsdGVkX18iGTSABpggNVIWWkFh6mMPldsGGa/8V3Qu+af4h///W8qAnil4iX+TS2XMOiy3wE76HahqFnvTz1HvPi+cyjns0J4Gv3Qx2bzZwMH772isa8B/xZnIAmiecpufiBB/fOPcIALdlMFFKCcdxXyBLCZhMM+Ji9jIj/UN8Q+LrWm0In6nRaGGXnXsyzx0milBOYtTOnywAi9iHvYHsS0R5DP1TLzjDRonUAnVFPQV030hgpFRnOK6z+o5lu0YI/cjIztE8KGXSRpvCrwwn8wemlpfx4MYfQJNNCsAKsAUnbXtYAcS184mU89U9WnpLGn0fVLIGzA0wwv0xaA5IkGFQHw+bwGUs3U8k9jc3IzSEl5ejEHZwyIDsx8JNwVf9A+FkWGwrq1jOd+bLlMC8mBP83GQ4Y5PHYY96w99m1WVa0cUw2niLy1PwIxG5+paNDvy2Qy//odin4KSZZMa6h70L240AGSHqViKcWJxlOQNY07nQrsa5Gpf1jTgekPEMu/hPWtcclFuiLB/sH4qbX+9YjJZG/O0V6x92cDB75SkyO96jhj0PzhR5HJcCxbsDgN+PvmqrrDKzw58YVFq4qs56tx7yR7P56zwCCLY//YCEfvIF4Vjx7CIAwVTiLFOwmVXBsuJ0lG3f9EWGpuGxQ5qY7xbXZ05zuafifZRJlbhkYANK8lPlIA43E5WnIvNaELJF4sKfE7JYrhPGLcYHvgTZOpwgIyAQgoh2sXklZycsEygPnBLt2kXa1YCwUAjs9hI6/s8XByUfNht/KafQ9zgaMy60uqJuQ92ZD8GUFmjjBCXgwykzGQ9DwzDjvInzeRETS3oat4OxCBXjhYrS+2mRw/pCJax0QtUBtobxOT3bhokLpKquMEVJewEKJuDxHGJPWsfRJNX1VeLUkWorc+FJ1JU6YUcX9cQpDmBOFptdrtlVOJ/nMoEKKHF603Y5fqs8TvK6GXBY1i 5WxKHOX0 rS7v8by8td07SGnsPPdZoWZrG2jZwUFZpYs7wxQlmii4m2L27Q2N43SBcexHfJby33WECcdajj5m5NzHIQS6xPJG/DaTun041ZHT5HYZcOEeBtpSMkGyV1Ml8V//gnGAyZVp1bt7aq7x2Vzz0vHdZvlcSmRLycnGyyvo5DP9onJAwJwXrIqi3Ll99GhLDIFuGHJi9WzbF2UESMjI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This reverts commit 7a7f094635349a7d0314364ad50bdeb770b6df4f. Reviewed-by: Suren Baghdasaryan --- MAINTAINERS | 1 - include/linux/net_mm.h | 17 ---------------- include/net/tcp.h | 1 - mm/memory.c | 7 +++---- net/ipv4/tcp.c | 45 ++++++++---------------------------------- 5 files changed, 11 insertions(+), 60 deletions(-) delete mode 100644 include/linux/net_mm.h diff --git a/MAINTAINERS b/MAINTAINERS index 18cd0ce2c7d2..00047800cff1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -14816,7 +14816,6 @@ NETWORKING [TCP] M: Eric Dumazet L: netdev@vger.kernel.org S: Maintained -F: include/linux/net_mm.h F: include/linux/tcp.h F: include/net/tcp.h F: include/trace/events/tcp.h diff --git a/include/linux/net_mm.h b/include/linux/net_mm.h deleted file mode 100644 index b298998bd5a0..000000000000 --- a/include/linux/net_mm.h +++ /dev/null @@ -1,17 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -#ifdef CONFIG_MMU - -#ifdef CONFIG_INET -extern const struct vm_operations_struct tcp_vm_ops; -static inline bool vma_is_tcp(const struct vm_area_struct *vma) -{ - return vma->vm_ops == &tcp_vm_ops; -} -#else -static inline bool vma_is_tcp(const struct vm_area_struct *vma) -{ - return false; -} -#endif /* CONFIG_INET*/ - -#endif /* CONFIG_MMU */ diff --git a/include/net/tcp.h b/include/net/tcp.h index 226bce6d1e8c..95e4507febed 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -45,7 +45,6 @@ #include #include #include -#include extern struct inet_hashinfo tcp_hashinfo; diff --git a/mm/memory.c b/mm/memory.c index 0a265ac6246e..2c7967632866 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -77,7 +77,6 @@ #include #include #include -#include #include @@ -5419,12 +5418,12 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - /* Only anonymous and tcp vmas are supported for now */ - if (!vma_is_anonymous(vma) && !vma_is_tcp(vma)) + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) goto inval; /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ - if (!vma->anon_vma && !vma_is_tcp(vma)) + if (!vma->anon_vma) goto inval; if (!vma_start_read(vma)) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index e03e08745308..1542de3f66f7 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1739,7 +1739,7 @@ void tcp_update_recv_tstamps(struct sk_buff *skb, } #ifdef CONFIG_MMU -const struct vm_operations_struct tcp_vm_ops = { +static const struct vm_operations_struct tcp_vm_ops = { }; int tcp_mmap(struct file *file, struct socket *sock, @@ -2038,34 +2038,6 @@ static void tcp_zc_finalize_rx_tstamp(struct sock *sk, } } -static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, - unsigned long address, - bool *mmap_locked) -{ - struct vm_area_struct *vma = NULL; - -#ifdef CONFIG_PER_VMA_LOCK - vma = lock_vma_under_rcu(mm, address); -#endif - if (vma) { - if (!vma_is_tcp(vma)) { - vma_end_read(vma); - return NULL; - } - *mmap_locked = false; - return vma; - } - - mmap_read_lock(mm); - vma = vma_lookup(mm, address); - if (!vma || !vma_is_tcp(vma)) { - mmap_read_unlock(mm); - return NULL; - } - *mmap_locked = true; - return vma; -} - #define TCP_ZEROCOPY_PAGE_BATCH_SIZE 32 static int tcp_zerocopy_receive(struct sock *sk, struct tcp_zerocopy_receive *zc, @@ -2083,7 +2055,6 @@ static int tcp_zerocopy_receive(struct sock *sk, u32 seq = tp->copied_seq; u32 total_bytes_to_map; int inq = tcp_inq(sk); - bool mmap_locked; int ret; zc->copybuf_len = 0; @@ -2108,10 +2079,13 @@ static int tcp_zerocopy_receive(struct sock *sk, return 0; } - vma = find_tcp_vma(current->mm, address, &mmap_locked); - if (!vma) - return -EINVAL; + mmap_read_lock(current->mm); + vma = vma_lookup(current->mm, address); + if (!vma || vma->vm_ops != &tcp_vm_ops) { + mmap_read_unlock(current->mm); + return -EINVAL; + } vma_len = min_t(unsigned long, zc->length, vma->vm_end - address); avail_len = min_t(u32, vma_len, inq); total_bytes_to_map = avail_len & ~(PAGE_SIZE - 1); @@ -2185,10 +2159,7 @@ static int tcp_zerocopy_receive(struct sock *sk, zc, total_bytes_to_map); } out: - if (mmap_locked) - mmap_read_unlock(current->mm); - else - vma_end_read(vma); + mmap_read_unlock(current->mm); /* Try to copy straggler data. */ if (!ret) copylen = tcp_zc_handle_leftover(zc, sk, skb, &seq, copybuf_len, tss); From patchwork Tue Jul 11 20:20:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8EA3EB64DC for ; Tue, 11 Jul 2023 20:21:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E4516B0078; Tue, 11 Jul 2023 16:21:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36CAA8D0002; Tue, 11 Jul 2023 16:21:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20D2A6B007D; Tue, 11 Jul 2023 16:21:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12D006B0078 for ; Tue, 11 Jul 2023 16:21:07 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DD137A0327 for ; Tue, 11 Jul 2023 20:21:06 +0000 (UTC) X-FDA: 81000450132.05.651C759 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 303EF1A0016 for ; Tue, 11 Jul 2023 20:21:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=N2FB4LpM; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106865; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X97NNzauf5O0p+HCWC6dKF0cIqUU0VbFEqEvNM0A4i4=; b=ygbO2SgGc171gqzKfUhtZUBjYTnLsrz8e8Lp06moIaGVonTdrktk1J1GXtp4hlLBDstUvo 16K+2S/sgdCUE7BL0roDBsi9EOvl5SHiqkCUF6EFLP4CEMsKTIY9g6pcLeIEqaXrX28EHt 0DqspvIziHn2BCU9gpydw1hkRzcP70c= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=N2FB4LpM; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106865; a=rsa-sha256; cv=none; b=LDqhEDV7griCYxdY68waWtwRhsEB9VjLbBFCpg6802hVMrViOskSgkTx62sf4SmPId1CUi lQMyqbVTuL0ZAwCR7oGWVX/NEcbgnta8ogdqznYvSZb1+SqIPN7Ow821RdVJLdG0eM2lYy OumyR2M4ymzsifGna8RmUjBGQlZOkhg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=X97NNzauf5O0p+HCWC6dKF0cIqUU0VbFEqEvNM0A4i4=; b=N2FB4LpMKm0mT+slcEozTMrM51 Mzt8L4tH3hlsu8n+Qea08POT5eAZDE9TFcS3oWwRYevUqOyXF/Rxy4n/DXbo8yPxPUMSnF1bTd3Jq gf0xyKh8oVCeUwqgzPW/PsA43KHs15iBRAqVrbt6ZknhdhH5OHOKUD2dh1GgoMCR3PZslcddURI8l EpKHwR6htyRucMHqyS7bSSc7VDt5IeV4qcQWX1bvGuIyveIrqWC1wYlfiFy9LhIHbt2dgordQS3wd qxSzFkAwiJG2L7M8XeQxRR36/4ziRMElrEsLItSW0Wk/DindbqfqMSYGzo0dTdKqjhSGNcQyQBCg4 HcX1v2wQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqr-00G1Q8-HB; Tue, 11 Jul 2023 20:20:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 2/9] mm: Allow per-VMA locks on file-backed VMAs Date: Tue, 11 Jul 2023 21:20:40 +0100 Message-Id: <20230711202047.3818697-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 303EF1A0016 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: xihkupwfigz7prtbd5gpg57bn6apdsh4 X-HE-Tag: 1689106864-167981 X-HE-Meta: U2FsdGVkX1/jWT29EzrVh9WTBc9Qn/n1nQyAyqlZNKgDoSQ25YFWSqK1KF6NMMAvzAzu01gj9UH1bFAxH+v6r+6FT0SgPWfrf3ACZQVVGzNzf9VXfG7ru8NykXrz6eBX5rNck6IjSIOOz+dDdrzoNiB381xdCKHs/0t4UHf/JoHsMRE/nzpYRu6U8Hiy24JEWGZRKnCMaZqMksUaysmwNB1mdNBYH7B06XhlJqYnKwET9UlC5zic+LC/MS0no4r2qQ0JSdouXs5XKM+cPxx2G0LpRId2x8UuDafWMMpPiU5VURHTZejpBlIKFCLNByg0Hjofmt9hCnGhx+J8VD943/IE+5D8LQbx+Nx/DGix4i4UvQw/xpJsgVaBZKRkGrD+u+PNvNF5kxpUDdtCTMUk33TEGUztn4+NtjvU6qiNhrivxwiWgGwoBEoxeoe14iDILYxkUsc/QXfBtqwV8ZSFqPTKivgd+a1E7TLbV8VEEBfqPuabLsu2b1wfOglXHl42IvJ6CwDAoCSJoubxfhTfsRLpjbbrmbw2DLfX78r6OTVK+C5kGCqu8odZoY4NCURAFI1pqznjiAt/mPWMzLyk2Fw4qBgWTAuJ35OdotmLQmZyNeor2i2T0d/DE4lcabR4u4FPzm99EDd8uXRNLrCt95AU03xwkaFuMQBGBI+psdwH6ExvuiSL2KP1XHrcvWNhN+OifxVGJ++JDp6jdhNWN89sT4Wna3CPCarnqfvfuYSg8Mi716b5HA44cr6RmwRVTMS0Dbr48JZ4yxDZUqbwn3JgEYt+XW+MowAEaPYHvZhVemQVKj2b1zWhGI+xOfu1+O4+RdZd46lktpL45i7aUduvZGJ1pDgritvE5fG/GEEBmDD828XmvqR6vFtpLfrQWSAZdrScNLL4ik840Vd67wrzZbHECMQPLIga3DZacWK7+0wE7RO+6jRKZn5tsnd/n3fcQcK5lNTfoa7whf6 eVuIRJwy /4w8PJxFxjYpHDI/p2d0wSbR/99PaStsEcMIz2V1kHdn+EluRI6Lz1l40cPMBgI9SzysRXepMN8TPnOTezGGn3U0ZuH9G+R1j8DqT9iB68xU+2qIV5vHHKdG28BaVhtFP9FhXCnXMpAcbbgJHDWxYLfeU8+mOsMYmtnDo6N1yFWeewvLOlWvPqMsNgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The fault path will immediately fail in handle_mm_fault(), so this is the minimal step which allows the per-VMA lock to be taken on file-backed VMAs. There may be a small performance reduction as a little unnecessary work will be done on each page fault. See later patches for the improvement. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2c7967632866..f2dcc695f54e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5247,6 +5247,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, goto out; } + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. @@ -5418,12 +5423,8 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - /* Only anonymous vmas are supported for now */ - if (!vma_is_anonymous(vma)) - goto inval; - /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ - if (!vma->anon_vma) + if (vma_is_anonymous(vma) && !vma->anon_vma) goto inval; if (!vma_start_read(vma)) From patchwork Tue Jul 11 20:20:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFF6AEB64DD for ; Tue, 11 Jul 2023 20:21:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14B466B0072; Tue, 11 Jul 2023 16:21:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AD926B0075; Tue, 11 Jul 2023 16:21:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3796B0078; Tue, 11 Jul 2023 16:20:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CFE846B0072 for ; Tue, 11 Jul 2023 16:20:59 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AD0A1B020B for ; Tue, 11 Jul 2023 20:20:59 +0000 (UTC) X-FDA: 81000449838.06.05AE24F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 071E11A0003 for ; Tue, 11 Jul 2023 20:20:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wQ2iTu3C; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GLnKPguWFW7/+2NrpNM1tuYQ/qesy2XQRrzmH9bU/NA=; b=DpWnO/s+gEKXi27wf90/5VQGveJlhYevCydzMbKLXEoyAEUyyDl5zxcOmM3Ze8afrKt32P vJ+/GodffFAFIxyWlyIU8DihnMwcwok7mrBbDW+5Z4AcxKMJE4MHg9YKknmevaydvBBP3U PwSyVPeD6/XtWDYiJhHzc1PmXysMG/8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wQ2iTu3C; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106858; a=rsa-sha256; cv=none; b=zk0y2Ip0sJkaG8m54A8vDzPFPSQFOmhqSrTEKRFFyItosYRPLuRKkgflZvFZU/mvHaqOww bM5aTvpuHacEc7b1Fnf8FBWgw7jRUeb6hGjAS2n2PVaWd+B2Ipinu7Yc53H1qeBUa+XgJf 5Ejbq2egTEy5/XYCFmP3aAGMPvzwGtQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GLnKPguWFW7/+2NrpNM1tuYQ/qesy2XQRrzmH9bU/NA=; b=wQ2iTu3CFpFz2xaUFdin9qQcLg Zo6SBBpIiOLLIMV6swSA+k3EKgZiUTrGU0JtQ0GqAtC1zBFq/djG4qxyKgxXL2rELIMTjX7PST0+t wEF09/oTC9SpZ1eYMICRhU549nBmS1JJGjx1yl6+G1tVkWh32Nnm4jfwsuZ1yc1aLIxLimEAr0Trj agjHHSe1c13m7UzUObkj9h9Ps3CS52CsrM3yseZX/Eq7zh8qaB/t5l99NlgY7UH+vEG1mGH8Nt+LF Xh2jqTBfKjMtc1j99Is0Levj2AGXa208pRFmDKx3Lc0dmOmQC3OHcoSsJ7qGFeTWMSqRSo0MyyldI H8WIznMA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqr-00G1QB-Ju; Tue, 11 Jul 2023 20:20:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 3/9] mm: Move FAULT_FLAG_VMA_LOCK check from handle_mm_fault() Date: Tue, 11 Jul 2023 21:20:41 +0100 Message-Id: <20230711202047.3818697-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 071E11A0003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gg6g7tyziuf8ibtksy3ycftdyah5oqen X-HE-Tag: 1689106857-283340 X-HE-Meta: U2FsdGVkX1+hxvZagdc935ai0QbuhjhHPz4s12DrKX0vch8GTzmC33Rd4NmAeJdMi0KJl4Bo/PIyQqOkr4gAMgvE2MBaM3q4kxvClZgJg4uyne9X/APsCwohiHeP7X6uiaEUNSqkyKd7iYqsqR+jNExI09kqcOmjXJdt6HvNLJilZuP/qr5l8Bk1ft1LaaNuJYairqWgwrFXSq6VzGNVdNh1vy78FgPjWwOuwe/g+fdzGI8ABvCp5BbmOXOD1JNLi7RFXSKrLebYYc22RO4czV4RvnD3SF1RFmT61/DPY+HliM8gCZxsNPxOkVRaCv8vlXvFbezL3I8DI52eHDxd8bB7I6BhrJNra02j2xHTwCuZwkbn4TUGXPY7N36PvyuhqP0S1hoSSnyLxSDPzXLxfH+gEBPI6SmGySZhdCEFnBILGHahi0gygVbTKPBAUVrTkwJMFijGPlPIHcFPQMcH2xwe8QBqYepR1fgrY/iJGvuJDkJwBJUhRg3Yo+eZW+WM+GIT3BNRKBelmj+ERocr1s35fKQHdqx8CR7gq5lZIwisRWFolvHK5Yb+T/PkbZfz11hdxCe0w0F8Nhj9i2CU35dZrkQl1z0sLEaMmxHZvDFo8dcfsWuFd/69ac2mRjqLgWWkZfZ0re11mqFMcZ7M3RR9+N5VjcD1OcLYcoX2NZX9dBsc+GprTTFWv+VYgevwv+5w+AR59lBc+jDrrjesGYUeakU5zm6lNVzbvmYDscgpvvl2DCLlkF42LbIfBEhk6/0sW65MiLAzdwdviN+tPRMhhopmT73rLkB1R21hTnGrSxEEwB8lJAY0KViHtiSQoC9U0rjSd2HGyZO9o55DYUASkxrhCgWSu2/1XFOdxg02DgtzaNhUhLDkf+TF/Al6tpaEdXo1gPlmGsOO0sHw/ObeEImcVR5oPdyeWwztvrcV0gZ8Mts/yFYDMOdeASxPrFmWWCxnZgnFoloT7Kz pvcz+myj T+bLUIF+tEeuUocJNgcesvvXdBM0FUa/LjyhfzSyOKKf3jbtK5B8uu0W0TC9ockFr8aLZc4g3lxwoMHC2nseSu4u4wkE7u1sn9x0tEAyx9emjn511PlprYLCWPNwZp/tBnWz/OiE/p3EM7LBzT/Ix3aj5b4bLdbfZjhbUSZ92Qfu7wTjrzVfTwMOOkz0K9cZJD/aEw7J2/fTxhB5Oy1dRSZNz86cx02J/I0Bg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle a little more of the page fault path outside the mmap sem. The hugetlb path doesn't need to check whether the VMA is anonymous; the VM_HUGETLB flag is only set on hugetlbfs VMAs. There should be no performance change from the previous commit; this is simply a step to ease bisection of any problems. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/hugetlb.c | 6 ++++++ mm/memory.c | 18 +++++++++--------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e4a28ce0667f..109e1ff92bc8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6063,6 +6063,12 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + /* TODO: Handle faults under the VMA lock */ + if (flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate diff --git a/mm/memory.c b/mm/memory.c index f2dcc695f54e..6eda5c5f2069 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4998,10 +4998,10 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) } /* - * By the time we get here, we already hold the mm semaphore - * - * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __folio_lock_or_retry(). + * On entry, we hold either the VMA lock or the mmap_lock + * (FAULT_FLAG_VMA_LOCK tells you which). If VM_FAULT_RETRY is set in + * the result, the mmap_lock is not held on exit. See filemap_fault() + * and __folio_lock_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) @@ -5020,6 +5020,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -5247,11 +5252,6 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, goto out; } - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. From patchwork Tue Jul 11 20:20:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A966EB64DC for ; Tue, 11 Jul 2023 20:21:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DEDD6B0075; Tue, 11 Jul 2023 16:21:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 591966B0078; Tue, 11 Jul 2023 16:21:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4301E6B007B; Tue, 11 Jul 2023 16:21:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 344BC6B0075 for ; Tue, 11 Jul 2023 16:21:01 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D7DD140397 for ; Tue, 11 Jul 2023 20:21:00 +0000 (UTC) X-FDA: 81000449880.29.C72F838 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 1DC521C0022 for ; Tue, 11 Jul 2023 20:20:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="W/ZMMD3u"; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106859; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SWi2hnJa6UVy5RDmiU5nvVsgG2SVJpIZTkG5op8he4U=; b=p1obY8DoZN23BRX+pD0AYbRRgEjKgD9zltsNhPtq1kYNVLKjZrJP48BBSxV62FTYKhWmoR q4GeRQq2u1SUdcrsUgYRYl/2Wuyq1aUCBr7vwzSixYA951mzLM6wCzPuc44g4qjzW+AfNB wSjhYc3IQpgQdoqnVcezwbExx6s/SS0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="W/ZMMD3u"; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106859; a=rsa-sha256; cv=none; b=NnptW8rIg+kS+xgmEHspQfP1jLfeUMtCkf3ivWGM7bEXEnJLeel7EW2gM9CYO8jWqCKL4x pCKo9C4gEK9JfKglnMF+yIkoy0gB0wPZBf+lZV5Xs7WSIiZQ5ax25NUJTpkKU6umvlYzzE q8mU3WhuubEXn9UuEXswEDX8+9GyujE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=SWi2hnJa6UVy5RDmiU5nvVsgG2SVJpIZTkG5op8he4U=; b=W/ZMMD3uCnkA6CMhMyDdz/rsx8 js1sAmaboeOuGlO/QfP/tkIY64sHyudD8yzOcDG8w95UyCCrvW3+lRjx/ndAFDyBdq1yILWDVxX8s +xR3hsJqKfIvR/awLGd+z8FNyTkV10J7mb6UVwioKberqclMIvX8E8is2PY1eBNf85tLDI9xFcP6J ojVNyWZKBjenQ1R0trgEulrgdvA3/JGfcRNl/lFbRuHNBJHBPsQg4e2GGLqbbjN8nrFGqOYg/Pafl pfiG+sUyiO7cHtXp32JUp31f8x4BgNQtw5g4lLaF2ybHg3RjpLPYkq3hVEu9oje9EIX+X7z7w25Lp j1KRSO/g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqr-00G1QD-Un; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 4/9] mm: Move FAULT_FLAG_VMA_LOCK check into handle_pte_fault() Date: Tue, 11 Jul 2023 21:20:42 +0100 Message-Id: <20230711202047.3818697-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: th8ehwzomugdt7816x7q5gr4zx687n5o X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1DC521C0022 X-HE-Tag: 1689106858-585703 X-HE-Meta: U2FsdGVkX1+UzyxcysV/cyzOKRR3/KcqBZtR2mP+yLuq2RMwAxoSZGw9jG26obKcWZ5wwnRp+5SCrK7c3FKmBwsiYwOd3NY5nc6fh7ulDuEOzV163XlG4D75GaokqDTe6KoDTYIfPuAhVrf599OVE0L9Pp9DJ/SpIcUXDhZZNyfanGdIiTPPxX0H/7tHu4LZJniWbSj7MtovvvrVASHD3ElKdAR72rwxAyeFTFoEZ73DthY3Ri5iZZM2s89fHSRx/0jkUtl1BIdVeq0CDsrR3SmiYclZ8vcaQEsTBIrv+WSm+ltvK+mPzyuvGlBpx+OZBn5yY9ePbR9YIXGNwsCdL+NcbKQMBKFQgEPJqa6iLn76DERST65qgt5FwulMDuAULTgTEkX/CFOMYXEoLN9AaxStft2rQyNU0x95JE8hk1oVaVr51/j3vwrudWsL/1EJloxOOV+Q1l74vwBdraGof35UzUGUjujOrP0Ht+0a4YaAKqwls0sIgtUJVXVVzVUZp+dZlK4nV0QJqTUVmyD8ORZLs9LplhtSMV6pATzo82vLdBmzYzLaKAppboTDqL+AXZ99OPTDVBpoA4GqkkIgQTuJ6ACzHoO3C+jHFc+U0i0bS4sxwyqCKMMzqfkK7voWd2oT1L51P90r8NfbHH+Pw0dPJebVF2AUekzG2sPTbBJpFVf/rdZAKcxrmHZg0ZOb/mYaketaq/bxGVCuYAVhrB2Kkv5ibO4fbbl+LH1O2zOMD1qYyXs5KMljkF3KgkncuHmet0rABmW771tp7oBOjyRUkvFBBvv3dhLUG+L/wcLbRcspfOjb97jSj42oXbnvAGwy2MRhFCXkoGIK9lp2yfgKeV70mQjPTv85+4sOoa+8DxMvPQpwbJnv6i6e88A2x/ISL2JKMsamj5Hr7411r5ETSbwrfqwceMok3QwA/xopqJTXd5fZinYnUXDnPqtL0l23LCKxZMce0+fVEeL 7wfvMpAK SR3ZmoH81mbo3ajS9XWxAibbRrTJYaVVEeyfTZGZRT3M2PUtL/31Ws2VlKcxZkfA/ce7T12a2kFCX7IJKwApdQgKMDcoNsUkxyEyTKonwUV10CGNPLNBX3gzSutyFtjBb6LHysIRUiudD/RGVf64B+aFEpzOCZ5PbSOz7UeX1g8YBe9zKSecexoZcCD6AK1Xmp6son9B85hu+ohub8MkdBaAsY88UD7sYQCdU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Push the check down from __handle_mm_fault(). There's a mild upside to this patch in that we'll allocate the page tables while under the VMA lock rather than the mmap lock, reducing the hold time on the mmap lock, since the retry will find the page tables already populated. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6eda5c5f2069..52f7fdd78380 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4924,6 +4924,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -5020,11 +5025,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) From patchwork Tue Jul 11 20:20:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15350C001DE for ; Tue, 11 Jul 2023 20:21:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A24458D0006; Tue, 11 Jul 2023 16:21:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D4558D0002; Tue, 11 Jul 2023 16:21:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4198D0006; Tue, 11 Jul 2023 16:21:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7B8FD8D0002 for ; Tue, 11 Jul 2023 16:21:16 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 30B7EA039B for ; Tue, 11 Jul 2023 20:21:16 +0000 (UTC) X-FDA: 81000450552.05.D524F1B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 6D3754001B for ; Tue, 11 Jul 2023 20:21:14 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="N/UAfb14"; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106874; a=rsa-sha256; cv=none; b=8Coj2DSjK54wwtiEcFRsPtOO7VRt++GHJQvQOwbvHbw7mqgLMQlu6TyoZuQGisEw3C6cMK cWuvns6db5Jng/r2EMpgXDI1Uv1EEKqlPQr5Q3V/fpguHK08VuKlr9fjflBVwdw8bXfHsp l8yhABv7il7+1ab1mX3n9c2KljtuxtA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="N/UAfb14"; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106874; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1J6AQ/unSE4IC1W70poNN/OYLEC8RIrhsKKg+sT5K2M=; b=SO8PR+o5n1RjBsHEEQ2AzsMWBSiGJOkWH+WdyFwwE6p6gl9yfiolRhGzRUtkAcwHBbLd6Q 50I1cgX4WOvYToTZpod7IW6QecEK2jNhMT7RslIfUYkSbbA2jrRohHVLPRDPdnjxOXPht3 HRytFT0JxAxLxAUMQTjpZ1nMvF9FT2Q= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=1J6AQ/unSE4IC1W70poNN/OYLEC8RIrhsKKg+sT5K2M=; b=N/UAfb14mocbi+tYXm4qnqDyG2 54o1KXDavmlW/bXWhMvMxUY0YE4V7uNEqi2dinDdzSmQaNlP0d+Yo6FQ2sFbcl0CI4HlzHr8GgyI3 1XULhcv0Vfox/1xF4oknDaW6xBMDp4a8gImwj39X+NOkn+YB4nvufbnG1EEQvBTxBx9D5U2xysqUK 7lgn9jqfuC3V29CCgIJTSn59h4STqEJYBLzWetZR48XDpxEouAj6oMF1+SnvSA0dsDRjjqbB7/ymW bq5CuSOEJW5NWZQrReS0SsFhu2hMZx+h3GJZFO0tayozQN0qenw3amuyVgKEuBcMmwrOV8B3rWlCf 6//Jqs+Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqs-00G1QF-2B; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 5/9] mm: Move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault() Date: Tue, 11 Jul 2023 21:20:43 +0100 Message-Id: <20230711202047.3818697-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6D3754001B X-Stat-Signature: o31559jsbzyof8sk5xi5qtjh6hw1pgpd X-Rspam-User: X-HE-Tag: 1689106874-909667 X-HE-Meta: U2FsdGVkX1/1Gf0MdwQL8iNta3iluxTEE8tVPu7FStCqGrmxxsYL/8tFpGVRqHsUqj/NMpCkhtCrtScmNZEDhRsxNyEU3TDBMeKsY4FRYqeIr4pWC0rY96ViVCiRwsd10cidXN7F4OzT+n5NAb64D6Cj0RXCA5gvUKchJb8ktaJkxpORRpJbtH8n3qrxwyMLQhufgY7MGz4qeBKumUNz0RI6l0U1wSL1gdtnR0I/62ZIpzMpMYtSIbJ9mPzATCgDIMYgAuq6y8ES4MEYcqnBni556dFjCFxrMz3Vc3tu2m3SGejM0cqwfO+8RtyPBjQrx+XpsPsjvrctKDnJBBHP33C/3mYIeuUINVLtqkADn0LF3ZUPE+nB1CC9QBkiWuIDXxkUUwwUnuseG/OZT/NOoBOxwzxuuo2WHPlFSW1N22QkL8BU9k11wHjy841b0rpdB3JC6inyqjG7qqLpI86xfXm4HSgg2EZg/zI6kasuj3VZPGQCxo+2C2M3RPSqSl6NjD0S3vjXQ6oWhexl0h1w+OP2kOBFSdhXHXhiaE1atfQ0hN+r2s2uTVG96Q5yo/7KUxEht5x/GvQwi/UZkjhh2/rJ4fCMhCUs/EN8t7TEPv8X03bBeL2ruS/HNhGO0eEa2yfpWjND/IPTlL5hZJnvezkzOGNtPz9OBg3LXfLbST3Vojfm4CayQ0oT2x2oUptj++CPB3eIgbiYe1JJ1L1v1Ml7yMH/yWnT2mOqt/NozCQC2YMxDdAjqkhru2UIgOgo2UwEsCgAC+M25vorY0+dl6iKs0T0ONFFYTBy0Nn6eHHZcKf0r3C7NcIcxN1uQovRkPh2eymQ8vG1sJWuCmjY6s2+MZt71Adu2Jgaxjys3b5beS5CxjXecBJHPnSgJS+Qu2P/uyKB20YYbibffr+/DqUH43hBIEP/bLnPZHG4H35dEq7U+gfKQwdPx9rvCnkQkS5+k2CtO6/h+17reeg 724mkyDD 8nmVZKp6ijp3Gvu72yVQKV7cR80CXRwbHduC3ihkfybfWlYUvir3j8+/jspTIqzc7aUSZDaMhGo8cLvxp2SpVn80v9nWXko2fO6gXAw2PTEFSVi4D+zKw0UmmJ8B0ZL2iusKMIRvDei0SoXqHe91GvD9vy44Hp0/fYYKkhKwqNZszgbfKCenbsSPeQqIsn0ep3tTpn5cpUI/lFNrwFSI5Hz2vWu6XfjmYGj/N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call do_pte_missing() under the VMA lock ... then immediately retry in do_fault(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 52f7fdd78380..88cf9860f17e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4661,6 +4661,11 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_VMA_LOCK){ + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ @@ -4924,11 +4929,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -4961,6 +4961,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!vmf->pte) return do_pte_missing(vmf); + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); From patchwork Tue Jul 11 20:20:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13DECC001B0 for ; Tue, 11 Jul 2023 20:21:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40AA78D0001; Tue, 11 Jul 2023 16:21:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BA886B007B; Tue, 11 Jul 2023 16:21:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2825B8D0001; Tue, 11 Jul 2023 16:21:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1860B6B0078 for ; Tue, 11 Jul 2023 16:21:04 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D76651203A1 for ; Tue, 11 Jul 2023 20:21:03 +0000 (UTC) X-FDA: 81000450006.03.5425139 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 050D34000F for ; Tue, 11 Jul 2023 20:21:01 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="ZX0x/yAQ"; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106862; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y6MqKlK6OxY+tonB29qlEWdduvckRCFfUllkRcgCchQ=; b=BVbe04bfUvuJpfbCr4avuEGeGln+EHFCmpnuITWqty3IfO6Hp7wgJYCgasaJh1UJF4A8eq RUjdFLloHn3amwrIPAoR91sRvgppmv5uZ0RX79+iy/20bN5er0iu/jIv01pXS8fv5jByLQ jHAzfi2RZov6VYCO86qMZU0JAfEQg7k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106862; a=rsa-sha256; cv=none; b=mjrgYrccSrTHAKOLp8SeO6ag0Q4s67RZq7kJFIsoFLsTVnAOLRObCnxt/8ueYadP70hYTJ sVURqcvWOXVO+lusL1Z2wgQcMKOBpa4CdNPO3UVf/zOGkcnVuADzRkbzI8/WZj9R+AjNwi T002FNDRsAwowmwH+RJG8tdxA4MCK2E= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="ZX0x/yAQ"; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=y6MqKlK6OxY+tonB29qlEWdduvckRCFfUllkRcgCchQ=; b=ZX0x/yAQj25a5I0W1lufq/qwUP oXson9J+2ubQdjXaXjjRQ+O80fML4BiNDyo6y8jRSU1mXCImAXYb7ruFGwoRefM3EqSQc1BDVRPo2 ppz+hwLnF9JLwe05Hcdxyrw3BWuVIpaMCFMjDmpKW1LoHIwy9KeZbvl1DlfsSmjuvhHINYGd8vruW wwfs5AZ3GNemhwSJxsakWnNgZpxjJnyTa/d7nKlahUEur99F9UV6GUfq7l/OxED//pTMUJ0Bk9pGb qK6l4TnIwmYQpiv/bo7yufDpeMmcqZxl85DCZrq+hr9QYvbfBAf+PWdagfdqEdA21O7KpOeUCFSP8 HfNHy3Gg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqs-00G1QH-5O; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 6/9] mm: Move the FAULT_FLAG_VMA_LOCK check down from do_fault() Date: Tue, 11 Jul 2023 21:20:44 +0100 Message-Id: <20230711202047.3818697-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 888yyyf5tbz6wgubfc53u336t7pedbds X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 050D34000F X-Rspam-User: X-HE-Tag: 1689106861-428889 X-HE-Meta: U2FsdGVkX1+ApfogpcE0k0Jtvc4x7jnHCZJ2Umc3Al+f0cWgx28wK5xWjOyd83WeMYY1taSmcbL96c8bYyLkyGaq9M1BlRBnn5sQzBgO2C1VcOo02K+2kVa5B3auq1B7DsUNmFZlwh30mYVyAk4Ix0ImqvDClNyp7GDIopooXCTq2EqHhmV9vFtGLHn4BT4twOkBB+9fEgXLD/Pfz61YyXoyeBFuit/deY0nQEPBe4KIoMqp71NuFdzIwiIUbhYxrL/nwWdqmd8FDDUlkdUOIoelpA68WOioHUhZTubru8EGeu0ovardQlWniuo/ozUx2f+cYawVlU2cN47IERYaUqknzcJfkTM20sg50v/rB2QQlf3hDbvcDeBm+8g0vr+yaZ5r2b7nMCJRIRjkZgpnn8PHfSqtOwmiHcHSgWlzm7/k5O38YlGDhgaqoshYVg/MKP26uYo2BkqG0TtIh+qf8VMAN2zZl6BQhhuesH1Wm4ka1zc3r9TwyvfH6B8bz5GwUr94d0HfrmHcL/wiRpY2ArePBiy9R+gVF220fPfbrRBbQ9hMt/w4nmF6TrMAC3Bbq9JVDN4I6bLFOkKidHTNniRPT27zgU4jSVmzaf6xPjGhZAG41XMEjW5Pevr4lhADQ7l3PpZRLNd+e3xjYhGSkWwk2Au3/Lk8Ocpc9VSx3EQR0YvE41e4+eHP+HW+76iZtASyZ3sDk8KG3oNyGkDNJQxPqlHuQUPZmhTUNI8dKuvBDMlhw9JtrMD4YgdQSz1dzADrWJKlYHQz+2sopb8ojyjDbEbFk5n5n0U6wpRlDHyCMRkiNMLTQSPOuAf2qUrVqh9nmHyBjwCL5VsJ+qK+5nOlMIYxAJbdL1WknO7Oyv5WD5aWDqQSuXBT9/oPJE+OIGc15vES7n009QbYSEWY/SQWqPTqW5P4Qm4jU/yTWDyJwV0fD8QolWQokjSh8ZWRfN7IiNZm/98UB0QQOld XaEShdKd dS+rW44qwFia2c+rg3kHVzBZITAY9PuUHCFqw/7AEnxYSsh2MvcWmS+PR5DBEq7ttktSNkU801rHPnNILactJ9SafcdaaxV6qL6NRuXv9W49XUh26+Z0iZvMDsIy+OudGKYkH3wyfJLmG93RZH6VhbDQvTp6m/2kFMVIqbBXm2GKSUe6YeaYfqrBE844Y2a2TnD/mN4iP5RCOUfZKqmCsyi4kME+FcJwdVwOw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Perform the check at the start of do_read_fault(), do_cow_fault() and do_shared_fault() instead. Should be no performance change from the last commit. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 88cf9860f17e..709bffee8aa2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4547,6 +4547,11 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) vm_fault_t ret = 0; struct folio *folio; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4575,6 +4580,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; @@ -4615,6 +4625,11 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) vm_fault_t ret, tmp; struct folio *folio; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; @@ -4661,11 +4676,6 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; - if (vmf->flags & FAULT_FLAG_VMA_LOCK){ - vma_end_read(vma); - return VM_FAULT_RETRY; - } - /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ From patchwork Tue Jul 11 20:20:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E931CC001B0 for ; Tue, 11 Jul 2023 20:20:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A9F36B0071; Tue, 11 Jul 2023 16:20:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25AAA6B0072; Tue, 11 Jul 2023 16:20:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 122AA6B0075; Tue, 11 Jul 2023 16:20:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 024A26B0071 for ; Tue, 11 Jul 2023 16:20:58 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AA2901203B2 for ; Tue, 11 Jul 2023 20:20:57 +0000 (UTC) X-FDA: 81000449754.22.33FE311 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 1A693160005 for ; Tue, 11 Jul 2023 20:20:54 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TzHT2edh; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v94Ubxfh0lfpoM1pvfa/PhiO08lrJc9bG/pAuDnDu5E=; b=wCRbo+ztPmLZssH1uDCJPI01O9I07ImdJ4SJj82p01FjtboeCHF7Jp364eqZ8z3wFA3u5N UcqIn6Pz5OaXXePkQ9kaPpi87iuK6JH8YQE3Ak3SebmMtc1vtqbJqye0PA8AGWDSIvMnai aka9Qk6F/y5li3n6s4z0kuDyIG2QPAE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TzHT2edh; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106855; a=rsa-sha256; cv=none; b=VvNwdxgCFxplVKhsWvRX7SeE8uglvfHhPD97n6AS84rhSDa6h296Wd1BN0rC9la2IEauq9 GYKw0qBqJJWvZbe+I7twUfbuXjCwqlQyFabaAgX2QvEF5VLHnXMSsqH94eFNasaau7rl6u RU0SOlcFjZeaMXGTF+haoXfdfYZoPXI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=v94Ubxfh0lfpoM1pvfa/PhiO08lrJc9bG/pAuDnDu5E=; b=TzHT2edho2+AoBnEaZG6rSN+Xz HqdS52OSxRjhfnGl0GcPjNDDaW+puGj9MFK6PDZdT0nHLkUZZHVZia6dFjj4fCyS/krNwe1XiBJ5W fG8eF6uJHrHh/n19hVXzIQE2sFtiQSswSN30D3B9vOlrHmVQ5q51J7cxkBk0/VfmnHySgMxOdbGO5 lPK6MQnwfgOKQrrf+YXuHBuIkBte0N1ZLRiCRnpASNkDN1JdOSpcNqw2mwsAfZ7OuF9U6HU9PLIZ/ YsAHUMECuSPk8nMWNQYQyiCeg/QspuKbmiWuO24oJQwEh5tQaIjAA63lIhKQPa68IinpMHnL73dW7 q4aTheXA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqs-00G1QJ-7t; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 7/9] mm: Run the fault-around code under the VMA lock Date: Tue, 11 Jul 2023 21:20:45 +0100 Message-Id: <20230711202047.3818697-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1A693160005 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: orjqq1yaifctph6ff15wrebccqieexi3 X-HE-Tag: 1689106854-7263 X-HE-Meta: U2FsdGVkX1+FCoGl9z4+ksixBGhue70Y0pZ20GAnPxaGCud1RwvPqELTkkOyaqZ1uQlfIsyanVeBdafj/oo9+zC8qfbkOn33TuW+U/9O+B5rIWjza+dD8csQCBmzqXIAC1U+rYxWkbQSvnWT4rnPJMAQVJ6BmcG+7696t/yMz6J5tmYgnTTGnx7cSC6zuRSi4NvKSHKNffdfmTHE8FUEmI+S40OYk1YcbdSajgqP/HjhzC/46DiPiZcvwgDJJXCeIm+Gik3IRX2aYuyeZIHujX9EQ7qaD6X8rf20DCbvgGcl9+GPx0ke/kvrCN9Zhe2Ns13WtsabF/oiAKGGtTLVoqhhWaK7OXX0N8kafEAIZCAbvGvPqVX1GyQ7HtzNHsTtZ4umDg33Yad7AK8XkQvXG18UAoyJ5EjFRMFjwJpFp6ZBCz+PkewfYCiY8QDgRwkFOjopOAHw5iAn5VezYkeY3WsV/d76yEtaZHeIHHWyLAa8LKnlIMM8O/Oe3waoFGJmI5L1zY1vYrBQjUbEAYjsgQ3hD100NRyJDi7XTa4yDU/7u6tREFi6YShFTjdqGEeVOoImhge07M+qUDLSN0l3/U+269x17Pd5qVc8xahzcKImhKEqWZ8tK2rlB8EUA2Xv+L7qQBQAooXmZmN0HOk1qzcoHgdD6t2DJ1mKjE6/SPO9ZNWisE8isWZxeBRtB2hGtrgZSuEB4FYdVGWOsnTztxBtfjVA4wO4kK7b/6pUZBa73iMoG8G0zYT7wskO8yXrkTtW8s/L7GgHA9R0St9KpB1zIFdHI+Df+zl+e5JGitJjwvpKH8T76SBfkWq772l3hhxs5dfRJyaifj8tzOGP9FEIe1p6+5L0fD1Dhix0aj3FXyPloOsJywvUT1TeowSEFwy2PV6NG+cYk8hoXB6iSul+sLtBXDcZaIL2Jln6HUrljQDTq/ux+17V6cbIMg7rCAELB+40Jb4ri1CwIcT mNPj+q3G fgAJFTrpu/T5b5uV8G3J+N1qGyGUxrVbhh2pIchxWGqRK17hCg0GMOwCpubQzghbZXKOzFrQQhQG6puMr7sI9wfD2dUmgb1dwSbs0N1bAzsMIf0G0YFV2Y2g8doihBokMkc1DB17KE5YuxuOxXDkgDwOm/1+8bHWsbjBVK1SmYceJVKjirM29OHyBBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The map_pages fs method should be safe to run under the VMA lock instead of the mmap lock. This should have a measurable reduction in contention on the mmap lock. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 709bffee8aa2..0a4e363b0605 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4547,11 +4547,6 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) vm_fault_t ret = 0; struct folio *folio; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4563,6 +4558,11 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) return ret; } + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; From patchwork Tue Jul 11 20:20:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C60FCEB64DD for ; Tue, 11 Jul 2023 20:21:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 606538D0005; Tue, 11 Jul 2023 16:21:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B8568D0002; Tue, 11 Jul 2023 16:21:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47EC18D0005; Tue, 11 Jul 2023 16:21:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 39DCB8D0002 for ; Tue, 11 Jul 2023 16:21:13 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EC87CB0211 for ; Tue, 11 Jul 2023 20:21:12 +0000 (UTC) X-FDA: 81000450384.08.77453F8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 341D180021 for ; Tue, 11 Jul 2023 20:21:10 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cXXWg8Bu; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106871; a=rsa-sha256; cv=none; b=yt6/85TyBmxkpLkU+uan3jyPOvZQYZ9LyBThHgabSayu3dzOSUzsh7jR/jYxzSeeI/HVZG BvKDfz2BHo3XWwqXDOvpn6NCwo4bjgyCKqYtT8zUbUyHoIglfIOAVjYOQp+2Pvkz1TwSMc HbokKVF+ILlF5R3iEd+JhgyCNEu1vlY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cXXWg8Bu; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106871; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oydEjCEkyZj8+gutTzPsBmFt0r4FHmvMPjaQ23PbRgc=; b=XKZwuwNVpBFcdsDm0hVTs7BeF9tz/11jacLYsC8iqaZhdEry46ZAwHXBKhXTGZgD15g/jD OSgYYfWci32Yt5yQWG3j4fjKJEXgEvSJaEIxGIVXzlusMI6eK0vqRtfFDChR/lHS4xKesg 2CHSsY09ywhRbQ/G19EGbxt11d4qQIU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oydEjCEkyZj8+gutTzPsBmFt0r4FHmvMPjaQ23PbRgc=; b=cXXWg8BuZO0XqlvdRgjjzr84up U8ERFG7x3TwJYwHvp5ZfHxkfuMe5agMVZN+MrgkEHvY58Ub8lYv03fWL+T227H7pgedOQfEJG9i+Y ny9Z37VhKs93IJeDFrg9FolL8ayGmHoQ3sQQM7VsqTk3DK/TJVXztIu5IZ0T3dm1zwnVp9C/URPmO NlWMbj/9ZcTXccrHoDs0LxlptMyivsKD5t1zRgmN9/NZEvfXmHpeta+MjmGtFT7ZkLEFRd18BNAtz j8ux2UC+USp+KPRGF8TeHcLInvUIx25O41hcZyxkRGgAEupYP1XcneHPoS2ak3MRrBOFF90mnIpWq QRicDFBg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqs-00G1QL-AJ; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v2 8/9] mm: Remove CONFIG_PER_VMA_LOCK ifdefs Date: Tue, 11 Jul 2023 21:20:46 +0100 Message-Id: <20230711202047.3818697-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 341D180021 X-Stat-Signature: yk6cgqyunk1wuidb4s8zhwaed78m7d6i X-HE-Tag: 1689106870-701387 X-HE-Meta: U2FsdGVkX199qi4UMtwjgtfJLQq/lSsQsHVt0BgLzPvWRDHOqlCqWpWt2kq1QM9wrjN34/gjWzWAjfxzFrh4nKcchBzBfRpZibK9Q9Bh2LA+S1A0Lib1OQUTG/alYtREhmIxdlKs7feWkAsV1dwZYB+NNs1XAE3cfWOFBZDgcxhYHOejvjYC1DGhgou7kFNigaZYtCa8d/6ZFxWaw2ftgEUERVgYgxPtqhKpLuK4HPeyyNPWHzfp9pgd4zHr/6qzEDJM0OTdOp31Rl6GCHHe88R7cjVdzPSPzbT+mUgAc71osJHJqvAySWVzp9+hjdveb+OCV16kNbM7ywCBef+GzPdoaE5pe01Ft+Mf0iwAMC3zkxYBWkW1hJVMBZRKbwLktjqpHavmeBkYfCpXB+Cf0Z5EUeg1SObYZhj3hLCGoJZ4LbvKYq1v1pGZLkTABlNgUXHtj0CiOgcd9L/l0ppzfM1TkbUixow0u9H34xcQEb2MN2Cxi+YRJgheS8IaQksZmyQoa3FvPkkCQjzlHwr/Iq+ycSWiT4ITjbFG2VVgvWF5H7lwADphILxm9j96Jn1j9clyEYGckARJp1OCfNgJP5GmRu/BcjHzdRdqv+MXPsoEm1ZNx8lWFy3sviHtXugENblDiGomcb/eaIU5BYwWMQ9HJE5/iw5aeaRzCrIk9SdDEHSjYytsGyAmd8kVXsFC/BJcXAZguyyO4TD9n64e160S7yrQNLoJy8gEn03Gruiv2QX9CdVh36Nw1e8GC2y/DIaljSsPcRawcYNTA8CfkuR8K9c/IqebqEkeQZKHntihhQkyIbIyyrkhHg3FTb6j9TTaiaVqtZSfl06KmcaFnE+KX8sKSDu0Byb00BX7bPlAiD7bLjjTEotTjkcGO473W3+iYtefUAHfML6j2ZcyVfHACtxfczBP9mM8lsk3p/p8Onb9ng8zQhP6KfCF07hw32LZtHNQqs78bkt8vqM O3M9inmI I88YArHBnJwrf6NT0okUqPHmB2bkQqeoyZYBMd7a+jRvFvxEqnBtfS3y2EEpuQPnIRqaMAuIhc1zteA9e5SgLaCybIOefS0kRvQAw48Lc0pp1u1b0bBSkGVOGIqsLmltlWiWUjrgdZV9iLEwylPHjRdTkZxMeaXDIaRyIOB1nPn3YW9SzCrn0V6+ekqPlR444L+L3vSJlpjOeY0OjxaGHo7Zhh2TelkHNZ9V2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide lock_vma_under_rcu() when CONFIG_PER_VMA_LOCK is not defined to eliminate ifdefs in the users. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 2 -- arch/powerpc/mm/fault.c | 4 ---- arch/riscv/mm/fault.c | 4 ---- arch/s390/mm/fault.c | 2 -- arch/x86/mm/fault.c | 4 ---- include/linux/mm.h | 6 ++++++ 6 files changed, 6 insertions(+), 16 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index b8c80f7b8a5f..2e5d1e238af9 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -587,7 +587,6 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); -#ifdef CONFIG_PER_VMA_LOCK if (!(mm_flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -616,7 +615,6 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, return 0; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, addr, regs); diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 82954d0e6906..b1723094d464 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,7 +469,6 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -502,7 +501,6 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, return user_mode(regs) ? 0 : SIGBUS; lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the @@ -552,9 +550,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 6ea2cce4cc17..046732fcb48c 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -283,7 +283,6 @@ void handle_page_fault(struct pt_regs *regs) flags |= FAULT_FLAG_WRITE; else if (cause == EXC_INST_PAGE_FAULT) flags |= FAULT_FLAG_INSTRUCTION; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -311,7 +310,6 @@ void handle_page_fault(struct pt_regs *regs) return; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, addr, regs); @@ -368,9 +366,7 @@ void handle_page_fault(struct pt_regs *regs) mmap_read_unlock(mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (unlikely(fault & VM_FAULT_ERROR)) { tsk->thread.bad_cause = cause; mm_fault_error(regs, addr, fault); diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 40a71063949b..ac8351f172bb 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -407,7 +407,6 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) access = VM_WRITE; if (access == VM_WRITE) flags |= FAULT_FLAG_WRITE; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; vma = lock_vma_under_rcu(mm, address); @@ -431,7 +430,6 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ mmap_read_lock(mm); gmap = NULL; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index b0f7add07aa5..ab778eac1952 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1350,7 +1350,6 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -1381,7 +1380,6 @@ void do_user_addr_fault(struct pt_regs *regs, return; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, address, regs); @@ -1441,9 +1439,7 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (likely(!(fault & VM_FAULT_ERROR))) return; diff --git a/include/linux/mm.h b/include/linux/mm.h index 46c442855df7..3c923a4bf213 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -813,6 +813,12 @@ static inline void assert_fault_locked(struct vm_fault *vmf) mmap_assert_locked(vmf->vma->vm_mm); } +static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + return NULL; +} + #endif /* CONFIG_PER_VMA_LOCK */ /* From patchwork Tue Jul 11 20:20:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13309350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6803DC001B0 for ; Tue, 11 Jul 2023 20:21:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0601D8D0007; Tue, 11 Jul 2023 16:21:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0103B8D0002; Tue, 11 Jul 2023 16:21:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1ADF8D0007; Tue, 11 Jul 2023 16:21:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D4A2C8D0002 for ; Tue, 11 Jul 2023 16:21:21 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B17634035E for ; Tue, 11 Jul 2023 20:21:21 +0000 (UTC) X-FDA: 81000450762.13.843AC30 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 285D240013 for ; Tue, 11 Jul 2023 20:21:19 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oRIKjCgd; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689106880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/8ipsblXmqhBdjmpy7pSUutiQ9uV/NjS0hvvOESjZ3g=; b=2azREM208lcaCRJJtKJvLIuFte2Qhx5BprfJcxzBu1CkUtjhsV4BluYitMBcO4W7zBRTFK 2ZqVAwYRbop0b3kC612SGbt+gi/oa1jvDfRtHfiJsZJRuqcPIAcPiT/JJ8s1WeAM/M0f8M sD7kvfojZ2yXyNuNmSsiZFrt/Kv8rTw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689106880; a=rsa-sha256; cv=none; b=snLOYbqFP8JWRvSV6udMEZXE0VUR7Zx4HEU+9WJLvRoEZFHaaH0gq1t8YCkKfvu4Tz49jE Q7FEsq7zhXqx//XpuSLpOM2jJ6xlPpxgHeAdawZeufjCYJojEo/XMUXj4DG/+CGYIUaAvM NM15omgVqi4kDiijWdq3kI+Ye+YH7LE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oRIKjCgd; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/8ipsblXmqhBdjmpy7pSUutiQ9uV/NjS0hvvOESjZ3g=; b=oRIKjCgdAvLsraMKft9V0w2oKZ gA8+g5HvWxvz4J9SdjP/9qtxqWDS4jIJdweAuxxJ07yxaaSCu0MhExZ19AW2OuvPf+CA6aC+NYOqT QsgdUoumFuLkq5/WvVKVt/mYvxzsa7oZ1+my3VBjHQ7LrQYRRkh7iEd+/l6iwx5E4Lo3bhWFbXcen HXyLgW8Wd4jARBTL4Z6MZxVI6s9jQrSqITbeKUWsnlYMYZms9rU81kzYHlX62UPOop87gF1igohrk 6TJx6U6NbyQJBf3mNlRd3N7lTbJ8mI8y5XtZtR6ZhMIGyIzri2Zmj1h68QaAbAoenY2dJY9625Dzi 2yTOA1Xg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qJJqs-00G1QN-D5; Tue, 11 Jul 2023 20:20:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: Arjun Roy , Eric Dumazet , Suren Baghdasaryan , linux-fsdevel@vger.kernel.org, Punit Agrawal , "David S . Miller" , Matthew Wilcox Subject: [PATCH v2 9/9] tcp: Use per-vma locking for receive zerocopy Date: Tue, 11 Jul 2023 21:20:47 +0100 Message-Id: <20230711202047.3818697-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230711202047.3818697-1-willy@infradead.org> References: <20230711202047.3818697-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: zuee998mchhsdf6yq4s11g6xwzmwxshd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 285D240013 X-Rspam-User: X-HE-Tag: 1689106879-300544 X-HE-Meta: U2FsdGVkX1+gzWleNaMF3BzSdHavxgppdiU+s34nX8inL+MmmgTbdv+E41ht9ivdMhmrZbYoIO5oaRcuYlt68ByDk7ZHW0P3a9Gtxkb4hElQXTArUwReHWpD02xIp5LVn/czIMqIzVXXAC1OW1AVlDSJVtCGJ4ppni7JlqhG+EfapKAlfTEgVplP38UWAv5PpCQupbqjKwJBFXM3WUA3tB73LswdjNUSUB14Ub0lRLh9i2I8VFIGqyIzBwhj00qc2Uzxl7pcbQEpWc4a894e+an0ZBc0wt6iDv1ZRSN0pLj0dFhs6KNuUjBCdDf6sSxeAp+CTOsEcs6crH0pncFr7jyQ9LuOlXWjdEEjUiUUQJ20Hgj3ZkfnwZlOKvFyEzONiRxhV93wEd47gTuk8vpkFnGG0fAsDhZkUPpYtwnURYBtWUNgHgaXoYURdVvInfCyE22wcO1mV9EXnrw8ehplqTCwBBDVF6zxvQkU++1UKKTVrIq0Yr+uRE2Uy8T5QfAQC8J6klmDhlnNenhjl14BN44Zq4KpF2Gl1YpRaDW3nJLccXCngRcQ4a9GM5OhJy9s3LceWiOz3gO0/PaqiNxA9O2zRxP87bGLhcSFhwby9A44s0ww/jfZ/BGS6iNs6u1uvzPojsxFIaDc1vzld8C+NjbOpomC4rdazjQ5xK53nYk/uhC28W+N2VS3IQ4dl/KYc3RMjIFnADYYuEC5bX64pZVs4ilGRBM4HGqBYDOOcTEKnPDvPkUxlx4dA5UUv4xgxpYxm70T0N6MdNqZMW3T4kPTaZdVR4tKHHEDZ3VLjLEUnPA1MVq0HDIp/aVLtX1p9m32PIvloPnoEnq1Evad8kBZMoI4Ou6UMPRI1icock4qY7bCbQuVWKswYgPTp0jvPhXPA/meQXmwZBq5/9IzZX8lx8BnXDPpccjzTJxBEWdZE7+VG6fFJheszCCxRnWn101uLfE3jbo0kAEbDcU HfYXT6bZ ysmz3O2w1RiLAFSWUAspNeRSGnAOIEIiVOcbX/7JfHGYV1L2N2bWoV3q+LVFgw/bcc7CxAU07kQsjHoTn0CvTz2jxZdk60In2WKmPGyFyRsN4uOBTnimB8yEM+A7GE9aB9quWgjxI5MEJ8cjbux6k7TaxcmbXnF/lmBIvKUq3lnzPs5U3Kqt7SrFUYEBFYRQiRb11nD5RkCAIgyecJMeBmN68no6Oa5wWcmBmhCEberGBjAI2rzuO0w7smk9JioIw1oVzMFjhhYzJ2L8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Arjun Roy Per-VMA locking allows us to lock a struct vm_area_struct without taking the process-wide mmap lock in read mode. Consider a process workload where the mmap lock is taken constantly in write mode. In this scenario, all zerocopy receives are periodically blocked during that period of time - though in principle, the memory ranges being used by TCP are not touched by the operations that need the mmap write lock. This results in performance degradation. Now consider another workload where the mmap lock is never taken in write mode, but there are many TCP connections using receive zerocopy that are concurrently receiving. These connections all take the mmap lock in read mode, but this does induce a lot of contention and atomic ops for this process-wide lock. This results in additional CPU overhead caused by contending on the cache line for this lock. However, with per-vma locking, both of these problems can be avoided. As a test, I ran an RPC-style request/response workload with 4KB payloads and receive zerocopy enabled, with 100 simultaneous TCP connections. I measured perf cycles within the find_tcp_vma/mmap_read_lock/mmap_read_unlock codepath, with and without per-vma locking enabled. When using process-wide mmap semaphore read locking, about 1% of measured perf cycles were within this path. With per-VMA locking, this value dropped to about 0.45%. Signed-off-by: Arjun Roy Reviewed-by: Eric Dumazet Signed-off-by: David S. Miller Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- net/ipv4/tcp.c | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 1542de3f66f7..7118ec6cf886 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2038,6 +2038,30 @@ static void tcp_zc_finalize_rx_tstamp(struct sock *sk, } } +static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, + unsigned long address, bool *mmap_locked) +{ + struct vm_area_struct *vma = lock_vma_under_rcu(mm, address); + + if (vma) { + if (vma->vm_ops != &tcp_vm_ops) { + vma_end_read(vma); + return NULL; + } + *mmap_locked = false; + return vma; + } + + mmap_read_lock(mm); + vma = vma_lookup(mm, address); + if (!vma || vma->vm_ops != &tcp_vm_ops) { + mmap_read_unlock(mm); + return NULL; + } + *mmap_locked = true; + return vma; +} + #define TCP_ZEROCOPY_PAGE_BATCH_SIZE 32 static int tcp_zerocopy_receive(struct sock *sk, struct tcp_zerocopy_receive *zc, @@ -2055,6 +2079,7 @@ static int tcp_zerocopy_receive(struct sock *sk, u32 seq = tp->copied_seq; u32 total_bytes_to_map; int inq = tcp_inq(sk); + bool mmap_locked; int ret; zc->copybuf_len = 0; @@ -2079,13 +2104,10 @@ static int tcp_zerocopy_receive(struct sock *sk, return 0; } - mmap_read_lock(current->mm); - - vma = vma_lookup(current->mm, address); - if (!vma || vma->vm_ops != &tcp_vm_ops) { - mmap_read_unlock(current->mm); + vma = find_tcp_vma(current->mm, address, &mmap_locked); + if (!vma) return -EINVAL; - } + vma_len = min_t(unsigned long, zc->length, vma->vm_end - address); avail_len = min_t(u32, vma_len, inq); total_bytes_to_map = avail_len & ~(PAGE_SIZE - 1); @@ -2159,7 +2181,10 @@ static int tcp_zerocopy_receive(struct sock *sk, zc, total_bytes_to_map); } out: - mmap_read_unlock(current->mm); + if (mmap_locked) + mmap_read_unlock(current->mm); + else + vma_end_read(vma); /* Try to copy straggler data. */ if (!ret) copylen = tcp_zc_handle_leftover(zc, sk, skb, &seq, copybuf_len, tss);