From patchwork Wed Sep 27 05:25:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13399935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E984AE80A89 for ; Wed, 27 Sep 2023 05:25:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BA658D007C; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 869E88D0002; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 759978D007C; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 60A5D8D0002 for ; Wed, 27 Sep 2023 01:25:19 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 35764A0F54 for ; Wed, 27 Sep 2023 05:25:19 +0000 (UTC) X-FDA: 81281239158.28.12FF488 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 9AB6640021 for ; Wed, 27 Sep 2023 05:25:17 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BSQTXxPD; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695792317; a=rsa-sha256; cv=none; b=YHW4REZpMBNB/FENtLXGvSvXDcTCbS852Sswz5XRhMx2pyejyvy4jozkwTq390fMA1haph YeZxN07RimpruUq11PgRwWcQZ/n2/sisY18hyv084widFO9WwsYaj95rkn2OxL+NLwVebN Yo/yMOM6eQTeHnZqhu/h5xomvqJ8nwU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BSQTXxPD; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695792317; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vACvD6A8iEuqKKhqCoG2NbL9weZtzxwL09B4YNYsPv4=; b=U742L2YsxgXUjMk1Z7w9NctxqKXIcvMKmTidWK3E5GL26WFqDcIWLuXUtLKn67K9zrPRp1 cNidF042/ywgkF1MjMRlT/JaQbCX+X9J8kmZVqVmuXNc6NR/aNuVEYTe9+qZUAEGgtA3Hr ezKKCN/RCBXKzekd9Heugvr4Cflxeog= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vACvD6A8iEuqKKhqCoG2NbL9weZtzxwL09B4YNYsPv4=; b=BSQTXxPDX+1Kl9mwmPhVKnB0Qo 8edzTyOe58NeuMrcsFv4tBjacPPzioLUtdkCLEhAOlwV73RJshNAK/RKmdA78JgFk3duPJ+Am4P+E GPnfH8IpPeAo/KHiK+cWKYnNCGqTip+ASc0apsJkJjWBebwT4gXrD/QjRFD9Ce/r3tCSJlJA/yevq xKKaMrDUz2BTQnTOM7IE04xi8kgWfTO9CErNXvnoFEl7Z32S950cf4jAD/sLAbKp3fOVkXPMqbPFL Pf51d24HsbQRKAISvJ/vZ7JG/5ieLLzrGMwI3lYdr/xs3C4cLSMkM9TSYKMw10+6qTRjaTi6XVH1u tj3B++YQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qlN2o-00Byws-Ls; Wed, 27 Sep 2023 05:25:06 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH 3/6] mm: Handle shared faults under the VMA lock Date: Wed, 27 Sep 2023 06:25:01 +0100 Message-Id: <20230927052505.2855872-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230927052505.2855872-1-willy@infradead.org> References: <20230927052505.2855872-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9AB6640021 X-Stat-Signature: 38x7ao55cet146f7jrb6k4k83ns9ospi X-HE-Tag: 1695792317-986811 X-HE-Meta: U2FsdGVkX18nKZP61u822h4jY71AhMNbcGmU3X3+Pw6u4XnoVJcQa+tXlqWWkR0S92gWGg2axoIv+sJapBFzRbgduVWjXbeuGzub8AFzhW5bQTSSeoLN37TDe2kWJubJ/Ky3ZiB++HVugI1xIc9zLb5fRn4n7sT4p1ve/beSuTt9HJW8L2h0liy8/CrcTBRe8Jn6gV5ndGkDPX3eSGGi8V/ZQ56KhmcfVw7V/CL7L2V5zGs7W3LPcn/C8ZFVpme88EqwFNWk+0MRhMhPAx9bOiH3zhS8O9P+dmS+9yRccqg29mihazxeIG8GYeMFf+0P8y+52BHU/iQ9AbUjs7CZ2s7APxn+yw2PY2jtDq5n8tdrhrMsMw0p26kTEUjk67Gvq9GtPEVkCJLqH/QETmrdNXElWfwb9S0zlBOoa7jvbsi3aYfK28OIqgLAQn3GwTKHouSsz1poAftos0rEgVrS6156b/6cKKRtYPZuI5HAI+q+KHfXwQlxAIl2yxwp1CG4J9Ub4+SEytfjrV+VMqpQ9IaM392SMHMJgpXaVY1amPCvNadGwkN9RLIgFClci67iN6YsM9W9ZOjnLbyXtzmQV1rKsKkzWGgrWgVRCy6LJpzgiSjn3koYWk5+VedvBuxMAPJZl3ReP0dv3uLOgQW7mZySCakmT1ATqwoYBINILCit6IiAkM+BTBnFZH2hYbypXANAbHKO/yrZlvXa/f+Ki8rA7e/CQePfIWpGMOwlqIP7m89zOgMRf8o2mVasL+StA1LJLtRN5Mc3y2y6oWylFPN2lHlQ6Q4NXcH9iARHapo2yZuhIT/66ltOz+tbCg6cX51O5vSf1MZbYRtqS4vRKf3mxuDGewRg4mOnGd6ps9De6H5oeeG5Q6winWsU0eeOJQhTgmur2XdL4ZlMDS4EVWq+GW8uDN27pFcQMlo4Lrtx9XrH03grkevrNzkhZRFE0E0Yl71w0GItp43llax 2H+EzLBD 9Iujh3aeDG84KOYwKjzwJ2g7E1yAhuXShiGV4CFpq2CaHGBRU5CkR3rR4QZ9e725z+ipfVo8QgJhE7Nvbik0D3Y6x84CN2a3Q6cTnh3gPeh+4Qj8mIX0+C06acyLn46khYp/MbTAyBzG02tj5NS0gps4Xqd8VAppY6gMEa2inHUr1i/TvyJlqRyrtDQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are many implementations of ->fault and some of them depend on mmap_lock being held. All vm_ops that implement ->map_pages() end up calling filemap_fault(), which I have audited to be sure it does not rely on mmap_lock. So (for now) key off ->map_pages existing as being a flag to indicate that it's safe to call ->fault while only holding the vma lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index cff78c496728..0f3da4889230 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +/* + * We could add a bitflag somewhere, but for now, we know that all + * vm_ops that have a ->map_pages have been audited and don't need + * the mmap_lock to be held. + */ +static inline vm_fault_t vmf_maybe_unlock_vma(const struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK)) + return 0; + vma_end_read(vma); + return VM_FAULT_RETRY; +} + static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -4669,10 +4684,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) vm_fault_t ret, tmp; struct folio *folio; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } + ret = vmf_maybe_unlock_vma(vmf); + if (ret) + return ret; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))