From patchwork Fri Aug 18 20:23:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13358314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D979EE4992 for ; Fri, 18 Aug 2023 20:23:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66AE6280066; Fri, 18 Aug 2023 16:23:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 619DF940012; Fri, 18 Aug 2023 16:23:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50857280066; Fri, 18 Aug 2023 16:23:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 41C19940012 for ; Fri, 18 Aug 2023 16:23:41 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 15973411B2 for ; Fri, 18 Aug 2023 20:23:41 +0000 (UTC) X-FDA: 81138351042.01.D860A58 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 345161C0008 for ; Fri, 18 Aug 2023 20:23:38 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ni7NRHRx; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692390219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sVojqgZC5Oa87bAeAa92zxKwzVUJMpto0yGZWvUp4SY=; b=MV5eUr+xj7b9qbdXEGjQiZD0xjNwCI8hNY6f6M9WxvybisWcDJqr9UF2QC6CPVyBLAwZ6F 3o9OHFxcZzYe/A2h+EJ2D9b4LBu/Ge0DjekKn6O9M9DIPivr67inQuEm5bfBDzJO53FLIE JIn870vLf/4pFrs2qZC/Ko0apq4OdHg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692390219; a=rsa-sha256; cv=none; b=t4KkLs/lbMJIHNEdPq3VwbiatpUhqG/ceBmeLQEO+e2WXV0+HLQ0aW1oE3mpTrBdrq5W9L cPL29UGIt/SIhfoDtoHQvS9sz9VXqor429irj5L1mlxYDrZKuLVNaeTkL7I8DJkV373+IQ x6qv/N4cismVtUVJdln7pmAxtAFP1V0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ni7NRHRx; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=sVojqgZC5Oa87bAeAa92zxKwzVUJMpto0yGZWvUp4SY=; b=Ni7NRHRx5b4t2IQB3qQ7Z/MJ+K rSPa32veDfubMM2LaVaFgaDZ9EG8WY7JCYvpiZd5wcDX3SJHlbcPS2R7PMoPQSRv7GMItrELodmcG 78UTPKkBQ/4W9BWT/7Bvn4kN4axTzsmjXtRn8QNlSoOMjvxZJY7FDrRL6VGJl4i7f7NNZGRSEuKdd ktnz/4HBjBdmapNOEB3mqZQhF8haDBsQN4X4RwHpt4w2DWfVj22IKjTVkVN9HSENKl6wAAS7e92SM BgkzDRvgln9Yn18HO5sw5HqewyrjfWuEEWNsSOcTYRmHHkNeC46ndMWYbw5iiN3q5HvChUV4xBEJ8 HfvWiwkw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qX60P-00BUiS-0x; Fri, 18 Aug 2023 20:23:37 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 1/3] mm: Move PMD_ORDER to pgtable.h Date: Fri, 18 Aug 2023 21:23:33 +0100 Message-Id: <20230818202335.2739663-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230818202335.2739663-1-willy@infradead.org> References: <20230818202335.2739663-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 345161C0008 X-Rspam-User: X-Stat-Signature: ff56n4g6rkq7198h6xm6zdnkau9hbgo8 X-Rspamd-Server: rspam03 X-HE-Tag: 1692390218-534615 X-HE-Meta: U2FsdGVkX1+1fp1DEufO9t1JwW/ASICqwdIYPXv5pzOJoGaidCN8Mcdo+Tqj/THAmwsU6Y/yvUW7oOTDmEXcbY+S5CRbiWUoPjiHe0fQETzXYy3600slqJZQ+XvZKEk9Wmh+vTP/MGxy53J2tCNBqXEWTKyWNqLblE/dj3bY0+6WRWQ4o5fyI/xQHeWNG2zn/BCUnC3djREy+V9Q3cHeKTMFV7LDUz8jlUwsUVzx/OrxUgCwI9DWVmvtRC+szo0uP2URHixBnfdQfvtwgMOKmtpwKjKRW19YeCVEXCSORpVKUw1ESjOzCNvxjGeL659kbB35TIGoNp/WqI4srrwe3bScUDgtImLxJA3nGgvxgfAGu9rkOCnbvuyzqXhpklvVpmhn96Pg64DLEknoge7qL/2CMgodHKNBFPAtazphQHivV+SsnM06XTcK1nk85FrC4F09k0qh8H8EqdzfuqmU5PyTC5UPUYgqMYRtPHUK9Epr2JovAfP8xlDZTNw/ytoC6420EZ8EfOivlutVOffqKghysGuv9792udOgzPJ0ZK/+Ui9838cmyrqJFFFN0R3LAk3e2xrqHaYMvyhKKFSAuN8MY04uv9LpCwLw8I9ZXyd+uz4cSIX9WwM5rwH1uPp0UvC1bvNce8w8ir8iOK2CguP/j+Pxf31nY9yuUgiqIkKl4VQLR2QLkYhpPvvqPddaXwBE1i5FRHGNmDPjL+PYQP6v7C1oub/FM55ee2yBhmTmWaO90VeYIAYGynURrxcaN3UrcTEwxX/RnwSX3xIQa+HZVw5FM2TlageKYGKuPQGI0WHSmRLDab1s96RNwAaj5FciPisLKsxzNv9XCiZiVjEDy1evHiuY5D3YZ4IxHNvB+A75Tw2j7UQUd9vky3U5aEX95j42Y3h9ALuQL9Vp/Z3JRy1SfQA0RDJES0Qq/gH4t61JydGa0b4bECHnsQBXNTxyBMwQybxrDvwxLXc 3MqmGNB9 UGOboc+o1X8FYZCjuM1G+IddgfkV/2u2SGQWAFVWAm+zxammEUsp5g4MmPj+RensUDx1kmfhrv00dMXTZi1lVVKCNg5ln0eX3ZLCCHjQMeovMUICUCmioCZ06b7/iRcTcRjsqxK4OaPsQ0STspbrfG1vaQ3aBR6c8w7VVJOfrB/k3KMtpyiGKvW0jWB4+boSfyU+LaQzCJ9rQcNQahFXEypM+VEVNBZReTryt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow this to be used in generic code. Also add PUD_ORDER. Signed-off-by: Matthew Wilcox (Oracle) --- fs/dax.c | 3 --- include/linux/pgtable.h | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 906ecbd541a3..88bb13643117 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -49,9 +49,6 @@ static inline unsigned int pe_order(enum page_entry_size pe_size) #define PG_PMD_COLOUR ((PMD_SIZE >> PAGE_SHIFT) - 1) #define PG_PMD_NR (PMD_SIZE >> PAGE_SHIFT) -/* The order of a PMD entry */ -#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) - static wait_queue_head_t wait_table[DAX_WAIT_TABLE_ENTRIES]; static int __init init_dax_wait_table(void) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index db4fe642b3e2..cb5c1fad1078 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -5,6 +5,9 @@ #include #include +#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) +#define PUD_ORDER (PUD_SHIFT - PAGE_SHIFT) + #ifndef __ASSEMBLY__ #ifdef CONFIG_MMU From patchwork Fri Aug 18 20:23:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13358315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D06A0EE4997 for ; Fri, 18 Aug 2023 20:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E30B940012; Fri, 18 Aug 2023 16:23:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 46BE3280067; Fri, 18 Aug 2023 16:23:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30E29940065; Fri, 18 Aug 2023 16:23:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2076C940012 for ; Fri, 18 Aug 2023 16:23:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id ACEE580D40 for ; Fri, 18 Aug 2023 20:23:43 +0000 (UTC) X-FDA: 81138351126.09.C14CE63 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 1EFE440005 for ; Fri, 18 Aug 2023 20:23:41 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="D/314gsG"; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692390222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eMSOQeab9BMo/9wblj2pEdrCD6yXcLeZBSZb6jVLwvY=; b=m7Qy5COpzwXZVgNeH29jQNsMIhAOFNa2gumyC2ih+5mROELdVA2CVBmH7UyVyO2l9HdgDR FX/uoyKjJbL5/XYTDOpcrd2e2k4PdQoglA0noMjCEY1sIStO2xsr6fStYzZqlxyb1a4ePG mi9UqAk8ijCzYi5671PpZ00YWbH466Y= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="D/314gsG"; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692390222; a=rsa-sha256; cv=none; b=E+A5jkIlZh/PbcdDRlW9XITtoJBwG549m6juPmFz0Kb/r1uYdZQ4ptkZRbtL0IN6rIfYZo Frj3oX/BcnkvIA8bLCQZYEIG3cJrZcdTqDrsMTpgRxKh3FRCO4TDWhZmSk1LEMGImHdh+h DLBEqxngdyW5R4ZiEePVrE/kld7r8ms= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=eMSOQeab9BMo/9wblj2pEdrCD6yXcLeZBSZb6jVLwvY=; b=D/314gsGws9n8E4pcG8TaD4Upb B/aeVE9EnQEU5U5bIS+9Wx7NMB1uw9QfrUAFgHVm8LNa6YVQhDwtQwfTw+6VL7fWIJ6FlsBedVVuM APDQUCF9Ww47RzqxQhpemYDKcdQ/PgyJ0zpEF5AOaYlRu9oODngH45xoPmSx/tMLeFmEGf5yVRq+g 1L/2ihXHtKbVfmnpPvpoWg2KKKjW+vK3sJFij/G2mZAa7Vn3HwNYjNsfwKtHBtJMWFtR/tYvkITQb fC9pzJwH8DBjXdKhHuNo4VsivOx5Q7cUam7Bd/6qO0BAxoHpkSpMo2Z0hEOnl2xG2UI2ey42Vxdh8 lBBbRcdQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qX60P-00BUiU-3u; Fri, 18 Aug 2023 20:23:37 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 2/3] mm: Allow ->huge_fault() to be called without the mmap_lock held Date: Fri, 18 Aug 2023 21:23:34 +0100 Message-Id: <20230818202335.2739663-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230818202335.2739663-1-willy@infradead.org> References: <20230818202335.2739663-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 8tu5syks6tzirximbagrjoqgh1wify4d X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1EFE440005 X-HE-Tag: 1692390221-897257 X-HE-Meta: U2FsdGVkX188X180g3SDDV1CgD8ixKQL39NJ4xx5LPbnHywAmsYpNidjJRsbAubqNHk3Gqllof8SyIOP9AsSjDHrgJd38piCoU2lWYmWTmU4jfxrDg66wvImOy1iTekgwRArkA372cAAyV9dlGq4bHMx06/hBPTUfRgvicj48Bn7TqlT/uS5qtsvByMbqgkO3pWKHIzvnmMOI1dfRoyLZo3Jl2s9tQVuyfZPV6pVK1EQdZ8DH6g4tD8I1Y++7/t54r7xzC9N4Giau1Qy3xrpJr4tOtkhABv00Z6s15t0AiVUWEV6CYCMStZA+J4yCRKsKbfe8tNm5HO61iwRZfs9YanI+7Bc+zm6++7ZrT+F+ORZwcUlPMdTsG+L+qBvARhmTOCcZCc0E5ramIvULYEodyqIK0YsxhjXl/cnwwwgTQmBXvVaPy5InG2uLqKQIEYiEgKsjGXka0ere0Mx5lqOQTBuKvs+gGA1QPQtPW9RHQyVbWpRmxVadACyktc+2sk7S0D0+P7FeFKvtas3Nly/pqANMUH6uTRl1A3craD82KhiynvNHEAlHzbWSgf74VoYBTCM2nrl++k5RiFZLgSlaHL/Zxsve06s6GAYAT0PQDGKBiivvNwp4HsL8rDfFs1GtmJjev6lHo+Joi8auuZn1UP27EL+PF2dcQIx+QwMqU7JlOqfN6mWnQKnXnFwp3bAL81SfjtTTUZFnd8fbjo+MBCkUwnVX8XC1l80wXO20pQdfBrc/30jpyVdq1bZffadEf+m4Rt4jBPRLyuJ8DH7tBcK7BKTuhbUos9cadSV6/hBiKDy+3oUyyafNA0LTBBaz0Z++GmRMilgG2xNYD80LAnKUiCBKsv+FRn3wzKEKYwOPa0rXXJ/pbhE3mw07ic5H3ghNnqC0ZrEPW0CLAGjkGgwYbEiU+hOg3+3QRHCwt+YkiqfWxlVTXTXM59Jk/WY/8NYcQSTkOlEnZH9ELU qAkM4cBv gcNkxhDF/FBAwT97W0U3v+tgL6Gvu6uCndBiab/RgYDu4otdKur5L4PfN3Mqud0IxJWhKRXgS61KdFVJxMy4YnEM9et+DUhS2mIN9UI9jCeNd8ZQQlaGl4YDOsHSE3RBm7IpG8khRmht5QnA073qMDPWFy5lQLDwO9/FnjmuGBipaK8x+rekIT06xAy7wMpVQbaFrRExH0lIhw2wePgddnW3C3URx09VIbJcF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the checks for the VMA lock being held, allowing the page fault path to call into the filesystem instead of retrying with the mmap_lock held. This will improve scalability for DAX page faults. Also update the documentation to match (and fix some other changes that have happened recently). Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/filesystems/locking.rst | 36 +++++++++++++++++---------- Documentation/filesystems/porting.rst | 11 ++++++++ mm/memory.c | 22 ++-------------- 3 files changed, 36 insertions(+), 33 deletions(-) diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index ab64356eff1a..7be2900806c8 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -635,26 +635,29 @@ vm_operations_struct prototypes:: - void (*open)(struct vm_area_struct*); - void (*close)(struct vm_area_struct*); - vm_fault_t (*fault)(struct vm_area_struct*, struct vm_fault *); + void (*open)(struct vm_area_struct *); + void (*close)(struct vm_area_struct *); + vm_fault_t (*fault)(struct vm_fault *); + vm_fault_t (*huge_fault)(struct vm_fault *, unsigned int order); + vm_fault_t (*map_pages)(struct vm_fault *, pgoff_t start, pgoff_t end); vm_fault_t (*page_mkwrite)(struct vm_area_struct *, struct vm_fault *); vm_fault_t (*pfn_mkwrite)(struct vm_area_struct *, struct vm_fault *); int (*access)(struct vm_area_struct *, unsigned long, void*, int, int); locking rules: -============= ========= =========================== +============= ========== =========================== ops mmap_lock PageLocked(page) -============= ========= =========================== -open: yes -close: yes -fault: yes can return with page locked -map_pages: read -page_mkwrite: yes can return with page locked -pfn_mkwrite: yes -access: yes -============= ========= =========================== +============= ========== =========================== +open: write +close: read/write +fault: read can return with page locked +huge_fault: maybe-read +map_pages: maybe-read +page_mkwrite: read can return with page locked +pfn_mkwrite: read +access: read +============= ========== =========================== ->fault() is called when a previously not present pte is about to be faulted in. The filesystem must find and return the page associated with the passed in @@ -664,6 +667,13 @@ then ensure the page is not already truncated (invalidate_lock will block subsequent truncate), and then return with VM_FAULT_LOCKED, and the page locked. The VM will unlock the page. +->huge_fault() is called when there is no PUD or PMD entry present. This +gives the filesystem the opportunity to install a PUD or PMD sized page. +Filesystems can also use the ->fault method to return a PMD sized page, +so implementing this function may not be necessary. In particular, +filesystems should not call filemap_fault() from ->huge_fault(). +The mmap_lock may not be held when this method is called. + ->map_pages() is called when VM asks to map easy accessible pages. Filesystem should find and map pages associated with offsets from "start_pgoff" till "end_pgoff". ->map_pages() is called with the RCU lock held and must diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index 0f5da78ef4f9..98969d713e2e 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -938,3 +938,14 @@ file pointer instead of struct dentry pointer. d_tmpfile() is similarly changed to simplify callers. The passed file is in a non-open state and on success must be opened before returning (e.g. by calling finish_open_simple()). + +--- + +**mandatory** + +Calling convention for ->huge_fault has changed. It now takes a page +order instead of an enum page_entry_size, and it may be called without the +mmap_lock held. All in-tree users have been audited and do not seem to +depend on the mmap_lock being held, but out of tree users should verify +for themselves. If they do need it, they can return VM_FAULT_RETRY to +be called with the mmap_lock held. diff --git a/mm/memory.c b/mm/memory.c index 3b4aaa0d2fff..254ee9c0e8c4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4873,13 +4873,8 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; if (vma_is_anonymous(vma)) return do_huge_pmd_anonymous_page(vmf); - if (vma->vm_ops->huge_fault) { - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } + if (vma->vm_ops->huge_fault) return vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); - } return VM_FAULT_FALLBACK; } @@ -4899,10 +4894,6 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { if (vma->vm_ops->huge_fault) { - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); if (!(ret & VM_FAULT_FALLBACK)) return ret; @@ -4923,13 +4914,8 @@ static vm_fault_t create_huge_pud(struct vm_fault *vmf) /* No support for anonymous transparent PUD pages yet */ if (vma_is_anonymous(vma)) return VM_FAULT_FALLBACK; - if (vma->vm_ops->huge_fault) { - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } + if (vma->vm_ops->huge_fault) return vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); - } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ return VM_FAULT_FALLBACK; } @@ -4946,10 +4932,6 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) goto split; if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { if (vma->vm_ops->huge_fault) { - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); if (!(ret & VM_FAULT_FALLBACK)) return ret; From patchwork Fri Aug 18 20:23:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13358317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 757D9EE4992 for ; Fri, 18 Aug 2023 20:23:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1399D280069; Fri, 18 Aug 2023 16:23:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EA70280067; Fri, 18 Aug 2023 16:23:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA59F280069; Fri, 18 Aug 2023 16:23:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DB70E280067 for ; Fri, 18 Aug 2023 16:23:49 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B4496C1111 for ; Fri, 18 Aug 2023 20:23:49 +0000 (UTC) X-FDA: 81138351378.16.31B23CE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id D9AB418001A for ; Fri, 18 Aug 2023 20:23:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BfrqYlMm; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692390228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IGCdtpVtb4NRh6pMmgyKhRtYamKEKXc9g3bMSYmI4rA=; b=M2Who8wL1pVlssQLKGDhvgq0tawz2Zk0pmRTkUFz4Bk5sFfZNviY0bg2zrCy6rpMTtWX95 ZwvcslP13y8po4zZikUz14vP6Wt5R6GAfigEUXlEWAji/dJPx73NI+fhXzs+7FqbCPr+4D NmsyC4dzLcjGTzzwITWnulexF7jNNKc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692390228; a=rsa-sha256; cv=none; b=Vhh/SaIQ0m2O5uuHpVqjrJqE16SwONZpZucEJC5fOkPHgEtfDzyfK4Thl0wem3nkduOdns Bddt4J/dtiS3Hp5W5EClaEL9u3zQL4FSPKMRR+VknJ6o6DxJt5eniiyDon8u37gSNMMidQ u6qN1puFdmOv0kK1jRvYUkGYbSfGusg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BfrqYlMm; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IGCdtpVtb4NRh6pMmgyKhRtYamKEKXc9g3bMSYmI4rA=; b=BfrqYlMmqycpmXf9zsZxW8ODKt /HF9stVk6l/xZjTMbtWHuGvwl79l+Z1nNGny+a4RQvrposzOhEPpJ3YBOXvy3K3H0LjQq4qP6Cgae fc3k89y+7p4JVukpwQqn3nEo7uySSfkhDu9JZ4mCo2ymSJ9gGABUitXLjSTPOHqWLadOS7mzlFrXv QvQXLO/IYz/WqNPt3UJDQZbGLPC1m9+MdHeI6DVM2ueZFn+F8CfT/eeAnWw/oZNSygdaCVW/G690N xpOok5aIRepqa0s7RMj0XYazudVWief4hOYp0/aJzT4ZpQ2zWp+j7zv+mBMq3BnP3nlvT1TZzb/fF 1fDQQZWg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qX60P-00BUiW-6c; Fri, 18 Aug 2023 20:23:37 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 3/3] mm: Remove enum page_entry_size Date: Fri, 18 Aug 2023 21:23:35 +0100 Message-Id: <20230818202335.2739663-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230818202335.2739663-1-willy@infradead.org> References: <20230818202335.2739663-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 5o39jc4wmn3o6jpx8tkyawpyraprxihe X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D9AB418001A X-Rspam-User: X-HE-Tag: 1692390227-560380 X-HE-Meta: U2FsdGVkX1/8zSP3xXEmzY6cpt0QD8p3TPKnpyIEF9j6xs5eishloZCFjzyNUQzpW3LgLg6gbLksueI9+VVE98MsQMgmWjEEW7vpA4wn6+/UbebWG9L80BtbWmuMmVa4q/dvnpJ3ZsSg2ogS75As62i+bDVLah7RdrLU/5SbFqSzv7ymEvk3fswveMUAh0SAS+zjNPKh2bllyX7nQaEdhp7a/VXARA+EefJBCeaJSoffcPBc+Vwyi0YR50xbs6A9vij4pz7itMtPHxiXmspwStmmHBQvOtiolb9uOjjTSOhbpRBtfAgb+ZdTCtMwDJeEX9F2O3+/ti/edhxuY4vnc+y9Puhp3/N6ZACBT58ItGrUZ1QxN+otVoBanweJyjmoZ2+Fq05Zne4WDXAN6OBA3qa/SerqyNm7Po1evLOyhDSoAqpkHRHdHMzUZJMgpYV4k023kfRGzB3klP2y77oS5/gTL4t3XCq8rY9HHjv3/toZFkm6/UEJxwj8eWs6T1qg70YR8s0OhcC/JkhTjrV51erYtF9VuiMEMcKeoJoPj6ZgHbUJVnHThE20KNeq6+gahilDIMyaVY3bMnA6oReljUz2/rDfQh4uiuw2sHNnCSAB62dvvErww9rR+lWx+bGufNKSO4bfb+25qLV21drxdAr8TZnCDGEyfcLbFJqM3B4xUubQpcJY93WGK3/g5wHC//11tARBP1z2K3SirhaL+QqxAmbAJcNFTetptNARS4mnvVPUCuB1pkgleYmRa7mYE7O39OsQYhJD7aM5XKDm5cn38h6ZFxSNyxhgzuAY8Jb1zypNvMmE/y35SVtRFd4Y2ML9jwfELs0H2BmvhSh7eV/H+8Hay4XbE4J1HzT3vXn7oYCAv6tzEtf1XLdqXYXuQXLF93fQ8ynzwjtpktOUmmIZUyjkg3C3gF+oGWb2j6eUlMa9c9o/gJn9Asos5rQhf/jRfFlBozWCAnHO8Gt jF6feQqZ uzYZ451RLGoHlfQ8FEkaxtC9cTgUz3sv+8gSp7zUcnF0kbBpHizzqx2RyU6oRQ3qbXr1M6LA+EDIgOZe9/O7H44aGDhf5SC/pFNCTfKQyDgd5ElYKQW+QTt0D5/JChUeU83f3KcKlu+TpfZFOvUPOC0lnfJBVnNF+mpkNO4kBj8ksbRNHVXSCiX2lnsJWa7m6imcnc/oJ2OIVd7WHEcke2gPYxbiexh8fnUjD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the unnecessary encoding of page order into an enum and pass the page order directly. That lets us get rid of pe_order(). The switch constructs have to be changed to if/else constructs to prevent GCC from warning on builds with 3-level page tables where PMD_ORDER and PUD_ORDER have the same value. If you are looking at this commit because your driver stopped compiling, look at the previous commit as well and audit your driver to be sure it doesn't depend on mmap_lock being held in its ->huge_fault method. Signed-off-by: Matthew Wilcox (Oracle) --- drivers/dax/device.c | 22 ++++++++-------------- fs/dax.c | 30 ++++++++---------------------- fs/erofs/data.c | 6 +++--- fs/ext2/file.c | 2 +- fs/ext4/file.c | 11 +++++------ fs/fuse/dax.c | 20 +++++++++----------- fs/xfs/xfs_file.c | 24 ++++++++++++------------ fs/xfs/xfs_trace.h | 22 ++++++++++------------ include/linux/dax.h | 4 ++-- include/linux/mm.h | 10 +--------- mm/memory.c | 8 ++++---- 11 files changed, 63 insertions(+), 96 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 30665a3ff6ea..93ebedc5ec8c 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -228,32 +228,26 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, - enum page_entry_size pe_size) +static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) { struct file *filp = vmf->vma->vm_file; vm_fault_t rc = VM_FAULT_SIGBUS; int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) size = %d\n", current->comm, + dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, pe_size); + vmf->vma->vm_start, vmf->vma->vm_end, order); id = dax_read_lock(); - switch (pe_size) { - case PE_SIZE_PTE: + if (order == 0) rc = __dev_dax_pte_fault(dev_dax, vmf); - break; - case PE_SIZE_PMD: + else if (order == PMD_ORDER) rc = __dev_dax_pmd_fault(dev_dax, vmf); - break; - case PE_SIZE_PUD: + else if (order == PUD_ORDER) rc = __dev_dax_pud_fault(dev_dax, vmf); - break; - default: + else rc = VM_FAULT_SIGBUS; - } dax_read_unlock(id); @@ -262,7 +256,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, static vm_fault_t dev_dax_fault(struct vm_fault *vmf) { - return dev_dax_huge_fault(vmf, PE_SIZE_PTE); + return dev_dax_huge_fault(vmf, 0); } static int dev_dax_may_split(struct vm_area_struct *vma, unsigned long addr) diff --git a/fs/dax.c b/fs/dax.c index 88bb13643117..8fafecbe42b1 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -30,17 +30,6 @@ #define CREATE_TRACE_POINTS #include -static inline unsigned int pe_order(enum page_entry_size pe_size) -{ - if (pe_size == PE_SIZE_PTE) - return PAGE_SHIFT - PAGE_SHIFT; - if (pe_size == PE_SIZE_PMD) - return PMD_SHIFT - PAGE_SHIFT; - if (pe_size == PE_SIZE_PUD) - return PUD_SHIFT - PAGE_SHIFT; - return ~0; -} - /* We choose 4096 entries - same as per-zone page wait tables */ #define DAX_WAIT_TABLE_BITS 12 #define DAX_WAIT_TABLE_ENTRIES (1 << DAX_WAIT_TABLE_BITS) @@ -1905,7 +1894,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, pfn_t *pfnp, /** * dax_iomap_fault - handle a page fault on a DAX file * @vmf: The description of the fault - * @pe_size: Size of the page to fault in + * @order: Order of the page to fault in * @pfnp: PFN to insert for synchronous faults if fsync is required * @iomap_errp: Storage for detailed error code in case of error * @ops: Iomap ops passed from the file system @@ -1915,17 +1904,15 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, pfn_t *pfnp, * has done all the necessary locking for page fault to proceed * successfully. */ -vm_fault_t dax_iomap_fault(struct vm_fault *vmf, enum page_entry_size pe_size, +vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order, pfn_t *pfnp, int *iomap_errp, const struct iomap_ops *ops) { - switch (pe_size) { - case PE_SIZE_PTE: + if (order == 0) return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, ops); - case PE_SIZE_PMD: + else if (order == PMD_ORDER) return dax_iomap_pmd_fault(vmf, pfnp, ops); - default: + else return VM_FAULT_FALLBACK; - } } EXPORT_SYMBOL_GPL(dax_iomap_fault); @@ -1976,19 +1963,18 @@ dax_insert_pfn_mkwrite(struct vm_fault *vmf, pfn_t pfn, unsigned int order) /** * dax_finish_sync_fault - finish synchronous page fault * @vmf: The description of the fault - * @pe_size: Size of entry to be inserted + * @order: Order of entry to be inserted * @pfn: PFN to insert * * This function ensures that the file range touched by the page fault is * stored persistently on the media and handles inserting of appropriate page * table entry. */ -vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf, - enum page_entry_size pe_size, pfn_t pfn) +vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf, unsigned int order, + pfn_t pfn) { int err; loff_t start = ((loff_t)vmf->pgoff) << PAGE_SHIFT; - unsigned int order = pe_order(pe_size); size_t len = PAGE_SIZE << order; err = vfs_fsync_range(vmf->vma->vm_file, start, start + len - 1, 1); diff --git a/fs/erofs/data.c b/fs/erofs/data.c index db5e4b7636ec..0c2c99c58b5e 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -413,14 +413,14 @@ const struct address_space_operations erofs_raw_access_aops = { #ifdef CONFIG_FS_DAX static vm_fault_t erofs_dax_huge_fault(struct vm_fault *vmf, - enum page_entry_size pe_size) + unsigned int order) { - return dax_iomap_fault(vmf, pe_size, NULL, NULL, &erofs_iomap_ops); + return dax_iomap_fault(vmf, order, NULL, NULL, &erofs_iomap_ops); } static vm_fault_t erofs_dax_fault(struct vm_fault *vmf) { - return erofs_dax_huge_fault(vmf, PE_SIZE_PTE); + return erofs_dax_huge_fault(vmf, 0); } static const struct vm_operations_struct erofs_dax_vm_ops = { diff --git a/fs/ext2/file.c b/fs/ext2/file.c index 0b4c91c62e1f..1039e5bf90af 100644 --- a/fs/ext2/file.c +++ b/fs/ext2/file.c @@ -103,7 +103,7 @@ static vm_fault_t ext2_dax_fault(struct vm_fault *vmf) } filemap_invalidate_lock_shared(inode->i_mapping); - ret = dax_iomap_fault(vmf, PE_SIZE_PTE, NULL, NULL, &ext2_iomap_ops); + ret = dax_iomap_fault(vmf, 0, NULL, NULL, &ext2_iomap_ops); filemap_invalidate_unlock_shared(inode->i_mapping); if (write) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index c457c8517f0f..2dc3f8301225 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -723,8 +723,7 @@ ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from) } #ifdef CONFIG_FS_DAX -static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, - enum page_entry_size pe_size) +static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, unsigned int order) { int error = 0; vm_fault_t result; @@ -740,7 +739,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, * read-only. * * We check for VM_SHARED rather than vmf->cow_page since the latter is - * unset for pe_size != PE_SIZE_PTE (i.e. only in do_cow_fault); for + * unset for order != 0 (i.e. only in do_cow_fault); for * other sizes, dax_iomap_fault will handle splitting / fallback so that * we eventually come back with a COW page. */ @@ -764,7 +763,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, } else { filemap_invalidate_lock_shared(mapping); } - result = dax_iomap_fault(vmf, pe_size, &pfn, &error, &ext4_iomap_ops); + result = dax_iomap_fault(vmf, order, &pfn, &error, &ext4_iomap_ops); if (write) { ext4_journal_stop(handle); @@ -773,7 +772,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, goto retry; /* Handling synchronous page fault? */ if (result & VM_FAULT_NEEDDSYNC) - result = dax_finish_sync_fault(vmf, pe_size, pfn); + result = dax_finish_sync_fault(vmf, order, pfn); filemap_invalidate_unlock_shared(mapping); sb_end_pagefault(sb); } else { @@ -785,7 +784,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, static vm_fault_t ext4_dax_fault(struct vm_fault *vmf) { - return ext4_dax_huge_fault(vmf, PE_SIZE_PTE); + return ext4_dax_huge_fault(vmf, 0); } static const struct vm_operations_struct ext4_dax_vm_ops = { diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c index 8e74f278a3f6..23904a6a9a96 100644 --- a/fs/fuse/dax.c +++ b/fs/fuse/dax.c @@ -784,8 +784,8 @@ static int fuse_dax_writepages(struct address_space *mapping, return dax_writeback_mapping_range(mapping, fc->dax->dev, wbc); } -static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, - enum page_entry_size pe_size, bool write) +static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, unsigned int order, + bool write) { vm_fault_t ret; struct inode *inode = file_inode(vmf->vma->vm_file); @@ -809,7 +809,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, * to populate page cache or access memory we are trying to free. */ filemap_invalidate_lock_shared(inode->i_mapping); - ret = dax_iomap_fault(vmf, pe_size, &pfn, &error, &fuse_iomap_ops); + ret = dax_iomap_fault(vmf, order, &pfn, &error, &fuse_iomap_ops); if ((ret & VM_FAULT_ERROR) && error == -EAGAIN) { error = 0; retry = true; @@ -818,7 +818,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, } if (ret & VM_FAULT_NEEDDSYNC) - ret = dax_finish_sync_fault(vmf, pe_size, pfn); + ret = dax_finish_sync_fault(vmf, order, pfn); filemap_invalidate_unlock_shared(inode->i_mapping); if (write) @@ -829,24 +829,22 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, static vm_fault_t fuse_dax_fault(struct vm_fault *vmf) { - return __fuse_dax_fault(vmf, PE_SIZE_PTE, - vmf->flags & FAULT_FLAG_WRITE); + return __fuse_dax_fault(vmf, 0, vmf->flags & FAULT_FLAG_WRITE); } -static vm_fault_t fuse_dax_huge_fault(struct vm_fault *vmf, - enum page_entry_size pe_size) +static vm_fault_t fuse_dax_huge_fault(struct vm_fault *vmf, unsigned int order) { - return __fuse_dax_fault(vmf, pe_size, vmf->flags & FAULT_FLAG_WRITE); + return __fuse_dax_fault(vmf, order, vmf->flags & FAULT_FLAG_WRITE); } static vm_fault_t fuse_dax_page_mkwrite(struct vm_fault *vmf) { - return __fuse_dax_fault(vmf, PE_SIZE_PTE, true); + return __fuse_dax_fault(vmf, 0, true); } static vm_fault_t fuse_dax_pfn_mkwrite(struct vm_fault *vmf) { - return __fuse_dax_fault(vmf, PE_SIZE_PTE, true); + return __fuse_dax_fault(vmf, 0, true); } static const struct vm_operations_struct fuse_dax_vm_ops = { diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 4f502219ae4f..203700278ddb 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1287,11 +1287,11 @@ xfs_file_llseek( static inline vm_fault_t xfs_dax_fault( struct vm_fault *vmf, - enum page_entry_size pe_size, + unsigned int order, bool write_fault, pfn_t *pfn) { - return dax_iomap_fault(vmf, pe_size, pfn, NULL, + return dax_iomap_fault(vmf, order, pfn, NULL, (write_fault && !vmf->cow_page) ? &xfs_dax_write_iomap_ops : &xfs_read_iomap_ops); @@ -1300,7 +1300,7 @@ xfs_dax_fault( static inline vm_fault_t xfs_dax_fault( struct vm_fault *vmf, - enum page_entry_size pe_size, + unsigned int order, bool write_fault, pfn_t *pfn) { @@ -1322,14 +1322,14 @@ xfs_dax_fault( static vm_fault_t __xfs_filemap_fault( struct vm_fault *vmf, - enum page_entry_size pe_size, + unsigned int order, bool write_fault) { struct inode *inode = file_inode(vmf->vma->vm_file); struct xfs_inode *ip = XFS_I(inode); vm_fault_t ret; - trace_xfs_filemap_fault(ip, pe_size, write_fault); + trace_xfs_filemap_fault(ip, order, write_fault); if (write_fault) { sb_start_pagefault(inode->i_sb); @@ -1340,9 +1340,9 @@ __xfs_filemap_fault( pfn_t pfn; xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED); - ret = xfs_dax_fault(vmf, pe_size, write_fault, &pfn); + ret = xfs_dax_fault(vmf, order, write_fault, &pfn); if (ret & VM_FAULT_NEEDDSYNC) - ret = dax_finish_sync_fault(vmf, pe_size, pfn); + ret = dax_finish_sync_fault(vmf, order, pfn); xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED); } else { if (write_fault) { @@ -1373,7 +1373,7 @@ xfs_filemap_fault( struct vm_fault *vmf) { /* DAX can shortcut the normal fault path on write faults! */ - return __xfs_filemap_fault(vmf, PE_SIZE_PTE, + return __xfs_filemap_fault(vmf, 0, IS_DAX(file_inode(vmf->vma->vm_file)) && xfs_is_write_fault(vmf)); } @@ -1381,13 +1381,13 @@ xfs_filemap_fault( static vm_fault_t xfs_filemap_huge_fault( struct vm_fault *vmf, - enum page_entry_size pe_size) + unsigned int order) { if (!IS_DAX(file_inode(vmf->vma->vm_file))) return VM_FAULT_FALLBACK; /* DAX can shortcut the normal fault path on write faults! */ - return __xfs_filemap_fault(vmf, pe_size, + return __xfs_filemap_fault(vmf, order, xfs_is_write_fault(vmf)); } @@ -1395,7 +1395,7 @@ static vm_fault_t xfs_filemap_page_mkwrite( struct vm_fault *vmf) { - return __xfs_filemap_fault(vmf, PE_SIZE_PTE, true); + return __xfs_filemap_fault(vmf, 0, true); } /* @@ -1408,7 +1408,7 @@ xfs_filemap_pfn_mkwrite( struct vm_fault *vmf) { - return __xfs_filemap_fault(vmf, PE_SIZE_PTE, true); + return __xfs_filemap_fault(vmf, 0, true); } static const struct vm_operations_struct xfs_file_vm_ops = { diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h index f3cc204bb4bf..1904eaf7a2e9 100644 --- a/fs/xfs/xfs_trace.h +++ b/fs/xfs/xfs_trace.h @@ -802,36 +802,34 @@ DEFINE_INODE_EVENT(xfs_inode_inactivating); * ring buffer. Somehow this was only worth mentioning in the ftrace sample * code. */ -TRACE_DEFINE_ENUM(PE_SIZE_PTE); -TRACE_DEFINE_ENUM(PE_SIZE_PMD); -TRACE_DEFINE_ENUM(PE_SIZE_PUD); +TRACE_DEFINE_ENUM(PMD_ORDER); +TRACE_DEFINE_ENUM(PUD_ORDER); TRACE_DEFINE_ENUM(XFS_REFC_DOMAIN_SHARED); TRACE_DEFINE_ENUM(XFS_REFC_DOMAIN_COW); TRACE_EVENT(xfs_filemap_fault, - TP_PROTO(struct xfs_inode *ip, enum page_entry_size pe_size, - bool write_fault), - TP_ARGS(ip, pe_size, write_fault), + TP_PROTO(struct xfs_inode *ip, unsigned int order, bool write_fault), + TP_ARGS(ip, order, write_fault), TP_STRUCT__entry( __field(dev_t, dev) __field(xfs_ino_t, ino) - __field(enum page_entry_size, pe_size) + __field(unsigned int, order) __field(bool, write_fault) ), TP_fast_assign( __entry->dev = VFS_I(ip)->i_sb->s_dev; __entry->ino = ip->i_ino; - __entry->pe_size = pe_size; + __entry->order = order; __entry->write_fault = write_fault; ), TP_printk("dev %d:%d ino 0x%llx %s write_fault %d", MAJOR(__entry->dev), MINOR(__entry->dev), __entry->ino, - __print_symbolic(__entry->pe_size, - { PE_SIZE_PTE, "PTE" }, - { PE_SIZE_PMD, "PMD" }, - { PE_SIZE_PUD, "PUD" }), + __print_symbolic(__entry->order, + { 0, "PTE" }, + { PMD_ORDER, "PMD" }, + { PUD_ORDER, "PUD" }), __entry->write_fault) ) diff --git a/include/linux/dax.h b/include/linux/dax.h index 261944ec0887..22cd9902345d 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -241,10 +241,10 @@ void dax_flush(struct dax_device *dax_dev, void *addr, size_t size); ssize_t dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter, const struct iomap_ops *ops); -vm_fault_t dax_iomap_fault(struct vm_fault *vmf, enum page_entry_size pe_size, +vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order, pfn_t *pfnp, int *errp, const struct iomap_ops *ops); vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf, - enum page_entry_size pe_size, pfn_t pfn); + unsigned int order, pfn_t pfn); int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index); int dax_invalidate_mapping_entry_sync(struct address_space *mapping, pgoff_t index); diff --git a/include/linux/mm.h b/include/linux/mm.h index 3f764e84e567..840bae5f23b6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -551,13 +551,6 @@ struct vm_fault { */ }; -/* page entry size for vm->huge_fault() */ -enum page_entry_size { - PE_SIZE_PTE = 0, - PE_SIZE_PMD, - PE_SIZE_PUD, -}; - /* * These are the virtual MM functions - opening of an area, closing and * unmapping it (needed to keep files on disk up-to-date etc), pointer @@ -581,8 +574,7 @@ struct vm_operations_struct { int (*mprotect)(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long newflags); vm_fault_t (*fault)(struct vm_fault *vmf); - vm_fault_t (*huge_fault)(struct vm_fault *vmf, - enum page_entry_size pe_size); + vm_fault_t (*huge_fault)(struct vm_fault *vmf, unsigned int order); vm_fault_t (*map_pages)(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff); unsigned long (*pagesize)(struct vm_area_struct * area); diff --git a/mm/memory.c b/mm/memory.c index 254ee9c0e8c4..2947fbc558f6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4874,7 +4874,7 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) if (vma_is_anonymous(vma)) return do_huge_pmd_anonymous_page(vmf); if (vma->vm_ops->huge_fault) - return vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); + return vma->vm_ops->huge_fault(vmf, PMD_ORDER); return VM_FAULT_FALLBACK; } @@ -4894,7 +4894,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { if (vma->vm_ops->huge_fault) { - ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); + ret = vma->vm_ops->huge_fault(vmf, PMD_ORDER); if (!(ret & VM_FAULT_FALLBACK)) return ret; } @@ -4915,7 +4915,7 @@ static vm_fault_t create_huge_pud(struct vm_fault *vmf) if (vma_is_anonymous(vma)) return VM_FAULT_FALLBACK; if (vma->vm_ops->huge_fault) - return vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + return vma->vm_ops->huge_fault(vmf, PUD_ORDER); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ return VM_FAULT_FALLBACK; } @@ -4932,7 +4932,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) goto split; if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { if (vma->vm_ops->huge_fault) { - ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + ret = vma->vm_ops->huge_fault(vmf, PUD_ORDER); if (!(ret & VM_FAULT_FALLBACK)) return ret; }