From patchwork Fri Sep 16 03:36:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12978068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D556C32771 for ; Fri, 16 Sep 2022 03:36:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF51480009; Thu, 15 Sep 2022 23:36:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA4E68D0002; Thu, 15 Sep 2022 23:36:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1F7380009; Thu, 15 Sep 2022 23:36:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AEEDA8D0002 for ; Thu, 15 Sep 2022 23:36:46 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8B8BF1A06C5 for ; Fri, 16 Sep 2022 03:36:46 +0000 (UTC) X-FDA: 79916536812.24.F58A43D Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf07.hostedemail.com (Postfix) with ESMTP id 085F240094 for ; Fri, 16 Sep 2022 03:36:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663299406; x=1694835406; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=R5iLP9zAHTpb9OkjwMC0Bw2xXqlB6pFYCCoagiNt1Hw=; b=mnaB6T4LghDIN7deuks9qBR2WV2AWID9qCPocoVsTWLRfj3vS3hV7tR5 xNSy1mhZjMDRR3KDVrdRCwn0FPogz2sjahV0Sf/u7Rmum1WQsBiA0Bfig 3fiwDhjjfYKFn+yPX7Uk1XuANZtAw9zark8rPCMsdgrL+x1evcBer9UJD hG1mvaGP520moZkc/vhCDg2w7y5K+j5RAle2pj0/Tkcc/KCRmWoTC1M+Z AEiJZc98eO8D+JypMOF6sn62/S35bBAYPLW1l1ynTaeXfb0lWl37s6BfC CT7Zz4JAPTUL48n2RLWWgjgbc8luf6SpZwSXJDm/H2e+nGazmEUFRXIBG Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="362867080" X-IronPort-AV: E=Sophos;i="5.93,319,1654585200"; d="scan'208";a="362867080" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 20:36:44 -0700 X-IronPort-AV: E=Sophos;i="5.93,319,1654585200"; d="scan'208";a="679809589" Received: from colinlix-mobl.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.209.29.52]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 20:36:43 -0700 Subject: [PATCH v2 16/18] mm/memremap_pages: Support initializing pages to a zero reference count From: Dan Williams To: akpm@linux-foundation.org Cc: Matthew Wilcox , Jan Kara , "Darrick J. Wong" , Jason Gunthorpe , Christoph Hellwig , John Hubbard , linux-fsdevel@vger.kernel.org, nvdimm@lists.linux.dev, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org Date: Thu, 15 Sep 2022 20:36:43 -0700 Message-ID: <166329940343.2786261.6047770378829215962.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <166329930818.2786261.6086109734008025807.stgit@dwillia2-xfh.jf.intel.com> References: <166329930818.2786261.6086109734008025807.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663299406; a=rsa-sha256; cv=none; b=IiUlWo7GbJOrpION6Eb21mrglFNVJzWh30qq1Kb/Juk+T22oV2CVnUAFL8V22fzkw8F7ra lbMI4/JeZYdReM02Sfia3OwecVuNshqvXDq1NEe/LHFdVKQ/VlagciYVs38EJ1+lUuzdCc 1PxkdnHR+RwIQ7cs0aOLBriQEY/D3fc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=mnaB6T4L; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of dan.j.williams@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663299406; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KPg/2jdS7BngvojjU/GNniiwhZh+I091wdut/ByZ5FY=; b=iWXpRN4YhtoW493vU9+k4wO38P7VKTJJnwYg4uVbuy5RS53cVKXkRQkW9UmBY/qMNfhaZh 8DF6D1H2LLTHNimgthZLUfj6Y/A3qNTrKknaZ5kP/kLFje7rNskZdy92V0xJsx0UHd7ckS SogVYDPXto1pbytAk5B8CB010hu0Fjk= X-Stat-Signature: a49gusc7tecmfpqddftqodb9p4i6n5ym X-Rspamd-Queue-Id: 085F240094 Authentication-Results: imf07.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=mnaB6T4L; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf07.hostedemail.com: domain of dan.j.williams@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com X-Rspamd-Server: rspam01 X-Rspam-User: X-HE-Tag: 1663299405-925076 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The initial memremap_pages() implementation inherited the __init_single_page() default of pages starting life with an elevated reference count. This originally allowed for the page->pgmap pointer to alias with the storage for page->lru since a page was only allowed to be on an lru list when its reference count was zero. Since then, 'struct page' definition cleanups have arranged for dedicated space for the ZONE_DEVICE page metadata, and the MEMORY_DEVICE_{PRIVATE,COHERENT} work has arranged for the 1 -> 0 page->_refcount transition to route the page to free_zone_device_page() and not the core-mm page-free. With those cleanups in place and with filesystem-dax and device-dax now converted to take and drop references at map and truncate time, it is possible to start MEMORY_DEVICE_FS_DAX and MEMORY_DEVICE_GENERIC reference counts at 0. MEMORY_DEVICE_{PRIVATE,COHERENT} still expect that their ZONE_DEVICE pages start life at _refcount 1, so make that the default if pgmap->init_mode is left at zero. Cc: Matthew Wilcox Cc: Jan Kara Cc: "Darrick J. Wong" Cc: Jason Gunthorpe Cc: Christoph Hellwig Cc: John Hubbard Signed-off-by: Dan Williams --- drivers/dax/device.c | 1 + drivers/nvdimm/pmem.c | 2 ++ include/linux/dax.h | 2 +- include/linux/memremap.h | 5 +++++ mm/memremap.c | 15 ++++++++++----- mm/page_alloc.c | 2 ++ 6 files changed, 21 insertions(+), 6 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 7f306939807e..8a7281d16c99 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -460,6 +460,7 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + pgmap->init_mode = INIT_PAGEMAP_IDLE; if (dev_dax->align > PAGE_SIZE) pgmap->vmemmap_shift = order_base_2(dev_dax->align >> PAGE_SHIFT); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 7e88cd242380..9c98dcb9f33d 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -529,6 +529,7 @@ static int pmem_attach_disk(struct device *dev, pmem->pfn_flags = PFN_DEV; if (is_nd_pfn(dev)) { pmem->pgmap.type = MEMORY_DEVICE_FS_DAX; + pmem->pgmap.init_mode = INIT_PAGEMAP_IDLE; pmem->pgmap.ops = &fsdax_pagemap_ops; addr = devm_memremap_pages(dev, &pmem->pgmap); pfn_sb = nd_pfn->pfn_sb; @@ -543,6 +544,7 @@ static int pmem_attach_disk(struct device *dev, pmem->pgmap.range.end = res->end; pmem->pgmap.nr_range = 1; pmem->pgmap.type = MEMORY_DEVICE_FS_DAX; + pmem->pgmap.init_mode = INIT_PAGEMAP_IDLE; pmem->pgmap.ops = &fsdax_pagemap_ops; addr = devm_memremap_pages(dev, &pmem->pgmap); pmem->pfn_flags |= PFN_MAP; diff --git a/include/linux/dax.h b/include/linux/dax.h index 3a27fecf072a..b9fdd8951e06 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -235,7 +235,7 @@ static inline void dax_unlock_mapping_entry(struct address_space *mapping, */ static inline bool dax_page_idle(struct page *page) { - return page_ref_count(page) == 1; + return page_ref_count(page) == 0; } bool dax_alive(struct dax_device *dax_dev); diff --git a/include/linux/memremap.h b/include/linux/memremap.h index e5d30eec3bf1..9f1a57efd371 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -116,6 +116,7 @@ struct dev_pagemap_ops { * representation. A bigger value will set up compound struct pages * of the requested order value. * @ops: method table + * @init_mode: initial reference count mode * @owner: an opaque pointer identifying the entity that manages this * instance. Used by various helpers to make sure that no * foreign ZONE_DEVICE memory is accessed. @@ -131,6 +132,10 @@ struct dev_pagemap { unsigned int flags; unsigned long vmemmap_shift; const struct dev_pagemap_ops *ops; + enum { + INIT_PAGEMAP_BUSY = 0, /* default / historical */ + INIT_PAGEMAP_IDLE, + } init_mode; void *owner; int nr_range; union { diff --git a/mm/memremap.c b/mm/memremap.c index 83c5e6fafd84..b6a7a95339b3 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -467,8 +467,10 @@ EXPORT_SYMBOL_GPL(get_dev_pagemap_many); void free_zone_device_page(struct page *page) { - if (WARN_ON_ONCE(!page->pgmap->ops || !page->pgmap->ops->page_free)) - return; + struct dev_pagemap *pgmap = page->pgmap; + + /* wake filesystem 'break dax layouts' waiters */ + wake_up_var(page); mem_cgroup_uncharge(page_folio(page)); @@ -503,12 +505,15 @@ void free_zone_device_page(struct page *page) * to clear page->mapping. */ page->mapping = NULL; - page->pgmap->ops->page_free(page); + if (pgmap->ops && pgmap->ops->page_free) + pgmap->ops->page_free(page); /* - * Reset the page count to 1 to prepare for handing out the page again. + * Reset the page count to the @init_mode value to prepare for + * handing out the page again. */ - set_page_count(page, 1); + if (pgmap->init_mode == INIT_PAGEMAP_BUSY) + set_page_count(page, 1); } #ifdef CONFIG_FS_DAX diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e5486d47406e..8ee52992055b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6719,6 +6719,8 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, { __init_single_page(page, pfn, zone_idx, nid); + if (pgmap->init_mode == INIT_PAGEMAP_IDLE) + set_page_count(page, 0); /* * Mark page reserved as it will need to wait for onlining