From patchwork Sun Sep 4 02:16:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 12965099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE76EC6FA82 for ; Sun, 4 Sep 2022 02:16:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FCEF80175; Sat, 3 Sep 2022 22:16:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AD348015A; Sat, 3 Sep 2022 22:16:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54EEB80175; Sat, 3 Sep 2022 22:16:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 442658015A for ; Sat, 3 Sep 2022 22:16:42 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2455140219 for ; Sun, 4 Sep 2022 02:16:42 +0000 (UTC) X-FDA: 79872789444.19.6BAB395 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf10.hostedemail.com (Postfix) with ESMTP id A3802C0059 for ; Sun, 4 Sep 2022 02:16:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662257801; x=1693793801; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xVQvjYo1znKhjHAgKaHx9Jqu5HLN8DrX7hpEaKrWHI0=; b=ZUZhLybVCfHKu+D2vwBgFxihVHvLHTdr8DSNIk4/P1VwvIdtq61seYN3 ddp0k4+j/7V3kXmKkeQ3y1bS1VPj6q0+hM5zfKBKt8SvIbA8k6L+JBDja 2wuCIGinhVIejn5DFbtuAcWWOM3mQ0yQwBSRpuv83LFV8z47gFAp9NMO3 uR9iAEndycXBdXO5OrqkNDI4ot4Q8eUipBdHCmCrNB0eEkSRCE6d6ZTyv 9aEaaCzR7tEJXuAA3GbCdEIpZE7Fm1912jPuMudJsABSkGEsv4rqshc/1 YJjwO8/BQUm8h4zsPJBPdnIdnk28LpVxlpxhcY79qyxehMIu23l2vv0Vd Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10459"; a="297501610" X-IronPort-AV: E=Sophos;i="5.93,288,1654585200"; d="scan'208";a="297501610" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Sep 2022 19:16:41 -0700 X-IronPort-AV: E=Sophos;i="5.93,288,1654585200"; d="scan'208";a="643384601" Received: from pg4-mobl3.amr.corp.intel.com (HELO dwillia2-xfh.jf.intel.com) ([10.212.132.198]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Sep 2022 19:16:41 -0700 Subject: [PATCH 07/13] fsdax: Manage pgmap references at entry insertion and deletion From: Dan Williams To: akpm@linux-foundation.org Cc: Matthew Wilcox , Jan Kara , "Darrick J. Wong" , Jason Gunthorpe , Christoph Hellwig , John Hubbard , linux-mm@kvack.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org Date: Sat, 03 Sep 2022 19:16:40 -0700 Message-ID: <166225780079.2351842.2977488896744053462.stgit@dwillia2-xfh.jf.intel.com> In-Reply-To: <166225775968.2351842.11156458342486082012.stgit@dwillia2-xfh.jf.intel.com> References: <166225775968.2351842.11156458342486082012.stgit@dwillia2-xfh.jf.intel.com> User-Agent: StGit/0.18-3-g996c MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=ZUZhLybV; spf=softfail (imf10.hostedemail.com: 134.134.136.65 is neither permitted nor denied by domain of dan.j.williams@intel.com) smtp.mailfrom=dan.j.williams@intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662257801; a=rsa-sha256; cv=none; b=aYSn782VB53qRzAGqRz29IpDzTiAHcb9Bu5W/cIB2qiDyQZ51XtwP7q+8JKSRSRyRxTigK Fspdr0zNzcXnSQ6Qpjl51SGvHJ1s+VfZ1JJhl3QfgfRZQ7GK/1Wd9zJtC74UMBNpwLDc9A 7KrCCw01rDi8OAEo2BvFgQFuN2Rn55g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662257801; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zrsd3AC/94bqT598M9+OFrl+WmLF1FtxxgeMeLW6DJ8=; b=mfk7CT064+2EXoEiniolPD97LD5FvhkuwC3bV6Siaf+ZHuIqDZd1ZT7HCFTKCKsgeyP7/b VrbxlSqxJ3STkwWQV4uks/2EWWEFq+jZWgyhqqD4eSRTZM1jeANo0IIlWV9fjAvzsVYDRP JVHTjkv/ewjrXBOxtDlWiccQWP6sCaY= X-Stat-Signature: rut9tayib7joib5tuuorag3om338baxb X-Rspamd-Queue-Id: A3802C0059 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=ZUZhLybV; spf=softfail (imf10.hostedemail.com: 134.134.136.65 is neither permitted nor denied by domain of dan.j.williams@intel.com) smtp.mailfrom=dan.j.williams@intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) X-Rspamd-Server: rspam07 X-HE-Tag: 1662257801-611943 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The percpu_ref in 'struct dev_pagemap' is used to coordinate active mappings of device-memory with the device-removal / unbind path. It enables the semantic that initiating device-removal (or device-driver-unbind) blocks new mapping and DMA attempts, and waits for mapping revocation or inflight DMA to complete. Expand the scope of the reference count to pin the DAX device active at mapping time and not later at the first gup event. With a device reference being held while any page on that device is mapped the need to manage pgmap reference counts in the gup code is eliminated. That cleanup is saved for a follow-on change. For now, teach dax_insert_entry() and dax_delete_mapping_entry() to take and drop pgmap references respectively. Cc: Matthew Wilcox Cc: Jan Kara Cc: "Darrick J. Wong" Cc: Jason Gunthorpe Cc: Christoph Hellwig Cc: John Hubbard Signed-off-by: Dan Williams --- fs/dax.c | 30 ++++++++++++++++++++++++------ include/linux/memremap.h | 18 ++++++++++++++---- mm/memremap.c | 13 ++++++++----- 3 files changed, 46 insertions(+), 15 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 65d55c5ecdef..b23222b0dae4 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -370,14 +370,24 @@ static inline void dax_mapping_set_cow(struct page *page) * whether this entry is shared by multiple files. If so, set the page->mapping * FS_DAX_MAPPING_COW, and use page->index as refcount. */ -static void dax_associate_entry(void *entry, struct address_space *mapping, - struct vm_fault *vmf, unsigned long flags) +static vm_fault_t dax_associate_entry(void *entry, + struct address_space *mapping, + struct vm_fault *vmf, unsigned long flags) { unsigned long size = dax_entry_size(entry), pfn, index; + struct dev_pagemap *pgmap; int i = 0; if (IS_ENABLED(CONFIG_FS_DAX_LIMITED)) - return; + return 0; + + if (!size) + return 0; + + pfn = dax_to_pfn(entry); + pgmap = get_dev_pagemap_many(pfn, NULL, PHYS_PFN(size)); + if (!pgmap) + return VM_FAULT_SIGBUS; index = linear_page_index(vmf->vma, ALIGN(vmf->address, size)); for_each_mapped_pfn(entry, pfn) { @@ -391,19 +401,27 @@ static void dax_associate_entry(void *entry, struct address_space *mapping, page->index = index + i++; } } + + return 0; } static void dax_disassociate_entry(void *entry, struct address_space *mapping, bool trunc) { - unsigned long pfn; + unsigned long size = dax_entry_size(entry), pfn; + struct page *page; if (IS_ENABLED(CONFIG_FS_DAX_LIMITED)) return; - for_each_mapped_pfn(entry, pfn) { - struct page *page = pfn_to_page(pfn); + if (!size) + return; + + page = pfn_to_page(dax_to_pfn(entry)); + put_dev_pagemap_many(page->pgmap, PHYS_PFN(size)); + for_each_mapped_pfn(entry, pfn) { + page = pfn_to_page(pfn); WARN_ON_ONCE(trunc && page_maybe_dma_pinned(page)); if (dax_mapping_is_cow(page->mapping)) { /* keep the CoW flag if this page is still shared */ diff --git a/include/linux/memremap.h b/include/linux/memremap.h index c3b4cc84877b..fd57407e7f3d 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -191,8 +191,13 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid); void memunmap_pages(struct dev_pagemap *pgmap); void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap); void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap); -struct dev_pagemap *get_dev_pagemap(unsigned long pfn, - struct dev_pagemap *pgmap); +struct dev_pagemap *get_dev_pagemap_many(unsigned long pfn, + struct dev_pagemap *pgmap, int refs); +static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn, + struct dev_pagemap *pgmap) +{ + return get_dev_pagemap_many(pfn, pgmap, 1); +} bool pgmap_pfn_valid(struct dev_pagemap *pgmap, unsigned long pfn); unsigned long vmem_altmap_offset(struct vmem_altmap *altmap); @@ -244,10 +249,15 @@ static inline unsigned long memremap_compat_align(void) } #endif /* CONFIG_ZONE_DEVICE */ -static inline void put_dev_pagemap(struct dev_pagemap *pgmap) +static inline void put_dev_pagemap_many(struct dev_pagemap *pgmap, int refs) { if (pgmap) - percpu_ref_put(&pgmap->ref); + percpu_ref_put_many(&pgmap->ref, refs); +} + +static inline void put_dev_pagemap(struct dev_pagemap *pgmap) +{ + put_dev_pagemap_many(pgmap, 1); } #endif /* _LINUX_MEMREMAP_H_ */ diff --git a/mm/memremap.c b/mm/memremap.c index 433500e955fb..4debe7b211ae 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -430,15 +430,16 @@ void vmem_altmap_free(struct vmem_altmap *altmap, unsigned long nr_pfns) } /** - * get_dev_pagemap() - take a new live reference on the dev_pagemap for @pfn + * get_dev_pagemap_many() - take new live references(s) on the dev_pagemap for @pfn * @pfn: page frame number to lookup page_map * @pgmap: optional known pgmap that already has a reference + * @refs: number of references to take * * If @pgmap is non-NULL and covers @pfn it will be returned as-is. If @pgmap * is non-NULL but does not cover @pfn the reference to it will be released. */ -struct dev_pagemap *get_dev_pagemap(unsigned long pfn, - struct dev_pagemap *pgmap) +struct dev_pagemap *get_dev_pagemap_many(unsigned long pfn, + struct dev_pagemap *pgmap, int refs) { resource_size_t phys = PFN_PHYS(pfn); @@ -454,13 +455,15 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn, /* fall back to slow path lookup */ rcu_read_lock(); pgmap = xa_load(&pgmap_array, PHYS_PFN(phys)); - if (pgmap && !percpu_ref_tryget_live(&pgmap->ref)) + if (pgmap && !percpu_ref_tryget_live_rcu(&pgmap->ref)) pgmap = NULL; + if (pgmap && refs > 1) + percpu_ref_get_many(&pgmap->ref, refs - 1); rcu_read_unlock(); return pgmap; } -EXPORT_SYMBOL_GPL(get_dev_pagemap); +EXPORT_SYMBOL_GPL(get_dev_pagemap_many); void free_zone_device_page(struct page *page) {