From patchwork Tue May 7 23:56:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933895 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 41FDD16C1 for ; Wed, 8 May 2019 00:09:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32E4A28643 for ; Wed, 8 May 2019 00:09:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 27482286A0; Wed, 8 May 2019 00:09:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 118442864E for ; Wed, 8 May 2019 00:09:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 058FF6B0006; Tue, 7 May 2019 20:09:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 008FE6B0008; Tue, 7 May 2019 20:09:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3BB06B000A; Tue, 7 May 2019 20:09:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id AEB946B0006 for ; Tue, 7 May 2019 20:09:53 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id f1so3243368pfb.0 for ; Tue, 07 May 2019 17:09:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=QPUnx65moTf9orYAxQfca8uCnxz2On+XPnlOyTMkP5g=; b=QEMg49i3xcWwFKsSl09MymEr0BcK1oN1w1i/dBEULfmzk2QS8vLcUQ/qp90fbFzCpJ nwH0JfktcOzzz5w8ZOpioRcy/0QJNdXtekrMczvKV+8D0kdwDMbW/1sZo52MCFWK9vqL LYQxmH/MvvoTXt/yy5pX3ji1qq0+2Zj3gXjeQLriZLVLI559BYRKH1dqYZogJ35QBaMT z/u6H/T2Eucb06X5vnP7VF/JDZlkyxvZce/X4bz3Rm7jAzZ0NIKW8IzAwstnqVWh7vy8 2GOA9zqNcVC2cqB9Stkqcxl+556BC/Rw+XavABXMC4w9c9JZffRDq1VCEG2vsApWTT5q QUUg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAV8HFio8pN/ZvCXW3VlvWwbZnxj+xfcPIOhtNU08b4uOvljfBI4 5udCO3jzFxvkIet7qj3vGZsojtwWSD5xD+vTIuRrX8Uhyu4mWHdvaoweDDcd8u0RkArtHrPZhsQ aMgwPvCODv3GrHu2HloFaWWyDCYDM9DWmT+/ayBnGN/Z64Jou5cinq9jTUcpo5kDZnw== X-Received: by 2002:a63:754b:: with SMTP id f11mr4138275pgn.32.1557274193359; Tue, 07 May 2019 17:09:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqzYPsfFvC5Z/JP9x2hEpNKjuhnHhhda5NLijjVhbvfCkaD1ye2ZK0kObp8uUONvtXvQniS1 X-Received: by 2002:a63:754b:: with SMTP id f11mr4138208pgn.32.1557274192527; Tue, 07 May 2019 17:09:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274192; cv=none; d=google.com; s=arc-20160816; b=RsiJSpVaFoMU13ddlmwa/8BItmDoJhTsZxvf7MpbdLAPdQ7fw1QDZnW2OuZsVpISz+ rlPibrFNiqMnGXeezGnR/JKWhjRPVG2lw491wflCPRwKMb6pBeqQuk5USF+ExBWCYedE JN03lbF99UTZwwGuvU469vM103AJ74TuBWLaVRPh/jwH/A0bEabV/aqut1gkWQ09kwsQ Xk5ZmI3uB8MsGo0bnhZCioEFkxFGz4IB2AYE3mfnIUo3HcZhRh667P8xD3gUWS9xmYYx rk+0cz2Acj3QhOn8stDamZxNvjv/Z/TdUbCzAbv1VDPGUyZ5Knjqu0g7w6KG5RbPtST2 QJWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=QPUnx65moTf9orYAxQfca8uCnxz2On+XPnlOyTMkP5g=; b=Dyv5M1cWKiSKKNujd2WaNpxgPe0q6nOlVFaBtMoQGGAylJJVWClis8Ky9TI+QcDPPI jcWSOs+YMgjbXLnjrn5PSoP58XjwifyUCQwB0vaFTLtqz8AVZOZJ4ULZMVvHGQvE5GZr oIE2+JfhVEl1LAm3tNJVodcnmrLTULqu8E4hGdug8BlHIb0sG/H/e47ZDXNBzqtMo4su k0RrWdFZJDwZ4N+KlCLnw7VDwNGbp1b+n+hPvMPmGVxr7C8o9KOEkoRCbhtP9LXLKZgB T+wDgzd3Jve6NegEE/bLZxDUF7djVhDzvx/sBZnDFq8J2vJcK2H+Fbr4msgS/KXq8wBe oPjg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id 144si11128746pgh.524.2019.05.07.17.09.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:09:52 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:09:52 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga001.fm.intel.com with ESMTP; 07 May 2019 17:09:51 -0700 Subject: [PATCH v2 1/6] drivers/base/devres: Introduce devm_release_action() From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Bjorn Helgaas , Christoph Hellwig , Greg Kroah-Hartman , "Rafael J. Wysocki" , Ira Weiny , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:05 -0700 Message-ID: <155727336530.292046.2926860263201336366.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The devm_add_action() facility allows a resource allocation routine to add custom devm semantics. One such user is devm_memremap_pages(). There is now a need to manually trigger devm_memremap_pages_release(). Introduce devm_release_action() so the release action can be triggered via a new devm_memunmap_pages() api in a follow-on change. Cc: Logan Gunthorpe Cc: Bjorn Helgaas Cc: Christoph Hellwig Cc: Greg Kroah-Hartman Cc: "Rafael J. Wysocki" Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- drivers/base/devres.c | 24 +++++++++++++++++++++++- include/linux/device.h | 1 + 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/base/devres.c b/drivers/base/devres.c index e038e2b3b7ea..0bbb328bd17f 100644 --- a/drivers/base/devres.c +++ b/drivers/base/devres.c @@ -755,10 +755,32 @@ void devm_remove_action(struct device *dev, void (*action)(void *), void *data) WARN_ON(devres_destroy(dev, devm_action_release, devm_action_match, &devres)); - } EXPORT_SYMBOL_GPL(devm_remove_action); +/** + * devm_release_action() - release previously added custom action + * @dev: Device that owns the action + * @action: Function implementing the action + * @data: Pointer to data passed to @action implementation + * + * Releases and removes instance of @action previously added by + * devm_add_action(). Both action and data should match one of the + * existing entries. + */ +void devm_release_action(struct device *dev, void (*action)(void *), void *data) +{ + struct action_devres devres = { + .data = data, + .action = action, + }; + + WARN_ON(devres_release(dev, devm_action_release, devm_action_match, + &devres)); + +} +EXPORT_SYMBOL_GPL(devm_release_action); + /* * Managed kmalloc/kfree */ diff --git a/include/linux/device.h b/include/linux/device.h index 4e6987e11f68..6d7fd5370f3d 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -713,6 +713,7 @@ void __iomem *devm_of_iomap(struct device *dev, /* allows to add/remove a custom action to devres stack */ int devm_add_action(struct device *dev, void (*action)(void *), void *data); void devm_remove_action(struct device *dev, void (*action)(void *), void *data); +void devm_release_action(struct device *dev, void (*action)(void *), void *data); static inline int devm_add_action_or_reset(struct device *dev, void (*action)(void *), void *data) From patchwork Tue May 7 23:56:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC2631575 for ; Wed, 8 May 2019 00:10:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBD3026256 for ; Wed, 8 May 2019 00:10:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C051728643; Wed, 8 May 2019 00:10:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA3A728485 for ; Wed, 8 May 2019 00:10:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA38D6B0008; Tue, 7 May 2019 20:09:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B532B6B000A; Tue, 7 May 2019 20:09:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6B166B000C; Tue, 7 May 2019 20:09:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 70C3C6B0008 for ; Tue, 7 May 2019 20:09:59 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id f3so10316947plb.17 for ; Tue, 07 May 2019 17:09:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=rly5zRrCSESMX4aeor/m6oZVFemBQhV/3bqm72CMmro=; b=evZAC5d/qg/tbJjd9o/3TGPnt56Uj3OdKt6Djw/+YnRnttXDhpdcfL8MaLN+lesv3z LveaQHyqFI5qAlPRCv7BiQF1S6GIlfOCMpY2Fo4IkHZGKYz/P02slPV6Ob9qvvM8I08k p58VM+v/K0XkDbSVe0hw0mNjc2hyro7FhkoVIc0KNYComzol3rYTYtO7WkqCaWwU8cY/ BwGfdGcStsClELZQsC1WLGgGhIkQrB4sJuItf3jhzhiCRWEMzkaj19YEMbC+hMBHCvVd 0ev0cOeFo/ARdkBxVX4D1U1CwiM0huMtgPZrWL0gqZVHzkfCxJKtmA75FZM4A6GPhkti IKNA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWWgq3/F2cW87g+oy1B3JCGP1mZHSLJ7urOdaBxd7zfTjPc4mXL hZ7N474qesdHEJFYasnOuxDA1TXRfbdL0fPVk1FHhuuKm2omT9jtDQHVBnkqPusJzma6kj8+OPh n852+7kOzMjP+mayfJcqXGJfCrK2KIO9Hqv8iDWHy9ST/yXJ6uGwrUyw0xGUL1ZdCfQ== X-Received: by 2002:a62:1d0d:: with SMTP id d13mr44869428pfd.96.1557274199121; Tue, 07 May 2019 17:09:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqxPOGVo3nFokpjOzBT/S1s1YhcIDRT9V96XyQ+LfHv+Z7Yf0dbY7P4tmkkJlYMSH7j4J3JY X-Received: by 2002:a62:1d0d:: with SMTP id d13mr44869353pfd.96.1557274198127; Tue, 07 May 2019 17:09:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274198; cv=none; d=google.com; s=arc-20160816; b=JayrvN/+XTB1e/gl9DutF70zEhapyObC0I3RhqU3GG/tzQvS7hrrrl/HZPO1fAl9cQ tu2ogPCGauTrL5f/oQR9uWmfYWA0ZclgzM+4xTJ93cVbMMMg2kt3zOK6GgM5X5QZnGUI K5zsOw/S5znkMqgxrNkD0MBBR4Ab8wrERxY/4Qn3y7W77mdARF4JX2d3NZPZnSOc3rgm klNqXBssVtfJoQ+juHrXJObgErBJ30/MWwN58fa+hidVP9l2ktjQb/0Ps3C2MTE7nmOO Z7K4AfpAaEdFRPCFQyixlcyO0HgJeTczcigCKdkMvddRjKlVuePEy9nQTnuvWWEQ/+qy hJ1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=rly5zRrCSESMX4aeor/m6oZVFemBQhV/3bqm72CMmro=; b=THYb4QIKotpZjgOk9SCJ7jR/hx7hmElhnT8Jc0/G03lZobUxy24ipZ0iywJ/2pChs7 NaJh96ARtuitbpdeDFzIdOSycK2Y97U3dDCtSxbLX5v/hfy+XkmpSPb8RIIjGt3CMtoh wPd7X78L4AuoDMY2kPHWLZk/6PnsrKo91s1Nk+H/bii2u7srvTcsn6vHwsEbyXEadwq9 Yv4gDQuUmSWOxtSIQ0kuQemRahOMoko53s1UzTR5SEueJXs5/I5tVzVSV7wC4OhCbOl7 YIIANqxwUcWSrOq7ZHcsF4tcdQskbABASwmWOXhH3ebYx75s8GeTkqfXTO1xl5Geq9IF QlyA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga14.intel.com (mga14.intel.com. [192.55.52.115]) by mx.google.com with ESMTPS id v20si13622756pgn.266.2019.05.07.17.09.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:09:58 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.115 as permitted sender) client-ip=192.55.52.115; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:09:57 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga002.jf.intel.com with ESMTP; 07 May 2019 17:09:57 -0700 Subject: [PATCH v2 2/6] mm/devm_memremap_pages: Introduce devm_memunmap_pages From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Bjorn Helgaas , Christoph Hellwig , Ira Weiny , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:10 -0700 Message-ID: <155727337088.292046.5774214552136776763.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use the new devm_relase_action() facility to allow devm_memremap_pages_release() to be manually triggered. Cc: Logan Gunthorpe Cc: Bjorn Helgaas Cc: Christoph Hellwig Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- include/linux/memremap.h | 6 ++++++ kernel/memremap.c | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index f0628660d541..7601ee314c4a 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -100,6 +100,7 @@ struct dev_pagemap { #ifdef CONFIG_ZONE_DEVICE void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap); +void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap); struct dev_pagemap *get_dev_pagemap(unsigned long pfn, struct dev_pagemap *pgmap); @@ -118,6 +119,11 @@ static inline void *devm_memremap_pages(struct device *dev, return ERR_PTR(-ENXIO); } +static inline void devm_memunmap_pages(struct device *dev, + struct dev_pagemap *pgmap) +{ +} + static inline struct dev_pagemap *get_dev_pagemap(unsigned long pfn, struct dev_pagemap *pgmap) { diff --git a/kernel/memremap.c b/kernel/memremap.c index a856cb5ff192..65afbacab44e 100644 --- a/kernel/memremap.c +++ b/kernel/memremap.c @@ -266,6 +266,12 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) } EXPORT_SYMBOL_GPL(devm_memremap_pages); +void devm_memunmap_pages(struct device *dev, struct dev_pagemap *pgmap) +{ + devm_release_action(dev, devm_memremap_pages_release, pgmap); +} +EXPORT_SYMBOL_GPL(devm_memunmap_pages); + unsigned long vmem_altmap_offset(struct vmem_altmap *altmap) { /* number of pfns from base where pfn_to_page() is valid */ From patchwork Tue May 7 23:56:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EB52912 for ; Wed, 8 May 2019 00:10:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F7E42873B for ; Wed, 8 May 2019 00:10:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 136CE2875F; Wed, 8 May 2019 00:10:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88DA3286A0 for ; Wed, 8 May 2019 00:10:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2D716B000A; Tue, 7 May 2019 20:10:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CB3FF6B000C; Tue, 7 May 2019 20:10:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7AD16B000E; Tue, 7 May 2019 20:10:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 7CEDE6B000A for ; Tue, 7 May 2019 20:10:04 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id a90so10331788plc.7 for ; Tue, 07 May 2019 17:10:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=W6mjT0eRfCwfLc+G2YCpsPlEeRwl807aZwyCVJxSnaI=; b=oRCdgABVztA/NFEAxObfzZDRdrjoVrpkMYA+wLlf4Hw4v+Xt/ea2dASbrN2Kwh7N8K P+zd5auMTd6CYZru69DFe7X86Dc7ZuKaBNBRoq1f2KaWheQNvM7bOu4QVuE/w/pffKNb zqjsQLaunuqqzi3UDoOCnSoGJwUh584ZUFdRhJLMYc0J9+2Pq1X9I75RNY90hPbvdCbM kuuZm+vXBdBUlPvPjHdmmGI0CMfesatzK5C9JkN9QrM5DEdHaQJ+BodLCWi5RcbNOGub ulWMo/RtdgTb7BfFFGcdbovDU6UvJ8rmfVXwRpMsu23aX1oeC87FcFFm1cynTPBChC+0 MluQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWup8JvVpvGSGU1eEizxoy6xUngVCT5zLISZZgs+909j5MULV8j 9nWF98fij80TfAlrCpc5W2xbzIbnV4xpeQWDojMWChatk8u8Za9WxlSBeuSeed0nLvx7Dj11yiG JAxSGlZ8YmF0SuE4AYe/e4m7dVvzcp/An41rXqr39Yp4HV+3uvA/gOkHw+TRMNTxQsA== X-Received: by 2002:a62:1ec5:: with SMTP id e188mr45388417pfe.242.1557274204171; Tue, 07 May 2019 17:10:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqzpOt8lqHoMGTjz85m67yFEueLy+oZJd+tFF2kNST4RExQA8G1mDl8aB1XOiRduwG3xxjAP X-Received: by 2002:a62:1ec5:: with SMTP id e188mr45388335pfe.242.1557274203238; Tue, 07 May 2019 17:10:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274203; cv=none; d=google.com; s=arc-20160816; b=br6fDSzAtBsls+RwT9Zle0A+toi5tC7sG7ynaBwRKyiu91qhzqezmTvXZ0Nb+syAgq NO+sC+vMBAzQzLuB6R81zgi/lFn7eTZQxA/WEp2GuA+ERN4sv3c1zb3IlbVnb+gzGhUh +Z/OXj3Eak+zCgp07U43R1jr6ytjxuRWameAdy5Hl9C34Yn+mATHbf/KYDx63+VGxx8e EFCeOxcx5BsvO/dc1u0ckVcfxN6vjygyRkPF44+gyRyDDe8VLxdEUnONeWEVIHR6SWQH Rhl4cYiDrpI/5wKljhJB/+ty8TGkv779KRELI4hyYpj/+Zv5CHMwTzeQGsPXSO9YNF2Z uq7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=W6mjT0eRfCwfLc+G2YCpsPlEeRwl807aZwyCVJxSnaI=; b=LB7xxMFltj3u2ud+16F1FQ3xUVLh2WR1CAqlBYtQxkfk1I2Ez8QYpCm0VbIBMGGxhq 6igNPncevLxxu+J+mwfwsUz3SOx70qHbWl7FbnXNLlXOR4bd/4tbfQhB/Efn7ndd5Rwe +fpE6pyjeZ1duxE+677Rz7O4iZOU1XMLJtE7RAbnww/lXMyzVJFRmAVtpT5z8rjBzRKb T1uF4L1ihtAgTnB6jrJryXgqd8ZRGC3QgqnJb1yoT6lPOnv4ahXmbdO5dGP1C6Obr3wl 2sFBVGPj9+nUovSbEwv2SdjjA4ayq5MLvMn5B/w5orIy9xeEM/8N6ZUQBiU5fdgpndOa wsLQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id 10si13164298pgm.332.2019.05.07.17.10.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:10:03 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:10:02 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga001.jf.intel.com with ESMTP; 07 May 2019 17:10:02 -0700 Subject: [PATCH v2 3/6] PCI/P2PDMA: Fix the gen_pool_add_virt() failure path From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Christoph Hellwig , Ira Weiny , Bjorn Helgaas , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:16 -0700 Message-ID: <155727337603.292046.13101332703665246702.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The pci_p2pdma_add_resource() implementation immediately frees the pgmap if gen_pool_add_virt() fails. However, that means that when @dev triggers a devres release devm_memremap_pages_release() will crash trying to access the freed @pgmap. Use the new devm_memunmap_pages() to manually free the mapping in the error path. Fixes: 52916982af48 ("PCI/P2PDMA: Support peer-to-peer memory") Cc: Logan Gunthorpe Cc: Christoph Hellwig Reviewed-by: Ira Weiny Acked-by: Bjorn Helgaas Signed-off-by: Dan Williams --- drivers/pci/p2pdma.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index c52298d76e64..595a534bd749 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -208,13 +208,15 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pci_bus_address(pdev, bar) + offset, resource_size(&pgmap->res), dev_to_node(&pdev->dev)); if (error) - goto pgmap_free; + goto pages_free; pci_info(pdev, "added peer-to-peer DMA memory %pR\n", &pgmap->res); return 0; +pages_free: + devm_memunmap_pages(&pdev->dev, pgmap); pgmap_free: devm_kfree(&pdev->dev, pgmap); return error; From patchwork Tue May 7 23:56:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84AF1912 for ; Wed, 8 May 2019 00:10:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7606E28643 for ; Wed, 8 May 2019 00:10:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 69FD5287A6; Wed, 8 May 2019 00:10:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71F1A2873B for ; Wed, 8 May 2019 00:10:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5144E6B000C; Tue, 7 May 2019 20:10:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4C3C16B000D; Tue, 7 May 2019 20:10:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DBFB6B000E; Tue, 7 May 2019 20:10:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 03E056B000C for ; Tue, 7 May 2019 20:10:10 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id i8so8932254pfo.21 for ; Tue, 07 May 2019 17:10:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=BRMli1SPmPT1M1ZzEoZnww9z5elObfh02aZx1Lu8wmY=; b=K5ciTR2LzwRLmV68MI23MdWfw7G3/3uk6ESjVwwyRJ75bqcwVpY4lK83SwpHfLxAAl cvUKxKH8TYkVRGhAWWMOdE3xHvpXOiqK1N5Fq6DqGvzqF+sRDM+q/A5mYyExy5Vg5Dmm 8AxaZaN4OvmvwfQV0DmBPAJU8SuWqkAAN73KUDqqQhzzlL28qUTq2vU6S2wgj/0/Z0Iz SIQBy3XI6ijiw+25Bo+6fqXVWHae0n3xTJfxzWWeiCe4SZcxiZhHb9h9VSIxSFTiRefF RfBcq/1kS0pcbd9ymzYAwR2SD8tAbeY0NrsoRgyboNCaskGg7VSYZxE/MRieaMHBZG9o EPlQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUtqzCF1lXDjcml6CTo6jG7syRI78UvR3DUY5bcfKrZbUPamtCe HDEephfLFr6VnwxkvJVPh0FjemCAc12bkXGGXvMQRWpIIhGvYG68u0U6r1QAWA7hLMyEwQEKhCg Ex1LKSTHJVPjC9ER23DQj4QtER+YduUYSYA2mwBiZ4cy3UlgwX7m71wO5Asmmx2CdvQ== X-Received: by 2002:a62:4e86:: with SMTP id c128mr44202596pfb.39.1557274209626; Tue, 07 May 2019 17:10:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqxzj1o8FXLpUH1mn3fsRKZ81700kiNxZ+tIB2b340IRI9rxbz2HGybnXg6j9h6RGvMkFl/w X-Received: by 2002:a62:4e86:: with SMTP id c128mr44202508pfb.39.1557274208447; Tue, 07 May 2019 17:10:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274208; cv=none; d=google.com; s=arc-20160816; b=E4Ush/5NoK0b9j4VArUt9sRwFymM4JtzpFOVpOsM0DLCmjC4m8OGHsh3lwlL54Yh/q SO1mZ2mSH1bQCjr7b+/sHftTsm25Fq3OghcZ2dQXsxoGRvzGbmS2JYW/ifRPl74wD3ND 223W4WBbw9OeD83WjArvR330BguTBpAY8AQMGVxyCRkz2b1/72c7TZe7Dzq3KOBcb09P KAZQ+eW3RR0GdCReS1hTkm1OAALmbbDoSEVbaff5L+YUPRm1ct6JOmno/mHob7+fMAjU LoDQMtYKflS0P8he/cM/YzJ2nETf1r8I5rVktom4J33JW55A2ylQ2ZPgB5VpIhF3l/8R R5xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=BRMli1SPmPT1M1ZzEoZnww9z5elObfh02aZx1Lu8wmY=; b=PytntgVwb8jpqQQ8EgqMiI6/v/fmGNAcj4Np9T8Lbhjx/TM92Y3ahJvNDGmt9I54tT 1hB1IF2NzHwnA5Sf2GU2D1+dFlzllNa8fNUQeFdsKuXoVln+3YRTZ+wRCnoCuFt9PF6e pb1uYivWPdvDl4i1kIvaU6qR/es6lkEWBMqcfPl2afAi3g/PTwIHUqXhsFTpZn/Kixnc 7q8W9vFvaNliCP/4uR5HwhRiCJ3MPt70pxeS5/hlCBylx7KCOlt4aDVypcY2E+stbth0 C0QF6tJTQEDaZXa3vzvWfN0rGN97QB+HdHENHpDhElMn3N32j4VERDaBSSK74YYbZavh utXQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id b6si4973500pgk.279.2019.05.07.17.10.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:10:08 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:10:07 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga004.jf.intel.com with ESMTP; 07 May 2019 17:10:07 -0700 Subject: [PATCH v2 4/6] lib/genalloc: Introduce chunk owners From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Bjorn Helgaas , =?utf-8?b?SsOpcsO0bWU=?= Glisse , Christoph Hellwig , Ira Weiny , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:21 -0700 Message-ID: <155727338118.292046.13407378933221579644.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The p2pdma facility enables a provider to publish a pool of dma addresses for a consumer to allocate. A genpool is used internally by p2pdma to collect dma resources, 'chunks', to be handed out to consumers. Whenever a consumer allocates a resource it needs to pin the 'struct dev_pagemap' instance that backs the chunk selected by pci_alloc_p2pmem(). Currently that reference is taken globally on the entire provider device. That sets up a lifetime mismatch whereby the p2pdma core needs to maintain hacks to make sure the percpu_ref is not released twice. This lifetime mismatch also stands in the way of a fix to devm_memremap_pages() whereby devm_memremap_pages_release() must wait for the percpu_ref ->release() callback to complete before it can proceed to teardown pages. So, towards fixing this situation, introduce the ability to store a 'chunk owner' at gen_pool_add() time, and a facility to retrieve the owner at gen_pool_{alloc,free}() time. For p2pdma this will be used to store and recall individual dev_pagemap reference counter instances per-chunk. Cc: Logan Gunthorpe Cc: Bjorn Helgaas Cc: "Jérôme Glisse" Cc: Christoph Hellwig Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- include/linux/genalloc.h | 55 +++++++++++++++++++++++++++++++++++++++++----- lib/genalloc.c | 51 +++++++++++++++++++++---------------------- 2 files changed, 74 insertions(+), 32 deletions(-) diff --git a/include/linux/genalloc.h b/include/linux/genalloc.h index dd0a452373e7..a337313e064f 100644 --- a/include/linux/genalloc.h +++ b/include/linux/genalloc.h @@ -75,6 +75,7 @@ struct gen_pool_chunk { struct list_head next_chunk; /* next chunk in pool */ atomic_long_t avail; phys_addr_t phys_addr; /* physical starting address of memory chunk */ + void *owner; /* private data to retrieve at alloc time */ unsigned long start_addr; /* start address of memory chunk */ unsigned long end_addr; /* end address of memory chunk (inclusive) */ unsigned long bits[0]; /* bitmap for allocating memory chunk */ @@ -96,8 +97,15 @@ struct genpool_data_fixed { extern struct gen_pool *gen_pool_create(int, int); extern phys_addr_t gen_pool_virt_to_phys(struct gen_pool *pool, unsigned long); -extern int gen_pool_add_virt(struct gen_pool *, unsigned long, phys_addr_t, - size_t, int); +extern int gen_pool_add_owner(struct gen_pool *, unsigned long, phys_addr_t, + size_t, int, void *); + +static inline int gen_pool_add_virt(struct gen_pool *pool, unsigned long addr, + phys_addr_t phys, size_t size, int nid) +{ + return gen_pool_add_owner(pool, addr, phys, size, nid, NULL); +} + /** * gen_pool_add - add a new chunk of special memory to the pool * @pool: pool to add new memory chunk to @@ -116,12 +124,47 @@ static inline int gen_pool_add(struct gen_pool *pool, unsigned long addr, return gen_pool_add_virt(pool, addr, -1, size, nid); } extern void gen_pool_destroy(struct gen_pool *); -extern unsigned long gen_pool_alloc(struct gen_pool *, size_t); -extern unsigned long gen_pool_alloc_algo(struct gen_pool *, size_t, - genpool_algo_t algo, void *data); +unsigned long gen_pool_alloc_algo_owner(struct gen_pool *pool, size_t size, + genpool_algo_t algo, void *data, void **owner); + +static inline unsigned long gen_pool_alloc_owner(struct gen_pool *pool, + size_t size, void **owner) +{ + return gen_pool_alloc_algo_owner(pool, size, pool->algo, pool->data, + owner); +} + +static inline unsigned long gen_pool_alloc_algo(struct gen_pool *pool, + size_t size, genpool_algo_t algo, void *data) +{ + return gen_pool_alloc_algo_owner(pool, size, algo, data, NULL); +} + +/** + * gen_pool_alloc - allocate special memory from the pool + * @pool: pool to allocate from + * @size: number of bytes to allocate from the pool + * + * Allocate the requested number of bytes from the specified pool. + * Uses the pool allocation function (with first-fit algorithm by default). + * Can not be used in NMI handler on architectures without + * NMI-safe cmpxchg implementation. + */ +static inline unsigned long gen_pool_alloc(struct gen_pool *pool, size_t size) +{ + return gen_pool_alloc_algo(pool, size, pool->algo, pool->data); +} + extern void *gen_pool_dma_alloc(struct gen_pool *pool, size_t size, dma_addr_t *dma); -extern void gen_pool_free(struct gen_pool *, unsigned long, size_t); +extern void gen_pool_free_owner(struct gen_pool *pool, unsigned long addr, + size_t size, void **owner); +static inline void gen_pool_free(struct gen_pool *pool, unsigned long addr, + size_t size) +{ + gen_pool_free_owner(pool, addr, size, NULL); +} + extern void gen_pool_for_each_chunk(struct gen_pool *, void (*)(struct gen_pool *, struct gen_pool_chunk *, void *), void *); extern size_t gen_pool_avail(struct gen_pool *); diff --git a/lib/genalloc.c b/lib/genalloc.c index 7e85d1e37a6e..770c769d7cb7 100644 --- a/lib/genalloc.c +++ b/lib/genalloc.c @@ -168,20 +168,21 @@ struct gen_pool *gen_pool_create(int min_alloc_order, int nid) EXPORT_SYMBOL(gen_pool_create); /** - * gen_pool_add_virt - add a new chunk of special memory to the pool + * gen_pool_add_owner- add a new chunk of special memory to the pool * @pool: pool to add new memory chunk to * @virt: virtual starting address of memory chunk to add to pool * @phys: physical starting address of memory chunk to add to pool * @size: size in bytes of the memory chunk to add to pool * @nid: node id of the node the chunk structure and bitmap should be * allocated on, or -1 + * @owner: private data the publisher would like to recall at alloc time * * Add a new chunk of special memory to the specified pool. * * Returns 0 on success or a -ve errno on failure. */ -int gen_pool_add_virt(struct gen_pool *pool, unsigned long virt, phys_addr_t phys, - size_t size, int nid) +int gen_pool_add_owner(struct gen_pool *pool, unsigned long virt, phys_addr_t phys, + size_t size, int nid, void *owner) { struct gen_pool_chunk *chunk; int nbits = size >> pool->min_alloc_order; @@ -195,6 +196,7 @@ int gen_pool_add_virt(struct gen_pool *pool, unsigned long virt, phys_addr_t phy chunk->phys_addr = phys; chunk->start_addr = virt; chunk->end_addr = virt + size - 1; + chunk->owner = owner; atomic_long_set(&chunk->avail, size); spin_lock(&pool->lock); @@ -203,7 +205,7 @@ int gen_pool_add_virt(struct gen_pool *pool, unsigned long virt, phys_addr_t phy return 0; } -EXPORT_SYMBOL(gen_pool_add_virt); +EXPORT_SYMBOL(gen_pool_add_owner); /** * gen_pool_virt_to_phys - return the physical address of memory @@ -260,35 +262,20 @@ void gen_pool_destroy(struct gen_pool *pool) EXPORT_SYMBOL(gen_pool_destroy); /** - * gen_pool_alloc - allocate special memory from the pool - * @pool: pool to allocate from - * @size: number of bytes to allocate from the pool - * - * Allocate the requested number of bytes from the specified pool. - * Uses the pool allocation function (with first-fit algorithm by default). - * Can not be used in NMI handler on architectures without - * NMI-safe cmpxchg implementation. - */ -unsigned long gen_pool_alloc(struct gen_pool *pool, size_t size) -{ - return gen_pool_alloc_algo(pool, size, pool->algo, pool->data); -} -EXPORT_SYMBOL(gen_pool_alloc); - -/** - * gen_pool_alloc_algo - allocate special memory from the pool + * gen_pool_alloc_algo_owner - allocate special memory from the pool * @pool: pool to allocate from * @size: number of bytes to allocate from the pool * @algo: algorithm passed from caller * @data: data passed to algorithm + * @owner: optionally retrieve the chunk owner * * Allocate the requested number of bytes from the specified pool. * Uses the pool allocation function (with first-fit algorithm by default). * Can not be used in NMI handler on architectures without * NMI-safe cmpxchg implementation. */ -unsigned long gen_pool_alloc_algo(struct gen_pool *pool, size_t size, - genpool_algo_t algo, void *data) +unsigned long gen_pool_alloc_algo_owner(struct gen_pool *pool, size_t size, + genpool_algo_t algo, void *data, void **owner) { struct gen_pool_chunk *chunk; unsigned long addr = 0; @@ -299,6 +286,9 @@ unsigned long gen_pool_alloc_algo(struct gen_pool *pool, size_t size, BUG_ON(in_nmi()); #endif + if (owner) + *owner = NULL; + if (size == 0) return 0; @@ -326,12 +316,14 @@ unsigned long gen_pool_alloc_algo(struct gen_pool *pool, size_t size, addr = chunk->start_addr + ((unsigned long)start_bit << order); size = nbits << order; atomic_long_sub(size, &chunk->avail); + if (owner) + *owner = chunk->owner; break; } rcu_read_unlock(); return addr; } -EXPORT_SYMBOL(gen_pool_alloc_algo); +EXPORT_SYMBOL(gen_pool_alloc_algo_owner); /** * gen_pool_dma_alloc - allocate special memory from the pool for DMA usage @@ -367,12 +359,14 @@ EXPORT_SYMBOL(gen_pool_dma_alloc); * @pool: pool to free to * @addr: starting address of memory to free back to pool * @size: size in bytes of memory to free + * @owner: private data stashed at gen_pool_add() time * * Free previously allocated special memory back to the specified * pool. Can not be used in NMI handler on architectures without * NMI-safe cmpxchg implementation. */ -void gen_pool_free(struct gen_pool *pool, unsigned long addr, size_t size) +void gen_pool_free_owner(struct gen_pool *pool, unsigned long addr, size_t size, + void **owner) { struct gen_pool_chunk *chunk; int order = pool->min_alloc_order; @@ -382,6 +376,9 @@ void gen_pool_free(struct gen_pool *pool, unsigned long addr, size_t size) BUG_ON(in_nmi()); #endif + if (owner) + *owner = NULL; + nbits = (size + (1UL << order) - 1) >> order; rcu_read_lock(); list_for_each_entry_rcu(chunk, &pool->chunks, next_chunk) { @@ -392,6 +389,8 @@ void gen_pool_free(struct gen_pool *pool, unsigned long addr, size_t size) BUG_ON(remain); size = nbits << order; atomic_long_add(size, &chunk->avail); + if (owner) + *owner = chunk->owner; rcu_read_unlock(); return; } @@ -399,7 +398,7 @@ void gen_pool_free(struct gen_pool *pool, unsigned long addr, size_t size) rcu_read_unlock(); BUG(); } -EXPORT_SYMBOL(gen_pool_free); +EXPORT_SYMBOL(gen_pool_free_owner); /** * gen_pool_for_each_chunk - call func for every chunk of generic memory pool From patchwork Tue May 7 23:56:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4FB2912 for ; Wed, 8 May 2019 00:10:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4E3D28758 for ; Wed, 8 May 2019 00:10:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B976C286A7; Wed, 8 May 2019 00:10:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E5AD287A6 for ; Wed, 8 May 2019 00:10:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED80F6B000E; Tue, 7 May 2019 20:10:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E88C66B0266; Tue, 7 May 2019 20:10:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9E356B026A; Tue, 7 May 2019 20:10:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 9A8D76B000E for ; Tue, 7 May 2019 20:10:14 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id d21so11359731pfr.3 for ; Tue, 07 May 2019 17:10:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=KmRyp9iGH2ia/dM4+L+AlPWU24ubf//fuPEOuKJI6UE=; b=entb8gHfzxqhKRaPCTJ6E25MzhJiqGCYEySZWk73Ozq4ZlCjJ3y7vtQCoBxpmMwr04 oLASs22tD61QZJNzhlkdWJQyWM0Woe011VyT+Di/p6PEfBa7+C90tCLM/5F9Atw6nige 1cExxWiWOCJlcb8Lu9IUpnz3pheY03bKC3hn68YxFr0K73c8DDVjbpvRl/Y7QqM+oS4d SoTRSQFwfyzHs3My/u5S5T81sn8ATzAXZesuQtmSokb/5vup6CgDSPWANBRCSLno0MMv BZx+XkaMirz5Z2qIIKnzqJvzdch6OA5JPaZIjT/rHawNie5NUFS0UoPoTCSxaI9Jyve3 wKlA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVIk3n/tCzl1Wnw67ge+j8C9krlN2XCWeTGR5x4OocwOR6u6ArQ iB/2ywDM0FYt4aEexmg/f+kAfLYLadmKSe2geeVZFHq6PpBSsd4Pf0Bx1xw2MMbDm9qCf8B8qSE mzykS0hNoO3AiqXTfCclFUT7ONqqmrvqaE47QkAy9X8zFtVotHNq/XDIwlQbhb0kjiw== X-Received: by 2002:a65:4c0b:: with SMTP id u11mr43592673pgq.405.1557274214252; Tue, 07 May 2019 17:10:14 -0700 (PDT) X-Google-Smtp-Source: APXvYqz0HbcH8t0P7Lc2rWuK+hsh0oly+4k9DrTMYFBkvUKKCqE/aEE+OMdjj0WyFsvHI2jCN2sx X-Received: by 2002:a65:4c0b:: with SMTP id u11mr43592605pgq.405.1557274213406; Tue, 07 May 2019 17:10:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274213; cv=none; d=google.com; s=arc-20160816; b=rCbZ0H5GpQn/TFcqSNx74JV2VTV8464T0p/2sqXSVBkqbPNv9WrUVexxkNUa6+e6A+ ksHRH4WuYVAr9Oe2DZX0Kei9qnZgpQ9bmRIcNSg/2jhW8PxRLWCc/44IY/ZCK8noJBco BygDG33fyneb7NogVA21DpuBceEOW9Zc7RDdaRnaSuSgE4nku7p9RkPMulmuPzOaIx3I 9nRLvhjbMGp80RNchJHfngMq5oP75u8rGXpFgHvUv2Rhbk6p7o2HQqZXjskVXFufkvxP m7/Ah1TmSdgbbYzmjyAu49Jo8SX9VhXxGuc4IEOzbjYjNctrgYElvG19gEDKy1h1itM7 +bBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=KmRyp9iGH2ia/dM4+L+AlPWU24ubf//fuPEOuKJI6UE=; b=IGKLdcBcxDXCVihXio2uOWGMV6hZ+5W+5MYxOpfxnAZ8MfODmg87JBjBL1IW7KTMud IRIsbPTnml01nlNHKm8nCZhaRUektU17fzaGJn+CUnNEKU+ahQZpZyhkDlTmSOFmIjUD vV6lqjE9Ah4cO1/jf3cguqIDuxcjuVzSK1FvBzkheWvif74CZueEjR7p0lRkpXiyV09X +M43Hxbz9t1ha9SxQ29Jk0qyt6Qjpv7QjWMxD0kG3Ja/w1y42HgbEvIEY33w9n4t6mEA 3jyZ8/1yXi7jLzKtfwmoRSS57qeqfPLpukNohF/mZufIDFxX9jctveHk8uOCaMoJERI9 3D7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id x28si15777860pff.104.2019.05.07.17.10.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:10:13 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:10:13 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga006.jf.intel.com with ESMTP; 07 May 2019 17:10:12 -0700 Subject: [PATCH v2 5/6] PCI/P2PDMA: Track pgmap references per resource, not globally From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Bjorn Helgaas , Christoph Hellwig , Ira Weiny , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:26 -0700 Message-ID: <155727338646.292046.9922678317501435597.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In preparation for fixing a race between devm_memremap_pages_release() and the final put of a page from the device-page-map, allocate a percpu-ref per p2pdma resource mapping. Cc: Logan Gunthorpe Cc: Bjorn Helgaas Cc: Christoph Hellwig Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- drivers/pci/p2pdma.c | 124 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 81 insertions(+), 43 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 595a534bd749..54d475569058 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -20,12 +20,16 @@ #include struct pci_p2pdma { - struct percpu_ref devmap_ref; - struct completion devmap_ref_done; struct gen_pool *pool; bool p2pmem_published; }; +struct p2pdma_pagemap { + struct dev_pagemap pgmap; + struct percpu_ref ref; + struct completion ref_done; +}; + static ssize_t size_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -74,41 +78,45 @@ static const struct attribute_group p2pmem_group = { .name = "p2pmem", }; +static struct p2pdma_pagemap *to_p2p_pgmap(struct percpu_ref *ref) +{ + return container_of(ref, struct p2pdma_pagemap, ref); +} + static void pci_p2pdma_percpu_release(struct percpu_ref *ref) { - struct pci_p2pdma *p2p = - container_of(ref, struct pci_p2pdma, devmap_ref); + struct p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(ref); - complete_all(&p2p->devmap_ref_done); + complete(&p2p_pgmap->ref_done); } static void pci_p2pdma_percpu_kill(struct percpu_ref *ref) { - /* - * pci_p2pdma_add_resource() may be called multiple times - * by a driver and may register the percpu_kill devm action multiple - * times. We only want the first action to actually kill the - * percpu_ref. - */ - if (percpu_ref_is_dying(ref)) - return; - percpu_ref_kill(ref); } +static void pci_p2pdma_percpu_cleanup(void *ref) +{ + struct p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(ref); + + wait_for_completion(&p2p_pgmap->ref_done); + percpu_ref_exit(&p2p_pgmap->ref); +} + static void pci_p2pdma_release(void *data) { struct pci_dev *pdev = data; + struct pci_p2pdma *p2pdma = pdev->p2pdma; - if (!pdev->p2pdma) + if (!p2pdma) return; - wait_for_completion(&pdev->p2pdma->devmap_ref_done); - percpu_ref_exit(&pdev->p2pdma->devmap_ref); + /* Flush and disable pci_alloc_p2p_mem() */ + pdev->p2pdma = NULL; + synchronize_rcu(); - gen_pool_destroy(pdev->p2pdma->pool); + gen_pool_destroy(p2pdma->pool); sysfs_remove_group(&pdev->dev.kobj, &p2pmem_group); - pdev->p2pdma = NULL; } static int pci_p2pdma_setup(struct pci_dev *pdev) @@ -124,12 +132,6 @@ static int pci_p2pdma_setup(struct pci_dev *pdev) if (!p2p->pool) goto out; - init_completion(&p2p->devmap_ref_done); - error = percpu_ref_init(&p2p->devmap_ref, - pci_p2pdma_percpu_release, 0, GFP_KERNEL); - if (error) - goto out_pool_destroy; - error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev); if (error) goto out_pool_destroy; @@ -163,6 +165,7 @@ static int pci_p2pdma_setup(struct pci_dev *pdev) int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, u64 offset) { + struct p2pdma_pagemap *p2p_pgmap; struct dev_pagemap *pgmap; void *addr; int error; @@ -185,14 +188,32 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, return error; } - pgmap = devm_kzalloc(&pdev->dev, sizeof(*pgmap), GFP_KERNEL); - if (!pgmap) + p2p_pgmap = devm_kzalloc(&pdev->dev, sizeof(*p2p_pgmap), GFP_KERNEL); + if (!p2p_pgmap) return -ENOMEM; + init_completion(&p2p_pgmap->ref_done); + error = percpu_ref_init(&p2p_pgmap->ref, + pci_p2pdma_percpu_release, 0, GFP_KERNEL); + if (error) + goto pgmap_free; + + /* + * FIXME: the percpu_ref_exit needs to be coordinated internal + * to devm_memremap_pages_release(). Duplicate the same ordering + * as other devm_memremap_pages() users for now. + */ + error = devm_add_action(&pdev->dev, pci_p2pdma_percpu_cleanup, + &p2p_pgmap->ref); + if (error) + goto ref_cleanup; + + pgmap = &p2p_pgmap->pgmap; + pgmap->res.start = pci_resource_start(pdev, bar) + offset; pgmap->res.end = pgmap->res.start + size - 1; pgmap->res.flags = pci_resource_flags(pdev, bar); - pgmap->ref = &pdev->p2pdma->devmap_ref; + pgmap->ref = &p2p_pgmap->ref; pgmap->type = MEMORY_DEVICE_PCI_P2PDMA; pgmap->pci_p2pdma_bus_offset = pci_bus_address(pdev, bar) - pci_resource_start(pdev, bar); @@ -201,12 +222,13 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, addr = devm_memremap_pages(&pdev->dev, pgmap); if (IS_ERR(addr)) { error = PTR_ERR(addr); - goto pgmap_free; + goto ref_exit; } - error = gen_pool_add_virt(pdev->p2pdma->pool, (unsigned long)addr, + error = gen_pool_add_owner(pdev->p2pdma->pool, (unsigned long)addr, pci_bus_address(pdev, bar) + offset, - resource_size(&pgmap->res), dev_to_node(&pdev->dev)); + resource_size(&pgmap->res), dev_to_node(&pdev->dev), + &p2p_pgmap->ref); if (error) goto pages_free; @@ -217,8 +239,10 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pages_free: devm_memunmap_pages(&pdev->dev, pgmap); +ref_cleanup: + percpu_ref_exit(&p2p_pgmap->ref); pgmap_free: - devm_kfree(&pdev->dev, pgmap); + devm_kfree(&pdev->dev, p2p_pgmap); return error; } EXPORT_SYMBOL_GPL(pci_p2pdma_add_resource); @@ -555,19 +579,30 @@ EXPORT_SYMBOL_GPL(pci_p2pmem_find_many); */ void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size) { - void *ret; + void *ret = NULL; + struct percpu_ref *ref; + /* + * Pairs with synchronize_rcu() in pci_p2pdma_release() to + * ensure pdev->p2pdma is non-NULL for the duration of the + * read-lock. + */ + rcu_read_lock(); if (unlikely(!pdev->p2pdma)) - return NULL; - - if (unlikely(!percpu_ref_tryget_live(&pdev->p2pdma->devmap_ref))) - return NULL; - - ret = (void *)gen_pool_alloc(pdev->p2pdma->pool, size); + goto out; - if (unlikely(!ret)) - percpu_ref_put(&pdev->p2pdma->devmap_ref); + ret = (void *)gen_pool_alloc_owner(pdev->p2pdma->pool, size, + (void **) &ref); + if (!ret) + goto out; + if (unlikely(!percpu_ref_tryget_live(ref))) { + gen_pool_free(pdev->p2pdma->pool, (unsigned long) ret, size); + ret = NULL; + goto out; + } +out: + rcu_read_unlock(); return ret; } EXPORT_SYMBOL_GPL(pci_alloc_p2pmem); @@ -580,8 +615,11 @@ EXPORT_SYMBOL_GPL(pci_alloc_p2pmem); */ void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size) { - gen_pool_free(pdev->p2pdma->pool, (uintptr_t)addr, size); - percpu_ref_put(&pdev->p2pdma->devmap_ref); + struct percpu_ref *ref; + + gen_pool_free_owner(pdev->p2pdma->pool, (uintptr_t)addr, size, + (void **) &ref); + percpu_ref_put(ref); } EXPORT_SYMBOL_GPL(pci_free_p2pmem); From patchwork Tue May 7 23:56:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 10933915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8DEC16C1 for ; Wed, 8 May 2019 00:10:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA24128485 for ; Wed, 8 May 2019 00:10:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE72E286E3; Wed, 8 May 2019 00:10:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A546C28746 for ; Wed, 8 May 2019 00:10:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1C296B026A; Tue, 7 May 2019 20:10:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9CCBE6B026B; Tue, 7 May 2019 20:10:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E2FC6B026C; Tue, 7 May 2019 20:10:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 50DAD6B026A for ; Tue, 7 May 2019 20:10:20 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id s26so11307317pfm.18 for ; Tue, 07 May 2019 17:10:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=L02iG7m5SsIhvrUm8qkkcbYbvcncK7iQivvm3wqwLGM=; b=P/k9Is8gwosxUNuFN86WepxFQp0PJWJZBSYj1zusjyNyJ7ro3WRm5DYRsLQoMZWpMl jp7dtuJsknwK7FQN0E98h48pB1V0XeLTaX/qnqDOXk6No7uLHhf7vHiR+L0ofew6kyOY d8ToOPq9OfwlvQBvhizXqfqY93Tj+M8XO4hnNo9k8k/3b0ocmz4kxAf4NEfQ0wB21nmb vH8z3ArTYA5kBpcp8Q4ACGBGErzOHE83GxmHvBFpspeeQGDLy8wpfDEBV8n1/X6nUxAL uTZaxlbrmqc2XqFJrIlHCVcSNkLOdWuLzd1LHU7DEu9eoavrpxW5BCYiVVBjko53m7vK zAhg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVlNXBwiDBvy9GblpRfIoLlIsAHzJ/Wg7E7VfA3ZtxZQJtJs9zo j3ZE6KIEZWNq0MLIkspoYsay4cVQlrGf+oqN5R79n4AsA0ErPOq4LCnCs9WXTYoIRR0f6WOoS+g lwKbVH9pEQ0NqqcAyVqEDLKD7V1k0pPtlqbbJ0pWxEkUPmUzvSg2euETtAuwwn3gc1w== X-Received: by 2002:a63:690:: with SMTP id 138mr42674539pgg.415.1557274219934; Tue, 07 May 2019 17:10:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqwImg+JH0Mnq9R71/OS4zqJOoR0mEgCQh2YdQ7CuNa/BN0CjPYxtwyHKSyaqrE7xNsZGTRB X-Received: by 2002:a63:690:: with SMTP id 138mr42674450pgg.415.1557274218801; Tue, 07 May 2019 17:10:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557274218; cv=none; d=google.com; s=arc-20160816; b=rg053JvOzu3tn0WQEEVK3395pZ/Pxgc+aAJs0nfPcE/o6ZsF3LYE/YMVi0GquhMU5O vBC/2ucF1DKXRHSIRWuPaN6ZFxn7o9Fod98x5IJWrKZRzbaR//eQo36N37rsjWkE1nJB EwthW/hCtXjc+CYeGeqqjdzx4Dnwm8Umlxx7pXDNfrI6lGoNGnd9QbI3KposHX2urVJP D5sjiMKYdfb+mJjUdulSw0LvpRHXC6cMV+NsguN2BEapx7HhUvkHsKrPwc9NOqW/Hk6t 5elHsRm8Ak1Gr6NTOXUqvA6muawKxSPAijudxnwWlwaWowG9Y9bX/r/KYA0LTbDfnYJv ik6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=L02iG7m5SsIhvrUm8qkkcbYbvcncK7iQivvm3wqwLGM=; b=K2zEIijC0RQkDUizhJy2tXRAnWGx2aLAgEeVO+g3thARzwCxIiWvtQWyBAWOOviqKU ttbb+bjc+Q8FWp54Oit7TQfdGiqDcaeGTJhtmxhcs6F6uFqJ85vY4SNxFhzDOcYPXIxI 1qK5qFuLyRyar+VY5c1TxeeU8jacWQJzhPeKM2td6icpQk0FufkRDaOI86lfNCb33Tin 5cnAVaR/xydfipOG0J+tfesARgv0JNlFVf2R/WSpl06T8cdkqF62klXSjeI0dp9liU6/ 4n8pklIZ+/h+utguI6EBC4aLgtMRQFeBpwirZrWz7iNbhpRdp9r+f/1QDFMQStMTX7aG /Tsw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id o64si20563938pfa.274.2019.05.07.17.10.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 May 2019 17:10:18 -0700 (PDT) Received-SPF: pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan.j.williams@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=dan.j.williams@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 May 2019 17:10:18 -0700 X-ExtLoop1: 1 Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga008.jf.intel.com with ESMTP; 07 May 2019 17:10:18 -0700 Subject: [PATCH v2 6/6] mm/devm_memremap_pages: Fix final page put race From: Dan Williams To: akpm@linux-foundation.org Cc: Logan Gunthorpe , Bjorn Helgaas , =?utf-8?b?SsOpcsO0bWU=?= Glisse , Christoph Hellwig , Ira Weiny , linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org Date: Tue, 07 May 2019 16:56:31 -0700 Message-ID: <155727339156.292046.5432007428235387859.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> References: <155727335978.292046.12068191395005445711.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.18-2-gc94f MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Logan noticed that devm_memremap_pages_release() kills the percpu_ref drops all the page references that were acquired at init and then immediately proceeds to unplug, arch_remove_memory(), the backing pages for the pagemap. If for some reason device shutdown actually collides with a busy / elevated-ref-count page then arch_remove_memory() should be deferred until after that reference is dropped. As it stands the "wait for last page ref drop" happens *after* devm_memremap_pages_release() returns, which is obviously too late and can lead to crashes. Fix this situation by assigning the responsibility to wait for the percpu_ref to go idle to devm_memremap_pages() with a new ->cleanup() callback. Implement the new cleanup callback for all devm_memremap_pages() users: pmem, devdax, hmm, and p2pdma. Reported-by: Logan Gunthorpe Fixes: 41e94a851304 ("add devm_memremap_pages") Cc: Bjorn Helgaas Cc: "Jérôme Glisse" Cc: Christoph Hellwig Reviewed-by: Ira Weiny Signed-off-by: Dan Williams --- drivers/dax/device.c | 13 +++---------- drivers/nvdimm/pmem.c | 17 +++++++++++++---- drivers/pci/p2pdma.c | 17 +++-------------- include/linux/memremap.h | 2 ++ kernel/memremap.c | 17 ++++++++++++----- mm/hmm.c | 14 +++----------- tools/testing/nvdimm/test/iomap.c | 2 ++ 7 files changed, 38 insertions(+), 44 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index e428468ab661..e3aa78dd1bb0 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -27,9 +27,8 @@ static void dev_dax_percpu_release(struct percpu_ref *ref) complete(&dev_dax->cmp); } -static void dev_dax_percpu_exit(void *data) +static void dev_dax_percpu_exit(struct percpu_ref *ref) { - struct percpu_ref *ref = data; struct dev_dax *dev_dax = ref_to_dev_dax(ref); dev_dbg(&dev_dax->dev, "%s\n", __func__); @@ -468,18 +467,12 @@ int dev_dax_probe(struct device *dev) if (rc) return rc; - rc = devm_add_action_or_reset(dev, dev_dax_percpu_exit, &dev_dax->ref); - if (rc) - return rc; - dev_dax->pgmap.ref = &dev_dax->ref; dev_dax->pgmap.kill = dev_dax_percpu_kill; + dev_dax->pgmap.cleanup = dev_dax_percpu_exit; addr = devm_memremap_pages(dev, &dev_dax->pgmap); - if (IS_ERR(addr)) { - devm_remove_action(dev, dev_dax_percpu_exit, &dev_dax->ref); - percpu_ref_exit(&dev_dax->ref); + if (IS_ERR(addr)) return PTR_ERR(addr); - } inode = dax_inode(dax_dev); cdev = inode->i_cdev; diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 0279eb1da3ef..1c9181712fa4 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -304,11 +304,19 @@ static const struct attribute_group *pmem_attribute_groups[] = { NULL, }; -static void pmem_release_queue(void *q) +static void __pmem_release_queue(struct percpu_ref *ref) { + struct request_queue *q; + + q = container_of(ref, typeof(*q), q_usage_counter); blk_cleanup_queue(q); } +static void pmem_release_queue(void *ref) +{ + __pmem_release_queue(ref); +} + static void pmem_freeze_queue(struct percpu_ref *ref) { struct request_queue *q; @@ -400,12 +408,10 @@ static int pmem_attach_disk(struct device *dev, if (!q) return -ENOMEM; - if (devm_add_action_or_reset(dev, pmem_release_queue, q)) - return -ENOMEM; - pmem->pfn_flags = PFN_DEV; pmem->pgmap.ref = &q->q_usage_counter; pmem->pgmap.kill = pmem_freeze_queue; + pmem->pgmap.cleanup = __pmem_release_queue; if (is_nd_pfn(dev)) { if (setup_pagemap_fsdax(dev, &pmem->pgmap)) return -ENOMEM; @@ -426,6 +432,9 @@ static int pmem_attach_disk(struct device *dev, pmem->pfn_flags |= PFN_MAP; memcpy(&bb_res, &pmem->pgmap.res, sizeof(bb_res)); } else { + if (devm_add_action_or_reset(dev, pmem_release_queue, + &q->q_usage_counter)) + return -ENOMEM; addr = devm_memremap(dev, pmem->phys_addr, pmem->size, ARCH_MEMREMAP_PMEM); memcpy(&bb_res, &nsio->res, sizeof(bb_res)); diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 54d475569058..a7a66b958720 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -95,7 +95,7 @@ static void pci_p2pdma_percpu_kill(struct percpu_ref *ref) percpu_ref_kill(ref); } -static void pci_p2pdma_percpu_cleanup(void *ref) +static void pci_p2pdma_percpu_cleanup(struct percpu_ref *ref) { struct p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(ref); @@ -198,16 +198,6 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, if (error) goto pgmap_free; - /* - * FIXME: the percpu_ref_exit needs to be coordinated internal - * to devm_memremap_pages_release(). Duplicate the same ordering - * as other devm_memremap_pages() users for now. - */ - error = devm_add_action(&pdev->dev, pci_p2pdma_percpu_cleanup, - &p2p_pgmap->ref); - if (error) - goto ref_cleanup; - pgmap = &p2p_pgmap->pgmap; pgmap->res.start = pci_resource_start(pdev, bar) + offset; @@ -218,11 +208,12 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->pci_p2pdma_bus_offset = pci_bus_address(pdev, bar) - pci_resource_start(pdev, bar); pgmap->kill = pci_p2pdma_percpu_kill; + pgmap->cleanup = pci_p2pdma_percpu_cleanup; addr = devm_memremap_pages(&pdev->dev, pgmap); if (IS_ERR(addr)) { error = PTR_ERR(addr); - goto ref_exit; + goto pgmap_free; } error = gen_pool_add_owner(pdev->p2pdma->pool, (unsigned long)addr, @@ -239,8 +230,6 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pages_free: devm_memunmap_pages(&pdev->dev, pgmap); -ref_cleanup: - percpu_ref_exit(&p2p_pgmap->ref); pgmap_free: devm_kfree(&pdev->dev, p2p_pgmap); return error; diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 7601ee314c4a..1732dea030b2 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -81,6 +81,7 @@ typedef void (*dev_page_free_t)(struct page *page, void *data); * @res: physical address range covered by @ref * @ref: reference count that pins the devm_memremap_pages() mapping * @kill: callback to transition @ref to the dead state + * @cleanup: callback to wait for @ref to be idle and reap it * @dev: host device of the mapping for debug * @data: private data pointer for page_free() * @type: memory type: see MEMORY_* in memory_hotplug.h @@ -92,6 +93,7 @@ struct dev_pagemap { struct resource res; struct percpu_ref *ref; void (*kill)(struct percpu_ref *ref); + void (*cleanup)(struct percpu_ref *ref); struct device *dev; void *data; enum memory_type type; diff --git a/kernel/memremap.c b/kernel/memremap.c index 65afbacab44e..05d1af5a2f15 100644 --- a/kernel/memremap.c +++ b/kernel/memremap.c @@ -96,6 +96,7 @@ static void devm_memremap_pages_release(void *data) pgmap->kill(pgmap->ref); for_each_device_pfn(pfn, pgmap) put_page(pfn_to_page(pfn)); + pgmap->cleanup(pgmap->ref); /* pages are dead and unused, undo the arch mapping */ align_start = res->start & ~(SECTION_SIZE - 1); @@ -134,8 +135,8 @@ static void devm_memremap_pages_release(void *data) * 2/ The altmap field may optionally be initialized, in which case altmap_valid * must be set to true * - * 3/ pgmap->ref must be 'live' on entry and will be killed at - * devm_memremap_pages_release() time, or if this routine fails. + * 3/ pgmap->ref must be 'live' on entry and will be killed and reaped + * at devm_memremap_pages_release() time, or if this routine fails. * * 4/ res is expected to be a host memory range that could feasibly be * treated as a "System RAM" range, i.e. not a device mmio range, but @@ -151,8 +152,10 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) pgprot_t pgprot = PAGE_KERNEL; int error, nid, is_ram; - if (!pgmap->ref || !pgmap->kill) + if (!pgmap->ref || !pgmap->kill || !pgmap->cleanup) { + WARN(1, "Missing reference count teardown definition\n"); return ERR_PTR(-EINVAL); + } align_start = res->start & ~(SECTION_SIZE - 1); align_size = ALIGN(res->start + resource_size(res), SECTION_SIZE) @@ -163,14 +166,16 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) if (conflict_pgmap) { dev_WARN(dev, "Conflicting mapping in same section\n"); put_dev_pagemap(conflict_pgmap); - return ERR_PTR(-ENOMEM); + error = -ENOMEM; + goto err_array; } conflict_pgmap = get_dev_pagemap(PHYS_PFN(align_end), NULL); if (conflict_pgmap) { dev_WARN(dev, "Conflicting mapping in same section\n"); put_dev_pagemap(conflict_pgmap); - return ERR_PTR(-ENOMEM); + error = -ENOMEM; + goto err_array; } is_ram = region_intersects(align_start, align_size, @@ -262,6 +267,8 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) pgmap_array_delete(res); err_array: pgmap->kill(pgmap->ref); + pgmap->cleanup(pgmap->ref); + return ERR_PTR(error); } EXPORT_SYMBOL_GPL(devm_memremap_pages); diff --git a/mm/hmm.c b/mm/hmm.c index fe1cd87e49ac..225ade644058 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -975,9 +975,8 @@ static void hmm_devmem_ref_release(struct percpu_ref *ref) complete(&devmem->completion); } -static void hmm_devmem_ref_exit(void *data) +static void hmm_devmem_ref_exit(struct percpu_ref *ref) { - struct percpu_ref *ref = data; struct hmm_devmem *devmem; devmem = container_of(ref, struct hmm_devmem, ref); @@ -1054,10 +1053,6 @@ struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, if (ret) return ERR_PTR(ret); - ret = devm_add_action_or_reset(device, hmm_devmem_ref_exit, &devmem->ref); - if (ret) - return ERR_PTR(ret); - size = ALIGN(size, PA_SECTION_SIZE); addr = min((unsigned long)iomem_resource.end, (1UL << MAX_PHYSMEM_BITS) - 1); @@ -1096,6 +1091,7 @@ struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, devmem->pagemap.ref = &devmem->ref; devmem->pagemap.data = devmem; devmem->pagemap.kill = hmm_devmem_ref_kill; + devmem->pagemap.cleanup = hmm_devmem_ref_exit; result = devm_memremap_pages(devmem->device, &devmem->pagemap); if (IS_ERR(result)) @@ -1133,11 +1129,6 @@ struct hmm_devmem *hmm_devmem_add_resource(const struct hmm_devmem_ops *ops, if (ret) return ERR_PTR(ret); - ret = devm_add_action_or_reset(device, hmm_devmem_ref_exit, - &devmem->ref); - if (ret) - return ERR_PTR(ret); - devmem->pfn_first = devmem->resource->start >> PAGE_SHIFT; devmem->pfn_last = devmem->pfn_first + (resource_size(devmem->resource) >> PAGE_SHIFT); @@ -1150,6 +1141,7 @@ struct hmm_devmem *hmm_devmem_add_resource(const struct hmm_devmem_ops *ops, devmem->pagemap.ref = &devmem->ref; devmem->pagemap.data = devmem; devmem->pagemap.kill = hmm_devmem_ref_kill; + devmem->pagemap.cleanup = hmm_devmem_ref_exit; result = devm_memremap_pages(devmem->device, &devmem->pagemap); if (IS_ERR(result)) diff --git a/tools/testing/nvdimm/test/iomap.c b/tools/testing/nvdimm/test/iomap.c index c6635fee27d8..219dd0a1cb08 100644 --- a/tools/testing/nvdimm/test/iomap.c +++ b/tools/testing/nvdimm/test/iomap.c @@ -108,7 +108,9 @@ static void nfit_test_kill(void *_pgmap) { struct dev_pagemap *pgmap = _pgmap; + WARN_ON(!pgmap || !pgmap->ref || !pgmap->kill || !pgmap->cleanup); pgmap->kill(pgmap->ref); + pgmap->cleanup(pgmap->ref); } void *__wrap_devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap)