From patchwork Sat Jan 13 06:52:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Kasireddy X-Patchwork-Id: 13518885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1099C47422 for ; Sat, 13 Jan 2024 07:17:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C08FB940007; Sat, 13 Jan 2024 02:16:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B912D94000A; Sat, 13 Jan 2024 02:16:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E597940007; Sat, 13 Jan 2024 02:16:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7F78B94000A for ; Sat, 13 Jan 2024 02:16:47 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5D41C401E9 for ; Sat, 13 Jan 2024 07:16:47 +0000 (UTC) X-FDA: 81673430454.29.21E4FD7 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) by imf03.hostedemail.com (Postfix) with ESMTP id 41BBA20019 for ; Sat, 13 Jan 2024 07:16:45 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="dzMPM/wO"; spf=pass (imf03.hostedemail.com: domain of vivek.kasireddy@intel.com designates 192.198.163.11 as permitted sender) smtp.mailfrom=vivek.kasireddy@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705130205; a=rsa-sha256; cv=none; b=unQoVOQ4LYWAAIQBrNXzP104iMNSTKIz6axNWyO0mGcGp/KngoNb9mUbtUeelhEDAGXH21 BR6jcnAb1sqcdi2AOn6pFZ0QbcLq6cKrdJkSrGG2i+XGYNCnXgr98GvMksXcBYhiONUq48 sgri2qWp5hspN/VcjZTink5okwUTM6c= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="dzMPM/wO"; spf=pass (imf03.hostedemail.com: domain of vivek.kasireddy@intel.com designates 192.198.163.11 as permitted sender) smtp.mailfrom=vivek.kasireddy@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705130205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yOgBVQKmKDj4PD+KJHINMdpEC7EG7X90VypbiNBAVbQ=; b=JA6M25Gw5jjmB/5s0f4q9qILG2Pp4NVStEYL+akhaxPCVGk4xU7h6FdjDRD3aMzeEj7bTo Ncq4SAgJqka2W0mMoqTorqYeET8klTGUis/s8FwIBf7mLHmnDJf7gkkiQyczTZ8odtx4wn dJ4S5kn/ku7RVgxJXM8KpAGLYZOh/g0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1705130205; x=1736666205; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CPdbE6dv4gTz9/ceu4Ozo3TWtNEi41mmHjF0rn+A6NA=; b=dzMPM/wOOFauVryvh44Jm2gt1awjdqiURREAZkxyOAlTPPPy05mX5mbG C0/1Qfr4meCZ6MdD6tNu6gT/OFljGw8yBQQXaekx14VCkBBB37qI69QvM cHWoeFy/bHFsdEbpqMlcOmRi0ILLL9pElIjEd9ULbO4UdzYWvieY4Qyn8 QPASdoe0c7jye/EttRcJaJ4g20P7yysKCMCAB1DTWpr9Qg7tZCZWuq7nW i86xjp1Q6VsoB+U/ENs/L2L+9KqqkFhBTCV6/YFtjzJ2Smxi9Tq68pJcO lpDSMm3DbJpmFAiVxvRkTiosY8AHm/glfpSD2wmQVdlx2mPSu7LyQ0l8g A==; X-IronPort-AV: E=McAfee;i="6600,9927,10951"; a="6078158" X-IronPort-AV: E=Sophos;i="6.04,191,1695711600"; d="scan'208";a="6078158" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jan 2024 23:16:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10951"; a="783269515" X-IronPort-AV: E=Sophos;i="6.04,191,1695711600"; d="scan'208";a="783269515" Received: from vkasired-desk2.fm.intel.com ([10.105.128.132]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jan 2024 23:16:40 -0800 From: Vivek Kasireddy To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org Cc: Vivek Kasireddy , David Hildenbrand , Matthew Wilcox , Daniel Vetter , Mike Kravetz , Hugh Dickins , Peter Xu , Jason Gunthorpe , Gerd Hoffmann , Dongwon Kim , Junxiao Chang Subject: [PATCH v11 7/8] udmabuf: Pin the pages using memfd_pin_folios() API Date: Fri, 12 Jan 2024 22:52:22 -0800 Message-Id: <20240113065223.1532987-8-vivek.kasireddy@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240113065223.1532987-1-vivek.kasireddy@intel.com> References: <20240113065223.1532987-1-vivek.kasireddy@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 41BBA20019 X-Stat-Signature: o7yx357qrgkjacs9o4y4yebjkm4yrb1c X-Rspam-User: X-HE-Tag: 1705130205-32594 X-HE-Meta: U2FsdGVkX191YE8ms0awWwoOKzVq7DGD2AZW9L4XVoPpMHaBYsWVnmelXKLpEIA/r/Q44lTw1WBnFFnQFFPUeXFMGTsgWVNf8T2xkhd2OY+Dx8FweONo3cmV+MLqAsra00PjLt0ctdeEqzCr71uHCNZJpUQjXJR0/ID7DGJAWYwY/aaNA58jzyjh69zqczyUqHyNIRJegv9GRQK0NOBPJfMUakaBbxEmwv1Fn/XDZQNZpZFA0ADTTtUVJQEjmFvBuwwcHp+IGB1R9kFhqwO0gAyD6Pj/gMbXDIWB8kU7oMcGdWA1gXT39R7r0VTVtUF/+UNeqThLRT1iePY8ao0h5eZ/v6kjbNLfzSBcOZOHRRnPR0VHRPdw75FFO8FGGhkPAE6/FNkLpRia9yDVrTvi6RNM9j7VlBO4n+IYLQjMEeA3UQUJ+buS8+g5BXAIBI3zy/mgb5MLmBZC/SXAGnBrN9WWSaHGr7tyjmZq5F2N7Ndf8Atk5bhtaSqEdV0KyC6jS6BWNAjPDM7aJAGoXcLJP2oWBoziJhty9maBB8yx9om/nlmvXKBTZ4dFjyvc9r5EELJkP7MVALmTPpwZVPvMU2bO5pw091nip7+Kr7M9siOlVBG3dU7iPSiMKy4ytgoNBAM9ZijLtc2LcEMplnaObyIMklTM/GCVa6HD8OLdoY0MpIyFSrZXQo/nkgb0hzMkXD3QwRu/5UBQ9SKthHobeDb1s0UlUcJ6vDp2p12jpzYu3Y95ARGqK9MHDtBBVvbblwilRgfDB8NjRgxUGHl1Hdu33y4NtO5a6JlksmcTlBKbFGtOZm+DV40rY+YLhz8h9WYGF5HqMQ+iESyavCPbLmFffX862jE87wyLQEtbJ4LOFPdRj0+ucLZ0/usQo3pvh0p4up7EX+LWi+bwCemqOXvHx1/cMCCVygA2IefKge1sxi0Z/qXo85Pl5EkZrzxNdYFm7yueQVtRDIB3YFc QQPC+8F1 jXEVPie6sPw7+9QVaAYMYV3wKWeLphqWoFc7i3pHW3DnY0cdYns1rWwk7DCwluqwrezkIyMuJwcMVWpOVqJSOiw79weqv2pDK1Y8fJUw/YOMGXTkLqZvL9EE6DPAknevtLSaPUX2tuqbYyxFH+RyUsFcOPP62ngGVRFXT1ZB0QT1X0K9PHAGxy8WH6OgKNfmKfHhwwuhaQHtD0WvFDQh+sEmUasrR1AG1xRdiDe4GXwRvMUMjK/zMmBURM6OK1Eupd2pahivMe828FkBxDrukRGe2ZWn1Ax1jLb5s6e8eOVpLMQzgvKiTBD/50EHXMAhYQYW4WyrgBsmwYjyuQRXQqKA56lfplcxRqyX6Hmq3urGbI5A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Using memfd_pin_folios() will ensure that the pages are pinned correctly using FOLL_PIN. And, this also ensures that we don't accidentally break features such as memory hotunplug as it would not allow pinning pages in the movable zone. Using this new API also simplifies the code as we no longer have to deal with extracting individual pages from their mappings or handle shmem and hugetlb cases separately. Cc: David Hildenbrand Cc: Matthew Wilcox Cc: Daniel Vetter Cc: Mike Kravetz Cc: Hugh Dickins Cc: Peter Xu Cc: Jason Gunthorpe Cc: Gerd Hoffmann Cc: Dongwon Kim Cc: Junxiao Chang Signed-off-by: Vivek Kasireddy --- drivers/dma-buf/udmabuf.c | 153 +++++++++++++++++++------------------- 1 file changed, 78 insertions(+), 75 deletions(-) diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c index a8f3af61f7f2..afa8bfd2a2a9 100644 --- a/drivers/dma-buf/udmabuf.c +++ b/drivers/dma-buf/udmabuf.c @@ -30,6 +30,12 @@ struct udmabuf { struct sg_table *sg; struct miscdevice *device; pgoff_t *offsets; + struct list_head unpin_list; +}; + +struct udmabuf_folio { + struct folio *folio; + struct list_head list; }; static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) @@ -153,17 +159,43 @@ static void unmap_udmabuf(struct dma_buf_attachment *at, return put_sg_table(at->dev, sg, direction); } +static void unpin_all_folios(struct list_head *unpin_list) +{ + struct udmabuf_folio *ubuf_folio; + + while (!list_empty(unpin_list)) { + ubuf_folio = list_first_entry(unpin_list, + struct udmabuf_folio, list); + unpin_folio(ubuf_folio->folio); + + list_del(&ubuf_folio->list); + kfree(ubuf_folio); + } +} + +static int add_to_unpin_list(struct list_head *unpin_list, + struct folio *folio) +{ + struct udmabuf_folio *ubuf_folio; + + ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL); + if (!ubuf_folio) + return -ENOMEM; + + ubuf_folio->folio = folio; + list_add_tail(&ubuf_folio->list, unpin_list); + return 0; +} + static void release_udmabuf(struct dma_buf *buf) { struct udmabuf *ubuf = buf->priv; struct device *dev = ubuf->device->this_device; - pgoff_t pg; if (ubuf->sg) put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL); - for (pg = 0; pg < ubuf->pagecount; pg++) - folio_put(ubuf->folios[pg]); + unpin_all_folios(&ubuf->unpin_list); kfree(ubuf->offsets); kfree(ubuf->folios); kfree(ubuf); @@ -218,64 +250,6 @@ static const struct dma_buf_ops udmabuf_ops = { #define SEALS_WANTED (F_SEAL_SHRINK) #define SEALS_DENIED (F_SEAL_WRITE) -static int handle_hugetlb_pages(struct udmabuf *ubuf, struct file *memfd, - pgoff_t offset, pgoff_t pgcnt, - pgoff_t *pgbuf) -{ - struct hstate *hpstate = hstate_file(memfd); - pgoff_t mapidx = offset >> huge_page_shift(hpstate); - pgoff_t subpgoff = (offset & ~huge_page_mask(hpstate)) >> PAGE_SHIFT; - pgoff_t maxsubpgs = huge_page_size(hpstate) >> PAGE_SHIFT; - struct folio *folio = NULL; - pgoff_t pgidx; - - mapidx <<= huge_page_order(hpstate); - for (pgidx = 0; pgidx < pgcnt; pgidx++) { - if (!folio) { - folio = __filemap_get_folio(memfd->f_mapping, - mapidx, - FGP_ACCESSED, 0); - if (IS_ERR(folio)) - return PTR_ERR(folio); - } - - folio_get(folio); - ubuf->folios[*pgbuf] = folio; - ubuf->offsets[*pgbuf] = subpgoff << PAGE_SHIFT; - (*pgbuf)++; - if (++subpgoff == maxsubpgs) { - folio_put(folio); - folio = NULL; - subpgoff = 0; - mapidx += pages_per_huge_page(hpstate); - } - } - - if (folio) - folio_put(folio); - - return 0; -} - -static int handle_shmem_pages(struct udmabuf *ubuf, struct file *memfd, - pgoff_t offset, pgoff_t pgcnt, - pgoff_t *pgbuf) -{ - pgoff_t pgidx, pgoff = offset >> PAGE_SHIFT; - struct folio *folio = NULL; - - for (pgidx = 0; pgidx < pgcnt; pgidx++) { - folio = shmem_read_folio(memfd->f_mapping, pgoff + pgidx); - if (IS_ERR(folio)) - return PTR_ERR(folio); - - ubuf->folios[*pgbuf] = folio; - (*pgbuf)++; - } - - return 0; -} - static int check_memfd_seals(struct file *memfd) { int seals; @@ -321,16 +295,19 @@ static long udmabuf_create(struct miscdevice *device, struct udmabuf_create_list *head, struct udmabuf_create_item *list) { - pgoff_t pgcnt, pgbuf = 0, pglimit; + pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0; + long nr_folios, ret = -EINVAL; struct file *memfd = NULL; + struct folio **folios; struct udmabuf *ubuf; - int ret = -EINVAL; - u32 i, flags; + u32 i, j, k, flags; + loff_t end; ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL); if (!ubuf) return -ENOMEM; + INIT_LIST_HEAD(&ubuf->unpin_list); pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT; for (i = 0; i < head->count; i++) { if (!IS_ALIGNED(list[i].offset, PAGE_SIZE)) @@ -366,17 +343,44 @@ static long udmabuf_create(struct miscdevice *device, goto err; pgcnt = list[i].size >> PAGE_SHIFT; - if (is_file_hugepages(memfd)) - ret = handle_hugetlb_pages(ubuf, memfd, - list[i].offset, - pgcnt, &pgbuf); - else - ret = handle_shmem_pages(ubuf, memfd, - list[i].offset, - pgcnt, &pgbuf); - if (ret < 0) + folios = kmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL); + if (!folios) { + ret = -ENOMEM; goto err; + } + end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1; + ret = memfd_pin_folios(memfd, list[i].offset, end, + folios, pgcnt, &pgoff); + if (ret < 0) { + kfree(folios); + goto err; + } + + nr_folios = ret; + pgoff >>= PAGE_SHIFT; + for (j = 0, k = 0; j < pgcnt; j++) { + ubuf->folios[pgbuf] = folios[k]; + ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT; + + if (j == 0 || ubuf->folios[pgbuf-1] != folios[k]) { + ret = add_to_unpin_list(&ubuf->unpin_list, + folios[k]); + if (ret < 0) { + kfree(folios); + goto err; + } + } + + pgbuf++; + if (++pgoff == folio_nr_pages(folios[k])) { + pgoff = 0; + if (++k == nr_folios) + break; + } + } + + kfree(folios); fput(memfd); } @@ -388,10 +392,9 @@ static long udmabuf_create(struct miscdevice *device, return ret; err: - while (pgbuf > 0) - folio_put(ubuf->folios[--pgbuf]); if (memfd) fput(memfd); + unpin_all_folios(&ubuf->unpin_list); kfree(ubuf->offsets); kfree(ubuf->folios); kfree(ubuf);