From patchwork Thu Jan 26 14:16:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5088C05027 for ; Thu, 26 Jan 2023 14:16:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FC3C6B0072; Thu, 26 Jan 2023 09:16:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 284166B0073; Thu, 26 Jan 2023 09:16:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FE388E0001; Thu, 26 Jan 2023 09:16:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 01D446B0072 for ; Thu, 26 Jan 2023 09:16:43 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BE86FAB2A3 for ; Thu, 26 Jan 2023 14:16:43 +0000 (UTC) X-FDA: 80397151086.26.D724A17 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 72F614002C for ; Thu, 26 Jan 2023 14:16:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iQnq7a5M; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742600; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E9Jt6FaYFdS1OFI4/wM9JLR5TBUxMH/4qOao0tXSDe4=; b=yhwHc7reNqrS4JdeR/VOGVrYYAFvtzsLiGY3KVTtC1pWpug4jcVYyfQCv4tEh9+F1pLa6x gZ/HaQ6M4jXKIa2bcU/HRUX4DM7spi2THvnqFtI/u5/fsdw0NOl07whe1oJkXIN59LMNTg fZz+gcKALdK4FInBnm2yQKXs8Ugxl7Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iQnq7a5M; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742600; a=rsa-sha256; cv=none; b=kAtn4wSHVCeTCG4zGHac/KDvrvOSlxbAtUC/YuK1F0s6r+n7dAZ2sesQ1FxDZccDmy/fh3 hLYrM655jW0j27zKehh/+ePFeewVd/gsh1rSKO6Rn95+ye/+QCBp6HT3mLEEcl8kAqOY55 INgYYhi3cYKtvayG1shMqJkxJdpmi/0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E9Jt6FaYFdS1OFI4/wM9JLR5TBUxMH/4qOao0tXSDe4=; b=iQnq7a5MjRdC8ulKtVRgk5PTV4F3wVNHdmMNYttKsfsqe9jRLuNmFAbNRzXQjP6HYsITn+ virXM7QSkSz2DD7oq7mVb0SE2hflXwRoz42PNbvv5qE7wIue7hr6San/bcz+W2VeHvmDp9 8awtpJseiC+dwo0Fmk6p76IIRIe5mho= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-20-ZJTJgo9oNrO2PTfmKjBjXw-1; Thu, 26 Jan 2023 09:16:36 -0500 X-MC-Unique: ZJTJgo9oNrO2PTfmKjBjXw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7C84E18E0923; Thu, 26 Jan 2023 14:16:35 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7288DC15BA0; Thu, 26 Jan 2023 14:16:33 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 1/8] iov_iter: Define flags to qualify page extraction. Date: Thu, 26 Jan 2023 14:16:19 +0000 Message-Id: <20230126141626.2809643-2-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 72F614002C X-Stat-Signature: ez6p8p1xshg5fua9yyg447fun8q9j3uy X-Rspam-User: X-HE-Tag: 1674742600-110134 X-HE-Meta: U2FsdGVkX186+TIv7tNz/la1ABxWzkb0QqnwOYdGRfOfA/gL9+0Po9/gkWpPub91NjdGXLVJ5OGZGxT44qSu2E7Jv0hcILS86yQxgiw+ETFOY2zssqp58fQXM8X/U65bEybMkdn2bdQArxqLJsPrMFlWLfabUr0xFmxJXc5CqE0nmw5MJ99R6+lRzHGZ0CDAFOASJ81oGSSA+kV91Lsn04aOoldROMFK1qcG1la8/WjyoN9mWMVzKF2MOD2yO1NMTyA3qNfCsBOWu8o72QNmiHbQFEENQkT81pJ+eSpfUUGLlk8zShHiTyXHIhUDe1lrLa6BfLQtDqZDPDDSNC/TrWg9UlBmRPwyOvcLfIN0Tll0qsHfp9Wbvpb+W5s/0hQsYmkahztRUYjw/DD1eUFdvlvIcex8gY/vlqGjUVd2cyu18m/VgMklvNyDkyoAVPnH3RWbKvgd3q94supnUyXcoujVLJApx3Po8MQY531FdqS8c8appVv6by2nTHq37pWL4/paINdrIGozcvJdvZUXVJM3ejcIcucxsZTyy/wqDp+9jH0ipgsDLeHcmNs0qmZGijAdwdDfOp+3mh5ty/U5a/mcDefxU63T1bl8VOhPwwDH7jubMrI3+QnCjQyluJnaQadrWyT5ULCFKw2ktC/WQS4eJy+UiCMPBPRljU3fcD6zJBJ1HgUQG1+o7GxEsRkSHU5DaSj+MKvE+NN5HgQyD1OlIvA9F3eKnFBqJc0yw8dDSCOcLbGxTjz4NAi5tuqcuuF3TlN4jUvCaEL0Tbztgu5H1tkm57I+i3KzR9OXiF4fyU/nSzOcqhnWElfXzbdXMDCZlbvasPRDY4I/LEujH6UW7l0jCOC4JzyyCHftgqrylE/W5RHI7fCw1N25mOI3Y38VX0i2jxUL1dYHg/PJaAUNXOXy0agQl9vafJJJFAHm+HwlxHh7AtbUo8/4o3QYUmHAOnWLBi+Zgrqw6vJ ei37YrzP H31wslI7qMH5hXP6PjnBnKEdMiZS/dSlrAO//vezO8YLoFddD80JkPNSZ8/c8O8mhmbWtZWuQD4PQu+CzMR7H/EBZeljdZao+iHfb5M6zmy6fzva+UceOUk8pKtFb4joRJR3pSsZnZ9zhwt+px92btYpIM0vfvFUIJ3SrxSoXQBkHYYHHL0gejSiQ3DP9EPjeqsBFgqF4DE6mlNSj+7GUcRzs+SUPqphsYD54GtrYQc0ju70ALnA7T1NVBanFsRE85sLkXZBceDaEyge4E891rbifJw1HsMQduzicW5ZqhgwuTiJzg6xo0rYhDrizvbAtZZouMUg0liB6IjpZyZ4UIYV3w5wdgLFRQVrp+qtPP0np5utBVlYX6BHmy5WTmQ6wo3nZ9vlKYj26OI8qGD41vrBHkvd+xipfjS94k6CZt2vTVOxJTQm2+ndUnWVbzuggfgJl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags. For now only a flag to allow peer-to-peer DMA is supported. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Logan Gunthorpe cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org --- Notes: ver #11) - Use __bitwise for the extraction flags ver #9) - Change extract_flags to extraction_flags. ver #7) - Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*(). - Drop the I/O direction constants for now. block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 10 ++++++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 23 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index ab59a491a883..b97f3991c904 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1245,11 +1245,11 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { + iov_iter_extraction_t extraction_flags = 0; unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt; unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - unsigned int gup_flags = 0; ssize_t size, left; unsigned len, i = 0; size_t offset, trim; @@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; /* * Each segment in the iov is required to be a block size multiple. @@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) */ size = iov_iter_get_pages(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extraction_flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/blk-map.c b/block/blk-map.c index 19940c978c73..080dd60485be 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -265,9 +265,9 @@ static struct bio *blk_rq_map_bio_alloc(struct request *rq, static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, gfp_t gfp_mask) { + iov_iter_extraction_t extraction_flags = 0; unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); - unsigned int gup_flags = 0; struct bio *bio; int ret; int j; @@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; if (blk_queue_pci_p2pdma(rq->q)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; @@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (nr_vecs <= ARRAY_SIZE(stack_pages)) { pages = stack_pages; bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extraction_flags); } else { bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, gup_flags); + LONG_MAX, &offs, extraction_flags); } if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; diff --git a/include/linux/uio.h b/include/linux/uio.h index 9f158238edba..bf77cd3d5fb1 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -13,6 +13,8 @@ struct page; struct pipe_inode_info; +typedef unsigned int iov_iter_extraction_t; + struct kvec { void *iov_base; /* and that should *never* hold a userland pointer */ size_t iov_len; @@ -252,12 +254,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start, - unsigned gup_flags); + iov_iter_extraction_t extraction_flags); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start, - unsigned gup_flags); + iov_iter_extraction_t extraction_flags); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); @@ -360,4 +362,8 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +/* Allow P2PDMA on the extracted pages */ +#define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01) + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f9a3ff37ecd1..553afc870866 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1432,9 +1432,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i, static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start, - unsigned int gup_flags) + iov_iter_extraction_t extraction_flags) { - unsigned int n; + unsigned int n, gup_flags = 0; if (maxsize > i->count) maxsize = i->count; @@ -1442,6 +1442,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; if (likely(user_backed_iter(i))) { unsigned long addr; @@ -1495,14 +1497,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start, unsigned gup_flags) + size_t *start, iov_iter_extraction_t extraction_flags) { if (!maxpages) return 0; BUG_ON(!pages); return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, - start, gup_flags); + start, extraction_flags); } EXPORT_SYMBOL_GPL(iov_iter_get_pages); @@ -1515,14 +1517,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - size_t *start, unsigned gup_flags) + size_t *start, iov_iter_extraction_t extraction_flags) { ssize_t len; *pages = NULL; len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, - gup_flags); + extraction_flags); if (len <= 0) { kvfree(*pages); *pages = NULL; From patchwork Thu Jan 26 14:16:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DA55C61D97 for ; Thu, 26 Jan 2023 14:16:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 175236B0073; Thu, 26 Jan 2023 09:16:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 124FE6B0075; Thu, 26 Jan 2023 09:16:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE1A58E0001; Thu, 26 Jan 2023 09:16:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D499C6B0073 for ; Thu, 26 Jan 2023 09:16:45 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 639AB1C5B55 for ; Thu, 26 Jan 2023 14:16:45 +0000 (UTC) X-FDA: 80397151170.05.9EBA648 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 41D2D40021 for ; Thu, 26 Jan 2023 14:16:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UmBSngJK; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yDwFcqZk6I5FlMB5P/c+emcqWOnXr9CEIGCYw1ruBS0=; b=8CJ73c6iqu779fmwn+jnPKawxx/zkL+x8zbGGgr39ZKZnVikk6CHEomrCHMEnJ0E1E6cYZ f70ZDafoyabOSC0t1NQ3pR8foPpyHRv36XnbO3/PN23X4fEZwgM8v2K3VoACQHE8dWnN95 CHUGbLCXtzykEbu87vqawBrjR2q/9Lo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UmBSngJK; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742603; a=rsa-sha256; cv=none; b=EMQ5/ZZor/TFTFkV4WoOjuyithw6FB0BWdgErcutu2pux9bixoPD7Yg+wUgvnmOgtd9eST 3QP7BsRPmGLl/b4GzSANeEAq6j0Vx0tVdmn599qtAMSTZzFSpHyL1Z5sSt9uV6kzSdjr6c sIu6Jw4lBheEadQjkML2H8dxiN7MjvE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yDwFcqZk6I5FlMB5P/c+emcqWOnXr9CEIGCYw1ruBS0=; b=UmBSngJKsj1H9T4Cle3u18U0inCxq3JqfGF2XdobYSqrQDFSBoiNcc3J+BHxeRtD+J93C7 5pBiad4wHaTAt0IX3NZeQv6DJYzscM+VINZGXyX72A9vM1FVBu4xOjTGg/FdjIvkbbsl7q ss8yMtKpkUuQLQljKfMUt5sfHKRvms0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-570-N9o5qYQkPlqwiwd24vhK_Q-1; Thu, 26 Jan 2023 09:16:39 -0500 X-MC-Unique: N9o5qYQkPlqwiwd24vhK_Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6F650886C61; Thu, 26 Jan 2023 14:16:38 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4EF152026D4B; Thu, 26 Jan 2023 14:16:36 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 2/8] iov_iter: Add a function to extract a page list from an iterator Date: Thu, 26 Jan 2023 14:16:20 +0000 Message-Id: <20230126141626.2809643-3-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 41D2D40021 X-Rspam-User: X-Stat-Signature: f1wyokpbpukwt1wy6ofp95u8zcbuckxy X-HE-Tag: 1674742603-945075 X-HE-Meta: U2FsdGVkX182HB6aswMDanGlbJnTIm/pdYXMy/mOHCbNIGVmJLQhfmzq8/yBDuj8M3UTFhT4S2f4BY+Zebihh3gBbCcb7jO8+30wQ0DQ0V6S+9Ihqr3dZnPzgKRFsTL+m0Ay5cFgcsHyFlQ3sW381zPo+5Qks3NeNTtK7R4ZMCr4yeaFwo6LUZGZC7JnJecQsa1wnv6h0UOoNLhcDJU0Z/O3nKc3hAm2gOw27x9wuPpgLekpze3nlqVjaMXO0FeBmp5s1zdw/SUFNtNSwqpr8HzPt/v46gGEKRaWfJBPphdkS3uh6tqQB8iQ0TYNjhoFgphK3fy5IOEl8H5rrP/u0s+zfQWJSfBCM/JjYH4WY7ErL2NF/IJgr1pCZJ1m0WTAxDpQm0REaOjk1y9mMsRijW1LWbfJ630OKw0fcSHOrUmdNJSIT38FYs7a71TI+WWv7rPkhbfPGnUFrM/7faY7JMzQk8sKDeU3EJwUYiKjP21/kALbqsUiUW0XA3EsNgjOExLvaRmvgu+NM41EGDBs81XtScHMez/qlGME99QkKwG+ZRsieEaUKAc7Ie+lDhI+dO/BVqA5Aw6U0ow/hOcMrA2GGWh0qVkRaVo29EONC540YZqwWEqVqcrzPjdhYIXQBP2TFP+X1bfirshLhUXatq+DRb+EffoXyap5tXQuQ2oVToXf3+a0s33+zKhVMkuVMi5e78zF9ZxU8NK+Al6XGosRbwv8rBqosedLe4NGX70X7bUBhBbaLBWrQHZY3v6g1Tqacy6O5A5K+66xLMIFAF6ro1IiVJM4F+kbQC/UQ5XIHt1T7DKY/FyFohCcQED8BcA1BigvLyi7SPyKwcNETlXTJ28VJwpegTUiOctFEfSZuVcrm0KgG8AQIxx2aiP6xcnni1J5yH/jbGzhVwV66swPZP2oiwp7dDUB55V1nqlrySQtnDjKWqKZol07mUucDQv0odrcILIlidfTeh6 8crMc9yH AsH/+CHXVBbU9S1EWasphjjXJ+7smOXPJ1oxIhCt8zE306NCl7n1+n7HmeA6kqlnfzXt+11cv2DkjoSF43sut6zW14ZMB1RAjy72rIGK35qRbBbEOpTdP1lAQ5zm+hw23rY+Nc6t2GHi36GLEq7HqsupNXN7IRbmCoVx9G0B3NGE3VKjZ5Zfwin+NFigqYblUy49pFr6ea4sDkP4h+NX3/HACQs+FhezvJqMppaLZmgZzoLuxdCM18s6+vbzlEF0694EcmBnKG12QBEUgL+DrT47NJZm1ToKFSUQEbYkH8Ce6zoE6bpnC1M/NQ3YezPLtvHIUtN8cSlOBlr3Uh9+J6OHe4QSRAT0osQX/ymy5ccX4LHqFVzLlvl66ZSr5p0uui2PdfnaZy0ufFEGmj07KAe7rfZBv58LuQnLluudok1ACY4uBFEOVSodQ+8FH/8CuoKZk9kPBuwFghqXDPIZeo38+XtHUqkkglFbSDhm+PPXBcx8SR7gP3nZvO1M1I2zttpRnpgYg9qC8FyZYMVFM+wcbR75HWUmjBRCBigei6SVNF7eL+4x2/GPBPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a pin added or nothing, depending on the type of iterator. Add a second function, iov_iter_extract_will_pin(), to determine how the cleanup should be done. There are two cases: (1) ITER_IOVEC or ITER_UBUF iterator. Extracted pages will have pins (FOLL_PIN) obtained on them so that a concurrent fork() will forcibly copy the page so that DMA is done to/from the parent's buffer and is unavailable to/unaffected by the child process. iov_iter_extract_will_pin() will return true for this case. The caller should use something like unpin_user_page() to dispose of the page. (2) Any other sort of iterator. No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention. iov_iter_extract_will_pin() will return false. The pages don't need additional disposal. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- Notes: ver #11) - Fix iov_iter_extract_kvec_pages() to include the offset into the page in the returned starting offset. - Use __bitwise for the extraction flags ver #10) - Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec. ver #9) - Rename iov_iter_extract_mode() to iov_iter_extract_will_pin() and make it return true/false not FOLL_PIN/0 as FOLL_PIN is going to be made private to mm/. - Change extract_flags to extraction_flags. ver #8) - It seems that all DIO is supposed to be done under FOLL_PIN now, and not FOLL_GET, so switch to only using pin_user_pages() for user-backed iters. - Wrap an argument in brackets in the iov_iter_extract_mode() macro. - Drop the extract_flags argument to iov_iter_extract_mode() for now [hch]. ver #7) - Switch to passing in iter-specific flags rather than FOLL_* flags. - Drop the direction flags for now. - Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA. - Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter. - Add support for extraction from KVEC-type iters. - Use iov_iter_advance() rather than open-coding it. - Make BVEC- and KVEC-type skip over initial empty vectors. ver #6) - Add back the function to indicate the cleanup mode. - Drop the cleanup_mode return arg to iov_iter_extract_pages(). - Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source. ver #4) - Use ITER_SOURCE/DEST instead of WRITE/READ. - Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in. ver #3) - Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1]. include/linux/uio.h | 27 +++- lib/iov_iter.c | 321 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 347 insertions(+), 1 deletion(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index bf77cd3d5fb1..b1be128bb2fa 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -361,9 +361,34 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, .count = count }; } - /* Flags for iov_iter_get/extract_pages*() */ /* Allow P2PDMA on the extracted pages */ #define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01) +ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages, + size_t maxsize, unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0); + +/** + * iov_iter_extract_will_pin - Indicate how pages from the iterator will be retained + * @iter: The iterator + * + * Examine the iterator and indicate by returning true or false as to how, if + * at all, pages extracted from the iterator will be retained by the extraction + * function. + * + * %true indicates that the pages will have a pin placed in them that the + * caller must unpin. This is must be done for DMA/async DIO to force fork() + * to forcibly copy a page for the child (the parent must retain the original + * page). + * + * %false indicates that no measures are taken and that it's up to the caller + * to retain the pages. + */ +static inline bool iov_iter_extract_will_pin(const struct iov_iter *iter) +{ + return user_backed_iter(iter); +} + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 553afc870866..d69a05950555 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1916,3 +1916,324 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) i->iov -= state->nr_segs - i->nr_segs; i->nr_segs = state->nr_segs; } + +/* + * Extract a list of contiguous pages from an ITER_PIPE iterator. This does + * not get references of its own on the pages, nor does it get a pin on them. + * If there's a partial page, it adds that first and will then allocate and add + * pages into the pipe to make up the buffer space to the amount required. + * + * The caller must hold the pipe locked and only transferring into a pipe is + * supported. + */ +static ssize_t iov_iter_extract_pipe_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + unsigned int nr, offset, chunk, j; + struct page **p; + size_t left; + + if (!sanity(i)) + return -EFAULT; + + offset = pipe_npages(i, &nr); + if (!nr) + return -EFAULT; + *offset0 = offset; + + maxpages = min_t(size_t, nr, maxpages); + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + left = maxsize; + for (j = 0; j < maxpages; j++) { + struct page *page = append_pipe(i, left, &offset); + if (!page) + break; + chunk = min_t(size_t, left, PAGE_SIZE - offset); + left -= chunk; + *p++ = page; + } + if (!j) + return -EFAULT; + return maxsize - left; +} + +/* + * Extract a list of contiguous pages from an ITER_XARRAY iterator. This does not + * get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_xarray_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page *page, **p; + unsigned int nr = 0, offset; + loff_t pos = i->xarray_start + i->iov_offset; + pgoff_t index = pos >> PAGE_SHIFT; + XA_STATE(xas, i->xarray, index); + + offset = pos & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + rcu_read_lock(); + for (page = xas_load(&xas); page; page = xas_next(&xas)) { + if (xas_retry(&xas, page)) + continue; + + /* Has the page moved or been split? */ + if (unlikely(page != xas_reload(&xas))) { + xas_reset(&xas); + continue; + } + + p[nr++] = find_subpage(page, xas.xa_index); + if (nr == maxpages) + break; + } + rcu_read_unlock(); + + maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from an ITER_BVEC iterator. This does + * not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + size_t skip = i->iov_offset, offset; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->bvec->bv_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->bvec++; + skip = 0; + } + + skip += i->bvec->bv_offset; + page = i->bvec->bv_page + skip / PAGE_SIZE; + offset = skip % PAGE_SIZE; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + for (k = 0; k < maxpages; k++) + p[k] = page + k; + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of virtually contiguous pages from an ITER_KVEC iterator. + * This does not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + const void *kaddr; + size_t skip = i->iov_offset, offset, len; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->kvec->iov_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->kvec++; + skip = 0; + } + + kaddr = i->kvec->iov_base + skip; + offset = (unsigned long)kaddr & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + kaddr -= offset; + len = offset + maxsize; + for (k = 0; k < maxpages; k++) { + size_t seg = min_t(size_t, len, PAGE_SIZE); + + if (is_vmalloc_or_module_addr(kaddr)) + page = vmalloc_to_page(kaddr); + else + page = virt_to_page(kaddr); + + p[k] = page; + len -= seg; + kaddr += PAGE_SIZE; + } + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from a user iterator and get a pin on + * each of them. This should only be used if the iterator is user-backed + * (IOBUF/UBUF). + * + * It does not get refs on the pages, but the pages must be unpinned by the + * caller once the transfer is complete. + * + * This is safe to be used where background IO/DMA *is* going to be modifying + * the buffer; using a pin rather than a ref makes forces fork() to give the + * child a copy of the page. + */ +static ssize_t iov_iter_extract_user_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + unsigned long addr; + unsigned int gup_flags = FOLL_PIN; + size_t offset; + int res; + + if (i->data_source == ITER_DEST) + gup_flags |= FOLL_WRITE; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; + if (i->nofault) + gup_flags |= FOLL_NOFAULT; + + addr = first_iovec_segment(i, &maxsize); + *offset0 = offset = addr % PAGE_SIZE; + addr &= PAGE_MASK; + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + res = pin_user_pages_fast(addr, maxpages, gup_flags, *pages); + if (unlikely(res <= 0)) + return res; + maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/** + * iov_iter_extract_pages - Extract a list of contiguous pages from an iterator + * @i: The iterator to extract from + * @pages: Where to return the list of pages + * @maxsize: The maximum amount of iterator to extract + * @maxpages: The maximum size of the list of pages + * @extraction_flags: Flags to qualify request + * @offset0: Where to return the starting offset into (*@pages)[0] + * + * Extract a list of contiguous pages from the current point of the iterator, + * advancing the iterator. The maximum number of pages and the maximum amount + * of page contents can be set. + * + * If *@pages is NULL, a page list will be allocated to the required size and + * *@pages will be set to its base. If *@pages is not NULL, it will be assumed + * that the caller allocated a page list at least @maxpages in size and this + * will be filled in. + * + * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA + * be allowed on the pages extracted. + * + * The iov_iter_extract_will_pin() function can be used to query how cleanup + * should be performed. + * + * Extra refs or pins on the pages may be obtained as follows: + * + * (*) If the iterator is user-backed (ITER_IOVEC/ITER_UBUF), pins will be + * added to the pages, but refs will not be taken. + * iov_iter_extract_will_pin() will return true. + * + * (*) If the iterator is ITER_PIPE, this must describe a destination for the + * data. Additional pages may be allocated and added to the pipe (which + * will hold the refs), but pins will not be obtained for the caller. The + * caller must hold the pipe lock. iov_iter_extract_will_pin() will + * return false. + * + * (*) If the iterator is ITER_KVEC, ITER_BVEC or ITER_XARRAY, the pages are + * merely listed; no extra refs or pins are obtained. + * iov_iter_extract_will_pin() will return 0. + * + * Note also: + * + * (*) Use with ITER_DISCARD is not supported as that has no content. + * + * On success, the function sets *@pages to the new pagelist, if allocated, and + * sets *offset0 to the offset into the first page. + * + * It may also return -ENOMEM and -EFAULT. + */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + maxsize = min_t(size_t, min_t(size_t, maxsize, i->count), MAX_RW_COUNT); + if (!maxsize) + return 0; + + if (likely(user_backed_iter(i))) + return iov_iter_extract_user_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_kvec(i)) + return iov_iter_extract_kvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_bvec(i)) + return iov_iter_extract_bvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_pipe(i)) + return iov_iter_extract_pipe_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_xarray(i)) + return iov_iter_extract_xarray_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + return -EFAULT; +} +EXPORT_SYMBOL_GPL(iov_iter_extract_pages); From patchwork Thu Jan 26 14:16:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 079E9C54E94 for ; Thu, 26 Jan 2023 14:16:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87A916B0075; Thu, 26 Jan 2023 09:16:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DBF56B0078; Thu, 26 Jan 2023 09:16:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62DDE8E0001; Thu, 26 Jan 2023 09:16:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 536E96B0075 for ; Thu, 26 Jan 2023 09:16:48 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2F56EAAFAD for ; Thu, 26 Jan 2023 14:16:48 +0000 (UTC) X-FDA: 80397151296.19.F454111 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 7A3A416001A for ; Thu, 26 Jan 2023 14:16:46 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V3QiqFn8; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742606; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=UUeroDmTl+H3UYSLKfx3UyVjFR2p+V0BTEnxRsCOJP+EYOtuDy1kR/RKo6soVsQ4P5CYom u+BAjKnMLodEz8qq0dll6aOb7xKYuB4iKgc4QInqHCgBg1u4NPPAbrN9LvkLISbDLGayJp 0UUXrdFu+gN+hfHVZzoHQNn2u+TL1vg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V3QiqFn8; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742606; a=rsa-sha256; cv=none; b=1XwSvYHgMkngKUZ6KhkmmilCI36qE7KPm4LpwzqvrJCxO0F4vKwGWt1qE6Yncj97R7nLmj IC17DJ6zoIaLf8rqNVaJBYMyua/aUPCCuQ1aogsYy6FPdC9HMRqE6YzTOmWsSAi1R86Jt+ E/tEk0mthBDof+jXNV8njlgz+mpexvw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742605; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=V3QiqFn8d02UZ1ovwc4hMqXd14d4PnPtEqGQfgjZTp6jopZ8v2zeE/oH10u7GqyZpB6oKJ TQGEDRJ+trca/8I7ufB8sOF4fVOxHLXX1/oo66YKcbJin9+GlRBgIu67HR3JW7jYO9mfHv 7n5dKsMloCAn0Cm0s2Mz6X6W8XqLV8M= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-660-24XiFfRTO8-Jed3P6qNVRg-1; Thu, 26 Jan 2023 09:16:42 -0500 X-MC-Unique: 24XiFfRTO8-Jed3P6qNVRg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 013E41C29D45; Thu, 26 Jan 2023 14:16:41 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B45E492B01; Thu, 26 Jan 2023 14:16:39 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Hubbard Subject: [PATCH v11 3/8] iomap: Don't get an reference on ZERO_PAGE for direct I/O block zeroing Date: Thu, 26 Jan 2023 14:16:21 +0000 Message-Id: <20230126141626.2809643-4-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7A3A416001A X-Rspam-User: X-Stat-Signature: hbrfgxehbrpz46jffneudmrp7ttc9ygw X-HE-Tag: 1674742606-546009 X-HE-Meta: U2FsdGVkX1/vYz22u1tavRLqEqZk75nWX5ciEyCjGywSrSmXmM1gcuDeKDPNAJqbz0PCWP6x/P36wFGrktv7cnHp6Kt+VgHwsq+/mVe/zt4m9wWFrDLEW+NancNiH57JE39oBa71cVvywIwfykUp8ZlPVXFDfyHfoTgVU2Ti4DWUvlPpyjTCz4luvOtcLbaBBMuW9n8GJlG6UGHsqi3PkOMuKp/DQIdY/xxrEh7OoOr8iRwM5B+nmIiLpG4zarqvvSoi7DoX5JDdQpHIzRNXdHOm3kX4ae6jl74nJGzUD+RuFDsKB8btj90SjExBTDtDRJkIqJ05IGCF1SogFopppK3GI9cY5Q5DqcFa98T4Tj9MWnrjWnALyocq4sGcxb0gdZFWJ+Ns8zpXH/YTKKR3X7AOhfaWNoXi0EtlpXwJjDLUOTX3yEjRm7+BdxvmP6VpsnnfZpKdjjNLqph866ozBYRF7c/RH7ulLrM+hL7+3Jz0HHtBNC/SlUKCt/Rs4iLQxJtE3WlrjpaQ1Jt3XziingjIJPOAIMl2YvGH+Avrt6cVXjp3/TxuOmdcgfd0C0ayNmPEOLVb3qwodvYwM2TiZeSUxKztGYwXo/SIULg+3rUbt5OaSMK8EKvJIk/+a1Jp6hF+V2pjECRI5Y/Vbh9lQO6wtbUYdAosIG5AqOPMPPG8aUzG0ZRv+fBrB1czPl5uphT1sfrm+eYPmMqhwqSCvEwy6Rmx9kuSvaNws4O5kYhWkxqeocS0fz7sSKf41ZSLS8J0drB5YCodVmwT8ldxc1y6GPEP87XuT8dBzaxqZGwUSkpUp/Otmha0K7HOf6VO4bcbUEa06UFxltUAq8grI2P3ERjdTOgNH1tUATx1v8a7dR6XK0AiyllQvTdDi3NyvHQ4K9N+8GSMMQwZX1f7zdX5fvYUuk55iAXfw6p4yCqPxrG1JxYHhPQE6Gj7XhRZlmFrGXMculqf7QVzple mQSHnWvM s+t5CIuD9iu7UBaLVv1Dh80m3OAVty4cC9uq5wN3s+sLqVODtXy3uvYiEw4xwmrwPMvAf/m2eJbmoRJWLiUNhSxI9OlwWOlvf5RWG7vEiUXJcPegeerdcuDexUgyEObdGWKaLk06pIu/S1kp4thC/VivKegMjurfOU1lP1KwwIiDBLqSwxJbFXXVcGCyLUfG6T6vg2xGVaTjiZFEbwLlgILFieHayOC2Nnv5kunnLPBQ77GciRJm9eenQ4Hy8dcC4pNbdvQdK0kITIObVXhve9we7eAme+HK6gE7x6F/aSo8bKVpTuPZbcm2eg/gnhd3Bn5GY8eioDvw8npVQztvXYknAJpkObUo+iyK8cHKCqIw475Npz/mrvdFelarCYtJlEYbd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ZERO_PAGE can't go away, no need to hold an extra reference. Signed-off-by: David Howells Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard cc: Al Viro cc: David Hildenbrand cc: linux-fsdevel@vger.kernel.org Reviewed-by: Christoph Hellwig --- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 9804714b1751..47db4ead1e74 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - get_page(page); + bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } From patchwork Thu Jan 26 14:16:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73ED6C54E94 for ; Thu, 26 Jan 2023 14:16:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1749B6B0078; Thu, 26 Jan 2023 09:16:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 124FF8E0001; Thu, 26 Jan 2023 09:16:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F079B6B007D; Thu, 26 Jan 2023 09:16:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E27E06B0078 for ; Thu, 26 Jan 2023 09:16:51 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9B07940CF7 for ; Thu, 26 Jan 2023 14:16:51 +0000 (UTC) X-FDA: 80397151422.13.32708A1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id DF9E840013 for ; Thu, 26 Jan 2023 14:16:48 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="YADvt9l/"; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=H6WOtGjsn2aez7ttVp7/Gslbr79b6juUusL2rHFqO3TvBIsevbSpiDWXWM28On5HaeiKIA dW5tcbz6sKP4zK2u1L+NmBfYNX7Kv0tmxsVvVwrvlvx8RiXHj/OjFrIKRSnpqgRqwibDRO I0+Dr5xX5+SeEWBhvua8lQ9DQGj0oSE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="YADvt9l/"; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742608; a=rsa-sha256; cv=none; b=64JqN8WpQDPJY0P5CuWvbnOpvaZbnM2rfNfRe5PXRzWAHyyZM98PdeUC4IBlVd2SuDhhNw nbKrX2/IYsJXGZCNk+eqgrXM7cSMg8l9DTZ2fORdVImUIxeqz3rEnr6pG7BjtN4NZqTyKp /eDH3PJvxWNK6kZiBoiMlgbgL1I9XG4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=YADvt9l/TC3vAJPKaNcfrWVumeZ93LNJfKTlOnUTikwOuhAHdUvMVxYLD+q6OCcbzWdIOV NkKQ85fSDazyWnifJB3LFwlkBYl+NH4zr4V4mQnFJC4xwjoEE235eebJ2a5JE3B+9iAJvT 49pHoTa8CGLK4aiiEF3te8674m962i4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-632-TDWE-n9FPgCznZ4yl1rJMA-1; Thu, 26 Jan 2023 09:16:44 -0500 X-MC-Unique: TDWE-n9FPgCznZ4yl1rJMA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3DD3857F82; Thu, 26 Jan 2023 14:16:43 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id C432D2026D4B; Thu, 26 Jan 2023 14:16:41 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 4/8] block: Fix bio_flagged() so that gcc can better optimise it Date: Thu, 26 Jan 2023 14:16:22 +0000 Message-Id: <20230126141626.2809643-5-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: p33khu35jri349ug8qeh9zizdmxe37xq X-Rspam-User: X-Rspamd-Queue-Id: DF9E840013 X-Rspamd-Server: rspam06 X-HE-Tag: 1674742608-451937 X-HE-Meta: U2FsdGVkX19QGcBzB0SE1eKKYq101euQffNgaErh2yag2Vm6sCm/BmObTTTl810oyUbuvVTdKa7PQWNFqCCYVI/zDX+i58eJatscxMo4xsTKbKYzL9/WKt0czUbxdW/Cx6MSfYCJWecxLwIrXRU/0Y1g44NPBTNCOstKrerwJUNFAr9JNLJ3sRCamKUfQQEvRraTmYVwCm0J/J9IchM2NZzinHE7BuqZFq1PUz3Jev41aDpFnZow6gxfXp+lpsA7tyci/yv0lhD+9vtHM3fCuCCd06KpeY/0Dtu9P0evcZnppQHE9YichREFlymz2zBr/bP54WOY6eub1S5uMUBEeueBkyck07HpMgHRD/5mGdiTQ2Sf3F6dnhl8WOxE9YVkgSRXiHSUtmF4sq03JMx2Ylq1YSC6flz/Hyi8iIHoRVwFEqUHyDK3Lul9WUAYAOX6t3cJm74f6RrceLP4WAxOsd1Z1Il59v2vxSQxdxtQWwzeqsqznzJPqpJEKxD8CJ4uX0W1ee5PnfwrsowLsDzI/R0fBrVwlUqTIb7Vgc53Q8GbvKA6dr3ZeDl5bEZOorfLnZobqegPk18WaH5/EFX51DskOIJBHPhMiEuGRC5IuzcndSP+vpXZqcGJpKrFKvDEFakuqTC1Of+WVRNpRMv9D1LT0nkk/a3wANTdP6oQCOWWbdgNRwl4R6IpEWLff0uvbl77p3qmE7ICFdpfgizyE36/h8VArCgqIWLUpLvEoWyoUoZsOUv4uUPXihWh4EZmkpWLcrfiD8fF4IxnCkiXTMrISnnkU6SnPpvMORRWy5ZOMJdvVmLRKiJTfuiyWGSk4KR9hxKtbSciyFBocv8cBC9KXUccXt35aWbOdF5CUJF5ztTw3eIvUtch4b19fHKAc0FlQUEbW8mG7T/rHcToZp20IncTwrtSbgy7bwxBuQ3GPA/lnSHLC9MUmm+a3FlMADZtx76wlOzdy+90QtI KF5O+0wW tQuLhEGr0mNy6g8GOlPtHelQW8Q0BwdKJuGwMFhpu06pqFdWEYAOM5fuJgjTaOgp1+TatHBw5Ss9vIboaCHTkZG4HVCkeZeAkgN/LmAL8QPZeR1Ndj4zO1ISkGZZhm90VNzcYpGgzZtGkfcipsSUkj/MJK6PQ2Jqb/CrYXXNFpKNVl1kZoA7gVoqisiI5nzvympSQljTxIAQ8j1vptUldn0C9bwGsbiffYnO7M3coL7wrWbm+6rMyJ3dvQgatclruQ3JbLUK2pJk+42JJgwRYLt/ofORv+6yhXrwX/M2/dMzwz2UgldobxOSkAY8e36Aro6Vho2AdHSq4fBjUbxyaS/yFpocbsacW9QHjzLob1jnmM8PqhtUvZ7RyGOh/p46ZThEfmNNfWtCdGz0/pFdtoy9cM6oxTmK+OxpVPH5vUYg0hNfb0ewxwVF3NU+rqVu0R7Rg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix bio_flagged() so that multiple instances of it, such as: if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED)) can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]). The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like: <+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk> However, the test is superfluous as the return type is bool. Removing it results in: <+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk> instead. Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Jens Axboe cc: linux-block@vger.kernel.org Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108370 [1] Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108371 [2] Link: https://lore.kernel.org/r/167391056756.2311931.356007731815807265.stgit@warthog.procyon.org.uk/ # v6 --- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index c1da63f6c808..10366b8bdb13 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count) static inline bool bio_flagged(struct bio *bio, unsigned int bit) { - return (bio->bi_flags & (1U << bit)) != 0; + return bio->bi_flags & (1U << bit); } static inline void bio_set_flag(struct bio *bio, unsigned int bit) From patchwork Thu Jan 26 14:16:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B08BC54E94 for ; Thu, 26 Jan 2023 14:16:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE6186B007B; Thu, 26 Jan 2023 09:16:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6EC76B007D; Thu, 26 Jan 2023 09:16:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE8E96B007E; Thu, 26 Jan 2023 09:16:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9EAD56B007B for ; Thu, 26 Jan 2023 09:16:55 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 63A99160E69 for ; Thu, 26 Jan 2023 14:16:55 +0000 (UTC) X-FDA: 80397151590.24.A387D40 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 8E1361C001E for ; Thu, 26 Jan 2023 14:16:53 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fqvm7GPT; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dqniJi1fcm9DuVb2jGUzwy7ATEE/wnT//KjRosSsKEA=; b=lwCf62GrYo24IqhLVZafZn1ufBntVscj3CSFKFOtG8PMBwZ8vL9GaQNcrfgAQymcQBNkiD JGZa2ZYz/5J5G63gav4AbmFW5KAHQW1w0xGz0XjW7RJ3atsttRwPqm0nOt5YH9FNCnpyft sZjI09Rz3MHflv47lb0aB1IoR4KyzZk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fqvm7GPT; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742613; a=rsa-sha256; cv=none; b=dMSCpoofMwyFE7ifpYKKwWh8++rnNtuSyNWvqXF851qbA1+08C7D3H20WG15ALNPgn0v3p uN8gpe1amXuJhlGcU0vazQ4hDfiwdlipdFAqnvW0DpvkLRaEZpgoqJK2uskZLbjcIUqEDt iOD2D618d65rrL3t4DDwWNmYqUNr/hY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742613; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dqniJi1fcm9DuVb2jGUzwy7ATEE/wnT//KjRosSsKEA=; b=Fqvm7GPTdxZ6g/ZuzUgv/6Kj78xXDQbNjt+WAXhZWJpmVBdsuEr2eVmexyDdtJ2aW1QRuc hvpawrM6yOTF/Em3o7+Ncm2ZV1zilCOZgsceVyNLLBZmLoFY/G9j5t9BSOExc1425aIYNz frY+N9PIl3QAo6v4jzptiIwllMY63HM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-U4AecB_RMYiOOnaZf-fd3w-1; Thu, 26 Jan 2023 09:16:47 -0500 X-MC-Unique: U4AecB_RMYiOOnaZf-fd3w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 82D6E811E6E; Thu, 26 Jan 2023 14:16:46 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F31C4221F; Thu, 26 Jan 2023 14:16:44 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic Date: Thu, 26 Jan 2023 14:16:23 +0000 Message-Id: <20230126141626.2809643-6-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8E1361C001E X-Stat-Signature: dkqqdmkraai868as8is519upwmq7n7hh X-HE-Tag: 1674742613-339820 X-HE-Meta: U2FsdGVkX1+WCFpR3DQS2OST+St7udlbGoSDVa5mRt+rz1FBS9Ej2eVht7YiobM05KtsHZ+gY5AVNhu4whg1aVV2nxuvlmrZyFO3KXrwLalTjKbG5T+jTz1ayLs1Y4hA+NF5TdEt2BAvea573TT5+5rwLhj50Sg5KPBX4fjJFrtJmskWPmzlQ/xjQZttYaGx+6RRoDyaRgmKqJ3DJmNwmtcNY3SjyUeDtYFV70yG47wYqh6bKETWMuch20zZnW5fIZLuyIgd5/k3zkC1FNUBUGv/mJd4f5f3ckScry/j7mFpiNW7w8ONTE8+06ZztTn3HpTpzujWkY77gmkevpQk0OSQ4856KFkmOBuvHErEMspKn0YZXVgXvRvTUppQxc0DM0qbGynzWDCm6AlROISPTLAYrip7aRw3whw3AfzAWUVPvYkYgrHjrRnMcuTkhHydugVY1etI0+7utXFrvly2enksneHpOQgORL0G5AB+SEMHQRtKOE8sIm/N8dEa6nyCyThhmQKpUlz+LGPQDGJOhPdzCIatxzkgXdWCenZsrIHRPxwm1+bykmT3kXMo41BlnqA+yTeHZxyL2LHwkRfkNU4PMEzeaETsuCULsrbE2016TFPEsfKHT01s4DDNyOv8coKQeTtN1hNAbX27hOeUZ1c4V715aC9zvDKOf8Kl/hxMkaWBcEY6/rsmkvnZBs70Yl4FhdqjiDA05B/WwzwNCo43gqRVNJ8bhV5kGsswUd3FnRTrAcD7w70l5FjRsKqMrYR8Qzn/tMaZcKD84uwxWw1/iLqULmGG1AH1I6lECubiP7rhr1QmOY7Wc9Rll9SY7CuotUK4GlQIMKWfdLyOvOWc8SxLUTXC8dPrPB5fE3coWjbOZolJ6itjuEgxXcTB5ggnnwEDaiHSTRRpEIaZSajVTmGJpSGsvvEnYdWHTpt8abFGsN5USVUsW9TurdsumBjnJG3NXOkpc4A10vH lTufcEX2 n76wvM5CdV5I/6IoDccY6Vtsq5GQNDt0jOhthK9jB5RrLooeKIBXD9d8x78tNZ5Bh/1B/VshUdVBdhoRWbibEgScXbBdGziqlvpcDo4M0msXB/u6C9Byc59kUVugtMcNTOnVXQIu6Uwav9TO8ZEEjGyO7MfBdvggY5CRgYLB2nEGUNcL/8CPgDjIynysD9STFLvnY3dkqhRgsZsGVxfTQgJhIPQftKey3Nos5tP/6R5mXgjecYkM7HT0KdErv78gT1lXj9IgPkn+LPn+1kUacvts8z+23adNHos5Qj9qlcUYADqGNpnQQj5xJz9ywbH+wm9CmhHJVhWwsl72ebO0pRguPZxz5v2JT2f2k5oVD1shXXstuu3hjFaLnlQ70CVlE6U3q/X3BkMkl2qF0F9mN3Om+bNeuw6RnWTtJcgdgITIJxid3IRBvH79nm5Pc4uP4rJvfXAdY6JEzR7tELaTJ0u0obce5B+FpBtiedaRSfadI0MpyBiyboBuGpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Christoph Hellwig Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be released by bio_release_pages(). Signed-off-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #8) - Split out from another patch [hch]. - Don't default to BIO_PAGE_REFFED [hch]. ver #5) - Split from patch that uses iov_iter_extract_pages(). block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index b97f3991c904..bf9bf53232be 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1198,7 +1198,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; - bio_set_flag(bio, BIO_NO_PAGE_REF); bio_set_flag(bio, BIO_CLONED); } @@ -1343,6 +1342,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } + bio_set_flag(bio, BIO_PAGE_REFFED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); diff --git a/block/blk-map.c b/block/blk-map.c index 080dd60485be..f1f70b50388d 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -282,6 +282,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; ssize_t bytes; diff --git a/fs/direct-io.c b/fs/direct-io.c index 03d381377ae1..07810465fc9d 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; + /* for now require references for all pages */ + bio_set_flag(bio, BIO_PAGE_REFFED); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; } diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 47db4ead1e74..c0e75900e754 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } diff --git a/include/linux/bio.h b/include/linux/bio.h index 10366b8bdb13..805957c99147 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,7 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (!bio_flagged(bio, BIO_NO_PAGE_REF)) + if (bio_flagged(bio, BIO_PAGE_REFFED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..7daa261f4f98 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,7 +318,7 @@ struct bio { * bio flags */ enum { - BIO_NO_PAGE_REF, /* don't put release vec pages */ + BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ BIO_QUIET, /* Make BIO Quiet */ From patchwork Thu Jan 26 14:16:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ACC4C05027 for ; Thu, 26 Jan 2023 14:16:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14D106B007D; Thu, 26 Jan 2023 09:16:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D76F6B007E; Thu, 26 Jan 2023 09:16:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6B9C6B0080; Thu, 26 Jan 2023 09:16:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D79376B007D for ; Thu, 26 Jan 2023 09:16:57 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A1462C02E2 for ; Thu, 26 Jan 2023 14:16:57 +0000 (UTC) X-FDA: 80397151674.16.4DB14EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id DB9141A001C for ; Thu, 26 Jan 2023 14:16:55 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MepWsr+p; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GZNnQ4PFOAio/M36ha2lSqPmGocJT5Xn1bbdC3s23DY=; b=cDQB7KhIDyXSziSn/GI35cAKUCVQsG972STYDRbkBkFVcrOHqEwwuL/A8awpIl2bFqBDqW jdKfu2sW+Sv40X4Ej4mRF+aDm9/jUMWdLPJ16bXSY1Qc/PSYnWVmEYPC6Ny4ksvlrxQ61b C9wVPSrxEA+0uufkxD/oSCHCJHaVjMk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MepWsr+p; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742615; a=rsa-sha256; cv=none; b=3SkFAVPM2I3t5hJmv4DcEdmlrTX4NNnha+YkCMgrLb1kG/3AvJXMZEb03uvLzlsXEXr+oy 2bndU42OUAHDvAtwnMWhXlSe472OiDIglWqdmcdPG1aAw5hhYzBHlP7GQe+aJ4M7TD3shU iq9JDI76ZNtL+doIE8sc/ayuxUhfT1Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742615; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GZNnQ4PFOAio/M36ha2lSqPmGocJT5Xn1bbdC3s23DY=; b=MepWsr+pksZ++PfXyzByEi2+fYMP9mm2Twf+uxXL3FxbtteEK4h+zWMgTlBcZXrgc9Pfuu Ee18c2SeTLbkZnKqy5Gqvsz+HfuoT8xAagcmKzdCskgv6ryOgEXyVx5RtqxHSk5UFRmXc9 rw1R6rrWMTQla0/o74nOZeuiotx2RyE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-240-3eI7SoLlONeQMLaTsdqU5g-1; Thu, 26 Jan 2023 09:16:50 -0500 X-MC-Unique: 3eI7SoLlONeQMLaTsdqU5g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DEA25857D0D; Thu, 26 Jan 2023 14:16:49 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E20540C2064; Thu, 26 Jan 2023 14:16:47 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 6/8] block: Add BIO_PAGE_PINNED and associated infrastructure Date: Thu, 26 Jan 2023 14:16:24 +0000 Message-Id: <20230126141626.2809643-7-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: asq8x758jdqgfmfi5w5qa9rjbysfarc4 X-Rspamd-Queue-Id: DB9141A001C X-HE-Tag: 1674742615-94669 X-HE-Meta: U2FsdGVkX1/CP5eZVh2oo66pbwZ5JJ7/LHy/njc2xN1+tWGX3UwJKJEMV23F+EGpD0zgU5FPj4uNndTiRGYWhNXzv/6IA6GSEmANjVpTKkIiiGwdH1OrAsY3w11ciE1Y0C02P4YUNKC7S9lBLZ1kJXqu9OzOfeU5Frz8VCM5eQzbF3eb6wSswHeHKKm5/DP7thHwQbw2q7Ti8lqYiRKkKnkVkigi2Txr0wXcFVmr/49C7dVvrzhVfMZlBPRVaUNOtJ11tzVtiO3/etQ36ASgGnMXrHFyGm8cOPDnbj+v1Cj/En9un94s4RzpAi9t5Vh05RzOhdVIKMK/YDR2o1y7XKfEubL1djAzXOZ0Ukx9YTp95SP5QDqROU1PAA5a1+F4Ygn/ndyPV8GpySqT4K0Oz17zQtF50QIsFBx2qgoUmQ6m2JcyjLVjOGY2XtedUxVStPJnWC4Z9QSo/nccXv7RfyR7KLfiiAgm8yqRqa2RJ6giURFj7y+JGqDyg2c2UKTV5i1bLUAQRKVdd0jcHj5H1s8oOIPLD8TdZb1D9mKOJ4iIV9R8Zj4Bv9/lGr5Ny2p+5+U1UECX/yzH5I1Ii8RwOwogymr36FQroIOGQaeTDsYvCBQJUzeY6tA/kepU4B2axQiDZ82veZ2aTKTrAJKdDhBqgpwgqWjKiQA6MvYDIG06SJ5qvji0GJtmqe2Y0dAmkluHu5Q3afTXWgi7mJsDUnw50L7MHwQt2Vz1Y/YBeLdTtHXqKtFnculTPv54IlnxOciIaUSk6ecnhlFhvDL/u/6pR91sOohRnFH/8KYXD/wErY6GZZj48EPlVOsDo0zXc455eZVAP1AMlz9cbRYVfDbnm4xtz37NrNg+vyEWJXDfrESZGDwg13FmTfIzKTeKglv4pvuGxViQeZ7ClKv6O7FpQAUoRg198cT5i5b0o7cjSUT6Ea3swWyy5m810VCRA1XdYmdHJCwpqCXrv9L Nlcq/SlX KQ8j08yreKLnAz7X3I5Bzhu5OJ6oAjy7XnEUBwCe2YkRKZRsmSgztQVx6JIWfMUyufD63l4gvXPe6D5cwXx4Dmvu60WhIwSwcUe+5i8LqsKcegvvqafb/IUxgaXThmhPcf/6NZyQVCKL7PRjsl5kwDK8Wfe+zWQ7qf1Yi022wbXzcfA3GDCE2JDf4DyXa7rdPklW+k2Z5/xY+Sqb/d7CgP2+LR9Lt44jGHoNVBhc1p7AKjm7vDwyOdePWlejrOK1GJ0WJ9W7igu82t5wg0dO9ts2AJdwEhLOJkWrtQVjj8kZn9StiWVZVn68RY1sLuyteBfALWHgTj2cD8fl6sDvimoyBIi5TCoAUSi+T8j3SZILLV6r3eW2EoVJMwY482505Md1tDOJ8EIoQMS06WRKb66fZNIWw5cNsjllSQRcAd70r3/klXB1xwttltOr9ck7AY5y3yQHXeiGVHr5K4eY9XTbIOv7wjTxZlY4akolH/ecVD6p870+GeM9a5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #9) - Only consider pinning in bio_set_cleanup_mode(). Ref'ing pages in struct bio is going away. - page_put_unpin() is removed; call unpin_user_page() and put_page() directly. - Use bio_release_page() in __bio_release_pages(). - BIO_PAGE_PINNED and BIO_PAGE_REFFED can't both be set, so use if-else when testing both of them. ver #8) - Move the infrastructure to clean up pinned pages to this patch [hch]. - Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first. block/bio.c | 6 +++--- block/blk.h | 12 ++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index bf9bf53232be..547e38883934 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1176,7 +1176,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + bio_release_page(bio, bvec->bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1496,8 +1496,8 @@ void bio_set_pages_dirty(struct bio *bio) * the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * here on. It will unpin each page and will run one bio_put() against the + * BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..f02381405311 100644 --- a/block/blk.h +++ b/block/blk.h @@ -425,6 +425,18 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* + * Clean up a page appropriately, where the page may be pinned, may have a + * ref taken on it or neither. + */ +static inline void bio_release_page(struct bio *bio, struct page *page) +{ + if (bio_flagged(bio, BIO_PAGE_PINNED)) + unpin_user_page(page); + else if (bio_flagged(bio, BIO_PAGE_REFFED)) + put_page(page); +} + struct request_queue *blk_alloc_queue(int node_id); int disk_scan_partitions(struct gendisk *disk, fmode_t mode, void *owner); diff --git a/include/linux/bio.h b/include/linux/bio.h index 805957c99147..b2c09997d79c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,8 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (bio_flagged(bio, BIO_PAGE_REFFED)) + if (bio_flagged(bio, BIO_PAGE_REFFED) || + bio_flagged(bio, BIO_PAGE_PINNED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 7daa261f4f98..a0e339ff3d09 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,6 +318,7 @@ struct bio { * bio flags */ enum { + BIO_PAGE_PINNED, /* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ From patchwork Thu Jan 26 14:16:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21D8FC54E94 for ; Thu, 26 Jan 2023 14:17:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B38EC6B007E; Thu, 26 Jan 2023 09:16:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE8426B0080; Thu, 26 Jan 2023 09:16:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 914B76B0081; Thu, 26 Jan 2023 09:16:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 818436B007E for ; Thu, 26 Jan 2023 09:16:59 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 37E9DC0CF9 for ; Thu, 26 Jan 2023 14:16:59 +0000 (UTC) X-FDA: 80397151758.28.5D02117 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 5E3A1140017 for ; Thu, 26 Jan 2023 14:16:57 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R6XRf0t2; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742617; a=rsa-sha256; cv=none; b=05a0fkatuLnuhGQJNqbb4RQEzvMaZ2bi+k/CxFyrV1+9FAaT0VYgD8NK64iT9d94/TlBjb eZBKsq+856Q8BwYpaaJiZ4fbWI0M2q5BIgomiU8awj1bA/nI65U2vRlQmzyXeujauMT2uQ V9gAqxhLzNN/QCRLMxlMwhYIEuNh5Ag= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R6XRf0t2; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742617; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YX3eOMnEnHmNO2fsUm7QGJ9rpDZ86GbDPShQEUjns7k=; b=nrcK9ZskeFTwwKTYhdqisJeQ+flRMyEjUA2PZJCrKFNDN9sIl3M32PW44yA1+agRjvu94w bj4DGLYcZ91WbGl1pRW2oijXINrfVQv0w+Fp8XC/8ICXNk/c5a5o8qbZS1i1Ww+y3P0HYC etOJKsDK/wHkIlGAUogjV2AKVr86gUs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742616; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YX3eOMnEnHmNO2fsUm7QGJ9rpDZ86GbDPShQEUjns7k=; b=R6XRf0t2MQIQE7djLvhDVQ4oBBc3c1n+INKIN+7L1NYuFhGPGL+2nCAxvn4ACqjRl/jyzj 2XzIeT34eX6YdEoxFc6cHR3B8IpAA59PfhRm4hARgfnXS7C0ICc1cuUprwpcwHWd8S5fOF +ekT2DMEZBfGlFDGUIZPSbiMwsbzwaE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-257-pCAwC_fgPBOmzVMPFYtKxA-1; Thu, 26 Jan 2023 09:16:53 -0500 X-MC-Unique: pCAwC_fgPBOmzVMPFYtKxA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8F8A0885623; Thu, 26 Jan 2023 14:16:52 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 906022026D4B; Thu, 26 Jan 2023 14:16:50 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 7/8] block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages Date: Thu, 26 Jan 2023 14:16:25 +0000 Message-Id: <20230126141626.2809643-8-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Queue-Id: 5E3A1140017 X-Rspamd-Server: rspam01 X-Stat-Signature: mjx36xu6un43emmsbm9pxts63yozbunm X-HE-Tag: 1674742617-682242 X-HE-Meta: U2FsdGVkX18GV1OceOf8PManV3vcD/Sy3DB4UmT+ZZfuRfvlbN01jyFTppuj9d8MhP9upX6n6SbykOitfIN35sjLQQiSYSudsGWqEdWKb5616WWeSBhS3e0wePmdFNJocL4Rd4LZn+GBmA8eD9g2HvSd+Qxu6zXo26EbkfjsupxZQb78JuDMHO38ADktN8PACKAVYt+8b+ijptoOkF/TZUAxip5eQWn92pANfdxSi0p8R0oUwEPeyAwcwjyKXCUG+mIWnMmRS4yrgBqsZR7bxC2NaD+e3xD4X6ZZNEBZZ2vmV1K1DdLyfX5U+fbOIiPWdFKmLoPIVsUX/KVDJFHwmdwDoY7xFywJPxoIN1K77fkB2Zefrlto8wA1XdsgQZC0Ip7G5pyslGp3BH0FniYE+e3qapiYpLqjyJUimjVBjsGoL7/e354U6V4cXDnQtzQwLwhKIkPdPH5DYRaBLUqLE7jC12zeZkO3LAIxSCUYRqwjeaiosTBHjOovcSop64VR+Zefgurx+F67OeJ/3pk9w4/fDsB5mj5abby6CvhV/VlXqTkvsPl235mEBx7oXiknzdSeGmwVjNmavLTq04JFPgTj/5BhG0l38EfCUjBYjDABWmCxQZVPLBmR1Zwp//4ZoKgzwkQ1ynq3q2InpS4EQJ/BAF8IQ+j/Wptgaa4X7zGHF697U7Pr6a/vXiX3feWujmoBA+IJNGCxqz6Zt8wx7OI3UIHsYie9oiMILLeXleCT5vfw8t4G68g9zVLZ49GVTA/DrwzX32KjaQlL+1jFVLJzaN9Ycb9AR6SGK+V0UKda5wiWSJMEZPIK+4YBizEGw4mlUy7WkpP+EDdi9To+kxGIl6Fy+U/tLHo2ZoAjQ2YTTWA1Re7MqIqXGf1Y8iyPcilWGeGgORYdtilmOxZGVt+TH58zGqrf1UNgcZUJftYi6sHhxzCTCSXpF+d7E+PN3sI2J14yIhLjYGlayfp 3BG6O8fO DBDSH1eAfD9fJjpy2MUopVUm3DEQhwLpvW9BcPS6Yo/5G29VWlbw5FE8zvUtYrlngCIIAXqL8oDICCDw3WXAS09y0yTpdqGZHCQ0laCydvUrhKpmYORevDz12VllBg9qgYjHigDk2W/zSqNQ2z4CM8bqC+hMWQH2uTEmQTDuid8cnecfbJn8psE8QZYUs/ifWDHAWianiOqCKr7/qn52HV5dpOOeKr3ln70MBPkSFJuurA26FnciKo0jnEm6BSNhYTvPFWZu150lOSWgydoXWoZdfYg6gp3TmeonLK5cX0C6tVKfUbuaAt1WupBQeCSQCjvJ8/E0C3VDwNE1/jKGWl0/CfYBsPtyJ1+MBxFRlncSDwfolvtc25T8BQ06eRZaVwReXTYNPIJMYHZOWIunEN1IIs+N3X32B30W8JUieY4jPj6/Uds1p2qbDldgCGzhjfobvILTpECWlwZBZSuSbUECz0CJr6FXIFNCVZR+wsJHyzaUz6KJu7A2/eA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/bio.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/block/bio.c b/block/bio.c index 547e38883934..fc57f0aa098e 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1212,7 +1212,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1226,7 +1226,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1237,10 +1237,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Extracts pages from *iter and appends them to @bio's bvec array. The pages + * will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. + * For a multi-segment *iter, this function only adds pages from the next + * non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1272,9 +1272,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages(iter, pages, - UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extraction_flags); + size = iov_iter_extract_pages(iter, &pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extraction_flags, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1307,7 +1307,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + bio_release_page(bio, pages[i++]); return ret; } @@ -1342,7 +1342,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } - bio_set_flag(bio, BIO_PAGE_REFFED); + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); From patchwork Thu Jan 26 14:16:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13117199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8E7CC61D97 for ; Thu, 26 Jan 2023 14:17:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78F276B0080; Thu, 26 Jan 2023 09:17:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73E426B0081; Thu, 26 Jan 2023 09:17:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 606DB8E0001; Thu, 26 Jan 2023 09:17:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 518156B0080 for ; Thu, 26 Jan 2023 09:17:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 22448AB462 for ; Thu, 26 Jan 2023 14:17:03 +0000 (UTC) X-FDA: 80397151926.06.F7739EC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 534CA100017 for ; Thu, 26 Jan 2023 14:17:01 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RO2vMiL3; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674742621; a=rsa-sha256; cv=none; b=VeoB77kBv4keMXGVR/Iw8R7ocCfgmV4ynO5QeOmel21SRXlV0Kr0NzuyM8obFF2BLMUbkS gC0coEFZeMm6C2CiEkoJN27KSVJ4QhF9jyibrfLZwXPOgoGRD7eJKHNDZbkefDWjXMpw8u UBv0qEvfQdKC2TAdXY81aOBSP3GcoNw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RO2vMiL3; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674742621; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7RYncx3d2QCFno76YxQ26A3gARIW2WsBk/g5n5z32ew=; b=dIBDv8sHwAgT3F6AFYPvy43qfCXPFvcg30mZS/08xd8/OEzL7iLvdE4beBb2QZ1LiVmRSS aAzvVMB9nx3L9G9yOLjHs1zPe1er3g5jwC2nAlNnQPDxra9O+7ItXN9AwuqgpmROZYNUuN +hchkzFh8iAlW+qiW1cmmmj4X44JnoE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674742620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7RYncx3d2QCFno76YxQ26A3gARIW2WsBk/g5n5z32ew=; b=RO2vMiL3XB5fy5j06VapbtYiaTUQ8ADrEQozUND+3v1ZQhXNEKvnhA64hLZ+0Zd94T6V1Q zxiU5QId4ZoVS8fjnt7qbWEN5awHb2tQQEZZrTRD5oSvK/0DCG+2b1cdoRGpC/5NzHiGv4 lmKUhkp533ELlV2tQ4O0ukw1XsRL7tk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-567-vMncZTdFMO2Ma3wz53HZSA-1; Thu, 26 Jan 2023 09:16:56 -0500 X-MC-Unique: vMncZTdFMO2Ma3wz53HZSA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6072B886C60; Thu, 26 Jan 2023 14:16:55 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5BCAA2166B26; Thu, 26 Jan 2023 14:16:53 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v11 8/8] block: convert bio_map_user_iov to use iov_iter_extract_pages Date: Thu, 26 Jan 2023 14:16:26 +0000 Message-Id: <20230126141626.2809643-9-dhowells@redhat.com> In-Reply-To: <20230126141626.2809643-1-dhowells@redhat.com> References: <20230126141626.2809643-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Queue-Id: 534CA100017 X-Rspamd-Server: rspam01 X-Stat-Signature: shhf7k6mu5m3ebro3bsw9u71bp747jez X-HE-Tag: 1674742621-543314 X-HE-Meta: U2FsdGVkX1/46B5G7bdpCLUZtS5eWjgyvd3nrkFzP4oz+bL/ZldS+OvUVAlwXrK/FmXQLbIRh55s0msyZenUIIBAn8hOO4attY8YoXPRiTNZHegHgTCLTaXVJC81guRl6c75kq2pYJpOoOSLlDJOfjeAQgxM9ztBnl7qy8lYXzakJEe6thdsgxxywL2YnykEGouEUag3AMwFkNlvJEs2eK5+I7WuN0uVmjQmcy4K7AxHgDT3IgL5p9XNVuCTFyV5c3Myra+zB3d3Nycjbg/hPF0poqo5Xtl5LsjMsg0rNCUYd1f0AKIkmFHwOc9ARplsJEurp5akWsaMLjqJao1wNEnAQIDqVMVMCEWY2cZMUR+fCtUOvO8nuAGv7SWnrrrNdVVhIiaiGx+p/t3HAAnaj2IE81jibSjiC6usrCk5GFNacq6JLAnVLb7oUy7ov7bIIeZiSw3Tu6olsuO2HaLhA1WQ3A4/e3H4ccZBtsFZ1E/un3Y4JM7bggIeaiZrH+JL0RXVnv/fG0iOfBvVSjLjJGLd8ePf6UEH7UWM9WKvmmNGEzehbwVY+toWXeBALXMVNhYhQ8bMdiMs7NcU747wXaK8VJHNOwc2iaTkkM0r7rk5I6clpbDPbm0p1vBIZAl7YHF7rYymhf8/EpvVVu97fri/izb1c3vMoOhdNRK/3C3f6Pp9nC8VynPbDvaYJLcJcjliSn2vXWaX1uUkY5XkdJVvUg2UZf1kiW2zpsY5E9e0a0qBo6coys3kYM1gy+MQPIwwX7m/xuowlFTzyoDqtA+90Bd/hXObbdIo787Jl/Af878zU6Ujczqa6/bRZCLD6rm3E2F5x1rdGkm7waKigJBOmYdRnIXiUr0N8nKgD38PDeYnX+N0uCJsbvCMt8ItXD+BEwF0x5lPfwRsuGheit0s4VcAGRUIB8v+5sDyRBgIOxp5/JvfOvgxWoYe1LS8ET1Y6dCOu+QzRwUtTko S4kYeF0u bARRNEe+nO+t9I+CFTZHajkWYj3DrN5Ejd9QqgZswLwEFHyvTXzNDiUnijabz10iEsP7/XYgZ1WIl5+r49L1CYlohIJ125vcH32fbbZw294/igF3xJHaZQ4Yh4Mm4xOz1kcTzqYRgG0bTHANmhwTbpBv/Iy04v0Qf+ThJ6WW9dx2olY8XrPdtlKJLPQZFhUgbB7s01nWBjV5flDvGm0/pNeySsE9VWWC+bhsaPhEgEl/40JyB8RKQR/CQDN+0gUAfyf/kegnl2d7SRy1JSmUQoj1QXWW1+txrZN0luUEIsNlzwsbYPqwNRg4vprO2V6gjZ6XJJlFxhwx5nXoUasDMJdBEkt760tsbXD6lN0Ydmuda8l9Op0fN32JKA00w2Fyycx3q7WqrZD2clKLSRpvr5Y4IWzGZJGBZrgBnPGZ9s8maP/uFZWpExMpL9Z48Qi8RfDw4u0OBg4Ckr3ntR0zKOJ/HvCubTQmwk2b5OyNjFictgE0UXRLGwdQIQQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/blk-map.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index f1f70b50388d..0f1593e144da 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -281,22 +281,21 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); - bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { - struct page **pages, *stack_pages[UIO_FASTIOV]; + struct page *stack_pages[UIO_FASTIOV]; + struct page **pages = stack_pages; ssize_t bytes; size_t offs; int npages; - if (nr_vecs <= ARRAY_SIZE(stack_pages)) { - pages = stack_pages; - bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, extraction_flags); - } else { - bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, extraction_flags); - } + if (nr_vecs > ARRAY_SIZE(stack_pages)) + pages = NULL; + + bytes = iov_iter_extract_pages(iter, &pages, LONG_MAX, + nr_vecs, extraction_flags, &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -318,7 +317,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!bio_add_hw_page(rq->q, bio, page, n, offs, max_sectors, &same_page)) { if (same_page) - put_page(page); + bio_release_page(bio, page); break; } @@ -330,7 +329,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, * release the pages we didn't map into the bio, if any */ while (j < npages) - put_page(pages[j++]); + bio_release_page(bio, pages[j++]); if (pages != stack_pages) kvfree(pages); /* couldn't stuff something into bio? */