From patchwork Thu Oct 19 11:01:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428720 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E3751DFDA for ; Thu, 19 Oct 2023 12:04:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="hNMIyW0O" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231019120353epoutp035e3a91bdff90a2b3bcd6c8c1f6babbbd~PgKWxtJy42845028450epoutp032 for ; Thu, 19 Oct 2023 12:03:53 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231019120353epoutp035e3a91bdff90a2b3bcd6c8c1f6babbbd~PgKWxtJy42845028450epoutp032 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717033; bh=Pnjs48GpM2fNI5vQ32r+zx38m0Ym/uFN2UPMQUh9Xn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hNMIyW0Oq3/gGpQGpRI0tjVbaNSM8+QLf+o4o1288jicIa1X8wQILVUQAJvaciOaD aEKXBw4RwrQUWomhnd7uF8jLaUY/k+CW5yt0ej7pWd/73OnIYvYX3h2ASNGOkR3Nha ySdBTyi0kSuTFBsaGUV8dC+oFdQnbxj9JK3DdXKM= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120353epcas5p2c3d376275f88aa749617249515d4aaba~PgKWSn9ME0651406514epcas5p2_; Thu, 19 Oct 2023 12:03:53 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SB5wR2W7jz4x9Ps; Thu, 19 Oct 2023 12:03:51 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 41.78.10009.72B11356; Thu, 19 Oct 2023 21:03:51 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231019110822epcas5p4e09aab7295e48ef885f82dbd0576a584~PfZ4p1gfs2520825208epcas5p4Q; Thu, 19 Oct 2023 11:08:22 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110822epsmtrp23bd0179aca24fb5d4f8cc84fff709c52~PfZ4oxCrn1571815718epsmtrp27; Thu, 19 Oct 2023 11:08:22 +0000 (GMT) X-AuditID: b6c32a4a-261fd70000002719-f5-65311b271e9e Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id FA.57.07368.62E01356; Thu, 19 Oct 2023 20:08:22 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110819epsmtip16145c708b1f5884719ace0f791f0c46c~PfZ1SRj0v0329303293epsmtip1W; Thu, 19 Oct 2023 11:08:19 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Kanchan Joshi , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 01/12] block: Introduce queue limits and sysfs for copy-offload support Date: Thu, 19 Oct 2023 16:31:29 +0530 Message-Id: <20231019110147.31672-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHPfe2t4UMvKDEw2Mb6eIMGGgLLRxA3IZs3Lm5sJltkcRAR69A oKX0wWNEBZGRoTzksYVO3qCzbMAAGQ87CKxDIQRN5TmtQ1uDMvGFLAjCWq5s/vf5fc/vffLj 4s4VHDduglxNK+WSJB5hz+oc9PL22eUupAV9xtdQy/AfOHr8bIWFThS/wFHTzSICzQ8+Acjc nwdQTV0lC033d2PoUl0Jhi40GTBkWH9AoJKBCYAs41oM6Wd2o9pvGljokv4KCxl7zhKo+pyF g05NdhHo/NAahqaKLQB1mbMB6lypxlHz/EMWujzjjsZeDLHfdaO6tTc51JjpFxZlHNVQbbpv Caq94Tg1114BqN7pLIKqLyxlUwU5CwT12DLDoh7+Nk5QhR06QLWPZFJP296g2swPsMitUYl7 4mmJlFZ60vLYZGmCPC6U99HB6H3R4gCB0EcYhAJ5nnKJjA7lhX8c6fNBQpJ1JTzPVEmSxipF SlQqHn/vHmWyRk17xier1KE8WiFNUogUviqJTKWRx/nKaXWwUCDwE1sdYxLjFwuPKvJC042l XVgW6PDPB3ZcSIrgdfP3bBs7k70Arpjs84G9lZ8A2FBgZDPGEoBL8yfYmxHZc6dx5kEPYOl1 I4cxcjFYdOZnkA+4XILcDUfWuTZ9O5mLw74bfwObgZODODTU3iZsqbaRh+E1XS3LxixyJzy/ vr5RwoEMhrrFCY4tEST5sOiWk022I0Ng66k6nHFxglcqzBuhOPkmzLn4A85012sH524cYTgc 3mpaAAxvg/eHOjgMu8GnC3qC4TR4oexHwtYbJE8CqJ3Uvgx4B+YOF+G2HnDSC7b08Bn5dVg+ 3IwxdR1hwYoZY3QH2FW1yW/Bn1pqXuZ3hRP/ZBPMKBS89quMWXUhgJ1lMcXAU/vKNNpXptH+ X7gG4DrgSitUsjhaJVb4yem0/744NlnWBjZuxHt/F5j965HvAMC4YABALs7b7rCTEtDODlJJ xte0MjlaqUmiVQNAbN32GdzNJTbZemRydbRQFCQQBQQEiIL8A4S8HQ7zuZVSZzJOoqYTaVpB KzfjMK6dWxb2SeJ7xzLGAqdNgdPL1fYHx5Pq6QgQM2gWHPuiLurstL7xHsfv7mBeS6X45LJv rirlfZf8xv0iOHDo9NUD/Zfl37X0e/eIQ6bTymcMRepnLL7Hc3VV+fA5wYe3W8M8UiJ+t+Ol uejxYFeZwvGuMGdf5dU7zWt7/1yV10RPugateFlmY3UGfmoZ5vXoXubUVxxhRv9ShiakJHNh 6lPp0P1JpUxbuMAbVWRUGcq2fnbEaa1wzDLL9rCw3zbZOzaOHz4eMdPa1jdSZxLVL0fJF1dz /MJGy74U+aZWrzr5hx+6M7ZlKDh+hyw0rDhFpVz4fNfRuAPpfGV3uqn7+Zb8MveL+TyWKl4i 9MaVKsm/7stzqawEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrIIsWRmVeSWpSXmKPExsWy7bCSnK4an2GqQcMHdYv1p44xW3z8+pvF omnCX2aL1Xf72SxeH/7EaPHkQDujxYJFc1ksbh7YyWSxZ9EkJouVq48yWRz9/5bNYtKha4wW T6/OYrLYe0vbYmHbEhaLPXtPslhc3jWHzWL+sqfsFt3Xd7BZLD/+j8nixoSnjBY7njQyWmz7 PZ/ZYt3r9ywWJ25JW5z/e5zVQcpj56y77B7n721k8bh8ttRj06pONo/NS+o9Xmyeyeix+2YD m8fivsmsHr3N79g8Pj69xeLxft9VNo++LasYPTafrvb4vEnOY9OTt0wB/FFcNimpOZllqUX6 dglcGV/6agvabSsuT97B1MC4xbiLkZNDQsBEovFFD3MXIxeHkMBuRomlx36yQyQkJZb9PcIM YQtLrPz3nB2iqJlJ4tnx30AJDg42AW2J0/85QOIiAv3MEu/+TmcCcZgFLjNLTFt4hBGkSFgg WuL/WkOQQSwCqhLL//9nBbF5BawkVn25xg5SIiGgL9F/XxAkzClgLbGhexHYXiGgkgcLHrND lAtKnJz5hAXEZhaQl2jeOpt5AqPALCSpWUhSCxiZVjFKphYU56bnJhsWGOallusVJ+YWl+al 6yXn525iBMe6lsYOxnvz/+kdYmTiYDzEKMHBrCTCq+phkCrEm5JYWZValB9fVJqTWnyIUZqD RUmc13DG7BQhgfTEktTs1NSC1CKYLBMHp1QD08mbnuafNYTf982rcOHdHagd3H95gYJsesr3 hfq3hXc1/vU7HLX162/tysQNHyXmPTzPoXAzi2vx4urV/bJ+i3y6z/K7TVhq0v78pZpS+myW pN/vlerfrln6JyepOuN3fKucenOhwtSb/w7//rxv6zWz+BOSLxJ7PnnZpO7Yd/9LfPbbJcoH fs/x4Z+t2r88q1/J3XJPoFIvb3fhyY+TZx1e9WVTSndvl8K9WUvsP757ECC+mmFHkw9n/rRg 0Xv/LTzSD8acrPx2fJaToceJc0+dbXetkfO8WGMpuU6fdYGj8cXWdVsWBp2bKWlv7cqZ9zy2 58JqXzd1r8zVRbF5swPf8z+0Xy0bc//QTS0mbSWW4oxEQy3mouJEAF6pBXxkAwAA X-CMS-MailID: 20231019110822epcas5p4e09aab7295e48ef885f82dbd0576a584 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110822epcas5p4e09aab7295e48ef885f82dbd0576a584 References: <20231019110147.31672-1-nj.shetty@samsung.com> Add device limits as sysfs entries, - copy_max_bytes (RW) - copy_max_hw_bytes (RO) Above limits help to split the copy payload in block layer. copy_max_bytes: maximum total length of copy in single payload. copy_max_hw_bytes: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Reviewed-by: Luis Chamberlain Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 23 ++++++++++++++++++ block/blk-settings.c | 24 +++++++++++++++++++ block/blk-sysfs.c | 36 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 13 ++++++++++ 4 files changed, 96 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index 1fe9a553c37b..96ba701e57da 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,29 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_max_bytes +Date: August 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes that the block layer + will allow for a copy request. This is always smaller or + equal to the maximum size allowed by the hardware, indicated by + 'copy_max_hw_bytes'. An attempt to set a value higher than + 'copy_max_hw_bytes' will truncate this to 'copy_max_hw_bytes'. + Writing '0' to this file will disable offloading copies for this + device, instead copy is done via emulation. + + +What: /sys/block//queue/copy_max_hw_bytes +Date: August 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes that the hardware + will allow for single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 0046b447268f..4441711ac364 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_hw_sectors = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_hw_sectors = UINT_MAX; + lim->max_copy_sectors = UINT_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/* + * blk_queue_max_copy_hw_sectors - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + */ +void blk_queue_max_copy_hw_sectors(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (BLK_COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = BLK_COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_hw_sectors = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_hw_sectors); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_hw_sectors = min(t->max_copy_hw_sectors, + b->max_copy_hw_sectors); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 63e481262336..4840e21adefa 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -199,6 +199,37 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_hw_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_hw_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, const char *page, + size_t count) +{ + unsigned long max_copy; + ssize_t ret = queue_var_store(&max_copy, page, count); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + q->limits.max_copy_sectors = min_t(unsigned int, max_copy, + q->limits.max_copy_hw_sectors); + + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -517,6 +548,9 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RO_ENTRY(queue_copy_hw_max, "copy_max_hw_bytes"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -633,6 +667,8 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_hw_max_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index eef450f25982..7548f1685ee9 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -309,6 +309,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned int max_copy_hw_sectors; + unsigned int max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -893,6 +896,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_hw_sectors(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1211,6 +1216,14 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +/* maximum copy offload length, this is set to 128MB based on current testing */ +#define BLK_COPY_MAX_BYTES (1 << 27) + +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { From patchwork Thu Oct 19 11:01:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428722 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C6781DFF4 for ; Thu, 19 Oct 2023 12:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="StV9toHC" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231019120357epoutp030068fee3e1fdf389ffc54f3cfaf1931d~PgKaAL1_G2846628466epoutp03A for ; Thu, 19 Oct 2023 12:03:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231019120357epoutp030068fee3e1fdf389ffc54f3cfaf1931d~PgKaAL1_G2846628466epoutp03A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717037; bh=A08fzSS5yF5x8HdE6Ju9IHdNmMRpxoEL3T7ujwB/X78=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=StV9toHCyuqYLjAEH8PlVsTG4PkKInbfvTITRB7b1PqdSNxk6eCs8NOPcXww/QdtF pabMoEp2kYlHvIuMs1BsVawi+NbjlsQ/xCGjYDw+pZdfrLk103MdqH8/vroJbXu4Tb LstKFIZk1XhM/6rxg0AEwrV1ct4lQP6kge0iz98g= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20231019120356epcas5p1995aa540862c36a48598bcc177b777df~PgKZgM6eJ3087230872epcas5p1t; Thu, 19 Oct 2023 12:03:56 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.178]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4SB5wV661tz4x9Pv; Thu, 19 Oct 2023 12:03:54 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id F8.32.09672.A2B11356; Thu, 19 Oct 2023 21:03:54 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20231019110833epcas5p39702b3bd2c06fdce04e261c2e79f8bdd~PfaDDcegz1682516825epcas5p3t; Thu, 19 Oct 2023 11:08:33 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231019110833epsmtrp10bd306eac66f3302c6058a0d2577cd12~PfaDCTmm12693626936epsmtrp1g; Thu, 19 Oct 2023 11:08:33 +0000 (GMT) X-AuditID: b6c32a4b-60bfd700000025c8-98-65311b2a0ea9 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 7E.57.07368.13E01356; Thu, 19 Oct 2023 20:08:33 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110830epsmtip16ffe486e013330eff7e42444a5ab9598~PfZ-8_6HU1829018290epsmtip1b; Thu, 19 Oct 2023 11:08:30 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 02/12] Add infrastructure for copy offload in block and request layer. Date: Thu, 19 Oct 2023 16:31:30 +0530 Message-Id: <20231019110147.31672-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbVBUZRSe9967yyIDXVCml1URNxkSA3blwxeEcgZzbhEMTWEfNuEOewOG ZXfbuxsYQ+4CC4HBLhqYq8hnKKBQQM4iEgxICIViKAhJfLSIhIFCkMRqwe5S/nvO855znuec dw4Hd9LZcDnxEgUtlwjFPPYG4lLHTg8vz80Cmt+5/BKq6/kBR48WVwiUpnuCo5oRLRvNdMwD ZGzLAqikrIhAQ21NGKqq6cTQ8fYBgCZv6zHUMrwLlWZWEOhKSzeB+i+fYaPiykkbdGzQwEbn up5i6I5uEiCDUQ3QpZViHNXOzBHo2vBmdONJF2sfpJr0IzbUjV+/Jaj+XiVVX53NphoqjlL3 G04BqnlIxabK806wqNz0WTb1aHKYoOa+v82m8hqrAbVQ70rVG//AIh3eTwiOo4UiWu5GS2Kk onhJbAgv7K3o0Gj/AL7ASxCI9vDcJMJEOoS3/41IrwPx4tUl8Nw+EYqVq1SkkGF4Pi8Hy6VK Be0WJ2UUITxaJhLL/GTejDCRUUpivSW0IkjA5+/2X008nBDXka1jyVTByRUnsggVuMnPAbYc SPrBxoEmkAM2cJzIZgC/KDSwLME8gJqJEsISLAF4b6KHWC9RZ/Vbs1oALBwYwS2BBoOm8mos B3A4bHIX/PEfzhq/idTgsPXuA7MITpoweHyqnbXWaiP5ARyrPImvYYJ0h7qRAjO2J4NgX0st vtYIkj5QO+q4RtuSe+E3x8qsKY6w+5TR7Agnt8H0706bTUCy3BY+fvwLZrG6H7ad1lptb4S/ dzXaWDAXTmszrTgJVn15nm0pzgBQP6gHlodXoKZHazaBkzth3WUfC70VFvTUYhZhB5i7YrRq 2UPD2XX8ArxQV8K2YBc48JfaiimYf/IWZtlWHoB9C2ksHXDTPzOQ/pmB9P9LlwC8GrjQMiYx lmb8Zb4SOum/f46RJtYD82l4hhnAxNhD73aAcUA7gByct8neneLTTvYi4ZFPabk0Wq4U00w7 8F9deD7OdY6Rrt6WRBEt8Avk+wUEBPgF+gYIeM/bz2iKRE5krFBBJ9C0jJav12EcW64K07mo lPZyuwhe40L8529Of5zn6JBa4DOvSr1i+pmb8p7cVKU+70JdbOwPPbgvKTCsqKhmS8/rxS9m p+SOjn51sPJ64Z9+KRLunahio9303+MfujsHlTo5996K2GY3dyjd5Hwk/+J2rVI05hVdX+3h Su3uK+8ueLCstr3wzrnlZrV6PGK5tGZxyHQI5p2JEHZm7G3oC7lZYuP6Np3c8Joh3DA7k6rp 3KLNTO81+GKxUeFfn33YOi6uH/jNd+vV9HfL9gjdPRZ9Q6saOluzrt9N3t+XsuPAq58xnkwa q+y5ivsixpioXdred++px5JkcCrf7dpPGRL/jzzCZ3c0HL7qqzUenYriEUycUOCJyxnhv6dR HqyjBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra1BMYRzG5z3ndPaUKcfW8KpRLMVEtQfDG8nlA4cPbqMo43LUmTJtu2t3 cx8224iKdjSh+9JOaIlaNV2E2UhJUyyVZiLsVjRqKsZ1d9kaM749/+f3PM+XP4ULHxPe1H6p ildIOYmIdCOqGkS+QYwHw4sL1Ay6/bQRRyNffxHolNaGI0NPJokGG0YBsjxMBUh3tYBArx/W YOiG4TGGLpg6ALK+ysVQffd8dOW0nkD36psJZK7NJ1FRiVWA0jurSXTtiR1DXVorQNWWZICq fhXhqGxwmEBN3T6ozfbEZRVka3J7BGzbm3KCNbcmsRWlZ0nWqD/JDhhzAFv3Wk2yxeezXNhz miGSHbF2E+zw/Vcke/5uKWDHKnzZCstnbLNHtFtYLC/Zf5BXhITvdYtvOKt1kavDDuuzUgk1 eC5OA64UpBfD5FSzSxpwo4R0HYB1rS/ABJgOS2yP8AntCW/Y+wUTIQ0GUxr7/zYoiqTnwxYH 5fS96EwcDtkuYc4Dp9NweMFhEzjbnnQ0PJV/ZnyVoP2htid7fNWdXgbb68tw5xCkQ2Dm2ylO 25VeDu+kXx2PCP9GenUfBBPxKbA5x0I4NU77QU1lHq4FdO5/KPc/pANYKZjOy5WJcYkxjJyR 8oeClVyiMkkaFxwjS6wA458OnFcN3hTZg00Ao4AJQAoXebn7s2Je6B7LHTnKK2R7FEkSXmkC PhQhmubOXM6LFdJxnIpP4Hk5r/hHMcrVW40dS89oj2yGppaXkRkLjaPH8kH1etHPjC1+nMpY ZCAf3PaXG9pKji9YSqfIPjw1R2IxaGv784Qo9cW5rYx+sqC1b/aK3o1XfETZtV9GoiQWh6Ot b3v417GMroDr7yaPfe8V/CSCrKF+Ui5+0mX9oCX4RId3++z6zuHOhPIXvzvKfuy0h92dU76g 4Og+j9VfbvlaZ8l1moKmaStbys2Cd8kD1jmZxQbP0ggiXiYOXEOXeX88YFy7PPVbWN+MrcPw 04Goym0zszUR3z+twx2N62uWBPjKwvuGdvmr9NzBmpzdU7s+bipcgmpDsnYsSult4jacDmLe q07AwpsBz/ShKRJMRCjjOSYQVyi5P0Gc9qVYAwAA X-CMS-MailID: 20231019110833epcas5p39702b3bd2c06fdce04e261c2e79f8bdd X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110833epcas5p39702b3bd2c06fdce04e261c2e79f8bdd References: <20231019110147.31672-1-nj.shetty@samsung.com> We add two new opcode REQ_OP_COPY_SRC, REQ_OP_COPY_DST. Since copy is a composite operation involving src and dst sectors/lba, each needs to be represented by a separate bio to make it compatible with device mapper. We expect caller to take a plug and send bio with source information, followed by bio with destination information. Once the src bio arrives we form a request and wait for destination bio. Upon arrival of destination we merge these two bio's and send corresponding request down to device driver. Merging non copy offload bio is avoided by checking for copy specific opcodes in merge function. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-core.c | 7 +++++++ block/blk-merge.c | 41 +++++++++++++++++++++++++++++++++++++++ block/blk.h | 16 +++++++++++++++ block/elevator.h | 1 + include/linux/bio.h | 6 +----- include/linux/blk_types.h | 10 ++++++++++ 6 files changed, 76 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9d51e9894ece..33aadafdb7f9 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -121,6 +121,8 @@ static const char *const blk_op_name[] = { REQ_OP_NAME(ZONE_FINISH), REQ_OP_NAME(ZONE_APPEND), REQ_OP_NAME(WRITE_ZEROES), + REQ_OP_NAME(COPY_SRC), + REQ_OP_NAME(COPY_DST), REQ_OP_NAME(DRV_IN), REQ_OP_NAME(DRV_OUT), }; @@ -792,6 +794,11 @@ void submit_bio_noacct(struct bio *bio) if (!q->limits.max_write_zeroes_sectors) goto not_supported; break; + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: + if (!q->limits.max_copy_sectors) + goto not_supported; + break; default: break; } diff --git a/block/blk-merge.c b/block/blk-merge.c index 65e75efa9bd3..bcb55ba48107 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -158,6 +158,20 @@ static struct bio *bio_split_write_zeroes(struct bio *bio, return bio_split(bio, lim->max_write_zeroes_sectors, GFP_NOIO, bs); } +static struct bio *bio_split_copy(struct bio *bio, + const struct queue_limits *lim, + unsigned int *nsegs) +{ + *nsegs = 1; + if (bio_sectors(bio) <= lim->max_copy_sectors) + return NULL; + /* + * We don't support splitting for a copy bio. End it with EIO if + * splitting is required and return an error pointer. + */ + return ERR_PTR(-EIO); +} + /* * Return the maximum number of sectors from the start of a bio that may be * submitted as a single request to a block device. If enough sectors remain, @@ -366,6 +380,12 @@ struct bio *__bio_split_to_limits(struct bio *bio, case REQ_OP_WRITE_ZEROES: split = bio_split_write_zeroes(bio, lim, nr_segs, bs); break; + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: + split = bio_split_copy(bio, lim, nr_segs); + if (IS_ERR(split)) + return NULL; + break; default: split = bio_split_rw(bio, lim, nr_segs, bs, get_max_io_size(bio, lim) << SECTOR_SHIFT); @@ -922,6 +942,9 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (!rq_mergeable(rq) || !bio_mergeable(bio)) return false; + if (blk_copy_offload_mergable(rq, bio)) + return true; + if (req_op(rq) != bio_op(bio)) return false; @@ -951,6 +974,8 @@ enum elv_merge blk_try_merge(struct request *rq, struct bio *bio) { if (blk_discard_mergable(rq)) return ELEVATOR_DISCARD_MERGE; + else if (blk_copy_offload_mergable(rq, bio)) + return ELEVATOR_COPY_OFFLOAD_MERGE; else if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector) return ELEVATOR_BACK_MERGE; else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector) @@ -1053,6 +1078,20 @@ static enum bio_merge_status bio_attempt_discard_merge(struct request_queue *q, return BIO_MERGE_FAILED; } +static enum bio_merge_status bio_attempt_copy_offload_merge(struct request *req, + struct bio *bio) +{ + if (req->__data_len != bio->bi_iter.bi_size) + return BIO_MERGE_FAILED; + + req->biotail->bi_next = bio; + req->biotail = bio; + req->nr_phys_segments++; + req->__data_len += bio->bi_iter.bi_size; + + return BIO_MERGE_OK; +} + static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, struct request *rq, struct bio *bio, @@ -1073,6 +1112,8 @@ static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, break; case ELEVATOR_DISCARD_MERGE: return bio_attempt_discard_merge(q, rq, bio); + case ELEVATOR_COPY_OFFLOAD_MERGE: + return bio_attempt_copy_offload_merge(rq, bio); default: return BIO_MERGE_NONE; } diff --git a/block/blk.h b/block/blk.h index 08a358bc0919..b0c17ad635a5 100644 --- a/block/blk.h +++ b/block/blk.h @@ -159,6 +159,20 @@ static inline bool blk_discard_mergable(struct request *req) return false; } +/* + * Copy offload sends a pair of bio with REQ_OP_COPY_SRC and REQ_OP_COPY_DST + * operation by taking a plug. + * Initially SRC bio is sent which forms a request and + * waits for DST bio to arrive. Once DST bio arrives + * we merge it and send request down to driver. + */ +static inline bool blk_copy_offload_mergable(struct request *req, + struct bio *bio) +{ + return (req_op(req) == REQ_OP_COPY_SRC && + bio_op(bio) == REQ_OP_COPY_DST); +} + static inline unsigned int blk_rq_get_max_segments(struct request *rq) { if (req_op(rq) == REQ_OP_DISCARD) @@ -300,6 +314,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, case REQ_OP_DISCARD: case REQ_OP_SECURE_ERASE: case REQ_OP_WRITE_ZEROES: + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: return true; /* non-trivial splitting decisions */ default: break; diff --git a/block/elevator.h b/block/elevator.h index 7ca3d7b6ed82..eec442bbf384 100644 --- a/block/elevator.h +++ b/block/elevator.h @@ -18,6 +18,7 @@ enum elv_merge { ELEVATOR_FRONT_MERGE = 1, ELEVATOR_BACK_MERGE = 2, ELEVATOR_DISCARD_MERGE = 3, + ELEVATOR_COPY_OFFLOAD_MERGE = 4, }; struct blk_mq_alloc_data; diff --git a/include/linux/bio.h b/include/linux/bio.h index 41d417ee1349..ed746738755a 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -53,11 +53,7 @@ static inline unsigned int bio_max_segs(unsigned int nr_segs) */ static inline bool bio_has_data(struct bio *bio) { - if (bio && - bio->bi_iter.bi_size && - bio_op(bio) != REQ_OP_DISCARD && - bio_op(bio) != REQ_OP_SECURE_ERASE && - bio_op(bio) != REQ_OP_WRITE_ZEROES) + if (bio && (bio_op(bio) == REQ_OP_READ || bio_op(bio) == REQ_OP_WRITE)) return true; return false; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d5c5e59ddbd2..78624e8f4ab4 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -393,6 +393,10 @@ enum req_op { /* reset all the zone present on the device */ REQ_OP_ZONE_RESET_ALL = (__force blk_opf_t)17, + /* copy offload dst and src operation */ + REQ_OP_COPY_SRC = (__force blk_opf_t)19, + REQ_OP_COPY_DST = (__force blk_opf_t)21, + /* Driver private requests */ REQ_OP_DRV_IN = (__force blk_opf_t)34, REQ_OP_DRV_OUT = (__force blk_opf_t)35, @@ -481,6 +485,12 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return ((op & REQ_OP_MASK) == REQ_OP_COPY_SRC || + (op & REQ_OP_MASK) == REQ_OP_COPY_DST); +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. From patchwork Thu Oct 19 11:01:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428721 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96A171DFE3 for ; Thu, 19 Oct 2023 12:04:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="uiqtkGxH" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231019120401epoutp018c711447c684f6de596c47183a52c85b~PgKeHQwwh2665326653epoutp01d for ; Thu, 19 Oct 2023 12:04:01 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231019120401epoutp018c711447c684f6de596c47183a52c85b~PgKeHQwwh2665326653epoutp01d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717041; bh=8RJPLhVhgPuqIj+OvdyJBFvN6LyTk+71evbw3zEgBx4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uiqtkGxHUIp468+GHlMMjJrlhSgoLOorvR2Ize4OhYXGZysMpskFWWjD01LIl/eoD kcU4/s3fgKNMwsamxT/ZpeU4uyRBxJkaZtW2zNyA8+wT1uTmW3JjrgSepzqEBfnXxa QhjxB+AOfpJDttEpDRkBeIr5HK83a9Hb9JAJ8DUQ= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120401epcas5p2b2f1ab51b5b85d60cb2d6e22f253eaf6~PgKdncFD00920309203epcas5p2A; Thu, 19 Oct 2023 12:04:01 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.179]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SB5wZ1vkFz4x9Pq; Thu, 19 Oct 2023 12:03:58 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F5.78.10009.E2B11356; Thu, 19 Oct 2023 21:03:58 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20231019110843epcas5p18cc398bc87c141630677ef41d266004f~PfaMXe37q1154811548epcas5p1Q; Thu, 19 Oct 2023 11:08:43 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110843epsmtrp204a2a549b824a4abc8c32b937c4d5518~PfaMWR9O01629616296epsmtrp2B; Thu, 19 Oct 2023 11:08:43 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-0a-65311b2ec2be Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B6.C9.08755.B3E01356; Thu, 19 Oct 2023 20:08:43 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110840epsmtip1f494d1d0f2e7edb701ac308bef6b8144~PfaI2f1zk2963429634epsmtip14; Thu, 19 Oct 2023 11:08:40 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 03/12] block: add copy offload support Date: Thu, 19 Oct 2023 16:31:31 +0530 Message-Id: <20231019110147.31672-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbVBUVRju3Hu5u+BsXBfNA0xFi6iAwC4u2+HL0Ji6lT8Y/aFlhAvcgAF2 190FyyIgROP7e4gFFgSGAhpAIOQbZTG+ptkmQISARBdiJEDUtGlHbJeL5b/nfc7zvM953zOH i/OLOXbcKJmaUcqkMQLSimjXOTu7uduLGOFqmSVqGv0JRxt/GQn0de5THDXM5ZBoRfcAIMO1 SwBVVpUTaPpaJ4Z6qvIxVNdwA0P5AzcBWpzUYKh3xhVdvlhDoJ7eEQKNd5WRqKJ2kYMypjpI 9N3QJoZu5S4C1GFIBqjdWIGjxpV1Ag3P2CP90yGLAFu6UzPHofXzVwh6/Oc4uqU+jaRbaxLp 5dYSQHdPJ5F0dXaBBZ2VskbSG4szBL3eN0nS2W31gG4d+4J+2PIa3WJYxYKsP4r2i2Sk4YzS gZGFycOjZBH+gg9OhLwd4iURitxE3uhNgYNMGsv4CwKPBbm9ExVj2obAIV4aE2eigqQqlcDj sJ9SHqdmHCLlKrW/gFGExyjECneVNFYVJ4twlzFqH5FQ6OllEp6Jjrzcn0MoCgM+W/2mDSSB DHE6sORCSgzzipY56cCKy6e6Acwe7gJs8QDAemMzyRaPARz+dckk425ZUvIJlu8F8O5y2rY9 FYO163lbIpJyhWPPuGZ+F5WKw/7ZP7fa4pQWh613hjBzuA3lAzN/qSXNBoJygkX642aaZ6LL 5y8RbJgHzPl9p5m2pHxhc0YVzkp2wpESA2HGOPU6TPmxFDe3h1SDJVwqXSPZ2QKh7lGFBYtt 4L2hNg6L7eDDtd5tzTlYV/g9yZovAKiZ0gD24C2YOpqDmy+BU86wqcuDpV+FRaONGBv8Mswy GjCW58EO7XPsCH9oqtzubwtvPknexjTsy8zc3lw2gHXaAotc4KB5YSDNCwNp/o+uBHg9sGUU qtgIRuWl8JQx5/575TB5bAvY+iEu73eAhdv33QcAxgUDAHJxwS6eEy1k+Lxw6efnGaU8RBkX w6gGgJdp33m43e4wuemLydQhIrG3UCyRSMTehyQiwR7eSmp5OJ+KkKqZaIZRMMrnPoxraZeE Oa+ceiMqb2CPonvDSuvUdPfvLKHdUqJsf/zVpYyT0c9qTu9zGUvodM0OG8wL2s/XyjK/1F4t +sNgY2N/Pe0rx/cmZs9KvY99a3P7+on2iYL1x4O/Xdh3K/iQYbrXeMpzcPKT6R20LtRR4hts 1bORsCMgkZeQvHk6pfRAY+3Bl4rH/9Hfm/0wqvvonTrrwsUZlW62P3+SP/ZpH6esZ/Rk6fHg CZ/4g7q2sftH1zffPax4ggdVnrWdzJw/M9O8dyOwhLLXa1P5OuusUEz0sdwYXLy7fs0xLAKs LvS31J7fvOgZan+jP+eVOueltQN+ngvVlXP+ex/5UtVIUMIbmVrxv3JkQ58uIFSRUpELrlRJ /wVpXyDtqgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RbTBUYRiG5z3ndfYsszqWxtlUpk0/tONjpXr7tEMf76SRavqhSbXDGQmb 2Y2iJmorX8sixqCNMIQmWhLCSlIxtT5GQqkpYmpYKv3IqGxqpn/3XPd9P88z89CksAsuo0MV pzmlQh4upixh3SPxSpct1lLO/VeyG6rq7CDRzOwcRJfS50lU+UZLoc+PvgA02poAUGGRDqLB 1gYCNRVlEqi88jGBMtteAjTWn0eg5iEJunm1BKKm5mcQ9TVep1BB6RgPpQzUU6jsyU8CvUof A6h+9CJAdXMFJLrz2QTR0yEHZJx/YiET4Ya8NzxsHLkLcd/zKKyvSKJwTUkcnqjJBfjBYDyF i9OuWeBU9RSFZ8aGIDa19FM4rbYC4Jquc/irfiXWj04S/ksOW24N5sJDozml2/bjliduGrQw Mkt2djKxFsSDFM9kQNMs48mqM2EysKSFzAPA6k2vyGTAX+AitnS+/a+2Zct/jvMWQ2qCNTZ8 J8xlipGwXb9oM7djtCQ7NZ9DmAskc5tka3UCs7ZlNrOa7lLKnIfMGjbbeMCMBQtYN5IAF29w Y7VvbcyYz2xhq1OK/qwVLkTeFX7gLcZt2Ge5o3BxuiOrvpdPpgMm7z8r7z+rEBAVQMRFqiJC IlTSSKmCO+OqkkeoohQhrkGnIvTgz7/XOteD+xXTrm2AoEEbYGlSbCdYg905oSBYHhPLKU8d U0aFc6o24EBDsb3AfiI1WMiEyE9zYRwXySn/uQTNXxZP5EuL+7u9B6Y3ei3X7M3wmhiX5Qdo 3t+I4etaE2I1Vk7rO5N8jaHtSb7Nik64zXlPYmOLZCbKx3p1ztym2c5VA1Z21s7j0bs29O9/ LvYMahkxyddJrvhQ7RbZw2GBoiBeufMPLtYpgCqJ6/X2Gyy9kGWS6L753ZKpP5WVhTVV+SlT jxjKkz7K7LlD+1x7HsYO5MT0jHtIV4RKiOnz0jmc//HKzGZ+kwdhqI3zD3x9NGvY9FXrHxi+ zcvW4NIV7wE1Dgc39ZrIb8W+uJs/69KzdLKasRPt1j39ER3Skeh7ucC9Q/iuIHDf1Dp/fPzC TkP9ydzHmHmRcc5xR8YHsF0MVSfk0rWkUiX/DSFZMc5eAwAA X-CMS-MailID: 20231019110843epcas5p18cc398bc87c141630677ef41d266004f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110843epcas5p18cc398bc87c141630677ef41d266004f References: <20231019110147.31672-1-nj.shetty@samsung.com> Introduce blkdev_copy_offload to perform copy offload. Issue REQ_OP_COPY_SRC with source info along with taking a plug. This flows till request layer and waits for dst bio to arrive. Issue REQ_OP_COPY_DST with destination info and this bio reaches request layer and merges with src request. For any reason, if a request comes to the driver with only one of src/dst bio, we fail the copy offload. Larger copy will be divided, based on max_copy_sectors limit. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 204 +++++++++++++++++++++++++++++++++++++++++ include/linux/blkdev.h | 4 + 2 files changed, 208 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..0c5763095f7b 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -10,6 +10,22 @@ #include "blk.h" +/* Keeps track of all outstanding copy IO */ +struct blkdev_copy_io { + atomic_t refcount; + ssize_t copied; + int status; + struct task_struct *waiter; + void (*endio)(void *private, int status, ssize_t copied); + void *private; +}; + +/* Keeps track of single outstanding copy offload IO */ +struct blkdev_copy_offload_io { + struct blkdev_copy_io *cio; + loff_t offset; +}; + static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); @@ -115,6 +131,194 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +static inline ssize_t blkdev_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, + struct block_device *bdev_out, + loff_t pos_out, size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= BLK_COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +static inline void blkdev_copy_endio(struct blkdev_copy_io *cio) +{ + if (cio->endio) { + cio->endio(cio->private, cio->status, cio->copied); + kfree(cio); + } else { + struct task_struct *waiter = cio->waiter; + + WRITE_ONCE(cio->waiter, NULL); + blk_wake_io_task(waiter); + } +} + +/* + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to complete. + * Returns the length of bytes copied or error + */ +static ssize_t blkdev_copy_wait_for_completion_io(struct blkdev_copy_io *cio) +{ + ssize_t ret; + + for (;;) { + __set_current_state(TASK_UNINTERRUPTIBLE); + if (!READ_ONCE(cio->waiter)) + break; + blk_io_schedule(); + } + __set_current_state(TASK_RUNNING); + ret = cio->copied; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_dst_endio(struct bio *bio) +{ + struct blkdev_copy_offload_io *offload_io = bio->bi_private; + struct blkdev_copy_io *cio = offload_io->cio; + + if (bio->bi_status) { + cio->copied = min_t(ssize_t, offload_io->offset, cio->copied); + if (!cio->status) + cio->status = blk_status_to_errno(bio->bi_status); + } + bio_put(bio); + kfree(offload_io); + + if (atomic_dec_and_test(&cio->refcount)) + blkdev_copy_endio(cio); +} + +/* + * @bdev: block device + * @pos_in: source offset + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, + * for synchronous operation this should be NULL + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * For synchronous operation returns the length of bytes copied or error + * For asynchronous operation returns -EIOCBQUEUED or error + * + * Description: + * Copy source offset to destination offset within block device, using + * device's native copy offload feature. + * We perform copy operation using 2 bio's. + * 1. We take a plug and send a REQ_OP_COPY_SRC bio along with source + * sector and length. Once this bio reaches request layer, we form a + * request and wait for dst bio to arrive. + * 2. We issue REQ_OP_COPY_DST bio along with destination sector, length. + * Once this bio reaches request layer and find a request with previously + * sent source info we merge the destination bio and return. + * 3. Release the plug and request is sent to driver + * This design works only for drivers with request queue. + */ +ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, + loff_t pos_out, size_t len, + void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp) +{ + struct blkdev_copy_io *cio; + struct blkdev_copy_offload_io *offload_io; + struct bio *src_bio, *dst_bio; + size_t rem, chunk; + size_t max_copy_bytes = bdev_max_copy_sectors(bdev) << SECTOR_SHIFT; + ssize_t ret; + struct blk_plug plug; + + if (!max_copy_bytes) + return -EOPNOTSUPP; + + ret = blkdev_copy_sanity_check(bdev, pos_in, bdev, pos_out, len); + if (ret) + return ret; + + cio = kzalloc(sizeof(*cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 1); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + /* + * If there is a error, copied will be set to least successfully + * completed copied length + */ + cio->copied = len; + for (rem = len; rem > 0; rem -= chunk) { + chunk = min(rem, max_copy_bytes); + + offload_io = kzalloc(sizeof(*offload_io), GFP_KERNEL); + if (!offload_io) + goto err_free_cio; + offload_io->cio = cio; + /* + * For partial completion, we use offload_io->offset to truncate + * successful copy length + */ + offload_io->offset = len - rem; + + src_bio = bio_alloc(bdev, 0, REQ_OP_COPY_SRC, gfp); + if (!src_bio) + goto err_free_offload_io; + src_bio->bi_iter.bi_size = chunk; + src_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + + blk_start_plug(&plug); + dst_bio = blk_next_bio(src_bio, bdev, 0, REQ_OP_COPY_DST, gfp); + if (!dst_bio) + goto err_free_src_bio; + dst_bio->bi_iter.bi_size = chunk; + dst_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + dst_bio->bi_end_io = blkdev_copy_offload_dst_endio; + dst_bio->bi_private = offload_io; + + atomic_inc(&cio->refcount); + submit_bio(dst_bio); + blk_finish_plug(&plug); + pos_in += chunk; + pos_out += chunk; + } + + if (atomic_dec_and_test(&cio->refcount)) + blkdev_copy_endio(cio); + if (endio) + return -EIOCBQUEUED; + + return blkdev_copy_wait_for_completion_io(cio); + +err_free_src_bio: + bio_put(src_bio); +err_free_offload_io: + kfree(offload_io); +err_free_cio: + cio->copied = min_t(ssize_t, cio->copied, (len - rem)); + cio->status = -ENOMEM; + if (rem == len) { + ret = cio->status; + kfree(cio); + return ret; + } + if (cio->endio) + return cio->status; + + return blkdev_copy_wait_for_completion_io(cio); +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 7548f1685ee9..5405499bcf22 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1042,6 +1042,10 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, + loff_t pos_out, size_t len, + void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Thu Oct 19 11:01:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428723 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A7CC1DFFB for ; Thu, 19 Oct 2023 12:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="i11LTIcM" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231019120403epoutp01c80b51210803c7c2edbc6377b0c0bc68~PgKfsa7-a2715027150epoutp01E for ; Thu, 19 Oct 2023 12:04:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231019120403epoutp01c80b51210803c7c2edbc6377b0c0bc68~PgKfsa7-a2715027150epoutp01E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717043; bh=co+9TgP5yUjiGDtqASAf2IAo56nzxtAjKELuu/tyfJ8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i11LTIcM5cFi8sl1rOqX7OBe1SuPTR/HuZ25TftuVYcUq5TGdQX3t/Lgr3qEN3rsD kRS4N9aYhro6+jNVT5w+O7M/eOj59jFuNiSsBPvS5eWueENS9rfSCHUGBR9mIjKaYb WDTFkl1+/2Q3Tb9tiDq9sdwnbLBMezlzPZshX7y0= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20231019120402epcas5p464d0e2b3f29e245840d41a609a9d8fc9~PgKfVJbXe0467704677epcas5p4r; Thu, 19 Oct 2023 12:04:02 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SB5wd1Hxhz4x9Py; Thu, 19 Oct 2023 12:04:01 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id A8.78.10009.13B11356; Thu, 19 Oct 2023 21:04:01 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231019110853epcas5p42d50a676bb3168956edc4b87f3dd80c2~PfaVrxnb-2742527425epcas5p45; Thu, 19 Oct 2023 11:08:53 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110853epsmtrp280eadbad3ddb1c91c5ea10419eb3cc7c~PfaVq3v5O1571815718epsmtrp2d; Thu, 19 Oct 2023 11:08:53 +0000 (GMT) X-AuditID: b6c32a4a-261fd70000002719-14-65311b31649f Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F8.C9.08755.54E01356; Thu, 19 Oct 2023 20:08:53 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110850epsmtip15c04b58c22d3d2d4eeac6a0963c066f0~PfaSfSIGK2856028560epsmtip1G; Thu, 19 Oct 2023 11:08:50 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Vincent Fu , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 04/12] block: add emulation for copy Date: Thu, 19 Oct 2023 16:31:32 +0530 Message-Id: <20231019110147.31672-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHPb23l6KruzzUA2SuqzGbdkDLgB1QNl8zFzUGppnJYgYNvaEI tE1bEM2moKtbgQIKE6gIDGFCmYDASC0UtCAVFmCGCchTpWxsaDt8LUCYayls/vf5fX/v38lh Ye55Lt6sOImSlkuECVxiNd7UvuVdX4GPgOZbq5motrsTQ7MvFnB0JmcRQ9Vj2QSaaX8KkOXm NwCVll3G0f2bNxioqvo2A10wDQA0dU/LQMZhHvr+XDmOWoxdOOo3FBGo5IcpF5QxqCfQVfM/ DDSUMwWQ3pIGUNNCCYZqZmw4ujPsgyYzvgWob9HM3OFF3dCOuVB949dxqr8niarXqQmqofw0 Nd1QCKjm+6kEdSUrl0lpzloJanZqGKdsrfcIKqtRB6hn9RupessTRsTaz+O3i2mhiJZzaEmM VBQniQ3j7j8UtTsqKJgv8BWEoA+5HIkwkQ7j7jkQ4bs3LsF+CS4nWZiQZJcihAoF1/+j7XJp kpLmiKUKZRiXlokSZIEyP4UwUZEkifWT0MpQAZ8fEGQPjI4X320fYsoGdqX80TqCp4K5oHTg yoJkINRPZjIc7E42A7hgpNPBajs/BXCwvgVzGi8BnFfn4isZxfPtyw4jgPmlNbjTUDHgYnEe Mx2wWATJgz+/Yjl0T1KFwbbRx8BhYOQVDBrHSwhHKQ8SwRFr/hLj5GY4fmeS6WA2GQr7VY+W CkHSH2ZPuDlkV3IbrMsow5whbrCr0LI0EUa+Dc/+dGlpIkg2uMLe8qblUffAqq4SppM94J/m Rhcne8NnViPh5OOwKq+ScCZ/DaB2UAucjo+hqjsbcwyBkVtgrcHfKb8Fv+uuYTgbr4WaBQvD qbOhvniFN8Efa0uX63vBgb/TlpmC8x3NhPNaWQC+/M1I5ACO9rWFtK8tpP2/dSnAdMCLlikS Y2lFkCxAQh//75ljpIn1YOl7bN2nBw8f/OVnAgwWMAHIwrie7M0Un3Zni4QnTtJyaZQ8KYFW mECQ/eDnMe91MVL7/5IoowSBIfzA4ODgwJAPggXcDewZ1WWROxkrVNLxNC2j5St5DJardyrj Ypdp94k1oV35jS8CK0reuVZ4e81GtN7cfKT/kDWnA12fv2hIzT2g958Vp7QWnpHd7VXlH1P/ fvjRfJuuQJy2/trzVx5jB8c0LRONxziqUV3y2VMmdpX5dLpqOvFhd8Wq0VMF3ofDSt7jf2Ky dY5Yej/NA0XhX/R0FvLq0E7145PRBpedojptbum+UVaxuOBBWuQ57oVW9dyi5tav+ls6W8fR zKGjdRk+G0RvVBoaeb+YP2sNf1/9ZHZG+eY6S6TLV2V7iYUj+vOeZZeE2zQBKVfbNDF6GD5t GGnaMeyW3MMrmPpyrmKitjLEylvM3JQ/EWk7iFU9twVFW4m+0VVF+zO4uEIsFGzF5Arhv0uY BAynBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrBIsWRmVeSWpSXmKPExsWy7bCSnK4rn2GqwfkPxhbrTx1jtvj49TeL RdOEv8wWq+/2s1m8PvyJ0eLJgXZGiwWL5rJY3Dywk8li5eqjTBaTDl1jtHh6dRaTxd5b2hYL 25awWOzZe5LF4vKuOWwW85c9Zbfovr6DzWL58X9MFjcmPGW02PGkkdFi2+/5zBbrXr9nsThx S9ricXcHo8X5v8dZHSQ9ds66y+5x/t5GFo/LZ0s9Nq3qZPPYvKTe48XmmYweu282sHks7pvM 6tHb/I7N4+PTWywe7/ddZfPo27KK0ePzJjmPTU/eMgXwRXHZpKTmZJalFunbJXBlXDx8g7Xg mlPFy323WRoYf5p2MXJySAiYSMz7dZi5i5GLQ0hgN6PEp+3bWCESkhLL/h5hhrCFJVb+e84O YgsJNDNJdJ0P72Lk4GAT0JY4/Z8DpFdEoJ9Z4t3f6UwgNcwCG5klLh0zA7GFBSwkbr+bzgZi swioStw78RhsPq+AlcTl1kesIHMkBPQl+u8LgoQ5BawlNnQvYoZYZSXxYMFjdohyQYmTM5+w QIyXl2jeOpt5AqPALCSpWUhSCxiZVjFKphYU56bnFhsWGOallusVJ+YWl+al6yXn525iBEe2 luYOxu2rPugdYmTiYDzEKMHBrCTCq+phkCrEm5JYWZValB9fVJqTWnyIUZqDRUmcV/xFb4qQ QHpiSWp2ampBahFMlomDU6qByfXKjztLdp+1F1p+OPbU/fzHNRN0RJKfL5xyb4PlbIEOgwyF F7eu3VVk4HHjnseUwTv15MHTsTZT9O/0edW4LN+ZviVTet7jTz6bv3DcuSjw+fZS43xVRZ1f fyP/tMZsOMm6WqkyPWqp+QWneRH+WZy39JKsTK46zt6d2b1T/sNi9iDVhI1r5vdZOpht4mH5 nK1YXNJ92vHrhSOd0pxr2/91dDzZUXspZMeyxlgj4ZVnCtRdt07S2eBdySO74kH+VTFT13WL VVYx6YXcZb/Yc4/xUHDkqX9Wko+CwvMC/pxzcumbZVF6SO174Rnz15tOBr5f9j5q8qHDVQlt opZxNcZn3/4zXThnx4UXKzQnOiqxFGckGmoxFxUnAgCMYAK4WwMAAA== X-CMS-MailID: 20231019110853epcas5p42d50a676bb3168956edc4b87f3dd80c2 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110853epcas5p42d50a676bb3168956edc4b87f3dd80c2 References: <20231019110147.31672-1-nj.shetty@samsung.com> For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination. At present in kernel user of emulation is fabrics. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 223 +++++++++++++++++++++++++++++++++++++++++ include/linux/blkdev.h | 4 + 2 files changed, 227 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index 0c5763095f7b..25cba4878517 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -26,6 +26,20 @@ struct blkdev_copy_offload_io { loff_t offset; }; +/* Keeps track of single outstanding copy emulation IO */ +struct blkdev_copy_emulation_io { + struct blkdev_copy_io *cio; + struct work_struct emulation_work; + void *buf; + ssize_t buf_len; + loff_t pos_in; + loff_t pos_out; + ssize_t len; + struct block_device *bdev_in; + struct block_device *bdev_out; + gfp_t gfp; +}; + static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); @@ -319,6 +333,215 @@ ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, } EXPORT_SYMBOL_GPL(blkdev_copy_offload); +static void *blkdev_copy_alloc_buf(ssize_t req_size, ssize_t *alloc_size, + gfp_t gfp) +{ + int min_size = PAGE_SIZE; + char *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp); + if (buf) { + *alloc_size = req_size; + return buf; + } + req_size >>= 1; + } + + return NULL; +} + +static struct bio *bio_map_buf(void *data, unsigned int len, gfp_t gfp) +{ + unsigned long kaddr = (unsigned long)data; + unsigned long end = (kaddr + len + PAGE_SIZE - 1) >> PAGE_SHIFT; + unsigned long start = kaddr >> PAGE_SHIFT; + const int nr_pages = end - start; + bool is_vmalloc = is_vmalloc_addr(data); + struct page *page; + int offset, i; + struct bio *bio; + + bio = bio_kmalloc(nr_pages, gfp); + if (!bio) + return ERR_PTR(-ENOMEM); + bio_init(bio, NULL, bio->bi_inline_vecs, nr_pages, 0); + + if (is_vmalloc) { + flush_kernel_vmap_range(data, len); + bio->bi_private = data; + } + + offset = offset_in_page(kaddr); + for (i = 0; i < nr_pages; i++) { + unsigned int bytes = PAGE_SIZE - offset; + + if (len <= 0) + break; + + if (bytes > len) + bytes = len; + + if (!is_vmalloc) + page = virt_to_page(data); + else + page = vmalloc_to_page(data); + if (bio_add_page(bio, page, bytes, offset) < bytes) { + /* we don't support partial mappings */ + bio_uninit(bio); + kfree(bio); + return ERR_PTR(-EINVAL); + } + + data += bytes; + len -= bytes; + offset = 0; + } + + return bio; +} + +static void blkdev_copy_emulation_work(struct work_struct *work) +{ + struct blkdev_copy_emulation_io *emulation_io = container_of(work, + struct blkdev_copy_emulation_io, emulation_work); + struct blkdev_copy_io *cio = emulation_io->cio; + struct bio *read_bio, *write_bio; + loff_t pos_in = emulation_io->pos_in, pos_out = emulation_io->pos_out; + ssize_t rem, chunk; + int ret = 0; + + for (rem = emulation_io->len; rem > 0; rem -= chunk) { + chunk = min_t(int, emulation_io->buf_len, rem); + + read_bio = bio_map_buf(emulation_io->buf, + emulation_io->buf_len, + emulation_io->gfp); + if (IS_ERR(read_bio)) { + ret = PTR_ERR(read_bio); + break; + } + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, emulation_io->bdev_in); + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = chunk; + ret = submit_bio_wait(read_bio); + kfree(read_bio); + if (ret) + break; + + write_bio = bio_map_buf(emulation_io->buf, + emulation_io->buf_len, + emulation_io->gfp); + if (IS_ERR(write_bio)) { + ret = PTR_ERR(write_bio); + break; + } + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, emulation_io->bdev_out); + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_iter.bi_size = chunk; + ret = submit_bio_wait(write_bio); + kfree(write_bio); + if (ret) + break; + + pos_in += chunk; + pos_out += chunk; + } + cio->status = ret; + kvfree(emulation_io->buf); + kfree(emulation_io); + blkdev_copy_endio(cio); +} + +static inline ssize_t queue_max_hw_bytes(struct request_queue *q) +{ + return min_t(ssize_t, queue_max_hw_sectors(q) << SECTOR_SHIFT, + queue_max_segments(q) << PAGE_SHIFT); +} +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, + * for synchronous operation this should be NULL + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * For synchronous operation returns the length of bytes copied or error + * For asynchronous operation returns -EIOCBQUEUED or error + * + * Description: + * If native copy offload feature is absent, caller can use this function + * to perform copy. + * We store information required to perform the copy along with temporary + * buffer allocation. We async punt copy emulation to a worker. And worker + * performs copy in 2 steps. + * 1. Read data from source to temporary buffer + * 2. Write data to destination from temporary buffer + */ +ssize_t blkdev_copy_emulation(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct blkdev_copy_emulation_io *emulation_io; + struct blkdev_copy_io *cio; + ssize_t ret; + size_t max_hw_bytes = min(queue_max_hw_bytes(in), + queue_max_hw_bytes(out)); + + ret = blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + cio = kzalloc(sizeof(*cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + + cio->waiter = current; + cio->copied = len; + cio->endio = endio; + cio->private = private; + + emulation_io = kzalloc(sizeof(*emulation_io), gfp); + if (!emulation_io) + goto err_free_cio; + emulation_io->cio = cio; + INIT_WORK(&emulation_io->emulation_work, blkdev_copy_emulation_work); + emulation_io->pos_in = pos_in; + emulation_io->pos_out = pos_out; + emulation_io->len = len; + emulation_io->bdev_in = bdev_in; + emulation_io->bdev_out = bdev_out; + emulation_io->gfp = gfp; + + emulation_io->buf = blkdev_copy_alloc_buf(min(max_hw_bytes, len), + &emulation_io->buf_len, gfp); + if (!emulation_io->buf) + goto err_free_emulation_io; + + schedule_work(&emulation_io->emulation_work); + + if (cio->endio) + return -EIOCBQUEUED; + + return blkdev_copy_wait_for_completion_io(cio); + +err_free_emulation_io: + kfree(emulation_io); +err_free_cio: + kfree(cio); + return -ENOMEM; +} +EXPORT_SYMBOL_GPL(blkdev_copy_emulation); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 5405499bcf22..e0a832a1c3a7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1046,6 +1046,10 @@ ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, loff_t pos_out, size_t len, void (*endio)(void *, int, ssize_t), void *private, gfp_t gfp_mask); +ssize_t blkdev_copy_emulation(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Thu Oct 19 11:01:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428788 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 090FF1EB5D for ; Thu, 19 Oct 2023 12:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="dpB5VAyL" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231019120408epoutp0211713a0ef85adbe17681f11e968f5e39~PgKkcPemD2730227302epoutp02l for ; Thu, 19 Oct 2023 12:04:08 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231019120408epoutp0211713a0ef85adbe17681f11e968f5e39~PgKkcPemD2730227302epoutp02l DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717048; bh=WyEZwCEcMZEgd+uPDvb2WQHlAKDzDTA2b4v6jEJOw5k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dpB5VAyLh2w6dyA/2YUmJl/6JWe52S3MiUwSIq8sHPoL9b3r/gxrSIza+bBTzVTeY Fc2pwcxdcVnsSlT2pBXDOLwTlOAxojfv5GUFoP3p88Uf2VY5MiP2mNmg2njJVWh74k yM1iZdxyuwN8zC2YAcY9vI64ThnymqaQUHdFERUI= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120407epcas5p2f1c529c296e63180f5848cbe11a535ea~PgKkB9THD0651406514epcas5p2n; Thu, 19 Oct 2023 12:04:07 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SB5wk18s5z4x9Px; Thu, 19 Oct 2023 12:04:06 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3A.73.08567.63B11356; Thu, 19 Oct 2023 21:04:06 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231019110904epcas5p4028e52ae5c6cefbb73a63f36d7f225e5~PfafYtRX22423124231epcas5p4H; Thu, 19 Oct 2023 11:09:04 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110904epsmtrp2989081c25c3dde0f27403aeba46f8ea3~PfafV-tTY1629616296epsmtrp2V; Thu, 19 Oct 2023 11:09:04 +0000 (GMT) X-AuditID: b6c32a44-3abff70000002177-33-65311b36bb14 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 05.67.07368.05E01356; Thu, 19 Oct 2023 20:09:04 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110900epsmtip19922c5e976f68e719c2e85f492299e40~PfacF3Tkv2963429634epsmtip15; Thu, 19 Oct 2023 11:09:00 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Anuj Gupta , Hannes Reinecke , Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 05/12] fs/read_write: Enable copy_file_range for block device. Date: Thu, 19 Oct 2023 16:31:33 +0530 Message-Id: <20231019110147.31672-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxj2O+dwemBhHAvED8g2UuccGi5F6D4cINmIOXGYsUk2M63YlZNC gLbpBTeBjCq3gRREx0IHchkOAbkMEAG5uKpcp6AEsITLCIVN2SDIxnBEGKVl89/zPN/7Pu/l y0vh3HyOMxUlVbEKqSiGR9oQTXfc9roLXPis14ObBKrt68LR0l9rBDqX8wJHVRPZJJq/8wwg 4+00gIpLCwlkuN2CobbSXAxVVN3DUK5+BKDZYR2G2sf2o5LUMgK1tfcSaKi1gERFP8xyUOZo M4nKu9cx9DhnFqBmowagprUiHNXMLxKoZ8wFDbzotgpyYlp0ExxmYPJHghm6r2bqK78mmYay r5jfGvIBc8uQRDLfay9ZMVnnF0hmaXaMYBY7hklG21gJmIb+eGa5/nWm3vgHFmr3WbR/JCuK YBWurFQsi4iSSgJ4HxwLfz/cV+DFd+f7oXd4rlJRLBvACw4JdT8cFbO5DZ5rnChGvSmFipRK nmegv0KmVrGukTKlKoDHyiNi5D5yD6UoVqmWSjykrOog38vL23cz8HR05IZmzkq+YPvFQtc0 JwkMvpIBrClI+8D+vktWGcCG4tK3ACxPfsIxk2cAap4uW8gKgDOXH4HtlMmCHstDO4BTORuk maRgMPkXzWYURZH0fti/QZl0BzoFh53jvwMTwekrOJzbKN6ysqfD4MA/PYQJE/QeuJq3sYVt 6YNwYsbAMRlB2hNmT+00ydb0u7AusxQ3h+yEvfnGrXCcfgOev/EdbvKHdIM1HP35IWZuNRim LqWTZmwPn3Y3cszYGS4vtFv0M7Di8jXSnJwMoG5UZ5nzEEzpy8ZNTeC0G6xt9TTLr8Fv+mow c+FXYdaa0VLLFjZf2ca74fXaYou/Exz5W2PBDKy+brCsTgtgY2EpJwe46l4aSPfSQLr/SxcD vBI4sXJlrIQV+8r5UvbMf/8slsXWg60b2RfcDB4XrXvoAUYBPYAUznOw3cN4sVzbCNGXZ1mF LFyhjmGVeuC7ufCLuLOjWLZ5ZFJVON/Hz8tHIBD4+B0Q8Hm7bOdTCiO4tESkYqNZVs4qtvMw yto5CSMTKsbShTfj0tr0u20W16kbZxOm/Ts6Z5rCrj483GKYqcQ6j9iVXPs2bmg6t7BBNBT3 Zu0BXQ6VHMiELVfrqneECIJWdgm1R8sKqjXC5+XJ6R7yvB0nuKTqXmKtDhpia8R+dp8Lh2Pf K/JNJ+7H6/3ZqrdWuUJJXUFg3ZPjDs7ejqM2b8cfT76bpdZ2Sz/9STB6KskxoDg16JTxtPUF e88HLtPcjAvkyKBEPP7xVGDe87I/jecy42VXB+3cLs5rJ1HHaqnw6K8nV/TK8Y9SMvaeTJ+b 6WsLKfnkWOp6WvddXxet96HWjZrRE7mLa13lg4N52eSjXvuMhAryw0RqRpHII5SRIv4+XKEU /QtMTKkLrAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLIsWRmVeSWpSXmKPExsWy7bCSnG4An2GqQetKAYv1p44xW3z8+pvF omnCX2aL1Xf72SxeH/7EaPHkQDujxYJFc1ksbh7YyWSxZ9EkJouVq48yWUw6dI3R4unVWUwW e29pWyxsW8JisWfvSRaLy7vmsFnMX/aU3aL7+g42i+XH/zFZ3JjwlNFix5NGRottv+czW6x7 /Z7F4sQtaYvzf4+zOkh67Jx1l93j/L2NLB6Xz5Z6bFrVyeaxeUm9x4vNMxk9dt9sYPNY3DeZ 1aO3+R2bx8ent1g83u+7yubRt2UVo8fm09UenzfJeWx68pYpgD+KyyYlNSezLLVI3y6BK+N/ 4zPWgne8Fe+OPWRvYLzA3cXIySEhYCJxb84J9i5GLg4hgd2MEk/7P7JBJCQllv09wgxhC0us /PccqqiZSeLmumagIg4ONgFtidP/OUDiIgL9zBLv/k5nAnGYBdYwS1y5OhOsSFggSKLxcTTI IBYBVYkf0/6zgNi8AlYSdx/fZAcpkRDQl+i/LwgS5hSwltjQvQhsrxBQyYMFj9khygUlTs58 AtbKLCAv0bx1NvMERoFZSFKzkKQWMDKtYpRMLSjOTc9NNiwwzEst1ytOzC0uzUvXS87P3cQI jnEtjR2M9+b/0zvEyMTBeIhRgoNZSYRX1cMgVYg3JbGyKrUoP76oNCe1+BCjNAeLkjiv4YzZ KUIC6YklqdmpqQWpRTBZJg5OqQYmpSu/VuloSHa4LjSXTuP/u6l21TbeOalH3ymez+blNZ3A fKrvTGGb9Mo7sx1MgvrmzVf5yKqT2MV+v4dxYdb2BNdEqTK5FqPFXs8+VcwofrpebWJF1pfn d//YJQkvfnDzwC67kwHVLJekSyNft1jfKQw/cMbzxwyRSkbB0xU6vNf+Jjce7Erf90L/nvUJ n4cKKt9D+KQeH3gt8/NMd0LGFb7fHlt/tF0UcX4851C2eoOWyY5s64U/2H/9TQg+Zf1O98D6 61t6eQ6z5V9eyNN9Mj+Pb+2SBV9viDW/uS64cmr9siel3gEzNP+sKmZ1nrUr7O3hdQmz/X2C LyQmthYIvDhb8yzX6vXrFyfaZUveKrEUZyQaajEXFScCAHuzOrxgAwAA X-CMS-MailID: 20231019110904epcas5p4028e52ae5c6cefbb73a63f36d7f225e5 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110904epcas5p4028e52ae5c6cefbb73a63f36d7f225e5 References: <20231019110147.31672-1-nj.shetty@samsung.com> From: Anuj Gupta This is a prep patch. Allow copy_file_range to work for block devices. Relaxing generic_copy_file_checks allows us to reuse the existing infra, instead of adding a new user interface for block copy offload. Change generic_copy_file_checks to use ->f_mapping->host for both inode_in and inode_out. Allow block device in generic_file_rw_checks. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- fs/read_write.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/fs/read_write.c b/fs/read_write.c index 4771701c896b..f0f52bf48f57 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -1405,8 +1405,8 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, struct file *file_out, loff_t pos_out, size_t *req_count, unsigned int flags) { - struct inode *inode_in = file_inode(file_in); - struct inode *inode_out = file_inode(file_out); + struct inode *inode_in = file_in->f_mapping->host; + struct inode *inode_out = file_out->f_mapping->host; uint64_t count = *req_count; loff_t size_in; int ret; @@ -1708,7 +1708,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + if (!S_ISREG(inode_in->i_mode) && !S_ISBLK(inode_in->i_mode)) + return -EINVAL; + if ((inode_in->i_mode & S_IFMT) != (inode_out->i_mode & S_IFMT)) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || From patchwork Thu Oct 19 11:01:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428787 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAF951EB51 for ; Thu, 19 Oct 2023 12:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="l9kXyZA0" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231019120412epoutp02d4670d91efe8a22b5e4d54db99f4508a~PgKogBclf2727627276epoutp02l for ; Thu, 19 Oct 2023 12:04:12 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231019120412epoutp02d4670d91efe8a22b5e4d54db99f4508a~PgKogBclf2727627276epoutp02l DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717052; bh=WRb5UebBvbY1xsMYZIZvAmNJpBcvY4g9cTSXPzDv7NY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l9kXyZA01NEQvEoLklbSfUIHi58bUKSEbX0gp+bGFNT+eUfW7xwQmEfYDQd1aaFaT gPJi/vnX/aoyyhxbEt+NsIejN4FzgRAp3Ed0i/lCBDBfqvPWZQByedk6Is8KM2QuGk +K2MrMtCMpMxmNymAOqSSonPYm0rdJKOciCLx2oU= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120411epcas5p22ef9512956a82b118280209dff7c334c~PgKn1cIU20651406514epcas5p2x; Thu, 19 Oct 2023 12:04:11 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SB5wp34jmz4x9Ps; Thu, 19 Oct 2023 12:04:10 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 23.88.10009.A3B11356; Thu, 19 Oct 2023 21:04:10 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20231019110914epcas5p2ce15c70c6f443895bb458dfbdcc50e3b~PfaoryLJd0518805188epcas5p24; Thu, 19 Oct 2023 11:09:14 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231019110914epsmtrp15aa5ab9511ae4a9c05a6e2716dbfecb7~PfaopVnvm2693626936epsmtrp1E; Thu, 19 Oct 2023 11:09:14 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-2d-65311b3a36a3 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id EE.C9.08755.A5E01356; Thu, 19 Oct 2023 20:09:14 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110910epsmtip1f1eb25b3163831303f673e11c1294f9e~PfalPVO2s2856028560epsmtip1M; Thu, 19 Oct 2023 11:09:10 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 06/12] fs, block: copy_file_range for def_blk_ops for direct block device Date: Thu, 19 Oct 2023 16:31:34 +0530 Message-Id: <20231019110147.31672-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Tf1DTZRzH7/l+v3wZHOQXlHiAIlpRhwZsMsYDiRYQfQ/JozjupK5ba3xv EGPb7Ye/OGskg4L4kSLlEuRXJSBMQHT8Ug4OECiX4RAQlGhUxgLFUytC2tgo/3s97+f9fH49 92Hhnl86+7IypCpGIRVK2KQrcb4/6MXgKD8uw8kZ8UL6kUEc3b2/QqCPS1dx1DhTQqKF/mWA zL35AFXVVBBosrcDQ901RzFU3ziAoaN94wDNm3QY6pnahqrz6gjU3TNMoLHOkyQ69c28Myq8 biDRt0OPMDRROg+QwZwD0PmVUzhqXlgi0OUpP2RcHXJ6xYfu0M0408abLQQ99r2abm34lKTb 6j6if2s7AeiuSQ1J1xYfc6KLjiyS9N35KYJeumgi6eJzDYBuG82m77X6063mP7CkTW9n7khn hGmMIoCRimRpGVJxNHt3siBWEM7ncIO5kSiCHSAVZjHR7LjEpOD4DIl1GuyAfUKJ2iolCZVK dujOHQqZWsUEpMuUqmg2I0+TyHnyEKUwS6mWikOkjCqKy+FsD7ca38tM79CXEvKuTQfG+yec NGDIrQC4sCDFg7MtuaAAuLI8qS4ANYYBzH5YBrC1vNFx8wBAY3GT88aTqa9LHK4eq6t+wuHS YrBurcCpALBYJLUNjq6xbPoWSovDS9OWdRNOVeKwbW4Is4XaTAngvbnadSaoQHj56jhuY3cq Cl57aCZtgSAVCktuedhkF+pleLawxmHxgMMnzISNceoZeKT9K9xend4Flk+52zkOrhqPk3be DH8fOufowBfeLslz8H5YX3aatNUGqVwAddd1wH6xC2pHSnBbDTgVBPWdoXb5aXh8pBmz530C Fq2YMbvuDg2VG/wcPKOvcuT1geMPcxxMw/zuWtw+rGIAqyt+dCoFAbrH+tE91o/u/9RVAG8A PoxcmSVmlOHy7VJm/3/fLJJltYL1FdmaYAA/zd4J6QMYC/QByMLZW9wDaQ7j6Z4mPHiIUcgE CrWEUfaBcOu8P8d9vUQy645JVQIuL5LD4/P5vMgwPpft7b6grUjzpMRCFZPJMHJGsfEOY7n4 arDA+wq/l3im1JQ/E7Wu7YpiZWZ+P2HInTi0WzOWGfbPjZUEy8+E5P2K9Gn1a4NxM+bUFBn/ +SZvPPuNZJk68DOT56x3UPThH67yjW5k+YctB/19nuL9agram31peizwwgeFu8QxX9Qslp9+ fbLmTm/UDRD1i9xkaXoQvT9OoP/EjI1yJpdl1c/GJ1fs3JNS5t+8p/dVkQb6BlfclKfGuzat RqyFXbl1MvH2cNSZ2PIn+zhBktQX/l4pU3U3Dx6WLl57dzHmkXhfzlmvY5YDosV3EpKuRFiM xMBFUV/LtPjNTh/GQwR7aFGMUIH2trtJ5zrGv5uojC3S/rWUx3kr74JlpoZNKNOF3K24Qin8 F0tvCoSrBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Rf0yMcRzHfZ/nueee4vR0Ud9kzOGPfrvV1ndqYWP7zkz+YCw/T/f0Y666 3an8mo4sXHQpOneVcgkdI3c5/VD6gVIahlMtMV2JViQUdsV1s/Xfe+/X6/P+58OQwg5qAZOQ tJ9TJElkItqVsjSLFgVGzxVzK/pKBeh222MSjf74Q6HjOXYS3XirodFQ8zeAbA0nASoxFFGo q6GaQPcNuQQqv/GIQLlNVoD6X+sJVNftjy5nXqHQ/bonFHpZU0ij4qv9fJT1popG11omCdSZ 0w9Qle0YQJY/xSS6NfSFQq3dPuiZvYW32htX69/y8bPeOxR+2ZGCTcbTNDZfSceDZh3AtV0q Gpdm5/Hw2YwRGo/2d1P4S/1rGmdXGgE2tx/GY6ZF2GQbJja5RbtGSDlZQiqnCI7c4xpffTuH kte6HbA2d/JUoGWOGrgwkA2F3WUaQg1cGSFbC+DP5jLCCbzhVftD0pk9YPnkR75TyiBgQ/kr Sg0Yhmb9YfsU4+jnsRoSjti108cke5OElUUCR/Zgd0KtTk87MsUuh60vrNOjAnYlfDVuox07 kA2GmnfujtqFDYcVWYZpRfhPeV/Sx3fq7vCJzkY55xfDjLsFZA5g9TOQfgYqAYQReHNyZWJc olIsFydxaUFKSaIyJSkuKCY50QSmP+7nWwXuGb8GNQGCAU0AMqRonmA5XsEJBVLJwUOcInm3 IkXGKZuAD0OJvAReg2elQjZOsp/bx3FyTvGfEozLAhXhNuI1NejzaX4NSzTamJDUcfGP0dKb wQb+0EDMwajaUE+eprBodo1w9oFrq7OnNvd9t2SZQ3SPCqKLZe6B9b5hPbzWM1FHOn1jFqdd Prr3xZ6QwPW9po7Y4g3Lkj/4fmoHlp6e86q1wgo7E65VrGLurnsw4RkwFpL5OW/8QSTlucYD k5Fbon5fP9VaCEXb3KwXWe1zz7S8tfk/F3ZIpem/s0+MqFPFlnMXvubPN3g3cuqA6qW/rBvb XM5N+OsEk70V28K045mz8hsrP8/dWmD3T9hQEGEuO2luM3/bnb5lSay8bGBX0NPJdONA5PPH YadllyRLJ4ZptH2H34l420fVXhGljJeI/UiFUvIXeN3BjGADAAA= X-CMS-MailID: 20231019110914epcas5p2ce15c70c6f443895bb458dfbdcc50e3b X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110914epcas5p2ce15c70c6f443895bb458dfbdcc50e3b References: <20231019110147.31672-1-nj.shetty@samsung.com> For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, or use generic_copy_file_range in case device copy offload capability is absent or the device files are not open with O_DIRECT. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/fops.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/block/fops.c b/block/fops.c index 73e42742543f..662d36a251a3 100644 --- a/block/fops.c +++ b/block/fops.c @@ -735,6 +735,30 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + ssize_t copied = 0; + + if ((in_bdev == out_bdev) && bdev_max_copy_sectors(in_bdev) && + (file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) { + copied = blkdev_copy_offload(in_bdev, pos_in, pos_out, len, + NULL, NULL, GFP_KERNEL); + if (copied < 0) + copied = 0; + } else { + copied = generic_copy_file_range(file_in, pos_in + copied, + file_out, pos_out + copied, + len - copied, flags); + } + + return copied; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -839,6 +863,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) From patchwork Thu Oct 19 11:01:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428724 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06DB51DFDA for ; Thu, 19 Oct 2023 12:04:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="VuWn14vd" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231019120415epoutp0190fcf152dc16cf07a9994a7f81b47170~PgKq6Iqbz2591425914epoutp01C for ; Thu, 19 Oct 2023 12:04:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231019120415epoutp0190fcf152dc16cf07a9994a7f81b47170~PgKq6Iqbz2591425914epoutp01C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717055; bh=9HUxsYJcUqY59rvCQaWh3ErxEvPd5vF6B6n+YdK8rBY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VuWn14vdXddRe36URDVj4i0DNuNMIqSTjC85mT2yajib/A6aKraBeWeT0rhXHQ614 WsS3RJsbFlhzFWhwWIyQXW9CI11nVTqTTSBcN4U5rh13R8Q2cQz3CfXmf5SksSqVAq tNnKOiejoKjOSH/870vdMi8jtlAKIjK7DthkplyI= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120414epcas5p23f41604468125d167e09033622acfc14~PgKqfULOv0920309203epcas5p2W; Thu, 19 Oct 2023 12:04:14 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.176]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4SB5ws2S8Kz4x9Pv; Thu, 19 Oct 2023 12:04:13 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 90.DB.19369.D3B11356; Thu, 19 Oct 2023 21:04:13 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20231019110926epcas5p231dae80a47f8e25149460e538584dd79~PfazjyXC72430724307epcas5p2T; Thu, 19 Oct 2023 11:09:26 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110926epsmtrp2c84cc332f14c1289496b1ed2a81fe73f~Pfazin6Le1629616296epsmtrp2k; Thu, 19 Oct 2023 11:09:26 +0000 (GMT) X-AuditID: b6c32a50-9e1ff70000004ba9-05-65311b3d2012 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id B2.88.08817.56E01356; Thu, 19 Oct 2023 20:09:25 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110920epsmtip1a379a6a69e0f9b25335feb170883121e~Pfau2L9Lp0452004520epsmtip1o; Thu, 19 Oct 2023 11:09:20 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Kanchan Joshi , =?utf-8?q?Javier_Gonz=C3=A1lez?= , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 07/12] nvme: add copy offload support Date: Thu, 19 Oct 2023 16:31:35 +0530 Message-Id: <20231019110147.31672-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTVxzOufdyWzAdV2DZESI2ncwAgbYC9aACSyDmDiXAli0bWYAG7oBR StcHOs0yHoMJyGNlbKPKYwXGeCgvh0VgMuQxIMwgCEJgm6MoWkB5ExFda2Hzv+/7fr/v9zo5 bNymnGXPjpMqGblULOGRVkTLTedDbj4OQkagSeWg+oFeHC2tbREoNX8bR7XTeSQy3FwGSN/5 NUBl2mICTXS2Yqhdq8ZQdW0PhmamVlio58UCidRdYwDN3tFgqGPSFf2YUUGg9o5+Ao1cv0Si 0p9mWSh7XEeiqr7nGLqbPwuQTp8CUMtWKY6uGB4T6PdJB3Rru8/ibQe6VTPNom/92UjQI0Mq uqkmk6SbK76k55qLAN02kUzS5bkFFnRO2iJJL81OEvTjX++QdO7VGkA3D56jV5oc6Sb9AhZi HRZ/PJYRRzNyLiONSoyOk8b48E6+F+Ef4SUSCN2E3ugIjysVJzA+vIBTIW4n4iTGu/C4SWKJ yiiFiBUKHt/3uDxRpWS4sYkKpQ+PkUVLZJ4yd4U4QaGSxrhLGeVRoUBw2MuYGBkfO7/RDWSX Tp2p0q+ykkGVbxawZEPKE9Zc6CaygBXbhmoHMKO8nzSTZQCLVp9iZrIO4PMnPaxdy/ZaC24O dAD4rKHXwkzSMfjUkGO0sNkk5QoHX7BNuh2VjsMbU/PARHAqjYBjdbmkqZQt5Q23Wg2YCROU E9QYOnET5lBH4f3hOcJUCFJ8mPfXXpNsSR2DDdnanZS9sL9IT5gwTh2Aab9cfDkRpP6whJl9 pmYmbwB88pBtntoWPuq7urOBPVxZ7CDN+DSs/vZn0uz9CkDNuAaYA34wfSAPN9XBKWdYf51v lvfDwoErmLnvazBnS4+ZdQ7UleziN2FdfdlO/X1wbCNlB9NwceHezrFzAdS1puD5gKt5ZR/N K/to/m9dBvAaYM/IFAkxTJSXTOgmZU7/985RiQlN4OVvcQnRgdqGbfcugLFBF4BsnGfHcaIF jA0nWvz5WUaeGCFXSRhFF/AyHvwb3P71qETjd5MqI4Se3gJPkUjk6e0hEvLe4BjSi6NtqBix kolnGBkj3/VhbEv7ZEy9vBLJ9fOfnfgh2CrTcdlDPDgd+vFA1ginh3vbdk9A+qawUhUmOVzx 7B3sfI/C1yNI+352QXpd5UdRU1X+757JG6/1Ju45qWrtrCfUkyOVv01d2xPsPKzKCv+ipjAp lN926FoP4A/VvUWNuoUeCfYrKtYqRguQdePlB1792Z95PYworTobZCG537jPstg28m78xQ/2 Z3R9om07uPaAfDSfmpnUET5Rqms8FnYwo3v6+09HxYEnZOdc9K5Dl2+PDaPeuX8uLFVaKzba w9f+DgyaXhGo15zXo+sOnJ8pmXKVidZzqBk7w3eTJSfdVmMdWzfHNiMlFhp1YKEooPrDUu4N fgWPUMSKhS64XCH+F4fjH0C2BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA03Sf0yMcRwHcN/neTz3XLo8XehbTea22KJLZL5IGOM70zRbDJsc97iOq85d KQzRFjK6heWOSH40ZdKldnU5dZQfLT/W0o9F2l3E1RV31aZdx10z/vt83p/X3n99GFJop4IZ eUoap0qRKES0D1X9TBQawflFcYueti9B5a+bSPRjZJxCpzUuEpV9zKOR7dlPgKz1ZwAqKi6k UGd9DYHqivMJdL+skUCWbgcPNboHaZRv/gBQX5uOQE+6FqBbOXcoVPfkFYVaa6/T6Oa9Ph46 326gUcmLCQJ1aPoAMlhPAVQ9fpNED21DFHrZFYLeul5MXROCa3QfefjtpwoKt7akY33pORpX 3jmJ+yu1ABs7s2h8++KlqfhCtp3GP/q6KDxkaqPxxcelAFc2H8MOfSjWWweJ+Ok7fWKknEJ+ mFNFxu7xSRoYew6U1zdnllidvCxQEpsL+Axko6FrpJrMBT6MkDUC+GWwC0weguA913Nycg6A 9ye+8iZRNgFt7iEiFzAMzS6AzW7Gk89g80hodxUQnoVkL1GwoveNtymAXQ7Ha2yEZ6bYMKiz 1XtbBewK+OV9P+UpgmwkzOvx98R8diV8dL7YS4R/yOciC2+S+8NXWquXk+x8WH5D6IlJdg7M rrpGaoC/7j+l+6d0/6kiQJaCIE6pTpYlq6OUi1O4DLFakqxOT5GJ96Um64H3JcLDDaCudFhs BgQDzAAypGiGIAwv4oQCqeTIUU6VmqhKV3BqMwhhKFGgYNR2QSpkZZI07iDHKTnV3yvB8IOz iMdjDf7oan6E7rv4V22caP2OhJOrVvWMptltIxrpbXpcHuge6xjINMRkdhRYcxrOTLSYHG36 NGmZMWyLONZi2pZ4aFrxQTvmF+ZlzPrUPXdZgyEnzri08YPINj9+tEEWg58eNUcEEEvaVpT4 7tdpfQvloU2bD6n6E7Vlm3ZuyDR9O9vb3LnJLTKtPe5nypBcDn436jiVMOwSOALjW7YH7fs1 ciV+Hb+d1mfdrdoY53xQ90g45UBVgrGpvMIJa/1mauNm/6zeM8+5N6mVNKwbrqiUR/N65ygs Mo1zYJkiNDrSdyAqJ4bZFT7c0tSz+2v71u6z274vrJEpZ6WesK62iCh1kiQqnFSpJb8BxYAz iIEDAAA= X-CMS-MailID: 20231019110926epcas5p231dae80a47f8e25149460e538584dd79 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110926epcas5p231dae80a47f8e25149460e538584dd79 References: <20231019110147.31672-1-nj.shetty@samsung.com> Current design only supports single source range. We receive a request with REQ_OP_COPY_SRC. Parse this request which consists of src(1st) and dst(2nd) bios. Form a copy command (TP 4065) trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Reviewed-by: Hannes Reinecke Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 79 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/trace.c | 19 +++++++++ include/linux/blkdev.h | 1 + include/linux/nvme.h | 43 +++++++++++++++++-- 5 files changed, 140 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index 20f46c230885..2f504a2b1fe8 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 21783aa2ee8e..4522c702610b 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -763,6 +763,63 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline blk_status_t nvme_setup_copy_offload(struct nvme_ns *ns, + struct request *req, + struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio; + u64 dst_lba = 0, src_lba, n_lba; + u16 nr_range = 1, control = 0, seg = 1; + + if (blk_rq_nr_phys_segments(req) != BLK_COPY_MAX_SEGMENTS) + return BLK_STS_IOERR; + + /* + * First bio contains information about source and last bio contains + * information about destination. + */ + __rq_for_each_bio(bio, req) { + if (seg == blk_rq_nr_phys_segments(req)) { + dst_lba = nvme_sect_to_lba(ns, bio->bi_iter.bi_sector); + if (n_lba != bio->bi_iter.bi_size >> ns->lba_shift) + return BLK_STS_IOERR; + } else { + src_lba = nvme_sect_to_lba(ns, bio->bi_iter.bi_sector); + n_lba = bio->bi_iter.bi_size >> ns->lba_shift; + } + seg++; + } + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.control = cpu_to_le16(control); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + cmnd->copy.nr_range = 0; + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -1005,6 +1062,11 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); break; + case REQ_OP_COPY_SRC: + ret = nvme_setup_copy_offload(ns, req, cmd); + break; + case REQ_OP_COPY_DST: + return BLK_STS_IOERR; default: WARN_ON_ONCE(1); return BLK_STS_IOERR; @@ -1745,6 +1807,21 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_hw_sectors(q, 0); + return; + } + + blk_queue_max_copy_hw_sectors(q, nvme_lba_to_sect(ns, + le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1944,6 +2021,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -4654,6 +4732,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..82c6aef77c31 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 sdlba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "sdlba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + sdlba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e0a832a1c3a7..ce2009b693c8 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1226,6 +1226,7 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) /* maximum copy offload length, this is set to 128MB based on current testing */ #define BLK_COPY_MAX_BYTES (1 << 27) +#define BLK_COPY_MAX_SEGMENTS 2 static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) { diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 26dd3f859d9d..7744538c4ca4 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -831,6 +835,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -854,7 +859,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -1031,6 +1037,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1792,6 +1828,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Thu Oct 19 11:01:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428744 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDACA1DFDA for ; Thu, 19 Oct 2023 12:04:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="GLoaLhQH" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231019120418epoutp01a4e84d30537d493d60712d7342ba31d8~PgKuPoezK2715027150epoutp01P for ; Thu, 19 Oct 2023 12:04:18 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231019120418epoutp01a4e84d30537d493d60712d7342ba31d8~PgKuPoezK2715027150epoutp01P DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717058; bh=1M3rppJEeb1Hb0d/CfRhd346wXXsHkpIV1s1W43J98s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GLoaLhQHUk4ko0yBKeYIjFoEdHbQkUEIQcMpXKVJ41gYh3Hj4W7lX6wwvJ0jnQ6JY QcrJPN/lWc0aBtXQDSy8Yct6QJQXdkf2s3JTJNTcFKPbdh2UUVNPHBZBLKJl8J3zZD GwfRfHdWzNpdd5cv02g7j+59GiZx/OnKZiktZ3WI= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231019120418epcas5p3e463315f81450788c658628417955a9c~PgKt8ealE2680326803epcas5p31; Thu, 19 Oct 2023 12:04:18 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.177]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SB5wx2nBJz4x9Pv; Thu, 19 Oct 2023 12:04:17 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 04.DB.19369.14B11356; Thu, 19 Oct 2023 21:04:17 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20231019110935epcas5p2f956b8376cdb5ff2f5afe8791cd9d490~Pfa8sWl6O2430724307epcas5p2k; Thu, 19 Oct 2023 11:09:35 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231019110935epsmtrp1f326a512658c3218b7d34cb950579947~Pfa8rJSnD2693626936epsmtrp1N; Thu, 19 Oct 2023 11:09:35 +0000 (GMT) X-AuditID: b6c32a50-c99ff70000004ba9-10-65311b41eeb9 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 05.88.08817.F6E01356; Thu, 19 Oct 2023 20:09:35 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110932epsmtip1586db37cacc1620b684c422cd6806447~Pfa5dYO9l2755627556epsmtip17; Thu, 19 Oct 2023 11:09:32 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 08/12] nvmet: add copy command support for bdev and file ns Date: Thu, 19 Oct 2023 16:31:36 +0530 Message-Id: <20231019110147.31672-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHc+5tL4UNvRQ2DiVz7CLbgEBbge6AVvGReRcXBnFs6tywoTcU gbbrYzA2M9AVByLlHel4iQSBgtWWR0VAg1EGaNiCohLFSIDAAB8oTsJwo7Ru/vf5fvP7ne/5 /U4OB+eWOfE4iXINo5JLkinChdV22T8gaKu3kBEUZ3ORqf8qjp4sLLHQ4fxlHBnv6Qk0c3ke oPFLRwGqrqlgoTuXzmOos6YQQw3GKxgq7BkGaOKmAUNdI4HoZFYtC3V29bHQUEc5garqJpzQ sVtWAp3ufYmh2/kTAFnHMwFqW6rC0ZmZRyz024g3GlzuZUd60ecN95zowdFzLHroupY2N2YT tKX2J3rKUgboC3cyCPpUXhGbPn7kIUE/mRhh0Y+6bxJ0XksjoC0DP9BPzeto8/gcFr12X9Im GSORMiofRh6vkCbKE8TUrt1x2+PCRAJhkDAcfUT5yCUpjJja8Wl00MeJySvboHy+kyRrV6xo iVpN8TdvUim0GsZHplBrxBSjlCYrQ5XBakmKWitPCJYzmgihQLAhbKXwQJLM0mAklKadabU1 R0EGKNqYA5w5kAyFlVazUw5w4XDJTgAv/PGAsIt5AEdzOxziOYDD7cVYDuCstvRc/MLudwFY 057FtgsdBvOyFglbEUEGwoF/ODbfg9Th8OLdWWATOFmJQ8tYL2YLdydj4C8TdWwbs0g/2Pw0 e5VdyQj4eMhA2NP4UH/fzWY7kxvh2WM1uL3EDfaVjbNsjJPvwiOtv+K28yFpcoa5k2PAPtwO mGkxO9gd/tnb4mRnHpzWZzk4FTYU1xP25p8BNNwyOBq2QF2/HrddAif9oamDb7ffgSX9ZzB7 8Bp4fGkcs/uu0Fr5in1hk6masLMXHP4r08E0nNPrHMvOA/ClboGdD3wMrw1keG0gw//R1QBv BDxGqU5JYOLDlMIgOZP63zvHK1LMYPWPBERbgfHscnAPwDigB0AOTnm4+tEChusqlXyfzqgU cSptMqPuAWErGy/AeW/FK1Y+mVwTJwwNF4SKRKLQ8BCRkPJ0ndFVSLlkgkTDJDGMklG96sM4 zrwM7MP96sjI7n2NnMFWub5ufZqxpD496kBnSOD+/ODSwjqmKHKZa9zyuKw4fDJ8urt18xyu EsgkbaKhqB/7eMZSULroIZY08apO7SmHO/vChN/+ndvRtNWHr37b95lsdoz6Joo1uBBDeaSm 1baPnP7yq99n4/2LX1Q8/0RrLXnvwcBtWfPnOUx6wd6WWI4LbhIvuzPlbfB6TNJVl/ujh66g gy9G1+zKmi7r2j1gPuTnOYX5d468eeON9wMWY/nYtmsfnAgJjrj2tarELeKzmHMDG+ZPeuV5 uRWwY+92tU0VcpZPeK2Tpk7fsBxMOby2FveOohJ81z/cs/fZpGeHOKKZ37C9nmKpZRJhAK5S S/4F6Qv4K6wEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNIsWRmVeSWpSXmKPExsWy7bCSnG4+n2GqwfPpmhbrTx1jtvj49TeL RdOEv8wWq+/2s1m8PvyJ0eLJgXZGiwWL5rJY3Dywk8liz6JJTBYrVx9lsph06BqjxdOrs5gs 9t7StljYtoTFYs/ekywWl3fNYbOYv+wpu0X39R1sFsuP/2OyuDHhKaPFjieNjBbbfs9ntlj3 +j2LxYlb0hbn/x5ndZD02DnrLrvH+XsbWTwuny312LSqk81j85J6jxebZzJ67L7ZwOaxuG8y q0dv8zs2j49Pb7F4vN93lc2jb8sqRo/Np6s9Pm+S89j05C1TAH8Ul01Kak5mWWqRvl0CV8bm lavZCta7VyxZ1M7YwDjZuouRg0NCwETi0P6wLkYuDiGB3YwSB47NYO5i5ASKS0os+3sEyhaW WPnvOTtEUTOTxM7Di1lBmtkEtCVO/+cAiYsI9DNLvPs7nQmkgVlgDbPElrm8IDXCAv4SK3tt QMIsAqoSaz93soLYvAJWEh8uz2KDuEFfov++IEiYU8BaYkP3IrC1QkAlDxY8ZocoF5Q4OfMJ C8R0eYnmrbOZJzAKzEKSmoUktYCRaRWjZGpBcW56brFhgVFearlecWJucWleul5yfu4mRnB8 a2ntYNyz6oPeIUYmDsZDjBIczEoivKoeBqlCvCmJlVWpRfnxRaU5qcWHGKU5WJTEeb+97k0R EkhPLEnNTk0tSC2CyTJxcEo1MPF+PZC7Pn3JQhlNr9CpxSwBD5LZ5QNdWr78X5M/Kzlzy/IN ubUTnqoXn0roWMa0qaBaYduj3wqsPw0d2kXijpT4f9Zoztu5uCas8us9EzbtVZ6LvmhVL2iq baza5leuwJMc/uvULNlvD27J3W03mNyTwC5p9WjOnSx/I+Y3b9r//5F8lXRFTyOA9cK29b1G G/KPtu9bpRz6f9Vb1ofXm5WK24/NXxwfEhnG0bYxjWt/9QZXc/24qUwL/kw5ePvoXlnhfyUn 3cXKty74GHO2J+eb6gtOt9KXpd/9veoWz2wRXis4p9km3rrt0i3hBbJLE8M/2L8S4nwdIznB iElSI7eUc2c4x5xz6Wpzu1d1KrEUZyQaajEXFScCAA9/0a9eAwAA X-CMS-MailID: 20231019110935epcas5p2f956b8376cdb5ff2f5afe8791cd9d490 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110935epcas5p2f956b8376cdb5ff2f5afe8791cd9d490 References: <20231019110147.31672-1-nj.shetty@samsung.com> Add support for handling nvme_cmd_copy command on target. For bdev-ns if backing device supports copy offload we call device copy offload (blkdev_copy_offload). In case of absence of device copy offload capability, we use copy emulation (blkdev_copy_emulation) For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. loop target has copy support, which can be used to test copy offload. trace event support for nvme_cmd_copy. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++- drivers/nvme/target/io-cmd-bdev.c | 71 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 50 ++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/trace.c | 19 +++++++++ 5 files changed, 148 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 39cb570f833d..4e1a6ca09937 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index 468833675cc9..e51978514546 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,18 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -449,6 +461,61 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +static void nvmet_bdev_copy_endio(void *private, int status, + ssize_t copied) +{ + struct nvmet_req *rq = (struct nvmet_req *)private; + u16 nvme_status; + + if (copied == rq->copy_len) + rq->cqe->result.u32 = cpu_to_le32(1); + else + rq->cqe->result.u32 = cpu_to_le32(0); + + nvme_status = errno_to_nvme_status(rq, status); + nvmet_req_complete(rq, nvme_status); +} + +/* + * At present we handle only one range entry, since copy offload is aligned with + * copy_file_range, only one entry is passed from block layer. + */ +static void nvmet_bdev_execute_copy(struct nvmet_req *rq) +{ + struct nvme_copy_range range; + struct nvme_command *cmd = rq->cmd; + ssize_t ret; + off_t dst, src; + + u16 status; + + status = nvmet_copy_from_sgl(rq, 0, &range, sizeof(range)); + if (status) + goto err_rq_complete; + + dst = le64_to_cpu(cmd->copy.sdlba) << rq->ns->blksize_shift; + src = le64_to_cpu(range.slba) << rq->ns->blksize_shift; + rq->copy_len = (range.nlb + 1) << rq->ns->blksize_shift; + + if (bdev_max_copy_sectors(rq->ns->bdev)) { + ret = blkdev_copy_offload(rq->ns->bdev, dst, src, rq->copy_len, + nvmet_bdev_copy_endio, + (void *)rq, GFP_KERNEL); + } else { + ret = blkdev_copy_emulation(rq->ns->bdev, dst, + rq->ns->bdev, src, rq->copy_len, + nvmet_bdev_copy_endio, + (void *)rq, GFP_KERNEL); + } + if (ret == -EIOCBQUEUED) + return; + + rq->cqe->result.u32 = cpu_to_le32(0); + status = errno_to_nvme_status(rq, ret); +err_rq_complete: + nvmet_req_complete(rq, status); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -467,6 +534,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..4524cfffa4c6 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,47 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range = req->cmd->copy.nr_range + 1; + u16 status = 0; + int src, id; + ssize_t len, ret; + loff_t pos; + + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + status = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (status) + break; + + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); + pos += ret; + if (ret != len) { + req->cqe->result.u32 = cpu_to_le32(id); + if (ret < 0) + status = errno_to_nvme_status(req, ret); + else + status = errno_to_nvme_status(req, -EIO); + break; + } + } + + nvmet_req_complete(req, status); +} + static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +371,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +423,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 8cfd60f3b564..395f3af28413 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 diff --git a/drivers/nvme/target/trace.c b/drivers/nvme/target/trace.c index bff454d46255..551fdf029381 100644 --- a/drivers/nvme/target/trace.c +++ b/drivers/nvme/target/trace.c @@ -92,6 +92,23 @@ static const char *nvmet_trace_dsm(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvmet_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 sdlba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "sdlba=%llu, nr_range=%u, ctrl=1x%x, dsmgmt=%u, reftag=%u", + sdlba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvmet_trace_common(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -129,6 +146,8 @@ const char *nvmet_trace_parse_nvm_cmd(struct trace_seq *p, return nvmet_trace_read_write(p, cdw10); case nvme_cmd_dsm: return nvmet_trace_dsm(p, cdw10); + case nvme_cmd_copy: + return nvmet_trace_copy(p, cdw10); default: return nvmet_trace_common(p, cdw10); } From patchwork Thu Oct 19 11:01:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428745 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 352D71DFFF for ; Thu, 19 Oct 2023 12:04:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="JDl4vw1L" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231019120422epoutp03ecf3b65440e5f33a46053baf88f5aec7~PgKxWEe0N3013130131epoutp03E for ; Thu, 19 Oct 2023 12:04:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231019120422epoutp03ecf3b65440e5f33a46053baf88f5aec7~PgKxWEe0N3013130131epoutp03E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717062; bh=zohq6qFcyNQPlL9CW1gp72Nbdr99RwoaaLJ8/DxbinM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JDl4vw1LaT2LyinfulQYW6csi/HhxvNaCruB4fMxU2JB9Sp2yqzg4uwWbwmmz3FOl 0Po/tS0qLhKvFyq/RjO7iraHxyGCobJbG4YWXzHmD+6bwNTtb1KGF8zjdKH/TFDL/0 Bs0Q/ifOnlEV5WeKSJfHbEp3Vi2B1Gpbp8DvzixM= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231019120421epcas5p2ca9ad98c31f6d0eb8fdfe0d602c01635~PgKw7Ku2B1623216232epcas5p2p; Thu, 19 Oct 2023 12:04:21 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SB5x04FSzz4x9Py; Thu, 19 Oct 2023 12:04:20 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id FF.73.08567.44B11356; Thu, 19 Oct 2023 21:04:20 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20231019110945epcas5p3760e2ce59477ab804b05248a54f107c9~PfbFu3DLz0645906459epcas5p3z; Thu, 19 Oct 2023 11:09:45 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110945epsmtrp2f0b61f4e94ea6e37abaf3e68083fd493~PfbFt7v-Q1629616296epsmtrp2s; Thu, 19 Oct 2023 11:09:45 +0000 (GMT) X-AuditID: b6c32a44-617fd70000002177-54-65311b44fb8c Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id E9.17.18939.97E01356; Thu, 19 Oct 2023 20:09:45 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110941epsmtip1fbaf6e8874504f8fef365afa997ee3ce~PfbCS2Oja2755327553epsmtip1i; Thu, 19 Oct 2023 11:09:41 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 09/12] dm: Add support for copy offload Date: Thu, 19 Oct 2023 16:31:37 +0530 Message-Id: <20231019110147.31672-10-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxj2O+dwaEkgBwr6AZs2ZyEbMKAdUL+qbIsSPBv9wXQ/trmMdfSs rUBb2yJsiQGHwAbj4pA4CgoTNxTRTkBSLIymchlsQAiTmxZdAMdtMOSyS4Wttej897zP9zzv 7cvLwX1y3QM4SpWe1aqkKTTpQTTfCg4Oiw0UsoJmUyAy9nbhaHnNTqArtmISzd96CNCUJQ+g 6gvnCDRmacHQ5SudGPrKOgzQ9G0DhtrGQ9E3uRcJ1NrWQ6Chm5Ukqvpu2h0VjJhIVNu9iaHR kmmATFMnAWq2V+Ho2vwSgX4cD0QDG91ur+9gWgw2d2Zg4jrBDPWlMQ11X5BM48VMZqaxHDDm sSySqSkqdWMKsxdJZnl6nGCWfrhNMkVNdYBZadjJNEz9jiV4vZe8T8FKZayWz6qS1DKlSh5D xx9OPJAYLRIIw4RitJvmq6SpbAwdK0kIi1OmOOan+celKWkOKkGq09ERr+7TqtP0LF+h1ulj aFYjS9FEacJ10lRdmkoermL1e4QCwSvRDuGHyYrP+xdxTYd/Rv3kIJ4FHvjmAw4HUlFw7jqd Dzw4PpQZwNU+M+YKHgK4NtFOPg2GO3rxJ46r9c/lA66DbwHwjx6eS5ODwa6ODcKpIalQ+NO/ HCfvS+XgsP3uAnAGODWGwb72Zszp5lF7YY29ws2JCSoIzpavk07s6eDryrq3ikXA4nveTprr oL8vuIC7JN6wp3yKcGKc2gWzb1TgzvyQquDC+skF0uWNhRPW3U4NpHhwrrvJ3YUD4MpiG+nC 6fDymUuky3sKQMOIAbgeXoM5vcWPe8CpYGi8GeGin4dlvdcwV10vWGifwly8JzSdf4JfgPXG 6q38/nD4z5Nb7TBwrmBrn0UALpXMkCWAb3hmHMMz4xj+r1wN8Drgz2p0qXI2KVojVLHpT784 SZ3aAB4fREisCYxWbYZbAcYBVgA5OO3rGcQIWB9PmfSTT1mtOlGblsLqrCDase7TeIBfktpx USp9ojBKLIgSiURR4kiRkN7hOZ9zTuZDyaV6NpllNaz2iQ/jcAOysIKjh3oNf7UU3Vjttxjf Gn15PLJNbhTb/N9Y0t1fzz6yXgnfMZW28awV1X7udbYHf8fZznbGHfqNe/DbDz7y2n7Vu6+r JFovZixhHZVW5ZDRi2taHZGYm+x3WrMGuxdm6PXlsMlHyo8Lc2dfmpesUn4Zh5ODAs8M7a/J aF8bivBiEgYP8GL6FZv3cZFsM+TLnz9LrKj95+s4xYuxrZKsg5Hb7r1feYo80T9ruBNvt8ST ysa7e47S4jw8qTivwzw2vqto535bpqTUQ1z7Zujxjc6z2QPbQo4FmR7x5LFrOfHvZp5ozL/0 S32/odCywDdvnz5Sdj7z170rErPk2NudGkVVOk3oFFJhCK7VSf8DGJSVg5kEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrNIsWRmVeSWpSXmKPExsWy7bCSnG4ln2GqwZK7MhbrTx1jtvj49TeL xeq7/WwWrw9/YrR4cqCd0WLBorksFjcP7GSyWLn6KJPFpEPXGC2eXp3FZLH3lrbFwrYlLBZ7 9p5ksbi8aw6bxfxlT9ktuq/vYLNYfvwfk8WNCU8ZLXY8aWS02PZ7PrPFutfvWSxO3JK2OP/3 OKuDuMfOWXfZPc7f28jicflsqcemVZ1sHpuX1Hu82DyT0WP3zQY2j8V9k1k9epvfsXl8fHqL xeP9vqtsHn1bVjF6fN4k57HpyVumAL4oLpuU1JzMstQifbsEroyOc++YC45IVqx5fJG5gfGZ SBcjB4eEgInE2jUyXYxcHEIC2xklVrT0sXUxcgLFJSWW/T3CDGELS6z895wdoqiZSaLvfQcT SDObgLbE6f8cIHERgX5miXd/pzOBOMwCT5kkPv78xgjSLSxgLbH492xWEJtFQFXi5cxvYBt4 geKrph5nhrhCX6L/viBImBMovKF7EdhiIQEriQcLHrNDlAtKnJz5hAXEZhaQl2jeOpt5AqPA LCSpWUhSCxiZVjGKphYU56bnJhcY6hUn5haX5qXrJefnbmIEx69W0A7GZev/6h1iZOJgPMQo wcGsJMKr6mGQKsSbklhZlVqUH19UmpNafIhRmoNFSZxXOaczRUggPbEkNTs1tSC1CCbLxMEp 1cA0IeS2M/dbgwuNn6xZF6wyehPxxWLf7ZdnlE8/fa99OLSzalGIw63n03R52qMlIs6JMiz9 8X3HpYXf0qQ1OF5pXmU8rPGu4Jnnpc27FLM2+oqtfbxsyu+XAoc8fVXuzbVuKOI7bv6Yt+r9 hRs1lv7l6895OL46utpuPY+zysEC3RJRnycTpoR3ykdrvdtrfuNapc7uovAA2ayjbN5tX9ZK 52lyuM63ubH2ZOjnqsMJVWGJvmxXtrd1mx2fq5McMTm6/9MW079yaz7fW/pdYekNTd0Y/dgW 9gkzJqpeXKJpKz/FINPiiLDUVJ6/H09OuXu251NT2u+aqAd79k/ua/54ZpWxYMa0RS4rVxSt DJ01RYmlOCPRUIu5qDgRACRDesFOAwAA X-CMS-MailID: 20231019110945epcas5p3760e2ce59477ab804b05248a54f107c9 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110945epcas5p3760e2ce59477ab804b05248a54f107c9 References: <20231019110147.31672-1-nj.shetty@samsung.com> Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 37 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 +++++++ include/linux/device-mapper.h | 3 +++ 3 files changed, 47 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 37b48f63ae6a..8803c351624c 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1878,6 +1878,38 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !q->limits.max_copy_sectors; +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1960,6 +1992,11 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + q->limits.max_copy_sectors = 0; + q->limits.max_copy_hw_sectors = 0; + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 64a1f306c96c..eca336487d44 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1714,6 +1714,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 69d0435c7ebb..98db52d1c773 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -396,6 +396,9 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* copy offload is supported */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Thu Oct 19 11:01:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428746 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6ACFB1F93E for ; Thu, 19 Oct 2023 12:04:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="cOITCODN" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231019120425epoutp03b75f52f34897ca922cc75bd2bafa644d~PgK0nSENU2846628466epoutp03V for ; Thu, 19 Oct 2023 12:04:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231019120425epoutp03b75f52f34897ca922cc75bd2bafa644d~PgK0nSENU2846628466epoutp03V DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717065; bh=W6Cvqjs8GJhwgBfxrn3JFatd+tYHBlqUZSV4Q7H1XFM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cOITCODNXXuYwgLQ6hyqZYYLH3GJY/7fPZzF/sdr6RHAL+atAmR8035wbBK7PTHfO EXsUuQMhgjAF3ITxKiEVZVxl0OgT/b5Yh/Y06YT90ChXO3+AMzQUIb1Owrmbh/4Fgw G7zSuNMG68blBewu27/NB+JxxA5XIdOp93yC8R1A= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231019120425epcas5p39a9a0961718caa950f5d37351b93c490~PgK0GSnQG1178011780epcas5p3O; Thu, 19 Oct 2023 12:04:25 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.174]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SB5x3443yz4x9Pp; Thu, 19 Oct 2023 12:04:23 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id F7.DB.19369.74B11356; Thu, 19 Oct 2023 21:04:23 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20231019110954epcas5p132f1188382b57c5e6cd071e52100b60f~PfbOf8T-e2531825318epcas5p1C; Thu, 19 Oct 2023 11:09:54 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019110954epsmtrp2dead90b55f80f656007a5524af4403c2~PfbOfE_Un1629616296epsmtrp2u; Thu, 19 Oct 2023 11:09:54 +0000 (GMT) X-AuditID: b6c32a50-c99ff70000004ba9-22-65311b4779c4 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 08.88.08817.28E01356; Thu, 19 Oct 2023 20:09:54 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019110951epsmtip1cdd2d4cc218dfbcf4982a24ae7888de3~PfbLbvW240548505485epsmtip1H; Thu, 19 Oct 2023 11:09:51 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 10/12] dm: Enable copy offload for dm-linear target Date: Thu, 19 Oct 2023 16:31:38 +0530 Message-Id: <20231019110147.31672-11-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbUxTZxTOe+/tpTA7LkXDuzI2UkIIMqDdKHtBkE0N3o39wGjcQia1gbtC gLZry1CYAQRkIF9zGKHyUTrCVwk4vkYZzK6oSBkjzg8GypymZEwQFSawEexais5/z3lynnOe c04OG+dWOvHYyTI1o5RJUvmkC9E37O8XuN9TyAiqSr1Qp/kKjp48XSeQfqacRPPDSwBZjIUA aXW1BJoyGjA0qDuDoVb9ZQydMd0CaPamBkND0wGo4VQjgQaHRgl0faCGRPVNs07o9GQ/iZpH nmHot4pZgPotuQD1rdfjqGP+EYGuTnuiiY0R1nuQNmhmnOiJ378j6Ovj6XRXWxFJdzdm03Pd 1YD+YSqHpL8t+4ZFl+YtkvST2WmCfvTjTZIu62kDdPdYFr3c9QbdZXmIxbrGpUQkMZJERunN yBLkickyaSQ/5qB4r1gUKhAGCsPQu3xvmSSNieTv+yg2MDo51bYIvvcXktR0GxUrUan4wbsj lPJ0NeOdJFepI/mMIjFVEaIIUknSVOkyaZCMUYcLBYK3RbbEoylJxjkjpuhgHau6XQlygJEo Bs5sSIXAqSIdXgxc2FxqEMCKxzOkI1gCcPmXSSdHsAJg842NFxKL9Y8tyRCAs6W3tyQFGLxb +DOrGLDZJBUAx6xsO7+dKsDhxTsLwB7g1D8YzGu5Cuyl3Kn9MH+mhGXHBOUL7zdaN3kOtQuO fV+L2QtBKhiW33Wz0842+sJpu1l7ihscrbZsOsKpN2Fe7/lNR5DSO8P7q4PAYXUfPLVkZTmw O3ww0uPkwDy4vDhEOnAGbK1sIR3ifAA1k5otcRQsMJfjdhM45Q87B4IdtBc8a+7AHI1fhaXr FszBc2B/3XPsA9s7tVv1X4O3VnO3MA2rSoc3MZcqA/DG0NEK4K15aR7NS/No/u+sBXgb4DEK VZqUSRAphIEyJuPFnRPkaV1g8z12xvYD/YWNIBPA2MAEIBvnb+f40gKGy0mUHM9klHKxMj2V UZmAyLbwr3HejgS57b9karEwJEwQEhoaGhL2TqiQ78GZL6hN5FJSiZpJYRgFo3yuw9jOvBzs EHciO0P/qSE7qy7G75iXci36r5gDqmHyUkmtVdvk+2vhiHvv5x2cPZJtH8rCORlGLjvwpIvJ z2fxMTfF5P9Wy0ZmvO4OIaZETbvChZl8M+ve3uiEj+PjelbMTWZxU6f24EbvqvuVBUNVciN7 1NX4yQms9eHc8eoaGefSWT1Hu5r1tLnKZ/ya8eI5a+SXnz3R/3To/J+GcsOBQt7lgC6xV11W t8+Jf3cfznnfkxvh6tI3phtjxRj8pP7THq7SyoYMt3H52gcP/s5vcdu2wlmr6Y4Slew5Eufc fu31IknEEdXAyCtZ55KfxS98pWPMuYdPRjVme3iT9+oLfdsbanZE9LXwCVWSRLgTV6ok/wFO aP8OpwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RfUzMcRzH9/39fvfr123Hr8N8i3VzZiiuwvSdh8rD5rsR1kyeJqe+q6au 3PXkOUJzyZ0Uitx1JVwoVxJOOycP5w7zEMryeEdzKnkeq7hi89/78369Pw/bh6PFNsaPS1Sk EaVCniRlhUz9Nan/xB2DQkjwseOTUfXtGzTq/vqLQVVtGha5r30CyGnJBUhvKGVQi+UihcyG AgqdqrpOoQLrY4BczSUUutIaiMp2VzDIfMXGoIeXjrJIV+nyQnlPGlh04mYvhZ5qXQA1OLcD VP9LR6Oz7i4G3Wodge713BREQHyxpM0L33t+jsEP76Rjk3EPi2srtuH22mKAL7dks7h83wEB zs/pZHG3q5XBXY3NLN5XZwS41r4Jfzb5Y5Ozg1o8eIVwRhxJSswgyqCwNcIES7uFSj0ryDr8 rBBkAwujBt4c5KdAZ99LWg2EnJi/DGDjrg/0APCFlT1Nf/UQeKr3nddAKIeCTXoHqwYcx/KB 0N7HefyhvIaGnT2HKE9B87k0tL93U57uIfw8uLNtr8CjGX4MfF3RBzxaxE+H9gullGcQ5IOg 5oWPx/b+Y9fkGfoXi/lp8KX+jddA3Afaip39V9O8BOacP0JrAV/yHyr5D+kBZQS+JFWVHJ+s CkmdpCCZMpU8WZWuiJfFpiSbQP+3AwIagNn4UWYFFAesAHK0dKhoDA4mYlGcfMNGokyJUaYn EZUVjOAY6XDRN3d+nJiPl6eRdYSkEuU/SnHeftlUZLtk+s93b0ur1iR+jsoaVV5Aolt5q8qY FT7BbLz9KO2OYsPJtK2hwqIM0e4ZY4OQTSeRTiq7/wrPMo97Ejl7ap1T+jUsAZ0ZPWyR/uqe WTsKHQn+HcvIhY5nhnUv3JqluTXRgvrMmJj5Mx8Ri7ZoyYL98dmxd+sekHRhUsaXExmXJr6d 6z3tWFyjpDtvTrgsczDwWXjGb1RMuV17TlcZJLE54I+Pc8aGH+ydLRObPqkXzq+eF+WInLky 5EdgxOn1VdpITWiUw652vmp57bvZFNUi1o3ftXzkWlRzpHPLMk1ORH5h86qDYZzcUF2YlxLb uzr/fe74va7gD12hxU3fsVTKqBLkIQG0UiX/DageFnZcAwAA X-CMS-MailID: 20231019110954epcas5p132f1188382b57c5e6cd071e52100b60f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019110954epcas5p132f1188382b57c5e6cd071e52100b60f References: <20231019110147.31672-1-nj.shetty@samsung.com> Setting copy_offload_supported flag to enable offload. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index f4448d520ee9..1d1ee30bbefb 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Thu Oct 19 11:01:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428747 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EAD021368 for ; Thu, 19 Oct 2023 12:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="dO0aohtD" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231019120429epoutp02845e7da942668d3838c1b09264fd1e7c~PgK4QSdei2730227302epoutp02x for ; Thu, 19 Oct 2023 12:04:29 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231019120429epoutp02845e7da942668d3838c1b09264fd1e7c~PgK4QSdei2730227302epoutp02x DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717069; bh=ddG+iMTBPykY7aem2/d5RFRU6pD1r9HGia7WdJCIOGI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dO0aohtDW8Y+Qo5zwCcjT305ZBTbZgibKLbv4F2+TzGAZV1opHVstGZ6uaD5yvx3n txG4SaChvJGZnGweFXbE+VeBqunzfkUepszUaQr6wYUFi3uQeNkyqpD1V5Ab68rdFK KtBzAf+bgJ7gxVo5w0hNTogGUlaePznMI1D6gzcA= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20231019120428epcas5p4d723ca2957f7203c4d2701934872abe7~PgK3kT7Kh2715127151epcas5p4B; Thu, 19 Oct 2023 12:04:28 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SB5x71dYBz4x9Pp; Thu, 19 Oct 2023 12:04:27 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id BA.88.10009.B4B11356; Thu, 19 Oct 2023 21:04:27 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231019111004epcas5p4ff1f40fbc56715d0b1033fa47adca93f~PfbXjg6vj2423124231epcas5p4n; Thu, 19 Oct 2023 11:10:04 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019111004epsmtrp205ee1fd77aee25a662712da74be81aae~PfbXiRvXZ1629616296epsmtrp2_; Thu, 19 Oct 2023 11:10:04 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-55-65311b4bc577 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D5.D9.08755.C8E01356; Thu, 19 Oct 2023 20:10:04 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019111001epsmtip1a2133c2992c6f76748a0a7ac4ecaac12~PfbUZyLMq0547905479epsmtip1j; Thu, 19 Oct 2023 11:10:01 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 11/12] null: Enable trace capability for null block Date: Thu, 19 Oct 2023 16:31:39 +0530 Message-Id: <20231019110147.31672-12-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TezBcVxyec+/du4vZuh6dHqTJZjMyJYNdzyMobdK6JWaMzJiRPuTW3sFg 7ezdTfpIJyIeU4YVj5QViYpEifeqIhaViqApqSBMaKK0HeTFtJpIqtalzX/f7zu/7/ud73fm iHDrYqG9KF6pYdVKJlFKmhOt15ycXUId5Kzs0ldC1DB4HUdP/lwjUGreCxxdmdaRaPHaMkBz PZkAlVeUEWiypx1DnRX5GKq+0oeh/N5xgObH9BgyTu1DX2dUEqjTOECg0Y5zJLpweV6Isifa SFTV/w+G7uTNA9Q2dwqg1rULOKpffESgG1MOaPhFvyDIjm7XTwvp4Zkmgh69qaWba74kaUPl SfoPQwmgr06mkPTF3AIBnXP6IUk/mZ8i6EddYySd21IDaMPQ5/RK8066ee4BFm55JME/jmUU rFrCKmOSFfHK2ABp6OHoA9Fe3jK5i9wX+UglSiaJDZAePBTu8m584sY2pJJjTKJ2gwpnOE7q 9qa/OlmrYSVxyZwmQMqqFIkqT5UrxyRxWmWsq5LV7JfLZO5eG41HE+LGizOFqkybT243/oyl gEIqC5iJIOUJyxZLhFnAXGRNXQXQWFUl4ItlAIcX6jC++AvAzKbbYFuSerOG5A+MAI4OfLcl Scfgb7oZPAuIRCS1Dw6ti0y8LZWOw+67S8BU4NR5HBpm+zGTlQ0VDIfqFwQmTFCOsHqpbZMX U37wTlU/YTKClBvU/WJlos026MbsCpxvsYIDJXOECePULnj621Lc5A+pBjPYNT4GeO1B2JVh zt/aBi70twh5bA9XHhpJHh+H1YXfkLw2DUD9hH4rZiBMH9RthsEpJ9jQ4cbTr8OiwXqMn/sK zFmbw3heDNvOb+M9sLahfMvfDo6vntrCNFzoTCP4ZeUCWGt4LswDEv1LefQv5dH/P7oc4DXA jlVxSbEs56VyV7LH/3vmmOSkZrD5RZxD2sD9e49dewEmAr0AinCprdiRlrHWYgXz6WesOjla rU1kuV7gtbHvM7j9qzHJG39MqYmWe/rKPL29vT19Pbzl0tfEi+llCmsqltGwCSyrYtXbOkxk Zp+CXUzYHfbFiUqtZvleJPvjunlgg09RcJiF/0i6Y3bl8Dut90PmJgNXima9IgR1T7PcInJu tek/6nzr+w7vjrfhr7qgwZLs2fmApajVqGYnC6/EJaf2xoxz/vmrZ0urcz1aY/bMLNqGZD97 o8VCh0Yc7lruFUe5B7nLLkcZVFSa2PUD55Xh62d27O+tsh+ovdHHHXJ772PjT767JmlZahDh tHtvRb7kCPT7XTcdyvhIyJ113UWRBb1+BSeZyKSzJ6KPHuP+dh+XGEsP933YMzLx9AcPS+16 za2x9YjHzPuKSu7AJU134I6VpQdr1cErFnYTqucBTYZnsZIOq8KqaUG/pK84TEpwcYzcGVdz zL+ZEfCSqwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RfUzMcRzHfX+/3/3uV9vp57L1vY7YTZOa6pbxnacyma+nhj88rzr66UEP t9+VCCs1RelhNJKH6pweDkWXnESpdMiNiXQ3YdwRHdXFmrXEdbP577336/15f/54M6TYSHky sYnJHJ+oiJfRrlRju8xr/smpci5waNgb1T3pJNHwzzEKZRaNk+hqXyGNBtrtAFlacwAqV1+k kKn1DoGa1acIVHP1IYFOtfUAZH1VSqB7Zj9Uka2hUPO9xxTqbrpAo7JKqxDlvdbTqMrwm0C9 RVaA9JajADWOlZGodmCQQo/MUvRs3CAIkeA7pX1C/OztTQp3G1NwvfYEjXWadNyvOwfwXVMG jS8XnBbg/KzvNB62mik8eP8VjQsatADrug7hkXovXG/5Rmx02+G6NIqLj93P8QHLI11jekpy hMoc9wMvb7wgMkAxmwtcGMgugJlGLZ0LXBkxexdAU3cdcAIJrBzvIJ3aHdb8/ix0hrIIWFhV LcgFDEOzfrBrgnH409lCEn4fP0s4Dkj2GgkbLooc2p1dDbtqvwocmmK9YY1NP5kRsUtgb5WB cvRANgAWvpvmsF3+2jfy1JN/xexi+L78o9AZnwYfn7NQzvpZMOvWebIIsKX/odL/UDkgtEDC KVUJ0QkquVKeyKX6qxQJqpTEaP89SQn1YHJx33l6cFs75N8GCAa0AciQsukibxzIiUVRioNp HJ8UwafEc6o2IGUomYfIoz8/SsxGK5K5fRyn5Ph/lGBcPDOIyKfSOT+mjM7VBYX7ffDZMHaY +NT/pNUgjkk6ougw2m1xRdbUxl+pZjJiS1C7ZvnUm2eGBlttoQ0ZApeC4Ecn364M/rRrmVWt J6njYctA6MFjS6R5w2np4Zd2tGxND7YtXJOPRkpyn7/v2N3SbbrFbw/S29e+6Y9Ytd4tbuFO N5s5syYzHy+SdIY32EJg2GHLSFdlsdf1soFq+Q+te/UuLls5oSHDpLPnezTFDLZ8GP1M2gWx Rl2oT23kXsGKuuymlg3NFRKwJ2qxKW+imI2vmnHFN0Szzn8l7xm3LUCkDp73xbQpbeZmUS9p 7xml/favM1DPA5P73qxV93Q+WMPLKFWMQu5L8irFH+TAt8BgAwAA X-CMS-MailID: 20231019111004epcas5p4ff1f40fbc56715d0b1033fa47adca93f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019111004epcas5p4ff1f40fbc56715d0b1033fa47adca93f References: <20231019110147.31672-1-nj.shetty@samsung.com> This is a prep patch to enable copy trace capability. At present only zoned null_block is using trace, so we decoupled trace and zoned dependency to make it usable in null_blk driver also. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/block/null_blk/Makefile | 2 -- drivers/block/null_blk/main.c | 3 +++ drivers/block/null_blk/trace.h | 2 ++ drivers/block/null_blk/zoned.c | 1 - 4 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/block/null_blk/Makefile b/drivers/block/null_blk/Makefile index 84c36e512ab8..672adcf0ad24 100644 --- a/drivers/block/null_blk/Makefile +++ b/drivers/block/null_blk/Makefile @@ -5,7 +5,5 @@ ccflags-y += -I$(src) obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk.o null_blk-objs := main.o -ifeq ($(CONFIG_BLK_DEV_ZONED), y) null_blk-$(CONFIG_TRACING) += trace.o -endif null_blk-$(CONFIG_BLK_DEV_ZONED) += zoned.o diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 968090935eb2..c56bef0edc5e 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -11,6 +11,9 @@ #include #include "null_blk.h" +#define CREATE_TRACE_POINTS +#include "trace.h" + #undef pr_fmt #define pr_fmt(fmt) "null_blk: " fmt diff --git a/drivers/block/null_blk/trace.h b/drivers/block/null_blk/trace.h index 6b2b370e786f..91446c34eac2 100644 --- a/drivers/block/null_blk/trace.h +++ b/drivers/block/null_blk/trace.h @@ -30,6 +30,7 @@ static inline void __assign_disk_name(char *name, struct gendisk *disk) } #endif +#ifdef CONFIG_BLK_DEV_ZONED TRACE_EVENT(nullb_zone_op, TP_PROTO(struct nullb_cmd *cmd, unsigned int zone_no, unsigned int zone_cond), @@ -67,6 +68,7 @@ TRACE_EVENT(nullb_report_zones, TP_printk("%s nr_zones=%u", __print_disk_name(__entry->disk), __entry->nr_zones) ); +#endif /* CONFIG_BLK_DEV_ZONED */ #endif /* _TRACE_NULLB_H */ diff --git a/drivers/block/null_blk/zoned.c b/drivers/block/null_blk/zoned.c index 55c5b48bc276..9694461a31a4 100644 --- a/drivers/block/null_blk/zoned.c +++ b/drivers/block/null_blk/zoned.c @@ -3,7 +3,6 @@ #include #include "null_blk.h" -#define CREATE_TRACE_POINTS #include "trace.h" #undef pr_fmt From patchwork Thu Oct 19 11:01:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13428748 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 237ED225B8 for ; Thu, 19 Oct 2023 12:04:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="bi8CLoPC" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20231019120432epoutp0425e7cac704ef0bbd8d6ce7d2839dac83~PgK7PhpSv1483514835epoutp04K for ; Thu, 19 Oct 2023 12:04:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20231019120432epoutp0425e7cac704ef0bbd8d6ce7d2839dac83~PgK7PhpSv1483514835epoutp04K DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1697717072; bh=OJzVFQCb3WofjpZljeXnGBqYi+B32Cvdl/hefyKURJs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bi8CLoPCBUmwkFvI/JVM4HyjYompNST0RPr4v+pEa4iOAJpzqflbnS3PLdfmS00jH pRxadRjgOzlKgeRCJ//8cDRHLV0oPmW5Fh4d27nBbbDljV6Mec9a/8QQMCfC/DaJLJ 3evzsc1yY3IVK/bJE1geQygBVy7z3E0cnA7IEKPE= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231019120432epcas5p358fff75117ff942f9f13e08d787f00c3~PgK6pE52f1178011780epcas5p3f; Thu, 19 Oct 2023 12:04:32 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SB5xC01K8z4x9Pr; Thu, 19 Oct 2023 12:04:31 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 55.42.09672.E4B11356; Thu, 19 Oct 2023 21:04:30 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231019111014epcas5p461d1c994f489b9c71a6baf18922146b5~PfbhDQAmH2188221882epcas5p45; Thu, 19 Oct 2023 11:10:14 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231019111014epsmtrp206f0641b0b89b0ccb652fcd40d91ffb5~PfbhA7aMl1629616296epsmtrp2T; Thu, 19 Oct 2023 11:10:14 +0000 (GMT) X-AuditID: b6c32a4b-60bfd700000025c8-e8-65311b4e2145 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 11.77.07368.69E01356; Thu, 19 Oct 2023 20:10:14 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20231019111011epsmtip107621e0d42a07c717dbe65573354f61f~Pfbdemp6B2869028690epsmtip1R; Thu, 19 Oct 2023 11:10:10 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Damien Le Moal , Anuj Gupta , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v17 12/12] null_blk: add support for copy offload Date: Thu, 19 Oct 2023 16:31:40 +0530 Message-Id: <20231019110147.31672-13-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231019110147.31672-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbVBUZRSe9967l4WJvHxNL8uEzKqZEOyusPhiUKDkXNMJZsqZ6M+6sNcF gWXbjwDLAmFDIAElKheVDwkUkoXlI1D5GBAWsYVZyQWZELNdP9qCwDJHINp1sfz3nOec5zzv Oe8cNu7Z4MJhp8hUjEImTuOSbkTn4JbNwe/4CRh+YV8o0o0O42jhryUCHSlbwVHTTCmJbIOL AFn6CwBaMo7jqLr2NIFu9ndj6HLtCQydbxrC0IkBM0DWG1oM9UwHoZrP6wh0uecqgSYuniJR Vb3VBRVPdpGowfAPhqbKrAB1WXIB6lyqwlGzbZ5AI9N+6JfiowCNrxhY0X50t3bGhR6/1UrQ E0Y1rW8sJOm2us/o+20nAX3pZg5Jny0pZ9HH8uZIulszy6IXrNMEPd97g6RL2hsB3XbtY/qh 3p/WW37H4qkPUiOTGbGEUQQwsqQMSYpMGsXd865op0gYzhcECyLQNm6ATJzORHFj98YH70pJ sy+HG/CROE1tp+LFSiWX90akIkOtYgKSM5SqKC4jl6TJw+QhSnG6Ui2ThsgY1XYBn79VaC/c n5pc8cUIITftyrrbYmLlgObtRcCVDakwuDTVRRYBN7YndQnAtvvzLGewCGCNuQFzBo8ALG4v xZ9JVsqHCGeiB8ALvUOYI+FJaTDY9OPGIsBmk1QQvLbKdtR4Uxoc9v30G3AEOPUYh2cNBtIh 8KJi4JQ5z8WBCWoTbDp+9yl2p16HhZYrpKMRpHiwdNbDQbva6ZbiWtxZ4gGvnrQQDoxT62Fe RyXu6A+pMVdoXjiHOV8aC2tyO1hO7AV/NbS7ODEHPpzrIZ04E57/8hzpFOcDqJ3UAmfiTagZ dYzMtjtsgbqLPCf9MqwYbcacxi/CY0uWNS932HXmGd4Av9NVr/X3hea/c9cwDU2mxbWVlgCo X60ny0CA9rmBtM8NpP3fuhrgjcCXkSvTpYxSKA+VMZn/fXNSRroePL2YwD1d4M7tP0IGAMYG AwCyca63+yaaz3i6S8TZhxhFhkihTmOUA0BoX/hxnOOTlGE/OZlKJAiL4IeFh4eHRYSGC7gv uds0pyWelFSsYlIZRs4onukwtisnB+t4oe2eRzOsM7klzM3mShNmDSmdB4J5DzQ+VdetXnHq 3qwn/tm5K8O7XaMLloPlWW+59WGc+kPKVV39EXmm0WNkK9xdWSeKPfjDQtKrIaOFbx8c09Ua W0FLyUxUXfnt/IqIQlHTt7q9R7N38OAGo+7Kvo3VWduGFyYkhxNjIj+13fqq/5uvo5fNOxPG 7vF6Dj9ZFyka7eeRRX/qP4xkkZs5gdP++z9pQ9b56xfivs9dlyxcHzC23Fpw5v2QoJZHjw8A AeP7ikGaH4UWw8fjbYuDeOVrO4zcTJ+8GJ/35r2iY7xL7tgE2sLOlLjE3vGpfRXZiYOTp4qF D7KY7p8jTEEKLqFMFgsCcYVS/C/00qlQugQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrLIsWRmVeSWpSXmKPExsWy7bCSnO40PsNUg637tSzWnzrGbPHx628W i6YJf5ktVt/tZ7N4ffgTo8WTA+2MFr/Pnme2WLBoLovFzQM7mSz2LJrEZLFy9VEmi0mHrjFa PL06i8li7y1ti4VtS1gs9uw9yWJxedccNov5y56yW3Rf38Fmsfz4PyaLGxOeMlrseNLIaLHt 93xmi3Wv37NYnLglbfG4u4PR4vzf46wO0h47Z91l9zh/byOLx+WzpR6bVnWyeWxeUu/xYvNM Ro/dNxvYPBb3TWb16G1+x+axs/U+q8fHp7dYPN7vu8rm0bdlFaPH5tPVHp83yXlsevKWKUAg issmJTUnsyy1SN8ugStjas8JloKLbhXPNlxkbWBcZ9XFyMkhIWAi8XfyUZYuRi4OIYHdjBIf 7l1hgkhISiz7e4QZwhaWWPnvOTtEUTOTxKmGdqAiDg42AW2J0/85QOIiAv3MEu/+TmcCcZgF OlgkLmzbyw7SLSzgKHHjWjOYzSKgKrF64jMwm1fAWqLzyRE2kEESAvoS/fcFQcKcQOEN3YvA FgsJWEk8WPAYqlxQ4uTMJywgNrOAvETz1tnMExgFZiFJzUKSWsDItIpRMrWgODc9N9mwwDAv tVyvODG3uDQvXS85P3cTIzgFaGnsYLw3/5/eIUYmDsZDjBIczEoivKoeBqlCvCmJlVWpRfnx RaU5qcWHGKU5WJTEeQ1nzE4REkhPLEnNTk0tSC2CyTJxcEo1MHEfmLBVP128dcb8Nd5Z4sw7 b/w4cLp9mn79k+1yRrNXle/1+Jqt/7Hw013ddxU8Sb80TN/WTV36eaK+//ant6UFcybc2GGk GfvrbzPvIpkle+eEvjhxqky+PbfOrPC3XJ3x/V+y10/+nz4zyPqp7VSTnb+1197YvKmshf20 1dqtCUeKw79daFNeu2db7qLKd6fmSh7JrRI7PjsoxeO3/JeuFa9LmkJ5DLXU8wWuLnFZUXPN xuTT1FaekqPBZ0V0rI2Vn77KX7lG48tdjxaelDNMc1nuiTQfXnm47YXd9s2bZj1/H3Bgy5N5 1fd2q749t6P6uN3y01dydksIrfuofeL9G84Pi9Xe5zTp+Hfv4IgIVmIpzkg01GIuKk4EAPOw tuxwAwAA X-CMS-MailID: 20231019111014epcas5p461d1c994f489b9c71a6baf18922146b5 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231019111014epcas5p461d1c994f489b9c71a6baf18922146b5 References: <20231019110147.31672-1-nj.shetty@samsung.com> Implementation is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Only request based queue mode will support for copy offload. Added tracefs support to copy IO tracing. Reviewed-by: Hannes Reinecke Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- Documentation/block/null_blk.rst | 5 ++ drivers/block/null_blk/main.c | 97 ++++++++++++++++++++++++++++++- drivers/block/null_blk/null_blk.h | 1 + drivers/block/null_blk/trace.h | 23 ++++++++ 4 files changed, 123 insertions(+), 3 deletions(-) diff --git a/Documentation/block/null_blk.rst b/Documentation/block/null_blk.rst index 4dd78f24d10a..6153e02fcf13 100644 --- a/Documentation/block/null_blk.rst +++ b/Documentation/block/null_blk.rst @@ -149,3 +149,8 @@ zone_size=[MB]: Default: 256 zone_nr_conv=[nr_conv]: Default: 0 The number of conventional zones to create when block device is zoned. If zone_nr_conv >= nr_zones, it will be reduced to nr_zones - 1. + +copy_max_bytes=[size in bytes]: Default: COPY_MAX_BYTES + A module and configfs parameter which can be used to set hardware/driver + supported maximum copy offload limit. + COPY_MAX_BYTES(=128MB at present) is defined in fs.h diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index c56bef0edc5e..22361f4d5f71 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -160,6 +160,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static unsigned long g_copy_max_bytes = BLK_COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, ulong, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -412,6 +416,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -553,6 +558,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -659,7 +665,8 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," - "zone_nr_conv,zone_offline,zone_readonly,zone_size\n"); + "zone_nr_conv,zone_offline,zone_readonly,zone_size," + "copy_max_bytes\n"); } CONFIGFS_ATTR_RO(memb_group_, features); @@ -725,6 +732,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1274,6 +1282,81 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline int nullb_setup_copy(struct nullb *nullb, struct request *req, + bool is_fua) +{ + sector_t sector_in = 0, sector_out = 0; + loff_t offset_in, offset_out; + void *in, *out; + ssize_t chunk, rem = 0; + struct bio *bio; + struct nullb_page *t_page_in, *t_page_out; + u16 seg = 1; + int status = -EIO; + + if (blk_rq_nr_phys_segments(req) != BLK_COPY_MAX_SEGMENTS) + return status; + + /* + * First bio contains information about source and last bio contains + * information about destination. + */ + __rq_for_each_bio(bio, req) { + if (seg == blk_rq_nr_phys_segments(req)) { + sector_out = bio->bi_iter.bi_sector; + if (rem != bio->bi_iter.bi_size) + return status; + } else { + sector_in = bio->bi_iter.bi_sector; + rem = bio->bi_iter.bi_size; + } + seg++; + } + + trace_nullb_copy_op(req, sector_out << SECTOR_SHIFT, + sector_in << SECTOR_SHIFT, rem); + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + chunk = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || + is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, chunk); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= chunk; + sector_in += chunk >> SECTOR_SHIFT; + sector_out += chunk >> SECTOR_SHIFT; + } + + status = 0; +err: + spin_unlock_irq(&nullb->lock); + return status; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1283,13 +1366,16 @@ static int null_handle_rq(struct nullb_cmd *cmd) sector_t sector = blk_rq_pos(rq); struct req_iterator iter; struct bio_vec bvec; + bool fua = rq->cmd_flags & REQ_FUA; + + if (op_is_copy(req_op(rq))) + return nullb_setup_copy(nullb, rq, fua); spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(req_op(rq)), sector, - rq->cmd_flags & REQ_FUA); + op_is_write(req_op(rq)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -2053,6 +2139,9 @@ static int null_validate_conf(struct nullb_device *dev) return -EINVAL; } + if (dev->queue_mode == NULL_Q_BIO) + dev->copy_max_bytes = 0; + return 0; } @@ -2172,6 +2261,8 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_hw_sectors(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index 929f659dd255..e82e53a2e2df 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -107,6 +107,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */ diff --git a/drivers/block/null_blk/trace.h b/drivers/block/null_blk/trace.h index 91446c34eac2..2f2c1d1c2b48 100644 --- a/drivers/block/null_blk/trace.h +++ b/drivers/block/null_blk/trace.h @@ -70,6 +70,29 @@ TRACE_EVENT(nullb_report_zones, ); #endif /* CONFIG_BLK_DEV_ZONED */ +TRACE_EVENT(nullb_copy_op, + TP_PROTO(struct request *req, + sector_t dst, sector_t src, size_t len), + TP_ARGS(req, dst, src, len), + TP_STRUCT__entry( + __array(char, disk, DISK_NAME_LEN) + __field(enum req_op, op) + __field(sector_t, dst) + __field(sector_t, src) + __field(size_t, len) + ), + TP_fast_assign( + __entry->op = req_op(req); + __assign_disk_name(__entry->disk, req->q->disk); + __entry->dst = dst; + __entry->src = src; + __entry->len = len; + ), + TP_printk("%s req=%-15s: dst=%llu, src=%llu, len=%lu", + __print_disk_name(__entry->disk), + blk_op_str(__entry->op), + __entry->dst, __entry->src, __entry->len) +); #endif /* _TRACE_NULLB_H */ #undef TRACE_INCLUDE_PATH