From patchwork Fri Dec 22 06:12:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503054 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79B95168DC for ; Fri, 22 Dec 2023 07:31:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="q83dtxB9" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231222073151epoutp02cc220c9563509ea1b90b5f0183c87a11~jFvGvVccp1674516745epoutp02G for ; Fri, 22 Dec 2023 07:31:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231222073151epoutp02cc220c9563509ea1b90b5f0183c87a11~jFvGvVccp1674516745epoutp02G DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230311; bh=4LYQ4Z8s8Ktq3w5D+hMl2cEBoY0WGnDBr6jEXEQrqz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q83dtxB9YIyXJRgt801wVaoW+Ss00prinXLjkI7Z1xRirEB/dzzhuVTHiHRw+/qyr UuyrOoeiUG5Yu0bx+T6K4tKrxtK12Tm26ED8zhaEVJ3hOy5S4pXYDdy2kYz/d1L2C2 Mqi6z8Nv1c1fh0brAzcT2PBb5gc6x2wIZ53xHOqU= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231222073150epcas5p3ad1077464d13a44c29f119619c5633ab~jFvGKb1H-2805628056epcas5p3Z; Fri, 22 Dec 2023 07:31:50 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.179]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SxJs03BD7z4x9Q2; Fri, 22 Dec 2023 07:31:48 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id C5.34.08567.46B35856; Fri, 22 Dec 2023 16:31:48 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20231222062055epcas5p2f8d3738628d50a52c1b7da80ca6d68fe~jExLWFClN0448404484epcas5p2V; Fri, 22 Dec 2023 06:20:55 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231222062055epsmtrp29f9c2de2937929e82acecb1a65de64b8~jExLVECoj1637116371epsmtrp2W; Fri, 22 Dec 2023 06:20:55 +0000 (GMT) X-AuditID: b6c32a44-617fd70000002177-43-65853b64a85c Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id A9.B8.08817.7CA25856; Fri, 22 Dec 2023 15:20:55 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062051epsmtip26ae1566d805a06d00a45a4a05eea4f16~jExHx61j_0362603626epsmtip2F; Fri, 22 Dec 2023 06:20:51 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Kanchan Joshi , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 01/12] block: Introduce queue limits and sysfs for copy-offload support Date: Fri, 22 Dec 2023 11:42:55 +0530 Message-Id: <20231222061313.12260-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta1BUZRjuO+dwWFDkgFifq9l2NAy57eqyfIuQTDHOGdNkqH7IDMHGngFa 2N12F9HUxLgoFBcxDRaQ6xQXE4GFWJRkQe6hY4TAjtDEQCo3AxRUBoh1ofz3vM/7vPd5Obh9 viWXEyHXsCq5JJImrYnaZicnV+neBJY/YbRDFZ2tOJp+ukCgb9IXcVQ+mEai8eYZgEYazwKU X5hLoIFGPYZuFGZgqLS8BUMty5Mkymi6B9BorxZDDUZnVJBYTKAbDR0E6qnPIVHej6OW6Nu+ OhL91LaEof70UYDOn+vFUN3IGYBqF/JwdHX8MYHajVvQncU2C98tjF47aMncGaokmJ7uaKaq LIlkqotPMw+rswBzfSCWZIpSL1gwKXFTJDM9aiSYx7/2kkyqrgww1V0nmNmqbUzVyCTmbxso 8w5nJVJWxWPloQpphDzMh/7w4+APgj1EfIGrQIw8aZ5cEsX60H4H/V33R0Su7IXmHZVERq9Q /hK1mnZ/z1uliNawvHCFWuNDs0pppFKodFNLotTR8jA3OavxEvD5uz1WhCGy8MyuHEJ51ufY fOJfeCzQ7UkGVhxICeGkIZtIBtYce+o6gCUTD1aNGQAHhzIszMYcgBmFP4C1kO96y4HZ0QBg 48w/hMlhTyVgMG9JnAw4HJJyhl3LHBPtQF3Bob5SYNLjVDMOWwqGSZNmIxUEK2f8TRqCegem /16Cm7AN5QVvjk1ZmCSQcodpf9qZaCtqL3zwosjCLLGDHVkjL6vi1FswriYbN6WH1E0r+Dzt LmHu0w9O6ppJM94Ix9p0lmbMhbNTDat8DCz9voQ0B8cDqO3Trg65DyZ0puGmJnDKCVbUu5vp N+HFzquYufAGmLIwgpl5G1h3eQ1vh1cq8lfzb4b35s+sYgaW6+pJ895SAUzoKQPpgKd9ZSDt KwNp/y+dD/AysJlVqqPC2FAPpUDOxvx35FBFVBV4+Sq7/OpAf96SWxPAOKAJQA5OO9goXOJZ exup5PhXrEoRrIqOZNVNwGNl4edx7qZQxcqvyTXBAqGYLxSJRELxHpGAfsNmPCFXak+FSTSs jGWVrGotDuNYcWMxGTNhta+7tl3fXVp7+Y9uT7cDdPsmZ33YfccuOss6Zf7AjhOW5PJnPxt2 x/Nk9wvy113sONymcOkJfvvgl7cvSJ/ZHknV97k8Mw7bbtj63KFwIOAXrlXh8tDdS7cCIpLL jU/qbvVcC/m8Yj6I61H1qafB1unk0WMNSXGGr0+3Fom3OS3+lqk7F5+dWDKdceRkZvdHMGj4 E77vwim/dVT9zpy6/ACXL2oCjcUtrwc9Gl6KS5K1qLwf0YEv0mU+c02n3j2U03pbv963y9Fm rH/cvuDwom5rliG5MkIutnut+dCOfnSJa5gTzr5f/Pd+u8kn29c7ikLGH4YcH6/xiuG47HR7 mnttlCbU4RLBLlyllvwL8ni+HrMEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG+75zPDuOVsdl9KmQMLCL1cpV8HVRArNOWBQhYVdberyUm2tT u0GtVpp2W0oXN8XUstrIaCtZTrtMbZpZoVlqWWpbVtPsHrXUmhL03/M+v+d9eOGlCWEr6U8n yVM5pVyaLKL4ZHm1KHCGPfgQN+tSXyC+ev8egT99c5P4gHaAwMaOExR2VX8G2HEnE+BzxQUk brtzE+LK4hyILxtrIa4d6qNwju0pwM4WHcRV7dNwUcZ5EldW1ZO4uSKfwoWlTh4+8sxC4Yv2 QYhbtU6ATx5ugdji2A9wubuQwGWufhLXtQfgRwN2r0UB7E1dB4999PIayTY3prEmQxbFms/v Y9+a8wBrbVNTbMnxXC/2mOYDxX5ytpNs/60Wij1+3QBYc8Me9otpImty9MFVY9fxF8ZxyUnp nHJm2GZ+4tmGfFKRGbrzR0YXoQbXZ2cDbxoxc9DRFiPIBnxayFgBeljXSY4AP1Q6UEOM6HHo 8mAPbySkgehI89u/gKYpZhpqGKI9vi9jIdDPKxroGQimmUCni2qAZ3scsx6VH6wc1iQThLRN l4ZbBcx8dPv9By9PEWJmohOvfDy2N7MA9fwq8fJo4d9IfY+ZGon7oPo8x/BxBBOINDf0hBYw uv+Q7j90DkAD8OMUKlmCTBWikMi5HWKVVKZKkyeIY1NkJjD8/OBgC6g0fBTbAKSBDSCaEPkK UqYf5ISCOOmu3ZwyJUaZlsypbCCAJkUTBN9dx+KETII0ldvGcQpO+Y9C2ttfDbO2MY6wxAfX wiVb8iPlZFr479SKx2ujWyPfIPe65Zb0tmXjRz+OyD1lNT1VN3g33W+cWxoLfXl4w+vdblfm F+arVmqQXdhQ2+8XNbjVMrTPT98ZxCwvnvJOJwqr4fFxfM1UxSb1Gom50/58lC1aG1r4a/Xo vc/HbwwtnzyvQr+/JBIc3jNYJJkQWFDo6qpyqeAL+K4uoLfRmeVvja7v3qLJb7XHX61NgtkV SNAdtSN+U5DbkGOtWjzbMIbPH2v4vaLjyZOVYssktqysff7Sh73upqb0iO3GhXfzlsxJTp8R ExWZZOyOjeiS6Kdn2KJEMeI6xawzvRvD9bklbp9GEalKlIYEE0qV9A8ulR+iawMAAA== X-CMS-MailID: 20231222062055epcas5p2f8d3738628d50a52c1b7da80ca6d68fe X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062055epcas5p2f8d3738628d50a52c1b7da80ca6d68fe References: <20231222061313.12260-1-nj.shetty@samsung.com> Add device limits as sysfs entries, - copy_max_bytes (RW) - copy_max_hw_bytes (RO) Above limits help to split the copy payload in block layer. copy_max_bytes: maximum total length of copy in single payload. copy_max_hw_bytes: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Reviewed-by: Luis Chamberlain Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 23 ++++++++++++++++++ block/blk-settings.c | 24 +++++++++++++++++++ block/blk-sysfs.c | 36 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 13 ++++++++++ 4 files changed, 96 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index 1fe9a553c37b..96ba701e57da 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,29 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_max_bytes +Date: August 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes that the block layer + will allow for a copy request. This is always smaller or + equal to the maximum size allowed by the hardware, indicated by + 'copy_max_hw_bytes'. An attempt to set a value higher than + 'copy_max_hw_bytes' will truncate this to 'copy_max_hw_bytes'. + Writing '0' to this file will disable offloading copies for this + device, instead copy is done via emulation. + + +What: /sys/block//queue/copy_max_hw_bytes +Date: August 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes that the hardware + will allow for single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 0046b447268f..4441711ac364 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_hw_sectors = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_hw_sectors = UINT_MAX; + lim->max_copy_sectors = UINT_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/* + * blk_queue_max_copy_hw_sectors - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + */ +void blk_queue_max_copy_hw_sectors(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (BLK_COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = BLK_COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_hw_sectors = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_hw_sectors); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_hw_sectors = min(t->max_copy_hw_sectors, + b->max_copy_hw_sectors); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 0b2d04766324..7193a1f015f9 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -199,6 +199,37 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_hw_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_hw_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, const char *page, + size_t count) +{ + unsigned long max_copy; + ssize_t ret = queue_var_store(&max_copy, page, count); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + q->limits.max_copy_sectors = min_t(unsigned int, max_copy, + q->limits.max_copy_hw_sectors); + + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -517,6 +548,9 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RO_ENTRY(queue_copy_hw_max, "copy_max_hw_bytes"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -634,6 +668,8 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_hw_max_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 51fa7ffdee83..d5019967a908 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -309,6 +309,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned int max_copy_hw_sectors; + unsigned int max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -893,6 +896,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_hw_sectors(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1211,6 +1216,14 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +/* maximum copy offload length, this is set to 128MB based on current testing */ +#define BLK_COPY_MAX_BYTES (1 << 27) + +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { From patchwork Fri Dec 22 06:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503055 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C51BA171C7 for ; Fri, 22 Dec 2023 07:31:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="phC5LFnU" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231222073158epoutp03867a8809900d5e1a6cfdfc66cad889e3~jFvNCAHsT2966029660epoutp03E for ; Fri, 22 Dec 2023 07:31:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231222073158epoutp03867a8809900d5e1a6cfdfc66cad889e3~jFvNCAHsT2966029660epoutp03E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230318; bh=zNOpXkz6e9K4rFrmmQNgX1f5/CjWsYrydMLP3nTXSzc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=phC5LFnUhaYBtBLfoaA92T9OVJdUd8/M5ylcTrjUzn620b5adR5b+qTX9qAd72zom /tdo8Ah7uVci5ca9oHben0r41djzySP0+daH/gj8VI5Vv6Tzh++1bdswcFikr1Opcw acaTfpXVDhVq/helfZEWh9dtZR+h2PICUilyFSRI= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231222073157epcas5p34cf86fc659d43f490607f3bc1d3b26ce~jFvMhFlNH0585105851epcas5p3V; Fri, 22 Dec 2023 07:31:57 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.178]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SxJs76bj4z4x9Q0; Fri, 22 Dec 2023 07:31:55 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id E5.37.10009.B6B35856; Fri, 22 Dec 2023 16:31:55 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20231222062105epcas5p1c21613f0c44451d579ae4cd24003cca2~jExUqR4Ov1748217482epcas5p19; Fri, 22 Dec 2023 06:21:05 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231222062105epsmtrp23d848ed4420c0c59a8032b1b8568ec1d~jExUpUc2E1647616476epsmtrp2S; Fri, 22 Dec 2023 06:21:05 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-e4-65853b6b8346 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 11.93.08755.1DA25856; Fri, 22 Dec 2023 15:21:05 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062102epsmtip2e916b9b662132bc0cb3cccba89ce6c13~jExRPlMhy0303903039epsmtip2W; Fri, 22 Dec 2023 06:21:01 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 02/12] Add infrastructure for copy offload in block and request layer. Date: Fri, 22 Dec 2023 11:42:56 +0530 Message-Id: <20231222061313.12260-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TfVCTdRz39zzPHjYTewSKHxA2V5mIwEawfryJV0aPRScd3VUKBwsegQO2 tReU6gpS4kXkZQLGcIwXzUBkCUSADmlUxHt3wNAlKXebehCKcGJCVFuD8r/P9/v9fL6v92Xj Ticd3NkpYgUjE4vSeORGor3Xa4dPakgOw7dc90e6gZ9wdP/BCoE+L1nF0fmpYhLN9i4AZO7J BaimTkOgaz2dGGo4/yOGVAYjQJYJNYb0Jm9U+8UZAl3W9xNorOs0ibRfWRzQ8ckOEp3r+wtD V0ssAJXmTWCow5wNUPuKFkfNs/cI9LPJA42u9rH2uNGd6ikHevS3iwQ9NqykWxrzSbr1zGf0 ndZKQF+6lkXS9UUnWfSJo3dJ+r7FRND3uidIuqitEdCLLVvpFvMcFrX5QGpoMiNKZGRcRpwg SUwRJ4Xx3oyOezUuUMgX+AiC0Ms8rliUzoTx9kZG+USkpFk3weNmiNKUVleUSC7n+e0OlUmU CoabLJErwniMNDFNGiD1lYvS5Upxkq+YUQQL+Hz/QCsxPjW5v+oXUroccqSifZWVBQb4BYDD hlQArB05BQrARrYTdQnA4dkKB7uxAODvXVmkjeVELQF49frr64oqVR1pJ+kBHLqhZ9mNHAwu nbtjlbPZJOUNB/9m2wQuVBMOOy8KbByc+hODqtsGli3gTMXA/i+XMBsmqBfg0GA1sGFHKhiO Nv9B2PJAyg8W39hic3OoEHh7uZ5lp2yB/ZVmwoZx6ll49Nsq3JYfUjoOLFusJe2d7oULx+pY duwMZ/raHOzYHS7e1a9xDsOGsq9Ju/gYgOpJNbAHwmHOQDFuawKnvKCuy8/u9oTlA82YvfBm eGLFjNn9jrCjeh0/B5t0NWv53aDxYfYapmGu7sLa5ooAbCnvxUoAV/3YQOrHBlL/X7oG4I3A jZHK05MYeaDUX8wc/u/KCZL0FvDvd+x8owNM35z3NQCMDQwAsnGei6Nk1zHGyTFRlPkRI5PE yZRpjNwAAq0LL8Xdn0qQWN9LrIgTBATxA4RCYUDQS0IBz9VxNkeT6EQliRRMKsNIGdm6DmNz 3LOw7O1NjaeMU6uzm0yuw1UztzThmg9Nvbvf22+4+e73WtXxdz6dlL74liWggcmIC35bzC8Q YvsiDCtdi/GFVaf7tvXFKmPPikYGqz+xnny/j/n5psKg8SMfdA1qvV+BzPx2Syh4xF5RzYzv UJSPR8rL8gX95Zndhz6u/W7fwbYHA8qtxDaj0f1sYfT0wQ1t3Ysd3dFFS5wDjyos6c9EVqpc hEz8kx4/cNp/1Y/tCiqltHM6hScWn/9Ebs8cMx8TY3yt1SfaJXdTpPTCXNPTGhn0TOKGHlIS GqHJ8I02JNkrInYkz7leNxQzfTmvsmDZIyU4/kr48oTHrUzD+xkPN7heGeQR8mSRYCcuk4v+ AaEHZsSmBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG+845Oztaq+MU/DRIGESpZS2FvjTMKOqUlEEoEUUe2ski59bm uhjVykuppbZuOKt5XTpLaTOv2cXKtSy0zNJ1M9qaJFmrFt2mdZKg/37v8zzvywMvhYttRDC1 LS2dU6WxqRLSl2i8JQmZ/TAsm5t78uw0VH+vE0duz08CHSry4qj2RSGJhm99Ashx4zBApeXn CDRwowVDNbV3MKTreAKQs0+PoXZ7OCrLqSTQ1XYbgXpbz5LIYHQKUf7TZhJdsI5iqL/ICdDx I30YanYcBKjxpwFHdcMfCHTXPhV1e62CuCCmRf9CyHS/vEwwvQ80jNmUSzKWygPMkKUYMG0D WpKpKDghYI5ljpCM22knmA/X+kimoMEEmM/maYzZ8R5bM3m970IZl7ptJ6eaE5vsu9VW0kMq f8TsPt3oFWjBvbl5wIeCdBQs0ZWTPIvpNgCzPbJxPQgavbfxcfaHNaMuYR7w/ZPJxODQwVKQ ByiKpMNh1xjF6wF0Mw6/X8rE+AGn83CoG/MK+W1/ej18ZegU8EzQ0+H9rvOAZxEdDbvrvhH8 IUjPgYWv/HjZh46Brh8VgvFC0dDmspDjcT9oK3YQPON0CMy8UoIXAVr/n6X/zyoFmAkEcUq1 PEWuliqladyuCDUrV2vSUiI2K+Rm8PfbYaHNoMn0MaIDYBToAJDCJQEixawsTiySsXsyOJVi k0qTyqk7wFSKkASKAoeOycR0CpvObec4Jaf652KUT7AWa7GHaU+J31ZXmaUtCZPMS+YbWjeL do60DV0LeGRJr83a4ro46fZykz7fc/Nx7FKB99ODRe++HrevTXEHi+Kt4g1PK458iTrjvr67 rHHVSO6v9ndvx06WdykOtcqTwBbTMm1rZM7hR/pQH1cyNb0HhLtDWVVu3+vooKPxLzX1E/cP S6zsknU5GYGF0voZkbEr+z8LCyjo78jtn4LvwxITZsYpU2can+0Q+nUnJtWtWfDcEGesJiUN 8+veLMe2s9mKvfmnDU2KhsH4mgk9iijT0ox12QkbKj0fB1c7NU03z3t6jKOjxaJvMpdqhW7e ypDF+5ObNjreD0ysGq5+4tJZLRJCvZWVhuEqNfsbD7KTz1wDAAA= X-CMS-MailID: 20231222062105epcas5p1c21613f0c44451d579ae4cd24003cca2 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062105epcas5p1c21613f0c44451d579ae4cd24003cca2 References: <20231222061313.12260-1-nj.shetty@samsung.com> We add two new opcode REQ_OP_COPY_DST, REQ_OP_COPY_SRC. Since copy is a composite operation involving src and dst sectors/lba, each needs to be represented by a separate bio to make it compatible with device mapper. We expect caller to take a plug and send bio with destination information, followed by bio with source information. Once the dst bio arrives we form a request and wait for source bio. Upon arrival of source bio we merge these two bio's and send corresponding request down to device driver. Merging non copy offload bio is avoided by checking for copy specific opcodes in merge function. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-core.c | 7 +++++++ block/blk-merge.c | 41 +++++++++++++++++++++++++++++++++++++++ block/blk.h | 16 +++++++++++++++ block/elevator.h | 1 + include/linux/bio.h | 6 +----- include/linux/blk_types.h | 10 ++++++++++ 6 files changed, 76 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 2eca76ccf4ee..51c6cc3022f4 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -121,6 +121,8 @@ static const char *const blk_op_name[] = { REQ_OP_NAME(ZONE_FINISH), REQ_OP_NAME(ZONE_APPEND), REQ_OP_NAME(WRITE_ZEROES), + REQ_OP_NAME(COPY_SRC), + REQ_OP_NAME(COPY_DST), REQ_OP_NAME(DRV_IN), REQ_OP_NAME(DRV_OUT), }; @@ -800,6 +802,11 @@ void submit_bio_noacct(struct bio *bio) if (!q->limits.max_write_zeroes_sectors) goto not_supported; break; + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: + if (!q->limits.max_copy_sectors) + goto not_supported; + break; default: break; } diff --git a/block/blk-merge.c b/block/blk-merge.c index 65e75efa9bd3..bcb55ba48107 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -158,6 +158,20 @@ static struct bio *bio_split_write_zeroes(struct bio *bio, return bio_split(bio, lim->max_write_zeroes_sectors, GFP_NOIO, bs); } +static struct bio *bio_split_copy(struct bio *bio, + const struct queue_limits *lim, + unsigned int *nsegs) +{ + *nsegs = 1; + if (bio_sectors(bio) <= lim->max_copy_sectors) + return NULL; + /* + * We don't support splitting for a copy bio. End it with EIO if + * splitting is required and return an error pointer. + */ + return ERR_PTR(-EIO); +} + /* * Return the maximum number of sectors from the start of a bio that may be * submitted as a single request to a block device. If enough sectors remain, @@ -366,6 +380,12 @@ struct bio *__bio_split_to_limits(struct bio *bio, case REQ_OP_WRITE_ZEROES: split = bio_split_write_zeroes(bio, lim, nr_segs, bs); break; + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: + split = bio_split_copy(bio, lim, nr_segs); + if (IS_ERR(split)) + return NULL; + break; default: split = bio_split_rw(bio, lim, nr_segs, bs, get_max_io_size(bio, lim) << SECTOR_SHIFT); @@ -922,6 +942,9 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (!rq_mergeable(rq) || !bio_mergeable(bio)) return false; + if (blk_copy_offload_mergable(rq, bio)) + return true; + if (req_op(rq) != bio_op(bio)) return false; @@ -951,6 +974,8 @@ enum elv_merge blk_try_merge(struct request *rq, struct bio *bio) { if (blk_discard_mergable(rq)) return ELEVATOR_DISCARD_MERGE; + else if (blk_copy_offload_mergable(rq, bio)) + return ELEVATOR_COPY_OFFLOAD_MERGE; else if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector) return ELEVATOR_BACK_MERGE; else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector) @@ -1053,6 +1078,20 @@ static enum bio_merge_status bio_attempt_discard_merge(struct request_queue *q, return BIO_MERGE_FAILED; } +static enum bio_merge_status bio_attempt_copy_offload_merge(struct request *req, + struct bio *bio) +{ + if (req->__data_len != bio->bi_iter.bi_size) + return BIO_MERGE_FAILED; + + req->biotail->bi_next = bio; + req->biotail = bio; + req->nr_phys_segments++; + req->__data_len += bio->bi_iter.bi_size; + + return BIO_MERGE_OK; +} + static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, struct request *rq, struct bio *bio, @@ -1073,6 +1112,8 @@ static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, break; case ELEVATOR_DISCARD_MERGE: return bio_attempt_discard_merge(q, rq, bio); + case ELEVATOR_COPY_OFFLOAD_MERGE: + return bio_attempt_copy_offload_merge(rq, bio); default: return BIO_MERGE_NONE; } diff --git a/block/blk.h b/block/blk.h index 08a358bc0919..3d51d7827abb 100644 --- a/block/blk.h +++ b/block/blk.h @@ -159,6 +159,20 @@ static inline bool blk_discard_mergable(struct request *req) return false; } +/* + * Copy offload sends a pair of bio with REQ_OP_COPY_DST and REQ_OP_COPY_SRC + * operation by taking a plug. + * Initially DST bio is sent which forms a request and + * waits for SRC bio to arrive. Once SRC bio arrives + * we merge it and send request down to driver. + */ +static inline bool blk_copy_offload_mergable(struct request *req, + struct bio *bio) +{ + return (req_op(req) == REQ_OP_COPY_DST && + bio_op(bio) == REQ_OP_COPY_SRC); +} + static inline unsigned int blk_rq_get_max_segments(struct request *rq) { if (req_op(rq) == REQ_OP_DISCARD) @@ -300,6 +314,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, case REQ_OP_DISCARD: case REQ_OP_SECURE_ERASE: case REQ_OP_WRITE_ZEROES: + case REQ_OP_COPY_SRC: + case REQ_OP_COPY_DST: return true; /* non-trivial splitting decisions */ default: break; diff --git a/block/elevator.h b/block/elevator.h index 7ca3d7b6ed82..eec442bbf384 100644 --- a/block/elevator.h +++ b/block/elevator.h @@ -18,6 +18,7 @@ enum elv_merge { ELEVATOR_FRONT_MERGE = 1, ELEVATOR_BACK_MERGE = 2, ELEVATOR_DISCARD_MERGE = 3, + ELEVATOR_COPY_OFFLOAD_MERGE = 4, }; struct blk_mq_alloc_data; diff --git a/include/linux/bio.h b/include/linux/bio.h index 41d417ee1349..ed746738755a 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -53,11 +53,7 @@ static inline unsigned int bio_max_segs(unsigned int nr_segs) */ static inline bool bio_has_data(struct bio *bio) { - if (bio && - bio->bi_iter.bi_size && - bio_op(bio) != REQ_OP_DISCARD && - bio_op(bio) != REQ_OP_SECURE_ERASE && - bio_op(bio) != REQ_OP_WRITE_ZEROES) + if (bio && (bio_op(bio) == REQ_OP_READ || bio_op(bio) == REQ_OP_WRITE)) return true; return false; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index b29ebd53417d..344dd3740a87 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -395,6 +395,10 @@ enum req_op { /* reset all the zone present on the device */ REQ_OP_ZONE_RESET_ALL = (__force blk_opf_t)17, + /* copy offload src and dst operation */ + REQ_OP_COPY_SRC = (__force blk_opf_t)18, + REQ_OP_COPY_DST = (__force blk_opf_t)19, + /* Driver private requests */ REQ_OP_DRV_IN = (__force blk_opf_t)34, REQ_OP_DRV_OUT = (__force blk_opf_t)35, @@ -483,6 +487,12 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return ((op & REQ_OP_MASK) == REQ_OP_COPY_SRC || + (op & REQ_OP_MASK) == REQ_OP_COPY_DST); +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. From patchwork Fri Dec 22 06:12:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503056 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC50918646 for ; Fri, 22 Dec 2023 07:32:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="hHNBJwOC" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231222073203epoutp021cb75cde9cb0661db8d53aee3d24a62f~jFvRw-M3C1676516765epoutp02Z for ; Fri, 22 Dec 2023 07:32:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231222073203epoutp021cb75cde9cb0661db8d53aee3d24a62f~jFvRw-M3C1676516765epoutp02Z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230323; bh=in2AYpVTNT76lhCvnjMvG4KI3Pxm2pxWCaKDykGs2IQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hHNBJwOCVuRDfHAagqT9Z+x9WW0s/NAstiErN3maDCSWCPPRSz7umM5rC0x9Q9XLr +taDxjQO2p5WLWjHkm9UymzFsclp/wajTfoITBCLpZZWMENq6l2u956fSqz3hDpfsK dr8k/QBVQnI74VF53gAhrKnlsS5ziyPulGaTVbhQ= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231222073202epcas5p208adfa9360db1984ca7359d690abbe15~jFvQzndsb1857018570epcas5p2M; Fri, 22 Dec 2023 07:32:02 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SxJsD1KdBz4x9Pr; Fri, 22 Dec 2023 07:32:00 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 17.90.09672.F6B35856; Fri, 22 Dec 2023 16:31:59 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20231222062116epcas5p2ff54b405039a65f107b2a570b113c501~jExeRsijg0448404484epcas5p2K; Fri, 22 Dec 2023 06:21:16 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062116epsmtrp12c1cdf9606a1bb1630bcab4b6c5d24e0~jExeQkTWF1607116071epsmtrp1n; Fri, 22 Dec 2023 06:21:16 +0000 (GMT) X-AuditID: b6c32a4b-60bfd700000025c8-69-65853b6f38e2 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F1.C8.08817.BDA25856; Fri, 22 Dec 2023 15:21:15 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062112epsmtip24391c108c8dd8663d9345d5960bcf949~jExawUa6O0303903039epsmtip2a; Fri, 22 Dec 2023 06:21:12 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 03/12] block: add copy offload support Date: Fri, 22 Dec 2023 11:42:57 +0530 Message-Id: <20231222061313.12260-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHc+5tLwUtXgqGQ1HsurgNEGilwOEp85WbsSxsi/9gGDb0hkKh bdoyx8YmiIBjQUBgw4I8hCgvYVDGeD9KGApzMhGZTISxoisvhziMMuZaC5v/fX/f8/md3+Pk sHBOkRWXFSvX0Cq5OJ5P2DBaB1xdPZSB6bTglhNqHP4BRyt/rTPQ6dwNHNVN5RBoYeAxQIa+ TIDKL11koLt97RjqunQeQzV1gxg6r78D0Ny4FkPdk+6oIqOKgbq6rzPQWEcJgcouz1mhryba CHRl6B8M/ZI7B1De2XEMtRlSAWpdL8NRw8IjBro26YxubgwxQ7lUu3bKirp5v4lBjd1IpJpr vyQoXdUp6g/dBUB13k0hqMpz+UwqO22ZoFbmJhnUo55xgjrXUgso3chn1GqzC9VsWMLCd0TI gqS0WEKreLQ8WiGJlccE88M+jDoU5eMrEHoI/ZEfnycXJ9DB/MPvhnscjY03bYTP+1gcn2iy wsVqNd8rJEilSNTQPKlCrQnm00pJvFKk9FSLE9SJ8hhPOa0JEAoE+31M4AmZ9EFVM6HMCP3E eKXJKgVseGcBaxYkRXD83hgzC9iwOGQngP1TpZvBYwAL799imCkOuQZgSpn/VsbtomuYBeoG cHglFViCdAwO5FVYZQEWiyDd4cgLljnBgazHYXuT0MzgZCkOdbNDmJmxJwPgzOW9ZoZB7oW/ 9k8SZs022XOFz5lmBJJeMGfazmxbk4Hw4fNKpgWxg9cvGF72hpN7YNp3xbilt++t4eJ8uCX1 MBxoCLHY9nB+qMXKornQmJOxqU/CmoJqwtwZJM8AqJ3QAsvBAZg+nIOb78FJV9jY4WWxd8PC 4QbMUtYWZq8bMIvPhm2lW/p1WN9YTli0E7zzNHVTU7C4Sb+5tnMAzrYugVzA074yjvaVcbT/ ly4HeC1wopXqhBha7aP0ltMn/3viaEVCM3j5RdzC2sDszJ+eeoCxgB5AFs53YCv2naE5bIk4 6VNapYhSJcbTaj3wMa07D+fujFaY/phcEyUU+QtEvr6+In9vXyHfkb2QflHCIWPEGlpG00pa tZWHsay5KVhuWfRu472vj/yUdHy+y37XysLTb2qV/FNOV2/YVu8s+SIi8re4d944Nkpu1+Vy Pir2jPBaVgh+9DUccJ8erKzId372tt1Us+OTkjxF2aHC6vZy7nvbl2NfO92/pHFpUf9c8OZQ Y+Vil8RLRkhu9zJkSbscaiYH3bxZYw2ZvZ8v2I72uI64OKpEabz1icb1hr4jTkmZ5D72W/lS nl9ccvq3CmOqX53zmmxH62rkuPF9xdVtKmGoTrq/SD/LmtmzbTUikFuSvBgy2hHw7EHadFye CgVl9wh/z/87Wf2k84QN1PUaH4YxDx41nvU5bmDyp15oP3A85lwQ2VafwkleDRQlp6zxGWqp WOiGq9TifwGNNZNSqwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrAIsWRmVeSWpSXmKPExsWy7bCSvO5trdZUgz+njSzWnzrGbPHx628W i6YJf5ktVt/tZ7N4ffgTo8WTA+2MFgsWzWWxuHlgJ5PFnkWTmCxWrj7KZDHp0DVGi6dXZzFZ 7L2lbbGwbQmLxZ69J1ksLu+aw2Yxf9lTdovu6zvYLJYf/8dkcWPCU0aLiR1XmSx2PGlktNj2 ez6zxbrX71ksTtyStjj/9zirg5THzll32T3O39vI4nH5bKnHplWdbB6bl9R7vNg8k9Fj980G No/FfZNZPXqb37F5fHx6i8Xj/b6rbB59W1Yxemw+Xe3xeZOcx6Ynb5kC+KO4bFJSczLLUov0 7RK4Mp4t2cRW0OZQ8XL5RvYGxr/GXYycHBICJhJXZpxgArGFBHYzSszcqQgRl5RY9vcIM4Qt LLHy33P2LkYuoJpmJol/vc8Yuxg5ONgEtCVO/+cAiYsI7GCW+Lm2GWwQs8AaZoktc3lBaoQF rCQeLFMFCbMIqErcPniLDcTmBQo/nfqLFaREQkBfov++IEiYU8Ba4vmvxawQ51hJnHy+Gapc UOLkzCcsENPlJZq3zmaewCgwC0lqFpLUAkamVYySqQXFuem5xYYFRnmp5XrFibnFpXnpesn5 uZsYwZGupbWDcc+qD3qHGJk4GA8xSnAwK4nw5uu0pArxpiRWVqUW5ccXleakFh9ilOZgURLn /fa6N0VIID2xJDU7NbUgtQgmy8TBKdXAJMeQc+Q/y5wpekluO7lKGbb13Lybz62sqBNjaX7o uHCxpIyF+A5O3ff5HKvT38/OOCV34OLK6K/5Gy6WOjv7Ht3lxf7EqPFImzTj1/Rvi6aduW3z cuXVXi1nvnNnZB0su4OvyVxdfPKJ38GvkgeW//jHt5F9kdNTj3Uxpl5+5jUrgmZ+WmCXPtP4 la/E/4tXtqi2ray+Xbvr2dZ04XsLdln7PNetMTAQ4fzcJvu4jf2aQcZX5yMxjubNvx6U7+a2 2eX4b5Pop1+FlzYFnfdunZDLc1vgQfusO5e/5z2ZeYN9ZdenRYYGpmXH58Rv3KJSwMt0VDY8 6FO/iOeOuflXPv90nj1JzD8kUDHhyCXfjUosxRmJhlrMRcWJAO1ue45jAwAA X-CMS-MailID: 20231222062116epcas5p2ff54b405039a65f107b2a570b113c501 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062116epcas5p2ff54b405039a65f107b2a570b113c501 References: <20231222061313.12260-1-nj.shetty@samsung.com> Introduce blkdev_copy_offload to perform copy offload. Issue REQ_OP_COPY_DST with destination info along with taking a plug. This flows till request layer and waits for src bio to arrive. Issue REQ_OP_COPY_SRC with source info and this bio reaches request layer and merges with dst request. For any reason, if a request comes to the driver with only one of src/dst bio, we fail the copy offload. Larger copy will be divided, based on max_copy_sectors limit. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 204 +++++++++++++++++++++++++++++++++++++++++ include/linux/blkdev.h | 4 + 2 files changed, 208 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..f03cb4bc3134 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -10,6 +10,22 @@ #include "blk.h" +/* Keeps track of all outstanding copy IO */ +struct blkdev_copy_io { + atomic_t refcount; + ssize_t copied; + int status; + struct task_struct *waiter; + void (*endio)(void *private, int status, ssize_t copied); + void *private; +}; + +/* Keeps track of single outstanding copy offload IO */ +struct blkdev_copy_offload_io { + struct blkdev_copy_io *cio; + loff_t offset; +}; + static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); @@ -115,6 +131,194 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +static inline ssize_t blkdev_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, + struct block_device *bdev_out, + loff_t pos_out, size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= BLK_COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +static inline void blkdev_copy_endio(struct blkdev_copy_io *cio) +{ + if (cio->endio) { + cio->endio(cio->private, cio->status, cio->copied); + kfree(cio); + } else { + struct task_struct *waiter = cio->waiter; + + WRITE_ONCE(cio->waiter, NULL); + blk_wake_io_task(waiter); + } +} + +/* + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to complete. + * Returns the length of bytes copied or error + */ +static ssize_t blkdev_copy_wait_for_completion_io(struct blkdev_copy_io *cio) +{ + ssize_t ret; + + for (;;) { + __set_current_state(TASK_UNINTERRUPTIBLE); + if (!READ_ONCE(cio->waiter)) + break; + blk_io_schedule(); + } + __set_current_state(TASK_RUNNING); + ret = cio->copied; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_src_endio(struct bio *bio) +{ + struct blkdev_copy_offload_io *offload_io = bio->bi_private; + struct blkdev_copy_io *cio = offload_io->cio; + + if (bio->bi_status) { + cio->copied = min_t(ssize_t, offload_io->offset, cio->copied); + if (!cio->status) + cio->status = blk_status_to_errno(bio->bi_status); + } + bio_put(bio); + kfree(offload_io); + + if (atomic_dec_and_test(&cio->refcount)) + blkdev_copy_endio(cio); +} + +/* + * @bdev: block device + * @pos_in: source offset + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, + * for synchronous operation this should be NULL + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * For synchronous operation returns the length of bytes copied or error + * For asynchronous operation returns -EIOCBQUEUED or error + * + * Description: + * Copy source offset to destination offset within block device, using + * device's native copy offload feature. + * We perform copy operation using 2 bio's. + * 1. We take a plug and send a REQ_OP_COPY_DST bio along with destination + * sector and length. Once this bio reaches request layer, we form a + * request and wait for dst bio to arrive. + * 2. We issue REQ_OP_COPY_SRC bio along with source sector, length. + * Once this bio reaches request layer and find a request with previously + * sent destination info we merge the source bio and return. + * 3. Release the plug and request is sent to driver + * This design works only for drivers with request queue. + */ +ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, + loff_t pos_out, size_t len, + void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp) +{ + struct blkdev_copy_io *cio; + struct blkdev_copy_offload_io *offload_io; + struct bio *src_bio, *dst_bio; + size_t rem, chunk; + size_t max_copy_bytes = bdev_max_copy_sectors(bdev) << SECTOR_SHIFT; + ssize_t ret; + struct blk_plug plug; + + if (!max_copy_bytes) + return -EOPNOTSUPP; + + ret = blkdev_copy_sanity_check(bdev, pos_in, bdev, pos_out, len); + if (ret) + return ret; + + cio = kzalloc(sizeof(*cio), gfp); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 1); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + /* + * If there is a error, copied will be set to least successfully + * completed copied length + */ + cio->copied = len; + for (rem = len; rem > 0; rem -= chunk) { + chunk = min(rem, max_copy_bytes); + + offload_io = kzalloc(sizeof(*offload_io), gfp); + if (!offload_io) + goto err_free_cio; + offload_io->cio = cio; + /* + * For partial completion, we use offload_io->offset to truncate + * successful copy length + */ + offload_io->offset = len - rem; + + dst_bio = bio_alloc(bdev, 0, REQ_OP_COPY_DST, gfp); + if (!dst_bio) + goto err_free_offload_io; + dst_bio->bi_iter.bi_size = chunk; + dst_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + + blk_start_plug(&plug); + src_bio = blk_next_bio(dst_bio, bdev, 0, REQ_OP_COPY_SRC, gfp); + if (!src_bio) + goto err_free_dst_bio; + src_bio->bi_iter.bi_size = chunk; + src_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + src_bio->bi_end_io = blkdev_copy_offload_src_endio; + src_bio->bi_private = offload_io; + + atomic_inc(&cio->refcount); + submit_bio(src_bio); + blk_finish_plug(&plug); + pos_in += chunk; + pos_out += chunk; + } + + if (atomic_dec_and_test(&cio->refcount)) + blkdev_copy_endio(cio); + if (endio) + return -EIOCBQUEUED; + + return blkdev_copy_wait_for_completion_io(cio); + +err_free_dst_bio: + bio_put(dst_bio); +err_free_offload_io: + kfree(offload_io); +err_free_cio: + cio->copied = min_t(ssize_t, cio->copied, (len - rem)); + cio->status = -ENOMEM; + if (rem == len) { + ret = cio->status; + kfree(cio); + return ret; + } + if (cio->endio) + return cio->status; + + return blkdev_copy_wait_for_completion_io(cio); +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index d5019967a908..cd940de16750 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1042,6 +1042,10 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, + loff_t pos_out, size_t len, + void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Fri Dec 22 06:12:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503057 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49E291CA9F for ; Fri, 22 Dec 2023 07:32:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="MBG6W+jJ" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231222073210epoutp035aa218f5ca20d0536cdfff2ffc84c3fa~jFvYwpNlE2967629676epoutp03I for ; Fri, 22 Dec 2023 07:32:10 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231222073210epoutp035aa218f5ca20d0536cdfff2ffc84c3fa~jFvYwpNlE2967629676epoutp03I DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230330; bh=9frU+oN9cW84tUmQ8PszwC/fCsF2sdAWZkWnbtQtafI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MBG6W+jJ6LXb2iwYngYt7WcqC/DsvTBvk2L1UI78B7u8NPXmC+f+gTDa/bK0P/BIC pC6W8eqA12L3T7vCfjCl+LynObzh9uf/iyXCjjYX9PtXoiToD10Mj1T/av/jys8hNz wIB/Vntuws6no3thzjAHjGnyvRYU5Sdlj7CS8QGQ= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231222073210epcas5p2d828fcd26c2acef7da1ed7d5eaed9d97~jFvYP30bG2130021300epcas5p2M; Fri, 22 Dec 2023 07:32:10 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.174]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SxJsM5XBzz4x9Q9; Fri, 22 Dec 2023 07:32:07 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 0F.D2.09634.77B35856; Fri, 22 Dec 2023 16:32:07 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20231222062126epcas5p382f390cbedd5351c49adf07121d18448~jExoFy5wl0194201942epcas5p3T; Fri, 22 Dec 2023 06:21:26 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062126epsmtrp131e61b114be345c1c8edb06d53480a52~jExoEtGIY1607116071epsmtrp1s; Fri, 22 Dec 2023 06:21:26 +0000 (GMT) X-AuditID: b6c32a49-159fd700000025a2-6d-65853b773aa6 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 8B.DE.18939.6EA25856; Fri, 22 Dec 2023 15:21:26 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062122epsmtip234bd0da50619d5d996eaf8c8abad787c~jExkq0fXB0344503445epsmtip2p; Fri, 22 Dec 2023 06:21:22 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Vincent Fu , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 04/12] block: add emulation for copy Date: Fri, 22 Dec 2023 11:42:58 +0530 Message-Id: <20231222061313.12260-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TaVBTVxjtfe/xCMxEn2jbm1QwE0sdF5bYJF5EilbF50QrTqd1pq2lGXiF DNnMAlbaKQgBDbJWbAkiVKgLIluQYSkWgRBFlHYUUGYAWxNKQUuBQasgNiFo/Xe+851zv+XO x8K9fnDnsmRKHaNRSuV80pOoa1u9yi8+2MAEHrOsQpWdHTiamJ4h0OHsZzi6MJBForG2SYBs LWkAFZ8uJNDdlgYMnb9gwVBuay9A9h4Thpr716IfU0sJ9HPzNQLdajxJoqIzdneU3ldPorPW OQzdybYDlHOkB0P1tiSA6maKcFQxNk6gq/1vofvpRwDqfmZ128ylG0wD7nT3YDVB37qhp2vK jpK0ufRbesScD+imu4kkXZL5nRudkfw3SU/Y+wl6/HIPSWfWlgF6qsaHrrE9xMIXfRK7KYaR RjEaHqOMVEXJlNEhfMmHEVsjROJAgZ8gCG3g85RSBRPC37Yr3C9MJnesg8+Lk8r1DipcqtXy A97bpFHpdQwvRqXVhfAZdZRcLVT7a6UKrV4Z7a9kdBsFgYHrRQ7hF7ExjyyzmLr7/YNDN21E IpgUGYEHC1JC+IehEzMCT5YX1QRgZ0eFmyuYBHD4shF7GdiNRuKFJbXBjLsSDQAOPT65oDJg cOTBFYeKxSKptfD6c5bTsIwqx2FDtcCpwakSHDYPFpHOxFIKQcOU1c2JCcoXzqUWzFdgUxvh qeuFpPMdSAXArKElTtqDCoZ/Pi1xc0mWwGv5tnk5Tq2AyZcK5huCVKUHbMzJW+h0G0xp68Jd eCkctda6uzAX/pWVuoDj4fnj50iXOQVAU58JuBKh0NCZhTubwKnVsLIxwEV7w7zOCsxVeBHM mLFhLp4N60+9wCtheWUx6cIc2Ps4aQHTcCata/55LyoTwF/GQrMBz/TKPKZX5jH9X7kY4GWA w6i1imhGK1ILlEz8y1+OVClqwPyJrNlZDwbu/ePfCjAWaAWQhfOXsVXrUhgvdpT0q0OMRhWh 0csZbSsQOfadg3Nfj1Q5bkypixAIgwKFYrFYGPSuWMB/kz1mKIzyoqKlOiaWYdSM5oUPY3lw EzGUwKP3lwdwfjftv3LIzuoLitij1Ncm7W2ffZ7zdKQ7NKtG0rHFmrvjdHVlz0qBZ8DXeX77 GieeeMuMceOffZNpbVHPpfl2ejOKf8vjD8CmoyfMbaVbMtqzD8duP56/nTibIH/N19Iikkmq uc0bpkc+3/2oK1YynLzrgE/xvYLcg2HDPyVYf+Ml7O77VHNx8YmHijR7xx5s5sb9d7LXCz/Q HxuWFNzeKd4hu2queoNTxInbN3QmtLDKfKdqLuwiOyma8wCte5tjCZ7194msvd1xqX3zYs+h jMn0c7/WVY9a9rbmFg6WjXpTyhrOiqmPp/DpJ5MJN7/0KOU3VWiWTy3/vvcjCZ/QxkgFa3CN Vvof1rR34KsEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra1BMYRzG5z3ndDqtyrG5vIpWa8hkRa7vuA0xnD6YwciMxbBjj2q07dpt U2qo3XTZSJHJblKUaEm1S8qlmi2lm8aliww7o13dkGrcosWKGd9+8zy///PlT+HcRsKdCgkL Z+VholA+ySHKavi8BW99TrCLUjshKm6sw9HQp+8EUqWN4ej6q9MkGqgZBshSnQhQ7uVsAr2o rsBQ4fWHGDpjagfI2qbD0IOu+ehSQj6B7j9oINCzuxdIlFNgdUQpHeUkulpvw1BnmhWg9KQ2 DJVb4gAq+56Do5sDgwR61OWBulOSAGodq3dY585U6F45Mq2vSwnmWYuSMeiTScaYf5zpNWoB c+9FLMnkpZ51YE6pP5DMkLWLYAYr20gm9ZYeMCMGT8ZgeY9tdRVyVovZ0JAIVr5w7X5O8OeH PzBZq3+k+bGFiAXDyzTAiYL0UphQYcQ1gENx6TsANr1NwMeL6bBgrPYvu8FCW4/juKTG4L2y KgcNoCiSng+bflL2fDJdjsNvRWrMfoDTpTh8Wrfczm40gidG6h3sTNBzoC0hi7CzC70SXmzK Ju07kF4IT5sn2WMnehXsGc37o3N/Kw09RnJcnwQbtBZifJ4H1bez8DRA6/6rdP9VuQDTgyms TCEJkhyQ+fkqRBKFMizI94BUYgB/3u2zvRwUFI/5mgBGAROAFM6f7CIVxLNcF7Eo6igrl+6T K0NZhQl4UAR/msvs0GQxlw4ShbOHWFbGyv+1GOXkHot5O28dWMFJbaiMSowbUR0Mr7NkZDTv yKxR6TU/v+yoHVL1zzTtqk8O3lgYZKHbpQGLBXOrVvMH1kdsbpEFjG75Ia7kKTNV0V5Ci8fH vGMdObn7PL96CSIXJQbGi5LO6hOvvQzs1B4+Y5VE+vcWZe9ekhHQJ98rNL+Z0JfsHH9rf0SK MVxYiq5EK24s8V83sUd97vimzc07gWZ4dFDc66lcHG/wfg6iLsTM0jWLn2zof1nm1UnPeyc4 OVVyJDJGqC6pDTPfD+l+02J0pbvPayvjPkmc+UvbtzfClXuac3i2kXyetmTG57YF5nTXbVJF i+1OwaWhR8Vr4jKgoGZj4OguPqEIFvn54HKF6Beo0twiXQMAAA== X-CMS-MailID: 20231222062126epcas5p382f390cbedd5351c49adf07121d18448 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062126epcas5p382f390cbedd5351c49adf07121d18448 References: <20231222061313.12260-1-nj.shetty@samsung.com> For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination. At present in kernel user of emulation is fabrics. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 223 +++++++++++++++++++++++++++++++++++++++++ include/linux/blkdev.h | 4 + 2 files changed, 227 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index f03cb4bc3134..9d5fe177308c 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -26,6 +26,20 @@ struct blkdev_copy_offload_io { loff_t offset; }; +/* Keeps track of single outstanding copy emulation IO */ +struct blkdev_copy_emulation_io { + struct blkdev_copy_io *cio; + struct work_struct emulation_work; + void *buf; + ssize_t buf_len; + loff_t pos_in; + loff_t pos_out; + ssize_t len; + struct block_device *bdev_in; + struct block_device *bdev_out; + gfp_t gfp; +}; + static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); @@ -319,6 +333,215 @@ ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, } EXPORT_SYMBOL_GPL(blkdev_copy_offload); +static void *blkdev_copy_alloc_buf(ssize_t req_size, ssize_t *alloc_size, + gfp_t gfp) +{ + int min_size = PAGE_SIZE; + char *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp); + if (buf) { + *alloc_size = req_size; + return buf; + } + req_size >>= 1; + } + + return NULL; +} + +static struct bio *bio_map_buf(void *data, unsigned int len, gfp_t gfp) +{ + unsigned long kaddr = (unsigned long)data; + unsigned long end = (kaddr + len + PAGE_SIZE - 1) >> PAGE_SHIFT; + unsigned long start = kaddr >> PAGE_SHIFT; + const int nr_pages = end - start; + bool is_vmalloc = is_vmalloc_addr(data); + struct page *page; + int offset, i; + struct bio *bio; + + bio = bio_kmalloc(nr_pages, gfp); + if (!bio) + return ERR_PTR(-ENOMEM); + bio_init(bio, NULL, bio->bi_inline_vecs, nr_pages, 0); + + if (is_vmalloc) { + flush_kernel_vmap_range(data, len); + bio->bi_private = data; + } + + offset = offset_in_page(kaddr); + for (i = 0; i < nr_pages; i++) { + unsigned int bytes = PAGE_SIZE - offset; + + if (len <= 0) + break; + + if (bytes > len) + bytes = len; + + if (!is_vmalloc) + page = virt_to_page(data); + else + page = vmalloc_to_page(data); + if (bio_add_page(bio, page, bytes, offset) < bytes) { + /* we don't support partial mappings */ + bio_uninit(bio); + kfree(bio); + return ERR_PTR(-EINVAL); + } + + data += bytes; + len -= bytes; + offset = 0; + } + + return bio; +} + +static void blkdev_copy_emulation_work(struct work_struct *work) +{ + struct blkdev_copy_emulation_io *emulation_io = container_of(work, + struct blkdev_copy_emulation_io, emulation_work); + struct blkdev_copy_io *cio = emulation_io->cio; + struct bio *read_bio, *write_bio; + loff_t pos_in = emulation_io->pos_in, pos_out = emulation_io->pos_out; + ssize_t rem, chunk; + int ret = 0; + + for (rem = emulation_io->len; rem > 0; rem -= chunk) { + chunk = min_t(int, emulation_io->buf_len, rem); + + read_bio = bio_map_buf(emulation_io->buf, + emulation_io->buf_len, + emulation_io->gfp); + if (IS_ERR(read_bio)) { + ret = PTR_ERR(read_bio); + break; + } + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, emulation_io->bdev_in); + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = chunk; + ret = submit_bio_wait(read_bio); + kfree(read_bio); + if (ret) + break; + + write_bio = bio_map_buf(emulation_io->buf, + emulation_io->buf_len, + emulation_io->gfp); + if (IS_ERR(write_bio)) { + ret = PTR_ERR(write_bio); + break; + } + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, emulation_io->bdev_out); + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_iter.bi_size = chunk; + ret = submit_bio_wait(write_bio); + kfree(write_bio); + if (ret) + break; + + pos_in += chunk; + pos_out += chunk; + } + cio->status = ret; + kvfree(emulation_io->buf); + kfree(emulation_io); + blkdev_copy_endio(cio); +} + +static inline ssize_t queue_max_hw_bytes(struct request_queue *q) +{ + return min_t(ssize_t, queue_max_hw_sectors(q) << SECTOR_SHIFT, + queue_max_segments(q) << PAGE_SHIFT); +} +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, + * for synchronous operation this should be NULL + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * For synchronous operation returns the length of bytes copied or error + * For asynchronous operation returns -EIOCBQUEUED or error + * + * Description: + * If native copy offload feature is absent, caller can use this function + * to perform copy. + * We store information required to perform the copy along with temporary + * buffer allocation. We async punt copy emulation to a worker. And worker + * performs copy in 2 steps. + * 1. Read data from source to temporary buffer + * 2. Write data to destination from temporary buffer + */ +ssize_t blkdev_copy_emulation(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct blkdev_copy_emulation_io *emulation_io; + struct blkdev_copy_io *cio; + ssize_t ret; + size_t max_hw_bytes = min(queue_max_hw_bytes(in), + queue_max_hw_bytes(out)); + + ret = blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + cio = kzalloc(sizeof(*cio), gfp); + if (!cio) + return -ENOMEM; + + cio->waiter = current; + cio->copied = len; + cio->endio = endio; + cio->private = private; + + emulation_io = kzalloc(sizeof(*emulation_io), gfp); + if (!emulation_io) + goto err_free_cio; + emulation_io->cio = cio; + INIT_WORK(&emulation_io->emulation_work, blkdev_copy_emulation_work); + emulation_io->pos_in = pos_in; + emulation_io->pos_out = pos_out; + emulation_io->len = len; + emulation_io->bdev_in = bdev_in; + emulation_io->bdev_out = bdev_out; + emulation_io->gfp = gfp; + + emulation_io->buf = blkdev_copy_alloc_buf(min(max_hw_bytes, len), + &emulation_io->buf_len, gfp); + if (!emulation_io->buf) + goto err_free_emulation_io; + + schedule_work(&emulation_io->emulation_work); + + if (cio->endio) + return -EIOCBQUEUED; + + return blkdev_copy_wait_for_completion_io(cio); + +err_free_emulation_io: + kfree(emulation_io); +err_free_cio: + kfree(cio); + return -ENOMEM; +} +EXPORT_SYMBOL_GPL(blkdev_copy_emulation); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index cd940de16750..80867c9fd602 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1046,6 +1046,10 @@ ssize_t blkdev_copy_offload(struct block_device *bdev, loff_t pos_in, loff_t pos_out, size_t len, void (*endio)(void *, int, ssize_t), void *private, gfp_t gfp_mask); +ssize_t blkdev_copy_emulation(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, void (*endio)(void *, int, ssize_t), + void *private, gfp_t gfp); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Fri Dec 22 06:12:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503058 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CACC51EB5A for ; Fri, 22 Dec 2023 07:32:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="aFFNXEWc" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20231222073216epoutp04bd0546d516c75b4f24be866a814365ae~jFvds2XCi1647216472epoutp04f for ; Fri, 22 Dec 2023 07:32:16 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20231222073216epoutp04bd0546d516c75b4f24be866a814365ae~jFvds2XCi1647216472epoutp04f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230336; bh=WyEZwCEcMZEgd+uPDvb2WQHlAKDzDTA2b4v6jEJOw5k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aFFNXEWcDSD08Jt4soITZ4CqvtFquJiYPLx0JiGHGyIuKL+tH5o9yZKacBbvZ7Xhv PrUqQ07BqhP1aOUJ7RAM0MbQO44oIDoqpipf5559/APFr+SAQWTyYAsfmfsIKF1JcH heaUv64PVHtIzO+3iUTb9CURktKTFXKaXl5vVvvg= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231222073215epcas5p3d382a2a311619cd6cbaeddbeb22c4db8~jFvdKbJmw0585105851epcas5p31; Fri, 22 Dec 2023 07:32:15 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SxJsT44Cmz4x9Q1; Fri, 22 Dec 2023 07:32:13 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 9A.54.08567.D7B35856; Fri, 22 Dec 2023 16:32:13 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20231222062136epcas5p18f39a4dea2e66c56c652f29c0dc85a15~jExxKy2F72292622926epcas5p1C; Fri, 22 Dec 2023 06:21:36 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231222062136epsmtrp2e00b9a0c775dac9476ac12096c995dc0~jExxJ2QAp1637116371epsmtrp2K; Fri, 22 Dec 2023 06:21:36 +0000 (GMT) X-AuditID: b6c32a44-3abff70000002177-7e-65853b7d75e0 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 20.3D.07368.0FA25856; Fri, 22 Dec 2023 15:21:36 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062132epsmtip2ec4c90b353e75c0a4e1c894584ccc4c5~jExtyQ-Ok0362303623epsmtip2U; Fri, 22 Dec 2023 06:21:32 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Anuj Gupta , Hannes Reinecke , Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 05/12] fs/read_write: Enable copy_file_range for block device. Date: Fri, 22 Dec 2023 11:42:59 +0530 Message-Id: <20231222061313.12260-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te1BUZRTvu3f3smALFyT7XIZiNmkQXGCL3T5UhFG0O8VMBGiN0wzcYa/A ALvbPjJsQhCQR8oCYhNL8hKpBUaUl8tjzUDeKRnyLEpHIHINB8ggGZdYLpT//c7vnN/vfOd8 c3i4U7GNgBcn1zAqOZ0gJOw4zZ07PUWf78lgfOtWBKiuvxtH809WOOhU3jMc1UzqCGTuXABo 6kYmQGUVFzho/EYLhtorCjBkqOnCUEHHCEDTw3oMmSa8UPnpSg5qN/Vx0FDr1wQqrZq2QV+M Ggn0TY8FQ2N50wDlZw1jyDiVClDzSimOLpsfc1DvhAsafNbDDRJQLfpJG2rw16scauiWlqqv ziaohsqT1GxDEaDaxlMI6mLuOS51Nm2OoOanJzjU4+vDBJXbWA2ohoHPqMX6V6j6qT+xUIej 8XtjGVrGqNwYebRCFiePCRC+Gx55IFIi9RWLxP7oLaGbnE5kAoTBIaGiQ3EJaysRun1CJ2jX qFBarRb67NurUmg1jFusQq0JEDJKWYLST+mtphPVWnmMt5zR7Bb7+r4hWSuMio9dTZ3hKuf4 n85137dJAT9uyQG2PEj6QXO9GbNiJ7INwNv36Rxgt4YXADQ8+oHLBn8DOFb8BN9UFKSVYGzC BGB6toXDBhkYPJ/5D5EDeDyC9IIDqzyrwJmsxWHLVbG1BidLcDizWgasia1kBHxwqZ9jxRzS HVaduUJYMZ/cDR8uTK77QNIH6n5ztNK25B74+9OLXLbEEfYVTa1LcfJVmNZUjFv9IXnNFj7I a9rQBsOKAgX76K3wYU+jDYsFcHHORLD4ODQUfkuw2nQA9aN6wCYCYUa/Drf64OROWNfqw9Ku 8Hz/ZYztaw/PrkxhLM+HxpJN/BqsrSvb8N8OR5ZSNzAFh7//CbC7ygXw56FWmzzgpn9uHv1z 8+j/b10G8GqwnVGqE2OYaIlSLGeO//fJ0YrEerB+JZ7BRjBWavHuABgPdADIw4XOfMWudMaJ L6OTTjAqRaRKm8CoO4Bkbd/5uOClaMXamck1kWI/f18/qVTq5/+mVCx8mW/OuCBzImNoDRPP MEpGtanDeLaCFMw1/mnbtagjFLfFFGcoDKgcLcsqad4W5kHMv3j0sHnZW+u4avyoTXTA9x3K w3hPcmjgbmaI69zyHY3IIki+bbmUuU/od8/OvqtkNmz1XD9aDHo7NqwHK9iVd8ekfsG5u9Z+ i+rUtGRwadm9XDQb+uHrqvnIaYFra7m9Z0YSlnwiqT7k/blbomNZhqqanizGfOb6oMohyGV/ 9v7Fbe6NdwexqEDXX8hA90ceEe+RV/yr8otOw84ol+qvRO3SssMWOj9Zd3JJNxMeXqjs+YD7 3aKo1yT00q0ea3dylPUe8T8oc4wo71J/qaV3ZIxg464eTTtuZkZ8/NfNBQfeH6OReX0GIUcd S4s9cZWa/hfUNTGLrgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RezBUYRjG+845exxbW8dm6otK7ZRKRSZTn+4zmul0G7rMFLrtcEbG2t3O 0n3KtoUkl9VUVlEKsbrZGBJp2YSWZlgilGGTmohGM4VVVs303zPP73l//7wULqwnHKhgaRjL ScUSEcknCspFTku/uVxgl2k7BOhh9Usc9Q8OEehcwgiOtG3xJPpSPgBQV1kUQLfSbxKouawI Q8/S1RjK1howpNY3AmQ2aTBU0rIY3Y68S6BnJVUEqn96g0RpmWYbdKmpkERZlRYMvU0wA5QY bcJQYZcSoIKhNBw9+NJHoFctjqhupJK3wYEp0rTZMHXtjwmm3hjO5OVcJBnd3bPMJ10yYIqb I0jmTlwSj7ms6iWZfnMLwfSVmkgm7kkOYHQ1p5jvebOZvK6vmM8UP/6aQFYSfJTl3NYd4h8e VX7kyXsFx3tfdthEgDcTY4AtBWkPqFalYjGATwnpYgD7Rn8Q42AGzBypwMfzVJht6bYZH6kw OJz6mIwBFEXSi2HNKDXW29OFOPx5X2U14XQuDhtMyeTY9VR6J1R+zbdaCXo+zIx9ZO0F9Cr4 eaDNKoK0G4x/bzdW29KrYfevO7yxLPwzqerW/Z3bwarkLqsGp52gKj8FTwC05j+k+Q/dAlgO mMHKFaFBoQHucncpe8xVIQ5VhEuDXANkoXnA+neXhYWgPc3iqgcYBfQAUrjIXiBbcp4VCgLF J06ynOwgFy5hFXrgSBGi6QL36ymBQjpIHMaGsKyc5f5RjLJ1iMD2eBVrf1aXRGaXukQZL/kU 7zIeSKhavqnTL2Rlu2/GhmHzlQABX+ok+Bjrf9lw9gp5sGJOpykEE7V6vGhodZ77euKZnu07 yvfWaNWDPM98vclB+anH+2jtHENR0izLcb6/94P+k9NaPb+5JQ7YymupSV7O33nG8n2OK/wH Ugd9worClySZ9mddXZvbEMJxk9bbR9xs2t2zNQ75Nsos+YsMhvQ1ubl+Rh/UH7B53QkUyW0M njl5dsOB6OlK/VVZ56pfBu0HybYY3DIvujrlGmZfyx2J7T3j6ZVY6f2uY4V5o+uCQfWj7VKJ 7t6EpurSgkqZUpmBRwXPf45ltZ7eQp7TiAjFYbG7C84pxL8BvynQ1GYDAAA= X-CMS-MailID: 20231222062136epcas5p18f39a4dea2e66c56c652f29c0dc85a15 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062136epcas5p18f39a4dea2e66c56c652f29c0dc85a15 References: <20231222061313.12260-1-nj.shetty@samsung.com> From: Anuj Gupta This is a prep patch. Allow copy_file_range to work for block devices. Relaxing generic_copy_file_checks allows us to reuse the existing infra, instead of adding a new user interface for block copy offload. Change generic_copy_file_checks to use ->f_mapping->host for both inode_in and inode_out. Allow block device in generic_file_rw_checks. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- fs/read_write.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/fs/read_write.c b/fs/read_write.c index 4771701c896b..f0f52bf48f57 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -1405,8 +1405,8 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, struct file *file_out, loff_t pos_out, size_t *req_count, unsigned int flags) { - struct inode *inode_in = file_inode(file_in); - struct inode *inode_out = file_inode(file_out); + struct inode *inode_in = file_in->f_mapping->host; + struct inode *inode_out = file_out->f_mapping->host; uint64_t count = *req_count; loff_t size_in; int ret; @@ -1708,7 +1708,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + if (!S_ISREG(inode_in->i_mode) && !S_ISBLK(inode_in->i_mode)) + return -EINVAL; + if ((inode_in->i_mode & S_IFMT) != (inode_out->i_mode & S_IFMT)) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || From patchwork Fri Dec 22 06:13:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503059 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92FC8C8CA for ; Fri, 22 Dec 2023 07:32:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="dKhqRgVS" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20231222073234epoutp03b7e851ecb29b6d00c224553c5535232c~jFvudbYfT0074700747epoutp03M for ; Fri, 22 Dec 2023 07:32:34 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20231222073234epoutp03b7e851ecb29b6d00c224553c5535232c~jFvudbYfT0074700747epoutp03M DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230354; bh=5DXoDTvrI2mWMm1I9eg06D4Nr/8IY38fu2o2A+1efIw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dKhqRgVSXcXQyiilTsaCkVy5pRIDAvEC7ErrwQ8xpx1l/xEBHZSgeYNCrucPei9Zo JqGj21zakV4ygG3C30OG52w6UAxMH5o4BsoakpX3f0WARVihgP7yK2q8JknhLTyrmw OYXbGZ8LkWaOxWMummLjDLgppUD0MrivG1m2VSCI= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231222073233epcas5p2c6275650c592fa37f2a425f2acec9f3e~jFvt4zDsg2130021300epcas5p2q; Fri, 22 Dec 2023 07:32:33 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.181]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SxJsq3yRfz4x9Q1; Fri, 22 Dec 2023 07:32:31 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 4F.A9.19369.F8B35856; Fri, 22 Dec 2023 16:32:31 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231222062147epcas5p4353805c045bac87c459f488d0f5b8c86~jEx7MZwT80435204352epcas5p4J; Fri, 22 Dec 2023 06:21:47 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231222062147epsmtrp219873d3dc5f8da5a57edb2fc5bba60ce~jEx7KnPV21637116371epsmtrp2Q; Fri, 22 Dec 2023 06:21:47 +0000 (GMT) X-AuditID: b6c32a50-c99ff70000004ba9-12-65853b8f7f92 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 85.3D.07368.AFA25856; Fri, 22 Dec 2023 15:21:46 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062143epsmtip29f2629f9af58673581da4b45db15f459~jEx3wdbaS0362603626epsmtip2P; Fri, 22 Dec 2023 06:21:43 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 06/12] fs, block: copy_file_range for def_blk_ops for direct block device Date: Fri, 22 Dec 2023 11:43:00 +0530 Message-Id: <20231222061313.12260-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te1BUZRSf797L3bvmyuXh+LFMhhsMgcGyudCHCORozm1sAsfpHyiXHbg8 WvbR7pJoU0GwukosC0rlFg+BInk/xAFcxCB5GQkxrEARTbNgEygKLkwQGstC+d/v/M75fef8 zjeHwl2LOHwqRaFl1QppqoDcRlzr9vMPyAvTsUElfXxUP9CDo0e2VQJ9alzDUfVkHolmuxcA st48C1BpWRGBxm+2YchcVoChK9W3MFTQZQFoetSEoY6JvejymQoCmTv6CTTS/jWJSr6d5qCc u60kqux9gqEx4zRA+fpRDLVaMwG6tlqCo7rZeQL1TXiiO2u9Tq/xmTbTJIe581sjwYwMpjFN VedIprniE+bP5kuAuT6eQTLlhgtOTG7WA5J5ND1BMPM3RknGcLUKMM23P2QWm3YzTdb7WLRz jOxAMitNYNVerCJemZCiSAoXHD0uOSQJDgkSBYhC0asCL4VUzoYLDr8ZHXAkJXV9JQKvD6Sp aetUtFSjEQgjDqiVaVrWK1mp0YYLWFVCqkqsCtRI5Zo0RVKggtXuFwUFvRK8XhgnS+68P0So rjunj9wIzwC9288DLgVpMazp+xk/D7ZRrrQZwH5bjpMjWADwckU7xxEsAVjZcIuzJZmfadmU dABoKarEHIEOg82DmesZiiLpvfD2U8oucKdrcNjWKLLX4HQxDpv/6MXsCTdaAq16C7BjgvaB NvMwacc8ej/s/GGOY38H0kKYN+Vip7l0GLy3Uu7kKHGB/ZeshB3j9Aswq+WrjYEg3cyFdb+O Acekh+HCxQbCgd3gX71XNx3w4eKDDtKBT8IrF78jHeJsAE13TZviSKgbyNswg9N+sL5d6KCf h4UDdZij8Q6Yu2rFHDwPthZv4RdhTX3p5vse0LKcuYkZ+PvD4s2VGgAsX+gARuBlesaQ6RlD pv9blwK8CvBZlUaexMYHq0QBCvbkf98cr5Q3gY078Y9uBdUNa4FdAKNAF4AULnDnKV/OZl15 CdJTp1m1UqJOS2U1XSB4feP5OH9nvHL90BRaiUgcGiQOCQkRh+4LEQl28WZ1RQmudJJUy8pY VsWqt3QYxeVnYAcT9HMetntZsbldhf67BwrknYd8vjz2ud8vndXYTyMD7554R2dQxw7FZh/Z B6mlE88xS3UeGufajEnj9PsfxWHKmc8iwr+JyvMRJh7TF5wyxLho3y5t76z1fhpj22W0LMnc +bW8vPJhnSXWsCexrsQjSWYjz65E+Sq6/xn09T5YHKlddDsXVmOKP/pj0sMvvn9J3xdV6Nxk 2BN57vVJfdwM97F1xNzTlvP4Y8+w9PqS93h839MS73zZckP2UG5V1BR6429G17pCbWfkWfXK xuM7lmWSlpzVOS47bGbeShf5lD1xMkYoxJmeO6MkwgsFQv9YajxxLDtbPe/TM5V1hiMgNMlS kT+u1kj/BUBCJ9ewBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRjHec85Ho/W8rQpvdp9Isksa1Dx0o0Iq9PKyvoQFGIjjxp5WZsr MyVrlbVIlxfIqa3MJtpF28VZXlnW2iy01iql2QWX3VQs1FJntSzo25//7/c8PPBQOLeDCKL2 J6ey0mRxIp/0JWrv8ecuGhWcYpcUORGqtj3A0eDQGIFOqNw4uu7MJdHne18B6mnJBuhyWSmB OlvuYKihLA9DldfvYyjP/Bwgl0ONocauMHTldDmBGhqtBLLfLSGRRuvyRude1JGowjKBoZcq F0AXzjgwVNdzHKDaMQ2Obn0eINDDrpmo3W3xWhvE3FE7vZn27tsEY38sZ3RVZ0lGX36M+aAv Akx9ZxbJXM3J92LOK/pJZtDVRTADTQ6SyTFUAUbfdpT5ppvD6Hr6sO1+u31XxbKJ+w+x0sVr 9vomNPd1EJJ6vzR70+osYJmqBD4UpJfCgfdGXAl8KS5dD6DJ8chrEgRCrbsVn8w8WDnR6z0p KTA4PnbpN6Aokg6DbT8pT+9P1+Hwx00F5hnA6Rs4NJRyPA6PjobvtCJPTdAhcKjhCenJHHoF bG794u1RIL0Y5r6e7ql96JWwd/TqnxO4vxVrr/6vPh1ai3qIye1zocJYjKsArf4Pqf9DlwFW BQJZiSwpPmmfUCJMZg+Hy8RJMnlyfPi+lCQd+PN0QWgd6NZMhJsBRgEzgBTO9+ekLDzJcjmx 4iPprDQlRipPZGVmMJMi+DM4wovFsVw6XpzKHmBZCSv9RzHKJygLU/OqQ3abm/tWlnUctOnc ji2qYHdIxUiL5YC9MmbKsqcFG2rfpJovaoNfFWTGjkYN+y2/YQ5N31E4Q3Nf1FWSLR2xuXjP OGEBTTtFkraATy+pdKOrtdqoBLPdq05bC53guWBPBs+Urr40zdQWudWl9P8umjWqWT8vxzmh wq8dFa1Tx+/KN7QSmzvtt2SFJeMWXKN8ckUyRZC/LaO21zAsLx04F2yy2R9EhibO2RW0KUT7 vYK7jdvuFRe9MebN9rfCoULzx+Vu+jEZV6Oe/yg6Yrg/A1Z0K2wBCYbxExEf7taU57zO3dO4 OZcwRfkZ5TWzbbR2xBo1lLkAT9Ov4ROyBLFQgEtl4l80qWJJYwMAAA== X-CMS-MailID: 20231222062147epcas5p4353805c045bac87c459f488d0f5b8c86 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062147epcas5p4353805c045bac87c459f488d0f5b8c86 References: <20231222061313.12260-1-nj.shetty@samsung.com> For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, or use generic_copy_file_range in case device copy offload capability is absent or the device files are not open with O_DIRECT. Reviewed-by: Hannes Reinecke Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/fops.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/block/fops.c b/block/fops.c index 0abaac705daf..6ca46ea1f358 100644 --- a/block/fops.c +++ b/block/fops.c @@ -747,6 +747,30 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + ssize_t copied = 0; + + if ((in_bdev == out_bdev) && bdev_max_copy_sectors(in_bdev) && + (file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) { + copied = blkdev_copy_offload(in_bdev, pos_in, pos_out, len, + NULL, NULL, GFP_KERNEL); + if (copied < 0) + copied = 0; + } else { + copied = generic_copy_file_range(file_in, pos_in + copied, + file_out, pos_out + copied, + len - copied, flags); + } + + return copied; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -851,6 +875,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) From patchwork Fri Dec 22 06:13:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503060 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B120E566 for ; Fri, 22 Dec 2023 07:32:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="HQdnt7LE" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231222073236epoutp029c33bee55d000b817bce96bdb784f0b5~jFvxDYIYM1875918759epoutp02v for ; Fri, 22 Dec 2023 07:32:36 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231222073236epoutp029c33bee55d000b817bce96bdb784f0b5~jFvxDYIYM1875918759epoutp02v DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230356; bh=p/9yh3tUv4Curw5H7Y3DMkbb1qxhGWe31pfXNHDD36M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HQdnt7LExveo9gWi23t+Xe5XS+IlQxTZ+0nVye9NYm32s2ajX3fTa2m17YiEX9OOs crcsUz5U3MaFCqy4lWL+CpOGotPPSE9qDZ3kwNDKeAH1jRHiYbNrMyq0HHMDG22Q4K xDt+pJvJhlhzDFOmv/8ReXsqH+nzWd4h1r+dpHJE= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231222073235epcas5p29cfc218b2f42d06223c8e55091a2b289~jFvvzs2Qz2132821328epcas5p2w; Fri, 22 Dec 2023 07:32:35 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4SxJst1H3Zz4x9Px; Fri, 22 Dec 2023 07:32:34 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 6E.47.10009.19B35856; Fri, 22 Dec 2023 16:32:34 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20231222062157epcas5p35537549a1610815931e43618e045f3ff~jEyE_mD8h0194101941epcas5p3I; Fri, 22 Dec 2023 06:21:57 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062157epsmtrp106db83d3fe8e2b7e66b731d8cdac586c~jEyE9HsE41657416574epsmtrp1C; Fri, 22 Dec 2023 06:21:57 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-3e-65853b91625e Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 0B.C8.08817.50B25856; Fri, 22 Dec 2023 15:21:57 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062153epsmtip2458d39b115507ab8971489f478b6b9ab~jEyBP8YTh0362603626epsmtip2S; Fri, 22 Dec 2023 06:21:53 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Kanchan Joshi , =?utf-8?q?Javier_Gonz=C3=A1lez?= , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 07/12] nvme: add copy offload support Date: Fri, 22 Dec 2023 11:43:01 +0530 Message-Id: <20231222061313.12260-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta1ATVxTu3d1sAjW6RqsXRGViW6oIJBrojQh1xNZtYRg6Hdsp0ykNZAUE kjQPseIUkIcahrcohKIIaUVAUaCUp5WXUaBSyxAaKiA1VBGpAlVnQKSEQOu/75zvfPd859w5 HJz3PdueEy5TM0qZJJJP2hI1rZudXLI8kxjBT+MYqui4jqOJpzMEOpoxi6OygXQSjbVOAmS+ dgygwqICApmu1WGosSgLQxfK2jF0784UG7XPjZMoq8UI0EivDkNN/c7oXLKeQI1NNwnUU/8d ic7+MMJGKX21JDpveImh3zNGAMo83ouhWnM8QDUzZ3F0aewxgW70r0PdswbWLge6TjfAprsH rxB0zy8aurL0BElX6WPpB1V5gG4wxZF0cVo2i05N+JukJ0b6Cfrx1V6STqsuBXRVZww9VbmB rjSPYwErAiN2hjESKaN0ZGQhcmm4LNSL7/tJkE+Qu4dA6CIUo3f5jjJJFOPF3+MX4PJBeOT8 cviOByWRmvlUgESl4rt571TKNWrGMUyuUnvxGYU0UiFSuKokUSqNLNRVxqh3CAWCbe7zhV9F hL34Q6HQ+x1qLvyVHQfKvLXAhgMpEWyMP0FqgS2HRzUA+PPJM4Q1mARwurkGWINnAP6VocWW JG2Xn+BWognAjOpjuIXgUUkYLNb7aAGHQ1LOsHOOY0mvpspxWHdFaKnHqQQCGsvTSAuxihJD w62BBS1BvQXTU41sC+ZSO2D2aDFueQdSbjB9aKUlbUN5wvvTxSxryUp4M89MWDBObYQJP+Yv +IHUnzYwZXKCZTW6BxqH2hdNr4IPDdVsK7aHo+nJizgaXjhZQlrFiQDq+nTASrwHkzrSF0zg 1GZYUe9mTa+HOR2XMGvj5TB1xrz4PhfWnlnCm2B5RSFpxXbQ+Dx+EdMwM/vW4uLSAMwtbmNl AEfdKwPpXhlI93/rQoCXAjtGoYoKZVTuim0yJvq/Tw6RR1WChXvZ8lEtGL77xLUFYBzQAiAH 56/myrcmMjyuVPLNYUYpD1JqIhlVC3CfX3gmbv9GiHz+4GTqIKFILBB5eHiIxNs9hPy13LGk AimPCpWomQiGUTDKJR3GsbGPw/j1H4ovm6JZdmtsrmKkfYzA6TR+b0N3UFfb7nq/1unWKaJf k0jldx499WY6yCto/ScO866U+sYErGHdHqwsanoh/IxllG48HPgof9mc3fCA7WmCt89ptC/O MDfGEhvc9pdvv45zk7VPDWK7+3NGg++NlNeeK3O7Hj461KB6J/DrvS7HA32+PRA2FBBwfm25 VrMibO8XrX5J8bni25+KTcMJ76OufQZTbOxF598+PgJz1p9rTxuc7XY125fEu6f6B+t7HzTz 3z7yzMQ98LmPQ8+wCMsNrnJ4/a5/tlBfS7w0xBtzlvtH681RdM1+w8Xy3Zs8Sw6uW6YWuPXp TwV/eWeriE+owiTCLbhSJfkX4kWDbrgEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02SaUxTaRSG/e693N6ixEtB58OimEbjpCBLY+KnA0ajwvcDUUP8Q9SxkcsS aekCjMuYYWBEQdkEo9QFkIpCI0irWGQRy1IWBR2HqmgxYhuGIFZgXBgGGQsx+u897/ucN+fH YUjBOLWEiZcncSq5NEFEu1K1LSKf1S6+x7jA0brFqLqrnURj76colJY3TSKdNZdGIy3jANma jwNUcvkihZ411xGo4fJpAlXo2gj0+sUED7XNjNLotMkCkL1PQ6DGfl9UmqGlUENjJ4Ue37lA o+JyOw+dfGKk0VXzZwI9zbMDlH+ij0BG2+8A1U4Vk6hqxEGhjn4h6p02u2z0xnUaKw/3DtRQ +PGDZKyvzKSxQfsb/ttQBHD9s1Qal+UUuODs9Lc0HrP3U9jR1EfjnJuVABu6j+AJ/TKst40S OxZGuQZHcwnxKZwqYMM+17j/nisU2vCD90oe8lKBbkMW4DOQXQNbb7wjs4ArI2DrASwfL3SZ C7xg+XQrOac9YMXnId4clE7A0nErkQUYhmZ9YfcM4/Q9WSMJJ6+nE86BZAsoWPOqBzi3Pdh1 0NxjnW2i2JUwN9vCc2o3dj0sGC4jnUWQDYC5L92dNp/9CQ79WzZ7hOAL0jlkoOdwd9hZZKOc OMmugtWXBE6bZH1g+q3zZB5w13xHab5Rmu+oEkBWAi9OoZbFytRBComc+8VfLZWpk+Wx/vsT ZXow+xZisRE0VL7zNwGCASYAGVLk6Zbo9wcncIuWHjrMqRJ/ViUncGoTEDKU6Ae3DyPZ0QI2 VprEHeA4Baf6mhIMf0kq8Wj9LlVgPeZ0mXfFZYNocGXI+TBD1YdbZmVxdO7GiJq0hm1/ja5b YZl4IIypiJneHXJiZjhJ+7RFNHzbGowk3jvqVj/Z2h4YNnbUUbj/rU/VgIQLixx7c+6jNvJ+ kGgo6EW/dP68tB6PwuBLMXrvw/8U+U1cWLTXb1ISSloy3nyyyemmxIH8nZu9WiLMm67saTyl XGaK1DaLzy6vrb62NkZ5o0v+pw4re/ndwqmcQR+6sT0zqtNuTskQLpwXukVktJR2hUsiGGVr YdcCaa2KqXB0UBd74o83OTK3a/Kj1mr8kkPXOMKLNleF5PB/fe8gtknalva1yn48k9L2HBVP iih1nDRITKrU0v8B4p9TkYUDAAA= X-CMS-MailID: 20231222062157epcas5p35537549a1610815931e43618e045f3ff X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062157epcas5p35537549a1610815931e43618e045f3ff References: <20231222061313.12260-1-nj.shetty@samsung.com> Current design only supports single source range. We receive a request with REQ_OP_COPY_DST. Parse this request which consists of dst(1st) and src(2nd) bios. Form a copy command (TP 4065) trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Reviewed-by: Hannes Reinecke Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 79 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/trace.c | 19 +++++++++ include/linux/blkdev.h | 1 + include/linux/nvme.h | 43 +++++++++++++++++-- 5 files changed, 140 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index 20f46c230885..2f504a2b1fe8 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 8ebdfd623e0f..c398546e4edc 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -765,6 +765,63 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline blk_status_t nvme_setup_copy_offload(struct nvme_ns *ns, + struct request *req, + struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio; + u64 dst_lba = 0, src_lba = 0, n_lba = 0; + u16 nr_range = 1, control = 0, seg = 1; + + if (blk_rq_nr_phys_segments(req) != BLK_COPY_MAX_SEGMENTS) + return BLK_STS_IOERR; + + /* + * First bio contains information about destination and last bio + * contains information about source. + */ + __rq_for_each_bio(bio, req) { + if (seg == blk_rq_nr_phys_segments(req)) { + src_lba = nvme_sect_to_lba(ns, bio->bi_iter.bi_sector); + if (n_lba != bio->bi_iter.bi_size >> ns->lba_shift) + return BLK_STS_IOERR; + } else { + dst_lba = nvme_sect_to_lba(ns, bio->bi_iter.bi_sector); + n_lba = bio->bi_iter.bi_size >> ns->lba_shift; + } + seg++; + } + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.control = cpu_to_le16(control); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + cmnd->copy.nr_range = 0; + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -1007,6 +1064,11 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); break; + case REQ_OP_COPY_DST: + ret = nvme_setup_copy_offload(ns, req, cmd); + break; + case REQ_OP_COPY_SRC: + return BLK_STS_IOERR; default: WARN_ON_ONCE(1); return BLK_STS_IOERR; @@ -1758,6 +1820,21 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_hw_sectors(q, 0); + return; + } + + blk_queue_max_copy_hw_sectors(q, nvme_lba_to_sect(ns, + le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1961,6 +2038,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -4708,6 +4786,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..82c6aef77c31 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 sdlba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "sdlba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + sdlba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 80867c9fd602..24016fbcd409 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1226,6 +1226,7 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) /* maximum copy offload length, this is set to 128MB based on current testing */ #define BLK_COPY_MAX_BYTES (1 << 27) +#define BLK_COPY_MAX_SEGMENTS 2 static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) { diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 44325c068b6a..4afd2cd1a629 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -344,7 +344,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -372,6 +372,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -421,7 +422,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -838,6 +842,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -861,7 +866,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -1038,6 +1044,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1802,6 +1838,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Fri Dec 22 06:13:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503061 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDC0514A86 for ; Fri, 22 Dec 2023 07:32:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="FplPGCKR" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231222073240epoutp017c2b4342b976e7ea88ae0db90a898b36~jFv0Re3jI0228402284epoutp01U for ; Fri, 22 Dec 2023 07:32:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231222073240epoutp017c2b4342b976e7ea88ae0db90a898b36~jFv0Re3jI0228402284epoutp01U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230360; bh=vH0s1GVcNF5IA3K6MXsproiU6tEBb+c/9OtwFqb8YN4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FplPGCKR029A4lfxdOC+GddH0APSg6svbDnNIZR3sS97/Gy9XougBczm2xFGJdEkW deJ6j7Fnybm1uxPVJWhAbed8IshbiwcheX9JU98o97RenB7Fb6RG7Jhbmu20vkWP6b QgNSNp+sGEY2QXshxR5Y3gFONEwHPEwlaSMWNmtM= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20231222073239epcas5p373b8f0edde693b3ef67ca791b2c270db~jFvzYYFg51077210772epcas5p3C; Fri, 22 Dec 2023 07:32:39 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SxJsx6LPqz4x9Pr; Fri, 22 Dec 2023 07:32:37 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F9.64.08567.59B35856; Fri, 22 Dec 2023 16:32:37 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231222062207epcas5p4a5e0db868dd96cf4ed614a0283d11c7e~jEyOfFzYV2962329623epcas5p4a; Fri, 22 Dec 2023 06:22:07 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062207epsmtrp1c653c77027174202dee60229bbf4fedd~jEyOcuX9p1657416574epsmtrp1Y; Fri, 22 Dec 2023 06:22:07 +0000 (GMT) X-AuditID: b6c32a44-617fd70000002177-c5-65853b9595a8 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 6E.93.08755.F0B25856; Fri, 22 Dec 2023 15:22:07 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062204epsmtip2614200aae659e182071c956838100583~jEyK_zSUv0303503035epsmtip2K; Fri, 22 Dec 2023 06:22:03 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 08/12] nvmet: add copy command support for bdev and file ns Date: Fri, 22 Dec 2023 11:43:02 +0530 Message-Id: <20231222061313.12260-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTZxT3u/fSFreOS9H4Uac2dVtEhLZCy8dLF+ngJm4ZhixuhAUaekMJ 0HZ9TJ1s42F5RkDQbZQ36MbDwHgMi4AayGBiRAwDBKNuSHUMAZFJDQiupWXzv9855/f7fuec L4eFc0qZXFa8UkdrlLJEPmMz0d7r4eF1LtBAC2tnBKhpoA9HC89XCJRWsIqjhnv5DDTT+wyg qWuZAFVWlxFo/FoHhrqqCzFU1/Arhgp7RgEyjxgx1D3hiaoyzhOoq/s6gYYvlzJQxY9mJsod MzHQT/1rGLpTYAboTNYIhkxTqQC1r1TgqHFmnkC/TWxHt1b7nd7nUh3Ge0zq1v1mghq+qada 6rMZVOv5b6m/WosB1TmewqBq8oqcqNPpcwxqwTxBUPNXRhhUXls9oFpvnKQWW3ZSLVOzWLhL ZEKQgpbJaQ2PVsaq5PHKuGD+4YjokGixRCjyEvkjPz5PKUuig/nSD8O9QuMTrSvh876UJeqt qXCZVssXHAjSqPQ6mqdQaXXBfFotT1T7qr21siStXhnnraR1ASKhcL/YSoxJUFT3DjHVg2HH LZ3qFNAcmAOcWZD0habMS0QO2MzikJ0ALk08drIHzwAc6sx2VJYAnP2jj9iQzP8+6WB1A5jd dJZpDwwYTJ8bAjmAxWKQnvDGK5ZNsIW8iMOOZpGNg5PlOGyd7MdsBTfyCCwrfrb+KkG+C3Nr anEbZpMB8Ha7zYFldRPA/AeutrQzGQgfL9c42Smu8Hrx1LoUJ3fB9F9KcNv7kLzkDLOqJhn2 TqXwyVwN047d4N/9bQ7MhdP5GQ58DNadrWXYxacANI4Zgb1wEBoG8nFbEzjpAZsuC+zpHfDc QCNmN34Lnl6Zwux5NjSVb+Dd8GJTpaMHdzhqSXVgCl7JvuBYVh6AY8vPsQLAM742kPG1gYz/ W1cCvB6402ptUhwdK1aLlPSx/345VpXUAtbPZK/UBO5UrHn3AIwFegBk4fwtbNW+UzSHLZed +IrWqKI1+kRa2wPE1oWfwblbY1XWO1PqokW+/kJfiUTi6+8jEfG3sWcMZXIOGSfT0Qk0raY1 GzqM5cxNweoqO/a88UVx21Gx2nny0Ue6NS+x6pNcJ2zTTqZ08U/ODumj+m+CZn22pxXNWUbr 7i6/zJRGeflEFpJ+qVyLqvH+qsJ116LhxWdhVZ8fKp+NyhntDk2+2tY/KIogoWRy+ISufQU+ 9XRJiGKXFPwjYBueLlgiY64Kkruaf6B6TKXk9NJD9fSBd2JWsp54t2PmcTkv133fBxmfHucP MIVJES4g8yWfSC6qn4Otd4tjXDkhr0o+frHHb5v+dp3b7qMPFXrOoaitbwZ63PT53uj6gPvd 4d6GC1k/h71dqm/5GoT0Go4EDBR4HxwU+mRNmvra3psNpS2MoBRzBOWWtinoZPh+PqFVyER7 cY1W9i+HOC9KrwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrIIsWRmVeSWpSXmKPExsWy7bCSvC6/dmuqweIeZYv1p44xW3z8+pvF omnCX2aL1Xf72SxeH/7EaPHkQDujxYJFc1ksbh7YyWSxZ9EkJouVq48yWUw6dI3R4unVWUwW e29pWyxsW8JisWfvSRaLy7vmsFnMX/aU3aL7+g42i+XH/zFZ3JjwlNFiYsdVJosdTxoZLbb9 ns9sse71exaLE7ekLc7/Pc7qIOWxc9Zddo/z9zayeFw+W+qxaVUnm8fmJfUeLzbPZPTYfbOB zWNx32RWj97md2weH5/eYvF4v+8qm0ffllWMHptPV3t83iTnsenJW6YA/igum5TUnMyy1CJ9 uwSujEWHL7AXnHOv+L67oIFxo3UXIyeHhICJxPsrj1i7GLk4hAR2M0p82PKZBSIhKbHs7xFm CFtYYuW/5+wQRc1MEhebfwB1cHCwCWhLnP7PARIXEdjBLPFzbTMTSAOzwBpmiS1zeUFsYQF/ idfbpzGC2CwCqhLdi1eADeUVsJK4uO0R2BwJAX2J/vuCIGFOAWuJ578Ws4LYQkAlJ59vZoMo F5Q4OfMJC8R4eYnmrbOZJzAKzEKSmoUktYCRaRWjZGpBcW56brFhgWFearlecWJucWleul5y fu4mRnCsa2nuYNy+6oPeIUYmDsZDjBIczEoivPk6LalCvCmJlVWpRfnxRaU5qcWHGKU5WJTE ecVf9KYICaQnlqRmp6YWpBbBZJk4OKUamJS8Yk/HzgjoEDrOEmWouFrf/hL/7IzqLV0VxaWf m50/V0ts0YxMLin0iVs1UfyhAvf+Qttj2em6fJVPGrZqX78pmvHJa9f9mHWB/0M/XG8rkvPJ 5W+6cJXJvzihuCpZSuCQtmdMhXSz6KyHLFOepcgutK5ivxL7a4FMhrhcsfrkpm+S52+3HPs7 b9VGfsaSl+9P2EXHWaa/WahioDRRsNo7zHGSA9dJkc/3Xp/5brNy19G9kss0D83k3KoY/ydn BuMX6ZypXz3+Z+1aJVZzXmtn0zS5l0//nFLyP8VQWLF0yf7wW75evyozfkun7Jiz+M5dvdtr A8Q/vt84WaXWXmf7spjF9eHqMziWBO9foMRSnJFoqMVcVJwIAHdlVgtkAwAA X-CMS-MailID: 20231222062207epcas5p4a5e0db868dd96cf4ed614a0283d11c7e X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062207epcas5p4a5e0db868dd96cf4ed614a0283d11c7e References: <20231222061313.12260-1-nj.shetty@samsung.com> Add support for handling nvme_cmd_copy command on target. For bdev-ns if backing device supports copy offload we call device copy offload (blkdev_copy_offload). In case of absence of device copy offload capability, we use copy emulation (blkdev_copy_emulation) For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. loop target has copy support, which can be used to test copy offload. trace event support for nvme_cmd_copy. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++- drivers/nvme/target/io-cmd-bdev.c | 71 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 50 ++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/trace.c | 19 +++++++++ 5 files changed, 148 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 39cb570f833d..4e1a6ca09937 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index f11400a908f2..f974858ae5a0 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,18 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -451,6 +463,61 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +static void nvmet_bdev_copy_endio(void *private, int status, + ssize_t copied) +{ + struct nvmet_req *rq = (struct nvmet_req *)private; + u16 nvme_status; + + if (copied == rq->copy_len) + rq->cqe->result.u32 = cpu_to_le32(1); + else + rq->cqe->result.u32 = cpu_to_le32(0); + + nvme_status = errno_to_nvme_status(rq, status); + nvmet_req_complete(rq, nvme_status); +} + +/* + * At present we handle only one range entry, since copy offload is aligned with + * copy_file_range, only one entry is passed from block layer. + */ +static void nvmet_bdev_execute_copy(struct nvmet_req *rq) +{ + struct nvme_copy_range range; + struct nvme_command *cmd = rq->cmd; + ssize_t ret; + off_t dst, src; + + u16 status; + + status = nvmet_copy_from_sgl(rq, 0, &range, sizeof(range)); + if (status) + goto err_rq_complete; + + dst = le64_to_cpu(cmd->copy.sdlba) << rq->ns->blksize_shift; + src = le64_to_cpu(range.slba) << rq->ns->blksize_shift; + rq->copy_len = ((__force size_t)range.nlb + 1) << rq->ns->blksize_shift; + + if (bdev_max_copy_sectors(rq->ns->bdev)) { + ret = blkdev_copy_offload(rq->ns->bdev, dst, src, rq->copy_len, + nvmet_bdev_copy_endio, + (void *)rq, GFP_KERNEL); + } else { + ret = blkdev_copy_emulation(rq->ns->bdev, dst, + rq->ns->bdev, src, rq->copy_len, + nvmet_bdev_copy_endio, + (void *)rq, GFP_KERNEL); + } + if (ret == -EIOCBQUEUED) + return; + + rq->cqe->result.u32 = cpu_to_le32(0); + status = errno_to_nvme_status(rq, ret); +err_rq_complete: + nvmet_req_complete(rq, status); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -469,6 +536,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..0a8337596f0c 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,47 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range = req->cmd->copy.nr_range + 1; + u16 status = 0; + int src, id; + ssize_t len, ret; + loff_t pos; + + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + status = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (status) + break; + + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, COPY_FILE_SPLICE); + pos += ret; + if (ret != len) { + req->cqe->result.u32 = cpu_to_le32(id); + if (ret < 0) + status = errno_to_nvme_status(req, ret); + else + status = errno_to_nvme_status(req, -EIO); + break; + } + } + + nvmet_req_complete(req, status); +} + static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +371,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +423,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 6c8acebe1a1a..b648baeb52cf 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -405,6 +405,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 diff --git a/drivers/nvme/target/trace.c b/drivers/nvme/target/trace.c index bff454d46255..551fdf029381 100644 --- a/drivers/nvme/target/trace.c +++ b/drivers/nvme/target/trace.c @@ -92,6 +92,23 @@ static const char *nvmet_trace_dsm(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvmet_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 sdlba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "sdlba=%llu, nr_range=%u, ctrl=1x%x, dsmgmt=%u, reftag=%u", + sdlba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvmet_trace_common(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -129,6 +146,8 @@ const char *nvmet_trace_parse_nvm_cmd(struct trace_seq *p, return nvmet_trace_read_write(p, cdw10); case nvme_cmd_dsm: return nvmet_trace_dsm(p, cdw10); + case nvme_cmd_copy: + return nvmet_trace_copy(p, cdw10); default: return nvmet_trace_common(p, cdw10); } From patchwork Fri Dec 22 06:13:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503062 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A627D18AE4 for ; Fri, 22 Dec 2023 07:32:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="TSOhTFeF" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20231222073242epoutp04bd564965640090188cdc79c1bdbfdac2~jFv2zdAMg1382513825epoutp04V for ; Fri, 22 Dec 2023 07:32:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20231222073242epoutp04bd564965640090188cdc79c1bdbfdac2~jFv2zdAMg1382513825epoutp04V DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230363; bh=pMi2KJ6tLswDpsWtY8Cv4RkPJ8GSgmmvRnoKG+rEJEY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TSOhTFeFlnbAmcgPAmTSIxpXkzJbWyYVOR/BbHGhrYVoDYjcpgFDMErK79OtWaF1r S/e08ibmE3EDAv/rf/gvr1l0UkSE9DKQC63yTSmD+Wyq9+lCRaWOZkJa/9t9bGeQ9t b4aJUrpKB885cTMXUedwABL2WXe8JTSjrOoXzdxE= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20231222073242epcas5p2e4a4a09ff56f17c4452a1166e77b29c0~jFv2NhXCV1857018570epcas5p2Q; Fri, 22 Dec 2023 07:32:42 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4SxJt10fm4z4x9Pp; Fri, 22 Dec 2023 07:32:41 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F2.57.10009.89B35856; Fri, 22 Dec 2023 16:32:40 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20231222062217epcas5p1263b3a88c9e94933ab44be1df86ccfbe~jEyXpfPu-1003410034epcas5p1v; Fri, 22 Dec 2023 06:22:17 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062217epsmtrp18acad7daa76e95f3898780e78c18a5a2~jEyXoMX6s1657416574epsmtrp1e; Fri, 22 Dec 2023 06:22:17 +0000 (GMT) X-AuditID: b6c32a4a-ff1ff70000002719-50-65853b988906 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 4B.EE.18939.91B25856; Fri, 22 Dec 2023 15:22:17 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062214epsmtip2890709eff5394d61ae1d575ca7ebbd62~jEyUbT6GN0344503445epsmtip2_; Fri, 22 Dec 2023 06:22:13 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 09/12] dm: Add support for copy offload Date: Fri, 22 Dec 2023 11:43:03 +0530 Message-Id: <20231222061313.12260-10-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTdxTH97v39lLYqpcC2w+cWjs1gIO2ULofCnMZZN69EEfiHy5ZqXAH hNI2LRXZS96yMh6iGKkgDAhMIHSFjhWYE3GjgjLisCBsuqEF5xpF2BgLBFhL0fnf93zOOb/z +OWwcW6xmx87RZHOqBUyOZ/0IDovBwQEndmTzwi770Ugw2A/jmbnlwjUcquURPbLcwDZeo8D VFtXTaDx3i4MnW/5EUPlfaMATVn1GLowsQt9WdBAoO8uDBBopLuKRDWNU26oaMxMoibLCoZu lk0BdKLQiiGzLRugzqUaHLXZZwh0ZWITGl62sF6DdJf+lhs9fNtI0CNDWrq9+XOS7mg4Rv/R UQnonvEskq4vOcmii3MfkvTs1ARBz3xvJekSUzOg/2rfQrfbHmCxGw6lRiQzskRGzWMUCcrE FEVSJP/tOGmUNEwiFAWJwtErfJ5ClsZE8qPfiQ16I0XuWAKfd0Qm1zpQrEyj4QtejVArtekM L1mpSY/kM6pEuUqsCtbI0jRaRVKwgknfLRIKQ8IcgfGpySs5s7jqB9+j1l9+JbPAtLcOuLMh JYaGLD1LBzzYXKoHwOzJ6XVjDsCy3OPYE2PpRjmuA+y1lGWrzMW7ALQ/+pd0PsWl8jFYMXPU GUNSu+DVVbYTe1OtOOwyipzxODWOwaGLnZjT4UXtgYaFMZZTE9QOmD1XssY5Dp4zcHG9lgCW /ubpxO4OfG+xnuUK8YQDlTbCqXFqK8z95izufB9SNe6wtWWcdI0WDTsWJ3CX9oJ/WkxuLu0H 75cWrOsMeP7UV6QrOQ9A/ZgeuBx7Yf5g6VoTOBUADd0CF94MKwbbMFfhDbB4yYa5OAeazz3W L8FWQ+16D75wdCF7XdOwYd5CuhZXAqBlcRQrAzz9UwPpnxpI/3/pWoA3A19GpUlLYjRhqhAF k/HkkxOUae1g7S4C3zKDyd8fBfcBjA36AGTjfG+O8uU8hstJlGV+xKiVUrVWzmj6QJhj4Sdw P58EpeOwFOlSkThcKJZIJOLwUImI/wLHnl+dyKWSZOlMKsOoGPXjPIzt7peFbX8u7tNBakwY y/Ng6jd+sol3Laz5Cl0V3LNU5xFo7xDsjPtHUhlxzbh/Y0JT3PVvRZ8VRllj7h/LMv3tP3Lw tHb54PNBX/t75Xjtbtb1yumic36te7dltt6RluzrRx+sVJ/qN84uTFd0LqJLobfTDuvuvFno c+aQ3RMrMp30RdtG+WX53pUtKY0/t0uNipDcxuuvv3iDGxVKzw9+rDt7NXofIJL3dheaV4YE w5n765VugqY+n5ln3l89QJtZytXhdx/m3g3Y6R9/aTKaU/dT/Om8957tPVJgsxbrqox11TGH F6q4MUlf3MwQb2EOTG//sM1jwrD17o7iBssD3uam8i7GZOITmmSZKBBXa2T/AagvnICgBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPIsWRmVeSWpSXmKPExsWy7bCSvK6kdmuqwYqNMhbrTx1jtvj49TeL xeq7/WwWrw9/YrR4cqCd0WLBorksFjcP7GSyWLn6KJPFpEPXGC2eXp3FZLH3lrbFwrYlLBZ7 9p5ksbi8aw6bxfxlT9ktuq/vYLNYfvwfk8WNCU8ZLSZ2XGWy2PGkkdFi2+/5zBbrXr9nsThx S9ri/N/jrA4SHjtn3WX3OH9vI4vH5bOlHptWdbJ5bF5S7/Fi80xGj903G9g8FvdNZvXobX7H 5vHx6S0Wj/f7rrJ59G1ZxejxeZOcx6Ynb5kC+KK4bFJSczLLUov07RK4Mv41fWQuOCJZcfX2 HbYGxmciXYwcHBICJhJ/ryZ2MXJxCAlsZ5T4ve0dWxcjJ1BcUmLZ3yPMELawxMp/z9lBbCGB ZiaJjadCQXrZBLQlTv/nAOkVEdjBLPFzbTMTiMMs8JRJ4uPPb4wgDcIC1hLrv19nBbFZBFQl Gj/1MYHYvEDxppP7mSGO0Jfovy8IEuYECj//tZgVYpeVxMnnm9kgygUlTs58wgJiMwvISzRv nc08gVFgFpLULCSpBYxMqxhFUwuKc9NzkwsM9YoTc4tL89L1kvNzNzGC41graAfjsvV/9Q4x MnEwHmKU4GBWEuHN12lJFeJNSaysSi3Kjy8qzUktPsQozcGiJM6rnNOZIiSQnliSmp2aWpBa BJNl4uCUamDyvDzpJpvQCxYttUijjrW6LRwrEqYus7r+hGmukf9yg8oS4/dswt2sh5++k5Gt f8gqzXVnsfWZLa0ye1S1lb5UhxVLhm123qo+9fKS7i3838V+HU4U1Fdv+sv/TO+qfrDRjib9 lv8qaiZfBBY6P8mdaLg55m/QrOXf29Mcsp6rP7XgXDoxMJCtoCj8h687H/ueGT1ycg1+XZ8Z G1ZfX6ZRUv9IeXvS/v/u4n7X5/JOe6Rkn/ay70nlc6cpi1YKb1YT7zoq4qhg27ihZM1im6Ye /9h0hknPxVRlZsxs9RHg9t8en/5T108sLoKhXUSiYuGm+NsPlSe1XZovwjyx63H2Fh4h5fuJ 6s9Yl7x4pcRSnJFoqMVcVJwIAKujTP5SAwAA X-CMS-MailID: 20231222062217epcas5p1263b3a88c9e94933ab44be1df86ccfbe X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062217epcas5p1263b3a88c9e94933ab44be1df86ccfbe References: <20231222061313.12260-1-nj.shetty@samsung.com> Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 37 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 +++++++ include/linux/device-mapper.h | 3 +++ 3 files changed, 47 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 198d38b53322..a3e1695d111f 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1899,6 +1899,38 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !q->limits.max_copy_sectors; +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1981,6 +2013,11 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + q->limits.max_copy_sectors = 0; + q->limits.max_copy_hw_sectors = 0; + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 23c32cd1f1d8..c98bb3908bd4 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1721,6 +1721,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 772ab4d74d94..d840ed38516c 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -397,6 +397,9 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* copy offload is supported */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Fri Dec 22 06:13:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503063 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1943C22336 for ; Fri, 22 Dec 2023 07:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="Q21mH9Zm" Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20231222073246epoutp02666d9899b24118a10859a3e187b8b231~jFv6G_um91675316753epoutp02O for ; Fri, 22 Dec 2023 07:32:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20231222073246epoutp02666d9899b24118a10859a3e187b8b231~jFv6G_um91675316753epoutp02O DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230366; bh=ve1xdmhje+5EA/lKMdfQIcdO/aY9KnYalMJf/3V4ggw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q21mH9Zm9xMHWN1RohqpyRB/kDqKo80T9NL+ATnKx4Tw5i9a8GaEhAiBgZ4QgAKNs qce2pKdwswSXK1VZ6ZRttNyLGg8V0tNpExZQj2BI4HOtAWp9E9qd33KCq28BlW+VRk sCHrEN+mZZeX3qdh92eXhnUHX1UMCgbYmjTjK4Mg= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20231222073245epcas5p1b3c6e576644a2b573d2d065012ce9e43~jFv5ghx-m2925829258epcas5p1O; Fri, 22 Dec 2023 07:32:45 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4SxJt40yHnz4x9Py; Fri, 22 Dec 2023 07:32:44 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D4.23.09634.B9B35856; Fri, 22 Dec 2023 16:32:44 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20231222062227epcas5p34a8e2395fc03b456b5fdb4b858e01d15~jEygs7JbK0954409544epcas5p3_; Fri, 22 Dec 2023 06:22:27 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062227epsmtrp14c1dc293656af35d18e50f68a873c1d4~jEygr3APY1655116551epsmtrp1C; Fri, 22 Dec 2023 06:22:27 +0000 (GMT) X-AuditID: b6c32a49-eebff700000025a2-ca-65853b9ba4cb Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E5.A3.08755.32B25856; Fri, 22 Dec 2023 15:22:27 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062223epsmtip2fb23343255634a8809e53ce3053828e7~jEydWR4t50300703007epsmtip2R; Fri, 22 Dec 2023 06:22:23 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 10/12] dm: Enable copy offload for dm-linear target Date: Fri, 22 Dec 2023 11:43:04 +0530 Message-Id: <20231222061313.12260-11-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TaVBTVxTufe/l5cEQfIC2F9CWSaedAdnSAl42i1XjE5yBqaXj2FqahlfC AEnIItT+KAi4YAWJtcWwiEgtokNGgpS1ZbHG4AKtxUDaKJ2yUwSxyiiLTQi0/vvOOd93z/nO nUPhrqe4HlSyVMUqpKJUPulINHR5e/uVhuexgUd6caTrvoajh4/nCXTRUkiiya5ZgIbaDwNU UVlGoIH2Jgy1VmowdOHizxjSdN4FaLhPi6E280Z09lAVgVrbjAS601xKojPnh7nomKmRRN8b ljDUf2IYoKIjfRhqHMoGqGH+DI5qJ6cJdN3siXoWDZwod6ZJa+EyPfcuE8ydW2qmruYoyeir vmTG9KcB0zKQRTLnCk5ymOM5D0jm4bCZYKZ/7COZgvoawOhvfME8qnuVqRuawuLW7E2JkLCi RFbhxUrFssRkaVIkP2Z3wtaE4JBAgZ8gFG3ie0lFaWwkf9uuOD9hcqp1G3yv/aJUtTUVJ1Iq +QGbIxQytYr1ksiUqkg+K09MlQfJ/ZWiNKVamuQvZVVhgsDAt4KtxE9SJFf+ngPyWk7mzZJe LAu0E/nAgYJ0EDTf7+XkA0fKlW4B8KzJRNiDWQDLWg7iNpYr/QRAfY9iVVEyWo7ZSW0AdpZU cu1BHga7RopBPqAokt4IbzynbIK19CUcNl0W2Dg4/RSDOdXXga3gRu+ADwaNy3MQ9Btw/OQ1 jk3Lo8Nh+69ONgjpAFh438XGcLBmR5+d49gwj3aBxtNDy0qcfg3mXCnBbc9DWu8AqweGV6xt g3WDs5gdu8EJQz3Xjj3geOGhFZwBL3xdTdrFuQBqTVpgL7wD87oLcdsQOO0Ndc0B9vQGeKq7 FrM3dobH54dW3ufBxvJV/Dq8pKsg7dgd3p3LJu1eGLj00yb7qgoAtPxzlXsCeGlf8KN9wY/2 /84VAK8B7qxcmZbEKoPlAimb8d8fi2VpdWD5Pnx2NgLL4Ix/J8Ao0AkghfPX8mS+uawrL1H0 +QFWIUtQqFNZZScItq67CPdYJ5ZZD0yqShAEhQYGhYSEBIW+HSLgv8KbzCtLdKWTRCo2hWXl rGJVh1EOHllYRk2p4eixz7bs2qOJcvEdnOnfp3s2uWessMNTL4zJ0/vuVU50hGaraXejOHYp YHE9YT6ginmU8m40d/ds5kLKc4Mb0zgu+eZ9STZwiK6PyB3bnLFjlJyomEvasD1cq4tUnHc3 Oq3R/iKf8i/3XJ++D9uyP+Vx0QiveMqxQNh6802fvg7pknHih5GYYL8Pbs9MObff+ktoStc8 eSktPcCUm29u+Ej7tFn46R+ZPoZ7/Rr2Y9PC7Q/FyG06uvrl3sMap+LYeOuFrjvw3Vwy2/yb ZZEnuKqgJWL5e18thLX8rqvodo7606xXTNUKd1Kx8WbLwa3ZXVQYVBTFz4RWmeXf8gmlRCTw wRVK0b/TbozHqAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG+c45Ho+r2Wka+7QwWwW12cqK+jLtQrcTGERB0PojRzvMyuna NLOgtKVW5h0ip2VpKM4y27osXbLmLbOa5aUysqjNebdlXkq6uEnQfw/P73ne94WXwnkvCH/q cHQsq4qWRglIDvGgVjB/2UJRMruitMsL3XnWgCPn6CSByj9kkqi/9htANnMqQNeLrhLonfkR hkxFORgqK6/HUI6lAyB7uxZDjztF6EbKTQKZHjcRqLWqgESFJXZPlPbGSKLSxt8YeptlByj7 fDuGjLYkgB5MFuKoon+YQE875yLrr0aPTX7MI+0HT8badZdgWl/EMXrdBZIx3DzD9BjyAFP9 LpFkijNyPZh0zRDJOO2dBDNc004yGfd0gDE0n2JG9AGM3jaI7Z4l4YTK2KjDx1nV8g0RnMj7 A+NAWeFx4nl+C5YIzMRF4EVBejXMd1zDLgIOxaOrAfw+NOQxDfxgya86fFr7wLLfDs/pkAaD d7pHpkIURdIi2PyHcvm+tBGHP25r3JNwOhWHzX39mKvtQ++AQ5+a3OsIejHszW1wl7n0emh+ PdMlIb0cZn6c7Up4TbmOn8XuG3h0CGxyGEiX5tKzYVOezT0Fp+dDzf18PAvQ2v+Q9j90HWA6 4Mcq1Qq5Qh2sDI5m48VqqUIdFy0XH4pR6IH748KlRvBQ91VsARgFLABSuMCXGxN0juVxZdKE k6wq5qAqLopVW8BcihDwufyedBmPlktj2aMsq2RV/yhGefknYlsiLO3hsWCgtsqkkdXw7ZaE /vjLgdr0NTmSzRmvjn9NNQVcCZL1eYvG/7zn7MdC7o6LOdsWLsjuEYYIjUWx6/3ZSyMrxwxP kiKb1YrTzlVp3qPyL0G5xS9jJgaz94V2eY8EbE2ZIR38UVWnb5ME7d0i26iT9JisOxaFcm7J C3ZfDVsdMm/O25yNkjFtmiqwt7Jk9HOt1TL8zHqsvHPySVty3dqiiUM+4c6KvbvOruOV/Twy UNnavWd7zeDZrdTOvnpz4IH0UkLcfS+FdYSrRLeXNnYXdPguuTY5lpmh3JxkG5i1LksUrE+s Tn7asJiN4IskQn5LfGXhJ7/whjBvZ72AUEdKg4W4Si39C4HaAd1gAwAA X-CMS-MailID: 20231222062227epcas5p34a8e2395fc03b456b5fdb4b858e01d15 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062227epcas5p34a8e2395fc03b456b5fdb4b858e01d15 References: <20231222061313.12260-1-nj.shetty@samsung.com> Setting copy_offload_supported flag to enable offload. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index 2d3e186ca87e..cfec2fac28e1 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Fri Dec 22 06:13:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503064 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F23722EF6 for ; Fri, 22 Dec 2023 07:32:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="u6/pyKiR" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20231222073251epoutp04edfd354d77443d77c636065ddb1b7245~jFv_qU5lb1645016450epoutp043 for ; Fri, 22 Dec 2023 07:32:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20231222073251epoutp04edfd354d77443d77c636065ddb1b7245~jFv_qU5lb1645016450epoutp043 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230371; bh=Q0pzwoO8FGWSOp2naz3g5BT4IBzXcQeY3AEFwmS1OLc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u6/pyKiR4nJilbs/nHNIzIALHBG3JbtDMF4BLoWRbQ8HSC2Ny6pfO3uIivnnbAO3D wewRSM0hc0eFPgPLaj+Xf49wYGe+aY3vHzKbrg4q24eMHWt/RpG4+F5KbNXGwpeoa6 bAu3PPNiHJpFtm4lMFevAXea+l6V48SLDWXOvxZ4= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20231222073250epcas5p14c52bc5a11e8cbb01b1668064dc1999b~jFv9cgZb32275422754epcas5p1e; Fri, 22 Dec 2023 07:32:50 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.174]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SxJt81Fsyz4x9Q7; Fri, 22 Dec 2023 07:32:48 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 7F.B9.19369.0AB35856; Fri, 22 Dec 2023 16:32:48 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231222062237epcas5p4ec8dd02068efc7f0fca133a4241ca308~jEyqMMj4Z0902509025epcas5p4S; Fri, 22 Dec 2023 06:22:37 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20231222062237epsmtrp213b0b000da4a86971afba36caf47b248~jEyqK1Qpd1720617206epsmtrp2L; Fri, 22 Dec 2023 06:22:37 +0000 (GMT) X-AuditID: b6c32a50-9e1ff70000004ba9-46-65853ba0b2af Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F7.D8.08817.D2B25856; Fri, 22 Dec 2023 15:22:37 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062233epsmtip22a1d209c2a01750ffee5de9c57802f9f~jEymvdoyd0303903039epsmtip2u; Fri, 22 Dec 2023 06:22:33 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 11/12] null: Enable trace capability for null block Date: Fri, 22 Dec 2023 11:43:05 +0530 Message-Id: <20231222061313.12260-12-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0zTVxTHd3+/8qNQCz8eZte6MVLDNmDQVkq9vNRFgj91JN3YJG5mUOGX lgBt0wIyZVuZAo43KG7W8Zi6MaCRRxF5VaEGERrGH4Y62GRh0o2CkwFDo6RuLYXN/z73e8/3 nHvOzWHi3hddOcw0eRatkksyuIQ7o+tW4JshDVEFNL9M64JaR2/jaGl1jYG+qLThqOV+BYEW bi0DNDtQBFDDpVoGmhzowVD/pWoMNbUMYajaaAbIMqHFkGEqGH1beIWB+g0jDHS39xsC1X9v cUUl97oJ1Dj8HEM/VVoAqjozgaHu2XyAutbqcXR1YZGB7kxtR+O2YZe9HKpHe9+VGp9uZ1B3 x7KpjuYvCUp/5XNqTn8BUH2TGoK6XH7WhSo79YiglixTDGrxxgRBlXc2A0pvOkmtdPhRHbN/ YmLPD9OjZbQklVb50/IURWqaXBrDPZSQtC8pXMQXhAgi0C6uv1ySScdwY98Rh8SlZdhHwvXP kWRk2yWxRK3m8nZHqxTZWbS/TKHOiuHSytQMpVAZqpZkqrPl0lA5nRUp4PN3htsDk9NlhTfH MGWRT27f2gyhAefIYuDGhKQQLnU1MRzsTfYDaJ07UAzc7bwMYE3tQ8J5eAzgqrmOsemob60E zgsDgI3tFuC0F2BwsT2+GDCZBBkMTf8wHbIvqcNhT7vAEY+TdTjU/zaMOS58yP3weWPbelIG GQAXr5swh5dNRkHr7QMOhCQPVvzq5Yhws6t/PLvs4mA26QVHLsyuO3HyNXjq2kXckR6Sejf4 181eV+c7Y+Hy4DRwsg+cH+7c0Dlw5ZGBcPJx2HTuB8JpPg2g9p52w7AHFoxW4I5H4GQgbO3l OeVXYc3oVcxZ2AOWrc1iTp0Nu+s2eQfUtTZs5N8GzU/yN5iCtutzuHNu5QBOV1tBJfDXvtCQ 9oWGtP+XbgB4M+DQSnWmlE4JVwpC5PTx/z45RZHZAda3JEjcDVrabKFGgDGBEUAmzvVlK946 TXuzUyWfnKBViiRVdgatNoJw+8CrcM7WFIV9zeRZSQJhBF8oEomEEWEiAfdl9kJBbao3KZVk 0ek0raRVmz6M6cbRYDVa5qHxIk92Ws78MT/r5FzhE4/RpaczPjLPT/eUnNirnMzrw6MOj+/X Mzs9z448NBeW8Sz5CXkexfTq1yPPrL3JOp3vAPWuZlQii+f5NcU3jL1Ps+KMb1Ov+29bePDg YP5XiTOaw0G2rdv9guvT423u1GrYmVLXtqaWQYEhWs76mV/F4rV5fRQgRqIlfnjojfOBvzzV FR2J1Ll9Fjuc8KOo1JR7ZNbvpUCx0BxwTWfekWeKjfPa+/fQFsgS7BQffW9flJHTEXInkixt FaedZG2xqrxy26WP5z8weVgGkU/iMe+Pd69+l3wwB2iOsgxvaM2BQyVhOSuv+Pzen7hLOsZl qGUSQRCuUkv+BXEBq4muBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRjHe885Ho+D0XETfNUw2ipoc9oi6O1m9aE6BVF0USiiVju5ldO5 qd1xuWqtWqkV4crmfaVpNU1nmxnTMrNRalaaWcHMTNLMC5S5aknQt9/zvzw88FA4r4UIpZQJ yawmQRYvIDlEVb1gukQiPsHOHTUEo1tPHuFoaHScQOkZEzgqfXueRP313wDyPDAAlJufQ6CO BzUYcuZnYehG6UMMZbleAtTTbsZQbacY5Z0sJJCztolAbfeukshS3OOPzryyk8ja6MXQ64we gDJPtWPI7jkGUNW4BUfl/YMEetwZhp5NNPotD2VqzG/9mWfddwimzZ3C2EqMJFNRmMZ8qsgG jKNDRzIF5y74MSb9AMkM9XQSzOD9dpI5V1kCmIrmw8ywLZyxeb5gG6Zu5SyRs/HKVFYTFb2T ozhZ58bUBv4Bx/gHUgcu0qdBAAXp+dByKwOcBhyKRzsAPDvWQE4aIbB4ogGfZD684e31nwzp Meh+qf8zUBRJi2HzL8qnB9F2HH4v02O+Ak7fxGFlDtfHfHo19FpvEz4m6FlwsLoZ83W59GLY 92iNDyEdBc+/C/QlAv6ovT8K/HzMoxfBpt6Kv+dw6UDYlO0hJrdPh/q7V/AMQJv/s8z/WbkA KwEhrFqrilNppep5Cez+SK1MpU1JiIvcnaiygb9fF4nswFnyNdIFMAq4AKRwQRA3MeI4y+PK ZQcPsZrEHZqUeFbrAmEUIQjmjvWb5Dw6TpbM7mNZNav552JUQKgOm6nau6xxQFh9anGtcESA 5qAz0bONiueZ3lp5+q91S9vNXvHtjVnVhd2t5WUvGiTk0YtGN7lK2n3NT1yYyt51KUQrLa2c Krjks+3wz7SzK6QjqYN2ZZ0ldvxCzCWmxhSrqB5Vb1YGPa17M0MQnARetSmijyaFr71pqrRK Pa7sbc68Yk/6nuG4tD5Dy6Y+qiPnfevAWKrj4HbTim0xkvDrsUufLsxvkQ8tXBCiM65PSuao PxoN9ULOV71cunMgLIK7peCdVTbsyJkmPDGCdd137ZIk89NjwfYi60hrUcQ13eXMPZnsfP7s aYFubdb7UpvziFLo6uqaEpDYR4uiBIRWIZOKcI1W9hvNZliTZAMAAA== X-CMS-MailID: 20231222062237epcas5p4ec8dd02068efc7f0fca133a4241ca308 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062237epcas5p4ec8dd02068efc7f0fca133a4241ca308 References: <20231222061313.12260-1-nj.shetty@samsung.com> This is a prep patch to enable copy trace capability. At present only zoned null_block is using trace, so we decoupled trace and zoned dependency to make it usable in null_blk driver also. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/block/null_blk/Makefile | 2 -- drivers/block/null_blk/main.c | 3 +++ drivers/block/null_blk/trace.h | 2 ++ drivers/block/null_blk/zoned.c | 1 - 4 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/block/null_blk/Makefile b/drivers/block/null_blk/Makefile index 84c36e512ab8..672adcf0ad24 100644 --- a/drivers/block/null_blk/Makefile +++ b/drivers/block/null_blk/Makefile @@ -5,7 +5,5 @@ ccflags-y += -I$(src) obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk.o null_blk-objs := main.o -ifeq ($(CONFIG_BLK_DEV_ZONED), y) null_blk-$(CONFIG_TRACING) += trace.o -endif null_blk-$(CONFIG_BLK_DEV_ZONED) += zoned.o diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 3021d58ca51c..1b40c674f62b 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -11,6 +11,9 @@ #include #include "null_blk.h" +#define CREATE_TRACE_POINTS +#include "trace.h" + #undef pr_fmt #define pr_fmt(fmt) "null_blk: " fmt diff --git a/drivers/block/null_blk/trace.h b/drivers/block/null_blk/trace.h index 6b2b370e786f..91446c34eac2 100644 --- a/drivers/block/null_blk/trace.h +++ b/drivers/block/null_blk/trace.h @@ -30,6 +30,7 @@ static inline void __assign_disk_name(char *name, struct gendisk *disk) } #endif +#ifdef CONFIG_BLK_DEV_ZONED TRACE_EVENT(nullb_zone_op, TP_PROTO(struct nullb_cmd *cmd, unsigned int zone_no, unsigned int zone_cond), @@ -67,6 +68,7 @@ TRACE_EVENT(nullb_report_zones, TP_printk("%s nr_zones=%u", __print_disk_name(__entry->disk), __entry->nr_zones) ); +#endif /* CONFIG_BLK_DEV_ZONED */ #endif /* _TRACE_NULLB_H */ diff --git a/drivers/block/null_blk/zoned.c b/drivers/block/null_blk/zoned.c index 55c5b48bc276..9694461a31a4 100644 --- a/drivers/block/null_blk/zoned.c +++ b/drivers/block/null_blk/zoned.c @@ -3,7 +3,6 @@ #include #include "null_blk.h" -#define CREATE_TRACE_POINTS #include "trace.h" #undef pr_fmt From patchwork Fri Dec 22 06:13:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13503065 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DEA22374E for ; Fri, 22 Dec 2023 07:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="cHTeEEHD" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20231222073254epoutp0140cbc37295f5eb8530e8feab0d8b59ba~jFwBvucDC0228602286epoutp01Y for ; Fri, 22 Dec 2023 07:32:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20231222073254epoutp0140cbc37295f5eb8530e8feab0d8b59ba~jFwBvucDC0228602286epoutp01Y DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1703230374; bh=gdF/wOMKugvH3378GVUkXSQTheU2+tEf4koM7BLSTqc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cHTeEEHD4GNkzWNv6eTZqmKPtPXdJKUZPn/655lcCQOFn2Z6uQmJfSn1Th03+STr/ Ho6KlLBKpF15Aso4B3fAmwrWCpQBvaa7ZfyV7pDeZTYLeKn9JTVcy1UFqlnJibpGBi m9vq38tx4NtPjfT+hZ2qsOct3qDZcOUmZh2V6hRk= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20231222073253epcas5p412bcd55be87879719d1db2fdf5a4af8e~jFwBAl_tL2514625146epcas5p4K; Fri, 22 Dec 2023 07:32:53 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4SxJtD5PGNz4x9Q1; Fri, 22 Dec 2023 07:32:52 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id C9.74.08567.4AB35856; Fri, 22 Dec 2023 16:32:52 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20231222062248epcas5p4623168200a168ae2f895df4221368d66~jEy0En_fu0903809038epcas5p4h; Fri, 22 Dec 2023 06:22:48 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20231222062248epsmtrp1088c62ec57e799bad9ace7097f36eaf2~jEy0DmXzk1655116551epsmtrp1W; Fri, 22 Dec 2023 06:22:48 +0000 (GMT) X-AuditID: b6c32a44-3abff70000002177-e8-65853ba4252f Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 51.FE.18939.73B25856; Fri, 22 Dec 2023 15:22:48 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20231222062244epsmtip26cf587d2c278da63c0ced776226aa9da~jEywWeSb_0303503035epsmtip2O; Fri, 22 Dec 2023 06:22:44 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , dm-devel@lists.linux.dev, Keith Busch , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, nitheshshetty@gmail.com, anuj1072538@gmail.com, gost.dev@samsung.com, mcgrof@kernel.org, Nitesh Shetty , Hannes Reinecke , Damien Le Moal , Anuj Gupta , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v19 12/12] null_blk: add support for copy offload Date: Fri, 22 Dec 2023 11:43:06 +0530 Message-Id: <20231222061313.12260-13-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20231222061313.12260-1-nj.shetty@samsung.com> Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta1BUZRie75yzZxca6gCin4SJa0moXBaX7eOWMlw8TQ5DpcboD9zgCAzs xb1Ixjgutw0hASUcWeMiMAlIy8jN5SKjkFyzrQRcYYDSxQxCAlM0Atr1oPnveZ/ved7bNy8P d6jhOvMSpCpGIRUn8UlbornL3c2jMiCT8a44twXV9XfjaO7xIoHS8pdwdGksj0TTXfMAma99 CdDiTSOOysqLCXTnWguG2svPYKj60g0MnekcBmhySIehqyPb0AVtJYHar/YR6FbrNyQq/XaS i3JuG0h0sWcZQ6b8SYBOZw1hyGBOBah5sRRH+ulZAvWOvInu5WQBZFzq4exyoVt0Y1zaOH6Z oG/dVNP1NSdJuqHyBP2goQjQbXc0JF2RW8ChT6U/JOmWzAkOPTc5QtCzHUMkndtYA+iGgRT6 Uf1bdL15BoukDiQGxjPiWEbhykhjZLEJ0rgg/oefRIdE+4q8BR4CP/Qe31UqljBB/NA9kR7h CUmWDfFdj4qT1BYqUqxU8r3eD1TI1CrGNV6mVAXxGXlsklwo91SKJUq1NM5Tyqj8Bd7ePr4W 4aHE+Kn+blL+c/jnVU9+5WqA3j8b8HiQEsJnJ12ygS3PgWoDcPziAMkG8wAWrpi4LwPzn/OW wOa547e0Uox9aAGw1tQH2CATg8bUGa41L0ltgwMrPKthDVWLw5bLAqsGp57isKKnh7RqHKlg WFISadUQ1Duw+ayWY8V2VADM67tOsO15wbwJeyttY6F//6diVWIP+4rMhBXj1EaY3nQet6aH VJcNHP3uJGAbDYWPHxUSLHaEUz2NqwM4wz/ytKs4GVZ/XUWy5gwAdbd1q+adMLM/D7c2gVPu sK7Vi6U3wMJ+PcYWfh2eWjRjLG8HDSUv8GZYW1dGsng9HF5IXcU0LB/Q4uyucgHMvqvD8oGr 7pWBdK8MpPu/dBnAa8B6Rq6UxDExvnKBlEl++ckxMkk9eH40W0MNwFS67NkJMB7oBJCH89fY ybZnMA52seJjXzAKWbRCncQoO4GvZeGncWenGJnl6qSqaIHQz1soEomEfjtEAv46u+nM4lgH Kk6sYhIZRs4oXvgwno2zBst4Y/vkbColGmwwp3S4rWPCtRlr3/YJRvuznl73SIg4qpsoQIl7 c47L/o1KvrDb3se88CBNrt3b/33auX1RQXM1IXP6wzs2930cUDcsdc8YmVhbWTBTHTHqWP5D zfF9URXL8Zuc/XI0kuLXXP7K8t3yUdsV46eDNiFmv186ME5YeliT05A9RrgUF5btMUzd0F35 27/RKDIcWXm46aAhy/bEhiOOo6KlFP2xQ88iHN9tHOT8KDJNud2f2T0bKFTualcYzx8ujDA5 bZTYj98b14yfHVkprQ796aA+fudn2m5+a9F+nyr13Wxpn1tv7oLW7kCjS1rTE/X9gKrerzTp OWEfBLeP8QllvFiwFVcoxf8B7C2wGr0EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02SfyyUcQDG+77ve++9bt32dlTfcstc9Q86as2+xVJrtTfK0toqrXF450d3 2F1KtBLHSeUulnLidKcTDeUk59c4ScRUNxWSLDeVFaU03fTjWFv/PXs+nz1/PRQuGCFWU7Hx J1h5vEQqInlEfYfIbQPyzGR9lJMA1fQ8wtGX73YCpWvmcXRnRE2iyY6vAI23qQCy9/XjqFRf TKDBNjOGmvV5GKq404mhPMsLgGwDWgy1DHmim1llBGpu6SaQtfEGiXRGGxddfNlAovKuXxh6 pbEBdCV7AEMN4+cBqrfrcFQ9OUWgx0Ou6N3FbID657s424WMWTvCZfrf3CMYa18SU1t5gWRM ZeeY96ZCwDQNppGMITefw1zO+Ewy5sxRDvPFNkQwU60DJJNbVwkY05NUZqZ2DVM7/gnbT4fy /KNYaexJVu69LZwX87HnEZn4bHfy7dm33DRQvTUHOFGQ3gzH0nVYDuBRAvoBgKqnH8hFsAoa 5x/ii9kZVvya4C5KGRi8PjxB5ACKImlP+OQ35ehd6AYczlVlLCzhdDYBn9a3cB2SM70DlpTs dwwR9HpYX5DFcWQ+7QfV3e0LO5D2hurRZY7a6W898dOwoAjorbB7wkQu6stgd+E44cg47QYz 7hfhGkBr/0Pa/1ApwCrBcjZRIYuWRSZuFCskMkVSfLQ4MkFWCxYe4HGgARhr5sUWgFHAAiCF i1z4CV5KVsCPkpxOYeUJYfIkKauwAFeKEK3kr5VeiBLQ0ZIT7HGWTWTl/yhGOa1Ow652Vqzr 1XQFTrl37ak5ONint/i6r+vYZOfdTqraUjK7dAspMcwEufgp248RRYfyW5Pj3hXmlllvxVnV BpVK7rtiZcC0NWX9SS/zDbUHiggQ9ynJMY/Xn9+6iavSi8MvaQvdmyaLOjX+4e1+wm9jISap Mz/C+qKRa7Pq65qb1mRrp30iBWLtcG+ysmff0V0681mj4fKOgvIjJfbpoL0puw/f5bwJUY66 n7nedO2H9HX/R1e9sb1GuIuaWxVYGpB1yll2TBEdlBomDhXFpcZ2KpRh56qfW5bkN89tzznC TRHuEc7WxfirhjPFmw7ofhcJy2fY2JCbZ4uDd+58mOUfLCIUMZKNHrhcIfkDEazyDnADAAA= X-CMS-MailID: 20231222062248epcas5p4623168200a168ae2f895df4221368d66 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20231222062248epcas5p4623168200a168ae2f895df4221368d66 References: <20231222061313.12260-1-nj.shetty@samsung.com> Implementation is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Only request based queue mode will support for copy offload. Added tracefs support to copy IO tracing. Reviewed-by: Hannes Reinecke Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- Documentation/block/null_blk.rst | 5 ++ drivers/block/null_blk/main.c | 97 ++++++++++++++++++++++++++++++- drivers/block/null_blk/null_blk.h | 1 + drivers/block/null_blk/trace.h | 23 ++++++++ 4 files changed, 123 insertions(+), 3 deletions(-) diff --git a/Documentation/block/null_blk.rst b/Documentation/block/null_blk.rst index 4dd78f24d10a..6153e02fcf13 100644 --- a/Documentation/block/null_blk.rst +++ b/Documentation/block/null_blk.rst @@ -149,3 +149,8 @@ zone_size=[MB]: Default: 256 zone_nr_conv=[nr_conv]: Default: 0 The number of conventional zones to create when block device is zoned. If zone_nr_conv >= nr_zones, it will be reduced to nr_zones - 1. + +copy_max_bytes=[size in bytes]: Default: COPY_MAX_BYTES + A module and configfs parameter which can be used to set hardware/driver + supported maximum copy offload limit. + COPY_MAX_BYTES(=128MB at present) is defined in fs.h diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 1b40c674f62b..e6f476ad45a6 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -160,6 +160,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static unsigned long g_copy_max_bytes = BLK_COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, ulong, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -412,6 +416,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -553,6 +558,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -659,7 +665,8 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," - "zone_nr_conv,zone_offline,zone_readonly,zone_size\n"); + "zone_nr_conv,zone_offline,zone_readonly,zone_size," + "copy_max_bytes\n"); } CONFIGFS_ATTR_RO(memb_group_, features); @@ -725,6 +732,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1274,6 +1282,81 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline int nullb_setup_copy(struct nullb *nullb, struct request *req, + bool is_fua) +{ + sector_t sector_in = 0, sector_out = 0; + loff_t offset_in, offset_out; + void *in, *out; + ssize_t chunk, rem = 0; + struct bio *bio; + struct nullb_page *t_page_in, *t_page_out; + u16 seg = 1; + int status = -EIO; + + if (blk_rq_nr_phys_segments(req) != BLK_COPY_MAX_SEGMENTS) + return status; + + /* + * First bio contains information about destination and last bio + * contains information about source. + */ + __rq_for_each_bio(bio, req) { + if (seg == blk_rq_nr_phys_segments(req)) { + sector_in = bio->bi_iter.bi_sector; + if (rem != bio->bi_iter.bi_size) + return status; + } else { + sector_out = bio->bi_iter.bi_sector; + rem = bio->bi_iter.bi_size; + } + seg++; + } + + trace_nullb_copy_op(req, sector_out << SECTOR_SHIFT, + sector_in << SECTOR_SHIFT, rem); + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + chunk = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || + is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, chunk); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= chunk; + sector_in += chunk >> SECTOR_SHIFT; + sector_out += chunk >> SECTOR_SHIFT; + } + + status = 0; +err: + spin_unlock_irq(&nullb->lock); + return status; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1283,13 +1366,16 @@ static int null_handle_rq(struct nullb_cmd *cmd) sector_t sector = blk_rq_pos(rq); struct req_iterator iter; struct bio_vec bvec; + bool fua = rq->cmd_flags & REQ_FUA; + + if (op_is_copy(req_op(rq))) + return nullb_setup_copy(nullb, rq, fua); spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(req_op(rq)), sector, - rq->cmd_flags & REQ_FUA); + op_is_write(req_op(rq)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -2074,6 +2160,9 @@ static int null_validate_conf(struct nullb_device *dev) return -EINVAL; } + if (dev->queue_mode == NULL_Q_BIO) + dev->copy_max_bytes = 0; + return 0; } @@ -2193,6 +2282,8 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_hw_sectors(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index 929f659dd255..e82e53a2e2df 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -107,6 +107,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */ diff --git a/drivers/block/null_blk/trace.h b/drivers/block/null_blk/trace.h index 91446c34eac2..2f2c1d1c2b48 100644 --- a/drivers/block/null_blk/trace.h +++ b/drivers/block/null_blk/trace.h @@ -70,6 +70,29 @@ TRACE_EVENT(nullb_report_zones, ); #endif /* CONFIG_BLK_DEV_ZONED */ +TRACE_EVENT(nullb_copy_op, + TP_PROTO(struct request *req, + sector_t dst, sector_t src, size_t len), + TP_ARGS(req, dst, src, len), + TP_STRUCT__entry( + __array(char, disk, DISK_NAME_LEN) + __field(enum req_op, op) + __field(sector_t, dst) + __field(sector_t, src) + __field(size_t, len) + ), + TP_fast_assign( + __entry->op = req_op(req); + __assign_disk_name(__entry->disk, req->q->disk); + __entry->dst = dst; + __entry->src = src; + __entry->len = len; + ), + TP_printk("%s req=%-15s: dst=%llu, src=%llu, len=%lu", + __print_disk_name(__entry->disk), + blk_op_str(__entry->op), + __entry->dst, __entry->src, __entry->len) +); #endif /* _TRACE_NULLB_H */ #undef TRACE_INCLUDE_PATH