From patchwork Sat May 11 05:52:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13662283 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6553A41 for ; Sat, 11 May 2024 06:30:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.24 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409051; cv=none; b=KaR0ZgwVTE1eD5M4xbI1t/NYkx1hjWtWsRJyPvpZeBU6EgFk+niviL0hmwiGjPWeLEIJzTe084vzFKaGLd4sVdW/9QdMI4jhJsFl0Gv593lNCv4g37BjLQ9LpfJogl0w8TC1zuPEPnFt9RmBE4xeAlcACE2WFmk699OW6EKAWfU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409051; c=relaxed/simple; bh=nwvNmN5Gee5PTeE9GxoI+Qf5rwANgehvwTJo1UQZ7Co=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=nr60QcxRbH6b6Ij9VrT3+N7GEU1fIrdxS8gniDrTsLBXW6hUeSqhb2RV+jOP4JWCBZ2DT5jcVfcFcq26G55IMja8AlrwNNu8rwl4awkLvBVxRW5o3a7M+eTZJq0dgUQYN5p0iQXt1Wt3iykxlxf3HHWjx5tuiW4GKi/BdNX1FJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=H9bWYAPV; arc=none smtp.client-ip=203.254.224.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="H9bWYAPV" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20240511063040epoutp01e908103276350dd6c034b73bc476ef77~OW28J2q9j1876718767epoutp01T for ; Sat, 11 May 2024 06:30:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20240511063040epoutp01e908103276350dd6c034b73bc476ef77~OW28J2q9j1876718767epoutp01T DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715409040; bh=nFNDFrMBrJOO6iC/RO0A43jv+b7q7YeIenXeJG1B8ck=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H9bWYAPVge/vIIHJ1w93HPPJPsN1ALNanp/f07Y7IP41K4QC3LAvTSMK/jvGudD60 R9sgi7wp4vydmf4XA+9BHuOK5pprf7Euzp6k4j+7Xkc8HBKAp4wVqHi/lda2FlAVOt Gw/7dHlipX1pw3OQZgLWodEgJz+7Zpr497acM2jo= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20240511063040epcas5p3bf830e9688047a3bcf9f87282e010270~OW272tzww0244302443epcas5p3n; Sat, 11 May 2024 06:30:40 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.181]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4VbwqL43WZz4x9Pv; Sat, 11 May 2024 06:30:38 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 89.35.09665.E801F366; Sat, 11 May 2024 15:30:38 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240511055243epcas5p291fc5f72baf211a79475ec36682e170d~OWVz8WDpu2182521825epcas5p2z; Sat, 11 May 2024 05:52:43 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240511055243epsmtrp12f48971b06dfa58b49a06c64b11479f8~OWVz7qlWM2071620716epsmtrp1S; Sat, 11 May 2024 05:52:43 +0000 (GMT) X-AuditID: b6c32a4b-829fa700000025c1-82-663f108e55d2 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 35.33.08390.BA70F366; Sat, 11 May 2024 14:52:43 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240511055242epsmtip1ac7cd7a434f15bb496e157f5688b9a66~OWVzBX5Rc2084420844epsmtip1Z; Sat, 11 May 2024 05:52:42 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v2 1/4] io_uring/rsrc: add hugepage buffer coalesce helpers Date: Sat, 11 May 2024 13:52:26 +0800 Message-Id: <20240511055229.352481-2-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240511055229.352481-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprBJsWRmVeSWpSXmKPExsWy7bCmpm6fgH2awZy58hZzVm1jtFh9t5/N 4vTfxywWNw/sZLJ413qOxeLo/7dsFr+67zJabP3yldXi2V5Oi7MTPrA6cHnsnHWX3ePy2VKP vi2rGD0+b5ILYInKtslITUxJLVJIzUvOT8nMS7dV8g6Od443NTMw1DW0tDBXUshLzE21VXLx CdB1y8wBOkdJoSwxpxQoFJBYXKykb2dTlF9akqqQkV9cYquUWpCSU2BSoFecmFtcmpeul5da YmVoYGBkClSYkJ0x81kra8FFuYqV0+YzNzBekOhi5OSQEDCR2HfrEVsXIxeHkMBuRonbT24y giSEBD4xSqxrL4FIfGOUOHNyKTtMx8LeU+wQib2MEsvunYdyfjFKfG77CNbOJqAj8XvFLxYQ W0RAW+L146ksIEXMAksYJXZ1LgcrEhbwljh/4R4TiM0ioCpx7M96MJtXwFZi1/yNTBDr5CX2 HzzLDGJzCthJHH7ZzgZRIyhxcuYTsAXMQDXNW2czgyyQEPjJLnFnxx82iGYXiTunJ7JC2MIS r45vgfpBSuJlfxuQzQFkF0ssWycH0dvCKPH+3RxGiBpriX9X9rCA1DALaEqs36UPEZaVmHpq HRPEXj6J3t9PoO7kldgxD8ZWlbhwcBvUKmmJtRO2MkPYHhKLf+5igoTvREaJni1pExgVZiF5 ZxaSd2YhbF7AyLyKUTK1oDg3PbXYtMA4L7UcHsvJ+bmbGMFJVMt7B+OjBx/0DjEycTAeYpTg YFYS4a2qsU4T4k1JrKxKLcqPLyrNSS0+xGgKDO+JzFKiyfnANJ5XEm9oYmlgYmZmZmJpbGao JM77unVuipBAemJJanZqakFqEUwfEwenVAPToganGhWBX6l/dZYXvVv0uT2lO2UBF++yrA36 YT9DHVeUvquyCuB5vkf4ui2f/82XO1LYxSvMul+cvu+UVVRql8206+57Zv8Mv5N7GSLWbWoR MbE1Ouz07WvjLb/aoF37pP36xXtZZQ8sn5PvdlCnV1b8hv3BL/y6dx15O1aurN+8hyn3reK6 tHQuizWS7F0Cz/fv/PzmU96+bS+ecxcrT1pwaceJQ8wn4jYp/xFz3fZKdgZP0NfHj5csvni2 QjXAySPNplro6qu9+aUzOvQmT/kxjZU/0INz0qrDHouOh1uI3TUwcGw+yrW5epur29XwDVpr Ch5Z8W9U6P12USdY+rlzX1bxvHa9xNM2t2OUWIozEg21mIuKEwFJisv5KwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsWy7bCSnO5qdvs0g9uTzSzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlTHzWStrwUW5ipXT5jM3MF6Q6GLk5JAQMJFY2HuK HcQWEtjNKHH+cCZEXFqi41ArO4QtLLHy33Mgmwuo5gejxP2Na8ASbAI6Er9X/GLpYuTgEBHQ lWi8qwBSwyywilHi6vt2VpAaYQFvifMX7jGB2CwCqhLH/qwHs3kFbCV2zd/IBLFAXmL/wbPM IDangJ3E4ZftbBAH2UqcmnqGGaJeUOLkzCcsIDYzUH3z1tnMExgFZiFJzUKSWsDItIpRMrWg ODc9t9iwwCgvtVyvODG3uDQvXS85P3cTIzjItbR2MO5Z9UHvECMTB+MhRgkOZiUR3qoa6zQh 3pTEyqrUovz4otKc1OJDjNIcLErivN9e96YICaQnlqRmp6YWpBbBZJk4OKUamK7UPi5ate4X /x/Db0XsapL/awTTtXmNZE9mZ7xrPN33+ENpo66Tiol0hfOmR+c22HA47q7b6/ZRpfLR3XvJ v0+LH7h5IEk6l0lYqm//1Fu1jI5rz79K+1UZt/LmxTP7xf48OfFNaUlzjqN55+3PF1bmznzi ejB75U63rvf3OfZPPbR1Rq/hKYXSPF3/9X1GPx5NEcg5IpkcpLw5J0r/9ZbdDYfuP1pk6LBz wwF+bYGZz5Wex1zisXpuvmiBa6T1TauVB3kYvjS/2Kx/9dzrebe8WnwSxH97eH5hX7/g71+z JF6r6nwjK8F14TKWRb4Vn6dHuUc/eFB1PtPxRvz7FYuOaZzj/DbRyfTD75JZh/4qsRRnJBpq MRcVJwIAvDMb9OECAAA= X-CMS-MailID: 20240511055243epcas5p291fc5f72baf211a79475ec36682e170d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240511055243epcas5p291fc5f72baf211a79475ec36682e170d References: <20240511055229.352481-1-cliang01.li@samsung.com> This patch introduces helper functions to check whether a buffer can be coalesced or not, and gather folio data for later use. The coalescing optimizes time and space consumption caused by mapping and storing multi-hugepage fixed buffers. A coalescable multi-hugepage buffer should fully cover its folios (except potentially the first and last one), and these folios should have the same size. These requirements are for easier later process, also we need same size'd chunks in io_import_fixed for fast iov_iter adjust. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 10 +++++++ 2 files changed, 88 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 65417c9553b1..d08224c0c5b0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,84 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + struct folio *folio = page_folio(pages[0]); + unsigned int count = 1; + int i; + + data->nr_pages_mid = folio_nr_pages(folio); + if (data->nr_pages_mid == 1) + return false; + + data->folio_shift = folio_shift(folio); + data->folio_size = folio_size(folio); + data->nr_folios = 1; + /* + * Check if pages are contiguous inside a folio, and all folios have + * the same page count except for the head and tail. + */ + for (i = 1; i < nr_pages; i++) { + if (page_folio(pages[i]) == folio && + pages[i] == pages[i-1] + 1) { + count++; + continue; + } + + if (data->nr_folios == 1) + data->nr_pages_head = count; + else if (count != data->nr_pages_mid) + return false; + + folio = page_folio(pages[i]); + if (folio_size(folio) != data->folio_size) + return false; + + count = 1; + data->nr_folios++; + } + if (data->nr_folios == 1) + data->nr_pages_head = count; + + return true; +} + +static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + int i, j; + + if (nr_pages <= 1 || + !__io_sqe_buffer_try_coalesce(pages, nr_pages, data)) + return false; + + /* + * The pages are bound to the folio, it doesn't + * actually unpin them but drops all but one reference, + * which is usually put down by io_buffer_unmap(). + * Note, needs a better helper. + */ + if (data->nr_pages_head > 1) + unpin_user_pages(&pages[1], data->nr_pages_head - 1); + + j = data->nr_pages_head; + nr_pages -= data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + unsigned int nr_unpin; + + nr_unpin = min_t(unsigned int, nr_pages - 1, + data->nr_pages_mid - 1); + if (nr_unpin == 0) + break; + unpin_user_pages(&pages[j+1], nr_unpin); + j += data->nr_pages_mid; + nr_pages -= data->nr_pages_mid; + } + + return true; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c032ca3436ca..b2a9d66b76dd 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -50,6 +50,16 @@ struct io_mapped_ubuf { struct bio_vec bvec[] __counted_by(nr_bvecs); }; +struct io_imu_folio_data { + /* Head folio can be partially included in the fixed buf */ + unsigned int nr_pages_head; + /* For non-head/tail folios, has to be fully included */ + unsigned int nr_pages_mid; + unsigned int nr_folios; + unsigned int folio_shift; + size_t folio_size; +}; + void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); From patchwork Sat May 11 05:52:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13662282 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77BE91C01 for ; Sat, 11 May 2024 06:30:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409049; cv=none; b=Nydk05A9ZhcMaNWoRjy27mAZ3G1anucHMdWkt7Qn3FD5vqDBgW+n2LnS/33x+qENi43PpwAz3UzG2ESW1av0BASpKaq/w97ZzOyA1LQXAAbku5JZyV8tDxnvW+bLxgIkHSZddOIj7Me3v4uSuMPnq2temmLT7wrB58WGAODcFLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409049; c=relaxed/simple; bh=v98Mo+MpYBU0qzaAkf44WS1u6iYjnUbvkJq6cTR5f6I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=IxkVV/AkltsaPCL5EikWKUHvFQS0bcbqNc4qCzfaDguo48pZGIvZUe/ecLlzJCWcMGNX0YIRE1NZ9dKU/SbdKV0yZjl8gkPopb6sE/vz5QL0noTpUz4ur8Uux1IAwKQ/VLXhcdriqP2qXRzS0q+fFANKpts2FECB3pPvHM6d/X0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=fM8DoD9D; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="fM8DoD9D" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240511063044epoutp022fdad395a280e8ec40c2fddbf6b5f53a~OW2-jOdW11125511255epoutp02a for ; Sat, 11 May 2024 06:30:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240511063044epoutp022fdad395a280e8ec40c2fddbf6b5f53a~OW2-jOdW11125511255epoutp02a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715409044; bh=9J3qY+gr/EQFxqtdizhx5LUqyyO9iMd1OdFM5PunXLE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fM8DoD9Du8yVIjmVrqaMqlBlTyzTANe0FlueWK+TplUjKmpcl0l5Fbn57JwcqvaBB 79ftzGfBBJWJafE3UpORI9Jb0NI/ZPyLKV2OwF4LGYcBOOpUyu4HPOzlZyjv9XqoRY ZY8hDuIJrtA/7fhxXRgwxsmWv0c2Jkp0hkgOStVI= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240511063043epcas5p2a6667af7b215d1b1e659daabbc522795~OW2-F_76_3223632236epcas5p2B; Sat, 11 May 2024 06:30:43 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.179]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4VbwqP6ntnz4x9Pr; Sat, 11 May 2024 06:30:41 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 62.82.08600.1901F366; Sat, 11 May 2024 15:30:41 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20240511055245epcas5p407cdbc005fb5f0fe2d9bbb8da423ff28~OWV1c4NLN3151631516epcas5p4I; Sat, 11 May 2024 05:52:45 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240511055245epsmtrp19da6b7953e41a8a34bf71b6600bcc92a~OWV1cM7FX2071620716epsmtrp1T; Sat, 11 May 2024 05:52:45 +0000 (GMT) X-AuditID: b6c32a44-921fa70000002198-b7-663f1091369d Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 99.C2.19234.DA70F366; Sat, 11 May 2024 14:52:45 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240511055244epsmtip110a6ec42fc837cc2ac8a55e2b14d6a56~OWV0gUinR0560305603epsmtip1a; Sat, 11 May 2024 05:52:44 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v2 2/4] io_uring/rsrc: store folio shift and mask into imu Date: Sat, 11 May 2024 13:52:27 +0800 Message-Id: <20240511055229.352481-3-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240511055229.352481-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJJsWRmVeSWpSXmKPExsWy7bCmuu5EAfs0g0f/TC3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITvj1PoVzAXPeCr6rzUyNjDO5upi5OSQEDCRmNa0hb2LkYtDSGA3o8S73U1s EM4nRoneB5+Y4Jyuy9NYYFo+bnsAldjJKDH1dyNU/y9Gie5DF1lBqtgEdCR+r/gF1iEioC3x +vFUFpAiZoEljBK7OpczgiSEBbwkNm6+CDSKg4NFQFVi23pekDCvgK3E+Z4DbBDb5CX2HzzL DGJzCthJHH7ZzgZRIyhxcuYTsPnMQDXNW2czg8yXEPjILnH//1cmiGYXiXerIJolBIQlXh0H +RTElpJ42d/GDrJXQqBYYtk6OYjeFkaJ9+/mMELUWEv8u7KHBaSGWUBTYv0ufYiwrMTUU+uY IPbySfT+fgK1ildixzwYW1XiwsFtUKukJdZO2Ap1gofEp4MXGCGBNZFR4vLKF2wTGBVmIfln FpJ/ZiGsXsDIvIpRMrWgODc9Ndm0wDAvtRwezcn5uZsYwWlUy2UH4435//QOMTJxMB5ilOBg VhLhraqxThPiTUmsrEotyo8vKs1JLT7EaAoM74nMUqLJ+cBEnlcSb2hiaWBiZmZmYmlsZqgk zvu6dW6KkEB6YklqdmpqQWoRTB8TB6dUA5NuT5jqDt8Gp1cXZzC+O785ILLr6q72h5MV3zR1 8B1MkZ1zbV7n9bJ7V47NVazIcXWZtbMg+cU3R7clj+MuW+YkZsxmXb5Qfwsjo6rBZ4moVXzn ueedOquufjdE+GPs8+pSvcsXU0U1nmeYz1zydc6fyQ/dNi+6bnDtS13qdrfSTIMMFXPXSfO2 WH6WCo9LbgvX2rFuF6N/go6A9kKuU28W3F1zesud/jX1ycsmpdbGvrnnONnC7+Q3/5D4G87N pZotogKRD7MT677f3Xr+Jp/5G883TXcerj6ftLmjqfHtjrTMDc/ELx++/vxhW0fxkXsPc1QO PHy6L4onalvkcleWM29d9qd7m3x5vFPAhiNNiaU4I9FQi7moOBEAyqq9BywEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFLMWRmVeSWpSXmKPExsWy7bCSnO5advs0g6fvZS3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlXFq/Qrmgmc8Ff3XGhkbGGdzdTFyckgImEh83PaA qYuRi0NIYDujxJR1y1kgEtISHYda2SFsYYmV/56zQxT9YJTY0/AeLMEmoCPxe8UvoAYODhEB XYnGuwogNcwCqxglrr5vZwWpERbwkti4+SITSA2LgKrEtvW8IGFeAVuJ8z0H2CDmy0vsP3iW GcTmFLCTOPyyHSwuBFRzauoZZoh6QYmTM5+A3cYMVN+8dTbzBEaBWUhSs5CkFjAyrWIUTS0o zk3PTS4w1CtOzC0uzUvXS87P3cQIDm6toB2My9b/1TvEyMTBeIhRgoNZSYS3qsY6TYg3JbGy KrUoP76oNCe1+BCjNAeLkjivck5nipBAemJJanZqakFqEUyWiYNTqoFpYc+hz85/Fygavz4R dObl7rux8X+6F9dtXvtjf8ffT+bRZczK5bL2brsnrX7u5m/fYxoXMkFHa5m7UGD2hhMMmR3b UvSbMvi2i378NLWfYzd7h1i0U5mQkvLhunt3i77vV55aLVLTPNVJ55NUfvNWA38jE68ZmguL oubfne3g/2HSmpolzxtuPvhlMLlLLuhi5+/wmjK1PcEnbZY+Z73FzBoh6t+wMLOfx6BQ357t f5PjfsHnX9burim++XeG6VSf7fGL8sIKcj9NUNZ/4uDiYsrIvK9reop7i9TWxgmHtqdzbmW9 d4194pP6HS6zmaQDWNNZ7zcGT83Rmf9Mw7j4oeHvp+/W5W6tnXx7zy4lluKMREMt5qLiRAC6 OQ173QIAAA== X-CMS-MailID: 20240511055245epcas5p407cdbc005fb5f0fe2d9bbb8da423ff28 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240511055245epcas5p407cdbc005fb5f0fe2d9bbb8da423ff28 References: <20240511055229.352481-1-cliang01.li@samsung.com> Store the folio shift and folio mask into imu struct and use it in iov_iter adjust, as we will have non PAGE_SIZE'd chunks if a multi-hugepage buffer get coalesced. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 6 ++++-- io_uring/rsrc.h | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d08224c0c5b0..578d382ca9bc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1015,6 +1015,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->ubuf = (unsigned long) iov->iov_base; imu->ubuf_end = imu->ubuf + iov->iov_len; imu->nr_bvecs = nr_pages; + imu->folio_shift = PAGE_SHIFT; + imu->folio_mask = PAGE_MASK; *pimu = imu; ret = 0; @@ -1153,12 +1155,12 @@ int io_import_fixed(int ddir, struct iov_iter *iter, /* skip first vec */ offset -= bvec->bv_len; - seg_skip = 1 + (offset >> PAGE_SHIFT); + seg_skip = 1 + (offset >> imu->folio_shift); iter->bvec = bvec + seg_skip; iter->nr_segs -= seg_skip; iter->count -= bvec->bv_len + offset; - iter->iov_offset = offset & ~PAGE_MASK; + iter->iov_offset = offset & ~imu->folio_mask; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index b2a9d66b76dd..93da02e652bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -46,7 +46,9 @@ struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; unsigned int nr_bvecs; + unsigned int folio_shift; unsigned long acct_pages; + unsigned long folio_mask; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Sat May 11 05:52:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13662284 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA801C01 for ; Sat, 11 May 2024 06:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409058; cv=none; b=K4TGib3L2qscfW01mHz+40POb28DfNt1Dydve2MYtfWEoYUJrg70u02lVOvr1wt9gCWUvKNJ9oLwkSi/7SJLX9/JKdcwBLv8qXa7mzBbkC06lmP7WdTGzPO1YCghGeDnZSbpXXcnIkSEzH7M/FyT/RoiTF7oAq0/yKrAzkZUsxI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409058; c=relaxed/simple; bh=qQDVbo2zNhRgCypeLLC78vrJ0ywWDK++kIZPUGQTr1w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=NtEC8NaN4GdZK0WdkR6OMBCp3X+T1Zw0v9wJnXJjWxsN3XiEavAUzlrGBDEWSEIIKgGJM/bG2P0mWm3TfEu6cTnot3TJmhk3vbgNG5uC8FwY6CvLL8OqpnoznUrlcUyGrYqKYooMEI8CNUHatX9pZILHFo4ceV2dP1Mzl/A2iCg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=a612tyK/; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="a612tyK/" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240511063047epoutp042401555103319cabdeb26d2aedd32bdd~OW3C6OZ1v1078310783epoutp04y for ; Sat, 11 May 2024 06:30:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240511063047epoutp042401555103319cabdeb26d2aedd32bdd~OW3C6OZ1v1078310783epoutp04y DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715409047; bh=+bAmOvxlNrI4OUAx/tDTCXzBPY5PuA60g2e4gjWolnY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a612tyK/cL7FG3sp4mWHJoWOLCB/8g/6TXWzCUadq89x7nkf8IrFkKqxGwqGb/5zg OLGPwmOzszbvaGPkuiAMWClZTSIgmDVofLP8sCIl6qjOqyIR9GszlAQQdBWhvFK7tm /obr3MwVpZRWSeL1OtzPIEaoPniUFM0JZE+NoQcM= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20240511063047epcas5p4569bbff7478877081d1c62d3c8ccad3d~OW3CmsrCG2475724757epcas5p4P; Sat, 11 May 2024 06:30:47 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.179]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4VbwqT3GxQz4x9Pp; Sat, 11 May 2024 06:30:45 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 8E.F5.09666.5901F366; Sat, 11 May 2024 15:30:45 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240511055247epcas5p2a54e23b6dddd11dda962733d259a10af~OWV3BQgr93263732637epcas5p2x; Sat, 11 May 2024 05:52:47 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240511055247epsmtrp25b1a1dd968f2e16627343bfc0a013f60~OWV3Ajb_k2153021530epsmtrp2N; Sat, 11 May 2024 05:52:47 +0000 (GMT) X-AuditID: b6c32a49-cefff700000025c2-04-663f10956289 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 06.33.08390.FA70F366; Sat, 11 May 2024 14:52:47 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240511055246epsmtip1d8d50fa25572c791a6b8a2f42aaa28a5~OWV2Cl48I2084420844epsmtip1a; Sat, 11 May 2024 05:52:46 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v2 3/4] io_uring/rsrc: add init and account functions for coalesced imus Date: Sat, 11 May 2024 13:52:28 +0800 Message-Id: <20240511055229.352481-4-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240511055229.352481-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprGJsWRmVeSWpSXmKPExsWy7bCmhu5UAfs0g607hS3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITtjyd5upoKXUhU9uy+wNTDOFO1i5OCQEDCRWHKApYuRi0NIYDejxIP7L9kh nE+MEh8PPmeFcL4xSqyefom5i5ETrOPqpx3sILaQwF5GiSmr+SGKfjFKHP4wgwUkwSagI/F7 xS8wW0RAW+L146lgO5gFljBK7OpczgiSEBaIkvj5YgXYVBYBVYmjqzcxgdi8ArYSl/98ZYLY Ji+x/+BZsBpOATuJwy/b2SBqBCVOznwCtoAZqKZ562yo676yS7y76Qthu0gsuXmTFcIWlnh1 fAs7hC0l8fndXjaI/4sllq2TA7lNQqCFUeL9uzmMEDXWEv+u7GEBqWEW0JRYv0sfIiwrMfXU OiaItXwSvb+fQJ3JK7FjHoytKnHh4DaoVdISaydshTrNQ2Ldse/MkICbyCjxtpt9AqPCLCTf zELyzSyEzQsYmVcxSqYWFOempxabFhjmpZbD4zg5P3cTIziBannuYLz74IPeIUYmDsZDjBIc zEoivFU11mlCvCmJlVWpRfnxRaU5qcWHGE2BwT2RWUo0OR+YwvNK4g1NLA1MzMzMTCyNzQyV xHlft85NERJITyxJzU5NLUgtgulj4uCUamBS39ZecyAj7Hncp8PuffXcJ9tEGC9NsJD5Hs8Z UPT/7DLL3qLF5qbmldyTqpn3Lj69V+ndY9VbK8wXHPi/0L2K4aOo2fyEit6JeVOy7yYmVoVZ Pc/3DnKYFTHx5LsbcULvN/m8dEy4cW9b4eKMNovlM8tT5apC2+r982sqvb6vq0q0PVLQv3Ja /HlJJ9Zjr2u7d+uabvtTNYdNSOb0w41n4t8/qItm3Py117j0zD99/aiW0CuHeLbNO8wS6izD ePrh8X9/3j5fr3cte+r0109NWFeuc31kss7kkPiZC9NfndJ6dzGSqXZR1+1Eccv3M/cm3mmx Vn8kcyBv4YbebYVFJhYzFvvIN5S/nmv35jy/EktxRqKhFnNRcSIAse5U3ykEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSnO56dvs0g3ff2S3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlbFkbzdTwUupip7dF9gaGGeKdjFyckgImEhc/bSD vYuRi0NIYDejxJMn/1ggEtISHYda2SFsYYmV/55DFf1glHjwaxErSIJNQEfi94pfQA0cHCIC uhKNdxVAapgFVjFKXH3fDlYjLBAhsWZnH9hQFgFViaOrNzGB2LwCthKX/3xlglggL7H/4Flm EJtTwE7i8Mt2NhBbCKjm1NQzzBD1ghInZz4Bm8MMVN+8dTbzBEaBWUhSs5CkFjAyrWKUTC0o zk3PLTYsMMpLLdcrTswtLs1L10vOz93ECA5zLa0djHtWfdA7xMjEwXiIUYKDWUmEt6rGOk2I NyWxsiq1KD++qDQntfgQozQHi5I477fXvSlCAumJJanZqakFqUUwWSYOTqkGpuPJLhYtjj7/ 5s61j8h9+umYOePyy6U3OZrs7mVW7p3OUb7W8Ht4fMY3rdpVCbLMr/vbFDs0OGxZjujm34qf F2czd+Z2nTc5tTu5Y86f2/w2WYJzid0pm/+1s+RnnuF+JDvvvkx/T7HP+7sKsT0V/zbpF1Ye YTe0dXX83/D/b/yrz2HnS1csknJ1vXRBa6VT3UrmT7IT02vP8ot3tze8k2hvnvd40+pZ019N fhuiJRKr0JX5fU/idJZv6WdSZjVdeqYuWlp/0CEz4VPZn1VOzz8zuy5wbbp6NPRf4MfXxQaT KkJPT3yrcazywPpzjtvPXdjN8zP7woZy2U/Kwod/dSftZVZJeqmUrJS+7bTxbiWW4oxEQy3m ouJEAAN5tgziAgAA X-CMS-MailID: 20240511055247epcas5p2a54e23b6dddd11dda962733d259a10af X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240511055247epcas5p2a54e23b6dddd11dda962733d259a10af References: <20240511055229.352481-1-cliang01.li@samsung.com> This patch depends on patch 1 and 2. Introduces two functions to separate the coalesced imu alloc and accounting path from the original one. This helps to keep the original code path clean. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 578d382ca9bc..7f95eba72f1c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,42 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static int io_coalesced_buffer_account_pin(struct io_ring_ctx *ctx, + struct page **pages, + struct io_mapped_ubuf *imu, + struct page **last_hpage, + struct io_imu_folio_data *data) +{ + int i, j, ret; + + imu->acct_pages = 0; + j = 0; + for (i = 0; i < data->nr_folios; i++) { + struct page *hpage = pages[j]; + + if (hpage == *last_hpage) + continue; + *last_hpage = hpage; + /* + * Already checked the page array in try coalesce, + * so pass in nr_pages=0 here to waive that. + */ + if (headpage_already_acct(ctx, pages, 0, hpage)) + continue; + imu->acct_pages += data->nr_pages_mid; + j += (i == 0) ? + data->nr_pages_head : data->nr_pages_mid; + } + + if (!imu->acct_pages) + return 0; + + ret = io_account_mem(ctx, imu->acct_pages); + if (ret) + imu->acct_pages = 0; + return ret; +} + static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, struct io_imu_folio_data *data) { @@ -949,6 +985,56 @@ static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, return true; } +static int io_coalesced_imu_alloc(struct io_ring_ctx *ctx, struct iovec *iov, + struct io_mapped_ubuf **pimu, + struct page **last_hpage, struct page **pages, + struct io_imu_folio_data *data) +{ + struct io_mapped_ubuf *imu = NULL; + unsigned long off; + size_t size, vec_len; + int ret, i, j; + + ret = -ENOMEM; + imu = kvmalloc(struct_size(imu, bvec, data->nr_folios), GFP_KERNEL); + if (!imu) + return ret; + + ret = io_coalesced_buffer_account_pin(ctx, pages, imu, last_hpage, + data); + if (ret) { + j = 0; + for (i = 0; i < data->nr_folios; i++) { + unpin_user_page(pages[j]); + j += (i == 0) ? + data->nr_pages_head : data->nr_pages_mid; + } + return ret; + } + off = (unsigned long) iov->iov_base & ~PAGE_MASK; + size = iov->iov_len; + /* store original address for later verification */ + imu->ubuf = (unsigned long) iov->iov_base; + imu->ubuf_end = imu->ubuf + iov->iov_len; + imu->nr_bvecs = data->nr_folios; + imu->folio_shift = data->folio_shift; + imu->folio_mask = ~((1UL << data->folio_shift) - 1); + *pimu = imu; + ret = 0; + + vec_len = min_t(size_t, size, PAGE_SIZE * data->nr_pages_head - off); + bvec_set_page(&imu->bvec[0], pages[0], vec_len, off); + size -= vec_len; + j = data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + vec_len = min_t(size_t, size, data->folio_size); + bvec_set_page(&imu->bvec[i], pages[j], vec_len, 0); + size -= vec_len; + j += data->nr_pages_mid; + } + return ret; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) From patchwork Sat May 11 05:52:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13662285 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE7D8380 for ; Sat, 11 May 2024 06:30:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409060; cv=none; b=tJDhoIYBPJiIVcwaYqR3IsZJRVm09bl+0S7KECXfa+UcqD8sCN88xZG9bN/lvWQt2Hcloahq0aViBd6owRVO48aITnpWbGqLHcagC7fhsOxe4LVZellJnLvdwkJbOp/TLf+86ydVRkb8RBycxXLOqXqLkSDLvGaP8rm/FycLPe8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715409060; c=relaxed/simple; bh=RKNM6mcKJHQrN1gXAWf9OWz2eNKa20RONK4dATr/IHA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=mIbKQ8kV++jxIiGAfTPhf0v6GF68kdG+E5fPL2LnnqA+B9Ze1Et90myi1VjcmONH5PKsuKl1YksxSwkkNwcrurp1+4dYFQ6p9HyCWOagjAuWGVWUsb7IWmWE8Rs6vlAgDMQokKKqjcueRwMwE9GUc4RnAWW3w6ZR5SHt8hU8KkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=O9GbEOA/; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="O9GbEOA/" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240511063050epoutp04ca91251544d7c9735d9ef2680ed6ba7c~OW3FJK3JT1078310783epoutp04z for ; Sat, 11 May 2024 06:30:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240511063050epoutp04ca91251544d7c9735d9ef2680ed6ba7c~OW3FJK3JT1078310783epoutp04z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715409050; bh=1gUqNGiqVvQbEYPG08LCY1EgsRDR8KnwG+EpwqPYvBk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O9GbEOA/bpO8uNT7vyqxscsQPN7gOzV53RYcbLUYzM2bI/TpUa4XdZq5fDZbMCcN5 yj5DMF6+cvAlrJFIlT0SrDxhp6ucspftV0XBKCjX0daKje3kQDvCRjQoXyawkljIrL WfXmkN5mX3BKqdvr+Ri9/7aFztrLFVuGcuQHpd6k= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240511063049epcas5p2598bb35194851823a5c227a1225c6541~OW3ExPuIE3223532235epcas5p2P; Sat, 11 May 2024 06:30:49 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.176]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4VbwqX0PSxz4x9Pw; Sat, 11 May 2024 06:30:48 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id FF.F5.09666.7901F366; Sat, 11 May 2024 15:30:47 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240511055248epcas5p287b7dfdab3162033744badc71fd084e1~OWV4dsZC73263732637epcas5p2z; Sat, 11 May 2024 05:52:48 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240511055248epsmtrp1c6501a09b228020dc84989394e2f0a69~OWV4c-VeE2071620716epsmtrp1U; Sat, 11 May 2024 05:52:48 +0000 (GMT) X-AuditID: b6c32a49-cefff700000025c2-0a-663f10976a5f Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id 8A.C2.19234.0B70F366; Sat, 11 May 2024 14:52:48 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20240511055247epsmtip125b36d14c4052d72974b983be33ad514~OWV3iXHCO0364503645epsmtip1d; Sat, 11 May 2024 05:52:47 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v2 4/4] io_uring/rsrc: enable multi-hugepage buffer coalescing Date: Sat, 11 May 2024 13:52:29 +0800 Message-Id: <20240511055229.352481-5-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240511055229.352481-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJJsWRmVeSWpSXmKPExsWy7bCmlu50Afs0g0sfrCzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITvjzMEmloLHkhX777WyNTAeFe5i5OSQEDCR2DhnPnsXIxeHkMBuRolv274z QjifGCX+fznMBue8eLiVFaZlzoXnrBCJnYwSC9cuhKr6xSjx+3QTI0gVm4COxO8Vv1hAbBEB bYnXj6eygBQxCyxhlNjVuRysSFggQOLgwg9A3RwcLAKqEoeeWoOEeQVsJW6u3cgGsU1eYv/B s8wgNqeAncThl+1sEDWCEidnPgGbzwxU07x1NjPIfAmBt+wSU349YoRodpHoaLjPAmELS7w6 voUdwpaSeNnfxg6yV0KgWGLZOjmI3hZGiffv5kD1Wkv8u7KHBaSGWUBTYv0ufYiwrMTUU+uY IPbySfT+fsIEEeeV2DEPxlaVuHBwG9QqaYm1E7YyQ9geEt/ubYIG9kRGiSOnGpgmMCrMQvLP LCT/zEJYvYCReRWjZGpBcW56arFpgWFeajk8mpPzczcxgtOolucOxrsPPugdYmTiYDzEKMHB rCTCW1VjnSbEm5JYWZValB9fVJqTWnyI0RQY3hOZpUST84GJPK8k3tDE0sDEzMzMxNLYzFBJ nPd169wUIYH0xJLU7NTUgtQimD4mDk6pBqaQC4575qZk1X8qvO8fvi3inVwf43Tfug+1Kco8 K3qWrZnllF54XebEz4XLd+/asG6J21eXuxlVpzfElJ9+avP03ZyryZdcQm7nx6xwV0tvby6/ eMHeMfOjy27lk29Xzj7951ST2ye2GWXPBRkts/Lv5R174t/+uW2h37Q/5WpG+yQndK9N4TW+ Ond1Fefx2DlMK9/GXeVe3m7+/Bdj+Y0CrikOVyvVje6fP7v5we+dXabcKXeOq80+KyCqMD/p jPr2LytrzrJ2NN/N/GT4YcvfFc2f4k50i8+RSJZ3np/scXLZaQ7nl9szS10edf9VPdnUmdcV ym1ldkfcKKiCKfz2exXR6petan5pbUYBW+cpsRRnJBpqMRcVJwIA6WiGniwEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrBLMWRmVeSWpSXmKPExsWy7bCSnO4Gdvs0g3/XdC3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlXHmYBNLwWPJiv33WtkaGI8KdzFyckgImEjMufCc tYuRi0NIYDujxMMjG1khEtISHYda2SFsYYmV/54D2RxART8YJS4ogITZBHQkfq/4xQISFhHQ lWi8qwAyhllgFaPE1fftYGOEBfwk2h8fYwapYRFQlTj01BokzCtgK3Fz7UY2iOnyEvsPnmUG sTkF7CQOv2wHiwsB1ZyaeoYZol5Q4uTMJywgNjNQffPW2cwTGAVmIUnNQpJawMi0ilE0taA4 Nz03ucBQrzgxt7g0L10vOT93EyM4sLWCdjAuW/9X7xAjEwfjIUYJDmYlEd6qGus0Id6UxMqq 1KL8+KLSnNTiQ4zSHCxK4rzKOZ0pQgLpiSWp2ampBalFMFkmDk6pBiZfj57wPUHPXeZZaj/w 8pwQ135ufsTlz+FbHeTe9Ew/ZhJ8VLm6XKi3cvE/xWdzim/6vvM6rbLq06wKD60nbmoru4W5 jhfX3rxx9ubRDVv589qcHuw5uFP4dkvWsTvme0JSF4btnnyfuSmrpaPsTMaPuPefnxzo+j4/ KYqP88W101JSMwW1s6U9T0Tm9h2xXrv+gpd+85wnoQ+m7PknaVcdHZW7vlq4R/hPh1dVh8Ze s5OamWuXTsrtNno7n2G2rseGFQG8hekpsnUpT68/WKFh1t/0fupxd649IteD5xXlpvBs3imm yRb1zj1d9VCkO+eWaoa6vliFuUtUxU8dKZDlyRMxfnw3Ve7O7DZRRyWW4oxEQy3mouJEAEOI YK/bAgAA X-CMS-MailID: 20240511055248epcas5p287b7dfdab3162033744badc71fd084e1 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240511055248epcas5p287b7dfdab3162033744badc71fd084e1 References: <20240511055229.352481-1-cliang01.li@samsung.com> This patch depends on patch 1, 2, 3. It modifies the original buffer registration path to expand the one-hugepage coalescing feature to work with multi-hugepage buffers. Separated from previous patches to make it more easily reviewed. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 44 ++++++++------------------------------------ 1 file changed, 8 insertions(+), 36 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7f95eba72f1c..70acc76ff27c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1044,7 +1044,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; - struct folio *folio = NULL; + struct io_imu_folio_data data; *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; if (!iov->iov_base) @@ -1059,30 +1059,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - /* If it's a huge page, try to coalesce them into a single bvec entry */ - if (nr_pages > 1) { - folio = page_folio(pages[0]); - for (i = 1; i < nr_pages; i++) { - /* - * Pages must be consecutive and on the same folio for - * this to work - */ - if (page_folio(pages[i]) != folio || - pages[i] != pages[i - 1] + 1) { - folio = NULL; - break; - } - } - if (folio) { - /* - * The pages are bound to the folio, it doesn't - * actually unpin them but drops all but one reference, - * which is usually put down by io_buffer_unmap(). - * Note, needs a better helper. - */ - unpin_user_pages(&pages[1], nr_pages - 1); - nr_pages = 1; - } + /* If it's huge page(s), try to coalesce them into fewer bvec entries */ + if (io_sqe_buffer_try_coalesce(pages, nr_pages, &data)) { + ret = io_coalesced_imu_alloc(ctx, iov, pimu, last_hpage, + pages, &data); + goto done; } imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); @@ -1106,10 +1087,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, *pimu = imu; ret = 0; - if (folio) { - bvec_set_page(&imu->bvec[0], pages[0], size, off); - goto done; - } for (i = 0; i < nr_pages; i++) { size_t vec_len; @@ -1215,23 +1192,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter, * we know that: * * 1) it's a BVEC iter, we set it up - * 2) all bvecs are PAGE_SIZE in size, except potentially the + * 2) all bvecs are the same in size, except potentially the * first and last bvec * * So just find our index, and adjust the iterator afterwards. * If the offset is within the first bvec (or the whole first * bvec, just use iov_iter_advance(). This makes it easier * since we can just skip the first segment, which may not - * be PAGE_SIZE aligned. + * be folio_size aligned. */ const struct bio_vec *bvec = imu->bvec; if (offset < bvec->bv_len) { - /* - * Note, huge pages buffers consists of one large - * bvec entry and should always go this way. The other - * branch doesn't expect non PAGE_SIZE'd chunks. - */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset;