From patchwork Mon May 13 08:22:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663186 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C078146A91 for ; Mon, 13 May 2024 08:29:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588983; cv=none; b=f/+vYgb+0QJvrT1IBNp8f+3iSIjqWjdYNbUYMBdxYMb6LIvWI/CbG/hLT7am1D2GTJ06h6amhOhKJq/2IhLnp3k2XSmCXinPcSReAHukGrPgQo2ySqbaKbCDrKVbUkhmZUDeqX96JKZrq97yswMfZaoK40Ey79gxNfeD2lhX0Ck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588983; c=relaxed/simple; bh=GgB79gShNLQoKk+gewD6WVXUhfoRMFUqTI993fcouUM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=pLLp/Mh3DsvkPS/f/pbsksUXLZJsMCpNci5xknHwIQEIRdufnwL1cGFrXCGt4CssvzjnnEnfvWfzFooqRz9cRZQNjZwf7nqyX4e4YhF5a2go/1O/vDib95hMEb/WmK+IS00tSwBa34IWzGCH6ijVObN7feFA4OD5pV0ycYLSOLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=b+kzygwh; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="b+kzygwh" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240513082937epoutp02577a157fe7b0fe8ae3e22730b9c71122~O-xXwjyZ31177211772epoutp020 for ; Mon, 13 May 2024 08:29:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240513082937epoutp02577a157fe7b0fe8ae3e22730b9c71122~O-xXwjyZ31177211772epoutp020 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715588977; bh=luky4jv5sJLLkurx2YzjEwBLcK+91Q/SXM6pUFX2CzU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b+kzygwhYtWbv9s6gSfRLjEWjInOYWc0w0LIVZEvtHO8He8cOxAfQLpbvMK1w1mAI Vjop1dCpu3eoL1btBwdnkhr18u4or8Iu4n7xGzWms9LqDpodBoF9JvPqCktIGf/eic kV4kd63iUcTpyBv6K+aXNR1uIsk7qLXCCsniSgYw= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20240513082937epcas5p1dae1544e187c9b4240615cb47de2f062~O-xXHAykX0073900739epcas5p1o; Mon, 13 May 2024 08:29:37 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.183]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4VdCMg2cPTz4x9Q3; Mon, 13 May 2024 08:29:35 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 70.0D.19431.F6FC1466; Mon, 13 May 2024 17:29:35 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240513082308epcas5p3c38ce4d44fa1613988bbae84eaec41b9~O-rtG6zkc2476724767epcas5p3j; Mon, 13 May 2024 08:23:08 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240513082308epsmtrp2401a2d08f6cb8aecad9ecda9907a7843~O-rtGCodg1459414594epsmtrp2p; Mon, 13 May 2024 08:23:08 +0000 (GMT) X-AuditID: b6c32a50-ccbff70000004be7-c4-6641cf6fa0ab Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id BA.23.09238.CEDC1466; Mon, 13 May 2024 17:23:08 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240513082307epsmtip241d24af930fcbb5745ddf0c1252668c2~O-rsLhlen1150211502epsmtip2Y; Mon, 13 May 2024 08:23:07 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v3 1/5] io_uring/rsrc: add hugepage buffer coalesce helpers Date: Mon, 13 May 2024 16:22:56 +0800 Message-Id: <20240513082300.515905-2-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513082300.515905-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJJsWRmVeSWpSXmKPExsWy7bCmpm7+ecc0g3m9ghZzVm1jtFh9t5/N 4vTfxywWNw/sZLJ413qOxeLo/7dsFr+67zJabP3yldXi2V5Oi7MTPrA6cHnsnHWX3ePy2VKP vi2rGD0+b5ILYInKtslITUxJLVJIzUvOT8nMS7dV8g6Od443NTMw1DW0tDBXUshLzE21VXLx CdB1y8wBOkdJoSwxpxQoFJBYXKykb2dTlF9akqqQkV9cYquUWpCSU2BSoFecmFtcmpeul5da YmVoYGBkClSYkJ0xb/oO1oKjchVXV15jbWA8JtHFyMkhIWAi0XfnH1sXIxeHkMAeRonfSw4z QzifGCVerJ7FDuF8Y5S4MquHEaZlzaa3LBCJvYwSy6+8ZAJJCAn8YpRYsMcHxGYT0JH4veIX C4gtIqAt8frxVLAGZoEljBK7OpeDTRIW8JbY2v+MHcRmEVCVeHJgIdggXgFbiX2XD7FDbJOX 2H/wLDOIzSlgJ7Fzzx9miBpBiZMzn4AtYAaqad46mxmi/iu7xIEpthC2i8SR3/NZIGxhiVfH t0DNlJL4/G4v0NMcQHaxxLJ1ciC3SQi0MEq8fzcH6ktriX9X9rCA1DALaEqs36UPEZaVmHpq HRPEWj6J3t9PmCDivBI75sHYqhIXDm6DWiUtsXbCVqjTPCT23LkMDeuJjBJ9Lb+YJjAqzELy ziwk78xCWL2AkXkVo1RqQXFuemqyaYGhbl5qOTyak/NzNzGC06hWwA7G1Rv+6h1iZOJgPMQo wcGsJMLrUGifJsSbklhZlVqUH19UmpNafIjRFBjgE5mlRJPzgYk8ryTe0MTSwMTMzMzE0tjM UEmc93Xr3BQhgfTEktTs1NSC1CKYPiYOTqkGpqLWvXdXm9Sczrh5eMWpeaXqp81vHI1duvF1 e++B5pJJ917sO+eZIfnK/hDn1qlul7VfxM8VK13+c7G2wNXnbpf2z9ygE710kZl+9UKpXVbP ClNtt8cVZ3PPkDgacfmEEbvknlXMDxbt+T7rVDXPMueii+s86yZaX+pufeRct/fqnfKzwmeW d5yqjW4z85vJNeFJykSJtK9Tvl1pOrvkecdSzZaDIoU81xK/GXZ/SxQwnsxs/+VZioOClvL0 sG15e9+c3JnwweNlrkCl+BU36QsVPW6bj53OnqhiNe8as+S0Zu/rdcyHdxy4rHrX9XeMv/rr A7kb5x83k/rsOefoKgsJ19V8PnybLn9amP5IwodHiaU4I9FQi7moOBEAa3BWsSwEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvO6bs45pBmcahSzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlTFv+g7WgqNyFVdXXmNtYDwm0cXIySEhYCKxZtNb li5GLg4hgd2MEt+f97FAJKQlOg61skPYwhIr/z1nhyj6wSixZddrsCI2AR2J3yt+AdkcHCIC uhKNdxVAapgFVjFKXH3fzgpSIyzgLbG1/xnYIBYBVYknBxYygdi8ArYS+y4fglogL7H/4Flm EJtTwE5i554/YLYQUM2pw5vZIOoFJU7OfAK2lxmovnnrbOYJjAKzkKRmIUktYGRaxSiZWlCc m56bbFhgmJdarlecmFtcmpeul5yfu4kRHOZaGjsY783/p3eIkYmD8RCjBAezkgivQ6F9mhBv SmJlVWpRfnxRaU5q8SFGaQ4WJXFewxmzU4QE0hNLUrNTUwtSi2CyTBycUg1MutK9fGYpWvkJ bzf6tM+TTXn60KT1yAOudTMjvitrXa+y6DxsIfFXybL5Tma6bZ137SrH2Td6lf6lf7RqjvtW W6bf2XJWVnLnmYczlii4tB/9k38y44VB8M8Ajjz5p8VhSw9xzKk22FilWTXrsM9UqZ8sqy+y TuPmfHNH6voH5l3Tm7hNbkkvr4rvnv1t+YHrO+Vcu545BPBGLt3IUnVEd5VxSMFbwZP/lpVo HjybPDUtTKL68INGL70pr+IUpqj+4Evc8Yz11JzqybMnv9jb3fLg162Y/scv0zlibJZ+Wrtm yqYjFcyPNM9b5p4L620UaG5pXmGtnzotWtOcLerO4338M/MvT5V4GiJyNWqVEktxRqKhFnNR cSIAorbt3eICAAA= X-CMS-MailID: 20240513082308epcas5p3c38ce4d44fa1613988bbae84eaec41b9 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240513082308epcas5p3c38ce4d44fa1613988bbae84eaec41b9 References: <20240513082300.515905-1-cliang01.li@samsung.com> Introduce helper functions to check whether a buffer can be coalesced or not, and gather folio data for later use. The coalescing optimizes time and space consumption caused by mapping and storing multi-hugepage fixed buffers. A coalescable multi-hugepage buffer should fully cover its folios (except potentially the first and last one), and these folios should have the same size. These requirements are for easier later process, also we need same size'd chunks in io_import_fixed for fast iov_iter adjust. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 10 +++++++ 2 files changed, 88 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 65417c9553b1..d08224c0c5b0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,84 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + struct folio *folio = page_folio(pages[0]); + unsigned int count = 1; + int i; + + data->nr_pages_mid = folio_nr_pages(folio); + if (data->nr_pages_mid == 1) + return false; + + data->folio_shift = folio_shift(folio); + data->folio_size = folio_size(folio); + data->nr_folios = 1; + /* + * Check if pages are contiguous inside a folio, and all folios have + * the same page count except for the head and tail. + */ + for (i = 1; i < nr_pages; i++) { + if (page_folio(pages[i]) == folio && + pages[i] == pages[i-1] + 1) { + count++; + continue; + } + + if (data->nr_folios == 1) + data->nr_pages_head = count; + else if (count != data->nr_pages_mid) + return false; + + folio = page_folio(pages[i]); + if (folio_size(folio) != data->folio_size) + return false; + + count = 1; + data->nr_folios++; + } + if (data->nr_folios == 1) + data->nr_pages_head = count; + + return true; +} + +static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, + struct io_imu_folio_data *data) +{ + int i, j; + + if (nr_pages <= 1 || + !__io_sqe_buffer_try_coalesce(pages, nr_pages, data)) + return false; + + /* + * The pages are bound to the folio, it doesn't + * actually unpin them but drops all but one reference, + * which is usually put down by io_buffer_unmap(). + * Note, needs a better helper. + */ + if (data->nr_pages_head > 1) + unpin_user_pages(&pages[1], data->nr_pages_head - 1); + + j = data->nr_pages_head; + nr_pages -= data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + unsigned int nr_unpin; + + nr_unpin = min_t(unsigned int, nr_pages - 1, + data->nr_pages_mid - 1); + if (nr_unpin == 0) + break; + unpin_user_pages(&pages[j+1], nr_unpin); + j += data->nr_pages_mid; + nr_pages -= data->nr_pages_mid; + } + + return true; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c032ca3436ca..b2a9d66b76dd 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -50,6 +50,16 @@ struct io_mapped_ubuf { struct bio_vec bvec[] __counted_by(nr_bvecs); }; +struct io_imu_folio_data { + /* Head folio can be partially included in the fixed buf */ + unsigned int nr_pages_head; + /* For non-head/tail folios, has to be fully included */ + unsigned int nr_pages_mid; + unsigned int nr_folios; + unsigned int folio_shift; + size_t folio_size; +}; + void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); From patchwork Mon May 13 08:22:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663187 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E998F1474A2 for ; Mon, 13 May 2024 08:29:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588985; cv=none; b=bdHzhRiJAdJvkNVGTVWf2Ts2ZYsYW7m/0XX2EGWZWVrhtXfKysW8aJg+2ofPa6gLToJz2SYKHbVa412kArXKS0EmPM3ivCzZsZC692lWer7kDFSLIfVKRFLto9koE+BUSsgFLXPH8/+gxD/mRy3h8ACCl8VwzcNK6P+ADcA9aGQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588985; c=relaxed/simple; bh=v98Mo+MpYBU0qzaAkf44WS1u6iYjnUbvkJq6cTR5f6I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=ZW59epwUDjT3Mx7Isk+g7zcEIq7xSBMooDTxQ2O5Cw4S1LUs2tS5BK13nkLXuVkisb9tBf6QTs1QGhRYgstdUCvaSWQFdCHN5Nvs/Ms/YSZs/DHxvp/JJWMrQFejMD1306es9T6QAIMSNem9ju4UQNdXOudEF5FtMx+u8BR/KQg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=bkxRbHnD; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="bkxRbHnD" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240513082940epoutp022f8d198c3784496126d9950fe94c00b8~O-xaBftbp1135311353epoutp022 for ; Mon, 13 May 2024 08:29:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240513082940epoutp022f8d198c3784496126d9950fe94c00b8~O-xaBftbp1135311353epoutp022 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715588980; bh=9J3qY+gr/EQFxqtdizhx5LUqyyO9iMd1OdFM5PunXLE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bkxRbHnDFPbn/Womho2vSf396FPYF7pJ+NBeOE6AhMRxylprhf1JC+Giun0l6Utoy DW6PfQCaDFyyvBoj3ORwE/02g8cXSKchm1XJfBlNkUwPshpHOcoWcqcvnlj6QjqwEJ IuPikktpGGcScEpoV44UgDAYh8y9liY/+mlt+Fvw= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240513082939epcas5p215274d484d2a8367f537b04e9cb9c7d9~O-xZqNy2J2025720257epcas5p2B; Mon, 13 May 2024 08:29:39 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4VdCMk0VZ7z4x9Q0; Mon, 13 May 2024 08:29:38 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 12.0D.19431.17FC1466; Mon, 13 May 2024 17:29:37 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240513082310epcas5p27576a80eae3ee09e40102b179ce46fa9~O-ruo0HN-3055430554epcas5p2V; Mon, 13 May 2024 08:23:10 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240513082310epsmtrp28b0a33d949586dbfc88762977aa1037a~O-ruoAuP_1439014390epsmtrp2J; Mon, 13 May 2024 08:23:10 +0000 (GMT) X-AuditID: b6c32a50-58472a8000004be7-cf-6641cf7195c0 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 9C.6C.08390.DEDC1466; Mon, 13 May 2024 17:23:09 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240513082308epsmtip2e368ab2b59d386810dd727a912e9737a~O-rtonvxn0931209312epsmtip2T; Mon, 13 May 2024 08:23:08 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v3 2/5] io_uring/rsrc: store folio shift and mask into imu Date: Mon, 13 May 2024 16:22:57 +0800 Message-Id: <20240513082300.515905-3-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513082300.515905-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJJsWRmVeSWpSXmKPExsWy7bCmlm7hecc0g2OHrSzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITvj1PoVzAXPeCr6rzUyNjDO5upi5OSQEDCRePJhElsXIxeHkMAeRond55Yy QjifGCW2n5nBAlIlJPCNUeLIRGmYjgufO1kgivYySnS2/4ZyfjFK7HqzlR2kik1AR+L3il9g 3SIC2hKvH08FK2IWWAJU1LmcESQhLOAlsfTBJKAEBweLgKrE3t2SIGFeAVuJdT8usEJsk5fY f/AsM4jNKWAnsXPPH2aIGkGJkzOfgM1nBqpp3jqbGWS+hMBHdonJnxazQzS7SKy78oQRwhaW eHV8C1RcSuLzu71sIHslBIollq2Tg+htYZR4/24OVL21xL8re8BuYxbQlFi/Sx8iLCsx9dQ6 Joi9fBK9v58wQcR5JXbMg7FVJS4c3Aa1Slpi7YStzBC2h8Ti+7OhYT2RUWLN1H7GCYwKs5D8 MwvJP7MQVi9gZF7FKJVaUJybnppsWmCom5daDo/m5PzcTYzgNKoVsINx9Ya/eocYmTgYDzFK cDArifA6FNqnCfGmJFZWpRblxxeV5qQWH2I0BQb4RGYp0eR8YCLPK4k3NLE0MDEzMzOxNDYz VBLnfd06N0VIID2xJDU7NbUgtQimj4mDU6qBKdlNMD8g1skwkM3WfcHO/Njgta5qgakZ92Mn bGu9dOd54OyvFxusl7YYV9ybZ1i5tlFr6sOlJy0E1RVlVk13Nq3dvJlTx8vsndeCubfmsftl eH81s/3h/ahug8TDhqmZGl1iStFiE28v48ho+7xg+2rjhgmZL1dJCeXM3eG/82gVp2mfhsHO utLFPtLrVI2ifeJvG6ZMj7ynqPdU14A1b7WskugBEw8Hm6s9JZYGT1lnO4s+rfifGbUxIabp eOye9XNTo0633dbS8v95dV6s0YowBeY3DxkaWSq3dn07t+LJ4s4PMd8cnp3JM3S+oHR411Rz NZfKtVUWRTH7GpimeHiafJjlJdYZ0SnTsk+JpTgj0VCLuag4EQDuOds+LAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsWy7bCSvO7bs45pBodazCzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlXFq/Qrmgmc8Ff3XGhkbGGdzdTFyckgImEhc+NzJ 0sXIxSEksJtR4ur0B6wQCWmJjkOt7BC2sMTKf8/ZIYp+MEpMe/6GCSTBJqAj8XvFL6BuDg4R AV2JxrsKIDXMAquABr1vBxskLOAlsfTBJLAaFgFVib27JUHCvAK2Eut+XIDaJS+x/+BZZhCb U8BOYueeP2C2EFDNqcOb2SDqBSVOznzCAmIzA9U3b53NPIFRYBaS1CwkqQWMTKsYJVMLinPT c4sNC4zyUsv1ihNzi0vz0vWS83M3MYKDXEtrB+OeVR/0DjEycTAeYpTgYFYS4XUotE8T4k1J rKxKLcqPLyrNSS0+xCjNwaIkzvvtdW+KkEB6YklqdmpqQWoRTJaJg1OqgSnv357Tj5/kPbmp n31OaIFAbszVTIm0xb9fMP48wa0mfWfN6SefQqQLE++WrHxhV2Rz6cAegZM5Kao6eUtvaWRv OH1aPPfdA+4+JgGF4jBu76nfXT/7GrbsnrrT6k6Cf6fEy6nWVyXnzZ2g+L/ba83ar+I64fqt q36eMF/7/WPQ43ieA/GJintdNfaabxcwXvKxnNtk5m+OCPXtD6yrmhq3vxV3dT4klL2UZ72c kp0Ik8+cD8YC2ZIO//PP+b3vXp5dV747t+L9FJu3WfUF7ry3lhnybJiwhoE1OrXlrgHTo6Tj 1/r9fjwWUC0UkJnW7pPSfaVqseFP9Rh/9WOqPSdKNW8lXZ8Xk/Hf3PmajBJLcUaioRZzUXEi AFWp7pzhAgAA X-CMS-MailID: 20240513082310epcas5p27576a80eae3ee09e40102b179ce46fa9 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240513082310epcas5p27576a80eae3ee09e40102b179ce46fa9 References: <20240513082300.515905-1-cliang01.li@samsung.com> Store the folio shift and folio mask into imu struct and use it in iov_iter adjust, as we will have non PAGE_SIZE'd chunks if a multi-hugepage buffer get coalesced. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 6 ++++-- io_uring/rsrc.h | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d08224c0c5b0..578d382ca9bc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1015,6 +1015,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->ubuf = (unsigned long) iov->iov_base; imu->ubuf_end = imu->ubuf + iov->iov_len; imu->nr_bvecs = nr_pages; + imu->folio_shift = PAGE_SHIFT; + imu->folio_mask = PAGE_MASK; *pimu = imu; ret = 0; @@ -1153,12 +1155,12 @@ int io_import_fixed(int ddir, struct iov_iter *iter, /* skip first vec */ offset -= bvec->bv_len; - seg_skip = 1 + (offset >> PAGE_SHIFT); + seg_skip = 1 + (offset >> imu->folio_shift); iter->bvec = bvec + seg_skip; iter->nr_segs -= seg_skip; iter->count -= bvec->bv_len + offset; - iter->iov_offset = offset & ~PAGE_MASK; + iter->iov_offset = offset & ~imu->folio_mask; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index b2a9d66b76dd..93da02e652bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -46,7 +46,9 @@ struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; unsigned int nr_bvecs; + unsigned int folio_shift; unsigned long acct_pages; + unsigned long folio_mask; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Mon May 13 08:22:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663188 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18C371474A2 for ; Mon, 13 May 2024 08:29:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.24 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588993; cv=none; b=rpENufXIrc9/B0FEG8JpR3QIsMCf2352YpKRCHtjICfaJBqNNQ7H9tcv0y2efUy4Gq6zu9ZLzRpLcI+0ib/6548icfXhR2TzRYaVv6RiuwoAP53DRXg3XGtgGm+A5q0TvoNcjHI1WhVYmpuf6KM3Zxh9R79My/83oV6DEe3iypA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588993; c=relaxed/simple; bh=L04S/T0HduAOk5jic8D4zPDYPUsOlWlq6z5J2Sfh0tI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=afbtVaY9d1YKMXlPMK+7seBN0sc0cUuu2LRsssoDaeTMMqqMKNHaL6kfvqAbRKBd9HyxyesxonT9igU2kz621RNlTPdl5sLSObq0GzqwE8qKILimNW+W/A+nmvEJxTUkewudc/CXirAbvt3yzY8njFlSV3S1FVdGFKlktTrLv0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=YlhD9tyI; arc=none smtp.client-ip=203.254.224.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="YlhD9tyI" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20240513082942epoutp013a8d7e74c6fb8085be81e0050922360f~O-xcQxuEe1953319533epoutp01U for ; Mon, 13 May 2024 08:29:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20240513082942epoutp013a8d7e74c6fb8085be81e0050922360f~O-xcQxuEe1953319533epoutp01U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715588982; bh=oXAeKS3l9SfB9JAX17XgdEuqjbyD8WlhE++s0l1R6lk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YlhD9tyIFIYfRbpDGyBQk1iPGlInbU3AnmVVly5KsS4tRZB/M449uVSByWACAries +uUUBhEsK8zMCB1bTrMR5/Ty+93sIMrudmIAM/0Yiuk40ow1O8VQjiWnGEd9bNsS09 RufEvra6SgbavzMn4659u2afj9Gy8OKSLkSYfNzg= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240513082942epcas5p29c83675a161e291e8b4a5aa0f40cf4ee~O-xb3ojjq2028620286epcas5p2D; Mon, 13 May 2024 08:29:42 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.174]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4VdCMm3Bpxz4x9Py; Mon, 13 May 2024 08:29:40 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 23.E5.09688.47FC1466; Mon, 13 May 2024 17:29:40 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240513082311epcas5p3556d301a1f1faca0c6b613555324861e~O-rwIj0Ga2725827258epcas5p3O; Mon, 13 May 2024 08:23:11 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240513082311epsmtrp15dd4da51428071c821337d214cdc0a09~O-rwHvkYh1297912979epsmtrp1I; Mon, 13 May 2024 08:23:11 +0000 (GMT) X-AuditID: b6c32a4a-5dbff700000025d8-bb-6641cf745a24 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 7E.6C.08390.FEDC1466; Mon, 13 May 2024 17:23:11 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240513082310epsmtip23953c50dfd194e42ffa8b22fd018ed8a~O-rvCjmCr0931209312epsmtip2V; Mon, 13 May 2024 08:23:10 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v3 3/5] io_uring/rsrc: add init and account functions for coalesced imus Date: Mon, 13 May 2024 16:22:58 +0800 Message-Id: <20240513082300.515905-4-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513082300.515905-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprOJsWRmVeSWpSXmKPExsWy7bCmum7Jecc0g8vfhS3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITvj9MMFbAXPpSrWftnI3sA4R7SLkZNDQsBE4tSJhYwgtpDAbkaJZ4vjIexP jBKHpvN1MXIB2d8YJV7/XMwC0zDn8VUmiMReoIYD65khnF+MEnOuLwGrYhPQkfi94heYLSKg LfH68VQWkCJmgSWMErs6l4PtExaIkvg0cyVYEYuAqsS2pbvAbF4BW4l/66ZBrZOX2H/wLDOI zSlgJ7Fzzx9miBpBiZMzn4DVMAPVNG+dDXaFhMBXdol1i56zdzFyADkuEtOvMULMEZZ4dXwL O4QtJfGyvw2qpFhi2To5iNYWRon37+ZA1VtL/LuyhwWkhllAU2L9Ln2IsKzE1FPrmCDW8kn0 /n7CBBHnldgxD8ZWlbhwcBvUKmmJtRO2MkPYHhITl/5ihQTWREaJUz1/GScwKsxC8s4sJO/M Qli9gJF5FaNkakFxbnpqsWmBUV5qOTySk/NzNzGCU6iW1w7Ghw8+6B1iZOJgPMQowcGsJMLr UGifJsSbklhZlVqUH19UmpNafIjRFBjeE5mlRJPzgUk8ryTe0MTSwMTMzMzE0tjMUEmc93Xr 3BQhgfTEktTs1NSC1CKYPiYOTqkGpqVzVvkuj0lWfvfyxbmDvi/vujscPxDI7+i8b3bQTPsd jIneQUuWx5W66cvdlF1dbeq+e/HeCcnb+WfFP167co7ZhXfGP5YkCTf5Gx/P6FKedPmHl/iW +/5Je/mkOCtilSOYar8ahsfbS/2bZXvW/q5Y4EqhtK54waKwaQrHojrPKPJLnT5ruyhlN/dM iQWliz737w2xdPhpVvN84/JdEdNm5Uw9ybpISkTV1Cn81PK5evYnjQpEv3v/iAvbqxeyaJm9 A/f07CM8f102aNpzvtHtfONc0x/0td79262STROX/KmdufECx8NXHwR3LitRLnLZdPXwT6/T Qmfl1nxcKypwLk4kdJpP//JjCjda3yixFGckGmoxFxUnAgAxCMHbKgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvO77s45pBsceKljMWbWN0WL13X42 i9N/H7NY3Dywk8niXes5Fouj/9+yWfzqvstosfXLV1aLZ3s5Lc5O+MDqwOWxc9Zddo/LZ0s9 +rasYvT4vEkugCWKyyYlNSezLLVI3y6BK+P0wwVsBc+lKtZ+2cjewDhHtIuRk0NCwERizuOr TF2MXBxCArsZJZ49nMwMkZCW6DjUyg5hC0us/PecHaLoB6PEju0HWEASbAI6Er9X/AKyOThE BHQlGu8qgNQwC6xilLj6vp0VpEZYIEJi1tpFTCA2i4CqxLalu8B6eQVsJf6tm8YCsUBeYv/B s2CLOQXsJHbu+QNmCwHVnDq8mQ2iXlDi5MwnYPXMQPXNW2czT2AUmIUkNQtJagEj0ypGydSC 4tz03GLDAqO81HK94sTc4tK8dL3k/NxNjOAw19Lawbhn1Qe9Q4xMHIyHGCU4mJVEeB0K7dOE eFMSK6tSi/Lji0pzUosPMUpzsCiJ83573ZsiJJCeWJKanZpakFoEk2Xi4JRqYIoTaAm883tH u2VDbPl6fxcvl4u6DIdONLCav4qsctc6eGvfbdOHB5fprrvostvo/PWA80JKydXPX2SZaN6q siis6N3tJZvySoehO1WiPc6tIetjYOKfH/l39u6Qe6H9f6JaT8K5vpR7OdpdUde9jGS2s8q8 7TLnMdGd2v0/JuOensDpX7v/XI6yX7bk7ZQNc9/zBboUlaTOe9bTmf7VqFTySu7ktaKFlXnM UyUXurEKzjox/0WAQtS+rA/JudEb2dlP/7mSac2xk9ljT2fF8+CwRy6nGW9MP7PkVInWA565 Jvfa3bZNlH9ZV6DaEuf9KmFjReAbXZft37bZSE676rTa/1nO58Xmy17f9lvKocRSnJFoqMVc VJwIAPTAEMjiAgAA X-CMS-MailID: 20240513082311epcas5p3556d301a1f1faca0c6b613555324861e X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240513082311epcas5p3556d301a1f1faca0c6b613555324861e References: <20240513082300.515905-1-cliang01.li@samsung.com> Introduce two functions to separate the coalesced imu alloc and accounting path from the original one. This helps to keep the original code path clean. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 578d382ca9bc..53fac5f27bbf 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -871,6 +871,45 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static int io_coalesced_buffer_account_pin(struct io_ring_ctx *ctx, + struct page **pages, + struct io_mapped_ubuf *imu, + struct page **last_hpage, + struct io_imu_folio_data *data) +{ + int i, j, ret; + + imu->acct_pages = 0; + j = 0; + for (i = 0; i < data->nr_folios; i++) { + struct page *hpage = pages[j]; + + if (hpage == *last_hpage) + continue; + *last_hpage = hpage; + /* + * Already checked the page array in try coalesce, + * so pass in nr_pages=0 here to waive that. + */ + if (headpage_already_acct(ctx, pages, 0, hpage)) + continue; + imu->acct_pages += data->nr_pages_mid; + if (i) + j += data->nr_pages_mid; + else + j = data->nr_pages_head; + } + + if (!imu->acct_pages) + return 0; + + ret = io_account_mem(ctx, imu->acct_pages); + if (!ret) + return 0; + imu->acct_pages = 0; + return ret; +} + static bool __io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, struct io_imu_folio_data *data) { @@ -949,6 +988,56 @@ static bool io_sqe_buffer_try_coalesce(struct page **pages, int nr_pages, return true; } +static int io_coalesced_imu_alloc(struct io_ring_ctx *ctx, struct iovec *iov, + struct io_mapped_ubuf **pimu, + struct page **last_hpage, struct page **pages, + struct io_imu_folio_data *data) +{ + struct io_mapped_ubuf *imu = NULL; + unsigned long off; + size_t size, vec_len; + int ret, i, j; + + ret = -ENOMEM; + imu = kvmalloc(struct_size(imu, bvec, data->nr_folios), GFP_KERNEL); + if (!imu) + return ret; + + ret = io_coalesced_buffer_account_pin(ctx, pages, imu, last_hpage, + data); + if (ret) { + unpin_user_page(pages[0]); + j = data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + unpin_user_page(pages[j]); + j += data->nr_pages_mid; + } + return ret; + } + off = (unsigned long) iov->iov_base & ~PAGE_MASK; + size = iov->iov_len; + /* store original address for later verification */ + imu->ubuf = (unsigned long) iov->iov_base; + imu->ubuf_end = imu->ubuf + iov->iov_len; + imu->nr_bvecs = data->nr_folios; + imu->folio_shift = data->folio_shift; + imu->folio_mask = ~((1UL << data->folio_shift) - 1); + *pimu = imu; + ret = 0; + + vec_len = min_t(size_t, size, PAGE_SIZE * data->nr_pages_head - off); + bvec_set_page(&imu->bvec[0], pages[0], vec_len, off); + size -= vec_len; + j = data->nr_pages_head; + for (i = 1; i < data->nr_folios; i++) { + vec_len = min_t(size_t, size, data->folio_size); + bvec_set_page(&imu->bvec[i], pages[j], vec_len, 0); + size -= vec_len; + j += data->nr_pages_mid; + } + return ret; +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) From patchwork Mon May 13 08:22:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663190 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B6CA1482FE for ; Mon, 13 May 2024 08:33:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715589202; cv=none; b=G1/A/z/hX8eHn6sswOtJW3u1pvbJH7DpJjH4aZjrc953atpKpnV15xm2JseNOvXYCALd45RNFlCdPNMxMs/dmpQzRN6s1+p4M16y5WW2Kbnfty4EMqg7GwyznqhlipuCcMEa7cSgZRbYsx5d+a5b70Te9puFNlUIQ6YjU6vjgzA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715589202; c=relaxed/simple; bh=x1lM9kZ9BGnwdm1+MInf5Pe+/XsY67EMndVqggG8gr0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=rTCHqOtp/EWZ0LogE4xFTSJ+1QtkvEOqMHjBTid15XA9v1xf9K5+4Vh7GlSxgsdKFJA02rbKQ/X3PMqHdN8xYd3uCfi8Hh/vDmbr+KClrN0Z1DecoRcdObbFuE4viD0g8BKC+hjNqsWHOk+RTni0ydFCqLe0X093IPGdnZrznQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=vh6Nn3wf; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="vh6Nn3wf" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240513083315epoutp04e351f265cc1d1b99d55c9fdcb9c867d4~O-0iHRfq11434114341epoutp042 for ; Mon, 13 May 2024 08:33:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240513083315epoutp04e351f265cc1d1b99d55c9fdcb9c867d4~O-0iHRfq11434114341epoutp042 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715589195; bh=v+WaDfn/RpYboI5Yj9LJJNwQgkVytlZFrMVuD73n/0I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vh6Nn3wfWu+IKelfJLV3TqsniNwSUKoO8duvyCRcvSwenKqhWUt/Ze2mSSnABuf6Y 9v2OPBy3JOF3cYFuJ6/7UvTCY9aIBYjB5PuplmMpnR3mxEd0Qlii70smE16VBFRIyS I3DFpoK/EorU7SKUhutTh9yttDy9Ja87dguzfwvc= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240513083314epcas5p27a08524ef77f627f7da46e75c61461d1~O-0hmoRud2904529045epcas5p2_; Mon, 13 May 2024 08:33:14 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4VdCRr5KFnz4x9QJ; Mon, 13 May 2024 08:33:12 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 2B.02.09665.840D1466; Mon, 13 May 2024 17:33:12 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20240513082313epcas5p2a781d3393e4bf92d13d04ac62bb28fb7~O-rxpcuJ22123521235epcas5p2Q; Mon, 13 May 2024 08:23:13 +0000 (GMT) Received: from epsmgmcp1.samsung.com (unknown [182.195.42.82]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240513082313epsmtrp1a9380dc713c4cfd1b446392d7b647a0b~O-rxnxfX61295312953epsmtrp1g; Mon, 13 May 2024 08:23:13 +0000 (GMT) X-AuditID: b6c32a4b-829fa700000025c1-b8-6641d048970b Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmcp1.samsung.com (Symantec Messaging Gateway) with SMTP id B9.AB.19234.1FDC1466; Mon, 13 May 2024 17:23:13 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240513082312epsmtip2add8b99aa1bf7445aaaa0113f8a58d50~O-rwmIjEU1126111261epsmtip2z; Mon, 13 May 2024 08:23:12 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v3 4/5] io_uring/rsrc: enable multi-hugepage buffer coalescing Date: Mon, 13 May 2024 16:22:59 +0800 Message-Id: <20240513082300.515905-5-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513082300.515905-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprJJsWRmVeSWpSXmKPExsWy7bCmhq7HBcc0g3V3NSzmrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITuj7fF1toKdkhV9c++xNDAuEe5i5OSQEDCRODN7MmMXIxeHkMBuRonmT2fY IZxPjBKbFtxhgXNebv7FDtOy7sVKZojETkaJ69/XQDm/GCVOHjnIClLFJqAj8XvFLxYQW0RA W+L146lgo5gFljBK7OpczgiSEBYIkJg5bQkbiM0ioCqx4VAD2ApeAVuJna2rWSHWyUvsP3iW GcTmFLCT2LnnDzNEjaDEyZlPwBYwA9U0b50NdoWEwFt2iTUX3jBBNLtINP1sZoOwhSVeHd8C 9YOUxMv+NiCbA8gulli2Tg6it4VR4v27OYwQNdYS/67sYQGpYRbQlFi/Sx8iLCsx9dQ6Joi9 fBK9v59AreKV2DEPxlaVuHBwG9QqaYm1E7YyQ6zykDjUng0JrImMEl9mNrFPYFSYheSdWUje mYWweQEj8ypGydSC4tz01GLTAuO81HJ4NCfn525iBKdRLe8djI8efNA7xMjEwXiIUYKDWUmE 16HQPk2INyWxsiq1KD++qDQntfgQoykwvCcyS4km5wMTeV5JvKGJpYGJmZmZiaWxmaGSOO/r 1rkpQgLpiSWp2ampBalFMH1MHJxSDUzLf/DEc3VeExcUkmx+ZjtdpVMvfOc7h/kCT4Q2hUac TnyquD2g/I+67f3T5+YLJmxwm7xfv/CzlElYK3/FkukHtY/ofl6s/vfe54qbQaF/97k8YX+W HsL5a23Q0vneZxbXN8ponmB50W/DK/1lTWTXibA2TotXWkmyRsrpvh0lFz7/FK1TcTymXHf9 633Znw8KtR98KTp98sotJ2PHc+FPvf97z/yZeq1J6HPLU7F7jw7N7je7bRX5z2GnYruVtPJN ZfUPLWJ9u8Q0eY0knrb+dVF929QSxCVYLi2X5fBuwz+jT9VuS25YTEp+4FQQ9eh567ZPDZqp j6ccl320+nJ9x7YaQZW1K6rOCnJGeSixFGckGmoxFxUnAgCxR3IrLAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFLMWRmVeSWpSXmKPExsWy7bCSvO7Hs45pBlM2cVnMWbWN0WL13X42 i9N/H7NY3Dywk8niXes5Fouj/9+yWfzqvstosfXLV1aLZ3s5Lc5O+MDqwOWxc9Zddo/LZ0s9 +rasYvT4vEkugCWKyyYlNSezLLVI3y6BK6Pt8XW2gp2SFX1z77E0MC4R7mLk5JAQMJFY92Il cxcjF4eQwHZGiQUnNjFDJKQlOg61skPYwhIr/z0Hs4UEfjBKnD4kBGKzCehI/F7xi6WLkYND REBXovGuAsgcZoFVjBJX37ezgtQIC/hJrH/TC2azCKhKbDjUADaHV8BWYmfralaI+fIS+w+e BdvLKWAnsXPPH2aIXbYSpw5vZoOoF5Q4OfMJC4jNDFTfvHU28wRGgVlIUrOQpBYwMq1iFE0t KM5Nz00uMNQrTswtLs1L10vOz93ECA5uraAdjMvW/9U7xMjEwXiIUYKDWUmE16HQPk2INyWx siq1KD++qDQntfgQozQHi5I4r3JOZ4qQQHpiSWp2ampBahFMlomDU6qBqf/1Po+Da0Sj1a2k 8iutTyfdDNgSI9pwgd/00qRls5w16i/M4Cj2lTCa+zJeccHabXedpBX57lfvdYjmEI/VOCN/ 083l0Yb/NbkaVQfmWjkxLmldlZE0wahZ9oDr7zzrUi3N9bZ7Dp8Mi3Xhbb4kl+/gbtvtatN1 WCM0b6Wb/hpbyz8bFiistqma6NOc25Z7nWsaz81u22LBOUsDOx38trTfmV7xRSY80Ntzf6o4 78WTbneNw2qb7cQU/3PnJkW9PKuaxmm/6XCItrX+9wcic0wWnr94n39R9FsL392H9E0/qhpP fXf5dPk5m92Cmk8tNfOzNrPXyHZZuPUv8l/dXrqgIOS2tZzg160P5JRYijMSDbWYi4oTAbNo r+bdAgAA X-CMS-MailID: 20240513082313epcas5p2a781d3393e4bf92d13d04ac62bb28fb7 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240513082313epcas5p2a781d3393e4bf92d13d04ac62bb28fb7 References: <20240513082300.515905-1-cliang01.li@samsung.com> Modify the original buffer registration path to expand the one-hugepage coalescing feature to work with multi-hugepage buffers. Separated from previous patches to make it more easily reviewed. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 44 ++++++++------------------------------------ 1 file changed, 8 insertions(+), 36 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 53fac5f27bbf..5e5c1d6f3501 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1047,7 +1047,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; - struct folio *folio = NULL; + struct io_imu_folio_data data; *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; if (!iov->iov_base) @@ -1062,30 +1062,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - /* If it's a huge page, try to coalesce them into a single bvec entry */ - if (nr_pages > 1) { - folio = page_folio(pages[0]); - for (i = 1; i < nr_pages; i++) { - /* - * Pages must be consecutive and on the same folio for - * this to work - */ - if (page_folio(pages[i]) != folio || - pages[i] != pages[i - 1] + 1) { - folio = NULL; - break; - } - } - if (folio) { - /* - * The pages are bound to the folio, it doesn't - * actually unpin them but drops all but one reference, - * which is usually put down by io_buffer_unmap(). - * Note, needs a better helper. - */ - unpin_user_pages(&pages[1], nr_pages - 1); - nr_pages = 1; - } + /* If it's huge page(s), try to coalesce them into fewer bvec entries */ + if (io_sqe_buffer_try_coalesce(pages, nr_pages, &data)) { + ret = io_coalesced_imu_alloc(ctx, iov, pimu, last_hpage, + pages, &data); + goto done; } imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); @@ -1109,10 +1090,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, *pimu = imu; ret = 0; - if (folio) { - bvec_set_page(&imu->bvec[0], pages[0], size, off); - goto done; - } for (i = 0; i < nr_pages; i++) { size_t vec_len; @@ -1218,23 +1195,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter, * we know that: * * 1) it's a BVEC iter, we set it up - * 2) all bvecs are PAGE_SIZE in size, except potentially the + * 2) all bvecs are the same in size, except potentially the * first and last bvec * * So just find our index, and adjust the iterator afterwards. * If the offset is within the first bvec (or the whole first * bvec, just use iov_iter_advance(). This makes it easier * since we can just skip the first segment, which may not - * be PAGE_SIZE aligned. + * be folio_size aligned. */ const struct bio_vec *bvec = imu->bvec; if (offset < bvec->bv_len) { - /* - * Note, huge pages buffers consists of one large - * bvec entry and should always go this way. The other - * branch doesn't expect non PAGE_SIZE'd chunks. - */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset; From patchwork Mon May 13 08:23:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13663189 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78FEF1474B8 for ; Mon, 13 May 2024 08:29:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.33 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588994; cv=none; b=gtJeIBWL0uHBG+hCpAtIkLYk1gx8Wwwo9jc7UBVk+X86lmaoftU5LPAYCTvGqntZzF0eQWPapNHstJTZDhC6kqr3NsAA3my51idMHybxWJaf1mvGmOJLnkU4As3Q5kPwaXGQEL2twWD4XMebQRryviVexnWhzbQU15yauJvz8ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588994; c=relaxed/simple; bh=RsIuGX5WN/HtIDzBQRHz23ESiivoJG7bJjYvkqY+Rvg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=is8T/aCr8rTn8qn4YIQnJjAD/8vbXsnYk4VyrqgxH3maeBRfr5ukPtXV0bZfHjjzXoM4dit6m7uLyFa0M/iX2KiIJqCPU2v4sDt3vF6ylEDO8eGlNT8qHbpX4/GquyReXu3UUU1vcH7C3eMQDFmiJngVkj2q5mBlDRf+gQGwHx4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=Yk8h64Rh; arc=none smtp.client-ip=203.254.224.33 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="Yk8h64Rh" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20240513082947epoutp03fd717ae62c14c46eda40acf9f8460c43~O-xgqYlqc2099120991epoutp03Q for ; Mon, 13 May 2024 08:29:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20240513082947epoutp03fd717ae62c14c46eda40acf9f8460c43~O-xgqYlqc2099120991epoutp03Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1715588987; bh=O5AQlYRpYZGPgRa5PdwrIVzhA8/wXpy8vWaqjRaCDMk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yk8h64RhXS1Xeg8y6bPifF+VP/h8TDsDfzJvNlJf/ykMLSz8Rh4jHnsha94G+8f5Z IPUbAPwAHAouVSUip+aQZHqlCdaR7EXb4e+24sDMpc/FUvIqi9s+Er7XLvdoryQQPK VtWNJibNZVPFbWECVd8uEtNn9yoSzA34FiSLsshc= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240513082946epcas5p201718fa46699cdfe188994d8c99a9d06~O-xgUxdnJ2028620286epcas5p2T; Mon, 13 May 2024 08:29:46 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.174]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4VdCMs1K3lz4x9Pr; Mon, 13 May 2024 08:29:45 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3F.B3.09666.97FC1466; Mon, 13 May 2024 17:29:45 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20240513082314epcas5p309fa70575596792b5c9923ce76a3778f~O-rzNiBop2725827258epcas5p3e; Mon, 13 May 2024 08:23:14 +0000 (GMT) Received: from epsmgms1p2new.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240513082314epsmtrp278845615213ca1252df3c529eb31a824~O-rzMygWQ1459414594epsmtrp2J; Mon, 13 May 2024 08:23:14 +0000 (GMT) X-AuditID: b6c32a49-cefff700000025c2-e8-6641cf7931f6 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2new.samsung.com (Symantec Messaging Gateway) with SMTP id F0.7C.08390.2FDC1466; Mon, 13 May 2024 17:23:14 +0900 (KST) Received: from testpc118124.samsungds.net (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240513082313epsmtip20c9e5f86e0da51704fb9ff9c0c9bb9a9~O-ryIsfUZ0931209312epsmtip2a; Mon, 13 May 2024 08:23:13 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [PATCH v3 5/5] liburing: add test cases for hugepage registered buffers Date: Mon, 13 May 2024 16:23:00 +0800 Message-Id: <20240513082300.515905-6-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513082300.515905-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprOJsWRmVeSWpSXmKPExsWy7bCmpm7lecc0g80fWC3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBKVbZORmpiSWqSQmpecn5KZl26r5B0c7xxvamZgqGtoaWGupJCXmJtqq+Ti E6DrlpkDdI6SQlliTilQKCCxuFhJ386mKL+0JFUhI7+4xFYptSAlp8CkQK84Mbe4NC9dLy+1 xMrQwMDIFKgwITtj9qyFjAVbjSrurn7A1MD4QL2LkZNDQsBEYs3Hc8wgtpDAbkaJQxeLIOxP jBK/N0p3MXIB2d8YJbrnT2OCaeh4NJ8FIrGXUaLv/zRGiI5fjBI901JBbDYBHYnfK36xgNgi AtoSrx9PBWtgFljCKLGrczlYg7BAkMTKM4vZQGwWAVWJSf+/gJ3BK2ArsfDlJFaIbfIS+w+e BYtzCthJ7NzzB6pGUOLkzCdgC5iBapq3zmYGWSAh8JNd4mDjDzaIZheJCyf3sEDYwhKvjm9h h7ClJD6/2wtUwwFkF0ssWycH0dvCKPH+3RxGiBpriX9XQHo5gBZoSqzfpQ8RlpWYemodE8Re Pone30+gocIrsWMejK0qceHgNqhV0hJrJ2xlhrA9JN4sugsNuYmMEqs/nWSfwKgwC8k/s5D8 Mwth9QJG5lWMkqkFxbnpqcWmBYZ5qeXwSE7Oz93ECE6hWp47GO8++KB3iJGJg/EQowQHs5II r0OhfZoQb0piZVVqUX58UWlOavEhRlNggE9klhJNzgcm8bySeEMTSwMTMzMzE0tjM0Mlcd7X rXNThATSE0tSs1NTC1KLYPqYODilGpj4bihtuvFjw5SG5UV8YQFFelpfcmx3Cs196D/zgvGn W7MC/zdIHFTj/vHK96vYrM2uIbGijcLOmu3d/Mab/O4dXvz4q0Don6mbpOv/PvpkeDPmwaeJ SxTjKw90PgjQb7wpKfJTx++F3TvuuTI/vYXsHx7bdeHwtUydiOvbQoVuXrqx55a6oVXg50sN e7d++W31npt5goinuI5RxtGkPFl5fia3F4+Fn95MWyfxeUu6cdzb1i0tzg+3rFqfF8uhKyGz JI6x4n+w16IFWzjMzk+Z9cm5afvRF5wNFx1ONm3p/fD81+e3tx7d/ugZ55G56lnTEWWf3K33 fIrNrcM+l845ohCb9n2m82rJmNd82vdmK7EUZyQaajEXFScCABGgs0YqBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsWy7bCSvO6ns45pBu0r9S3mrNrGaLH6bj+b xem/j1ksbh7YyWTxrvUci8XR/2/ZLH5132W02PrlK6vFs72cFmcnfGB14PLYOesuu8fls6Ue fVtWMXp83iQXwBLFZZOSmpNZllqkb5fAlTF71kLGgq1GFXdXP2BqYHyg3sXIySEhYCLR8Wg+ SxcjF4eQwG5GiUP3V7JDJKQlOg61QtnCEiv/PWeHKPrBKLHh9GMmkASbgI7E7xW/gLo5OEQE dCUa7yqA1DALrGKUuPq+nRWkRlggQOLMiXksIDaLgKrEpP9fmEFsXgFbiYUvJ7FCLJCX2H/w LFicU8BOYueeP2C2EFDNqcOb2SDqBSVOznwCNocZqL5562zmCYwCs5CkZiFJLWBkWsUomVpQ nJueW2xYYJSXWq5XnJhbXJqXrpecn7uJERzmWlo7GPes+qB3iJGJg/EQowQHs5IIr0OhfZoQ b0piZVVqUX58UWlOavEhRmkOFiVx3m+ve1OEBNITS1KzU1MLUotgskwcnFINTFwqQcdPx0+e v8uckelm2F/2hRzc8lWr35RpKNh17fvFfX+hgfbkjTYtH0uvVK5rlgpjfJUSP/V2UpZg+Gul i1z8LQLLn/1Iy5HQXDdjGrME94SZJlf438QtYtQsvzPjgtukKsbFtgKOYXOd8g/d4nr0Tqjo wvNrFSuXdN8M19IXWjBt+dwpZw777P94NHim5SrVtvqI6qqYl2/5dzxVaPtxVYn9uzlb1I0r LidKH8967pLx2zb0ue/Wv29WJC9N+SXb+Or84mW9xqd7zNqrXr0zuVK2blU3m42O8+LHFzfF 9Vw93bcv7CLnRGf5tMvCcnO23vjJK2aqzb/64KOFeS2fsj5e5E25NP1Qi8sfm3wlluKMREMt 5qLiRACb0a8e4gIAAA== X-CMS-MailID: 20240513082314epcas5p309fa70575596792b5c9923ce76a3778f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240513082314epcas5p309fa70575596792b5c9923ce76a3778f References: <20240513082300.515905-1-cliang01.li@samsung.com> Add a test file for hugepage registered buffers, to make sure the fixed buffer coalescing feature works safe and soundly. Testcases include reading/writing with single/multiple hugepage buffers. Signed-off-by: Chenliang Li --- test/Makefile | 1 + test/fixed-hugepage.c | 236 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 237 insertions(+) create mode 100644 test/fixed-hugepage.c base-commit: 23b43f44f882e48a725ad87f8f22722c40743dec diff --git a/test/Makefile b/test/Makefile index 94bdc25..364514d 100644 --- a/test/Makefile +++ b/test/Makefile @@ -88,6 +88,7 @@ test_srcs := \ file-update.c \ file-verify.c \ fixed-buf-iter.c \ + fixed-hugepage.c \ fixed-link.c \ fixed-reuse.c \ fpos.c \ diff --git a/test/fixed-hugepage.c b/test/fixed-hugepage.c new file mode 100644 index 0000000..4fe45a2 --- /dev/null +++ b/test/fixed-hugepage.c @@ -0,0 +1,236 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Test fixed buffers consisting of hugepages. + */ +#include +#include +#include +#include +#include + +#include "liburing.h" +#include "helpers.h" + +#define PAGE_SIZE 4096 +#define HUGEPAGE_SIZE (512 * PAGE_SIZE) +#define NR_BUFS 1 +#define IN_FD "/dev/urandom" +#define OUT_FD "/dev/zero" + +static int open_files(int *fd_in, int *fd_out) +{ + *fd_in = open(IN_FD, O_RDONLY, 0644); + if (*fd_in < 0) { + perror("open in"); + return 1; + } + + *fd_out = open(OUT_FD, O_RDWR, 0644); + if (*fd_out < 0) { + perror("open out"); + return 1; + } + + return 0; +} + +static int mmap_hugebufs(struct iovec *iov, int nr_bufs, size_t buf_size) +{ + int i; + + for (i = 0; i < nr_bufs; i++) { + void *base = NULL; + + base = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); + if (!base || base == MAP_FAILED) { + fprintf(stderr, "Error in mmapping the %dth buffer", i); + return 1; + } + + iov[i].iov_base = base; + iov[i].iov_len = buf_size; + memset(iov[i].iov_base, 0, buf_size); + } + + return 0; +} + +static int do_read(struct io_uring *ring, int fd, struct iovec *iov, + int nr_bufs, size_t buf_size) +{ + struct io_uring_sqe *sqe; + struct io_uring_cqe *cqe; + int i, ret; + + for (i = 0; i < nr_bufs; i++) { + sqe = io_uring_get_sqe(ring); + if (!sqe) { + fprintf(stderr, "Could not get SQE.\n"); + return 1; + } + + io_uring_prep_read_fixed(sqe, fd, iov[i].iov_base, buf_size, 0, i); + io_uring_submit(ring); + + ret = io_uring_wait_cqe(ring, &cqe); + if (ret < 0) { + fprintf(stderr, "Error waiting for completion: %s\n", strerror(-ret)); + return 1; + } + + if (cqe->res < 0) { + fprintf(stderr, "Error in async operation: %s\n", strerror(-cqe->res)); + return 1; + } + + io_uring_cqe_seen(ring, cqe); + } + + return 0; +} + +static int do_write(struct io_uring *ring, int fd, struct iovec *iov, + int nr_bufs, size_t buf_size) +{ + struct io_uring_sqe *sqe; + struct io_uring_cqe *cqe; + int i, ret; + + for (i = 0; i < nr_bufs; i++) { + sqe = io_uring_get_sqe(ring); + if (!sqe) { + fprintf(stderr, "Could not get SQE.\n"); + return 1; + } + + io_uring_prep_write_fixed(sqe, fd, iov[i].iov_base, buf_size, 0, i); + io_uring_submit(ring); + + ret = io_uring_wait_cqe(ring, &cqe); + if (ret < 0) { + fprintf(stderr, "Error waiting for completion: %s\n", strerror(-ret)); + return 1; + } + + if (cqe->res < 0) { + fprintf(stderr, "Error in async operation: %s\n", strerror(-cqe->res)); + return 1; + } + + io_uring_cqe_seen(ring, cqe); + } + + return 0; +} + +static int test_one_hugepage(struct io_uring *ring, int fd_in, int fd_out) +{ + struct iovec iov[NR_BUFS]; + int ret; + size_t buf_size = HUGEPAGE_SIZE; + + ret = mmap_hugebufs(iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "Buffer allocating for one hugepage failed."); + return 1; + } + + ret = io_uring_register_buffers(ring, iov, NR_BUFS); + if (ret) { + fprintf(stderr, "Error registering buffers for one hugepage test: %s", strerror(-ret)); + return 1; + } + + ret = do_read(ring, fd_in, iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "One hugepage read test failed"); + return ret; + } + + ret = do_write(ring, fd_out, iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "One hugepages write test failed"); + return ret; + } + + ret = io_uring_unregister_buffers(ring); + if (ret) { + fprintf(stderr, "Error unregistering buffers for one hugepage test: %s", strerror(-ret)); + return 1; + } + + return 0; +} + +static int test_multi_hugepages(struct io_uring *ring, int fd_in, int fd_out) +{ + struct iovec iov[NR_BUFS]; + int ret; + size_t buf_size = 4 * HUGEPAGE_SIZE; + + ret = mmap_hugebufs(iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "mmap multi hugepages failed."); + return 1; + } + + ret = io_uring_register_buffers(ring, iov, NR_BUFS); + if (ret) { + fprintf(stderr, "Error registering buffers for multi hugepages test: %s", strerror(-ret)); + return 1; + } + + ret = do_read(ring, fd_in, iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "multi hugepages read test failed"); + return ret; + } + + ret = do_write(ring, fd_out, iov, NR_BUFS, buf_size); + if (ret) { + fprintf(stderr, "multi hugepages write test failed"); + return ret; + } + + ret = io_uring_unregister_buffers(ring); + if (ret) { + fprintf(stderr, "Error unregistering buffers for multi hugepages test: %s", strerror(-ret)); + return 1; + } + + return 0; +} + +int main(int argc, char *argv[]) +{ + struct io_uring ring; + int ret, fd_in, fd_out; + + if (argc > 1) + return T_EXIT_SKIP; + + if (open_files(&fd_in, &fd_out)) + return T_EXIT_FAIL; + + ret = t_create_ring(8, &ring, 0); + if (ret == T_SETUP_SKIP) + return T_EXIT_SKIP; + else if (ret < 0) + return T_EXIT_FAIL; + + ret = test_one_hugepage(&ring, fd_in, fd_out); + if (ret) { + fprintf(stderr, "Test one hugepage failed"); + return T_EXIT_FAIL; + } + + ret = test_multi_hugepages(&ring, fd_in, fd_out); + if (ret) { + fprintf(stderr, "Test multi hugepages failed"); + return T_EXIT_FAIL; + } + + io_uring_queue_exit(&ring); + return T_EXIT_PASS; +}