From patchwork Tue Jul 23 05:56:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13739380 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3187C13D8B3 for ; Tue, 23 Jul 2024 06:28:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721716117; cv=none; b=eTI7K/VYUq0Jm1Vj0Mjfg7HFDdaexYVo2/+5fVd/jm7JGfml5jeh3zbUa1EDWf8IqLY4Xz4WGklE76x3b7QbgozxwnneirFQdcsvLgycLOYnBNMkIgyYe6Rm4rJxMkym4TDoppcgecfQdcCutHLNhHquRwYjmA6ezjk5lbrmmqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721716117; c=relaxed/simple; bh=isk6Xphu7apU7I5W/dqwpKBlxbGhc14gmtvZjJp8L60=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=NF5uHDEdVlcNtG4McQ2CJnXE76Bn/PpTv/ztHeMLrzhB6K69GdhhjHw0uC3kGrLwuEhqKkv8v8UZA2eqEn1MOucZe2CajWZFReV7hefhhPyMREvHV43fcHYZfWwMKAlvCOa5FcFE/VPYlC6gaK0UN0yY6SjVRSLpBVoiSRKMcw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=RYwDvYOM; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="RYwDvYOM" Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20240723062827epoutp0277013092548a239d9e9c76877702ec84~kw61uoMPJ2267922679epoutp02k for ; Tue, 23 Jul 2024 06:28:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20240723062827epoutp0277013092548a239d9e9c76877702ec84~kw61uoMPJ2267922679epoutp02k DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1721716107; bh=vvDmPE9fHChoq3JiMipwa+hpODyfu2IxOaDRAQSotjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RYwDvYOM5nnkL6l/YUq6km06iOlwqcP40GnOEK7jvLUk1R9xgZ3LbZNd7GIeMOpYi NCh1x/x2UoWoVTgqePqwrwmtr5L8xjXxiMNrjBaFAx7M4X1zpL4xaSmM+0+OI3YRWA WuNZwv5zhtV3PVaunsBH9wJ2chd/F9lfmJyUSsf4= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20240723062826epcas5p18c6bdd38342267ba16b1a023531e26a7~kw61ZbPvZ1238012380epcas5p1D; Tue, 23 Jul 2024 06:28:26 +0000 (GMT) Received: from epsmgec5p1-new.samsung.com (unknown [182.195.38.178]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4WSnK41p8rz4x9Pp; Tue, 23 Jul 2024 06:28:24 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmgec5p1-new.samsung.com (Symantec Messaging Gateway) with SMTP id 3B.30.19863.88D4F966; Tue, 23 Jul 2024 15:28:24 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20240723055625epcas5p105f302ac966b9bef5925b1e718acd453~kwe35po7Z2316223162epcas5p1w; Tue, 23 Jul 2024 05:56:25 +0000 (GMT) Received: from epsmgmc1p1new.samsung.com (unknown [182.195.42.40]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20240723055625epsmtrp17ea97a623aad47242adc6d4b29dfdab3~kwe34_1wG1884918849epsmtrp1Y; Tue, 23 Jul 2024 05:56:25 +0000 (GMT) X-AuditID: b6c32a50-c73ff70000004d97-10-669f4d8838eb Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgmc1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F5.B3.07567.9064F966; Tue, 23 Jul 2024 14:56:25 +0900 (KST) Received: from lcl-Standard-PC-i440FX-PIIX-1996.. (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240723055623epsmtip2bb15e081f7b1027c6dad1a8ee347bc88~kwe2jVPKl1406214062epsmtip2t; Tue, 23 Jul 2024 05:56:23 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [Patch v7 1/2] io_uring/rsrc: store folio shift and mask into imu Date: Tue, 23 Jul 2024 13:56:15 +0800 Message-Id: <20240723055616.2362-2-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240723055616.2362-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprDJsWRmVeSWpSXmKPExsWy7bCmlm6H7/w0g41/OSyaJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IyX8w+yFTSKVtxbvoO9gXG2QBcjJ4eEgInEzQPH2UFsIYE9jBIn b3tB2J8YJa7t4YewvzFK7DqbBFM/+eFsxi5GLqD4XkaJGZ8mMkM4TUwSq56cYQKpYhPQkfi9 4hcLiC0ioC3x+vFUMJtZYBejxMJzUiC2sICXxKxfWxlBbBYBVYlpf++AXcErYC0xf88hVoht 8hL7D55lBrE5BWwkHp+bxAhRIyhxcuYTqJnyEs1bZ4MdISHwl12ibd9tNohmF4mn7UcYIWxh iVfHt7BD2FISL/vbgGwOILtYYtk6OYjeFkaJ9+/mQNVbS/y7socFpIZZQFNi/S59iLCsxNRT 65gg9vJJ9P5+wgQR55XYMQ/GVpW4cHAb1CppibUTtjJD2B4Suz79ZYUEVj8wpC9OYZnAqDAL yT+zkPwzC2H1AkbmVYxSqQXFuempyaYFhrp5qeXwSE7Oz93ECE6mWgE7GFdv+Kt3iJGJg/EQ owQHs5II75NXc9OEeFMSK6tSi/Lji0pzUosPMZoCQ3wis5Rocj4wneeVxBuaWBqYmJmZmVga mxkqifO+bp2bIiSQnliSmp2aWpBaBNPHxMEp1cA0u2BmyUWHxVY9qWFOV01WBQY+7OCqFJhq p6/xd4am8NyGlMgM0SniSmm9yX+mPL18g7Nm0TzX9WXK+69c+LrQ9v2OGbJ/C2tunOaMvnei 6LCashCb32LrF/Mnqj5undqt7hKc1VPq27NxxucPS/S6pTY+ONqvzL72ywU/zdwZq8WnXt+e N7//cBjPpI8a835az/Vcsf7RyTU3vZIMgreUz33Uz+Ew+8KsRRGzOWSYOjlK/8r9vno+6fz2 80+eKno0T8kUehkoqvOLLSIzSsP4oDKDk+G1G/d5ne7mPBU/8fz5x80Hr2dmf43Zp6r27b9Y sfiDO6nPeNe/uCg6iV2448E9jfc/CvgE5Q+VNLWLKLEUZyQaajEXFScCAEWt3vAvBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrCLMWRmVeSWpSXmKPExsWy7bCSvC6n2/w0gxNzRC2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxsv5B9kKGkUr7i3fwd7AOFugi5GTQ0LA RGLyw9mMXYxcHEICuxkl3l9ewwqRkJboONTKDmELS6z895wdoqiBSeLupC4WkASbgI7E7xW/ gGwODhEBXYnGuwogNcwChxglmjc0M4LUCAt4Scz6tRXMZhFQlZj29w7YUF4Ba4n5ew5BLZOX 2H/wLDOIzSlgI/H43CSweiGgmsmb1jBC1AtKnJz5BGwvM1B989bZzBMYBWYhSc1CklrAyLSK UTK1oDg3PTfZsMAwL7Vcrzgxt7g0L10vOT93EyM44LU0djDem/9P7xAjEwcj0NEczEoivE9e zU0T4k1JrKxKLcqPLyrNSS0+xCjNwaIkzms4Y3aKkEB6YklqdmpqQWoRTJaJg1OqgWm2+Y5z mlsfB4dOjT37nf/IZMVkpx28mSfYr7XrHYnxY7jkWBXgcq4u40jpL/ktYbF7Fzn5RVnrrFYo Twva94iZjffnDbOoDU2Zu/oK/lS/OPGCU+wR+zP+E6ksVb9NZCcHPhKKWHM9YVGt5LnfTD++ LHWy9Ora1ZJ+RPzX6aN2LFavguvuTTb45eP2ZNXSSvabNbeSz2x9e3/Zl4xyhgtrAirS+Gvf 7PNQNjHO9xBZfSxn9f+JX6fYTLZYbqT4YbvBg2Mzn+z1zkhsOHPZXLL74Z76lx6BOacyajM0 lHy1XjOJbr4rcKew8SWLyr8fsofEfK52C9xmu1lxvjC703ujr6LdLSP9+k2F+ey7PyqxFGck GmoxFxUnAgBdnCLA5wIAAA== X-CMS-MailID: 20240723055625epcas5p105f302ac966b9bef5925b1e718acd453 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240723055625epcas5p105f302ac966b9bef5925b1e718acd453 References: <20240723055616.2362-1-cliang01.li@samsung.com> Store the folio shift and folio mask into imu struct and use it in iov_iter adjust, as we will have non PAGE_SIZE'd chunks if a multi-hugepage buffer get coalesced. Signed-off-by: Chenliang Li Reviewed-by: Anuj Gupta --- io_uring/rsrc.c | 15 ++++++--------- io_uring/rsrc.h | 2 ++ 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 60c00144471a..0d6cda92ba46 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -915,6 +915,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->ubuf = (unsigned long) iov->iov_base; imu->ubuf_end = imu->ubuf + iov->iov_len; imu->nr_bvecs = nr_pages; + imu->folio_shift = PAGE_SHIFT; + imu->folio_mask = PAGE_MASK; *pimu = imu; ret = 0; @@ -1031,23 +1033,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter, * we know that: * * 1) it's a BVEC iter, we set it up - * 2) all bvecs are PAGE_SIZE in size, except potentially the + * 2) all bvecs are the same in size, except potentially the * first and last bvec * * So just find our index, and adjust the iterator afterwards. * If the offset is within the first bvec (or the whole first * bvec, just use iov_iter_advance(). This makes it easier * since we can just skip the first segment, which may not - * be PAGE_SIZE aligned. + * be folio_size aligned. */ const struct bio_vec *bvec = imu->bvec; if (offset < bvec->bv_len) { - /* - * Note, huge pages buffers consists of one large - * bvec entry and should always go this way. The other - * branch doesn't expect non PAGE_SIZE'd chunks. - */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset; @@ -1057,12 +1054,12 @@ int io_import_fixed(int ddir, struct iov_iter *iter, /* skip first vec */ offset -= bvec->bv_len; - seg_skip = 1 + (offset >> PAGE_SHIFT); + seg_skip = 1 + (offset >> imu->folio_shift); iter->bvec = bvec + seg_skip; iter->nr_segs -= seg_skip; iter->count -= bvec->bv_len + offset; - iter->iov_offset = offset & ~PAGE_MASK; + iter->iov_offset = offset & ~imu->folio_mask; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c032ca3436ca..8b029c53d325 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -46,7 +46,9 @@ struct io_mapped_ubuf { u64 ubuf; u64 ubuf_end; unsigned int nr_bvecs; + unsigned int folio_shift; unsigned long acct_pages; + unsigned long folio_mask; struct bio_vec bvec[] __counted_by(nr_bvecs); }; From patchwork Tue Jul 23 05:56:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chenliang Li X-Patchwork-Id: 13739379 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 719C013D8B3 for ; Tue, 23 Jul 2024 06:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721716114; cv=none; b=u8LtR6/EmIciPSvLY2VOYeZOzWyjytAbAnD6xRxUUDxzaA4Jjhoc6WECcgPoGe5F4Sp5A593DSNcKlkVPqPqT1/0wHZ+w7U0Jzf9W73m6pA1qcZra42GJLdHelJmXChL+RvoHgL6nVEHPhFnON9NnjXWP/wXfsO4rzbkIRhhGg8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721716114; c=relaxed/simple; bh=rwSlEH5POIeRfeBUzdsicMXJRZ+TjVooAEvRCUzEhkQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:MIME-Version: Content-Type:References; b=NLGFBnoaWV37TfvJf5GnN5fcqubiUwp19i7NXWSUYuvGLL9Qz5f87LGk7UawR3VQnvFQDjzRm5rVHC/UB24VE+LgF8LNRKKETUL4eijSvD/PY9t+0vn6KjrEKna2y1y+GJY6EmwS7GW5oXT8E5bwWZsxAvZLAGmnsaGWypE+8fA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=O3XyFIqJ; arc=none smtp.client-ip=203.254.224.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="O3XyFIqJ" Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20240723062829epoutp04ac2efb16be0c2e638c224525a4d5cf99~kw630iHYN0182301823epoutp04D for ; Tue, 23 Jul 2024 06:28:29 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20240723062829epoutp04ac2efb16be0c2e638c224525a4d5cf99~kw630iHYN0182301823epoutp04D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1721716109; bh=dRZHvZ8LVGP9ldTit2RBbw55OLuz4Nu4t6v9O5KqMsU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O3XyFIqJIPEIzGYzubCPCuxuQW1qW2JF2BBZ4TK5qSlLqugeiPeWQZjtgPtIdYRD5 8F9kK0BddJj28nverc7/4Qe8G/rtkOWvpn5eVQvLspMWWB6Gd7didPEwetEXpoQ0Ia JDp7CymgamzO4Qjb64ZceCvcJUqxCFKZRdW7lWO4= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20240723062828epcas5p2b74e2e7d5fc661c910a356ecb6ab37a7~kw63LXoDF1918419184epcas5p2j; Tue, 23 Jul 2024 06:28:28 +0000 (GMT) Received: from epsmgec5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4WSnK65w5Kz4x9Q7; Tue, 23 Jul 2024 06:28:26 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmgec5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id AD.E5.08855.A8D4F966; Tue, 23 Jul 2024 15:28:26 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20240723055627epcas5p4e61911110cd5aa41d80f1570d7796bd0~kwe6DJOjs1800918009epcas5p4k; Tue, 23 Jul 2024 05:56:27 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240723055627epsmtrp2503a76cc451a072ba2e220bdf453c70c~kwe6Brb7E2364623646epsmtrp2W; Tue, 23 Jul 2024 05:56:27 +0000 (GMT) X-AuditID: b6c32a44-107ff70000002297-c5-669f4d8a0099 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B8.72.08964.B064F966; Tue, 23 Jul 2024 14:56:27 +0900 (KST) Received: from lcl-Standard-PC-i440FX-PIIX-1996.. (unknown [109.105.118.124]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240723055626epsmtip2f02b9168a5caa0145bda4e2bf840c27d~kwe4g3Zdu1451714517epsmtip2G; Tue, 23 Jul 2024 05:56:25 +0000 (GMT) From: Chenliang Li To: axboe@kernel.dk, asml.silence@gmail.com Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com, joshi.k@samsung.com, kundan.kumar@samsung.com, anuj20.g@samsung.com, gost.dev@samsung.com, Chenliang Li Subject: [Patch v7 2/2] io_uring/rsrc: enable multi-hugepage buffer coalescing Date: Tue, 23 Jul 2024 13:56:16 +0800 Message-Id: <20240723055616.2362-3-cliang01.li@samsung.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240723055616.2362-1-cliang01.li@samsung.com> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNJsWRmVeSWpSXmKPExsWy7bCmum6X7/w0g2cv1C2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wAS1S2TUZqYkpqkUJqXnJ+SmZeuq2Sd3C8c7ypmYGhrqGlhbmSQl5i bqqtkotPgK5bZg7QTUoKZYk5pUChgMTiYiV9O5ui/NKSVIWM/OISW6XUgpScApMCveLE3OLS vHS9vNQSK0MDAyNToMKE7IwnWxYzFhw2rbhw0bKBcZdWFyMnh4SAicSkff1sXYxcHEICuxkl jr5sZYRwPjFKLFtznAXC+cYo8fHObxaYlvkXbkK17AVqWfSAHSQhJNDEJLHnrzyIzSagI/F7 xS+wBhEBbYnXj6eC2cwCuxglFp6TArGFBQIk+v5+A+rl4GARUJVoXOEMEuYVsJb4+uAlO8Qu eYn9B88yg9icAjYSj89NYoSoEZQ4OfMJ1Eh5ieats5lB7pEQ+MkusWjidWaIZheJJ9cuskHY whKvjm+BGiol8fndXjaQvRICxRLL1slB9LYwSrx/N4cRosZa4t+VPSwgNcwCmhLrd+lDhGUl pp5axwSxl0+i9/cTJog4r8SOeTC2qsSFg9ugVklLrJ2wFeocD4m/n1qZIeHWzyjx9Fov4wRG hVlI/pmF5J9ZCKsXMDKvYpRMLSjOTU9NNi0wzEsth8dxcn7uJkZwKtVy2cF4Y/4/vUOMTByM hxglOJiVRHifvJqbJsSbklhZlVqUH19UmpNafIjRFBjeE5mlRJPzgck8ryTe0MTSwMTMzMzE 0tjMUEmc93Xr3BQhgfTEktTs1NSC1CKYPiYOTqkGJmXfAPu7nq13XgvvvN5b9Mny+MnTe3P2 8dcuvle65P8pt01ruA1UP7wI+PN+o3/C5BQfncAvdw5d6joZtSmvptB0dVbdmf3fln1mO/Lk w9kTX+s5JxjMDQ4JEQ5c8zdnGRfnDJPgxEi5LtM24ROXbe61hXhvfNfE2LviS8K6io6/Eq8L HNaF5Zs8nNbh/FGuqVRg6u/6Ncudd/XInZ3NHmjxdxLP5M1cjjmCS6Nu3NnP1vHI8vuM905B hYbqSv17d16/1sG69c8/8a2n55fdSNl20C53i9Ca1FmV190OrmDUOnliLXNVcR9Lxt39Hkah AkLs6+dOu54ttKZScUOdw/fNb8LrDhdsX/o+re+QZKcSS3FGoqEWc1FxIgB2MCzELgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrILMWRmVeSWpSXmKPExsWy7bCSvC632/w0g8M/lS2aJvxltpizahuj xeq7/WwWp/8+ZrG4eWAnk8W71nMsFkf/v2Wz+NV9l9Fi65evrBbP9nJanJ3wgdWB22PnrLvs HpfPlnr0bVnF6PF5k1wASxSXTUpqTmZZapG+XQJXxpMtixkLDptWXLho2cC4S6uLkZNDQsBE Yv6Fm2wgtpDAbkaJle+VIeLSEh2HWtkhbGGJlf+eA9lcQDUNTBJXvtxlBUmwCehI/F7xi6WL kYNDREBXovGuAkgNs8AhRonmDc2MIDXCAn4SW/euZgOpYRFQlWhc4QwS5hWwlvj64CXUfHmJ /QfPMoPYnAI2Eo/PTWKEuMdaYvKmNYwQ9YISJ2c+YQGxmYHqm7fOZp7AKDALSWoWktQCRqZV jJKpBcW56bnFhgWGeanlesWJucWleel6yfm5mxjBoa6luYNx+6oPeocYmTgYgW7mYFYS4X3y am6aEG9KYmVValF+fFFpTmrxIUZpDhYlcV7xF70pQgLpiSWp2ampBalFMFkmDk6pBqatR30z JKftC29+qXsqa+Oka18DNrFV5a5UucI9OYdnUUnOsV+7ez6n/K982fbV/WJijGadq7K9gegN porwAkXhWW+mV++I0u++xlV1eqm0jMvmQIkFpq/uPvt+340h9bGkToy773kLO02LhVISB2TF 87+/5fY04P+Zza17QcjOfN6BHzv4dl9N4QyWKlitw1U747Vz5yHz82c+7Vpm9HS6oWf11w3T Wmxaz2TZRwetfDDhYkJCmV11elHq8fL/zYmeApwCuxiPnlkgeNPwlOlJZeGQtNi+b+dkpokV 9C76sexKtqp+m5h8og/Pow012bvSm2svnE6RO/u2pDzO8vqi8o9awvMqVJPcrjZmK7EUZyQa ajEXFScCAIHo6SjkAgAA X-CMS-MailID: 20240723055627epcas5p4e61911110cd5aa41d80f1570d7796bd0 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20240723055627epcas5p4e61911110cd5aa41d80f1570d7796bd0 References: <20240723055616.2362-1-cliang01.li@samsung.com> Add support for checking and coalescing multi-hugepage-backed fixed buffers. The coalescing optimizes both time and space consumption caused by mapping and storing multi-hugepage fixed buffers. A coalescable multi-hugepage buffer should fully cover its folios (except potentially the first and last one), and these folios should have the same size. These requirements are for easier processing later, also we need same size'd chunks in io_import_fixed for fast iov_iter adjust. Signed-off-by: Chenliang Li --- io_uring/rsrc.c | 134 ++++++++++++++++++++++++++++++++++++------------ io_uring/rsrc.h | 8 +++ 2 files changed, 110 insertions(+), 32 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 0d6cda92ba46..794c046b9247 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -849,6 +849,98 @@ static int io_buffer_account_pin(struct io_ring_ctx *ctx, struct page **pages, return ret; } +static bool io_do_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data, int nr_folios) +{ + struct page **page_array = *pages, **new_array = NULL; + int nr_pages_left = *nr_pages, i, j; + + /* Store head pages only*/ + new_array = kvmalloc_array(nr_folios, sizeof(struct page *), + GFP_KERNEL); + if (!new_array) + return false; + + new_array[0] = compound_head(page_array[0]); + /* + * The pages are bound to the folio, it doesn't + * actually unpin them but drops all but one reference, + * which is usually put down by io_buffer_unmap(). + * Note, needs a better helper. + */ + if (data->nr_pages_head > 1) + unpin_user_pages(&page_array[1], data->nr_pages_head - 1); + + j = data->nr_pages_head; + nr_pages_left -= data->nr_pages_head; + for (i = 1; i < nr_folios; i++) { + unsigned int nr_unpin; + + new_array[i] = page_array[j]; + nr_unpin = min_t(unsigned int, nr_pages_left - 1, + data->nr_pages_mid - 1); + if (nr_unpin) + unpin_user_pages(&page_array[j+1], nr_unpin); + j += data->nr_pages_mid; + nr_pages_left -= data->nr_pages_mid; + } + kvfree(page_array); + *pages = new_array; + *nr_pages = nr_folios; + return true; +} + +static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, + struct io_imu_folio_data *data) +{ + struct page **page_array = *pages; + struct folio *folio = page_folio(page_array[0]); + unsigned int count = 1, nr_folios = 1; + int i; + + if (*nr_pages <= 1) + return false; + + data->nr_pages_mid = folio_nr_pages(folio); + if (data->nr_pages_mid == 1) + return false; + + data->folio_shift = folio_shift(folio); + /* + * Check if pages are contiguous inside a folio, and all folios have + * the same page count except for the head and tail. + */ + for (i = 1; i < *nr_pages; i++) { + if (page_folio(page_array[i]) == folio && + page_array[i] == page_array[i-1] + 1) { + count++; + continue; + } + + if (nr_folios == 1) { + if (folio_page_idx(folio, page_array[i-1]) != + data->nr_pages_mid - 1) + return false; + + data->nr_pages_head = count; + } else if (count != data->nr_pages_mid) { + return false; + } + + folio = page_folio(page_array[i]); + if (folio_size(folio) != (1UL << data->folio_shift) || + folio_page_idx(folio, page_array[i]) != 0) + return false; + + count = 1; + nr_folios++; + } + if (nr_folios == 1) + data->nr_pages_head = count; + + return io_do_coalesce_buffer(pages, nr_pages, data, nr_folios); +} + static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage) @@ -858,7 +950,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; - struct folio *folio = NULL; + struct io_imu_folio_data data; + bool coalesced; *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; if (!iov->iov_base) @@ -873,31 +966,8 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - /* If it's a huge page, try to coalesce them into a single bvec entry */ - if (nr_pages > 1) { - folio = page_folio(pages[0]); - for (i = 1; i < nr_pages; i++) { - /* - * Pages must be consecutive and on the same folio for - * this to work - */ - if (page_folio(pages[i]) != folio || - pages[i] != pages[i - 1] + 1) { - folio = NULL; - break; - } - } - if (folio) { - /* - * The pages are bound to the folio, it doesn't - * actually unpin them but drops all but one reference, - * which is usually put down by io_buffer_unmap(). - * Note, needs a better helper. - */ - unpin_user_pages(&pages[1], nr_pages - 1); - nr_pages = 1; - } - } + /* If it's huge page(s), try to coalesce them into fewer bvec entries */ + coalesced = io_try_coalesce_buffer(&pages, &nr_pages, &data); imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) @@ -909,7 +979,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } - off = (unsigned long) iov->iov_base & ~PAGE_MASK; size = iov->iov_len; /* store original address for later verification */ imu->ubuf = (unsigned long) iov->iov_base; @@ -917,17 +986,18 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->nr_bvecs = nr_pages; imu->folio_shift = PAGE_SHIFT; imu->folio_mask = PAGE_MASK; + if (coalesced) { + imu->folio_shift = data.folio_shift; + imu->folio_mask = ~((1UL << data.folio_shift) - 1); + } + off = (unsigned long) iov->iov_base & ~imu->folio_mask; *pimu = imu; ret = 0; - if (folio) { - bvec_set_page(&imu->bvec[0], pages[0], size, off); - goto done; - } for (i = 0; i < nr_pages; i++) { size_t vec_len; - vec_len = min_t(size_t, size, PAGE_SIZE - off); + vec_len = min_t(size_t, size, (1UL << imu->folio_shift) - off); bvec_set_page(&imu->bvec[i], pages[i], vec_len, off); off = 0; size -= vec_len; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8b029c53d325..048fd6959f7a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -52,6 +52,14 @@ struct io_mapped_ubuf { struct bio_vec bvec[] __counted_by(nr_bvecs); }; +struct io_imu_folio_data { + /* Head folio can be partially included in the fixed buf */ + unsigned int nr_pages_head; + /* For non-head/tail folios, has to be fully included */ + unsigned int nr_pages_mid; + unsigned int folio_shift; +}; + void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx);