From patchwork Sat Dec 14 03:10:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luis Chamberlain X-Patchwork-Id: 13908275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD98BE7716A for ; Sat, 14 Dec 2024 03:11:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41F786B009C; Fri, 13 Dec 2024 22:11:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 386206B009B; Fri, 13 Dec 2024 22:11:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11DEA6B009A; Fri, 13 Dec 2024 22:11:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DAEB96B008C for ; Fri, 13 Dec 2024 22:11:01 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9D8A01C844F for ; Sat, 14 Dec 2024 03:11:01 +0000 (UTC) X-FDA: 82892087670.29.2AC07F6 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf30.hostedemail.com (Postfix) with ESMTP id 1B53380006 for ; Sat, 14 Dec 2024 03:10:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=VHdDK0Rw; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=quarantine); spf=none (imf30.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734145847; a=rsa-sha256; cv=none; b=1grP4AJwQ2n7WVYV2Nbe9aH38QiYng+QiXzVKaCymhJ8R0UaPml3DtqdI+U913fEGpG4AN VUALdM7GDZxB5l48hSVvz9AcvIVnhJAuMDX5B7Uv2VZvkHtM39K8/Ohys0En5+dez2mY0a OMz4Culz6jukHuQ10Gt1wX5v98XU3xY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=VHdDK0Rw; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=quarantine); spf=none (imf30.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734145847; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=17Q0gy/g/R6GF/kNACXWfTzKrmUaCrFTxm3DWtltqxI=; b=BQCw9Nozsy0CqMLj1RSkGLXfdaEEax4pWNK5+eLosE/MvU6BcfHlOtRdKBzazh2uUi1XVF 8uMCoSlLTbhV/9ql3EQDG/wWUOhf/hPSNNelDOJy+9kTTtzvsEvJwmTAsUragme4WqRVKG XkNu1fmcipHiDDw1z6D7XJLRirDjv9w= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=17Q0gy/g/R6GF/kNACXWfTzKrmUaCrFTxm3DWtltqxI=; b=VHdDK0RwjffflcX0z4byIMeij+ aBC6gTkms+qYDlNfhyjSs4Iliz0FzPjXIe39juSBMwSojG8tpjFyDSN1F1sOdoq2Hbh9yXAmmtlOt usRJ2g2lmhXmusq/xZg+cKiTaGMTY+FQVlDo3OPNgLXemfBWpiTctwVZz8NZvXSWH9g8b9OeoGLXE 5H0355Vu+iAVkzf1E2XE6CSW/DqoZK/gq2UnpMD2T5zGEPbqz6W6d8F2PgzjrXvnaBVpXzpwxgwPW sLKjc5IS86MBxhWVC7oEkQMS+svl9Wx4GyxjP1KXnek3QBbi0tPj2ClewMr8FHSTRGQw/xqCPLI3r u9L99Mgg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tMIYN-00000005c3b-3W4y; Sat, 14 Dec 2024 03:10:51 +0000 From: Luis Chamberlain To: willy@infradead.org, hch@lst.de, hare@suse.de, dave@stgolabs.net, david@fromorbit.com, djwong@kernel.org Cc: john.g.garry@oracle.com, ritesh.list@gmail.com, kbusch@kernel.org, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, gost.dev@samsung.com, p.raghav@samsung.com, da.gomez@samsung.com, kernel@pankajraghav.com, mcgrof@kernel.org Subject: [RFC v2 03/11] fs/buffer: add iteration support for block_read_full_folio() Date: Fri, 13 Dec 2024 19:10:41 -0800 Message-ID: <20241214031050.1337920-4-mcgrof@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241214031050.1337920-1-mcgrof@kernel.org> References: <20241214031050.1337920-1-mcgrof@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1B53380006 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: udmkk5smitntzpd48grpd6sbkj3xxyfo X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1734145808-952775 X-HE-Meta: U2FsdGVkX1/5fpz0Tqrk7Kb2WPHs1Dv/DFcSVGimv2jhnf/ptB+p3UKyAbKmSddogiI2J7kV6QRR8LImoLjD3TfCe20GjH/EYYJqF13iZdr5C933DMTSYab/+848lSbknEziuSxD0gWhENBJmbj3VhdiW9Bajc0KaSZjcXlZIRivDyBLB4lA7jZPdDH0GdL0cdWIOQPKGoKlpmsj39ORbxU59av1mdqXa8gQlJfvtIzanJ7uxTZks7R+4/DLujJ5f0pCIBmGFjlrVi6+r8cS2/9vVW61AfPqZREswzEiUHKtLvAk0Zdcnw9wnXClfeuL3TEszbihMLnYeWecJGuoIIJTTtoEMu6BitwfMPJubYfhs0RAOGhy7COxY9xuYTPDEIexJ8yj/vVAXIrPbIQKixq5v8YVk1T0FcaWCTuqZWzdtxGPJwJvF88BCUQMTCcyC0tHVsk0fNGCZMHFhAKkr44U4QoBZ6HIdCfXKZ/5bkGQ8GGPNnidIw+92i+tqvgpqcOBL+vzzZ94KihaU6Dp7cK/nbj2BRJtjG6sgqb4HpSXUDrysSWloHA0Ue1ZdlmE13GstHPpoaLpkWgHSH9YF5GXmzG/ZeT27Ttb5NrphxUp0jCX/tcmrdCZlQkDeMSvq0C8Qse5R6Et9Xa3tNdhmAJT2yP+3dnndfmx1+TdKy1PdQ7cettUb8so29TXQdUDj1nkTPp+v6I/ANfZWL5LRGFkwiBTy4aWwSULIKzm7f/IHFtOuNxSPOBS/xqL9zJKE53jkELWkfHxgWoGPjjhdUjqtwJleGMVazLKOVv+6pUk19+ganXa4TxIFPbwB/VvnCRuKcDmuxivb5NcASRGg+O9k2WuJtE3L3iCntQeUyY0ZVuY5/nloPAD0X3v5ZjN22j8AJP5IazdCVa6mvwNRhHJjSAi+lFUgXug/fSNr7fvTL3ElFUhYzmeq4NihUPOKlBsgHxwkEo1zjN+7CE YnOR/2Fq mWrNcCfA+1pasLyR62nCyQ2vv22l6pv85QJTT/nJDzLGf5uuN8MFkqspF74+/7FM3bnrI5MzERIG47Ejs3mKI0jT7ygFOxI/eQsDgokXCzaA3OcMXWfLpsFWphY4E1jCrjtFZ0k8Mo/ZLzhDBW0PodVQjitg+uZ8oEh1jiqpJgm/0rTQX6WLtctmBWQZfbhTgNfsyiZmi1pHpW9FV4zQMxMDjoWT09afLw8QOVMMxTXumtWzXHqS5GkyvrItGRQg5um/0nQ+8cQTYbouPipUrCnaDi/GINgf7l4AO8ustTDK4uK2b83AnHk//T+2EA8g3yZcaKxPGurkW9zXRMOADMKU4gVLZAXyenO32WvjRmYHiLgCBrXi19AhF8v9j2HvcO+unhu7MMKh9vLNC8RFf5nagjZ3n2LjR3MyQztcXDgniqI5bUjMjbBkDHfuxTm5wHo+qXM1WgfvhoAmuFvt8LKzOp+bhWp2FwHgM1YH/ZKFYS1kjYjRU6VruzipKBsqm6oFG2TvaEugKq93RV/TJz8cxXq87wYNlphOpLaBj+9yUVRM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide a helper to iterate on buffer heads on a folio. We do this as a preliminary step so to make the subsequent changes easier to read. Right now we use an array on stack to loop over all buffer heads in a folio of size MAX_BUF_PER_PAGE, however on CPUs where the system page size is quite larger like Hexagon with 256 KiB page size support this can mean the kernel can end up spewing spews stack growth warnings. To be able to break this down into smaller array chunks add support for processing smaller array chunks of buffer heads at a time. The used array size is not changed yet, that will be done in a subsequent patch, this just adds the iterator support and logic. While at it clarify the booleans used on bh_read_batch_async() and how they are only valid in consideration when we've processed all buffer-heads of a folio, that is when we're on the last buffer head in a folio: * bh_folio_reads * unmapped Signed-off-by: Luis Chamberlain Reviewed-by: Hannes Reinecke --- fs/buffer.c | 130 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 94 insertions(+), 36 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 108e1c36fc1a..f8e6a5454dbb 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -2397,57 +2397,64 @@ static void bh_read_batch_async(struct folio *folio, #define bh_next(__bh, __head) \ (bh_is_last(__bh, __head) ? NULL : (__bh)->b_this_page) +/* Starts from a pivot which you initialize */ +#define for_each_bh_pivot(__pivot, __last, __head) \ + for ((__pivot) = __last = (__pivot); \ + (__pivot); \ + (__pivot) = bh_next(__pivot, __head), \ + (__last) = (__pivot) ? (__pivot) : (__last)) + /* Starts from the provided head */ #define for_each_bh(__tmp, __head) \ for ((__tmp) = (__head); \ (__tmp); \ (__tmp) = bh_next(__tmp, __head)) +struct bh_iter { + sector_t iblock; + get_block_t *get_block; + bool any_get_block_error; + int unmapped; + int bh_folio_reads; +}; + /* - * Generic "read_folio" function for block devices that have the normal - * get_block functionality. This is most of the block device filesystems. - * Reads the folio asynchronously --- the unlock_buffer() and - * set/clear_buffer_uptodate() functions propagate buffer state into the - * folio once IO has completed. + * Reads up to MAX_BUF_PER_PAGE buffer heads at a time on a folio on the given + * block range iblock to lblock and helps update the number of buffer-heads + * which were not uptodate or unmapped for which we issued an async read for + * on iter->bh_folio_reads for the full folio. Returns the last buffer-head we + * worked on. */ -int block_read_full_folio(struct folio *folio, get_block_t *get_block) -{ - struct inode *inode = folio->mapping->host; - sector_t iblock, lblock; - struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE]; - size_t blocksize; - int nr, i; - int fully_mapped = 1; - bool page_error = false; - loff_t limit = i_size_read(inode); - - /* This is needed for ext4. */ - if (IS_ENABLED(CONFIG_FS_VERITY) && IS_VERITY(inode)) - limit = inode->i_sb->s_maxbytes; - - VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - head = folio_create_buffers(folio, inode, 0); - blocksize = head->b_size; +static struct buffer_head *bh_read_iter(struct folio *folio, + struct buffer_head *pivot, + struct buffer_head *head, + struct inode *inode, + struct bh_iter *iter, sector_t lblock) +{ + struct buffer_head *arr[MAX_BUF_PER_PAGE]; + struct buffer_head *bh = pivot, *last; + int nr = 0, i = 0; + size_t blocksize = head->b_size; + bool no_reads = false; + bool fully_mapped = false; + + /* collect buffers not uptodate and not mapped yet */ + for_each_bh_pivot(bh, last, head) { + BUG_ON(nr >= MAX_BUF_PER_PAGE); - iblock = div_u64(folio_pos(folio), blocksize); - lblock = div_u64(limit + blocksize - 1, blocksize); - nr = 0; - i = 0; - - for_each_bh(bh, head) { if (buffer_uptodate(bh)) continue; if (!buffer_mapped(bh)) { int err = 0; - fully_mapped = 0; - if (iblock < lblock) { + iter->unmapped++; + if (iter->iblock < lblock) { WARN_ON(bh->b_size != blocksize); - err = get_block(inode, iblock, bh, 0); + err = iter->get_block(inode, iter->iblock, + bh, 0); if (err) - page_error = true; + iter->any_get_block_error = true; } if (!buffer_mapped(bh)) { folio_zero_range(folio, i * blocksize, @@ -2465,10 +2472,61 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) } arr[nr++] = bh; i++; - iblock++; + iter->iblock++; + } + + iter->bh_folio_reads += nr; + + WARN_ON_ONCE(!bh_is_last(last, head)); + + if (bh_is_last(last, head)) { + if (!iter->bh_folio_reads) + no_reads = true; + if (!iter->unmapped) + fully_mapped = true; } - bh_read_batch_async(folio, nr, arr, fully_mapped, nr == 0, page_error); + bh_read_batch_async(folio, nr, arr, fully_mapped, no_reads, + iter->any_get_block_error); + + return last; +} + +/* + * Generic "read_folio" function for block devices that have the normal + * get_block functionality. This is most of the block device filesystems. + * Reads the folio asynchronously --- the unlock_buffer() and + * set/clear_buffer_uptodate() functions propagate buffer state into the + * folio once IO has completed. + */ +int block_read_full_folio(struct folio *folio, get_block_t *get_block) +{ + struct inode *inode = folio->mapping->host; + sector_t lblock; + size_t blocksize; + struct buffer_head *bh, *head; + struct bh_iter iter = { + .get_block = get_block, + .unmapped = 0, + .any_get_block_error = false, + .bh_folio_reads = 0, + }; + loff_t limit = i_size_read(inode); + + /* This is needed for ext4. */ + if (IS_ENABLED(CONFIG_FS_VERITY) && IS_VERITY(inode)) + limit = inode->i_sb->s_maxbytes; + + VM_BUG_ON_FOLIO(folio_test_large(folio), folio); + + head = folio_create_buffers(folio, inode, 0); + blocksize = head->b_size; + + iter.iblock = div_u64(folio_pos(folio), blocksize); + lblock = div_u64(limit + blocksize - 1, blocksize); + + for_each_bh(bh, head) + bh = bh_read_iter(folio, bh, head, inode, &iter, lblock); return 0; }