From patchwork Wed Oct 4 16:53:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13409077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B682EE7C4E2 for ; Wed, 4 Oct 2023 16:53:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C08E6B028C; Wed, 4 Oct 2023 12:53:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D11E6B028E; Wed, 4 Oct 2023 12:53:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 197B56B028F; Wed, 4 Oct 2023 12:53:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0B5596B028C for ; Wed, 4 Oct 2023 12:53:38 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D96DA120395 for ; Wed, 4 Oct 2023 16:53:37 +0000 (UTC) X-FDA: 81308375274.20.353B13B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 4BBF6140004 for ; Wed, 4 Oct 2023 16:53:36 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=P9PqackI; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696438416; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D/8R0cQXGxymytv8qeJZP60gdDV9xdbcF13gSUAJ8s0=; b=nqQNHMVGTEIz6yNz13+XAtFMUMGO5+KRPPlO5tp+qflmswiJUfSoFPu8cMVtt01fjBZZOE V3GHJggmo/3JRLGzCmPLWwaiAif9MkgZwvLhZWPzvh6M2ZzsCZtghUplQ65JIPdJTfD47D zwsfYHr7GMiFEQfbW4z0jFw2P2G1fQw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696438416; a=rsa-sha256; cv=none; b=h5WxJNXgWOqAWhQcnH2C7iht3k887TRmq7ckksSUplABt3KlzZSiWQFRNC18zpIA1cTkBS tXsgScu2GRxxC6/By/2NdtrW1tH75uZCt2sN/ihQdf4LgvM8xWDmLzJV5SWUutqu9OITFS BOyn0xFNxWxbIk6OzT71+RD+zOUk6g4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=P9PqackI; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D/8R0cQXGxymytv8qeJZP60gdDV9xdbcF13gSUAJ8s0=; b=P9PqackIT0b7fZvk2PUiwq2eEL ogL4pXi4p90LlQsy6dqVmscrou91ZCSdQPbrsXf44oAxDyO/BxwBm5Z2vIwKUlQlUDpjFbnFPG9fM BrJ63AckCUHqeJPCN9sjwAn+cG1XEsPXgyCNIWA2qvjPyp6VYgJA83cYUMVSJesCNkv4zWzv7yoTQ gP2Z1xQ6jEcNIyFm3dsMavIcKirjm580itF3FOEGmJRHUDqy4pF9sGf2LiGyEorzm8grd31hr/VVK KTTbbG5C8guCMxuwevbECmyH8mc5IYYAvCK+Lb8Nkenf0880jROkyzHaXctQk/+6ULTCzBapwqBs3 erU9zo2g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qo57f-004SEv-92; Wed, 04 Oct 2023 16:53:19 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, npiggin@gmail.com Subject: [PATCH v2 02/17] iomap: Protect read_bytes_pending with the state_lock Date: Wed, 4 Oct 2023 17:53:02 +0100 Message-Id: <20231004165317.1061855-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231004165317.1061855-1-willy@infradead.org> References: <20231004165317.1061855-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BBF6140004 X-Rspam-User: X-Stat-Signature: adqjfwomuodr97oq8e1pgmke61tz4fgo X-Rspamd-Server: rspam03 X-HE-Tag: 1696438416-486160 X-HE-Meta: U2FsdGVkX1/09d84drkvUY8SGUAnNnl7KmEBpm0da7d2IEUJq9toV3pwE3LiDGP4wLrv27xpXrvc6QIzOw+EMV8ZgnqHlKqxmJLSjgvtIEjLkphJ3qayAGJ2CeZv0Q+0RnDc9TXHDUrEu9boxUDaBmAk+TSXcRryhATu4MltC4kB1XKsudYl102unPMbuojvPGWqdTZNL1K1Hht079B5XjiVVl0KK70y3yo0mlKz2ECENtyVN1QjvGvg+eP41+jO5oUyFOZzde5nlrc7zyWPsu8kcqjQZRfJ7DmnAIK6IbJZ+C1s393/dUnHEQ78qhR/7mEPPWm9ouB8a6jfoRDdRkfleQARjSNfLvW+kJqYhaYNTVRQuxx8u4jA4/nuJZDkTutshbuLyBnl3KNYZTFyePMFbXaekBRDGlI78+wMALA/9EDJw8EO2TRTJ6qoh9g/cG6zG4hkTYHDadMIEa1WoKHY3M7sUa0VI+xSVDbzH4HGfWVrsO7zBBSdtcuhI1rav+LVgDoKVAJHq5UvcI3CaSb2E6/Qi5DGtMVEhSN52zuwhd7/pluBwl+rrqy2C/JzPcQVEhFQ7il6bSHeSLZv7dS2m/CIYXcuqGvOUYjcP/YcA519FjQT9OIbAaOO93yPInjoohWjJvdk8ZQdIx2h+52+GwZkX36hniZfGd583jmwlhD8/cetDLMV538UEIgIi0Y2YEPVN5YMAkGRjQ8pUj9qJvzXKym5GcSgp7Kj5OK6AC+wXF8viMM9m13gN9O9dakqCcGzwgL96YIUuhAcZ8KokezUg8CGqHeljfkgv47X1HT3jzPzBUtZw9tClGsI7V6bboX6NqpxxSNyqn6lzgN2EmPqO7YUJBC4bZEpWvUew3gyLt0Fq7tvuH8GMrSE6QuIY6tv/rdLty9fppGmRn+TXt2jdeY0fVLOmeDozLvtWg7SJBsYtV2tp8F+D569RRawFt5nv+kjjPX4tKC 4y8MHaYJ 9uEHtCBZVvP+g/fHPkFS4oSDEGaEHzrMxU2pq/wven0BFwR0spyEFsCOPUberzSe4+9H1ZJOkxeNHczOuao0S/90hEqGF7Lv57d7TU1Jee/RW0JoukaPQAtLfX8ueEcWAftxqzFt9d6/vI0U3q6yUwEcHgEA3q9nvv9VVV3Dopu/CrQwLUjZk8LP3S63YYGjEhZKAEZDm87TiZs9zE1XIrZY7IZnM6cunQ+HR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Perform one atomic operation (acquiring the spinlock) instead of two (spinlock & atomic_sub) per read completion. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 6e780ca64ce3..4a996c5327ef 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -29,9 +29,9 @@ typedef int (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length); * and I/O completions. */ struct iomap_folio_state { - atomic_t read_bytes_pending; - atomic_t write_bytes_pending; spinlock_t state_lock; + unsigned int read_bytes_pending; + atomic_t write_bytes_pending; /* * Each block has two bits in this bitmap: @@ -183,7 +183,7 @@ static void ifs_free(struct folio *folio) if (!ifs) return; - WARN_ON_ONCE(atomic_read(&ifs->read_bytes_pending)); + WARN_ON_ONCE(ifs->read_bytes_pending != 0); WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending)); WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) != folio_test_uptodate(folio)); @@ -250,19 +250,29 @@ static void iomap_adjust_read_range(struct inode *inode, struct folio *folio, *lenp = plen; } -static void iomap_finish_folio_read(struct folio *folio, size_t offset, +static void iomap_finish_folio_read(struct folio *folio, size_t off, size_t len, int error) { struct iomap_folio_state *ifs = folio->private; + bool uptodate = !error; + bool finished = true; - if (unlikely(error)) { - folio_clear_uptodate(folio); - folio_set_error(folio); - } else { - iomap_set_range_uptodate(folio, offset, len); + if (ifs) { + unsigned long flags; + + spin_lock_irqsave(&ifs->state_lock, flags); + if (!error) + uptodate = ifs_set_range_uptodate(folio, ifs, off, len); + ifs->read_bytes_pending -= len; + finished = !ifs->read_bytes_pending; + spin_unlock_irqrestore(&ifs->state_lock, flags); } - if (!ifs || atomic_sub_and_test(len, &ifs->read_bytes_pending)) + if (error) + folio_set_error(folio); + if (uptodate) + folio_mark_uptodate(folio); + if (finished) folio_unlock(folio); } @@ -360,8 +370,11 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, } ctx->cur_folio_in_bio = true; - if (ifs) - atomic_add(plen, &ifs->read_bytes_pending); + if (ifs) { + spin_lock_irq(&ifs->state_lock); + ifs->read_bytes_pending += plen; + spin_unlock_irq(&ifs->state_lock); + } sector = iomap_sector(iomap, pos); if (!ctx->bio ||