From patchwork Wed Jan 25 13:34:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3954C54EAA for ; Wed, 25 Jan 2023 13:35:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A8906B007B; Wed, 25 Jan 2023 08:34:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B50E6B007D; Wed, 25 Jan 2023 08:34:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E90F6B0081; Wed, 25 Jan 2023 08:34:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E97426B007E for ; Wed, 25 Jan 2023 08:34:57 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AD2A2C0994 for ; Wed, 25 Jan 2023 13:34:57 +0000 (UTC) X-FDA: 80393417034.02.99F1D46 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf04.hostedemail.com (Postfix) with ESMTP id 0686840020 for ; Wed, 25 Jan 2023 13:34:55 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=nDXFh8LW; spf=none (imf04.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eatZ3C2tdEkdvrb64DPJf57lpiFsGds2EGzrS1xbOrI=; b=zdE1D+jttmMGSKOU3JJPkyBrE+ATYRBcT0NSWGvrhFlSL7jjxvBLPQUD9yDnu+HES8waxD eR9iy/juxZpQ5gA5n/rp/KF4rPiqz8azZdO+EqP0HN4MWZ2qPh354MYKf3VCIwctnv1fDp Do9TibW1W3PJd/55Qh3XJ/h8IJDMJzw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=nDXFh8LW; spf=none (imf04.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653696; a=rsa-sha256; cv=none; b=AkRLdrs5PneYFA1oEsTwXDD40LgNByn5Blmis1IIB1AdrPN4Iu9AHSk12cBEFfHzIJWjnL 2l/U6HLO80Q/lhAwBcenMKFlHShziN7Ap3TupND9HyGB4NsdE83kncXBxgUUk+yO51YSKZ zmoT4pen3w3fzqg0V/RMVzdc1b4QMHo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=eatZ3C2tdEkdvrb64DPJf57lpiFsGds2EGzrS1xbOrI=; b=nDXFh8LWA1LGc5RQsEKZ0eXn4y OPVffqBjS24TQGQz3FdPT3r1QlObRWw1zBGgczQcuqrM0hVwHR1p0QQkfDBgl5PyOJyUC8uSlN6Wf BoqxcuHsf364qZ/DO7d/ayY38C4qDJBWDh+4mXV9yD9aEAhXVFcAw2z3K9tZvszL7qIpdy+dKJTYQ QOSZLn9G7dp+lkYy6obFD//bxcg0YVpfoFLdrw51es6H1n+EPKRp3mzEUMLp8a61KTfEmYnJBhZKf 4t0GUiY15/ca16Qlgzfa0UHylFIR51q2529/y2EoCY9/qv3XSmxpru60Gd0vznQCPMOXZAtAR4o/b gJyUIuNw==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvH-007P1T-Eg; Wed, 25 Jan 2023 13:34:44 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/7] mpage: stop using bdev_{read,write}_page Date: Wed, 25 Jan 2023 14:34:30 +0100 Message-Id: <20230125133436.447864-2-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Stat-Signature: 9r999kem61zt39ywn7kwxhh6tfao767s X-Rspam-User: X-Rspamd-Queue-Id: 0686840020 X-Rspamd-Server: rspam06 X-HE-Tag: 1674653695-198292 X-HE-Meta: U2FsdGVkX1/vAo1ObwxnAmWeIvFJe3lvYSI7I44Ff1wFXkqRLV1/8xiCbNMzrSOL0JAsDLYOAnQxu3GGK1H18Q38qeKra7Yikxcf5kjdaIHWg1c+J5B3UI+wd8lsW7PPkMGGERrw0BSIFq392uVVOHBcPbl35ZGLsd0pD177L8by/pWZh/ZzeWpHp/Eaac6HDL/kFAJX6Iq716b7lXV9+T4S6vi9bMoWpJBCxQiZoMTQ0QWc0+Xb0D6aBQT4jAzAxiHPbjgPTKbivoTfw1QYlwlB45GsqoYFY2N+U7ZurPfBuFaiEq+pA9IsiksyQvBvGZBTYZMUe5aIq+h8ZdYjjNhZqHjUMvDppOnz1ecH/yxGjJ1FTzIPQQhtJx5rLRFoo5RZSSsatbfSSZnP/H+SmvMZXjVtCuKtcAFPJBkV34DBrgFwpvK5KsdIMkme4cr6LSmoe/F7vev7IaeMB71I0MQEaFrqS6adWWsYAJLtnpyKZ6WOOziLIzQAHJRaTV/0G6nI2w8GqYpkHQ8lceaHOFLfi1CRrHGjw6vo16aCBM7JryuTFEbWnyxyzI+bp/Wx6TsQXl0GQSR3geoRQH9fXmmf9HRVkwZJSOJliQfphybttnp6PAJM5TL0bG4p4CA6J9hfZ9KvJwE9HTkkbBBkpu3MzqTmeQFTFRFIh4peybDAvrUBFhxFZ/zTqwbyYGab2P5O5T7xVA6JSDkrOdo6w7Ye9jXPGZbrhJaoZVabVDTUQD6vi4Y1N2KlQohKMR2HxWqbK7GOKscAJfIi1itLvFREbLI1boZcP8vnP0qLk4D0NQfmDUn6/ID5WerVGJevEMYrgFxGE3P1s68i4TNzgBbGXoRxUuTFfNw6O2zOL/Zx/XYPlfJYgYaX0Dom821nLcoNcGH+m/Eq2SLZFNvfZ3TV+rXLBBJhZRpEMBrkKjQyWtFMWH99T4yqCwlQ0VDGOadoTd4SGI+MlBEOEuE /EVTaiIe OTzGszwgaCMHflnwuQEt8QHxccR2/D8ZcH+gmbvX4fJv4lnjcdIKCirUoKZgP0Y2O045ZDQwkmbPk8b9Dlq82XRcwPA9gU1LgJVQ/4Yh/nQiflqblRKbeYnTrBzU0BhO5mo2MsYXE3GuWLfInhpJ0qncHjYm7cPWzLDMNrSZYRE9Iohd0dXbw6CnAiYs14fh+jMesAi1+KeAKkYMssRA69ylfyVbtZzFQ+siGC86di/33jEQrpBT8lON0CVi7ARrfyWuy/UBUdcoCLaoMnYcnIqZp+mXBXScqWjTh9+h3Aiw21azV04W+6r4ZzWn+zV1BMLIub8TFQ5lYxlfiYPZhQHiu5eFzAW8sle3f045E8MlJlwubsVlZRsdQWdAtOKDiyyx/1EvHBLpZY3R5MdKu8Z/wRH1ggRcsFY2ttV6+OG3WmLxY1TgRIRLWv5s92rxodoVDMnRiYviayUFFr3XXQ5Yb8i/VfsHSzIZG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are micro-optimizations for synchronous I/O, which do not matter compared to all the other inefficiencies in the legacy buffer_head based mpage code. Signed-off-by: Christoph Hellwig Reviewed-by: Dan Williams --- fs/mpage.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/fs/mpage.c b/fs/mpage.c index 0f8ae954a57903..124550cfac4a70 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -269,11 +269,6 @@ static struct bio *do_mpage_readpage(struct mpage_readpage_args *args) alloc_new: if (args->bio == NULL) { - if (first_hole == blocks_per_page) { - if (!bdev_read_page(bdev, blocks[0] << (blkbits - 9), - &folio->page)) - goto out; - } args->bio = bio_alloc(bdev, bio_max_segs(args->nr_pages), opf, gfp); if (args->bio == NULL) @@ -579,11 +574,6 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc, alloc_new: if (bio == NULL) { - if (first_unmapped == blocks_per_page) { - if (!bdev_write_page(bdev, blocks[0] << (blkbits - 9), - page, wbc)) - goto out; - } bio = bio_alloc(bdev, BIO_MAX_VECS, REQ_OP_WRITE | wbc_to_write_flags(wbc), GFP_NOFS); From patchwork Wed Jan 25 13:34:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96503C61DA7 for ; Wed, 25 Jan 2023 13:34:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 115606B0075; Wed, 25 Jan 2023 08:34:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C22E6B0080; Wed, 25 Jan 2023 08:34:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE0D06B007B; Wed, 25 Jan 2023 08:34:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C86356B0075 for ; Wed, 25 Jan 2023 08:34:57 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 92E7CA0518 for ; Wed, 25 Jan 2023 13:34:57 +0000 (UTC) X-FDA: 80393417034.05.FAB592E Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf25.hostedemail.com (Postfix) with ESMTP id 3C66DA0004 for ; Wed, 25 Jan 2023 13:34:55 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=1OkYkmeC; spf=none (imf25.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653695; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T2C8sCZx2VTip67EbxZ++0B2OBKUBUFgCJpLY05V/n4=; b=zwt8ZKK5iPnNm9A/0JS0k0J2BGUTTkmQmefacH72okvq+YLDILvP4/hU7EJ/Ajua6snxIw 3QRSJDXxkG6YimQpMA4Z7uCBBDiOAMXxKzYyxUGxFZNOu5kI6D5xYh97slniRmWtAUw/k9 f4IapEywUewj+hvQ9TCGekEGRcF7bjc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=1OkYkmeC; spf=none (imf25.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653695; a=rsa-sha256; cv=none; b=Jn/3YDKyKRy8UnTbVix6Emlkuxptu7vqrCfDG6NI2E8dWXnNdrXxWbgXS2+Ux9d7QbKrFi 52xgcAju7/xssadSgoOC0arfBRVSnp6So/8HL6w0TPqB3A2/jM3ERGDCqfzwsarsr1axmk fY0n0LXBjoX57TB7cCgIQol3E7YeDC0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=T2C8sCZx2VTip67EbxZ++0B2OBKUBUFgCJpLY05V/n4=; b=1OkYkmeC3ChehQGkCBhUBzHODi QSMoBl9rcdqYF7decsoyhscAM9vM9Hpnw/L4odKQ1gkjYKFwq/0oZkYxYu1aloIMtWAc08/FcY9ZL nSts4tqADKXgGbJWMAO8oSrFfsj2Yu7AtBHdLX8S1kDMvbf1Q1zrwua6XOWKJjmu1hStbMqfuaBsY tOmLvdjNBUEzgRogLqECDJPUjmGuxm3RCnXRaS/03B4XTS5sSIRIYxGfSgQ/f+q+LMmQ7E7fVG/ux gn9BdYL2Lq9oN8+PWNupsXpCM8PU0ttxtNTKJ6HOlMnWPrY11+sneV5vmaRixQQ0r4aNAUW5GTqng TN1Ky4mw==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvK-007P1y-Ni; Wed, 25 Jan 2023 13:34:47 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/7] mm: remove the swap_readpage return value Date: Wed, 25 Jan 2023 14:34:31 +0100 Message-Id: <20230125133436.447864-3-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3C66DA0004 X-Stat-Signature: xyypa6xjr3sh6e533pwjpy86tyuhobz9 X-Rspam-User: X-HE-Tag: 1674653695-763678 X-HE-Meta: U2FsdGVkX19z7zKRJfnEGRGY2uJMMG5q/akDdObePZ6M62MchHa3x2VNckZTbBnEKLf8NNH5yxvoRxqYGmthfiml5uuR+0C5otCxUArBJ9R8UTxc2NO5SAbz/4FdSX+0vLIQdv97uSJ5FcJq6sF/9KJcYq4t9hyvtRjMwVVtGIqRvP8ZKq+D+vLH1V/D8f+ljFIS4fM6PT036TSOLqSkfZFQ8AQWMzwcGmcWs2MZGWt8S9N6N8GcWR+aphDv7adN/SAPDYtRsMr78QyrgJy3UGhkILHNJ2b3TZbm1CNtGsJcUcUWDWWcGIXxc/CYofcBNq5cc8enP+VDLkWiPEUlV7oiK+Yxxvincm3E5uBxuS/qpJyPYeeuOXNp4g+xjGcyCGg+g8aDDODmPdpYUs6ZFzNiMfCD3VgPquqvy3Nct59/O3RFbe/ZVsu+mpTzlwAZSQqhu+J8fF2v4W6UUBNu38e3YLhK2SjbS2b18MxcIL2b2blxND2lg+qFOqD3GVqfrZZYZTVQcmiWREj37mKbN/sDSSUPzZ9apQ5OhvuJVra+I6XBxNPvMHg2ONFTUEnRKqZXepjJg0/xEr7J5QfpR4i1aglEL0VRCbaUWmaQyhE+1kJsp6HNVKxWo3mGNRT9JxT16cyUbD3nOKxOrTITxqZ1IMujuT8ve9AC/s0177hTyQp3q2wk2TyXJ2+5n0+aka2UJEphuzW/srrRTs9mLaP9fmL4LQFRYXKshNu/CDFV7U4uJjj8oZ1nFkRRwLdm82lKFUXtdMck5i1HRQNkmFak1UFKzsMazZ5dGG5Hb0ge3kKQAl2oq+cEW8nFmO5Fdhr7AHGm9xt0t9MyhXE/6pxSNSpHhYGpSSz6BV0WO2rh0MS6+ljGzJLEDUm6QQ1BGOWG4Ct9ei1p862fR+9qUlfPD1o1QQErse2ZeJdOSE+GKA0EVXFTs5yr3aEgnAj1cQe5Nnkz/JajPqwd4A3 toWsNsfj ASeuxFLE+AjRiDflCwKYQ12yfxoYUIuKGBUrFj9myBvTotZj5+oBgqpfTrXK/Brv44PTTCq3EPEbFL6+kNFYxNBLeM5AOQ/WdxIRJoEInJPQPXLKJeCBJumugGrg66bW4MZ3FomhW6o3Bg3xvHItBuLQUySSC5XWShNC9ky6uXeLnH91H+or3dplTj27PjURIzndKH62faIJpRMEgZmkaM2iikOQ5qGdvcBTyw/9IvfbJeKh/zK/+bsBGubRpJQ+UrHpckYzxnUPUP7E3GDT38d/SZmvETZiUO//Vdn2SWedEzJw9a5NxVOzJtRaRb1EevEDMc0DLjvTHHpiPzA+XRtFeh6Mfl6k37MFaFROPw+ouuwQ2rV38K4U/KYAvZZGwvxTXJLayY2sE0cNbezyV8nQVP2QJ4nKdgMm7uEeIA75O6LTFCzpUF32KD8ae4jbQwcTnX4nXPrqpebcxFg8yByCLt9a7VA3YhSW2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: swap_readpage always returns 0, and no caller checks the return value. Signed-off-by: Christoph Hellwig Reviewed-by: Dan Williams --- mm/page_io.c | 16 +++++----------- mm/swap.h | 7 +++---- 2 files changed, 8 insertions(+), 15 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 3a5f921b932e82..6f7166fdc4b2bb 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -445,11 +445,9 @@ static void swap_readpage_fs(struct page *page, *plug = sio; } -int swap_readpage(struct page *page, bool synchronous, - struct swap_iocb **plug) +void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) { struct bio *bio; - int ret = 0; struct swap_info_struct *sis = page_swap_info(page); bool workingset = PageWorkingset(page); unsigned long pflags; @@ -481,15 +479,12 @@ int swap_readpage(struct page *page, bool synchronous, goto out; } - if (sis->flags & SWP_SYNCHRONOUS_IO) { - ret = bdev_read_page(sis->bdev, swap_page_sector(page), page); - if (!ret) { - count_vm_event(PSWPIN); - goto out; - } + if ((sis->flags & SWP_SYNCHRONOUS_IO) && + !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { + count_vm_event(PSWPIN); + goto out; } - ret = 0; bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); bio->bi_iter.bi_sector = swap_page_sector(page); bio->bi_end_io = end_swap_bio_read; @@ -521,7 +516,6 @@ int swap_readpage(struct page *page, bool synchronous, psi_memstall_leave(&pflags); } delayacct_swapin_end(); - return ret; } void __swap_read_unplug(struct swap_iocb *sio) diff --git a/mm/swap.h b/mm/swap.h index f78065c8ef524b..f5eb5069d28c2e 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -8,8 +8,7 @@ /* linux/mm/page_io.c */ int sio_pool_init(void); struct swap_iocb; -int swap_readpage(struct page *page, bool do_poll, - struct swap_iocb **plug); +void swap_readpage(struct page *page, bool do_poll, struct swap_iocb **plug); void __swap_read_unplug(struct swap_iocb *plug); static inline void swap_read_unplug(struct swap_iocb *plug) { @@ -64,8 +63,8 @@ static inline unsigned int folio_swap_flags(struct folio *folio) } #else /* CONFIG_SWAP */ struct swap_iocb; -static inline int swap_readpage(struct page *page, bool do_poll, - struct swap_iocb **plug) +static inline void swap_readpage(struct page *page, bool do_poll, + struct swap_iocb **plug) { return 0; } From patchwork Wed Jan 25 13:34:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 766FDC54E94 for ; Wed, 25 Jan 2023 13:34:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37FBF6B0078; Wed, 25 Jan 2023 08:34:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 30C2B6B007D; Wed, 25 Jan 2023 08:34:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECC196B0078; Wed, 25 Jan 2023 08:34:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C40A46B0073 for ; Wed, 25 Jan 2023 08:34:57 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8DF991201D8 for ; Wed, 25 Jan 2023 13:34:57 +0000 (UTC) X-FDA: 80393417034.09.7E2FAE1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf22.hostedemail.com (Postfix) with ESMTP id DD7E5C0004 for ; Wed, 25 Jan 2023 13:34:55 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=d2zsfJpD; spf=none (imf22.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653695; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ROS65tCompeGK2agNW/ZuvBq/QEcxYeTNpBB9DoF018=; b=w8E1K0caVOb4gcsMjCjawaplw3gJQ9Hf7+UZ8aPhx4Na+r72UY8g0iJbfDHwgryp6bGfTb JEpBcW07SeUozcp+2tUMEbmkj1/qhwJwYmZ40fKIY68yKtKFjeEgyMVqCqrcG2JM7dEaxQ uc97dWPbwgq6EIw20Z8KyQdLyaCG8g8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=d2zsfJpD; spf=none (imf22.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653695; a=rsa-sha256; cv=none; b=DAehqyQJpyuSmc+xpoph4s87MaefrqvNOiz4Yzmt8SOQwZu0Os/iuyHmI40OrsCXHZIFTe phqFnGP4DKk5txEZGsd12d/GKClD+Y+Hw7QZjpMWj/emNO/SqtEP1APdff9EXu6JnewEJ9 TRuYuF4KCsc8ux/KFflf+hHjiMmFO4s= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=ROS65tCompeGK2agNW/ZuvBq/QEcxYeTNpBB9DoF018=; b=d2zsfJpDRFFrzt10Ijye3rXQc2 Gjn+l3XF0voJlrGvZJbRaoDc0JsfwL53ZGXWNBzJ6miUPcn0kp8xAKeRout6JP7hGUgUeBnU8SnCL ZwXxcM7tOQPx16uvKwZ1a7QYhmAyFXfhWMcHRZLzuQtiHmQQ2dJ1SFoxLWl1VG9hqGtiqbKNwbZ13 az/1JlLHspaGSvs/82k74pRav2m9Z0bMZHevtFB7Ke82yYJE5KNarfHvapG+FrtjCXWOQUJII3U+0 mhXAdCeOuBAczTry7ksQCAQEQQBQKsezhy8QB3hprWYOCwBFc+wH4SeUtfu/DXDkl/f2QTnVwuwAK psNJQ/uQ==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvO-007P2P-RA; Wed, 25 Jan 2023 13:34:51 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/7] mm: factor out a swap_readpage_bdev helper Date: Wed, 25 Jan 2023 14:34:32 +0100 Message-Id: <20230125133436.447864-4-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DD7E5C0004 X-Rspam-User: X-Stat-Signature: kr8fjgcgdb5cktd3ad9857fsarrugqsf X-HE-Tag: 1674653695-785563 X-HE-Meta: U2FsdGVkX1+CjMtQjhWW64TtUat1ZkBEABPmi0I4cH6dA6nczHV1qjgCRFiX0gvV7FLUZB6UIully6PbO92gQVwSXHhGNeyLiqA3koS+3E8hSfp5IKggCBS4FLaBvMIqD4miCsyryCpp0U6/OtxN4WkEmQ5gcJutLj4rFGQ9dMoMT82mQNBfXSxV40rWMcXhAIYhYN0dogYVq3559gxR4Nj8RdcbhQ8tZAcy6lx5fpfTjLRsZcxHxz7suqxGpFZ8mnRhtGLKBcIVLpl0vHMl2qRI5VnNtvJdR0a3PWKI2OSb5QhxDtRnteeG8nNNiE8uV6Fukb2nLKSQribGO9uM5YL57jBLCil1ojiDOPeZsWyq2rXNy1jn+U7ZfOjHkySegXBsi0076E/jDACESRAjg94OZNc+nfBSO+QkbWFhRix3ghcpdWiLzkv4kbxUeE2PI8NXQsH42CPCvSs516vUC599g6GXrviAZK6cxeixHSdeETmwn3xrlyOsCfWH1xDqZOml6/K0g20mqOYNd+Fb/54mQuktmsjnXS5yH1b/2Ik/b/zFLxzybd+LSan2k7LRyOwOC/l214vmCeTRyGww8ItDPslHpE08yUaLLD6SpUx8srMOaUyBVG0Yz7WFLa8mts2Bapa+KhsuJF5CM6WHtRgKpaMBPqBr9HfRzfSA/YckjRxD+jVoZG6c5Z3Pule/fu/k/KiE9Gn6MnJOyPMi1baXqwPzwJvL0C/tlDogUaHZ8bMf6BP+4BtTgm7rqA0ASFZ+SJa3WiFIFiVp1KadYVgn9LW8CALBVvbhgaAwUakxPo2GlAaz/ZBD2+PK+UrSBbSQidjZAu5bO/n0G0fLFIzhv9ekolwiUmm1oPH4Gr2GX32H3N7yKYpdwuFD1qC6LVv9LfV0rah99qNM68GjeO2OseWu1FIRtNQtG6mQzseRk0LaRkfawebWU+bB+izBfbGfz4J8cCbneYNv+Z6 w6eOOnOI gxi581MCYUfeFTT5cqz7Aa6gnOW/y9Qx/NdRiuFExFFFAjXaqzdZN0d6dRsLy57hMsDbre8cMyIy0jbU9vt31P2/FVHW8SIR3wyjtGQf1+CPTcTFZp5EyM5eNzNsQpDUbtXnwxlIUpsbl8bzkb4Tuir4MFpQ+rvvLhLlEVhC8Oh/KUJlWBmKD2PeY1nC/+zKT3LMAfN2EzhAbYBEzfNuYGsOhfwlhWDe6Vd/9QVx0D9lgqF26No7YpuiS0HfQe/SuylG/1XrDn4ywVXnUGQLFEfMYamCWMuumtnIQMLWqSoGLjPoHzxBe+5Z6YFzQQqkuzxRMIABHfeJYQbfHzHNJVPj8lFoFuXd/GXSLIzKzUt6fsgU4i2jhdtK14T+TOr73PJeMrxVcfzyn8c3Csp37kU/nCKqx/88juwO2M5wWBoAM6VS1DRxbA4Qz74DD7Pmd3KsakQ16un9HVicU4P+7ymEC/jZRGzPN+n1R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Split the block device case from swap_readpage into a separate helper, following the abstraction for file based swap and frontswap. Signed-off-by: Christoph Hellwig Reviewed-by: Dan Williams --- mm/page_io.c | 68 +++++++++++++++++++++++++++------------------------- 1 file changed, 35 insertions(+), 33 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 6f7166fdc4b2bb..ce0b3638094f85 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -445,44 +445,15 @@ static void swap_readpage_fs(struct page *page, *plug = sio; } -void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) +static void swap_readpage_bdev(struct page *page, bool synchronous, + struct swap_info_struct *sis) { struct bio *bio; - struct swap_info_struct *sis = page_swap_info(page); - bool workingset = PageWorkingset(page); - unsigned long pflags; - bool in_thrashing; - - VM_BUG_ON_PAGE(!PageSwapCache(page) && !synchronous, page); - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageUptodate(page), page); - - /* - * Count submission time as memory stall and delay. When the device - * is congested, or the submitting cgroup IO-throttled, submission - * can be a significant part of overall IO time. - */ - if (workingset) { - delayacct_thrashing_start(&in_thrashing); - psi_memstall_enter(&pflags); - } - delayacct_swapin_start(); - - if (frontswap_load(page) == 0) { - SetPageUptodate(page); - unlock_page(page); - goto out; - } - - if (data_race(sis->flags & SWP_FS_OPS)) { - swap_readpage_fs(page, plug); - goto out; - } if ((sis->flags & SWP_SYNCHRONOUS_IO) && !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { count_vm_event(PSWPIN); - goto out; + return; } bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); @@ -509,8 +480,39 @@ void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) } __set_current_state(TASK_RUNNING); bio_put(bio); +} + +void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) +{ + struct swap_info_struct *sis = page_swap_info(page); + bool workingset = PageWorkingset(page); + unsigned long pflags; + bool in_thrashing; + + VM_BUG_ON_PAGE(!PageSwapCache(page) && !synchronous, page); + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(PageUptodate(page), page); + + /* + * Count submission time as memory stall and delay. When the device + * is congested, or the submitting cgroup IO-throttled, submission + * can be a significant part of overall IO time. + */ + if (workingset) { + delayacct_thrashing_start(&in_thrashing); + psi_memstall_enter(&pflags); + } + delayacct_swapin_start(); + + if (frontswap_load(page) == 0) { + SetPageUptodate(page); + unlock_page(page); + } else if (data_race(sis->flags & SWP_FS_OPS)) { + swap_readpage_fs(page, plug); + } else { + swap_readpage_bdev(page, synchronous, sis); + } -out: if (workingset) { delayacct_thrashing_end(&in_thrashing); psi_memstall_leave(&pflags); From patchwork Wed Jan 25 13:34:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CB37C27C76 for ; Wed, 25 Jan 2023 13:35:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A13C76B007D; Wed, 25 Jan 2023 08:35:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 974946B007E; Wed, 25 Jan 2023 08:35:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83C4D6B0080; Wed, 25 Jan 2023 08:35:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 700686B007D for ; Wed, 25 Jan 2023 08:35:02 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4456880238 for ; Wed, 25 Jan 2023 13:35:02 +0000 (UTC) X-FDA: 80393417244.17.123E270 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf05.hostedemail.com (Postfix) with ESMTP id 0078D10001D for ; Wed, 25 Jan 2023 13:34:59 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jpPymmd9; spf=none (imf05.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jVkXgiX++B6ZRBSVBAQLBWXDo+TKW/LQnC8PttvWfoI=; b=ANmSaHOdvibZTnmw9mzCZzKtRksME/Pp5bWyooHv4luhFCWqWrVtJYZGFMBsaGz8G79HpO VBN+apGNnIf1mrVg6VB7JiSThsd8FKUPypbhI/th1bgHHlKm9VRxIBtj0xYy2o+aeb8qv+ GmSGw4MpaNVAHMDLVjue/DyR2ryFGtE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jpPymmd9; spf=none (imf05.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653700; a=rsa-sha256; cv=none; b=vFYRSMAK5Sx2rTAHVr2voGF/jcUoDyrgBEpeGf1gKpaO50wxpJw+DdY/hGC6wUVK4MOHph +7JJb0Tk2/tEy4E73rQNnQ/gNSEE4OD29Nv4e2FsrWue99AsS1BwTKEthGdX7la2+xKWLV t7mjCaA7WSiiHtmJaQXYQkU8D5anfEQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=jVkXgiX++B6ZRBSVBAQLBWXDo+TKW/LQnC8PttvWfoI=; b=jpPymmd9O3sK/QGD+ihLjY6MQw DujfARMhHkJboFbptXRG3wcOjROYaih0gEIyYQ62LZVDUAgArAKCyD8OCwjzY6z/naqsY2qirNzk6 ilVJSvbs/au6I8kbHKpxEMoXr4kQ+PIXBx1Xx05PUmL47nkfz1F+kf+Vv2AKaDc+l3ObIsNC38cn8 JxvQTVAJfCpZs6htcmZCXlun2hpg6aZGiqB+MjWGH2YZ1xt9fR7clMZ5B8g0ihcuH/72NKZHCUQHv wY9Mvjy501914NspkIOys3opWFybPv4UVzJIlny+xpsxGrYLH50vRDjbdL9W0IIis2vQexz/hNkUn YYxaSGBA==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvS-007P3i-N5; Wed, 25 Jan 2023 13:34:55 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/7] mm: use an on-stack bio for synchronous swapin Date: Wed, 25 Jan 2023 14:34:33 +0100 Message-Id: <20230125133436.447864-5-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Stat-Signature: 46umxtkemwc9cmpahzkwzrbxntkduf61 X-Rspam-User: X-Rspamd-Queue-Id: 0078D10001D X-Rspamd-Server: rspam06 X-HE-Tag: 1674653699-20864 X-HE-Meta: U2FsdGVkX19h4wGPykADunXy+rkMlPDn3eEc+GuJdlmcNGpVpcmu7fpSflymQO95nIhRqhSFDam55czDqUp2PuG2TuT2GKO+6O3yzicFTEPv+lDTCkTcvXPMvE/25wJdD0+Cey2kejHgqQg7YuwFq6djBT+VhfJCZjwhFmLI4aBgYHeuAnj+iRqKK9wIsiB4+39vM/V9u8/x3MzlUZgaoi1wv4KHRsdfm3bRhdvvAZosgNRWRY8ohL69ZdMBzk8Tv2M1MkWLhYMp5P57Rgjymzm1v7x3wvZ1h5tLDPF1f2lx61gxYU+i0ZKOvnE+OveqtEv0FJbcOb5tplpASQG9f104VbCIe+qrK5f3CXCOt8x66pLAQmy2PMJw+m7+y0e7La3WehV1ehSqh9ogWodGYq2d4dpfLjXReu97yorRD7mNwRDMThCTVS/Wzerbm8czAmfTSUEIYujdvdX/zQGdKyGjLzxGp9OvWzTccM9/A0hhatfK78S7amN6pdFNVd5QNMPNnXGI2hAKboFsEeRxu+Ci9kxB9oghmrpBluvVhthyt6L+6/XWoJjJJzPHGozhZ3zA7giSdHY3npaIA/sVIaKRh6mTaoV3ZyMpQeTsBYE5KchzwslRrknG43+ITa2ZAtXB/9f428fB0MuviRLYRvs3fJD0OxYJrccJZgLnjIzSXTVQLXx6vqN6W6K+wsr669K7DBiIgnJ9xkl4Ds/Lxz4xXWWNdziTAnJArIt4Z3k83AUIJVp5mDdz8LWLq6uJv0zTbHasH6ahbMcRwtZGbSMKSAackmeB8B/xBdJkDDvsHTW3tlHXcz4zV7/fNREcV7vP5UAUFg5NgFMYEydJDoNwm68QFoGKrLdqX+4aj4pCtENiEMoNdRcJaR5D8bmr/R7MEBozcega4ikElEUYba2nf4TAJGQrGOJh+6x0QA1T+B4kaSMUxK0qBMoc+2IjTRS1Cy8voZaUG2iebQo bgvWP9My uU+XL7tm0mj0P/+ML82oCIx1abSfr9XJiSK2bZCh7b1zIXgPfieCstOKjuWiUeu+yvo39/GKfkZd0mIsdJmwCmEnn2vJu53oIH3yBmm1Sf1/eI8PkVoV6ujJRMdBS8+kJpKCoNPj5GuZTYqJUZYderJbFJ3PZNEvPMtGOrpil/vRl1K84sL0YpgxTU2crb59fVB09ugXVnLf5tBDTBebwpHomVktInkjP0T0R06qFSOqXzbZeeKgOQ93lyuyjj5PKWbb5d/F3hLjoKF+uVYcaWvG7he8Uw9IgXruI4wSQZPAdkIVugCWgN8CO6+V6FEZYxzWi/v7ltDZaKuy1DQ6tH4o77uD1Bk3Cu9a6y1lleZRRoFa/yGWynB78hAGHpCw9hFGB85QrtKL9YSWHEPax3JbYdu01KPnsmtWVWseC17qfqX/BW3qaj5oarg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Optimize the synchronous swap in case by using an on-stack bio instead of allocating one using bio_alloc. Signed-off-by: Christoph Hellwig --- mm/page_io.c | 69 +++++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 31 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index ce0b3638094f85..21ce4505f00607 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -52,10 +52,9 @@ static void end_swap_bio_write(struct bio *bio) bio_put(bio); } -static void end_swap_bio_read(struct bio *bio) +static void __end_swap_bio_read(struct bio *bio) { struct page *page = bio_first_page_all(bio); - struct task_struct *waiter = bio->bi_private; if (bio->bi_status) { SetPageError(page); @@ -63,18 +62,16 @@ static void end_swap_bio_read(struct bio *bio) pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n", MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), (unsigned long long)bio->bi_iter.bi_sector); - goto out; + } else { + SetPageUptodate(page); } - - SetPageUptodate(page); -out: unlock_page(page); - WRITE_ONCE(bio->bi_private, NULL); +} + +static void end_swap_bio_read(struct bio *bio) +{ + __end_swap_bio_read(bio); bio_put(bio); - if (waiter) { - blk_wake_io_task(waiter); - put_task_struct(waiter); - } } int generic_swapfile_activate(struct swap_info_struct *sis, @@ -445,10 +442,11 @@ static void swap_readpage_fs(struct page *page, *plug = sio; } -static void swap_readpage_bdev(struct page *page, bool synchronous, +static void swap_readpage_bdev_sync(struct page *page, struct swap_info_struct *sis) { - struct bio *bio; + struct bio_vec bv; + struct bio bio; if ((sis->flags & SWP_SYNCHRONOUS_IO) && !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { @@ -456,30 +454,37 @@ static void swap_readpage_bdev(struct page *page, bool synchronous, return; } - bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); - bio->bi_iter.bi_sector = swap_page_sector(page); - bio->bi_end_io = end_swap_bio_read; - bio_add_page(bio, page, thp_size(page), 0); + bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ); + bio.bi_iter.bi_sector = swap_page_sector(page); + bio_add_page(&bio, page, thp_size(page), 0); /* * Keep this task valid during swap readpage because the oom killer may * attempt to access it in the page fault retry time check. */ - if (synchronous) { - get_task_struct(current); - bio->bi_private = current; - } + get_task_struct(current); count_vm_event(PSWPIN); - bio_get(bio); - submit_bio(bio); - while (synchronous) { - set_current_state(TASK_UNINTERRUPTIBLE); - if (!READ_ONCE(bio->bi_private)) - break; + submit_bio_wait(&bio); + __end_swap_bio_read(&bio); + put_task_struct(current); +} + +static void swap_readpage_bdev_async(struct page *page, + struct swap_info_struct *sis) +{ + struct bio *bio; - blk_io_schedule(); + if ((sis->flags & SWP_SYNCHRONOUS_IO) && + !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { + count_vm_event(PSWPIN); + return; } - __set_current_state(TASK_RUNNING); - bio_put(bio); + + bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); + bio->bi_iter.bi_sector = swap_page_sector(page); + bio->bi_end_io = end_swap_bio_read; + bio_add_page(bio, page, thp_size(page), 0); + count_vm_event(PSWPIN); + submit_bio(bio); } void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) @@ -509,8 +514,10 @@ void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) unlock_page(page); } else if (data_race(sis->flags & SWP_FS_OPS)) { swap_readpage_fs(page, plug); + } else if (synchronous) { + swap_readpage_bdev_sync(page, sis); } else { - swap_readpage_bdev(page, synchronous, sis); + swap_readpage_bdev_async(page, sis); } if (workingset) { From patchwork Wed Jan 25 13:34:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6CB4C61D9D for ; Wed, 25 Jan 2023 13:35:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B991C6B007E; Wed, 25 Jan 2023 08:35:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B496B6B0080; Wed, 25 Jan 2023 08:35:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EA936B0081; Wed, 25 Jan 2023 08:35:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 87C236B007E for ; Wed, 25 Jan 2023 08:35:05 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5DDF21C62BB for ; Wed, 25 Jan 2023 13:35:05 +0000 (UTC) X-FDA: 80393417370.20.DFBC0B5 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf02.hostedemail.com (Postfix) with ESMTP id B54A780015 for ; Wed, 25 Jan 2023 13:35:03 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=Qa7NsUs+; spf=none (imf02.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653703; a=rsa-sha256; cv=none; b=1FJwmbn51JZkzW+l1IpfRqWYz7n+qS6pihQmYyWIeAGQpLEZrlyGrRLdVVa1pKw6u0ACtY gqdDalh+Mujow4cGpUtuKyonNyaQrDQG8H5LLDq64szdFXLLTY6Qb0oyAcL4oyXh9/n/UE TAPjHokCnTBY+Ir9Sg+v4HtFQqDAPZA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=Qa7NsUs+; spf=none (imf02.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TjTCQ4W/29YQuTOitm932bfpCXinG0reI1NrYTz/RJs=; b=L9BqFAvRaK6Xha6TK0eaAubWKu602R/27FJcXrmVRIIsYQze5tZ1L9psoJEqclFXgqRJ+Z YHlSBESGZlefCT7gcgLkdOsrVliheqmt4doCnWn6ZNaaPEuUP3FuB82wTh1HBdCBVANP0g paopsXVi+MzOFryNZmaqIFnsJuvHfdA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=TjTCQ4W/29YQuTOitm932bfpCXinG0reI1NrYTz/RJs=; b=Qa7NsUs+tvjV6sn3Kr0WIEmeG9 geCnOc/IfyJxD9jHyWR/aN/A+TdXVZUfPy08CvfCxirk0mn9RmZVet9GdDBUb+pZ15qgVGPBGtzso QHEZQzT6kWyev3o6cAh6dAdjgTHxhev/QUsLJgkklZ0nXSKrdnTAlRntR9SiFK1438rz6lAE2Hpri h6CQfd/Z3iewk34wp9Lp40RKfXjVgKdofOMupCfP8jC+sr9umVu4jN3RDTewqJq03Y6kYUhiohgiI c7CoSzjjxFRQfN6MC3tuXDmzqdxnv2m5DTBHHzUWYe4RaojFo0cRMQ1D8vXnpnI6y8MLcrcx2vNri ukqR28IQ==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvW-007P4q-JB; Wed, 25 Jan 2023 13:34:59 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 5/7] mm: remove the __swap_writepage return value Date: Wed, 25 Jan 2023 14:34:34 +0100 Message-Id: <20230125133436.447864-6-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspam-User: X-Rspamd-Queue-Id: B54A780015 X-Rspamd-Server: rspam01 X-Stat-Signature: fweppowjuegcgdahkywpjerg818fotcs X-HE-Tag: 1674653703-279777 X-HE-Meta: U2FsdGVkX1/NNX3lrC7IzhWhv0lxoJ88sf1PEQpxAZXUfZTrEJnucYIiIZl8NtIjqoBDMxphmP5eU3vxoi1Y9Y4nrb7oxSBuFLqu0Vt7HhppGML4LU2OGh4gFRucVN3PTqHcxnZcSVDFU/t8PcG41AwJo0Cpn578e+1FGhBXFf5hYnrNrBP6ndphyB4RyRSHgyhBEF4OouYQGk2i33yd8mkeKTFzJykkPbXUFzyJ8lZlT6FBji88XfDyHsJUH6wQOGAaCnYljnNktcgsNMyGmdW5ZA23Ak3S4m2iI4+jPmeJN+3MXHGjIuIagolHnwXeAghF1z7/0uqDq75aVHsf1I9gqELuRBTVPdaZXEDvFgT8EgMykdILgm5+6SIl/1XGG29xfCU4w7o4Cjo4P6aY7/auKYidMKB91RfbOT0Es5SA7FSBLf9vZogVAcHPuxQIukX5AOfUsPwv8JMp2ADvtXXjJ6n+R8Vkk1yPQc/ZRJ18ilo+ftozVWgpkLfZPz59KG+hoY1wean6f7TD+adNC4J/NG3ZYK/Fuy9W3YgxRK+jymCnB3fRjVkwyhmn+rlL+fu+jVvwyzkTGSrw9bTsccxY9nY1Zssvth3GEHhJcAExLx5AsF6A0PTv27QAmrfvwXQ4YQGvk4b0oHE8B1TBL8xcPyNXqFOUzMsxgGQy7f3FQvwVfkazWMk4QpvLm3fitaEthzWAe8W1hCkwnOhCbevSlWClB1Qq1nCM69dxjGFgQY6YASFrfqX4shqRwV2gKV/+sy2dwzMNiecVO29EBKdTB6+gMkMTspqw3FNYQyzoaaem0s92/frgyoyvZi5tXm9y4jJKUeKuJxagfuhongDWsc+x8KEOWQXcmqsyzaBWhDjV62zJIQtLPnEL3T/yVx2x3HkOTsPxyktqKjPYeWIHnco+haAa2TFteW+DVjhb0ebFuc2cHGUCILzU4srnnmlXSSHgEsv2aye3uDk bv6CygfN ++cY0jP0Biu1RHyPLErwwTe8uBqLFOcrjZQ4qqMx889vU8ku9roea5yWyNcNdRgYVDM1oyJ9grDe5kyhfS3eGCqiMSQVCQUf8kL2vfb55z318U9sHAxKRh/b4dC3SvMklTqTxoz1tBMRKHSlT+KaRmWcR5w5jN476RX04F1apjkjRd3H6p0RtJdj96a2AiaXAyIyCV/lMdsRRTLX0axLEM729AG88By1jg32LxDkODjFcvRx4zOW+F2cEI/xlk6bVafuJtPehZhXR8Se5kXonf7Sl3oDBvNp85AsRuVn+WQ8KHGRVvKx/gmbNldf6ZTYVR9vkjq+TbFimdGexP6dF2RfxxN3KK5OZsHPTITZvSsrTc4sbJyh9hnsaT6I2xzvBX3gnAkuA2Q0hbWzgqRN6J6WZHyjmiAztLlzPc4eslZ7Vl7kGd0Sn+l7VoHjP+iAqDAnXGyJ6FpU5/Mj3J8o9NXAFgoQ8Y5xtJy8O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __swap_writepage always returns 0. Signed-off-by: Christoph Hellwig --- mm/page_io.c | 23 +++++++++-------------- mm/swap.h | 2 +- 2 files changed, 10 insertions(+), 15 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 21ce4505f00607..c373d5694cdffd 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -178,11 +178,11 @@ int generic_swapfile_activate(struct swap_info_struct *sis, int swap_writepage(struct page *page, struct writeback_control *wbc) { struct folio *folio = page_folio(page); - int ret = 0; + int ret; if (folio_free_swap(folio)) { folio_unlock(folio); - goto out; + return 0; } /* * Arch code may have to preserve more data than just the page @@ -192,17 +192,16 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) if (ret) { folio_mark_dirty(folio); folio_unlock(folio); - goto out; + return ret; } if (frontswap_store(&folio->page) == 0) { folio_start_writeback(folio); folio_unlock(folio); folio_end_writeback(folio); - goto out; + return 0; } - ret = __swap_writepage(&folio->page, wbc); -out: - return ret; + __swap_writepage(&folio->page, wbc); + return 0; } static inline void count_swpout_vm_event(struct page *page) @@ -289,7 +288,7 @@ static void sio_write_complete(struct kiocb *iocb, long ret) mempool_free(sio, sio_pool); } -static int swap_writepage_fs(struct page *page, struct writeback_control *wbc) +static void swap_writepage_fs(struct page *page, struct writeback_control *wbc) { struct swap_iocb *sio = NULL; struct swap_info_struct *sis = page_swap_info(page); @@ -326,11 +325,9 @@ static int swap_writepage_fs(struct page *page, struct writeback_control *wbc) } if (wbc->swap_plug) *wbc->swap_plug = sio; - - return 0; } -int __swap_writepage(struct page *page, struct writeback_control *wbc) +void __swap_writepage(struct page *page, struct writeback_control *wbc) { struct bio *bio; int ret; @@ -348,7 +345,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc) ret = bdev_write_page(sis->bdev, swap_page_sector(page), page, wbc); if (!ret) { count_swpout_vm_event(page); - return 0; + return; } bio = bio_alloc(sis->bdev, 1, @@ -363,8 +360,6 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc) set_page_writeback(page); unlock_page(page); submit_bio(bio); - - return 0; } void swap_write_unplug(struct swap_iocb *sio) diff --git a/mm/swap.h b/mm/swap.h index f5eb5069d28c2e..28be6cb3277fa4 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -17,7 +17,7 @@ static inline void swap_read_unplug(struct swap_iocb *plug) } void swap_write_unplug(struct swap_iocb *sio); int swap_writepage(struct page *page, struct writeback_control *wbc); -int __swap_writepage(struct page *page, struct writeback_control *wbc); +void __swap_writepage(struct page *page, struct writeback_control *wbc); /* linux/mm/swap_state.c */ /* One swap address space for each 64M swap space */ From patchwork Wed Jan 25 13:34:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D18AAC54EAA for ; Wed, 25 Jan 2023 13:35:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B7196B0080; Wed, 25 Jan 2023 08:35:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6682E6B0081; Wed, 25 Jan 2023 08:35:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E19A6B0082; Wed, 25 Jan 2023 08:35:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3DF546B0080 for ; Wed, 25 Jan 2023 08:35:10 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 11379A0B99 for ; Wed, 25 Jan 2023 13:35:10 +0000 (UTC) X-FDA: 80393417580.07.8514FFF Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf27.hostedemail.com (Postfix) with ESMTP id 428A840009 for ; Wed, 25 Jan 2023 13:35:08 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=FZLS7LER; spf=none (imf27.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653708; a=rsa-sha256; cv=none; b=D8X9D2M/0NltbU1/51MBGE0idXZm2XP3xO4Qo6nmazYZp2rVy7Mevaeu0glMxm0EWYWpWv johiyIL9a9HndzAGpIpL/2KtchbCw9gmu1dwcq+bdDGgB3y1GHqZnNEsw2VfVM43L6tqdb cYa63ETu1c03VgU+m7VqoyEGv+WIvpo= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=FZLS7LER; spf=none (imf27.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O8RUMOLSrBkE/AeLTaJ5NRi7CBbI3GgiJS4cpgVv33A=; b=0MqpFXm+kf2tGQfRTHAOofbNLTGlLI6bx8ikYVzLzo7WD3cmN0HCP79GizpWrTY1jzlrjU pW7Qr1DTMlz8awYbSd711S4sj1MaQn+ktgrW1fwG852JCs2BWbf+F5HDXylhdpgXjyp2jL vaA+IwmKAD1VATRf0MZ3D6upgfBUkqA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=O8RUMOLSrBkE/AeLTaJ5NRi7CBbI3GgiJS4cpgVv33A=; b=FZLS7LERrZSdq80qX1+1MvYfm+ tHSMY9a2OAXiuep3YUjzxChyTd7mWMpjgGGK1IWSZCHOrsBc4mYr9O0fcAuBRVeVf/HygYJyfUxBK 6zaLL47c+sAcntjQ0kgrkDmgJ5LiUUvvcUQ8ROLe5lvCOY66Gmlpi0BFWsebpRD9KlaIh5wQ2fCMA jTDLpbWC61h8bEnaKdenCM22qooLGyurBJsPxnpvIj3NMRRDkq7+3Sqyzxa3wXqlXJiM1niLjZMZQ pM7HXBQ9xd74o4URQAYlOsC7tXJS0grlmJQts2LwPMdvETtcJh13rKH47OPAJQn7JzpP5EkFPeOgJ r5n6fmsg==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvZ-007P6r-P0; Wed, 25 Jan 2023 13:35:02 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 6/7] mm: factor out a swap_writepage_bdev helper Date: Wed, 25 Jan 2023 14:34:35 +0100 Message-Id: <20230125133436.447864-7-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspam-User: X-Rspamd-Queue-Id: 428A840009 X-Rspamd-Server: rspam01 X-Stat-Signature: mqa5tkmmd6us1sjkh97hj1u43b9zaues X-HE-Tag: 1674653708-87602 X-HE-Meta: U2FsdGVkX1+Jqf97t1OsgHejakiWC6ihz1cbGVr1p0uzPeM2MnUMX4YfSSvsChq6ZHoYEn3MUdG64wOSQtwQm5y24OITUq62QbXmn/xEt2Gogj/lYpxhCg1dHqK6qwaXHOwp3AGDB/tNoWzeMrw4e/j8p0h64jMnIHvUwFHz0huJYaWP2OW/YL+ECSivRSbnwNwi9owMvySLFdTeXuGFtTfOfoWB2IH34xGElG1kbvNeSdl6Bl7e3GipSY94BGoraznIY11BtQhLWUMtRRClihbV9Lq5XKzRWBPyVGkMU3Ea+PJz27U9N0xH+0Aw+eHQxecRZm2oIwZgIJOPqn1BvCrD+JmMW5NL8R+pkd8EFFC8AD26DHMlx3/9vX9KvVKy3YRnvZI2liuNtT3UzSMh/JMGTu8Ha1Gq5GGzFWj4GNHYFF3dCYNpUPR+tMYEmDOECYCVBFY63IeyNPsRZcBI6RwJMZ8inzpWGzQJz0CtiD+tck2sz7Qko2NkHM7ZJ2QsBWWFISaZ1VhVx1/qgqHmN8QaOUlkQ5H6GPvJ/xLdI+cZye5XPR5oo9wGy/DrMomwgDiRt30qzNYppc4ferZF9RHNqa9qGneZOArWZuG98BUw0BiJwmA6kC5WWddVqurvXIfEIHa9qOkNi7pOsVPRXbBnjDhmdao304allxL3dehjzCqCGqa50TXIeEqMZNPPXLzjfzhLjKh1q9IaWxkn+Np3tWAgZrlejLWkrUqlOrc/pPd61+41eb9KDzDd1HzrxxwgsAwF/oOU4aLewi5Sv9G2kXtRIa0n43nJPZFsJhW1pcfLu8n9tYzv5joCfsCfECwi3aCiRoMv6VSz1h4Lwlkf1GmlYTU6RODmg0yEqf7WNx/F8EENc6Y+OtE6K7y9Nl3kDo6il7d0WT/2YeOanPgXgjTQSJWwSmheCZVklbYyRzhqNgqlgdytcXDlm1GFJo3kGYTRUHuHwT3OM74 xGE1x0DG 2Pm8yI7fSeLrE2CxGtfWXoEdL+/MArNmhuJpVJfZI7jWELTSOwX3gDlHj/IRAHWEYoE5+S0oIWQXBGDO0ijT/IhF/Id6pJUdpr9MDDH4odmK1wFHdxK+RZMGqNFjznTr6RG5sjOpLNN4x2QALp0xk6HWnt7kwRZ8yotFH8Qudoto1gyhxchAVnOkI+xvdESjvf/zlO0HCsdSWI/jWzcap6HTWXKLzEibwc8kzQNNtG8lDf7IElkdg6Y2E9V+TsNNdhzItG3WOU8w8dH8J7f+gdYdHBU5pFu6uf2kuyWI4Z+MPxGls/kseGtK/pX/UowNeVqfEGeT8G5bAxeyyHd34qPWwJ1Bg9kmJ8nTFUjepNYvMe8bxyPFGwkPNrKjWUzpknSKp+0OBNnrY6G7TnAjONpg7pOoD+65eFwaMmu2705zIdyXaiMNnmdqSYk+w3VgfZbYEcG9iIN30n+zthtYD4nNcYYy/Gp6Ujz7f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Split the block device case from swap_readpage into a separate helper, following the abstraction for file based swap. Signed-off-by: Christoph Hellwig --- mm/page_io.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index c373d5694cdffd..2ee2bfe5de0386 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -327,23 +327,12 @@ static void swap_writepage_fs(struct page *page, struct writeback_control *wbc) *wbc->swap_plug = sio; } -void __swap_writepage(struct page *page, struct writeback_control *wbc) +static void swap_writepage_bdev(struct page *page, + struct writeback_control *wbc, struct swap_info_struct *sis) { struct bio *bio; - int ret; - struct swap_info_struct *sis = page_swap_info(page); - - VM_BUG_ON_PAGE(!PageSwapCache(page), page); - /* - * ->flags can be updated non-atomicially (scan_swap_map_slots), - * but that will never affect SWP_FS_OPS, so the data_race - * is safe. - */ - if (data_race(sis->flags & SWP_FS_OPS)) - return swap_writepage_fs(page, wbc); - ret = bdev_write_page(sis->bdev, swap_page_sector(page), page, wbc); - if (!ret) { + if (!bdev_write_page(sis->bdev, swap_page_sector(page), page, wbc)) { count_swpout_vm_event(page); return; } @@ -362,6 +351,22 @@ void __swap_writepage(struct page *page, struct writeback_control *wbc) submit_bio(bio); } +void __swap_writepage(struct page *page, struct writeback_control *wbc) +{ + struct swap_info_struct *sis = page_swap_info(page); + + VM_BUG_ON_PAGE(!PageSwapCache(page), page); + /* + * ->flags can be updated non-atomicially (scan_swap_map_slots), + * but that will never affect SWP_FS_OPS, so the data_race + * is safe. + */ + if (data_race(sis->flags & SWP_FS_OPS)) + swap_writepage_fs(page, wbc); + else + swap_writepage_bdev(page, wbc, sis); +} + void swap_write_unplug(struct swap_iocb *sio) { struct iov_iter from; From patchwork Wed Jan 25 13:34:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35F26C27C76 for ; Wed, 25 Jan 2023 13:35:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAA016B0081; Wed, 25 Jan 2023 08:35:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3CC66B0082; Wed, 25 Jan 2023 08:35:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD3826B0083; Wed, 25 Jan 2023 08:35:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9B8886B0081 for ; Wed, 25 Jan 2023 08:35:15 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 55854A0ADB for ; Wed, 25 Jan 2023 13:35:15 +0000 (UTC) X-FDA: 80393417790.16.1749F6F Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf29.hostedemail.com (Postfix) with ESMTP id 95074120007 for ; Wed, 25 Jan 2023 13:35:13 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=HbM04clQ; spf=none (imf29.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653713; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5e3umRwOKACXFKw82QaWO4SWjP0rM/pbj8DPbGZpua8=; b=vkA+Ja8XOwxXXYHeC+t/HgUAlyd1Q8Ihe6TkmtPIyLyBkvCzb7Z5BF2DrifGN3e6dJfnnT 6Wbyjwkyb3bO93nvuJ7t67xkWc5DVyBFnlQoJLlZAPmimns311iGrHOvvMKP84lohWHhM9 Lh+0eUToQKk9wvHhwdkABV2fyuEwgYU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=HbM04clQ; spf=none (imf29.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653713; a=rsa-sha256; cv=none; b=0yqw7VKTt64qMzDswf80PP/+qKKcdhAjwCxV24NEVnQ7HdUa9u1CYdvl9FM6r/iBW8lU0l 8Hn9lq+R/hBnNtw9W0IqHQaqAppyW836hRL+j05J+TcPEj6B2hWymYAhqdIKww1DR82K5C 0990dG5c9pttGmdyWnWmkDprw6Q3SJc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=5e3umRwOKACXFKw82QaWO4SWjP0rM/pbj8DPbGZpua8=; b=HbM04clQ1tr1H4+Fsuq4fwwRDr BR3RaFyH5ACZMGS8ouq+xMWOJRmcTGO9gU27KDtTnGpiOXvOiJi7m50oIzEXOmoeTS9nBev3VcbPt UK3z/Ug2vtctMFcCxVVFyyhGknUvg8rIpcpXNZ2yoIpvfFQSJ1etXKiG2IjYyZewqnPcYrDZXdNPl qVuW+PKfjWdalrcHBZX+KfUcuKUldfz3JJGXV267Pw15umOs3kzU6X2HXmkECCtMllgl8CZcmg8xk CAQciVZ2kwsUL9nBtVzdU86xxsUKFTRdnonWJprknyXjh5Z81bERiAbC73dYTyvtR9xAM2W64lJvP hmkPEhfw==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvd-007P9D-SE; Wed, 25 Jan 2023 13:35:06 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 7/7] block: remove ->rw_page Date: Wed, 25 Jan 2023 14:34:36 +0100 Message-Id: <20230125133436.447864-8-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 95074120007 X-Rspam-User: X-Stat-Signature: gwx4nompb1ajhqno35ics8aimbymydqo X-HE-Tag: 1674653713-14411 X-HE-Meta: U2FsdGVkX19mVRjqAPjywXs6v7obEGryd+bPQj9x8skTgnp2BB0sz2a30hCkdnRaq9z46Miwlb6KdM8tYe0rsykgUI7Kb+ysijd2kGRr6OjQlLNPuR1KNjp0/HGw69ev/26sz0Cn0j5rU3RXMt4Vsv2dONjpc08GW/fHMdsiLc47Ft2zL3+jxzTxN4E37KNCas3w8cB1X3QVtBQc0CsP0AqHLmOKncL5Ts8VyYWceuJzBEKLLRGdLvNaetqItiCNJfITNDeSI4qfmX9SH4wBnVR57M7FNGbDPc2Z3262OuCSuUrk00p4aPaLsndbuNnOoiiH+fjBuc3+ZJzg5grO2m5rIL9jjPIuPA+y3qMcVq0XWIBlc2CzIlF5qtEKBwmObV0r5tKL9td+FnBZSWaLewux7V4/ZveYNTV2SOrMOmWRePMmemQBrHZv5JAj5nZsEyVmKZPeJi9B96SUMSi/Hkojb96L/ZrzbG+sDyZ6w8818rSLijheWVN648yzhhn9ketxiOGTb4Kou1jlg35k3CWCYEkUFz6HDHuqfoo7drzUQreXy96j9VsXWEzY4oPSAByiQEo4bcew4Wupo3CRIUoFfx16Z01TZr01qAOTDaCmv3HmFloe5oOW0OHUr/6AsqP2W8PKMue0ALP7EzzGAUBmZLeEw9RwuMAcMDUx7n9vNARXpavxMguMcl4SHKGbb/wpE9SNDYUyEN6Hv/4rkMhRJsniqPIKTSss5kFjhrlexPQmEuiYDpBUuvkmvWV6GqT9qWcgehniyC6cd+jLrsmsUxO03d2LTCGgGgBgu+0sh7VwsasWMPP2Cs9y0g2/a4TMvRqIf/G5Yd+ODENZU5u4kkjncGBSmrbO9LBKccJmowrGmxoUA2xtIrFAj8sA6u/QyjpOeXQBZQwWDX5df9+clfndmqkPt7yN59Q63iULmQN/vyRypp8sjUxb2P1oAbs+oi+0OuIClcsOQ5a IMeUs1Ld NrliKoXdIHSuOWTV6Q+fEpf7O5H1fcSiQDMDtkN6JDjnin3qymwfloqwmLvS90MYXU1iCI6RT37jpHa48nNkqQLq8KjgUDrNjyQqUXgwIXeNLyvkGTlwfHPbRg4ey84UM6jccDEcNauX6/POWgqTHXCwMrdptIioRchYp4o+9YcB+YJH2AmLcXkO9Unb7Pql8GHdMajD1wI/EvQaqCey+vRDuPlehLH4W9qL0nFtBbaEB8PgXR0WbIwYaxb9TN2DE0ZOvM0KLEM88sFjL7N+EfCBOgkA7RyZgsquZx+3NiJ2A2V3gGbagvNJJWVsKmXvU74Fj5lLGHR/oqdaBADUGwGFt845iYR+WMkEJ0KsJfuslzrJj2WWVPXroqItETp9UPu+z9624FuSZqMEdjv4TJclE8jYjB+/Dy288XdsxHOAdIY+rL5bP9fNuqPkSmC0Ztwos6L56sx4Ep/lEB7xC8hbkdhzdcPHc7+3vmqhGPDMxjAGNGGPoJfNyAud3eg8yZ+Xd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The ->rw_page method is a special purpose bypass of the usual bio handling path that is limited to single-page reads and writes and synchronous which causes a lot of extra code in the drivers, callers and the block layer. The only remaining user is the MM swap code. Switch that swap code to simply submit a single-vec on-stack bio an synchronously wait on it based on a newly added QUEUE_FLAG_SYNCHRONOUS flag set by the drivers that currently implement ->rw_page instead. While this touches one extra cache line and executes extra code, it simplifies the block layer and drivers and ensures that all feastures are properly supported by all drivers, e.g. right now ->rw_page bypassed cgroup writeback entirely. Signed-off-by: Christoph Hellwig Reviewed-by: Dan Williams --- block/bdev.c | 78 ----------------------------------- drivers/block/brd.c | 15 +------ drivers/block/zram/zram_drv.c | 61 +-------------------------- drivers/nvdimm/btt.c | 16 +------ drivers/nvdimm/pmem.c | 24 +---------- include/linux/blkdev.h | 12 +++--- mm/page_io.c | 53 ++++++++++++++---------- mm/swapfile.c | 2 +- 8 files changed, 44 insertions(+), 217 deletions(-) diff --git a/block/bdev.c b/block/bdev.c index edc110d90df404..1795c7d4b99efa 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -304,84 +304,6 @@ int thaw_bdev(struct block_device *bdev) } EXPORT_SYMBOL(thaw_bdev); -/** - * bdev_read_page() - Start reading a page from a block device - * @bdev: The device to read the page from - * @sector: The offset on the device to read the page to (need not be aligned) - * @page: The page to read - * - * On entry, the page should be locked. It will be unlocked when the page - * has been read. If the block driver implements rw_page synchronously, - * that will be true on exit from this function, but it need not be. - * - * Errors returned by this function are usually "soft", eg out of memory, or - * queue full; callers should try a different route to read this page rather - * than propagate an error back up the stack. - * - * Return: negative errno if an error occurs, 0 if submission was successful. - */ -int bdev_read_page(struct block_device *bdev, sector_t sector, - struct page *page) -{ - const struct block_device_operations *ops = bdev->bd_disk->fops; - int result = -EOPNOTSUPP; - - if (!ops->rw_page || bdev_get_integrity(bdev)) - return result; - - result = blk_queue_enter(bdev_get_queue(bdev), 0); - if (result) - return result; - result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, - REQ_OP_READ); - blk_queue_exit(bdev_get_queue(bdev)); - return result; -} - -/** - * bdev_write_page() - Start writing a page to a block device - * @bdev: The device to write the page to - * @sector: The offset on the device to write the page to (need not be aligned) - * @page: The page to write - * @wbc: The writeback_control for the write - * - * On entry, the page should be locked and not currently under writeback. - * On exit, if the write started successfully, the page will be unlocked and - * under writeback. If the write failed already (eg the driver failed to - * queue the page to the device), the page will still be locked. If the - * caller is a ->writepage implementation, it will need to unlock the page. - * - * Errors returned by this function are usually "soft", eg out of memory, or - * queue full; callers should try a different route to write this page rather - * than propagate an error back up the stack. - * - * Return: negative errno if an error occurs, 0 if submission was successful. - */ -int bdev_write_page(struct block_device *bdev, sector_t sector, - struct page *page, struct writeback_control *wbc) -{ - int result; - const struct block_device_operations *ops = bdev->bd_disk->fops; - - if (!ops->rw_page || bdev_get_integrity(bdev)) - return -EOPNOTSUPP; - result = blk_queue_enter(bdev_get_queue(bdev), 0); - if (result) - return result; - - set_page_writeback(page); - result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, - REQ_OP_WRITE); - if (result) { - end_page_writeback(page); - } else { - clean_page_buffers(page); - unlock_page(page); - } - blk_queue_exit(bdev_get_queue(bdev)); - return result; -} - /* * pseudo-fs */ diff --git a/drivers/block/brd.c b/drivers/block/brd.c index 20acc4a1fd6def..37dce184eb56c6 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -309,23 +309,9 @@ static void brd_submit_bio(struct bio *bio) bio_endio(bio); } -static int brd_rw_page(struct block_device *bdev, sector_t sector, - struct page *page, enum req_op op) -{ - struct brd_device *brd = bdev->bd_disk->private_data; - int err; - - if (PageTransHuge(page)) - return -ENOTSUPP; - err = brd_do_bvec(brd, page, PAGE_SIZE, 0, op, sector); - page_endio(page, op_is_write(op), err); - return err; -} - static const struct block_device_operations brd_fops = { .owner = THIS_MODULE, .submit_bio = brd_submit_bio, - .rw_page = brd_rw_page, }; /* @@ -411,6 +397,7 @@ static int brd_alloc(int i) /* Tell the block layer that this is not a rotational device */ blk_queue_flag_set(QUEUE_FLAG_NONROT, disk->queue); + blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, disk->queue); blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM, disk->queue); err = add_disk(disk); if (err) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e290d6d970474e..4d5af9bbbea649 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1453,10 +1453,6 @@ static int __zram_bvec_read(struct zram *zram, struct page *page, u32 index, /* Slot should be unlocked before the function call */ zram_slot_unlock(zram, index); - /* A null bio means rw_page was used, we must fallback to bio */ - if (!bio) - return -EOPNOTSUPP; - ret = zram_bvec_read_from_bdev(zram, page, index, bio, partial_io); } @@ -2081,61 +2077,6 @@ static void zram_slot_free_notify(struct block_device *bdev, zram_slot_unlock(zram, index); } -static int zram_rw_page(struct block_device *bdev, sector_t sector, - struct page *page, enum req_op op) -{ - int offset, ret; - u32 index; - struct zram *zram; - struct bio_vec bv; - unsigned long start_time; - - if (PageTransHuge(page)) - return -ENOTSUPP; - zram = bdev->bd_disk->private_data; - - if (!valid_io_request(zram, sector, PAGE_SIZE)) { - atomic64_inc(&zram->stats.invalid_io); - ret = -EINVAL; - goto out; - } - - index = sector >> SECTORS_PER_PAGE_SHIFT; - offset = (sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; - - bv.bv_page = page; - bv.bv_len = PAGE_SIZE; - bv.bv_offset = 0; - - start_time = bdev_start_io_acct(bdev->bd_disk->part0, - SECTORS_PER_PAGE, op, jiffies); - ret = zram_bvec_rw(zram, &bv, index, offset, op, NULL); - bdev_end_io_acct(bdev->bd_disk->part0, op, start_time); -out: - /* - * If I/O fails, just return error(ie, non-zero) without - * calling page_endio. - * It causes resubmit the I/O with bio request by upper functions - * of rw_page(e.g., swap_readpage, __swap_writepage) and - * bio->bi_end_io does things to handle the error - * (e.g., SetPageError, set_page_dirty and extra works). - */ - if (unlikely(ret < 0)) - return ret; - - switch (ret) { - case 0: - page_endio(page, op_is_write(op), 0); - break; - case 1: - ret = 0; - break; - default: - WARN_ON(1); - } - return ret; -} - static void zram_destroy_comps(struct zram *zram) { u32 prio; @@ -2290,7 +2231,6 @@ static const struct block_device_operations zram_devops = { .open = zram_open, .submit_bio = zram_submit_bio, .swap_slot_free_notify = zram_slot_free_notify, - .rw_page = zram_rw_page, .owner = THIS_MODULE }; @@ -2389,6 +2329,7 @@ static int zram_add(void) set_capacity(zram->disk, 0); /* zram devices sort of resembles non-rotational disks */ blk_queue_flag_set(QUEUE_FLAG_NONROT, zram->disk->queue); + blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, zram->disk->queue); blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM, zram->disk->queue); /* diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c index 0297b7882e33bd..d5593b0dc7009c 100644 --- a/drivers/nvdimm/btt.c +++ b/drivers/nvdimm/btt.c @@ -1482,20 +1482,6 @@ static void btt_submit_bio(struct bio *bio) bio_endio(bio); } -static int btt_rw_page(struct block_device *bdev, sector_t sector, - struct page *page, enum req_op op) -{ - struct btt *btt = bdev->bd_disk->private_data; - int rc; - - rc = btt_do_bvec(btt, NULL, page, thp_size(page), 0, op, sector); - if (rc == 0) - page_endio(page, op_is_write(op), 0); - - return rc; -} - - static int btt_getgeo(struct block_device *bd, struct hd_geometry *geo) { /* some standard values */ @@ -1508,7 +1494,6 @@ static int btt_getgeo(struct block_device *bd, struct hd_geometry *geo) static const struct block_device_operations btt_fops = { .owner = THIS_MODULE, .submit_bio = btt_submit_bio, - .rw_page = btt_rw_page, .getgeo = btt_getgeo, }; @@ -1530,6 +1515,7 @@ static int btt_blk_init(struct btt *btt) blk_queue_logical_block_size(btt->btt_disk->queue, btt->sector_size); blk_queue_max_hw_sectors(btt->btt_disk->queue, UINT_MAX); blk_queue_flag_set(QUEUE_FLAG_NONROT, btt->btt_disk->queue); + blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, btt->btt_disk->queue); if (btt_meta_size(btt)) { rc = nd_integrity_init(btt->btt_disk, btt_meta_size(btt)); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 96e6e9a5f235d8..ceea55f621cc7f 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -238,28 +238,6 @@ static void pmem_submit_bio(struct bio *bio) bio_endio(bio); } -static int pmem_rw_page(struct block_device *bdev, sector_t sector, - struct page *page, enum req_op op) -{ - struct pmem_device *pmem = bdev->bd_disk->private_data; - blk_status_t rc; - - if (op_is_write(op)) - rc = pmem_do_write(pmem, page, 0, sector, thp_size(page)); - else - rc = pmem_do_read(pmem, page, 0, sector, thp_size(page)); - /* - * The ->rw_page interface is subtle and tricky. The core - * retries on any error, so we can only invoke page_endio() in - * the successful completion case. Otherwise, we'll see crashes - * caused by double completion. - */ - if (rc == 0) - page_endio(page, op_is_write(op), 0); - - return blk_status_to_errno(rc); -} - /* see "strong" declaration in tools/testing/nvdimm/pmem-dax.c */ __weak long __pmem_direct_access(struct pmem_device *pmem, pgoff_t pgoff, long nr_pages, enum dax_access_mode mode, void **kaddr, @@ -310,7 +288,6 @@ __weak long __pmem_direct_access(struct pmem_device *pmem, pgoff_t pgoff, static const struct block_device_operations pmem_fops = { .owner = THIS_MODULE, .submit_bio = pmem_submit_bio, - .rw_page = pmem_rw_page, }; static int pmem_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff, @@ -565,6 +542,7 @@ static int pmem_attach_disk(struct device *dev, blk_queue_logical_block_size(q, pmem_sector_size(ndns)); blk_queue_max_hw_sectors(q, UINT_MAX); blk_queue_flag_set(QUEUE_FLAG_NONROT, q); + blk_queue_flag_set(QUEUE_FLAG_SYNCHRONOUS, q); if (pmem->pfn_flags & PFN_MAP) blk_queue_flag_set(QUEUE_FLAG_DAX, q); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 89f51d68c68ad6..1bffe8f44ae9a8 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -555,6 +555,7 @@ struct request_queue { #define QUEUE_FLAG_IO_STAT 7 /* do disk/partitions IO accounting */ #define QUEUE_FLAG_NOXMERGES 9 /* No extended merges */ #define QUEUE_FLAG_ADD_RANDOM 10 /* Contributes to random pool */ +#define QUEUE_FLAG_SYNCHRONOUS 11 /* always complets in submit context */ #define QUEUE_FLAG_SAME_FORCE 12 /* force complete on same CPU */ #define QUEUE_FLAG_INIT_DONE 14 /* queue is initialized */ #define QUEUE_FLAG_STABLE_WRITES 15 /* don't modify blks until WB is done */ @@ -1252,6 +1253,12 @@ static inline bool bdev_nonrot(struct block_device *bdev) return blk_queue_nonrot(bdev_get_queue(bdev)); } +static inline bool bdev_synchronous(struct block_device *bdev) +{ + return test_bit(QUEUE_FLAG_SYNCHRONOUS, + &bdev_get_queue(bdev)->queue_flags); +} + static inline bool bdev_stable_writes(struct block_device *bdev) { return test_bit(QUEUE_FLAG_STABLE_WRITES, @@ -1396,7 +1403,6 @@ struct block_device_operations { unsigned int flags); int (*open) (struct block_device *, fmode_t); void (*release) (struct gendisk *, fmode_t); - int (*rw_page)(struct block_device *, sector_t, struct page *, enum req_op); int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long); int (*compat_ioctl) (struct block_device *, fmode_t, unsigned, unsigned long); unsigned int (*check_events) (struct gendisk *disk, @@ -1431,10 +1437,6 @@ extern int blkdev_compat_ptr_ioctl(struct block_device *, fmode_t, #define blkdev_compat_ptr_ioctl NULL #endif -extern int bdev_read_page(struct block_device *, sector_t, struct page *); -extern int bdev_write_page(struct block_device *, sector_t, struct page *, - struct writeback_control *); - static inline void blk_wake_io_task(struct task_struct *waiter) { /* diff --git a/mm/page_io.c b/mm/page_io.c index 2ee2bfe5de0386..69931fcbe7ce6e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -28,7 +28,7 @@ #include #include "swap.h" -static void end_swap_bio_write(struct bio *bio) +static void __end_swap_bio_write(struct bio *bio) { struct page *page = bio_first_page_all(bio); @@ -49,6 +49,11 @@ static void end_swap_bio_write(struct bio *bio) ClearPageReclaim(page); } end_page_writeback(page); +} + +static void end_swap_bio_write(struct bio *bio) +{ + __end_swap_bio_write(bio); bio_put(bio); } @@ -327,15 +332,31 @@ static void swap_writepage_fs(struct page *page, struct writeback_control *wbc) *wbc->swap_plug = sio; } -static void swap_writepage_bdev(struct page *page, +static void swap_writepage_bdev_sync(struct page *page, struct writeback_control *wbc, struct swap_info_struct *sis) { - struct bio *bio; + struct bio_vec bv; + struct bio bio; - if (!bdev_write_page(sis->bdev, swap_page_sector(page), page, wbc)) { - count_swpout_vm_event(page); - return; - } + bio_init(&bio, sis->bdev, &bv, 1, + REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc)); + bio.bi_iter.bi_sector = swap_page_sector(page); + bio_add_page(&bio, page, thp_size(page), 0); + + bio_associate_blkg_from_page(&bio, page); + count_swpout_vm_event(page); + + set_page_writeback(page); + unlock_page(page); + + submit_bio_wait(&bio); + __end_swap_bio_write(&bio); +} + +static void swap_writepage_bdev_async(struct page *page, + struct writeback_control *wbc, struct swap_info_struct *sis) +{ + struct bio *bio; bio = bio_alloc(sis->bdev, 1, REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc), @@ -363,8 +384,10 @@ void __swap_writepage(struct page *page, struct writeback_control *wbc) */ if (data_race(sis->flags & SWP_FS_OPS)) swap_writepage_fs(page, wbc); + else if (sis->flags & SWP_SYNCHRONOUS_IO) + swap_writepage_bdev_sync(page, wbc, sis); else - swap_writepage_bdev(page, wbc, sis); + swap_writepage_bdev_async(page, wbc, sis); } void swap_write_unplug(struct swap_iocb *sio) @@ -448,12 +471,6 @@ static void swap_readpage_bdev_sync(struct page *page, struct bio_vec bv; struct bio bio; - if ((sis->flags & SWP_SYNCHRONOUS_IO) && - !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { - count_vm_event(PSWPIN); - return; - } - bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ); bio.bi_iter.bi_sector = swap_page_sector(page); bio_add_page(&bio, page, thp_size(page), 0); @@ -473,12 +490,6 @@ static void swap_readpage_bdev_async(struct page *page, { struct bio *bio; - if ((sis->flags & SWP_SYNCHRONOUS_IO) && - !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { - count_vm_event(PSWPIN); - return; - } - bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); bio->bi_iter.bi_sector = swap_page_sector(page); bio->bi_end_io = end_swap_bio_read; @@ -514,7 +525,7 @@ void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) unlock_page(page); } else if (data_race(sis->flags & SWP_FS_OPS)) { swap_readpage_fs(page, plug); - } else if (synchronous) { + } else if (synchronous || (sis->flags & SWP_SYNCHRONOUS_IO)) { swap_readpage_bdev_sync(page, sis); } else { swap_readpage_bdev_async(page, sis); diff --git a/mm/swapfile.c b/mm/swapfile.c index 908a529bca12c9..dc61050a46d8ef 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3069,7 +3069,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (p->bdev && bdev_stable_writes(p->bdev)) p->flags |= SWP_STABLE_WRITES; - if (p->bdev && p->bdev->bd_disk->fops->rw_page) + if (p->bdev && bdev_synchronous(p->bdev)) p->flags |= SWP_SYNCHRONOUS_IO; if (p->bdev && bdev_nonrot(p->bdev)) {