From patchwork Wed Jan 25 13:34:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13115638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CB37C27C76 for ; Wed, 25 Jan 2023 13:35:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A13C76B007D; Wed, 25 Jan 2023 08:35:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 974946B007E; Wed, 25 Jan 2023 08:35:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83C4D6B0080; Wed, 25 Jan 2023 08:35:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 700686B007D for ; Wed, 25 Jan 2023 08:35:02 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4456880238 for ; Wed, 25 Jan 2023 13:35:02 +0000 (UTC) X-FDA: 80393417244.17.123E270 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf05.hostedemail.com (Postfix) with ESMTP id 0078D10001D for ; Wed, 25 Jan 2023 13:34:59 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jpPymmd9; spf=none (imf05.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674653700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jVkXgiX++B6ZRBSVBAQLBWXDo+TKW/LQnC8PttvWfoI=; b=ANmSaHOdvibZTnmw9mzCZzKtRksME/Pp5bWyooHv4luhFCWqWrVtJYZGFMBsaGz8G79HpO VBN+apGNnIf1mrVg6VB7JiSThsd8FKUPypbhI/th1bgHHlKm9VRxIBtj0xYy2o+aeb8qv+ GmSGw4MpaNVAHMDLVjue/DyR2ryFGtE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jpPymmd9; spf=none (imf05.hostedemail.com: domain of BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+926f35216cf1e93e6ac9+7094+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674653700; a=rsa-sha256; cv=none; b=vFYRSMAK5Sx2rTAHVr2voGF/jcUoDyrgBEpeGf1gKpaO50wxpJw+DdY/hGC6wUVK4MOHph +7JJb0Tk2/tEy4E73rQNnQ/gNSEE4OD29Nv4e2FsrWue99AsS1BwTKEthGdX7la2+xKWLV t7mjCaA7WSiiHtmJaQXYQkU8D5anfEQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=jVkXgiX++B6ZRBSVBAQLBWXDo+TKW/LQnC8PttvWfoI=; b=jpPymmd9O3sK/QGD+ihLjY6MQw DujfARMhHkJboFbptXRG3wcOjROYaih0gEIyYQ62LZVDUAgArAKCyD8OCwjzY6z/naqsY2qirNzk6 ilVJSvbs/au6I8kbHKpxEMoXr4kQ+PIXBx1Xx05PUmL47nkfz1F+kf+Vv2AKaDc+l3ObIsNC38cn8 JxvQTVAJfCpZs6htcmZCXlun2hpg6aZGiqB+MjWGH2YZ1xt9fR7clMZ5B8g0ihcuH/72NKZHCUQHv wY9Mvjy501914NspkIOys3opWFybPv4UVzJIlny+xpsxGrYLH50vRDjbdL9W0IIis2vQexz/hNkUn YYxaSGBA==; Received: from [2001:4bb8:19a:27af:c78f:9b0d:b95c:d248] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pKfvS-007P3i-N5; Wed, 25 Jan 2023 13:34:55 +0000 From: Christoph Hellwig To: Jens Axboe , Minchan Kim , Sergey Senozhatsky , Dan Williams , Vishal Verma , Dave Jiang , Ira Weiny Cc: Andrew Morton , linux-block@vger.kernel.org, nvdimm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/7] mm: use an on-stack bio for synchronous swapin Date: Wed, 25 Jan 2023 14:34:33 +0100 Message-Id: <20230125133436.447864-5-hch@lst.de> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230125133436.447864-1-hch@lst.de> References: <20230125133436.447864-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Stat-Signature: 46umxtkemwc9cmpahzkwzrbxntkduf61 X-Rspam-User: X-Rspamd-Queue-Id: 0078D10001D X-Rspamd-Server: rspam06 X-HE-Tag: 1674653699-20864 X-HE-Meta: U2FsdGVkX19h4wGPykADunXy+rkMlPDn3eEc+GuJdlmcNGpVpcmu7fpSflymQO95nIhRqhSFDam55czDqUp2PuG2TuT2GKO+6O3yzicFTEPv+lDTCkTcvXPMvE/25wJdD0+Cey2kejHgqQg7YuwFq6djBT+VhfJCZjwhFmLI4aBgYHeuAnj+iRqKK9wIsiB4+39vM/V9u8/x3MzlUZgaoi1wv4KHRsdfm3bRhdvvAZosgNRWRY8ohL69ZdMBzk8Tv2M1MkWLhYMp5P57Rgjymzm1v7x3wvZ1h5tLDPF1f2lx61gxYU+i0ZKOvnE+OveqtEv0FJbcOb5tplpASQG9f104VbCIe+qrK5f3CXCOt8x66pLAQmy2PMJw+m7+y0e7La3WehV1ehSqh9ogWodGYq2d4dpfLjXReu97yorRD7mNwRDMThCTVS/Wzerbm8czAmfTSUEIYujdvdX/zQGdKyGjLzxGp9OvWzTccM9/A0hhatfK78S7amN6pdFNVd5QNMPNnXGI2hAKboFsEeRxu+Ci9kxB9oghmrpBluvVhthyt6L+6/XWoJjJJzPHGozhZ3zA7giSdHY3npaIA/sVIaKRh6mTaoV3ZyMpQeTsBYE5KchzwslRrknG43+ITa2ZAtXB/9f428fB0MuviRLYRvs3fJD0OxYJrccJZgLnjIzSXTVQLXx6vqN6W6K+wsr669K7DBiIgnJ9xkl4Ds/Lxz4xXWWNdziTAnJArIt4Z3k83AUIJVp5mDdz8LWLq6uJv0zTbHasH6ahbMcRwtZGbSMKSAackmeB8B/xBdJkDDvsHTW3tlHXcz4zV7/fNREcV7vP5UAUFg5NgFMYEydJDoNwm68QFoGKrLdqX+4aj4pCtENiEMoNdRcJaR5D8bmr/R7MEBozcega4ikElEUYba2nf4TAJGQrGOJh+6x0QA1T+B4kaSMUxK0qBMoc+2IjTRS1Cy8voZaUG2iebQo bgvWP9My uU+XL7tm0mj0P/+ML82oCIx1abSfr9XJiSK2bZCh7b1zIXgPfieCstOKjuWiUeu+yvo39/GKfkZd0mIsdJmwCmEnn2vJu53oIH3yBmm1Sf1/eI8PkVoV6ujJRMdBS8+kJpKCoNPj5GuZTYqJUZYderJbFJ3PZNEvPMtGOrpil/vRl1K84sL0YpgxTU2crb59fVB09ugXVnLf5tBDTBebwpHomVktInkjP0T0R06qFSOqXzbZeeKgOQ93lyuyjj5PKWbb5d/F3hLjoKF+uVYcaWvG7he8Uw9IgXruI4wSQZPAdkIVugCWgN8CO6+V6FEZYxzWi/v7ltDZaKuy1DQ6tH4o77uD1Bk3Cu9a6y1lleZRRoFa/yGWynB78hAGHpCw9hFGB85QrtKL9YSWHEPax3JbYdu01KPnsmtWVWseC17qfqX/BW3qaj5oarg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Optimize the synchronous swap in case by using an on-stack bio instead of allocating one using bio_alloc. Signed-off-by: Christoph Hellwig --- mm/page_io.c | 69 +++++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 31 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index ce0b3638094f85..21ce4505f00607 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -52,10 +52,9 @@ static void end_swap_bio_write(struct bio *bio) bio_put(bio); } -static void end_swap_bio_read(struct bio *bio) +static void __end_swap_bio_read(struct bio *bio) { struct page *page = bio_first_page_all(bio); - struct task_struct *waiter = bio->bi_private; if (bio->bi_status) { SetPageError(page); @@ -63,18 +62,16 @@ static void end_swap_bio_read(struct bio *bio) pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n", MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), (unsigned long long)bio->bi_iter.bi_sector); - goto out; + } else { + SetPageUptodate(page); } - - SetPageUptodate(page); -out: unlock_page(page); - WRITE_ONCE(bio->bi_private, NULL); +} + +static void end_swap_bio_read(struct bio *bio) +{ + __end_swap_bio_read(bio); bio_put(bio); - if (waiter) { - blk_wake_io_task(waiter); - put_task_struct(waiter); - } } int generic_swapfile_activate(struct swap_info_struct *sis, @@ -445,10 +442,11 @@ static void swap_readpage_fs(struct page *page, *plug = sio; } -static void swap_readpage_bdev(struct page *page, bool synchronous, +static void swap_readpage_bdev_sync(struct page *page, struct swap_info_struct *sis) { - struct bio *bio; + struct bio_vec bv; + struct bio bio; if ((sis->flags & SWP_SYNCHRONOUS_IO) && !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { @@ -456,30 +454,37 @@ static void swap_readpage_bdev(struct page *page, bool synchronous, return; } - bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); - bio->bi_iter.bi_sector = swap_page_sector(page); - bio->bi_end_io = end_swap_bio_read; - bio_add_page(bio, page, thp_size(page), 0); + bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ); + bio.bi_iter.bi_sector = swap_page_sector(page); + bio_add_page(&bio, page, thp_size(page), 0); /* * Keep this task valid during swap readpage because the oom killer may * attempt to access it in the page fault retry time check. */ - if (synchronous) { - get_task_struct(current); - bio->bi_private = current; - } + get_task_struct(current); count_vm_event(PSWPIN); - bio_get(bio); - submit_bio(bio); - while (synchronous) { - set_current_state(TASK_UNINTERRUPTIBLE); - if (!READ_ONCE(bio->bi_private)) - break; + submit_bio_wait(&bio); + __end_swap_bio_read(&bio); + put_task_struct(current); +} + +static void swap_readpage_bdev_async(struct page *page, + struct swap_info_struct *sis) +{ + struct bio *bio; - blk_io_schedule(); + if ((sis->flags & SWP_SYNCHRONOUS_IO) && + !bdev_read_page(sis->bdev, swap_page_sector(page), page)) { + count_vm_event(PSWPIN); + return; } - __set_current_state(TASK_RUNNING); - bio_put(bio); + + bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); + bio->bi_iter.bi_sector = swap_page_sector(page); + bio->bi_end_io = end_swap_bio_read; + bio_add_page(bio, page, thp_size(page), 0); + count_vm_event(PSWPIN); + submit_bio(bio); } void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) @@ -509,8 +514,10 @@ void swap_readpage(struct page *page, bool synchronous, struct swap_iocb **plug) unlock_page(page); } else if (data_race(sis->flags & SWP_FS_OPS)) { swap_readpage_fs(page, plug); + } else if (synchronous) { + swap_readpage_bdev_sync(page, sis); } else { - swap_readpage_bdev(page, synchronous, sis); + swap_readpage_bdev_async(page, sis); } if (workingset) {