From patchwork Sat Oct 23 16:21:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12579643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F3ADC433FE for ; Sat, 23 Oct 2021 16:21:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01D4E61073 for ; Sat, 23 Oct 2021 16:21:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230104AbhJWQYI (ORCPT ); Sat, 23 Oct 2021 12:24:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbhJWQYH (ORCPT ); Sat, 23 Oct 2021 12:24:07 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23D7AC061714 for ; Sat, 23 Oct 2021 09:21:48 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id v17so2517211wrv.9 for ; Sat, 23 Oct 2021 09:21:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iECOiVFDdSc/w8PF/u9qtqfiQcqmaH9XDcaCDgobCk4=; b=ZHxZNgBebO+dDTnZtt+NsEuzIOhHFK3KIAYgH3seFJz9MpCWWTV4OmYNlSHwFxzxmv Xdwdk3ZnrHOGk75I1yuFWBRysAixsEBJANphvmS5ohmifdrmgYpMoGQBGV3kBgKE/SC9 7ms38yTOTsthdCR+2Q6tDz49iXCPo1WIa20m4EouQX/aDwanvpovEXR1taa5TZjFFgUu RLRnU5uXZBKm+WrM+q85JKLxzoqLAP3UguarlgJSuqmkOdqQm1t62yFZbSmacicAJrMG d48f1YjkKyfKIx4igQEF/OH3/peBEjgEaX6qGpVd2OC8lPkWTu1Rh+oShjz1uC4Z6Qvb 3A5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iECOiVFDdSc/w8PF/u9qtqfiQcqmaH9XDcaCDgobCk4=; b=1unlA4vrQE737Q218aHoYz4XbpOxWy9/9ocGXkqgZkfjCgCZ5MD0jx/XT1/LLlM5EJ ot3yfy6/qy9Gw3Q+7zMs5jj4JThYYNQRJcxuYQ1/sc8yYcQFX0kb1hTheygMrOeW/CKF L9AGCw/QYIoYIrlkEdDOEunZHPO+vMtD9yBjpWjwr4o98S8IZ5Wqx46nRXbU4acDklky dsOWd13wT+42cdeTzVhu1Dnf0qgfLoJSpgHhaKdF+NoN613TlGIdysmRdlBJSmTRehYn uetXMN56xfz3QS+mgWDa2h562vu1kl4jrQ7ZJEIY+CH/1Lf3myDiTMS63LiXQ2pjn+Q6 gXDg== X-Gm-Message-State: AOAM532FMizIyAwU/AV1DIVmvatfbz1id6cM72Ze79bbzx3O7RPPgiQe cXCYw2RJXAu7D2qyidERHTnMxm4bcaQ= X-Google-Smtp-Source: ABdhPJz8zKQQjdGq4YhNbU/nd38OL9B1KTuHdLZ7sJM4Hzmu1NDBak6bhZgcdVW9eGd3uc9ngxp6OQ== X-Received: by 2002:adf:e306:: with SMTP id b6mr8688279wrj.244.1635006106497; Sat, 23 Oct 2021 09:21:46 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.133.195]) by smtp.gmail.com with ESMTPSA id c16sm2174799wrt.43.2021.10.23.09.21.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Oct 2021 09:21:46 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH 1/5] block: add single bio async direct IO helper Date: Sat, 23 Oct 2021 17:21:32 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org As with __blkdev_direct_IO_simple(), we can implement direct IO more efficiently if there is only one bio. Add __blkdev_direct_IO_async() and blkdev_bio_end_io_async(). This patch brings me from 4.45-4.5 MIOPS with nullblk to 4.7+. Signed-off-by: Pavel Begunkov --- block/fops.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 84 insertions(+), 3 deletions(-) diff --git a/block/fops.c b/block/fops.c index 396537598e3e..a7b328296912 100644 --- a/block/fops.c +++ b/block/fops.c @@ -305,6 +305,85 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, return ret; } +static void blkdev_bio_end_io_async(struct bio *bio) +{ + struct blkdev_dio *dio = container_of(bio, struct blkdev_dio, bio); + struct kiocb *iocb = dio->iocb; + ssize_t ret; + + if (likely(!bio->bi_status)) { + ret = dio->size; + iocb->ki_pos += ret; + } else { + ret = blk_status_to_errno(bio->bi_status); + } + + iocb->ki_complete(iocb, ret, 0); + + if (dio->flags & DIO_SHOULD_DIRTY) { + bio_check_pages_dirty(bio); + } else { + bio_release_pages(bio, false); + bio_put(bio); + } +} + +static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, + struct iov_iter *iter, + unsigned int nr_pages) +{ + struct block_device *bdev = iocb->ki_filp->private_data; + struct blkdev_dio *dio; + struct bio *bio; + loff_t pos = iocb->ki_pos; + int ret = 0; + + if ((pos | iov_iter_alignment(iter)) & + (bdev_logical_block_size(bdev) - 1)) + return -EINVAL; + + bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); + dio = container_of(bio, struct blkdev_dio, bio); + dio->flags = 0; + dio->iocb = iocb; + bio_set_dev(bio, bdev); + bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT; + bio->bi_write_hint = iocb->ki_hint; + bio->bi_end_io = blkdev_bio_end_io_async; + bio->bi_ioprio = iocb->ki_ioprio; + + ret = bio_iov_iter_get_pages(bio, iter); + if (unlikely(ret)) { + bio->bi_status = BLK_STS_IOERR; + bio_endio(bio); + return ret; + } + dio->size = bio->bi_iter.bi_size; + + if (iov_iter_rw(iter) == READ) { + bio->bi_opf = REQ_OP_READ; + if (iter_is_iovec(iter)) { + dio->flags |= DIO_SHOULD_DIRTY; + bio_set_pages_dirty(bio); + } + } else { + bio->bi_opf = dio_bio_write_op(iocb); + task_io_account_write(bio->bi_iter.bi_size); + } + + if (iocb->ki_flags & IOCB_NOWAIT) + bio->bi_opf |= REQ_NOWAIT; + + if (iocb->ki_flags & IOCB_HIPRI) { + bio_set_polled(bio, iocb); + submit_bio(bio); + WRITE_ONCE(iocb->private, bio); + } else { + submit_bio(bio); + } + return -EIOCBQUEUED; +} + static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) { unsigned int nr_pages; @@ -313,9 +392,11 @@ static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) return 0; nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS + 1); - if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_VECS) - return __blkdev_direct_IO_simple(iocb, iter, nr_pages); - + if (likely(nr_pages <= BIO_MAX_VECS)) { + if (is_sync_kiocb(iocb)) + return __blkdev_direct_IO_simple(iocb, iter, nr_pages); + return __blkdev_direct_IO_async(iocb, iter, nr_pages); + } return __blkdev_direct_IO(iocb, iter, bio_max_segs(nr_pages)); } From patchwork Sat Oct 23 16:21:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12579645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 276C5C433F5 for ; Sat, 23 Oct 2021 16:21:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0CD4761073 for ; Sat, 23 Oct 2021 16:21:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230142AbhJWQYJ (ORCPT ); Sat, 23 Oct 2021 12:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbhJWQYI (ORCPT ); Sat, 23 Oct 2021 12:24:08 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04C4FC061714 for ; Sat, 23 Oct 2021 09:21:49 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id u18so382485wrg.5 for ; Sat, 23 Oct 2021 09:21:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1EgAyDTNJczgY/QgtEiRbb/2pP/JWJ0lkUXSaV8aVUk=; b=ZLbf0mHJW3xWQZwiGxpN3fhoNcyUcsHPSJ7QdHJzvvJqMsU67s5UlZQCR6ml28PrWO 8oJiAou3SVcsXjeig9fBJv5HGMB4sR44aS/1U2Eq3Qhb+Ynujnz1Jz7Sm8LCRgwYu1nw hjG7myFPLiucAn0HlafBz97MLB2sBvbZSekznH+DHaqyfde3CcBVAMsEmX6/FtJxLAVD dDjg0I+LGFV31mKdCYN5EJqhvUlmXsmx9MwDFtqlPRayoDZIlnWUm0tHtZLbDiBIiEpe 8HW8lgNBxoBsoc+hpz25A2Ukei+MeOOO5+2O1/OrPsNC73lPna2O3YAgAW56+W5JUl3m AyqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1EgAyDTNJczgY/QgtEiRbb/2pP/JWJ0lkUXSaV8aVUk=; b=NqBguU+RKlDaySrlHwxUXPL/6qnW4/H8Z534x7X5/IdHmglLWGSXljdLNUWps0iD3v OI9mta84FxPbHCQGEDdKK5ZF/TfTMueaw2cVd1/By+aRcwCmGvSfbb3qHYl/KvtxGbmG a1hj9Ommx9KwZryWqVcvufjk5w2sebXmrfhoQQduUwWuQySaAepBLKYgzFu4xWOpDEx+ FpC/gU+Plq/kJqZptdQKU/tjTq6XjaIi7DJKDhjKOdMDUyp3gvLbJ9mDCgy8mhxFEJfS b7d68gOpnd3r1tcWdDvatJmZThTFaw6t3aaYt6YjuJl3jdgE9nd+hRAW8BpdqPv7jel6 9lDw== X-Gm-Message-State: AOAM531YMZ9SNwSvakjXPhKLWGylSosTkkkDQ77FO9eWPlDlUvRyxshZ +FPmsmyXvOK1VCXO1SSlp1XgBjKeBh4= X-Google-Smtp-Source: ABdhPJx9NZz8tzieoIkN4hgClxa3OS+JyAVAXBPZPQoaPTaI8ieKKKpTBQQqMLzjrzWIhim9gKOxKA== X-Received: by 2002:adf:d1ee:: with SMTP id g14mr9080943wrd.264.1635006107430; Sat, 23 Oct 2021 09:21:47 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.133.195]) by smtp.gmail.com with ESMTPSA id c16sm2174799wrt.43.2021.10.23.09.21.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Oct 2021 09:21:47 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH 2/5] block: refactor bio_iov_bvec_set() Date: Sat, 23 Oct 2021 17:21:33 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Combine bio_iov_bvec_set() and bio_iov_bvec_set_append() and let the caller to do iov_iter_advance(). Also get rid of __bio_iov_bvec_set(), which was duplicated in the final binary, and replace a weird iov_iter_truncate() of a temporal iter copy with min() better reflecting the intention. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/bio.c | 37 ++++++++++++++----------------------- 1 file changed, 14 insertions(+), 23 deletions(-) diff --git a/block/bio.c b/block/bio.c index 46a87c72d2b4..ead1f8a9ff5e 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1046,36 +1046,27 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(__bio_release_pages); -static void __bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) +static void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { + size_t size = iov_iter_count(iter); + WARN_ON_ONCE(bio->bi_max_vecs); + if (bio_op(bio) == REQ_OP_ZONE_APPEND) { + struct request_queue *q = bdev_get_queue(bio->bi_bdev); + size_t max_sectors = queue_max_zone_append_sectors(q); + + size = min(size, max_sectors << SECTOR_SHIFT); + } + bio->bi_vcnt = iter->nr_segs; bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; - bio->bi_iter.bi_size = iter->count; + bio->bi_iter.bi_size = size; bio_set_flag(bio, BIO_NO_PAGE_REF); bio_set_flag(bio, BIO_CLONED); } -static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) -{ - __bio_iov_bvec_set(bio, iter); - iov_iter_advance(iter, iter->count); - return 0; -} - -static int bio_iov_bvec_set_append(struct bio *bio, struct iov_iter *iter) -{ - struct request_queue *q = bdev_get_queue(bio->bi_bdev); - struct iov_iter i = *iter; - - iov_iter_truncate(&i, queue_max_zone_append_sectors(q) << 9); - __bio_iov_bvec_set(bio, &i); - iov_iter_advance(iter, i.count); - return 0; -} - static void bio_put_pages(struct page **pages, size_t size, size_t off) { size_t i, nr = DIV_ROUND_UP(size + (off & ~PAGE_MASK), PAGE_SIZE); @@ -1217,9 +1208,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) int ret = 0; if (iov_iter_is_bvec(iter)) { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) - return bio_iov_bvec_set_append(bio, iter); - return bio_iov_bvec_set(bio, iter); + bio_iov_bvec_set(bio, iter); + iov_iter_advance(iter, bio->bi_iter.bi_size); + return 0; } do { From patchwork Sat Oct 23 16:21:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12579647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A862C433F5 for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4BB4461090 for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230230AbhJWQYJ (ORCPT ); Sat, 23 Oct 2021 12:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbhJWQYJ (ORCPT ); Sat, 23 Oct 2021 12:24:09 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2927EC061714 for ; Sat, 23 Oct 2021 09:21:50 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id r7so1753212wrc.10 for ; Sat, 23 Oct 2021 09:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VuvUv0whHecDeqlvFKkvrtxzrfXxRHmWygKkSMZW+pM=; b=Tb7DZF6P3KW9hx4cOdeMQiDpKR+VdeRVD/qEVatLbqFqy2LE3PdA1xLMDiYuW/xO2j CgoKh0bkJZrFWXP51nigXeaTJyQdfytGJ4mQH4jx8SSnFqzKi3aippRYvNUeOj3b9LUA uKBr8FWoXjW8Tkl1yNkCWAA3bn3gF+ntagU/d2QwJSZxOSS4JwfYQnw6ZXR7yt8cAKW9 UoAdJtjRQhCroiM/AfKN4GQ0dlx39qJw4KtYanwNP7f3N5uoB742L4+opgYwFd2eM5dZ KBF0kKg8c6yDrF9yq/UNR+e/o6eIYmO6Vd8bgUyYGRugefwBlk3qjC4daThfzUcg7OfX NEjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VuvUv0whHecDeqlvFKkvrtxzrfXxRHmWygKkSMZW+pM=; b=vo/Hpxbee3ab+AcNJiztJzX16jjUFw85AMkaaM/weNArLe7V70B36iBvm8h0CRBWvm qerphsm4nrtmB+8IiSiWE6poio5+wix5zBadc/SJLzQtwf/fE2H2HxE2wfI4gseevFxe TVEpU+5dknPT+KDWiDLaRLGHEjyJTKEkdV75/tM8aZTbNMSaot4oi5uUAY3vw1yqaAoQ mrRZsVZaNdR6gTOnN5j/kXZQHFnnS9ukb/2p6WM9iVAQRirpdV9OO4k+ybU3xLQl9s90 QoQaPGifebgD4KROZ/y2y2DL4pzwhg329nRKHkgi0MFGa81SRwJpmCsQN4WV2pfugYH8 tErA== X-Gm-Message-State: AOAM530/7or+Y9PANPTX2o6oIvxfzZqXj9vhibp36nv01hHxxW4+ew5Z GoXt6o3dH0Fp1ARRhq/c3k0Bu2eu+2M= X-Google-Smtp-Source: ABdhPJzxdn4o62XnTDS6JmdDUHUg+Y6K+6bQ3mx9SR0+tHw5LcZgp6pkiwxUAZAvmJrzCrvjbcR0UQ== X-Received: by 2002:a5d:5287:: with SMTP id c7mr8980826wrv.236.1635006108534; Sat, 23 Oct 2021 09:21:48 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.133.195]) by smtp.gmail.com with ESMTPSA id c16sm2174799wrt.43.2021.10.23.09.21.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Oct 2021 09:21:48 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH 3/5] block: avoid extra iter advance with async iocb Date: Sat, 23 Oct 2021 17:21:34 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Nobody cares about iov iterators state if we return -EIOCBQUEUED, so as the we now have __blkdev_direct_IO_async(), which gets pages only once, we can skip expensive iov_iter_advance(). It's around 1-2% of all CPU spent. Signed-off-by: Pavel Begunkov --- block/bio.c | 2 +- block/fops.c | 20 +++++++++++++++----- include/linux/bio.h | 1 + 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/block/bio.c b/block/bio.c index ead1f8a9ff5e..15ab0d6d1c06 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1046,7 +1046,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(__bio_release_pages); -static void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) +void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { size_t size = iov_iter_count(iter); diff --git a/block/fops.c b/block/fops.c index a7b328296912..8800b0ad5c29 100644 --- a/block/fops.c +++ b/block/fops.c @@ -352,11 +352,21 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, bio->bi_end_io = blkdev_bio_end_io_async; bio->bi_ioprio = iocb->ki_ioprio; - ret = bio_iov_iter_get_pages(bio, iter); - if (unlikely(ret)) { - bio->bi_status = BLK_STS_IOERR; - bio_endio(bio); - return ret; + if (!iov_iter_is_bvec(iter)) { + ret = bio_iov_iter_get_pages(bio, iter); + if (unlikely(ret)) { + bio->bi_status = BLK_STS_IOERR; + bio_endio(bio); + return ret; + } + } else { + /* + * Users don't rely on the iterator being in any particular + * state for async I/O returning -EIOCBQUEUED, hence we can + * avoid expensive iov_iter_advance(). Bypass + * bio_iov_iter_get_pages() and set the bvec directly. + */ + bio_iov_bvec_set(bio, iter); } dio->size = bio->bi_iter.bi_size; diff --git a/include/linux/bio.h b/include/linux/bio.h index c88700d1bdc3..fe6bdfbbef66 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -417,6 +417,7 @@ int bio_add_zone_append_page(struct bio *bio, struct page *page, void __bio_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int off); int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter); +void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter); void __bio_release_pages(struct bio *bio, bool mark_dirty); extern void bio_set_pages_dirty(struct bio *bio); extern void bio_check_pages_dirty(struct bio *bio); From patchwork Sat Oct 23 16:21:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12579649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8DD1C433EF for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A30986108B for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230323AbhJWQYM (ORCPT ); Sat, 23 Oct 2021 12:24:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbhJWQYK (ORCPT ); Sat, 23 Oct 2021 12:24:10 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DC79C061714 for ; Sat, 23 Oct 2021 09:21:51 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id a20-20020a1c7f14000000b003231d13ee3cso7492063wmd.3 for ; Sat, 23 Oct 2021 09:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NgYgX1V6iC19ADVcmUcsDZUvqPBrKkuru1MqVCti3TU=; b=X7ATy8fS+Tk+24i7UD8yvviCV+fAyagdrDlTwyPi1kzq7pXIwLp4GeQUodSVVuz+Fa /PBE+0oF7W+QIhsy7/ySAnrBia9/Hs3BNA0Fw4bPXih0BgERu722axHHLIYXrSyKfY4P xMVdugntHEKiKGolg/MAhs6BsZjQngVMXNArY/BpeudXcROIiEpsSthQdjHE5LBN1JAz Mt3htajwYy9F0oeePCB7WEA6PUkZJZF6Z19i2Nq87U+8ct1SmyIQ5KXVIgj2xqiM4yOQ Dz4PGaMocY6+HYv5lltbDpdaCJvOKWMTiW1vc0kvLB3Rv0+kGbtrYp+Woo2mjKQziCtG lNrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NgYgX1V6iC19ADVcmUcsDZUvqPBrKkuru1MqVCti3TU=; b=qYVQLRkJZ/U/UrD4WpyexAoo510SVph+sOGnzyl4f/vfijscpYeNp62seNlOoWY1iC h9CanQL8LcMs0yPIu5vACLnnLiABIFMFpZk3bJYoXXjPzTdSVfVJNX/sZBDvVEzI8GI0 83yhH4S5BMlrXuIf7MhMchF2BDkaQA9fqq27bjxnOToDCWC8+nPsLmVIyvwkcRj0omU7 x88w58/xy3D81vRaDnF0up1g4YlKDdHLZ7ai/3UNAuOVesf43zlVBHlSroBY+uXTEqne 4Hv5yiK5h7kBVFiawxPJkOUNraGmgK2ZyGcAQjISg9/hYTPDer5baevXfXlEpGnBMGQO txDg== X-Gm-Message-State: AOAM53134BAiosJFYWu4CH4QaOsLSrhKMc80KnTeyhMASOoGAhcK6CpH xcw/nWjkhL/HWS4JNPxQfX6s4aA/X6k= X-Google-Smtp-Source: ABdhPJx/tQbRK/dvkmbE1iCqTCL17aIHSRaW4PmJ0Q8PP2mfidkQFKXG0CHruItDU72q+D71Ecs3lg== X-Received: by 2002:a1c:4e18:: with SMTP id g24mr2904290wmh.180.1635006109555; Sat, 23 Oct 2021 09:21:49 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.133.195]) by smtp.gmail.com with ESMTPSA id c16sm2174799wrt.43.2021.10.23.09.21.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Oct 2021 09:21:49 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH 4/5] block: kill unused polling bits in __blkdev_direct_IO() Date: Sat, 23 Oct 2021 17:21:35 +0100 Message-Id: <2e63549f6bce3442c27997fae83082f1c9f4e6c3.1635006010.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org With addition of __blkdev_direct_IO_async(), __blkdev_direct_IO() now serves only multio-bio I/O, which we don't poll. Now we can remove anything related to I/O polling from it. Signed-off-by: Pavel Begunkov --- block/fops.c | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/block/fops.c b/block/fops.c index 8800b0ad5c29..997904963a9d 100644 --- a/block/fops.c +++ b/block/fops.c @@ -190,7 +190,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, struct blk_plug plug; struct blkdev_dio *dio; struct bio *bio; - bool do_poll = (iocb->ki_flags & IOCB_HIPRI); bool is_read = (iov_iter_rw(iter) == READ), is_sync; loff_t pos = iocb->ki_pos; int ret = 0; @@ -216,12 +215,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, if (is_read && iter_is_iovec(iter)) dio->flags |= DIO_SHOULD_DIRTY; - /* - * Don't plug for HIPRI/polled IO, as those should go straight - * to issue - */ - if (!(iocb->ki_flags & IOCB_HIPRI)) - blk_start_plug(&plug); + blk_start_plug(&plug); for (;;) { bio_set_dev(bio, bdev); @@ -254,11 +248,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS); if (!nr_pages) { - if (do_poll) - bio_set_polled(bio, iocb); submit_bio(bio); - if (do_poll) - WRITE_ONCE(iocb->private, bio); break; } if (!(dio->flags & DIO_MULTI_BIO)) { @@ -271,7 +261,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bio_get(bio); dio->flags |= DIO_MULTI_BIO; atomic_set(&dio->ref, 2); - do_poll = false; } else { atomic_inc(&dio->ref); } @@ -280,8 +269,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bio = bio_alloc(GFP_KERNEL, nr_pages); } - if (!(iocb->ki_flags & IOCB_HIPRI)) - blk_finish_plug(&plug); + blk_finish_plug(&plug); if (!is_sync) return -EIOCBQUEUED; @@ -290,9 +278,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, set_current_state(TASK_UNINTERRUPTIBLE); if (!READ_ONCE(dio->waiter)) break; - - if (!do_poll || !bio_poll(bio, NULL, 0)) - blk_io_schedule(); + blk_io_schedule(); } __set_current_state(TASK_RUNNING); From patchwork Sat Oct 23 16:21:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12579651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF317C433FE for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B6421610A6 for ; Sat, 23 Oct 2021 16:21:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230327AbhJWQYM (ORCPT ); Sat, 23 Oct 2021 12:24:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230253AbhJWQYL (ORCPT ); Sat, 23 Oct 2021 12:24:11 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 196DFC061764 for ; Sat, 23 Oct 2021 09:21:52 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id v17so2517422wrv.9 for ; Sat, 23 Oct 2021 09:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4M1lw0oXTZ8+DVGrY/pbRVuISTz3oZ6JQgweXSxGY2Y=; b=KodvKt5qiS4nEb1Vb8dVCxloMUQoq3Z3jeuAcx9IUhdhiPooLKHxbSR/ctGenZ26eM hU9I6Zh65k2KUjM101/Fwxc0pf2U0jpzwD3Cw161MJbqklKKvlDcfpM+8jfvgYRxhB5H gwtmoWxgG1LLuQPYPN/zNA5ST8U/RYVoMLbpFQz298DMXQs78xWMPW/I6Rr0HNV/ZLOq Eqof/8Tq2+X8aPLtiIvVUwfUKDmnnZa94ph+NLGFjqdhyBUhOMJWh89Ocb0KoyAbFiq7 4riJceRf4H5EfmvWJvXIex2zEIsP8GrCdVDtLInDZMtQ020M4VwI+4tY58Ta+kBkYJX2 kDsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4M1lw0oXTZ8+DVGrY/pbRVuISTz3oZ6JQgweXSxGY2Y=; b=NyQZ9Ca9gCZK8em5thSL0Pc8N+8ZB3zDV/E8cXj1X1dgJin3U/VfEnFM0c6PijgwVD s1q8J4EOnagOyPccK87CnuHaPdW0OYebJsqEHP5C0JVIrE3BtYP90xpSpCSt1+XdiJU4 UBBhKKLbI/FmBmG1UFB2DwhUKJFEo/pvw65l/AxLn0/zVCdLvkdBMDemPE00bOSjrahJ tvTqCPHawB7mmUry73VtSndJdJOg4Z4Hgfn07yZkScWqFzP0ZzJx/pZ82LxO5gLkDRTW m6RrtjpYr3oaS/AC7wrk5t6raxBQIVQUVSd6sb95ZOHVfkBUrq/qAnsa1xvULqYzLADZ ksKA== X-Gm-Message-State: AOAM5315COhFz37ed3E7U2NlVgSX4ppDvU99W1DuXKss281dllYfNsie aKzZ5zeO2Ha+lrA+S+0yAy165yvQBkY= X-Google-Smtp-Source: ABdhPJwBAKpSfirIC9/OuAAXwiMNHMinGVaNq/CboS7GHbs8S0/wt9IqHE3j5PElg/wMTAVCjtdKmQ== X-Received: by 2002:adf:f208:: with SMTP id p8mr8589955wro.209.1635006110551; Sat, 23 Oct 2021 09:21:50 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.133.195]) by smtp.gmail.com with ESMTPSA id c16sm2174799wrt.43.2021.10.23.09.21.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Oct 2021 09:21:50 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Pavel Begunkov Subject: [PATCH 5/5] block: add async version of bio_set_polled Date: Sat, 23 Oct 2021 17:21:36 +0100 Message-Id: X-Mailer: git-send-email 2.33.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If we know that a iocb is async we can optimise bio_set_polled() a bit, add a new helper bio_set_polled_async(). Signed-off-by: Pavel Begunkov --- block/fops.c | 7 +++---- include/linux/bio.h | 5 +++++ 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/block/fops.c b/block/fops.c index 997904963a9d..9cb436de92bb 100644 --- a/block/fops.c +++ b/block/fops.c @@ -367,14 +367,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, task_io_account_write(bio->bi_iter.bi_size); } - if (iocb->ki_flags & IOCB_NOWAIT) - bio->bi_opf |= REQ_NOWAIT; - if (iocb->ki_flags & IOCB_HIPRI) { - bio_set_polled(bio, iocb); + bio_set_polled_async(bio, iocb); submit_bio(bio); WRITE_ONCE(iocb->private, bio); } else { + if (iocb->ki_flags & IOCB_NOWAIT) + bio->bi_opf |= REQ_NOWAIT; submit_bio(bio); } return -EIOCBQUEUED; diff --git a/include/linux/bio.h b/include/linux/bio.h index fe6bdfbbef66..b64161473f3e 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -738,6 +738,11 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) bio->bi_opf |= REQ_NOWAIT; } +static inline void bio_set_polled_async(struct bio *bio, struct kiocb *kiocb) +{ + bio->bi_opf |= REQ_POLLED | REQ_NOWAIT; +} + struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp); #endif /* __LINUX_BIO_H */