From patchwork Wed Oct 13 16:54:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C3D1C433FE for ; Wed, 13 Oct 2021 16:54:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71E3361027 for ; Wed, 13 Oct 2021 16:54:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237470AbhJMQ4Y (ORCPT ); Wed, 13 Oct 2021 12:56:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237193AbhJMQ4Y (ORCPT ); Wed, 13 Oct 2021 12:56:24 -0400 Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EBD7C061570 for ; Wed, 13 Oct 2021 09:54:20 -0700 (PDT) Received: by mail-il1-x12a.google.com with SMTP id s3so477520ild.0 for ; Wed, 13 Oct 2021 09:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/HxmRu2SrFMjxnbbGwX0Q0a2H0TAm1oB9K6LjRRzJwo=; b=ge82IDkCAO34j14tCKoHv2ZrI1SAmm8yMepOnP8dufan9alc7mpGpJ+Bka9Eb3RX0J 1PN1sx7T96kS661+CGew+EIqGCsr2KFAnvbXbUUOQJt773BgGZC4CcxEr8JYxsIKaBh3 JkcM5ViegjazL91JsGCy2+keJYjR7CZV8nIk67HALx8cAenvKKh2uBS+57sku3jltB+o +QBrldav12tswZ4kbYSOwRmDB8bl+2v1SQnQbwKz09rM+iHH8FTl5ubHu/pxXJumR0q0 Q30CagyAjngRIRNGuy6G9ZDsvy4rDByPhy1Jr/fti6J538DsrAlx+IlvVqKOlOO5te6X 0R3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/HxmRu2SrFMjxnbbGwX0Q0a2H0TAm1oB9K6LjRRzJwo=; b=ECmsr4hvjT3yV9AfjeccBrcUastUmivpxIWIbWZEAkMQul2gUCSIQYQ2L5EjdI3nXj OGaUaV2iFRm/mlF0GyxeLCR7rdl+mKuHKugw/lEMUMnuxgakZ2Hk0tqQ7oPc4kLRaTbG +6nAc4TZ8/y7kP0+RKtdlpt7UOLHkqLeL3TyZDF2lc9SPJiAr8dyEtZrbauZKoR5vtaG pV4zpml0HIAD82m0Qd+7dzMQ+eb7W4GV/rDhvtUR8dHmlP47a4kDvD9SE3xhffeZ7qQ/ stm7BoWfddMyJ9j/bRuROaalYFdVBNcCw1JI/AYuqpLtLsVs1sVyMdhPJXYvwBxL0HKH yLhQ== X-Gm-Message-State: AOAM533PrWsqg5P08s5YS7At0CFozwgrq7EdcywzDTko0Dxzro46s+Tx D6S3XfJcKEku8CLmW71RjXnDGGeP95QgPA== X-Google-Smtp-Source: ABdhPJzPUvV7QQuurN48GtuuWaEmZc9Vw6wdiL+DBGwzzGQ8R11U5u6Ax2AE3CU0J0xMF4VsvMrTMA== X-Received: by 2002:a92:2001:: with SMTP id j1mr140876ile.84.1634144059768; Wed, 13 Oct 2021 09:54:19 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:19 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/9] block: define io_batch structure Date: Wed, 13 Oct 2021 10:54:08 -0600 Message-Id: <20211013165416.985696-2-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This adds the io_batch structure, and a helper for defining one on the stack. It's meant to be used for collecting requests for completion, with a completion handler defined to be called at the end. Signed-off-by: Jens Axboe --- include/linux/blkdev.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 2a8689e949b4..b39b19dbe7b6 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1298,4 +1298,14 @@ int fsync_bdev(struct block_device *bdev); int freeze_bdev(struct block_device *bdev); int thaw_bdev(struct block_device *bdev); +struct io_batch { + struct request *req_list; + void (*complete)(struct io_batch *); +}; + +#define DEFINE_IO_BATCH(name) \ + struct io_batch name = { \ + .req_list = NULL \ + } + #endif /* _LINUX_BLKDEV_H */ From patchwork Wed Oct 13 16:54:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99377C433F5 for ; Wed, 13 Oct 2021 16:54:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8114161027 for ; Wed, 13 Oct 2021 16:54:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237532AbhJMQ4Z (ORCPT ); Wed, 13 Oct 2021 12:56:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237193AbhJMQ4Y (ORCPT ); Wed, 13 Oct 2021 12:56:24 -0400 Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 847B0C061570 for ; Wed, 13 Oct 2021 09:54:21 -0700 (PDT) Received: by mail-il1-x136.google.com with SMTP id k3so432675ilu.2 for ; Wed, 13 Oct 2021 09:54:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SkBDBNE+5k1MVCj3zB0CAXAb+tTd9ocFtwkuNqiyPpQ=; b=YdqlwxfuKt6J6KSLWqgu36QyD4TcIO1aHpiczu4k7mXFupT7kTBXPkDknnQ5RHG5Sw BjwKL3gQuk+oSgIfw5CXkroxXoP+iJtaCP4vXFLEKbSLEtWiqVYAh736zxVdlSe3/V2a Rqj6xZ/I1DQe51j/wibHRsuoD6vm3ELpIVZldR9ZOh3tvaAQbtTqLU++yA28wFsjzeAT diXiBJN7OfvJ4+BwwcTHtWw+PkiGZMgMFI6/92MYlke74IfSCYupV/52lfYPEtUESWqE Z9avrpJCJMwaZFbt+M0DL2b/m+6hvTiJBPHIzEX7X3XtuimKRDHkW2W493OO8790pBA+ UvaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SkBDBNE+5k1MVCj3zB0CAXAb+tTd9ocFtwkuNqiyPpQ=; b=WZvFyWZEQLsauRjf6UIDnazOfep2bvLvgiFIcQyADQkRRHb+E/yEqLp/CkR/goTcD1 8SvrXXbgS6pKcxr7nAdQXFXN1W0SbUhIsgTsZokDHVdLqquZCa7gV711RphTX+JO6FfD iVER3MldD2f6HhHjOaD8Ic1AgSs5RDpbeDzsu8mbtGs/a168jpvlJ7Lc4EQhNdPtgIWM S6ocITjKi9nCMmqpyXjesm/saIlRgaXcDBcbDWq4s3XaNNs4cVWhdeT3BthxtaWviUQV lz87r3DsugTWeEm9Mt/OhzCPyJrR1HXzAc60yrFhyvbutMR11PjRJxiS/G29uP0/h5ZG kZCg== X-Gm-Message-State: AOAM53021DW36JwHGEc0+5ktfVWe0eJiXbdM2UU4X5LRHsD19AhHWn+w shoPYdoKqSpWv/zee511yRbb106NIP7FfQ== X-Google-Smtp-Source: ABdhPJzAzs0etI0uUzHILC+7T1NZZ5iMfPR63GE8PtgKGL/4vFq1nGaciG9tqKt7UuL8QZ6r/HS5bg== X-Received: by 2002:a05:6e02:b2e:: with SMTP id e14mr116714ilu.47.1634144060602; Wed, 13 Oct 2021 09:54:20 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:20 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/9] block: add a struct io_batch argument to fops->iopoll() Date: Wed, 13 Oct 2021 10:54:09 -0600 Message-Id: <20211013165416.985696-3-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org struct io_batch contains a list head and a completion handler, which will allow completions to more effciently completed batches of IO. For now, no functional changes in this patch, we just add the argument to the file_operations iopoll handler Signed-off-by: Jens Axboe --- block/blk-core.c | 9 +++++---- block/blk-exec.c | 2 +- block/blk-mq.c | 9 +++++---- block/blk-mq.h | 3 ++- block/fops.c | 4 ++-- drivers/block/null_blk/main.c | 2 +- drivers/block/rnbd/rnbd-clt.c | 2 +- drivers/nvme/host/pci.c | 4 ++-- drivers/nvme/host/rdma.c | 2 +- drivers/nvme/host/tcp.c | 2 +- drivers/scsi/scsi_lib.c | 2 +- fs/io_uring.c | 2 +- fs/iomap/direct-io.c | 2 +- include/linux/blk-mq.h | 2 +- include/linux/blkdev.h | 5 +++-- include/linux/fs.h | 3 ++- mm/page_io.c | 2 +- 17 files changed, 31 insertions(+), 26 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index b199579c5f1f..67ec3f06c559 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1002,7 +1002,7 @@ EXPORT_SYMBOL(submit_bio); * Note: the caller must either be the context that submitted @bio, or * be in a RCU critical section to prevent freeing of @bio. */ -int bio_poll(struct bio *bio, unsigned int flags) +int bio_poll(struct bio *bio, struct io_batch *iob, unsigned int flags) { struct request_queue *q = bio->bi_bdev->bd_disk->queue; blk_qc_t cookie = READ_ONCE(bio->bi_cookie); @@ -1020,7 +1020,7 @@ int bio_poll(struct bio *bio, unsigned int flags) if (WARN_ON_ONCE(!queue_is_mq(q))) ret = 0; /* not yet implemented, should not happen */ else - ret = blk_mq_poll(q, cookie, flags); + ret = blk_mq_poll(q, cookie, iob, flags); blk_queue_exit(q); return ret; } @@ -1030,7 +1030,8 @@ EXPORT_SYMBOL_GPL(bio_poll); * Helper to implement file_operations.iopoll. Requires the bio to be stored * in iocb->private, and cleared before freeing the bio. */ -int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags) +int iocb_bio_iopoll(struct kiocb *kiocb, struct io_batch *iob, + unsigned int flags) { struct bio *bio; int ret = 0; @@ -1058,7 +1059,7 @@ int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags) rcu_read_lock(); bio = READ_ONCE(kiocb->private); if (bio && bio->bi_bdev) - ret = bio_poll(bio, flags); + ret = bio_poll(bio, iob, flags); rcu_read_unlock(); return ret; diff --git a/block/blk-exec.c b/block/blk-exec.c index 55f0cd34b37b..1b8b47f6e79b 100644 --- a/block/blk-exec.c +++ b/block/blk-exec.c @@ -77,7 +77,7 @@ static bool blk_rq_is_poll(struct request *rq) static void blk_rq_poll_completion(struct request *rq, struct completion *wait) { do { - bio_poll(rq->bio, 0); + bio_poll(rq->bio, NULL, 0); cond_resched(); } while (!completion_done(wait)); } diff --git a/block/blk-mq.c b/block/blk-mq.c index 069837a020fe..6eac10fd244e 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -4229,7 +4229,7 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, blk_qc_t qc) } static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie, - unsigned int flags) + struct io_batch *iob, unsigned int flags) { struct blk_mq_hw_ctx *hctx = blk_qc_to_hctx(q, cookie); long state = get_current_state(); @@ -4240,7 +4240,7 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie, do { hctx->poll_invoked++; - ret = q->mq_ops->poll(hctx); + ret = q->mq_ops->poll(hctx, iob); if (ret > 0) { hctx->poll_success++; __set_current_state(TASK_RUNNING); @@ -4261,14 +4261,15 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie, return 0; } -int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags) +int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, struct io_batch *iob, + unsigned int flags) { if (!(flags & BLK_POLL_NOSLEEP) && q->poll_nsec != BLK_MQ_POLL_CLASSIC) { if (blk_mq_poll_hybrid(q, cookie)) return 1; } - return blk_mq_poll_classic(q, cookie, flags); + return blk_mq_poll_classic(q, cookie, iob, flags); } unsigned int blk_mq_rq_cpu(struct request *rq) diff --git a/block/blk-mq.h b/block/blk-mq.h index ceed0a001c76..924b537b2d52 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -38,7 +38,8 @@ struct blk_mq_ctx { } ____cacheline_aligned_in_smp; void blk_mq_submit_bio(struct bio *bio); -int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, unsigned int flags); +int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, struct io_batch *iob, + unsigned int flags); void blk_mq_exit_queue(struct request_queue *q); int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr); void blk_mq_wake_waiters(struct request_queue *q); diff --git a/block/fops.c b/block/fops.c index 551b71af6d90..89e1eae8f89a 100644 --- a/block/fops.c +++ b/block/fops.c @@ -105,7 +105,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, set_current_state(TASK_UNINTERRUPTIBLE); if (!READ_ONCE(bio.bi_private)) break; - if (!(iocb->ki_flags & IOCB_HIPRI) || !bio_poll(&bio, 0)) + if (!(iocb->ki_flags & IOCB_HIPRI) || !bio_poll(&bio, NULL, 0)) blk_io_schedule(); } __set_current_state(TASK_RUNNING); @@ -285,7 +285,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, if (!READ_ONCE(dio->waiter)) break; - if (!do_poll || !bio_poll(bio, 0)) + if (!do_poll || !bio_poll(bio, NULL, 0)) blk_io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 7ce911e2289d..161faf172527 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -1494,7 +1494,7 @@ static int null_map_queues(struct blk_mq_tag_set *set) return 0; } -static int null_poll(struct blk_mq_hw_ctx *hctx) +static int null_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct nullb_queue *nq = hctx->driver_data; LIST_HEAD(list); diff --git a/drivers/block/rnbd/rnbd-clt.c b/drivers/block/rnbd/rnbd-clt.c index bd4a41afbbfc..5ccd6c602673 100644 --- a/drivers/block/rnbd/rnbd-clt.c +++ b/drivers/block/rnbd/rnbd-clt.c @@ -1176,7 +1176,7 @@ static blk_status_t rnbd_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } -static int rnbd_rdma_poll(struct blk_mq_hw_ctx *hctx) +static int rnbd_rdma_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct rnbd_queue *q = hctx->driver_data; struct rnbd_clt_dev *dev = q->dev; diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 0dd4b44b59cd..9db6e23f41ef 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1092,7 +1092,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq) enable_irq(pci_irq_vector(pdev, nvmeq->cq_vector)); } -static int nvme_poll(struct blk_mq_hw_ctx *hctx) +static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct nvme_queue *nvmeq = hctx->driver_data; bool found; @@ -1274,7 +1274,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) * Did we miss an interrupt? */ if (test_bit(NVMEQ_POLLED, &nvmeq->flags)) - nvme_poll(req->mq_hctx); + nvme_poll(req->mq_hctx, NULL); else nvme_poll_irqdisable(nvmeq); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 40317e1b9183..f02c1411c8e9 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2106,7 +2106,7 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } -static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx) +static int nvme_rdma_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct nvme_rdma_queue *queue = hctx->driver_data; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 3c1c29dd3020..a094920a6914 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2429,7 +2429,7 @@ static int nvme_tcp_map_queues(struct blk_mq_tag_set *set) return 0; } -static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx) +static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct nvme_tcp_queue *queue = hctx->driver_data; struct sock *sk = queue->sock->sk; diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 33fd9a01330c..0b382b2ad41b 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1784,7 +1784,7 @@ static void scsi_mq_exit_request(struct blk_mq_tag_set *set, struct request *rq, } -static int scsi_mq_poll(struct blk_mq_hw_ctx *hctx) +static int scsi_mq_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct Scsi_Host *shost = hctx->driver_data; diff --git a/fs/io_uring.c b/fs/io_uring.c index e43e130a0e92..082ff64c1bcb 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2412,7 +2412,7 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) if (READ_ONCE(req->iopoll_completed)) break; - ret = kiocb->ki_filp->f_op->iopoll(kiocb, poll_flags); + ret = kiocb->ki_filp->f_op->iopoll(kiocb, NULL, poll_flags); if (unlikely(ret < 0)) return ret; else if (ret) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 8efab177011d..83ecfba53abe 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -630,7 +630,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, break; if (!dio->submit.poll_bio || - !bio_poll(dio->submit.poll_bio, 0)) + !bio_poll(dio->submit.poll_bio, NULL, 0)) blk_io_schedule(); } __set_current_state(TASK_RUNNING); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index c05560524841..5106c4cc411a 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -563,7 +563,7 @@ struct blk_mq_ops { /** * @poll: Called to poll for completion of a specific tag. */ - int (*poll)(struct blk_mq_hw_ctx *); + int (*poll)(struct blk_mq_hw_ctx *, struct io_batch *); /** * @complete: Mark the request as complete. diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index b39b19dbe7b6..f626ab192ea2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -569,8 +569,9 @@ blk_status_t errno_to_blk_status(int errno); #define BLK_POLL_ONESHOT (1 << 0) /* do not sleep to wait for the expected completion time */ #define BLK_POLL_NOSLEEP (1 << 1) -int bio_poll(struct bio *bio, unsigned int flags); -int iocb_bio_iopoll(struct kiocb *kiocb, unsigned int flags); +int bio_poll(struct bio *bio, struct io_batch *iob, unsigned int flags); +int iocb_bio_iopoll(struct kiocb *kiocb, struct io_batch *iob, + unsigned int flags); static inline struct request_queue *bdev_get_queue(struct block_device *bdev) { diff --git a/include/linux/fs.h b/include/linux/fs.h index f98a361a6752..0ba86433797b 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -48,6 +48,7 @@ struct backing_dev_info; struct bdi_writeback; struct bio; +struct io_batch; struct export_operations; struct fiemap_extent_info; struct hd_geometry; @@ -2075,7 +2076,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); - int (*iopoll)(struct kiocb *kiocb, unsigned int flags); + int (*iopoll)(struct kiocb *kiocb, struct io_batch *, unsigned int flags); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *); diff --git a/mm/page_io.c b/mm/page_io.c index a68faab5b310..6010fb07f231 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -424,7 +424,7 @@ int swap_readpage(struct page *page, bool synchronous) if (!READ_ONCE(bio->bi_private)) break; - if (!bio_poll(bio, 0)) + if (!bio_poll(bio, NULL, 0)) blk_io_schedule(); } __set_current_state(TASK_RUNNING); From patchwork Wed Oct 13 16:54:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE6A1C433EF for ; Wed, 13 Oct 2021 16:54:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7AA56052B for ; Wed, 13 Oct 2021 16:54:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237598AbhJMQ40 (ORCPT ); Wed, 13 Oct 2021 12:56:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237563AbhJMQ4Z (ORCPT ); Wed, 13 Oct 2021 12:56:25 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F60BC061570 for ; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id r134so426479iod.11 for ; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ahqldeCtg66uzCKobXV+3ZUDAi97hwCQoUdjJg1AbOs=; b=VwyWvr7wdYOL9keMvXL/QDfI4sEWrwF/oRAC5cY/WZcR+VdeTnb7a5DnWPvWg7Freo O4S7GaL6xuhltdgGs6p2n1jNukmALmWE+/Q+QtL5UUdcExqax2i38AqBIggPDidVIzPd FyBOraM8sOT5bSFbiaqC9RY+Pi8PrUr8peK0F7JuCy8H9l9U9OtRBArl/+HfDtWsOLBv XdZkg/MBC8eUx/9dDpBChsmJx8lq2bhYIsL1mLxkyUGXINNw0JwCYtHQwig2+xizOsEr yDzGsj0h3mdf21tR8PKFZAFoM9lunBLTGD1RlwywclHPoqwlGukbSffMWHM0rx6lTaI1 Wuvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ahqldeCtg66uzCKobXV+3ZUDAi97hwCQoUdjJg1AbOs=; b=tDJfy7snpdsdGxhBLIJym+HVQvY4DA33zlM711mJ0wc12q2YAxfrSU/1PiGtNIOWDj 4fnh1K5f9qFvZbnMSki3qCWx5YLGmUczydTcfrnyGssR706wzhKjediogPvC0x93qonQ Zmiqt026z4WeBMlWQCYvw9S/Wc2onI4o8jhq+gsJo4XuMG7Sz0sXpWGwzjGB9CXcNVnC vLkN0w3d3RoGlF+ReysGIiV1P9Fi05VPzsF3g44AlIhM4rd/wgMf1ISGxPuuRcTeZBxr rDhRjT1OvOAPnaJeDDqJceH8VJLmf+88dVn/LlyAZyQRMd2ZEXqXxVcG2BBSDp37ohVG aaug== X-Gm-Message-State: AOAM533mfs4NlIkjSuVBFWRl5eSezwWSHt1I6jSlrORvi/yQkI32tfT8 8ya8mBHKwYI2sESowoDVhMk1fTgxyk4= X-Google-Smtp-Source: ABdhPJz88CJrC6mMo/Pl/ZmskF0eZpMfTcw1jcn3B4HJHPADH2RgtDR+21imLn1NJfl0hOpxlEVYaQ== X-Received: by 2002:a05:6602:2ac3:: with SMTP id m3mr306844iov.138.1634144061391; Wed, 13 Oct 2021 09:54:21 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:20 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/9] sbitmap: add helper to clear a batch of tags Date: Wed, 13 Oct 2021 10:54:10 -0600 Message-Id: <20211013165416.985696-4-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org sbitmap currently only supports clearing tags one-by-one, add a helper that allows the caller to pass in an array of tags to clear. Signed-off-by: Jens Axboe --- include/linux/sbitmap.h | 11 +++++++++++ lib/sbitmap.c | 44 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 52 insertions(+), 3 deletions(-) diff --git a/include/linux/sbitmap.h b/include/linux/sbitmap.h index e30b56023ead..4a6ff274335a 100644 --- a/include/linux/sbitmap.h +++ b/include/linux/sbitmap.h @@ -528,6 +528,17 @@ void sbitmap_queue_min_shallow_depth(struct sbitmap_queue *sbq, void sbitmap_queue_clear(struct sbitmap_queue *sbq, unsigned int nr, unsigned int cpu); +/** + * sbitmap_queue_clear_batch() - Free a batch of allocated bits + * &struct sbitmap_queue. + * @sbq: Bitmap to free from. + * @offset: offset for each tag in array + * @tags: array of tags + * @nr_tags: number of tags in array + */ +void sbitmap_queue_clear_batch(struct sbitmap_queue *sbq, int offset, + int *tags, int nr_tags); + static inline int sbq_index_inc(int index) { return (index + 1) & (SBQ_WAIT_QUEUES - 1); diff --git a/lib/sbitmap.c b/lib/sbitmap.c index f398e0ae548e..c6e2f1f2c4d2 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -628,6 +628,46 @@ void sbitmap_queue_wake_up(struct sbitmap_queue *sbq) } EXPORT_SYMBOL_GPL(sbitmap_queue_wake_up); +static inline void sbitmap_update_cpu_hint(struct sbitmap *sb, int cpu, int tag) +{ + if (likely(!sb->round_robin && tag < sb->depth)) + *per_cpu_ptr(sb->alloc_hint, cpu) = tag; +} + +void sbitmap_queue_clear_batch(struct sbitmap_queue *sbq, int offset, + int *tags, int nr_tags) +{ + struct sbitmap *sb = &sbq->sb; + unsigned long *addr = NULL; + unsigned long mask = 0; + int i; + + smp_mb__before_atomic(); + for (i = 0; i < nr_tags; i++) { + const int tag = tags[i] - offset; + unsigned long *this_addr; + + /* since we're clearing a batch, skip the deferred map */ + this_addr = &sb->map[SB_NR_TO_INDEX(sb, tag)].word; + if (!addr) { + addr = this_addr; + } else if (addr != this_addr) { + atomic_long_andnot(mask, (atomic_long_t *) addr); + mask = 0; + addr = this_addr; + } + mask |= (1UL << SB_NR_TO_BIT(sb, tag)); + } + + if (mask) + atomic_long_andnot(mask, (atomic_long_t *) addr); + + smp_mb__after_atomic(); + sbitmap_queue_wake_up(sbq); + sbitmap_update_cpu_hint(&sbq->sb, raw_smp_processor_id(), + tags[nr_tags - 1] - offset); +} + void sbitmap_queue_clear(struct sbitmap_queue *sbq, unsigned int nr, unsigned int cpu) { @@ -652,9 +692,7 @@ void sbitmap_queue_clear(struct sbitmap_queue *sbq, unsigned int nr, */ smp_mb__after_atomic(); sbitmap_queue_wake_up(sbq); - - if (likely(!sbq->sb.round_robin && nr < sbq->sb.depth)) - *per_cpu_ptr(sbq->sb.alloc_hint, cpu) = nr; + sbitmap_update_cpu_hint(&sbq->sb, cpu, nr); } EXPORT_SYMBOL_GPL(sbitmap_queue_clear); From patchwork Wed Oct 13 16:54:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E173C4332F for ; Wed, 13 Oct 2021 16:54:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 29AAE6112D for ; Wed, 13 Oct 2021 16:54:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237563AbhJMQ40 (ORCPT ); Wed, 13 Oct 2021 12:56:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237193AbhJMQ40 (ORCPT ); Wed, 13 Oct 2021 12:56:26 -0400 Received: from mail-il1-x12b.google.com (mail-il1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D79E1C061749 for ; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) Received: by mail-il1-x12b.google.com with SMTP id h10so426257ilq.3 for ; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UWWbxxl1nqVuRPyRl4TrQ64PvIRN1ySCGLzomh2MwOY=; b=CflgU2lNxFISUs7Wvc/XFgDqwIo5d6uA52jlkovsmf2d3JZ4uAkoCdFnq8suCN0ic/ Oh0lXdUwWq0YKA/gQES0XUq3P81BmEdIDQpxFiRGjkKitiUtafjm8ksVrehZby7DWjY0 GW1hrVL+KIuyhCiKIcso7Drlp/6KtYHYEKCbvVaI+EEFs74aO3ihzw9fXzXMhosEJ8wH rLu3xh9661EdHfkcy+hcH0GoPhIzUUSWT2+kjwxCmSFqk7DRjX3s4WBXSkNJoy0h8ZIW iEsKCx/9EGVZ7rSWjS/IjGPIMp8L0ih1aMF6iR1gN87fVQbd6ts7AZWTn06ZWodaj/5c 8Tmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UWWbxxl1nqVuRPyRl4TrQ64PvIRN1ySCGLzomh2MwOY=; b=pi7fbkf2Sm/BJXrCuiIlggDwl1KuTImIlYoSdshnw0AxXljIsstjV9jXIZzaPZ2IqP hemOX5j8PphXfzbCtB85c2fDLpRf5dKxsjp6VzMsvZzYs0gH0fnuEEjC/mGXn2awsXVU vsLRnPR/dJVov9fRiNM02B4uamucrE/AJTnARiTtqTKF2fudPKCxh5dR9ffT9dOCCSUM FHrCcozs84dMX4wXuRrZyFaA4nepJebKMJUoG6hRZXgFw96zUxQeVAuAzh3rhg1DyXkA 1v/rqt6MOeVimlXm5qKN+Tdd1Yoqc3X99LfSIbc/GelzXDcTohZ+M1gKJR1YzLemZNoI zilw== X-Gm-Message-State: AOAM533RMka8m5eg1hQuYo+hYjFxEHr2my/cVuTcqfFX+BCENIYcY8Ou LFYKpuk8JKzHZIEYWOU6cJN2rY+xsltaKg== X-Google-Smtp-Source: ABdhPJz0XpqF3ukDwj/1tNTioMwvr6SBUJaon891lNXBWnGTFI6QRM5p1x/cay79NvB7a/TEdCoywA== X-Received: by 2002:a05:6e02:1569:: with SMTP id k9mr89672ilu.317.1634144062169; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:21 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/9] sbitmap: test bit before calling test_and_set_bit() Date: Wed, 13 Oct 2021 10:54:11 -0600 Message-Id: <20211013165416.985696-5-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If we come across bits that are already set, then it's quicker to test that first and gate the test_and_set_bit() operation on the result of the bit test. Signed-off-by: Jens Axboe --- lib/sbitmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index c6e2f1f2c4d2..11b244a8d00f 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -166,7 +166,7 @@ static int __sbitmap_get_word(unsigned long *word, unsigned long depth, return -1; } - if (!test_and_set_bit_lock(nr, word)) + if (!test_bit(nr, word) && !test_and_set_bit_lock(nr, word)) break; hint = nr + 1; From patchwork Wed Oct 13 16:54:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18041C43217 for ; Wed, 13 Oct 2021 16:54:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05D176052B for ; Wed, 13 Oct 2021 16:54:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237621AbhJMQ41 (ORCPT ); Wed, 13 Oct 2021 12:56:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237193AbhJMQ40 (ORCPT ); Wed, 13 Oct 2021 12:56:26 -0400 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F184C061746 for ; Wed, 13 Oct 2021 09:54:23 -0700 (PDT) Received: by mail-il1-x12d.google.com with SMTP id d11so400993ilc.8 for ; Wed, 13 Oct 2021 09:54:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YaoawFoet/Xh6gz7KFsZtmgLG0LWFGggNfKqp0lP/Uk=; b=0dMqOIgJyr7YB+4S1XeW5qzegP8WwSexIsSv6u7JsoRr5aUc64Fj3M0pvIcR4b4fkf n+5wzVVmSz60SkA8Zh4EieodX2KvvaMusQ5npgqC7aN4U3MyL3fDgSRWbTMYRnhjwMeU Owh0b/tFvq8PDhc52xjIAVWFWbQEGx/LHGwbJVUpTavn7NtFogj2eUFRfKzyKsxHpWTg sKsf8Uur1VMkfHzKsbxWFMj2zz1A+v4Jq8P+AXsU0YHuX+VtS3JtBb9yES04khE2Wavr SPv1lVpmDeLASi/bhTsqZo37VNv2OO9ArI3uNcRsyD+UQ9JEln6mDZl4jGZoZ9KarRLq SS/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YaoawFoet/Xh6gz7KFsZtmgLG0LWFGggNfKqp0lP/Uk=; b=4x0WE2D8qSSssSzEGHNsskoizgeA6DrNaEbJvvNnDQlheKTRbAuI2qRQtUnHXAcUBD 4D9+5wFRMjOurlR0rBlEtxcIiVabMGEALWMEjhfmN7tg4lUu9pYAweTNiouafV39nKSt O/FORmZYN8z+a3mppysX7cAJgBxUZJolr3aDyFrYF+dVSY+ZzSHtm6HGZDZLPR1x6GRJ BoPnqamENClkpEsD82lufSxy1V3tP09stkDJo/ImVqcEas11E4dzOdOJppjR+iVJZyUN D4ehvPm3GhZT/kIY7zhEK8kb7Bp0bBpThZRswiGVvSQSPFYevrxKcu2/3/olBXYossrR Jr5w== X-Gm-Message-State: AOAM532tAcI+MiJFRc5wvaPt1c4b0cn8cc6uzK+tqYVcxxzyGMpaWd+q 4kLX6R2ub2+WVKjWnYjPRwhQThxf08e/tA== X-Google-Smtp-Source: ABdhPJze2azlIySMUtPxQScXsIYo09k7MYazvufALtZKvrccaaYjpdERCraZHg2H6mD1YVj3PXw0yw== X-Received: by 2002:a05:6e02:12c3:: with SMTP id i3mr109123ilm.145.1634144062818; Wed, 13 Oct 2021 09:54:22 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:22 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/9] block: add support for blk_mq_end_request_batch() Date: Wed, 13 Oct 2021 10:54:12 -0600 Message-Id: <20211013165416.985696-6-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Instead of calling blk_mq_end_request() on a single request, add a helper that takes the new struct io_batch and completes any request stored in there. Signed-off-by: Jens Axboe --- block/blk-mq-tag.c | 6 +++ block/blk-mq-tag.h | 1 + block/blk-mq.c | 85 +++++++++++++++++++++++++++++++++++++----- include/linux/blk-mq.h | 13 +++++++ 4 files changed, 95 insertions(+), 10 deletions(-) diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index c43b97201161..b94c3e8ef392 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -207,6 +207,12 @@ void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, } } +void blk_mq_put_tags(struct blk_mq_tags *tags, int *tag_array, int nr_tags) +{ + sbitmap_queue_clear_batch(&tags->bitmap_tags, tags->nr_reserved_tags, + tag_array, nr_tags); +} + struct bt_iter_data { struct blk_mq_hw_ctx *hctx; busy_iter_fn *fn; diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 71c2f7d8e9b7..78ae2fb8e2a4 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -42,6 +42,7 @@ unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags, unsigned int *offset); extern void blk_mq_put_tag(struct blk_mq_tags *tags, struct blk_mq_ctx *ctx, unsigned int tag); +void blk_mq_put_tags(struct blk_mq_tags *tags, int *tag_array, int nr_tags); extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx, struct blk_mq_tags **tags, unsigned int depth, bool can_grow); diff --git a/block/blk-mq.c b/block/blk-mq.c index 6eac10fd244e..d603703cf272 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -603,19 +603,22 @@ void blk_mq_free_plug_rqs(struct blk_plug *plug) } } -inline void __blk_mq_end_request(struct request *rq, blk_status_t error) +static inline void __blk_mq_end_request_acct(struct request *rq, + blk_status_t error, u64 now) { - if (blk_mq_need_time_stamp(rq)) { - u64 now = ktime_get_ns(); + if (rq->rq_flags & RQF_STATS) { + blk_mq_poll_stats_start(rq->q); + blk_stat_add(rq, now); + } - if (rq->rq_flags & RQF_STATS) { - blk_mq_poll_stats_start(rq->q); - blk_stat_add(rq, now); - } + blk_mq_sched_completed_request(rq, now); + blk_account_io_done(rq, now); +} - blk_mq_sched_completed_request(rq, now); - blk_account_io_done(rq, now); - } +inline void __blk_mq_end_request(struct request *rq, blk_status_t error) +{ + if (blk_mq_need_time_stamp(rq)) + __blk_mq_end_request_acct(rq, error, ktime_get_ns()); if (rq->end_io) { rq_qos_done(rq->q, rq); @@ -848,6 +851,68 @@ void blk_mq_end_request(struct request *rq, blk_status_t error) } EXPORT_SYMBOL(blk_mq_end_request); +#define TAG_COMP_BATCH 32 +#define TAG_SCHED_BATCH (TAG_COMP_BATCH >> 1) + +static inline void blk_mq_flush_tag_batch(struct blk_mq_hw_ctx *hctx, + int *tag_array, int nr_tags) +{ + struct request_queue *q = hctx->queue; + + blk_mq_put_tags(hctx->tags, tag_array, nr_tags); + if (q->elevator) + blk_mq_put_tags(hctx->sched_tags, &tag_array[TAG_SCHED_BATCH], + nr_tags); + percpu_ref_put_many(&q->q_usage_counter, nr_tags); + blk_mq_sched_restart(hctx); +} + +void blk_mq_end_request_batch(struct io_batch *iob) +{ + int tags[TAG_COMP_BATCH], nr_tags = 0, acct_tags = 0; + struct blk_mq_hw_ctx *last_hctx = NULL; + struct request *rq; + u64 now = 0; + + while ((rq = rq_list_pop(&iob->req_list)) != NULL) { + if (!now && blk_mq_need_time_stamp(rq)) + now = ktime_get_ns(); + blk_update_request(rq, rq->status, blk_rq_bytes(rq)); + __blk_mq_end_request_acct(rq, rq->status, now); + + if (rq->q->elevator) { + blk_mq_free_request(rq); + continue; + } + + if (!refcount_dec_and_test(&rq->ref)) + continue; + + blk_crypto_free_request(rq); + blk_pm_mark_last_busy(rq); + rq_qos_done(rq->q, rq); + WRITE_ONCE(rq->state, MQ_RQ_IDLE); + + if (acct_tags == TAG_COMP_BATCH || + (last_hctx && last_hctx != rq->mq_hctx)) { + blk_mq_flush_tag_batch(last_hctx, tags, nr_tags); + acct_tags = nr_tags = 0; + } + tags[nr_tags] = rq->tag; + last_hctx = rq->mq_hctx; + if (last_hctx->queue->elevator) { + tags[nr_tags + TAG_SCHED_BATCH] = rq->internal_tag; + acct_tags++; + } + nr_tags++; + acct_tags++; + } + + if (nr_tags) + blk_mq_flush_tag_batch(last_hctx, tags, nr_tags); +} +EXPORT_SYMBOL_GPL(blk_mq_end_request_batch); + static void blk_complete_reqs(struct llist_head *list) { struct llist_node *entry = llist_reverse_order(llist_del_all(list)); diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 5106c4cc411a..aea7d866a34c 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -183,9 +183,16 @@ struct request { unsigned int timeout; unsigned long deadline; + /* + * csd is used for remote completions, fifo_time at scheduler time. + * They are mutually exclusive. result is used at completion time + * like csd, but for batched IO. Batched IO does not use IPI + * completions. + */ union { struct __call_single_data csd; u64 fifo_time; + blk_status_t status; }; /* @@ -570,6 +577,11 @@ struct blk_mq_ops { */ void (*complete)(struct request *); + /** + * @complete_batch: Mark list of requests as complete + */ + void (*complete_batch)(struct io_batch *); + /** * @init_hctx: Called when the block layer side of a hardware queue has * been set up, allowing the driver to allocate/init matching @@ -759,6 +771,7 @@ static inline void blk_mq_set_request_complete(struct request *rq) void blk_mq_start_request(struct request *rq); void blk_mq_end_request(struct request *rq, blk_status_t error); void __blk_mq_end_request(struct request *rq, blk_status_t error); +void blk_mq_end_request_batch(struct io_batch *ib); void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list); void blk_mq_kick_requeue_list(struct request_queue *q); From patchwork Wed Oct 13 16:54:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE4C0C433EF for ; Wed, 13 Oct 2021 16:54:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C503861027 for ; Wed, 13 Oct 2021 16:54:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237782AbhJMQ42 (ORCPT ); Wed, 13 Oct 2021 12:56:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237650AbhJMQ42 (ORCPT ); Wed, 13 Oct 2021 12:56:28 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAC43C061746 for ; Wed, 13 Oct 2021 09:54:24 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id i189so484042ioa.1 for ; Wed, 13 Oct 2021 09:54:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=doaFOKk5ebYYoE13YQULp5d2Lpok+sMDrhuPP3+ZyVc=; b=HM0nqPvHkVUfP00UfLItNwBwB/yXC02o2juVZqb/BQTJbzmkIyZsulQNGSOupWRuQ0 bKGCqpgsXI15fgEeiGUpZfHSj0PcmometO7whNuTCfKXbNa5vXi6YVqysLACMaf0+UpL QTrgnpcpaoSQ3PMuhVxxRTKKrdM/fGh7wfGKqwzFmP5I22HLBN5Y4F/mKn8SDaRWwLOk IjNWB5Sg7oR91SZa1uVahJf4AuqfXcXUo/BqKhgb4ELtkYBxIH7Q0Euav/8OTt4raxne qKFrsGMU+exQ6whXgh4wP9C+D27mASSXXJie12JWgVoyqUkFQ4COtDrzrA+zj8LFfVbP 0IVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=doaFOKk5ebYYoE13YQULp5d2Lpok+sMDrhuPP3+ZyVc=; b=DDlDLUwuX7jGFNp3160eDnmsToOQr/Uy7l3sVOkTJvMl70zPWZMdcXKC6P98jA8RKe daqrlbXVlWWtdoFBekXGFtv2O8Nu6w85nmvHWfjQmuHyg/lWJvBvNXtnSdjwbjf+YkcE bR2xO7OEblAPunfDGS6GJ9hiqWzwW0Wc2mlPsX3o8PjsKWIpkQYxbxd4KAnwajAOf9xe VMmEzaQoe7Fs+UPj3pzCKcj0ZBn0TzUb3O2rE0tnwBDMsxMnCIR21oRpYnWG+JxVSmz7 5dbxUBnsRkh4i/70iQ7HKaTnpLFkw1GFNCctowMrdYyv7BZ9cOmZ8xxKeQpX21GxCE2j HVAw== X-Gm-Message-State: AOAM532oEkapoL4VMUALvcuOXtTEaLbHcuexusn9zJ5pu9wsl7hU7t+k KeLX7gEm353pu2WfbeiAqWzVyghuL1dlIw== X-Google-Smtp-Source: ABdhPJwHuFFlPk7IEEyBjbyv2uqYu7un0eXfsDF/M2jVKAifwkKwbfV9n7CxGEOot5qj3MrKTONXZw== X-Received: by 2002:a5e:9612:: with SMTP id a18mr282315ioq.57.1634144063921; Wed, 13 Oct 2021 09:54:23 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:23 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/9] nvme: add support for batched completion of polled IO Date: Wed, 13 Oct 2021 10:54:13 -0600 Message-Id: <20211013165416.985696-7-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take advantage of struct io_batch, if passed in to the nvme poll handler. If it's set, rather than complete each request individually inline, store them in the io_batch list. We only do so for requests that will complete successfully, anything else will be completed inline as before. Add an mq_ops->complete_batch() handler to do the post-processing of the io_batch list once polling is complete. Signed-off-by: Jens Axboe --- drivers/nvme/host/core.c | 18 +++++++++++--- drivers/nvme/host/nvme.h | 17 +++++++++++++ drivers/nvme/host/pci.c | 54 +++++++++++++++++++++++++++++++++++----- 3 files changed, 80 insertions(+), 9 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index c2c2e8545292..4b14258a3bac 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -346,15 +346,19 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req) return RETRY; } -static inline void nvme_end_req(struct request *req) +static inline void nvme_end_req_zoned(struct request *req) { - blk_status_t status = nvme_error_status(nvme_req(req)->status); - if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) && req_op(req) == REQ_OP_ZONE_APPEND) req->__sector = nvme_lba_to_sect(req->q->queuedata, le64_to_cpu(nvme_req(req)->result.u64)); +} + +static inline void nvme_end_req(struct request *req) +{ + blk_status_t status = nvme_error_status(nvme_req(req)->status); + nvme_end_req_zoned(req); nvme_trace_bio_complete(req); blk_mq_end_request(req, status); } @@ -381,6 +385,14 @@ void nvme_complete_rq(struct request *req) } EXPORT_SYMBOL_GPL(nvme_complete_rq); +void nvme_complete_batch_req(struct request *req) +{ + nvme_cleanup_cmd(req); + nvme_end_req_zoned(req); + req->status = BLK_STS_OK; +} +EXPORT_SYMBOL_GPL(nvme_complete_batch_req); + /* * Called to unwind from ->queue_rq on a failed command submission so that the * multipathing code gets called to potentially failover to another path. diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index ed79a6c7e804..e0c079f704cf 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -638,6 +638,23 @@ static inline bool nvme_is_aen_req(u16 qid, __u16 command_id) } void nvme_complete_rq(struct request *req); +void nvme_complete_batch_req(struct request *req); + +static __always_inline void nvme_complete_batch(struct io_batch *iob, + void (*fn)(struct request *rq)) +{ + struct request *req; + + req = rq_list_peek(&iob->req_list); + while (req) { + fn(req); + nvme_complete_batch_req(req); + req = rq_list_next(req); + } + + blk_mq_end_request_batch(iob); +} + blk_status_t nvme_host_path_error(struct request *req); bool nvme_cancel_request(struct request *req, void *data, bool reserved); void nvme_cancel_tagset(struct nvme_ctrl *ctrl); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 9db6e23f41ef..ae253f6f5c80 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -959,7 +959,7 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } -static void nvme_pci_complete_rq(struct request *req) +static __always_inline void nvme_pci_unmap_rq(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); struct nvme_dev *dev = iod->nvmeq->dev; @@ -969,9 +969,19 @@ static void nvme_pci_complete_rq(struct request *req) rq_integrity_vec(req)->bv_len, rq_data_dir(req)); if (blk_rq_nr_phys_segments(req)) nvme_unmap_data(dev, req); +} + +static void nvme_pci_complete_rq(struct request *req) +{ + nvme_pci_unmap_rq(req); nvme_complete_rq(req); } +static void nvme_pci_complete_batch(struct io_batch *iob) +{ + nvme_complete_batch(iob, nvme_pci_unmap_rq); +} + /* We read the CQE phase first to check if the rest of the entry is valid */ static inline bool nvme_cqe_pending(struct nvme_queue *nvmeq) { @@ -996,7 +1006,8 @@ static inline struct blk_mq_tags *nvme_queue_tagset(struct nvme_queue *nvmeq) return nvmeq->dev->tagset.tags[nvmeq->qid - 1]; } -static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx) +static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, + struct io_batch *iob, u16 idx) { struct nvme_completion *cqe = &nvmeq->cqes[idx]; __u16 command_id = READ_ONCE(cqe->command_id); @@ -1023,8 +1034,17 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx) } trace_nvme_sq(req, cqe->sq_head, nvmeq->sq_tail); - if (!nvme_try_complete_req(req, cqe->status, cqe->result)) - nvme_pci_complete_rq(req); + if (!nvme_try_complete_req(req, cqe->status, cqe->result)) { + /* + * Do normal inline completion if we don't have a batch + * list, if we have an end_io handler, or if the status of + * the request isn't just normal success. + */ + if (!iob || req->end_io || nvme_req(req)->status) + nvme_pci_complete_rq(req); + else + rq_list_add_tail(&iob->req_list, req); + } } static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) @@ -1050,7 +1070,7 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq) * the cqe requires a full read memory barrier */ dma_rmb(); - nvme_handle_cqe(nvmeq, nvmeq->cq_head); + nvme_handle_cqe(nvmeq, NULL, nvmeq->cq_head); nvme_update_cq_head(nvmeq); } @@ -1092,6 +1112,27 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq) enable_irq(pci_irq_vector(pdev, nvmeq->cq_vector)); } +static inline int nvme_poll_cq(struct nvme_queue *nvmeq, struct io_batch *iob) +{ + int found = 0; + + while (nvme_cqe_pending(nvmeq)) { + found++; + /* + * load-load control dependency between phase and the rest of + * the cqe requires a full read memory barrier + */ + dma_rmb(); + nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head); + nvme_update_cq_head(nvmeq); + } + + if (found) + nvme_ring_cq_doorbell(nvmeq); + return found; +} + + static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) { struct nvme_queue *nvmeq = hctx->driver_data; @@ -1101,7 +1142,7 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_batch *iob) return 0; spin_lock(&nvmeq->cq_poll_lock); - found = nvme_process_cq(nvmeq); + found = nvme_poll_cq(nvmeq, iob); spin_unlock(&nvmeq->cq_poll_lock); return found; @@ -1639,6 +1680,7 @@ static const struct blk_mq_ops nvme_mq_admin_ops = { static const struct blk_mq_ops nvme_mq_ops = { .queue_rq = nvme_queue_rq, .complete = nvme_pci_complete_rq, + .complete_batch = nvme_pci_complete_batch, .commit_rqs = nvme_commit_rqs, .init_hctx = nvme_init_hctx, .init_request = nvme_init_request, From patchwork Wed Oct 13 16:54:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 046CAC433F5 for ; Wed, 13 Oct 2021 16:54:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DBB596052B for ; Wed, 13 Oct 2021 16:54:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237763AbhJMQ4b (ORCPT ); Wed, 13 Oct 2021 12:56:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237650AbhJMQ43 (ORCPT ); Wed, 13 Oct 2021 12:56:29 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC2E6C061570 for ; Wed, 13 Oct 2021 09:54:25 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id 5so435532iov.9 for ; Wed, 13 Oct 2021 09:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JoLh29ngTtiaznbE3jjvzkQQaGtKKfQvQt74EVggIEg=; b=ojflR0MeYp8GskxfZ4BThN1zUppaAVA4lOkPGpGCfYKft+2ELg5/bkgwffXeuwECCr Nq4jg6szTxj/VIsNT2qtWYPyp0ZM6MrPiJebuL4ThQq2tYsOQqD7HnlzAIClKGUYlDXm lPZMbiPqEksiP2wqu0QEMOeJP4tDxUAO33S8blp4t+fzfBdNPwjYGs97/Ai9qho5vGw2 OHqDZhFefSwBDWZOp8GDBLPGreBUGpt3LXlgsyDOAEHCPg37vGpqGBi6glBIEQM+RJgF 2IuwRkaE4RuKf2OmGeQJIqL2mANeSH7PGwx7yT527Bw5w4Sp0IxqVwtZTSOFjw5VCu0c 8qOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JoLh29ngTtiaznbE3jjvzkQQaGtKKfQvQt74EVggIEg=; b=esVgsN98nmpZGTrMGp56oBDgTu7prqM1bIbbFAFqdop8Z2h8VMKRAb7OG6dgozXT99 1PV6T0NqhM9+cnt/PNUc/IgdipykbCON/V+900q1by6Vw1n3XMrITu6s1Mp/s42zKb60 O/1zbGK+GXnxrb9B4nzTHG2yhWR3nLNTTm891EgxpDwV1guF1UUwHaHYqMZw3LsvuLoA 0DWwCK8vAOm1kKMM3vIaTzXzwEYzw5cf9WjWX9+Qo7UNDDAXoMrbp/LRtAtz4My2qbYB cdk927XEb8rD59e3H4fUuP9D1Z+zwtQ9KIZP7AP4N55im9/vYEFQW8WY5iIlpAsKgvh2 Ac8w== X-Gm-Message-State: AOAM532fLfywv8p4SVjlThxIKD6W6uOuV77WWPuaHGhbT4kf8L8wvzff zOH/QlRgNXnr9RY19jTnarZgnZeWkxIstA== X-Google-Smtp-Source: ABdhPJwap4nbDUwXK8iiUx0ok2sTq8nOtjmolPqIuC5/koPY9Cjcx2UfkEWIk00d4nbRyFZSWm7LlA== X-Received: by 2002:a05:6602:2c07:: with SMTP id w7mr320453iov.122.1634144064610; Wed, 13 Oct 2021 09:54:24 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:24 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 7/9] block: assign batch completion handler in blk_poll() Date: Wed, 13 Oct 2021 10:54:14 -0600 Message-Id: <20211013165416.985696-8-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If an io_batch is passed in to blk_poll(), we need to assign the batch handler associated with this queue. This allows callers to complete an io_batch handler on by calling it. Signed-off-by: Jens Axboe --- block/blk-mq.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index d603703cf272..deafd444761d 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -4302,6 +4302,19 @@ static int blk_mq_poll_classic(struct request_queue *q, blk_qc_t cookie, hctx->poll_considered++; + /* + * If batching is requested but the target doesn't support batched + * completions, then just clear ib and completions will be handled + * normally. + */ + if (iob) { + iob->complete = q->mq_ops->complete_batch; + if (!iob->complete) { + WARN_ON_ONCE(iob->req_list); + iob = NULL; + } + } + do { hctx->poll_invoked++; From patchwork Wed Oct 13 16:54:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0A85C433EF for ; Wed, 13 Oct 2021 16:54:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B715F6052B for ; Wed, 13 Oct 2021 16:54:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237193AbhJMQ4e (ORCPT ); Wed, 13 Oct 2021 12:56:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237785AbhJMQ43 (ORCPT ); Wed, 13 Oct 2021 12:56:29 -0400 Received: from mail-il1-x12c.google.com (mail-il1-x12c.google.com [IPv6:2607:f8b0:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D601C061570 for ; Wed, 13 Oct 2021 09:54:26 -0700 (PDT) Received: by mail-il1-x12c.google.com with SMTP id g2so437864ild.1 for ; Wed, 13 Oct 2021 09:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NQbLAMYZvChZM1eTcfqVnB+m/wLbIopxB4L7QBJlFdU=; b=7NpYDl9IUxlyrxikKFILhgejJP22cql/pkHdWWS6DKS4yvufoevkSOfVrVThBLmKfF gw0PLyLS6IFHx1NmzcOzkkpdFsEkz9ZC80Mb8lf+XgyCRos5PkC471nYhhIz6Em2PGG4 bo2a9CKBs6170FetKuHruhnh5X8pPrZ24HvBy1mwJp38xNGQQDEecYe8TRITKeiG/PEY QMUOqWVb8Oyk2dk3g88sWdPJp0iWj9PmPTATz5X4jpwnxs4mFpBfj9hQ0czuB9M7Ypdu B1cVqEJeGjK4mhXcBrn6io8v5JmasLxtOzGPKEIb3+Pa1I3Vqjzv0nyIWAARJdtK2Ct4 +QpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NQbLAMYZvChZM1eTcfqVnB+m/wLbIopxB4L7QBJlFdU=; b=RkpTEyDUl55vdS//8FQwIALpE3xSh7q4g/dMvH4TXLk/xuWRyWwUsKRmn2xEkQC6s0 sj4RuzkQBCf5lIVmlNkmVGCVXT1EUaPtAxccv6bW4y20MVwretguq1tHVGyWQWSKyA1A O0C7PLLk4+lbyeGYZbmLpRyGvTLz93RMJvWi7Gj6kp4TPf83nKWh6IsDcJzS9/z1WBLm G2zLNcwPKi7HejN5XRG/ont7Xzjieg55QNWve3cAepObReQCqMV0HeOFBQYqVwo6xeuR QWbYqn4IqkRpu9/YQR34CLSOigcqLSfk1Zik+2gvisIXq2pYtxOikLArA4Gg7qpoCAAp h2TA== X-Gm-Message-State: AOAM530hmJ7aovDAUAEmVRY/5j3+34ZSK7fPotIdGvDBsYF/axMVFDAi zDQ8oFC1bhWgMIYspKq3GgbkTxQlO9fRtA== X-Google-Smtp-Source: ABdhPJxgFrCNDC46tAEXdRHVTjTHJxfmS5C7xXMo/PBovScHJw5ot4naQhCMSWaI8d5B0jWrVv95qA== X-Received: by 2002:a05:6e02:148f:: with SMTP id n15mr107849ilk.121.1634144065783; Wed, 13 Oct 2021 09:54:25 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:24 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 8/9] io_uring: utilize the io_batch infrastructure for more efficient polled IO Date: Wed, 13 Oct 2021 10:54:15 -0600 Message-Id: <20211013165416.985696-9-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Wire up using an io_batch for f_op->iopoll(). If the lower stack supports it, we can handle high rates of polled IO more efficiently. This raises the single core efficiency on my system from ~6.1M IOPS to ~6.6M IOPS running a random read workload at depth 128 on two gen2 Optane drives. Signed-off-by: Jens Axboe --- fs/io_uring.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 082ff64c1bcb..5c031ab8f77f 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2390,6 +2390,8 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) { struct io_wq_work_node *pos, *start, *prev; unsigned int poll_flags = BLK_POLL_NOSLEEP; + struct file *file = NULL; + DEFINE_IO_BATCH(iob); int nr_events = 0; /* @@ -2404,6 +2406,11 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) struct kiocb *kiocb = &req->rw.kiocb; int ret; + if (!file) + file = kiocb->ki_filp; + else if (file != kiocb->ki_filp) + break; + /* * Move completed and retryable entries to our local lists. * If we find a request that requires polling, break out @@ -2412,19 +2419,21 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) if (READ_ONCE(req->iopoll_completed)) break; - ret = kiocb->ki_filp->f_op->iopoll(kiocb, NULL, poll_flags); + ret = kiocb->ki_filp->f_op->iopoll(kiocb, &iob, poll_flags); if (unlikely(ret < 0)) return ret; else if (ret) poll_flags |= BLK_POLL_ONESHOT; /* iopoll may have completed current req */ - if (READ_ONCE(req->iopoll_completed)) + if (iob.req_list || READ_ONCE(req->iopoll_completed)) break; } - if (!pos) + if (!pos && !iob.req_list) return 0; + if (iob.req_list) + iob.complete(&iob); prev = start; wq_list_for_each_resume(pos, prev) { From patchwork Wed Oct 13 16:54:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12556383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6175C433F5 for ; Wed, 13 Oct 2021 16:54:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C2D5061053 for ; Wed, 13 Oct 2021 16:54:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237855AbhJMQ4g (ORCPT ); Wed, 13 Oct 2021 12:56:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237819AbhJMQ4a (ORCPT ); Wed, 13 Oct 2021 12:56:30 -0400 Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30571C06174E for ; Wed, 13 Oct 2021 09:54:27 -0700 (PDT) Received: by mail-il1-x133.google.com with SMTP id d11so401171ilc.8 for ; Wed, 13 Oct 2021 09:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=u6GVj+VGurrqDHVBNA0HHjEZpwWFZJrLtzTyFowWRbw=; b=eVuUAlCM8/qj0n/oCc6R3BfPwPWPesXR04LH38vjObMabE6ziFFbde54TMF0QqkKvW IydqXTfV9KQg3/UkMQc0qFhdHMCD6pCMJa97fB53IUkuZhOswaUwCZikdmBM9qydhTJp P2691ofF/uhHa0KP51590b/z44nNxDlxkDn9AWFQIHcDuwkq/fwcGuQpZ2GAIoLaLN4M ijA8NxlZ6xd0UaQAFj/nijbctA1sNsqI4iAQiAIH+CP9Sbd7gIGM4FwDCsh1EFOGjzu/ NfMmZpyAUec9CjyS34yuyaPKAxtEIetGg3FJfXg766Mr4MocZwFgVOBWYuxmWVqnxEqk 7VEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u6GVj+VGurrqDHVBNA0HHjEZpwWFZJrLtzTyFowWRbw=; b=on+/v7OUS5XkAVlQtyudhfARjxBIIbcFK+aNFTpjNriqDu7xKkLP9yKYn5vXugo2jj 1uFs1Q1BG6vg2N6aFsWFsozFxrT6QvsN026Gsdc9/mJ+AayXMGZvKDBYNwpO8jSI8dUu vQFMFFkfNs6vjnxTqEdjYZQhVjLuM4gekMEd9KsaV0mEfmzPmIkwgAMrNZs7aniNC4n3 DNbY433fdxTC3aESIOsyXp/Mt7zjjq9bxGSSfK2wu7G2svkQR+zhe3UUQPsudukjaMJl ftUNATO8IF5KmN9WFeSlUkwEHEHby023+0oGbm/8t0fu9iZT+Hsp2k3iKE7Iz01cIG8M rruQ== X-Gm-Message-State: AOAM530iaM0kKDWt/ts/2bp87mLSWhv6GHew+QEbYzKLYnwywLvPtmZb +AfRu8WaW9kERmg1P06aeW6pScgKmpd0lA== X-Google-Smtp-Source: ABdhPJxYZ73FtvBxrm+5xjOAmv8nj52rXg//pBZfA2+hllzFjassYveU3xzMbCGiFBmx0EHj3Mm79A== X-Received: by 2002:a92:c244:: with SMTP id k4mr124927ilo.3.1634144066482; Wed, 13 Oct 2021 09:54:26 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id r7sm65023ior.25.2021.10.13.09.54.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:54:26 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 9/9] nvme: wire up completion batching for the IRQ path Date: Wed, 13 Oct 2021 10:54:16 -0600 Message-Id: <20211013165416.985696-10-axboe@kernel.dk> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013165416.985696-1-axboe@kernel.dk> References: <20211013165416.985696-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Trivial to do now, just need our own io_batch on the stack and pass that in to the usual command completion handling. I pondered making this dependent on how many entries we had to process, but even for a single entry there's no discernable difference in performance or latency. Running a sync workload over io_uring: t/io_uring -b512 -d1 -s1 -c1 -p0 -F1 -B1 -n2 /dev/nvme1n1 /dev/nvme2n1 yields the below performance before the patch: IOPS=254820, BW=124MiB/s, IOS/call=1/1, inflight=(1 1) IOPS=251174, BW=122MiB/s, IOS/call=1/1, inflight=(1 1) IOPS=250806, BW=122MiB/s, IOS/call=1/1, inflight=(1 1) and the following after: IOPS=255972, BW=124MiB/s, IOS/call=1/1, inflight=(1 1) IOPS=251920, BW=123MiB/s, IOS/call=1/1, inflight=(1 1) IOPS=251794, BW=122MiB/s, IOS/call=1/1, inflight=(1 1) which definitely isn't slower, about the same if you factor in a bit of variance. For peak performance workloads, benchmarking shows a 2% improvement. Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index ae253f6f5c80..061f0b1cb0ec 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1061,6 +1061,7 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq) static inline int nvme_process_cq(struct nvme_queue *nvmeq) { + DEFINE_IO_BATCH(iob); int found = 0; while (nvme_cqe_pending(nvmeq)) { @@ -1070,12 +1071,15 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq) * the cqe requires a full read memory barrier */ dma_rmb(); - nvme_handle_cqe(nvmeq, NULL, nvmeq->cq_head); + nvme_handle_cqe(nvmeq, &iob, nvmeq->cq_head); nvme_update_cq_head(nvmeq); } - if (found) + if (found) { + if (iob.req_list) + nvme_pci_complete_batch(&iob); nvme_ring_cq_doorbell(nvmeq); + } return found; }