From patchwork Thu Jun 18 14:43:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612379 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A0ECB618 for ; Thu, 18 Jun 2020 14:46:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 81D1620899 for ; Thu, 18 Jun 2020 14:46:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="XjGUImD9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731210AbgFROq2 (ORCPT ); Thu, 18 Jun 2020 10:46:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730926AbgFROoB (ORCPT ); Thu, 18 Jun 2020 10:44:01 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67047C0613EF for ; Thu, 18 Jun 2020 07:44:01 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id i12so2641774pju.3 for ; Thu, 18 Jun 2020 07:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=AMBC//GM9TUmKqc/Ee7MUTX4MVxXaivvBIm8XTKWtjI=; b=XjGUImD9rS9VeH3ATYR5UN46KsG/kGd5i8ObOVULAJ3bWbAHLUKz+Nt/bzRQeCC7k+ q4pmWPc7rGRsnlJnmLarBBAj7QbxKCn+OtJBti2v/itk1DKcuvDFfV26CpsOgPwwVj46 oKdSZT+E6O3bgsFXk2UysYjSDRy6zj4yUK78Rv1eqtIwgcAQ5nn/tpiH68Y8GD4/iHUw 3O1/2ihPs06whdoc10VZDpvDnnO0QVgM3L+4vMq/X6KTuGPCEjHx9ydMZTrOPjVK9Bkv Qkf+MhMg3TkdzkCd8QX51qCa7+tmj/0j1O1H3Kj4Zv1sGjsxfa6mxtY9/zGOSIjiER3+ piRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=AMBC//GM9TUmKqc/Ee7MUTX4MVxXaivvBIm8XTKWtjI=; b=odpzdXd0Dx0jtaiEAP72AooAF2Szu9jPW9wTBljpLx8AiemLHumtl5F/sh2Mc+33ij 3vedHzk6nwlsKQD2Pf7s/4TqWd0TuQhv4PULIXqxHkBODnjMF4RxGgcFWypMk3Q5kZ5X 5M63jn5rToQID8F7lfYv9nze67i5K1uUpuBONieNDRlhtEbMYrfJdiUpz1pFaTEjisS7 lfkO7wRJxLZ462u8qWIVXKDokfwVxCiGrYleP45X/0a9tzvqzS9yhpmVkgfS5/jrFcpr rmZFmLH0/9HxInafa0W3rJ/qBIUH+R2iQ6VGw3FIjAu5FyzaxWofTNtpJ/D005LztFCM wFqw== X-Gm-Message-State: AOAM533ZL48gNmjzwQZ9epbYwDuo8R4FHEweM1aqH/0y1K/3b0QlM9Sd fL6z37lD96W8wGy1oR5YkW1SsA== X-Google-Smtp-Source: ABdhPJxPg9RpVdrGQ4Z6ww3Iq/Db4vGVv54cHwt8FNsBk1NOsUw3hrbUnoL/Xc/AmyjDRE+S7btjoQ== X-Received: by 2002:a17:90b:3004:: with SMTP id hg4mr5050010pjb.208.1592491440947; Thu, 18 Jun 2020 07:44:00 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:00 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 01/15] block: provide plug based way of signaling forced no-wait semantics Date: Thu, 18 Jun 2020 08:43:41 -0600 Message-Id: <20200618144355.17324-2-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Provide a way for the caller to specify that IO should be marked with REQ_NOWAIT to avoid blocking on allocation. Signed-off-by: Jens Axboe --- block/blk-core.c | 6 ++++++ include/linux/blkdev.h | 1 + 2 files changed, 7 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index 03252af8c82c..62a4904db921 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -958,6 +958,7 @@ generic_make_request_checks(struct bio *bio) struct request_queue *q; int nr_sectors = bio_sectors(bio); blk_status_t status = BLK_STS_IOERR; + struct blk_plug *plug; char b[BDEVNAME_SIZE]; might_sleep(); @@ -971,6 +972,10 @@ generic_make_request_checks(struct bio *bio) goto end_io; } + plug = blk_mq_plug(q, bio); + if (plug && plug->nowait) + bio->bi_opf |= REQ_NOWAIT; + /* * For a REQ_NOWAIT based request, return -EOPNOTSUPP * if queue is not a request based queue. @@ -1800,6 +1805,7 @@ void blk_start_plug(struct blk_plug *plug) INIT_LIST_HEAD(&plug->cb_list); plug->rq_count = 0; plug->multiple_queues = false; + plug->nowait = false; /* * Store ordering should not be needed here, since a potential diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 8fd900998b4e..6e067dca94cf 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1189,6 +1189,7 @@ struct blk_plug { struct list_head cb_list; /* md requires an unplug callback */ unsigned short rq_count; bool multiple_queues; + bool nowait; }; #define BLK_MAX_REQUEST_COUNT 16 #define BLK_PLUG_FLUSH_SIZE (128 * 1024) From patchwork Thu Jun 18 14:43:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612325 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99EBE913 for ; Thu, 18 Jun 2020 14:44:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80FCE208B3 for ; Thu, 18 Jun 2020 14:44:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="yzp2+WvV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730958AbgFROoJ (ORCPT ); Thu, 18 Jun 2020 10:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730931AbgFROoD (ORCPT ); Thu, 18 Jun 2020 10:44:03 -0400 Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1691AC06174E for ; Thu, 18 Jun 2020 07:44:03 -0700 (PDT) Received: by mail-pj1-x1042.google.com with SMTP id jz3so2649903pjb.0 for ; Thu, 18 Jun 2020 07:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=aZaDFCszZJWwvw+YU15551W6DS/tp5Q0zZ/e2mzcPKg=; b=yzp2+WvVYDYAjxYEzGcL7PkavHxDy3cpr9qzW1IWcotTTR9f4VhesgN5dhZI94+PsU D9XrCxqsfO+3+KrzN5/i59EGbZklLXhJ8EEfRkcKjS+RSy9l3JJAAb3DNGy49QESvyup y+54KKaMpJCnfzL+F+KC8Nq7RiZ+0vIr9IiDsumf++0aVkCJenAnaKTtA3JP5ygXQhYB 2HbmH52rlEtjDxPlJY6OwsA0PBg77wgg7/2VGxIMl44vnzaycw7+hUNxlA6Vq9Gzvp1p BDytFguC2D8/8IQpvQ3NiTjxRjsGSXn+74N6mA5zAWSn0EOSGif2RsltVaUOkZ/y9GnI oyUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=aZaDFCszZJWwvw+YU15551W6DS/tp5Q0zZ/e2mzcPKg=; b=XlZrXGmqzi2rBP8ICJYkvoVtMt3KpogxEnVF+OghrM2UP3EnLuy6KXP2GFhixGjPHR N2q3RnMkk4xOinKugp+IRL1AJvmFUBWfCe0nMfOB/12+mcpOKT4fkr4MmXqmEW2wyp34 X6Bgv9QUjAZY2GvFR3TbHE4RSDKd0jxv+juJWZAaj3k9BmLtuvMp8doBWBQsWHqbZRLR C3HiwQCovdyKN0Dhq+lyGXqXXQwlj60+YEy+RzpALIkYfobWAUlr9BZsSEgLh8H4bvgB fJhFjbwhDZaxr9sosYepU0brOM8lyM5stNWPDJsE+Y8s5mfiq4KZPRaaC0LFhGiYNQwn Y19Q== X-Gm-Message-State: AOAM531JXkxj7LtxJ7pW7j7rs/Z89hRVjzadd2L/Tm+C+2+Zz4kqzOci A+7Pq+db47lNXiEb7oWbbwl0sQ== X-Google-Smtp-Source: ABdhPJwcpBuWJToBprbE+kBnwBY9JxdUaCcM4SRk7uPRoXVsDbkErSY+i9Q0PI2mKLg9/3NhPQrRsQ== X-Received: by 2002:a17:90b:915:: with SMTP id bo21mr4702163pjb.52.1592491442581; Thu, 18 Jun 2020 07:44:02 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:01 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 02/15] io_uring: always plug for any number of IOs Date: Thu, 18 Jun 2020 08:43:42 -0600 Message-Id: <20200618144355.17324-3-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently we only plug if we're doing more than two request. We're going to be relying on always having the plug there to pass down information, so plug unconditionally. Signed-off-by: Jens Axboe --- fs/io_uring.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index b14a8e6a0e15..ca78dd7c79da 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -676,7 +676,6 @@ struct io_kiocb { }; }; -#define IO_PLUG_THRESHOLD 2 #define IO_IOPOLL_BATCH 8 struct io_submit_state { @@ -5914,7 +5913,7 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, struct file *ring_file, int ring_fd) { - struct io_submit_state state, *statep = NULL; + struct io_submit_state state; struct io_kiocb *link = NULL; int i, submitted = 0; @@ -5931,10 +5930,7 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, if (!percpu_ref_tryget_many(&ctx->refs, nr)) return -EAGAIN; - if (nr > IO_PLUG_THRESHOLD) { - io_submit_state_start(&state, nr); - statep = &state; - } + io_submit_state_start(&state, nr); ctx->ring_fd = ring_fd; ctx->ring_file = ring_file; @@ -5949,14 +5945,14 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, io_consume_sqe(ctx); break; } - req = io_alloc_req(ctx, statep); + req = io_alloc_req(ctx, &state); if (unlikely(!req)) { if (!submitted) submitted = -EAGAIN; break; } - err = io_init_req(ctx, req, sqe, statep); + err = io_init_req(ctx, req, sqe, &state); io_consume_sqe(ctx); /* will complete beyond this point, count as submitted */ submitted++; @@ -5982,8 +5978,7 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, } if (link) io_queue_link_head(link); - if (statep) - io_submit_state_end(&state); + io_submit_state_end(&state); /* Commit SQ ring head once we've consumed and submitted all SQEs */ io_commit_sqring(ctx); From patchwork Thu Jun 18 14:43:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612331 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5EBE14E3 for ; Thu, 18 Jun 2020 14:44:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B913820897 for ; Thu, 18 Jun 2020 14:44:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="uL0fwmVf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730969AbgFROoO (ORCPT ); Thu, 18 Jun 2020 10:44:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730943AbgFROoF (ORCPT ); Thu, 18 Jun 2020 10:44:05 -0400 Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4599FC061795 for ; Thu, 18 Jun 2020 07:44:04 -0700 (PDT) Received: by mail-pj1-x1041.google.com with SMTP id ne5so2633345pjb.5 for ; Thu, 18 Jun 2020 07:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=HUMIpAnpDXTm7durxzsxq78uIppsf6NP0VS0tAu1naU=; b=uL0fwmVf7za35bF7WmhIeuLwTRYPYomNz3qpWAcNPJ4MG8GTRnh40j/G/FSQGZeqHn yQkV1IezTLpFjD50TYMgv1HY78G4RgY498zLaJMuA+l8xEee6zj/2I82z6xH0BShFzod Jw88wuY/pILeFUtVbhY9sbTrVoWY7AC3u3FJAQooqqxLaL58sSt1DjQd+DiakJiej111 3QKNAaz9kpVK0JkkwmB/tU2PK7EUI1l3/XQW2Hhdy1LlQkxSISsyAWXCpJiSM6HpaavS QcI6gfKDo0Ou2qvSuNEV0Pw2sksh+dDzkS5v9GpxMofkwta1H+6MYdHHM9azIlc5CFhg lSog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=HUMIpAnpDXTm7durxzsxq78uIppsf6NP0VS0tAu1naU=; b=gPLMzfRRD0eYpLH5DUR/AD6Pe1KTvb12wXQTtv6WmJdkMzHQGef+QV/e25KAtI3AeH jXRxjcG4SwhAWYOdeewMCNYeA+bcZkiu8XXFM4HSdOxLQykNq1UdT0WiT5GavmNFxj/O paLjWAjRWmSYbrzNQJF5pAmAWodHsnWDqZa7L1hzRgLnX4sRQIW+YROOPHpmTs3irV4R vbVsszPdqVm6K8adu+oY3uSmqQIzpdAGnCkhml50W40UOOEOF+lV025xS0o9jWt64MbC lcyNRtNVYThk6f1W3Pk2lzuUo1f4Y8h8A6ifU5AZAzO28Ksj3sZZ957aEy0M1SR2L8LP 9xRQ== X-Gm-Message-State: AOAM530nKzVwrUz5R9XrhgBe/RIHPd9ppR8IOSIXOrBOmcrCkWACZS5v u4wyyEEMVL8lhiZfC/XOEFNNFQ== X-Google-Smtp-Source: ABdhPJzURLyFzMozy7yIPWEvSTKNlwHInRbt5PeVmy/Fa+Ew58olXOewrYlOoVFbj5T5wAt30tH4HQ== X-Received: by 2002:a17:90a:de1:: with SMTP id 88mr4809422pjv.124.1592491443796; Thu, 18 Jun 2020 07:44:03 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:03 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 03/15] io_uring: catch -EIO from buffered issue request failure Date: Thu, 18 Jun 2020 08:43:43 -0600 Message-Id: <20200618144355.17324-4-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org -EIO bubbles up like -EAGAIN if we fail to allocate a request at the lower level. Play it safe and treat it like -EAGAIN in terms of sync retry, to avoid passing back an errant -EIO. Catch some of these early for block based file, as non-mq devices generally do not support NOWAIT. That saves us some overhead by not first trying, then retrying from async context. We can go straight to async punt instead. Signed-off-by: Jens Axboe --- fs/io_uring.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index ca78dd7c79da..2e257c5a1866 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2088,6 +2088,15 @@ static struct file *__io_file_get(struct io_submit_state *state, int fd) return state->file; } +static bool io_bdev_nowait(struct block_device *bdev) +{ +#ifdef CONFIG_BLOCK + return !bdev || queue_is_mq(bdev_get_queue(bdev)); +#else + return true; +#endif +} + /* * If we tracked the file through the SCM inflight mechanism, we could support * any file. For now, just ensure that anything potentially problematic is done @@ -2097,10 +2106,19 @@ static bool io_file_supports_async(struct file *file, int rw) { umode_t mode = file_inode(file)->i_mode; - if (S_ISBLK(mode) || S_ISCHR(mode) || S_ISSOCK(mode)) - return true; - if (S_ISREG(mode) && file->f_op != &io_uring_fops) + if (S_ISBLK(mode)) { + if (io_bdev_nowait(file->f_inode->i_bdev)) + return true; + return false; + } + if (S_ISCHR(mode) || S_ISSOCK(mode)) return true; + if (S_ISREG(mode)) { + if (io_bdev_nowait(file->f_inode->i_sb->s_bdev) && + file->f_op != &io_uring_fops) + return true; + return false; + } /* any ->read/write should understand O_NONBLOCK */ if (file->f_flags & O_NONBLOCK) @@ -2650,7 +2668,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(READ, req->file, &kiocb->ki_pos, iov_count); if (!ret) { - ssize_t ret2; + ssize_t ret2 = 0; if (req->file->f_op->read_iter) ret2 = call_read_iter(req->file, kiocb, &iter); @@ -2658,7 +2676,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) ret2 = loop_rw_iter(READ, req->file, kiocb, &iter); /* Catch -EAGAIN return for forced non-blocking submission */ - if (!force_nonblock || ret2 != -EAGAIN) { + if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { kiocb_done(kiocb, ret2); } else { copy_iov: From patchwork Thu Jun 18 14:43:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612377 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48DAE159A for ; Thu, 18 Jun 2020 14:46:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A01220890 for ; Thu, 18 Jun 2020 14:46:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="veVE78EJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731186AbgFROqT (ORCPT ); Thu, 18 Jun 2020 10:46:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726478AbgFROoF (ORCPT ); Thu, 18 Jun 2020 10:44:05 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86A12C061794 for ; Thu, 18 Jun 2020 07:44:05 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id h185so2895584pfg.2 for ; Thu, 18 Jun 2020 07:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=tVJM5ezTdcIMG+aKDW1osETe1l5d87INy1YG0FUdQcM=; b=veVE78EJYXVACUi1db6AgzNJIzh1hHgYOEDwJ41A29iyHkJ0tRhmtt2MLXV57P+t9T huAegqWFdBcpK4nimSirWfSC0b9sPD6lnc/YrwOmnptgxWF7i9cd+CEONozpAruie+nd wPqh4hcqGMRiDgDp72UNJ9uJRJ3Tm6xYgQAXv7jwxFhQBMol00k7MbnVC0E69rPw6FIM 7UKG7C5tnxjAntploc2dKWqJuB+isxJpInUjNRMdqtbp8QrgmRcJRGjgnIdC2OqwERSf xgxD3JLAUc+AU3m+cc8OrVO8os02InMtoRgHoCVgMKC/eW5Z+iV6e3xMpQFkEs3n9M84 vhKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=tVJM5ezTdcIMG+aKDW1osETe1l5d87INy1YG0FUdQcM=; b=DengvcEFfpnjO2LTdApox/DbyFfbqEHFNblzBlJ2yCQElOQvUavqZLLENo2+wgS20Y uQ+DMZHWhAhYV/i7byiJlnSdUIeBBBQo9RKbnjTnKWtKp2+WJZl5ZtuyEuDw5jG9WKUo anXZd5tiR0/gzbgKF3M+DGsDVKXcIG8sBNTmbBTqvRRlDLvtCklrGL7zupVG7VsVa5lz v/+/6oLBQ8kD+njBpfFze92ADVw9iNe/Ug2xsUs/ZUrDut6Tu63uzaKw0vap0gMta/v3 4EkjxkR57LQsRasnyiCvb6Jn+WjtnU3yqCPFcMDpGt7Xy2Zjl7hAHCHlMxhnVznrzzKF 9PkQ== X-Gm-Message-State: AOAM532uWNGWkLrFeg+ffjODQhezPCVTwBwNNNEVunTWU3uJGHfB+1GX 65tR7mVAcqOzz9p1EA2PgAR8Kw== X-Google-Smtp-Source: ABdhPJybozkuJOyPuS+aF/bntPw1nY8EdCmOZ0hRiMVA9zO7UZT6xbjz8eMsCaZwMXp2bvsabGcJ4A== X-Received: by 2002:aa7:9537:: with SMTP id c23mr3848900pfp.149.1592491444963; Thu, 18 Jun 2020 07:44:04 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:04 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 04/15] io_uring: re-issue block requests that failed because of resources Date: Thu, 18 Jun 2020 08:43:44 -0600 Message-Id: <20200618144355.17324-5-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Mark the plug with nowait == true, which will cause requests to avoid blocking on request allocation. If they do, we catch them and reissue them from a task_work based handler. Normally we can catch -EAGAIN directly, but the hard case is for split requests. As an example, the application issues a 512KB request. The block core will split this into 128KB if that's the max size for the device. The first request issues just fine, but we run into -EAGAIN for some latter splits for the same request. As the bio is split, we don't get to see the -EAGAIN until one of the actual reads complete, and hence we cannot handle it inline as part of submission. This does potentially cause re-reads of parts of the range, as the whole request is reissued. There's currently no better way to handle this. Signed-off-by: Jens Axboe --- fs/io_uring.c | 148 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 124 insertions(+), 24 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 2e257c5a1866..40413fb9d07b 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -900,6 +900,13 @@ static int io_file_get(struct io_submit_state *state, struct io_kiocb *req, static void __io_queue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe); +static ssize_t io_import_iovec(int rw, struct io_kiocb *req, + struct iovec **iovec, struct iov_iter *iter, + bool needs_lock); +static int io_setup_async_rw(struct io_kiocb *req, ssize_t io_size, + struct iovec *iovec, struct iovec *fast_iov, + struct iov_iter *iter); + static struct kmem_cache *req_cachep; static const struct file_operations io_uring_fops; @@ -1978,12 +1985,115 @@ static void io_complete_rw_common(struct kiocb *kiocb, long res) __io_cqring_add_event(req, res, cflags); } +static void io_sq_thread_drop_mm(struct io_ring_ctx *ctx) +{ + struct mm_struct *mm = current->mm; + + if (mm) { + kthread_unuse_mm(mm); + mmput(mm); + } +} + +static int io_sq_thread_acquire_mm(struct io_ring_ctx *ctx, + struct io_kiocb *req) +{ + if (io_op_defs[req->opcode].needs_mm && !current->mm) { + if (unlikely(!mmget_not_zero(ctx->sqo_mm))) + return -EFAULT; + kthread_use_mm(ctx->sqo_mm); + } + + return 0; +} + +#ifdef CONFIG_BLOCK +static bool io_resubmit_prep(struct io_kiocb *req, int error) +{ + struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; + ssize_t ret = -ECANCELED; + struct iov_iter iter; + int rw; + + if (error) { + ret = error; + goto end_req; + } + + switch (req->opcode) { + case IORING_OP_READV: + case IORING_OP_READ_FIXED: + case IORING_OP_READ: + rw = READ; + break; + case IORING_OP_WRITEV: + case IORING_OP_WRITE_FIXED: + case IORING_OP_WRITE: + rw = WRITE; + break; + default: + printk_once(KERN_WARNING "io_uring: bad opcode in resubmit %d\n", + req->opcode); + goto end_req; + } + + ret = io_import_iovec(rw, req, &iovec, &iter, false); + if (ret < 0) + goto end_req; + ret = io_setup_async_rw(req, ret, iovec, inline_vecs, &iter); + if (!ret) + return true; + kfree(iovec); +end_req: + io_cqring_add_event(req, ret); + req_set_fail_links(req); + io_put_req(req); + return false; +} + +static void io_rw_resubmit(struct callback_head *cb) +{ + struct io_kiocb *req = container_of(cb, struct io_kiocb, task_work); + struct io_ring_ctx *ctx = req->ctx; + int err; + + __set_current_state(TASK_RUNNING); + + err = io_sq_thread_acquire_mm(ctx, req); + + if (io_resubmit_prep(req, err)) { + refcount_inc(&req->refs); + io_queue_async_work(req); + } +} +#endif + +static bool io_rw_reissue(struct io_kiocb *req, long res) +{ +#ifdef CONFIG_BLOCK + struct task_struct *tsk; + int ret; + + if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) + return false; + + tsk = req->task; + init_task_work(&req->task_work, io_rw_resubmit); + ret = task_work_add(tsk, &req->task_work, true); + if (!ret) + return true; +#endif + return false; +} + static void io_complete_rw(struct kiocb *kiocb, long res, long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw.kiocb); - io_complete_rw_common(kiocb, res); - io_put_req(req); + if (!io_rw_reissue(req, res)) { + io_complete_rw_common(kiocb, res); + io_put_req(req); + } } static void io_complete_rw_iopoll(struct kiocb *kiocb, long res, long res2) @@ -2169,6 +2279,9 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, if (kiocb->ki_flags & IOCB_NOWAIT) req->flags |= REQ_F_NOWAIT; + if (kiocb->ki_flags & IOCB_DIRECT) + io_get_req_task(req); + if (force_nonblock) kiocb->ki_flags |= IOCB_NOWAIT; @@ -2668,6 +2781,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(READ, req->file, &kiocb->ki_pos, iov_count); if (!ret) { + unsigned long nr_segs = iter.nr_segs; ssize_t ret2 = 0; if (req->file->f_op->read_iter) @@ -2679,6 +2793,8 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { kiocb_done(kiocb, ret2); } else { + iter.count = iov_count; + iter.nr_segs = nr_segs; copy_iov: ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); @@ -2765,6 +2881,7 @@ static int io_write(struct io_kiocb *req, bool force_nonblock) iov_count = iov_iter_count(&iter); ret = rw_verify_area(WRITE, req->file, &kiocb->ki_pos, iov_count); if (!ret) { + unsigned long nr_segs = iter.nr_segs; ssize_t ret2; /* @@ -2802,6 +2919,8 @@ static int io_write(struct io_kiocb *req, bool force_nonblock) if (!force_nonblock || ret2 != -EAGAIN) { kiocb_done(kiocb, ret2); } else { + iter.count = iov_count; + iter.nr_segs = nr_segs; copy_iov: ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); @@ -4282,28 +4401,6 @@ static void io_async_queue_proc(struct file *file, struct wait_queue_head *head, __io_queue_proc(&pt->req->apoll->poll, pt, head); } -static void io_sq_thread_drop_mm(struct io_ring_ctx *ctx) -{ - struct mm_struct *mm = current->mm; - - if (mm) { - kthread_unuse_mm(mm); - mmput(mm); - } -} - -static int io_sq_thread_acquire_mm(struct io_ring_ctx *ctx, - struct io_kiocb *req) -{ - if (io_op_defs[req->opcode].needs_mm && !current->mm) { - if (unlikely(!mmget_not_zero(ctx->sqo_mm))) - return -EFAULT; - kthread_use_mm(ctx->sqo_mm); - } - - return 0; -} - static void io_async_task_func(struct callback_head *cb) { struct io_kiocb *req = container_of(cb, struct io_kiocb, task_work); @@ -5814,6 +5911,9 @@ static void io_submit_state_start(struct io_submit_state *state, unsigned int max_ios) { blk_start_plug(&state->plug); +#ifdef CONFIG_BLOCK + state->plug.nowait = true; +#endif state->free_reqs = 0; state->file = NULL; state->ios_left = max_ios; From patchwork Thu Jun 18 14:43:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612341 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A882E913 for ; Thu, 18 Jun 2020 14:44:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D33820C09 for ; Thu, 18 Jun 2020 14:44:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="TF887Y4c" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731018AbgFROo3 (ORCPT ); Thu, 18 Jun 2020 10:44:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730961AbgFROoJ (ORCPT ); Thu, 18 Jun 2020 10:44:09 -0400 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B95FDC061797 for ; Thu, 18 Jun 2020 07:44:06 -0700 (PDT) Received: by mail-pl1-x642.google.com with SMTP id bh7so2511690plb.11 for ; Thu, 18 Jun 2020 07:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=kQSBvSrh7q7DQwM3udW0mgxqiTGah8xT/7K+LzTiHfI=; b=TF887Y4cr//mvXhTI5c5VEIRfZElOkM2eJr4igs7tYzM3I8rEROoL5414jXEsJNQ1d 6jCCrNh5fQhPQ07otOWMKTLuwned62gpZzcQSnGz+AtcAsFIRf8kVJYMZrFYkSHJEstI EF+Ln2Q+pUkq3tXurup9X6YD2LWffkOldc8tPFPnSONqddrmM/DgTfaNjc16nR3q5d4t hPjBvp3o4ycBlrYr8QUDl2U7jqa+BlqD3TbE19T3/QnC95yw4ECXM/85GcpiBdM2DbqP 8OM2osLFz5ebGolhbCSR09DpxT7wJ8dhnp+kF1wWM32/wq3hn9RugfvhthZyfIx3RxdO JS+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=kQSBvSrh7q7DQwM3udW0mgxqiTGah8xT/7K+LzTiHfI=; b=jF6uE6TehWXwzo/YydT8Rd4WyG5e4t2+WU6W6Bt0MRQR+QSsQ+RhjaWliqH9+ItoAW nyhhKRmJpUQyk8gvfTLbwWM33E3ZkqF98P7qxEGV/5smkZaKvhA87vNonfqRiUa6Wzgg tLFAp++utLjHazZT4IgcDs6seXYYcQDU4gxe/HKyLpe+ML9sBUTweFMX38epoDXgSEQD iHPTE3tjXPL633l8n+NdmM38o12Cmmq77Gm1hj5oRmqB73WpRaFH4pM9u4BTcUjbqFGS AbnRGSFdHiyYk6lKigs7cr6wvTNVsz6Eal+nQjY7AGRvU3T1Bc8oKJIg5cC1uf5QTNxd exZQ== X-Gm-Message-State: AOAM533bvb8zy8kf/Lyrbm915ZuXWa+9AeSE8eLOkm5OzHk+YULmMB3Y 5oE7FB1X5pbsV5WZj8elTclWm4jhjf78Tg== X-Google-Smtp-Source: ABdhPJxFiInSxSM7iPuZuYhM5oN9cIihl9Wh6hIowfIxeUrII9w3vL4Xc8G06h5ebNdfyunElMZGlg== X-Received: by 2002:a17:902:eed1:: with SMTP id h17mr4012771plb.172.1592491446303; Thu, 18 Jun 2020 07:44:06 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:05 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 05/15] mm: allow read-ahead with IOCB_NOWAIT set Date: Thu, 18 Jun 2020 08:43:45 -0600 Message-Id: <20200618144355.17324-6-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The read-ahead shouldn't block, so allow it to be done even if IOCB_NOWAIT is set in the kiocb. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- mm/filemap.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f0ae9a6308cb..3378d4fca883 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2028,8 +2028,6 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, page = find_get_page(mapping, index); if (!page) { - if (iocb->ki_flags & IOCB_NOWAIT) - goto would_block; page_cache_sync_readahead(mapping, ra, filp, index, last_index - index); From patchwork Thu Jun 18 14:43:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612375 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 08DD3618 for ; Thu, 18 Jun 2020 14:46:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E00DF20890 for ; Thu, 18 Jun 2020 14:46:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="YeULpx9I" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730810AbgFROqO (ORCPT ); Thu, 18 Jun 2020 10:46:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730934AbgFROoJ (ORCPT ); Thu, 18 Jun 2020 10:44:09 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CE7AC0613ED for ; Thu, 18 Jun 2020 07:44:08 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id h22so2777831pjf.1 for ; Thu, 18 Jun 2020 07:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=L8y2vzO4eFciFziI46a/ttSbeGYaIdwkqPPuYZk9V/Y=; b=YeULpx9IMVYlet06hXXtF3Dyovfb6OWV7znBhPslocYv61JEKGOzA86Uf7KlkwQUxA 5veTkQm0U3FiHje/G2qBCvINtR+5H6+EjnG+6Sf5DBiwhwEgiK3ZySkgoWKeRlArsl6Z KII1sM40sgw0BBBEt/0RNUdVdIscLTYaAk11C3WKb8/EKEdqvKbfubfLIO/VBQ2Xx9WL CEYBtGHqlJYku+rfQdHayepPiSv6ME4oZxtjXE8I56CTLxFCqrzXuobRRnwbzTHobf+i uKAVIMjqM417ECasSBzSLduZSGdbDtTFAHLkWJ12coSXUTQ7pSi5y5D2Y4Jzync6tV9q xrBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=L8y2vzO4eFciFziI46a/ttSbeGYaIdwkqPPuYZk9V/Y=; b=fc+GGrNGV4AY0fTyHvhwgZKfAQgfgaFqVLQAoootBFj9NTmGfhpPrvOrDrmESu8zHo xrtD8D8gYl+SzafYbC3UJnmarqz3ozAUnBdbOQNwgJGU85+ybreM6HQu+LydbaPy9yxX s+G6JyFReFWorEkca7u3dtcmSXFb/eSIzIJTMr/9q8uvRJnZGjOP8gYy7PAvMpRnApAu f1aK3NbwH1rqNluSlr9LLoHh5ZFhmqctqoCisYT9kwbrcTjwEMXN3s+kKAW0VJs+KfcY z9lwjHZruuQABJNSf42i/B47mCFCZaJMUxUJDXaDhT6+b8kHMalGubp4t6X0NxcITtU+ wriw== X-Gm-Message-State: AOAM532I35bGIHWrQ5OXyH6FsPAetF42s2RB13fB2ugYmz+i8fjyCODe 91HjUlJeZsbtt7x3KDMHon7jtw== X-Google-Smtp-Source: ABdhPJwx9um0vmY0DnYUG9kEzc1v1xMH5bH+K7HVKhmpcv+yl+Hyx+jkxHpVYBs3czBGcr7qfQ+u8g== X-Received: by 2002:a17:902:8690:: with SMTP id g16mr4021455plo.257.1592491447688; Thu, 18 Jun 2020 07:44:07 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:06 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 06/15] mm: abstract out wake_page_match() from wake_page_function() Date: Thu, 18 Jun 2020 08:43:46 -0600 Message-Id: <20200618144355.17324-7-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org No functional changes in this patch, just in preparation for allowing more callers. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 37 +++++++++++++++++++++++++++++++++++++ mm/filemap.c | 35 ++++------------------------------- 2 files changed, 41 insertions(+), 31 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index cf2468da68e9..2f18221bb5c8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -496,6 +496,43 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, return pgoff; } +/* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ +struct wait_page_key { + struct page *page; + int bit_nr; + int page_match; +}; + +struct wait_page_queue { + struct page *page; + int bit_nr; + wait_queue_entry_t wait; +}; + +static inline int wake_page_match(struct wait_page_queue *wait_page, + struct wait_page_key *key) +{ + if (wait_page->page != key->page) + return 0; + key->page_match = 1; + + if (wait_page->bit_nr != key->bit_nr) + return 0; + + /* + * Stop walking if it's locked. + * Is this safe if put_and_wait_on_page_locked() is in use? + * Yes: the waker must hold a reference to this page, and if PG_locked + * has now already been set by another task, that task must also hold + * a reference to the *same usage* of this page; so there is no need + * to walk on to wake even the put_and_wait_on_page_locked() callers. + */ + if (test_bit(key->bit_nr, &key->page->flags)) + return -1; + + return 1; +} + extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, diff --git a/mm/filemap.c b/mm/filemap.c index 3378d4fca883..c3175dbd8fba 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -987,43 +987,16 @@ void __init pagecache_init(void) page_writeback_init(); } -/* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ -struct wait_page_key { - struct page *page; - int bit_nr; - int page_match; -}; - -struct wait_page_queue { - struct page *page; - int bit_nr; - wait_queue_entry_t wait; -}; - static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg) { struct wait_page_key *key = arg; struct wait_page_queue *wait_page = container_of(wait, struct wait_page_queue, wait); + int ret; - if (wait_page->page != key->page) - return 0; - key->page_match = 1; - - if (wait_page->bit_nr != key->bit_nr) - return 0; - - /* - * Stop walking if it's locked. - * Is this safe if put_and_wait_on_page_locked() is in use? - * Yes: the waker must hold a reference to this page, and if PG_locked - * has now already been set by another task, that task must also hold - * a reference to the *same usage* of this page; so there is no need - * to walk on to wake even the put_and_wait_on_page_locked() callers. - */ - if (test_bit(key->bit_nr, &key->page->flags)) - return -1; - + ret = wake_page_match(wait_page, key); + if (ret != 1) + return ret; return autoremove_wake_function(wait, mode, sync, key); } From patchwork Thu Jun 18 14:43:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612369 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24966159A for ; Thu, 18 Jun 2020 14:45:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0708120897 for ; Thu, 18 Jun 2020 14:45:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="nPiXsRi6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731136AbgFROpn (ORCPT ); Thu, 18 Jun 2020 10:45:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730931AbgFROoK (ORCPT ); Thu, 18 Jun 2020 10:44:10 -0400 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8547BC06179A for ; Thu, 18 Jun 2020 07:44:09 -0700 (PDT) Received: by mail-pf1-x443.google.com with SMTP id d66so2884307pfd.6 for ; Thu, 18 Jun 2020 07:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=ugefse5kKTqXQ4eIaIDMHVyZsS3OmAuYWslFQFrKGAg=; b=nPiXsRi6BRt5JXEn23LLT0VlfowWc9FFrkRNPpQKyLz9yJSlB/8/wAL82qAGNedSkF cFv9Mr/dmOS/jD0ExP7JdeFsVG0cpLkPOn+Mv46eMO76ZpXYm8G+eaVtyJaSY5qfTAid FGe99XxOvLrha9M1k68N9nt6NIVNYnuM6FbUXYkyHG4WXFzvEDR8h5yR+ORXQIzt6rOY nlAC2fElI7Ax5Ja40mz8CQ3+pneQvvbUW7ssoIRcg++V8szsXNlw7tYb2iFTM3zhNZ0R BV6S7vWTAo03tso3K4liAsMATX93QS+pb1jxyiHuCxXSSpR0IvQ9l4/kjEcYZ79DaA9F dzDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=ugefse5kKTqXQ4eIaIDMHVyZsS3OmAuYWslFQFrKGAg=; b=SChT1JbvEmSMph3IHWYd/bszA8BgfvYLyAdKSCxNhDdX+EHt7aLm2OhxQZT4FBE+ls 2t1thKftfLsobVXj5TW/lTrhHGMsxiTg8iK3WogjTYtvyoRF/f7LPNXWUuC+Y0SnNdjy xHynQURenhUv7HanCWWjP0XpXvVLiB6cG3l/mYO0klfMNiMjKi+eiNTcenYOMm5v3h+T SEkp3/X6/cgQSkdFnzDfL6BOH+gRpYHzXUnqNUMYeUlZLCOyXYIkHPgKgP6WeMzm4gAG 9QKKklCbH16veShb4+yeJrKCV9ACNpE5myPAXlY7A+hyT1HIoE7mXwn9sxkPj5aWPiuX JjBw== X-Gm-Message-State: AOAM531fNm/BGRxmQ6IdOZ2g1NKgiQl0nLatO1p7uTBqxehI0U0YrgMA 5ofVYCld5E1CsZJ/iVv6H0asyA== X-Google-Smtp-Source: ABdhPJzwVrGYYhzPh7B/cl4K86rqVhqHG34HLw3paGpNlY/2JChA4DR4YVjWC/vOkk6B5Jf1Y+/RRQ== X-Received: by 2002:a63:d858:: with SMTP id k24mr3368619pgj.288.1592491448987; Thu, 18 Jun 2020 07:44:08 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:08 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 07/15] mm: add support for async page locking Date: Thu, 18 Jun 2020 08:43:47 -0600 Message-Id: <20200618144355.17324-8-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Normally waiting for a page to become unlocked, or locking the page, requires waiting for IO to complete. Add support for lock_page_async() and wait_on_page_locked_async(), which are callback based instead. This allows a caller to get notified when a page becomes unlocked, rather than wait for it. We add a new iocb field, ki_waitq, to pass in the necessary data for this to happen. We can unionize this with ki_cookie, since that is only used for polled IO. Polled IO can never co-exist with async callbacks, as it is (by definition) polled completions. struct wait_page_key is made public, and we define struct wait_page_async as the interface between the caller and the core. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/fs.h | 7 ++++++- include/linux/pagemap.h | 17 ++++++++++++++++ mm/filemap.c | 45 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 6c4ab4dc1cd7..6ac919b40596 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -315,6 +315,8 @@ enum rw_hint { #define IOCB_SYNC (1 << 5) #define IOCB_WRITE (1 << 6) #define IOCB_NOWAIT (1 << 7) +/* iocb->ki_waitq is valid */ +#define IOCB_WAITQ (1 << 8) struct kiocb { struct file *ki_filp; @@ -328,7 +330,10 @@ struct kiocb { int ki_flags; u16 ki_hint; u16 ki_ioprio; /* See linux/ioprio.h */ - unsigned int ki_cookie; /* for ->iopoll */ + union { + unsigned int ki_cookie; /* for ->iopoll */ + struct wait_page_queue *ki_waitq; /* for async buffered IO */ + }; randomized_struct_fields_end }; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2f18221bb5c8..e053e1d9a4d7 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -535,6 +535,7 @@ static inline int wake_page_match(struct wait_page_queue *wait_page, extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); +extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); extern void unlock_page(struct page *page); @@ -571,6 +572,22 @@ static inline int lock_page_killable(struct page *page) return 0; } +/* + * lock_page_async - Lock the page, unless this would block. If the page + * is already locked, then queue a callback when the page becomes unlocked. + * This callback can then retry the operation. + * + * Returns 0 if the page is locked successfully, or -EIOCBQUEUED if the page + * was already locked and the callback defined in 'wait' was queued. + */ +static inline int lock_page_async(struct page *page, + struct wait_page_queue *wait) +{ + if (!trylock_page(page)) + return __lock_page_async(page, wait); + return 0; +} + /* * lock_page_or_retry - Lock the page, unless this would block and the * caller indicated that it can handle a retry. diff --git a/mm/filemap.c b/mm/filemap.c index c3175dbd8fba..e8aaf43bee9f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1180,6 +1180,36 @@ int wait_on_page_bit_killable(struct page *page, int bit_nr) } EXPORT_SYMBOL(wait_on_page_bit_killable); +static int __wait_on_page_locked_async(struct page *page, + struct wait_page_queue *wait, bool set) +{ + struct wait_queue_head *q = page_waitqueue(page); + int ret = 0; + + wait->page = page; + wait->bit_nr = PG_locked; + + spin_lock_irq(&q->lock); + __add_wait_queue_entry_tail(q, &wait->wait); + SetPageWaiters(page); + if (set) + ret = !trylock_page(page); + else + ret = PageLocked(page); + /* + * If we were succesful now, we know we're still on the + * waitqueue as we're still under the lock. This means it's + * safe to remove and return success, we know the callback + * isn't going to trigger. + */ + if (!ret) + __remove_wait_queue(q, &wait->wait); + else + ret = -EIOCBQUEUED; + spin_unlock_irq(&q->lock); + return ret; +} + /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked * @page: The page to wait for. @@ -1342,6 +1372,11 @@ int __lock_page_killable(struct page *__page) } EXPORT_SYMBOL_GPL(__lock_page_killable); +int __lock_page_async(struct page *page, struct wait_page_queue *wait) +{ + return __wait_on_page_locked_async(page, wait, true); +} + /* * Return values: * 1 - page is locked; mmap_lock is still held. @@ -2131,6 +2166,11 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } readpage: + if (iocb->ki_flags & IOCB_NOWAIT) { + unlock_page(page); + put_page(page); + goto would_block; + } /* * A previous I/O error may have been due to temporary * failures, eg. multipath errors. @@ -2150,7 +2190,10 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } if (!PageUptodate(page)) { - error = lock_page_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) + error = lock_page_async(page, iocb->ki_waitq); + else + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; if (!PageUptodate(page)) { From patchwork Thu Jun 18 14:43:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAFE2618 for ; Thu, 18 Jun 2020 14:45:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DC1A208B3 for ; Thu, 18 Jun 2020 14:45:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="hm30VxEN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731130AbgFROpk (ORCPT ); Thu, 18 Jun 2020 10:45:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730968AbgFROoM (ORCPT ); Thu, 18 Jun 2020 10:44:12 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13A8DC0617B9 for ; Thu, 18 Jun 2020 07:44:10 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id x207so2891175pfc.5 for ; Thu, 18 Jun 2020 07:44:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=UWm+xgAJblrksR6J1tnnSoleBzj4zdmtzBmjubOyR4o=; b=hm30VxENfZId4iIBXYzmK7NVLqG1L452OFVsVzBNxCcZsqrbma8q5o0CvvJElY7+8p Vs3+SDv61eWDJjcgEcNeNVdRAryioKuqAY0VZ5KUIBPne3UmEh/fpO8LsZt7zLrRhbVF r+xLfZ3r0wJVLTx1JAQn8WOQ4kpy0vI/zExZIzz2SQwYuNPeL12OnD2ggCtzoLOv29Ic dmRtYj8kmpp8on0fqufKHzxL4rQ9UdLZGMuQ6WCXYxQiYiLwh6uMWp6HgQzG+k87uazV SyI82aOrSqK4C/Ej6hlCmtBJKK+kxn+LVz6hW8Too2R1mZqI9Blhbsn1DdgKctf5HYDC n8lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=UWm+xgAJblrksR6J1tnnSoleBzj4zdmtzBmjubOyR4o=; b=QKQ7bSqnnqUY9MrROgXssQVwV4Ic5BvTmzqF1HQZq4oUnuj340GQmZOYCo6fT25+Jn VZB7447JieOM3bUQ6QpCqiI3uucxkAyWYS8Lxzp+QO2kzTmTeJKiMyQI/rNkRlyKJzb0 g+CA9o83n4R1cHWupSly4TyZp5npkK/H6xgfPsU8qrJoh3WSYRb9QFld15rTf6MMaQq5 cgGjCgKHkv1oibHzlUqk47Z2fZ6VHFOrWii84SJXlaKuP0N29TmRpO3PaOE/N4UjjQ9N d40mElvrdu2oV9OM88G6auoYwABAp/qbDhiTfHCxwnT4n8JpDjb1fL2d/n7sh3Hs3q0h QDTA== X-Gm-Message-State: AOAM533p7tlMAcNNPvwMOiuP6Cn/dxvL/hmy7/uNBGuP/p7cCRuve1Bw bixiuQA0KU+rECki1CWQ1fOpCw== X-Google-Smtp-Source: ABdhPJw0BKfA7CBOC1lO1/wFQlqefNyxHA+f5iR+FuKvtS2xAVdcBGbcEewx+yzDvjoPb4DPsIu+/g== X-Received: by 2002:a63:931b:: with SMTP id b27mr3478644pge.135.1592491450319; Thu, 18 Jun 2020 07:44:10 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:09 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 08/15] mm: support async buffered reads in generic_file_buffered_read() Date: Thu, 18 Jun 2020 08:43:48 -0600 Message-Id: <20200618144355.17324-9-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Use the async page locking infrastructure, if IOCB_WAITQ is set in the passed in iocb. The caller must expect an -EIOCBQUEUED return value, which means that IO is started but not done yet. This is similar to how O_DIRECT signals the same operation. Once the callback is received by the caller for IO completion, the caller must retry the operation. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- mm/filemap.c | 38 +++++++++++++++++++++++++++----------- 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index e8aaf43bee9f..a5b1fa8f7ce4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1210,6 +1210,14 @@ static int __wait_on_page_locked_async(struct page *page, return ret; } +static int wait_on_page_locked_async(struct page *page, + struct wait_page_queue *wait) +{ + if (!PageLocked(page)) + return 0; + return __wait_on_page_locked_async(compound_head(page), wait, false); +} + /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked * @page: The page to wait for. @@ -2049,17 +2057,25 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, index, last_index - index); } if (!PageUptodate(page)) { - if (iocb->ki_flags & IOCB_NOWAIT) { - put_page(page); - goto would_block; - } - /* * See comment in do_read_cache_page on why * wait_on_page_locked is used to avoid unnecessarily * serialisations and why it's safe. */ - error = wait_on_page_locked_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) { + if (written) { + put_page(page); + goto out; + } + error = wait_on_page_locked_async(page, + iocb->ki_waitq); + } else { + if (iocb->ki_flags & IOCB_NOWAIT) { + put_page(page); + goto would_block; + } + error = wait_on_page_locked_killable(page); + } if (unlikely(error)) goto readpage_error; if (PageUptodate(page)) @@ -2147,7 +2163,10 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, page_not_up_to_date: /* Get exclusive access to the page ... */ - error = lock_page_killable(page); + if (iocb->ki_flags & IOCB_WAITQ) + error = lock_page_async(page, iocb->ki_waitq); + else + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; @@ -2190,10 +2209,7 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, } if (!PageUptodate(page)) { - if (iocb->ki_flags & IOCB_WAITQ) - error = lock_page_async(page, iocb->ki_waitq); - else - error = lock_page_killable(page); + error = lock_page_killable(page); if (unlikely(error)) goto readpage_error; if (!PageUptodate(page)) { From patchwork Thu Jun 18 14:43:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612355 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79965913 for ; Thu, 18 Jun 2020 14:44:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5FE1320888 for ; Thu, 18 Jun 2020 14:44:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="e7NOJWPg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731038AbgFROof (ORCPT ); Thu, 18 Jun 2020 10:44:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730980AbgFROoQ (ORCPT ); Thu, 18 Jun 2020 10:44:16 -0400 Received: from mail-pj1-x1044.google.com (mail-pj1-x1044.google.com [IPv6:2607:f8b0:4864:20::1044]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 117BCC0617BC for ; Thu, 18 Jun 2020 07:44:12 -0700 (PDT) Received: by mail-pj1-x1044.google.com with SMTP id ga6so2648637pjb.1 for ; Thu, 18 Jun 2020 07:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=SizPTgUtV3iXJFQF9neCUoo2HH9dTA+Tsyik+Qs6Rtw=; b=e7NOJWPgpDilY8hAgu/E3UR0koeaR+7iSH91nRdwNwiag9SfhIjXt+DofFu1kfxmld sonG2ndDZmPtvlzTnmdVx6QC5xI0mKU+4Aw8UB4nmkxFWzScSGoM7yga5MuFL/2PZJnh Pi2uLHZ7iG/+7QjsmbzNkUTAyheEYTR5mB2REZBiiFdN8gvdYfMgHGneVhhGnbnz7xqE NBqAPOV/8kOoplTYc3iFVcW/Ztt8ElL2AgGcFfzG4+02vqZh/PMroO9cQupkyyW6unos 0oclD5apnWHqq9iEooOIUY0aKLpEh6E4q1f5yO/zY9SvBTrL+8900x9yRlsIPiwznjZd LoCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=SizPTgUtV3iXJFQF9neCUoo2HH9dTA+Tsyik+Qs6Rtw=; b=SgmDChjVJWHirVQgSvCBlzpE8qtone5Em3/LVOPz1voMfUyj7xUNZKkM5TrpVGuoIA 299YIjHQuZa+lm8AJW4fAPTw2kMuOS9pn6JwdgpsCl0B7Wc0CbdDr0fkVNj13QQzM1kg mI7bdL2+uhUmhSKjL6Y35yXsRiqZ7rdlrcxSpJwst0lF3F0KpBMSSEXPxc94ZbomzEd3 7oZWqeR6S340oXichq7hUG2Yznwyefm7ni4Aw7ovuXHKWsvUgPjTSJLNulhLUryfIG2f YcVxatZNjsN+xBDOFkgZEkqOyDaqx6YeSNVNCtzjx41gK0dnZezOJUFkW+hUfg/sYTuO KaMw== X-Gm-Message-State: AOAM530YqeQ9w0TtfBOcLaibJ29ZXv3if2751xc0FUqG0KZo5xkJkqwX pObowYuI4y2R/gKaaWTlCo7hyQ== X-Google-Smtp-Source: ABdhPJytALdF6/WXpXXo/EALA/GzyJrmCsd6yIDilF8kXfMl19bKdv+RXhHgzV2NYKcaqUbkTdgmNQ== X-Received: by 2002:a17:90a:b013:: with SMTP id x19mr4762711pjq.229.1592491451503; Thu, 18 Jun 2020 07:44:11 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:11 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 09/15] fs: add FMODE_BUF_RASYNC Date: Thu, 18 Jun 2020 08:43:49 -0600 Message-Id: <20200618144355.17324-10-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org If set, this indicates that the file system supports IOCB_WAITQ for buffered reads. Signed-off-by: Jens Axboe --- include/linux/fs.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 6ac919b40596..3f9de90e0266 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -175,6 +175,9 @@ typedef int (dio_iodone_t)(struct kiocb *iocb, loff_t offset, /* File does not contribute to nr_files count */ #define FMODE_NOACCOUNT ((__force fmode_t)0x20000000) +/* File supports async buffered reads */ +#define FMODE_BUF_RASYNC ((__force fmode_t)0x40000000) + /* * Flag for rw_copy_check_uvector and compat_rw_copy_check_uvector * that indicates that they should check the contents of the iovec are From patchwork Thu Jun 18 14:43:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612363 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 437AE618 for ; Thu, 18 Jun 2020 14:45:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 283C020890 for ; Thu, 18 Jun 2020 14:45:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="ynTJluJ6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731087AbgFROpO (ORCPT ); Thu, 18 Jun 2020 10:45:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730983AbgFROoQ (ORCPT ); Thu, 18 Jun 2020 10:44:16 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50A7FC0617BF for ; Thu, 18 Jun 2020 07:44:13 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id b16so2861520pfi.13 for ; Thu, 18 Jun 2020 07:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=lmn4EHxlosWk2LKyZRbh3uHZDbpZvbZJJ5zRlTFAsKo=; b=ynTJluJ6Q99yEvMTa6y10W+u5+Z89pdb0A+GuJkEPIQuhyxpm60zb8aV1UgTzM49Cg 7mYXB2z8mtKRwGfC7oa8zmXN0NeqZcMk0nfEZAsWCQC7qS69O7EfAy7bvV4ntFSDUnKd s1HYlH3UUvMSO05JqooeCBbZY0ZRIjrWWqeXC7QjivnX5Jt0UXn8Nku5Kkav004uevwG 8PUXkLfmKjS6G8DCuicClsmZmNXcIm7PbrbS7/OqZObQQHmkmS+ttktTCCcqchSw4mb3 huoY15kQX2Ms3nHgl6XAnHvSp6STkjvZO6hPs2x+GSzE549wVTux9mlLhmmuwEge7kyC Whmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=lmn4EHxlosWk2LKyZRbh3uHZDbpZvbZJJ5zRlTFAsKo=; b=Tm8NFa79typLIzDuhAaHsLft9kBQisy8lJ0kwaqXzDQ5bRBW5n0hwNPX2Zp2bBnBqN 5uLRcPE5/jwRbSLizOrJBEALOIEWwAofmAvzDzPCuMLiD+GJSdAOl/XsZDQkj/vobvL0 CpDnGnjmsiWSgGWJj9O83XtC3JCQoc+Kjc6tr/Pit6PPNut+M0PYR8srAC/VqUUqpcn1 mA8po7o2ozyqYlpVqNTPFzo1kWezxrmDwNd4ts6RLwWPJ+ld5DanNIm19dJ9UbSnSOGU 35qojmEBBKeHR2qug6gLEzIwcVS5e/jeFISn3FNqTSkY89xKn2XsZ40vSocV4iOV/xS7 PTQw== X-Gm-Message-State: AOAM533NOTfIUB2xK04+ufYgEofoAyBvsT/tAx35eztM4AG+TeNo1n/X hHH1p7YJIBXEr2otnGinCGpe78xljZNB6g== X-Google-Smtp-Source: ABdhPJzBNjEb2EomZLOu+f8/v3tw76mKzCcpM8kJeYEd8Lb2xxqO66ZxbvFvynHQ19qXPAAGDtGMUA== X-Received: by 2002:a63:1e60:: with SMTP id p32mr3518283pgm.172.1592491452832; Thu, 18 Jun 2020 07:44:12 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:12 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 10/15] block: flag block devices as supporting IOCB_WAITQ Date: Thu, 18 Jun 2020 08:43:50 -0600 Message-Id: <20200618144355.17324-11-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Signed-off-by: Jens Axboe --- fs/block_dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 47860e589388..54720c90dad0 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1850,7 +1850,7 @@ static int blkdev_open(struct inode * inode, struct file * filp) */ filp->f_flags |= O_LARGEFILE; - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; if (filp->f_flags & O_NDELAY) filp->f_mode |= FMODE_NDELAY; From patchwork Thu Jun 18 14:43:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612349 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DB81159A for ; Thu, 18 Jun 2020 14:44:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8362F208D5 for ; Thu, 18 Jun 2020 14:44:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="HvZc/ixP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730978AbgFROog (ORCPT ); Thu, 18 Jun 2020 10:44:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730977AbgFROoP (ORCPT ); Thu, 18 Jun 2020 10:44:15 -0400 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 893DEC061794 for ; Thu, 18 Jun 2020 07:44:14 -0700 (PDT) Received: by mail-pg1-x544.google.com with SMTP id e18so3002614pgn.7 for ; Thu, 18 Jun 2020 07:44:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=KEZHLow75x1WGZauUbfw6ciLmnTzXmpQXtEtlinmygw=; b=HvZc/ixPovm/kRhvl4SJUiwM/PVM+gQUGpw4gn7GL1Ek48GAAcFBxFJBO4YBWLb0Y9 Oiqk0i8f6vQ1Em168554rqM+AcEyk7rObxLac2g8tH1Pd6I83C8/2mPcrM6EC5HB+yuI ZQq+xXWBQ17rDR3i6if4Yr3L8h/fWDRW9Sjj870yRZqr2dNo7/szXK+hje9neRqzDGj1 k+P4aQd8v6FJFSOymCFraV22X6dCtjZ03CNMccoXsgYKxxTrdgIEeGkBWmuwX9eMXQVf 6oKU1HJcojpsQ/gOVihItBajLbiifqA4r48bx/wgZ3Vs2f3Le4J1aEEGs7KW+RdevsH2 /nIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=KEZHLow75x1WGZauUbfw6ciLmnTzXmpQXtEtlinmygw=; b=VWT/S7BTn9KSHk56HzbAu9JmQvlAUY1YQKfH2BcddmtdMsB6iw5faxLj8b9MEQyA71 UfAwkJXttQoKRHX9gU4Hbk1W/NNS6ZMBEj2Q8+7iEhbxA6NK0xrFBvB9NCc/an6zAKMG m+ox9vO2tUqWNWIT7gXZtYsaF4iRbLlYw8el1ZjXtVdxxcM7xWGhr7s2n1tzSkuK3YkB 2ZL+EHsqyG/8tMuV4U4I6qVbo/qYjRM8Esd/X3SA3yxLz5RpDrkp/+zR5C4rJt8jcyZN CDcNivs/yzVh17Lc/LLiOKgN8b1hf3uDGytjmdaWAzvKXMvHD/c93oYnUSAoBJQj0pJh ARsA== X-Gm-Message-State: AOAM533o+noAfYItvU4J19N+LNqOBrul1J3zgsAJpqB+MCjbM7QqgzID tkRQazLRasO+b8MTXU9ya0e8/g== X-Google-Smtp-Source: ABdhPJybJIEFYk2ErRp3fvA+q2tyQLM9BD6LGpYjoii03ztTUTqqX4O2JwO+0Ij8wpxOd+bSneCDiQ== X-Received: by 2002:a63:205b:: with SMTP id r27mr3555018pgm.326.1592491454092; Thu, 18 Jun 2020 07:44:14 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:13 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , "Darrick J . Wong" Subject: [PATCH 11/15] xfs: flag files as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:51 -0600 Message-Id: <20200618144355.17324-12-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org XFS uses generic_file_read_iter(), which already supports this. Acked-by: Darrick J. Wong Signed-off-by: Jens Axboe --- fs/xfs/xfs_file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 00db81eac80d..fdbff4860d61 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1080,7 +1080,7 @@ xfs_file_open( return -EFBIG; if (XFS_FORCED_SHUTDOWN(XFS_M(inode->i_sb))) return -EIO; - file->f_mode |= FMODE_NOWAIT; + file->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return 0; } From patchwork Thu Jun 18 14:43:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9034618 for ; Thu, 18 Jun 2020 14:45:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AEFDD20890 for ; Thu, 18 Jun 2020 14:45:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="JsQWJAr0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731083AbgFROpM (ORCPT ); Thu, 18 Jun 2020 10:45:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731000AbgFROoW (ORCPT ); Thu, 18 Jun 2020 10:44:22 -0400 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBD60C0613EE for ; Thu, 18 Jun 2020 07:44:16 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id a127so2866910pfa.12 for ; Thu, 18 Jun 2020 07:44:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=qn8Ub7OYxVXKid0SneeEv6VlK8roF4pAhw3PBRQGk58=; b=JsQWJAr0eX5i4/3VH0JUUcbrX1hvyikPQulQivfaSDiMps3VzdLvSQq6dC4JKqWrJw KNxWMk6jyBDtv6zAbNsi8EXWu/yzVZs9PhGYrbDhl/fbSE8G4wRID9uPFVW7kqxk71SV 0TMZ2olNJApcg9LZpxFyKjUmCe4HMfr+UszmCR8CNyND2jnymuGfLJggvQZrlKsGiP5H olUIpqHBAvyxiNTFfvQ13/71tmHFNQyLLqdFLI0ERuAdJTxyxlimD9W8/4OLmmmYJWbC NDlNdD05mAC1WxkMhISVU5ejMHN0Tp6Wbi9rsSKqKY3u8ZOVaCzS4KBi9ffc4A0l9I/a Yiuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=qn8Ub7OYxVXKid0SneeEv6VlK8roF4pAhw3PBRQGk58=; b=FQnxOi7tspte+t5Z5BCTSOo7P12I+/LJjDMvHFqZtxt/VOFTIXUvVuoJXenCD9TmFq iBuBToRSrJvMmOkSaNcWccopIRn8eqxap+zc99YBsvYrbHJrvzY+JCDYou1b7dM/Owp/ nTePk/09BbX76UccvKRz+c2NT/COPZF8j0feL4o4FJDdfUICuAekMbqXnXNASQUwpQrt NnkBPB0OpBN4a8NWBIohWRvDlS5SZ0SgYmlyDCZh6oflUYAqJY5IHRF2lazQeEx+7ADg OoOIBiy/xR+oORKBgjaAcljD67lpufkt6Wc8GOC94xAarIqHP/pKiYOYeyZbFEtw3LQw yZZg== X-Gm-Message-State: AOAM533fgx3OJXu4jNRYxfS1ca2QPVSxZS+kiLyGEGwVjAG7CPL5vDfE co5Dcjb36HhIhDKoNheCOM6NBw== X-Google-Smtp-Source: ABdhPJxqRDix9IaoTIWmuLrGVEzpKy6hFma+ZgZz9Se6n9okXrRAwv8gfsXZq8bmQZ3+Gotx4PSctg== X-Received: by 2002:a65:6119:: with SMTP id z25mr3404328pgu.52.1592491456371; Thu, 18 Jun 2020 07:44:16 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:14 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Chris Mason Subject: [PATCH 12/15] btrfs: flag files as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:52 -0600 Message-Id: <20200618144355.17324-13-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org btrfs uses generic_file_read_iter(), which already supports this. Acked-by: Chris Mason Signed-off-by: Jens Axboe --- fs/btrfs/file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 2c14312b05e8..234a418eb6da 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -3472,7 +3472,7 @@ static loff_t btrfs_file_llseek(struct file *file, loff_t offset, int whence) static int btrfs_file_open(struct inode *inode, struct file *filp) { - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return generic_file_open(inode, filp); } From patchwork Thu Jun 18 14:43:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09F14618 for ; Thu, 18 Jun 2020 14:45:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF89C20890 for ; Thu, 18 Jun 2020 14:45:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="CmsR4pnk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731061AbgFROpL (ORCPT ); Thu, 18 Jun 2020 10:45:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731005AbgFROoY (ORCPT ); Thu, 18 Jun 2020 10:44:24 -0400 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED9FCC061796 for ; Thu, 18 Jun 2020 07:44:17 -0700 (PDT) Received: by mail-pg1-x541.google.com with SMTP id e9so2991133pgo.9 for ; Thu, 18 Jun 2020 07:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=agiR+Byeck2s775I8LSKf8tm0BQVQsWoO+u3QsmIwcE=; b=CmsR4pnk1FMjkbrTWikiw1ru36jrxC1tfShgKpQNL5fwTTR48LJzuB0WFeslN0A8R7 yVhdczhT7rCHZGNSPImdCvg2GFHYjPZq3Lo8LN55YdVfIqbcMnyIHIfCWRTp7nIlWoCP 8mIA0QOEOodS5hrggItuRz50XlDx1FRg2U51ti0K2K6RLhv04afTVM85LjEQHy1K3put aih63C1nllRb0Y93OtUHEJJE5MygBtOKi6TQwE2EGgx2382rGAnQZJ70Af+bxcWW78if p90wfGKQkPEoYPs0wXfDt35iVSEZ+5uCYlZohmz4//abZsAjr1a8x2PskDZN8j4J5CQD CYxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=agiR+Byeck2s775I8LSKf8tm0BQVQsWoO+u3QsmIwcE=; b=tnljA0G5yysSotr5D7W3Kz3LElR+5y7zVr6a3D8R5JXx94ldyuRRdb6weVWsJv3l0l eLa6Jm5JnTGHg7IWy+uPwH4Uxf4BKyCFb0GXd+nzBUdrx/8M8nmrxABPTVyiK/Rv2euq sulCLLhdJVP4ff6DKQLZvwL21QT/5/dTanHH12EvmPwzU8yL4MMXydnyymkC9wCQcQ6B Y4dcw/C1A3sReZqqJp/izWU0qRxTOrycnAJNrwZ85Zs6VDJWzw5abiRGF2a9azn1/cv+ 0FAbHghvAKAB9cT8q61yYqS2OwBxQ7RRPFGTvxgUf4tF8+qWbvbPGi2lzo4qtS3BVvJC 6I/w== X-Gm-Message-State: AOAM531BNOlj+Vd9JxDj8cxLkrNRdSGpdY4D/XpFAz15jhRQxcpQ4EaC ioPmp+eLfOnfIlsGQ7sNLeLUXQ== X-Google-Smtp-Source: ABdhPJyNr3+kHnVN8fM6oIP5PSOO4gsldaD6y/PdYSuuK7cpSwrVAwO5ubxT1Qc0fzuBScSDRUwL4w== X-Received: by 2002:a65:6558:: with SMTP id a24mr1850398pgw.110.1592491457531; Thu, 18 Jun 2020 07:44:17 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:17 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Theodore Ts'o Subject: [PATCH 13/15] ext4: flag as supporting buffered async reads Date: Thu, 18 Jun 2020 08:43:53 -0600 Message-Id: <20200618144355.17324-14-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ext4 uses generic_file_read_iter(), which already supports this. Cc: Theodore Ts'o Signed-off-by: Jens Axboe --- fs/ext4/file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 2a01e31a032c..1e827410e9e1 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -839,7 +839,7 @@ static int ext4_file_open(struct inode * inode, struct file * filp) return ret; } - filp->f_mode |= FMODE_NOWAIT; + filp->f_mode |= FMODE_NOWAIT | FMODE_BUF_RASYNC; return dquot_file_open(inode, filp); } From patchwork Thu Jun 18 14:43:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612359 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27499618 for ; Thu, 18 Jun 2020 14:45:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0E50D20897 for ; Thu, 18 Jun 2020 14:45:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="bUiZ2Fxa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730982AbgFROpH (ORCPT ); Thu, 18 Jun 2020 10:45:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731016AbgFROo2 (ORCPT ); Thu, 18 Jun 2020 10:44:28 -0400 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FE68C06125F for ; Thu, 18 Jun 2020 07:44:19 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id x207so2891379pfc.5 for ; Thu, 18 Jun 2020 07:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=qG+PVAwYXEH3Rowu54+BkyZS7fslCJIulLcAGKfzRB8=; b=bUiZ2FxaCsZtyTAaJBoFcJMrX/CUga9qC0v2B0H5JcW44TVjLNQskpGK86a5Zb631M p0I2rj2UBPJFfNusO9HGLjnYw/iM43LqQ928bwfe2TjZbjF/4Hg7nfa5uPlMNvTn3vO1 apcy+p2q8hky9Ls8gdUEdvgAwIA3LtH/8apm463k0TATUTGX0gwMdBhkDB52mw3XPrHh GrIGx+AghUZ13e4QtMHcv9dXOv4lssAk0Hv512Bg1on9er9ClFRT03k4kQjM4RRxOSPW 6hvmoIIT3WZGyFLsTEshOvym+xTAnpq2veuMMOcqepVgDlq37cK5gOQh2kGyUaORNsIr A/uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=qG+PVAwYXEH3Rowu54+BkyZS7fslCJIulLcAGKfzRB8=; b=IhpSvheuqxvMVbQc4YPoJmXVWjcr9Mir3R7PQVWgkly+jwV3tYl23eaBvL090sRu/O CKMBbsXeVZGuOya0piFw8Hl59r7xuw+728tby9+3RMXUKR/pk00O9OZA+luqOsJLDF74 kJ+z32ih7B6JzJ5Gs6POtKUa5IT223osrVVOs4/f8N3GJ4kzx+BUERz116lZ6aLr4DGF u5l9Bv8Jv7G7c1kn9wC98ITQynKOHb/dMJdEKFhcvFRgF+8ZD5x6j5TkgXNS1P1r70rO lcuozq/DrycrlEJG4wEn5reIZiLa4KiPH++irLWXlQKnybWQCnHyThfBZ3smLV5UP5yh M1ng== X-Gm-Message-State: AOAM532p8wp37uMFG2Ve4VUZ4ac5xUbVVY4EYgYanwEMxMB2Hdh6J1G2 QQf8cu9EofEhuIq62gNI5Z2zYA== X-Google-Smtp-Source: ABdhPJzzGfgUvN38nQlE0ltPql9r24L3d5fAFgRA+DRt5giXUwPYRi3iuK4YpxPwlcYuqcOquHT1Ag== X-Received: by 2002:a62:7e95:: with SMTP id z143mr3898104pfc.108.1592491458734; Thu, 18 Jun 2020 07:44:18 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:18 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe , Johannes Weiner Subject: [PATCH 14/15] mm: add kiocb_wait_page_queue_init() helper Date: Thu, 18 Jun 2020 08:43:54 -0600 Message-Id: <20200618144355.17324-15-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Checks if the file supports it, and initializes the values that we need. Caller passes in 'data' pointer, if any, and the callback function to be used. Acked-by: Johannes Weiner Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e053e1d9a4d7..7386bc67cc5a 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -533,6 +533,27 @@ static inline int wake_page_match(struct wait_page_queue *wait_page, return 1; } +static inline int kiocb_wait_page_queue_init(struct kiocb *kiocb, + struct wait_page_queue *wait, + wait_queue_func_t func, + void *data) +{ + /* Can't support async wakeup with polled IO */ + if (kiocb->ki_flags & IOCB_HIPRI) + return -EINVAL; + if (kiocb->ki_filp->f_mode & FMODE_BUF_RASYNC) { + wait->wait.func = func; + wait->wait.private = data; + wait->wait.flags = 0; + INIT_LIST_HEAD(&wait->wait.entry); + kiocb->ki_flags |= IOCB_WAITQ; + kiocb->ki_waitq = wait; + return 0; + } + + return -EOPNOTSUPP; +} + extern void __lock_page(struct page *page); extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); From patchwork Thu Jun 18 14:43:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11612357 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65C4314E3 for ; Thu, 18 Jun 2020 14:44:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4773320890 for ; Thu, 18 Jun 2020 14:44:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel-dk.20150623.gappssmtp.com header.i=@kernel-dk.20150623.gappssmtp.com header.b="RfkCMfGN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731022AbgFROow (ORCPT ); Thu, 18 Jun 2020 10:44:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731026AbgFROob (ORCPT ); Thu, 18 Jun 2020 10:44:31 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7357CC061798 for ; Thu, 18 Jun 2020 07:44:20 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id y17so2516635plb.8 for ; Thu, 18 Jun 2020 07:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=hItvrdMXW3x4/4yokWms4yIKM/BxkkcMkxbyn/WrUe0=; b=RfkCMfGNHVXXALcOZPZ2jdMi+X2j7BbBbajuLvtbMQgxMG/R/SN9Lmnx9NaM1whfmk FoFat2Ffm7ozF2Am9lqvkz0jsEoF+28OalJ8pxWgXrDzHjm9PzijBHVdQeoN953xZdDY tf/dvYtXZA04B8ODzL5vUKx87iaBZufPMtd6kedtozYO9BMoAcWeY9LWTHO0Ida6RpP3 x1eoT/nTeXvRj4v/5yKnIft1RpIV2avzGEzDdkbF3szc11YLa/Lb2tqMcnSEDn0GFTjr Fg3FMcJwXY/ZGtqbb19AHc5aPFxfUxJG8rl9k5i052NZMKop0iiA8EZgxFfSHlvxzSqN C9BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=hItvrdMXW3x4/4yokWms4yIKM/BxkkcMkxbyn/WrUe0=; b=j6OzmbffXfPVccJZRrAeYUlVItvMLRXWICF8xv4oF+KGVMdW0tTb9/r25i4jl46/UZ vYOBSXeWLfYPSETFDibGNDzDk7vO0uxT7lyCPNyVFO9m+idUn0KC3VeIGj49+/O7v+LF pJOef9felwyxN4Eo5OdRIBbpfgc/nL4g1HWUEI6MLJo5vRP6R1C5OqnpXQMwrzCKslm5 thIZKsgvpXxEgssxJsMmXVNYd3YjxCPb2nSWV54a4Qe22rKmMtFXdgawi+PyfHf8rSCg IQNVNJGgsyVOOwFKWDwLWneeHreU6iLBHkBi6RoGqa8wyrZy8E+WviNJG4xOVmpbZAXT Vojw== X-Gm-Message-State: AOAM533puI8NInLKm25ozOaEJamcyYXqI6NJLJ8XOvpRcWqKAWs0nrws TT5MbvXHyFtYZttvZLhiDg5HWZPqd1YYOQ== X-Google-Smtp-Source: ABdhPJxS5EFTVcmIXFEiggPIpu8uYctGJadye2v8tGqmJWPLNu+hH9ccfCYUZJj+BdxVYR30oM/QCw== X-Received: by 2002:a17:902:7c8f:: with SMTP id y15mr3899856pll.95.1592491459962; Thu, 18 Jun 2020 07:44:19 -0700 (PDT) Received: from x1.localdomain ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id g9sm3127197pfm.151.2020.06.18.07.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 07:44:19 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, Jens Axboe Subject: [PATCH 15/15] io_uring: support true async buffered reads, if file provides it Date: Thu, 18 Jun 2020 08:43:55 -0600 Message-Id: <20200618144355.17324-16-axboe@kernel.dk> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618144355.17324-1-axboe@kernel.dk> References: <20200618144355.17324-1-axboe@kernel.dk> Reply-To: "[PATCHSET v7 0/15]"@vger.kernel.org, Add@vger.kernel.org, support@vger.kernel.org, for@vger.kernel.org, async@vger.kernel.org, buffered@vger.kernel.org, reads@vger.kernel.org MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org If the file is flagged with FMODE_BUF_RASYNC, then we don't have to punt the buffered read to an io-wq worker. Instead we can rely on page unlocking callbacks to support retry based async IO. This is a lot more efficient than doing async thread offload. The retry is done similarly to how we handle poll based retry. From the unlock callback, we simply queue the retry to a task_work based handler. Signed-off-by: Jens Axboe --- fs/io_uring.c | 145 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 137 insertions(+), 8 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 40413fb9d07b..94282be1c413 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -78,6 +78,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -503,6 +504,8 @@ struct io_async_rw { struct iovec *iov; ssize_t nr_segs; ssize_t size; + struct wait_page_queue wpq; + struct callback_head task_work; }; struct io_async_ctx { @@ -2750,6 +2753,126 @@ static int io_read_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe, return 0; } +static void __io_async_buf_error(struct io_kiocb *req, int error) +{ + struct io_ring_ctx *ctx = req->ctx; + + spin_lock_irq(&ctx->completion_lock); + io_cqring_fill_event(req, error); + io_commit_cqring(ctx); + spin_unlock_irq(&ctx->completion_lock); + + io_cqring_ev_posted(ctx); + req_set_fail_links(req); + io_double_put_req(req); +} + +static void io_async_buf_cancel(struct callback_head *cb) +{ + struct io_async_rw *rw; + struct io_kiocb *req; + + rw = container_of(cb, struct io_async_rw, task_work); + req = rw->wpq.wait.private; + __io_async_buf_error(req, -ECANCELED); +} + +static void io_async_buf_retry(struct callback_head *cb) +{ + struct io_async_rw *rw; + struct io_ring_ctx *ctx; + struct io_kiocb *req; + + rw = container_of(cb, struct io_async_rw, task_work); + req = rw->wpq.wait.private; + ctx = req->ctx; + + __set_current_state(TASK_RUNNING); + if (!io_sq_thread_acquire_mm(ctx, req)) { + mutex_lock(&ctx->uring_lock); + __io_queue_sqe(req, NULL); + mutex_unlock(&ctx->uring_lock); + } else { + __io_async_buf_error(req, -EFAULT); + } +} + +static int io_async_buf_func(struct wait_queue_entry *wait, unsigned mode, + int sync, void *arg) +{ + struct wait_page_queue *wpq; + struct io_kiocb *req = wait->private; + struct io_async_rw *rw = &req->io->rw; + struct wait_page_key *key = arg; + struct task_struct *tsk; + int ret; + + wpq = container_of(wait, struct wait_page_queue, wait); + + ret = wake_page_match(wpq, key); + if (ret != 1) + return ret; + + list_del_init(&wait->entry); + + init_task_work(&rw->task_work, io_async_buf_retry); + /* submit ref gets dropped, acquire a new one */ + refcount_inc(&req->refs); + tsk = req->task; + ret = task_work_add(tsk, &rw->task_work, true); + if (unlikely(ret)) { + /* queue just for cancelation */ + init_task_work(&rw->task_work, io_async_buf_cancel); + tsk = io_wq_get_task(req->ctx->io_wq); + task_work_add(tsk, &rw->task_work, true); + } + wake_up_process(tsk); + return 1; +} + +static bool io_rw_should_retry(struct io_kiocb *req) +{ + struct kiocb *kiocb = &req->rw.kiocb; + int ret; + + /* never retry for NOWAIT, we just complete with -EAGAIN */ + if (req->flags & REQ_F_NOWAIT) + return false; + + /* already tried, or we're doing O_DIRECT */ + if (kiocb->ki_flags & (IOCB_DIRECT | IOCB_WAITQ)) + return false; + /* + * just use poll if we can, and don't attempt if the fs doesn't + * support callback based unlocks + */ + if (file_can_poll(req->file) || !(req->file->f_mode & FMODE_BUF_RASYNC)) + return false; + + /* + * If request type doesn't require req->io to defer in general, + * we need to allocate it here + */ + if (!req->io && __io_alloc_async_ctx(req)) + return false; + + ret = kiocb_wait_page_queue_init(kiocb, &req->io->rw.wpq, + io_async_buf_func, req); + if (!ret) { + io_get_req_task(req); + return true; + } + + return false; +} + +static int io_iter_do_read(struct io_kiocb *req, struct iov_iter *iter) +{ + if (req->file->f_op->read_iter) + return call_read_iter(req->file, &req->rw.kiocb, iter); + return loop_rw_iter(READ, req->file, &req->rw.kiocb, iter); +} + static int io_read(struct io_kiocb *req, bool force_nonblock) { struct iovec inline_vecs[UIO_FASTIOV], *iovec = inline_vecs; @@ -2784,10 +2907,7 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) unsigned long nr_segs = iter.nr_segs; ssize_t ret2 = 0; - if (req->file->f_op->read_iter) - ret2 = call_read_iter(req->file, kiocb, &iter); - else - ret2 = loop_rw_iter(READ, req->file, kiocb, &iter); + ret2 = io_iter_do_read(req, &iter); /* Catch -EAGAIN return for forced non-blocking submission */ if (!force_nonblock || (ret2 != -EAGAIN && ret2 != -EIO)) { @@ -2799,17 +2919,26 @@ static int io_read(struct io_kiocb *req, bool force_nonblock) ret = io_setup_async_rw(req, io_size, iovec, inline_vecs, &iter); if (ret) - goto out_free; + goto out; /* any defer here is final, must blocking retry */ if (!(req->flags & REQ_F_NOWAIT) && !file_can_poll(req->file)) req->flags |= REQ_F_MUST_PUNT; + /* if we can retry, do so with the callbacks armed */ + if (io_rw_should_retry(req)) { + ret2 = io_iter_do_read(req, &iter); + if (ret2 == -EIOCBQUEUED) { + goto out; + } else if (ret2 != -EAGAIN) { + kiocb_done(kiocb, ret2); + goto out; + } + } + kiocb->ki_flags &= ~IOCB_WAITQ; return -EAGAIN; } } -out_free: - kfree(iovec); - req->flags &= ~REQ_F_NEED_CLEANUP; +out: return ret; }