From patchwork Mon Nov 19 03:51:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688177 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B22ED109C for ; Mon, 19 Nov 2018 03:51:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A06E0286CA for ; Mon, 19 Nov 2018 03:51:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 94AA429AC9; Mon, 19 Nov 2018 03:51:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54236286CA for ; Mon, 19 Nov 2018 03:51:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728141AbeKSONx (ORCPT ); Mon, 19 Nov 2018 09:13:53 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62727 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726903AbeKSONx (ORCPT ); Mon, 19 Nov 2018 09:13:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599497; x=1574135497; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MdGOl0NUx0ZiANgGuUGhQRcuAwTeVGsY7Kq8hiZZ72A=; b=Zc2FFtxlW3xunGA7P1Mu5s2ueEn9X5Vs6tA7Rxka6OsVxP6LvmtmS5yS VbMNGQaW+xbaq74a/LgC932AA23bBYTAx1QdCFch1j9xljUhxTEd7IYOI Ap3/8zfLGwBUt/AzwnQhd/tHVfk0CA5tz5BKtts1hkU+AcsGV9INUbmQL bpnF30wiQku32wsZ97Lqz3QGPOHHN3OG5OGUu2uJbuoI6rfqQDeMl2Tp7 fYdf6rMLvrG1K9wm2opDdgs6Vmicd6dWzrA9liL+Mv/D8DIkiufs83twj iGeOqpPCIXbSzLN3ip5VBozRHAwIhNfekjYOrpSgORwBWScY7ETmc3w7n A==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272447" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:37 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:36 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:36 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 1/7] aio: Comment use of IOCB_FLAG_IOPRIO aio flag Date: Mon, 19 Nov 2018 12:51:25 +0900 Message-Id: <20181119035131.11255-2-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Comment the use of the IOCB_FLAG_IOPRIO aio flag similarly to the IOCB_FLAG_RESFD flag. Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn --- include/uapi/linux/aio_abi.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/uapi/linux/aio_abi.h b/include/uapi/linux/aio_abi.h index ce43d340f010..8387e0af0f76 100644 --- a/include/uapi/linux/aio_abi.h +++ b/include/uapi/linux/aio_abi.h @@ -50,6 +50,8 @@ enum { * * IOCB_FLAG_RESFD - Set if the "aio_resfd" member of the "struct iocb" * is valid. + * IOCB_FLAG_IOPRIO - Set if the "aio_reqprio" member of the "struct iocb" + * is valid. */ #define IOCB_FLAG_RESFD (1 << 0) #define IOCB_FLAG_IOPRIO (1 << 1) From patchwork Mon Nov 19 03:51:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688193 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4857D13B5 for ; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31CF3286CA for ; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 260D529AC4; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2D7329ACA for ; Mon, 19 Nov 2018 03:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728243AbeKSOOQ (ORCPT ); Mon, 19 Nov 2018 09:14:16 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62740 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728192AbeKSOOP (ORCPT ); Mon, 19 Nov 2018 09:14:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599519; x=1574135519; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2bteIgvU+Xj6dWadF96GiJ3sWnt5ZWQMBkBQRMZuUng=; b=Fe4ng7vClIJcZMps+TkB1SSK5DhO0JR7RPrpOf2qUgSURPXdXvaVEhW0 PZUtd468Lk5S/R8B/C18MR2genHjKrUqxVDIYAAI/y+0c5SvYDczULx1J 1a9fpJBhvQiQLwbI03cKJhggVO8Ml1X2Jd3Mxif55kJ21VsONH/vj/TLQ M2J7UpflNoB2qFNdXzdZqesG0wxRUM36CHcKImXubyyAMutWRDibztAMq lu+J13U3YuX6DdxWxHkr5vnr7OgCUHN4SrwfshqHsoPxZD3HrGGdLVZdA HtaAQQwKDZ+ay8ZkF4iaFZHwLiUBInd98zk95muQ8GtOOjwuv0dHhW4TG g==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272449" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:40 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:38 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:38 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 2/7] block: Remove bio->bi_ioc Date: Mon, 19 Nov 2018 12:51:26 +0900 Message-Id: <20181119035131.11255-3-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP bio->bi_ioc is never set so always NULL. Remove references to it in bio_disassociate_task() and in rq_ioc() and delete this field from struct bio. With this change, rq_ioc() always returns current->io_context without the need for a bio argument. Further simplify the code and make it more readable by also removing this helper, which also allows to simplify blk_mq_sched_assign_ioc() by removing its bio argument. Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn Reviewed-by: Adam Manzanares Tested-by: Ming Lei --- block/bio.c | 4 ---- block/blk-core.c | 2 +- block/blk-mq-sched.c | 4 ++-- block/blk-mq-sched.h | 2 +- block/blk-mq.c | 4 ++-- block/blk.h | 16 ---------------- include/linux/blk_types.h | 3 +-- 7 files changed, 7 insertions(+), 28 deletions(-) diff --git a/block/bio.c b/block/bio.c index 4f4d9884443b..03895cc0d74a 100644 --- a/block/bio.c +++ b/block/bio.c @@ -2027,10 +2027,6 @@ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) */ void bio_disassociate_task(struct bio *bio) { - if (bio->bi_ioc) { - put_io_context(bio->bi_ioc); - bio->bi_ioc = NULL; - } if (bio->bi_css) { css_put(bio->bi_css); bio->bi_css = NULL; diff --git a/block/blk-core.c b/block/blk-core.c index d6e8ab9ca99d..492648c96992 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -813,7 +813,7 @@ unsigned int blk_plug_queued_count(struct request_queue *q) void blk_init_request_from_bio(struct request *req, struct bio *bio) { - struct io_context *ioc = rq_ioc(bio); + struct io_context *ioc = current->io_context; if (bio->bi_opf & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index d084f731d104..13b8dc332541 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -31,10 +31,10 @@ void blk_mq_sched_free_hctx_data(struct request_queue *q, } EXPORT_SYMBOL_GPL(blk_mq_sched_free_hctx_data); -void blk_mq_sched_assign_ioc(struct request *rq, struct bio *bio) +void blk_mq_sched_assign_ioc(struct request *rq) { struct request_queue *q = rq->q; - struct io_context *ioc = rq_ioc(bio); + struct io_context *ioc = current->io_context; struct io_cq *icq; spin_lock_irq(&q->queue_lock); diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 7ff5671bf128..0f719c8532ae 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -8,7 +8,7 @@ void blk_mq_sched_free_hctx_data(struct request_queue *q, void (*exit)(struct blk_mq_hw_ctx *)); -void blk_mq_sched_assign_ioc(struct request *rq, struct bio *bio); +void blk_mq_sched_assign_ioc(struct request *rq); void blk_mq_sched_request_inserted(struct request *rq); bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, diff --git a/block/blk-mq.c b/block/blk-mq.c index 32b246ed44c0..636f80b96fa6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -389,8 +389,8 @@ static struct request *blk_mq_get_request(struct request_queue *q, if (!op_is_flush(data->cmd_flags)) { rq->elv.icq = NULL; if (e && e->type->ops.prepare_request) { - if (e->type->icq_cache && rq_ioc(bio)) - blk_mq_sched_assign_ioc(rq, bio); + if (e->type->icq_cache) + blk_mq_sched_assign_ioc(rq); e->type->ops.prepare_request(rq, bio); rq->rq_flags |= RQF_ELVPRIV; diff --git a/block/blk.h b/block/blk.h index 816a9abb87cd..610948157a5b 100644 --- a/block/blk.h +++ b/block/blk.h @@ -254,22 +254,6 @@ void ioc_clear_queue(struct request_queue *q); int create_task_io_context(struct task_struct *task, gfp_t gfp_mask, int node); -/** - * rq_ioc - determine io_context for request allocation - * @bio: request being allocated is for this bio (can be %NULL) - * - * Determine io_context to use for request allocation for @bio. May return - * %NULL if %current->io_context doesn't exist. - */ -static inline struct io_context *rq_ioc(struct bio *bio) -{ -#ifdef CONFIG_BLK_CGROUP - if (bio && bio->bi_ioc) - return bio->bi_ioc; -#endif - return current->io_context; -} - /** * create_io_context - try to create task->io_context * @gfp_mask: allocation mask diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index dbdbfbd6a987..c0ba1a038ff3 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -174,10 +174,9 @@ struct bio { void *bi_private; #ifdef CONFIG_BLK_CGROUP /* - * Optional ioc and css associated with this bio. Put on bio + * Optional css associated with this bio. Put on bio * release. Read comment on top of bio_associate_current(). */ - struct io_context *bi_ioc; struct cgroup_subsys_state *bi_css; struct blkcg_gq *bi_blkg; struct bio_issue bi_issue; From patchwork Mon Nov 19 03:51:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688179 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B0BB109C for ; Mon, 19 Nov 2018 03:52:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36DE9286CA for ; Mon, 19 Nov 2018 03:52:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2BB5E29ACA; Mon, 19 Nov 2018 03:52:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D82A1286CA for ; Mon, 19 Nov 2018 03:51:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728205AbeKSOOP (ORCPT ); Mon, 19 Nov 2018 09:14:15 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62740 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726903AbeKSOOP (ORCPT ); Mon, 19 Nov 2018 09:14:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599519; x=1574135519; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZIIjB7gNVpAOpRwCqgvSYsejmZ9E4J4t+YZiNAL7w7c=; b=OJNtPItSLtcFEoda6vmOWP9Y9R0PUUV6q+jKlkTXOeGADMEMyw4jnoLs jF66TVzA+ct5/cDOCguSfT7CKT7/evxWvsIqiCI48PWIvxpDw0qVEBpus YAiURgewuQoo60qbmek0Dhcy9Fi3ruyuxvl2JJ7W6+I9+FLgZJ7IQo9i2 PH0kRJMEfDm6DmNC8eVp6bekrkkWC6SsJFZ3cCN23vpZHNnM6Gh3exT6T YZ5zZWYvAyCWWXMjvkqRc/avhAfZe+96m+NMXuQO4qTChATAK+bAlGInh wL77q2on5vXKcV3238YssicsAlNstmVIjEkAjOL3M49FtMgHKkIEBfs62 A==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272451" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:42 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:41 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:40 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 3/7] block: Fix get_task_ioprio() default return value Date: Mon, 19 Nov 2018 12:51:27 +0900 Message-Id: <20181119035131.11255-4-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As explained in ioprio_get() and ionice man pages, the default I/O priority class for processes which have not set an I/O priority using ioprio_set() is IOPRIO_CLASS_BE and not IOPRIO_CLASS_NONE. Signed-off-by: Damien Le Moal --- block/ioprio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/ioprio.c b/block/ioprio.c index f9821080c92c..ea5be206eb26 100644 --- a/block/ioprio.c +++ b/block/ioprio.c @@ -163,7 +163,7 @@ static int get_task_ioprio(struct task_struct *p) ret = security_task_getioprio(p); if (ret) goto out; - ret = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, IOPRIO_NORM); + ret = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_NORM); task_lock(p); if (p->io_context) ret = p->io_context->ioprio; From patchwork Mon Nov 19 03:51:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2ADD817D2 for ; Mon, 19 Nov 2018 03:52:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1544B286CA for ; Mon, 19 Nov 2018 03:52:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 09AB729AC4; Mon, 19 Nov 2018 03:52:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9D9829ACA for ; Mon, 19 Nov 2018 03:52:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728253AbeKSOOR (ORCPT ); Mon, 19 Nov 2018 09:14:17 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62743 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726903AbeKSOOQ (ORCPT ); Mon, 19 Nov 2018 09:14:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599520; x=1574135520; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=opAle5GdR+avzfDAyhIIsXOTcGEeTgm1Sxii3yZO8es=; b=pujLVFEqkMSeuI8cyKxumuf9iHU+7KwXnG7ZHL2zHYKPyuhW1pamJfx9 QeddaARwOSJ161QYKOe/R+to1hffJDv+5dtqR0FTJsNNOIpInpC7QXN6x ezjucM9cZzcdMygEepInqfkGIZ1jE8Z87mFZ3zgVu6BRV26T9GwDD6EIE c+IAJjFu/gTLSXRduiEvRZ2cr8Xxr5RlaS9X6uuxfRwazefaXnXw3vcmW LwSWxqzlPXe2r4x5yEkvucRXQ2iaIZooLScDu/omk6TBnExBk9jke/TjL 061MACbd2CkqMU+Y7hXbPaiYnQl6jFj73qr+ZDL9rj/0Q1+LtlDWErYI9 w==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272456" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:45 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:43 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:43 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 4/7] block: Introduce get_current_ioprio() Date: Mon, 19 Nov 2018 12:51:28 +0900 Message-Id: <20181119035131.11255-5-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Define get_current_ioprio() as an inline helper to obtain the caller I/O priority from its task I/O context. Use this helper in blk_init_request_from_bio() to set a request ioprio. Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn --- block/blk-core.c | 6 +----- include/linux/ioprio.h | 13 +++++++++++++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 492648c96992..4450d3c08f25 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -813,18 +813,14 @@ unsigned int blk_plug_queued_count(struct request_queue *q) void blk_init_request_from_bio(struct request *req, struct bio *bio) { - struct io_context *ioc = current->io_context; - if (bio->bi_opf & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; req->__sector = bio->bi_iter.bi_sector; if (ioprio_valid(bio_prio(bio))) req->ioprio = bio_prio(bio); - else if (ioc) - req->ioprio = ioc->ioprio; else - req->ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); + req->ioprio = get_current_ioprio(); req->write_hint = bio->bi_write_hint; blk_rq_bio_prep(req->q, req, bio); } diff --git a/include/linux/ioprio.h b/include/linux/ioprio.h index 9e30ed6443db..e9bfe6972aed 100644 --- a/include/linux/ioprio.h +++ b/include/linux/ioprio.h @@ -70,6 +70,19 @@ static inline int task_nice_ioclass(struct task_struct *task) return IOPRIO_CLASS_BE; } +/* + * If the calling process has set an I/O priority, use that. Otherwise, return + * the default I/O priority. + */ +static inline int get_current_ioprio(void) +{ + struct io_context *ioc = current->io_context; + + if (ioc) + return ioc->ioprio; + return IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); +} + /* * For inheritance, return the highest of the two given priorities */ From patchwork Mon Nov 19 03:51:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688201 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6CA55109C for ; Mon, 19 Nov 2018 03:52:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A964286CA for ; Mon, 19 Nov 2018 03:52:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4EA2729AC9; Mon, 19 Nov 2018 03:52:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 004E6286CA for ; Mon, 19 Nov 2018 03:52:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728293AbeKSOOU (ORCPT ); Mon, 19 Nov 2018 09:14:20 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62740 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728188AbeKSOOQ (ORCPT ); Mon, 19 Nov 2018 09:14:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599520; x=1574135520; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=D3K+H7awC5p/e5QtEP4eZu2ImmI8Bnbh0jwo9Zz/Kj0=; b=qFajbTMpXgltTFW/zRZEXSHNSJ+VlhAwAoNPKRuk/U+l/facHiD0gMPH Y6iYFYHF0+2xe1lKugUNPQfM7NRDNV5HgeWDhvs+/M1HpedDhNbESyTHg veI25k2KyDg8FAqPum6TmhFM0EJLxEvCRFAARMuWme8ewkJU/ohWRX+zm vNxLPNLot/xdEIDVanBy+dtS3gA9p/85HXaHVtkN5jkrcASUDZUJrYXUO QsMhyGYYvx9C5+bk3c+9wsdkbctosLrylwLPRy6lhTUYv9RC42vlkt455 +hrENF49UVK7qR2rRZ2PLcaTyj8szweak8F8yqesksUFl3DWgGyNptxsc w==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272461" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:47 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:45 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:45 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 5/7] aio: Fix fallback I/O priority value Date: Mon, 19 Nov 2018 12:51:29 +0900 Message-Id: <20181119035131.11255-6-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For cases when the application does not specify aio_reqprio for an aio, fallback to use get_current_ioprio() to obtain the task I/O priority last set using ioprio_set() rather than the hardcoded IOPRIO_CLASS_NONE value. Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Adam Manzanares --- fs/aio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/aio.c b/fs/aio.c index 301e6314183b..b984918be4b7 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1441,7 +1441,7 @@ static int aio_prep_rw(struct kiocb *req, struct iocb *iocb) req->ki_ioprio = iocb->aio_reqprio; } else - req->ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); + req->ki_ioprio = get_current_ioprio(); ret = kiocb_set_rw_flags(req, iocb->aio_rw_flags); if (unlikely(ret)) From patchwork Mon Nov 19 03:51:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688191 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB5F8109C for ; Mon, 19 Nov 2018 03:52:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9E6C286CA for ; Mon, 19 Nov 2018 03:52:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E64629AC9; Mon, 19 Nov 2018 03:52:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4B47C286CA for ; Mon, 19 Nov 2018 03:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728266AbeKSOOS (ORCPT ); Mon, 19 Nov 2018 09:14:18 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62743 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728217AbeKSOOR (ORCPT ); Mon, 19 Nov 2018 09:14:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599520; x=1574135520; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Gxw4QXlXWHHW6qd8AwCOFH6CaAG1U3Df7ye+7Qiwl/E=; b=EKOKAZJR2fs3D+cq0u5omemTAovkz2KSJJpiqlXWFGGjTEAqdb4CtKgs I/DB3Le80ml3kRgMocWVhXx5wD9oEgihuqB+8MfPwe2ABiUJaT1yZAdDy sGAStYliQ3gJZJBXeuY5fggPno1qp6nfzb/MbbLDXdI0c2YuHdqFcWSJZ b6je00aIWZzKfXy2YikPcwPF83863mWORgVod0JW5bnCp06tP6RcOaMDv N3OqYt4cPsTvOpG3WDOGxWnhALmATVPNL6ul5YAw5GfrB8oVHTMCuCbSm iDr4l+YySa/FhY2LBAFCvS8NZ47z1M9i4BSio1Ukr8mLw4OCoqTu6sRtp g==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272466" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:49 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:48 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:48 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 6/7] block: prevent merging of requests with different priorities Date: Mon, 19 Nov 2018 12:51:30 +0900 Message-Id: <20181119035131.11255-7-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Growing in size a high priority request by merging it with a lower priority BIO or request will increase the request execution time. This is the opposite result of the desired effect of high I/O priorities, namely getting low I/O latencies. Prevent merging of requests and BIOs that have different I/O priorities to fix this. Signed-off-by: Damien Le Moal Reviewed-by: Johannes Thumshirn --- block/blk-core.c | 3 --- block/blk-merge.c | 14 ++++++++++++-- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 4450d3c08f25..dde30b08aa14 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -662,7 +662,6 @@ bool bio_attempt_back_merge(struct request_queue *q, struct request *req, req->biotail->bi_next = bio; req->biotail = bio; req->__data_len += bio->bi_iter.bi_size; - req->ioprio = ioprio_best(req->ioprio, bio_prio(bio)); blk_account_io_start(req, false); return true; @@ -686,7 +685,6 @@ bool bio_attempt_front_merge(struct request_queue *q, struct request *req, req->__sector = bio->bi_iter.bi_sector; req->__data_len += bio->bi_iter.bi_size; - req->ioprio = ioprio_best(req->ioprio, bio_prio(bio)); blk_account_io_start(req, false); return true; @@ -706,7 +704,6 @@ bool bio_attempt_discard_merge(struct request_queue *q, struct request *req, req->biotail->bi_next = bio; req->biotail = bio; req->__data_len += bio->bi_iter.bi_size; - req->ioprio = ioprio_best(req->ioprio, bio_prio(bio)); req->nr_phys_segments = segments + 1; blk_account_io_start(req, false); diff --git a/block/blk-merge.c b/block/blk-merge.c index b1df622cbd85..129f554d250d 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -752,6 +752,12 @@ static struct request *attempt_merge(struct request_queue *q, if (req->write_hint != next->write_hint) return NULL; + /* + * Don't allow merge of different I/O priorities. + */ + if (req->ioprio != next->ioprio) + return NULL; + /* * If we are allowed to merge, then append bio list * from next to rq and release next. merge_requests_fn @@ -807,8 +813,6 @@ static struct request *attempt_merge(struct request_queue *q, */ blk_account_io_merge(next); - req->ioprio = ioprio_best(req->ioprio, next->ioprio); - /* * ownership of bio passed from next to req, return 'next' for * the caller to free @@ -883,6 +887,12 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (rq->write_hint != bio->bi_write_hint) return false; + /* + * Don't allow merge of different I/O priorities. + */ + if (rq->ioprio != bio_prio(bio)) + return false; + return true; } From patchwork Mon Nov 19 03:51:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Damien Le Moal X-Patchwork-Id: 10688197 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D86FD1923 for ; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C67E3286CA for ; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA75329AC9; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E1FA286CA for ; Mon, 19 Nov 2018 03:52:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728279AbeKSOOT (ORCPT ); Mon, 19 Nov 2018 09:14:19 -0500 Received: from esa6.hgst.iphmx.com ([216.71.154.45]:62746 "EHLO esa6.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726903AbeKSOOR (ORCPT ); Mon, 19 Nov 2018 09:14:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1542599521; x=1574135521; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0Xnqma5d37U1rxyzamL4z6UnYEUN6VPkgOpnFiZGCoA=; b=eenaSNxEhz1ZvQhwtXTiIw71spRoU3YDFv5ckm5x7o6MDuPng/AxifcA U01s0Cx1q+1oLhtVgklv6WDRyO8tYbyF5zGa4tClf+SZpKqHMmk4j9dcE KORPtjdjZKtrCKU98+493zMg0jKR5yCPQBV167x+UON0Nn5rikq3gRIw5 ZRVNoUn5QvCd/frCbPts2+Azx6mANImJsGasSQ5nF9xfaFf2T1/ViVZwM n8a9Sje5UVG3N11FRcatWUBedL7G1hKKOpNzn3OLNNvQcbpDOaVq9V4Xa FZavbTL6sAL9iB71Y8Ovr69DU9JEHGL+NArUGJkPjTxRsKXWaRC9pRdDJ Q==; X-IronPort-AV: E=Sophos;i="5.56,250,1539619200"; d="scan'208";a="96272470" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 19 Nov 2018 11:51:52 +0800 Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP; 18 Nov 2018 19:34:50 -0800 Received: from washi.fujisawa.hgst.com ([10.149.53.254]) by uls-op-cesaip02.wdc.com with ESMTP; 18 Nov 2018 19:51:50 -0800 From: Damien Le Moal To: linux-block@vger.kernel.org, Jens Axboe Cc: Adam Manzanares , Alexander Viro , linux-fsdevel@vger.kernel.org Subject: [PATCH 7/7] block: Initialize BIO I/O priority early Date: Mon, 19 Nov 2018 12:51:31 +0900 Message-Id: <20181119035131.11255-8-damien.lemoal@wdc.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119035131.11255-1-damien.lemoal@wdc.com> References: <20181119035131.11255-1-damien.lemoal@wdc.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For the synchronous I/O path case (read(), write() etc system calls), a BIO I/O priority is not initialized until the execution of blk_init_request_from_bio() when the BIO is submitted and a request initialized for the BIO execution. This is due to the ki_ioprio field of the struct kiocb defined on stack being always initialized to IOPRIO_CLASS_NONE, regardless of the calling process I/O context ioprio value set with ioprio_set(). This late initialization can result in the BIO being merged to pending requests even when the I/O priorities differ. Fix this by initializing the ki_iopriority field of on stack struct kiocb using the get_current_ioprio() helper, ensuring that all BIOs allocated and submitted for the system call execution see the correct intended I/O priority early. With this, since a BIO I/O priority is always set to the intended effective value for both the sync and async path, blk_init_request_from_bio() can be simplified. Signed-off-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Adam Manzanares --- block/blk-core.c | 5 +---- include/linux/fs.h | 2 +- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index dde30b08aa14..04f5be473638 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -814,10 +814,7 @@ void blk_init_request_from_bio(struct request *req, struct bio *bio) req->cmd_flags |= REQ_FAILFAST_MASK; req->__sector = bio->bi_iter.bi_sector; - if (ioprio_valid(bio_prio(bio))) - req->ioprio = bio_prio(bio); - else - req->ioprio = get_current_ioprio(); + req->ioprio = bio_prio(bio); req->write_hint = bio->bi_write_hint; blk_rq_bio_prep(req->q, req, bio); } diff --git a/include/linux/fs.h b/include/linux/fs.h index c95c0807471f..a1ab233e6469 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2021,7 +2021,7 @@ static inline void init_sync_kiocb(struct kiocb *kiocb, struct file *filp) .ki_filp = filp, .ki_flags = iocb_flags(filp), .ki_hint = ki_hint_validate(file_write_hint(filp)), - .ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0), + .ki_ioprio = get_current_ioprio(), }; }