From patchwork Mon Oct 18 17:51:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0368BC433FE for ; Mon, 18 Oct 2021 17:51:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC52961027 for ; Mon, 18 Oct 2021 17:51:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234207AbhJRRxZ (ORCPT ); Mon, 18 Oct 2021 13:53:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233383AbhJRRxY (ORCPT ); Mon, 18 Oct 2021 13:53:24 -0400 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47DC7C06161C for ; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id x1so17256265iof.7 for ; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C6KyFvc9/cRpsVEldC9hLwi1Bm5WYTo8xsA740Oti4I=; b=uY070Cu39UZOIfxSbB8Ls4g+mTGycigkPKHFSZswHBChYP5IuXhNE7WISmhMgAPJ5M xGeaqr7+x53EuZzbN5Xk7yyvE/PKCy64Nm/QtZ+LqTiutpI1/JumLMU74f2k5ybrwr1r LI56K2s6DibElVZCAHcuxXsOSCRDZx2SfiiIgtBx7j2sBUq0GYXv1bCGtjM9GcEOJLQO fvFyogLgOu1dpsOf2P1bCrNKEhovbBIv/+NoAY/fXvB7tnsdNj4xDW7QUXAvXuzB8xzf +L2y/S29xuR9GUzj0XEnwhu8tfyBsmo6wXRLfHghxcY3k7YDVReYAjfJ6bmnAXXoA8nk PlBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C6KyFvc9/cRpsVEldC9hLwi1Bm5WYTo8xsA740Oti4I=; b=JSHtzWI1mRQyo2xuvNjKPGpasBgYO68E63McGkkfDTbZ0s1WUyUuhB/ACNcvIU2f/3 TYr2Oq44X4t27QhirjuPUBKPByPQcvYknqvFL/NXWZIIz4j8QQ/iuum0ZSwIsPiiW2tO tMK371VjNbwA9/HGiwM6/HHHQBQlZUOZXDmwrgnEg/YlvE6IYcSG23xOMIvaGcozUHRA ycVo/PTj9fpuIsRwB6lmIkArG2k1wN66BHxeDFEp520sm7OBiWS8SfX+QpqlpOVHuQxp 8Y1To4GkgeSZ7YClyx6258Z4KJPsqt5NetY+AjAmUifLT1wA6IS307VimcV65FZdCY7S StgA== X-Gm-Message-State: AOAM530IjiHcBhknwIWBeWwkgqPAp+Eyez6leZqco51q4Ok63io2yjS1 up71IlgNB96VfBziY8w4Bb77ey3Hwc5TIw== X-Google-Smtp-Source: ABdhPJzvDtv/FgT6zwhpuNOr2Re9j+qLdfFFODD2j5sNJGm70gR7RXrvBBna6V8t2zp4cfqJUP0Mbw== X-Received: by 2002:a6b:8f44:: with SMTP id r65mr15458436iod.156.1634579472392; Mon, 18 Oct 2021 10:51:12 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:12 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 1/6] block: don't call blk_status_to_errno in blk_update_request Date: Mon, 18 Oct 2021 11:51:04 -0600 Message-Id: <20211018175109.401292-2-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Christoph Hellwig We only need to call it to resolve the blk_status_t -> errno mapping for tracing, so move the conversion into the tracepoints that are not called at all when tracing isn't enabled. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- block/blk-mq.c | 2 +- include/trace/events/block.h | 6 +++--- kernel/trace/blktrace.c | 7 ++++--- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f7c610f5f3e7..e3ef55f76701 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -672,7 +672,7 @@ bool blk_update_request(struct request *req, blk_status_t error, { int total_bytes; - trace_block_rq_complete(req, blk_status_to_errno(error), nr_bytes); + trace_block_rq_complete(req, error, nr_bytes); if (!req->bio) return false; diff --git a/include/trace/events/block.h b/include/trace/events/block.h index cc5ab96a7471..a95daa4d4caa 100644 --- a/include/trace/events/block.h +++ b/include/trace/events/block.h @@ -114,7 +114,7 @@ TRACE_EVENT(block_rq_requeue, */ TRACE_EVENT(block_rq_complete, - TP_PROTO(struct request *rq, int error, unsigned int nr_bytes), + TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes), TP_ARGS(rq, error, nr_bytes), @@ -122,7 +122,7 @@ TRACE_EVENT(block_rq_complete, __field( dev_t, dev ) __field( sector_t, sector ) __field( unsigned int, nr_sector ) - __field( int, error ) + __field( int , error ) __array( char, rwbs, RWBS_LEN ) __dynamic_array( char, cmd, 1 ) ), @@ -131,7 +131,7 @@ TRACE_EVENT(block_rq_complete, __entry->dev = rq->rq_disk ? disk_devt(rq->rq_disk) : 0; __entry->sector = blk_rq_pos(rq); __entry->nr_sector = nr_bytes >> 9; - __entry->error = error; + __entry->error = blk_status_to_errno(error); blk_fill_rwbs(__entry->rwbs, rq->cmd_flags); __get_str(cmd)[0] = '\0'; diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index fa91f398f28b..1183c88634aa 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -816,7 +816,7 @@ blk_trace_request_get_cgid(struct request *rq) * Records an action against a request. Will log the bio offset + size. * **/ -static void blk_add_trace_rq(struct request *rq, int error, +static void blk_add_trace_rq(struct request *rq, blk_status_t error, unsigned int nr_bytes, u32 what, u64 cgid) { struct blk_trace *bt; @@ -834,7 +834,8 @@ static void blk_add_trace_rq(struct request *rq, int error, what |= BLK_TC_ACT(BLK_TC_FS); __blk_add_trace(bt, blk_rq_trace_sector(rq), nr_bytes, req_op(rq), - rq->cmd_flags, what, error, 0, NULL, cgid); + rq->cmd_flags, what, blk_status_to_errno(error), 0, + NULL, cgid); rcu_read_unlock(); } @@ -863,7 +864,7 @@ static void blk_add_trace_rq_requeue(void *ignore, struct request *rq) } static void blk_add_trace_rq_complete(void *ignore, struct request *rq, - int error, unsigned int nr_bytes) + blk_status_t error, unsigned int nr_bytes) { blk_add_trace_rq(rq, error, nr_bytes, BLK_TA_COMPLETE, blk_trace_request_get_cgid(rq)); From patchwork Mon Oct 18 17:51:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93A0AC4332F for ; Mon, 18 Oct 2021 17:51:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B109610C8 for ; Mon, 18 Oct 2021 17:51:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229980AbhJRRxZ (ORCPT ); Mon, 18 Oct 2021 13:53:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234135AbhJRRxZ (ORCPT ); Mon, 18 Oct 2021 13:53:25 -0400 Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3477C06161C for ; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) Received: by mail-io1-xd2f.google.com with SMTP id y67so17227253iof.10 for ; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nHdWo5JlPcrT5ny9e2xXLs7Ss2obuQA7vRyruQpssLs=; b=dh+8RZ+gUJBSgu3ijYn8rYkrAMNcTC1VEK2abmNh6zBMaNjF24PxT3h4z++MyjvYH1 TCrkqkZ5X8MR44nhaWzYPuwRVZmnSWqwcs3LSGJULSSmO53Vp+SITASt3Ghlc44jFNb7 CykIXr7XuKx+NHAXp7u6vtDl3v7JV3Xf5Jn2kr91nUOmHWlZSKqH0q9NDS8W+HW4q3Mc V4zPnDsAZbmypH0cFuA/svtwTs96yh1yaSB2jocP6wIFoNavINQJa5OeytRx7lbFIoA5 1lBRwZzF+dJQx3lRmusKC8OEswZKmJ/T7mxLhREmJB2Twqoz0x22Ku6K7GdYoWgXUR8D csPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nHdWo5JlPcrT5ny9e2xXLs7Ss2obuQA7vRyruQpssLs=; b=dM8NE6m0tcmysXyGIaKjEG/zbMJRr55FaVy2kGSO9IzFQOUHhakffh6aIlZYkDRJqH dQCdV7GrHiqYdrhbZ5U/hGu7vUDTRMZN/x6hZA1KhzEiY/ETkAGC3xYk3iuEj1vAC2Ap lfBeNkqb/e3QmUf69/+JEXfgi5f069sohFU/dwV0eyo0ILDYV+XLa+bk0/bGEjFUWMPx 3Ul4agDVUoB9qWAt+vkkkcyJZTPL26xqgn9+uH6QEVtZ4LR3uQsUgZ9m3/c91M+99mDE Rswj4PfLLcBP7izA7opwjo+qyxK1kd3jEGVUxDVEZQvZ088qDZK7nowlRBHz5d3vIkVs L/WA== X-Gm-Message-State: AOAM530ZBPUdH95G6ggxgFhrSZgzZi1t5jynC0hvd0EuIp61jOdLfFWY wkZfVcLiaFLyzBICG5jNwyKyJOS9Og4T/A== X-Google-Smtp-Source: ABdhPJxa/HaNYOdtT2bPcoWkljamtjQ+kisWTeNylgcFXnep42QnNj+4W5IVt3CGtIdeMJ5dUFPXIA== X-Received: by 2002:a05:6602:1546:: with SMTP id h6mr15133840iow.125.1634579473143; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:12 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 2/6] block: return whether or not to unplug through boolean Date: Mon, 18 Oct 2021 11:51:05 -0600 Message-Id: <20211018175109.401292-3-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Instead of returning the same queue request through a request pointer, use a boolean to accomplish the same. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-merge.c | 11 +++++------ block/blk-mq.c | 16 +++++++++------- block/blk.h | 2 +- 3 files changed, 15 insertions(+), 14 deletions(-) diff --git a/block/blk-merge.c b/block/blk-merge.c index ec727234ac48..c273b58378ce 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -1067,9 +1067,8 @@ static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, * @q: request_queue new bio is being queued at * @bio: new bio being queued * @nr_segs: number of segments in @bio - * @same_queue_rq: pointer to &struct request that gets filled in when - * another request associated with @q is found on the plug list - * (optional, may be %NULL) + * @same_queue_rq: output value, will be true if there's an existing request + * from the passed in @q already in the plug list * * Determine whether @bio being queued on @q can be merged with the previous * request on %current's plugged list. Returns %true if merge was successful, @@ -1085,7 +1084,7 @@ static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q, * Caller must ensure !blk_queue_nomerges(q) beforehand. */ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, - unsigned int nr_segs, struct request **same_queue_rq) + unsigned int nr_segs, bool *same_queue_rq) { struct blk_plug *plug; struct request *rq; @@ -1096,12 +1095,12 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, /* check the previously added entry for a quick merge attempt */ rq = list_last_entry(&plug->mq_list, struct request, queuelist); - if (rq->q == q && same_queue_rq) { + if (rq->q == q) { /* * Only blk-mq multiple hardware queues case checks the rq in * the same queue, there should be only one such rq in a queue */ - *same_queue_rq = rq; + *same_queue_rq = true; } if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) == BIO_MERGE_OK) return true; diff --git a/block/blk-mq.c b/block/blk-mq.c index e3ef55f76701..d957b6812a98 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2422,7 +2422,7 @@ void blk_mq_submit_bio(struct bio *bio) const int is_flush_fua = op_is_flush(bio->bi_opf); struct request *rq; struct blk_plug *plug; - struct request *same_queue_rq = NULL; + bool same_queue_rq = false; unsigned int nr_segs = 1; blk_status_t ret; @@ -2515,6 +2515,8 @@ void blk_mq_submit_bio(struct bio *bio) /* Insert the request at the IO scheduler queue */ blk_mq_sched_insert_request(rq, false, true, true); } else if (plug && !blk_queue_nomerges(q)) { + struct request *next_rq = NULL; + /* * We do limited plugging. If the bio can be merged, do that. * Otherwise the existing request in the plug list will be @@ -2522,19 +2524,19 @@ void blk_mq_submit_bio(struct bio *bio) * The plug list might get flushed before this. If that happens, * the plug list is empty, and same_queue_rq is invalid. */ - if (list_empty(&plug->mq_list)) - same_queue_rq = NULL; if (same_queue_rq) { - list_del_init(&same_queue_rq->queuelist); + next_rq = list_last_entry(&plug->mq_list, + struct request, + queuelist); + list_del_init(&next_rq->queuelist); plug->rq_count--; } blk_add_rq_to_plug(plug, rq); trace_block_plug(q); - if (same_queue_rq) { + if (next_rq) { trace_block_unplug(q, 1, true); - blk_mq_try_issue_directly(same_queue_rq->mq_hctx, - same_queue_rq); + blk_mq_try_issue_directly(next_rq->mq_hctx, next_rq); } } else if ((q->nr_hw_queues > 1 && is_sync) || !rq->mq_hctx->dispatch_busy) { diff --git a/block/blk.h b/block/blk.h index e80350327e6d..b9729c12fd62 100644 --- a/block/blk.h +++ b/block/blk.h @@ -218,7 +218,7 @@ void blk_add_timer(struct request *req); void blk_print_req_error(struct request *req, blk_status_t status); bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, - unsigned int nr_segs, struct request **same_queue_rq); + unsigned int nr_segs, bool *same_queue_rq); bool blk_bio_list_merge(struct request_queue *q, struct list_head *list, struct bio *bio, unsigned int nr_segs); From patchwork Mon Oct 18 17:51:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09C35C433EF for ; Mon, 18 Oct 2021 17:51:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EF02E61077 for ; Mon, 18 Oct 2021 17:51:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234135AbhJRRx0 (ORCPT ); Mon, 18 Oct 2021 13:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233383AbhJRRxZ (ORCPT ); Mon, 18 Oct 2021 13:53:25 -0400 Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86020C06161C for ; Mon, 18 Oct 2021 10:51:14 -0700 (PDT) Received: by mail-io1-xd29.google.com with SMTP id n7so17352427iod.0 for ; Mon, 18 Oct 2021 10:51:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZRo+0Nv/5yTlZsp5PnfL7gTFHW11KoE6fwo/PS73/zs=; b=HibWV3m6ESAwvkPj/RgK2oFibmOMs4F+q8KgbZuCmfRM7+N90UfAl1fKB2GM3Ozoti 8SaoUU4Zu/fX1H6rx1SaU07DHlgriZhlwvAbue/vI6n3YRS59Npn805l5Y+Tg2iCIXSc ewK7rnPD1lE0vFS5SWxaWRXWjYwhubPynpl6MXYbQWSDzgPTBesEy5FRsEeh7Lr5OOfE JmUYZweea6o0fj/HFR0HYOOn9/g4XzcMkkGSTQu25a4WjYWAD0mjiPNrEspVQp018IL6 /gjU13HJqFl+L3GnA6agb35/RHBBrbZ5H4KpG+K3rGJw7YFL3n7cI90X8PwOa5CFtZxe AQmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZRo+0Nv/5yTlZsp5PnfL7gTFHW11KoE6fwo/PS73/zs=; b=5UbiN2HI9e9Ra6jpWywhFvPtxUoaB5WLfGMiK4DkvKjyb1rgy/rbXZHru3AQ9BMPkC Rm9VH889MyuRnYoiWCuDv9J/fEOGGxiTNzPjhcdLHXY0eGJMTzLxaEtuwYR4sCmdRWxk LGfKoDVejWtzfX+zkUm7sT9SekIfkHsqrUNECYJ9GB/kRWND2cXI0najqcSJeqTghcH+ Cd1vIo/Q2dfPfrLB3sEzee0fqmrzSYtWxbbyXyMAtC215iFB8jLlLKixSU2BwugNgtBl KaVJRVoQKTXRsh4BZNjGTngJmwQqWpa/Wxk9lEkq3wPmlXEtfd0Zx10cgAXUDAWVuG6z KLVg== X-Gm-Message-State: AOAM532R9INSxu9QTQVPlmtHppSOoMRUJYSQSRDfQIsetuAyb3PQzXWH MZRH8X9GclvKV4d+yTZei5crKWsKri50CQ== X-Google-Smtp-Source: ABdhPJxI8vnh38YET74M68PBh5L1tmD3cm+0adazAppsZChoHR/6+JLC7i90tCJP1cK/DMx+jwdGfw== X-Received: by 2002:a05:6602:2f0a:: with SMTP id q10mr15759272iow.76.1634579473787; Mon, 18 Oct 2021 10:51:13 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:13 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 3/6] block: get rid of plug list sorting Date: Mon, 18 Oct 2021 11:51:06 -0600 Message-Id: <20211018175109.401292-4-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Even if we have multiple queues in the plug list, chances that they are very interspersed is minimal. Don't bother spending CPU cycles sorting the list. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index d957b6812a98..58774267dd95 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include #include @@ -2151,20 +2150,6 @@ void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx, spin_unlock(&ctx->lock); } -static int plug_rq_cmp(void *priv, const struct list_head *a, - const struct list_head *b) -{ - struct request *rqa = container_of(a, struct request, queuelist); - struct request *rqb = container_of(b, struct request, queuelist); - - if (rqa->mq_ctx != rqb->mq_ctx) - return rqa->mq_ctx > rqb->mq_ctx; - if (rqa->mq_hctx != rqb->mq_hctx) - return rqa->mq_hctx > rqb->mq_hctx; - - return blk_rq_pos(rqa) > blk_rq_pos(rqb); -} - void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { LIST_HEAD(list); @@ -2172,10 +2157,6 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) if (list_empty(&plug->mq_list)) return; list_splice_init(&plug->mq_list, &list); - - if (plug->rq_count > 2 && plug->multiple_queues) - list_sort(NULL, &list, plug_rq_cmp); - plug->rq_count = 0; do { From patchwork Mon Oct 18 17:51:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFFBCC43217 for ; Mon, 18 Oct 2021 17:51:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9BC4761077 for ; Mon, 18 Oct 2021 17:51:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233383AbhJRRx1 (ORCPT ); Mon, 18 Oct 2021 13:53:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234216AbhJRRx0 (ORCPT ); Mon, 18 Oct 2021 13:53:26 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EEBBC06161C for ; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id s17so17219433ioa.13 for ; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pFknZAQ8BmeVCGAxMw2E9Pvn0JC2j+fmOPqQbcmwYTI=; b=U4gFvTFEXkWmnmgH7WaiwZlMiDi72DxEM9/HKKX67ySJqOn8fi8d/ojfZiE0tk7+Bq PcwpE1ilGdsZM2tO4D4lYV/nX38CzItUF++fMxhNXYcnVg+lizWicZt3dPozHgAS6hFk GeCAisWomGLJRxN/v16TSnOQMgUKcIMi7rR7wf9NzmybhrzDiLUf7QLWq13G26Gg2Dyi F3Ri7Kz1dz8OGy86XB89QULcrqvfjg4G5a/znQUVDYFmM52lIUiBIGTf3jZ0ZAGIYBzx 83CXTP8hWJ7bntxPqyRr9WZ2GXhx27ski7y2P/T7k79eF1YAQgz1qvopsg96W7xZYB1G 5PZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pFknZAQ8BmeVCGAxMw2E9Pvn0JC2j+fmOPqQbcmwYTI=; b=NdDSiO26xi5n0Go/8EDUPL69EdxyO18oggEW5WNIQw5AjR39eKrjA11Ap9XKpabE9S ajMPTndN+ScGq23V99wOCr4C0b7QBkLNFx5tijFNVwAsY2UwErqBaTKlrmEM+EJm+SR6 NLi3neF9uqWOsj0FkESymQp2Dugh0ubaIzPOmP5QzW2qSASyk57tNEz05UKFFqfLxrDd Hss1dCe+iLHscmqveZGr0kVL5SgCrtp5MI8UPRCoednNiuSYIpMulkxAKF2ClShQ5e9k gn+1QA5NKAVTWy+UHbAUhLpTX7o04duEE/lrFhh91k/uG54WvQyAxv1cAaZz+wIxkqN/ dGzA== X-Gm-Message-State: AOAM533gCdQftAgkV1B7CoFSJGvtZFMgL2ZIFdylDOemKqROlmw8Ze0/ WLV7HwJS6z/qAqW9e2ttKPSw73oaTJA2QA== X-Google-Smtp-Source: ABdhPJzLQy19b6TpTjv/qAi2m6QQnb2g/RPZE8XZipMMWrRjeG5DQkqgfJyRCfeFgOzTtE7gMwcnAg== X-Received: by 2002:a6b:7604:: with SMTP id g4mr15454564iom.162.1634579474468; Mon, 18 Oct 2021 10:51:14 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:14 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 4/6] block: change plugging to use a singly linked list Date: Mon, 18 Oct 2021 11:51:07 -0600 Message-Id: <20211018175109.401292-5-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Use a singly linked list for the blk_plug. This saves 8 bytes in the blk_plug struct, and makes for faster list manipulations than doubly linked lists. As we don't use the doubly linked lists for anything, singly linked is just fine. This yields a bump in default (merging enabled) performance from 7.0 to 7.1M IOPS, and ~7.5M IOPS with merging disabled. Signed-off-by: Jens Axboe --- block/blk-core.c | 5 +- block/blk-merge.c | 4 +- block/blk-mq.c | 140 ++++++++++++++++++++++++++++++----------- include/linux/blkdev.h | 6 +- 4 files changed, 113 insertions(+), 42 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index d0c2e11411d0..e6ad5b51d0c3 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1550,11 +1550,12 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) if (tsk->plug) return; - INIT_LIST_HEAD(&plug->mq_list); + plug->mq_list = NULL; plug->cached_rq = NULL; plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); plug->rq_count = 0; plug->multiple_queues = false; + plug->has_elevator = false; plug->nowait = false; INIT_LIST_HEAD(&plug->cb_list); @@ -1640,7 +1641,7 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) { flush_plug_callbacks(plug, from_schedule); - if (!list_empty(&plug->mq_list)) + if (!rq_list_empty(plug->mq_list)) blk_mq_flush_plug_list(plug, from_schedule); if (unlikely(!from_schedule && plug->cached_rq)) blk_mq_free_plug_rqs(plug); diff --git a/block/blk-merge.c b/block/blk-merge.c index c273b58378ce..3e6fa449caff 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -1090,11 +1090,11 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, struct request *rq; plug = blk_mq_plug(q, bio); - if (!plug || list_empty(&plug->mq_list)) + if (!plug || rq_list_empty(plug->mq_list)) return false; /* check the previously added entry for a quick merge attempt */ - rq = list_last_entry(&plug->mq_list, struct request, queuelist); + rq = rq_list_peek(&plug->mq_list); if (rq->q == q) { /* * Only blk-mq multiple hardware queues case checks the rq in diff --git a/block/blk-mq.c b/block/blk-mq.c index 58774267dd95..3d5010d93059 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2150,36 +2150,106 @@ void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx, spin_unlock(&ctx->lock); } +static void blk_mq_commit_rqs(struct blk_mq_hw_ctx *hctx, int *queued, + bool from_schedule) +{ + if (hctx->queue->mq_ops->commit_rqs) { + trace_block_unplug(hctx->queue, *queued, !from_schedule); + hctx->queue->mq_ops->commit_rqs(hctx); + } + *queued = 0; +} + +static void blk_mq_plug_issue_direct(struct blk_plug *plug, bool from_schedule) +{ + struct blk_mq_hw_ctx *hctx = NULL; + struct request *rq; + int queued = 0; + int errors = 0; + + while ((rq = rq_list_pop(&plug->mq_list))) { + bool last = rq_list_empty(plug->mq_list); + blk_status_t ret; + + if (hctx != rq->mq_hctx) { + if (hctx) + blk_mq_commit_rqs(hctx, &queued, from_schedule); + hctx = rq->mq_hctx; + } + + ret = blk_mq_request_issue_directly(rq, last); + switch (ret) { + case BLK_STS_OK: + queued++; + break; + case BLK_STS_RESOURCE: + case BLK_STS_DEV_RESOURCE: + blk_mq_request_bypass_insert(rq, false, last); + blk_mq_commit_rqs(hctx, &queued, from_schedule); + return; + default: + blk_mq_end_request(rq, ret); + errors++; + break; + } + } + + /* + * If we didn't flush the entire list, we could have told the driver + * there was more coming, but that turned out to be a lie. + */ + if (errors) + blk_mq_commit_rqs(hctx, &queued, from_schedule); +} + void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { + struct blk_mq_hw_ctx *this_hctx; + struct blk_mq_ctx *this_ctx; + unsigned int depth; LIST_HEAD(list); - if (list_empty(&plug->mq_list)) + if (rq_list_empty(plug->mq_list)) return; - list_splice_init(&plug->mq_list, &list); plug->rq_count = 0; + if (!plug->multiple_queues && !plug->has_elevator) { + blk_mq_plug_issue_direct(plug, from_schedule); + if (rq_list_empty(plug->mq_list)) + return; + } + + this_hctx = NULL; + this_ctx = NULL; + depth = 0; do { - struct list_head rq_list; - struct request *rq, *head_rq = list_entry_rq(list.next); - struct list_head *pos = &head_rq->queuelist; /* skip first */ - struct blk_mq_hw_ctx *this_hctx = head_rq->mq_hctx; - struct blk_mq_ctx *this_ctx = head_rq->mq_ctx; - unsigned int depth = 1; - - list_for_each_continue(pos, &list) { - rq = list_entry_rq(pos); - BUG_ON(!rq->q); - if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) - break; - depth++; + struct request *rq; + + rq = rq_list_pop(&plug->mq_list); + + if (!this_hctx) { + this_hctx = rq->mq_hctx; + this_ctx = rq->mq_ctx; + } else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) { + trace_block_unplug(this_hctx->queue, depth, + !from_schedule); + blk_mq_sched_insert_requests(this_hctx, this_ctx, + &list, from_schedule); + depth = 0; + this_hctx = rq->mq_hctx; + this_ctx = rq->mq_ctx; + } - list_cut_before(&rq_list, &list, pos); - trace_block_unplug(head_rq->q, depth, !from_schedule); - blk_mq_sched_insert_requests(this_hctx, this_ctx, &rq_list, + list_add(&rq->queuelist, &list); + depth++; + } while (!rq_list_empty(plug->mq_list)); + + if (!list_empty(&list)) { + trace_block_unplug(this_hctx->queue, depth, !from_schedule); + blk_mq_sched_insert_requests(this_hctx, this_ctx, &list, from_schedule); - } while(!list_empty(&list)); + } } static void blk_mq_bio_to_request(struct request *rq, struct bio *bio, @@ -2359,16 +2429,17 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq) { - list_add_tail(&rq->queuelist, &plug->mq_list); - plug->rq_count++; - if (!plug->multiple_queues && !list_is_singular(&plug->mq_list)) { - struct request *tmp; + if (!plug->multiple_queues) { + struct request *nxt = rq_list_peek(&plug->mq_list); - tmp = list_first_entry(&plug->mq_list, struct request, - queuelist); - if (tmp->q != rq->q) + if (nxt && nxt->q != rq->q) plug->multiple_queues = true; } + if (!plug->has_elevator && (rq->rq_flags & RQF_ELV)) + plug->has_elevator = true; + rq->rq_next = NULL; + rq_list_add(&plug->mq_list, rq); + plug->rq_count++; } /* @@ -2480,13 +2551,15 @@ void blk_mq_submit_bio(struct bio *bio) unsigned int request_count = plug->rq_count; struct request *last = NULL; - if (!request_count) + if (!request_count) { trace_block_plug(q); - else - last = list_entry_rq(plug->mq_list.prev); + } else if (!blk_queue_nomerges(q)) { + last = rq_list_peek(&plug->mq_list); + if (blk_rq_bytes(last) < BLK_PLUG_FLUSH_SIZE) + last = NULL; + } - if (request_count >= blk_plug_max_rq_count(plug) || (last && - blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) { + if (request_count >= blk_plug_max_rq_count(plug) || last) { blk_flush_plug_list(plug, false); trace_block_plug(q); } @@ -2506,10 +2579,7 @@ void blk_mq_submit_bio(struct bio *bio) * the plug list is empty, and same_queue_rq is invalid. */ if (same_queue_rq) { - next_rq = list_last_entry(&plug->mq_list, - struct request, - queuelist); - list_del_init(&next_rq->queuelist); + next_rq = rq_list_pop(&plug->mq_list); plug->rq_count--; } blk_add_rq_to_plug(plug, rq); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index abe721591e80..2e93682f8f68 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -711,7 +711,7 @@ extern void blk_set_queue_dying(struct request_queue *); * schedule() where blk_schedule_flush_plug() is called. */ struct blk_plug { - struct list_head mq_list; /* blk-mq requests */ + struct request *mq_list; /* blk-mq requests */ /* if ios_left is > 1, we can batch tag/rq allocations */ struct request *cached_rq; @@ -720,6 +720,7 @@ struct blk_plug { unsigned short rq_count; bool multiple_queues; + bool has_elevator; bool nowait; struct list_head cb_list; /* md requires an unplug callback */ @@ -760,8 +761,7 @@ static inline bool blk_needs_flush_plug(struct task_struct *tsk) struct blk_plug *plug = tsk->plug; return plug && - (!list_empty(&plug->mq_list) || - !list_empty(&plug->cb_list)); + (plug->mq_list || !list_empty(&plug->cb_list)); } int blkdev_issue_flush(struct block_device *bdev); From patchwork Mon Oct 18 17:51:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B10FC433F5 for ; Mon, 18 Oct 2021 17:51:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23B9661074 for ; Mon, 18 Oct 2021 17:51:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234216AbhJRRx1 (ORCPT ); Mon, 18 Oct 2021 13:53:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234242AbhJRRx1 (ORCPT ); Mon, 18 Oct 2021 13:53:27 -0400 Received: from mail-il1-x12e.google.com (mail-il1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2B7DC06161C for ; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) Received: by mail-il1-x12e.google.com with SMTP id a8so15737918ilj.10 for ; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t+DV643unX74TTLFqfijj2SncIcFWXHZQrz3bUct0Ho=; b=UTPFJjvMgIDuDbmHWe2U/33gugkvPjp9ARfEt+FaCr3eg7wdEazFK63BAuSlTWABkw ofi97GJ59e2Vkdwp3fxQIH+d3AHnDS4X8mqb2WQp48lyMMXLMtdQynIHf05TUL/lSu5g RhcKkInMbSeOZV6GcYV8DwbzkJdy106SikXKkdew9i892ourcMl3DuKaPat/+yofdlzZ 6Do1JHdlFqEyWaxYy0XAb3kaYExIU0Ovhp+6vUJZt10ZKdhN3bVpt/K65k95JkwDJ7Im zYfyiI1IrXHuYK0aKDO3mkfh0ZuRWGOWUX38pcBgaHdq9dwC5yAbuWQNHkFTSagLZSOj cObg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t+DV643unX74TTLFqfijj2SncIcFWXHZQrz3bUct0Ho=; b=0ReZ7Lhx6GzDwIrEiZCe5gEZE8hv5jO0D0AmyVRe2yX9FPcFuy35UejpFG8Q6bnuDl vqf27LFZcO9uGZaPYAxCY3qBZG0DJpsw44Z6HK9NV9ZCLRuDwircHxbvF6VlEOtJ8ytN hUBShAKhlvZtYA/gP12Q5nCRaGAS9HV2/DQxXV6PxdtfnyMgazDb7g8zF2MibVe7qdM3 xljiUlfYuf421m2xkPkb+vMZUyHNtoLcN33rZhY0pU7a0t7e+tneF+aSqFQtH5MQElZr IVmjbLjxgzp2plYQM/Fe49d1F1i6eCGuMtbiGgv5MEOVs+EGMEI9AlkNqelD8SYRAeXp VB4Q== X-Gm-Message-State: AOAM533k6ZEVEz34gkrXzKC5q9hEveZaPR/GhIYS7ItbG2a8OBiqPdVU xHcnYYaZM4NJNqo591OZ4wz+/AVm09Nh+g== X-Google-Smtp-Source: ABdhPJxRp6Ndx4iw/ooQVho1nvW7bcPR29OHvmj0fA9gTf3U8qZDIX95eQZWAwmoCkLsR7Rp+YQxLA== X-Received: by 2002:a92:cd89:: with SMTP id r9mr15573669ilb.261.1634579475095; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:14 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 5/6] block: move blk_mq_tag_to_rq() inline Date: Mon, 18 Oct 2021 11:51:08 -0600 Message-Id: <20211018175109.401292-6-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This is in the fast path of driver issue or completion, and it's a single array index operation. Move it inline to avoid a function call for it. This does mean making struct blk_mq_tags block layer public, but there's not really much in there. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-mq-tag.h | 23 ----------------------- block/blk-mq.c | 11 ----------- include/linux/blk-mq.h | 36 +++++++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 35 deletions(-) diff --git a/block/blk-mq-tag.h b/block/blk-mq-tag.h index 78ae2fb8e2a4..df787b5a23bd 100644 --- a/block/blk-mq-tag.h +++ b/block/blk-mq-tag.h @@ -4,29 +4,6 @@ struct blk_mq_alloc_data; -/* - * Tag address space map. - */ -struct blk_mq_tags { - unsigned int nr_tags; - unsigned int nr_reserved_tags; - - atomic_t active_queues; - - struct sbitmap_queue bitmap_tags; - struct sbitmap_queue breserved_tags; - - struct request **rqs; - struct request **static_rqs; - struct list_head page_list; - - /* - * used to clear request reference in rqs[] before freeing one - * request pool - */ - spinlock_t lock; -}; - extern struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags, unsigned int reserved_tags, int node, int alloc_policy); diff --git a/block/blk-mq.c b/block/blk-mq.c index 3d5010d93059..9a73e2c3ce32 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1112,17 +1112,6 @@ void blk_mq_delay_kick_requeue_list(struct request_queue *q, } EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list); -struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag) -{ - if (tag < tags->nr_tags) { - prefetch(tags->rqs[tag]); - return tags->rqs[tag]; - } - - return NULL; -} -EXPORT_SYMBOL(blk_mq_tag_to_rq); - static bool blk_mq_rq_inflight(struct blk_mq_hw_ctx *hctx, struct request *rq, void *priv, bool reserved) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 656fe34bdb6c..6cf35de151a9 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -7,6 +7,7 @@ #include #include #include +#include struct blk_mq_tags; struct blk_flush_queue; @@ -675,7 +676,40 @@ struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op, struct request *blk_mq_alloc_request_hctx(struct request_queue *q, unsigned int op, blk_mq_req_flags_t flags, unsigned int hctx_idx); -struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag); + +/* + * Tag address space map. + */ +struct blk_mq_tags { + unsigned int nr_tags; + unsigned int nr_reserved_tags; + + atomic_t active_queues; + + struct sbitmap_queue bitmap_tags; + struct sbitmap_queue breserved_tags; + + struct request **rqs; + struct request **static_rqs; + struct list_head page_list; + + /* + * used to clear request reference in rqs[] before freeing one + * request pool + */ + spinlock_t lock; +}; + +static inline struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, + unsigned int tag) +{ + if (tag < tags->nr_tags) { + prefetch(tags->rqs[tag]); + return tags->rqs[tag]; + } + + return NULL; +} enum { BLK_MQ_UNIQUE_TAG_BITS = 16, From patchwork Mon Oct 18 17:51:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12567525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F35C7C433FE for ; Mon, 18 Oct 2021 17:51:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DEF5361027 for ; Mon, 18 Oct 2021 17:51:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234252AbhJRRx2 (ORCPT ); Mon, 18 Oct 2021 13:53:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233830AbhJRRx1 (ORCPT ); Mon, 18 Oct 2021 13:53:27 -0400 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D75BC061765 for ; Mon, 18 Oct 2021 10:51:16 -0700 (PDT) Received: by mail-io1-xd2d.google.com with SMTP id n7so17352541iod.0 for ; Mon, 18 Oct 2021 10:51:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nlhkTVMRTDUIg6xLkz4mvLFJzbvrzB5LrVKzz1g0kp8=; b=q7dIawP9BMGEUvv0nz3zPT66kOxfZ+7PLCOG8Z5ZmdlL1dFDEHcndYGVf45pKY5GXv Knoj1wtYK1I+a3ICzhzwEzJCwqY6qZeqCQQI/kcWqo0KxOn+MQKoa1rYBURntA1Dnsfv VDpyRuoCrgll8s8pOhl0z4jda7FOt8ezjaOiwW73AYRSaQnQ2gJAWaczQ77bIho4kc6F tlPf7Tp+rQe8ClG79Bwg9ysV3kkoKkM1wuUDuhr7UOjzNM2UWYZkOaYyqNQdIP7DWkwr YVwK0DCatLhUzETwyQqLqHJyo3YjJfoXiDWGa0Quk/nF+5MTetA4AcEJ4xJ/lM4OQs3U plEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nlhkTVMRTDUIg6xLkz4mvLFJzbvrzB5LrVKzz1g0kp8=; b=yyGYaCN0/uMgKqw9rbWdqghIFrFL3Zh9uDTVwuRRoxyjpuy+wiNJxY3iEJvswCF6S6 CwqoJk+uZFk1IE0Jll47nxB71gmUaGr9TMRJVAYZUGFSucYR2qR0NssPQri3jqnGjcfD lZyzaQFDtYSK7IJxJOQybCQKZlb2C85kzx4pHnU/jXIo/kUPBS9LNgsDmGPGV7IQQqX1 6U63SsHmRZIqWBkKwneY/t7zOWf39KdV6EOcgYrg4GhAnMdCqEQl1nN/hPRQWqMtkQVo ZmSwO22j+93dDY0s1WpBY9tY1MvKIvkaFZa2x1Xe0aOUwKb8VjiPUN3KbqeSQJl1Qjhu g5Pg== X-Gm-Message-State: AOAM5330d+LHGij4yDzNygtCTg1q76MYZRWau5IuRs+akhw0zy0xusay 2+vkukeBr6q/B+LY9V8WYcfXqkCBTZA= X-Google-Smtp-Source: ABdhPJw2D0VTGptY83uacStKNDIQi77yURzxzLQGUQuo+ogCij9iocKXK4u5PESrYMVs4Rf4u1/A/Q== X-Received: by 2002:a05:6602:2c88:: with SMTP id i8mr15696227iow.48.1634579475906; Mon, 18 Oct 2021 10:51:15 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id v17sm7380017ilh.67.2021.10.18.10.51.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Oct 2021 10:51:15 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: hch@lst.de, Jens Axboe Subject: [PATCH 6/6] block: align blkdev_dio inlined bio to a cacheline Date: Mon, 18 Oct 2021 11:51:09 -0600 Message-Id: <20211018175109.401292-7-axboe@kernel.dk> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211018175109.401292-1-axboe@kernel.dk> References: <20211018175109.401292-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We get all sorts of unreliable and funky results since the bio is designed to align on a cacheline, which it does not when inlined like this. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/fops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/fops.c b/block/fops.c index 2ae8a7bd2b84..e30082ea219a 100644 --- a/block/fops.c +++ b/block/fops.c @@ -137,7 +137,7 @@ struct blkdev_dio { size_t size; atomic_t ref; unsigned int flags; - struct bio bio; + struct bio bio ____cacheline_aligned_in_smp; }; static struct bio_set blkdev_dio_pool;