From patchwork Thu Nov 28 21:11:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11266603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 051A6921 for ; Thu, 28 Nov 2019 21:12:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D6ED921781 for ; Thu, 28 Nov 2019 21:12:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JXtMTc7o" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726692AbfK1VMX (ORCPT ); Thu, 28 Nov 2019 16:12:23 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:42029 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726569AbfK1VMX (ORCPT ); Thu, 28 Nov 2019 16:12:23 -0500 Received: by mail-wr1-f66.google.com with SMTP id a15so32647501wrf.9; Thu, 28 Nov 2019 13:12:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=fIuQVSLMX/1X5Z+UmWPRtwbnLZp5pF/aLYF24EqxQ1U=; b=JXtMTc7ocPvqqjhWGwL4N5XfTDj8IvidBJQBGqSy8ay6tVNlfhLMbwLCktvdaCLDUj g2jup+T3ajILxGPZ5JSmGNH83XvXWl9GqSDsdG2Ww81oJyOhmWbFWH8LUZwJmsorW++T WfD8SUnBT2p9Mc05X0HTlWYzTN+YpltX/UNH3zVBmWfmjMEmbaB0chJJxQb3KhoN5icz 7DJGUMHpKJLlqL/IfLWQ0YLKaElaF9gSOfTGPVUsVUKwMtMVlPdsZgtYd0FwS9M/wmNb Z5OOPpGvRYhfFgQvyVlmaDhNqGxNxSGw2MG2zA/8zdSuLZEo/K5nFPyBuMIl/ssCrNQS lGtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fIuQVSLMX/1X5Z+UmWPRtwbnLZp5pF/aLYF24EqxQ1U=; b=rDQkJB5Vno0PiIJwwR8D5II81bx4VC7Itu0czFIaQs3ImAwi7ja9IfP304nFNPkKOl a0hNgWZA1RL/4BNV/71hpZBi+BC42jJFWJux9B1kS42NO0voCEBOXQlc7Oz4IOPHXuhi BF5NWL/Q1mtszU32OT0cXWsHZg0srm5rTDVMumA1uHQ+kFRfC9mHQQQCUaMV3ubQcAew b1pjDND1Jc2h4+v5LLhaQ09R8XhqEREulwSpKo5HDSFaLFp/p+1Znlla9p2ui7h9RfcG g81ZHAio+x0U2dvExbDRr8rqvFkTybh5MHD6rRiiaDB7lGaOaU+tP5f5AtCuSZnZ7gOU dsmw== X-Gm-Message-State: APjAAAXGZawhaoAhvSUKfczkuYg7kTDLk3Iib2BegKARVKMpTwfXDgsa VeV7/ihlJi/pQNhcBeNr92c= X-Google-Smtp-Source: APXvYqw9YPzhZ5G3E2CqjZguJrFN2rKInPTXGrl7fNNH7DDUc1clMMTdpDu41ANSrPWnVFCvDl5mKw== X-Received: by 2002:adf:ec48:: with SMTP id w8mr2500286wrn.19.1574975539807; Thu, 28 Nov 2019 13:12:19 -0800 (PST) Received: from localhost.localdomain ([109.126.143.74]) by smtp.gmail.com with ESMTPSA id l26sm11620809wmj.48.2019.11.28.13.12.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Nov 2019 13:12:19 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] blk-mq: optimise rq sort function Date: Fri, 29 Nov 2019 00:11:53 +0300 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Check "!=" in multi-layer comparisons. The same memory usage, fewer instructions, and 2 from 4 jumps are replaced with SETcc. Note, that list_sort() doesn't differ 0 and <0. Signed-off-by: Pavel Begunkov Reviewed-by: Nikolay Borisov --- block/blk-mq.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 323c9cb28066..f32a3cfdd34e 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1668,14 +1668,10 @@ static int plug_rq_cmp(void *priv, struct list_head *a, struct list_head *b) struct request *rqa = container_of(a, struct request, queuelist); struct request *rqb = container_of(b, struct request, queuelist); - if (rqa->mq_ctx < rqb->mq_ctx) - return -1; - else if (rqa->mq_ctx > rqb->mq_ctx) - return 1; - else if (rqa->mq_hctx < rqb->mq_hctx) - return -1; - else if (rqa->mq_hctx > rqb->mq_hctx) - return 1; + if (rqa->mq_ctx != rqb->mq_ctx) + return rqa->mq_ctx > rqb->mq_ctx; + if (rqa->mq_hctx != rqb->mq_hctx) + return rqa->mq_hctx > rqb->mq_hctx; return blk_rq_pos(rqa) > blk_rq_pos(rqb); } From patchwork Thu Nov 28 21:11:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11266601 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C122015AB for ; Thu, 28 Nov 2019 21:12:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9F97D21781 for ; Thu, 28 Nov 2019 21:12:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X8vfNFSy" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726569AbfK1VMZ (ORCPT ); Thu, 28 Nov 2019 16:12:25 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:42033 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726703AbfK1VMZ (ORCPT ); Thu, 28 Nov 2019 16:12:25 -0500 Received: by mail-wr1-f66.google.com with SMTP id a15so32647563wrf.9; Thu, 28 Nov 2019 13:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Lg7L77IP4bx5QKviUaKZB+T2xlEmOdr7ElK44pBo7WA=; b=X8vfNFSy73lLY1V6LiT2zN+ER6oI601XUGtHVkH1zsQ/ca6cHz+Ma9znwE+QeDm6eJ XeQyKxNr4YYJIf632sihj6wyRB7LcgIMtBRCoBpO88NzSx8h+zSTJ3J1B0N6QLdkGozw 7y3ux5z4sflj2Xjo7XSREL2Illbc1C7bX8b0KmoZsaYogQ1ECD7ttBjWT3/9bHCMFjnX sbTVySr29u+1Pbb/k93zTan11Sbpr2J2vZ1eoq33ZRZBPBEL47jvN+sMcLstRNdF2d16 7zX9+1FKUNocz/APdluQxW0nsVEw6c6N8p/5SaXf4KHNarEDnCADy92tfCMHUuE6lki3 9MUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Lg7L77IP4bx5QKviUaKZB+T2xlEmOdr7ElK44pBo7WA=; b=FqcfrBRCZ4K5U68qkwaWP00Qfi8pG+4EFmxsSTIXBADjLZfNlO7hhdvfaFjJc1u83E GCBi3inz54yXrGUxqnB3rMAEBQgqWeivTf8dZmgBllYPQ6Vn6SvNC9fMf7YQYIb+gx/o xC+Of8DjvNJKH/9AVw2kBQZkXPL7VGpAi0jnJMGaNLxvGGqts7P9xZCdzYwYiEOyEi5R zVC255HFQGrUAdO5B/9f4PVg1fP43OI1yNI7l4vQBo+KRb7d5VXxND7sm3yIk7/8g93c ScaohYMLPiQGZ6FhJPzcore2+xcRY8u7dGSbBMDuw9kY/T1xJyo6Z/um9e/lkDSd1pPC cM9g== X-Gm-Message-State: APjAAAWYRM7pq7ZvautQcDQAhJmbmrwsQ6aVgB9jdH1bFjaSfJ5BnG9E iPrdhreFLlyqufnjno1JjoFui5gI X-Google-Smtp-Source: APXvYqy2mXbXFjqiXd4Q8h9H7IaLhmGXNdXzQ3Lb1se3s9RqLQDiXKrlRaGJ1HeXBn32u87XMHow9Q== X-Received: by 2002:a5d:6887:: with SMTP id h7mr19806568wru.397.1574975541563; Thu, 28 Nov 2019 13:12:21 -0800 (PST) Received: from localhost.localdomain ([109.126.143.74]) by smtp.gmail.com with ESMTPSA id l26sm11620809wmj.48.2019.11.28.13.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Nov 2019 13:12:21 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] list: introduce list_for_each_continue() Date: Fri, 29 Nov 2019 00:11:54 +0300 Message-Id: <2b54707c0640114d24a3e08f57fce2576e05bd27.1574974577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org As other *continue() helpers, this continues iteration from a given position. Signed-off-by: Pavel Begunkov --- include/linux/list.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/list.h b/include/linux/list.h index 85c92555e31f..3c391bbd03c3 100644 --- a/include/linux/list.h +++ b/include/linux/list.h @@ -538,6 +538,16 @@ static inline void list_splice_tail_init(struct list_head *list, #define list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next) +/** + * list_for_each_continue - continue iteration over a list + * @pos: the &struct list_head to use as a loop cursor. + * @head: the head for your list. + * + * Continue to iterate over a list, continuing after the current position. + */ +#define list_for_each_continue(pos, head) \ + for (pos = pos->next; pos != (head); pos = pos->next) + /** * list_for_each_prev - iterate over a list backwards * @pos: the &struct list_head to use as a loop cursor. From patchwork Thu Nov 28 21:11:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11266599 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78A32138C for ; Thu, 28 Nov 2019 21:12:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5694C217BA for ; Thu, 28 Nov 2019 21:12:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oScINBvj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726856AbfK1VM3 (ORCPT ); Thu, 28 Nov 2019 16:12:29 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:39429 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726729AbfK1VM0 (ORCPT ); Thu, 28 Nov 2019 16:12:26 -0500 Received: by mail-wr1-f66.google.com with SMTP id y11so29515744wrt.6; Thu, 28 Nov 2019 13:12:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=OJIbPjHQmKLscXvEELpJQJK3dKB4Msg/Vy9NKsHE9co=; b=oScINBvjs475mEFUHeEgMihefU3ksmRy37nar6Vl5ofyc3H8GsvODLgyv7EbjbTTAu C7qJftZnPwY8JsCeJ54p3BT0xdQc0lP3hqzPV1Sb1IgFn6s+wrjbpdUIHs8+VERyREG/ YQxYsQ5Gvlv9EPlyx425Fx8oVieuiTLqKwP/nkg59M4nixZKMdsN+MnEoSGSN2Q2GG5x BXw708vtdD/OMP5J0g4wpWghf9ueR4unYLV1QdQRcpEPZWtG3jiAAZyasSBXa3y4Ibq2 9eLm8uiT0aAXlBX4TWK6dTAHADdQYzpgLPJY5TTKsWWfAaiFEU+Eypxy8NosMsFMKrNj 9OZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OJIbPjHQmKLscXvEELpJQJK3dKB4Msg/Vy9NKsHE9co=; b=e75Fbj1SkZ2y6noyFotdFmY9w6x3EEBWeVRm+iG18GnHBI4w2W6jtjbaZe6DFFKutt uQrlVRwXVPIZVZa7i3ttugackS3x9WnHn0ofNwJxi3ojDuuzOHeXMWrgBO5CwP8HH+tu LhiYTNsp5C5SS5yZk4bPciJH3rl05gurDVB0H5TT4vUw9NYO9DnS/iA2xVE/UnEixCvB goP1uYHQyowDrsqZijS2K6IOuajTSMO6t8LGqb52ud16f5N+JTiLAgfPV8KRwy5ERCm1 CRrF6Z/L+YcPN0su8+LjnIujcGUCtiq0j/XnkItWlNIp5UdXs/0BXmQBKavS1STgfhH2 Y6cg== X-Gm-Message-State: APjAAAXIQKFplENd94lggtxU6ryjPxoCtrvDGTDchaAYdkzLLLjwd4ii I7Gu91hrNfpr8D886XjrxhU= X-Google-Smtp-Source: APXvYqywCK1v1R526VWzc1R7MqbdJ8g7i/DSrkXD9DaShCzlSMYNArDB3n7VzKCAm4r2B0I3W65amQ== X-Received: by 2002:adf:dc02:: with SMTP id t2mr8725230wri.39.1574975543344; Thu, 28 Nov 2019 13:12:23 -0800 (PST) Received: from localhost.localdomain ([109.126.143.74]) by smtp.gmail.com with ESMTPSA id l26sm11620809wmj.48.2019.11.28.13.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Nov 2019 13:12:22 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] blk-mq: optimise blk_mq_flush_plug_list() Date: Fri, 29 Nov 2019 00:11:55 +0300 Message-Id: <021a4aad9f3f9ee13661006e945d642eafd69d9d.1574974577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Instead of using list_del_init() in a loop, that generates a lot of unnecessary memory read/writes, iterate from the first request of a batch and cut out a sublist with list_cut_before(). Apart from removing the list node initialisation part, this is more register-friendly, and the assembly uses the stack less intensively. list_empty() at the beginning is done with hope, that the compiler can optimise out the same check in the following list_splice_init(). Signed-off-by: Pavel Begunkov --- block/blk-mq.c | 57 +++++++++++++++++--------------------------------- 1 file changed, 19 insertions(+), 38 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f32a3cfdd34e..3c71d52b6401 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1678,14 +1678,10 @@ static int plug_rq_cmp(void *priv, struct list_head *a, struct list_head *b) void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { - struct blk_mq_hw_ctx *this_hctx; - struct blk_mq_ctx *this_ctx; - struct request_queue *this_q; - struct request *rq; LIST_HEAD(list); - LIST_HEAD(rq_list); - unsigned int depth; + if (list_empty(&plug->mq_list)) + return; list_splice_init(&plug->mq_list, &list); if (plug->rq_count > 2 && plug->multiple_queues) @@ -1693,42 +1689,27 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) plug->rq_count = 0; - this_q = NULL; - this_hctx = NULL; - this_ctx = NULL; - depth = 0; - - while (!list_empty(&list)) { - rq = list_entry_rq(list.next); - list_del_init(&rq->queuelist); - BUG_ON(!rq->q); - if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) { - if (this_hctx) { - trace_block_unplug(this_q, depth, !from_schedule); - blk_mq_sched_insert_requests(this_hctx, this_ctx, - &rq_list, - from_schedule); - } - - this_q = rq->q; - this_ctx = rq->mq_ctx; - this_hctx = rq->mq_hctx; - depth = 0; + do { + struct list_head rq_list; + struct request *rq, *head_rq = list_entry_rq(list.next); + struct list_head *pos = &head_rq->queuelist; /* skip first */ + struct blk_mq_hw_ctx *this_hctx = head_rq->mq_hctx; + struct blk_mq_ctx *this_ctx = head_rq->mq_ctx; + unsigned int depth = 1; + + list_for_each_continue(pos, &list) { + rq = list_entry_rq(pos); + BUG_ON(!rq->q); + if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) + break; + depth++; } - depth++; - list_add_tail(&rq->queuelist, &rq_list); - } - - /* - * If 'this_hctx' is set, we know we have entries to complete - * on 'rq_list'. Do those. - */ - if (this_hctx) { - trace_block_unplug(this_q, depth, !from_schedule); + list_cut_before(&rq_list, &list, pos); + trace_block_unplug(head_rq->q, depth, !from_schedule); blk_mq_sched_insert_requests(this_hctx, this_ctx, &rq_list, from_schedule); - } + } while(!list_empty(&list)); } static void blk_mq_bio_to_request(struct request *rq, struct bio *bio,