From patchwork Sat Aug 6 15:20:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12938066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0A17C25B0C for ; Sat, 6 Aug 2022 15:20:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230326AbiHFPUK (ORCPT ); Sat, 6 Aug 2022 11:20:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231196AbiHFPUK (ORCPT ); Sat, 6 Aug 2022 11:20:10 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D9A511A12 for ; Sat, 6 Aug 2022 08:20:09 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id v16-20020a17090abb9000b001f25244c65dso10777369pjr.2 for ; Sat, 06 Aug 2022 08:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=KQkxJ7Xt9ACG3CLAZzFZxHFZSW93HMroBHCMM5ScDhY=; b=PpOGzcizOZkg0RFbAfq/VBwaHTwNveQxnWvIJZX5jiO4YFP8V7TcnI3N+Z2MEq6isr ElX9mDWtl5+mDewuU3MpARtITiJNl1UvYZpgihG2/g+A/0bfIwldZaXAnJDU6MhO9xq1 O28LnFdujA+s7nVTaW4GgCtnS3/TM3f8DZREJeegD+yXeyGpdAcaz0Tbu0Q8Rrr2P8R9 0yvzJtGgVRMUiLwLqMw4hQofm+eh6n94b/VvBO/QSQdNa1g/MuADCST9LbRtU9panUDW BksTxRIIi50RRGPIkO0HbBIHcRs4gIS165Kr6V6P6tvljM+9zM9YxCGmn4SSpeEAp/0s Bgsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=KQkxJ7Xt9ACG3CLAZzFZxHFZSW93HMroBHCMM5ScDhY=; b=63krSit32QCgcVPOKkqqjU34d9d19XYBASbWjt/2u14JFdTLJGyuMJhUrKs7EmdpnI UMhB9ed+j3mNIv3UUFeLaXuK5XVQvfd/I/tsJdXWJ4STW37/w+NjGKRml6MM/uwPGwNP TV2QcdU3yuoSHkSU3mlXZgj40D1vUtXLGct2Pl+PO7fshK1fqs5GvGBPk4AdzuyEBkh1 ETPKlq8ehrMwuE/5C7wJLT6dmNEYrKkYPDfqu+J9/U86UiYrplGlg5ik+LnIUaWnnlb6 T6mh1rbvCTbhcpuv6GNcVdN7DOUq/m/IeYdX2AK4A8NgzqeBLNNx3qPg6WwMd4EcxUfj Yxlg== X-Gm-Message-State: ACgBeo0ZzJUlRNHobAPFzYJrUuLzrb4luFCTbLVpWWGk9+hmx3FmCh6l ocGIRq6ITgmFChs0uIOhFtu2zkj4zJYJAg== X-Google-Smtp-Source: AA6agR4Zyu06NMsedS3qStSFFnwVOul4lTIFw2LFfPB2a43FI8JEvX61ZP2eJ2NZnK0CMl4daxdZgw== X-Received: by 2002:a17:90b:4a4e:b0:1f5:431c:54f8 with SMTP id lb14-20020a17090b4a4e00b001f5431c54f8mr20943801pjb.161.1659799208925; Sat, 06 Aug 2022 08:20:08 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id h6-20020a170902f7c600b0016d1f6d1b99sm5158661plw.49.2022.08.06.08.20.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 08:20:08 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: joshi.k@samsung.com, kbusch@kernel.org, Jens Axboe Subject: [PATCH 1/3] block: shrink rq_map_data a bit Date: Sat, 6 Aug 2022 09:20:02 -0600 Message-Id: <20220806152004.382170-2-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220806152004.382170-1-axboe@kernel.dk> References: <20220806152004.382170-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We don't need full ints for several of these members. Change the page_order and nr_entries to unsigned shorts, and the true/false from_user and null_mapped to booleans. This shrinks the struct from 32 to 24 bytes on 64-bit archs. Signed-off-by: Jens Axboe Reviewed-by: Chaitanya Kulkarni --- block/blk-map.c | 2 +- include/linux/blk-mq.h | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index df8b066cd548..4043c5809cd4 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -158,7 +158,7 @@ static int bio_copy_user_iov(struct request *rq, struct rq_map_data *map_data, bio_init(bio, NULL, bio->bi_inline_vecs, nr_pages, req_op(rq)); if (map_data) { - nr_pages = 1 << map_data->page_order; + nr_pages = 1U << map_data->page_order; i = map_data->offset / PAGE_SIZE; } while (len) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index effee1dc715a..1f21590439d4 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -964,11 +964,11 @@ blk_status_t blk_insert_cloned_request(struct request *rq); struct rq_map_data { struct page **pages; - int page_order; - int nr_entries; unsigned long offset; - int null_mapped; - int from_user; + unsigned short page_order; + unsigned short nr_entries; + bool null_mapped; + bool from_user; }; int blk_rq_map_user(struct request_queue *, struct request *, From patchwork Sat Aug 6 15:20:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12938067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85DA5C19F2D for ; Sat, 6 Aug 2022 15:20:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231302AbiHFPUN (ORCPT ); Sat, 6 Aug 2022 11:20:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231196AbiHFPUM (ORCPT ); Sat, 6 Aug 2022 11:20:12 -0400 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96A6BD5 for ; Sat, 6 Aug 2022 08:20:10 -0700 (PDT) Received: by mail-pf1-x433.google.com with SMTP id u133so4615641pfc.10 for ; Sat, 06 Aug 2022 08:20:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Is5g6lb6jj60GAOGQvztYldquLbOFIsb6gndMPsDvCc=; b=sO3dKXLjxpIn9RdifvvGk/uZk0/RRqOTSVDvgRUOJ4a5OWL30gQdJVFvqqTiNLHgyW VL5U38tIyPRdqlQXRypNjxw2HtiQVkpqUP123TY1K8bzgaWIoXj/GaWGdN9xIBRjBPaV /Tk3B31Oz1VsA2vhV5VdeahH4HnX+VNpSQHDvhN8GnhGtGRLdGGm1q0cMQAvhfPnQfT9 arbXf7mvGlildykuwYS6zfj2pT3hyrkxg0mH1gcnCjsW270sGP6WX/r8Ge9ck7KXCA5c WC1WxfkF7za3W0hErq6l7NnQq8gP2IcWo2cO7njz5ytreh5k6GA5R4fLgFmnV08myiUt hMdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Is5g6lb6jj60GAOGQvztYldquLbOFIsb6gndMPsDvCc=; b=HFd5nBpnssGiW7qfc0JjjxbGXpgKaZenu9rlGK3Y6kXoabaPqK+g4T9IrRYJ+XnW/V rOebJvyN6MiYT1pYL+ABIfHJJNpxI1l/lKGn2ENdyqza+sxWaF0f3CvAE8qYOXzyYFSL N4QBhDL83bW+GonuAP4cY9GVCb9DJkr5OPu7J+WhnREgR/39kaWdZru+HLJtoSKqaIiB HDQV6dPtG/WRYP6DrdgrdRLxGcbPgLcDvDyxH/oiC6TcwFufMKSNl0ztKSDnQdgczkbm cwn+ETCElAszwGjbKlbHhVkiQNQYg91fQEro6vw/w5+O5WjBYuq2xAeR5n3M99DIPWFe 36xg== X-Gm-Message-State: ACgBeo2bUFWOkpg8gCaeb4GtSWy0qpAWhCAiGryvmD0RIVmZUJmksPsB fcrBbpe6o6756tzhC5xmImxUQzWl/MdEbg== X-Google-Smtp-Source: AA6agR7HJzNOrYBhoT1aLsMsaWj5XkSpXpzxmOlKhG8ioFFEc4F2Zm/mDN3jZjCASytKKUI2Q02wrQ== X-Received: by 2002:a05:6a00:c96:b0:52e:979c:dd63 with SMTP id a22-20020a056a000c9600b0052e979cdd63mr10713198pfv.50.1659799209858; Sat, 06 Aug 2022 08:20:09 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id h6-20020a170902f7c600b0016d1f6d1b99sm5158661plw.49.2022.08.06.08.20.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 08:20:09 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: joshi.k@samsung.com, kbusch@kernel.org, Jens Axboe Subject: [PATCH 2/3] block: enable bio caching use for passthru IO Date: Sat, 6 Aug 2022 09:20:03 -0600 Message-Id: <20220806152004.382170-3-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220806152004.382170-1-axboe@kernel.dk> References: <20220806152004.382170-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org bdev based polled O_DIRECT is currently quite a bit faster than passthru on the same device, and one of the reaons is that we're not able to use the bio caching for passthru IO. If REQ_POLLED is set on the request, use the fs bio set for grabbing a bio from the caches, if available. This saves 5-6% of CPU over head for polled passthru IO. Signed-off-by: Jens Axboe Reviewed-by: Chaitanya Kulkarni --- block/blk-map.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index 4043c5809cd4..5da03f2614eb 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -231,6 +231,16 @@ static int bio_copy_user_iov(struct request *rq, struct rq_map_data *map_data, return ret; } +static void bio_map_put(struct bio *bio) +{ + if (bio->bi_opf & REQ_ALLOC_CACHE) { + bio_put(bio); + } else { + bio_uninit(bio); + kfree(bio); + } +} + static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, gfp_t gfp_mask) { @@ -243,10 +253,19 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!iov_iter_count(iter)) return -EINVAL; - bio = bio_kmalloc(nr_vecs, gfp_mask); - if (!bio) - return -ENOMEM; - bio_init(bio, NULL, bio->bi_inline_vecs, nr_vecs, req_op(rq)); + if (rq->cmd_flags & REQ_POLLED) { + blk_opf_t opf = rq->cmd_flags | REQ_ALLOC_CACHE; + + bio = bio_alloc_bioset(NULL, nr_vecs, opf, gfp_mask, + &fs_bio_set); + if (!bio) + return -ENOMEM; + } else { + bio = bio_kmalloc(nr_vecs, gfp_mask); + if (!bio) + return -ENOMEM; + bio_init(bio, NULL, bio->bi_inline_vecs, nr_vecs, req_op(rq)); + } while (iov_iter_count(iter)) { struct page **pages; @@ -304,8 +323,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, out_unmap: bio_release_pages(bio, false); - bio_uninit(bio); - kfree(bio); + bio_map_put(bio); return ret; } @@ -610,8 +628,7 @@ int blk_rq_unmap_user(struct bio *bio) next_bio = bio; bio = bio->bi_next; - bio_uninit(next_bio); - kfree(next_bio); + bio_map_put(next_bio); } return ret; From patchwork Sat Aug 6 15:20:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12938068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BCACC25B0D for ; Sat, 6 Aug 2022 15:20:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231196AbiHFPUN (ORCPT ); Sat, 6 Aug 2022 11:20:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231245AbiHFPUM (ORCPT ); Sat, 6 Aug 2022 11:20:12 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 679BE27C for ; Sat, 6 Aug 2022 08:20:11 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id t2so5043927ply.2 for ; Sat, 06 Aug 2022 08:20:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=x+rg3l5+RTbusQB0ppV1F1CJZjLcHIeSUkGmaVShyQg=; b=amXbFbvlCtFJUaDcAuJ9wxOjQ5Rgd0ElkRczUWjoM1lTm/eFGGlqeypxjX83hfOSrv Sl5J83BI2sirvZ/oZJaGksPo/x4airPZYQnnROQEloWoG/GEvOhjJmCWdUZqKG94XXHm no9wLiwAqTh/W0q84w+QtEifEYC0TcvuSPKjXpjbOwQ9Fum6wDrKUOKvqs6pnUEG6vRi pUQAXF+rY23fEDMnjfndYc5bzRs3qZ7sXrOZ1Y8iStKb56ctkdnPH2hVLlllpu+i0DD5 ivg5pie76Go1U14B9+J+EQYoO9Po2v18+qHPRStzTmguz2cgd/9DlBbX+OVbeavJ3C6x SHAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=x+rg3l5+RTbusQB0ppV1F1CJZjLcHIeSUkGmaVShyQg=; b=I4kwyygvY5TI3NdqA53xlclA5tcWIOCOzKwCF0Z7787IFHiZXjO7xcJaN8Po0l5cQt 9mW9lQgRx4DHcq8UijxwNbt/CiVZaTP7fhqVpNr1e+xq7mry44Py2bGqrR/Z3oVnQGau OiL26fddPMK6yXkTrl8xqC4PA+k3Wg5gFxPzKtrsFyC19MdtOyf53yat6fs1XckkZiNQ MDTezP9xfSJBXeymSHmLD7wPFx15i66mBKJKwk1OLEelf243Kca/LHYnCyveljyuf9T0 dJVGG/XLarnNL4/wJNiTCGs+fzJG0lfeJk7rsQQXtxlyfluKjpUmM+9C2eDuq4YDyjf5 mXEA== X-Gm-Message-State: ACgBeo0pRiM26vbpQjJuanQ8n4QgJIqnNjHkIpuymAo+u0kEIH/fwY6W WYzf0IwNSwddhx+ZpEJTN6RXp2XJtoQG0A== X-Google-Smtp-Source: AA6agR69ElA9pX1wqKT3t6N3Lg0yicQ9k3YjsVneHyXjDvCfRO5ANknujnzjjMIf/E0sb3ZFV6WM9g== X-Received: by 2002:a17:90a:150:b0:1f3:1dc7:c9f0 with SMTP id z16-20020a17090a015000b001f31dc7c9f0mr12773936pje.237.1659799210813; Sat, 06 Aug 2022 08:20:10 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id h6-20020a170902f7c600b0016d1f6d1b99sm5158661plw.49.2022.08.06.08.20.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 08:20:10 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org Cc: joshi.k@samsung.com, kbusch@kernel.org, Jens Axboe Subject: [PATCH 3/3] block: use on-stack page vec for <= UIO_FASTIOV Date: Sat, 6 Aug 2022 09:20:04 -0600 Message-Id: <20220806152004.382170-4-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220806152004.382170-1-axboe@kernel.dk> References: <20220806152004.382170-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Avoid a kmalloc+kfree for each page array, if we only have a few pages that are mapped. An alloc+free for each IO is quite expensive, and it's pretty pointless if we're only dealing with 1 or a few vecs. Use UIO_FASTIOV like we do in other spots to set a sane limit for how big of an IO we want to avoid allocations for. Signed-off-by: Jens Axboe Reviewed-by: Chaitanya Kulkarni --- block/blk-map.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index 5da03f2614eb..d0ff80a9902e 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -268,12 +268,19 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, } while (iov_iter_count(iter)) { - struct page **pages; + struct page **pages, *stack_pages[UIO_FASTIOV]; ssize_t bytes; size_t offs, added = 0; int npages; - bytes = iov_iter_get_pages_alloc(iter, &pages, LONG_MAX, &offs); + if (nr_vecs < ARRAY_SIZE(stack_pages)) { + pages = stack_pages; + bytes = iov_iter_get_pages(iter, pages, LONG_MAX, + nr_vecs, &offs); + } else { + bytes = iov_iter_get_pages_alloc(iter, &pages, LONG_MAX, + &offs); + } if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -310,7 +317,8 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, */ while (j < npages) put_page(pages[j++]); - kvfree(pages); + if (pages != stack_pages) + kvfree(pages); /* couldn't stuff something into bio? */ if (bytes) break;