From patchwork Thu Feb 16 21:22:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13143983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B47CCC636CC for ; Thu, 16 Feb 2023 21:22:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EEFA6B0074; Thu, 16 Feb 2023 16:22:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 39EC76B0075; Thu, 16 Feb 2023 16:22:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28DD16B0078; Thu, 16 Feb 2023 16:22:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1987F6B0074 for ; Thu, 16 Feb 2023 16:22:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DB173C0A8E for ; Thu, 16 Feb 2023 21:22:54 +0000 (UTC) X-FDA: 80474429868.20.F1411E7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 15F8740016 for ; Thu, 16 Feb 2023 21:22:52 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Z7avsjaH; spf=pass (imf04.hostedemail.com: domain of mpatocka@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mpatocka@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676582573; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Wf4v/tNzjT2VRs1P16c/6f29cG3mDdqrWzEZUinry/Q=; b=dAChHZaUZFFw2Lo3rvkX52o99ZiFYGu/4Si5F13EKmGmDXL0FiqWKp4u90fLWdVRXQvT1s 0fCQEtbQvIldjUIQOpTvyUZqkkysVGzGRdA8nE5pamQhvCZm2TqaSNq4IPBXtUVvp7OD7T j6bcNGH1z5GAP/CQ3WrarGyz1KtolfE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Z7avsjaH; spf=pass (imf04.hostedemail.com: domain of mpatocka@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mpatocka@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676582573; a=rsa-sha256; cv=none; b=AvOulK4Xo8Iy0Xs2tQ+hJ8J+ExmMAyvk6wyIpco3HZzaexW9AI9iDDWU+xCaSTrAUjzlkU hG9/fA152VNhUQ9F1mbR2ygrhzXbJPAotZa03aw9nIU5O0BAurQQiyWiSntRVfn5J/iOa7 eIuDY3UXGLUqYmQvZzsHURzlQe5dSdk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676582572; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=Wf4v/tNzjT2VRs1P16c/6f29cG3mDdqrWzEZUinry/Q=; b=Z7avsjaHz0Ipj25c3fZKiPzo2mohV+difgQHlePZ7Eajmfbyp/tEWMKBh0qMowOUQMnKG1 fDfkH9ur9ZAd3IL8r3ntMVJPJ5DaE+LAIxyoUHRdYyKXDuacYVcYnfSRq4yf/BbfMTIa0f 1fIwJmT4X3owoL1Km/qE3qebQJSgYpg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-zENzyEZKM0KIUmysLL-VZA-1; Thu, 16 Feb 2023 16:22:47 -0500 X-MC-Unique: zENzyEZKM0KIUmysLL-VZA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 952611C0758A; Thu, 16 Feb 2023 21:22:46 +0000 (UTC) Received: from file01.intranet.prod.int.rdu2.redhat.com (file01.intranet.prod.int.rdu2.redhat.com [10.11.5.7]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 86A0440CF8E4; Thu, 16 Feb 2023 21:22:46 +0000 (UTC) Received: from file01.intranet.prod.int.rdu2.redhat.com (localhost [127.0.0.1]) by file01.intranet.prod.int.rdu2.redhat.com (8.14.4/8.14.4) with ESMTP id 31GLMkxg007334; Thu, 16 Feb 2023 16:22:46 -0500 Received: from localhost (mpatocka@localhost) by file01.intranet.prod.int.rdu2.redhat.com (8.14.4/8.14.4/Submit) with ESMTP id 31GLMksq007330; Thu, 16 Feb 2023 16:22:46 -0500 X-Authentication-Warning: file01.intranet.prod.int.rdu2.redhat.com: mpatocka owned process doing -bs Date: Thu, 16 Feb 2023 16:22:46 -0500 (EST) From: Mikulas Patocka X-X-Sender: mpatocka@file01.intranet.prod.int.rdu2.redhat.com To: Matthew Wilcox cc: snitzer@kernel.org, Yang Shi , mgorman@techsingularity.net, agk@redhat.com, dm-devel@redhat.com, akpm@linux-foundation.org, linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2] dm-crypt: allocate compound pages if possible Message-ID: User-Agent: Alpine 2.21 (LRH 202 2017-01-01) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 15F8740016 X-Stat-Signature: n4jrxtihwshosijx3xn5ydw7swzuzfkd X-HE-Tag: 1676582572-831397 X-HE-Meta: U2FsdGVkX1+6TlY9odmsbqX/bgfhsFzfv4Mxx2LgNv81e3E+Bm12TW61RzGVp52UNGCn6XB1GVZF9LwoRFfIDrGuwPpGfO8GR9U5ws7dNiwtBZfECwSCx8VHtxwrSXEss8joufzeyClY560dt59IJOSbJUpcyhffO8l6CyNbIpC2b/H4cTvdeHLu3zJigWTbFnVDNdg8CgggNeFLq6FFqvOgrquBl8jbjn2wTCPKDYgAMVz/4eOx4iMpcgGiyTe6RsBtsTWAs7fSRhgBsic7U8i6e7RVX/G1TnfaIfuXMqzIwP63jW6stDC/VvBGTAF0w+gl4xQcL95BTJ9awUIgkoKaKkuOZrkqEWvItboFPlYR2YtiQYg/pOQ7aRELgvyYTo+7XbIYhdCqK8GOfJAkuhxTSN+vyb1MDGdtTwunXjfiwOnYdVRGjPwI0F3v+gjO7OwPuqcFHzXm8aW+Ul048n92QC4sJdEHhGATidN23B2gtPAB4ZgIrwJgE7Qac56xjOJ8GHwfVDzUFz2Jt76O6sQfPcCXF5VaWLSXUl6XbOym4X+5gEwg6vM/dlpeFDd4UUzYtKnvYt5QG6g7OLLds1z9tepvDAJ89cYubpiffC6NJTX0vE5lZSfXR71HiXNDsaVOgJKnBOJVfZsJ9kRoEAS1F4WgEcOf/M+T86PF9AcgRLRKBKfoz5QhQ65LizQPPOBoH/fv/Mb57Sa0CevqlVr2eA9nWKIC01sqfGkmY1/nzFCgUNbd2RtSJLNHzdBX1qBv0GASxfos7cla6fQJ07Xgoy8mJSlxWwftQbvqL3TlE/MLEpBl7vWfzzL0CR3f1Y0YrucO+yMBUlkV6jdXfuA3SD/fIx2kNB+UySHVf16gu8axjYw0GOY8Ww09xxTRVgxKoEkqK0cjrGJ7sLyziMGFxjSxm5luWyFUY5l0SSiZnhc7icTB9EiSJmT0btyutgy4u5fwSANaKwtl57G KJcwqyH4 RBzj0D0hbYor/RXibC/q2bmvSRJdc4FBGDqfMY1xAkIOH6QcqxtfC4s6jNhqWG+/a7ofk5KrHcfQf1iUCOUIWC2LJ38RNuvIOgztKCFzmGYju4Cez0D4b1kMktbPb6Le83akek0gISsJV8oNkDshNCllb2g5m2YKq6CPOJUoi6wqip1nNj+8MLGt8Lt7/UxdQ/TdYmn8PIkXydM+Gcf0SR2I/pNHoHOxjf1by7rVhTn3uQXdpogES6YH7pXfRAkD6ER/L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It was reported that allocating pages for the write buffer in dm-crypt causes measurable overhead [1]. This patch changes dm-crypt to allocate compound pages if they are available. If not, we fall back to the mempool. [1] https://listman.redhat.com/archives/dm-devel/2023-February/053284.html Signed-off-by: Mikulas Patocka --- drivers/md/dm-crypt.c | 48 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 15 deletions(-) Index: linux-2.6/drivers/md/dm-crypt.c =================================================================== --- linux-2.6.orig/drivers/md/dm-crypt.c 2023-02-16 20:40:15.000000000 +0100 +++ linux-2.6/drivers/md/dm-crypt.c 2023-02-16 21:56:34.000000000 +0100 @@ -1657,6 +1657,9 @@ static void crypt_free_buffer_pages(stru * In order to not degrade performance with excessive locking, we try * non-blocking allocations without a mutex first but on failure we fallback * to blocking allocations with a mutex. + * + * In order to reduce allocation overhead, we try to allocate compound pages in + * the first pass. If they are not available, we fall back to the mempool. */ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned size) { @@ -1664,8 +1667,8 @@ static struct bio *crypt_alloc_buffer(st struct bio *clone; unsigned int nr_iovecs = (size + PAGE_SIZE - 1) >> PAGE_SHIFT; gfp_t gfp_mask = GFP_NOWAIT | __GFP_HIGHMEM; - unsigned i, len, remaining_size; - struct page *page; + unsigned remaining_size; + unsigned order = MAX_ORDER - 1; retry: if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) @@ -1678,20 +1681,34 @@ retry: remaining_size = size; - for (i = 0; i < nr_iovecs; i++) { - page = mempool_alloc(&cc->page_pool, gfp_mask); - if (!page) { + while (remaining_size) { + struct page *pages; + unsigned size_to_add; + unsigned remaining_order = __fls((remaining_size + PAGE_SIZE - 1) >> PAGE_SHIFT); + order = min(order, remaining_order); + + while (order > 0) { + pages = alloc_pages(gfp_mask + | __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN | __GFP_COMP, + order); + if (likely(pages != NULL)) + goto have_pages; + order--; + } + + pages = mempool_alloc(&cc->page_pool, gfp_mask); + if (!pages) { crypt_free_buffer_pages(cc, clone); bio_put(clone); gfp_mask |= __GFP_DIRECT_RECLAIM; + order = 0; goto retry; } - len = (remaining_size > PAGE_SIZE) ? PAGE_SIZE : remaining_size; - - bio_add_page(clone, page, len, 0); - - remaining_size -= len; +have_pages: + size_to_add = min((unsigned)PAGE_SIZE << order, remaining_size); + bio_add_page(clone, pages, size_to_add, 0); + remaining_size -= size_to_add; } /* Allocate space for integrity tags */ @@ -1709,12 +1726,13 @@ retry: static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone) { - struct bio_vec *bv; - struct bvec_iter_all iter_all; + struct folio_iter fi; - bio_for_each_segment_all(bv, clone, iter_all) { - BUG_ON(!bv->bv_page); - mempool_free(bv->bv_page, &cc->page_pool); + bio_for_each_folio_all(fi, clone) { + if (folio_test_large(fi.folio)) + folio_put(fi.folio); + else + mempool_free(&fi.folio->page, &cc->page_pool); } }