From patchwork Wed Nov 15 15:49:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13457017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FFD0C54FB9 for ; Wed, 15 Nov 2023 15:50:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 850DB8001F; Wed, 15 Nov 2023 10:50:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AF6880018; Wed, 15 Nov 2023 10:50:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DE6A8001F; Wed, 15 Nov 2023 10:50:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4AE0380018 for ; Wed, 15 Nov 2023 10:50:29 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 00E90404F1 for ; Wed, 15 Nov 2023 15:50:28 +0000 (UTC) X-FDA: 81460625778.15.C085ECE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 3455C1C0017 for ; Wed, 15 Nov 2023 15:50:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eP9G4ntn; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700063427; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SHVN5dprl5SXzKPIzGJgk8i0Q8IZLJJ1is7dtiWAZyQ=; b=pcJzwlm43Z+8jvSucwj3pwrX/bfKMb5edY5bM5xS86BGUSrDyJhZ74oNFy/sMJJyHS9qLP H7WO/aEy2FJaJIqNKb7aVGV6zMbPMVdQ6EsPUEuC/V0dEuHW5Gvbkf3se1NbIRLSlMnHJt CuT7fcisJQntPuUk9knTOoZiPSfoDlM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700063427; a=rsa-sha256; cv=none; b=HXtO+/i2f0FlHWZuseVmK5rJ8911chYz+egP2/ZN6Aq2ps5G+ZqhOITvQ14qoFEjhEN4vP lAlu+Js8F46Z8n6qaEncx1vtC7rQfMTcX83mpSjdcPNnuoW/lV6xzTA7ExmRSdq2A+84aW zLqUY1X114n+ZcW+KaOJ1eYkQJ89ozY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eP9G4ntn; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700063426; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SHVN5dprl5SXzKPIzGJgk8i0Q8IZLJJ1is7dtiWAZyQ=; b=eP9G4ntnT1QWScbZU/njXxc/bytFl46AS/AlMvI2L/CP8MwXfOQH9ljLJXaB5FLBOsV7om oaNZBBVMsuDH/aAoqejpe9TWPwEpMfa/Rj8GfBlDTRxaAZEI1vPH/1ClqGn71G3pa2e0yU 95ptXZqPXpwgyQ9v6OOJxRz7IkWBEA0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-630-9oNthIjAOCex8BE_jP80JQ-1; Wed, 15 Nov 2023 10:50:23 -0500 X-MC-Unique: 9oNthIjAOCex8BE_jP80JQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F270038143B7; Wed, 15 Nov 2023 15:50:21 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76E1240C6EB9; Wed, 15 Nov 2023 15:50:18 +0000 (UTC) From: David Howells To: Christian Brauner Cc: David Howells , Jens Axboe , Al Viro , Linus Torvalds , Christoph Hellwig , David Laight , Matthew Wilcox , Brendan Higgins , David Gow , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, Christian Brauner , David Hildenbrand , John Hubbard Subject: [PATCH v3 08/10] iov_iter: Add benchmarking kunit tests Date: Wed, 15 Nov 2023 15:49:44 +0000 Message-ID: <20231115154946.3933808-9-dhowells@redhat.com> In-Reply-To: <20231115154946.3933808-1-dhowells@redhat.com> References: <20231115154946.3933808-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Rspamd-Queue-Id: 3455C1C0017 X-Rspam-User: X-Stat-Signature: 4yabjf19ftkuza4u8796dwd8jm1839hu X-Rspamd-Server: rspam03 X-HE-Tag: 1700063426-660110 X-HE-Meta: U2FsdGVkX197eTJfJE1QpsF2hoC15EngOfYjX5lN9oMAJtSy18F17IMMbpWQBdBceDPWrAAxsDWzh0O2TX/Q3WcDx/xGhQ7wm0f9vXtllrEYoGuWy2KFl7I+bdC5n5XBIR5rYQOC03dBdVQP/yJgT8rxqdVy8zGIqsb9TORW0de1BK/ELMXwfNa67JsUnxVXcYqzsUuLUE3bjpECUNgVwFQjs2biH6bk0MwIdzNKJOyRuUnafuIPcqrU2k5ZlDTzMmr0b7PEglkHjs30VUsMZOnV3iWQB3UTD8GYSf7wNovVcBNDCUpS4all9rsWdRAglTLhJgz8ua6Bm4F0faKTBowJQZ57QXMVcp6yH0oGhSn0pHkhkdGOg0e9HiJt1uEfNvOsnkJb7TAM5MgyN9oE1qWrcpsTo728Y9cEsycRDZC6Ca/g6GuD85dIvcmaW4G1dfkDzHQGyfEoIXSQ8QyxbCFJcTM5C4TfQa4pg/Xy1zhuVyrPYme+/55iN82gQNTbJfDnXj9dFr2arB78IAcA8Wt83uOCjAdN5w0wGNwUZUfPeaZ357D6Jo6fnECiZXOKWFSoMwxMWyuEc1PYIv9Soo9z8ExJgeGNJ5V8B5kx/2OYCGf0pSr78oz73wYx65aU+yxZAxjBOXqrw9iXjlB4P9G+ONc0+gauNZlMMc4cUsFJDnv7bn6rwpTlAbdViGQo6lanXVXgmgoD358387OxDft50xaL72etxb/idnsqWmU5fPnvQXY6Bia1Bc1PW6w6x1aq1is5k0gDLm1UJUHbQrDy3kPabNiKbgH/rydV+oVQ259TXKzZilE6KHahzpGvMormA0144/R9cZpe6kXqXpHhmYae1jGqmTYlwrG89+1vU78PcVni0FLXEASaqBwIhlnC5higrTrBd48j3SLXLm8eNTQXDyIw53BoAeaqjRXSah8OH1E3QrbHMuK2A0L9+cXypANStWZiwn9an64 fHuV6QaT B2yfq+eaSJePXdyGtHUV6NU6kkMaIwSX9iATLaIHYvMMuCsQg6LjRexxPf8JLwyxdv6CPJESpz2l3COlRQPAbkb4DG/qhT1jAFMlA8P/KPx8c5GFUThLKGqLtYc3hvCGGm7/T9XlXYgqH9WdmCYsoG5v6DwAYaILhYRlBcok7hC2IMI8o80pVe3/lfRkwxtyIc/NaDFl0e97gal51nLSik+UGNWisKavSwysR0lsW2arJJEzi7R2mRvivowu71m7jhZoWyrD/eukttd9vkWiLE3nv19oOSxzGx6kVjwqYGQO4U9xcAsoIicmh+ZxjCrbDCthYsdRWc+xxVAGpA9UChh/g9OZ8s2qRL6JuHixxOvXkobDLCx0CD4Y1hRa+057Z5rXPaTm2Zc3mFn6KmZoBVONfumG5lPSQDZKT+bSRbF7NJeyrBPATc3r3v0ICvfzU6FND5xh4LSG8MoDPDAoq0ZVqqbtXJIxUZdoQF2hIh5MQ9ZTKXhZIF0WRxigUcOXhpiQ7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add kunit tests to benchmark 256MiB copies to a KVEC iterator, a BVEC iterator, an XARRAY iterator and to a loop that allocates 256-page BVECs and fills them in (similar to a maximal bio struct being set up). Signed-off-by: David Howells cc: Christoph Hellwig cc: Christian Brauner cc: Jens Axboe cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: Brendan Higgins cc: David Gow cc: linux-kselftest@vger.kernel.org cc: kunit-dev@googlegroups.com cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- lib/kunit_iov_iter.c | 251 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c index fdf598e49c0b..1a43e9518a63 100644 --- a/lib/kunit_iov_iter.c +++ b/lib/kunit_iov_iter.c @@ -1262,6 +1262,253 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) KUNIT_SUCCEED(); } +static void iov_kunit_free_page(void *data) +{ + __free_page(data); +} + +#define IOV_KUNIT_NR_SAMPLES 16 +static void __init iov_kunit_benchmark_print_stats(struct kunit *test, + unsigned int *samples) +{ + unsigned long long sumsq = 0; + unsigned long total = 0, mean, stddev; + unsigned int n = IOV_KUNIT_NR_SAMPLES; + int i; + + //for (i = 0; i < n; i++) + // kunit_info(test, "run %x: %u uS\n", i, samples[i]); + + /* Ignore the 0th sample as that may include extra overhead such as + * setting up PTEs. + */ + samples++; + n--; + for (i = 0; i < n; i++) + total += samples[i]; + mean = total / n; + + for (i = 0; i < n; i++) { + long s = samples[i] - mean; + + sumsq += s * s; + } + stddev = int_sqrt64(sumsq); + + kunit_info(test, "avg %lu uS, stddev %lu uS\n", mean, stddev); +} + +/* + * Create a source buffer for benchmarking. + */ +static void *__init iov_kunit_create_source(struct kunit *test, size_t npages) +{ + struct page *page, **pages; + void *scratch; + size_t i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) { + pages[i] = page; + get_page(page); + } + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + return scratch; +} + +/* + * Time copying 256MiB through an ITER_KVEC. + */ +static void __init iov_kunit_benchmark_kvec(struct kunit *test) +{ + struct iov_iter iter; + struct kvec kvec[8]; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE, part; + void *scratch, *buffer; + int i; + + /* Allocate a huge buffer and populate it with pages. */ + buffer = iov_kunit_create_source(test, npages); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Split the target over a number of kvecs */ + copied = 0; + for (i = 0; i < ARRAY_SIZE(kvec); i++) { + part = size / ARRAY_SIZE(kvec); + kvec[i].iov_base = buffer + copied; + kvec[i].iov_len = part; + copied += part; + } + kvec[i - 1].iov_len += size - part; + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over KVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_kvec(&iter, ITER_SOURCE, kvec, ARRAY_SIZE(kvec), size); + + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC. + */ +static void __init iov_kunit_benchmark_bvec(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (i = 0; i < npages; i++) + bvec_set_page(&bvec[i], page, PAGE_SIZE, 0); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC in 256 page chunks. + */ +static void __init iov_kunit_benchmark_bvec_split(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size, npages = 64; + void *scratch; + int i, j; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + size = 256 * 1024 * 1024; + a = ktime_get_real(); + do { + size_t part = min_t(size_t, size, npages * PAGE_SIZE); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (j = 0; j < npages; j++) + bvec_set_page(&bvec[j], page, PAGE_SIZE, 0); + + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, part); + copied = copy_from_iter(scratch, part, &iter); + KUNIT_EXPECT_EQ(test, copied, part); + size -= part; + } while (size > 0); + b = ktime_get_real(); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_XARRAY. + */ +static void __init iov_kunit_benchmark_xarray(struct kunit *test) +{ + struct iov_iter iter; + struct xarray *xarray; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + xarray = iov_kunit_create_xarray(test); + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, page, GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + } + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over XARRAY:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_xarray(&iter, ITER_SOURCE, xarray, 0, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_ubuf), KUNIT_CASE(iov_kunit_copy_from_ubuf), @@ -1278,6 +1525,10 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_kvec), KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_benchmark_kvec), + KUNIT_CASE(iov_kunit_benchmark_bvec), + KUNIT_CASE(iov_kunit_benchmark_bvec_split), + KUNIT_CASE(iov_kunit_benchmark_xarray), {} };