From patchwork Fri Sep 22 11:30:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13395607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF9ACD4F49 for ; Fri, 22 Sep 2023 11:31:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEB7C6B02B7; Fri, 22 Sep 2023 07:31:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9B926B02B8; Fri, 22 Sep 2023 07:31:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C9306B02B9; Fri, 22 Sep 2023 07:31:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7606E6B02B7 for ; Fri, 22 Sep 2023 07:31:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 36AC3C10DC for ; Fri, 22 Sep 2023 11:31:17 +0000 (UTC) X-FDA: 81264017394.14.070FB80 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 7C9851C0014 for ; Fri, 22 Sep 2023 11:31:15 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dc8gP70f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695382275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SHVN5dprl5SXzKPIzGJgk8i0Q8IZLJJ1is7dtiWAZyQ=; b=ocWbTyi8xKc+WzLlL77n3jZtJUdSyG6T2gH7W/ho+s9xyBBh8l6LAEPY6zYvjCaeuxnHmD 5KBCw2cpE4W/uigu/G/KYw/TNNAkfC70xkh7r4EzbU4RePnLxM+xg+l8tAzgqqcNsYtTQV +Di6OpiJ0ufnqkqtmBizR/Z8yZzB+NE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dc8gP70f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695382275; a=rsa-sha256; cv=none; b=d1Spk3hEzZYCNdQDo6g/eVjjkLeuxZdZj2buhKxA3QqhhozPrpzQ2j/ZuL60zpjzqablXv oPADApHIipyJCfR0F8hTBHmCIwgi29tYpf+ACwSr2lIEmdrlYtS9RvEclnmmFtxe6hrnDa i++lkfJd+R/coUA4+c4jgTpJzciuGd4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695382274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SHVN5dprl5SXzKPIzGJgk8i0Q8IZLJJ1is7dtiWAZyQ=; b=Dc8gP70fw+PpY0LFzkjFPpUKXiBnqQC4c1NIhzMOIWINr6XGN5iOJdqyG5faKx+tqGNfUg iMnKy/w4w805gpzHHZZDJuTJizV+wQVAdr58+JsGnsGuslFnr8JYzMT/FfQrtutOQuRr7Q harjG0I7qxVxiQrf1vueL9AejD1jhyQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-487-T_FPJATDP5C2zR1rQTrR3g-1; Fri, 22 Sep 2023 07:31:10 -0400 X-MC-Unique: T_FPJATDP5C2zR1rQTrR3g-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 77D47811E7D; Fri, 22 Sep 2023 11:31:09 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3738940C6EBF; Fri, 22 Sep 2023 11:31:07 +0000 (UTC) From: David Howells To: Jens Axboe Cc: David Howells , Al Viro , Linus Torvalds , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Brendan Higgins , David Gow , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, Christian Brauner , David Hildenbrand , John Hubbard Subject: [PATCH v3 08/10] iov_iter: Add benchmarking kunit tests Date: Fri, 22 Sep 2023 12:30:36 +0100 Message-ID: <20230922113038.1135236-9-dhowells@redhat.com> In-Reply-To: <20230922113038.1135236-1-dhowells@redhat.com> References: <20230922113038.1135236-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7C9851C0014 X-Stat-Signature: 1q6hcqzrqe4qxqzno39d6mymzk6ceomx X-Rspam-User: X-HE-Tag: 1695382275-823566 X-HE-Meta: U2FsdGVkX1+B+yfz2YIj46mQcyd3Dzdn6i6nFsSi1uB21jP3EfFxa9VK5O6Y26pdmeGHfXhBl3wPYWCj1vouiZFIABVJGfyDPooJyjdjq4S7opbQkpHsJ89mve69UGRP3juHtPvn9EUW7h/RADWv2bB79/Tt9sqb03aOCWpOZNksifsDew/1OtyewWoIEsuDrHIiG8c9wC5K6SDtXd8NJNWf6qPUDDlZ+EHtWYpCFdDWYprbG1kf5DvnjN2PpRMBRLwDrUNe3X9JxEJpoguTP2SzJj0QfwmPiLyA98COUPMAlzApJar/KzsU+kJCWqco3Ic+vskLtVkmXjV6MWuXP6QwQ2tSd/QX6+H3U6njYrW1AqpuCNbndHVbH3Qrp2oOymuwwlxlDTEQDW9bzsr11L49Mc2KIaQOEzZB6eaS47jczd5pcLtMFu9LSSuDzmfwfS2fSQTMCHmDpWduC95HML8O7MEAUyRytybF2F3MgIIj7rJ1HAQmLbPnmOpX4GU3UYMjKIVTfcGyCtXItaTWda27rLmXG3QpZtFKw62+PF82UAY0tesd3XsXgjxY/Y8MpwcoQbNEtfX8/6PTxESAcPKAPOIxQtx/PJRdC/qdwKTtx8O78trmdGNGbbrsC0AEcedHKPdcZSITpp58wGzN8HRc+ZRw+raHJhp1QecvjYKTXvVPkhFg5JHC7AIbp9RNQA/Eij4B9hFYKfISeksLVi2u6UApSr79woE031tuJ3k3FzgTnfMmJmWArY0VnWV5DrBBpLlP4rARjC9t+T3sdowJmueja02hCaNA7z2GuHcWv9Ez32l3rpvB428rngLDzherODHsQcq+LJid6hwLMPeeHnQ2rlnnQtWCNO+aNn6u17MyVhcvlN0u6hT+R26xlIhgbju3CGa0C/RBT/va02NG/dD9easDdRPjDblEj4yyoMQPhXA1Ks3RkqV+4XB64yAuH8tQDvtrIhqC38H +w6uUdUF YO4rniHnvk6R5rS3wDSVUgDwuVGbe7/EuxaEC5AEoDTnqOlBhoiKN1c/fwasLuQEqNHndireVgZjbmUHEtxovAcTcDBnLGtAYJgRpUwnzQGvkGmVERz5lOXhPoPTH7bC8v6usP6UPMn1TxRJYXl+nzkDbCzoX6O7Pn0YL+2CtfxpG3uhhnX3laAscINONYup3bj1bHCfw+WMwHrGc64youj6roYeORWxj6s/8o4/fllzY4wH4szeZFlNiCMaviH5s9o8k4XwrXHvblbEhg26WChHseKUKb4D6VfgMkJ6CRUiQ+p+3rrz4pgjbBDA+LiZdFWEoUjCJyIN5tT9UNURC5Xg8lMLgcQkOC0A/c/emPvZGvuHqOq2EN0vcI9bZ7CeGpWSKEO1ik1DB0v+3WESq0qG31Flh82Amb3S1RD30Fqv0p7lU4RZnnrKX1BqvnP4CZywGTkqHsgoIsYCj0R09I/an0FIr5zy5Zp8GYjCkOsL8FqwHlQodpf8mY5SzZb3zhO8G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add kunit tests to benchmark 256MiB copies to a KVEC iterator, a BVEC iterator, an XARRAY iterator and to a loop that allocates 256-page BVECs and fills them in (similar to a maximal bio struct being set up). Signed-off-by: David Howells cc: Christoph Hellwig cc: Christian Brauner cc: Jens Axboe cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: Brendan Higgins cc: David Gow cc: linux-kselftest@vger.kernel.org cc: kunit-dev@googlegroups.com cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- lib/kunit_iov_iter.c | 251 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c index fdf598e49c0b..1a43e9518a63 100644 --- a/lib/kunit_iov_iter.c +++ b/lib/kunit_iov_iter.c @@ -1262,6 +1262,253 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) KUNIT_SUCCEED(); } +static void iov_kunit_free_page(void *data) +{ + __free_page(data); +} + +#define IOV_KUNIT_NR_SAMPLES 16 +static void __init iov_kunit_benchmark_print_stats(struct kunit *test, + unsigned int *samples) +{ + unsigned long long sumsq = 0; + unsigned long total = 0, mean, stddev; + unsigned int n = IOV_KUNIT_NR_SAMPLES; + int i; + + //for (i = 0; i < n; i++) + // kunit_info(test, "run %x: %u uS\n", i, samples[i]); + + /* Ignore the 0th sample as that may include extra overhead such as + * setting up PTEs. + */ + samples++; + n--; + for (i = 0; i < n; i++) + total += samples[i]; + mean = total / n; + + for (i = 0; i < n; i++) { + long s = samples[i] - mean; + + sumsq += s * s; + } + stddev = int_sqrt64(sumsq); + + kunit_info(test, "avg %lu uS, stddev %lu uS\n", mean, stddev); +} + +/* + * Create a source buffer for benchmarking. + */ +static void *__init iov_kunit_create_source(struct kunit *test, size_t npages) +{ + struct page *page, **pages; + void *scratch; + size_t i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) { + pages[i] = page; + get_page(page); + } + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + return scratch; +} + +/* + * Time copying 256MiB through an ITER_KVEC. + */ +static void __init iov_kunit_benchmark_kvec(struct kunit *test) +{ + struct iov_iter iter; + struct kvec kvec[8]; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE, part; + void *scratch, *buffer; + int i; + + /* Allocate a huge buffer and populate it with pages. */ + buffer = iov_kunit_create_source(test, npages); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Split the target over a number of kvecs */ + copied = 0; + for (i = 0; i < ARRAY_SIZE(kvec); i++) { + part = size / ARRAY_SIZE(kvec); + kvec[i].iov_base = buffer + copied; + kvec[i].iov_len = part; + copied += part; + } + kvec[i - 1].iov_len += size - part; + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over KVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_kvec(&iter, ITER_SOURCE, kvec, ARRAY_SIZE(kvec), size); + + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC. + */ +static void __init iov_kunit_benchmark_bvec(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (i = 0; i < npages; i++) + bvec_set_page(&bvec[i], page, PAGE_SIZE, 0); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC in 256 page chunks. + */ +static void __init iov_kunit_benchmark_bvec_split(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size, npages = 64; + void *scratch; + int i, j; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + size = 256 * 1024 * 1024; + a = ktime_get_real(); + do { + size_t part = min_t(size_t, size, npages * PAGE_SIZE); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (j = 0; j < npages; j++) + bvec_set_page(&bvec[j], page, PAGE_SIZE, 0); + + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, part); + copied = copy_from_iter(scratch, part, &iter); + KUNIT_EXPECT_EQ(test, copied, part); + size -= part; + } while (size > 0); + b = ktime_get_real(); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_XARRAY. + */ +static void __init iov_kunit_benchmark_xarray(struct kunit *test) +{ + struct iov_iter iter; + struct xarray *xarray; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + xarray = iov_kunit_create_xarray(test); + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, page, GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + } + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over XARRAY:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_xarray(&iter, ITER_SOURCE, xarray, 0, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_ubuf), KUNIT_CASE(iov_kunit_copy_from_ubuf), @@ -1278,6 +1525,10 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_kvec), KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_benchmark_kvec), + KUNIT_CASE(iov_kunit_benchmark_bvec), + KUNIT_CASE(iov_kunit_benchmark_bvec_split), + KUNIT_CASE(iov_kunit_benchmark_xarray), {} };