From patchwork Tue Jul 13 16:19:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 12374457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39703C07E95 for ; Tue, 13 Jul 2021 16:19:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2346861175 for ; Tue, 13 Jul 2021 16:19:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230516AbhGMQWJ (ORCPT ); Tue, 13 Jul 2021 12:22:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:35550 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230254AbhGMQWI (ORCPT ); Tue, 13 Jul 2021 12:22:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626193157; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=awbwz8Mi1G1+f2fwGt3ZEpkOa0ZEsi8ROYjzYe/Okus=; b=PoxG3ZFQEIknPgusP3jAlPopWFK8MOplxhvPlQ2MYBymohwpwqs7RRESBlrr/PIQ7r1MgN yILgJUWeVdPoaJ6k3pgEBvY9JfrFKROZeJ3IzsQJxrGxSoWecbADwrshRumoIVF7xWhlPn brxHJX0zBCcu+boIfCx7zS3+gHE2gD0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-456-JG1pSGTjO-m9PADZ5-L7Wg-1; Tue, 13 Jul 2021 12:19:16 -0400 X-MC-Unique: JG1pSGTjO-m9PADZ5-L7Wg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 258DB1B2C983; Tue, 13 Jul 2021 16:19:15 +0000 (UTC) Received: from localhost (ovpn-112-172.ams2.redhat.com [10.36.112.172]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9C1F819C44; Tue, 13 Jul 2021 16:19:14 +0000 (UTC) From: Stefan Hajnoczi To: linux-kernel@vger.kernel.org Cc: Daniel Lezcano , Stefano Garzarella , Ming Lei , "Michael S . Tsirkin" , Marcelo Tosatti , Jens Axboe , Jason Wang , linux-block@vger.kernel.org, "Rafael J. Wysocki" , virtualization@lists.linux-foundation.org, linux-pm@vger.kernel.org, Christoph Hellwig , Stefan Hajnoczi Subject: [RFC 0/3] cpuidle: add poll_source API and virtio vq polling Date: Tue, 13 Jul 2021 17:19:03 +0100 Message-Id: <20210713161906.457857-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org These patches are not polished yet but I would like request feedback on this approach and share performance results with you. Idle CPUs tentatively enter a busy wait loop before halting when the cpuidle haltpoll driver is enabled inside a virtual machine. This reduces wakeup latency for events that occur soon after the vCPU becomes idle. This patch series extends the cpuidle busy wait loop with the new poll_source API so drivers can participate in polling. Such polling-aware drivers disable their device's irq during the busy wait loop to avoid the cost of interrupts. This reduces latency further than regular cpuidle haltpoll, which still relies on irqs. Virtio drivers are modified to use the poll_source API so all virtio device types get this feature. The following virtio-blk fio benchmark results show the improvement: IOPS (numjobs=4, iodepth=1, 4 virtqueues) before poll_source io_poll 4k randread 167102 186049 (+11%) 186654 (+11%) 4k randwrite 162204 181214 (+11%) 181850 (+12%) 4k randrw 159520 177071 (+11%) 177928 (+11%) The comparison against io_poll shows that cpuidle poll_source achieves equivalent performance to the block layer's io_poll feature (which I implemented in a separate patch series [1]). The advantage of poll_source is that applications do not need to explicitly set the RWF_HIPRI I/O request flag. The poll_source approach is attractive because few applications actually use RWF_HIPRI and it takes advantage of CPU cycles we would have spent in cpuidle haltpoll anyway. The current series does not improve virtio-net. I haven't investigated deeply, but it is possible that NAPI and poll_source do not combine. See the final patch for a starting point on making the two work together. I have not tried this on bare metal but it might help there too. The cost of disabling a device's irq must be less than the savings from avoiding irq handling for this optimization to make sense. [1] https://lore.kernel.org/linux-block/20210520141305.355961-1-stefanha@redhat.com/ Stefan Hajnoczi (3): cpuidle: add poll_source API virtio: add poll_source virtqueue polling softirq: participate in cpuidle polling drivers/cpuidle/Makefile | 1 + drivers/virtio/virtio_pci_common.h | 7 ++ include/linux/interrupt.h | 2 + include/linux/poll_source.h | 53 +++++++++++++++ include/linux/virtio.h | 2 + include/linux/virtio_config.h | 2 + drivers/cpuidle/poll_source.c | 102 +++++++++++++++++++++++++++++ drivers/cpuidle/poll_state.c | 6 ++ drivers/virtio/virtio.c | 34 ++++++++++ drivers/virtio/virtio_pci_common.c | 86 ++++++++++++++++++++++++ drivers/virtio/virtio_pci_modern.c | 2 + kernel/softirq.c | 14 ++++ 12 files changed, 311 insertions(+) create mode 100644 include/linux/poll_source.h create mode 100644 drivers/cpuidle/poll_source.c