From patchwork Sat Nov 11 01:49:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Dai X-Patchwork-Id: 13452891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ACF2C4332F for ; Sat, 11 Nov 2023 01:49:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230101AbjKKBty (ORCPT ); Fri, 10 Nov 2023 20:49:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230071AbjKKBtx (ORCPT ); Fri, 10 Nov 2023 20:49:53 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60FB0449A for ; Fri, 10 Nov 2023 17:49:49 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5b053454aeeso36558697b3.0 for ; Fri, 10 Nov 2023 17:49:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1699667388; x=1700272188; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:from:to:cc:subject:date:message-id:reply-to; bh=xSVkGOQ6Je2Nvd4/Iw08tctHiARRNVZTWxkSrnRPly4=; b=fudsWKeWi/4dJZzg/r2Z+mvQOKy6Vo45qFSELBv1xieC8SBgTOR1yowiwqOcn98Owr XMGfqwo87qxugnqTOenu4Gm9Vk1iluFbAjqFYkGNwEOfVCNbRBJwG8ILG4rJHs0S7ICK Ch/NAta3CETcWIgsrLhW7UVA6ROk8ABDcvxYEb1H4Qw5EkynG9aea1ig20XhWv07A9Mr jaK3aOxOPM9FWaqFU23QMsOkb11D/5SgrSApsLqHvU2rc4MWRt0+4lmO5etvzSv4Eftk ZvX67vj0LdxJ8R7bYvlooqZhDj6sb7PtTVsl6yAcwDggBz/eeobE1DVT+doXXu0OUlcO RZhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699667388; x=1700272188; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=xSVkGOQ6Je2Nvd4/Iw08tctHiARRNVZTWxkSrnRPly4=; b=jp8L0QcJ2Ho0jhMDrUTaA2qLYb4iRkTxS0F975rUAEmpQPlrKX/0HicXfP1OXHh3sT 7kf1HC+H+DMLZFitNNGizJyPC1JJHfoVjMVX9/kd/aePrCEkrJbhfcSIPOY5RLvM/MdC lbQwn24Ldj2VSQTn13iJ9JaIn7nc+PjaFb8e7168atXKiOQC9szj2iqmAKjZbhCfIPN7 L/Diz2WdNUJuJeKrpxRlFx6v/XV02OnJ7oe5eifrpRQLQGK1t1qlt3fDsGELuOrYPaF0 u958P/vYQig2Br6wSEXb9msq2t2Vz6ugd0X1JyEzbeJWn2fpS6Sms+4hzsGaECdI1D0p ufmQ== X-Gm-Message-State: AOJu0YzI15lD1TApVyHzsuhV/sy6JlEKxB4AAA/+auYWFyrAFrmpftMm AmZshx9MyYDaZLLzT9lUuWU1b+v1M4jz X-Google-Smtp-Source: AGHT+IFSKHK8rNVtGq6xSzwzFHWKLOlchqnfAcYRbzjW2Gxy8VQh38Q0Vvu6iYAwEy08vUf+okshZZlASqdj X-Received: from davidai2.mtv.corp.google.com ([2620:15c:211:201:77fa:5ee:8100:77]) (user=davidai job=sendgmr) by 2002:a25:318a:0:b0:d9a:36cd:482e with SMTP id x132-20020a25318a000000b00d9a36cd482emr19289ybx.13.1699667388440; Fri, 10 Nov 2023 17:49:48 -0800 (PST) Date: Fri, 10 Nov 2023 17:49:28 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.42.0.869.gea05f2083d-goog Message-ID: <20231111014933.1934562-1-davidai@google.com> Subject: [PATCH v4 0/2] Improve VM CPUfreq and task placement behavior From: David Dai To: "Rafael J. Wysocki" , Viresh Kumar , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Sudeep Holla , David Dai , Saravana Kannan Cc: Quentin Perret , Masami Hiramatsu , Will Deacon , Peter Zijlstra , Vincent Guittot , Marc Zyngier , Oliver Upton , Dietmar Eggemann , Pavan Kondeti , Gupta Pankaj , Mel Gorman , kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Hi, This patch series is a continuation of the talk Saravana gave at LPC 2022 titled "CPUfreq/sched and VM guest workload problems" [1][2][3]. The gist of the talk is that workloads running in a guest VM get terrible task placement and CPUfreq behavior when compared to running the same workload in the host. Effectively, no EAS(Energy Aware Scheduling) for threads inside VMs. This would make power and performance terrible just by running the workload in a VM even if we assume there is zero virtualization overhead. With this series, a workload running in a VM gets the same task placement and CPUfreq behavior as it would when running in the host. The idea is to improve VM CPUfreq/sched behavior by: - Having guest kernel do accurate load tracking by taking host CPU arch/type and frequency into account. - Sharing vCPU frequency requirements with the host so that the host can do proper frequency scaling and task placement on the host side. Based on feedback from RFC v1 proposal[4], we've revised our implementation to using MMIO reads and writes to pass information from/to host instead of using hypercalls. In our example, the VMM(Virtual Machine Manager) translates the frequency requests into Uclamp_min and applies it to the vCPU thread as a hint to the host kernel. To achieve the results below, configure the host to: - Affine vCPUs to specific clusters. - Set vCPU capacity to match the host CPU they are running on. To make it easy for folks to try this out with CrosVM, we have put up userspace patches[5][6]. With those patches, you can configure CrosVM correctly by adding the options "--host-cpu-topology" and "--virt-cpufreq". Results: ======== Here are some side-by-side comparisons of RFC v1 proposal vs the current patch series and are labelled as follows. - (RFC v1) UtilHyp = hypercall + util_guest - (current) UClampMMIO = MMIO + UClamp_min Use cases running a minimal system inside a VM on a Pixel 6: ============================================================ FIO Higher is better +-------------------+----------+---------+--------+------------+--------+ | Usecase(avg MB/s) | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +-------------------+----------+---------+--------+------------+--------+ | Seq Write | 13.3 | 16.4 | +23% | 13.4 | +1% | +-------------------+----------+---------+--------+------------+--------+ | Rand Write | 11.2 | 12.9 | +15% | 11.2 | 0% | +-------------------+----------+---------+--------+------------+--------+ | Seq Read | 100 | 168 | +68% | 136 | +36% | +-------------------+----------+---------+--------+------------+--------+ | Rand Read | 20.5 | 35.6 | +74% | 29.5 | +44% | +-------------------+----------+---------+--------+------------+--------+ CPU-based ML Inference Benchmark Lower is better +----------------+----------+------------+--------+------------+--------+ | Test Case (ms) | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +----------------+----------+------------+--------+------------+--------+ | Cached Sample | | | | | | | Inference | 3.40 | 2.37 | -30% | 2.97 | -13% | +----------------+----------+------------+--------+------------+--------+ | Small Sample | | | | | | | Inference | 9.87 | 6.78 | -31% | 7.92 | -20% | +----------------+----------+------------+--------+------------+--------+ | Large Sample | | | | | | | Inference | 33.35 | 26.74 | -20% | 31.48 | -6% | +----------------+----------+------------+--------+------------+--------+ Use cases running Android inside a VM on a Chromebook: ====================================================== PCMark (Emulates real world usecases) Higher is better +-------------------+----------+---------+--------+------------+--------+ | Test Case (score) | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +-------------------+----------+---------+--------+------------+--------+ | Weighted Total | 5970 | 7162 | +20% | 6782 | +14% | +-------------------+----------+---------+--------+------------+--------+ | Web Browsing | 5558 | 5877 | +6% | 5729 | +3% | +-------------------+----------+---------+--------+------------+--------+ | Video Editing | 4921 | 5140 | +4% | 5079 | +3% | +-------------------+----------+---------+--------+------------+--------+ | Writing | 6864 | 9111 | +33% | 8171 | +10% | +-------------------+----------+---------+--------+------------+--------+ | Photo Editing | 7983 | 11349 | +42% | 10313 | +29% | +-------------------+----------+---------+--------+------------+--------+ | Data Manipulation | 5814 | 6051 | +4% | 6051 | +1% | +-------------------+----------+---------+--------+------------+--------+ PCMark Performance/mAh Higher is better +-------------------+----------+---------+--------+------------+--------+ | | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +-------------------+----------+---------+--------+------------+--------+ | Score/mAh | 85 | 102 | +20% | 94 | 10% | +-------------------+----------+---------+--------+------------+--------+ Roblox Higher is better +-------------------+----------+---------+--------+------------+--------+ | | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +-------------------+----------+---------+--------+------------+--------+ | FPS | 20.88 | 25.64 | +23% | 24.05 | +15% | +-------------------+----------+---------+--------+------------+--------+ Roblox Frames/mAh Higher is better +-------------------+----------+---------+--------+------------+--------+ | | Baseline | UtilHyp | %delta | UClampMMIO | %delta | +-------------------+----------+---------+--------+------------+--------+ | Frames/mAh | 85.29 | 102.31 | +20% | 94.20 | 10% | +-------------------+----------+---------+--------+------------+--------+ We've simplified our implementation based on community feedback to make it less intrusive and to use a more generic MMIO interface for communication with the host. The results show that the current design still has tangible improvements over baseline. We'll continue looking into ways to reduce the overhead of the MMIO read/writes and submit separate and generic patches for that if we find any good optimizations. Thanks, David & Saravana Cc: Saravana Kannan Cc: Quentin Perret Cc: Masami Hiramatsu Cc: Will Deacon Cc: Peter Zijlstra Cc: Vincent Guittot Cc: Marc Zyngier Cc: Oliver Upton Cc: Dietmar Eggemann Cc: Pavan Kondeti Cc: Gupta Pankaj Cc: Mel Gorman v3 -> v4: -Fixed dt-binding formatting issues -Added additional dt-binding descriptions for “HW interfaces” -Changed dt-binding to “qemu,virtual-cpufreq” -Fixed Kconfig formatting issues -Removed frequency downscaling when requesting frequency updates -Removed ops and cpufreq driver data -Added check to limit freq_scale to 1024 -Added KHZ in the register offset naming -Added comments to explain FIE and not allowing dvfs_possible_from_any_cpu v2 -> v3: - Dropped patches adding new hypercalls - Dropped patch adding util_guest in sched/fair - Cpufreq driver now populates frequency using opp bindings - Removed transition_delay_us=1 cpufreq setting as it was configured too agressively and resulted in poor I/O performance - Modified guest cpufreq driver to read/write MMIO regions instead of using hypercalls to communicate with the host - Modified guest cpufreq driver to pass frequency info instead of utilization of the current vCPU's runqueue which now takes iowait_boost into account from the schedutil governor - Updated DT bindings for a virtual CPU frequency device Userspace changes: - Updated CrosVM patches to emulate a virtual cpufreq device - Updated to newer userspace binaries when collecting more recent benchmark data v1 -> v2: - No functional changes. - Added description for EAS and removed DVFS in coverletter. - Added a v2 tag to the subject. - Fixed up the inconsistent "units" between tables. - Made sure everyone is To/Cc-ed for all the patches in the series. [1] - https://lpc.events/event/16/contributions/1195/ [2] - https://lpc.events/event/16/contributions/1195/attachments/970/1893/LPC%202022%20-%20VM%20DVFS.pdf [3] - https://www.youtube.com/watch?v=hIg_5bg6opU [4] - https://lore.kernel.org/all/20230331014356.1033759-1-davidai@google.com/ [5] - https://chromium-review.googlesource.com/c/crosvm/crosvm/+/4208668 [6] - https://chromium-review.googlesource.com/q/topic:%22virtcpufreq-v4%22 David Dai (2): dt-bindings: cpufreq: add virtual cpufreq device cpufreq: add virtual-cpufreq driver .../cpufreq/qemu,cpufreq-virtual.yaml | 99 +++++++++ drivers/cpufreq/Kconfig | 15 ++ drivers/cpufreq/Makefile | 1 + drivers/cpufreq/virtual-cpufreq.c | 201 ++++++++++++++++++ include/linux/arch_topology.h | 1 + 5 files changed, 317 insertions(+) create mode 100644 Documentation/devicetree/bindings/cpufreq/qemu,cpufreq-virtual.yaml create mode 100644 drivers/cpufreq/virtual-cpufreq.c