From patchwork Tue Aug 1 20:47:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sudarshan Rajagopalan X-Patchwork-Id: 13337241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 639B4C0015E for ; Tue, 1 Aug 2023 20:48:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5C402800F6; Tue, 1 Aug 2023 16:48:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE6B42800C8; Tue, 1 Aug 2023 16:48:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 969FE2800F6; Tue, 1 Aug 2023 16:48:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8081B2800C8 for ; Tue, 1 Aug 2023 16:48:50 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 535DF140642 for ; Tue, 1 Aug 2023 20:48:50 +0000 (UTC) X-FDA: 81076724820.24.B4EA1EE Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by imf26.hostedemail.com (Postfix) with ESMTP id 112AE140006 for ; Tue, 1 Aug 2023 20:48:47 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=DRFYyVHK; spf=pass (imf26.hostedemail.com: domain of sudaraja@qualcomm.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=sudaraja@qualcomm.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690922928; a=rsa-sha256; cv=none; b=z0vDDKE9YqriLHVnbsdkyCRyJ2hnhzuZsHTyW5WN8oCkzIx2du7I3nzaezeBBh/XLJsgdw 1uB5ARHoKNMGObNLoSuU6jPIsO+SfiI/qQ5pWX4tUBAfArrvrzuVhnW4G0KMSJoTuP1Nq4 fXI+SAr2U2FwI4DFTYROFYgG/g2n7gg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=DRFYyVHK; spf=pass (imf26.hostedemail.com: domain of sudaraja@qualcomm.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=sudaraja@qualcomm.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690922928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references:dkim-signature; bh=gQu+xJ9b2wewbPXTvV8WorriSxzvGoVmvD2FKbEEJMo=; b=Zd0Rgo9sGnG9Y9jMoECzoEkRGEnApYU1IdUZOt37tJcaW2cCmuMewLVEIiSlZdDVNZrLSX rMuVwhu0B+LvAvET4sKnGVE0NUYX5R0/+1d6ug7SS33EpKTA89scrE/gCtjEjU6Zgz89bq gCk8rLCUuhpvGJxf4/NobeIsK1gNTQM= Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 371Kehqs024310; Tue, 1 Aug 2023 20:48:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=qcppdkim1; bh=gQu+xJ9b2wewbPXTvV8WorriSxzvGoVmvD2FKbEEJMo=; b=DRFYyVHKtOuFXLg4/u1X2pQ/AY/61nbZTl7ICs+aeNOddm9FbqUltAnUgcCILmR48662 neXurx+5ow2yxqm9KU5D1mjERsbfUOdDivh2CUhuFJx/AhUKfgby7pE2m2+MabhYuW57 6B/zoQiGJ3F9OFsxEpzt422qP7ZzLZEgCXiMZmHNw0mLFpdJm8fCJRi3sf2LviNGBcpH t2r/ITfCy4PMNEE/8z+AujP6WZq10IlQvcrS5d4RQFHKTD18P+msJLIYxyM8Fp5/t3c3 Jjf8QzpKZHZ1txm521aRc/2xQ/5wg3utv5UYbsDuoQzPT9P6D8o0Ax2JpyxuheivXv4o tw== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3s75b30hyv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Aug 2023 20:48:39 +0000 Received: from pps.filterd (NASANPPMTA02.qualcomm.com [127.0.0.1]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTP id 371Kf6Pm019361; Tue, 1 Aug 2023 20:48:38 GMT Received: from pps.reinject (localhost [127.0.0.1]) by NASANPPMTA02.qualcomm.com (PPS) with ESMTP id 3s776xs2us-1; Tue, 01 Aug 2023 20:48:38 +0000 Received: from NASANPPMTA02.qualcomm.com (NASANPPMTA02.qualcomm.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 371KmcXs027428; Tue, 1 Aug 2023 20:48:38 GMT Received: from th-lint-014.qualcomm.com (th-lint-014.qualcomm.com [10.63.177.51]) by NASANPPMTA02.qualcomm.com (PPS) with ESMTP id 371Kmc1Q027424; Tue, 01 Aug 2023 20:48:38 +0000 Received: by th-lint-014.qualcomm.com (Postfix, from userid 455663) id E43166846; Tue, 1 Aug 2023 13:48:37 -0700 (PDT) From: Sudarshan Rajagopalan To: "T . J . Alumbaugh" , David Hildenbrand , Johannes Weiner , Suren Baghdasaryan , Mike Rapoport , Oscar Salvador , Anshuman Khandual , "mark . rutland @ arm . com" , "will @ kernel . org" , "virtualization @ lists . linux-foundation . org" , "linux-mm @ kvack . org" , "linux-kernel @ vger . kernel . org" , "linux-arm-kernel @ lists . infradead . org" , "linux-arm-msm @ vger . kernel . org" Cc: Sudarshan Rajagopalan , Trilok Soni , Sukadev Bhattiprolu , Srivatsa Vaddagiri , Patrick Daly , Elliot Berman Subject: [RFC PATCH 2/2] vmrd: add implementation of memory plugin interfaces Date: Tue, 1 Aug 2023 13:47:14 -0700 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: In-Reply-To: References: X-QCInternal: smtphost X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 7ke5VV3D8Y36hUICoVnv7TPsYPwCmpx9 X-Proofpoint-ORIG-GUID: 7ke5VV3D8Y36hUICoVnv7TPsYPwCmpx9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-08-01_19,2023-08-01_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 adultscore=0 suspectscore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 mlxscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308010185 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 112AE140006 X-Stat-Signature: otpf5teb3dwpihkxg44rkdq6xtfnogru X-Rspam-User: X-HE-Tag: 1690922927-38374 X-HE-Meta: U2FsdGVkX19NjOsUJqJyzCNbkiX/Mc7oUsw31ubpi0Iom5+QUsjN1XP7yiGoP044/gP/mSqJiySBmMWJ986YaYqi0zRsLJUdgiuFcQcwEfikV6rf0lwT4YYkck+kbZlIkk+px09CNbP4tCMQn8MTVfH7Bo8yp3BLQu0FB1sAnk1Hjf8JtHZxi6tFI3CLxbdg2wwOvszlyvY8UCNdwBWPBsvFe6zSGYG9K/Yco5F6aFiVdpJA1OiIkyqlsB80SO4QYcEizvQfSO/8FxHPiFEtSFUCGUbJpsFwQh0KAglYW2eltbMaXF3lyH9JybFj/1wGynsgEuwZufO5NWNesC8ZXnyGgxxPUWiURSXyKaOeth4VUXNB0lYwpi0dcIeQcGMEw7rhwrPPa/bcWX3yVheEpuLjEOBAycgdhfX5B2MVJmZtZczD5nic1OKQYhZRSV5pX6U4LNzjHRJYsjjVR+gUcA/xmlDlaK3FhjGJRoPjo2UXKiO4z5ojM1gJ7eZOp9srWZsYLcxt6wWBgEZmsMOku6HcTy9YpbrCiGHI5p5E1zNZGSZvbO/givvPgAukOH4YoG8dJ0Jfcn5uts6KPibYGEnuiN8Pnlyj9tmsogr3d5IEADpOFiKtHWftTL6C5jG//94lM0VaqLE6J3648MwanmK/bHlJ4vlq/pc4VxYFzopcyqzfO9b2oVBMrpKegVA76Q/xk7q+jJzYLoOBucCkcgFrwdzZVZBUNv5qvjBOkzXmmXsHZwc7+mf/zuBiFmo7D124CivSH59hXfyeTtJbo0Kf+9ovoYE5AMynZNivyR91VIZYxE+xNA+LoPAzPvCh4fFpGI6aXieQ+XNKSiiYVAOxe5IGK83+IavTFZ0t9NpUrUu4A5njnyc7cyDfnQ7n9Myqa70OSqR0ryuvVoNX1MbXYeQ2ojOaBr0lAG+2VbYai75g2KixhxnoiwRsLXs8rMrgwdyMzDqdeZwt6O3 31VGgvlU eKC3XGUa+tVBAl/NPJkgKhN8uQeTGjZBil/7q6Sl7/1idbKYDA0KsRCKcuUm/QvxCTV7DETyY14eT/dI5vb7X59W+ivM6/oON34BWvjl4CcoIZku3BbZjyVArGIlocC313icgBTpOZXbWHD17/lFfDuSUraBlEUalzq4DxUP4HAIqtYvWkaLSv350Lr9omJsWIpgxPbVYvO1ZWq0rQlEBxFcMLDu7bgBfiB2jlr5m69/wITzRU8jQyt/iv8kZcqBjGulZlrJ10V80NxAlG8hd9uXJiVUZvMvDuq6a59H+FFKrT7CeTVat+xLcMEhig5R9YFJmTVOAmJepwbUTTI10ryCwWiKgT/e8rLDgU6CbezJDWa2ymDPUekyBs/Mska2TdL/0nHctaIYDcmKWpvepuDh9cQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implementation of memory plugin interfaces such as memory_plug_request, memory_unplug_request etc. using virtio_mem kernel driver. The userspace daemon makes ioctl calls to kernel requesting for adding/ removing memory to the VM. The size request is aligned to virtio-mem device size. Modified version of virtio-mem driver is used that supports memory_on_hotplug feature and add/remove memory requests via ioctl calls. Link to the ioctl handling function is below: https://git.codelinaro.org/clo/la/kernel/msm-5.15/-/blob/kernel.lnx.5.15.r33-rel/drivers/virtio/qti_virtio_mem.c#L185 modified virtio-mem driver: https://git.codelinaro.org/clo/la/kernel/msm-5.15/-/blob/kernel.lnx.5.15.r33-rel/drivers/virtio/virtio_mem.c We use a kernel driver called mem-buf for communication between the VMs. The memory plug request by virtio-mem is made by calling mem_buf_alloc each of virtio-mem block size iteratively to accommodate the requested size. https://git.codelinaro.org/clo/la/kernel/msm-5.15/-/blob/kernel.lnx.5.15.r33-rel/drivers/virtio/virtio_mem.c#L1373 The mem-buf driver uses to Linux Gunyah driver (mem_buf_map_mem_s2) to communicate with Hypervisor requesting to map the memory into S2 page-tables. This is currently tested only on Gunyah Hypervisor and not on other Hypervisors. mem_buf_map_mem_s2: https://git.codelinaro.org/clo/la/kernel/msm-5.15/-/blob/kernel.lnx.5.15.r33-rel/drivers/soc/qcom/mem_buf/mem-buf-dev-gh.c#L207 The daemon also gets virtio-mem device configuration details such as device_bloc_size, max_threshold of resizing the VM etc. from the kernel via sysfs nodes of virtio-mem device. --- vmrd.cpp | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 117 insertions(+), 12 deletions(-) diff --git a/vmrd.cpp b/vmrd.cpp index 1bf5812..090f90a 100644 --- a/vmrd.cpp +++ b/vmrd.cpp @@ -19,11 +19,18 @@ #include #include #include +#include #include #include #include #include +#include #include +#include +#include + +/* qti_virtio_mem uapi header */ +#include #define SIZE_1MB 0x00100000 #define SIZE_1KB 0x00000400 @@ -205,23 +212,71 @@ using namespace std; * are needed to support the functionality of vmrd. */ +/* qti_virtio_mem device fd */ +static int virtio_mem_fd = -1; + +/* mem_buf fds returned by virtio-mem driver */ +static vector array_memfd; + +#define QVM_VIRTIO_MEM_DEV_PATH "/dev/qti_virtio_mem" +#define QVM_SYS_DEVICE_PATH "/sys/devices/virtual/qti_virtio_mem/qti_virtio_mem" +#define QVM_BLOCK_SIZE_PATH QVM_SYS_DEVICE_PATH"/device_block_size" +#define QVM_MAX_PLUGIN_THRES_PATH QVM_SYS_DEVICE_PATH"/max_plugin_threshold" +#define QVM_NUM_BLOCK_PLUGGED_PATH QVM_SYS_DEVICE_PATH"/device_block_plugged" + +static int virtio_mem_plug_memory(int64_t size, const std::string& name) +{ + struct qti_virtio_mem_ioc_hint_create_arg arg = {}; + int ret; + + if (virtio_mem_fd < 0) + return -ENOTTY; + + arg.size = size; + strlcpy(arg.name, name.c_str(), sizeof(arg.name)); + + ret = ioctl(virtio_mem_fd, QTI_VIRTIO_MEM_IOC_HINT_CREATE, &arg); + if (ret) { + LOGE("MemorySizeHint() failed"); + return ret; + } + + return arg.fd; +} + static int memory_plug_init(void) { - LOGE("memory plug request not supported"); - return -ENOTTY; + virtio_mem_fd = open(QVM_VIRTIO_MEM_DEV_PATH, O_RDONLY | O_CLOEXEC); + if (virtio_mem_fd < 0) { + LOGE("Unable to open %s: %s", QVM_VIRTIO_MEM_DEV_PATH, strerror(errno)); + return errno; + } + + return 0; } static void memory_plug_deinit(void) { - LOGE("memory plug request not supported"); + if (virtio_mem_fd >= 0) + close(virtio_mem_fd); } /* * Plugs in memory of given size into the system by requesting it from host VM. * This call is expected to be blocking call. */ + static int memory_plug_request(uint64_t size) { - (void) size; - LOGE("Memory plug request not supported"); - return -ENOTTY; + int memfd; + + memfd = virtio_mem_plug_memory(size * SIZE_1MB, "vmrd"); + if (memfd < 0) { + LOGE("failed to suggest memory size hint"); + return -1; + } + + LOGI("Memory of size %lu MB plugged-in successfully", size); + array_memfd.push_back(memfd); + + return 0; } /* @@ -230,8 +285,18 @@ static int memory_plug_request(uint64_t size) { * his call is expected to be blocking call. */ static int memory_unplug_request(uint64_t size) { - (void) size; - LOGE("Memory unplug request not supported"); + int res; + + if (array_memfd.size()) { + res = close(array_memfd.back()); + array_memfd.pop_back(); + if (res) + LOGE("Failed to unplug one memory chunk of size %lu MB", size); + + return res; + } + + LOGE("No memory available to unplug"); return -ENOTTY; } @@ -243,8 +308,32 @@ static int memory_unplug_request(uint64_t size) { * This call is expected to be blocking call. */ static int __unused memory_unplug_all_request(void) { - LOGE("Memory unplug all request not supported"); - return -ENOTTY; + uint64_t initial_count, unplugged_count = 0, res; + + initial_count = array_memfd.size(); + if (!initial_count) { + LOGE("No memory available to unplug"); + return 0; + } + + while (array_memfd.size()) { + LOGI("releasing one memory chunk to host VM"); + res = close(array_memfd.back()); + array_memfd.pop_back(); + if (res) + LOGE("failed to unplug one memory chunk"); + else + unplugged_count++; + } + + if (unplugged_count < initial_count) + LOGI("not all memory chunks were unplugged. initial_count: %lu unplugged_count: %lu", + initial_count, unplugged_count); + else + LOGI("Successfully unplugged all memory chunks. unplugged_count: %lu", + unplugged_count); + + return unplugged_count; } static int write_file(const char *file_path, char *s) { @@ -305,7 +394,15 @@ static char *read_file(const char *file_path) { * Memory block size or resolution. */ static int get_memory_plugin_resolution(uint64_t *plugin_resolution_mb) { - *plugin_resolution_mb = DEFAULT_PLUGIN_RESOLUTION_MB; + char *buf; + + buf = read_file(QVM_BLOCK_SIZE_PATH); + if (!buf) + return -EINVAL; + + *plugin_resolution_mb = strtoul(buf, 0, 10); + *plugin_resolution_mb /= SIZE_1MB; + return 0; } @@ -313,7 +410,15 @@ static int get_memory_plugin_resolution(uint64_t *plugin_resolution_mb) { * Total max memory that the system (guest VM) allows to be pluuged-in. */ static int get_max_memory_plugin_allowed(uint64_t *max_memory_plugin_mb) { - *max_memory_plugin_mb = DEFAULT_MAX_MEMORY_PLUGIN_MB; + char *buf; + + buf = read_file(QVM_MAX_PLUGIN_THRES_PATH); + if (!buf) + return -EINVAL; + + *max_memory_plugin_mb = strtoul(buf, 0, 10); + *max_memory_plugin_mb /= SIZE_1MB; + return 0; }