From patchwork Tue Jan 9 19:37:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliot Berman X-Patchwork-Id: 13515276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15101C4707B for ; Tue, 9 Jan 2024 19:38:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89D6B8D0005; Tue, 9 Jan 2024 14:38:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FC628D0001; Tue, 9 Jan 2024 14:38:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 652808D0005; Tue, 9 Jan 2024 14:38:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4C44C8D0001 for ; Tue, 9 Jan 2024 14:38:07 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1E21A1C1356 for ; Tue, 9 Jan 2024 19:38:07 +0000 (UTC) X-FDA: 81660783414.09.3BB6308 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by imf11.hostedemail.com (Postfix) with ESMTP id E104F40002 for ; Tue, 9 Jan 2024 19:38:04 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=TDAArpDa; spf=pass (imf11.hostedemail.com: domain of quic_eberman@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_eberman@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704829085; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xWdZbdG7dSafWq3cabzRvyRx9NAZngj+f9YcIbeZBfY=; b=ZL1E37ztEPFlkU0x5bHLSoIn3Kibtlc4YobaNl1KnzhM0kPzSL1PL1wCAWTbCojh87tfGM pzIQgE9a5YZbNMGtgq1PZKXU1cCP/FKsDW5GqiW1SxjEKViN0s1H5ajMIK8U5BQQUuDZrn CjRMZVZGMdhr4IJ0LJ+sZlHoyXXag18= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704829085; a=rsa-sha256; cv=none; b=4fRtuZm/sASx17OOWn0QLbt0sjqmoAp4bYI25AwtJyiZGeb7u/D/W6F4wzcmhiE4RrFBFJ DFlEJXAklXB7qQJmDV1R5VekJ3PeR26kryVrW443ZLlBpWxqkpyRI0JJ4dhBvj6+b9nega c8A8lZpNpdrbtSY4TZZb4Gl0a7qGacE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=TDAArpDa; spf=pass (imf11.hostedemail.com: domain of quic_eberman@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_eberman@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com Received: from pps.filterd (m0279868.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 409GaJ18010710; Tue, 9 Jan 2024 19:37:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h= from:date:subject:mime-version:content-type :content-transfer-encoding:message-id:references:in-reply-to:to :cc; s=qcppdkim1; bh=xWdZbdG7dSafWq3cabzRvyRx9NAZngj+f9YcIbeZBfY =; b=TDAArpDahUKJYT7Ji4NE1rBHgi9T9MX6/TKw57bQQmvVzjuO7uQw0AHQLOS Ru7UbgJaSCXKlurAPEVVoULnJ8dyMtzYsYRJfxwGW1YVhFS+laYSAyr1cEWK1wH7 FnWH/fBjD7+iEyaaUwUeonBaw23vg0oR0ZEkjdz57ZBxnC4Imxor1MArNGW8BoE3 i4Lx7BTIvLf/7bOvorKdKiJggh3qm0CJUl7xYCguAHA0oHGKSyUmTMnluXLj1nKd k/obzOk5EFohZVl1J4FW+JLwuFaLDd/sgPojysn7Q1gIHCKUSMZ+AbAn+ZzSZC2j 39vNCH9hRwOh7UuVwR+iRBmpwdg== Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3vh9ta0dw8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 09 Jan 2024 19:37:57 +0000 (GMT) Received: from nasanex01b.na.qualcomm.com (nasanex01b.na.qualcomm.com [10.46.141.250]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 409JbuQD011419 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 9 Jan 2024 19:37:56 GMT Received: from hu-eberman-lv.qualcomm.com (10.49.16.6) by nasanex01b.na.qualcomm.com (10.46.141.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Tue, 9 Jan 2024 11:37:55 -0800 From: Elliot Berman Date: Tue, 9 Jan 2024 11:37:48 -0800 Subject: [PATCH v16 10/34] gunyah: vm_mgr: Add VM start/stop MIME-Version: 1.0 Message-ID: <20240109-gunyah-v16-10-634904bf4ce9@quicinc.com> References: <20240109-gunyah-v16-0-634904bf4ce9@quicinc.com> In-Reply-To: <20240109-gunyah-v16-0-634904bf4ce9@quicinc.com> To: Alex Elder , Srinivas Kandagatla , Murali Nalajal , Trilok Soni , Srivatsa Vaddagiri , Carl van Schaik , Philip Derrin , Prakruthi Deepak Heragu , Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Catalin Marinas , Will Deacon , Konrad Dybcio , Bjorn Andersson , Dmitry Baryshkov , "Fuad Tabba" , Sean Christopherson , "Andrew Morton" CC: , , , , , , Elliot Berman X-Mailer: b4 0.13-dev X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01b.na.qualcomm.com (10.46.141.250) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: xkWEhD3cCcd4_26fH5x9GQrV4OQNq86C X-Proofpoint-ORIG-GUID: xkWEhD3cCcd4_26fH5x9GQrV4OQNq86C X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-09_02,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 impostorscore=0 malwarescore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2311290000 definitions=main-2401090158 X-Stat-Signature: fc545n1ostcyc5qwa1em3sjx868hackw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: E104F40002 X-Rspam-User: X-HE-Tag: 1704829084-871526 X-HE-Meta: U2FsdGVkX19GuURkJ0F/of7ZF7MA0sHuz4UAgWbI6VAfL9V/nYi49ZGx6PLv3j49MbY6r/Q62ZfvL59VZO8+ebLY/Tic0S8H0zfPdORb2yEYRiWcTaQ27T4n/ZpGmH4VDNbSw5wu2EPD+ZGvmf+4DTIuAwHA33HvHK8AMm/+lM5oMLv63YpXaBU0pONHqwng7r2RWMvcGU31JzDbMZDm4Juickxu/5/6xmMuXYgUuoNIhAZLCfsYAY2v1X4QQrW025WYDYxylzVrrw8qTrT3NaVCR5SH4wVdjbySmE644VlDfZ4T98Yue1qQ3lNQjUVG7LoUBA1UdPqFq7j+3+wsNA85umT0El0NnKR94B8khDQGrFV0mkpH0ag29ddFezbBSAWFq7Rkg2krJrGSaNOlU1xGTGniP+GVxRLlAl6svgXa0dgrFFERrDi0t0nrIn2xl/cY/fPaX34S/vNDKk5hbHL0xBJ26X/brkPLrpq+b0Pg9Pa6YR83piNVVjsMw0X+feV80mNHU4kB08tZfRuuCJXSvdjF6t7TvjU3Cii+8HhhW5Po9wgSRlbyMg2mKX7OQ/Svc2kVu/EBYnCVrQZFHyAm6MHWIsmB8l0vTunQVUlvvgvjcdNa97E+pAR/lmUei7Hqky7nPwY8IBp1riT0SCbNIQn7V7P4Dcop2FAMPseyeQ82EBnLL3BBuq1xWGl+yN2U9QRwsI5Mp2LZ7zLxWV60mhl1xkAys6mEvln2wjh4DREYSqDWKZnhinhsklrY8vcTmZ26SzUNu8jF8ypuGsQdUgwp9P4Ob2HlqH9fCSWEBvcTv9s/T7jDqpYd/3aLaxAt7XUL5WLIFphEozfZNkiLgJNNvaXvNPW0VgM4ogjEpsNpCYDmE2vrprwAO4RDzNu6vY3BD2Pb2D3RxvXiYL9KfdRzxVOTHqsNPc3mjrFuBnTz/mG+p2pE8EYLM9FQoJpa+feSAceW8sMenLZ p1h4ST2o im04xVMyzpqFiks4z1sBk7b0bYT6fgjdY46ihVf8sJ3L3dxhJ5+9MNryziHQ5WOWGgLUW4u4rOvjsz/9PnQkhilF1fMMQ+StiQKQ6fJzX/ELFKzBPY/DEZ/XLrdR5AvKEFz+GEtVWO5H+tzz59WqxmlaRcB/wG4FMFNr4VJwfG6WJozJqDjf76kY9gNUlEkarHCBLhYtUzSwL7UV9Q32wNUb0utVivJdjHERgwNkMJ2pRi0DCnkkaIivl7RWJnTJCUHOt7ZlxA84r/SXJdkj+Rr1pE7RS5jn/QabDIEVF/EXZxONr7mLA0Lm1sVot8UhuM+UopVDC0nYuIANPX3OJzYYiwDmvqEk7l0NDn+8gpDH0wZZWj16l+eMOdh+HSVIaKZ0SLM0YZ6L3U6+jrAURVIQfKd9hpOfJdyWP0ad3veIXUME= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add ioctl to trigger the start of a Gunyah virtual machine. Subsequent commits will provide memory to the virtual machine and add ability to interact with the resources (capabilities) of the virtual machine. Although start of the virtual machine can be done implicitly on the first vCPU run for proxy-schedule virtual machines, there is a non-trivial number of calls to Gunyah: a more precise error can be given to userspace which calls VM_START without looking at kernel logs because userspace can detect that the VM start failed instead of "couldn't run the vCPU". Co-developed-by: Prakruthi Deepak Heragu Signed-off-by: Prakruthi Deepak Heragu Signed-off-by: Elliot Berman --- drivers/virt/gunyah/vm_mgr.c | 198 +++++++++++++++++++++++++++++++++++++++++++ drivers/virt/gunyah/vm_mgr.h | 19 +++++ include/uapi/linux/gunyah.h | 5 ++ 3 files changed, 222 insertions(+) diff --git a/drivers/virt/gunyah/vm_mgr.c b/drivers/virt/gunyah/vm_mgr.c index e9dff733e35e..f6e6b5669aae 100644 --- a/drivers/virt/gunyah/vm_mgr.c +++ b/drivers/virt/gunyah/vm_mgr.c @@ -15,6 +15,68 @@ #include "rsc_mgr.h" #include "vm_mgr.h" +static int gunyah_vm_rm_notification_status(struct gunyah_vm *ghvm, void *data) +{ + struct gunyah_rm_vm_status_payload *payload = data; + + if (le16_to_cpu(payload->vmid) != ghvm->vmid) + return NOTIFY_OK; + + /* All other state transitions are synchronous to a corresponding RM call */ + if (payload->vm_status == GUNYAH_RM_VM_STATUS_RESET) { + down_write(&ghvm->status_lock); + ghvm->vm_status = payload->vm_status; + up_write(&ghvm->status_lock); + wake_up(&ghvm->vm_status_wait); + } + + return NOTIFY_DONE; +} + +static int gunyah_vm_rm_notification_exited(struct gunyah_vm *ghvm, void *data) +{ + struct gunyah_rm_vm_exited_payload *payload = data; + + if (le16_to_cpu(payload->vmid) != ghvm->vmid) + return NOTIFY_OK; + + down_write(&ghvm->status_lock); + ghvm->vm_status = GUNYAH_RM_VM_STATUS_EXITED; + up_write(&ghvm->status_lock); + wake_up(&ghvm->vm_status_wait); + + return NOTIFY_DONE; +} + +static int gunyah_vm_rm_notification(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct gunyah_vm *ghvm = container_of(nb, struct gunyah_vm, nb); + + switch (action) { + case GUNYAH_RM_NOTIFICATION_VM_STATUS: + return gunyah_vm_rm_notification_status(ghvm, data); + case GUNYAH_RM_NOTIFICATION_VM_EXITED: + return gunyah_vm_rm_notification_exited(ghvm, data); + default: + return NOTIFY_OK; + } +} + +static void gunyah_vm_stop(struct gunyah_vm *ghvm) +{ + int ret; + + if (ghvm->vm_status == GUNYAH_RM_VM_STATUS_RUNNING) { + ret = gunyah_rm_vm_stop(ghvm->rm, ghvm->vmid); + if (ret) + dev_warn(ghvm->parent, "Failed to stop VM: %d\n", ret); + } + + wait_event(ghvm->vm_status_wait, + ghvm->vm_status != GUNYAH_RM_VM_STATUS_RUNNING); +} + static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gunyah_rm *rm) { struct gunyah_vm *ghvm; @@ -24,14 +86,148 @@ static __must_check struct gunyah_vm *gunyah_vm_alloc(struct gunyah_rm *rm) return ERR_PTR(-ENOMEM); ghvm->parent = gunyah_rm_get(rm); + ghvm->vmid = GUNYAH_VMID_INVAL; ghvm->rm = rm; + init_rwsem(&ghvm->status_lock); + init_waitqueue_head(&ghvm->vm_status_wait); + ghvm->vm_status = GUNYAH_RM_VM_STATUS_NO_STATE; + return ghvm; } +static int gunyah_vm_start(struct gunyah_vm *ghvm) +{ + int ret; + + down_write(&ghvm->status_lock); + if (ghvm->vm_status != GUNYAH_RM_VM_STATUS_NO_STATE) { + up_write(&ghvm->status_lock); + return 0; + } + + ghvm->nb.notifier_call = gunyah_vm_rm_notification; + ret = gunyah_rm_notifier_register(ghvm->rm, &ghvm->nb); + if (ret) + goto err; + + ret = gunyah_rm_alloc_vmid(ghvm->rm, 0); + if (ret < 0) { + gunyah_rm_notifier_unregister(ghvm->rm, &ghvm->nb); + goto err; + } + ghvm->vmid = ret; + ghvm->vm_status = GUNYAH_RM_VM_STATUS_LOAD; + + ret = gunyah_rm_vm_configure(ghvm->rm, ghvm->vmid, ghvm->auth, 0, 0, 0, + 0, 0); + if (ret) { + dev_warn(ghvm->parent, "Failed to configure VM: %d\n", ret); + goto err; + } + + ret = gunyah_rm_vm_init(ghvm->rm, ghvm->vmid); + if (ret) { + ghvm->vm_status = GUNYAH_RM_VM_STATUS_INIT_FAILED; + dev_warn(ghvm->parent, "Failed to initialize VM: %d\n", ret); + goto err; + } + ghvm->vm_status = GUNYAH_RM_VM_STATUS_READY; + + ret = gunyah_rm_vm_start(ghvm->rm, ghvm->vmid); + if (ret) { + dev_warn(ghvm->parent, "Failed to start VM: %d\n", ret); + goto err; + } + + ghvm->vm_status = GUNYAH_RM_VM_STATUS_RUNNING; + up_write(&ghvm->status_lock); + return ret; +err: + /* gunyah_vm_free will handle releasing resources and reclaiming memory */ + up_write(&ghvm->status_lock); + return ret; +} + +static int gunyah_vm_ensure_started(struct gunyah_vm *ghvm) +{ + int ret; + + ret = down_read_interruptible(&ghvm->status_lock); + if (ret) + return ret; + + /* Unlikely because VM is typically started */ + if (unlikely(ghvm->vm_status == GUNYAH_RM_VM_STATUS_NO_STATE)) { + up_read(&ghvm->status_lock); + ret = gunyah_vm_start(ghvm); + if (ret) + return ret; + /** gunyah_vm_start() is guaranteed to bring status out of + * GUNYAH_RM_VM_STATUS_LOAD, thus infinitely recursive call is not + * possible + */ + return gunyah_vm_ensure_started(ghvm); + } + + /* Unlikely because VM is typically running */ + if (unlikely(ghvm->vm_status != GUNYAH_RM_VM_STATUS_RUNNING)) + ret = -ENODEV; + + up_read(&ghvm->status_lock); + return ret; +} + +static long gunyah_vm_ioctl(struct file *filp, unsigned int cmd, + unsigned long arg) +{ + struct gunyah_vm *ghvm = filp->private_data; + long r; + + switch (cmd) { + case GUNYAH_VM_START: { + r = gunyah_vm_ensure_started(ghvm); + break; + } + default: + r = -ENOTTY; + break; + } + + return r; +} + static int gunyah_vm_release(struct inode *inode, struct file *filp) { struct gunyah_vm *ghvm = filp->private_data; + int ret; + + /** + * We might race with a VM exit notification, but that's ok: + * gh_rm_vm_stop() will just return right away. + */ + if (ghvm->vm_status == GUNYAH_RM_VM_STATUS_RUNNING) + gunyah_vm_stop(ghvm); + + if (ghvm->vm_status != GUNYAH_RM_VM_STATUS_NO_STATE && + ghvm->vm_status != GUNYAH_RM_VM_STATUS_LOAD && + ghvm->vm_status != GUNYAH_RM_VM_STATUS_RESET) { + ret = gunyah_rm_vm_reset(ghvm->rm, ghvm->vmid); + if (ret) + dev_err(ghvm->parent, "Failed to reset the vm: %d\n", + ret); + wait_event(ghvm->vm_status_wait, + ghvm->vm_status == GUNYAH_RM_VM_STATUS_RESET); + } + + if (ghvm->vm_status > GUNYAH_RM_VM_STATUS_NO_STATE) { + gunyah_rm_notifier_unregister(ghvm->rm, &ghvm->nb); + + ret = gunyah_rm_dealloc_vmid(ghvm->rm, ghvm->vmid); + if (ret) + dev_warn(ghvm->parent, + "Failed to deallocate vmid: %d\n", ret); + } gunyah_rm_put(ghvm->rm); kfree(ghvm); @@ -40,6 +236,8 @@ static int gunyah_vm_release(struct inode *inode, struct file *filp) static const struct file_operations gunyah_vm_fops = { .owner = THIS_MODULE, + .unlocked_ioctl = gunyah_vm_ioctl, + .compat_ioctl = compat_ptr_ioctl, .release = gunyah_vm_release, .llseek = noop_llseek, }; diff --git a/drivers/virt/gunyah/vm_mgr.h b/drivers/virt/gunyah/vm_mgr.h index 50790d402676..e6cc9aead0b6 100644 --- a/drivers/virt/gunyah/vm_mgr.h +++ b/drivers/virt/gunyah/vm_mgr.h @@ -7,6 +7,8 @@ #define _GUNYAH_VM_MGR_PRIV_H #include +#include +#include #include @@ -17,12 +19,29 @@ long gunyah_dev_vm_mgr_ioctl(struct gunyah_rm *rm, unsigned int cmd, /** * struct gunyah_vm - Main representation of a Gunyah Virtual machine + * @vmid: Gunyah's VMID for this virtual machine * @rm: Pointer to the resource manager struct to make RM calls * @parent: For logging + * @nb: Notifier block for RM notifications + * @vm_status: Current state of the VM, as last reported by RM + * @vm_status_wait: Wait queue for status @vm_status changes + * @status_lock: Serializing state transitions + * @auth: Authentication mechanism to be used by resource manager when + * launching the VM + * + * Members are grouped by hot path. */ struct gunyah_vm { + u16 vmid; struct gunyah_rm *rm; + + struct notifier_block nb; + enum gunyah_rm_vm_status vm_status; + wait_queue_head_t vm_status_wait; + struct rw_semaphore status_lock; + struct device *parent; + enum gunyah_rm_vm_auth_mechanism auth; }; #endif diff --git a/include/uapi/linux/gunyah.h b/include/uapi/linux/gunyah.h index ac338ec4b85d..31e7f79a6c39 100644 --- a/include/uapi/linux/gunyah.h +++ b/include/uapi/linux/gunyah.h @@ -20,4 +20,9 @@ */ #define GUNYAH_CREATE_VM _IO(GUNYAH_IOCTL_TYPE, 0x0) /* Returns a Gunyah VM fd */ +/* + * ioctls for gunyah-vm fds (returned by GUNYAH_CREATE_VM) + */ +#define GUNYAH_VM_START _IO(GUNYAH_IOCTL_TYPE, 0x3) + #endif