From patchwork Mon Aug 19 12:08:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eslam Elnikety X-Patchwork-Id: 11100759 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42E1013B1 for ; Mon, 19 Aug 2019 12:10:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 319CF283DA for ; Mon, 19 Aug 2019 12:10:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 25E762832D; Mon, 19 Aug 2019 12:10:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_INVALID,DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6C66827FAC for ; Mon, 19 Aug 2019 12:10:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hzgTB-00045k-Gl; Mon, 19 Aug 2019 12:09:05 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hzgT9-00045f-EF for xen-devel@lists.xenproject.org; Mon, 19 Aug 2019 12:09:03 +0000 X-Inumbo-ID: 211ae566-c27a-11e9-8bec-12813bfff9fa Received: from smtp-fw-9102.amazon.com (unknown [207.171.184.29]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 211ae566-c27a-11e9-8bec-12813bfff9fa; Mon, 19 Aug 2019 12:09:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1566216542; x=1597752542; h=from:to:cc:subject:date:message-id:mime-version; bh=QTxb/0qIEprP7GUsr8SHh3WiSudeFiwE1ayGLpx7DVc=; b=akT+G2jYSN2n4F2PjZbuf0nQ/enleJ5dUVd8oCvujjgbwL8k202y4ypI BcAxMTmAiguXTouYjTsLF53S2cTPqgYVhmR65UCrl8lJwJMmgPYU6VQYn dRqrY+eEKrhZrvLmFDd4+MJ0D6X7VkTtQZG+1DCUgG/LMlQSAtofrwo93 k=; X-IronPort-AV: E=Sophos;i="5.64,403,1559520000"; d="scan'208";a="695110408" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 19 Aug 2019 12:08:57 +0000 Received: from EX13MTAUEA001.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com (Postfix) with ESMTPS id 810D3A2488; Mon, 19 Aug 2019 12:08:55 +0000 (UTC) Received: from EX13D08UEE003.ant.amazon.com (10.43.62.118) by EX13MTAUEA001.ant.amazon.com (10.43.61.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 19 Aug 2019 12:08:55 +0000 Received: from EX13MTAUEE001.ant.amazon.com (10.43.62.200) by EX13D08UEE003.ant.amazon.com (10.43.62.118) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 19 Aug 2019 12:08:55 +0000 Received: from dev-dsk-elnikety-1c-28a13cba.eu-west-1.amazon.com (10.15.5.186) by mail-relay.amazon.com (10.43.62.226) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 19 Aug 2019 12:08:55 +0000 Received: by dev-dsk-elnikety-1c-28a13cba.eu-west-1.amazon.com (Postfix, from userid 6438462) id 074B646551; Mon, 19 Aug 2019 12:08:55 +0000 (UTC) From: Eslam Elnikety To: Date: Mon, 19 Aug 2019 12:08:45 +0000 Message-ID: <20190819120845.18846-1-elnikety@amazon.com> X-Mailer: git-send-email 2.15.3.AMZN MIME-Version: 1.0 Precedence: Bulk Subject: [Xen-devel] [PATCH v3] evtchn: Introduce a per-guest knob to control FIFO ABI X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Eslam Elnikety , Tim Deegan , Julien Grall , Jan Beulich , Anthony PERARD , David Woodhouse Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Support for FIFO event channel ABI was first introduced in Xen 4.4 (see 88910061ec6). Make this support tunable, since the choice of which event channel ABI has implications for hibernation. Consider resuming a pre Xen 4.4 hibernated Linux guest. During resume from hibernation, there are two kernels involved: the "boot" kernel and the "resume" kernel. The guest boot kernel may default to FIFO ABI and instruct Xen via an EVTCHNOP_init_control to switch from 2L to FIFO. On the other hand, the resume kernel keeps assuming 2L. This, in turn, results in Xen and the resumed kernel talking past each other (due to different protocols FIFO vs 2L). A knob to control the support level for event channel ABIs per guest would allow an administrator to work around this issue. This patch introduces this knob. In order to announce to guests that the event channel ABI does not support FIFO, the hypervisor returns EPERM on init_control operation. When this operation fails, the guest should continue to use the 2L event channel ABI. For example, in Linux drivers/xen/events/events_base.c: if (fifo_events) ret = xen_evtchn_fifo_init(); if (ret < 0) xen_evtchn_2l_init(); and xen_evtchn_fifo_init fails when EVTCHNOP_init_control fails. This commit does not change the current default behaviour: announce FIFO event channels ABI support for guests unless explicitly stated otherwise at domaincreate. Signed-off-by: Eslam Elnikety --- Changes in v2: - Reworded the commit message - Return EPERM instead of ENOSYS when forcing 2L ABI - Use d->options instead of replicating the flag Changes in v3: - Reworded the commit subject - Check at evtchn_fifo_init_control, rather than EVTCHNOP_init_control --- docs/man/xl.cfg.5.pod.in | 5 +++++ tools/libxl/libxl.h | 8 ++++++++ tools/libxl/libxl_create.c | 5 +++++ tools/libxl/libxl_types.idl | 1 + tools/xl/xl_parse.c | 2 ++ tools/xl/xl_sxp.c | 2 ++ xen/common/event_channel.c | 1 + xen/common/event_fifo.c | 4 ++++ xen/include/public/domctl.h | 3 +++ 9 files changed, 31 insertions(+) diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in index c99d40307e..f204d8b4f0 100644 --- a/docs/man/xl.cfg.5.pod.in +++ b/docs/man/xl.cfg.5.pod.in @@ -1262,6 +1262,11 @@ FIFO-based event channel ABI support up to 131,071 event channels. Other guests are limited to 4095 (64-bit x86 and ARM) or 1023 (32-bit x86). +=item B + +Indicates if support for FIFO event channel ABI is disabled. The default +is false (0). + =item B Specifies the virtual display devices to be provided to the guest. diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h index 9bacfb97f0..9ee80d74a6 100644 --- a/tools/libxl/libxl.h +++ b/tools/libxl/libxl.h @@ -169,6 +169,14 @@ */ #define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1 +/* + * LIBXL_HAVE_BUILDINFO_DISABLE_EVTCHN_FIFO indicates that the + * libxl_domain_build_info structure contains a boolean + * disable_evtchn_fifo which instructs libxl to enable/disable + * support for FIFO event channel ABI at create time. + */ +#define LIBXL_HAVE_BUILDINFO_DISABLE_EVTCHN_FIFO 1 + /* * libxl_domain_build_info has the u.hvm.ms_vm_genid field. */ diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index 03ce166f4f..aa87e45643 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -217,6 +217,8 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc, if (!b_info->event_channels) b_info->event_channels = 1023; + libxl_defbool_setdefault(&b_info->disable_evtchn_fifo, false); + libxl__arch_domain_build_info_setdefault(gc, b_info); libxl_defbool_setdefault(&b_info->dm_restrict, false); @@ -564,6 +566,9 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, libxl_defbool_val(info->oos) ? 0 : XEN_DOMCTL_CDF_oos_off; } + if (libxl_defbool_val(b_info->disable_evtchn_fifo)) + create.flags |= XEN_DOMCTL_CDF_disable_fifo; + /* Ultimately, handle is an array of 16 uint8_t, same as uuid */ libxl_uuid_copy(ctx, (libxl_uuid *)&create.handle, &info->uuid); diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl index b61399ce36..5f30570443 100644 --- a/tools/libxl/libxl_types.idl +++ b/tools/libxl/libxl_types.idl @@ -521,6 +521,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("iomem", Array(libxl_iomem_range, "num_iomem")), ("claim_mode", libxl_defbool), ("event_channels", uint32), + ("disable_evtchn_fifo", libxl_defbool), ("kernel", string), ("cmdline", string), ("ramdisk", string), diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c index e105bda2bb..bcf16c31d4 100644 --- a/tools/xl/xl_parse.c +++ b/tools/xl/xl_parse.c @@ -1496,6 +1496,8 @@ void parse_config_data(const char *config_source, if (!xlu_cfg_get_long(config, "max_event_channels", &l, 0)) b_info->event_channels = l; + xlu_cfg_get_defbool(config, "disable_evtchn_fifo", + &b_info->disable_evtchn_fifo, 0); xlu_cfg_replace_string (config, "kernel", &b_info->kernel, 0); xlu_cfg_replace_string (config, "ramdisk", &b_info->ramdisk, 0); xlu_cfg_replace_string (config, "device_tree", &b_info->device_tree, 0); diff --git a/tools/xl/xl_sxp.c b/tools/xl/xl_sxp.c index 359a001570..52e98c6c61 100644 --- a/tools/xl/xl_sxp.c +++ b/tools/xl/xl_sxp.c @@ -71,6 +71,8 @@ void printf_info_sexp(int domid, libxl_domain_config *d_config, FILE *fh) fprintf(fh, "\t(target_memkb %"PRId64")\n", b_info->target_memkb); fprintf(fh, "\t(nomigrate %s)\n", libxl_defbool_to_string(b_info->disable_migrate)); + fprintf(fh, "\t(disable_evtchn_fifo %s)\n", + libxl_defbool_to_string(b_info->disable_evtchn_fifo)); if (c_info->type == LIBXL_DOMAIN_TYPE_PV && b_info->bootloader) { fprintf(fh, "\t(bootloader %s)\n", b_info->bootloader); diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c index e86e2bfab0..58415e6f93 100644 --- a/xen/common/event_channel.c +++ b/xen/common/event_channel.c @@ -1170,6 +1170,7 @@ long do_event_channel_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) case EVTCHNOP_init_control: { struct evtchn_init_control init_control; + if ( copy_from_guest(&init_control, arg, 1) != 0 ) return -EFAULT; rc = evtchn_fifo_init_control(&init_control); diff --git a/xen/common/event_fifo.c b/xen/common/event_fifo.c index 230f440f14..3a470c97f8 100644 --- a/xen/common/event_fifo.c +++ b/xen/common/event_fifo.c @@ -513,6 +513,10 @@ int evtchn_fifo_init_control(struct evtchn_init_control *init_control) struct vcpu *v; int rc; + /* Fail init_control for domains that cannot use FIFO ABI */ + if ( d->options & XEN_DOMCTL_CDF_disable_fifo ) + return -EPERM; + init_control->link_bits = EVTCHN_FIFO_LINK_BITS; vcpu_id = init_control->vcpu; diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h index 19486d5e32..654b4fdd22 100644 --- a/xen/include/public/domctl.h +++ b/xen/include/public/domctl.h @@ -64,6 +64,9 @@ struct xen_domctl_createdomain { /* Is this a xenstore domain? */ #define _XEN_DOMCTL_CDF_xs_domain 4 #define XEN_DOMCTL_CDF_xs_domain (1U<<_XEN_DOMCTL_CDF_xs_domain) + /* Disable FIFO event channels? */ +#define _XEN_DOMCTL_CDF_disable_fifo 5 +#define XEN_DOMCTL_CDF_disable_fifo (1U<<_XEN_DOMCTL_CDF_disable_fifo) uint32_t flags; /*