From patchwork Mon Jul 1 11:16:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 11025617 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC276746 for ; Mon, 1 Jul 2019 11:19:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC48328640 for ; Mon, 1 Jul 2019 11:19:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B03592866D; Mon, 1 Jul 2019 11:19:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 97EEC28640 for ; Mon, 1 Jul 2019 11:19:07 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hhuJH-0000Hz-CQ; Mon, 01 Jul 2019 11:17:23 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hhuJG-0000Ht-6j for xen-devel@lists.xenproject.org; Mon, 01 Jul 2019 11:17:22 +0000 X-Inumbo-ID: c9f823b7-9bf1-11e9-8980-bc764e045a96 Received: from m9a0001g.houston.softwaregrp.com (unknown [15.124.64.66]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id c9f823b7-9bf1-11e9-8980-bc764e045a96; Mon, 01 Jul 2019 11:17:20 +0000 (UTC) Received: FROM m9a0001g.houston.softwaregrp.com (15.121.0.190) BY m9a0001g.houston.softwaregrp.com WITH ESMTP; Mon, 1 Jul 2019 11:17:13 +0000 Received: from M4W0334.microfocus.com (2002:f78:1192::f78:1192) by M9W0067.microfocus.com (2002:f79:be::f79:be) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1591.10; Mon, 1 Jul 2019 11:16:19 +0000 Received: from NAM02-BL2-obe.outbound.protection.outlook.com (15.124.8.10) by M4W0334.microfocus.com (15.120.17.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1591.10 via Frontend Transport; Mon, 1 Jul 2019 11:16:19 +0000 Received: from BY5PR18MB3394.namprd18.prod.outlook.com (10.255.139.95) by BY5PR18MB3186.namprd18.prod.outlook.com (10.255.137.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2032.20; Mon, 1 Jul 2019 11:16:17 +0000 Received: from BY5PR18MB3394.namprd18.prod.outlook.com ([fe80::2005:4b02:1d60:d1bc]) by BY5PR18MB3394.namprd18.prod.outlook.com ([fe80::2005:4b02:1d60:d1bc%3]) with mapi id 15.20.2008.020; Mon, 1 Jul 2019 11:16:17 +0000 From: Jan Beulich To: "xen-devel@lists.xenproject.org" Thread-Topic: [PATCH v9 01/23] x86emul: support AVX512{F,_VBMI2} compress/expand insns Thread-Index: AQHVL/5nbIXBRx/CpUWnISqWjglTYA== Date: Mon, 1 Jul 2019 11:16:17 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LO2P265CA0110.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::26) To BY5PR18MB3394.namprd18.prod.outlook.com (2603:10b6:a03:194::31) authentication-results: spf=none (sender IP is ) smtp.mailfrom=JBeulich@suse.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [87.234.252.170] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7da2c696-84b6-4a47-ca4b-08d6fe158987 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:BY5PR18MB3186; x-ms-traffictypediagnostic: BY5PR18MB3186: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2399; x-forefront-prvs: 00851CA28B x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(4636009)(346002)(366004)(39860400002)(376002)(396003)(136003)(189003)(199004)(2351001)(8676002)(186003)(102836004)(2906002)(54906003)(256004)(6512007)(14444005)(6916009)(71190400001)(2501003)(66946007)(6506007)(66476007)(66556008)(81156014)(3846002)(53936002)(52116002)(486006)(71200400001)(6116002)(80792005)(11346002)(305945005)(4326008)(53946003)(81166006)(446003)(26005)(386003)(31696002)(73956011)(64756008)(66446008)(2616005)(36756003)(476003)(5640700003)(68736007)(5660300002)(6436002)(31686004)(86362001)(7736002)(66066001)(478600001)(72206003)(6486002)(30864003)(99286004)(8936002)(14454004)(25786009)(316002)(76176011); DIR:OUT; SFP:1102; SCL:1; SRVR:BY5PR18MB3186; H:BY5PR18MB3394.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: suse.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 7hCUPKRCSsZrXPJrgK4WCoa+40z6tIVlGbnnpMHbO3yoP2wlLUM3S1amsdL+jDjFRwv02ycwbaQEnmeBoSKAzNsX0YyACk0JiaP4ei545F0ergSt9RyD51ODD8+8cyNxQ0ix5yWQdvQ4QexRyLzvCcX3vLq+YHEWzQQXQ4FkvGX/RTYU7wMTAX+k7nTf+fzdzMaAxT9gxjXA99obfHTbA8L7PAhllvxmwovQ2XfcLP7DQcD0HDdzqKS+LY4XGLxOImaeeKcuC4zEgfbJIr71oG4ZDLPobRgQlT7Vzmr2kvzDaHu+OyiicK0BETSAyHt3QB26kG5W5G2SlY4rcjkhHv6EPlaWdFMZcrQugLCA89m/OwFhU/Dz49CCvvb7iU83Y5oniNFDZSIJXXwpJGkyrkzpTM3HtIJngTxGHUxB2gM= Content-ID: <22100741BA9D9449870769531E7C35B6@namprd18.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 7da2c696-84b6-4a47-ca4b-08d6fe158987 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2019 11:16:17.7098 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 856b813c-16e5-49a5-85ec-6f081e13b527 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JBeulich@suse.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR18MB3186 X-OriginatorOrg: suse.com Subject: [Xen-devel] [PATCH v9 01/23] x86emul: support AVX512{F, _VBMI2} compress/expand insns X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , RogerPau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Jan Beulich Acked-by: Andrew Cooper --- v9: Re-base. Drop "(solely)" from a comment. v7: Re-base. v6: Re-base. Add tests for the byte/word forms. v5: New. --- a/tools/tests/x86_emulator/evex-disp8.c +++ b/tools/tests/x86_emulator/evex-disp8.c @@ -109,6 +109,7 @@ static const struct test avx512f_all[] = INSN_FP(cmp, 0f, c2), INSN(comisd, 66, 0f, 2f, el, q, el), INSN(comiss, , 0f, 2f, el, d, el), + INSN(compress, 66, 0f38, 8a, vl, sd, el), INSN(cvtdq2pd, f3, 0f, e6, vl_2, d, vl), INSN(cvtdq2ps, , 0f, 5b, vl, d, vl), INSN(cvtpd2dq, f2, 0f, e6, vl, q, vl), @@ -140,6 +141,7 @@ static const struct test avx512f_all[] = INSN(cvtusi2sd, f2, 0f, 7b, el, dq64, el), INSN(cvtusi2ss, f3, 0f, 7b, el, dq64, el), INSN_FP(div, 0f, 5e), + INSN(expand, 66, 0f38, 88, vl, sd, el), INSN(fixupimm, 66, 0f3a, 54, vl, sd, vl), INSN(fixupimm, 66, 0f3a, 55, el, sd, el), INSN(fmadd132, 66, 0f38, 98, vl, sd, vl), @@ -214,6 +216,7 @@ static const struct test avx512f_all[] = INSN(pcmpgtd, 66, 0f, 66, vl, d, vl), INSN(pcmpgtq, 66, 0f38, 37, vl, q, vl), INSN(pcmpu, 66, 0f3a, 1e, vl, dq, vl), + INSN(pcompress, 66, 0f38, 8b, vl, dq, el), INSN(permi2, 66, 0f38, 76, vl, dq, vl), INSN(permi2, 66, 0f38, 77, vl, sd, vl), INSN(permilpd, 66, 0f38, 0d, vl, q, vl), @@ -222,6 +225,7 @@ static const struct test avx512f_all[] = INSN(permilps, 66, 0f3a, 04, vl, d, vl), INSN(permt2, 66, 0f38, 7e, vl, dq, vl), INSN(permt2, 66, 0f38, 7f, vl, sd, vl), + INSN(pexpand, 66, 0f38, 89, vl, dq, el), INSN(pmaxs, 66, 0f38, 3d, vl, dq, vl), INSN(pmaxu, 66, 0f38, 3f, vl, dq, vl), INSN(pmins, 66, 0f38, 39, vl, dq, vl), @@ -509,6 +513,11 @@ static const struct test avx512_vbmi_all INSN(permt2b, 66, 0f38, 7d, vl, b, vl), }; +static const struct test avx512_vbmi2_all[] = { + INSN(pcompress, 66, 0f38, 63, vl, bw, el), + INSN(pexpand, 66, 0f38, 62, vl, bw, el), +}; + static const unsigned char vl_all[] = { VL_512, VL_128, VL_256 }; static const unsigned char vl_128[] = { VL_128 }; static const unsigned char vl_no128[] = { VL_512, VL_256 }; @@ -865,4 +874,5 @@ void evex_disp8_test(void *instr, struct RUN(avx512dq, 512); RUN(avx512er, 512); RUN(avx512_vbmi, all); + RUN(avx512_vbmi2, all); } --- a/tools/tests/x86_emulator/test_x86_emulator.c +++ b/tools/tests/x86_emulator/test_x86_emulator.c @@ -3995,6 +3995,227 @@ int main(int argc, char **argv) else printf("skipped\n"); + /* + * The following compress/expand tests are not only making sure the + * accessed data is correct, but they also verify (by placing operands + * on the mapping boundaries) that elements controlled by clear mask + * bits don't get accessed. + */ + if ( stack_exec && cpu_has_avx512f ) + { + decl_insn(vpcompressd); + decl_insn(vpcompressq); + decl_insn(vpexpandd); + decl_insn(vpexpandq); + static const struct { + unsigned int d[16]; + } dsrc = { { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 } }; + static const struct { + unsigned long long q[8]; + } qsrc = { { 0, 1, 2, 3, 4, 5, 6, 7 } }; + unsigned int *ptr = res + MMAP_SZ / sizeof(*res) - 32; + + printf("%-40s", "Testing vpcompressd %zmm1,24*4(%ecx){%k2}..."); + asm volatile ( "kmovw %1, %%k2\n\t" + "vmovdqu32 %2, %%zmm1\n" + put_insn(vpcompressd, + "vpcompressd %%zmm1, 24*4(%0)%{%%k2%}") + :: "c" (NULL), "r" (0x55aa), "m" (dsrc) ); + + memset(ptr, 0xdb, 32 * 4); + set_insn(vpcompressd); + regs.ecx = (unsigned long)ptr; + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpcompressd) || + memcmp(ptr, ptr + 8, 16 * 4) ) + goto fail; + for ( i = 0; i < 4; ++i ) + if ( ptr[24 + i] != 2 * i + 1 ) + goto fail; + for ( ; i < 8; ++i ) + if ( ptr[24 + i] != 2 * i ) + goto fail; + printf("okay\n"); + + printf("%-40s", "Testing vpexpandd 8*4(%edx),%zmm3{%k2}{z}..."); + asm volatile ( "vpternlogd $0x81, %%zmm3, %%zmm3, %%zmm3\n" + put_insn(vpexpandd, + "vpexpandd 8*4(%0), %%zmm3%{%%k2%}%{z%}") + :: "d" (NULL) ); + set_insn(vpexpandd); + regs.edx = (unsigned long)(ptr + 16); + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpexpandd) ) + goto fail; + asm ( "vmovdqa32 %%zmm1, %%zmm2%{%%k2%}%{z%}\n\t" + "vpcmpeqd %%zmm2, %%zmm3, %%k0\n\t" + "kmovw %%k0, %0" + : "=r" (rc) ); + if ( rc != 0xffff ) + goto fail; + printf("okay\n"); + + printf("%-40s", "Testing vpcompressq %zmm4,12*8(%edx){%k3}..."); + asm volatile ( "kmovw %1, %%k3\n\t" + "vmovdqu64 %2, %%zmm4\n" + put_insn(vpcompressq, + "vpcompressq %%zmm4, 12*8(%0)%{%%k3%}") + :: "d" (NULL), "r" (0x5a), "m" (qsrc) ); + + memset(ptr, 0xdb, 16 * 8); + set_insn(vpcompressq); + regs.edx = (unsigned long)ptr; + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpcompressq) || + memcmp(ptr, ptr + 8, 8 * 8) ) + goto fail; + for ( i = 0; i < 2; ++i ) + { + if ( ptr[(12 + i) * 2] != 2 * i + 1 || + ptr[(12 + i) * 2 + 1] ) + goto fail; + } + for ( ; i < 4; ++i ) + { + if ( ptr[(12 + i) * 2] != 2 * i || + ptr[(12 + i) * 2 + 1] ) + goto fail; + } + printf("okay\n"); + + printf("%-40s", "Testing vpexpandq 4*8(%ecx),%zmm5{%k3}{z}..."); + asm volatile ( "vpternlogq $0x81, %%zmm5, %%zmm5, %%zmm5\n" + put_insn(vpexpandq, + "vpexpandq 4*8(%0), %%zmm5%{%%k3%}%{z%}") + :: "c" (NULL) ); + set_insn(vpexpandq); + regs.ecx = (unsigned long)(ptr + 16); + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpexpandq) ) + goto fail; + asm ( "vmovdqa64 %%zmm4, %%zmm6%{%%k3%}%{z%}\n\t" + "vpcmpeqq %%zmm5, %%zmm6, %%k0\n\t" + "kmovw %%k0, %0" + : "=r" (rc) ); + if ( rc != 0xff ) + goto fail; + printf("okay\n"); + } + +#if __GNUC__ > 7 /* can't check for __AVX512VBMI2__ here */ + if ( stack_exec && cpu_has_avx512_vbmi2 ) + { + decl_insn(vpcompressb); + decl_insn(vpcompressw); + decl_insn(vpexpandb); + decl_insn(vpexpandw); + static const struct { + unsigned char b[64]; + } bsrc = { { 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15, + 16, 17, 18, 19, 20, 21, 22, 23, + 24, 25, 26, 27, 28, 29, 30, 31, + 32, 33, 34, 35, 36, 37, 38, 39, + 40, 41, 42, 43, 44, 45, 46, 47, + 48, 49, 50, 51, 52, 53, 54, 55, + 56, 57, 58, 59, 60, 61, 62, 63 } }; + static const struct { + unsigned short w[32]; + } wsrc = { { 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15, + 16, 17, 18, 19, 20, 21, 22, 23, + 24, 25, 26, 27, 28, 29, 30, 31 } }; + unsigned char *ptr = (void *)res + MMAP_SZ - 128; + unsigned long long w = 0x55555555aaaaaaaaULL; + + printf("%-40s", "Testing vpcompressb %zmm1,96*1(%ecx){%k2}..."); + asm volatile ( "kmovq %1, %%k2\n\t" + "vmovdqu8 %2, %%zmm1\n" + put_insn(vpcompressb, + "vpcompressb %%zmm1, 96*1(%0)%{%%k2%}") + :: "c" (NULL), "m" (w), "m" (bsrc) ); + + memset(ptr, 0xdb, 128 * 1); + set_insn(vpcompressb); + regs.ecx = (unsigned long)ptr; + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpcompressb) || + memcmp(ptr, ptr + 32, 64 * 1) ) + goto fail; + for ( i = 0; i < 16; ++i ) + if ( ptr[96 + i] != 2 * i + 1 ) + goto fail; + for ( ; i < 32; ++i ) + if ( ptr[96 + i] != 2 * i ) + goto fail; + printf("okay\n"); + + printf("%-40s", "Testing vpexpandb 32*1(%edx),%zmm3{%k2}{z}..."); + asm volatile ( "vpternlogd $0x81, %%zmm3, %%zmm3, %%zmm3\n" + put_insn(vpexpandb, + "vpexpandb 32*1(%0), %%zmm3%{%%k2%}%{z%}") + :: "d" (NULL) ); + set_insn(vpexpandb); + regs.edx = (unsigned long)(ptr + 64); + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpexpandb) ) + goto fail; + asm ( "vmovdqu8 %%zmm1, %%zmm2%{%%k2%}%{z%}\n\t" + "vpcmpeqb %%zmm2, %%zmm3, %%k0\n\t" + "kmovq %%k0, %0" + : "=m" (w) ); + if ( w != 0xffffffffffffffffULL ) + goto fail; + printf("okay\n"); + + printf("%-40s", "Testing vpcompressw %zmm4,48*2(%edx){%k3}..."); + asm volatile ( "kmovd %1, %%k3\n\t" + "vmovdqu16 %2, %%zmm4\n" + put_insn(vpcompressw, + "vpcompressw %%zmm4, 48*2(%0)%{%%k3%}") + :: "d" (NULL), "r" (0x5555aaaa), "m" (wsrc) ); + + memset(ptr, 0xdb, 64 * 2); + set_insn(vpcompressw); + regs.edx = (unsigned long)ptr; + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpcompressw) || + memcmp(ptr, ptr + 32, 32 * 2) ) + goto fail; + for ( i = 0; i < 8; ++i ) + { + if ( ptr[(48 + i) * 2] != 2 * i + 1 || + ptr[(48 + i) * 2 + 1] ) + goto fail; + } + for ( ; i < 16; ++i ) + { + if ( ptr[(48 + i) * 2] != 2 * i || + ptr[(48 + i) * 2 + 1] ) + goto fail; + } + printf("okay\n"); + + printf("%-40s", "Testing vpexpandw 16*2(%ecx),%zmm5{%k3}{z}..."); + asm volatile ( "vpternlogd $0x81, %%zmm5, %%zmm5, %%zmm5\n" + put_insn(vpexpandw, + "vpexpandw 16*2(%0), %%zmm5%{%%k3%}%{z%}") + :: "c" (NULL) ); + set_insn(vpexpandw); + regs.ecx = (unsigned long)(ptr + 64); + rc = x86_emulate(&ctxt, &emulops); + if ( rc != X86EMUL_OKAY || !check_eip(vpexpandw) ) + goto fail; + asm ( "vmovdqu16 %%zmm4, %%zmm6%{%%k3%}%{z%}\n\t" + "vpcmpeqw %%zmm5, %%zmm6, %%k0\n\t" + "kmovq %%k0, %0" + : "=m" (w) ); + if ( w != 0xffffffff ) + goto fail; + printf("okay\n"); + } +#endif + #undef decl_insn #undef put_insn #undef set_insn --- a/tools/tests/x86_emulator/x86-emulate.h +++ b/tools/tests/x86_emulator/x86-emulate.h @@ -59,6 +59,9 @@ (type *)((char *)mptr__ - offsetof(type, member)); \ }) +#define hweight32 __builtin_popcount +#define hweight64 __builtin_popcountll + #define is_canonical_address(x) (((int64_t)(x) >> 47) == ((int64_t)(x) >> 63)) extern uint32_t mxcsr_mask; @@ -138,6 +141,7 @@ static inline bool xcr0_mask(uint64_t ma #define cpu_has_avx512bw (cp.feat.avx512bw && xcr0_mask(0xe6)) #define cpu_has_avx512vl (cp.feat.avx512vl && xcr0_mask(0xe6)) #define cpu_has_avx512_vbmi (cp.feat.avx512_vbmi && xcr0_mask(0xe6)) +#define cpu_has_avx512_vbmi2 (cp.feat.avx512_vbmi2 && xcr0_mask(0xe6)) #define cpu_has_xgetbv1 (cpu_has_xsave && cp.xstate.xgetbv1) --- a/xen/arch/x86/x86_emulate/x86_emulate.c +++ b/xen/arch/x86/x86_emulate/x86_emulate.c @@ -482,6 +482,8 @@ static const struct ext0f38_table { [0x59] = { .simd_size = simd_other, .two_op = 1, .d8s = 3 }, [0x5a] = { .simd_size = simd_128, .two_op = 1, .d8s = 4 }, [0x5b] = { .simd_size = simd_256, .two_op = 1, .d8s = d8s_vl_by_2 }, + [0x62] = { .simd_size = simd_packed_int, .two_op = 1, .d8s = d8s_bw }, + [0x63] = { .simd_size = simd_packed_int, .to_mem = 1, .two_op = 1, .d8s = d8s_bw }, [0x75 ... 0x76] = { .simd_size = simd_packed_int, .d8s = d8s_vl }, [0x77] = { .simd_size = simd_packed_fp, .d8s = d8s_vl }, [0x78] = { .simd_size = simd_other, .two_op = 1 }, @@ -489,6 +491,10 @@ static const struct ext0f38_table { [0x7a ... 0x7c] = { .simd_size = simd_none, .two_op = 1 }, [0x7d ... 0x7e] = { .simd_size = simd_packed_int, .d8s = d8s_vl }, [0x7f] = { .simd_size = simd_packed_fp, .d8s = d8s_vl }, + [0x88] = { .simd_size = simd_packed_fp, .two_op = 1, .d8s = d8s_dq }, + [0x89] = { .simd_size = simd_packed_int, .two_op = 1, .d8s = d8s_dq }, + [0x8a] = { .simd_size = simd_packed_fp, .to_mem = 1, .two_op = 1, .d8s = d8s_dq }, + [0x8b] = { .simd_size = simd_packed_int, .to_mem = 1, .two_op = 1, .d8s = d8s_dq }, [0x8c] = { .simd_size = simd_packed_int }, [0x8d] = { .simd_size = simd_packed_int, .d8s = d8s_vl }, [0x8e] = { .simd_size = simd_packed_int, .to_mem = 1 }, @@ -1868,6 +1874,7 @@ in_protmode( #define vcpu_has_avx512bw() (ctxt->cpuid->feat.avx512bw) #define vcpu_has_avx512vl() (ctxt->cpuid->feat.avx512vl) #define vcpu_has_avx512_vbmi() (ctxt->cpuid->feat.avx512_vbmi) +#define vcpu_has_avx512_vbmi2() (ctxt->cpuid->feat.avx512_vbmi2) #define vcpu_has_rdpid() (ctxt->cpuid->feat.rdpid) #define vcpu_must_have(feat) \ @@ -8881,6 +8888,36 @@ x86_emulate( generate_exception_if(ea.type != OP_MEM || !vex.l || vex.w, EXC_UD); goto simd_0f_avx2; + case X86EMUL_OPC_EVEX_66(0x0f38, 0x62): /* vpexpand{b,w} [xyz]mm/mem,[xyz]mm{k} */ + case X86EMUL_OPC_EVEX_66(0x0f38, 0x63): /* vpcompress{b,w} [xyz]mm,[xyz]mm/mem{k} */ + host_and_vcpu_must_have(avx512_vbmi2); + elem_bytes = 1 << evex.w; + /* fall through */ + case X86EMUL_OPC_EVEX_66(0x0f38, 0x88): /* vexpandp{s,d} [xyz]mm/mem,[xyz]mm{k} */ + case X86EMUL_OPC_EVEX_66(0x0f38, 0x89): /* vpexpand{d,q} [xyz]mm/mem,[xyz]mm{k} */ + case X86EMUL_OPC_EVEX_66(0x0f38, 0x8a): /* vcompressp{s,d} [xyz]mm,[xyz]mm/mem{k} */ + case X86EMUL_OPC_EVEX_66(0x0f38, 0x8b): /* vpcompress{d,q} [xyz]mm,[xyz]mm/mem{k} */ + host_and_vcpu_must_have(avx512f); + generate_exception_if(evex.brs, EXC_UD); + avx512_vlen_check(false); + /* + * For the respective code below the main switch() to work we need to + * compact op_mask here: Memory accesses are non-sparse even if the + * mask register has sparsely set bits. + */ + if ( likely(fault_suppression) ) + { + n = 1 << ((b & 8 ? 2 : 4) + evex.lr - evex.w); + EXPECT(elem_bytes > 0); + ASSERT(op_bytes == n * elem_bytes); + op_mask &= ~0ULL >> (64 - n); + n = hweight64(op_mask); + op_bytes = n * elem_bytes; + if ( n ) + op_mask = ~0ULL >> (64 - n); + } + goto simd_zmm; + case X86EMUL_OPC_EVEX_66(0x0f38, 0x75): /* vpermi2{b,w} [xyz]mm/mem,[xyz]mm,[xyz]mm{k} */ case X86EMUL_OPC_EVEX_66(0x0f38, 0x7d): /* vpermt2{b,w} [xyz]mm/mem,[xyz]mm,[xyz]mm{k} */ case X86EMUL_OPC_EVEX_66(0x0f38, 0x8d): /* vperm{b,w} [xyz]mm/mem,[xyz]mm,[xyz]mm{k} */ --- a/xen/include/asm-x86/cpufeature.h +++ b/xen/include/asm-x86/cpufeature.h @@ -108,6 +108,7 @@ /* CPUID level 0x00000007:0.ecx */ #define cpu_has_avx512_vbmi boot_cpu_has(X86_FEATURE_AVX512_VBMI) +#define cpu_has_avx512_vbmi2 boot_cpu_has(X86_FEATURE_AVX512_VBMI2) #define cpu_has_rdpid boot_cpu_has(X86_FEATURE_RDPID) /* CPUID level 0x80000007.edx */ --- a/xen/include/public/arch-x86/cpufeatureset.h +++ b/xen/include/public/arch-x86/cpufeatureset.h @@ -228,6 +228,7 @@ XEN_CPUFEATURE(AVX512_VBMI, 6*32+ 1) / XEN_CPUFEATURE(UMIP, 6*32+ 2) /*S User Mode Instruction Prevention */ XEN_CPUFEATURE(PKU, 6*32+ 3) /*H Protection Keys for Userspace */ XEN_CPUFEATURE(OSPKE, 6*32+ 4) /*! OS Protection Keys Enable */ +XEN_CPUFEATURE(AVX512_VBMI2, 6*32+ 6) /*A Additional AVX-512 Vector Byte Manipulation Instrs */ XEN_CPUFEATURE(AVX512_VPOPCNTDQ, 6*32+14) /*A POPCNT for vectors of DW/QW */ XEN_CPUFEATURE(RDPID, 6*32+22) /*A RDPID instruction */ --- a/xen/tools/gen-cpuid.py +++ b/xen/tools/gen-cpuid.py @@ -265,10 +265,10 @@ def crunch_numbers(state): AVX512BW, AVX512VL, AVX512_4VNNIW, AVX512_4FMAPS, AVX512_VPOPCNTDQ], - # AVX512 extensions acting solely on vectors of bytes/words are made + # AVX512 extensions acting on vectors of bytes/words are made # dependents of AVX512BW (as to requiring wider than 16-bit mask # registers), despite the SDM not formally making this connection. - AVX512BW: [AVX512_BF16, AVX512_VBMI], + AVX512BW: [AVX512_BF16, AVX512_VBMI, AVX512_VBMI2], # The features: # * Single Thread Indirect Branch Predictors