From patchwork Mon Apr 22 18:14:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53CAFC10F1A for ; Mon, 22 Apr 2024 18:14:54 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710193.1109312 (Exim 4.92) (envelope-from ) id 1ryyBe-0000iX-TO; Mon, 22 Apr 2024 18:14:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710193.1109312; Mon, 22 Apr 2024 18:14:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBe-0000g7-N5; Mon, 22 Apr 2024 18:14:42 +0000 Received: by outflank-mailman (input) for mailman id 710193; Mon, 22 Apr 2024 18:14:41 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBd-0000Oz-FA for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:41 +0000 Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [2a00:1450:4864:20::530]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 2eaaf11d-00d4-11ef-b4bb-af5377834399; Mon, 22 Apr 2024 20:14:38 +0200 (CEST) Received: by mail-ed1-x530.google.com with SMTP id 4fb4d7f45d1cf-56e136cbcecso6788378a12.3 for ; Mon, 22 Apr 2024 11:14:38 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:37 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2eaaf11d-00d4-11ef-b4bb-af5377834399 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809678; x=1714414478; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6RRljp8ilqBOCq7+riUQu2IxAcNGdCvaCKnCfHGbuws=; b=u73wAZeaC42onoCzpA2HsXTCOhDXT4Y7CH8on5ZJOI8H5KnR91YSaSgn0cO63+pCc8 jL/8itLcWkrLbZvNjWphMaib1bQtDNmUfH4GmUiUPCUzryL90KpKuft+CxGDVaXjneWo 2HUUrh5sErCC3FziPwUDMLMfP8O0PGIaGsK8c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809678; x=1714414478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6RRljp8ilqBOCq7+riUQu2IxAcNGdCvaCKnCfHGbuws=; b=RHJV3rB8EF5RXAKd3cKU1XgWmG82ie52JZqqRVOvuz2un+K0lD+T5vdkXjRU05CvDd hVhU2jdFCtNuPqXi5OtPE5EkjhkIDO0otbnSCy+yG7ZSb3smoxFcGksF3VmKVvyN9L4k 76dBnjJUPZ+4ZCu+LiaJ72+exu8ShsWmM5pTLNC1+V8tTKiMqJWSlZ10Nub/ljiSNdXa 2YPKEXs1yub8csebe7P/4k+gWjBIwBTWCS4ld4nJ9f5lYVA6CdLiTY+b7CitZPghyi28 YXjYYzvzeKr0hA0B8n4jRDi2unDiee6ffRqtX5g0ec8i7ThXipHB01Nf5hB84ynZuNTw nxfQ== X-Gm-Message-State: AOJu0Yw9NidK+KP2Q57ra+jcEv4PrqDfveRTI48HjHxnDVyZaLEcL763 nawJjWanaRICJqZ3tPcSAoSX1mx2iuUTz0zI59J+AoCKLB0fr2yDDFK9qFTAjLQejjmI3adY+Mg 9nnI= X-Google-Smtp-Source: AGHT+IGXtcazIcbQnNsHc45oDqFJsnTRiN7QccNtav/aj2Lc2owoQkx7kwbiFs6TnAPwRJubHQtmzg== X-Received: by 2002:a17:906:eb4f:b0:a58:725b:9fea with SMTP id mc15-20020a170906eb4f00b00a58725b9feamr846018ejb.21.1713809677900; Mon, 22 Apr 2024 11:14:37 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 1/6] x86: Introduce x86_decode_lite() Date: Mon, 22 Apr 2024 19:14:29 +0100 Message-Id: <20240422181434.3463252-2-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 In order to relocate all IP-relative fields in an alternative replacement block, we need to decode the instructions enough to obtain their length and any relative fields. Full x86_decode() is far too heavyweight, so introduce a minimal form which can make several simplifying assumptions. This logic is sufficient to decode all alternative blocks that exist in Xen right now. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/x86_emulate/Makefile | 1 + xen/arch/x86/x86_emulate/decode-lite.c | 245 +++++++++++++++++++++++++ xen/arch/x86/x86_emulate/private.h | 2 + xen/arch/x86/x86_emulate/x86_emulate.h | 17 ++ 4 files changed, 265 insertions(+) create mode 100644 xen/arch/x86/x86_emulate/decode-lite.c diff --git a/xen/arch/x86/x86_emulate/Makefile b/xen/arch/x86/x86_emulate/Makefile index 2e20d65d788a..f06731913d67 100644 --- a/xen/arch/x86/x86_emulate/Makefile +++ b/xen/arch/x86/x86_emulate/Makefile @@ -3,6 +3,7 @@ obj-y += 0fae.o obj-y += 0fc7.o obj-y += blk.o obj-y += decode.o +obj-y += decode-lite.o obj-$(CONFIG_HVM) += fpu.o obj-y += util.o obj-y += util-xen.o diff --git a/xen/arch/x86/x86_emulate/decode-lite.c b/xen/arch/x86/x86_emulate/decode-lite.c new file mode 100644 index 000000000000..050b0d8cf4a8 --- /dev/null +++ b/xen/arch/x86/x86_emulate/decode-lite.c @@ -0,0 +1,245 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include "private.h" + +#define Imm8 (1 << 0) +#define Imm (1 << 1) +#define Branch (1 << 5) /* ... that we care about */ +/* ModRM (1 << 6) */ +#define Known (1 << 7) + +#define ALU_OPS \ + (Known|ModRM), \ + (Known|ModRM), \ + (Known|ModRM), \ + (Known|ModRM), \ + (Known|Imm8), \ + (Known|Imm) + +static const uint8_t init_or_livepatch_const onebyte[256] = { + [0x00] = ALU_OPS, /* ADD */ [0x08] = ALU_OPS, /* OR */ + [0x10] = ALU_OPS, /* ADC */ [0x18] = ALU_OPS, /* SBB */ + [0x20] = ALU_OPS, /* AND */ [0x28] = ALU_OPS, /* SUB */ + [0x30] = ALU_OPS, /* XOR */ [0x38] = ALU_OPS, /* CMP */ + + [0x50 ... 0x5f] = (Known), /* PUSH/POP %reg */ + + [0x70 ... 0x7f] = (Known|Branch|Imm8), /* Jcc disp8 */ + [0x80] = (Known|ModRM|Imm8), + [0x81] = (Known|ModRM|Imm), + + [0x83] = (Known|ModRM|Imm8), + [0x84 ... 0x8e] = (Known|ModRM), /* TEST/XCHG/MOV/MOV-SREG/LEA r/rm */ + + [0xb0 ... 0xb7] = (Known|Imm8), /* MOV $imm8, %reg */ + [0xb8 ... 0xbf] = (Known|Imm), /* MOV $imm32, %reg */ + + [0xcc] = (Known), /* INT3 */ + [0xcd] = (Known|Imm8), /* INT $imm8 */ + + [0xe8 ... 0xe9] = (Known|Branch|Imm), /* CALL/JMP disp32 */ + [0xeb] = (Known|Branch|Imm8), /* JMP disp8 */ + + [0xf1] = (Known), /* ICEBP */ + [0xf4] = (Known), /* HLT */ + [0xf5] = (Known), /* CMC */ + [0xf6 ... 0xf7] = (Known|ModRM), /* Grp3 */ + [0xf8 ... 0xfd] = (Known), /* CLC ... STD */ + [0xfe ... 0xff] = (Known|ModRM), /* Grp4 */ +}; +static const uint8_t init_or_livepatch_const twobyte[256] = { + [0x00 ... 0x01] = (Known|ModRM), /* Grp6/Grp7 */ + + [0x18 ... 0x1f] = (Known|ModRM), /* Grp16 (Hint Nop) */ + + [0x20 ... 0x23] = (Known|ModRM), /* MOV CR/DR */ + + [0x30 ... 0x34] = (Known), /* WRMSR ... RDPMC */ + [0x40 ... 0x4f] = (Known|ModRM), /* CMOVcc */ + + [0x80 ... 0x8f] = (Known|Branch|Imm), /* Jcc disp32 */ + [0x90 ... 0x9f] = (Known|ModRM), /* SETcc */ + + [0xab] = (Known|ModRM), /* BTS */ + [0xac] = (Known|ModRM|Imm8), /* SHRD $imm8 */ + [0xad ... 0xaf] = (Known|ModRM), /* SHRD %cl / Grp15 / IMUL */ + + [0xb8 ... 0xb9] = (Known|ModRM), /* POPCNT/Grp10 (UD1) */ + [0xba] = (Known|ModRM|Imm8), /* Grp8 */ + [0xbb ... 0xbf] = (Known|ModRM), /* BSR/BSF/BSR/MOVSX */ +}; + +/* + * Bare minimum x86 instruction decoder to parse the alternative replacement + * instructions and locate the IP-relative references that may need updating. + * + * These are: + * - disp8/32 from near branches + * - RIP-relative memory references + * + * The following simplifications are used: + * - All code is 64bit, and the instruction stream is safe to read. + * - The 67 prefix is not implemented, so the address size is only 64bit. + * + * Inputs: + * @ip The position to start decoding from. + * @end End of the replacement block. Exceeding this is considered an error. + * + * Returns: x86_decode_lite_t + * - On failure, length of -1. + * - On success, length > 0 and REL_TYPE_*. For REL_TYPE != NONE, rel points + * at the relative field in the instruction stream. + */ +x86_decode_lite_t init_or_livepatch x86_decode_lite(void *ip, void *end) +{ + void *start = ip, *rel = NULL; + unsigned int opc, type = REL_TYPE_NONE; + uint8_t b, d, osize = 4; + +#define OPC_TWOBYTE (1 << 8) + + /* Mutates IP, uses END. */ +#define FETCH(ty) \ + ({ \ + ty _val; \ + \ + if ( (ip + sizeof(ty)) > end ) \ + goto overrun; \ + _val = *(ty *)ip; \ + ip += sizeof(ty); \ + _val; \ + }) + + for ( ;; ) /* Prefixes */ + { + switch ( b = FETCH(uint8_t) ) + { + case 0x26: /* ES override */ + case 0x2e: /* CS override */ + case 0x36: /* DS override */ + case 0x3e: /* SS override */ + case 0x64: /* FS override */ + case 0x65: /* GS override */ + case 0xf0: /* Lock */ + case 0xf2: /* REPNE */ + case 0xf3: /* REP */ + break; + + case 0x66: /* Operand size override */ + osize = 2; + break; + + /* case 0x67: Address size override, not implemented */ + + case 0x40 ... 0x4f: /* REX */ + continue; + + default: + goto prefixes_done; + } + } + prefixes_done: + + /* Fetch the main opcode byte(s) */ + if ( b == 0x0f ) + { + b = FETCH(uint8_t); + opc = OPC_TWOBYTE | b; + + d = twobyte[b]; + } + else + { + opc = b; + d = onebyte[b]; + } + + if ( unlikely(!(d & Known)) ) + goto unknown; + + if ( d & ModRM ) + { + uint8_t modrm = FETCH(uint8_t); + uint8_t mod = modrm >> 6; + uint8_t reg = (modrm >> 3) & 7; + uint8_t rm = modrm & 7; + + /* ModRM/SIB decode */ + if ( mod == 0 && rm == 5 ) /* RIP relative */ + { + rel = ip; + type = REL_TYPE_d32; + FETCH(uint32_t); + } + else if ( mod != 3 && rm == 4 ) /* SIB */ + FETCH(uint8_t); + + if ( mod == 1 ) /* disp8 */ + FETCH(uint8_t); + else if ( mod == 2 ) /* disp32 */ + FETCH(uint32_t); + + /* ModRM based decode adjustements */ + switch ( opc ) + { + case 0xf6: /* Grp3 TEST(s) have extra Imm8 */ + if ( reg == 0 || reg == 1 ) + d |= Imm8; + break; + case 0xf7: /* Grp3 TEST(s) have extra Imm */ + if ( reg == 0 || reg == 1 ) + d |= Imm; + break; + } + } + + if ( d & Branch ) + { + /* + * Don't tolerate 66-prefixed call/jmp in alternatives. Some are + * genuinely decoded differently between Intel and AMD CPUs. + */ + if ( osize != 4 ) + goto bad_osize; + + rel = ip; + type = (d & Imm8) ? REL_TYPE_d8 : REL_TYPE_d32; + } + + if ( d & (Imm|Imm8) ) + { + if ( d & Imm8 ) + osize = 1; + + switch ( osize ) + { + case 1: FETCH(uint8_t); break; + case 2: FETCH(uint16_t); break; + case 4: FETCH(uint32_t); break; + default: goto bad_osize; + } + } + + return (x86_decode_lite_t){ ip - start, type, rel }; + + bad_osize: + printk(XENLOG_ERR "%s() Bad osize %u in %*ph\n", + __func__, osize, + (int)(unsigned long)(end - start), start); + return (x86_decode_lite_t){ -1, REL_TYPE_NONE, NULL }; + + unknown: + printk(XENLOG_ERR "%s() Unknown opcode in %*ph <%02x> %*ph\n", + __func__, + (int)(unsigned long)(ip - 1 - start), start, b, + (int)(unsigned long)(end - ip), ip); + return (x86_decode_lite_t){ -1, REL_TYPE_NONE, NULL }; + + overrun: + printk(XENLOG_ERR "%s() Decode overrun, got %*ph\n", + __func__, + (int)(unsigned long)(end - start), start); + return (x86_decode_lite_t){ -1, REL_TYPE_NONE, NULL }; + +#undef FETCH +} diff --git a/xen/arch/x86/x86_emulate/private.h b/xen/arch/x86/x86_emulate/private.h index 0fa26ba00aa5..792b04889ce6 100644 --- a/xen/arch/x86/x86_emulate/private.h +++ b/xen/arch/x86/x86_emulate/private.h @@ -9,7 +9,9 @@ #ifdef __XEN__ # include +# include # include +# include # include # include # include diff --git a/xen/arch/x86/x86_emulate/x86_emulate.h b/xen/arch/x86/x86_emulate/x86_emulate.h index d92be69d84d9..b7cbcd8b70ba 100644 --- a/xen/arch/x86/x86_emulate/x86_emulate.h +++ b/xen/arch/x86/x86_emulate/x86_emulate.h @@ -842,4 +842,21 @@ static inline void x86_emul_reset_event(struct x86_emulate_ctxt *ctxt) ctxt->event = (struct x86_event){}; } +/* + * x86_decode_lite(). Very minimal decoder for managing alternatives. + * + * @len is -1 on error, or positive on success. If the instruction has a + * relative field, REL_TYPE_* gives the size, and @rel points at the field. + */ +typedef struct { + int8_t len; + uint8_t rel_type; +#define REL_TYPE_NONE 0 +#define REL_TYPE_d8 1 +#define REL_TYPE_d32 2 + void *rel; +} x86_decode_lite_t; + +x86_decode_lite_t x86_decode_lite(void *ip, void *end); + #endif /* __X86_EMULATE_H__ */ From patchwork Mon Apr 22 18:14:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D050C1746D for ; Mon, 22 Apr 2024 18:14:55 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710191.1109300 (Exim 4.92) (envelope-from ) id 1ryyBe-0000aL-9I; Mon, 22 Apr 2024 18:14:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710191.1109300; Mon, 22 Apr 2024 18:14:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBe-0000aE-63; Mon, 22 Apr 2024 18:14:42 +0000 Received: by outflank-mailman (input) for mailman id 710191; Mon, 22 Apr 2024 18:14:40 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBc-0000IX-66 for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:40 +0000 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [2a00:1450:4864:20::430]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 2edcb1ec-00d4-11ef-909a-e314d9c70b13; Mon, 22 Apr 2024 20:14:39 +0200 (CEST) Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-34b1e35155aso1321394f8f.3 for ; Mon, 22 Apr 2024 11:14:39 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:38 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2edcb1ec-00d4-11ef-909a-e314d9c70b13 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809678; x=1714414478; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MQHzbKjDaYMcDY7B8SUQVz+z/Tfqym852xnTGj/zaZg=; b=fkoAkh4o4FnDA99uQNTZb7WraJWXKC2aG6unhYS0rrZIufQCDn5N70MWM/DTyZSA5z NeAjzvm/PJimblHVJgb4bHXzM9Rr0w7UkGOTJJ+nuzOZmXh9lx9GQJw22NhP/Tj8+/5R ONR1uM/azVB0mFU3xpualaMwOxWW1MqD8zjLU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809678; x=1714414478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MQHzbKjDaYMcDY7B8SUQVz+z/Tfqym852xnTGj/zaZg=; b=Z7KNvKuGyja0RlV16mPa1yOseYZr+SWfrDlIsD7bMyGklWTHPTswUlpk60ISuWE/vP dc6QSsNuoPQANdB/hkBgM2FPllFzgn/iebrIsBhUzxbZQxBySnuJdu4AgoZDcGQqRIRQ wwni3mbGpvxsTfsAZxYhWLO2I8FnlyxRVcZgS8DftwgzDcgfCVGywSAhYZc0tCUXvTW6 TzsoOdz2cmRbtRxA38NEkKe4bW6EofTpZ2tXGJHDQ65YZnfFRg8cywtAJkZG367sjW2F 4oMZ7mch5BHDJNvmkGaZiVm+ccdGx3XhC5zxiXoB3fMM5PsQmBTUg7ezuShTlyJshJdr 7Cag== X-Gm-Message-State: AOJu0YxiIHH3elVtiVQoD3vnH7iU0qSBo0jS+i1B6QYMJG+U4DX56ZRW Xg4d6SnOQJbQzmOLIVog4+Q0/c3PKIiHtfyPN3AhK2azt8gN/eYHk4/Ow4bQnC+7rDkD1LKPwpp HXPE= X-Google-Smtp-Source: AGHT+IEW6kofzVvscdUp6IMKGrRkIje/J7JBh/Nc5qR/k31CnySuyyiwls0zSTGzIDDorCf57TnjJQ== X-Received: by 2002:a5d:4a81:0:b0:341:cd0d:b78a with SMTP id o1-20020a5d4a81000000b00341cd0db78amr8767835wrq.48.1713809678533; Mon, 22 Apr 2024 11:14:38 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 2/6] x86/alternative: Walk all replacements in debug builds Date: Mon, 22 Apr 2024 19:14:30 +0100 Message-Id: <20240422181434.3463252-3-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 In debug builds, walk all alternative replacements with x86_decode_lite(). This checks that we can decode all instructions, and also lets us check that disp8's don't leave the replacement block. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/alternative.c | 49 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/xen/arch/x86/alternative.c b/xen/arch/x86/alternative.c index 2e7ba6e0b833..5bd256365def 100644 --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #define MAX_PATCH_LEN (255-1) @@ -464,6 +465,54 @@ static void __init _alternative_instructions(bool force) void __init alternative_instructions(void) { arch_init_ideal_nops(); + + /* + * Walk all replacement instructions with x86_decode_lite(). This checks + * both that we can decode all instructions within the replacement, and + * that any near branch with a disp8 stays within the alternative itself. + */ + if ( IS_ENABLED(CONFIG_DEBUG) ) + { + struct alt_instr *a; + + for ( a = __alt_instructions; + a < __alt_instructions_end; ++a ) + { + void *repl = ALT_REPL_PTR(a); + void *ip = repl, *end = ip + a->repl_len; + + if ( !a->repl_len ) + continue; + + for ( x86_decode_lite_t res; ip < end; ip += res.len ) + { + res = x86_decode_lite(ip, end); + + if ( res.len <= 0 ) + { + printk("Alternative for %ps [%*ph]\n", + ALT_ORIG_PTR(a), a->repl_len, repl); + panic("Unable to decode instruction at +%u in alternative\n", + (unsigned int)(unsigned long)(ip - repl)); + } + + if ( res.rel_type == REL_TYPE_d8 ) + { + int8_t *d8 = res.rel; + void *target = ip + res.len + *d8; + + if ( target < repl || target > end ) + { + printk("Alternative for %ps [%*ph]\n", + ALT_ORIG_PTR(a), a->repl_len, repl); + panic("'JMP/Jcc disp8' at +%u leaves alternative block\n", + (unsigned int)(unsigned long)(ip - repl)); + } + } + } + } + } + _alternative_instructions(false); } From patchwork Mon Apr 22 18:14:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A961C41513 for ; Mon, 22 Apr 2024 18:14:55 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710192.1109306 (Exim 4.92) (envelope-from ) id 1ryyBe-0000cG-Kx; Mon, 22 Apr 2024 18:14:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710192.1109306; Mon, 22 Apr 2024 18:14:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBe-0000bt-E3; Mon, 22 Apr 2024 18:14:42 +0000 Received: by outflank-mailman (input) for mailman id 710192; Mon, 22 Apr 2024 18:14:41 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBd-0000IX-6N for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:41 +0000 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [2a00:1450:4864:20::636]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 2f64d62a-00d4-11ef-909a-e314d9c70b13; Mon, 22 Apr 2024 20:14:40 +0200 (CEST) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a5872b74c44so46930866b.3 for ; Mon, 22 Apr 2024 11:14:40 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:38 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2f64d62a-00d4-11ef-909a-e314d9c70b13 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809679; x=1714414479; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=31yUgd5YK+pN4dRkn1LMnihd6GkdbjUm1b+8i5iacEg=; b=R8PB8SYRh8z3v+Cn85Jjbb74Z10LovtcXj6ymwCs2nfNJZki8WLNxy/auZMP+8np1g 3UrKFthyTFQU4JYfs7JkbUYkNsTBzzy5D5aqEAA08QulrinS90Bqd6DckYwBhlL4jigh Lsn50J/FNrHi9cBv5RTpwIxvV3ENJ71hnw6ZU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809679; x=1714414479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=31yUgd5YK+pN4dRkn1LMnihd6GkdbjUm1b+8i5iacEg=; b=oM2Zm/nKzwWEWmg9mARcP99h3lv4HWNNCHi+Lpv2rkSFjPysZSzcb+ewU0WosV8jwn SuISFefjdc9tvZcGx/wdq26bNpM2Kf8JgMDpM+3Z0HvIAqSZ1Y/vTBrjKguUMZsTC2Ic 0iwV+POn0begZM5WI25cr7kuYyXCAxblY9syDpjiYuR8yzjxjdS1Ozpwbmr9ReeSEdH1 aXtukBXWyvvBV/yPaMF724KuQQD1YBsJ7T4XLHf5V/nO+k4qG7EHoAWhnrCOmCiZ8PQ1 1xMtvJ6b6QVQ+AbcOyz5zbLJeDRzDkbI9uoE6stycf4117DYukwDqXwP0Xpa2Z26VPle /IOQ== X-Gm-Message-State: AOJu0YzZk8fI+mWgpVG6674mu7AFPXHXoiIimJ6kfcmp1FtJlpUvbYVz VtA7kzFu3+SZAn6BnPyOxjWO4BBYhUZfXkvQU8N78rGZi5p3Kn3KmhlTJGYbRtHFz+QJW78JYcA 1Q2E= X-Google-Smtp-Source: AGHT+IG+1HMvF1SwFEh23Uw5BDoD1AN36ExhwVuxXRvPOWRV9QQwPpCghdo+Q9ydNqUU6Qfd296HkA== X-Received: by 2002:a17:906:7708:b0:a51:969e:ba0 with SMTP id q8-20020a170906770800b00a51969e0ba0mr6257278ejm.44.1713809679358; Mon, 22 Apr 2024 11:14:39 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 3/6] x86/alternative: Intend the relocation logic Date: Mon, 22 Apr 2024 19:14:31 +0100 Message-Id: <20240422181434.3463252-4-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 ... to make subsequent patches legible. No functional change. Signed-off-by: Andrew Cooper Acked-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/alternative.c | 126 +++++++++++++++++++------------------ 1 file changed, 64 insertions(+), 62 deletions(-) diff --git a/xen/arch/x86/alternative.c b/xen/arch/x86/alternative.c index 5bd256365def..2ca4dfd569bc 100644 --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -244,78 +244,80 @@ static void init_or_livepatch _apply_alternatives(struct alt_instr *start, memcpy(buf, repl, a->repl_len); - /* 0xe8/0xe9 are relative branches; fix the offset. */ - if ( a->repl_len >= 5 && (*buf & 0xfe) == 0xe8 ) { - /* - * Detect the special case of indirect-to-direct branch patching: - * - replacement is a direct CALL/JMP (opcodes 0xE8/0xE9; already - * checked above), - * - replacement's displacement is -5 (pointing back at the very - * insn, which makes no sense in a real replacement insn), - * - original is an indirect CALL/JMP (opcodes 0xFF/2 or 0xFF/4) - * using RIP-relative addressing. - * Some branch destinations may still be NULL when we come here - * the first time. Defer patching of those until the post-presmp- - * initcalls re-invocation (with force set to true). If at that - * point the branch destination is still NULL, insert "UD2; UD0" - * (for ease of recognition) instead of CALL/JMP. - */ - if ( a->cpuid == X86_FEATURE_ALWAYS && - *(int32_t *)(buf + 1) == -5 && - a->orig_len >= 6 && - orig[0] == 0xff && - orig[1] == (*buf & 1 ? 0x25 : 0x15) ) + /* 0xe8/0xe9 are relative branches; fix the offset. */ + if ( a->repl_len >= 5 && (*buf & 0xfe) == 0xe8 ) { - long disp = *(int32_t *)(orig + 2); - const uint8_t *dest = *(void **)(orig + 6 + disp); - - if ( dest ) + /* + * Detect the special case of indirect-to-direct branch patching: + * - replacement is a direct CALL/JMP (opcodes 0xE8/0xE9; already + * checked above), + * - replacement's displacement is -5 (pointing back at the very + * insn, which makes no sense in a real replacement insn), + * - original is an indirect CALL/JMP (opcodes 0xFF/2 or 0xFF/4) + * using RIP-relative addressing. + * Some branch destinations may still be NULL when we come here + * the first time. Defer patching of those until the post-presmp- + * initcalls re-invocation (with force set to true). If at that + * point the branch destination is still NULL, insert "UD2; UD0" + * (for ease of recognition) instead of CALL/JMP. + */ + if ( a->cpuid == X86_FEATURE_ALWAYS && + *(int32_t *)(buf + 1) == -5 && + a->orig_len >= 6 && + orig[0] == 0xff && + orig[1] == (*buf & 1 ? 0x25 : 0x15) ) { - /* - * When building for CET-IBT, all function pointer targets - * should have an endbr64 instruction. - * - * If this is not the case, leave a warning because - * something is probably wrong with the build. A CET-IBT - * enabled system might have exploded already. - * - * Otherwise, skip the endbr64 instruction. This is a - * marginal perf improvement which saves on instruction - * decode bandwidth. - */ - if ( IS_ENABLED(CONFIG_XEN_IBT) ) + long disp = *(int32_t *)(orig + 2); + const uint8_t *dest = *(void **)(orig + 6 + disp); + + if ( dest ) { - if ( is_endbr64(dest) ) - dest += ENDBR64_LEN; - else - printk(XENLOG_WARNING - "altcall %ps dest %ps has no endbr64\n", - orig, dest); + /* + * When building for CET-IBT, all function pointer targets + * should have an endbr64 instruction. + * + * If this is not the case, leave a warning because + * something is probably wrong with the build. A CET-IBT + * enabled system might have exploded already. + * + * Otherwise, skip the endbr64 instruction. This is a + * marginal perf improvement which saves on instruction + * decode bandwidth. + */ + if ( IS_ENABLED(CONFIG_XEN_IBT) ) + { + if ( is_endbr64(dest) ) + dest += ENDBR64_LEN; + else + printk(XENLOG_WARNING + "altcall %ps dest %ps has no endbr64\n", + orig, dest); + } + + disp = dest - (orig + 5); + ASSERT(disp == (int32_t)disp); + *(int32_t *)(buf + 1) = disp; } - - disp = dest - (orig + 5); - ASSERT(disp == (int32_t)disp); - *(int32_t *)(buf + 1) = disp; - } - else if ( force ) - { - buf[0] = 0x0f; - buf[1] = 0x0b; - buf[2] = 0x0f; - buf[3] = 0xff; - buf[4] = 0xff; + else if ( force ) + { + buf[0] = 0x0f; + buf[1] = 0x0b; + buf[2] = 0x0f; + buf[3] = 0xff; + buf[4] = 0xff; + } + else + continue; } + else if ( force && system_state < SYS_STATE_active ) + ASSERT_UNREACHABLE(); else - continue; + *(int32_t *)(buf + 1) += repl - orig; } - else if ( force && system_state < SYS_STATE_active ) + else if ( force && system_state < SYS_STATE_active ) ASSERT_UNREACHABLE(); - else - *(int32_t *)(buf + 1) += repl - orig; } - else if ( force && system_state < SYS_STATE_active ) - ASSERT_UNREACHABLE(); a->priv = 1; From patchwork Mon Apr 22 18:14:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A83AC07C79 for ; Mon, 22 Apr 2024 18:14:54 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710194.1109320 (Exim 4.92) (envelope-from ) id 1ryyBf-0000pk-9U; Mon, 22 Apr 2024 18:14:43 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710194.1109320; Mon, 22 Apr 2024 18:14:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBf-0000oY-1G; Mon, 22 Apr 2024 18:14:43 +0000 Received: by outflank-mailman (input) for mailman id 710194; Mon, 22 Apr 2024 18:14:42 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBe-0000IX-6f for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:42 +0000 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [2a00:1450:4864:20::634]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 2ffeed76-00d4-11ef-909a-e314d9c70b13; Mon, 22 Apr 2024 20:14:41 +0200 (CEST) Received: by mail-ej1-x634.google.com with SMTP id a640c23a62f3a-a55ab922260so203926466b.3 for ; Mon, 22 Apr 2024 11:14:41 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:39 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2ffeed76-00d4-11ef-909a-e314d9c70b13 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809680; x=1714414480; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FaF1+jq0lOG0Zt+SDR+w7NABaf8lTc4H+2KHEf8aMyg=; b=BJFT2JallZ/j//2MXYLr2gSSAMRBSs91NowYyRZw2r13Q85exIpfAnABt4KqaiUoV9 0Vg30zKnMkh7CSHkaXkjpQmHLqMv/eMn/dG6qDzr4n7wIL8+9NM4h8tNpEpACGpMMUVi 2IF4ZFViGMHFOKUmCUZNq3Cab7WnNgcCkC+bU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809680; x=1714414480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FaF1+jq0lOG0Zt+SDR+w7NABaf8lTc4H+2KHEf8aMyg=; b=JZuVlHWOy8PWnTFXnKhTy85wWu56b2o6nIHlo7rLskynZ+KP2v3B00CjHAttAzd7wo g2SFKt70VgrhytUxHDVEuATy1lCw0vy0cNBiOqfWwuaCUZRO5CXqBzVzavjp7EXWLHdL qcB5YpHWdt9JPbd9DryaGaWOGs3S0LzhrYBBAcJpaIRuvyV5gIA+jhPKISlxPrz6rWfE pfiJe9P9Vs8gH/NHOke4iblbd+VkQKHfYaqOj7V/bIiP3z2njEL0aXfcQkvHAWzeQx29 H6i7pvn0a8sjNZ3V2Kj7jmuP9ztHBSEzl7VsclLlVXPR6K1aBJpFRCEQy5WH11SeFSxT K2Ng== X-Gm-Message-State: AOJu0Yy80mRYXNI8DOw+y1h1t3mqlIhPVyK2Egz8VlOk/pcRpgke57Qu kY0z5d+/Y8RkiBRK1rRQSwvocAhdEN8BXh8VAqxELbsluSXH/jB/LbpLnT42H0vm83EgjLyz+sn ZRr4= X-Google-Smtp-Source: AGHT+IHmqFdmdVeYAFVG6c/DuwA7Wp3vBh4M41S2ewoBL7esKFiKG2mvjUAjL/oIHx92bkOiFta4Xg== X-Received: by 2002:a17:906:1d47:b0:a58:7507:d4ed with SMTP id o7-20020a1709061d4700b00a587507d4edmr440811ejh.15.1713809680447; Mon, 22 Apr 2024 11:14:40 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 4/6] x86/alternative: Replace a continue with a goto Date: Mon, 22 Apr 2024 19:14:32 +0100 Message-Id: <20240422181434.3463252-5-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 A subsequent patch is going to insert a loop, which interferes with the continue in the devirtualisation logic. Replace it with a goto, and a paragraph explaining why we intentionally avoid setting a->priv = 1. No functional change. Signed-off-by: Andrew Cooper Acked-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/alternative.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/alternative.c b/xen/arch/x86/alternative.c index 2ca4dfd569bc..c86ea235e865 100644 --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -308,7 +308,15 @@ static void init_or_livepatch _apply_alternatives(struct alt_instr *start, buf[4] = 0xff; } else - continue; + { + /* + * The function pointer we're wanting to devirtualise + * is still NULL, and we're not sealing yet. Leave + * the alternative fully un-processed, in order to + * process it the next time around. + */ + goto skip_this_alternative; + } } else if ( force && system_state < SYS_STATE_active ) ASSERT_UNREACHABLE(); @@ -323,6 +331,7 @@ static void init_or_livepatch _apply_alternatives(struct alt_instr *start, add_nops(buf + a->repl_len, total_len - a->repl_len); text_poke(orig, buf, total_len); + skip_this_alternative:; } /* From patchwork Mon Apr 22 18:14:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30D68C4345F for ; Mon, 22 Apr 2024 18:14:54 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710195.1109341 (Exim 4.92) (envelope-from ) id 1ryyBh-0001aA-HA; Mon, 22 Apr 2024 18:14:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710195.1109341; Mon, 22 Apr 2024 18:14:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBh-0001Zz-Dp; Mon, 22 Apr 2024 18:14:45 +0000 Received: by outflank-mailman (input) for mailman id 710195; Mon, 22 Apr 2024 18:14:43 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBf-0000Oz-If for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:43 +0000 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [2a00:1450:4864:20::631]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 30822915-00d4-11ef-b4bb-af5377834399; Mon, 22 Apr 2024 20:14:42 +0200 (CEST) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a5872b74c44so46933066b.3 for ; Mon, 22 Apr 2024 11:14:42 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:40 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 30822915-00d4-11ef-b4bb-af5377834399 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809681; x=1714414481; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qnv/4DDYEmGIANJ07M3XY3Ct96pjBr7ZvuszXz6nRKc=; b=QzpJtI54AXreZkGZCRZZWsDlIs9qxLUDDh/+69vjiW9tEa6Gan13OcnMTcnl2GBhLJ CtUdJNSCtiiW0pMXxkR7GhaMosMBBj4oilGVCrFvaZ/CX6M7pIDI8vEP3hAp6cO/+/FA ltgIqXVkf0Ybz8/1fTA2dUqFp01Q2YIFJG/Mk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809681; x=1714414481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qnv/4DDYEmGIANJ07M3XY3Ct96pjBr7ZvuszXz6nRKc=; b=UV3y7Ab06TOJMUZW5M3QB05GIvIt7ukm4F1PAk0/e9gwG67NAS9At8CLF3IWgXI14N 13AczqnawMxz4DE6a6A2KcryJA5/NpvClssuHw2shom85xsZha+wG/Bu76/tdYPPSn6Z YxtIi/dmr5Ggyyxs5QB4OvRFjp4iFtnB9CLLSDSzS77hjFZ+2H3jEUMLw7scFewgiY5R /MEe/ZxOPmfvoKH4zJWDVw4rRQZ7tp1uiKeQrwrt12gmiP25qrZ4CRo1FKgyovY1o/WC XWWWXLufWzLA9WAXbUXggq7fuoM52YAZ2d5mdALLxpKx/LZnKV0J5XDDoIsofjmUy7zM ZNWA== X-Gm-Message-State: AOJu0Yw6J/krc7Kv9OPWmHiLH+GkI1w45lajDjaF6NBWzl2CTcB9m0Vl 0pRTxk1nZ38hkoUqKUO2PVBt1QstdJHktHFBnXSkUf5zFMB60o5n+s6/zMeWkEHyryRNjQBdLy4 jXds= X-Google-Smtp-Source: AGHT+IENu/P+oqdgoIPmfD0QCWOFEWg+4TnWMcAxsx2C9L/ECrh0opFdNPWnQ9cnJbJNMSXQoIR0AQ== X-Received: by 2002:a17:906:a0b:b0:a52:33b0:fcb1 with SMTP id w11-20020a1709060a0b00b00a5233b0fcb1mr6665155ejf.32.1713809681291; Mon, 22 Apr 2024 11:14:41 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 5/6] x86/alternative: Relocate all insn-relative fields Date: Mon, 22 Apr 2024 19:14:33 +0100 Message-Id: <20240422181434.3463252-6-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Right now, relocation of displacements is restricted to finding 0xe8/e9 as the first byte of the replacement, but this is overly restrictive. Use x86_decode_lite() to find and adjust all insn-relative fields. As with disp8's not leaving the replacemnet block, some disp32's don't either. e.g. the RSB stuffing loop. These stay unmodified. For now, leave the altcall devirtualisation alone. These require more care to transform into the new scheme. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/alternative.c | 46 +++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/xen/arch/x86/alternative.c b/xen/arch/x86/alternative.c index c86ea235e865..4d7dc9418cf0 100644 --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -244,10 +244,31 @@ static void init_or_livepatch _apply_alternatives(struct alt_instr *start, memcpy(buf, repl, a->repl_len); + /* Walk buf[] and adjust any insn-relative operands. */ + if ( a->repl_len ) { - /* 0xe8/0xe9 are relative branches; fix the offset. */ - if ( a->repl_len >= 5 && (*buf & 0xfe) == 0xe8 ) + uint8_t *ip = buf, *end = ip + a->repl_len; + + for ( x86_decode_lite_t res; ip < end; ip += res.len ) { + int32_t *d32; + uint8_t *target; + + res = x86_decode_lite(ip, end); + + if ( res.len <= 0 ) + { + printk("Alternative for %ps [%*ph]\n", + ALT_ORIG_PTR(a), a->repl_len, repl); + printk("Unable to decode instruction in alternative - ignoring.\n"); + goto skip_this_alternative; + } + + if ( res.rel_type != REL_TYPE_d32 ) + continue; + + d32 = res.rel; + /* * Detect the special case of indirect-to-direct branch patching: * - replacement is a direct CALL/JMP (opcodes 0xE8/0xE9; already @@ -317,14 +338,23 @@ static void init_or_livepatch _apply_alternatives(struct alt_instr *start, */ goto skip_this_alternative; } + + continue; } - else if ( force && system_state < SYS_STATE_active ) - ASSERT_UNREACHABLE(); - else - *(int32_t *)(buf + 1) += repl - orig; + + target = ip + res.len + *d32; + + if ( target >= buf && target <= end ) + { + /* + * Target doesn't leave the replacement block. e.g. RSB + * stuffing. Leave it unmodified. + */ + continue; + } + + *d32 += repl - orig; } - else if ( force && system_state < SYS_STATE_active ) - ASSERT_UNREACHABLE(); } a->priv = 1; From patchwork Mon Apr 22 18:14:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 13638801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5226DC18E72 for ; Mon, 22 Apr 2024 18:14:55 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.710196.1109346 (Exim 4.92) (envelope-from ) id 1ryyBh-0001dp-TO; Mon, 22 Apr 2024 18:14:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 710196.1109346; Mon, 22 Apr 2024 18:14:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBh-0001cX-Ml; Mon, 22 Apr 2024 18:14:45 +0000 Received: by outflank-mailman (input) for mailman id 710196; Mon, 22 Apr 2024 18:14:43 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ryyBf-0000IX-OD for xen-devel@lists.xenproject.org; Mon, 22 Apr 2024 18:14:43 +0000 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [2a00:1450:4864:20::629]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 313ada63-00d4-11ef-909a-e314d9c70b13; Mon, 22 Apr 2024 20:14:43 +0200 (CEST) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a559b919303so249730966b.1 for ; Mon, 22 Apr 2024 11:14:43 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id z15-20020a170906668f00b00a51d073da7esm5994224ejo.82.2024.04.22.11.14.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Apr 2024 11:14:41 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 313ada63-00d4-11ef-909a-e314d9c70b13 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713809682; x=1714414482; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dB+pEYt/xsRXM0pv02BxjDs01ExeTPOKtKLGLzOzv0s=; b=m/YD7PW9Cinj9EGqD1JjCAgTChK87yX6Uvl6xvYJxsOl35BZPNaiEJAng9P1i57KOa gaZh+J2Ef3hUfcbgW6d01L0UpVYiLgX6baUO0KnTnZV0wgYOPAjIB24VwRZgo9BcaeWq yxIeEpAGtfYCnaOsSKNN+8W0Dz0CMi9L08mkQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713809682; x=1714414482; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dB+pEYt/xsRXM0pv02BxjDs01ExeTPOKtKLGLzOzv0s=; b=O+hyWgCzHxiPtOv9ZnkfVqlvje80IYYtmlIrBUYejy1k/r9pT863FXu2zyg2yCUZMa xBPB0qkNFwNilMIPRqe9+Mpb2TdhyaasY8t4X0JtIjt5SpEVZcRgjDuXXJizAZWYlgcg rKTjPPp56IGaMnavArrNOBaBY2HcHeKGglXG+bwFvSYkKbLxi1vOx0wXiF33aRmjdV1E 0wQSsdXVJpS62DC99mSByl7YWUQvIauVkfWwqEZddeTmdwD2X/nEy864f5OL8BTMdPWc SycWpd8lSfwbSfaGZ1+tgicvQ3OGsWBX2lIXtQHqeCshXQfmfrf0Urx/HhvjJuHaL+77 VPwQ== X-Gm-Message-State: AOJu0YwF6/8QhXhe7xBsz9KWpSctht05QZKdoNoQ4z5ebrq8EoztWUyb YSJKbDLXXMWiEl795QtbxRwFMeYe2/z19+bDdyBCnWd8c7TurCsBn0Zkh/fHc9/9HHJoEcM+tzO sfQU= X-Google-Smtp-Source: AGHT+IGqd/HqMIOYUnziS1lqHwIAw5CNOJQ9KnM2G4JVD+h+CP14AE9vAz3vT5t6yXjZ6d/aUx+h6Q== X-Received: by 2002:a17:906:b092:b0:a55:387b:ef07 with SMTP id x18-20020a170906b09200b00a55387bef07mr7177532ejy.13.1713809682179; Mon, 22 Apr 2024 11:14:42 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Subject: [PATCH 6/6] x86/spec-ctrl: Introduce and use DO_COND_BHB_SEQ Date: Mon, 22 Apr 2024 19:14:34 +0100 Message-Id: <20240422181434.3463252-7-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240422181434.3463252-1-andrew.cooper3@citrix.com> References: <20240422181434.3463252-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Now that alternatives can fix up call displacements even when they're not the first instruction of the replacement, move the SCF_entry_bhb conditional inside the replacement block. This removes a conditional branch from the fastpaths of BHI-unaffected hardware. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné --- xen/arch/x86/hvm/vmx/entry.S | 12 +++---- xen/arch/x86/include/asm/spec_ctrl_asm.h | 43 +++++++++++++----------- 2 files changed, 30 insertions(+), 25 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/entry.S b/xen/arch/x86/hvm/vmx/entry.S index 7233e771d88a..77bf9ea564ea 100644 --- a/xen/arch/x86/hvm/vmx/entry.S +++ b/xen/arch/x86/hvm/vmx/entry.S @@ -62,12 +62,12 @@ ENTRY(vmx_asm_vmexit_handler) * Clear the BHB to mitigate BHI. Used on eIBRS parts, and uses RETs * itself so must be after we've perfomed all the RET-safety we can. */ - testb $SCF_entry_bhb, CPUINFO_scf(%rsp) - jz .L_skip_bhb - ALTERNATIVE_2 "", \ - "call clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ - "call clear_bhb_tsx", X86_SPEC_BHB_TSX -.L_skip_bhb: + .macro VMX_BHB_SEQ fn:req + DO_COND_BHB_SEQ \fn scf=CPUINFO_scf(%rsp) + .endm + ALTERNATIVE_2 "", \ + "VMX_BHB_SEQ fn=clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ + "VMX_BHB_SEQ fn=clear_bhb_tsx", X86_SPEC_BHB_TSX ALTERNATIVE "lfence", "", X86_SPEC_NO_LFENCE_ENTRY_VMX /* WARNING! `ret`, `call *`, `jmp *` not safe before this point. */ diff --git a/xen/arch/x86/include/asm/spec_ctrl_asm.h b/xen/arch/x86/include/asm/spec_ctrl_asm.h index 729a830411eb..559dad88f967 100644 --- a/xen/arch/x86/include/asm/spec_ctrl_asm.h +++ b/xen/arch/x86/include/asm/spec_ctrl_asm.h @@ -92,6 +92,21 @@ .L\@_skip: .endm +.macro DO_COND_BHB_SEQ fn:req, scf=%bl +/* + * Requires SCF (defaults to %rbx), fn=clear_bhb_{loops,tsx} + * Clobbers %rax, %rcx + * + * Conditionally use a BHB clearing software sequence. + */ + testb $SCF_entry_bhb, \scf + jz .L\@_skip_bhb + + call \fn + +.L\@_skip_bhb: +.endm + .macro DO_OVERWRITE_RSB tmp=rax, xu /* * Requires nothing @@ -277,12 +292,9 @@ * Clear the BHB to mitigate BHI. Used on eIBRS parts, and uses RETs * itself so must be after we've perfomed all the RET-safety we can. */ - testb $SCF_entry_bhb, %bl - jz .L\@_skip_bhb - ALTERNATIVE_2 "", \ - "call clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ - "call clear_bhb_tsx", X86_SPEC_BHB_TSX -.L\@_skip_bhb: + ALTERNATIVE_2 "", \ + "DO_COND_BHB_SEQ clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ + "DO_COND_BHB_SEQ clear_bhb_tsx", X86_SPEC_BHB_TSX ALTERNATIVE "lfence", "", X86_SPEC_NO_LFENCE_ENTRY_PV .endm @@ -322,12 +334,9 @@ ALTERNATIVE "", __stringify(DO_SPEC_CTRL_ENTRY maybexen=1), \ X86_FEATURE_SC_MSR_PV - testb $SCF_entry_bhb, %bl - jz .L\@_skip_bhb - ALTERNATIVE_2 "", \ - "call clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ - "call clear_bhb_tsx", X86_SPEC_BHB_TSX -.L\@_skip_bhb: + ALTERNATIVE_2 "", \ + "DO_COND_BHB_SEQ clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ + "DO_COND_BHB_SEQ clear_bhb_tsx", X86_SPEC_BHB_TSX ALTERNATIVE "lfence", "", X86_SPEC_NO_LFENCE_ENTRY_INTR .endm @@ -433,13 +442,9 @@ * Clear the BHB to mitigate BHI. Used on eIBRS parts, and uses RETs * itself so must be after we've perfomed all the RET-safety we can. */ - testb $SCF_entry_bhb, %bl - jz .L\@_skip_bhb - - ALTERNATIVE_2 "", \ - "call clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ - "call clear_bhb_tsx", X86_SPEC_BHB_TSX -.L\@_skip_bhb: + ALTERNATIVE_2 "", \ + "DO_COND_BHB_SEQ clear_bhb_loops", X86_SPEC_BHB_LOOPS, \ + "DO_COND_BHB_SEQ clear_bhb_tsx", X86_SPEC_BHB_TSX lfence .endm