From patchwork Wed Jun 19 09:22:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13703569 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60FD679DC7 for ; Wed, 19 Jun 2024 09:22:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718788944; cv=none; b=Fu0DTzYJK9l/CNLP0HoHNm9BH0uV1vEqih7SLtNsYQL/8/HjZBijoU+hXseh2oxZ8E1My96f8ABPGg27PTZqQ3OhdbhjQp6NLhm8ghxQbDsQEjuIhs32Kt+UriO9JHufXHMI974LIQiCoVF7fPjWcmmGLhtNolDXMy4fnGEpqiM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718788944; c=relaxed/simple; bh=mY5SBIVa14u2VjPWIp7hVxyy5bTzHfSg1McACT/etHc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VvcBEdxBuIwUInZui+W1rpqwO3x+pHw4OtdlQlJjzxUqShm29LaDQIY1SyeeTJ+MXEGgGp7y+V+Ct9zritRhYWyFZTbEbPSdQBSRkvEAQBtHMUitLCmzupHJV6RAHqtXTUJL33Xkp7n7gphV3L9iBDjG6Uap5X/u8ONKg+HY0s8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BV5DdU4Q; arc=none smtp.client-ip=209.85.208.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BV5DdU4Q" Received: by mail-lj1-f196.google.com with SMTP id 38308e7fff4ca-2ebe785b234so63314541fa.1 for ; Wed, 19 Jun 2024 02:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718788940; x=1719393740; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M619BJr1wfMGHjyjRuE2IRDAIUZLIDAA97Lw+i23gJk=; b=BV5DdU4QAD9/WH9wSW2SW4J7XUfzLMg6ZSWyew9XkgWreDpbfa0dNu2jvkFxv8fUVO RRxNc65+10wBxPEJZcFXnbg+h0mJE0Ovx4CtTnRoqY9gOjqaGWsQfYNUuF4gZkCQ0Nog K/bJJm8EC4eAJHqDSTdba6uKShOV5cOUp9pQAECf1LJMgXuSZi1y89FQ4oRiX6ZxUkJn cQ0eK1zmkWnI806+8tsQpg5mBTgidwVuts5rGW5w8YRgATsx6sqa+4srN/pf0+5Y34Eh NZgTho7zfTwobDHOAeUwlPkSe1KmSoCFxOrHbnohoBxfZ5ncE023ChKFe0zkkxL+XO7b ZN2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718788940; x=1719393740; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M619BJr1wfMGHjyjRuE2IRDAIUZLIDAA97Lw+i23gJk=; b=NXlP2QLpZyQBGUPk1xi4VGQe2NLStJUFfSlmCJUDVLVqdR8jMeb4qBlruYW0r5oLqv Amu3WYmu/1+kvpQUknWEj3aO0tbIsGXnP9fwV/GsYsBN4YuZWXxTQ2Fgg9iyqaoHzCrV q3kCszi6m/d9NTzoR3vj7gfXP5Ao7bgxCPKOgDzeD0+Hi/LNCq9cjXiYHQEW/9aThf9/ ACUloVZXPlGqurCQ3N1L+DUOf+yVbY7xAGCY3Z/M0r3Y7SnmjKGVw38nFggM8x7ImqXn gsXzxFLiFIidY+ML3zlkBIdKiIl7QqXNzbe2xIoc5BEIuSpE5P6JrZBqdVdMvL8y56LC HFgg== X-Gm-Message-State: AOJu0Yx5C+tiJZ2DjcqJJnlNPvBCMBeLrI2fhiLlsVsMcCHtUdeEIImk ghMziFgh+Zonp/plvARQMF8KwX6gBmaWPlKA5nqRF4C8qPeC7CIlFYUczdpE X-Google-Smtp-Source: AGHT+IHIhDmykuO7OTR09KFRKgWaiwEc7Nv2jcXLOO48baXdhcxciBC4Bzv+SQt1W9QeldA13fg2lg== X-Received: by 2002:a2e:2416:0:b0:2ec:1042:fb02 with SMTP id 38308e7fff4ca-2ec3cea5765mr16171261fa.8.1718788939631; Wed, 19 Jun 2024 02:22:19 -0700 (PDT) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-422870e9676sm255312915e9.24.2024.06.19.02.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jun 2024 02:22:18 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Puranjay Mohan , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Rishabh Iyer , Sanidhya Kashyap Subject: [PATCH bpf-next v2 1/2] x86: Perform BPF exception fixup in do_user_addr_fault Date: Wed, 19 Jun 2024 09:22:15 +0000 Message-ID: <20240619092216.1780946-2-memxor@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240619092216.1780946-1-memxor@gmail.com> References: <20240619092216.1780946-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2965; i=memxor@gmail.com; h=from:subject; bh=mY5SBIVa14u2VjPWIp7hVxyy5bTzHfSg1McACT/etHc=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBmcqMXCedlnqJwMQhJ6Hv4aBvbkxoXFOpWeO6nQ s2OQVPhaBWJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZnKjFwAKCRBM4MiGSL8R ymULD/wI69aqIShM0H4jjq3JBkDpMT9RoUADPyMFTS91TvH8rgMizwtWUZeEac4pdUHzzqMNgFK 2MrnPcwRSNR5prerJjeRSbt6JJjd7GJ1PqLCFol8extbvfdufft4suV7ZbOPyMtOLSdWuQQCGMp bslQi9Smwlf9zh2qiQaUCpLxLBCWCkkoIKOwYNeNFvDbkX6h9qNFTztqU5oM/0WsOKvKi8DnQuf 7GixjGHPiJDlSp+AxagNG/oNEN/h4nCktZkbSZhpU/+uH+hA1zYQr02gfecr+Hf8LaDbI9fC2Ia gjNYuAskmP/+faMIMiM3AQpqrPw3h8tqx0flLn3A8SFh0l1uOh8koPydZ/l1jKAra3f+y2UQB9j l+NmX23tdRdsc2L8i/3Jfim+zSPgeWi4crqAdDiVKM9CeuIVbsM9SKvuCygLKzROxPzl2oUrIDD OWehKcinuBqNRoZzhHMxSmTL7z+uJN4AM4Hh4MqydTqYxorBxXVoHNYpDKB3/aXb3kmO74c3ooI /ahPT3GfSr4/HBkgh7MwPaHotEDTk/76GP7F33QPN2R8MMVYc23xQ3NqxDgBOuMDY/c+/Mvcvc0 QGVScEpvbwqgBS969k2N1Y2auJSmJ4wf1NmtE2cDwwgZxA4Mb2Ob53uQdcsm0r4Z+j9QiG3Dq7W qkprKuuhqBOLAcA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Currently, on x86, when SMAP is enabled, and a page fault occurs in kernel mode for accessing a user address, the kernel will rightly panic as no valid kernel code can cause such a page fault (unless buggy). There is no valid correct kernel code that can generate such a fault, therefore this behavior would be correct. BPF programs that currently encounter user addresses when doing PROBE_MEM loads (load instructions which are allowed to read any kernel address, only available for root users) avoid a page fault by performing bounds checking on the address. This requires the JIT to emit a jump over each PROBE_MEM load instruction to avoid hitting page faults. We would prefer avoiding these jump instructions to improve performance of programs which use PROBE_MEM loads pervasively. For correct behavior, programs already rely on the kernel addresses being valid when they are executing, but BPF's safety properties must still ensure kernel safety in presence of invalid addresses. Therefore, for correct programs, the bounds checking is an added cost meant to ensure kernel safety. If the do_user_addr_fault handler could perform fixups for the BPF program in such a case, the bounds checking could be eliminated, the load instruction could be emitted directly without any checking. Thus, in case SMAP is enabled (which would mean the kernel traps on accessing a user address), and the instruction pointer belongs to a BPF program, perform fixup for the access by searching exception tables. All BPF programs already execute with SMAP protection. When SMAP is not enabled, the BPF JIT will continue to emit bounds checking instructions. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/mm/fault.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index e6c469b323cc..189e93d88bd4 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -21,6 +21,7 @@ #include #include /* find_and_lock_vma() */ #include +#include /* is_bpf_text_address() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1257,6 +1258,16 @@ void do_user_addr_fault(struct pt_regs *regs, if (unlikely(cpu_feature_enabled(X86_FEATURE_SMAP) && !(error_code & X86_PF_USER) && !(regs->flags & X86_EFLAGS_AC))) { + /* + * If the kernel access happened to an invalid user pointer + * under SMAP by a BPF program, we will have an extable entry + * here, and need to perform the fixup. + */ + if (is_bpf_text_address(regs->ip)) { + kernelmode_fixup_or_oops(regs, error_code, address, + 0, 0, ARCH_DEFAULT_PKEY); + return; + } /* * No extable entry here. This was a kernel access to an * invalid pointer. get_kernel_nofault() will not get here. From patchwork Wed Jun 19 09:22:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13703570 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B85867F7C6 for ; Wed, 19 Jun 2024 09:22:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.67 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718788945; cv=none; b=j7c7FzEwVKvs5etY62mg5H8B4sLkT5AjxGBcwF5NtGceizoI8Lx7nhkaonDuEa/+ldOsUycAcyCAldAqOAoE17tVpFuswsDEonDQwCQix+e8LfbOIni3Ebh2Cy0G4Kt9X3e2OQiJ8DEFqae8rccFBB3qyAJkgX8oAgJutiM8wws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718788945; c=relaxed/simple; bh=c5oTgP9QkkqMhl0mhuFCPOTyp5qmTdwIBTcPx9gqqZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M2zheKs+rdeQF9sXMyWFad+5WDTmOZ+RIPv5xNNFCnUUhpfe7tYKwCj0LdR15Zz+JWM+cQGSrS4OjUMOJztfGPs4imDQQQC4vnBiP27LO1mGjpt2VQJ/paPRx6lRg5RqKmP1MXlJeeynhtEpDVaN+uyCtnln0GetFHy+IuL/nAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WiwbKVSc; arc=none smtp.client-ip=209.85.221.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WiwbKVSc" Received: by mail-wr1-f67.google.com with SMTP id ffacd0b85a97d-35f090093d8so4964543f8f.0 for ; Wed, 19 Jun 2024 02:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718788941; x=1719393741; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CuFNtRiPUxiY2ssVsIPz8qk6a5jINgTHJnFo8uiNf6E=; b=WiwbKVScTYDjXfoJa0nRuUtcRiy0RcLl95oJhVipq1J7kLUlZJdPry13GvNnXR/8Ij NSZZ+iwoB5ao5VN4aIUpNTjMziwsO6MjTWidqmUBcwdb8Rqm74scrjqORiG0f/kbTxSO s168OJXU7RyZ0bxxGHsQgB0PCb/bazk4Ibw43nQvyngddF+vF5e5QLzELo4tpBH2lqPy aENBKEHj7mfXQZL6q2miAY6gQySY84enJb8HFtMIo9OQw2TDhO3QRT1eT9X28swZVBs3 0dwRUWrdBLdEbOVANEf7Sca8/RY19LjpmwMvSb/nq9eLQL4KsPP1Tqla9sCo/kK9nDzJ bOwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718788941; x=1719393741; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CuFNtRiPUxiY2ssVsIPz8qk6a5jINgTHJnFo8uiNf6E=; b=oX6BIEHnG4q0CrdJX3cVfH1hc6kapV68NGM0VyoPynUt0DvA8PD6AG5XJB+YbcvH7h 7XvZNca+2aRJyLA1bhymJ7epu2UVrU+o/CggoNsYt+8RRqpeuYZ19wj7U7HlurCV/rF0 d5YoAlDajg4tVye89u6j5vF31TBAcm9wadUZWDqx3o3tVG7SkksGtQsNfEmi1NRkIdpo vZpuOottXkLp37zUblMWKUEjH3e5V0Si0KfH8KSyEIPOc99ILDseKeJEscwXFz14CopB UfGIs2O42y6JEZT1M0orCE6nnlK5YOxqkYPRv3XDnIBMUeiUST8wElbHv5dmmuzsi5ns 2pDA== X-Gm-Message-State: AOJu0YycQHq4CQZ1URf1orRryf22E+BMwENbuNxKH3lw2FmOZJemRIXU q0RaEiHPha0tYeUArOYLyT7lqaDDJO2vX3Ew2KnTYUpXhHWBj8es0kUq8/wK X-Google-Smtp-Source: AGHT+IEi+t6LCzQjl+9fEB5xZGJaMXkkQhn5DC8PrgPixXxp61yBKXreDrlHZ4R5kfn92Q8VNc6qfw== X-Received: by 2002:a5d:49c7:0:b0:35f:488:6d3d with SMTP id ffacd0b85a97d-3631998ee26mr1491524f8f.58.1718788941234; Wed, 19 Jun 2024 02:22:21 -0700 (PDT) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-360750935ecsm16669902f8f.3.2024.06.19.02.22.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jun 2024 02:22:20 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Puranjay Mohan , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Rishabh Iyer , Sanidhya Kashyap Subject: [PATCH bpf-next v2 2/2] bpf, x86: Skip bounds checking for PROBE_MEM with SMAP Date: Wed, 19 Jun 2024 09:22:16 +0000 Message-ID: <20240619092216.1780946-3-memxor@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240619092216.1780946-1-memxor@gmail.com> References: <20240619092216.1780946-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2699; i=memxor@gmail.com; h=from:subject; bh=c5oTgP9QkkqMhl0mhuFCPOTyp5qmTdwIBTcPx9gqqZM=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBmcqMXHVXLEcjnMz1ZjXdYJR6KEUpM7hl4PD0Iv CImrToA1vSJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZnKjFwAKCRBM4MiGSL8R ym2eEADC44zzO+zr1+TWeQwuGhXhsBTsAHFm7riuiIMzqmQb+uEXzOKdYoH1jRz4ktnspPO7Jt7 /JFbwihLTtMdTAjaIbbje7fbXjfZzfvnT2Kk0jdFHWSPJtvtgx4o+N0UWIH1YDz5yi4xwx3wUsr AvyEU7+KYASDCkq3wN7FYKnyTJz4aL5TRkcAVyAQrQPU+yO9fHrF7G0as30iJEEO+EboJwuqP3Z 9I2cTmZifbJ1b3YXsG5U2rimxub9X8HnF4pBPxcdqvm7Nab6ecfgUI12sOWe3GSzj7OCk7xUDbh zIsOF0UhfbQKzUYgYaCrqv00LnAeZGJVU7P7CaOPjdQnN555NVFqtt+jfZNcy5RJ4TU99Vq8jO/ K+N/nsZRTG9JZFPEsViCyPzpjMCrQSuhSyzw/LfrsqC0NwSulxDMljnuhDWT2Mpkknmp8GaE5WW 3c+rDsecaxzJX/S1wRQQ5CeyktaS8RuWf1fJGE/kCfg9NdG+H0luBFibbBYNmYepBTVU6XBabTP lzPNoe6Fwfb9PYs3v7Zq6yRg6GZc3eHRhgFqmLmybPWzJ/FYDo9drW+AypvIA1kNLypQ7zoV/U4 mMLj6Aky9Ns+h7rs8AJunzQeJ2KMxGz/2Jtv6EDEemBr+Nq2QWHEH0m/u+8PF/WIj5JY05Kmq7O jDdnfv67kL6NwKg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net The previous patch changed the do_user_addr_fault page fault handler to invoke BPF's fixup routines (by searching exception tables and calling ex_handler_bpf). This would only occur when SMAP is enabled, such that any user address access from BPF programs running in kernel mode would reach this path and invoke the fixup routines. Relying on this behavior, disable any bounds checking instrumentation in the BPF JIT for x86 when X86_FEATURE_SMAP is available. All BPF programs execute with SMAP enabled, therefore when this feature is available, we can assume that SMAP will be enabled during program execution at runtime. This optimizes PROBE_MEM loads down to a normal unchecked load instruction. Any page faults for user or kernel addresses will be handled using the fixup routines, and the generation exception table entries for such load instructions. All in all, this ensures that PROBE_MEM loads will now incur no runtime overhead, and become practically free. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 5159c7a22922..f8a39189cddc 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1864,8 +1864,8 @@ st: if (is_imm8(insn->off)) case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: insn_off = insn->off; - if (BPF_MODE(insn->code) == BPF_PROBE_MEM || - BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { + if ((BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEMSX) && !cpu_feature_enabled(X86_FEATURE_SMAP)) { /* Conservatively check that src_reg + insn->off is a kernel address: * src_reg + insn->off > TASK_SIZE_MAX + PAGE_SIZE * and @@ -1912,6 +1912,9 @@ st: if (is_imm8(insn->off)) /* populate jmp_offset for JAE above to jump to start_of_ldx */ start_of_ldx = prog; end_of_jmp[-1] = start_of_ldx - end_of_jmp; + } else if ((BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEMSX)) { + start_of_ldx = prog; } if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX || BPF_MODE(insn->code) == BPF_MEMSX) @@ -1924,9 +1927,13 @@ st: if (is_imm8(insn->off)) u8 *_insn = image + proglen + (start_of_ldx - temp); s64 delta; + if (cpu_feature_enabled(X86_FEATURE_SMAP)) + goto extable_fixup; + /* populate jmp_offset for JMP above */ start_of_ldx[-1] = prog - start_of_ldx; + extable_fixup: if (!bpf_prog->aux->extable) break;