From patchwork Sun Nov 3 19:35:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13860560 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1179175D45 for ; Sun, 3 Nov 2024 19:35:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.66 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730662523; cv=none; b=jOochxY5vWerFXBnUley9EdxA7PZHmyr+Bl/DouJOg2qfB6Kle8sk4v7Y2X7qg4KBxZo1ZZOBECNxXE1jAslzJUL1+xav9pHtQLKN7hIYnWcmUGk1IAhPWvS2mNdYJU77UuPywFhv6pMOtVl76TUmbaLovyx9N/7DpPZCzFPBKg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730662523; c=relaxed/simple; bh=1HXCG3nTLc7zR8zdspgk2xsHWUxX+SKe9U3lnmN6z60=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i3osWtUa4h4hHEQ8cG8xmianVxpYgGsyKBGc9lJURH6CZLwDQGxqrDoOG0eiNBiCHTSg1kMlffXmSRRha1bTMSWgq1eWfs7A7+12GHJgryj6ksKJl7Ym+Z18wD4o9EYz+NfcKEx0daoghlpMCRgE81cqBmgYdZ+OLSXuZTmHy28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U9YVOaym; arc=none smtp.client-ip=209.85.128.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U9YVOaym" Received: by mail-wm1-f66.google.com with SMTP id 5b1f17b1804b1-4314f38d274so43900005e9.1 for ; Sun, 03 Nov 2024 11:35:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730662520; x=1731267320; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tKuf0piAxx3HVnImVjpuo//lYXPD5MuJPh3iJyvRmQ4=; b=U9YVOaym8iBhXaMMJQUz4Ak5LYt7RiC+9XRmP6InD/boNV+C7XXCdiqVsuCdPAC/OT lPCrsHKHIspPNKNmq3qvVOBhx8QvdrJT3D5snxOuKAiRzWWVpQL3i7o9/0NkkRuhAiFG KUxsJ+I/sf6TjoyImfBYmKQh4DchhjGMxn/4706Kk6FyxH+l/4rmFHVNS2aXl1Pb3tNQ 28NSY+A9PrWU9aGie7wSS4XTO6KcF3Wun8b643Wo2paqi1xn0yxj0MQneOC7WNZOleA0 7fdjUJ3pfbQOWcJhDy5Gu3pCj1oQsifnVL0GA03tZcsDwXYSu0W/UlUxU+RLMhkgh2pW 8ZgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730662520; x=1731267320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tKuf0piAxx3HVnImVjpuo//lYXPD5MuJPh3iJyvRmQ4=; b=JUbwTfR2ujUl9HPYJV2SeJ2CRfisHsqlaWtZf7h6d6EqXLsoQoaG/kiIgh6g6DGmBH A2hpYlNFCbGJpCQKoXygTCX9E4SW2Dathw+RoJojGud7ts4tnQtkPwIk8hiZmw61WI61 8XEC0tYJ4PUUmoPOMWUDDrhrYwNBK/NHNywjhpTDhQ5lPbsZcQ+6VpavE/LZlG/AQoTp M4w+krkJaCMea9TQUjtQuDvNb++6Vg+RQubJwBblApH4V16+gPkRE1PSEwL+Y5eEJm4b fFEaO8JhmFdm7z47DjMfCkEhqQQhfluuv8O94c/elCipFpbb/jPvWC6Y7wyZN0AY91YD GxTg== X-Gm-Message-State: AOJu0YxtKPCo2mNYw3/Mgu687om743d7WjJCT8rd8Ef/0QXw14GeBRwD SO8ekV5VzjDxInBGqKikWK+jSpCUkdYxmCLFvRtZjpOkunXmn+wuUl3ZPKIy/BM3qA== X-Google-Smtp-Source: AGHT+IGkSKwVVKFCfxgnTcqoxre81NbZndQpdbUfb7NuEFa2a9cdJOsEanClHaH/oxeoudx7CDK61w== X-Received: by 2002:a05:600c:4683:b0:431:5ba1:a520 with SMTP id 5b1f17b1804b1-432942dd802mr38288695e9.3.1730662518308; Sun, 03 Nov 2024 11:35:18 -0800 (PST) Received: from localhost (fwdproxy-cln-016.fbsv.net. [2a03:2880:31ff:10::face:b00c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431bd8e9145sm160603235e9.1.2024.11.03.11.35.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 11:35:17 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: kkd@meta.com, Puranjay Mohan , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Alexander Shishkin , "Kirill A. Shutemov" , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Rishabh Iyer , Sanidhya Kashyap , x86@kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 1/2] x86: Perform BPF exception fixup in do_user_addr_fault Date: Sun, 3 Nov 2024 11:35:11 -0800 Message-ID: <20241103193512.4076710-2-memxor@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241103193512.4076710-1-memxor@gmail.com> References: <20241103193512.4076710-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3013; h=from:subject; bh=1HXCG3nTLc7zR8zdspgk2xsHWUxX+SKe9U3lnmN6z60=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBnJ8wNJpmWYQgs2zmTqWQ+16u+4dc30s6k6QyeMWqQ ejFeS6OJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZyfMDQAKCRBM4MiGSL8RypCFD/ 0er51/TVp6u82UCrtbFrIWSdhFBfnv+Q14bfLFbjFd1/ruILK9Hfo4/z+8M5m/ngoh0BeWr9zLaGuA YBzYCDp7PSLj+FBA7ANAeQOgr3tG2q7PJVP/avdAxfMg2W4fAdjB3ISJBNh1B9IN0l9/dm3GFQUq0C hOUuiNhGaWxcxTrqM2sho27OJnAeCC6JgALXOFja49FqGA3rqrN3so8hnTA8digiFkFrKjbJPBkvE5 wkxiEkUbKGdn7k3cijYLdkOj/Fspdbh/lTk8fG6JSmVKB+HJw8cFlW2KrKkVkys1zDczeL4KOED0pI jebGQKkqTFVy5cRPqw5ph6eIHD0z2wDexFNxHIcvH9xHuC4T+NmJ/CBTQFxl64uW4OjUjAU7+axZw0 A2q3l9gkAGyk2UtbFISCAdqOZJr8npTmK5p0cvXvhPXSJ2Itaa+cDk+IcbncxQWxDPTNaDPL1v9wls +GVTS9e0ZcJQKjphDt2W5yBmLtDj+i/tnME8N0edPn2TwPL6rNBPdTetrZUuPR4/jd3gmbRwkX+axC rPOb/ebEyfzW16SCfXtHfjdLusiCB4jmXJjFPK0aAn/OARQweAyBzTP10doCDtAYpDdy85WUFbUViA yDyJtniwAmLeNFYPX2A2qnD6G8HrESOEOYLPweq1pmRNKn4jQ4Amjlqp//UQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Currently, on x86, when SMAP is enabled, and a page fault occurs in kernel mode for accessing a user address, the kernel will rightly panic as no valid kernel code can cause such a page fault (unless buggy). There is no valid correct kernel code that can generate such a fault, therefore this behavior would be correct. BPF programs that currently encounter user addresses when doing PROBE_MEM loads (load instructions which are allowed to read any kernel address, only available for root users) avoid a page fault by performing bounds checking on the address. This requires the JIT to emit a jump over each PROBE_MEM load instruction to avoid hitting page faults. We would prefer avoiding these jump instructions to improve performance of programs which use PROBE_MEM loads pervasively. For correct behavior, programs already rely on the kernel addresses being valid when they are executing, but BPF's safety properties must still ensure kernel safety in presence of invalid addresses. Therefore, for correct programs, the bounds checking is an added cost meant to ensure kernel safety. If the do_user_addr_fault handler could perform fixups for the BPF program in such a case, the bounds checking could be eliminated, the load instruction could be emitted directly without any checking. Thus, in case SMAP is enabled (which would mean the kernel traps on accessing a user address), and the instruction pointer belongs to a BPF program, perform fixup for the access by searching exception tables. All BPF programs already execute with SMAP protection. When SMAP is not enabled, the BPF JIT will continue to emit bounds checking instructions. Acked-by: Puranjay Mohan Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/mm/fault.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index e6c469b323cc..189e93d88bd4 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -21,6 +21,7 @@ #include #include /* find_and_lock_vma() */ #include +#include /* is_bpf_text_address() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1257,6 +1258,16 @@ void do_user_addr_fault(struct pt_regs *regs, if (unlikely(cpu_feature_enabled(X86_FEATURE_SMAP) && !(error_code & X86_PF_USER) && !(regs->flags & X86_EFLAGS_AC))) { + /* + * If the kernel access happened to an invalid user pointer + * under SMAP by a BPF program, we will have an extable entry + * here, and need to perform the fixup. + */ + if (is_bpf_text_address(regs->ip)) { + kernelmode_fixup_or_oops(regs, error_code, address, + 0, 0, ARCH_DEFAULT_PKEY); + return; + } /* * No extable entry here. This was a kernel access to an * invalid pointer. get_kernel_nofault() will not get here. From patchwork Sun Nov 3 19:35:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13860561 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 331EC18756A for ; Sun, 3 Nov 2024 19:35:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730662523; cv=none; b=fuYnWIxdejCdAr2Znzq3s/M1hQHjtX1YKACsnZa421TPVieJqcvLt99HvGt7HpwourzNadScKVb9k76O71q/vIx0yWmgxw2sPAbZ2fE04ePtvY9zkiiraZERcacMrw/iGd+zZA9NFBTKuoempTAIbRoWyNWXv7ypZ66S5VyLxRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730662523; c=relaxed/simple; bh=24tNqLBMPAstXcEG50Y99Zp8jdqY3afmUcIQKWuiHBI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kbznUZpnUtKmr6JhLihE6vq4/N/o9uDgdQ2+vvODVYhxdrqV+fipwxZPQpRqIUXHAvomDxhpBWnZAS/rV9ElGjgo4jZSm8BeOkRg1sc20y/adhux8fgt2CukiwGRlypUIady8Nak6M3dYXMVFOk9Wbz3bAA9S1qfxvUm4iF6QP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=W/azD1fr; arc=none smtp.client-ip=209.85.128.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="W/azD1fr" Received: by mail-wm1-f68.google.com with SMTP id 5b1f17b1804b1-4315c1c7392so30293525e9.1 for ; Sun, 03 Nov 2024 11:35:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730662520; x=1731267320; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A5p2hCLjfzedLrY/PuAn7mcGY9AOeiqSMiTahDqn2M8=; b=W/azD1frCNpPIWjOLx2Puu+ddP64FthtG6D4oRkPSBk5RtBZcwcI6VmhkYUfasEKgW YlRO7xVG3NClmCtwinG483e41S96JpLRTdCI11W2dzJ59pOMh60BMA4UG/1r24+ZLtrV qtRIslacaprLzvrSKct367qL6DALMtFUu/7tJvScPMVMVK9gxzi+zdne8i7NjJdT6QQ1 JfTs0lSWGQPXuLSbygCDurVyLgzJYRM+6agCUV+OkwfEpNJ6xSVNGHe3edz5/Dgs8S+2 v2pNHHdGCs4o3cA0u0XQPgL4sBdtT2D7/OrcKRLmJJs8FE2FBgzjA/Vo2/helekpUV2A JuWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730662520; x=1731267320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A5p2hCLjfzedLrY/PuAn7mcGY9AOeiqSMiTahDqn2M8=; b=hFz7EoUZqFxMtNYpx4ATqnv2I4ac4xz39Qdx2K4X6e2tIDdEFEOMMQHwtCa2rEeLfu IIOo2q3Wt73PN+zS6Cbl3nAPHqiRgVufiUulOtFLx8NVbbCVxmL/fgNVlCwT6s+uuQP6 nN53g/w/P7FcTFnaIMzLpBvz5gslPF+DIKAacmy897ENAePVsugUZv9/AvrYks6GPnHh nPnvfybej25Ed61Pch4e4Wpr0q83a5ZbKiv9VRt7uMgY5MG8+t4h4a4tlkFlPl6OVLu5 O2vJfqASTjYNGo4zvax4r9vv61ddz4WVSAAYV4nV9eQYHShdzl8TVZonBzCFcNzBYzfG BHNQ== X-Gm-Message-State: AOJu0Yz+JpQv5TgvF2/9Oo3cmPtd/E5/KXnD3cNpbIYrrTvHogWaC2Tw OtVR/T/0npX7tafnw0pNLkoh7UGTRAgeAH3PQ/ODlctOuPLPQmG0wRYAHVN/MJz7Eg== X-Google-Smtp-Source: AGHT+IFbr402wBj+hpQPvD2i9jDstKL9Q+NVWiMoDudAjJ/gOXraSmEbbdg6yxu3CCQbP5KgCAR7MQ== X-Received: by 2002:a05:600c:a41:b0:42a:a6d2:3270 with SMTP id 5b1f17b1804b1-432832562aemr83498845e9.21.1730662519697; Sun, 03 Nov 2024 11:35:19 -0800 (PST) Received: from localhost (fwdproxy-cln-004.fbsv.net. [2a03:2880:31ff:4::face:b00c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d6984e3sm135561255e9.48.2024.11.03.11.35.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 11:35:19 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: kkd@meta.com, Puranjay Mohan , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Alexander Shishkin , "Kirill A. Shutemov" , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Rishabh Iyer , Sanidhya Kashyap , x86@kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 2/2] bpf, x86: Skip bounds checking for PROBE_MEM with SMAP Date: Sun, 3 Nov 2024 11:35:12 -0800 Message-ID: <20241103193512.4076710-3-memxor@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241103193512.4076710-1-memxor@gmail.com> References: <20241103193512.4076710-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2747; h=from:subject; bh=24tNqLBMPAstXcEG50Y99Zp8jdqY3afmUcIQKWuiHBI=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBnJ8wNYj993Q8YuJ3N3/xm4JC6LFtTeTnFQeqBwPFH hLHLzWOJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZyfMDQAKCRBM4MiGSL8RykKCEA CYpwoa6CNkRVcqatnsqU30bDVX7ZP++vFvX94oVWuRuT/vCebs8vtOHD6DTbnh4QnFaoNYuJoIKYdG BwCaCk4QZSqTNMWUUbKMRJ+Tt8wTcsg4Fpyy6hEWwNCfodvp00g/IRzXE/ljgQvVa447PBDGy7ugla 7f0aZSCz6hQudOIbHqxM8fK+T4PmpNzbXJWndNLTaUQmLpy5MEuSuK+uJ/cgTb/eleQW5eSLot/TxA ANgZ6UjlRj7bH0dmpj0o2mOzCWtB9MnY8rbj0OrkiWpdU1IXP2LM78uwLPPy6Kp3hweTlAdBumJRl2 kAZqLFluRLty1JeDV4mal4r26vMjz5UyHKLgVghNnR0IHZvwM81vtyagoJ9iCJu1QDYaqZVyFIBBpu DMGm6Q5gX3x9NGV0jdvgndIbrLqf2NqliFEqsgMkujKnmdxfAhrcq0M+icN3Xs0EUyOu0Nba/MqM0t PXGRW/ZndSHZzrt7Yf3bjWLWhl56I00xndV+5sCFtOdJxnSoNSo2gm4Ysrb9MzYyuFVJ8yCYjvNJwo I6o2nXk4OCmoFOYMT4GXvZFoPHOp2GkfP9KjtOHAYmjPnkVvuRoppcBwB88/wttexPQe7YfWyJWsHh TdWg58LFlayUgMgvNuOFJXvIe4/zuyjwB0m38+mtuD8+Ny4VtRgvI6TgareQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net The previous patch changed the do_user_addr_fault page fault handler to invoke BPF's fixup routines (by searching exception tables and calling ex_handler_bpf). This would only occur when SMAP is enabled, such that any user address access from BPF programs running in kernel mode would reach this path and invoke the fixup routines. Relying on this behavior, disable any bounds checking instrumentation in the BPF JIT for x86 when X86_FEATURE_SMAP is available. All BPF programs execute with SMAP enabled, therefore when this feature is available, we can assume that SMAP will be enabled during program execution at runtime. This optimizes PROBE_MEM loads down to a normal unchecked load instruction. Any page faults for user or kernel addresses will be handled using the fixup routines, and the generation exception table entries for such load instructions. All in all, this ensures that PROBE_MEM loads will now incur no runtime overhead, and become practically free. Acked-by: Puranjay Mohan Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 06b080b61aa5..7e3bd589efc3 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1954,8 +1954,8 @@ st: if (is_imm8(insn->off)) case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: insn_off = insn->off; - if (BPF_MODE(insn->code) == BPF_PROBE_MEM || - BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { + if ((BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEMSX) && !cpu_feature_enabled(X86_FEATURE_SMAP)) { /* Conservatively check that src_reg + insn->off is a kernel address: * src_reg + insn->off > TASK_SIZE_MAX + PAGE_SIZE * and @@ -2002,6 +2002,9 @@ st: if (is_imm8(insn->off)) /* populate jmp_offset for JAE above to jump to start_of_ldx */ start_of_ldx = prog; end_of_jmp[-1] = start_of_ldx - end_of_jmp; + } else if ((BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEMSX)) { + start_of_ldx = prog; } if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX || BPF_MODE(insn->code) == BPF_MEMSX) @@ -2014,9 +2017,13 @@ st: if (is_imm8(insn->off)) u8 *_insn = image + proglen + (start_of_ldx - temp); s64 delta; + if (cpu_feature_enabled(X86_FEATURE_SMAP)) + goto extable_fixup; + /* populate jmp_offset for JMP above */ start_of_ldx[-1] = prog - start_of_ldx; + extable_fixup: if (!bpf_prog->aux->extable) break;