From patchwork Tue Oct 5 08:26:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535817 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD777C433EF for ; Tue, 5 Oct 2021 08:30:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7715613AC for ; Tue, 5 Oct 2021 08:30:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232365AbhJEIbx (ORCPT ); Tue, 5 Oct 2021 04:31:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230286AbhJEIbw (ORCPT ); Tue, 5 Oct 2021 04:31:52 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5A59C061745; Tue, 5 Oct 2021 01:30:02 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id b22so1827034pls.1; Tue, 05 Oct 2021 01:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l/X3D8axCR8R6Iv8FHRMGZCljg4SY7rI3XtaUgh1lNc=; b=Nsn0Y4fntJPCapUQIivLcByF5/yz08ne98pBVKNURr5S119buvbXXVg9Cg+PfiAVCY vW+yUX0rm/dZGQakisBI9WGHB+vNJrGwMzAviJhZLWr9+HK1ZGssZnld/VH1jle8/xRB mhe2b1ahLM0qCBkU00C0RUjQksK4eH1aD/kLSWqCLIKHugqKeNB6Ad8xXG0s+JZJG1RL X11K+FvTfr35PJT6RTlhU9BZRSx5zkD0Ip8R7sehA5HkeXbjnmQvBKCqA4jbc7vagCQu aQjc6+sUDnRcjkLrd/rvZOF6H9WJZXXkaXyREkU44S7Bgvik3YRF2arPWG8Y7lutlZxX uojg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l/X3D8axCR8R6Iv8FHRMGZCljg4SY7rI3XtaUgh1lNc=; b=wm8006xIDXIudvGYhM1S2MZnliuPGSJx00RsJjTp4/5KvGAEGy5XVGS7VOU/Uku9s/ ycZmF2zVG72A4Bx/yxj3L1Dzck3yKxzbZOnU6OYxGIka+LlOnQfsknknkdgqYCYCMNJb 3tagraf+BJq+YZBDV4VWBhTZUMojNt+pZdkYnF2wNczQr4k58/MT1VdXfEx2sbwc9Vfu p9VpJQ1JO/MTOLRv90PyOhnnryI8fQgS5y3sTzB4YP8ak3eesMkV+LKmVZ/lv24LdPzT MaYI2pbinqOY2mJ3EZMh0RqtQ3ZlJVv0dYIg3yBu+QDv1BoKHXd7jq7a9QuY/H/LyjNF hhHQ== X-Gm-Message-State: AOAM530eMXHqbUmR0erg0q10RllW/NUNHouhhu1tnPBuQlvJtPLXb48/ J9PAZpJk0ZX4q4+hyUJWYQk= X-Google-Smtp-Source: ABdhPJxo2Q4SBzL3ZP1UWnJkgImTo3uNqIpC7e6zID71Bvd0KWgJm6FHNjNaqLW/NTgEhxBVDUvr6g== X-Received: by 2002:a17:902:8a83:b0:13d:9572:86c2 with SMTP id p3-20020a1709028a8300b0013d957286c2mr4173829plo.48.1633422602452; Tue, 05 Oct 2021 01:30:02 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:02 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 01/16] MIPS: eBPF: support BPF_TAIL_CALL in JIT static analysis Date: Tue, 5 Oct 2021 01:26:45 -0700 Message-Id: <05a0d17bef1394521e0aec4b33d2ea3c29c715e5.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add support in reg_val_propagate_range() for BPF_TAIL_CALL, fixing many kernel log WARNINGs ("Unhandled BPF_JMP case") seen during JIT testing. Treat BPF_TAIL_CALL like a NOP, falling through as if the tail call failed. Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 3a73e9375712..0e99cb790564 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1717,6 +1717,9 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, for (reg = BPF_REG_0; reg <= BPF_REG_5; reg++) set_reg_val_type(&exit_rvt, reg, REG_64BIT); + rvt[idx] |= RVT_DONE; + break; + case BPF_TAIL_CALL: rvt[idx] |= RVT_DONE; break; default: From patchwork Tue Oct 5 08:26:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535819 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A9FDC433FE for ; Tue, 5 Oct 2021 08:30:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 00B49613D5 for ; Tue, 5 Oct 2021 08:30:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232511AbhJEIcF (ORCPT ); Tue, 5 Oct 2021 04:32:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232761AbhJEIcE (ORCPT ); Tue, 5 Oct 2021 04:32:04 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7D7EC061745; Tue, 5 Oct 2021 01:30:14 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id b22so1827385pls.1; Tue, 05 Oct 2021 01:30:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zzGe5Wy3mDkKEKYCT9h+VBJHOg/ijD2uRySpX5DGZX4=; b=LDJs7JXie2s8SUMGrGQq0hEMaanjTjMnRkTtWaBRu1FZSWgF9Dx1nnAMLm2+QYU8dq jNVwoVPuuv6zHydYnMfH5iGuI8LE6O/5s8t1mJ7+/ZXRxbKTLinSoniLfOVqUBPlqI9z zAWmwvkWgmeRJtYILVfZOzEnrWyVYgyh5uXIZ++WKMZUzDw4QHPYHjkGc4gROCpry342 qXn17Ropc0dmtt3OybtFms7D8pinUvVKIfVwPN305DpR8M6HSZIGky5BNl8kcDh8eq8W WQ5mUv5YrXmXGWT9lmqURmoWrIAUM+7JAgiaO0KyOdGiFPiregxVSmnCIUl8ibLhDXqG AOoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zzGe5Wy3mDkKEKYCT9h+VBJHOg/ijD2uRySpX5DGZX4=; b=xkeYGI3hySRqzfmUFb8aa7t/Tx8TH9RliTANda5TgJb1ntGWByZ91wJCwLualxvTFG l/b+860o0uHI2LqH7aDjDTB3EIUeCgis3vm1vkNiaNrpjf0YXu5zLe5OhCfskZbM6wN6 c9za1BDZZHLJIm8uQZWFn81yiLPPw3/6ubS1ORXvBA89QTse2J91s39Ez3a40lEVmso8 iKlg/UjL5WO8TikrZ7ty5eyOHskzXr4HaeVjMh2mLMiP/vKcrwBf05na3o3Yh7+7BMWV 37oDNs0z8D8krkv/RcWJ1s1yLqZMWcuELLF4ewynHbHyKVUxdaXv0QlZCbDNPfdlY+Q8 //JA== X-Gm-Message-State: AOAM5310+LlP4yQa25E75ORmwMPS67GGlSMmCidWbhDzV2DO25Fz3cRV wYBLUe5Qia8DMxmGrZQzI/M= X-Google-Smtp-Source: ABdhPJzrq+Z2N/ppb2aKdvI31i8JGUgD1xJW9WP49KjOZpIHNwyQa9rjT21w/d2v6Qkd02EQPI/uEg== X-Received: by 2002:a17:90a:9f91:: with SMTP id o17mr2263689pjp.225.1633422614497; Tue, 05 Oct 2021 01:30:14 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:14 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 02/16] MIPS: eBPF: mask 32-bit index for tail calls Date: Tue, 5 Oct 2021 01:26:46 -0700 Message-Id: <15d2aab0133231aea254bd7422528d4a765d5f0f.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The program array index for tail-calls should be 32-bit, so zero-extend to sanitize the value. This fixes failures seen for test_verifier test: 852/p runtime/jit: pass > 32bit index to tail_call FAIL retval 2 != 42 Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 0e99cb790564..82ea20399b70 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -611,6 +611,8 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx) * if (index >= array->map.max_entries) * goto out; */ + /* Mask index as 32-bit */ + emit_instr(ctx, dinsu, MIPS_R_A2, MIPS_R_ZERO, 32, 32); off = offsetof(struct bpf_array, map.max_entries); emit_instr(ctx, lwu, MIPS_R_T5, off, MIPS_R_A1); emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_T5, MIPS_R_A2); From patchwork Tue Oct 5 08:26:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535821 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EED8C433FE for ; Tue, 5 Oct 2021 08:30:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0017E61425 for ; Tue, 5 Oct 2021 08:30:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232979AbhJEIcV (ORCPT ); Tue, 5 Oct 2021 04:32:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232658AbhJEIcU (ORCPT ); Tue, 5 Oct 2021 04:32:20 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C4D1C061745; Tue, 5 Oct 2021 01:30:30 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id cs11-20020a17090af50b00b0019fe3df3dddso1531759pjb.0; Tue, 05 Oct 2021 01:30:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dGdTtYrmJ1H2+tRpCqMyGgjc5iCxgfgYhTTRd7bVUbY=; b=LApmQoVGQRm/nto+Z1864v0TQawWYqePSZJO1E2kwEQhXpKjtGZWgx8cjMH1ARXqG2 x/HvmD4JNIpoKXt7Wwbtt9gGQsTwbdzYFEke9nAi7qv4Rq5AovshjC/L18Dr4GHjsKje rFhUIXnrZci+QHORiJxsPQRclgg+beOraJA6+BRZbh0TstyuFwSfHUpuxcEMgEWL/4DC v8nAjxEo30toJNgNSPlDYHE78NBYS7CoPwpI7DbHbPzgpZ/jBpXXspYUAH7GIB27O36h l+nyT5ItJJUjP9tLncNnGgW6QI6bXqhSzVzN5s2yM3RnNHN+V3ISDPblIM6ojcOo1oQz k4XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dGdTtYrmJ1H2+tRpCqMyGgjc5iCxgfgYhTTRd7bVUbY=; b=hbA6TeyV/Vl/29oIPoMGq0CaoQazBTtADNJnqpyBwqwTdq9vQe+xgAFZAyPImfh8ya 79jhilEqv45A3o5d/dLsxF/YoIRllnHmweDDBR1bRDMIyYNZ85e+Oug75uKN0bt+0Kuz TPMSFjZc+dNxkmrof0t670pNtQTnmsCP62CER0bdGLFFF9e/M7y/m82PvgKXck5sExIP WSYBSdsjBPo14jxknMkpjkR6Ofio8NUdw3H00vzSCvlhLFI7LPQ8Qqfcmx4rueysGAIK REWih7NISA/JTRlT49HlVfE1teWwMH3n4EdUzK9/jn2u902WFnYPQHIhZUuxOABCtqn+ Idwg== X-Gm-Message-State: AOAM530NsnwM1/rNu8WuQxz5o3d5fyhmsBPR+4/dkkf9IciQqSlqp99f Zyi71fyVJklG1DxbOCa99EU= X-Google-Smtp-Source: ABdhPJyb50NUklWBG1NtqkaFC+LDythq55+NAZsQJiXK6F2yOU48aIpwY/7hwkUqfdSd6Gqp4TRxvw== X-Received: by 2002:a17:902:7297:b0:13e:6650:a4ca with SMTP id d23-20020a170902729700b0013e6650a4camr4169302pll.37.1633422630289; Tue, 05 Oct 2021 01:30:30 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:29 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 03/16] MIPS: eBPF: fix BPF_ALU|ARSH handling in JIT static analysis Date: Tue, 5 Oct 2021 01:26:47 -0700 Message-Id: <1f88a0febdee93743eb438cfd28b82bbe17c2650.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Update reg_val_propagate_range() to add the missing case for BPF_ALU|ARSH, otherwise zero-extension breaks for this opcode. Resolves failures for test_verifier tests: 963/p arsh32 reg zero extend check FAIL retval -1 != 0 964/u arsh32 imm zero extend check FAIL retval -1 != 0 964/p arsh32 imm zero extend check FAIL retval -1 != 0 Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 82ea20399b70..b41ebcfb90c4 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1590,6 +1590,7 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, case BPF_AND: case BPF_LSH: case BPF_RSH: + case BPF_ARSH: case BPF_NEG: case BPF_MOD: case BPF_XOR: From patchwork Tue Oct 5 08:26:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535823 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18F75C433FE for ; Tue, 5 Oct 2021 08:30:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE82361425 for ; Tue, 5 Oct 2021 08:30:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232658AbhJEIc2 (ORCPT ); Tue, 5 Oct 2021 04:32:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233318AbhJEIc2 (ORCPT ); Tue, 5 Oct 2021 04:32:28 -0400 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C36FDC061749; Tue, 5 Oct 2021 01:30:37 -0700 (PDT) Received: by mail-pg1-x52e.google.com with SMTP id v11so6509478pgb.8; Tue, 05 Oct 2021 01:30:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fBG696NEpSwIBlXH0TMuXB909IPwtFq8p/NuyOUiOsc=; b=BKMM/GwyCarVz0hwngv5UYK5qMe4WkDZ/1BzG/8T/F9CD1/Hl5nut7LDe/ddtScEOx kt2zL4wNzs2P+QjlInsdup5DqqVNx0DLFXN7/rw3i8dGTlRZRv299Xe90KZvKHDM1Pkr zwUGf5pTKUCj3afYs51uVZVd3KZqvz1zq6RT5JiSAUmw+MV8cSBMCUIBDiLXAAwyB+bE UCuBiLvllVDu5hjAecjbz8a+1KZrjQrmQL0FokivnfSbW60R4tL7JK4ov5+LK7qc5NgA MWOzOySfcufv3Twi1vUCYBUuEQN3VLkbAbnUhW8nMbKCNU8cpairduMyCHOIhE/7qWhl owjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fBG696NEpSwIBlXH0TMuXB909IPwtFq8p/NuyOUiOsc=; b=QxtsAREiX/YwVuTyGU84c1MwLVFmnQ3+XyTfAmnepc0VCUTCwhS0mIy9jZi5dQqwit jwO90I8PIZPjXUDuYqg04aLsyBoamEd0FgbIy6dfstogcOVHClbEaw7cXEtT4qTeakg9 xqMq+NYUGNVcrXytwOLxTClKs0WeB5eC8HN8rVqLMjRiX1d/OteTSETeL0Pd9134c+eP uu/BoUjRNP7jq9SU4f0y+h2FPjhTEW29E0EBofeSi/p2VqFhFMmRH2jFRRhC03dfA4Pj CQnOxb78FmZtpBUQladmM7aKa2KnVqUe8K/7shXjh9j1HbhdNB4gClB1Zy10JdpUVEyh O5KA== X-Gm-Message-State: AOAM530QwibgBg5w1qawnCGpNyJslDz9/2iVkyisqndV09zLi5vq4ep4 6AakBc8545XGHyEQKIEx9Kk= X-Google-Smtp-Source: ABdhPJwhMimbJz9LYq+FXgDCm1WqNWV59ayamwzKcBveNHS/UiEkra2KlzEfbrjeP80lIYeSs6M8ow== X-Received: by 2002:aa7:9005:0:b0:44b:fa4e:95c2 with SMTP id m5-20020aa79005000000b0044bfa4e95c2mr28003296pfo.29.1633422637435; Tue, 05 Oct 2021 01:30:37 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:37 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 04/16] MIPS: eBPF: support BPF_JMP32 in JIT static analysis Date: Tue, 5 Oct 2021 01:26:48 -0700 Message-Id: <8ca5dfaa9ad749e643f9246d2434c12a1c21150a.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC While the MIPS64 JIT rejects programs with JMP32 insns, it still performs initial static analysis. Add support in reg_val_propagate_range() for BPF_JMP32, fixing kernel log WARNINGs ("Unhandled BPF_JMP case") seen during JIT testing. Handle code BPF_JMP32 the same as BPF_JMP. Fixes: 092ed0968bb6 ("bpf: verifier support JMP32") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index b41ebcfb90c4..dbde5d6eefa6 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1686,6 +1686,7 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, rvt[idx] |= RVT_DONE; break; case BPF_JMP: + case BPF_JMP32: switch (BPF_OP(insn->code)) { case BPF_EXIT: rvt[idx] = RVT_DONE | exit_rvt; From patchwork Tue Oct 5 08:26:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535825 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B29BC433F5 for ; Tue, 5 Oct 2021 08:30:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 072A2613AC for ; Tue, 5 Oct 2021 08:30:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232723AbhJEIch (ORCPT ); Tue, 5 Oct 2021 04:32:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233382AbhJEIce (ORCPT ); Tue, 5 Oct 2021 04:32:34 -0400 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C4E0C061760; Tue, 5 Oct 2021 01:30:44 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id u7so16739932pfg.13; Tue, 05 Oct 2021 01:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0qfPr+t/+gvCFeroYYa6+cQZP2abmSI5vv080z1ELL4=; b=O0n8t2sTB1QY2tNqoK3WMNgRGHTfcpwbjiBUjvCP3PMXTK56gk1+ScMSupxUaZOQVy iOof4U78vD9QnNGxO0DyQM7d3vqp5AeiJiaq/baAWiEM8yuIoYpClTyWMsG6c0y5U6bf 9v2qOUkqPDs+8m6KzBP1Wqn5ZSZq7Lq+nYVOQ1kWHZMcWp2Pe9irg+OfqMty4owzILHM Wnzk4evor8JKaTUkvSGW44OkTA29/uGyHCV8r76dYIYnsdJuJK8TyIDt6dMHq5L5MUbe vruikwMpFVHhiCNKqt1jEaofPv/Q5UTH4zF76V0ZHOY5cXcI9T/oI2fgT/cBsa7SQAGL hs/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0qfPr+t/+gvCFeroYYa6+cQZP2abmSI5vv080z1ELL4=; b=JdKoJWG8LYy+9TXrz4U8KCSyJUYPb1WINep4OBLiUz9Nu6EXmXzKvasXqCpFGaIZ/O /KZjQYa+HPFziegDKkD0lYpZ5gLYUaP6A1X8xkL40mr0XEunLnV0wErElWBq1PZDMGnG j+AMGaItYuEuscKn3cz+yfXDTZBM9q6ALK8O4jfUwyfddVvMpz4VbFS+FP7xApCUvuSg Jb/6+MQEL9f1dNYGAWZ6/36TK7/yz7PznIt8KZ5dDAA2TgwZKAxtVxS9mUcQ6mLQ35H5 vcNl46vdvFrvc2grbfieTDk8fy3yCVVzHJSBzkdhiol0R1Zhq8guXcU5vYUDQ6QpCInb ztaA== X-Gm-Message-State: AOAM531ifiZRXmdOqC9olGHTHLqn3IN/W9AmWceHLaSwTjF3vZJpuGgM NTGslCfeR/dz36Yti49NV7U= X-Google-Smtp-Source: ABdhPJw+PHqLUKzhmcgXiLynAZiC4lepD11qnAgaBvzWHaTQNcFxJ9B3/wT772P03lgTTaZUIsT8VA== X-Received: by 2002:a62:3881:0:b0:44b:6639:6c20 with SMTP id f123-20020a623881000000b0044b66396c20mr29442728pfa.78.1633422643878; Tue, 05 Oct 2021 01:30:43 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:43 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 05/16] MIPS: eBPF: fix system hang with verifier dead-code patching Date: Tue, 5 Oct 2021 01:26:49 -0700 Message-Id: <97887c36e932374626a3022f5d84e527414d106c.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Commit 2a5418a13fcf changed verifier dead code handling from patching with NOPs to using a loop trap made with BPF_JMP_IMM(BPF_JA, 0, 0, -1). This confuses the JIT static analysis, which follows the loop assuming the verifier passed safe code, and results in a system hang and RCU stall. Update reg_val_propagate_range() to fall through these trap insns. Trigger the bug using test_verifier "check known subreg with unknown reg". Fixes: 2a5418a13fcf ("bpf: improve dead code sanitizing") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index dbde5d6eefa6..0928d86cb3b0 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1694,6 +1694,14 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, return idx; case BPF_JA: rvt[idx] |= RVT_DONE; + /* + * Verifier dead code patching can use + * infinite-loop traps, causing hangs and + * RCU stalls here. Treat traps as nops + * if detected and fall through. + */ + if (insn->off == -1) + break; idx += insn->off; break; case BPF_JEQ: From patchwork Tue Oct 5 08:26:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535827 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC9C3C433F5 for ; Tue, 5 Oct 2021 08:30:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 983D8613AC for ; Tue, 5 Oct 2021 08:30:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233400AbhJEIcm (ORCPT ); Tue, 5 Oct 2021 04:32:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233391AbhJEIcl (ORCPT ); Tue, 5 Oct 2021 04:32:41 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57801C061760; Tue, 5 Oct 2021 01:30:51 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id pi19-20020a17090b1e5300b0019fdd3557d3so1819069pjb.5; Tue, 05 Oct 2021 01:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i5TQ/GNjv5Hjjqp29xrTfJl5zRBuzrKjJ4+0fxARHB4=; b=TEkGbWWht6AFtnUdWjagkEsYiTdhi0l6fWIL/tVtwmpAfaacCiy0nioxisv5i0fns7 8PxgIKYmB/QK4OZ8uUZObGpADSi01Ww6VU5+ZYbalZuvaa+codryJhKcrt/FbzL//slW 0SLCp4iUuz+16nupkmImvU3AAyV2t3t46QAhzsM4v8FNQWWClt9zOWhAFUsRUePXAB5r 5QZVJfzQZ+S0OnaRheMIZEewdyInN5O+QjGpQAkW6ICtzG9OFYbgR7d9lZKJNV1sts59 uYuGTtixwnk0uBgAXGGsNXO1PfjWL1fj+ts472O7XG2fpRZaUTdSFOCSKjjLKZi3hKMz Zhhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i5TQ/GNjv5Hjjqp29xrTfJl5zRBuzrKjJ4+0fxARHB4=; b=aYSpidB/D8qY1lPpCiIZFo8mPI5h+taaBidLH9g4Ud6HsC6sxm96kKH1OspobrnX0H tQK2li37qN6Sk3I5m2NNpH5snCyGV7uASekl2H0Jrly3+k5N7nmHf6sGT+i9XAcNKIaP ZCQdfOVR+3d9HDkkLsYWr0wHTCmR+V3Cpmb+tVDdWeEr5XyGL0VyVBrNuXj+E3Hq4iZe TCX49ki4fGJvxSaNOdv80IsUDiAlSdnZ9hSNRQ1wWsKBva25v/OwpVR9havi7/IAA+Wg V6wQJcZ7Motl/wiyZJ5vsirJocQYuEQpF2VcgOkDtXD69WkhPAOC36ReG0TpNniKaFT/ TGbA== X-Gm-Message-State: AOAM530ZxNfT8Ab9w90GjkdCA8vBK2QlISfeiLiSpMkCknqMCqIXVetf ivNq10CtjSJPR3qltBHlu3I= X-Google-Smtp-Source: ABdhPJzfWZqsHKyOuRjJE94Hjnly/yfdOhN5F1c+IBzYfDEZxI/EMIq65wLlbPIJXgno3w4B9Ssuig== X-Received: by 2002:a17:90b:a17:: with SMTP id gg23mr2277722pjb.18.1633422650965; Tue, 05 Oct 2021 01:30:50 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:50 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 06/16] MIPS: eBPF: fix JIT static analysis hang with bounded loops Date: Tue, 5 Oct 2021 01:26:50 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Support for bounded loops allowed the verifier to output backward jumps such as BPF_JMP_A(-4). These trap the JIT's static analysis in a loop, resulting in a system hang and eventual RCU stall. Fix by updating reg_val_propagate_range() to skip backward jumps when in fallthrough mode and if the jump target has been visited already. Trigger the bug using the test_verifier test "bounded loop that jumps out rather than in". Fixes: 2589726d12a1 ("bpf: introduce bounded loops") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 0928d86cb3b0..b8dc6cebefab 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1693,6 +1693,10 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, rvt[prog->len] = exit_rvt; return idx; case BPF_JA: + { + int tgt = idx + 1 + insn->off; + bool visited = (rvt[tgt] & RVT_FALL_THROUGH); + rvt[idx] |= RVT_DONE; /* * Verifier dead code patching can use @@ -1702,8 +1706,16 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, */ if (insn->off == -1) break; + /* + * Bounded loops cause the same issues in + * fallthrough mode; follow only if jump + * target is unvisited to mitigate. + */ + if (insn->off < 0 && !follow_taken && visited) + break; idx += insn->off; break; + } case BPF_JEQ: case BPF_JGT: case BPF_JGE: From patchwork Tue Oct 5 08:26:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535829 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC01DC433FE for ; Tue, 5 Oct 2021 08:31:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BAF57613D5 for ; Tue, 5 Oct 2021 08:31:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233441AbhJEIcu (ORCPT ); Tue, 5 Oct 2021 04:32:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233366AbhJEIcs (ORCPT ); Tue, 5 Oct 2021 04:32:48 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 824FBC061745; Tue, 5 Oct 2021 01:30:58 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id oa6-20020a17090b1bc600b0019ffc4b9c51so1364821pjb.2; Tue, 05 Oct 2021 01:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DAGNoLYAM13FCAsShiHivQ1p6RlYfr33Xe38II2bD8U=; b=cBDmZU424tuQmQIoC3gjJYq5sHDqh+yCBNy2EH1IMKIbT5rkl9vk+5RTjfb8AQ7MFI dC2ULIApzwL8Wk1GHv9W9SXUogddOGtt1oFvKrVuQQg8ucZcztM1fJYzdfzRsSG6r/GE QDaMLodiHKPyOMm+iFfzxrjhs4A/dHn34XWXbRUPfql3ACKK7JG3LIQFA6ADa1s5/R65 bB8d3hTaqQ/cAc6apMOpbiycNiA9pYskY+XC96U9eKL+lnoNRvT9Mioelkkf640EoQhz l+hn8d0lgu3KoJ5yGaUHs/6nEOJ5DDlNsbR757s6pUqSMZhE1Y+g+TvO/xZltr79xca5 5dng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DAGNoLYAM13FCAsShiHivQ1p6RlYfr33Xe38II2bD8U=; b=GpqIbrh+2hAm9jxdl6tykLu1nMyWh+lPCCJkGRvP3or/IHsJqcnQEXD0UbiJjNnd5Y gl1MsLaE4jMH4PXKZ2Oa64imLu7LPBwzJyulXNboi+ufoSTlfWZzIS4TjdY4Kk2ZGfWe NV1NxAGzr+CbSo3Pg39eX9XWCXohIPyMyOknZtl4XfMo4TYPBpFE5dn63vXtfw8+BL9q 1LGkhKDNPPNYBF6F+UWnTOwXVJyRmJFHQcx1qfhjJhf49zgrrqyxdZEvgYhAeeg/vXKb lA8DCPfB4VZV8Kuclbp13+XxfGelhvZknpzG0imlTCYQAtgzzvus77To7xxMx87Y2Wzl kB+Q== X-Gm-Message-State: AOAM531Pq8u8J25zMhhml3QSpfWLDjNzwUrO4qztSwzWVTh74moz/xXq tR/2vuuQ6M/v0pxLDzu/Icn//Eo9xT/bRw== X-Google-Smtp-Source: ABdhPJz/t0tF4B5I4040OsOvyCt2OxNMZtEY5lMth30IA55yWXN5QOuG8MDZSgY1PeOduk0kdTtOiQ== X-Received: by 2002:a17:90b:3901:: with SMTP id ob1mr2329767pjb.12.1633422658158; Tue, 05 Oct 2021 01:30:58 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.30.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:30:57 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 07/16] MIPS: eBPF: fix MOD64 insn on R6 ISA Date: Tue, 5 Oct 2021 01:26:51 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The BPF_ALU64 | BPF_MOD implementation is broken on MIPS64R6 due to use of a 32-bit "modu" insn, as shown by the test_verifier failures: 455/p MOD64 overflow, check 1 FAIL retval 0 != 1 (run 1/1) 456/p MOD64 overflow, check 2 FAIL retval 0 != 1 (run 1/1) Resolve by using the 64-bit "dmodu" instead. Fixes: 6c2c8a188868 ("MIPS: eBPF: Provide eBPF support for MIPS64R6") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index b8dc6cebefab..00dc20bc0def 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -800,7 +800,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, if (bpf_op == BPF_DIV) emit_instr(ctx, ddivu_r6, dst, dst, MIPS_R_AT); else - emit_instr(ctx, modu, dst, dst, MIPS_R_AT); + emit_instr(ctx, dmodu, dst, dst, MIPS_R_AT); break; } emit_instr(ctx, ddivu, dst, MIPS_R_AT); @@ -882,7 +882,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit_instr(ctx, ddivu_r6, dst, dst, src); else - emit_instr(ctx, modu, dst, dst, src); + emit_instr(ctx, dmodu, dst, dst, src); break; } emit_instr(ctx, ddivu, dst, src); From patchwork Tue Oct 5 08:26:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535831 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0A9CC433FE for ; Tue, 5 Oct 2021 08:31:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 91F97613D5 for ; Tue, 5 Oct 2021 08:31:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233071AbhJEIdH (ORCPT ); Tue, 5 Oct 2021 04:33:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233271AbhJEIdG (ORCPT ); Tue, 5 Oct 2021 04:33:06 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94F3CC061745; Tue, 5 Oct 2021 01:31:15 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id l6so1808276plh.9; Tue, 05 Oct 2021 01:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AF+nX8jdqSnU6pR8M/rTaNNAOHPmdiA0ekgYVMAQg7U=; b=VuWemK4OU4D/CDs2SLRYR8s2Vou6A3r15u9bTN2BBh4Hdqah6uEWajAXp4Y5wfK4R5 1Df6KCYU8g+PxcFOLMaOs/v3H1SdG91JfrE9+3zDqldvS3OleKR2sKYu8J4L9ooyWE8N 0Mqmq008yZapjhwfraTFU8bzdRSDTyD3IpIe2ISFYvDh/DR+uUAYdTkF+6rNfNWZnTFu KWWjwZs62SpXdZmI+gEDZRw9WSUn4cyQsuoOTIoZTN+4nwmgkkb6gsHkJ2R+y5fO0duG V9URrQxc7PI2LMkXgVTow9BYHLcIYP0ODBI3kIyqvAwcJRCTjcahOnPHB96pYk1sU/vw BhuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AF+nX8jdqSnU6pR8M/rTaNNAOHPmdiA0ekgYVMAQg7U=; b=PsaalxM3FscRvYmavqqeCtMhoYYfTQY7OWMdFlyp6yheMwZVTeRvi5bM0RAr52PAxw c7cilt3hbpHMkqtojFU0aBWLPsG0hLMUtjRyRIlP2vRHB54wAnXBC5TYeUtncsRls+xl Tk/d6+Cd3/XMp1oQ7Z5WJ+RfoYveoFMzKIzdTr7K9QgVSx2ZO0JfqjpUDxQNCf3RxvWJ gbByMCr1toZkvX98mP6+qtcivPdjSPMPRy31nL2Obv3FcTV51ML1vrqUjI+NXNo4zDqN 5W/XQhd4RITHbZ9DVdhAsM0X+cNwbXecNKAioyQbDHPdrbz+7AsS5K9vP1XtFgW5vnHN xnGw== X-Gm-Message-State: AOAM532E0diGaIzgTRPJQXIicaBMiLlAaM7oLaZdaNQzdbMR1hunrb8C DQT5M3ROOTbuTkk8u3Y07u8= X-Google-Smtp-Source: ABdhPJyj+7wBDt83NXZwPR2zsCsc5Ncr8NwQgxrP/zDbR5bWN/UxB9GvDiYMcxeGmYzH1bdCSHBg1w== X-Received: by 2002:a17:90a:9f91:: with SMTP id o17mr2268834pjp.225.1633422675254; Tue, 05 Oct 2021 01:31:15 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:31:14 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 08/16] MIPS: eBPF: support long jump for BPF_JMP|EXIT Date: Tue, 5 Oct 2021 01:26:52 -0700 Message-Id: <8f7207410035a7d2b1f5e0d87695880313f5adf6.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Existing JIT code supports only short (18-bit) branches for BPF EXIT, and results in some tests from module test_bpf not being jited. Update code to fall back to long (28-bit) jumps if short branches are insufficient. Before: test_bpf: #296 BPF_MAXINSNS: exec all MSH jited:0 1556004 PASS test_bpf: #297 BPF_MAXINSNS: ld_abs+get_processor_id jited:0 824957 PASS test_bpf: Summary: 378 PASSED, 0 FAILED, [364/366 JIT'ed] After: test_bpf: #296 BPF_MAXINSNS: exec all MSH jited:1 221998 PASS test_bpf: #297 BPF_MAXINSNS: ld_abs+get_processor_id jited:1 490507 PASS test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed] Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 00dc20bc0def..7252cd44ff63 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -994,9 +994,14 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_EXIT: if (this_idx + 1 < exit_idx) { b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - emit_instr(ctx, beq, MIPS_R_ZERO, MIPS_R_ZERO, b_off); + if (is_bad_offset(b_off)) { + target = j_target(ctx, exit_idx); + if (target == (unsigned int)-1) + return -E2BIG; + emit_instr(ctx, j, target); + } else { + emit_instr(ctx, b, b_off); + } emit_instr(ctx, nop); } break; From patchwork Tue Oct 5 08:26:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535833 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5735C433F5 for ; Tue, 5 Oct 2021 08:31:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 90A576128A for ; Tue, 5 Oct 2021 08:31:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233077AbhJEIdQ (ORCPT ); Tue, 5 Oct 2021 04:33:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233473AbhJEIdN (ORCPT ); Tue, 5 Oct 2021 04:33:13 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2028C061745; Tue, 5 Oct 2021 01:31:23 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id np13so3526493pjb.4; Tue, 05 Oct 2021 01:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0vOpPFSSAKxNlwx3QHOSMlCuTCb3URMfQFvAF0txKJ4=; b=CfosujBa/4JwNAB8WcZDL/oa/QlmnyT8I+zR9lRan+iLW2TMdNkMW3s3maB8L4GkZ7 i5rVx9vwpn5Vw+ri0rIygne4doaqrWMQ8RmnIPVIFVAVz0J2NzpYalFPAKmZEHFFs7l7 fg/AzWdjAs4xDMnyvctFyhbSC1sm8SZyZJqpdgM3wDJeoEmYdeSlJJJGo3pYil3ZiSNG +CLkW6cAvgtdQf2SsvmEaZrjtqhv/UBeU3BE/iayKFJl4PeicV4Wtjtl2wKOVmb8cAo7 7WoyqYX6+ykdeq8nb6ftXrL78LQ6d9FCnCDov9IRbkoeqZKaQyE0RSYnv8d/SEqlYYBw 9emQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vOpPFSSAKxNlwx3QHOSMlCuTCb3URMfQFvAF0txKJ4=; b=J6WNAGon8DqFrgmHOClGAlEmKOfYAUo9sWaQj+elMBaovQVPkvhmQXXMe3gSXv1c89 BXNpaCShM26EofLG0GY6KuSt+K2txut9G6D7W+dRVS1GSMQ9GP9aNUThd58JutEeZWvM NgZdPL4wFfuMLKkh8/+PmHjR5iiaVa0PzgOiD3hFzjfyk/2DjaeNA958iYP2TzzbOALm V3RqnnhfdIFIy6fjGKcGxMD+e0bHQ/Rd+n+QaB+edmAhcukrhtI9u7JYIqACmPqkbEHz SCPGc/nha3/BORm/X2XpwL3Ml4bUJxQqEitrmnrEoW9IdR//BXemav66eVh3zg84LvdC YnPg== X-Gm-Message-State: AOAM5330itpkoZNVuPKlFC8LMTDEi8x2iXvatlWShbjoJiaLwkpl0IFM WooDM8UJxV8Zz3HXdhe0GJ0= X-Google-Smtp-Source: ABdhPJz+TUhiFSR80/1njDH7LkPnLoejQesZP+h3Qfm7y4T9y3JAEeFkgTki3zQ/Oja4FjPstU9HRA== X-Received: by 2002:a17:90a:4a04:: with SMTP id e4mr2247240pjh.51.1633422683340; Tue, 05 Oct 2021 01:31:23 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:31:23 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 09/16] MIPS: eBPF: drop src_reg restriction in BPF_LD|BPF_DW|BPF_IMM Date: Tue, 5 Oct 2021 01:26:53 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Stop enforcing (insn->src_reg == 0) and allow special verifier insns such as "BPF_LD_MAP_FD(BPF_REG_2, 0)", used to refer to a process-local map fd and introduced in 0246e64d9a5f ("bpf: handle pseudo BPF_LD_IMM64 insn"). This is consistent with other JITs such as riscv32 and also used by test_verifier (e.g. "runtime/jit: tail_call within bounds, prog once"). Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 7252cd44ff63..7fbd4e371c80 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1303,8 +1303,6 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit_instr(ctx, nop); break; case BPF_LD | BPF_DW | BPF_IMM: - if (insn->src_reg != 0) - return -EINVAL; dst = ebpf_to_mips_reg(ctx, insn, dst_reg); if (dst < 0) return dst; From patchwork Tue Oct 5 08:26:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535835 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E36E8C433F5 for ; Tue, 5 Oct 2021 08:31:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BE908611F0 for ; Tue, 5 Oct 2021 08:31:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233391AbhJEIdd (ORCPT ); Tue, 5 Oct 2021 04:33:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233492AbhJEIdc (ORCPT ); Tue, 5 Oct 2021 04:33:32 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD593C061745; Tue, 5 Oct 2021 01:31:39 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id na16-20020a17090b4c1000b0019f5bb661f9so1857069pjb.0; Tue, 05 Oct 2021 01:31:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f3iSI2vxyaOsgRcdG56J0IbiqxH5KKykuTvfOZPZCoc=; b=UbVsAMrONqPJTfAXlF5O8lh4M6IZjUdj5jgcD008JzWLNmtFeCHsW8givzreHJAwuM qbIXS0y3cAYyW80ttB07CfFxO7knLuOY8OSbn8Ji6Fcv3Uu2AiExBVxmqf3PymGDnFUM wmVpvpj8k2Hokkjz/RN+5ak87I/ultBu23ZTFf7HlTdYxnS64Wj1HUKd3SQAiZrFbAR0 yDkRp9FSWuWlvkrjUcZCV2wXYJGEt0LxPApPziVgwZ/qiCKYtof4fsc3lLf/t/WLTTwR s0hw7fZQiCJ3J4oJzE7cvXcwcVfkn3U7TUrl5/N/iSzgpugeEivnx7sqhJfD9xZBBWuE dvvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f3iSI2vxyaOsgRcdG56J0IbiqxH5KKykuTvfOZPZCoc=; b=nBlvg1YKA64nyFvFXVCAJQbhDNavU5wrXGYeTm8KYl0lqRHSjBoUcogO/ZXvCMc7Eo m0Ivm5UQgJfQZ2aDKp85aKhvQ+UmosA/fLc8pU37rLT7f/kqSP9HEwSRAxIeo43GDPbg xe5tanfwLaAMol0mZz4gwZlV9VS27Sak70R2wLIM1hxZhOAT0eTgRrALCznIvs/aM3Jn 2TcJMKLDaUfLAGS74tz8+jKWJKsbydt6FyyCht+koGOCNqxMmtzI1CQyN09RMMrLqe40 +3wSshT/O6Rjm8w9i1/h24tPwx8f21rIzL9+TdpROk0Iv/LuAGBg5q5agjigXIwTDQca QtnA== X-Gm-Message-State: AOAM530yegZJEE3dMZj+UDOFsoTkov24ivj/d6c58v947/eKYyCOnDYB IYS3ljDbPKHxGdrNUiLcqQk= X-Google-Smtp-Source: ABdhPJxFmgfBtla3FXGEoqCkSVFjsKdIbKEuMaFlbgBkvejPB4bCdqJaxeDYMac4ELRFRe7kD5HzfA== X-Received: by 2002:a17:902:fe82:b0:13e:7271:92dc with SMTP id x2-20020a170902fe8200b0013e727192dcmr4193279plm.0.1633422698779; Tue, 05 Oct 2021 01:31:38 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:31:38 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 10/16] MIPS: eBPF: add core support for 32/64-bit systems Date: Tue, 5 Oct 2021 01:26:54 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Update register definitions and flags for both 32/64-bit operation. Add a common register lookup table, modifying ebpf_to_mips_reg() to use this, and update enum and literals for ebpf_to_mips_reg() to be more consistent and less confusing. Add is64bit() and isbigend() common helper functions. On MIPS32, BPF registers are held in register pairs defined by the base register. Word-size and endian-aware helper macros select 32-bit registers from a pair and generate 32-bit word memory storage offsets. The BPF TCC is stored to the stack due to register pressure. Update bpf_int_jit_compile() to properly enable BPF2BPF calls, by adding support for the extra pass needed to fix up function target addresses. Also provide bpf line info by calling bpf_prog_fill_jited_linfo(). Modify build_int_prologue() and build_int_epilogue() to handle MIPS32 registers and any adjustments needed during program entry/exit/transfer when transitioning between the native N64/O32 ABI and the BPF 64-bit ABI. Also ensure ABI-consistent stack alignment and use the verifier-provided stack depth during setup, saving considerable stack space. Update emit_const_to_reg() to work across MIPS64 and MIPS32 systems and optimize gen_imm_to_reg() to only set the lower halfword if needed. Rework emit_bpf_tail_call() to also support MIPS32 usage and add common helpers, emit_bpf_call() and emit_push_args(), handling TCC and ABI variations on MIPS32/MIPS64. Add tail_call_present() and update tailcall handling to support mixing BPF2BPF subprograms and tailcalls. Add sign and zero-extension helpers usable with verifier zext insertion, gen_zext_insn() and gen_sext_insn(). Add common functions emit_caller_save() and emit_caller_restore(), which push and pop all caller-saved BPF registers to the stack, for use with JIT-internal kernel calls such as those needed for BPF insns unsupported by native system ISA opcodes. Since these calls would be hidden from any BPF C compiler, which would normally spill needed registers during a call, the JIT must handle save/restore itself. Adopt a dedicated BPF FP (in MIPS_R_S8), and relax FP usage within insns. This reduces ad-hoc code doing $sp manipulation with temp registers, and allows wider usage of BPF FP for comparison and arithmetic. For example, the following tests from test_verifier are now jited but not previously: 939/p store PTR_TO_STACK in R10 to array map using BPF_B 981/p unpriv: cmp pointer with pointer 984/p unpriv: indirectly pass pointer on stack to helper function 985/p unpriv: mangle pointer on stack 1 986/p unpriv: mangle pointer on stack 2 1001/p unpriv: partial copy of pointer 1097/p xadd/w check whether src/dst got mangled, 1 1098/p xadd/w check whether src/dst got mangled, 2 Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 1008 +++++++++++++++++++++++++++----------- 1 file changed, 729 insertions(+), 279 deletions(-) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 7fbd4e371c80..7d8ed8bb19ab 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1,11 +1,13 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * Just-In-Time compiler for eBPF filters on MIPS - * - * Copyright (c) 2017 Cavium, Inc. + * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 + * Copyright (c) 2021 Tony Ambardar * * Based on code from: * + * Copyright (c) 2017 Cavium, Inc. + * Author: David Daney + * * Copyright (c) 2014 Imagination Technologies Ltd. * Author: Markos Chandras */ @@ -22,31 +24,42 @@ #include #include -/* Registers used by JIT */ +/* Registers used by JIT: (MIPS32) (MIPS64) */ #define MIPS_R_ZERO 0 #define MIPS_R_AT 1 -#define MIPS_R_V0 2 /* BPF_R0 */ -#define MIPS_R_V1 3 -#define MIPS_R_A0 4 /* BPF_R1 */ -#define MIPS_R_A1 5 /* BPF_R2 */ -#define MIPS_R_A2 6 /* BPF_R3 */ -#define MIPS_R_A3 7 /* BPF_R4 */ -#define MIPS_R_A4 8 /* BPF_R5 */ -#define MIPS_R_T4 12 /* BPF_AX */ -#define MIPS_R_T5 13 -#define MIPS_R_T6 14 -#define MIPS_R_T7 15 -#define MIPS_R_S0 16 /* BPF_R6 */ -#define MIPS_R_S1 17 /* BPF_R7 */ -#define MIPS_R_S2 18 /* BPF_R8 */ -#define MIPS_R_S3 19 /* BPF_R9 */ -#define MIPS_R_S4 20 /* BPF_TCC */ -#define MIPS_R_S5 21 -#define MIPS_R_S6 22 -#define MIPS_R_S7 23 -#define MIPS_R_T8 24 -#define MIPS_R_T9 25 +#define MIPS_R_V0 2 /* BPF_R0 BPF_R0 */ +#define MIPS_R_V1 3 /* BPF_R0 BPF_TCC */ +#define MIPS_R_A0 4 /* BPF_R1 BPF_R1 */ +#define MIPS_R_A1 5 /* BPF_R1 BPF_R2 */ +#define MIPS_R_A2 6 /* BPF_R2 BPF_R3 */ +#define MIPS_R_A3 7 /* BPF_R2 BPF_R4 */ + +/* MIPS64 replaces T0-T3 scratch regs with extra arguments A4-A7. */ +#ifdef CONFIG_64BIT +# define MIPS_R_A4 8 /* (n/a) BPF_R5 */ +#else +# define MIPS_R_T0 8 /* BPF_R3 (n/a) */ +# define MIPS_R_T1 9 /* BPF_R3 (n/a) */ +# define MIPS_R_T2 10 /* BPF_R4 (n/a) */ +# define MIPS_R_T3 11 /* BPF_R4 (n/a) */ +#endif + +#define MIPS_R_T4 12 /* BPF_R5 BPF_AX */ +#define MIPS_R_T5 13 /* BPF_R5 (free) */ +#define MIPS_R_T6 14 /* BPF_AX (used) */ +#define MIPS_R_T7 15 /* BPF_AX (free) */ +#define MIPS_R_S0 16 /* BPF_R6 BPF_R6 */ +#define MIPS_R_S1 17 /* BPF_R6 BPF_R7 */ +#define MIPS_R_S2 18 /* BPF_R7 BPF_R8 */ +#define MIPS_R_S3 19 /* BPF_R7 BPF_R9 */ +#define MIPS_R_S4 20 /* BPF_R8 BPF_TCC */ +#define MIPS_R_S5 21 /* BPF_R8 (free) */ +#define MIPS_R_S6 22 /* BPF_R9 (free) */ +#define MIPS_R_S7 23 /* BPF_R9 (free) */ +#define MIPS_R_T8 24 /* (used) (used) */ +#define MIPS_R_T9 25 /* (used) (used) */ #define MIPS_R_SP 29 +#define MIPS_R_S8 30 /* BPF_R10 BPF_R10 */ #define MIPS_R_RA 31 /* eBPF flags */ @@ -55,10 +68,117 @@ #define EBPF_SAVE_S2 BIT(2) #define EBPF_SAVE_S3 BIT(3) #define EBPF_SAVE_S4 BIT(4) -#define EBPF_SAVE_RA BIT(5) -#define EBPF_SEEN_FP BIT(6) -#define EBPF_SEEN_TC BIT(7) -#define EBPF_TCC_IN_V1 BIT(8) +#define EBPF_SAVE_S5 BIT(5) +#define EBPF_SAVE_S6 BIT(6) +#define EBPF_SAVE_S7 BIT(7) +#define EBPF_SAVE_S8 BIT(8) +#define EBPF_SAVE_RA BIT(9) +#define EBPF_SEEN_FP BIT(10) +#define EBPF_SEEN_TC BIT(11) +#define EBPF_TCC_IN_RUN BIT(12) + +/* + * Extra JIT registers dedicated to holding TCC during runtime or saving + * across calls. + */ +enum { + JIT_RUN_TCC = MAX_BPF_JIT_REG, + JIT_SAV_TCC +}; +/* Temporary register for passing TCC if nothing dedicated. */ +#define TEMP_PASS_TCC MIPS_R_T8 + +/* + * Word-size and endianness-aware helpers for building MIPS32 vs MIPS64 + * tables and selecting 32-bit subregisters from a register pair base. + * Simplify use by emulating MIPS_R_SP and MIPS_R_ZERO as register pairs + * and adding HI/LO word memory offsets. + */ +#ifdef CONFIG_64BIT +# define HI(reg) (reg) +# define LO(reg) (reg) +# define OFFHI(mem) (mem) +# define OFFLO(mem) (mem) +#else /* CONFIG_32BIT */ +# ifdef __BIG_ENDIAN +# define HI(reg) ((reg) == MIPS_R_SP ? MIPS_R_ZERO : \ + (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ + (reg)) +# define LO(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ + (reg) == MIPS_R_SP ? (reg) : \ + (reg) == MIPS_R_S8 ? (reg) : \ + (reg) + 1) +# define OFFHI(mem) (mem) +# define OFFLO(mem) ((mem) + sizeof(long)) +# else /* __LITTLE_ENDIAN */ +# define HI(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ + (reg) == MIPS_R_SP ? MIPS_R_ZERO : \ + (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ + (reg) + 1) +# define LO(reg) (reg) +# define OFFHI(mem) ((mem) + sizeof(long)) +# define OFFLO(mem) (mem) +# endif +#endif + +#ifdef CONFIG_64BIT +# define M(expr32, expr64) (expr64) +#else +# define M(expr32, expr64) (expr32) +#endif +const struct { + /* Register or pair base */ + int reg; + /* Register flags */ + u32 flags; + /* Usage table: (MIPS32) (MIPS64) */ +} bpf2mips[] = { + /* Return value from in-kernel function, and exit value from eBPF. */ + [BPF_REG_0] = {M(MIPS_R_V0, MIPS_R_V0)}, + /* Arguments from eBPF program to in-kernel/BPF functions. */ + [BPF_REG_1] = {M(MIPS_R_A0, MIPS_R_A0)}, + [BPF_REG_2] = {M(MIPS_R_A2, MIPS_R_A1)}, + [BPF_REG_3] = {M(MIPS_R_T0, MIPS_R_A2)}, + [BPF_REG_4] = {M(MIPS_R_T2, MIPS_R_A3)}, + [BPF_REG_5] = {M(MIPS_R_T4, MIPS_R_A4)}, + /* Callee-saved registers preserved by in-kernel/BPF functions. */ + [BPF_REG_6] = {M(MIPS_R_S0, MIPS_R_S0), + M(EBPF_SAVE_S0|EBPF_SAVE_S1, EBPF_SAVE_S0)}, + [BPF_REG_7] = {M(MIPS_R_S2, MIPS_R_S1), + M(EBPF_SAVE_S2|EBPF_SAVE_S3, EBPF_SAVE_S1)}, + [BPF_REG_8] = {M(MIPS_R_S4, MIPS_R_S2), + M(EBPF_SAVE_S4|EBPF_SAVE_S5, EBPF_SAVE_S2)}, + [BPF_REG_9] = {M(MIPS_R_S6, MIPS_R_S3), + M(EBPF_SAVE_S6|EBPF_SAVE_S7, EBPF_SAVE_S3)}, + [BPF_REG_10] = {M(MIPS_R_S8, MIPS_R_S8), + M(EBPF_SAVE_S8|EBPF_SEEN_FP, EBPF_SAVE_S8|EBPF_SEEN_FP)}, + /* Internal register for rewriting insns during JIT blinding. */ + [BPF_REG_AX] = {M(MIPS_R_T6, MIPS_R_T4)}, + /* + * Internal registers for TCC runtime holding and saving during + * calls. A zero save register indicates using scratch space on + * the stack for storage during calls. A zero hold register means + * no dedicated register holds TCC during runtime (but a temp reg + * still passes TCC to tailcall or bpf2bpf call). + */ + [JIT_RUN_TCC] = {M(0, MIPS_R_V1)}, + [JIT_SAV_TCC] = {M(0, MIPS_R_S4), + M(0, EBPF_SAVE_S4)} +}; +#undef M + +static inline bool is64bit(void) +{ + return IS_ENABLED(CONFIG_64BIT); +} + +static inline bool isbigend(void) +{ + return IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); +} + +/* Stack region alignment under N64 and O32 ABIs */ +#define STACK_ALIGN (2 * sizeof(long)) /* * For the mips64 ISA, we need to track the value range or type for @@ -89,17 +209,21 @@ enum reg_val_type { /** * struct jit_ctx - JIT context - * @skf: The sk_filter + * @prog: The program * @stack_size: eBPF stack size + * @bpf_stack_off: eBPF FP offset + * @prolog_skip: Prologue insns to skip by BPF caller * @idx: Instruction index * @flags: JIT flags * @offsets: Instruction offsets - * @target: Memory location for the compiled filter - * @reg_val_types Packed enum reg_val_type for each register. + * @target: Memory location for compiled instructions + * @reg_val_types: Packed enum reg_val_type for each register */ struct jit_ctx { - const struct bpf_prog *skf; + const struct bpf_prog *prog; int stack_size; + int bpf_stack_off; + int prolog_skip; u32 idx; u32 flags; u32 *offsets; @@ -177,132 +301,192 @@ static u32 b_imm(unsigned int tgt, struct jit_ctx *ctx) (ctx->idx * 4) - 4; } -enum which_ebpf_reg { - src_reg, - src_reg_no_fp, - dst_reg, - dst_reg_fp_ok +/* Sign-extend dst register or HI 32-bit reg of pair. */ +static inline void gen_sext_insn(int dst, struct jit_ctx *ctx) +{ + if (is64bit()) + emit_instr(ctx, sll, dst, dst, 0); + else + emit_instr(ctx, sra, HI(dst), LO(dst), 31); +} + +/* + * Zero-extend dst register or HI 32-bit reg of pair, if either forced + * or the BPF verifier does not insert its own zext insns. + */ +static inline void gen_zext_insn(int dst, bool force, struct jit_ctx *ctx) +{ + if (!ctx->prog->aux->verifier_zext || force) { + if (is64bit()) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + else + emit_instr(ctx, and, HI(dst), MIPS_R_ZERO, MIPS_R_ZERO); + } +} + +static inline bool tail_call_present(struct jit_ctx *ctx) +{ + return ctx->flags & EBPF_SEEN_TC || ctx->prog->aux->tail_call_reachable; +} + +enum reg_usage { + REG_SRC_FP_OK, + REG_SRC_NO_FP, + REG_DST_FP_OK, + REG_DST_NO_FP }; /* * For eBPF, the register mapping naturally falls out of the - * requirements of eBPF and the MIPS n64 ABI. We don't maintain a - * separate frame pointer, so BPF_REG_10 relative accesses are - * adjusted to be $sp relative. + * requirements of eBPF and the MIPS N64/O32 ABIs. We also maintain + * a separate frame pointer, setting BPF_REG_10 relative to $sp. */ static int ebpf_to_mips_reg(struct jit_ctx *ctx, const struct bpf_insn *insn, - enum which_ebpf_reg w) + enum reg_usage u) { - int ebpf_reg = (w == src_reg || w == src_reg_no_fp) ? + int ebpf_reg = (u == REG_SRC_FP_OK || u == REG_SRC_NO_FP) ? insn->src_reg : insn->dst_reg; switch (ebpf_reg) { case BPF_REG_0: - return MIPS_R_V0; case BPF_REG_1: - return MIPS_R_A0; case BPF_REG_2: - return MIPS_R_A1; case BPF_REG_3: - return MIPS_R_A2; case BPF_REG_4: - return MIPS_R_A3; case BPF_REG_5: - return MIPS_R_A4; case BPF_REG_6: - ctx->flags |= EBPF_SAVE_S0; - return MIPS_R_S0; case BPF_REG_7: - ctx->flags |= EBPF_SAVE_S1; - return MIPS_R_S1; case BPF_REG_8: - ctx->flags |= EBPF_SAVE_S2; - return MIPS_R_S2; case BPF_REG_9: - ctx->flags |= EBPF_SAVE_S3; - return MIPS_R_S3; + case BPF_REG_AX: + ctx->flags |= bpf2mips[ebpf_reg].flags; + return bpf2mips[ebpf_reg].reg; case BPF_REG_10: - if (w == dst_reg || w == src_reg_no_fp) + if (u == REG_DST_NO_FP || u == REG_SRC_NO_FP) goto bad_reg; - ctx->flags |= EBPF_SEEN_FP; - /* - * Needs special handling, return something that - * cannot be clobbered just in case. - */ - return MIPS_R_ZERO; - case BPF_REG_AX: - return MIPS_R_T4; + ctx->flags |= bpf2mips[ebpf_reg].flags; + return bpf2mips[ebpf_reg].reg; default: bad_reg: WARN(1, "Illegal bpf reg: %d\n", ebpf_reg); return -EINVAL; } } + /* * eBPF stack frame will be something like: * * Entry $sp ------> +--------------------------------+ * | $ra (optional) | * +--------------------------------+ - * | $s0 (optional) | + * | $s8 (optional) | * +--------------------------------+ - * | $s1 (optional) | + * | $s7 (optional) | * +--------------------------------+ - * | $s2 (optional) | + * | $s6 (optional) | * +--------------------------------+ - * | $s3 (optional) | + * | $s5 (optional) | * +--------------------------------+ * | $s4 (optional) | * +--------------------------------+ - * | tmp-storage (if $ra saved) | - * $sp + tmp_offset --> +--------------------------------+ <--BPF_REG_10 + * | $s3 (optional) | + * +--------------------------------+ + * | $s2 (optional) | + * +--------------------------------+ + * | $s1 (optional) | + * +--------------------------------+ + * | $s0 (optional) | + * +--------------------------------+ + * | tmp-storage (optional) | + * $sp + bpf_stack_off->+--------------------------------+ <--BPF_REG_10 * | BPF_REG_10 relative storage | * | MAX_BPF_STACK (optional) | * | . | * | . | * | . | - * $sp --------> +--------------------------------+ + * $sp ------> +--------------------------------+ * * If BPF_REG_10 is never referenced, then the MAX_BPF_STACK sized * area is not allocated. */ -static int gen_int_prologue(struct jit_ctx *ctx) +static int build_int_prologue(struct jit_ctx *ctx) { + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + const struct bpf_prog *prog = ctx->prog; + int r10 = bpf2mips[BPF_REG_10].reg; + int r1 = bpf2mips[BPF_REG_1].reg; int stack_adjust = 0; int store_offset; int locals_size; + int start_idx; if (ctx->flags & EBPF_SAVE_RA) - /* - * If RA we are doing a function call and may need - * extra 8-byte tmp area. - */ - stack_adjust += 2 * sizeof(long); - if (ctx->flags & EBPF_SAVE_S0) stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S1) + if (ctx->flags & EBPF_SAVE_S8) stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S2) + if (ctx->flags & EBPF_SAVE_S7) stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S3) + if (ctx->flags & EBPF_SAVE_S6) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S5) stack_adjust += sizeof(long); if (ctx->flags & EBPF_SAVE_S4) stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S3) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S2) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S1) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S0) + stack_adjust += sizeof(long); + if (tail_call_present(ctx) && + !(ctx->flags & EBPF_TCC_IN_RUN) && !tcc_sav) + /* Allocate scratch space for holding TCC if needed. */ + stack_adjust += sizeof(long); + + stack_adjust = ALIGN(stack_adjust, STACK_ALIGN); - BUILD_BUG_ON(MAX_BPF_STACK & 7); - locals_size = (ctx->flags & EBPF_SEEN_FP) ? MAX_BPF_STACK : 0; + locals_size = (ctx->flags & EBPF_SEEN_FP) ? prog->aux->stack_depth : 0; + locals_size = ALIGN(locals_size, STACK_ALIGN); stack_adjust += locals_size; ctx->stack_size = stack_adjust; + ctx->bpf_stack_off = locals_size; /* - * First instruction initializes the tail call count (TCC). - * On tail call we skip this instruction, and the TCC is - * passed in $v1 from the caller. + * First instruction initializes the tail call count (TCC) and + * assumes a call from kernel using the native ABI. Calls made + * using the BPF ABI (bpf2bpf or tail call) will skip this insn + * and pass the TCC via register. */ - emit_instr(ctx, addiu, MIPS_R_V1, MIPS_R_ZERO, MAX_TAIL_CALL_CNT); + start_idx = ctx->idx; + emit_instr(ctx, addiu, tcc_run, MIPS_R_ZERO, MAX_TAIL_CALL_CNT); + + /* + * When called from kernel under O32 ABI we must set up BPF R1 + * context, since BPF R1 is an endian-order regster pair ($a0:$a1 + * or $a1:$a0) but context is always passed in $a0 as a 32-bit + * pointer. As above, bpf2bpf and tail calls will skip these insns + * since all registers are correctly set up already. + */ + if (!is64bit()) { + if (isbigend()) + emit_instr(ctx, move, LO(r1), MIPS_R_A0); + /* Sanitize upper 32-bit reg */ + gen_zext_insn(r1, true, ctx); + } + /* + * Calls using BPF ABI (bpf2bpf and tail calls) will skip TCC + * initialization and R1 context fixup needed by kernel calls. + */ + ctx->prolog_skip = (ctx->idx - start_idx) * 4; + if (stack_adjust) emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adjust); @@ -316,24 +500,24 @@ static int gen_int_prologue(struct jit_ctx *ctx) MIPS_R_RA, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S0) { + if (ctx->flags & EBPF_SAVE_S8) { emit_instr_long(ctx, sd, sw, - MIPS_R_S0, store_offset, MIPS_R_SP); + MIPS_R_S8, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S1) { + if (ctx->flags & EBPF_SAVE_S7) { emit_instr_long(ctx, sd, sw, - MIPS_R_S1, store_offset, MIPS_R_SP); + MIPS_R_S7, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S2) { + if (ctx->flags & EBPF_SAVE_S6) { emit_instr_long(ctx, sd, sw, - MIPS_R_S2, store_offset, MIPS_R_SP); + MIPS_R_S6, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S3) { + if (ctx->flags & EBPF_SAVE_S5) { emit_instr_long(ctx, sd, sw, - MIPS_R_S3, store_offset, MIPS_R_SP); + MIPS_R_S5, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } if (ctx->flags & EBPF_SAVE_S4) { @@ -341,27 +525,95 @@ static int gen_int_prologue(struct jit_ctx *ctx) MIPS_R_S4, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } + if (ctx->flags & EBPF_SAVE_S3) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S3, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S2) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S2, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S1) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S1, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S0) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S0, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + + /* Store TCC in backup register or stack scratch space if indicated. */ + if (tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { + if (tcc_sav) + emit_instr(ctx, move, tcc_sav, tcc_run); + else + emit_instr_long(ctx, sd, sw, + tcc_run, ctx->bpf_stack_off, MIPS_R_SP); + } - if ((ctx->flags & EBPF_SEEN_TC) && !(ctx->flags & EBPF_TCC_IN_V1)) - emit_instr_long(ctx, daddu, addu, - MIPS_R_S4, MIPS_R_V1, MIPS_R_ZERO); + /* Prepare BPF FP as single-reg ptr, emulate upper 32-bits as needed.*/ + if (ctx->flags & EBPF_SEEN_FP) + emit_instr_long(ctx, daddiu, addiu, r10, + MIPS_R_SP, ctx->bpf_stack_off); return 0; } static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg) { - const struct bpf_prog *prog = ctx->skf; + const struct bpf_prog *prog = ctx->prog; int stack_adjust = ctx->stack_size; int store_offset = stack_adjust - sizeof(long); + int ax = bpf2mips[BPF_REG_AX].reg; + int r0 = bpf2mips[BPF_REG_0].reg; enum reg_val_type td; - int r0 = MIPS_R_V0; - if (dest_reg == MIPS_R_RA) { - /* Don't let zero extended value escape. */ - td = get_reg_val_type(ctx, prog->len, BPF_REG_0); - if (td == REG_64BIT) - emit_instr(ctx, sll, r0, r0, 0); + /* + * As in prologue code, we default to assuming exit to the kernel. + * Returns to the kernel follow the N64 or O32 ABI. For N64, the + * BPF R0 return value may need to be sign-extended, while O32 may + * need fixup of BPF R0 to place the 32-bit return value in MIPS V0. + * + * Returns to BPF2BPF callers consistently use the BPF 64-bit ABI, + * so register usage and mapping between JIT and OS is unchanged. + * Accommodate by saving unmodified R0 register data to allow a + * BPF caller to restore R0 after we return. + */ + if (dest_reg == MIPS_R_RA) { /* kernel or bpf2bpf function return */ + if (is64bit()) { + /* + * Backup BPF R0 to AX, allowing the caller to + * restore it in case this is a BPF2BPF rather + * than a kernel return. + */ + emit_instr(ctx, move, ax, r0); + /* + * Don't let zero-extended R0 value escape to + * kernel on return, so sign-extend if needed. + */ + td = get_reg_val_type(ctx, prog->len, BPF_REG_0); + if (td == REG_64BIT) + gen_sext_insn(r0, ctx); + } else if (isbigend()) { /* and 32-bit */ + /* + * Backup high 32-bit register of BPF R0 to AX, + * since it occupies MIPS_R_V0 which needs to be + * clobbered for a kernel return. + */ + emit_instr(ctx, move, HI(ax), HI(r0)); + /* + * O32 ABI specifies 32-bit return value always + * placed in MIPS_R_V0 regardless of the native + * endianness. This would be in the wrong position + * in a BPF R0 reg pair on big-endian systems, so + * we must relocate. + */ + emit_instr(ctx, move, MIPS_R_V0, LO(r0)); + } } if (ctx->flags & EBPF_SAVE_RA) { @@ -369,24 +621,24 @@ static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg) MIPS_R_RA, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S0) { + if (ctx->flags & EBPF_SAVE_S8) { emit_instr_long(ctx, ld, lw, - MIPS_R_S0, store_offset, MIPS_R_SP); + MIPS_R_S8, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S1) { + if (ctx->flags & EBPF_SAVE_S7) { emit_instr_long(ctx, ld, lw, - MIPS_R_S1, store_offset, MIPS_R_SP); + MIPS_R_S7, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S2) { + if (ctx->flags & EBPF_SAVE_S6) { emit_instr_long(ctx, ld, lw, - MIPS_R_S2, store_offset, MIPS_R_SP); + MIPS_R_S6, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } - if (ctx->flags & EBPF_SAVE_S3) { + if (ctx->flags & EBPF_SAVE_S5) { emit_instr_long(ctx, ld, lw, - MIPS_R_S3, store_offset, MIPS_R_SP); + MIPS_R_S5, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } if (ctx->flags & EBPF_SAVE_S4) { @@ -394,8 +646,29 @@ static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg) MIPS_R_S4, store_offset, MIPS_R_SP); store_offset -= sizeof(long); } + if (ctx->flags & EBPF_SAVE_S3) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S3, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S2) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S2, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S1) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S1, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S0) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S0, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } emit_instr(ctx, jr, dest_reg); + /* Delay slot */ if (stack_adjust) emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, stack_adjust); @@ -415,7 +688,9 @@ static void gen_imm_to_reg(const struct bpf_insn *insn, int reg, int upper = insn->imm - lower; emit_instr(ctx, lui, reg, upper >> 16); - emit_instr(ctx, addiu, reg, reg, lower); + /* lui already clears lower halfword */ + if (lower) + emit_instr(ctx, addiu, reg, reg, lower); } } @@ -423,7 +698,7 @@ static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, int idx) { int upper_bound, lower_bound; - int dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + int dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; @@ -564,12 +839,12 @@ static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return 0; } -static void emit_const_to_reg(struct jit_ctx *ctx, int dst, u64 value) +static void emit_const_to_reg(struct jit_ctx *ctx, int dst, unsigned long value) { - if (value >= 0xffffffffffff8000ull || value < 0x8000ull) { - emit_instr(ctx, daddiu, dst, MIPS_R_ZERO, (int)value); - } else if (value >= 0xffffffff80000000ull || - (value < 0x80000000 && value > 0xffff)) { + if (value >= S16_MIN || value <= S16_MAX) { + emit_instr_long(ctx, daddiu, addiu, dst, MIPS_R_ZERO, (int)value); + } else if (value >= S32_MIN || + (value <= S32_MAX && value > U16_MAX)) { emit_instr(ctx, lui, dst, (s32)(s16)(value >> 16)); emit_instr(ctx, ori, dst, dst, (unsigned int)(value & 0xffff)); } else { @@ -601,54 +876,167 @@ static void emit_const_to_reg(struct jit_ctx *ctx, int dst, u64 value) } } +/* + * Push BPF regs R3-R5 to the stack, skipping BPF regs R1-R2 which are + * passed via MIPS register pairs in $a0-$a3. Register order within pairs + * and the memory storage order are identical i.e. endian native. + */ +static void emit_push_args(struct jit_ctx *ctx) +{ + int store_offset = 2 * sizeof(u64); /* Skip R1-R2 in $a0-$a3 */ + int bpf, reg; + + for (bpf = BPF_REG_3; bpf <= BPF_REG_5; bpf++) { + reg = bpf2mips[bpf].reg; + + emit_instr(ctx, sw, LO(reg), OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, sw, HI(reg), OFFHI(store_offset), MIPS_R_SP); + store_offset += sizeof(u64); + } +} + +/* + * Common helper for BPF_CALL insn, handling TCC and ABI variations. + * Kernel calls under O32 ABI require arguments passed on the stack, + * while BPF2BPF calls need the TCC passed via register as expected + * by the subprog's prologue. + * + * Under MIPS32 O32 ABI calling convention, u64 BPF regs R1-R2 are passed + * via reg pairs in $a0-$a3, while BPF regs R3-R5 are passed via the stack. + * Stack space is still reserved for $a0-$a3, and the whole area aligned. + */ +#define ARGS_SIZE (5 * sizeof(u64)) + +void emit_bpf_call(struct jit_ctx *ctx, const struct bpf_insn *insn) +{ + int stack_adjust = ALIGN(ARGS_SIZE, STACK_ALIGN); + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + int ax = bpf2mips[BPF_REG_AX].reg; + int r0 = bpf2mips[BPF_REG_0].reg; + long func_addr; + + ctx->flags |= EBPF_SAVE_RA; + + /* Ensure TCC passed into BPF subprog */ + if ((insn->src_reg == BPF_PSEUDO_CALL) && + tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { + /* Set TCC from reg or stack */ + if (tcc_sav) + emit_instr(ctx, move, tcc_run, tcc_sav); + else + emit_instr_long(ctx, ld, lw, tcc_run, + ctx->bpf_stack_off, MIPS_R_SP); + } + + /* Push O32 stack args for kernel call */ + if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) { + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adjust); + emit_push_args(ctx); + } + + func_addr = (long)__bpf_call_base + insn->imm; + + /* Skip TCC init and R1 register fixup with BPF ABI. */ + if (insn->src_reg == BPF_PSEUDO_CALL) + func_addr += ctx->prolog_skip; + + emit_const_to_reg(ctx, MIPS_R_T9, func_addr); + emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); + /* Delay slot */ + emit_instr(ctx, nop); + + /* Restore stack */ + if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, stack_adjust); + + /* + * Assuming a kernel return, a MIPS64 function epilogue may + * sign-extend R0, while MIPS32BE mangles the R0 register pair. + * Undo both for a bpf2bpf call return. + */ + if (insn->src_reg == BPF_PSEUDO_CALL) { + /* Restore BPF R0 from AX */ + if (is64bit()) { + emit_instr(ctx, move, r0, ax); + } else if (isbigend()) { /* and 32-bit */ + emit_instr(ctx, move, LO(r0), MIPS_R_V0); + emit_instr(ctx, move, HI(r0), HI(ax)); + } + } +} + +/* + * Tail call helper arguments passed via BPF ABI as u64 parameters. On + * MIPS64 N64 ABI systems these are native regs, while on MIPS32 O32 ABI + * systems these are reg pairs: + * + * R1 -> &ctx + * R2 -> &array + * R3 -> index + */ static int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx) { + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + int r2 = bpf2mips[BPF_REG_2].reg; + int r3 = bpf2mips[BPF_REG_3].reg; int off, b_off; - int tcc_reg; + int tcc; ctx->flags |= EBPF_SEEN_TC; /* * if (index >= array->map.max_entries) * goto out; */ - /* Mask index as 32-bit */ - emit_instr(ctx, dinsu, MIPS_R_A2, MIPS_R_ZERO, 32, 32); + if (is64bit()) + /* Mask index as 32-bit */ + gen_zext_insn(r3, true, ctx); off = offsetof(struct bpf_array, map.max_entries); - emit_instr(ctx, lwu, MIPS_R_T5, off, MIPS_R_A1); - emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_T5, MIPS_R_A2); + emit_instr_long(ctx, lwu, lw, MIPS_R_AT, off, LO(r2)); + emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_AT, LO(r3)); b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, bne, MIPS_R_AT, MIPS_R_ZERO, b_off); + emit_instr(ctx, bnez, MIPS_R_AT, b_off); /* * if (TCC-- < 0) * goto out; */ /* Delay slot */ - tcc_reg = (ctx->flags & EBPF_TCC_IN_V1) ? MIPS_R_V1 : MIPS_R_S4; - emit_instr(ctx, daddiu, MIPS_R_T5, tcc_reg, -1); + tcc = (ctx->flags & EBPF_TCC_IN_RUN) ? tcc_run : tcc_sav; + /* Get TCC from reg or stack */ + if (tcc) + emit_instr(ctx, move, MIPS_R_T8, tcc); + else + emit_instr_long(ctx, ld, lw, MIPS_R_T8, + ctx->bpf_stack_off, MIPS_R_SP); b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, bltz, tcc_reg, b_off); + emit_instr(ctx, bltz, MIPS_R_T8, b_off); /* * prog = array->ptrs[index]; * if (prog == NULL) * goto out; */ /* Delay slot */ - emit_instr(ctx, dsll, MIPS_R_T8, MIPS_R_A2, 3); - emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, MIPS_R_A1); + emit_instr_long(ctx, dsll, sll, MIPS_R_AT, LO(r3), ilog2(sizeof(long))); + emit_instr_long(ctx, daddu, addu, MIPS_R_AT, MIPS_R_AT, LO(r2)); off = offsetof(struct bpf_array, ptrs); - emit_instr(ctx, ld, MIPS_R_AT, off, MIPS_R_T8); + emit_instr_long(ctx, ld, lw, MIPS_R_AT, off, MIPS_R_AT); b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, beq, MIPS_R_AT, MIPS_R_ZERO, b_off); + emit_instr(ctx, beqz, MIPS_R_AT, b_off); /* Delay slot */ emit_instr(ctx, nop); - /* goto *(prog->bpf_func + 4); */ + /* goto *(prog->bpf_func + skip); */ off = offsetof(struct bpf_prog, bpf_func); - emit_instr(ctx, ld, MIPS_R_T9, off, MIPS_R_AT); - /* All systems are go... propagate TCC */ - emit_instr(ctx, daddu, MIPS_R_V1, MIPS_R_T5, MIPS_R_ZERO); - /* Skip first instruction (TCC initialization) */ - emit_instr(ctx, daddiu, MIPS_R_T9, MIPS_R_T9, 4); + emit_instr_long(ctx, ld, lw, MIPS_R_T9, off, MIPS_R_AT); + /* All systems are go... decrement and propagate TCC */ + emit_instr_long(ctx, daddiu, addiu, tcc_run, MIPS_R_T8, -1); + /* Skip first instructions (TCC init and R1 fixup) */ + emit_instr_long(ctx, daddiu, addiu, MIPS_R_T9, MIPS_R_T9, ctx->prolog_skip); return build_int_epilogue(ctx, MIPS_R_T9); } @@ -696,7 +1084,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return r; break; case BPF_ALU64 | BPF_MUL | BPF_K: /* ALU64_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) @@ -712,7 +1100,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; case BPF_ALU64 | BPF_NEG | BPF_K: /* ALU64_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) @@ -720,7 +1108,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit_instr(ctx, dsubu, dst, MIPS_R_ZERO, dst); break; case BPF_ALU | BPF_MUL | BPF_K: /* ALU_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -739,7 +1127,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; case BPF_ALU | BPF_NEG | BPF_K: /* ALU_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -753,7 +1141,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ALU | BPF_MOD | BPF_K: /* ALU_IMM */ if (insn->imm == 0) return -EINVAL; - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -784,7 +1172,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ALU64 | BPF_MOD | BPF_K: /* ALU_IMM */ if (insn->imm == 0) return -EINVAL; - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) @@ -821,22 +1209,14 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ALU64 | BPF_LSH | BPF_X: /* ALU64_REG */ case BPF_ALU64 | BPF_RSH | BPF_X: /* ALU64_REG */ case BPF_ALU64 | BPF_ARSH | BPF_X: /* ALU64_REG */ - src = ebpf_to_mips_reg(ctx, insn, src_reg); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (src < 0 || dst < 0) return -EINVAL; if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); did_move = false; - if (insn->src_reg == BPF_REG_10) { - if (bpf_op == BPF_MOV) { - emit_instr(ctx, daddiu, dst, MIPS_R_SP, MAX_BPF_STACK); - did_move = true; - } else { - emit_instr(ctx, daddiu, MIPS_R_AT, MIPS_R_SP, MAX_BPF_STACK); - src = MIPS_R_AT; - } - } else if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { int tmp_reg = MIPS_R_AT; if (bpf_op == BPF_MOV) { @@ -917,8 +1297,8 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ALU | BPF_LSH | BPF_X: /* ALU_REG */ case BPF_ALU | BPF_RSH | BPF_X: /* ALU_REG */ case BPF_ALU | BPF_ARSH | BPF_X: /* ALU_REG */ - src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (src < 0 || dst < 0) return -EINVAL; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -1008,7 +1388,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */ case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */ cmp_eq = (bpf_op == BPF_JEQ); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; if (insn->imm == 0) { @@ -1029,8 +1409,8 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JGT | BPF_X: case BPF_JMP | BPF_JGE | BPF_X: case BPF_JMP | BPF_JSET | BPF_X: - src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (src < 0 || dst < 0) return -EINVAL; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -1160,7 +1540,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JSLT | BPF_K: /* JMP_IMM */ case BPF_JMP | BPF_JSLE | BPF_K: /* JMP_IMM */ cmp_eq = (bpf_op == BPF_JSGE); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; @@ -1235,7 +1615,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JLT | BPF_K: case BPF_JMP | BPF_JLE | BPF_K: cmp_eq = (bpf_op == BPF_JGE); - dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; /* @@ -1258,7 +1638,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, goto jeq_common; case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; @@ -1303,7 +1683,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit_instr(ctx, nop); break; case BPF_LD | BPF_DW | BPF_IMM: - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; t64 = ((u64)(u32)insn->imm) | ((u64)(insn + 1)->imm << 32); @@ -1311,12 +1691,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return 2; /* Double slot insn */ case BPF_JMP | BPF_CALL: - ctx->flags |= EBPF_SAVE_RA; - t64s = (s64)insn->imm + (long)__bpf_call_base; - emit_const_to_reg(ctx, MIPS_R_T9, (u64)t64s); - emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); - /* delay slot */ - emit_instr(ctx, nop); + emit_bpf_call(ctx, insn); break; case BPF_JMP | BPF_TAIL_CALL: @@ -1326,7 +1701,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ALU | BPF_END | BPF_FROM_BE: case BPF_ALU | BPF_END | BPF_FROM_LE: - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); if (dst < 0) return dst; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); @@ -1367,16 +1742,10 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_ST | BPF_H | BPF_MEM: case BPF_ST | BPF_W | BPF_MEM: case BPF_ST | BPF_DW | BPF_MEM: - if (insn->dst_reg == BPF_REG_10) { - ctx->flags |= EBPF_SEEN_FP; - dst = MIPS_R_SP; - mem_off = insn->off + MAX_BPF_STACK; - } else { - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); - if (dst < 0) - return dst; - mem_off = insn->off; - } + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + mem_off = insn->off; gen_imm_to_reg(insn, MIPS_R_AT, ctx); switch (BPF_SIZE(insn->code)) { case BPF_B: @@ -1398,19 +1767,11 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_LDX | BPF_H | BPF_MEM: case BPF_LDX | BPF_W | BPF_MEM: case BPF_LDX | BPF_DW | BPF_MEM: - if (insn->src_reg == BPF_REG_10) { - ctx->flags |= EBPF_SEEN_FP; - src = MIPS_R_SP; - mem_off = insn->off + MAX_BPF_STACK; - } else { - src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp); - if (src < 0) - return src; - mem_off = insn->off; - } - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); - if (dst < 0) - return dst; + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (dst < 0 || src < 0) + return -EINVAL; + mem_off = insn->off; switch (BPF_SIZE(insn->code)) { case BPF_B: emit_instr(ctx, lbu, dst, mem_off, src); @@ -1433,25 +1794,16 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_STX | BPF_DW | BPF_MEM: case BPF_STX | BPF_W | BPF_ATOMIC: case BPF_STX | BPF_DW | BPF_ATOMIC: - if (insn->dst_reg == BPF_REG_10) { - ctx->flags |= EBPF_SEEN_FP; - dst = MIPS_R_SP; - mem_off = insn->off + MAX_BPF_STACK; - } else { - dst = ebpf_to_mips_reg(ctx, insn, dst_reg); - if (dst < 0) - return dst; - mem_off = insn->off; - } - src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp); - if (src < 0) - return src; + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; if (BPF_MODE(insn->code) == BPF_ATOMIC) { if (insn->imm != BPF_ADD) { pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); return -EINVAL; } - /* * If mem_off does not fit within the 9 bit ll/sc * instruction immediate field, use a temp reg. @@ -1530,7 +1882,7 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, static int build_int_body(struct jit_ctx *ctx) { - const struct bpf_prog *prog = ctx->skf; + const struct bpf_prog *prog = ctx->prog; const struct bpf_insn *insn; int i, r; @@ -1572,7 +1924,7 @@ static int build_int_body(struct jit_ctx *ctx) static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, int start_idx, bool follow_taken) { - const struct bpf_prog *prog = ctx->skf; + const struct bpf_prog *prog = ctx->prog; const struct bpf_insn *insn; u64 exit_rvt = initial_rvt; u64 *rvt = ctx->reg_val_types; @@ -1773,7 +2125,7 @@ static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, */ static int reg_val_propagate(struct jit_ctx *ctx) { - const struct bpf_prog *prog = ctx->skf; + const struct bpf_prog *prog = ctx->prog; u64 exit_rvt; int reg; int i; @@ -1832,23 +2184,86 @@ static void jit_fill_hole(void *area, unsigned int size) uasm_i_break(&p, BRK_BUG); /* Increments p */ } +/* + * Save and restore the BPF VM state across a direct kernel call. This + * includes the caller-saved registers used for BPF_REG_0 .. BPF_REG_5 + * and BPF_REG_AX used by the verifier for blinding and other dark arts. + * Restore avoids clobbering bpf_ret, which holds the call return value. + * BPF_REG_6 .. BPF_REG_10 and TCC are already callee-saved or on stack. + */ +static const int bpf_caller_save[] = { + BPF_REG_0, + BPF_REG_1, + BPF_REG_2, + BPF_REG_3, + BPF_REG_4, + BPF_REG_5, + BPF_REG_AX, +}; + +#define CALLER_ENV_SIZE (ARRAY_SIZE(bpf_caller_save) * sizeof(u64)) + +void emit_caller_save(struct jit_ctx *ctx) +{ + int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); + int i, bpf, reg, store_offset; + + emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adj); + + for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { + bpf = bpf_caller_save[i]; + reg = bpf2mips[bpf].reg; + store_offset = i * sizeof(u64); + + if (is64bit()) { + emit_instr(ctx, sd, reg, store_offset, MIPS_R_SP); + } else { + emit_instr(ctx, sw, LO(reg), + OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, sw, HI(reg), + OFFHI(store_offset), MIPS_R_SP); + } + } +} + +void emit_caller_restore(struct jit_ctx *ctx, int bpf_ret) +{ + int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); + int i, bpf, reg, store_offset; + + for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { + bpf = bpf_caller_save[i]; + reg = bpf2mips[bpf].reg; + store_offset = i * sizeof(u64); + if (bpf == bpf_ret) + continue; + + if (is64bit()) { + emit_instr(ctx, ld, reg, store_offset, MIPS_R_SP); + } else { + emit_instr(ctx, lw, LO(reg), + OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, lw, HI(reg), + OFFHI(store_offset), MIPS_R_SP); + } + } + + emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, stack_adj); +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { - struct bpf_prog *orig_prog = prog; - bool tmp_blinded = false; - struct bpf_prog *tmp; + bool tmp_blinded = false, extra_pass = false; + struct bpf_prog *tmp, *orig_prog = prog; struct bpf_binary_header *header = NULL; - struct jit_ctx ctx; - unsigned int image_size; - u8 *image_ptr; + unsigned int image_size, pass = 3; + struct jit_ctx *ctx; if (!prog->jit_requested) - return prog; + return orig_prog; + /* Attempt blinding but fall back to the interpreter on failure. */ tmp = bpf_jit_blind_constants(prog); - /* If blinding was requested and we failed during blinding, - * we must fall back to the interpreter. - */ if (IS_ERR(tmp)) return orig_prog; if (tmp != prog) { @@ -1856,50 +2271,75 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = tmp; } - memset(&ctx, 0, sizeof(ctx)); + ctx = prog->aux->jit_data; + if (!ctx) { + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) { + prog = orig_prog; + goto out; + } + } - preempt_disable(); - switch (current_cpu_type()) { - case CPU_CAVIUM_OCTEON: - case CPU_CAVIUM_OCTEON_PLUS: - case CPU_CAVIUM_OCTEON2: - case CPU_CAVIUM_OCTEON3: - ctx.use_bbit_insns = 1; - break; - default: - ctx.use_bbit_insns = 0; + /* + * Assume extra pass needed for patching addresses if previous + * ctx exists in saved jit_data, so skip to code generation. + */ + if (ctx->offsets) { + extra_pass = true; + pass++; + image_size = 4 * ctx->idx; + header = bpf_jit_binary_hdr(ctx->prog); + goto skip_init_ctx; } - preempt_enable(); - ctx.offsets = kcalloc(prog->len + 1, sizeof(*ctx.offsets), GFP_KERNEL); - if (ctx.offsets == NULL) + ctx->prog = prog; + ctx->offsets = kcalloc(prog->len + 1, + sizeof(*ctx->offsets), + GFP_KERNEL); + if (!ctx->offsets) goto out_err; - ctx.reg_val_types = kcalloc(prog->len + 1, sizeof(*ctx.reg_val_types), GFP_KERNEL); - if (ctx.reg_val_types == NULL) - goto out_err; + /* Check Octeon bbit ops only for MIPS64. */ + if (is64bit()) { + preempt_disable(); + switch (current_cpu_type()) { + case CPU_CAVIUM_OCTEON: + case CPU_CAVIUM_OCTEON_PLUS: + case CPU_CAVIUM_OCTEON2: + case CPU_CAVIUM_OCTEON3: + ctx->use_bbit_insns = 1; + break; + default: + ctx->use_bbit_insns = 0; + } + preempt_enable(); + } - ctx.skf = prog; + ctx->reg_val_types = kcalloc(prog->len + 1, + sizeof(*ctx->reg_val_types), + GFP_KERNEL); + if (!ctx->reg_val_types) + goto out_err; - if (reg_val_propagate(&ctx)) + if (reg_val_propagate(ctx)) goto out_err; /* * First pass discovers used resources and instruction offsets * assuming short branches are used. */ - if (build_int_body(&ctx)) + if (build_int_body(ctx)) goto out_err; /* - * If no calls are made (EBPF_SAVE_RA), then tail call count - * in $v1, else we must save in n$s4. + * If no calls are made (EBPF_SAVE_RA), then tailcall count located + * in runtime reg if defined, else we backup to save reg or stack. */ - if (ctx.flags & EBPF_SEEN_TC) { - if (ctx.flags & EBPF_SAVE_RA) - ctx.flags |= EBPF_SAVE_S4; - else - ctx.flags |= EBPF_TCC_IN_V1; + if (tail_call_present(ctx)) { + if (ctx->flags & EBPF_SAVE_RA) + ctx->flags |= bpf2mips[JIT_SAV_TCC].flags; + else if (bpf2mips[JIT_RUN_TCC].reg) + ctx->flags |= EBPF_TCC_IN_RUN; } /* @@ -1910,59 +2350,69 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * necessary. */ do { - ctx.idx = 0; - ctx.gen_b_offsets = 1; - ctx.long_b_conversion = 0; - if (gen_int_prologue(&ctx)) + ctx->idx = 0; + ctx->gen_b_offsets = 1; + ctx->long_b_conversion = 0; + if (build_int_prologue(ctx)) goto out_err; - if (build_int_body(&ctx)) + if (build_int_body(ctx)) goto out_err; - if (build_int_epilogue(&ctx, MIPS_R_RA)) + if (build_int_epilogue(ctx, MIPS_R_RA)) goto out_err; - } while (ctx.long_b_conversion); + } while (ctx->long_b_conversion); - image_size = 4 * ctx.idx; + image_size = 4 * ctx->idx; - header = bpf_jit_binary_alloc(image_size, &image_ptr, + header = bpf_jit_binary_alloc(image_size, (void *)&ctx->target, sizeof(u32), jit_fill_hole); - if (header == NULL) + if (!header) goto out_err; - ctx.target = (u32 *)image_ptr; +skip_init_ctx: - /* Third pass generates the code */ - ctx.idx = 0; - if (gen_int_prologue(&ctx)) + /* Third pass generates the code (fourth patches call addresses) */ + ctx->idx = 0; + if (build_int_prologue(ctx)) goto out_err; - if (build_int_body(&ctx)) + if (build_int_body(ctx)) goto out_err; - if (build_int_epilogue(&ctx, MIPS_R_RA)) + if (build_int_epilogue(ctx, MIPS_R_RA)) goto out_err; - /* Update the icache */ - flush_icache_range((unsigned long)ctx.target, - (unsigned long)&ctx.target[ctx.idx]); - if (bpf_jit_enable > 1) /* Dump JIT code */ - bpf_jit_dump(prog->len, image_size, 2, ctx.target); + bpf_jit_dump(prog->len, image_size, pass, ctx->target); + + /* Update the icache */ + flush_icache_range((unsigned long)ctx->target, + (unsigned long)&ctx->target[ctx->idx]); + + if (!prog->is_func || extra_pass) + bpf_jit_binary_lock_ro(header); + else + prog->aux->jit_data = ctx; - bpf_jit_binary_lock_ro(header); - prog->bpf_func = (void *)ctx.target; + prog->bpf_func = (void *)ctx->target; prog->jited = 1; prog->jited_len = image_size; -out_normal: + + if (!prog->is_func || extra_pass) { + bpf_prog_fill_jited_linfo(prog, ctx->offsets + 1); +out_ctx: + kfree(ctx->offsets); + kfree(ctx->reg_val_types); + kfree(ctx); + prog->aux->jit_data = NULL; + } +out: if (tmp_blinded) bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); - kfree(ctx.offsets); - kfree(ctx.reg_val_types); - return prog; out_err: prog = orig_prog; if (header) bpf_jit_binary_free(header); - goto out_normal; + goto out_ctx; } From patchwork Tue Oct 5 08:26:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535837 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 572F5C433EF for ; Tue, 5 Oct 2021 08:31:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3FFDD61056 for ; Tue, 5 Oct 2021 08:31:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232954AbhJEIdo (ORCPT ); Tue, 5 Oct 2021 04:33:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233460AbhJEIdj (ORCPT ); Tue, 5 Oct 2021 04:33:39 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A660C061755; Tue, 5 Oct 2021 01:31:49 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id q201so6452085pgq.12; Tue, 05 Oct 2021 01:31:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6XuRo9+dsLwMOEdoGhGnTKjYpBra+3KZ9lusvAfMqso=; b=TGpO7+Hsx23ekJIj0k//9H6Ms7oIBOKCV7BNtfOSpzZ3fgi7kZgQ4unSXlfxuSpP1K A87CBuOguCjsaSt7k7CXc/T2yrsT2uwM/RBeRdl2kbjuYEGrvwLXskyL+vHaS/YKDKFM Iql5sBSPCXZ4f4+AYkSbgJ82a6/HUS20Nd6r8B0p2o4+J52aEW6lSxKKoa8k3CjQWkhY 1bvfTciU7O+wwbYX/I7BSBCcfG8VNuhuSxZyobFtpYHDszSzmITqP16lDFqopmzXhlPk klyKaoVcRXtAhNdi4OUA6R39srOLsiBaTUjBbpUxc08MuOtxvHngcrYHz3Jcb7qt/8Dz DVUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6XuRo9+dsLwMOEdoGhGnTKjYpBra+3KZ9lusvAfMqso=; b=RJplqjCsY9VXglpzdKpHVEOaKWOVuwxNo/MwEApMaVyuJo1zDkJeu6iQEBfm8u5E2G z4HuNzC3zCLtIVZaWEoL284lBq09Kk/GMYLAkgP1Q5nwagyLWZWw6xnxSxlROICU/sOL 8UWhDaCcz5XD5dRdPLuyZGg9fEl3YEYr2yMyCHXioGeB/T3mSyesgdbMIWx0rdKnY/RS zKeZL11Bzhy939UjK2CyEWygfk8ktS3Z8hJ8vn94uEKwFADWgQBa6sZ8ob3pxllWgdwl C7tWlduOaZ8vlfJszdEBLTgaOZX6oLu0GPJ7pJs/g6YUj+hVb8auLP38EEPUk6Rrb+xi jcRQ== X-Gm-Message-State: AOAM532KbXbcvuD1Ylu7VVk8Wmi6HeMOXZ/VGN78qByX/H6jfmB5CkQM BmiJvFfuoyGjTJeY7VW1Nmc= X-Google-Smtp-Source: ABdhPJyLhqxZxibIepJ0YDGhNBSgdPWdkxN8oH95W/evrMgLubNxfcHaUIUoUPFOSQ7ymi8quwwpnQ== X-Received: by 2002:a63:7543:: with SMTP id f3mr14725608pgn.449.1633422709011; Tue, 05 Oct 2021 01:31:49 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:31:48 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 11/16] bpf: allow tailcalls in subprograms for MIPS64/MIPS32 Date: Tue, 5 Oct 2021 01:26:55 -0700 Message-Id: <77dfea2d224e7545e5e4d3f350721d27e5a77b0d.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The BPF core/verifier is hard-coded to permit mixing bpf2bpf and tail calls for only x86-64. Change the logic to instead rely on a new weak function 'bool bpf_jit_supports_subprog_tailcalls(void)', which a capable JIT backend can override. Update the x86-64 eBPF JIT to reflect this, and also enable the feature for the MIPS64/MIPS32 JIT. Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit.c | 6 ++++++ arch/x86/net/bpf_jit_comp.c | 6 ++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 6 ++++++ kernel/bpf/verifier.c | 3 ++- 5 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 7d8ed8bb19ab..501c1d532be6 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -2416,3 +2416,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_binary_free(header); goto out_ctx; } + +/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return true; +} diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 3ae729bb2475..2d78588fb5b5 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2367,3 +2367,9 @@ bool bpf_jit_supports_kfunc_call(void) { return true; } + +/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return true; +} diff --git a/include/linux/filter.h b/include/linux/filter.h index 16e5cebea82c..50b50fb271b5 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -933,6 +933,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); +bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_helper_changes_pkt_data(void *func); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 189934d2a3f2..c82b48ed0005 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2366,6 +2366,12 @@ bool __weak bpf_jit_needs_zext(void) return false; } +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ +bool __weak bpf_jit_supports_subprog_tailcalls(void) +{ + return false; +} + bool __weak bpf_jit_supports_kfunc_call(void) { return false; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6e2ebcb0d66f..267de5428fad 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5144,7 +5144,8 @@ static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id) static bool allow_tail_call_in_subprogs(struct bpf_verifier_env *env) { - return env->prog->jit_requested && IS_ENABLED(CONFIG_X86_64); + return env->prog->jit_requested && + bpf_jit_supports_subprog_tailcalls(); } static int check_map_func_compatibility(struct bpf_verifier_env *env, From patchwork Tue Oct 5 08:26:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535873 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F608C433F5 for ; Tue, 5 Oct 2021 08:32:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D763861056 for ; Tue, 5 Oct 2021 08:32:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233477AbhJEIdw (ORCPT ); Tue, 5 Oct 2021 04:33:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233449AbhJEIds (ORCPT ); Tue, 5 Oct 2021 04:33:48 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EE1AC061745; Tue, 5 Oct 2021 01:31:57 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id j15so1814260plh.7; Tue, 05 Oct 2021 01:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h7TtWjC1cjgQ0uTKQ3v5YHSgW+i9emfeomIkS0qfipQ=; b=IaUkqaiAY5NdbYjNPUPzxEobJZSg94irTysF5UBW89WyYgtAIw4D9DN/9eYJDTXFsB AdNolS8juYIyCnlwHub9P5+B8hUugoC6//1TeNydJaO9Iiq5E4w2Fv1+9XEQryGQukND aSE3Jsytp0fruFgSXwUgDsuvyW8ya94cnE+kTZd3+EgO6x6CrE8BiI3UBMKnryp7DsKE 9HOUuI6Bed1shypghzsgR/i/tsJKa4Z9RPdUNeLAEjJmvt3K4wTmBAAeEjpxl1+6BEDw Px0QfhF2Yjq4sqRDEWYHPpwBENr7L74XzAnQI5IMNHykg96Is37wQqd8gKWYUp2eAMkm 7M3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h7TtWjC1cjgQ0uTKQ3v5YHSgW+i9emfeomIkS0qfipQ=; b=GEINHX6Bwb6aeMJRt4YcY1a5w3crGA61KAFJ8usM8z2I3ZJBLIpBUvY1xI/lvpTts/ dcmSxPAmFSXwfEWIvmR0IO6NDUNhNMAso5m0F5q3HACBb6EQCDAvCNEedQkllX+DSPy+ TUXpssHpew/AuTtF7n3b5aJIgeOTehEpvHMA6XozJ5QktORXc4SBt+cm1LM2ouAKDrLF OnAxmnIL2Az6+vl15uK00el9JO7dU5Aiek7kQDmhNhERzagmLQUHVpilJMQzpjG0uQZb WV2DCwoY54V81aSfmIVXFnRTt7+79lPTe770vEAm1OegwoujjOZgU0ge/v24qB1N1kmE sn2Q== X-Gm-Message-State: AOAM532IV1kJbQmFEvHrMZH/imotOf2u3g+TVgSAlTPoZW4+pdDsgsUT fywdeNzC4rENuu6Tb2g7yfg= X-Google-Smtp-Source: ABdhPJw7YR6ac64fOw9orTUmPWekSn5RxkQKUHg6Z1lImzC/MMggkIaO+wRKNQT41C+KCO4N/mT4mA== X-Received: by 2002:a17:902:6f01:b0:138:9aca:efda with SMTP id w1-20020a1709026f0100b001389acaefdamr4162295plk.19.1633422715189; Tue, 05 Oct 2021 01:31:55 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:31:54 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 12/16] MIPS: eBPF: refactor common 32/64-bit functions and headers Date: Tue, 5 Oct 2021 01:26:56 -0700 Message-Id: <56f0d08169bed47303a482a5c6bff67fe1e4bdee.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Move core functions and headers to ebpf_jit_core.c and ebpf_jit.h, and relocate the MIPS64 specific build_one_insn() to ebpf_jit_comp64.c. Signed-off-by: Tony Ambardar --- arch/mips/net/Makefile | 2 +- arch/mips/net/ebpf_jit.c | 2424 ------------------------------- arch/mips/net/ebpf_jit.h | 297 ++++ arch/mips/net/ebpf_jit_comp64.c | 990 +++++++++++++ arch/mips/net/ebpf_jit_core.c | 1189 +++++++++++++++ 5 files changed, 2477 insertions(+), 2425 deletions(-) delete mode 100644 arch/mips/net/ebpf_jit.c create mode 100644 arch/mips/net/ebpf_jit.h create mode 100644 arch/mips/net/ebpf_jit_comp64.c create mode 100644 arch/mips/net/ebpf_jit_core.c diff --git a/arch/mips/net/Makefile b/arch/mips/net/Makefile index d55912349039..de42f4a4db56 100644 --- a/arch/mips/net/Makefile +++ b/arch/mips/net/Makefile @@ -2,4 +2,4 @@ # MIPS networking code obj-$(CONFIG_MIPS_CBPF_JIT) += bpf_jit.o bpf_jit_asm.o -obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit.o +obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit_core.o ebpf_jit_comp64.o diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c deleted file mode 100644 index 501c1d532be6..000000000000 --- a/arch/mips/net/ebpf_jit.c +++ /dev/null @@ -1,2424 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 - * Copyright (c) 2021 Tony Ambardar - * - * Based on code from: - * - * Copyright (c) 2017 Cavium, Inc. - * Author: David Daney - * - * Copyright (c) 2014 Imagination Technologies Ltd. - * Author: Markos Chandras - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* Registers used by JIT: (MIPS32) (MIPS64) */ -#define MIPS_R_ZERO 0 -#define MIPS_R_AT 1 -#define MIPS_R_V0 2 /* BPF_R0 BPF_R0 */ -#define MIPS_R_V1 3 /* BPF_R0 BPF_TCC */ -#define MIPS_R_A0 4 /* BPF_R1 BPF_R1 */ -#define MIPS_R_A1 5 /* BPF_R1 BPF_R2 */ -#define MIPS_R_A2 6 /* BPF_R2 BPF_R3 */ -#define MIPS_R_A3 7 /* BPF_R2 BPF_R4 */ - -/* MIPS64 replaces T0-T3 scratch regs with extra arguments A4-A7. */ -#ifdef CONFIG_64BIT -# define MIPS_R_A4 8 /* (n/a) BPF_R5 */ -#else -# define MIPS_R_T0 8 /* BPF_R3 (n/a) */ -# define MIPS_R_T1 9 /* BPF_R3 (n/a) */ -# define MIPS_R_T2 10 /* BPF_R4 (n/a) */ -# define MIPS_R_T3 11 /* BPF_R4 (n/a) */ -#endif - -#define MIPS_R_T4 12 /* BPF_R5 BPF_AX */ -#define MIPS_R_T5 13 /* BPF_R5 (free) */ -#define MIPS_R_T6 14 /* BPF_AX (used) */ -#define MIPS_R_T7 15 /* BPF_AX (free) */ -#define MIPS_R_S0 16 /* BPF_R6 BPF_R6 */ -#define MIPS_R_S1 17 /* BPF_R6 BPF_R7 */ -#define MIPS_R_S2 18 /* BPF_R7 BPF_R8 */ -#define MIPS_R_S3 19 /* BPF_R7 BPF_R9 */ -#define MIPS_R_S4 20 /* BPF_R8 BPF_TCC */ -#define MIPS_R_S5 21 /* BPF_R8 (free) */ -#define MIPS_R_S6 22 /* BPF_R9 (free) */ -#define MIPS_R_S7 23 /* BPF_R9 (free) */ -#define MIPS_R_T8 24 /* (used) (used) */ -#define MIPS_R_T9 25 /* (used) (used) */ -#define MIPS_R_SP 29 -#define MIPS_R_S8 30 /* BPF_R10 BPF_R10 */ -#define MIPS_R_RA 31 - -/* eBPF flags */ -#define EBPF_SAVE_S0 BIT(0) -#define EBPF_SAVE_S1 BIT(1) -#define EBPF_SAVE_S2 BIT(2) -#define EBPF_SAVE_S3 BIT(3) -#define EBPF_SAVE_S4 BIT(4) -#define EBPF_SAVE_S5 BIT(5) -#define EBPF_SAVE_S6 BIT(6) -#define EBPF_SAVE_S7 BIT(7) -#define EBPF_SAVE_S8 BIT(8) -#define EBPF_SAVE_RA BIT(9) -#define EBPF_SEEN_FP BIT(10) -#define EBPF_SEEN_TC BIT(11) -#define EBPF_TCC_IN_RUN BIT(12) - -/* - * Extra JIT registers dedicated to holding TCC during runtime or saving - * across calls. - */ -enum { - JIT_RUN_TCC = MAX_BPF_JIT_REG, - JIT_SAV_TCC -}; -/* Temporary register for passing TCC if nothing dedicated. */ -#define TEMP_PASS_TCC MIPS_R_T8 - -/* - * Word-size and endianness-aware helpers for building MIPS32 vs MIPS64 - * tables and selecting 32-bit subregisters from a register pair base. - * Simplify use by emulating MIPS_R_SP and MIPS_R_ZERO as register pairs - * and adding HI/LO word memory offsets. - */ -#ifdef CONFIG_64BIT -# define HI(reg) (reg) -# define LO(reg) (reg) -# define OFFHI(mem) (mem) -# define OFFLO(mem) (mem) -#else /* CONFIG_32BIT */ -# ifdef __BIG_ENDIAN -# define HI(reg) ((reg) == MIPS_R_SP ? MIPS_R_ZERO : \ - (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ - (reg)) -# define LO(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ - (reg) == MIPS_R_SP ? (reg) : \ - (reg) == MIPS_R_S8 ? (reg) : \ - (reg) + 1) -# define OFFHI(mem) (mem) -# define OFFLO(mem) ((mem) + sizeof(long)) -# else /* __LITTLE_ENDIAN */ -# define HI(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ - (reg) == MIPS_R_SP ? MIPS_R_ZERO : \ - (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ - (reg) + 1) -# define LO(reg) (reg) -# define OFFHI(mem) ((mem) + sizeof(long)) -# define OFFLO(mem) (mem) -# endif -#endif - -#ifdef CONFIG_64BIT -# define M(expr32, expr64) (expr64) -#else -# define M(expr32, expr64) (expr32) -#endif -const struct { - /* Register or pair base */ - int reg; - /* Register flags */ - u32 flags; - /* Usage table: (MIPS32) (MIPS64) */ -} bpf2mips[] = { - /* Return value from in-kernel function, and exit value from eBPF. */ - [BPF_REG_0] = {M(MIPS_R_V0, MIPS_R_V0)}, - /* Arguments from eBPF program to in-kernel/BPF functions. */ - [BPF_REG_1] = {M(MIPS_R_A0, MIPS_R_A0)}, - [BPF_REG_2] = {M(MIPS_R_A2, MIPS_R_A1)}, - [BPF_REG_3] = {M(MIPS_R_T0, MIPS_R_A2)}, - [BPF_REG_4] = {M(MIPS_R_T2, MIPS_R_A3)}, - [BPF_REG_5] = {M(MIPS_R_T4, MIPS_R_A4)}, - /* Callee-saved registers preserved by in-kernel/BPF functions. */ - [BPF_REG_6] = {M(MIPS_R_S0, MIPS_R_S0), - M(EBPF_SAVE_S0|EBPF_SAVE_S1, EBPF_SAVE_S0)}, - [BPF_REG_7] = {M(MIPS_R_S2, MIPS_R_S1), - M(EBPF_SAVE_S2|EBPF_SAVE_S3, EBPF_SAVE_S1)}, - [BPF_REG_8] = {M(MIPS_R_S4, MIPS_R_S2), - M(EBPF_SAVE_S4|EBPF_SAVE_S5, EBPF_SAVE_S2)}, - [BPF_REG_9] = {M(MIPS_R_S6, MIPS_R_S3), - M(EBPF_SAVE_S6|EBPF_SAVE_S7, EBPF_SAVE_S3)}, - [BPF_REG_10] = {M(MIPS_R_S8, MIPS_R_S8), - M(EBPF_SAVE_S8|EBPF_SEEN_FP, EBPF_SAVE_S8|EBPF_SEEN_FP)}, - /* Internal register for rewriting insns during JIT blinding. */ - [BPF_REG_AX] = {M(MIPS_R_T6, MIPS_R_T4)}, - /* - * Internal registers for TCC runtime holding and saving during - * calls. A zero save register indicates using scratch space on - * the stack for storage during calls. A zero hold register means - * no dedicated register holds TCC during runtime (but a temp reg - * still passes TCC to tailcall or bpf2bpf call). - */ - [JIT_RUN_TCC] = {M(0, MIPS_R_V1)}, - [JIT_SAV_TCC] = {M(0, MIPS_R_S4), - M(0, EBPF_SAVE_S4)} -}; -#undef M - -static inline bool is64bit(void) -{ - return IS_ENABLED(CONFIG_64BIT); -} - -static inline bool isbigend(void) -{ - return IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); -} - -/* Stack region alignment under N64 and O32 ABIs */ -#define STACK_ALIGN (2 * sizeof(long)) - -/* - * For the mips64 ISA, we need to track the value range or type for - * each JIT register. The BPF machine requires zero extended 32-bit - * values, but the mips64 ISA requires sign extended 32-bit values. - * At each point in the BPF program we track the state of every - * register so that we can zero extend or sign extend as the BPF - * semantics require. - */ -enum reg_val_type { - /* uninitialized */ - REG_UNKNOWN, - /* not known to be 32-bit compatible. */ - REG_64BIT, - /* 32-bit compatible, no truncation needed for 64-bit ops. */ - REG_64BIT_32BIT, - /* 32-bit compatible, need truncation for 64-bit ops. */ - REG_32BIT, - /* 32-bit no sign/zero extension needed. */ - REG_32BIT_POS -}; - -/* - * high bit of offsets indicates if long branch conversion done at - * this insn. - */ -#define OFFSETS_B_CONV BIT(31) - -/** - * struct jit_ctx - JIT context - * @prog: The program - * @stack_size: eBPF stack size - * @bpf_stack_off: eBPF FP offset - * @prolog_skip: Prologue insns to skip by BPF caller - * @idx: Instruction index - * @flags: JIT flags - * @offsets: Instruction offsets - * @target: Memory location for compiled instructions - * @reg_val_types: Packed enum reg_val_type for each register - */ -struct jit_ctx { - const struct bpf_prog *prog; - int stack_size; - int bpf_stack_off; - int prolog_skip; - u32 idx; - u32 flags; - u32 *offsets; - u32 *target; - u64 *reg_val_types; - unsigned int long_b_conversion:1; - unsigned int gen_b_offsets:1; - unsigned int use_bbit_insns:1; -}; - -static void set_reg_val_type(u64 *rvt, int reg, enum reg_val_type type) -{ - *rvt &= ~(7ull << (reg * 3)); - *rvt |= ((u64)type << (reg * 3)); -} - -static enum reg_val_type get_reg_val_type(const struct jit_ctx *ctx, - int index, int reg) -{ - return (ctx->reg_val_types[index] >> (reg * 3)) & 7; -} - -/* Simply emit the instruction if the JIT memory space has been allocated */ -#define emit_instr_long(ctx, func64, func32, ...) \ -do { \ - if ((ctx)->target != NULL) { \ - u32 *p = &(ctx)->target[ctx->idx]; \ - if (IS_ENABLED(CONFIG_64BIT)) \ - uasm_i_##func64(&p, ##__VA_ARGS__); \ - else \ - uasm_i_##func32(&p, ##__VA_ARGS__); \ - } \ - (ctx)->idx++; \ -} while (0) - -#define emit_instr(ctx, func, ...) \ - emit_instr_long(ctx, func, func, ##__VA_ARGS__) - -static unsigned int j_target(struct jit_ctx *ctx, int target_idx) -{ - unsigned long target_va, base_va; - unsigned int r; - - if (!ctx->target) - return 0; - - base_va = (unsigned long)ctx->target; - target_va = base_va + (ctx->offsets[target_idx] & ~OFFSETS_B_CONV); - - if ((base_va & ~0x0ffffffful) != (target_va & ~0x0ffffffful)) - return (unsigned int)-1; - r = target_va & 0x0ffffffful; - return r; -} - -/* Compute the immediate value for PC-relative branches. */ -static u32 b_imm(unsigned int tgt, struct jit_ctx *ctx) -{ - if (!ctx->gen_b_offsets) - return 0; - - /* - * We want a pc-relative branch. tgt is the instruction offset - * we want to jump to. - - * Branch on MIPS: - * I: target_offset <- sign_extend(offset) - * I+1: PC += target_offset (delay slot) - * - * ctx->idx currently points to the branch instruction - * but the offset is added to the delay slot so we need - * to subtract 4. - */ - return (ctx->offsets[tgt] & ~OFFSETS_B_CONV) - - (ctx->idx * 4) - 4; -} - -/* Sign-extend dst register or HI 32-bit reg of pair. */ -static inline void gen_sext_insn(int dst, struct jit_ctx *ctx) -{ - if (is64bit()) - emit_instr(ctx, sll, dst, dst, 0); - else - emit_instr(ctx, sra, HI(dst), LO(dst), 31); -} - -/* - * Zero-extend dst register or HI 32-bit reg of pair, if either forced - * or the BPF verifier does not insert its own zext insns. - */ -static inline void gen_zext_insn(int dst, bool force, struct jit_ctx *ctx) -{ - if (!ctx->prog->aux->verifier_zext || force) { - if (is64bit()) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - else - emit_instr(ctx, and, HI(dst), MIPS_R_ZERO, MIPS_R_ZERO); - } -} - -static inline bool tail_call_present(struct jit_ctx *ctx) -{ - return ctx->flags & EBPF_SEEN_TC || ctx->prog->aux->tail_call_reachable; -} - -enum reg_usage { - REG_SRC_FP_OK, - REG_SRC_NO_FP, - REG_DST_FP_OK, - REG_DST_NO_FP -}; - -/* - * For eBPF, the register mapping naturally falls out of the - * requirements of eBPF and the MIPS N64/O32 ABIs. We also maintain - * a separate frame pointer, setting BPF_REG_10 relative to $sp. - */ -static int ebpf_to_mips_reg(struct jit_ctx *ctx, - const struct bpf_insn *insn, - enum reg_usage u) -{ - int ebpf_reg = (u == REG_SRC_FP_OK || u == REG_SRC_NO_FP) ? - insn->src_reg : insn->dst_reg; - - switch (ebpf_reg) { - case BPF_REG_0: - case BPF_REG_1: - case BPF_REG_2: - case BPF_REG_3: - case BPF_REG_4: - case BPF_REG_5: - case BPF_REG_6: - case BPF_REG_7: - case BPF_REG_8: - case BPF_REG_9: - case BPF_REG_AX: - ctx->flags |= bpf2mips[ebpf_reg].flags; - return bpf2mips[ebpf_reg].reg; - case BPF_REG_10: - if (u == REG_DST_NO_FP || u == REG_SRC_NO_FP) - goto bad_reg; - ctx->flags |= bpf2mips[ebpf_reg].flags; - return bpf2mips[ebpf_reg].reg; - default: -bad_reg: - WARN(1, "Illegal bpf reg: %d\n", ebpf_reg); - return -EINVAL; - } -} - -/* - * eBPF stack frame will be something like: - * - * Entry $sp ------> +--------------------------------+ - * | $ra (optional) | - * +--------------------------------+ - * | $s8 (optional) | - * +--------------------------------+ - * | $s7 (optional) | - * +--------------------------------+ - * | $s6 (optional) | - * +--------------------------------+ - * | $s5 (optional) | - * +--------------------------------+ - * | $s4 (optional) | - * +--------------------------------+ - * | $s3 (optional) | - * +--------------------------------+ - * | $s2 (optional) | - * +--------------------------------+ - * | $s1 (optional) | - * +--------------------------------+ - * | $s0 (optional) | - * +--------------------------------+ - * | tmp-storage (optional) | - * $sp + bpf_stack_off->+--------------------------------+ <--BPF_REG_10 - * | BPF_REG_10 relative storage | - * | MAX_BPF_STACK (optional) | - * | . | - * | . | - * | . | - * $sp ------> +--------------------------------+ - * - * If BPF_REG_10 is never referenced, then the MAX_BPF_STACK sized - * area is not allocated. - */ -static int build_int_prologue(struct jit_ctx *ctx) -{ - int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? - bpf2mips[JIT_RUN_TCC].reg : - TEMP_PASS_TCC; - int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; - const struct bpf_prog *prog = ctx->prog; - int r10 = bpf2mips[BPF_REG_10].reg; - int r1 = bpf2mips[BPF_REG_1].reg; - int stack_adjust = 0; - int store_offset; - int locals_size; - int start_idx; - - if (ctx->flags & EBPF_SAVE_RA) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S8) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S7) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S6) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S5) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S4) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S3) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S2) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S1) - stack_adjust += sizeof(long); - if (ctx->flags & EBPF_SAVE_S0) - stack_adjust += sizeof(long); - if (tail_call_present(ctx) && - !(ctx->flags & EBPF_TCC_IN_RUN) && !tcc_sav) - /* Allocate scratch space for holding TCC if needed. */ - stack_adjust += sizeof(long); - - stack_adjust = ALIGN(stack_adjust, STACK_ALIGN); - - locals_size = (ctx->flags & EBPF_SEEN_FP) ? prog->aux->stack_depth : 0; - locals_size = ALIGN(locals_size, STACK_ALIGN); - - stack_adjust += locals_size; - - ctx->stack_size = stack_adjust; - ctx->bpf_stack_off = locals_size; - - /* - * First instruction initializes the tail call count (TCC) and - * assumes a call from kernel using the native ABI. Calls made - * using the BPF ABI (bpf2bpf or tail call) will skip this insn - * and pass the TCC via register. - */ - start_idx = ctx->idx; - emit_instr(ctx, addiu, tcc_run, MIPS_R_ZERO, MAX_TAIL_CALL_CNT); - - /* - * When called from kernel under O32 ABI we must set up BPF R1 - * context, since BPF R1 is an endian-order regster pair ($a0:$a1 - * or $a1:$a0) but context is always passed in $a0 as a 32-bit - * pointer. As above, bpf2bpf and tail calls will skip these insns - * since all registers are correctly set up already. - */ - if (!is64bit()) { - if (isbigend()) - emit_instr(ctx, move, LO(r1), MIPS_R_A0); - /* Sanitize upper 32-bit reg */ - gen_zext_insn(r1, true, ctx); - } - /* - * Calls using BPF ABI (bpf2bpf and tail calls) will skip TCC - * initialization and R1 context fixup needed by kernel calls. - */ - ctx->prolog_skip = (ctx->idx - start_idx) * 4; - - if (stack_adjust) - emit_instr_long(ctx, daddiu, addiu, - MIPS_R_SP, MIPS_R_SP, -stack_adjust); - else - return 0; - - store_offset = stack_adjust - sizeof(long); - - if (ctx->flags & EBPF_SAVE_RA) { - emit_instr_long(ctx, sd, sw, - MIPS_R_RA, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S8) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S8, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S7) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S7, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S6) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S6, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S5) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S5, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S4) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S4, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S3) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S3, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S2) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S2, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S1) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S1, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S0) { - emit_instr_long(ctx, sd, sw, - MIPS_R_S0, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - - /* Store TCC in backup register or stack scratch space if indicated. */ - if (tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { - if (tcc_sav) - emit_instr(ctx, move, tcc_sav, tcc_run); - else - emit_instr_long(ctx, sd, sw, - tcc_run, ctx->bpf_stack_off, MIPS_R_SP); - } - - /* Prepare BPF FP as single-reg ptr, emulate upper 32-bits as needed.*/ - if (ctx->flags & EBPF_SEEN_FP) - emit_instr_long(ctx, daddiu, addiu, r10, - MIPS_R_SP, ctx->bpf_stack_off); - - return 0; -} - -static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg) -{ - const struct bpf_prog *prog = ctx->prog; - int stack_adjust = ctx->stack_size; - int store_offset = stack_adjust - sizeof(long); - int ax = bpf2mips[BPF_REG_AX].reg; - int r0 = bpf2mips[BPF_REG_0].reg; - enum reg_val_type td; - - /* - * As in prologue code, we default to assuming exit to the kernel. - * Returns to the kernel follow the N64 or O32 ABI. For N64, the - * BPF R0 return value may need to be sign-extended, while O32 may - * need fixup of BPF R0 to place the 32-bit return value in MIPS V0. - * - * Returns to BPF2BPF callers consistently use the BPF 64-bit ABI, - * so register usage and mapping between JIT and OS is unchanged. - * Accommodate by saving unmodified R0 register data to allow a - * BPF caller to restore R0 after we return. - */ - if (dest_reg == MIPS_R_RA) { /* kernel or bpf2bpf function return */ - if (is64bit()) { - /* - * Backup BPF R0 to AX, allowing the caller to - * restore it in case this is a BPF2BPF rather - * than a kernel return. - */ - emit_instr(ctx, move, ax, r0); - /* - * Don't let zero-extended R0 value escape to - * kernel on return, so sign-extend if needed. - */ - td = get_reg_val_type(ctx, prog->len, BPF_REG_0); - if (td == REG_64BIT) - gen_sext_insn(r0, ctx); - } else if (isbigend()) { /* and 32-bit */ - /* - * Backup high 32-bit register of BPF R0 to AX, - * since it occupies MIPS_R_V0 which needs to be - * clobbered for a kernel return. - */ - emit_instr(ctx, move, HI(ax), HI(r0)); - /* - * O32 ABI specifies 32-bit return value always - * placed in MIPS_R_V0 regardless of the native - * endianness. This would be in the wrong position - * in a BPF R0 reg pair on big-endian systems, so - * we must relocate. - */ - emit_instr(ctx, move, MIPS_R_V0, LO(r0)); - } - } - - if (ctx->flags & EBPF_SAVE_RA) { - emit_instr_long(ctx, ld, lw, - MIPS_R_RA, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S8) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S8, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S7) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S7, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S6) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S6, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S5) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S5, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S4) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S4, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S3) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S3, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S2) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S2, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S1) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S1, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - if (ctx->flags & EBPF_SAVE_S0) { - emit_instr_long(ctx, ld, lw, - MIPS_R_S0, store_offset, MIPS_R_SP); - store_offset -= sizeof(long); - } - emit_instr(ctx, jr, dest_reg); - - /* Delay slot */ - if (stack_adjust) - emit_instr_long(ctx, daddiu, addiu, - MIPS_R_SP, MIPS_R_SP, stack_adjust); - else - emit_instr(ctx, nop); - - return 0; -} - -static void gen_imm_to_reg(const struct bpf_insn *insn, int reg, - struct jit_ctx *ctx) -{ - if (insn->imm >= S16_MIN && insn->imm <= S16_MAX) { - emit_instr(ctx, addiu, reg, MIPS_R_ZERO, insn->imm); - } else { - int lower = (s16)(insn->imm & 0xffff); - int upper = insn->imm - lower; - - emit_instr(ctx, lui, reg, upper >> 16); - /* lui already clears lower halfword */ - if (lower) - emit_instr(ctx, addiu, reg, reg, lower); - } -} - -static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, - int idx) -{ - int upper_bound, lower_bound; - int dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - - if (dst < 0) - return dst; - - switch (BPF_OP(insn->code)) { - case BPF_MOV: - case BPF_ADD: - upper_bound = S16_MAX; - lower_bound = S16_MIN; - break; - case BPF_SUB: - upper_bound = -(int)S16_MIN; - lower_bound = -(int)S16_MAX; - break; - case BPF_AND: - case BPF_OR: - case BPF_XOR: - upper_bound = 0xffff; - lower_bound = 0; - break; - case BPF_RSH: - case BPF_LSH: - case BPF_ARSH: - /* Shift amounts are truncated, no need for bounds */ - upper_bound = S32_MAX; - lower_bound = S32_MIN; - break; - default: - return -EINVAL; - } - - /* - * Immediate move clobbers the register, so no sign/zero - * extension needed. - */ - if (BPF_CLASS(insn->code) == BPF_ALU64 && - BPF_OP(insn->code) != BPF_MOV && - get_reg_val_type(ctx, idx, insn->dst_reg) == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - /* BPF_ALU | BPF_LSH doesn't need separate sign extension */ - if (BPF_CLASS(insn->code) == BPF_ALU && - BPF_OP(insn->code) != BPF_LSH && - BPF_OP(insn->code) != BPF_MOV && - get_reg_val_type(ctx, idx, insn->dst_reg) != REG_32BIT) - emit_instr(ctx, sll, dst, dst, 0); - - if (insn->imm >= lower_bound && insn->imm <= upper_bound) { - /* single insn immediate case */ - switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { - case BPF_ALU64 | BPF_MOV: - emit_instr(ctx, daddiu, dst, MIPS_R_ZERO, insn->imm); - break; - case BPF_ALU64 | BPF_AND: - case BPF_ALU | BPF_AND: - emit_instr(ctx, andi, dst, dst, insn->imm); - break; - case BPF_ALU64 | BPF_OR: - case BPF_ALU | BPF_OR: - emit_instr(ctx, ori, dst, dst, insn->imm); - break; - case BPF_ALU64 | BPF_XOR: - case BPF_ALU | BPF_XOR: - emit_instr(ctx, xori, dst, dst, insn->imm); - break; - case BPF_ALU64 | BPF_ADD: - emit_instr(ctx, daddiu, dst, dst, insn->imm); - break; - case BPF_ALU64 | BPF_SUB: - emit_instr(ctx, daddiu, dst, dst, -insn->imm); - break; - case BPF_ALU64 | BPF_RSH: - emit_instr(ctx, dsrl_safe, dst, dst, insn->imm & 0x3f); - break; - case BPF_ALU | BPF_RSH: - emit_instr(ctx, srl, dst, dst, insn->imm & 0x1f); - break; - case BPF_ALU64 | BPF_LSH: - emit_instr(ctx, dsll_safe, dst, dst, insn->imm & 0x3f); - break; - case BPF_ALU | BPF_LSH: - emit_instr(ctx, sll, dst, dst, insn->imm & 0x1f); - break; - case BPF_ALU64 | BPF_ARSH: - emit_instr(ctx, dsra_safe, dst, dst, insn->imm & 0x3f); - break; - case BPF_ALU | BPF_ARSH: - emit_instr(ctx, sra, dst, dst, insn->imm & 0x1f); - break; - case BPF_ALU | BPF_MOV: - emit_instr(ctx, addiu, dst, MIPS_R_ZERO, insn->imm); - break; - case BPF_ALU | BPF_ADD: - emit_instr(ctx, addiu, dst, dst, insn->imm); - break; - case BPF_ALU | BPF_SUB: - emit_instr(ctx, addiu, dst, dst, -insn->imm); - break; - default: - return -EINVAL; - } - } else { - /* multi insn immediate case */ - if (BPF_OP(insn->code) == BPF_MOV) { - gen_imm_to_reg(insn, dst, ctx); - } else { - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { - case BPF_ALU64 | BPF_AND: - case BPF_ALU | BPF_AND: - emit_instr(ctx, and, dst, dst, MIPS_R_AT); - break; - case BPF_ALU64 | BPF_OR: - case BPF_ALU | BPF_OR: - emit_instr(ctx, or, dst, dst, MIPS_R_AT); - break; - case BPF_ALU64 | BPF_XOR: - case BPF_ALU | BPF_XOR: - emit_instr(ctx, xor, dst, dst, MIPS_R_AT); - break; - case BPF_ALU64 | BPF_ADD: - emit_instr(ctx, daddu, dst, dst, MIPS_R_AT); - break; - case BPF_ALU64 | BPF_SUB: - emit_instr(ctx, dsubu, dst, dst, MIPS_R_AT); - break; - case BPF_ALU | BPF_ADD: - emit_instr(ctx, addu, dst, dst, MIPS_R_AT); - break; - case BPF_ALU | BPF_SUB: - emit_instr(ctx, subu, dst, dst, MIPS_R_AT); - break; - default: - return -EINVAL; - } - } - } - - return 0; -} - -static void emit_const_to_reg(struct jit_ctx *ctx, int dst, unsigned long value) -{ - if (value >= S16_MIN || value <= S16_MAX) { - emit_instr_long(ctx, daddiu, addiu, dst, MIPS_R_ZERO, (int)value); - } else if (value >= S32_MIN || - (value <= S32_MAX && value > U16_MAX)) { - emit_instr(ctx, lui, dst, (s32)(s16)(value >> 16)); - emit_instr(ctx, ori, dst, dst, (unsigned int)(value & 0xffff)); - } else { - int i; - bool seen_part = false; - int needed_shift = 0; - - for (i = 0; i < 4; i++) { - u64 part = (value >> (16 * (3 - i))) & 0xffff; - - if (seen_part && needed_shift > 0 && (part || i == 3)) { - emit_instr(ctx, dsll_safe, dst, dst, needed_shift); - needed_shift = 0; - } - if (part) { - if (i == 0 || (!seen_part && i < 3 && part < 0x8000)) { - emit_instr(ctx, lui, dst, (s32)(s16)part); - needed_shift = -16; - } else { - emit_instr(ctx, ori, dst, - seen_part ? dst : MIPS_R_ZERO, - (unsigned int)part); - } - seen_part = true; - } - if (seen_part) - needed_shift += 16; - } - } -} - -/* - * Push BPF regs R3-R5 to the stack, skipping BPF regs R1-R2 which are - * passed via MIPS register pairs in $a0-$a3. Register order within pairs - * and the memory storage order are identical i.e. endian native. - */ -static void emit_push_args(struct jit_ctx *ctx) -{ - int store_offset = 2 * sizeof(u64); /* Skip R1-R2 in $a0-$a3 */ - int bpf, reg; - - for (bpf = BPF_REG_3; bpf <= BPF_REG_5; bpf++) { - reg = bpf2mips[bpf].reg; - - emit_instr(ctx, sw, LO(reg), OFFLO(store_offset), MIPS_R_SP); - emit_instr(ctx, sw, HI(reg), OFFHI(store_offset), MIPS_R_SP); - store_offset += sizeof(u64); - } -} - -/* - * Common helper for BPF_CALL insn, handling TCC and ABI variations. - * Kernel calls under O32 ABI require arguments passed on the stack, - * while BPF2BPF calls need the TCC passed via register as expected - * by the subprog's prologue. - * - * Under MIPS32 O32 ABI calling convention, u64 BPF regs R1-R2 are passed - * via reg pairs in $a0-$a3, while BPF regs R3-R5 are passed via the stack. - * Stack space is still reserved for $a0-$a3, and the whole area aligned. - */ -#define ARGS_SIZE (5 * sizeof(u64)) - -void emit_bpf_call(struct jit_ctx *ctx, const struct bpf_insn *insn) -{ - int stack_adjust = ALIGN(ARGS_SIZE, STACK_ALIGN); - int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? - bpf2mips[JIT_RUN_TCC].reg : - TEMP_PASS_TCC; - int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; - int ax = bpf2mips[BPF_REG_AX].reg; - int r0 = bpf2mips[BPF_REG_0].reg; - long func_addr; - - ctx->flags |= EBPF_SAVE_RA; - - /* Ensure TCC passed into BPF subprog */ - if ((insn->src_reg == BPF_PSEUDO_CALL) && - tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { - /* Set TCC from reg or stack */ - if (tcc_sav) - emit_instr(ctx, move, tcc_run, tcc_sav); - else - emit_instr_long(ctx, ld, lw, tcc_run, - ctx->bpf_stack_off, MIPS_R_SP); - } - - /* Push O32 stack args for kernel call */ - if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) { - emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adjust); - emit_push_args(ctx); - } - - func_addr = (long)__bpf_call_base + insn->imm; - - /* Skip TCC init and R1 register fixup with BPF ABI. */ - if (insn->src_reg == BPF_PSEUDO_CALL) - func_addr += ctx->prolog_skip; - - emit_const_to_reg(ctx, MIPS_R_T9, func_addr); - emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); - /* Delay slot */ - emit_instr(ctx, nop); - - /* Restore stack */ - if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) - emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, stack_adjust); - - /* - * Assuming a kernel return, a MIPS64 function epilogue may - * sign-extend R0, while MIPS32BE mangles the R0 register pair. - * Undo both for a bpf2bpf call return. - */ - if (insn->src_reg == BPF_PSEUDO_CALL) { - /* Restore BPF R0 from AX */ - if (is64bit()) { - emit_instr(ctx, move, r0, ax); - } else if (isbigend()) { /* and 32-bit */ - emit_instr(ctx, move, LO(r0), MIPS_R_V0); - emit_instr(ctx, move, HI(r0), HI(ax)); - } - } -} - -/* - * Tail call helper arguments passed via BPF ABI as u64 parameters. On - * MIPS64 N64 ABI systems these are native regs, while on MIPS32 O32 ABI - * systems these are reg pairs: - * - * R1 -> &ctx - * R2 -> &array - * R3 -> index - */ -static int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx) -{ - int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? - bpf2mips[JIT_RUN_TCC].reg : - TEMP_PASS_TCC; - int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; - int r2 = bpf2mips[BPF_REG_2].reg; - int r3 = bpf2mips[BPF_REG_3].reg; - int off, b_off; - int tcc; - - ctx->flags |= EBPF_SEEN_TC; - /* - * if (index >= array->map.max_entries) - * goto out; - */ - if (is64bit()) - /* Mask index as 32-bit */ - gen_zext_insn(r3, true, ctx); - off = offsetof(struct bpf_array, map.max_entries); - emit_instr_long(ctx, lwu, lw, MIPS_R_AT, off, LO(r2)); - emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_AT, LO(r3)); - b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, bnez, MIPS_R_AT, b_off); - /* - * if (TCC-- < 0) - * goto out; - */ - /* Delay slot */ - tcc = (ctx->flags & EBPF_TCC_IN_RUN) ? tcc_run : tcc_sav; - /* Get TCC from reg or stack */ - if (tcc) - emit_instr(ctx, move, MIPS_R_T8, tcc); - else - emit_instr_long(ctx, ld, lw, MIPS_R_T8, - ctx->bpf_stack_off, MIPS_R_SP); - b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, bltz, MIPS_R_T8, b_off); - /* - * prog = array->ptrs[index]; - * if (prog == NULL) - * goto out; - */ - /* Delay slot */ - emit_instr_long(ctx, dsll, sll, MIPS_R_AT, LO(r3), ilog2(sizeof(long))); - emit_instr_long(ctx, daddu, addu, MIPS_R_AT, MIPS_R_AT, LO(r2)); - off = offsetof(struct bpf_array, ptrs); - emit_instr_long(ctx, ld, lw, MIPS_R_AT, off, MIPS_R_AT); - b_off = b_imm(this_idx + 1, ctx); - emit_instr(ctx, beqz, MIPS_R_AT, b_off); - /* Delay slot */ - emit_instr(ctx, nop); - - /* goto *(prog->bpf_func + skip); */ - off = offsetof(struct bpf_prog, bpf_func); - emit_instr_long(ctx, ld, lw, MIPS_R_T9, off, MIPS_R_AT); - /* All systems are go... decrement and propagate TCC */ - emit_instr_long(ctx, daddiu, addiu, tcc_run, MIPS_R_T8, -1); - /* Skip first instructions (TCC init and R1 fixup) */ - emit_instr_long(ctx, daddiu, addiu, MIPS_R_T9, MIPS_R_T9, ctx->prolog_skip); - return build_int_epilogue(ctx, MIPS_R_T9); -} - -static bool is_bad_offset(int b_off) -{ - return b_off > 0x1ffff || b_off < -0x20000; -} - -/* Returns the number of insn slots consumed. */ -static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, - int this_idx, int exit_idx) -{ - int src, dst, r, td, ts, mem_off, b_off; - bool need_swap, did_move, cmp_eq; - unsigned int target = 0; - u64 t64; - s64 t64s; - int bpf_op = BPF_OP(insn->code); - - if (IS_ENABLED(CONFIG_32BIT) && ((BPF_CLASS(insn->code) == BPF_ALU64) - || (bpf_op == BPF_DW))) - return -EINVAL; - - switch (insn->code) { - case BPF_ALU64 | BPF_ADD | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_SUB | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_OR | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_AND | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_LSH | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_RSH | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_XOR | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_ARSH | BPF_K: /* ALU64_IMM */ - case BPF_ALU64 | BPF_MOV | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_MOV | BPF_K: /* ALU32_IMM */ - case BPF_ALU | BPF_ADD | BPF_K: /* ALU32_IMM */ - case BPF_ALU | BPF_SUB | BPF_K: /* ALU32_IMM */ - case BPF_ALU | BPF_OR | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_AND | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_LSH | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_RSH | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_XOR | BPF_K: /* ALU64_IMM */ - case BPF_ALU | BPF_ARSH | BPF_K: /* ALU64_IMM */ - r = gen_imm_insn(insn, ctx, this_idx); - if (r < 0) - return r; - break; - case BPF_ALU64 | BPF_MUL | BPF_K: /* ALU64_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - if (insn->imm == 1) /* Mult by 1 is a nop */ - break; - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - if (MIPS_ISA_REV >= 6) { - emit_instr(ctx, dmulu, dst, dst, MIPS_R_AT); - } else { - emit_instr(ctx, dmultu, MIPS_R_AT, dst); - emit_instr(ctx, mflo, dst); - } - break; - case BPF_ALU64 | BPF_NEG | BPF_K: /* ALU64_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - emit_instr(ctx, dsubu, dst, MIPS_R_ZERO, dst); - break; - case BPF_ALU | BPF_MUL | BPF_K: /* ALU_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - if (td == REG_64BIT) { - /* sign extend */ - emit_instr(ctx, sll, dst, dst, 0); - } - if (insn->imm == 1) /* Mult by 1 is a nop */ - break; - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - if (MIPS_ISA_REV >= 6) { - emit_instr(ctx, mulu, dst, dst, MIPS_R_AT); - } else { - emit_instr(ctx, multu, dst, MIPS_R_AT); - emit_instr(ctx, mflo, dst); - } - break; - case BPF_ALU | BPF_NEG | BPF_K: /* ALU_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - if (td == REG_64BIT) { - /* sign extend */ - emit_instr(ctx, sll, dst, dst, 0); - } - emit_instr(ctx, subu, dst, MIPS_R_ZERO, dst); - break; - case BPF_ALU | BPF_DIV | BPF_K: /* ALU_IMM */ - case BPF_ALU | BPF_MOD | BPF_K: /* ALU_IMM */ - if (insn->imm == 0) - return -EINVAL; - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - if (td == REG_64BIT) - /* sign extend */ - emit_instr(ctx, sll, dst, dst, 0); - if (insn->imm == 1) { - /* div by 1 is a nop, mod by 1 is zero */ - if (bpf_op == BPF_MOD) - emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO); - break; - } - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - if (MIPS_ISA_REV >= 6) { - if (bpf_op == BPF_DIV) - emit_instr(ctx, divu_r6, dst, dst, MIPS_R_AT); - else - emit_instr(ctx, modu, dst, dst, MIPS_R_AT); - break; - } - emit_instr(ctx, divu, dst, MIPS_R_AT); - if (bpf_op == BPF_DIV) - emit_instr(ctx, mflo, dst); - else - emit_instr(ctx, mfhi, dst); - break; - case BPF_ALU64 | BPF_DIV | BPF_K: /* ALU_IMM */ - case BPF_ALU64 | BPF_MOD | BPF_K: /* ALU_IMM */ - if (insn->imm == 0) - return -EINVAL; - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - if (insn->imm == 1) { - /* div by 1 is a nop, mod by 1 is zero */ - if (bpf_op == BPF_MOD) - emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO); - break; - } - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - if (MIPS_ISA_REV >= 6) { - if (bpf_op == BPF_DIV) - emit_instr(ctx, ddivu_r6, dst, dst, MIPS_R_AT); - else - emit_instr(ctx, dmodu, dst, dst, MIPS_R_AT); - break; - } - emit_instr(ctx, ddivu, dst, MIPS_R_AT); - if (bpf_op == BPF_DIV) - emit_instr(ctx, mflo, dst); - else - emit_instr(ctx, mfhi, dst); - break; - case BPF_ALU64 | BPF_MOV | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_ADD | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_SUB | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_XOR | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_OR | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_AND | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_MUL | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_DIV | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_MOD | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_LSH | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_RSH | BPF_X: /* ALU64_REG */ - case BPF_ALU64 | BPF_ARSH | BPF_X: /* ALU64_REG */ - src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (src < 0 || dst < 0) - return -EINVAL; - if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - did_move = false; - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - int tmp_reg = MIPS_R_AT; - - if (bpf_op == BPF_MOV) { - tmp_reg = dst; - did_move = true; - } - emit_instr(ctx, daddu, tmp_reg, src, MIPS_R_ZERO); - emit_instr(ctx, dinsu, tmp_reg, MIPS_R_ZERO, 32, 32); - src = MIPS_R_AT; - } - switch (bpf_op) { - case BPF_MOV: - if (!did_move) - emit_instr(ctx, daddu, dst, src, MIPS_R_ZERO); - break; - case BPF_ADD: - emit_instr(ctx, daddu, dst, dst, src); - break; - case BPF_SUB: - emit_instr(ctx, dsubu, dst, dst, src); - break; - case BPF_XOR: - emit_instr(ctx, xor, dst, dst, src); - break; - case BPF_OR: - emit_instr(ctx, or, dst, dst, src); - break; - case BPF_AND: - emit_instr(ctx, and, dst, dst, src); - break; - case BPF_MUL: - if (MIPS_ISA_REV >= 6) { - emit_instr(ctx, dmulu, dst, dst, src); - } else { - emit_instr(ctx, dmultu, dst, src); - emit_instr(ctx, mflo, dst); - } - break; - case BPF_DIV: - case BPF_MOD: - if (MIPS_ISA_REV >= 6) { - if (bpf_op == BPF_DIV) - emit_instr(ctx, ddivu_r6, - dst, dst, src); - else - emit_instr(ctx, dmodu, dst, dst, src); - break; - } - emit_instr(ctx, ddivu, dst, src); - if (bpf_op == BPF_DIV) - emit_instr(ctx, mflo, dst); - else - emit_instr(ctx, mfhi, dst); - break; - case BPF_LSH: - emit_instr(ctx, dsllv, dst, dst, src); - break; - case BPF_RSH: - emit_instr(ctx, dsrlv, dst, dst, src); - break; - case BPF_ARSH: - emit_instr(ctx, dsrav, dst, dst, src); - break; - default: - pr_err("ALU64_REG NOT HANDLED\n"); - return -EINVAL; - } - break; - case BPF_ALU | BPF_MOV | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_ADD | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_SUB | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_XOR | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_OR | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_AND | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_MUL | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_DIV | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_MOD | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_LSH | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_RSH | BPF_X: /* ALU_REG */ - case BPF_ALU | BPF_ARSH | BPF_X: /* ALU_REG */ - src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (src < 0 || dst < 0) - return -EINVAL; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - if (td == REG_64BIT) { - /* sign extend */ - emit_instr(ctx, sll, dst, dst, 0); - } - did_move = false; - ts = get_reg_val_type(ctx, this_idx, insn->src_reg); - if (ts == REG_64BIT) { - int tmp_reg = MIPS_R_AT; - - if (bpf_op == BPF_MOV) { - tmp_reg = dst; - did_move = true; - } - /* sign extend */ - emit_instr(ctx, sll, tmp_reg, src, 0); - src = MIPS_R_AT; - } - switch (bpf_op) { - case BPF_MOV: - if (!did_move) - emit_instr(ctx, addu, dst, src, MIPS_R_ZERO); - break; - case BPF_ADD: - emit_instr(ctx, addu, dst, dst, src); - break; - case BPF_SUB: - emit_instr(ctx, subu, dst, dst, src); - break; - case BPF_XOR: - emit_instr(ctx, xor, dst, dst, src); - break; - case BPF_OR: - emit_instr(ctx, or, dst, dst, src); - break; - case BPF_AND: - emit_instr(ctx, and, dst, dst, src); - break; - case BPF_MUL: - emit_instr(ctx, mul, dst, dst, src); - break; - case BPF_DIV: - case BPF_MOD: - if (MIPS_ISA_REV >= 6) { - if (bpf_op == BPF_DIV) - emit_instr(ctx, divu_r6, dst, dst, src); - else - emit_instr(ctx, modu, dst, dst, src); - break; - } - emit_instr(ctx, divu, dst, src); - if (bpf_op == BPF_DIV) - emit_instr(ctx, mflo, dst); - else - emit_instr(ctx, mfhi, dst); - break; - case BPF_LSH: - emit_instr(ctx, sllv, dst, dst, src); - break; - case BPF_RSH: - emit_instr(ctx, srlv, dst, dst, src); - break; - case BPF_ARSH: - emit_instr(ctx, srav, dst, dst, src); - break; - default: - pr_err("ALU_REG NOT HANDLED\n"); - return -EINVAL; - } - break; - case BPF_JMP | BPF_EXIT: - if (this_idx + 1 < exit_idx) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) { - target = j_target(ctx, exit_idx); - if (target == (unsigned int)-1) - return -E2BIG; - emit_instr(ctx, j, target); - } else { - emit_instr(ctx, b, b_off); - } - emit_instr(ctx, nop); - } - break; - case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */ - cmp_eq = (bpf_op == BPF_JEQ); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (dst < 0) - return dst; - if (insn->imm == 0) { - src = MIPS_R_ZERO; - } else { - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - src = MIPS_R_AT; - } - goto jeq_common; - case BPF_JMP | BPF_JEQ | BPF_X: /* JMP_REG */ - case BPF_JMP | BPF_JNE | BPF_X: - case BPF_JMP | BPF_JSLT | BPF_X: - case BPF_JMP | BPF_JSLE | BPF_X: - case BPF_JMP | BPF_JSGT | BPF_X: - case BPF_JMP | BPF_JSGE | BPF_X: - case BPF_JMP | BPF_JLT | BPF_X: - case BPF_JMP | BPF_JLE | BPF_X: - case BPF_JMP | BPF_JGT | BPF_X: - case BPF_JMP | BPF_JGE | BPF_X: - case BPF_JMP | BPF_JSET | BPF_X: - src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (src < 0 || dst < 0) - return -EINVAL; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - ts = get_reg_val_type(ctx, this_idx, insn->src_reg); - if (td == REG_32BIT && ts != REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, src, 0); - src = MIPS_R_AT; - } else if (ts == REG_32BIT && td != REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, dst, 0); - dst = MIPS_R_AT; - } - if (bpf_op == BPF_JSET) { - emit_instr(ctx, and, MIPS_R_AT, dst, src); - cmp_eq = false; - dst = MIPS_R_AT; - src = MIPS_R_ZERO; - } else if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLE) { - emit_instr(ctx, dsubu, MIPS_R_AT, dst, src); - if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - if (bpf_op == BPF_JSGT) - emit_instr(ctx, blez, MIPS_R_AT, b_off); - else - emit_instr(ctx, bgtz, MIPS_R_AT, b_off); - emit_instr(ctx, nop); - return 2; /* We consumed the exit. */ - } - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - if (bpf_op == BPF_JSGT) - emit_instr(ctx, bgtz, MIPS_R_AT, b_off); - else - emit_instr(ctx, blez, MIPS_R_AT, b_off); - emit_instr(ctx, nop); - break; - } else if (bpf_op == BPF_JSGE || bpf_op == BPF_JSLT) { - emit_instr(ctx, slt, MIPS_R_AT, dst, src); - cmp_eq = bpf_op == BPF_JSGE; - dst = MIPS_R_AT; - src = MIPS_R_ZERO; - } else if (bpf_op == BPF_JGT || bpf_op == BPF_JLE) { - /* dst or src could be AT */ - emit_instr(ctx, dsubu, MIPS_R_T8, dst, src); - emit_instr(ctx, sltu, MIPS_R_AT, dst, src); - /* SP known to be non-zero, movz becomes boolean not */ - if (MIPS_ISA_REV >= 6) { - emit_instr(ctx, seleqz, MIPS_R_T9, - MIPS_R_SP, MIPS_R_T8); - } else { - emit_instr(ctx, movz, MIPS_R_T9, - MIPS_R_SP, MIPS_R_T8); - emit_instr(ctx, movn, MIPS_R_T9, - MIPS_R_ZERO, MIPS_R_T8); - } - emit_instr(ctx, or, MIPS_R_AT, MIPS_R_T9, MIPS_R_AT); - cmp_eq = bpf_op == BPF_JGT; - dst = MIPS_R_AT; - src = MIPS_R_ZERO; - } else if (bpf_op == BPF_JGE || bpf_op == BPF_JLT) { - emit_instr(ctx, sltu, MIPS_R_AT, dst, src); - cmp_eq = bpf_op == BPF_JGE; - dst = MIPS_R_AT; - src = MIPS_R_ZERO; - } else { /* JNE/JEQ case */ - cmp_eq = (bpf_op == BPF_JEQ); - } -jeq_common: - /* - * If the next insn is EXIT and we are jumping arround - * only it, invert the sense of the compare and - * conditionally jump to the exit. Poor man's branch - * chaining. - */ - if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) { - target = j_target(ctx, exit_idx); - if (target == (unsigned int)-1) - return -E2BIG; - cmp_eq = !cmp_eq; - b_off = 4 * 3; - if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { - ctx->offsets[this_idx] |= OFFSETS_B_CONV; - ctx->long_b_conversion = 1; - } - } - - if (cmp_eq) - emit_instr(ctx, bne, dst, src, b_off); - else - emit_instr(ctx, beq, dst, src, b_off); - emit_instr(ctx, nop); - if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { - emit_instr(ctx, j, target); - emit_instr(ctx, nop); - } - return 2; /* We consumed the exit. */ - } - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) { - target = j_target(ctx, this_idx + insn->off + 1); - if (target == (unsigned int)-1) - return -E2BIG; - cmp_eq = !cmp_eq; - b_off = 4 * 3; - if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { - ctx->offsets[this_idx] |= OFFSETS_B_CONV; - ctx->long_b_conversion = 1; - } - } - - if (cmp_eq) - emit_instr(ctx, beq, dst, src, b_off); - else - emit_instr(ctx, bne, dst, src, b_off); - emit_instr(ctx, nop); - if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { - emit_instr(ctx, j, target); - emit_instr(ctx, nop); - } - break; - case BPF_JMP | BPF_JSGT | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSGE | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSLT | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSLE | BPF_K: /* JMP_IMM */ - cmp_eq = (bpf_op == BPF_JSGE); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (dst < 0) - return dst; - - if (insn->imm == 0) { - if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - switch (bpf_op) { - case BPF_JSGT: - emit_instr(ctx, blez, dst, b_off); - break; - case BPF_JSGE: - emit_instr(ctx, bltz, dst, b_off); - break; - case BPF_JSLT: - emit_instr(ctx, bgez, dst, b_off); - break; - case BPF_JSLE: - emit_instr(ctx, bgtz, dst, b_off); - break; - } - emit_instr(ctx, nop); - return 2; /* We consumed the exit. */ - } - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - switch (bpf_op) { - case BPF_JSGT: - emit_instr(ctx, bgtz, dst, b_off); - break; - case BPF_JSGE: - emit_instr(ctx, bgez, dst, b_off); - break; - case BPF_JSLT: - emit_instr(ctx, bltz, dst, b_off); - break; - case BPF_JSLE: - emit_instr(ctx, blez, dst, b_off); - break; - } - emit_instr(ctx, nop); - break; - } - /* - * only "LT" compare available, so we must use imm + 1 - * to generate "GT" and imm -1 to generate LE - */ - if (bpf_op == BPF_JSGT) - t64s = insn->imm + 1; - else if (bpf_op == BPF_JSLE) - t64s = insn->imm + 1; - else - t64s = insn->imm; - - cmp_eq = bpf_op == BPF_JSGT || bpf_op == BPF_JSGE; - if (t64s >= S16_MIN && t64s <= S16_MAX) { - emit_instr(ctx, slti, MIPS_R_AT, dst, (int)t64s); - src = MIPS_R_AT; - dst = MIPS_R_ZERO; - goto jeq_common; - } - emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); - emit_instr(ctx, slt, MIPS_R_AT, dst, MIPS_R_AT); - src = MIPS_R_AT; - dst = MIPS_R_ZERO; - goto jeq_common; - - case BPF_JMP | BPF_JGT | BPF_K: - case BPF_JMP | BPF_JGE | BPF_K: - case BPF_JMP | BPF_JLT | BPF_K: - case BPF_JMP | BPF_JLE | BPF_K: - cmp_eq = (bpf_op == BPF_JGE); - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (dst < 0) - return dst; - /* - * only "LT" compare available, so we must use imm + 1 - * to generate "GT" and imm -1 to generate LE - */ - if (bpf_op == BPF_JGT) - t64s = (u64)(u32)(insn->imm) + 1; - else if (bpf_op == BPF_JLE) - t64s = (u64)(u32)(insn->imm) + 1; - else - t64s = (u64)(u32)(insn->imm); - - cmp_eq = bpf_op == BPF_JGT || bpf_op == BPF_JGE; - - emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); - emit_instr(ctx, sltu, MIPS_R_AT, dst, MIPS_R_AT); - src = MIPS_R_AT; - dst = MIPS_R_ZERO; - goto jeq_common; - - case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */ - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (dst < 0) - return dst; - - if (ctx->use_bbit_insns && hweight32((u32)insn->imm) == 1) { - if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - emit_instr(ctx, bbit0, dst, ffs((u32)insn->imm) - 1, b_off); - emit_instr(ctx, nop); - return 2; /* We consumed the exit. */ - } - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - emit_instr(ctx, bbit1, dst, ffs((u32)insn->imm) - 1, b_off); - emit_instr(ctx, nop); - break; - } - t64 = (u32)insn->imm; - emit_const_to_reg(ctx, MIPS_R_AT, t64); - emit_instr(ctx, and, MIPS_R_AT, dst, MIPS_R_AT); - src = MIPS_R_AT; - dst = MIPS_R_ZERO; - cmp_eq = false; - goto jeq_common; - - case BPF_JMP | BPF_JA: - /* - * Prefer relative branch for easier debugging, but - * fall back if needed. - */ - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) { - target = j_target(ctx, this_idx + insn->off + 1); - if (target == (unsigned int)-1) - return -E2BIG; - emit_instr(ctx, j, target); - } else { - emit_instr(ctx, b, b_off); - } - emit_instr(ctx, nop); - break; - case BPF_LD | BPF_DW | BPF_IMM: - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - t64 = ((u64)(u32)insn->imm) | ((u64)(insn + 1)->imm << 32); - emit_const_to_reg(ctx, dst, t64); - return 2; /* Double slot insn */ - - case BPF_JMP | BPF_CALL: - emit_bpf_call(ctx, insn); - break; - - case BPF_JMP | BPF_TAIL_CALL: - if (emit_bpf_tail_call(ctx, this_idx)) - return -EINVAL; - break; - - case BPF_ALU | BPF_END | BPF_FROM_BE: - case BPF_ALU | BPF_END | BPF_FROM_LE: - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - if (dst < 0) - return dst; - td = get_reg_val_type(ctx, this_idx, insn->dst_reg); - if (insn->imm == 64 && td == REG_32BIT) - emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); - - if (insn->imm != 64 && td == REG_64BIT) { - /* sign extend */ - emit_instr(ctx, sll, dst, dst, 0); - } - -#ifdef __BIG_ENDIAN - need_swap = (BPF_SRC(insn->code) == BPF_FROM_LE); -#else - need_swap = (BPF_SRC(insn->code) == BPF_FROM_BE); -#endif - if (insn->imm == 16) { - if (need_swap) - emit_instr(ctx, wsbh, dst, dst); - emit_instr(ctx, andi, dst, dst, 0xffff); - } else if (insn->imm == 32) { - if (need_swap) { - emit_instr(ctx, wsbh, dst, dst); - emit_instr(ctx, rotr, dst, dst, 16); - } - } else { /* 64-bit*/ - if (need_swap) { - emit_instr(ctx, dsbh, dst, dst); - emit_instr(ctx, dshd, dst, dst); - } - } - break; - - case BPF_ST | BPF_NOSPEC: /* speculation barrier */ - break; - - case BPF_ST | BPF_B | BPF_MEM: - case BPF_ST | BPF_H | BPF_MEM: - case BPF_ST | BPF_W | BPF_MEM: - case BPF_ST | BPF_DW | BPF_MEM: - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - if (dst < 0) - return dst; - mem_off = insn->off; - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - switch (BPF_SIZE(insn->code)) { - case BPF_B: - emit_instr(ctx, sb, MIPS_R_AT, mem_off, dst); - break; - case BPF_H: - emit_instr(ctx, sh, MIPS_R_AT, mem_off, dst); - break; - case BPF_W: - emit_instr(ctx, sw, MIPS_R_AT, mem_off, dst); - break; - case BPF_DW: - emit_instr(ctx, sd, MIPS_R_AT, mem_off, dst); - break; - } - break; - - case BPF_LDX | BPF_B | BPF_MEM: - case BPF_LDX | BPF_H | BPF_MEM: - case BPF_LDX | BPF_W | BPF_MEM: - case BPF_LDX | BPF_DW | BPF_MEM: - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); - src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); - if (dst < 0 || src < 0) - return -EINVAL; - mem_off = insn->off; - switch (BPF_SIZE(insn->code)) { - case BPF_B: - emit_instr(ctx, lbu, dst, mem_off, src); - break; - case BPF_H: - emit_instr(ctx, lhu, dst, mem_off, src); - break; - case BPF_W: - emit_instr(ctx, lw, dst, mem_off, src); - break; - case BPF_DW: - emit_instr(ctx, ld, dst, mem_off, src); - break; - } - break; - - case BPF_STX | BPF_B | BPF_MEM: - case BPF_STX | BPF_H | BPF_MEM: - case BPF_STX | BPF_W | BPF_MEM: - case BPF_STX | BPF_DW | BPF_MEM: - case BPF_STX | BPF_W | BPF_ATOMIC: - case BPF_STX | BPF_DW | BPF_ATOMIC: - dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); - src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); - if (src < 0 || dst < 0) - return -EINVAL; - mem_off = insn->off; - if (BPF_MODE(insn->code) == BPF_ATOMIC) { - if (insn->imm != BPF_ADD) { - pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); - return -EINVAL; - } - /* - * If mem_off does not fit within the 9 bit ll/sc - * instruction immediate field, use a temp reg. - */ - if (MIPS_ISA_REV >= 6 && - (mem_off >= BIT(8) || mem_off < -BIT(8))) { - emit_instr(ctx, daddiu, MIPS_R_T6, - dst, mem_off); - mem_off = 0; - dst = MIPS_R_T6; - } - switch (BPF_SIZE(insn->code)) { - case BPF_W: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, src, 0); - src = MIPS_R_AT; - } - emit_instr(ctx, ll, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, addu, MIPS_R_T8, MIPS_R_T8, src); - emit_instr(ctx, sc, MIPS_R_T8, mem_off, dst); - /* - * On failure back up to LL (-4 - * instructions of 4 bytes each - */ - emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); - emit_instr(ctx, nop); - break; - case BPF_DW: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); - emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); - src = MIPS_R_AT; - } - emit_instr(ctx, lld, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, src); - emit_instr(ctx, scd, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); - emit_instr(ctx, nop); - break; - } - } else { /* BPF_MEM */ - switch (BPF_SIZE(insn->code)) { - case BPF_B: - emit_instr(ctx, sb, src, mem_off, dst); - break; - case BPF_H: - emit_instr(ctx, sh, src, mem_off, dst); - break; - case BPF_W: - emit_instr(ctx, sw, src, mem_off, dst); - break; - case BPF_DW: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); - emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); - src = MIPS_R_AT; - } - emit_instr(ctx, sd, src, mem_off, dst); - break; - } - } - break; - - default: - pr_err("NOT HANDLED %d - (%02x)\n", - this_idx, (unsigned int)insn->code); - return -EINVAL; - } - return 1; -} - -#define RVT_VISITED_MASK 0xc000000000000000ull -#define RVT_FALL_THROUGH 0x4000000000000000ull -#define RVT_BRANCH_TAKEN 0x8000000000000000ull -#define RVT_DONE (RVT_FALL_THROUGH | RVT_BRANCH_TAKEN) - -static int build_int_body(struct jit_ctx *ctx) -{ - const struct bpf_prog *prog = ctx->prog; - const struct bpf_insn *insn; - int i, r; - - for (i = 0; i < prog->len; ) { - insn = prog->insnsi + i; - if ((ctx->reg_val_types[i] & RVT_VISITED_MASK) == 0) { - /* dead instruction, don't emit it. */ - i++; - continue; - } - - if (ctx->target == NULL) - ctx->offsets[i] = (ctx->offsets[i] & OFFSETS_B_CONV) | (ctx->idx * 4); - - r = build_one_insn(insn, ctx, i, prog->len); - if (r < 0) - return r; - i += r; - } - /* epilogue offset */ - if (ctx->target == NULL) - ctx->offsets[i] = ctx->idx * 4; - - /* - * All exits have an offset of the epilogue, some offsets may - * not have been set due to banch-around threading, so set - * them now. - */ - if (ctx->target == NULL) - for (i = 0; i < prog->len; i++) { - insn = prog->insnsi + i; - if (insn->code == (BPF_JMP | BPF_EXIT)) - ctx->offsets[i] = ctx->idx * 4; - } - return 0; -} - -/* return the last idx processed, or negative for error */ -static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, - int start_idx, bool follow_taken) -{ - const struct bpf_prog *prog = ctx->prog; - const struct bpf_insn *insn; - u64 exit_rvt = initial_rvt; - u64 *rvt = ctx->reg_val_types; - int idx; - int reg; - - for (idx = start_idx; idx < prog->len; idx++) { - rvt[idx] = (rvt[idx] & RVT_VISITED_MASK) | exit_rvt; - insn = prog->insnsi + idx; - switch (BPF_CLASS(insn->code)) { - case BPF_ALU: - switch (BPF_OP(insn->code)) { - case BPF_ADD: - case BPF_SUB: - case BPF_MUL: - case BPF_DIV: - case BPF_OR: - case BPF_AND: - case BPF_LSH: - case BPF_RSH: - case BPF_ARSH: - case BPF_NEG: - case BPF_MOD: - case BPF_XOR: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - break; - case BPF_MOV: - if (BPF_SRC(insn->code)) { - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - } else { - /* IMM to REG move*/ - if (insn->imm >= 0) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - else - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - } - break; - case BPF_END: - if (insn->imm == 64) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - else if (insn->imm == 32) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - else /* insn->imm == 16 */ - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - break; - } - rvt[idx] |= RVT_DONE; - break; - case BPF_ALU64: - switch (BPF_OP(insn->code)) { - case BPF_MOV: - if (BPF_SRC(insn->code)) { - /* REG to REG move*/ - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - } else { - /* IMM to REG move*/ - if (insn->imm >= 0) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - else - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT); - } - break; - default: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - } - rvt[idx] |= RVT_DONE; - break; - case BPF_LD: - switch (BPF_SIZE(insn->code)) { - case BPF_DW: - if (BPF_MODE(insn->code) == BPF_IMM) { - s64 val; - - val = (s64)((u32)insn->imm | ((u64)(insn + 1)->imm << 32)); - if (val > 0 && val <= S32_MAX) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - else if (val >= S32_MIN && val <= S32_MAX) - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT); - else - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - rvt[idx] |= RVT_DONE; - idx++; - } else { - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - } - break; - case BPF_B: - case BPF_H: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - break; - case BPF_W: - if (BPF_MODE(insn->code) == BPF_IMM) - set_reg_val_type(&exit_rvt, insn->dst_reg, - insn->imm >= 0 ? REG_32BIT_POS : REG_32BIT); - else - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - break; - } - rvt[idx] |= RVT_DONE; - break; - case BPF_LDX: - switch (BPF_SIZE(insn->code)) { - case BPF_DW: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); - break; - case BPF_B: - case BPF_H: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); - break; - case BPF_W: - set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); - break; - } - rvt[idx] |= RVT_DONE; - break; - case BPF_JMP: - case BPF_JMP32: - switch (BPF_OP(insn->code)) { - case BPF_EXIT: - rvt[idx] = RVT_DONE | exit_rvt; - rvt[prog->len] = exit_rvt; - return idx; - case BPF_JA: - { - int tgt = idx + 1 + insn->off; - bool visited = (rvt[tgt] & RVT_FALL_THROUGH); - - rvt[idx] |= RVT_DONE; - /* - * Verifier dead code patching can use - * infinite-loop traps, causing hangs and - * RCU stalls here. Treat traps as nops - * if detected and fall through. - */ - if (insn->off == -1) - break; - /* - * Bounded loops cause the same issues in - * fallthrough mode; follow only if jump - * target is unvisited to mitigate. - */ - if (insn->off < 0 && !follow_taken && visited) - break; - idx += insn->off; - break; - } - case BPF_JEQ: - case BPF_JGT: - case BPF_JGE: - case BPF_JLT: - case BPF_JLE: - case BPF_JSET: - case BPF_JNE: - case BPF_JSGT: - case BPF_JSGE: - case BPF_JSLT: - case BPF_JSLE: - if (follow_taken) { - rvt[idx] |= RVT_BRANCH_TAKEN; - idx += insn->off; - follow_taken = false; - } else { - rvt[idx] |= RVT_FALL_THROUGH; - } - break; - case BPF_CALL: - set_reg_val_type(&exit_rvt, BPF_REG_0, REG_64BIT); - /* Upon call return, argument registers are clobbered. */ - for (reg = BPF_REG_0; reg <= BPF_REG_5; reg++) - set_reg_val_type(&exit_rvt, reg, REG_64BIT); - - rvt[idx] |= RVT_DONE; - break; - case BPF_TAIL_CALL: - rvt[idx] |= RVT_DONE; - break; - default: - WARN(1, "Unhandled BPF_JMP case.\n"); - rvt[idx] |= RVT_DONE; - break; - } - break; - default: - rvt[idx] |= RVT_DONE; - break; - } - } - return idx; -} - -/* - * Track the value range (i.e. 32-bit vs. 64-bit) of each register at - * each eBPF insn. This allows unneeded sign and zero extension - * operations to be omitted. - * - * Doesn't handle yet confluence of control paths with conflicting - * ranges, but it is good enough for most sane code. - */ -static int reg_val_propagate(struct jit_ctx *ctx) -{ - const struct bpf_prog *prog = ctx->prog; - u64 exit_rvt; - int reg; - int i; - - /* - * 11 registers * 3 bits/reg leaves top bits free for other - * uses. Bit-62..63 used to see if we have visited an insn. - */ - exit_rvt = 0; - - /* Upon entry, argument registers are 64-bit. */ - for (reg = BPF_REG_1; reg <= BPF_REG_5; reg++) - set_reg_val_type(&exit_rvt, reg, REG_64BIT); - - /* - * First follow all conditional branches on the fall-through - * edge of control flow.. - */ - reg_val_propagate_range(ctx, exit_rvt, 0, false); -restart_search: - /* - * Then repeatedly find the first conditional branch where - * both edges of control flow have not been taken, and follow - * the branch taken edge. We will end up restarting the - * search once per conditional branch insn. - */ - for (i = 0; i < prog->len; i++) { - u64 rvt = ctx->reg_val_types[i]; - - if ((rvt & RVT_VISITED_MASK) == RVT_DONE || - (rvt & RVT_VISITED_MASK) == 0) - continue; - if ((rvt & RVT_VISITED_MASK) == RVT_FALL_THROUGH) { - reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, true); - } else { /* RVT_BRANCH_TAKEN */ - WARN(1, "Unexpected RVT_BRANCH_TAKEN case.\n"); - reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, false); - } - goto restart_search; - } - /* - * Eventually all conditional branches have been followed on - * both branches and we are done. Any insn that has not been - * visited at this point is dead. - */ - - return 0; -} - -static void jit_fill_hole(void *area, unsigned int size) -{ - u32 *p; - - /* We are guaranteed to have aligned memory. */ - for (p = area; size >= sizeof(u32); size -= sizeof(u32)) - uasm_i_break(&p, BRK_BUG); /* Increments p */ -} - -/* - * Save and restore the BPF VM state across a direct kernel call. This - * includes the caller-saved registers used for BPF_REG_0 .. BPF_REG_5 - * and BPF_REG_AX used by the verifier for blinding and other dark arts. - * Restore avoids clobbering bpf_ret, which holds the call return value. - * BPF_REG_6 .. BPF_REG_10 and TCC are already callee-saved or on stack. - */ -static const int bpf_caller_save[] = { - BPF_REG_0, - BPF_REG_1, - BPF_REG_2, - BPF_REG_3, - BPF_REG_4, - BPF_REG_5, - BPF_REG_AX, -}; - -#define CALLER_ENV_SIZE (ARRAY_SIZE(bpf_caller_save) * sizeof(u64)) - -void emit_caller_save(struct jit_ctx *ctx) -{ - int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); - int i, bpf, reg, store_offset; - - emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adj); - - for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { - bpf = bpf_caller_save[i]; - reg = bpf2mips[bpf].reg; - store_offset = i * sizeof(u64); - - if (is64bit()) { - emit_instr(ctx, sd, reg, store_offset, MIPS_R_SP); - } else { - emit_instr(ctx, sw, LO(reg), - OFFLO(store_offset), MIPS_R_SP); - emit_instr(ctx, sw, HI(reg), - OFFHI(store_offset), MIPS_R_SP); - } - } -} - -void emit_caller_restore(struct jit_ctx *ctx, int bpf_ret) -{ - int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); - int i, bpf, reg, store_offset; - - for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { - bpf = bpf_caller_save[i]; - reg = bpf2mips[bpf].reg; - store_offset = i * sizeof(u64); - if (bpf == bpf_ret) - continue; - - if (is64bit()) { - emit_instr(ctx, ld, reg, store_offset, MIPS_R_SP); - } else { - emit_instr(ctx, lw, LO(reg), - OFFLO(store_offset), MIPS_R_SP); - emit_instr(ctx, lw, HI(reg), - OFFHI(store_offset), MIPS_R_SP); - } - } - - emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, stack_adj); -} - -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) -{ - bool tmp_blinded = false, extra_pass = false; - struct bpf_prog *tmp, *orig_prog = prog; - struct bpf_binary_header *header = NULL; - unsigned int image_size, pass = 3; - struct jit_ctx *ctx; - - if (!prog->jit_requested) - return orig_prog; - - /* Attempt blinding but fall back to the interpreter on failure. */ - tmp = bpf_jit_blind_constants(prog); - if (IS_ERR(tmp)) - return orig_prog; - if (tmp != prog) { - tmp_blinded = true; - prog = tmp; - } - - ctx = prog->aux->jit_data; - if (!ctx) { - ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); - if (!ctx) { - prog = orig_prog; - goto out; - } - } - - /* - * Assume extra pass needed for patching addresses if previous - * ctx exists in saved jit_data, so skip to code generation. - */ - if (ctx->offsets) { - extra_pass = true; - pass++; - image_size = 4 * ctx->idx; - header = bpf_jit_binary_hdr(ctx->prog); - goto skip_init_ctx; - } - - ctx->prog = prog; - ctx->offsets = kcalloc(prog->len + 1, - sizeof(*ctx->offsets), - GFP_KERNEL); - if (!ctx->offsets) - goto out_err; - - /* Check Octeon bbit ops only for MIPS64. */ - if (is64bit()) { - preempt_disable(); - switch (current_cpu_type()) { - case CPU_CAVIUM_OCTEON: - case CPU_CAVIUM_OCTEON_PLUS: - case CPU_CAVIUM_OCTEON2: - case CPU_CAVIUM_OCTEON3: - ctx->use_bbit_insns = 1; - break; - default: - ctx->use_bbit_insns = 0; - } - preempt_enable(); - } - - ctx->reg_val_types = kcalloc(prog->len + 1, - sizeof(*ctx->reg_val_types), - GFP_KERNEL); - if (!ctx->reg_val_types) - goto out_err; - - if (reg_val_propagate(ctx)) - goto out_err; - - /* - * First pass discovers used resources and instruction offsets - * assuming short branches are used. - */ - if (build_int_body(ctx)) - goto out_err; - - /* - * If no calls are made (EBPF_SAVE_RA), then tailcall count located - * in runtime reg if defined, else we backup to save reg or stack. - */ - if (tail_call_present(ctx)) { - if (ctx->flags & EBPF_SAVE_RA) - ctx->flags |= bpf2mips[JIT_SAV_TCC].flags; - else if (bpf2mips[JIT_RUN_TCC].reg) - ctx->flags |= EBPF_TCC_IN_RUN; - } - - /* - * Second pass generates offsets, if any branches are out of - * range a jump-around long sequence is generated, and we have - * to try again from the beginning to generate the new - * offsets. This is done until no additional conversions are - * necessary. - */ - do { - ctx->idx = 0; - ctx->gen_b_offsets = 1; - ctx->long_b_conversion = 0; - if (build_int_prologue(ctx)) - goto out_err; - if (build_int_body(ctx)) - goto out_err; - if (build_int_epilogue(ctx, MIPS_R_RA)) - goto out_err; - } while (ctx->long_b_conversion); - - image_size = 4 * ctx->idx; - - header = bpf_jit_binary_alloc(image_size, (void *)&ctx->target, - sizeof(u32), jit_fill_hole); - if (!header) - goto out_err; - -skip_init_ctx: - - /* Third pass generates the code (fourth patches call addresses) */ - ctx->idx = 0; - if (build_int_prologue(ctx)) - goto out_err; - if (build_int_body(ctx)) - goto out_err; - if (build_int_epilogue(ctx, MIPS_R_RA)) - goto out_err; - - if (bpf_jit_enable > 1) - /* Dump JIT code */ - bpf_jit_dump(prog->len, image_size, pass, ctx->target); - - /* Update the icache */ - flush_icache_range((unsigned long)ctx->target, - (unsigned long)&ctx->target[ctx->idx]); - - if (!prog->is_func || extra_pass) - bpf_jit_binary_lock_ro(header); - else - prog->aux->jit_data = ctx; - - prog->bpf_func = (void *)ctx->target; - prog->jited = 1; - prog->jited_len = image_size; - - if (!prog->is_func || extra_pass) { - bpf_prog_fill_jited_linfo(prog, ctx->offsets + 1); -out_ctx: - kfree(ctx->offsets); - kfree(ctx->reg_val_types); - kfree(ctx); - prog->aux->jit_data = NULL; - } -out: - if (tmp_blinded) - bpf_jit_prog_release_other(prog, prog == orig_prog ? - tmp : orig_prog); - return prog; - -out_err: - prog = orig_prog; - if (header) - bpf_jit_binary_free(header); - goto out_ctx; -} - -/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ -bool bpf_jit_supports_subprog_tailcalls(void) -{ - return true; -} diff --git a/arch/mips/net/ebpf_jit.h b/arch/mips/net/ebpf_jit.h new file mode 100644 index 000000000000..82227e16e503 --- /dev/null +++ b/arch/mips/net/ebpf_jit.h @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 + * Copyright (c) 2021 Tony Ambardar + * + * Based on code from: + * + * Copyright (c) 2017 Cavium, Inc. + * Author: David Daney + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras + */ + +#ifndef _EBPF_JIT_H +#define _EBPF_JIT_H + +#include +#include +#include +#include + +/* Registers used by JIT: (MIPS32) (MIPS64) */ +#define MIPS_R_ZERO 0 +#define MIPS_R_AT 1 +#define MIPS_R_V0 2 /* BPF_R0 BPF_R0 */ +#define MIPS_R_V1 3 /* BPF_R0 BPF_TCC */ +#define MIPS_R_A0 4 /* BPF_R1 BPF_R1 */ +#define MIPS_R_A1 5 /* BPF_R1 BPF_R2 */ +#define MIPS_R_A2 6 /* BPF_R2 BPF_R3 */ +#define MIPS_R_A3 7 /* BPF_R2 BPF_R4 */ +/* MIPS64 swaps T0-T3 regs for extra args A4-A7. */ +#ifdef CONFIG_64BIT +# define MIPS_R_A4 8 /* (n/a) BPF_R5 */ +#else /* CONFIG_32BIT */ +# define MIPS_R_T0 8 /* BPF_R3 (n/a) */ +# define MIPS_R_T1 9 /* BPF_R3 (n/a) */ +# define MIPS_R_T2 10 /* BPF_R4 (n/a) */ +# define MIPS_R_T3 11 /* BPF_R4 (n/a) */ +#endif +#define MIPS_R_T4 12 /* BPF_R5 BPF_AX */ +#define MIPS_R_T5 13 /* BPF_R5 (free) */ +#define MIPS_R_T6 14 /* BPF_AX (used) */ +#define MIPS_R_T7 15 /* BPF_AX (free) */ +#define MIPS_R_S0 16 /* BPF_R6 BPF_R6 */ +#define MIPS_R_S1 17 /* BPF_R6 BPF_R7 */ +#define MIPS_R_S2 18 /* BPF_R7 BPF_R8 */ +#define MIPS_R_S3 19 /* BPF_R7 BPF_R9 */ +#define MIPS_R_S4 20 /* BPF_R8 BPF_TCC */ +#define MIPS_R_S5 21 /* BPF_R8 (free) */ +#define MIPS_R_S6 22 /* BPF_R9 (free) */ +#define MIPS_R_S7 23 /* BPF_R9 (free) */ +#define MIPS_R_T8 24 /* (used) (used) */ +#define MIPS_R_T9 25 /* (used) (used) */ +#define MIPS_R_SP 29 +#define MIPS_R_S8 30 /* BPF_R10 BPF_R10 */ +#define MIPS_R_RA 31 + +/* eBPF flags */ +#define EBPF_SAVE_S0 BIT(0) +#define EBPF_SAVE_S1 BIT(1) +#define EBPF_SAVE_S2 BIT(2) +#define EBPF_SAVE_S3 BIT(3) +#define EBPF_SAVE_S4 BIT(4) +#define EBPF_SAVE_S5 BIT(5) +#define EBPF_SAVE_S6 BIT(6) +#define EBPF_SAVE_S7 BIT(7) +#define EBPF_SAVE_S8 BIT(8) +#define EBPF_SAVE_RA BIT(9) +#define EBPF_SEEN_FP BIT(10) +#define EBPF_SEEN_TC BIT(11) +#define EBPF_TCC_IN_RUN BIT(12) + +/* + * Word-size and endianness-aware helpers for building MIPS32 vs MIPS64 + * tables and selecting 32-bit subregisters from a register pair base. + * Simplify use by emulating MIPS_R_SP and MIPS_R_ZERO as register pairs + * and adding HI/LO word memory offsets. + */ +#ifdef CONFIG_64BIT +# define HI(reg) (reg) +# define LO(reg) (reg) +# define OFFHI(mem) (mem) +# define OFFLO(mem) (mem) +#else /* CONFIG_32BIT */ +# ifdef __BIG_ENDIAN +# define HI(reg) ((reg) == MIPS_R_SP ? MIPS_R_ZERO : \ + (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ + (reg)) +# define LO(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ + (reg) == MIPS_R_SP ? (reg) : \ + (reg) == MIPS_R_S8 ? (reg) : \ + (reg) + 1) +# define OFFHI(mem) (mem) +# define OFFLO(mem) ((mem) + sizeof(long)) +# else /* __LITTLE_ENDIAN */ +# define HI(reg) ((reg) == MIPS_R_ZERO ? (reg) : \ + (reg) == MIPS_R_SP ? MIPS_R_ZERO : \ + (reg) == MIPS_R_S8 ? MIPS_R_ZERO : \ + (reg) + 1) +# define LO(reg) (reg) +# define OFFHI(mem) ((mem) + sizeof(long)) +# define OFFLO(mem) (mem) +# endif +#endif + +static inline bool is64bit(void) +{ + return IS_ENABLED(CONFIG_64BIT); +} + +static inline bool isbigend(void) +{ + return IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); +} + +/* + * For the mips64 ISA, we need to track the value range or type for + * each JIT register. The BPF machine requires zero extended 32-bit + * values, but the mips64 ISA requires sign extended 32-bit values. + * At each point in the BPF program we track the state of every + * register so that we can zero extend or sign extend as the BPF + * semantics require. + */ +enum reg_val_type { + /* uninitialized */ + REG_UNKNOWN, + /* not known to be 32-bit compatible. */ + REG_64BIT, + /* 32-bit compatible, no truncation needed for 64-bit ops. */ + REG_64BIT_32BIT, + /* 32-bit compatible, need truncation for 64-bit ops. */ + REG_32BIT, + /* 32-bit no sign/zero extension needed. */ + REG_32BIT_POS +}; + +/** + * struct jit_ctx - JIT context + * @prog: The program + * @stack_size: eBPF stack size + * @bpf_stack_off: eBPF FP offset + * @idx: Instruction index + * @flags: JIT flags + * @offsets: Instruction offsets + * @target: Memory location for compiled instructions + * @reg_val_types Packed enum reg_val_type for each register + */ +struct jit_ctx { + const struct bpf_prog *prog; + int stack_size; + int bpf_stack_off; + int prolog_skip; + u32 idx; + u32 flags; + u32 *offsets; + u32 *target; + u64 *reg_val_types; + unsigned int long_b_conversion:1; + unsigned int gen_b_offsets:1; + unsigned int use_bbit_insns:1; +}; + +static inline void set_reg_val_type(u64 *rvt, int reg, enum reg_val_type type) +{ + *rvt &= ~(7ull << (reg * 3)); + *rvt |= ((u64)type << (reg * 3)); +} + +static inline enum reg_val_type get_reg_val_type(const struct jit_ctx *ctx, + int index, int reg) +{ + return (ctx->reg_val_types[index] >> (reg * 3)) & 7; +} + +/* Simply emit the instruction if the JIT memory space has been allocated */ +#define emit_instr_long(ctx, func64, func32, ...) \ +do { \ + if ((ctx)->target != NULL) { \ + u32 *p = &(ctx)->target[ctx->idx]; \ + if (IS_ENABLED(CONFIG_64BIT)) \ + uasm_i_##func64(&p, ##__VA_ARGS__); \ + else \ + uasm_i_##func32(&p, ##__VA_ARGS__); \ + } \ + (ctx)->idx++; \ +} while (0) + +#define emit_instr(ctx, func, ...) \ + emit_instr_long(ctx, func, func, ##__VA_ARGS__) + +/* + * High bit of offsets indicates if long branch conversion done at + * this insn. + */ +#define OFFSETS_B_CONV BIT(31) + +static inline unsigned int j_target(struct jit_ctx *ctx, int target_idx) +{ + unsigned long target_va, base_va; + unsigned int r; + + if (!ctx->target) + return 0; + + base_va = (unsigned long)ctx->target; + target_va = base_va + (ctx->offsets[target_idx] & ~OFFSETS_B_CONV); + + if ((base_va & ~0x0ffffffful) != (target_va & ~0x0ffffffful)) + return (unsigned int)-1; + r = target_va & 0x0ffffffful; + return r; +} + +/* Compute the immediate value for PC-relative branches. */ +static inline u32 b_imm(unsigned int tgt, struct jit_ctx *ctx) +{ + if (!ctx->gen_b_offsets) + return 0; + + /* + * We want a pc-relative branch. tgt is the instruction offset + * we want to jump to. + + * Branch on MIPS: + * I: target_offset <- sign_extend(offset) + * I+1: PC += target_offset (delay slot) + * + * ctx->idx currently points to the branch instruction + * but the offset is added to the delay slot so we need + * to subtract 4. + */ + return (ctx->offsets[tgt] & ~OFFSETS_B_CONV) - + (ctx->idx * 4) - 4; +} + +static inline bool tail_call_present(struct jit_ctx *ctx) +{ + return ctx->flags & EBPF_SEEN_TC || ctx->prog->aux->tail_call_reachable; +} + +static inline bool is_bad_offset(int b_off) +{ + return b_off > 0x1ffff || b_off < -0x20000; +} + +/* Sign-extend dst register or HI 32-bit reg of pair. */ +static inline void gen_sext_insn(int dst, struct jit_ctx *ctx) +{ + if (is64bit()) + emit_instr(ctx, sll, dst, dst, 0); + else + emit_instr(ctx, sra, HI(dst), LO(dst), 31); +} + +/* + * Zero-extend dst register or HI 32-bit reg of pair, if either forced + * or the BPF verifier does not insert its own zext insns. + */ +static inline void gen_zext_insn(int dst, bool force, struct jit_ctx *ctx) +{ + if (!ctx->prog->aux->verifier_zext || force) { + if (is64bit()) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + else + emit_instr(ctx, and, HI(dst), MIPS_R_ZERO, MIPS_R_ZERO); + } +} + +enum reg_usage { + REG_SRC_FP_OK, + REG_SRC_NO_FP, + REG_DST_FP_OK, + REG_DST_NO_FP +}; + +extern int ebpf_to_mips_reg(struct jit_ctx *ctx, + const struct bpf_insn *insn, + enum reg_usage u); + +extern void gen_imm_to_reg(const struct bpf_insn *insn, int reg, + struct jit_ctx *ctx); + +extern void emit_const_to_reg(struct jit_ctx *ctx, int dst, unsigned long value); + +extern void emit_bpf_call(struct jit_ctx *ctx, const struct bpf_insn *insn); + +extern int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx); + +extern void emit_caller_save(struct jit_ctx *ctx); + +extern void emit_caller_restore(struct jit_ctx *ctx, int bpf_ret); + +extern int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, + int this_idx, int exit_idx); + +#endif /* _EBPF_JIT_H */ diff --git a/arch/mips/net/ebpf_jit_comp64.c b/arch/mips/net/ebpf_jit_comp64.c new file mode 100644 index 000000000000..c38d93d37ce3 --- /dev/null +++ b/arch/mips/net/ebpf_jit_comp64.c @@ -0,0 +1,990 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 + * Copyright (c) 2021 Tony Ambardar + * + * Based on code from: + * + * Copyright (c) 2017 Cavium, Inc. + * Author: David Daney + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras + */ + +#include +#include +#include + +#include "ebpf_jit.h" + +static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, + int idx) +{ + int upper_bound, lower_bound; + int dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + + if (dst < 0) + return dst; + + switch (BPF_OP(insn->code)) { + case BPF_MOV: + case BPF_ADD: + upper_bound = S16_MAX; + lower_bound = S16_MIN; + break; + case BPF_SUB: + upper_bound = -(int)S16_MIN; + lower_bound = -(int)S16_MAX; + break; + case BPF_AND: + case BPF_OR: + case BPF_XOR: + upper_bound = 0xffff; + lower_bound = 0; + break; + case BPF_RSH: + case BPF_LSH: + case BPF_ARSH: + /* Shift amounts are truncated, no need for bounds */ + upper_bound = S32_MAX; + lower_bound = S32_MIN; + break; + default: + return -EINVAL; + } + + /* + * Immediate move clobbers the register, so no sign/zero + * extension needed. + */ + if (BPF_CLASS(insn->code) == BPF_ALU64 && + BPF_OP(insn->code) != BPF_MOV && + get_reg_val_type(ctx, idx, insn->dst_reg) == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + /* BPF_ALU | BPF_LSH doesn't need separate sign extension */ + if (BPF_CLASS(insn->code) == BPF_ALU && + BPF_OP(insn->code) != BPF_LSH && + BPF_OP(insn->code) != BPF_MOV && + get_reg_val_type(ctx, idx, insn->dst_reg) != REG_32BIT) + emit_instr(ctx, sll, dst, dst, 0); + + if (insn->imm >= lower_bound && insn->imm <= upper_bound) { + /* single insn immediate case */ + switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { + case BPF_ALU64 | BPF_MOV: + emit_instr(ctx, daddiu, dst, MIPS_R_ZERO, insn->imm); + break; + case BPF_ALU64 | BPF_AND: + case BPF_ALU | BPF_AND: + emit_instr(ctx, andi, dst, dst, insn->imm); + break; + case BPF_ALU64 | BPF_OR: + case BPF_ALU | BPF_OR: + emit_instr(ctx, ori, dst, dst, insn->imm); + break; + case BPF_ALU64 | BPF_XOR: + case BPF_ALU | BPF_XOR: + emit_instr(ctx, xori, dst, dst, insn->imm); + break; + case BPF_ALU64 | BPF_ADD: + emit_instr(ctx, daddiu, dst, dst, insn->imm); + break; + case BPF_ALU64 | BPF_SUB: + emit_instr(ctx, daddiu, dst, dst, -insn->imm); + break; + case BPF_ALU64 | BPF_RSH: + emit_instr(ctx, dsrl_safe, dst, dst, insn->imm & 0x3f); + break; + case BPF_ALU | BPF_RSH: + emit_instr(ctx, srl, dst, dst, insn->imm & 0x1f); + break; + case BPF_ALU64 | BPF_LSH: + emit_instr(ctx, dsll_safe, dst, dst, insn->imm & 0x3f); + break; + case BPF_ALU | BPF_LSH: + emit_instr(ctx, sll, dst, dst, insn->imm & 0x1f); + break; + case BPF_ALU64 | BPF_ARSH: + emit_instr(ctx, dsra_safe, dst, dst, insn->imm & 0x3f); + break; + case BPF_ALU | BPF_ARSH: + emit_instr(ctx, sra, dst, dst, insn->imm & 0x1f); + break; + case BPF_ALU | BPF_MOV: + emit_instr(ctx, addiu, dst, MIPS_R_ZERO, insn->imm); + break; + case BPF_ALU | BPF_ADD: + emit_instr(ctx, addiu, dst, dst, insn->imm); + break; + case BPF_ALU | BPF_SUB: + emit_instr(ctx, addiu, dst, dst, -insn->imm); + break; + default: + return -EINVAL; + } + } else { + /* multi insn immediate case */ + if (BPF_OP(insn->code) == BPF_MOV) { + gen_imm_to_reg(insn, dst, ctx); + } else { + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { + case BPF_ALU64 | BPF_AND: + case BPF_ALU | BPF_AND: + emit_instr(ctx, and, dst, dst, MIPS_R_AT); + break; + case BPF_ALU64 | BPF_OR: + case BPF_ALU | BPF_OR: + emit_instr(ctx, or, dst, dst, MIPS_R_AT); + break; + case BPF_ALU64 | BPF_XOR: + case BPF_ALU | BPF_XOR: + emit_instr(ctx, xor, dst, dst, MIPS_R_AT); + break; + case BPF_ALU64 | BPF_ADD: + emit_instr(ctx, daddu, dst, dst, MIPS_R_AT); + break; + case BPF_ALU64 | BPF_SUB: + emit_instr(ctx, dsubu, dst, dst, MIPS_R_AT); + break; + case BPF_ALU | BPF_ADD: + emit_instr(ctx, addu, dst, dst, MIPS_R_AT); + break; + case BPF_ALU | BPF_SUB: + emit_instr(ctx, subu, dst, dst, MIPS_R_AT); + break; + default: + return -EINVAL; + } + } + } + + return 0; +} + +/* Returns the number of insn slots consumed. */ +int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, + int this_idx, int exit_idx) +{ + int src, dst, r, td, ts, mem_off, b_off; + bool need_swap, did_move, cmp_eq; + unsigned int target = 0; + u64 t64; + s64 t64s; + int bpf_op = BPF_OP(insn->code); + + switch (insn->code) { + case BPF_ALU64 | BPF_ADD | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_SUB | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_OR | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_AND | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_LSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_RSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_XOR | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_ARSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_MOV | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_MOV | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_ADD | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_SUB | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_OR | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_AND | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_LSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_RSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_XOR | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_ARSH | BPF_K: /* ALU64_IMM */ + r = gen_imm_insn(insn, ctx, this_idx); + if (r < 0) + return r; + break; + case BPF_ALU64 | BPF_MUL | BPF_K: /* ALU64_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + if (insn->imm == 1) /* Mult by 1 is a nop */ + break; + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, dmulu, dst, dst, MIPS_R_AT); + } else { + emit_instr(ctx, dmultu, MIPS_R_AT, dst); + emit_instr(ctx, mflo, dst); + } + break; + case BPF_ALU64 | BPF_NEG | BPF_K: /* ALU64_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + emit_instr(ctx, dsubu, dst, MIPS_R_ZERO, dst); + break; + case BPF_ALU | BPF_MUL | BPF_K: /* ALU_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + if (td == REG_64BIT) { + /* sign extend */ + emit_instr(ctx, sll, dst, dst, 0); + } + if (insn->imm == 1) /* Mult by 1 is a nop */ + break; + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, mulu, dst, dst, MIPS_R_AT); + } else { + emit_instr(ctx, multu, dst, MIPS_R_AT); + emit_instr(ctx, mflo, dst); + } + break; + case BPF_ALU | BPF_NEG | BPF_K: /* ALU_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + if (td == REG_64BIT) { + /* sign extend */ + emit_instr(ctx, sll, dst, dst, 0); + } + emit_instr(ctx, subu, dst, MIPS_R_ZERO, dst); + break; + case BPF_ALU | BPF_DIV | BPF_K: /* ALU_IMM */ + case BPF_ALU | BPF_MOD | BPF_K: /* ALU_IMM */ + if (insn->imm == 0) + return -EINVAL; + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + if (td == REG_64BIT) + /* sign extend */ + emit_instr(ctx, sll, dst, dst, 0); + if (insn->imm == 1) { + /* div by 1 is a nop, mod by 1 is zero */ + if (bpf_op == BPF_MOD) + emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO); + break; + } + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, divu_r6, dst, dst, MIPS_R_AT); + else + emit_instr(ctx, modu, dst, dst, MIPS_R_AT); + break; + } + emit_instr(ctx, divu, dst, MIPS_R_AT); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, dst); + else + emit_instr(ctx, mfhi, dst); + break; + case BPF_ALU64 | BPF_DIV | BPF_K: /* ALU_IMM */ + case BPF_ALU64 | BPF_MOD | BPF_K: /* ALU_IMM */ + if (insn->imm == 0) + return -EINVAL; + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + if (insn->imm == 1) { + /* div by 1 is a nop, mod by 1 is zero */ + if (bpf_op == BPF_MOD) + emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO); + break; + } + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, ddivu_r6, dst, dst, MIPS_R_AT); + else + emit_instr(ctx, dmodu, dst, dst, MIPS_R_AT); + break; + } + emit_instr(ctx, ddivu, dst, MIPS_R_AT); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, dst); + else + emit_instr(ctx, mfhi, dst); + break; + case BPF_ALU64 | BPF_MOV | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_ADD | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_SUB | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_XOR | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_OR | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_AND | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_MUL | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_DIV | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_MOD | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_LSH | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_RSH | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_ARSH | BPF_X: /* ALU64_REG */ + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (src < 0 || dst < 0) + return -EINVAL; + if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + did_move = false; + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + int tmp_reg = MIPS_R_AT; + + if (bpf_op == BPF_MOV) { + tmp_reg = dst; + did_move = true; + } + emit_instr(ctx, daddu, tmp_reg, src, MIPS_R_ZERO); + emit_instr(ctx, dinsu, tmp_reg, MIPS_R_ZERO, 32, 32); + src = MIPS_R_AT; + } + switch (bpf_op) { + case BPF_MOV: + if (!did_move) + emit_instr(ctx, daddu, dst, src, MIPS_R_ZERO); + break; + case BPF_ADD: + emit_instr(ctx, daddu, dst, dst, src); + break; + case BPF_SUB: + emit_instr(ctx, dsubu, dst, dst, src); + break; + case BPF_XOR: + emit_instr(ctx, xor, dst, dst, src); + break; + case BPF_OR: + emit_instr(ctx, or, dst, dst, src); + break; + case BPF_AND: + emit_instr(ctx, and, dst, dst, src); + break; + case BPF_MUL: + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, dmulu, dst, dst, src); + } else { + emit_instr(ctx, dmultu, dst, src); + emit_instr(ctx, mflo, dst); + } + break; + case BPF_DIV: + case BPF_MOD: + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, ddivu_r6, + dst, dst, src); + else + emit_instr(ctx, dmodu, dst, dst, src); + break; + } + emit_instr(ctx, ddivu, dst, src); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, dst); + else + emit_instr(ctx, mfhi, dst); + break; + case BPF_LSH: + emit_instr(ctx, dsllv, dst, dst, src); + break; + case BPF_RSH: + emit_instr(ctx, dsrlv, dst, dst, src); + break; + case BPF_ARSH: + emit_instr(ctx, dsrav, dst, dst, src); + break; + default: + pr_err("ALU64_REG NOT HANDLED\n"); + return -EINVAL; + } + break; + case BPF_ALU | BPF_MOV | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_ADD | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_SUB | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_XOR | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_OR | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_AND | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_MUL | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_DIV | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_MOD | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_LSH | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_RSH | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_ARSH | BPF_X: /* ALU_REG */ + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (src < 0 || dst < 0) + return -EINVAL; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + if (td == REG_64BIT) { + /* sign extend */ + emit_instr(ctx, sll, dst, dst, 0); + } + did_move = false; + ts = get_reg_val_type(ctx, this_idx, insn->src_reg); + if (ts == REG_64BIT) { + int tmp_reg = MIPS_R_AT; + + if (bpf_op == BPF_MOV) { + tmp_reg = dst; + did_move = true; + } + /* sign extend */ + emit_instr(ctx, sll, tmp_reg, src, 0); + src = MIPS_R_AT; + } + switch (bpf_op) { + case BPF_MOV: + if (!did_move) + emit_instr(ctx, addu, dst, src, MIPS_R_ZERO); + break; + case BPF_ADD: + emit_instr(ctx, addu, dst, dst, src); + break; + case BPF_SUB: + emit_instr(ctx, subu, dst, dst, src); + break; + case BPF_XOR: + emit_instr(ctx, xor, dst, dst, src); + break; + case BPF_OR: + emit_instr(ctx, or, dst, dst, src); + break; + case BPF_AND: + emit_instr(ctx, and, dst, dst, src); + break; + case BPF_MUL: + emit_instr(ctx, mul, dst, dst, src); + break; + case BPF_DIV: + case BPF_MOD: + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, divu_r6, dst, dst, src); + else + emit_instr(ctx, modu, dst, dst, src); + break; + } + emit_instr(ctx, divu, dst, src); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, dst); + else + emit_instr(ctx, mfhi, dst); + break; + case BPF_LSH: + emit_instr(ctx, sllv, dst, dst, src); + break; + case BPF_RSH: + emit_instr(ctx, srlv, dst, dst, src); + break; + case BPF_ARSH: + emit_instr(ctx, srav, dst, dst, src); + break; + default: + pr_err("ALU_REG NOT HANDLED\n"); + return -EINVAL; + } + break; + case BPF_JMP | BPF_EXIT: + if (this_idx + 1 < exit_idx) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, exit_idx); + if (target == (unsigned int)-1) + return -E2BIG; + emit_instr(ctx, j, target); + } else { + emit_instr(ctx, b, b_off); + } + emit_instr(ctx, nop); + } + break; + case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */ + cmp_eq = (bpf_op == BPF_JEQ); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + if (insn->imm == 0) { + src = MIPS_R_ZERO; + } else { + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + src = MIPS_R_AT; + } + goto jeq_common; + case BPF_JMP | BPF_JEQ | BPF_X: /* JMP_REG */ + case BPF_JMP | BPF_JNE | BPF_X: + case BPF_JMP | BPF_JSLT | BPF_X: + case BPF_JMP | BPF_JSLE | BPF_X: + case BPF_JMP | BPF_JSGT | BPF_X: + case BPF_JMP | BPF_JSGE | BPF_X: + case BPF_JMP | BPF_JLT | BPF_X: + case BPF_JMP | BPF_JLE | BPF_X: + case BPF_JMP | BPF_JGT | BPF_X: + case BPF_JMP | BPF_JGE | BPF_X: + case BPF_JMP | BPF_JSET | BPF_X: + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + ts = get_reg_val_type(ctx, this_idx, insn->src_reg); + if (td == REG_32BIT && ts != REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_AT, src, 0); + src = MIPS_R_AT; + } else if (ts == REG_32BIT && td != REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_AT, dst, 0); + dst = MIPS_R_AT; + } + if (bpf_op == BPF_JSET) { + emit_instr(ctx, and, MIPS_R_AT, dst, src); + cmp_eq = false; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; + } else if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLE) { + emit_instr(ctx, dsubu, MIPS_R_AT, dst, src); + if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + if (bpf_op == BPF_JSGT) + emit_instr(ctx, blez, MIPS_R_AT, b_off); + else + emit_instr(ctx, bgtz, MIPS_R_AT, b_off); + emit_instr(ctx, nop); + return 2; /* We consumed the exit. */ + } + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + if (bpf_op == BPF_JSGT) + emit_instr(ctx, bgtz, MIPS_R_AT, b_off); + else + emit_instr(ctx, blez, MIPS_R_AT, b_off); + emit_instr(ctx, nop); + break; + } else if (bpf_op == BPF_JSGE || bpf_op == BPF_JSLT) { + emit_instr(ctx, slt, MIPS_R_AT, dst, src); + cmp_eq = bpf_op == BPF_JSGE; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; + } else if (bpf_op == BPF_JGT || bpf_op == BPF_JLE) { + /* dst or src could be AT */ + emit_instr(ctx, dsubu, MIPS_R_T8, dst, src); + emit_instr(ctx, sltu, MIPS_R_AT, dst, src); + /* SP known to be non-zero, movz becomes boolean not */ + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, seleqz, MIPS_R_T9, + MIPS_R_SP, MIPS_R_T8); + } else { + emit_instr(ctx, movz, MIPS_R_T9, + MIPS_R_SP, MIPS_R_T8); + emit_instr(ctx, movn, MIPS_R_T9, + MIPS_R_ZERO, MIPS_R_T8); + } + emit_instr(ctx, or, MIPS_R_AT, MIPS_R_T9, MIPS_R_AT); + cmp_eq = bpf_op == BPF_JGT; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; + } else if (bpf_op == BPF_JGE || bpf_op == BPF_JLT) { + emit_instr(ctx, sltu, MIPS_R_AT, dst, src); + cmp_eq = bpf_op == BPF_JGE; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; + } else { /* JNE/JEQ case */ + cmp_eq = (bpf_op == BPF_JEQ); + } +jeq_common: + /* + * If the next insn is EXIT and we are jumping arround + * only it, invert the sense of the compare and + * conditionally jump to the exit. Poor man's branch + * chaining. + */ + if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, exit_idx); + if (target == (unsigned int)-1) + return -E2BIG; + cmp_eq = !cmp_eq; + b_off = 4 * 3; + if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { + ctx->offsets[this_idx] |= OFFSETS_B_CONV; + ctx->long_b_conversion = 1; + } + } + + if (cmp_eq) + emit_instr(ctx, bne, dst, src, b_off); + else + emit_instr(ctx, beq, dst, src, b_off); + emit_instr(ctx, nop); + if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { + emit_instr(ctx, j, target); + emit_instr(ctx, nop); + } + return 2; /* We consumed the exit. */ + } + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, this_idx + insn->off + 1); + if (target == (unsigned int)-1) + return -E2BIG; + cmp_eq = !cmp_eq; + b_off = 4 * 3; + if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { + ctx->offsets[this_idx] |= OFFSETS_B_CONV; + ctx->long_b_conversion = 1; + } + } + + if (cmp_eq) + emit_instr(ctx, beq, dst, src, b_off); + else + emit_instr(ctx, bne, dst, src, b_off); + emit_instr(ctx, nop); + if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { + emit_instr(ctx, j, target); + emit_instr(ctx, nop); + } + break; + case BPF_JMP | BPF_JSGT | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JSGE | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JSLT | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JSLE | BPF_K: /* JMP_IMM */ + cmp_eq = (bpf_op == BPF_JSGE); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + + if (insn->imm == 0) { + if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + switch (bpf_op) { + case BPF_JSGT: + emit_instr(ctx, blez, dst, b_off); + break; + case BPF_JSGE: + emit_instr(ctx, bltz, dst, b_off); + break; + case BPF_JSLT: + emit_instr(ctx, bgez, dst, b_off); + break; + case BPF_JSLE: + emit_instr(ctx, bgtz, dst, b_off); + break; + } + emit_instr(ctx, nop); + return 2; /* We consumed the exit. */ + } + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + switch (bpf_op) { + case BPF_JSGT: + emit_instr(ctx, bgtz, dst, b_off); + break; + case BPF_JSGE: + emit_instr(ctx, bgez, dst, b_off); + break; + case BPF_JSLT: + emit_instr(ctx, bltz, dst, b_off); + break; + case BPF_JSLE: + emit_instr(ctx, blez, dst, b_off); + break; + } + emit_instr(ctx, nop); + break; + } + /* + * only "LT" compare available, so we must use imm + 1 + * to generate "GT" and imm -1 to generate LE + */ + if (bpf_op == BPF_JSGT) + t64s = insn->imm + 1; + else if (bpf_op == BPF_JSLE) + t64s = insn->imm + 1; + else + t64s = insn->imm; + + cmp_eq = bpf_op == BPF_JSGT || bpf_op == BPF_JSGE; + if (t64s >= S16_MIN && t64s <= S16_MAX) { + emit_instr(ctx, slti, MIPS_R_AT, dst, (int)t64s); + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + } + emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); + emit_instr(ctx, slt, MIPS_R_AT, dst, MIPS_R_AT); + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP | BPF_JGT | BPF_K: + case BPF_JMP | BPF_JGE | BPF_K: + case BPF_JMP | BPF_JLT | BPF_K: + case BPF_JMP | BPF_JLE | BPF_K: + cmp_eq = (bpf_op == BPF_JGE); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + /* + * only "LT" compare available, so we must use imm + 1 + * to generate "GT" and imm -1 to generate LE + */ + if (bpf_op == BPF_JGT) + t64s = (u64)(u32)(insn->imm) + 1; + else if (bpf_op == BPF_JLE) + t64s = (u64)(u32)(insn->imm) + 1; + else + t64s = (u64)(u32)(insn->imm); + + cmp_eq = bpf_op == BPF_JGT || bpf_op == BPF_JGE; + + emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); + emit_instr(ctx, sltu, MIPS_R_AT, dst, MIPS_R_AT); + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + + if (ctx->use_bbit_insns && hweight32((u32)insn->imm) == 1) { + if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + emit_instr(ctx, bbit0, dst, ffs((u32)insn->imm) - 1, b_off); + emit_instr(ctx, nop); + return 2; /* We consumed the exit. */ + } + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) + return -E2BIG; + emit_instr(ctx, bbit1, dst, ffs((u32)insn->imm) - 1, b_off); + emit_instr(ctx, nop); + break; + } + t64 = (u32)insn->imm; + emit_const_to_reg(ctx, MIPS_R_AT, t64); + emit_instr(ctx, and, MIPS_R_AT, dst, MIPS_R_AT); + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + cmp_eq = false; + goto jeq_common; + + case BPF_JMP | BPF_JA: + /* + * Prefer relative branch for easier debugging, but + * fall back if needed. + */ + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, this_idx + insn->off + 1); + if (target == (unsigned int)-1) + return -E2BIG; + emit_instr(ctx, j, target); + } else { + emit_instr(ctx, b, b_off); + } + emit_instr(ctx, nop); + break; + case BPF_LD | BPF_DW | BPF_IMM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + t64 = ((u64)(u32)insn->imm) | ((u64)(insn + 1)->imm << 32); + emit_const_to_reg(ctx, dst, t64); + return 2; /* Double slot insn */ + + case BPF_JMP | BPF_CALL: + emit_bpf_call(ctx, insn); + break; + + case BPF_JMP | BPF_TAIL_CALL: + if (emit_bpf_tail_call(ctx, this_idx)) + return -EINVAL; + break; + + case BPF_ALU | BPF_END | BPF_FROM_BE: + case BPF_ALU | BPF_END | BPF_FROM_LE: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + td = get_reg_val_type(ctx, this_idx, insn->dst_reg); + if (insn->imm == 64 && td == REG_32BIT) + emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32); + + if (insn->imm != 64 && td == REG_64BIT) { + /* sign extend */ + emit_instr(ctx, sll, dst, dst, 0); + } + +#ifdef __BIG_ENDIAN + need_swap = (BPF_SRC(insn->code) == BPF_FROM_LE); +#else + need_swap = (BPF_SRC(insn->code) == BPF_FROM_BE); +#endif + if (insn->imm == 16) { + if (need_swap) + emit_instr(ctx, wsbh, dst, dst); + emit_instr(ctx, andi, dst, dst, 0xffff); + } else if (insn->imm == 32) { + if (need_swap) { + emit_instr(ctx, wsbh, dst, dst); + emit_instr(ctx, rotr, dst, dst, 16); + } + } else { /* 64-bit*/ + if (need_swap) { + emit_instr(ctx, dsbh, dst, dst); + emit_instr(ctx, dshd, dst, dst); + } + } + break; + + case BPF_ST | BPF_NOSPEC: /* speculation barrier */ + break; + + case BPF_ST | BPF_B | BPF_MEM: + case BPF_ST | BPF_H | BPF_MEM: + case BPF_ST | BPF_W | BPF_MEM: + case BPF_ST | BPF_DW | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + mem_off = insn->off; + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + switch (BPF_SIZE(insn->code)) { + case BPF_B: + emit_instr(ctx, sb, MIPS_R_AT, mem_off, dst); + break; + case BPF_H: + emit_instr(ctx, sh, MIPS_R_AT, mem_off, dst); + break; + case BPF_W: + emit_instr(ctx, sw, MIPS_R_AT, mem_off, dst); + break; + case BPF_DW: + emit_instr(ctx, sd, MIPS_R_AT, mem_off, dst); + break; + } + break; + + case BPF_LDX | BPF_B | BPF_MEM: + case BPF_LDX | BPF_H | BPF_MEM: + case BPF_LDX | BPF_W | BPF_MEM: + case BPF_LDX | BPF_DW | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (dst < 0 || src < 0) + return -EINVAL; + mem_off = insn->off; + switch (BPF_SIZE(insn->code)) { + case BPF_B: + emit_instr(ctx, lbu, dst, mem_off, src); + break; + case BPF_H: + emit_instr(ctx, lhu, dst, mem_off, src); + break; + case BPF_W: + emit_instr(ctx, lw, dst, mem_off, src); + break; + case BPF_DW: + emit_instr(ctx, ld, dst, mem_off, src); + break; + } + break; + + case BPF_STX | BPF_B | BPF_MEM: + case BPF_STX | BPF_H | BPF_MEM: + case BPF_STX | BPF_W | BPF_MEM: + case BPF_STX | BPF_DW | BPF_MEM: + case BPF_STX | BPF_W | BPF_ATOMIC: + case BPF_STX | BPF_DW | BPF_ATOMIC: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + if (BPF_MODE(insn->code) == BPF_ATOMIC) { + if (insn->imm != BPF_ADD) { + pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); + return -EINVAL; + } + /* + * If mem_off does not fit within the 9 bit ll/sc + * instruction immediate field, use a temp reg. + */ + if (MIPS_ISA_REV >= 6 && + (mem_off >= BIT(8) || mem_off < -BIT(8))) { + emit_instr(ctx, daddiu, MIPS_R_T6, + dst, mem_off); + mem_off = 0; + dst = MIPS_R_T6; + } + switch (BPF_SIZE(insn->code)) { + case BPF_W: + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_AT, src, 0); + src = MIPS_R_AT; + } + emit_instr(ctx, ll, MIPS_R_T8, mem_off, dst); + emit_instr(ctx, addu, MIPS_R_T8, MIPS_R_T8, src); + emit_instr(ctx, sc, MIPS_R_T8, mem_off, dst); + /* + * On failure back up to LL (-4 + * instructions of 4 bytes each + */ + emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); + emit_instr(ctx, nop); + break; + case BPF_DW: + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); + emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); + src = MIPS_R_AT; + } + emit_instr(ctx, lld, MIPS_R_T8, mem_off, dst); + emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, src); + emit_instr(ctx, scd, MIPS_R_T8, mem_off, dst); + emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); + emit_instr(ctx, nop); + break; + } + } else { /* BPF_MEM */ + switch (BPF_SIZE(insn->code)) { + case BPF_B: + emit_instr(ctx, sb, src, mem_off, dst); + break; + case BPF_H: + emit_instr(ctx, sh, src, mem_off, dst); + break; + case BPF_W: + emit_instr(ctx, sw, src, mem_off, dst); + break; + case BPF_DW: + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); + emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); + src = MIPS_R_AT; + } + emit_instr(ctx, sd, src, mem_off, dst); + break; + } + } + break; + + default: + pr_err("NOT HANDLED %d - (%02x)\n", + this_idx, (unsigned int)insn->code); + return -EINVAL; + } + return 1; +} diff --git a/arch/mips/net/ebpf_jit_core.c b/arch/mips/net/ebpf_jit_core.c new file mode 100644 index 000000000000..37b496f47ddb --- /dev/null +++ b/arch/mips/net/ebpf_jit_core.c @@ -0,0 +1,1189 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 + * Copyright (c) 2021 Tony Ambardar + * + * Based on code from: + * + * Copyright (c) 2017 Cavium, Inc. + * Author: David Daney + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras + */ + +#include +#include +#include +#include +#include + +#include "ebpf_jit.h" + +/* + * Extra JIT registers dedicated to holding TCC during runtime or saving + * across calls. + */ +enum { + JIT_RUN_TCC = MAX_BPF_JIT_REG, + JIT_SAV_TCC +}; +/* Temporary register for passing TCC if nothing dedicated. */ +#define TEMP_PASS_TCC MIPS_R_T8 + +#ifdef CONFIG_64BIT +# define M(expr32, expr64) (expr64) +#else +# define M(expr32, expr64) (expr32) +#endif +static const struct { + /* Register or pair base */ + int reg; + /* Register flags */ + u32 flags; + /* Usage table: (MIPS32) (MIPS64) */ +} bpf2mips[] = { + /* Return value from in-kernel function, and exit value from eBPF. */ + [BPF_REG_0] = {M(MIPS_R_V0, MIPS_R_V0)}, + /* Arguments from eBPF program to in-kernel/BPF functions. */ + [BPF_REG_1] = {M(MIPS_R_A0, MIPS_R_A0)}, + [BPF_REG_2] = {M(MIPS_R_A2, MIPS_R_A1)}, + [BPF_REG_3] = {M(MIPS_R_T0, MIPS_R_A2)}, + [BPF_REG_4] = {M(MIPS_R_T2, MIPS_R_A3)}, + [BPF_REG_5] = {M(MIPS_R_T4, MIPS_R_A4)}, + /* Callee-saved registers preserved by in-kernel/BPF functions. */ + [BPF_REG_6] = {M(MIPS_R_S0, MIPS_R_S0), + M(EBPF_SAVE_S0|EBPF_SAVE_S1, EBPF_SAVE_S0)}, + [BPF_REG_7] = {M(MIPS_R_S2, MIPS_R_S1), + M(EBPF_SAVE_S2|EBPF_SAVE_S3, EBPF_SAVE_S1)}, + [BPF_REG_8] = {M(MIPS_R_S4, MIPS_R_S2), + M(EBPF_SAVE_S4|EBPF_SAVE_S5, EBPF_SAVE_S2)}, + [BPF_REG_9] = {M(MIPS_R_S6, MIPS_R_S3), + M(EBPF_SAVE_S6|EBPF_SAVE_S7, EBPF_SAVE_S3)}, + [BPF_REG_10] = {M(MIPS_R_S8, MIPS_R_S8), + M(EBPF_SAVE_S8|EBPF_SEEN_FP, EBPF_SAVE_S8|EBPF_SEEN_FP)}, + /* Internal register for rewriting insns during JIT blinding. */ + [BPF_REG_AX] = {M(MIPS_R_T6, MIPS_R_T4)}, + /* + * Internal registers for TCC runtime holding and saving during + * calls. A zero save register indicates using scratch space on + * the stack for storage during calls. A zero hold register means + * no dedicated register holds TCC during runtime (but a temp reg + * still passes TCC to tailcall or bpf2bpf call). + */ + [JIT_RUN_TCC] = {M(0, MIPS_R_V1)}, + [JIT_SAV_TCC] = {M(0, MIPS_R_S4), + M(0, EBPF_SAVE_S4)} +}; +#undef M + +/* + * For eBPF, the register mapping naturally falls out of the + * requirements of eBPF and MIPS N64/O32 ABIs. We also maintain + * a separate frame pointer, setting BPF_REG_10 relative to $sp. + */ +int ebpf_to_mips_reg(struct jit_ctx *ctx, + const struct bpf_insn *insn, + enum reg_usage u) +{ + int ebpf_reg = (u == REG_SRC_FP_OK || u == REG_SRC_NO_FP) ? + insn->src_reg : insn->dst_reg; + + switch (ebpf_reg) { + case BPF_REG_0: + case BPF_REG_1: + case BPF_REG_2: + case BPF_REG_3: + case BPF_REG_4: + case BPF_REG_5: + case BPF_REG_6: + case BPF_REG_7: + case BPF_REG_8: + case BPF_REG_9: + case BPF_REG_AX: + ctx->flags |= bpf2mips[ebpf_reg].flags; + return bpf2mips[ebpf_reg].reg; + case BPF_REG_10: + if (u == REG_DST_NO_FP || u == REG_SRC_NO_FP) + goto bad_reg; + ctx->flags |= bpf2mips[ebpf_reg].flags; + return bpf2mips[ebpf_reg].reg; + default: +bad_reg: + WARN(1, "Illegal bpf reg: %d\n", ebpf_reg); + return -EINVAL; + } +} + +void gen_imm_to_reg(const struct bpf_insn *insn, int reg, + struct jit_ctx *ctx) +{ + if (insn->imm >= S16_MIN && insn->imm <= S16_MAX) { + emit_instr(ctx, addiu, reg, MIPS_R_ZERO, insn->imm); + } else { + int lower = (s16)(insn->imm & 0xffff); + int upper = insn->imm - lower; + + emit_instr(ctx, lui, reg, upper >> 16); + /* lui already clears lower halfword */ + if (lower) + emit_instr(ctx, addiu, reg, reg, lower); + } +} + +void emit_const_to_reg(struct jit_ctx *ctx, int dst, unsigned long value) +{ + if (value >= S16_MIN || value <= S16_MAX) { + emit_instr_long(ctx, daddiu, addiu, dst, MIPS_R_ZERO, (int)value); + } else if (value >= S32_MIN || + (value <= S32_MAX && value > U16_MAX)) { + emit_instr(ctx, lui, dst, (s32)(s16)(value >> 16)); + emit_instr(ctx, ori, dst, dst, (unsigned int)(value & 0xffff)); + } else { + int i; + bool seen_part = false; + int needed_shift = 0; + + for (i = 0; i < 4; i++) { + u64 part = (value >> (16 * (3 - i))) & 0xffff; + + if (seen_part && needed_shift > 0 && (part || i == 3)) { + emit_instr(ctx, dsll_safe, dst, dst, needed_shift); + needed_shift = 0; + } + if (part) { + if (i == 0 || (!seen_part && i < 3 && part < 0x8000)) { + emit_instr(ctx, lui, dst, (s32)(s16)part); + needed_shift = -16; + } else { + emit_instr(ctx, ori, dst, + seen_part ? dst : MIPS_R_ZERO, + (unsigned int)part); + } + seen_part = true; + } + if (seen_part) + needed_shift += 16; + } + } +} + +#define RVT_VISITED_MASK 0xc000000000000000ull +#define RVT_FALL_THROUGH 0x4000000000000000ull +#define RVT_BRANCH_TAKEN 0x8000000000000000ull +#define RVT_DONE (RVT_FALL_THROUGH | RVT_BRANCH_TAKEN) + +/* return the last idx processed, or negative for error */ +static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt, + int start_idx, bool follow_taken) +{ + const struct bpf_prog *prog = ctx->prog; + const struct bpf_insn *insn; + u64 exit_rvt = initial_rvt; + u64 *rvt = ctx->reg_val_types; + int idx; + int reg; + + for (idx = start_idx; idx < prog->len; idx++) { + rvt[idx] = (rvt[idx] & RVT_VISITED_MASK) | exit_rvt; + insn = prog->insnsi + idx; + switch (BPF_CLASS(insn->code)) { + case BPF_ALU: + switch (BPF_OP(insn->code)) { + case BPF_ADD: + case BPF_SUB: + case BPF_MUL: + case BPF_DIV: + case BPF_OR: + case BPF_AND: + case BPF_LSH: + case BPF_RSH: + case BPF_ARSH: + case BPF_NEG: + case BPF_MOD: + case BPF_XOR: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + break; + case BPF_MOV: + if (BPF_SRC(insn->code)) { + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + } else { + /* IMM to REG move*/ + if (insn->imm >= 0) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + else + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + } + break; + case BPF_END: + if (insn->imm == 64) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + else if (insn->imm == 32) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + else /* insn->imm == 16 */ + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + break; + } + rvt[idx] |= RVT_DONE; + break; + case BPF_ALU64: + switch (BPF_OP(insn->code)) { + case BPF_MOV: + if (BPF_SRC(insn->code)) { + /* REG to REG move*/ + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + } else { + /* IMM to REG move*/ + if (insn->imm >= 0) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + else + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT); + } + break; + default: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + } + rvt[idx] |= RVT_DONE; + break; + case BPF_LD: + switch (BPF_SIZE(insn->code)) { + case BPF_DW: + if (BPF_MODE(insn->code) == BPF_IMM) { + s64 val; + + val = (s64)((u32)insn->imm | ((u64)(insn + 1)->imm << 32)); + if (val > 0 && val <= S32_MAX) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + else if (val >= S32_MIN && val <= S32_MAX) + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT); + else + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + rvt[idx] |= RVT_DONE; + idx++; + } else { + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + } + break; + case BPF_B: + case BPF_H: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + break; + case BPF_W: + if (BPF_MODE(insn->code) == BPF_IMM) + set_reg_val_type(&exit_rvt, insn->dst_reg, + insn->imm >= 0 ? REG_32BIT_POS : REG_32BIT); + else + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + break; + } + rvt[idx] |= RVT_DONE; + break; + case BPF_LDX: + switch (BPF_SIZE(insn->code)) { + case BPF_DW: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT); + break; + case BPF_B: + case BPF_H: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS); + break; + case BPF_W: + set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT); + break; + } + rvt[idx] |= RVT_DONE; + break; + case BPF_JMP: + case BPF_JMP32: + switch (BPF_OP(insn->code)) { + case BPF_EXIT: + rvt[idx] = RVT_DONE | exit_rvt; + rvt[prog->len] = exit_rvt; + return idx; + case BPF_JA: + { + int tgt = idx + 1 + insn->off; + bool visited = (rvt[tgt] & RVT_FALL_THROUGH); + + rvt[idx] |= RVT_DONE; + /* + * Verifier dead code patching can use + * infinite-loop traps, causing hangs and + * RCU stalls here. Treat traps as nops + * if detected and fall through. + */ + if (insn->off == -1) + break; + /* + * Bounded loops cause the same issues in + * fallthrough mode; follow only if jump + * target is unvisited to mitigate. + */ + if (insn->off < 0 && !follow_taken && visited) + break; + idx += insn->off; + break; + } + case BPF_JEQ: + case BPF_JGT: + case BPF_JGE: + case BPF_JLT: + case BPF_JLE: + case BPF_JSET: + case BPF_JNE: + case BPF_JSGT: + case BPF_JSGE: + case BPF_JSLT: + case BPF_JSLE: + if (follow_taken) { + rvt[idx] |= RVT_BRANCH_TAKEN; + idx += insn->off; + follow_taken = false; + } else { + rvt[idx] |= RVT_FALL_THROUGH; + } + break; + case BPF_CALL: + set_reg_val_type(&exit_rvt, BPF_REG_0, REG_64BIT); + /* Upon call return, argument registers are clobbered. */ + for (reg = BPF_REG_0; reg <= BPF_REG_5; reg++) + set_reg_val_type(&exit_rvt, reg, REG_64BIT); + + rvt[idx] |= RVT_DONE; + break; + case BPF_TAIL_CALL: + rvt[idx] |= RVT_DONE; + break; + default: + WARN(1, "Unhandled BPF_JMP case.\n"); + rvt[idx] |= RVT_DONE; + break; + } + break; + default: + rvt[idx] |= RVT_DONE; + break; + } + } + return idx; +} + +/* + * Track the value range (i.e. 32-bit vs. 64-bit) of each register at + * each eBPF insn. This allows unneeded sign and zero extension + * operations to be omitted. + * + * Doesn't handle yet confluence of control paths with conflicting + * ranges, but it is good enough for most sane code. + */ +static int reg_val_propagate(struct jit_ctx *ctx) +{ + const struct bpf_prog *prog = ctx->prog; + u64 exit_rvt; + int reg; + int i; + + /* + * 11 registers * 3 bits/reg leaves top bits free for other + * uses. Bit-62..63 used to see if we have visited an insn. + */ + exit_rvt = 0; + + /* Upon entry, argument registers are 64-bit. */ + for (reg = BPF_REG_1; reg <= BPF_REG_5; reg++) + set_reg_val_type(&exit_rvt, reg, REG_64BIT); + + /* + * First follow all conditional branches on the fall-through + * edge of control flow.. + */ + reg_val_propagate_range(ctx, exit_rvt, 0, false); +restart_search: + /* + * Then repeatedly find the first conditional branch where + * both edges of control flow have not been taken, and follow + * the branch taken edge. We will end up restarting the + * search once per conditional branch insn. + */ + for (i = 0; i < prog->len; i++) { + u64 rvt = ctx->reg_val_types[i]; + + if ((rvt & RVT_VISITED_MASK) == RVT_DONE || + (rvt & RVT_VISITED_MASK) == 0) + continue; + if ((rvt & RVT_VISITED_MASK) == RVT_FALL_THROUGH) { + reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, true); + } else { /* RVT_BRANCH_TAKEN */ + WARN(1, "Unexpected RVT_BRANCH_TAKEN case.\n"); + reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, false); + } + goto restart_search; + } + /* + * Eventually all conditional branches have been followed on + * both branches and we are done. Any insn that has not been + * visited at this point is dead. + */ + + return 0; +} + +static void jit_fill_hole(void *area, unsigned int size) +{ + u32 *p; + + /* We are guaranteed to have aligned memory. */ + for (p = area; size >= sizeof(u32); size -= sizeof(u32)) + uasm_i_break(&p, BRK_BUG); /* Increments p */ +} + +/* Stack region alignment under N64 and O32 ABIs */ +#define STACK_ALIGN (2 * sizeof(long)) + +/* + * eBPF stack frame will be something like: + * + * Entry $sp ------> +--------------------------------+ + * | $ra (optional) | + * +--------------------------------+ + * | $s8 (optional) | + * +--------------------------------+ + * | $s7 (optional) | + * +--------------------------------+ + * | $s6 (optional) | + * +--------------------------------+ + * | $s5 (optional) | + * +--------------------------------+ + * | $s4 (optional) | + * +--------------------------------+ + * | $s3 (optional) | + * +--------------------------------+ + * | $s2 (optional) | + * +--------------------------------+ + * | $s1 (optional) | + * +--------------------------------+ + * | $s0 (optional) | + * +--------------------------------+ + * | tmp-storage (optional) | + * $sp + bpf_stack_off->+--------------------------------+ <--BPF_REG_10 + * | BPF_REG_10 relative storage | + * | MAX_BPF_STACK (optional) | + * | . | + * | . | + * | . | + * $sp ------> +--------------------------------+ + * + * If BPF_REG_10 is never referenced, then the MAX_BPF_STACK sized + * area is not allocated. + */ +static int build_int_prologue(struct jit_ctx *ctx) +{ + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + const struct bpf_prog *prog = ctx->prog; + int r10 = bpf2mips[BPF_REG_10].reg; + int r1 = bpf2mips[BPF_REG_1].reg; + int stack_adjust = 0; + int store_offset; + int locals_size; + int start_idx; + + if (ctx->flags & EBPF_SAVE_RA) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S8) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S7) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S6) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S5) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S4) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S3) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S2) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S1) + stack_adjust += sizeof(long); + if (ctx->flags & EBPF_SAVE_S0) + stack_adjust += sizeof(long); + if (tail_call_present(ctx) && + !(ctx->flags & EBPF_TCC_IN_RUN) && !tcc_sav) + /* Allocate scratch space for holding TCC if needed. */ + stack_adjust += sizeof(long); + + stack_adjust = ALIGN(stack_adjust, STACK_ALIGN); + + locals_size = (ctx->flags & EBPF_SEEN_FP) ? prog->aux->stack_depth : 0; + locals_size = ALIGN(locals_size, STACK_ALIGN); + + stack_adjust += locals_size; + + ctx->stack_size = stack_adjust; + ctx->bpf_stack_off = locals_size; + + /* + * First instruction initializes the tail call count (TCC) and + * assumes a call from kernel using the native ABI. Calls made + * using the BPF ABI (bpf2bpf or tail call) will skip this insn + * and pass the TCC via register. + */ + start_idx = ctx->idx; + emit_instr(ctx, addiu, tcc_run, MIPS_R_ZERO, MAX_TAIL_CALL_CNT); + + /* + * When called from kernel under O32 ABI we must set up BPF R1 + * context, since BPF R1 is an endian-order regster pair ($a0:$a1 + * or $a1:$a0) but context is always passed in $a0 as a 32-bit + * pointer. As above, bpf2bpf and tail calls will skip these insns + * since all registers are correctly set up already. + */ + if (!is64bit()) { + if (isbigend()) + emit_instr(ctx, move, LO(r1), MIPS_R_A0); + /* Sanitize upper 32-bit reg */ + gen_zext_insn(r1, true, ctx); + } + /* + * Calls using BPF ABI (bpf2bpf and tail calls) will skip TCC + * initialization and R1 context fixup needed by kernel calls. + */ + ctx->prolog_skip = (ctx->idx - start_idx) * 4; + + if (stack_adjust) + emit_instr_long(ctx, daddiu, addiu, + MIPS_R_SP, MIPS_R_SP, -stack_adjust); + else + return 0; + + store_offset = stack_adjust - sizeof(long); + + if (ctx->flags & EBPF_SAVE_RA) { + emit_instr_long(ctx, sd, sw, + MIPS_R_RA, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S8) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S8, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S7) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S7, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S6) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S6, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S5) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S5, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S4) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S4, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S3) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S3, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S2) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S2, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S1) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S1, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S0) { + emit_instr_long(ctx, sd, sw, + MIPS_R_S0, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + + /* Store TCC in backup register or stack scratch space if indicated. */ + if (tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { + if (tcc_sav) + emit_instr(ctx, move, tcc_sav, tcc_run); + else + emit_instr_long(ctx, sd, sw, + tcc_run, ctx->bpf_stack_off, MIPS_R_SP); + } + + /* Prepare BPF FP as single-reg ptr, emulate upper 32-bits as needed.*/ + if (ctx->flags & EBPF_SEEN_FP) + emit_instr_long(ctx, daddiu, addiu, r10, + MIPS_R_SP, ctx->bpf_stack_off); + + return 0; +} + +static int build_int_body(struct jit_ctx *ctx) +{ + const struct bpf_prog *prog = ctx->prog; + const struct bpf_insn *insn; + int i, r; + + for (i = 0; i < prog->len; ) { + insn = prog->insnsi + i; + if ((ctx->reg_val_types[i] & RVT_VISITED_MASK) == 0) { + /* dead instruction, don't emit it. */ + i++; + continue; + } + + if (ctx->target == NULL) + ctx->offsets[i] = (ctx->offsets[i] & OFFSETS_B_CONV) | (ctx->idx * 4); + + r = build_one_insn(insn, ctx, i, prog->len); + if (r < 0) + return r; + i += r; + } + /* epilogue offset */ + if (ctx->target == NULL) + ctx->offsets[i] = ctx->idx * 4; + + /* + * All exits have an offset of the epilogue, some offsets may + * not have been set due to banch-around threading, so set + * them now. + */ + if (ctx->target == NULL) + for (i = 0; i < prog->len; i++) { + insn = prog->insnsi + i; + if (insn->code == (BPF_JMP | BPF_EXIT)) + ctx->offsets[i] = ctx->idx * 4; + } + return 0; +} + +static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg) +{ + const struct bpf_prog *prog = ctx->prog; + int stack_adjust = ctx->stack_size; + int store_offset = stack_adjust - sizeof(long); + int ax = bpf2mips[BPF_REG_AX].reg; + int r0 = bpf2mips[BPF_REG_0].reg; + enum reg_val_type td; + + /* + * As in prologue code, we default to assuming exit to the kernel. + * Returns to the kernel follow the N64 or O32 ABI. For N64, the + * BPF R0 return value may need to be sign-extended, while O32 may + * need fixup of BPF R0 to place the 32-bit return value in MIPS V0. + * + * Returns to BPF2BPF callers consistently use the BPF 64-bit ABI, + * so register usage and mapping between JIT and OS is unchanged. + * Accommodate by saving unmodified R0 register data to allow a + * BPF caller to restore R0 after we return. + */ + if (dest_reg == MIPS_R_RA) { /* kernel or bpf2bpf function return */ + if (is64bit()) { + /* + * Backup BPF R0 to AX, allowing the caller to + * restore it in case this is a BPF2BPF rather + * than a kernel return. + */ + emit_instr(ctx, move, ax, r0); + /* + * Don't let zero-extended R0 value escape to + * kernel on return, so sign-extend if needed. + */ + td = get_reg_val_type(ctx, prog->len, BPF_REG_0); + if (td == REG_64BIT) + gen_sext_insn(r0, ctx); + } else if (isbigend()) { /* and 32-bit */ + /* + * Backup high 32-bit register of BPF R0 to AX, + * since it occupies MIPS_R_V0 which needs to be + * clobbered for a kernel return. + */ + emit_instr(ctx, move, HI(ax), HI(r0)); + /* + * O32 ABI specifies 32-bit return value always + * placed in MIPS_R_V0 regardless of the native + * endianness. This would be in the wrong position + * in a BPF R0 reg pair on big-endian systems, so + * we must relocate. + */ + emit_instr(ctx, move, MIPS_R_V0, LO(r0)); + } + } + + + if (ctx->flags & EBPF_SAVE_RA) { + emit_instr_long(ctx, ld, lw, + MIPS_R_RA, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S8) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S8, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S7) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S7, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S6) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S6, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S5) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S5, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S4) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S4, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S3) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S3, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S2) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S2, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S1) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S1, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + if (ctx->flags & EBPF_SAVE_S0) { + emit_instr_long(ctx, ld, lw, + MIPS_R_S0, store_offset, MIPS_R_SP); + store_offset -= sizeof(long); + } + emit_instr(ctx, jr, dest_reg); + + /* Delay slot */ + if (stack_adjust) + emit_instr_long(ctx, daddiu, addiu, + MIPS_R_SP, MIPS_R_SP, stack_adjust); + else + emit_instr(ctx, nop); + + return 0; +} + +/* + * Push BPF regs R3-R5 to the stack, skipping BPF regs R1-R2 which are + * passed via MIPS register pairs in $a0-$a3. Register order within pairs + * and the memory storage order are identical i.e. endian native. + */ +static void emit_push_args(struct jit_ctx *ctx) +{ + int store_offset = 2 * sizeof(u64); /* Skip R1-R2 in $a0-$a3 */ + int bpf, reg; + + for (bpf = BPF_REG_3; bpf <= BPF_REG_5; bpf++) { + reg = bpf2mips[bpf].reg; + + emit_instr(ctx, sw, LO(reg), OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, sw, HI(reg), OFFHI(store_offset), MIPS_R_SP); + store_offset += sizeof(u64); + } +} + +/* + * Common helper for BPF_CALL insn, handling TCC and ABI variations. + * Kernel calls under O32 ABI require arguments passed on the stack, + * while BPF2BPF calls need the TCC passed via register as expected + * by the subprog's prologue. + * + * Under MIPS32 O32 ABI calling convention, u64 BPF regs R1-R2 are passed + * via reg pairs in $a0-$a3, while BPF regs R3-R5 are passed via the stack. + * Stack space is still reserved for $a0-$a3, and the whole area aligned. + */ +#define ARGS_SIZE (5 * sizeof(u64)) + +void emit_bpf_call(struct jit_ctx *ctx, const struct bpf_insn *insn) +{ + int stack_adjust = ALIGN(ARGS_SIZE, STACK_ALIGN); + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + int ax = bpf2mips[BPF_REG_AX].reg; + int r0 = bpf2mips[BPF_REG_0].reg; + long func_addr; + + ctx->flags |= EBPF_SAVE_RA; + + /* Ensure TCC passed into BPF subprog */ + if ((insn->src_reg == BPF_PSEUDO_CALL) && + tail_call_present(ctx) && !(ctx->flags & EBPF_TCC_IN_RUN)) { + /* Set TCC from reg or stack */ + if (tcc_sav) + emit_instr(ctx, move, tcc_run, tcc_sav); + else + emit_instr_long(ctx, ld, lw, tcc_run, + ctx->bpf_stack_off, MIPS_R_SP); + } + + /* Push O32 stack args for kernel call */ + if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) { + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adjust); + emit_push_args(ctx); + } + + func_addr = (long)__bpf_call_base + insn->imm; + + /* Skip TCC init and R1 register fixup with BPF ABI. */ + if (insn->src_reg == BPF_PSEUDO_CALL) + func_addr += ctx->prolog_skip; + + emit_const_to_reg(ctx, MIPS_R_T9, func_addr); + emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); + /* Delay slot */ + emit_instr(ctx, nop); + + /* Restore stack */ + if (!is64bit() && (insn->src_reg != BPF_PSEUDO_CALL)) + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, stack_adjust); + + /* + * Assuming a kernel return, a MIPS64 function epilogue may + * sign-extend R0, while MIPS32BE mangles the R0 register pair. + * Undo both for a bpf2bpf call return. + */ + if (insn->src_reg == BPF_PSEUDO_CALL) { + /* Restore BPF R0 from AX */ + if (is64bit()) { + emit_instr(ctx, move, r0, ax); + } else if (isbigend()) { /* and 32-bit */ + emit_instr(ctx, move, LO(r0), MIPS_R_V0); + emit_instr(ctx, move, HI(r0), HI(ax)); + } + } +} + +/* + * Tail call helper arguments passed via BPF ABI as u64 parameters. On + * MIPS64 N64 ABI systems these are native regs, while on MIPS32 O32 ABI + * systems these are reg pairs: + * + * R1 -> &ctx + * R2 -> &array + * R3 -> index + */ +int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx) +{ + int tcc_run = bpf2mips[JIT_RUN_TCC].reg ? + bpf2mips[JIT_RUN_TCC].reg : + TEMP_PASS_TCC; + int tcc_sav = bpf2mips[JIT_SAV_TCC].reg; + int r2 = bpf2mips[BPF_REG_2].reg; + int r3 = bpf2mips[BPF_REG_3].reg; + int off, b_off; + int tcc; + + ctx->flags |= EBPF_SEEN_TC; + /* + * if (index >= array->map.max_entries) + * goto out; + */ + if (is64bit()) + /* Mask index as 32-bit */ + gen_zext_insn(r3, true, ctx); + off = offsetof(struct bpf_array, map.max_entries); + emit_instr_long(ctx, lwu, lw, MIPS_R_AT, off, LO(r2)); + emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_AT, LO(r3)); + b_off = b_imm(this_idx + 1, ctx); + emit_instr(ctx, bnez, MIPS_R_AT, b_off); + /* + * if (TCC-- < 0) + * goto out; + */ + /* Delay slot */ + tcc = (ctx->flags & EBPF_TCC_IN_RUN) ? tcc_run : tcc_sav; + /* Get TCC from reg or stack */ + if (tcc) + emit_instr(ctx, move, MIPS_R_T8, tcc); + else + emit_instr_long(ctx, ld, lw, MIPS_R_T8, + ctx->bpf_stack_off, MIPS_R_SP); + b_off = b_imm(this_idx + 1, ctx); + emit_instr(ctx, bltz, MIPS_R_T8, b_off); + /* + * prog = array->ptrs[index]; + * if (prog == NULL) + * goto out; + */ + /* Delay slot */ + emit_instr_long(ctx, dsll, sll, MIPS_R_AT, LO(r3), ilog2(sizeof(long))); + emit_instr_long(ctx, daddu, addu, MIPS_R_AT, MIPS_R_AT, LO(r2)); + off = offsetof(struct bpf_array, ptrs); + emit_instr_long(ctx, ld, lw, MIPS_R_AT, off, MIPS_R_AT); + b_off = b_imm(this_idx + 1, ctx); + emit_instr(ctx, beqz, MIPS_R_AT, b_off); + /* Delay slot */ + emit_instr(ctx, nop); + + /* goto *(prog->bpf_func + skip); */ + off = offsetof(struct bpf_prog, bpf_func); + emit_instr_long(ctx, ld, lw, MIPS_R_T9, off, MIPS_R_AT); + /* All systems are go... decrement and propagate TCC */ + emit_instr_long(ctx, daddiu, addiu, tcc_run, MIPS_R_T8, -1); + /* Skip first instructions (TCC init and R1 fixup) */ + emit_instr_long(ctx, daddiu, addiu, MIPS_R_T9, MIPS_R_T9, ctx->prolog_skip); + return build_int_epilogue(ctx, MIPS_R_T9); +} + +/* + * Save and restore the BPF VM state across a direct kernel call. This + * includes the caller-saved registers used for BPF_REG_0 .. BPF_REG_5 + * and BPF_REG_AX used by the verifier for blinding and other dark arts. + * Restore avoids clobbering bpf_ret, which holds the call return value. + * BPF_REG_6 .. BPF_REG_10 and TCC are already callee-saved or on stack. + */ +static const int bpf_caller_save[] = { + BPF_REG_0, + BPF_REG_1, + BPF_REG_2, + BPF_REG_3, + BPF_REG_4, + BPF_REG_5, + BPF_REG_AX, +}; + +#define CALLER_ENV_SIZE (ARRAY_SIZE(bpf_caller_save) * sizeof(u64)) + +void emit_caller_save(struct jit_ctx *ctx) +{ + int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); + int i, bpf, reg, store_offset; + + emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adj); + + for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { + bpf = bpf_caller_save[i]; + reg = bpf2mips[bpf].reg; + store_offset = i * sizeof(u64); + + if (is64bit()) { + emit_instr(ctx, sd, reg, store_offset, MIPS_R_SP); + } else { + emit_instr(ctx, sw, LO(reg), + OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, sw, HI(reg), + OFFHI(store_offset), MIPS_R_SP); + } + } +} + +void emit_caller_restore(struct jit_ctx *ctx, int bpf_ret) +{ + int stack_adj = ALIGN(CALLER_ENV_SIZE, STACK_ALIGN); + int i, bpf, reg, store_offset; + + for (i = 0; i < ARRAY_SIZE(bpf_caller_save); i++) { + bpf = bpf_caller_save[i]; + reg = bpf2mips[bpf].reg; + store_offset = i * sizeof(u64); + if (bpf == bpf_ret) + continue; + + if (is64bit()) { + emit_instr(ctx, ld, reg, store_offset, MIPS_R_SP); + } else { + emit_instr(ctx, lw, LO(reg), + OFFLO(store_offset), MIPS_R_SP); + emit_instr(ctx, lw, HI(reg), + OFFHI(store_offset), MIPS_R_SP); + } + } + + emit_instr_long(ctx, daddiu, addiu, MIPS_R_SP, MIPS_R_SP, stack_adj); +} + +struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) +{ + bool tmp_blinded = false, extra_pass = false; + struct bpf_prog *tmp, *orig_prog = prog; + struct bpf_binary_header *header = NULL; + unsigned int image_size, pass = 3; + struct jit_ctx *ctx; + + if (!prog->jit_requested) + return orig_prog; + + /* Attempt blinding but fall back to the interpreter on failure. */ + tmp = bpf_jit_blind_constants(prog); + if (IS_ERR(tmp)) + return orig_prog; + if (tmp != prog) { + tmp_blinded = true; + prog = tmp; + } + + ctx = prog->aux->jit_data; + if (!ctx) { + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) { + prog = orig_prog; + goto out; + } + } + + /* + * Assume extra pass needed for patching addresses if previous + * ctx exists in saved jit_data, so skip to code generation. + */ + if (ctx->offsets) { + extra_pass = true; + pass++; + image_size = 4 * ctx->idx; + header = bpf_jit_binary_hdr(ctx->prog); + goto skip_init_ctx; + } + + ctx->prog = prog; + ctx->offsets = kcalloc(prog->len + 1, + sizeof(*ctx->offsets), + GFP_KERNEL); + if (!ctx->offsets) + goto out_err; + + /* Check Octeon bbit ops only for MIPS64. */ + if (is64bit()) { + preempt_disable(); + switch (current_cpu_type()) { + case CPU_CAVIUM_OCTEON: + case CPU_CAVIUM_OCTEON_PLUS: + case CPU_CAVIUM_OCTEON2: + case CPU_CAVIUM_OCTEON3: + ctx->use_bbit_insns = 1; + break; + default: + ctx->use_bbit_insns = 0; + } + preempt_enable(); + } + + ctx->reg_val_types = kcalloc(prog->len + 1, + sizeof(*ctx->reg_val_types), + GFP_KERNEL); + if (!ctx->reg_val_types) + goto out_err; + + if (reg_val_propagate(ctx)) + goto out_err; + + /* + * First pass discovers used resources and instruction offsets + * assuming short branches are used. + */ + if (build_int_body(ctx)) + goto out_err; + + /* + * If no calls are made (EBPF_SAVE_RA), then tailcall count located + * in runtime reg if defined, else we backup to save reg or stack. + */ + if (tail_call_present(ctx)) { + if (ctx->flags & EBPF_SAVE_RA) + ctx->flags |= bpf2mips[JIT_SAV_TCC].flags; + else if (bpf2mips[JIT_RUN_TCC].reg) + ctx->flags |= EBPF_TCC_IN_RUN; + } + + /* + * Second pass generates offsets, if any branches are out of + * range a jump-around long sequence is generated, and we have + * to try again from the beginning to generate the new + * offsets. This is done until no additional conversions are + * necessary. + */ + do { + ctx->idx = 0; + ctx->gen_b_offsets = 1; + ctx->long_b_conversion = 0; + if (build_int_prologue(ctx)) + goto out_err; + if (build_int_body(ctx)) + goto out_err; + if (build_int_epilogue(ctx, MIPS_R_RA)) + goto out_err; + } while (ctx->long_b_conversion); + + image_size = 4 * ctx->idx; + + header = bpf_jit_binary_alloc(image_size, (void *)&ctx->target, + sizeof(u32), jit_fill_hole); + if (!header) + goto out_err; + +skip_init_ctx: + + /* Third pass generates the code (fourth patches call addresses) */ + ctx->idx = 0; + if (build_int_prologue(ctx)) + goto out_err; + if (build_int_body(ctx)) + goto out_err; + if (build_int_epilogue(ctx, MIPS_R_RA)) + goto out_err; + + if (bpf_jit_enable > 1) + /* Dump JIT code */ + bpf_jit_dump(prog->len, image_size, pass, ctx->target); + + /* Update the icache */ + flush_icache_range((unsigned long)ctx->target, + (unsigned long)&ctx->target[ctx->idx]); + + if (!prog->is_func || extra_pass) + bpf_jit_binary_lock_ro(header); + else + prog->aux->jit_data = ctx; + + prog->bpf_func = (void *)ctx->target; + prog->jited = 1; + prog->jited_len = image_size; + + if (!prog->is_func || extra_pass) { + bpf_prog_fill_jited_linfo(prog, ctx->offsets + 1); +out_ctx: + kfree(ctx->offsets); + kfree(ctx->reg_val_types); + kfree(ctx); + prog->aux->jit_data = NULL; + } +out: + if (tmp_blinded) + bpf_jit_prog_release_other(prog, prog == orig_prog ? + tmp : orig_prog); + return prog; + +out_err: + prog = orig_prog; + if (header) + bpf_jit_binary_free(header); + goto out_ctx; +} + +/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return true; +} From patchwork Tue Oct 5 08:26:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535839 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7CA7C433FE for ; Tue, 5 Oct 2021 08:32:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CDFFC611F2 for ; Tue, 5 Oct 2021 08:32:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233449AbhJEIdx (ORCPT ); Tue, 5 Oct 2021 04:33:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233460AbhJEIdv (ORCPT ); Tue, 5 Oct 2021 04:33:51 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 431FBC061755; Tue, 5 Oct 2021 01:32:01 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id np13so3527450pjb.4; Tue, 05 Oct 2021 01:32:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=raomSAqq/0SNS5jx0FmmwAcCbX5cgON15ayIk5USH+k=; b=hbof3e598NnjQTLsseHPXr0JhjjW+mKHGxdc3ae3XF1e2LgFS+T34g41CRf6dYseEL L3ZyiqTYYSXWz8CCc7rCONfPMUTb0byWzftp7MJgzWvzNJDSRHW2p/4Nz+hQusTp2bcr LJbUKPgavN7rhoqSwJcfHLUWKeM/XllEPiEMvUMNlZG1ZlXJdNSvumUg6VSCoStrARYa 4XVzQNhLsP6rZ0tvm+Cq8UrhhyOtjIs4+ycyyfg7Znlw0EFDtg2VnGJTRc1wyPhRWLaQ Uo7SVTzcRQpixrJBvsyxyS+lQAPm/Xiv9KcaYV9yzLi7YpOL2kAI7g9f2B8V/hpgukOE 8vvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=raomSAqq/0SNS5jx0FmmwAcCbX5cgON15ayIk5USH+k=; b=D5cGKoGCIEuaS5IaAkO0vRbDmsUBXFO3s5TA8A5cGdnjKtrzo/ULxoqSfKSjCSX5jq CRZq8UzoWNZiUzufsBaJ/g1Krea5M4zB3AvbVKBdC9laNR4Y0AGS3phzyt5Hk3Ktza4M d00uM9v2eZda+fJCIXiGi5hwAkG0Y6GK8wRCpBp/chd6jRLr+tAdLzszVD54eyk1DRao f3xLvET95KnHqBaZlFKbWDe+/+40mL8C2wNScR0miyoCRZPZV+rNzWvwee4NXlOR4VrQ SgnD+H7+G2JrYPHbICsKrAGDgL5umxykOntHn5Wh2YTBIZ6Blw71Gt3BWgj9WyexCJRX 9LQA== X-Gm-Message-State: AOAM532gMK776PLmhwWRyu6CPbfdTzexxBG0ceuC2+KCWyinAJ4J+dsJ MkWUcJfVR3HQTFdSLrpQWayTBBEiIollLA== X-Google-Smtp-Source: ABdhPJxlNiWNVI/6/Rqd2XUpaETVGe7ea05cNnZ37AKZA9CCKX40im3MSK3H8YVsMN+CP6KLeAWewQ== X-Received: by 2002:a17:90b:4f88:: with SMTP id qe8mr2220996pjb.223.1633422720675; Tue, 05 Oct 2021 01:32:00 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.31.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:32:00 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 13/16] MIPS: eBPF64: support BPF_JMP32 conditionals Date: Tue, 5 Oct 2021 01:26:57 -0700 Message-Id: <8af45dc454a5f1dcf024e9b511bb65b8600c65be.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Simplify BPF_JGT/JLE/JSGT/JSLE and related code by dropping unneeded usage of MIPS_R_T8/T9 registers and jump-around branches, and extra comparison arithmetic. Also reorganize var declarations and add 'bpf_class' helper constant. Implement BPF_JMP32 branches using sign or zero-extended temporaries as needed for comparisons. This enables JITing of many more BPF programs, and provides greater test coverage by e.g. 'test_verifier'. Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit_comp64.c | 206 +++++++++++++++++++------------- 1 file changed, 122 insertions(+), 84 deletions(-) diff --git a/arch/mips/net/ebpf_jit_comp64.c b/arch/mips/net/ebpf_jit_comp64.c index c38d93d37ce3..842e516ce749 100644 --- a/arch/mips/net/ebpf_jit_comp64.c +++ b/arch/mips/net/ebpf_jit_comp64.c @@ -167,12 +167,13 @@ static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, int this_idx, int exit_idx) { - int src, dst, r, td, ts, mem_off, b_off; + const int bpf_class = BPF_CLASS(insn->code); + const int bpf_op = BPF_OP(insn->code); bool need_swap, did_move, cmp_eq; unsigned int target = 0; + int src, dst, r, td, ts; + int mem_off, b_off; u64 t64; - s64 t64s; - int bpf_op = BPF_OP(insn->code); switch (insn->code) { case BPF_ALU64 | BPF_ADD | BPF_K: /* ALU64_IMM */ @@ -500,7 +501,9 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JNE | BPF_K: + case BPF_JMP32 | BPF_JEQ | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JNE | BPF_K: cmp_eq = (bpf_op == BPF_JEQ); dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) @@ -511,6 +514,16 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, gen_imm_to_reg(insn, MIPS_R_AT, ctx); src = MIPS_R_AT; } + if (bpf_class == BPF_JMP32) { + emit_instr(ctx, move, MIPS_R_T8, dst); + emit_instr(ctx, dinsu, MIPS_R_T8, MIPS_R_ZERO, 32, 32); + dst = MIPS_R_T8; + if (src != MIPS_R_ZERO) { + emit_instr(ctx, move, MIPS_R_T9, src); + emit_instr(ctx, dinsu, MIPS_R_T9, MIPS_R_ZERO, 32, 32); + src = MIPS_R_T9; + } + } goto jeq_common; case BPF_JMP | BPF_JEQ | BPF_X: /* JMP_REG */ case BPF_JMP | BPF_JNE | BPF_X: @@ -523,18 +536,46 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JGT | BPF_X: case BPF_JMP | BPF_JGE | BPF_X: case BPF_JMP | BPF_JSET | BPF_X: + case BPF_JMP32 | BPF_JEQ | BPF_X: /* JMP_REG */ + case BPF_JMP32 | BPF_JNE | BPF_X: + case BPF_JMP32 | BPF_JSLT | BPF_X: + case BPF_JMP32 | BPF_JSLE | BPF_X: + case BPF_JMP32 | BPF_JSGT | BPF_X: + case BPF_JMP32 | BPF_JSGE | BPF_X: + case BPF_JMP32 | BPF_JLT | BPF_X: + case BPF_JMP32 | BPF_JLE | BPF_X: + case BPF_JMP32 | BPF_JGT | BPF_X: + case BPF_JMP32 | BPF_JGE | BPF_X: + case BPF_JMP32 | BPF_JSET | BPF_X: src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (src < 0 || dst < 0) return -EINVAL; td = get_reg_val_type(ctx, this_idx, insn->dst_reg); ts = get_reg_val_type(ctx, this_idx, insn->src_reg); - if (td == REG_32BIT && ts != REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, src, 0); - src = MIPS_R_AT; - } else if (ts == REG_32BIT && td != REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, dst, 0); - dst = MIPS_R_AT; + if (bpf_class == BPF_JMP) { + if (td == REG_32BIT && ts != REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_AT, src, 0); + src = MIPS_R_AT; + } else if (ts == REG_32BIT && td != REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_AT, dst, 0); + dst = MIPS_R_AT; + } + } else { /* BPF_JMP32 */ + if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLT || + bpf_op == BPF_JSGE || bpf_op == BPF_JSLE) { + emit_instr(ctx, sll, MIPS_R_T8, dst, 0); + emit_instr(ctx, sll, MIPS_R_T9, src, 0); + dst = MIPS_R_T8; + src = MIPS_R_T9; + } else { + emit_instr(ctx, move, MIPS_R_T8, dst); + emit_instr(ctx, move, MIPS_R_T9, src); + emit_instr(ctx, dinsu, MIPS_R_T8, MIPS_R_ZERO, 32, 32); + emit_instr(ctx, dinsu, MIPS_R_T9, MIPS_R_ZERO, 32, 32); + dst = MIPS_R_T8; + src = MIPS_R_T9; + } } if (bpf_op == BPF_JSET) { emit_instr(ctx, and, MIPS_R_AT, dst, src); @@ -542,48 +583,20 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, dst = MIPS_R_AT; src = MIPS_R_ZERO; } else if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLE) { - emit_instr(ctx, dsubu, MIPS_R_AT, dst, src); - if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - if (bpf_op == BPF_JSGT) - emit_instr(ctx, blez, MIPS_R_AT, b_off); - else - emit_instr(ctx, bgtz, MIPS_R_AT, b_off); - emit_instr(ctx, nop); - return 2; /* We consumed the exit. */ - } - b_off = b_imm(this_idx + insn->off + 1, ctx); - if (is_bad_offset(b_off)) - return -E2BIG; - if (bpf_op == BPF_JSGT) - emit_instr(ctx, bgtz, MIPS_R_AT, b_off); - else - emit_instr(ctx, blez, MIPS_R_AT, b_off); - emit_instr(ctx, nop); - break; + /* swap dst and src to simplify comparison */ + emit_instr(ctx, slt, MIPS_R_AT, src, dst); + cmp_eq = bpf_op == BPF_JSLE; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; } else if (bpf_op == BPF_JSGE || bpf_op == BPF_JSLT) { emit_instr(ctx, slt, MIPS_R_AT, dst, src); cmp_eq = bpf_op == BPF_JSGE; dst = MIPS_R_AT; src = MIPS_R_ZERO; } else if (bpf_op == BPF_JGT || bpf_op == BPF_JLE) { - /* dst or src could be AT */ - emit_instr(ctx, dsubu, MIPS_R_T8, dst, src); - emit_instr(ctx, sltu, MIPS_R_AT, dst, src); - /* SP known to be non-zero, movz becomes boolean not */ - if (MIPS_ISA_REV >= 6) { - emit_instr(ctx, seleqz, MIPS_R_T9, - MIPS_R_SP, MIPS_R_T8); - } else { - emit_instr(ctx, movz, MIPS_R_T9, - MIPS_R_SP, MIPS_R_T8); - emit_instr(ctx, movn, MIPS_R_T9, - MIPS_R_ZERO, MIPS_R_T8); - } - emit_instr(ctx, or, MIPS_R_AT, MIPS_R_T9, MIPS_R_AT); - cmp_eq = bpf_op == BPF_JGT; + /* swap dst and src to simplify comparison */ + emit_instr(ctx, sltu, MIPS_R_AT, src, dst); + cmp_eq = bpf_op == BPF_JLE; dst = MIPS_R_AT; src = MIPS_R_ZERO; } else if (bpf_op == BPF_JGE || bpf_op == BPF_JLT) { @@ -650,14 +663,20 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; case BPF_JMP | BPF_JSGT | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSGE | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSLT | BPF_K: /* JMP_IMM */ - case BPF_JMP | BPF_JSLE | BPF_K: /* JMP_IMM */ - cmp_eq = (bpf_op == BPF_JSGE); + case BPF_JMP | BPF_JSGE | BPF_K: + case BPF_JMP | BPF_JSLT | BPF_K: + case BPF_JMP | BPF_JSLE | BPF_K: + case BPF_JMP32 | BPF_JSGT | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JSGE | BPF_K: + case BPF_JMP32 | BPF_JSLT | BPF_K: + case BPF_JMP32 | BPF_JSLE | BPF_K: dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; - + if (bpf_class == BPF_JMP32) { + emit_instr(ctx, sll, MIPS_R_T8, dst, 0); + dst = MIPS_R_T8; + } if (insn->imm == 0) { if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { b_off = b_imm(exit_idx, ctx); @@ -700,26 +719,24 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit_instr(ctx, nop); break; } - /* - * only "LT" compare available, so we must use imm + 1 - * to generate "GT" and imm -1 to generate LE - */ - if (bpf_op == BPF_JSGT) - t64s = insn->imm + 1; - else if (bpf_op == BPF_JSLE) - t64s = insn->imm + 1; - else - t64s = insn->imm; - cmp_eq = bpf_op == BPF_JSGT || bpf_op == BPF_JSGE; - if (t64s >= S16_MIN && t64s <= S16_MAX) { - emit_instr(ctx, slti, MIPS_R_AT, dst, (int)t64s); - src = MIPS_R_AT; - dst = MIPS_R_ZERO; - goto jeq_common; + gen_imm_to_reg(insn, MIPS_R_T9, ctx); + src = MIPS_R_T9; + cmp_eq = bpf_op == BPF_JSLE || bpf_op == BPF_JSGE; + switch (bpf_op) { + case BPF_JSGE: + emit_instr(ctx, slt, MIPS_R_AT, dst, src); + break; + case BPF_JSLT: + emit_instr(ctx, slt, MIPS_R_AT, dst, src); + break; + case BPF_JSGT: + emit_instr(ctx, slt, MIPS_R_AT, src, dst); + break; + case BPF_JSLE: + emit_instr(ctx, slt, MIPS_R_AT, src, dst); + break; } - emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); - emit_instr(ctx, slt, MIPS_R_AT, dst, MIPS_R_AT); src = MIPS_R_AT; dst = MIPS_R_ZERO; goto jeq_common; @@ -728,33 +745,52 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_JMP | BPF_JGE | BPF_K: case BPF_JMP | BPF_JLT | BPF_K: case BPF_JMP | BPF_JLE | BPF_K: - cmp_eq = (bpf_op == BPF_JGE); + case BPF_JMP32 | BPF_JGT | BPF_K: + case BPF_JMP32 | BPF_JGE | BPF_K: + case BPF_JMP32 | BPF_JLT | BPF_K: + case BPF_JMP32 | BPF_JLE | BPF_K: dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; - /* - * only "LT" compare available, so we must use imm + 1 - * to generate "GT" and imm -1 to generate LE - */ - if (bpf_op == BPF_JGT) - t64s = (u64)(u32)(insn->imm) + 1; - else if (bpf_op == BPF_JLE) - t64s = (u64)(u32)(insn->imm) + 1; - else - t64s = (u64)(u32)(insn->imm); - - cmp_eq = bpf_op == BPF_JGT || bpf_op == BPF_JGE; + if (bpf_class == BPF_JMP32) { + emit_instr(ctx, move, MIPS_R_T8, dst); + gen_zext_insn(MIPS_R_T8, true, ctx); + dst = MIPS_R_T8; + } + gen_imm_to_reg(insn, MIPS_R_T9, ctx); + if (bpf_class == BPF_JMP32 && insn->imm < 0) + gen_zext_insn(MIPS_R_T9, true, ctx); + src = MIPS_R_T9; - emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s); - emit_instr(ctx, sltu, MIPS_R_AT, dst, MIPS_R_AT); + cmp_eq = bpf_op == BPF_JLE || bpf_op == BPF_JGE; + switch (bpf_op) { + case BPF_JGE: + emit_instr(ctx, sltu, MIPS_R_AT, dst, src); + break; + case BPF_JLT: + emit_instr(ctx, sltu, MIPS_R_AT, dst, src); + break; + case BPF_JGT: + emit_instr(ctx, sltu, MIPS_R_AT, src, dst); + break; + case BPF_JLE: + emit_instr(ctx, sltu, MIPS_R_AT, src, dst); + break; + } src = MIPS_R_AT; dst = MIPS_R_ZERO; goto jeq_common; case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JSET | BPF_K: /* JMP_IMM */ dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); if (dst < 0) return dst; + if (bpf_class == BPF_JMP32) { + emit_instr(ctx, move, MIPS_R_T8, dst); + emit_instr(ctx, dinsu, MIPS_R_T8, MIPS_R_ZERO, 32, 32); + dst = MIPS_R_T8; + } if (ctx->use_bbit_insns && hweight32((u32)insn->imm) == 1) { if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { @@ -774,6 +810,8 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } t64 = (u32)insn->imm; emit_const_to_reg(ctx, MIPS_R_AT, t64); + if (bpf_class == BPF_JMP32) + emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); emit_instr(ctx, and, MIPS_R_AT, dst, MIPS_R_AT); src = MIPS_R_AT; dst = MIPS_R_ZERO; From patchwork Tue Oct 5 08:26:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535881 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B58B9C433FE for ; Tue, 5 Oct 2021 08:39:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 98DD561130 for ; Tue, 5 Oct 2021 08:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233271AbhJEIlJ (ORCPT ); Tue, 5 Oct 2021 04:41:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233002AbhJEIlJ (ORCPT ); Tue, 5 Oct 2021 04:41:09 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9462C061745; Tue, 5 Oct 2021 01:39:18 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id d18so23363401iof.13; Tue, 05 Oct 2021 01:39:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vRAxmubyADiVGnedQuvE2MHzC9GyBk/oPvl54a5kj8k=; b=H3/bD4ruK7c89f4rRt0oMJD3H5C6OYZpYeTFmO74MrOWGrYEDs9gx5A0eZKOLHeFfP VhXme9AvQalS9rekOpaUf3xVVIe/N774f6tonzu3R/6ZVvUuJ4nnozm4cUId59fA+2cD IyuweqfHJmCTL3JpjhiV8E5oG/AEDPTtIi/Yp1pJhNL9RIIHP3/6flaBGurRCOgJdRqr UbHz/QLLP0mM3S4CVlN2WxJcNEylO6QorqPLkZrqUABWY6mDLSoclkPFasW+GWOkzpB2 NPe57of2oibWf1IuaWJSC29MQngnFs5wtSBoJ42F6y28P8qk1KbLsRGQlSCL63D+esSl h34Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vRAxmubyADiVGnedQuvE2MHzC9GyBk/oPvl54a5kj8k=; b=B60fhZe0Lbi55nvWkoxYQDcdGynAsUIAFv4Y14WEEdaLQYAIsJ1lYCdErhTHQ5S9Sv KoH+IO8upyY9oOLjIe2XCv8cC6zTNjDyNrZu5yQp7oxmi7jMVNLcX1zeM7AtdwmogU0h aoGTAH9Ge945bNHNKtpKXCZpC8R4zLzIxs/F1SnqJH8ecPqFnhEK1ZqIdEFJweyeuINE wDmFz8BxbIbkMNsM9mNoXeYCpRE+JUEF9ZZp5xPeJE8FzkdVPm2DQSF/Duloqb0p2OZf 9jstF320mzFMFwuVlluHKz/J0ghPd64MgXP+3Z8I2yb6jareUWkDCzzD4mZfjP9Qvzsm AUgQ== X-Gm-Message-State: AOAM5316H/9wSFSPm7c8HWDfUd8EwgMr+MYqNBPavyAJo460G8qMcHI/ YqpQTD0kO5Vc2YCBYeD6gxFiJKsAA3T2Vw== X-Google-Smtp-Source: ABdhPJyAn4+mgqMuaK99WTG4Qb3Q4N8lzJX6ftX6PCSdMN8k4MrclknyAt1h2SD/t/JgJ3sDw9zZxg== X-Received: by 2002:a63:b34a:: with SMTP id x10mr14611613pgt.473.1633422728971; Tue, 05 Oct 2021 01:32:08 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.32.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:32:08 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 14/16] MIPS: eBPF64: implement all BPF_ATOMIC ops Date: Tue, 5 Oct 2021 01:26:58 -0700 Message-Id: <3499e013eb8994077c8f642fb1907a49d6a0afd7.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Reorganize code for BPF_ATOMIC and BPF_MEM, and add the atomic ops AND, OR, XOR, XCHG and CMPXCHG, with support for BPF_FETCH. Signed-off-by: Tony Ambardar --- arch/mips/net/ebpf_jit_comp64.c | 181 +++++++++++++++++++++----------- 1 file changed, 119 insertions(+), 62 deletions(-) diff --git a/arch/mips/net/ebpf_jit_comp64.c b/arch/mips/net/ebpf_jit_comp64.c index 842e516ce749..35c8c8307b64 100644 --- a/arch/mips/net/ebpf_jit_comp64.c +++ b/arch/mips/net/ebpf_jit_comp64.c @@ -167,7 +167,15 @@ static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, int this_idx, int exit_idx) { + /* + * Since CMPXCHG uses R0 implicitly, outside of a passed + * bpf_insn, we fake a lookup to get the MIPS base reg. + */ + const struct bpf_insn r0_insn = {.src_reg = BPF_REG_0}; + const int r0 = ebpf_to_mips_reg(ctx, &r0_insn, + REG_SRC_NO_FP); const int bpf_class = BPF_CLASS(insn->code); + const int bpf_size = BPF_SIZE(insn->code); const int bpf_op = BPF_OP(insn->code); bool need_swap, did_move, cmp_eq; unsigned int target = 0; @@ -944,6 +952,32 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, case BPF_STX | BPF_H | BPF_MEM: case BPF_STX | BPF_W | BPF_MEM: case BPF_STX | BPF_DW | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + switch (BPF_SIZE(insn->code)) { + case BPF_B: + emit_instr(ctx, sb, src, mem_off, dst); + break; + case BPF_H: + emit_instr(ctx, sh, src, mem_off, dst); + break; + case BPF_W: + emit_instr(ctx, sw, src, mem_off, dst); + break; + case BPF_DW: + if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { + emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); + emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); + src = MIPS_R_AT; + } + emit_instr(ctx, sd, src, mem_off, dst); + break; + } + break; + case BPF_STX | BPF_W | BPF_ATOMIC: case BPF_STX | BPF_DW | BPF_ATOMIC: dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); @@ -951,71 +985,94 @@ int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, if (src < 0 || dst < 0) return -EINVAL; mem_off = insn->off; - if (BPF_MODE(insn->code) == BPF_ATOMIC) { - if (insn->imm != BPF_ADD) { - pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); - return -EINVAL; + /* + * If mem_off does not fit within the 9 bit ll/sc + * instruction immediate field, use a temp reg. + */ + if (MIPS_ISA_REV >= 6 && + (mem_off >= BIT(8) || mem_off < -BIT(8))) { + emit_instr(ctx, daddiu, MIPS_R_T6, dst, mem_off); + mem_off = 0; + dst = MIPS_R_T6; + } + /* Copy or adjust 32-bit src regs based on BPF op size. */ + ts = get_reg_val_type(ctx, this_idx, insn->src_reg); + if (bpf_size == BPF_W) { + if (ts == REG_32BIT) { + emit_instr(ctx, sll, MIPS_R_T9, src, 0); + src = MIPS_R_T9; } + /* Ensure proper old == new comparison .*/ + if (insn->imm == BPF_CMPXCHG) + emit_instr(ctx, sll, r0, r0, 0); + } + if (bpf_size == BPF_DW && ts == REG_32BIT) { + emit_instr(ctx, move, MIPS_R_T9, src); + emit_instr(ctx, dinsu, MIPS_R_T9, MIPS_R_ZERO, 32, 32); + src = MIPS_R_T9; + } + +/* Helper to simplify using BPF_DW/BPF_W atomic opcodes. */ +#define emit_instr_size(ctx, func64, func32, ...) \ +do { \ + if (bpf_size == BPF_DW) \ + emit_instr(ctx, func64, ##__VA_ARGS__); \ + else \ + emit_instr(ctx, func32, ##__VA_ARGS__); \ +} while (0) + + /* Track variable branch offset due to CMPXCHG. */ + b_off = ctx->idx; + emit_instr_size(ctx, lld, ll, MIPS_R_AT, mem_off, dst); + switch (insn->imm) { + case BPF_AND | BPF_FETCH: + case BPF_AND: + emit_instr(ctx, and, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_OR | BPF_FETCH: + case BPF_OR: + emit_instr(ctx, or, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_XOR | BPF_FETCH: + case BPF_XOR: + emit_instr(ctx, xor, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_ADD | BPF_FETCH: + case BPF_ADD: + emit_instr_size(ctx, daddu, addu, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_XCHG: + emit_instr_size(ctx, daddu, addu, MIPS_R_T8, MIPS_R_ZERO, src); + break; + case BPF_CMPXCHG: /* - * If mem_off does not fit within the 9 bit ll/sc - * instruction immediate field, use a temp reg. + * If R0 != old_val then break out of LL/SC loop */ - if (MIPS_ISA_REV >= 6 && - (mem_off >= BIT(8) || mem_off < -BIT(8))) { - emit_instr(ctx, daddiu, MIPS_R_T6, - dst, mem_off); - mem_off = 0; - dst = MIPS_R_T6; - } - switch (BPF_SIZE(insn->code)) { - case BPF_W: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, sll, MIPS_R_AT, src, 0); - src = MIPS_R_AT; - } - emit_instr(ctx, ll, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, addu, MIPS_R_T8, MIPS_R_T8, src); - emit_instr(ctx, sc, MIPS_R_T8, mem_off, dst); - /* - * On failure back up to LL (-4 - * instructions of 4 bytes each - */ - emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); - emit_instr(ctx, nop); - break; - case BPF_DW: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); - emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); - src = MIPS_R_AT; - } - emit_instr(ctx, lld, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, src); - emit_instr(ctx, scd, MIPS_R_T8, mem_off, dst); - emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4); - emit_instr(ctx, nop); - break; - } - } else { /* BPF_MEM */ - switch (BPF_SIZE(insn->code)) { - case BPF_B: - emit_instr(ctx, sb, src, mem_off, dst); - break; - case BPF_H: - emit_instr(ctx, sh, src, mem_off, dst); - break; - case BPF_W: - emit_instr(ctx, sw, src, mem_off, dst); - break; - case BPF_DW: - if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) { - emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO); - emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); - src = MIPS_R_AT; - } - emit_instr(ctx, sd, src, mem_off, dst); - break; - } + emit_instr(ctx, bne, r0, MIPS_R_AT, 4 * 4); + /* Delay slot */ + emit_instr_size(ctx, daddu, addu, MIPS_R_T8, MIPS_R_ZERO, src); + /* Return old_val in R0 */ + src = r0; + break; + default: + pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); + return -EINVAL; + } + emit_instr_size(ctx, scd, sc, MIPS_R_T8, mem_off, dst); +#undef emit_instr_size + /* + * On failure back up to LL (calculate # insns) + */ + b_off = (b_off - ctx->idx - 1) * 4; + emit_instr(ctx, beqz, MIPS_R_T8, b_off); + emit_instr(ctx, nop); + /* + * Using fetch returns old value in src or R0 + */ + if (insn->imm & BPF_FETCH) { + if (bpf_size == BPF_W) + emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32); + emit_instr(ctx, move, src, MIPS_R_AT); } break; From patchwork Tue Oct 5 08:26:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535841 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA29C433EF for ; Tue, 5 Oct 2021 08:32:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7FACF611F2 for ; Tue, 5 Oct 2021 08:32:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233285AbhJEIea (ORCPT ); Tue, 5 Oct 2021 04:34:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232108AbhJEIe3 (ORCPT ); Tue, 5 Oct 2021 04:34:29 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50A1FC061745; Tue, 5 Oct 2021 01:32:39 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id g14so16824395pfm.1; Tue, 05 Oct 2021 01:32:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oB80oYMulhrt43Nc7h4b8WTmJ925bEkBYiIlwoCVavE=; b=LgU8mqTxrXP9w5FDf88vbtQB9pIkvHyP7Snhffc78rg77kw1jneS0Sy7xl5qFPvSpD IEXqLMHsQvmL8Xh0eh5EbfhUgR06klJAs6a4bBHMTpfqktP7HfLTny/NBLAqMwXBGmC5 t+ROAI14vIpQpsHOH4QDgAz1e3pNCqdtzLM98HdCyrLSlfGmvk5QrmNtH8mccgCzO13I Om5erayhNB9sS/ul8EcqUCkw1NwmE6crq4qIWNSIxMWF3ae7w/oFgYfGKwQCkW9oVSsW HuMqyE14TgXm7+b9PMCG8kbC7flvvRWJlkYvHBpynpl+IzvK38i6WZLAGq+Rce5vKYuB EWug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oB80oYMulhrt43Nc7h4b8WTmJ925bEkBYiIlwoCVavE=; b=CIkjmJVk65mWiFnxsfskbvwRAhg1ehLxExzr3xDOy4R/NH9/ZPw/N9uVQs6l7+3tru dY9ljidW05fcWvCpvSsN98R2H2mAYrCbNG0x/67emL9EdEz9rOM3phBk2w6zmAomKOvT 0JcyJ6tNk/BVpxsu2e/QuJl12Tq2162cXT6ubNowkH/rr2oQapUhV39FAq7NC6EkGpVJ 5k9Pyaj1GV1OxbC86utjfqVaDaFa5Y57IJ0fbyphvtvriTAmqTdfZf13lzsHGjjUyES/ GOD6dpXJ5AVTne6f82lphhKjWo2N3cMPPyzyazQ5XljwbCvEWzKcaj+vJ50nQAwXLwq7 nOrg== X-Gm-Message-State: AOAM5337DhUSYMp+9pH8lHfI9Wn9AVx4HS3yiHpZoRZK5V6f8CqwnH9m gtW+ERX8jj2oeH1T/P3lnaE= X-Google-Smtp-Source: ABdhPJwadmGEPkjSjq1bvQexkQA0RWI7UEAOIDW4pkpcLRXMENbOdsYnuFGNaqU/vrA8TFzFq61suA== X-Received: by 2002:a05:6a00:98b:b0:44b:df34:c17b with SMTP id u11-20020a056a00098b00b0044bdf34c17bmr29657149pfg.34.1633422758941; Tue, 05 Oct 2021 01:32:38 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:32:38 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 15/16] MIPS: uasm: Enable muhu opcode for MIPS R6 Date: Tue, 5 Oct 2021 01:26:59 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Enable the 'muhu' instruction, complementing the existing 'mulu', needed to implement a MIPS32 BPF JIT. Also fix a typo in the existing definition of 'dmulu'. Signed-off-by: Tony Ambardar --- arch/mips/include/asm/uasm.h | 1 + arch/mips/mm/uasm-mips.c | 4 +++- arch/mips/mm/uasm.c | 3 ++- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h index f7effca791a5..5efa4e2dc9ab 100644 --- a/arch/mips/include/asm/uasm.h +++ b/arch/mips/include/asm/uasm.h @@ -145,6 +145,7 @@ Ip_u1(_mtlo); Ip_u3u1u2(_mul); Ip_u1u2(_multu); Ip_u3u1u2(_mulu); +Ip_u3u1u2(_muhu); Ip_u3u1u2(_nor); Ip_u3u1u2(_or); Ip_u2u1u3(_ori); diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c index 7154a1d99aad..e15c6700cd08 100644 --- a/arch/mips/mm/uasm-mips.c +++ b/arch/mips/mm/uasm-mips.c @@ -90,7 +90,7 @@ static const struct insn insn_table[insn_invalid] = { RS | RT | RD}, [insn_dmtc0] = {M(cop0_op, dmtc_op, 0, 0, 0, 0), RT | RD | SET}, [insn_dmultu] = {M(spec_op, 0, 0, 0, 0, dmultu_op), RS | RT}, - [insn_dmulu] = {M(spec_op, 0, 0, 0, dmult_dmul_op, dmultu_op), + [insn_dmulu] = {M(spec_op, 0, 0, 0, dmultu_dmulu_op, dmultu_op), RS | RT | RD}, [insn_drotr] = {M(spec_op, 1, 0, 0, 0, dsrl_op), RT | RD | RE}, [insn_drotr32] = {M(spec_op, 1, 0, 0, 0, dsrl32_op), RT | RD | RE}, @@ -150,6 +150,8 @@ static const struct insn insn_table[insn_invalid] = { [insn_mtlo] = {M(spec_op, 0, 0, 0, 0, mtlo_op), RS}, [insn_mulu] = {M(spec_op, 0, 0, 0, multu_mulu_op, multu_op), RS | RT | RD}, + [insn_muhu] = {M(spec_op, 0, 0, 0, multu_muhu_op, multu_op), + RS | RT | RD}, #ifndef CONFIG_CPU_MIPSR6 [insn_mul] = {M(spec2_op, 0, 0, 0, 0, mul_op), RS | RT | RD}, #else diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c index 81dd226d6b6b..125140979d62 100644 --- a/arch/mips/mm/uasm.c +++ b/arch/mips/mm/uasm.c @@ -59,7 +59,7 @@ enum opcode { insn_lddir, insn_ldpte, insn_ldx, insn_lh, insn_lhu, insn_ll, insn_lld, insn_lui, insn_lw, insn_lwu, insn_lwx, insn_mfc0, insn_mfhc0, insn_mfhi, insn_mflo, insn_modu, insn_movn, insn_movz, insn_mtc0, insn_mthc0, - insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu, insn_nor, + insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu, insn_muhu, insn_nor, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sb, insn_sc, insn_scd, insn_seleqz, insn_selnez, insn_sd, insn_sh, insn_sll, insn_sllv, insn_slt, insn_slti, insn_sltiu, insn_sltu, insn_sra, @@ -344,6 +344,7 @@ I_u1(_mtlo) I_u3u1u2(_mul) I_u1u2(_multu) I_u3u1u2(_mulu) +I_u3u1u2(_muhu) I_u3u1u2(_nor) I_u3u1u2(_or) I_u2u1u3(_ori) From patchwork Tue Oct 5 08:27:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Ambardar X-Patchwork-Id: 12535843 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60E63C433FE for ; Tue, 5 Oct 2021 08:32:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 43DBD61056 for ; Tue, 5 Oct 2021 08:32:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232997AbhJEIeq (ORCPT ); Tue, 5 Oct 2021 04:34:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232108AbhJEIeq (ORCPT ); Tue, 5 Oct 2021 04:34:46 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E69BC061745; Tue, 5 Oct 2021 01:32:56 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id on6so2667245pjb.5; Tue, 05 Oct 2021 01:32:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WGzG2E6uBjhCwvl7ERMbgVWMOyPXLCcfpUeO/GgX/OI=; b=bzNH7UHFnSWXArXwt2Gi9qEVPORCCJ+mXd7v+IV9gj0sa97KzbjfVIrCnhkHGZV/BK FT14uBOBFA/rM26d0fk4y8hgGbHnDgXPp+k9LxQe73nGefjTX84YhMw05XSsV+tMXSTR 03zdNmo4KDtEYLhFvxkpYT8hwT+tVqtCDcAJ8jzexZ21Ofjm3JksMpEhG1x92vsJ2+Kz DzakeNjzu8cf9CeJgz48BeBuNpZuM1S8CnHZPDero0thG9S1bpyTbk1mQzPV5oQowEtX Xgey0k8oGYypC3XHyO4w9GEdAEfEacYUGtkFIM6iH6+KUERj4b3RPM4V8hweFoLwDwbz kU2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WGzG2E6uBjhCwvl7ERMbgVWMOyPXLCcfpUeO/GgX/OI=; b=peL9EhxJoBVpbYiC1igO7NtCcd2PYfXUU59y+F3lKg35hO1RF3oZHo8KhMexPX3FsT mSeTwDpCETTuR28QtVQSk/t/AKHBpRyIhtW2mYt0x6VhO39uVxiPa/ou25pcW+6RR5TB 2f/pno3RSt8y5Q5odwUuWl7j6sbLPVci3nNqlE4tWuhb36OaEdEXDoKH7gar8qhb0okn JZgzKNWuO0uA+fc2AmcDcD5eNrUDwxAkiPw0P9cwXiHEmwXNmXpozhorw2+qqx3PPGVl 6+e43+GXfOHIg72wqJHkuRbvpPXUmpVUbITXvZUqhS5CBcm6YEkcfaJjBwDBlmXfOpur vaiQ== X-Gm-Message-State: AOAM531RL/82ewugR1fOcRKnrrhvgmb/lRH0H0NvQHUeeXIRQyMUErIM Mk6piJcVH5ZXwFV+TDd8W00= X-Google-Smtp-Source: ABdhPJyb0a9MkoHkUJDlTMm5dJ8YQ+fs8ELC7Y450w9iejUxC0N9nZLf8ha1yfk3lsok7XH2nT+s5g== X-Received: by 2002:a17:90b:1c81:: with SMTP id oo1mr2220449pjb.97.1633422775320; Tue, 05 Oct 2021 01:32:55 -0700 (PDT) Received: from localhost.localdomain ([2001:470:e92d:10:78ba:4bcc:a59a:2284]) by smtp.gmail.com with ESMTPSA id a15sm4941257pfg.53.2021.10.05.01.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 01:32:55 -0700 (PDT) From: Tony Ambardar X-Google-Original-From: Tony Ambardar To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Thomas Bogendoerfer , Paul Burton Cc: Tony Ambardar , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mips@vger.kernel.org, Johan Almbladh , Tiezhu Yang , Hassan Naveed , David Daney , Luke Nelson , Serge Semin , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [RFC PATCH bpf-next v2 16/16] MIPS: eBPF: add MIPS32 JIT Date: Tue, 5 Oct 2021 01:27:00 -0700 Message-Id: <1f04a1d0d303e6690e56eb8c5139d68be4e9280a.1633392335.git.Tony.Ambardar@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add a new variant of build_one_insn() supporting MIPS32, leveraging the previously added common functions, and disable static analysis as unneeded on MIPS32. Also define bpf_jit_needs_zext() to request verifier zext insertion. Handle these zext insns, and add conditional zext for all ALU32 and LDX word-size operations for cases where the verifier is unable to do so (e.g. test_bpf bypasses verifier). Aside from mapping 64-bit BPF registers to pairs of 32-bit MIPS registers, notable changes from the MIPS64 version of build_one_insn() include: BPF_ALU{,64} | {DIV,MOD} | BPF_K: drop divide-by-zero guard as the insns underlying do not raise exceptions. BPF_JMP | JSET | BPF_K: drop bbit insns only usable on MIPS64 Octeon. In addition to BPF_ADD, implement atomic insns AND, OR, XOR, XCHG and CMPXCHG, together with support for BPF_FETCH, in mode BPF_W. The MIPS32 ISA does not include 64-bit div/mod or atomic opcodes. Add the emit_bpf_divmod64() and emit_bpf_atomic64() functions, which use built-in kernel functions to implement the following BPF insns: BPF_STX | BPF_DW | BPF_ATOMIC (AND, OR, XOR, XCHG, CMPXCHG, FETCH) BPF_ALU64 | BPF_DIV | BPF_X BPF_ALU64 | BPF_DIV | BPF_K BPF_ALU64 | BPF_MOD | BPF_X BPF_ALU64 | BPF_MOD | BPF_K Test and development primarily used LTS kernel 5.10.x and then 5.13.x, running under QEMU. Test suites included the 'test_bpf' module and the 'test_verifier' program from kselftests. Testing with 'test_progs' from kselftests was not possible in general since cross-compilation depends on libbpf/bpftool, which does not support cross-endian builds (see also [1]). The matrix of test configurations executed for this series covers: MIPSWORD={64-bit,32-bit} x MIPSISA={R2,R6} x JIT={off,on,hardened} On MIPS32BE and MIPS32LE there was general parity between the results of interpreter vs. JIT-backed tests with respect to the numbers of PASSED, SKIPPED, and FAILED tests. A sample of results on QEMU/MIPS32LE from kernel 5.13.x: root@OpenWrt:~# sysctl net.core.bpf_jit_enable=1 root@OpenWrt:~# modprobe test_bpf ... test_bpf: Summary: 378 PASSED, 0 FAILED, [366/366 JIT'ed] root@OpenWrt:~# ./test_verifier 0 884 ... Summary: 1231 PASSED, 0 SKIPPED, 20 FAILED root@OpenWrt:~# ./test_verifier 886 1184 ... Summary: 459 PASSED, 1 SKIPPED, 2 FAILED root@OpenWrt:~# ./test_progs -n 105,106 ... 105 subprogs:OK 106/1 tailcall_1:OK 106/2 tailcall_2:OK 106/3 tailcall_3:OK 106/4 tailcall_4:OK 106/5 tailcall_5:OK 106/6 tailcall_bpf2bpf_1:OK 106/7 tailcall_bpf2bpf_2:OK 106/8 tailcall_bpf2bpf_3:OK 106/9 tailcall_bpf2bpf_4:OK 106 tailcalls:OK Summary: 2/9 PASSED, 0 SKIPPED, 0 FAILED Link: [1] https://lore.kernel.org/bpf/CAEf4BzZCnP3oB81w4BDL4TCmvO3vPw8MucOTbVnjbW8UuCtejw@mail.gmail.com/ Signed-off-by: Tony Ambardar --- Documentation/admin-guide/sysctl/net.rst | 6 +- Documentation/networking/filter.rst | 6 +- arch/mips/Kconfig | 4 +- arch/mips/net/Makefile | 8 +- arch/mips/net/ebpf_jit_comp32.c | 1398 ++++++++++++++++++++++ arch/mips/net/ebpf_jit_core.c | 20 +- 6 files changed, 1426 insertions(+), 16 deletions(-) create mode 100644 arch/mips/net/ebpf_jit_comp32.c diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst index 4150f74c521a..099e3efbf38e 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -66,14 +66,16 @@ two flavors of JITs, the newer eBPF JIT currently supported on: - ppc64 - ppc32 - sparc64 - - mips64 + - mips64 (R2+) + - mips32 (R2+) - s390x - riscv64 - riscv32 And the older cBPF JIT supported on the following archs: - - mips + - mips64 (R1) + - mips32 (R1) - sparc eBPF JITs are a superset of cBPF JITs, meaning the kernel will diff --git a/Documentation/networking/filter.rst b/Documentation/networking/filter.rst index 3e2221f4abe4..31101411da0e 100644 --- a/Documentation/networking/filter.rst +++ b/Documentation/networking/filter.rst @@ -637,9 +637,9 @@ skb pointer). All constraints and restrictions from bpf_check_classic() apply before a conversion to the new layout is being done behind the scenes! Currently, the classic BPF format is being used for JITing on most -32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64, -sparc64, arm32, riscv64, riscv32 perform JIT compilation from eBPF -instruction set. +32-bit architectures, whereas x86-64, aarch64, s390x, powerpc64, sparc64, +mips64, riscv64, arm32, riscv32, and mips32 perform JIT compilation from +eBPF instruction set. Some core changes of the new internal format: diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index ed51970c08e7..d096d2332fe4 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -55,7 +55,7 @@ config MIPS select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE if CPU_SUPPORTS_HUGEPAGES select HAVE_ASM_MODVERSIONS - select HAVE_CBPF_JIT if !64BIT && !CPU_MICROMIPS + select HAVE_CBPF_JIT if !CPU_MICROMIPS && TARGET_ISA_REV < 2 select HAVE_CONTEXT_TRACKING select HAVE_TIF_NOHZ select HAVE_C_RECORDMCOUNT @@ -63,7 +63,7 @@ config MIPS select HAVE_DEBUG_STACKOVERFLOW select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE - select HAVE_EBPF_JIT if 64BIT && !CPU_MICROMIPS && TARGET_ISA_REV >= 2 + select HAVE_EBPF_JIT if !CPU_MICROMIPS && TARGET_ISA_REV >= 2 select HAVE_EXIT_THREAD select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD diff --git a/arch/mips/net/Makefile b/arch/mips/net/Makefile index de42f4a4db56..5f804bc54629 100644 --- a/arch/mips/net/Makefile +++ b/arch/mips/net/Makefile @@ -2,4 +2,10 @@ # MIPS networking code obj-$(CONFIG_MIPS_CBPF_JIT) += bpf_jit.o bpf_jit_asm.o -obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit_core.o ebpf_jit_comp64.o + +obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit_core.o +ifeq ($(CONFIG_CPU_MIPS64),y) + obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit_comp64.o +else + obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit_comp32.o +endif diff --git a/arch/mips/net/ebpf_jit_comp32.c b/arch/mips/net/ebpf_jit_comp32.c new file mode 100644 index 000000000000..13623224f78e --- /dev/null +++ b/arch/mips/net/ebpf_jit_comp32.c @@ -0,0 +1,1398 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Just-In-Time compiler for eBPF filters on MIPS32/MIPS64 + * Copyright (c) 2021 Tony Ambardar + * + * Based on code from: + * + * Copyright (c) 2017 Cavium, Inc. + * Author: David Daney + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras + */ + +#include +#include +#include + +#include "ebpf_jit.h" + +static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, + int idx) +{ + int dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + int upper_bound, lower_bound, shamt; + int imm = insn->imm; + + if (dst < 0) + return dst; + + switch (BPF_OP(insn->code)) { + case BPF_MOV: + case BPF_ADD: + upper_bound = S16_MAX; + lower_bound = S16_MIN; + break; + case BPF_SUB: + upper_bound = -(int)S16_MIN; + lower_bound = -(int)S16_MAX; + break; + case BPF_AND: + case BPF_OR: + case BPF_XOR: + upper_bound = 0xffff; + lower_bound = 0; + break; + case BPF_RSH: + case BPF_LSH: + case BPF_ARSH: + /* Shift amounts are truncated, no need for bounds */ + upper_bound = S32_MAX; + lower_bound = S32_MIN; + break; + default: + return -EINVAL; + } + + /* + * Immediate move clobbers the register, so no sign/zero + * extension needed. + */ + if (lower_bound <= imm && imm <= upper_bound) { + /* single insn immediate case */ + switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { + case BPF_ALU64 | BPF_MOV: + emit_instr(ctx, addiu, LO(dst), MIPS_R_ZERO, imm); + if (imm < 0) + gen_sext_insn(dst, ctx); + else + gen_zext_insn(dst, true, ctx); + break; + case BPF_ALU | BPF_MOV: + emit_instr(ctx, addiu, LO(dst), MIPS_R_ZERO, imm); + break; + case BPF_ALU64 | BPF_AND: + if (imm >= 0) + gen_zext_insn(dst, true, ctx); + fallthrough; + case BPF_ALU | BPF_AND: + emit_instr(ctx, andi, LO(dst), LO(dst), imm); + break; + case BPF_ALU64 | BPF_OR: + if (imm < 0) + emit_instr(ctx, nor, HI(dst), + MIPS_R_ZERO, MIPS_R_ZERO); + fallthrough; + case BPF_ALU | BPF_OR: + emit_instr(ctx, ori, LO(dst), LO(dst), imm); + break; + case BPF_ALU64 | BPF_XOR: + if (imm < 0) + emit_instr(ctx, nor, HI(dst), + HI(dst), MIPS_R_ZERO); + fallthrough; + case BPF_ALU | BPF_XOR: + emit_instr(ctx, xori, LO(dst), LO(dst), imm); + break; + case BPF_ALU64 | BPF_ADD: + emit_instr(ctx, addiu, LO(dst), LO(dst), imm); + if (imm < 0) + emit_instr(ctx, addiu, HI(dst), HI(dst), -1); + emit_instr(ctx, sltiu, MIPS_R_AT, LO(dst), imm); + emit_instr(ctx, addu, HI(dst), HI(dst), MIPS_R_AT); + break; + case BPF_ALU64 | BPF_SUB: + emit_instr(ctx, addiu, MIPS_R_AT, LO(dst), -imm); + if (imm < 0) + emit_instr(ctx, addiu, HI(dst), HI(dst), 1); + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), MIPS_R_AT); + emit_instr(ctx, subu, HI(dst), HI(dst), MIPS_R_AT); + emit_instr(ctx, addiu, LO(dst), LO(dst), -imm); + break; + case BPF_ALU64 | BPF_ARSH: + shamt = imm & 0x3f; + if (shamt >= 32) { + emit_instr(ctx, sra, LO(dst), + HI(dst), shamt - 32); + emit_instr(ctx, sra, HI(dst), HI(dst), 31); + } else if (shamt > 0) { + emit_instr(ctx, srl, LO(dst), LO(dst), shamt); + emit_instr(ctx, ins, LO(dst), HI(dst), + 32 - shamt, shamt); + emit_instr(ctx, sra, HI(dst), HI(dst), shamt); + } + break; + case BPF_ALU64 | BPF_RSH: + shamt = imm & 0x3f; + if (shamt >= 32) { + emit_instr(ctx, srl, LO(dst), + HI(dst), shamt - 32); + emit_instr(ctx, and, HI(dst), + HI(dst), MIPS_R_ZERO); + } else if (shamt > 0) { + emit_instr(ctx, srl, LO(dst), LO(dst), shamt); + emit_instr(ctx, ins, LO(dst), HI(dst), + 32 - shamt, shamt); + emit_instr(ctx, srl, HI(dst), HI(dst), shamt); + } + break; + case BPF_ALU64 | BPF_LSH: + shamt = imm & 0x3f; + if (shamt >= 32) { + emit_instr(ctx, sll, HI(dst), + LO(dst), shamt - 32); + emit_instr(ctx, and, LO(dst), + LO(dst), MIPS_R_ZERO); + } else if (shamt > 0) { + emit_instr(ctx, srl, MIPS_R_AT, + LO(dst), 32 - shamt); + emit_instr(ctx, sll, HI(dst), HI(dst), shamt); + emit_instr(ctx, sll, LO(dst), LO(dst), shamt); + emit_instr(ctx, or, HI(dst), + HI(dst), MIPS_R_AT); + } + break; + case BPF_ALU | BPF_RSH: + emit_instr(ctx, srl, LO(dst), LO(dst), imm & 0x1f); + break; + case BPF_ALU | BPF_LSH: + emit_instr(ctx, sll, LO(dst), LO(dst), imm & 0x1f); + break; + case BPF_ALU | BPF_ARSH: + emit_instr(ctx, sra, LO(dst), LO(dst), imm & 0x1f); + break; + case BPF_ALU | BPF_ADD: + emit_instr(ctx, addiu, LO(dst), LO(dst), imm); + break; + case BPF_ALU | BPF_SUB: + emit_instr(ctx, addiu, LO(dst), LO(dst), -imm); + break; + default: + return -EINVAL; + } + } else { + /* multi insn immediate case */ + if (BPF_OP(insn->code) == BPF_MOV) { + gen_imm_to_reg(insn, LO(dst), ctx); + if (BPF_CLASS(insn->code) == BPF_ALU64) { + if (imm < 0) + gen_sext_insn(dst, ctx); + else + gen_zext_insn(dst, true, ctx); + } + } else { + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) { + case BPF_ALU64 | BPF_AND: + if (imm >= 0) + gen_zext_insn(dst, true, ctx); + fallthrough; + case BPF_ALU | BPF_AND: + emit_instr(ctx, and, LO(dst), LO(dst), + MIPS_R_AT); + break; + case BPF_ALU64 | BPF_OR: + if (imm < 0) + emit_instr(ctx, nor, HI(dst), + MIPS_R_ZERO, MIPS_R_ZERO); + fallthrough; + case BPF_ALU | BPF_OR: + emit_instr(ctx, or, LO(dst), LO(dst), + MIPS_R_AT); + break; + case BPF_ALU64 | BPF_XOR: + if (imm < 0) + emit_instr(ctx, nor, HI(dst), + HI(dst), MIPS_R_ZERO); + fallthrough; + case BPF_ALU | BPF_XOR: + emit_instr(ctx, xor, LO(dst), LO(dst), + MIPS_R_AT); + break; + case BPF_ALU64 | BPF_ADD: + emit_instr(ctx, addu, LO(dst), + LO(dst), MIPS_R_AT); + if (imm < 0) + emit_instr(ctx, addiu, HI(dst), HI(dst), -1); + emit_instr(ctx, sltu, MIPS_R_AT, + LO(dst), MIPS_R_AT); + emit_instr(ctx, addu, HI(dst), + HI(dst), MIPS_R_AT); + break; + case BPF_ALU64 | BPF_SUB: + emit_instr(ctx, subu, LO(dst), + LO(dst), MIPS_R_AT); + if (imm < 0) + emit_instr(ctx, addiu, HI(dst), HI(dst), 1); + emit_instr(ctx, sltu, MIPS_R_AT, + MIPS_R_AT, LO(dst)); + emit_instr(ctx, subu, HI(dst), + HI(dst), MIPS_R_AT); + break; + case BPF_ALU | BPF_ADD: + emit_instr(ctx, addu, LO(dst), LO(dst), + MIPS_R_AT); + break; + case BPF_ALU | BPF_SUB: + emit_instr(ctx, subu, LO(dst), LO(dst), + MIPS_R_AT); + break; + default: + return -EINVAL; + } + } + } + + return 0; +} + +/* + * Implement 64-bit BPF div/mod insns on 32-bit systems by calling the + * equivalent built-in kernel function. The function args may be mixed + * 64/32-bit types, unlike the uniform u64 args of BPF kernel helpers. + * Func proto: u64 div64_u64_rem(u64 dividend, u64 divisor, u64 *remainder) + */ +static int emit_bpf_divmod64(struct jit_ctx *ctx, const struct bpf_insn *insn) +{ + const int bpf_src = BPF_SRC(insn->code); + const int bpf_op = BPF_OP(insn->code); + int rem_off, arg_off; + int src, dst, tmp; + u32 func_addr; + + ctx->flags |= EBPF_SAVE_RA; + + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + + if (bpf_src == BPF_X) { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0) + return -EINVAL; + /* + * Use MIPS_R_T8 as temp reg pair to avoid target + * of dst from clobbering src. + */ + if (src == MIPS_R_A0) { + tmp = MIPS_R_T8; + emit_instr(ctx, move, LO(tmp), LO(src)); + emit_instr(ctx, move, HI(tmp), HI(src)); + src = tmp; + } + } + + /* Save caller registers */ + emit_caller_save(ctx); + /* Push O32 stack, aligned space for u64, u64, u64 *, u64 */ + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -32); + + func_addr = (u32) &div64_u64_rem; + /* Move u64 dst to arg 1 as needed */ + if (dst != MIPS_R_A0) { + emit_instr(ctx, move, LO(MIPS_R_A0), LO(dst)); + emit_instr(ctx, move, HI(MIPS_R_A0), HI(dst)); + } + /* Load imm or move u64 src to arg 2 as needed */ + if (bpf_src == BPF_K) { + gen_imm_to_reg(insn, LO(MIPS_R_A2), ctx); + gen_sext_insn(MIPS_R_A2, ctx); + } else if (src != MIPS_R_A2) { /* BPF_X */ + emit_instr(ctx, move, LO(MIPS_R_A2), LO(src)); + emit_instr(ctx, move, HI(MIPS_R_A2), HI(src)); + } + /* Set up stack arg 3 as ptr to u64 remainder on stack */ + arg_off = 16; + rem_off = 24; + emit_instr(ctx, addiu, MIPS_R_AT, MIPS_R_SP, rem_off); + emit_instr(ctx, sw, MIPS_R_AT, arg_off, MIPS_R_SP); + + emit_const_to_reg(ctx, MIPS_R_T9, func_addr); + emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); + /* Delay slot */ + emit_instr(ctx, nop); + + /* Move return value to dst as needed */ + switch (bpf_op) { + case BPF_DIV: + /* Quotient in MIPS_R_V0 reg pair */ + if (dst != MIPS_R_V0) { + emit_instr(ctx, move, LO(dst), LO(MIPS_R_V0)); + emit_instr(ctx, move, HI(dst), HI(MIPS_R_V0)); + } + break; + case BPF_MOD: + /* Remainder on stack */ + emit_instr(ctx, lw, LO(dst), OFFLO(rem_off), MIPS_R_SP); + emit_instr(ctx, lw, HI(dst), OFFHI(rem_off), MIPS_R_SP); + break; + } + + /* Pop O32 call stack */ + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, 32); + /* Restore all caller registers except call return value*/ + emit_caller_restore(ctx, insn->dst_reg); + + return 0; +} + +/* + * Implement 64-bit BPF atomic insns on 32-bit systems by calling the + * equivalent built-in kernel function. The function args may be mixed + * 64/32-bit types, unlike the uniform u64 args of BPF kernel helpers. + */ +static int emit_bpf_atomic64(struct jit_ctx *ctx, const struct bpf_insn *insn) +{ + /* + * Since CMPXCHG uses R0 implicitly, outside of a passed + * bpf_insn, we fake a lookup to get the MIPS base reg. + */ + const struct bpf_insn r0_insn = {.src_reg = BPF_REG_0}; + const int r0 = ebpf_to_mips_reg(ctx, &r0_insn, + REG_SRC_NO_FP); + int mem_off, arg_off, stack_adj; + int src, dst, fetch; + u32 func_addr; + + + ctx->flags |= EBPF_SAVE_RA; + + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + + /* Save caller registers */ + emit_caller_save(ctx); + + /* + * Push O32 ABI stack, noting CMPXCHG requires 1 u64 arg passed + * on the stack. + */ + stack_adj = insn->imm == BPF_CMPXCHG ? 24 : 16; + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -stack_adj); + + switch (insn->imm) { + case BPF_AND | BPF_FETCH: + /* void atomic64_fetch_and(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_fetch_and; + goto args_common; + case BPF_OR | BPF_FETCH: + /* void atomic64_fetch_or(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_fetch_or; + goto args_common; + case BPF_XOR | BPF_FETCH: + /* void atomic64_fetch_xor(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_fetch_xor; + goto args_common; + case BPF_ADD | BPF_FETCH: + /* void atomic64_fetch_add(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_fetch_add; + goto args_common; + case BPF_AND: + /* void atomic64_and(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_and; + goto args_common; + case BPF_OR: + /* void atomic64_or(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_or; + goto args_common; + case BPF_XOR: + /* void atomic64_xor(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_xor; + goto args_common; + case BPF_ADD: + /* void atomic64_add(s64 a, atomic64_t *v) */ + func_addr = (u32) &atomic64_add; +args_common: + fetch = src; + /* Set up dst ptr in tmp reg if conflicting */ + if (dst == MIPS_R_A0) { + emit_instr(ctx, move, LO(MIPS_R_T8), LO(dst)); + dst = MIPS_R_T8; + } + /* Move s64 src to arg 1 as needed */ + if (src != MIPS_R_A0) { + emit_instr(ctx, move, LO(MIPS_R_A0), LO(src)); + emit_instr(ctx, move, HI(MIPS_R_A0), HI(src)); + } + /* Set up dst ptr in arg 2 base register*/ + emit_instr(ctx, addiu, MIPS_R_A2, LO(dst), mem_off); + break; + case BPF_XCHG: + /* s64 atomic64_xchg(atomic64_t *v, s64 i) */ + func_addr = (u32) &atomic64_xchg; + fetch = src; + /* Set up dst ptr in tmp reg if conflicting */ + if (dst == MIPS_R_A2) { + emit_instr(ctx, move, LO(MIPS_R_T8), LO(dst)); + dst = MIPS_R_T8; + } + emit_instr(ctx, addiu, MIPS_R_AT, LO(dst), mem_off); + /* Move s64 src to arg 2 as needed */ + if (src != MIPS_R_A2) { + emit_instr(ctx, move, LO(MIPS_R_A2), LO(src)); + emit_instr(ctx, move, HI(MIPS_R_A2), HI(src)); + } + /* Set up dst ptr in arg 1 base register*/ + emit_instr(ctx, addiu, MIPS_R_A0, LO(dst), mem_off); + break; + case BPF_CMPXCHG: + /* s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) */ + func_addr = (u32) &atomic64_cmpxchg; + fetch = r0; + /* Move s64 src to arg 3 on stack */ + arg_off = 16; + emit_instr(ctx, sw, LO(src), OFFLO(arg_off), MIPS_R_SP); + emit_instr(ctx, sw, HI(src), OFFHI(arg_off), MIPS_R_SP); + /* Set up dst ptr in arg 1 base register*/ + emit_instr(ctx, addiu, MIPS_R_A0, LO(dst), mem_off); + /* Move s64 R0 to arg 2 */ + emit_instr(ctx, move, LO(MIPS_R_A2), LO(r0)); + emit_instr(ctx, move, HI(MIPS_R_A2), HI(r0)); + break; + default: + pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); + return -EINVAL; + } + + emit_const_to_reg(ctx, MIPS_R_T9, func_addr); + emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); + /* Delay slot */ + emit_instr(ctx, nop); + + /* Pop O32 stack */ + emit_instr(ctx, addiu, MIPS_R_SP, MIPS_R_SP, stack_adj); + + if (insn->imm & BPF_FETCH) { + /* Set up returned value */ + if (fetch != MIPS_R_V0) { + emit_instr(ctx, move, LO(fetch), LO(MIPS_R_V0)); + emit_instr(ctx, move, HI(fetch), HI(MIPS_R_V0)); + } + /* Restore all caller registers except one fetched */ + if (insn->imm == BPF_CMPXCHG) + emit_caller_restore(ctx, BPF_REG_0); + else + emit_caller_restore(ctx, insn->src_reg); + } else { + /* Restore all caller registers since none clobbered */ + emit_caller_restore(ctx, BPF_REG_FP); + } + + return 0; +} + +/* Returns the number of insn slots consumed. */ +int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, + int this_idx, int exit_idx) +{ + /* + * Since CMPXCHG uses R0 implicitly, outside of a passed + * bpf_insn, we fake a lookup to get the MIPS base reg. + */ + const struct bpf_insn r0_insn = {.src_reg = BPF_REG_0}; + const int r0 = ebpf_to_mips_reg(ctx, &r0_insn, + REG_SRC_NO_FP); + const int bpf_class = BPF_CLASS(insn->code); + const int bpf_size = BPF_SIZE(insn->code); + const int bpf_src = BPF_SRC(insn->code); + const int bpf_op = BPF_OP(insn->code); + int src, dst, r, mem_off, b_off; + bool need_swap, cmp_eq; + unsigned int target = 0; + + switch (insn->code) { + case BPF_ALU64 | BPF_ADD | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_SUB | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_LSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_RSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_ARSH | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_XOR | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_MOV | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_OR | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_AND | BPF_K: /* ALU64_IMM */ + case BPF_ALU | BPF_MOV | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_ADD | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_SUB | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_OR | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_AND | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_LSH | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_RSH | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_XOR | BPF_K: /* ALU32_IMM */ + case BPF_ALU | BPF_ARSH | BPF_K: /* ALU32_IMM */ + r = gen_imm_insn(insn, ctx, this_idx); + if (r < 0) + return r; + break; + case BPF_ALU64 | BPF_MUL | BPF_K: /* ALU64_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (insn->imm == 1) /* Mult by 1 is a nop */ + break; + src = MIPS_R_T8; /* Use tmp reg pair for imm */ + gen_imm_to_reg(insn, LO(src), ctx); + emit_instr(ctx, sra, HI(src), LO(src), 31); + goto case_alu64_mul_x; + + case BPF_ALU64 | BPF_NEG | BPF_K: /* ALU64_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + emit_instr(ctx, subu, LO(dst), MIPS_R_ZERO, LO(dst)); + emit_instr(ctx, subu, HI(dst), MIPS_R_ZERO, HI(dst)); + emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_ZERO, LO(dst)); + emit_instr(ctx, subu, HI(dst), HI(dst), MIPS_R_AT); + break; + case BPF_ALU | BPF_MUL | BPF_K: /* ALU_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (insn->imm == 1) /* Mult by 1 is a nop */ + break; + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, mulu, LO(dst), LO(dst), MIPS_R_AT); + } else { + emit_instr(ctx, multu, LO(dst), MIPS_R_AT); + emit_instr(ctx, mflo, LO(dst)); + } + break; + case BPF_ALU | BPF_NEG | BPF_K: /* ALU_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + emit_instr(ctx, subu, LO(dst), MIPS_R_ZERO, LO(dst)); + break; + case BPF_ALU | BPF_DIV | BPF_K: /* ALU_IMM */ + case BPF_ALU | BPF_MOD | BPF_K: /* ALU_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + if (insn->imm == 1) { + /* div by 1 is a nop, mod by 1 is zero */ + if (bpf_op == BPF_MOD) + emit_instr(ctx, move, LO(dst), MIPS_R_ZERO); + break; + } + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, divu_r6, LO(dst), + LO(dst), MIPS_R_AT); + else + emit_instr(ctx, modu, LO(dst), + LO(dst), MIPS_R_AT); + break; + } + emit_instr(ctx, divu, LO(dst), MIPS_R_AT); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, LO(dst)); + else + emit_instr(ctx, mfhi, LO(dst)); + break; + case BPF_ALU64 | BPF_DIV | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_MOD | BPF_K: /* ALU64_IMM */ + case BPF_ALU64 | BPF_DIV | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_MOD | BPF_X: /* ALU64_REG */ + r = emit_bpf_divmod64(ctx, insn); + if (r < 0) + return r; + break; + case BPF_ALU64 | BPF_MUL | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_ADD | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_SUB | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_MOV | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_XOR | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_OR | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_AND | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_LSH | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_RSH | BPF_X: /* ALU64_REG */ + case BPF_ALU64 | BPF_ARSH | BPF_X: /* ALU64_REG */ + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (src < 0 || dst < 0) + return -EINVAL; + switch (bpf_op) { + case BPF_MOV: + emit_instr(ctx, move, LO(dst), LO(src)); + emit_instr(ctx, move, HI(dst), HI(src)); + break; + case BPF_ADD: + emit_instr(ctx, addu, HI(dst), HI(dst), HI(src)); + emit_instr(ctx, addu, MIPS_R_AT, LO(dst), LO(src)); + emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_AT, LO(dst)); + emit_instr(ctx, addu, HI(dst), HI(dst), MIPS_R_AT); + emit_instr(ctx, addu, LO(dst), LO(dst), LO(src)); + break; + case BPF_SUB: + emit_instr(ctx, subu, HI(dst), HI(dst), HI(src)); + emit_instr(ctx, subu, MIPS_R_AT, LO(dst), LO(src)); + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), MIPS_R_AT); + emit_instr(ctx, subu, HI(dst), HI(dst), MIPS_R_AT); + emit_instr(ctx, subu, LO(dst), LO(dst), LO(src)); + break; + case BPF_XOR: + emit_instr(ctx, xor, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, xor, HI(dst), HI(dst), HI(src)); + break; + case BPF_OR: + emit_instr(ctx, or, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, or, HI(dst), HI(dst), HI(src)); + break; + case BPF_AND: + emit_instr(ctx, and, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, and, HI(dst), HI(dst), HI(src)); + break; + case BPF_MUL: +case_alu64_mul_x: + emit_instr(ctx, mul, HI(dst), HI(dst), LO(src)); + emit_instr(ctx, mul, MIPS_R_AT, LO(dst), HI(src)); + emit_instr(ctx, addu, HI(dst), HI(dst), MIPS_R_AT); + if (MIPS_ISA_REV >= 6) { + emit_instr(ctx, muhu, MIPS_R_AT, LO(dst), LO(src)); + emit_instr(ctx, mul, LO(dst), LO(dst), LO(src)); + } else { + emit_instr(ctx, multu, LO(dst), LO(src)); + emit_instr(ctx, mfhi, MIPS_R_AT); + emit_instr(ctx, mflo, LO(dst)); + } + emit_instr(ctx, addu, HI(dst), HI(dst), MIPS_R_AT); + break; + case BPF_LSH: + emit_instr(ctx, beqz, LO(src), 11 * 4); + emit_instr(ctx, addiu, MIPS_R_AT, LO(src), -32); + emit_instr(ctx, bltz, MIPS_R_AT, 4 * 4); + emit_instr(ctx, nop); + emit_instr(ctx, sllv, HI(dst), LO(dst), MIPS_R_AT); + emit_instr(ctx, and, LO(dst), LO(dst), MIPS_R_ZERO); + emit_instr(ctx, b, 5 * 4); + emit_instr(ctx, subu, MIPS_R_AT, MIPS_R_ZERO, MIPS_R_AT); + emit_instr(ctx, srlv, MIPS_R_AT, LO(dst), MIPS_R_AT); + emit_instr(ctx, sllv, HI(dst), HI(dst), LO(src)); + emit_instr(ctx, sllv, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, or, HI(dst), HI(dst), MIPS_R_AT); + break; + case BPF_RSH: + emit_instr(ctx, beqz, LO(src), 11 * 4); + emit_instr(ctx, addiu, MIPS_R_AT, LO(src), -32); + emit_instr(ctx, bltz, MIPS_R_AT, 4 * 4); + emit_instr(ctx, nop); + emit_instr(ctx, srlv, LO(dst), HI(dst), MIPS_R_AT); + emit_instr(ctx, and, HI(dst), HI(dst), MIPS_R_ZERO); + emit_instr(ctx, b, 5 * 4); + emit_instr(ctx, subu, MIPS_R_AT, MIPS_R_ZERO, MIPS_R_AT); + emit_instr(ctx, sllv, MIPS_R_AT, HI(dst), MIPS_R_AT); + emit_instr(ctx, srlv, HI(dst), HI(dst), LO(src)); + emit_instr(ctx, srlv, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, or, LO(dst), LO(dst), MIPS_R_AT); + break; + case BPF_ARSH: + emit_instr(ctx, beqz, LO(src), 11 * 4); + emit_instr(ctx, addiu, MIPS_R_AT, LO(src), -32); + emit_instr(ctx, bltz, MIPS_R_AT, 4 * 4); + emit_instr(ctx, nop); + emit_instr(ctx, srav, LO(dst), HI(dst), MIPS_R_AT); + emit_instr(ctx, sra, HI(dst), HI(dst), 31); + emit_instr(ctx, b, 5 * 4); + emit_instr(ctx, subu, MIPS_R_AT, MIPS_R_ZERO, MIPS_R_AT); + emit_instr(ctx, sllv, MIPS_R_AT, HI(dst), MIPS_R_AT); + emit_instr(ctx, srav, HI(dst), HI(dst), LO(src)); + emit_instr(ctx, srlv, LO(dst), LO(dst), LO(src)); + emit_instr(ctx, or, LO(dst), LO(dst), MIPS_R_AT); + break; + default: + pr_err("ALU64_REG NOT HANDLED\n"); + return -EINVAL; + } + break; + case BPF_ALU | BPF_MOV | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_ADD | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_SUB | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_XOR | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_OR | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_AND | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_MUL | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_DIV | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_MOD | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_LSH | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_RSH | BPF_X: /* ALU_REG */ + case BPF_ALU | BPF_ARSH | BPF_X: /* ALU_REG */ + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (src < 0 || dst < 0) + return -EINVAL; + /* Special BPF_MOV zext insn from verifier. */ + if (insn_is_zext(insn)) { + gen_zext_insn(dst, true, ctx); + break; + } + switch (bpf_op) { + case BPF_MOV: + emit_instr(ctx, move, LO(dst), LO(src)); + break; + case BPF_ADD: + emit_instr(ctx, addu, LO(dst), LO(dst), LO(src)); + break; + case BPF_SUB: + emit_instr(ctx, subu, LO(dst), LO(dst), LO(src)); + break; + case BPF_XOR: + emit_instr(ctx, xor, LO(dst), LO(dst), LO(src)); + break; + case BPF_OR: + emit_instr(ctx, or, LO(dst), LO(dst), LO(src)); + break; + case BPF_AND: + emit_instr(ctx, and, LO(dst), LO(dst), LO(src)); + break; + case BPF_MUL: + emit_instr(ctx, mul, LO(dst), LO(dst), LO(src)); + break; + case BPF_DIV: + case BPF_MOD: + if (MIPS_ISA_REV >= 6) { + if (bpf_op == BPF_DIV) + emit_instr(ctx, divu_r6, LO(dst), + LO(dst), LO(src)); + else + emit_instr(ctx, modu, LO(dst), + LO(dst), LO(src)); + break; + } + emit_instr(ctx, divu, LO(dst), LO(src)); + if (bpf_op == BPF_DIV) + emit_instr(ctx, mflo, LO(dst)); + else + emit_instr(ctx, mfhi, LO(dst)); + break; + case BPF_LSH: + emit_instr(ctx, sllv, LO(dst), LO(dst), LO(src)); + break; + case BPF_RSH: + emit_instr(ctx, srlv, LO(dst), LO(dst), LO(src)); + break; + case BPF_ARSH: + emit_instr(ctx, srav, LO(dst), LO(dst), LO(src)); + break; + default: + pr_err("ALU_REG NOT HANDLED\n"); + return -EINVAL; + } + break; + case BPF_JMP | BPF_EXIT: + if (this_idx + 1 < exit_idx) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, exit_idx); + if (target == (unsigned int)-1) + return -E2BIG; + emit_instr(ctx, j, target); + } else { + emit_instr(ctx, b, b_off); + } + emit_instr(ctx, nop); + } + break; + case BPF_JMP32 | BPF_JSLT | BPF_X: + case BPF_JMP32 | BPF_JSLE | BPF_X: + case BPF_JMP32 | BPF_JSGT | BPF_X: + case BPF_JMP32 | BPF_JSGE | BPF_X: + case BPF_JMP32 | BPF_JSGT | BPF_K: + case BPF_JMP32 | BPF_JSGE | BPF_K: + case BPF_JMP32 | BPF_JSLT | BPF_K: + case BPF_JMP32 | BPF_JSLE | BPF_K: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + + if (bpf_src == BPF_X) { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + if (src < 0) + return -EINVAL; + } else if (insn->imm == 0) { /* and BPF_K */ + src = MIPS_R_ZERO; + } else { + src = MIPS_R_T8; + gen_imm_to_reg(insn, LO(src), ctx); + } + + cmp_eq = bpf_op == BPF_JSLE || bpf_op == BPF_JSGE; + switch (bpf_op) { + case BPF_JSGE: + emit_instr(ctx, slt, MIPS_R_AT, LO(dst), LO(src)); + break; + case BPF_JSLT: + emit_instr(ctx, slt, MIPS_R_AT, LO(dst), LO(src)); + break; + case BPF_JSGT: + emit_instr(ctx, slt, MIPS_R_AT, LO(src), LO(dst)); + break; + case BPF_JSLE: + emit_instr(ctx, slt, MIPS_R_AT, LO(src), LO(dst)); + break; + } + + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP | BPF_JSLT | BPF_X: + case BPF_JMP | BPF_JSLE | BPF_X: + case BPF_JMP | BPF_JSGT | BPF_X: + case BPF_JMP | BPF_JSGE | BPF_X: + case BPF_JMP | BPF_JSGT | BPF_K: + case BPF_JMP | BPF_JSGE | BPF_K: + case BPF_JMP | BPF_JSLT | BPF_K: + case BPF_JMP | BPF_JSLE | BPF_K: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + + if (bpf_src == BPF_X) { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + if (src < 0) + return -EINVAL; + } else if (insn->imm == 0) { /* and BPF_K */ + src = MIPS_R_ZERO; + } else { + src = MIPS_R_T8; + gen_imm_to_reg(insn, LO(src), ctx); + if (insn->imm < 0) + gen_sext_insn(src, ctx); + else + gen_zext_insn(src, true, ctx); + } + + cmp_eq = bpf_op == BPF_JSGT || bpf_op == BPF_JSGE; + + if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLE) { + /* Check dst <= src */ + emit_instr(ctx, bne, HI(dst), HI(src), 4 * 4); + /* Delay slot */ + emit_instr(ctx, slt, MIPS_R_AT, HI(dst), HI(src)); + emit_instr(ctx, bne, LO(dst), LO(src), 2 * 4); + /* Delay slot */ + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + emit_instr(ctx, nor, MIPS_R_AT, MIPS_R_ZERO, MIPS_R_AT); + } else { + /* Check dst < src */ + emit_instr(ctx, bne, HI(dst), HI(src), 2 * 4); + /* Delay slot */ + emit_instr(ctx, slt, MIPS_R_AT, HI(dst), HI(src)); + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + } + + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP | BPF_JLT | BPF_X: + case BPF_JMP | BPF_JLE | BPF_X: + case BPF_JMP | BPF_JGT | BPF_X: + case BPF_JMP | BPF_JGE | BPF_X: + case BPF_JMP | BPF_JGT | BPF_K: + case BPF_JMP | BPF_JGE | BPF_K: + case BPF_JMP | BPF_JLT | BPF_K: + case BPF_JMP | BPF_JLE | BPF_K: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + + if (bpf_src == BPF_X) { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + if (src < 0) + return -EINVAL; + } else if (insn->imm == 0) { /* and BPF_K */ + src = MIPS_R_ZERO; + } else { + src = MIPS_R_T8; + gen_imm_to_reg(insn, LO(src), ctx); + if (insn->imm < 0) + gen_sext_insn(src, ctx); + else + gen_zext_insn(src, true, ctx); + } + + cmp_eq = bpf_op == BPF_JGT || bpf_op == BPF_JGE; + + if (bpf_op == BPF_JGT || bpf_op == BPF_JLE) { + /* Check dst <= src */ + emit_instr(ctx, bne, HI(dst), HI(src), 4 * 4); + /* Delay slot */ + emit_instr(ctx, sltu, MIPS_R_AT, HI(dst), HI(src)); + emit_instr(ctx, bne, LO(dst), LO(src), 2 * 4); + /* Delay slot */ + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + emit_instr(ctx, nor, MIPS_R_AT, MIPS_R_ZERO, MIPS_R_AT); + } else { + /* Check dst < src */ + emit_instr(ctx, bne, HI(dst), HI(src), 2 * 4); + /* Delay slot */ + emit_instr(ctx, sltu, MIPS_R_AT, HI(dst), HI(src)); + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + } + + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP32 | BPF_JLT | BPF_X: + case BPF_JMP32 | BPF_JLE | BPF_X: + case BPF_JMP32 | BPF_JGT | BPF_X: + case BPF_JMP32 | BPF_JGE | BPF_X: + case BPF_JMP32 | BPF_JGT | BPF_K: + case BPF_JMP32 | BPF_JGE | BPF_K: + case BPF_JMP32 | BPF_JLT | BPF_K: + case BPF_JMP32 | BPF_JLE | BPF_K: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + + if (bpf_src == BPF_X) { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + if (src < 0) + return -EINVAL; + } else if (insn->imm == 0) { /* and BPF_K */ + src = MIPS_R_ZERO; + } else { + src = MIPS_R_T8; + gen_imm_to_reg(insn, LO(src), ctx); + } + + cmp_eq = bpf_op == BPF_JLE || bpf_op == BPF_JGE; + switch (bpf_op) { + case BPF_JGE: + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + break; + case BPF_JLT: + emit_instr(ctx, sltu, MIPS_R_AT, LO(dst), LO(src)); + break; + case BPF_JGT: + emit_instr(ctx, sltu, MIPS_R_AT, LO(src), LO(dst)); + break; + case BPF_JLE: + emit_instr(ctx, sltu, MIPS_R_AT, LO(src), LO(dst)); + break; + } + + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + goto jeq_common; + + case BPF_JMP | BPF_JEQ | BPF_X: /* JMP_REG */ + case BPF_JMP | BPF_JNE | BPF_X: + case BPF_JMP32 | BPF_JEQ | BPF_X: + case BPF_JMP32 | BPF_JNE | BPF_X: + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + + cmp_eq = (bpf_op == BPF_JEQ); + if (bpf_class == BPF_JMP) { + emit_instr(ctx, beq, HI(dst), HI(src), 2 * 4); + /* Delay slot */ + emit_instr(ctx, move, MIPS_R_AT, LO(src)); + /* Make low words unequal if high word unequal. */ + emit_instr(ctx, addu, MIPS_R_AT, LO(dst), MIPS_R_SP); + dst = LO(dst); + src = MIPS_R_AT; + } else { /* BPF_JMP32 */ + dst = LO(dst); + src = LO(src); + } + goto jeq_common; + + case BPF_JMP | BPF_JSET | BPF_X: /* JMP_REG */ + case BPF_JMP32 | BPF_JSET | BPF_X: + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (src < 0 || dst < 0) + return -EINVAL; + emit_instr(ctx, and, MIPS_R_AT, LO(dst), LO(src)); + if (bpf_class == BPF_JMP) { + emit_instr(ctx, and, MIPS_R_T8, HI(dst), HI(src)); + emit_instr(ctx, or, MIPS_R_AT, MIPS_R_AT, MIPS_R_T8); + } + cmp_eq = false; + dst = MIPS_R_AT; + src = MIPS_R_ZERO; +jeq_common: + /* + * If the next insn is EXIT and we are jumping arround + * only it, invert the sense of the compare and + * conditionally jump to the exit. Poor man's branch + * chaining. + */ + if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) { + b_off = b_imm(exit_idx, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, exit_idx); + if (target == (unsigned int)-1) + return -E2BIG; + cmp_eq = !cmp_eq; + b_off = 4 * 3; + if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { + ctx->offsets[this_idx] |= OFFSETS_B_CONV; + ctx->long_b_conversion = 1; + } + } + + if (cmp_eq) + emit_instr(ctx, bne, dst, src, b_off); + else + emit_instr(ctx, beq, dst, src, b_off); + emit_instr(ctx, nop); + if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { + emit_instr(ctx, j, target); + emit_instr(ctx, nop); + } + return 2; /* We consumed the exit. */ + } + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, this_idx + insn->off + 1); + if (target == (unsigned int)-1) + return -E2BIG; + cmp_eq = !cmp_eq; + b_off = 4 * 3; + if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { + ctx->offsets[this_idx] |= OFFSETS_B_CONV; + ctx->long_b_conversion = 1; + } + } + + if (cmp_eq) + emit_instr(ctx, beq, dst, src, b_off); + else + emit_instr(ctx, bne, dst, src, b_off); + emit_instr(ctx, nop); + if (ctx->offsets[this_idx] & OFFSETS_B_CONV) { + emit_instr(ctx, j, target); + emit_instr(ctx, nop); + } + break; + + case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */ + case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JEQ | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JNE | BPF_K: /* JMP_IMM */ + cmp_eq = (bpf_op == BPF_JEQ); + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return dst; + if (insn->imm == 0) { + src = MIPS_R_ZERO; + if (bpf_class == BPF_JMP32) { + dst = LO(dst); + } else { /* BPF_JMP */ + emit_instr(ctx, or, MIPS_R_AT, LO(dst), HI(dst)); + dst = MIPS_R_AT; + } + } else if (bpf_class == BPF_JMP32) { + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + src = MIPS_R_AT; + dst = LO(dst); + } else { /* BPF_JMP */ + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + /* If low words equal, check high word vs imm sign. */ + emit_instr(ctx, beq, LO(dst), MIPS_R_AT, 2 * 4); + emit_instr(ctx, nop); + /* Make high word signs unequal if low words unequal. */ + emit_instr(ctx, nor, MIPS_R_AT, MIPS_R_ZERO, HI(dst)); + emit_instr(ctx, sra, MIPS_R_AT, MIPS_R_AT, 31); + src = MIPS_R_AT; + dst = HI(dst); + } + goto jeq_common; + + case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */ + case BPF_JMP32 | BPF_JSET | BPF_K: /* JMP_IMM */ + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + emit_instr(ctx, and, MIPS_R_AT, LO(dst), MIPS_R_AT); + if (bpf_class == BPF_JMP && insn->imm < 0) + emit_instr(ctx, or, MIPS_R_AT, MIPS_R_AT, HI(dst)); + src = MIPS_R_AT; + dst = MIPS_R_ZERO; + cmp_eq = false; + goto jeq_common; + + case BPF_JMP | BPF_JA: + /* + * Prefer relative branch for easier debugging, but + * fall back if needed. + */ + b_off = b_imm(this_idx + insn->off + 1, ctx); + if (is_bad_offset(b_off)) { + target = j_target(ctx, this_idx + insn->off + 1); + if (target == (unsigned int)-1) + return -E2BIG; + emit_instr(ctx, j, target); + } else { + emit_instr(ctx, b, b_off); + } + emit_instr(ctx, nop); + break; + case BPF_LD | BPF_DW | BPF_IMM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; + gen_imm_to_reg(insn, LO(dst), ctx); + gen_imm_to_reg(insn+1, HI(dst), ctx); + return 2; /* Double slot insn */ + + case BPF_JMP | BPF_CALL: + emit_bpf_call(ctx, insn); + break; + case BPF_JMP | BPF_TAIL_CALL: + if (emit_bpf_tail_call(ctx, this_idx)) + return -EINVAL; + break; + + case BPF_ALU | BPF_END | BPF_FROM_BE: + case BPF_ALU | BPF_END | BPF_FROM_LE: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return dst; +#ifdef __BIG_ENDIAN + need_swap = (bpf_src == BPF_FROM_LE); +#else + need_swap = (bpf_src == BPF_FROM_BE); +#endif + if (insn->imm == 16) { + if (need_swap) + emit_instr(ctx, wsbh, LO(dst), LO(dst)); + emit_instr(ctx, andi, LO(dst), LO(dst), 0xffff); + } else if (insn->imm == 32) { + if (need_swap) { + emit_instr(ctx, wsbh, LO(dst), LO(dst)); + emit_instr(ctx, rotr, LO(dst), LO(dst), 16); + } + } else { /* 64-bit*/ + if (need_swap) { + emit_instr(ctx, wsbh, MIPS_R_AT, LO(dst)); + emit_instr(ctx, wsbh, LO(dst), HI(dst)); + emit_instr(ctx, rotr, HI(dst), MIPS_R_AT, 16); + emit_instr(ctx, rotr, LO(dst), LO(dst), 16); + } + } + break; + + case BPF_ST | BPF_NOSPEC: /* speculation barrier */ + break; + + case BPF_ST | BPF_B | BPF_MEM: + case BPF_ST | BPF_H | BPF_MEM: + case BPF_ST | BPF_W | BPF_MEM: + case BPF_ST | BPF_DW | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + if (dst < 0) + return -EINVAL; + mem_off = insn->off; + gen_imm_to_reg(insn, MIPS_R_AT, ctx); + + switch (bpf_size) { + case BPF_B: + emit_instr(ctx, sb, MIPS_R_AT, mem_off, LO(dst)); + break; + case BPF_H: + emit_instr(ctx, sh, MIPS_R_AT, mem_off, LO(dst)); + break; + case BPF_W: + emit_instr(ctx, sw, MIPS_R_AT, mem_off, LO(dst)); + break; + case BPF_DW: + /* Memory order == register order in pair */ + emit_instr(ctx, sw, MIPS_R_AT, OFFLO(mem_off), LO(dst)); + if (insn->imm < 0) { + emit_instr(ctx, nor, MIPS_R_AT, + MIPS_R_ZERO, MIPS_R_ZERO); + emit_instr(ctx, sw, MIPS_R_AT, + OFFHI(mem_off), LO(dst)); + } else { + emit_instr(ctx, sw, MIPS_R_ZERO, + OFFHI(mem_off), LO(dst)); + } + break; + } + break; + + case BPF_LDX | BPF_B | BPF_MEM: + case BPF_LDX | BPF_H | BPF_MEM: + case BPF_LDX | BPF_W | BPF_MEM: + case BPF_LDX | BPF_DW | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + + switch (bpf_size) { + case BPF_B: + emit_instr(ctx, lbu, LO(dst), mem_off, LO(src)); + break; + case BPF_H: + emit_instr(ctx, lhu, LO(dst), mem_off, LO(src)); + break; + case BPF_W: + emit_instr(ctx, lw, LO(dst), mem_off, LO(src)); + break; + case BPF_DW: + /* + * Careful: update HI(dst) first in case dst == src, + * since only LO(src) is the usable pointer. + */ + emit_instr(ctx, lw, HI(dst), OFFHI(mem_off), LO(src)); + emit_instr(ctx, lw, LO(dst), OFFLO(mem_off), LO(src)); + break; + } + break; + + case BPF_STX | BPF_DW | BPF_ATOMIC: + r = emit_bpf_atomic64(ctx, insn); + if (r < 0) + return r; + break; + case BPF_STX | BPF_W | BPF_ATOMIC: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + /* + * Drop reg pair scheme for more efficient temp register usage + * given BPF_W mode. + */ + dst = LO(dst); + src = LO(src); + /* + * If mem_off does not fit within the 9 bit ll/sc instruction + * immediate field, use a temp reg. + */ + if (MIPS_ISA_REV >= 6 && + (mem_off >= BIT(8) || mem_off < -BIT(8))) { + emit_instr(ctx, addiu, MIPS_R_T9, dst, mem_off); + mem_off = 0; + dst = MIPS_R_T9; + } + /* Track variable branch offset due to CMPXCHG. */ + b_off = ctx->idx; + emit_instr(ctx, ll, MIPS_R_AT, mem_off, dst); + switch (insn->imm) { + case BPF_AND | BPF_FETCH: + case BPF_AND: + emit_instr(ctx, and, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_OR | BPF_FETCH: + case BPF_OR: + emit_instr(ctx, or, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_XOR | BPF_FETCH: + case BPF_XOR: + emit_instr(ctx, xor, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_ADD | BPF_FETCH: + case BPF_ADD: + emit_instr(ctx, addu, MIPS_R_T8, MIPS_R_AT, src); + break; + case BPF_XCHG: + emit_instr(ctx, move, MIPS_R_T8, src); + break; + case BPF_CMPXCHG: + /* + * If R0 != old_val then break out of LL/SC loop + */ + emit_instr(ctx, bne, LO(r0), MIPS_R_AT, 4 * 4); + /* Delay slot */ + emit_instr(ctx, move, MIPS_R_T8, src); + /* Return old_val in R0 */ + src = LO(r0); + break; + default: + pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm); + return -EINVAL; + } + emit_instr(ctx, sc, MIPS_R_T8, mem_off, dst); + /* + * On failure back up to LL (calculate # insns) + */ + b_off = (b_off - ctx->idx - 1) * 4; + emit_instr(ctx, beqz, MIPS_R_AT, b_off); + emit_instr(ctx, nop); + /* + * Using fetch returns old value in src or R0 + */ + if (insn->imm & BPF_FETCH) + emit_instr(ctx, move, src, MIPS_R_AT); + break; + + case BPF_STX | BPF_DW | BPF_MEM: + case BPF_STX | BPF_B | BPF_MEM: + case BPF_STX | BPF_H | BPF_MEM: + case BPF_STX | BPF_W | BPF_MEM: + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_FP_OK); + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_FP_OK); + if (src < 0 || dst < 0) + return -EINVAL; + mem_off = insn->off; + + switch (bpf_size) { + case BPF_B: + emit_instr(ctx, sb, LO(src), mem_off, LO(dst)); + break; + case BPF_H: + emit_instr(ctx, sh, LO(src), mem_off, LO(dst)); + break; + case BPF_W: + emit_instr(ctx, sw, LO(src), mem_off, LO(dst)); + break; + case BPF_DW: + emit_instr(ctx, sw, HI(src), OFFHI(mem_off), LO(dst)); + emit_instr(ctx, sw, LO(src), OFFLO(mem_off), LO(dst)); + break; + } + break; + + default: + pr_err("NOT HANDLED %d - (%02x)\n", + this_idx, (unsigned int)insn->code); + return -EINVAL; + } + /* + * Handle zero-extension if the verifier is unable to patch and + * insert it's own special zext insns. + */ + if ((bpf_class == BPF_ALU && !(bpf_op == BPF_END && insn->imm == 64)) || + (bpf_class == BPF_LDX && bpf_size != BPF_DW)) { + dst = ebpf_to_mips_reg(ctx, insn, REG_DST_NO_FP); + if (dst < 0) + return -EINVAL; + gen_zext_insn(dst, false, ctx); + } + if (insn->code == (BPF_STX|BPF_ATOMIC|BPF_W) && insn->imm & BPF_FETCH) { + if (insn->imm == BPF_CMPXCHG) { + gen_zext_insn(r0, false, ctx); + } else { + src = ebpf_to_mips_reg(ctx, insn, REG_SRC_NO_FP); + if (src < 0) + return -EINVAL; + gen_zext_insn(src, false, ctx); + } + } + + return 1; +} + +/* Enable the verifier to insert zext insn for ALU32 ops as needed. */ +bool bpf_jit_needs_zext(void) +{ + return true; +} diff --git a/arch/mips/net/ebpf_jit_core.c b/arch/mips/net/ebpf_jit_core.c index 37b496f47ddb..6eac8150ce2a 100644 --- a/arch/mips/net/ebpf_jit_core.c +++ b/arch/mips/net/ebpf_jit_core.c @@ -637,7 +637,8 @@ static int build_int_body(struct jit_ctx *ctx) for (i = 0; i < prog->len; ) { insn = prog->insnsi + i; - if ((ctx->reg_val_types[i] & RVT_VISITED_MASK) == 0) { + if (is64bit() && (ctx->reg_val_types[i] & + RVT_VISITED_MASK) == 0) { /* dead instruction, don't emit it. */ i++; continue; @@ -1080,14 +1081,17 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) preempt_enable(); } - ctx->reg_val_types = kcalloc(prog->len + 1, - sizeof(*ctx->reg_val_types), - GFP_KERNEL); - if (!ctx->reg_val_types) - goto out_err; + /* Static analysis only used for MIPS64. */ + if (is64bit()) { + ctx->reg_val_types = kcalloc(prog->len + 1, + sizeof(*ctx->reg_val_types), + GFP_KERNEL); + if (!ctx->reg_val_types) + goto out_err; - if (reg_val_propagate(ctx)) - goto out_err; + if (reg_val_propagate(ctx)) + goto out_err; + } /* * First pass discovers used resources and instruction offsets